2024-03-28T21:49:07.482 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-28T21:49:07.483 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} --archive /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627081 --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627081/orig.config.yaml 2024-03-28T21:49:07.544 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-28T21:49:07.643 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627081 branch: squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7627081' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - slow metadata IO - SLOW_OPS - slow request sha1: 323d23dc228bbcb7f16f188fdc9c0dcb801844be subvols: create: 2 subvol_options: --namespace-isolated ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: 323d23dc228bbcb7f16f188fdc9c0dcb801844be kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - wsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: 55f5138942aa4866e49329c9103468889cec3f73 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 7214 sha1: 323d23dc228bbcb7f16f188fdc9c0dcb801844be sleep_before_teardown: 0 subset: 39/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_55f5138942aa4866e49329c9103468889cec3f73/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 55f5138942aa4866e49329c9103468889cec3f73 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi016.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBALRE86IiYBSvcxgSkbfx534/kRLCWStoyl6wAPDRl4nc3gdufJHPiscJ+bHK493R7gtQKDv2W7wkh+bP78Mi14= smithi053.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM44VrSbAkAdZ43rcJvDCkX44DcpMU2oZk2CR5TPqYqHHdYimKvUvbqpfps/UbdzGC+6zQbJKOyiFYYufqvMNu8= smithi063.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEkuF2zVDPcz+eNFgDlEeVqrx7lY4M0fPlPf9F4EV5kLZ4JP9oA3g8gk9DFuASMkR4P2mAXA8O8T1rStkYerqTM= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - workunit: clients: all: - suites/fsx.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-27_21:24:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-28T21:49:07.643 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_55f5138942aa4866e49329c9103468889cec3f73/qa; will attempt to use it 2024-03-28T21:49:07.644 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_55f5138942aa4866e49329c9103468889cec3f73/qa/tasks 2024-03-28T21:49:07.644 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-28T21:49:07.646 INFO:teuthology.task.internal:Checking packages... 2024-03-28T21:49:07.669 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '323d23dc228bbcb7f16f188fdc9c0dcb801844be' 2024-03-28T21:49:07.669 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-28T21:49:07.669 INFO:teuthology.packaging:ref: None 2024-03-28T21:49:07.669 INFO:teuthology.packaging:tag: None 2024-03-28T21:49:07.669 INFO:teuthology.packaging:branch: squid 2024-03-28T21:49:07.669 INFO:teuthology.packaging:sha1: 323d23dc228bbcb7f16f188fdc9c0dcb801844be 2024-03-28T21:49:07.669 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-28T21:49:07.907 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1959.g8da23624 2024-03-28T21:49:07.908 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-28T21:49:07.915 INFO:teuthology.task.internal:no buildpackages task found 2024-03-28T21:49:07.915 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-28T21:49:07.921 INFO:teuthology.task.internal:Saving configuration 2024-03-28T21:49:07.946 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-28T21:49:07.952 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T21:49:07.971 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627081', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 21:43:11.861994', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBALRE86IiYBSvcxgSkbfx534/kRLCWStoyl6wAPDRl4nc3gdufJHPiscJ+bHK493R7gtQKDv2W7wkh+bP78Mi14='} 2024-03-28T21:49:07.988 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi053.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627081', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 21:43:11.859516', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM44VrSbAkAdZ43rcJvDCkX44DcpMU2oZk2CR5TPqYqHHdYimKvUvbqpfps/UbdzGC+6zQbJKOyiFYYufqvMNu8='} 2024-03-28T21:49:08.005 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi063.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627081', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 21:43:11.860884', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEkuF2zVDPcz+eNFgDlEeVqrx7lY4M0fPlPf9F4EV5kLZ4JP9oA3g8gk9DFuASMkR4P2mAXA8O8T1rStkYerqTM='} 2024-03-28T21:49:08.006 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-28T21:49:08.013 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-28T21:49:08.014 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-28T21:49:08.014 INFO:teuthology.task.internal:roles: ubuntu@smithi063.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-28T21:49:08.014 INFO:teuthology.run_tasks:Running task console_log... 2024-03-28T21:49:08.102 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbebac2b160>, signals=[15]) 2024-03-28T21:49:08.102 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-28T21:49:08.108 INFO:teuthology.task.internal:Opening connections... 2024-03-28T21:49:08.108 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi016.front.sepia.ceph.com 2024-03-28T21:49:08.110 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T21:49:08.216 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi053.front.sepia.ceph.com 2024-03-28T21:49:08.217 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T21:49:08.321 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi063.front.sepia.ceph.com 2024-03-28T21:49:08.322 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T21:49:08.426 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-28T21:49:08.433 DEBUG:teuthology.orchestra.run.smithi016:> uname -m 2024-03-28T21:49:08.451 INFO:teuthology.orchestra.run.smithi016.stdout:x86_64 2024-03-28T21:49:08.451 DEBUG:teuthology.orchestra.run.smithi016:> cat /etc/os-release 2024-03-28T21:49:08.509 INFO:teuthology.orchestra.run.smithi016.stdout:NAME="CentOS Stream" 2024-03-28T21:49:08.509 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION="9" 2024-03-28T21:49:08.510 INFO:teuthology.orchestra.run.smithi016.stdout:ID="centos" 2024-03-28T21:49:08.510 INFO:teuthology.orchestra.run.smithi016.stdout:ID_LIKE="rhel fedora" 2024-03-28T21:49:08.510 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION_ID="9" 2024-03-28T21:49:08.510 INFO:teuthology.orchestra.run.smithi016.stdout:PLATFORM_ID="platform:el9" 2024-03-28T21:49:08.510 INFO:teuthology.orchestra.run.smithi016.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T21:49:08.510 INFO:teuthology.orchestra.run.smithi016.stdout:ANSI_COLOR="0;31" 2024-03-28T21:49:08.510 INFO:teuthology.orchestra.run.smithi016.stdout:LOGO="fedora-logo-icon" 2024-03-28T21:49:08.510 INFO:teuthology.orchestra.run.smithi016.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T21:49:08.510 INFO:teuthology.orchestra.run.smithi016.stdout:HOME_URL="https://centos.org/" 2024-03-28T21:49:08.510 INFO:teuthology.orchestra.run.smithi016.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T21:49:08.510 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T21:49:08.510 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T21:49:08.511 INFO:teuthology.lock.ops:Updating smithi016.front.sepia.ceph.com on lock server 2024-03-28T21:49:08.532 DEBUG:teuthology.orchestra.run.smithi053:> uname -m 2024-03-28T21:49:08.551 INFO:teuthology.orchestra.run.smithi053.stdout:x86_64 2024-03-28T21:49:08.551 DEBUG:teuthology.orchestra.run.smithi053:> cat /etc/os-release 2024-03-28T21:49:08.610 INFO:teuthology.orchestra.run.smithi053.stdout:NAME="CentOS Stream" 2024-03-28T21:49:08.610 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION="9" 2024-03-28T21:49:08.610 INFO:teuthology.orchestra.run.smithi053.stdout:ID="centos" 2024-03-28T21:49:08.610 INFO:teuthology.orchestra.run.smithi053.stdout:ID_LIKE="rhel fedora" 2024-03-28T21:49:08.610 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION_ID="9" 2024-03-28T21:49:08.610 INFO:teuthology.orchestra.run.smithi053.stdout:PLATFORM_ID="platform:el9" 2024-03-28T21:49:08.611 INFO:teuthology.orchestra.run.smithi053.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T21:49:08.611 INFO:teuthology.orchestra.run.smithi053.stdout:ANSI_COLOR="0;31" 2024-03-28T21:49:08.611 INFO:teuthology.orchestra.run.smithi053.stdout:LOGO="fedora-logo-icon" 2024-03-28T21:49:08.611 INFO:teuthology.orchestra.run.smithi053.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T21:49:08.611 INFO:teuthology.orchestra.run.smithi053.stdout:HOME_URL="https://centos.org/" 2024-03-28T21:49:08.611 INFO:teuthology.orchestra.run.smithi053.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T21:49:08.611 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T21:49:08.611 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T21:49:08.611 INFO:teuthology.lock.ops:Updating smithi053.front.sepia.ceph.com on lock server 2024-03-28T21:49:08.640 DEBUG:teuthology.orchestra.run.smithi063:> uname -m 2024-03-28T21:49:08.659 INFO:teuthology.orchestra.run.smithi063.stdout:x86_64 2024-03-28T21:49:08.659 DEBUG:teuthology.orchestra.run.smithi063:> cat /etc/os-release 2024-03-28T21:49:08.716 INFO:teuthology.orchestra.run.smithi063.stdout:NAME="CentOS Stream" 2024-03-28T21:49:08.716 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION="9" 2024-03-28T21:49:08.716 INFO:teuthology.orchestra.run.smithi063.stdout:ID="centos" 2024-03-28T21:49:08.716 INFO:teuthology.orchestra.run.smithi063.stdout:ID_LIKE="rhel fedora" 2024-03-28T21:49:08.716 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION_ID="9" 2024-03-28T21:49:08.717 INFO:teuthology.orchestra.run.smithi063.stdout:PLATFORM_ID="platform:el9" 2024-03-28T21:49:08.717 INFO:teuthology.orchestra.run.smithi063.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T21:49:08.717 INFO:teuthology.orchestra.run.smithi063.stdout:ANSI_COLOR="0;31" 2024-03-28T21:49:08.717 INFO:teuthology.orchestra.run.smithi063.stdout:LOGO="fedora-logo-icon" 2024-03-28T21:49:08.717 INFO:teuthology.orchestra.run.smithi063.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T21:49:08.717 INFO:teuthology.orchestra.run.smithi063.stdout:HOME_URL="https://centos.org/" 2024-03-28T21:49:08.717 INFO:teuthology.orchestra.run.smithi063.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T21:49:08.717 INFO:teuthology.orchestra.run.smithi063.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T21:49:08.717 INFO:teuthology.orchestra.run.smithi063.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T21:49:08.717 INFO:teuthology.lock.ops:Updating smithi063.front.sepia.ceph.com on lock server 2024-03-28T21:49:08.732 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-28T21:49:08.742 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-28T21:49:08.749 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-28T21:49:08.749 DEBUG:teuthology.orchestra.run.smithi016:> test '!' -e /home/ubuntu/cephtest 2024-03-28T21:49:08.752 DEBUG:teuthology.orchestra.run.smithi053:> test '!' -e /home/ubuntu/cephtest 2024-03-28T21:49:08.754 DEBUG:teuthology.orchestra.run.smithi063:> test '!' -e /home/ubuntu/cephtest 2024-03-28T21:49:08.773 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-28T21:49:08.778 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-28T21:49:08.779 DEBUG:teuthology.orchestra.run.smithi016:> test -z $(ls -A /var/lib/ceph) 2024-03-28T21:49:08.812 DEBUG:teuthology.orchestra.run.smithi053:> test -z $(ls -A /var/lib/ceph) 2024-03-28T21:49:08.816 DEBUG:teuthology.orchestra.run.smithi063:> test -z $(ls -A /var/lib/ceph) 2024-03-28T21:49:08.843 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-28T21:49:08.935 INFO:teuthology.run_tasks:Running task kernel... 2024-03-28T21:49:08.950 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-03-28T21:49:08.951 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-03-28T21:49:08.951 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-03-28T21:49:08.951 DEBUG:teuthology.orchestra.run.smithi016:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T21:49:08.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T21:49:08.968 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2024-03-28T21:49:09.027 INFO:teuthology.orchestra.run.smithi016.stdout:5.14.0-361.el9.x86_64 2024-03-28T21:49:09.027 INFO:teuthology.task.kernel:Running kernel on smithi016: 5.14.0-361.el9.x86_64 2024-03-28T21:49:09.027 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum install -y kernel 2024-03-28T21:49:11.125 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-28T21:49:12.575 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS-9 - BaseOS 7.1 MB/s | 2.0 MB 00:00 2024-03-28T21:49:14.157 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - BaseOS 6.6 MB/s | 8.0 MB 00:01 2024-03-28T21:49:16.142 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-03-28T21:49:21.067 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - CRB 5.2 MB/s | 6.1 MB 00:01 2024-03-28T21:49:22.752 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 16 kB 00:00 2024-03-28T21:49:29.039 INFO:teuthology.orchestra.run.smithi016.stdout:Extra Packages for Enterprise Linux 3.4 MB/s | 21 MB 00:06 2024-03-28T21:49:34.269 INFO:teuthology.orchestra.run.smithi016.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-28T21:49:36.635 INFO:teuthology.orchestra.run.smithi016.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-28T21:49:36.635 INFO:teuthology.orchestra.run.smithi016.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-28T21:49:36.724 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-28T21:49:36.734 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-28T21:49:36.734 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-03-28T21:49:36.734 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-28T21:49:36.734 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-28T21:49:36.734 INFO:teuthology.orchestra.run.smithi016.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-28T21:49:36.734 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-03-28T21:49:36.734 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-28T21:49:36.734 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-28T21:49:36.735 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-28T21:49:36.735 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T21:49:36.735 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-28T21:49:36.735 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-28T21:49:36.735 INFO:teuthology.orchestra.run.smithi016.stdout:Install 4 Packages 2024-03-28T21:49:36.735 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T21:49:36.735 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 76 M 2024-03-28T21:49:36.735 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 123 M 2024-03-28T21:49:36.735 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-28T21:49:36.929 INFO:teuthology.orchestra.run.smithi016.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 990 kB/s | 192 kB 00:00 2024-03-28T21:49:37.846 INFO:teuthology.orchestra.run.smithi016.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 14 MB/s | 15 MB 00:01 2024-03-28T21:49:38.346 INFO:teuthology.orchestra.run.smithi016.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 21 MB/s | 34 MB 00:01 2024-03-28T21:49:38.821 INFO:teuthology.orchestra.run.smithi016.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 15 MB/s | 28 MB 00:01 2024-03-28T21:49:38.824 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-28T21:49:38.824 INFO:teuthology.orchestra.run.smithi016.stdout:Total 37 MB/s | 76 MB 00:02 2024-03-28T21:49:38.824 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-28T21:49:39.229 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-28T21:49:39.302 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-28T21:49:40.083 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-28T21:49:40.083 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-28T21:49:42.515 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-28T21:49:43.144 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-28T21:49:43.230 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-28T21:49:46.161 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-28T21:49:46.299 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-28T21:49:51.370 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-28T21:49:51.416 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-28T21:49:56.522 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-28T21:50:27.450 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-28T21:50:27.450 INFO:teuthology.orchestra.run.smithi016.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-28T21:50:27.450 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T21:50:27.495 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-28T21:50:27.886 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-28T21:50:27.887 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-28T21:50:27.887 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-28T21:50:27.887 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-28T21:50:28.305 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-28T21:50:28.305 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T21:50:28.305 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-28T21:50:28.305 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-28T21:50:28.305 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-28T21:50:28.305 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-28T21:50:28.305 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-28T21:50:28.305 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T21:50:28.305 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-28T21:50:28.632 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q kernel | sort -rV | head -n 1 2024-03-28T21:50:28.782 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-28T21:50:28.783 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-28T21:50:28.783 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-28T21:50:28.783 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-28T21:50:28.783 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-28T21:50:28.783 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-03-28T21:50:28.783 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi016.front.sepia.ceph.com, path=None, version=distro) 2024-03-28T21:50:28.783 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q kernel | sort -rV | head -n 1 2024-03-28T21:50:28.866 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-28T21:50:28.866 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-28T21:50:28.867 DEBUG:teuthology.orchestra.run.smithi016:> sudo rpm -qi grub2-tools 2024-03-28T21:50:28.916 INFO:teuthology.orchestra.run.smithi016.stdout:Name : grub2-tools 2024-03-28T21:50:28.916 INFO:teuthology.orchestra.run.smithi016.stdout:Epoch : 1 2024-03-28T21:50:28.916 INFO:teuthology.orchestra.run.smithi016.stdout:Version : 2.06 2024-03-28T21:50:28.916 INFO:teuthology.orchestra.run.smithi016.stdout:Release : 68.el9 2024-03-28T21:50:28.916 INFO:teuthology.orchestra.run.smithi016.stdout:Architecture: x86_64 2024-03-28T21:50:28.916 INFO:teuthology.orchestra.run.smithi016.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-28T21:50:28.916 INFO:teuthology.orchestra.run.smithi016.stdout:Group : Unspecified 2024-03-28T21:50:28.916 INFO:teuthology.orchestra.run.smithi016.stdout:Size : 8264254 2024-03-28T21:50:28.916 INFO:teuthology.orchestra.run.smithi016.stdout:License : GPLv3+ 2024-03-28T21:50:28.916 INFO:teuthology.orchestra.run.smithi016.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-28T21:50:28.916 INFO:teuthology.orchestra.run.smithi016.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-28T21:50:28.917 INFO:teuthology.orchestra.run.smithi016.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-28T21:50:28.917 INFO:teuthology.orchestra.run.smithi016.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-28T21:50:28.917 INFO:teuthology.orchestra.run.smithi016.stdout:Packager : builder@centos.org 2024-03-28T21:50:28.917 INFO:teuthology.orchestra.run.smithi016.stdout:Vendor : CentOS 2024-03-28T21:50:28.917 INFO:teuthology.orchestra.run.smithi016.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-28T21:50:28.917 INFO:teuthology.orchestra.run.smithi016.stdout:Summary : Support tools for GRUB. 2024-03-28T21:50:28.917 INFO:teuthology.orchestra.run.smithi016.stdout:Description : 2024-03-28T21:50:28.917 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T21:50:28.917 INFO:teuthology.orchestra.run.smithi016.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-28T21:50:28.917 INFO:teuthology.orchestra.run.smithi016.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-28T21:50:28.917 INFO:teuthology.orchestra.run.smithi016.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-28T21:50:28.917 INFO:teuthology.orchestra.run.smithi016.stdout:hardware devices. 2024-03-28T21:50:28.917 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T21:50:28.917 INFO:teuthology.orchestra.run.smithi016.stdout:This subpackage provides tools for support of all platforms. 2024-03-28T21:50:28.918 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-28T21:50:28.918 INFO:teuthology.task.kernel:Updating grub on smithi016 to boot 5.14.0-434.el9.x86_64 2024-03-28T21:50:28.918 DEBUG:teuthology.orchestra.run.smithi016:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-28T21:50:29.490 INFO:teuthology.orchestra.run.smithi016.stderr:Generating grub configuration file ... 2024-03-28T21:50:30.965 INFO:teuthology.orchestra.run.smithi016.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-28T21:50:31.014 INFO:teuthology.orchestra.run.smithi016.stderr:done 2024-03-28T21:50:31.016 DEBUG:teuthology.orchestra.run.smithi016:> mktemp 2024-03-28T21:50:31.032 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.TM9YBP83Dk 2024-03-28T21:50:31.032 DEBUG:teuthology.orchestra.run.smithi016:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.TM9YBP83Dk 2024-03-28T21:50:31.106 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.TM9YBP83Dk 2024-03-28T21:50:31.251 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.TM9YBP83Dk is 6KB 2024-03-28T21:50:31.273 DEBUG:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.TM9YBP83Dk 2024-03-28T21:50:31.290 DEBUG:teuthology.orchestra.run.smithi016:> sudo /bin/ls /boot/loader/entries || true 2024-03-28T21:50:31.373 INFO:teuthology.orchestra.run.smithi016.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-28T21:50:31.373 INFO:teuthology.orchestra.run.smithi016.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-28T21:50:31.373 INFO:teuthology.orchestra.run.smithi016.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-28T21:50:31.373 INFO:teuthology.orchestra.run.smithi016.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-28T21:50:31.374 DEBUG:teuthology.orchestra.run.smithi016:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-28T21:50:31.532 DEBUG:teuthology.orchestra.run.smithi016:> sudo shutdown -r now 2024-03-28T21:51:01.535 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-28T21:51:01.536 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-03-28T21:51:01.537 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T21:52:01.539 DEBUG:teuthology.orchestra.remote:timed out 2024-03-28T21:52:10.540 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-03-28T21:52:10.542 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T21:52:10.911 DEBUG:teuthology.orchestra.run.smithi016:> true 2024-03-28T21:52:11.335 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-03-28T21:52:11.335 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-434.el9.x86_64"... 2024-03-28T21:52:11.335 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2024-03-28T21:52:11.349 INFO:teuthology.orchestra.run.smithi016.stdout:5.14.0-434.el9.x86_64 2024-03-28T21:52:11.349 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-28T21:52:11.349 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-28T21:52:11.349 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-28T21:52:12.351 DEBUG:teuthology.parallel:result is None 2024-03-28T21:52:12.351 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-28T21:52:12.358 INFO:teuthology.task.internal:Creating test directory... 2024-03-28T21:52:12.359 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T21:52:12.361 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T21:52:12.364 DEBUG:teuthology.orchestra.run.smithi063:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T21:52:12.382 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-28T21:52:12.389 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-28T21:52:12.398 INFO:teuthology.task.internal:Creating archive directory... 2024-03-28T21:52:12.398 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T21:52:12.420 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T21:52:12.422 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T21:52:12.456 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-28T21:52:12.463 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-28T21:52:12.463 DEBUG:teuthology.orchestra.run.smithi053:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T21:52:12.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T21:52:12.505 DEBUG:teuthology.orchestra.run.smithi063:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T21:52:12.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T21:52:12.522 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T21:52:12.524 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T21:52:12.548 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T21:52:12.584 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T21:52:12.585 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T21:52:12.608 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T21:52:12.623 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T21:52:12.634 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T21:52:12.645 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T21:52:12.647 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-28T21:52:12.656 INFO:teuthology.task.internal:Configuring sudo... 2024-03-28T21:52:12.656 DEBUG:teuthology.orchestra.run.smithi016:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T21:52:12.659 DEBUG:teuthology.orchestra.run.smithi053:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T21:52:12.662 DEBUG:teuthology.orchestra.run.smithi063:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T21:52:12.696 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-28T21:52:12.708 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-28T21:52:12.709 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T21:52:12.730 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T21:52:12.734 DEBUG:teuthology.orchestra.run.smithi063:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T21:52:12.753 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T21:52:12.826 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T21:52:12.887 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T21:52:12.973 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T21:52:13.020 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-28T21:52:13.020 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T21:52:13.092 DEBUG:teuthology.orchestra.run.smithi053:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T21:52:13.136 DEBUG:teuthology.orchestra.run.smithi053:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T21:52:13.197 DEBUG:teuthology.orchestra.run.smithi053:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T21:52:13.238 DEBUG:teuthology.orchestra.run.smithi053:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T21:52:13.291 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-28T21:52:13.291 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T21:52:13.364 DEBUG:teuthology.orchestra.run.smithi063:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T21:52:13.410 DEBUG:teuthology.orchestra.run.smithi063:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T21:52:13.473 DEBUG:teuthology.orchestra.run.smithi063:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T21:52:13.515 DEBUG:teuthology.orchestra.run.smithi063:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T21:52:13.566 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-28T21:52:13.566 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T21:52:13.639 DEBUG:teuthology.orchestra.run.smithi016:> sudo service rsyslog restart 2024-03-28T21:52:13.641 DEBUG:teuthology.orchestra.run.smithi053:> sudo service rsyslog restart 2024-03-28T21:52:13.643 DEBUG:teuthology.orchestra.run.smithi063:> sudo service rsyslog restart 2024-03-28T21:52:13.682 INFO:teuthology.orchestra.run.smithi053.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T21:52:13.723 INFO:teuthology.orchestra.run.smithi063.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T21:52:13.742 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T21:52:14.209 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-28T21:52:14.217 INFO:teuthology.task.internal:Starting timer... 2024-03-28T21:52:14.217 INFO:teuthology.run_tasks:Running task pcp... 2024-03-28T21:52:14.237 INFO:teuthology.run_tasks:Running task selinux... 2024-03-28T21:52:14.246 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-28T21:52:14.246 DEBUG:teuthology.orchestra.run.smithi016:> sudo service auditd rotate 2024-03-28T21:52:14.352 INFO:teuthology.orchestra.run.smithi016.stdout:Rotating logs: 2024-03-28T21:52:14.354 DEBUG:teuthology.orchestra.run.smithi053:> sudo service auditd rotate 2024-03-28T21:52:14.406 INFO:teuthology.orchestra.run.smithi053.stdout:Rotating logs: 2024-03-28T21:52:14.408 DEBUG:teuthology.orchestra.run.smithi063:> sudo service auditd rotate 2024-03-28T21:52:14.477 INFO:teuthology.orchestra.run.smithi063.stdout:Rotating logs: 2024-03-28T21:52:14.479 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-28T21:52:14.479 DEBUG:teuthology.orchestra.run.smithi016:> /usr/sbin/getenforce 2024-03-28T21:52:14.531 INFO:teuthology.orchestra.run.smithi016.stdout:Permissive 2024-03-28T21:52:14.532 DEBUG:teuthology.orchestra.run.smithi053:> /usr/sbin/getenforce 2024-03-28T21:52:14.563 INFO:teuthology.orchestra.run.smithi053.stdout:Permissive 2024-03-28T21:52:14.564 DEBUG:teuthology.orchestra.run.smithi063:> /usr/sbin/getenforce 2024-03-28T21:52:14.583 INFO:teuthology.orchestra.run.smithi063.stdout:Permissive 2024-03-28T21:52:14.583 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi016.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi053.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi063.front.sepia.ceph.com': 'permissive'} 2024-03-28T21:52:14.583 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T21:52:14.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T21:52:14.611 DEBUG:teuthology.orchestra.run.smithi053:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T21:52:14.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T21:52:14.644 DEBUG:teuthology.orchestra.run.smithi063:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T21:52:14.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T21:52:14.677 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-28T21:52:14.677 DEBUG:teuthology.orchestra.run.smithi016:> sudo /usr/sbin/setenforce permissive 2024-03-28T21:52:14.984 DEBUG:teuthology.orchestra.run.smithi053:> sudo /usr/sbin/setenforce permissive 2024-03-28T21:52:15.032 DEBUG:teuthology.orchestra.run.smithi063:> sudo /usr/sbin/setenforce permissive 2024-03-28T21:52:15.069 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-28T21:52:15.082 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-28T21:52:15.087 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-28T21:52:15.210 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-28T21:52:15.225 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-28T21:52:15.227 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi016.front.sepia.ceph.com,smithi053.front.sepia.ceph.com,smithi063.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-28T21:57:23.772 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi016.front.sepia.ceph.com'), Remote(name='ubuntu@smithi053.front.sepia.ceph.com'), Remote(name='ubuntu@smithi063.front.sepia.ceph.com')] 2024-03-28T21:57:23.774 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-03-28T21:57:23.775 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T21:57:23.869 DEBUG:teuthology.orchestra.run.smithi016:> true 2024-03-28T21:57:23.950 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-03-28T21:57:23.951 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-03-28T21:57:23.951 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T21:57:24.055 DEBUG:teuthology.orchestra.run.smithi053:> true 2024-03-28T21:57:24.127 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-03-28T21:57:24.127 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-03-28T21:57:24.128 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T21:57:24.275 DEBUG:teuthology.orchestra.run.smithi063:> true 2024-03-28T21:57:24.346 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-03-28T21:57:24.346 INFO:teuthology.run_tasks:Running task clock... 2024-03-28T21:57:24.357 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-28T21:57:24.357 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T21:57:24.358 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T21:57:24.361 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T21:57:24.361 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T21:57:24.363 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T21:57:24.363 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T21:57:24.394 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T21:57:24.400 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T21:57:24.412 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T21:57:24.421 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T21:57:24.422 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T21:57:24.443 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T21:57:24.443 INFO:teuthology.orchestra.run.smithi016.stderr:sudo: ntpd: command not found 2024-03-28T21:57:24.458 INFO:teuthology.orchestra.run.smithi016.stdout:506 Cannot talk to daemon 2024-03-28T21:57:24.459 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: ntpd: command not found 2024-03-28T21:57:24.475 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T21:57:24.477 INFO:teuthology.orchestra.run.smithi053.stdout:506 Cannot talk to daemon 2024-03-28T21:57:24.479 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: ntpd: command not found 2024-03-28T21:57:24.492 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T21:57:24.496 INFO:teuthology.orchestra.run.smithi063.stdout:506 Cannot talk to daemon 2024-03-28T21:57:24.497 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T21:57:24.516 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T21:57:24.517 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T21:57:24.536 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T21:57:24.550 INFO:teuthology.orchestra.run.smithi016.stderr:bash: line 1: ntpq: command not found 2024-03-28T21:57:24.553 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T21:57:24.553 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2024-03-28T21:57:24.553 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T21:57:24.553 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T21:57:24.553 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T21:57:24.553 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T21:57:24.565 INFO:teuthology.orchestra.run.smithi053.stderr:bash: line 1: ntpq: command not found 2024-03-28T21:57:24.569 INFO:teuthology.orchestra.run.smithi053.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T21:57:24.569 INFO:teuthology.orchestra.run.smithi053.stdout:=============================================================================== 2024-03-28T21:57:24.569 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T21:57:24.569 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T21:57:24.569 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T21:57:24.570 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T21:57:24.585 INFO:teuthology.orchestra.run.smithi063.stderr:bash: line 1: ntpq: command not found 2024-03-28T21:57:24.588 INFO:teuthology.orchestra.run.smithi063.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T21:57:24.588 INFO:teuthology.orchestra.run.smithi063.stdout:=============================================================================== 2024-03-28T21:57:24.588 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T21:57:24.588 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T21:57:24.588 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T21:57:24.588 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T21:57:24.589 INFO:teuthology.run_tasks:Running task install... 2024-03-28T21:57:24.598 DEBUG:teuthology.task.install:project ceph 2024-03-28T21:57:24.598 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '323d23dc228bbcb7f16f188fdc9c0dcb801844be'}} 2024-03-28T21:57:24.598 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '323d23dc228bbcb7f16f188fdc9c0dcb801844be'} 2024-03-28T21:57:24.598 INFO:teuthology.task.install:Using flavor: default 2024-03-28T21:57:24.604 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-28T21:57:24.604 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-28T21:57:24.605 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '323d23dc228bbcb7f16f188fdc9c0dcb801844be', 'tag': None, 'wait_for_package': False} 2024-03-28T21:57:24.605 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=323d23dc228bbcb7f16f188fdc9c0dcb801844be 2024-03-28T21:57:24.608 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '323d23dc228bbcb7f16f188fdc9c0dcb801844be', 'tag': None, 'wait_for_package': False} 2024-03-28T21:57:24.608 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=323d23dc228bbcb7f16f188fdc9c0dcb801844be 2024-03-28T21:57:24.609 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '323d23dc228bbcb7f16f188fdc9c0dcb801844be', 'tag': None, 'wait_for_package': False} 2024-03-28T21:57:24.609 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=323d23dc228bbcb7f16f188fdc9c0dcb801844be 2024-03-28T21:57:24.755 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/flavors/default/ 2024-03-28T21:57:24.756 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1899.g323d23dc 2024-03-28T21:57:24.820 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/flavors/default/ 2024-03-28T21:57:24.820 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1899.g323d23dc 2024-03-28T21:57:24.832 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/flavors/default/ 2024-03-28T21:57:24.832 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1899.g323d23dc 2024-03-28T21:57:25.052 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T21:57:25.052 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-28T21:57:25.052 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T21:57:25.084 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T21:57:25.084 DEBUG:teuthology.orchestra.run.smithi063:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/323d23dc228bbcb7f16f188fdc9c0dcb801844be/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T21:57:25.099 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T21:57:25.099 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-28T21:57:25.099 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T21:57:25.122 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T21:57:25.122 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-28T21:57:25.122 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T21:57:25.132 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T21:57:25.132 DEBUG:teuthology.orchestra.run.smithi016:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/323d23dc228bbcb7f16f188fdc9c0dcb801844be/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T21:57:25.155 DEBUG:teuthology.orchestra.run.smithi063:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T21:57:25.157 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T21:57:25.157 DEBUG:teuthology.orchestra.run.smithi053:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/323d23dc228bbcb7f16f188fdc9c0dcb801844be/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T21:57:25.202 DEBUG:teuthology.orchestra.run.smithi016:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T21:57:25.231 DEBUG:teuthology.orchestra.run.smithi053:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T21:57:25.243 DEBUG:teuthology.orchestra.run.smithi063:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T21:57:25.273 INFO:teuthology.orchestra.run.smithi063.stdout:check_obsoletes = 1 2024-03-28T21:57:25.275 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum clean all 2024-03-28T21:57:25.285 DEBUG:teuthology.orchestra.run.smithi016:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T21:57:25.320 DEBUG:teuthology.orchestra.run.smithi053:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T21:57:25.350 INFO:teuthology.orchestra.run.smithi053.stdout:check_obsoletes = 1 2024-03-28T21:57:25.352 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum clean all 2024-03-28T21:57:25.353 INFO:teuthology.orchestra.run.smithi016.stdout:check_obsoletes = 1 2024-03-28T21:57:25.354 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum clean all 2024-03-28T21:57:25.524 INFO:teuthology.orchestra.run.smithi063.stdout:74 files removed 2024-03-28T21:57:25.552 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T21:57:25.600 INFO:teuthology.orchestra.run.smithi053.stdout:74 files removed 2024-03-28T21:57:25.606 INFO:teuthology.orchestra.run.smithi016.stdout:74 files removed 2024-03-28T21:57:25.627 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T21:57:25.633 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T21:57:26.666 INFO:teuthology.orchestra.run.smithi063.stdout:ceph packages for x86_64 103 kB/s | 83 kB 00:00 2024-03-28T21:57:26.831 INFO:teuthology.orchestra.run.smithi016.stdout:ceph packages for x86_64 93 kB/s | 83 kB 00:00 2024-03-28T21:57:26.832 INFO:teuthology.orchestra.run.smithi053.stdout:ceph packages for x86_64 93 kB/s | 83 kB 00:00 2024-03-28T21:57:27.433 INFO:teuthology.orchestra.run.smithi063.stdout:ceph noarch packages 16 kB/s | 11 kB 00:00 2024-03-28T21:57:27.565 INFO:teuthology.orchestra.run.smithi016.stdout:ceph noarch packages 17 kB/s | 11 kB 00:00 2024-03-28T21:57:27.593 INFO:teuthology.orchestra.run.smithi053.stdout:ceph noarch packages 16 kB/s | 11 kB 00:00 2024-03-28T21:57:28.192 INFO:teuthology.orchestra.run.smithi063.stdout:ceph source packages 2.6 kB/s | 1.9 kB 00:00 2024-03-28T21:57:28.241 INFO:teuthology.orchestra.run.smithi053.stdout:ceph source packages 3.1 kB/s | 1.9 kB 00:00 2024-03-28T21:57:28.291 INFO:teuthology.orchestra.run.smithi016.stdout:ceph source packages 2.8 kB/s | 1.9 kB 00:00 2024-03-28T21:57:28.927 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-28T21:57:29.086 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS-9 - AppStream 7.7 MB/s | 6.7 MB 00:00 2024-03-28T21:57:29.119 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS-9 - AppStream 7.9 MB/s | 6.7 MB 00:00 2024-03-28T21:57:30.376 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2024-03-28T21:57:30.529 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS-9 - BaseOS 7.1 MB/s | 2.0 MB 00:00 2024-03-28T21:57:30.561 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2024-03-28T21:57:31.392 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-28T21:57:32.586 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.0 MB 00:01 2024-03-28T21:57:33.796 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-03-28T21:57:34.762 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:01 2024-03-28T21:57:38.778 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - CRB 4.9 MB/s | 6.1 MB 00:01 2024-03-28T21:57:39.746 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - CRB 4.8 MB/s | 6.1 MB 00:01 2024-03-28T21:57:39.942 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - BaseOS 911 kB/s | 8.0 MB 00:09 2024-03-28T21:57:40.278 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - Extras packages 57 kB/s | 16 kB 00:00 2024-03-28T21:57:42.313 INFO:teuthology.orchestra.run.smithi016.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-28T21:57:42.447 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - Extras packages 10 kB/s | 16 kB 00:01 2024-03-28T21:57:44.215 INFO:teuthology.orchestra.run.smithi063.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-28T21:57:44.435 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - AppStream 5.7 MB/s | 19 MB 00:03 2024-03-28T21:57:47.388 INFO:teuthology.orchestra.run.smithi016.stdout:lab-extras 21 kB/s | 1.7 kB 00:00 2024-03-28T21:57:48.763 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - CRB 9.2 MB/s | 6.1 MB 00:00 2024-03-28T21:57:49.490 INFO:teuthology.orchestra.run.smithi063.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-28T21:57:50.075 INFO:teuthology.orchestra.run.smithi016.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T21:57:50.076 INFO:teuthology.orchestra.run.smithi016.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T21:57:50.091 INFO:teuthology.orchestra.run.smithi016.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T21:57:50.092 INFO:teuthology.orchestra.run.smithi016.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T21:57:50.092 INFO:teuthology.orchestra.run.smithi016.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T21:57:50.093 INFO:teuthology.orchestra.run.smithi016.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T21:57:50.095 INFO:teuthology.orchestra.run.smithi016.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T21:57:50.098 INFO:teuthology.orchestra.run.smithi016.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T21:57:50.099 INFO:teuthology.orchestra.run.smithi016.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T21:57:50.185 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-28T21:57:50.197 INFO:teuthology.orchestra.run.smithi016.stdout:============================================================================================ 2024-03-28T21:57:50.198 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-03-28T21:57:50.198 INFO:teuthology.orchestra.run.smithi016.stdout:============================================================================================ 2024-03-28T21:57:50.198 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-28T21:57:50.198 INFO:teuthology.orchestra.run.smithi016.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T21:57:50.198 INFO:teuthology.orchestra.run.smithi016.stdout: ceph x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 6.6 k 2024-03-28T21:57:50.198 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 5.3 M 2024-03-28T21:57:50.198 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 889 k 2024-03-28T21:57:50.198 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 145 k 2024-03-28T21:57:50.198 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 1.5 M 2024-03-28T21:57:50.198 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 138 k 2024-03-28T21:57:50.198 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 3.5 M 2024-03-28T21:57:50.198 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 7.4 M 2024-03-28T21:57:50.199 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 49 k 2024-03-28T21:57:50.199 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 9.8 M 2024-03-28T21:57:50.199 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 47 M 2024-03-28T21:57:50.199 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-volume noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 264 k 2024-03-28T21:57:50.199 INFO:teuthology.orchestra.run.smithi016.stdout: cephadm noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 748 k 2024-03-28T21:57:50.199 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-mirror x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 221 k 2024-03-28T21:57:50.199 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-shell x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 27 k 2024-03-28T21:57:50.199 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-top noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 18 k 2024-03-28T21:57:50.199 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T21:57:50.199 INFO:teuthology.orchestra.run.smithi016.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T21:57:50.199 INFO:teuthology.orchestra.run.smithi016.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T21:57:50.199 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 32 k 2024-03-28T21:57:50.199 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 746 k 2024-03-28T21:57:50.199 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 127 k 2024-03-28T21:57:50.200 INFO:teuthology.orchestra.run.smithi016.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T21:57:50.200 INFO:teuthology.orchestra.run.smithi016.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T21:57:50.200 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T21:57:50.200 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql x86_64 13.14-1.el9 CentOS-AppStream 1.5 M 2024-03-28T21:57:50.200 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-contrib x86_64 13.14-1.el9 CentOS-AppStream 817 k 2024-03-28T21:57:50.200 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-server x86_64 13.14-1.el9 CentOS-AppStream 5.7 M 2024-03-28T21:57:50.200 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 162 k 2024-03-28T21:57:50.200 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 321 k 2024-03-28T21:57:50.200 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 299 k 2024-03-28T21:57:50.200 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 100 k 2024-03-28T21:57:50.200 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 86 k 2024-03-28T21:57:50.200 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 3.1 M 2024-03-28T21:57:50.200 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 171 k 2024-03-28T21:57:50.201 INFO:teuthology.orchestra.run.smithi016.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T21:57:50.201 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrading: 2024-03-28T21:57:50.201 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager x86_64 1:1.47.2-1.el9 CentOS-BaseOS 2.3 M 2024-03-28T21:57:50.201 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 CentOS-BaseOS 1.8 M 2024-03-28T21:57:50.201 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 CentOS-BaseOS 41 k 2024-03-28T21:57:50.201 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 CentOS-BaseOS 247 k 2024-03-28T21:57:50.201 INFO:teuthology.orchestra.run.smithi016.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T21:57:50.201 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T21:57:50.201 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T21:57:50.201 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T21:57:50.201 INFO:teuthology.orchestra.run.smithi016.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T21:57:50.201 INFO:teuthology.orchestra.run.smithi016.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T21:57:50.201 INFO:teuthology.orchestra.run.smithi016.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T21:57:50.201 INFO:teuthology.orchestra.run.smithi016.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T21:57:50.202 INFO:teuthology.orchestra.run.smithi016.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T21:57:50.202 INFO:teuthology.orchestra.run.smithi016.stdout: librados2 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 3.4 M 2024-03-28T21:57:50.202 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 3.1 M 2024-03-28T21:57:50.202 INFO:teuthology.orchestra.run.smithi016.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T21:57:50.202 INFO:teuthology.orchestra.run.smithi016.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T21:57:50.202 INFO:teuthology.orchestra.run.smithi016.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T21:57:50.202 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T21:57:50.202 INFO:teuthology.orchestra.run.smithi016.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T21:57:50.202 INFO:teuthology.orchestra.run.smithi016.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T21:57:50.202 INFO:teuthology.orchestra.run.smithi016.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T21:57:50.202 INFO:teuthology.orchestra.run.smithi016.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T21:57:50.202 INFO:teuthology.orchestra.run.smithi016.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T21:57:50.202 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-03-28T21:57:50.202 INFO:teuthology.orchestra.run.smithi016.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T21:57:50.203 INFO:teuthology.orchestra.run.smithi016.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T21:57:50.203 INFO:teuthology.orchestra.run.smithi016.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T21:57:50.203 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 21 M 2024-03-28T21:57:50.203 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 31 k 2024-03-28T21:57:50.203 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 2.4 M 2024-03-28T21:57:50.203 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 248 k 2024-03-28T21:57:50.203 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 4.7 M 2024-03-28T21:57:50.203 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 17 M 2024-03-28T21:57:50.203 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 15 k 2024-03-28T21:57:50.203 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 25 k 2024-03-28T21:57:50.203 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T21:57:50.203 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T21:57:50.203 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T21:57:50.203 INFO:teuthology.orchestra.run.smithi016.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T21:57:50.203 INFO:teuthology.orchestra.run.smithi016.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T21:57:50.204 INFO:teuthology.orchestra.run.smithi016.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T21:57:50.204 INFO:teuthology.orchestra.run.smithi016.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T21:57:50.204 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T21:57:50.204 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T21:57:50.204 INFO:teuthology.orchestra.run.smithi016.stdout: libcephsqlite x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 163 k 2024-03-28T21:57:50.204 INFO:teuthology.orchestra.run.smithi016.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T21:57:50.204 INFO:teuthology.orchestra.run.smithi016.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T21:57:50.204 INFO:teuthology.orchestra.run.smithi016.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T21:57:50.204 INFO:teuthology.orchestra.run.smithi016.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T21:57:50.204 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T21:57:50.204 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 501 k 2024-03-28T21:57:50.204 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T21:57:50.204 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 5.1 M 2024-03-28T21:57:50.204 INFO:teuthology.orchestra.run.smithi016.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T21:57:50.204 INFO:teuthology.orchestra.run.smithi016.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T21:57:50.205 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T21:57:50.205 INFO:teuthology.orchestra.run.smithi016.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T21:57:50.205 INFO:teuthology.orchestra.run.smithi016.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T21:57:50.205 INFO:teuthology.orchestra.run.smithi016.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T21:57:50.205 INFO:teuthology.orchestra.run.smithi016.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T21:57:50.205 INFO:teuthology.orchestra.run.smithi016.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T21:57:50.205 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T21:57:50.205 INFO:teuthology.orchestra.run.smithi016.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T21:57:50.205 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-private-libs x86_64 13.14-1.el9 CentOS-AppStream 132 k 2024-03-28T21:57:50.205 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T21:57:50.205 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T21:57:50.205 INFO:teuthology.orchestra.run.smithi016.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T21:57:50.205 INFO:teuthology.orchestra.run.smithi016.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T21:57:50.206 INFO:teuthology.orchestra.run.smithi016.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T21:57:50.206 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T21:57:50.206 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T21:57:50.206 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 45 k 2024-03-28T21:57:50.206 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 131 k 2024-03-28T21:57:50.206 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T21:57:50.206 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T21:57:50.206 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T21:57:50.206 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T21:57:50.206 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T21:57:50.206 INFO:teuthology.orchestra.run.smithi016.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T21:57:50.206 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T21:57:50.206 INFO:teuthology.orchestra.run.smithi016.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T21:57:50.206 INFO:teuthology.orchestra.run.smithi016.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T21:57:50.207 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T21:57:50.207 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T21:57:50.207 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T21:57:50.207 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T21:57:50.207 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T21:57:50.207 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T21:57:50.207 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T21:57:50.207 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T21:57:50.207 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T21:57:50.207 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T21:57:50.207 INFO:teuthology.orchestra.run.smithi016.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T21:57:50.207 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T21:57:50.207 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T21:57:50.207 INFO:teuthology.orchestra.run.smithi016.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T21:57:50.208 INFO:teuthology.orchestra.run.smithi016.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T21:57:50.208 INFO:teuthology.orchestra.run.smithi016.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T21:57:50.208 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T21:57:50.208 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T21:57:50.208 INFO:teuthology.orchestra.run.smithi016.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T21:57:50.208 INFO:teuthology.orchestra.run.smithi016.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T21:57:50.208 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T21:57:50.208 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T21:57:50.208 INFO:teuthology.orchestra.run.smithi016.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T21:57:50.208 INFO:teuthology.orchestra.run.smithi016.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T21:57:50.208 INFO:teuthology.orchestra.run.smithi016.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T21:57:50.208 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T21:57:50.208 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T21:57:50.208 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T21:57:50.208 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T21:57:50.209 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T21:57:50.209 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T21:57:50.209 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T21:57:50.209 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T21:57:50.209 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T21:57:50.209 INFO:teuthology.orchestra.run.smithi016.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T21:57:50.209 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T21:57:50.209 INFO:teuthology.orchestra.run.smithi016.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T21:57:50.209 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T21:57:50.209 INFO:teuthology.orchestra.run.smithi016.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T21:57:50.209 INFO:teuthology.orchestra.run.smithi016.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T21:57:50.209 INFO:teuthology.orchestra.run.smithi016.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T21:57:50.209 INFO:teuthology.orchestra.run.smithi016.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T21:57:50.209 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T21:57:50.209 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T21:57:50.210 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T21:57:50.210 INFO:teuthology.orchestra.run.smithi016.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T21:57:50.210 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-28T21:57:50.210 INFO:teuthology.orchestra.run.smithi016.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-28T21:57:50.210 INFO:teuthology.orchestra.run.smithi016.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T21:57:50.210 INFO:teuthology.orchestra.run.smithi016.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T21:57:50.210 INFO:teuthology.orchestra.run.smithi016.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T21:57:50.210 INFO:teuthology.orchestra.run.smithi016.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T21:57:50.210 INFO:teuthology.orchestra.run.smithi016.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T21:57:50.210 INFO:teuthology.orchestra.run.smithi016.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T21:57:50.210 INFO:teuthology.orchestra.run.smithi016.stdout:Installing weak dependencies: 2024-03-28T21:57:50.210 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-28T21:57:50.210 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T21:57:50.211 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-28T21:57:50.211 INFO:teuthology.orchestra.run.smithi016.stdout:============================================================================================ 2024-03-28T21:57:50.211 INFO:teuthology.orchestra.run.smithi016.stdout:Install 150 Packages 2024-03-28T21:57:50.211 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrade 24 Packages 2024-03-28T21:57:50.211 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T21:57:50.211 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 226 M 2024-03-28T21:57:50.211 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-28T21:57:50.386 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 16 kB 00:00 2024-03-28T21:57:51.389 INFO:teuthology.orchestra.run.smithi016.stdout:(1/174): ceph-19.0.0-1899.g323d23dc.el9.x86_64. 21 kB/s | 6.6 kB 00:00 2024-03-28T21:57:52.140 INFO:teuthology.orchestra.run.smithi063.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T21:57:52.141 INFO:teuthology.orchestra.run.smithi063.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T21:57:52.156 INFO:teuthology.orchestra.run.smithi063.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T21:57:52.157 INFO:teuthology.orchestra.run.smithi063.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T21:57:52.158 INFO:teuthology.orchestra.run.smithi063.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T21:57:52.160 INFO:teuthology.orchestra.run.smithi063.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T21:57:52.162 INFO:teuthology.orchestra.run.smithi063.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T21:57:52.165 INFO:teuthology.orchestra.run.smithi063.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T21:57:52.166 INFO:teuthology.orchestra.run.smithi063.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T21:57:52.196 INFO:teuthology.orchestra.run.smithi053.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-28T21:57:52.255 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-28T21:57:52.273 INFO:teuthology.orchestra.run.smithi063.stdout:============================================================================================ 2024-03-28T21:57:52.274 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-03-28T21:57:52.274 INFO:teuthology.orchestra.run.smithi063.stdout:============================================================================================ 2024-03-28T21:57:52.274 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-28T21:57:52.274 INFO:teuthology.orchestra.run.smithi063.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T21:57:52.274 INFO:teuthology.orchestra.run.smithi063.stdout: ceph x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 6.6 k 2024-03-28T21:57:52.274 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-base x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 5.3 M 2024-03-28T21:57:52.274 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-fuse x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 889 k 2024-03-28T21:57:52.274 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 145 k 2024-03-28T21:57:52.274 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 1.5 M 2024-03-28T21:57:52.274 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 138 k 2024-03-28T21:57:52.274 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 3.5 M 2024-03-28T21:57:52.274 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 7.4 M 2024-03-28T21:57:52.275 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-rook noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 49 k 2024-03-28T21:57:52.275 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-radosgw x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 9.8 M 2024-03-28T21:57:52.275 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-test x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 47 M 2024-03-28T21:57:52.275 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-volume noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 264 k 2024-03-28T21:57:52.275 INFO:teuthology.orchestra.run.smithi063.stdout: cephadm noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 748 k 2024-03-28T21:57:52.275 INFO:teuthology.orchestra.run.smithi063.stdout: cephfs-mirror x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 221 k 2024-03-28T21:57:52.275 INFO:teuthology.orchestra.run.smithi063.stdout: cephfs-shell x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 27 k 2024-03-28T21:57:52.275 INFO:teuthology.orchestra.run.smithi063.stdout: cephfs-top noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 18 k 2024-03-28T21:57:52.275 INFO:teuthology.orchestra.run.smithi063.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T21:57:52.275 INFO:teuthology.orchestra.run.smithi063.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T21:57:52.275 INFO:teuthology.orchestra.run.smithi063.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T21:57:52.275 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs-devel x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 32 k 2024-03-28T21:57:52.275 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs2 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 746 k 2024-03-28T21:57:52.275 INFO:teuthology.orchestra.run.smithi063.stdout: librados-devel x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 127 k 2024-03-28T21:57:52.276 INFO:teuthology.orchestra.run.smithi063.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T21:57:52.276 INFO:teuthology.orchestra.run.smithi063.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T21:57:52.276 INFO:teuthology.orchestra.run.smithi063.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T21:57:52.276 INFO:teuthology.orchestra.run.smithi063.stdout: postgresql x86_64 13.14-1.el9 CentOS-AppStream 1.5 M 2024-03-28T21:57:52.276 INFO:teuthology.orchestra.run.smithi063.stdout: postgresql-contrib x86_64 13.14-1.el9 CentOS-AppStream 817 k 2024-03-28T21:57:52.276 INFO:teuthology.orchestra.run.smithi063.stdout: postgresql-server x86_64 13.14-1.el9 CentOS-AppStream 5.7 M 2024-03-28T21:57:52.276 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cephfs x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 162 k 2024-03-28T21:57:52.276 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rados x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 321 k 2024-03-28T21:57:52.276 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rbd x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 299 k 2024-03-28T21:57:52.276 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rgw x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 100 k 2024-03-28T21:57:52.276 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-fuse x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 86 k 2024-03-28T21:57:52.276 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-mirror x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 3.1 M 2024-03-28T21:57:52.276 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-nbd x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 171 k 2024-03-28T21:57:52.276 INFO:teuthology.orchestra.run.smithi063.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T21:57:52.277 INFO:teuthology.orchestra.run.smithi063.stdout:Upgrading: 2024-03-28T21:57:52.277 INFO:teuthology.orchestra.run.smithi063.stdout: NetworkManager x86_64 1:1.47.2-1.el9 CentOS-BaseOS 2.3 M 2024-03-28T21:57:52.277 INFO:teuthology.orchestra.run.smithi063.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 CentOS-BaseOS 1.8 M 2024-03-28T21:57:52.277 INFO:teuthology.orchestra.run.smithi063.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 CentOS-BaseOS 41 k 2024-03-28T21:57:52.277 INFO:teuthology.orchestra.run.smithi063.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 CentOS-BaseOS 247 k 2024-03-28T21:57:52.277 INFO:teuthology.orchestra.run.smithi063.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T21:57:52.277 INFO:teuthology.orchestra.run.smithi063.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T21:57:52.277 INFO:teuthology.orchestra.run.smithi063.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T21:57:52.277 INFO:teuthology.orchestra.run.smithi063.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T21:57:52.277 INFO:teuthology.orchestra.run.smithi063.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T21:57:52.277 INFO:teuthology.orchestra.run.smithi063.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T21:57:52.277 INFO:teuthology.orchestra.run.smithi063.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T21:57:52.277 INFO:teuthology.orchestra.run.smithi063.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T21:57:52.277 INFO:teuthology.orchestra.run.smithi063.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T21:57:52.278 INFO:teuthology.orchestra.run.smithi063.stdout: librados2 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 3.4 M 2024-03-28T21:57:52.278 INFO:teuthology.orchestra.run.smithi063.stdout: librbd1 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 3.1 M 2024-03-28T21:57:52.278 INFO:teuthology.orchestra.run.smithi063.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T21:57:52.278 INFO:teuthology.orchestra.run.smithi063.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T21:57:52.278 INFO:teuthology.orchestra.run.smithi063.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T21:57:52.278 INFO:teuthology.orchestra.run.smithi063.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T21:57:52.278 INFO:teuthology.orchestra.run.smithi063.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T21:57:52.278 INFO:teuthology.orchestra.run.smithi063.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T21:57:52.278 INFO:teuthology.orchestra.run.smithi063.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T21:57:52.278 INFO:teuthology.orchestra.run.smithi063.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T21:57:52.278 INFO:teuthology.orchestra.run.smithi063.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T21:57:52.278 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-03-28T21:57:52.278 INFO:teuthology.orchestra.run.smithi063.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T21:57:52.279 INFO:teuthology.orchestra.run.smithi063.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T21:57:52.279 INFO:teuthology.orchestra.run.smithi063.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T21:57:52.279 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-common x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 21 M 2024-03-28T21:57:52.279 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 31 k 2024-03-28T21:57:52.279 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mds x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 2.4 M 2024-03-28T21:57:52.279 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 248 k 2024-03-28T21:57:52.279 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mon x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 4.7 M 2024-03-28T21:57:52.279 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-osd x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 17 M 2024-03-28T21:57:52.279 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 15 k 2024-03-28T21:57:52.279 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-selinux x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 25 k 2024-03-28T21:57:52.279 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T21:57:52.279 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T21:57:52.279 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T21:57:52.279 INFO:teuthology.orchestra.run.smithi063.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T21:57:52.280 INFO:teuthology.orchestra.run.smithi063.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T21:57:52.280 INFO:teuthology.orchestra.run.smithi063.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T21:57:52.280 INFO:teuthology.orchestra.run.smithi063.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T21:57:52.280 INFO:teuthology.orchestra.run.smithi063.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T21:57:52.280 INFO:teuthology.orchestra.run.smithi063.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T21:57:52.280 INFO:teuthology.orchestra.run.smithi063.stdout: libcephsqlite x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 163 k 2024-03-28T21:57:52.280 INFO:teuthology.orchestra.run.smithi063.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T21:57:52.280 INFO:teuthology.orchestra.run.smithi063.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T21:57:52.280 INFO:teuthology.orchestra.run.smithi063.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T21:57:52.280 INFO:teuthology.orchestra.run.smithi063.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T21:57:52.280 INFO:teuthology.orchestra.run.smithi063.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T21:57:52.280 INFO:teuthology.orchestra.run.smithi063.stdout: libradosstriper1 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 501 k 2024-03-28T21:57:52.280 INFO:teuthology.orchestra.run.smithi063.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T21:57:52.280 INFO:teuthology.orchestra.run.smithi063.stdout: librgw2 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 5.1 M 2024-03-28T21:57:52.281 INFO:teuthology.orchestra.run.smithi063.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T21:57:52.281 INFO:teuthology.orchestra.run.smithi063.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T21:57:52.281 INFO:teuthology.orchestra.run.smithi063.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T21:57:52.281 INFO:teuthology.orchestra.run.smithi063.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T21:57:52.281 INFO:teuthology.orchestra.run.smithi063.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T21:57:52.281 INFO:teuthology.orchestra.run.smithi063.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T21:57:52.282 INFO:teuthology.orchestra.run.smithi063.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T21:57:52.282 INFO:teuthology.orchestra.run.smithi063.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T21:57:52.282 INFO:teuthology.orchestra.run.smithi063.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T21:57:52.282 INFO:teuthology.orchestra.run.smithi063.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T21:57:52.282 INFO:teuthology.orchestra.run.smithi063.stdout: postgresql-private-libs x86_64 13.14-1.el9 CentOS-AppStream 132 k 2024-03-28T21:57:52.282 INFO:teuthology.orchestra.run.smithi063.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T21:57:52.282 INFO:teuthology.orchestra.run.smithi063.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T21:57:52.282 INFO:teuthology.orchestra.run.smithi063.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T21:57:52.283 INFO:teuthology.orchestra.run.smithi063.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T21:57:52.283 INFO:teuthology.orchestra.run.smithi063.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T21:57:52.283 INFO:teuthology.orchestra.run.smithi063.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T21:57:52.283 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T21:57:52.283 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-argparse x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 45 k 2024-03-28T21:57:52.283 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-common x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 131 k 2024-03-28T21:57:52.283 INFO:teuthology.orchestra.run.smithi063.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T21:57:52.283 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T21:57:52.283 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T21:57:52.283 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T21:57:52.283 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T21:57:52.283 INFO:teuthology.orchestra.run.smithi063.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T21:57:52.283 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T21:57:52.283 INFO:teuthology.orchestra.run.smithi063.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T21:57:52.284 INFO:teuthology.orchestra.run.smithi063.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T21:57:52.284 INFO:teuthology.orchestra.run.smithi063.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T21:57:52.284 INFO:teuthology.orchestra.run.smithi063.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T21:57:52.284 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T21:57:52.284 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T21:57:52.284 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T21:57:52.284 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T21:57:52.284 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T21:57:52.284 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T21:57:52.284 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T21:57:52.284 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T21:57:52.284 INFO:teuthology.orchestra.run.smithi063.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T21:57:52.284 INFO:teuthology.orchestra.run.smithi063.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T21:57:52.285 INFO:teuthology.orchestra.run.smithi063.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T21:57:52.285 INFO:teuthology.orchestra.run.smithi063.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T21:57:52.285 INFO:teuthology.orchestra.run.smithi063.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T21:57:52.285 INFO:teuthology.orchestra.run.smithi063.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T21:57:52.285 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T21:57:52.285 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T21:57:52.285 INFO:teuthology.orchestra.run.smithi063.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T21:57:52.285 INFO:teuthology.orchestra.run.smithi063.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T21:57:52.285 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T21:57:52.285 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T21:57:52.285 INFO:teuthology.orchestra.run.smithi063.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T21:57:52.285 INFO:teuthology.orchestra.run.smithi063.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T21:57:52.285 INFO:teuthology.orchestra.run.smithi063.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T21:57:52.285 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T21:57:52.285 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T21:57:52.286 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T21:57:52.286 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T21:57:52.286 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T21:57:52.286 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T21:57:52.286 INFO:teuthology.orchestra.run.smithi063.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T21:57:52.286 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T21:57:52.286 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T21:57:52.286 INFO:teuthology.orchestra.run.smithi063.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T21:57:52.286 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T21:57:52.286 INFO:teuthology.orchestra.run.smithi063.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T21:57:52.286 INFO:teuthology.orchestra.run.smithi063.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T21:57:52.286 INFO:teuthology.orchestra.run.smithi063.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T21:57:52.286 INFO:teuthology.orchestra.run.smithi063.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T21:57:52.287 INFO:teuthology.orchestra.run.smithi063.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T21:57:52.287 INFO:teuthology.orchestra.run.smithi063.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T21:57:52.287 INFO:teuthology.orchestra.run.smithi063.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T21:57:52.287 INFO:teuthology.orchestra.run.smithi063.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T21:57:52.287 INFO:teuthology.orchestra.run.smithi063.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T21:57:52.287 INFO:teuthology.orchestra.run.smithi063.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T21:57:52.287 INFO:teuthology.orchestra.run.smithi063.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-28T21:57:52.287 INFO:teuthology.orchestra.run.smithi063.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-28T21:57:52.287 INFO:teuthology.orchestra.run.smithi063.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T21:57:52.287 INFO:teuthology.orchestra.run.smithi063.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T21:57:52.287 INFO:teuthology.orchestra.run.smithi063.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T21:57:52.287 INFO:teuthology.orchestra.run.smithi063.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T21:57:52.287 INFO:teuthology.orchestra.run.smithi063.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T21:57:52.287 INFO:teuthology.orchestra.run.smithi063.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T21:57:52.287 INFO:teuthology.orchestra.run.smithi063.stdout:Installing weak dependencies: 2024-03-28T21:57:52.288 INFO:teuthology.orchestra.run.smithi063.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-28T21:57:52.288 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-28T21:57:52.288 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-28T21:57:52.288 INFO:teuthology.orchestra.run.smithi063.stdout:============================================================================================ 2024-03-28T21:57:52.288 INFO:teuthology.orchestra.run.smithi063.stdout:Install 150 Packages 2024-03-28T21:57:52.288 INFO:teuthology.orchestra.run.smithi063.stdout:Upgrade 24 Packages 2024-03-28T21:57:52.288 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-28T21:57:52.289 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 226 M 2024-03-28T21:57:52.289 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-28T21:57:52.615 INFO:teuthology.orchestra.run.smithi016.stdout:(2/174): ceph-base-19.0.0-1899.g323d23dc.el9.x8 3.5 MB/s | 5.3 MB 00:01 2024-03-28T21:57:52.657 INFO:teuthology.orchestra.run.smithi016.stdout:(3/174): ceph-fuse-19.0.0-1899.g323d23dc.el9.x8 702 kB/s | 889 kB 00:01 2024-03-28T21:57:52.715 INFO:teuthology.orchestra.run.smithi016.stdout:(4/174): ceph-immutable-object-cache-19.0.0-189 1.4 MB/s | 145 kB 00:00 2024-03-28T21:57:53.040 INFO:teuthology.orchestra.run.smithi016.stdout:(5/174): ceph-mgr-19.0.0-1899.g323d23dc.el9.x86 4.6 MB/s | 1.5 MB 00:00 2024-03-28T21:57:53.117 INFO:teuthology.orchestra.run.smithi063.stdout:(1/174): ceph-19.0.0-1899.g323d23dc.el9.x86_64. 13 kB/s | 6.6 kB 00:00 2024-03-28T21:57:53.949 INFO:teuthology.orchestra.run.smithi016.stdout:(6/174): ceph-mon-19.0.0-1899.g323d23dc.el9.x86 5.2 MB/s | 4.7 MB 00:00 2024-03-28T21:57:54.027 INFO:teuthology.orchestra.run.smithi063.stdout:(2/174): ceph-fuse-19.0.0-1899.g323d23dc.el9.x8 979 kB/s | 889 kB 00:00 2024-03-28T21:57:54.182 INFO:teuthology.orchestra.run.smithi016.stdout:(7/174): ceph-mds-19.0.0-1899.g323d23dc.el9.x86 1.5 MB/s | 2.4 MB 00:01 2024-03-28T21:57:54.217 INFO:teuthology.orchestra.run.smithi063.stdout:(3/174): ceph-immutable-object-cache-19.0.0-189 756 kB/s | 145 kB 00:00 2024-03-28T21:57:54.426 INFO:teuthology.orchestra.run.smithi063.stdout:(4/174): ceph-base-19.0.0-1899.g323d23dc.el9.x8 3.0 MB/s | 5.3 MB 00:01 2024-03-28T21:57:54.967 INFO:teuthology.orchestra.run.smithi063.stdout:(5/174): ceph-mgr-19.0.0-1899.g323d23dc.el9.x86 2.8 MB/s | 1.5 MB 00:00 2024-03-28T21:57:55.309 INFO:teuthology.orchestra.run.smithi063.stdout:(6/174): ceph-mds-19.0.0-1899.g323d23dc.el9.x86 2.2 MB/s | 2.4 MB 00:01 2024-03-28T21:57:56.818 INFO:teuthology.orchestra.run.smithi063.stdout:(7/174): ceph-mon-19.0.0-1899.g323d23dc.el9.x86 2.5 MB/s | 4.7 MB 00:01 2024-03-28T21:57:57.338 INFO:teuthology.orchestra.run.smithi053.stdout:lab-extras 47 kB/s | 1.7 kB 00:00 2024-03-28T21:57:57.491 INFO:teuthology.orchestra.run.smithi016.stdout:(8/174): ceph-radosgw-19.0.0-1899.g323d23dc.el9 3.0 MB/s | 9.8 MB 00:03 2024-03-28T21:57:57.991 INFO:teuthology.orchestra.run.smithi016.stdout:(9/174): ceph-selinux-19.0.0-1899.g323d23dc.el9 50 kB/s | 25 kB 00:00 2024-03-28T21:57:58.308 INFO:teuthology.orchestra.run.smithi016.stdout:(10/174): ceph-osd-19.0.0-1899.g323d23dc.el9.x8 3.9 MB/s | 17 MB 00:04 2024-03-28T21:57:58.708 INFO:teuthology.orchestra.run.smithi016.stdout:(11/174): cephfs-mirror-19.0.0-1899.g323d23dc.e 554 kB/s | 221 kB 00:00 2024-03-28T21:57:58.825 INFO:teuthology.orchestra.run.smithi016.stdout:(12/174): cephfs-shell-19.0.0-1899.g323d23dc.el 232 kB/s | 27 kB 00:00 2024-03-28T21:57:58.925 INFO:teuthology.orchestra.run.smithi016.stdout:(13/174): libcephfs-devel-19.0.0-1899.g323d23dc 317 kB/s | 32 kB 00:00 2024-03-28T21:57:59.767 INFO:teuthology.orchestra.run.smithi016.stdout:(14/174): libcephfs2-19.0.0-1899.g323d23dc.el9. 886 kB/s | 746 kB 00:00 2024-03-28T21:57:59.942 INFO:teuthology.orchestra.run.smithi053.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T21:57:59.943 INFO:teuthology.orchestra.run.smithi053.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T21:57:59.956 INFO:teuthology.orchestra.run.smithi053.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T21:57:59.957 INFO:teuthology.orchestra.run.smithi053.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T21:57:59.957 INFO:teuthology.orchestra.run.smithi053.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T21:57:59.958 INFO:teuthology.orchestra.run.smithi053.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T21:57:59.960 INFO:teuthology.orchestra.run.smithi053.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T21:57:59.965 INFO:teuthology.orchestra.run.smithi053.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T21:57:59.966 INFO:teuthology.orchestra.run.smithi053.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T21:57:59.993 INFO:teuthology.orchestra.run.smithi016.stdout:(15/174): libcephsqlite-19.0.0-1899.g323d23dc.e 725 kB/s | 163 kB 00:00 2024-03-28T21:58:00.049 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-28T21:58:00.076 INFO:teuthology.orchestra.run.smithi053.stdout:============================================================================================ 2024-03-28T21:58:00.076 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-03-28T21:58:00.076 INFO:teuthology.orchestra.run.smithi053.stdout:============================================================================================ 2024-03-28T21:58:00.076 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-28T21:58:00.076 INFO:teuthology.orchestra.run.smithi053.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T21:58:00.076 INFO:teuthology.orchestra.run.smithi053.stdout: ceph x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 6.6 k 2024-03-28T21:58:00.076 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-base x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 5.3 M 2024-03-28T21:58:00.076 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-fuse x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 889 k 2024-03-28T21:58:00.077 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 145 k 2024-03-28T21:58:00.077 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 1.5 M 2024-03-28T21:58:00.077 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 138 k 2024-03-28T21:58:00.077 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 3.5 M 2024-03-28T21:58:00.077 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 7.4 M 2024-03-28T21:58:00.077 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-rook noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 49 k 2024-03-28T21:58:00.077 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-radosgw x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 9.8 M 2024-03-28T21:58:00.077 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-test x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 47 M 2024-03-28T21:58:00.077 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-volume noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 264 k 2024-03-28T21:58:00.077 INFO:teuthology.orchestra.run.smithi053.stdout: cephadm noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 748 k 2024-03-28T21:58:00.077 INFO:teuthology.orchestra.run.smithi053.stdout: cephfs-mirror x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 221 k 2024-03-28T21:58:00.077 INFO:teuthology.orchestra.run.smithi053.stdout: cephfs-shell x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 27 k 2024-03-28T21:58:00.077 INFO:teuthology.orchestra.run.smithi053.stdout: cephfs-top noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 18 k 2024-03-28T21:58:00.078 INFO:teuthology.orchestra.run.smithi053.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T21:58:00.078 INFO:teuthology.orchestra.run.smithi053.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T21:58:00.078 INFO:teuthology.orchestra.run.smithi053.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T21:58:00.078 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs-devel x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 32 k 2024-03-28T21:58:00.078 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs2 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 746 k 2024-03-28T21:58:00.078 INFO:teuthology.orchestra.run.smithi053.stdout: librados-devel x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 127 k 2024-03-28T21:58:00.078 INFO:teuthology.orchestra.run.smithi053.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T21:58:00.078 INFO:teuthology.orchestra.run.smithi053.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T21:58:00.078 INFO:teuthology.orchestra.run.smithi053.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T21:58:00.078 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql x86_64 13.14-1.el9 CentOS-AppStream 1.5 M 2024-03-28T21:58:00.078 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-contrib x86_64 13.14-1.el9 CentOS-AppStream 817 k 2024-03-28T21:58:00.079 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-server x86_64 13.14-1.el9 CentOS-AppStream 5.7 M 2024-03-28T21:58:00.079 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cephfs x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 162 k 2024-03-28T21:58:00.079 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rados x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 321 k 2024-03-28T21:58:00.079 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rbd x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 299 k 2024-03-28T21:58:00.079 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rgw x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 100 k 2024-03-28T21:58:00.079 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-fuse x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 86 k 2024-03-28T21:58:00.079 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-mirror x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 3.1 M 2024-03-28T21:58:00.079 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-nbd x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 171 k 2024-03-28T21:58:00.079 INFO:teuthology.orchestra.run.smithi053.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T21:58:00.079 INFO:teuthology.orchestra.run.smithi053.stdout:Upgrading: 2024-03-28T21:58:00.079 INFO:teuthology.orchestra.run.smithi053.stdout: NetworkManager x86_64 1:1.47.2-1.el9 CentOS-BaseOS 2.3 M 2024-03-28T21:58:00.079 INFO:teuthology.orchestra.run.smithi053.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 CentOS-BaseOS 1.8 M 2024-03-28T21:58:00.079 INFO:teuthology.orchestra.run.smithi053.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 CentOS-BaseOS 41 k 2024-03-28T21:58:00.080 INFO:teuthology.orchestra.run.smithi053.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 CentOS-BaseOS 247 k 2024-03-28T21:58:00.080 INFO:teuthology.orchestra.run.smithi053.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T21:58:00.080 INFO:teuthology.orchestra.run.smithi053.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T21:58:00.080 INFO:teuthology.orchestra.run.smithi053.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T21:58:00.080 INFO:teuthology.orchestra.run.smithi053.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T21:58:00.080 INFO:teuthology.orchestra.run.smithi053.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T21:58:00.080 INFO:teuthology.orchestra.run.smithi053.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T21:58:00.080 INFO:teuthology.orchestra.run.smithi053.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T21:58:00.080 INFO:teuthology.orchestra.run.smithi053.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T21:58:00.080 INFO:teuthology.orchestra.run.smithi053.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T21:58:00.080 INFO:teuthology.orchestra.run.smithi053.stdout: librados2 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 3.4 M 2024-03-28T21:58:00.080 INFO:teuthology.orchestra.run.smithi053.stdout: librbd1 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 3.1 M 2024-03-28T21:58:00.080 INFO:teuthology.orchestra.run.smithi053.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T21:58:00.081 INFO:teuthology.orchestra.run.smithi053.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T21:58:00.081 INFO:teuthology.orchestra.run.smithi053.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T21:58:00.081 INFO:teuthology.orchestra.run.smithi053.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T21:58:00.081 INFO:teuthology.orchestra.run.smithi053.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T21:58:00.081 INFO:teuthology.orchestra.run.smithi053.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T21:58:00.081 INFO:teuthology.orchestra.run.smithi053.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T21:58:00.081 INFO:teuthology.orchestra.run.smithi053.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T21:58:00.081 INFO:teuthology.orchestra.run.smithi053.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T21:58:00.081 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-03-28T21:58:00.081 INFO:teuthology.orchestra.run.smithi053.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T21:58:00.081 INFO:teuthology.orchestra.run.smithi053.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T21:58:00.081 INFO:teuthology.orchestra.run.smithi053.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T21:58:00.082 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-common x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 21 M 2024-03-28T21:58:00.082 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 31 k 2024-03-28T21:58:00.082 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mds x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 2.4 M 2024-03-28T21:58:00.082 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 248 k 2024-03-28T21:58:00.082 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mon x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 4.7 M 2024-03-28T21:58:00.082 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-osd x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 17 M 2024-03-28T21:58:00.082 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 15 k 2024-03-28T21:58:00.082 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-selinux x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 25 k 2024-03-28T21:58:00.082 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T21:58:00.082 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T21:58:00.082 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T21:58:00.082 INFO:teuthology.orchestra.run.smithi053.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T21:58:00.083 INFO:teuthology.orchestra.run.smithi053.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T21:58:00.083 INFO:teuthology.orchestra.run.smithi053.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T21:58:00.083 INFO:teuthology.orchestra.run.smithi053.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T21:58:00.083 INFO:teuthology.orchestra.run.smithi053.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T21:58:00.083 INFO:teuthology.orchestra.run.smithi053.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T21:58:00.083 INFO:teuthology.orchestra.run.smithi053.stdout: libcephsqlite x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 163 k 2024-03-28T21:58:00.083 INFO:teuthology.orchestra.run.smithi053.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T21:58:00.083 INFO:teuthology.orchestra.run.smithi053.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T21:58:00.083 INFO:teuthology.orchestra.run.smithi053.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T21:58:00.083 INFO:teuthology.orchestra.run.smithi053.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T21:58:00.083 INFO:teuthology.orchestra.run.smithi053.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T21:58:00.083 INFO:teuthology.orchestra.run.smithi053.stdout: libradosstriper1 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 501 k 2024-03-28T21:58:00.084 INFO:teuthology.orchestra.run.smithi053.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T21:58:00.084 INFO:teuthology.orchestra.run.smithi053.stdout: librgw2 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 5.1 M 2024-03-28T21:58:00.084 INFO:teuthology.orchestra.run.smithi053.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T21:58:00.084 INFO:teuthology.orchestra.run.smithi053.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T21:58:00.084 INFO:teuthology.orchestra.run.smithi053.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T21:58:00.084 INFO:teuthology.orchestra.run.smithi053.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T21:58:00.084 INFO:teuthology.orchestra.run.smithi053.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T21:58:00.084 INFO:teuthology.orchestra.run.smithi053.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T21:58:00.084 INFO:teuthology.orchestra.run.smithi053.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T21:58:00.084 INFO:teuthology.orchestra.run.smithi053.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T21:58:00.084 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T21:58:00.084 INFO:teuthology.orchestra.run.smithi053.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T21:58:00.084 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-private-libs x86_64 13.14-1.el9 CentOS-AppStream 132 k 2024-03-28T21:58:00.085 INFO:teuthology.orchestra.run.smithi053.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T21:58:00.085 INFO:teuthology.orchestra.run.smithi053.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T21:58:00.085 INFO:teuthology.orchestra.run.smithi053.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T21:58:00.085 INFO:teuthology.orchestra.run.smithi053.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T21:58:00.085 INFO:teuthology.orchestra.run.smithi053.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T21:58:00.085 INFO:teuthology.orchestra.run.smithi053.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T21:58:00.085 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T21:58:00.085 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-argparse x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 45 k 2024-03-28T21:58:00.085 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-common x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 131 k 2024-03-28T21:58:00.085 INFO:teuthology.orchestra.run.smithi053.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T21:58:00.085 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T21:58:00.085 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T21:58:00.085 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T21:58:00.086 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T21:58:00.086 INFO:teuthology.orchestra.run.smithi053.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T21:58:00.086 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T21:58:00.086 INFO:teuthology.orchestra.run.smithi053.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T21:58:00.086 INFO:teuthology.orchestra.run.smithi053.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T21:58:00.086 INFO:teuthology.orchestra.run.smithi053.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T21:58:00.086 INFO:teuthology.orchestra.run.smithi053.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T21:58:00.086 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T21:58:00.086 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T21:58:00.086 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T21:58:00.086 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T21:58:00.087 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T21:58:00.087 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T21:58:00.087 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T21:58:00.087 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T21:58:00.087 INFO:teuthology.orchestra.run.smithi053.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T21:58:00.087 INFO:teuthology.orchestra.run.smithi053.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T21:58:00.087 INFO:teuthology.orchestra.run.smithi053.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T21:58:00.087 INFO:teuthology.orchestra.run.smithi053.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T21:58:00.087 INFO:teuthology.orchestra.run.smithi053.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T21:58:00.087 INFO:teuthology.orchestra.run.smithi053.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T21:58:00.087 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T21:58:00.087 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T21:58:00.087 INFO:teuthology.orchestra.run.smithi053.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T21:58:00.088 INFO:teuthology.orchestra.run.smithi053.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T21:58:00.088 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T21:58:00.088 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T21:58:00.088 INFO:teuthology.orchestra.run.smithi053.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T21:58:00.088 INFO:teuthology.orchestra.run.smithi053.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T21:58:00.088 INFO:teuthology.orchestra.run.smithi053.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T21:58:00.088 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T21:58:00.088 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T21:58:00.088 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T21:58:00.088 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T21:58:00.088 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T21:58:00.088 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T21:58:00.088 INFO:teuthology.orchestra.run.smithi053.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T21:58:00.089 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T21:58:00.089 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T21:58:00.089 INFO:teuthology.orchestra.run.smithi053.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T21:58:00.089 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T21:58:00.089 INFO:teuthology.orchestra.run.smithi053.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T21:58:00.089 INFO:teuthology.orchestra.run.smithi053.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T21:58:00.089 INFO:teuthology.orchestra.run.smithi053.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T21:58:00.089 INFO:teuthology.orchestra.run.smithi053.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T21:58:00.089 INFO:teuthology.orchestra.run.smithi053.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T21:58:00.089 INFO:teuthology.orchestra.run.smithi053.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T21:58:00.089 INFO:teuthology.orchestra.run.smithi053.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T21:58:00.089 INFO:teuthology.orchestra.run.smithi053.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T21:58:00.089 INFO:teuthology.orchestra.run.smithi053.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T21:58:00.089 INFO:teuthology.orchestra.run.smithi053.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T21:58:00.090 INFO:teuthology.orchestra.run.smithi053.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-28T21:58:00.090 INFO:teuthology.orchestra.run.smithi053.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-28T21:58:00.090 INFO:teuthology.orchestra.run.smithi053.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T21:58:00.090 INFO:teuthology.orchestra.run.smithi053.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T21:58:00.090 INFO:teuthology.orchestra.run.smithi053.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T21:58:00.090 INFO:teuthology.orchestra.run.smithi053.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T21:58:00.090 INFO:teuthology.orchestra.run.smithi053.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T21:58:00.090 INFO:teuthology.orchestra.run.smithi053.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T21:58:00.090 INFO:teuthology.orchestra.run.smithi053.stdout:Installing weak dependencies: 2024-03-28T21:58:00.090 INFO:teuthology.orchestra.run.smithi053.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-28T21:58:00.090 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-28T21:58:00.090 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-28T21:58:00.090 INFO:teuthology.orchestra.run.smithi053.stdout:============================================================================================ 2024-03-28T21:58:00.091 INFO:teuthology.orchestra.run.smithi053.stdout:Install 150 Packages 2024-03-28T21:58:00.091 INFO:teuthology.orchestra.run.smithi053.stdout:Upgrade 24 Packages 2024-03-28T21:58:00.091 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-28T21:58:00.091 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 226 M 2024-03-28T21:58:00.092 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-28T21:58:00.368 INFO:teuthology.orchestra.run.smithi016.stdout:(16/174): librados-devel-19.0.0-1899.g323d23dc. 338 kB/s | 127 kB 00:00 2024-03-28T21:58:00.989 INFO:teuthology.orchestra.run.smithi053.stdout:(1/174): ceph-19.0.0-1899.g323d23dc.el9.x86_64. 14 kB/s | 6.6 kB 00:00 2024-03-28T21:58:01.026 INFO:teuthology.orchestra.run.smithi016.stdout:(17/174): libradosstriper1-19.0.0-1899.g323d23d 761 kB/s | 501 kB 00:00 2024-03-28T21:58:01.178 INFO:teuthology.orchestra.run.smithi063.stdout:(8/174): ceph-osd-19.0.0-1899.g323d23dc.el9.x86 2.9 MB/s | 17 MB 00:05 2024-03-28T21:58:01.301 INFO:teuthology.orchestra.run.smithi063.stdout:(9/174): ceph-selinux-19.0.0-1899.g323d23dc.el9 206 kB/s | 25 kB 00:00 2024-03-28T21:58:02.298 INFO:teuthology.orchestra.run.smithi053.stdout:(2/174): ceph-fuse-19.0.0-1899.g323d23dc.el9.x8 679 kB/s | 889 kB 00:01 2024-03-28T21:58:02.573 INFO:teuthology.orchestra.run.smithi053.stdout:(3/174): ceph-immutable-object-cache-19.0.0-189 527 kB/s | 145 kB 00:00 2024-03-28T21:58:03.749 INFO:teuthology.orchestra.run.smithi053.stdout:(4/174): ceph-base-19.0.0-1899.g323d23dc.el9.x8 1.7 MB/s | 5.3 MB 00:03 2024-03-28T21:58:04.449 INFO:teuthology.orchestra.run.smithi053.stdout:(5/174): ceph-mgr-19.0.0-1899.g323d23dc.el9.x86 2.1 MB/s | 1.5 MB 00:00 2024-03-28T21:58:05.924 INFO:teuthology.orchestra.run.smithi053.stdout:(6/174): ceph-mds-19.0.0-1899.g323d23dc.el9.x86 722 kB/s | 2.4 MB 00:03 2024-03-28T21:58:06.001 INFO:teuthology.orchestra.run.smithi063.stdout:(10/174): ceph-common-19.0.0-1899.g323d23dc.el9 1.5 MB/s | 21 MB 00:13 2024-03-28T21:58:06.110 INFO:teuthology.orchestra.run.smithi016.stdout:(18/174): librgw2-19.0.0-1899.g323d23dc.el9.x86 1.0 MB/s | 5.1 MB 00:05 2024-03-28T21:58:06.257 INFO:teuthology.orchestra.run.smithi053.stdout:(7/174): ceph-mon-19.0.0-1899.g323d23dc.el9.x86 2.6 MB/s | 4.7 MB 00:01 2024-03-28T21:58:06.276 INFO:teuthology.orchestra.run.smithi063.stdout:(11/174): cephfs-mirror-19.0.0-1899.g323d23dc.e 806 kB/s | 221 kB 00:00 2024-03-28T21:58:06.293 INFO:teuthology.orchestra.run.smithi016.stdout:(19/174): python3-ceph-argparse-19.0.0-1899.g32 246 kB/s | 45 kB 00:00 2024-03-28T21:58:06.393 INFO:teuthology.orchestra.run.smithi063.stdout:(12/174): cephfs-shell-19.0.0-1899.g323d23dc.el 232 kB/s | 27 kB 00:00 2024-03-28T21:58:06.477 INFO:teuthology.orchestra.run.smithi016.stdout:(20/174): python3-ceph-common-19.0.0-1899.g323d 717 kB/s | 131 kB 00:00 2024-03-28T21:58:06.502 INFO:teuthology.orchestra.run.smithi063.stdout:(13/174): libcephfs-devel-19.0.0-1899.g323d23dc 293 kB/s | 32 kB 00:00 2024-03-28T21:58:06.677 INFO:teuthology.orchestra.run.smithi016.stdout:(21/174): python3-cephfs-19.0.0-1899.g323d23dc. 809 kB/s | 162 kB 00:00 2024-03-28T21:58:06.978 INFO:teuthology.orchestra.run.smithi016.stdout:(22/174): python3-rados-19.0.0-1899.g323d23dc.e 1.0 MB/s | 321 kB 00:00 2024-03-28T21:58:07.294 INFO:teuthology.orchestra.run.smithi063.stdout:(14/174): libcephfs2-19.0.0-1899.g323d23dc.el9. 942 kB/s | 746 kB 00:00 2024-03-28T21:58:07.311 INFO:teuthology.orchestra.run.smithi016.stdout:(23/174): python3-rbd-19.0.0-1899.g323d23dc.el9 896 kB/s | 299 kB 00:00 2024-03-28T21:58:07.486 INFO:teuthology.orchestra.run.smithi063.stdout:(15/174): libcephsqlite-19.0.0-1899.g323d23dc.e 852 kB/s | 163 kB 00:00 2024-03-28T21:58:07.503 INFO:teuthology.orchestra.run.smithi016.stdout:(24/174): python3-rgw-19.0.0-1899.g323d23dc.el9 520 kB/s | 100 kB 00:00 2024-03-28T21:58:07.612 INFO:teuthology.orchestra.run.smithi016.stdout:(25/174): rbd-fuse-19.0.0-1899.g323d23dc.el9.x8 792 kB/s | 86 kB 00:00 2024-03-28T21:58:07.686 INFO:teuthology.orchestra.run.smithi063.stdout:(16/174): librados-devel-19.0.0-1899.g323d23dc. 634 kB/s | 127 kB 00:00 2024-03-28T21:58:08.269 INFO:teuthology.orchestra.run.smithi063.stdout:(17/174): libradosstriper1-19.0.0-1899.g323d23d 859 kB/s | 501 kB 00:00 2024-03-28T21:58:09.270 INFO:teuthology.orchestra.run.smithi063.stdout:(18/174): ceph-radosgw-19.0.0-1899.g323d23dc.el 805 kB/s | 9.8 MB 00:12 2024-03-28T21:58:09.370 INFO:teuthology.orchestra.run.smithi063.stdout:(19/174): python3-ceph-argparse-19.0.0-1899.g32 452 kB/s | 45 kB 00:00 2024-03-28T21:58:09.537 INFO:teuthology.orchestra.run.smithi063.stdout:(20/174): python3-ceph-common-19.0.0-1899.g323d 788 kB/s | 131 kB 00:00 2024-03-28T21:58:09.820 INFO:teuthology.orchestra.run.smithi063.stdout:(21/174): python3-cephfs-19.0.0-1899.g323d23dc. 572 kB/s | 162 kB 00:00 2024-03-28T21:58:10.153 INFO:teuthology.orchestra.run.smithi016.stdout:(26/174): rbd-mirror-19.0.0-1899.g323d23dc.el9. 1.2 MB/s | 3.1 MB 00:02 2024-03-28T21:58:10.212 INFO:teuthology.orchestra.run.smithi063.stdout:(22/174): python3-rados-19.0.0-1899.g323d23dc.e 819 kB/s | 321 kB 00:00 2024-03-28T21:58:10.378 INFO:teuthology.orchestra.run.smithi016.stdout:(27/174): rbd-nbd-19.0.0-1899.g323d23dc.el9.x86 760 kB/s | 171 kB 00:00 2024-03-28T21:58:10.454 INFO:teuthology.orchestra.run.smithi063.stdout:(23/174): python3-rbd-19.0.0-1899.g323d23dc.el9 1.2 MB/s | 299 kB 00:00 2024-03-28T21:58:10.504 INFO:teuthology.orchestra.run.smithi016.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1899.g 251 kB/s | 31 kB 00:00 2024-03-28T21:58:10.571 INFO:teuthology.orchestra.run.smithi063.stdout:(24/174): python3-rgw-19.0.0-1899.g323d23dc.el9 852 kB/s | 100 kB 00:00 2024-03-28T21:58:10.672 INFO:teuthology.orchestra.run.smithi063.stdout:(25/174): rbd-fuse-19.0.0-1899.g323d23dc.el9.x8 857 kB/s | 86 kB 00:00 2024-03-28T21:58:10.721 INFO:teuthology.orchestra.run.smithi016.stdout:(29/174): ceph-mgr-cephadm-19.0.0-1899.g323d23d 636 kB/s | 138 kB 00:00 2024-03-28T21:58:11.099 INFO:teuthology.orchestra.run.smithi053.stdout:(8/174): ceph-radosgw-19.0.0-1899.g323d23dc.el9 2.0 MB/s | 9.8 MB 00:04 2024-03-28T21:58:11.174 INFO:teuthology.orchestra.run.smithi053.stdout:(9/174): ceph-selinux-19.0.0-1899.g323d23dc.el9 337 kB/s | 25 kB 00:00 2024-03-28T21:58:13.055 INFO:teuthology.orchestra.run.smithi063.stdout:(26/174): librgw2-19.0.0-1899.g323d23dc.el9.x86 1.1 MB/s | 5.1 MB 00:04 2024-03-28T21:58:13.174 INFO:teuthology.orchestra.run.smithi053.stdout:(10/174): ceph-common-19.0.0-1899.g323d23dc.el9 1.6 MB/s | 21 MB 00:12 2024-03-28T21:58:13.264 INFO:teuthology.orchestra.run.smithi063.stdout:(27/174): rbd-mirror-19.0.0-1899.g323d23dc.el9. 1.2 MB/s | 3.1 MB 00:02 2024-03-28T21:58:13.289 INFO:teuthology.orchestra.run.smithi063.stdout:(28/174): rbd-nbd-19.0.0-1899.g323d23dc.el9.x86 731 kB/s | 171 kB 00:00 2024-03-28T21:58:13.348 INFO:teuthology.orchestra.run.smithi063.stdout:(29/174): ceph-grafana-dashboards-19.0.0-1899.g 376 kB/s | 31 kB 00:00 2024-03-28T21:58:13.476 INFO:teuthology.orchestra.run.smithi053.stdout:(11/174): cephfs-mirror-19.0.0-1899.g323d23dc.e 736 kB/s | 221 kB 00:00 2024-03-28T21:58:13.498 INFO:teuthology.orchestra.run.smithi063.stdout:(30/174): ceph-mgr-cephadm-19.0.0-1899.g323d23d 662 kB/s | 138 kB 00:00 2024-03-28T21:58:13.616 INFO:teuthology.orchestra.run.smithi053.stdout:(12/174): cephfs-shell-19.0.0-1899.g323d23dc.el 193 kB/s | 27 kB 00:00 2024-03-28T21:58:13.750 INFO:teuthology.orchestra.run.smithi053.stdout:(13/174): libcephfs-devel-19.0.0-1899.g323d23dc 238 kB/s | 32 kB 00:00 2024-03-28T21:58:13.963 INFO:teuthology.orchestra.run.smithi016.stdout:(30/174): ceph-mgr-dashboard-19.0.0-1899.g323d2 1.1 MB/s | 3.5 MB 00:03 2024-03-28T21:58:14.183 INFO:teuthology.orchestra.run.smithi053.stdout:(14/174): libcephfs2-19.0.0-1899.g323d23dc.el9. 1.7 MB/s | 746 kB 00:00 2024-03-28T21:58:14.400 INFO:teuthology.orchestra.run.smithi053.stdout:(15/174): libcephsqlite-19.0.0-1899.g323d23dc.e 753 kB/s | 163 kB 00:00 2024-03-28T21:58:14.534 INFO:teuthology.orchestra.run.smithi053.stdout:(16/174): librados-devel-19.0.0-1899.g323d23dc. 951 kB/s | 127 kB 00:00 2024-03-28T21:58:14.943 INFO:teuthology.orchestra.run.smithi053.stdout:(17/174): libradosstriper1-19.0.0-1899.g323d23d 1.2 MB/s | 501 kB 00:00 2024-03-28T21:58:15.580 INFO:teuthology.orchestra.run.smithi016.stdout:(31/174): ceph-common-19.0.0-1899.g323d23dc.el9 860 kB/s | 21 MB 00:24 2024-03-28T21:58:15.946 INFO:teuthology.orchestra.run.smithi016.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1899.g32 678 kB/s | 248 kB 00:00 2024-03-28T21:58:15.990 INFO:teuthology.orchestra.run.smithi063.stdout:(31/174): ceph-mgr-dashboard-19.0.0-1899.g323d2 1.3 MB/s | 3.5 MB 00:02 2024-03-28T21:58:16.055 INFO:teuthology.orchestra.run.smithi016.stdout:(33/174): ceph-mgr-rook-19.0.0-1899.g323d23dc.e 456 kB/s | 49 kB 00:00 2024-03-28T21:58:16.164 INFO:teuthology.orchestra.run.smithi016.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1899.g3 141 kB/s | 15 kB 00:00 2024-03-28T21:58:16.232 INFO:teuthology.orchestra.run.smithi063.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1899.g32 1.0 MB/s | 248 kB 00:00 2024-03-28T21:58:16.332 INFO:teuthology.orchestra.run.smithi063.stdout:(33/174): ceph-mgr-rook-19.0.0-1899.g323d23dc.e 493 kB/s | 49 kB 00:00 2024-03-28T21:58:16.424 INFO:teuthology.orchestra.run.smithi063.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1899.g3 167 kB/s | 15 kB 00:00 2024-03-28T21:58:16.605 INFO:teuthology.orchestra.run.smithi016.stdout:(35/174): ceph-volume-19.0.0-1899.g323d23dc.el9 597 kB/s | 264 kB 00:00 2024-03-28T21:58:16.657 INFO:teuthology.orchestra.run.smithi063.stdout:(35/174): ceph-volume-19.0.0-1899.g323d23dc.el9 1.1 MB/s | 264 kB 00:00 2024-03-28T21:58:17.141 INFO:teuthology.orchestra.run.smithi063.stdout:(36/174): cephadm-19.0.0-1899.g323d23dc.el9.noa 1.5 MB/s | 748 kB 00:00 2024-03-28T21:58:17.233 INFO:teuthology.orchestra.run.smithi063.stdout:(37/174): cephfs-top-19.0.0-1899.g323d23dc.el9. 200 kB/s | 18 kB 00:00 2024-03-28T21:58:17.367 INFO:teuthology.orchestra.run.smithi063.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 4.9 MB/s | 662 kB 00:00 2024-03-28T21:58:17.422 INFO:teuthology.orchestra.run.smithi016.stdout:(36/174): cephadm-19.0.0-1899.g323d23dc.el9.noa 915 kB/s | 748 kB 00:00 2024-03-28T21:58:17.433 INFO:teuthology.orchestra.run.smithi063.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 14 MB/s | 920 kB 00:00 2024-03-28T21:58:17.467 INFO:teuthology.orchestra.run.smithi063.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-03-28T21:58:17.509 INFO:teuthology.orchestra.run.smithi063.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-28T21:58:17.523 INFO:teuthology.orchestra.run.smithi016.stdout:(37/174): cephfs-top-19.0.0-1899.g323d23dc.el9. 182 kB/s | 18 kB 00:00 2024-03-28T21:58:17.543 INFO:teuthology.orchestra.run.smithi063.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 950 kB/s | 32 kB 00:00 2024-03-28T21:58:17.651 INFO:teuthology.orchestra.run.smithi063.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 28 MB/s | 3.0 MB 00:00 2024-03-28T21:58:17.685 INFO:teuthology.orchestra.run.smithi063.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 514 kB/s | 17 kB 00:00 2024-03-28T21:58:17.698 INFO:teuthology.orchestra.run.smithi016.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 3.7 MB/s | 662 kB 00:00 2024-03-28T21:58:17.718 INFO:teuthology.orchestra.run.smithi063.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 303 kB/s | 10 kB 00:00 2024-03-28T21:58:17.744 INFO:teuthology.orchestra.run.smithi063.stdout:(46/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 6.2 MB/s | 159 kB 00:00 2024-03-28T21:58:17.748 INFO:teuthology.orchestra.run.smithi016.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 18 MB/s | 920 kB 00:00 2024-03-28T21:58:17.777 INFO:teuthology.orchestra.run.smithi063.stdout:(47/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-28T21:58:17.790 INFO:teuthology.orchestra.run.smithi016.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 106 kB 00:00 2024-03-28T21:58:17.819 INFO:teuthology.orchestra.run.smithi063.stdout:(48/174): librdkafka-1.6.1-102.el9.x86_64.rpm 15 MB/s | 661 kB 00:00 2024-03-28T21:58:17.832 INFO:teuthology.orchestra.run.smithi016.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-28T21:58:17.859 INFO:teuthology.orchestra.run.smithi053.stdout:(18/174): librgw2-19.0.0-1899.g323d23dc.el9.x86 1.8 MB/s | 5.1 MB 00:02 2024-03-28T21:58:17.866 INFO:teuthology.orchestra.run.smithi016.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 953 kB/s | 32 kB 00:00 2024-03-28T21:58:17.870 INFO:teuthology.orchestra.run.smithi063.stdout:(49/174): libtool-2.4.6-45.el9.x86_64.rpm 11 MB/s | 578 kB 00:00 2024-03-28T21:58:17.912 INFO:teuthology.orchestra.run.smithi063.stdout:(50/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-03-28T21:58:17.962 INFO:teuthology.orchestra.run.smithi063.stdout:(51/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-28T21:58:17.966 INFO:teuthology.orchestra.run.smithi016.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 30 MB/s | 3.0 MB 00:00 2024-03-28T21:58:17.976 INFO:teuthology.orchestra.run.smithi053.stdout:(19/174): python3-ceph-argparse-19.0.0-1899.g32 387 kB/s | 45 kB 00:00 2024-03-28T21:58:18.008 INFO:teuthology.orchestra.run.smithi016.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 413 kB/s | 17 kB 00:00 2024-03-28T21:58:18.012 INFO:teuthology.orchestra.run.smithi063.stdout:(52/174): lua-5.4.4-4.el9.x86_64.rpm 3.6 MB/s | 187 kB 00:00 2024-03-28T21:58:18.042 INFO:teuthology.orchestra.run.smithi016.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 301 kB/s | 10 kB 00:00 2024-03-28T21:58:18.055 INFO:teuthology.orchestra.run.smithi063.stdout:(53/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2024-03-28T21:58:18.084 INFO:teuthology.orchestra.run.smithi016.stdout:(46/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-28T21:58:18.093 INFO:teuthology.orchestra.run.smithi053.stdout:(20/174): python3-ceph-common-19.0.0-1899.g323d 1.1 MB/s | 131 kB 00:00 2024-03-28T21:58:18.105 INFO:teuthology.orchestra.run.smithi063.stdout:(54/174): openblas-0.3.21-2.el9.x86_64.rpm 683 kB/s | 34 kB 00:00 2024-03-28T21:58:18.134 INFO:teuthology.orchestra.run.smithi016.stdout:(47/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 884 kB/s | 44 kB 00:00 2024-03-28T21:58:18.193 INFO:teuthology.orchestra.run.smithi016.stdout:(48/174): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-28T21:58:18.234 INFO:teuthology.orchestra.run.smithi016.stdout:(49/174): libtool-2.4.6-45.el9.x86_64.rpm 14 MB/s | 578 kB 00:00 2024-03-28T21:58:18.260 INFO:teuthology.orchestra.run.smithi016.stdout:(50/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.8 MB/s | 46 kB 00:00 2024-03-28T21:58:18.285 INFO:teuthology.orchestra.run.smithi053.stdout:(21/174): python3-cephfs-19.0.0-1899.g323d23dc. 845 kB/s | 162 kB 00:00 2024-03-28T21:58:18.301 INFO:teuthology.orchestra.run.smithi016.stdout:(51/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-28T21:58:18.343 INFO:teuthology.orchestra.run.smithi016.stdout:(52/174): lua-5.4.4-4.el9.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2024-03-28T21:58:18.380 INFO:teuthology.orchestra.run.smithi063.stdout:(55/174): openblas-openmp-0.3.21-2.el9.x86_64.r 17 MB/s | 4.7 MB 00:00 2024-03-28T21:58:18.385 INFO:teuthology.orchestra.run.smithi016.stdout:(53/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2024-03-28T21:58:18.419 INFO:teuthology.orchestra.run.smithi016.stdout:(54/174): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-28T21:58:18.447 INFO:teuthology.orchestra.run.smithi063.stdout:(56/174): postgresql-13.14-1.el9.x86_64.rpm 23 MB/s | 1.5 MB 00:00 2024-03-28T21:58:18.502 INFO:teuthology.orchestra.run.smithi053.stdout:(22/174): python3-rados-19.0.0-1899.g323d23dc.e 1.4 MB/s | 321 kB 00:00 2024-03-28T21:58:18.523 INFO:teuthology.orchestra.run.smithi063.stdout:(57/174): postgresql-contrib-13.14-1.el9.x86_64 11 MB/s | 817 kB 00:00 2024-03-28T21:58:18.561 INFO:teuthology.orchestra.run.smithi016.stdout:(55/174): openblas-openmp-0.3.21-2.el9.x86_64.r 33 MB/s | 4.7 MB 00:00 2024-03-28T21:58:18.564 INFO:teuthology.orchestra.run.smithi063.stdout:(58/174): postgresql-private-libs-13.14-1.el9.x 3.1 MB/s | 132 kB 00:00 2024-03-28T21:58:18.637 INFO:teuthology.orchestra.run.smithi016.stdout:(56/174): postgresql-13.14-1.el9.x86_64.rpm 20 MB/s | 1.5 MB 00:00 2024-03-28T21:58:18.702 INFO:teuthology.orchestra.run.smithi053.stdout:(23/174): python3-rbd-19.0.0-1899.g323d23dc.el9 1.5 MB/s | 299 kB 00:00 2024-03-28T21:58:18.712 INFO:teuthology.orchestra.run.smithi016.stdout:(57/174): postgresql-contrib-13.14-1.el9.x86_64 11 MB/s | 817 kB 00:00 2024-03-28T21:58:18.731 INFO:teuthology.orchestra.run.smithi063.stdout:(59/174): postgresql-server-13.14-1.el9.x86_64. 34 MB/s | 5.7 MB 00:00 2024-03-28T21:58:18.746 INFO:teuthology.orchestra.run.smithi016.stdout:(58/174): postgresql-private-libs-13.14-1.el9.x 3.9 MB/s | 132 kB 00:00 2024-03-28T21:58:18.790 INFO:teuthology.orchestra.run.smithi063.stdout:(60/174): protobuf-3.14.0-13.el9.x86_64.rpm 17 MB/s | 1.0 MB 00:00 2024-03-28T21:58:18.827 INFO:teuthology.orchestra.run.smithi053.stdout:(24/174): python3-rgw-19.0.0-1899.g323d23dc.el9 798 kB/s | 100 kB 00:00 2024-03-28T21:58:18.848 INFO:teuthology.orchestra.run.smithi063.stdout:(61/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-28T21:58:18.912 INFO:teuthology.orchestra.run.smithi016.stdout:(59/174): postgresql-server-13.14-1.el9.x86_64. 34 MB/s | 5.7 MB 00:00 2024-03-28T21:58:18.930 INFO:teuthology.orchestra.run.smithi053.stdout:(25/174): rbd-fuse-19.0.0-1899.g323d23dc.el9.x8 856 kB/s | 86 kB 00:00 2024-03-28T21:58:18.979 INFO:teuthology.orchestra.run.smithi016.stdout:(60/174): protobuf-3.14.0-13.el9.x86_64.rpm 15 MB/s | 1.0 MB 00:00 2024-03-28T21:58:19.013 INFO:teuthology.orchestra.run.smithi016.stdout:(61/174): python3-attrs-20.3.0-7.el9.noarch.rpm 2.3 MB/s | 78 kB 00:00 2024-03-28T21:58:19.058 INFO:teuthology.orchestra.run.smithi063.stdout:(62/174): python3-babel-2.9.1-2.el9.noarch.rpm 28 MB/s | 5.8 MB 00:00 2024-03-28T21:58:19.116 INFO:teuthology.orchestra.run.smithi063.stdout:(63/174): python3-jsonpatch-1.21-16.el9.noarch. 420 kB/s | 24 kB 00:00 2024-03-28T21:58:19.158 INFO:teuthology.orchestra.run.smithi063.stdout:(64/174): python3-jsonpointer-2.0-4.el9.noarch. 436 kB/s | 18 kB 00:00 2024-03-28T21:58:19.172 INFO:teuthology.orchestra.run.smithi016.stdout:(62/174): python3-babel-2.9.1-2.el9.noarch.rpm 36 MB/s | 5.8 MB 00:00 2024-03-28T21:58:19.197 INFO:teuthology.orchestra.run.smithi016.stdout:(63/174): python3-jsonpatch-1.21-16.el9.noarch. 980 kB/s | 24 kB 00:00 2024-03-28T21:58:19.200 INFO:teuthology.orchestra.run.smithi063.stdout:(65/174): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-28T21:58:19.222 INFO:teuthology.orchestra.run.smithi016.stdout:(64/174): python3-jsonpointer-2.0-4.el9.noarch. 723 kB/s | 18 kB 00:00 2024-03-28T21:58:19.234 INFO:teuthology.orchestra.run.smithi063.stdout:(66/174): python3-markupsafe-1.1.1-12.el9.x86_6 942 kB/s | 32 kB 00:00 2024-03-28T21:58:19.256 INFO:teuthology.orchestra.run.smithi016.stdout:(65/174): python3-mako-1.1.4-6.el9.noarch.rpm 4.2 MB/s | 144 kB 00:00 2024-03-28T21:58:19.289 INFO:teuthology.orchestra.run.smithi016.stdout:(66/174): python3-markupsafe-1.1.1-12.el9.x86_6 944 kB/s | 32 kB 00:00 2024-03-28T21:58:19.401 INFO:teuthology.orchestra.run.smithi063.stdout:(67/174): ceph-mgr-diskprediction-local-19.0.0- 1.3 MB/s | 7.4 MB 00:05 2024-03-28T21:58:19.448 INFO:teuthology.orchestra.run.smithi016.stdout:(67/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 30 MB/s | 4.8 MB 00:00 2024-03-28T21:58:19.468 INFO:teuthology.orchestra.run.smithi063.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-28T21:58:19.490 INFO:teuthology.orchestra.run.smithi016.stdout:(68/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 5.1 MB/s | 217 kB 00:00 2024-03-28T21:58:19.501 INFO:teuthology.orchestra.run.smithi063.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.1 MB/s | 217 kB 00:00 2024-03-28T21:58:19.515 INFO:teuthology.orchestra.run.smithi016.stdout:(69/174): python3-oauthlib-3.1.1-5.el9.noarch.r 6.9 MB/s | 174 kB 00:00 2024-03-28T21:58:19.526 INFO:teuthology.orchestra.run.smithi063.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 2.9 MB/s | 174 kB 00:00 2024-03-28T21:58:19.549 INFO:teuthology.orchestra.run.smithi016.stdout:(70/174): python3-packaging-20.9-5.el9.noarch.r 2.0 MB/s | 69 kB 00:00 2024-03-28T21:58:19.552 INFO:teuthology.orchestra.run.smithi063.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-28T21:58:19.577 INFO:teuthology.orchestra.run.smithi063.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 820 kB/s | 41 kB 00:00 2024-03-28T21:58:19.582 INFO:teuthology.orchestra.run.smithi016.stdout:(71/174): python3-prettytable-0.7.2-27.el9.noar 1.2 MB/s | 41 kB 00:00 2024-03-28T21:58:19.602 INFO:teuthology.orchestra.run.smithi063.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-03-28T21:58:19.624 INFO:teuthology.orchestra.run.smithi016.stdout:(72/174): python3-protobuf-3.14.0-13.el9.noarch 5.6 MB/s | 238 kB 00:00 2024-03-28T21:58:19.628 INFO:teuthology.orchestra.run.smithi063.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-28T21:58:19.653 INFO:teuthology.orchestra.run.smithi063.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-28T21:58:19.658 INFO:teuthology.orchestra.run.smithi016.stdout:(73/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.9 MB/s | 132 kB 00:00 2024-03-28T21:58:19.678 INFO:teuthology.orchestra.run.smithi063.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 948 kB/s | 47 kB 00:00 2024-03-28T21:58:19.700 INFO:teuthology.orchestra.run.smithi016.stdout:(74/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.9 MB/s | 211 kB 00:00 2024-03-28T21:58:19.704 INFO:teuthology.orchestra.run.smithi063.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 855 kB/s | 43 kB 00:00 2024-03-28T21:58:19.734 INFO:teuthology.orchestra.run.smithi016.stdout:(75/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-28T21:58:19.792 INFO:teuthology.orchestra.run.smithi016.stdout:(76/174): python3-requests-oauthlib-1.3.0-12.el 733 kB/s | 43 kB 00:00 2024-03-28T21:58:19.962 INFO:teuthology.orchestra.run.smithi063.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 151 kB/s | 39 kB 00:00 2024-03-28T21:58:20.162 INFO:teuthology.orchestra.run.smithi053.stdout:(26/174): ceph-osd-19.0.0-1899.g323d23dc.el9.x8 1.2 MB/s | 17 MB 00:14 2024-03-28T21:58:20.196 INFO:teuthology.orchestra.run.smithi063.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 30 MB/s | 16 MB 00:00 2024-03-28T21:58:20.221 INFO:teuthology.orchestra.run.smithi063.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 160 kB/s | 41 kB 00:00 2024-03-28T21:58:20.246 INFO:teuthology.orchestra.run.smithi063.stdout:(81/174): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-03-28T21:58:20.253 INFO:teuthology.orchestra.run.smithi053.stdout:(27/174): rbd-nbd-19.0.0-1899.g323d23dc.el9.x86 1.8 MB/s | 171 kB 00:00 2024-03-28T21:58:20.273 INFO:teuthology.orchestra.run.smithi063.stdout:(82/174): qatlib-service-24.02.0-1.el9.x86_64.r 700 kB/s | 35 kB 00:00 2024-03-28T21:58:20.293 INFO:teuthology.orchestra.run.smithi016.stdout:(77/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 31 MB/s | 16 MB 00:00 2024-03-28T21:58:20.297 INFO:teuthology.orchestra.run.smithi063.stdout:(83/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 923 kB/s | 46 kB 00:00 2024-03-28T21:58:20.331 INFO:teuthology.orchestra.run.smithi063.stdout:(84/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-28T21:58:20.351 INFO:teuthology.orchestra.run.smithi016.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 664 kB/s | 39 kB 00:00 2024-03-28T21:58:20.356 INFO:teuthology.orchestra.run.smithi063.stdout:(85/174): uuid-1.6.2-55.el9.x86_64.rpm 963 kB/s | 56 kB 00:00 2024-03-28T21:58:20.362 INFO:teuthology.orchestra.run.smithi053.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1899.g 290 kB/s | 31 kB 00:00 2024-03-28T21:58:20.381 INFO:teuthology.orchestra.run.smithi063.stdout:(86/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-28T21:58:20.385 INFO:teuthology.orchestra.run.smithi016.stdout:(79/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.2 MB/s | 41 kB 00:00 2024-03-28T21:58:20.406 INFO:teuthology.orchestra.run.smithi063.stdout:(87/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 880 kB/s | 44 kB 00:00 2024-03-28T21:58:20.432 INFO:teuthology.orchestra.run.smithi063.stdout:(88/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-28T21:58:20.444 INFO:teuthology.orchestra.run.smithi016.stdout:(80/174): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 220 kB 00:00 2024-03-28T21:58:20.474 INFO:teuthology.orchestra.run.smithi063.stdout:(89/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-28T21:58:20.479 INFO:teuthology.orchestra.run.smithi053.stdout:(29/174): ceph-mgr-cephadm-19.0.0-1899.g323d23d 1.2 MB/s | 138 kB 00:00 2024-03-28T21:58:20.519 INFO:teuthology.orchestra.run.smithi016.stdout:(81/174): qatlib-service-24.02.0-1.el9.x86_64.r 465 kB/s | 35 kB 00:00 2024-03-28T21:58:20.533 INFO:teuthology.orchestra.run.smithi063.stdout:(90/174): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.1 MB 00:00 2024-03-28T21:58:20.594 INFO:teuthology.orchestra.run.smithi016.stdout:(82/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 617 kB/s | 46 kB 00:00 2024-03-28T21:58:20.616 INFO:teuthology.orchestra.run.smithi063.stdout:(91/174): python3-ply-3.11-14.el9.noarch.rpm 723 kB/s | 103 kB 00:00 2024-03-28T21:58:20.641 INFO:teuthology.orchestra.run.smithi063.stdout:(92/174): python3-pycparser-2.20-6.el9.noarch.r 1.1 MB/s | 124 kB 00:00 2024-03-28T21:58:20.678 INFO:teuthology.orchestra.run.smithi016.stdout:(83/174): socat-1.7.4.1-5.el9.x86_64.rpm 3.5 MB/s | 300 kB 00:00 2024-03-28T21:58:20.708 INFO:teuthology.orchestra.run.smithi063.stdout:(93/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.7 MB/s | 187 kB 00:00 2024-03-28T21:58:20.745 INFO:teuthology.orchestra.run.smithi016.stdout:(84/174): uuid-1.6.2-55.el9.x86_64.rpm 844 kB/s | 56 kB 00:00 2024-03-28T21:58:20.795 INFO:teuthology.orchestra.run.smithi016.stdout:(85/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-28T21:58:20.796 INFO:teuthology.orchestra.run.smithi053.stdout:(30/174): rbd-mirror-19.0.0-1899.g323d23dc.el9. 1.7 MB/s | 3.1 MB 00:01 2024-03-28T21:58:20.829 INFO:teuthology.orchestra.run.smithi016.stdout:(86/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-28T21:58:20.871 INFO:teuthology.orchestra.run.smithi016.stdout:(87/174): fuse-2.9.9-15.el9.x86_64.rpm 1.9 MB/s | 79 kB 00:00 2024-03-28T21:58:20.892 INFO:teuthology.orchestra.run.smithi063.stdout:(94/174): jq-1.6-16.el9.x86_64.rpm 748 kB/s | 187 kB 00:00 2024-03-28T21:58:20.929 INFO:teuthology.orchestra.run.smithi016.stdout:(88/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-28T21:58:21.030 INFO:teuthology.orchestra.run.smithi016.stdout:(89/174): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.1 MB 00:00 2024-03-28T21:58:21.034 INFO:teuthology.orchestra.run.smithi063.stdout:(95/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-03-28T21:58:21.076 INFO:teuthology.orchestra.run.smithi063.stdout:(96/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 803 kB 00:00 2024-03-28T21:58:21.088 INFO:teuthology.orchestra.run.smithi016.stdout:(90/174): python3-ply-3.11-14.el9.noarch.rpm 1.7 MB/s | 103 kB 00:00 2024-03-28T21:58:21.110 INFO:teuthology.orchestra.run.smithi063.stdout:(97/174): python3-requests-2.25.1-8.el9.noarch. 1.6 MB/s | 125 kB 00:00 2024-03-28T21:58:21.139 INFO:teuthology.orchestra.run.smithi016.stdout:(91/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-28T21:58:21.197 INFO:teuthology.orchestra.run.smithi016.stdout:(92/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.2 MB/s | 187 kB 00:00 2024-03-28T21:58:21.268 INFO:teuthology.orchestra.run.smithi063.stdout:(98/174): elfutils-libelf-devel-0.190-2.el9.x86 112 kB/s | 21 kB 00:00 2024-03-28T21:58:21.419 INFO:teuthology.orchestra.run.smithi063.stdout:(99/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 82 kB/s | 25 kB 00:00 2024-03-28T21:58:21.744 INFO:teuthology.orchestra.run.smithi063.stdout:(100/174): python3-devel-3.9.17-2.el9.x86_64.rp 755 kB/s | 245 kB 00:00 2024-03-28T21:58:21.797 INFO:teuthology.orchestra.run.smithi016.stdout:(93/174): jq-1.6-16.el9.x86_64.rpm 312 kB/s | 187 kB 00:00 2024-03-28T21:58:21.903 INFO:teuthology.orchestra.run.smithi063.stdout:(101/174): python3-jinja2-2.11.3-5.el9.noarch.r 1.5 MB/s | 248 kB 00:00 2024-03-28T21:58:21.946 INFO:teuthology.orchestra.run.smithi053.stdout:(31/174): ceph-mgr-dashboard-19.0.0-1899.g323d2 2.4 MB/s | 3.5 MB 00:01 2024-03-28T21:58:21.961 INFO:teuthology.orchestra.run.smithi063.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.6 MB/s | 94 kB 00:00 2024-03-28T21:58:21.964 INFO:teuthology.orchestra.run.smithi016.stdout:(94/174): ceph-mgr-diskprediction-local-19.0.0- 946 kB/s | 7.4 MB 00:08 2024-03-28T21:58:22.020 INFO:teuthology.orchestra.run.smithi063.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 382 kB/s | 22 kB 00:00 2024-03-28T21:58:22.080 INFO:teuthology.orchestra.run.smithi053.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1899.g32 1.8 MB/s | 248 kB 00:00 2024-03-28T21:58:22.128 INFO:teuthology.orchestra.run.smithi063.stdout:(104/174): protobuf-compiler-3.14.0-13.el9.x86_ 7.8 MB/s | 863 kB 00:00 2024-03-28T21:58:22.180 INFO:teuthology.orchestra.run.smithi053.stdout:(33/174): ceph-mgr-rook-19.0.0-1899.g323d23dc.e 494 kB/s | 49 kB 00:00 2024-03-28T21:58:22.254 INFO:teuthology.orchestra.run.smithi063.stdout:(105/174): abseil-cpp-20211102.0-3.el9.x86_64.r 4.3 MB/s | 548 kB 00:00 2024-03-28T21:58:22.289 INFO:teuthology.orchestra.run.smithi053.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1899.g3 142 kB/s | 15 kB 00:00 2024-03-28T21:58:22.290 INFO:teuthology.orchestra.run.smithi016.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 803 kB 00:00 2024-03-28T21:58:22.295 INFO:teuthology.orchestra.run.smithi063.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 7.2 MB/s | 309 kB 00:00 2024-03-28T21:58:22.329 INFO:teuthology.orchestra.run.smithi063.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 583 kB/s | 19 kB 00:00 2024-03-28T21:58:22.390 INFO:teuthology.orchestra.run.smithi016.stdout:(96/174): python3-requests-2.25.1-8.el9.noarch. 1.2 MB/s | 125 kB 00:00 2024-03-28T21:58:22.396 INFO:teuthology.orchestra.run.smithi063.stdout:(108/174): openssl-devel-3.0.7-27.el9.x86_64.rp 3.7 MB/s | 4.1 MB 00:01 2024-03-28T21:58:22.406 INFO:teuthology.orchestra.run.smithi053.stdout:(35/174): ceph-volume-19.0.0-1899.g323d23dc.el9 2.2 MB/s | 264 kB 00:00 2024-03-28T21:58:22.459 INFO:teuthology.orchestra.run.smithi063.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 417 kB/s | 26 kB 00:00 2024-03-28T21:58:22.474 INFO:teuthology.orchestra.run.smithi016.stdout:(97/174): elfutils-libelf-devel-0.190-2.el9.x86 258 kB/s | 21 kB 00:00 2024-03-28T21:58:22.499 INFO:teuthology.orchestra.run.smithi016.stdout:(98/174): libquadmath-11.4.1-3.el9.x86_64.rpm 366 kB/s | 195 kB 00:00 2024-03-28T21:58:22.516 INFO:teuthology.orchestra.run.smithi016.stdout:(99/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 608 kB/s | 25 kB 00:00 2024-03-28T21:58:22.525 INFO:teuthology.orchestra.run.smithi063.stdout:(110/174): libarrow-9.0.0-10.el9.x86_64.rpm 23 MB/s | 4.4 MB 00:00 2024-03-28T21:58:22.551 INFO:teuthology.orchestra.run.smithi063.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 530 kB/s | 49 kB 00:00 2024-03-28T21:58:22.576 INFO:teuthology.orchestra.run.smithi063.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-03-28T21:58:22.624 INFO:teuthology.orchestra.run.smithi016.stdout:(100/174): python3-devel-3.9.17-2.el9.x86_64.rp 2.2 MB/s | 245 kB 00:00 2024-03-28T21:58:22.677 INFO:teuthology.orchestra.run.smithi063.stdout:(113/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.2 MB/s | 838 kB 00:00 2024-03-28T21:58:22.691 INFO:teuthology.orchestra.run.smithi016.stdout:(101/174): python3-jinja2-2.11.3-5.el9.noarch.r 3.6 MB/s | 248 kB 00:00 2024-03-28T21:58:22.702 INFO:teuthology.orchestra.run.smithi063.stdout:(114/174): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-03-28T21:58:22.727 INFO:teuthology.orchestra.run.smithi063.stdout:(115/174): python3-asyncssh-2.13.2-5.el9.noarch 11 MB/s | 548 kB 00:00 2024-03-28T21:58:22.742 INFO:teuthology.orchestra.run.smithi016.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-03-28T21:58:22.761 INFO:teuthology.orchestra.run.smithi063.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 739 kB/s | 43 kB 00:00 2024-03-28T21:58:22.778 INFO:teuthology.orchestra.run.smithi063.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 639 kB/s | 32 kB 00:00 2024-03-28T21:58:22.795 INFO:teuthology.orchestra.run.smithi063.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 446 kB/s | 15 kB 00:00 2024-03-28T21:58:22.806 INFO:teuthology.orchestra.run.smithi053.stdout:(36/174): cephadm-19.0.0-1899.g323d23dc.el9.noa 1.8 MB/s | 748 kB 00:00 2024-03-28T21:58:22.817 INFO:teuthology.orchestra.run.smithi016.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 298 kB/s | 22 kB 00:00 2024-03-28T21:58:22.820 INFO:teuthology.orchestra.run.smithi063.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 4.0 MB/s | 172 kB 00:00 2024-03-28T21:58:22.845 INFO:teuthology.orchestra.run.smithi063.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-03-28T21:58:22.871 INFO:teuthology.orchestra.run.smithi063.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.9 MB/s | 303 kB 00:00 2024-03-28T21:58:22.896 INFO:teuthology.orchestra.run.smithi063.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 702 kB/s | 35 kB 00:00 2024-03-28T21:58:22.907 INFO:teuthology.orchestra.run.smithi053.stdout:(37/174): cephfs-top-19.0.0-1899.g323d23dc.el9. 183 kB/s | 18 kB 00:00 2024-03-28T21:58:22.921 INFO:teuthology.orchestra.run.smithi063.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 4.0 MB/s | 203 kB 00:00 2024-03-28T21:58:22.955 INFO:teuthology.orchestra.run.smithi063.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 4.3 MB/s | 144 kB 00:00 2024-03-28T21:58:22.996 INFO:teuthology.orchestra.run.smithi063.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 20 MB/s | 2.0 MB 00:00 2024-03-28T21:58:23.017 INFO:teuthology.orchestra.run.smithi016.stdout:(104/174): openssl-devel-3.0.7-27.el9.x86_64.rp 8.0 MB/s | 4.1 MB 00:00 2024-03-28T21:58:23.022 INFO:teuthology.orchestra.run.smithi063.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 160 kB/s | 11 kB 00:00 2024-03-28T21:58:23.047 INFO:teuthology.orchestra.run.smithi063.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 352 kB/s | 18 kB 00:00 2024-03-28T21:58:23.049 INFO:teuthology.orchestra.run.smithi053.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 4.6 MB/s | 662 kB 00:00 2024-03-28T21:58:23.072 INFO:teuthology.orchestra.run.smithi063.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 462 kB/s | 23 kB 00:00 2024-03-28T21:58:23.098 INFO:teuthology.orchestra.run.smithi063.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 387 kB/s | 19 kB 00:00 2024-03-28T21:58:23.123 INFO:teuthology.orchestra.run.smithi063.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 391 kB/s | 20 kB 00:00 2024-03-28T21:58:23.140 INFO:teuthology.orchestra.run.smithi053.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 9.8 MB/s | 920 kB 00:00 2024-03-28T21:58:23.142 INFO:teuthology.orchestra.run.smithi016.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.6 MB/s | 863 kB 00:00 2024-03-28T21:58:23.157 INFO:teuthology.orchestra.run.smithi063.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 1.4 MB/s | 46 kB 00:00 2024-03-28T21:58:23.174 INFO:teuthology.orchestra.run.smithi053.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-03-28T21:58:23.199 INFO:teuthology.orchestra.run.smithi063.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 10 MB/s | 1.0 MB 00:00 2024-03-28T21:58:23.216 INFO:teuthology.orchestra.run.smithi053.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-28T21:58:23.223 INFO:teuthology.orchestra.run.smithi063.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.2 MB/s | 79 kB 00:00 2024-03-28T21:58:23.249 INFO:teuthology.orchestra.run.smithi063.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-03-28T21:58:23.249 INFO:teuthology.orchestra.run.smithi053.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 953 kB/s | 32 kB 00:00 2024-03-28T21:58:23.274 INFO:teuthology.orchestra.run.smithi063.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-03-28T21:58:23.299 INFO:teuthology.orchestra.run.smithi063.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 327 kB/s | 16 kB 00:00 2024-03-28T21:58:23.318 INFO:teuthology.orchestra.run.smithi016.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.8 MB/s | 548 kB 00:00 2024-03-28T21:58:23.324 INFO:teuthology.orchestra.run.smithi063.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2024-03-28T21:58:23.350 INFO:teuthology.orchestra.run.smithi053.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 30 MB/s | 3.0 MB 00:00 2024-03-28T21:58:23.350 INFO:teuthology.orchestra.run.smithi063.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 562 kB/s | 28 kB 00:00 2024-03-28T21:58:23.368 INFO:teuthology.orchestra.run.smithi016.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 389 kB/s | 19 kB 00:00 2024-03-28T21:58:23.375 INFO:teuthology.orchestra.run.smithi063.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 612 kB/s | 31 kB 00:00 2024-03-28T21:58:23.400 INFO:teuthology.orchestra.run.smithi063.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 3.7 MB/s | 188 kB 00:00 2024-03-28T21:58:23.417 INFO:teuthology.orchestra.run.smithi053.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 257 kB/s | 17 kB 00:00 2024-03-28T21:58:23.426 INFO:teuthology.orchestra.run.smithi063.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-03-28T21:58:23.450 INFO:teuthology.orchestra.run.smithi053.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 301 kB/s | 10 kB 00:00 2024-03-28T21:58:23.451 INFO:teuthology.orchestra.run.smithi063.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 712 kB/s | 36 kB 00:00 2024-03-28T21:58:23.476 INFO:teuthology.orchestra.run.smithi063.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-03-28T21:58:23.493 INFO:teuthology.orchestra.run.smithi053.stdout:(46/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-28T21:58:23.510 INFO:teuthology.orchestra.run.smithi063.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-03-28T21:58:23.527 INFO:teuthology.orchestra.run.smithi053.stdout:(47/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-28T21:58:23.535 INFO:teuthology.orchestra.run.smithi063.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-03-28T21:58:23.560 INFO:teuthology.orchestra.run.smithi063.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 8.3 MB/s | 427 kB 00:00 2024-03-28T21:58:23.586 INFO:teuthology.orchestra.run.smithi063.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 397 kB/s | 20 kB 00:00 2024-03-28T21:58:23.594 INFO:teuthology.orchestra.run.smithi053.stdout:(48/174): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-28T21:58:23.632 INFO:teuthology.orchestra.run.smithi063.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 2.6 MB/s | 192 kB 00:00 2024-03-28T21:58:23.653 INFO:teuthology.orchestra.run.smithi053.stdout:(49/174): libtool-2.4.6-45.el9.x86_64.rpm 9.6 MB/s | 578 kB 00:00 2024-03-28T21:58:23.660 INFO:teuthology.orchestra.run.smithi016.stdout:(108/174): libarrow-9.0.0-10.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-03-28T21:58:23.687 INFO:teuthology.orchestra.run.smithi053.stdout:(50/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-28T21:58:23.703 INFO:teuthology.orchestra.run.smithi063.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-03-28T21:58:23.710 INFO:teuthology.orchestra.run.smithi016.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 520 kB/s | 26 kB 00:00 2024-03-28T21:58:23.737 INFO:teuthology.orchestra.run.smithi053.stdout:(51/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-28T21:58:23.761 INFO:teuthology.orchestra.run.smithi016.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 973 kB/s | 49 kB 00:00 2024-03-28T21:58:23.770 INFO:teuthology.orchestra.run.smithi053.stdout:(52/174): lua-5.4.4-4.el9.x86_64.rpm 5.5 MB/s | 187 kB 00:00 2024-03-28T21:58:23.813 INFO:teuthology.orchestra.run.smithi053.stdout:(53/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2024-03-28T21:58:23.819 INFO:teuthology.orchestra.run.smithi016.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-28T21:58:23.854 INFO:teuthology.orchestra.run.smithi053.stdout:(54/174): openblas-0.3.21-2.el9.x86_64.rpm 823 kB/s | 34 kB 00:00 2024-03-28T21:58:23.886 INFO:teuthology.orchestra.run.smithi016.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-03-28T21:58:23.996 INFO:teuthology.orchestra.run.smithi016.stdout:(113/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.5 MB/s | 838 kB 00:00 2024-03-28T21:58:24.063 INFO:teuthology.orchestra.run.smithi053.stdout:(55/174): openblas-openmp-0.3.21-2.el9.x86_64.r 23 MB/s | 4.7 MB 00:00 2024-03-28T21:58:24.087 INFO:teuthology.orchestra.run.smithi016.stdout:(114/174): python3-asyncssh-2.13.2-5.el9.noarch 5.8 MB/s | 548 kB 00:00 2024-03-28T21:58:24.154 INFO:teuthology.orchestra.run.smithi016.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 653 kB/s | 43 kB 00:00 2024-03-28T21:58:24.174 INFO:teuthology.orchestra.run.smithi053.stdout:(56/174): postgresql-13.14-1.el9.x86_64.rpm 14 MB/s | 1.5 MB 00:00 2024-03-28T21:58:24.221 INFO:teuthology.orchestra.run.smithi016.stdout:(116/174): python3-cachetools-4.2.4-1.el9.noarc 482 kB/s | 32 kB 00:00 2024-03-28T21:58:24.256 INFO:teuthology.orchestra.run.smithi053.stdout:(57/174): postgresql-contrib-13.14-1.el9.x86_64 9.6 MB/s | 817 kB 00:00 2024-03-28T21:58:24.289 INFO:teuthology.orchestra.run.smithi053.stdout:(58/174): postgresql-private-libs-13.14-1.el9.x 3.9 MB/s | 132 kB 00:00 2024-03-28T21:58:24.431 INFO:teuthology.orchestra.run.smithi053.stdout:(59/174): postgresql-server-13.14-1.el9.x86_64. 40 MB/s | 5.7 MB 00:00 2024-03-28T21:58:24.482 INFO:teuthology.orchestra.run.smithi053.stdout:(60/174): protobuf-3.14.0-13.el9.x86_64.rpm 20 MB/s | 1.0 MB 00:00 2024-03-28T21:58:24.513 INFO:teuthology.orchestra.run.smithi016.stdout:(117/174): python3-certifi-2021.10.8-2.el9.noar 51 kB/s | 15 kB 00:00 2024-03-28T21:58:24.515 INFO:teuthology.orchestra.run.smithi053.stdout:(61/174): python3-attrs-20.3.0-7.el9.noarch.rpm 2.3 MB/s | 78 kB 00:00 2024-03-28T21:58:24.674 INFO:teuthology.orchestra.run.smithi053.stdout:(62/174): python3-babel-2.9.1-2.el9.noarch.rpm 36 MB/s | 5.8 MB 00:00 2024-03-28T21:58:24.733 INFO:teuthology.orchestra.run.smithi053.stdout:(63/174): python3-jsonpatch-1.21-16.el9.noarch. 419 kB/s | 24 kB 00:00 2024-03-28T21:58:24.766 INFO:teuthology.orchestra.run.smithi053.stdout:(64/174): python3-jsonpointer-2.0-4.el9.noarch. 540 kB/s | 18 kB 00:00 2024-03-28T21:58:24.800 INFO:teuthology.orchestra.run.smithi053.stdout:(65/174): python3-mako-1.1.4-6.el9.noarch.rpm 4.2 MB/s | 144 kB 00:00 2024-03-28T21:58:24.825 INFO:teuthology.orchestra.run.smithi053.stdout:(66/174): python3-markupsafe-1.1.1-12.el9.x86_6 1.2 MB/s | 32 kB 00:00 2024-03-28T21:58:24.838 INFO:teuthology.orchestra.run.smithi016.stdout:(118/174): python3-cheroot-8.6.0-4.el9.noarch.r 528 kB/s | 172 kB 00:00 2024-03-28T21:58:24.930 INFO:teuthology.orchestra.run.smithi016.stdout:(119/174): python3-cherrypy-18.6.1-2.el9.noarch 3.8 MB/s | 358 kB 00:00 2024-03-28T21:58:24.959 INFO:teuthology.orchestra.run.smithi053.stdout:(67/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-28T21:58:25.001 INFO:teuthology.orchestra.run.smithi053.stdout:(68/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 5.1 MB/s | 217 kB 00:00 2024-03-28T21:58:25.005 INFO:teuthology.orchestra.run.smithi016.stdout:(120/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.0 MB/s | 303 kB 00:00 2024-03-28T21:58:25.035 INFO:teuthology.orchestra.run.smithi053.stdout:(69/174): python3-oauthlib-3.1.1-5.el9.noarch.r 5.1 MB/s | 174 kB 00:00 2024-03-28T21:58:25.064 INFO:teuthology.orchestra.run.smithi016.stdout:(121/174): python3-colorama-0.4.4-12.el9.noarch 605 kB/s | 35 kB 00:00 2024-03-28T21:58:25.068 INFO:teuthology.orchestra.run.smithi053.stdout:(70/174): python3-packaging-20.9-5.el9.noarch.r 2.0 MB/s | 69 kB 00:00 2024-03-28T21:58:25.102 INFO:teuthology.orchestra.run.smithi053.stdout:(71/174): python3-prettytable-0.7.2-27.el9.noar 1.2 MB/s | 41 kB 00:00 2024-03-28T21:58:25.131 INFO:teuthology.orchestra.run.smithi016.stdout:(122/174): python3-google-auth-2.28.2-1.el9.noa 3.0 MB/s | 203 kB 00:00 2024-03-28T21:58:25.144 INFO:teuthology.orchestra.run.smithi053.stdout:(72/174): python3-protobuf-3.14.0-13.el9.noarch 5.6 MB/s | 238 kB 00:00 2024-03-28T21:58:25.177 INFO:teuthology.orchestra.run.smithi053.stdout:(73/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.9 MB/s | 132 kB 00:00 2024-03-28T21:58:25.219 INFO:teuthology.orchestra.run.smithi053.stdout:(74/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.9 MB/s | 211 kB 00:00 2024-03-28T21:58:25.261 INFO:teuthology.orchestra.run.smithi053.stdout:(75/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-28T21:58:25.295 INFO:teuthology.orchestra.run.smithi053.stdout:(76/174): python3-requests-oauthlib-1.3.0-12.el 1.3 MB/s | 43 kB 00:00 2024-03-28T21:58:25.298 INFO:teuthology.orchestra.run.smithi016.stdout:(123/174): python3-grpcio-1.46.7-10.el9.x86_64. 12 MB/s | 2.0 MB 00:00 2024-03-28T21:58:25.365 INFO:teuthology.orchestra.run.smithi016.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 2.1 MB/s | 144 kB 00:00 2024-03-28T21:58:25.432 INFO:teuthology.orchestra.run.smithi016.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 160 kB/s | 11 kB 00:00 2024-03-28T21:58:25.490 INFO:teuthology.orchestra.run.smithi016.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 302 kB/s | 18 kB 00:00 2024-03-28T21:58:25.549 INFO:teuthology.orchestra.run.smithi016.stdout:(127/174): python3-jaraco-collections-3.0.0-8.e 396 kB/s | 23 kB 00:00 2024-03-28T21:58:25.608 INFO:teuthology.orchestra.run.smithi016.stdout:(128/174): python3-jaraco-functools-3.5.0-2.el9 333 kB/s | 19 kB 00:00 2024-03-28T21:58:25.645 INFO:teuthology.orchestra.run.smithi053.stdout:(77/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 45 MB/s | 16 MB 00:00 2024-03-28T21:58:25.666 INFO:teuthology.orchestra.run.smithi016.stdout:(129/174): python3-jaraco-text-3.2.0-6.el9.noar 337 kB/s | 20 kB 00:00 2024-03-28T21:58:25.679 INFO:teuthology.orchestra.run.smithi053.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-28T21:58:25.713 INFO:teuthology.orchestra.run.smithi053.stdout:(79/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.2 MB/s | 41 kB 00:00 2024-03-28T21:58:25.763 INFO:teuthology.orchestra.run.smithi053.stdout:(80/174): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-03-28T21:58:25.775 INFO:teuthology.orchestra.run.smithi016.stdout:(130/174): python3-kubernetes-26.1.0-2.el9.noar 9.4 MB/s | 1.0 MB 00:00 2024-03-28T21:58:25.830 INFO:teuthology.orchestra.run.smithi053.stdout:(81/174): qatlib-service-24.02.0-1.el9.x86_64.r 522 kB/s | 35 kB 00:00 2024-03-28T21:58:25.833 INFO:teuthology.orchestra.run.smithi016.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 793 kB/s | 46 kB 00:00 2024-03-28T21:58:25.888 INFO:teuthology.orchestra.run.smithi053.stdout:(82/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 796 kB/s | 46 kB 00:00 2024-03-28T21:58:25.900 INFO:teuthology.orchestra.run.smithi016.stdout:(132/174): python3-more-itertools-8.12.0-2.el9. 1.2 MB/s | 79 kB 00:00 2024-03-28T21:58:25.930 INFO:teuthology.orchestra.run.smithi053.stdout:(83/174): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-03-28T21:58:25.959 INFO:teuthology.orchestra.run.smithi016.stdout:(133/174): python3-natsort-7.1.1-5.el9.noarch.r 990 kB/s | 58 kB 00:00 2024-03-28T21:58:25.964 INFO:teuthology.orchestra.run.smithi053.stdout:(84/174): uuid-1.6.2-55.el9.x86_64.rpm 1.7 MB/s | 56 kB 00:00 2024-03-28T21:58:25.995 INFO:teuthology.orchestra.run.smithi063.stdout:(150/174): librbd1-19.0.0-1899.g323d23dc.el9.x8 1.4 MB/s | 3.1 MB 00:02 2024-03-28T21:58:25.998 INFO:teuthology.orchestra.run.smithi053.stdout:(85/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-28T21:58:26.031 INFO:teuthology.orchestra.run.smithi053.stdout:(86/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-28T21:58:26.034 INFO:teuthology.orchestra.run.smithi016.stdout:(134/174): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-03-28T21:58:26.045 INFO:teuthology.orchestra.run.smithi063.stdout:(151/174): librados2-19.0.0-1899.g323d23dc.el9. 1.4 MB/s | 3.4 MB 00:02 2024-03-28T21:58:26.065 INFO:teuthology.orchestra.run.smithi053.stdout:(87/174): fuse-2.9.9-15.el9.x86_64.rpm 2.3 MB/s | 79 kB 00:00 2024-03-28T21:58:26.093 INFO:teuthology.orchestra.run.smithi016.stdout:(135/174): python3-portend-3.1.0-2.el9.noarch.r 282 kB/s | 16 kB 00:00 2024-03-28T21:58:26.103 INFO:teuthology.orchestra.run.smithi063.stdout:(152/174): NetworkManager-1.47.2-1.el9.x86_64.r 21 MB/s | 2.3 MB 00:00 2024-03-28T21:58:26.107 INFO:teuthology.orchestra.run.smithi053.stdout:(88/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-28T21:58:26.152 INFO:teuthology.orchestra.run.smithi016.stdout:(136/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.5 MB/s | 90 kB 00:00 2024-03-28T21:58:26.174 INFO:teuthology.orchestra.run.smithi053.stdout:(89/174): python3-cryptography-36.0.1-4.el9.x86 17 MB/s | 1.1 MB 00:00 2024-03-28T21:58:26.179 INFO:teuthology.orchestra.run.smithi063.stdout:(153/174): NetworkManager-libnm-1.47.2-1.el9.x8 14 MB/s | 1.8 MB 00:00 2024-03-28T21:58:26.204 INFO:teuthology.orchestra.run.smithi063.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 415 kB/s | 41 kB 00:00 2024-03-28T21:58:26.208 INFO:teuthology.orchestra.run.smithi053.stdout:(90/174): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-28T21:58:26.210 INFO:teuthology.orchestra.run.smithi016.stdout:(137/174): python3-pyperclip-1.8.2-2.el9.noarch 483 kB/s | 28 kB 00:00 2024-03-28T21:58:26.229 INFO:teuthology.orchestra.run.smithi063.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 4.8 MB/s | 247 kB 00:00 2024-03-28T21:58:26.242 INFO:teuthology.orchestra.run.smithi053.stdout:(91/174): python3-pycparser-2.20-6.el9.noarch.r 3.6 MB/s | 124 kB 00:00 2024-03-28T21:58:26.263 INFO:teuthology.orchestra.run.smithi063.stdout:(156/174): xfsdump-3.1.12-4.el9.x86_64.rpm 5.4 MB/s | 321 kB 00:00 2024-03-28T21:58:26.269 INFO:teuthology.orchestra.run.smithi016.stdout:(138/174): python3-repoze-lru-0.7-16.el9.noarch 528 kB/s | 31 kB 00:00 2024-03-28T21:58:26.275 INFO:teuthology.orchestra.run.smithi053.stdout:(92/174): python3-urllib3-1.26.5-5.el9.noarch.r 5.5 MB/s | 187 kB 00:00 2024-03-28T21:58:26.336 INFO:teuthology.orchestra.run.smithi016.stdout:(139/174): python3-routes-2.5.1-5.el9.noarch.rp 2.8 MB/s | 188 kB 00:00 2024-03-28T21:58:26.355 INFO:teuthology.orchestra.run.smithi063.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 398 kB/s | 36 kB 00:00 2024-03-28T21:58:26.380 INFO:teuthology.orchestra.run.smithi063.stdout:(158/174): acl-2.3.1-4.el9.x86_64.rpm 476 kB/s | 71 kB 00:00 2024-03-28T21:58:26.394 INFO:teuthology.orchestra.run.smithi016.stdout:(140/174): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-03-28T21:58:26.420 INFO:teuthology.orchestra.run.smithi016.stdout:(141/174): gperftools-libs-2.9.1-2.el9.x86_64.r 94 kB/s | 309 kB 00:03 2024-03-28T21:58:26.442 INFO:teuthology.orchestra.run.smithi053.stdout:(93/174): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-03-28T21:58:26.453 INFO:teuthology.orchestra.run.smithi016.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 609 kB/s | 36 kB 00:00 2024-03-28T21:58:26.464 INFO:teuthology.orchestra.run.smithi063.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.7 MB/s | 192 kB 00:00 2024-03-28T21:58:26.479 INFO:teuthology.orchestra.run.smithi016.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 869 kB/s | 51 kB 00:00 2024-03-28T21:58:26.506 INFO:teuthology.orchestra.run.smithi063.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 551 kB/s | 23 kB 00:00 2024-03-28T21:58:26.521 INFO:teuthology.orchestra.run.smithi016.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.3 MB/s | 230 kB 00:00 2024-03-28T21:58:26.531 INFO:teuthology.orchestra.run.smithi063.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 1.7 MB/s | 256 kB 00:00 2024-03-28T21:58:26.546 INFO:teuthology.orchestra.run.smithi016.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.3 MB/s | 90 kB 00:00 2024-03-28T21:58:26.573 INFO:teuthology.orchestra.run.smithi063.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 108 kB 00:00 2024-03-28T21:58:26.596 INFO:teuthology.orchestra.run.smithi016.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.6 MB/s | 427 kB 00:00 2024-03-28T21:58:26.598 INFO:teuthology.orchestra.run.smithi063.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 2.3 MB/s | 155 kB 00:00 2024-03-28T21:58:26.621 INFO:teuthology.orchestra.run.smithi016.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 265 kB/s | 20 kB 00:00 2024-03-28T21:58:26.640 INFO:teuthology.orchestra.run.smithi063.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 2.0 MB/s | 136 kB 00:00 2024-03-28T21:58:26.666 INFO:teuthology.orchestra.run.smithi063.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 952 kB/s | 64 kB 00:00 2024-03-28T21:58:26.672 INFO:teuthology.orchestra.run.smithi016.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-28T21:58:26.691 INFO:teuthology.orchestra.run.smithi063.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 572 kB/s | 29 kB 00:00 2024-03-28T21:58:26.759 INFO:teuthology.orchestra.run.smithi053.stdout:(94/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 803 kB 00:00 2024-03-28T21:58:26.820 INFO:teuthology.orchestra.run.smithi016.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 8.0 MB/s | 1.6 MB 00:00 2024-03-28T21:58:26.846 INFO:teuthology.orchestra.run.smithi053.stdout:(95/174): libquadmath-11.4.1-3.el9.x86_64.rpm 2.2 MB/s | 195 kB 00:00 2024-03-28T21:58:26.896 INFO:teuthology.orchestra.run.smithi053.stdout:(96/174): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-03-28T21:58:26.899 INFO:teuthology.orchestra.run.smithi063.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 5.2 MB/s | 1.2 MB 00:00 2024-03-28T21:58:26.988 INFO:teuthology.orchestra.run.smithi053.stdout:(97/174): elfutils-libelf-devel-0.190-2.el9.x86 234 kB/s | 21 kB 00:00 2024-03-28T21:58:27.038 INFO:teuthology.orchestra.run.smithi053.stdout:(98/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 509 kB/s | 25 kB 00:00 2024-03-28T21:58:27.100 INFO:teuthology.orchestra.run.smithi063.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 5.3 MB/s | 2.2 MB 00:00 2024-03-28T21:58:27.205 INFO:teuthology.orchestra.run.smithi063.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 4.3 MB/s | 465 kB 00:00 2024-03-28T21:58:27.250 INFO:teuthology.orchestra.run.smithi063.stdout:(170/174): util-linux-2.37.4-18.el9.x86_64.rpm 6.5 MB/s | 2.3 MB 00:00 2024-03-28T21:58:27.267 INFO:teuthology.orchestra.run.smithi063.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 508 kB/s | 31 kB 00:00 2024-03-28T21:58:27.434 INFO:teuthology.orchestra.run.smithi063.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 5.9 MB/s | 1.1 MB 00:00 2024-03-28T21:58:27.459 INFO:teuthology.orchestra.run.smithi063.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 387 kB/s | 74 kB 00:00 2024-03-28T21:58:28.013 INFO:teuthology.orchestra.run.smithi053.stdout:(99/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 4.2 MB/s | 4.1 MB 00:00 2024-03-28T21:58:28.064 INFO:teuthology.orchestra.run.smithi053.stdout:(100/174): python3-devel-3.9.17-2.el9.x86_64.rp 4.8 MB/s | 245 kB 00:00 2024-03-28T21:58:28.131 INFO:teuthology.orchestra.run.smithi053.stdout:(101/174): python3-jinja2-2.11.3-5.el9.noarch.r 3.6 MB/s | 248 kB 00:00 2024-03-28T21:58:28.181 INFO:teuthology.orchestra.run.smithi053.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-03-28T21:58:28.356 INFO:teuthology.orchestra.run.smithi053.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 127 kB/s | 22 kB 00:00 2024-03-28T21:58:28.565 INFO:teuthology.orchestra.run.smithi053.stdout:(104/174): protobuf-compiler-3.14.0-13.el9.x86_ 4.0 MB/s | 863 kB 00:00 2024-03-28T21:58:28.890 INFO:teuthology.orchestra.run.smithi053.stdout:(105/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.6 MB/s | 548 kB 00:00 2024-03-28T21:58:28.982 INFO:teuthology.orchestra.run.smithi053.stdout:(106/174): ceph-mgr-diskprediction-local-19.0.0 925 kB/s | 7.4 MB 00:08 2024-03-28T21:58:29.024 INFO:teuthology.orchestra.run.smithi053.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 2.3 MB/s | 309 kB 00:00 2024-03-28T21:58:29.116 INFO:teuthology.orchestra.run.smithi053.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 146 kB/s | 19 kB 00:00 2024-03-28T21:58:29.225 INFO:teuthology.orchestra.run.smithi053.stdout:(109/174): libarrow-9.0.0-10.el9.x86_64.rpm 22 MB/s | 4.4 MB 00:00 2024-03-28T21:58:29.250 INFO:teuthology.orchestra.run.smithi053.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 194 kB/s | 26 kB 00:00 2024-03-28T21:58:29.284 INFO:teuthology.orchestra.run.smithi053.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 826 kB/s | 49 kB 00:00 2024-03-28T21:58:29.334 INFO:teuthology.orchestra.run.smithi053.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 3.0 MB/s | 151 kB 00:00 2024-03-28T21:58:29.360 INFO:teuthology.orchestra.run.smithi053.stdout:(113/174): libunwind-1.6.2-1.el9.x86_64.rpm 620 kB/s | 67 kB 00:00 2024-03-28T21:58:29.410 INFO:teuthology.orchestra.run.smithi053.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 11 MB/s | 838 kB 00:00 2024-03-28T21:58:29.468 INFO:teuthology.orchestra.run.smithi053.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 746 kB/s | 43 kB 00:00 2024-03-28T21:58:29.527 INFO:teuthology.orchestra.run.smithi053.stdout:(116/174): python3-cachetools-4.2.4-1.el9.noarc 551 kB/s | 32 kB 00:00 2024-03-28T21:58:29.552 INFO:teuthology.orchestra.run.smithi053.stdout:(117/174): python3-asyncssh-2.13.2-5.el9.noarch 2.8 MB/s | 548 kB 00:00 2024-03-28T21:58:29.568 INFO:teuthology.orchestra.run.smithi063.stdout:(174/174): ceph-test-19.0.0-1899.g323d23dc.el9. 1.7 MB/s | 47 MB 00:28 2024-03-28T21:58:29.573 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-28T21:58:29.573 INFO:teuthology.orchestra.run.smithi063.stdout:Total 6.1 MB/s | 226 MB 00:37 2024-03-28T21:58:29.577 INFO:teuthology.orchestra.run.smithi053.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 300 kB/s | 15 kB 00:00 2024-03-28T21:58:29.619 INFO:teuthology.orchestra.run.smithi053.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-03-28T21:58:29.645 INFO:teuthology.orchestra.run.smithi053.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 5.2 MB/s | 358 kB 00:00 2024-03-28T21:58:29.695 INFO:teuthology.orchestra.run.smithi053.stdout:(121/174): python3-colorama-0.4.4-12.el9.noarch 704 kB/s | 35 kB 00:00 2024-03-28T21:58:29.729 INFO:teuthology.orchestra.run.smithi053.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.7 MB/s | 303 kB 00:00 2024-03-28T21:58:29.754 INFO:teuthology.orchestra.run.smithi053.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 3.4 MB/s | 203 kB 00:00 2024-03-28T21:58:29.838 INFO:teuthology.orchestra.run.smithi053.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 1.7 MB/s | 144 kB 00:00 2024-03-28T21:58:29.888 INFO:teuthology.orchestra.run.smithi053.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 13 MB/s | 2.0 MB 00:00 2024-03-28T21:58:29.913 INFO:teuthology.orchestra.run.smithi053.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 142 kB/s | 11 kB 00:00 2024-03-28T21:58:29.947 INFO:teuthology.orchestra.run.smithi053.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 302 kB/s | 18 kB 00:00 2024-03-28T21:58:29.963 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-28T21:58:29.972 INFO:teuthology.orchestra.run.smithi053.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 394 kB/s | 23 kB 00:00 2024-03-28T21:58:29.997 INFO:teuthology.orchestra.run.smithi016.stdout:(150/174): librbd1-19.0.0-1899.g323d23dc.el9.x8 1.0 MB/s | 3.1 MB 00:03 2024-03-28T21:58:30.006 INFO:teuthology.orchestra.run.smithi053.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 333 kB/s | 19 kB 00:00 2024-03-28T21:58:30.040 INFO:teuthology.orchestra.run.smithi053.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 294 kB/s | 20 kB 00:00 2024-03-28T21:58:30.072 INFO:teuthology.orchestra.run.smithi016.stdout:(151/174): librados2-19.0.0-1899.g323d23dc.el9. 1.0 MB/s | 3.4 MB 00:03 2024-03-28T21:58:30.090 INFO:teuthology.orchestra.run.smithi053.stdout:(131/174): python3-kubernetes-26.1.0-2.el9.noar 12 MB/s | 1.0 MB 00:00 2024-03-28T21:58:30.115 INFO:teuthology.orchestra.run.smithi053.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 615 kB/s | 46 kB 00:00 2024-03-28T21:58:30.146 INFO:teuthology.orchestra.run.smithi016.stdout:(152/174): NetworkManager-1.47.2-1.el9.x86_64.r 15 MB/s | 2.3 MB 00:00 2024-03-28T21:58:30.149 INFO:teuthology.orchestra.run.smithi053.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-03-28T21:58:30.174 INFO:teuthology.orchestra.run.smithi053.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 986 kB/s | 58 kB 00:00 2024-03-28T21:58:30.198 INFO:teuthology.orchestra.run.smithi016.stdout:(153/174): NetworkManager-libnm-1.47.2-1.el9.x8 15 MB/s | 1.8 MB 00:00 2024-03-28T21:58:30.216 INFO:teuthology.orchestra.run.smithi053.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-03-28T21:58:30.223 INFO:teuthology.orchestra.run.smithi016.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 551 kB/s | 41 kB 00:00 2024-03-28T21:58:30.241 INFO:teuthology.orchestra.run.smithi053.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 245 kB/s | 16 kB 00:00 2024-03-28T21:58:30.248 INFO:teuthology.orchestra.run.smithi016.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 4.9 MB/s | 247 kB 00:00 2024-03-28T21:58:30.259 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-28T21:58:30.259 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-28T21:58:30.274 INFO:teuthology.orchestra.run.smithi016.stdout:(156/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-03-28T21:58:30.275 INFO:teuthology.orchestra.run.smithi053.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.5 MB/s | 90 kB 00:00 2024-03-28T21:58:30.302 INFO:teuthology.orchestra.run.smithi053.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 466 kB/s | 28 kB 00:00 2024-03-28T21:58:30.327 INFO:teuthology.orchestra.run.smithi053.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 586 kB/s | 31 kB 00:00 2024-03-28T21:58:30.361 INFO:teuthology.orchestra.run.smithi053.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 3.1 MB/s | 188 kB 00:00 2024-03-28T21:58:30.386 INFO:teuthology.orchestra.run.smithi053.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-03-28T21:58:30.420 INFO:teuthology.orchestra.run.smithi053.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 611 kB/s | 36 kB 00:00 2024-03-28T21:58:30.445 INFO:teuthology.orchestra.run.smithi053.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 871 kB/s | 51 kB 00:00 2024-03-28T21:58:30.479 INFO:teuthology.orchestra.run.smithi053.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-03-28T21:58:30.504 INFO:teuthology.orchestra.run.smithi053.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-03-28T21:58:30.532 INFO:teuthology.orchestra.run.smithi016.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 141 kB/s | 36 kB 00:00 2024-03-28T21:58:30.538 INFO:teuthology.orchestra.run.smithi053.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 7.1 MB/s | 427 kB 00:00 2024-03-28T21:58:30.633 INFO:teuthology.orchestra.run.smithi016.stdout:(158/174): acl-2.3.1-4.el9.x86_64.rpm 186 kB/s | 71 kB 00:00 2024-03-28T21:58:30.791 INFO:teuthology.orchestra.run.smithi016.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 743 kB/s | 192 kB 00:00 2024-03-28T21:58:30.901 INFO:teuthology.orchestra.run.smithi016.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 210 kB/s | 23 kB 00:00 2024-03-28T21:58:30.975 INFO:teuthology.orchestra.run.smithi016.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 747 kB/s | 256 kB 00:00 2024-03-28T21:58:31.013 INFO:teuthology.orchestra.run.smithi053.stdout:(147/174): ceph-test-19.0.0-1899.g323d23dc.el9. 2.4 MB/s | 47 MB 00:19 2024-03-28T21:58:31.017 INFO:teuthology.orchestra.run.smithi016.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 937 kB/s | 108 kB 00:00 2024-03-28T21:58:31.030 INFO:teuthology.orchestra.run.smithi053.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 38 kB/s | 20 kB 00:00 2024-03-28T21:58:31.055 INFO:teuthology.orchestra.run.smithi053.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 371 kB/s | 192 kB 00:00 2024-03-28T21:58:31.076 INFO:teuthology.orchestra.run.smithi016.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.5 MB/s | 155 kB 00:00 2024-03-28T21:58:31.126 INFO:teuthology.orchestra.run.smithi016.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 1.2 MB/s | 136 kB 00:00 2024-03-28T21:58:31.176 INFO:teuthology.orchestra.run.smithi016.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 637 kB/s | 64 kB 00:00 2024-03-28T21:58:31.227 INFO:teuthology.orchestra.run.smithi016.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 286 kB/s | 29 kB 00:00 2024-03-28T21:58:31.347 INFO:teuthology.orchestra.run.smithi053.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 4.7 MB/s | 1.6 MB 00:00 2024-03-28T21:58:31.439 INFO:teuthology.orchestra.run.smithi053.stdout:(151/174): NetworkManager-1.47.2-1.el9.x86_64.r 25 MB/s | 2.3 MB 00:00 2024-03-28T21:58:31.515 INFO:teuthology.orchestra.run.smithi053.stdout:(152/174): NetworkManager-libnm-1.47.2-1.el9.x8 24 MB/s | 1.8 MB 00:00 2024-03-28T21:58:31.532 INFO:teuthology.orchestra.run.smithi016.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 3.4 MB/s | 1.2 MB 00:00 2024-03-28T21:58:31.548 INFO:teuthology.orchestra.run.smithi053.stdout:(153/174): NetworkManager-team-1.47.2-1.el9.x86 1.2 MB/s | 41 kB 00:00 2024-03-28T21:58:31.598 INFO:teuthology.orchestra.run.smithi053.stdout:(154/174): NetworkManager-tui-1.47.2-1.el9.x86_ 4.8 MB/s | 247 kB 00:00 2024-03-28T21:58:31.648 INFO:teuthology.orchestra.run.smithi053.stdout:(155/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-03-28T21:58:31.690 INFO:teuthology.orchestra.run.smithi016.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 4.7 MB/s | 2.2 MB 00:00 2024-03-28T21:58:31.724 INFO:teuthology.orchestra.run.smithi053.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 951 kB/s | 71 kB 00:00 2024-03-28T21:58:31.774 INFO:teuthology.orchestra.run.smithi053.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 729 kB/s | 36 kB 00:00 2024-03-28T21:58:31.858 INFO:teuthology.orchestra.run.smithi053.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.3 MB/s | 192 kB 00:00 2024-03-28T21:58:31.941 INFO:teuthology.orchestra.run.smithi053.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 3.0 MB/s | 256 kB 00:00 2024-03-28T21:58:32.000 INFO:teuthology.orchestra.run.smithi053.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 396 kB/s | 23 kB 00:00 2024-03-28T21:58:32.041 INFO:teuthology.orchestra.run.smithi016.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 1.3 MB/s | 465 kB 00:00 2024-03-28T21:58:32.050 INFO:teuthology.orchestra.run.smithi053.stdout:(161/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.1 MB/s | 108 kB 00:00 2024-03-28T21:58:32.073 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-28T21:58:32.073 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-28T21:58:32.091 INFO:teuthology.orchestra.run.smithi016.stdout:(170/174): util-linux-2.37.4-18.el9.x86_64.rpm 4.1 MB/s | 2.3 MB 00:00 2024-03-28T21:58:32.109 INFO:teuthology.orchestra.run.smithi053.stdout:(162/174): libfdisk-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 155 kB 00:00 2024-03-28T21:58:32.133 INFO:teuthology.orchestra.run.smithi016.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 338 kB/s | 31 kB 00:00 2024-03-28T21:58:32.159 INFO:teuthology.orchestra.run.smithi053.stdout:(163/174): libmount-2.37.4-18.el9.x86_64.rpm 2.7 MB/s | 136 kB 00:00 2024-03-28T21:58:32.209 INFO:teuthology.orchestra.run.smithi016.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 9.2 MB/s | 1.1 MB 00:00 2024-03-28T21:58:32.209 INFO:teuthology.orchestra.run.smithi053.stdout:(164/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.2 MB/s | 64 kB 00:00 2024-03-28T21:58:32.234 INFO:teuthology.orchestra.run.smithi016.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 742 kB/s | 74 kB 00:00 2024-03-28T21:58:32.259 INFO:teuthology.orchestra.run.smithi053.stdout:(165/174): libuuid-2.37.4-18.el9.x86_64.rpm 576 kB/s | 29 kB 00:00 2024-03-28T21:58:32.510 INFO:teuthology.orchestra.run.smithi053.stdout:(166/174): openssl-3.0.7-27.el9.x86_64.rpm 4.8 MB/s | 1.2 MB 00:00 2024-03-28T21:58:32.810 INFO:teuthology.orchestra.run.smithi053.stdout:(167/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 7.2 MB/s | 2.2 MB 00:00 2024-03-28T21:58:33.085 INFO:teuthology.orchestra.run.smithi053.stdout:(168/174): util-linux-2.37.4-18.el9.x86_64.rpm 8.3 MB/s | 2.3 MB 00:00 2024-03-28T21:58:33.185 INFO:teuthology.orchestra.run.smithi053.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 4.5 MB/s | 465 kB 00:00 2024-03-28T21:58:33.236 INFO:teuthology.orchestra.run.smithi053.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 622 kB/s | 31 kB 00:00 2024-03-28T21:58:33.344 INFO:teuthology.orchestra.run.smithi053.stdout:(171/174): librbd1-19.0.0-1899.g323d23dc.el9.x8 1.4 MB/s | 3.1 MB 00:02 2024-03-28T21:58:33.378 INFO:teuthology.orchestra.run.smithi053.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 7.6 MB/s | 1.1 MB 00:00 2024-03-28T21:58:33.478 INFO:teuthology.orchestra.run.smithi053.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 558 kB/s | 74 kB 00:00 2024-03-28T21:58:33.784 INFO:teuthology.orchestra.run.smithi016.stdout:(174/174): ceph-test-19.0.0-1899.g323d23dc.el9. 1.3 MB/s | 47 MB 00:35 2024-03-28T21:58:33.788 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-28T21:58:33.788 INFO:teuthology.orchestra.run.smithi016.stdout:Total 5.2 MB/s | 226 MB 00:43 2024-03-28T21:58:34.146 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-28T21:58:34.443 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-28T21:58:34.444 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-28T21:58:36.212 INFO:teuthology.orchestra.run.smithi053.stdout:(174/174): librados2-19.0.0-1899.g323d23dc.el9. 669 kB/s | 3.4 MB 00:05 2024-03-28T21:58:36.217 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-28T21:58:36.217 INFO:teuthology.orchestra.run.smithi053.stdout:Total 6.2 MB/s | 226 MB 00:36 2024-03-28T21:58:36.536 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-28T21:58:36.536 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-28T21:58:36.596 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-28T21:58:36.901 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-28T21:58:36.901 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-28T21:58:38.739 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-28T21:58:38.740 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-28T21:58:39.136 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-28T21:58:39.263 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T21:58:39.298 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T21:58:39.328 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T21:58:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T21:58:39.378 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T21:58:39.440 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T21:58:39.474 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T21:58:39.507 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T21:58:39.550 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T21:58:39.896 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T21:58:39.941 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : librados2-2:19.0.0-1899.g323d23dc.el9.x86_64 10/198 2024-03-28T21:58:40.056 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librados2-2:19.0.0-1899.g323d23dc.el9.x86_64 10/198 2024-03-28T21:58:40.084 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libcephfs2-2:19.0.0-1899.g323d23dc.el9.x86_64 11/198 2024-03-28T21:58:40.176 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libcephfs2-2:19.0.0-1899.g323d23dc.el9.x86_64 11/198 2024-03-28T21:58:40.204 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rados-2:19.0.0-1899.g323d23dc.el9.x86_64 12/198 2024-03-28T21:58:40.245 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T21:58:40.430 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T21:58:40.477 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T21:58:40.496 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libcephsqlite-2:19.0.0-1899.g323d23dc.el9.x86_64 16/198 2024-03-28T21:58:40.585 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1899.g323d23dc.el9.x86_64 16/198 2024-03-28T21:58:40.614 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libradosstriper1-2:19.0.0-1899.g323d23dc.el9.x86 17/198 2024-03-28T21:58:40.678 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1899.g323d23dc.el9.x86 17/198 2024-03-28T21:58:40.718 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T21:58:40.741 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T21:58:40.758 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T21:58:40.924 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T21:58:40.954 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T21:58:40.986 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T21:58:41.093 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T21:58:41.182 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T21:58:41.240 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T21:58:41.317 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T21:58:41.366 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T21:58:41.595 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T21:58:41.708 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T21:58:41.802 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T21:58:41.838 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T21:58:41.865 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T21:58:41.900 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T21:58:41.978 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T21:58:42.029 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T21:58:42.083 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T21:58:42.150 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T21:58:42.199 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T21:58:42.243 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T21:58:42.275 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T21:58:42.302 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T21:58:42.375 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T21:58:42.407 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-ceph-common-2:19.0.0-1899.g323d23dc.el9. 39/198 2024-03-28T21:58:42.451 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-ceph-argparse-2:19.0.0-1899.g323d23dc.el 40/198 2024-03-28T21:58:42.479 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cephfs-2:19.0.0-1899.g323d23dc.el9.x86_6 41/198 2024-03-28T21:58:42.552 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T21:58:42.594 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T21:58:42.670 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T21:58:42.823 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T21:58:42.880 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T21:58:43.434 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T21:58:43.511 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T21:58:43.604 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T21:58:43.644 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T21:58:43.654 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T21:58:43.684 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T21:58:43.725 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T21:58:43.763 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librados-devel-2:19.0.0-1899.g323d23dc.el9.x86_6 53/198 2024-03-28T21:58:43.792 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T21:58:43.796 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-28T21:58:43.907 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T21:58:43.940 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T21:58:43.969 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T21:58:43.979 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T21:58:44.024 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T21:58:44.028 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T21:58:44.083 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T21:58:44.123 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T21:58:44.149 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T21:58:44.185 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T21:58:44.227 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T21:58:44.286 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T21:58:44.330 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T21:58:44.477 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T21:58:44.524 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T21:58:44.526 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T21:58:44.580 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T21:58:44.583 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : librados2-2:19.0.0-1899.g323d23dc.el9.x86_64 10/198 2024-03-28T21:58:44.654 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T21:58:44.691 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T21:58:44.713 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-2:19.0.0-1899.g323d23dc.el9.x86_64 10/198 2024-03-28T21:58:44.727 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T21:58:44.742 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs2-2:19.0.0-1899.g323d23dc.el9.x86_64 11/198 2024-03-28T21:58:44.776 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T21:58:44.819 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T21:58:44.834 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephfs2-2:19.0.0-1899.g323d23dc.el9.x86_64 11/198 2024-03-28T21:58:44.854 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T21:58:44.863 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rados-2:19.0.0-1899.g323d23dc.el9.x86_64 12/198 2024-03-28T21:58:44.893 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T21:58:44.911 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T21:58:45.033 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T21:58:45.083 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T21:58:45.102 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T21:58:45.112 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T21:58:45.154 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T21:58:45.164 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T21:58:45.190 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephsqlite-2:19.0.0-1899.g323d23dc.el9.x86_64 16/198 2024-03-28T21:58:45.252 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T21:58:45.279 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1899.g323d23dc.el9.x86_64 16/198 2024-03-28T21:58:45.295 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T21:58:45.308 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libradosstriper1-2:19.0.0-1899.g323d23dc.el9.x86 17/198 2024-03-28T21:58:45.335 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T21:58:45.360 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T21:58:45.364 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1899.g323d23dc.el9.x86 17/198 2024-03-28T21:58:45.385 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T21:58:45.403 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T21:58:45.426 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T21:58:45.441 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T21:58:45.461 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T21:58:45.590 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T21:58:45.602 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T21:58:45.624 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T21:58:45.655 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T21:58:45.701 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T21:58:45.743 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T21:58:45.801 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T21:58:45.805 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-28T21:58:45.842 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T21:58:45.857 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T21:58:45.899 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T21:58:45.933 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T21:58:45.962 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T21:58:45.972 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T21:58:45.981 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T21:58:46.009 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T21:58:46.022 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T21:58:46.049 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T21:58:46.076 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T21:58:46.096 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T21:58:46.104 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T21:58:46.109 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T21:58:46.143 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T21:58:46.153 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T21:58:46.185 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T21:58:46.234 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T21:58:46.278 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T21:58:46.310 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T21:58:46.340 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T21:58:46.357 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T21:58:46.369 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T21:58:46.403 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T21:58:46.443 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T21:58:46.468 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T21:58:46.474 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T21:58:46.504 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T21:58:46.565 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T21:58:46.583 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T21:58:46.590 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T21:58:46.599 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T21:58:46.614 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T21:58:46.626 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T21:58:46.627 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : librados2-2:19.0.0-1899.g323d23dc.el9.x86_64 10/198 2024-03-28T21:58:46.638 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T21:58:46.670 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T21:58:46.697 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T21:58:46.731 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T21:58:46.738 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T21:58:46.751 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librados2-2:19.0.0-1899.g323d23dc.el9.x86_64 10/198 2024-03-28T21:58:46.778 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephfs2-2:19.0.0-1899.g323d23dc.el9.x86_64 11/198 2024-03-28T21:58:46.799 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T21:58:46.823 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T21:58:46.836 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T21:58:46.853 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libcephfs2-2:19.0.0-1899.g323d23dc.el9.x86_64 11/198 2024-03-28T21:58:46.882 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rados-2:19.0.0-1899.g323d23dc.el9.x86_64 12/198 2024-03-28T21:58:46.903 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T21:58:46.912 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T21:58:46.919 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T21:58:46.936 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T21:58:46.954 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T21:58:46.979 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T21:58:46.985 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T21:58:47.011 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T21:58:47.032 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T21:58:47.039 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T21:58:47.107 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T21:58:47.111 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T21:58:47.141 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T21:58:47.146 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ceph-common-2:19.0.0-1899.g323d23dc.el9. 39/198 2024-03-28T21:58:47.163 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T21:58:47.189 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ceph-argparse-2:19.0.0-1899.g323d23dc.el 40/198 2024-03-28T21:58:47.190 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephsqlite-2:19.0.0-1899.g323d23dc.el9.x86_64 16/198 2024-03-28T21:58:47.196 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T21:58:47.216 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cephfs-2:19.0.0-1899.g323d23dc.el9.x86_6 41/198 2024-03-28T21:58:47.287 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T21:58:47.296 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1899.g323d23dc.el9.x86_64 16/198 2024-03-28T21:58:47.325 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libradosstriper1-2:19.0.0-1899.g323d23dc.el9.x86 17/198 2024-03-28T21:58:47.330 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T21:58:47.389 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1899.g323d23dc.el9.x86 17/198 2024-03-28T21:58:47.408 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T21:58:47.437 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T21:58:47.460 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T21:58:47.476 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T21:58:47.553 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T21:58:47.609 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T21:58:47.652 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T21:58:47.682 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T21:58:47.714 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T21:58:47.814 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T21:58:47.900 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T21:58:47.959 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T21:58:47.968 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T21:58:47.999 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T21:58:48.033 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T21:58:48.042 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T21:58:48.082 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T21:58:48.094 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T21:58:48.162 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T21:58:48.173 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T21:58:48.209 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T21:58:48.239 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T21:58:48.324 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T21:58:48.328 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T21:58:48.373 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T21:58:48.382 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T21:58:48.393 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T21:58:48.417 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T21:58:48.428 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T21:58:48.440 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T21:58:48.456 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T21:58:48.494 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librados-devel-2:19.0.0-1899.g323d23dc.el9.x86_6 53/198 2024-03-28T21:58:48.520 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T21:58:48.525 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T21:58:48.525 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T21:58:48.557 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T21:58:48.587 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T21:58:48.620 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T21:58:48.698 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T21:58:48.749 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T21:58:48.790 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T21:58:48.821 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T21:58:48.864 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T21:58:48.902 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T21:58:48.946 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T21:58:48.978 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T21:58:48.991 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T21:58:49.006 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T21:58:49.068 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T21:58:49.078 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T21:58:49.111 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-ceph-common-2:19.0.0-1899.g323d23dc.el9. 39/198 2024-03-28T21:58:49.113 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T21:58:49.155 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-ceph-argparse-2:19.0.0-1899.g323d23dc.el 40/198 2024-03-28T21:58:49.183 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cephfs-2:19.0.0-1899.g323d23dc.el9.x86_6 41/198 2024-03-28T21:58:49.188 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T21:58:49.256 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T21:58:49.258 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T21:58:49.298 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T21:58:49.306 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T21:58:49.307 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T21:58:49.355 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T21:58:49.375 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T21:58:49.429 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T21:58:49.466 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T21:58:49.503 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T21:58:49.514 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T21:58:49.551 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T21:58:49.577 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T21:58:49.586 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T21:58:49.629 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T21:58:49.671 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T21:58:49.802 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T21:58:49.844 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T21:58:49.873 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T21:58:49.916 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T21:58:50.013 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T21:58:50.057 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T21:58:50.097 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T21:58:50.122 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T21:58:50.130 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T21:58:50.147 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T21:58:50.215 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T21:58:50.226 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T21:58:50.296 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T21:58:50.348 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T21:58:50.352 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T21:58:50.357 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T21:58:50.388 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T21:58:50.429 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T21:58:50.460 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T21:58:50.466 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librados-devel-2:19.0.0-1899.g323d23dc.el9.x86_6 53/198 2024-03-28T21:58:50.495 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T21:58:50.498 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T21:58:50.548 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T21:58:50.592 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T21:58:50.612 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T21:58:50.656 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T21:58:50.687 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T21:58:50.718 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T21:58:50.735 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T21:58:50.746 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T21:58:50.802 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T21:58:50.852 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T21:58:50.914 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T21:58:50.973 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T21:58:50.995 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T21:58:51.018 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T21:58:51.033 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T21:58:51.065 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T21:58:51.075 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T21:58:51.096 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T21:58:51.109 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : librbd1-2:19.0.0-1899.g323d23dc.el9.x86_64 124/198 2024-03-28T21:58:51.124 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T21:58:51.154 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T21:58:51.159 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T21:58:51.198 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T21:58:51.200 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librbd1-2:19.0.0-1899.g323d23dc.el9.x86_64 124/198 2024-03-28T21:58:51.201 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T21:58:51.230 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T21:58:51.230 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rbd-2:19.0.0-1899.g323d23dc.el9.x86_64 125/198 2024-03-28T21:58:51.267 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T21:58:51.276 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T21:58:51.292 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T21:58:51.340 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T21:58:51.355 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T21:58:51.385 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T21:58:51.386 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T21:58:51.420 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T21:58:51.422 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T21:58:51.469 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T21:58:51.479 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T21:58:51.504 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T21:58:51.505 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T21:58:51.548 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T21:58:51.590 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T21:58:51.598 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T21:58:51.685 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T21:58:51.728 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T21:58:51.733 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T21:58:51.767 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T21:58:51.772 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T21:58:51.778 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T21:58:51.803 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librgw2-2:19.0.0-1899.g323d23dc.el9.x86_64 128/198 2024-03-28T21:58:51.808 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T21:58:51.818 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T21:58:51.851 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T21:58:51.874 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librgw2-2:19.0.0-1899.g323d23dc.el9.x86_64 128/198 2024-03-28T21:58:51.896 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rgw-2:19.0.0-1899.g323d23dc.el9.x86_64 129/198 2024-03-28T21:58:51.921 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T21:58:51.950 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T21:58:51.974 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T21:58:51.983 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T21:58:52.023 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T21:58:52.049 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T21:58:52.073 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T21:58:52.149 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T21:58:52.280 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T21:58:52.386 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T21:58:52.432 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T21:58:52.490 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T21:58:52.554 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T21:58:52.738 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T21:58:52.759 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T21:58:52.786 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T21:58:52.802 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T21:58:52.820 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T21:58:52.870 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T21:58:52.949 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T21:58:52.962 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T21:58:52.984 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T21:58:52.996 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T21:58:53.023 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T21:58:53.054 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T21:58:53.083 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T21:58:53.117 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T21:58:53.157 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T21:58:53.178 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T21:58:53.188 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T21:58:53.202 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T21:58:53.217 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T21:58:53.283 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T21:58:53.297 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T21:58:53.328 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T21:58:53.361 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T21:58:53.395 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T21:58:53.428 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T21:58:53.523 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T21:58:53.612 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T21:58:53.660 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T21:58:53.691 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T21:58:53.747 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T21:58:53.859 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T21:58:53.910 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T21:58:54.117 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T21:58:54.228 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T21:58:54.362 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 130/198 2024-03-28T21:58:54.401 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 130/198 2024-03-28T21:58:54.678 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T21:58:54.731 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T21:58:54.765 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T21:58:54.814 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T21:58:54.904 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T21:58:54.940 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T21:58:54.971 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 130/198 2024-03-28T21:58:55.004 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-base-2:19.0.0-1899.g323d23dc.el9.x86_64 131/198 2024-03-28T21:58:55.086 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-base-2:19.0.0-1899.g323d23dc.el9.x86_64 131/198 2024-03-28T21:58:55.086 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T21:58:55.086 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T21:58:55.086 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-28T21:58:55.126 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T21:58:55.154 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T21:58:55.213 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-selinux-2:19.0.0-1899.g323d23dc.el9.x86_64 132/198 2024-03-28T21:58:55.234 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T21:58:55.515 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T21:58:55.595 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T21:58:55.618 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T21:58:55.641 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T21:58:55.668 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T21:58:55.912 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T21:58:55.995 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T21:58:56.023 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : librbd1-2:19.0.0-1899.g323d23dc.el9.x86_64 124/198 2024-03-28T21:58:56.032 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T21:58:56.114 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-2:19.0.0-1899.g323d23dc.el9.x86_64 124/198 2024-03-28T21:58:56.145 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rbd-2:19.0.0-1899.g323d23dc.el9.x86_64 125/198 2024-03-28T21:58:56.181 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T21:58:56.658 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T21:58:56.684 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librgw2-2:19.0.0-1899.g323d23dc.el9.x86_64 128/198 2024-03-28T21:58:56.739 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librgw2-2:19.0.0-1899.g323d23dc.el9.x86_64 128/198 2024-03-28T21:58:56.761 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rgw-2:19.0.0-1899.g323d23dc.el9.x86_64 129/198 2024-03-28T21:58:57.321 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T21:58:57.396 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T21:58:57.427 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T21:58:57.458 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T21:58:57.485 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T21:58:57.814 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T21:58:57.840 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : librbd1-2:19.0.0-1899.g323d23dc.el9.x86_64 124/198 2024-03-28T21:58:57.931 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librbd1-2:19.0.0-1899.g323d23dc.el9.x86_64 124/198 2024-03-28T21:58:57.962 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rbd-2:19.0.0-1899.g323d23dc.el9.x86_64 125/198 2024-03-28T21:58:57.998 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T21:58:58.478 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T21:58:58.510 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librgw2-2:19.0.0-1899.g323d23dc.el9.x86_64 128/198 2024-03-28T21:58:58.564 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librgw2-2:19.0.0-1899.g323d23dc.el9.x86_64 128/198 2024-03-28T21:58:58.594 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rgw-2:19.0.0-1899.g323d23dc.el9.x86_64 129/198 2024-03-28T21:58:59.081 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 130/198 2024-03-28T21:58:59.118 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 130/198 2024-03-28T21:58:59.685 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 130/198 2024-03-28T21:58:59.722 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-base-2:19.0.0-1899.g323d23dc.el9.x86_64 131/198 2024-03-28T21:58:59.814 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-base-2:19.0.0-1899.g323d23dc.el9.x86_64 131/198 2024-03-28T21:58:59.814 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T21:58:59.814 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T21:58:59.814 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T21:58:59.849 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-selinux-2:19.0.0-1899.g323d23dc.el9.x86_64 132/198 2024-03-28T21:59:00.861 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 130/198 2024-03-28T21:59:00.900 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 130/198 2024-03-28T21:59:01.468 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 130/198 2024-03-28T21:59:01.512 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-base-2:19.0.0-1899.g323d23dc.el9.x86_64 131/198 2024-03-28T21:59:01.577 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-base-2:19.0.0-1899.g323d23dc.el9.x86_64 131/198 2024-03-28T21:59:01.577 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T21:59:01.578 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T21:59:01.578 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-28T21:59:01.605 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-selinux-2:19.0.0-1899.g323d23dc.el9.x86_64 132/198 2024-03-28T21:59:07.287 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1899.g323d23dc.el9.x86_64 132/198 2024-03-28T21:59:07.287 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /sys 2024-03-28T21:59:07.288 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /proc 2024-03-28T21:59:07.288 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /mnt 2024-03-28T21:59:07.288 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /var/tmp 2024-03-28T21:59:07.288 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /home 2024-03-28T21:59:07.288 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /root 2024-03-28T21:59:07.288 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /tmp 2024-03-28T21:59:07.288 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-28T21:59:07.450 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-volume-2:19.0.0-1899.g323d23dc.el9.noarch 133/198 2024-03-28T21:59:07.466 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-volume-2:19.0.0-1899.g323d23dc.el9.noarch 133/198 2024-03-28T21:59:07.466 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T21:59:07.466 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T21:59:07.466 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-28T21:59:09.003 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-osd-2:19.0.0-1899.g323d23dc.el9.x86_64 134/198 2024-03-28T21:59:09.029 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-osd-2:19.0.0-1899.g323d23dc.el9.x86_64 134/198 2024-03-28T21:59:09.030 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T21:59:09.030 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T21:59:09.030 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T21:59:09.030 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T21:59:09.030 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-28T21:59:09.262 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mds-2:19.0.0-1899.g323d23dc.el9.x86_64 135/198 2024-03-28T21:59:09.285 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mds-2:19.0.0-1899.g323d23dc.el9.x86_64 135/198 2024-03-28T21:59:09.286 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T21:59:09.286 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T21:59:09.286 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T21:59:09.286 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T21:59:09.286 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-28T21:59:09.756 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mon-2:19.0.0-1899.g323d23dc.el9.x86_64 136/198 2024-03-28T21:59:09.780 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mon-2:19.0.0-1899.g323d23dc.el9.x86_64 136/198 2024-03-28T21:59:09.780 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T21:59:09.780 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T21:59:09.780 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T21:59:09.780 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T21:59:09.780 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-28T21:59:09.874 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T21:59:10.251 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 138/198 2024-03-28T21:59:10.281 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 138/198 2024-03-28T21:59:10.313 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1899.g323d23dc.e 139/198 2024-03-28T21:59:10.375 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1899.g323d23dc. 140/198 2024-03-28T21:59:10.404 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1899.g323d23dc.el9.noa 141/198 2024-03-28T21:59:11.495 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1899.g323d23dc.el9.noa 141/198 2024-03-28T21:59:11.530 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1899.g323d23dc.el9.n 142/198 2024-03-28T21:59:12.195 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1899.g323d23dc.el9.x86_64 132/198 2024-03-28T21:59:12.196 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /sys 2024-03-28T21:59:12.196 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /proc 2024-03-28T21:59:12.196 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /mnt 2024-03-28T21:59:12.196 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /var/tmp 2024-03-28T21:59:12.196 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /home 2024-03-28T21:59:12.196 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /root 2024-03-28T21:59:12.196 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /tmp 2024-03-28T21:59:12.196 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T21:59:12.363 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-volume-2:19.0.0-1899.g323d23dc.el9.noarch 133/198 2024-03-28T21:59:12.377 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-volume-2:19.0.0-1899.g323d23dc.el9.noarch 133/198 2024-03-28T21:59:12.378 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T21:59:12.378 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T21:59:12.378 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T21:59:12.549 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1899.g323d23dc.el9.n 142/198 2024-03-28T21:59:12.573 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1899.g323 143/198 2024-03-28T21:59:12.679 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1899.g323 143/198 2024-03-28T21:59:12.861 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1899.g323d23dc.el 144/198 2024-03-28T21:59:12.891 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-2:19.0.0-1899.g323d23dc.el9.x86_64 145/198 2024-03-28T21:59:12.915 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1899.g323d23dc.el9.x86_64 145/198 2024-03-28T21:59:12.915 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T21:59:12.916 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T21:59:12.916 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T21:59:12.916 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T21:59:12.916 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-28T21:59:12.967 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-rook-2:19.0.0-1899.g323d23dc.el9.noarch 146/198 2024-03-28T21:59:12.978 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1899.g323d23dc.el9.noarch 146/198 2024-03-28T21:59:13.023 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-2:19.0.0-1899.g323d23dc.el9.x86_64 147/198 2024-03-28T21:59:13.600 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1899.g323d23dc.el9.x86_64 132/198 2024-03-28T21:59:13.601 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /sys 2024-03-28T21:59:13.601 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /proc 2024-03-28T21:59:13.601 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /mnt 2024-03-28T21:59:13.601 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /var/tmp 2024-03-28T21:59:13.601 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /home 2024-03-28T21:59:13.601 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /root 2024-03-28T21:59:13.601 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /tmp 2024-03-28T21:59:13.601 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-28T21:59:13.765 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-volume-2:19.0.0-1899.g323d23dc.el9.noarch 133/198 2024-03-28T21:59:13.781 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-volume-2:19.0.0-1899.g323d23dc.el9.noarch 133/198 2024-03-28T21:59:13.781 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T21:59:13.781 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T21:59:13.781 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-28T21:59:13.899 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-osd-2:19.0.0-1899.g323d23dc.el9.x86_64 134/198 2024-03-28T21:59:13.924 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-osd-2:19.0.0-1899.g323d23dc.el9.x86_64 134/198 2024-03-28T21:59:13.924 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T21:59:13.924 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T21:59:13.924 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T21:59:13.924 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T21:59:13.924 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T21:59:13.931 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T21:59:13.960 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-radosgw-2:19.0.0-1899.g323d23dc.el9.x86_64 149/198 2024-03-28T21:59:13.984 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1899.g323d23dc.el9.x86_64 149/198 2024-03-28T21:59:13.984 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T21:59:13.984 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T21:59:13.984 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T21:59:13.984 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T21:59:13.985 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-28T21:59:14.028 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1899.g323d2 150/198 2024-03-28T21:59:14.051 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1899.g323d2 150/198 2024-03-28T21:59:14.051 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T21:59:14.051 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T21:59:14.051 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-28T21:59:14.103 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : cephfs-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 151/198 2024-03-28T21:59:14.127 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 151/198 2024-03-28T21:59:14.127 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T21:59:14.127 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T21:59:14.127 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T21:59:14.127 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T21:59:14.127 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-28T21:59:14.150 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mds-2:19.0.0-1899.g323d23dc.el9.x86_64 135/198 2024-03-28T21:59:14.172 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mds-2:19.0.0-1899.g323d23dc.el9.x86_64 135/198 2024-03-28T21:59:14.172 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T21:59:14.172 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T21:59:14.172 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T21:59:14.172 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T21:59:14.172 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T21:59:14.454 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 152/198 2024-03-28T21:59:14.478 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 152/198 2024-03-28T21:59:14.478 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T21:59:14.478 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T21:59:14.478 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T21:59:14.478 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T21:59:14.479 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-28T21:59:14.634 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mon-2:19.0.0-1899.g323d23dc.el9.x86_64 136/198 2024-03-28T21:59:14.656 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mon-2:19.0.0-1899.g323d23dc.el9.x86_64 136/198 2024-03-28T21:59:14.656 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T21:59:14.656 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T21:59:14.656 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T21:59:14.656 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T21:59:14.657 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T21:59:14.753 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T21:59:15.145 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 138/198 2024-03-28T21:59:15.168 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 138/198 2024-03-28T21:59:15.200 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1899.g323d23dc.e 139/198 2024-03-28T21:59:15.262 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1899.g323d23dc. 140/198 2024-03-28T21:59:15.291 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1899.g323d23dc.el9.noa 141/198 2024-03-28T21:59:15.318 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-osd-2:19.0.0-1899.g323d23dc.el9.x86_64 134/198 2024-03-28T21:59:15.345 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-osd-2:19.0.0-1899.g323d23dc.el9.x86_64 134/198 2024-03-28T21:59:15.345 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T21:59:15.345 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T21:59:15.345 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T21:59:15.346 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T21:59:15.346 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-28T21:59:15.585 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mds-2:19.0.0-1899.g323d23dc.el9.x86_64 135/198 2024-03-28T21:59:15.608 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mds-2:19.0.0-1899.g323d23dc.el9.x86_64 135/198 2024-03-28T21:59:15.609 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T21:59:15.609 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T21:59:15.609 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T21:59:15.609 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T21:59:15.609 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-28T21:59:16.069 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mon-2:19.0.0-1899.g323d23dc.el9.x86_64 136/198 2024-03-28T21:59:16.093 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mon-2:19.0.0-1899.g323d23dc.el9.x86_64 136/198 2024-03-28T21:59:16.093 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T21:59:16.093 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T21:59:16.093 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T21:59:16.093 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T21:59:16.093 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-28T21:59:16.188 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T21:59:16.376 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1899.g323d23dc.el9.noa 141/198 2024-03-28T21:59:16.409 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1899.g323d23dc.el9.n 142/198 2024-03-28T21:59:16.615 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 138/198 2024-03-28T21:59:16.644 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 138/198 2024-03-28T21:59:16.676 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1899.g323d23dc.e 139/198 2024-03-28T21:59:16.738 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1899.g323d23dc. 140/198 2024-03-28T21:59:16.767 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1899.g323d23dc.el9.noa 141/198 2024-03-28T21:59:17.418 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1899.g323d23dc.el9.n 142/198 2024-03-28T21:59:17.443 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1899.g323 143/198 2024-03-28T21:59:17.549 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1899.g323 143/198 2024-03-28T21:59:17.713 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1899.g323d23dc.el 144/198 2024-03-28T21:59:17.746 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-2:19.0.0-1899.g323d23dc.el9.x86_64 145/198 2024-03-28T21:59:17.768 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1899.g323d23dc.el9.x86_64 145/198 2024-03-28T21:59:17.768 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T21:59:17.768 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T21:59:17.768 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T21:59:17.768 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T21:59:17.769 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T21:59:17.804 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-rook-2:19.0.0-1899.g323d23dc.el9.noarch 146/198 2024-03-28T21:59:17.815 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1899.g323d23dc.el9.noarch 146/198 2024-03-28T21:59:17.857 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-2:19.0.0-1899.g323d23dc.el9.x86_64 147/198 2024-03-28T21:59:17.861 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1899.g323d23dc.el9.noa 141/198 2024-03-28T21:59:17.893 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1899.g323d23dc.el9.n 142/198 2024-03-28T21:59:18.760 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T21:59:18.790 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-radosgw-2:19.0.0-1899.g323d23dc.el9.x86_64 149/198 2024-03-28T21:59:18.812 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1899.g323d23dc.el9.x86_64 149/198 2024-03-28T21:59:18.812 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T21:59:18.812 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T21:59:18.812 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T21:59:18.812 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T21:59:18.812 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T21:59:18.858 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1899.g323d2 150/198 2024-03-28T21:59:18.879 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1899.g323d2 150/198 2024-03-28T21:59:18.880 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T21:59:18.880 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T21:59:18.880 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T21:59:18.909 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1899.g323d23dc.el9.n 142/198 2024-03-28T21:59:18.933 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephfs-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 151/198 2024-03-28T21:59:18.944 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1899.g323 143/198 2024-03-28T21:59:18.955 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 151/198 2024-03-28T21:59:18.955 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T21:59:18.955 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T21:59:18.955 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T21:59:18.955 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T21:59:18.955 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T21:59:19.052 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1899.g323 143/198 2024-03-28T21:59:19.192 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-test-2:19.0.0-1899.g323d23dc.el9.x86_64 153/198 2024-03-28T21:59:19.223 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1899.g323d23dc.el 144/198 2024-03-28T21:59:19.233 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 154/198 2024-03-28T21:59:19.246 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-2:19.0.0-1899.g323d23dc.el9.x86_64 145/198 2024-03-28T21:59:19.263 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-nbd-2:19.0.0-1899.g323d23dc.el9.x86_64 155/198 2024-03-28T21:59:19.267 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 152/198 2024-03-28T21:59:19.271 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1899.g323d23dc.el9.x86_64 145/198 2024-03-28T21:59:19.271 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T21:59:19.271 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T21:59:19.271 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T21:59:19.271 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T21:59:19.271 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-28T21:59:19.289 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 152/198 2024-03-28T21:59:19.289 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T21:59:19.289 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T21:59:19.289 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T21:59:19.289 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T21:59:19.289 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T21:59:19.291 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T21:59:19.322 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-rook-2:19.0.0-1899.g323d23dc.el9.noarch 146/198 2024-03-28T21:59:19.333 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1899.g323d23dc.el9.noarch 146/198 2024-03-28T21:59:19.370 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-2:19.0.0-1899.g323d23dc.el9.x86_64 147/198 2024-03-28T21:59:19.480 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : cephfs-shell-2:19.0.0-1899.g323d23dc.el9.x86_64 157/198 2024-03-28T21:59:19.602 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T21:59:19.648 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 159/198 2024-03-28T21:59:19.686 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T21:59:19.730 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T21:59:19.781 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T21:59:19.814 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libcephfs-devel-2:19.0.0-1899.g323d23dc.el9.x86_ 163/198 2024-03-28T21:59:19.854 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T21:59:19.884 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T21:59:19.938 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T21:59:19.999 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T21:59:20.032 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T21:59:20.274 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T21:59:20.307 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-radosgw-2:19.0.0-1899.g323d23dc.el9.x86_64 149/198 2024-03-28T21:59:20.331 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1899.g323d23dc.el9.x86_64 149/198 2024-03-28T21:59:20.331 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T21:59:20.331 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T21:59:20.331 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T21:59:20.331 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T21:59:20.331 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-28T21:59:20.374 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1899.g323d2 150/198 2024-03-28T21:59:20.397 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1899.g323d2 150/198 2024-03-28T21:59:20.397 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T21:59:20.397 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T21:59:20.397 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-28T21:59:20.441 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : cephfs-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 151/198 2024-03-28T21:59:20.465 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 151/198 2024-03-28T21:59:20.465 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T21:59:20.465 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T21:59:20.465 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T21:59:20.466 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T21:59:20.466 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-28T21:59:20.626 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T21:59:20.663 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T21:59:20.683 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T21:59:20.792 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 152/198 2024-03-28T21:59:20.816 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 152/198 2024-03-28T21:59:20.816 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T21:59:20.816 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T21:59:20.816 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T21:59:20.816 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T21:59:20.816 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-28T21:59:21.572 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : cephfs-top-2:19.0.0-1899.g323d23dc.el9.noarch 170/198 2024-03-28T21:59:21.666 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T21:59:21.708 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T21:59:21.792 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T21:59:21.828 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T21:59:21.853 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T21:59:21.878 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T21:59:21.904 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T21:59:21.905 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T21:59:21.984 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T21:59:22.014 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T21:59:22.014 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T21:59:22.099 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T21:59:22.124 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T21:59:22.158 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T21:59:22.183 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T21:59:22.208 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T21:59:22.241 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T21:59:22.280 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T21:59:22.312 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T21:59:22.337 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T21:59:22.363 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T21:59:22.388 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T21:59:22.413 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T21:59:22.420 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T21:59:22.420 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T21:59:22.608 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T21:59:22.608 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T21:59:22.682 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T21:59:22.707 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T21:59:22.732 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T21:59:22.759 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T21:59:22.783 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T21:59:22.851 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T21:59:23.989 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-test-2:19.0.0-1899.g323d23dc.el9.x86_64 153/198 2024-03-28T21:59:24.031 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 154/198 2024-03-28T21:59:24.061 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-nbd-2:19.0.0-1899.g323d23dc.el9.x86_64 155/198 2024-03-28T21:59:24.088 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T21:59:24.283 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephfs-shell-2:19.0.0-1899.g323d23dc.el9.x86_64 157/198 2024-03-28T21:59:24.400 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T21:59:24.445 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 159/198 2024-03-28T21:59:24.499 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T21:59:24.553 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T21:59:24.596 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T21:59:24.629 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs-devel-2:19.0.0-1899.g323d23dc.el9.x86_ 163/198 2024-03-28T21:59:24.661 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T21:59:24.691 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T21:59:24.759 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T21:59:24.838 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T21:59:24.872 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T21:59:25.507 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-test-2:19.0.0-1899.g323d23dc.el9.x86_64 153/198 2024-03-28T21:59:25.545 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T21:59:25.548 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 154/198 2024-03-28T21:59:25.578 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-nbd-2:19.0.0-1899.g323d23dc.el9.x86_64 155/198 2024-03-28T21:59:25.579 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T21:59:25.599 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T21:59:25.605 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T21:59:25.797 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : cephfs-shell-2:19.0.0-1899.g323d23dc.el9.x86_64 157/198 2024-03-28T21:59:25.920 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T21:59:25.962 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 159/198 2024-03-28T21:59:26.001 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T21:59:26.045 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T21:59:26.079 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T21:59:26.112 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephfs-devel-2:19.0.0-1899.g323d23dc.el9.x86_ 163/198 2024-03-28T21:59:26.152 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T21:59:26.182 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T21:59:26.236 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T21:59:26.297 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T21:59:26.330 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T21:59:26.474 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephfs-top-2:19.0.0-1899.g323d23dc.el9.noarch 170/198 2024-03-28T21:59:26.558 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T21:59:26.600 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T21:59:26.725 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T21:59:26.753 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T21:59:26.779 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T21:59:26.812 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T21:59:26.841 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T21:59:26.841 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T21:59:26.862 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T21:59:26.895 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T21:59:26.916 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T21:59:26.929 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T21:59:26.967 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T21:59:26.967 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T21:59:27.051 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T21:59:27.087 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T21:59:27.119 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T21:59:27.145 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T21:59:27.170 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T21:59:27.205 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T21:59:27.251 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T21:59:27.283 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T21:59:27.325 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T21:59:27.350 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T21:59:27.376 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T21:59:27.401 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T21:59:27.407 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T21:59:27.407 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T21:59:27.596 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T21:59:27.596 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T21:59:27.679 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T21:59:27.703 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T21:59:27.735 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T21:59:27.763 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T21:59:27.788 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T21:59:27.818 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : cephfs-top-2:19.0.0-1899.g323d23dc.el9.noarch 170/198 2024-03-28T21:59:27.856 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T21:59:27.906 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T21:59:27.948 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T21:59:28.032 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T21:59:28.068 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T21:59:28.093 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T21:59:28.118 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T21:59:28.145 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T21:59:28.145 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T21:59:28.241 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T21:59:28.271 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T21:59:28.271 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T21:59:28.301 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T21:59:28.301 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-2:19.0.0-1899.g323d23dc.el9.x86_64 1/198 2024-03-28T21:59:28.301 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-base-2:19.0.0-1899.g323d23dc.el9.x86_64 2/198 2024-03-28T21:59:28.301 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 3/198 2024-03-28T21:59:28.301 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 4/198 2024-03-28T21:59:28.301 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1899.g323d2 5/198 2024-03-28T21:59:28.301 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mds-2:19.0.0-1899.g323d23dc.el9.x86_64 6/198 2024-03-28T21:59:28.301 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-2:19.0.0-1899.g323d23dc.el9.x86_64 7/198 2024-03-28T21:59:28.301 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mon-2:19.0.0-1899.g323d23dc.el9.x86_64 8/198 2024-03-28T21:59:28.302 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-osd-2:19.0.0-1899.g323d23dc.el9.x86_64 9/198 2024-03-28T21:59:28.302 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-radosgw-2:19.0.0-1899.g323d23dc.el9.x86_64 10/198 2024-03-28T21:59:28.302 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-selinux-2:19.0.0-1899.g323d23dc.el9.x86_64 11/198 2024-03-28T21:59:28.302 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-test-2:19.0.0-1899.g323d23dc.el9.x86_64 12/198 2024-03-28T21:59:28.302 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : cephfs-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 13/198 2024-03-28T21:59:28.302 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : cephfs-shell-2:19.0.0-1899.g323d23dc.el9.x86_64 14/198 2024-03-28T21:59:28.302 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libcephfs-devel-2:19.0.0-1899.g323d23dc.el9.x86_ 15/198 2024-03-28T21:59:28.302 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libcephfs2-2:19.0.0-1899.g323d23dc.el9.x86_64 16/198 2024-03-28T21:59:28.302 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libcephsqlite-2:19.0.0-1899.g323d23dc.el9.x86_64 17/198 2024-03-28T21:59:28.302 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados-devel-2:19.0.0-1899.g323d23dc.el9.x86_6 18/198 2024-03-28T21:59:28.302 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libradosstriper1-2:19.0.0-1899.g323d23dc.el9.x86 19/198 2024-03-28T21:59:28.302 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librgw2-2:19.0.0-1899.g323d23dc.el9.x86_64 20/198 2024-03-28T21:59:28.302 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1899.g323d23dc.el 21/198 2024-03-28T21:59:28.303 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-ceph-common-2:19.0.0-1899.g323d23dc.el9. 22/198 2024-03-28T21:59:28.303 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cephfs-2:19.0.0-1899.g323d23dc.el9.x86_6 23/198 2024-03-28T21:59:28.303 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rados-2:19.0.0-1899.g323d23dc.el9.x86_64 24/198 2024-03-28T21:59:28.303 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rbd-2:19.0.0-1899.g323d23dc.el9.x86_64 25/198 2024-03-28T21:59:28.303 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rgw-2:19.0.0-1899.g323d23dc.el9.x86_64 26/198 2024-03-28T21:59:28.303 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 27/198 2024-03-28T21:59:28.303 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 28/198 2024-03-28T21:59:28.303 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-nbd-2:19.0.0-1899.g323d23dc.el9.x86_64 29/198 2024-03-28T21:59:28.303 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1899.g323d23dc. 30/198 2024-03-28T21:59:28.303 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1899.g323d23dc.el9.noa 31/198 2024-03-28T21:59:28.303 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1899.g323d23dc.el9.n 32/198 2024-03-28T21:59:28.303 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1899.g323 33/198 2024-03-28T21:59:28.303 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1899.g323d23dc.el 34/198 2024-03-28T21:59:28.304 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1899.g323d23dc.el9.noarch 35/198 2024-03-28T21:59:28.304 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1899.g323d23dc.e 36/198 2024-03-28T21:59:28.304 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-volume-2:19.0.0-1899.g323d23dc.el9.noarch 37/198 2024-03-28T21:59:28.304 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 38/198 2024-03-28T21:59:28.304 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : cephfs-top-2:19.0.0-1899.g323d23dc.el9.noarch 39/198 2024-03-28T21:59:28.304 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T21:59:28.304 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T21:59:28.304 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T21:59:28.304 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T21:59:28.304 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T21:59:28.304 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T21:59:28.304 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T21:59:28.304 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T21:59:28.305 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T21:59:28.305 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T21:59:28.305 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T21:59:28.305 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T21:59:28.305 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T21:59:28.305 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T21:59:28.305 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T21:59:28.305 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T21:59:28.305 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T21:59:28.306 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T21:59:28.306 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : postgresql-13.14-1.el9.x86_64 58/198 2024-03-28T21:59:28.306 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 59/198 2024-03-28T21:59:28.306 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 60/198 2024-03-28T21:59:28.306 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 61/198 2024-03-28T21:59:28.306 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-28T21:59:28.306 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-28T21:59:28.306 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-28T21:59:28.306 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-28T21:59:28.306 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-28T21:59:28.306 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-28T21:59:28.306 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-28T21:59:28.306 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-28T21:59:28.307 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-28T21:59:28.307 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-28T21:59:28.307 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-28T21:59:28.307 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-28T21:59:28.307 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-28T21:59:28.307 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-28T21:59:28.307 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-28T21:59:28.307 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-28T21:59:28.307 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-28T21:59:28.307 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-28T21:59:28.307 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-28T21:59:28.307 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-28T21:59:28.307 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 82/198 2024-03-28T21:59:28.308 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 83/198 2024-03-28T21:59:28.308 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 84/198 2024-03-28T21:59:28.308 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 85/198 2024-03-28T21:59:28.308 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 86/198 2024-03-28T21:59:28.308 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 87/198 2024-03-28T21:59:28.308 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 88/198 2024-03-28T21:59:28.308 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T21:59:28.308 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 90/198 2024-03-28T21:59:28.308 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 91/198 2024-03-28T21:59:28.308 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-ply-3.11-14.el9.noarch 92/198 2024-03-28T21:59:28.308 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 93/198 2024-03-28T21:59:28.308 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/198 2024-03-28T21:59:28.308 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : jq-1.6-16.el9.x86_64 95/198 2024-03-28T21:59:28.308 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 96/198 2024-03-28T21:59:28.308 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 97/198 2024-03-28T21:59:28.309 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 98/198 2024-03-28T21:59:28.309 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 99/198 2024-03-28T21:59:28.309 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 100/198 2024-03-28T21:59:28.309 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 101/198 2024-03-28T21:59:28.309 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 102/198 2024-03-28T21:59:28.309 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 103/198 2024-03-28T21:59:28.309 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T21:59:28.309 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T21:59:28.309 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T21:59:28.309 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T21:59:28.309 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T21:59:28.309 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T21:59:28.309 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T21:59:28.309 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T21:59:28.310 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T21:59:28.310 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T21:59:28.310 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T21:59:28.310 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T21:59:28.310 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T21:59:28.310 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T21:59:28.310 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T21:59:28.310 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T21:59:28.310 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T21:59:28.310 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T21:59:28.310 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T21:59:28.310 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T21:59:28.310 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T21:59:28.310 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T21:59:28.311 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T21:59:28.311 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T21:59:28.311 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T21:59:28.311 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T21:59:28.311 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T21:59:28.311 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T21:59:28.311 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T21:59:28.311 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T21:59:28.311 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T21:59:28.311 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T21:59:28.311 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T21:59:28.311 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T21:59:28.311 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T21:59:28.311 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T21:59:28.312 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T21:59:28.312 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T21:59:28.312 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T21:59:28.312 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T21:59:28.312 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T21:59:28.312 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T21:59:28.312 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T21:59:28.312 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T21:59:28.312 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T21:59:28.312 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T21:59:28.312 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T21:59:28.312 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados2-2:19.0.0-1899.g323d23dc.el9.x86_64 151/198 2024-03-28T21:59:28.312 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T21:59:28.312 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librbd1-2:19.0.0-1899.g323d23dc.el9.x86_64 153/198 2024-03-28T21:59:28.313 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T21:59:28.313 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 155/198 2024-03-28T21:59:28.313 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-03-28T21:59:28.313 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T21:59:28.313 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T21:59:28.313 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T21:59:28.313 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T21:59:28.313 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T21:59:28.313 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T21:59:28.313 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 163/198 2024-03-28T21:59:28.313 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 164/198 2024-03-28T21:59:28.313 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T21:59:28.313 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T21:59:28.313 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T21:59:28.313 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T21:59:28.314 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T21:59:28.314 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T21:59:28.314 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T21:59:28.314 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T21:59:28.314 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T21:59:28.314 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T21:59:28.314 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T21:59:28.314 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T21:59:28.314 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T21:59:28.314 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T21:59:28.314 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T21:59:28.314 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T21:59:28.314 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T21:59:28.314 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T21:59:28.315 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T21:59:28.315 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T21:59:28.315 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T21:59:28.315 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T21:59:28.315 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T21:59:28.315 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T21:59:28.315 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T21:59:28.315 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T21:59:28.315 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T21:59:28.315 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T21:59:28.315 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T21:59:28.315 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T21:59:28.315 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T21:59:28.316 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T21:59:28.316 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T21:59:28.363 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T21:59:28.390 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T21:59:28.423 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T21:59:28.448 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T21:59:28.473 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T21:59:28.507 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T21:59:28.554 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T21:59:28.595 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T21:59:28.620 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T21:59:28.645 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T21:59:28.670 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T21:59:28.695 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T21:59:28.702 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T21:59:28.702 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T21:59:28.890 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T21:59:28.890 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T21:59:28.956 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T21:59:28.981 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T21:59:29.014 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T21:59:29.040 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T21:59:29.066 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T21:59:29.133 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T21:59:31.341 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T21:59:31.342 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-28T21:59:31.342 INFO:teuthology.orchestra.run.smithi063.stdout:Upgraded: 2024-03-28T21:59:31.342 INFO:teuthology.orchestra.run.smithi063.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T21:59:31.342 INFO:teuthology.orchestra.run.smithi063.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T21:59:31.342 INFO:teuthology.orchestra.run.smithi063.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T21:59:31.342 INFO:teuthology.orchestra.run.smithi063.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T21:59:31.342 INFO:teuthology.orchestra.run.smithi063.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T21:59:31.342 INFO:teuthology.orchestra.run.smithi063.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T21:59:31.342 INFO:teuthology.orchestra.run.smithi063.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T21:59:31.342 INFO:teuthology.orchestra.run.smithi063.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T21:59:31.342 INFO:teuthology.orchestra.run.smithi063.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T21:59:31.342 INFO:teuthology.orchestra.run.smithi063.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T21:59:31.342 INFO:teuthology.orchestra.run.smithi063.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T21:59:31.342 INFO:teuthology.orchestra.run.smithi063.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T21:59:31.343 INFO:teuthology.orchestra.run.smithi063.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T21:59:31.343 INFO:teuthology.orchestra.run.smithi063.stdout: librados2-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:31.343 INFO:teuthology.orchestra.run.smithi063.stdout: librbd1-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:31.343 INFO:teuthology.orchestra.run.smithi063.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T21:59:31.343 INFO:teuthology.orchestra.run.smithi063.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T21:59:31.343 INFO:teuthology.orchestra.run.smithi063.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T21:59:31.343 INFO:teuthology.orchestra.run.smithi063.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T21:59:31.343 INFO:teuthology.orchestra.run.smithi063.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T21:59:31.343 INFO:teuthology.orchestra.run.smithi063.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T21:59:31.343 INFO:teuthology.orchestra.run.smithi063.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T21:59:31.343 INFO:teuthology.orchestra.run.smithi063.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T21:59:31.343 INFO:teuthology.orchestra.run.smithi063.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T21:59:31.343 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-28T21:59:31.343 INFO:teuthology.orchestra.run.smithi063.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T21:59:31.344 INFO:teuthology.orchestra.run.smithi063.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T21:59:31.344 INFO:teuthology.orchestra.run.smithi063.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T21:59:31.344 INFO:teuthology.orchestra.run.smithi063.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T21:59:31.344 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:31.344 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-base-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:31.344 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:31.344 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:31.344 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-grafana-dashboards-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T21:59:31.344 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-immutable-object-cache-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:31.344 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mds-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:31.344 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:31.344 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T21:59:31.344 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-dashboard-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T21:59:31.344 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T21:59:31.345 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-modules-core-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T21:59:31.345 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-rook-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T21:59:31.345 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mon-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:31.345 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-osd-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:31.345 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-prometheus-alerts-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T21:59:31.345 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-radosgw-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:31.345 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-selinux-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:31.345 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-test-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:31.345 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-volume-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T21:59:31.345 INFO:teuthology.orchestra.run.smithi063.stdout: cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T21:59:31.345 INFO:teuthology.orchestra.run.smithi063.stdout: cephfs-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:31.345 INFO:teuthology.orchestra.run.smithi063.stdout: cephfs-shell-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:31.345 INFO:teuthology.orchestra.run.smithi063.stdout: cephfs-top-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T21:59:31.345 INFO:teuthology.orchestra.run.smithi063.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T21:59:31.346 INFO:teuthology.orchestra.run.smithi063.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T21:59:31.346 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T21:59:31.346 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T21:59:31.346 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T21:59:31.346 INFO:teuthology.orchestra.run.smithi063.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T21:59:31.346 INFO:teuthology.orchestra.run.smithi063.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T21:59:31.346 INFO:teuthology.orchestra.run.smithi063.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T21:59:31.346 INFO:teuthology.orchestra.run.smithi063.stdout: jq-1.6-16.el9.x86_64 2024-03-28T21:59:31.346 INFO:teuthology.orchestra.run.smithi063.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T21:59:31.346 INFO:teuthology.orchestra.run.smithi063.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T21:59:31.346 INFO:teuthology.orchestra.run.smithi063.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T21:59:31.346 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs-devel-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:31.347 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs2-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:31.347 INFO:teuthology.orchestra.run.smithi063.stdout: libcephsqlite-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:31.347 INFO:teuthology.orchestra.run.smithi063.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T21:59:31.347 INFO:teuthology.orchestra.run.smithi063.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T21:59:31.347 INFO:teuthology.orchestra.run.smithi063.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T21:59:31.347 INFO:teuthology.orchestra.run.smithi063.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T21:59:31.347 INFO:teuthology.orchestra.run.smithi063.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T21:59:31.347 INFO:teuthology.orchestra.run.smithi063.stdout: librados-devel-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:31.347 INFO:teuthology.orchestra.run.smithi063.stdout: libradosstriper1-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:31.347 INFO:teuthology.orchestra.run.smithi063.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T21:59:31.347 INFO:teuthology.orchestra.run.smithi063.stdout: librgw2-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:31.347 INFO:teuthology.orchestra.run.smithi063.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T21:59:31.347 INFO:teuthology.orchestra.run.smithi063.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T21:59:31.347 INFO:teuthology.orchestra.run.smithi063.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T21:59:31.348 INFO:teuthology.orchestra.run.smithi063.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T21:59:31.348 INFO:teuthology.orchestra.run.smithi063.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T21:59:31.348 INFO:teuthology.orchestra.run.smithi063.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T21:59:31.348 INFO:teuthology.orchestra.run.smithi063.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T21:59:31.348 INFO:teuthology.orchestra.run.smithi063.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T21:59:31.348 INFO:teuthology.orchestra.run.smithi063.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T21:59:31.348 INFO:teuthology.orchestra.run.smithi063.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T21:59:31.348 INFO:teuthology.orchestra.run.smithi063.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T21:59:31.348 INFO:teuthology.orchestra.run.smithi063.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T21:59:31.348 INFO:teuthology.orchestra.run.smithi063.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T21:59:31.348 INFO:teuthology.orchestra.run.smithi063.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T21:59:31.348 INFO:teuthology.orchestra.run.smithi063.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T21:59:31.348 INFO:teuthology.orchestra.run.smithi063.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T21:59:31.348 INFO:teuthology.orchestra.run.smithi063.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T21:59:31.349 INFO:teuthology.orchestra.run.smithi063.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T21:59:31.349 INFO:teuthology.orchestra.run.smithi063.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T21:59:31.349 INFO:teuthology.orchestra.run.smithi063.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T21:59:31.349 INFO:teuthology.orchestra.run.smithi063.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T21:59:31.349 INFO:teuthology.orchestra.run.smithi063.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T21:59:31.349 INFO:teuthology.orchestra.run.smithi063.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T21:59:31.349 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T21:59:31.349 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-argparse-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:31.349 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:31.349 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cephfs-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:31.349 INFO:teuthology.orchestra.run.smithi063.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T21:59:31.349 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T21:59:31.349 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T21:59:31.349 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T21:59:31.350 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T21:59:31.350 INFO:teuthology.orchestra.run.smithi063.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T21:59:31.350 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T21:59:31.350 INFO:teuthology.orchestra.run.smithi063.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T21:59:31.350 INFO:teuthology.orchestra.run.smithi063.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T21:59:31.350 INFO:teuthology.orchestra.run.smithi063.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T21:59:31.350 INFO:teuthology.orchestra.run.smithi063.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T21:59:31.350 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T21:59:31.350 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T21:59:31.350 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T21:59:31.350 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T21:59:31.351 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T21:59:31.351 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T21:59:31.351 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T21:59:31.351 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T21:59:31.352 INFO:teuthology.orchestra.run.smithi063.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T21:59:31.352 INFO:teuthology.orchestra.run.smithi063.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T21:59:31.352 INFO:teuthology.orchestra.run.smithi063.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T21:59:31.352 INFO:teuthology.orchestra.run.smithi063.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T21:59:31.352 INFO:teuthology.orchestra.run.smithi063.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T21:59:31.352 INFO:teuthology.orchestra.run.smithi063.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T21:59:31.352 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T21:59:31.352 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T21:59:31.352 INFO:teuthology.orchestra.run.smithi063.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T21:59:31.352 INFO:teuthology.orchestra.run.smithi063.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T21:59:31.352 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T21:59:31.352 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T21:59:31.352 INFO:teuthology.orchestra.run.smithi063.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T21:59:31.352 INFO:teuthology.orchestra.run.smithi063.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T21:59:31.353 INFO:teuthology.orchestra.run.smithi063.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T21:59:31.353 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T21:59:31.353 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T21:59:31.353 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T21:59:31.353 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T21:59:31.353 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T21:59:31.353 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T21:59:31.353 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rados-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:31.353 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rbd-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:31.353 INFO:teuthology.orchestra.run.smithi063.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T21:59:31.354 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T21:59:31.354 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T21:59:31.354 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rgw-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:31.354 INFO:teuthology.orchestra.run.smithi063.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T21:59:31.354 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T21:59:31.354 INFO:teuthology.orchestra.run.smithi063.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T21:59:31.354 INFO:teuthology.orchestra.run.smithi063.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T21:59:31.354 INFO:teuthology.orchestra.run.smithi063.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T21:59:31.354 INFO:teuthology.orchestra.run.smithi063.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T21:59:31.354 INFO:teuthology.orchestra.run.smithi063.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T21:59:31.354 INFO:teuthology.orchestra.run.smithi063.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T21:59:31.354 INFO:teuthology.orchestra.run.smithi063.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T21:59:31.354 INFO:teuthology.orchestra.run.smithi063.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T21:59:31.355 INFO:teuthology.orchestra.run.smithi063.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T21:59:31.355 INFO:teuthology.orchestra.run.smithi063.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T21:59:31.355 INFO:teuthology.orchestra.run.smithi063.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T21:59:31.355 INFO:teuthology.orchestra.run.smithi063.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T21:59:31.355 INFO:teuthology.orchestra.run.smithi063.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T21:59:31.355 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:31.355 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:31.355 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-nbd-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:31.355 INFO:teuthology.orchestra.run.smithi063.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T21:59:31.355 INFO:teuthology.orchestra.run.smithi063.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T21:59:31.355 INFO:teuthology.orchestra.run.smithi063.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T21:59:31.355 INFO:teuthology.orchestra.run.smithi063.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T21:59:31.355 INFO:teuthology.orchestra.run.smithi063.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T21:59:31.355 INFO:teuthology.orchestra.run.smithi063.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T21:59:31.356 INFO:teuthology.orchestra.run.smithi063.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T21:59:31.356 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-28T21:59:31.356 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-28T21:59:31.791 DEBUG:teuthology.parallel:result is None 2024-03-28T21:59:33.219 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T21:59:33.219 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-2:19.0.0-1899.g323d23dc.el9.x86_64 1/198 2024-03-28T21:59:33.219 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-base-2:19.0.0-1899.g323d23dc.el9.x86_64 2/198 2024-03-28T21:59:33.219 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 3/198 2024-03-28T21:59:33.219 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 4/198 2024-03-28T21:59:33.219 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1899.g323d2 5/198 2024-03-28T21:59:33.219 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mds-2:19.0.0-1899.g323d23dc.el9.x86_64 6/198 2024-03-28T21:59:33.219 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-2:19.0.0-1899.g323d23dc.el9.x86_64 7/198 2024-03-28T21:59:33.220 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mon-2:19.0.0-1899.g323d23dc.el9.x86_64 8/198 2024-03-28T21:59:33.220 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-osd-2:19.0.0-1899.g323d23dc.el9.x86_64 9/198 2024-03-28T21:59:33.220 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-radosgw-2:19.0.0-1899.g323d23dc.el9.x86_64 10/198 2024-03-28T21:59:33.220 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-selinux-2:19.0.0-1899.g323d23dc.el9.x86_64 11/198 2024-03-28T21:59:33.220 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-test-2:19.0.0-1899.g323d23dc.el9.x86_64 12/198 2024-03-28T21:59:33.220 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephfs-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 13/198 2024-03-28T21:59:33.220 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephfs-shell-2:19.0.0-1899.g323d23dc.el9.x86_64 14/198 2024-03-28T21:59:33.220 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs-devel-2:19.0.0-1899.g323d23dc.el9.x86_ 15/198 2024-03-28T21:59:33.220 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs2-2:19.0.0-1899.g323d23dc.el9.x86_64 16/198 2024-03-28T21:59:33.220 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephsqlite-2:19.0.0-1899.g323d23dc.el9.x86_64 17/198 2024-03-28T21:59:33.220 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados-devel-2:19.0.0-1899.g323d23dc.el9.x86_6 18/198 2024-03-28T21:59:33.220 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libradosstriper1-2:19.0.0-1899.g323d23dc.el9.x86 19/198 2024-03-28T21:59:33.220 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librgw2-2:19.0.0-1899.g323d23dc.el9.x86_64 20/198 2024-03-28T21:59:33.220 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1899.g323d23dc.el 21/198 2024-03-28T21:59:33.221 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ceph-common-2:19.0.0-1899.g323d23dc.el9. 22/198 2024-03-28T21:59:33.221 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cephfs-2:19.0.0-1899.g323d23dc.el9.x86_6 23/198 2024-03-28T21:59:33.221 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rados-2:19.0.0-1899.g323d23dc.el9.x86_64 24/198 2024-03-28T21:59:33.221 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rbd-2:19.0.0-1899.g323d23dc.el9.x86_64 25/198 2024-03-28T21:59:33.221 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rgw-2:19.0.0-1899.g323d23dc.el9.x86_64 26/198 2024-03-28T21:59:33.221 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 27/198 2024-03-28T21:59:33.221 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 28/198 2024-03-28T21:59:33.221 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-nbd-2:19.0.0-1899.g323d23dc.el9.x86_64 29/198 2024-03-28T21:59:33.237 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1899.g323d23dc. 30/198 2024-03-28T21:59:33.237 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1899.g323d23dc.el9.noa 31/198 2024-03-28T21:59:33.237 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1899.g323d23dc.el9.n 32/198 2024-03-28T21:59:33.237 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1899.g323 33/198 2024-03-28T21:59:33.237 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1899.g323d23dc.el 34/198 2024-03-28T21:59:33.237 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1899.g323d23dc.el9.noarch 35/198 2024-03-28T21:59:33.238 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1899.g323d23dc.e 36/198 2024-03-28T21:59:33.238 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-volume-2:19.0.0-1899.g323d23dc.el9.noarch 37/198 2024-03-28T21:59:33.238 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 38/198 2024-03-28T21:59:33.238 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephfs-top-2:19.0.0-1899.g323d23dc.el9.noarch 39/198 2024-03-28T21:59:33.238 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T21:59:33.238 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T21:59:33.238 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T21:59:33.238 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T21:59:33.238 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T21:59:33.238 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T21:59:33.238 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T21:59:33.238 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T21:59:33.238 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T21:59:33.238 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T21:59:33.239 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T21:59:33.239 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T21:59:33.239 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T21:59:33.239 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T21:59:33.239 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T21:59:33.239 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T21:59:33.239 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T21:59:33.239 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T21:59:33.239 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : postgresql-13.14-1.el9.x86_64 58/198 2024-03-28T21:59:33.239 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 59/198 2024-03-28T21:59:33.239 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 60/198 2024-03-28T21:59:33.239 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 61/198 2024-03-28T21:59:33.239 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-28T21:59:33.240 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-28T21:59:33.240 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-28T21:59:33.240 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-28T21:59:33.240 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-28T21:59:33.240 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-28T21:59:33.240 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-28T21:59:33.240 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-28T21:59:33.240 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-28T21:59:33.240 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-28T21:59:33.240 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-28T21:59:33.240 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-28T21:59:33.240 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-28T21:59:33.240 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-28T21:59:33.240 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-28T21:59:33.241 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-28T21:59:33.241 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-28T21:59:33.241 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-28T21:59:33.241 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-28T21:59:33.241 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-28T21:59:33.241 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 82/198 2024-03-28T21:59:33.241 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 83/198 2024-03-28T21:59:33.241 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 84/198 2024-03-28T21:59:33.241 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 85/198 2024-03-28T21:59:33.241 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 86/198 2024-03-28T21:59:33.241 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 87/198 2024-03-28T21:59:33.241 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 88/198 2024-03-28T21:59:33.241 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T21:59:33.241 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 90/198 2024-03-28T21:59:33.242 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 91/198 2024-03-28T21:59:33.242 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ply-3.11-14.el9.noarch 92/198 2024-03-28T21:59:33.242 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 93/198 2024-03-28T21:59:33.242 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/198 2024-03-28T21:59:33.242 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : jq-1.6-16.el9.x86_64 95/198 2024-03-28T21:59:33.242 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 96/198 2024-03-28T21:59:33.242 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 97/198 2024-03-28T21:59:33.242 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 98/198 2024-03-28T21:59:33.242 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 99/198 2024-03-28T21:59:33.242 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 100/198 2024-03-28T21:59:33.242 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 101/198 2024-03-28T21:59:33.242 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 102/198 2024-03-28T21:59:33.242 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 103/198 2024-03-28T21:59:33.243 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T21:59:33.243 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T21:59:33.243 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T21:59:33.243 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T21:59:33.243 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T21:59:33.243 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T21:59:33.243 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T21:59:33.243 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T21:59:33.243 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T21:59:33.243 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T21:59:33.243 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T21:59:33.243 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T21:59:33.243 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T21:59:33.243 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T21:59:33.244 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T21:59:33.247 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T21:59:33.247 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T21:59:33.247 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T21:59:33.247 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T21:59:33.247 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T21:59:33.247 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T21:59:33.247 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T21:59:33.247 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T21:59:33.247 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T21:59:33.248 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T21:59:33.248 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T21:59:33.248 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T21:59:33.248 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T21:59:33.248 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T21:59:33.248 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T21:59:33.248 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T21:59:33.248 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T21:59:33.248 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T21:59:33.248 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T21:59:33.248 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T21:59:33.248 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T21:59:33.248 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T21:59:33.249 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T21:59:33.249 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T21:59:33.249 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T21:59:33.249 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T21:59:33.249 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T21:59:33.249 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T21:59:33.249 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T21:59:33.249 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T21:59:33.249 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T21:59:33.249 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T21:59:33.249 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-2:19.0.0-1899.g323d23dc.el9.x86_64 151/198 2024-03-28T21:59:33.249 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T21:59:33.249 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-2:19.0.0-1899.g323d23dc.el9.x86_64 153/198 2024-03-28T21:59:33.250 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T21:59:33.250 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 155/198 2024-03-28T21:59:33.250 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-03-28T21:59:33.250 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T21:59:33.250 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T21:59:33.250 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T21:59:33.250 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T21:59:33.250 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T21:59:33.250 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T21:59:33.250 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 163/198 2024-03-28T21:59:33.250 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 164/198 2024-03-28T21:59:33.250 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T21:59:33.250 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T21:59:33.250 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T21:59:33.251 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T21:59:33.251 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T21:59:33.251 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T21:59:33.251 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T21:59:33.251 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T21:59:33.251 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T21:59:33.251 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T21:59:33.251 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T21:59:33.251 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T21:59:33.251 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T21:59:33.251 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T21:59:33.251 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T21:59:33.251 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T21:59:33.252 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T21:59:33.252 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T21:59:33.252 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T21:59:33.252 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T21:59:33.252 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T21:59:33.252 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T21:59:33.252 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T21:59:33.252 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T21:59:33.252 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T21:59:33.252 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T21:59:33.252 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T21:59:33.252 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T21:59:33.252 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T21:59:33.252 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T21:59:33.253 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T21:59:33.253 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T21:59:33.253 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T21:59:34.410 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T21:59:34.410 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-2:19.0.0-1899.g323d23dc.el9.x86_64 1/198 2024-03-28T21:59:34.410 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-base-2:19.0.0-1899.g323d23dc.el9.x86_64 2/198 2024-03-28T21:59:34.410 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 3/198 2024-03-28T21:59:34.410 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 4/198 2024-03-28T21:59:34.410 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1899.g323d2 5/198 2024-03-28T21:59:34.411 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mds-2:19.0.0-1899.g323d23dc.el9.x86_64 6/198 2024-03-28T21:59:34.411 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-2:19.0.0-1899.g323d23dc.el9.x86_64 7/198 2024-03-28T21:59:34.411 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mon-2:19.0.0-1899.g323d23dc.el9.x86_64 8/198 2024-03-28T21:59:34.411 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-osd-2:19.0.0-1899.g323d23dc.el9.x86_64 9/198 2024-03-28T21:59:34.411 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-radosgw-2:19.0.0-1899.g323d23dc.el9.x86_64 10/198 2024-03-28T21:59:34.411 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-selinux-2:19.0.0-1899.g323d23dc.el9.x86_64 11/198 2024-03-28T21:59:34.411 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-test-2:19.0.0-1899.g323d23dc.el9.x86_64 12/198 2024-03-28T21:59:34.411 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : cephfs-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 13/198 2024-03-28T21:59:34.411 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : cephfs-shell-2:19.0.0-1899.g323d23dc.el9.x86_64 14/198 2024-03-28T21:59:34.411 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephfs-devel-2:19.0.0-1899.g323d23dc.el9.x86_ 15/198 2024-03-28T21:59:34.411 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephfs2-2:19.0.0-1899.g323d23dc.el9.x86_64 16/198 2024-03-28T21:59:34.411 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephsqlite-2:19.0.0-1899.g323d23dc.el9.x86_64 17/198 2024-03-28T21:59:34.411 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados-devel-2:19.0.0-1899.g323d23dc.el9.x86_6 18/198 2024-03-28T21:59:34.411 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libradosstriper1-2:19.0.0-1899.g323d23dc.el9.x86 19/198 2024-03-28T21:59:34.411 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librgw2-2:19.0.0-1899.g323d23dc.el9.x86_64 20/198 2024-03-28T21:59:34.412 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1899.g323d23dc.el 21/198 2024-03-28T21:59:34.412 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-ceph-common-2:19.0.0-1899.g323d23dc.el9. 22/198 2024-03-28T21:59:34.412 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cephfs-2:19.0.0-1899.g323d23dc.el9.x86_6 23/198 2024-03-28T21:59:34.412 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rados-2:19.0.0-1899.g323d23dc.el9.x86_64 24/198 2024-03-28T21:59:34.412 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rbd-2:19.0.0-1899.g323d23dc.el9.x86_64 25/198 2024-03-28T21:59:34.412 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rgw-2:19.0.0-1899.g323d23dc.el9.x86_64 26/198 2024-03-28T21:59:34.412 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 27/198 2024-03-28T21:59:34.412 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 28/198 2024-03-28T21:59:34.412 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-nbd-2:19.0.0-1899.g323d23dc.el9.x86_64 29/198 2024-03-28T21:59:34.427 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1899.g323d23dc. 30/198 2024-03-28T21:59:34.427 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1899.g323d23dc.el9.noa 31/198 2024-03-28T21:59:34.427 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1899.g323d23dc.el9.n 32/198 2024-03-28T21:59:34.427 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1899.g323 33/198 2024-03-28T21:59:34.427 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1899.g323d23dc.el 34/198 2024-03-28T21:59:34.427 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1899.g323d23dc.el9.noarch 35/198 2024-03-28T21:59:34.427 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1899.g323d23dc.e 36/198 2024-03-28T21:59:34.427 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-volume-2:19.0.0-1899.g323d23dc.el9.noarch 37/198 2024-03-28T21:59:34.427 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 38/198 2024-03-28T21:59:34.428 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : cephfs-top-2:19.0.0-1899.g323d23dc.el9.noarch 39/198 2024-03-28T21:59:34.428 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T21:59:34.428 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T21:59:34.428 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T21:59:34.428 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T21:59:34.428 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T21:59:34.428 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T21:59:34.428 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T21:59:34.428 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T21:59:34.428 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T21:59:34.428 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T21:59:34.428 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T21:59:34.428 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T21:59:34.428 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T21:59:34.429 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T21:59:34.429 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T21:59:34.429 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T21:59:34.429 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T21:59:34.429 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T21:59:34.429 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : postgresql-13.14-1.el9.x86_64 58/198 2024-03-28T21:59:34.429 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 59/198 2024-03-28T21:59:34.429 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 60/198 2024-03-28T21:59:34.429 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 61/198 2024-03-28T21:59:34.429 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-28T21:59:34.429 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-28T21:59:34.429 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-28T21:59:34.429 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-28T21:59:34.429 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-28T21:59:34.430 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-28T21:59:34.430 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-28T21:59:34.430 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-28T21:59:34.430 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-28T21:59:34.430 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-28T21:59:34.430 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-28T21:59:34.430 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-28T21:59:34.430 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-28T21:59:34.430 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-28T21:59:34.430 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-28T21:59:34.430 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-28T21:59:34.430 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-28T21:59:34.430 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-28T21:59:34.430 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-28T21:59:34.431 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-28T21:59:34.431 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 82/198 2024-03-28T21:59:34.431 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 83/198 2024-03-28T21:59:34.431 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 84/198 2024-03-28T21:59:34.431 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 85/198 2024-03-28T21:59:34.431 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 86/198 2024-03-28T21:59:34.431 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 87/198 2024-03-28T21:59:34.431 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 88/198 2024-03-28T21:59:34.431 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T21:59:34.431 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 90/198 2024-03-28T21:59:34.431 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 91/198 2024-03-28T21:59:34.431 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-ply-3.11-14.el9.noarch 92/198 2024-03-28T21:59:34.431 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 93/198 2024-03-28T21:59:34.431 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/198 2024-03-28T21:59:34.431 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : jq-1.6-16.el9.x86_64 95/198 2024-03-28T21:59:34.432 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 96/198 2024-03-28T21:59:34.432 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 97/198 2024-03-28T21:59:34.432 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 98/198 2024-03-28T21:59:34.432 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 99/198 2024-03-28T21:59:34.432 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 100/198 2024-03-28T21:59:34.432 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 101/198 2024-03-28T21:59:34.432 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 102/198 2024-03-28T21:59:34.432 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 103/198 2024-03-28T21:59:34.432 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T21:59:34.432 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T21:59:34.432 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T21:59:34.432 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T21:59:34.432 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T21:59:34.432 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T21:59:34.432 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T21:59:34.433 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T21:59:34.433 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T21:59:34.433 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T21:59:34.433 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T21:59:34.433 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T21:59:34.433 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T21:59:34.433 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T21:59:34.433 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T21:59:34.433 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T21:59:34.433 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T21:59:34.433 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T21:59:34.433 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T21:59:34.433 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T21:59:34.433 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T21:59:34.433 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T21:59:34.434 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T21:59:34.434 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T21:59:34.434 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T21:59:34.434 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T21:59:34.434 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T21:59:34.434 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T21:59:34.434 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T21:59:34.434 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T21:59:34.434 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T21:59:34.434 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T21:59:34.434 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T21:59:34.434 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T21:59:34.435 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T21:59:34.435 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T21:59:34.435 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T21:59:34.435 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T21:59:34.435 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T21:59:34.435 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T21:59:34.435 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T21:59:34.435 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T21:59:34.435 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T21:59:34.435 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T21:59:34.435 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T21:59:34.435 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T21:59:34.435 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T21:59:34.435 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados2-2:19.0.0-1899.g323d23dc.el9.x86_64 151/198 2024-03-28T21:59:34.435 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T21:59:34.436 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librbd1-2:19.0.0-1899.g323d23dc.el9.x86_64 153/198 2024-03-28T21:59:34.436 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T21:59:34.436 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 155/198 2024-03-28T21:59:34.436 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-03-28T21:59:34.436 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T21:59:34.436 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T21:59:34.436 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T21:59:34.436 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T21:59:34.436 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T21:59:34.436 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T21:59:34.436 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 163/198 2024-03-28T21:59:34.436 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 164/198 2024-03-28T21:59:34.436 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T21:59:34.436 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T21:59:34.437 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T21:59:34.437 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T21:59:34.437 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T21:59:34.437 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T21:59:34.437 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T21:59:34.437 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T21:59:34.437 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T21:59:34.437 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T21:59:34.437 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T21:59:34.437 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T21:59:34.437 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T21:59:34.437 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T21:59:34.437 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T21:59:34.437 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T21:59:34.437 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T21:59:34.438 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T21:59:34.438 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T21:59:34.438 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T21:59:34.438 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T21:59:34.438 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T21:59:34.438 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T21:59:34.438 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T21:59:34.438 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T21:59:34.438 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T21:59:34.438 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T21:59:34.438 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T21:59:34.438 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T21:59:34.438 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T21:59:34.438 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T21:59:34.438 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T21:59:34.439 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T21:59:36.216 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T21:59:36.216 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T21:59:36.216 INFO:teuthology.orchestra.run.smithi016.stdout:Upgraded: 2024-03-28T21:59:36.216 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T21:59:36.217 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T21:59:36.217 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T21:59:36.217 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T21:59:36.217 INFO:teuthology.orchestra.run.smithi016.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T21:59:36.217 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T21:59:36.217 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T21:59:36.217 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T21:59:36.217 INFO:teuthology.orchestra.run.smithi016.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T21:59:36.217 INFO:teuthology.orchestra.run.smithi016.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T21:59:36.217 INFO:teuthology.orchestra.run.smithi016.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T21:59:36.217 INFO:teuthology.orchestra.run.smithi016.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T21:59:36.217 INFO:teuthology.orchestra.run.smithi016.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T21:59:36.217 INFO:teuthology.orchestra.run.smithi016.stdout: librados2-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:36.218 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:36.218 INFO:teuthology.orchestra.run.smithi016.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T21:59:36.218 INFO:teuthology.orchestra.run.smithi016.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T21:59:36.218 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T21:59:36.218 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T21:59:36.218 INFO:teuthology.orchestra.run.smithi016.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T21:59:36.218 INFO:teuthology.orchestra.run.smithi016.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T21:59:36.218 INFO:teuthology.orchestra.run.smithi016.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T21:59:36.218 INFO:teuthology.orchestra.run.smithi016.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T21:59:36.218 INFO:teuthology.orchestra.run.smithi016.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T21:59:36.218 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-28T21:59:36.218 INFO:teuthology.orchestra.run.smithi016.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T21:59:36.218 INFO:teuthology.orchestra.run.smithi016.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T21:59:36.218 INFO:teuthology.orchestra.run.smithi016.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T21:59:36.219 INFO:teuthology.orchestra.run.smithi016.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T21:59:36.219 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:36.219 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:36.219 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:36.219 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:36.219 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T21:59:36.219 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:36.219 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:36.219 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:36.219 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T21:59:36.219 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T21:59:36.219 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T21:59:36.219 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T21:59:36.220 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T21:59:36.220 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:36.220 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:36.220 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-prometheus-alerts-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T21:59:36.220 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:36.220 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:36.220 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:36.220 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-volume-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T21:59:36.220 INFO:teuthology.orchestra.run.smithi016.stdout: cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T21:59:36.220 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:36.220 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-shell-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:36.220 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-top-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T21:59:36.220 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T21:59:36.221 INFO:teuthology.orchestra.run.smithi016.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T21:59:36.221 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T21:59:36.221 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T21:59:36.221 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T21:59:36.221 INFO:teuthology.orchestra.run.smithi016.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T21:59:36.221 INFO:teuthology.orchestra.run.smithi016.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T21:59:36.221 INFO:teuthology.orchestra.run.smithi016.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T21:59:36.221 INFO:teuthology.orchestra.run.smithi016.stdout: jq-1.6-16.el9.x86_64 2024-03-28T21:59:36.221 INFO:teuthology.orchestra.run.smithi016.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T21:59:36.221 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T21:59:36.221 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T21:59:36.221 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:36.221 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:36.222 INFO:teuthology.orchestra.run.smithi016.stdout: libcephsqlite-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:36.222 INFO:teuthology.orchestra.run.smithi016.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T21:59:36.222 INFO:teuthology.orchestra.run.smithi016.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T21:59:36.222 INFO:teuthology.orchestra.run.smithi016.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T21:59:36.222 INFO:teuthology.orchestra.run.smithi016.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T21:59:36.222 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T21:59:36.223 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:36.223 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:36.223 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T21:59:36.223 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:36.223 INFO:teuthology.orchestra.run.smithi016.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T21:59:36.223 INFO:teuthology.orchestra.run.smithi016.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T21:59:36.223 INFO:teuthology.orchestra.run.smithi016.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T21:59:36.223 INFO:teuthology.orchestra.run.smithi016.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T21:59:36.223 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T21:59:36.224 INFO:teuthology.orchestra.run.smithi016.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T21:59:36.224 INFO:teuthology.orchestra.run.smithi016.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T21:59:36.224 INFO:teuthology.orchestra.run.smithi016.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T21:59:36.224 INFO:teuthology.orchestra.run.smithi016.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T21:59:36.224 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T21:59:36.224 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T21:59:36.224 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T21:59:36.224 INFO:teuthology.orchestra.run.smithi016.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T21:59:36.224 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T21:59:36.224 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T21:59:36.225 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T21:59:36.225 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T21:59:36.225 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T21:59:36.225 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T21:59:36.225 INFO:teuthology.orchestra.run.smithi016.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T21:59:36.225 INFO:teuthology.orchestra.run.smithi016.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T21:59:36.225 INFO:teuthology.orchestra.run.smithi016.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T21:59:36.225 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T21:59:36.225 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T21:59:36.225 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:36.225 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:36.225 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:36.225 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T21:59:36.225 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T21:59:36.226 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T21:59:36.226 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T21:59:36.226 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T21:59:36.226 INFO:teuthology.orchestra.run.smithi016.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T21:59:36.226 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T21:59:36.226 INFO:teuthology.orchestra.run.smithi016.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T21:59:36.226 INFO:teuthology.orchestra.run.smithi016.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T21:59:36.226 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T21:59:36.226 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T21:59:36.226 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T21:59:36.226 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T21:59:36.227 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T21:59:36.227 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T21:59:36.227 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T21:59:36.227 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T21:59:36.227 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T21:59:36.227 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T21:59:36.227 INFO:teuthology.orchestra.run.smithi016.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T21:59:36.227 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T21:59:36.227 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T21:59:36.227 INFO:teuthology.orchestra.run.smithi016.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T21:59:36.227 INFO:teuthology.orchestra.run.smithi016.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T21:59:36.228 INFO:teuthology.orchestra.run.smithi016.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T21:59:36.228 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T21:59:36.228 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T21:59:36.228 INFO:teuthology.orchestra.run.smithi016.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T21:59:36.228 INFO:teuthology.orchestra.run.smithi016.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T21:59:36.228 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T21:59:36.228 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T21:59:36.228 INFO:teuthology.orchestra.run.smithi016.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T21:59:36.228 INFO:teuthology.orchestra.run.smithi016.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T21:59:36.228 INFO:teuthology.orchestra.run.smithi016.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T21:59:36.228 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T21:59:36.228 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T21:59:36.229 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T21:59:36.229 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T21:59:36.229 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T21:59:36.229 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T21:59:36.229 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:36.229 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:36.229 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T21:59:36.229 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T21:59:36.229 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T21:59:36.229 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:36.229 INFO:teuthology.orchestra.run.smithi016.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T21:59:36.229 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T21:59:36.229 INFO:teuthology.orchestra.run.smithi016.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T21:59:36.230 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T21:59:36.230 INFO:teuthology.orchestra.run.smithi016.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T21:59:36.230 INFO:teuthology.orchestra.run.smithi016.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T21:59:36.230 INFO:teuthology.orchestra.run.smithi016.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T21:59:36.230 INFO:teuthology.orchestra.run.smithi016.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T21:59:36.230 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T21:59:36.230 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T21:59:36.230 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T21:59:36.230 INFO:teuthology.orchestra.run.smithi016.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T21:59:36.230 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T21:59:36.230 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T21:59:36.231 INFO:teuthology.orchestra.run.smithi016.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T21:59:36.231 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:36.231 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:36.231 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:36.231 INFO:teuthology.orchestra.run.smithi016.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T21:59:36.231 INFO:teuthology.orchestra.run.smithi016.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T21:59:36.231 INFO:teuthology.orchestra.run.smithi016.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T21:59:36.231 INFO:teuthology.orchestra.run.smithi016.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T21:59:36.231 INFO:teuthology.orchestra.run.smithi016.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T21:59:36.232 INFO:teuthology.orchestra.run.smithi016.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T21:59:36.232 INFO:teuthology.orchestra.run.smithi016.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T21:59:36.232 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T21:59:36.232 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-28T21:59:36.671 DEBUG:teuthology.parallel:result is None 2024-03-28T21:59:37.584 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T21:59:37.584 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-28T21:59:37.584 INFO:teuthology.orchestra.run.smithi053.stdout:Upgraded: 2024-03-28T21:59:37.584 INFO:teuthology.orchestra.run.smithi053.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T21:59:37.584 INFO:teuthology.orchestra.run.smithi053.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T21:59:37.584 INFO:teuthology.orchestra.run.smithi053.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T21:59:37.584 INFO:teuthology.orchestra.run.smithi053.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T21:59:37.584 INFO:teuthology.orchestra.run.smithi053.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T21:59:37.585 INFO:teuthology.orchestra.run.smithi053.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T21:59:37.585 INFO:teuthology.orchestra.run.smithi053.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T21:59:37.585 INFO:teuthology.orchestra.run.smithi053.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T21:59:37.585 INFO:teuthology.orchestra.run.smithi053.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T21:59:37.585 INFO:teuthology.orchestra.run.smithi053.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T21:59:37.585 INFO:teuthology.orchestra.run.smithi053.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T21:59:37.585 INFO:teuthology.orchestra.run.smithi053.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T21:59:37.585 INFO:teuthology.orchestra.run.smithi053.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T21:59:37.585 INFO:teuthology.orchestra.run.smithi053.stdout: librados2-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:37.585 INFO:teuthology.orchestra.run.smithi053.stdout: librbd1-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:37.585 INFO:teuthology.orchestra.run.smithi053.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T21:59:37.585 INFO:teuthology.orchestra.run.smithi053.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T21:59:37.585 INFO:teuthology.orchestra.run.smithi053.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T21:59:37.585 INFO:teuthology.orchestra.run.smithi053.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T21:59:37.585 INFO:teuthology.orchestra.run.smithi053.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T21:59:37.586 INFO:teuthology.orchestra.run.smithi053.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T21:59:37.586 INFO:teuthology.orchestra.run.smithi053.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T21:59:37.586 INFO:teuthology.orchestra.run.smithi053.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T21:59:37.586 INFO:teuthology.orchestra.run.smithi053.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T21:59:37.586 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-28T21:59:37.586 INFO:teuthology.orchestra.run.smithi053.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T21:59:37.586 INFO:teuthology.orchestra.run.smithi053.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T21:59:37.586 INFO:teuthology.orchestra.run.smithi053.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T21:59:37.586 INFO:teuthology.orchestra.run.smithi053.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T21:59:37.586 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:37.586 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-base-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:37.586 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:37.586 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:37.587 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-grafana-dashboards-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T21:59:37.587 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-immutable-object-cache-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:37.587 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mds-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:37.587 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:37.587 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T21:59:37.587 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-dashboard-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T21:59:37.587 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T21:59:37.587 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-modules-core-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T21:59:37.587 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-rook-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T21:59:37.587 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mon-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:37.587 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-osd-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:37.587 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-prometheus-alerts-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T21:59:37.587 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-radosgw-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:37.588 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-selinux-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:37.588 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-test-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:37.588 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-volume-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T21:59:37.588 INFO:teuthology.orchestra.run.smithi053.stdout: cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T21:59:37.588 INFO:teuthology.orchestra.run.smithi053.stdout: cephfs-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:37.588 INFO:teuthology.orchestra.run.smithi053.stdout: cephfs-shell-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:37.588 INFO:teuthology.orchestra.run.smithi053.stdout: cephfs-top-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T21:59:37.588 INFO:teuthology.orchestra.run.smithi053.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T21:59:37.588 INFO:teuthology.orchestra.run.smithi053.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T21:59:37.588 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T21:59:37.588 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T21:59:37.588 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T21:59:37.588 INFO:teuthology.orchestra.run.smithi053.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T21:59:37.589 INFO:teuthology.orchestra.run.smithi053.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T21:59:37.589 INFO:teuthology.orchestra.run.smithi053.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T21:59:37.589 INFO:teuthology.orchestra.run.smithi053.stdout: jq-1.6-16.el9.x86_64 2024-03-28T21:59:37.589 INFO:teuthology.orchestra.run.smithi053.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T21:59:37.589 INFO:teuthology.orchestra.run.smithi053.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T21:59:37.589 INFO:teuthology.orchestra.run.smithi053.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T21:59:37.589 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs-devel-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:37.589 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs2-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:37.589 INFO:teuthology.orchestra.run.smithi053.stdout: libcephsqlite-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:37.589 INFO:teuthology.orchestra.run.smithi053.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T21:59:37.589 INFO:teuthology.orchestra.run.smithi053.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T21:59:37.589 INFO:teuthology.orchestra.run.smithi053.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T21:59:37.589 INFO:teuthology.orchestra.run.smithi053.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T21:59:37.589 INFO:teuthology.orchestra.run.smithi053.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T21:59:37.590 INFO:teuthology.orchestra.run.smithi053.stdout: librados-devel-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:37.590 INFO:teuthology.orchestra.run.smithi053.stdout: libradosstriper1-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:37.590 INFO:teuthology.orchestra.run.smithi053.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T21:59:37.590 INFO:teuthology.orchestra.run.smithi053.stdout: librgw2-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:37.590 INFO:teuthology.orchestra.run.smithi053.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T21:59:37.590 INFO:teuthology.orchestra.run.smithi053.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T21:59:37.590 INFO:teuthology.orchestra.run.smithi053.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T21:59:37.590 INFO:teuthology.orchestra.run.smithi053.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T21:59:37.590 INFO:teuthology.orchestra.run.smithi053.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T21:59:37.590 INFO:teuthology.orchestra.run.smithi053.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T21:59:37.590 INFO:teuthology.orchestra.run.smithi053.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T21:59:37.590 INFO:teuthology.orchestra.run.smithi053.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T21:59:37.590 INFO:teuthology.orchestra.run.smithi053.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T21:59:37.590 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T21:59:37.591 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T21:59:37.591 INFO:teuthology.orchestra.run.smithi053.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T21:59:37.591 INFO:teuthology.orchestra.run.smithi053.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T21:59:37.591 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T21:59:37.591 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T21:59:37.591 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T21:59:37.591 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T21:59:37.591 INFO:teuthology.orchestra.run.smithi053.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T21:59:37.591 INFO:teuthology.orchestra.run.smithi053.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T21:59:37.591 INFO:teuthology.orchestra.run.smithi053.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T21:59:37.591 INFO:teuthology.orchestra.run.smithi053.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T21:59:37.591 INFO:teuthology.orchestra.run.smithi053.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T21:59:37.591 INFO:teuthology.orchestra.run.smithi053.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T21:59:37.591 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T21:59:37.592 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-argparse-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:37.592 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:37.592 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cephfs-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:37.592 INFO:teuthology.orchestra.run.smithi053.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T21:59:37.592 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T21:59:37.592 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T21:59:37.592 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T21:59:37.592 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T21:59:37.592 INFO:teuthology.orchestra.run.smithi053.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T21:59:37.592 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T21:59:37.592 INFO:teuthology.orchestra.run.smithi053.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T21:59:37.592 INFO:teuthology.orchestra.run.smithi053.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T21:59:37.592 INFO:teuthology.orchestra.run.smithi053.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T21:59:37.593 INFO:teuthology.orchestra.run.smithi053.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T21:59:37.593 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T21:59:37.593 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T21:59:37.593 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T21:59:37.593 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T21:59:37.593 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T21:59:37.593 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T21:59:37.593 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T21:59:37.593 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T21:59:37.593 INFO:teuthology.orchestra.run.smithi053.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T21:59:37.593 INFO:teuthology.orchestra.run.smithi053.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T21:59:37.593 INFO:teuthology.orchestra.run.smithi053.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T21:59:37.593 INFO:teuthology.orchestra.run.smithi053.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T21:59:37.593 INFO:teuthology.orchestra.run.smithi053.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T21:59:37.594 INFO:teuthology.orchestra.run.smithi053.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T21:59:37.594 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T21:59:37.594 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T21:59:37.594 INFO:teuthology.orchestra.run.smithi053.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T21:59:37.594 INFO:teuthology.orchestra.run.smithi053.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T21:59:37.594 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T21:59:37.594 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T21:59:37.594 INFO:teuthology.orchestra.run.smithi053.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T21:59:37.594 INFO:teuthology.orchestra.run.smithi053.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T21:59:37.594 INFO:teuthology.orchestra.run.smithi053.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T21:59:37.594 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T21:59:37.594 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T21:59:37.594 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T21:59:37.595 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T21:59:37.595 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T21:59:37.595 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T21:59:37.595 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rados-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:37.595 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rbd-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:37.595 INFO:teuthology.orchestra.run.smithi053.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T21:59:37.595 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T21:59:37.595 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T21:59:37.595 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rgw-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:37.595 INFO:teuthology.orchestra.run.smithi053.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T21:59:37.595 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T21:59:37.595 INFO:teuthology.orchestra.run.smithi053.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T21:59:37.595 INFO:teuthology.orchestra.run.smithi053.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T21:59:37.595 INFO:teuthology.orchestra.run.smithi053.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T21:59:37.596 INFO:teuthology.orchestra.run.smithi053.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T21:59:37.596 INFO:teuthology.orchestra.run.smithi053.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T21:59:37.596 INFO:teuthology.orchestra.run.smithi053.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T21:59:37.596 INFO:teuthology.orchestra.run.smithi053.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T21:59:37.596 INFO:teuthology.orchestra.run.smithi053.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T21:59:37.596 INFO:teuthology.orchestra.run.smithi053.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T21:59:37.596 INFO:teuthology.orchestra.run.smithi053.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T21:59:37.596 INFO:teuthology.orchestra.run.smithi053.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T21:59:37.596 INFO:teuthology.orchestra.run.smithi053.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T21:59:37.596 INFO:teuthology.orchestra.run.smithi053.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T21:59:37.596 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:37.596 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:37.596 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-nbd-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T21:59:37.597 INFO:teuthology.orchestra.run.smithi053.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T21:59:37.597 INFO:teuthology.orchestra.run.smithi053.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T21:59:37.597 INFO:teuthology.orchestra.run.smithi053.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T21:59:37.597 INFO:teuthology.orchestra.run.smithi053.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T21:59:37.597 INFO:teuthology.orchestra.run.smithi053.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T21:59:37.597 INFO:teuthology.orchestra.run.smithi053.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T21:59:37.597 INFO:teuthology.orchestra.run.smithi053.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T21:59:37.597 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-28T21:59:37.597 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-28T21:59:38.052 DEBUG:teuthology.parallel:result is None 2024-03-28T21:59:38.052 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=323d23dc228bbcb7f16f188fdc9c0dcb801844be 2024-03-28T21:59:38.194 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T21:59:38.219 INFO:teuthology.orchestra.run.smithi016.stdout:19.0.0-1899.g323d23dc.el9 2024-03-28T21:59:38.219 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1899.g323d23dc.el9 2024-03-28T21:59:38.219 INFO:teuthology.task.install:The correct ceph version 19.0.0-1899.g323d23dc is installed. 2024-03-28T21:59:38.221 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=323d23dc228bbcb7f16f188fdc9c0dcb801844be 2024-03-28T21:59:38.370 DEBUG:teuthology.orchestra.run.smithi053:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T21:59:38.396 INFO:teuthology.orchestra.run.smithi053.stdout:19.0.0-1899.g323d23dc.el9 2024-03-28T21:59:38.396 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1899.g323d23dc.el9 2024-03-28T21:59:38.396 INFO:teuthology.task.install:The correct ceph version 19.0.0-1899.g323d23dc is installed. 2024-03-28T21:59:38.400 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=323d23dc228bbcb7f16f188fdc9c0dcb801844be 2024-03-28T21:59:38.538 DEBUG:teuthology.orchestra.run.smithi063:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T21:59:38.565 INFO:teuthology.orchestra.run.smithi063.stdout:19.0.0-1899.g323d23dc.el9 2024-03-28T21:59:38.566 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1899.g323d23dc.el9 2024-03-28T21:59:38.566 INFO:teuthology.task.install:The correct ceph version 19.0.0-1899.g323d23dc is installed. 2024-03-28T21:59:38.567 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-28T21:59:38.568 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-28T21:59:38.568 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T21:59:38.602 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-28T21:59:38.602 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T21:59:38.642 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-28T21:59:38.642 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T21:59:38.681 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-28T21:59:38.681 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-28T21:59:38.681 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T21:59:38.711 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T21:59:38.780 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-28T21:59:38.781 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T21:59:38.817 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T21:59:38.891 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-28T21:59:38.891 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T21:59:38.927 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T21:59:39.003 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-28T21:59:39.004 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-28T21:59:39.004 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T21:59:39.035 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T21:59:39.106 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-28T21:59:39.106 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T21:59:39.142 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T21:59:39.215 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-28T21:59:39.215 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T21:59:39.251 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T21:59:39.325 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-28T21:59:39.326 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-28T21:59:39.326 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T21:59:39.357 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T21:59:39.427 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-28T21:59:39.427 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T21:59:39.462 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T21:59:39.539 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-28T21:59:39.539 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T21:59:39.574 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T21:59:39.647 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-28T21:59:39.761 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '323d23dc228bbcb7f16f188fdc9c0dcb801844be', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-03-28T21:59:39.761 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be 2024-03-28T21:59:39.762 INFO:tasks.cephadm:Cluster fsid is 798d9884-ed4e-11ee-95d0-87774f69a715 2024-03-28T21:59:39.762 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-28T21:59:39.762 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.16', 'mon.b': '172.21.15.53', 'mon.c': '172.21.15.63'} 2024-03-28T21:59:39.762 INFO:tasks.cephadm:First mon is mon.a on smithi016 2024-03-28T21:59:39.762 INFO:tasks.cephadm:First mgr is x 2024-03-28T21:59:39.762 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-28T21:59:39.763 DEBUG:teuthology.orchestra.run.smithi016:> sudo hostname $(hostname -s) 2024-03-28T21:59:39.795 DEBUG:teuthology.orchestra.run.smithi053:> sudo hostname $(hostname -s) 2024-03-28T21:59:39.832 DEBUG:teuthology.orchestra.run.smithi063:> sudo hostname $(hostname -s) 2024-03-28T21:59:39.866 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-28T21:59:39.866 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=323d23dc228bbcb7f16f188fdc9c0dcb801844be 2024-03-28T21:59:40.004 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '323d23dc228bbcb7f16f188fdc9c0dcb801844be', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54719/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1899-g323d23dc', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1899.g323d23dc'}, 'url': 'https://4.chacra.ceph.com/r/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-27 18:23:47.768164', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-28T21:59:40.139 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref squid, sha1 323d23dc228bbcb7f16f188fdc9c0dcb801844be from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=323d23dc228bbcb7f16f188fdc9c0dcb801844be 2024-03-28T21:59:40.140 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/x86_64/flavors/default/cephadm 2024-03-28T21:59:40.141 DEBUG:teuthology.orchestra.run.smithi016:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-28T21:59:41.221 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r--. 1 ubuntu ubuntu 766659 Mar 28 21:59 /home/ubuntu/cephtest/cephadm 2024-03-28T21:59:41.221 DEBUG:teuthology.orchestra.run.smithi053:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-28T21:59:42.331 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r--. 1 ubuntu ubuntu 766659 Mar 28 21:59 /home/ubuntu/cephtest/cephadm 2024-03-28T21:59:42.331 DEBUG:teuthology.orchestra.run.smithi063:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-28T21:59:43.208 INFO:teuthology.orchestra.run.smithi063.stdout:-rw-r--r--. 1 ubuntu ubuntu 766659 Mar 28 21:59 /home/ubuntu/cephtest/cephadm 2024-03-28T21:59:43.208 DEBUG:teuthology.orchestra.run.smithi016:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-28T21:59:43.227 DEBUG:teuthology.orchestra.run.smithi053:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-28T21:59:43.244 DEBUG:teuthology.orchestra.run.smithi063:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-28T21:59:43.279 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be on all hosts... 2024-03-28T21:59:43.279 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be pull 2024-03-28T21:59:43.282 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be pull 2024-03-28T21:59:43.287 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be pull 2024-03-28T21:59:44.094 INFO:teuthology.orchestra.run.smithi053.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be... 2024-03-28T21:59:44.166 INFO:teuthology.orchestra.run.smithi016.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be... 2024-03-28T21:59:44.231 INFO:teuthology.orchestra.run.smithi063.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be... 2024-03-28T22:00:12.362 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-03-28T22:00:12.362 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_version": "ceph version 19.0.0-1899-g323d23dc (323d23dc228bbcb7f16f188fdc9c0dcb801844be) squid (dev)", 2024-03-28T22:00:12.362 INFO:teuthology.orchestra.run.smithi053.stdout: "image_id": "bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52", 2024-03-28T22:00:12.362 INFO:teuthology.orchestra.run.smithi053.stdout: "repo_digests": [ 2024-03-28T22:00:12.363 INFO:teuthology.orchestra.run.smithi053.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15" 2024-03-28T22:00:12.363 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-03-28T22:00:12.363 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-03-28T22:00:16.209 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-28T22:00:16.210 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_version": "ceph version 19.0.0-1899-g323d23dc (323d23dc228bbcb7f16f188fdc9c0dcb801844be) squid (dev)", 2024-03-28T22:00:16.210 INFO:teuthology.orchestra.run.smithi063.stdout: "image_id": "bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52", 2024-03-28T22:00:16.210 INFO:teuthology.orchestra.run.smithi063.stdout: "repo_digests": [ 2024-03-28T22:00:16.210 INFO:teuthology.orchestra.run.smithi063.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15" 2024-03-28T22:00:16.210 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-03-28T22:00:16.210 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-28T22:00:17.014 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:00:17.014 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 19.0.0-1899-g323d23dc (323d23dc228bbcb7f16f188fdc9c0dcb801844be) squid (dev)", 2024-03-28T22:00:17.015 INFO:teuthology.orchestra.run.smithi016.stdout: "image_id": "bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52", 2024-03-28T22:00:17.015 INFO:teuthology.orchestra.run.smithi016.stdout: "repo_digests": [ 2024-03-28T22:00:17.015 INFO:teuthology.orchestra.run.smithi016.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15" 2024-03-28T22:00:17.015 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-03-28T22:00:17.015 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:00:17.041 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /etc/ceph 2024-03-28T22:00:17.079 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /etc/ceph 2024-03-28T22:00:17.119 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /etc/ceph 2024-03-28T22:00:17.159 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 777 /etc/ceph 2024-03-28T22:00:17.187 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 777 /etc/ceph 2024-03-28T22:00:17.223 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 777 /etc/ceph 2024-03-28T22:00:17.257 INFO:tasks.cephadm:Writing seed config... 2024-03-28T22:00:17.258 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-28T22:00:17.259 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-28T22:00:17.259 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-28T22:00:17.259 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-28T22:00:17.259 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-28T22:00:17.259 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-28T22:00:17.259 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-28T22:00:17.259 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-28T22:00:17.259 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-28T22:00:17.259 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-28T22:00:17.259 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-28T22:00:17.260 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-28T22:00:17.260 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-28T22:00:17.260 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-28T22:00:17.260 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-28T22:00:17.260 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-28T22:00:17.260 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-28T22:00:17.260 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-03-28T22:00:17.260 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-03-28T22:00:17.260 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-28T22:00:17.260 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-28T22:00:17.260 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-28T22:00:17.260 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-28T22:00:17.260 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-03-28T22:00:17.260 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-03-28T22:00:17.261 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-28T22:00:17.261 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-28T22:00:17.261 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-28T22:00:17.261 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-28T22:00:17.261 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-28T22:00:17.261 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-03-28T22:00:17.261 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-28T22:00:17.261 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-03-28T22:00:17.261 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-03-28T22:00:17.261 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-03-28T22:00:17.261 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-28T22:00:17.261 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-28T22:00:17.262 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-03-28T22:00:17.262 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-03-28T22:00:17.262 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-28T22:00:17.262 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-28T22:00:17.262 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-03-28T22:00:17.262 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-28T22:00:17.262 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-28T22:00:17.262 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-28T22:00:17.262 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-28T22:00:17.262 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-28T22:00:17.262 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-28T22:00:17.262 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-28T22:00:17.262 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-03-28T22:00:17.263 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-28T22:00:17.263 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-28T22:00:17.278 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 798d9884-ed4e-11ee-95d0-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-28T22:00:17.279 DEBUG:teuthology.orchestra.run.smithi016:mon.a> sudo journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mon.a.service 2024-03-28T22:00:17.321 DEBUG:teuthology.orchestra.run.smithi016:mgr.x> sudo journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mgr.x.service 2024-03-28T22:00:17.365 INFO:tasks.cephadm:Bootstrapping... 2024-03-28T22:00:17.365 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be -v bootstrap --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.16 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-28T22:00:17.561 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-28T22:00:17.561 INFO:teuthology.orchestra.run.smithi016.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be', '-v', 'bootstrap', '--fsid', '798d9884-ed4e-11ee-95d0-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.16', '--skip-admin-label'] 2024-03-28T22:00:17.585 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-28T22:00:17.585 INFO:teuthology.orchestra.run.smithi016.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-28T22:00:17.586 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying podman|docker is present... 2024-03-28T22:00:17.609 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-28T22:00:17.609 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying lvm2 is present... 2024-03-28T22:00:17.609 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying time synchronization is in place... 2024-03-28T22:00:17.616 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-28T22:00:17.616 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-28T22:00:17.622 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-28T22:00:17.622 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-03-28T22:00:17.628 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout enabled 2024-03-28T22:00:17.635 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout active 2024-03-28T22:00:17.635 INFO:teuthology.orchestra.run.smithi016.stdout:Unit chronyd.service is enabled and running 2024-03-28T22:00:17.635 INFO:teuthology.orchestra.run.smithi016.stdout:Repeating the final host check... 2024-03-28T22:00:17.657 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-28T22:00:17.657 INFO:teuthology.orchestra.run.smithi016.stdout:podman (/bin/podman) version 4.9.4 is present 2024-03-28T22:00:17.657 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl is present 2024-03-28T22:00:17.657 INFO:teuthology.orchestra.run.smithi016.stdout:lvcreate is present 2024-03-28T22:00:17.663 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-28T22:00:17.663 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-28T22:00:17.669 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-28T22:00:17.669 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-03-28T22:00:17.675 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout enabled 2024-03-28T22:00:17.681 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout active 2024-03-28T22:00:17.681 INFO:teuthology.orchestra.run.smithi016.stdout:Unit chronyd.service is enabled and running 2024-03-28T22:00:17.681 INFO:teuthology.orchestra.run.smithi016.stdout:Host looks OK 2024-03-28T22:00:17.681 INFO:teuthology.orchestra.run.smithi016.stdout:Cluster fsid: 798d9884-ed4e-11ee-95d0-87774f69a715 2024-03-28T22:00:17.681 INFO:teuthology.orchestra.run.smithi016.stdout:Acquiring lock 139996229447632 on /run/cephadm/798d9884-ed4e-11ee-95d0-87774f69a715.lock 2024-03-28T22:00:17.682 INFO:teuthology.orchestra.run.smithi016.stdout:Lock 139996229447632 acquired on /run/cephadm/798d9884-ed4e-11ee-95d0-87774f69a715.lock 2024-03-28T22:00:17.682 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying IP 172.21.15.16 port 3300 ... 2024-03-28T22:00:17.682 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying IP 172.21.15.16 port 6789 ... 2024-03-28T22:00:17.682 INFO:teuthology.orchestra.run.smithi016.stdout:Base mon IP(s) is [172.21.15.16:3300, 172.21.15.16:6789], mon addrv is [v2:172.21.15.16:3300,v1:172.21.15.16:6789] 2024-03-28T22:00:17.685 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.16 metric 100 2024-03-28T22:00:17.685 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.16 metric 100 2024-03-28T22:00:17.688 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-28T22:00:17.688 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-28T22:00:17.688 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-28T22:00:17.690 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-28T22:00:17.690 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-28T22:00:17.691 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-28T22:00:17.691 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-28T22:00:17.691 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1564/64 scope link noprefixroute 2024-03-28T22:00:17.691 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-28T22:00:17.691 INFO:teuthology.orchestra.run.smithi016.stdout:Mon IP `172.21.15.16` is in CIDR network `172.21.0.0/20` 2024-03-28T22:00:17.691 INFO:teuthology.orchestra.run.smithi016.stdout:Mon IP `172.21.15.16` is in CIDR network `172.21.0.0/20` 2024-03-28T22:00:17.692 INFO:teuthology.orchestra.run.smithi016.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-28T22:00:17.692 INFO:teuthology.orchestra.run.smithi016.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-28T22:00:17.693 INFO:teuthology.orchestra.run.smithi016.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be... 2024-03-28T22:00:19.117 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 2024-03-28T22:00:19.117 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be... 2024-03-28T22:00:19.118 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Getting image source signatures 2024-03-28T22:00:19.118 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Copying blob sha256:f2898dd983f857f34092ebd65fc601696375e5e42bb673c7cb1e443cfa2e5da5 2024-03-28T22:00:19.118 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Copying blob sha256:11da15500eca1f826c328eb3265caa30b0a3730510e2a32023084d6fe9cee20c 2024-03-28T22:00:19.118 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Copying config sha256:bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 2024-03-28T22:00:19.118 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-28T22:00:20.121 INFO:teuthology.orchestra.run.smithi016.stdout:ceph: stdout ceph version 19.0.0-1899-g323d23dc (323d23dc228bbcb7f16f188fdc9c0dcb801844be) squid (dev) 2024-03-28T22:00:20.122 INFO:teuthology.orchestra.run.smithi016.stdout:Ceph version: ceph version 19.0.0-1899-g323d23dc (323d23dc228bbcb7f16f188fdc9c0dcb801844be) squid (dev) 2024-03-28T22:00:20.122 INFO:teuthology.orchestra.run.smithi016.stdout:Extracting ceph user uid/gid from container image... 2024-03-28T22:00:21.035 INFO:teuthology.orchestra.run.smithi016.stdout:stat: stdout 167 167 2024-03-28T22:00:21.035 INFO:teuthology.orchestra.run.smithi016.stdout:Creating initial keys... 2024-03-28T22:00:21.940 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph-authtool: stdout AQB16AVmL1IFFxAAkimD7orgpsHiJ8YOg7CBUA== 2024-03-28T22:00:22.929 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph-authtool: stdout AQB26AVmV+9HDxAAci8g8ejzF8vcCeUVWVpPaw== 2024-03-28T22:00:23.943 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph-authtool: stdout AQB36AVm2RmoDhAAXDC76SWznW5CipPJQf7Zmw== 2024-03-28T22:00:23.944 INFO:teuthology.orchestra.run.smithi016.stdout:Creating initial monmap... 2024-03-28T22:00:24.824 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-28T22:00:24.824 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-28T22:00:24.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 798d9884-ed4e-11ee-95d0-87774f69a715 2024-03-28T22:00:24.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-28T22:00:24.825 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool for a [v2:172.21.15.16:3300,v1:172.21.15.16:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-28T22:00:24.825 INFO:teuthology.orchestra.run.smithi016.stdout:setting min_mon_release = quincy 2024-03-28T22:00:24.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: set fsid to 798d9884-ed4e-11ee-95d0-87774f69a715 2024-03-28T22:00:24.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-28T22:00:24.825 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:00:24.825 INFO:teuthology.orchestra.run.smithi016.stdout:Creating mon... 2024-03-28T22:00:25.773 INFO:teuthology.orchestra.run.smithi016.stdout:create mon.a on 2024-03-28T22:00:25.988 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-28T22:00:26.131 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-28T22:00:26.330 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-798d9884-ed4e-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-798d9884-ed4e-11ee-95d0-87774f69a715.target. 2024-03-28T22:00:26.330 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-798d9884-ed4e-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-798d9884-ed4e-11ee-95d0-87774f69a715.target. 2024-03-28T22:00:26.561 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mon.a 2024-03-28T22:00:26.561 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to reset failed state of unit ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mon.a.service: Unit ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mon.a.service not loaded. 2024-03-28T22:00:26.722 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-798d9884-ed4e-11ee-95d0-87774f69a715.target.wants/ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mon.a.service → /etc/systemd/system/ceph-798d9884-ed4e-11ee-95d0-87774f69a715@.service. 2024-03-28T22:00:27.248 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T22:00:27.248 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-03-28T22:00:27.255 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T22:00:27.255 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-03-28T22:00:27.255 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-03-28T22:00:27.255 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to enable service . firewalld.service is not available 2024-03-28T22:00:27.255 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mon to start... 2024-03-28T22:00:27.256 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mon... 2024-03-28T22:00:28.257 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout cluster: 2024-03-28T22:00:28.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout id: 798d9884-ed4e-11ee-95d0-87774f69a715 2024-03-28T22:00:28.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-28T22:00:28.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-03-28T22:00:28.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout services: 2024-03-28T22:00:28.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.525375s) 2024-03-28T22:00:28.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-28T22:00:28.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-28T22:00:28.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-03-28T22:00:28.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout data: 2024-03-28T22:00:28.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-28T22:00:28.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-28T22:00:28.258 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-28T22:00:28.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout pgs: 2024-03-28T22:00:28.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-03-28T22:00:28.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.659+0000 7f2f24151700 1 Processor -- start 2024-03-28T22:00:28.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.659+0000 7f2f24151700 1 -- start start 2024-03-28T22:00:28.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.659+0000 7f2f24151700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f1c107f20 0x7f2f1c108340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:00:28.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.659+0000 7f2f24151700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1c108880 con 0x7f2f1c107f20 2024-03-28T22:00:28.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.660+0000 7f2f21eed700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f1c107f20 0x7f2f1c108340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:00:28.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.660+0000 7f2f21eed700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f1c107f20 0x7f2f1c108340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58920/0 (socket says 172.21.15.16:58920) 2024-03-28T22:00:28.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.660+0000 7f2f21eed700 1 -- 172.21.15.16:0/2915911754 learned_addr learned my addr 172.21.15.16:0/2915911754 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:00:28.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.660+0000 7f2f21eed700 1 -- 172.21.15.16:0/2915911754 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f1c108fe0 con 0x7f2f1c107f20 2024-03-28T22:00:28.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.660+0000 7f2f21eed700 1 --2- 172.21.15.16:0/2915911754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f1c107f20 0x7f2f1c108340 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2f14009e80 tx=0x7f2f1400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dba0abb4b467370 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:00:28.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.660+0000 7f2f20eeb700 1 -- 172.21.15.16:0/2915911754 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f14017070 con 0x7f2f1c107f20 2024-03-28T22:00:28.259 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.660+0000 7f2f20eeb700 1 -- 172.21.15.16:0/2915911754 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f14003a20 con 0x7f2f1c107f20 2024-03-28T22:00:28.260 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.660+0000 7f2f20eeb700 1 -- 172.21.15.16:0/2915911754 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f14003d20 con 0x7f2f1c107f20 2024-03-28T22:00:28.260 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.661+0000 7f2f24151700 1 -- 172.21.15.16:0/2915911754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f1c107f20 msgr2=0x7f2f1c108340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:00:28.260 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.661+0000 7f2f24151700 1 --2- 172.21.15.16:0/2915911754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f1c107f20 0x7f2f1c108340 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2f14009e80 tx=0x7f2f1400b330 comp rx=0 tx=0).stop 2024-03-28T22:00:28.260 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.661+0000 7f2f24151700 1 -- 172.21.15.16:0/2915911754 shutdown_connections 2024-03-28T22:00:28.260 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.661+0000 7f2f24151700 1 --2- 172.21.15.16:0/2915911754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f1c107f20 0x7f2f1c108340 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:00:28.260 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.661+0000 7f2f24151700 1 -- 172.21.15.16:0/2915911754 >> 172.21.15.16:0/2915911754 conn(0x7f2f1c1034a0 msgr2=0x7f2f1c105900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:00:28.260 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.661+0000 7f2f24151700 1 -- 172.21.15.16:0/2915911754 shutdown_connections 2024-03-28T22:00:28.260 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.661+0000 7f2f24151700 1 -- 172.21.15.16:0/2915911754 wait complete. 2024-03-28T22:00:28.260 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.661+0000 7f2f24151700 1 Processor -- start 2024-03-28T22:00:28.260 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.662+0000 7f2f24151700 1 -- start start 2024-03-28T22:00:28.260 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.662+0000 7f2f24151700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f1c107f20 0x7f2f1c19b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:00:28.260 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.662+0000 7f2f24151700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1c1094b0 con 0x7f2f1c107f20 2024-03-28T22:00:28.260 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.662+0000 7f2f21eed700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f1c107f20 0x7f2f1c19b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:00:28.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.662+0000 7f2f21eed700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f1c107f20 0x7f2f1c19b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58932/0 (socket says 172.21.15.16:58932) 2024-03-28T22:00:28.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.662+0000 7f2f21eed700 1 -- 172.21.15.16:0/4266796854 learned_addr learned my addr 172.21.15.16:0/4266796854 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:00:28.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.662+0000 7f2f21eed700 1 -- 172.21.15.16:0/4266796854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f1c19bab0 con 0x7f2f1c107f20 2024-03-28T22:00:28.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.662+0000 7f2f21eed700 1 --2- 172.21.15.16:0/4266796854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f1c107f20 0x7f2f1c19b570 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c108bb0 tx=0x7f2f14004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:00:28.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.662+0000 7f2f12ffd700 1 -- 172.21.15.16:0/4266796854 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f14022020 con 0x7f2f1c107f20 2024-03-28T22:00:28.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.662+0000 7f2f12ffd700 1 -- 172.21.15.16:0/4266796854 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f14018040 con 0x7f2f1c107f20 2024-03-28T22:00:28.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.662+0000 7f2f24151700 1 -- 172.21.15.16:0/4266796854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f1c19bd40 con 0x7f2f1c107f20 2024-03-28T22:00:28.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.662+0000 7f2f12ffd700 1 -- 172.21.15.16:0/4266796854 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f14017070 con 0x7f2f1c107f20 2024-03-28T22:00:28.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.663+0000 7f2f24151700 1 -- 172.21.15.16:0/4266796854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f1c19ea30 con 0x7f2f1c107f20 2024-03-28T22:00:28.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.663+0000 7f2f12ffd700 1 -- 172.21.15.16:0/4266796854 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2f14025020 con 0x7f2f1c107f20 2024-03-28T22:00:28.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.663+0000 7f2f12ffd700 1 -- 172.21.15.16:0/4266796854 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2f1400f050 con 0x7f2f1c107f20 2024-03-28T22:00:28.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.663+0000 7f2f24151700 1 -- 172.21.15.16:0/4266796854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f1c063870 con 0x7f2f1c107f20 2024-03-28T22:00:28.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.666+0000 7f2f12ffd700 1 -- 172.21.15.16:0/4266796854 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f2f1402c030 con 0x7f2f1c107f20 2024-03-28T22:00:28.262 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.739+0000 7f2f24151700 1 -- 172.21.15.16:0/4266796854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f2f1c02dcc0 con 0x7f2f1c107f20 2024-03-28T22:00:28.262 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.740+0000 7f2f12ffd700 1 -- 172.21.15.16:0/4266796854 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f2f1401e480 con 0x7f2f1c107f20 2024-03-28T22:00:28.262 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.741+0000 7f2f24151700 1 -- 172.21.15.16:0/4266796854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f1c107f20 msgr2=0x7f2f1c19b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:00:28.262 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.741+0000 7f2f24151700 1 --2- 172.21.15.16:0/4266796854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f1c107f20 0x7f2f1c19b570 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c108bb0 tx=0x7f2f14004ad0 comp rx=0 tx=0).stop 2024-03-28T22:00:28.262 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.741+0000 7f2f24151700 1 -- 172.21.15.16:0/4266796854 shutdown_connections 2024-03-28T22:00:28.262 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.741+0000 7f2f24151700 1 --2- 172.21.15.16:0/4266796854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f1c107f20 0x7f2f1c19b570 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:00:28.262 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.741+0000 7f2f24151700 1 -- 172.21.15.16:0/4266796854 >> 172.21.15.16:0/4266796854 conn(0x7f2f1c1034a0 msgr2=0x7f2f1c1921d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:00:28.262 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.741+0000 7f2f24151700 1 -- 172.21.15.16:0/4266796854 shutdown_connections 2024-03-28T22:00:28.262 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:27.741+0000 7f2f24151700 1 -- 172.21.15.16:0/4266796854 wait complete. 2024-03-28T22:00:28.262 INFO:teuthology.orchestra.run.smithi016.stdout:mon is available 2024-03-28T22:00:28.262 INFO:teuthology.orchestra.run.smithi016.stdout:Assimilating anything we can from ceph.conf... 2024-03-28T22:00:29.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-03-28T22:00:29.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [global] 2024-03-28T22:00:29.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout fsid = 798d9884-ed4e-11ee-95d0-87774f69a715 2024-03-28T22:00:29.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.16:3300,v1:172.21.15.16:6789] 2024-03-28T22:00:29.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-28T22:00:29.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-28T22:00:29.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-28T22:00:29.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-28T22:00:29.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-03-28T22:00:29.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-28T22:00:29.338 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-28T22:00:29.339 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-03-28T22:00:29.339 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [osd] 2024-03-28T22:00:29.339 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-28T22:00:29.339 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-28T22:00:29.339 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-28T22:00:29.339 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-28T22:00:29.339 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-28T22:00:29.339 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.702+0000 7f220a12c700 1 Processor -- start 2024-03-28T22:00:29.339 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.702+0000 7f220a12c700 1 -- start start 2024-03-28T22:00:29.339 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.702+0000 7f220a12c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2204108920 0x7f220410ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:00:29.339 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.702+0000 7f220a12c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f220407be50 con 0x7f2204108920 2024-03-28T22:00:29.339 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.702+0000 7f22037fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2204108920 0x7f220410ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:00:29.339 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.702+0000 7f22037fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2204108920 0x7f220410ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58946/0 (socket says 172.21.15.16:58946) 2024-03-28T22:00:29.339 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.702+0000 7f22037fe700 1 -- 172.21.15.16:0/1749955198 learned_addr learned my addr 172.21.15.16:0/1749955198 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:00:29.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.702+0000 7f22037fe700 1 -- 172.21.15.16:0/1749955198 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f220410b250 con 0x7f2204108920 2024-03-28T22:00:29.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.702+0000 7f22037fe700 1 --2- 172.21.15.16:0/1749955198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2204108920 0x7f220410ad10 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f21f40098a0 tx=0x7f21f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0f3bb2b7f3eb37c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:00:29.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.703+0000 7f22027fc700 1 -- 172.21.15.16:0/1749955198 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f21f4004160 con 0x7f2204108920 2024-03-28T22:00:29.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.703+0000 7f22027fc700 1 -- 172.21.15.16:0/1749955198 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21f4004300 con 0x7f2204108920 2024-03-28T22:00:29.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.703+0000 7f22027fc700 1 -- 172.21.15.16:0/1749955198 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f21f40036a0 con 0x7f2204108920 2024-03-28T22:00:29.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.703+0000 7f220a12c700 1 -- 172.21.15.16:0/1749955198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2204108920 msgr2=0x7f220410ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:00:29.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.703+0000 7f220a12c700 1 --2- 172.21.15.16:0/1749955198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2204108920 0x7f220410ad10 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f21f40098a0 tx=0x7f21f4009bb0 comp rx=0 tx=0).stop 2024-03-28T22:00:29.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.703+0000 7f220a12c700 1 -- 172.21.15.16:0/1749955198 shutdown_connections 2024-03-28T22:00:29.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.703+0000 7f220a12c700 1 --2- 172.21.15.16:0/1749955198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2204108920 0x7f220410ad10 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:00:29.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.703+0000 7f220a12c700 1 -- 172.21.15.16:0/1749955198 >> 172.21.15.16:0/1749955198 conn(0x7f2204102310 msgr2=0x7f2204104750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:00:29.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.703+0000 7f220a12c700 1 -- 172.21.15.16:0/1749955198 shutdown_connections 2024-03-28T22:00:29.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.703+0000 7f220a12c700 1 -- 172.21.15.16:0/1749955198 wait complete. 2024-03-28T22:00:29.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.704+0000 7f220a12c700 1 Processor -- start 2024-03-28T22:00:29.340 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.704+0000 7f220a12c700 1 -- start start 2024-03-28T22:00:29.341 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.704+0000 7f220a12c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2204108920 0x7f220419b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:00:29.341 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.704+0000 7f220a12c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f220410b6c0 con 0x7f2204108920 2024-03-28T22:00:29.341 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.704+0000 7f22037fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2204108920 0x7f220419b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:00:29.341 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.704+0000 7f22037fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2204108920 0x7f220419b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58962/0 (socket says 172.21.15.16:58962) 2024-03-28T22:00:29.341 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.704+0000 7f22037fe700 1 -- 172.21.15.16:0/1980894560 learned_addr learned my addr 172.21.15.16:0/1980894560 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:00:29.341 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.704+0000 7f22037fe700 1 -- 172.21.15.16:0/1980894560 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f220419bab0 con 0x7f2204108920 2024-03-28T22:00:29.341 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.705+0000 7f22037fe700 1 --2- 172.21.15.16:0/1980894560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2204108920 0x7f220419b570 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f21f4000c00 tx=0x7f21f4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:00:29.341 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.705+0000 7f2200ff9700 1 -- 172.21.15.16:0/1980894560 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f21f4003db0 con 0x7f2204108920 2024-03-28T22:00:29.341 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.705+0000 7f2200ff9700 1 -- 172.21.15.16:0/1980894560 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21f4011630 con 0x7f2204108920 2024-03-28T22:00:29.341 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.705+0000 7f220a12c700 1 -- 172.21.15.16:0/1980894560 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f220419bd40 con 0x7f2204108920 2024-03-28T22:00:29.341 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.705+0000 7f2200ff9700 1 -- 172.21.15.16:0/1980894560 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f21f4011950 con 0x7f2204108920 2024-03-28T22:00:29.341 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.705+0000 7f220a12c700 1 -- 172.21.15.16:0/1980894560 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f220419e9e0 con 0x7f2204108920 2024-03-28T22:00:29.341 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.705+0000 7f2200ff9700 1 -- 172.21.15.16:0/1980894560 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f21f4011af0 con 0x7f2204108920 2024-03-28T22:00:29.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.705+0000 7f2200ff9700 1 -- 172.21.15.16:0/1980894560 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f21f401c450 con 0x7f2204108920 2024-03-28T22:00:29.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.706+0000 7f220a12c700 1 -- 172.21.15.16:0/1980894560 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2204063870 con 0x7f2204108920 2024-03-28T22:00:29.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.708+0000 7f2200ff9700 1 -- 172.21.15.16:0/1980894560 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f21f4032020 con 0x7f2204108920 2024-03-28T22:00:29.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.778+0000 7f220a12c700 1 -- 172.21.15.16:0/1980894560 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f220402dda0 con 0x7f2204108920 2024-03-28T22:00:29.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.781+0000 7f2200ff9700 1 -- 172.21.15.16:0/1980894560 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+429 (secure 0 0 0) 0x7f21f4004260 con 0x7f2204108920 2024-03-28T22:00:29.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.782+0000 7f2200ff9700 1 -- 172.21.15.16:0/1980894560 <== mon.0 v2:172.21.15.16:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f21f401cb60 con 0x7f2204108920 2024-03-28T22:00:29.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.783+0000 7f220a12c700 1 -- 172.21.15.16:0/1980894560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2204108920 msgr2=0x7f220419b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:00:29.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.783+0000 7f220a12c700 1 --2- 172.21.15.16:0/1980894560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2204108920 0x7f220419b570 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f21f4000c00 tx=0x7f21f4004750 comp rx=0 tx=0).stop 2024-03-28T22:00:29.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.783+0000 7f220a12c700 1 -- 172.21.15.16:0/1980894560 shutdown_connections 2024-03-28T22:00:29.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.783+0000 7f220a12c700 1 --2- 172.21.15.16:0/1980894560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2204108920 0x7f220419b570 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:00:29.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.783+0000 7f220a12c700 1 -- 172.21.15.16:0/1980894560 >> 172.21.15.16:0/1980894560 conn(0x7f2204102310 msgr2=0x7f2204102740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:00:29.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.783+0000 7f220a12c700 1 -- 172.21.15.16:0/1980894560 shutdown_connections 2024-03-28T22:00:29.342 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:28.783+0000 7f220a12c700 1 -- 172.21.15.16:0/1980894560 wait complete. 2024-03-28T22:00:29.343 INFO:teuthology.orchestra.run.smithi016.stdout:Generating new minimal ceph.conf... 2024-03-28T22:00:30.419 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:30 smithi016 ceph-mon[31996]: from='client.? 172.21.15.16:0/1683016696' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:00:30.426 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.751+0000 7fed8b641700 1 Processor -- start 2024-03-28T22:00:30.426 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.751+0000 7fed8b641700 1 -- start start 2024-03-28T22:00:30.426 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.751+0000 7fed8b641700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed8407ddc0 0x7fed8407c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:00:30.426 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.751+0000 7fed8b641700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fed8407c800 con 0x7fed8407ddc0 2024-03-28T22:00:30.426 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.752+0000 7fed893dd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed8407ddc0 0x7fed8407c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:00:30.426 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.752+0000 7fed893dd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed8407ddc0 0x7fed8407c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58968/0 (socket says 172.21.15.16:58968) 2024-03-28T22:00:30.426 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.752+0000 7fed893dd700 1 -- 172.21.15.16:0/1850212341 learned_addr learned my addr 172.21.15.16:0/1850212341 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:00:30.426 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.752+0000 7fed893dd700 1 -- 172.21.15.16:0/1850212341 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed8407c980 con 0x7fed8407ddc0 2024-03-28T22:00:30.426 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.752+0000 7fed893dd700 1 --2- 172.21.15.16:0/1850212341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed8407ddc0 0x7fed8407c2c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fed800098a0 tx=0x7fed80009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5a954ef096b9134 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:00:30.426 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.752+0000 7fed7bfff700 1 -- 172.21.15.16:0/1850212341 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed80004160 con 0x7fed8407ddc0 2024-03-28T22:00:30.426 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.752+0000 7fed7bfff700 1 -- 172.21.15.16:0/1850212341 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fed80004300 con 0x7fed8407ddc0 2024-03-28T22:00:30.427 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.752+0000 7fed7bfff700 1 -- 172.21.15.16:0/1850212341 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed80011b50 con 0x7fed8407ddc0 2024-03-28T22:00:30.427 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.753+0000 7fed8b641700 1 -- 172.21.15.16:0/1850212341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed8407ddc0 msgr2=0x7fed8407c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:00:30.427 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.753+0000 7fed8b641700 1 --2- 172.21.15.16:0/1850212341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed8407ddc0 0x7fed8407c2c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fed800098a0 tx=0x7fed80009bb0 comp rx=0 tx=0).stop 2024-03-28T22:00:30.427 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.753+0000 7fed8b641700 1 -- 172.21.15.16:0/1850212341 shutdown_connections 2024-03-28T22:00:30.427 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.753+0000 7fed8b641700 1 --2- 172.21.15.16:0/1850212341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed8407ddc0 0x7fed8407c2c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:00:30.427 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.753+0000 7fed8b641700 1 -- 172.21.15.16:0/1850212341 >> 172.21.15.16:0/1850212341 conn(0x7fed841034a0 msgr2=0x7fed84105900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:00:30.427 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.753+0000 7fed8b641700 1 -- 172.21.15.16:0/1850212341 shutdown_connections 2024-03-28T22:00:30.427 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.753+0000 7fed8b641700 1 -- 172.21.15.16:0/1850212341 wait complete. 2024-03-28T22:00:30.427 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.753+0000 7fed8b641700 1 Processor -- start 2024-03-28T22:00:30.427 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.754+0000 7fed8b641700 1 -- start start 2024-03-28T22:00:30.427 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.754+0000 7fed8b641700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed8407ddc0 0x7fed8419e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:00:30.427 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.754+0000 7fed8b641700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fed8410e490 con 0x7fed8407ddc0 2024-03-28T22:00:30.427 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.754+0000 7fed893dd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed8407ddc0 0x7fed8419e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:00:30.427 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.754+0000 7fed893dd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed8407ddc0 0x7fed8419e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58970/0 (socket says 172.21.15.16:58970) 2024-03-28T22:00:30.428 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.754+0000 7fed893dd700 1 -- 172.21.15.16:0/1683016696 learned_addr learned my addr 172.21.15.16:0/1683016696 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:00:30.428 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.754+0000 7fed893dd700 1 -- 172.21.15.16:0/1683016696 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed8419e540 con 0x7fed8407ddc0 2024-03-28T22:00:30.428 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.754+0000 7fed893dd700 1 --2- 172.21.15.16:0/1683016696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed8407ddc0 0x7fed8419e000 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fed80011670 tx=0x7fed80011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:00:30.428 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.754+0000 7fed7a7fc700 1 -- 172.21.15.16:0/1683016696 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed800039c0 con 0x7fed8407ddc0 2024-03-28T22:00:30.428 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.754+0000 7fed7a7fc700 1 -- 172.21.15.16:0/1683016696 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fed80011e50 con 0x7fed8407ddc0 2024-03-28T22:00:30.428 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.754+0000 7fed7a7fc700 1 -- 172.21.15.16:0/1683016696 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed8001a5c0 con 0x7fed8407ddc0 2024-03-28T22:00:30.428 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.754+0000 7fed8b641700 1 -- 172.21.15.16:0/1683016696 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed8419e7d0 con 0x7fed8407ddc0 2024-03-28T22:00:30.428 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.754+0000 7fed8b641700 1 -- 172.21.15.16:0/1683016696 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed841a14c0 con 0x7fed8407ddc0 2024-03-28T22:00:30.428 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.755+0000 7fed7a7fc700 1 -- 172.21.15.16:0/1683016696 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fed8001ac00 con 0x7fed8407ddc0 2024-03-28T22:00:30.428 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.755+0000 7fed7a7fc700 1 -- 172.21.15.16:0/1683016696 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fed80019700 con 0x7fed8407ddc0 2024-03-28T22:00:30.428 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.755+0000 7fed8b641700 1 -- 172.21.15.16:0/1683016696 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed84063870 con 0x7fed8407ddc0 2024-03-28T22:00:30.428 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.758+0000 7fed7a7fc700 1 -- 172.21.15.16:0/1683016696 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fed80003b60 con 0x7fed8407ddc0 2024-03-28T22:00:30.428 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.828+0000 7fed8b641700 1 -- 172.21.15.16:0/1683016696 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fed841a1690 con 0x7fed8407ddc0 2024-03-28T22:00:30.428 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.829+0000 7fed7a7fc700 1 -- 172.21.15.16:0/1683016696 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7fed800199a0 con 0x7fed8407ddc0 2024-03-28T22:00:30.428 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.829+0000 7fed8b641700 1 -- 172.21.15.16:0/1683016696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed8407ddc0 msgr2=0x7fed8419e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:00:30.429 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.829+0000 7fed8b641700 1 --2- 172.21.15.16:0/1683016696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed8407ddc0 0x7fed8419e000 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fed80011670 tx=0x7fed80011b60 comp rx=0 tx=0).stop 2024-03-28T22:00:30.429 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.829+0000 7fed8b641700 1 -- 172.21.15.16:0/1683016696 shutdown_connections 2024-03-28T22:00:30.429 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.829+0000 7fed8b641700 1 --2- 172.21.15.16:0/1683016696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed8407ddc0 0x7fed8419e000 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:00:30.429 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.829+0000 7fed8b641700 1 -- 172.21.15.16:0/1683016696 >> 172.21.15.16:0/1683016696 conn(0x7fed841034a0 msgr2=0x7fed84104050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:00:30.429 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.830+0000 7fed8b641700 1 -- 172.21.15.16:0/1683016696 shutdown_connections 2024-03-28T22:00:30.429 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:29.830+0000 7fed8b641700 1 -- 172.21.15.16:0/1683016696 wait complete. 2024-03-28T22:00:30.429 INFO:teuthology.orchestra.run.smithi016.stdout:Restarting the monitor... 2024-03-28T22:00:30.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:30 smithi016 systemd[1]: Stopping Ceph mon.a for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:00:31.180 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:30 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mon-a[31992]: 2024-03-28T22:00:30.694+0000 7fa203e3f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:00:31.180 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:30 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mon-a[31992]: 2024-03-28T22:00:30.695+0000 7fa203e3f700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-28T22:00:31.180 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:30 smithi016 podman[32243]: 2024-03-28 22:00:30.930196987 +0000 UTC m=+0.341061874 container died 3a1a27d7fbd555c2f97fbc8abd1598c47c6dbc4bd66d5005d608cf3f5843c6f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mon-a, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-323d23d, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-28T22:00:31.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:31 smithi016 podman[32243]: 2024-03-28 22:00:31.286625776 +0000 UTC m=+0.697490663 container cleanup 3a1a27d7fbd555c2f97fbc8abd1598c47c6dbc4bd66d5005d608cf3f5843c6f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mon-a, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-323d23d, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux ) 2024-03-28T22:00:31.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:31 smithi016 bash[32243]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mon-a 2024-03-28T22:00:31.715 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:31 smithi016 podman[32256]: 2024-03-28 22:00:31.462405004 +0000 UTC m=+0.527583081 container remove 3a1a27d7fbd555c2f97fbc8abd1598c47c6dbc4bd66d5005d608cf3f5843c6f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mon-a, RELEASE=squid-323d23d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-28T22:00:32.040 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:31 smithi016 systemd[1]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-28T22:00:32.041 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:31 smithi016 systemd[1]: Stopped Ceph mon.a for 798d9884-ed4e-11ee-95d0-87774f69a715. 2024-03-28T22:00:32.041 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:31 smithi016 systemd[1]: Starting Ceph mon.a for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:00:32.041 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:31 smithi016 podman[32355]: 2024-03-28 22:00:31.829590886 +0000 UTC m=+0.017752886 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be 2024-03-28T22:00:32.041 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:31 smithi016 podman[32355]: 2024-03-28 22:00:31.929768089 +0000 UTC m=+0.117930077 container create 89528bfea73262e13c40e4724f4059497f435302350965eed236cd717fbd566c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mon-a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-323d23d) 2024-03-28T22:00:32.246 INFO:teuthology.orchestra.run.smithi016.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-28T22:00:32.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 podman[32355]: 2024-03-28 22:00:32.182321351 +0000 UTC m=+0.370483353 container init 89528bfea73262e13c40e4724f4059497f435302350965eed236cd717fbd566c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mon-a, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_CLEAN=True, RELEASE=squid-323d23d, org.label-schema.schema-version=1.0, ceph=True) 2024-03-28T22:00:32.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 podman[32355]: 2024-03-28 22:00:32.185226311 +0000 UTC m=+0.373388301 container start 89528bfea73262e13c40e4724f4059497f435302350965eed236cd717fbd566c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mon-a, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, RELEASE=squid-323d23d, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-28T22:00:32.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: set uid:gid to 167:167 (ceph:ceph) 2024-03-28T22:00:32.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: ceph version 19.0.0-1899-g323d23dc (323d23dc228bbcb7f16f188fdc9c0dcb801844be) squid (dev), process ceph-mon, pid 2 2024-03-28T22:00:32.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: pidfile_write: ignore empty --pid-file 2024-03-28T22:00:32.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: load: jerasure load: lrc 2024-03-28T22:00:32.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: RocksDB version: 7.9.2 2024-03-28T22:00:32.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Git sha 0 2024-03-28T22:00:32.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Compile date 2024-03-27 16:17:23 2024-03-28T22:00:32.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: DB SUMMARY 2024-03-28T22:00:32.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: DB Session ID: VKKD6P4FFAJM7OKN3314 2024-03-28T22:00:32.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: CURRENT file: CURRENT 2024-03-28T22:00:32.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: IDENTITY file: IDENTITY 2024-03-28T22:00:32.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-28T22:00:32.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-28T22:00:32.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 106680 ; 2024-03-28T22:00:32.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.error_if_exists: 0 2024-03-28T22:00:32.438 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.create_if_missing: 0 2024-03-28T22:00:32.438 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.paranoid_checks: 1 2024-03-28T22:00:32.438 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-28T22:00:32.438 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-28T22:00:32.438 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-28T22:00:32.438 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.env: 0x55c1d2d36c60 2024-03-28T22:00:32.438 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.fs: PosixFileSystem 2024-03-28T22:00:32.438 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.info_log: 0x55c1d4e6dae0 2024-03-28T22:00:32.438 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.max_file_opening_threads: 16 2024-03-28T22:00:32.438 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.statistics: (nil) 2024-03-28T22:00:32.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.use_fsync: 0 2024-03-28T22:00:32.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.max_log_file_size: 0 2024-03-28T22:00:32.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-28T22:00:32.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-28T22:00:32.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.keep_log_file_num: 1000 2024-03-28T22:00:32.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.recycle_log_file_num: 0 2024-03-28T22:00:32.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.allow_fallocate: 1 2024-03-28T22:00:32.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.allow_mmap_reads: 0 2024-03-28T22:00:32.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.allow_mmap_writes: 0 2024-03-28T22:00:32.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.use_direct_reads: 0 2024-03-28T22:00:32.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-28T22:00:32.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.create_missing_column_families: 0 2024-03-28T22:00:32.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.db_log_dir: 2024-03-28T22:00:32.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.wal_dir: 2024-03-28T22:00:32.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-28T22:00:32.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-28T22:00:32.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-28T22:00:32.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-28T22:00:32.444 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-28T22:00:32.444 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-28T22:00:32.444 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.advise_random_on_open: 1 2024-03-28T22:00:32.444 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.db_write_buffer_size: 0 2024-03-28T22:00:32.444 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.write_buffer_manager: 0x55c1d4e401e0 2024-03-28T22:00:32.444 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-28T22:00:32.444 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-28T22:00:32.444 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-28T22:00:32.444 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.rate_limiter: (nil) 2024-03-28T22:00:32.444 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-28T22:00:32.444 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.wal_recovery_mode: 2 2024-03-28T22:00:32.444 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.enable_thread_tracking: 0 2024-03-28T22:00:32.444 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.enable_pipelined_write: 0 2024-03-28T22:00:32.444 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.unordered_write: 0 2024-03-28T22:00:32.444 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-28T22:00:32.444 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-28T22:00:32.445 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-28T22:00:32.445 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-28T22:00:32.445 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.row_cache: None 2024-03-28T22:00:32.445 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.wal_filter: None 2024-03-28T22:00:32.445 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-28T22:00:32.445 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.allow_ingest_behind: 0 2024-03-28T22:00:32.445 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.two_write_queues: 0 2024-03-28T22:00:32.445 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.manual_wal_flush: 0 2024-03-28T22:00:32.445 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.wal_compression: 0 2024-03-28T22:00:32.445 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.atomic_flush: 0 2024-03-28T22:00:32.445 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-28T22:00:32.445 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-28T22:00:32.445 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-28T22:00:32.445 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.log_readahead_size: 0 2024-03-28T22:00:32.445 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-28T22:00:32.446 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.best_efforts_recovery: 0 2024-03-28T22:00:32.446 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-28T22:00:32.446 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-28T22:00:32.446 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.allow_data_in_errors: 0 2024-03-28T22:00:32.446 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.db_host_id: __hostname__ 2024-03-28T22:00:32.446 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-28T22:00:32.446 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.max_background_jobs: 2 2024-03-28T22:00:32.446 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.max_background_compactions: -1 2024-03-28T22:00:32.446 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.max_subcompactions: 1 2024-03-28T22:00:32.446 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-28T22:00:32.446 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-28T22:00:32.446 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-28T22:00:32.446 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.max_total_wal_size: 0 2024-03-28T22:00:32.446 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-28T22:00:32.446 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-28T22:00:32.447 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-28T22:00:32.447 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-28T22:00:32.447 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.max_open_files: -1 2024-03-28T22:00:32.447 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.bytes_per_sync: 0 2024-03-28T22:00:32.447 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-28T22:00:32.447 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-28T22:00:32.447 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.compaction_readahead_size: 0 2024-03-28T22:00:32.447 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.max_background_flushes: -1 2024-03-28T22:00:32.447 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Compression algorithms supported: 2024-03-28T22:00:32.447 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-28T22:00:32.447 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: kZSTD supported: 0 2024-03-28T22:00:32.447 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: kXpressCompression supported: 0 2024-03-28T22:00:32.447 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: kLZ4HCCompression supported: 1 2024-03-28T22:00:32.447 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: kZlibCompression supported: 1 2024-03-28T22:00:32.447 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: kSnappyCompression supported: 1 2024-03-28T22:00:32.447 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: kLZ4Compression supported: 1 2024-03-28T22:00:32.448 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: kBZip2Compression supported: 0 2024-03-28T22:00:32.448 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-28T22:00:32.448 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-28T22:00:32.448 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-28T22:00:32.448 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-28T22:00:32.448 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-28T22:00:32.448 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.merge_operator: 2024-03-28T22:00:32.448 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.compaction_filter: None 2024-03-28T22:00:32.448 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.compaction_filter_factory: None 2024-03-28T22:00:32.448 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.sst_partitioner_factory: None 2024-03-28T22:00:32.448 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-28T22:00:32.448 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-28T22:00:32.448 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c1d4e6dbe0) 2024-03-28T22:00:32.449 INFO:journalctl@ceph.mon.a.smithi016.stdout: cache_index_and_filter_blocks: 1 2024-03-28T22:00:32.449 INFO:journalctl@ceph.mon.a.smithi016.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-28T22:00:32.449 INFO:journalctl@ceph.mon.a.smithi016.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-28T22:00:32.449 INFO:journalctl@ceph.mon.a.smithi016.stdout: pin_top_level_index_and_filter: 1 2024-03-28T22:00:32.449 INFO:journalctl@ceph.mon.a.smithi016.stdout: index_type: 0 2024-03-28T22:00:32.449 INFO:journalctl@ceph.mon.a.smithi016.stdout: data_block_index_type: 0 2024-03-28T22:00:32.449 INFO:journalctl@ceph.mon.a.smithi016.stdout: index_shortening: 1 2024-03-28T22:00:32.449 INFO:journalctl@ceph.mon.a.smithi016.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-28T22:00:32.449 INFO:journalctl@ceph.mon.a.smithi016.stdout: checksum: 4 2024-03-28T22:00:32.449 INFO:journalctl@ceph.mon.a.smithi016.stdout: no_block_cache: 0 2024-03-28T22:00:32.449 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache: 0x55c1d4e71090 2024-03-28T22:00:32.449 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache_name: BinnedLRUCache 2024-03-28T22:00:32.449 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache_options: 2024-03-28T22:00:32.449 INFO:journalctl@ceph.mon.a.smithi016.stdout: capacity : 536870912 2024-03-28T22:00:32.449 INFO:journalctl@ceph.mon.a.smithi016.stdout: num_shard_bits : 4 2024-03-28T22:00:32.450 INFO:journalctl@ceph.mon.a.smithi016.stdout: strict_capacity_limit : 0 2024-03-28T22:00:32.450 INFO:journalctl@ceph.mon.a.smithi016.stdout: high_pri_pool_ratio: 0.000 2024-03-28T22:00:32.450 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache_compressed: (nil) 2024-03-28T22:00:32.450 INFO:journalctl@ceph.mon.a.smithi016.stdout: persistent_cache: (nil) 2024-03-28T22:00:32.450 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_size: 4096 2024-03-28T22:00:32.450 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_size_deviation: 10 2024-03-28T22:00:32.450 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_restart_interval: 16 2024-03-28T22:00:32.450 INFO:journalctl@ceph.mon.a.smithi016.stdout: index_block_restart_interval: 1 2024-03-28T22:00:32.450 INFO:journalctl@ceph.mon.a.smithi016.stdout: metadata_block_size: 4096 2024-03-28T22:00:32.450 INFO:journalctl@ceph.mon.a.smithi016.stdout: partition_filters: 0 2024-03-28T22:00:32.450 INFO:journalctl@ceph.mon.a.smithi016.stdout: use_delta_encoding: 1 2024-03-28T22:00:32.450 INFO:journalctl@ceph.mon.a.smithi016.stdout: filter_policy: bloomfilter 2024-03-28T22:00:32.450 INFO:journalctl@ceph.mon.a.smithi016.stdout: whole_key_filtering: 1 2024-03-28T22:00:32.450 INFO:journalctl@ceph.mon.a.smithi016.stdout: verify_compression: 0 2024-03-28T22:00:32.451 INFO:journalctl@ceph.mon.a.smithi016.stdout: read_amp_bytes_per_bit: 0 2024-03-28T22:00:32.451 INFO:journalctl@ceph.mon.a.smithi016.stdout: format_version: 5 2024-03-28T22:00:32.451 INFO:journalctl@ceph.mon.a.smithi016.stdout: enable_index_compression: 1 2024-03-28T22:00:32.451 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_align: 0 2024-03-28T22:00:32.451 INFO:journalctl@ceph.mon.a.smithi016.stdout: max_auto_readahead_size: 262144 2024-03-28T22:00:32.451 INFO:journalctl@ceph.mon.a.smithi016.stdout: prepopulate_block_cache: 0 2024-03-28T22:00:32.451 INFO:journalctl@ceph.mon.a.smithi016.stdout: initial_auto_readahead_size: 8192 2024-03-28T22:00:32.451 INFO:journalctl@ceph.mon.a.smithi016.stdout: num_file_reads_for_auto_readahead: 2 2024-03-28T22:00:32.451 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.write_buffer_size: 33554432 2024-03-28T22:00:32.451 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.max_write_buffer_number: 2 2024-03-28T22:00:32.451 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.compression: NoCompression 2024-03-28T22:00:32.451 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.bottommost_compression: Disabled 2024-03-28T22:00:32.451 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.prefix_extractor: nullptr 2024-03-28T22:00:32.451 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-28T22:00:32.452 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.num_levels: 7 2024-03-28T22:00:32.452 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-28T22:00:32.452 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-28T22:00:32.452 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-28T22:00:32.452 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-28T22:00:32.452 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-28T22:00:32.452 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-28T22:00:32.452 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-28T22:00:32.452 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-28T22:00:32.452 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-28T22:00:32.452 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-28T22:00:32.452 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-28T22:00:32.452 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-28T22:00:32.452 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-28T22:00:32.452 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.compression_opts.level: 32767 2024-03-28T22:00:32.453 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.compression_opts.strategy: 0 2024-03-28T22:00:32.453 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-28T22:00:32.453 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-28T22:00:32.453 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-28T22:00:32.453 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-28T22:00:32.453 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.compression_opts.enabled: false 2024-03-28T22:00:32.453 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-28T22:00:32.453 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-28T22:00:32.453 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-28T22:00:32.453 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-28T22:00:32.453 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.target_file_size_base: 67108864 2024-03-28T22:00:32.453 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-28T22:00:32.453 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-28T22:00:32.453 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-28T22:00:32.454 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-28T22:00:32.454 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-28T22:00:32.454 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-28T22:00:32.454 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-28T22:00:32.454 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-28T22:00:32.454 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-28T22:00:32.454 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-28T22:00:32.454 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-28T22:00:32.454 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-28T22:00:32.454 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-28T22:00:32.454 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-28T22:00:32.454 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.arena_block_size: 1048576 2024-03-28T22:00:32.454 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-28T22:00:32.455 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-28T22:00:32.455 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.disable_auto_compactions: 0 2024-03-28T22:00:32.455 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-28T22:00:32.455 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-28T22:00:32.455 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-28T22:00:32.455 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-28T22:00:32.455 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-28T22:00:32.455 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-28T22:00:32.455 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-28T22:00:32.455 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-28T22:00:32.455 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-28T22:00:32.455 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-28T22:00:32.455 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.table_properties_collectors: 2024-03-28T22:00:32.455 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.inplace_update_support: 0 2024-03-28T22:00:32.455 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-28T22:00:32.456 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-28T22:00:32.456 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-28T22:00:32.456 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-28T22:00:32.456 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.bloom_locality: 0 2024-03-28T22:00:32.456 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.max_successive_merges: 0 2024-03-28T22:00:32.456 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-28T22:00:32.456 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.paranoid_file_checks: 0 2024-03-28T22:00:32.456 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.force_consistency_checks: 1 2024-03-28T22:00:32.456 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.report_bg_io_stats: 0 2024-03-28T22:00:32.456 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.ttl: 2592000 2024-03-28T22:00:32.456 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-28T22:00:32.456 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-28T22:00:32.456 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-28T22:00:32.456 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.enable_blob_files: false 2024-03-28T22:00:32.457 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.min_blob_size: 0 2024-03-28T22:00:32.457 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.blob_file_size: 268435456 2024-03-28T22:00:32.457 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-28T22:00:32.457 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-28T22:00:32.457 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-28T22:00:32.457 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-28T22:00:32.457 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-28T22:00:32.457 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.blob_file_starting_level: 0 2024-03-28T22:00:32.457 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-28T22:00:32.457 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-28T22:00:32.457 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-28T22:00:32.457 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 42affade-edc4-4812-b77b-86f5b529d0db 2024-03-28T22:00:32.457 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711663232214739, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-28T22:00:32.457 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-28T22:00:32.457 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711663232216186, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 101539, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 334, "table_properties": {"data_size": 99457, "index_size": 300, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14657, "raw_average_key_size": 50, "raw_value_size": 91568, "raw_average_value_size": 312, "num_data_blocks": 12, "num_entries": 293, "num_filter_entries": 293, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711663232, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "42affade-edc4-4812-b77b-86f5b529d0db", "db_session_id": "VKKD6P4FFAJM7OKN3314", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-28T22:00:32.458 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711663232216297, "job": 1, "event": "recovery_finished"} 2024-03-28T22:00:32.458 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-28T22:00:32.458 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-28T22:00:32.458 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55c1d4f62000 2024-03-28T22:00:32.458 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: DB pointer 0x55c1d4f4c000 2024-03-28T22:00:32.458 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-28T22:00:32.458 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-28T22:00:32.458 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** DB Stats ** 2024-03-28T22:00:32.458 INFO:journalctl@ceph.mon.a.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-28T22:00:32.458 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-28T22:00:32.458 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-28T22:00:32.458 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-28T22:00:32.458 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-28T22:00:32.459 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-28T22:00:32.459 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-28T22:00:32.459 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-03-28T22:00:32.459 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** Compaction Stats [default] ** 2024-03-28T22:00:32.459 INFO:journalctl@ceph.mon.a.smithi016.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-28T22:00:32.459 INFO:journalctl@ceph.mon.a.smithi016.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-28T22:00:32.459 INFO:journalctl@ceph.mon.a.smithi016.stdout: L0 2/0 100.99 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 115.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T22:00:32.459 INFO:journalctl@ceph.mon.a.smithi016.stdout: Sum 2/0 100.99 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 115.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T22:00:32.459 INFO:journalctl@ceph.mon.a.smithi016.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 115.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T22:00:32.459 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-03-28T22:00:32.459 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** Compaction Stats [default] ** 2024-03-28T22:00:32.459 INFO:journalctl@ceph.mon.a.smithi016.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-28T22:00:32.459 INFO:journalctl@ceph.mon.a.smithi016.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-28T22:00:32.459 INFO:journalctl@ceph.mon.a.smithi016.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 115.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T22:00:32.460 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-03-28T22:00:32.460 INFO:journalctl@ceph.mon.a.smithi016.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-28T22:00:32.460 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-03-28T22:00:32.460 INFO:journalctl@ceph.mon.a.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-28T22:00:32.460 INFO:journalctl@ceph.mon.a.smithi016.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-28T22:00:32.460 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-28T22:00:32.460 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-28T22:00:32.460 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-28T22:00:32.460 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-28T22:00:32.460 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative compaction: 0.00 GB write, 22.92 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-28T22:00:32.460 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval compaction: 0.00 GB write, 22.92 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-28T22:00:32.460 INFO:journalctl@ceph.mon.a.smithi016.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-28T22:00:32.460 INFO:journalctl@ceph.mon.a.smithi016.stdout: Block cache BinnedLRUCache@0x55c1d4e71090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-28T22:00:32.460 INFO:journalctl@ceph.mon.a.smithi016.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-28T22:00:32.461 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-03-28T22:00:32.461 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-28T22:00:32.461 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: starting mon.a rank 0 at public addrs [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] at bind addrs [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 798d9884-ed4e-11ee-95d0-87774f69a715 2024-03-28T22:00:32.461 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: mon.a@-1(???) e1 preinit fsid 798d9884-ed4e-11ee-95d0-87774f69a715 2024-03-28T22:00:32.461 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: mon.a@-1(???).mds e1 new map 2024-03-28T22:00:32.461 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: mon.a@-1(???).mds e1 print_map 2024-03-28T22:00:32.461 INFO:journalctl@ceph.mon.a.smithi016.stdout: e1 2024-03-28T22:00:32.461 INFO:journalctl@ceph.mon.a.smithi016.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-28T22:00:32.461 INFO:journalctl@ceph.mon.a.smithi016.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-28T22:00:32.461 INFO:journalctl@ceph.mon.a.smithi016.stdout: legacy client fscid: -1 2024-03-28T22:00:32.461 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-03-28T22:00:32.461 INFO:journalctl@ceph.mon.a.smithi016.stdout: No filesystems configured 2024-03-28T22:00:32.461 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-28T22:00:32.461 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T22:00:32.461 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T22:00:32.461 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T22:00:32.462 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-28T22:00:32.462 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 bash[32355]: 89528bfea73262e13c40e4724f4059497f435302350965eed236cd717fbd566c 2024-03-28T22:00:32.462 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 systemd[1]: Started Ceph mon.a for 798d9884-ed4e-11ee-95d0-87774f69a715. 2024-03-28T22:00:32.462 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-28T22:00:32.462 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: monmap epoch 1 2024-03-28T22:00:32.462 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: fsid 798d9884-ed4e-11ee-95d0-87774f69a715 2024-03-28T22:00:32.462 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: last_changed 2024-03-28T22:00:24.266479+0000 2024-03-28T22:00:32.462 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: created 2024-03-28T22:00:24.266479+0000 2024-03-28T22:00:32.462 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: min_mon_release 19 (squid) 2024-03-28T22:00:32.462 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: election_strategy: 1 2024-03-28T22:00:32.462 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-03-28T22:00:32.462 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: fsmap 2024-03-28T22:00:32.462 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: osdmap e1: 0 total, 0 up, 0 in 2024-03-28T22:00:32.463 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:32 smithi016 ceph-mon[32374]: mgrmap e1: no daemons active 2024-03-28T22:00:33.283 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.648+0000 7f783b25f700 1 Processor -- start 2024-03-28T22:00:33.289 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.648+0000 7f783b25f700 1 -- start start 2024-03-28T22:00:33.289 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.648+0000 7f783b25f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7834107f20 0x7f7834108340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:00:33.289 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.648+0000 7f783b25f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7834108880 con 0x7f7834107f20 2024-03-28T22:00:33.289 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.649+0000 7f7838ffb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7834107f20 0x7f7834108340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:00:33.289 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.649+0000 7f7838ffb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7834107f20 0x7f7834108340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57182/0 (socket says 172.21.15.16:57182) 2024-03-28T22:00:33.289 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.649+0000 7f7838ffb700 1 -- 172.21.15.16:0/817535113 learned_addr learned my addr 172.21.15.16:0/817535113 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:00:33.289 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.649+0000 7f7838ffb700 1 -- 172.21.15.16:0/817535113 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7834108fe0 con 0x7f7834107f20 2024-03-28T22:00:33.289 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.649+0000 7f7838ffb700 1 --2- 172.21.15.16:0/817535113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7834107f20 0x7f7834108340 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f78280098a0 tx=0x7f7828009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f788b6648302e40 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:00:33.289 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.649+0000 7f78337fe700 1 -- 172.21.15.16:0/817535113 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7828004160 con 0x7f7834107f20 2024-03-28T22:00:33.289 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.649+0000 7f78337fe700 1 -- 172.21.15.16:0/817535113 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7828004300 con 0x7f7834107f20 2024-03-28T22:00:33.289 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.649+0000 7f78337fe700 1 -- 172.21.15.16:0/817535113 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7828011b50 con 0x7f7834107f20 2024-03-28T22:00:33.290 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.650+0000 7f783b25f700 1 -- 172.21.15.16:0/817535113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7834107f20 msgr2=0x7f7834108340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:00:33.290 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.650+0000 7f783b25f700 1 --2- 172.21.15.16:0/817535113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7834107f20 0x7f7834108340 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f78280098a0 tx=0x7f7828009bb0 comp rx=0 tx=0).stop 2024-03-28T22:00:33.290 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.650+0000 7f783b25f700 1 -- 172.21.15.16:0/817535113 shutdown_connections 2024-03-28T22:00:33.290 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.650+0000 7f783b25f700 1 --2- 172.21.15.16:0/817535113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7834107f20 0x7f7834108340 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:00:33.290 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.650+0000 7f783b25f700 1 -- 172.21.15.16:0/817535113 >> 172.21.15.16:0/817535113 conn(0x7f78341034a0 msgr2=0x7f7834105900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:00:33.290 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.650+0000 7f783b25f700 1 -- 172.21.15.16:0/817535113 shutdown_connections 2024-03-28T22:00:33.290 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.650+0000 7f783b25f700 1 -- 172.21.15.16:0/817535113 wait complete. 2024-03-28T22:00:33.290 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.651+0000 7f783b25f700 1 Processor -- start 2024-03-28T22:00:33.290 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.651+0000 7f783b25f700 1 -- start start 2024-03-28T22:00:33.290 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.651+0000 7f783b25f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7834107f20 0x7f783407db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:00:33.290 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.651+0000 7f783b25f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f783410c420 con 0x7f7834107f20 2024-03-28T22:00:33.290 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.651+0000 7f7838ffb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7834107f20 0x7f783407db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:00:33.290 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.651+0000 7f7838ffb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7834107f20 0x7f783407db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57188/0 (socket says 172.21.15.16:57188) 2024-03-28T22:00:33.291 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.651+0000 7f7838ffb700 1 -- 172.21.15.16:0/4097537696 learned_addr learned my addr 172.21.15.16:0/4097537696 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:00:33.291 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.651+0000 7f7838ffb700 1 -- 172.21.15.16:0/4097537696 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f783407e090 con 0x7f7834107f20 2024-03-28T22:00:33.291 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.651+0000 7f7838ffb700 1 --2- 172.21.15.16:0/4097537696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7834107f20 0x7f783407db50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7828011b60 tx=0x7f7828011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:00:33.291 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.651+0000 7f7831ffb700 1 -- 172.21.15.16:0/4097537696 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7828003b20 con 0x7f7834107f20 2024-03-28T22:00:33.291 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.652+0000 7f783b25f700 1 -- 172.21.15.16:0/4097537696 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f783407c280 con 0x7f7834107f20 2024-03-28T22:00:33.291 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.652+0000 7f7831ffb700 1 -- 172.21.15.16:0/4097537696 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7828003cc0 con 0x7f7834107f20 2024-03-28T22:00:33.291 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.652+0000 7f7831ffb700 1 -- 172.21.15.16:0/4097537696 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7828024840 con 0x7f7834107f20 2024-03-28T22:00:33.291 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.652+0000 7f783b25f700 1 -- 172.21.15.16:0/4097537696 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f783407c7c0 con 0x7f7834107f20 2024-03-28T22:00:33.291 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.652+0000 7f7831ffb700 1 -- 172.21.15.16:0/4097537696 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7828022020 con 0x7f7834107f20 2024-03-28T22:00:33.291 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.652+0000 7f7831ffb700 1 -- 172.21.15.16:0/4097537696 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7828011e00 con 0x7f7834107f20 2024-03-28T22:00:33.291 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.653+0000 7f783b25f700 1 -- 172.21.15.16:0/4097537696 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7834063870 con 0x7f7834107f20 2024-03-28T22:00:33.291 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.655+0000 7f7831ffb700 1 -- 172.21.15.16:0/4097537696 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f7828043dd0 con 0x7f7834107f20 2024-03-28T22:00:33.291 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.725+0000 7f783b25f700 1 -- 172.21.15.16:0/4097537696 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f783407cb00 con 0x7f7834107f20 2024-03-28T22:00:33.292 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.727+0000 7f7831ffb700 1 -- 172.21.15.16:0/4097537696 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f7828032030 con 0x7f7834107f20 2024-03-28T22:00:33.292 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.727+0000 7f783b25f700 1 -- 172.21.15.16:0/4097537696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7834107f20 msgr2=0x7f783407db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:00:33.292 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.727+0000 7f783b25f700 1 --2- 172.21.15.16:0/4097537696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7834107f20 0x7f783407db50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7828011b60 tx=0x7f7828011c40 comp rx=0 tx=0).stop 2024-03-28T22:00:33.292 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.728+0000 7f783b25f700 1 -- 172.21.15.16:0/4097537696 shutdown_connections 2024-03-28T22:00:33.292 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.728+0000 7f783b25f700 1 --2- 172.21.15.16:0/4097537696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7834107f20 0x7f783407db50 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:00:33.292 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.728+0000 7f783b25f700 1 -- 172.21.15.16:0/4097537696 >> 172.21.15.16:0/4097537696 conn(0x7f78341034a0 msgr2=0x7f7834103fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:00:33.292 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.728+0000 7f783b25f700 1 -- 172.21.15.16:0/4097537696 shutdown_connections 2024-03-28T22:00:33.292 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:32.728+0000 7f783b25f700 1 -- 172.21.15.16:0/4097537696 wait complete. 2024-03-28T22:00:33.292 INFO:teuthology.orchestra.run.smithi016.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-28T22:00:33.314 INFO:teuthology.orchestra.run.smithi016.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-28T22:00:33.314 INFO:teuthology.orchestra.run.smithi016.stdout:Creating mgr... 2024-03-28T22:00:33.314 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-28T22:00:33.315 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-28T22:00:33.570 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mgr.x 2024-03-28T22:00:33.570 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to reset failed state of unit ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mgr.x.service: Unit ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mgr.x.service not loaded. 2024-03-28T22:00:33.716 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-798d9884-ed4e-11ee-95d0-87774f69a715.target.wants/ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-798d9884-ed4e-11ee-95d0-87774f69a715@.service. 2024-03-28T22:00:33.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:33 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/4097537696' entity='client.admin' 2024-03-28T22:00:33.937 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:33 smithi016 systemd[1]: Starting Ceph mgr.x for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:00:34.194 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:33 smithi016 podman[32589]: 2024-03-28 22:00:33.940051395 +0000 UTC m=+0.105118304 container create ebef8c7b1dbd41105072d4cb6ef197b32ec610643cbee18ebaa6bf57aa591597 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, RELEASE=squid-323d23d, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True) 2024-03-28T22:00:34.194 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:33 smithi016 podman[32589]: 2024-03-28 22:00:33.852377443 +0000 UTC m=+0.017444336 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be 2024-03-28T22:00:34.194 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:34 smithi016 podman[32589]: 2024-03-28 22:00:34.125986086 +0000 UTC m=+0.291052959 container init ebef8c7b1dbd41105072d4cb6ef197b32ec610643cbee18ebaa6bf57aa591597 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-323d23d, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326) 2024-03-28T22:00:34.194 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:34 smithi016 podman[32589]: 2024-03-28 22:00:34.128330351 +0000 UTC m=+0.293397230 container start ebef8c7b1dbd41105072d4cb6ef197b32ec610643cbee18ebaa6bf57aa591597 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-323d23d, ceph=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T22:00:34.195 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:34 smithi016 bash[32589]: ebef8c7b1dbd41105072d4cb6ef197b32ec610643cbee18ebaa6bf57aa591597 2024-03-28T22:00:34.207 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T22:00:34.207 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-03-28T22:00:34.215 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T22:00:34.215 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-03-28T22:00:34.215 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-03-28T22:00:34.215 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to enable service . firewalld.service is not available 2024-03-28T22:00:34.224 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T22:00:34.224 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-03-28T22:00:34.232 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T22:00:34.232 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-03-28T22:00:34.232 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-03-28T22:00:34.232 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-28T22:00:34.232 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr to start... 2024-03-28T22:00:34.232 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr... 2024-03-28T22:00:34.464 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:34 smithi016 systemd[1]: Started Ceph mgr.x for 798d9884-ed4e-11ee-95d0-87774f69a715. 2024-03-28T22:00:34.753 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:34 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:34.699+0000 7f46127ea200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T22:00:35.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:34 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3658820558' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T22:00:35.186 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:34 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:34.809+0000 7f46127ea200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T22:00:35.273 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-03-28T22:00:35.273 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-03-28T22:00:35.274 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsid": "798d9884-ed4e-11ee-95d0-87774f69a715", 2024-03-28T22:00:35.274 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T22:00:35.274 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T22:00:35.274 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T22:00:35.274 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T22:00:35.274 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:00:35.274 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T22:00:35.274 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T22:00:35.274 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 0 2024-03-28T22:00:35.274 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-03-28T22:00:35.274 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T22:00:35.274 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T22:00:35.274 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-03-28T22:00:35.275 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-28T22:00:35.275 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T22:00:35.275 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T22:00:35.275 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T22:00:35.275 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T22:00:35.275 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:00:35.275 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T22:00:35.275 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T22:00:35.275 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T22:00:35.275 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T22:00:35.275 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T22:00:35.283 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T22:00:35.283 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T22:00:35.283 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T22:00:35.283 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:00:35.283 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T22:00:35.283 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T22:00:35.283 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T22:00:35.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T22:00:35.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T22:00:35.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T22:00:35.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T22:00:35.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T22:00:35.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T22:00:35.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:00:35.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T22:00:35.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T22:00:35.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T22:00:35.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T22:00:35.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:00:35.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T22:00:35.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-28T22:00:35.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T22:00:35.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T22:00:35.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T22:00:35.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T22:00:35.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T22:00:35.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-03-28T22:00:35.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T22:00:35.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:00:35.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T22:00:35.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T22:00:35.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T22:00:27.216568+0000", 2024-03-28T22:00:35.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T22:00:35.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:00:35.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T22:00:35.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-03-28T22:00:35.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.671+0000 7fbbb0282700 1 Processor -- start 2024-03-28T22:00:35.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.671+0000 7fbbb0282700 1 -- start start 2024-03-28T22:00:35.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.671+0000 7fbbb0282700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbba8106d80 0x7fbba81071a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:00:35.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.671+0000 7fbbb0282700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbba8107770 con 0x7fbba8106d80 2024-03-28T22:00:35.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.671+0000 7fbbae01e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbba8106d80 0x7fbba81071a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:00:35.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.672+0000 7fbbae01e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbba8106d80 0x7fbba81071a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57214/0 (socket says 172.21.15.16:57214) 2024-03-28T22:00:35.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.672+0000 7fbbae01e700 1 -- 172.21.15.16:0/3582793966 learned_addr learned my addr 172.21.15.16:0/3582793966 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:00:35.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.672+0000 7fbbae01e700 1 -- 172.21.15.16:0/3582793966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbba8107f30 con 0x7fbba8106d80 2024-03-28T22:00:35.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.672+0000 7fbbae01e700 1 --2- 172.21.15.16:0/3582793966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbba8106d80 0x7fbba81071a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fbba40098a0 tx=0x7fbba4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a06f8f0776729ec7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:00:35.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.672+0000 7fbbad01c700 1 -- 172.21.15.16:0/3582793966 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbba4004160 con 0x7fbba8106d80 2024-03-28T22:00:35.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.672+0000 7fbbad01c700 1 -- 172.21.15.16:0/3582793966 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbba4004300 con 0x7fbba8106d80 2024-03-28T22:00:35.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.672+0000 7fbbad01c700 1 -- 172.21.15.16:0/3582793966 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbba4011b50 con 0x7fbba8106d80 2024-03-28T22:00:35.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.673+0000 7fbbb0282700 1 -- 172.21.15.16:0/3582793966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbba8106d80 msgr2=0x7fbba81071a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:00:35.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.673+0000 7fbbb0282700 1 --2- 172.21.15.16:0/3582793966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbba8106d80 0x7fbba81071a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fbba40098a0 tx=0x7fbba4009bb0 comp rx=0 tx=0).stop 2024-03-28T22:00:35.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.673+0000 7fbbb0282700 1 -- 172.21.15.16:0/3582793966 shutdown_connections 2024-03-28T22:00:35.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.673+0000 7fbbb0282700 1 --2- 172.21.15.16:0/3582793966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbba8106d80 0x7fbba81071a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:00:35.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.673+0000 7fbbb0282700 1 -- 172.21.15.16:0/3582793966 >> 172.21.15.16:0/3582793966 conn(0x7fbba8102360 msgr2=0x7fbba8104780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:00:35.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.673+0000 7fbbb0282700 1 -- 172.21.15.16:0/3582793966 shutdown_connections 2024-03-28T22:00:35.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.673+0000 7fbbb0282700 1 -- 172.21.15.16:0/3582793966 wait complete. 2024-03-28T22:00:35.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.673+0000 7fbbb0282700 1 Processor -- start 2024-03-28T22:00:35.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.674+0000 7fbbb0282700 1 -- start start 2024-03-28T22:00:35.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.674+0000 7fbbb0282700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbba8106d80 0x7fbba807ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:00:35.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.674+0000 7fbbb0282700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbba810b2e0 con 0x7fbba8106d80 2024-03-28T22:00:35.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.674+0000 7fbbae01e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbba8106d80 0x7fbba807ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:00:35.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.674+0000 7fbbae01e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbba8106d80 0x7fbba807ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57218/0 (socket says 172.21.15.16:57218) 2024-03-28T22:00:35.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.674+0000 7fbbae01e700 1 -- 172.21.15.16:0/3658820558 learned_addr learned my addr 172.21.15.16:0/3658820558 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:00:35.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.674+0000 7fbbae01e700 1 -- 172.21.15.16:0/3658820558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbba819ebc0 con 0x7fbba8106d80 2024-03-28T22:00:35.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.674+0000 7fbbae01e700 1 --2- 172.21.15.16:0/3658820558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbba8106d80 0x7fbba807ba30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fbba4000c00 tx=0x7fbba4011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:00:35.288 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.674+0000 7fbb9f7fe700 1 -- 172.21.15.16:0/3658820558 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbba4003b20 con 0x7fbba8106d80 2024-03-28T22:00:35.288 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.674+0000 7fbb9f7fe700 1 -- 172.21.15.16:0/3658820558 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbba4003cc0 con 0x7fbba8106d80 2024-03-28T22:00:35.288 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.674+0000 7fbbb0282700 1 -- 172.21.15.16:0/3658820558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbba819ed90 con 0x7fbba8106d80 2024-03-28T22:00:35.288 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.674+0000 7fbb9f7fe700 1 -- 172.21.15.16:0/3658820558 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbba4024910 con 0x7fbba8106d80 2024-03-28T22:00:35.288 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.675+0000 7fbbb0282700 1 -- 172.21.15.16:0/3658820558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbba819f130 con 0x7fbba8106d80 2024-03-28T22:00:35.288 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.675+0000 7fbb9f7fe700 1 -- 172.21.15.16:0/3658820558 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fbba4022020 con 0x7fbba8106d80 2024-03-28T22:00:35.288 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.675+0000 7fbb9f7fe700 1 -- 172.21.15.16:0/3658820558 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fbba4011e00 con 0x7fbba8106d80 2024-03-28T22:00:35.288 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.675+0000 7fbbb0282700 1 -- 172.21.15.16:0/3658820558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbba8063870 con 0x7fbba8106d80 2024-03-28T22:00:35.288 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.678+0000 7fbb9f7fe700 1 -- 172.21.15.16:0/3658820558 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fbba4042dd0 con 0x7fbba8106d80 2024-03-28T22:00:35.288 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.751+0000 7fbbb0282700 1 -- 172.21.15.16:0/3658820558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fbba819f4f0 con 0x7fbba8106d80 2024-03-28T22:00:35.288 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.752+0000 7fbb9f7fe700 1 -- 172.21.15.16:0/3658820558 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fbba4031020 con 0x7fbba8106d80 2024-03-28T22:00:35.288 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.753+0000 7fbbb0282700 1 -- 172.21.15.16:0/3658820558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbba8106d80 msgr2=0x7fbba807ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:00:35.288 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.753+0000 7fbbb0282700 1 --2- 172.21.15.16:0/3658820558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbba8106d80 0x7fbba807ba30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fbba4000c00 tx=0x7fbba4011c10 comp rx=0 tx=0).stop 2024-03-28T22:00:35.289 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.753+0000 7fbbb0282700 1 -- 172.21.15.16:0/3658820558 shutdown_connections 2024-03-28T22:00:35.289 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.753+0000 7fbbb0282700 1 --2- 172.21.15.16:0/3658820558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbba8106d80 0x7fbba807ba30 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:00:35.289 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.753+0000 7fbbb0282700 1 -- 172.21.15.16:0/3658820558 >> 172.21.15.16:0/3658820558 conn(0x7fbba8102360 msgr2=0x7fbba8102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:00:35.289 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.753+0000 7fbbb0282700 1 -- 172.21.15.16:0/3658820558 shutdown_connections 2024-03-28T22:00:35.289 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:34.753+0000 7fbbb0282700 1 -- 172.21.15.16:0/3658820558 wait complete. 2024-03-28T22:00:35.289 INFO:teuthology.orchestra.run.smithi016.stdout:mgr not available, waiting (1/15)... 2024-03-28T22:00:35.576 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:35 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:35.478+0000 7f46127ea200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T22:00:35.576 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:35 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:35.574+0000 7f46127ea200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T22:00:35.899 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:35 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:35.716+0000 7f46127ea200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T22:00:35.899 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:35 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:35.813+0000 7f46127ea200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T22:00:35.899 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:35 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:35.897+0000 7f46127ea200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T22:00:36.436 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:36 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:36.164+0000 7f46127ea200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T22:00:36.436 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:36 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:36.276+0000 7f46127ea200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T22:00:36.436 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:36 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:36.366+0000 7f46127ea200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T22:00:36.936 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:36 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:36.462+0000 7f46127ea200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T22:00:36.936 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:36 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:36.639+0000 7f46127ea200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T22:00:38.172 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:37 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3462357081' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T22:00:38.172 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:37 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:37.690+0000 7f46127ea200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T22:00:38.172 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:37 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:37.830+0000 7f46127ea200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T22:00:38.172 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:38 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:38.170+0000 7f46127ea200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T22:00:38.431 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:38 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:38.262+0000 7f46127ea200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T22:00:38.444 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-03-28T22:00:38.444 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-03-28T22:00:38.444 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsid": "798d9884-ed4e-11ee-95d0-87774f69a715", 2024-03-28T22:00:38.444 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T22:00:38.444 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T22:00:38.444 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T22:00:38.444 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T22:00:38.444 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:00:38.444 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T22:00:38.444 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T22:00:38.444 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 0 2024-03-28T22:00:38.445 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-03-28T22:00:38.445 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T22:00:38.445 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T22:00:38.445 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-03-28T22:00:38.445 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-28T22:00:38.445 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T22:00:38.445 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T22:00:38.445 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T22:00:38.445 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T22:00:38.445 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:00:38.445 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T22:00:38.445 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T22:00:38.445 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T22:00:38.445 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T22:00:38.445 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T22:00:38.446 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T22:00:38.446 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T22:00:38.446 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T22:00:38.446 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:00:38.446 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T22:00:38.446 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T22:00:38.446 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T22:00:38.446 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T22:00:38.446 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T22:00:38.446 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T22:00:38.446 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T22:00:38.446 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T22:00:38.446 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T22:00:38.446 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:00:38.447 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T22:00:38.447 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T22:00:38.447 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T22:00:38.447 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T22:00:38.447 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:00:38.447 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T22:00:38.447 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-28T22:00:38.447 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T22:00:38.447 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T22:00:38.447 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T22:00:38.447 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T22:00:38.447 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T22:00:38.447 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-03-28T22:00:38.447 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T22:00:38.448 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:00:38.448 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T22:00:38.448 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T22:00:38.448 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T22:00:27.216568+0000", 2024-03-28T22:00:38.448 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T22:00:38.448 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:00:38.448 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T22:00:38.448 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-03-28T22:00:38.448 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.713+0000 7fabca5d0700 1 Processor -- start 2024-03-28T22:00:38.448 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.713+0000 7fabca5d0700 1 -- start start 2024-03-28T22:00:38.448 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.713+0000 7fabca5d0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabc4108920 0x7fabc410ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:00:38.448 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.713+0000 7fabca5d0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fabc407be50 con 0x7fabc4108920 2024-03-28T22:00:38.448 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.714+0000 7fabc3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabc4108920 0x7fabc410ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:00:38.448 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.714+0000 7fabc3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabc4108920 0x7fabc410ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57224/0 (socket says 172.21.15.16:57224) 2024-03-28T22:00:38.448 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.714+0000 7fabc3fff700 1 -- 172.21.15.16:0/1481232596 learned_addr learned my addr 172.21.15.16:0/1481232596 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:00:38.449 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.714+0000 7fabc3fff700 1 -- 172.21.15.16:0/1481232596 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabc410b250 con 0x7fabc4108920 2024-03-28T22:00:38.449 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.714+0000 7fabc3fff700 1 --2- 172.21.15.16:0/1481232596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabc4108920 0x7fabc410ad10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fabb40098a0 tx=0x7fabb4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=471d616f175b46a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:00:38.449 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.714+0000 7fabc2ffd700 1 -- 172.21.15.16:0/1481232596 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fabb4004160 con 0x7fabc4108920 2024-03-28T22:00:38.449 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.714+0000 7fabc2ffd700 1 -- 172.21.15.16:0/1481232596 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fabb4004300 con 0x7fabc4108920 2024-03-28T22:00:38.449 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.714+0000 7fabc2ffd700 1 -- 172.21.15.16:0/1481232596 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fabb4011b50 con 0x7fabc4108920 2024-03-28T22:00:38.449 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.715+0000 7fabca5d0700 1 -- 172.21.15.16:0/1481232596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabc4108920 msgr2=0x7fabc410ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:00:38.449 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.715+0000 7fabca5d0700 1 --2- 172.21.15.16:0/1481232596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabc4108920 0x7fabc410ad10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fabb40098a0 tx=0x7fabb4009bb0 comp rx=0 tx=0).stop 2024-03-28T22:00:38.449 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.715+0000 7fabca5d0700 1 -- 172.21.15.16:0/1481232596 shutdown_connections 2024-03-28T22:00:38.449 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.715+0000 7fabca5d0700 1 --2- 172.21.15.16:0/1481232596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabc4108920 0x7fabc410ad10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:00:38.449 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.715+0000 7fabca5d0700 1 -- 172.21.15.16:0/1481232596 >> 172.21.15.16:0/1481232596 conn(0x7fabc4102310 msgr2=0x7fabc4104750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:00:38.449 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.715+0000 7fabca5d0700 1 -- 172.21.15.16:0/1481232596 shutdown_connections 2024-03-28T22:00:38.449 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.715+0000 7fabca5d0700 1 -- 172.21.15.16:0/1481232596 wait complete. 2024-03-28T22:00:38.449 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.716+0000 7fabca5d0700 1 Processor -- start 2024-03-28T22:00:38.449 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.716+0000 7fabca5d0700 1 -- start start 2024-03-28T22:00:38.450 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.716+0000 7fabca5d0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabc4108920 0x7fabc419bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:00:38.450 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.716+0000 7fabca5d0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fabc410c270 con 0x7fabc4108920 2024-03-28T22:00:38.450 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.716+0000 7fabc3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabc4108920 0x7fabc419bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:00:38.450 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.716+0000 7fabc3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabc4108920 0x7fabc419bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57232/0 (socket says 172.21.15.16:57232) 2024-03-28T22:00:38.450 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.716+0000 7fabc3fff700 1 -- 172.21.15.16:0/3462357081 learned_addr learned my addr 172.21.15.16:0/3462357081 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:00:38.450 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.717+0000 7fabc3fff700 1 -- 172.21.15.16:0/3462357081 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabc419c300 con 0x7fabc4108920 2024-03-28T22:00:38.450 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.717+0000 7fabc3fff700 1 --2- 172.21.15.16:0/3462357081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabc4108920 0x7fabc419bdc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fabb4011b60 tx=0x7fabb4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:00:38.450 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.717+0000 7fabc17fa700 1 -- 172.21.15.16:0/3462357081 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fabb4003b20 con 0x7fabc4108920 2024-03-28T22:00:38.450 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.717+0000 7fabca5d0700 1 -- 172.21.15.16:0/3462357081 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabc419c590 con 0x7fabc4108920 2024-03-28T22:00:38.450 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.717+0000 7fabc17fa700 1 -- 172.21.15.16:0/3462357081 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fabb4003cc0 con 0x7fabc4108920 2024-03-28T22:00:38.450 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.717+0000 7fabc17fa700 1 -- 172.21.15.16:0/3462357081 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fabb4024840 con 0x7fabc4108920 2024-03-28T22:00:38.450 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.717+0000 7fabca5d0700 1 -- 172.21.15.16:0/3462357081 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabc419f280 con 0x7fabc4108920 2024-03-28T22:00:38.450 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.717+0000 7fabc17fa700 1 -- 172.21.15.16:0/3462357081 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fabb4022020 con 0x7fabc4108920 2024-03-28T22:00:38.451 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.717+0000 7fabc17fa700 1 -- 172.21.15.16:0/3462357081 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fabb4011e00 con 0x7fabc4108920 2024-03-28T22:00:38.451 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.718+0000 7fabca5d0700 1 -- 172.21.15.16:0/3462357081 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabc4063870 con 0x7fabc4108920 2024-03-28T22:00:38.451 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.720+0000 7fabc17fa700 1 -- 172.21.15.16:0/3462357081 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fabb4043dd0 con 0x7fabc4108920 2024-03-28T22:00:38.451 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.794+0000 7fabca5d0700 1 -- 172.21.15.16:0/3462357081 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fabc419f7d0 con 0x7fabc4108920 2024-03-28T22:00:38.451 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.794+0000 7fabc17fa700 1 -- 172.21.15.16:0/3462357081 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fabb4032020 con 0x7fabc4108920 2024-03-28T22:00:38.451 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.795+0000 7fabca5d0700 1 -- 172.21.15.16:0/3462357081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabc4108920 msgr2=0x7fabc419bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:00:38.451 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.795+0000 7fabca5d0700 1 --2- 172.21.15.16:0/3462357081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabc4108920 0x7fabc419bdc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fabb4011b60 tx=0x7fabb4011c40 comp rx=0 tx=0).stop 2024-03-28T22:00:38.451 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.795+0000 7fabca5d0700 1 -- 172.21.15.16:0/3462357081 shutdown_connections 2024-03-28T22:00:38.451 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.795+0000 7fabca5d0700 1 --2- 172.21.15.16:0/3462357081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabc4108920 0x7fabc419bdc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:00:38.451 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.795+0000 7fabca5d0700 1 -- 172.21.15.16:0/3462357081 >> 172.21.15.16:0/3462357081 conn(0x7fabc4102310 msgr2=0x7fabc4102e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:00:38.451 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.795+0000 7fabca5d0700 1 -- 172.21.15.16:0/3462357081 shutdown_connections 2024-03-28T22:00:38.451 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:37.795+0000 7fabca5d0700 1 -- 172.21.15.16:0/3462357081 wait complete. 2024-03-28T22:00:38.451 INFO:teuthology.orchestra.run.smithi016.stdout:mgr not available, waiting (2/15)... 2024-03-28T22:00:38.686 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:38 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:38.463+0000 7f46127ea200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T22:00:38.686 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:38 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:38.653+0000 7f46127ea200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T22:00:39.206 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:38 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:38.942+0000 7f46127ea200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T22:00:39.206 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:39 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:39.204+0000 7f46127ea200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T22:00:39.618 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:39 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:39.452+0000 7f46127ea200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T22:00:40.571 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:40 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:40.318+0000 7f46127ea200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T22:00:41.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:40 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3903640544' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T22:00:41.186 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:40 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:40.917+0000 7f46127ea200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T22:00:41.186 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:41 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:41.021+0000 7f46127ea200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T22:00:41.686 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:41 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:41.246+0000 7f46127ea200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T22:00:42.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:41 smithi016 ceph-mon[32374]: Activating manager daemon x 2024-03-28T22:00:42.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:41 smithi016 ceph-mon[32374]: mgrmap e2: x(active, starting, since 0.00318096s) 2024-03-28T22:00:42.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:41 smithi016 ceph-mon[32374]: from='mgr.14100 172.21.15.16:0/530553737' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T22:00:42.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:41 smithi016 ceph-mon[32374]: from='mgr.14100 172.21.15.16:0/530553737' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T22:00:42.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:41 smithi016 ceph-mon[32374]: from='mgr.14100 172.21.15.16:0/530553737' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T22:00:42.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:41 smithi016 ceph-mon[32374]: from='mgr.14100 172.21.15.16:0/530553737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:00:42.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:41 smithi016 ceph-mon[32374]: from='mgr.14100 172.21.15.16:0/530553737' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T22:00:42.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:41 smithi016 ceph-mon[32374]: Manager daemon x is now available 2024-03-28T22:00:42.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:41 smithi016 ceph-mon[32374]: from='mgr.14100 172.21.15.16:0/530553737' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T22:00:42.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:41 smithi016 ceph-mon[32374]: from='mgr.14100 172.21.15.16:0/530553737' entity='mgr.x' 2024-03-28T22:00:42.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:41 smithi016 ceph-mon[32374]: from='mgr.14100 172.21.15.16:0/530553737' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T22:00:42.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:41 smithi016 ceph-mon[32374]: from='mgr.14100 172.21.15.16:0/530553737' entity='mgr.x' 2024-03-28T22:00:42.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:41 smithi016 ceph-mon[32374]: from='mgr.14100 172.21.15.16:0/530553737' entity='mgr.x' 2024-03-28T22:00:43.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:43 smithi016 ceph-mon[32374]: mgrmap e3: x(active, since 1.00767s) 2024-03-28T22:00:44.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:44 smithi016 ceph-mon[32374]: mgrmap e4: x(active, since 2s) 2024-03-28T22:00:46.894 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-03-28T22:00:46.894 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-03-28T22:00:46.894 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsid": "798d9884-ed4e-11ee-95d0-87774f69a715", 2024-03-28T22:00:46.894 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T22:00:46.894 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T22:00:46.895 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T22:00:46.895 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T22:00:46.895 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:00:46.895 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T22:00:46.895 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T22:00:46.895 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 0 2024-03-28T22:00:46.895 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-03-28T22:00:46.895 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T22:00:46.895 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T22:00:46.895 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-03-28T22:00:46.895 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-28T22:00:46.895 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T22:00:46.895 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T22:00:46.895 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T22:00:46.895 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T22:00:46.896 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:00:46.896 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T22:00:46.896 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T22:00:46.896 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T22:00:46.896 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T22:00:46.896 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T22:00:46.904 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T22:00:46.904 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T22:00:46.904 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T22:00:46.904 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:00:46.904 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T22:00:46.904 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T22:00:46.904 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T22:00:46.904 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T22:00:46.905 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T22:00:46.905 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T22:00:46.905 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T22:00:46.905 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T22:00:46.905 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T22:00:46.905 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:00:46.905 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T22:00:46.905 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T22:00:46.905 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T22:00:46.905 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T22:00:46.905 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:00:46.905 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T22:00:46.905 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-28T22:00:46.905 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T22:00:46.905 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T22:00:46.906 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T22:00:46.906 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T22:00:46.906 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T22:00:46.906 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-03-28T22:00:46.906 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T22:00:46.906 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:00:46.906 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T22:00:46.906 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T22:00:46.906 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T22:00:27.216568+0000", 2024-03-28T22:00:46.906 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T22:00:46.906 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:00:46.906 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T22:00:46.906 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-03-28T22:00:46.907 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.850+0000 7f4d9c558700 1 Processor -- start 2024-03-28T22:00:46.907 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.850+0000 7f4d9c558700 1 -- start start 2024-03-28T22:00:46.907 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.850+0000 7f4d9c558700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d94108360 0x7f4d94108780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:00:46.907 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.850+0000 7f4d9c558700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4d94108d50 con 0x7f4d94108360 2024-03-28T22:00:46.907 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.850+0000 7f4d9a2f4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d94108360 0x7f4d94108780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:00:46.907 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.850+0000 7f4d9a2f4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d94108360 0x7f4d94108780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38132/0 (socket says 172.21.15.16:38132) 2024-03-28T22:00:46.907 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.850+0000 7f4d9a2f4700 1 -- 172.21.15.16:0/2490902818 learned_addr learned my addr 172.21.15.16:0/2490902818 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:00:46.907 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.851+0000 7f4d9a2f4700 1 -- 172.21.15.16:0/2490902818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d94108ed0 con 0x7f4d94108360 2024-03-28T22:00:46.907 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.851+0000 7f4d9a2f4700 1 --2- 172.21.15.16:0/2490902818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d94108360 0x7f4d94108780 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4d900098a0 tx=0x7f4d90009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e3356f9605a276f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:00:46.907 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.851+0000 7f4d992f2700 1 -- 172.21.15.16:0/2490902818 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d90004160 con 0x7f4d94108360 2024-03-28T22:00:46.907 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.851+0000 7f4d992f2700 1 -- 172.21.15.16:0/2490902818 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4d90004300 con 0x7f4d94108360 2024-03-28T22:00:46.907 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.851+0000 7f4d992f2700 1 -- 172.21.15.16:0/2490902818 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d90011b50 con 0x7f4d94108360 2024-03-28T22:00:46.907 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.852+0000 7f4d9c558700 1 -- 172.21.15.16:0/2490902818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d94108360 msgr2=0x7f4d94108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:00:46.907 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.852+0000 7f4d9c558700 1 --2- 172.21.15.16:0/2490902818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d94108360 0x7f4d94108780 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4d900098a0 tx=0x7f4d90009bb0 comp rx=0 tx=0).stop 2024-03-28T22:00:46.907 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.852+0000 7f4d9c558700 1 -- 172.21.15.16:0/2490902818 shutdown_connections 2024-03-28T22:00:46.907 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.852+0000 7f4d9c558700 1 --2- 172.21.15.16:0/2490902818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d94108360 0x7f4d94108780 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:00:46.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.852+0000 7f4d9c558700 1 -- 172.21.15.16:0/2490902818 >> 172.21.15.16:0/2490902818 conn(0x7f4d94101ad0 msgr2=0x7f4d94103ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:00:46.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.852+0000 7f4d9c558700 1 -- 172.21.15.16:0/2490902818 shutdown_connections 2024-03-28T22:00:46.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.852+0000 7f4d9c558700 1 -- 172.21.15.16:0/2490902818 wait complete. 2024-03-28T22:00:46.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.853+0000 7f4d9c558700 1 Processor -- start 2024-03-28T22:00:46.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.853+0000 7f4d9c558700 1 -- start start 2024-03-28T22:00:46.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.853+0000 7f4d9c558700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d94108360 0x7f4d9419e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:00:46.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.853+0000 7f4d9c558700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4d94106c60 con 0x7f4d94108360 2024-03-28T22:00:46.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.853+0000 7f4d9a2f4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d94108360 0x7f4d9419e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:00:46.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.853+0000 7f4d9a2f4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d94108360 0x7f4d9419e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38134/0 (socket says 172.21.15.16:38134) 2024-03-28T22:00:46.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.853+0000 7f4d9a2f4700 1 -- 172.21.15.16:0/3903640544 learned_addr learned my addr 172.21.15.16:0/3903640544 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:00:46.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.853+0000 7f4d9a2f4700 1 -- 172.21.15.16:0/3903640544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d9419e560 con 0x7f4d94108360 2024-03-28T22:00:46.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.854+0000 7f4d9a2f4700 1 --2- 172.21.15.16:0/3903640544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d94108360 0x7f4d9419e020 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4d90011670 tx=0x7f4d90011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:00:46.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.854+0000 7f4d8b7fe700 1 -- 172.21.15.16:0/3903640544 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d900039c0 con 0x7f4d94108360 2024-03-28T22:00:46.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.854+0000 7f4d8b7fe700 1 -- 172.21.15.16:0/3903640544 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4d90011e50 con 0x7f4d94108360 2024-03-28T22:00:46.908 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.854+0000 7f4d9c558700 1 -- 172.21.15.16:0/3903640544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d9419e850 con 0x7f4d94108360 2024-03-28T22:00:46.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.854+0000 7f4d8b7fe700 1 -- 172.21.15.16:0/3903640544 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d9001a5c0 con 0x7f4d94108360 2024-03-28T22:00:46.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.854+0000 7f4d9c558700 1 -- 172.21.15.16:0/3903640544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d941a1490 con 0x7f4d94108360 2024-03-28T22:00:46.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.854+0000 7f4d8b7fe700 1 -- 172.21.15.16:0/3903640544 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4d9001ac50 con 0x7f4d94108360 2024-03-28T22:00:46.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.854+0000 7f4d8b7fe700 1 -- 172.21.15.16:0/3903640544 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4d9000b720 con 0x7f4d94108360 2024-03-28T22:00:46.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.855+0000 7f4d9c558700 1 -- 172.21.15.16:0/3903640544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d94063870 con 0x7f4d94108360 2024-03-28T22:00:46.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.859+0000 7f4d8b7fe700 1 -- 172.21.15.16:0/3903640544 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f4d90003b60 con 0x7f4d94108360 2024-03-28T22:00:46.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.932+0000 7f4d9c558700 1 -- 172.21.15.16:0/3903640544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f4d94197580 con 0x7f4d94108360 2024-03-28T22:00:46.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.933+0000 7f4d8b7fe700 1 -- 172.21.15.16:0/3903640544 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f4d9000b9c0 con 0x7f4d94108360 2024-03-28T22:00:46.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.933+0000 7f4d9c558700 1 -- 172.21.15.16:0/3903640544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d94108360 msgr2=0x7f4d9419e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:00:46.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.933+0000 7f4d9c558700 1 --2- 172.21.15.16:0/3903640544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d94108360 0x7f4d9419e020 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4d90011670 tx=0x7f4d90011b60 comp rx=0 tx=0).stop 2024-03-28T22:00:46.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.934+0000 7f4d9c558700 1 -- 172.21.15.16:0/3903640544 shutdown_connections 2024-03-28T22:00:46.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.934+0000 7f4d9c558700 1 --2- 172.21.15.16:0/3903640544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d94108360 0x7f4d9419e020 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:00:46.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.934+0000 7f4d9c558700 1 -- 172.21.15.16:0/3903640544 >> 172.21.15.16:0/3903640544 conn(0x7f4d94101ad0 msgr2=0x7f4d94194640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:00:46.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.934+0000 7f4d9c558700 1 -- 172.21.15.16:0/3903640544 shutdown_connections 2024-03-28T22:00:46.909 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:40.934+0000 7f4d9c558700 1 -- 172.21.15.16:0/3903640544 wait complete. 2024-03-28T22:00:46.909 INFO:teuthology.orchestra.run.smithi016.stdout:mgr not available, waiting (3/15)... 2024-03-28T22:00:53.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:52 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2518395973' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T22:00:53.460 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-03-28T22:00:53.460 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-03-28T22:00:53.460 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsid": "798d9884-ed4e-11ee-95d0-87774f69a715", 2024-03-28T22:00:53.460 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T22:00:53.460 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T22:00:53.460 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T22:00:53.460 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T22:00:53.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:00:53.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T22:00:53.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T22:00:53.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 0 2024-03-28T22:00:53.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-03-28T22:00:53.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T22:00:53.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T22:00:53.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-03-28T22:00:53.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_age": 20, 2024-03-28T22:00:53.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T22:00:53.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T22:00:53.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T22:00:53.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T22:00:53.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:00:53.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T22:00:53.462 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T22:00:53.462 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T22:00:53.462 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T22:00:53.462 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T22:00:53.470 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T22:00:53.470 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T22:00:53.470 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T22:00:53.471 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:00:53.471 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T22:00:53.471 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T22:00:53.471 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T22:00:53.471 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T22:00:53.471 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T22:00:53.471 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T22:00:53.471 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T22:00:53.471 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T22:00:53.471 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T22:00:53.471 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:00:53.471 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T22:00:53.471 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T22:00:53.471 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T22:00:53.471 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T22:00:53.472 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:00:53.472 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T22:00:53.472 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-28T22:00:53.472 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T22:00:53.472 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T22:00:53.472 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T22:00:53.472 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T22:00:53.472 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T22:00:53.472 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-03-28T22:00:53.472 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T22:00:53.472 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:00:53.472 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T22:00:53.472 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T22:00:53.473 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T22:00:27.216568+0000", 2024-03-28T22:00:53.473 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T22:00:53.473 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:00:53.473 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T22:00:53.473 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-03-28T22:00:53.473 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.459+0000 7fbf01a90700 1 Processor -- start 2024-03-28T22:00:53.473 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.459+0000 7fbf01a90700 1 -- start start 2024-03-28T22:00:53.473 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.459+0000 7fbf01a90700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbefc108940 0x7fbefc10ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:00:53.473 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.459+0000 7fbf01a90700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbefc07be30 con 0x7fbefc108940 2024-03-28T22:00:53.473 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.460+0000 7fbefb7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbefc108940 0x7fbefc10ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:00:53.473 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.460+0000 7fbefb7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbefc108940 0x7fbefc10ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59768/0 (socket says 172.21.15.16:59768) 2024-03-28T22:00:53.473 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.460+0000 7fbefb7fe700 1 -- 172.21.15.16:0/3968210610 learned_addr learned my addr 172.21.15.16:0/3968210610 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:00:53.474 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.460+0000 7fbefb7fe700 1 -- 172.21.15.16:0/3968210610 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbefc10b270 con 0x7fbefc108940 2024-03-28T22:00:53.474 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.460+0000 7fbefb7fe700 1 --2- 172.21.15.16:0/3968210610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbefc108940 0x7fbefc10ad30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbeec009e80 tx=0x7fbeec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=933f0d81d425d029 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:00:53.474 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.460+0000 7fbefa7fc700 1 -- 172.21.15.16:0/3968210610 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbeec017070 con 0x7fbefc108940 2024-03-28T22:00:53.474 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.460+0000 7fbefa7fc700 1 -- 172.21.15.16:0/3968210610 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbeec00b8c0 con 0x7fbefc108940 2024-03-28T22:00:53.474 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.460+0000 7fbefa7fc700 1 -- 172.21.15.16:0/3968210610 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbeec0045c0 con 0x7fbefc108940 2024-03-28T22:00:53.474 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.461+0000 7fbf01a90700 1 -- 172.21.15.16:0/3968210610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbefc108940 msgr2=0x7fbefc10ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:00:53.474 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.461+0000 7fbf01a90700 1 --2- 172.21.15.16:0/3968210610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbefc108940 0x7fbefc10ad30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbeec009e80 tx=0x7fbeec00b330 comp rx=0 tx=0).stop 2024-03-28T22:00:53.474 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.461+0000 7fbf01a90700 1 -- 172.21.15.16:0/3968210610 shutdown_connections 2024-03-28T22:00:53.474 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.461+0000 7fbf01a90700 1 --2- 172.21.15.16:0/3968210610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbefc108940 0x7fbefc10ad30 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:00:53.474 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.461+0000 7fbf01a90700 1 -- 172.21.15.16:0/3968210610 >> 172.21.15.16:0/3968210610 conn(0x7fbefc102310 msgr2=0x7fbefc104750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:00:53.474 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.461+0000 7fbf01a90700 1 -- 172.21.15.16:0/3968210610 shutdown_connections 2024-03-28T22:00:53.474 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.461+0000 7fbf01a90700 1 -- 172.21.15.16:0/3968210610 wait complete. 2024-03-28T22:00:53.474 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.462+0000 7fbf01a90700 1 Processor -- start 2024-03-28T22:00:53.474 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.462+0000 7fbf01a90700 1 -- start start 2024-03-28T22:00:53.475 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.462+0000 7fbf01a90700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbefc108940 0x7fbefc19bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:00:53.475 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.462+0000 7fbf01a90700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbefc10c240 con 0x7fbefc108940 2024-03-28T22:00:53.475 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.462+0000 7fbefb7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbefc108940 0x7fbefc19bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:00:53.475 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.462+0000 7fbefb7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbefc108940 0x7fbefc19bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59780/0 (socket says 172.21.15.16:59780) 2024-03-28T22:00:53.475 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.462+0000 7fbefb7fe700 1 -- 172.21.15.16:0/2518395973 learned_addr learned my addr 172.21.15.16:0/2518395973 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:00:53.475 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.462+0000 7fbefb7fe700 1 -- 172.21.15.16:0/2518395973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbefc19c2a0 con 0x7fbefc108940 2024-03-28T22:00:53.475 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.463+0000 7fbefb7fe700 1 --2- 172.21.15.16:0/2518395973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbefc108940 0x7fbefc19bd60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbeec004350 tx=0x7fbeec00bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:00:53.475 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.463+0000 7fbef8ff9700 1 -- 172.21.15.16:0/2518395973 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbeec020020 con 0x7fbefc108940 2024-03-28T22:00:53.475 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.463+0000 7fbef8ff9700 1 -- 172.21.15.16:0/2518395973 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbeec003c10 con 0x7fbefc108940 2024-03-28T22:00:53.475 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.463+0000 7fbf01a90700 1 -- 172.21.15.16:0/2518395973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbefc19c530 con 0x7fbefc108940 2024-03-28T22:00:53.475 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.463+0000 7fbef8ff9700 1 -- 172.21.15.16:0/2518395973 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbeec017040 con 0x7fbefc108940 2024-03-28T22:00:53.475 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.463+0000 7fbf01a90700 1 -- 172.21.15.16:0/2518395973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbefc19f220 con 0x7fbefc108940 2024-03-28T22:00:53.475 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.464+0000 7fbef8ff9700 1 -- 172.21.15.16:0/2518395973 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 4) ==== 45917+0+0 (secure 0 0 0) 0x7fbeec023020 con 0x7fbefc108940 2024-03-28T22:00:53.475 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.464+0000 7fbf01a90700 1 -- 172.21.15.16:0/2518395973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbefc063870 con 0x7fbefc108940 2024-03-28T22:00:53.475 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.464+0000 7fbef8ff9700 1 --2- 172.21.15.16:0/2518395973 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7fbee8038b50 0x7fbee803b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:00:53.476 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.464+0000 7fbef8ff9700 1 -- 172.21.15.16:0/2518395973 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fbeec00f050 con 0x7fbefc108940 2024-03-28T22:00:53.476 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.464+0000 7fbefaffd700 1 --2- 172.21.15.16:0/2518395973 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7fbee8038b50 0x7fbee803b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:00:53.476 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.465+0000 7fbefaffd700 1 --2- 172.21.15.16:0/2518395973 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7fbee8038b50 0x7fbee803b010 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fbee4006fd0 tx=0x7fbee4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:00:53.476 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.469+0000 7fbef8ff9700 1 -- 172.21.15.16:0/2518395973 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fbeec02a020 con 0x7fbefc108940 2024-03-28T22:00:53.476 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.753+0000 7fbf01a90700 1 -- 172.21.15.16:0/2518395973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fbefc19f910 con 0x7fbefc108940 2024-03-28T22:00:53.476 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.753+0000 7fbef8ff9700 1 -- 172.21.15.16:0/2518395973 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fbeec01b530 con 0x7fbefc108940 2024-03-28T22:00:53.476 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.756+0000 7fbf01a90700 1 -- 172.21.15.16:0/2518395973 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7fbee8038b50 msgr2=0x7fbee803b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:00:53.476 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.756+0000 7fbf01a90700 1 --2- 172.21.15.16:0/2518395973 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7fbee8038b50 0x7fbee803b010 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fbee4006fd0 tx=0x7fbee4006e40 comp rx=0 tx=0).stop 2024-03-28T22:00:53.476 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.756+0000 7fbf01a90700 1 -- 172.21.15.16:0/2518395973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbefc108940 msgr2=0x7fbefc19bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:00:53.476 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.756+0000 7fbf01a90700 1 --2- 172.21.15.16:0/2518395973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbefc108940 0x7fbefc19bd60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbeec004350 tx=0x7fbeec00bd50 comp rx=0 tx=0).stop 2024-03-28T22:00:53.476 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.756+0000 7fbf01a90700 1 -- 172.21.15.16:0/2518395973 shutdown_connections 2024-03-28T22:00:53.476 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.756+0000 7fbf01a90700 1 --2- 172.21.15.16:0/2518395973 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7fbee8038b50 0x7fbee803b010 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:00:53.476 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.756+0000 7fbf01a90700 1 --2- 172.21.15.16:0/2518395973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbefc108940 0x7fbefc19bd60 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:00:53.476 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.756+0000 7fbf01a90700 1 -- 172.21.15.16:0/2518395973 >> 172.21.15.16:0/2518395973 conn(0x7fbefc102310 msgr2=0x7fbefc102dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:00:53.476 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.756+0000 7fbf01a90700 1 -- 172.21.15.16:0/2518395973 shutdown_connections 2024-03-28T22:00:53.476 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:52.756+0000 7fbf01a90700 1 -- 172.21.15.16:0/2518395973 wait complete. 2024-03-28T22:00:53.477 INFO:teuthology.orchestra.run.smithi016.stdout:mgr is available 2024-03-28T22:00:54.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:54 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/567740002' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-28T22:00:54.738 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-03-28T22:00:54.738 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [global] 2024-03-28T22:00:54.738 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout fsid = 798d9884-ed4e-11ee-95d0-87774f69a715 2024-03-28T22:00:54.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-28T22:00:54.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-28T22:00:54.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-28T22:00:54.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-28T22:00:54.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-03-28T22:00:54.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-28T22:00:54.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-28T22:00:54.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-03-28T22:00:54.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [osd] 2024-03-28T22:00:54.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-28T22:00:54.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-28T22:00:54.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-28T22:00:54.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-28T22:00:54.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-28T22:00:54.740 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.870+0000 7f9482e5f700 1 Processor -- start 2024-03-28T22:00:54.740 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.870+0000 7f9482e5f700 1 -- start start 2024-03-28T22:00:54.740 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.870+0000 7f9482e5f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f947c106d80 0x7f947c1071a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:00:54.740 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.870+0000 7f9482e5f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f947c107770 con 0x7f947c106d80 2024-03-28T22:00:54.740 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.870+0000 7f9480bfb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f947c106d80 0x7f947c1071a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:00:54.740 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.870+0000 7f9480bfb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f947c106d80 0x7f947c1071a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59796/0 (socket says 172.21.15.16:59796) 2024-03-28T22:00:54.740 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.870+0000 7f9480bfb700 1 -- 172.21.15.16:0/69046848 learned_addr learned my addr 172.21.15.16:0/69046848 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:00:54.740 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.870+0000 7f9480bfb700 1 -- 172.21.15.16:0/69046848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f947c107f30 con 0x7f947c106d80 2024-03-28T22:00:54.740 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.870+0000 7f9480bfb700 1 --2- 172.21.15.16:0/69046848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f947c106d80 0x7f947c1071a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f94700098a0 tx=0x7f9470009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fd5e7ce4974618f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:00:54.740 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.871+0000 7f947b7fe700 1 -- 172.21.15.16:0/69046848 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9470004160 con 0x7f947c106d80 2024-03-28T22:00:54.740 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.871+0000 7f947b7fe700 1 -- 172.21.15.16:0/69046848 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9470004300 con 0x7f947c106d80 2024-03-28T22:00:54.740 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.871+0000 7f947b7fe700 1 -- 172.21.15.16:0/69046848 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9470011b50 con 0x7f947c106d80 2024-03-28T22:00:54.740 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.871+0000 7f9482e5f700 1 -- 172.21.15.16:0/69046848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f947c106d80 msgr2=0x7f947c1071a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:00:54.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.871+0000 7f9482e5f700 1 --2- 172.21.15.16:0/69046848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f947c106d80 0x7f947c1071a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f94700098a0 tx=0x7f9470009bb0 comp rx=0 tx=0).stop 2024-03-28T22:00:54.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.871+0000 7f9482e5f700 1 -- 172.21.15.16:0/69046848 shutdown_connections 2024-03-28T22:00:54.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.871+0000 7f9482e5f700 1 --2- 172.21.15.16:0/69046848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f947c106d80 0x7f947c1071a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:00:54.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.871+0000 7f9482e5f700 1 -- 172.21.15.16:0/69046848 >> 172.21.15.16:0/69046848 conn(0x7f947c102360 msgr2=0x7f947c104780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:00:54.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.871+0000 7f9482e5f700 1 -- 172.21.15.16:0/69046848 shutdown_connections 2024-03-28T22:00:54.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.871+0000 7f9482e5f700 1 -- 172.21.15.16:0/69046848 wait complete. 2024-03-28T22:00:54.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.872+0000 7f9482e5f700 1 Processor -- start 2024-03-28T22:00:54.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.872+0000 7f9482e5f700 1 -- start start 2024-03-28T22:00:54.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.872+0000 7f9482e5f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f947c106d80 0x7f947c19bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:00:54.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.872+0000 7f9482e5f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f947c10b2e0 con 0x7f947c106d80 2024-03-28T22:00:54.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.872+0000 7f9480bfb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f947c106d80 0x7f947c19bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:00:54.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.872+0000 7f9480bfb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f947c106d80 0x7f947c19bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59804/0 (socket says 172.21.15.16:59804) 2024-03-28T22:00:54.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.872+0000 7f9480bfb700 1 -- 172.21.15.16:0/567740002 learned_addr learned my addr 172.21.15.16:0/567740002 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:00:54.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.873+0000 7f9480bfb700 1 -- 172.21.15.16:0/567740002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f947c19c3e0 con 0x7f947c106d80 2024-03-28T22:00:54.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.873+0000 7f9480bfb700 1 --2- 172.21.15.16:0/567740002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f947c106d80 0x7f947c19bea0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9470011670 tx=0x7f9470011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:00:54.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.873+0000 7f9479ffb700 1 -- 172.21.15.16:0/567740002 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94700039c0 con 0x7f947c106d80 2024-03-28T22:00:54.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.873+0000 7f9479ffb700 1 -- 172.21.15.16:0/567740002 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9470011e50 con 0x7f947c106d80 2024-03-28T22:00:54.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.873+0000 7f9479ffb700 1 -- 172.21.15.16:0/567740002 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f947001a5c0 con 0x7f947c106d80 2024-03-28T22:00:54.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.873+0000 7f9482e5f700 1 -- 172.21.15.16:0/567740002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f947c19c670 con 0x7f947c106d80 2024-03-28T22:00:54.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.873+0000 7f9482e5f700 1 -- 172.21.15.16:0/567740002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f947c19f360 con 0x7f947c106d80 2024-03-28T22:00:54.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.874+0000 7f9479ffb700 1 -- 172.21.15.16:0/567740002 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 4) ==== 45917+0+0 (secure 0 0 0) 0x7f9470024430 con 0x7f947c106d80 2024-03-28T22:00:54.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.874+0000 7f9479ffb700 1 --2- 172.21.15.16:0/567740002 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7f946c038b00 0x7f946c03afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:00:54.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.874+0000 7f9479ffb700 1 -- 172.21.15.16:0/567740002 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f947004cdc0 con 0x7f947c106d80 2024-03-28T22:00:54.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.874+0000 7f9482e5f700 1 -- 172.21.15.16:0/567740002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f947c063870 con 0x7f947c106d80 2024-03-28T22:00:54.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.874+0000 7f947bfff700 1 --2- 172.21.15.16:0/567740002 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7f946c038b00 0x7f946c03afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:00:54.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.875+0000 7f947bfff700 1 --2- 172.21.15.16:0/567740002 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7f946c038b00 0x7f946c03afc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9468006fd0 tx=0x7f9468006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:00:54.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:53.879+0000 7f9479ffb700 1 -- 172.21.15.16:0/567740002 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9470003b60 con 0x7f947c106d80 2024-03-28T22:00:54.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:54.083+0000 7f9482e5f700 1 -- 172.21.15.16:0/567740002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f947c02de30 con 0x7f947c106d80 2024-03-28T22:00:54.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:54.084+0000 7f9479ffb700 1 -- 172.21.15.16:0/567740002 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7f947004b310 con 0x7f947c106d80 2024-03-28T22:00:54.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:54.086+0000 7f9482e5f700 1 -- 172.21.15.16:0/567740002 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7f946c038b00 msgr2=0x7f946c03afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:00:54.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:54.086+0000 7f9482e5f700 1 --2- 172.21.15.16:0/567740002 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7f946c038b00 0x7f946c03afc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9468006fd0 tx=0x7f9468006e40 comp rx=0 tx=0).stop 2024-03-28T22:00:54.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:54.086+0000 7f9482e5f700 1 -- 172.21.15.16:0/567740002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f947c106d80 msgr2=0x7f947c19bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:00:54.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:54.086+0000 7f9482e5f700 1 --2- 172.21.15.16:0/567740002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f947c106d80 0x7f947c19bea0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9470011670 tx=0x7f9470011b60 comp rx=0 tx=0).stop 2024-03-28T22:00:54.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:54.086+0000 7f9482e5f700 1 -- 172.21.15.16:0/567740002 shutdown_connections 2024-03-28T22:00:54.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:54.086+0000 7f9482e5f700 1 --2- 172.21.15.16:0/567740002 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7f946c038b00 0x7f946c03afc0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:00:54.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:54.086+0000 7f9482e5f700 1 --2- 172.21.15.16:0/567740002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f947c106d80 0x7f947c19bea0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:00:54.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:54.086+0000 7f9482e5f700 1 -- 172.21.15.16:0/567740002 >> 172.21.15.16:0/567740002 conn(0x7f947c102360 msgr2=0x7f947c103260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:00:54.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:54.086+0000 7f9482e5f700 1 -- 172.21.15.16:0/567740002 shutdown_connections 2024-03-28T22:00:54.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:54.086+0000 7f9482e5f700 1 -- 172.21.15.16:0/567740002 wait complete. 2024-03-28T22:00:54.743 INFO:teuthology.orchestra.run.smithi016.stdout:Enabling cephadm module... 2024-03-28T22:00:55.808 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:55 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2725279089' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-28T22:00:55.809 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:55 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: ignoring --setuser ceph since I am not root 2024-03-28T22:00:55.809 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:55 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: ignoring --setgroup ceph since I am not root 2024-03-28T22:00:56.076 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:56 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:56.055+0000 7f5cbf71b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T22:00:56.084 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.163+0000 7f58bec43700 1 Processor -- start 2024-03-28T22:00:56.084 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.163+0000 7f58bec43700 1 -- start start 2024-03-28T22:00:56.084 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.163+0000 7f58bec43700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58b8108900 0x7f58b810acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:00:56.084 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.163+0000 7f58bec43700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f58b807be70 con 0x7f58b8108900 2024-03-28T22:00:56.084 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.163+0000 7f58bc9df700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58b8108900 0x7f58b810acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:00:56.084 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.163+0000 7f58bc9df700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58b8108900 0x7f58b810acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59820/0 (socket says 172.21.15.16:59820) 2024-03-28T22:00:56.084 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.163+0000 7f58bc9df700 1 -- 172.21.15.16:0/1490626898 learned_addr learned my addr 172.21.15.16:0/1490626898 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:00:56.084 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.163+0000 7f58bc9df700 1 -- 172.21.15.16:0/1490626898 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58b810b230 con 0x7f58b8108900 2024-03-28T22:00:56.084 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.164+0000 7f58bc9df700 1 --2- 172.21.15.16:0/1490626898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58b8108900 0x7f58b810acf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f58ac0098a0 tx=0x7f58ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8253f30b21a53368 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:00:56.084 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.164+0000 7f58b77fe700 1 -- 172.21.15.16:0/1490626898 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58ac004160 con 0x7f58b8108900 2024-03-28T22:00:56.084 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.164+0000 7f58b77fe700 1 -- 172.21.15.16:0/1490626898 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f58ac004300 con 0x7f58b8108900 2024-03-28T22:00:56.084 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.164+0000 7f58b77fe700 1 -- 172.21.15.16:0/1490626898 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58ac011b50 con 0x7f58b8108900 2024-03-28T22:00:56.084 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.164+0000 7f58bec43700 1 -- 172.21.15.16:0/1490626898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58b8108900 msgr2=0x7f58b810acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:00:56.085 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.164+0000 7f58bec43700 1 --2- 172.21.15.16:0/1490626898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58b8108900 0x7f58b810acf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f58ac0098a0 tx=0x7f58ac009bb0 comp rx=0 tx=0).stop 2024-03-28T22:00:56.085 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.164+0000 7f58bec43700 1 -- 172.21.15.16:0/1490626898 shutdown_connections 2024-03-28T22:00:56.085 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.164+0000 7f58bec43700 1 --2- 172.21.15.16:0/1490626898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58b8108900 0x7f58b810acf0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:00:56.085 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.164+0000 7f58bec43700 1 -- 172.21.15.16:0/1490626898 >> 172.21.15.16:0/1490626898 conn(0x7f58b8102310 msgr2=0x7f58b8104750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:00:56.085 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.164+0000 7f58bec43700 1 -- 172.21.15.16:0/1490626898 shutdown_connections 2024-03-28T22:00:56.085 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.164+0000 7f58bec43700 1 -- 172.21.15.16:0/1490626898 wait complete. 2024-03-28T22:00:56.085 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.165+0000 7f58bec43700 1 Processor -- start 2024-03-28T22:00:56.085 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.165+0000 7f58bec43700 1 -- start start 2024-03-28T22:00:56.085 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.165+0000 7f58bec43700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58b8108900 0x7f58b819bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:00:56.085 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.165+0000 7f58bec43700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f58b810c1e0 con 0x7f58b8108900 2024-03-28T22:00:56.085 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.165+0000 7f58bc9df700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58b8108900 0x7f58b819bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:00:56.085 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.165+0000 7f58bc9df700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58b8108900 0x7f58b819bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59830/0 (socket says 172.21.15.16:59830) 2024-03-28T22:00:56.085 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.165+0000 7f58bc9df700 1 -- 172.21.15.16:0/2725279089 learned_addr learned my addr 172.21.15.16:0/2725279089 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:00:56.085 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.166+0000 7f58bc9df700 1 -- 172.21.15.16:0/2725279089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58b819c2e0 con 0x7f58b8108900 2024-03-28T22:00:56.086 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.166+0000 7f58bc9df700 1 --2- 172.21.15.16:0/2725279089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58b8108900 0x7f58b819bda0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f58ac011670 tx=0x7f58ac011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:00:56.086 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.166+0000 7f58b5ffb700 1 -- 172.21.15.16:0/2725279089 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58ac0039c0 con 0x7f58b8108900 2024-03-28T22:00:56.086 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.166+0000 7f58b5ffb700 1 -- 172.21.15.16:0/2725279089 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f58ac011e50 con 0x7f58b8108900 2024-03-28T22:00:56.086 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.166+0000 7f58bec43700 1 -- 172.21.15.16:0/2725279089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58b819c5d0 con 0x7f58b8108900 2024-03-28T22:00:56.086 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.166+0000 7f58b5ffb700 1 -- 172.21.15.16:0/2725279089 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58ac01a5c0 con 0x7f58b8108900 2024-03-28T22:00:56.086 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.166+0000 7f58bec43700 1 -- 172.21.15.16:0/2725279089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58b819f220 con 0x7f58b8108900 2024-03-28T22:00:56.086 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.167+0000 7f58b5ffb700 1 -- 172.21.15.16:0/2725279089 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 4) ==== 45917+0+0 (secure 0 0 0) 0x7f58ac01a8c0 con 0x7f58b8108900 2024-03-28T22:00:56.086 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.167+0000 7f58bec43700 1 -- 172.21.15.16:0/2725279089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58b8063870 con 0x7f58b8108900 2024-03-28T22:00:56.086 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.167+0000 7f58b5ffb700 1 --2- 172.21.15.16:0/2725279089 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7f58a8038b00 0x7f58a803afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:00:56.086 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.167+0000 7f58b5ffb700 1 -- 172.21.15.16:0/2725279089 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f58ac04cda0 con 0x7f58b8108900 2024-03-28T22:00:56.086 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.168+0000 7f58b7fff700 1 --2- 172.21.15.16:0/2725279089 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7f58a8038b00 0x7f58a803afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:00:56.086 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.168+0000 7f58b7fff700 1 --2- 172.21.15.16:0/2725279089 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7f58a8038b00 0x7f58a803afc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f58a4006fd0 tx=0x7f58a4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:00:56.087 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.172+0000 7f58b5ffb700 1 -- 172.21.15.16:0/2725279089 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f58ac019c70 con 0x7f58b8108900 2024-03-28T22:00:56.087 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.438+0000 7f58bec43700 1 -- 172.21.15.16:0/2725279089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f58b819fa10 con 0x7f58b8108900 2024-03-28T22:00:56.087 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.490+0000 7f58b5ffb700 1 -- 172.21.15.16:0/2725279089 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f58ac04c7c0 con 0x7f58b8108900 2024-03-28T22:00:56.087 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.491+0000 7f58b5ffb700 1 -- 172.21.15.16:0/2725279089 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 5) ==== 45928+0+0 (secure 0 0 0) 0x7f58ac01ac60 con 0x7f58b8108900 2024-03-28T22:00:56.087 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.492+0000 7f58bec43700 1 -- 172.21.15.16:0/2725279089 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7f58a8038b00 msgr2=0x7f58a803afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:00:56.087 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.492+0000 7f58bec43700 1 --2- 172.21.15.16:0/2725279089 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7f58a8038b00 0x7f58a803afc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f58a4006fd0 tx=0x7f58a4006e40 comp rx=0 tx=0).stop 2024-03-28T22:00:56.087 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.492+0000 7f58bec43700 1 -- 172.21.15.16:0/2725279089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58b8108900 msgr2=0x7f58b819bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:00:56.087 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.492+0000 7f58bec43700 1 --2- 172.21.15.16:0/2725279089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58b8108900 0x7f58b819bda0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f58ac011670 tx=0x7f58ac011b60 comp rx=0 tx=0).stop 2024-03-28T22:00:56.087 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.493+0000 7f58bec43700 1 -- 172.21.15.16:0/2725279089 shutdown_connections 2024-03-28T22:00:56.087 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.493+0000 7f58bec43700 1 --2- 172.21.15.16:0/2725279089 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7f58a8038b00 0x7f58a803afc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:00:56.087 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.493+0000 7f58bec43700 1 --2- 172.21.15.16:0/2725279089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58b8108900 0x7f58b819bda0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:00:56.087 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.493+0000 7f58bec43700 1 -- 172.21.15.16:0/2725279089 >> 172.21.15.16:0/2725279089 conn(0x7f58b8102310 msgr2=0x7f58b8102e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:00:56.087 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.493+0000 7f58bec43700 1 -- 172.21.15.16:0/2725279089 shutdown_connections 2024-03-28T22:00:56.087 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:55.493+0000 7f58bec43700 1 -- 172.21.15.16:0/2725279089 wait complete. 2024-03-28T22:00:56.362 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:56 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:56.157+0000 7f5cbf71b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T22:00:56.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:56 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2725279089' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-28T22:00:56.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:56 smithi016 ceph-mon[32374]: mgrmap e5: x(active, since 14s) 2024-03-28T22:00:56.920 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:56 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:56.824+0000 7f5cbf71b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T22:00:56.920 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:56 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:56.917+0000 7f5cbf71b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T22:00:57.186 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:57 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:57.063+0000 7f5cbf71b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T22:00:57.186 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:57 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:57.160+0000 7f5cbf71b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T22:00:57.282 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-03-28T22:00:57.282 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-28T22:00:57.282 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-28T22:00:57.282 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-28T22:00:57.283 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-28T22:00:57.283 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-03-28T22:00:57.283 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.460+0000 7f0e9967e700 1 Processor -- start 2024-03-28T22:00:57.283 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.460+0000 7f0e9967e700 1 -- start start 2024-03-28T22:00:57.283 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.460+0000 7f0e9967e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e94106d80 0x7f0e941071a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:00:57.283 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.460+0000 7f0e9967e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0e94107770 con 0x7f0e94106d80 2024-03-28T22:00:57.283 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.460+0000 7f0e92ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e94106d80 0x7f0e941071a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:00:57.283 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.460+0000 7f0e92ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e94106d80 0x7f0e941071a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59866/0 (socket says 172.21.15.16:59866) 2024-03-28T22:00:57.283 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.460+0000 7f0e92ffd700 1 -- 172.21.15.16:0/3719012789 learned_addr learned my addr 172.21.15.16:0/3719012789 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:00:57.283 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.460+0000 7f0e92ffd700 1 -- 172.21.15.16:0/3719012789 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e94107f80 con 0x7f0e94106d80 2024-03-28T22:00:57.283 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.461+0000 7f0e92ffd700 1 --2- 172.21.15.16:0/3719012789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e94106d80 0x7f0e941071a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0e840098a0 tx=0x7f0e84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d16894f28c768ce3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:00:57.283 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.461+0000 7f0e91ffb700 1 -- 172.21.15.16:0/3719012789 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e84004160 con 0x7f0e94106d80 2024-03-28T22:00:57.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.461+0000 7f0e91ffb700 1 -- 172.21.15.16:0/3719012789 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0e84004300 con 0x7f0e94106d80 2024-03-28T22:00:57.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.461+0000 7f0e91ffb700 1 -- 172.21.15.16:0/3719012789 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e84011b50 con 0x7f0e94106d80 2024-03-28T22:00:57.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.461+0000 7f0e9967e700 1 -- 172.21.15.16:0/3719012789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e94106d80 msgr2=0x7f0e941071a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:00:57.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.461+0000 7f0e9967e700 1 --2- 172.21.15.16:0/3719012789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e94106d80 0x7f0e941071a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0e840098a0 tx=0x7f0e84009bb0 comp rx=0 tx=0).stop 2024-03-28T22:00:57.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.462+0000 7f0e9967e700 1 -- 172.21.15.16:0/3719012789 shutdown_connections 2024-03-28T22:00:57.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.462+0000 7f0e9967e700 1 --2- 172.21.15.16:0/3719012789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e94106d80 0x7f0e941071a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:00:57.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.462+0000 7f0e9967e700 1 -- 172.21.15.16:0/3719012789 >> 172.21.15.16:0/3719012789 conn(0x7f0e94102340 msgr2=0x7f0e94104760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:00:57.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.462+0000 7f0e9967e700 1 -- 172.21.15.16:0/3719012789 shutdown_connections 2024-03-28T22:00:57.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.462+0000 7f0e9967e700 1 -- 172.21.15.16:0/3719012789 wait complete. 2024-03-28T22:00:57.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.462+0000 7f0e9967e700 1 Processor -- start 2024-03-28T22:00:57.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.462+0000 7f0e9967e700 1 -- start start 2024-03-28T22:00:57.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.462+0000 7f0e9967e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e94106d80 0x7f0e94191f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:00:57.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.463+0000 7f0e9967e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0e9410b330 con 0x7f0e94106d80 2024-03-28T22:00:57.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.463+0000 7f0e92ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e94106d80 0x7f0e94191f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:00:57.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.463+0000 7f0e92ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e94106d80 0x7f0e94191f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59876/0 (socket says 172.21.15.16:59876) 2024-03-28T22:00:57.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.463+0000 7f0e92ffd700 1 -- 172.21.15.16:0/2086883384 learned_addr learned my addr 172.21.15.16:0/2086883384 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:00:57.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.463+0000 7f0e92ffd700 1 -- 172.21.15.16:0/2086883384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e94192470 con 0x7f0e94106d80 2024-03-28T22:00:57.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.463+0000 7f0e92ffd700 1 --2- 172.21.15.16:0/2086883384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e94106d80 0x7f0e94191f30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0e84011670 tx=0x7f0e84011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:00:57.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.463+0000 7f0e7bfff700 1 -- 172.21.15.16:0/2086883384 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e840039c0 con 0x7f0e94106d80 2024-03-28T22:00:57.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.463+0000 7f0e9967e700 1 -- 172.21.15.16:0/2086883384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e94190660 con 0x7f0e94106d80 2024-03-28T22:00:57.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.463+0000 7f0e7bfff700 1 -- 172.21.15.16:0/2086883384 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0e84011e50 con 0x7f0e94106d80 2024-03-28T22:00:57.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.463+0000 7f0e7bfff700 1 -- 172.21.15.16:0/2086883384 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e8401a5c0 con 0x7f0e94106d80 2024-03-28T22:00:57.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.463+0000 7f0e9967e700 1 -- 172.21.15.16:0/2086883384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e94190ba0 con 0x7f0e94106d80 2024-03-28T22:00:57.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.464+0000 7f0e7bfff700 1 -- 172.21.15.16:0/2086883384 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 45928+0+0 (secure 0 0 0) 0x7f0e8401a8c0 con 0x7f0e94106d80 2024-03-28T22:00:57.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.464+0000 7f0e7bfff700 1 --2- 172.21.15.16:0/2086883384 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7f0e7c038b50 0x7f0e7c03b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:00:57.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.464+0000 7f0e7bfff700 1 -- 172.21.15.16:0/2086883384 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0e8404ddc0 con 0x7f0e94106d80 2024-03-28T22:00:57.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.464+0000 7f0e927fc700 1 -- 172.21.15.16:0/2086883384 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7f0e7c038b50 msgr2=0x7f0e7c03b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/2859950725 2024-03-28T22:00:57.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.464+0000 7f0e927fc700 1 --2- 172.21.15.16:0/2086883384 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7f0e7c038b50 0x7f0e7c03b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:00:57.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.465+0000 7f0e9967e700 1 -- 172.21.15.16:0/2086883384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e94063870 con 0x7f0e94106d80 2024-03-28T22:00:57.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.470+0000 7f0e7bfff700 1 -- 172.21.15.16:0/2086883384 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0e84019c70 con 0x7f0e94106d80 2024-03-28T22:00:57.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.665+0000 7f0e927fc700 1 -- 172.21.15.16:0/2086883384 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7f0e7c038b50 msgr2=0x7f0e7c03b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/2859950725 2024-03-28T22:00:57.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.665+0000 7f0e927fc700 1 --2- 172.21.15.16:0/2086883384 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7f0e7c038b50 0x7f0e7c03b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:00:57.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.733+0000 7f0e9967e700 1 -- 172.21.15.16:0/2086883384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f0e94195c60 con 0x7f0e94106d80 2024-03-28T22:00:57.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.733+0000 7f0e7bfff700 1 -- 172.21.15.16:0/2086883384 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f0e84020240 con 0x7f0e94106d80 2024-03-28T22:00:57.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.735+0000 7f0e9967e700 1 -- 172.21.15.16:0/2086883384 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7f0e7c038b50 msgr2=0x7f0e7c03b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T22:00:57.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.735+0000 7f0e9967e700 1 --2- 172.21.15.16:0/2086883384 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7f0e7c038b50 0x7f0e7c03b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:00:57.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.735+0000 7f0e9967e700 1 -- 172.21.15.16:0/2086883384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e94106d80 msgr2=0x7f0e94191f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:00:57.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.735+0000 7f0e9967e700 1 --2- 172.21.15.16:0/2086883384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e94106d80 0x7f0e94191f30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0e84011670 tx=0x7f0e84011b60 comp rx=0 tx=0).stop 2024-03-28T22:00:57.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.735+0000 7f0e9967e700 1 -- 172.21.15.16:0/2086883384 shutdown_connections 2024-03-28T22:00:57.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.735+0000 7f0e9967e700 1 --2- 172.21.15.16:0/2086883384 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7f0e7c038b50 0x7f0e7c03b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:00:57.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.735+0000 7f0e9967e700 1 --2- 172.21.15.16:0/2086883384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e94106d80 0x7f0e94191f30 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:00:57.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.735+0000 7f0e9967e700 1 -- 172.21.15.16:0/2086883384 >> 172.21.15.16:0/2086883384 conn(0x7f0e94102340 msgr2=0x7f0e941031c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:00:57.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.736+0000 7f0e9967e700 1 -- 172.21.15.16:0/2086883384 shutdown_connections 2024-03-28T22:00:57.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:56.736+0000 7f0e9967e700 1 -- 172.21.15.16:0/2086883384 wait complete. 2024-03-28T22:00:57.287 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for the mgr to restart... 2024-03-28T22:00:57.287 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr epoch 5... 2024-03-28T22:00:57.510 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:00:57 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2086883384' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-28T22:00:57.511 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:57 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:57.246+0000 7f5cbf71b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T22:00:57.829 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:57 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:57.526+0000 7f5cbf71b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T22:00:57.829 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:57 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:57.638+0000 7f5cbf71b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T22:00:57.829 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:57 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:57.728+0000 7f5cbf71b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T22:00:58.186 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:57 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:57.826+0000 7f5cbf71b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T22:00:58.186 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:58 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:58.004+0000 7f5cbf71b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T22:00:59.436 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:58 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:58.982+0000 7f5cbf71b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T22:00:59.436 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:59 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:59.114+0000 7f5cbf71b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T22:00:59.747 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:59 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:59.451+0000 7f5cbf71b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T22:00:59.747 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:59 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:59.544+0000 7f5cbf71b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T22:00:59.747 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:59 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:59.745+0000 7f5cbf71b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T22:01:00.186 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:00:59 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:00:59.935+0000 7f5cbf71b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T22:01:00.486 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:00 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:01:00.224+0000 7f5cbf71b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T22:01:00.487 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:00 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:01:00.484+0000 7f5cbf71b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T22:01:00.744 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:00 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:01:00.726+0000 7f5cbf71b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T22:01:01.936 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:01 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:01:01.548+0000 7f5cbf71b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T22:01:02.436 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:02 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:01:02.125+0000 7f5cbf71b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T22:01:02.436 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:02 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:01:02.225+0000 7f5cbf71b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T22:01:02.936 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:02 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:01:02.446+0000 7f5cbf71b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T22:01:02.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:02 smithi016 ceph-mon[32374]: Active manager daemon x restarted 2024-03-28T22:01:02.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:02 smithi016 ceph-mon[32374]: Activating manager daemon x 2024-03-28T22:01:02.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:02 smithi016 ceph-mon[32374]: osdmap e2: 0 total, 0 up, 0 in 2024-03-28T22:01:02.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:02 smithi016 ceph-mon[32374]: mgrmap e6: x(active, starting, since 0.003438s) 2024-03-28T22:01:02.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:02 smithi016 ceph-mon[32374]: from='mgr.14120 172.21.15.16:0/3672307959' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:01:02.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:02 smithi016 ceph-mon[32374]: from='mgr.14120 172.21.15.16:0/3672307959' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T22:01:02.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:02 smithi016 ceph-mon[32374]: from='mgr.14120 172.21.15.16:0/3672307959' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T22:01:02.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:02 smithi016 ceph-mon[32374]: from='mgr.14120 172.21.15.16:0/3672307959' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T22:01:02.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:02 smithi016 ceph-mon[32374]: from='mgr.14120 172.21.15.16:0/3672307959' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T22:01:02.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:02 smithi016 ceph-mon[32374]: Manager daemon x is now available 2024-03-28T22:01:02.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:02 smithi016 ceph-mon[32374]: from='mgr.14120 172.21.15.16:0/3672307959' entity='mgr.x' 2024-03-28T22:01:02.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:02 smithi016 ceph-mon[32374]: from='mgr.14120 172.21.15.16:0/3672307959' entity='mgr.x' 2024-03-28T22:01:02.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:02 smithi016 ceph-mon[32374]: from='mgr.14120 172.21.15.16:0/3672307959' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:01:02.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:02 smithi016 ceph-mon[32374]: from='mgr.14120 172.21.15.16:0/3672307959' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:01:02.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:02 smithi016 ceph-mon[32374]: from='mgr.14120 172.21.15.16:0/3672307959' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:01:02.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:02 smithi016 ceph-mon[32374]: from='mgr.14120 172.21.15.16:0/3672307959' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T22:01:02.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:02 smithi016 ceph-mon[32374]: from='mgr.14120 172.21.15.16:0/3672307959' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T22:01:03.771 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:03 smithi016 ceph-mon[32374]: Found migration_current of "None". Setting to last migration. 2024-03-28T22:01:03.772 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:03 smithi016 ceph-mon[32374]: mgrmap e7: x(active, since 1.00536s) 2024-03-28T22:01:04.032 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-03-28T22:01:04.033 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-28T22:01:04.033 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-28T22:01:04.033 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-03-28T22:01:04.033 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.733+0000 7f513eb03700 1 Processor -- start 2024-03-28T22:01:04.033 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.733+0000 7f513eb03700 1 -- start start 2024-03-28T22:01:04.033 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.733+0000 7f513eb03700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5138108910 0x7f513810ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:04.033 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.733+0000 7f513eb03700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f513807be20 con 0x7f5138108910 2024-03-28T22:01:04.033 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.733+0000 7f513c89f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5138108910 0x7f513810ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:04.033 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.733+0000 7f513c89f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5138108910 0x7f513810ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59892/0 (socket says 172.21.15.16:59892) 2024-03-28T22:01:04.033 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.733+0000 7f513c89f700 1 -- 172.21.15.16:0/1906361717 learned_addr learned my addr 172.21.15.16:0/1906361717 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:04.033 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.734+0000 7f513c89f700 1 -- 172.21.15.16:0/1906361717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f513810b240 con 0x7f5138108910 2024-03-28T22:01:04.034 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.734+0000 7f513c89f700 1 --2- 172.21.15.16:0/1906361717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5138108910 0x7f513810ad00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f512c0098a0 tx=0x7f512c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9f8d97cf93c90ac server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:04.034 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.734+0000 7f51377fe700 1 -- 172.21.15.16:0/1906361717 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f512c004160 con 0x7f5138108910 2024-03-28T22:01:04.034 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.734+0000 7f51377fe700 1 -- 172.21.15.16:0/1906361717 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f512c004300 con 0x7f5138108910 2024-03-28T22:01:04.034 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.734+0000 7f51377fe700 1 -- 172.21.15.16:0/1906361717 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f512c011b50 con 0x7f5138108910 2024-03-28T22:01:04.034 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.734+0000 7f513eb03700 1 -- 172.21.15.16:0/1906361717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5138108910 msgr2=0x7f513810ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:04.034 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.734+0000 7f513eb03700 1 --2- 172.21.15.16:0/1906361717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5138108910 0x7f513810ad00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f512c0098a0 tx=0x7f512c009bb0 comp rx=0 tx=0).stop 2024-03-28T22:01:04.034 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.735+0000 7f513eb03700 1 -- 172.21.15.16:0/1906361717 shutdown_connections 2024-03-28T22:01:04.034 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.735+0000 7f513eb03700 1 --2- 172.21.15.16:0/1906361717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5138108910 0x7f513810ad00 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:04.034 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.735+0000 7f513eb03700 1 -- 172.21.15.16:0/1906361717 >> 172.21.15.16:0/1906361717 conn(0x7f5138102360 msgr2=0x7f5138104780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:04.034 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.735+0000 7f513eb03700 1 -- 172.21.15.16:0/1906361717 shutdown_connections 2024-03-28T22:01:04.034 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.735+0000 7f513eb03700 1 -- 172.21.15.16:0/1906361717 wait complete. 2024-03-28T22:01:04.034 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.735+0000 7f513eb03700 1 Processor -- start 2024-03-28T22:01:04.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.735+0000 7f513eb03700 1 -- start start 2024-03-28T22:01:04.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.735+0000 7f513eb03700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5138108910 0x7f513819be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:04.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.735+0000 7f513eb03700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f513810c200 con 0x7f5138108910 2024-03-28T22:01:04.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.736+0000 7f513c89f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5138108910 0x7f513819be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:04.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.736+0000 7f513c89f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5138108910 0x7f513819be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59900/0 (socket says 172.21.15.16:59900) 2024-03-28T22:01:04.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.736+0000 7f513c89f700 1 -- 172.21.15.16:0/3898943914 learned_addr learned my addr 172.21.15.16:0/3898943914 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:04.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.736+0000 7f513c89f700 1 -- 172.21.15.16:0/3898943914 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f513819c340 con 0x7f5138108910 2024-03-28T22:01:04.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.736+0000 7f513c89f700 1 --2- 172.21.15.16:0/3898943914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5138108910 0x7f513819be00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f512c011670 tx=0x7f512c011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:04.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.736+0000 7f5135ffb700 1 -- 172.21.15.16:0/3898943914 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f512c0039c0 con 0x7f5138108910 2024-03-28T22:01:04.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.736+0000 7f5135ffb700 1 -- 172.21.15.16:0/3898943914 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f512c011e50 con 0x7f5138108910 2024-03-28T22:01:04.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.736+0000 7f513eb03700 1 -- 172.21.15.16:0/3898943914 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f513819c5d0 con 0x7f5138108910 2024-03-28T22:01:04.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.736+0000 7f5135ffb700 1 -- 172.21.15.16:0/3898943914 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f512c01a5c0 con 0x7f5138108910 2024-03-28T22:01:04.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.736+0000 7f513eb03700 1 -- 172.21.15.16:0/3898943914 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f513819f2c0 con 0x7f5138108910 2024-03-28T22:01:04.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.737+0000 7f5135ffb700 1 -- 172.21.15.16:0/3898943914 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 45928+0+0 (secure 0 0 0) 0x7f512c01a870 con 0x7f5138108910 2024-03-28T22:01:04.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.737+0000 7f5135ffb700 1 --2- 172.21.15.16:0/3898943914 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7f5128038ba0 0x7f512803b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:04.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.737+0000 7f5135ffb700 1 -- 172.21.15.16:0/3898943914 --> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f512803b770 con 0x7f5128038ba0 2024-03-28T22:01:04.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.737+0000 7f5135ffb700 1 -- 172.21.15.16:0/3898943914 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f512c04cdf0 con 0x7f5138108910 2024-03-28T22:01:04.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.737+0000 7f5137fff700 1 -- 172.21.15.16:0/3898943914 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7f5128038ba0 msgr2=0x7f512803b060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/2859950725 2024-03-28T22:01:04.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.738+0000 7f5137fff700 1 --2- 172.21.15.16:0/3898943914 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7f5128038ba0 0x7f512803b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:01:04.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.939+0000 7f5137fff700 1 -- 172.21.15.16:0/3898943914 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7f5128038ba0 msgr2=0x7f512803b060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/2859950725 2024-03-28T22:01:04.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:57.939+0000 7f5137fff700 1 --2- 172.21.15.16:0/3898943914 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7f5128038ba0 0x7f512803b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:01:04.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:58.340+0000 7f5137fff700 1 -- 172.21.15.16:0/3898943914 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7f5128038ba0 msgr2=0x7f512803b060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/2859950725 2024-03-28T22:01:04.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:58.340+0000 7f5137fff700 1 --2- 172.21.15.16:0/3898943914 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7f5128038ba0 0x7f512803b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:01:04.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:59.141+0000 7f5137fff700 1 -- 172.21.15.16:0/3898943914 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7f5128038ba0 msgr2=0x7f512803b060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/2859950725 2024-03-28T22:01:04.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:00:59.141+0000 7f5137fff700 1 --2- 172.21.15.16:0/3898943914 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7f5128038ba0 0x7f512803b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:01:04.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:00.742+0000 7f5137fff700 1 -- 172.21.15.16:0/3898943914 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7f5128038ba0 msgr2=0x7f512803b060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/2859950725 2024-03-28T22:01:04.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:00.742+0000 7f5137fff700 1 --2- 172.21.15.16:0/3898943914 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7f5128038ba0 0x7f512803b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-28T22:01:04.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:02.449+0000 7f5135ffb700 1 -- 172.21.15.16:0/3898943914 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mgrmap(e 6) ==== 45695+0+0 (secure 0 0 0) 0x7f512c0248d0 con 0x7f5138108910 2024-03-28T22:01:04.037 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:02.449+0000 7f5135ffb700 1 -- 172.21.15.16:0/3898943914 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7f5128038ba0 msgr2=0x7f512803b060 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T22:01:04.037 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:02.449+0000 7f5135ffb700 1 --2- 172.21.15.16:0/3898943914 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7f5128038ba0 0x7f512803b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:04.037 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:03.450+0000 7f5135ffb700 1 -- 172.21.15.16:0/3898943914 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 7) ==== 45822+0+0 (secure 0 0 0) 0x7f512c04e510 con 0x7f5138108910 2024-03-28T22:01:04.037 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:03.450+0000 7f5135ffb700 1 --2- 172.21.15.16:0/3898943914 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f512803c4d0 0x7f512803e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:04.037 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:03.451+0000 7f5135ffb700 1 -- 172.21.15.16:0/3898943914 --> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f512c008ea0 con 0x7f512803c4d0 2024-03-28T22:01:04.037 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:03.451+0000 7f5137fff700 1 --2- 172.21.15.16:0/3898943914 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f512803c4d0 0x7f512803e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:04.037 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:03.452+0000 7f5137fff700 1 --2- 172.21.15.16:0/3898943914 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f512803c4d0 0x7f512803e8c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f5124003a10 tx=0x7f51240092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:04.037 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:03.452+0000 7f5135ffb700 1 -- 172.21.15.16:0/3898943914 <== mgr.14120 v2:172.21.15.16:6800/1832422779 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f512c008ea0 con 0x7f512803c4d0 2024-03-28T22:01:04.037 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:03.464+0000 7f513eb03700 1 -- 172.21.15.16:0/3898943914 --> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f5138063870 con 0x7f512803c4d0 2024-03-28T22:01:04.037 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:03.464+0000 7f5135ffb700 1 -- 172.21.15.16:0/3898943914 <== mgr.14120 v2:172.21.15.16:6800/1832422779 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f5138063870 con 0x7f512803c4d0 2024-03-28T22:01:04.037 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:03.464+0000 7f513eb03700 1 -- 172.21.15.16:0/3898943914 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f512803c4d0 msgr2=0x7f512803e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:04.037 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:03.464+0000 7f513eb03700 1 --2- 172.21.15.16:0/3898943914 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f512803c4d0 0x7f512803e8c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f5124003a10 tx=0x7f51240092b0 comp rx=0 tx=0).stop 2024-03-28T22:01:04.037 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:03.464+0000 7f513eb03700 1 -- 172.21.15.16:0/3898943914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5138108910 msgr2=0x7f513819be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:04.038 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:03.464+0000 7f513eb03700 1 --2- 172.21.15.16:0/3898943914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5138108910 0x7f513819be00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f512c011670 tx=0x7f512c011b60 comp rx=0 tx=0).stop 2024-03-28T22:01:04.038 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:03.464+0000 7f513eb03700 1 -- 172.21.15.16:0/3898943914 shutdown_connections 2024-03-28T22:01:04.038 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:03.464+0000 7f513eb03700 1 --2- 172.21.15.16:0/3898943914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5138108910 0x7f513819be00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:04.038 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:03.464+0000 7f513eb03700 1 --2- 172.21.15.16:0/3898943914 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f512803c4d0 0x7f512803e8c0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:04.038 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:03.464+0000 7f513eb03700 1 --2- 172.21.15.16:0/3898943914 >> [v2:172.21.15.16:6800/2859950725,v1:172.21.15.16:6801/2859950725] conn(0x7f5128038ba0 0x7f512803b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:04.038 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:03.464+0000 7f513eb03700 1 -- 172.21.15.16:0/3898943914 >> 172.21.15.16:0/3898943914 conn(0x7f5138102360 msgr2=0x7f5138102f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:04.038 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:03.464+0000 7f513eb03700 1 -- 172.21.15.16:0/3898943914 shutdown_connections 2024-03-28T22:01:04.038 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:03.464+0000 7f513eb03700 1 -- 172.21.15.16:0/3898943914 wait complete. 2024-03-28T22:01:04.038 INFO:teuthology.orchestra.run.smithi016.stdout:mgr epoch 5 is available 2024-03-28T22:01:04.038 INFO:teuthology.orchestra.run.smithi016.stdout:Setting orchestrator backend to cephadm... 2024-03-28T22:01:04.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:04 smithi016 ceph-mon[32374]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-28T22:01:04.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:04 smithi016 ceph-mon[32374]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-28T22:01:04.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:04 smithi016 ceph-mon[32374]: from='mgr.14120 172.21.15.16:0/3672307959' entity='mgr.x' 2024-03-28T22:01:04.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:04 smithi016 ceph-mon[32374]: from='mgr.14120 172.21.15.16:0/3672307959' entity='mgr.x' 2024-03-28T22:01:04.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:04 smithi016 ceph-mon[32374]: from='mgr.14120 172.21.15.16:0/3672307959' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:01:05.242 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.457+0000 7f6486d8b700 1 Processor -- start 2024-03-28T22:01:05.243 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.457+0000 7f6486d8b700 1 -- start start 2024-03-28T22:01:05.243 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.457+0000 7f6486d8b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f648007dd40 0x7f648007c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:05.243 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.457+0000 7f6486d8b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f648007c840 con 0x7f648007dd40 2024-03-28T22:01:05.243 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.457+0000 7f6484b27700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f648007dd40 0x7f648007c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:05.243 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.457+0000 7f6484b27700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f648007dd40 0x7f648007c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57748/0 (socket says 172.21.15.16:57748) 2024-03-28T22:01:05.244 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.458+0000 7f6484b27700 1 -- 172.21.15.16:0/3758037236 learned_addr learned my addr 172.21.15.16:0/3758037236 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:05.244 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.458+0000 7f6484b27700 1 -- 172.21.15.16:0/3758037236 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f648007c9c0 con 0x7f648007dd40 2024-03-28T22:01:05.244 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.458+0000 7f6484b27700 1 --2- 172.21.15.16:0/3758037236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f648007dd40 0x7f648007c300 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6478009e80 tx=0x7f647800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc49fa2948685ad3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:05.244 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.458+0000 7f647f7fe700 1 -- 172.21.15.16:0/3758037236 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6478017070 con 0x7f648007dd40 2024-03-28T22:01:05.244 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.458+0000 7f647f7fe700 1 -- 172.21.15.16:0/3758037236 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f647800b8c0 con 0x7f648007dd40 2024-03-28T22:01:05.244 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.458+0000 7f647f7fe700 1 -- 172.21.15.16:0/3758037236 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f64780045c0 con 0x7f648007dd40 2024-03-28T22:01:05.244 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.459+0000 7f6486d8b700 1 -- 172.21.15.16:0/3758037236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f648007dd40 msgr2=0x7f648007c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:05.245 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.459+0000 7f6486d8b700 1 --2- 172.21.15.16:0/3758037236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f648007dd40 0x7f648007c300 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6478009e80 tx=0x7f647800b330 comp rx=0 tx=0).stop 2024-03-28T22:01:05.245 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.459+0000 7f6486d8b700 1 -- 172.21.15.16:0/3758037236 shutdown_connections 2024-03-28T22:01:05.245 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.459+0000 7f6486d8b700 1 --2- 172.21.15.16:0/3758037236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f648007dd40 0x7f648007c300 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:05.245 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.459+0000 7f6486d8b700 1 -- 172.21.15.16:0/3758037236 >> 172.21.15.16:0/3758037236 conn(0x7f6480103550 msgr2=0x7f6480105970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:05.245 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.459+0000 7f6486d8b700 1 -- 172.21.15.16:0/3758037236 shutdown_connections 2024-03-28T22:01:05.245 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.459+0000 7f6486d8b700 1 -- 172.21.15.16:0/3758037236 wait complete. 2024-03-28T22:01:05.245 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.459+0000 7f6486d8b700 1 Processor -- start 2024-03-28T22:01:05.245 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.459+0000 7f6486d8b700 1 -- start start 2024-03-28T22:01:05.245 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.459+0000 7f6486d8b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f648007dd40 0x7f648019df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:05.246 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.459+0000 7f6486d8b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f648010e3a0 con 0x7f648007dd40 2024-03-28T22:01:05.246 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.460+0000 7f6484b27700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f648007dd40 0x7f648019df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:05.246 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.460+0000 7f6484b27700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f648007dd40 0x7f648019df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57758/0 (socket says 172.21.15.16:57758) 2024-03-28T22:01:05.246 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.460+0000 7f6484b27700 1 -- 172.21.15.16:0/3138436245 learned_addr learned my addr 172.21.15.16:0/3138436245 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:05.246 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.460+0000 7f6484b27700 1 -- 172.21.15.16:0/3138436245 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f648019e4d0 con 0x7f648007dd40 2024-03-28T22:01:05.246 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.460+0000 7f6484b27700 1 --2- 172.21.15.16:0/3138436245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f648007dd40 0x7f648019df90 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6478004350 tx=0x7f647800bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:05.247 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.460+0000 7f647dffb700 1 -- 172.21.15.16:0/3138436245 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6478020020 con 0x7f648007dd40 2024-03-28T22:01:05.247 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.460+0000 7f647dffb700 1 -- 172.21.15.16:0/3138436245 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6478003c10 con 0x7f648007dd40 2024-03-28T22:01:05.247 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.460+0000 7f6486d8b700 1 -- 172.21.15.16:0/3138436245 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f648019e760 con 0x7f648007dd40 2024-03-28T22:01:05.247 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.460+0000 7f647dffb700 1 -- 172.21.15.16:0/3138436245 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6478017040 con 0x7f648007dd40 2024-03-28T22:01:05.247 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.460+0000 7f6486d8b700 1 -- 172.21.15.16:0/3138436245 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64801a1450 con 0x7f648007dd40 2024-03-28T22:01:05.247 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.461+0000 7f647dffb700 1 -- 172.21.15.16:0/3138436245 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 7) ==== 45822+0+0 (secure 0 0 0) 0x7f6478023020 con 0x7f648007dd40 2024-03-28T22:01:05.247 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.461+0000 7f647dffb700 1 --2- 172.21.15.16:0/3138436245 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f6468038ad0 0x7f646803af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:05.247 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.461+0000 7f647dffb700 1 -- 172.21.15.16:0/3138436245 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f647800f050 con 0x7f648007dd40 2024-03-28T22:01:05.247 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.461+0000 7f6486d8b700 1 -- 172.21.15.16:0/3138436245 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6480063870 con 0x7f648007dd40 2024-03-28T22:01:05.248 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.461+0000 7f647ffff700 1 --2- 172.21.15.16:0/3138436245 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f6468038ad0 0x7f646803af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:05.248 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.462+0000 7f647ffff700 1 --2- 172.21.15.16:0/3138436245 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f6468038ad0 0x7f646803af90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6474006fd0 tx=0x7f6474006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:05.248 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.467+0000 7f647dffb700 1 -- 172.21.15.16:0/3138436245 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f647802a020 con 0x7f648007dd40 2024-03-28T22:01:05.248 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.563+0000 7f647dffb700 1 -- 172.21.15.16:0/3138436245 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7f6478045d10 con 0x7f648007dd40 2024-03-28T22:01:05.248 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.688+0000 7f6486d8b700 1 -- 172.21.15.16:0/3138436245 --> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f6480104770 con 0x7f6468038ad0 2024-03-28T22:01:05.248 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.693+0000 7f647dffb700 1 -- 172.21.15.16:0/3138436245 <== mgr.14120 v2:172.21.15.16:6800/1832422779 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f6480104770 con 0x7f6468038ad0 2024-03-28T22:01:05.248 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.696+0000 7f6486d8b700 1 -- 172.21.15.16:0/3138436245 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f6468038ad0 msgr2=0x7f646803af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:05.249 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.696+0000 7f6486d8b700 1 --2- 172.21.15.16:0/3138436245 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f6468038ad0 0x7f646803af90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6474006fd0 tx=0x7f6474006e40 comp rx=0 tx=0).stop 2024-03-28T22:01:05.249 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.696+0000 7f6486d8b700 1 -- 172.21.15.16:0/3138436245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f648007dd40 msgr2=0x7f648019df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:05.249 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.696+0000 7f6486d8b700 1 --2- 172.21.15.16:0/3138436245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f648007dd40 0x7f648019df90 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6478004350 tx=0x7f647800bd50 comp rx=0 tx=0).stop 2024-03-28T22:01:05.249 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.696+0000 7f6486d8b700 1 -- 172.21.15.16:0/3138436245 shutdown_connections 2024-03-28T22:01:05.249 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.696+0000 7f6486d8b700 1 --2- 172.21.15.16:0/3138436245 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f6468038ad0 0x7f646803af90 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:05.249 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.696+0000 7f6486d8b700 1 --2- 172.21.15.16:0/3138436245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f648007dd40 0x7f648019df90 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:05.249 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.696+0000 7f6486d8b700 1 -- 172.21.15.16:0/3138436245 >> 172.21.15.16:0/3138436245 conn(0x7f6480103550 msgr2=0x7f6480104060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:05.249 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.696+0000 7f6486d8b700 1 -- 172.21.15.16:0/3138436245 shutdown_connections 2024-03-28T22:01:05.250 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:04.696+0000 7f6486d8b700 1 -- 172.21.15.16:0/3138436245 wait complete. 2024-03-28T22:01:05.642 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:05 smithi016 ceph-mon[32374]: [28/Mar/2024:22:01:03] ENGINE Bus STARTING 2024-03-28T22:01:05.642 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:05 smithi016 ceph-mon[32374]: [28/Mar/2024:22:01:04] ENGINE Serving on https://172.21.15.16:7150 2024-03-28T22:01:05.642 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:05 smithi016 ceph-mon[32374]: [28/Mar/2024:22:01:04] ENGINE Serving on http://172.21.15.16:8765 2024-03-28T22:01:05.642 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:05 smithi016 ceph-mon[32374]: [28/Mar/2024:22:01:04] ENGINE Bus STARTED 2024-03-28T22:01:05.642 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:05 smithi016 ceph-mon[32374]: mgrmap e8: x(active, since 2s) 2024-03-28T22:01:05.642 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:05 smithi016 ceph-mon[32374]: from='mgr.14120 172.21.15.16:0/3672307959' entity='mgr.x' 2024-03-28T22:01:05.642 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:05 smithi016 ceph-mon[32374]: from='mgr.14120 172.21.15.16:0/3672307959' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:01:06.479 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-28T22:01:06.479 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.661+0000 7fda6e575700 1 Processor -- start 2024-03-28T22:01:06.479 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.661+0000 7fda6e575700 1 -- start start 2024-03-28T22:01:06.479 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.661+0000 7fda6e575700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda68106d70 0x7fda68107190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:06.479 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.661+0000 7fda6e575700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fda68107760 con 0x7fda68106d70 2024-03-28T22:01:06.480 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.661+0000 7fda67fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda68106d70 0x7fda68107190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:06.480 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.661+0000 7fda67fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda68106d70 0x7fda68107190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57760/0 (socket says 172.21.15.16:57760) 2024-03-28T22:01:06.480 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.661+0000 7fda67fff700 1 -- 172.21.15.16:0/1851729954 learned_addr learned my addr 172.21.15.16:0/1851729954 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:06.480 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.662+0000 7fda67fff700 1 -- 172.21.15.16:0/1851729954 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda68107f70 con 0x7fda68106d70 2024-03-28T22:01:06.480 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.662+0000 7fda67fff700 1 --2- 172.21.15.16:0/1851729954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda68106d70 0x7fda68107190 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fda500098a0 tx=0x7fda50009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c746f6e5eade6d62 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:06.480 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.662+0000 7fda66ffd700 1 -- 172.21.15.16:0/1851729954 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fda50004160 con 0x7fda68106d70 2024-03-28T22:01:06.480 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.662+0000 7fda66ffd700 1 -- 172.21.15.16:0/1851729954 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fda50004300 con 0x7fda68106d70 2024-03-28T22:01:06.480 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.662+0000 7fda66ffd700 1 -- 172.21.15.16:0/1851729954 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fda50011b50 con 0x7fda68106d70 2024-03-28T22:01:06.480 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.663+0000 7fda6e575700 1 -- 172.21.15.16:0/1851729954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda68106d70 msgr2=0x7fda68107190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:06.480 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.663+0000 7fda6e575700 1 --2- 172.21.15.16:0/1851729954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda68106d70 0x7fda68107190 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fda500098a0 tx=0x7fda50009bb0 comp rx=0 tx=0).stop 2024-03-28T22:01:06.480 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.663+0000 7fda6e575700 1 -- 172.21.15.16:0/1851729954 shutdown_connections 2024-03-28T22:01:06.480 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.663+0000 7fda6e575700 1 --2- 172.21.15.16:0/1851729954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda68106d70 0x7fda68107190 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:06.480 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.663+0000 7fda6e575700 1 -- 172.21.15.16:0/1851729954 >> 172.21.15.16:0/1851729954 conn(0x7fda68102310 msgr2=0x7fda68104750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:06.481 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.663+0000 7fda6e575700 1 -- 172.21.15.16:0/1851729954 shutdown_connections 2024-03-28T22:01:06.481 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.663+0000 7fda6e575700 1 -- 172.21.15.16:0/1851729954 wait complete. 2024-03-28T22:01:06.481 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.663+0000 7fda6e575700 1 Processor -- start 2024-03-28T22:01:06.481 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.663+0000 7fda6e575700 1 -- start start 2024-03-28T22:01:06.481 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.664+0000 7fda6e575700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda68106d70 0x7fda6819ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:06.481 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.664+0000 7fda6e575700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fda6810b320 con 0x7fda68106d70 2024-03-28T22:01:06.481 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.664+0000 7fda67fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda68106d70 0x7fda6819ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:06.481 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.664+0000 7fda67fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda68106d70 0x7fda6819ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57776/0 (socket says 172.21.15.16:57776) 2024-03-28T22:01:06.481 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.664+0000 7fda67fff700 1 -- 172.21.15.16:0/474866192 learned_addr learned my addr 172.21.15.16:0/474866192 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:06.481 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.664+0000 7fda67fff700 1 -- 172.21.15.16:0/474866192 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda6819b260 con 0x7fda68106d70 2024-03-28T22:01:06.481 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.664+0000 7fda67fff700 1 --2- 172.21.15.16:0/474866192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda68106d70 0x7fda6819ad20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fda50000c00 tx=0x7fda50011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:06.481 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.664+0000 7fda657fa700 1 -- 172.21.15.16:0/474866192 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fda50003b20 con 0x7fda68106d70 2024-03-28T22:01:06.481 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.664+0000 7fda657fa700 1 -- 172.21.15.16:0/474866192 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fda50003cc0 con 0x7fda68106d70 2024-03-28T22:01:06.482 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.664+0000 7fda6e575700 1 -- 172.21.15.16:0/474866192 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda6819b4f0 con 0x7fda68106d70 2024-03-28T22:01:06.482 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.664+0000 7fda657fa700 1 -- 172.21.15.16:0/474866192 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fda50024910 con 0x7fda68106d70 2024-03-28T22:01:06.482 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.664+0000 7fda6e575700 1 -- 172.21.15.16:0/474866192 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda6819e1d0 con 0x7fda68106d70 2024-03-28T22:01:06.482 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.665+0000 7fda657fa700 1 -- 172.21.15.16:0/474866192 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7fda50022020 con 0x7fda68106d70 2024-03-28T22:01:06.482 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.665+0000 7fda6e575700 1 -- 172.21.15.16:0/474866192 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda68063870 con 0x7fda68106d70 2024-03-28T22:01:06.482 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.666+0000 7fda657fa700 1 --2- 172.21.15.16:0/474866192 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7fda54038bf0 0x7fda5403b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:06.482 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.666+0000 7fda657fa700 1 -- 172.21.15.16:0/474866192 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fda50011e00 con 0x7fda68106d70 2024-03-28T22:01:06.482 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.666+0000 7fda677fe700 1 --2- 172.21.15.16:0/474866192 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7fda54038bf0 0x7fda5403b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:06.482 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.666+0000 7fda677fe700 1 --2- 172.21.15.16:0/474866192 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7fda54038bf0 0x7fda5403b0b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fda58006fd0 tx=0x7fda58006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:06.482 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.671+0000 7fda657fa700 1 -- 172.21.15.16:0/474866192 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fda5002cdf0 con 0x7fda68106d70 2024-03-28T22:01:06.482 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.877+0000 7fda6e575700 1 -- 172.21.15.16:0/474866192 --> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fda68103540 con 0x7fda54038bf0 2024-03-28T22:01:06.482 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.878+0000 7fda657fa700 1 -- 172.21.15.16:0/474866192 <== mgr.14120 v2:172.21.15.16:6800/1832422779 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fda68103540 con 0x7fda54038bf0 2024-03-28T22:01:06.483 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.880+0000 7fda6e575700 1 -- 172.21.15.16:0/474866192 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7fda54038bf0 msgr2=0x7fda5403b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:06.483 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.880+0000 7fda6e575700 1 --2- 172.21.15.16:0/474866192 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7fda54038bf0 0x7fda5403b0b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fda58006fd0 tx=0x7fda58006e40 comp rx=0 tx=0).stop 2024-03-28T22:01:06.483 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.880+0000 7fda6e575700 1 -- 172.21.15.16:0/474866192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda68106d70 msgr2=0x7fda6819ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:06.483 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.880+0000 7fda6e575700 1 --2- 172.21.15.16:0/474866192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda68106d70 0x7fda6819ad20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fda50000c00 tx=0x7fda50011c10 comp rx=0 tx=0).stop 2024-03-28T22:01:06.483 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.880+0000 7fda6e575700 1 -- 172.21.15.16:0/474866192 shutdown_connections 2024-03-28T22:01:06.483 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.880+0000 7fda6e575700 1 --2- 172.21.15.16:0/474866192 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7fda54038bf0 0x7fda5403b0b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:06.483 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.880+0000 7fda6e575700 1 --2- 172.21.15.16:0/474866192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda68106d70 0x7fda6819ad20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:06.483 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.880+0000 7fda6e575700 1 -- 172.21.15.16:0/474866192 >> 172.21.15.16:0/474866192 conn(0x7fda68102310 msgr2=0x7fda68102e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:06.483 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.880+0000 7fda6e575700 1 -- 172.21.15.16:0/474866192 shutdown_connections 2024-03-28T22:01:06.483 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:05.880+0000 7fda6e575700 1 -- 172.21.15.16:0/474866192 wait complete. 2024-03-28T22:01:06.483 INFO:teuthology.orchestra.run.smithi016.stdout:Generating ssh key... 2024-03-28T22:01:06.814 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:06 smithi016 ceph-mon[32374]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:01:07.417 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:07 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: Generating public/private rsa key pair. 2024-03-28T22:01:07.417 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:07 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: Your identification has been saved in /tmp/tmpclcnpv_8/key. 2024-03-28T22:01:07.417 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:07 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: Your public key has been saved in /tmp/tmpclcnpv_8/key.pub. 2024-03-28T22:01:07.417 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:07 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: The key fingerprint is: 2024-03-28T22:01:07.417 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:07 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: SHA256:83a0bXZxxl8hqsYhz5b12HsWKh4ev84gxkMhdYfuEQY ceph-798d9884-ed4e-11ee-95d0-87774f69a715 2024-03-28T22:01:07.417 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:07 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: The key's randomart image is: 2024-03-28T22:01:07.417 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:07 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: +---[RSA 3072]----+ 2024-03-28T22:01:07.418 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:07 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: | E.... | 2024-03-28T22:01:07.418 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:07 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: | . .+. | 2024-03-28T22:01:07.418 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:07 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: | . .o .. . | 2024-03-28T22:01:07.418 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:07 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: | . .o. ...| 2024-03-28T22:01:07.418 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:07 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: | .So.oo .=| 2024-03-28T22:01:07.418 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:07 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: | Bo=o+o o=| 2024-03-28T22:01:07.418 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:07 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: | &o=oo= +| 2024-03-28T22:01:07.418 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:07 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: | +.+o*+.+ | 2024-03-28T22:01:07.418 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:07 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: | .oo*+ | 2024-03-28T22:01:07.418 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:07 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: +----[SHA256]-----+ 2024-03-28T22:01:07.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:07 smithi016 ceph-mon[32374]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:01:07.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:07 smithi016 ceph-mon[32374]: from='mgr.14120 172.21.15.16:0/3672307959' entity='mgr.x' 2024-03-28T22:01:07.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:07 smithi016 ceph-mon[32374]: from='mgr.14120 172.21.15.16:0/3672307959' entity='mgr.x' 2024-03-28T22:01:07.708 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.840+0000 7fe8dfcf5700 1 Processor -- start 2024-03-28T22:01:07.709 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.840+0000 7fe8dfcf5700 1 -- start start 2024-03-28T22:01:07.709 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.840+0000 7fe8dfcf5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8d8107f20 0x7fe8d8108340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:07.709 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.840+0000 7fe8dfcf5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe8d8108880 con 0x7fe8d8107f20 2024-03-28T22:01:07.709 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.840+0000 7fe8dda91700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8d8107f20 0x7fe8d8108340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:07.709 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.840+0000 7fe8dda91700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8d8107f20 0x7fe8d8108340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57786/0 (socket says 172.21.15.16:57786) 2024-03-28T22:01:07.709 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.840+0000 7fe8dda91700 1 -- 172.21.15.16:0/1211824398 learned_addr learned my addr 172.21.15.16:0/1211824398 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:07.709 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.840+0000 7fe8dda91700 1 -- 172.21.15.16:0/1211824398 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8d8108fe0 con 0x7fe8d8107f20 2024-03-28T22:01:07.709 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.841+0000 7fe8dda91700 1 --2- 172.21.15.16:0/1211824398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8d8107f20 0x7fe8d8108340 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe8d40098a0 tx=0x7fe8d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a712d43e6e100e22 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:07.709 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.841+0000 7fe8dca8f700 1 -- 172.21.15.16:0/1211824398 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe8d4004160 con 0x7fe8d8107f20 2024-03-28T22:01:07.709 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.841+0000 7fe8dca8f700 1 -- 172.21.15.16:0/1211824398 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe8d4004300 con 0x7fe8d8107f20 2024-03-28T22:01:07.709 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.841+0000 7fe8dca8f700 1 -- 172.21.15.16:0/1211824398 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe8d4011b50 con 0x7fe8d8107f20 2024-03-28T22:01:07.709 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.841+0000 7fe8dfcf5700 1 -- 172.21.15.16:0/1211824398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8d8107f20 msgr2=0x7fe8d8108340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:07.709 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.841+0000 7fe8dfcf5700 1 --2- 172.21.15.16:0/1211824398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8d8107f20 0x7fe8d8108340 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe8d40098a0 tx=0x7fe8d4009bb0 comp rx=0 tx=0).stop 2024-03-28T22:01:07.710 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.841+0000 7fe8dfcf5700 1 -- 172.21.15.16:0/1211824398 shutdown_connections 2024-03-28T22:01:07.710 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.841+0000 7fe8dfcf5700 1 --2- 172.21.15.16:0/1211824398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8d8107f20 0x7fe8d8108340 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:07.710 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.841+0000 7fe8dfcf5700 1 -- 172.21.15.16:0/1211824398 >> 172.21.15.16:0/1211824398 conn(0x7fe8d81034a0 msgr2=0x7fe8d8105900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:07.710 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.841+0000 7fe8dfcf5700 1 -- 172.21.15.16:0/1211824398 shutdown_connections 2024-03-28T22:01:07.710 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.841+0000 7fe8dfcf5700 1 -- 172.21.15.16:0/1211824398 wait complete. 2024-03-28T22:01:07.710 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.842+0000 7fe8dfcf5700 1 Processor -- start 2024-03-28T22:01:07.710 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.842+0000 7fe8dfcf5700 1 -- start start 2024-03-28T22:01:07.710 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.842+0000 7fe8dfcf5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8d8107f20 0x7fe8d819be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:07.710 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.842+0000 7fe8dfcf5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe8d810c420 con 0x7fe8d8107f20 2024-03-28T22:01:07.710 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.842+0000 7fe8dda91700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8d8107f20 0x7fe8d819be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:07.710 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.842+0000 7fe8dda91700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8d8107f20 0x7fe8d819be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57800/0 (socket says 172.21.15.16:57800) 2024-03-28T22:01:07.710 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.842+0000 7fe8dda91700 1 -- 172.21.15.16:0/297510155 learned_addr learned my addr 172.21.15.16:0/297510155 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:07.710 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.843+0000 7fe8dda91700 1 -- 172.21.15.16:0/297510155 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8d819c340 con 0x7fe8d8107f20 2024-03-28T22:01:07.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.843+0000 7fe8dda91700 1 --2- 172.21.15.16:0/297510155 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8d8107f20 0x7fe8d819be00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe8d4011670 tx=0x7fe8d4011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:07.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.843+0000 7fe8ceffd700 1 -- 172.21.15.16:0/297510155 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe8d40039c0 con 0x7fe8d8107f20 2024-03-28T22:01:07.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.843+0000 7fe8ceffd700 1 -- 172.21.15.16:0/297510155 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe8d4011e50 con 0x7fe8d8107f20 2024-03-28T22:01:07.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.843+0000 7fe8ceffd700 1 -- 172.21.15.16:0/297510155 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe8d401a5c0 con 0x7fe8d8107f20 2024-03-28T22:01:07.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.843+0000 7fe8dfcf5700 1 -- 172.21.15.16:0/297510155 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8d819c5d0 con 0x7fe8d8107f20 2024-03-28T22:01:07.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.843+0000 7fe8dfcf5700 1 -- 172.21.15.16:0/297510155 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8d819f2c0 con 0x7fe8d8107f20 2024-03-28T22:01:07.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.844+0000 7fe8ceffd700 1 -- 172.21.15.16:0/297510155 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7fe8d4024430 con 0x7fe8d8107f20 2024-03-28T22:01:07.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.844+0000 7fe8dfcf5700 1 -- 172.21.15.16:0/297510155 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8d8063870 con 0x7fe8d8107f20 2024-03-28T22:01:07.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.844+0000 7fe8ceffd700 1 --2- 172.21.15.16:0/297510155 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7fe8c4038ba0 0x7fe8c403b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:07.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.844+0000 7fe8dd290700 1 --2- 172.21.15.16:0/297510155 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7fe8c4038ba0 0x7fe8c403b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:07.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.844+0000 7fe8ceffd700 1 -- 172.21.15.16:0/297510155 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe8d404ce30 con 0x7fe8d8107f20 2024-03-28T22:01:07.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.844+0000 7fe8dd290700 1 --2- 172.21.15.16:0/297510155 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7fe8c4038ba0 0x7fe8c403b060 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe8c8006fd0 tx=0x7fe8c8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:07.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:06.849+0000 7fe8ceffd700 1 -- 172.21.15.16:0/297510155 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe8d4003b60 con 0x7fe8d8107f20 2024-03-28T22:01:07.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:07.055+0000 7fe8dfcf5700 1 -- 172.21.15.16:0/297510155 --> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fe8d81046c0 con 0x7fe8c4038ba0 2024-03-28T22:01:07.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:07.121+0000 7fe8ceffd700 1 -- 172.21.15.16:0/297510155 <== mgr.14120 v2:172.21.15.16:6800/1832422779 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fe8d81046c0 con 0x7fe8c4038ba0 2024-03-28T22:01:07.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:07.123+0000 7fe8dfcf5700 1 -- 172.21.15.16:0/297510155 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7fe8c4038ba0 msgr2=0x7fe8c403b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:07.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:07.123+0000 7fe8dfcf5700 1 --2- 172.21.15.16:0/297510155 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7fe8c4038ba0 0x7fe8c403b060 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe8c8006fd0 tx=0x7fe8c8006e40 comp rx=0 tx=0).stop 2024-03-28T22:01:07.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:07.123+0000 7fe8dfcf5700 1 -- 172.21.15.16:0/297510155 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8d8107f20 msgr2=0x7fe8d819be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:07.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:07.123+0000 7fe8dfcf5700 1 --2- 172.21.15.16:0/297510155 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8d8107f20 0x7fe8d819be00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe8d4011670 tx=0x7fe8d4011b60 comp rx=0 tx=0).stop 2024-03-28T22:01:07.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:07.123+0000 7fe8dfcf5700 1 -- 172.21.15.16:0/297510155 shutdown_connections 2024-03-28T22:01:07.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:07.123+0000 7fe8dfcf5700 1 --2- 172.21.15.16:0/297510155 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7fe8c4038ba0 0x7fe8c403b060 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:07.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:07.123+0000 7fe8dfcf5700 1 --2- 172.21.15.16:0/297510155 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8d8107f20 0x7fe8d819be00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:07.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:07.123+0000 7fe8dfcf5700 1 -- 172.21.15.16:0/297510155 >> 172.21.15.16:0/297510155 conn(0x7fe8d81034a0 msgr2=0x7fe8d8103fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:07.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:07.123+0000 7fe8dfcf5700 1 -- 172.21.15.16:0/297510155 shutdown_connections 2024-03-28T22:01:07.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:07.123+0000 7fe8dfcf5700 1 -- 172.21.15.16:0/297510155 wait complete. 2024-03-28T22:01:08.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:08 smithi016 ceph-mon[32374]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:01:08.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:08 smithi016 ceph-mon[32374]: Generating ssh key... 2024-03-28T22:01:08.941 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqpS+LL1RstHXIb1tCrS18jre6EvgvPhh/oXAEzmEQZB46e/fgm/6qia0uQXMNyq3hE6y4a5IiWOrXZzYt9zwqC5hBpOXkTChog0KCAk9hAvf4QyGZdwLwURSPYPlsGr0pUHum+nKQMw7LUBTR0n7HrrMR2a4faZyq9UPc76DyC4RXtyuT5GXeygB2muIjn4zVTyB/FiS8e6+7E7UJEQYErRyuDu/Mj0hgXxZWelvt1cq4YLJoCfbxREl77DyOCBCyQDRYqrMqxTzWqbuFdZ/3CVndEbzIT5ubqkrBArl092CkD7FZ/M4weUnLGbHBhXA7OYKxdIOSucvo3FbpGDWNGM02KyZ7xiESsrYCzvKEXygOTp4rVhjXuz4rYv8m74v2PTKTNiYiB0SwkHXpMV8tlPcByzcADb0k9kHZzvlzH31Anfx7WVJ7oevTt48HbxvtBS99PVQ/wXYBc2Fl0p/7QsSC6bRMB6OmDr8qGt1Smh7Hpyhy54LdSIWLvwDZE/M= ceph-798d9884-ed4e-11ee-95d0-87774f69a715 2024-03-28T22:01:08.941 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.143+0000 7f5126284700 1 Processor -- start 2024-03-28T22:01:08.942 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.143+0000 7f5126284700 1 -- start start 2024-03-28T22:01:08.942 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.143+0000 7f5126284700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5120108900 0x7f512010acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:08.942 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.143+0000 7f5126284700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f512007be70 con 0x7f5120108900 2024-03-28T22:01:08.942 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.143+0000 7f511ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5120108900 0x7f512010acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:08.942 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.143+0000 7f511ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5120108900 0x7f512010acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57806/0 (socket says 172.21.15.16:57806) 2024-03-28T22:01:08.942 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.143+0000 7f511ffff700 1 -- 172.21.15.16:0/1657746740 learned_addr learned my addr 172.21.15.16:0/1657746740 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:08.942 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.143+0000 7f511ffff700 1 -- 172.21.15.16:0/1657746740 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f512010b230 con 0x7f5120108900 2024-03-28T22:01:08.942 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.144+0000 7f511ffff700 1 --2- 172.21.15.16:0/1657746740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5120108900 0x7f512010acf0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f51100098a0 tx=0x7f5110009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60b43aad2e212872 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:08.942 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.144+0000 7f511effd700 1 -- 172.21.15.16:0/1657746740 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5110004160 con 0x7f5120108900 2024-03-28T22:01:08.942 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.144+0000 7f511effd700 1 -- 172.21.15.16:0/1657746740 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5110004300 con 0x7f5120108900 2024-03-28T22:01:08.942 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.144+0000 7f511effd700 1 -- 172.21.15.16:0/1657746740 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5110011b50 con 0x7f5120108900 2024-03-28T22:01:08.942 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.144+0000 7f5126284700 1 -- 172.21.15.16:0/1657746740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5120108900 msgr2=0x7f512010acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:08.942 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.144+0000 7f5126284700 1 --2- 172.21.15.16:0/1657746740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5120108900 0x7f512010acf0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f51100098a0 tx=0x7f5110009bb0 comp rx=0 tx=0).stop 2024-03-28T22:01:08.943 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.144+0000 7f5126284700 1 -- 172.21.15.16:0/1657746740 shutdown_connections 2024-03-28T22:01:08.943 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.144+0000 7f5126284700 1 --2- 172.21.15.16:0/1657746740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5120108900 0x7f512010acf0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:08.943 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.144+0000 7f5126284700 1 -- 172.21.15.16:0/1657746740 >> 172.21.15.16:0/1657746740 conn(0x7f5120102310 msgr2=0x7f5120104750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:08.943 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.145+0000 7f5126284700 1 -- 172.21.15.16:0/1657746740 shutdown_connections 2024-03-28T22:01:08.943 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.145+0000 7f5126284700 1 -- 172.21.15.16:0/1657746740 wait complete. 2024-03-28T22:01:08.943 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.145+0000 7f5126284700 1 Processor -- start 2024-03-28T22:01:08.943 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.145+0000 7f5126284700 1 -- start start 2024-03-28T22:01:08.943 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.145+0000 7f5126284700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5120108900 0x7f512019bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:08.943 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.145+0000 7f5126284700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f512010c1e0 con 0x7f5120108900 2024-03-28T22:01:08.943 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.146+0000 7f511ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5120108900 0x7f512019bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:08.943 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.146+0000 7f511ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5120108900 0x7f512019bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57808/0 (socket says 172.21.15.16:57808) 2024-03-28T22:01:08.943 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.146+0000 7f511ffff700 1 -- 172.21.15.16:0/561322471 learned_addr learned my addr 172.21.15.16:0/561322471 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:08.943 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.146+0000 7f511ffff700 1 -- 172.21.15.16:0/561322471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f512019c2a0 con 0x7f5120108900 2024-03-28T22:01:08.943 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.146+0000 7f511ffff700 1 --2- 172.21.15.16:0/561322471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5120108900 0x7f512019bd60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5110011b60 tx=0x7f5110011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:08.944 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.146+0000 7f511d7fa700 1 -- 172.21.15.16:0/561322471 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5110003b20 con 0x7f5120108900 2024-03-28T22:01:08.944 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.146+0000 7f511d7fa700 1 -- 172.21.15.16:0/561322471 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5110003cc0 con 0x7f5120108900 2024-03-28T22:01:08.944 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.146+0000 7f5126284700 1 -- 172.21.15.16:0/561322471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f512019c590 con 0x7f5120108900 2024-03-28T22:01:08.944 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.146+0000 7f511d7fa700 1 -- 172.21.15.16:0/561322471 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5110024840 con 0x7f5120108900 2024-03-28T22:01:08.944 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.146+0000 7f5126284700 1 -- 172.21.15.16:0/561322471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f512019f220 con 0x7f5120108900 2024-03-28T22:01:08.944 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.147+0000 7f511d7fa700 1 -- 172.21.15.16:0/561322471 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7f5110022020 con 0x7f5120108900 2024-03-28T22:01:08.944 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.147+0000 7f511d7fa700 1 --2- 172.21.15.16:0/561322471 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f5108038bf0 0x7f510803b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:08.944 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.147+0000 7f511d7fa700 1 -- 172.21.15.16:0/561322471 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f5110011e00 con 0x7f5120108900 2024-03-28T22:01:08.944 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.147+0000 7f5126284700 1 -- 172.21.15.16:0/561322471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5120063870 con 0x7f5120108900 2024-03-28T22:01:08.944 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.148+0000 7f511f7fe700 1 --2- 172.21.15.16:0/561322471 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f5108038bf0 0x7f510803b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:08.944 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.148+0000 7f511f7fe700 1 --2- 172.21.15.16:0/561322471 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f5108038bf0 0x7f510803b0b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5114006fd0 tx=0x7f5114006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:08.944 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.153+0000 7f511d7fa700 1 -- 172.21.15.16:0/561322471 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f511004c630 con 0x7f5120108900 2024-03-28T22:01:08.944 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.357+0000 7f5126284700 1 -- 172.21.15.16:0/561322471 --> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f5120103540 con 0x7f5108038bf0 2024-03-28T22:01:08.944 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.358+0000 7f511d7fa700 1 -- 172.21.15.16:0/561322471 <== mgr.14120 v2:172.21.15.16:6800/1832422779 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f5120103540 con 0x7f5108038bf0 2024-03-28T22:01:08.944 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.361+0000 7f5126284700 1 -- 172.21.15.16:0/561322471 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f5108038bf0 msgr2=0x7f510803b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:08.945 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.361+0000 7f5126284700 1 --2- 172.21.15.16:0/561322471 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f5108038bf0 0x7f510803b0b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5114006fd0 tx=0x7f5114006e40 comp rx=0 tx=0).stop 2024-03-28T22:01:08.945 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.361+0000 7f5126284700 1 -- 172.21.15.16:0/561322471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5120108900 msgr2=0x7f512019bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:08.945 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.361+0000 7f5126284700 1 --2- 172.21.15.16:0/561322471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5120108900 0x7f512019bd60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5110011b60 tx=0x7f5110011c40 comp rx=0 tx=0).stop 2024-03-28T22:01:08.945 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.361+0000 7f5126284700 1 -- 172.21.15.16:0/561322471 shutdown_connections 2024-03-28T22:01:08.945 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.361+0000 7f5126284700 1 --2- 172.21.15.16:0/561322471 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f5108038bf0 0x7f510803b0b0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:08.945 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.361+0000 7f5126284700 1 --2- 172.21.15.16:0/561322471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5120108900 0x7f512019bd60 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:08.945 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.361+0000 7f5126284700 1 -- 172.21.15.16:0/561322471 >> 172.21.15.16:0/561322471 conn(0x7f5120102310 msgr2=0x7f5120102e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:08.945 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.361+0000 7f5126284700 1 -- 172.21.15.16:0/561322471 shutdown_connections 2024-03-28T22:01:08.945 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:08.361+0000 7f5126284700 1 -- 172.21.15.16:0/561322471 wait complete. 2024-03-28T22:01:08.945 INFO:teuthology.orchestra.run.smithi016.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-28T22:01:08.945 INFO:teuthology.orchestra.run.smithi016.stdout:Adding key to root@localhost authorized_keys... 2024-03-28T22:01:08.945 INFO:teuthology.orchestra.run.smithi016.stdout:Adding host smithi016... 2024-03-28T22:01:09.711 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:09 smithi016 ceph-mon[32374]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:01:10.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:10 smithi016 ceph-mon[32374]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi016", "addr": "172.21.15.16", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:01:10.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:10 smithi016 ceph-mon[32374]: Deploying cephadm binary to smithi016 2024-03-28T22:01:14.432 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Added host 'smithi016' with addr '172.21.15.16' 2024-03-28T22:01:14.432 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.348+0000 7f9aae3f3700 1 Processor -- start 2024-03-28T22:01:14.432 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.348+0000 7f9aae3f3700 1 -- start start 2024-03-28T22:01:14.432 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.348+0000 7f9aae3f3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9aa807ddc0 0x7f9aa807c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:14.432 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.348+0000 7f9aae3f3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa807c800 con 0x7f9aa807ddc0 2024-03-28T22:01:14.432 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.349+0000 7f9aa7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9aa807ddc0 0x7f9aa807c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:14.433 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.349+0000 7f9aa7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9aa807ddc0 0x7f9aa807c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57820/0 (socket says 172.21.15.16:57820) 2024-03-28T22:01:14.433 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.349+0000 7f9aa7fff700 1 -- 172.21.15.16:0/2719813740 learned_addr learned my addr 172.21.15.16:0/2719813740 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:14.433 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.349+0000 7f9aa7fff700 1 -- 172.21.15.16:0/2719813740 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aa807c980 con 0x7f9aa807ddc0 2024-03-28T22:01:14.433 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.349+0000 7f9aa7fff700 1 --2- 172.21.15.16:0/2719813740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9aa807ddc0 0x7f9aa807c2c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9a900098a0 tx=0x7f9a90009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7763db9c9c77344 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:14.433 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.349+0000 7f9aa6ffd700 1 -- 172.21.15.16:0/2719813740 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a90004160 con 0x7f9aa807ddc0 2024-03-28T22:01:14.433 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.349+0000 7f9aa6ffd700 1 -- 172.21.15.16:0/2719813740 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9a90004300 con 0x7f9aa807ddc0 2024-03-28T22:01:14.433 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.349+0000 7f9aa6ffd700 1 -- 172.21.15.16:0/2719813740 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a90011b50 con 0x7f9aa807ddc0 2024-03-28T22:01:14.433 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.350+0000 7f9aae3f3700 1 -- 172.21.15.16:0/2719813740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9aa807ddc0 msgr2=0x7f9aa807c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:14.433 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.350+0000 7f9aae3f3700 1 --2- 172.21.15.16:0/2719813740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9aa807ddc0 0x7f9aa807c2c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9a900098a0 tx=0x7f9a90009bb0 comp rx=0 tx=0).stop 2024-03-28T22:01:14.433 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.350+0000 7f9aae3f3700 1 -- 172.21.15.16:0/2719813740 shutdown_connections 2024-03-28T22:01:14.433 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.350+0000 7f9aae3f3700 1 --2- 172.21.15.16:0/2719813740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9aa807ddc0 0x7f9aa807c2c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:14.433 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.350+0000 7f9aae3f3700 1 -- 172.21.15.16:0/2719813740 >> 172.21.15.16:0/2719813740 conn(0x7f9aa81034a0 msgr2=0x7f9aa8105900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:14.433 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.350+0000 7f9aae3f3700 1 -- 172.21.15.16:0/2719813740 shutdown_connections 2024-03-28T22:01:14.434 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.350+0000 7f9aae3f3700 1 -- 172.21.15.16:0/2719813740 wait complete. 2024-03-28T22:01:14.434 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.351+0000 7f9aae3f3700 1 Processor -- start 2024-03-28T22:01:14.434 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.351+0000 7f9aae3f3700 1 -- start start 2024-03-28T22:01:14.434 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.351+0000 7f9aae3f3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9aa807ddc0 0x7f9aa819e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:14.434 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.351+0000 7f9aae3f3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa810e490 con 0x7f9aa807ddc0 2024-03-28T22:01:14.434 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.351+0000 7f9aa7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9aa807ddc0 0x7f9aa819e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:14.434 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.351+0000 7f9aa7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9aa807ddc0 0x7f9aa819e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57828/0 (socket says 172.21.15.16:57828) 2024-03-28T22:01:14.434 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.351+0000 7f9aa7fff700 1 -- 172.21.15.16:0/549572134 learned_addr learned my addr 172.21.15.16:0/549572134 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:14.434 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.351+0000 7f9aa7fff700 1 -- 172.21.15.16:0/549572134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aa819e540 con 0x7f9aa807ddc0 2024-03-28T22:01:14.434 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.351+0000 7f9aa7fff700 1 --2- 172.21.15.16:0/549572134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9aa807ddc0 0x7f9aa819e000 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9a90011670 tx=0x7f9a90011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:14.434 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.352+0000 7f9aa57fa700 1 -- 172.21.15.16:0/549572134 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a900039c0 con 0x7f9aa807ddc0 2024-03-28T22:01:14.434 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.352+0000 7f9aa57fa700 1 -- 172.21.15.16:0/549572134 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9a90011e50 con 0x7f9aa807ddc0 2024-03-28T22:01:14.434 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.352+0000 7f9aae3f3700 1 -- 172.21.15.16:0/549572134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9aa819e7d0 con 0x7f9aa807ddc0 2024-03-28T22:01:14.435 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.352+0000 7f9aa57fa700 1 -- 172.21.15.16:0/549572134 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a9001a5c0 con 0x7f9aa807ddc0 2024-03-28T22:01:14.435 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.352+0000 7f9aae3f3700 1 -- 172.21.15.16:0/549572134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9aa81a14c0 con 0x7f9aa807ddc0 2024-03-28T22:01:14.435 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.353+0000 7f9aa57fa700 1 -- 172.21.15.16:0/549572134 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7f9a90024430 con 0x7f9aa807ddc0 2024-03-28T22:01:14.435 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.353+0000 7f9aae3f3700 1 -- 172.21.15.16:0/549572134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9aa8063870 con 0x7f9aa807ddc0 2024-03-28T22:01:14.435 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.353+0000 7f9aa57fa700 1 --2- 172.21.15.16:0/549572134 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f9a94038b50 0x7f9a9403b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:14.435 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.353+0000 7f9aa57fa700 1 -- 172.21.15.16:0/549572134 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9a9004ce50 con 0x7f9aa807ddc0 2024-03-28T22:01:14.435 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.353+0000 7f9aa77fe700 1 --2- 172.21.15.16:0/549572134 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f9a94038b50 0x7f9a9403b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:14.435 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.354+0000 7f9aa77fe700 1 --2- 172.21.15.16:0/549572134 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f9a94038b50 0x7f9a9403b010 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9a98006fd0 tx=0x7f9a98006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:14.435 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.358+0000 7f9aa57fa700 1 -- 172.21.15.16:0/549572134 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9a9004c7a0 con 0x7f9aa807ddc0 2024-03-28T22:01:14.435 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:09.569+0000 7f9aae3f3700 1 -- 172.21.15.16:0/549572134 --> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi016", "addr": "172.21.15.16", "target": ["mon-mgr", ""]}) -- 0x7f9aa81046f0 con 0x7f9a94038b50 2024-03-28T22:01:14.435 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:13.901+0000 7f9aa57fa700 1 -- 172.21.15.16:0/549572134 <== mgr.14120 v2:172.21.15.16:6800/1832422779 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f9aa81046f0 con 0x7f9a94038b50 2024-03-28T22:01:14.435 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:13.904+0000 7f9aae3f3700 1 -- 172.21.15.16:0/549572134 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f9a94038b50 msgr2=0x7f9a9403b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:14.436 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:13.904+0000 7f9aae3f3700 1 --2- 172.21.15.16:0/549572134 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f9a94038b50 0x7f9a9403b010 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9a98006fd0 tx=0x7f9a98006e40 comp rx=0 tx=0).stop 2024-03-28T22:01:14.436 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:13.904+0000 7f9aae3f3700 1 -- 172.21.15.16:0/549572134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9aa807ddc0 msgr2=0x7f9aa819e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:14.436 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:13.904+0000 7f9aae3f3700 1 --2- 172.21.15.16:0/549572134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9aa807ddc0 0x7f9aa819e000 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9a90011670 tx=0x7f9a90011b60 comp rx=0 tx=0).stop 2024-03-28T22:01:14.436 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:13.904+0000 7f9aae3f3700 1 -- 172.21.15.16:0/549572134 shutdown_connections 2024-03-28T22:01:14.436 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:13.904+0000 7f9aae3f3700 1 --2- 172.21.15.16:0/549572134 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f9a94038b50 0x7f9a9403b010 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:14.436 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:13.904+0000 7f9aae3f3700 1 --2- 172.21.15.16:0/549572134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9aa807ddc0 0x7f9aa819e000 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:14.436 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:13.904+0000 7f9aae3f3700 1 -- 172.21.15.16:0/549572134 >> 172.21.15.16:0/549572134 conn(0x7f9aa81034a0 msgr2=0x7f9aa8103fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:14.436 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:13.905+0000 7f9aae3f3700 1 -- 172.21.15.16:0/549572134 shutdown_connections 2024-03-28T22:01:14.436 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:13.905+0000 7f9aae3f3700 1 -- 172.21.15.16:0/549572134 wait complete. 2024-03-28T22:01:14.436 INFO:teuthology.orchestra.run.smithi016.stdout:Deploying unmanaged mon service... 2024-03-28T22:01:15.052 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:14 smithi016 ceph-mon[32374]: from='mgr.14120 172.21.15.16:0/3672307959' entity='mgr.x' 2024-03-28T22:01:15.052 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:14 smithi016 ceph-mon[32374]: Added host smithi016 2024-03-28T22:01:15.052 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:14 smithi016 ceph-mon[32374]: from='mgr.14120 172.21.15.16:0/3672307959' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:01:15.667 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-28T22:01:15.667 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.820+0000 7f46f99d4700 1 Processor -- start 2024-03-28T22:01:15.667 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.820+0000 7f46f99d4700 1 -- start start 2024-03-28T22:01:15.667 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.820+0000 7f46f99d4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46f4108960 0x7f46f410ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:15.667 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.820+0000 7f46f99d4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f46f407be10 con 0x7f46f4108960 2024-03-28T22:01:15.667 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.820+0000 7f46f2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46f4108960 0x7f46f410ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:15.667 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.820+0000 7f46f2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46f4108960 0x7f46f410ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47038/0 (socket says 172.21.15.16:47038) 2024-03-28T22:01:15.667 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.820+0000 7f46f2ffd700 1 -- 172.21.15.16:0/4064672443 learned_addr learned my addr 172.21.15.16:0/4064672443 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:15.667 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.821+0000 7f46f2ffd700 1 -- 172.21.15.16:0/4064672443 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46f410b290 con 0x7f46f4108960 2024-03-28T22:01:15.667 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.821+0000 7f46f2ffd700 1 --2- 172.21.15.16:0/4064672443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46f4108960 0x7f46f410ad50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f46e40098a0 tx=0x7f46e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=575530d4569e3b1b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:15.668 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.821+0000 7f46f1ffb700 1 -- 172.21.15.16:0/4064672443 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f46e4004160 con 0x7f46f4108960 2024-03-28T22:01:15.668 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.821+0000 7f46f1ffb700 1 -- 172.21.15.16:0/4064672443 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f46e4004300 con 0x7f46f4108960 2024-03-28T22:01:15.668 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.821+0000 7f46f1ffb700 1 -- 172.21.15.16:0/4064672443 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f46e4011b50 con 0x7f46f4108960 2024-03-28T22:01:15.668 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.821+0000 7f46f99d4700 1 -- 172.21.15.16:0/4064672443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46f4108960 msgr2=0x7f46f410ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:15.668 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.821+0000 7f46f99d4700 1 --2- 172.21.15.16:0/4064672443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46f4108960 0x7f46f410ad50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f46e40098a0 tx=0x7f46e4009bb0 comp rx=0 tx=0).stop 2024-03-28T22:01:15.668 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.822+0000 7f46f99d4700 1 -- 172.21.15.16:0/4064672443 shutdown_connections 2024-03-28T22:01:15.668 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.822+0000 7f46f99d4700 1 --2- 172.21.15.16:0/4064672443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46f4108960 0x7f46f410ad50 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:15.668 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.822+0000 7f46f99d4700 1 -- 172.21.15.16:0/4064672443 >> 172.21.15.16:0/4064672443 conn(0x7f46f4102310 msgr2=0x7f46f4104750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:15.668 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.822+0000 7f46f99d4700 1 -- 172.21.15.16:0/4064672443 shutdown_connections 2024-03-28T22:01:15.668 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.822+0000 7f46f99d4700 1 -- 172.21.15.16:0/4064672443 wait complete. 2024-03-28T22:01:15.668 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.822+0000 7f46f99d4700 1 Processor -- start 2024-03-28T22:01:15.668 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.822+0000 7f46f99d4700 1 -- start start 2024-03-28T22:01:15.668 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.822+0000 7f46f99d4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46f4108960 0x7f46f419bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:15.669 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.822+0000 7f46f99d4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f46f410c250 con 0x7f46f4108960 2024-03-28T22:01:15.669 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.823+0000 7f46f2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46f4108960 0x7f46f419bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:15.669 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.823+0000 7f46f2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46f4108960 0x7f46f419bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47042/0 (socket says 172.21.15.16:47042) 2024-03-28T22:01:15.669 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.823+0000 7f46f2ffd700 1 -- 172.21.15.16:0/3957863831 learned_addr learned my addr 172.21.15.16:0/3957863831 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:15.669 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.823+0000 7f46f2ffd700 1 -- 172.21.15.16:0/3957863831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46f419c330 con 0x7f46f4108960 2024-03-28T22:01:15.669 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.823+0000 7f46f2ffd700 1 --2- 172.21.15.16:0/3957863831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46f4108960 0x7f46f419bdf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f46e4011b60 tx=0x7f46e4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:15.669 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.823+0000 7f46f89d2700 1 -- 172.21.15.16:0/3957863831 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f46e4003b20 con 0x7f46f4108960 2024-03-28T22:01:15.669 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.823+0000 7f46f89d2700 1 -- 172.21.15.16:0/3957863831 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f46e4003cc0 con 0x7f46f4108960 2024-03-28T22:01:15.669 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.823+0000 7f46f99d4700 1 -- 172.21.15.16:0/3957863831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46f419c5c0 con 0x7f46f4108960 2024-03-28T22:01:15.669 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.823+0000 7f46f89d2700 1 -- 172.21.15.16:0/3957863831 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f46e4024840 con 0x7f46f4108960 2024-03-28T22:01:15.669 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.823+0000 7f46f99d4700 1 -- 172.21.15.16:0/3957863831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46f419f2b0 con 0x7f46f4108960 2024-03-28T22:01:15.669 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.824+0000 7f46f89d2700 1 -- 172.21.15.16:0/3957863831 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7f46e4022020 con 0x7f46f4108960 2024-03-28T22:01:15.669 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.824+0000 7f46f99d4700 1 -- 172.21.15.16:0/3957863831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46f4063870 con 0x7f46f4108960 2024-03-28T22:01:15.669 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.824+0000 7f46f89d2700 1 --2- 172.21.15.16:0/3957863831 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f46dc038ba0 0x7f46dc03b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:15.670 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.824+0000 7f46f89d2700 1 -- 172.21.15.16:0/3957863831 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f46e4011e00 con 0x7f46f4108960 2024-03-28T22:01:15.670 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.825+0000 7f46f27fc700 1 --2- 172.21.15.16:0/3957863831 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f46dc038ba0 0x7f46dc03b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:15.670 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.825+0000 7f46f27fc700 1 --2- 172.21.15.16:0/3957863831 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f46dc038ba0 0x7f46dc03b060 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f46e8006fd0 tx=0x7f46e8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:15.670 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:14.829+0000 7f46f89d2700 1 -- 172.21.15.16:0/3957863831 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f46e402d450 con 0x7f46f4108960 2024-03-28T22:01:15.670 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:15.046+0000 7f46f99d4700 1 -- 172.21.15.16:0/3957863831 --> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f46f4103560 con 0x7f46dc038ba0 2024-03-28T22:01:15.670 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:15.050+0000 7f46f89d2700 1 -- 172.21.15.16:0/3957863831 <== mgr.14120 v2:172.21.15.16:6800/1832422779 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f46f4103560 con 0x7f46dc038ba0 2024-03-28T22:01:15.670 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:15.052+0000 7f46f99d4700 1 -- 172.21.15.16:0/3957863831 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f46dc038ba0 msgr2=0x7f46dc03b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:15.670 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:15.052+0000 7f46f99d4700 1 --2- 172.21.15.16:0/3957863831 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f46dc038ba0 0x7f46dc03b060 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f46e8006fd0 tx=0x7f46e8006e40 comp rx=0 tx=0).stop 2024-03-28T22:01:15.670 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:15.052+0000 7f46f99d4700 1 -- 172.21.15.16:0/3957863831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46f4108960 msgr2=0x7f46f419bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:15.670 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:15.052+0000 7f46f99d4700 1 --2- 172.21.15.16:0/3957863831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46f4108960 0x7f46f419bdf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f46e4011b60 tx=0x7f46e4011c40 comp rx=0 tx=0).stop 2024-03-28T22:01:15.670 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:15.052+0000 7f46f99d4700 1 -- 172.21.15.16:0/3957863831 shutdown_connections 2024-03-28T22:01:15.670 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:15.052+0000 7f46f99d4700 1 --2- 172.21.15.16:0/3957863831 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f46dc038ba0 0x7f46dc03b060 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:15.671 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:15.052+0000 7f46f99d4700 1 --2- 172.21.15.16:0/3957863831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46f4108960 0x7f46f419bdf0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:15.671 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:15.052+0000 7f46f99d4700 1 -- 172.21.15.16:0/3957863831 >> 172.21.15.16:0/3957863831 conn(0x7f46f4102310 msgr2=0x7f46f4102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:15.671 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:15.053+0000 7f46f99d4700 1 -- 172.21.15.16:0/3957863831 shutdown_connections 2024-03-28T22:01:15.671 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:15.053+0000 7f46f99d4700 1 -- 172.21.15.16:0/3957863831 wait complete. 2024-03-28T22:01:15.671 INFO:teuthology.orchestra.run.smithi016.stdout:Deploying unmanaged mgr service... 2024-03-28T22:01:16.269 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:16 smithi016 ceph-mon[32374]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:01:16.270 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:16 smithi016 ceph-mon[32374]: Saving service mon spec with placement count:5 2024-03-28T22:01:16.270 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:16 smithi016 ceph-mon[32374]: from='mgr.14120 172.21.15.16:0/3672307959' entity='mgr.x' 2024-03-28T22:01:17.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-28T22:01:17.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.088+0000 7f39d592c700 1 Processor -- start 2024-03-28T22:01:17.284 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.088+0000 7f39d592c700 1 -- start start 2024-03-28T22:01:17.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.088+0000 7f39d592c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39d0107f20 0x7f39d0108340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:17.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.088+0000 7f39d592c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f39d0108880 con 0x7f39d0107f20 2024-03-28T22:01:17.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.088+0000 7f39ceffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39d0107f20 0x7f39d0108340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:17.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.088+0000 7f39ceffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39d0107f20 0x7f39d0108340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47044/0 (socket says 172.21.15.16:47044) 2024-03-28T22:01:17.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.089+0000 7f39ceffd700 1 -- 172.21.15.16:0/3338238574 learned_addr learned my addr 172.21.15.16:0/3338238574 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:17.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.089+0000 7f39ceffd700 1 -- 172.21.15.16:0/3338238574 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39d0108fe0 con 0x7f39d0107f20 2024-03-28T22:01:17.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.089+0000 7f39ceffd700 1 --2- 172.21.15.16:0/3338238574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39d0107f20 0x7f39d0108340 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f39c8009e80 tx=0x7f39c800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c9b7055af6c94fc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:17.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.089+0000 7f39cdffb700 1 -- 172.21.15.16:0/3338238574 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f39c8017070 con 0x7f39d0107f20 2024-03-28T22:01:17.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.089+0000 7f39cdffb700 1 -- 172.21.15.16:0/3338238574 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f39c800b8c0 con 0x7f39d0107f20 2024-03-28T22:01:17.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.089+0000 7f39cdffb700 1 -- 172.21.15.16:0/3338238574 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f39c80045c0 con 0x7f39d0107f20 2024-03-28T22:01:17.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.090+0000 7f39d592c700 1 -- 172.21.15.16:0/3338238574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39d0107f20 msgr2=0x7f39d0108340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:17.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.090+0000 7f39d592c700 1 --2- 172.21.15.16:0/3338238574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39d0107f20 0x7f39d0108340 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f39c8009e80 tx=0x7f39c800b330 comp rx=0 tx=0).stop 2024-03-28T22:01:17.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.090+0000 7f39d592c700 1 -- 172.21.15.16:0/3338238574 shutdown_connections 2024-03-28T22:01:17.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.090+0000 7f39d592c700 1 --2- 172.21.15.16:0/3338238574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39d0107f20 0x7f39d0108340 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:17.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.090+0000 7f39d592c700 1 -- 172.21.15.16:0/3338238574 >> 172.21.15.16:0/3338238574 conn(0x7f39d01034a0 msgr2=0x7f39d0105900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:17.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.090+0000 7f39d592c700 1 -- 172.21.15.16:0/3338238574 shutdown_connections 2024-03-28T22:01:17.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.090+0000 7f39d592c700 1 -- 172.21.15.16:0/3338238574 wait complete. 2024-03-28T22:01:17.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.090+0000 7f39d592c700 1 Processor -- start 2024-03-28T22:01:17.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.091+0000 7f39d592c700 1 -- start start 2024-03-28T22:01:17.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.091+0000 7f39d592c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39d0107f20 0x7f39d007db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:17.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.091+0000 7f39d592c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f39d010c420 con 0x7f39d0107f20 2024-03-28T22:01:17.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.091+0000 7f39ceffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39d0107f20 0x7f39d007db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:17.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.091+0000 7f39ceffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39d0107f20 0x7f39d007db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47052/0 (socket says 172.21.15.16:47052) 2024-03-28T22:01:17.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.091+0000 7f39ceffd700 1 -- 172.21.15.16:0/2633492903 learned_addr learned my addr 172.21.15.16:0/2633492903 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:17.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.091+0000 7f39ceffd700 1 -- 172.21.15.16:0/2633492903 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39d007e090 con 0x7f39d0107f20 2024-03-28T22:01:17.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.091+0000 7f39ceffd700 1 --2- 172.21.15.16:0/2633492903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39d0107f20 0x7f39d007db50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f39c8004350 tx=0x7f39c800bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:17.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.091+0000 7f39d492a700 1 -- 172.21.15.16:0/2633492903 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f39c8020020 con 0x7f39d0107f20 2024-03-28T22:01:17.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.092+0000 7f39d492a700 1 -- 172.21.15.16:0/2633492903 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f39c8003c10 con 0x7f39d0107f20 2024-03-28T22:01:17.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.092+0000 7f39d592c700 1 -- 172.21.15.16:0/2633492903 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39d007c280 con 0x7f39d0107f20 2024-03-28T22:01:17.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.092+0000 7f39d492a700 1 -- 172.21.15.16:0/2633492903 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f39c8017040 con 0x7f39d0107f20 2024-03-28T22:01:17.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.092+0000 7f39d592c700 1 -- 172.21.15.16:0/2633492903 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39d007c7c0 con 0x7f39d0107f20 2024-03-28T22:01:17.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.092+0000 7f39d492a700 1 -- 172.21.15.16:0/2633492903 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7f39c8023020 con 0x7f39d0107f20 2024-03-28T22:01:17.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.093+0000 7f39d492a700 1 --2- 172.21.15.16:0/2633492903 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f39bc038ba0 0x7f39bc03b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:17.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.093+0000 7f39d492a700 1 -- 172.21.15.16:0/2633492903 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f39c800f050 con 0x7f39d0107f20 2024-03-28T22:01:17.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.093+0000 7f39d592c700 1 -- 172.21.15.16:0/2633492903 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39d0063870 con 0x7f39d0107f20 2024-03-28T22:01:17.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.093+0000 7f39ce7fc700 1 --2- 172.21.15.16:0/2633492903 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f39bc038ba0 0x7f39bc03b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:17.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.093+0000 7f39ce7fc700 1 --2- 172.21.15.16:0/2633492903 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f39bc038ba0 0x7f39bc03b060 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f39b8006fd0 tx=0x7f39b8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:17.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.098+0000 7f39d492a700 1 -- 172.21.15.16:0/2633492903 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f39c802a020 con 0x7f39d0107f20 2024-03-28T22:01:17.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.317+0000 7f39d592c700 1 -- 172.21.15.16:0/2633492903 --> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f39d007cef0 con 0x7f39bc038ba0 2024-03-28T22:01:17.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.321+0000 7f39d492a700 1 -- 172.21.15.16:0/2633492903 <== mgr.14120 v2:172.21.15.16:6800/1832422779 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f39d007cef0 con 0x7f39bc038ba0 2024-03-28T22:01:17.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.323+0000 7f39d592c700 1 -- 172.21.15.16:0/2633492903 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f39bc038ba0 msgr2=0x7f39bc03b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:17.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.323+0000 7f39d592c700 1 --2- 172.21.15.16:0/2633492903 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f39bc038ba0 0x7f39bc03b060 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f39b8006fd0 tx=0x7f39b8006e40 comp rx=0 tx=0).stop 2024-03-28T22:01:17.288 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.323+0000 7f39d592c700 1 -- 172.21.15.16:0/2633492903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39d0107f20 msgr2=0x7f39d007db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:17.288 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.323+0000 7f39d592c700 1 --2- 172.21.15.16:0/2633492903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39d0107f20 0x7f39d007db50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f39c8004350 tx=0x7f39c800bd50 comp rx=0 tx=0).stop 2024-03-28T22:01:17.288 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.323+0000 7f39d592c700 1 -- 172.21.15.16:0/2633492903 shutdown_connections 2024-03-28T22:01:17.288 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.323+0000 7f39d592c700 1 --2- 172.21.15.16:0/2633492903 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f39bc038ba0 0x7f39bc03b060 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:17.288 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.323+0000 7f39d592c700 1 --2- 172.21.15.16:0/2633492903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39d0107f20 0x7f39d007db50 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:17.288 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.323+0000 7f39d592c700 1 -- 172.21.15.16:0/2633492903 >> 172.21.15.16:0/2633492903 conn(0x7f39d01034a0 msgr2=0x7f39d0103fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:17.288 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.324+0000 7f39d592c700 1 -- 172.21.15.16:0/2633492903 shutdown_connections 2024-03-28T22:01:17.288 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:16.324+0000 7f39d592c700 1 -- 172.21.15.16:0/2633492903 wait complete. 2024-03-28T22:01:17.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:17 smithi016 ceph-mon[32374]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:01:17.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:17 smithi016 ceph-mon[32374]: Saving service mgr spec with placement count:2 2024-03-28T22:01:17.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:17 smithi016 ceph-mon[32374]: from='mgr.14120 172.21.15.16:0/3672307959' entity='mgr.x' 2024-03-28T22:01:18.753 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.710+0000 7f977c6c6700 1 Processor -- start 2024-03-28T22:01:18.753 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.710+0000 7f977c6c6700 1 -- start start 2024-03-28T22:01:18.753 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.710+0000 7f977c6c6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97741088e0 0x7f977410acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:18.753 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.710+0000 7f977c6c6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f977407be90 con 0x7f97741088e0 2024-03-28T22:01:18.753 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.710+0000 7f977a462700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97741088e0 0x7f977410acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:18.753 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.710+0000 7f977a462700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97741088e0 0x7f977410acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47068/0 (socket says 172.21.15.16:47068) 2024-03-28T22:01:18.754 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.710+0000 7f977a462700 1 -- 172.21.15.16:0/1584936127 learned_addr learned my addr 172.21.15.16:0/1584936127 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:18.754 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.710+0000 7f977a462700 1 -- 172.21.15.16:0/1584936127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f977410b210 con 0x7f97741088e0 2024-03-28T22:01:18.754 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.711+0000 7f977a462700 1 --2- 172.21.15.16:0/1584936127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97741088e0 0x7f977410acd0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f97700098a0 tx=0x7f9770009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afa2f72ba999766c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:18.754 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.711+0000 7f9779460700 1 -- 172.21.15.16:0/1584936127 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9770004160 con 0x7f97741088e0 2024-03-28T22:01:18.754 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.711+0000 7f9779460700 1 -- 172.21.15.16:0/1584936127 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9770004300 con 0x7f97741088e0 2024-03-28T22:01:18.754 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.711+0000 7f9779460700 1 -- 172.21.15.16:0/1584936127 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9770011b50 con 0x7f97741088e0 2024-03-28T22:01:18.754 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.711+0000 7f977c6c6700 1 -- 172.21.15.16:0/1584936127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97741088e0 msgr2=0x7f977410acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:18.754 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.711+0000 7f977c6c6700 1 --2- 172.21.15.16:0/1584936127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97741088e0 0x7f977410acd0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f97700098a0 tx=0x7f9770009bb0 comp rx=0 tx=0).stop 2024-03-28T22:01:18.754 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.711+0000 7f977c6c6700 1 -- 172.21.15.16:0/1584936127 shutdown_connections 2024-03-28T22:01:18.754 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.711+0000 7f977c6c6700 1 --2- 172.21.15.16:0/1584936127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97741088e0 0x7f977410acd0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:18.754 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.711+0000 7f977c6c6700 1 -- 172.21.15.16:0/1584936127 >> 172.21.15.16:0/1584936127 conn(0x7f9774102310 msgr2=0x7f9774104750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:18.754 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.711+0000 7f977c6c6700 1 -- 172.21.15.16:0/1584936127 shutdown_connections 2024-03-28T22:01:18.755 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.711+0000 7f977c6c6700 1 -- 172.21.15.16:0/1584936127 wait complete. 2024-03-28T22:01:18.755 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.712+0000 7f977c6c6700 1 Processor -- start 2024-03-28T22:01:18.755 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.712+0000 7f977c6c6700 1 -- start start 2024-03-28T22:01:18.755 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.712+0000 7f977c6c6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97741088e0 0x7f977419bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:18.755 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.712+0000 7f977c6c6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f977410c240 con 0x7f97741088e0 2024-03-28T22:01:18.755 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.712+0000 7f977a462700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97741088e0 0x7f977419bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:18.755 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.712+0000 7f977a462700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97741088e0 0x7f977419bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47070/0 (socket says 172.21.15.16:47070) 2024-03-28T22:01:18.755 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.712+0000 7f977a462700 1 -- 172.21.15.16:0/3508058724 learned_addr learned my addr 172.21.15.16:0/3508058724 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:18.755 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.713+0000 7f977a462700 1 -- 172.21.15.16:0/3508058724 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f977419c320 con 0x7f97741088e0 2024-03-28T22:01:18.755 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.713+0000 7f977a462700 1 --2- 172.21.15.16:0/3508058724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97741088e0 0x7f977419bde0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9770011670 tx=0x7f9770011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:18.755 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.713+0000 7f976b7fe700 1 -- 172.21.15.16:0/3508058724 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f97700039c0 con 0x7f97741088e0 2024-03-28T22:01:18.755 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.713+0000 7f976b7fe700 1 -- 172.21.15.16:0/3508058724 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9770011e50 con 0x7f97741088e0 2024-03-28T22:01:18.755 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.713+0000 7f977c6c6700 1 -- 172.21.15.16:0/3508058724 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f977419c5b0 con 0x7f97741088e0 2024-03-28T22:01:18.755 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.713+0000 7f977c6c6700 1 -- 172.21.15.16:0/3508058724 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f977419f2a0 con 0x7f97741088e0 2024-03-28T22:01:18.756 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.713+0000 7f976b7fe700 1 -- 172.21.15.16:0/3508058724 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f977001a5c0 con 0x7f97741088e0 2024-03-28T22:01:18.756 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.714+0000 7f976b7fe700 1 -- 172.21.15.16:0/3508058724 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7f977001a870 con 0x7f97741088e0 2024-03-28T22:01:18.756 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.714+0000 7f977c6c6700 1 -- 172.21.15.16:0/3508058724 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9774063870 con 0x7f97741088e0 2024-03-28T22:01:18.756 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.714+0000 7f976b7fe700 1 --2- 172.21.15.16:0/3508058724 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f9760038ba0 0x7f976003b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:18.756 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.714+0000 7f976b7fe700 1 -- 172.21.15.16:0/3508058724 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f977004cf80 con 0x7f97741088e0 2024-03-28T22:01:18.756 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.714+0000 7f9779c61700 1 --2- 172.21.15.16:0/3508058724 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f9760038ba0 0x7f976003b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:18.756 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.715+0000 7f9779c61700 1 --2- 172.21.15.16:0/3508058724 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f9760038ba0 0x7f976003b060 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9764006fd0 tx=0x7f9764006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:18.756 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.719+0000 7f976b7fe700 1 -- 172.21.15.16:0/3508058724 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9770019c70 con 0x7f97741088e0 2024-03-28T22:01:18.756 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.920+0000 7f977c6c6700 1 -- 172.21.15.16:0/3508058724 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f977419f8f0 con 0x7f97741088e0 2024-03-28T22:01:18.756 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.923+0000 7f976b7fe700 1 -- 172.21.15.16:0/3508058724 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v6)=0 v6) ==== 142+0+0 (secure 0 0 0) 0x7f9770003b60 con 0x7f97741088e0 2024-03-28T22:01:18.756 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.925+0000 7f977c6c6700 1 -- 172.21.15.16:0/3508058724 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f9760038ba0 msgr2=0x7f976003b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:18.756 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.925+0000 7f977c6c6700 1 --2- 172.21.15.16:0/3508058724 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f9760038ba0 0x7f976003b060 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9764006fd0 tx=0x7f9764006e40 comp rx=0 tx=0).stop 2024-03-28T22:01:18.756 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.925+0000 7f977c6c6700 1 -- 172.21.15.16:0/3508058724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97741088e0 msgr2=0x7f977419bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:18.756 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.925+0000 7f977c6c6700 1 --2- 172.21.15.16:0/3508058724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97741088e0 0x7f977419bde0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9770011670 tx=0x7f9770011b60 comp rx=0 tx=0).stop 2024-03-28T22:01:18.757 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.925+0000 7f977c6c6700 1 -- 172.21.15.16:0/3508058724 shutdown_connections 2024-03-28T22:01:18.757 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.925+0000 7f977c6c6700 1 --2- 172.21.15.16:0/3508058724 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f9760038ba0 0x7f976003b060 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:18.757 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.925+0000 7f977c6c6700 1 --2- 172.21.15.16:0/3508058724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97741088e0 0x7f977419bde0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:18.757 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.925+0000 7f977c6c6700 1 -- 172.21.15.16:0/3508058724 >> 172.21.15.16:0/3508058724 conn(0x7f9774102310 msgr2=0x7f9774102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:18.757 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.925+0000 7f977c6c6700 1 -- 172.21.15.16:0/3508058724 shutdown_connections 2024-03-28T22:01:18.757 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:17.925+0000 7f977c6c6700 1 -- 172.21.15.16:0/3508058724 wait complete. 2024-03-28T22:01:19.023 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:18 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3508058724' entity='client.admin' 2024-03-28T22:01:19.023 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:18 smithi016 ceph-mon[32374]: from='mgr.14120 172.21.15.16:0/3672307959' entity='mgr.x' 2024-03-28T22:01:19.023 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:18 smithi016 ceph-mon[32374]: from='mgr.14120 172.21.15.16:0/3672307959' entity='mgr.x' 2024-03-28T22:01:20.252 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.216+0000 7f1dfa4eb700 1 Processor -- start 2024-03-28T22:01:20.252 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.216+0000 7f1dfa4eb700 1 -- start start 2024-03-28T22:01:20.252 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.216+0000 7f1dfa4eb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1df4107ea0 0x7f1df41082c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:20.252 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.216+0000 7f1dfa4eb700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1df4108800 con 0x7f1df4107ea0 2024-03-28T22:01:20.252 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.217+0000 7f1df3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1df4107ea0 0x7f1df41082c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:20.252 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.217+0000 7f1df3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1df4107ea0 0x7f1df41082c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47080/0 (socket says 172.21.15.16:47080) 2024-03-28T22:01:20.253 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.217+0000 7f1df3fff700 1 -- 172.21.15.16:0/2851012605 learned_addr learned my addr 172.21.15.16:0/2851012605 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:20.253 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.217+0000 7f1df3fff700 1 -- 172.21.15.16:0/2851012605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1df4108980 con 0x7f1df4107ea0 2024-03-28T22:01:20.253 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.217+0000 7f1df3fff700 1 --2- 172.21.15.16:0/2851012605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1df4107ea0 0x7f1df41082c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc0098a0 tx=0x7f1ddc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a434b1176ddcab45 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:20.253 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.217+0000 7f1df2ffd700 1 -- 172.21.15.16:0/2851012605 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1ddc004160 con 0x7f1df4107ea0 2024-03-28T22:01:20.253 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.217+0000 7f1df2ffd700 1 -- 172.21.15.16:0/2851012605 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ddc004300 con 0x7f1df4107ea0 2024-03-28T22:01:20.253 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.217+0000 7f1df2ffd700 1 -- 172.21.15.16:0/2851012605 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1ddc011b20 con 0x7f1df4107ea0 2024-03-28T22:01:20.253 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.218+0000 7f1dfa4eb700 1 -- 172.21.15.16:0/2851012605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1df4107ea0 msgr2=0x7f1df41082c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:20.253 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.218+0000 7f1dfa4eb700 1 --2- 172.21.15.16:0/2851012605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1df4107ea0 0x7f1df41082c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc0098a0 tx=0x7f1ddc009bb0 comp rx=0 tx=0).stop 2024-03-28T22:01:20.253 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.218+0000 7f1dfa4eb700 1 -- 172.21.15.16:0/2851012605 shutdown_connections 2024-03-28T22:01:20.253 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.218+0000 7f1dfa4eb700 1 --2- 172.21.15.16:0/2851012605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1df4107ea0 0x7f1df41082c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:20.253 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.218+0000 7f1dfa4eb700 1 -- 172.21.15.16:0/2851012605 >> 172.21.15.16:0/2851012605 conn(0x7f1df4103500 msgr2=0x7f1df4105920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:20.253 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.218+0000 7f1dfa4eb700 1 -- 172.21.15.16:0/2851012605 shutdown_connections 2024-03-28T22:01:20.253 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.218+0000 7f1dfa4eb700 1 -- 172.21.15.16:0/2851012605 wait complete. 2024-03-28T22:01:20.254 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.219+0000 7f1dfa4eb700 1 Processor -- start 2024-03-28T22:01:20.254 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.219+0000 7f1dfa4eb700 1 -- start start 2024-03-28T22:01:20.254 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.220+0000 7f1dfa4eb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1df4107ea0 0x7f1df419bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:20.254 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.220+0000 7f1dfa4eb700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1df410c310 con 0x7f1df4107ea0 2024-03-28T22:01:20.254 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.220+0000 7f1df3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1df4107ea0 0x7f1df419bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:20.254 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.220+0000 7f1df3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1df4107ea0 0x7f1df419bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47096/0 (socket says 172.21.15.16:47096) 2024-03-28T22:01:20.254 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.220+0000 7f1df3fff700 1 -- 172.21.15.16:0/965281165 learned_addr learned my addr 172.21.15.16:0/965281165 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:20.254 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.220+0000 7f1df3fff700 1 -- 172.21.15.16:0/965281165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1df419c290 con 0x7f1df4107ea0 2024-03-28T22:01:20.254 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.220+0000 7f1df3fff700 1 --2- 172.21.15.16:0/965281165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1df4107ea0 0x7f1df419bd50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc0117e0 tx=0x7f1ddc011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:20.255 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.220+0000 7f1df17fa700 1 -- 172.21.15.16:0/965281165 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1ddc003bf0 con 0x7f1df4107ea0 2024-03-28T22:01:20.255 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.220+0000 7f1df17fa700 1 -- 172.21.15.16:0/965281165 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ddc01a910 con 0x7f1df4107ea0 2024-03-28T22:01:20.255 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.220+0000 7f1dfa4eb700 1 -- 172.21.15.16:0/965281165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1df419c580 con 0x7f1df4107ea0 2024-03-28T22:01:20.255 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.220+0000 7f1df17fa700 1 -- 172.21.15.16:0/965281165 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1ddc024970 con 0x7f1df4107ea0 2024-03-28T22:01:20.255 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.220+0000 7f1dfa4eb700 1 -- 172.21.15.16:0/965281165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1df419f210 con 0x7f1df4107ea0 2024-03-28T22:01:20.255 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.221+0000 7f1df17fa700 1 -- 172.21.15.16:0/965281165 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7f1ddc022020 con 0x7f1df4107ea0 2024-03-28T22:01:20.255 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.221+0000 7f1df17fa700 1 --2- 172.21.15.16:0/965281165 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f1de0038ba0 0x7f1de003b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:20.255 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.221+0000 7f1df17fa700 1 -- 172.21.15.16:0/965281165 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f1ddc04d170 con 0x7f1df4107ea0 2024-03-28T22:01:20.255 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.221+0000 7f1dfa4eb700 1 -- 172.21.15.16:0/965281165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1df4063870 con 0x7f1df4107ea0 2024-03-28T22:01:20.255 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.222+0000 7f1df37fe700 1 --2- 172.21.15.16:0/965281165 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f1de0038ba0 0x7f1de003b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:20.255 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.222+0000 7f1df37fe700 1 --2- 172.21.15.16:0/965281165 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f1de0038ba0 0x7f1de003b060 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1de4006fd0 tx=0x7f1de4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:20.255 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.231+0000 7f1df17fa700 1 -- 172.21.15.16:0/965281165 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1ddc01a430 con 0x7f1df4107ea0 2024-03-28T22:01:20.255 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.458+0000 7f1dfa4eb700 1 -- 172.21.15.16:0/965281165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f1df419f3e0 con 0x7f1df4107ea0 2024-03-28T22:01:20.255 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.460+0000 7f1df17fa700 1 -- 172.21.15.16:0/965281165 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f1ddc0320c0 con 0x7f1df4107ea0 2024-03-28T22:01:20.255 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.462+0000 7f1dfa4eb700 1 -- 172.21.15.16:0/965281165 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f1de0038ba0 msgr2=0x7f1de003b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:20.256 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.462+0000 7f1dfa4eb700 1 --2- 172.21.15.16:0/965281165 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f1de0038ba0 0x7f1de003b060 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1de4006fd0 tx=0x7f1de4006e40 comp rx=0 tx=0).stop 2024-03-28T22:01:20.256 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.463+0000 7f1dfa4eb700 1 -- 172.21.15.16:0/965281165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1df4107ea0 msgr2=0x7f1df419bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:20.256 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.463+0000 7f1dfa4eb700 1 --2- 172.21.15.16:0/965281165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1df4107ea0 0x7f1df419bd50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc0117e0 tx=0x7f1ddc011ba0 comp rx=0 tx=0).stop 2024-03-28T22:01:20.256 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.463+0000 7f1dfa4eb700 1 -- 172.21.15.16:0/965281165 shutdown_connections 2024-03-28T22:01:20.256 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.463+0000 7f1dfa4eb700 1 --2- 172.21.15.16:0/965281165 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f1de0038ba0 0x7f1de003b060 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:20.256 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.463+0000 7f1dfa4eb700 1 --2- 172.21.15.16:0/965281165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1df4107ea0 0x7f1df419bd50 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:20.256 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.463+0000 7f1dfa4eb700 1 -- 172.21.15.16:0/965281165 >> 172.21.15.16:0/965281165 conn(0x7f1df4103500 msgr2=0x7f1df4103f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:20.256 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.463+0000 7f1dfa4eb700 1 -- 172.21.15.16:0/965281165 shutdown_connections 2024-03-28T22:01:20.256 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:19.463+0000 7f1dfa4eb700 1 -- 172.21.15.16:0/965281165 wait complete. 2024-03-28T22:01:20.256 INFO:teuthology.orchestra.run.smithi016.stdout:Enabling the dashboard module... 2024-03-28T22:01:20.504 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:20 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/965281165' entity='client.admin' 2024-03-28T22:01:22.270 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:21 smithi016 ceph-mon[32374]: from='mgr.14120 172.21.15.16:0/3672307959' entity='mgr.x' 2024-03-28T22:01:22.270 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:21 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/4117335387' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-28T22:01:22.270 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:22 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: ignoring --setuser ceph since I am not root 2024-03-28T22:01:22.270 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:22 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: ignoring --setgroup ceph since I am not root 2024-03-28T22:01:22.570 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.716+0000 7f6c20579700 1 Processor -- start 2024-03-28T22:01:22.570 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.716+0000 7f6c20579700 1 -- start start 2024-03-28T22:01:22.570 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.717+0000 7f6c20579700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c18107f20 0x7f6c18108340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:22.570 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.717+0000 7f6c20579700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6c18108880 con 0x7f6c18107f20 2024-03-28T22:01:22.570 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.717+0000 7f6c1e315700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c18107f20 0x7f6c18108340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:22.570 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.717+0000 7f6c1e315700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c18107f20 0x7f6c18108340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55716/0 (socket says 172.21.15.16:55716) 2024-03-28T22:01:22.571 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.717+0000 7f6c1e315700 1 -- 172.21.15.16:0/2177516390 learned_addr learned my addr 172.21.15.16:0/2177516390 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:22.571 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.717+0000 7f6c1e315700 1 -- 172.21.15.16:0/2177516390 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c18108fe0 con 0x7f6c18107f20 2024-03-28T22:01:22.571 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.717+0000 7f6c1e315700 1 --2- 172.21.15.16:0/2177516390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c18107f20 0x7f6c18108340 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6c140098a0 tx=0x7f6c14009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7224a61fbf9c2294 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:22.571 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.717+0000 7f6c1d313700 1 -- 172.21.15.16:0/2177516390 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c14004160 con 0x7f6c18107f20 2024-03-28T22:01:22.571 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.718+0000 7f6c1d313700 1 -- 172.21.15.16:0/2177516390 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c14004300 con 0x7f6c18107f20 2024-03-28T22:01:22.571 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.718+0000 7f6c1d313700 1 -- 172.21.15.16:0/2177516390 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c14011b20 con 0x7f6c18107f20 2024-03-28T22:01:22.571 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.718+0000 7f6c20579700 1 -- 172.21.15.16:0/2177516390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c18107f20 msgr2=0x7f6c18108340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:22.571 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.718+0000 7f6c20579700 1 --2- 172.21.15.16:0/2177516390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c18107f20 0x7f6c18108340 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6c140098a0 tx=0x7f6c14009bb0 comp rx=0 tx=0).stop 2024-03-28T22:01:22.571 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.718+0000 7f6c20579700 1 -- 172.21.15.16:0/2177516390 shutdown_connections 2024-03-28T22:01:22.571 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.718+0000 7f6c20579700 1 --2- 172.21.15.16:0/2177516390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c18107f20 0x7f6c18108340 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:22.571 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.718+0000 7f6c20579700 1 -- 172.21.15.16:0/2177516390 >> 172.21.15.16:0/2177516390 conn(0x7f6c181034a0 msgr2=0x7f6c18105900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:22.571 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.718+0000 7f6c20579700 1 -- 172.21.15.16:0/2177516390 shutdown_connections 2024-03-28T22:01:22.571 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.718+0000 7f6c20579700 1 -- 172.21.15.16:0/2177516390 wait complete. 2024-03-28T22:01:22.571 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.719+0000 7f6c20579700 1 Processor -- start 2024-03-28T22:01:22.571 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.719+0000 7f6c20579700 1 -- start start 2024-03-28T22:01:22.572 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.719+0000 7f6c20579700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c18107f20 0x7f6c1807db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:22.572 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.719+0000 7f6c20579700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1810c420 con 0x7f6c18107f20 2024-03-28T22:01:22.572 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.719+0000 7f6c1e315700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c18107f20 0x7f6c1807db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:22.572 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.719+0000 7f6c1e315700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c18107f20 0x7f6c1807db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55722/0 (socket says 172.21.15.16:55722) 2024-03-28T22:01:22.572 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.719+0000 7f6c1e315700 1 -- 172.21.15.16:0/4117335387 learned_addr learned my addr 172.21.15.16:0/4117335387 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:22.572 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.719+0000 7f6c1e315700 1 -- 172.21.15.16:0/4117335387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c1807e090 con 0x7f6c18107f20 2024-03-28T22:01:22.572 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.719+0000 7f6c1e315700 1 --2- 172.21.15.16:0/4117335387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c18107f20 0x7f6c1807db50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6c140117e0 tx=0x7f6c14011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:22.572 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.719+0000 7f6c0f7fe700 1 -- 172.21.15.16:0/4117335387 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c14003ba0 con 0x7f6c18107f20 2024-03-28T22:01:22.572 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.720+0000 7f6c20579700 1 -- 172.21.15.16:0/4117335387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c1807c280 con 0x7f6c18107f20 2024-03-28T22:01:22.572 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.720+0000 7f6c0f7fe700 1 -- 172.21.15.16:0/4117335387 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c1401a910 con 0x7f6c18107f20 2024-03-28T22:01:22.572 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.720+0000 7f6c0f7fe700 1 -- 172.21.15.16:0/4117335387 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c140248f0 con 0x7f6c18107f20 2024-03-28T22:01:22.572 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.720+0000 7f6c20579700 1 -- 172.21.15.16:0/4117335387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c1807c740 con 0x7f6c18107f20 2024-03-28T22:01:22.572 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.720+0000 7f6c0f7fe700 1 -- 172.21.15.16:0/4117335387 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7f6c14022020 con 0x7f6c18107f20 2024-03-28T22:01:22.572 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.721+0000 7f6c0f7fe700 1 --2- 172.21.15.16:0/4117335387 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f6c04038ba0 0x7f6c0403b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:22.573 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.721+0000 7f6c20579700 1 -- 172.21.15.16:0/4117335387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c18063870 con 0x7f6c18107f20 2024-03-28T22:01:22.573 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.721+0000 7f6c0f7fe700 1 -- 172.21.15.16:0/4117335387 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f6c1404d320 con 0x7f6c18107f20 2024-03-28T22:01:22.573 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.721+0000 7f6c1db14700 1 --2- 172.21.15.16:0/4117335387 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f6c04038ba0 0x7f6c0403b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:22.573 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.721+0000 7f6c1db14700 1 --2- 172.21.15.16:0/4117335387 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f6c04038ba0 0x7f6c0403b060 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6c08006fd0 tx=0x7f6c08006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:22.573 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.726+0000 7f6c0f7fe700 1 -- 172.21.15.16:0/4117335387 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6c1404ccb0 con 0x7f6c18107f20 2024-03-28T22:01:22.573 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:20.993+0000 7f6c20579700 1 -- 172.21.15.16:0/4117335387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f6c1807c9c0 con 0x7f6c18107f20 2024-03-28T22:01:22.573 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:21.947+0000 7f6c0f7fe700 1 -- 172.21.15.16:0/4117335387 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f6c14017ca0 con 0x7f6c18107f20 2024-03-28T22:01:22.573 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:21.948+0000 7f6c0f7fe700 1 -- 172.21.15.16:0/4117335387 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 9) ==== 45941+0+0 (secure 0 0 0) 0x7f6c1402d3f0 con 0x7f6c18107f20 2024-03-28T22:01:22.573 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:21.950+0000 7f6c20579700 1 -- 172.21.15.16:0/4117335387 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f6c04038ba0 msgr2=0x7f6c0403b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:22.573 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:21.950+0000 7f6c20579700 1 --2- 172.21.15.16:0/4117335387 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f6c04038ba0 0x7f6c0403b060 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6c08006fd0 tx=0x7f6c08006e40 comp rx=0 tx=0).stop 2024-03-28T22:01:22.573 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:21.950+0000 7f6c20579700 1 -- 172.21.15.16:0/4117335387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c18107f20 msgr2=0x7f6c1807db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:22.573 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:21.950+0000 7f6c20579700 1 --2- 172.21.15.16:0/4117335387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c18107f20 0x7f6c1807db50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6c140117e0 tx=0x7f6c14011ba0 comp rx=0 tx=0).stop 2024-03-28T22:01:22.573 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:21.950+0000 7f6c20579700 1 -- 172.21.15.16:0/4117335387 shutdown_connections 2024-03-28T22:01:22.574 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:21.950+0000 7f6c20579700 1 --2- 172.21.15.16:0/4117335387 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f6c04038ba0 0x7f6c0403b060 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:22.574 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:21.950+0000 7f6c20579700 1 --2- 172.21.15.16:0/4117335387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c18107f20 0x7f6c1807db50 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:22.574 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:21.950+0000 7f6c20579700 1 -- 172.21.15.16:0/4117335387 >> 172.21.15.16:0/4117335387 conn(0x7f6c181034a0 msgr2=0x7f6c18103ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:22.574 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:21.950+0000 7f6c20579700 1 -- 172.21.15.16:0/4117335387 shutdown_connections 2024-03-28T22:01:22.574 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:21.950+0000 7f6c20579700 1 -- 172.21.15.16:0/4117335387 wait complete. 2024-03-28T22:01:22.864 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:22 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:01:22.543+0000 7ff223950200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T22:01:22.864 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:22 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:01:22.642+0000 7ff223950200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T22:01:23.172 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:22 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/4117335387' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-28T22:01:23.172 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:22 smithi016 ceph-mon[32374]: mgrmap e9: x(active, since 19s) 2024-03-28T22:01:23.436 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:23 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:01:23.318+0000 7ff223950200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T22:01:23.436 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:23 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:01:23.413+0000 7ff223950200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T22:01:23.936 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:23 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:01:23.558+0000 7ff223950200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T22:01:23.937 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:23 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:01:23.656+0000 7ff223950200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T22:01:23.937 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:23 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:01:23.741+0000 7ff223950200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T22:01:23.977 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-03-28T22:01:23.977 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-28T22:01:23.977 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-28T22:01:23.977 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-28T22:01:23.977 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-28T22:01:23.977 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-03-28T22:01:23.977 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.965+0000 7fc3183a5700 1 Processor -- start 2024-03-28T22:01:23.977 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.965+0000 7fc3183a5700 1 -- start start 2024-03-28T22:01:23.977 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.965+0000 7fc3183a5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc310108950 0x7fc31010ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:23.977 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.965+0000 7fc3183a5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc31007bde0 con 0x7fc310108950 2024-03-28T22:01:23.977 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.965+0000 7fc316141700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc310108950 0x7fc31010ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:23.978 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.965+0000 7fc316141700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc310108950 0x7fc31010ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55754/0 (socket says 172.21.15.16:55754) 2024-03-28T22:01:23.978 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.965+0000 7fc316141700 1 -- 172.21.15.16:0/1570033171 learned_addr learned my addr 172.21.15.16:0/1570033171 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:23.978 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.966+0000 7fc316141700 1 -- 172.21.15.16:0/1570033171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc31010b280 con 0x7fc310108950 2024-03-28T22:01:23.978 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.966+0000 7fc316141700 1 --2- 172.21.15.16:0/1570033171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc310108950 0x7fc31010ad40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc30c0098a0 tx=0x7fc30c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dac86303489ada6b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:23.978 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.966+0000 7fc31513f700 1 -- 172.21.15.16:0/1570033171 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc30c004160 con 0x7fc310108950 2024-03-28T22:01:23.978 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.966+0000 7fc31513f700 1 -- 172.21.15.16:0/1570033171 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc30c004300 con 0x7fc310108950 2024-03-28T22:01:23.978 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.966+0000 7fc31513f700 1 -- 172.21.15.16:0/1570033171 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc30c011b20 con 0x7fc310108950 2024-03-28T22:01:23.978 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.967+0000 7fc3183a5700 1 -- 172.21.15.16:0/1570033171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc310108950 msgr2=0x7fc31010ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:23.978 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.967+0000 7fc3183a5700 1 --2- 172.21.15.16:0/1570033171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc310108950 0x7fc31010ad40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc30c0098a0 tx=0x7fc30c009bb0 comp rx=0 tx=0).stop 2024-03-28T22:01:23.978 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.967+0000 7fc3183a5700 1 -- 172.21.15.16:0/1570033171 shutdown_connections 2024-03-28T22:01:23.978 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.967+0000 7fc3183a5700 1 --2- 172.21.15.16:0/1570033171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc310108950 0x7fc31010ad40 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:23.978 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.967+0000 7fc3183a5700 1 -- 172.21.15.16:0/1570033171 >> 172.21.15.16:0/1570033171 conn(0x7fc310102360 msgr2=0x7fc310104780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:23.978 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.967+0000 7fc3183a5700 1 -- 172.21.15.16:0/1570033171 shutdown_connections 2024-03-28T22:01:23.979 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.967+0000 7fc3183a5700 1 -- 172.21.15.16:0/1570033171 wait complete. 2024-03-28T22:01:23.979 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.967+0000 7fc3183a5700 1 Processor -- start 2024-03-28T22:01:23.979 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.967+0000 7fc3183a5700 1 -- start start 2024-03-28T22:01:23.979 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.967+0000 7fc3183a5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc310108950 0x7fc31019bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:23.979 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.967+0000 7fc3183a5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc31010c280 con 0x7fc310108950 2024-03-28T22:01:23.979 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.968+0000 7fc316141700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc310108950 0x7fc31019bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:23.979 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.968+0000 7fc316141700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc310108950 0x7fc31019bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55756/0 (socket says 172.21.15.16:55756) 2024-03-28T22:01:23.979 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.968+0000 7fc316141700 1 -- 172.21.15.16:0/541848972 learned_addr learned my addr 172.21.15.16:0/541848972 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:23.979 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.968+0000 7fc316141700 1 -- 172.21.15.16:0/541848972 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc31019c2e0 con 0x7fc310108950 2024-03-28T22:01:23.979 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.968+0000 7fc316141700 1 --2- 172.21.15.16:0/541848972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc310108950 0x7fc31019bda0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc30c0117e0 tx=0x7fc30c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:23.979 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.968+0000 7fc3077fe700 1 -- 172.21.15.16:0/541848972 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc30c003ba0 con 0x7fc310108950 2024-03-28T22:01:23.979 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.968+0000 7fc3077fe700 1 -- 172.21.15.16:0/541848972 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc30c01a910 con 0x7fc310108950 2024-03-28T22:01:23.979 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.968+0000 7fc3183a5700 1 -- 172.21.15.16:0/541848972 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc31019c570 con 0x7fc310108950 2024-03-28T22:01:23.979 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.968+0000 7fc3077fe700 1 -- 172.21.15.16:0/541848972 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc30c0248f0 con 0x7fc310108950 2024-03-28T22:01:23.980 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.968+0000 7fc3183a5700 1 -- 172.21.15.16:0/541848972 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc31019c9d0 con 0x7fc310108950 2024-03-28T22:01:23.980 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.969+0000 7fc3183a5700 1 -- 172.21.15.16:0/541848972 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc310063870 con 0x7fc310108950 2024-03-28T22:01:23.980 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.969+0000 7fc3077fe700 1 -- 172.21.15.16:0/541848972 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) ==== 45941+0+0 (secure 0 0 0) 0x7fc30c022020 con 0x7fc310108950 2024-03-28T22:01:23.980 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.969+0000 7fc3077fe700 1 --2- 172.21.15.16:0/541848972 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7fc2fc038bf0 0x7fc2fc03b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:23.980 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.969+0000 7fc3077fe700 1 -- 172.21.15.16:0/541848972 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc30c04d480 con 0x7fc310108950 2024-03-28T22:01:23.980 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.969+0000 7fc315940700 1 -- 172.21.15.16:0/541848972 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7fc2fc038bf0 msgr2=0x7fc2fc03b0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1832422779 2024-03-28T22:01:23.980 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.970+0000 7fc315940700 1 --2- 172.21.15.16:0/541848972 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7fc2fc038bf0 0x7fc2fc03b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:01:23.980 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:22.974+0000 7fc3077fe700 1 -- 172.21.15.16:0/541848972 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc30c018350 con 0x7fc310108950 2024-03-28T22:01:23.980 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:23.171+0000 7fc315940700 1 -- 172.21.15.16:0/541848972 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7fc2fc038bf0 msgr2=0x7fc2fc03b0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1832422779 2024-03-28T22:01:23.980 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:23.171+0000 7fc315940700 1 --2- 172.21.15.16:0/541848972 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7fc2fc038bf0 0x7fc2fc03b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:01:23.980 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:23.237+0000 7fc3183a5700 1 -- 172.21.15.16:0/541848972 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fc31019f380 con 0x7fc310108950 2024-03-28T22:01:23.980 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:23.237+0000 7fc3077fe700 1 -- 172.21.15.16:0/541848972 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7fc30c00bd30 con 0x7fc310108950 2024-03-28T22:01:23.980 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:23.239+0000 7fc3183a5700 1 -- 172.21.15.16:0/541848972 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7fc2fc038bf0 msgr2=0x7fc2fc03b0b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T22:01:23.981 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:23.239+0000 7fc3183a5700 1 --2- 172.21.15.16:0/541848972 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7fc2fc038bf0 0x7fc2fc03b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:23.981 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:23.239+0000 7fc3183a5700 1 -- 172.21.15.16:0/541848972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc310108950 msgr2=0x7fc31019bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:23.981 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:23.239+0000 7fc3183a5700 1 --2- 172.21.15.16:0/541848972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc310108950 0x7fc31019bda0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc30c0117e0 tx=0x7fc30c011ba0 comp rx=0 tx=0).stop 2024-03-28T22:01:23.981 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:23.239+0000 7fc3183a5700 1 -- 172.21.15.16:0/541848972 shutdown_connections 2024-03-28T22:01:23.981 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:23.239+0000 7fc3183a5700 1 --2- 172.21.15.16:0/541848972 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7fc2fc038bf0 0x7fc2fc03b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:23.981 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:23.239+0000 7fc3183a5700 1 --2- 172.21.15.16:0/541848972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc310108950 0x7fc31019bda0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:23.981 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:23.239+0000 7fc3183a5700 1 -- 172.21.15.16:0/541848972 >> 172.21.15.16:0/541848972 conn(0x7fc310102360 msgr2=0x7fc310102e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:23.981 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:23.239+0000 7fc3183a5700 1 -- 172.21.15.16:0/541848972 shutdown_connections 2024-03-28T22:01:23.981 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:23.239+0000 7fc3183a5700 1 -- 172.21.15.16:0/541848972 wait complete. 2024-03-28T22:01:23.981 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for the mgr to restart... 2024-03-28T22:01:23.981 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr epoch 9... 2024-03-28T22:01:24.202 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:23 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/541848972' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-28T22:01:24.202 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:24 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:01:24.021+0000 7ff223950200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T22:01:24.203 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:24 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:01:24.134+0000 7ff223950200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T22:01:24.511 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:24 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:01:24.228+0000 7ff223950200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T22:01:24.511 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:24 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:01:24.326+0000 7ff223950200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T22:01:24.511 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:24 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:01:24.509+0000 7ff223950200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T22:01:25.833 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:25 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:01:25.497+0000 7ff223950200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T22:01:25.834 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:25 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:01:25.632+0000 7ff223950200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T22:01:26.186 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:25 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:01:25.972+0000 7ff223950200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T22:01:26.186 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:26 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:01:26.064+0000 7ff223950200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T22:01:26.686 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:26 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:01:26.267+0000 7ff223950200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T22:01:26.686 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:26 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:01:26.457+0000 7ff223950200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T22:01:27.013 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:26 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:01:26.750+0000 7ff223950200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T22:01:27.014 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:27 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:01:27.012+0000 7ff223950200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T22:01:27.435 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:27 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:01:27.255+0000 7ff223950200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T22:01:28.436 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:28 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:01:28.082+0000 7ff223950200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T22:01:28.936 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:28 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:01:28.665+0000 7ff223950200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T22:01:28.936 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:28 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:01:28.767+0000 7ff223950200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T22:01:29.436 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:01:28 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:01:28.991+0000 7ff223950200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T22:01:29.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:29 smithi016 ceph-mon[32374]: Active manager daemon x restarted 2024-03-28T22:01:29.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:29 smithi016 ceph-mon[32374]: Activating manager daemon x 2024-03-28T22:01:29.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:29 smithi016 ceph-mon[32374]: osdmap e3: 0 total, 0 up, 0 in 2024-03-28T22:01:29.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:29 smithi016 ceph-mon[32374]: mgrmap e10: x(active, starting, since 0.00329358s) 2024-03-28T22:01:29.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:29 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:01:29.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:29 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T22:01:29.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:29 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T22:01:29.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:29 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T22:01:29.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:29 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T22:01:29.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:29 smithi016 ceph-mon[32374]: Manager daemon x is now available 2024-03-28T22:01:29.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:29 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:01:29.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:29 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:01:30.253 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:30 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T22:01:30.253 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:30 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T22:01:30.253 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:30 smithi016 ceph-mon[32374]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-28T22:01:30.253 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:30 smithi016 ceph-mon[32374]: mgrmap e11: x(active, since 1.00637s) 2024-03-28T22:01:30.254 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:30 smithi016 ceph-mon[32374]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-28T22:01:31.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-03-28T22:01:31.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-28T22:01:31.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-28T22:01:31.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-03-28T22:01:31.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.424+0000 7f87ead63700 1 Processor -- start 2024-03-28T22:01:31.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.424+0000 7f87ead63700 1 -- start start 2024-03-28T22:01:31.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.424+0000 7f87ead63700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e4108910 0x7f87e410ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:31.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.424+0000 7f87ead63700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f87e407be20 con 0x7f87e4108910 2024-03-28T22:01:31.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.424+0000 7f87e8aff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e4108910 0x7f87e410ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:31.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.424+0000 7f87e8aff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e4108910 0x7f87e410ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55758/0 (socket says 172.21.15.16:55758) 2024-03-28T22:01:31.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.424+0000 7f87e8aff700 1 -- 172.21.15.16:0/2112911354 learned_addr learned my addr 172.21.15.16:0/2112911354 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:31.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.424+0000 7f87e8aff700 1 -- 172.21.15.16:0/2112911354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87e410b240 con 0x7f87e4108910 2024-03-28T22:01:31.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.424+0000 7f87e8aff700 1 --2- 172.21.15.16:0/2112911354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e4108910 0x7f87e410ad00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f87d80098a0 tx=0x7f87d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c985a07ba1c2f566 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:31.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.425+0000 7f87e37fe700 1 -- 172.21.15.16:0/2112911354 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f87d8004160 con 0x7f87e4108910 2024-03-28T22:01:31.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.425+0000 7f87e37fe700 1 -- 172.21.15.16:0/2112911354 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87d8004300 con 0x7f87e4108910 2024-03-28T22:01:31.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.425+0000 7f87e37fe700 1 -- 172.21.15.16:0/2112911354 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f87d8011b20 con 0x7f87e4108910 2024-03-28T22:01:31.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.426+0000 7f87ead63700 1 -- 172.21.15.16:0/2112911354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e4108910 msgr2=0x7f87e410ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:31.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.426+0000 7f87ead63700 1 --2- 172.21.15.16:0/2112911354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e4108910 0x7f87e410ad00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f87d80098a0 tx=0x7f87d8009bb0 comp rx=0 tx=0).stop 2024-03-28T22:01:31.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.426+0000 7f87ead63700 1 -- 172.21.15.16:0/2112911354 shutdown_connections 2024-03-28T22:01:31.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.426+0000 7f87ead63700 1 --2- 172.21.15.16:0/2112911354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e4108910 0x7f87e410ad00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:31.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.426+0000 7f87ead63700 1 -- 172.21.15.16:0/2112911354 >> 172.21.15.16:0/2112911354 conn(0x7f87e4102360 msgr2=0x7f87e4104780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:31.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.426+0000 7f87ead63700 1 -- 172.21.15.16:0/2112911354 shutdown_connections 2024-03-28T22:01:31.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.426+0000 7f87ead63700 1 -- 172.21.15.16:0/2112911354 wait complete. 2024-03-28T22:01:31.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.426+0000 7f87ead63700 1 Processor -- start 2024-03-28T22:01:31.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.426+0000 7f87ead63700 1 -- start start 2024-03-28T22:01:31.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.427+0000 7f87ead63700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e4108910 0x7f87e419bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:31.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.427+0000 7f87ead63700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f87e410c240 con 0x7f87e4108910 2024-03-28T22:01:31.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.427+0000 7f87e8aff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e4108910 0x7f87e419bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:31.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.427+0000 7f87e8aff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e4108910 0x7f87e419bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55770/0 (socket says 172.21.15.16:55770) 2024-03-28T22:01:31.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.427+0000 7f87e8aff700 1 -- 172.21.15.16:0/1782925531 learned_addr learned my addr 172.21.15.16:0/1782925531 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:31.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.427+0000 7f87e8aff700 1 -- 172.21.15.16:0/1782925531 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87e419c2e0 con 0x7f87e4108910 2024-03-28T22:01:31.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.427+0000 7f87e8aff700 1 --2- 172.21.15.16:0/1782925531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e4108910 0x7f87e419bda0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f87d80117e0 tx=0x7f87d8011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:31.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.427+0000 7f87e1ffb700 1 -- 172.21.15.16:0/1782925531 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f87d8003bf0 con 0x7f87e4108910 2024-03-28T22:01:31.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.427+0000 7f87e1ffb700 1 -- 172.21.15.16:0/1782925531 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87d801a910 con 0x7f87e4108910 2024-03-28T22:01:31.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.427+0000 7f87e1ffb700 1 -- 172.21.15.16:0/1782925531 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f87d8024970 con 0x7f87e4108910 2024-03-28T22:01:31.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.427+0000 7f87ead63700 1 -- 172.21.15.16:0/1782925531 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87e419c570 con 0x7f87e4108910 2024-03-28T22:01:31.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.428+0000 7f87ead63700 1 -- 172.21.15.16:0/1782925531 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87e419c9d0 con 0x7f87e4108910 2024-03-28T22:01:31.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.428+0000 7f87e1ffb700 1 -- 172.21.15.16:0/1782925531 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) ==== 45941+0+0 (secure 0 0 0) 0x7f87d8022020 con 0x7f87e4108910 2024-03-28T22:01:31.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.429+0000 7f87e1ffb700 1 --2- 172.21.15.16:0/1782925531 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f87d4038c00 0x7f87d403b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:31.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.429+0000 7f87e1ffb700 1 -- 172.21.15.16:0/1782925531 --> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f87d403b7d0 con 0x7f87d4038c00 2024-03-28T22:01:31.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.429+0000 7f87e1ffb700 1 -- 172.21.15.16:0/1782925531 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f87d804cdb0 con 0x7f87e4108910 2024-03-28T22:01:31.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.429+0000 7f87e3fff700 1 -- 172.21.15.16:0/1782925531 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f87d4038c00 msgr2=0x7f87d403b0c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1832422779 2024-03-28T22:01:31.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.429+0000 7f87e3fff700 1 --2- 172.21.15.16:0/1782925531 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f87d4038c00 0x7f87d403b0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:01:31.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.630+0000 7f87e3fff700 1 -- 172.21.15.16:0/1782925531 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f87d4038c00 msgr2=0x7f87d403b0c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1832422779 2024-03-28T22:01:31.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:24.630+0000 7f87e3fff700 1 --2- 172.21.15.16:0/1782925531 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f87d4038c00 0x7f87d403b0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:01:31.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:25.031+0000 7f87e3fff700 1 -- 172.21.15.16:0/1782925531 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f87d4038c00 msgr2=0x7f87d403b0c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1832422779 2024-03-28T22:01:31.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:25.031+0000 7f87e3fff700 1 --2- 172.21.15.16:0/1782925531 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f87d4038c00 0x7f87d403b0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:01:31.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:25.832+0000 7f87e3fff700 1 -- 172.21.15.16:0/1782925531 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f87d4038c00 msgr2=0x7f87d403b0c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1832422779 2024-03-28T22:01:31.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:25.832+0000 7f87e3fff700 1 --2- 172.21.15.16:0/1782925531 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f87d4038c00 0x7f87d403b0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:01:31.010 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:27.433+0000 7f87e3fff700 1 -- 172.21.15.16:0/1782925531 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f87d4038c00 msgr2=0x7f87d403b0c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1832422779 2024-03-28T22:01:31.010 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:27.433+0000 7f87e3fff700 1 --2- 172.21.15.16:0/1782925531 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f87d4038c00 0x7f87d403b0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-28T22:01:31.010 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:28.995+0000 7f87e1ffb700 1 -- 172.21.15.16:0/1782925531 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mgrmap(e 10) ==== 45708+0+0 (secure 0 0 0) 0x7f87d801a430 con 0x7f87e4108910 2024-03-28T22:01:31.010 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:28.995+0000 7f87e1ffb700 1 -- 172.21.15.16:0/1782925531 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f87d4038c00 msgr2=0x7f87d403b0c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T22:01:31.010 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:28.995+0000 7f87e1ffb700 1 --2- 172.21.15.16:0/1782925531 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f87d4038c00 0x7f87d403b0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:31.010 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:29.997+0000 7f87e1ffb700 1 -- 172.21.15.16:0/1782925531 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 11) ==== 45835+0+0 (secure 0 0 0) 0x7f87d804e400 con 0x7f87e4108910 2024-03-28T22:01:31.010 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:29.997+0000 7f87e1ffb700 1 --2- 172.21.15.16:0/1782925531 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f87d403c6f0 0x7f87d403eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:31.010 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:29.997+0000 7f87e1ffb700 1 -- 172.21.15.16:0/1782925531 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f87d8018e50 con 0x7f87d403c6f0 2024-03-28T22:01:31.010 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:29.997+0000 7f87e3fff700 1 --2- 172.21.15.16:0/1782925531 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f87d403c6f0 0x7f87d403eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:31.010 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:29.998+0000 7f87e3fff700 1 --2- 172.21.15.16:0/1782925531 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f87d403c6f0 0x7f87d403eae0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f87d0003a10 tx=0x7f87d0009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:31.010 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:29.999+0000 7f87e1ffb700 1 -- 172.21.15.16:0/1782925531 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f87d8018e50 con 0x7f87d403c6f0 2024-03-28T22:01:31.010 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:30.005+0000 7f87ead63700 1 -- 172.21.15.16:0/1782925531 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f87e4063870 con 0x7f87d403c6f0 2024-03-28T22:01:31.010 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:30.006+0000 7f87e1ffb700 1 -- 172.21.15.16:0/1782925531 <== mgr.14152 v2:172.21.15.16:6800/21986004 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f87e4063870 con 0x7f87d403c6f0 2024-03-28T22:01:31.010 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:30.006+0000 7f87ead63700 1 -- 172.21.15.16:0/1782925531 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f87d403c6f0 msgr2=0x7f87d403eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:31.011 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:30.006+0000 7f87ead63700 1 --2- 172.21.15.16:0/1782925531 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f87d403c6f0 0x7f87d403eae0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f87d0003a10 tx=0x7f87d0009270 comp rx=0 tx=0).stop 2024-03-28T22:01:31.011 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:30.006+0000 7f87ead63700 1 -- 172.21.15.16:0/1782925531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e4108910 msgr2=0x7f87e419bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:31.011 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:30.006+0000 7f87ead63700 1 --2- 172.21.15.16:0/1782925531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e4108910 0x7f87e419bda0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f87d80117e0 tx=0x7f87d8011c10 comp rx=0 tx=0).stop 2024-03-28T22:01:31.011 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:30.006+0000 7f87ead63700 1 -- 172.21.15.16:0/1782925531 shutdown_connections 2024-03-28T22:01:31.011 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:30.006+0000 7f87ead63700 1 --2- 172.21.15.16:0/1782925531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e4108910 0x7f87e419bda0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:31.011 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:30.006+0000 7f87ead63700 1 --2- 172.21.15.16:0/1782925531 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f87d403c6f0 0x7f87d403eae0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:31.011 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:30.006+0000 7f87ead63700 1 --2- 172.21.15.16:0/1782925531 >> [v2:172.21.15.16:6800/1832422779,v1:172.21.15.16:6801/1832422779] conn(0x7f87d4038c00 0x7f87d403b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:31.011 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:30.006+0000 7f87ead63700 1 -- 172.21.15.16:0/1782925531 >> 172.21.15.16:0/1782925531 conn(0x7f87e4102360 msgr2=0x7f87e4102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:31.011 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:30.006+0000 7f87ead63700 1 -- 172.21.15.16:0/1782925531 shutdown_connections 2024-03-28T22:01:31.011 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:30.006+0000 7f87ead63700 1 -- 172.21.15.16:0/1782925531 wait complete. 2024-03-28T22:01:31.011 INFO:teuthology.orchestra.run.smithi016.stdout:mgr epoch 9 is available 2024-03-28T22:01:31.011 INFO:teuthology.orchestra.run.smithi016.stdout:Generating a dashboard self-signed certificate... 2024-03-28T22:01:32.123 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:31 smithi016 ceph-mon[32374]: [28/Mar/2024:22:01:30] ENGINE Bus STARTING 2024-03-28T22:01:32.123 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:31 smithi016 ceph-mon[32374]: [28/Mar/2024:22:01:30] ENGINE Serving on http://172.21.15.16:8765 2024-03-28T22:01:32.123 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:31 smithi016 ceph-mon[32374]: [28/Mar/2024:22:01:30] ENGINE Serving on https://172.21.15.16:7150 2024-03-28T22:01:32.123 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:31 smithi016 ceph-mon[32374]: [28/Mar/2024:22:01:30] ENGINE Bus STARTED 2024-03-28T22:01:32.123 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:31 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:01:32.409 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-28T22:01:32.410 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.422+0000 7f95a00cf700 1 Processor -- start 2024-03-28T22:01:32.410 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.422+0000 7f95a00cf700 1 -- start start 2024-03-28T22:01:32.410 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.422+0000 7f95a00cf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9598106770 0x7f9598106b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:32.410 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.423+0000 7f95a00cf700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f959807bdf0 con 0x7f9598106770 2024-03-28T22:01:32.410 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.423+0000 7f959de6b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9598106770 0x7f9598106b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:32.410 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.423+0000 7f959de6b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9598106770 0x7f9598106b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38976/0 (socket says 172.21.15.16:38976) 2024-03-28T22:01:32.410 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.423+0000 7f959de6b700 1 -- 172.21.15.16:0/936351489 learned_addr learned my addr 172.21.15.16:0/936351489 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:32.410 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.423+0000 7f959de6b700 1 -- 172.21.15.16:0/936351489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95981070d0 con 0x7f9598106770 2024-03-28T22:01:32.410 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.423+0000 7f959de6b700 1 --2- 172.21.15.16:0/936351489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9598106770 0x7f9598106b90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f95940098a0 tx=0x7f9594009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b6a83b68e35c3d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:32.410 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.424+0000 7f959ce69700 1 -- 172.21.15.16:0/936351489 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9594004160 con 0x7f9598106770 2024-03-28T22:01:32.410 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.424+0000 7f959ce69700 1 -- 172.21.15.16:0/936351489 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9594004300 con 0x7f9598106770 2024-03-28T22:01:32.410 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.424+0000 7f959ce69700 1 -- 172.21.15.16:0/936351489 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9594011b20 con 0x7f9598106770 2024-03-28T22:01:32.411 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.424+0000 7f95a00cf700 1 -- 172.21.15.16:0/936351489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9598106770 msgr2=0x7f9598106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:32.411 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.424+0000 7f95a00cf700 1 --2- 172.21.15.16:0/936351489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9598106770 0x7f9598106b90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f95940098a0 tx=0x7f9594009bb0 comp rx=0 tx=0).stop 2024-03-28T22:01:32.411 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.424+0000 7f95a00cf700 1 -- 172.21.15.16:0/936351489 shutdown_connections 2024-03-28T22:01:32.411 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.424+0000 7f95a00cf700 1 --2- 172.21.15.16:0/936351489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9598106770 0x7f9598106b90 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:32.411 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.424+0000 7f95a00cf700 1 -- 172.21.15.16:0/936351489 >> 172.21.15.16:0/936351489 conn(0x7f9598102310 msgr2=0x7f9598104750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:32.411 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.424+0000 7f95a00cf700 1 -- 172.21.15.16:0/936351489 shutdown_connections 2024-03-28T22:01:32.411 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.424+0000 7f95a00cf700 1 -- 172.21.15.16:0/936351489 wait complete. 2024-03-28T22:01:32.411 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.425+0000 7f95a00cf700 1 Processor -- start 2024-03-28T22:01:32.411 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.425+0000 7f95a00cf700 1 -- start start 2024-03-28T22:01:32.411 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.425+0000 7f95a00cf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9598106770 0x7f959819df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:32.411 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.425+0000 7f95a00cf700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f95981081a0 con 0x7f9598106770 2024-03-28T22:01:32.411 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.425+0000 7f959de6b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9598106770 0x7f959819df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:32.411 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.425+0000 7f959de6b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9598106770 0x7f959819df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38978/0 (socket says 172.21.15.16:38978) 2024-03-28T22:01:32.412 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.425+0000 7f959de6b700 1 -- 172.21.15.16:0/2095443002 learned_addr learned my addr 172.21.15.16:0/2095443002 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:32.412 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.426+0000 7f959de6b700 1 -- 172.21.15.16:0/2095443002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f959819e4d0 con 0x7f9598106770 2024-03-28T22:01:32.412 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.426+0000 7f959de6b700 1 --2- 172.21.15.16:0/2095443002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9598106770 0x7f959819df90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f95940117e0 tx=0x7f9594011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:32.412 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.426+0000 7f958effd700 1 -- 172.21.15.16:0/2095443002 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9594003bf0 con 0x7f9598106770 2024-03-28T22:01:32.412 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.426+0000 7f95a00cf700 1 -- 172.21.15.16:0/2095443002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f959819e760 con 0x7f9598106770 2024-03-28T22:01:32.412 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.426+0000 7f958effd700 1 -- 172.21.15.16:0/2095443002 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f959401a910 con 0x7f9598106770 2024-03-28T22:01:32.412 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.426+0000 7f958effd700 1 -- 172.21.15.16:0/2095443002 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9594024970 con 0x7f9598106770 2024-03-28T22:01:32.412 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.426+0000 7f95a00cf700 1 -- 172.21.15.16:0/2095443002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f959819ebc0 con 0x7f9598106770 2024-03-28T22:01:32.412 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.427+0000 7f958effd700 1 -- 172.21.15.16:0/2095443002 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 11) ==== 45835+0+0 (secure 0 0 0) 0x7f9594022020 con 0x7f9598106770 2024-03-28T22:01:32.412 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.427+0000 7f95a00cf700 1 -- 172.21.15.16:0/2095443002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f957c005360 con 0x7f9598106770 2024-03-28T22:01:32.412 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.427+0000 7f958effd700 1 --2- 172.21.15.16:0/2095443002 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f9584038a90 0x7f958403af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:32.412 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.427+0000 7f958effd700 1 -- 172.21.15.16:0/2095443002 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f959404ccd0 con 0x7f9598106770 2024-03-28T22:01:32.412 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.427+0000 7f959d66a700 1 --2- 172.21.15.16:0/2095443002 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f9584038a90 0x7f958403af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:32.413 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.428+0000 7f959d66a700 1 --2- 172.21.15.16:0/2095443002 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f9584038a90 0x7f958403af50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9588006fd0 tx=0x7f9588006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:32.413 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.433+0000 7f958effd700 1 -- 172.21.15.16:0/2095443002 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9594011e20 con 0x7f9598106770 2024-03-28T22:01:32.413 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.642+0000 7f95a00cf700 1 -- 172.21.15.16:0/2095443002 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f957c000bf0 con 0x7f9584038a90 2024-03-28T22:01:32.413 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.754+0000 7f958effd700 1 -- 172.21.15.16:0/2095443002 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 12) ==== 45941+0+0 (secure 0 0 0) 0x7f9594024cd0 con 0x7f9598106770 2024-03-28T22:01:32.413 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.793+0000 7f958effd700 1 -- 172.21.15.16:0/2095443002 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f957c000bf0 con 0x7f9584038a90 2024-03-28T22:01:32.413 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.795+0000 7f95a00cf700 1 -- 172.21.15.16:0/2095443002 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f9584038a90 msgr2=0x7f958403af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:32.413 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.795+0000 7f95a00cf700 1 --2- 172.21.15.16:0/2095443002 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f9584038a90 0x7f958403af50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9588006fd0 tx=0x7f9588006e40 comp rx=0 tx=0).stop 2024-03-28T22:01:32.413 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.795+0000 7f95a00cf700 1 -- 172.21.15.16:0/2095443002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9598106770 msgr2=0x7f959819df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:32.413 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.795+0000 7f95a00cf700 1 --2- 172.21.15.16:0/2095443002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9598106770 0x7f959819df90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f95940117e0 tx=0x7f9594011c10 comp rx=0 tx=0).stop 2024-03-28T22:01:32.413 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.795+0000 7f95a00cf700 1 -- 172.21.15.16:0/2095443002 shutdown_connections 2024-03-28T22:01:32.413 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.795+0000 7f95a00cf700 1 --2- 172.21.15.16:0/2095443002 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f9584038a90 0x7f958403af50 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:32.413 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.795+0000 7f95a00cf700 1 --2- 172.21.15.16:0/2095443002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9598106770 0x7f959819df90 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:32.413 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.795+0000 7f95a00cf700 1 -- 172.21.15.16:0/2095443002 >> 172.21.15.16:0/2095443002 conn(0x7f9598102310 msgr2=0x7f9598194c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:32.413 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.795+0000 7f95a00cf700 1 -- 172.21.15.16:0/2095443002 shutdown_connections 2024-03-28T22:01:32.414 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:31.795+0000 7f95a00cf700 1 -- 172.21.15.16:0/2095443002 wait complete. 2024-03-28T22:01:32.414 INFO:teuthology.orchestra.run.smithi016.stdout:Creating initial admin user... 2024-03-28T22:01:33.043 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:32 smithi016 ceph-mon[32374]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:01:33.043 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:32 smithi016 ceph-mon[32374]: mgrmap e12: x(active, since 2s) 2024-03-28T22:01:33.043 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:32 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:01:33.043 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:32 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:01:33.900 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$qns13rxshREUtM964T6VM.UY3PAMY0eqQKlK/wGjmk.xtziPT1.di", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711663293, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-28T22:01:33.900 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.796+0000 7f446460e700 1 Processor -- start 2024-03-28T22:01:33.900 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.796+0000 7f446460e700 1 -- start start 2024-03-28T22:01:33.901 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.796+0000 7f446460e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f445c107f20 0x7f445c108340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:33.901 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.796+0000 7f446460e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f445c108880 con 0x7f445c107f20 2024-03-28T22:01:33.901 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.797+0000 7f44623aa700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f445c107f20 0x7f445c108340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:33.901 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.797+0000 7f44623aa700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f445c107f20 0x7f445c108340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38986/0 (socket says 172.21.15.16:38986) 2024-03-28T22:01:33.901 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.797+0000 7f44623aa700 1 -- 172.21.15.16:0/3258670840 learned_addr learned my addr 172.21.15.16:0/3258670840 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:33.901 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.797+0000 7f44623aa700 1 -- 172.21.15.16:0/3258670840 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f445c108fe0 con 0x7f445c107f20 2024-03-28T22:01:33.901 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.797+0000 7f44623aa700 1 --2- 172.21.15.16:0/3258670840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f445c107f20 0x7f445c108340 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4454009e80 tx=0x7f445400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12e487306740d1b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:33.901 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.797+0000 7f44613a8700 1 -- 172.21.15.16:0/3258670840 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4454017070 con 0x7f445c107f20 2024-03-28T22:01:33.901 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.797+0000 7f44613a8700 1 -- 172.21.15.16:0/3258670840 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f445400b8c0 con 0x7f445c107f20 2024-03-28T22:01:33.901 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.797+0000 7f44613a8700 1 -- 172.21.15.16:0/3258670840 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f44540045c0 con 0x7f445c107f20 2024-03-28T22:01:33.901 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.798+0000 7f446460e700 1 -- 172.21.15.16:0/3258670840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f445c107f20 msgr2=0x7f445c108340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:33.901 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.798+0000 7f446460e700 1 --2- 172.21.15.16:0/3258670840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f445c107f20 0x7f445c108340 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4454009e80 tx=0x7f445400b330 comp rx=0 tx=0).stop 2024-03-28T22:01:33.902 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.798+0000 7f446460e700 1 -- 172.21.15.16:0/3258670840 shutdown_connections 2024-03-28T22:01:33.902 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.798+0000 7f446460e700 1 --2- 172.21.15.16:0/3258670840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f445c107f20 0x7f445c108340 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:33.902 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.798+0000 7f446460e700 1 -- 172.21.15.16:0/3258670840 >> 172.21.15.16:0/3258670840 conn(0x7f445c1034a0 msgr2=0x7f445c105900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:33.902 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.798+0000 7f446460e700 1 -- 172.21.15.16:0/3258670840 shutdown_connections 2024-03-28T22:01:33.902 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.798+0000 7f446460e700 1 -- 172.21.15.16:0/3258670840 wait complete. 2024-03-28T22:01:33.902 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.799+0000 7f446460e700 1 Processor -- start 2024-03-28T22:01:33.902 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.799+0000 7f446460e700 1 -- start start 2024-03-28T22:01:33.902 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.799+0000 7f446460e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f445c107f20 0x7f445c19bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:33.902 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.799+0000 7f446460e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f445c10c420 con 0x7f445c107f20 2024-03-28T22:01:33.902 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.799+0000 7f44623aa700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f445c107f20 0x7f445c19bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:33.902 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.799+0000 7f44623aa700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f445c107f20 0x7f445c19bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38996/0 (socket says 172.21.15.16:38996) 2024-03-28T22:01:33.902 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.799+0000 7f44623aa700 1 -- 172.21.15.16:0/2025179929 learned_addr learned my addr 172.21.15.16:0/2025179929 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:33.903 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.799+0000 7f44623aa700 1 -- 172.21.15.16:0/2025179929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f445c19c310 con 0x7f445c107f20 2024-03-28T22:01:33.903 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.799+0000 7f44623aa700 1 --2- 172.21.15.16:0/2025179929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f445c107f20 0x7f445c19bdd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f445400b7b0 tx=0x7f445400bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:33.903 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.799+0000 7f445b7fe700 1 -- 172.21.15.16:0/2025179929 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4454020020 con 0x7f445c107f20 2024-03-28T22:01:33.903 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.799+0000 7f445b7fe700 1 -- 172.21.15.16:0/2025179929 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4454003d90 con 0x7f445c107f20 2024-03-28T22:01:33.903 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.799+0000 7f445b7fe700 1 -- 172.21.15.16:0/2025179929 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4454017040 con 0x7f445c107f20 2024-03-28T22:01:33.903 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.799+0000 7f446460e700 1 -- 172.21.15.16:0/2025179929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f445c19c5a0 con 0x7f445c107f20 2024-03-28T22:01:33.903 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.800+0000 7f446460e700 1 -- 172.21.15.16:0/2025179929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f445c19f290 con 0x7f445c107f20 2024-03-28T22:01:33.903 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.800+0000 7f445b7fe700 1 -- 172.21.15.16:0/2025179929 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) ==== 45941+0+0 (secure 0 0 0) 0x7f4454023020 con 0x7f445c107f20 2024-03-28T22:01:33.903 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.801+0000 7f445b7fe700 1 --2- 172.21.15.16:0/2025179929 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4444038ba0 0x7f444403b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:33.903 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.801+0000 7f446460e700 1 -- 172.21.15.16:0/2025179929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f445c063870 con 0x7f445c107f20 2024-03-28T22:01:33.903 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.801+0000 7f445b7fe700 1 -- 172.21.15.16:0/2025179929 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f445400f050 con 0x7f445c107f20 2024-03-28T22:01:33.903 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.801+0000 7f4461ba9700 1 --2- 172.21.15.16:0/2025179929 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4444038ba0 0x7f444403b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:33.904 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.801+0000 7f4461ba9700 1 --2- 172.21.15.16:0/2025179929 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4444038ba0 0x7f444403b060 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4450006fd0 tx=0x7f4450006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:33.904 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:32.806+0000 7f445b7fe700 1 -- 172.21.15.16:0/2025179929 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f445402a020 con 0x7f445c107f20 2024-03-28T22:01:33.904 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:33.041+0000 7f446460e700 1 -- 172.21.15.16:0/2025179929 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f445c104620 con 0x7f4444038ba0 2024-03-28T22:01:33.904 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:33.287+0000 7f445b7fe700 1 -- 172.21.15.16:0/2025179929 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f445c104620 con 0x7f4444038ba0 2024-03-28T22:01:33.904 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:33.289+0000 7f446460e700 1 -- 172.21.15.16:0/2025179929 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4444038ba0 msgr2=0x7f444403b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:33.904 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:33.289+0000 7f446460e700 1 --2- 172.21.15.16:0/2025179929 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4444038ba0 0x7f444403b060 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4450006fd0 tx=0x7f4450006e40 comp rx=0 tx=0).stop 2024-03-28T22:01:33.904 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:33.289+0000 7f446460e700 1 -- 172.21.15.16:0/2025179929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f445c107f20 msgr2=0x7f445c19bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:33.904 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:33.289+0000 7f446460e700 1 --2- 172.21.15.16:0/2025179929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f445c107f20 0x7f445c19bdd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f445400b7b0 tx=0x7f445400bd50 comp rx=0 tx=0).stop 2024-03-28T22:01:33.904 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:33.289+0000 7f446460e700 1 -- 172.21.15.16:0/2025179929 shutdown_connections 2024-03-28T22:01:33.904 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:33.289+0000 7f446460e700 1 --2- 172.21.15.16:0/2025179929 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4444038ba0 0x7f444403b060 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:33.904 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:33.289+0000 7f446460e700 1 --2- 172.21.15.16:0/2025179929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f445c107f20 0x7f445c19bdd0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:33.904 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:33.289+0000 7f446460e700 1 -- 172.21.15.16:0/2025179929 >> 172.21.15.16:0/2025179929 conn(0x7f445c1034a0 msgr2=0x7f445c103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:33.904 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:33.289+0000 7f446460e700 1 -- 172.21.15.16:0/2025179929 shutdown_connections 2024-03-28T22:01:33.905 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:33.290+0000 7f446460e700 1 -- 172.21.15.16:0/2025179929 wait complete. 2024-03-28T22:01:33.905 INFO:teuthology.orchestra.run.smithi016.stdout:Fetching dashboard port number... 2024-03-28T22:01:34.261 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:33 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:01:34.261 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:33 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:01:35.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:34 smithi016 ceph-mon[32374]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:01:35.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:34 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2979399212' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-28T22:01:35.476 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 8443 2024-03-28T22:01:35.476 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.360+0000 7fcb0c3fb700 1 Processor -- start 2024-03-28T22:01:35.477 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.360+0000 7fcb0c3fb700 1 -- start start 2024-03-28T22:01:35.477 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.360+0000 7fcb0c3fb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb04106d70 0x7fcb04107190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:35.477 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.360+0000 7fcb0c3fb700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcb04107760 con 0x7fcb04106d70 2024-03-28T22:01:35.477 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.361+0000 7fcb0a197700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb04106d70 0x7fcb04107190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:35.477 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.361+0000 7fcb0a197700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb04106d70 0x7fcb04107190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39002/0 (socket says 172.21.15.16:39002) 2024-03-28T22:01:35.477 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.361+0000 7fcb0a197700 1 -- 172.21.15.16:0/961644400 learned_addr learned my addr 172.21.15.16:0/961644400 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:35.477 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.361+0000 7fcb0a197700 1 -- 172.21.15.16:0/961644400 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb04107f70 con 0x7fcb04106d70 2024-03-28T22:01:35.477 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.361+0000 7fcb0a197700 1 --2- 172.21.15.16:0/961644400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb04106d70 0x7fcb04107190 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcb000098a0 tx=0x7fcb00009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=709f307803d32674 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:35.477 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.361+0000 7fcb09195700 1 -- 172.21.15.16:0/961644400 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcb00004160 con 0x7fcb04106d70 2024-03-28T22:01:35.477 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.361+0000 7fcb09195700 1 -- 172.21.15.16:0/961644400 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb00004300 con 0x7fcb04106d70 2024-03-28T22:01:35.477 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.361+0000 7fcb09195700 1 -- 172.21.15.16:0/961644400 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcb00011b20 con 0x7fcb04106d70 2024-03-28T22:01:35.477 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.362+0000 7fcb0c3fb700 1 -- 172.21.15.16:0/961644400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb04106d70 msgr2=0x7fcb04107190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:35.477 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.362+0000 7fcb0c3fb700 1 --2- 172.21.15.16:0/961644400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb04106d70 0x7fcb04107190 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcb000098a0 tx=0x7fcb00009bb0 comp rx=0 tx=0).stop 2024-03-28T22:01:35.477 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.362+0000 7fcb0c3fb700 1 -- 172.21.15.16:0/961644400 shutdown_connections 2024-03-28T22:01:35.477 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.362+0000 7fcb0c3fb700 1 --2- 172.21.15.16:0/961644400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb04106d70 0x7fcb04107190 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:35.478 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.362+0000 7fcb0c3fb700 1 -- 172.21.15.16:0/961644400 >> 172.21.15.16:0/961644400 conn(0x7fcb04102310 msgr2=0x7fcb04104750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:35.478 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.362+0000 7fcb0c3fb700 1 -- 172.21.15.16:0/961644400 shutdown_connections 2024-03-28T22:01:35.478 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.362+0000 7fcb0c3fb700 1 -- 172.21.15.16:0/961644400 wait complete. 2024-03-28T22:01:35.478 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.363+0000 7fcb0c3fb700 1 Processor -- start 2024-03-28T22:01:35.478 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.363+0000 7fcb0c3fb700 1 -- start start 2024-03-28T22:01:35.478 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.363+0000 7fcb0c3fb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb04106d70 0x7fcb0407ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:35.478 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.363+0000 7fcb0c3fb700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0410b320 con 0x7fcb04106d70 2024-03-28T22:01:35.478 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.363+0000 7fcb0a197700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb04106d70 0x7fcb0407ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:35.478 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.363+0000 7fcb0a197700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb04106d70 0x7fcb0407ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39008/0 (socket says 172.21.15.16:39008) 2024-03-28T22:01:35.478 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.363+0000 7fcb0a197700 1 -- 172.21.15.16:0/2979399212 learned_addr learned my addr 172.21.15.16:0/2979399212 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:35.478 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.363+0000 7fcb0a197700 1 -- 172.21.15.16:0/2979399212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb0419ea90 con 0x7fcb04106d70 2024-03-28T22:01:35.478 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.363+0000 7fcb0a197700 1 --2- 172.21.15.16:0/2979399212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb04106d70 0x7fcb0407ba90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcb000117e0 tx=0x7fcb00011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:35.478 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.364+0000 7fcafb7fe700 1 -- 172.21.15.16:0/2979399212 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcb00003bf0 con 0x7fcb04106d70 2024-03-28T22:01:35.479 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.364+0000 7fcb0c3fb700 1 -- 172.21.15.16:0/2979399212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb0419ec60 con 0x7fcb04106d70 2024-03-28T22:01:35.479 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.364+0000 7fcafb7fe700 1 -- 172.21.15.16:0/2979399212 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb0001a910 con 0x7fcb04106d70 2024-03-28T22:01:35.479 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.364+0000 7fcafb7fe700 1 -- 172.21.15.16:0/2979399212 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcb000249a0 con 0x7fcb04106d70 2024-03-28T22:01:35.479 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.364+0000 7fcb0c3fb700 1 -- 172.21.15.16:0/2979399212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb0419f000 con 0x7fcb04106d70 2024-03-28T22:01:35.479 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.364+0000 7fcafb7fe700 1 -- 172.21.15.16:0/2979399212 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) ==== 45941+0+0 (secure 0 0 0) 0x7fcb00022020 con 0x7fcb04106d70 2024-03-28T22:01:35.479 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.365+0000 7fcb0c3fb700 1 -- 172.21.15.16:0/2979399212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb04063870 con 0x7fcb04106d70 2024-03-28T22:01:35.479 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.365+0000 7fcafb7fe700 1 --2- 172.21.15.16:0/2979399212 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fcaf0038c00 0x7fcaf003b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:35.479 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.365+0000 7fcafb7fe700 1 -- 172.21.15.16:0/2979399212 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fcb0004cf10 con 0x7fcb04106d70 2024-03-28T22:01:35.479 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.365+0000 7fcb09996700 1 --2- 172.21.15.16:0/2979399212 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fcaf0038c00 0x7fcaf003b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:35.479 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.365+0000 7fcb09996700 1 --2- 172.21.15.16:0/2979399212 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fcaf0038c00 0x7fcaf003b0c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4006fd0 tx=0x7fcaf4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:35.479 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.370+0000 7fcafb7fe700 1 -- 172.21.15.16:0/2979399212 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fcb00011e20 con 0x7fcb04106d70 2024-03-28T22:01:35.479 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.578+0000 7fcb0c3fb700 1 -- 172.21.15.16:0/2979399212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7fcb0419f7f0 con 0x7fcb04106d70 2024-03-28T22:01:35.479 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.579+0000 7fcafb7fe700 1 -- 172.21.15.16:0/2979399212 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7fcb0000b3e0 con 0x7fcb04106d70 2024-03-28T22:01:35.479 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.581+0000 7fcb0c3fb700 1 -- 172.21.15.16:0/2979399212 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fcaf0038c00 msgr2=0x7fcaf003b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:35.480 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.581+0000 7fcb0c3fb700 1 --2- 172.21.15.16:0/2979399212 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fcaf0038c00 0x7fcaf003b0c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4006fd0 tx=0x7fcaf4006e40 comp rx=0 tx=0).stop 2024-03-28T22:01:35.480 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.581+0000 7fcb0c3fb700 1 -- 172.21.15.16:0/2979399212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb04106d70 msgr2=0x7fcb0407ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:35.480 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.581+0000 7fcb0c3fb700 1 --2- 172.21.15.16:0/2979399212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb04106d70 0x7fcb0407ba90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcb000117e0 tx=0x7fcb00011c10 comp rx=0 tx=0).stop 2024-03-28T22:01:35.480 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.581+0000 7fcb0c3fb700 1 -- 172.21.15.16:0/2979399212 shutdown_connections 2024-03-28T22:01:35.480 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.581+0000 7fcb0c3fb700 1 --2- 172.21.15.16:0/2979399212 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fcaf0038c00 0x7fcaf003b0c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:35.480 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.581+0000 7fcb0c3fb700 1 --2- 172.21.15.16:0/2979399212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb04106d70 0x7fcb0407ba90 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:35.480 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.581+0000 7fcb0c3fb700 1 -- 172.21.15.16:0/2979399212 >> 172.21.15.16:0/2979399212 conn(0x7fcb04102310 msgr2=0x7fcb04102db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:35.480 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.581+0000 7fcb0c3fb700 1 -- 172.21.15.16:0/2979399212 shutdown_connections 2024-03-28T22:01:35.480 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:34.581+0000 7fcb0c3fb700 1 -- 172.21.15.16:0/2979399212 wait complete. 2024-03-28T22:01:35.486 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T22:01:35.486 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-03-28T22:01:35.494 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T22:01:35.494 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-03-28T22:01:35.494 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-03-28T22:01:35.494 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-28T22:01:35.495 INFO:teuthology.orchestra.run.smithi016.stdout:Ceph Dashboard is now available at: 2024-03-28T22:01:35.495 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:01:35.496 INFO:teuthology.orchestra.run.smithi016.stdout: URL: https://smithi016.front.sepia.ceph.com:8443/ 2024-03-28T22:01:35.496 INFO:teuthology.orchestra.run.smithi016.stdout: User: admin 2024-03-28T22:01:35.496 INFO:teuthology.orchestra.run.smithi016.stdout: Password: 0n5qkdqpaz 2024-03-28T22:01:35.496 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:01:35.496 INFO:teuthology.orchestra.run.smithi016.stdout:Saving cluster configuration to /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config directory 2024-03-28T22:01:35.496 INFO:teuthology.orchestra.run.smithi016.stdout:Enabling autotune for osd_memory_target 2024-03-28T22:01:37.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:36 smithi016 ceph-mon[32374]: mgrmap e13: x(active, since 6s) 2024-03-28T22:01:37.169 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.244+0000 7f0d548d1700 1 Processor -- start 2024-03-28T22:01:37.169 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.244+0000 7f0d548d1700 1 -- start start 2024-03-28T22:01:37.169 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.244+0000 7f0d548d1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d50107f20 0x7f0d50108340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:37.169 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.244+0000 7f0d548d1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0d50108880 con 0x7f0d50107f20 2024-03-28T22:01:37.170 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.244+0000 7f0d4e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d50107f20 0x7f0d50108340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:37.170 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.244+0000 7f0d4e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d50107f20 0x7f0d50108340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39014/0 (socket says 172.21.15.16:39014) 2024-03-28T22:01:37.170 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.244+0000 7f0d4e59c700 1 -- 172.21.15.16:0/1030768273 learned_addr learned my addr 172.21.15.16:0/1030768273 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:37.170 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.244+0000 7f0d4e59c700 1 -- 172.21.15.16:0/1030768273 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d50108fe0 con 0x7f0d50107f20 2024-03-28T22:01:37.170 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.245+0000 7f0d4e59c700 1 --2- 172.21.15.16:0/1030768273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d50107f20 0x7f0d50108340 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0d400098a0 tx=0x7f0d40009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a320382da4123a2b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:37.170 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.245+0000 7f0d4d59a700 1 -- 172.21.15.16:0/1030768273 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d40004160 con 0x7f0d50107f20 2024-03-28T22:01:37.170 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.245+0000 7f0d4d59a700 1 -- 172.21.15.16:0/1030768273 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d40004300 con 0x7f0d50107f20 2024-03-28T22:01:37.170 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.245+0000 7f0d4d59a700 1 -- 172.21.15.16:0/1030768273 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d40011b20 con 0x7f0d50107f20 2024-03-28T22:01:37.170 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.245+0000 7f0d548d1700 1 -- 172.21.15.16:0/1030768273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d50107f20 msgr2=0x7f0d50108340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:37.170 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.245+0000 7f0d548d1700 1 --2- 172.21.15.16:0/1030768273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d50107f20 0x7f0d50108340 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0d400098a0 tx=0x7f0d40009bb0 comp rx=0 tx=0).stop 2024-03-28T22:01:37.170 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.245+0000 7f0d548d1700 1 -- 172.21.15.16:0/1030768273 shutdown_connections 2024-03-28T22:01:37.170 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.246+0000 7f0d548d1700 1 --2- 172.21.15.16:0/1030768273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d50107f20 0x7f0d50108340 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:37.170 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.246+0000 7f0d548d1700 1 -- 172.21.15.16:0/1030768273 >> 172.21.15.16:0/1030768273 conn(0x7f0d501034a0 msgr2=0x7f0d50105900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:37.171 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.246+0000 7f0d548d1700 1 -- 172.21.15.16:0/1030768273 shutdown_connections 2024-03-28T22:01:37.171 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.246+0000 7f0d548d1700 1 -- 172.21.15.16:0/1030768273 wait complete. 2024-03-28T22:01:37.171 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.246+0000 7f0d548d1700 1 Processor -- start 2024-03-28T22:01:37.171 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.246+0000 7f0d548d1700 1 -- start start 2024-03-28T22:01:37.171 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.247+0000 7f0d548d1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d50107f20 0x7f0d5007db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:37.171 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.247+0000 7f0d548d1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0d5010c420 con 0x7f0d50107f20 2024-03-28T22:01:37.171 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.247+0000 7f0d4e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d50107f20 0x7f0d5007db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:37.171 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.247+0000 7f0d4e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d50107f20 0x7f0d5007db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39024/0 (socket says 172.21.15.16:39024) 2024-03-28T22:01:37.171 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.247+0000 7f0d4e59c700 1 -- 172.21.15.16:0/3114084215 learned_addr learned my addr 172.21.15.16:0/3114084215 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:37.171 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.247+0000 7f0d4e59c700 1 -- 172.21.15.16:0/3114084215 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d5007e090 con 0x7f0d50107f20 2024-03-28T22:01:37.171 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.247+0000 7f0d4e59c700 1 --2- 172.21.15.16:0/3114084215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d50107f20 0x7f0d5007db50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0d400117e0 tx=0x7f0d40011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:37.171 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.247+0000 7f0d3f7fe700 1 -- 172.21.15.16:0/3114084215 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d40003ba0 con 0x7f0d50107f20 2024-03-28T22:01:37.171 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.247+0000 7f0d3f7fe700 1 -- 172.21.15.16:0/3114084215 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d4001a910 con 0x7f0d50107f20 2024-03-28T22:01:37.171 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.247+0000 7f0d3f7fe700 1 -- 172.21.15.16:0/3114084215 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d400248f0 con 0x7f0d50107f20 2024-03-28T22:01:37.172 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.247+0000 7f0d548d1700 1 -- 172.21.15.16:0/3114084215 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d5007c280 con 0x7f0d50107f20 2024-03-28T22:01:37.172 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.247+0000 7f0d548d1700 1 -- 172.21.15.16:0/3114084215 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d5007c740 con 0x7f0d50107f20 2024-03-28T22:01:37.172 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.248+0000 7f0d3f7fe700 1 -- 172.21.15.16:0/3114084215 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7f0d40022020 con 0x7f0d50107f20 2024-03-28T22:01:37.172 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.248+0000 7f0d548d1700 1 -- 172.21.15.16:0/3114084215 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d50063870 con 0x7f0d50107f20 2024-03-28T22:01:37.172 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.248+0000 7f0d3f7fe700 1 --2- 172.21.15.16:0/3114084215 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0d38038c90 0x7f0d3803b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:37.172 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.248+0000 7f0d3f7fe700 1 -- 172.21.15.16:0/3114084215 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f0d4004d4b0 con 0x7f0d50107f20 2024-03-28T22:01:37.172 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.249+0000 7f0d4dd9b700 1 --2- 172.21.15.16:0/3114084215 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0d38038c90 0x7f0d3803b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:37.172 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.249+0000 7f0d4dd9b700 1 --2- 172.21.15.16:0/3114084215 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0d38038c90 0x7f0d3803b150 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0d44006fd0 tx=0x7f0d44006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:37.172 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.253+0000 7f0d3f7fe700 1 -- 172.21.15.16:0/3114084215 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0d4001ab70 con 0x7f0d50107f20 2024-03-28T22:01:37.172 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.455+0000 7f0d548d1700 1 -- 172.21.15.16:0/3114084215 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f0d5007c9c0 con 0x7f0d50107f20 2024-03-28T22:01:37.172 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.455+0000 7f0d3f7fe700 1 -- 172.21.15.16:0/3114084215 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f0d40010380 con 0x7f0d50107f20 2024-03-28T22:01:37.172 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.457+0000 7f0d548d1700 1 -- 172.21.15.16:0/3114084215 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0d38038c90 msgr2=0x7f0d3803b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:37.172 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.457+0000 7f0d548d1700 1 --2- 172.21.15.16:0/3114084215 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0d38038c90 0x7f0d3803b150 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0d44006fd0 tx=0x7f0d44006e40 comp rx=0 tx=0).stop 2024-03-28T22:01:37.173 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.457+0000 7f0d548d1700 1 -- 172.21.15.16:0/3114084215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d50107f20 msgr2=0x7f0d5007db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:37.173 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.457+0000 7f0d548d1700 1 --2- 172.21.15.16:0/3114084215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d50107f20 0x7f0d5007db50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0d400117e0 tx=0x7f0d40011ba0 comp rx=0 tx=0).stop 2024-03-28T22:01:37.173 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.457+0000 7f0d548d1700 1 -- 172.21.15.16:0/3114084215 shutdown_connections 2024-03-28T22:01:37.173 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.457+0000 7f0d548d1700 1 --2- 172.21.15.16:0/3114084215 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0d38038c90 0x7f0d3803b150 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:37.173 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.457+0000 7f0d548d1700 1 --2- 172.21.15.16:0/3114084215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d50107f20 0x7f0d5007db50 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:37.173 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.457+0000 7f0d548d1700 1 -- 172.21.15.16:0/3114084215 >> 172.21.15.16:0/3114084215 conn(0x7f0d501034a0 msgr2=0x7f0d50103ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:37.173 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.457+0000 7f0d548d1700 1 -- 172.21.15.16:0/3114084215 shutdown_connections 2024-03-28T22:01:37.173 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:36.457+0000 7f0d548d1700 1 -- 172.21.15.16:0/3114084215 wait complete. 2024-03-28T22:01:38.553 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.608+0000 7f83f0d99700 1 Processor -- start 2024-03-28T22:01:38.553 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.608+0000 7f83f0d99700 1 -- start start 2024-03-28T22:01:38.553 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.609+0000 7f83f0d99700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83ec106d70 0x7f83ec107190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:38.553 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.609+0000 7f83f0d99700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f83ec107760 con 0x7f83ec106d70 2024-03-28T22:01:38.553 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.609+0000 7f83ea59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83ec106d70 0x7f83ec107190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:38.553 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.609+0000 7f83ea59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83ec106d70 0x7f83ec107190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39028/0 (socket says 172.21.15.16:39028) 2024-03-28T22:01:38.553 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.609+0000 7f83ea59c700 1 -- 172.21.15.16:0/2818928978 learned_addr learned my addr 172.21.15.16:0/2818928978 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:38.553 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.609+0000 7f83ea59c700 1 -- 172.21.15.16:0/2818928978 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83ec107f70 con 0x7f83ec106d70 2024-03-28T22:01:38.553 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.610+0000 7f83ea59c700 1 --2- 172.21.15.16:0/2818928978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83ec106d70 0x7f83ec107190 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f83e00098a0 tx=0x7f83e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1af9e3f9ac1f67d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:38.554 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.610+0000 7f83e959a700 1 -- 172.21.15.16:0/2818928978 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f83e0004160 con 0x7f83ec106d70 2024-03-28T22:01:38.554 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.610+0000 7f83e959a700 1 -- 172.21.15.16:0/2818928978 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83e0004300 con 0x7f83ec106d70 2024-03-28T22:01:38.554 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.610+0000 7f83e959a700 1 -- 172.21.15.16:0/2818928978 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f83e0011b20 con 0x7f83ec106d70 2024-03-28T22:01:38.554 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.610+0000 7f83f0d99700 1 -- 172.21.15.16:0/2818928978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83ec106d70 msgr2=0x7f83ec107190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:38.554 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.610+0000 7f83f0d99700 1 --2- 172.21.15.16:0/2818928978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83ec106d70 0x7f83ec107190 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f83e00098a0 tx=0x7f83e0009bb0 comp rx=0 tx=0).stop 2024-03-28T22:01:38.554 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.610+0000 7f83f0d99700 1 -- 172.21.15.16:0/2818928978 shutdown_connections 2024-03-28T22:01:38.554 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.610+0000 7f83f0d99700 1 --2- 172.21.15.16:0/2818928978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83ec106d70 0x7f83ec107190 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:38.554 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.610+0000 7f83f0d99700 1 -- 172.21.15.16:0/2818928978 >> 172.21.15.16:0/2818928978 conn(0x7f83ec102310 msgr2=0x7f83ec104750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:38.554 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.610+0000 7f83f0d99700 1 -- 172.21.15.16:0/2818928978 shutdown_connections 2024-03-28T22:01:38.554 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.611+0000 7f83f0d99700 1 -- 172.21.15.16:0/2818928978 wait complete. 2024-03-28T22:01:38.554 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.611+0000 7f83f0d99700 1 Processor -- start 2024-03-28T22:01:38.554 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.611+0000 7f83f0d99700 1 -- start start 2024-03-28T22:01:38.554 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.611+0000 7f83f0d99700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83ec106d70 0x7f83ec191f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:38.555 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.611+0000 7f83f0d99700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f83ec10b320 con 0x7f83ec106d70 2024-03-28T22:01:38.555 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.612+0000 7f83ea59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83ec106d70 0x7f83ec191f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:38.555 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.612+0000 7f83ea59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83ec106d70 0x7f83ec191f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39042/0 (socket says 172.21.15.16:39042) 2024-03-28T22:01:38.555 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.612+0000 7f83ea59c700 1 -- 172.21.15.16:0/237409721 learned_addr learned my addr 172.21.15.16:0/237409721 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:38.555 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.612+0000 7f83ea59c700 1 -- 172.21.15.16:0/237409721 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83ec192460 con 0x7f83ec106d70 2024-03-28T22:01:38.555 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.612+0000 7f83ea59c700 1 --2- 172.21.15.16:0/237409721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83ec106d70 0x7f83ec191f20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f83e00117e0 tx=0x7f83e0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:38.555 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.612+0000 7f83db7fe700 1 -- 172.21.15.16:0/237409721 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f83e0003bf0 con 0x7f83ec106d70 2024-03-28T22:01:38.555 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.612+0000 7f83f0d99700 1 -- 172.21.15.16:0/237409721 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83ec190650 con 0x7f83ec106d70 2024-03-28T22:01:38.555 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.612+0000 7f83db7fe700 1 -- 172.21.15.16:0/237409721 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83e001a910 con 0x7f83ec106d70 2024-03-28T22:01:38.555 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.612+0000 7f83db7fe700 1 -- 172.21.15.16:0/237409721 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f83e0024970 con 0x7f83ec106d70 2024-03-28T22:01:38.556 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.612+0000 7f83f0d99700 1 -- 172.21.15.16:0/237409721 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83ec190b10 con 0x7f83ec106d70 2024-03-28T22:01:38.556 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.613+0000 7f83db7fe700 1 -- 172.21.15.16:0/237409721 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7f83e0022020 con 0x7f83ec106d70 2024-03-28T22:01:38.556 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.613+0000 7f83f0d99700 1 -- 172.21.15.16:0/237409721 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83ec063870 con 0x7f83ec106d70 2024-03-28T22:01:38.556 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.613+0000 7f83db7fe700 1 --2- 172.21.15.16:0/237409721 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f83d4038c90 0x7f83d403b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:38.556 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.613+0000 7f83db7fe700 1 -- 172.21.15.16:0/237409721 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f83e004d4d0 con 0x7f83ec106d70 2024-03-28T22:01:38.556 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.613+0000 7f83e9d9b700 1 --2- 172.21.15.16:0/237409721 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f83d4038c90 0x7f83d403b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:38.556 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.614+0000 7f83e9d9b700 1 --2- 172.21.15.16:0/237409721 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f83d4038c90 0x7f83d403b150 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f83dc006fd0 tx=0x7f83dc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:38.556 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.618+0000 7f83db7fe700 1 -- 172.21.15.16:0/237409721 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f83e000b3e0 con 0x7f83ec106d70 2024-03-28T22:01:38.556 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.932+0000 7f83f0d99700 1 -- 172.21.15.16:0/237409721 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f83ec195c70 con 0x7f83ec106d70 2024-03-28T22:01:38.556 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.934+0000 7f83db7fe700 1 -- 172.21.15.16:0/237409721 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7f83e001ab30 con 0x7f83ec106d70 2024-03-28T22:01:38.556 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.937+0000 7f83f0d99700 1 -- 172.21.15.16:0/237409721 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f83d4038c90 msgr2=0x7f83d403b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:38.556 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.937+0000 7f83f0d99700 1 --2- 172.21.15.16:0/237409721 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f83d4038c90 0x7f83d403b150 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f83dc006fd0 tx=0x7f83dc006e40 comp rx=0 tx=0).stop 2024-03-28T22:01:38.556 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.937+0000 7f83f0d99700 1 -- 172.21.15.16:0/237409721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83ec106d70 msgr2=0x7f83ec191f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:38.557 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.937+0000 7f83f0d99700 1 --2- 172.21.15.16:0/237409721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83ec106d70 0x7f83ec191f20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f83e00117e0 tx=0x7f83e0011ba0 comp rx=0 tx=0).stop 2024-03-28T22:01:38.557 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.937+0000 7f83f0d99700 1 -- 172.21.15.16:0/237409721 shutdown_connections 2024-03-28T22:01:38.557 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.937+0000 7f83f0d99700 1 --2- 172.21.15.16:0/237409721 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f83d4038c90 0x7f83d403b150 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:38.557 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.937+0000 7f83f0d99700 1 --2- 172.21.15.16:0/237409721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83ec106d70 0x7f83ec191f20 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:38.557 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.937+0000 7f83f0d99700 1 -- 172.21.15.16:0/237409721 >> 172.21.15.16:0/237409721 conn(0x7f83ec102310 msgr2=0x7f83ec103140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:38.557 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.937+0000 7f83f0d99700 1 -- 172.21.15.16:0/237409721 shutdown_connections 2024-03-28T22:01:38.557 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-28T22:01:37.937+0000 7f83f0d99700 1 -- 172.21.15.16:0/237409721 wait complete. 2024-03-28T22:01:38.557 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-28T22:01:38.557 INFO:teuthology.orchestra.run.smithi016.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-28T22:01:38.557 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:01:38.557 INFO:teuthology.orchestra.run.smithi016.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-28T22:01:38.557 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:01:38.557 INFO:teuthology.orchestra.run.smithi016.stdout:Or, if you are only running a single cluster on this host: 2024-03-28T22:01:38.557 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:01:38.558 INFO:teuthology.orchestra.run.smithi016.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-28T22:01:38.558 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:01:38.558 INFO:teuthology.orchestra.run.smithi016.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-28T22:01:38.558 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:01:38.558 INFO:teuthology.orchestra.run.smithi016.stdout: ceph telemetry on 2024-03-28T22:01:38.558 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:01:38.558 INFO:teuthology.orchestra.run.smithi016.stdout:For more information see: 2024-03-28T22:01:38.558 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:01:38.558 INFO:teuthology.orchestra.run.smithi016.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-28T22:01:38.558 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:01:38.558 INFO:teuthology.orchestra.run.smithi016.stdout:Bootstrap complete. 2024-03-28T22:01:38.595 INFO:tasks.cephadm:Fetching config... 2024-03-28T22:01:38.596 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-28T22:01:38.596 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-28T22:01:38.612 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-28T22:01:38.612 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-28T22:01:38.612 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-28T22:01:38.667 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-28T22:01:38.667 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-28T22:01:38.667 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-28T22:01:38.734 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-28T22:01:38.735 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-28T22:01:38.735 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-28T22:01:38.790 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-28T22:01:38.790 DEBUG:teuthology.orchestra.run.smithi016:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqpS+LL1RstHXIb1tCrS18jre6EvgvPhh/oXAEzmEQZB46e/fgm/6qia0uQXMNyq3hE6y4a5IiWOrXZzYt9zwqC5hBpOXkTChog0KCAk9hAvf4QyGZdwLwURSPYPlsGr0pUHum+nKQMw7LUBTR0n7HrrMR2a4faZyq9UPc76DyC4RXtyuT5GXeygB2muIjn4zVTyB/FiS8e6+7E7UJEQYErRyuDu/Mj0hgXxZWelvt1cq4YLJoCfbxREl77DyOCBCyQDRYqrMqxTzWqbuFdZ/3CVndEbzIT5ubqkrBArl092CkD7FZ/M4weUnLGbHBhXA7OYKxdIOSucvo3FbpGDWNGM02KyZ7xiESsrYCzvKEXygOTp4rVhjXuz4rYv8m74v2PTKTNiYiB0SwkHXpMV8tlPcByzcADb0k9kHZzvlzH31Anfx7WVJ7oevTt48HbxvtBS99PVQ/wXYBc2Fl0p/7QsSC6bRMB6OmDr8qGt1Smh7Hpyhy54LdSIWLvwDZE/M= ceph-798d9884-ed4e-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-28T22:01:38.871 INFO:teuthology.orchestra.run.smithi016.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqpS+LL1RstHXIb1tCrS18jre6EvgvPhh/oXAEzmEQZB46e/fgm/6qia0uQXMNyq3hE6y4a5IiWOrXZzYt9zwqC5hBpOXkTChog0KCAk9hAvf4QyGZdwLwURSPYPlsGr0pUHum+nKQMw7LUBTR0n7HrrMR2a4faZyq9UPc76DyC4RXtyuT5GXeygB2muIjn4zVTyB/FiS8e6+7E7UJEQYErRyuDu/Mj0hgXxZWelvt1cq4YLJoCfbxREl77DyOCBCyQDRYqrMqxTzWqbuFdZ/3CVndEbzIT5ubqkrBArl092CkD7FZ/M4weUnLGbHBhXA7OYKxdIOSucvo3FbpGDWNGM02KyZ7xiESsrYCzvKEXygOTp4rVhjXuz4rYv8m74v2PTKTNiYiB0SwkHXpMV8tlPcByzcADb0k9kHZzvlzH31Anfx7WVJ7oevTt48HbxvtBS99PVQ/wXYBc2Fl0p/7QsSC6bRMB6OmDr8qGt1Smh7Hpyhy54LdSIWLvwDZE/M= ceph-798d9884-ed4e-11ee-95d0-87774f69a715 2024-03-28T22:01:38.886 DEBUG:teuthology.orchestra.run.smithi053:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqpS+LL1RstHXIb1tCrS18jre6EvgvPhh/oXAEzmEQZB46e/fgm/6qia0uQXMNyq3hE6y4a5IiWOrXZzYt9zwqC5hBpOXkTChog0KCAk9hAvf4QyGZdwLwURSPYPlsGr0pUHum+nKQMw7LUBTR0n7HrrMR2a4faZyq9UPc76DyC4RXtyuT5GXeygB2muIjn4zVTyB/FiS8e6+7E7UJEQYErRyuDu/Mj0hgXxZWelvt1cq4YLJoCfbxREl77DyOCBCyQDRYqrMqxTzWqbuFdZ/3CVndEbzIT5ubqkrBArl092CkD7FZ/M4weUnLGbHBhXA7OYKxdIOSucvo3FbpGDWNGM02KyZ7xiESsrYCzvKEXygOTp4rVhjXuz4rYv8m74v2PTKTNiYiB0SwkHXpMV8tlPcByzcADb0k9kHZzvlzH31Anfx7WVJ7oevTt48HbxvtBS99PVQ/wXYBc2Fl0p/7QsSC6bRMB6OmDr8qGt1Smh7Hpyhy54LdSIWLvwDZE/M= ceph-798d9884-ed4e-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-28T22:01:38.942 INFO:teuthology.orchestra.run.smithi053.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqpS+LL1RstHXIb1tCrS18jre6EvgvPhh/oXAEzmEQZB46e/fgm/6qia0uQXMNyq3hE6y4a5IiWOrXZzYt9zwqC5hBpOXkTChog0KCAk9hAvf4QyGZdwLwURSPYPlsGr0pUHum+nKQMw7LUBTR0n7HrrMR2a4faZyq9UPc76DyC4RXtyuT5GXeygB2muIjn4zVTyB/FiS8e6+7E7UJEQYErRyuDu/Mj0hgXxZWelvt1cq4YLJoCfbxREl77DyOCBCyQDRYqrMqxTzWqbuFdZ/3CVndEbzIT5ubqkrBArl092CkD7FZ/M4weUnLGbHBhXA7OYKxdIOSucvo3FbpGDWNGM02KyZ7xiESsrYCzvKEXygOTp4rVhjXuz4rYv8m74v2PTKTNiYiB0SwkHXpMV8tlPcByzcADb0k9kHZzvlzH31Anfx7WVJ7oevTt48HbxvtBS99PVQ/wXYBc2Fl0p/7QsSC6bRMB6OmDr8qGt1Smh7Hpyhy54LdSIWLvwDZE/M= ceph-798d9884-ed4e-11ee-95d0-87774f69a715 2024-03-28T22:01:38.962 DEBUG:teuthology.orchestra.run.smithi063:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqpS+LL1RstHXIb1tCrS18jre6EvgvPhh/oXAEzmEQZB46e/fgm/6qia0uQXMNyq3hE6y4a5IiWOrXZzYt9zwqC5hBpOXkTChog0KCAk9hAvf4QyGZdwLwURSPYPlsGr0pUHum+nKQMw7LUBTR0n7HrrMR2a4faZyq9UPc76DyC4RXtyuT5GXeygB2muIjn4zVTyB/FiS8e6+7E7UJEQYErRyuDu/Mj0hgXxZWelvt1cq4YLJoCfbxREl77DyOCBCyQDRYqrMqxTzWqbuFdZ/3CVndEbzIT5ubqkrBArl092CkD7FZ/M4weUnLGbHBhXA7OYKxdIOSucvo3FbpGDWNGM02KyZ7xiESsrYCzvKEXygOTp4rVhjXuz4rYv8m74v2PTKTNiYiB0SwkHXpMV8tlPcByzcADb0k9kHZzvlzH31Anfx7WVJ7oevTt48HbxvtBS99PVQ/wXYBc2Fl0p/7QsSC6bRMB6OmDr8qGt1Smh7Hpyhy54LdSIWLvwDZE/M= ceph-798d9884-ed4e-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-28T22:01:39.018 INFO:teuthology.orchestra.run.smithi063.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqpS+LL1RstHXIb1tCrS18jre6EvgvPhh/oXAEzmEQZB46e/fgm/6qia0uQXMNyq3hE6y4a5IiWOrXZzYt9zwqC5hBpOXkTChog0KCAk9hAvf4QyGZdwLwURSPYPlsGr0pUHum+nKQMw7LUBTR0n7HrrMR2a4faZyq9UPc76DyC4RXtyuT5GXeygB2muIjn4zVTyB/FiS8e6+7E7UJEQYErRyuDu/Mj0hgXxZWelvt1cq4YLJoCfbxREl77DyOCBCyQDRYqrMqxTzWqbuFdZ/3CVndEbzIT5ubqkrBArl092CkD7FZ/M4weUnLGbHBhXA7OYKxdIOSucvo3FbpGDWNGM02KyZ7xiESsrYCzvKEXygOTp4rVhjXuz4rYv8m74v2PTKTNiYiB0SwkHXpMV8tlPcByzcADb0k9kHZzvlzH31Anfx7WVJ7oevTt48HbxvtBS99PVQ/wXYBc2Fl0p/7QsSC6bRMB6OmDr8qGt1Smh7Hpyhy54LdSIWLvwDZE/M= ceph-798d9884-ed4e-11ee-95d0-87774f69a715 2024-03-28T22:01:39.037 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-28T22:01:39.123 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:38 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/237409721' entity='client.admin' 2024-03-28T22:01:39.298 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:01:41.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.456+0000 7f5557712700 1 -- 172.21.15.16:0/76152952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5550101c90 msgr2=0x7f55501020b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:41.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.456+0000 7f5557712700 1 --2- 172.21.15.16:0/76152952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5550101c90 0x7f55501020b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5544009910 tx=0x7f5544009c20 comp rx=0 tx=0).stop 2024-03-28T22:01:41.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.456+0000 7f5557712700 1 -- 172.21.15.16:0/76152952 shutdown_connections 2024-03-28T22:01:41.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.456+0000 7f5557712700 1 --2- 172.21.15.16:0/76152952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5550101c90 0x7f55501020b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.456+0000 7f5557712700 1 -- 172.21.15.16:0/76152952 >> 172.21.15.16:0/76152952 conn(0x7f55500fd230 msgr2=0x7f55500ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.456+0000 7f5557712700 1 -- 172.21.15.16:0/76152952 shutdown_connections 2024-03-28T22:01:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.456+0000 7f5557712700 1 -- 172.21.15.16:0/76152952 wait complete. 2024-03-28T22:01:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.457+0000 7f5557712700 1 Processor -- start 2024-03-28T22:01:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.457+0000 7f5557712700 1 -- start start 2024-03-28T22:01:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.457+0000 7f5557712700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5550101c90 0x7f5550199580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.457+0000 7f5557712700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5550107d20 con 0x7f5550101c90 2024-03-28T22:01:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.457+0000 7f55554ae700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5550101c90 0x7f5550199580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.457+0000 7f55554ae700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5550101c90 0x7f5550199580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34514/0 (socket says 172.21.15.16:34514) 2024-03-28T22:01:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.457+0000 7f55554ae700 1 -- 172.21.15.16:0/4148610799 learned_addr learned my addr 172.21.15.16:0/4148610799 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.457+0000 7f55554ae700 1 -- 172.21.15.16:0/4148610799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5550199ac0 con 0x7f5550101c90 2024-03-28T22:01:41.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.457+0000 7f55554ae700 1 --2- 172.21.15.16:0/4148610799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5550101c90 0x7f5550199580 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5544006010 tx=0x7f5544005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:41.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.458+0000 7f55427fc700 1 -- 172.21.15.16:0/4148610799 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5544027070 con 0x7f5550101c90 2024-03-28T22:01:41.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.458+0000 7f5557712700 1 -- 172.21.15.16:0/4148610799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5550199d50 con 0x7f5550101c90 2024-03-28T22:01:41.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.458+0000 7f55427fc700 1 -- 172.21.15.16:0/4148610799 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f554400be40 con 0x7f5550101c90 2024-03-28T22:01:41.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.458+0000 7f55427fc700 1 -- 172.21.15.16:0/4148610799 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f554400f460 con 0x7f5550101c90 2024-03-28T22:01:41.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.458+0000 7f5557712700 1 -- 172.21.15.16:0/4148610799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f555019ca40 con 0x7f5550101c90 2024-03-28T22:01:41.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.459+0000 7f55427fc700 1 -- 172.21.15.16:0/4148610799 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7f554402a070 con 0x7f5550101c90 2024-03-28T22:01:41.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.459+0000 7f5557712700 1 -- 172.21.15.16:0/4148610799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55501084a0 con 0x7f5550101c90 2024-03-28T22:01:41.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.459+0000 7f55427fc700 1 --2- 172.21.15.16:0/4148610799 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f553c038c40 0x7f553c03b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:41.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.459+0000 7f55427fc700 1 -- 172.21.15.16:0/4148610799 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f554401e030 con 0x7f5550101c90 2024-03-28T22:01:41.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.459+0000 7f5554cad700 1 --2- 172.21.15.16:0/4148610799 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f553c038c40 0x7f553c03b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:41.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.459+0000 7f5554cad700 1 --2- 172.21.15.16:0/4148610799 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f553c038c40 0x7f553c03b100 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f554c006fd0 tx=0x7f554c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:41.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.464+0000 7f55427fc700 1 -- 172.21.15.16:0/4148610799 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5544013070 con 0x7f5550101c90 2024-03-28T22:01:41.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.666+0000 7f5557712700 1 -- 172.21.15.16:0/4148610799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f555002dc40 con 0x7f5550101c90 2024-03-28T22:01:41.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.668+0000 7f55427fc700 1 -- 172.21.15.16:0/4148610799 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f55440310c0 con 0x7f5550101c90 2024-03-28T22:01:41.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.671+0000 7f5557712700 1 -- 172.21.15.16:0/4148610799 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f553c038c40 msgr2=0x7f553c03b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:41.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.671+0000 7f5557712700 1 --2- 172.21.15.16:0/4148610799 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f553c038c40 0x7f553c03b100 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f554c006fd0 tx=0x7f554c006e40 comp rx=0 tx=0).stop 2024-03-28T22:01:41.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.671+0000 7f5557712700 1 -- 172.21.15.16:0/4148610799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5550101c90 msgr2=0x7f5550199580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:41.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.671+0000 7f5557712700 1 --2- 172.21.15.16:0/4148610799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5550101c90 0x7f5550199580 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5544006010 tx=0x7f5544005050 comp rx=0 tx=0).stop 2024-03-28T22:01:41.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.671+0000 7f5557712700 1 -- 172.21.15.16:0/4148610799 shutdown_connections 2024-03-28T22:01:41.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.671+0000 7f5557712700 1 --2- 172.21.15.16:0/4148610799 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f553c038c40 0x7f553c03b100 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:41.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.671+0000 7f5557712700 1 --2- 172.21.15.16:0/4148610799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5550101c90 0x7f5550199580 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:41.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.671+0000 7f5557712700 1 -- 172.21.15.16:0/4148610799 >> 172.21.15.16:0/4148610799 conn(0x7f55500fd230 msgr2=0x7f55500ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:41.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.671+0000 7f5557712700 1 -- 172.21.15.16:0/4148610799 shutdown_connections 2024-03-28T22:01:41.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:41.671+0000 7f5557712700 1 -- 172.21.15.16:0/4148610799 wait complete. 2024-03-28T22:01:41.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:41 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:01:41.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:41 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:01:41.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:41 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi016", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:01:41.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:41 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:01:41.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:41 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:01:41.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:41 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:01:41.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:41 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/4148610799' entity='client.admin' 2024-03-28T22:01:42.267 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-28T22:01:42.267 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-28T22:01:42.505 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:01:42.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:42 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:01:42.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:42 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:01:43.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:43.980+0000 7f74a14b3700 1 -- 172.21.15.16:0/3641696644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f749c101c90 msgr2=0x7f749c1020b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:43.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:43.980+0000 7f74a14b3700 1 --2- 172.21.15.16:0/3641696644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f749c101c90 0x7f749c1020b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f748c009910 tx=0x7f748c009c20 comp rx=0 tx=0).stop 2024-03-28T22:01:43.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:43.980+0000 7f74a14b3700 1 -- 172.21.15.16:0/3641696644 shutdown_connections 2024-03-28T22:01:43.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:43.980+0000 7f74a14b3700 1 --2- 172.21.15.16:0/3641696644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f749c101c90 0x7f749c1020b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:43.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:43.980+0000 7f74a14b3700 1 -- 172.21.15.16:0/3641696644 >> 172.21.15.16:0/3641696644 conn(0x7f749c0fd230 msgr2=0x7f749c0ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:43.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:43.980+0000 7f74a14b3700 1 -- 172.21.15.16:0/3641696644 shutdown_connections 2024-03-28T22:01:43.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:43.980+0000 7f74a14b3700 1 -- 172.21.15.16:0/3641696644 wait complete. 2024-03-28T22:01:43.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:43.981+0000 7f74a14b3700 1 Processor -- start 2024-03-28T22:01:43.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:43.981+0000 7f74a14b3700 1 -- start start 2024-03-28T22:01:43.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:43.981+0000 7f74a14b3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f749c101c90 0x7f749c197420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:43.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:43.981+0000 7f74a14b3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f749c107d20 con 0x7f749c101c90 2024-03-28T22:01:43.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:43.981+0000 7f749affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f749c101c90 0x7f749c197420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:43.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:43.981+0000 7f749affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f749c101c90 0x7f749c197420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34538/0 (socket says 172.21.15.16:34538) 2024-03-28T22:01:43.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:43.981+0000 7f749affd700 1 -- 172.21.15.16:0/3633781537 learned_addr learned my addr 172.21.15.16:0/3633781537 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:43.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:43.982+0000 7f749affd700 1 -- 172.21.15.16:0/3633781537 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f749c197960 con 0x7f749c101c90 2024-03-28T22:01:43.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:43.982+0000 7f749affd700 1 --2- 172.21.15.16:0/3633781537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f749c101c90 0x7f749c197420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f748c006010 tx=0x7f748c005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:43.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:43.982+0000 7f7483fff700 1 -- 172.21.15.16:0/3633781537 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f748c027070 con 0x7f749c101c90 2024-03-28T22:01:43.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:43.982+0000 7f7483fff700 1 -- 172.21.15.16:0/3633781537 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f748c00be40 con 0x7f749c101c90 2024-03-28T22:01:43.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:43.982+0000 7f74a14b3700 1 -- 172.21.15.16:0/3633781537 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f749c197bf0 con 0x7f749c101c90 2024-03-28T22:01:43.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:43.982+0000 7f7483fff700 1 -- 172.21.15.16:0/3633781537 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f748c00f460 con 0x7f749c101c90 2024-03-28T22:01:43.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:43.982+0000 7f74a14b3700 1 -- 172.21.15.16:0/3633781537 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f749c19a890 con 0x7f749c101c90 2024-03-28T22:01:43.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:43.983+0000 7f7483fff700 1 -- 172.21.15.16:0/3633781537 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7f748c02a070 con 0x7f749c101c90 2024-03-28T22:01:43.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:43.983+0000 7f7483fff700 1 --2- 172.21.15.16:0/3633781537 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f7484038ca0 0x7f748403b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:43.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:43.983+0000 7f74a14b3700 1 -- 172.21.15.16:0/3633781537 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f749c1084a0 con 0x7f749c101c90 2024-03-28T22:01:43.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:43.983+0000 7f7483fff700 1 -- 172.21.15.16:0/3633781537 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f748c01e030 con 0x7f749c101c90 2024-03-28T22:01:43.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:43.983+0000 7f749a7fc700 1 --2- 172.21.15.16:0/3633781537 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f7484038ca0 0x7f748403b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:43.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:43.984+0000 7f749a7fc700 1 --2- 172.21.15.16:0/3633781537 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f7484038ca0 0x7f748403b160 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7490006fd0 tx=0x7f7490006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:43.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:43.988+0000 7f7483fff700 1 -- 172.21.15.16:0/3633781537 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f748c013070 con 0x7f749c101c90 2024-03-28T22:01:44.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:44.196+0000 7f74a14b3700 1 -- 172.21.15.16:0/3633781537 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f749c02dbd0 con 0x7f7484038ca0 2024-03-28T22:01:44.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:44.199+0000 7f7483fff700 1 -- 172.21.15.16:0/3633781537 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f749c02dbd0 con 0x7f7484038ca0 2024-03-28T22:01:44.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:44.201+0000 7f74a14b3700 1 -- 172.21.15.16:0/3633781537 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f7484038ca0 msgr2=0x7f748403b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:44.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:44.201+0000 7f74a14b3700 1 --2- 172.21.15.16:0/3633781537 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f7484038ca0 0x7f748403b160 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7490006fd0 tx=0x7f7490006e40 comp rx=0 tx=0).stop 2024-03-28T22:01:44.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:44.201+0000 7f74a14b3700 1 -- 172.21.15.16:0/3633781537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f749c101c90 msgr2=0x7f749c197420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:44.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:44.201+0000 7f74a14b3700 1 --2- 172.21.15.16:0/3633781537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f749c101c90 0x7f749c197420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f748c006010 tx=0x7f748c005050 comp rx=0 tx=0).stop 2024-03-28T22:01:44.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:44.201+0000 7f74a14b3700 1 -- 172.21.15.16:0/3633781537 shutdown_connections 2024-03-28T22:01:44.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:44.201+0000 7f74a14b3700 1 --2- 172.21.15.16:0/3633781537 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f7484038ca0 0x7f748403b160 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:44.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:44.201+0000 7f74a14b3700 1 --2- 172.21.15.16:0/3633781537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f749c101c90 0x7f749c197420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:44.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:44.201+0000 7f74a14b3700 1 -- 172.21.15.16:0/3633781537 >> 172.21.15.16:0/3633781537 conn(0x7f749c0fd230 msgr2=0x7f749c0fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:44.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:44.202+0000 7f74a14b3700 1 -- 172.21.15.16:0/3633781537 shutdown_connections 2024-03-28T22:01:44.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:44.202+0000 7f74a14b3700 1 -- 172.21.15.16:0/3633781537 wait complete. 2024-03-28T22:01:44.951 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi053 2024-03-28T22:01:44.951 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-28T22:01:44.951 DEBUG:teuthology.orchestra.run.smithi053:> dd of=/etc/ceph/ceph.conf 2024-03-28T22:01:44.970 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-28T22:01:44.970 DEBUG:teuthology.orchestra.run.smithi053:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T22:01:45.027 INFO:tasks.cephadm:Adding host smithi053 to orchestrator... 2024-03-28T22:01:45.028 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph orch host add smithi053 2024-03-28T22:01:45.268 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:01:45.389 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:45 smithi016 ceph-mon[32374]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:01:45.389 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:45 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:01:45.389 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:45 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:01:45.389 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:45 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:01:45.389 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:45 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:01:45.389 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:45 smithi016 ceph-mon[32374]: Updating smithi016:/etc/ceph/ceph.conf 2024-03-28T22:01:45.389 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:45 smithi016 ceph-mon[32374]: Updating smithi016:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:01:45.390 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:45 smithi016 ceph-mon[32374]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-03-28T22:01:45.390 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:45 smithi016 ceph-mon[32374]: Updating smithi016:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T22:01:45.390 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:45 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:01:45.390 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:45 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:01:45.390 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:45 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:01:46.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:46.845+0000 7f24527a2700 1 -- 172.21.15.16:0/2011698711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f244c107a60 msgr2=0x7f244c107e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:46.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:46.845+0000 7f24527a2700 1 --2- 172.21.15.16:0/2011698711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f244c107a60 0x7f244c107e40 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2448009910 tx=0x7f2448009c20 comp rx=0 tx=0).stop 2024-03-28T22:01:46.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:46.845+0000 7f24527a2700 1 -- 172.21.15.16:0/2011698711 shutdown_connections 2024-03-28T22:01:46.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:46.845+0000 7f24527a2700 1 --2- 172.21.15.16:0/2011698711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f244c107a60 0x7f244c107e40 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:46.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:46.845+0000 7f24527a2700 1 -- 172.21.15.16:0/2011698711 >> 172.21.15.16:0/2011698711 conn(0x7f244c0fd570 msgr2=0x7f244c0ff990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:46.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:46.845+0000 7f24527a2700 1 -- 172.21.15.16:0/2011698711 shutdown_connections 2024-03-28T22:01:46.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:46.845+0000 7f24527a2700 1 -- 172.21.15.16:0/2011698711 wait complete. 2024-03-28T22:01:46.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:46.846+0000 7f24527a2700 1 Processor -- start 2024-03-28T22:01:46.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:46.846+0000 7f24527a2700 1 -- start start 2024-03-28T22:01:46.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:46.846+0000 7f24527a2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f244c107a60 0x7f244c074540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:46.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:46.846+0000 7f24527a2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f244c10c0a0 con 0x7f244c107a60 2024-03-28T22:01:46.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:46.846+0000 7f24517a0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f244c107a60 0x7f244c074540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:46.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:46.846+0000 7f24517a0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f244c107a60 0x7f244c074540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34556/0 (socket says 172.21.15.16:34556) 2024-03-28T22:01:46.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:46.846+0000 7f24517a0700 1 -- 172.21.15.16:0/896943949 learned_addr learned my addr 172.21.15.16:0/896943949 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:46.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:46.846+0000 7f24517a0700 1 -- 172.21.15.16:0/896943949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f244c076430 con 0x7f244c107a60 2024-03-28T22:01:46.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:46.847+0000 7f24517a0700 1 --2- 172.21.15.16:0/896943949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f244c107a60 0x7f244c074540 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2448005850 tx=0x7f2448005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:46.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:46.847+0000 7f24427fc700 1 -- 172.21.15.16:0/896943949 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2448026070 con 0x7f244c107a60 2024-03-28T22:01:46.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:46.847+0000 7f24427fc700 1 -- 172.21.15.16:0/896943949 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f244800bc30 con 0x7f244c107a60 2024-03-28T22:01:46.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:46.847+0000 7f24427fc700 1 -- 172.21.15.16:0/896943949 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2448003a20 con 0x7f244c107a60 2024-03-28T22:01:46.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:46.847+0000 7f24527a2700 1 -- 172.21.15.16:0/896943949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f244c074ae0 con 0x7f244c107a60 2024-03-28T22:01:46.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:46.847+0000 7f24527a2700 1 -- 172.21.15.16:0/896943949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f244c074fa0 con 0x7f244c107a60 2024-03-28T22:01:46.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:46.848+0000 7f24427fc700 1 -- 172.21.15.16:0/896943949 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7f2448029070 con 0x7f244c107a60 2024-03-28T22:01:46.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:46.848+0000 7f24527a2700 1 -- 172.21.15.16:0/896943949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f244c1031a0 con 0x7f244c107a60 2024-03-28T22:01:46.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:46.848+0000 7f24427fc700 1 --2- 172.21.15.16:0/896943949 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f2438038ca0 0x7f243803b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:46.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:46.848+0000 7f24427fc700 1 -- 172.21.15.16:0/896943949 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f244801e070 con 0x7f244c107a60 2024-03-28T22:01:46.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:46.848+0000 7f2450f9f700 1 --2- 172.21.15.16:0/896943949 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f2438038ca0 0x7f243803b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:46.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:46.848+0000 7f2450f9f700 1 --2- 172.21.15.16:0/896943949 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f2438038ca0 0x7f243803b160 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f243c006fd0 tx=0x7f243c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:46.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:46.853+0000 7f24427fc700 1 -- 172.21.15.16:0/896943949 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2448013070 con 0x7f244c107a60 2024-03-28T22:01:47.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:47.064+0000 7f24527a2700 1 -- 172.21.15.16:0/896943949 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi053", "target": ["mon-mgr", ""]}) -- 0x7f244c075a70 con 0x7f2438038ca0 2024-03-28T22:01:48.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:48 smithi016 ceph-mon[32374]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi053", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:01:48.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:48 smithi016 ceph-mon[32374]: Deploying cephadm binary to smithi053 2024-03-28T22:01:50.378 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:50 smithi016 ceph-mon[32374]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:01:51.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:51.429+0000 7f24427fc700 1 -- 172.21.15.16:0/896943949 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f244c075a70 con 0x7f2438038ca0 2024-03-28T22:01:51.432 INFO:teuthology.orchestra.run.smithi016.stdout:Added host 'smithi053' with addr '172.21.15.53' 2024-03-28T22:01:51.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:51.432+0000 7f24527a2700 1 -- 172.21.15.16:0/896943949 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f2438038ca0 msgr2=0x7f243803b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:51.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:51.432+0000 7f24527a2700 1 --2- 172.21.15.16:0/896943949 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f2438038ca0 0x7f243803b160 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f243c006fd0 tx=0x7f243c006e40 comp rx=0 tx=0).stop 2024-03-28T22:01:51.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:51.432+0000 7f24527a2700 1 -- 172.21.15.16:0/896943949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f244c107a60 msgr2=0x7f244c074540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:51.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:51.432+0000 7f24527a2700 1 --2- 172.21.15.16:0/896943949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f244c107a60 0x7f244c074540 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2448005850 tx=0x7f2448005030 comp rx=0 tx=0).stop 2024-03-28T22:01:51.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:51.432+0000 7f24527a2700 1 -- 172.21.15.16:0/896943949 shutdown_connections 2024-03-28T22:01:51.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:51.432+0000 7f24527a2700 1 --2- 172.21.15.16:0/896943949 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f2438038ca0 0x7f243803b160 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:51.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:51.432+0000 7f24527a2700 1 --2- 172.21.15.16:0/896943949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f244c107a60 0x7f244c074540 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:51.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:51.432+0000 7f24527a2700 1 -- 172.21.15.16:0/896943949 >> 172.21.15.16:0/896943949 conn(0x7f244c0fd570 msgr2=0x7f244c1095b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:51.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:51.432+0000 7f24527a2700 1 -- 172.21.15.16:0/896943949 shutdown_connections 2024-03-28T22:01:51.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:51.432+0000 7f24527a2700 1 -- 172.21.15.16:0/896943949 wait complete. 2024-03-28T22:01:52.023 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-28T22:01:52.267 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:01:52.389 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:52 smithi016 ceph-mon[32374]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:01:52.389 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:52 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:01:52.389 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:52 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:01:52.389 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:52 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:01:53.351 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:53 smithi016 ceph-mon[32374]: Added host smithi053 2024-03-28T22:01:53.352 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:53 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:01:53.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.671+0000 7f513c7d8700 1 -- 172.21.15.16:0/64128543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5134074560 msgr2=0x7f5134074940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:53.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.671+0000 7f513c7d8700 1 --2- 172.21.15.16:0/64128543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5134074560 0x7f5134074940 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f512c00b0a0 tx=0x7f512c00b3b0 comp rx=0 tx=0).stop 2024-03-28T22:01:53.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.671+0000 7f513c7d8700 1 -- 172.21.15.16:0/64128543 shutdown_connections 2024-03-28T22:01:53.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.671+0000 7f513c7d8700 1 --2- 172.21.15.16:0/64128543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5134074560 0x7f5134074940 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:53.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.671+0000 7f513c7d8700 1 -- 172.21.15.16:0/64128543 >> 172.21.15.16:0/64128543 conn(0x7f51340fd730 msgr2=0x7f51340ffb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:53.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.671+0000 7f513c7d8700 1 -- 172.21.15.16:0/64128543 shutdown_connections 2024-03-28T22:01:53.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.671+0000 7f513c7d8700 1 -- 172.21.15.16:0/64128543 wait complete. 2024-03-28T22:01:53.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.672+0000 7f513c7d8700 1 Processor -- start 2024-03-28T22:01:53.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.672+0000 7f513c7d8700 1 -- start start 2024-03-28T22:01:53.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.672+0000 7f513c7d8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5134074560 0x7f513419f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:53.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.672+0000 7f513c7d8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f513410e4b0 con 0x7f5134074560 2024-03-28T22:01:53.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.672+0000 7f513a574700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5134074560 0x7f513419f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:53.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.672+0000 7f513a574700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5134074560 0x7f513419f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48152/0 (socket says 172.21.15.16:48152) 2024-03-28T22:01:53.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.672+0000 7f513a574700 1 -- 172.21.15.16:0/2070188932 learned_addr learned my addr 172.21.15.16:0/2070188932 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:53.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.672+0000 7f513a574700 1 -- 172.21.15.16:0/2070188932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f513419fbb0 con 0x7f5134074560 2024-03-28T22:01:53.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.672+0000 7f513a574700 1 --2- 172.21.15.16:0/2070188932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5134074560 0x7f513419f670 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f512c000f80 tx=0x7f512c00bd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:53.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.673+0000 7f51337fe700 1 -- 172.21.15.16:0/2070188932 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f512c020020 con 0x7f5134074560 2024-03-28T22:01:53.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.673+0000 7f51337fe700 1 -- 172.21.15.16:0/2070188932 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f512c018040 con 0x7f5134074560 2024-03-28T22:01:53.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.673+0000 7f513c7d8700 1 -- 172.21.15.16:0/2070188932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f513419fe40 con 0x7f5134074560 2024-03-28T22:01:53.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.673+0000 7f51337fe700 1 -- 172.21.15.16:0/2070188932 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f512c017070 con 0x7f5134074560 2024-03-28T22:01:53.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.673+0000 7f513c7d8700 1 -- 172.21.15.16:0/2070188932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5134199740 con 0x7f5134074560 2024-03-28T22:01:53.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.674+0000 7f51337fe700 1 -- 172.21.15.16:0/2070188932 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7f512c023020 con 0x7f5134074560 2024-03-28T22:01:53.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.674+0000 7f513c7d8700 1 -- 172.21.15.16:0/2070188932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5134075ca0 con 0x7f5134074560 2024-03-28T22:01:53.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.674+0000 7f51337fe700 1 --2- 172.21.15.16:0/2070188932 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f511c038c90 0x7f511c03b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:53.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.674+0000 7f51337fe700 1 -- 172.21.15.16:0/2070188932 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f512c00f050 con 0x7f5134074560 2024-03-28T22:01:53.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.674+0000 7f5139d73700 1 --2- 172.21.15.16:0/2070188932 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f511c038c90 0x7f511c03b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:53.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.674+0000 7f5139d73700 1 --2- 172.21.15.16:0/2070188932 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f511c038c90 0x7f511c03b150 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5128006fd0 tx=0x7f5128006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:53.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.682+0000 7f51337fe700 1 -- 172.21.15.16:0/2070188932 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f512c02a020 con 0x7f5134074560 2024-03-28T22:01:53.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.891+0000 7f513c7d8700 1 -- 172.21.15.16:0/2070188932 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f513419ac70 con 0x7f511c038c90 2024-03-28T22:01:53.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.892+0000 7f51337fe700 1 -- 172.21.15.16:0/2070188932 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7f513419ac70 con 0x7f511c038c90 2024-03-28T22:01:53.895 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:01:53.895 INFO:teuthology.orchestra.run.smithi016.stdout:[{"addr": "172.21.15.16", "hostname": "smithi016", "labels": [], "status": ""}, {"addr": "172.21.15.53", "hostname": "smithi053", "labels": [], "status": ""}] 2024-03-28T22:01:53.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.894+0000 7f513c7d8700 1 -- 172.21.15.16:0/2070188932 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f511c038c90 msgr2=0x7f511c03b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:53.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.894+0000 7f513c7d8700 1 --2- 172.21.15.16:0/2070188932 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f511c038c90 0x7f511c03b150 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5128006fd0 tx=0x7f5128006e40 comp rx=0 tx=0).stop 2024-03-28T22:01:53.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.894+0000 7f513c7d8700 1 -- 172.21.15.16:0/2070188932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5134074560 msgr2=0x7f513419f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:53.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.894+0000 7f513c7d8700 1 --2- 172.21.15.16:0/2070188932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5134074560 0x7f513419f670 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f512c000f80 tx=0x7f512c00bd20 comp rx=0 tx=0).stop 2024-03-28T22:01:53.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.894+0000 7f513c7d8700 1 -- 172.21.15.16:0/2070188932 shutdown_connections 2024-03-28T22:01:53.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.894+0000 7f513c7d8700 1 --2- 172.21.15.16:0/2070188932 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f511c038c90 0x7f511c03b150 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:53.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.894+0000 7f513c7d8700 1 --2- 172.21.15.16:0/2070188932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5134074560 0x7f513419f670 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:53.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.894+0000 7f513c7d8700 1 -- 172.21.15.16:0/2070188932 >> 172.21.15.16:0/2070188932 conn(0x7f51340fd730 msgr2=0x7f5134105f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:53.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.894+0000 7f513c7d8700 1 -- 172.21.15.16:0/2070188932 shutdown_connections 2024-03-28T22:01:53.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:53.894+0000 7f513c7d8700 1 -- 172.21.15.16:0/2070188932 wait complete. 2024-03-28T22:01:54.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:54 smithi016 ceph-mon[32374]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:01:54.550 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi063 2024-03-28T22:01:54.551 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-28T22:01:54.551 DEBUG:teuthology.orchestra.run.smithi063:> dd of=/etc/ceph/ceph.conf 2024-03-28T22:01:54.569 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-28T22:01:54.570 DEBUG:teuthology.orchestra.run.smithi063:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T22:01:54.628 INFO:tasks.cephadm:Adding host smithi063 to orchestrator... 2024-03-28T22:01:54.629 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph orch host add smithi063 2024-03-28T22:01:54.868 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:01:55.308 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:55 smithi016 ceph-mon[32374]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:01:55.309 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:55 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:01:56.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:56 smithi016 ceph-mon[32374]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:01:56.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:56.360+0000 7f344acbb700 1 -- 172.21.15.16:0/138106959 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3444106c00 msgr2=0x7f3444106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:01:56.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:56.360+0000 7f344acbb700 1 --2- 172.21.15.16:0/138106959 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3444106c00 0x7f3444106fe0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3438009910 tx=0x7f3438009c20 comp rx=0 tx=0).stop 2024-03-28T22:01:56.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:56.360+0000 7f344acbb700 1 -- 172.21.15.16:0/138106959 shutdown_connections 2024-03-28T22:01:56.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:56.360+0000 7f344acbb700 1 --2- 172.21.15.16:0/138106959 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3444106c00 0x7f3444106fe0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:01:56.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:56.360+0000 7f344acbb700 1 -- 172.21.15.16:0/138106959 >> 172.21.15.16:0/138106959 conn(0x7f34440fc640 msgr2=0x7f34440fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:01:56.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:56.360+0000 7f344acbb700 1 -- 172.21.15.16:0/138106959 shutdown_connections 2024-03-28T22:01:56.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:56.360+0000 7f344acbb700 1 -- 172.21.15.16:0/138106959 wait complete. 2024-03-28T22:01:56.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:56.360+0000 7f344acbb700 1 Processor -- start 2024-03-28T22:01:56.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:56.361+0000 7f344acbb700 1 -- start start 2024-03-28T22:01:56.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:56.361+0000 7f344acbb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3444106c00 0x7f344419d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:56.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:56.361+0000 7f344acbb700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f344410b1c0 con 0x7f3444106c00 2024-03-28T22:01:56.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:56.361+0000 7f3448a57700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3444106c00 0x7f344419d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:56.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:56.361+0000 7f3448a57700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3444106c00 0x7f344419d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48164/0 (socket says 172.21.15.16:48164) 2024-03-28T22:01:56.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:56.361+0000 7f3448a57700 1 -- 172.21.15.16:0/1543910422 learned_addr learned my addr 172.21.15.16:0/1543910422 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:01:56.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:56.361+0000 7f3448a57700 1 -- 172.21.15.16:0/1543910422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f344419d960 con 0x7f3444106c00 2024-03-28T22:01:56.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:56.361+0000 7f3448a57700 1 --2- 172.21.15.16:0/1543910422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3444106c00 0x7f344419d420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3438005850 tx=0x7f3438005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:56.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:56.361+0000 7f3441ffb700 1 -- 172.21.15.16:0/1543910422 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3438026070 con 0x7f3444106c00 2024-03-28T22:01:56.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:56.361+0000 7f3441ffb700 1 -- 172.21.15.16:0/1543910422 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f343800bc30 con 0x7f3444106c00 2024-03-28T22:01:56.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:56.361+0000 7f3441ffb700 1 -- 172.21.15.16:0/1543910422 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3438003a20 con 0x7f3444106c00 2024-03-28T22:01:56.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:56.361+0000 7f344acbb700 1 -- 172.21.15.16:0/1543910422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f344419dbf0 con 0x7f3444106c00 2024-03-28T22:01:56.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:56.362+0000 7f344acbb700 1 -- 172.21.15.16:0/1543910422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34441975e0 con 0x7f3444106c00 2024-03-28T22:01:56.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:56.363+0000 7f3441ffb700 1 -- 172.21.15.16:0/1543910422 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7f3438029070 con 0x7f3444106c00 2024-03-28T22:01:56.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:56.363+0000 7f344acbb700 1 -- 172.21.15.16:0/1543910422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3444102200 con 0x7f3444106c00 2024-03-28T22:01:56.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:56.363+0000 7f3441ffb700 1 --2- 172.21.15.16:0/1543910422 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f3434038ca0 0x7f343403b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:01:56.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:56.363+0000 7f3441ffb700 1 -- 172.21.15.16:0/1543910422 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f343801e070 con 0x7f3444106c00 2024-03-28T22:01:56.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:56.363+0000 7f3443fff700 1 --2- 172.21.15.16:0/1543910422 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f3434038ca0 0x7f343403b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:01:56.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:56.363+0000 7f3443fff700 1 --2- 172.21.15.16:0/1543910422 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f3434038ca0 0x7f343403b160 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3430006fd0 tx=0x7f3430006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:01:56.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:56.368+0000 7f3441ffb700 1 -- 172.21.15.16:0/1543910422 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3438013070 con 0x7f3444106c00 2024-03-28T22:01:56.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:01:56.575+0000 7f344acbb700 1 -- 172.21.15.16:0/1543910422 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi063", "target": ["mon-mgr", ""]}) -- 0x7f3444063870 con 0x7f3434038ca0 2024-03-28T22:01:58.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:58 smithi016 ceph-mon[32374]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi063", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:01:58.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:58 smithi016 ceph-mon[32374]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:01:58.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:58 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:01:58.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:58 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:01:58.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:58 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:01:58.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:58 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:01:58.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:58 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi053", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:01:58.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:58 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:01:58.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:58 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:01:59.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:59 smithi016 ceph-mon[32374]: Updating smithi053:/etc/ceph/ceph.conf 2024-03-28T22:01:59.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:59 smithi016 ceph-mon[32374]: Deploying cephadm binary to smithi063 2024-03-28T22:01:59.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:59 smithi016 ceph-mon[32374]: Updating smithi053:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:01:59.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:01:59 smithi016 ceph-mon[32374]: Updating smithi053:/etc/ceph/ceph.client.admin.keyring 2024-03-28T22:02:00.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:00 smithi016 ceph-mon[32374]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:00.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:00.918+0000 7f3441ffb700 1 -- 172.21.15.16:0/1543910422 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f3444063870 con 0x7f3434038ca0 2024-03-28T22:02:00.922 INFO:teuthology.orchestra.run.smithi016.stdout:Added host 'smithi063' with addr '172.21.15.63' 2024-03-28T22:02:00.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:00.921+0000 7f344acbb700 1 -- 172.21.15.16:0/1543910422 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f3434038ca0 msgr2=0x7f343403b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:02:00.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:00.921+0000 7f344acbb700 1 --2- 172.21.15.16:0/1543910422 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f3434038ca0 0x7f343403b160 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3430006fd0 tx=0x7f3430006e40 comp rx=0 tx=0).stop 2024-03-28T22:02:00.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:00.921+0000 7f344acbb700 1 -- 172.21.15.16:0/1543910422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3444106c00 msgr2=0x7f344419d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:02:00.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:00.921+0000 7f344acbb700 1 --2- 172.21.15.16:0/1543910422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3444106c00 0x7f344419d420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3438005850 tx=0x7f3438005030 comp rx=0 tx=0).stop 2024-03-28T22:02:00.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:00.921+0000 7f344acbb700 1 -- 172.21.15.16:0/1543910422 shutdown_connections 2024-03-28T22:02:00.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:00.921+0000 7f344acbb700 1 --2- 172.21.15.16:0/1543910422 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f3434038ca0 0x7f343403b160 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:00.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:00.921+0000 7f344acbb700 1 --2- 172.21.15.16:0/1543910422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3444106c00 0x7f344419d420 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:00.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:00.921+0000 7f344acbb700 1 -- 172.21.15.16:0/1543910422 >> 172.21.15.16:0/1543910422 conn(0x7f34440fc640 msgr2=0x7f344406a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:02:00.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:00.921+0000 7f344acbb700 1 -- 172.21.15.16:0/1543910422 shutdown_connections 2024-03-28T22:02:00.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:00.921+0000 7f344acbb700 1 -- 172.21.15.16:0/1543910422 wait complete. 2024-03-28T22:02:01.516 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-28T22:02:01.756 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:02:01.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:01 smithi016 ceph-mon[32374]: Updating smithi053:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T22:02:01.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:01 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:01.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:01 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:01.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:01 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:01.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:01 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:01.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:01 smithi016 ceph-mon[32374]: Added host smithi063 2024-03-28T22:02:01.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:01 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:02:01.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:01 smithi016 ceph-mon[32374]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:01.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:01 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:03.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:02 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:03.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.253+0000 7fb24500c700 1 -- 172.21.15.16:0/4000772193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb240106c20 msgr2=0x7fb240107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:02:03.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.253+0000 7fb24500c700 1 --2- 172.21.15.16:0/4000772193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb240106c20 0x7fb240107000 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb230009910 tx=0x7fb230009c20 comp rx=0 tx=0).stop 2024-03-28T22:02:03.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.253+0000 7fb24500c700 1 -- 172.21.15.16:0/4000772193 shutdown_connections 2024-03-28T22:02:03.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.253+0000 7fb24500c700 1 --2- 172.21.15.16:0/4000772193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb240106c20 0x7fb240107000 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:03.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.253+0000 7fb24500c700 1 -- 172.21.15.16:0/4000772193 >> 172.21.15.16:0/4000772193 conn(0x7fb2400fbd90 msgr2=0x7fb2400fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:02:03.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.253+0000 7fb24500c700 1 -- 172.21.15.16:0/4000772193 shutdown_connections 2024-03-28T22:02:03.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.253+0000 7fb24500c700 1 -- 172.21.15.16:0/4000772193 wait complete. 2024-03-28T22:02:03.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.254+0000 7fb24500c700 1 Processor -- start 2024-03-28T22:02:03.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.254+0000 7fb24500c700 1 -- start start 2024-03-28T22:02:03.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.254+0000 7fb24500c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb240106c20 0x7fb24019d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:02:03.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.254+0000 7fb24500c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb24010c3a0 con 0x7fb240106c20 2024-03-28T22:02:03.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.254+0000 7fb23ed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb240106c20 0x7fb24019d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:02:03.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.254+0000 7fb23ed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb240106c20 0x7fb24019d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37996/0 (socket says 172.21.15.16:37996) 2024-03-28T22:02:03.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.254+0000 7fb23ed9d700 1 -- 172.21.15.16:0/1784579320 learned_addr learned my addr 172.21.15.16:0/1784579320 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:02:03.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.255+0000 7fb23ed9d700 1 -- 172.21.15.16:0/1784579320 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb24019d910 con 0x7fb240106c20 2024-03-28T22:02:03.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.255+0000 7fb23ed9d700 1 --2- 172.21.15.16:0/1784579320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb240106c20 0x7fb24019d3d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb230005850 tx=0x7fb230005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:02:03.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.255+0000 7fb237fff700 1 -- 172.21.15.16:0/1784579320 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb230026070 con 0x7fb240106c20 2024-03-28T22:02:03.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.255+0000 7fb237fff700 1 -- 172.21.15.16:0/1784579320 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb23000bc30 con 0x7fb240106c20 2024-03-28T22:02:03.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.255+0000 7fb24500c700 1 -- 172.21.15.16:0/1784579320 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb24019dba0 con 0x7fb240106c20 2024-03-28T22:02:03.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.255+0000 7fb237fff700 1 -- 172.21.15.16:0/1784579320 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb230003a20 con 0x7fb240106c20 2024-03-28T22:02:03.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.255+0000 7fb24500c700 1 -- 172.21.15.16:0/1784579320 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb240197590 con 0x7fb240106c20 2024-03-28T22:02:03.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.256+0000 7fb237fff700 1 -- 172.21.15.16:0/1784579320 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7fb230029070 con 0x7fb240106c20 2024-03-28T22:02:03.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.256+0000 7fb24500c700 1 -- 172.21.15.16:0/1784579320 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb240103470 con 0x7fb240106c20 2024-03-28T22:02:03.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.256+0000 7fb237fff700 1 --2- 172.21.15.16:0/1784579320 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb22c038c40 0x7fb22c03b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:02:03.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.256+0000 7fb237fff700 1 -- 172.21.15.16:0/1784579320 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb23001e070 con 0x7fb240106c20 2024-03-28T22:02:03.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.256+0000 7fb23e59c700 1 --2- 172.21.15.16:0/1784579320 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb22c038c40 0x7fb22c03b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:02:03.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.257+0000 7fb23e59c700 1 --2- 172.21.15.16:0/1784579320 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb22c038c40 0x7fb22c03b100 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb228006fd0 tx=0x7fb228006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:02:03.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.261+0000 7fb237fff700 1 -- 172.21.15.16:0/1784579320 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb230013070 con 0x7fb240106c20 2024-03-28T22:02:03.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.471+0000 7fb24500c700 1 -- 172.21.15.16:0/1784579320 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb240198ac0 con 0x7fb22c038c40 2024-03-28T22:02:03.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.472+0000 7fb237fff700 1 -- 172.21.15.16:0/1784579320 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+238 (secure 0 0 0) 0x7fb240198ac0 con 0x7fb22c038c40 2024-03-28T22:02:03.474 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:02:03.474 INFO:teuthology.orchestra.run.smithi016.stdout:[{"addr": "172.21.15.16", "hostname": "smithi016", "labels": [], "status": ""}, {"addr": "172.21.15.53", "hostname": "smithi053", "labels": [], "status": ""}, {"addr": "172.21.15.63", "hostname": "smithi063", "labels": [], "status": ""}] 2024-03-28T22:02:03.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.474+0000 7fb24500c700 1 -- 172.21.15.16:0/1784579320 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb22c038c40 msgr2=0x7fb22c03b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:02:03.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.474+0000 7fb24500c700 1 --2- 172.21.15.16:0/1784579320 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb22c038c40 0x7fb22c03b100 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb228006fd0 tx=0x7fb228006e40 comp rx=0 tx=0).stop 2024-03-28T22:02:03.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.474+0000 7fb24500c700 1 -- 172.21.15.16:0/1784579320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb240106c20 msgr2=0x7fb24019d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:02:03.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.474+0000 7fb24500c700 1 --2- 172.21.15.16:0/1784579320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb240106c20 0x7fb24019d3d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb230005850 tx=0x7fb230005030 comp rx=0 tx=0).stop 2024-03-28T22:02:03.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.474+0000 7fb24500c700 1 -- 172.21.15.16:0/1784579320 shutdown_connections 2024-03-28T22:02:03.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.474+0000 7fb24500c700 1 --2- 172.21.15.16:0/1784579320 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb22c038c40 0x7fb22c03b100 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:03.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.474+0000 7fb24500c700 1 --2- 172.21.15.16:0/1784579320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb240106c20 0x7fb24019d3d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:03.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.474+0000 7fb24500c700 1 -- 172.21.15.16:0/1784579320 >> 172.21.15.16:0/1784579320 conn(0x7fb2400fbd90 msgr2=0x7fb240109700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:02:03.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.474+0000 7fb24500c700 1 -- 172.21.15.16:0/1784579320 shutdown_connections 2024-03-28T22:02:03.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:03.474+0000 7fb24500c700 1 -- 172.21.15.16:0/1784579320 wait complete. 2024-03-28T22:02:04.088 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:03 smithi016 ceph-mon[32374]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:04.088 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:03 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:04.118 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-28T22:02:04.118 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph osd crush tunables default 2024-03-28T22:02:04.356 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:02:05.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:04 smithi016 ceph-mon[32374]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:02:05.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:05.895+0000 7fdac6eed700 1 -- 172.21.15.16:0/1173624920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdac0100510 msgr2=0x7fdac01008f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:02:05.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:05.895+0000 7fdac6eed700 1 --2- 172.21.15.16:0/1173624920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdac0100510 0x7fdac01008f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdab4009910 tx=0x7fdab4009c20 comp rx=0 tx=0).stop 2024-03-28T22:02:05.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:05.895+0000 7fdac6eed700 1 -- 172.21.15.16:0/1173624920 shutdown_connections 2024-03-28T22:02:05.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:05.895+0000 7fdac6eed700 1 --2- 172.21.15.16:0/1173624920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdac0100510 0x7fdac01008f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:05.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:05.895+0000 7fdac6eed700 1 -- 172.21.15.16:0/1173624920 >> 172.21.15.16:0/1173624920 conn(0x7fdac00771f0 msgr2=0x7fdac0077600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:02:05.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:05.895+0000 7fdac6eed700 1 -- 172.21.15.16:0/1173624920 shutdown_connections 2024-03-28T22:02:05.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:05.895+0000 7fdac6eed700 1 -- 172.21.15.16:0/1173624920 wait complete. 2024-03-28T22:02:05.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:05.895+0000 7fdac6eed700 1 Processor -- start 2024-03-28T22:02:05.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:05.896+0000 7fdac6eed700 1 -- start start 2024-03-28T22:02:05.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:05.896+0000 7fdac6eed700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdac0100510 0x7fdac01985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:02:05.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:05.896+0000 7fdac6eed700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdac010a8e0 con 0x7fdac0100510 2024-03-28T22:02:05.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:05.896+0000 7fdac4c89700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdac0100510 0x7fdac01985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:02:05.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:05.896+0000 7fdac4c89700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdac0100510 0x7fdac01985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38016/0 (socket says 172.21.15.16:38016) 2024-03-28T22:02:05.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:05.896+0000 7fdac4c89700 1 -- 172.21.15.16:0/3175355794 learned_addr learned my addr 172.21.15.16:0/3175355794 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:02:05.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:05.896+0000 7fdac4c89700 1 -- 172.21.15.16:0/3175355794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdac0198b20 con 0x7fdac0100510 2024-03-28T22:02:05.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:05.896+0000 7fdac4c89700 1 --2- 172.21.15.16:0/3175355794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdac0100510 0x7fdac01985e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdab4005850 tx=0x7fdab4005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:02:05.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:05.896+0000 7fdabdffb700 1 -- 172.21.15.16:0/3175355794 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdab4026070 con 0x7fdac0100510 2024-03-28T22:02:05.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:05.897+0000 7fdabdffb700 1 -- 172.21.15.16:0/3175355794 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdab400bbd0 con 0x7fdac0100510 2024-03-28T22:02:05.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:05.897+0000 7fdac6eed700 1 -- 172.21.15.16:0/3175355794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdac0198e10 con 0x7fdac0100510 2024-03-28T22:02:05.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:05.897+0000 7fdabdffb700 1 -- 172.21.15.16:0/3175355794 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdab4003970 con 0x7fdac0100510 2024-03-28T22:02:05.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:05.897+0000 7fdac6eed700 1 -- 172.21.15.16:0/3175355794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdac01992d0 con 0x7fdac0100510 2024-03-28T22:02:05.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:05.898+0000 7fdabdffb700 1 -- 172.21.15.16:0/3175355794 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7fdab4029070 con 0x7fdac0100510 2024-03-28T22:02:05.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:05.898+0000 7fdac6eed700 1 -- 172.21.15.16:0/3175355794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdac01019f0 con 0x7fdac0100510 2024-03-28T22:02:05.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:05.898+0000 7fdabdffb700 1 --2- 172.21.15.16:0/3175355794 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdab0038ca0 0x7fdab003b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:02:05.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:05.898+0000 7fdabdffb700 1 -- 172.21.15.16:0/3175355794 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fdab401e070 con 0x7fdac0100510 2024-03-28T22:02:05.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:05.898+0000 7fdabffff700 1 --2- 172.21.15.16:0/3175355794 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdab0038ca0 0x7fdab003b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:02:05.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:05.898+0000 7fdabffff700 1 --2- 172.21.15.16:0/3175355794 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdab0038ca0 0x7fdab003b160 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fdaac006fd0 tx=0x7fdaac006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:02:05.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:05.903+0000 7fdabdffb700 1 -- 172.21.15.16:0/3175355794 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fdab4013070 con 0x7fdac0100510 2024-03-28T22:02:06.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:06.110+0000 7fdac6eed700 1 -- 172.21.15.16:0/3175355794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fdac0063870 con 0x7fdac0100510 2024-03-28T22:02:06.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:05 smithi016 ceph-mon[32374]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:06.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:06.980+0000 7fdabdffb700 1 -- 172.21.15.16:0/3175355794 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7fdab4030090 con 0x7fdac0100510 2024-03-28T22:02:06.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:06.983+0000 7fdac6eed700 1 -- 172.21.15.16:0/3175355794 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdab0038ca0 msgr2=0x7fdab003b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:02:06.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:06.983+0000 7fdac6eed700 1 --2- 172.21.15.16:0/3175355794 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdab0038ca0 0x7fdab003b160 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fdaac006fd0 tx=0x7fdaac006e40 comp rx=0 tx=0).stop 2024-03-28T22:02:06.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:06.983+0000 7fdac6eed700 1 -- 172.21.15.16:0/3175355794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdac0100510 msgr2=0x7fdac01985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:02:06.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:06.983+0000 7fdac6eed700 1 --2- 172.21.15.16:0/3175355794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdac0100510 0x7fdac01985e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdab4005850 tx=0x7fdab4005030 comp rx=0 tx=0).stop 2024-03-28T22:02:06.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:06.983+0000 7fdac6eed700 1 -- 172.21.15.16:0/3175355794 shutdown_connections 2024-03-28T22:02:06.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:06.983+0000 7fdac6eed700 1 --2- 172.21.15.16:0/3175355794 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdab0038ca0 0x7fdab003b160 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:06.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:06.983+0000 7fdac6eed700 1 --2- 172.21.15.16:0/3175355794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdac0100510 0x7fdac01985e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:06.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:06.983+0000 7fdac6eed700 1 -- 172.21.15.16:0/3175355794 >> 172.21.15.16:0/3175355794 conn(0x7fdac00771f0 msgr2=0x7fdac0108080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:02:06.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:06.984+0000 7fdac6eed700 1 -- 172.21.15.16:0/3175355794 shutdown_connections 2024-03-28T22:02:06.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:06.984+0000 7fdac6eed700 1 -- 172.21.15.16:0/3175355794 wait complete. 2024-03-28T22:02:06.987 INFO:teuthology.orchestra.run.smithi016.stderr:adjusted tunables profile to default 2024-03-28T22:02:07.369 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:06 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3175355794' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-28T22:02:07.369 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:06 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:07.370 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:06 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:07.370 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:06 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:07.370 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:06 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:07.370 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:06 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi063", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:02:07.370 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:06 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:07.370 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:06 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:02:07.705 INFO:tasks.cephadm:Adding mon.a on smithi016 2024-03-28T22:02:07.705 INFO:tasks.cephadm:Adding mon.b on smithi053 2024-03-28T22:02:07.705 INFO:tasks.cephadm:Adding mon.c on smithi063 2024-03-28T22:02:07.705 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph orch apply mon '3;smithi016:172.21.15.16=a;smithi053:172.21.15.53=b;smithi063:172.21.15.63=c' 2024-03-28T22:02:07.943 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:02:08.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:07 smithi016 ceph-mon[32374]: Updating smithi063:/etc/ceph/ceph.conf 2024-03-28T22:02:08.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:07 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3175355794' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-28T22:02:08.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:07 smithi016 ceph-mon[32374]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T22:02:08.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:07 smithi016 ceph-mon[32374]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:08.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:07 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:08.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:07 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:08.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:07 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:09.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.342+0000 7ff4ffc4b700 1 -- 172.21.15.63:0/3857811310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4f8101ec0 msgr2=0x7ff4f81022a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:02:09.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.342+0000 7ff4ffc4b700 1 --2- 172.21.15.63:0/3857811310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4f8101ec0 0x7ff4f81022a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8009910 tx=0x7ff4e8009c20 comp rx=0 tx=0).stop 2024-03-28T22:02:09.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.342+0000 7ff4ffc4b700 1 -- 172.21.15.63:0/3857811310 shutdown_connections 2024-03-28T22:02:09.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.342+0000 7ff4ffc4b700 1 --2- 172.21.15.63:0/3857811310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4f8101ec0 0x7ff4f81022a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:09.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.342+0000 7ff4ffc4b700 1 -- 172.21.15.63:0/3857811310 >> 172.21.15.63:0/3857811310 conn(0x7ff4f80fd770 msgr2=0x7ff4f80ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:02:09.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.342+0000 7ff4ffc4b700 1 -- 172.21.15.63:0/3857811310 shutdown_connections 2024-03-28T22:02:09.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.342+0000 7ff4ffc4b700 1 -- 172.21.15.63:0/3857811310 wait complete. 2024-03-28T22:02:09.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.342+0000 7ff4ffc4b700 1 Processor -- start 2024-03-28T22:02:09.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.342+0000 7ff4ffc4b700 1 -- start start 2024-03-28T22:02:09.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.343+0000 7ff4ffc4b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4f8101ec0 0x7ff4f8197750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:02:09.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.343+0000 7ff4ffc4b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f810c290 con 0x7ff4f8101ec0 2024-03-28T22:02:09.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.343+0000 7ff4fd9e7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4f8101ec0 0x7ff4f8197750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:02:09.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.343+0000 7ff4fd9e7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4f8101ec0 0x7ff4f8197750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.63:54104/0 (socket says 172.21.15.63:54104) 2024-03-28T22:02:09.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.343+0000 7ff4fd9e7700 1 -- 172.21.15.63:0/1799101515 learned_addr learned my addr 172.21.15.63:0/1799101515 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-28T22:02:09.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.343+0000 7ff4fd9e7700 1 -- 172.21.15.63:0/1799101515 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4f8197c90 con 0x7ff4f8101ec0 2024-03-28T22:02:09.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.343+0000 7ff4fd9e7700 1 --2- 172.21.15.63:0/1799101515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4f8101ec0 0x7ff4f8197750 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8005850 tx=0x7ff4e8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:02:09.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.343+0000 7ff4eeffd700 1 -- 172.21.15.63:0/1799101515 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff4e8026070 con 0x7ff4f8101ec0 2024-03-28T22:02:09.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.344+0000 7ff4eeffd700 1 -- 172.21.15.63:0/1799101515 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4e800bbd0 con 0x7ff4f8101ec0 2024-03-28T22:02:09.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.344+0000 7ff4ffc4b700 1 -- 172.21.15.63:0/1799101515 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4f8197f80 con 0x7ff4f8101ec0 2024-03-28T22:02:09.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.344+0000 7ff4eeffd700 1 -- 172.21.15.63:0/1799101515 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff4e8003970 con 0x7ff4f8101ec0 2024-03-28T22:02:09.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.344+0000 7ff4ffc4b700 1 -- 172.21.15.63:0/1799101515 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4f819c020 con 0x7ff4f8101ec0 2024-03-28T22:02:09.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.344+0000 7ff4ffc4b700 1 -- 172.21.15.63:0/1799101515 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4f81033a0 con 0x7ff4f8101ec0 2024-03-28T22:02:09.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.345+0000 7ff4eeffd700 1 -- 172.21.15.63:0/1799101515 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7ff4e8029070 con 0x7ff4f8101ec0 2024-03-28T22:02:09.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.345+0000 7ff4eeffd700 1 --2- 172.21.15.63:0/1799101515 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff4e4038c40 0x7ff4e403b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:02:09.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.345+0000 7ff4eeffd700 1 -- 172.21.15.63:0/1799101515 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ff4e801e070 con 0x7ff4f8101ec0 2024-03-28T22:02:09.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.345+0000 7ff4fd1e6700 1 --2- 172.21.15.63:0/1799101515 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff4e4038c40 0x7ff4e403b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:02:09.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.345+0000 7ff4fd1e6700 1 --2- 172.21.15.63:0/1799101515 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff4e4038c40 0x7ff4e403b100 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff4f4006fd0 tx=0x7ff4f4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:02:09.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.350+0000 7ff4eeffd700 1 -- 172.21.15.63:0/1799101515 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff4e8013070 con 0x7ff4f8101ec0 2024-03-28T22:02:09.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:08 smithi016 ceph-mon[32374]: Updating smithi063:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:02:09.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:08 smithi016 ceph-mon[32374]: Updating smithi063:/etc/ceph/ceph.client.admin.keyring 2024-03-28T22:02:09.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:08 smithi016 ceph-mon[32374]: Updating smithi063:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T22:02:09.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.567+0000 7ff4ffc4b700 1 -- 172.21.15.63:0/1799101515 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi016:172.21.15.16=a;smithi053:172.21.15.53=b;smithi063:172.21.15.63=c", "target": ["mon-mgr", ""]}) -- 0x7ff4f8198ab0 con 0x7ff4e4038c40 2024-03-28T22:02:09.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.573+0000 7ff4eeffd700 1 -- 172.21.15.63:0/1799101515 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7ff4f8198ab0 con 0x7ff4e4038c40 2024-03-28T22:02:09.575 INFO:teuthology.orchestra.run.smithi063.stdout:Scheduled mon update... 2024-03-28T22:02:09.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.575+0000 7ff4ffc4b700 1 -- 172.21.15.63:0/1799101515 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff4e4038c40 msgr2=0x7ff4e403b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:02:09.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.575+0000 7ff4ffc4b700 1 --2- 172.21.15.63:0/1799101515 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff4e4038c40 0x7ff4e403b100 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff4f4006fd0 tx=0x7ff4f4006e40 comp rx=0 tx=0).stop 2024-03-28T22:02:09.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.575+0000 7ff4ffc4b700 1 -- 172.21.15.63:0/1799101515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4f8101ec0 msgr2=0x7ff4f8197750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:02:09.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.575+0000 7ff4ffc4b700 1 --2- 172.21.15.63:0/1799101515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4f8101ec0 0x7ff4f8197750 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8005850 tx=0x7ff4e8005030 comp rx=0 tx=0).stop 2024-03-28T22:02:09.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.575+0000 7ff4ffc4b700 1 -- 172.21.15.63:0/1799101515 shutdown_connections 2024-03-28T22:02:09.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.575+0000 7ff4ffc4b700 1 --2- 172.21.15.63:0/1799101515 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff4e4038c40 0x7ff4e403b100 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:09.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.575+0000 7ff4ffc4b700 1 --2- 172.21.15.63:0/1799101515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4f8101ec0 0x7ff4f8197750 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:09.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.575+0000 7ff4ffc4b700 1 -- 172.21.15.63:0/1799101515 >> 172.21.15.63:0/1799101515 conn(0x7ff4f80fd770 msgr2=0x7ff4f8103890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:02:09.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.575+0000 7ff4ffc4b700 1 -- 172.21.15.63:0/1799101515 shutdown_connections 2024-03-28T22:02:09.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:09.575+0000 7ff4ffc4b700 1 -- 172.21.15.63:0/1799101515 wait complete. 2024-03-28T22:02:10.177 DEBUG:teuthology.orchestra.run.smithi053:mon.b> sudo journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mon.b.service 2024-03-28T22:02:10.180 DEBUG:teuthology.orchestra.run.smithi063:mon.c> sudo journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mon.c.service 2024-03-28T22:02:10.182 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-28T22:02:10.182 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-28T22:02:10.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:09 smithi016 ceph-mon[32374]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:10.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:09 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:10.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:09 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:02:10.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:09 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:10.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:09 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:02:10.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:09 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:10.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:09 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T22:02:10.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:09 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:11.011 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:02:11.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:10 smithi016 ceph-mon[32374]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi016:172.21.15.16=a;smithi053:172.21.15.53=b;smithi063:172.21.15.63=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:02:11.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:10 smithi016 ceph-mon[32374]: Saving service mon spec with placement smithi016:172.21.15.16=a;smithi053:172.21.15.53=b;smithi063:172.21.15.63=c;count:3 2024-03-28T22:02:11.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:10 smithi016 ceph-mon[32374]: Deploying daemon mon.c on smithi063 2024-03-28T22:02:12.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:11 smithi016 ceph-mon[32374]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:13.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:13 smithi016 ceph-mon[32374]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:14.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.046+0000 7f285f35f700 1 -- 172.21.15.63:0/1207929067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2858106be0 msgr2=0x7f2858106fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:02:14.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.046+0000 7f285f35f700 1 --2- 172.21.15.63:0/1207929067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2858106be0 0x7f2858106fc0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2848009910 tx=0x7f2848009c20 comp rx=0 tx=0).stop 2024-03-28T22:02:14.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.046+0000 7f285f35f700 1 -- 172.21.15.63:0/1207929067 shutdown_connections 2024-03-28T22:02:14.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.046+0000 7f285f35f700 1 --2- 172.21.15.63:0/1207929067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2858106be0 0x7f2858106fc0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:14.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.046+0000 7f285f35f700 1 -- 172.21.15.63:0/1207929067 >> 172.21.15.63:0/1207929067 conn(0x7f2858077780 msgr2=0x7f28580fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:02:14.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.046+0000 7f285f35f700 1 -- 172.21.15.63:0/1207929067 shutdown_connections 2024-03-28T22:02:14.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.046+0000 7f285f35f700 1 -- 172.21.15.63:0/1207929067 wait complete. 2024-03-28T22:02:14.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.047+0000 7f285f35f700 1 Processor -- start 2024-03-28T22:02:14.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.047+0000 7f285f35f700 1 -- start start 2024-03-28T22:02:14.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.048+0000 7f285f35f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2858106be0 0x7f2858197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:02:14.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.048+0000 7f285f35f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f285810b1a0 con 0x7f2858106be0 2024-03-28T22:02:14.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.048+0000 7f285d0fb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2858106be0 0x7f2858197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:02:14.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.048+0000 7f285d0fb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2858106be0 0x7f2858197700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.63:54110/0 (socket says 172.21.15.63:54110) 2024-03-28T22:02:14.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.048+0000 7f285d0fb700 1 -- 172.21.15.63:0/2237731616 learned_addr learned my addr 172.21.15.63:0/2237731616 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-28T22:02:14.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.048+0000 7f285d0fb700 1 -- 172.21.15.63:0/2237731616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2858197c40 con 0x7f2858106be0 2024-03-28T22:02:14.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.048+0000 7f285d0fb700 1 --2- 172.21.15.63:0/2237731616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2858106be0 0x7f2858197700 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2848006010 tx=0x7f2848005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:02:14.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.049+0000 7f284e7fc700 1 -- 172.21.15.63:0/2237731616 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2848026070 con 0x7f2858106be0 2024-03-28T22:02:14.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.049+0000 7f285f35f700 1 -- 172.21.15.63:0/2237731616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2858197ed0 con 0x7f2858106be0 2024-03-28T22:02:14.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.049+0000 7f285f35f700 1 -- 172.21.15.63:0/2237731616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f285819bfd0 con 0x7f2858106be0 2024-03-28T22:02:14.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.049+0000 7f284e7fc700 1 -- 172.21.15.63:0/2237731616 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f284800bc30 con 0x7f2858106be0 2024-03-28T22:02:14.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.049+0000 7f284e7fc700 1 -- 172.21.15.63:0/2237731616 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2848003950 con 0x7f2858106be0 2024-03-28T22:02:14.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.050+0000 7f285f35f700 1 -- 172.21.15.63:0/2237731616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f283c005330 con 0x7f2858106be0 2024-03-28T22:02:14.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.050+0000 7f284e7fc700 1 -- 172.21.15.63:0/2237731616 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7f2848029070 con 0x7f2858106be0 2024-03-28T22:02:14.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.050+0000 7f284e7fc700 1 --2- 172.21.15.63:0/2237731616 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f2844038c40 0x7f284403b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:02:14.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.050+0000 7f284e7fc700 1 -- 172.21.15.63:0/2237731616 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f284801e070 con 0x7f2858106be0 2024-03-28T22:02:14.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.050+0000 7f285c8fa700 1 --2- 172.21.15.63:0/2237731616 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f2844038c40 0x7f284403b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:02:14.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.051+0000 7f285c8fa700 1 --2- 172.21.15.63:0/2237731616 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f2844038c40 0x7f284403b100 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2854006fd0 tx=0x7f2854006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:02:14.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.056+0000 7f284e7fc700 1 -- 172.21.15.63:0/2237731616 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2848013070 con 0x7f2858106be0 2024-03-28T22:02:14.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.363+0000 7f285f35f700 1 -- 172.21.15.63:0/2237731616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f283c005620 con 0x7f2858106be0 2024-03-28T22:02:14.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.364+0000 7f284e7fc700 1 -- 172.21.15.63:0/2237731616 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f284800ecc0 con 0x7f2858106be0 2024-03-28T22:02:14.367 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-28T22:02:14.367 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":1,"fsid":"798d9884-ed4e-11ee-95d0-87774f69a715","modified":"2024-03-28T22:00:24.266479Z","created":"2024-03-28T22:00:24.266479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-28T22:02:14.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.367+0000 7f285f35f700 1 -- 172.21.15.63:0/2237731616 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f2844038c40 msgr2=0x7f284403b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:02:14.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.367+0000 7f285f35f700 1 --2- 172.21.15.63:0/2237731616 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f2844038c40 0x7f284403b100 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2854006fd0 tx=0x7f2854006e40 comp rx=0 tx=0).stop 2024-03-28T22:02:14.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.367+0000 7f285f35f700 1 -- 172.21.15.63:0/2237731616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2858106be0 msgr2=0x7f2858197700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:02:14.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.367+0000 7f285f35f700 1 --2- 172.21.15.63:0/2237731616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2858106be0 0x7f2858197700 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2848006010 tx=0x7f2848005030 comp rx=0 tx=0).stop 2024-03-28T22:02:14.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.367+0000 7f285f35f700 1 -- 172.21.15.63:0/2237731616 shutdown_connections 2024-03-28T22:02:14.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.367+0000 7f285f35f700 1 --2- 172.21.15.63:0/2237731616 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f2844038c40 0x7f284403b100 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:14.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.367+0000 7f285f35f700 1 --2- 172.21.15.63:0/2237731616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2858106be0 0x7f2858197700 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:14.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.367+0000 7f285f35f700 1 -- 172.21.15.63:0/2237731616 >> 172.21.15.63:0/2237731616 conn(0x7f2858077780 msgr2=0x7f2858069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:02:14.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.367+0000 7f285f35f700 1 -- 172.21.15.63:0/2237731616 shutdown_connections 2024-03-28T22:02:14.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:14.367+0000 7f285f35f700 1 -- 172.21.15.63:0/2237731616 wait complete. 2024-03-28T22:02:14.370 INFO:teuthology.orchestra.run.smithi063.stderr:dumped monmap epoch 1 2024-03-28T22:02:14.689 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:14 smithi016 ceph-mon[32374]: from='client.? 172.21.15.63:0/2237731616' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T22:02:15.519 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:15 smithi063 ceph-mon[36249]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-28T22:02:15.520 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:15 smithi063 bash[36227]: 1283d1c3415addeafafc8d38af49df66cfbc8c716c75f9eb2019ec3fe9823e8a 2024-03-28T22:02:15.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:15 smithi016 ceph-mon[32374]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:15.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:15 smithi063 systemd[1]: Started Ceph mon.c for 798d9884-ed4e-11ee-95d0-87774f69a715. 2024-03-28T22:02:16.557 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-28T22:02:16.557 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-28T22:02:16.815 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T22:02:20.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:20 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:02:20.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:20 smithi063 ceph-mon[36249]: mon.a calling monitor election 2024-03-28T22:02:20.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:20 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:02:20.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:20 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:02:20.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:20 smithi063 ceph-mon[36249]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:20.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:20 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:02:20.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:20 smithi063 ceph-mon[36249]: mon.c calling monitor election 2024-03-28T22:02:20.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:20 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:02:20.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:20 smithi063 ceph-mon[36249]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:20.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:20 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:02:20.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:20 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:02:20.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:20 smithi063 ceph-mon[36249]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T22:02:20.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:20 smithi063 ceph-mon[36249]: monmap epoch 2 2024-03-28T22:02:20.934 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:20 smithi063 ceph-mon[36249]: fsid 798d9884-ed4e-11ee-95d0-87774f69a715 2024-03-28T22:02:20.934 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:20 smithi063 ceph-mon[36249]: last_changed 2024-03-28T22:02:15.495715+0000 2024-03-28T22:02:20.934 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:20 smithi063 ceph-mon[36249]: created 2024-03-28T22:00:24.266479+0000 2024-03-28T22:02:20.934 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:20 smithi063 ceph-mon[36249]: min_mon_release 19 (squid) 2024-03-28T22:02:20.934 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:20 smithi063 ceph-mon[36249]: election_strategy: 1 2024-03-28T22:02:20.934 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:20 smithi063 ceph-mon[36249]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-03-28T22:02:20.934 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:20 smithi063 ceph-mon[36249]: 1: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.c 2024-03-28T22:02:20.934 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:20 smithi063 ceph-mon[36249]: fsmap 2024-03-28T22:02:20.934 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:20 smithi063 ceph-mon[36249]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T22:02:20.934 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:20 smithi063 ceph-mon[36249]: mgrmap e13: x(active, since 51s) 2024-03-28T22:02:20.934 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:20 smithi063 ceph-mon[36249]: overall HEALTH_OK 2024-03-28T22:02:20.934 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:20 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:20.935 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:20 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:20.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:20 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:02:20.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:20 smithi016 ceph-mon[32374]: mon.a calling monitor election 2024-03-28T22:02:20.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:20 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:02:20.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:20 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:02:20.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:20 smithi016 ceph-mon[32374]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:20.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:20 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:02:20.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:20 smithi016 ceph-mon[32374]: mon.c calling monitor election 2024-03-28T22:02:20.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:20 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:02:20.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:20 smithi016 ceph-mon[32374]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:20.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:20 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:02:20.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:20 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:02:20.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:20 smithi016 ceph-mon[32374]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T22:02:20.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:20 smithi016 ceph-mon[32374]: monmap epoch 2 2024-03-28T22:02:20.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:20 smithi016 ceph-mon[32374]: fsid 798d9884-ed4e-11ee-95d0-87774f69a715 2024-03-28T22:02:20.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:20 smithi016 ceph-mon[32374]: last_changed 2024-03-28T22:02:15.495715+0000 2024-03-28T22:02:20.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:20 smithi016 ceph-mon[32374]: created 2024-03-28T22:00:24.266479+0000 2024-03-28T22:02:20.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:20 smithi016 ceph-mon[32374]: min_mon_release 19 (squid) 2024-03-28T22:02:20.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:20 smithi016 ceph-mon[32374]: election_strategy: 1 2024-03-28T22:02:20.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:20 smithi016 ceph-mon[32374]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-03-28T22:02:20.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:20 smithi016 ceph-mon[32374]: 1: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.c 2024-03-28T22:02:20.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:20 smithi016 ceph-mon[32374]: fsmap 2024-03-28T22:02:20.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:20 smithi016 ceph-mon[32374]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T22:02:20.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:20 smithi016 ceph-mon[32374]: mgrmap e13: x(active, since 51s) 2024-03-28T22:02:20.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:20 smithi016 ceph-mon[32374]: overall HEALTH_OK 2024-03-28T22:02:20.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:20 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:20.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:20 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:21.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.223+0000 7feb45506700 1 -- 172.21.15.63:0/2483676883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb280035c0 msgr2=0x7feb28005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:02:21.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.223+0000 7feb45506700 1 --2- 172.21.15.63:0/2483676883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb280035c0 0x7feb28005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7feb40100b70 tx=0x7feb3400feb0 comp rx=0 tx=0).stop 2024-03-28T22:02:21.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.223+0000 7feb45506700 1 -- 172.21.15.63:0/2483676883 shutdown_connections 2024-03-28T22:02:21.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.223+0000 7feb45506700 1 --2- 172.21.15.63:0/2483676883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb280035c0 0x7feb28005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:21.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.223+0000 7feb45506700 1 -- 172.21.15.63:0/2483676883 >> 172.21.15.63:0/2483676883 conn(0x7feb400fc750 msgr2=0x7feb400feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:02:21.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.223+0000 7feb45506700 1 -- 172.21.15.63:0/2483676883 shutdown_connections 2024-03-28T22:02:21.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.223+0000 7feb45506700 1 -- 172.21.15.63:0/2483676883 wait complete. 2024-03-28T22:02:21.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.224+0000 7feb45506700 1 Processor -- start 2024-03-28T22:02:21.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.225+0000 7feb45506700 1 -- start start 2024-03-28T22:02:21.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.225+0000 7feb45506700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb280035c0 0x7feb40102940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:02:21.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.225+0000 7feb45506700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb40102e80 0x7feb40103300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:02:21.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.225+0000 7feb45506700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7feb40108cd0 con 0x7feb280035c0 2024-03-28T22:02:21.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.225+0000 7feb45506700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7feb40108b50 con 0x7feb40102e80 2024-03-28T22:02:21.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.225+0000 7feb3e7fc700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb40102e80 0x7feb40103300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:02:21.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.225+0000 7feb3e7fc700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb40102e80 0x7feb40103300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55686/0 (socket says 172.21.15.63:55686) 2024-03-28T22:02:21.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.225+0000 7feb3e7fc700 1 -- 172.21.15.63:0/1264586286 learned_addr learned my addr 172.21.15.63:0/1264586286 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-28T22:02:21.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.225+0000 7feb3effd700 1 --2- 172.21.15.63:0/1264586286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb280035c0 0x7feb40102940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:02:21.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.225+0000 7feb3e7fc700 1 -- 172.21.15.63:0/1264586286 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb40102e80 msgr2=0x7feb40103300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-28T22:02:21.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.225+0000 7feb3e7fc700 1 -- 172.21.15.63:0/1264586286 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb40102e80 msgr2=0x7feb40103300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-28T22:02:21.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.225+0000 7feb3e7fc700 1 --2- 172.21.15.63:0/1264586286 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb40102e80 0x7feb40103300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T22:02:21.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.225+0000 7feb3e7fc700 1 --2- 172.21.15.63:0/1264586286 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb40102e80 0x7feb40103300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:02:21.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.226+0000 7feb3effd700 1 -- 172.21.15.63:0/1264586286 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb40102e80 msgr2=0x7feb40103300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T22:02:21.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.226+0000 7feb3effd700 1 --2- 172.21.15.63:0/1264586286 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb40102e80 0x7feb40103300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:21.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.226+0000 7feb3effd700 1 -- 172.21.15.63:0/1264586286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb40104860 con 0x7feb280035c0 2024-03-28T22:02:21.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.226+0000 7feb3effd700 1 --2- 172.21.15.63:0/1264586286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb280035c0 0x7feb40102940 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7feb340059d0 tx=0x7feb340057d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:02:21.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.226+0000 7feb27fff700 1 -- 172.21.15.63:0/1264586286 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7feb34004b20 con 0x7feb280035c0 2024-03-28T22:02:21.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.226+0000 7feb27fff700 1 -- 172.21.15.63:0/1264586286 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb34013e40 con 0x7feb280035c0 2024-03-28T22:02:21.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.226+0000 7feb27fff700 1 -- 172.21.15.63:0/1264586286 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7feb34011cd0 con 0x7feb280035c0 2024-03-28T22:02:21.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.226+0000 7feb45506700 1 -- 172.21.15.63:0/1264586286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb401a3b80 con 0x7feb280035c0 2024-03-28T22:02:21.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.226+0000 7feb45506700 1 -- 172.21.15.63:0/1264586286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb401a40c0 con 0x7feb280035c0 2024-03-28T22:02:21.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.227+0000 7feb27fff700 1 -- 172.21.15.63:0/1264586286 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7feb34023070 con 0x7feb280035c0 2024-03-28T22:02:21.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.228+0000 7feb27fff700 1 --2- 172.21.15.63:0/1264586286 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7feb2c038a30 0x7feb2c03aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:02:21.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.227+0000 7feb45506700 1 -- 172.21.15.63:0/1264586286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb40102380 con 0x7feb280035c0 2024-03-28T22:02:21.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.228+0000 7feb27fff700 1 -- 172.21.15.63:0/1264586286 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7feb34020050 con 0x7feb280035c0 2024-03-28T22:02:21.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.228+0000 7feb3e7fc700 1 --2- 172.21.15.63:0/1264586286 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7feb2c038a30 0x7feb2c03aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:02:21.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.228+0000 7feb3e7fc700 1 --2- 172.21.15.63:0/1264586286 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7feb2c038a30 0x7feb2c03aef0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7feb30009180 tx=0x7feb30008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:02:21.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.233+0000 7feb27fff700 1 -- 172.21.15.63:0/1264586286 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7feb34031020 con 0x7feb280035c0 2024-03-28T22:02:21.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.529+0000 7feb45506700 1 -- 172.21.15.63:0/1264586286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7feb4002db30 con 0x7feb280035c0 2024-03-28T22:02:21.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.530+0000 7feb27fff700 1 -- 172.21.15.63:0/1264586286 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1011 (secure 0 0 0) 0x7feb34028030 con 0x7feb280035c0 2024-03-28T22:02:21.532 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-28T22:02:21.532 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":2,"fsid":"798d9884-ed4e-11ee-95d0-87774f69a715","modified":"2024-03-28T22:02:15.495715Z","created":"2024-03-28T22:00:24.266479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-28T22:02:21.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.532+0000 7feb45506700 1 -- 172.21.15.63:0/1264586286 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7feb2c038a30 msgr2=0x7feb2c03aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:02:21.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.532+0000 7feb45506700 1 --2- 172.21.15.63:0/1264586286 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7feb2c038a30 0x7feb2c03aef0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7feb30009180 tx=0x7feb30008040 comp rx=0 tx=0).stop 2024-03-28T22:02:21.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.532+0000 7feb45506700 1 -- 172.21.15.63:0/1264586286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb280035c0 msgr2=0x7feb40102940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:02:21.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.532+0000 7feb45506700 1 --2- 172.21.15.63:0/1264586286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb280035c0 0x7feb40102940 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7feb340059d0 tx=0x7feb340057d0 comp rx=0 tx=0).stop 2024-03-28T22:02:21.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.532+0000 7feb45506700 1 -- 172.21.15.63:0/1264586286 shutdown_connections 2024-03-28T22:02:21.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.532+0000 7feb45506700 1 --2- 172.21.15.63:0/1264586286 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7feb2c038a30 0x7feb2c03aef0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:21.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.532+0000 7feb45506700 1 --2- 172.21.15.63:0/1264586286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb280035c0 0x7feb40102940 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:21.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.532+0000 7feb45506700 1 --2- 172.21.15.63:0/1264586286 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb40102e80 0x7feb40103300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:21.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.532+0000 7feb45506700 1 -- 172.21.15.63:0/1264586286 >> 172.21.15.63:0/1264586286 conn(0x7feb400fc750 msgr2=0x7feb4006ac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:02:21.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.533+0000 7feb45506700 1 -- 172.21.15.63:0/1264586286 shutdown_connections 2024-03-28T22:02:21.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:21.533+0000 7feb45506700 1 -- 172.21.15.63:0/1264586286 wait complete. 2024-03-28T22:02:21.536 INFO:teuthology.orchestra.run.smithi063.stderr:dumped monmap epoch 2 2024-03-28T22:02:21.806 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:21 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:21.806 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:21 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T22:02:21.806 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:21 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:21.806 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:21 smithi063 ceph-mon[36249]: Deploying daemon mon.b on smithi053 2024-03-28T22:02:21.806 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:21 smithi063 ceph-mon[36249]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:21.806 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:21 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:02:21.806 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:21 smithi063 ceph-mon[36249]: from='client.? 172.21.15.63:0/1264586286' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T22:02:21.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:21 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:21.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:21 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T22:02:21.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:21 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:21.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:21 smithi016 ceph-mon[32374]: Deploying daemon mon.b on smithi053 2024-03-28T22:02:21.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:21 smithi016 ceph-mon[32374]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:21.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:21 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:02:21.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:21 smithi016 ceph-mon[32374]: from='client.? 172.21.15.63:0/1264586286' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T22:02:23.086 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-28T22:02:23.086 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-28T22:02:23.342 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T22:02:23.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:23 smithi063 ceph-mon[36249]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:23.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:23 smithi016 ceph-mon[32374]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:24.438 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: load: jerasure load: lrc 2024-03-28T22:02:24.760 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: RocksDB version: 7.9.2 2024-03-28T22:02:24.760 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Git sha 0 2024-03-28T22:02:24.760 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Compile date 2024-03-27 16:17:23 2024-03-28T22:02:24.760 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: DB SUMMARY 2024-03-28T22:02:24.761 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: DB Session ID: TG0BRZG1GFOGX728GOUV 2024-03-28T22:02:24.761 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: CURRENT file: CURRENT 2024-03-28T22:02:24.761 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: IDENTITY file: IDENTITY 2024-03-28T22:02:24.761 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-28T22:02:24.761 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-28T22:02:24.761 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-03-28T22:02:24.761 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.error_if_exists: 0 2024-03-28T22:02:24.761 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.create_if_missing: 0 2024-03-28T22:02:24.761 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.paranoid_checks: 1 2024-03-28T22:02:24.761 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-28T22:02:24.761 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-28T22:02:24.761 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-28T22:02:24.761 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.env: 0x555da0f1fc60 2024-03-28T22:02:24.762 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.fs: PosixFileSystem 2024-03-28T22:02:24.762 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.info_log: 0x555da1f27ae0 2024-03-28T22:02:24.762 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.max_file_opening_threads: 16 2024-03-28T22:02:24.762 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.statistics: (nil) 2024-03-28T22:02:24.762 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.use_fsync: 0 2024-03-28T22:02:24.762 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.max_log_file_size: 0 2024-03-28T22:02:24.762 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-28T22:02:24.762 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-28T22:02:24.762 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.keep_log_file_num: 1000 2024-03-28T22:02:24.762 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.recycle_log_file_num: 0 2024-03-28T22:02:24.762 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.allow_fallocate: 1 2024-03-28T22:02:24.762 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.allow_mmap_reads: 0 2024-03-28T22:02:24.762 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.allow_mmap_writes: 0 2024-03-28T22:02:24.763 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.use_direct_reads: 0 2024-03-28T22:02:24.763 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-28T22:02:24.763 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.create_missing_column_families: 0 2024-03-28T22:02:24.763 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.db_log_dir: 2024-03-28T22:02:24.763 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.wal_dir: 2024-03-28T22:02:24.763 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-28T22:02:24.763 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-28T22:02:24.763 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-28T22:02:24.763 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-28T22:02:24.763 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-28T22:02:24.763 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-28T22:02:24.763 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.advise_random_on_open: 1 2024-03-28T22:02:24.763 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.db_write_buffer_size: 0 2024-03-28T22:02:24.764 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.write_buffer_manager: 0x555da1efa1e0 2024-03-28T22:02:24.764 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-28T22:02:24.764 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-28T22:02:24.764 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-28T22:02:24.764 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.rate_limiter: (nil) 2024-03-28T22:02:24.764 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-28T22:02:24.764 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.wal_recovery_mode: 2 2024-03-28T22:02:24.764 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.enable_thread_tracking: 0 2024-03-28T22:02:24.764 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.enable_pipelined_write: 0 2024-03-28T22:02:24.764 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.unordered_write: 0 2024-03-28T22:02:24.764 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-28T22:02:24.764 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-28T22:02:24.764 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-28T22:02:24.764 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-28T22:02:24.765 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.row_cache: None 2024-03-28T22:02:24.765 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.wal_filter: None 2024-03-28T22:02:24.765 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-28T22:02:24.765 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.allow_ingest_behind: 0 2024-03-28T22:02:24.765 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.two_write_queues: 0 2024-03-28T22:02:24.765 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.manual_wal_flush: 0 2024-03-28T22:02:24.765 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.wal_compression: 0 2024-03-28T22:02:24.765 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.atomic_flush: 0 2024-03-28T22:02:24.765 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-28T22:02:24.765 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-28T22:02:24.765 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-28T22:02:24.765 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.log_readahead_size: 0 2024-03-28T22:02:24.765 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-28T22:02:24.765 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.best_efforts_recovery: 0 2024-03-28T22:02:24.766 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-28T22:02:24.766 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-28T22:02:24.766 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.allow_data_in_errors: 0 2024-03-28T22:02:24.766 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.db_host_id: __hostname__ 2024-03-28T22:02:24.766 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-28T22:02:24.766 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.max_background_jobs: 2 2024-03-28T22:02:24.766 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.max_background_compactions: -1 2024-03-28T22:02:24.766 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.max_subcompactions: 1 2024-03-28T22:02:24.766 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-28T22:02:24.768 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-28T22:02:24.768 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-28T22:02:24.768 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.max_total_wal_size: 0 2024-03-28T22:02:24.768 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-28T22:02:24.768 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-28T22:02:24.768 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-28T22:02:24.768 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-28T22:02:24.768 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.max_open_files: -1 2024-03-28T22:02:24.768 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.bytes_per_sync: 0 2024-03-28T22:02:24.768 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-28T22:02:24.769 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-28T22:02:24.769 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.compaction_readahead_size: 0 2024-03-28T22:02:24.769 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.max_background_flushes: -1 2024-03-28T22:02:24.769 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Compression algorithms supported: 2024-03-28T22:02:24.769 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-28T22:02:24.769 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: kZSTD supported: 0 2024-03-28T22:02:24.769 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: kXpressCompression supported: 0 2024-03-28T22:02:24.769 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: kLZ4HCCompression supported: 1 2024-03-28T22:02:24.769 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: kZlibCompression supported: 1 2024-03-28T22:02:24.769 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: kSnappyCompression supported: 1 2024-03-28T22:02:24.769 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: kLZ4Compression supported: 1 2024-03-28T22:02:24.769 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: kBZip2Compression supported: 0 2024-03-28T22:02:24.769 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-28T22:02:24.769 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-28T22:02:24.769 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-28T22:02:24.770 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-28T22:02:24.770 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-28T22:02:24.770 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.merge_operator: 2024-03-28T22:02:24.770 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.compaction_filter: None 2024-03-28T22:02:24.770 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.compaction_filter_factory: None 2024-03-28T22:02:24.770 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.sst_partitioner_factory: None 2024-03-28T22:02:24.770 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-28T22:02:24.770 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-28T22:02:24.770 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x555da1f27be0) 2024-03-28T22:02:24.770 INFO:journalctl@ceph.mon.b.smithi053.stdout: cache_index_and_filter_blocks: 1 2024-03-28T22:02:24.770 INFO:journalctl@ceph.mon.b.smithi053.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-28T22:02:24.770 INFO:journalctl@ceph.mon.b.smithi053.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-28T22:02:24.770 INFO:journalctl@ceph.mon.b.smithi053.stdout: pin_top_level_index_and_filter: 1 2024-03-28T22:02:24.771 INFO:journalctl@ceph.mon.b.smithi053.stdout: index_type: 0 2024-03-28T22:02:24.771 INFO:journalctl@ceph.mon.b.smithi053.stdout: data_block_index_type: 0 2024-03-28T22:02:24.771 INFO:journalctl@ceph.mon.b.smithi053.stdout: index_shortening: 1 2024-03-28T22:02:24.771 INFO:journalctl@ceph.mon.b.smithi053.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-28T22:02:24.771 INFO:journalctl@ceph.mon.b.smithi053.stdout: checksum: 4 2024-03-28T22:02:24.771 INFO:journalctl@ceph.mon.b.smithi053.stdout: no_block_cache: 0 2024-03-28T22:02:24.771 INFO:journalctl@ceph.mon.b.smithi053.stdout: block_cache: 0x555da1f2b090 2024-03-28T22:02:24.771 INFO:journalctl@ceph.mon.b.smithi053.stdout: block_cache_name: BinnedLRUCache 2024-03-28T22:02:24.771 INFO:journalctl@ceph.mon.b.smithi053.stdout: block_cache_options: 2024-03-28T22:02:24.771 INFO:journalctl@ceph.mon.b.smithi053.stdout: capacity : 536870912 2024-03-28T22:02:24.771 INFO:journalctl@ceph.mon.b.smithi053.stdout: num_shard_bits : 4 2024-03-28T22:02:24.771 INFO:journalctl@ceph.mon.b.smithi053.stdout: strict_capacity_limit : 0 2024-03-28T22:02:24.771 INFO:journalctl@ceph.mon.b.smithi053.stdout: high_pri_pool_ratio: 0.000 2024-03-28T22:02:24.771 INFO:journalctl@ceph.mon.b.smithi053.stdout: block_cache_compressed: (nil) 2024-03-28T22:02:24.772 INFO:journalctl@ceph.mon.b.smithi053.stdout: persistent_cache: (nil) 2024-03-28T22:02:24.772 INFO:journalctl@ceph.mon.b.smithi053.stdout: block_size: 4096 2024-03-28T22:02:24.772 INFO:journalctl@ceph.mon.b.smithi053.stdout: block_size_deviation: 10 2024-03-28T22:02:24.772 INFO:journalctl@ceph.mon.b.smithi053.stdout: block_restart_interval: 16 2024-03-28T22:02:24.772 INFO:journalctl@ceph.mon.b.smithi053.stdout: index_block_restart_interval: 1 2024-03-28T22:02:24.772 INFO:journalctl@ceph.mon.b.smithi053.stdout: metadata_block_size: 4096 2024-03-28T22:02:24.772 INFO:journalctl@ceph.mon.b.smithi053.stdout: partition_filters: 0 2024-03-28T22:02:24.772 INFO:journalctl@ceph.mon.b.smithi053.stdout: use_delta_encoding: 1 2024-03-28T22:02:24.772 INFO:journalctl@ceph.mon.b.smithi053.stdout: filter_policy: bloomfilter 2024-03-28T22:02:24.772 INFO:journalctl@ceph.mon.b.smithi053.stdout: whole_key_filtering: 1 2024-03-28T22:02:24.772 INFO:journalctl@ceph.mon.b.smithi053.stdout: verify_compression: 0 2024-03-28T22:02:24.772 INFO:journalctl@ceph.mon.b.smithi053.stdout: read_amp_bytes_per_bit: 0 2024-03-28T22:02:24.772 INFO:journalctl@ceph.mon.b.smithi053.stdout: format_version: 5 2024-03-28T22:02:24.772 INFO:journalctl@ceph.mon.b.smithi053.stdout: enable_index_compression: 1 2024-03-28T22:02:24.773 INFO:journalctl@ceph.mon.b.smithi053.stdout: block_align: 0 2024-03-28T22:02:24.773 INFO:journalctl@ceph.mon.b.smithi053.stdout: max_auto_readahead_size: 262144 2024-03-28T22:02:24.773 INFO:journalctl@ceph.mon.b.smithi053.stdout: prepopulate_block_cache: 0 2024-03-28T22:02:24.773 INFO:journalctl@ceph.mon.b.smithi053.stdout: initial_auto_readahead_size: 8192 2024-03-28T22:02:24.773 INFO:journalctl@ceph.mon.b.smithi053.stdout: num_file_reads_for_auto_readahead: 2 2024-03-28T22:02:24.773 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.write_buffer_size: 33554432 2024-03-28T22:02:24.773 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.max_write_buffer_number: 2 2024-03-28T22:02:24.773 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.compression: NoCompression 2024-03-28T22:02:24.773 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.bottommost_compression: Disabled 2024-03-28T22:02:24.773 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.prefix_extractor: nullptr 2024-03-28T22:02:24.773 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-28T22:02:24.773 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.num_levels: 7 2024-03-28T22:02:24.773 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-28T22:02:24.773 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-28T22:02:24.773 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-28T22:02:24.774 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-28T22:02:24.774 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-28T22:02:24.774 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-28T22:02:24.774 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-28T22:02:24.774 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-28T22:02:24.774 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-28T22:02:24.774 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-28T22:02:24.774 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-28T22:02:24.774 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-28T22:02:24.774 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-28T22:02:24.774 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.compression_opts.level: 32767 2024-03-28T22:02:24.774 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.compression_opts.strategy: 0 2024-03-28T22:02:24.774 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-28T22:02:24.775 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-28T22:02:24.775 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-28T22:02:24.775 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-28T22:02:24.775 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.compression_opts.enabled: false 2024-03-28T22:02:24.775 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-28T22:02:24.775 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-28T22:02:24.775 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-28T22:02:24.775 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-28T22:02:24.775 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.target_file_size_base: 67108864 2024-03-28T22:02:24.775 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-28T22:02:24.775 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-28T22:02:24.775 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-28T22:02:24.775 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-28T22:02:24.775 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-28T22:02:24.776 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-28T22:02:24.776 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-28T22:02:24.776 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-28T22:02:24.776 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-28T22:02:24.776 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-28T22:02:24.776 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-28T22:02:24.776 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-28T22:02:24.776 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-28T22:02:24.776 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-28T22:02:24.776 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.arena_block_size: 1048576 2024-03-28T22:02:24.776 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-28T22:02:24.776 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-28T22:02:24.776 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.disable_auto_compactions: 0 2024-03-28T22:02:24.776 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-28T22:02:24.777 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-28T22:02:24.777 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-28T22:02:24.777 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-28T22:02:24.777 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-28T22:02:24.777 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-28T22:02:24.777 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-28T22:02:24.777 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-28T22:02:24.777 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-28T22:02:24.777 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-28T22:02:24.777 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.table_properties_collectors: 2024-03-28T22:02:24.777 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.inplace_update_support: 0 2024-03-28T22:02:24.777 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-28T22:02:24.777 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-28T22:02:24.777 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-28T22:02:24.778 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-28T22:02:24.778 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.bloom_locality: 0 2024-03-28T22:02:24.778 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.max_successive_merges: 0 2024-03-28T22:02:24.778 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-28T22:02:24.778 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.paranoid_file_checks: 0 2024-03-28T22:02:24.778 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.force_consistency_checks: 1 2024-03-28T22:02:24.778 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.report_bg_io_stats: 0 2024-03-28T22:02:24.778 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.ttl: 2592000 2024-03-28T22:02:24.778 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-28T22:02:24.778 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-28T22:02:24.778 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-28T22:02:24.778 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.enable_blob_files: false 2024-03-28T22:02:24.778 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.min_blob_size: 0 2024-03-28T22:02:24.779 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.blob_file_size: 268435456 2024-03-28T22:02:24.779 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-28T22:02:24.779 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-28T22:02:24.779 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-28T22:02:24.779 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-28T22:02:24.779 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-28T22:02:24.779 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.blob_file_starting_level: 0 2024-03-28T22:02:24.779 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-28T22:02:24.779 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-28T22:02:24.779 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-28T22:02:24.779 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: aecbf8a1-3bf2-48b8-9b5e-c80c9966547b 2024-03-28T22:02:24.779 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711663344439357, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-28T22:02:24.779 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-28T22:02:24.779 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711663344439766, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711663344, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "aecbf8a1-3bf2-48b8-9b5e-c80c9966547b", "db_session_id": "TG0BRZG1GFOGX728GOUV", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-28T22:02:24.780 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711663344439851, "job": 1, "event": "recovery_finished"} 2024-03-28T22:02:24.780 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-28T22:02:24.780 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-28T22:02:24.780 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x555da201c000 2024-03-28T22:02:24.780 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: DB pointer 0x555da2006000 2024-03-28T22:02:24.780 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-28T22:02:24.780 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-28T22:02:24.780 INFO:journalctl@ceph.mon.b.smithi053.stdout: ** DB Stats ** 2024-03-28T22:02:24.780 INFO:journalctl@ceph.mon.b.smithi053.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-28T22:02:24.780 INFO:journalctl@ceph.mon.b.smithi053.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-28T22:02:24.780 INFO:journalctl@ceph.mon.b.smithi053.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-28T22:02:24.780 INFO:journalctl@ceph.mon.b.smithi053.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-28T22:02:24.780 INFO:journalctl@ceph.mon.b.smithi053.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-28T22:02:24.781 INFO:journalctl@ceph.mon.b.smithi053.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-28T22:02:24.781 INFO:journalctl@ceph.mon.b.smithi053.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-28T22:02:24.781 INFO:journalctl@ceph.mon.b.smithi053.stdout: 2024-03-28T22:02:24.781 INFO:journalctl@ceph.mon.b.smithi053.stdout: ** Compaction Stats [default] ** 2024-03-28T22:02:24.781 INFO:journalctl@ceph.mon.b.smithi053.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-28T22:02:24.781 INFO:journalctl@ceph.mon.b.smithi053.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-28T22:02:24.781 INFO:journalctl@ceph.mon.b.smithi053.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-28T22:02:24.781 INFO:journalctl@ceph.mon.b.smithi053.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-28T22:02:24.781 INFO:journalctl@ceph.mon.b.smithi053.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-28T22:02:24.781 INFO:journalctl@ceph.mon.b.smithi053.stdout: 2024-03-28T22:02:24.782 INFO:journalctl@ceph.mon.b.smithi053.stdout: ** Compaction Stats [default] ** 2024-03-28T22:02:24.782 INFO:journalctl@ceph.mon.b.smithi053.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-28T22:02:24.782 INFO:journalctl@ceph.mon.b.smithi053.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-28T22:02:24.782 INFO:journalctl@ceph.mon.b.smithi053.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-28T22:02:24.782 INFO:journalctl@ceph.mon.b.smithi053.stdout: 2024-03-28T22:02:24.782 INFO:journalctl@ceph.mon.b.smithi053.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-28T22:02:24.782 INFO:journalctl@ceph.mon.b.smithi053.stdout: 2024-03-28T22:02:24.782 INFO:journalctl@ceph.mon.b.smithi053.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-28T22:02:24.782 INFO:journalctl@ceph.mon.b.smithi053.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-28T22:02:24.782 INFO:journalctl@ceph.mon.b.smithi053.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-28T22:02:24.782 INFO:journalctl@ceph.mon.b.smithi053.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-28T22:02:24.782 INFO:journalctl@ceph.mon.b.smithi053.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-28T22:02:24.782 INFO:journalctl@ceph.mon.b.smithi053.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-28T22:02:24.782 INFO:journalctl@ceph.mon.b.smithi053.stdout: Cumulative compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-28T22:02:24.783 INFO:journalctl@ceph.mon.b.smithi053.stdout: Interval compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-28T22:02:24.783 INFO:journalctl@ceph.mon.b.smithi053.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-28T22:02:24.783 INFO:journalctl@ceph.mon.b.smithi053.stdout: Block cache BinnedLRUCache@0x555da1f2b090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-28T22:02:24.783 INFO:journalctl@ceph.mon.b.smithi053.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-28T22:02:24.783 INFO:journalctl@ceph.mon.b.smithi053.stdout: 2024-03-28T22:02:24.783 INFO:journalctl@ceph.mon.b.smithi053.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-28T22:02:24.783 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-28T22:02:24.783 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: using public_addr v2:172.21.15.53:0/0 -> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-03-28T22:02:24.783 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: starting mon.b rank -1 at public addrs [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] at bind addrs [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 798d9884-ed4e-11ee-95d0-87774f69a715 2024-03-28T22:02:24.783 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: mon.b@-1(???) e0 preinit fsid 798d9884-ed4e-11ee-95d0-87774f69a715 2024-03-28T22:02:24.783 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: mon.b@-1(synchronizing).mds e1 new map 2024-03-28T22:02:24.783 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-28T22:02:24.783 INFO:journalctl@ceph.mon.b.smithi053.stdout: e1 2024-03-28T22:02:24.783 INFO:journalctl@ceph.mon.b.smithi053.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-28T22:02:24.784 INFO:journalctl@ceph.mon.b.smithi053.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-28T22:02:24.784 INFO:journalctl@ceph.mon.b.smithi053.stdout: legacy client fscid: -1 2024-03-28T22:02:24.784 INFO:journalctl@ceph.mon.b.smithi053.stdout: 2024-03-28T22:02:24.784 INFO:journalctl@ceph.mon.b.smithi053.stdout: No filesystems configured 2024-03-28T22:02:24.784 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-28T22:02:24.784 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-28T22:02:24.784 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-28T22:02:24.784 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-28T22:02:24.784 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-28T22:02:24.784 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-28T22:02:24.784 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-28T22:02:24.784 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T22:02:24.784 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T22:02:24.785 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T22:02:24.785 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:02:24.785 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:24.785 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:24.785 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi063", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:02:24.785 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:24.785 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:24.785 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:24.785 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:24.785 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:24.785 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi053", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:02:24.785 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:24.785 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:02:24.785 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: Updating smithi053:/etc/ceph/ceph.conf 2024-03-28T22:02:24.785 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: Deploying cephadm binary to smithi063 2024-03-28T22:02:24.786 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: Updating smithi053:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:02:24.786 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: Updating smithi053:/etc/ceph/ceph.client.admin.keyring 2024-03-28T22:02:24.786 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:24.786 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: Updating smithi053:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T22:02:24.786 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:24.786 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:24.786 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:24.786 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:24.786 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: Added host smithi063 2024-03-28T22:02:24.786 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:02:24.786 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:24.786 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:24.786 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:24.786 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:24.787 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:24.787 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:02:24.787 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:24.787 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3175355794' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-28T22:02:24.787 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:24.787 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:24.787 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:24.787 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:24.787 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi063", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:02:24.787 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:24.787 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:02:24.787 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: Updating smithi063:/etc/ceph/ceph.conf 2024-03-28T22:02:24.787 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3175355794' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-28T22:02:24.787 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T22:02:24.788 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:24.788 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:24.788 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:24.788 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:24.788 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: Updating smithi063:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:02:24.788 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: Updating smithi063:/etc/ceph/ceph.client.admin.keyring 2024-03-28T22:02:24.788 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: Updating smithi063:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T22:02:24.788 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:24.788 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:24.788 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:02:24.788 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:24.788 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:02:24.788 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:24.788 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T22:02:24.789 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:24.789 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi016:172.21.15.16=a;smithi053:172.21.15.53=b;smithi063:172.21.15.63=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:02:24.789 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: Saving service mon spec with placement smithi016:172.21.15.16=a;smithi053:172.21.15.53=b;smithi063:172.21.15.63=c;count:3 2024-03-28T22:02:24.789 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: Deploying daemon mon.c on smithi063 2024-03-28T22:02:24.789 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:24.789 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:24.789 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='client.? 172.21.15.63:0/2237731616' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T22:02:24.789 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:24.789 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:02:24.789 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: mon.a calling monitor election 2024-03-28T22:02:24.789 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:02:24.789 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:02:24.789 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:24.789 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:02:24.789 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: mon.c calling monitor election 2024-03-28T22:02:24.790 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:02:24.790 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:24.790 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:02:24.790 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:02:24.790 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T22:02:24.790 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: monmap epoch 2 2024-03-28T22:02:24.790 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: fsid 798d9884-ed4e-11ee-95d0-87774f69a715 2024-03-28T22:02:24.790 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: last_changed 2024-03-28T22:02:15.495715+0000 2024-03-28T22:02:24.790 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: created 2024-03-28T22:00:24.266479+0000 2024-03-28T22:02:24.790 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: min_mon_release 19 (squid) 2024-03-28T22:02:24.790 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: election_strategy: 1 2024-03-28T22:02:24.790 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-03-28T22:02:24.790 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: 1: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.c 2024-03-28T22:02:24.791 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: fsmap 2024-03-28T22:02:24.791 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T22:02:24.791 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: mgrmap e13: x(active, since 51s) 2024-03-28T22:02:24.791 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: overall HEALTH_OK 2024-03-28T22:02:24.791 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:24.791 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:24.791 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:24.791 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T22:02:24.791 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:24.791 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: Deploying daemon mon.b on smithi053 2024-03-28T22:02:24.791 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:24.791 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:02:24.791 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: from='client.? 172.21.15.63:0/1264586286' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T22:02:24.791 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:24.792 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 ceph-mon[35979]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-28T22:02:24.792 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 bash[35959]: 1b0c77331568b2e2a6d2abee909eac4ac1c299df8fc8986de0cedaef3934fed2 2024-03-28T22:02:24.792 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:24 smithi053 systemd[1]: Started Ceph mon.b for 798d9884-ed4e-11ee-95d0-87774f69a715. 2024-03-28T22:02:29.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:29 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:02:29.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:29 smithi053 ceph-mon[35979]: mon.a calling monitor election 2024-03-28T22:02:29.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:29 smithi053 ceph-mon[35979]: mon.c calling monitor election 2024-03-28T22:02:29.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:29 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:02:29.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:29 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:02:29.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:29 smithi053 ceph-mon[35979]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:29.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:29 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:02:29.799 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:29 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:02:29.799 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:29 smithi053 ceph-mon[35979]: mon.b calling monitor election 2024-03-28T22:02:29.799 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:29 smithi053 ceph-mon[35979]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:29.799 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:29 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:02:29.799 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:29 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:02:29.799 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:29 smithi053 ceph-mon[35979]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:29.799 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:29 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:02:29.799 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:29 smithi053 ceph-mon[35979]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-28T22:02:29.799 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:29 smithi053 ceph-mon[35979]: monmap epoch 3 2024-03-28T22:02:29.799 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:29 smithi053 ceph-mon[35979]: fsid 798d9884-ed4e-11ee-95d0-87774f69a715 2024-03-28T22:02:29.799 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:29 smithi053 ceph-mon[35979]: last_changed 2024-03-28T22:02:24.465305+0000 2024-03-28T22:02:29.799 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:29 smithi053 ceph-mon[35979]: created 2024-03-28T22:00:24.266479+0000 2024-03-28T22:02:29.799 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:29 smithi053 ceph-mon[35979]: min_mon_release 19 (squid) 2024-03-28T22:02:29.799 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:29 smithi053 ceph-mon[35979]: election_strategy: 1 2024-03-28T22:02:29.800 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:29 smithi053 ceph-mon[35979]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-03-28T22:02:29.800 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:29 smithi053 ceph-mon[35979]: 1: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.c 2024-03-28T22:02:29.800 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:29 smithi053 ceph-mon[35979]: 2: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-28T22:02:29.800 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:29 smithi053 ceph-mon[35979]: fsmap 2024-03-28T22:02:29.800 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:29 smithi053 ceph-mon[35979]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T22:02:29.800 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:29 smithi053 ceph-mon[35979]: mgrmap e13: x(active, since 60s) 2024-03-28T22:02:29.800 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:29 smithi053 ceph-mon[35979]: overall HEALTH_OK 2024-03-28T22:02:29.800 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:29 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:29.800 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:29 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:29.800 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:29 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:29.800 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:29 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:29.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:29 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:02:29.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:29 smithi063 ceph-mon[36249]: mon.a calling monitor election 2024-03-28T22:02:29.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:29 smithi063 ceph-mon[36249]: mon.c calling monitor election 2024-03-28T22:02:29.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:29 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:02:29.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:29 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:02:29.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:29 smithi063 ceph-mon[36249]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:29.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:29 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:02:29.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:29 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:02:29.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:29 smithi063 ceph-mon[36249]: mon.b calling monitor election 2024-03-28T22:02:29.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:29 smithi063 ceph-mon[36249]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:29.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:29 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:02:29.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:29 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:02:29.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:29 smithi063 ceph-mon[36249]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:29.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:29 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:02:29.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:29 smithi063 ceph-mon[36249]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-28T22:02:29.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:29 smithi063 ceph-mon[36249]: monmap epoch 3 2024-03-28T22:02:29.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:29 smithi063 ceph-mon[36249]: fsid 798d9884-ed4e-11ee-95d0-87774f69a715 2024-03-28T22:02:29.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:29 smithi063 ceph-mon[36249]: last_changed 2024-03-28T22:02:24.465305+0000 2024-03-28T22:02:29.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:29 smithi063 ceph-mon[36249]: created 2024-03-28T22:00:24.266479+0000 2024-03-28T22:02:29.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:29 smithi063 ceph-mon[36249]: min_mon_release 19 (squid) 2024-03-28T22:02:29.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:29 smithi063 ceph-mon[36249]: election_strategy: 1 2024-03-28T22:02:29.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:29 smithi063 ceph-mon[36249]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-03-28T22:02:29.934 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:29 smithi063 ceph-mon[36249]: 1: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.c 2024-03-28T22:02:29.934 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:29 smithi063 ceph-mon[36249]: 2: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-28T22:02:29.934 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:29 smithi063 ceph-mon[36249]: fsmap 2024-03-28T22:02:29.934 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:29 smithi063 ceph-mon[36249]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T22:02:29.934 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:29 smithi063 ceph-mon[36249]: mgrmap e13: x(active, since 60s) 2024-03-28T22:02:29.934 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:29 smithi063 ceph-mon[36249]: overall HEALTH_OK 2024-03-28T22:02:29.934 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:29 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:29.934 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:29 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:29.934 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:29 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:29.934 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:29 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:29.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:29 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:02:29.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:29 smithi016 ceph-mon[32374]: mon.a calling monitor election 2024-03-28T22:02:29.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:29 smithi016 ceph-mon[32374]: mon.c calling monitor election 2024-03-28T22:02:29.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:29 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:02:29.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:29 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:02:29.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:29 smithi016 ceph-mon[32374]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:29.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:29 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:02:29.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:29 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:02:29.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:29 smithi016 ceph-mon[32374]: mon.b calling monitor election 2024-03-28T22:02:29.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:29 smithi016 ceph-mon[32374]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:29.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:29 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:02:29.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:29 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:02:29.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:29 smithi016 ceph-mon[32374]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:29.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:29 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:02:29.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:29 smithi016 ceph-mon[32374]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-28T22:02:29.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:29 smithi016 ceph-mon[32374]: monmap epoch 3 2024-03-28T22:02:29.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:29 smithi016 ceph-mon[32374]: fsid 798d9884-ed4e-11ee-95d0-87774f69a715 2024-03-28T22:02:29.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:29 smithi016 ceph-mon[32374]: last_changed 2024-03-28T22:02:24.465305+0000 2024-03-28T22:02:29.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:29 smithi016 ceph-mon[32374]: created 2024-03-28T22:00:24.266479+0000 2024-03-28T22:02:29.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:29 smithi016 ceph-mon[32374]: min_mon_release 19 (squid) 2024-03-28T22:02:29.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:29 smithi016 ceph-mon[32374]: election_strategy: 1 2024-03-28T22:02:29.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:29 smithi016 ceph-mon[32374]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-03-28T22:02:29.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:29 smithi016 ceph-mon[32374]: 1: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.c 2024-03-28T22:02:29.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:29 smithi016 ceph-mon[32374]: 2: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-28T22:02:29.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:29 smithi016 ceph-mon[32374]: fsmap 2024-03-28T22:02:29.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:29 smithi016 ceph-mon[32374]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T22:02:29.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:29 smithi016 ceph-mon[32374]: mgrmap e13: x(active, since 60s) 2024-03-28T22:02:29.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:29 smithi016 ceph-mon[32374]: overall HEALTH_OK 2024-03-28T22:02:29.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:29 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:29.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:29 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:29.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:29 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:29.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:29 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:30.779 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:30 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:02:30.780 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:30 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:02:30.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:30.777+0000 7fda54523700 1 -- 172.21.15.63:0/3369820283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda38005c90 msgr2=0x7fda38008120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:02:30.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:30.777+0000 7fda54523700 1 --2- 172.21.15.63:0/3369820283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda38005c90 0x7fda38008120 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fda4c102090 tx=0x7fda4800ff20 comp rx=0 tx=0).stop 2024-03-28T22:02:30.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:30.777+0000 7fda54523700 1 -- 172.21.15.63:0/3369820283 shutdown_connections 2024-03-28T22:02:30.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:30.777+0000 7fda54523700 1 --2- 172.21.15.63:0/3369820283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda38005c90 0x7fda38008120 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:30.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:30.777+0000 7fda54523700 1 -- 172.21.15.63:0/3369820283 >> 172.21.15.63:0/3369820283 conn(0x7fda4c0fc6f0 msgr2=0x7fda4c0feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:02:30.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:30.777+0000 7fda54523700 1 -- 172.21.15.63:0/3369820283 shutdown_connections 2024-03-28T22:02:30.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:30.777+0000 7fda54523700 1 -- 172.21.15.63:0/3369820283 wait complete. 2024-03-28T22:02:30.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:30.778+0000 7fda54523700 1 Processor -- start 2024-03-28T22:02:30.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:30.778+0000 7fda54523700 1 -- start start 2024-03-28T22:02:30.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:30.778+0000 7fda54523700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda38005c90 0x7fda4c19d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:02:30.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:30.778+0000 7fda54523700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda4c19dc80 0x7fda4c197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:02:30.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:30.778+0000 7fda54523700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda4c197e90 0x7fda4c198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:02:30.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:30.778+0000 7fda54523700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fda4c108c70 con 0x7fda4c197e90 2024-03-28T22:02:30.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:30.778+0000 7fda54523700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fda4c108af0 con 0x7fda4c19dc80 2024-03-28T22:02:30.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:30.778+0000 7fda54523700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fda4c108df0 con 0x7fda38005c90 2024-03-28T22:02:30.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:30.778+0000 7fda522bf700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda38005c90 0x7fda4c19d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:02:30.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:30.778+0000 7fda522bf700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda38005c90 0x7fda4c19d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38722/0 (socket says 172.21.15.63:38722) 2024-03-28T22:02:30.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:30.778+0000 7fda522bf700 1 -- 172.21.15.63:0/829649816 learned_addr learned my addr 172.21.15.63:0/829649816 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-28T22:02:30.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:30.779+0000 7fda51abe700 1 --2- 172.21.15.63:0/829649816 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda4c19dc80 0x7fda4c197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:02:30.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:30.779+0000 7fda52ac0700 1 --2- 172.21.15.63:0/829649816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda4c197e90 0x7fda4c198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:02:30.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:30.779+0000 7fda522bf700 1 -- 172.21.15.63:0/829649816 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda4c19dc80 msgr2=0x7fda4c197950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:02:30.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:30.779+0000 7fda522bf700 1 --2- 172.21.15.63:0/829649816 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda4c19dc80 0x7fda4c197950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:30.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:30.779+0000 7fda522bf700 1 -- 172.21.15.63:0/829649816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda4c197e90 msgr2=0x7fda4c198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:02:30.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:30.779+0000 7fda522bf700 1 --2- 172.21.15.63:0/829649816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda4c197e90 0x7fda4c198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:30.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:30.779+0000 7fda522bf700 1 -- 172.21.15.63:0/829649816 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda4c1a4610 con 0x7fda38005c90 2024-03-28T22:02:30.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:30.779+0000 7fda522bf700 1 --2- 172.21.15.63:0/829649816 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda38005c90 0x7fda4c19d740 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fda48015040 tx=0x7fda48012c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:02:30.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:30.779+0000 7fda437fe700 1 -- 172.21.15.63:0/829649816 <== mon.1 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda4802b070 con 0x7fda38005c90 2024-03-28T22:02:30.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:30.779+0000 7fda54523700 1 -- 172.21.15.63:0/829649816 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda4c1a47e0 con 0x7fda38005c90 2024-03-28T22:02:30.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:30.779+0000 7fda437fe700 1 -- 172.21.15.63:0/829649816 <== mon.1 v2:172.21.15.63:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda480048e0 con 0x7fda38005c90 2024-03-28T22:02:30.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:30.779+0000 7fda437fe700 1 -- 172.21.15.63:0/829649816 <== mon.1 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda4801d670 con 0x7fda38005c90 2024-03-28T22:02:30.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:30.779+0000 7fda54523700 1 -- 172.21.15.63:0/829649816 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda4c1a4cc0 con 0x7fda38005c90 2024-03-28T22:02:30.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:30.780+0000 7fda437fe700 1 -- 172.21.15.63:0/829649816 <== mon.1 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7fda4802e070 con 0x7fda38005c90 2024-03-28T22:02:30.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:30.780+0000 7fda437fe700 1 --2- 172.21.15.63:0/829649816 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fda300390d0 0x7fda3003b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:02:30.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:30.780+0000 7fda437fe700 1 -- 172.21.15.63:0/829649816 <== mon.1 v2:172.21.15.63:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fda4804ee10 con 0x7fda38005c90 2024-03-28T22:02:30.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:30.780+0000 7fda54523700 1 -- 172.21.15.63:0/829649816 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda4c063870 con 0x7fda38005c90 2024-03-28T22:02:30.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:30.781+0000 7fda51abe700 1 --2- 172.21.15.63:0/829649816 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fda300390d0 0x7fda3003b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:02:30.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:30.781+0000 7fda51abe700 1 --2- 172.21.15.63:0/829649816 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fda300390d0 0x7fda3003b590 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fda3c009ad0 tx=0x7fda3c009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:02:30.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:30.786+0000 7fda437fe700 1 -- 172.21.15.63:0/829649816 <== mon.1 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fda48004a90 con 0x7fda38005c90 2024-03-28T22:02:30.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:30 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:02:30.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:30 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:02:30.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:30 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:02:30.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:30 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:02:31.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:31.076+0000 7fda54523700 1 -- 172.21.15.63:0/829649816 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fda4c02db30 con 0x7fda38005c90 2024-03-28T22:02:31.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:31.077+0000 7fda437fe700 1 -- 172.21.15.63:0/829649816 <== mon.1 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1277 (secure 0 0 0) 0x7fda4800ee10 con 0x7fda38005c90 2024-03-28T22:02:31.079 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-28T22:02:31.079 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":3,"fsid":"798d9884-ed4e-11ee-95d0-87774f69a715","modified":"2024-03-28T22:02:24.465305Z","created":"2024-03-28T22:00:24.266479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-28T22:02:31.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:31.079+0000 7fda54523700 1 -- 172.21.15.63:0/829649816 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fda300390d0 msgr2=0x7fda3003b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:02:31.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:31.079+0000 7fda54523700 1 --2- 172.21.15.63:0/829649816 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fda300390d0 0x7fda3003b590 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fda3c009ad0 tx=0x7fda3c009210 comp rx=0 tx=0).stop 2024-03-28T22:02:31.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:31.079+0000 7fda54523700 1 -- 172.21.15.63:0/829649816 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda38005c90 msgr2=0x7fda4c19d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:02:31.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:31.079+0000 7fda54523700 1 --2- 172.21.15.63:0/829649816 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda38005c90 0x7fda4c19d740 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fda48015040 tx=0x7fda48012c60 comp rx=0 tx=0).stop 2024-03-28T22:02:31.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:31.079+0000 7fda54523700 1 -- 172.21.15.63:0/829649816 shutdown_connections 2024-03-28T22:02:31.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:31.079+0000 7fda54523700 1 --2- 172.21.15.63:0/829649816 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fda300390d0 0x7fda3003b590 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:31.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:31.079+0000 7fda54523700 1 --2- 172.21.15.63:0/829649816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda4c197e90 0x7fda4c198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:31.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:31.079+0000 7fda54523700 1 --2- 172.21.15.63:0/829649816 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda38005c90 0x7fda4c19d740 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:31.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:31.079+0000 7fda54523700 1 --2- 172.21.15.63:0/829649816 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda4c19dc80 0x7fda4c197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:31.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:31.079+0000 7fda54523700 1 -- 172.21.15.63:0/829649816 >> 172.21.15.63:0/829649816 conn(0x7fda4c0fc6f0 msgr2=0x7fda4c18ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:02:31.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:31.079+0000 7fda54523700 1 -- 172.21.15.63:0/829649816 shutdown_connections 2024-03-28T22:02:31.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:31.079+0000 7fda54523700 1 -- 172.21.15.63:0/829649816 wait complete. 2024-03-28T22:02:31.082 INFO:teuthology.orchestra.run.smithi063.stderr:dumped monmap epoch 3 2024-03-28T22:02:31.732 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-28T22:02:31.733 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph config generate-minimal-conf 2024-03-28T22:02:31.976 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:02:32.097 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:31 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.098 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:31 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.098 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:31 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.098 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:31 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.098 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:31 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:32.098 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:31 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:02:32.098 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:31 smithi016 ceph-mon[32374]: Updating smithi016:/etc/ceph/ceph.conf 2024-03-28T22:02:32.098 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:31 smithi016 ceph-mon[32374]: Updating smithi053:/etc/ceph/ceph.conf 2024-03-28T22:02:32.098 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:31 smithi016 ceph-mon[32374]: Updating smithi063:/etc/ceph/ceph.conf 2024-03-28T22:02:32.098 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:31 smithi016 ceph-mon[32374]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:32.098 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:31 smithi016 ceph-mon[32374]: from='client.? 172.21.15.63:0/829649816' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T22:02:32.098 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:31 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.098 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:31 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.098 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:31 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.099 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:31 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.099 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:31 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.099 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:31 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.099 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:31 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.099 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:31 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.099 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:31 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.099 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:31 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.099 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:31 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.099 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:31 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T22:02:32.099 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:31 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T22:02:32.099 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:31 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:32.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:31 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:31 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:31 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:31 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:31 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:32.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:31 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:02:32.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:31 smithi063 ceph-mon[36249]: Updating smithi016:/etc/ceph/ceph.conf 2024-03-28T22:02:32.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:31 smithi063 ceph-mon[36249]: Updating smithi053:/etc/ceph/ceph.conf 2024-03-28T22:02:32.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:31 smithi063 ceph-mon[36249]: Updating smithi063:/etc/ceph/ceph.conf 2024-03-28T22:02:32.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:31 smithi063 ceph-mon[36249]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:32.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:31 smithi063 ceph-mon[36249]: from='client.? 172.21.15.63:0/829649816' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T22:02:32.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:31 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:31 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:31 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:31 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:31 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:31 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:31 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:31 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:31 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:31 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:31 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:31 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T22:02:32.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:31 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T22:02:32.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:31 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:32.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:31 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:31 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:31 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:31 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:31 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:32.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:31 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:02:32.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:31 smithi053 ceph-mon[35979]: Updating smithi016:/etc/ceph/ceph.conf 2024-03-28T22:02:32.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:31 smithi053 ceph-mon[35979]: Updating smithi053:/etc/ceph/ceph.conf 2024-03-28T22:02:32.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:31 smithi053 ceph-mon[35979]: Updating smithi063:/etc/ceph/ceph.conf 2024-03-28T22:02:32.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:31 smithi053 ceph-mon[35979]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:32.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:31 smithi053 ceph-mon[35979]: from='client.? 172.21.15.63:0/829649816' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T22:02:32.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:31 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:31 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:31 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:31 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:31 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:31 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:31 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:31 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:31 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:31 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:31 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:32.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:31 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T22:02:32.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:31 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T22:02:32.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:31 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:33.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:32 smithi016 ceph-mon[32374]: Updating smithi016:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:02:33.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:32 smithi016 ceph-mon[32374]: Updating smithi053:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:02:33.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:32 smithi016 ceph-mon[32374]: Updating smithi063:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:02:33.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:32 smithi016 ceph-mon[32374]: Reconfiguring mon.a (unknown last config time)... 2024-03-28T22:02:33.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:32 smithi016 ceph-mon[32374]: Reconfiguring daemon mon.a on smithi016 2024-03-28T22:02:33.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:32 smithi053 ceph-mon[35979]: Updating smithi016:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:02:33.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:32 smithi053 ceph-mon[35979]: Updating smithi053:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:02:33.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:32 smithi053 ceph-mon[35979]: Updating smithi063:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:02:33.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:32 smithi053 ceph-mon[35979]: Reconfiguring mon.a (unknown last config time)... 2024-03-28T22:02:33.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:32 smithi053 ceph-mon[35979]: Reconfiguring daemon mon.a on smithi016 2024-03-28T22:02:33.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:32 smithi063 ceph-mon[36249]: Updating smithi016:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:02:33.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:32 smithi063 ceph-mon[36249]: Updating smithi053:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:02:33.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:32 smithi063 ceph-mon[36249]: Updating smithi063:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:02:33.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:32 smithi063 ceph-mon[36249]: Reconfiguring mon.a (unknown last config time)... 2024-03-28T22:02:33.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:32 smithi063 ceph-mon[36249]: Reconfiguring daemon mon.a on smithi016 2024-03-28T22:02:34.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:33 smithi016 ceph-mon[32374]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:34.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:33 smithi053 ceph-mon[35979]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:34.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:33 smithi063 ceph-mon[36249]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:34.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.596+0000 7f9f75023700 1 -- 172.21.15.16:0/4093487226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f70101dc0 msgr2=0x7f9f70105e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:02:34.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.596+0000 7f9f75023700 1 --2- 172.21.15.16:0/4093487226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f70101dc0 0x7f9f70105e10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9f600099c0 tx=0x7f9f60009cd0 comp rx=0 tx=0).stop 2024-03-28T22:02:34.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.596+0000 7f9f75023700 1 -- 172.21.15.16:0/4093487226 shutdown_connections 2024-03-28T22:02:34.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.596+0000 7f9f75023700 1 --2- 172.21.15.16:0/4093487226 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f701064f0 0x7f9f7010e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:34.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.596+0000 7f9f75023700 1 --2- 172.21.15.16:0/4093487226 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f70101410 0x7f9f701017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:34.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.596+0000 7f9f75023700 1 --2- 172.21.15.16:0/4093487226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f70101dc0 0x7f9f70105e10 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:34.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.596+0000 7f9f75023700 1 -- 172.21.15.16:0/4093487226 >> 172.21.15.16:0/4093487226 conn(0x7f9f70078270 msgr2=0x7f9f70078680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:02:34.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.596+0000 7f9f75023700 1 -- 172.21.15.16:0/4093487226 shutdown_connections 2024-03-28T22:02:34.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.596+0000 7f9f75023700 1 -- 172.21.15.16:0/4093487226 wait complete. 2024-03-28T22:02:34.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.597+0000 7f9f75023700 1 Processor -- start 2024-03-28T22:02:34.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.597+0000 7f9f75023700 1 -- start start 2024-03-28T22:02:34.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.597+0000 7f9f75023700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f70101410 0x7f9f7019e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:02:34.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.597+0000 7f9f75023700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f70101dc0 0x7f9f7019ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:02:34.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.597+0000 7f9f75023700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f701064f0 0x7f9f701988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:02:34.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.597+0000 7f9f75023700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9f70111500 con 0x7f9f70101dc0 2024-03-28T22:02:34.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.597+0000 7f9f75023700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9f70111380 con 0x7f9f701064f0 2024-03-28T22:02:34.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.597+0000 7f9f75023700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9f70111680 con 0x7f9f70101410 2024-03-28T22:02:34.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.598+0000 7f9f6e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f70101dc0 0x7f9f7019ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:02:34.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.598+0000 7f9f6e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f70101dc0 0x7f9f7019ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49468/0 (socket says 172.21.15.16:49468) 2024-03-28T22:02:34.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.598+0000 7f9f6e59c700 1 -- 172.21.15.16:0/29518337 learned_addr learned my addr 172.21.15.16:0/29518337 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:02:34.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.598+0000 7f9f6ed9d700 1 --2- 172.21.15.16:0/29518337 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f70101410 0x7f9f7019e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:02:34.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.598+0000 7f9f6f59e700 1 --2- 172.21.15.16:0/29518337 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f701064f0 0x7f9f701988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:02:34.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.598+0000 7f9f6e59c700 1 -- 172.21.15.16:0/29518337 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f70101410 msgr2=0x7f9f7019e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:02:34.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.598+0000 7f9f6e59c700 1 --2- 172.21.15.16:0/29518337 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f70101410 0x7f9f7019e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:34.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.598+0000 7f9f6e59c700 1 -- 172.21.15.16:0/29518337 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f701064f0 msgr2=0x7f9f701988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:02:34.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.598+0000 7f9f6e59c700 1 --2- 172.21.15.16:0/29518337 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f701064f0 0x7f9f701988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:34.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.598+0000 7f9f6e59c700 1 -- 172.21.15.16:0/29518337 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f70199040 con 0x7f9f70101dc0 2024-03-28T22:02:34.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.598+0000 7f9f6e59c700 1 --2- 172.21.15.16:0/29518337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f70101dc0 0x7f9f7019ec60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9f60003e00 tx=0x7f9f60003e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:02:34.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.598+0000 7f9f57fff700 1 -- 172.21.15.16:0/29518337 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f60009e60 con 0x7f9f70101dc0 2024-03-28T22:02:34.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.598+0000 7f9f57fff700 1 -- 172.21.15.16:0/29518337 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f60003680 con 0x7f9f70101dc0 2024-03-28T22:02:34.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.598+0000 7f9f75023700 1 -- 172.21.15.16:0/29518337 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f701992d0 con 0x7f9f70101dc0 2024-03-28T22:02:34.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.598+0000 7f9f57fff700 1 -- 172.21.15.16:0/29518337 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f6000fd40 con 0x7f9f70101dc0 2024-03-28T22:02:34.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.598+0000 7f9f75023700 1 -- 172.21.15.16:0/29518337 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f701a55f0 con 0x7f9f70101dc0 2024-03-28T22:02:34.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.599+0000 7f9f57fff700 1 -- 172.21.15.16:0/29518337 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7f9f6002b440 con 0x7f9f70101dc0 2024-03-28T22:02:34.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.599+0000 7f9f75023700 1 -- 172.21.15.16:0/29518337 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f70068f00 con 0x7f9f70101dc0 2024-03-28T22:02:34.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.599+0000 7f9f57fff700 1 --2- 172.21.15.16:0/29518337 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f9f58038e80 0x7f9f5803b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:02:34.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.599+0000 7f9f57fff700 1 -- 172.21.15.16:0/29518337 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f9f6004ca80 con 0x7f9f70101dc0 2024-03-28T22:02:34.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.599+0000 7f9f6ed9d700 1 --2- 172.21.15.16:0/29518337 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f9f58038e80 0x7f9f5803b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:02:34.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.600+0000 7f9f6ed9d700 1 --2- 172.21.15.16:0/29518337 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f9f58038e80 0x7f9f5803b340 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9f70199a20 tx=0x7f9f64009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:02:34.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.604+0000 7f9f57fff700 1 -- 172.21.15.16:0/29518337 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9f60013070 con 0x7f9f70101dc0 2024-03-28T22:02:34.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.809+0000 7f9f75023700 1 -- 172.21.15.16:0/29518337 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f9f7002db30 con 0x7f9f70101dc0 2024-03-28T22:02:34.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.810+0000 7f9f57fff700 1 -- 172.21.15.16:0/29518337 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+271 (secure 0 0 0) 0x7f9f600173f0 con 0x7f9f70101dc0 2024-03-28T22:02:34.811 INFO:teuthology.orchestra.run.smithi016.stdout:# minimal ceph.conf for 798d9884-ed4e-11ee-95d0-87774f69a715 2024-03-28T22:02:34.811 INFO:teuthology.orchestra.run.smithi016.stdout:[global] 2024-03-28T22:02:34.812 INFO:teuthology.orchestra.run.smithi016.stdout: fsid = 798d9884-ed4e-11ee-95d0-87774f69a715 2024-03-28T22:02:34.812 INFO:teuthology.orchestra.run.smithi016.stdout: mon_host = [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-03-28T22:02:34.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.812+0000 7f9f75023700 1 -- 172.21.15.16:0/29518337 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f9f58038e80 msgr2=0x7f9f5803b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:02:34.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.812+0000 7f9f75023700 1 --2- 172.21.15.16:0/29518337 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f9f58038e80 0x7f9f5803b340 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9f70199a20 tx=0x7f9f64009260 comp rx=0 tx=0).stop 2024-03-28T22:02:34.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.812+0000 7f9f75023700 1 -- 172.21.15.16:0/29518337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f70101dc0 msgr2=0x7f9f7019ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:02:34.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.812+0000 7f9f75023700 1 --2- 172.21.15.16:0/29518337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f70101dc0 0x7f9f7019ec60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9f60003e00 tx=0x7f9f60003e30 comp rx=0 tx=0).stop 2024-03-28T22:02:34.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.812+0000 7f9f75023700 1 -- 172.21.15.16:0/29518337 shutdown_connections 2024-03-28T22:02:34.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.812+0000 7f9f75023700 1 --2- 172.21.15.16:0/29518337 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f9f58038e80 0x7f9f5803b340 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:34.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.812+0000 7f9f75023700 1 --2- 172.21.15.16:0/29518337 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f701064f0 0x7f9f701988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:34.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.812+0000 7f9f75023700 1 --2- 172.21.15.16:0/29518337 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f70101410 0x7f9f7019e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:34.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.812+0000 7f9f75023700 1 --2- 172.21.15.16:0/29518337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f70101dc0 0x7f9f7019ec60 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:34.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.812+0000 7f9f75023700 1 -- 172.21.15.16:0/29518337 >> 172.21.15.16:0/29518337 conn(0x7f9f70078270 msgr2=0x7f9f7006ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:02:34.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.812+0000 7f9f75023700 1 -- 172.21.15.16:0/29518337 shutdown_connections 2024-03-28T22:02:34.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:34.812+0000 7f9f75023700 1 -- 172.21.15.16:0/29518337 wait complete. 2024-03-28T22:02:35.138 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:35 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:35.138 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:35 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:35.138 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:35 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:35.138 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:35 smithi053 ceph-mon[35979]: Reconfiguring mon.b (monmap changed)... 2024-03-28T22:02:35.138 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:35 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T22:02:35.138 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:35 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T22:02:35.139 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:35 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:35.139 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:35 smithi053 ceph-mon[35979]: Reconfiguring daemon mon.b on smithi053 2024-03-28T22:02:35.139 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:35 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/29518337' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:35.139 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:35 smithi053 ceph-mon[35979]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:35.319 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:35 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:35.319 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:35 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:35.319 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:35 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:35.319 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:35 smithi016 ceph-mon[32374]: Reconfiguring mon.b (monmap changed)... 2024-03-28T22:02:35.319 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:35 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T22:02:35.319 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:35 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T22:02:35.319 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:35 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:35.319 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:35 smithi016 ceph-mon[32374]: Reconfiguring daemon mon.b on smithi053 2024-03-28T22:02:35.319 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:35 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/29518337' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:35.319 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:35 smithi016 ceph-mon[32374]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:35.347 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-28T22:02:35.347 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-28T22:02:35.347 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.conf 2024-03-28T22:02:35.376 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-28T22:02:35.376 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T22:02:35.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:35 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:35.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:35 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:35.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:35 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:35.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:35 smithi063 ceph-mon[36249]: Reconfiguring mon.b (monmap changed)... 2024-03-28T22:02:35.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:35 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T22:02:35.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:35 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T22:02:35.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:35 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:35.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:35 smithi063 ceph-mon[36249]: Reconfiguring daemon mon.b on smithi053 2024-03-28T22:02:35.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:35 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/29518337' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:35.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:35 smithi063 ceph-mon[36249]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:35.443 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-28T22:02:35.443 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/ceph/ceph.conf 2024-03-28T22:02:35.478 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-28T22:02:35.478 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T22:02:35.550 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-28T22:02:35.550 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/ceph/ceph.conf 2024-03-28T22:02:35.589 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-28T22:02:35.590 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T22:02:35.665 INFO:tasks.cephadm:Adding mgr.x on smithi016 2024-03-28T22:02:35.665 INFO:tasks.cephadm:Adding mgr.y on smithi053 2024-03-28T22:02:35.665 INFO:tasks.cephadm:Adding mgr.z on smithi063 2024-03-28T22:02:35.665 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph orch apply mgr '3;smithi016=x;smithi053=y;smithi063=z' 2024-03-28T22:02:35.955 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T22:02:36.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:36 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:36.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:36 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:36.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:36 smithi016 ceph-mon[32374]: Reconfiguring mon.c (monmap changed)... 2024-03-28T22:02:36.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:36 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T22:02:36.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:36 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T22:02:36.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:36 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:36.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:36 smithi016 ceph-mon[32374]: Reconfiguring daemon mon.c on smithi063 2024-03-28T22:02:37.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:36 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:37.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:36 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:37.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:36 smithi053 ceph-mon[35979]: Reconfiguring mon.c (monmap changed)... 2024-03-28T22:02:37.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:36 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T22:02:37.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:36 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T22:02:37.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:36 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:37.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:36 smithi053 ceph-mon[35979]: Reconfiguring daemon mon.c on smithi063 2024-03-28T22:02:37.081 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:36 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:37.081 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:36 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:37.081 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:36 smithi063 ceph-mon[36249]: Reconfiguring mon.c (monmap changed)... 2024-03-28T22:02:37.081 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:36 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T22:02:37.081 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:36 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T22:02:37.081 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:36 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:37.081 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:36 smithi063 ceph-mon[36249]: Reconfiguring daemon mon.c on smithi063 2024-03-28T22:02:38.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.020+0000 7f225fa23700 1 -- 172.21.15.63:0/2878447204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2258101d90 msgr2=0x7f22581021b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:02:38.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.020+0000 7f225fa23700 1 --2- 172.21.15.63:0/2878447204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2258101d90 0x7f22581021b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2254009910 tx=0x7f2254009c20 comp rx=0 tx=0).stop 2024-03-28T22:02:38.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.020+0000 7f225fa23700 1 -- 172.21.15.63:0/2878447204 shutdown_connections 2024-03-28T22:02:38.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.020+0000 7f225fa23700 1 --2- 172.21.15.63:0/2878447204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2258101d90 0x7f22581021b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:38.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.020+0000 7f225fa23700 1 -- 172.21.15.63:0/2878447204 >> 172.21.15.63:0/2878447204 conn(0x7f22580fd330 msgr2=0x7f22580ff770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:02:38.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.020+0000 7f225fa23700 1 -- 172.21.15.63:0/2878447204 shutdown_connections 2024-03-28T22:02:38.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.020+0000 7f225fa23700 1 -- 172.21.15.63:0/2878447204 wait complete. 2024-03-28T22:02:38.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.020+0000 7f225fa23700 1 Processor -- start 2024-03-28T22:02:38.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.020+0000 7f225fa23700 1 -- start start 2024-03-28T22:02:38.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.021+0000 7f225fa23700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2258101d90 0x7f22581975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:02:38.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.021+0000 7f225fa23700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2258197af0 0x7f225819cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:02:38.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.021+0000 7f225fa23700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2258197f70 0x7f225819f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:02:38.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.021+0000 7f225fa23700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2258107dd0 con 0x7f2258197af0 2024-03-28T22:02:38.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.021+0000 7f225fa23700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f2258107c50 con 0x7f2258101d90 2024-03-28T22:02:38.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.021+0000 7f225fa23700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2258107f50 con 0x7f2258197f70 2024-03-28T22:02:38.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.021+0000 7f225dfc0700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2258197f70 0x7f225819f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:02:38.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.021+0000 7f225cfbe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2258197af0 0x7f225819cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:02:38.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.021+0000 7f225dfc0700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2258197f70 0x7f225819f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:57208/0 (socket says 172.21.15.63:57208) 2024-03-28T22:02:38.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.021+0000 7f225dfc0700 1 -- 172.21.15.63:0/3625283102 learned_addr learned my addr 172.21.15.63:0/3625283102 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-28T22:02:38.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.021+0000 7f225d7bf700 1 --2- 172.21.15.63:0/3625283102 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2258101d90 0x7f22581975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:02:38.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.021+0000 7f225cfbe700 1 -- 172.21.15.63:0/3625283102 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2258197f70 msgr2=0x7f225819f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:02:38.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.021+0000 7f225cfbe700 1 --2- 172.21.15.63:0/3625283102 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2258197f70 0x7f225819f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:38.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.021+0000 7f225cfbe700 1 -- 172.21.15.63:0/3625283102 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2258101d90 msgr2=0x7f22581975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:02:38.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.021+0000 7f225cfbe700 1 --2- 172.21.15.63:0/3625283102 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2258101d90 0x7f22581975b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:38.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.021+0000 7f225cfbe700 1 -- 172.21.15.63:0/3625283102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f225819f9c0 con 0x7f2258197af0 2024-03-28T22:02:38.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.021+0000 7f225dfc0700 1 --2- 172.21.15.63:0/3625283102 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2258197f70 0x7f225819f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:02:38.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.021+0000 7f225cfbe700 1 --2- 172.21.15.63:0/3625283102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2258197af0 0x7f225819cb60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f224800b550 tx=0x7f224800b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:02:38.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.022+0000 7f224e7fc700 1 -- 172.21.15.63:0/3625283102 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2248004230 con 0x7f2258197af0 2024-03-28T22:02:38.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.022+0000 7f224e7fc700 1 -- 172.21.15.63:0/3625283102 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22480043d0 con 0x7f2258197af0 2024-03-28T22:02:38.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.022+0000 7f225fa23700 1 -- 172.21.15.63:0/3625283102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f225819fcb0 con 0x7f2258197af0 2024-03-28T22:02:38.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.022+0000 7f224e7fc700 1 -- 172.21.15.63:0/3625283102 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2248011b20 con 0x7f2258197af0 2024-03-28T22:02:38.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.022+0000 7f225fa23700 1 -- 172.21.15.63:0/3625283102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22581a0260 con 0x7f2258197af0 2024-03-28T22:02:38.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.023+0000 7f224e7fc700 1 -- 172.21.15.63:0/3625283102 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7f2248011d00 con 0x7f2258197af0 2024-03-28T22:02:38.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.023+0000 7f225fa23700 1 -- 172.21.15.63:0/3625283102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2258063870 con 0x7f2258197af0 2024-03-28T22:02:38.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.023+0000 7f224e7fc700 1 --2- 172.21.15.63:0/3625283102 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f22440390d0 0x7f224403b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:02:38.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.023+0000 7f224e7fc700 1 -- 172.21.15.63:0/3625283102 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f224804b600 con 0x7f2258197af0 2024-03-28T22:02:38.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.023+0000 7f225d7bf700 1 --2- 172.21.15.63:0/3625283102 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f22440390d0 0x7f224403b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:02:38.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.024+0000 7f225d7bf700 1 --2- 172.21.15.63:0/3625283102 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f22440390d0 0x7f224403b590 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2254005200 tx=0x7f225401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:02:38.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.028+0000 7f224e7fc700 1 -- 172.21.15.63:0/3625283102 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f22480202a0 con 0x7f2258197af0 2024-03-28T22:02:38.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:37 smithi053 ceph-mon[35979]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:38.089 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:37 smithi063 ceph-mon[36249]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:38.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:37 smithi016 ceph-mon[32374]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:38.248 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.246+0000 7f225fa23700 1 -- 172.21.15.63:0/3625283102 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi016=x;smithi053=y;smithi063=z", "target": ["mon-mgr", ""]}) -- 0x7f2258062710 con 0x7f22440390d0 2024-03-28T22:02:38.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.253+0000 7f224e7fc700 1 -- 172.21.15.63:0/3625283102 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f2258062710 con 0x7f22440390d0 2024-03-28T22:02:38.255 INFO:teuthology.orchestra.run.smithi063.stdout:Scheduled mgr update... 2024-03-28T22:02:38.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.255+0000 7f225fa23700 1 -- 172.21.15.63:0/3625283102 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f22440390d0 msgr2=0x7f224403b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:02:38.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.255+0000 7f225fa23700 1 --2- 172.21.15.63:0/3625283102 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f22440390d0 0x7f224403b590 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2254005200 tx=0x7f225401a040 comp rx=0 tx=0).stop 2024-03-28T22:02:38.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.255+0000 7f225fa23700 1 -- 172.21.15.63:0/3625283102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2258197af0 msgr2=0x7f225819cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:02:38.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.255+0000 7f225fa23700 1 --2- 172.21.15.63:0/3625283102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2258197af0 0x7f225819cb60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f224800b550 tx=0x7f224800b860 comp rx=0 tx=0).stop 2024-03-28T22:02:38.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.255+0000 7f225fa23700 1 -- 172.21.15.63:0/3625283102 shutdown_connections 2024-03-28T22:02:38.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.255+0000 7f225fa23700 1 --2- 172.21.15.63:0/3625283102 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f22440390d0 0x7f224403b590 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:38.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.255+0000 7f225fa23700 1 --2- 172.21.15.63:0/3625283102 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2258197f70 0x7f225819f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:38.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.255+0000 7f225fa23700 1 --2- 172.21.15.63:0/3625283102 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2258101d90 0x7f22581975b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:38.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.255+0000 7f225fa23700 1 --2- 172.21.15.63:0/3625283102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2258197af0 0x7f225819cb60 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:38.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.255+0000 7f225fa23700 1 -- 172.21.15.63:0/3625283102 >> 172.21.15.63:0/3625283102 conn(0x7f22580fd330 msgr2=0x7f22580fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:02:38.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.255+0000 7f225fa23700 1 -- 172.21.15.63:0/3625283102 shutdown_connections 2024-03-28T22:02:38.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:02:38.255+0000 7f225fa23700 1 -- 172.21.15.63:0/3625283102 wait complete. 2024-03-28T22:02:38.923 DEBUG:teuthology.orchestra.run.smithi053:mgr.y> sudo journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mgr.y.service 2024-03-28T22:02:38.925 DEBUG:teuthology.orchestra.run.smithi063:mgr.z> sudo journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mgr.z.service 2024-03-28T22:02:38.928 INFO:tasks.cephadm:Deploying OSDs... 2024-03-28T22:02:38.928 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-28T22:02:38.928 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T22:02:38.944 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T22:02:38.944 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_1 2024-03-28T22:02:38.999 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T22:02:38.999 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T22:02:38.999 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-28T22:02:38.999 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T22:02:38.999 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T22:02:38.999 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-03-28 22:01:39.725141537 +0000 2024-03-28T22:02:38.999 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-03-28 21:56:20.835738087 +0000 2024-03-28T22:02:38.999 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-03-28 21:56:20.835738087 +0000 2024-03-28T22:02:38.999 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-03-28 21:56:20.835738087 +0000 2024-03-28T22:02:38.999 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T22:02:39.065 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-03-28T22:02:39.066 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-03-28T22:02:39.066 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000104264 s, 4.9 MB/s 2024-03-28T22:02:39.067 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T22:02:39.122 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_2 2024-03-28T22:02:39.176 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T22:02:39.176 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T22:02:39.176 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-28T22:02:39.177 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T22:02:39.177 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T22:02:39.177 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-03-28 22:01:39.758141025 +0000 2024-03-28T22:02:39.177 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-03-28 21:56:21.186732508 +0000 2024-03-28T22:02:39.177 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-03-28 21:56:21.186732508 +0000 2024-03-28T22:02:39.177 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-03-28 21:56:21.186732508 +0000 2024-03-28T22:02:39.177 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T22:02:39.241 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-03-28T22:02:39.241 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-03-28T22:02:39.241 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000102166 s, 5.0 MB/s 2024-03-28T22:02:39.242 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T22:02:39.298 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_3 2024-03-28T22:02:39.352 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T22:02:39.352 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T22:02:39.353 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-28T22:02:39.353 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T22:02:39.353 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T22:02:39.353 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-03-28 22:01:39.797140420 +0000 2024-03-28T22:02:39.353 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-03-28 21:56:21.517727247 +0000 2024-03-28T22:02:39.353 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-03-28 21:56:21.517727247 +0000 2024-03-28T22:02:39.353 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-03-28 21:56:21.517727247 +0000 2024-03-28T22:02:39.353 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T22:02:39.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:39 smithi016 ceph-mon[32374]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi016=x;smithi053=y;smithi063=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:02:39.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:39 smithi016 ceph-mon[32374]: Saving service mgr spec with placement smithi016=x;smithi053=y;smithi063=z;count:3 2024-03-28T22:02:39.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:39 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:39.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:39 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:39.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:39 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:39.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:39 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:02:39.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:39 smithi016 ceph-mon[32374]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:39.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:39 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:39.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:39 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:02:39.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:39 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:39.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:39 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T22:02:39.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:39 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T22:02:39.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:39 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T22:02:39.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:39 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:39.417 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-03-28T22:02:39.417 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-03-28T22:02:39.417 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 8.3257e-05 s, 6.1 MB/s 2024-03-28T22:02:39.418 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T22:02:39.473 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_4 2024-03-28T22:02:39.528 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T22:02:39.528 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T22:02:39.528 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-28T22:02:39.528 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T22:02:39.528 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T22:02:39.528 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-03-28 22:01:39.829139923 +0000 2024-03-28T22:02:39.528 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-03-28 21:56:21.851721939 +0000 2024-03-28T22:02:39.528 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-03-28 21:56:21.851721939 +0000 2024-03-28T22:02:39.528 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-03-28 21:56:21.851721939 +0000 2024-03-28T22:02:39.529 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T22:02:39.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:39 smithi053 ceph-mon[35979]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi016=x;smithi053=y;smithi063=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:02:39.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:39 smithi053 ceph-mon[35979]: Saving service mgr spec with placement smithi016=x;smithi053=y;smithi063=z;count:3 2024-03-28T22:02:39.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:39 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:39.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:39 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:39.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:39 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:39.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:39 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:02:39.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:39 smithi053 ceph-mon[35979]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:39.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:39 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:39.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:39 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:02:39.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:39 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:39.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:39 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T22:02:39.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:39 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T22:02:39.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:39 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T22:02:39.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:39 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:39.593 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-03-28T22:02:39.593 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-03-28T22:02:39.593 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 9.2705e-05 s, 5.5 MB/s 2024-03-28T22:02:39.594 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T22:02:39.649 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-28T22:02:39.650 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T22:02:39.653 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:39 smithi063 ceph-mon[36249]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi016=x;smithi053=y;smithi063=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:02:39.653 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:39 smithi063 ceph-mon[36249]: Saving service mgr spec with placement smithi016=x;smithi053=y;smithi063=z;count:3 2024-03-28T22:02:39.653 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:39 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:39.653 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:39 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:39.653 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:39 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:39.653 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:39 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:02:39.654 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:39 smithi063 ceph-mon[36249]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:39.654 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:39 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:39.654 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:39 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:02:39.654 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:39 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:39.654 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:39 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T22:02:39.654 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:39 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T22:02:39.654 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:39 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T22:02:39.654 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:39 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:39.669 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T22:02:39.669 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_1 2024-03-28T22:02:39.727 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T22:02:39.727 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T22:02:39.727 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 661 Links: 1 2024-03-28T22:02:39.727 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T22:02:39.727 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T22:02:39.727 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-03-28 22:02:23.882064403 +0000 2024-03-28T22:02:39.727 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-03-28 21:57:00.162974768 +0000 2024-03-28T22:02:39.727 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-03-28 21:57:00.162974768 +0000 2024-03-28T22:02:39.727 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-03-28 21:57:00.162974768 +0000 2024-03-28T22:02:39.728 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T22:02:39.800 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-03-28T22:02:39.800 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-03-28T22:02:39.800 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000127905 s, 4.0 MB/s 2024-03-28T22:02:39.802 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T22:02:39.859 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_2 2024-03-28T22:02:39.916 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T22:02:39.916 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T22:02:39.916 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 681 Links: 1 2024-03-28T22:02:39.916 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T22:02:39.916 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T22:02:39.917 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-03-28 22:02:23.882064403 +0000 2024-03-28T22:02:39.917 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-03-28 21:57:00.527967163 +0000 2024-03-28T22:02:39.917 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-03-28 21:57:00.527967163 +0000 2024-03-28T22:02:39.917 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-03-28 21:57:00.527967163 +0000 2024-03-28T22:02:39.917 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T22:02:39.988 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-03-28T22:02:39.988 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-03-28T22:02:39.988 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000120255 s, 4.3 MB/s 2024-03-28T22:02:39.990 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T22:02:40.047 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_3 2024-03-28T22:02:40.102 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T22:02:40.103 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T22:02:40.103 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 699 Links: 1 2024-03-28T22:02:40.103 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T22:02:40.103 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T22:02:40.103 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-03-28 22:02:23.882064403 +0000 2024-03-28T22:02:40.103 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-03-28 21:57:00.867960078 +0000 2024-03-28T22:02:40.103 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-03-28 21:57:00.867960078 +0000 2024-03-28T22:02:40.103 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-03-28 21:57:00.867960078 +0000 2024-03-28T22:02:40.103 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T22:02:40.189 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-03-28T22:02:40.189 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-03-28T22:02:40.189 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 9.8188e-05 s, 5.2 MB/s 2024-03-28T22:02:40.191 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T22:02:40.208 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_4 2024-03-28T22:02:40.263 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T22:02:40.263 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T22:02:40.263 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 717 Links: 1 2024-03-28T22:02:40.263 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T22:02:40.263 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T22:02:40.263 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-03-28 22:02:23.883064383 +0000 2024-03-28T22:02:40.263 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-03-28 21:57:01.204953056 +0000 2024-03-28T22:02:40.264 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-03-28 21:57:01.204953056 +0000 2024-03-28T22:02:40.264 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-03-28 21:57:01.204953056 +0000 2024-03-28T22:02:40.264 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T22:02:40.333 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-03-28T22:02:40.333 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-03-28T22:02:40.334 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000110342 s, 4.6 MB/s 2024-03-28T22:02:40.335 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T22:02:40.391 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-28T22:02:40.391 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T22:02:40.409 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T22:02:40.409 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_1 2024-03-28T22:02:40.458 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:40 smithi053 ceph-mon[35979]: Deploying daemon mgr.z on smithi063 2024-03-28T22:02:40.466 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T22:02:40.466 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T22:02:40.466 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-03-28T22:02:40.467 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T22:02:40.467 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T22:02:40.467 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-03-28 22:02:14.204719650 +0000 2024-03-28T22:02:40.467 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-03-28 21:56:32.920170038 +0000 2024-03-28T22:02:40.467 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-03-28 21:56:32.920170038 +0000 2024-03-28T22:02:40.467 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-03-28 21:56:32.920170038 +0000 2024-03-28T22:02:40.467 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T22:02:40.538 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-03-28T22:02:40.538 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-03-28T22:02:40.538 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000254142 s, 2.0 MB/s 2024-03-28T22:02:40.539 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T22:02:40.595 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_2 2024-03-28T22:02:40.650 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T22:02:40.650 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T22:02:40.650 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-03-28T22:02:40.650 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T22:02:40.650 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T22:02:40.651 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-03-28 22:02:14.204719650 +0000 2024-03-28T22:02:40.651 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-03-28 21:56:33.294166530 +0000 2024-03-28T22:02:40.651 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-03-28 21:56:33.294166530 +0000 2024-03-28T22:02:40.651 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-03-28 21:56:33.294166530 +0000 2024-03-28T22:02:40.651 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T22:02:40.656 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:40 smithi063 ceph-mon[36249]: Deploying daemon mgr.z on smithi063 2024-03-28T22:02:40.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:40 smithi016 ceph-mon[32374]: Deploying daemon mgr.z on smithi063 2024-03-28T22:02:40.840 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-03-28T22:02:40.841 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-03-28T22:02:40.841 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000196008 s, 2.6 MB/s 2024-03-28T22:02:40.842 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T22:02:40.858 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_3 2024-03-28T22:02:40.915 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T22:02:40.915 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T22:02:40.915 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 699 Links: 1 2024-03-28T22:02:40.915 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T22:02:40.915 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T22:02:40.915 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-03-28 22:02:40.774480823 +0000 2024-03-28T22:02:40.915 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-03-28 21:56:33.640163284 +0000 2024-03-28T22:02:40.915 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-03-28 21:56:33.640163284 +0000 2024-03-28T22:02:40.915 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-03-28 21:56:33.640163284 +0000 2024-03-28T22:02:40.916 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T22:02:41.060 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-03-28T22:02:41.060 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-03-28T22:02:41.060 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000200391 s, 2.6 MB/s 2024-03-28T22:02:41.062 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T22:02:41.079 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_4 2024-03-28T22:02:41.136 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T22:02:41.136 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T22:02:41.136 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-03-28T22:02:41.137 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T22:02:41.137 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T22:02:41.137 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-03-28 22:02:40.997478818 +0000 2024-03-28T22:02:41.137 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-03-28 21:56:33.985160048 +0000 2024-03-28T22:02:41.137 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-03-28 21:56:33.985160048 +0000 2024-03-28T22:02:41.137 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-03-28 21:56:33.985160048 +0000 2024-03-28T22:02:41.137 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T22:02:41.210 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-03-28T22:02:41.210 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-03-28T22:02:41.210 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000204034 s, 2.5 MB/s 2024-03-28T22:02:41.211 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T22:02:41.267 INFO:tasks.cephadm:Deploying osd.0 on smithi016 with /dev/vg_nvme/lv_4... 2024-03-28T22:02:41.268 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-28T22:02:41.515 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:02:41.571 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:41 smithi063 ceph-mon[36249]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:41.631 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:41 smithi016 ceph-mon[32374]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:41.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:41 smithi053 ceph-mon[35979]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:42.432 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:02:42 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:02:42.259+0000 7f76188f4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T22:02:42.741 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:42 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:42.741 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:42 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:42.741 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:42 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:42.741 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:42 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T22:02:42.741 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:42 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T22:02:42.741 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:42 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T22:02:42.741 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:42 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:42.741 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:42 smithi016 ceph-mon[32374]: Deploying daemon mgr.y on smithi053 2024-03-28T22:02:43.027 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:02:42 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:02:42.929+0000 7f76188f4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T22:02:43.027 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:42 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:43.027 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:42 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:43.027 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:42 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:43.027 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:42 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T22:02:43.027 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:42 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T22:02:43.027 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:42 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T22:02:43.027 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:42 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:43.028 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:42 smithi063 ceph-mon[36249]: Deploying daemon mgr.y on smithi053 2024-03-28T22:02:43.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:42 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:43.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:42 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:43.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:42 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:43.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:42 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T22:02:43.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:42 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T22:02:43.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:42 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T22:02:43.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:42 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:43.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:42 smithi053 ceph-mon[35979]: Deploying daemon mgr.y on smithi053 2024-03-28T22:02:43.281 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-28T22:02:43.281 INFO:teuthology.orchestra.run.smithi016.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-28T22:02:43.281 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10+0 records in 2024-03-28T22:02:43.281 INFO:teuthology.orchestra.run.smithi016.stderr:10+0 records out 2024-03-28T22:02:43.281 INFO:teuthology.orchestra.run.smithi016.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00850813 s, 1.2 GB/s 2024-03-28T22:02:43.281 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping successful for: 2024-03-28T22:02:43.356 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:02:43 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:02:43.024+0000 7f76188f4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T22:02:43.356 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:02:43 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:02:43.170+0000 7f76188f4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T22:02:43.356 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:02:43 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:02:43.268+0000 7f76188f4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T22:02:43.356 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:02:43 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:02:43.354+0000 7f76188f4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T22:02:43.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:43 smithi053 ceph-mon[35979]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:43.885 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:43 smithi016 ceph-mon[32374]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:43.911 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi016:vg_nvme/lv_4 2024-03-28T22:02:43.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:43 smithi063 ceph-mon[36249]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:43.932 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:02:43 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:02:43.626+0000 7f76188f4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T22:02:43.932 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:02:43 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:02:43.749+0000 7f76188f4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T22:02:43.932 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:02:43 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:02:43.838+0000 7f76188f4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T22:02:44.146 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:02:44 smithi053 podman[37362]: 2024-03-28 22:02:44.120936934 +0000 UTC m=+0.352336639 container start 6153d3a6250b7e53fd847d31e0337869267c967dec5cd64a264228f7aa490f72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=squid-323d23d, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-28T22:02:44.149 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:02:44.432 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:02:43 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:02:43.935+0000 7f76188f4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T22:02:44.432 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:02:44 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:02:44.116+0000 7f76188f4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T22:02:44.547 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:02:44 smithi053 ceph-mgr[37381]: -- 172.21.15.53:0/2231941698 <== mon.2 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e7901225a0 con 0x55e7900f7000 2024-03-28T22:02:44.547 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:02:44 smithi053 bash[37362]: 6153d3a6250b7e53fd847d31e0337869267c967dec5cd64a264228f7aa490f72 2024-03-28T22:02:44.547 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:02:44 smithi053 systemd[1]: Started Ceph mgr.y for 798d9884-ed4e-11ee-95d0-87774f69a715. 2024-03-28T22:02:45.047 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:02:44 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:02:44.694+0000 7f68e9bfd200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T22:02:45.047 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:02:44 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:02:44.792+0000 7f68e9bfd200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T22:02:45.306 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:45 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:45.306 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:45 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:45.306 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:45 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:45.306 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:45 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:45.306 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:45 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:02:45.306 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:45 smithi016 ceph-mon[32374]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:45.432 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:02:45 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:02:45.175+0000 7f76188f4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T22:02:45.432 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:02:45 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:02:45.313+0000 7f76188f4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T22:02:45.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:45 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:45.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:45 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:45.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:45 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:45.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:45 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:45.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:45 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:02:45.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:45 smithi063 ceph-mon[36249]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:45.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:45 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:45.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:45 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:45.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:45 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:45.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:45 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:45.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:45 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:02:45.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:45 smithi053 ceph-mon[35979]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:45.549 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:02:45 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:02:45.489+0000 7f68e9bfd200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T22:02:45.768 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:02:45 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:02:45.664+0000 7f76188f4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T22:02:45.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.790+0000 7f016688d700 1 -- 172.21.15.16:0/2374140038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0160103e30 msgr2=0x7f016010a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:02:45.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.790+0000 7f016688d700 1 --2- 172.21.15.16:0/2374140038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0160103e30 0x7f016010a6e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f015800b0d0 tx=0x7f015800b3e0 comp rx=0 tx=0).stop 2024-03-28T22:02:45.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.790+0000 7f016688d700 1 -- 172.21.15.16:0/2374140038 shutdown_connections 2024-03-28T22:02:45.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.790+0000 7f016688d700 1 --2- 172.21.15.16:0/2374140038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0160103e30 0x7f016010a6e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:45.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.790+0000 7f016688d700 1 --2- 172.21.15.16:0/2374140038 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0160102290 0x7f01601026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:45.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.790+0000 7f016688d700 1 --2- 172.21.15.16:0/2374140038 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0160103490 0x7f01601038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:45.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.790+0000 7f016688d700 1 -- 172.21.15.16:0/2374140038 >> 172.21.15.16:0/2374140038 conn(0x7f01600fd850 msgr2=0x7f01600ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:02:45.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.790+0000 7f016688d700 1 -- 172.21.15.16:0/2374140038 shutdown_connections 2024-03-28T22:02:45.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.790+0000 7f016688d700 1 -- 172.21.15.16:0/2374140038 wait complete. 2024-03-28T22:02:45.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.791+0000 7f016688d700 1 Processor -- start 2024-03-28T22:02:45.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.791+0000 7f016688d700 1 -- start start 2024-03-28T22:02:45.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.791+0000 7f016688d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0160102290 0x7f0160075fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:02:45.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.791+0000 7f016688d700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0160103490 0x7f0160074630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:02:45.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.791+0000 7f016688d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0160103e30 0x7f0160074b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:02:45.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.791+0000 7f016688d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f016010de70 con 0x7f0160103e30 2024-03-28T22:02:45.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.791+0000 7f016688d700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f016010dcf0 con 0x7f0160102290 2024-03-28T22:02:45.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.791+0000 7f016688d700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f016010dff0 con 0x7f0160103490 2024-03-28T22:02:45.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.791+0000 7f0164e2a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0160103e30 0x7f0160074b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:02:45.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.791+0000 7f015f7fe700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0160103490 0x7f0160074630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:02:45.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.791+0000 7f0164e2a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0160103e30 0x7f0160074b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45282/0 (socket says 172.21.15.16:45282) 2024-03-28T22:02:45.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.791+0000 7f0164e2a700 1 -- 172.21.15.16:0/3898727573 learned_addr learned my addr 172.21.15.16:0/3898727573 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:02:45.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.791+0000 7f015ffff700 1 --2- 172.21.15.16:0/3898727573 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0160102290 0x7f0160075fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:02:45.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.792+0000 7f0164e2a700 1 -- 172.21.15.16:0/3898727573 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0160103490 msgr2=0x7f0160074630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:02:45.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.792+0000 7f0164e2a700 1 --2- 172.21.15.16:0/3898727573 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0160103490 0x7f0160074630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:45.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.792+0000 7f0164e2a700 1 -- 172.21.15.16:0/3898727573 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0160102290 msgr2=0x7f0160075fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:02:45.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.792+0000 7f0164e2a700 1 --2- 172.21.15.16:0/3898727573 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0160102290 0x7f0160075fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:02:45.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.792+0000 7f0164e2a700 1 -- 172.21.15.16:0/3898727573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0160075420 con 0x7f0160103e30 2024-03-28T22:02:45.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.792+0000 7f0164e2a700 1 --2- 172.21.15.16:0/3898727573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0160103e30 0x7f0160074b70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f015800d010 tx=0x7f0158007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:02:45.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.792+0000 7f015d7fa700 1 -- 172.21.15.16:0/3898727573 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0158003c00 con 0x7f0160103e30 2024-03-28T22:02:45.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.792+0000 7f016688d700 1 -- 172.21.15.16:0/3898727573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01601a2500 con 0x7f0160103e30 2024-03-28T22:02:45.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.792+0000 7f015d7fa700 1 -- 172.21.15.16:0/3898727573 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0158003da0 con 0x7f0160103e30 2024-03-28T22:02:45.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.792+0000 7f015d7fa700 1 -- 172.21.15.16:0/3898727573 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015801b9a0 con 0x7f0160103e30 2024-03-28T22:02:45.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.792+0000 7f016688d700 1 -- 172.21.15.16:0/3898727573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01601a2900 con 0x7f0160103e30 2024-03-28T22:02:45.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.793+0000 7f015d7fa700 1 -- 172.21.15.16:0/3898727573 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7f0158026450 con 0x7f0160103e30 2024-03-28T22:02:45.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.793+0000 7f016688d700 1 -- 172.21.15.16:0/3898727573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0160068f00 con 0x7f0160103e30 2024-03-28T22:02:45.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.793+0000 7f015d7fa700 1 --2- 172.21.15.16:0/3898727573 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0148038e80 0x7f014803b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:02:45.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.793+0000 7f015d7fa700 1 -- 172.21.15.16:0/3898727573 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f015800f020 con 0x7f0160103e30 2024-03-28T22:02:45.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.793+0000 7f015ffff700 1 --2- 172.21.15.16:0/3898727573 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0148038e80 0x7f014803b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:02:45.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.794+0000 7f015ffff700 1 --2- 172.21.15.16:0/3898727573 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0148038e80 0x7f014803b340 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f01601032f0 tx=0x7f0150009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:02:45.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:45.798+0000 7f015d7fa700 1 -- 172.21.15.16:0/3898727573 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f015801a580 con 0x7f0160103e30 2024-03-28T22:02:45.840 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:02:45 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:02:45.584+0000 7f68e9bfd200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T22:02:45.840 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:02:45 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:02:45.737+0000 7f68e9bfd200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T22:02:45.841 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:02:45 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:02:45.838+0000 7f68e9bfd200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T22:02:46.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:46.015+0000 7f016688d700 1 -- 172.21.15.16:0/3898727573 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f0160062680 con 0x7f0148038e80 2024-03-28T22:02:46.163 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:02:45 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:02:45.766+0000 7f76188f4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T22:02:46.163 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:02:45 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:02:45.970+0000 7f76188f4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T22:02:46.163 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:02:46 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:02:46.160+0000 7f76188f4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T22:02:46.197 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:02:45 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:02:45.925+0000 7f68e9bfd200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T22:02:46.197 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:02:46 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:02:46.195+0000 7f68e9bfd200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T22:02:46.495 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:02:46 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:02:46.307+0000 7f68e9bfd200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T22:02:46.495 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:02:46 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:02:46.396+0000 7f68e9bfd200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T22:02:46.496 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:02:46 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:02:46.493+0000 7f68e9bfd200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T22:02:46.723 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:02:46 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:02:46.453+0000 7f76188f4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T22:02:46.723 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:02:46 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:02:46.721+0000 7f76188f4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T22:02:46.776 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:02:46 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:02:46.674+0000 7f68e9bfd200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T22:02:46.845 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:46 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:46.845 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:46 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:46.846 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:46 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:46.846 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:46 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:46.846 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:46 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:46.846 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:46 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:02:46.846 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:46 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:46.846 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:46 smithi016 ceph-mon[32374]: Reconfiguring mgr.x (unknown last config time)... 2024-03-28T22:02:46.847 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:46 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T22:02:46.847 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:46 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T22:02:46.847 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:46 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:46.847 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:46 smithi016 ceph-mon[32374]: Reconfiguring daemon mgr.x on smithi016 2024-03-28T22:02:46.847 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:46 smithi016 ceph-mon[32374]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:02:46.847 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:46 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:02:46.847 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:46 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:02:46.848 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:46 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:47.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:46 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:47.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:46 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:47.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:46 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:47.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:46 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:47.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:46 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:47.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:46 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:02:47.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:46 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:47.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:46 smithi053 ceph-mon[35979]: Reconfiguring mgr.x (unknown last config time)... 2024-03-28T22:02:47.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:46 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T22:02:47.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:46 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T22:02:47.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:46 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:47.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:46 smithi053 ceph-mon[35979]: Reconfiguring daemon mgr.x on smithi016 2024-03-28T22:02:47.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:46 smithi053 ceph-mon[35979]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:02:47.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:46 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:02:47.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:46 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:02:47.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:46 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:47.182 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:02:46 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:02:46.965+0000 7f76188f4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T22:02:47.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:46 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:47.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:46 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:47.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:46 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:47.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:46 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:47.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:46 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:47.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:46 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:02:47.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:46 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:47.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:46 smithi063 ceph-mon[36249]: Reconfiguring mgr.x (unknown last config time)... 2024-03-28T22:02:47.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:46 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T22:02:47.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:46 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T22:02:47.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:46 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:47.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:46 smithi063 ceph-mon[36249]: Reconfiguring daemon mgr.x on smithi016 2024-03-28T22:02:47.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:46 smithi063 ceph-mon[36249]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:02:47.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:46 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:02:47.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:46 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:02:47.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:46 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:47.923 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:47 smithi016 ceph-mon[32374]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:47.923 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:47 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:47.923 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:47 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:47.923 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:47 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:02:47.923 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:47 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:47.923 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:47 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:02:47.923 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:47 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:48.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:47 smithi053 ceph-mon[35979]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:48.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:47 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:48.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:47 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:48.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:47 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:02:48.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:47 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:48.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:47 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:02:48.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:47 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:48.048 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:02:47 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:02:47.692+0000 7f68e9bfd200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T22:02:48.048 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:02:47 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:02:47.827+0000 7f68e9bfd200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T22:02:48.182 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:02:47 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:02:47.804+0000 7f76188f4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T22:02:48.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:47 smithi063 ceph-mon[36249]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:48.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:47 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:48.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:47 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:48.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:47 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:02:48.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:47 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:48.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:47 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:02:48.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:47 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:48.463 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:02:48 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:02:48.170+0000 7f68e9bfd200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T22:02:48.463 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:02:48 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:02:48.262+0000 7f68e9bfd200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T22:02:48.463 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:02:48 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:02:48.461+0000 7f68e9bfd200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T22:02:48.681 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:02:48 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:02:48.395+0000 7f76188f4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T22:02:48.682 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:02:48 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:02:48.495+0000 7f76188f4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T22:02:48.780 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:02:48 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:02:48.650+0000 7f68e9bfd200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T22:02:48.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:48.786+0000 7f015d7fa700 1 -- 172.21.15.16:0/3898727573 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 14) ==== 91342+0+0 (secure 0 0 0) 0x7f015802ccf0 con 0x7f0160103e30 2024-03-28T22:02:49.005 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:48 smithi016 ceph-mon[32374]: Standby manager daemon z started 2024-03-28T22:02:49.005 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:48 smithi016 ceph-mon[32374]: from='mgr.? 172.21.15.63:0/2147989336' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T22:02:49.005 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:48 smithi016 ceph-mon[32374]: from='mgr.? 172.21.15.63:0/2147989336' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T22:02:49.005 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:48 smithi016 ceph-mon[32374]: from='mgr.? 172.21.15.63:0/2147989336' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T22:02:49.005 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:48 smithi016 ceph-mon[32374]: from='mgr.? 172.21.15.63:0/2147989336' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T22:02:49.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:48 smithi053 ceph-mon[35979]: Standby manager daemon z started 2024-03-28T22:02:49.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:48 smithi053 ceph-mon[35979]: from='mgr.? 172.21.15.63:0/2147989336' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T22:02:49.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:48 smithi053 ceph-mon[35979]: from='mgr.? 172.21.15.63:0/2147989336' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T22:02:49.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:48 smithi053 ceph-mon[35979]: from='mgr.? 172.21.15.63:0/2147989336' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T22:02:49.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:48 smithi053 ceph-mon[35979]: from='mgr.? 172.21.15.63:0/2147989336' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T22:02:49.048 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:02:48 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:02:48.939+0000 7f68e9bfd200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T22:02:49.182 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:02:48 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:02:48.718+0000 7f76188f4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T22:02:49.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:48 smithi063 ceph-mon[36249]: Standby manager daemon z started 2024-03-28T22:02:49.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:48 smithi063 ceph-mon[36249]: from='mgr.? 172.21.15.63:0/2147989336' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T22:02:49.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:48 smithi063 ceph-mon[36249]: from='mgr.? 172.21.15.63:0/2147989336' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T22:02:49.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:48 smithi063 ceph-mon[36249]: from='mgr.? 172.21.15.63:0/2147989336' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T22:02:49.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:48 smithi063 ceph-mon[36249]: from='mgr.? 172.21.15.63:0/2147989336' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T22:02:49.547 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:02:49 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:02:49.205+0000 7f68e9bfd200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T22:02:49.547 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:02:49 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:02:49.446+0000 7f68e9bfd200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T22:02:50.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:49 smithi053 ceph-mon[35979]: mgrmap e14: x(active, since 79s), standbys: z 2024-03-28T22:02:50.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:49 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T22:02:50.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:49 smithi053 ceph-mon[35979]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:50.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:49 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:50.181 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:49 smithi063 ceph-mon[36249]: mgrmap e14: x(active, since 79s), standbys: z 2024-03-28T22:02:50.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:49 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T22:02:50.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:49 smithi063 ceph-mon[36249]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:50.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:49 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:50.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:49 smithi016 ceph-mon[32374]: mgrmap e14: x(active, since 79s), standbys: z 2024-03-28T22:02:50.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:49 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T22:02:50.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:49 smithi016 ceph-mon[32374]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:50.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:49 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:02:50.547 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:02:50 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:02:50.273+0000 7f68e9bfd200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T22:02:51.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:50 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3553485938' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46e7c8f8-ecc8-43d6-ae53-0655728f3240"}]: dispatch 2024-03-28T22:02:51.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:50 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3553485938' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "46e7c8f8-ecc8-43d6-ae53-0655728f3240"}]': finished 2024-03-28T22:02:51.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:50 smithi053 ceph-mon[35979]: osdmap e5: 1 total, 0 up, 1 in 2024-03-28T22:02:51.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:50 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T22:02:51.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:50 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2609192644' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:02:51.048 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:02:50 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:02:50.861+0000 7f68e9bfd200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T22:02:51.048 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:02:50 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:02:50.963+0000 7f68e9bfd200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T22:02:51.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:50 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/3553485938' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46e7c8f8-ecc8-43d6-ae53-0655728f3240"}]: dispatch 2024-03-28T22:02:51.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:50 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/3553485938' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "46e7c8f8-ecc8-43d6-ae53-0655728f3240"}]': finished 2024-03-28T22:02:51.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:50 smithi063 ceph-mon[36249]: osdmap e5: 1 total, 0 up, 1 in 2024-03-28T22:02:51.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:50 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T22:02:51.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:50 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2609192644' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:02:51.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:50 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3553485938' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46e7c8f8-ecc8-43d6-ae53-0655728f3240"}]: dispatch 2024-03-28T22:02:51.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:50 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3553485938' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "46e7c8f8-ecc8-43d6-ae53-0655728f3240"}]': finished 2024-03-28T22:02:51.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:50 smithi016 ceph-mon[32374]: osdmap e5: 1 total, 0 up, 1 in 2024-03-28T22:02:51.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:50 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T22:02:51.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:50 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2609192644' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:02:51.490 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:02:51 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:02:51.183+0000 7f68e9bfd200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T22:02:51.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:02:51.800+0000 7f015d7fa700 1 -- 172.21.15.16:0/3898727573 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 15) ==== 136700+0+0 (secure 0 0 0) 0x7f0158022d00 con 0x7f0160103e30 2024-03-28T22:02:52.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:51 smithi063 ceph-mon[36249]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:52.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:51 smithi063 ceph-mon[36249]: Standby manager daemon y started 2024-03-28T22:02:52.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:51 smithi063 ceph-mon[36249]: from='mgr.? 172.21.15.53:0/1081772811' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T22:02:52.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:51 smithi063 ceph-mon[36249]: from='mgr.? 172.21.15.53:0/1081772811' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T22:02:52.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:51 smithi063 ceph-mon[36249]: from='mgr.? 172.21.15.53:0/1081772811' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T22:02:52.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:51 smithi063 ceph-mon[36249]: from='mgr.? 172.21.15.53:0/1081772811' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T22:02:52.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:51 smithi016 ceph-mon[32374]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:52.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:51 smithi016 ceph-mon[32374]: Standby manager daemon y started 2024-03-28T22:02:52.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:51 smithi016 ceph-mon[32374]: from='mgr.? 172.21.15.53:0/1081772811' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T22:02:52.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:51 smithi016 ceph-mon[32374]: from='mgr.? 172.21.15.53:0/1081772811' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T22:02:52.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:51 smithi016 ceph-mon[32374]: from='mgr.? 172.21.15.53:0/1081772811' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T22:02:52.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:51 smithi016 ceph-mon[32374]: from='mgr.? 172.21.15.53:0/1081772811' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T22:02:52.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:51 smithi053 ceph-mon[35979]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:52.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:51 smithi053 ceph-mon[35979]: Standby manager daemon y started 2024-03-28T22:02:52.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:51 smithi053 ceph-mon[35979]: from='mgr.? 172.21.15.53:0/1081772811' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T22:02:52.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:51 smithi053 ceph-mon[35979]: from='mgr.? 172.21.15.53:0/1081772811' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T22:02:52.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:51 smithi053 ceph-mon[35979]: from='mgr.? 172.21.15.53:0/1081772811' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T22:02:52.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:51 smithi053 ceph-mon[35979]: from='mgr.? 172.21.15.53:0/1081772811' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T22:02:53.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:52 smithi063 ceph-mon[36249]: mgrmap e15: x(active, since 82s), standbys: z, y 2024-03-28T22:02:53.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:52 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T22:02:53.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:52 smithi016 ceph-mon[32374]: mgrmap e15: x(active, since 82s), standbys: z, y 2024-03-28T22:02:53.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:52 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T22:02:53.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:52 smithi053 ceph-mon[35979]: mgrmap e15: x(active, since 82s), standbys: z, y 2024-03-28T22:02:53.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:52 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T22:02:53.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:53 smithi016 ceph-mon[32374]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:54.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:53 smithi063 ceph-mon[36249]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:54.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:53 smithi053 ceph-mon[35979]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:55.176 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:55 smithi016 ceph-mon[32374]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:55.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:55 smithi063 ceph-mon[36249]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:55.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:55 smithi053 ceph-mon[35979]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:57.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:57 smithi016 ceph-mon[32374]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:57.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:57 smithi063 ceph-mon[36249]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:57.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:57 smithi053 ceph-mon[35979]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:58.404 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:58 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-28T22:02:58.405 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:58 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:58.405 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:58 smithi016 ceph-mon[32374]: Deploying daemon osd.0 on smithi016 2024-03-28T22:02:58.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:58 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-28T22:02:58.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:58 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:58.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:58 smithi063 ceph-mon[36249]: Deploying daemon osd.0 on smithi016 2024-03-28T22:02:58.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:58 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-28T22:02:58.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:58 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:02:58.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:58 smithi053 ceph-mon[35979]: Deploying daemon osd.0 on smithi016 2024-03-28T22:02:59.388 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:02:59 smithi016 ceph-mon[32374]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:59.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:02:59 smithi063 ceph-mon[36249]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:02:59.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:02:59 smithi053 ceph-mon[35979]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:03:01.352 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:01 smithi016 ceph-mon[32374]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:03:01.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:01 smithi063 ceph-mon[36249]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:03:01.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:01 smithi053 ceph-mon[35979]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:03:03.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:03 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:03.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:03 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:03.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:03 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:03:03.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:03 smithi053 ceph-mon[35979]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:03:03.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:03 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:03.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:03 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:03.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:03 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:03:03.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:03 smithi063 ceph-mon[36249]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:03:03.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:03 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:03.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:03 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:03.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:03 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:03:03.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:03 smithi016 ceph-mon[32374]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:03:06.122 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:05 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:06.122 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:05 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:06.122 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:05 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:03:06.122 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:05 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:03:06.122 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:05 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:06.122 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:05 smithi016 ceph-mon[32374]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:03:06.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:05 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:06.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:05 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:06.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:05 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:03:06.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:05 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:03:06.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:05 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:06.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:05 smithi063 ceph-mon[36249]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:03:06.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:06.180+0000 7f015d7fa700 1 -- 172.21.15.16:0/3898727573 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0160062680 con 0x7f0148038e80 2024-03-28T22:03:06.183 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 0 on host 'smithi016' 2024-03-28T22:03:06.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:06.182+0000 7f016688d700 1 -- 172.21.15.16:0/3898727573 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0148038e80 msgr2=0x7f014803b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:03:06.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:06.182+0000 7f016688d700 1 --2- 172.21.15.16:0/3898727573 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0148038e80 0x7f014803b340 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f01601032f0 tx=0x7f0150009260 comp rx=0 tx=0).stop 2024-03-28T22:03:06.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:06.182+0000 7f016688d700 1 -- 172.21.15.16:0/3898727573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0160103e30 msgr2=0x7f0160074b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:03:06.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:06.182+0000 7f016688d700 1 --2- 172.21.15.16:0/3898727573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0160103e30 0x7f0160074b70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f015800d010 tx=0x7f0158007a20 comp rx=0 tx=0).stop 2024-03-28T22:03:06.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:06.182+0000 7f016688d700 1 -- 172.21.15.16:0/3898727573 shutdown_connections 2024-03-28T22:03:06.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:06.182+0000 7f016688d700 1 --2- 172.21.15.16:0/3898727573 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0148038e80 0x7f014803b340 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:03:06.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:06.182+0000 7f016688d700 1 --2- 172.21.15.16:0/3898727573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0160103e30 0x7f0160074b70 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:03:06.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:06.182+0000 7f016688d700 1 --2- 172.21.15.16:0/3898727573 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0160102290 0x7f0160075fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:03:06.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:06.182+0000 7f016688d700 1 --2- 172.21.15.16:0/3898727573 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0160103490 0x7f0160074630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:03:06.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:06.182+0000 7f016688d700 1 -- 172.21.15.16:0/3898727573 >> 172.21.15.16:0/3898727573 conn(0x7f01600fd850 msgr2=0x7f01601066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:03:06.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:06.182+0000 7f016688d700 1 -- 172.21.15.16:0/3898727573 shutdown_connections 2024-03-28T22:03:06.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:06.182+0000 7f016688d700 1 -- 172.21.15.16:0/3898727573 wait complete. 2024-03-28T22:03:06.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:05 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:06.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:05 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:06.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:05 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:03:06.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:05 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:03:06.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:05 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:06.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:05 smithi053 ceph-mon[35979]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:03:06.840 DEBUG:teuthology.orchestra.run.smithi016:osd.0> sudo journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.0.service 2024-03-28T22:03:06.842 INFO:tasks.cephadm:Deploying osd.1 on smithi053 with /dev/vg_nvme/lv_4... 2024-03-28T22:03:06.842 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-28T22:03:07.097 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T22:03:07.378 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:07 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:07.378 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:07 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:07.379 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:07 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:03:07.379 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:07 smithi016 ceph-mon[32374]: from='osd.0 [v2:172.21.15.16:6802/1589692483,v1:172.21.15.16:6803/1589692483]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-28T22:03:07.379 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:07 smithi016 ceph-mon[32374]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:03:07.408 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:07 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:07.408 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:07 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:07.408 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:07 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:03:07.408 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:07 smithi053 ceph-mon[35979]: from='osd.0 [v2:172.21.15.16:6802/1589692483,v1:172.21.15.16:6803/1589692483]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-28T22:03:07.409 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:07 smithi053 ceph-mon[35979]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:03:07.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:07 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:07.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:07 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:07.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:07 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:03:07.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:07 smithi063 ceph-mon[36249]: from='osd.0 [v2:172.21.15.16:6802/1589692483,v1:172.21.15.16:6803/1589692483]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-28T22:03:07.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:07 smithi063 ceph-mon[36249]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:03:08.399 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:08 smithi053 ceph-mon[35979]: from='osd.0 [v2:172.21.15.16:6802/1589692483,v1:172.21.15.16:6803/1589692483]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-28T22:03:08.399 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:08 smithi053 ceph-mon[35979]: osdmap e6: 1 total, 0 up, 1 in 2024-03-28T22:03:08.399 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:08 smithi053 ceph-mon[35979]: from='osd.0 [v2:172.21.15.16:6802/1589692483,v1:172.21.15.16:6803/1589692483]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-03-28T22:03:08.399 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:08 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T22:03:08.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:08 smithi063 ceph-mon[36249]: from='osd.0 [v2:172.21.15.16:6802/1589692483,v1:172.21.15.16:6803/1589692483]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-28T22:03:08.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:08 smithi063 ceph-mon[36249]: osdmap e6: 1 total, 0 up, 1 in 2024-03-28T22:03:08.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:08 smithi063 ceph-mon[36249]: from='osd.0 [v2:172.21.15.16:6802/1589692483,v1:172.21.15.16:6803/1589692483]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-03-28T22:03:08.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:08 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T22:03:08.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:08 smithi016 ceph-mon[32374]: from='osd.0 [v2:172.21.15.16:6802/1589692483,v1:172.21.15.16:6803/1589692483]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-28T22:03:08.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:08 smithi016 ceph-mon[32374]: osdmap e6: 1 total, 0 up, 1 in 2024-03-28T22:03:08.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:08 smithi016 ceph-mon[32374]: from='osd.0 [v2:172.21.15.16:6802/1589692483,v1:172.21.15.16:6803/1589692483]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-03-28T22:03:08.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:08 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T22:03:08.437 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 28 22:03:08 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-0[44837]: 2024-03-28T22:03:08.168+0000 7fb9d18e9700 -1 osd.0 0 waiting for initial osdmap 2024-03-28T22:03:08.437 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 28 22:03:08 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-0[44837]: 2024-03-28T22:03:08.173+0000 7fb9cdedd700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T22:03:08.962 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-28T22:03:08.962 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-28T22:03:08.962 INFO:teuthology.orchestra.run.smithi053.stderr: stderr: 10+0 records in 2024-03-28T22:03:08.962 INFO:teuthology.orchestra.run.smithi053.stderr:10+0 records out 2024-03-28T22:03:08.962 INFO:teuthology.orchestra.run.smithi053.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00825428 s, 1.3 GB/s 2024-03-28T22:03:08.962 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping successful for: 2024-03-28T22:03:09.263 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:09 smithi016 ceph-mon[32374]: from='osd.0 [v2:172.21.15.16:6802/1589692483,v1:172.21.15.16:6803/1589692483]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-03-28T22:03:09.263 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:09 smithi016 ceph-mon[32374]: osdmap e7: 1 total, 0 up, 1 in 2024-03-28T22:03:09.263 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:09 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T22:03:09.263 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:09 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T22:03:09.263 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:09 smithi016 ceph-mon[32374]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:03:09.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:09 smithi063 ceph-mon[36249]: from='osd.0 [v2:172.21.15.16:6802/1589692483,v1:172.21.15.16:6803/1589692483]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-03-28T22:03:09.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:09 smithi063 ceph-mon[36249]: osdmap e7: 1 total, 0 up, 1 in 2024-03-28T22:03:09.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:09 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T22:03:09.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:09 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T22:03:09.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:09 smithi063 ceph-mon[36249]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:03:09.455 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:09 smithi053 ceph-mon[35979]: from='osd.0 [v2:172.21.15.16:6802/1589692483,v1:172.21.15.16:6803/1589692483]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-03-28T22:03:09.455 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:09 smithi053 ceph-mon[35979]: osdmap e7: 1 total, 0 up, 1 in 2024-03-28T22:03:09.455 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:09 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T22:03:09.455 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:09 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T22:03:09.455 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:09 smithi053 ceph-mon[35979]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:03:09.572 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi053:vg_nvme/lv_4 2024-03-28T22:03:09.835 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T22:03:10.246 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:10 smithi053 ceph-mon[35979]: purged_snaps scrub starts 2024-03-28T22:03:10.246 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:10 smithi053 ceph-mon[35979]: purged_snaps scrub ok 2024-03-28T22:03:10.246 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:10 smithi053 ceph-mon[35979]: osd.0 [v2:172.21.15.16:6802/1589692483,v1:172.21.15.16:6803/1589692483] boot 2024-03-28T22:03:10.247 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:10 smithi053 ceph-mon[35979]: osdmap e8: 1 total, 1 up, 1 in 2024-03-28T22:03:10.247 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:10 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T22:03:10.247 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:10 smithi053 ceph-mon[35979]: Detected new or changed devices on smithi016 2024-03-28T22:03:10.247 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:10 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:10.247 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:10 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:10.247 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:10 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:03:10.247 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:10 smithi053 ceph-mon[35979]: Adjusting osd_memory_target on smithi016 to 17093M 2024-03-28T22:03:10.247 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:10 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:10.247 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:10 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:03:10.247 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:10 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:03:10.247 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:10 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:10.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:10 smithi063 ceph-mon[36249]: purged_snaps scrub starts 2024-03-28T22:03:10.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:10 smithi063 ceph-mon[36249]: purged_snaps scrub ok 2024-03-28T22:03:10.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:10 smithi063 ceph-mon[36249]: osd.0 [v2:172.21.15.16:6802/1589692483,v1:172.21.15.16:6803/1589692483] boot 2024-03-28T22:03:10.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:10 smithi063 ceph-mon[36249]: osdmap e8: 1 total, 1 up, 1 in 2024-03-28T22:03:10.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:10 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T22:03:10.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:10 smithi063 ceph-mon[36249]: Detected new or changed devices on smithi016 2024-03-28T22:03:10.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:10 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:10.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:10 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:10.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:10 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:03:10.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:10 smithi063 ceph-mon[36249]: Adjusting osd_memory_target on smithi016 to 17093M 2024-03-28T22:03:10.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:10 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:10.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:10 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:03:10.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:10 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:03:10.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:10 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:10.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:10 smithi016 ceph-mon[32374]: purged_snaps scrub starts 2024-03-28T22:03:10.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:10 smithi016 ceph-mon[32374]: purged_snaps scrub ok 2024-03-28T22:03:10.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:10 smithi016 ceph-mon[32374]: osd.0 [v2:172.21.15.16:6802/1589692483,v1:172.21.15.16:6803/1589692483] boot 2024-03-28T22:03:10.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:10 smithi016 ceph-mon[32374]: osdmap e8: 1 total, 1 up, 1 in 2024-03-28T22:03:10.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:10 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T22:03:10.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:10 smithi016 ceph-mon[32374]: Detected new or changed devices on smithi016 2024-03-28T22:03:10.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:10 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:10.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:10 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:10.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:10 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:03:10.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:10 smithi016 ceph-mon[32374]: Adjusting osd_memory_target on smithi016 to 17093M 2024-03-28T22:03:10.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:10 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:10.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:10 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:03:10.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:10 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:03:10.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:10 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:11.402 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:11 smithi053 ceph-mon[35979]: osdmap e9: 1 total, 1 up, 1 in 2024-03-28T22:03:11.402 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:11 smithi053 ceph-mon[35979]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:03:11.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:11 smithi016 ceph-mon[32374]: osdmap e9: 1 total, 1 up, 1 in 2024-03-28T22:03:11.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:11 smithi016 ceph-mon[32374]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:03:11.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.445+0000 7f8330d67700 1 -- 172.21.15.53:0/3744618486 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f832c103e40 msgr2=0x7f832c10a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:03:11.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.445+0000 7f8330d67700 1 --2- 172.21.15.53:0/3744618486 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f832c103e40 0x7f832c10a6f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f832400b0d0 tx=0x7f832400b3e0 comp rx=0 tx=0).stop 2024-03-28T22:03:11.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.445+0000 7f8330d67700 1 -- 172.21.15.53:0/3744618486 shutdown_connections 2024-03-28T22:03:11.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.445+0000 7f8330d67700 1 --2- 172.21.15.53:0/3744618486 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f832c103e40 0x7f832c10a6f0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:03:11.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.445+0000 7f8330d67700 1 --2- 172.21.15.53:0/3744618486 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f832c1022a0 0x7f832c1026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:03:11.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.445+0000 7f8330d67700 1 --2- 172.21.15.53:0/3744618486 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f832c1034a0 0x7f832c103900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:03:11.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.445+0000 7f8330d67700 1 -- 172.21.15.53:0/3744618486 >> 172.21.15.53:0/3744618486 conn(0x7f832c0fd840 msgr2=0x7f832c0ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:03:11.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.445+0000 7f8330d67700 1 -- 172.21.15.53:0/3744618486 shutdown_connections 2024-03-28T22:03:11.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.445+0000 7f8330d67700 1 -- 172.21.15.53:0/3744618486 wait complete. 2024-03-28T22:03:11.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.445+0000 7f8330d67700 1 Processor -- start 2024-03-28T22:03:11.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.446+0000 7f8330d67700 1 -- start start 2024-03-28T22:03:11.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.446+0000 7f8330d67700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f832c1022a0 0x7f832c198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:03:11.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.446+0000 7f8330d67700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f832c1034a0 0x7f832c198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:03:11.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.446+0000 7f8330d67700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f832c103e40 0x7f832c19f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:03:11.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.446+0000 7f8330d67700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f832c10de80 con 0x7f832c1034a0 2024-03-28T22:03:11.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.446+0000 7f8330d67700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f832c10dd00 con 0x7f832c1022a0 2024-03-28T22:03:11.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.446+0000 7f8330d67700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f832c10e000 con 0x7f832c103e40 2024-03-28T22:03:11.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.446+0000 7f832a59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f832c1022a0 0x7f832c198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:03:11.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.446+0000 7f832a59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f832c1022a0 0x7f832c198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:56464/0 (socket says 172.21.15.53:56464) 2024-03-28T22:03:11.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.446+0000 7f832a59c700 1 -- 172.21.15.53:0/2218421504 learned_addr learned my addr 172.21.15.53:0/2218421504 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-03-28T22:03:11.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.446+0000 7f8323fff700 1 --2- 172.21.15.53:0/2218421504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f832c1034a0 0x7f832c198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:03:11.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.446+0000 7f832ad9d700 1 --2- 172.21.15.53:0/2218421504 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f832c103e40 0x7f832c19f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:03:11.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.446+0000 7f832a59c700 1 -- 172.21.15.53:0/2218421504 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f832c103e40 msgr2=0x7f832c19f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:03:11.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.446+0000 7f832a59c700 1 --2- 172.21.15.53:0/2218421504 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f832c103e40 0x7f832c19f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:03:11.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.446+0000 7f832a59c700 1 -- 172.21.15.53:0/2218421504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f832c1034a0 msgr2=0x7f832c198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:03:11.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.446+0000 7f832a59c700 1 --2- 172.21.15.53:0/2218421504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f832c1034a0 0x7f832c198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:03:11.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.446+0000 7f832a59c700 1 -- 172.21.15.53:0/2218421504 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f832c1a0070 con 0x7f832c1022a0 2024-03-28T22:03:11.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.446+0000 7f8323fff700 1 --2- 172.21.15.53:0/2218421504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f832c1034a0 0x7f832c198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:03:11.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.446+0000 7f832ad9d700 1 --2- 172.21.15.53:0/2218421504 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f832c103e40 0x7f832c19f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:03:11.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.446+0000 7f832a59c700 1 --2- 172.21.15.53:0/2218421504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f832c1022a0 0x7f832c198410 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f831c00d980 tx=0x7f831c00dd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:03:11.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.447+0000 7f83237fe700 1 -- 172.21.15.53:0/2218421504 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f831c017030 con 0x7f832c1022a0 2024-03-28T22:03:11.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.447+0000 7f83237fe700 1 -- 172.21.15.53:0/2218421504 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f831c004500 con 0x7f832c1022a0 2024-03-28T22:03:11.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.447+0000 7f8330d67700 1 -- 172.21.15.53:0/2218421504 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f832c1a0360 con 0x7f832c1022a0 2024-03-28T22:03:11.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.447+0000 7f83237fe700 1 -- 172.21.15.53:0/2218421504 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f831c003b20 con 0x7f832c1022a0 2024-03-28T22:03:11.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.447+0000 7f8330d67700 1 -- 172.21.15.53:0/2218421504 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f832c1a08f0 con 0x7f832c1022a0 2024-03-28T22:03:11.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.448+0000 7f8330d67700 1 -- 172.21.15.53:0/2218421504 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f832c068f00 con 0x7f832c1022a0 2024-03-28T22:03:11.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.449+0000 7f83237fe700 1 -- 172.21.15.53:0/2218421504 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) ==== 136700+0+0 (secure 0 0 0) 0x7f831c01a070 con 0x7f832c1022a0 2024-03-28T22:03:11.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.449+0000 7f83237fe700 1 --2- 172.21.15.53:0/2218421504 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f83180a1e00 0x7f83180a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:03:11.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.449+0000 7f83237fe700 1 -- 172.21.15.53:0/2218421504 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f831c0cbc50 con 0x7f832c1022a0 2024-03-28T22:03:11.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.450+0000 7f8323fff700 1 --2- 172.21.15.53:0/2218421504 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f83180a1e00 0x7f83180a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:03:11.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.450+0000 7f8323fff700 1 --2- 172.21.15.53:0/2218421504 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f83180a1e00 0x7f83180a42c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8314005950 tx=0x7f8314005f30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:03:11.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.454+0000 7f83237fe700 1 -- 172.21.15.53:0/2218421504 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f831c009940 con 0x7f832c1022a0 2024-03-28T22:03:11.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:11.669+0000 7f8330d67700 1 -- 172.21.15.53:0/2218421504 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f832c062680 con 0x7f83180a1e00 2024-03-28T22:03:11.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:11 smithi063 ceph-mon[36249]: osdmap e9: 1 total, 1 up, 1 in 2024-03-28T22:03:11.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:11 smithi063 ceph-mon[36249]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:03:12.489 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:12 smithi053 ceph-mon[35979]: from='client.24155 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:03:12.489 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:12 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:03:12.489 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:12 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:03:12.489 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:12 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:03:12.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:12 smithi063 ceph-mon[36249]: from='client.24155 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:03:12.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:12 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:03:12.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:12 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:03:12.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:12 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:03:12.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:12 smithi016 ceph-mon[32374]: from='client.24155 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:03:12.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:12 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:03:12.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:12 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:03:12.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:12 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:03:13.483 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:13 smithi053 ceph-mon[35979]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:03:13.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:13 smithi063 ceph-mon[36249]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:03:13.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:13 smithi016 ceph-mon[32374]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:03:14.663 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:14 smithi053 ceph-mon[35979]: from='client.? 172.21.15.53:0/3000226109' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "af6d9c22-2830-4ce2-a547-011df0c6cbad"}]: dispatch 2024-03-28T22:03:14.663 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:14 smithi053 ceph-mon[35979]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "af6d9c22-2830-4ce2-a547-011df0c6cbad"}]: dispatch 2024-03-28T22:03:14.663 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:14 smithi053 ceph-mon[35979]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "af6d9c22-2830-4ce2-a547-011df0c6cbad"}]': finished 2024-03-28T22:03:14.663 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:14 smithi053 ceph-mon[35979]: osdmap e10: 2 total, 1 up, 2 in 2024-03-28T22:03:14.663 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:14 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:03:14.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:14 smithi063 ceph-mon[36249]: from='client.? 172.21.15.53:0/3000226109' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "af6d9c22-2830-4ce2-a547-011df0c6cbad"}]: dispatch 2024-03-28T22:03:14.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:14 smithi063 ceph-mon[36249]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "af6d9c22-2830-4ce2-a547-011df0c6cbad"}]: dispatch 2024-03-28T22:03:14.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:14 smithi063 ceph-mon[36249]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "af6d9c22-2830-4ce2-a547-011df0c6cbad"}]': finished 2024-03-28T22:03:14.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:14 smithi063 ceph-mon[36249]: osdmap e10: 2 total, 1 up, 2 in 2024-03-28T22:03:14.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:14 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:03:14.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:14 smithi016 ceph-mon[32374]: from='client.? 172.21.15.53:0/3000226109' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "af6d9c22-2830-4ce2-a547-011df0c6cbad"}]: dispatch 2024-03-28T22:03:14.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:14 smithi016 ceph-mon[32374]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "af6d9c22-2830-4ce2-a547-011df0c6cbad"}]: dispatch 2024-03-28T22:03:14.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:14 smithi016 ceph-mon[32374]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "af6d9c22-2830-4ce2-a547-011df0c6cbad"}]': finished 2024-03-28T22:03:14.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:14 smithi016 ceph-mon[32374]: osdmap e10: 2 total, 1 up, 2 in 2024-03-28T22:03:14.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:14 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:03:15.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:15 smithi063 ceph-mon[36249]: from='client.? 172.21.15.53:0/3205588311' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:03:15.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:15 smithi063 ceph-mon[36249]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:03:15.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:15 smithi016 ceph-mon[32374]: from='client.? 172.21.15.53:0/3205588311' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:03:15.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:15 smithi016 ceph-mon[32374]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:03:15.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:15 smithi053 ceph-mon[35979]: from='client.? 172.21.15.53:0/3205588311' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:03:15.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:15 smithi053 ceph-mon[35979]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:03:17.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:17 smithi063 ceph-mon[36249]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:03:17.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:17 smithi016 ceph-mon[32374]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:03:17.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:17 smithi053 ceph-mon[35979]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:03:19.378 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:19 smithi053 ceph-mon[35979]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:03:19.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:19 smithi063 ceph-mon[36249]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:03:19.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:19 smithi016 ceph-mon[32374]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:03:21.247 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:21 smithi053 ceph-mon[35979]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:03:21.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:21 smithi063 ceph-mon[36249]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:03:21.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:21 smithi016 ceph-mon[32374]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:03:22.174 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:22 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-28T22:03:22.175 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:22 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:03:22.175 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:22 smithi053 ceph-mon[35979]: Deploying daemon osd.1 on smithi053 2024-03-28T22:03:22.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:22 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-28T22:03:22.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:22 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:03:22.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:22 smithi063 ceph-mon[36249]: Deploying daemon osd.1 on smithi053 2024-03-28T22:03:22.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:22 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-28T22:03:22.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:22 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:03:22.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:22 smithi016 ceph-mon[32374]: Deploying daemon osd.1 on smithi053 2024-03-28T22:03:23.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:23 smithi053 ceph-mon[35979]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:03:23.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:23 smithi063 ceph-mon[36249]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:03:23.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:23 smithi016 ceph-mon[32374]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:03:25.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:25 smithi053 ceph-mon[35979]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:03:25.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:25 smithi063 ceph-mon[36249]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:03:25.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:25 smithi016 ceph-mon[32374]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:03:27.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:27 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:27.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:27 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:03:27.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:27 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:27.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:27 smithi053 ceph-mon[35979]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:03:27.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:27 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:27.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:27 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:03:27.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:27 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:27.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:27 smithi063 ceph-mon[36249]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:03:27.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:27 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:27.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:27 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:03:27.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:27 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:27.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:27 smithi016 ceph-mon[32374]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:03:29.389 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:29 smithi053 ceph-mon[35979]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:03:29.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:29 smithi063 ceph-mon[36249]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:03:29.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:29 smithi016 ceph-mon[32374]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:03:29.735 INFO:teuthology.orchestra.run.smithi053.stdout:Created osd(s) 1 on host 'smithi053' 2024-03-28T22:03:29.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:29.732+0000 7f83237fe700 1 -- 172.21.15.53:0/2218421504 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f832c062680 con 0x7f83180a1e00 2024-03-28T22:03:29.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:29.735+0000 7f8330d67700 1 -- 172.21.15.53:0/2218421504 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f83180a1e00 msgr2=0x7f83180a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:03:29.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:29.735+0000 7f8330d67700 1 --2- 172.21.15.53:0/2218421504 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f83180a1e00 0x7f83180a42c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8314005950 tx=0x7f8314005f30 comp rx=0 tx=0).stop 2024-03-28T22:03:29.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:29.735+0000 7f8330d67700 1 -- 172.21.15.53:0/2218421504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f832c1022a0 msgr2=0x7f832c198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:03:29.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:29.735+0000 7f8330d67700 1 --2- 172.21.15.53:0/2218421504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f832c1022a0 0x7f832c198410 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f831c00d980 tx=0x7f831c00dd40 comp rx=0 tx=0).stop 2024-03-28T22:03:29.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:29.735+0000 7f8330d67700 1 -- 172.21.15.53:0/2218421504 shutdown_connections 2024-03-28T22:03:29.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:29.735+0000 7f8330d67700 1 --2- 172.21.15.53:0/2218421504 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f83180a1e00 0x7f83180a42c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:03:29.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:29.735+0000 7f8330d67700 1 --2- 172.21.15.53:0/2218421504 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f832c103e40 0x7f832c19f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:03:29.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:29.735+0000 7f8330d67700 1 --2- 172.21.15.53:0/2218421504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f832c1022a0 0x7f832c198410 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:03:29.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:29.735+0000 7f8330d67700 1 --2- 172.21.15.53:0/2218421504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f832c1034a0 0x7f832c198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:03:29.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:29.735+0000 7f8330d67700 1 -- 172.21.15.53:0/2218421504 >> 172.21.15.53:0/2218421504 conn(0x7f832c0fd840 msgr2=0x7f832c106740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:03:29.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:29.735+0000 7f8330d67700 1 -- 172.21.15.53:0/2218421504 shutdown_connections 2024-03-28T22:03:29.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:03:29.735+0000 7f8330d67700 1 -- 172.21.15.53:0/2218421504 wait complete. 2024-03-28T22:03:30.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:30 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:30.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:30 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:30.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:30 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:03:30.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:30 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:03:30.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:30 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:30.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:30 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:30.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:30 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:30.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:30 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:03:30.363 DEBUG:teuthology.orchestra.run.smithi053:osd.1> sudo journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.1.service 2024-03-28T22:03:30.366 INFO:tasks.cephadm:Deploying osd.2 on smithi063 with /dev/vg_nvme/lv_4... 2024-03-28T22:03:30.366 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-28T22:03:30.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:30 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:30.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:30 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:30.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:30 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:03:30.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:30 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:03:30.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:30 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:30.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:30 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:30.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:30 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:30.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:30 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:03:30.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:30 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:30.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:30 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:30.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:30 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:03:30.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:30 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:03:30.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:30 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:30.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:30 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:30.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:30 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:30.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:30 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:03:30.624 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T22:03:31.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:31 smithi063 ceph-mon[36249]: from='osd.1 [v2:172.21.15.53:6800/795531234,v1:172.21.15.53:6801/795531234]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T22:03:31.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:31 smithi063 ceph-mon[36249]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T22:03:31.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:31 smithi063 ceph-mon[36249]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:03:31.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:31 smithi016 ceph-mon[32374]: from='osd.1 [v2:172.21.15.53:6800/795531234,v1:172.21.15.53:6801/795531234]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T22:03:31.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:31 smithi016 ceph-mon[32374]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T22:03:31.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:31 smithi016 ceph-mon[32374]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:03:31.545 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:31 smithi053 ceph-mon[35979]: from='osd.1 [v2:172.21.15.53:6800/795531234,v1:172.21.15.53:6801/795531234]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T22:03:31.545 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:31 smithi053 ceph-mon[35979]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T22:03:31.545 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:31 smithi053 ceph-mon[35979]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:03:32.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:32 smithi063 ceph-mon[36249]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-28T22:03:32.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:32 smithi063 ceph-mon[36249]: from='osd.1 [v2:172.21.15.53:6800/795531234,v1:172.21.15.53:6801/795531234]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-03-28T22:03:32.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:32 smithi063 ceph-mon[36249]: osdmap e11: 2 total, 1 up, 2 in 2024-03-28T22:03:32.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:32 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:03:32.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:32 smithi063 ceph-mon[36249]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-03-28T22:03:32.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:32 smithi016 ceph-mon[32374]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-28T22:03:32.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:32 smithi016 ceph-mon[32374]: from='osd.1 [v2:172.21.15.53:6800/795531234,v1:172.21.15.53:6801/795531234]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-03-28T22:03:32.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:32 smithi016 ceph-mon[32374]: osdmap e11: 2 total, 1 up, 2 in 2024-03-28T22:03:32.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:32 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:03:32.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:32 smithi016 ceph-mon[32374]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-03-28T22:03:32.475 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:32 smithi053 ceph-mon[35979]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-28T22:03:32.475 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:32 smithi053 ceph-mon[35979]: from='osd.1 [v2:172.21.15.53:6800/795531234,v1:172.21.15.53:6801/795531234]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-03-28T22:03:32.475 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:32 smithi053 ceph-mon[35979]: osdmap e11: 2 total, 1 up, 2 in 2024-03-28T22:03:32.475 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:32 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:03:32.475 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:32 smithi053 ceph-mon[35979]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-03-28T22:03:32.475 INFO:journalctl@ceph.osd.1.smithi053.stdout:Mar 28 22:03:32 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-1[40596]: 2024-03-28T22:03:32.168+0000 7f793e43e700 -1 osd.1 0 waiting for initial osdmap 2024-03-28T22:03:32.476 INFO:journalctl@ceph.osd.1.smithi053.stdout:Mar 28 22:03:32 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-1[40596]: 2024-03-28T22:03:32.173+0000 7f793aa32700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T22:03:32.638 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-28T22:03:32.638 INFO:teuthology.orchestra.run.smithi063.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-28T22:03:32.638 INFO:teuthology.orchestra.run.smithi063.stderr: stderr: 10+0 records in 2024-03-28T22:03:32.639 INFO:teuthology.orchestra.run.smithi063.stderr:10+0 records out 2024-03-28T22:03:32.639 INFO:teuthology.orchestra.run.smithi063.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111252 s, 943 MB/s 2024-03-28T22:03:32.639 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping successful for: 2024-03-28T22:03:33.213 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi063:vg_nvme/lv_4 2024-03-28T22:03:33.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:33 smithi063 ceph-mon[36249]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-03-28T22:03:33.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:33 smithi063 ceph-mon[36249]: osdmap e12: 2 total, 1 up, 2 in 2024-03-28T22:03:33.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:33 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:03:33.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:33 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:03:33.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:33 smithi063 ceph-mon[36249]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:03:33.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:33 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:33.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:33 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:33.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:33 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:03:33.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:33 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:33.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:33 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:03:33.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:33 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:03:33.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:33 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:33.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:33 smithi016 ceph-mon[32374]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-03-28T22:03:33.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:33 smithi016 ceph-mon[32374]: osdmap e12: 2 total, 1 up, 2 in 2024-03-28T22:03:33.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:33 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:03:33.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:33 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:03:33.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:33 smithi016 ceph-mon[32374]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:03:33.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:33 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:33.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:33 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:33.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:33 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:03:33.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:33 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:33.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:33 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:03:33.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:33 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:03:33.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:33 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:33.462 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T22:03:33.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:33 smithi053 ceph-mon[35979]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-03-28T22:03:33.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:33 smithi053 ceph-mon[35979]: osdmap e12: 2 total, 1 up, 2 in 2024-03-28T22:03:33.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:33 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:03:33.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:33 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:03:33.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:33 smithi053 ceph-mon[35979]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:03:33.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:33 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:33.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:33 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:33.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:33 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:03:33.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:33 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:33.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:33 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:03:33.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:33 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:03:33.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:33 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:34.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:34 smithi063 ceph-mon[36249]: purged_snaps scrub starts 2024-03-28T22:03:34.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:34 smithi063 ceph-mon[36249]: purged_snaps scrub ok 2024-03-28T22:03:34.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:34 smithi063 ceph-mon[36249]: Detected new or changed devices on smithi053 2024-03-28T22:03:34.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:34 smithi063 ceph-mon[36249]: Adjusting osd_memory_target on smithi053 to 17084M 2024-03-28T22:03:34.434 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:34 smithi063 ceph-mon[36249]: osd.1 [v2:172.21.15.53:6800/795531234,v1:172.21.15.53:6801/795531234] boot 2024-03-28T22:03:34.434 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:34 smithi063 ceph-mon[36249]: osdmap e13: 2 total, 2 up, 2 in 2024-03-28T22:03:34.434 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:34 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:03:34.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:34 smithi016 ceph-mon[32374]: purged_snaps scrub starts 2024-03-28T22:03:34.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:34 smithi016 ceph-mon[32374]: purged_snaps scrub ok 2024-03-28T22:03:34.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:34 smithi016 ceph-mon[32374]: Detected new or changed devices on smithi053 2024-03-28T22:03:34.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:34 smithi016 ceph-mon[32374]: Adjusting osd_memory_target on smithi053 to 17084M 2024-03-28T22:03:34.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:34 smithi016 ceph-mon[32374]: osd.1 [v2:172.21.15.53:6800/795531234,v1:172.21.15.53:6801/795531234] boot 2024-03-28T22:03:34.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:34 smithi016 ceph-mon[32374]: osdmap e13: 2 total, 2 up, 2 in 2024-03-28T22:03:34.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:34 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:03:34.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:34 smithi053 ceph-mon[35979]: purged_snaps scrub starts 2024-03-28T22:03:34.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:34 smithi053 ceph-mon[35979]: purged_snaps scrub ok 2024-03-28T22:03:34.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:34 smithi053 ceph-mon[35979]: Detected new or changed devices on smithi053 2024-03-28T22:03:34.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:34 smithi053 ceph-mon[35979]: Adjusting osd_memory_target on smithi053 to 17084M 2024-03-28T22:03:34.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:34 smithi053 ceph-mon[35979]: osd.1 [v2:172.21.15.53:6800/795531234,v1:172.21.15.53:6801/795531234] boot 2024-03-28T22:03:34.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:34 smithi053 ceph-mon[35979]: osdmap e13: 2 total, 2 up, 2 in 2024-03-28T22:03:34.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:34 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:03:34.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.848+0000 7f9e592ff700 1 -- 172.21.15.63:0/4221654211 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e541011e0 msgr2=0x7f9e54101600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:03:34.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.848+0000 7f9e592ff700 1 --2- 172.21.15.63:0/4221654211 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e541011e0 0x7f9e54101600 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9e44009990 tx=0x7f9e44009ca0 comp rx=0 tx=0).stop 2024-03-28T22:03:34.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.848+0000 7f9e592ff700 1 -- 172.21.15.63:0/4221654211 shutdown_connections 2024-03-28T22:03:34.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.848+0000 7f9e592ff700 1 --2- 172.21.15.63:0/4221654211 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e54102d80 0x7f9e54109630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:03:34.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.848+0000 7f9e592ff700 1 --2- 172.21.15.63:0/4221654211 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e541011e0 0x7f9e54101600 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:03:34.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.848+0000 7f9e592ff700 1 --2- 172.21.15.63:0/4221654211 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e541023e0 0x7f9e54102840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:03:34.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.848+0000 7f9e592ff700 1 -- 172.21.15.63:0/4221654211 >> 172.21.15.63:0/4221654211 conn(0x7f9e540fc7a0 msgr2=0x7f9e540febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:03:34.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.848+0000 7f9e592ff700 1 -- 172.21.15.63:0/4221654211 shutdown_connections 2024-03-28T22:03:34.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.848+0000 7f9e592ff700 1 -- 172.21.15.63:0/4221654211 wait complete. 2024-03-28T22:03:34.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.849+0000 7f9e592ff700 1 Processor -- start 2024-03-28T22:03:34.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.849+0000 7f9e592ff700 1 -- start start 2024-03-28T22:03:34.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.850+0000 7f9e592ff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e541011e0 0x7f9e54197420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:03:34.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.850+0000 7f9e592ff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e541023e0 0x7f9e54197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:03:34.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.850+0000 7f9e592ff700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e54102d80 0x7f9e5419e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:03:34.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.850+0000 7f9e592ff700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9e5410cdc0 con 0x7f9e541023e0 2024-03-28T22:03:34.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.850+0000 7f9e592ff700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9e5410cc40 con 0x7f9e541011e0 2024-03-28T22:03:34.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.850+0000 7f9e592ff700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9e5410cf40 con 0x7f9e54102d80 2024-03-28T22:03:34.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.850+0000 7f9e537fe700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e54102d80 0x7f9e5419e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:03:34.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.850+0000 7f9e537fe700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e54102d80 0x7f9e5419e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47362/0 (socket says 172.21.15.63:47362) 2024-03-28T22:03:34.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.850+0000 7f9e52ffd700 1 --2- 172.21.15.63:0/2596283328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e541011e0 0x7f9e54197420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:03:34.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.850+0000 7f9e537fe700 1 -- 172.21.15.63:0/2596283328 learned_addr learned my addr 172.21.15.63:0/2596283328 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-28T22:03:34.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.850+0000 7f9e527fc700 1 --2- 172.21.15.63:0/2596283328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e541023e0 0x7f9e54197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:03:34.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.850+0000 7f9e537fe700 1 -- 172.21.15.63:0/2596283328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e541011e0 msgr2=0x7f9e54197420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:03:34.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.850+0000 7f9e537fe700 1 --2- 172.21.15.63:0/2596283328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e541011e0 0x7f9e54197420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:03:34.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.850+0000 7f9e537fe700 1 -- 172.21.15.63:0/2596283328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e541023e0 msgr2=0x7f9e54197960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:03:34.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.850+0000 7f9e537fe700 1 --2- 172.21.15.63:0/2596283328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e541023e0 0x7f9e54197960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:03:34.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.850+0000 7f9e537fe700 1 -- 172.21.15.63:0/2596283328 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e5419f080 con 0x7f9e54102d80 2024-03-28T22:03:34.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.850+0000 7f9e537fe700 1 --2- 172.21.15.63:0/2596283328 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e54102d80 0x7f9e5419e9e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c007f40 tx=0x7f9e4c00f5c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:03:34.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.850+0000 7f9e3bfff700 1 -- 172.21.15.63:0/2596283328 <== mon.1 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e4c016030 con 0x7f9e54102d80 2024-03-28T22:03:34.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.850+0000 7f9e3bfff700 1 -- 172.21.15.63:0/2596283328 <== mon.1 v2:172.21.15.63:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e4c00fd30 con 0x7f9e54102d80 2024-03-28T22:03:34.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.851+0000 7f9e592ff700 1 -- 172.21.15.63:0/2596283328 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e5406a9a0 con 0x7f9e54102d80 2024-03-28T22:03:34.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.851+0000 7f9e3bfff700 1 -- 172.21.15.63:0/2596283328 <== mon.1 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e4c005020 con 0x7f9e54102d80 2024-03-28T22:03:34.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.851+0000 7f9e592ff700 1 -- 172.21.15.63:0/2596283328 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e5406af60 con 0x7f9e54102d80 2024-03-28T22:03:34.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.852+0000 7f9e592ff700 1 -- 172.21.15.63:0/2596283328 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e54068f00 con 0x7f9e54102d80 2024-03-28T22:03:34.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.853+0000 7f9e3bfff700 1 -- 172.21.15.63:0/2596283328 <== mon.1 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 15) ==== 136700+0+0 (secure 0 0 0) 0x7f9e4c004c40 con 0x7f9e54102d80 2024-03-28T22:03:34.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.853+0000 7f9e3bfff700 1 --2- 172.21.15.63:0/2596283328 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f9e3c0a1e00 0x7f9e3c0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:03:34.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.853+0000 7f9e3bfff700 1 -- 172.21.15.63:0/2596283328 <== mon.1 v2:172.21.15.63:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f9e4c0ce740 con 0x7f9e54102d80 2024-03-28T22:03:34.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.854+0000 7f9e52ffd700 1 --2- 172.21.15.63:0/2596283328 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f9e3c0a1e00 0x7f9e3c0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:03:34.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.854+0000 7f9e52ffd700 1 --2- 172.21.15.63:0/2596283328 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f9e3c0a1e00 0x7f9e3c0a42c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9e44003890 tx=0x7f9e44005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:03:34.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:34.858+0000 7f9e3bfff700 1 -- 172.21.15.63:0/2596283328 <== mon.1 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9e4c09d3c0 con 0x7f9e54102d80 2024-03-28T22:03:35.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:35.071+0000 7f9e592ff700 1 -- 172.21.15.63:0/2596283328 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f9e5406b210 con 0x7f9e3c0a1e00 2024-03-28T22:03:35.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:35 smithi063 ceph-mon[36249]: osdmap e14: 2 total, 2 up, 2 in 2024-03-28T22:03:35.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:35 smithi063 ceph-mon[36249]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:03:35.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:35 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:03:35.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:35 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:03:35.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:35 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:03:35.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:35 smithi016 ceph-mon[32374]: osdmap e14: 2 total, 2 up, 2 in 2024-03-28T22:03:35.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:35 smithi016 ceph-mon[32374]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:03:35.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:35 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:03:35.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:35 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:03:35.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:35 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:03:35.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:35 smithi053 ceph-mon[35979]: osdmap e14: 2 total, 2 up, 2 in 2024-03-28T22:03:35.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:35 smithi053 ceph-mon[35979]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:03:35.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:35 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:03:35.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:35 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:03:35.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:35 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:03:36.427 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:36 smithi063 ceph-mon[36249]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:03:36.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:36 smithi053 ceph-mon[35979]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:03:36.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:36 smithi016 ceph-mon[32374]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:03:37.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:37 smithi053 ceph-mon[35979]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:03:37.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:37 smithi063 ceph-mon[36249]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:03:37.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:37 smithi016 ceph-mon[32374]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:03:38.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:38 smithi063 ceph-mon[36249]: from='client.? 172.21.15.63:0/3449426922' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b10adca9-110c-4f87-85e0-b7542709d31a"}]: dispatch 2024-03-28T22:03:38.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:38 smithi063 ceph-mon[36249]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b10adca9-110c-4f87-85e0-b7542709d31a"}]: dispatch 2024-03-28T22:03:38.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:38 smithi063 ceph-mon[36249]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b10adca9-110c-4f87-85e0-b7542709d31a"}]': finished 2024-03-28T22:03:38.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:38 smithi063 ceph-mon[36249]: osdmap e15: 3 total, 2 up, 3 in 2024-03-28T22:03:38.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:38 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:03:38.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:38 smithi053 ceph-mon[35979]: from='client.? 172.21.15.63:0/3449426922' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b10adca9-110c-4f87-85e0-b7542709d31a"}]: dispatch 2024-03-28T22:03:38.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:38 smithi053 ceph-mon[35979]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b10adca9-110c-4f87-85e0-b7542709d31a"}]: dispatch 2024-03-28T22:03:38.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:38 smithi053 ceph-mon[35979]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b10adca9-110c-4f87-85e0-b7542709d31a"}]': finished 2024-03-28T22:03:38.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:38 smithi053 ceph-mon[35979]: osdmap e15: 3 total, 2 up, 3 in 2024-03-28T22:03:38.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:38 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:03:38.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:38 smithi016 ceph-mon[32374]: from='client.? 172.21.15.63:0/3449426922' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b10adca9-110c-4f87-85e0-b7542709d31a"}]: dispatch 2024-03-28T22:03:38.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:38 smithi016 ceph-mon[32374]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b10adca9-110c-4f87-85e0-b7542709d31a"}]: dispatch 2024-03-28T22:03:38.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:38 smithi016 ceph-mon[32374]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b10adca9-110c-4f87-85e0-b7542709d31a"}]': finished 2024-03-28T22:03:38.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:38 smithi016 ceph-mon[32374]: osdmap e15: 3 total, 2 up, 3 in 2024-03-28T22:03:38.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:38 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:03:39.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:39 smithi053 ceph-mon[35979]: from='client.? 172.21.15.63:0/3634342019' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:03:39.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:39 smithi053 ceph-mon[35979]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:03:39.586 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:39 smithi016 ceph-mon[32374]: from='client.? 172.21.15.63:0/3634342019' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:03:39.586 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:39 smithi016 ceph-mon[32374]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:03:39.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:39 smithi063 ceph-mon[36249]: from='client.? 172.21.15.63:0/3634342019' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:03:39.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:39 smithi063 ceph-mon[36249]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:03:41.349 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:41 smithi053 ceph-mon[35979]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:03:41.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:41 smithi063 ceph-mon[36249]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:03:41.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:41 smithi016 ceph-mon[32374]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:03:43.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:43 smithi063 ceph-mon[36249]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:03:43.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:43 smithi016 ceph-mon[32374]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:03:43.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:43 smithi053 ceph-mon[35979]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:03:45.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:45 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-28T22:03:45.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:45 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:03:45.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:45 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-28T22:03:45.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:45 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:03:45.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:45 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-28T22:03:45.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:45 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:03:46.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:46 smithi063 ceph-mon[36249]: Deploying daemon osd.2 on smithi063 2024-03-28T22:03:46.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:46 smithi063 ceph-mon[36249]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:03:46.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:46 smithi053 ceph-mon[35979]: Deploying daemon osd.2 on smithi063 2024-03-28T22:03:46.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:46 smithi053 ceph-mon[35979]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:03:46.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:46 smithi016 ceph-mon[32374]: Deploying daemon osd.2 on smithi063 2024-03-28T22:03:46.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:46 smithi016 ceph-mon[32374]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:03:48.163 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:48 smithi063 ceph-mon[36249]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:03:48.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:48 smithi053 ceph-mon[35979]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:03:48.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:48 smithi016 ceph-mon[32374]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:03:50.168 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:50 smithi063 ceph-mon[36249]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:03:50.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:50 smithi053 ceph-mon[35979]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:03:50.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:50 smithi016 ceph-mon[32374]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:03:51.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:51 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:51.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:51 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:51.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:51 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:03:51.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:51 smithi063 ceph-mon[36249]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:03:51.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:51 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:51.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:51 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:51.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:51 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:03:51.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:51 smithi016 ceph-mon[32374]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:03:51.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:51 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:51.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:51 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:51.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:51 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:03:51.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:51 smithi053 ceph-mon[35979]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:03:53.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:53 smithi063 ceph-mon[36249]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:03:53.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:53 smithi016 ceph-mon[32374]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:03:53.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:53 smithi053 ceph-mon[35979]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:03:53.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:53.842+0000 7f9e3bfff700 1 -- 172.21.15.63:0/2596283328 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9e5406b210 con 0x7f9e3c0a1e00 2024-03-28T22:03:53.845 INFO:teuthology.orchestra.run.smithi063.stdout:Created osd(s) 2 on host 'smithi063' 2024-03-28T22:03:53.846 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:53.844+0000 7f9e592ff700 1 -- 172.21.15.63:0/2596283328 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f9e3c0a1e00 msgr2=0x7f9e3c0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:03:53.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:53.844+0000 7f9e592ff700 1 --2- 172.21.15.63:0/2596283328 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f9e3c0a1e00 0x7f9e3c0a42c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9e44003890 tx=0x7f9e44005fd0 comp rx=0 tx=0).stop 2024-03-28T22:03:53.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:53.844+0000 7f9e592ff700 1 -- 172.21.15.63:0/2596283328 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e54102d80 msgr2=0x7f9e5419e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:03:53.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:53.844+0000 7f9e592ff700 1 --2- 172.21.15.63:0/2596283328 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e54102d80 0x7f9e5419e9e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c007f40 tx=0x7f9e4c00f5c0 comp rx=0 tx=0).stop 2024-03-28T22:03:53.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:53.844+0000 7f9e592ff700 1 -- 172.21.15.63:0/2596283328 shutdown_connections 2024-03-28T22:03:53.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:53.844+0000 7f9e592ff700 1 --2- 172.21.15.63:0/2596283328 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f9e3c0a1e00 0x7f9e3c0a42c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:03:53.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:53.844+0000 7f9e592ff700 1 --2- 172.21.15.63:0/2596283328 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e54102d80 0x7f9e5419e9e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:03:53.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:53.844+0000 7f9e592ff700 1 --2- 172.21.15.63:0/2596283328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e541011e0 0x7f9e54197420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:03:53.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:53.844+0000 7f9e592ff700 1 --2- 172.21.15.63:0/2596283328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e541023e0 0x7f9e54197960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:03:53.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:53.844+0000 7f9e592ff700 1 -- 172.21.15.63:0/2596283328 >> 172.21.15.63:0/2596283328 conn(0x7f9e540fc7a0 msgr2=0x7f9e54105780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:03:53.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:53.844+0000 7f9e592ff700 1 -- 172.21.15.63:0/2596283328 shutdown_connections 2024-03-28T22:03:53.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:03:53.844+0000 7f9e592ff700 1 -- 172.21.15.63:0/2596283328 wait complete. 2024-03-28T22:03:54.485 DEBUG:teuthology.orchestra.run.smithi063:osd.2> sudo journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.2.service 2024-03-28T22:03:54.487 INFO:tasks.cephadm:Deploying osd.3 on smithi016 with /dev/vg_nvme/lv_3... 2024-03-28T22:03:54.487 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-28T22:03:54.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:54 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:54.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:54 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:54.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:54 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:03:54.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:54 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:03:54.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:54 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:54.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:54 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:54.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:54 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:54.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:54 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:03:54.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:54 smithi063 ceph-mon[36249]: from='osd.2 [v2:172.21.15.63:6800/2214131086,v1:172.21.15.63:6801/2214131086]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T22:03:54.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:54 smithi063 ceph-mon[36249]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T22:03:54.737 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:03:54.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:54 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:54.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:54 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:54.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:54 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:03:54.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:54 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:03:54.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:54 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:54.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:54 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:54.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:54 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:54.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:54 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:03:54.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:54 smithi053 ceph-mon[35979]: from='osd.2 [v2:172.21.15.63:6800/2214131086,v1:172.21.15.63:6801/2214131086]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T22:03:54.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:54 smithi053 ceph-mon[35979]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T22:03:54.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:54 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:54.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:54 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:54.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:54 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:03:54.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:54 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:03:54.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:54 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:54.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:54 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:54.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:54 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:54.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:54 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:03:54.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:54 smithi016 ceph-mon[32374]: from='osd.2 [v2:172.21.15.63:6800/2214131086,v1:172.21.15.63:6801/2214131086]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T22:03:54.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:54 smithi016 ceph-mon[32374]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T22:03:56.086 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:55 smithi016 ceph-mon[32374]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-28T22:03:56.086 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:55 smithi016 ceph-mon[32374]: osdmap e16: 3 total, 2 up, 3 in 2024-03-28T22:03:56.086 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:55 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:03:56.086 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:55 smithi016 ceph-mon[32374]: from='osd.2 [v2:172.21.15.63:6800/2214131086,v1:172.21.15.63:6801/2214131086]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-28T22:03:56.086 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:55 smithi016 ceph-mon[32374]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-28T22:03:56.086 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:55 smithi016 ceph-mon[32374]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:03:56.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:55 smithi063 ceph-mon[36249]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-28T22:03:56.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:55 smithi063 ceph-mon[36249]: osdmap e16: 3 total, 2 up, 3 in 2024-03-28T22:03:56.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:55 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:03:56.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:55 smithi063 ceph-mon[36249]: from='osd.2 [v2:172.21.15.63:6800/2214131086,v1:172.21.15.63:6801/2214131086]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-28T22:03:56.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:55 smithi063 ceph-mon[36249]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-28T22:03:56.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:55 smithi063 ceph-mon[36249]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:03:56.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:55 smithi053 ceph-mon[35979]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-28T22:03:56.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:55 smithi053 ceph-mon[35979]: osdmap e16: 3 total, 2 up, 3 in 2024-03-28T22:03:56.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:55 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:03:56.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:55 smithi053 ceph-mon[35979]: from='osd.2 [v2:172.21.15.63:6800/2214131086,v1:172.21.15.63:6801/2214131086]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-28T22:03:56.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:55 smithi053 ceph-mon[35979]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-28T22:03:56.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:55 smithi053 ceph-mon[35979]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:03:56.630 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-28T22:03:56.630 INFO:teuthology.orchestra.run.smithi016.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-28T22:03:56.630 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10+0 records in 2024-03-28T22:03:56.630 INFO:teuthology.orchestra.run.smithi016.stderr:10+0 records out 2024-03-28T22:03:56.631 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00838563 s, 1.3 GB/s 2024-03-28T22:03:56.631 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping successful for: 2024-03-28T22:03:57.071 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:56 smithi063 ceph-mon[36249]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-03-28T22:03:57.072 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:56 smithi063 ceph-mon[36249]: osdmap e17: 3 total, 2 up, 3 in 2024-03-28T22:03:57.072 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:56 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:03:57.072 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:56 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:03:57.072 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:56 smithi063 ceph-mon[36249]: from='osd.2 ' entity='osd.2' 2024-03-28T22:03:57.072 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 28 22:03:56 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-2[41397]: 2024-03-28T22:03:56.735+0000 7f0141a69700 -1 osd.2 0 waiting for initial osdmap 2024-03-28T22:03:57.072 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 28 22:03:56 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-2[41397]: 2024-03-28T22:03:56.742+0000 7f013a055700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T22:03:57.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:56 smithi016 ceph-mon[32374]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-03-28T22:03:57.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:56 smithi016 ceph-mon[32374]: osdmap e17: 3 total, 2 up, 3 in 2024-03-28T22:03:57.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:56 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:03:57.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:56 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:03:57.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:56 smithi016 ceph-mon[32374]: from='osd.2 ' entity='osd.2' 2024-03-28T22:03:57.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:56 smithi053 ceph-mon[35979]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-03-28T22:03:57.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:56 smithi053 ceph-mon[35979]: osdmap e17: 3 total, 2 up, 3 in 2024-03-28T22:03:57.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:56 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:03:57.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:56 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:03:57.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:56 smithi053 ceph-mon[35979]: from='osd.2 ' entity='osd.2' 2024-03-28T22:03:57.327 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi016:vg_nvme/lv_3 2024-03-28T22:03:57.579 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:03:57.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:57 smithi016 ceph-mon[32374]: purged_snaps scrub starts 2024-03-28T22:03:57.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:57 smithi016 ceph-mon[32374]: purged_snaps scrub ok 2024-03-28T22:03:57.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:57 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:03:57.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:57 smithi016 ceph-mon[32374]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:03:57.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:57 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:57.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:57 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:57.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:57 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:03:57.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:57 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:57.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:57 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:03:57.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:57 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:03:57.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:57 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:57.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:57 smithi016 ceph-mon[32374]: osd.2 [v2:172.21.15.63:6800/2214131086,v1:172.21.15.63:6801/2214131086] boot 2024-03-28T22:03:57.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:57 smithi016 ceph-mon[32374]: osdmap e18: 3 total, 3 up, 3 in 2024-03-28T22:03:57.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:57 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:03:58.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:57 smithi063 ceph-mon[36249]: purged_snaps scrub starts 2024-03-28T22:03:58.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:57 smithi063 ceph-mon[36249]: purged_snaps scrub ok 2024-03-28T22:03:58.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:57 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:03:58.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:57 smithi063 ceph-mon[36249]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:03:58.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:57 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:58.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:57 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:58.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:57 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:03:58.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:57 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:58.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:57 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:03:58.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:57 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:03:58.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:57 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:58.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:57 smithi063 ceph-mon[36249]: osd.2 [v2:172.21.15.63:6800/2214131086,v1:172.21.15.63:6801/2214131086] boot 2024-03-28T22:03:58.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:57 smithi063 ceph-mon[36249]: osdmap e18: 3 total, 3 up, 3 in 2024-03-28T22:03:58.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:57 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:03:58.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:57 smithi053 ceph-mon[35979]: purged_snaps scrub starts 2024-03-28T22:03:58.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:57 smithi053 ceph-mon[35979]: purged_snaps scrub ok 2024-03-28T22:03:58.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:57 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:03:58.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:57 smithi053 ceph-mon[35979]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:03:58.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:57 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:58.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:57 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:58.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:57 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:03:58.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:57 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:58.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:57 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:03:58.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:57 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:03:58.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:57 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:03:58.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:57 smithi053 ceph-mon[35979]: osd.2 [v2:172.21.15.63:6800/2214131086,v1:172.21.15.63:6801/2214131086] boot 2024-03-28T22:03:58.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:57 smithi053 ceph-mon[35979]: osdmap e18: 3 total, 3 up, 3 in 2024-03-28T22:03:58.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:57 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:03:58.975 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:58 smithi016 ceph-mon[32374]: Detected new or changed devices on smithi063 2024-03-28T22:03:58.975 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:58 smithi016 ceph-mon[32374]: Adjusting osd_memory_target on smithi063 to 17080M 2024-03-28T22:03:59.181 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:58 smithi063 ceph-mon[36249]: Detected new or changed devices on smithi063 2024-03-28T22:03:59.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:58 smithi063 ceph-mon[36249]: Adjusting osd_memory_target on smithi063 to 17080M 2024-03-28T22:03:59.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:58 smithi053 ceph-mon[35979]: Detected new or changed devices on smithi063 2024-03-28T22:03:59.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:58 smithi053 ceph-mon[35979]: Adjusting osd_memory_target on smithi063 to 17080M 2024-03-28T22:03:59.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.327+0000 7fdb06a2b700 1 -- 172.21.15.16:0/691928424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb00103850 msgr2=0x7fdb00103c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:03:59.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.327+0000 7fdb06a2b700 1 --2- 172.21.15.16:0/691928424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb00103850 0x7fdb00103c70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdaf0009990 tx=0x7fdaf0009ca0 comp rx=0 tx=0).stop 2024-03-28T22:03:59.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.327+0000 7fdb06a2b700 1 -- 172.21.15.16:0/691928424 shutdown_connections 2024-03-28T22:03:59.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.327+0000 7fdb06a2b700 1 --2- 172.21.15.16:0/691928424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb00100c70 0x7fdb0010a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:03:59.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.327+0000 7fdb06a2b700 1 --2- 172.21.15.16:0/691928424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb00103850 0x7fdb00103c70 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:03:59.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.327+0000 7fdb06a2b700 1 --2- 172.21.15.16:0/691928424 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb001002d0 0x7fdb00100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:03:59.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.327+0000 7fdb06a2b700 1 -- 172.21.15.16:0/691928424 >> 172.21.15.16:0/691928424 conn(0x7fdb000fbeb0 msgr2=0x7fdb000fe2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:03:59.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.327+0000 7fdb06a2b700 1 -- 172.21.15.16:0/691928424 shutdown_connections 2024-03-28T22:03:59.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.327+0000 7fdb06a2b700 1 -- 172.21.15.16:0/691928424 wait complete. 2024-03-28T22:03:59.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.328+0000 7fdb06a2b700 1 Processor -- start 2024-03-28T22:03:59.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.328+0000 7fdb06a2b700 1 -- start start 2024-03-28T22:03:59.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.328+0000 7fdb06a2b700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb001002d0 0x7fdb001984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:03:59.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.328+0000 7fdb06a2b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb00100c70 0x7fdb00198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:03:59.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.328+0000 7fdb06a2b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb00103850 0x7fdb0019faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:03:59.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.328+0000 7fdb06a2b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdb0010dea0 con 0x7fdb00103850 2024-03-28T22:03:59.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.328+0000 7fdb06a2b700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fdb0010dd20 con 0x7fdb00100c70 2024-03-28T22:03:59.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.328+0000 7fdb06a2b700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fdb0010e020 con 0x7fdb001002d0 2024-03-28T22:03:59.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.328+0000 7fdb04fc8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb00103850 0x7fdb0019faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:03:59.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.328+0000 7fdb04fc8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb00103850 0x7fdb0019faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44764/0 (socket says 172.21.15.16:44764) 2024-03-28T22:03:59.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.328+0000 7fdb04fc8700 1 -- 172.21.15.16:0/2140646576 learned_addr learned my addr 172.21.15.16:0/2140646576 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:03:59.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.328+0000 7fdafffff700 1 --2- 172.21.15.16:0/2140646576 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb001002d0 0x7fdb001984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:03:59.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.328+0000 7fdaff7fe700 1 --2- 172.21.15.16:0/2140646576 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb00100c70 0x7fdb00198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:03:59.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.329+0000 7fdb04fc8700 1 -- 172.21.15.16:0/2140646576 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb001002d0 msgr2=0x7fdb001984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:03:59.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.329+0000 7fdb04fc8700 1 --2- 172.21.15.16:0/2140646576 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb001002d0 0x7fdb001984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:03:59.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.329+0000 7fdb04fc8700 1 -- 172.21.15.16:0/2140646576 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb00100c70 msgr2=0x7fdb00198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:03:59.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.329+0000 7fdb04fc8700 1 --2- 172.21.15.16:0/2140646576 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb00100c70 0x7fdb00198a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:03:59.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.329+0000 7fdb04fc8700 1 -- 172.21.15.16:0/2140646576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb001a0140 con 0x7fdb00103850 2024-03-28T22:03:59.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.329+0000 7fdb04fc8700 1 --2- 172.21.15.16:0/2140646576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb00103850 0x7fdb0019faa0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdaf800c3a0 tx=0x7fdaf800c6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:03:59.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.329+0000 7fdafd7fa700 1 -- 172.21.15.16:0/2140646576 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaf80092e0 con 0x7fdb00103850 2024-03-28T22:03:59.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.329+0000 7fdb06a2b700 1 -- 172.21.15.16:0/2140646576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb001a03d0 con 0x7fdb00103850 2024-03-28T22:03:59.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.329+0000 7fdafd7fa700 1 -- 172.21.15.16:0/2140646576 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdaf800cb90 con 0x7fdb00103850 2024-03-28T22:03:59.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.329+0000 7fdafd7fa700 1 -- 172.21.15.16:0/2140646576 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaf80047f0 con 0x7fdb00103850 2024-03-28T22:03:59.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.329+0000 7fdb06a2b700 1 -- 172.21.15.16:0/2140646576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb001a08c0 con 0x7fdb00103850 2024-03-28T22:03:59.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.330+0000 7fdb06a2b700 1 -- 172.21.15.16:0/2140646576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb00068f00 con 0x7fdb00103850 2024-03-28T22:03:59.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.331+0000 7fdafd7fa700 1 -- 172.21.15.16:0/2140646576 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 15) ==== 136700+0+0 (secure 0 0 0) 0x7fdaf801a020 con 0x7fdb00103850 2024-03-28T22:03:59.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.332+0000 7fdafd7fa700 1 --2- 172.21.15.16:0/2140646576 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdaec0a1d80 0x7fdaec0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:03:59.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.332+0000 7fdafffff700 1 --2- 172.21.15.16:0/2140646576 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdaec0a1d80 0x7fdaec0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:03:59.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.332+0000 7fdafd7fa700 1 -- 172.21.15.16:0/2140646576 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 2851+0+0 (secure 0 0 0) 0x7fdaf80cc600 con 0x7fdb00103850 2024-03-28T22:03:59.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.332+0000 7fdafffff700 1 --2- 172.21.15.16:0/2140646576 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdaec0a1d80 0x7fdaec0a4240 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdaf000b750 tx=0x7fdaf0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:03:59.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.336+0000 7fdafd7fa700 1 -- 172.21.15.16:0/2140646576 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fdaf8010040 con 0x7fdb00103850 2024-03-28T22:03:59.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:03:59.550+0000 7fdb06a2b700 1 -- 172.21.15.16:0/2140646576 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fdb00062680 con 0x7fdaec0a1d80 2024-03-28T22:04:00.150 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:59 smithi016 ceph-mon[32374]: osdmap e19: 3 total, 3 up, 3 in 2024-03-28T22:04:00.150 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:59 smithi016 ceph-mon[32374]: pgmap v84: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:04:00.150 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:59 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-28T22:04:00.150 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:59 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:04:00.150 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:59 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:04:00.150 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:03:59 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:04:00.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:59 smithi063 ceph-mon[36249]: osdmap e19: 3 total, 3 up, 3 in 2024-03-28T22:04:00.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:59 smithi063 ceph-mon[36249]: pgmap v84: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:04:00.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:59 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-28T22:04:00.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:59 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:04:00.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:59 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:04:00.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:03:59 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:04:00.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:59 smithi053 ceph-mon[35979]: osdmap e19: 3 total, 3 up, 3 in 2024-03-28T22:04:00.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:59 smithi053 ceph-mon[35979]: pgmap v84: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:04:00.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:59 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-28T22:04:00.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:59 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:04:00.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:59 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:04:00.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:03:59 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:04:01.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:00 smithi016 ceph-mon[32374]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:04:01.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:00 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-28T22:04:01.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:00 smithi016 ceph-mon[32374]: osdmap e20: 3 total, 3 up, 3 in 2024-03-28T22:04:01.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:00 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T22:04:01.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:00 smithi063 ceph-mon[36249]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:04:01.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:00 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-28T22:04:01.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:00 smithi063 ceph-mon[36249]: osdmap e20: 3 total, 3 up, 3 in 2024-03-28T22:04:01.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:00 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T22:04:01.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:00 smithi053 ceph-mon[35979]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:04:01.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:00 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-28T22:04:01.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:00 smithi053 ceph-mon[35979]: osdmap e20: 3 total, 3 up, 3 in 2024-03-28T22:04:01.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:00 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T22:04:01.377 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 28 22:04:01 smithi016 sudo[49287]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T22:04:01.377 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 28 22:04:01 smithi016 sudo[49287]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T22:04:01.377 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 28 22:04:01 smithi016 sudo[49287]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T22:04:01.378 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 28 22:04:01 smithi016 sudo[49287]: pam_unix(sudo:session): session closed for user root 2024-03-28T22:04:01.635 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 28 22:04:01 smithi016 sudo[49294]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T22:04:01.636 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 28 22:04:01 smithi016 sudo[49294]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T22:04:01.636 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 28 22:04:01 smithi016 sudo[49294]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T22:04:01.636 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 28 22:04:01 smithi016 sudo[49294]: pam_unix(sudo:session): session closed for user root 2024-03-28T22:04:01.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:04:01.862+0000 7fdafd7fa700 1 -- 172.21.15.16:0/2140646576 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fdaf8012070 con 0x7fdb00103850 2024-03-28T22:04:01.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:01 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-28T22:04:01.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:01 smithi016 ceph-mon[32374]: osdmap e21: 3 total, 3 up, 3 in 2024-03-28T22:04:01.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:01 smithi016 ceph-mon[32374]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:04:01.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:01 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:02.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:01 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-28T22:04:02.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:01 smithi063 ceph-mon[36249]: osdmap e21: 3 total, 3 up, 3 in 2024-03-28T22:04:02.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:01 smithi063 ceph-mon[36249]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:04:02.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:01 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:02.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:01 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-28T22:04:02.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:01 smithi053 ceph-mon[35979]: osdmap e21: 3 total, 3 up, 3 in 2024-03-28T22:04:02.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:01 smithi053 ceph-mon[35979]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:04:02.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:01 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:02.298 INFO:journalctl@ceph.osd.1.smithi053.stdout:Mar 28 22:04:01 smithi053 sudo[44347]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T22:04:02.298 INFO:journalctl@ceph.osd.1.smithi053.stdout:Mar 28 22:04:01 smithi053 sudo[44347]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T22:04:02.298 INFO:journalctl@ceph.osd.1.smithi053.stdout:Mar 28 22:04:01 smithi053 sudo[44347]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T22:04:02.298 INFO:journalctl@ceph.osd.1.smithi053.stdout:Mar 28 22:04:01 smithi053 sudo[44347]: pam_unix(sudo:session): session closed for user root 2024-03-28T22:04:02.797 INFO:journalctl@ceph.osd.1.smithi053.stdout:Mar 28 22:04:02 smithi053 sudo[44350]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T22:04:02.797 INFO:journalctl@ceph.osd.1.smithi053.stdout:Mar 28 22:04:02 smithi053 sudo[44350]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T22:04:02.797 INFO:journalctl@ceph.osd.1.smithi053.stdout:Mar 28 22:04:02 smithi053 sudo[44350]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T22:04:02.798 INFO:journalctl@ceph.osd.1.smithi053.stdout:Mar 28 22:04:02 smithi053 sudo[44350]: pam_unix(sudo:session): session closed for user root 2024-03-28T22:04:02.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:02 smithi016 ceph-mon[32374]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-28T22:04:02.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:02 smithi016 ceph-mon[32374]: osdmap e22: 3 total, 3 up, 3 in 2024-03-28T22:04:02.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:02 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2202967738' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c83cf4bb-395a-4786-abf7-74ca93a33eb3"}]: dispatch 2024-03-28T22:04:02.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:02 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2202967738' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c83cf4bb-395a-4786-abf7-74ca93a33eb3"}]': finished 2024-03-28T22:04:02.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:02 smithi016 ceph-mon[32374]: osdmap e23: 4 total, 3 up, 4 in 2024-03-28T22:04:02.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:02 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T22:04:02.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:02 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:03.182 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 28 22:04:02 smithi063 sudo[45470]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T22:04:03.182 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 28 22:04:02 smithi063 sudo[45470]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T22:04:03.182 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 28 22:04:02 smithi063 sudo[45470]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T22:04:03.182 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 28 22:04:02 smithi063 sudo[45470]: pam_unix(sudo:session): session closed for user root 2024-03-28T22:04:03.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:02 smithi063 ceph-mon[36249]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-28T22:04:03.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:02 smithi063 ceph-mon[36249]: osdmap e22: 3 total, 3 up, 3 in 2024-03-28T22:04:03.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:02 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2202967738' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c83cf4bb-395a-4786-abf7-74ca93a33eb3"}]: dispatch 2024-03-28T22:04:03.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:02 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2202967738' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c83cf4bb-395a-4786-abf7-74ca93a33eb3"}]': finished 2024-03-28T22:04:03.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:02 smithi063 ceph-mon[36249]: osdmap e23: 4 total, 3 up, 4 in 2024-03-28T22:04:03.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:02 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T22:04:03.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:02 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:03.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:02 smithi053 ceph-mon[35979]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-28T22:04:03.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:02 smithi053 ceph-mon[35979]: osdmap e22: 3 total, 3 up, 3 in 2024-03-28T22:04:03.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:02 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2202967738' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c83cf4bb-395a-4786-abf7-74ca93a33eb3"}]: dispatch 2024-03-28T22:04:03.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:02 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2202967738' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c83cf4bb-395a-4786-abf7-74ca93a33eb3"}]': finished 2024-03-28T22:04:03.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:02 smithi053 ceph-mon[35979]: osdmap e23: 4 total, 3 up, 4 in 2024-03-28T22:04:03.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:02 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T22:04:03.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:02 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:03.682 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 28 22:04:03 smithi063 sudo[45473]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T22:04:03.682 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 28 22:04:03 smithi063 sudo[45473]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T22:04:03.682 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 28 22:04:03 smithi063 sudo[45473]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T22:04:03.682 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 28 22:04:03 smithi063 sudo[45473]: pam_unix(sudo:session): session closed for user root 2024-03-28T22:04:03.870 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:03 smithi016 sudo[49613]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T22:04:03.871 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:03 smithi016 sudo[49613]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T22:04:03.871 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:03 smithi016 sudo[49613]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T22:04:03.871 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:03 smithi016 sudo[49613]: pam_unix(sudo:session): session closed for user root 2024-03-28T22:04:04.181 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:03 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/3302718861' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:04:04.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:03 smithi063 ceph-mon[36249]: pgmap v90: 1 pgs: 1 unknown; 0 B data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:04:04.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:03 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:04.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:03 smithi016 sudo[49800]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T22:04:04.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:03 smithi016 sudo[49800]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T22:04:04.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:03 smithi016 sudo[49800]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T22:04:04.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:03 smithi016 sudo[49800]: pam_unix(sudo:session): session closed for user root 2024-03-28T22:04:04.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:03 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3302718861' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:04:04.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:03 smithi016 ceph-mon[32374]: pgmap v90: 1 pgs: 1 unknown; 0 B data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:04:04.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:03 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:04.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:03 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3302718861' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:04:04.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:03 smithi053 ceph-mon[35979]: pgmap v90: 1 pgs: 1 unknown; 0 B data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:04:04.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:03 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:04.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:04 smithi063 sudo[45476]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T22:04:04.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:04 smithi063 sudo[45476]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T22:04:04.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:04 smithi063 sudo[45476]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T22:04:04.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:04 smithi063 sudo[45476]: pam_unix(sudo:session): session closed for user root 2024-03-28T22:04:05.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:04 smithi063 sudo[45479]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T22:04:05.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:04 smithi063 sudo[45479]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T22:04:05.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:04 smithi063 sudo[45479]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T22:04:05.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:04 smithi063 sudo[45479]: pam_unix(sudo:session): session closed for user root 2024-03-28T22:04:05.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:05 smithi053 sudo[44353]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T22:04:05.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:05 smithi053 sudo[44353]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T22:04:05.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:05 smithi053 sudo[44353]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T22:04:05.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:05 smithi053 sudo[44353]: pam_unix(sudo:session): session closed for user root 2024-03-28T22:04:05.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:05 smithi063 ceph-mon[36249]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T22:04:05.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:05 smithi063 ceph-mon[36249]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T22:04:05.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:05 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:04:05.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:05 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:04:05.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:05 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:04:05.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:05 smithi063 ceph-mon[36249]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T22:04:05.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:05 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:04:05.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:05 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:04:05.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:05 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:04:05.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:05 smithi063 ceph-mon[36249]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T22:04:05.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:05 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:04:05.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:05 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:04:05.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:05 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:04:05.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:05 smithi016 ceph-mon[32374]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T22:04:05.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:05 smithi016 ceph-mon[32374]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T22:04:05.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:05 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:04:05.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:05 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:04:05.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:05 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:04:05.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:05 smithi016 ceph-mon[32374]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T22:04:05.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:05 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:04:05.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:05 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:04:05.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:05 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:04:05.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:05 smithi016 ceph-mon[32374]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T22:04:05.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:05 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:04:05.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:05 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:04:05.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:05 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:04:06.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:05 smithi053 sudo[44356]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T22:04:06.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:05 smithi053 sudo[44356]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T22:04:06.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:05 smithi053 sudo[44356]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T22:04:06.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:05 smithi053 sudo[44356]: pam_unix(sudo:session): session closed for user root 2024-03-28T22:04:06.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:05 smithi053 ceph-mon[35979]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T22:04:06.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:05 smithi053 ceph-mon[35979]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T22:04:06.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:05 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:04:06.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:05 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:04:06.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:05 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:04:06.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:05 smithi053 ceph-mon[35979]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T22:04:06.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:05 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:04:06.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:05 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:04:06.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:05 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:04:06.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:05 smithi053 ceph-mon[35979]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T22:04:06.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:05 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:04:06.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:05 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:04:06.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:05 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:04:06.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:06 smithi063 ceph-mon[36249]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T22:04:06.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:06 smithi063 ceph-mon[36249]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:04:06.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:06 smithi063 ceph-mon[36249]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T22:04:06.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:06 smithi016 ceph-mon[32374]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T22:04:06.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:06 smithi016 ceph-mon[32374]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:04:06.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:06 smithi016 ceph-mon[32374]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T22:04:07.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:06 smithi053 ceph-mon[35979]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T22:04:07.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:06 smithi053 ceph-mon[35979]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:04:07.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:06 smithi053 ceph-mon[35979]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T22:04:07.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:07 smithi063 ceph-mon[36249]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:04:07.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:07 smithi016 ceph-mon[32374]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:04:08.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:07 smithi053 ceph-mon[35979]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:04:09.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:09 smithi016 ceph-mon[32374]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:04:09.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:09 smithi063 ceph-mon[36249]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:04:09.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:09 smithi053 ceph-mon[35979]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:04:10.406 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:10 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-28T22:04:10.406 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:10 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:04:10.406 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:10 smithi016 ceph-mon[32374]: Deploying daemon osd.3 on smithi016 2024-03-28T22:04:10.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:10 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-28T22:04:10.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:10 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:04:10.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:10 smithi063 ceph-mon[36249]: Deploying daemon osd.3 on smithi016 2024-03-28T22:04:10.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:10 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-28T22:04:10.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:10 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:04:10.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:10 smithi053 ceph-mon[35979]: Deploying daemon osd.3 on smithi016 2024-03-28T22:04:11.380 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:11 smithi016 ceph-mon[32374]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2024-03-28T22:04:11.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:11 smithi063 ceph-mon[36249]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2024-03-28T22:04:11.448 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:11 smithi053 ceph-mon[35979]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2024-03-28T22:04:13.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:13 smithi016 ceph-mon[32374]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2024-03-28T22:04:13.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:13 smithi063 ceph-mon[36249]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2024-03-28T22:04:13.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:13 smithi053 ceph-mon[35979]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2024-03-28T22:04:16.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:15 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:16.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:15 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:16.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:15 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:04:16.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:15 smithi063 ceph-mon[36249]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-28T22:04:16.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:15 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:16.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:15 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:16.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:15 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:04:16.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:15 smithi016 ceph-mon[32374]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-28T22:04:16.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:15 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:16.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:15 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:16.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:15 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:04:16.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:15 smithi053 ceph-mon[35979]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-28T22:04:17.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:17 smithi063 ceph-mon[36249]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-28T22:04:17.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:17 smithi016 ceph-mon[32374]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-28T22:04:17.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:17 smithi053 ceph-mon[35979]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-28T22:04:18.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:04:18.073+0000 7fdafd7fa700 1 -- 172.21.15.16:0/2140646576 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fdb00062680 con 0x7fdaec0a1d80 2024-03-28T22:04:18.077 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 3 on host 'smithi016' 2024-03-28T22:04:18.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:04:18.076+0000 7fdb06a2b700 1 -- 172.21.15.16:0/2140646576 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdaec0a1d80 msgr2=0x7fdaec0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:04:18.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:04:18.076+0000 7fdb06a2b700 1 --2- 172.21.15.16:0/2140646576 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdaec0a1d80 0x7fdaec0a4240 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdaf000b750 tx=0x7fdaf0005fd0 comp rx=0 tx=0).stop 2024-03-28T22:04:18.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:04:18.076+0000 7fdb06a2b700 1 -- 172.21.15.16:0/2140646576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb00103850 msgr2=0x7fdb0019faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:04:18.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:04:18.076+0000 7fdb06a2b700 1 --2- 172.21.15.16:0/2140646576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb00103850 0x7fdb0019faa0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdaf800c3a0 tx=0x7fdaf800c6b0 comp rx=0 tx=0).stop 2024-03-28T22:04:18.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:04:18.076+0000 7fdb06a2b700 1 -- 172.21.15.16:0/2140646576 shutdown_connections 2024-03-28T22:04:18.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:04:18.076+0000 7fdb06a2b700 1 --2- 172.21.15.16:0/2140646576 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdaec0a1d80 0x7fdaec0a4240 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:04:18.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:04:18.076+0000 7fdb06a2b700 1 --2- 172.21.15.16:0/2140646576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb00103850 0x7fdb0019faa0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:04:18.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:04:18.076+0000 7fdb06a2b700 1 --2- 172.21.15.16:0/2140646576 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb001002d0 0x7fdb001984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:04:18.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:04:18.076+0000 7fdb06a2b700 1 --2- 172.21.15.16:0/2140646576 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb00100c70 0x7fdb00198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:04:18.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:04:18.076+0000 7fdb06a2b700 1 -- 172.21.15.16:0/2140646576 >> 172.21.15.16:0/2140646576 conn(0x7fdb000fbeb0 msgr2=0x7fdb001068f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:04:18.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:04:18.076+0000 7fdb06a2b700 1 -- 172.21.15.16:0/2140646576 shutdown_connections 2024-03-28T22:04:18.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:04:18.076+0000 7fdb06a2b700 1 -- 172.21.15.16:0/2140646576 wait complete. 2024-03-28T22:04:18.699 DEBUG:teuthology.orchestra.run.smithi016:osd.3> sudo journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.3.service 2024-03-28T22:04:18.702 INFO:tasks.cephadm:Deploying osd.4 on smithi053 with /dev/vg_nvme/lv_3... 2024-03-28T22:04:18.702 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-28T22:04:18.936 INFO:journalctl@ceph.osd.3.smithi016.stdout:Mar 28 22:04:18 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-3[51407]: 2024-03-28T22:04:18.756+0000 7f1ad46837c0 -1 osd.3 0 log_to_monitors true 2024-03-28T22:04:18.959 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T22:04:19.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:19 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:19.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:19 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:19.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:19 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:19.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:19 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:19.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:19 smithi053 ceph-mon[35979]: from='osd.3 [v2:172.21.15.16:6810/13176789,v1:172.21.15.16:6811/13176789]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-28T22:04:19.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:19 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:19.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:19 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:19.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:19 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:19.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:19 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:19.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:19 smithi063 ceph-mon[36249]: from='osd.3 [v2:172.21.15.16:6810/13176789,v1:172.21.15.16:6811/13176789]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-28T22:04:19.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:19 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:19.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:19 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:19.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:19 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:19.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:19 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:19.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:19 smithi016 ceph-mon[32374]: from='osd.3 [v2:172.21.15.16:6810/13176789,v1:172.21.15.16:6811/13176789]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-28T22:04:20.239 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:20 smithi053 ceph-mon[35979]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-28T22:04:20.239 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:20 smithi053 ceph-mon[35979]: from='osd.3 [v2:172.21.15.16:6810/13176789,v1:172.21.15.16:6811/13176789]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-28T22:04:20.239 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:20 smithi053 ceph-mon[35979]: osdmap e24: 4 total, 3 up, 4 in 2024-03-28T22:04:20.239 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:20 smithi053 ceph-mon[35979]: from='osd.3 [v2:172.21.15.16:6810/13176789,v1:172.21.15.16:6811/13176789]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-03-28T22:04:20.239 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:20 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T22:04:20.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:20 smithi063 ceph-mon[36249]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-28T22:04:20.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:20 smithi063 ceph-mon[36249]: from='osd.3 [v2:172.21.15.16:6810/13176789,v1:172.21.15.16:6811/13176789]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-28T22:04:20.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:20 smithi063 ceph-mon[36249]: osdmap e24: 4 total, 3 up, 4 in 2024-03-28T22:04:20.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:20 smithi063 ceph-mon[36249]: from='osd.3 [v2:172.21.15.16:6810/13176789,v1:172.21.15.16:6811/13176789]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-03-28T22:04:20.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:20 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T22:04:20.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:20 smithi016 ceph-mon[32374]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-28T22:04:20.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:20 smithi016 ceph-mon[32374]: from='osd.3 [v2:172.21.15.16:6810/13176789,v1:172.21.15.16:6811/13176789]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-28T22:04:20.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:20 smithi016 ceph-mon[32374]: osdmap e24: 4 total, 3 up, 4 in 2024-03-28T22:04:20.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:20 smithi016 ceph-mon[32374]: from='osd.3 [v2:172.21.15.16:6810/13176789,v1:172.21.15.16:6811/13176789]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-03-28T22:04:20.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:20 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T22:04:20.437 INFO:journalctl@ceph.osd.3.smithi016.stdout:Mar 28 22:04:20 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-3[51407]: 2024-03-28T22:04:20.127+0000 7f1ac94e7700 -1 osd.3 0 waiting for initial osdmap 2024-03-28T22:04:20.437 INFO:journalctl@ceph.osd.3.smithi016.stdout:Mar 28 22:04:20 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-3[51407]: 2024-03-28T22:04:20.132+0000 7f1ac5adb700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T22:04:20.786 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-28T22:04:20.786 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-28T22:04:20.786 INFO:teuthology.orchestra.run.smithi053.stderr: stderr: 10+0 records in 2024-03-28T22:04:20.786 INFO:teuthology.orchestra.run.smithi053.stderr:10+0 records out 2024-03-28T22:04:20.786 INFO:teuthology.orchestra.run.smithi053.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00821013 s, 1.3 GB/s 2024-03-28T22:04:20.786 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping successful for: 2024-03-28T22:04:21.363 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi053:vg_nvme/lv_3 2024-03-28T22:04:21.394 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:21 smithi053 ceph-mon[35979]: from='osd.3 [v2:172.21.15.16:6810/13176789,v1:172.21.15.16:6811/13176789]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-03-28T22:04:21.394 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:21 smithi053 ceph-mon[35979]: osdmap e25: 4 total, 3 up, 4 in 2024-03-28T22:04:21.394 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:21 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T22:04:21.394 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:21 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T22:04:21.394 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:21 smithi053 ceph-mon[35979]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:04:21.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:21 smithi063 ceph-mon[36249]: from='osd.3 [v2:172.21.15.16:6810/13176789,v1:172.21.15.16:6811/13176789]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-03-28T22:04:21.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:21 smithi063 ceph-mon[36249]: osdmap e25: 4 total, 3 up, 4 in 2024-03-28T22:04:21.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:21 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T22:04:21.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:21 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T22:04:21.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:21 smithi063 ceph-mon[36249]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:04:21.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:21 smithi016 ceph-mon[32374]: from='osd.3 [v2:172.21.15.16:6810/13176789,v1:172.21.15.16:6811/13176789]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-03-28T22:04:21.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:21 smithi016 ceph-mon[32374]: osdmap e25: 4 total, 3 up, 4 in 2024-03-28T22:04:21.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:21 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T22:04:21.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:21 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T22:04:21.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:21 smithi016 ceph-mon[32374]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:04:21.621 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T22:04:22.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:22 smithi053 ceph-mon[35979]: purged_snaps scrub starts 2024-03-28T22:04:22.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:22 smithi053 ceph-mon[35979]: purged_snaps scrub ok 2024-03-28T22:04:22.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:22 smithi053 ceph-mon[35979]: osd.3 [v2:172.21.15.16:6810/13176789,v1:172.21.15.16:6811/13176789] boot 2024-03-28T22:04:22.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:22 smithi053 ceph-mon[35979]: osdmap e26: 4 total, 4 up, 4 in 2024-03-28T22:04:22.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:22 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T22:04:22.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:22 smithi053 ceph-mon[35979]: Detected new or changed devices on smithi016 2024-03-28T22:04:22.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:22 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:22.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:22 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:22.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:22 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:04:22.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:22 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:04:22.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:22 smithi053 ceph-mon[35979]: Adjusting osd_memory_target on smithi016 to 8546M 2024-03-28T22:04:22.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:22 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:22.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:22 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:04:22.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:22 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:04:22.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:22 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:22.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:22 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:04:22.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:22 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:04:22.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:22 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:04:22.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:22 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:22.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:22 smithi063 ceph-mon[36249]: purged_snaps scrub starts 2024-03-28T22:04:22.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:22 smithi063 ceph-mon[36249]: purged_snaps scrub ok 2024-03-28T22:04:22.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:22 smithi063 ceph-mon[36249]: osd.3 [v2:172.21.15.16:6810/13176789,v1:172.21.15.16:6811/13176789] boot 2024-03-28T22:04:22.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:22 smithi063 ceph-mon[36249]: osdmap e26: 4 total, 4 up, 4 in 2024-03-28T22:04:22.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:22 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T22:04:22.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:22 smithi063 ceph-mon[36249]: Detected new or changed devices on smithi016 2024-03-28T22:04:22.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:22 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:22.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:22 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:22.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:22 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:04:22.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:22 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:04:22.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:22 smithi063 ceph-mon[36249]: Adjusting osd_memory_target on smithi016 to 8546M 2024-03-28T22:04:22.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:22 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:22.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:22 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:04:22.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:22 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:04:22.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:22 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:22.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:22 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:04:22.434 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:22 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:04:22.434 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:22 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:04:22.434 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:22 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:22.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:22 smithi016 ceph-mon[32374]: purged_snaps scrub starts 2024-03-28T22:04:22.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:22 smithi016 ceph-mon[32374]: purged_snaps scrub ok 2024-03-28T22:04:22.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:22 smithi016 ceph-mon[32374]: osd.3 [v2:172.21.15.16:6810/13176789,v1:172.21.15.16:6811/13176789] boot 2024-03-28T22:04:22.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:22 smithi016 ceph-mon[32374]: osdmap e26: 4 total, 4 up, 4 in 2024-03-28T22:04:22.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:22 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T22:04:22.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:22 smithi016 ceph-mon[32374]: Detected new or changed devices on smithi016 2024-03-28T22:04:22.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:22 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:22.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:22 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:22.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:22 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:04:22.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:22 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:04:22.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:22 smithi016 ceph-mon[32374]: Adjusting osd_memory_target on smithi016 to 8546M 2024-03-28T22:04:22.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:22 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:22.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:22 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:04:22.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:22 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:04:22.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:22 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:22.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:22 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:04:22.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:22 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:04:22.438 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:22 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:04:22.438 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:22 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:23.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.112+0000 7f1777ffd700 1 -- 172.21.15.53:0/1432746339 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1770100ae0 msgr2=0x7f1770100f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:04:23.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.112+0000 7f1777ffd700 1 --2- 172.21.15.53:0/1432746339 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1770100ae0 0x7f1770100f00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f176000b0a0 tx=0x7f176000b3b0 comp rx=0 tx=0).stop 2024-03-28T22:04:23.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.112+0000 7f1777ffd700 1 -- 172.21.15.53:0/1432746339 shutdown_connections 2024-03-28T22:04:23.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.112+0000 7f1777ffd700 1 --2- 172.21.15.53:0/1432746339 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1770101e00 0x7f1770102280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:04:23.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.112+0000 7f1777ffd700 1 --2- 172.21.15.53:0/1432746339 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1770100ae0 0x7f1770100f00 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:04:23.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.112+0000 7f1777ffd700 1 --2- 172.21.15.53:0/1432746339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1770101440 0x7f17701018c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:04:23.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.112+0000 7f1777ffd700 1 -- 172.21.15.53:0/1432746339 >> 172.21.15.53:0/1432746339 conn(0x7f17700fc700 msgr2=0x7f17700feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:04:23.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.112+0000 7f1777ffd700 1 -- 172.21.15.53:0/1432746339 shutdown_connections 2024-03-28T22:04:23.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.112+0000 7f1777ffd700 1 -- 172.21.15.53:0/1432746339 wait complete. 2024-03-28T22:04:23.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.113+0000 7f1777ffd700 1 Processor -- start 2024-03-28T22:04:23.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.113+0000 7f1777ffd700 1 -- start start 2024-03-28T22:04:23.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.113+0000 7f1777ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1770100ae0 0x7f177019a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:04:23.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.113+0000 7f1777ffd700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1770101440 0x7f177019ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:04:23.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.113+0000 7f1777ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1770101e00 0x7f17701a1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:04:23.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.113+0000 7f1777ffd700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1770110090 con 0x7f1770100ae0 2024-03-28T22:04:23.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.113+0000 7f1777ffd700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f177010ff10 con 0x7f1770101e00 2024-03-28T22:04:23.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.113+0000 7f1777ffd700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1770110210 con 0x7f1770101440 2024-03-28T22:04:23.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.113+0000 7f177659a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1770101e00 0x7f17701a1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:04:23.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.113+0000 7f177659a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1770101e00 0x7f17701a1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:58004/0 (socket says 172.21.15.53:58004) 2024-03-28T22:04:23.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.113+0000 7f177659a700 1 -- 172.21.15.53:0/3420165479 learned_addr learned my addr 172.21.15.53:0/3420165479 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-03-28T22:04:23.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.113+0000 7f1775d99700 1 --2- 172.21.15.53:0/3420165479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1770100ae0 0x7f177019a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:04:23.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.113+0000 7f1775598700 1 --2- 172.21.15.53:0/3420165479 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1770101440 0x7f177019ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:04:23.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.113+0000 7f177659a700 1 -- 172.21.15.53:0/3420165479 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1770101440 msgr2=0x7f177019ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:04:23.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.113+0000 7f177659a700 1 --2- 172.21.15.53:0/3420165479 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1770101440 0x7f177019ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:04:23.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.113+0000 7f177659a700 1 -- 172.21.15.53:0/3420165479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1770100ae0 msgr2=0x7f177019a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:04:23.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.113+0000 7f177659a700 1 --2- 172.21.15.53:0/3420165479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1770100ae0 0x7f177019a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:04:23.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.113+0000 7f177659a700 1 -- 172.21.15.53:0/3420165479 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17701a2310 con 0x7f1770101e00 2024-03-28T22:04:23.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.113+0000 7f1775598700 1 --2- 172.21.15.53:0/3420165479 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1770101440 0x7f177019ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:04:23.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.114+0000 7f177659a700 1 --2- 172.21.15.53:0/3420165479 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1770101e00 0x7f17701a1c70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f176800b550 tx=0x7f176800b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:04:23.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.114+0000 7f176effd700 1 -- 172.21.15.53:0/3420165479 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1768015020 con 0x7f1770101e00 2024-03-28T22:04:23.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.114+0000 7f176effd700 1 -- 172.21.15.53:0/3420165479 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1768011d10 con 0x7f1770101e00 2024-03-28T22:04:23.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.114+0000 7f176effd700 1 -- 172.21.15.53:0/3420165479 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176800f2e0 con 0x7f1770101e00 2024-03-28T22:04:23.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.114+0000 7f1777ffd700 1 -- 172.21.15.53:0/3420165479 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17701a2600 con 0x7f1770101e00 2024-03-28T22:04:23.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.114+0000 7f1777ffd700 1 -- 172.21.15.53:0/3420165479 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17701a2af0 con 0x7f1770101e00 2024-03-28T22:04:23.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.115+0000 7f1777ffd700 1 -- 172.21.15.53:0/3420165479 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1770068f00 con 0x7f1770101e00 2024-03-28T22:04:23.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.116+0000 7f176effd700 1 -- 172.21.15.53:0/3420165479 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f1768018020 con 0x7f1770101e00 2024-03-28T22:04:23.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.117+0000 7f176effd700 1 --2- 172.21.15.53:0/3420165479 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f17580a1e90 0x7f17580a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:04:23.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.117+0000 7f176effd700 1 -- 172.21.15.53:0/3420165479 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 3698+0+0 (secure 0 0 0) 0x7f17680cc800 con 0x7f1770101e00 2024-03-28T22:04:23.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.117+0000 7f1775d99700 1 --2- 172.21.15.53:0/3420165479 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f17580a1e90 0x7f17580a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:04:23.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.118+0000 7f1775d99700 1 --2- 172.21.15.53:0/3420165479 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f17580a1e90 0x7f17580a4350 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1760007f00 tx=0x7f1760006230 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:04:23.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.120+0000 7f176effd700 1 -- 172.21.15.53:0/3420165479 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f176801f020 con 0x7f1770101e00 2024-03-28T22:04:23.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:23.334+0000 7f1777ffd700 1 -- 172.21.15.53:0/3420165479 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f1770062680 con 0x7f17580a1e90 2024-03-28T22:04:23.337 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:23 smithi053 ceph-mon[35979]: osdmap e27: 4 total, 4 up, 4 in 2024-03-28T22:04:23.337 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:23 smithi053 ceph-mon[35979]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:04:23.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:23 smithi063 ceph-mon[36249]: osdmap e27: 4 total, 4 up, 4 in 2024-03-28T22:04:23.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:23 smithi063 ceph-mon[36249]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:04:23.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:23 smithi016 ceph-mon[32374]: osdmap e27: 4 total, 4 up, 4 in 2024-03-28T22:04:23.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:23 smithi016 ceph-mon[32374]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:04:24.279 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:24 smithi053 ceph-mon[35979]: osdmap e28: 4 total, 4 up, 4 in 2024-03-28T22:04:24.279 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:24 smithi053 ceph-mon[35979]: from='client.24239 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:04:24.279 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:04:24.279 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:04:24.279 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:04:24.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:24 smithi063 ceph-mon[36249]: osdmap e28: 4 total, 4 up, 4 in 2024-03-28T22:04:24.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:24 smithi063 ceph-mon[36249]: from='client.24239 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:04:24.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:24 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:04:24.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:24 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:04:24.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:24 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:04:24.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:24 smithi016 ceph-mon[32374]: osdmap e28: 4 total, 4 up, 4 in 2024-03-28T22:04:24.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:24 smithi016 ceph-mon[32374]: from='client.24239 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:04:24.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:24 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:04:24.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:24 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:04:24.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:24 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:04:25.261 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:25 smithi053 ceph-mon[35979]: osdmap e29: 4 total, 4 up, 4 in 2024-03-28T22:04:25.262 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:25 smithi053 ceph-mon[35979]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T22:04:25.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:25 smithi063 ceph-mon[36249]: osdmap e29: 4 total, 4 up, 4 in 2024-03-28T22:04:25.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:25 smithi063 ceph-mon[36249]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T22:04:25.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:25 smithi016 ceph-mon[32374]: osdmap e29: 4 total, 4 up, 4 in 2024-03-28T22:04:25.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:25 smithi016 ceph-mon[32374]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T22:04:26.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:26 smithi063 ceph-mon[36249]: osdmap e30: 4 total, 4 up, 4 in 2024-03-28T22:04:26.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:26 smithi063 ceph-mon[36249]: from='client.? 172.21.15.53:0/3700864277' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "941b3e54-47d6-4e85-8069-e730c3fa8eec"}]: dispatch 2024-03-28T22:04:26.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:26 smithi063 ceph-mon[36249]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "941b3e54-47d6-4e85-8069-e730c3fa8eec"}]: dispatch 2024-03-28T22:04:26.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:26 smithi063 ceph-mon[36249]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "941b3e54-47d6-4e85-8069-e730c3fa8eec"}]': finished 2024-03-28T22:04:26.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:26 smithi063 ceph-mon[36249]: osdmap e31: 5 total, 4 up, 5 in 2024-03-28T22:04:26.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:26 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:04:26.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:26 smithi016 ceph-mon[32374]: osdmap e30: 4 total, 4 up, 4 in 2024-03-28T22:04:26.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:26 smithi016 ceph-mon[32374]: from='client.? 172.21.15.53:0/3700864277' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "941b3e54-47d6-4e85-8069-e730c3fa8eec"}]: dispatch 2024-03-28T22:04:26.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:26 smithi016 ceph-mon[32374]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "941b3e54-47d6-4e85-8069-e730c3fa8eec"}]: dispatch 2024-03-28T22:04:26.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:26 smithi016 ceph-mon[32374]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "941b3e54-47d6-4e85-8069-e730c3fa8eec"}]': finished 2024-03-28T22:04:26.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:26 smithi016 ceph-mon[32374]: osdmap e31: 5 total, 4 up, 5 in 2024-03-28T22:04:26.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:26 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:04:26.507 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:26 smithi053 ceph-mon[35979]: osdmap e30: 4 total, 4 up, 4 in 2024-03-28T22:04:26.507 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:26 smithi053 ceph-mon[35979]: from='client.? 172.21.15.53:0/3700864277' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "941b3e54-47d6-4e85-8069-e730c3fa8eec"}]: dispatch 2024-03-28T22:04:26.507 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:26 smithi053 ceph-mon[35979]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "941b3e54-47d6-4e85-8069-e730c3fa8eec"}]: dispatch 2024-03-28T22:04:26.508 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:26 smithi053 ceph-mon[35979]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "941b3e54-47d6-4e85-8069-e730c3fa8eec"}]': finished 2024-03-28T22:04:26.508 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:26 smithi053 ceph-mon[35979]: osdmap e31: 5 total, 4 up, 5 in 2024-03-28T22:04:26.508 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:26 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:04:27.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:27 smithi063 ceph-mon[36249]: from='client.? 172.21.15.53:0/1900488526' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:04:27.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:27 smithi063 ceph-mon[36249]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T22:04:27.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:27 smithi016 ceph-mon[32374]: from='client.? 172.21.15.53:0/1900488526' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:04:27.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:27 smithi016 ceph-mon[32374]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T22:04:27.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:27 smithi053 ceph-mon[35979]: from='client.? 172.21.15.53:0/1900488526' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:04:27.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:27 smithi053 ceph-mon[35979]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T22:04:29.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:29 smithi063 ceph-mon[36249]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-28T22:04:29.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:29 smithi016 ceph-mon[32374]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-28T22:04:29.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:29 smithi053 ceph-mon[35979]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-28T22:04:31.383 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:31 smithi053 ceph-mon[35979]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-03-28T22:04:31.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:31 smithi063 ceph-mon[36249]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-03-28T22:04:31.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:31 smithi016 ceph-mon[32374]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-03-28T22:04:33.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:33 smithi063 ceph-mon[36249]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:04:33.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:33 smithi016 ceph-mon[32374]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:04:33.495 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:33 smithi053 ceph-mon[35979]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:04:34.404 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:34 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-28T22:04:34.404 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:34 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:04:34.404 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:34 smithi053 ceph-mon[35979]: Deploying daemon osd.4 on smithi053 2024-03-28T22:04:34.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:34 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-28T22:04:34.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:34 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:04:34.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:34 smithi063 ceph-mon[36249]: Deploying daemon osd.4 on smithi053 2024-03-28T22:04:34.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:34 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-28T22:04:34.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:34 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:04:34.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:34 smithi016 ceph-mon[32374]: Deploying daemon osd.4 on smithi053 2024-03-28T22:04:35.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:35 smithi063 ceph-mon[36249]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:04:35.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:35 smithi016 ceph-mon[32374]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:04:35.493 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:35 smithi053 ceph-mon[35979]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:04:37.419 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:37 smithi053 ceph-mon[35979]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:04:37.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:37 smithi063 ceph-mon[36249]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:04:37.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:37 smithi016 ceph-mon[32374]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:04:39.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:39 smithi053 ceph-mon[35979]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:04:39.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:39 smithi063 ceph-mon[36249]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:04:39.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:39 smithi016 ceph-mon[32374]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:04:40.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:40 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:40.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:40 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:40.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:40 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:04:40.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:40 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:40.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:40 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:40.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:40 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:04:40.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:40 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:40.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:40 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:40.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:40 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:04:41.284 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:41 smithi053 ceph-mon[35979]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:04:41.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:41 smithi063 ceph-mon[36249]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:04:41.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:41 smithi016 ceph-mon[32374]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:04:42.603 INFO:teuthology.orchestra.run.smithi053.stdout:Created osd(s) 4 on host 'smithi053' 2024-03-28T22:04:42.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:42.600+0000 7f176effd700 1 -- 172.21.15.53:0/3420165479 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1770062680 con 0x7f17580a1e90 2024-03-28T22:04:42.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:42.603+0000 7f1777ffd700 1 -- 172.21.15.53:0/3420165479 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f17580a1e90 msgr2=0x7f17580a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:04:42.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:42.603+0000 7f1777ffd700 1 --2- 172.21.15.53:0/3420165479 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f17580a1e90 0x7f17580a4350 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1760007f00 tx=0x7f1760006230 comp rx=0 tx=0).stop 2024-03-28T22:04:42.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:42.603+0000 7f1777ffd700 1 -- 172.21.15.53:0/3420165479 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1770101e00 msgr2=0x7f17701a1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:04:42.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:42.603+0000 7f1777ffd700 1 --2- 172.21.15.53:0/3420165479 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1770101e00 0x7f17701a1c70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f176800b550 tx=0x7f176800b910 comp rx=0 tx=0).stop 2024-03-28T22:04:42.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:42.603+0000 7f1777ffd700 1 -- 172.21.15.53:0/3420165479 shutdown_connections 2024-03-28T22:04:42.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:42.603+0000 7f1777ffd700 1 --2- 172.21.15.53:0/3420165479 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f17580a1e90 0x7f17580a4350 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:04:42.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:42.603+0000 7f1777ffd700 1 --2- 172.21.15.53:0/3420165479 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1770101e00 0x7f17701a1c70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:04:42.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:42.603+0000 7f1777ffd700 1 --2- 172.21.15.53:0/3420165479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1770100ae0 0x7f177019a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:04:42.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:42.603+0000 7f1777ffd700 1 --2- 172.21.15.53:0/3420165479 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1770101440 0x7f177019ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:04:42.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:42.603+0000 7f1777ffd700 1 -- 172.21.15.53:0/3420165479 >> 172.21.15.53:0/3420165479 conn(0x7f17700fc700 msgr2=0x7f17700fe200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:04:42.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:42.603+0000 7f1777ffd700 1 -- 172.21.15.53:0/3420165479 shutdown_connections 2024-03-28T22:04:42.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:04:42.603+0000 7f1777ffd700 1 -- 172.21.15.53:0/3420165479 wait complete. 2024-03-28T22:04:43.240 DEBUG:teuthology.orchestra.run.smithi053:osd.4> sudo journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.4.service 2024-03-28T22:04:43.282 INFO:tasks.cephadm:Deploying osd.5 on smithi063 with /dev/vg_nvme/lv_3... 2024-03-28T22:04:43.282 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-28T22:04:43.540 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T22:04:43.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:43 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:43.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:43 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:43.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:43 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:43.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:43 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:43.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:43 smithi053 ceph-mon[35979]: from='osd.4 [v2:172.21.15.53:6808/772161849,v1:172.21.15.53:6809/772161849]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T22:04:43.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:43 smithi053 ceph-mon[35979]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T22:04:43.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:43 smithi053 ceph-mon[35979]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:04:43.880 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:43 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:43.880 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:43 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:43.880 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:43 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:43.880 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:43 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:43.880 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:43 smithi063 ceph-mon[36249]: from='osd.4 [v2:172.21.15.53:6808/772161849,v1:172.21.15.53:6809/772161849]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T22:04:43.880 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:43 smithi063 ceph-mon[36249]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T22:04:43.880 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:43 smithi063 ceph-mon[36249]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:04:43.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:43 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:43.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:43 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:43.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:43 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:43.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:43 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:43.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:43 smithi016 ceph-mon[32374]: from='osd.4 [v2:172.21.15.53:6808/772161849,v1:172.21.15.53:6809/772161849]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T22:04:43.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:43 smithi016 ceph-mon[32374]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T22:04:43.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:43 smithi016 ceph-mon[32374]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:04:44.699 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:44 smithi063 ceph-mon[36249]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-28T22:04:44.700 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:44 smithi063 ceph-mon[36249]: from='osd.4 [v2:172.21.15.53:6808/772161849,v1:172.21.15.53:6809/772161849]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-03-28T22:04:44.700 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:44 smithi063 ceph-mon[36249]: osdmap e32: 5 total, 4 up, 5 in 2024-03-28T22:04:44.700 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:44 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:04:44.700 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:44 smithi063 ceph-mon[36249]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-03-28T22:04:44.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:44 smithi053 ceph-mon[35979]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-28T22:04:44.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:44 smithi053 ceph-mon[35979]: from='osd.4 [v2:172.21.15.53:6808/772161849,v1:172.21.15.53:6809/772161849]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-03-28T22:04:44.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:44 smithi053 ceph-mon[35979]: osdmap e32: 5 total, 4 up, 5 in 2024-03-28T22:04:44.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:44 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:04:44.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:44 smithi053 ceph-mon[35979]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-03-28T22:04:44.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:44 smithi016 ceph-mon[32374]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-28T22:04:44.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:44 smithi016 ceph-mon[32374]: from='osd.4 [v2:172.21.15.53:6808/772161849,v1:172.21.15.53:6809/772161849]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-03-28T22:04:44.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:44 smithi016 ceph-mon[32374]: osdmap e32: 5 total, 4 up, 5 in 2024-03-28T22:04:44.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:44 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:04:44.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:44 smithi016 ceph-mon[32374]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-03-28T22:04:45.353 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-28T22:04:45.353 INFO:teuthology.orchestra.run.smithi063.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-28T22:04:45.353 INFO:teuthology.orchestra.run.smithi063.stderr: stderr: 10+0 records in 2024-03-28T22:04:45.353 INFO:teuthology.orchestra.run.smithi063.stderr:10+0 records out 2024-03-28T22:04:45.353 INFO:teuthology.orchestra.run.smithi063.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111491 s, 941 MB/s 2024-03-28T22:04:45.354 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping successful for: 2024-03-28T22:04:45.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:45 smithi053 ceph-mon[35979]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-03-28T22:04:45.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:45 smithi053 ceph-mon[35979]: osdmap e33: 5 total, 4 up, 5 in 2024-03-28T22:04:45.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:45 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:04:45.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:45 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:04:45.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:45 smithi053 ceph-mon[35979]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:04:45.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:45 smithi053 ceph-mon[35979]: from='osd.4 ' entity='osd.4' 2024-03-28T22:04:45.798 INFO:journalctl@ceph.osd.4.smithi053.stdout:Mar 28 22:04:45 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-4[47197]: 2024-03-28T22:04:45.470+0000 7ff1f06e1700 -1 osd.4 0 waiting for initial osdmap 2024-03-28T22:04:45.798 INFO:journalctl@ceph.osd.4.smithi053.stdout:Mar 28 22:04:45 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-4[47197]: 2024-03-28T22:04:45.477+0000 7ff1e74ca700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T22:04:45.922 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:45 smithi063 ceph-mon[36249]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-03-28T22:04:45.923 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:45 smithi063 ceph-mon[36249]: osdmap e33: 5 total, 4 up, 5 in 2024-03-28T22:04:45.923 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:45 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:04:45.923 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:45 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:04:45.923 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:45 smithi063 ceph-mon[36249]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:04:45.923 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:45 smithi063 ceph-mon[36249]: from='osd.4 ' entity='osd.4' 2024-03-28T22:04:45.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:45 smithi016 ceph-mon[32374]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-03-28T22:04:45.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:45 smithi016 ceph-mon[32374]: osdmap e33: 5 total, 4 up, 5 in 2024-03-28T22:04:45.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:45 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:04:45.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:45 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:04:45.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:45 smithi016 ceph-mon[32374]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:04:45.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:45 smithi016 ceph-mon[32374]: from='osd.4 ' entity='osd.4' 2024-03-28T22:04:45.951 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi063:vg_nvme/lv_3 2024-03-28T22:04:46.207 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T22:04:46.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:46 smithi063 ceph-mon[36249]: purged_snaps scrub starts 2024-03-28T22:04:46.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:46 smithi063 ceph-mon[36249]: purged_snaps scrub ok 2024-03-28T22:04:46.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:46 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:04:46.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:46 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:46.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:46 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:46.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:46 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:04:46.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:46 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:04:46.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:46 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:46.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:46 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:04:46.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:46 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:04:46.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:46 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:46.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:46 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:04:46.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:46 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:04:46.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:46 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:04:46.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:46 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:46.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:46 smithi063 ceph-mon[36249]: osd.4 [v2:172.21.15.53:6808/772161849,v1:172.21.15.53:6809/772161849] boot 2024-03-28T22:04:46.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:46 smithi063 ceph-mon[36249]: osdmap e34: 5 total, 5 up, 5 in 2024-03-28T22:04:46.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:46 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:04:46.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:46 smithi016 ceph-mon[32374]: purged_snaps scrub starts 2024-03-28T22:04:46.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:46 smithi016 ceph-mon[32374]: purged_snaps scrub ok 2024-03-28T22:04:46.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:46 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:04:46.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:46 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:46.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:46 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:46.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:46 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:04:46.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:46 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:04:46.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:46 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:46.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:46 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:04:46.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:46 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:04:46.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:46 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:46.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:46 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:04:46.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:46 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:04:46.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:46 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:04:46.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:46 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:46.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:46 smithi016 ceph-mon[32374]: osd.4 [v2:172.21.15.53:6808/772161849,v1:172.21.15.53:6809/772161849] boot 2024-03-28T22:04:46.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:46 smithi016 ceph-mon[32374]: osdmap e34: 5 total, 5 up, 5 in 2024-03-28T22:04:46.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:46 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:04:47.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:46 smithi053 ceph-mon[35979]: purged_snaps scrub starts 2024-03-28T22:04:47.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:46 smithi053 ceph-mon[35979]: purged_snaps scrub ok 2024-03-28T22:04:47.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:46 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:04:47.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:46 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:47.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:46 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:47.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:46 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:04:47.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:46 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:04:47.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:46 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:47.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:46 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:04:47.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:46 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:04:47.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:46 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:47.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:46 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:04:47.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:46 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:04:47.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:46 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:04:47.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:46 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:04:47.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:46 smithi053 ceph-mon[35979]: osd.4 [v2:172.21.15.53:6808/772161849,v1:172.21.15.53:6809/772161849] boot 2024-03-28T22:04:47.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:46 smithi053 ceph-mon[35979]: osdmap e34: 5 total, 5 up, 5 in 2024-03-28T22:04:47.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:46 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:04:47.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.740+0000 7f0c66ea8700 1 -- 172.21.15.63:0/3416900396 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0c601023a0 msgr2=0x7f0c60102800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:04:47.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.740+0000 7f0c66ea8700 1 --2- 172.21.15.63:0/3416900396 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0c601023a0 0x7f0c60102800 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c009990 tx=0x7f0c4c009ca0 comp rx=0 tx=0).stop 2024-03-28T22:04:47.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.740+0000 7f0c66ea8700 1 -- 172.21.15.63:0/3416900396 shutdown_connections 2024-03-28T22:04:47.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.740+0000 7f0c66ea8700 1 --2- 172.21.15.63:0/3416900396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c60102d40 0x7f0c601095f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:04:47.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.740+0000 7f0c66ea8700 1 --2- 172.21.15.63:0/3416900396 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c601011a0 0x7f0c601015c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:04:47.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.740+0000 7f0c66ea8700 1 --2- 172.21.15.63:0/3416900396 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0c601023a0 0x7f0c60102800 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:04:47.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.740+0000 7f0c66ea8700 1 -- 172.21.15.63:0/3416900396 >> 172.21.15.63:0/3416900396 conn(0x7f0c600fc760 msgr2=0x7f0c600feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:04:47.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.740+0000 7f0c66ea8700 1 -- 172.21.15.63:0/3416900396 shutdown_connections 2024-03-28T22:04:47.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.740+0000 7f0c66ea8700 1 -- 172.21.15.63:0/3416900396 wait complete. 2024-03-28T22:04:47.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.740+0000 7f0c66ea8700 1 Processor -- start 2024-03-28T22:04:47.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.741+0000 7f0c66ea8700 1 -- start start 2024-03-28T22:04:47.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.741+0000 7f0c66ea8700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0c601011a0 0x7f0c601984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:04:47.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.741+0000 7f0c66ea8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c601023a0 0x7f0c60198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:04:47.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.741+0000 7f0c66ea8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c60102d40 0x7f0c6019fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:04:47.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.741+0000 7f0c66ea8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0c6010cd70 con 0x7f0c601023a0 2024-03-28T22:04:47.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.741+0000 7f0c66ea8700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0c6010cbf0 con 0x7f0c60102d40 2024-03-28T22:04:47.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.741+0000 7f0c66ea8700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0c6010cef0 con 0x7f0c601011a0 2024-03-28T22:04:47.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.741+0000 7f0c64c44700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0c601011a0 0x7f0c601984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:04:47.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.741+0000 7f0c64c44700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0c601011a0 0x7f0c601984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42612/0 (socket says 172.21.15.63:42612) 2024-03-28T22:04:47.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.741+0000 7f0c64c44700 1 -- 172.21.15.63:0/2778493352 learned_addr learned my addr 172.21.15.63:0/2778493352 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-28T22:04:47.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.741+0000 7f0c65445700 1 --2- 172.21.15.63:0/2778493352 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c60102d40 0x7f0c6019fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:04:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.741+0000 7f0c5ffff700 1 --2- 172.21.15.63:0/2778493352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c601023a0 0x7f0c60198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:04:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.741+0000 7f0c64c44700 1 -- 172.21.15.63:0/2778493352 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c60102d40 msgr2=0x7f0c6019fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:04:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.741+0000 7f0c64c44700 1 --2- 172.21.15.63:0/2778493352 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c60102d40 0x7f0c6019fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:04:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.741+0000 7f0c64c44700 1 -- 172.21.15.63:0/2778493352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c601023a0 msgr2=0x7f0c60198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:04:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.741+0000 7f0c64c44700 1 --2- 172.21.15.63:0/2778493352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c601023a0 0x7f0c60198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:04:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.741+0000 7f0c64c44700 1 -- 172.21.15.63:0/2778493352 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c601a0130 con 0x7f0c601011a0 2024-03-28T22:04:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.741+0000 7f0c64c44700 1 --2- 172.21.15.63:0/2778493352 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0c601011a0 0x7f0c601984d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0c5400d9b0 tx=0x7f0c5400dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:04:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.741+0000 7f0c5dffb700 1 -- 172.21.15.63:0/2778493352 <== mon.1 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c54017030 con 0x7f0c601011a0 2024-03-28T22:04:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.742+0000 7f0c5dffb700 1 -- 172.21.15.63:0/2778493352 <== mon.1 v2:172.21.15.63:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c54004500 con 0x7f0c601011a0 2024-03-28T22:04:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.742+0000 7f0c66ea8700 1 -- 172.21.15.63:0/2778493352 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c601a0420 con 0x7f0c601011a0 2024-03-28T22:04:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.742+0000 7f0c5dffb700 1 -- 172.21.15.63:0/2778493352 <== mon.1 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c54003aa0 con 0x7f0c601011a0 2024-03-28T22:04:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.742+0000 7f0c66ea8700 1 -- 172.21.15.63:0/2778493352 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c601a09b0 con 0x7f0c601011a0 2024-03-28T22:04:47.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.743+0000 7f0c66ea8700 1 -- 172.21.15.63:0/2778493352 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c60068f00 con 0x7f0c601011a0 2024-03-28T22:04:47.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.744+0000 7f0c5dffb700 1 -- 172.21.15.63:0/2778493352 <== mon.1 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f0c5401a070 con 0x7f0c601011a0 2024-03-28T22:04:47.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.745+0000 7f0c5dffb700 1 --2- 172.21.15.63:0/2778493352 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0c500a1e90 0x7f0c500a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:04:47.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.745+0000 7f0c5dffb700 1 -- 172.21.15.63:0/2778493352 <== mon.1 v2:172.21.15.63:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 4101+0+0 (secure 0 0 0) 0x7f0c540cc5f0 con 0x7f0c601011a0 2024-03-28T22:04:47.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.745+0000 7f0c5ffff700 1 --2- 172.21.15.63:0/2778493352 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0c500a1e90 0x7f0c500a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:04:47.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.745+0000 7f0c5ffff700 1 --2- 172.21.15.63:0/2778493352 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0c500a1e90 0x7f0c500a4350 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c00b5c0 tx=0x7f0c4c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:04:47.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.750+0000 7f0c5dffb700 1 -- 172.21.15.63:0/2778493352 <== mon.1 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0c5409aa90 con 0x7f0c601011a0 2024-03-28T22:04:47.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:47 smithi063 ceph-mon[36249]: Detected new or changed devices on smithi053 2024-03-28T22:04:47.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:47 smithi063 ceph-mon[36249]: Adjusting osd_memory_target on smithi053 to 8542M 2024-03-28T22:04:47.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:47 smithi063 ceph-mon[36249]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:04:47.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:47 smithi016 ceph-mon[32374]: Detected new or changed devices on smithi053 2024-03-28T22:04:47.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:47 smithi016 ceph-mon[32374]: Adjusting osd_memory_target on smithi053 to 8542M 2024-03-28T22:04:47.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:47 smithi016 ceph-mon[32374]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:04:47.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:04:47.962+0000 7f0c66ea8700 1 -- 172.21.15.63:0/2778493352 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f0c600fd4a0 con 0x7f0c500a1e90 2024-03-28T22:04:48.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:47 smithi053 ceph-mon[35979]: Detected new or changed devices on smithi053 2024-03-28T22:04:48.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:47 smithi053 ceph-mon[35979]: Adjusting osd_memory_target on smithi053 to 8542M 2024-03-28T22:04:48.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:47 smithi053 ceph-mon[35979]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:04:48.901 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:48 smithi063 ceph-mon[36249]: osdmap e35: 5 total, 5 up, 5 in 2024-03-28T22:04:48.901 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:48 smithi063 ceph-mon[36249]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:04:48.901 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:48 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:04:48.901 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:48 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:04:48.901 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:48 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:04:48.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:48 smithi016 ceph-mon[32374]: osdmap e35: 5 total, 5 up, 5 in 2024-03-28T22:04:48.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:48 smithi016 ceph-mon[32374]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:04:48.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:48 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:04:48.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:48 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:04:48.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:48 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:04:49.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:48 smithi053 ceph-mon[35979]: osdmap e35: 5 total, 5 up, 5 in 2024-03-28T22:04:49.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:48 smithi053 ceph-mon[35979]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:04:49.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:48 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:04:49.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:48 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:04:49.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:48 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:04:49.791 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:49 smithi063 ceph-mon[36249]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:04:49.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:49 smithi016 ceph-mon[32374]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:04:50.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:49 smithi053 ceph-mon[35979]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:04:50.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:50 smithi063 ceph-mon[36249]: from='client.? 172.21.15.63:0/1195218654' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51d89826-1d60-40fc-8fd8-09bfe1ab4125"}]: dispatch 2024-03-28T22:04:50.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:50 smithi063 ceph-mon[36249]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51d89826-1d60-40fc-8fd8-09bfe1ab4125"}]: dispatch 2024-03-28T22:04:50.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:50 smithi063 ceph-mon[36249]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "51d89826-1d60-40fc-8fd8-09bfe1ab4125"}]': finished 2024-03-28T22:04:50.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:50 smithi063 ceph-mon[36249]: osdmap e36: 6 total, 5 up, 6 in 2024-03-28T22:04:50.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:50 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:04:50.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:50 smithi016 ceph-mon[32374]: from='client.? 172.21.15.63:0/1195218654' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51d89826-1d60-40fc-8fd8-09bfe1ab4125"}]: dispatch 2024-03-28T22:04:50.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:50 smithi016 ceph-mon[32374]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51d89826-1d60-40fc-8fd8-09bfe1ab4125"}]: dispatch 2024-03-28T22:04:50.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:50 smithi016 ceph-mon[32374]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "51d89826-1d60-40fc-8fd8-09bfe1ab4125"}]': finished 2024-03-28T22:04:50.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:50 smithi016 ceph-mon[32374]: osdmap e36: 6 total, 5 up, 6 in 2024-03-28T22:04:50.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:50 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:04:51.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:50 smithi053 ceph-mon[35979]: from='client.? 172.21.15.63:0/1195218654' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51d89826-1d60-40fc-8fd8-09bfe1ab4125"}]: dispatch 2024-03-28T22:04:51.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:50 smithi053 ceph-mon[35979]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51d89826-1d60-40fc-8fd8-09bfe1ab4125"}]: dispatch 2024-03-28T22:04:51.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:50 smithi053 ceph-mon[35979]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "51d89826-1d60-40fc-8fd8-09bfe1ab4125"}]': finished 2024-03-28T22:04:51.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:50 smithi053 ceph-mon[35979]: osdmap e36: 6 total, 5 up, 6 in 2024-03-28T22:04:51.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:50 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:04:52.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:51 smithi053 ceph-mon[35979]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:04:52.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:51 smithi053 ceph-mon[35979]: from='client.? 172.21.15.63:0/2032242732' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:04:52.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:51 smithi063 ceph-mon[36249]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:04:52.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:51 smithi063 ceph-mon[36249]: from='client.? 172.21.15.63:0/2032242732' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:04:52.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:51 smithi016 ceph-mon[32374]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:04:52.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:51 smithi016 ceph-mon[32374]: from='client.? 172.21.15.63:0/2032242732' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:04:53.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:53 smithi063 ceph-mon[36249]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:04:53.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:53 smithi016 ceph-mon[32374]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:04:53.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:53 smithi053 ceph-mon[35979]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:04:55.162 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:55 smithi063 ceph-mon[36249]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:04:55.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:55 smithi016 ceph-mon[32374]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:04:55.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:55 smithi053 ceph-mon[35979]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:04:57.331 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:57 smithi063 ceph-mon[36249]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:04:57.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:57 smithi016 ceph-mon[32374]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:04:57.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:57 smithi053 ceph-mon[35979]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:04:58.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:58 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-28T22:04:58.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:58 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:04:58.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:58 smithi063 ceph-mon[36249]: Deploying daemon osd.5 on smithi063 2024-03-28T22:04:58.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:58 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-28T22:04:58.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:58 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:04:58.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:58 smithi016 ceph-mon[32374]: Deploying daemon osd.5 on smithi063 2024-03-28T22:04:58.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:58 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-28T22:04:58.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:58 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:04:58.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:58 smithi053 ceph-mon[35979]: Deploying daemon osd.5 on smithi063 2024-03-28T22:04:59.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:04:59 smithi063 ceph-mon[36249]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:04:59.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:04:59 smithi016 ceph-mon[32374]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:04:59.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:04:59 smithi053 ceph-mon[35979]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:05:01.434 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:01 smithi063 ceph-mon[36249]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:05:01.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:01 smithi016 ceph-mon[32374]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:05:01.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:01 smithi053 ceph-mon[35979]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:05:03.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:03 smithi063 ceph-mon[36249]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:05:03.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:03 smithi016 ceph-mon[32374]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:05:03.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:03 smithi053 ceph-mon[35979]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:05:04.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:04 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:04.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:04 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:04.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:04 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:05:04.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:04 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:04.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:04 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:04.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:04 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:05:04.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:04 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:04.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:04 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:04.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:04 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:05:05.669 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:05 smithi063 ceph-mon[36249]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:05:05.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:05 smithi016 ceph-mon[32374]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:05:05.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:05 smithi053 ceph-mon[35979]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:05:06.731 INFO:teuthology.orchestra.run.smithi063.stdout:Created osd(s) 5 on host 'smithi063' 2024-03-28T22:05:06.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:05:06.729+0000 7f0c5dffb700 1 -- 172.21.15.63:0/2778493352 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0c600fd4a0 con 0x7f0c500a1e90 2024-03-28T22:05:06.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:05:06.730+0000 7f0c66ea8700 1 -- 172.21.15.63:0/2778493352 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0c500a1e90 msgr2=0x7f0c500a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:05:06.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:05:06.730+0000 7f0c66ea8700 1 --2- 172.21.15.63:0/2778493352 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0c500a1e90 0x7f0c500a4350 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c00b5c0 tx=0x7f0c4c005fb0 comp rx=0 tx=0).stop 2024-03-28T22:05:06.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:05:06.731+0000 7f0c66ea8700 1 -- 172.21.15.63:0/2778493352 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0c601011a0 msgr2=0x7f0c601984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:05:06.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:05:06.731+0000 7f0c66ea8700 1 --2- 172.21.15.63:0/2778493352 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0c601011a0 0x7f0c601984d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0c5400d9b0 tx=0x7f0c5400dcc0 comp rx=0 tx=0).stop 2024-03-28T22:05:06.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:05:06.731+0000 7f0c66ea8700 1 -- 172.21.15.63:0/2778493352 shutdown_connections 2024-03-28T22:05:06.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:05:06.731+0000 7f0c66ea8700 1 --2- 172.21.15.63:0/2778493352 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0c500a1e90 0x7f0c500a4350 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:05:06.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:05:06.731+0000 7f0c66ea8700 1 --2- 172.21.15.63:0/2778493352 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c60102d40 0x7f0c6019fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:05:06.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:05:06.731+0000 7f0c66ea8700 1 --2- 172.21.15.63:0/2778493352 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0c601011a0 0x7f0c601984d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:05:06.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:05:06.731+0000 7f0c66ea8700 1 --2- 172.21.15.63:0/2778493352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c601023a0 0x7f0c60198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:05:06.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:05:06.731+0000 7f0c66ea8700 1 -- 172.21.15.63:0/2778493352 >> 172.21.15.63:0/2778493352 conn(0x7f0c600fc760 msgr2=0x7f0c600fcd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:05:06.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:05:06.731+0000 7f0c66ea8700 1 -- 172.21.15.63:0/2778493352 shutdown_connections 2024-03-28T22:05:06.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:05:06.731+0000 7f0c66ea8700 1 -- 172.21.15.63:0/2778493352 wait complete. 2024-03-28T22:05:07.349 DEBUG:teuthology.orchestra.run.smithi063:osd.5> sudo journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.5.service 2024-03-28T22:05:07.352 INFO:tasks.cephadm:Deploying osd.6 on smithi016 with /dev/vg_nvme/lv_2... 2024-03-28T22:05:07.352 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-28T22:05:07.605 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:05:08.034 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:07 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:08.034 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:07 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:08.034 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:07 smithi016 ceph-mon[32374]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:05:08.034 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:07 smithi016 ceph-mon[32374]: from='osd.5 [v2:172.21.15.63:6808/1602554852,v1:172.21.15.63:6809/1602554852]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T22:05:08.034 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:07 smithi016 ceph-mon[32374]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T22:05:08.034 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:07 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:08.035 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:07 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:08.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:07 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:08.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:07 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:08.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:07 smithi053 ceph-mon[35979]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:05:08.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:07 smithi053 ceph-mon[35979]: from='osd.5 [v2:172.21.15.63:6808/1602554852,v1:172.21.15.63:6809/1602554852]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T22:05:08.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:07 smithi053 ceph-mon[35979]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T22:05:08.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:07 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:08.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:07 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:08.085 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:07 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:08.085 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:07 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:08.085 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:07 smithi063 ceph-mon[36249]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:05:08.085 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:07 smithi063 ceph-mon[36249]: from='osd.5 [v2:172.21.15.63:6808/1602554852,v1:172.21.15.63:6809/1602554852]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T22:05:08.085 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:07 smithi063 ceph-mon[36249]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T22:05:08.085 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:07 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:08.085 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:07 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:08.871 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:08 smithi016 ceph-mon[32374]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-28T22:05:08.871 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:08 smithi016 ceph-mon[32374]: osdmap e37: 6 total, 5 up, 6 in 2024-03-28T22:05:08.871 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:08 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:05:08.871 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:08 smithi016 ceph-mon[32374]: from='osd.5 [v2:172.21.15.63:6808/1602554852,v1:172.21.15.63:6809/1602554852]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-28T22:05:08.871 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:08 smithi016 ceph-mon[32374]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-28T22:05:09.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:08 smithi053 ceph-mon[35979]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-28T22:05:09.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:08 smithi053 ceph-mon[35979]: osdmap e37: 6 total, 5 up, 6 in 2024-03-28T22:05:09.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:08 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:05:09.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:08 smithi053 ceph-mon[35979]: from='osd.5 [v2:172.21.15.63:6808/1602554852,v1:172.21.15.63:6809/1602554852]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-28T22:05:09.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:08 smithi053 ceph-mon[35979]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-28T22:05:09.096 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:08 smithi063 ceph-mon[36249]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-28T22:05:09.096 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:08 smithi063 ceph-mon[36249]: osdmap e37: 6 total, 5 up, 6 in 2024-03-28T22:05:09.096 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:08 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:05:09.096 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:08 smithi063 ceph-mon[36249]: from='osd.5 [v2:172.21.15.63:6808/1602554852,v1:172.21.15.63:6809/1602554852]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-28T22:05:09.096 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:08 smithi063 ceph-mon[36249]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-28T22:05:09.097 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:05:08 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5[48377]: 2024-03-28T22:05:08.735+0000 7fc511dcb700 -1 osd.5 0 waiting for initial osdmap 2024-03-28T22:05:09.097 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:05:08 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5[48377]: 2024-03-28T22:05:08.744+0000 7fc50e3bf700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T22:05:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-28T22:05:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-28T22:05:09.713 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10+0 records in 2024-03-28T22:05:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:10+0 records out 2024-03-28T22:05:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0123622 s, 848 MB/s 2024-03-28T22:05:09.714 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 2024-03-28T22:05:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping successful for: 2024-03-28T22:05:09.991 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:09 smithi063 ceph-mon[36249]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-03-28T22:05:09.991 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:09 smithi063 ceph-mon[36249]: osdmap e38: 6 total, 5 up, 6 in 2024-03-28T22:05:09.991 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:09 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:05:09.991 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:09 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:05:09.991 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:09 smithi063 ceph-mon[36249]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:05:09.993 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:09 smithi016 ceph-mon[32374]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-03-28T22:05:09.993 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:09 smithi016 ceph-mon[32374]: osdmap e38: 6 total, 5 up, 6 in 2024-03-28T22:05:09.993 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:09 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:05:09.993 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:09 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:05:09.993 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:09 smithi016 ceph-mon[32374]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:05:10.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:09 smithi053 ceph-mon[35979]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-03-28T22:05:10.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:09 smithi053 ceph-mon[35979]: osdmap e38: 6 total, 5 up, 6 in 2024-03-28T22:05:10.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:09 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:05:10.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:09 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:05:10.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:09 smithi053 ceph-mon[35979]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:05:10.454 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi016:vg_nvme/lv_2 2024-03-28T22:05:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:05:10.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:10 smithi063 ceph-mon[36249]: purged_snaps scrub starts 2024-03-28T22:05:10.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:10 smithi063 ceph-mon[36249]: purged_snaps scrub ok 2024-03-28T22:05:10.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:10 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:05:10.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:10 smithi063 ceph-mon[36249]: osd.5 [v2:172.21.15.63:6808/1602554852,v1:172.21.15.63:6809/1602554852] boot 2024-03-28T22:05:10.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:10 smithi063 ceph-mon[36249]: osdmap e39: 6 total, 6 up, 6 in 2024-03-28T22:05:10.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:10 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:05:10.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:10 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:10.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:10 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:10.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:10 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:05:10.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:10 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:05:10.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:10 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:10.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:10 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:05:10.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:10 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:05:10.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:10 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:11.050 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:10 smithi053 ceph-mon[35979]: purged_snaps scrub starts 2024-03-28T22:05:11.050 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:10 smithi053 ceph-mon[35979]: purged_snaps scrub ok 2024-03-28T22:05:11.050 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:10 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:05:11.050 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:10 smithi053 ceph-mon[35979]: osd.5 [v2:172.21.15.63:6808/1602554852,v1:172.21.15.63:6809/1602554852] boot 2024-03-28T22:05:11.050 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:10 smithi053 ceph-mon[35979]: osdmap e39: 6 total, 6 up, 6 in 2024-03-28T22:05:11.050 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:10 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:05:11.050 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:10 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:11.050 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:10 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:11.050 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:10 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:05:11.050 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:10 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:05:11.050 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:10 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:11.051 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:10 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:05:11.051 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:10 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:05:11.051 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:10 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:11.080 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:10 smithi016 ceph-mon[32374]: purged_snaps scrub starts 2024-03-28T22:05:11.080 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:10 smithi016 ceph-mon[32374]: purged_snaps scrub ok 2024-03-28T22:05:11.080 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:10 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:05:11.080 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:10 smithi016 ceph-mon[32374]: osd.5 [v2:172.21.15.63:6808/1602554852,v1:172.21.15.63:6809/1602554852] boot 2024-03-28T22:05:11.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:10 smithi016 ceph-mon[32374]: osdmap e39: 6 total, 6 up, 6 in 2024-03-28T22:05:11.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:10 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:05:11.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:10 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:11.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:10 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:11.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:10 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:05:11.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:10 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:05:11.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:10 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:11.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:10 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:05:11.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:10 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:05:11.081 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:10 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:11.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:11 smithi016 ceph-mon[32374]: Detected new or changed devices on smithi063 2024-03-28T22:05:11.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:11 smithi016 ceph-mon[32374]: Adjusting osd_memory_target on smithi063 to 8540M 2024-03-28T22:05:11.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:11 smithi016 ceph-mon[32374]: osdmap e40: 6 total, 6 up, 6 in 2024-03-28T22:05:11.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:11 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:05:11.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:11 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:05:11.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:11 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:05:11.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:11 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:11.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:11 smithi016 ceph-mon[32374]: pgmap v141: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:05:12.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:11 smithi053 ceph-mon[35979]: Detected new or changed devices on smithi063 2024-03-28T22:05:12.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:11 smithi053 ceph-mon[35979]: Adjusting osd_memory_target on smithi063 to 8540M 2024-03-28T22:05:12.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:11 smithi053 ceph-mon[35979]: osdmap e40: 6 total, 6 up, 6 in 2024-03-28T22:05:12.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:11 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:05:12.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:11 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:05:12.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:11 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:05:12.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:11 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:12.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:11 smithi053 ceph-mon[35979]: pgmap v141: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:05:12.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:11 smithi063 ceph-mon[36249]: Detected new or changed devices on smithi063 2024-03-28T22:05:12.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:11 smithi063 ceph-mon[36249]: Adjusting osd_memory_target on smithi063 to 8540M 2024-03-28T22:05:12.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:11 smithi063 ceph-mon[36249]: osdmap e40: 6 total, 6 up, 6 in 2024-03-28T22:05:12.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:11 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:05:12.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:11 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:05:12.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:11 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:05:12.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:11 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:12.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:11 smithi063 ceph-mon[36249]: pgmap v141: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:05:12.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.241+0000 7fb3b8b93700 1 -- 172.21.15.16:0/1744080464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3b4103e00 msgr2=0x7fb3b410a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:05:12.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.241+0000 7fb3b8b93700 1 --2- 172.21.15.16:0/1744080464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3b4103e00 0x7fb3b410a6b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac00b0d0 tx=0x7fb3ac00b3e0 comp rx=0 tx=0).stop 2024-03-28T22:05:12.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.241+0000 7fb3b8b93700 1 -- 172.21.15.16:0/1744080464 shutdown_connections 2024-03-28T22:05:12.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.241+0000 7fb3b8b93700 1 --2- 172.21.15.16:0/1744080464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3b4103e00 0x7fb3b410a6b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:05:12.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.241+0000 7fb3b8b93700 1 --2- 172.21.15.16:0/1744080464 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3b4102260 0x7fb3b4102680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:05:12.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.241+0000 7fb3b8b93700 1 --2- 172.21.15.16:0/1744080464 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3b4103460 0x7fb3b41038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:05:12.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.241+0000 7fb3b8b93700 1 -- 172.21.15.16:0/1744080464 >> 172.21.15.16:0/1744080464 conn(0x7fb3b40fd820 msgr2=0x7fb3b40ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:05:12.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.241+0000 7fb3b8b93700 1 -- 172.21.15.16:0/1744080464 shutdown_connections 2024-03-28T22:05:12.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.241+0000 7fb3b8b93700 1 -- 172.21.15.16:0/1744080464 wait complete. 2024-03-28T22:05:12.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.241+0000 7fb3b8b93700 1 Processor -- start 2024-03-28T22:05:12.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.241+0000 7fb3b8b93700 1 -- start start 2024-03-28T22:05:12.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.242+0000 7fb3b8b93700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3b4102260 0x7fb3b4198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:05:12.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.242+0000 7fb3b8b93700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3b4103460 0x7fb3b41989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:05:12.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.242+0000 7fb3b8b93700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3b4103e00 0x7fb3b419fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:05:12.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.242+0000 7fb3b8b93700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb3b410dec0 con 0x7fb3b4102260 2024-03-28T22:05:12.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.242+0000 7fb3b8b93700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb3b410dd40 con 0x7fb3b4103460 2024-03-28T22:05:12.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.242+0000 7fb3b8b93700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb3b410e040 con 0x7fb3b4103e00 2024-03-28T22:05:12.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.242+0000 7fb3b259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3b4102260 0x7fb3b4198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:05:12.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.242+0000 7fb3b259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3b4102260 0x7fb3b4198480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49066/0 (socket says 172.21.15.16:49066) 2024-03-28T22:05:12.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.242+0000 7fb3b259c700 1 -- 172.21.15.16:0/3057098468 learned_addr learned my addr 172.21.15.16:0/3057098468 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:05:12.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.242+0000 7fb3b1d9b700 1 --2- 172.21.15.16:0/3057098468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3b4103460 0x7fb3b41989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:05:12.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.242+0000 7fb3b2d9d700 1 --2- 172.21.15.16:0/3057098468 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3b4103e00 0x7fb3b419fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:05:12.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.242+0000 7fb3b259c700 1 -- 172.21.15.16:0/3057098468 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3b4103e00 msgr2=0x7fb3b419fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:05:12.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.242+0000 7fb3b259c700 1 --2- 172.21.15.16:0/3057098468 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3b4103e00 0x7fb3b419fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:05:12.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.242+0000 7fb3b259c700 1 -- 172.21.15.16:0/3057098468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3b4103460 msgr2=0x7fb3b41989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:05:12.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.242+0000 7fb3b259c700 1 --2- 172.21.15.16:0/3057098468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3b4103460 0x7fb3b41989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:05:12.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.242+0000 7fb3b259c700 1 -- 172.21.15.16:0/3057098468 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3b41a00e0 con 0x7fb3b4102260 2024-03-28T22:05:12.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.242+0000 7fb3b259c700 1 --2- 172.21.15.16:0/3057098468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3b4102260 0x7fb3b4198480 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb3a400b530 tx=0x7fb3a400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:05:12.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.242+0000 7fb3a37fe700 1 -- 172.21.15.16:0/3057098468 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a4015020 con 0x7fb3b4102260 2024-03-28T22:05:12.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.242+0000 7fb3a37fe700 1 -- 172.21.15.16:0/3057098468 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3a4011d10 con 0x7fb3b4102260 2024-03-28T22:05:12.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.242+0000 7fb3b8b93700 1 -- 172.21.15.16:0/3057098468 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3b41a03d0 con 0x7fb3b4102260 2024-03-28T22:05:12.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.243+0000 7fb3a37fe700 1 -- 172.21.15.16:0/3057098468 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a4010d90 con 0x7fb3b4102260 2024-03-28T22:05:12.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.243+0000 7fb3b8b93700 1 -- 172.21.15.16:0/3057098468 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3b41a0990 con 0x7fb3b4102260 2024-03-28T22:05:12.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.243+0000 7fb3b8b93700 1 -- 172.21.15.16:0/3057098468 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3b4068f00 con 0x7fb3b4102260 2024-03-28T22:05:12.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.244+0000 7fb3a37fe700 1 -- 172.21.15.16:0/3057098468 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fb3a4018020 con 0x7fb3b4102260 2024-03-28T22:05:12.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.245+0000 7fb3a37fe700 1 --2- 172.21.15.16:0/3057098468 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb39c0a1e10 0x7fb39c0a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:05:12.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.245+0000 7fb3b1d9b700 1 --2- 172.21.15.16:0/3057098468 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb39c0a1e10 0x7fb39c0a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:05:12.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.245+0000 7fb3a37fe700 1 -- 172.21.15.16:0/3057098468 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4562+0+0 (secure 0 0 0) 0x7fb3a40cc5e0 con 0x7fb3b4102260 2024-03-28T22:05:12.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.246+0000 7fb3b1d9b700 1 --2- 172.21.15.16:0/3057098468 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb39c0a1e10 0x7fb39c0a42d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8005950 tx=0x7fb3a800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:05:12.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.250+0000 7fb3a37fe700 1 -- 172.21.15.16:0/3057098468 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb3a401f020 con 0x7fb3b4102260 2024-03-28T22:05:12.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:12.465+0000 7fb3b8b93700 1 -- 172.21.15.16:0/3057098468 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fb3b4062680 con 0x7fb39c0a1e10 2024-03-28T22:05:13.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:12 smithi053 ceph-mon[35979]: osdmap e41: 6 total, 6 up, 6 in 2024-03-28T22:05:13.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:12 smithi053 ceph-mon[35979]: osdmap e42: 6 total, 6 up, 6 in 2024-03-28T22:05:13.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:12 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:05:13.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:12 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:05:13.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:12 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:05:13.079 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:12 smithi016 ceph-mon[32374]: osdmap e41: 6 total, 6 up, 6 in 2024-03-28T22:05:13.079 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:12 smithi016 ceph-mon[32374]: osdmap e42: 6 total, 6 up, 6 in 2024-03-28T22:05:13.079 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:12 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:05:13.079 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:12 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:05:13.079 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:12 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:05:13.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:12 smithi063 ceph-mon[36249]: osdmap e41: 6 total, 6 up, 6 in 2024-03-28T22:05:13.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:12 smithi063 ceph-mon[36249]: osdmap e42: 6 total, 6 up, 6 in 2024-03-28T22:05:13.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:12 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:05:13.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:12 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:05:13.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:12 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:05:14.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:13 smithi053 ceph-mon[35979]: from='client.14346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:05:14.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:13 smithi053 ceph-mon[35979]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:05:14.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:13 smithi053 ceph-mon[35979]: osdmap e43: 6 total, 6 up, 6 in 2024-03-28T22:05:14.168 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:13 smithi016 ceph-mon[32374]: from='client.14346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:05:14.168 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:13 smithi016 ceph-mon[32374]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:05:14.168 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:13 smithi016 ceph-mon[32374]: osdmap e43: 6 total, 6 up, 6 in 2024-03-28T22:05:14.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:13 smithi063 ceph-mon[36249]: from='client.14346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:05:14.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:13 smithi063 ceph-mon[36249]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:05:14.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:13 smithi063 ceph-mon[36249]: osdmap e43: 6 total, 6 up, 6 in 2024-03-28T22:05:15.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:15 smithi063 ceph-mon[36249]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-03-28T22:05:15.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:15 smithi016 ceph-mon[32374]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-03-28T22:05:15.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:15 smithi053 ceph-mon[35979]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-03-28T22:05:16.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:16 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/1829925126' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "978efe5d-2c55-4ff8-8a1e-9f891388945e"}]: dispatch 2024-03-28T22:05:16.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:16 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/1829925126' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "978efe5d-2c55-4ff8-8a1e-9f891388945e"}]': finished 2024-03-28T22:05:16.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:16 smithi063 ceph-mon[36249]: osdmap e44: 7 total, 6 up, 7 in 2024-03-28T22:05:16.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:16 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:05:16.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:16 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/3329040622' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:05:16.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:16 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/1829925126' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "978efe5d-2c55-4ff8-8a1e-9f891388945e"}]: dispatch 2024-03-28T22:05:16.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:16 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/1829925126' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "978efe5d-2c55-4ff8-8a1e-9f891388945e"}]': finished 2024-03-28T22:05:16.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:16 smithi016 ceph-mon[32374]: osdmap e44: 7 total, 6 up, 7 in 2024-03-28T22:05:16.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:16 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:05:16.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:16 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3329040622' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:05:16.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:16 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/1829925126' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "978efe5d-2c55-4ff8-8a1e-9f891388945e"}]: dispatch 2024-03-28T22:05:16.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:16 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/1829925126' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "978efe5d-2c55-4ff8-8a1e-9f891388945e"}]': finished 2024-03-28T22:05:16.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:16 smithi053 ceph-mon[35979]: osdmap e44: 7 total, 6 up, 7 in 2024-03-28T22:05:16.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:16 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:05:16.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:16 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3329040622' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:05:17.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:17 smithi063 ceph-mon[36249]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-03-28T22:05:17.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:17 smithi016 ceph-mon[32374]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-03-28T22:05:17.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:17 smithi053 ceph-mon[35979]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-03-28T22:05:19.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:19 smithi063 ceph-mon[36249]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-28T22:05:19.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:19 smithi016 ceph-mon[32374]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-28T22:05:19.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:19 smithi053 ceph-mon[35979]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-28T22:05:21.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:21 smithi063 ceph-mon[36249]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T22:05:21.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:21 smithi016 ceph-mon[32374]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T22:05:21.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:21 smithi053 ceph-mon[35979]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T22:05:23.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:23 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-28T22:05:23.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:23 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:05:23.278 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:23 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-28T22:05:23.278 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:23 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:05:23.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:23 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-28T22:05:23.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:23 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:05:24.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:24 smithi016 ceph-mon[32374]: Deploying daemon osd.6 on smithi016 2024-03-28T22:05:24.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:24 smithi016 ceph-mon[32374]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-28T22:05:24.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:24 smithi053 ceph-mon[35979]: Deploying daemon osd.6 on smithi016 2024-03-28T22:05:24.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:24 smithi053 ceph-mon[35979]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-28T22:05:24.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:24 smithi063 ceph-mon[36249]: Deploying daemon osd.6 on smithi016 2024-03-28T22:05:24.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:24 smithi063 ceph-mon[36249]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-28T22:05:25.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:25 smithi063 ceph-mon[36249]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:05:25.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:25 smithi016 ceph-mon[32374]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:05:25.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:25 smithi053 ceph-mon[35979]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:05:27.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:27 smithi063 ceph-mon[36249]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:05:27.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:27 smithi016 ceph-mon[32374]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:05:27.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:27 smithi053 ceph-mon[35979]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:05:28.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:28 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:05:28.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:28 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:28.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:28 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:29.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:28 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:05:29.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:28 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:29.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:28 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:29.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:28 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:05:29.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:28 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:29.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:28 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:30.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:29 smithi063 ceph-mon[36249]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:05:30.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:29 smithi016 ceph-mon[32374]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:05:30.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:29 smithi053 ceph-mon[35979]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:05:31.377 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:31 smithi016 ceph-mon[32374]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:05:31.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:31 smithi063 ceph-mon[36249]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:05:31.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:31 smithi053 ceph-mon[35979]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:05:32.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:32.125+0000 7fb3a37fe700 1 -- 172.21.15.16:0/3057098468 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb3b4062680 con 0x7fb39c0a1e10 2024-03-28T22:05:32.128 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 6 on host 'smithi016' 2024-03-28T22:05:32.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:32.127+0000 7fb3b8b93700 1 -- 172.21.15.16:0/3057098468 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb39c0a1e10 msgr2=0x7fb39c0a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:05:32.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:32.127+0000 7fb3b8b93700 1 --2- 172.21.15.16:0/3057098468 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb39c0a1e10 0x7fb39c0a42d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8005950 tx=0x7fb3a800b410 comp rx=0 tx=0).stop 2024-03-28T22:05:32.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:32.127+0000 7fb3b8b93700 1 -- 172.21.15.16:0/3057098468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3b4102260 msgr2=0x7fb3b4198480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:05:32.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:32.127+0000 7fb3b8b93700 1 --2- 172.21.15.16:0/3057098468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3b4102260 0x7fb3b4198480 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb3a400b530 tx=0x7fb3a400b840 comp rx=0 tx=0).stop 2024-03-28T22:05:32.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:32.127+0000 7fb3b8b93700 1 -- 172.21.15.16:0/3057098468 shutdown_connections 2024-03-28T22:05:32.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:32.127+0000 7fb3b8b93700 1 --2- 172.21.15.16:0/3057098468 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb39c0a1e10 0x7fb39c0a42d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:05:32.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:32.127+0000 7fb3b8b93700 1 --2- 172.21.15.16:0/3057098468 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3b4103e00 0x7fb3b419fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:05:32.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:32.127+0000 7fb3b8b93700 1 --2- 172.21.15.16:0/3057098468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3b4102260 0x7fb3b4198480 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:05:32.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:32.127+0000 7fb3b8b93700 1 --2- 172.21.15.16:0/3057098468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3b4103460 0x7fb3b41989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:05:32.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:32.127+0000 7fb3b8b93700 1 -- 172.21.15.16:0/3057098468 >> 172.21.15.16:0/3057098468 conn(0x7fb3b40fd820 msgr2=0x7fb3b4106700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:05:32.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:32.127+0000 7fb3b8b93700 1 -- 172.21.15.16:0/3057098468 shutdown_connections 2024-03-28T22:05:32.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:05:32.127+0000 7fb3b8b93700 1 -- 172.21.15.16:0/3057098468 wait complete. 2024-03-28T22:05:32.815 DEBUG:teuthology.orchestra.run.smithi016:osd.6> sudo journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.6.service 2024-03-28T22:05:32.817 INFO:tasks.cephadm:Deploying osd.7 on smithi053 with /dev/vg_nvme/lv_2... 2024-03-28T22:05:32.818 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-28T22:05:33.078 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T22:05:33.380 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:33 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:33.380 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:33 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:33.380 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:33 smithi053 ceph-mon[35979]: from='osd.6 [v2:172.21.15.16:6818/4290459224,v1:172.21.15.16:6819/4290459224]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-28T22:05:33.381 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:33 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:33.381 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:33 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:33.381 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:33 smithi053 ceph-mon[35979]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:05:33.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:33 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:33.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:33 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:33.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:33 smithi063 ceph-mon[36249]: from='osd.6 [v2:172.21.15.16:6818/4290459224,v1:172.21.15.16:6819/4290459224]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-28T22:05:33.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:33 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:33.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:33 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:33.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:33 smithi063 ceph-mon[36249]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:05:33.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:33 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:33.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:33 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:33.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:33 smithi016 ceph-mon[32374]: from='osd.6 [v2:172.21.15.16:6818/4290459224,v1:172.21.15.16:6819/4290459224]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-28T22:05:33.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:33 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:33.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:33 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:33.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:33 smithi016 ceph-mon[32374]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:05:34.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:34 smithi063 ceph-mon[36249]: from='osd.6 [v2:172.21.15.16:6818/4290459224,v1:172.21.15.16:6819/4290459224]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-28T22:05:34.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:34 smithi063 ceph-mon[36249]: osdmap e45: 7 total, 6 up, 7 in 2024-03-28T22:05:34.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:34 smithi063 ceph-mon[36249]: from='osd.6 [v2:172.21.15.16:6818/4290459224,v1:172.21.15.16:6819/4290459224]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-03-28T22:05:34.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:34 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:05:34.434 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:34 smithi053 ceph-mon[35979]: from='osd.6 [v2:172.21.15.16:6818/4290459224,v1:172.21.15.16:6819/4290459224]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-28T22:05:34.435 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:34 smithi053 ceph-mon[35979]: osdmap e45: 7 total, 6 up, 7 in 2024-03-28T22:05:34.435 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:34 smithi053 ceph-mon[35979]: from='osd.6 [v2:172.21.15.16:6818/4290459224,v1:172.21.15.16:6819/4290459224]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-03-28T22:05:34.435 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:34 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:05:34.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:34 smithi016 ceph-mon[32374]: from='osd.6 [v2:172.21.15.16:6818/4290459224,v1:172.21.15.16:6819/4290459224]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-28T22:05:34.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:34 smithi016 ceph-mon[32374]: osdmap e45: 7 total, 6 up, 7 in 2024-03-28T22:05:34.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:34 smithi016 ceph-mon[32374]: from='osd.6 [v2:172.21.15.16:6818/4290459224,v1:172.21.15.16:6819/4290459224]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-03-28T22:05:34.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:34 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:05:35.118 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-28T22:05:35.118 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-28T22:05:35.118 INFO:teuthology.orchestra.run.smithi053.stderr: stderr: 10+0 records in 2024-03-28T22:05:35.118 INFO:teuthology.orchestra.run.smithi053.stderr:10+0 records out 2024-03-28T22:05:35.118 INFO:teuthology.orchestra.run.smithi053.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00844849 s, 1.2 GB/s 2024-03-28T22:05:35.118 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping successful for: 2024-03-28T22:05:35.125 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:05:34 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:05:34.791+0000 7f91d73e3700 -1 osd.6 0 waiting for initial osdmap 2024-03-28T22:05:35.126 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:05:34 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:05:34.801+0000 7f91cf9cf700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T22:05:35.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:35 smithi063 ceph-mon[36249]: from='osd.6 [v2:172.21.15.16:6818/4290459224,v1:172.21.15.16:6819/4290459224]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-03-28T22:05:35.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:35 smithi063 ceph-mon[36249]: osdmap e46: 7 total, 6 up, 7 in 2024-03-28T22:05:35.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:35 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:05:35.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:35 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:05:35.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:35 smithi063 ceph-mon[36249]: from='osd.6 [v2:172.21.15.16:6818/4290459224,v1:172.21.15.16:6819/4290459224]' entity='osd.6' 2024-03-28T22:05:35.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:35 smithi063 ceph-mon[36249]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:05:35.434 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:35 smithi053 ceph-mon[35979]: from='osd.6 [v2:172.21.15.16:6818/4290459224,v1:172.21.15.16:6819/4290459224]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-03-28T22:05:35.434 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:35 smithi053 ceph-mon[35979]: osdmap e46: 7 total, 6 up, 7 in 2024-03-28T22:05:35.434 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:35 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:05:35.434 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:35 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:05:35.434 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:35 smithi053 ceph-mon[35979]: from='osd.6 [v2:172.21.15.16:6818/4290459224,v1:172.21.15.16:6819/4290459224]' entity='osd.6' 2024-03-28T22:05:35.434 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:35 smithi053 ceph-mon[35979]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:05:35.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:35 smithi016 ceph-mon[32374]: from='osd.6 [v2:172.21.15.16:6818/4290459224,v1:172.21.15.16:6819/4290459224]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-03-28T22:05:35.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:35 smithi016 ceph-mon[32374]: osdmap e46: 7 total, 6 up, 7 in 2024-03-28T22:05:35.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:35 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:05:35.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:35 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:05:35.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:35 smithi016 ceph-mon[32374]: from='osd.6 [v2:172.21.15.16:6818/4290459224,v1:172.21.15.16:6819/4290459224]' entity='osd.6' 2024-03-28T22:05:35.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:35 smithi016 ceph-mon[32374]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:05:35.813 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi053:vg_nvme/lv_2 2024-03-28T22:05:36.069 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T22:05:36.272 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:36 smithi016 ceph-mon[32374]: purged_snaps scrub starts 2024-03-28T22:05:36.272 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:36 smithi016 ceph-mon[32374]: purged_snaps scrub ok 2024-03-28T22:05:36.272 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:36 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:05:36.272 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:36 smithi016 ceph-mon[32374]: osd.6 [v2:172.21.15.16:6818/4290459224,v1:172.21.15.16:6819/4290459224] boot 2024-03-28T22:05:36.272 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:36 smithi016 ceph-mon[32374]: osdmap e47: 7 total, 7 up, 7 in 2024-03-28T22:05:36.272 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:36 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:05:36.422 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:36 smithi053 ceph-mon[35979]: purged_snaps scrub starts 2024-03-28T22:05:36.422 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:36 smithi053 ceph-mon[35979]: purged_snaps scrub ok 2024-03-28T22:05:36.423 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:36 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:05:36.423 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:36 smithi053 ceph-mon[35979]: osd.6 [v2:172.21.15.16:6818/4290459224,v1:172.21.15.16:6819/4290459224] boot 2024-03-28T22:05:36.423 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:36 smithi053 ceph-mon[35979]: osdmap e47: 7 total, 7 up, 7 in 2024-03-28T22:05:36.423 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:36 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:05:36.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:36 smithi063 ceph-mon[36249]: purged_snaps scrub starts 2024-03-28T22:05:36.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:36 smithi063 ceph-mon[36249]: purged_snaps scrub ok 2024-03-28T22:05:36.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:36 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:05:36.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:36 smithi063 ceph-mon[36249]: osd.6 [v2:172.21.15.16:6818/4290459224,v1:172.21.15.16:6819/4290459224] boot 2024-03-28T22:05:36.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:36 smithi063 ceph-mon[36249]: osdmap e47: 7 total, 7 up, 7 in 2024-03-28T22:05:36.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:36 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:05:37.668 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:37 smithi053 ceph-mon[35979]: Detected new or changed devices on smithi016 2024-03-28T22:05:37.668 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:37 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:37.669 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:37 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:37.669 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:37 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:05:37.669 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:37 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:05:37.669 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:37 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:05:37.669 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:37 smithi053 ceph-mon[35979]: Adjusting osd_memory_target on smithi016 to 5697M 2024-03-28T22:05:37.669 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:37 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:37.669 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:37 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:05:37.669 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:37 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:05:37.669 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:37 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:37.669 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:37 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:05:37.669 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:37 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:05:37.669 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:37 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:05:37.669 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:37 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:37.669 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:37 smithi053 ceph-mon[35979]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:05:37.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.676+0000 7faea2e4b700 1 -- 172.21.15.53:0/483940068 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae9c1056f0 msgr2=0x7fae9c107ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:05:37.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.676+0000 7faea2e4b700 1 --2- 172.21.15.53:0/483940068 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae9c1056f0 0x7fae9c107ae0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fae88009990 tx=0x7fae88009ca0 comp rx=0 tx=0).stop 2024-03-28T22:05:37.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.676+0000 7faea2e4b700 1 -- 172.21.15.53:0/483940068 shutdown_connections 2024-03-28T22:05:37.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.676+0000 7faea2e4b700 1 --2- 172.21.15.53:0/483940068 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae9c108020 0x7fae9c10a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:05:37.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.676+0000 7faea2e4b700 1 --2- 172.21.15.53:0/483940068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae9c102dc0 0x7fae9c1051b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:05:37.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.676+0000 7faea2e4b700 1 --2- 172.21.15.53:0/483940068 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae9c1056f0 0x7fae9c107ae0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:05:37.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.676+0000 7faea2e4b700 1 -- 172.21.15.53:0/483940068 >> 172.21.15.53:0/483940068 conn(0x7fae9c0fc710 msgr2=0x7fae9c0feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:05:37.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.676+0000 7faea2e4b700 1 -- 172.21.15.53:0/483940068 shutdown_connections 2024-03-28T22:05:37.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.676+0000 7faea2e4b700 1 -- 172.21.15.53:0/483940068 wait complete. 2024-03-28T22:05:37.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.677+0000 7faea2e4b700 1 Processor -- start 2024-03-28T22:05:37.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.677+0000 7faea2e4b700 1 -- start start 2024-03-28T22:05:37.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.677+0000 7faea2e4b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae9c102dc0 0x7fae9c198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:05:37.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.677+0000 7faea2e4b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae9c1056f0 0x7fae9c1989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:05:37.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.677+0000 7faea2e4b700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae9c108020 0x7fae9c19fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:05:37.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.677+0000 7faea2e4b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fae9c10de70 con 0x7fae9c1056f0 2024-03-28T22:05:37.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.677+0000 7faea2e4b700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fae9c10dcf0 con 0x7fae9c102dc0 2024-03-28T22:05:37.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.677+0000 7faea2e4b700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fae9c10dff0 con 0x7fae9c108020 2024-03-28T22:05:37.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.677+0000 7faea0be7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae9c102dc0 0x7fae9c198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:05:37.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.677+0000 7faea0be7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae9c102dc0 0x7fae9c198480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47264/0 (socket says 172.21.15.53:47264) 2024-03-28T22:05:37.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.677+0000 7faea0be7700 1 -- 172.21.15.53:0/2681393122 learned_addr learned my addr 172.21.15.53:0/2681393122 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-03-28T22:05:37.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.677+0000 7fae9bfff700 1 --2- 172.21.15.53:0/2681393122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae9c1056f0 0x7fae9c1989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:05:37.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.677+0000 7faea13e8700 1 --2- 172.21.15.53:0/2681393122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae9c108020 0x7fae9c19fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:05:37.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.677+0000 7faea0be7700 1 -- 172.21.15.53:0/2681393122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae9c108020 msgr2=0x7fae9c19fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:05:37.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.677+0000 7faea0be7700 1 --2- 172.21.15.53:0/2681393122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae9c108020 0x7fae9c19fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:05:37.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.677+0000 7faea0be7700 1 -- 172.21.15.53:0/2681393122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae9c1056f0 msgr2=0x7fae9c1989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:05:37.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.677+0000 7faea0be7700 1 --2- 172.21.15.53:0/2681393122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae9c1056f0 0x7fae9c1989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:05:37.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.677+0000 7faea0be7700 1 -- 172.21.15.53:0/2681393122 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae9c1a00e0 con 0x7fae9c102dc0 2024-03-28T22:05:37.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.678+0000 7faea0be7700 1 --2- 172.21.15.53:0/2681393122 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae9c102dc0 0x7fae9c198480 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fae9000c920 tx=0x7fae9000cc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:05:37.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.678+0000 7fae99ffb700 1 -- 172.21.15.53:0/2681393122 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae90007b00 con 0x7fae9c102dc0 2024-03-28T22:05:37.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.678+0000 7faea2e4b700 1 -- 172.21.15.53:0/2681393122 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae9c1a03d0 con 0x7fae9c102dc0 2024-03-28T22:05:37.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.678+0000 7fae99ffb700 1 -- 172.21.15.53:0/2681393122 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae90004d10 con 0x7fae9c102dc0 2024-03-28T22:05:37.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.678+0000 7fae99ffb700 1 -- 172.21.15.53:0/2681393122 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae90005680 con 0x7fae9c102dc0 2024-03-28T22:05:37.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.678+0000 7faea2e4b700 1 -- 172.21.15.53:0/2681393122 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae9c1a0910 con 0x7fae9c102dc0 2024-03-28T22:05:37.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.679+0000 7faea2e4b700 1 -- 172.21.15.53:0/2681393122 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae9c068f00 con 0x7fae9c102dc0 2024-03-28T22:05:37.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:37 smithi063 ceph-mon[36249]: Detected new or changed devices on smithi016 2024-03-28T22:05:37.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:37 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:37.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:37 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:37.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:37 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:05:37.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:37 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:05:37.684 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:37 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:05:37.684 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:37 smithi063 ceph-mon[36249]: Adjusting osd_memory_target on smithi016 to 5697M 2024-03-28T22:05:37.684 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:37 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:37.684 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:37 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:05:37.684 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:37 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:05:37.684 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:37 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:37.684 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:37 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:05:37.684 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:37 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:05:37.684 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:37 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:05:37.684 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:37 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:37.684 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:37 smithi063 ceph-mon[36249]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:05:37.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.680+0000 7fae99ffb700 1 -- 172.21.15.53:0/2681393122 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fae90004750 con 0x7fae9c102dc0 2024-03-28T22:05:37.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.680+0000 7fae99ffb700 1 --2- 172.21.15.53:0/2681393122 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fae8c0a1e10 0x7fae8c0a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:05:37.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.681+0000 7fae99ffb700 1 -- 172.21.15.53:0/2681393122 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4965+0+0 (secure 0 0 0) 0x7fae900ccc70 con 0x7fae9c102dc0 2024-03-28T22:05:37.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.681+0000 7fae9bfff700 1 --2- 172.21.15.53:0/2681393122 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fae8c0a1e10 0x7fae8c0a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:05:37.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.681+0000 7fae9bfff700 1 --2- 172.21.15.53:0/2681393122 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fae8c0a1e10 0x7fae8c0a42d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fae8800b5c0 tx=0x7fae88003960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:05:37.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:37 smithi016 ceph-mon[32374]: Detected new or changed devices on smithi016 2024-03-28T22:05:37.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:37 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:37.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:37 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:37.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:37 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:05:37.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:37 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:05:37.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:37 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:05:37.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:37 smithi016 ceph-mon[32374]: Adjusting osd_memory_target on smithi016 to 5697M 2024-03-28T22:05:37.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:37 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:37.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:37 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:05:37.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:37 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:05:37.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:37 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:37.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:37 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:05:37.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:37 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:05:37.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:37 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:05:37.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:37 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:37.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:37 smithi016 ceph-mon[32374]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:05:37.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.685+0000 7fae99ffb700 1 -- 172.21.15.53:0/2681393122 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fae90014030 con 0x7fae9c102dc0 2024-03-28T22:05:37.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:37.900+0000 7faea2e4b700 1 -- 172.21.15.53:0/2681393122 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fae9c062680 con 0x7fae8c0a1e10 2024-03-28T22:05:38.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:38 smithi053 ceph-mon[35979]: osdmap e48: 7 total, 7 up, 7 in 2024-03-28T22:05:38.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:38 smithi053 ceph-mon[35979]: from='client.24305 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:05:38.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:38 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:05:38.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:38 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:05:38.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:38 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:05:38.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:38 smithi063 ceph-mon[36249]: osdmap e48: 7 total, 7 up, 7 in 2024-03-28T22:05:38.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:38 smithi063 ceph-mon[36249]: from='client.24305 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:05:38.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:38 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:05:38.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:38 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:05:38.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:38 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:05:38.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:38 smithi016 ceph-mon[32374]: osdmap e48: 7 total, 7 up, 7 in 2024-03-28T22:05:38.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:38 smithi016 ceph-mon[32374]: from='client.24305 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:05:38.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:38 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:05:38.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:38 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:05:38.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:38 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:05:39.676 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:39 smithi053 ceph-mon[35979]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:05:39.833 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:39 smithi063 ceph-mon[36249]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:05:39.837 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:39 smithi016 ceph-mon[32374]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:05:40.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:40 smithi053 ceph-mon[35979]: from='client.? 172.21.15.53:0/2383850790' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50d489cb-dbb8-48df-b868-b29edf12e04d"}]: dispatch 2024-03-28T22:05:40.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:40 smithi053 ceph-mon[35979]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50d489cb-dbb8-48df-b868-b29edf12e04d"}]: dispatch 2024-03-28T22:05:40.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:40 smithi053 ceph-mon[35979]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50d489cb-dbb8-48df-b868-b29edf12e04d"}]': finished 2024-03-28T22:05:40.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:40 smithi053 ceph-mon[35979]: osdmap e49: 8 total, 7 up, 8 in 2024-03-28T22:05:40.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:40 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:05:40.847 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:40 smithi063 ceph-mon[36249]: from='client.? 172.21.15.53:0/2383850790' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50d489cb-dbb8-48df-b868-b29edf12e04d"}]: dispatch 2024-03-28T22:05:40.847 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:40 smithi063 ceph-mon[36249]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50d489cb-dbb8-48df-b868-b29edf12e04d"}]: dispatch 2024-03-28T22:05:40.847 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:40 smithi063 ceph-mon[36249]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50d489cb-dbb8-48df-b868-b29edf12e04d"}]': finished 2024-03-28T22:05:40.847 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:40 smithi063 ceph-mon[36249]: osdmap e49: 8 total, 7 up, 8 in 2024-03-28T22:05:40.847 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:40 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:05:40.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:40 smithi016 ceph-mon[32374]: from='client.? 172.21.15.53:0/2383850790' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50d489cb-dbb8-48df-b868-b29edf12e04d"}]: dispatch 2024-03-28T22:05:40.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:40 smithi016 ceph-mon[32374]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50d489cb-dbb8-48df-b868-b29edf12e04d"}]: dispatch 2024-03-28T22:05:40.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:40 smithi016 ceph-mon[32374]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50d489cb-dbb8-48df-b868-b29edf12e04d"}]': finished 2024-03-28T22:05:40.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:40 smithi016 ceph-mon[32374]: osdmap e49: 8 total, 7 up, 8 in 2024-03-28T22:05:40.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:40 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:05:41.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:41 smithi053 ceph-mon[35979]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:05:41.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:41 smithi053 ceph-mon[35979]: from='client.? 172.21.15.53:0/4252072348' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:05:41.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:41 smithi063 ceph-mon[36249]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:05:41.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:41 smithi063 ceph-mon[36249]: from='client.? 172.21.15.53:0/4252072348' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:05:41.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:41 smithi016 ceph-mon[32374]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:05:41.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:41 smithi016 ceph-mon[32374]: from='client.? 172.21.15.53:0/4252072348' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:05:43.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:43 smithi063 ceph-mon[36249]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:05:43.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:43 smithi016 ceph-mon[32374]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:05:43.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:43 smithi053 ceph-mon[35979]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:05:45.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:45 smithi063 ceph-mon[36249]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:05:45.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:45 smithi016 ceph-mon[32374]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:05:45.443 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:45 smithi053 ceph-mon[35979]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:05:47.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:47 smithi053 ceph-mon[35979]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:05:47.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:47 smithi063 ceph-mon[36249]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:05:47.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:47 smithi016 ceph-mon[32374]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:05:48.381 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:48 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-28T22:05:48.381 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:48 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:05:48.381 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:48 smithi053 ceph-mon[35979]: Deploying daemon osd.7 on smithi053 2024-03-28T22:05:48.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:48 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-28T22:05:48.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:48 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:05:48.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:48 smithi063 ceph-mon[36249]: Deploying daemon osd.7 on smithi053 2024-03-28T22:05:48.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:48 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-28T22:05:48.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:48 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:05:48.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:48 smithi016 ceph-mon[32374]: Deploying daemon osd.7 on smithi053 2024-03-28T22:05:49.403 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:49 smithi053 ceph-mon[35979]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:05:49.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:49 smithi063 ceph-mon[36249]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:05:49.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:49 smithi016 ceph-mon[32374]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:05:51.339 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:51 smithi053 ceph-mon[35979]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:05:51.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:51 smithi063 ceph-mon[36249]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:05:51.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:51 smithi016 ceph-mon[32374]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:05:54.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:53 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:54.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:53 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:54.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:53 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:05:54.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:53 smithi053 ceph-mon[35979]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:05:54.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:53 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:54.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:53 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:54.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:53 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:05:54.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:53 smithi063 ceph-mon[36249]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:05:54.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:53 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:54.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:53 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:54.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:53 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:05:54.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:53 smithi016 ceph-mon[32374]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:05:55.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:55 smithi053 ceph-mon[35979]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:05:55.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:55 smithi063 ceph-mon[36249]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:05:55.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:55 smithi016 ceph-mon[32374]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:05:56.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:56.113+0000 7fae99ffb700 1 -- 172.21.15.53:0/2681393122 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fae9c062680 con 0x7fae8c0a1e10 2024-03-28T22:05:56.116 INFO:teuthology.orchestra.run.smithi053.stdout:Created osd(s) 7 on host 'smithi053' 2024-03-28T22:05:56.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:56.116+0000 7faea2e4b700 1 -- 172.21.15.53:0/2681393122 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fae8c0a1e10 msgr2=0x7fae8c0a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:05:56.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:56.116+0000 7faea2e4b700 1 --2- 172.21.15.53:0/2681393122 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fae8c0a1e10 0x7fae8c0a42d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fae8800b5c0 tx=0x7fae88003960 comp rx=0 tx=0).stop 2024-03-28T22:05:56.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:56.116+0000 7faea2e4b700 1 -- 172.21.15.53:0/2681393122 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae9c102dc0 msgr2=0x7fae9c198480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:05:56.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:56.116+0000 7faea2e4b700 1 --2- 172.21.15.53:0/2681393122 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae9c102dc0 0x7fae9c198480 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fae9000c920 tx=0x7fae9000cc30 comp rx=0 tx=0).stop 2024-03-28T22:05:56.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:56.116+0000 7faea2e4b700 1 -- 172.21.15.53:0/2681393122 shutdown_connections 2024-03-28T22:05:56.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:56.116+0000 7faea2e4b700 1 --2- 172.21.15.53:0/2681393122 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fae8c0a1e10 0x7fae8c0a42d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:05:56.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:56.116+0000 7faea2e4b700 1 --2- 172.21.15.53:0/2681393122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae9c108020 0x7fae9c19fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:05:56.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:56.116+0000 7faea2e4b700 1 --2- 172.21.15.53:0/2681393122 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae9c102dc0 0x7fae9c198480 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:05:56.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:56.116+0000 7faea2e4b700 1 --2- 172.21.15.53:0/2681393122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae9c1056f0 0x7fae9c1989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:05:56.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:56.116+0000 7faea2e4b700 1 -- 172.21.15.53:0/2681393122 >> 172.21.15.53:0/2681393122 conn(0x7fae9c0fc710 msgr2=0x7fae9c103820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:05:56.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:56.116+0000 7faea2e4b700 1 -- 172.21.15.53:0/2681393122 shutdown_connections 2024-03-28T22:05:56.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:05:56.116+0000 7faea2e4b700 1 -- 172.21.15.53:0/2681393122 wait complete. 2024-03-28T22:05:56.751 DEBUG:teuthology.orchestra.run.smithi053:osd.7> sudo journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.7.service 2024-03-28T22:05:56.753 INFO:tasks.cephadm:Deploying osd.8 on smithi063 with /dev/vg_nvme/lv_2... 2024-03-28T22:05:56.753 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-28T22:05:57.017 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T22:05:57.428 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:57 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:57.428 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:57 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:57.428 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:57 smithi063 ceph-mon[36249]: from='osd.7 [v2:172.21.15.53:6816/1056467525,v1:172.21.15.53:6817/1056467525]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T22:05:57.428 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:57 smithi063 ceph-mon[36249]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T22:05:57.428 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:57 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:57.428 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:57 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:57.428 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:57 smithi063 ceph-mon[36249]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:05:57.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:57 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:57.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:57 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:57.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:57 smithi016 ceph-mon[32374]: from='osd.7 [v2:172.21.15.53:6816/1056467525,v1:172.21.15.53:6817/1056467525]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T22:05:57.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:57 smithi016 ceph-mon[32374]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T22:05:57.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:57 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:57.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:57 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:57.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:57 smithi016 ceph-mon[32374]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:05:57.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:57 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:57.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:57 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:57.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:57 smithi053 ceph-mon[35979]: from='osd.7 [v2:172.21.15.53:6816/1056467525,v1:172.21.15.53:6817/1056467525]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T22:05:57.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:57 smithi053 ceph-mon[35979]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T22:05:57.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:57 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:57.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:57 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:05:57.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:57 smithi053 ceph-mon[35979]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:05:58.388 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:58 smithi063 ceph-mon[36249]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-28T22:05:58.388 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:58 smithi063 ceph-mon[36249]: from='osd.7 [v2:172.21.15.53:6816/1056467525,v1:172.21.15.53:6817/1056467525]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-03-28T22:05:58.388 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:58 smithi063 ceph-mon[36249]: osdmap e50: 8 total, 7 up, 8 in 2024-03-28T22:05:58.388 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:58 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:05:58.388 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:58 smithi063 ceph-mon[36249]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-03-28T22:05:58.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:58 smithi016 ceph-mon[32374]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-28T22:05:58.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:58 smithi016 ceph-mon[32374]: from='osd.7 [v2:172.21.15.53:6816/1056467525,v1:172.21.15.53:6817/1056467525]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-03-28T22:05:58.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:58 smithi016 ceph-mon[32374]: osdmap e50: 8 total, 7 up, 8 in 2024-03-28T22:05:58.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:58 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:05:58.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:58 smithi016 ceph-mon[32374]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-03-28T22:05:58.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:58 smithi053 ceph-mon[35979]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-28T22:05:58.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:58 smithi053 ceph-mon[35979]: from='osd.7 [v2:172.21.15.53:6816/1056467525,v1:172.21.15.53:6817/1056467525]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-03-28T22:05:58.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:58 smithi053 ceph-mon[35979]: osdmap e50: 8 total, 7 up, 8 in 2024-03-28T22:05:58.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:58 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:05:58.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:58 smithi053 ceph-mon[35979]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-03-28T22:05:59.096 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-28T22:05:59.096 INFO:teuthology.orchestra.run.smithi063.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-28T22:05:59.097 INFO:teuthology.orchestra.run.smithi063.stderr: stderr: 10+0 records in 2024-03-28T22:05:59.097 INFO:teuthology.orchestra.run.smithi063.stderr:10+0 records out 2024-03-28T22:05:59.097 INFO:teuthology.orchestra.run.smithi063.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127363 s, 823 MB/s 2024-03-28T22:05:59.097 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping successful for: 2024-03-28T22:05:59.104 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:05:58 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:05:58.817+0000 7f10a7a45700 -1 osd.7 0 waiting for initial osdmap 2024-03-28T22:05:59.104 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:05:58 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:05:58.828+0000 7f10a0031700 -1 osd.7 51 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T22:05:59.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:59 smithi063 ceph-mon[36249]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-03-28T22:05:59.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:59 smithi063 ceph-mon[36249]: osdmap e51: 8 total, 7 up, 8 in 2024-03-28T22:05:59.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:59 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:05:59.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:59 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:05:59.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:59 smithi063 ceph-mon[36249]: from='osd.7 ' entity='osd.7' 2024-03-28T22:05:59.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:05:59 smithi063 ceph-mon[36249]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:05:59.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:59 smithi016 ceph-mon[32374]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-03-28T22:05:59.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:59 smithi016 ceph-mon[32374]: osdmap e51: 8 total, 7 up, 8 in 2024-03-28T22:05:59.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:59 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:05:59.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:59 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:05:59.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:59 smithi016 ceph-mon[32374]: from='osd.7 ' entity='osd.7' 2024-03-28T22:05:59.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:05:59 smithi016 ceph-mon[32374]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:05:59.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:59 smithi053 ceph-mon[35979]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-03-28T22:05:59.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:59 smithi053 ceph-mon[35979]: osdmap e51: 8 total, 7 up, 8 in 2024-03-28T22:05:59.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:59 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:05:59.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:59 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:05:59.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:59 smithi053 ceph-mon[35979]: from='osd.7 ' entity='osd.7' 2024-03-28T22:05:59.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:05:59 smithi053 ceph-mon[35979]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:05:59.888 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi063:vg_nvme/lv_2 2024-03-28T22:06:00.144 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T22:06:00.209 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:00 smithi053 ceph-mon[35979]: purged_snaps scrub starts 2024-03-28T22:06:00.210 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:00 smithi053 ceph-mon[35979]: purged_snaps scrub ok 2024-03-28T22:06:00.210 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:00 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:06:00.210 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:00 smithi053 ceph-mon[35979]: osd.7 [v2:172.21.15.53:6816/1056467525,v1:172.21.15.53:6817/1056467525] boot 2024-03-28T22:06:00.210 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:00 smithi053 ceph-mon[35979]: osdmap e52: 8 total, 8 up, 8 in 2024-03-28T22:06:00.210 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:00 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:06:00.382 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:00 smithi063 ceph-mon[36249]: purged_snaps scrub starts 2024-03-28T22:06:00.382 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:00 smithi063 ceph-mon[36249]: purged_snaps scrub ok 2024-03-28T22:06:00.382 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:00 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:06:00.382 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:00 smithi063 ceph-mon[36249]: osd.7 [v2:172.21.15.53:6816/1056467525,v1:172.21.15.53:6817/1056467525] boot 2024-03-28T22:06:00.382 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:00 smithi063 ceph-mon[36249]: osdmap e52: 8 total, 8 up, 8 in 2024-03-28T22:06:00.383 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:00 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:06:00.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:00 smithi016 ceph-mon[32374]: purged_snaps scrub starts 2024-03-28T22:06:00.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:00 smithi016 ceph-mon[32374]: purged_snaps scrub ok 2024-03-28T22:06:00.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:00 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:06:00.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:00 smithi016 ceph-mon[32374]: osd.7 [v2:172.21.15.53:6816/1056467525,v1:172.21.15.53:6817/1056467525] boot 2024-03-28T22:06:00.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:00 smithi016 ceph-mon[32374]: osdmap e52: 8 total, 8 up, 8 in 2024-03-28T22:06:00.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:00 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:06:01.386 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:01 smithi063 ceph-mon[36249]: osdmap e53: 8 total, 8 up, 8 in 2024-03-28T22:06:01.387 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:01 smithi063 ceph-mon[36249]: Detected new or changed devices on smithi053 2024-03-28T22:06:01.387 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:01 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:01.387 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:01 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:01.387 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:01 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:06:01.387 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:01 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:06:01.387 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:01 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:06:01.387 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:01 smithi063 ceph-mon[36249]: Adjusting osd_memory_target on smithi053 to 5694M 2024-03-28T22:06:01.387 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:01 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:01.387 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:01 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:06:01.387 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:01 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:06:01.387 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:01 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:01.387 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:01 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:06:01.387 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:01 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:06:01.387 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:01 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:06:01.387 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:01 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:01.388 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:01 smithi063 ceph-mon[36249]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:06:01.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:01 smithi016 ceph-mon[32374]: osdmap e53: 8 total, 8 up, 8 in 2024-03-28T22:06:01.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:01 smithi016 ceph-mon[32374]: Detected new or changed devices on smithi053 2024-03-28T22:06:01.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:01 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:01.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:01 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:01.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:01 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:06:01.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:01 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:06:01.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:01 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:06:01.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:01 smithi016 ceph-mon[32374]: Adjusting osd_memory_target on smithi053 to 5694M 2024-03-28T22:06:01.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:01 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:01.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:01 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:06:01.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:01 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:06:01.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:01 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:01.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:01 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:06:01.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:01 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:06:01.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:01 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:06:01.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:01 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:01.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:01 smithi016 ceph-mon[32374]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:06:01.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:01 smithi053 ceph-mon[35979]: osdmap e53: 8 total, 8 up, 8 in 2024-03-28T22:06:01.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:01 smithi053 ceph-mon[35979]: Detected new or changed devices on smithi053 2024-03-28T22:06:01.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:01 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:01.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:01 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:01.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:01 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:06:01.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:01 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:06:01.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:01 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:06:01.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:01 smithi053 ceph-mon[35979]: Adjusting osd_memory_target on smithi053 to 5694M 2024-03-28T22:06:01.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:01 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:01.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:01 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:06:01.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:01 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:06:01.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:01 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:01.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:01 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:06:01.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:01 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:06:01.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:01 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:06:01.549 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:01 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:01.549 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:01 smithi053 ceph-mon[35979]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:06:01.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.709+0000 7f0bd4e5e700 1 -- 172.21.15.63:0/2083343125 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0bd0102280 msgr2=0x7f0bd01026a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:06:01.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.709+0000 7f0bd4e5e700 1 --2- 172.21.15.63:0/2083343125 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0bd0102280 0x7f0bd01026a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0009990 tx=0x7f0bc0009ca0 comp rx=0 tx=0).stop 2024-03-28T22:06:01.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.709+0000 7f0bd4e5e700 1 -- 172.21.15.63:0/2083343125 shutdown_connections 2024-03-28T22:06:01.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.709+0000 7f0bd4e5e700 1 --2- 172.21.15.63:0/2083343125 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0bd0103e20 0x7f0bd010a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:06:01.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.709+0000 7f0bd4e5e700 1 --2- 172.21.15.63:0/2083343125 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0bd0102280 0x7f0bd01026a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:06:01.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.709+0000 7f0bd4e5e700 1 --2- 172.21.15.63:0/2083343125 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bd0103480 0x7f0bd01038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:06:01.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.709+0000 7f0bd4e5e700 1 -- 172.21.15.63:0/2083343125 >> 172.21.15.63:0/2083343125 conn(0x7f0bd00fd7e0 msgr2=0x7f0bd00ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:06:01.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.709+0000 7f0bd4e5e700 1 -- 172.21.15.63:0/2083343125 shutdown_connections 2024-03-28T22:06:01.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.709+0000 7f0bd4e5e700 1 -- 172.21.15.63:0/2083343125 wait complete. 2024-03-28T22:06:01.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.710+0000 7f0bd4e5e700 1 Processor -- start 2024-03-28T22:06:01.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.710+0000 7f0bd4e5e700 1 -- start start 2024-03-28T22:06:01.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.710+0000 7f0bd4e5e700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0bd0102280 0x7f0bd01984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:06:01.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.710+0000 7f0bd4e5e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bd0103480 0x7f0bd01989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:06:01.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.710+0000 7f0bd4e5e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0bd0103e20 0x7f0bd019fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:06:01.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.710+0000 7f0bd4e5e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd010de70 con 0x7f0bd0103480 2024-03-28T22:06:01.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.710+0000 7f0bd4e5e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd010dcf0 con 0x7f0bd0103e20 2024-03-28T22:06:01.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.710+0000 7f0bd4e5e700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd010dff0 con 0x7f0bd0102280 2024-03-28T22:06:01.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.710+0000 7f0bce59c700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0bd0102280 0x7f0bd01984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:06:01.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.710+0000 7f0bce59c700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0bd0102280 0x7f0bd01984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46186/0 (socket says 172.21.15.63:46186) 2024-03-28T22:06:01.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.710+0000 7f0bce59c700 1 -- 172.21.15.63:0/315966124 learned_addr learned my addr 172.21.15.63:0/315966124 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-28T22:06:01.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.710+0000 7f0bced9d700 1 --2- 172.21.15.63:0/315966124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0bd0103e20 0x7f0bd019fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:06:01.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.711+0000 7f0bcdd9b700 1 --2- 172.21.15.63:0/315966124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bd0103480 0x7f0bd01989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:06:01.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.711+0000 7f0bce59c700 1 -- 172.21.15.63:0/315966124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0bd0103e20 msgr2=0x7f0bd019fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:06:01.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.711+0000 7f0bce59c700 1 --2- 172.21.15.63:0/315966124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0bd0103e20 0x7f0bd019fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:06:01.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.711+0000 7f0bce59c700 1 -- 172.21.15.63:0/315966124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bd0103480 msgr2=0x7f0bd01989f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:06:01.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.711+0000 7f0bce59c700 1 --2- 172.21.15.63:0/315966124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bd0103480 0x7f0bd01989f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:06:01.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.711+0000 7f0bce59c700 1 -- 172.21.15.63:0/315966124 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bd01a0110 con 0x7f0bd0102280 2024-03-28T22:06:01.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.711+0000 7f0bce59c700 1 --2- 172.21.15.63:0/315966124 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0bd0102280 0x7f0bd01984b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0000c00 tx=0x7f0bc00047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:06:01.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.711+0000 7f0bbf7fe700 1 -- 172.21.15.63:0/315966124 <== mon.1 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc0003c70 con 0x7f0bd0102280 2024-03-28T22:06:01.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.711+0000 7f0bbf7fe700 1 -- 172.21.15.63:0/315966124 <== mon.1 v2:172.21.15.63:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0bc0004830 con 0x7f0bd0102280 2024-03-28T22:06:01.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.711+0000 7f0bd4e5e700 1 -- 172.21.15.63:0/315966124 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bd01a03a0 con 0x7f0bd0102280 2024-03-28T22:06:01.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.711+0000 7f0bbf7fe700 1 -- 172.21.15.63:0/315966124 <== mon.1 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc0005330 con 0x7f0bd0102280 2024-03-28T22:06:01.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.712+0000 7f0bd4e5e700 1 -- 172.21.15.63:0/315966124 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bd01a0790 con 0x7f0bd0102280 2024-03-28T22:06:01.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.713+0000 7f0bd4e5e700 1 -- 172.21.15.63:0/315966124 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bd0068f00 con 0x7f0bd0102280 2024-03-28T22:06:01.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.713+0000 7f0bbf7fe700 1 -- 172.21.15.63:0/315966124 <== mon.1 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f0bc00049e0 con 0x7f0bd0102280 2024-03-28T22:06:01.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.714+0000 7f0bbf7fe700 1 --2- 172.21.15.63:0/315966124 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0bb80a1e10 0x7f0bb80a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:06:01.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.714+0000 7f0bbf7fe700 1 -- 172.21.15.63:0/315966124 <== mon.1 v2:172.21.15.63:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5426+0+0 (secure 0 0 0) 0x7f0bc00ce430 con 0x7f0bd0102280 2024-03-28T22:06:01.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.714+0000 7f0bcdd9b700 1 --2- 172.21.15.63:0/315966124 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0bb80a1e10 0x7f0bb80a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:06:01.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.715+0000 7f0bcdd9b700 1 --2- 172.21.15.63:0/315966124 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0bb80a1e10 0x7f0bb80a42d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4005950 tx=0x7f0bc4005cf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:06:01.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.718+0000 7f0bbf7fe700 1 -- 172.21.15.63:0/315966124 <== mon.1 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0bc005d280 con 0x7f0bd0102280 2024-03-28T22:06:01.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:01.933+0000 7f0bd4e5e700 1 -- 172.21.15.63:0/315966124 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f0bd0062740 con 0x7f0bb80a1e10 2024-03-28T22:06:02.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:02 smithi063 ceph-mon[36249]: osdmap e54: 8 total, 8 up, 8 in 2024-03-28T22:06:02.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:02 smithi063 ceph-mon[36249]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:06:02.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:02 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:06:02.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:02 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:06:02.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:02 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:06:02.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:02 smithi016 ceph-mon[32374]: osdmap e54: 8 total, 8 up, 8 in 2024-03-28T22:06:02.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:02 smithi016 ceph-mon[32374]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:06:02.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:02 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:06:02.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:02 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:06:02.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:02 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:06:02.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:02 smithi053 ceph-mon[35979]: osdmap e54: 8 total, 8 up, 8 in 2024-03-28T22:06:02.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:02 smithi053 ceph-mon[35979]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:06:02.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:02 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:06:02.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:02 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:06:02.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:02 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:06:03.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:03 smithi053 ceph-mon[35979]: osdmap e55: 8 total, 8 up, 8 in 2024-03-28T22:06:03.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:03 smithi053 ceph-mon[35979]: pgmap v182: 1 pgs: 1 remapped+peering; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:06:03.551 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:03 smithi063 ceph-mon[36249]: osdmap e55: 8 total, 8 up, 8 in 2024-03-28T22:06:03.551 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:03 smithi063 ceph-mon[36249]: pgmap v182: 1 pgs: 1 remapped+peering; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:06:03.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:03 smithi016 ceph-mon[32374]: osdmap e55: 8 total, 8 up, 8 in 2024-03-28T22:06:03.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:03 smithi016 ceph-mon[32374]: pgmap v182: 1 pgs: 1 remapped+peering; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:06:04.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:04 smithi053 ceph-mon[35979]: osdmap e56: 8 total, 8 up, 8 in 2024-03-28T22:06:04.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:04 smithi063 ceph-mon[36249]: osdmap e56: 8 total, 8 up, 8 in 2024-03-28T22:06:04.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:04 smithi016 ceph-mon[32374]: osdmap e56: 8 total, 8 up, 8 in 2024-03-28T22:06:05.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:05 smithi053 ceph-mon[35979]: from='client.? 172.21.15.63:0/742111356' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49b39414-cc9c-4057-ae3f-cc109cd979da"}]: dispatch 2024-03-28T22:06:05.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:05 smithi053 ceph-mon[35979]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49b39414-cc9c-4057-ae3f-cc109cd979da"}]: dispatch 2024-03-28T22:06:05.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:05 smithi053 ceph-mon[35979]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "49b39414-cc9c-4057-ae3f-cc109cd979da"}]': finished 2024-03-28T22:06:05.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:05 smithi053 ceph-mon[35979]: osdmap e57: 9 total, 8 up, 9 in 2024-03-28T22:06:05.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:05 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:06:05.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:05 smithi053 ceph-mon[35979]: pgmap v185: 1 pgs: 1 remapped+peering; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:06:05.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:05 smithi063 ceph-mon[36249]: from='client.? 172.21.15.63:0/742111356' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49b39414-cc9c-4057-ae3f-cc109cd979da"}]: dispatch 2024-03-28T22:06:05.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:05 smithi063 ceph-mon[36249]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49b39414-cc9c-4057-ae3f-cc109cd979da"}]: dispatch 2024-03-28T22:06:05.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:05 smithi063 ceph-mon[36249]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "49b39414-cc9c-4057-ae3f-cc109cd979da"}]': finished 2024-03-28T22:06:05.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:05 smithi063 ceph-mon[36249]: osdmap e57: 9 total, 8 up, 9 in 2024-03-28T22:06:05.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:05 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:06:05.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:05 smithi063 ceph-mon[36249]: pgmap v185: 1 pgs: 1 remapped+peering; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:06:05.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:05 smithi016 ceph-mon[32374]: from='client.? 172.21.15.63:0/742111356' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49b39414-cc9c-4057-ae3f-cc109cd979da"}]: dispatch 2024-03-28T22:06:05.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:05 smithi016 ceph-mon[32374]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49b39414-cc9c-4057-ae3f-cc109cd979da"}]: dispatch 2024-03-28T22:06:05.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:05 smithi016 ceph-mon[32374]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "49b39414-cc9c-4057-ae3f-cc109cd979da"}]': finished 2024-03-28T22:06:05.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:05 smithi016 ceph-mon[32374]: osdmap e57: 9 total, 8 up, 9 in 2024-03-28T22:06:05.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:05 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:06:05.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:05 smithi016 ceph-mon[32374]: pgmap v185: 1 pgs: 1 remapped+peering; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:06:06.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:06 smithi053 ceph-mon[35979]: from='client.? 172.21.15.63:0/3909256184' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:06:06.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:06 smithi063 ceph-mon[36249]: from='client.? 172.21.15.63:0/3909256184' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:06:06.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:06 smithi016 ceph-mon[32374]: from='client.? 172.21.15.63:0/3909256184' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:06:07.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:07 smithi053 ceph-mon[35979]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:06:07.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:07 smithi063 ceph-mon[36249]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:06:07.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:07 smithi016 ceph-mon[32374]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:06:09.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:09 smithi063 ceph-mon[36249]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:06:09.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:09 smithi016 ceph-mon[32374]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:06:09.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:09 smithi053 ceph-mon[35979]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:06:11.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:11 smithi063 ceph-mon[36249]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:06:11.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:11 smithi016 ceph-mon[32374]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:06:11.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:11 smithi053 ceph-mon[35979]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:06:12.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:12 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-28T22:06:12.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:12 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:06:12.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:12 smithi063 ceph-mon[36249]: Deploying daemon osd.8 on smithi063 2024-03-28T22:06:12.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:12 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-28T22:06:12.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:12 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:06:12.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:12 smithi016 ceph-mon[32374]: Deploying daemon osd.8 on smithi063 2024-03-28T22:06:12.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:12 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-28T22:06:12.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:12 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:06:12.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:12 smithi053 ceph-mon[35979]: Deploying daemon osd.8 on smithi063 2024-03-28T22:06:13.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:13 smithi063 ceph-mon[36249]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:06:13.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:13 smithi016 ceph-mon[32374]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:06:13.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:13 smithi053 ceph-mon[35979]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:06:15.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:15 smithi063 ceph-mon[36249]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:06:15.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:15 smithi016 ceph-mon[32374]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:06:15.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:15 smithi053 ceph-mon[35979]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:06:17.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:17 smithi063 ceph-mon[36249]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:06:17.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:17 smithi016 ceph-mon[32374]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:06:17.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:17 smithi053 ceph-mon[35979]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:06:18.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:18 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:18.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:18 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:18.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:18 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:06:18.820 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:18 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:18.820 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:18 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:18.820 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:18 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:06:18.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:18 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:18.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:18 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:18.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:18 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:06:19.503 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:19 smithi063 ceph-mon[36249]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:06:19.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:19 smithi053 ceph-mon[35979]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:06:19.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:19 smithi016 ceph-mon[32374]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:06:21.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:21.006+0000 7f0bbf7fe700 1 -- 172.21.15.63:0/315966124 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0bd0062740 con 0x7f0bb80a1e10 2024-03-28T22:06:21.009 INFO:teuthology.orchestra.run.smithi063.stdout:Created osd(s) 8 on host 'smithi063' 2024-03-28T22:06:21.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:21.008+0000 7f0bd4e5e700 1 -- 172.21.15.63:0/315966124 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0bb80a1e10 msgr2=0x7f0bb80a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:06:21.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:21.008+0000 7f0bd4e5e700 1 --2- 172.21.15.63:0/315966124 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0bb80a1e10 0x7f0bb80a42d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4005950 tx=0x7f0bc4005cf0 comp rx=0 tx=0).stop 2024-03-28T22:06:21.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:21.008+0000 7f0bd4e5e700 1 -- 172.21.15.63:0/315966124 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0bd0102280 msgr2=0x7f0bd01984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:06:21.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:21.008+0000 7f0bd4e5e700 1 --2- 172.21.15.63:0/315966124 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0bd0102280 0x7f0bd01984b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0000c00 tx=0x7f0bc00047b0 comp rx=0 tx=0).stop 2024-03-28T22:06:21.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:21.008+0000 7f0bd4e5e700 1 -- 172.21.15.63:0/315966124 shutdown_connections 2024-03-28T22:06:21.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:21.009+0000 7f0bd4e5e700 1 --2- 172.21.15.63:0/315966124 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0bb80a1e10 0x7f0bb80a42d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:06:21.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:21.009+0000 7f0bd4e5e700 1 --2- 172.21.15.63:0/315966124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0bd0103e20 0x7f0bd019fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:06:21.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:21.009+0000 7f0bd4e5e700 1 --2- 172.21.15.63:0/315966124 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0bd0102280 0x7f0bd01984b0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:06:21.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:21.009+0000 7f0bd4e5e700 1 --2- 172.21.15.63:0/315966124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bd0103480 0x7f0bd01989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:06:21.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:21.009+0000 7f0bd4e5e700 1 -- 172.21.15.63:0/315966124 >> 172.21.15.63:0/315966124 conn(0x7f0bd00fd7e0 msgr2=0x7f0bd01067f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:06:21.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:21.009+0000 7f0bd4e5e700 1 -- 172.21.15.63:0/315966124 shutdown_connections 2024-03-28T22:06:21.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:06:21.009+0000 7f0bd4e5e700 1 -- 172.21.15.63:0/315966124 wait complete. 2024-03-28T22:06:21.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:21 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:21.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:21 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:21.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:21 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:06:21.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:21 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:06:21.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:21 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:21.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:21 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:21.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:21 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:06:21.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:21 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:21.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:21 smithi016 ceph-mon[32374]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:06:21.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:21 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:21.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:21 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:21.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:21 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:06:21.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:21 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:06:21.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:21 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:21.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:21 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:21.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:21 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:06:21.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:21 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:21.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:21 smithi053 ceph-mon[35979]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:06:21.563 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:21 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:21.563 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:21 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:21.563 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:21 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:06:21.563 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:21 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:06:21.563 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:21 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:21.563 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:21 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:21.563 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:21 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:06:21.563 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:21 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:21.563 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:21 smithi063 ceph-mon[36249]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:06:21.591 DEBUG:teuthology.orchestra.run.smithi063:osd.8> sudo journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.8.service 2024-03-28T22:06:21.593 INFO:tasks.cephadm:Deploying osd.9 on smithi016 with /dev/vg_nvme/lv_1... 2024-03-28T22:06:21.593 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-28T22:06:21.873 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:06:22.298 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:22 smithi016 ceph-mon[32374]: from='osd.8 [v2:172.21.15.63:6816/2918516794,v1:172.21.15.63:6817/2918516794]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T22:06:22.298 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:22 smithi016 ceph-mon[32374]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T22:06:22.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:22 smithi053 ceph-mon[35979]: from='osd.8 [v2:172.21.15.63:6816/2918516794,v1:172.21.15.63:6817/2918516794]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T22:06:22.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:22 smithi053 ceph-mon[35979]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T22:06:22.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:22 smithi063 ceph-mon[36249]: from='osd.8 [v2:172.21.15.63:6816/2918516794,v1:172.21.15.63:6817/2918516794]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T22:06:22.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:22 smithi063 ceph-mon[36249]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T22:06:23.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:23 smithi016 ceph-mon[32374]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-28T22:06:23.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:23 smithi016 ceph-mon[32374]: osdmap e58: 9 total, 8 up, 9 in 2024-03-28T22:06:23.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:23 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:06:23.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:23 smithi016 ceph-mon[32374]: from='osd.8 [v2:172.21.15.63:6816/2918516794,v1:172.21.15.63:6817/2918516794]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-28T22:06:23.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:23 smithi016 ceph-mon[32374]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-28T22:06:23.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:23 smithi016 ceph-mon[32374]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:06:23.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:23 smithi053 ceph-mon[35979]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-28T22:06:23.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:23 smithi053 ceph-mon[35979]: osdmap e58: 9 total, 8 up, 9 in 2024-03-28T22:06:23.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:23 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:06:23.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:23 smithi053 ceph-mon[35979]: from='osd.8 [v2:172.21.15.63:6816/2918516794,v1:172.21.15.63:6817/2918516794]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-28T22:06:23.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:23 smithi053 ceph-mon[35979]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-28T22:06:23.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:23 smithi053 ceph-mon[35979]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:06:23.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:23 smithi063 ceph-mon[36249]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-28T22:06:23.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:23 smithi063 ceph-mon[36249]: osdmap e58: 9 total, 8 up, 9 in 2024-03-28T22:06:23.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:23 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:06:23.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:23 smithi063 ceph-mon[36249]: from='osd.8 [v2:172.21.15.63:6816/2918516794,v1:172.21.15.63:6817/2918516794]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-28T22:06:23.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:23 smithi063 ceph-mon[36249]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-28T22:06:23.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:23 smithi063 ceph-mon[36249]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:06:23.683 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:06:23 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:06:23.235+0000 7fc64b92a700 -1 osd.8 0 waiting for initial osdmap 2024-03-28T22:06:23.685 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:06:23 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:06:23.248+0000 7fc647f1e700 -1 osd.8 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T22:06:23.947 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-28T22:06:23.947 INFO:teuthology.orchestra.run.smithi016.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-28T22:06:23.947 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10+0 records in 2024-03-28T22:06:23.947 INFO:teuthology.orchestra.run.smithi016.stderr:10+0 records out 2024-03-28T22:06:23.947 INFO:teuthology.orchestra.run.smithi016.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00839922 s, 1.2 GB/s 2024-03-28T22:06:23.947 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping successful for: 2024-03-28T22:06:24.392 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:24 smithi063 ceph-mon[36249]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-03-28T22:06:24.392 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:24 smithi063 ceph-mon[36249]: osdmap e59: 9 total, 8 up, 9 in 2024-03-28T22:06:24.392 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:24 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:06:24.392 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:24 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:06:24.495 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:24 smithi016 ceph-mon[32374]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-03-28T22:06:24.496 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:24 smithi016 ceph-mon[32374]: osdmap e59: 9 total, 8 up, 9 in 2024-03-28T22:06:24.496 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:24 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:06:24.496 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:24 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:06:24.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:24 smithi053 ceph-mon[35979]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-03-28T22:06:24.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:24 smithi053 ceph-mon[35979]: osdmap e59: 9 total, 8 up, 9 in 2024-03-28T22:06:24.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:06:24.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:24 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:06:24.668 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi016:vg_nvme/lv_1 2024-03-28T22:06:24.914 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:06:25.296 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:25 smithi016 ceph-mon[32374]: purged_snaps scrub starts 2024-03-28T22:06:25.296 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:25 smithi016 ceph-mon[32374]: purged_snaps scrub ok 2024-03-28T22:06:25.296 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:25 smithi016 ceph-mon[32374]: osd.8 [v2:172.21.15.63:6816/2918516794,v1:172.21.15.63:6817/2918516794] boot 2024-03-28T22:06:25.296 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:25 smithi016 ceph-mon[32374]: osdmap e60: 9 total, 9 up, 9 in 2024-03-28T22:06:25.296 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:25 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:06:25.296 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:25 smithi016 ceph-mon[32374]: Detected new or changed devices on smithi063 2024-03-28T22:06:25.296 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:25 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:25.296 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:25 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:25.296 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:25 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:06:25.296 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:25 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:06:25.296 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:25 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:06:25.297 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:25 smithi016 ceph-mon[32374]: Adjusting osd_memory_target on smithi063 to 5693M 2024-03-28T22:06:25.297 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:25 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:25.297 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:25 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:06:25.297 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:25 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:06:25.297 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:25 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:25.297 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:25 smithi016 ceph-mon[32374]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:06:25.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:25 smithi053 ceph-mon[35979]: purged_snaps scrub starts 2024-03-28T22:06:25.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:25 smithi053 ceph-mon[35979]: purged_snaps scrub ok 2024-03-28T22:06:25.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:25 smithi053 ceph-mon[35979]: osd.8 [v2:172.21.15.63:6816/2918516794,v1:172.21.15.63:6817/2918516794] boot 2024-03-28T22:06:25.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:25 smithi053 ceph-mon[35979]: osdmap e60: 9 total, 9 up, 9 in 2024-03-28T22:06:25.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:25 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:06:25.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:25 smithi053 ceph-mon[35979]: Detected new or changed devices on smithi063 2024-03-28T22:06:25.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:25 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:25.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:25 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:25.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:25 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:06:25.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:25 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:06:25.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:25 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:06:25.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:25 smithi053 ceph-mon[35979]: Adjusting osd_memory_target on smithi063 to 5693M 2024-03-28T22:06:25.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:25 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:25.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:25 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:06:25.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:25 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:06:25.549 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:25 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:25.549 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:25 smithi053 ceph-mon[35979]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:06:25.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:25 smithi063 ceph-mon[36249]: purged_snaps scrub starts 2024-03-28T22:06:25.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:25 smithi063 ceph-mon[36249]: purged_snaps scrub ok 2024-03-28T22:06:25.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:25 smithi063 ceph-mon[36249]: osd.8 [v2:172.21.15.63:6816/2918516794,v1:172.21.15.63:6817/2918516794] boot 2024-03-28T22:06:25.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:25 smithi063 ceph-mon[36249]: osdmap e60: 9 total, 9 up, 9 in 2024-03-28T22:06:25.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:25 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:06:25.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:25 smithi063 ceph-mon[36249]: Detected new or changed devices on smithi063 2024-03-28T22:06:25.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:25 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:25.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:25 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:25.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:25 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:06:25.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:25 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:06:25.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:25 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:06:25.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:25 smithi063 ceph-mon[36249]: Adjusting osd_memory_target on smithi063 to 5693M 2024-03-28T22:06:25.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:25 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:25.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:25 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:06:25.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:25 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:06:25.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:25 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:25.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:25 smithi063 ceph-mon[36249]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:06:26.397 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:26 smithi016 ceph-mon[32374]: osdmap e61: 9 total, 9 up, 9 in 2024-03-28T22:06:26.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:26 smithi053 ceph-mon[35979]: osdmap e61: 9 total, 9 up, 9 in 2024-03-28T22:06:26.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.596+0000 7f2b45e2f700 1 -- 172.21.15.16:0/2873915883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b40105630 msgr2=0x7f2b40107a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:06:26.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.596+0000 7f2b45e2f700 1 --2- 172.21.15.16:0/2873915883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b40105630 0x7f2b40107a20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2b34009990 tx=0x7f2b34009ca0 comp rx=0 tx=0).stop 2024-03-28T22:06:26.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.596+0000 7f2b45e2f700 1 -- 172.21.15.16:0/2873915883 shutdown_connections 2024-03-28T22:06:26.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.596+0000 7f2b45e2f700 1 --2- 172.21.15.16:0/2873915883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b40107f60 0x7f2b4010a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:06:26.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.596+0000 7f2b45e2f700 1 --2- 172.21.15.16:0/2873915883 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b40102d00 0x7f2b401050f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:06:26.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.596+0000 7f2b45e2f700 1 --2- 172.21.15.16:0/2873915883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b40105630 0x7f2b40107a20 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:06:26.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.596+0000 7f2b45e2f700 1 -- 172.21.15.16:0/2873915883 >> 172.21.15.16:0/2873915883 conn(0x7f2b400fc710 msgr2=0x7f2b400feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:06:26.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.596+0000 7f2b45e2f700 1 -- 172.21.15.16:0/2873915883 shutdown_connections 2024-03-28T22:06:26.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.596+0000 7f2b45e2f700 1 -- 172.21.15.16:0/2873915883 wait complete. 2024-03-28T22:06:26.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.597+0000 7f2b45e2f700 1 Processor -- start 2024-03-28T22:06:26.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.597+0000 7f2b45e2f700 1 -- start start 2024-03-28T22:06:26.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.597+0000 7f2b45e2f700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b40102d00 0x7f2b40102660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:06:26.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.597+0000 7f2b45e2f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b40105630 0x7f2b40100cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:06:26.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.597+0000 7f2b45e2f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b40107f60 0x7f2b401011f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:06:26.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.597+0000 7f2b45e2f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2b4010de80 con 0x7f2b40105630 2024-03-28T22:06:26.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.597+0000 7f2b45e2f700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f2b4010dd00 con 0x7f2b40107f60 2024-03-28T22:06:26.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.597+0000 7f2b45e2f700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2b4010e000 con 0x7f2b40102d00 2024-03-28T22:06:26.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.597+0000 7f2b3effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b40105630 0x7f2b40100cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:06:26.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.597+0000 7f2b3effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b40105630 0x7f2b40100cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60974/0 (socket says 172.21.15.16:60974) 2024-03-28T22:06:26.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.597+0000 7f2b3effd700 1 -- 172.21.15.16:0/337506925 learned_addr learned my addr 172.21.15.16:0/337506925 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:06:26.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.597+0000 7f2b3ffff700 1 --2- 172.21.15.16:0/337506925 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b40107f60 0x7f2b401011f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:06:26.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.597+0000 7f2b3f7fe700 1 --2- 172.21.15.16:0/337506925 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b40102d00 0x7f2b40102660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:06:26.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.597+0000 7f2b3effd700 1 -- 172.21.15.16:0/337506925 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b40102d00 msgr2=0x7f2b40102660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:06:26.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.597+0000 7f2b3effd700 1 --2- 172.21.15.16:0/337506925 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b40102d00 0x7f2b40102660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:06:26.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.597+0000 7f2b3effd700 1 -- 172.21.15.16:0/337506925 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b40107f60 msgr2=0x7f2b401011f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:06:26.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.597+0000 7f2b3effd700 1 --2- 172.21.15.16:0/337506925 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b40107f60 0x7f2b401011f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:06:26.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.597+0000 7f2b3effd700 1 -- 172.21.15.16:0/337506925 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b40101aa0 con 0x7f2b40105630 2024-03-28T22:06:26.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.598+0000 7f2b3effd700 1 --2- 172.21.15.16:0/337506925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b40105630 0x7f2b40100cb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2b3400b5c0 tx=0x7f2b34009f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:06:26.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.598+0000 7f2b3cff9700 1 -- 172.21.15.16:0/337506925 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b34026070 con 0x7f2b40105630 2024-03-28T22:06:26.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.598+0000 7f2b45e2f700 1 -- 172.21.15.16:0/337506925 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b401a25b0 con 0x7f2b40105630 2024-03-28T22:06:26.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.598+0000 7f2b3cff9700 1 -- 172.21.15.16:0/337506925 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b34004150 con 0x7f2b40105630 2024-03-28T22:06:26.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.598+0000 7f2b3cff9700 1 -- 172.21.15.16:0/337506925 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b34004b20 con 0x7f2b40105630 2024-03-28T22:06:26.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.598+0000 7f2b45e2f700 1 -- 172.21.15.16:0/337506925 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b401a2a00 con 0x7f2b40105630 2024-03-28T22:06:26.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.599+0000 7f2b45e2f700 1 -- 172.21.15.16:0/337506925 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b40068f00 con 0x7f2b40105630 2024-03-28T22:06:26.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.600+0000 7f2b3cff9700 1 -- 172.21.15.16:0/337506925 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f2b34029070 con 0x7f2b40105630 2024-03-28T22:06:26.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.600+0000 7f2b3cff9700 1 --2- 172.21.15.16:0/337506925 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f2b280a1e10 0x7f2b280a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:06:26.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.601+0000 7f2b3f7fe700 1 --2- 172.21.15.16:0/337506925 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f2b280a1e10 0x7f2b280a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:06:26.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.601+0000 7f2b3cff9700 1 -- 172.21.15.16:0/337506925 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5858+0+0 (secure 0 0 0) 0x7f2b340cdf20 con 0x7f2b40105630 2024-03-28T22:06:26.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.601+0000 7f2b3f7fe700 1 --2- 172.21.15.16:0/337506925 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f2b280a1e10 0x7f2b280a42d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2b30005950 tx=0x7f2b3000a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:06:26.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.605+0000 7f2b3cff9700 1 -- 172.21.15.16:0/337506925 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2b3405d280 con 0x7f2b40105630 2024-03-28T22:06:26.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:26 smithi063 ceph-mon[36249]: osdmap e61: 9 total, 9 up, 9 in 2024-03-28T22:06:26.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:26.821+0000 7f2b45e2f700 1 -- 172.21.15.16:0/337506925 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f2b40062680 con 0x7f2b280a1e10 2024-03-28T22:06:27.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:27 smithi053 ceph-mon[35979]: osdmap e62: 9 total, 9 up, 9 in 2024-03-28T22:06:27.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:27 smithi053 ceph-mon[35979]: from='client.14412 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:06:27.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:27 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:06:27.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:27 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:06:27.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:27 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:06:27.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:27 smithi053 ceph-mon[35979]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:06:27.553 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:27 smithi016 ceph-mon[32374]: osdmap e62: 9 total, 9 up, 9 in 2024-03-28T22:06:27.554 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:27 smithi016 ceph-mon[32374]: from='client.14412 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:06:27.554 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:27 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:06:27.554 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:27 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:06:27.554 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:27 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:06:27.554 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:27 smithi016 ceph-mon[32374]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:06:27.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:27 smithi063 ceph-mon[36249]: osdmap e62: 9 total, 9 up, 9 in 2024-03-28T22:06:27.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:27 smithi063 ceph-mon[36249]: from='client.14412 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:06:27.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:27 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:06:27.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:27 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:06:27.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:27 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:06:27.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:27 smithi063 ceph-mon[36249]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:06:28.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:28 smithi053 ceph-mon[35979]: osdmap e63: 9 total, 9 up, 9 in 2024-03-28T22:06:28.551 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:28 smithi016 ceph-mon[32374]: osdmap e63: 9 total, 9 up, 9 in 2024-03-28T22:06:28.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:28 smithi063 ceph-mon[36249]: osdmap e63: 9 total, 9 up, 9 in 2024-03-28T22:06:29.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:29 smithi053 ceph-mon[35979]: osdmap e64: 9 total, 9 up, 9 in 2024-03-28T22:06:29.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:29 smithi053 ceph-mon[35979]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:06:29.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:29 smithi063 ceph-mon[36249]: osdmap e64: 9 total, 9 up, 9 in 2024-03-28T22:06:29.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:29 smithi063 ceph-mon[36249]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:06:29.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:29 smithi016 ceph-mon[32374]: osdmap e64: 9 total, 9 up, 9 in 2024-03-28T22:06:29.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:29 smithi016 ceph-mon[32374]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:06:30.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:30 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/707980417' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c873e51-b783-4eb6-83a3-fee31722d60b"}]: dispatch 2024-03-28T22:06:30.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:30 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/707980417' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c873e51-b783-4eb6-83a3-fee31722d60b"}]': finished 2024-03-28T22:06:30.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:30 smithi053 ceph-mon[35979]: osdmap e65: 10 total, 9 up, 10 in 2024-03-28T22:06:30.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:30 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T22:06:30.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:30 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3045899914' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:06:30.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:30 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/707980417' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c873e51-b783-4eb6-83a3-fee31722d60b"}]: dispatch 2024-03-28T22:06:30.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:30 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/707980417' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c873e51-b783-4eb6-83a3-fee31722d60b"}]': finished 2024-03-28T22:06:30.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:30 smithi063 ceph-mon[36249]: osdmap e65: 10 total, 9 up, 10 in 2024-03-28T22:06:30.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:30 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T22:06:30.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:30 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/3045899914' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:06:30.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:30 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/707980417' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c873e51-b783-4eb6-83a3-fee31722d60b"}]: dispatch 2024-03-28T22:06:30.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:30 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/707980417' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c873e51-b783-4eb6-83a3-fee31722d60b"}]': finished 2024-03-28T22:06:30.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:30 smithi016 ceph-mon[32374]: osdmap e65: 10 total, 9 up, 10 in 2024-03-28T22:06:30.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:30 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T22:06:30.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:30 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3045899914' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:06:31.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:31 smithi053 ceph-mon[35979]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-03-28T22:06:31.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:31 smithi063 ceph-mon[36249]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-03-28T22:06:31.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:31 smithi016 ceph-mon[32374]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-03-28T22:06:33.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:33 smithi016 ceph-mon[32374]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T22:06:33.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:33 smithi063 ceph-mon[36249]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T22:06:33.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:33 smithi053 ceph-mon[35979]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T22:06:35.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:35 smithi063 ceph-mon[36249]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-28T22:06:35.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:35 smithi016 ceph-mon[32374]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-28T22:06:35.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:35 smithi053 ceph-mon[35979]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-28T22:06:37.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:37 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-28T22:06:37.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:37 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:06:37.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:37 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-28T22:06:37.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:37 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:06:37.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:37 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-28T22:06:37.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:37 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:06:38.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:38 smithi016 ceph-mon[32374]: Deploying daemon osd.9 on smithi016 2024-03-28T22:06:38.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:38 smithi016 ceph-mon[32374]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-03-28T22:06:38.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:38 smithi063 ceph-mon[36249]: Deploying daemon osd.9 on smithi016 2024-03-28T22:06:38.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:38 smithi063 ceph-mon[36249]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-03-28T22:06:38.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:38 smithi053 ceph-mon[35979]: Deploying daemon osd.9 on smithi016 2024-03-28T22:06:38.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:38 smithi053 ceph-mon[35979]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-03-28T22:06:39.400 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:39 smithi016 ceph-mon[32374]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T22:06:39.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:39 smithi063 ceph-mon[36249]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T22:06:39.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:39 smithi053 ceph-mon[35979]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T22:06:41.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:41 smithi016 ceph-mon[32374]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-03-28T22:06:41.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:41 smithi063 ceph-mon[36249]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-03-28T22:06:41.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:41 smithi053 ceph-mon[35979]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-03-28T22:06:43.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:43 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:43.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:43 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:43.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:43 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:06:43.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:43 smithi053 ceph-mon[35979]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:06:43.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:43 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:43.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:43 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:43.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:43 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:06:43.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:43 smithi063 ceph-mon[36249]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:06:43.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:43 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:43.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:43 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:43.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:43 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:06:43.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:43 smithi016 ceph-mon[32374]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:06:45.228 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:45 smithi016 ceph-mon[32374]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:06:45.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:45 smithi063 ceph-mon[36249]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:06:45.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:45 smithi053 ceph-mon[35979]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:06:46.058 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 9 on host 'smithi016' 2024-03-28T22:06:46.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:46.055+0000 7f2b3cff9700 1 -- 172.21.15.16:0/337506925 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2b40062680 con 0x7f2b280a1e10 2024-03-28T22:06:46.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:46.057+0000 7f2b45e2f700 1 -- 172.21.15.16:0/337506925 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f2b280a1e10 msgr2=0x7f2b280a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:06:46.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:46.057+0000 7f2b45e2f700 1 --2- 172.21.15.16:0/337506925 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f2b280a1e10 0x7f2b280a42d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2b30005950 tx=0x7f2b3000a300 comp rx=0 tx=0).stop 2024-03-28T22:06:46.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:46.058+0000 7f2b45e2f700 1 -- 172.21.15.16:0/337506925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b40105630 msgr2=0x7f2b40100cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:06:46.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:46.058+0000 7f2b45e2f700 1 --2- 172.21.15.16:0/337506925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b40105630 0x7f2b40100cb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2b3400b5c0 tx=0x7f2b34009f50 comp rx=0 tx=0).stop 2024-03-28T22:06:46.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:46.058+0000 7f2b45e2f700 1 -- 172.21.15.16:0/337506925 shutdown_connections 2024-03-28T22:06:46.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:46.058+0000 7f2b45e2f700 1 --2- 172.21.15.16:0/337506925 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f2b280a1e10 0x7f2b280a42d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:06:46.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:46.058+0000 7f2b45e2f700 1 --2- 172.21.15.16:0/337506925 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b40107f60 0x7f2b401011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:06:46.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:46.058+0000 7f2b45e2f700 1 --2- 172.21.15.16:0/337506925 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b40102d00 0x7f2b40102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:06:46.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:46.058+0000 7f2b45e2f700 1 --2- 172.21.15.16:0/337506925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b40105630 0x7f2b40100cb0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:06:46.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:46.058+0000 7f2b45e2f700 1 -- 172.21.15.16:0/337506925 >> 172.21.15.16:0/337506925 conn(0x7f2b400fc710 msgr2=0x7f2b40103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:06:46.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:46.058+0000 7f2b45e2f700 1 -- 172.21.15.16:0/337506925 shutdown_connections 2024-03-28T22:06:46.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:06:46.058+0000 7f2b45e2f700 1 -- 172.21.15.16:0/337506925 wait complete. 2024-03-28T22:06:46.629 DEBUG:teuthology.orchestra.run.smithi016:osd.9> sudo journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.9.service 2024-03-28T22:06:46.632 INFO:tasks.cephadm:Deploying osd.10 on smithi053 with /dev/vg_nvme/lv_1... 2024-03-28T22:06:46.632 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-28T22:06:46.892 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T22:06:47.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:47 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:47.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:47 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:47.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:47 smithi053 ceph-mon[35979]: from='osd.9 [v2:172.21.15.16:6826/1585624367,v1:172.21.15.16:6827/1585624367]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-28T22:06:47.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:47 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:47.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:47 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:47.374 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:47 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:47.374 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:47 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:47.374 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:47 smithi016 ceph-mon[32374]: from='osd.9 [v2:172.21.15.16:6826/1585624367,v1:172.21.15.16:6827/1585624367]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-28T22:06:47.374 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:47 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:47.374 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:47 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:47.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:47 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:47.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:47 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:47.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:47 smithi063 ceph-mon[36249]: from='osd.9 [v2:172.21.15.16:6826/1585624367,v1:172.21.15.16:6827/1585624367]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-28T22:06:47.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:47 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:47.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:47 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:48.212 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:48 smithi053 ceph-mon[35979]: from='osd.9 [v2:172.21.15.16:6826/1585624367,v1:172.21.15.16:6827/1585624367]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-28T22:06:48.212 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:48 smithi053 ceph-mon[35979]: osdmap e66: 10 total, 9 up, 10 in 2024-03-28T22:06:48.212 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:48 smithi053 ceph-mon[35979]: from='osd.9 [v2:172.21.15.16:6826/1585624367,v1:172.21.15.16:6827/1585624367]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-03-28T22:06:48.212 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:48 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T22:06:48.212 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:48 smithi053 ceph-mon[35979]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:06:48.318 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:48 smithi016 ceph-mon[32374]: from='osd.9 [v2:172.21.15.16:6826/1585624367,v1:172.21.15.16:6827/1585624367]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-28T22:06:48.318 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:48 smithi016 ceph-mon[32374]: osdmap e66: 10 total, 9 up, 10 in 2024-03-28T22:06:48.318 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:48 smithi016 ceph-mon[32374]: from='osd.9 [v2:172.21.15.16:6826/1585624367,v1:172.21.15.16:6827/1585624367]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-03-28T22:06:48.318 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:48 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T22:06:48.319 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:48 smithi016 ceph-mon[32374]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:06:48.319 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:06:48 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:06:48.038+0000 7ff80078c700 -1 osd.9 0 waiting for initial osdmap 2024-03-28T22:06:48.319 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:06:48 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:06:48.053+0000 7ff7fcd80700 -1 osd.9 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T22:06:48.434 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:48 smithi063 ceph-mon[36249]: from='osd.9 [v2:172.21.15.16:6826/1585624367,v1:172.21.15.16:6827/1585624367]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-28T22:06:48.434 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:48 smithi063 ceph-mon[36249]: osdmap e66: 10 total, 9 up, 10 in 2024-03-28T22:06:48.435 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:48 smithi063 ceph-mon[36249]: from='osd.9 [v2:172.21.15.16:6826/1585624367,v1:172.21.15.16:6827/1585624367]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-03-28T22:06:48.435 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:48 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T22:06:48.435 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:48 smithi063 ceph-mon[36249]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:06:48.778 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-28T22:06:48.778 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-28T22:06:48.778 INFO:teuthology.orchestra.run.smithi053.stderr: stderr: 10+0 records in 2024-03-28T22:06:48.778 INFO:teuthology.orchestra.run.smithi053.stderr:10+0 records out 2024-03-28T22:06:48.778 INFO:teuthology.orchestra.run.smithi053.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00825609 s, 1.3 GB/s 2024-03-28T22:06:48.778 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping successful for: 2024-03-28T22:06:49.146 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:49 smithi053 ceph-mon[35979]: from='osd.9 [v2:172.21.15.16:6826/1585624367,v1:172.21.15.16:6827/1585624367]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-03-28T22:06:49.146 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:49 smithi053 ceph-mon[35979]: osdmap e67: 10 total, 9 up, 10 in 2024-03-28T22:06:49.146 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:49 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T22:06:49.146 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:49 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T22:06:49.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:49 smithi063 ceph-mon[36249]: from='osd.9 [v2:172.21.15.16:6826/1585624367,v1:172.21.15.16:6827/1585624367]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-03-28T22:06:49.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:49 smithi063 ceph-mon[36249]: osdmap e67: 10 total, 9 up, 10 in 2024-03-28T22:06:49.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:49 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T22:06:49.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:49 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T22:06:49.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:49 smithi016 ceph-mon[32374]: from='osd.9 [v2:172.21.15.16:6826/1585624367,v1:172.21.15.16:6827/1585624367]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-03-28T22:06:49.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:49 smithi016 ceph-mon[32374]: osdmap e67: 10 total, 9 up, 10 in 2024-03-28T22:06:49.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:49 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T22:06:49.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:49 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T22:06:49.522 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi053:vg_nvme/lv_1 2024-03-28T22:06:49.778 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T22:06:50.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:50 smithi016 ceph-mon[32374]: purged_snaps scrub starts 2024-03-28T22:06:50.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:50 smithi016 ceph-mon[32374]: purged_snaps scrub ok 2024-03-28T22:06:50.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:50 smithi016 ceph-mon[32374]: osd.9 [v2:172.21.15.16:6826/1585624367,v1:172.21.15.16:6827/1585624367] boot 2024-03-28T22:06:50.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:50 smithi016 ceph-mon[32374]: osdmap e68: 10 total, 10 up, 10 in 2024-03-28T22:06:50.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:50 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T22:06:50.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:50 smithi016 ceph-mon[32374]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:06:50.199 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:50 smithi053 ceph-mon[35979]: purged_snaps scrub starts 2024-03-28T22:06:50.199 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:50 smithi053 ceph-mon[35979]: purged_snaps scrub ok 2024-03-28T22:06:50.199 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:50 smithi053 ceph-mon[35979]: osd.9 [v2:172.21.15.16:6826/1585624367,v1:172.21.15.16:6827/1585624367] boot 2024-03-28T22:06:50.199 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:50 smithi053 ceph-mon[35979]: osdmap e68: 10 total, 10 up, 10 in 2024-03-28T22:06:50.199 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:50 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T22:06:50.199 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:50 smithi053 ceph-mon[35979]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:06:50.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:50 smithi063 ceph-mon[36249]: purged_snaps scrub starts 2024-03-28T22:06:50.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:50 smithi063 ceph-mon[36249]: purged_snaps scrub ok 2024-03-28T22:06:50.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:50 smithi063 ceph-mon[36249]: osd.9 [v2:172.21.15.16:6826/1585624367,v1:172.21.15.16:6827/1585624367] boot 2024-03-28T22:06:50.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:50 smithi063 ceph-mon[36249]: osdmap e68: 10 total, 10 up, 10 in 2024-03-28T22:06:50.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:50 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T22:06:50.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:50 smithi063 ceph-mon[36249]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:06:51.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:51 smithi053 ceph-mon[35979]: osdmap e69: 10 total, 10 up, 10 in 2024-03-28T22:06:51.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:51 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:51.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:51 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:51.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:51 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:06:51.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:51 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:06:51.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:51 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:06:51.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:51 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:06:51.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:51 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:51.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:51 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:06:51.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:51 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:06:51.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:51 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:51.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:51 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:06:51.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:51 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:06:51.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:51 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:06:51.299 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:51 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:51.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:51 smithi063 ceph-mon[36249]: osdmap e69: 10 total, 10 up, 10 in 2024-03-28T22:06:51.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:51 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:51.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:51 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:51.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:51 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:06:51.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:51 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:06:51.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:51 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:06:51.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:51 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:06:51.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:51 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:51.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:51 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:06:51.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:51 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:06:51.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:51 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:51.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:51 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:06:51.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:51 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:06:51.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:51 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:06:51.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:51 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:51.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:51 smithi016 ceph-mon[32374]: osdmap e69: 10 total, 10 up, 10 in 2024-03-28T22:06:51.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:51 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:51.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:51 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:51.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:51 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:06:51.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:51 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:06:51.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:51 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:06:51.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:51 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:06:51.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:51 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:51.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:51 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:06:51.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:51 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:06:51.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:51 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:51.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:51 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:06:51.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:51 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:06:51.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:51 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:06:51.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:51 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:06:51.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.439+0000 7f6084e7a700 1 -- 172.21.15.53:0/3580544299 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6080103e10 msgr2=0x7f608010a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:06:51.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.439+0000 7f6084e7a700 1 --2- 172.21.15.53:0/3580544299 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6080103e10 0x7f608010a6c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f607800b0d0 tx=0x7f607800b3e0 comp rx=0 tx=0).stop 2024-03-28T22:06:51.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.439+0000 7f6084e7a700 1 -- 172.21.15.53:0/3580544299 shutdown_connections 2024-03-28T22:06:51.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.439+0000 7f6084e7a700 1 --2- 172.21.15.53:0/3580544299 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6080103e10 0x7f608010a6c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:06:51.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.439+0000 7f6084e7a700 1 --2- 172.21.15.53:0/3580544299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6080102270 0x7f6080102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:06:51.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.439+0000 7f6084e7a700 1 --2- 172.21.15.53:0/3580544299 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6080103470 0x7f60801038d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:06:51.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.439+0000 7f6084e7a700 1 -- 172.21.15.53:0/3580544299 >> 172.21.15.53:0/3580544299 conn(0x7f60800fd850 msgr2=0x7f60800ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:06:51.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.439+0000 7f6084e7a700 1 -- 172.21.15.53:0/3580544299 shutdown_connections 2024-03-28T22:06:51.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.439+0000 7f6084e7a700 1 -- 172.21.15.53:0/3580544299 wait complete. 2024-03-28T22:06:51.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.440+0000 7f6084e7a700 1 Processor -- start 2024-03-28T22:06:51.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.440+0000 7f6084e7a700 1 -- start start 2024-03-28T22:06:51.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.440+0000 7f6084e7a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6080102270 0x7f6080198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:06:51.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.440+0000 7f6084e7a700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6080103470 0x7f60801988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:06:51.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.440+0000 7f6084e7a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6080103e10 0x7f608019f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:06:51.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.440+0000 7f6084e7a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f608010de30 con 0x7f6080102270 2024-03-28T22:06:51.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.440+0000 7f6084e7a700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f608010dcb0 con 0x7f6080103e10 2024-03-28T22:06:51.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.440+0000 7f6084e7a700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f608010dfb0 con 0x7f6080103470 2024-03-28T22:06:51.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.440+0000 7f607ed9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6080103e10 0x7f608019f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:06:51.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.440+0000 7f607dd9b700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6080103470 0x7f60801988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:06:51.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.440+0000 7f607ed9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6080103e10 0x7f608019f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45892/0 (socket says 172.21.15.53:45892) 2024-03-28T22:06:51.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.440+0000 7f607ed9d700 1 -- 172.21.15.53:0/1328462293 learned_addr learned my addr 172.21.15.53:0/1328462293 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-03-28T22:06:51.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.440+0000 7f607e59c700 1 --2- 172.21.15.53:0/1328462293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6080102270 0x7f6080198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:06:51.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.440+0000 7f607ed9d700 1 -- 172.21.15.53:0/1328462293 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6080103470 msgr2=0x7f60801988a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:06:51.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.440+0000 7f607ed9d700 1 --2- 172.21.15.53:0/1328462293 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6080103470 0x7f60801988a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:06:51.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.440+0000 7f607ed9d700 1 -- 172.21.15.53:0/1328462293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6080102270 msgr2=0x7f6080198360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:06:51.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.440+0000 7f607ed9d700 1 --2- 172.21.15.53:0/1328462293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6080102270 0x7f6080198360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:06:51.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.440+0000 7f607ed9d700 1 -- 172.21.15.53:0/1328462293 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f608019ffc0 con 0x7f6080103e10 2024-03-28T22:06:51.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.440+0000 7f607ed9d700 1 --2- 172.21.15.53:0/1328462293 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6080103e10 0x7f608019f920 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f607800d010 tx=0x7f6078007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:06:51.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.441+0000 7f606f7fe700 1 -- 172.21.15.53:0/1328462293 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6078003c00 con 0x7f6080103e10 2024-03-28T22:06:51.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.441+0000 7f606f7fe700 1 -- 172.21.15.53:0/1328462293 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6078003da0 con 0x7f6080103e10 2024-03-28T22:06:51.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.441+0000 7f6084e7a700 1 -- 172.21.15.53:0/1328462293 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60801a0250 con 0x7f6080103e10 2024-03-28T22:06:51.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.441+0000 7f606f7fe700 1 -- 172.21.15.53:0/1328462293 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f607801b990 con 0x7f6080103e10 2024-03-28T22:06:51.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.441+0000 7f6084e7a700 1 -- 172.21.15.53:0/1328462293 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60801a06a0 con 0x7f6080103e10 2024-03-28T22:06:51.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.442+0000 7f6084e7a700 1 -- 172.21.15.53:0/1328462293 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6080068f00 con 0x7f6080103e10 2024-03-28T22:06:51.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.443+0000 7f606f7fe700 1 -- 172.21.15.53:0/1328462293 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f6078005670 con 0x7f6080103e10 2024-03-28T22:06:51.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.443+0000 7f606f7fe700 1 --2- 172.21.15.53:0/1328462293 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f60680a1e10 0x7f60680a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:06:51.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.444+0000 7f606f7fe700 1 -- 172.21.15.53:0/1328462293 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6261+0+0 (secure 0 0 0) 0x7f607800f020 con 0x7f6080103e10 2024-03-28T22:06:51.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.444+0000 7f607e59c700 1 --2- 172.21.15.53:0/1328462293 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f60680a1e10 0x7f60680a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:06:51.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.444+0000 7f607e59c700 1 --2- 172.21.15.53:0/1328462293 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f60680a1e10 0x7f60680a42d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f60801032d0 tx=0x7f6070009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:06:51.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.448+0000 7f606f7fe700 1 -- 172.21.15.53:0/1328462293 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f607809c710 con 0x7f6080103e10 2024-03-28T22:06:51.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:06:51.663+0000 7f6084e7a700 1 -- 172.21.15.53:0/1328462293 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f6080062680 con 0x7f60680a1e10 2024-03-28T22:06:52.323 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:52 smithi053 ceph-mon[35979]: Detected new or changed devices on smithi016 2024-03-28T22:06:52.323 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:52 smithi053 ceph-mon[35979]: Adjusting osd_memory_target on smithi016 to 4273M 2024-03-28T22:06:52.323 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:52 smithi053 ceph-mon[35979]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:06:52.323 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:52 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:06:52.323 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:52 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:06:52.323 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:52 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:06:52.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:52 smithi063 ceph-mon[36249]: Detected new or changed devices on smithi016 2024-03-28T22:06:52.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:52 smithi063 ceph-mon[36249]: Adjusting osd_memory_target on smithi016 to 4273M 2024-03-28T22:06:52.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:52 smithi063 ceph-mon[36249]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:06:52.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:52 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:06:52.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:52 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:06:52.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:52 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:06:52.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:52 smithi016 ceph-mon[32374]: Detected new or changed devices on smithi016 2024-03-28T22:06:52.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:52 smithi016 ceph-mon[32374]: Adjusting osd_memory_target on smithi016 to 4273M 2024-03-28T22:06:52.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:52 smithi016 ceph-mon[32374]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:06:52.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:52 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:06:52.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:52 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:06:52.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:52 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:06:53.327 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:53 smithi053 ceph-mon[35979]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:06:53.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:53 smithi063 ceph-mon[36249]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:06:53.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:53 smithi016 ceph-mon[32374]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:06:54.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:54 smithi063 ceph-mon[36249]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:06:54.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:54 smithi016 ceph-mon[32374]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:06:54.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:54 smithi053 ceph-mon[35979]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:06:55.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:55 smithi063 ceph-mon[36249]: from='client.? 172.21.15.53:0/3324662808' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "085a78bf-e2ad-481a-85a3-71d1fc549a0c"}]: dispatch 2024-03-28T22:06:55.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:55 smithi063 ceph-mon[36249]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "085a78bf-e2ad-481a-85a3-71d1fc549a0c"}]: dispatch 2024-03-28T22:06:55.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:55 smithi063 ceph-mon[36249]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "085a78bf-e2ad-481a-85a3-71d1fc549a0c"}]': finished 2024-03-28T22:06:55.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:55 smithi063 ceph-mon[36249]: osdmap e70: 11 total, 10 up, 11 in 2024-03-28T22:06:55.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:55 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T22:06:55.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:55 smithi063 ceph-mon[36249]: from='client.? 172.21.15.53:0/3081764996' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:06:55.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:55 smithi016 ceph-mon[32374]: from='client.? 172.21.15.53:0/3324662808' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "085a78bf-e2ad-481a-85a3-71d1fc549a0c"}]: dispatch 2024-03-28T22:06:55.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:55 smithi016 ceph-mon[32374]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "085a78bf-e2ad-481a-85a3-71d1fc549a0c"}]: dispatch 2024-03-28T22:06:55.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:55 smithi016 ceph-mon[32374]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "085a78bf-e2ad-481a-85a3-71d1fc549a0c"}]': finished 2024-03-28T22:06:55.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:55 smithi016 ceph-mon[32374]: osdmap e70: 11 total, 10 up, 11 in 2024-03-28T22:06:55.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:55 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T22:06:55.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:55 smithi016 ceph-mon[32374]: from='client.? 172.21.15.53:0/3081764996' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:06:55.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:55 smithi053 ceph-mon[35979]: from='client.? 172.21.15.53:0/3324662808' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "085a78bf-e2ad-481a-85a3-71d1fc549a0c"}]: dispatch 2024-03-28T22:06:55.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:55 smithi053 ceph-mon[35979]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "085a78bf-e2ad-481a-85a3-71d1fc549a0c"}]: dispatch 2024-03-28T22:06:55.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:55 smithi053 ceph-mon[35979]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "085a78bf-e2ad-481a-85a3-71d1fc549a0c"}]': finished 2024-03-28T22:06:55.549 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:55 smithi053 ceph-mon[35979]: osdmap e70: 11 total, 10 up, 11 in 2024-03-28T22:06:55.549 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:55 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T22:06:55.549 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:55 smithi053 ceph-mon[35979]: from='client.? 172.21.15.53:0/3081764996' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:06:56.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:56 smithi063 ceph-mon[36249]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:06:56.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:56 smithi016 ceph-mon[32374]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:06:56.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:56 smithi053 ceph-mon[35979]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:06:58.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:06:58 smithi053 ceph-mon[35979]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:06:58.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:06:58 smithi063 ceph-mon[36249]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:06:58.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:06:58 smithi016 ceph-mon[32374]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:07:00.340 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:00 smithi053 ceph-mon[35979]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:07:00.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:00 smithi063 ceph-mon[36249]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:07:00.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:00 smithi016 ceph-mon[32374]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:07:02.390 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:02 smithi053 ceph-mon[35979]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:07:02.390 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:02 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-28T22:07:02.390 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:02 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:07:02.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:02 smithi063 ceph-mon[36249]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:07:02.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:02 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-28T22:07:02.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:02 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:07:02.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:02 smithi016 ceph-mon[32374]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:07:02.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:02 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-28T22:07:02.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:02 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:07:03.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:03 smithi053 ceph-mon[35979]: Deploying daemon osd.10 on smithi053 2024-03-28T22:07:03.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:03 smithi063 ceph-mon[36249]: Deploying daemon osd.10 on smithi053 2024-03-28T22:07:03.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:03 smithi016 ceph-mon[32374]: Deploying daemon osd.10 on smithi053 2024-03-28T22:07:04.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:04 smithi053 ceph-mon[35979]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:07:04.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:04 smithi063 ceph-mon[36249]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:07:04.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:04 smithi016 ceph-mon[32374]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:07:05.357 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:05 smithi053 ceph-mon[35979]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:07:05.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:05 smithi063 ceph-mon[36249]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:07:05.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:05 smithi016 ceph-mon[32374]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:07:07.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:07 smithi063 ceph-mon[36249]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:07:07.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:07 smithi016 ceph-mon[32374]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:07:07.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:07 smithi053 ceph-mon[35979]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:07:08.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:08 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:08.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:08 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:07:08.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:08 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:08.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:08 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:08.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:08 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:07:08.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:08 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:08.688 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:08 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:08.688 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:08 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:07:08.688 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:08 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:09.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:09 smithi053 ceph-mon[35979]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:07:09.607 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:09 smithi016 ceph-mon[32374]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:07:09.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:09 smithi063 ceph-mon[36249]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:07:10.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:07:10.777+0000 7f606f7fe700 1 -- 172.21.15.53:0/1328462293 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f6080062680 con 0x7f60680a1e10 2024-03-28T22:07:10.780 INFO:teuthology.orchestra.run.smithi053.stdout:Created osd(s) 10 on host 'smithi053' 2024-03-28T22:07:10.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:07:10.780+0000 7f6084e7a700 1 -- 172.21.15.53:0/1328462293 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f60680a1e10 msgr2=0x7f60680a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:10.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:07:10.780+0000 7f6084e7a700 1 --2- 172.21.15.53:0/1328462293 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f60680a1e10 0x7f60680a42d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f60801032d0 tx=0x7f6070009260 comp rx=0 tx=0).stop 2024-03-28T22:07:10.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:07:10.780+0000 7f6084e7a700 1 -- 172.21.15.53:0/1328462293 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6080103e10 msgr2=0x7f608019f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:10.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:07:10.780+0000 7f6084e7a700 1 --2- 172.21.15.53:0/1328462293 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6080103e10 0x7f608019f920 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f607800d010 tx=0x7f6078007a20 comp rx=0 tx=0).stop 2024-03-28T22:07:10.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:07:10.780+0000 7f6084e7a700 1 -- 172.21.15.53:0/1328462293 shutdown_connections 2024-03-28T22:07:10.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:07:10.780+0000 7f6084e7a700 1 --2- 172.21.15.53:0/1328462293 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f60680a1e10 0x7f60680a42d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:10.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:07:10.780+0000 7f6084e7a700 1 --2- 172.21.15.53:0/1328462293 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6080103e10 0x7f608019f920 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:10.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:07:10.780+0000 7f6084e7a700 1 --2- 172.21.15.53:0/1328462293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6080102270 0x7f6080198360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:10.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:07:10.780+0000 7f6084e7a700 1 --2- 172.21.15.53:0/1328462293 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6080103470 0x7f60801988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:10.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:07:10.780+0000 7f6084e7a700 1 -- 172.21.15.53:0/1328462293 >> 172.21.15.53:0/1328462293 conn(0x7f60800fd850 msgr2=0x7f6080106710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:07:10.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:07:10.780+0000 7f6084e7a700 1 -- 172.21.15.53:0/1328462293 shutdown_connections 2024-03-28T22:07:10.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-28T22:07:10.780+0000 7f6084e7a700 1 -- 172.21.15.53:0/1328462293 wait complete. 2024-03-28T22:07:11.404 DEBUG:teuthology.orchestra.run.smithi053:osd.10> sudo journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.10.service 2024-03-28T22:07:11.406 INFO:tasks.cephadm:Deploying osd.11 on smithi063 with /dev/vg_nvme/lv_1... 2024-03-28T22:07:11.407 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-28T22:07:11.671 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T22:07:12.009 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:11 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:12.010 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:11 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:12.010 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:11 smithi063 ceph-mon[36249]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:07:12.010 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:11 smithi063 ceph-mon[36249]: from='osd.10 [v2:172.21.15.53:6824/3357446484,v1:172.21.15.53:6825/3357446484]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T22:07:12.010 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:11 smithi063 ceph-mon[36249]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T22:07:12.010 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:11 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:12.010 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:11 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:12.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:11 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:12.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:11 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:12.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:11 smithi053 ceph-mon[35979]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:07:12.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:11 smithi053 ceph-mon[35979]: from='osd.10 [v2:172.21.15.53:6824/3357446484,v1:172.21.15.53:6825/3357446484]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T22:07:12.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:11 smithi053 ceph-mon[35979]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T22:07:12.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:11 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:12.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:11 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:12.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:11 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:12.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:11 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:12.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:11 smithi016 ceph-mon[32374]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:07:12.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:11 smithi016 ceph-mon[32374]: from='osd.10 [v2:172.21.15.53:6824/3357446484,v1:172.21.15.53:6825/3357446484]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T22:07:12.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:11 smithi016 ceph-mon[32374]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T22:07:12.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:11 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:12.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:11 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:13.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:12 smithi053 ceph-mon[35979]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-28T22:07:13.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:12 smithi053 ceph-mon[35979]: from='osd.10 [v2:172.21.15.53:6824/3357446484,v1:172.21.15.53:6825/3357446484]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-03-28T22:07:13.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:12 smithi053 ceph-mon[35979]: osdmap e71: 11 total, 10 up, 11 in 2024-03-28T22:07:13.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:12 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T22:07:13.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:12 smithi053 ceph-mon[35979]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-03-28T22:07:13.048 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:07:12 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:07:12.744+0000 7fad478eb700 -1 osd.10 0 waiting for initial osdmap 2024-03-28T22:07:13.048 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:07:12 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:07:12.766+0000 7fad43edf700 -1 osd.10 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T22:07:13.158 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:12 smithi063 ceph-mon[36249]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-28T22:07:13.159 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:12 smithi063 ceph-mon[36249]: from='osd.10 [v2:172.21.15.53:6824/3357446484,v1:172.21.15.53:6825/3357446484]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-03-28T22:07:13.159 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:12 smithi063 ceph-mon[36249]: osdmap e71: 11 total, 10 up, 11 in 2024-03-28T22:07:13.159 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:12 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T22:07:13.159 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:12 smithi063 ceph-mon[36249]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-03-28T22:07:13.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:12 smithi016 ceph-mon[32374]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-28T22:07:13.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:12 smithi016 ceph-mon[32374]: from='osd.10 [v2:172.21.15.53:6824/3357446484,v1:172.21.15.53:6825/3357446484]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-03-28T22:07:13.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:12 smithi016 ceph-mon[32374]: osdmap e71: 11 total, 10 up, 11 in 2024-03-28T22:07:13.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:12 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T22:07:13.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:12 smithi016 ceph-mon[32374]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-03-28T22:07:13.756 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-28T22:07:13.756 INFO:teuthology.orchestra.run.smithi063.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-28T22:07:13.756 INFO:teuthology.orchestra.run.smithi063.stderr: stderr: 10+0 records in 2024-03-28T22:07:13.757 INFO:teuthology.orchestra.run.smithi063.stderr:10+0 records out 2024-03-28T22:07:13.757 INFO:teuthology.orchestra.run.smithi063.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115105 s, 911 MB/s 2024-03-28T22:07:13.757 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping successful for: 2024-03-28T22:07:14.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:13 smithi053 ceph-mon[35979]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-03-28T22:07:14.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:13 smithi053 ceph-mon[35979]: osdmap e72: 11 total, 10 up, 11 in 2024-03-28T22:07:14.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:13 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T22:07:14.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:13 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T22:07:14.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:13 smithi053 ceph-mon[35979]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:07:14.110 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:13 smithi063 ceph-mon[36249]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-03-28T22:07:14.110 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:13 smithi063 ceph-mon[36249]: osdmap e72: 11 total, 10 up, 11 in 2024-03-28T22:07:14.110 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:13 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T22:07:14.110 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:13 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T22:07:14.110 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:13 smithi063 ceph-mon[36249]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:07:14.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:13 smithi016 ceph-mon[32374]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-03-28T22:07:14.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:13 smithi016 ceph-mon[32374]: osdmap e72: 11 total, 10 up, 11 in 2024-03-28T22:07:14.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:13 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T22:07:14.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:13 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T22:07:14.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:13 smithi016 ceph-mon[32374]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:07:14.545 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi063:vg_nvme/lv_1 2024-03-28T22:07:14.808 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T22:07:14.850 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:14 smithi063 ceph-mon[36249]: purged_snaps scrub starts 2024-03-28T22:07:14.850 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:14 smithi063 ceph-mon[36249]: purged_snaps scrub ok 2024-03-28T22:07:14.851 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:14 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T22:07:14.851 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:14 smithi063 ceph-mon[36249]: osd.10 [v2:172.21.15.53:6824/3357446484,v1:172.21.15.53:6825/3357446484] boot 2024-03-28T22:07:14.851 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:14 smithi063 ceph-mon[36249]: osdmap e73: 11 total, 11 up, 11 in 2024-03-28T22:07:14.851 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:14 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T22:07:15.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:14 smithi053 ceph-mon[35979]: purged_snaps scrub starts 2024-03-28T22:07:15.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:14 smithi053 ceph-mon[35979]: purged_snaps scrub ok 2024-03-28T22:07:15.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:14 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T22:07:15.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:14 smithi053 ceph-mon[35979]: osd.10 [v2:172.21.15.53:6824/3357446484,v1:172.21.15.53:6825/3357446484] boot 2024-03-28T22:07:15.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:14 smithi053 ceph-mon[35979]: osdmap e73: 11 total, 11 up, 11 in 2024-03-28T22:07:15.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:14 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T22:07:15.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:14 smithi016 ceph-mon[32374]: purged_snaps scrub starts 2024-03-28T22:07:15.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:14 smithi016 ceph-mon[32374]: purged_snaps scrub ok 2024-03-28T22:07:15.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:14 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T22:07:15.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:14 smithi016 ceph-mon[32374]: osd.10 [v2:172.21.15.53:6824/3357446484,v1:172.21.15.53:6825/3357446484] boot 2024-03-28T22:07:15.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:14 smithi016 ceph-mon[32374]: osdmap e73: 11 total, 11 up, 11 in 2024-03-28T22:07:15.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:14 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T22:07:16.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:15 smithi053 ceph-mon[35979]: osdmap e74: 11 total, 11 up, 11 in 2024-03-28T22:07:16.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:15 smithi053 ceph-mon[35979]: Detected new or changed devices on smithi053 2024-03-28T22:07:16.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:15 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:16.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:15 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:16.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:15 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:07:16.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:15 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:07:16.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:15 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:07:16.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:15 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:07:16.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:15 smithi053 ceph-mon[35979]: Adjusting osd_memory_target on smithi053 to 4271M 2024-03-28T22:07:16.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:15 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:16.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:15 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:07:16.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:15 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:07:16.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:15 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:16.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:15 smithi053 ceph-mon[35979]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:07:16.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:15 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:07:16.050 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:15 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:07:16.050 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:15 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:07:16.050 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:15 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:16.059 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:15 smithi063 ceph-mon[36249]: osdmap e74: 11 total, 11 up, 11 in 2024-03-28T22:07:16.060 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:15 smithi063 ceph-mon[36249]: Detected new or changed devices on smithi053 2024-03-28T22:07:16.060 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:15 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:16.060 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:15 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:16.060 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:15 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:07:16.060 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:15 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:07:16.060 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:15 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:07:16.060 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:15 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:07:16.060 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:15 smithi063 ceph-mon[36249]: Adjusting osd_memory_target on smithi053 to 4271M 2024-03-28T22:07:16.060 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:15 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:16.060 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:15 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:07:16.060 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:15 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:07:16.061 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:15 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:16.061 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:15 smithi063 ceph-mon[36249]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:07:16.061 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:15 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:07:16.061 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:15 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:07:16.061 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:15 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:07:16.061 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:15 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:16.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:15 smithi016 ceph-mon[32374]: osdmap e74: 11 total, 11 up, 11 in 2024-03-28T22:07:16.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:15 smithi016 ceph-mon[32374]: Detected new or changed devices on smithi053 2024-03-28T22:07:16.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:15 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:16.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:15 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:16.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:15 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:07:16.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:15 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:07:16.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:15 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:07:16.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:15 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:07:16.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:15 smithi016 ceph-mon[32374]: Adjusting osd_memory_target on smithi053 to 4271M 2024-03-28T22:07:16.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:15 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:16.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:15 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:07:16.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:15 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:07:16.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:15 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:16.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:15 smithi016 ceph-mon[32374]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:07:16.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:15 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:07:16.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:15 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:07:16.188 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:15 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:07:16.188 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:15 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:16.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.400+0000 7fc446a37700 1 -- 172.21.15.63:0/2218741934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc440102270 msgr2=0x7fc440102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:16.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.400+0000 7fc446a37700 1 --2- 172.21.15.63:0/2218741934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc440102270 0x7fc440102690 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc430009990 tx=0x7fc430009ca0 comp rx=0 tx=0).stop 2024-03-28T22:07:16.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.400+0000 7fc446a37700 1 -- 172.21.15.63:0/2218741934 shutdown_connections 2024-03-28T22:07:16.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.400+0000 7fc446a37700 1 --2- 172.21.15.63:0/2218741934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc440103e10 0x7fc44010a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:16.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.400+0000 7fc446a37700 1 --2- 172.21.15.63:0/2218741934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc440102270 0x7fc440102690 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:16.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.400+0000 7fc446a37700 1 --2- 172.21.15.63:0/2218741934 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc440103470 0x7fc4401038d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:16.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.400+0000 7fc446a37700 1 -- 172.21.15.63:0/2218741934 >> 172.21.15.63:0/2218741934 conn(0x7fc4400fd850 msgr2=0x7fc4400ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:07:16.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.400+0000 7fc446a37700 1 -- 172.21.15.63:0/2218741934 shutdown_connections 2024-03-28T22:07:16.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.400+0000 7fc446a37700 1 -- 172.21.15.63:0/2218741934 wait complete. 2024-03-28T22:07:16.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.401+0000 7fc446a37700 1 Processor -- start 2024-03-28T22:07:16.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.401+0000 7fc446a37700 1 -- start start 2024-03-28T22:07:16.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.401+0000 7fc446a37700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc440102270 0x7fc440075fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:16.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.401+0000 7fc446a37700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc440103470 0x7fc440074630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:16.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.401+0000 7fc446a37700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc440103e10 0x7fc440074b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:16.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.401+0000 7fc446a37700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc44010deb0 con 0x7fc440103470 2024-03-28T22:07:16.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.401+0000 7fc446a37700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc44010dd30 con 0x7fc440102270 2024-03-28T22:07:16.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.401+0000 7fc446a37700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc44010e030 con 0x7fc440103e10 2024-03-28T22:07:16.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.402+0000 7fc444fd4700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc440103e10 0x7fc440074b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:16.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.402+0000 7fc444fd4700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc440103e10 0x7fc440074b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38336/0 (socket says 172.21.15.63:38336) 2024-03-28T22:07:16.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.402+0000 7fc444fd4700 1 -- 172.21.15.63:0/3523774103 learned_addr learned my addr 172.21.15.63:0/3523774103 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-28T22:07:16.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.402+0000 7fc43f7fe700 1 --2- 172.21.15.63:0/3523774103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc440103470 0x7fc440074630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:16.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.402+0000 7fc43ffff700 1 --2- 172.21.15.63:0/3523774103 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc440102270 0x7fc440075fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:16.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.402+0000 7fc444fd4700 1 -- 172.21.15.63:0/3523774103 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc440102270 msgr2=0x7fc440075fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:16.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.402+0000 7fc444fd4700 1 --2- 172.21.15.63:0/3523774103 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc440102270 0x7fc440075fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:16.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.402+0000 7fc444fd4700 1 -- 172.21.15.63:0/3523774103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc440103470 msgr2=0x7fc440074630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:16.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.402+0000 7fc444fd4700 1 --2- 172.21.15.63:0/3523774103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc440103470 0x7fc440074630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:16.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.402+0000 7fc444fd4700 1 -- 172.21.15.63:0/3523774103 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc440075420 con 0x7fc440103e10 2024-03-28T22:07:16.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.402+0000 7fc444fd4700 1 --2- 172.21.15.63:0/3523774103 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc440103e10 0x7fc440074b70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc43800ada0 tx=0x7fc43800c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:07:16.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.402+0000 7fc43d7fa700 1 -- 172.21.15.63:0/3523774103 <== mon.1 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc438017020 con 0x7fc440103e10 2024-03-28T22:07:16.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.402+0000 7fc446a37700 1 -- 172.21.15.63:0/3523774103 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4401a2710 con 0x7fc440103e10 2024-03-28T22:07:16.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.402+0000 7fc43d7fa700 1 -- 172.21.15.63:0/3523774103 <== mon.1 v2:172.21.15.63:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc43800cdb0 con 0x7fc440103e10 2024-03-28T22:07:16.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.402+0000 7fc43d7fa700 1 -- 172.21.15.63:0/3523774103 <== mon.1 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4380053e0 con 0x7fc440103e10 2024-03-28T22:07:16.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.402+0000 7fc446a37700 1 -- 172.21.15.63:0/3523774103 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4401a2c50 con 0x7fc440103e10 2024-03-28T22:07:16.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.403+0000 7fc446a37700 1 -- 172.21.15.63:0/3523774103 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc440068f00 con 0x7fc440103e10 2024-03-28T22:07:16.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.404+0000 7fc43d7fa700 1 -- 172.21.15.63:0/3523774103 <== mon.1 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fc43800c8d0 con 0x7fc440103e10 2024-03-28T22:07:16.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.405+0000 7fc43d7fa700 1 --2- 172.21.15.63:0/3523774103 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc4280a1e10 0x7fc4280a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:16.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.405+0000 7fc43d7fa700 1 -- 172.21.15.63:0/3523774103 <== mon.1 v2:172.21.15.63:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6723+0+0 (secure 0 0 0) 0x7fc4380ccec0 con 0x7fc440103e10 2024-03-28T22:07:16.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.405+0000 7fc43ffff700 1 --2- 172.21.15.63:0/3523774103 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc4280a1e10 0x7fc4280a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:16.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.406+0000 7fc43ffff700 1 --2- 172.21.15.63:0/3523774103 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc4280a1e10 0x7fc4280a42d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc430003890 tx=0x7fc430005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:07:16.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.410+0000 7fc43d7fa700 1 -- 172.21.15.63:0/3523774103 <== mon.1 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc438021020 con 0x7fc440103e10 2024-03-28T22:07:16.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:16.623+0000 7fc446a37700 1 -- 172.21.15.63:0/3523774103 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fc440062680 con 0x7fc4280a1e10 2024-03-28T22:07:17.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:16 smithi053 ceph-mon[35979]: osdmap e75: 11 total, 11 up, 11 in 2024-03-28T22:07:17.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:16 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:07:17.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:16 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:07:17.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:16 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:07:17.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:16 smithi063 ceph-mon[36249]: osdmap e75: 11 total, 11 up, 11 in 2024-03-28T22:07:17.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:16 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:07:17.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:16 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:07:17.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:16 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:07:17.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:16 smithi016 ceph-mon[32374]: osdmap e75: 11 total, 11 up, 11 in 2024-03-28T22:07:17.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:16 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:07:17.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:16 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:07:17.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:16 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:07:17.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:17 smithi063 ceph-mon[36249]: from='client.24340 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:07:17.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:17 smithi063 ceph-mon[36249]: osdmap e76: 11 total, 11 up, 11 in 2024-03-28T22:07:17.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:17 smithi063 ceph-mon[36249]: pgmap v240: 1 pgs: 1 remapped+peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:07:18.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:17 smithi053 ceph-mon[35979]: from='client.24340 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:07:18.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:17 smithi053 ceph-mon[35979]: osdmap e76: 11 total, 11 up, 11 in 2024-03-28T22:07:18.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:17 smithi053 ceph-mon[35979]: pgmap v240: 1 pgs: 1 remapped+peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:07:18.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:17 smithi016 ceph-mon[32374]: from='client.24340 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:07:18.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:17 smithi016 ceph-mon[32374]: osdmap e76: 11 total, 11 up, 11 in 2024-03-28T22:07:18.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:17 smithi016 ceph-mon[32374]: pgmap v240: 1 pgs: 1 remapped+peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:07:18.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:18 smithi063 ceph-mon[36249]: osdmap e77: 11 total, 11 up, 11 in 2024-03-28T22:07:19.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:18 smithi053 ceph-mon[35979]: osdmap e77: 11 total, 11 up, 11 in 2024-03-28T22:07:19.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:18 smithi016 ceph-mon[32374]: osdmap e77: 11 total, 11 up, 11 in 2024-03-28T22:07:20.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:19 smithi053 ceph-mon[35979]: pgmap v242: 1 pgs: 1 remapped+peering; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:07:20.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:19 smithi053 ceph-mon[35979]: from='client.? 172.21.15.63:0/3605924754' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13302aa3-08b9-4639-9042-27d850b4d5f1"}]: dispatch 2024-03-28T22:07:20.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:19 smithi053 ceph-mon[35979]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13302aa3-08b9-4639-9042-27d850b4d5f1"}]: dispatch 2024-03-28T22:07:20.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:19 smithi053 ceph-mon[35979]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "13302aa3-08b9-4639-9042-27d850b4d5f1"}]': finished 2024-03-28T22:07:20.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:19 smithi053 ceph-mon[35979]: osdmap e78: 12 total, 11 up, 12 in 2024-03-28T22:07:20.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:19 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:07:20.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:19 smithi063 ceph-mon[36249]: pgmap v242: 1 pgs: 1 remapped+peering; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:07:20.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:19 smithi063 ceph-mon[36249]: from='client.? 172.21.15.63:0/3605924754' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13302aa3-08b9-4639-9042-27d850b4d5f1"}]: dispatch 2024-03-28T22:07:20.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:19 smithi063 ceph-mon[36249]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13302aa3-08b9-4639-9042-27d850b4d5f1"}]: dispatch 2024-03-28T22:07:20.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:19 smithi063 ceph-mon[36249]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "13302aa3-08b9-4639-9042-27d850b4d5f1"}]': finished 2024-03-28T22:07:20.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:19 smithi063 ceph-mon[36249]: osdmap e78: 12 total, 11 up, 12 in 2024-03-28T22:07:20.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:19 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:07:20.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:19 smithi016 ceph-mon[32374]: pgmap v242: 1 pgs: 1 remapped+peering; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:07:20.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:19 smithi016 ceph-mon[32374]: from='client.? 172.21.15.63:0/3605924754' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13302aa3-08b9-4639-9042-27d850b4d5f1"}]: dispatch 2024-03-28T22:07:20.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:19 smithi016 ceph-mon[32374]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13302aa3-08b9-4639-9042-27d850b4d5f1"}]: dispatch 2024-03-28T22:07:20.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:19 smithi016 ceph-mon[32374]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "13302aa3-08b9-4639-9042-27d850b4d5f1"}]': finished 2024-03-28T22:07:20.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:19 smithi016 ceph-mon[32374]: osdmap e78: 12 total, 11 up, 12 in 2024-03-28T22:07:20.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:19 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:07:21.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:20 smithi053 ceph-mon[35979]: from='client.? 172.21.15.63:0/4281479369' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:07:21.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:20 smithi063 ceph-mon[36249]: from='client.? 172.21.15.63:0/4281479369' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:07:21.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:20 smithi016 ceph-mon[32374]: from='client.? 172.21.15.63:0/4281479369' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:07:22.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:21 smithi053 ceph-mon[35979]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-03-28T22:07:22.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:21 smithi063 ceph-mon[36249]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-03-28T22:07:22.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:21 smithi016 ceph-mon[32374]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-03-28T22:07:23.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:23 smithi053 ceph-mon[35979]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-03-28T22:07:23.382 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:23 smithi063 ceph-mon[36249]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-03-28T22:07:23.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:23 smithi016 ceph-mon[32374]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-03-28T22:07:25.302 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:25 smithi063 ceph-mon[36249]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T22:07:25.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:25 smithi016 ceph-mon[32374]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T22:07:25.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:25 smithi053 ceph-mon[35979]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T22:07:27.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:27 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-28T22:07:27.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:27 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:07:27.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:27 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-28T22:07:27.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:27 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:07:27.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:27 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-28T22:07:27.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:27 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:07:28.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:28 smithi063 ceph-mon[36249]: Deploying daemon osd.11 on smithi063 2024-03-28T22:07:28.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:28 smithi063 ceph-mon[36249]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-28T22:07:28.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:28 smithi053 ceph-mon[35979]: Deploying daemon osd.11 on smithi063 2024-03-28T22:07:28.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:28 smithi053 ceph-mon[35979]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-28T22:07:28.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:28 smithi016 ceph-mon[32374]: Deploying daemon osd.11 on smithi063 2024-03-28T22:07:28.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:28 smithi016 ceph-mon[32374]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-28T22:07:29.337 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:29 smithi063 ceph-mon[36249]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T22:07:29.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:29 smithi016 ceph-mon[32374]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T22:07:29.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:29 smithi053 ceph-mon[35979]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T22:07:31.380 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:31 smithi063 ceph-mon[36249]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-03-28T22:07:31.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:31 smithi016 ceph-mon[32374]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-03-28T22:07:31.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:31 smithi053 ceph-mon[35979]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-03-28T22:07:33.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:33 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:33.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:33 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:33.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:33 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:07:33.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:33 smithi063 ceph-mon[36249]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:07:33.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:33 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:33.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:33 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:33.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:33 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:07:33.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:33 smithi016 ceph-mon[32374]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:07:34.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:33 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:34.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:33 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:34.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:33 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:07:34.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:33 smithi053 ceph-mon[35979]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:07:35.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:35 smithi063 ceph-mon[36249]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:07:35.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:35 smithi016 ceph-mon[32374]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:07:35.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:35 smithi053 ceph-mon[35979]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:07:36.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:36.395+0000 7fc43d7fa700 1 -- 172.21.15.63:0/3523774103 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fc440062680 con 0x7fc4280a1e10 2024-03-28T22:07:36.399 INFO:teuthology.orchestra.run.smithi063.stdout:Created osd(s) 11 on host 'smithi063' 2024-03-28T22:07:36.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:36.398+0000 7fc446a37700 1 -- 172.21.15.63:0/3523774103 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc4280a1e10 msgr2=0x7fc4280a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:36.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:36.398+0000 7fc446a37700 1 --2- 172.21.15.63:0/3523774103 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc4280a1e10 0x7fc4280a42d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc430003890 tx=0x7fc430005fd0 comp rx=0 tx=0).stop 2024-03-28T22:07:36.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:36.398+0000 7fc446a37700 1 -- 172.21.15.63:0/3523774103 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc440103e10 msgr2=0x7fc440074b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:36.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:36.398+0000 7fc446a37700 1 --2- 172.21.15.63:0/3523774103 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc440103e10 0x7fc440074b70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc43800ada0 tx=0x7fc43800c330 comp rx=0 tx=0).stop 2024-03-28T22:07:36.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:36.398+0000 7fc446a37700 1 -- 172.21.15.63:0/3523774103 shutdown_connections 2024-03-28T22:07:36.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:36.398+0000 7fc446a37700 1 --2- 172.21.15.63:0/3523774103 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc4280a1e10 0x7fc4280a42d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:36.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:36.398+0000 7fc446a37700 1 --2- 172.21.15.63:0/3523774103 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc440103e10 0x7fc440074b70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:36.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:36.398+0000 7fc446a37700 1 --2- 172.21.15.63:0/3523774103 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc440102270 0x7fc440075fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:36.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:36.398+0000 7fc446a37700 1 --2- 172.21.15.63:0/3523774103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc440103470 0x7fc440074630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:36.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:36.398+0000 7fc446a37700 1 -- 172.21.15.63:0/3523774103 >> 172.21.15.63:0/3523774103 conn(0x7fc4400fd850 msgr2=0x7fc440106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:07:36.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:36.398+0000 7fc446a37700 1 -- 172.21.15.63:0/3523774103 shutdown_connections 2024-03-28T22:07:36.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:36.398+0000 7fc446a37700 1 -- 172.21.15.63:0/3523774103 wait complete. 2024-03-28T22:07:36.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:36 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:36.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:36 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:36.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:36 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:07:36.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:36 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:07:36.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:36 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:36.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:36 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:36.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:36 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:36.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:36 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:07:36.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:36 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:07:36.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:36 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:36.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:36 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:36.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:36 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:36.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:36 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:07:36.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:36 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:07:36.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:36 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:37.063 DEBUG:teuthology.orchestra.run.smithi063:osd.11> sudo journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.11.service 2024-03-28T22:07:37.065 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-28T22:07:37.065 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph osd stat -f json 2024-03-28T22:07:37.319 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:07:37.626 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:37 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:37.626 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:37 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:37.626 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:37 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:07:37.626 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:37 smithi016 ceph-mon[32374]: from='osd.11 [v2:172.21.15.63:6824/2167866570,v1:172.21.15.63:6825/2167866570]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T22:07:37.626 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:37 smithi016 ceph-mon[32374]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T22:07:37.626 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:37 smithi016 ceph-mon[32374]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:07:37.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:37 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:37.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:37 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:37.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:37 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:07:37.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:37 smithi063 ceph-mon[36249]: from='osd.11 [v2:172.21.15.63:6824/2167866570,v1:172.21.15.63:6825/2167866570]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T22:07:37.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:37 smithi063 ceph-mon[36249]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T22:07:37.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:37 smithi063 ceph-mon[36249]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:07:37.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:37 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:37.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:37 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:37.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:37 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:07:37.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:37 smithi053 ceph-mon[35979]: from='osd.11 [v2:172.21.15.63:6824/2167866570,v1:172.21.15.63:6825/2167866570]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T22:07:37.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:37 smithi053 ceph-mon[35979]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T22:07:37.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:37 smithi053 ceph-mon[35979]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:07:38.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:38 smithi016 ceph-mon[32374]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-28T22:07:38.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:38 smithi016 ceph-mon[32374]: osdmap e79: 12 total, 11 up, 12 in 2024-03-28T22:07:38.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:38 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:07:38.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:38 smithi016 ceph-mon[32374]: from='osd.11 [v2:172.21.15.63:6824/2167866570,v1:172.21.15.63:6825/2167866570]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-28T22:07:38.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:38 smithi016 ceph-mon[32374]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-28T22:07:38.624 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:38 smithi063 ceph-mon[36249]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-28T22:07:38.625 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:38 smithi063 ceph-mon[36249]: osdmap e79: 12 total, 11 up, 12 in 2024-03-28T22:07:38.625 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:38 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:07:38.625 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:38 smithi063 ceph-mon[36249]: from='osd.11 [v2:172.21.15.63:6824/2167866570,v1:172.21.15.63:6825/2167866570]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-28T22:07:38.625 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:38 smithi063 ceph-mon[36249]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-28T22:07:38.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:38 smithi053 ceph-mon[35979]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-28T22:07:38.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:38 smithi053 ceph-mon[35979]: osdmap e79: 12 total, 11 up, 12 in 2024-03-28T22:07:38.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:38 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:07:38.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:38 smithi053 ceph-mon[35979]: from='osd.11 [v2:172.21.15.63:6824/2167866570,v1:172.21.15.63:6825/2167866570]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-28T22:07:38.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:38 smithi053 ceph-mon[35979]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-28T22:07:39.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.035+0000 7f579a7ce700 1 -- 172.21.15.16:0/1481841461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5794103e10 msgr2=0x7f579410a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:39.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.035+0000 7f579a7ce700 1 --2- 172.21.15.16:0/1481841461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5794103e10 0x7f579410a6c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f578c00b0d0 tx=0x7f578c00b3e0 comp rx=0 tx=0).stop 2024-03-28T22:07:39.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.035+0000 7f579a7ce700 1 -- 172.21.15.16:0/1481841461 shutdown_connections 2024-03-28T22:07:39.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.035+0000 7f579a7ce700 1 --2- 172.21.15.16:0/1481841461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5794103e10 0x7f579410a6c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:39.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.035+0000 7f579a7ce700 1 --2- 172.21.15.16:0/1481841461 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5794102270 0x7f5794102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:39.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.035+0000 7f579a7ce700 1 --2- 172.21.15.16:0/1481841461 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5794103470 0x7f57941038d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:39.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.035+0000 7f579a7ce700 1 -- 172.21.15.16:0/1481841461 >> 172.21.15.16:0/1481841461 conn(0x7f57940fd7f0 msgr2=0x7f57940ffc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:07:39.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.035+0000 7f579a7ce700 1 -- 172.21.15.16:0/1481841461 shutdown_connections 2024-03-28T22:07:39.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.035+0000 7f579a7ce700 1 -- 172.21.15.16:0/1481841461 wait complete. 2024-03-28T22:07:39.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.036+0000 7f579a7ce700 1 Processor -- start 2024-03-28T22:07:39.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.036+0000 7f579a7ce700 1 -- start start 2024-03-28T22:07:39.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.036+0000 7f579a7ce700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5794102270 0x7f57941984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:39.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.036+0000 7f579a7ce700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5794103470 0x7f57941989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:39.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.036+0000 7f579a7ce700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5794103e10 0x7f579419fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:39.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.036+0000 7f579a7ce700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f579410def0 con 0x7f5794102270 2024-03-28T22:07:39.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.036+0000 7f579a7ce700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f579410dd70 con 0x7f5794103470 2024-03-28T22:07:39.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.036+0000 7f579a7ce700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f579410e070 con 0x7f5794103e10 2024-03-28T22:07:39.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.036+0000 7f5793fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5794102270 0x7f57941984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:39.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.036+0000 7f5793fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5794102270 0x7f57941984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41194/0 (socket says 172.21.15.16:41194) 2024-03-28T22:07:39.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.036+0000 7f5793fff700 1 -- 172.21.15.16:0/2248913247 learned_addr learned my addr 172.21.15.16:0/2248913247 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:07:39.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.036+0000 7f5798d6b700 1 --2- 172.21.15.16:0/2248913247 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5794103e10 0x7f579419fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:39.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.037+0000 7f57937fe700 1 --2- 172.21.15.16:0/2248913247 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5794103470 0x7f57941989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:39.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.037+0000 7f5793fff700 1 -- 172.21.15.16:0/2248913247 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5794103e10 msgr2=0x7f579419fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:39.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.037+0000 7f5793fff700 1 --2- 172.21.15.16:0/2248913247 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5794103e10 0x7f579419fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:39.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.037+0000 7f5793fff700 1 -- 172.21.15.16:0/2248913247 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5794103470 msgr2=0x7f57941989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:39.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.037+0000 7f5793fff700 1 --2- 172.21.15.16:0/2248913247 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5794103470 0x7f57941989e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:39.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.037+0000 7f5793fff700 1 -- 172.21.15.16:0/2248913247 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57941a0100 con 0x7f5794102270 2024-03-28T22:07:39.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.037+0000 7f5793fff700 1 --2- 172.21.15.16:0/2248913247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5794102270 0x7f57941984a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f578400b530 tx=0x7f578400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:07:39.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.037+0000 7f57917fa700 1 -- 172.21.15.16:0/2248913247 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5784015020 con 0x7f5794102270 2024-03-28T22:07:39.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.037+0000 7f579a7ce700 1 -- 172.21.15.16:0/2248913247 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57941a03f0 con 0x7f5794102270 2024-03-28T22:07:39.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.037+0000 7f57917fa700 1 -- 172.21.15.16:0/2248913247 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5784011d10 con 0x7f5794102270 2024-03-28T22:07:39.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.037+0000 7f57917fa700 1 -- 172.21.15.16:0/2248913247 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5784010d90 con 0x7f5794102270 2024-03-28T22:07:39.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.037+0000 7f579a7ce700 1 -- 172.21.15.16:0/2248913247 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57941a0a00 con 0x7f5794102270 2024-03-28T22:07:39.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.038+0000 7f579a7ce700 1 -- 172.21.15.16:0/2248913247 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5794068f00 con 0x7f5794102270 2024-03-28T22:07:39.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.039+0000 7f57917fa700 1 -- 172.21.15.16:0/2248913247 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f5784018020 con 0x7f5794102270 2024-03-28T22:07:39.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.040+0000 7f57917fa700 1 --2- 172.21.15.16:0/2248913247 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f577c0a1e10 0x7f577c0a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:39.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.040+0000 7f57937fe700 1 --2- 172.21.15.16:0/2248913247 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f577c0a1e10 0x7f577c0a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:39.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.040+0000 7f57917fa700 1 -- 172.21.15.16:0/2248913247 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6847+0+0 (secure 0 0 0) 0x7f57840cd430 con 0x7f5794102270 2024-03-28T22:07:39.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.040+0000 7f57937fe700 1 --2- 172.21.15.16:0/2248913247 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f577c0a1e10 0x7f577c0a42d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5788004630 tx=0x7f578800b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:07:39.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.043+0000 7f57917fa700 1 -- 172.21.15.16:0/2248913247 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f578401f020 con 0x7f5794102270 2024-03-28T22:07:39.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.248+0000 7f579a7ce700 1 -- 172.21.15.16:0/2248913247 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f579402df80 con 0x7f5794102270 2024-03-28T22:07:39.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.248+0000 7f57917fa700 1 -- 172.21.15.16:0/2248913247 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) ==== 74+0+133 (secure 0 0 0) 0x7f578409ae20 con 0x7f5794102270 2024-03-28T22:07:39.251 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:07:39.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.251+0000 7f579a7ce700 1 -- 172.21.15.16:0/2248913247 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f577c0a1e10 msgr2=0x7f577c0a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:39.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.251+0000 7f579a7ce700 1 --2- 172.21.15.16:0/2248913247 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f577c0a1e10 0x7f577c0a42d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5788004630 tx=0x7f578800b3c0 comp rx=0 tx=0).stop 2024-03-28T22:07:39.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.251+0000 7f579a7ce700 1 -- 172.21.15.16:0/2248913247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5794102270 msgr2=0x7f57941984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:39.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.251+0000 7f579a7ce700 1 --2- 172.21.15.16:0/2248913247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5794102270 0x7f57941984a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f578400b530 tx=0x7f578400b840 comp rx=0 tx=0).stop 2024-03-28T22:07:39.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.251+0000 7f579a7ce700 1 -- 172.21.15.16:0/2248913247 shutdown_connections 2024-03-28T22:07:39.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.251+0000 7f579a7ce700 1 --2- 172.21.15.16:0/2248913247 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f577c0a1e10 0x7f577c0a42d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:39.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.251+0000 7f579a7ce700 1 --2- 172.21.15.16:0/2248913247 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5794103e10 0x7f579419fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:39.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.251+0000 7f579a7ce700 1 --2- 172.21.15.16:0/2248913247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5794102270 0x7f57941984a0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:39.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.251+0000 7f579a7ce700 1 --2- 172.21.15.16:0/2248913247 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5794103470 0x7f57941989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:39.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.251+0000 7f579a7ce700 1 -- 172.21.15.16:0/2248913247 >> 172.21.15.16:0/2248913247 conn(0x7f57940fd7f0 msgr2=0x7f5794106710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:07:39.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.251+0000 7f579a7ce700 1 -- 172.21.15.16:0/2248913247 shutdown_connections 2024-03-28T22:07:39.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:39.251+0000 7f579a7ce700 1 -- 172.21.15.16:0/2248913247 wait complete. 2024-03-28T22:07:39.463 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:39 smithi016 ceph-mon[32374]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-03-28T22:07:39.463 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:39 smithi016 ceph-mon[32374]: osdmap e80: 12 total, 11 up, 12 in 2024-03-28T22:07:39.463 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:39 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:07:39.463 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:39 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:07:39.463 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:39 smithi016 ceph-mon[32374]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:07:39.463 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:39 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2248913247' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T22:07:39.463 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:39 smithi016 ceph-mon[32374]: from='osd.11 ' entity='osd.11' 2024-03-28T22:07:39.532 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:39 smithi063 ceph-mon[36249]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-03-28T22:07:39.532 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:39 smithi063 ceph-mon[36249]: osdmap e80: 12 total, 11 up, 12 in 2024-03-28T22:07:39.532 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:39 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:07:39.532 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:39 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:07:39.532 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:39 smithi063 ceph-mon[36249]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:07:39.532 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:39 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2248913247' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T22:07:39.532 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:39 smithi063 ceph-mon[36249]: from='osd.11 ' entity='osd.11' 2024-03-28T22:07:39.533 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:07:39 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:07:39.248+0000 7f7fc160e700 -1 osd.11 0 waiting for initial osdmap 2024-03-28T22:07:39.533 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:07:39 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:07:39.264+0000 7f7fb83f7700 -1 osd.11 80 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T22:07:39.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:39 smithi053 ceph-mon[35979]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-03-28T22:07:39.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:39 smithi053 ceph-mon[35979]: osdmap e80: 12 total, 11 up, 12 in 2024-03-28T22:07:39.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:39 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:07:39.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:39 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:07:39.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:39 smithi053 ceph-mon[35979]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:07:39.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:39 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2248913247' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T22:07:39.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:39 smithi053 ceph-mon[35979]: from='osd.11 ' entity='osd.11' 2024-03-28T22:07:40.031 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":80,"num_osds":12,"num_up_osds":11,"osd_up_since":1711663633,"num_in_osds":12,"osd_in_since":1711663639,"num_remapped_pgs":0} 2024-03-28T22:07:40.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:40 smithi063 ceph-mon[36249]: purged_snaps scrub starts 2024-03-28T22:07:40.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:40 smithi063 ceph-mon[36249]: purged_snaps scrub ok 2024-03-28T22:07:40.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:40 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:07:40.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:40 smithi063 ceph-mon[36249]: osd.11 [v2:172.21.15.63:6824/2167866570,v1:172.21.15.63:6825/2167866570] boot 2024-03-28T22:07:40.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:40 smithi063 ceph-mon[36249]: osdmap e81: 12 total, 12 up, 12 in 2024-03-28T22:07:40.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:40 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:07:40.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:40 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:40.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:40 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:40.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:40 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:07:40.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:40 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:07:40.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:40 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:07:40.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:40 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:07:40.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:40 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:40.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:40 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:07:40.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:40 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:07:40.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:40 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:40.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:40 smithi016 ceph-mon[32374]: purged_snaps scrub starts 2024-03-28T22:07:40.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:40 smithi016 ceph-mon[32374]: purged_snaps scrub ok 2024-03-28T22:07:40.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:40 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:07:40.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:40 smithi016 ceph-mon[32374]: osd.11 [v2:172.21.15.63:6824/2167866570,v1:172.21.15.63:6825/2167866570] boot 2024-03-28T22:07:40.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:40 smithi016 ceph-mon[32374]: osdmap e81: 12 total, 12 up, 12 in 2024-03-28T22:07:40.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:40 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:07:40.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:40 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:40.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:40 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:40.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:40 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:07:40.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:40 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:07:40.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:40 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:07:40.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:40 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:07:40.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:40 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:40.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:40 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:07:40.688 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:40 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:07:40.688 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:40 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:40.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:40 smithi053 ceph-mon[35979]: purged_snaps scrub starts 2024-03-28T22:07:40.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:40 smithi053 ceph-mon[35979]: purged_snaps scrub ok 2024-03-28T22:07:40.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:40 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:07:40.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:40 smithi053 ceph-mon[35979]: osd.11 [v2:172.21.15.63:6824/2167866570,v1:172.21.15.63:6825/2167866570] boot 2024-03-28T22:07:40.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:40 smithi053 ceph-mon[35979]: osdmap e81: 12 total, 12 up, 12 in 2024-03-28T22:07:40.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:40 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:07:40.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:40 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:40.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:40 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:40.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:40 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:07:40.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:40 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:07:40.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:40 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:07:40.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:40 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:07:40.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:40 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:40.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:40 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:07:40.799 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:40 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:07:40.799 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:40 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:41.033 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph osd stat -f json 2024-03-28T22:07:41.290 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:07:41.656 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:41 smithi016 ceph-mon[32374]: Detected new or changed devices on smithi063 2024-03-28T22:07:41.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:41 smithi016 ceph-mon[32374]: Adjusting osd_memory_target on smithi063 to 4270M 2024-03-28T22:07:41.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:41 smithi016 ceph-mon[32374]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:07:41.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:41 smithi063 ceph-mon[36249]: Detected new or changed devices on smithi063 2024-03-28T22:07:41.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:41 smithi063 ceph-mon[36249]: Adjusting osd_memory_target on smithi063 to 4270M 2024-03-28T22:07:41.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:41 smithi063 ceph-mon[36249]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:07:41.699 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:41 smithi053 ceph-mon[35979]: Detected new or changed devices on smithi063 2024-03-28T22:07:41.699 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:41 smithi053 ceph-mon[35979]: Adjusting osd_memory_target on smithi063 to 4270M 2024-03-28T22:07:41.699 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:41 smithi053 ceph-mon[35979]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:07:42.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:42 smithi016 ceph-mon[32374]: osdmap e82: 12 total, 12 up, 12 in 2024-03-28T22:07:42.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:42 smithi063 ceph-mon[36249]: osdmap e82: 12 total, 12 up, 12 in 2024-03-28T22:07:42.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:42 smithi053 ceph-mon[35979]: osdmap e82: 12 total, 12 up, 12 in 2024-03-28T22:07:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.992+0000 7f33f2071700 1 -- 172.21.15.16:0/414031418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33ec103470 msgr2=0x7f33ec1038d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.992+0000 7f33f2071700 1 --2- 172.21.15.16:0/414031418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33ec103470 0x7f33ec1038d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f33e00099c0 tx=0x7f33e0009cd0 comp rx=0 tx=0).stop 2024-03-28T22:07:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.992+0000 7f33f2071700 1 -- 172.21.15.16:0/414031418 shutdown_connections 2024-03-28T22:07:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.992+0000 7f33f2071700 1 --2- 172.21.15.16:0/414031418 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33ec103e10 0x7f33ec10a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.992+0000 7f33f2071700 1 --2- 172.21.15.16:0/414031418 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33ec102270 0x7f33ec102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.992+0000 7f33f2071700 1 --2- 172.21.15.16:0/414031418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33ec103470 0x7f33ec1038d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.992+0000 7f33f2071700 1 -- 172.21.15.16:0/414031418 >> 172.21.15.16:0/414031418 conn(0x7f33ec0fd850 msgr2=0x7f33ec0ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:07:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.992+0000 7f33f2071700 1 -- 172.21.15.16:0/414031418 shutdown_connections 2024-03-28T22:07:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.992+0000 7f33f2071700 1 -- 172.21.15.16:0/414031418 wait complete. 2024-03-28T22:07:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.992+0000 7f33f2071700 1 Processor -- start 2024-03-28T22:07:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.993+0000 7f33f2071700 1 -- start start 2024-03-28T22:07:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.993+0000 7f33f2071700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33ec102270 0x7f33ec075fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.993+0000 7f33f2071700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33ec103470 0x7f33ec074630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.993+0000 7f33f2071700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33ec103e10 0x7f33ec074b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.993+0000 7f33f2071700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f33ec10de60 con 0x7f33ec103470 2024-03-28T22:07:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.993+0000 7f33f2071700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f33ec10dce0 con 0x7f33ec102270 2024-03-28T22:07:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.993+0000 7f33f2071700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f33ec10dfe0 con 0x7f33ec103e10 2024-03-28T22:07:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.993+0000 7f33eaffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33ec103470 0x7f33ec074630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.993+0000 7f33eaffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33ec103470 0x7f33ec074630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36382/0 (socket says 172.21.15.16:36382) 2024-03-28T22:07:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.993+0000 7f33eaffd700 1 -- 172.21.15.16:0/3142036052 learned_addr learned my addr 172.21.15.16:0/3142036052 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:07:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.993+0000 7f33eb7fe700 1 --2- 172.21.15.16:0/3142036052 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33ec102270 0x7f33ec075fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.993+0000 7f33ebfff700 1 --2- 172.21.15.16:0/3142036052 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33ec103e10 0x7f33ec074b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.993+0000 7f33eaffd700 1 -- 172.21.15.16:0/3142036052 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33ec103e10 msgr2=0x7f33ec074b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.993+0000 7f33eaffd700 1 --2- 172.21.15.16:0/3142036052 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33ec103e10 0x7f33ec074b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.993+0000 7f33eaffd700 1 -- 172.21.15.16:0/3142036052 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33ec102270 msgr2=0x7f33ec075fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.993+0000 7f33eaffd700 1 --2- 172.21.15.16:0/3142036052 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33ec102270 0x7f33ec075fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.993+0000 7f33eaffd700 1 -- 172.21.15.16:0/3142036052 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33ec075420 con 0x7f33ec103470 2024-03-28T22:07:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.993+0000 7f33ebfff700 1 --2- 172.21.15.16:0/3142036052 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33ec103e10 0x7f33ec074b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:07:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.993+0000 7f33eb7fe700 1 --2- 172.21.15.16:0/3142036052 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33ec102270 0x7f33ec075fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:07:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.994+0000 7f33eaffd700 1 --2- 172.21.15.16:0/3142036052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33ec103470 0x7f33ec074630 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f33e0003790 tx=0x7f33e000f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:07:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.994+0000 7f33e8ff9700 1 -- 172.21.15.16:0/3142036052 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e0009e60 con 0x7f33ec103470 2024-03-28T22:07:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.994+0000 7f33e8ff9700 1 -- 172.21.15.16:0/3142036052 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33e0003e40 con 0x7f33ec103470 2024-03-28T22:07:42.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.994+0000 7f33e8ff9700 1 -- 172.21.15.16:0/3142036052 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e00173f0 con 0x7f33ec103470 2024-03-28T22:07:42.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.994+0000 7f33f2071700 1 -- 172.21.15.16:0/3142036052 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33ec1a25e0 con 0x7f33ec103470 2024-03-28T22:07:42.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.994+0000 7f33f2071700 1 -- 172.21.15.16:0/3142036052 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33ec1a2a30 con 0x7f33ec103470 2024-03-28T22:07:43.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.995+0000 7f33f2071700 1 -- 172.21.15.16:0/3142036052 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33ec068f00 con 0x7f33ec103470 2024-03-28T22:07:43.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.996+0000 7f33e8ff9700 1 -- 172.21.15.16:0/3142036052 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f33e0003960 con 0x7f33ec103470 2024-03-28T22:07:43.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.996+0000 7f33e8ff9700 1 --2- 172.21.15.16:0/3142036052 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f33d40a1e10 0x7f33d40a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:43.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.997+0000 7f33eb7fe700 1 --2- 172.21.15.16:0/3142036052 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f33d40a1e10 0x7f33d40a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:43.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.997+0000 7f33e8ff9700 1 -- 172.21.15.16:0/3142036052 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f33e00ceb50 con 0x7f33ec103470 2024-03-28T22:07:43.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:42.997+0000 7f33eb7fe700 1 --2- 172.21.15.16:0/3142036052 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f33d40a1e10 0x7f33d40a42d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f33ec1032d0 tx=0x7f33dc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:07:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:43.000+0000 7f33e8ff9700 1 -- 172.21.15.16:0/3142036052 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f33e005d280 con 0x7f33ec103470 2024-03-28T22:07:43.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:43.204+0000 7f33f2071700 1 -- 172.21.15.16:0/3142036052 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f33ec062e50 con 0x7f33ec103470 2024-03-28T22:07:43.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:43.205+0000 7f33e8ff9700 1 -- 172.21.15.16:0/3142036052 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v82) ==== 74+0+133 (secure 0 0 0) 0x7f33e0013070 con 0x7f33ec103470 2024-03-28T22:07:43.207 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:07:43.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:43.207+0000 7f33f2071700 1 -- 172.21.15.16:0/3142036052 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f33d40a1e10 msgr2=0x7f33d40a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:43.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:43.207+0000 7f33f2071700 1 --2- 172.21.15.16:0/3142036052 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f33d40a1e10 0x7f33d40a42d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f33ec1032d0 tx=0x7f33dc009260 comp rx=0 tx=0).stop 2024-03-28T22:07:43.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:43.207+0000 7f33f2071700 1 -- 172.21.15.16:0/3142036052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33ec103470 msgr2=0x7f33ec074630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:43.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:43.207+0000 7f33f2071700 1 --2- 172.21.15.16:0/3142036052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33ec103470 0x7f33ec074630 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f33e0003790 tx=0x7f33e000f740 comp rx=0 tx=0).stop 2024-03-28T22:07:43.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:43.207+0000 7f33f2071700 1 -- 172.21.15.16:0/3142036052 shutdown_connections 2024-03-28T22:07:43.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:43.207+0000 7f33f2071700 1 --2- 172.21.15.16:0/3142036052 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f33d40a1e10 0x7f33d40a42d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:43.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:43.207+0000 7f33f2071700 1 --2- 172.21.15.16:0/3142036052 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33ec103e10 0x7f33ec074b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:43.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:43.207+0000 7f33f2071700 1 --2- 172.21.15.16:0/3142036052 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33ec102270 0x7f33ec075fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:43.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:43.207+0000 7f33f2071700 1 --2- 172.21.15.16:0/3142036052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33ec103470 0x7f33ec074630 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:43.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:43.207+0000 7f33f2071700 1 -- 172.21.15.16:0/3142036052 >> 172.21.15.16:0/3142036052 conn(0x7f33ec0fd850 msgr2=0x7f33ec106770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:07:43.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:43.207+0000 7f33f2071700 1 -- 172.21.15.16:0/3142036052 shutdown_connections 2024-03-28T22:07:43.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:43.207+0000 7f33f2071700 1 -- 172.21.15.16:0/3142036052 wait complete. 2024-03-28T22:07:43.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:43 smithi063 ceph-mon[36249]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:07:43.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:43 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/3142036052' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T22:07:43.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:43 smithi016 ceph-mon[32374]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:07:43.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:43 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3142036052' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T22:07:43.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:43 smithi053 ceph-mon[35979]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:07:43.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:43 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3142036052' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T22:07:43.875 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":82,"num_osds":12,"num_up_osds":12,"osd_up_since":1711663660,"num_in_osds":12,"osd_in_since":1711663639,"num_remapped_pgs":0} 2024-03-28T22:07:43.876 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-28T22:07:44.124 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:07:45.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:45 smithi016 ceph-mon[32374]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:07:45.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:45 smithi063 ceph-mon[36249]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:07:45.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:45 smithi053 ceph-mon[35979]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:07:45.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.778+0000 7f0f1ff3b700 1 -- 172.21.15.16:0/748642514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f1806a9c0 msgr2=0x7f0f1806ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:45.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.778+0000 7f0f1ff3b700 1 --2- 172.21.15.16:0/748642514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f1806a9c0 0x7f0f1806ae40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0f08009990 tx=0x7f0f08009ca0 comp rx=0 tx=0).stop 2024-03-28T22:07:45.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.778+0000 7f0f1ff3b700 1 -- 172.21.15.16:0/748642514 shutdown_connections 2024-03-28T22:07:45.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.778+0000 7f0f1ff3b700 1 --2- 172.21.15.16:0/748642514 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f1806b380 0x7f0f1810f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:45.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.778+0000 7f0f1ff3b700 1 --2- 172.21.15.16:0/748642514 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f18107250 0x7f0f18107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:45.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.778+0000 7f0f1ff3b700 1 --2- 172.21.15.16:0/748642514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f1806a9c0 0x7f0f1806ae40 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:45.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.778+0000 7f0f1ff3b700 1 -- 172.21.15.16:0/748642514 >> 172.21.15.16:0/748642514 conn(0x7f0f180781a0 msgr2=0x7f0f180785b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:07:45.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.778+0000 7f0f1ff3b700 1 -- 172.21.15.16:0/748642514 shutdown_connections 2024-03-28T22:07:45.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.778+0000 7f0f1ff3b700 1 -- 172.21.15.16:0/748642514 wait complete. 2024-03-28T22:07:45.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.778+0000 7f0f1ff3b700 1 Processor -- start 2024-03-28T22:07:45.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.778+0000 7f0f1ff3b700 1 -- start start 2024-03-28T22:07:45.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.779+0000 7f0f1ff3b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f1806a9c0 0x7f0f181011f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:45.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.779+0000 7f0f1ff3b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f1806b380 0x7f0f18101730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:45.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.779+0000 7f0f1ff3b700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f18107250 0x7f0f18101c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:45.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.779+0000 7f0f1ff3b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0f18112670 con 0x7f0f1806a9c0 2024-03-28T22:07:45.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.779+0000 7f0f1ff3b700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0f181124f0 con 0x7f0f1806b380 2024-03-28T22:07:45.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.779+0000 7f0f1ff3b700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0f181127f0 con 0x7f0f18107250 2024-03-28T22:07:45.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.779+0000 7f0f1dcd7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f1806a9c0 0x7f0f181011f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:45.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.779+0000 7f0f1dcd7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f1806a9c0 0x7f0f181011f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36392/0 (socket says 172.21.15.16:36392) 2024-03-28T22:07:45.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.779+0000 7f0f1dcd7700 1 -- 172.21.15.16:0/2527589297 learned_addr learned my addr 172.21.15.16:0/2527589297 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:07:45.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.779+0000 7f0f1e4d8700 1 --2- 172.21.15.16:0/2527589297 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f18107250 0x7f0f18101c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:45.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.779+0000 7f0f1d4d6700 1 --2- 172.21.15.16:0/2527589297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f1806b380 0x7f0f18101730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:45.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.779+0000 7f0f1dcd7700 1 -- 172.21.15.16:0/2527589297 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f18107250 msgr2=0x7f0f18101c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:45.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.779+0000 7f0f1dcd7700 1 --2- 172.21.15.16:0/2527589297 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f18107250 0x7f0f18101c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:45.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.779+0000 7f0f1dcd7700 1 -- 172.21.15.16:0/2527589297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f1806b380 msgr2=0x7f0f18101730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:45.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.779+0000 7f0f1dcd7700 1 --2- 172.21.15.16:0/2527589297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f1806b380 0x7f0f18101730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:45.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.779+0000 7f0f1dcd7700 1 -- 172.21.15.16:0/2527589297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f181a6f70 con 0x7f0f1806a9c0 2024-03-28T22:07:45.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.779+0000 7f0f1d4d6700 1 --2- 172.21.15.16:0/2527589297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f1806b380 0x7f0f18101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:07:45.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.779+0000 7f0f1dcd7700 1 --2- 172.21.15.16:0/2527589297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f1806a9c0 0x7f0f181011f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0f1400c9b0 tx=0x7f0f1400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:07:45.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.780+0000 7f0f0effd700 1 -- 172.21.15.16:0/2527589297 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f14007b00 con 0x7f0f1806a9c0 2024-03-28T22:07:45.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.780+0000 7f0f0effd700 1 -- 172.21.15.16:0/2527589297 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f14007ca0 con 0x7f0f1806a9c0 2024-03-28T22:07:45.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.780+0000 7f0f1ff3b700 1 -- 172.21.15.16:0/2527589297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f181a7260 con 0x7f0f1806a9c0 2024-03-28T22:07:45.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.780+0000 7f0f0effd700 1 -- 172.21.15.16:0/2527589297 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f14018730 con 0x7f0f1806a9c0 2024-03-28T22:07:45.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.780+0000 7f0f1ff3b700 1 -- 172.21.15.16:0/2527589297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f181a77a0 con 0x7f0f1806a9c0 2024-03-28T22:07:45.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.781+0000 7f0f1ff3b700 1 -- 172.21.15.16:0/2527589297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f18068f00 con 0x7f0f1806a9c0 2024-03-28T22:07:45.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.782+0000 7f0f0effd700 1 -- 172.21.15.16:0/2527589297 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f0f140188d0 con 0x7f0f1806a9c0 2024-03-28T22:07:45.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.782+0000 7f0f0effd700 1 --2- 172.21.15.16:0/2527589297 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0f040a1e10 0x7f0f040a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:45.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.782+0000 7f0f1d4d6700 1 --2- 172.21.15.16:0/2527589297 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0f040a1e10 0x7f0f040a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:45.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.782+0000 7f0f0effd700 1 -- 172.21.15.16:0/2527589297 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f0f140cd870 con 0x7f0f1806a9c0 2024-03-28T22:07:45.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.783+0000 7f0f1d4d6700 1 --2- 172.21.15.16:0/2527589297 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0f040a1e10 0x7f0f040a42d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0f08006010 tx=0x7f0f0800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:07:45.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.787+0000 7f0f0effd700 1 -- 172.21.15.16:0/2527589297 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0f14014030 con 0x7f0f1806a9c0 2024-03-28T22:07:45.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.991+0000 7f0f1ff3b700 1 -- 172.21.15.16:0/2527589297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0f1802db30 con 0x7f0f1806a9c0 2024-03-28T22:07:45.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.992+0000 7f0f0effd700 1 -- 172.21.15.16:0/2527589297 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18348 (secure 0 0 0) 0x7f0f140ba070 con 0x7f0f1806a9c0 2024-03-28T22:07:45.994 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:07:45.994 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":82,"fsid":"798d9884-ed4e-11ee-95d0-87774f69a715","created":"2024-03-28T22:00:27.216099+0000","modified":"2024-03-28T22:07:41.393002+0000","last_up_change":"2024-03-28T22:07:40.251141+0000","last_in_change":"2024-03-28T22:07:19.456451+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:03:59.327832+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"46e7c8f8-ecc8-43d6-ae53-0655728f3240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6803","nonce":1589692483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6805","nonce":1589692483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6809","nonce":1589692483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6807","nonce":1589692483}]},"public_addr":"172.21.15.16:6803/1589692483","cluster_addr":"172.21.15.16:6805/1589692483","heartbeat_back_addr":"172.21.15.16:6809/1589692483","heartbeat_front_addr":"172.21.15.16:6807/1589692483","state":["exists","up"]},{"osd":1,"uuid":"af6d9c22-2830-4ce2-a547-011df0c6cbad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6801","nonce":795531234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6803","nonce":795531234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6807","nonce":795531234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6805","nonce":795531234}]},"public_addr":"172.21.15.53:6801/795531234","cluster_addr":"172.21.15.53:6803/795531234","heartbeat_back_addr":"172.21.15.53:6807/795531234","heartbeat_front_addr":"172.21.15.53:6805/795531234","state":["exists","up"]},{"osd":2,"uuid":"b10adca9-110c-4f87-85e0-b7542709d31a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6801","nonce":2214131086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6803","nonce":2214131086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6807","nonce":2214131086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6805","nonce":2214131086}]},"public_addr":"172.21.15.63:6801/2214131086","cluster_addr":"172.21.15.63:6803/2214131086","heartbeat_back_addr":"172.21.15.63:6807/2214131086","heartbeat_front_addr":"172.21.15.63:6805/2214131086","state":["exists","up"]},{"osd":3,"uuid":"c83cf4bb-395a-4786-abf7-74ca93a33eb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6811","nonce":13176789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6813","nonce":13176789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6817","nonce":13176789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6815","nonce":13176789}]},"public_addr":"172.21.15.16:6811/13176789","cluster_addr":"172.21.15.16:6813/13176789","heartbeat_back_addr":"172.21.15.16:6817/13176789","heartbeat_front_addr":"172.21.15.16:6815/13176789","state":["exists","up"]},{"osd":4,"uuid":"941b3e54-47d6-4e85-8069-e730c3fa8eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6809","nonce":772161849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6811","nonce":772161849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6815","nonce":772161849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6813","nonce":772161849}]},"public_addr":"172.21.15.53:6809/772161849","cluster_addr":"172.21.15.53:6811/772161849","heartbeat_back_addr":"172.21.15.53:6815/772161849","heartbeat_front_addr":"172.21.15.53:6813/772161849","state":["exists","up"]},{"osd":5,"uuid":"51d89826-1d60-40fc-8fd8-09bfe1ab4125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6809","nonce":1602554852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6811","nonce":1602554852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6815","nonce":1602554852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6813","nonce":1602554852}]},"public_addr":"172.21.15.63:6809/1602554852","cluster_addr":"172.21.15.63:6811/1602554852","heartbeat_back_addr":"172.21.15.63:6815/1602554852","heartbeat_front_addr":"172.21.15.63:6813/1602554852","state":["exists","up"]},{"osd":6,"uuid":"978efe5d-2c55-4ff8-8a1e-9f891388945e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6819","nonce":4290459224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6821","nonce":4290459224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6825","nonce":4290459224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6823","nonce":4290459224}]},"public_addr":"172.21.15.16:6819/4290459224","cluster_addr":"172.21.15.16:6821/4290459224","heartbeat_back_addr":"172.21.15.16:6825/4290459224","heartbeat_front_addr":"172.21.15.16:6823/4290459224","state":["exists","up"]},{"osd":7,"uuid":"50d489cb-dbb8-48df-b868-b29edf12e04d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6817","nonce":1056467525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6819","nonce":1056467525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6823","nonce":1056467525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6821","nonce":1056467525}]},"public_addr":"172.21.15.53:6817/1056467525","cluster_addr":"172.21.15.53:6819/1056467525","heartbeat_back_addr":"172.21.15.53:6823/1056467525","heartbeat_front_addr":"172.21.15.53:6821/1056467525","state":["exists","up"]},{"osd":8,"uuid":"49b39414-cc9c-4057-ae3f-cc109cd979da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6817","nonce":2918516794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6819","nonce":2918516794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6823","nonce":2918516794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6821","nonce":2918516794}]},"public_addr":"172.21.15.63:6817/2918516794","cluster_addr":"172.21.15.63:6819/2918516794","heartbeat_back_addr":"172.21.15.63:6823/2918516794","heartbeat_front_addr":"172.21.15.63:6821/2918516794","state":["exists","up"]},{"osd":9,"uuid":"4c873e51-b783-4eb6-83a3-fee31722d60b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6827","nonce":1585624367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6829","nonce":1585624367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6833","nonce":1585624367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6831","nonce":1585624367}]},"public_addr":"172.21.15.16:6827/1585624367","cluster_addr":"172.21.15.16:6829/1585624367","heartbeat_back_addr":"172.21.15.16:6833/1585624367","heartbeat_front_addr":"172.21.15.16:6831/1585624367","state":["exists","up"]},{"osd":10,"uuid":"085a78bf-e2ad-481a-85a3-71d1fc549a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6825","nonce":3357446484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6827","nonce":3357446484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6831","nonce":3357446484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6829","nonce":3357446484}]},"public_addr":"172.21.15.53:6825/3357446484","cluster_addr":"172.21.15.53:6827/3357446484","heartbeat_back_addr":"172.21.15.53:6831/3357446484","heartbeat_front_addr":"172.21.15.53:6829/3357446484","state":["exists","up"]},{"osd":11,"uuid":"13302aa3-08b9-4639-9042-27d850b4d5f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6825","nonce":2167866570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6827","nonce":2167866570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6831","nonce":2167866570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6829","nonce":2167866570}]},"public_addr":"172.21.15.63:6825/2167866570","cluster_addr":"172.21.15.63:6827/2167866570","heartbeat_back_addr":"172.21.15.63:6831/2167866570","heartbeat_front_addr":"172.21.15.63:6829/2167866570","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:07.450323+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:31.363172+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:55.295021+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:19.794760+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:44.063492+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:08.195064+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:33.655328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:57.639414+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:22.402394+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:47.462050+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:12.226307+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:37.495625+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3821408150":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6801/1832422779":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/996598130":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/1062264551":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/317557779":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6801/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/3631746518":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6800/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/3116652786":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6800/1832422779":"2024-03-29T22:01:28.993434+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:07:45.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.994+0000 7f0f1ff3b700 1 -- 172.21.15.16:0/2527589297 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0f040a1e10 msgr2=0x7f0f040a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:45.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.994+0000 7f0f1ff3b700 1 --2- 172.21.15.16:0/2527589297 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0f040a1e10 0x7f0f040a42d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0f08006010 tx=0x7f0f0800b540 comp rx=0 tx=0).stop 2024-03-28T22:07:45.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.994+0000 7f0f1ff3b700 1 -- 172.21.15.16:0/2527589297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f1806a9c0 msgr2=0x7f0f181011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:45.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.994+0000 7f0f1ff3b700 1 --2- 172.21.15.16:0/2527589297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f1806a9c0 0x7f0f181011f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0f1400c9b0 tx=0x7f0f1400ccc0 comp rx=0 tx=0).stop 2024-03-28T22:07:45.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.994+0000 7f0f1ff3b700 1 -- 172.21.15.16:0/2527589297 shutdown_connections 2024-03-28T22:07:45.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.994+0000 7f0f1ff3b700 1 --2- 172.21.15.16:0/2527589297 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0f040a1e10 0x7f0f040a42d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:45.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.994+0000 7f0f1ff3b700 1 --2- 172.21.15.16:0/2527589297 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f18107250 0x7f0f18101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:45.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.994+0000 7f0f1ff3b700 1 --2- 172.21.15.16:0/2527589297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f1806a9c0 0x7f0f181011f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:45.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.994+0000 7f0f1ff3b700 1 --2- 172.21.15.16:0/2527589297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f1806b380 0x7f0f18101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:45.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.994+0000 7f0f1ff3b700 1 -- 172.21.15.16:0/2527589297 >> 172.21.15.16:0/2527589297 conn(0x7f0f180781a0 msgr2=0x7f0f180ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:07:45.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.994+0000 7f0f1ff3b700 1 -- 172.21.15.16:0/2527589297 shutdown_connections 2024-03-28T22:07:45.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:45.994+0000 7f0f1ff3b700 1 -- 172.21.15.16:0/2527589297 wait complete. 2024-03-28T22:07:46.288 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:46 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2527589297' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:07:46.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:46 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2527589297' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:07:46.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:46 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2527589297' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:07:46.625 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T22:03:59.327832+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T22:07:46.626 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-28T22:07:46.869 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:07:47.317 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:47 smithi016 ceph-mon[32374]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:07:47.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:47 smithi063 ceph-mon[36249]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:07:47.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:47 smithi053 ceph-mon[35979]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:07:48.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.411+0000 7f31ac0ae700 1 -- 172.21.15.16:0/2325863818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a4101120 msgr2=0x7f31a41015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:48.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.411+0000 7f31ac0ae700 1 --2- 172.21.15.16:0/2325863818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a4101120 0x7f31a41015a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3194009990 tx=0x7f3194009ca0 comp rx=0 tx=0).stop 2024-03-28T22:07:48.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.411+0000 7f31ac0ae700 1 -- 172.21.15.16:0/2325863818 shutdown_connections 2024-03-28T22:07:48.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.411+0000 7f31ac0ae700 1 --2- 172.21.15.16:0/2325863818 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f31a4101ae0 0x7f31a410df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:48.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.411+0000 7f31ac0ae700 1 --2- 172.21.15.16:0/2325863818 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31a41071d0 0x7f31a41075b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:48.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.411+0000 7f31ac0ae700 1 --2- 172.21.15.16:0/2325863818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a4101120 0x7f31a41015a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:48.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.411+0000 7f31ac0ae700 1 -- 172.21.15.16:0/2325863818 >> 172.21.15.16:0/2325863818 conn(0x7f31a4078150 msgr2=0x7f31a4078560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:07:48.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.411+0000 7f31ac0ae700 1 -- 172.21.15.16:0/2325863818 shutdown_connections 2024-03-28T22:07:48.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.411+0000 7f31ac0ae700 1 -- 172.21.15.16:0/2325863818 wait complete. 2024-03-28T22:07:48.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.411+0000 7f31ac0ae700 1 Processor -- start 2024-03-28T22:07:48.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.411+0000 7f31ac0ae700 1 -- start start 2024-03-28T22:07:48.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.412+0000 7f31ac0ae700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f31a4101120 0x7f31a41a07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:48.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.412+0000 7f31ac0ae700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a4101ae0 0x7f31a41a0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:48.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.412+0000 7f31ac0ae700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31a41071d0 0x7f31a419a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:48.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.412+0000 7f31ac0ae700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f31a4111410 con 0x7f31a4101ae0 2024-03-28T22:07:48.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.412+0000 7f31ac0ae700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f31a4111290 con 0x7f31a41071d0 2024-03-28T22:07:48.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.412+0000 7f31ac0ae700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f31a4111590 con 0x7f31a4101120 2024-03-28T22:07:48.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.412+0000 7f31a9649700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a4101ae0 0x7f31a41a0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:48.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.412+0000 7f31a9649700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a4101ae0 0x7f31a41a0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36414/0 (socket says 172.21.15.16:36414) 2024-03-28T22:07:48.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.412+0000 7f31a9649700 1 -- 172.21.15.16:0/3122525027 learned_addr learned my addr 172.21.15.16:0/3122525027 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:07:48.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.412+0000 7f31a9e4a700 1 --2- 172.21.15.16:0/3122525027 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f31a4101120 0x7f31a41a07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:48.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.412+0000 7f31aa64b700 1 --2- 172.21.15.16:0/3122525027 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31a41071d0 0x7f31a419a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:48.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.412+0000 7f31a9649700 1 -- 172.21.15.16:0/3122525027 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f31a4101120 msgr2=0x7f31a41a07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:48.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.412+0000 7f31a9649700 1 --2- 172.21.15.16:0/3122525027 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f31a4101120 0x7f31a41a07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:48.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.412+0000 7f31a9649700 1 -- 172.21.15.16:0/3122525027 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31a41071d0 msgr2=0x7f31a419a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:48.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.412+0000 7f31a9649700 1 --2- 172.21.15.16:0/3122525027 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31a41071d0 0x7f31a419a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:48.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.412+0000 7f31a9649700 1 -- 172.21.15.16:0/3122525027 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31a419b0e0 con 0x7f31a4101ae0 2024-03-28T22:07:48.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.412+0000 7f31aa64b700 1 --2- 172.21.15.16:0/3122525027 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31a41071d0 0x7f31a419a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:07:48.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.412+0000 7f31a9649700 1 --2- 172.21.15.16:0/3122525027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a4101ae0 0x7f31a41a0d00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3194000c00 tx=0x7f31940047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:07:48.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.413+0000 7f319affd700 1 -- 172.21.15.16:0/3122525027 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3194027070 con 0x7f31a4101ae0 2024-03-28T22:07:48.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.413+0000 7f319affd700 1 -- 172.21.15.16:0/3122525027 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3194003780 con 0x7f31a4101ae0 2024-03-28T22:07:48.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.413+0000 7f31ac0ae700 1 -- 172.21.15.16:0/3122525027 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31a419b370 con 0x7f31a4101ae0 2024-03-28T22:07:48.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.413+0000 7f319affd700 1 -- 172.21.15.16:0/3122525027 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3194022dd0 con 0x7f31a4101ae0 2024-03-28T22:07:48.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.413+0000 7f31ac0ae700 1 -- 172.21.15.16:0/3122525027 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31a41a7690 con 0x7f31a4101ae0 2024-03-28T22:07:48.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.414+0000 7f31ac0ae700 1 -- 172.21.15.16:0/3122525027 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31a4068f00 con 0x7f31a4101ae0 2024-03-28T22:07:48.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.415+0000 7f319affd700 1 -- 172.21.15.16:0/3122525027 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f319402a070 con 0x7f31a4101ae0 2024-03-28T22:07:48.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.415+0000 7f319affd700 1 --2- 172.21.15.16:0/3122525027 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f31900a1e90 0x7f31900a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:48.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.415+0000 7f31a9e4a700 1 --2- 172.21.15.16:0/3122525027 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f31900a1e90 0x7f31900a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:48.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.416+0000 7f319affd700 1 -- 172.21.15.16:0/3122525027 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f31940ce990 con 0x7f31a4101ae0 2024-03-28T22:07:48.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.416+0000 7f31a9e4a700 1 --2- 172.21.15.16:0/3122525027 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f31900a1e90 0x7f31900a4350 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f31a419bac0 tx=0x7f31a00073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:07:48.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.419+0000 7f319affd700 1 -- 172.21.15.16:0/3122525027 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3194013070 con 0x7f31a4101ae0 2024-03-28T22:07:48.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.619+0000 7f31ac0ae700 1 -- 172.21.15.16:0/3122525027 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f31a41a78c0 con 0x7f31a4101ae0 2024-03-28T22:07:48.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.620+0000 7f319affd700 1 -- 172.21.15.16:0/3122525027 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f319409c2e0 con 0x7f31a4101ae0 2024-03-28T22:07:48.622 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-28T22:07:48.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.622+0000 7f31ac0ae700 1 -- 172.21.15.16:0/3122525027 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f31900a1e90 msgr2=0x7f31900a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:48.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.622+0000 7f31ac0ae700 1 --2- 172.21.15.16:0/3122525027 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f31900a1e90 0x7f31900a4350 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f31a419bac0 tx=0x7f31a00073d0 comp rx=0 tx=0).stop 2024-03-28T22:07:48.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.622+0000 7f31ac0ae700 1 -- 172.21.15.16:0/3122525027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a4101ae0 msgr2=0x7f31a41a0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:48.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.622+0000 7f31ac0ae700 1 --2- 172.21.15.16:0/3122525027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a4101ae0 0x7f31a41a0d00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3194000c00 tx=0x7f31940047b0 comp rx=0 tx=0).stop 2024-03-28T22:07:48.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.622+0000 7f31ac0ae700 1 -- 172.21.15.16:0/3122525027 shutdown_connections 2024-03-28T22:07:48.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.622+0000 7f31ac0ae700 1 --2- 172.21.15.16:0/3122525027 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f31900a1e90 0x7f31900a4350 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:48.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.622+0000 7f31ac0ae700 1 --2- 172.21.15.16:0/3122525027 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31a41071d0 0x7f31a419a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:48.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.622+0000 7f31ac0ae700 1 --2- 172.21.15.16:0/3122525027 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f31a4101120 0x7f31a41a07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:48.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.622+0000 7f31ac0ae700 1 --2- 172.21.15.16:0/3122525027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a4101ae0 0x7f31a41a0d00 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:48.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.622+0000 7f31ac0ae700 1 -- 172.21.15.16:0/3122525027 >> 172.21.15.16:0/3122525027 conn(0x7f31a4078150 msgr2=0x7f31a406a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:07:48.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.622+0000 7f31ac0ae700 1 -- 172.21.15.16:0/3122525027 shutdown_connections 2024-03-28T22:07:48.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:48.622+0000 7f31ac0ae700 1 -- 172.21.15.16:0/3122525027 wait complete. 2024-03-28T22:07:48.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:48 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/3122525027' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:07:48.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:48 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3122525027' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:07:49.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:48 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3122525027' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:07:49.258 INFO:tasks.cephadm:Adding mds.a on smithi016 2024-03-28T22:07:49.259 INFO:tasks.cephadm:Adding mds.d on smithi016 2024-03-28T22:07:49.259 INFO:tasks.cephadm:Adding mds.g on smithi016 2024-03-28T22:07:49.259 INFO:tasks.cephadm:Adding mds.j on smithi016 2024-03-28T22:07:49.259 INFO:tasks.cephadm:Adding mds.b on smithi053 2024-03-28T22:07:49.259 INFO:tasks.cephadm:Adding mds.e on smithi053 2024-03-28T22:07:49.259 INFO:tasks.cephadm:Adding mds.h on smithi053 2024-03-28T22:07:49.259 INFO:tasks.cephadm:Adding mds.k on smithi053 2024-03-28T22:07:49.259 INFO:tasks.cephadm:Adding mds.c on smithi063 2024-03-28T22:07:49.259 INFO:tasks.cephadm:Adding mds.f on smithi063 2024-03-28T22:07:49.259 INFO:tasks.cephadm:Adding mds.i on smithi063 2024-03-28T22:07:49.259 INFO:tasks.cephadm:Adding mds.l on smithi063 2024-03-28T22:07:49.260 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph orch apply mds all '12;smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi053=b;smithi053=e;smithi053=h;smithi053=k;smithi063=c;smithi063=f;smithi063=i;smithi063=l' 2024-03-28T22:07:49.525 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T22:07:49.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:49 smithi063 ceph-mon[36249]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:07:49.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:49 smithi016 ceph-mon[32374]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:07:50.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:49 smithi053 ceph-mon[35979]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:07:50.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.948+0000 7f57590c2700 1 -- 172.21.15.63:0/1754183945 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5754105630 msgr2=0x7f5754107a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:50.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.948+0000 7f57590c2700 1 --2- 172.21.15.63:0/1754183945 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5754105630 0x7f5754107a20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5748009990 tx=0x7f5748009ca0 comp rx=0 tx=0).stop 2024-03-28T22:07:50.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.948+0000 7f57590c2700 1 -- 172.21.15.63:0/1754183945 shutdown_connections 2024-03-28T22:07:50.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.948+0000 7f57590c2700 1 --2- 172.21.15.63:0/1754183945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5754107f60 0x7f575410a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:50.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.948+0000 7f57590c2700 1 --2- 172.21.15.63:0/1754183945 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5754102d00 0x7f57541050f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:50.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.948+0000 7f57590c2700 1 --2- 172.21.15.63:0/1754183945 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5754105630 0x7f5754107a20 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:50.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.948+0000 7f57590c2700 1 -- 172.21.15.63:0/1754183945 >> 172.21.15.63:0/1754183945 conn(0x7f57540fc710 msgr2=0x7f57540feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:07:50.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.948+0000 7f57590c2700 1 -- 172.21.15.63:0/1754183945 shutdown_connections 2024-03-28T22:07:50.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.948+0000 7f57590c2700 1 -- 172.21.15.63:0/1754183945 wait complete. 2024-03-28T22:07:50.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.949+0000 7f57590c2700 1 Processor -- start 2024-03-28T22:07:50.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.949+0000 7f57590c2700 1 -- start start 2024-03-28T22:07:50.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.949+0000 7f57590c2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5754102d00 0x7f5754198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:50.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.949+0000 7f57590c2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5754105630 0x7f5754198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:50.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.949+0000 7f57590c2700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5754107f60 0x7f575419fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:50.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.949+0000 7f57590c2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f575410dec0 con 0x7f5754105630 2024-03-28T22:07:50.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.949+0000 7f57590c2700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f575410dd40 con 0x7f5754102d00 2024-03-28T22:07:50.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.949+0000 7f57590c2700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f575410e040 con 0x7f5754107f60 2024-03-28T22:07:50.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.949+0000 7f575359e700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5754107f60 0x7f575419fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:50.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.949+0000 7f575359e700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5754107f60 0x7f575419fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33540/0 (socket says 172.21.15.63:33540) 2024-03-28T22:07:50.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.949+0000 7f575259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5754105630 0x7f5754198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:50.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.949+0000 7f575359e700 1 -- 172.21.15.63:0/3933522761 learned_addr learned my addr 172.21.15.63:0/3933522761 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-28T22:07:50.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.949+0000 7f5752d9d700 1 --2- 172.21.15.63:0/3933522761 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5754102d00 0x7f5754198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:50.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.950+0000 7f575359e700 1 -- 172.21.15.63:0/3933522761 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5754102d00 msgr2=0x7f5754198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:50.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.950+0000 7f575359e700 1 --2- 172.21.15.63:0/3933522761 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5754102d00 0x7f5754198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:50.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.950+0000 7f575359e700 1 -- 172.21.15.63:0/3933522761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5754105630 msgr2=0x7f5754198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:50.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.950+0000 7f575359e700 1 --2- 172.21.15.63:0/3933522761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5754105630 0x7f5754198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:50.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.950+0000 7f575359e700 1 -- 172.21.15.63:0/3933522761 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57541a0180 con 0x7f5754107f60 2024-03-28T22:07:50.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.950+0000 7f575359e700 1 --2- 172.21.15.63:0/3933522761 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5754107f60 0x7f575419fae0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f574c00ba10 tx=0x7f574c00bd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:07:50.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.950+0000 7f573bfff700 1 -- 172.21.15.63:0/3933522761 <== mon.1 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f574c00d8e0 con 0x7f5754107f60 2024-03-28T22:07:50.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.950+0000 7f57590c2700 1 -- 172.21.15.63:0/3933522761 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57541a0470 con 0x7f5754107f60 2024-03-28T22:07:50.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.950+0000 7f573bfff700 1 -- 172.21.15.63:0/3933522761 <== mon.1 v2:172.21.15.63:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f574c015420 con 0x7f5754107f60 2024-03-28T22:07:50.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.950+0000 7f573bfff700 1 -- 172.21.15.63:0/3933522761 <== mon.1 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f574c014630 con 0x7f5754107f60 2024-03-28T22:07:50.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.950+0000 7f57590c2700 1 -- 172.21.15.63:0/3933522761 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57541a09b0 con 0x7f5754107f60 2024-03-28T22:07:50.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.951+0000 7f57590c2700 1 -- 172.21.15.63:0/3933522761 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5754068f00 con 0x7f5754107f60 2024-03-28T22:07:50.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.952+0000 7f573bfff700 1 -- 172.21.15.63:0/3933522761 <== mon.1 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f574c015980 con 0x7f5754107f60 2024-03-28T22:07:50.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.953+0000 7f573bfff700 1 --2- 172.21.15.63:0/3933522761 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f573c0a1e10 0x7f573c0a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:50.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.953+0000 7f573bfff700 1 -- 172.21.15.63:0/3933522761 <== mon.1 v2:172.21.15.63:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f574c0cd210 con 0x7f5754107f60 2024-03-28T22:07:50.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.953+0000 7f5752d9d700 1 --2- 172.21.15.63:0/3933522761 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f573c0a1e10 0x7f573c0a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:50.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.953+0000 7f5752d9d700 1 --2- 172.21.15.63:0/3933522761 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f573c0a1e10 0x7f573c0a42d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5744007c60 tx=0x7f57440073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:07:50.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:50.957+0000 7f573bfff700 1 -- 172.21.15.63:0/3933522761 <== mon.1 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f574c09aae0 con 0x7f5754107f60 2024-03-28T22:07:51.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:51.173+0000 7f57590c2700 1 -- 172.21.15.63:0/3933522761 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi053=b;smithi053=e;smithi053=h;smithi053=k;smithi063=c;smithi063=f;smithi063=i;smithi063=l", "target": ["mon-mgr", ""]}) -- 0x7f5754062840 con 0x7f573c0a1e10 2024-03-28T22:07:51.175 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:51 smithi063 ceph-mon[36249]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:07:51.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:51.180+0000 7f573bfff700 1 -- 172.21.15.63:0/3933522761 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f5754062840 con 0x7f573c0a1e10 2024-03-28T22:07:51.182 INFO:teuthology.orchestra.run.smithi063.stdout:Scheduled mds.all update... 2024-03-28T22:07:51.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:51.182+0000 7f57590c2700 1 -- 172.21.15.63:0/3933522761 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f573c0a1e10 msgr2=0x7f573c0a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:51.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:51.182+0000 7f57590c2700 1 --2- 172.21.15.63:0/3933522761 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f573c0a1e10 0x7f573c0a42d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5744007c60 tx=0x7f57440073d0 comp rx=0 tx=0).stop 2024-03-28T22:07:51.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:51.182+0000 7f57590c2700 1 -- 172.21.15.63:0/3933522761 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5754107f60 msgr2=0x7f575419fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:51.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:51.182+0000 7f57590c2700 1 --2- 172.21.15.63:0/3933522761 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5754107f60 0x7f575419fae0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f574c00ba10 tx=0x7f574c00bd20 comp rx=0 tx=0).stop 2024-03-28T22:07:51.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:51.182+0000 7f57590c2700 1 -- 172.21.15.63:0/3933522761 shutdown_connections 2024-03-28T22:07:51.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:51.182+0000 7f57590c2700 1 --2- 172.21.15.63:0/3933522761 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f573c0a1e10 0x7f573c0a42d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:51.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:51.182+0000 7f57590c2700 1 --2- 172.21.15.63:0/3933522761 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5754107f60 0x7f575419fae0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:51.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:51.182+0000 7f57590c2700 1 --2- 172.21.15.63:0/3933522761 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5754102d00 0x7f5754198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:51.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:51.182+0000 7f57590c2700 1 --2- 172.21.15.63:0/3933522761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5754105630 0x7f5754198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:51.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:51.182+0000 7f57590c2700 1 -- 172.21.15.63:0/3933522761 >> 172.21.15.63:0/3933522761 conn(0x7f57540fc710 msgr2=0x7f5754103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:07:51.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:51.183+0000 7f57590c2700 1 -- 172.21.15.63:0/3933522761 shutdown_connections 2024-03-28T22:07:51.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-28T22:07:51.183+0000 7f57590c2700 1 -- 172.21.15.63:0/3933522761 wait complete. 2024-03-28T22:07:51.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:51 smithi016 ceph-mon[32374]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:07:51.551 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:51 smithi053 ceph-mon[35979]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:07:51.790 DEBUG:teuthology.orchestra.run.smithi016:mds.a> sudo journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mds.a.service 2024-03-28T22:07:51.793 DEBUG:teuthology.orchestra.run.smithi016:mds.d> sudo journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mds.d.service 2024-03-28T22:07:51.835 DEBUG:teuthology.orchestra.run.smithi016:mds.g> sudo journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mds.g.service 2024-03-28T22:07:51.878 DEBUG:teuthology.orchestra.run.smithi016:mds.j> sudo journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mds.j.service 2024-03-28T22:07:51.921 DEBUG:teuthology.orchestra.run.smithi053:mds.b> sudo journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mds.b.service 2024-03-28T22:07:51.924 DEBUG:teuthology.orchestra.run.smithi053:mds.e> sudo journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mds.e.service 2024-03-28T22:07:51.967 DEBUG:teuthology.orchestra.run.smithi053:mds.h> sudo journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mds.h.service 2024-03-28T22:07:52.010 DEBUG:teuthology.orchestra.run.smithi053:mds.k> sudo journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mds.k.service 2024-03-28T22:07:52.054 DEBUG:teuthology.orchestra.run.smithi063:mds.c> sudo journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mds.c.service 2024-03-28T22:07:52.056 DEBUG:teuthology.orchestra.run.smithi063:mds.f> sudo journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mds.f.service 2024-03-28T22:07:52.099 DEBUG:teuthology.orchestra.run.smithi063:mds.i> sudo journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mds.i.service 2024-03-28T22:07:52.142 DEBUG:teuthology.orchestra.run.smithi063:mds.l> sudo journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mds.l.service 2024-03-28T22:07:52.185 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-28T22:07:52.185 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:07:52.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:52 smithi053 ceph-mon[35979]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi053=b;smithi053=e;smithi053=h;smithi053=k;smithi063=c;smithi063=f;smithi063=i;smithi063=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:07:52.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:52 smithi053 ceph-mon[35979]: Saving service mds.all spec with placement smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi053=b;smithi053=e;smithi053=h;smithi053=k;smithi063=c;smithi063=f;smithi063=i;smithi063=l;count:12 2024-03-28T22:07:52.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:52 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:52.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:52 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:07:52.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:52 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:07:52.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:52 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:07:52.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:52 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:52.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:52 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:07:52.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:52 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:07:52.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:52 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:07:52.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:52 smithi053 ceph-mon[35979]: Deploying daemon mds.l on smithi063 2024-03-28T22:07:52.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.300+0000 7f587828f640 1 -- 172.21.15.16:0/3363619558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58701029d0 msgr2=0x7f5870102e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:52.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.300+0000 7f587828f640 1 --2- 172.21.15.16:0/3363619558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58701029d0 0x7f5870102e50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5860009900 tx=0x7f586002bc30 comp rx=0 tx=0).stop 2024-03-28T22:07:52.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.300+0000 7f587828f640 1 -- 172.21.15.16:0/3363619558 shutdown_connections 2024-03-28T22:07:52.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.301+0000 7f587828f640 1 --2- 172.21.15.16:0/3363619558 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5870103390 0x7f5870109c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:52.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.301+0000 7f587828f640 1 --2- 172.21.15.16:0/3363619558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58701029d0 0x7f5870102e50 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:52.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.301+0000 7f587828f640 1 --2- 172.21.15.16:0/3363619558 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58701017e0 0x7f5870101c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:52.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.301+0000 7f587828f640 1 -- 172.21.15.16:0/3363619558 >> 172.21.15.16:0/3363619558 conn(0x7f58700fcd70 msgr2=0x7f58700ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:07:52.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.301+0000 7f587828f640 1 -- 172.21.15.16:0/3363619558 shutdown_connections 2024-03-28T22:07:52.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.301+0000 7f587828f640 1 -- 172.21.15.16:0/3363619558 wait complete. 2024-03-28T22:07:52.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.301+0000 7f587828f640 1 Processor -- start 2024-03-28T22:07:52.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.301+0000 7f587828f640 1 -- start start 2024-03-28T22:07:52.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.301+0000 7f587828f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58701017e0 0x7f5870197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:52.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.301+0000 7f587828f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58701029d0 0x7f5870197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:52.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.301+0000 7f587828f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5870103390 0x7f587019efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:52.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.301+0000 7f587828f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f587010d3f0 con 0x7f5870103390 2024-03-28T22:07:52.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.301+0000 7f587828f640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f587010d270 con 0x7f58701017e0 2024-03-28T22:07:52.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.301+0000 7f587828f640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f587010d570 con 0x7f58701029d0 2024-03-28T22:07:52.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.301+0000 7f587682c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5870103390 0x7f587019efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:52.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.301+0000 7f587682c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5870103390 0x7f587019efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38636/0 (socket says 172.21.15.16:38636) 2024-03-28T22:07:52.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.301+0000 7f587682c640 1 -- 172.21.15.16:0/1138220929 learned_addr learned my addr 172.21.15.16:0/1138220929 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:07:52.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.301+0000 7f587582a640 1 --2- 172.21.15.16:0/1138220929 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58701029d0 0x7f5870197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:52.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.302+0000 7f587602b640 1 --2- 172.21.15.16:0/1138220929 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58701017e0 0x7f5870197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:52.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.302+0000 7f587682c640 1 -- 172.21.15.16:0/1138220929 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58701029d0 msgr2=0x7f5870197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:52.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.302+0000 7f587682c640 1 --2- 172.21.15.16:0/1138220929 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58701029d0 0x7f5870197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:52.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.302+0000 7f587682c640 1 -- 172.21.15.16:0/1138220929 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58701017e0 msgr2=0x7f5870197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:52.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.302+0000 7f587682c640 1 --2- 172.21.15.16:0/1138220929 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58701017e0 0x7f5870197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:52.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.302+0000 7f587682c640 1 -- 172.21.15.16:0/1138220929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f587019f630 con 0x7f5870103390 2024-03-28T22:07:52.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.302+0000 7f587682c640 1 --2- 172.21.15.16:0/1138220929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5870103390 0x7f587019efc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f586800eb10 tx=0x7f586800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:07:52.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.302+0000 7f58677fe640 1 -- 172.21.15.16:0/1138220929 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5868004b20 con 0x7f5870103390 2024-03-28T22:07:52.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.302+0000 7f58677fe640 1 -- 172.21.15.16:0/1138220929 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5868004cc0 con 0x7f5870103390 2024-03-28T22:07:52.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.302+0000 7f587828f640 1 -- 172.21.15.16:0/1138220929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f587019f920 con 0x7f5870103390 2024-03-28T22:07:52.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.302+0000 7f58677fe640 1 -- 172.21.15.16:0/1138220929 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5868010760 con 0x7f5870103390 2024-03-28T22:07:52.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.302+0000 7f587828f640 1 -- 172.21.15.16:0/1138220929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f587019fe40 con 0x7f5870103390 2024-03-28T22:07:52.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.303+0000 7f587828f640 1 -- 172.21.15.16:0/1138220929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5870108aa0 con 0x7f5870103390 2024-03-28T22:07:52.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.304+0000 7f58677fe640 1 -- 172.21.15.16:0/1138220929 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f5868005020 con 0x7f5870103390 2024-03-28T22:07:52.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.304+0000 7f58677fe640 1 --2- 172.21.15.16:0/1138220929 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f585c0a1d80 0x7f585c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:52.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.305+0000 7f587602b640 1 --2- 172.21.15.16:0/1138220929 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f585c0a1d80 0x7f585c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:52.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.305+0000 7f58677fe640 1 -- 172.21.15.16:0/1138220929 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f58680ce0c0 con 0x7f5870103390 2024-03-28T22:07:52.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.305+0000 7f587602b640 1 --2- 172.21.15.16:0/1138220929 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f585c0a1d80 0x7f585c0a4240 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f586c0095f0 tx=0x7f586c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:07:52.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.309+0000 7f58677fe640 1 -- 172.21.15.16:0/1138220929 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f586809b990 con 0x7f5870103390 2024-03-28T22:07:52.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:52 smithi063 ceph-mon[36249]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi053=b;smithi053=e;smithi053=h;smithi053=k;smithi063=c;smithi063=f;smithi063=i;smithi063=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:07:52.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:52 smithi063 ceph-mon[36249]: Saving service mds.all spec with placement smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi053=b;smithi053=e;smithi053=h;smithi053=k;smithi063=c;smithi063=f;smithi063=i;smithi063=l;count:12 2024-03-28T22:07:52.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:52 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:52.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:52 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:07:52.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:52 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:07:52.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:52 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:07:52.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:52 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:52.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:52 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:07:52.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:52 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:07:52.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:52 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:07:52.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:52 smithi063 ceph-mon[36249]: Deploying daemon mds.l on smithi063 2024-03-28T22:07:52.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:52 smithi016 ceph-mon[32374]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi053=b;smithi053=e;smithi053=h;smithi053=k;smithi063=c;smithi063=f;smithi063=i;smithi063=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:07:52.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:52 smithi016 ceph-mon[32374]: Saving service mds.all spec with placement smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi053=b;smithi053=e;smithi053=h;smithi053=k;smithi063=c;smithi063=f;smithi063=i;smithi063=l;count:12 2024-03-28T22:07:52.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:52 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:52.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:52 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:07:52.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:52 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:07:52.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:52 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:07:52.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:52 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:52.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:52 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:07:52.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:52 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:07:52.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:52 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:07:52.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:52 smithi016 ceph-mon[32374]: Deploying daemon mds.l on smithi063 2024-03-28T22:07:52.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.491+0000 7f587828f640 1 -- 172.21.15.16:0/1138220929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5870062350 con 0x7f5870103390 2024-03-28T22:07:52.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.492+0000 7f58677fe640 1 -- 172.21.15.16:0/1138220929 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18348 (secure 0 0 0) 0x7f586809b290 con 0x7f5870103390 2024-03-28T22:07:52.495 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:07:52.495 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":82,"fsid":"798d9884-ed4e-11ee-95d0-87774f69a715","created":"2024-03-28T22:00:27.216099+0000","modified":"2024-03-28T22:07:41.393002+0000","last_up_change":"2024-03-28T22:07:40.251141+0000","last_in_change":"2024-03-28T22:07:19.456451+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:03:59.327832+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"46e7c8f8-ecc8-43d6-ae53-0655728f3240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6803","nonce":1589692483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6805","nonce":1589692483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6809","nonce":1589692483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6807","nonce":1589692483}]},"public_addr":"172.21.15.16:6803/1589692483","cluster_addr":"172.21.15.16:6805/1589692483","heartbeat_back_addr":"172.21.15.16:6809/1589692483","heartbeat_front_addr":"172.21.15.16:6807/1589692483","state":["exists","up"]},{"osd":1,"uuid":"af6d9c22-2830-4ce2-a547-011df0c6cbad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6801","nonce":795531234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6803","nonce":795531234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6807","nonce":795531234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6805","nonce":795531234}]},"public_addr":"172.21.15.53:6801/795531234","cluster_addr":"172.21.15.53:6803/795531234","heartbeat_back_addr":"172.21.15.53:6807/795531234","heartbeat_front_addr":"172.21.15.53:6805/795531234","state":["exists","up"]},{"osd":2,"uuid":"b10adca9-110c-4f87-85e0-b7542709d31a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6801","nonce":2214131086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6803","nonce":2214131086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6807","nonce":2214131086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6805","nonce":2214131086}]},"public_addr":"172.21.15.63:6801/2214131086","cluster_addr":"172.21.15.63:6803/2214131086","heartbeat_back_addr":"172.21.15.63:6807/2214131086","heartbeat_front_addr":"172.21.15.63:6805/2214131086","state":["exists","up"]},{"osd":3,"uuid":"c83cf4bb-395a-4786-abf7-74ca93a33eb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6811","nonce":13176789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6813","nonce":13176789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6817","nonce":13176789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6815","nonce":13176789}]},"public_addr":"172.21.15.16:6811/13176789","cluster_addr":"172.21.15.16:6813/13176789","heartbeat_back_addr":"172.21.15.16:6817/13176789","heartbeat_front_addr":"172.21.15.16:6815/13176789","state":["exists","up"]},{"osd":4,"uuid":"941b3e54-47d6-4e85-8069-e730c3fa8eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6809","nonce":772161849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6811","nonce":772161849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6815","nonce":772161849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6813","nonce":772161849}]},"public_addr":"172.21.15.53:6809/772161849","cluster_addr":"172.21.15.53:6811/772161849","heartbeat_back_addr":"172.21.15.53:6815/772161849","heartbeat_front_addr":"172.21.15.53:6813/772161849","state":["exists","up"]},{"osd":5,"uuid":"51d89826-1d60-40fc-8fd8-09bfe1ab4125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6809","nonce":1602554852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6811","nonce":1602554852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6815","nonce":1602554852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6813","nonce":1602554852}]},"public_addr":"172.21.15.63:6809/1602554852","cluster_addr":"172.21.15.63:6811/1602554852","heartbeat_back_addr":"172.21.15.63:6815/1602554852","heartbeat_front_addr":"172.21.15.63:6813/1602554852","state":["exists","up"]},{"osd":6,"uuid":"978efe5d-2c55-4ff8-8a1e-9f891388945e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6819","nonce":4290459224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6821","nonce":4290459224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6825","nonce":4290459224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6823","nonce":4290459224}]},"public_addr":"172.21.15.16:6819/4290459224","cluster_addr":"172.21.15.16:6821/4290459224","heartbeat_back_addr":"172.21.15.16:6825/4290459224","heartbeat_front_addr":"172.21.15.16:6823/4290459224","state":["exists","up"]},{"osd":7,"uuid":"50d489cb-dbb8-48df-b868-b29edf12e04d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6817","nonce":1056467525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6819","nonce":1056467525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6823","nonce":1056467525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6821","nonce":1056467525}]},"public_addr":"172.21.15.53:6817/1056467525","cluster_addr":"172.21.15.53:6819/1056467525","heartbeat_back_addr":"172.21.15.53:6823/1056467525","heartbeat_front_addr":"172.21.15.53:6821/1056467525","state":["exists","up"]},{"osd":8,"uuid":"49b39414-cc9c-4057-ae3f-cc109cd979da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6817","nonce":2918516794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6819","nonce":2918516794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6823","nonce":2918516794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6821","nonce":2918516794}]},"public_addr":"172.21.15.63:6817/2918516794","cluster_addr":"172.21.15.63:6819/2918516794","heartbeat_back_addr":"172.21.15.63:6823/2918516794","heartbeat_front_addr":"172.21.15.63:6821/2918516794","state":["exists","up"]},{"osd":9,"uuid":"4c873e51-b783-4eb6-83a3-fee31722d60b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6827","nonce":1585624367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6829","nonce":1585624367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6833","nonce":1585624367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6831","nonce":1585624367}]},"public_addr":"172.21.15.16:6827/1585624367","cluster_addr":"172.21.15.16:6829/1585624367","heartbeat_back_addr":"172.21.15.16:6833/1585624367","heartbeat_front_addr":"172.21.15.16:6831/1585624367","state":["exists","up"]},{"osd":10,"uuid":"085a78bf-e2ad-481a-85a3-71d1fc549a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6825","nonce":3357446484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6827","nonce":3357446484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6831","nonce":3357446484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6829","nonce":3357446484}]},"public_addr":"172.21.15.53:6825/3357446484","cluster_addr":"172.21.15.53:6827/3357446484","heartbeat_back_addr":"172.21.15.53:6831/3357446484","heartbeat_front_addr":"172.21.15.53:6829/3357446484","state":["exists","up"]},{"osd":11,"uuid":"13302aa3-08b9-4639-9042-27d850b4d5f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6825","nonce":2167866570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6827","nonce":2167866570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6831","nonce":2167866570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6829","nonce":2167866570}]},"public_addr":"172.21.15.63:6825/2167866570","cluster_addr":"172.21.15.63:6827/2167866570","heartbeat_back_addr":"172.21.15.63:6831/2167866570","heartbeat_front_addr":"172.21.15.63:6829/2167866570","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:07.450323+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:31.363172+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:55.295021+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:19.794760+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:44.063492+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:08.195064+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:33.655328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:57.639414+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:22.402394+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:47.462050+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:12.226307+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:37.495625+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3821408150":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6801/1832422779":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/996598130":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/1062264551":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/317557779":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6801/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/3631746518":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6800/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/3116652786":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6800/1832422779":"2024-03-29T22:01:28.993434+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:07:52.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.494+0000 7f587828f640 1 -- 172.21.15.16:0/1138220929 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f585c0a1d80 msgr2=0x7f585c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:52.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.494+0000 7f587828f640 1 --2- 172.21.15.16:0/1138220929 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f585c0a1d80 0x7f585c0a4240 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f586c0095f0 tx=0x7f586c005f90 comp rx=0 tx=0).stop 2024-03-28T22:07:52.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.494+0000 7f587828f640 1 -- 172.21.15.16:0/1138220929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5870103390 msgr2=0x7f587019efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:52.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.494+0000 7f587828f640 1 --2- 172.21.15.16:0/1138220929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5870103390 0x7f587019efc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f586800eb10 tx=0x7f586800c2d0 comp rx=0 tx=0).stop 2024-03-28T22:07:52.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.494+0000 7f587828f640 1 -- 172.21.15.16:0/1138220929 shutdown_connections 2024-03-28T22:07:52.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.495+0000 7f587828f640 1 --2- 172.21.15.16:0/1138220929 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f585c0a1d80 0x7f585c0a4240 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:52.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.495+0000 7f587828f640 1 --2- 172.21.15.16:0/1138220929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5870103390 0x7f587019efc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:52.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.495+0000 7f587828f640 1 --2- 172.21.15.16:0/1138220929 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58701029d0 0x7f5870197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:52.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.495+0000 7f587828f640 1 --2- 172.21.15.16:0/1138220929 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58701017e0 0x7f5870197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:52.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.495+0000 7f587828f640 1 -- 172.21.15.16:0/1138220929 >> 172.21.15.16:0/1138220929 conn(0x7f58700fcd70 msgr2=0x7f5870105c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:07:52.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.495+0000 7f587828f640 1 -- 172.21.15.16:0/1138220929 shutdown_connections 2024-03-28T22:07:52.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.495+0000 7f587828f640 1 -- 172.21.15.16:0/1138220929 wait complete. 2024-03-28T22:07:52.508 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T22:03:59.327832+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T22:07:52.508 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T22:07:52.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.612+0000 7fb851d7a640 1 -- 172.21.15.16:0/3467239056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb84c101840 msgr2=0x7fb84c101cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.612+0000 7fb851d7a640 1 --2- 172.21.15.16:0/3467239056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb84c101840 0x7fb84c101cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb840009900 tx=0x7fb84002bc30 comp rx=0 tx=0).stop 2024-03-28T22:07:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.612+0000 7fb851d7a640 1 -- 172.21.15.16:0/3467239056 shutdown_connections 2024-03-28T22:07:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.612+0000 7fb851d7a640 1 --2- 172.21.15.16:0/3467239056 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb84c102200 0x7fb84c108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.612+0000 7fb851d7a640 1 --2- 172.21.15.16:0/3467239056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb84c101840 0x7fb84c101cc0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.612+0000 7fb851d7a640 1 --2- 172.21.15.16:0/3467239056 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb84c100650 0x7fb84c100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.612+0000 7fb851d7a640 1 -- 172.21.15.16:0/3467239056 >> 172.21.15.16:0/3467239056 conn(0x7fb84c0fbbc0 msgr2=0x7fb84c0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:07:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.612+0000 7fb851d7a640 1 -- 172.21.15.16:0/3467239056 shutdown_connections 2024-03-28T22:07:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.612+0000 7fb851d7a640 1 -- 172.21.15.16:0/3467239056 wait complete. 2024-03-28T22:07:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.612+0000 7fb851d7a640 1 Processor -- start 2024-03-28T22:07:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.613+0000 7fb851d7a640 1 -- start start 2024-03-28T22:07:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.613+0000 7fb851d7a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb84c100650 0x7fb84c197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.613+0000 7fb851d7a640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb84c101840 0x7fb84c198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.613+0000 7fb851d7a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb84c102200 0x7fb84c19f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.613+0000 7fb851d7a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb84c10c2e0 con 0x7fb84c100650 2024-03-28T22:07:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.613+0000 7fb851d7a640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb84c10c160 con 0x7fb84c101840 2024-03-28T22:07:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.613+0000 7fb851d7a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb84c10c460 con 0x7fb84c102200 2024-03-28T22:07:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.613+0000 7fb84b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb84c100650 0x7fb84c197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.613+0000 7fb84b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb84c100650 0x7fb84c197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38654/0 (socket says 172.21.15.16:38654) 2024-03-28T22:07:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.613+0000 7fb84b7fe640 1 -- 172.21.15.16:0/1974591125 learned_addr learned my addr 172.21.15.16:0/1974591125 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:07:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.613+0000 7fb84affd640 1 --2- 172.21.15.16:0/1974591125 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb84c101840 0x7fb84c198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.613+0000 7fb84bfff640 1 --2- 172.21.15.16:0/1974591125 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb84c102200 0x7fb84c19f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.613+0000 7fb84b7fe640 1 -- 172.21.15.16:0/1974591125 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb84c102200 msgr2=0x7fb84c19f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.613+0000 7fb84b7fe640 1 --2- 172.21.15.16:0/1974591125 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb84c102200 0x7fb84c19f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.613+0000 7fb84b7fe640 1 -- 172.21.15.16:0/1974591125 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb84c101840 msgr2=0x7fb84c198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.613+0000 7fb84b7fe640 1 --2- 172.21.15.16:0/1974591125 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb84c101840 0x7fb84c198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.613+0000 7fb84b7fe640 1 -- 172.21.15.16:0/1974591125 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb84c19f6b0 con 0x7fb84c100650 2024-03-28T22:07:52.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.613+0000 7fb84bfff640 1 --2- 172.21.15.16:0/1974591125 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb84c102200 0x7fb84c19f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:07:52.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.613+0000 7fb84b7fe640 1 --2- 172.21.15.16:0/1974591125 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb84c100650 0x7fb84c197ae0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb83c00d770 tx=0x7fb83c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:07:52.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.614+0000 7fb848ff9640 1 -- 172.21.15.16:0/1974591125 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb83c017070 con 0x7fb84c100650 2024-03-28T22:07:52.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.614+0000 7fb851d7a640 1 -- 172.21.15.16:0/1974591125 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb84c19f9a0 con 0x7fb84c100650 2024-03-28T22:07:52.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.614+0000 7fb848ff9640 1 -- 172.21.15.16:0/1974591125 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb83c009c30 con 0x7fb84c100650 2024-03-28T22:07:52.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.614+0000 7fb848ff9640 1 -- 172.21.15.16:0/1974591125 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb83c003dd0 con 0x7fb84c100650 2024-03-28T22:07:52.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.614+0000 7fb851d7a640 1 -- 172.21.15.16:0/1974591125 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb84c19fec0 con 0x7fb84c100650 2024-03-28T22:07:52.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.615+0000 7fb851d7a640 1 -- 172.21.15.16:0/1974591125 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb84c1a00f0 con 0x7fb84c100650 2024-03-28T22:07:52.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.616+0000 7fb848ff9640 1 -- 172.21.15.16:0/1974591125 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fb83c003f70 con 0x7fb84c100650 2024-03-28T22:07:52.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.616+0000 7fb848ff9640 1 --2- 172.21.15.16:0/1974591125 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb8340a1d80 0x7fb8340a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:52.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.617+0000 7fb84affd640 1 --2- 172.21.15.16:0/1974591125 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb8340a1d80 0x7fb8340a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:52.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.617+0000 7fb848ff9640 1 -- 172.21.15.16:0/1974591125 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fb83c0cd710 con 0x7fb84c100650 2024-03-28T22:07:52.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.617+0000 7fb84affd640 1 --2- 172.21.15.16:0/1974591125 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb8340a1d80 0x7fb8340a4240 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb84002c7b0 tx=0x7fb840005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:07:52.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.620+0000 7fb848ff9640 1 -- 172.21.15.16:0/1974591125 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb83c009de0 con 0x7fb84c100650 2024-03-28T22:07:52.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.801+0000 7fb851d7a640 1 -- 172.21.15.16:0/1974591125 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb84c1a02c0 con 0x7fb84c100650 2024-03-28T22:07:52.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.802+0000 7fb848ff9640 1 -- 172.21.15.16:0/1974591125 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7fb83c09af40 con 0x7fb84c100650 2024-03-28T22:07:52.803 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-28T22:07:52.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.804+0000 7fb851d7a640 1 -- 172.21.15.16:0/1974591125 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb8340a1d80 msgr2=0x7fb8340a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:52.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.804+0000 7fb851d7a640 1 --2- 172.21.15.16:0/1974591125 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb8340a1d80 0x7fb8340a4240 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb84002c7b0 tx=0x7fb840005fd0 comp rx=0 tx=0).stop 2024-03-28T22:07:52.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.804+0000 7fb851d7a640 1 -- 172.21.15.16:0/1974591125 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb84c100650 msgr2=0x7fb84c197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:52.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.804+0000 7fb851d7a640 1 --2- 172.21.15.16:0/1974591125 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb84c100650 0x7fb84c197ae0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb83c00d770 tx=0x7fb83c00dc30 comp rx=0 tx=0).stop 2024-03-28T22:07:52.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.804+0000 7fb851d7a640 1 -- 172.21.15.16:0/1974591125 shutdown_connections 2024-03-28T22:07:52.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.804+0000 7fb851d7a640 1 --2- 172.21.15.16:0/1974591125 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb8340a1d80 0x7fb8340a4240 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:52.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.804+0000 7fb851d7a640 1 --2- 172.21.15.16:0/1974591125 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb84c102200 0x7fb84c19f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:52.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.804+0000 7fb851d7a640 1 --2- 172.21.15.16:0/1974591125 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb84c101840 0x7fb84c198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:52.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.804+0000 7fb851d7a640 1 --2- 172.21.15.16:0/1974591125 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb84c100650 0x7fb84c197ae0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:52.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.804+0000 7fb851d7a640 1 -- 172.21.15.16:0/1974591125 >> 172.21.15.16:0/1974591125 conn(0x7fb84c0fbbc0 msgr2=0x7fb84c0fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:07:52.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.804+0000 7fb851d7a640 1 -- 172.21.15.16:0/1974591125 shutdown_connections 2024-03-28T22:07:52.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.804+0000 7fb851d7a640 1 -- 172.21.15.16:0/1974591125 wait complete. 2024-03-28T22:07:52.817 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:07:52.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.918+0000 7fa451ba1640 1 -- 172.21.15.16:0/3989959331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa44c103390 msgr2=0x7fa44c109c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:52.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.918+0000 7fa451ba1640 1 --2- 172.21.15.16:0/3989959331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa44c103390 0x7fa44c109c20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa444009ca0 tx=0x7fa44402c0f0 comp rx=0 tx=0).stop 2024-03-28T22:07:52.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.918+0000 7fa451ba1640 1 -- 172.21.15.16:0/3989959331 shutdown_connections 2024-03-28T22:07:52.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.918+0000 7fa451ba1640 1 --2- 172.21.15.16:0/3989959331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa44c103390 0x7fa44c109c20 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:52.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.918+0000 7fa451ba1640 1 --2- 172.21.15.16:0/3989959331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa44c1029d0 0x7fa44c102e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:52.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.918+0000 7fa451ba1640 1 --2- 172.21.15.16:0/3989959331 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa44c1017e0 0x7fa44c101c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:52.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.918+0000 7fa451ba1640 1 -- 172.21.15.16:0/3989959331 >> 172.21.15.16:0/3989959331 conn(0x7fa44c0fcd70 msgr2=0x7fa44c0ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:07:52.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.918+0000 7fa451ba1640 1 -- 172.21.15.16:0/3989959331 shutdown_connections 2024-03-28T22:07:52.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.918+0000 7fa451ba1640 1 -- 172.21.15.16:0/3989959331 wait complete. 2024-03-28T22:07:52.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.918+0000 7fa451ba1640 1 Processor -- start 2024-03-28T22:07:52.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.919+0000 7fa451ba1640 1 -- start start 2024-03-28T22:07:52.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.919+0000 7fa451ba1640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa44c1017e0 0x7fa44c197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:52.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.919+0000 7fa451ba1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa44c1029d0 0x7fa44c197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:52.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.919+0000 7fa451ba1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa44c103390 0x7fa44c19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:52.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.919+0000 7fa451ba1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa44c10d3d0 con 0x7fa44c1029d0 2024-03-28T22:07:52.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.919+0000 7fa451ba1640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fa44c10d250 con 0x7fa44c1017e0 2024-03-28T22:07:52.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.919+0000 7fa451ba1640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa44c10d550 con 0x7fa44c103390 2024-03-28T22:07:52.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.919+0000 7fa44affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa44c1029d0 0x7fa44c197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:52.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.919+0000 7fa44affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa44c1029d0 0x7fa44c197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38670/0 (socket says 172.21.15.16:38670) 2024-03-28T22:07:52.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.919+0000 7fa44affd640 1 -- 172.21.15.16:0/3341692810 learned_addr learned my addr 172.21.15.16:0/3341692810 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:07:52.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.919+0000 7fa44bfff640 1 --2- 172.21.15.16:0/3341692810 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa44c103390 0x7fa44c19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:52.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.919+0000 7fa44b7fe640 1 --2- 172.21.15.16:0/3341692810 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa44c1017e0 0x7fa44c197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:52.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.919+0000 7fa44affd640 1 -- 172.21.15.16:0/3341692810 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa44c103390 msgr2=0x7fa44c19efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:52.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.919+0000 7fa44affd640 1 --2- 172.21.15.16:0/3341692810 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa44c103390 0x7fa44c19efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:52.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.919+0000 7fa44affd640 1 -- 172.21.15.16:0/3341692810 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa44c1017e0 msgr2=0x7fa44c197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:52.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.919+0000 7fa44affd640 1 --2- 172.21.15.16:0/3341692810 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa44c1017e0 0x7fa44c197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:52.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.919+0000 7fa44affd640 1 -- 172.21.15.16:0/3341692810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa44c19f650 con 0x7fa44c1029d0 2024-03-28T22:07:52.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.919+0000 7fa44bfff640 1 --2- 172.21.15.16:0/3341692810 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa44c103390 0x7fa44c19efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:07:52.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.919+0000 7fa44b7fe640 1 --2- 172.21.15.16:0/3341692810 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa44c1017e0 0x7fa44c197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:07:52.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.919+0000 7fa44affd640 1 --2- 172.21.15.16:0/3341692810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa44c1029d0 0x7fa44c197f70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa44000d950 tx=0x7fa44000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:07:52.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.919+0000 7fa448ff9640 1 -- 172.21.15.16:0/3341692810 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa440017030 con 0x7fa44c1029d0 2024-03-28T22:07:52.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.919+0000 7fa448ff9640 1 -- 172.21.15.16:0/3341692810 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa440004020 con 0x7fa44c1029d0 2024-03-28T22:07:52.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.919+0000 7fa451ba1640 1 -- 172.21.15.16:0/3341692810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa44c19f940 con 0x7fa44c1029d0 2024-03-28T22:07:52.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.920+0000 7fa448ff9640 1 -- 172.21.15.16:0/3341692810 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4400047a0 con 0x7fa44c1029d0 2024-03-28T22:07:52.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.920+0000 7fa451ba1640 1 -- 172.21.15.16:0/3341692810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa44c19fee0 con 0x7fa44c1029d0 2024-03-28T22:07:52.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.920+0000 7fa451ba1640 1 -- 172.21.15.16:0/3341692810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa44c068a20 con 0x7fa44c1029d0 2024-03-28T22:07:52.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.921+0000 7fa448ff9640 1 -- 172.21.15.16:0/3341692810 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fa4400041d0 con 0x7fa44c1029d0 2024-03-28T22:07:52.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.922+0000 7fa448ff9640 1 --2- 172.21.15.16:0/3341692810 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fa4340a1e00 0x7fa4340a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:52.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.922+0000 7fa448ff9640 1 -- 172.21.15.16:0/3341692810 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fa4400cd2e0 con 0x7fa44c1029d0 2024-03-28T22:07:52.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.922+0000 7fa44b7fe640 1 --2- 172.21.15.16:0/3341692810 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fa4340a1e00 0x7fa4340a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:52.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.922+0000 7fa44b7fe640 1 --2- 172.21.15.16:0/3341692810 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fa4340a1e00 0x7fa4340a42c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa43c005d00 tx=0x7fa43c005ef0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:07:52.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:52.927+0000 7fa448ff9640 1 -- 172.21.15.16:0/3341692810 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa44009abb0 con 0x7fa44c1029d0 2024-03-28T22:07:53.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:53.148+0000 7fa451ba1640 1 -- 172.21.15.16:0/3341692810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa44c108aa0 con 0x7fa44c1029d0 2024-03-28T22:07:53.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:53.148+0000 7fa448ff9640 1 -- 172.21.15.16:0/3341692810 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7fa44009a4b0 con 0x7fa44c1029d0 2024-03-28T22:07:53.150 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:07:53.150 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-28T22:07:53.151 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 1 2024-03-28T22:07:53.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:53.150+0000 7fa451ba1640 1 -- 172.21.15.16:0/3341692810 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fa4340a1e00 msgr2=0x7fa4340a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:53.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:53.150+0000 7fa451ba1640 1 --2- 172.21.15.16:0/3341692810 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fa4340a1e00 0x7fa4340a42c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa43c005d00 tx=0x7fa43c005ef0 comp rx=0 tx=0).stop 2024-03-28T22:07:53.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:53.150+0000 7fa451ba1640 1 -- 172.21.15.16:0/3341692810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa44c1029d0 msgr2=0x7fa44c197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:53.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:53.150+0000 7fa451ba1640 1 --2- 172.21.15.16:0/3341692810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa44c1029d0 0x7fa44c197f70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa44000d950 tx=0x7fa44000de10 comp rx=0 tx=0).stop 2024-03-28T22:07:53.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:53.151+0000 7fa451ba1640 1 -- 172.21.15.16:0/3341692810 shutdown_connections 2024-03-28T22:07:53.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:53.151+0000 7fa451ba1640 1 --2- 172.21.15.16:0/3341692810 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fa4340a1e00 0x7fa4340a42c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:53.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:53.151+0000 7fa451ba1640 1 --2- 172.21.15.16:0/3341692810 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa44c103390 0x7fa44c19efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:53.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:53.151+0000 7fa451ba1640 1 --2- 172.21.15.16:0/3341692810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa44c1029d0 0x7fa44c197f70 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:53.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:53.151+0000 7fa451ba1640 1 --2- 172.21.15.16:0/3341692810 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa44c1017e0 0x7fa44c197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:53.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:53.151+0000 7fa451ba1640 1 -- 172.21.15.16:0/3341692810 >> 172.21.15.16:0/3341692810 conn(0x7fa44c0fcd70 msgr2=0x7fa44c105c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:07:53.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:53.151+0000 7fa451ba1640 1 -- 172.21.15.16:0/3341692810 shutdown_connections 2024-03-28T22:07:53.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:53.151+0000 7fa451ba1640 1 -- 172.21.15.16:0/3341692810 wait complete. 2024-03-28T22:07:53.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:53 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/1138220929' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:07:53.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:53 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/1974591125' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:07:53.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:53 smithi053 ceph-mon[35979]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:07:53.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:53 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3341692810' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:07:53.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:53 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/1138220929' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:07:53.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:53 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/1974591125' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:07:53.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:53 smithi016 ceph-mon[32374]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:07:53.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:53 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3341692810' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:07:53.525 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:53 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/1138220929' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:07:53.525 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:53 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/1974591125' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:07:53.526 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:53 smithi063 ceph-mon[36249]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:07:53.526 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:53 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/3341692810' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:07:54.182 INFO:journalctl@ceph.mds.l.smithi063.stdout:Mar 28 22:07:54 smithi063 systemd[1]: Starting Ceph mds.l for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:07:54.571 INFO:journalctl@ceph.mds.l.smithi063.stdout:Mar 28 22:07:54 smithi063 podman[66224]: 2024-03-28 22:07:54.145580179 +0000 UTC m=+0.017870002 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:07:54.572 INFO:journalctl@ceph.mds.l.smithi063.stdout:Mar 28 22:07:54 smithi063 podman[66224]: 2024-03-28 22:07:54.259674143 +0000 UTC m=+0.131963961 container create b1246fcb93bfc98de573ce20067cfc91f54cb5dea9a836bf6ac126d8d28a1159 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-323d23d) 2024-03-28T22:07:54.572 INFO:journalctl@ceph.mds.l.smithi063.stdout:Mar 28 22:07:54 smithi063 podman[66224]: 2024-03-28 22:07:54.453587732 +0000 UTC m=+0.325877558 container init b1246fcb93bfc98de573ce20067cfc91f54cb5dea9a836bf6ac126d8d28a1159 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-323d23d) 2024-03-28T22:07:54.572 INFO:journalctl@ceph.mds.l.smithi063.stdout:Mar 28 22:07:54 smithi063 podman[66224]: 2024-03-28 22:07:54.456293027 +0000 UTC m=+0.328582842 container start b1246fcb93bfc98de573ce20067cfc91f54cb5dea9a836bf6ac126d8d28a1159 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-l, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, RELEASE=squid-323d23d) 2024-03-28T22:07:54.572 INFO:journalctl@ceph.mds.l.smithi063.stdout:Mar 28 22:07:54 smithi063 ceph-mds[66243]: -- 172.21.15.63:0/4280689853 <== mon.1 v2:172.21.15.63:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b03b4b9a40 con 0x55b03b48cc00 2024-03-28T22:07:54.572 INFO:journalctl@ceph.mds.l.smithi063.stdout:Mar 28 22:07:54 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-l[66239]: starting mds.l at 2024-03-28T22:07:54.572 INFO:journalctl@ceph.mds.l.smithi063.stdout:Mar 28 22:07:54 smithi063 bash[66224]: b1246fcb93bfc98de573ce20067cfc91f54cb5dea9a836bf6ac126d8d28a1159 2024-03-28T22:07:54.932 INFO:journalctl@ceph.mds.l.smithi063.stdout:Mar 28 22:07:54 smithi063 systemd[1]: Started Ceph mds.l for 798d9884-ed4e-11ee-95d0-87774f69a715. 2024-03-28T22:07:55.164 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:07:55.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.278+0000 7faf93d25640 1 -- 172.21.15.16:0/469309578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf8c103390 msgr2=0x7faf8c109c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:55.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.278+0000 7faf93d25640 1 --2- 172.21.15.16:0/469309578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf8c103390 0x7faf8c109c20 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7faf84009ca0 tx=0x7faf8402c0f0 comp rx=0 tx=0).stop 2024-03-28T22:07:55.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.278+0000 7faf93d25640 1 -- 172.21.15.16:0/469309578 shutdown_connections 2024-03-28T22:07:55.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.278+0000 7faf93d25640 1 --2- 172.21.15.16:0/469309578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf8c103390 0x7faf8c109c20 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:55.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.278+0000 7faf93d25640 1 --2- 172.21.15.16:0/469309578 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faf8c1029d0 0x7faf8c102e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:55.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.278+0000 7faf93d25640 1 --2- 172.21.15.16:0/469309578 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faf8c1017e0 0x7faf8c101c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:55.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.278+0000 7faf93d25640 1 -- 172.21.15.16:0/469309578 >> 172.21.15.16:0/469309578 conn(0x7faf8c0fcd70 msgr2=0x7faf8c0ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:07:55.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.278+0000 7faf93d25640 1 -- 172.21.15.16:0/469309578 shutdown_connections 2024-03-28T22:07:55.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.278+0000 7faf93d25640 1 -- 172.21.15.16:0/469309578 wait complete. 2024-03-28T22:07:55.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.279+0000 7faf93d25640 1 Processor -- start 2024-03-28T22:07:55.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.279+0000 7faf93d25640 1 -- start start 2024-03-28T22:07:55.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.279+0000 7faf93d25640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf8c1017e0 0x7faf8c197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:55.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.279+0000 7faf93d25640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faf8c1029d0 0x7faf8c197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:55.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.279+0000 7faf93d25640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faf8c103390 0x7faf8c19efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:55.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.279+0000 7faf93d25640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faf8c10d3f0 con 0x7faf8c1017e0 2024-03-28T22:07:55.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.279+0000 7faf93d25640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7faf8c10d270 con 0x7faf8c1029d0 2024-03-28T22:07:55.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.279+0000 7faf93d25640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7faf8c10d570 con 0x7faf8c103390 2024-03-28T22:07:55.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.279+0000 7faf91ac1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf8c1017e0 0x7faf8c197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:55.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.279+0000 7faf91ac1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf8c1017e0 0x7faf8c197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38690/0 (socket says 172.21.15.16:38690) 2024-03-28T22:07:55.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.279+0000 7faf91ac1640 1 -- 172.21.15.16:0/821830040 learned_addr learned my addr 172.21.15.16:0/821830040 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:07:55.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.279+0000 7faf912c0640 1 --2- 172.21.15.16:0/821830040 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faf8c1029d0 0x7faf8c197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:55.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.279+0000 7faf922c2640 1 --2- 172.21.15.16:0/821830040 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faf8c103390 0x7faf8c19efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:55.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.280+0000 7faf91ac1640 1 -- 172.21.15.16:0/821830040 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faf8c103390 msgr2=0x7faf8c19efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:55.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.280+0000 7faf91ac1640 1 --2- 172.21.15.16:0/821830040 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faf8c103390 0x7faf8c19efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:55.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.280+0000 7faf91ac1640 1 -- 172.21.15.16:0/821830040 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faf8c1029d0 msgr2=0x7faf8c197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:55.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.280+0000 7faf91ac1640 1 --2- 172.21.15.16:0/821830040 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faf8c1029d0 0x7faf8c197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:55.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.280+0000 7faf91ac1640 1 -- 172.21.15.16:0/821830040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf8c19f620 con 0x7faf8c1017e0 2024-03-28T22:07:55.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.280+0000 7faf91ac1640 1 --2- 172.21.15.16:0/821830040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf8c1017e0 0x7faf8c197a00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7faf8800e9c0 tx=0x7faf8800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:07:55.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.280+0000 7faf82ffd640 1 -- 172.21.15.16:0/821830040 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8800cd80 con 0x7faf8c1017e0 2024-03-28T22:07:55.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.280+0000 7faf93d25640 1 -- 172.21.15.16:0/821830040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf8c19f910 con 0x7faf8c1017e0 2024-03-28T22:07:55.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.280+0000 7faf82ffd640 1 -- 172.21.15.16:0/821830040 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf880035f0 con 0x7faf8c1017e0 2024-03-28T22:07:55.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.280+0000 7faf82ffd640 1 -- 172.21.15.16:0/821830040 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf88010690 con 0x7faf8c1017e0 2024-03-28T22:07:55.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.280+0000 7faf93d25640 1 -- 172.21.15.16:0/821830040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf8c19fe60 con 0x7faf8c1017e0 2024-03-28T22:07:55.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.281+0000 7faf93d25640 1 -- 172.21.15.16:0/821830040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf8c108aa0 con 0x7faf8c1017e0 2024-03-28T22:07:55.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.282+0000 7faf82ffd640 1 -- 172.21.15.16:0/821830040 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7faf880037a0 con 0x7faf8c1017e0 2024-03-28T22:07:55.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.283+0000 7faf82ffd640 1 --2- 172.21.15.16:0/821830040 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7faf780a1d80 0x7faf780a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:55.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.283+0000 7faf912c0640 1 --2- 172.21.15.16:0/821830040 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7faf780a1d80 0x7faf780a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:55.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.283+0000 7faf82ffd640 1 -- 172.21.15.16:0/821830040 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7faf880cde00 con 0x7faf8c1017e0 2024-03-28T22:07:55.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.283+0000 7faf912c0640 1 --2- 172.21.15.16:0/821830040 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7faf780a1d80 0x7faf780a4240 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7faf8c198f10 tx=0x7faf7c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:07:55.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.287+0000 7faf82ffd640 1 -- 172.21.15.16:0/821830040 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7faf8809b6d0 con 0x7faf8c1017e0 2024-03-28T22:07:55.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.513+0000 7faf93d25640 1 -- 172.21.15.16:0/821830040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faf8c02d170 con 0x7faf8c1017e0 2024-03-28T22:07:55.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.513+0000 7faf82ffd640 1 -- 172.21.15.16:0/821830040 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1393 (secure 0 0 0) 0x7faf8809afd0 con 0x7faf8c1017e0 2024-03-28T22:07:55.515 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:07:55.515 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-28T22:07:55.515 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 2 2024-03-28T22:07:55.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.515+0000 7faf93d25640 1 -- 172.21.15.16:0/821830040 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7faf780a1d80 msgr2=0x7faf780a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:55.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.515+0000 7faf93d25640 1 --2- 172.21.15.16:0/821830040 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7faf780a1d80 0x7faf780a4240 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7faf8c198f10 tx=0x7faf7c005f90 comp rx=0 tx=0).stop 2024-03-28T22:07:55.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.515+0000 7faf93d25640 1 -- 172.21.15.16:0/821830040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf8c1017e0 msgr2=0x7faf8c197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:55.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.515+0000 7faf93d25640 1 --2- 172.21.15.16:0/821830040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf8c1017e0 0x7faf8c197a00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7faf8800e9c0 tx=0x7faf8800ee80 comp rx=0 tx=0).stop 2024-03-28T22:07:55.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.515+0000 7faf93d25640 1 -- 172.21.15.16:0/821830040 shutdown_connections 2024-03-28T22:07:55.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.515+0000 7faf93d25640 1 --2- 172.21.15.16:0/821830040 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7faf780a1d80 0x7faf780a4240 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:55.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.515+0000 7faf93d25640 1 --2- 172.21.15.16:0/821830040 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faf8c103390 0x7faf8c19efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:55.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.515+0000 7faf93d25640 1 --2- 172.21.15.16:0/821830040 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faf8c1029d0 0x7faf8c197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:55.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.515+0000 7faf93d25640 1 --2- 172.21.15.16:0/821830040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf8c1017e0 0x7faf8c197a00 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:55.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.515+0000 7faf93d25640 1 -- 172.21.15.16:0/821830040 >> 172.21.15.16:0/821830040 conn(0x7faf8c0fcd70 msgr2=0x7faf8c105c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:07:55.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.516+0000 7faf93d25640 1 -- 172.21.15.16:0/821830040 shutdown_connections 2024-03-28T22:07:55.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:55.516+0000 7faf93d25640 1 -- 172.21.15.16:0/821830040 wait complete. 2024-03-28T22:07:55.628 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:55 smithi053 ceph-mon[35979]: mds.? [v2:172.21.15.63:6832/3228503882,v1:172.21.15.63:6833/3228503882] up:boot 2024-03-28T22:07:55.629 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:55 smithi053 ceph-mon[35979]: fsmap 1 up:standby 2024-03-28T22:07:55.629 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:55 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T22:07:55.629 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:55 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:55.629 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:55 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:55.629 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:55 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:55.629 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:55 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:07:55.629 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:55 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:07:55.629 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:55 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:07:55.629 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:55 smithi053 ceph-mon[35979]: Deploying daemon mds.k on smithi053 2024-03-28T22:07:55.629 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:55 smithi053 ceph-mon[35979]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:07:55.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:55 smithi063 ceph-mon[36249]: mds.? [v2:172.21.15.63:6832/3228503882,v1:172.21.15.63:6833/3228503882] up:boot 2024-03-28T22:07:55.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:55 smithi063 ceph-mon[36249]: fsmap 1 up:standby 2024-03-28T22:07:55.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:55 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T22:07:55.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:55 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:55.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:55 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:55.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:55 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:55.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:55 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:07:55.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:55 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:07:55.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:55 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:07:55.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:55 smithi063 ceph-mon[36249]: Deploying daemon mds.k on smithi053 2024-03-28T22:07:55.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:55 smithi063 ceph-mon[36249]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:07:55.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:55 smithi016 ceph-mon[32374]: mds.? [v2:172.21.15.63:6832/3228503882,v1:172.21.15.63:6833/3228503882] up:boot 2024-03-28T22:07:55.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:55 smithi016 ceph-mon[32374]: fsmap 1 up:standby 2024-03-28T22:07:55.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:55 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T22:07:55.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:55 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:55.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:55 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:55.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:55 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:55.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:55 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:07:55.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:55 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:07:55.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:55 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:07:55.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:55 smithi016 ceph-mon[32374]: Deploying daemon mds.k on smithi053 2024-03-28T22:07:55.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:55 smithi016 ceph-mon[32374]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:07:56.746 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:56 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/821830040' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:07:56.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:56 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/821830040' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:07:56.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:56 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/821830040' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:07:57.047 INFO:journalctl@ceph.mds.k.smithi053.stdout:Mar 28 22:07:56 smithi053 systemd[1]: Starting Ceph mds.k for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:07:57.446 INFO:journalctl@ceph.mds.k.smithi053.stdout:Mar 28 22:07:57 smithi053 podman[65489]: 2024-03-28 22:07:57.050911852 +0000 UTC m=+0.018168025 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:07:57.446 INFO:journalctl@ceph.mds.k.smithi053.stdout:Mar 28 22:07:57 smithi053 podman[65489]: 2024-03-28 22:07:57.200308916 +0000 UTC m=+0.167565126 container create 321408088b92313a15bc2b8aaf5ff1250c5defaeab95ecc4ab0a8105d08f4373 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-k, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-323d23d, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-28T22:07:57.446 INFO:journalctl@ceph.mds.k.smithi053.stdout:Mar 28 22:07:57 smithi053 podman[65489]: 2024-03-28 22:07:57.444235781 +0000 UTC m=+0.411491952 container init 321408088b92313a15bc2b8aaf5ff1250c5defaeab95ecc4ab0a8105d08f4373 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-k, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=squid-323d23d, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True) 2024-03-28T22:07:57.529 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:07:57.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.646+0000 7f5057078640 1 -- 172.21.15.16:0/2598839944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5050103350 msgr2=0x7f5050109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:57.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.646+0000 7f5057078640 1 --2- 172.21.15.16:0/2598839944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5050103350 0x7f5050109be0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5048009ca0 tx=0x7f504802c0f0 comp rx=0 tx=0).stop 2024-03-28T22:07:57.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.646+0000 7f5057078640 1 -- 172.21.15.16:0/2598839944 shutdown_connections 2024-03-28T22:07:57.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.646+0000 7f5057078640 1 --2- 172.21.15.16:0/2598839944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5050103350 0x7f5050109be0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:57.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.646+0000 7f5057078640 1 --2- 172.21.15.16:0/2598839944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5050102990 0x7f5050102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:57.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.646+0000 7f5057078640 1 --2- 172.21.15.16:0/2598839944 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50501017a0 0x7f5050101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:57.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.646+0000 7f5057078640 1 -- 172.21.15.16:0/2598839944 >> 172.21.15.16:0/2598839944 conn(0x7f50500fcd10 msgr2=0x7f50500ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:07:57.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.646+0000 7f5057078640 1 -- 172.21.15.16:0/2598839944 shutdown_connections 2024-03-28T22:07:57.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.646+0000 7f5057078640 1 -- 172.21.15.16:0/2598839944 wait complete. 2024-03-28T22:07:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.646+0000 7f5057078640 1 Processor -- start 2024-03-28T22:07:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.646+0000 7f5057078640 1 -- start start 2024-03-28T22:07:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.646+0000 7f5057078640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50501017a0 0x7f5050199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.647+0000 7f5057078640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5050102990 0x7f505019a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.647+0000 7f5057078640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5050103350 0x7f50501a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.647+0000 7f5057078640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f505010d400 con 0x7f5050103350 2024-03-28T22:07:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.647+0000 7f5057078640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f505010d280 con 0x7f5050102990 2024-03-28T22:07:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.647+0000 7f5057078640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f505010d580 con 0x7f50501017a0 2024-03-28T22:07:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.647+0000 7f5055615640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5050103350 0x7f50501a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.647+0000 7f5055615640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5050103350 0x7f50501a1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38706/0 (socket says 172.21.15.16:38706) 2024-03-28T22:07:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.647+0000 7f5055615640 1 -- 172.21.15.16:0/4193232714 learned_addr learned my addr 172.21.15.16:0/4193232714 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:07:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.647+0000 7f5047fff640 1 --2- 172.21.15.16:0/4193232714 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5050102990 0x7f505019a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.647+0000 7f5054e14640 1 --2- 172.21.15.16:0/4193232714 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50501017a0 0x7f5050199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.647+0000 7f5055615640 1 -- 172.21.15.16:0/4193232714 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50501017a0 msgr2=0x7f5050199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.647+0000 7f5055615640 1 --2- 172.21.15.16:0/4193232714 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50501017a0 0x7f5050199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.647+0000 7f5055615640 1 -- 172.21.15.16:0/4193232714 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5050102990 msgr2=0x7f505019a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.647+0000 7f5055615640 1 --2- 172.21.15.16:0/4193232714 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5050102990 0x7f505019a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.647+0000 7f5055615640 1 -- 172.21.15.16:0/4193232714 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50501a1830 con 0x7f5050103350 2024-03-28T22:07:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.647+0000 7f5055615640 1 --2- 172.21.15.16:0/4193232714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5050103350 0x7f50501a1190 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5048037040 tx=0x7f5048032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:07:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.647+0000 7f5045ffb640 1 -- 172.21.15.16:0/4193232714 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5048041020 con 0x7f5050103350 2024-03-28T22:07:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.647+0000 7f5045ffb640 1 -- 172.21.15.16:0/4193232714 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5048034d90 con 0x7f5050103350 2024-03-28T22:07:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.647+0000 7f5057078640 1 -- 172.21.15.16:0/4193232714 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50501a1ac0 con 0x7f5050103350 2024-03-28T22:07:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.647+0000 7f5045ffb640 1 -- 172.21.15.16:0/4193232714 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5048039040 con 0x7f5050103350 2024-03-28T22:07:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.648+0000 7f5057078640 1 -- 172.21.15.16:0/4193232714 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50501a1fb0 con 0x7f5050103350 2024-03-28T22:07:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.649+0000 7f5057078640 1 -- 172.21.15.16:0/4193232714 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50501a2180 con 0x7f5050103350 2024-03-28T22:07:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.650+0000 7f5045ffb640 1 -- 172.21.15.16:0/4193232714 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f5048044020 con 0x7f5050103350 2024-03-28T22:07:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.650+0000 7f5045ffb640 1 --2- 172.21.15.16:0/4193232714 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f50380a1d80 0x7f50380a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:07:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.651+0000 7f5054e14640 1 --2- 172.21.15.16:0/4193232714 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f50380a1d80 0x7f50380a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:07:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.651+0000 7f5045ffb640 1 -- 172.21.15.16:0/4193232714 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f50480ef9d0 con 0x7f5050103350 2024-03-28T22:07:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.651+0000 7f5054e14640 1 --2- 172.21.15.16:0/4193232714 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f50380a1d80 0x7f50380a4240 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f504c005d00 tx=0x7f504c005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:07:57.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.655+0000 7f5045ffb640 1 -- 172.21.15.16:0/4193232714 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f504804b020 con 0x7f5050103350 2024-03-28T22:07:57.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:57 smithi053 ceph-mon[35979]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:07:57.798 INFO:journalctl@ceph.mds.k.smithi053.stdout:Mar 28 22:07:57 smithi053 podman[65489]: 2024-03-28 22:07:57.447649299 +0000 UTC m=+0.414905467 container start 321408088b92313a15bc2b8aaf5ff1250c5defaeab95ecc4ab0a8105d08f4373 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-323d23d, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326) 2024-03-28T22:07:57.798 INFO:journalctl@ceph.mds.k.smithi053.stdout:Mar 28 22:07:57 smithi053 ceph-mds[65508]: -- 172.21.15.53:0/3680140240 <== mon.2 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56224b18fa40 con 0x56224b162800 2024-03-28T22:07:57.798 INFO:journalctl@ceph.mds.k.smithi053.stdout:Mar 28 22:07:57 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-k[65504]: starting mds.k at 2024-03-28T22:07:57.798 INFO:journalctl@ceph.mds.k.smithi053.stdout:Mar 28 22:07:57 smithi053 bash[65489]: 321408088b92313a15bc2b8aaf5ff1250c5defaeab95ecc4ab0a8105d08f4373 2024-03-28T22:07:57.798 INFO:journalctl@ceph.mds.k.smithi053.stdout:Mar 28 22:07:57 smithi053 systemd[1]: Started Ceph mds.k for 798d9884-ed4e-11ee-95d0-87774f69a715. 2024-03-28T22:07:57.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.879+0000 7f5057078640 1 -- 172.21.15.16:0/4193232714 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f50501a2350 con 0x7f5050103350 2024-03-28T22:07:57.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.879+0000 7f5045ffb640 1 -- 172.21.15.16:0/4193232714 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2218 (secure 0 0 0) 0x7f50480bd200 con 0x7f5050103350 2024-03-28T22:07:57.881 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:07:57.881 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-28T22:07:57.882 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 3 2024-03-28T22:07:57.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.882+0000 7f5057078640 1 -- 172.21.15.16:0/4193232714 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f50380a1d80 msgr2=0x7f50380a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:57.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.882+0000 7f5057078640 1 --2- 172.21.15.16:0/4193232714 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f50380a1d80 0x7f50380a4240 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f504c005d00 tx=0x7f504c005c70 comp rx=0 tx=0).stop 2024-03-28T22:07:57.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.882+0000 7f5057078640 1 -- 172.21.15.16:0/4193232714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5050103350 msgr2=0x7f50501a1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:07:57.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.882+0000 7f5057078640 1 --2- 172.21.15.16:0/4193232714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5050103350 0x7f50501a1190 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5048037040 tx=0x7f5048032f20 comp rx=0 tx=0).stop 2024-03-28T22:07:57.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.882+0000 7f5057078640 1 -- 172.21.15.16:0/4193232714 shutdown_connections 2024-03-28T22:07:57.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.882+0000 7f5057078640 1 --2- 172.21.15.16:0/4193232714 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f50380a1d80 0x7f50380a4240 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:57.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.882+0000 7f5057078640 1 --2- 172.21.15.16:0/4193232714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5050103350 0x7f50501a1190 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:57.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.882+0000 7f5057078640 1 --2- 172.21.15.16:0/4193232714 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5050102990 0x7f505019a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:57.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.882+0000 7f5057078640 1 --2- 172.21.15.16:0/4193232714 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50501017a0 0x7f5050199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:07:57.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.882+0000 7f5057078640 1 -- 172.21.15.16:0/4193232714 >> 172.21.15.16:0/4193232714 conn(0x7f50500fcd10 msgr2=0x7f5050073980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:07:57.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.882+0000 7f5057078640 1 -- 172.21.15.16:0/4193232714 shutdown_connections 2024-03-28T22:07:57.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:07:57.882+0000 7f5057078640 1 -- 172.21.15.16:0/4193232714 wait complete. 2024-03-28T22:07:57.893 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:57 smithi016 ceph-mon[32374]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:07:57.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:57 smithi063 ceph-mon[36249]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:07:58.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:58 smithi053 ceph-mon[35979]: mds.? [v2:172.21.15.53:6832/3748831330,v1:172.21.15.53:6833/3748831330] up:boot 2024-03-28T22:07:58.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:58 smithi053 ceph-mon[35979]: fsmap 2 up:standby 2024-03-28T22:07:58.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:58 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T22:07:58.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:58 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:58.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:58 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:58.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:58 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:58.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:58 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:07:58.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:58 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:07:58.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:58 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:07:58.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:58 smithi053 ceph-mon[35979]: Deploying daemon mds.j on smithi016 2024-03-28T22:07:58.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:58 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/4193232714' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:07:58.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:58 smithi063 ceph-mon[36249]: mds.? [v2:172.21.15.53:6832/3748831330,v1:172.21.15.53:6833/3748831330] up:boot 2024-03-28T22:07:58.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:58 smithi063 ceph-mon[36249]: fsmap 2 up:standby 2024-03-28T22:07:58.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:58 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T22:07:58.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:58 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:58.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:58 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:58.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:58 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:58.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:58 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:07:58.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:58 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:07:58.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:58 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:07:58.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:58 smithi063 ceph-mon[36249]: Deploying daemon mds.j on smithi016 2024-03-28T22:07:58.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:58 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/4193232714' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:07:58.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:58 smithi016 ceph-mon[32374]: mds.? [v2:172.21.15.53:6832/3748831330,v1:172.21.15.53:6833/3748831330] up:boot 2024-03-28T22:07:58.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:58 smithi016 ceph-mon[32374]: fsmap 2 up:standby 2024-03-28T22:07:58.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:58 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T22:07:58.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:58 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:58.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:58 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:58.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:58 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:07:58.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:58 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:07:58.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:58 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:07:58.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:58 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:07:58.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:58 smithi016 ceph-mon[32374]: Deploying daemon mds.j on smithi016 2024-03-28T22:07:58.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:58 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/4193232714' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:07:59.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:07:59 smithi053 ceph-mon[35979]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:07:59.895 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:07:59.921 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:07:59 smithi016 ceph-mon[32374]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:07:59.921 INFO:journalctl@ceph.mds.j.smithi016.stdout:Mar 28 22:07:59 smithi016 systemd[1]: Starting Ceph mds.j for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:07:59.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:07:59 smithi063 ceph-mon[36249]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:00.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.001+0000 7f3be68eb640 1 -- 172.21.15.16:0/3371626270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3be0101870 msgr2=0x7f3be0101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:00.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.001+0000 7f3be68eb640 1 --2- 172.21.15.16:0/3371626270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3be0101870 0x7f3be0101cf0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3bc8009900 tx=0x7f3bc802bc30 comp rx=0 tx=0).stop 2024-03-28T22:08:00.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.001+0000 7f3be68eb640 1 -- 172.21.15.16:0/3371626270 shutdown_connections 2024-03-28T22:08:00.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.001+0000 7f3be68eb640 1 --2- 172.21.15.16:0/3371626270 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3be0102230 0x7f3be0108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:00.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.001+0000 7f3be68eb640 1 --2- 172.21.15.16:0/3371626270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3be0101870 0x7f3be0101cf0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:00.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.001+0000 7f3be68eb640 1 --2- 172.21.15.16:0/3371626270 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3be0100680 0x7f3be0100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:00.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.001+0000 7f3be68eb640 1 -- 172.21.15.16:0/3371626270 >> 172.21.15.16:0/3371626270 conn(0x7f3be00fbbf0 msgr2=0x7f3be00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:00.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.001+0000 7f3be68eb640 1 -- 172.21.15.16:0/3371626270 shutdown_connections 2024-03-28T22:08:00.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.001+0000 7f3be68eb640 1 -- 172.21.15.16:0/3371626270 wait complete. 2024-03-28T22:08:00.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.002+0000 7f3be68eb640 1 Processor -- start 2024-03-28T22:08:00.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.002+0000 7f3be68eb640 1 -- start start 2024-03-28T22:08:00.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.002+0000 7f3be68eb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3be0100680 0x7f3be0196940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:00.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.002+0000 7f3be68eb640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3be0101870 0x7f3be0196e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:00.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.002+0000 7f3be68eb640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3be0102230 0x7f3be019def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:00.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.002+0000 7f3be68eb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3be010c300 con 0x7f3be0100680 2024-03-28T22:08:00.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.002+0000 7f3be68eb640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3be010c180 con 0x7f3be0101870 2024-03-28T22:08:00.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.002+0000 7f3be68eb640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3be010c480 con 0x7f3be0102230 2024-03-28T22:08:00.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.002+0000 7f3bdffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3be0100680 0x7f3be0196940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:00.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.002+0000 7f3bdffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3be0100680 0x7f3be0196940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38724/0 (socket says 172.21.15.16:38724) 2024-03-28T22:08:00.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.002+0000 7f3bdf7fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3be0101870 0x7f3be0196e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:00.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.002+0000 7f3bdffff640 1 -- 172.21.15.16:0/1713202609 learned_addr learned my addr 172.21.15.16:0/1713202609 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:00.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.002+0000 7f3be4e88640 1 --2- 172.21.15.16:0/1713202609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3be0102230 0x7f3be019def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:00.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.002+0000 7f3bdffff640 1 -- 172.21.15.16:0/1713202609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3be0102230 msgr2=0x7f3be019def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:00.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.002+0000 7f3bdffff640 1 --2- 172.21.15.16:0/1713202609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3be0102230 0x7f3be019def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:00.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.002+0000 7f3bdffff640 1 -- 172.21.15.16:0/1713202609 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3be0101870 msgr2=0x7f3be0196e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:00.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.002+0000 7f3bdffff640 1 --2- 172.21.15.16:0/1713202609 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3be0101870 0x7f3be0196e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:00.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.002+0000 7f3bdffff640 1 -- 172.21.15.16:0/1713202609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3be019e560 con 0x7f3be0100680 2024-03-28T22:08:00.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.002+0000 7f3bdf7fe640 1 --2- 172.21.15.16:0/1713202609 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3be0101870 0x7f3be0196e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:08:00.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.003+0000 7f3bdffff640 1 --2- 172.21.15.16:0/1713202609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3be0100680 0x7f3be0196940 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3bd000e9c0 tx=0x7f3bd000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:00.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.003+0000 7f3bdd7fa640 1 -- 172.21.15.16:0/1713202609 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd000cd80 con 0x7f3be0100680 2024-03-28T22:08:00.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.003+0000 7f3bdd7fa640 1 -- 172.21.15.16:0/1713202609 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bd00035f0 con 0x7f3be0100680 2024-03-28T22:08:00.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.003+0000 7f3be68eb640 1 -- 172.21.15.16:0/1713202609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3be019e850 con 0x7f3be0100680 2024-03-28T22:08:00.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.003+0000 7f3bdd7fa640 1 -- 172.21.15.16:0/1713202609 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd0010690 con 0x7f3be0100680 2024-03-28T22:08:00.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.003+0000 7f3be68eb640 1 -- 172.21.15.16:0/1713202609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3be019ed70 con 0x7f3be0100680 2024-03-28T22:08:00.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.004+0000 7f3be68eb640 1 -- 172.21.15.16:0/1713202609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3be0107940 con 0x7f3be0100680 2024-03-28T22:08:00.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.005+0000 7f3bdd7fa640 1 -- 172.21.15.16:0/1713202609 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f3bd00037a0 con 0x7f3be0100680 2024-03-28T22:08:00.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.005+0000 7f3bdd7fa640 1 --2- 172.21.15.16:0/1713202609 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f3bcc0a1e00 0x7f3bcc0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:00.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.005+0000 7f3bdf7fe640 1 --2- 172.21.15.16:0/1713202609 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f3bcc0a1e00 0x7f3bcc0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:00.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.005+0000 7f3bdd7fa640 1 -- 172.21.15.16:0/1713202609 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f3bd00cdd30 con 0x7f3be0100680 2024-03-28T22:08:00.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.006+0000 7f3bdf7fe640 1 --2- 172.21.15.16:0/1713202609 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f3bcc0a1e00 0x7f3bcc0a42c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3bc802c7b0 tx=0x7f3bc8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:00.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.010+0000 7f3bdd7fa640 1 -- 172.21.15.16:0/1713202609 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3bd00d3020 con 0x7f3be0100680 2024-03-28T22:08:00.186 INFO:journalctl@ceph.mds.j.smithi016.stdout:Mar 28 22:07:59 smithi016 podman[70716]: 2024-03-28 22:07:59.823030475 +0000 UTC m=+0.017337548 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:08:00.187 INFO:journalctl@ceph.mds.j.smithi016.stdout:Mar 28 22:07:59 smithi016 podman[70716]: 2024-03-28 22:07:59.954015182 +0000 UTC m=+0.148322238 container create 83061c52f5f61b02eb1c9a788df9845a9e56991953c010642c83c67fab95e06e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-j, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_CLEAN=True, RELEASE=squid-323d23d) 2024-03-28T22:08:00.187 INFO:journalctl@ceph.mds.j.smithi016.stdout:Mar 28 22:08:00 smithi016 podman[70716]: 2024-03-28 22:08:00.131373322 +0000 UTC m=+0.325680393 container init 83061c52f5f61b02eb1c9a788df9845a9e56991953c010642c83c67fab95e06e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-j, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-323d23d) 2024-03-28T22:08:00.187 INFO:journalctl@ceph.mds.j.smithi016.stdout:Mar 28 22:08:00 smithi016 podman[70716]: 2024-03-28 22:08:00.134090061 +0000 UTC m=+0.328397119 container start 83061c52f5f61b02eb1c9a788df9845a9e56991953c010642c83c67fab95e06e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-j, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, GIT_CLEAN=True, RELEASE=squid-323d23d, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-28T22:08:00.187 INFO:journalctl@ceph.mds.j.smithi016.stdout:Mar 28 22:08:00 smithi016 ceph-mds[70774]: -- 172.21.15.16:0/792260735 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555c12879a40 con 0x555c1284d000 2024-03-28T22:08:00.187 INFO:journalctl@ceph.mds.j.smithi016.stdout:Mar 28 22:08:00 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-j[70770]: starting mds.j at 2024-03-28T22:08:00.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.262+0000 7f3be68eb640 1 -- 172.21.15.16:0/1713202609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3be0062350 con 0x7f3be0100680 2024-03-28T22:08:00.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.262+0000 7f3bdd7fa640 1 -- 172.21.15.16:0/1713202609 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2218 (secure 0 0 0) 0x7f3bd009b560 con 0x7f3be0100680 2024-03-28T22:08:00.264 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:08:00.265 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-28T22:08:00.265 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 3 2024-03-28T22:08:00.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.265+0000 7f3be68eb640 1 -- 172.21.15.16:0/1713202609 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f3bcc0a1e00 msgr2=0x7f3bcc0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:00.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.265+0000 7f3be68eb640 1 --2- 172.21.15.16:0/1713202609 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f3bcc0a1e00 0x7f3bcc0a42c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3bc802c7b0 tx=0x7f3bc8005fd0 comp rx=0 tx=0).stop 2024-03-28T22:08:00.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.265+0000 7f3be68eb640 1 -- 172.21.15.16:0/1713202609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3be0100680 msgr2=0x7f3be0196940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:00.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.265+0000 7f3be68eb640 1 --2- 172.21.15.16:0/1713202609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3be0100680 0x7f3be0196940 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3bd000e9c0 tx=0x7f3bd000ee80 comp rx=0 tx=0).stop 2024-03-28T22:08:00.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.265+0000 7f3be68eb640 1 -- 172.21.15.16:0/1713202609 shutdown_connections 2024-03-28T22:08:00.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.265+0000 7f3be68eb640 1 --2- 172.21.15.16:0/1713202609 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f3bcc0a1e00 0x7f3bcc0a42c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:00.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.265+0000 7f3be68eb640 1 --2- 172.21.15.16:0/1713202609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3be0102230 0x7f3be019def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:00.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.265+0000 7f3be68eb640 1 --2- 172.21.15.16:0/1713202609 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3be0101870 0x7f3be0196e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:00.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.265+0000 7f3be68eb640 1 --2- 172.21.15.16:0/1713202609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3be0100680 0x7f3be0196940 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:00.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.265+0000 7f3be68eb640 1 -- 172.21.15.16:0/1713202609 >> 172.21.15.16:0/1713202609 conn(0x7f3be00fbbf0 msgr2=0x7f3be0104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:00.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.265+0000 7f3be68eb640 1 -- 172.21.15.16:0/1713202609 shutdown_connections 2024-03-28T22:08:00.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:00.265+0000 7f3be68eb640 1 -- 172.21.15.16:0/1713202609 wait complete. 2024-03-28T22:08:00.686 INFO:journalctl@ceph.mds.j.smithi016.stdout:Mar 28 22:08:00 smithi016 bash[70716]: 83061c52f5f61b02eb1c9a788df9845a9e56991953c010642c83c67fab95e06e 2024-03-28T22:08:00.686 INFO:journalctl@ceph.mds.j.smithi016.stdout:Mar 28 22:08:00 smithi016 systemd[1]: Started Ceph mds.j for 798d9884-ed4e-11ee-95d0-87774f69a715. 2024-03-28T22:08:01.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:01 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:01.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:01 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:01.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:01 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/1713202609' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:01.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:01 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:01.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:01 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:08:01.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:01 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:08:01.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:01 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:08:01.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:01 smithi016 ceph-mon[32374]: Deploying daemon mds.d on smithi016 2024-03-28T22:08:01.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:01 smithi016 ceph-mon[32374]: mds.? [v2:172.21.15.16:6834/2478458611,v1:172.21.15.16:6835/2478458611] up:boot 2024-03-28T22:08:01.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:01 smithi016 ceph-mon[32374]: fsmap 3 up:standby 2024-03-28T22:08:01.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:01 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T22:08:01.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:01 smithi016 ceph-mon[32374]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:01.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:01 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:01.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:01 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:01.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:01 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/1713202609' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:01.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:01 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:01.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:01 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:08:01.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:01 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:08:01.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:01 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:08:01.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:01 smithi053 ceph-mon[35979]: Deploying daemon mds.d on smithi016 2024-03-28T22:08:01.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:01 smithi053 ceph-mon[35979]: mds.? [v2:172.21.15.16:6834/2478458611,v1:172.21.15.16:6835/2478458611] up:boot 2024-03-28T22:08:01.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:01 smithi053 ceph-mon[35979]: fsmap 3 up:standby 2024-03-28T22:08:01.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:01 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T22:08:01.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:01 smithi053 ceph-mon[35979]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:01.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:01 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:01.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:01 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:01.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:01 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/1713202609' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:01.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:01 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:01.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:01 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:08:01.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:01 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:08:01.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:01 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:08:01.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:01 smithi063 ceph-mon[36249]: Deploying daemon mds.d on smithi016 2024-03-28T22:08:01.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:01 smithi063 ceph-mon[36249]: mds.? [v2:172.21.15.16:6834/2478458611,v1:172.21.15.16:6835/2478458611] up:boot 2024-03-28T22:08:01.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:01 smithi063 ceph-mon[36249]: fsmap 3 up:standby 2024-03-28T22:08:01.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:01 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T22:08:01.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:01 smithi063 ceph-mon[36249]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:02.278 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:08:02.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.497+0000 7f48c0d0d640 1 -- 172.21.15.16:0/3858772987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48bc1017a0 msgr2=0x7f48bc101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:02.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.497+0000 7f48c0d0d640 1 --2- 172.21.15.16:0/3858772987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48bc1017a0 0x7f48bc101bc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f48ac009900 tx=0x7f48ac02bc30 comp rx=0 tx=0).stop 2024-03-28T22:08:02.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.497+0000 7f48c0d0d640 1 -- 172.21.15.16:0/3858772987 shutdown_connections 2024-03-28T22:08:02.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.497+0000 7f48c0d0d640 1 --2- 172.21.15.16:0/3858772987 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48bc103350 0x7f48bc109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:02.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.497+0000 7f48c0d0d640 1 --2- 172.21.15.16:0/3858772987 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f48bc102990 0x7f48bc102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:02.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.497+0000 7f48c0d0d640 1 --2- 172.21.15.16:0/3858772987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48bc1017a0 0x7f48bc101bc0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:02.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.497+0000 7f48c0d0d640 1 -- 172.21.15.16:0/3858772987 >> 172.21.15.16:0/3858772987 conn(0x7f48bc0fcd10 msgr2=0x7f48bc0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:02.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.497+0000 7f48c0d0d640 1 -- 172.21.15.16:0/3858772987 shutdown_connections 2024-03-28T22:08:02.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.497+0000 7f48c0d0d640 1 -- 172.21.15.16:0/3858772987 wait complete. 2024-03-28T22:08:02.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.498+0000 7f48c0d0d640 1 Processor -- start 2024-03-28T22:08:02.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.498+0000 7f48c0d0d640 1 -- start start 2024-03-28T22:08:02.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.498+0000 7f48c0d0d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48bc1017a0 0x7f48bc197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:02.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.498+0000 7f48c0d0d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f48bc102990 0x7f48bc198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:02.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.498+0000 7f48c0d0d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48bc103350 0x7f48bc19f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:02.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.498+0000 7f48c0d0d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f48bc10d410 con 0x7f48bc1017a0 2024-03-28T22:08:02.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.498+0000 7f48c0d0d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f48bc10d290 con 0x7f48bc103350 2024-03-28T22:08:02.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.498+0000 7f48c0d0d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f48bc10d590 con 0x7f48bc102990 2024-03-28T22:08:02.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.498+0000 7f48ba59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48bc1017a0 0x7f48bc197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:02.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.498+0000 7f48ba59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48bc1017a0 0x7f48bc197af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47936/0 (socket says 172.21.15.16:47936) 2024-03-28T22:08:02.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.498+0000 7f48ba59c640 1 -- 172.21.15.16:0/1766426579 learned_addr learned my addr 172.21.15.16:0/1766426579 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:02.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.499+0000 7f48b9d9b640 1 --2- 172.21.15.16:0/1766426579 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f48bc102990 0x7f48bc198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:02.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.499+0000 7f48bad9d640 1 --2- 172.21.15.16:0/1766426579 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48bc103350 0x7f48bc19f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:02.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.499+0000 7f48ba59c640 1 -- 172.21.15.16:0/1766426579 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f48bc102990 msgr2=0x7f48bc198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:02.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.499+0000 7f48ba59c640 1 --2- 172.21.15.16:0/1766426579 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f48bc102990 0x7f48bc198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:02.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.499+0000 7f48ba59c640 1 -- 172.21.15.16:0/1766426579 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48bc103350 msgr2=0x7f48bc19f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:02.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.499+0000 7f48ba59c640 1 --2- 172.21.15.16:0/1766426579 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48bc103350 0x7f48bc19f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:02.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.499+0000 7f48ba59c640 1 -- 172.21.15.16:0/1766426579 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48bc19f710 con 0x7f48bc1017a0 2024-03-28T22:08:02.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.499+0000 7f48ba59c640 1 --2- 172.21.15.16:0/1766426579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48bc1017a0 0x7f48bc197af0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f48ac000c00 tx=0x7f48ac005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:02.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.499+0000 7f48ab7fe640 1 -- 172.21.15.16:0/1766426579 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48ac041070 con 0x7f48bc1017a0 2024-03-28T22:08:02.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.499+0000 7f48c0d0d640 1 -- 172.21.15.16:0/1766426579 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48bc19f9a0 con 0x7f48bc1017a0 2024-03-28T22:08:02.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.499+0000 7f48ab7fe640 1 -- 172.21.15.16:0/1766426579 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48ac0044d0 con 0x7f48bc1017a0 2024-03-28T22:08:02.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.499+0000 7f48ab7fe640 1 -- 172.21.15.16:0/1766426579 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48ac034440 con 0x7f48bc1017a0 2024-03-28T22:08:02.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.499+0000 7f48c0d0d640 1 -- 172.21.15.16:0/1766426579 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48bc19fee0 con 0x7f48bc1017a0 2024-03-28T22:08:02.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.500+0000 7f48c0d0d640 1 -- 172.21.15.16:0/1766426579 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48bc108ae0 con 0x7f48bc1017a0 2024-03-28T22:08:02.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.502+0000 7f48ab7fe640 1 -- 172.21.15.16:0/1766426579 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f48ac044070 con 0x7f48bc1017a0 2024-03-28T22:08:02.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.503+0000 7f48ab7fe640 1 --2- 172.21.15.16:0/1766426579 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f48a40a1e00 0x7f48a40a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:02.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.503+0000 7f48b9d9b640 1 --2- 172.21.15.16:0/1766426579 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f48a40a1e00 0x7f48a40a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:02.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.503+0000 7f48ab7fe640 1 -- 172.21.15.16:0/1766426579 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f48ac0ef560 con 0x7f48bc1017a0 2024-03-28T22:08:02.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.504+0000 7f48b9d9b640 1 --2- 172.21.15.16:0/1766426579 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f48a40a1e00 0x7f48a40a42c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f48bc199000 tx=0x7f48b0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:02.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.507+0000 7f48ab7fe640 1 -- 172.21.15.16:0/1766426579 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f48ac0bc7e0 con 0x7f48bc1017a0 2024-03-28T22:08:02.658 INFO:journalctl@ceph.mds.d.smithi016.stdout:Mar 28 22:08:02 smithi016 systemd[1]: Starting Ceph mds.d for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:08:02.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.743+0000 7f48c0d0d640 1 -- 172.21.15.16:0/1766426579 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f48bc02d2d0 con 0x7f48bc1017a0 2024-03-28T22:08:02.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.743+0000 7f48ab7fe640 1 -- 172.21.15.16:0/1766426579 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3043 (secure 0 0 0) 0x7f48ac0c0880 con 0x7f48bc1017a0 2024-03-28T22:08:02.745 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:08:02.745 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-28T22:08:02.745 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 4 2024-03-28T22:08:02.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.745+0000 7f48c0d0d640 1 -- 172.21.15.16:0/1766426579 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f48a40a1e00 msgr2=0x7f48a40a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:02.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.745+0000 7f48c0d0d640 1 --2- 172.21.15.16:0/1766426579 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f48a40a1e00 0x7f48a40a42c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f48bc199000 tx=0x7f48b0005f90 comp rx=0 tx=0).stop 2024-03-28T22:08:02.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.745+0000 7f48c0d0d640 1 -- 172.21.15.16:0/1766426579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48bc1017a0 msgr2=0x7f48bc197af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:02.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.745+0000 7f48c0d0d640 1 --2- 172.21.15.16:0/1766426579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48bc1017a0 0x7f48bc197af0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f48ac000c00 tx=0x7f48ac005d50 comp rx=0 tx=0).stop 2024-03-28T22:08:02.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.745+0000 7f48c0d0d640 1 -- 172.21.15.16:0/1766426579 shutdown_connections 2024-03-28T22:08:02.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.745+0000 7f48c0d0d640 1 --2- 172.21.15.16:0/1766426579 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f48a40a1e00 0x7f48a40a42c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:02.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.745+0000 7f48c0d0d640 1 --2- 172.21.15.16:0/1766426579 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48bc103350 0x7f48bc19f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:02.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.745+0000 7f48c0d0d640 1 --2- 172.21.15.16:0/1766426579 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f48bc102990 0x7f48bc198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:02.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.745+0000 7f48c0d0d640 1 --2- 172.21.15.16:0/1766426579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48bc1017a0 0x7f48bc197af0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:02.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.745+0000 7f48c0d0d640 1 -- 172.21.15.16:0/1766426579 >> 172.21.15.16:0/1766426579 conn(0x7f48bc0fcd10 msgr2=0x7f48bc105cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:02.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.745+0000 7f48c0d0d640 1 -- 172.21.15.16:0/1766426579 shutdown_connections 2024-03-28T22:08:02.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:02.745+0000 7f48c0d0d640 1 -- 172.21.15.16:0/1766426579 wait complete. 2024-03-28T22:08:02.929 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:02 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/1766426579' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:02.930 INFO:journalctl@ceph.mds.d.smithi016.stdout:Mar 28 22:08:02 smithi016 podman[71100]: 2024-03-28 22:08:02.559989646 +0000 UTC m=+0.025201439 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:08:02.930 INFO:journalctl@ceph.mds.d.smithi016.stdout:Mar 28 22:08:02 smithi016 podman[71100]: 2024-03-28 22:08:02.675663439 +0000 UTC m=+0.140875211 container create ef92b8d3de0cf5b93271e267ea76be0895d9de863f6a0870e6d33d438520ca1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-d, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-323d23d, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-28T22:08:02.930 INFO:journalctl@ceph.mds.d.smithi016.stdout:Mar 28 22:08:02 smithi016 podman[71100]: 2024-03-28 22:08:02.903260098 +0000 UTC m=+0.368471873 container init ef92b8d3de0cf5b93271e267ea76be0895d9de863f6a0870e6d33d438520ca1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-d, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-323d23d, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326) 2024-03-28T22:08:02.930 INFO:journalctl@ceph.mds.d.smithi016.stdout:Mar 28 22:08:02 smithi016 podman[71100]: 2024-03-28 22:08:02.905967284 +0000 UTC m=+0.371179057 container start ef92b8d3de0cf5b93271e267ea76be0895d9de863f6a0870e6d33d438520ca1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, RELEASE=squid-323d23d, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-28T22:08:02.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:02 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/1766426579' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:03.186 INFO:journalctl@ceph.mds.d.smithi016.stdout:Mar 28 22:08:02 smithi016 ceph-mds[71121]: -- 172.21.15.16:0/3043848163 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562d6c173a40 con 0x562d6c146c00 2024-03-28T22:08:03.186 INFO:journalctl@ceph.mds.d.smithi016.stdout:Mar 28 22:08:02 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-d[71117]: starting mds.d at 2024-03-28T22:08:03.186 INFO:journalctl@ceph.mds.d.smithi016.stdout:Mar 28 22:08:02 smithi016 bash[71100]: ef92b8d3de0cf5b93271e267ea76be0895d9de863f6a0870e6d33d438520ca1b 2024-03-28T22:08:03.186 INFO:journalctl@ceph.mds.d.smithi016.stdout:Mar 28 22:08:02 smithi016 systemd[1]: Started Ceph mds.d for 798d9884-ed4e-11ee-95d0-87774f69a715. 2024-03-28T22:08:03.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:02 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/1766426579' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:04.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:04 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:04.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:04 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:04.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:04 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:04.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:04 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:08:04.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:04 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:08:04.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:04 smithi053 ceph-mon[35979]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:04.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:04 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:08:04.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:04 smithi053 ceph-mon[35979]: Deploying daemon mds.e on smithi053 2024-03-28T22:08:04.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:04 smithi053 ceph-mon[35979]: mds.? [v2:172.21.15.16:6836/4019085314,v1:172.21.15.16:6837/4019085314] up:boot 2024-03-28T22:08:04.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:04 smithi053 ceph-mon[35979]: fsmap 4 up:standby 2024-03-28T22:08:04.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:04 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T22:08:04.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:04 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:04.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:04 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:04.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:04 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:04.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:04 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:08:04.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:04 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:08:04.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:04 smithi063 ceph-mon[36249]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:04.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:04 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:08:04.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:04 smithi063 ceph-mon[36249]: Deploying daemon mds.e on smithi053 2024-03-28T22:08:04.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:04 smithi063 ceph-mon[36249]: mds.? [v2:172.21.15.16:6836/4019085314,v1:172.21.15.16:6837/4019085314] up:boot 2024-03-28T22:08:04.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:04 smithi063 ceph-mon[36249]: fsmap 4 up:standby 2024-03-28T22:08:04.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:04 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T22:08:04.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:04 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:04.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:04 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:04.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:04 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:04.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:04 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:08:04.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:04 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:08:04.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:04 smithi016 ceph-mon[32374]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:04.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:04 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:08:04.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:04 smithi016 ceph-mon[32374]: Deploying daemon mds.e on smithi053 2024-03-28T22:08:04.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:04 smithi016 ceph-mon[32374]: mds.? [v2:172.21.15.16:6836/4019085314,v1:172.21.15.16:6837/4019085314] up:boot 2024-03-28T22:08:04.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:04 smithi016 ceph-mon[32374]: fsmap 4 up:standby 2024-03-28T22:08:04.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:04 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T22:08:04.759 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:08:04.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.874+0000 7f36e0393640 1 -- 172.21.15.16:0/2399240834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d8103390 msgr2=0x7f36d8109c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:04.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.874+0000 7f36e0393640 1 --2- 172.21.15.16:0/2399240834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d8103390 0x7f36d8109c20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f36d0009ca0 tx=0x7f36d002c0f0 comp rx=0 tx=0).stop 2024-03-28T22:08:04.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.874+0000 7f36e0393640 1 -- 172.21.15.16:0/2399240834 shutdown_connections 2024-03-28T22:08:04.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.874+0000 7f36e0393640 1 --2- 172.21.15.16:0/2399240834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d8103390 0x7f36d8109c20 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:04.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.874+0000 7f36e0393640 1 --2- 172.21.15.16:0/2399240834 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f36d81029d0 0x7f36d8102e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:04.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.874+0000 7f36e0393640 1 --2- 172.21.15.16:0/2399240834 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36d81017e0 0x7f36d8101c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:04.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.874+0000 7f36e0393640 1 -- 172.21.15.16:0/2399240834 >> 172.21.15.16:0/2399240834 conn(0x7f36d80fcd70 msgr2=0x7f36d80ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:04.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.874+0000 7f36e0393640 1 -- 172.21.15.16:0/2399240834 shutdown_connections 2024-03-28T22:08:04.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.875+0000 7f36e0393640 1 -- 172.21.15.16:0/2399240834 wait complete. 2024-03-28T22:08:04.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.875+0000 7f36e0393640 1 Processor -- start 2024-03-28T22:08:04.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.875+0000 7f36e0393640 1 -- start start 2024-03-28T22:08:04.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.875+0000 7f36e0393640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d81017e0 0x7f36d8197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:04.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.875+0000 7f36e0393640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f36d81029d0 0x7f36d8197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:04.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.875+0000 7f36e0393640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36d8103390 0x7f36d819efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:04.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.875+0000 7f36e0393640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f36d810d3f0 con 0x7f36d81017e0 2024-03-28T22:08:04.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.875+0000 7f36e0393640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f36d810d270 con 0x7f36d8103390 2024-03-28T22:08:04.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.875+0000 7f36e0393640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f36d810d570 con 0x7f36d81029d0 2024-03-28T22:08:04.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.875+0000 7f36de12f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d81017e0 0x7f36d8197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:04.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.875+0000 7f36de12f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d81017e0 0x7f36d8197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47986/0 (socket says 172.21.15.16:47986) 2024-03-28T22:08:04.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.875+0000 7f36de12f640 1 -- 172.21.15.16:0/2284859036 learned_addr learned my addr 172.21.15.16:0/2284859036 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:04.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.875+0000 7f36de930640 1 --2- 172.21.15.16:0/2284859036 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36d8103390 0x7f36d819efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:04.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.875+0000 7f36dd92e640 1 --2- 172.21.15.16:0/2284859036 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f36d81029d0 0x7f36d8197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:04.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.876+0000 7f36de12f640 1 -- 172.21.15.16:0/2284859036 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f36d81029d0 msgr2=0x7f36d8197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:04.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.876+0000 7f36de12f640 1 --2- 172.21.15.16:0/2284859036 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f36d81029d0 0x7f36d8197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:04.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.876+0000 7f36de12f640 1 -- 172.21.15.16:0/2284859036 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36d8103390 msgr2=0x7f36d819efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:04.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.876+0000 7f36de12f640 1 --2- 172.21.15.16:0/2284859036 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36d8103390 0x7f36d819efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:04.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.876+0000 7f36de12f640 1 -- 172.21.15.16:0/2284859036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36d819f620 con 0x7f36d81017e0 2024-03-28T22:08:04.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.876+0000 7f36de930640 1 --2- 172.21.15.16:0/2284859036 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36d8103390 0x7f36d819efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:08:04.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.876+0000 7f36de12f640 1 --2- 172.21.15.16:0/2284859036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d81017e0 0x7f36d8197a00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f36cc00e9c0 tx=0x7f36cc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:04.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.876+0000 7f36cb7fe640 1 -- 172.21.15.16:0/2284859036 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36cc00cd80 con 0x7f36d81017e0 2024-03-28T22:08:04.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.876+0000 7f36cb7fe640 1 -- 172.21.15.16:0/2284859036 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36cc0035f0 con 0x7f36d81017e0 2024-03-28T22:08:04.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.876+0000 7f36e0393640 1 -- 172.21.15.16:0/2284859036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36d819f910 con 0x7f36d81017e0 2024-03-28T22:08:04.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.876+0000 7f36cb7fe640 1 -- 172.21.15.16:0/2284859036 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36cc010690 con 0x7f36d81017e0 2024-03-28T22:08:04.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.876+0000 7f36e0393640 1 -- 172.21.15.16:0/2284859036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36d819fe60 con 0x7f36d81017e0 2024-03-28T22:08:04.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.877+0000 7f36e0393640 1 -- 172.21.15.16:0/2284859036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36d8108aa0 con 0x7f36d81017e0 2024-03-28T22:08:04.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.878+0000 7f36cb7fe640 1 -- 172.21.15.16:0/2284859036 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f36cc0037a0 con 0x7f36d81017e0 2024-03-28T22:08:04.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.879+0000 7f36cb7fe640 1 --2- 172.21.15.16:0/2284859036 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f36c40a1e00 0x7f36c40a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:04.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.879+0000 7f36dd92e640 1 --2- 172.21.15.16:0/2284859036 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f36c40a1e00 0x7f36c40a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:04.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.879+0000 7f36cb7fe640 1 -- 172.21.15.16:0/2284859036 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f36cc0cdd30 con 0x7f36d81017e0 2024-03-28T22:08:04.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.879+0000 7f36dd92e640 1 --2- 172.21.15.16:0/2284859036 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f36c40a1e00 0x7f36c40a42c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f36d8198f10 tx=0x7f36d4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:04.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:04.883+0000 7f36cb7fe640 1 -- 172.21.15.16:0/2284859036 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f36cc0d3020 con 0x7f36d81017e0 2024-03-28T22:08:05.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:05.107+0000 7f36e0393640 1 -- 172.21.15.16:0/2284859036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f36d802d3f0 con 0x7f36d81017e0 2024-03-28T22:08:05.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:05.107+0000 7f36cb7fe640 1 -- 172.21.15.16:0/2284859036 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3868 (secure 0 0 0) 0x7f36cc09b560 con 0x7f36d81017e0 2024-03-28T22:08:05.109 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:08:05.109 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-28T22:08:05.109 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 5 2024-03-28T22:08:05.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:05.109+0000 7f36e0393640 1 -- 172.21.15.16:0/2284859036 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f36c40a1e00 msgr2=0x7f36c40a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:05.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:05.109+0000 7f36e0393640 1 --2- 172.21.15.16:0/2284859036 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f36c40a1e00 0x7f36c40a42c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f36d8198f10 tx=0x7f36d4005f90 comp rx=0 tx=0).stop 2024-03-28T22:08:05.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:05.109+0000 7f36e0393640 1 -- 172.21.15.16:0/2284859036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d81017e0 msgr2=0x7f36d8197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:05.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:05.109+0000 7f36e0393640 1 --2- 172.21.15.16:0/2284859036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d81017e0 0x7f36d8197a00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f36cc00e9c0 tx=0x7f36cc00ee80 comp rx=0 tx=0).stop 2024-03-28T22:08:05.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:05.109+0000 7f36e0393640 1 -- 172.21.15.16:0/2284859036 shutdown_connections 2024-03-28T22:08:05.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:05.109+0000 7f36e0393640 1 --2- 172.21.15.16:0/2284859036 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f36c40a1e00 0x7f36c40a42c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:05.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:05.109+0000 7f36e0393640 1 --2- 172.21.15.16:0/2284859036 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36d8103390 0x7f36d819efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:05.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:05.109+0000 7f36e0393640 1 --2- 172.21.15.16:0/2284859036 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f36d81029d0 0x7f36d8197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:05.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:05.109+0000 7f36e0393640 1 --2- 172.21.15.16:0/2284859036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d81017e0 0x7f36d8197a00 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:05.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:05.109+0000 7f36e0393640 1 -- 172.21.15.16:0/2284859036 >> 172.21.15.16:0/2284859036 conn(0x7f36d80fcd70 msgr2=0x7f36d8105c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:05.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:05.109+0000 7f36e0393640 1 -- 172.21.15.16:0/2284859036 shutdown_connections 2024-03-28T22:08:05.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:05.109+0000 7f36e0393640 1 -- 172.21.15.16:0/2284859036 wait complete. 2024-03-28T22:08:05.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:05 smithi053 ceph-mon[35979]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:05.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:05 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2284859036' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:05.298 INFO:journalctl@ceph.mds.e.smithi053.stdout:Mar 28 22:08:05 smithi053 systemd[1]: Starting Ceph mds.e for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:08:05.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:05 smithi063 ceph-mon[36249]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:05.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:05 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2284859036' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:05.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:05 smithi016 ceph-mon[32374]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:05.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:05 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2284859036' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:05.639 INFO:journalctl@ceph.mds.e.smithi053.stdout:Mar 28 22:08:05 smithi053 podman[65814]: 2024-03-28 22:08:05.286571009 +0000 UTC m=+0.017823178 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:08:05.640 INFO:journalctl@ceph.mds.e.smithi053.stdout:Mar 28 22:08:05 smithi053 podman[65814]: 2024-03-28 22:08:05.386121276 +0000 UTC m=+0.117373430 container create 71e30f2bb06caa0523c2a575d9a073e33918894a1d13aea86e7b33c9f5b21f72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-e, GIT_BRANCH=HEAD, RELEASE=squid-323d23d, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T22:08:05.640 INFO:journalctl@ceph.mds.e.smithi053.stdout:Mar 28 22:08:05 smithi053 podman[65814]: 2024-03-28 22:08:05.638336975 +0000 UTC m=+0.369589136 container init 71e30f2bb06caa0523c2a575d9a073e33918894a1d13aea86e7b33c9f5b21f72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-e, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=squid-323d23d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-28T22:08:06.047 INFO:journalctl@ceph.mds.e.smithi053.stdout:Mar 28 22:08:05 smithi053 podman[65814]: 2024-03-28 22:08:05.641331604 +0000 UTC m=+0.372583760 container start 71e30f2bb06caa0523c2a575d9a073e33918894a1d13aea86e7b33c9f5b21f72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-e, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-323d23d, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-28T22:08:06.047 INFO:journalctl@ceph.mds.e.smithi053.stdout:Mar 28 22:08:05 smithi053 ceph-mds[65833]: -- 172.21.15.53:0/1677659939 <== mon.2 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b475cb7a40 con 0x55b475c8a800 2024-03-28T22:08:06.047 INFO:journalctl@ceph.mds.e.smithi053.stdout:Mar 28 22:08:05 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-e[65829]: starting mds.e at 2024-03-28T22:08:06.048 INFO:journalctl@ceph.mds.e.smithi053.stdout:Mar 28 22:08:05 smithi053 bash[65814]: 71e30f2bb06caa0523c2a575d9a073e33918894a1d13aea86e7b33c9f5b21f72 2024-03-28T22:08:06.048 INFO:journalctl@ceph.mds.e.smithi053.stdout:Mar 28 22:08:05 smithi053 systemd[1]: Started Ceph mds.e for 798d9884-ed4e-11ee-95d0-87774f69a715. 2024-03-28T22:08:07.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:06 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:07.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:06 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:07.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:06 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:07.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:06 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:08:07.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:06 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:08:07.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:06 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:08:07.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:06 smithi053 ceph-mon[35979]: Deploying daemon mds.f on smithi063 2024-03-28T22:08:07.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:06 smithi053 ceph-mon[35979]: mds.? [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] up:boot 2024-03-28T22:08:07.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:06 smithi053 ceph-mon[35979]: fsmap 5 up:standby 2024-03-28T22:08:07.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:06 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T22:08:07.124 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:08:07.152 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:06 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:07.152 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:06 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:07.152 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:06 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:07.152 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:06 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:08:07.152 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:06 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:08:07.153 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:06 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:08:07.153 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:06 smithi016 ceph-mon[32374]: Deploying daemon mds.f on smithi063 2024-03-28T22:08:07.153 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:06 smithi016 ceph-mon[32374]: mds.? [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] up:boot 2024-03-28T22:08:07.153 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:06 smithi016 ceph-mon[32374]: fsmap 5 up:standby 2024-03-28T22:08:07.153 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:06 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T22:08:07.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:06 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:07.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:06 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:07.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:06 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:07.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:06 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:08:07.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:06 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:08:07.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:06 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:08:07.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:06 smithi063 ceph-mon[36249]: Deploying daemon mds.f on smithi063 2024-03-28T22:08:07.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:06 smithi063 ceph-mon[36249]: mds.? [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] up:boot 2024-03-28T22:08:07.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:06 smithi063 ceph-mon[36249]: fsmap 5 up:standby 2024-03-28T22:08:07.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:06 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T22:08:07.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.238+0000 7f6c03747640 1 -- 172.21.15.16:0/3002154886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bfc0ffff0 msgr2=0x7f6bfc100410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:07.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.238+0000 7f6c03747640 1 --2- 172.21.15.16:0/3002154886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bfc0ffff0 0x7f6bfc100410 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6bf4009900 tx=0x7f6bf402bc30 comp rx=0 tx=0).stop 2024-03-28T22:08:07.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.238+0000 7f6c03747640 1 -- 172.21.15.16:0/3002154886 shutdown_connections 2024-03-28T22:08:07.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.238+0000 7f6c03747640 1 --2- 172.21.15.16:0/3002154886 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6bfc1012f0 0x7f6bfc101770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:07.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.238+0000 7f6c03747640 1 --2- 172.21.15.16:0/3002154886 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6bfc100950 0x7f6bfc100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:07.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.238+0000 7f6c03747640 1 --2- 172.21.15.16:0/3002154886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bfc0ffff0 0x7f6bfc100410 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:07.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.238+0000 7f6c03747640 1 -- 172.21.15.16:0/3002154886 >> 172.21.15.16:0/3002154886 conn(0x7f6bfc0fbbf0 msgr2=0x7f6bfc0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:07.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.238+0000 7f6c03747640 1 -- 172.21.15.16:0/3002154886 shutdown_connections 2024-03-28T22:08:07.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.238+0000 7f6c03747640 1 -- 172.21.15.16:0/3002154886 wait complete. 2024-03-28T22:08:07.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.238+0000 7f6c03747640 1 Processor -- start 2024-03-28T22:08:07.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.238+0000 7f6c03747640 1 -- start start 2024-03-28T22:08:07.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.238+0000 7f6c03747640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bfc0ffff0 0x7f6bfc199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:07.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.238+0000 7f6c03747640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6bfc100950 0x7f6bfc19a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:07.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.239+0000 7f6c03747640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6bfc1012f0 0x7f6bfc1a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:07.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.239+0000 7f6c03747640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6bfc10f5c0 con 0x7f6bfc0ffff0 2024-03-28T22:08:07.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.239+0000 7f6c03747640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f6bfc10f440 con 0x7f6bfc100950 2024-03-28T22:08:07.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.239+0000 7f6c03747640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6bfc10f740 con 0x7f6bfc1012f0 2024-03-28T22:08:07.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.239+0000 7f6c014e3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bfc0ffff0 0x7f6bfc199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:07.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.239+0000 7f6c014e3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bfc0ffff0 0x7f6bfc199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47996/0 (socket says 172.21.15.16:47996) 2024-03-28T22:08:07.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.239+0000 7f6c014e3640 1 -- 172.21.15.16:0/3371535256 learned_addr learned my addr 172.21.15.16:0/3371535256 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:07.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.239+0000 7f6c01ce4640 1 --2- 172.21.15.16:0/3371535256 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6bfc1012f0 0x7f6bfc1a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:07.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.239+0000 7f6c014e3640 1 -- 172.21.15.16:0/3371535256 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6bfc1012f0 msgr2=0x7f6bfc1a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:07.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.239+0000 7f6c014e3640 1 --2- 172.21.15.16:0/3371535256 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6bfc1012f0 0x7f6bfc1a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:07.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.239+0000 7f6c014e3640 1 -- 172.21.15.16:0/3371535256 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6bfc100950 msgr2=0x7f6bfc19a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:07.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.239+0000 7f6c014e3640 1 --2- 172.21.15.16:0/3371535256 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6bfc100950 0x7f6bfc19a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:07.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.239+0000 7f6c014e3640 1 -- 172.21.15.16:0/3371535256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bfc1a17f0 con 0x7f6bfc0ffff0 2024-03-28T22:08:07.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.239+0000 7f6c014e3640 1 --2- 172.21.15.16:0/3371535256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bfc0ffff0 0x7f6bfc199bd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6bf4009230 tx=0x7f6bf4005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:07.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.239+0000 7f6bf27fc640 1 -- 172.21.15.16:0/3371535256 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf4041070 con 0x7f6bfc0ffff0 2024-03-28T22:08:07.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.239+0000 7f6c03747640 1 -- 172.21.15.16:0/3371535256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bfc1a1a80 con 0x7f6bfc0ffff0 2024-03-28T22:08:07.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.239+0000 7f6bf27fc640 1 -- 172.21.15.16:0/3371535256 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6bf40054e0 con 0x7f6bfc0ffff0 2024-03-28T22:08:07.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.239+0000 7f6bf27fc640 1 -- 172.21.15.16:0/3371535256 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf4034440 con 0x7f6bfc0ffff0 2024-03-28T22:08:07.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.239+0000 7f6c03747640 1 -- 172.21.15.16:0/3371535256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bfc1a1fc0 con 0x7f6bfc0ffff0 2024-03-28T22:08:07.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.240+0000 7f6c03747640 1 -- 172.21.15.16:0/3371535256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bfc0685d0 con 0x7f6bfc0ffff0 2024-03-28T22:08:07.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.241+0000 7f6bf27fc640 1 -- 172.21.15.16:0/3371535256 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f6bf4044070 con 0x7f6bfc0ffff0 2024-03-28T22:08:07.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.242+0000 7f6bf27fc640 1 --2- 172.21.15.16:0/3371535256 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f6be80a1d80 0x7f6be80a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:07.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.242+0000 7f6c00ce2640 1 --2- 172.21.15.16:0/3371535256 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f6be80a1d80 0x7f6be80a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:07.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.242+0000 7f6bf27fc640 1 -- 172.21.15.16:0/3371535256 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f6bf40ef740 con 0x7f6bfc0ffff0 2024-03-28T22:08:07.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.243+0000 7f6c00ce2640 1 --2- 172.21.15.16:0/3371535256 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f6be80a1d80 0x7f6be80a4240 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc19b0e0 tx=0x7f6bec005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:07.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.247+0000 7f6bf27fc640 1 -- 172.21.15.16:0/3371535256 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6bf40bcf70 con 0x7f6bfc0ffff0 2024-03-28T22:08:07.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.466+0000 7f6c03747640 1 -- 172.21.15.16:0/3371535256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6bfc10ae90 con 0x7f6bfc0ffff0 2024-03-28T22:08:07.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.467+0000 7f6bf27fc640 1 -- 172.21.15.16:0/3371535256 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4690 (secure 0 0 0) 0x7f6bf40bc870 con 0x7f6bfc0ffff0 2024-03-28T22:08:07.469 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:08:07.469 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/350727634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":350727634},{"type":"v1","addr":"172.21.15.53:6835","nonce":350727634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-28T22:08:07.469 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 6 2024-03-28T22:08:07.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.469+0000 7f6c03747640 1 -- 172.21.15.16:0/3371535256 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f6be80a1d80 msgr2=0x7f6be80a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:07.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.469+0000 7f6c03747640 1 --2- 172.21.15.16:0/3371535256 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f6be80a1d80 0x7f6be80a4240 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc19b0e0 tx=0x7f6bec005c70 comp rx=0 tx=0).stop 2024-03-28T22:08:07.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.469+0000 7f6c03747640 1 -- 172.21.15.16:0/3371535256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bfc0ffff0 msgr2=0x7f6bfc199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:07.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.469+0000 7f6c03747640 1 --2- 172.21.15.16:0/3371535256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bfc0ffff0 0x7f6bfc199bd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6bf4009230 tx=0x7f6bf4005dd0 comp rx=0 tx=0).stop 2024-03-28T22:08:07.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.469+0000 7f6c03747640 1 -- 172.21.15.16:0/3371535256 shutdown_connections 2024-03-28T22:08:07.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.469+0000 7f6c03747640 1 --2- 172.21.15.16:0/3371535256 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f6be80a1d80 0x7f6be80a4240 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:07.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.469+0000 7f6c03747640 1 --2- 172.21.15.16:0/3371535256 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6bfc1012f0 0x7f6bfc1a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:07.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.469+0000 7f6c03747640 1 --2- 172.21.15.16:0/3371535256 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6bfc100950 0x7f6bfc19a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:07.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.469+0000 7f6c03747640 1 --2- 172.21.15.16:0/3371535256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bfc0ffff0 0x7f6bfc199bd0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:07.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.469+0000 7f6c03747640 1 -- 172.21.15.16:0/3371535256 >> 172.21.15.16:0/3371535256 conn(0x7f6bfc0fbbf0 msgr2=0x7f6bfc0fd2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:07.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.469+0000 7f6c03747640 1 -- 172.21.15.16:0/3371535256 shutdown_connections 2024-03-28T22:08:07.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:07.469+0000 7f6c03747640 1 -- 172.21.15.16:0/3371535256 wait complete. 2024-03-28T22:08:07.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:07 smithi063 ceph-mon[36249]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:07.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:07 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/3371535256' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:07.932 INFO:journalctl@ceph.mds.f.smithi063.stdout:Mar 28 22:08:07 smithi063 systemd[1]: Starting Ceph mds.f for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:08:08.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:07 smithi053 ceph-mon[35979]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:08.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:07 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3371535256' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:08.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:07 smithi016 ceph-mon[32374]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:08.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:07 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3371535256' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:08.275 INFO:journalctl@ceph.mds.f.smithi063.stdout:Mar 28 22:08:08 smithi063 podman[66531]: 2024-03-28 22:08:07.906259589 +0000 UTC m=+0.018432106 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:08:08.275 INFO:journalctl@ceph.mds.f.smithi063.stdout:Mar 28 22:08:08 smithi063 podman[66531]: 2024-03-28 22:08:08.019773486 +0000 UTC m=+0.131946034 container create ce5a887a2420436a5c5431b71ea4e726a1a579bf40bce69d678834ae0cef7b8d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-f, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-323d23d, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T22:08:08.275 INFO:journalctl@ceph.mds.f.smithi063.stdout:Mar 28 22:08:08 smithi063 podman[66531]: 2024-03-28 22:08:08.247246124 +0000 UTC m=+0.359418648 container init ce5a887a2420436a5c5431b71ea4e726a1a579bf40bce69d678834ae0cef7b8d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-f, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-323d23d, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-28T22:08:08.275 INFO:journalctl@ceph.mds.f.smithi063.stdout:Mar 28 22:08:08 smithi063 podman[66531]: 2024-03-28 22:08:08.250095584 +0000 UTC m=+0.362268094 container start ce5a887a2420436a5c5431b71ea4e726a1a579bf40bce69d678834ae0cef7b8d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-f, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-323d23d, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-28T22:08:08.276 INFO:journalctl@ceph.mds.f.smithi063.stdout:Mar 28 22:08:08 smithi063 ceph-mds[66550]: -- 172.21.15.63:0/583500298 <== mon.1 v2:172.21.15.63:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f64dc35a40 con 0x55f64dc09000 2024-03-28T22:08:08.682 INFO:journalctl@ceph.mds.f.smithi063.stdout:Mar 28 22:08:08 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-f[66546]: starting mds.f at 2024-03-28T22:08:08.682 INFO:journalctl@ceph.mds.f.smithi063.stdout:Mar 28 22:08:08 smithi063 bash[66531]: ce5a887a2420436a5c5431b71ea4e726a1a579bf40bce69d678834ae0cef7b8d 2024-03-28T22:08:08.682 INFO:journalctl@ceph.mds.f.smithi063.stdout:Mar 28 22:08:08 smithi063 systemd[1]: Started Ceph mds.f for 798d9884-ed4e-11ee-95d0-87774f69a715. 2024-03-28T22:08:09.483 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:08:09.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.596+0000 7f65b22c0640 1 -- 172.21.15.16:0/2754393588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65ac104b30 msgr2=0x7f65ac106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:09.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.596+0000 7f65b22c0640 1 --2- 172.21.15.16:0/2754393588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65ac104b30 0x7f65ac106f20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f65a0009900 tx=0x7f65a002bc30 comp rx=0 tx=0).stop 2024-03-28T22:08:09.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.596+0000 7f65b22c0640 1 -- 172.21.15.16:0/2754393588 shutdown_connections 2024-03-28T22:08:09.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.596+0000 7f65b22c0640 1 --2- 172.21.15.16:0/2754393588 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65ac107460 0x7f65ac1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:09.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.596+0000 7f65b22c0640 1 --2- 172.21.15.16:0/2754393588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65ac104b30 0x7f65ac106f20 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:09.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.596+0000 7f65b22c0640 1 --2- 172.21.15.16:0/2754393588 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65ac102200 0x7f65ac1045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:09.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.596+0000 7f65b22c0640 1 -- 172.21.15.16:0/2754393588 >> 172.21.15.16:0/2754393588 conn(0x7f65ac0fbbf0 msgr2=0x7f65ac0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:09.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.596+0000 7f65b22c0640 1 -- 172.21.15.16:0/2754393588 shutdown_connections 2024-03-28T22:08:09.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.596+0000 7f65b22c0640 1 -- 172.21.15.16:0/2754393588 wait complete. 2024-03-28T22:08:09.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.596+0000 7f65b22c0640 1 Processor -- start 2024-03-28T22:08:09.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.596+0000 7f65b22c0640 1 -- start start 2024-03-28T22:08:09.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.597+0000 7f65b22c0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65ac102200 0x7f65ac197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:09.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.597+0000 7f65b22c0640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65ac104b30 0x7f65ac198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:09.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.597+0000 7f65b22c0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65ac107460 0x7f65ac19f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:09.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.597+0000 7f65b22c0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f65ac10d400 con 0x7f65ac102200 2024-03-28T22:08:09.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.597+0000 7f65b22c0640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f65ac10d280 con 0x7f65ac104b30 2024-03-28T22:08:09.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.597+0000 7f65b22c0640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f65ac10d580 con 0x7f65ac107460 2024-03-28T22:08:09.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.597+0000 7f65abfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65ac102200 0x7f65ac197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.597+0000 7f65abfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65ac102200 0x7f65ac197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48018/0 (socket says 172.21.15.16:48018) 2024-03-28T22:08:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.597+0000 7f65abfff640 1 -- 172.21.15.16:0/2675458919 learned_addr learned my addr 172.21.15.16:0/2675458919 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.597+0000 7f65b085d640 1 --2- 172.21.15.16:0/2675458919 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65ac107460 0x7f65ac19f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.597+0000 7f65ab7fe640 1 --2- 172.21.15.16:0/2675458919 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65ac104b30 0x7f65ac198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.597+0000 7f65abfff640 1 -- 172.21.15.16:0/2675458919 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65ac107460 msgr2=0x7f65ac19f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.597+0000 7f65abfff640 1 --2- 172.21.15.16:0/2675458919 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65ac107460 0x7f65ac19f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.597+0000 7f65abfff640 1 -- 172.21.15.16:0/2675458919 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65ac104b30 msgr2=0x7f65ac198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.597+0000 7f65abfff640 1 --2- 172.21.15.16:0/2675458919 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65ac104b30 0x7f65ac198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.597+0000 7f65abfff640 1 -- 172.21.15.16:0/2675458919 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65ac19f700 con 0x7f65ac102200 2024-03-28T22:08:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.597+0000 7f65abfff640 1 --2- 172.21.15.16:0/2675458919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65ac102200 0x7f65ac197ae0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f659c00d950 tx=0x7f659c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.598+0000 7f65a97fa640 1 -- 172.21.15.16:0/2675458919 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f659c017070 con 0x7f65ac102200 2024-03-28T22:08:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.598+0000 7f65a97fa640 1 -- 172.21.15.16:0/2675458919 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f659c009c30 con 0x7f65ac102200 2024-03-28T22:08:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.598+0000 7f65b22c0640 1 -- 172.21.15.16:0/2675458919 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65ac19f9f0 con 0x7f65ac102200 2024-03-28T22:08:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.598+0000 7f65a97fa640 1 -- 172.21.15.16:0/2675458919 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f659c00f6b0 con 0x7f65ac102200 2024-03-28T22:08:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.598+0000 7f65b22c0640 1 -- 172.21.15.16:0/2675458919 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65ac19ff10 con 0x7f65ac102200 2024-03-28T22:08:09.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.599+0000 7f65b22c0640 1 -- 172.21.15.16:0/2675458919 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65ac0fd6e0 con 0x7f65ac102200 2024-03-28T22:08:09.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.600+0000 7f65a97fa640 1 -- 172.21.15.16:0/2675458919 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f659c0109d0 con 0x7f65ac102200 2024-03-28T22:08:09.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.600+0000 7f65a97fa640 1 --2- 172.21.15.16:0/2675458919 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f65940a1d80 0x7f65940a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:09.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.601+0000 7f65ab7fe640 1 --2- 172.21.15.16:0/2675458919 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f65940a1d80 0x7f65940a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:09.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.601+0000 7f65a97fa640 1 -- 172.21.15.16:0/2675458919 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f659c0cd490 con 0x7f65ac102200 2024-03-28T22:08:09.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.601+0000 7f65ab7fe640 1 --2- 172.21.15.16:0/2675458919 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f65940a1d80 0x7f65940a4240 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f65a0000c00 tx=0x7f65a0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:09.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.604+0000 7f65a97fa640 1 -- 172.21.15.16:0/2675458919 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f659c009de0 con 0x7f65ac102200 2024-03-28T22:08:09.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:09 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:09.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:09 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:09.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:09 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:09.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:09 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:08:09.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:09 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:08:09.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:09 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:08:09.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:09 smithi063 ceph-mon[36249]: Deploying daemon mds.c on smithi063 2024-03-28T22:08:09.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:09 smithi063 ceph-mon[36249]: mds.? [v2:172.21.15.63:6834/247249857,v1:172.21.15.63:6835/247249857] up:boot 2024-03-28T22:08:09.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:09 smithi063 ceph-mon[36249]: fsmap 6 up:standby 2024-03-28T22:08:09.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:09 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T22:08:09.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:09 smithi063 ceph-mon[36249]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:09.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:09 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:09.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:09 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:09.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:09 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:09.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:09 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:08:09.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:09 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:08:09.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:09 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:08:09.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:09 smithi016 ceph-mon[32374]: Deploying daemon mds.c on smithi063 2024-03-28T22:08:09.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:09 smithi016 ceph-mon[32374]: mds.? [v2:172.21.15.63:6834/247249857,v1:172.21.15.63:6835/247249857] up:boot 2024-03-28T22:08:09.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:09 smithi016 ceph-mon[32374]: fsmap 6 up:standby 2024-03-28T22:08:09.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:09 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T22:08:09.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:09 smithi016 ceph-mon[32374]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:09.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:09 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:09.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:09 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:09.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:09 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:09.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:09 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:08:09.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:09 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:08:09.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:09 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:08:09.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:09 smithi053 ceph-mon[35979]: Deploying daemon mds.c on smithi063 2024-03-28T22:08:09.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:09 smithi053 ceph-mon[35979]: mds.? [v2:172.21.15.63:6834/247249857,v1:172.21.15.63:6835/247249857] up:boot 2024-03-28T22:08:09.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:09 smithi053 ceph-mon[35979]: fsmap 6 up:standby 2024-03-28T22:08:09.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:09 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T22:08:09.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:09 smithi053 ceph-mon[35979]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:09.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.829+0000 7f65b22c0640 1 -- 172.21.15.16:0/2675458919 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f65ac02d1a0 con 0x7f65ac102200 2024-03-28T22:08:09.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.829+0000 7f65a97fa640 1 -- 172.21.15.16:0/2675458919 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5512 (secure 0 0 0) 0x7f659c09a710 con 0x7f65ac102200 2024-03-28T22:08:09.831 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:08:09.831 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/247249857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":247249857},{"type":"v1","addr":"172.21.15.63:6835","nonce":247249857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/350727634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":350727634},{"type":"v1","addr":"172.21.15.53:6835","nonce":350727634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-28T22:08:09.832 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 7 2024-03-28T22:08:09.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.831+0000 7f65b22c0640 1 -- 172.21.15.16:0/2675458919 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f65940a1d80 msgr2=0x7f65940a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:09.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.831+0000 7f65b22c0640 1 --2- 172.21.15.16:0/2675458919 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f65940a1d80 0x7f65940a4240 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f65a0000c00 tx=0x7f65a0005fd0 comp rx=0 tx=0).stop 2024-03-28T22:08:09.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.831+0000 7f65b22c0640 1 -- 172.21.15.16:0/2675458919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65ac102200 msgr2=0x7f65ac197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:09.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.831+0000 7f65b22c0640 1 --2- 172.21.15.16:0/2675458919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65ac102200 0x7f65ac197ae0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f659c00d950 tx=0x7f659c00de10 comp rx=0 tx=0).stop 2024-03-28T22:08:09.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.831+0000 7f65b22c0640 1 -- 172.21.15.16:0/2675458919 shutdown_connections 2024-03-28T22:08:09.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.831+0000 7f65b22c0640 1 --2- 172.21.15.16:0/2675458919 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f65940a1d80 0x7f65940a4240 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:09.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.831+0000 7f65b22c0640 1 --2- 172.21.15.16:0/2675458919 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65ac107460 0x7f65ac19f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:09.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.831+0000 7f65b22c0640 1 --2- 172.21.15.16:0/2675458919 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65ac104b30 0x7f65ac198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:09.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.831+0000 7f65b22c0640 1 --2- 172.21.15.16:0/2675458919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65ac102200 0x7f65ac197ae0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:09.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.831+0000 7f65b22c0640 1 -- 172.21.15.16:0/2675458919 >> 172.21.15.16:0/2675458919 conn(0x7f65ac0fbbf0 msgr2=0x7f65ac102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:09.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.831+0000 7f65b22c0640 1 -- 172.21.15.16:0/2675458919 shutdown_connections 2024-03-28T22:08:09.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:09.831+0000 7f65b22c0640 1 -- 172.21.15.16:0/2675458919 wait complete. 2024-03-28T22:08:10.682 INFO:journalctl@ceph.mds.c.smithi063.stdout:Mar 28 22:08:10 smithi063 systemd[1]: Starting Ceph mds.c for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:08:10.968 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:10 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2675458919' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:10.968 INFO:journalctl@ceph.mds.c.smithi063.stdout:Mar 28 22:08:10 smithi063 podman[66842]: 2024-03-28 22:08:10.705616095 +0000 UTC m=+0.096467963 container create 60896cd5fa5f2236598d0550ecee9e0778470a91d942c5d45f82678f40d2b6ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-c, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-323d23d, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True) 2024-03-28T22:08:10.968 INFO:journalctl@ceph.mds.c.smithi063.stdout:Mar 28 22:08:10 smithi063 podman[66842]: 2024-03-28 22:08:10.627783633 +0000 UTC m=+0.018635497 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:08:10.968 INFO:journalctl@ceph.mds.c.smithi063.stdout:Mar 28 22:08:10 smithi063 podman[66842]: 2024-03-28 22:08:10.941179498 +0000 UTC m=+0.332031348 container init 60896cd5fa5f2236598d0550ecee9e0778470a91d942c5d45f82678f40d2b6ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-c, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-323d23d, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True) 2024-03-28T22:08:10.969 INFO:journalctl@ceph.mds.c.smithi063.stdout:Mar 28 22:08:10 smithi063 podman[66842]: 2024-03-28 22:08:10.943876463 +0000 UTC m=+0.334728311 container start 60896cd5fa5f2236598d0550ecee9e0778470a91d942c5d45f82678f40d2b6ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_CLEAN=True, org.label-schema.build-date=20240326, ceph=True, RELEASE=squid-323d23d, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-28T22:08:10.969 INFO:journalctl@ceph.mds.c.smithi063.stdout:Mar 28 22:08:10 smithi063 ceph-mds[66861]: -- 172.21.15.63:0/769297016 <== mon.1 v2:172.21.15.63:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d033f89a40 con 0x55d033f5d000 2024-03-28T22:08:11.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:10 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2675458919' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:11.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:10 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2675458919' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:11.346 INFO:journalctl@ceph.mds.c.smithi063.stdout:Mar 28 22:08:10 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-c[66857]: starting mds.c at 2024-03-28T22:08:11.346 INFO:journalctl@ceph.mds.c.smithi063.stdout:Mar 28 22:08:11 smithi063 bash[66842]: 60896cd5fa5f2236598d0550ecee9e0778470a91d942c5d45f82678f40d2b6ea 2024-03-28T22:08:11.346 INFO:journalctl@ceph.mds.c.smithi063.stdout:Mar 28 22:08:11 smithi063 systemd[1]: Started Ceph mds.c for 798d9884-ed4e-11ee-95d0-87774f69a715. 2024-03-28T22:08:11.845 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:08:11.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.960+0000 7f0efcaae640 1 -- 172.21.15.16:0/1027571102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ef8102230 msgr2=0x7f0ef8108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:11.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.960+0000 7f0efcaae640 1 --2- 172.21.15.16:0/1027571102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ef8102230 0x7f0ef8108ac0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009900 tx=0x7f0ee802bc30 comp rx=0 tx=0).stop 2024-03-28T22:08:11.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.960+0000 7f0efcaae640 1 -- 172.21.15.16:0/1027571102 shutdown_connections 2024-03-28T22:08:11.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.960+0000 7f0efcaae640 1 --2- 172.21.15.16:0/1027571102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ef8102230 0x7f0ef8108ac0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:11.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.960+0000 7f0efcaae640 1 --2- 172.21.15.16:0/1027571102 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ef8101870 0x7f0ef8101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:11.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.960+0000 7f0efcaae640 1 --2- 172.21.15.16:0/1027571102 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ef8100680 0x7f0ef8100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:11.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.960+0000 7f0efcaae640 1 -- 172.21.15.16:0/1027571102 >> 172.21.15.16:0/1027571102 conn(0x7f0ef80fbc10 msgr2=0x7f0ef80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:11.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.960+0000 7f0efcaae640 1 -- 172.21.15.16:0/1027571102 shutdown_connections 2024-03-28T22:08:11.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.960+0000 7f0efcaae640 1 -- 172.21.15.16:0/1027571102 wait complete. 2024-03-28T22:08:11.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.961+0000 7f0efcaae640 1 Processor -- start 2024-03-28T22:08:11.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.961+0000 7f0efcaae640 1 -- start start 2024-03-28T22:08:11.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.961+0000 7f0efcaae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ef8100680 0x7f0ef8197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:11.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.961+0000 7f0efcaae640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ef8101870 0x7f0ef8197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:11.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.961+0000 7f0efcaae640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ef8102230 0x7f0ef819efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:11.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.961+0000 7f0efcaae640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef810c300 con 0x7f0ef8100680 2024-03-28T22:08:11.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.961+0000 7f0efcaae640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef810c180 con 0x7f0ef8102230 2024-03-28T22:08:11.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.961+0000 7f0efcaae640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef810c480 con 0x7f0ef8101870 2024-03-28T22:08:11.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.961+0000 7f0ef659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ef8100680 0x7f0ef8197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:11.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.961+0000 7f0ef659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ef8100680 0x7f0ef8197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48896/0 (socket says 172.21.15.16:48896) 2024-03-28T22:08:11.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.961+0000 7f0ef659c640 1 -- 172.21.15.16:0/56996033 learned_addr learned my addr 172.21.15.16:0/56996033 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:11.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.961+0000 7f0ef6d9d640 1 --2- 172.21.15.16:0/56996033 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ef8102230 0x7f0ef819efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:11.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.961+0000 7f0ef5d9b640 1 --2- 172.21.15.16:0/56996033 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ef8101870 0x7f0ef8197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:11.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.961+0000 7f0ef659c640 1 -- 172.21.15.16:0/56996033 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ef8101870 msgr2=0x7f0ef8197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:11.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.961+0000 7f0ef659c640 1 --2- 172.21.15.16:0/56996033 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ef8101870 0x7f0ef8197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:11.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.961+0000 7f0ef659c640 1 -- 172.21.15.16:0/56996033 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ef8102230 msgr2=0x7f0ef819efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:11.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.961+0000 7f0ef659c640 1 --2- 172.21.15.16:0/56996033 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ef8102230 0x7f0ef819efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:11.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.961+0000 7f0ef659c640 1 -- 172.21.15.16:0/56996033 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ef819f640 con 0x7f0ef8100680 2024-03-28T22:08:11.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.962+0000 7f0ef659c640 1 --2- 172.21.15.16:0/56996033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ef8100680 0x7f0ef8197a70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0ef000eb70 tx=0x7f0ef000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:11.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.962+0000 7f0eef7fe640 1 -- 172.21.15.16:0/56996033 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef0003b70 con 0x7f0ef8100680 2024-03-28T22:08:11.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.962+0000 7f0eef7fe640 1 -- 172.21.15.16:0/56996033 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ef000ee70 con 0x7f0ef8100680 2024-03-28T22:08:11.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.962+0000 7f0efcaae640 1 -- 172.21.15.16:0/56996033 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ef819f930 con 0x7f0ef8100680 2024-03-28T22:08:11.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.962+0000 7f0eef7fe640 1 -- 172.21.15.16:0/56996033 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef0010830 con 0x7f0ef8100680 2024-03-28T22:08:11.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.962+0000 7f0efcaae640 1 -- 172.21.15.16:0/56996033 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ef819fe30 con 0x7f0ef8100680 2024-03-28T22:08:11.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.963+0000 7f0efcaae640 1 -- 172.21.15.16:0/56996033 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ef81a0060 con 0x7f0ef8100680 2024-03-28T22:08:11.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.964+0000 7f0eef7fe640 1 -- 172.21.15.16:0/56996033 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f0ef0004a40 con 0x7f0ef8100680 2024-03-28T22:08:11.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.964+0000 7f0eef7fe640 1 --2- 172.21.15.16:0/56996033 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0ee40a1d80 0x7f0ee40a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:11.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.965+0000 7f0ef5d9b640 1 --2- 172.21.15.16:0/56996033 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0ee40a1d80 0x7f0ee40a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:11.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.965+0000 7f0eef7fe640 1 -- 172.21.15.16:0/56996033 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f0ef00ce080 con 0x7f0ef8100680 2024-03-28T22:08:11.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.965+0000 7f0ef5d9b640 1 --2- 172.21.15.16:0/56996033 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0ee40a1d80 0x7f0ee40a4240 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8198f30 tx=0x7f0ee000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:11.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:11.968+0000 7f0eef7fe640 1 -- 172.21.15.16:0/56996033 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0ef009b950 con 0x7f0ef8100680 2024-03-28T22:08:12.031 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:11 smithi053 ceph-mon[35979]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:12.031 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:11 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:12.031 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:11 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:12.031 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:11 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:12.031 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:11 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:08:12.031 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:11 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:08:12.031 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:11 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:08:12.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:11 smithi063 ceph-mon[36249]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:12.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:11 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:12.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:11 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:12.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:11 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:12.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:11 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:08:12.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:11 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:08:12.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:11 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:08:12.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:11 smithi016 ceph-mon[32374]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:12.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:11 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:12.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:11 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:12.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:11 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:12.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:11 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:08:12.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:11 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:08:12.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:11 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:08:12.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:12.190+0000 7f0efcaae640 1 -- 172.21.15.16:0/56996033 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0ef81a0230 con 0x7f0ef8100680 2024-03-28T22:08:12.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:12.190+0000 7f0eef7fe640 1 -- 172.21.15.16:0/56996033 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6337 (secure 0 0 0) 0x7f0ef009b250 con 0x7f0ef8100680 2024-03-28T22:08:12.192 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:08:12.192 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/247249857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":247249857},{"type":"v1","addr":"172.21.15.63:6835","nonce":247249857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/2324231032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":2324231032},{"type":"v1","addr":"172.21.15.63:6837","nonce":2324231032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/350727634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":350727634},{"type":"v1","addr":"172.21.15.53:6835","nonce":350727634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-28T22:08:12.192 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 8 2024-03-28T22:08:12.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:12.192+0000 7f0efcaae640 1 -- 172.21.15.16:0/56996033 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0ee40a1d80 msgr2=0x7f0ee40a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:12.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:12.192+0000 7f0efcaae640 1 --2- 172.21.15.16:0/56996033 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0ee40a1d80 0x7f0ee40a4240 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8198f30 tx=0x7f0ee000b390 comp rx=0 tx=0).stop 2024-03-28T22:08:12.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:12.192+0000 7f0efcaae640 1 -- 172.21.15.16:0/56996033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ef8100680 msgr2=0x7f0ef8197a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:12.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:12.192+0000 7f0efcaae640 1 --2- 172.21.15.16:0/56996033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ef8100680 0x7f0ef8197a70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0ef000eb70 tx=0x7f0ef000c6a0 comp rx=0 tx=0).stop 2024-03-28T22:08:12.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:12.192+0000 7f0efcaae640 1 -- 172.21.15.16:0/56996033 shutdown_connections 2024-03-28T22:08:12.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:12.192+0000 7f0efcaae640 1 --2- 172.21.15.16:0/56996033 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0ee40a1d80 0x7f0ee40a4240 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:12.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:12.192+0000 7f0efcaae640 1 --2- 172.21.15.16:0/56996033 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ef8102230 0x7f0ef819efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:12.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:12.192+0000 7f0efcaae640 1 --2- 172.21.15.16:0/56996033 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ef8101870 0x7f0ef8197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:12.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:12.192+0000 7f0efcaae640 1 --2- 172.21.15.16:0/56996033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ef8100680 0x7f0ef8197a70 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:12.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:12.192+0000 7f0efcaae640 1 -- 172.21.15.16:0/56996033 >> 172.21.15.16:0/56996033 conn(0x7f0ef80fbc10 msgr2=0x7f0ef80fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:12.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:12.192+0000 7f0efcaae640 1 -- 172.21.15.16:0/56996033 shutdown_connections 2024-03-28T22:08:12.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:12.192+0000 7f0efcaae640 1 -- 172.21.15.16:0/56996033 wait complete. 2024-03-28T22:08:13.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:12 smithi053 ceph-mon[35979]: Deploying daemon mds.h on smithi053 2024-03-28T22:08:13.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:12 smithi053 ceph-mon[35979]: mds.? [v2:172.21.15.63:6836/2324231032,v1:172.21.15.63:6837/2324231032] up:boot 2024-03-28T22:08:13.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:12 smithi053 ceph-mon[35979]: fsmap 7 up:standby 2024-03-28T22:08:13.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:12 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T22:08:13.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:12 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/56996033' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:13.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:12 smithi063 ceph-mon[36249]: Deploying daemon mds.h on smithi053 2024-03-28T22:08:13.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:12 smithi063 ceph-mon[36249]: mds.? [v2:172.21.15.63:6836/2324231032,v1:172.21.15.63:6837/2324231032] up:boot 2024-03-28T22:08:13.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:12 smithi063 ceph-mon[36249]: fsmap 7 up:standby 2024-03-28T22:08:13.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:12 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T22:08:13.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:12 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/56996033' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:13.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:12 smithi016 ceph-mon[32374]: Deploying daemon mds.h on smithi053 2024-03-28T22:08:13.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:12 smithi016 ceph-mon[32374]: mds.? [v2:172.21.15.63:6836/2324231032,v1:172.21.15.63:6837/2324231032] up:boot 2024-03-28T22:08:13.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:12 smithi016 ceph-mon[32374]: fsmap 7 up:standby 2024-03-28T22:08:13.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:12 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T22:08:13.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:12 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/56996033' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:13.547 INFO:journalctl@ceph.mds.h.smithi053.stdout:Mar 28 22:08:13 smithi053 systemd[1]: Starting Ceph mds.h for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:08:13.835 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:13 smithi053 ceph-mon[35979]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:13.835 INFO:journalctl@ceph.mds.h.smithi053.stdout:Mar 28 22:08:13 smithi053 podman[66136]: 2024-03-28 22:08:13.580494892 +0000 UTC m=+0.108787045 container create e38cdb5da5853660cc07004d82e98094409a198744ba5c12a1306f73730960d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-323d23d, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-28T22:08:13.836 INFO:journalctl@ceph.mds.h.smithi053.stdout:Mar 28 22:08:13 smithi053 podman[66136]: 2024-03-28 22:08:13.490337619 +0000 UTC m=+0.018629780 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:08:13.836 INFO:journalctl@ceph.mds.h.smithi053.stdout:Mar 28 22:08:13 smithi053 podman[66136]: 2024-03-28 22:08:13.80808017 +0000 UTC m=+0.336372331 container init e38cdb5da5853660cc07004d82e98094409a198744ba5c12a1306f73730960d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-h, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-323d23d, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-28T22:08:13.836 INFO:journalctl@ceph.mds.h.smithi053.stdout:Mar 28 22:08:13 smithi053 podman[66136]: 2024-03-28 22:08:13.811026606 +0000 UTC m=+0.339318759 container start e38cdb5da5853660cc07004d82e98094409a198744ba5c12a1306f73730960d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-h, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=squid-323d23d, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-28T22:08:14.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:13 smithi063 ceph-mon[36249]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:14.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:13 smithi016 ceph-mon[32374]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:14.206 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:08:14.297 INFO:journalctl@ceph.mds.h.smithi053.stdout:Mar 28 22:08:13 smithi053 ceph-mds[66155]: -- 172.21.15.53:0/485317226 <== mon.2 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e97a381a40 con 0x55e97a354c00 2024-03-28T22:08:14.297 INFO:journalctl@ceph.mds.h.smithi053.stdout:Mar 28 22:08:13 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-h[66151]: starting mds.h at 2024-03-28T22:08:14.298 INFO:journalctl@ceph.mds.h.smithi053.stdout:Mar 28 22:08:13 smithi053 bash[66136]: e38cdb5da5853660cc07004d82e98094409a198744ba5c12a1306f73730960d5 2024-03-28T22:08:14.298 INFO:journalctl@ceph.mds.h.smithi053.stdout:Mar 28 22:08:13 smithi053 systemd[1]: Started Ceph mds.h for 798d9884-ed4e-11ee-95d0-87774f69a715. 2024-03-28T22:08:14.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.318+0000 7f66dec0e640 1 -- 172.21.15.16:0/82021007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66d8101780 msgr2=0x7f66d8101ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:14.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.318+0000 7f66dec0e640 1 --2- 172.21.15.16:0/82021007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66d8101780 0x7f66d8101ba0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f66cc009900 tx=0x7f66cc02bc30 comp rx=0 tx=0).stop 2024-03-28T22:08:14.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.318+0000 7f66dec0e640 1 -- 172.21.15.16:0/82021007 shutdown_connections 2024-03-28T22:08:14.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.318+0000 7f66dec0e640 1 --2- 172.21.15.16:0/82021007 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66d8103330 0x7f66d8109bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:14.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.318+0000 7f66dec0e640 1 --2- 172.21.15.16:0/82021007 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66d8102970 0x7f66d8102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:14.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.318+0000 7f66dec0e640 1 --2- 172.21.15.16:0/82021007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66d8101780 0x7f66d8101ba0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:14.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.318+0000 7f66dec0e640 1 -- 172.21.15.16:0/82021007 >> 172.21.15.16:0/82021007 conn(0x7f66d80fcd30 msgr2=0x7f66d80ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:14.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.318+0000 7f66dec0e640 1 -- 172.21.15.16:0/82021007 shutdown_connections 2024-03-28T22:08:14.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.318+0000 7f66dec0e640 1 -- 172.21.15.16:0/82021007 wait complete. 2024-03-28T22:08:14.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.319+0000 7f66dec0e640 1 Processor -- start 2024-03-28T22:08:14.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.319+0000 7f66dec0e640 1 -- start start 2024-03-28T22:08:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.319+0000 7f66dec0e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66d8101780 0x7f66d8197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.319+0000 7f66dec0e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66d8102970 0x7f66d8197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.319+0000 7f66dec0e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66d8103330 0x7f66d819f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.319+0000 7f66dec0e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f66d810d400 con 0x7f66d8102970 2024-03-28T22:08:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.319+0000 7f66dec0e640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f66d810d280 con 0x7f66d8103330 2024-03-28T22:08:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.319+0000 7f66dec0e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f66d810d580 con 0x7f66d8101780 2024-03-28T22:08:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.319+0000 7f66d7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66d8102970 0x7f66d8197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.319+0000 7f66d7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66d8102970 0x7f66d8197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48910/0 (socket says 172.21.15.16:48910) 2024-03-28T22:08:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.319+0000 7f66d7fff640 1 -- 172.21.15.16:0/523169367 learned_addr learned my addr 172.21.15.16:0/523169367 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.319+0000 7f66dc9aa640 1 --2- 172.21.15.16:0/523169367 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66d8101780 0x7f66d8197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.319+0000 7f66dd1ab640 1 --2- 172.21.15.16:0/523169367 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66d8103330 0x7f66d819f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.319+0000 7f66d7fff640 1 -- 172.21.15.16:0/523169367 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66d8101780 msgr2=0x7f66d8197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.319+0000 7f66d7fff640 1 --2- 172.21.15.16:0/523169367 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66d8101780 0x7f66d8197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.319+0000 7f66d7fff640 1 -- 172.21.15.16:0/523169367 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66d8103330 msgr2=0x7f66d819f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.319+0000 7f66d7fff640 1 --2- 172.21.15.16:0/523169367 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66d8103330 0x7f66d819f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.319+0000 7f66d7fff640 1 -- 172.21.15.16:0/523169367 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66d819f6c0 con 0x7f66d8102970 2024-03-28T22:08:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.319+0000 7f66d7fff640 1 --2- 172.21.15.16:0/523169367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66d8102970 0x7f66d8197fe0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f66c400e9f0 tx=0x7f66c400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.320+0000 7f66d5ffb640 1 -- 172.21.15.16:0/523169367 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c400cd80 con 0x7f66d8102970 2024-03-28T22:08:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.320+0000 7f66dec0e640 1 -- 172.21.15.16:0/523169367 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66d819f9b0 con 0x7f66d8102970 2024-03-28T22:08:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.320+0000 7f66d5ffb640 1 -- 172.21.15.16:0/523169367 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66c4010910 con 0x7f66d8102970 2024-03-28T22:08:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.320+0000 7f66d5ffb640 1 -- 172.21.15.16:0/523169367 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c40188f0 con 0x7f66d8102970 2024-03-28T22:08:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.320+0000 7f66dec0e640 1 -- 172.21.15.16:0/523169367 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66d819ff50 con 0x7f66d8102970 2024-03-28T22:08:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.321+0000 7f66dec0e640 1 -- 172.21.15.16:0/523169367 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66d8108b20 con 0x7f66d8102970 2024-03-28T22:08:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.322+0000 7f66d5ffb640 1 -- 172.21.15.16:0/523169367 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f66c4010ac0 con 0x7f66d8102970 2024-03-28T22:08:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.323+0000 7f66d5ffb640 1 --2- 172.21.15.16:0/523169367 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f66c80a1d80 0x7f66c80a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.323+0000 7f66dc9aa640 1 --2- 172.21.15.16:0/523169367 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f66c80a1d80 0x7f66c80a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.323+0000 7f66d5ffb640 1 -- 172.21.15.16:0/523169367 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f66c40cddc0 con 0x7f66d8102970 2024-03-28T22:08:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.323+0000 7f66dc9aa640 1 --2- 172.21.15.16:0/523169367 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f66c80a1d80 0x7f66c80a4240 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f66cc000c00 tx=0x7f66cc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:14.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.327+0000 7f66d5ffb640 1 -- 172.21.15.16:0/523169367 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f66c409b5f0 con 0x7f66d8102970 2024-03-28T22:08:14.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.550+0000 7f66dec0e640 1 -- 172.21.15.16:0/523169367 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f66d802d170 con 0x7f66d8102970 2024-03-28T22:08:14.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.551+0000 7f66d5ffb640 1 -- 172.21.15.16:0/523169367 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6337 (secure 0 0 0) 0x7f66c409aef0 con 0x7f66d8102970 2024-03-28T22:08:14.554 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:08:14.554 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/247249857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":247249857},{"type":"v1","addr":"172.21.15.63:6835","nonce":247249857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/2324231032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":2324231032},{"type":"v1","addr":"172.21.15.63:6837","nonce":2324231032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/350727634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":350727634},{"type":"v1","addr":"172.21.15.53:6835","nonce":350727634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-28T22:08:14.554 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 8 2024-03-28T22:08:14.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.553+0000 7f66dec0e640 1 -- 172.21.15.16:0/523169367 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f66c80a1d80 msgr2=0x7f66c80a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:14.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.553+0000 7f66dec0e640 1 --2- 172.21.15.16:0/523169367 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f66c80a1d80 0x7f66c80a4240 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f66cc000c00 tx=0x7f66cc005fd0 comp rx=0 tx=0).stop 2024-03-28T22:08:14.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.553+0000 7f66dec0e640 1 -- 172.21.15.16:0/523169367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66d8102970 msgr2=0x7f66d8197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:14.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.553+0000 7f66dec0e640 1 --2- 172.21.15.16:0/523169367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66d8102970 0x7f66d8197fe0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f66c400e9f0 tx=0x7f66c400eeb0 comp rx=0 tx=0).stop 2024-03-28T22:08:14.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.553+0000 7f66dec0e640 1 -- 172.21.15.16:0/523169367 shutdown_connections 2024-03-28T22:08:14.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.553+0000 7f66dec0e640 1 --2- 172.21.15.16:0/523169367 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f66c80a1d80 0x7f66c80a4240 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:14.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.553+0000 7f66dec0e640 1 --2- 172.21.15.16:0/523169367 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66d8103330 0x7f66d819f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:14.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.553+0000 7f66dec0e640 1 --2- 172.21.15.16:0/523169367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66d8102970 0x7f66d8197fe0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:14.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.553+0000 7f66dec0e640 1 --2- 172.21.15.16:0/523169367 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66d8101780 0x7f66d8197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:14.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.553+0000 7f66dec0e640 1 -- 172.21.15.16:0/523169367 >> 172.21.15.16:0/523169367 conn(0x7f66d80fcd30 msgr2=0x7f66d8105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:14.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.553+0000 7f66dec0e640 1 -- 172.21.15.16:0/523169367 shutdown_connections 2024-03-28T22:08:14.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:14.553+0000 7f66dec0e640 1 -- 172.21.15.16:0/523169367 wait complete. 2024-03-28T22:08:15.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:14 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:15.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:14 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:15.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:14 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:15.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:14 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:08:15.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:14 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:08:15.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:14 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:08:15.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:14 smithi053 ceph-mon[35979]: Deploying daemon mds.i on smithi063 2024-03-28T22:08:15.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:14 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/523169367' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:15.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:14 smithi053 ceph-mon[35979]: mds.? [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] up:boot 2024-03-28T22:08:15.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:14 smithi053 ceph-mon[35979]: fsmap 8 up:standby 2024-03-28T22:08:15.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:14 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T22:08:15.298 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:14 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:15.299 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:14 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:15.299 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:14 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:15.299 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:14 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:08:15.299 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:14 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:08:15.299 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:14 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:08:15.299 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:14 smithi063 ceph-mon[36249]: Deploying daemon mds.i on smithi063 2024-03-28T22:08:15.299 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:14 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/523169367' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:15.299 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:14 smithi063 ceph-mon[36249]: mds.? [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] up:boot 2024-03-28T22:08:15.299 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:14 smithi063 ceph-mon[36249]: fsmap 8 up:standby 2024-03-28T22:08:15.299 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:14 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T22:08:15.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:14 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:15.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:14 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:15.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:14 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:15.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:14 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:08:15.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:14 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:08:15.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:14 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:08:15.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:14 smithi016 ceph-mon[32374]: Deploying daemon mds.i on smithi063 2024-03-28T22:08:15.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:14 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/523169367' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:15.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:14 smithi016 ceph-mon[32374]: mds.? [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] up:boot 2024-03-28T22:08:15.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:14 smithi016 ceph-mon[32374]: fsmap 8 up:standby 2024-03-28T22:08:15.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:14 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T22:08:16.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:15 smithi053 ceph-mon[35979]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:16.412 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:15 smithi063 ceph-mon[36249]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:16.412 INFO:journalctl@ceph.mds.i.smithi063.stdout:Mar 28 22:08:16 smithi063 systemd[1]: Starting Ceph mds.i for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:08:16.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:15 smithi016 ceph-mon[32374]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:16.568 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:08:16.682 INFO:journalctl@ceph.mds.i.smithi063.stdout:Mar 28 22:08:16 smithi063 podman[67164]: 2024-03-28 22:08:16.313278564 +0000 UTC m=+0.018193354 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:08:16.682 INFO:journalctl@ceph.mds.i.smithi063.stdout:Mar 28 22:08:16 smithi063 podman[67164]: 2024-03-28 22:08:16.416603057 +0000 UTC m=+0.121517848 container create 4092192cc725d3ddd0ae54b343e545724479d9337e701c096ba10bf06b99086e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-i, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, RELEASE=squid-323d23d, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-28T22:08:16.682 INFO:journalctl@ceph.mds.i.smithi063.stdout:Mar 28 22:08:16 smithi063 podman[67164]: 2024-03-28 22:08:16.626977265 +0000 UTC m=+0.331892045 container init 4092192cc725d3ddd0ae54b343e545724479d9337e701c096ba10bf06b99086e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-i, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_CLEAN=True, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-323d23d) 2024-03-28T22:08:16.682 INFO:journalctl@ceph.mds.i.smithi063.stdout:Mar 28 22:08:16 smithi063 podman[67164]: 2024-03-28 22:08:16.62992902 +0000 UTC m=+0.334843797 container start 4092192cc725d3ddd0ae54b343e545724479d9337e701c096ba10bf06b99086e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-i, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, org.label-schema.build-date=20240326, RELEASE=squid-323d23d, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-28T22:08:16.682 INFO:journalctl@ceph.mds.i.smithi063.stdout:Mar 28 22:08:16 smithi063 ceph-mds[67183]: -- 172.21.15.63:0/2003326257 <== mon.1 v2:172.21.15.63:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d9df8a1a40 con 0x55d9df874c00 2024-03-28T22:08:16.682 INFO:journalctl@ceph.mds.i.smithi063.stdout:Mar 28 22:08:16 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-i[67179]: starting mds.i at 2024-03-28T22:08:16.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.680+0000 7f535d8c0640 1 -- 172.21.15.16:0/906091149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5358101870 msgr2=0x7f5358101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:16.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.680+0000 7f535d8c0640 1 --2- 172.21.15.16:0/906091149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5358101870 0x7f5358101cf0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5340009900 tx=0x7f534002bc30 comp rx=0 tx=0).stop 2024-03-28T22:08:16.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.680+0000 7f535d8c0640 1 -- 172.21.15.16:0/906091149 shutdown_connections 2024-03-28T22:08:16.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.680+0000 7f535d8c0640 1 --2- 172.21.15.16:0/906091149 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5358102230 0x7f5358108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:16.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.680+0000 7f535d8c0640 1 --2- 172.21.15.16:0/906091149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5358101870 0x7f5358101cf0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:16.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.680+0000 7f535d8c0640 1 --2- 172.21.15.16:0/906091149 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5358100680 0x7f5358100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:16.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.680+0000 7f535d8c0640 1 -- 172.21.15.16:0/906091149 >> 172.21.15.16:0/906091149 conn(0x7f53580fbc10 msgr2=0x7f53580fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:16.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.680+0000 7f535d8c0640 1 -- 172.21.15.16:0/906091149 shutdown_connections 2024-03-28T22:08:16.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.680+0000 7f535d8c0640 1 -- 172.21.15.16:0/906091149 wait complete. 2024-03-28T22:08:16.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.680+0000 7f535d8c0640 1 Processor -- start 2024-03-28T22:08:16.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.680+0000 7f535d8c0640 1 -- start start 2024-03-28T22:08:16.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.681+0000 7f535d8c0640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5358100680 0x7f53581046e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:16.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.681+0000 7f535d8c0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5358101870 0x7f5358102d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:16.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.681+0000 7f535d8c0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5358102230 0x7f5358103260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:16.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.681+0000 7f535d8c0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f535810c300 con 0x7f5358102230 2024-03-28T22:08:16.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.681+0000 7f535d8c0640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f535810c180 con 0x7f5358100680 2024-03-28T22:08:16.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.681+0000 7f535d8c0640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f535810c480 con 0x7f5358101870 2024-03-28T22:08:16.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.681+0000 7f53577fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5358102230 0x7f5358103260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:16.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.681+0000 7f53577fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5358102230 0x7f5358103260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48918/0 (socket says 172.21.15.16:48918) 2024-03-28T22:08:16.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.681+0000 7f53577fe640 1 -- 172.21.15.16:0/524271691 learned_addr learned my addr 172.21.15.16:0/524271691 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:16.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.681+0000 7f534e7fc640 1 --2- 172.21.15.16:0/524271691 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5358101870 0x7f5358102d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:16.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.681+0000 7f5356ffd640 1 --2- 172.21.15.16:0/524271691 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5358100680 0x7f53581046e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:16.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.681+0000 7f53577fe640 1 -- 172.21.15.16:0/524271691 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5358101870 msgr2=0x7f5358102d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:16.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.681+0000 7f53577fe640 1 --2- 172.21.15.16:0/524271691 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5358101870 0x7f5358102d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:16.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.681+0000 7f53577fe640 1 -- 172.21.15.16:0/524271691 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5358100680 msgr2=0x7f53581046e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:16.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.681+0000 7f53577fe640 1 --2- 172.21.15.16:0/524271691 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5358100680 0x7f53581046e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:16.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.681+0000 7f53577fe640 1 -- 172.21.15.16:0/524271691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5358103af0 con 0x7f5358102230 2024-03-28T22:08:16.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.681+0000 7f5356ffd640 1 --2- 172.21.15.16:0/524271691 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5358100680 0x7f53581046e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:08:16.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.681+0000 7f53577fe640 1 --2- 172.21.15.16:0/524271691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5358102230 0x7f5358103260 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f535000eaa0 tx=0x7f535000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:16.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.681+0000 7f535c8be640 1 -- 172.21.15.16:0/524271691 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f535000cdb0 con 0x7f5358102230 2024-03-28T22:08:16.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.681+0000 7f535c8be640 1 -- 172.21.15.16:0/524271691 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5350005020 con 0x7f5358102230 2024-03-28T22:08:16.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.681+0000 7f535d8c0640 1 -- 172.21.15.16:0/524271691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53581a1bc0 con 0x7f5358102230 2024-03-28T22:08:16.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.682+0000 7f535c8be640 1 -- 172.21.15.16:0/524271691 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53500116b0 con 0x7f5358102230 2024-03-28T22:08:16.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.682+0000 7f535d8c0640 1 -- 172.21.15.16:0/524271691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53581a20e0 con 0x7f5358102230 2024-03-28T22:08:16.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.683+0000 7f535d8c0640 1 -- 172.21.15.16:0/524271691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5358068a20 con 0x7f5358102230 2024-03-28T22:08:16.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.684+0000 7f535c8be640 1 -- 172.21.15.16:0/524271691 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f5350005560 con 0x7f5358102230 2024-03-28T22:08:16.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.684+0000 7f535c8be640 1 --2- 172.21.15.16:0/524271691 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f53440a1d80 0x7f53440a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:16.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.684+0000 7f5356ffd640 1 --2- 172.21.15.16:0/524271691 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f53440a1d80 0x7f53440a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:16.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.684+0000 7f535c8be640 1 -- 172.21.15.16:0/524271691 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f53500cdb90 con 0x7f5358102230 2024-03-28T22:08:16.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.685+0000 7f5356ffd640 1 --2- 172.21.15.16:0/524271691 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f53440a1d80 0x7f53440a4240 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f53581016d0 tx=0x7f534800a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:16.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.689+0000 7f535c8be640 1 -- 172.21.15.16:0/524271691 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5350016030 con 0x7f5358102230 2024-03-28T22:08:16.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.913+0000 7f535d8c0640 1 -- 172.21.15.16:0/524271691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f53581a2310 con 0x7f5358102230 2024-03-28T22:08:16.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.913+0000 7f535c8be640 1 -- 172.21.15.16:0/524271691 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7159 (secure 0 0 0) 0x7f535009b460 con 0x7f5358102230 2024-03-28T22:08:16.916 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:08:16.916 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/247249857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":247249857},{"type":"v1","addr":"172.21.15.63:6835","nonce":247249857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/2324231032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":2324231032},{"type":"v1","addr":"172.21.15.63:6837","nonce":2324231032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/350727634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":350727634},{"type":"v1","addr":"172.21.15.53:6835","nonce":350727634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/349877573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":349877573},{"type":"v1","addr":"172.21.15.53:6837","nonce":349877573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-03-28T22:08:16.916 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 9 2024-03-28T22:08:16.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.915+0000 7f535d8c0640 1 -- 172.21.15.16:0/524271691 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f53440a1d80 msgr2=0x7f53440a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:16.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.915+0000 7f535d8c0640 1 --2- 172.21.15.16:0/524271691 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f53440a1d80 0x7f53440a4240 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f53581016d0 tx=0x7f534800a560 comp rx=0 tx=0).stop 2024-03-28T22:08:16.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.915+0000 7f535d8c0640 1 -- 172.21.15.16:0/524271691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5358102230 msgr2=0x7f5358103260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:16.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.915+0000 7f535d8c0640 1 --2- 172.21.15.16:0/524271691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5358102230 0x7f5358103260 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f535000eaa0 tx=0x7f535000ef60 comp rx=0 tx=0).stop 2024-03-28T22:08:16.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.915+0000 7f535d8c0640 1 -- 172.21.15.16:0/524271691 shutdown_connections 2024-03-28T22:08:16.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.915+0000 7f535d8c0640 1 --2- 172.21.15.16:0/524271691 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f53440a1d80 0x7f53440a4240 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:16.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.915+0000 7f535d8c0640 1 --2- 172.21.15.16:0/524271691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5358102230 0x7f5358103260 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:16.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.915+0000 7f535d8c0640 1 --2- 172.21.15.16:0/524271691 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5358101870 0x7f5358102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:16.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.915+0000 7f535d8c0640 1 --2- 172.21.15.16:0/524271691 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5358100680 0x7f53581046e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:16.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.915+0000 7f535d8c0640 1 -- 172.21.15.16:0/524271691 >> 172.21.15.16:0/524271691 conn(0x7f53580fbc10 msgr2=0x7f5358069e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:16.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.916+0000 7f535d8c0640 1 -- 172.21.15.16:0/524271691 shutdown_connections 2024-03-28T22:08:16.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:16.916+0000 7f535d8c0640 1 -- 172.21.15.16:0/524271691 wait complete. 2024-03-28T22:08:17.182 INFO:journalctl@ceph.mds.i.smithi063.stdout:Mar 28 22:08:16 smithi063 bash[67164]: 4092192cc725d3ddd0ae54b343e545724479d9337e701c096ba10bf06b99086e 2024-03-28T22:08:17.182 INFO:journalctl@ceph.mds.i.smithi063.stdout:Mar 28 22:08:16 smithi063 systemd[1]: Started Ceph mds.i for 798d9884-ed4e-11ee-95d0-87774f69a715. 2024-03-28T22:08:18.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:17 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:18.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:17 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:18.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:17 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:18.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:17 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:08:18.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:17 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:08:18.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:17 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:08:18.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:17 smithi053 ceph-mon[35979]: Deploying daemon mds.g on smithi016 2024-03-28T22:08:18.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:17 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/524271691' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:18.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:17 smithi053 ceph-mon[35979]: mds.? [v2:172.21.15.63:6838/1177197077,v1:172.21.15.63:6839/1177197077] up:boot 2024-03-28T22:08:18.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:17 smithi053 ceph-mon[35979]: fsmap 9 up:standby 2024-03-28T22:08:18.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:17 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T22:08:18.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:17 smithi053 ceph-mon[35979]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:18.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:17 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:18.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:17 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:18.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:17 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:18.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:17 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:08:18.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:17 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:08:18.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:17 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:08:18.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:17 smithi063 ceph-mon[36249]: Deploying daemon mds.g on smithi016 2024-03-28T22:08:18.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:17 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/524271691' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:18.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:17 smithi063 ceph-mon[36249]: mds.? [v2:172.21.15.63:6838/1177197077,v1:172.21.15.63:6839/1177197077] up:boot 2024-03-28T22:08:18.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:17 smithi063 ceph-mon[36249]: fsmap 9 up:standby 2024-03-28T22:08:18.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:17 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T22:08:18.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:17 smithi063 ceph-mon[36249]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:18.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:17 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:18.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:17 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:18.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:17 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:18.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:17 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:08:18.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:17 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:08:18.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:17 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:08:18.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:17 smithi016 ceph-mon[32374]: Deploying daemon mds.g on smithi016 2024-03-28T22:08:18.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:17 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/524271691' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:18.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:17 smithi016 ceph-mon[32374]: mds.? [v2:172.21.15.63:6838/1177197077,v1:172.21.15.63:6839/1177197077] up:boot 2024-03-28T22:08:18.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:17 smithi016 ceph-mon[32374]: fsmap 9 up:standby 2024-03-28T22:08:18.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:17 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T22:08:18.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:17 smithi016 ceph-mon[32374]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:18.930 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:08:19.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.050+0000 7f4e151b5640 1 -- 172.21.15.16:0/3328537282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e10104be0 msgr2=0x7f4e10106fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:19.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.050+0000 7f4e151b5640 1 --2- 172.21.15.16:0/3328537282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e10104be0 0x7f4e10106fd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4df8009900 tx=0x7f4df802bc30 comp rx=0 tx=0).stop 2024-03-28T22:08:19.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.050+0000 7f4e151b5640 1 -- 172.21.15.16:0/3328537282 shutdown_connections 2024-03-28T22:08:19.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.050+0000 7f4e151b5640 1 --2- 172.21.15.16:0/3328537282 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e10107510 0x7f4e10109900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:19.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.050+0000 7f4e151b5640 1 --2- 172.21.15.16:0/3328537282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e10104be0 0x7f4e10106fd0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:19.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.050+0000 7f4e151b5640 1 --2- 172.21.15.16:0/3328537282 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4e101022b0 0x7f4e101046a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:19.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.050+0000 7f4e151b5640 1 -- 172.21.15.16:0/3328537282 >> 172.21.15.16:0/3328537282 conn(0x7f4e100fbbe0 msgr2=0x7f4e100fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:19.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.050+0000 7f4e151b5640 1 -- 172.21.15.16:0/3328537282 shutdown_connections 2024-03-28T22:08:19.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.050+0000 7f4e151b5640 1 -- 172.21.15.16:0/3328537282 wait complete. 2024-03-28T22:08:19.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.051+0000 7f4e151b5640 1 Processor -- start 2024-03-28T22:08:19.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.051+0000 7f4e151b5640 1 -- start start 2024-03-28T22:08:19.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.051+0000 7f4e151b5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e101022b0 0x7f4e10197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:19.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.051+0000 7f4e151b5640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4e10104be0 0x7f4e10197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:19.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.051+0000 7f4e151b5640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e10107510 0x7f4e1019f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:19.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.051+0000 7f4e151b5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4e1010d400 con 0x7f4e101022b0 2024-03-28T22:08:19.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.051+0000 7f4e151b5640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4e1010d280 con 0x7f4e10107510 2024-03-28T22:08:19.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.051+0000 7f4e151b5640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4e1010d580 con 0x7f4e10104be0 2024-03-28T22:08:19.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.051+0000 7f4e0ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e101022b0 0x7f4e10197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:19.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.051+0000 7f4e0f59e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e10107510 0x7f4e1019f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:19.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.051+0000 7f4e0ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e101022b0 0x7f4e10197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48926/0 (socket says 172.21.15.16:48926) 2024-03-28T22:08:19.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.051+0000 7f4e0ed9d640 1 -- 172.21.15.16:0/3919835548 learned_addr learned my addr 172.21.15.16:0/3919835548 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:19.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.051+0000 7f4e0f59e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e10107510 0x7f4e1019f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:50762/0 (socket says 172.21.15.16:50762) 2024-03-28T22:08:19.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.051+0000 7f4e0e59c640 1 --2- 172.21.15.16:0/3919835548 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4e10104be0 0x7f4e10197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:19.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.052+0000 7f4e0f59e640 1 -- 172.21.15.16:0/3919835548 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4e10104be0 msgr2=0x7f4e10197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:19.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.052+0000 7f4e0f59e640 1 --2- 172.21.15.16:0/3919835548 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4e10104be0 0x7f4e10197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:19.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.052+0000 7f4e0f59e640 1 -- 172.21.15.16:0/3919835548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e101022b0 msgr2=0x7f4e10197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:19.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.052+0000 7f4e0f59e640 1 --2- 172.21.15.16:0/3919835548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e101022b0 0x7f4e10197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:19.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.052+0000 7f4e0f59e640 1 -- 172.21.15.16:0/3919835548 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e1019f690 con 0x7f4e10107510 2024-03-28T22:08:19.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.052+0000 7f4e0e59c640 1 --2- 172.21.15.16:0/3919835548 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4e10104be0 0x7f4e10197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:08:19.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.052+0000 7f4e0f59e640 1 --2- 172.21.15.16:0/3919835548 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e10107510 0x7f4e1019f020 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4e08009fd0 tx=0x7f4e0800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:19.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.052+0000 7f4e07fff640 1 -- 172.21.15.16:0/3919835548 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e0800ce20 con 0x7f4e10107510 2024-03-28T22:08:19.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.052+0000 7f4e151b5640 1 -- 172.21.15.16:0/3919835548 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e1019f980 con 0x7f4e10107510 2024-03-28T22:08:19.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.052+0000 7f4e07fff640 1 -- 172.21.15.16:0/3919835548 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e08005020 con 0x7f4e10107510 2024-03-28T22:08:19.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.052+0000 7f4e151b5640 1 -- 172.21.15.16:0/3919835548 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e1019fea0 con 0x7f4e10107510 2024-03-28T22:08:19.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.052+0000 7f4e07fff640 1 -- 172.21.15.16:0/3919835548 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e08010720 con 0x7f4e10107510 2024-03-28T22:08:19.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.053+0000 7f4e151b5640 1 -- 172.21.15.16:0/3919835548 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e10067d40 con 0x7f4e10107510 2024-03-28T22:08:19.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.055+0000 7f4e07fff640 1 -- 172.21.15.16:0/3919835548 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f4e08005540 con 0x7f4e10107510 2024-03-28T22:08:19.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.055+0000 7f4e07fff640 1 --2- 172.21.15.16:0/3919835548 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4dfc0a1d80 0x7f4dfc0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:19.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.055+0000 7f4e07fff640 1 -- 172.21.15.16:0/3919835548 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f4e080cdf90 con 0x7f4e10107510 2024-03-28T22:08:19.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.055+0000 7f4e0ed9d640 1 --2- 172.21.15.16:0/3919835548 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4dfc0a1d80 0x7f4dfc0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:19.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.056+0000 7f4e0ed9d640 1 --2- 172.21.15.16:0/3919835548 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4dfc0a1d80 0x7f4dfc0a4240 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4e00009cd0 tx=0x7f4e00005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:19.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.059+0000 7f4e07fff640 1 -- 172.21.15.16:0/3919835548 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4e0809b840 con 0x7f4e10107510 2024-03-28T22:08:19.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.289+0000 7f4e151b5640 1 -- 172.21.15.16:0/3919835548 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e100fd6d0 con 0x7f4e10107510 2024-03-28T22:08:19.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.290+0000 7f4e07fff640 1 -- 172.21.15.16:0/3919835548 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+7986 (secure 0 0 0) 0x7f4e0809b140 con 0x7f4e10107510 2024-03-28T22:08:19.293 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:08:19.293 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/247249857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":247249857},{"type":"v1","addr":"172.21.15.63:6835","nonce":247249857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/2324231032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":2324231032},{"type":"v1","addr":"172.21.15.63:6837","nonce":2324231032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24493,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/1177197077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":1177197077},{"type":"v1","addr":"172.21.15.63:6839","nonce":1177197077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/350727634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":350727634},{"type":"v1","addr":"172.21.15.53:6835","nonce":350727634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/349877573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":349877573},{"type":"v1","addr":"172.21.15.53:6837","nonce":349877573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-03-28T22:08:19.294 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 10 2024-03-28T22:08:19.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.293+0000 7f4e151b5640 1 -- 172.21.15.16:0/3919835548 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4dfc0a1d80 msgr2=0x7f4dfc0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:19.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.293+0000 7f4e151b5640 1 --2- 172.21.15.16:0/3919835548 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4dfc0a1d80 0x7f4dfc0a4240 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4e00009cd0 tx=0x7f4e00005f90 comp rx=0 tx=0).stop 2024-03-28T22:08:19.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.293+0000 7f4e151b5640 1 -- 172.21.15.16:0/3919835548 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e10107510 msgr2=0x7f4e1019f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:19.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.293+0000 7f4e151b5640 1 --2- 172.21.15.16:0/3919835548 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e10107510 0x7f4e1019f020 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4e08009fd0 tx=0x7f4e0800ef10 comp rx=0 tx=0).stop 2024-03-28T22:08:19.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.293+0000 7f4e151b5640 1 -- 172.21.15.16:0/3919835548 shutdown_connections 2024-03-28T22:08:19.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.293+0000 7f4e151b5640 1 --2- 172.21.15.16:0/3919835548 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4dfc0a1d80 0x7f4dfc0a4240 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:19.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.293+0000 7f4e151b5640 1 --2- 172.21.15.16:0/3919835548 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e10107510 0x7f4e1019f020 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:19.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.293+0000 7f4e151b5640 1 --2- 172.21.15.16:0/3919835548 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4e10104be0 0x7f4e10197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:19.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.293+0000 7f4e151b5640 1 --2- 172.21.15.16:0/3919835548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e101022b0 0x7f4e10197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:19.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.293+0000 7f4e151b5640 1 -- 172.21.15.16:0/3919835548 >> 172.21.15.16:0/3919835548 conn(0x7f4e100fbbe0 msgr2=0x7f4e10102d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:19.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.293+0000 7f4e151b5640 1 -- 172.21.15.16:0/3919835548 shutdown_connections 2024-03-28T22:08:19.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:19.293+0000 7f4e151b5640 1 -- 172.21.15.16:0/3919835548 wait complete. 2024-03-28T22:08:19.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:19 smithi016 ceph-mon[32374]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:19.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:19 smithi063 ceph-mon[36249]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:19.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:19 smithi053 ceph-mon[35979]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:19.686 INFO:journalctl@ceph.mds.g.smithi016.stdout:Mar 28 22:08:19 smithi016 systemd[1]: Started Ceph mds.g for 798d9884-ed4e-11ee-95d0-87774f69a715. 2024-03-28T22:08:20.428 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:20 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3919835548' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:20.428 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:20 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:20.428 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:20 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:20.428 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:20 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:20.429 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:20 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:08:20.429 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:20 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:08:20.429 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:20 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:08:20.429 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:20 smithi016 ceph-mon[32374]: Deploying daemon mds.a on smithi016 2024-03-28T22:08:20.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:20 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/3919835548' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:20.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:20 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:20.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:20 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:20.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:20 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:20.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:20 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:08:20.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:20 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:08:20.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:20 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:08:20.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:20 smithi063 ceph-mon[36249]: Deploying daemon mds.a on smithi016 2024-03-28T22:08:20.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:20 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3919835548' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:20.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:20 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:20.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:20 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:20.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:20 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:20.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:20 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:08:20.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:20 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:08:20.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:20 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:08:20.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:20 smithi053 ceph-mon[35979]: Deploying daemon mds.a on smithi016 2024-03-28T22:08:21.307 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:08:21.321 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:21 smithi016 ceph-mon[32374]: mds.? [v2:172.21.15.16:6838/1644581227,v1:172.21.15.16:6839/1644581227] up:boot 2024-03-28T22:08:21.322 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:21 smithi016 ceph-mon[32374]: fsmap 10 up:standby 2024-03-28T22:08:21.322 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:21 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T22:08:21.322 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:21 smithi016 ceph-mon[32374]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:21.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:21 smithi063 ceph-mon[36249]: mds.? [v2:172.21.15.16:6838/1644581227,v1:172.21.15.16:6839/1644581227] up:boot 2024-03-28T22:08:21.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:21 smithi063 ceph-mon[36249]: fsmap 10 up:standby 2024-03-28T22:08:21.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:21 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T22:08:21.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:21 smithi063 ceph-mon[36249]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:21.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:21 smithi053 ceph-mon[35979]: mds.? [v2:172.21.15.16:6838/1644581227,v1:172.21.15.16:6839/1644581227] up:boot 2024-03-28T22:08:21.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:21 smithi053 ceph-mon[35979]: fsmap 10 up:standby 2024-03-28T22:08:21.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:21 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T22:08:21.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:21 smithi053 ceph-mon[35979]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:21.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.562+0000 7fe550d3a640 1 -- 172.21.15.16:0/889807119 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe54c102230 msgr2=0x7fe54c108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:21.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.562+0000 7fe550d3a640 1 --2- 172.21.15.16:0/889807119 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe54c102230 0x7fe54c108ac0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe544009ca0 tx=0x7fe54402c0f0 comp rx=0 tx=0).stop 2024-03-28T22:08:21.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.562+0000 7fe550d3a640 1 -- 172.21.15.16:0/889807119 shutdown_connections 2024-03-28T22:08:21.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.562+0000 7fe550d3a640 1 --2- 172.21.15.16:0/889807119 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe54c102230 0x7fe54c108ac0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:21.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.562+0000 7fe550d3a640 1 --2- 172.21.15.16:0/889807119 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe54c101870 0x7fe54c101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:21.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.562+0000 7fe550d3a640 1 --2- 172.21.15.16:0/889807119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe54c100680 0x7fe54c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:21.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.562+0000 7fe550d3a640 1 -- 172.21.15.16:0/889807119 >> 172.21.15.16:0/889807119 conn(0x7fe54c0fbbf0 msgr2=0x7fe54c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:21.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.563+0000 7fe550d3a640 1 -- 172.21.15.16:0/889807119 shutdown_connections 2024-03-28T22:08:21.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.563+0000 7fe550d3a640 1 -- 172.21.15.16:0/889807119 wait complete. 2024-03-28T22:08:21.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.563+0000 7fe550d3a640 1 Processor -- start 2024-03-28T22:08:21.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.563+0000 7fe550d3a640 1 -- start start 2024-03-28T22:08:21.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.564+0000 7fe550d3a640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe54c100680 0x7fe54c0695b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:21.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.564+0000 7fe550d3a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe54c101870 0x7fe54c19c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:21.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.564+0000 7fe550d3a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe54c102230 0x7fe54c19ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:21.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.564+0000 7fe550d3a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe54c10c300 con 0x7fe54c102230 2024-03-28T22:08:21.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.564+0000 7fe550d3a640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe54c10c180 con 0x7fe54c100680 2024-03-28T22:08:21.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.564+0000 7fe550d3a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe54c10c480 con 0x7fe54c101870 2024-03-28T22:08:21.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.564+0000 7fe54ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe54c102230 0x7fe54c19ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:21.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.564+0000 7fe54ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe54c102230 0x7fe54c19ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53158/0 (socket says 172.21.15.16:53158) 2024-03-28T22:08:21.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.564+0000 7fe54ad9d640 1 -- 172.21.15.16:0/2336084145 learned_addr learned my addr 172.21.15.16:0/2336084145 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:21.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.564+0000 7fe541d9b640 1 --2- 172.21.15.16:0/2336084145 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe54c101870 0x7fe54c19c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:21.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.564+0000 7fe54a59c640 1 --2- 172.21.15.16:0/2336084145 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe54c100680 0x7fe54c0695b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:21.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.564+0000 7fe54ad9d640 1 -- 172.21.15.16:0/2336084145 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe54c101870 msgr2=0x7fe54c19c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:21.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.564+0000 7fe54ad9d640 1 --2- 172.21.15.16:0/2336084145 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe54c101870 0x7fe54c19c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:21.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.564+0000 7fe54ad9d640 1 -- 172.21.15.16:0/2336084145 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe54c100680 msgr2=0x7fe54c0695b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:21.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.564+0000 7fe54ad9d640 1 --2- 172.21.15.16:0/2336084145 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe54c100680 0x7fe54c0695b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:21.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.564+0000 7fe54ad9d640 1 -- 172.21.15.16:0/2336084145 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe54c19f640 con 0x7fe54c102230 2024-03-28T22:08:21.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.565+0000 7fe54ad9d640 1 --2- 172.21.15.16:0/2336084145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe54c102230 0x7fe54c19ed80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe544037040 tx=0x7fe544032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:21.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.565+0000 7fe543fff640 1 -- 172.21.15.16:0/2336084145 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe544041020 con 0x7fe54c102230 2024-03-28T22:08:21.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.565+0000 7fe550d3a640 1 -- 172.21.15.16:0/2336084145 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe54c19f8d0 con 0x7fe54c102230 2024-03-28T22:08:21.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.565+0000 7fe543fff640 1 -- 172.21.15.16:0/2336084145 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe544034cd0 con 0x7fe54c102230 2024-03-28T22:08:21.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.565+0000 7fe543fff640 1 -- 172.21.15.16:0/2336084145 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe544039040 con 0x7fe54c102230 2024-03-28T22:08:21.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.565+0000 7fe550d3a640 1 -- 172.21.15.16:0/2336084145 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe54c19fdc0 con 0x7fe54c102230 2024-03-28T22:08:21.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.566+0000 7fe550d3a640 1 -- 172.21.15.16:0/2336084145 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe54c1079a0 con 0x7fe54c102230 2024-03-28T22:08:21.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.567+0000 7fe543fff640 1 -- 172.21.15.16:0/2336084145 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fe544044020 con 0x7fe54c102230 2024-03-28T22:08:21.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.568+0000 7fe543fff640 1 --2- 172.21.15.16:0/2336084145 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fe5380a1d80 0x7fe5380a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:21.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.568+0000 7fe543fff640 1 -- 172.21.15.16:0/2336084145 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fe5440ef760 con 0x7fe54c102230 2024-03-28T22:08:21.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.568+0000 7fe54a59c640 1 --2- 172.21.15.16:0/2336084145 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fe5380a1d80 0x7fe5380a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:21.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.569+0000 7fe54a59c640 1 --2- 172.21.15.16:0/2336084145 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fe5380a1d80 0x7fe5380a4240 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe54c1016d0 tx=0x7fe53c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:21.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.573+0000 7fe543fff640 1 -- 172.21.15.16:0/2336084145 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe54404b020 con 0x7fe54c102230 2024-03-28T22:08:21.686 INFO:journalctl@ceph.mds.a.smithi016.stdout:Mar 28 22:08:21 smithi016 systemd[1]: Starting Ceph mds.a for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:08:21.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.808+0000 7fe550d3a640 1 -- 172.21.15.16:0/2336084145 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe54c062350 con 0x7fe54c102230 2024-03-28T22:08:21.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.808+0000 7fe543fff640 1 -- 172.21.15.16:0/2336084145 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8812 (secure 0 0 0) 0x7fe5440bd0c0 con 0x7fe54c102230 2024-03-28T22:08:21.811 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:08:21.811 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/1644581227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1644581227},{"type":"v1","addr":"172.21.15.16:6839","nonce":1644581227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/247249857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":247249857},{"type":"v1","addr":"172.21.15.63:6835","nonce":247249857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/2324231032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":2324231032},{"type":"v1","addr":"172.21.15.63:6837","nonce":2324231032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24493,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/1177197077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":1177197077},{"type":"v1","addr":"172.21.15.63:6839","nonce":1177197077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/350727634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":350727634},{"type":"v1","addr":"172.21.15.53:6835","nonce":350727634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/349877573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":349877573},{"type":"v1","addr":"172.21.15.53:6837","nonce":349877573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-03-28T22:08:21.811 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 11 2024-03-28T22:08:21.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.810+0000 7fe550d3a640 1 -- 172.21.15.16:0/2336084145 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fe5380a1d80 msgr2=0x7fe5380a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:21.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.810+0000 7fe550d3a640 1 --2- 172.21.15.16:0/2336084145 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fe5380a1d80 0x7fe5380a4240 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe54c1016d0 tx=0x7fe53c005f90 comp rx=0 tx=0).stop 2024-03-28T22:08:21.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.810+0000 7fe550d3a640 1 -- 172.21.15.16:0/2336084145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe54c102230 msgr2=0x7fe54c19ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:21.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.810+0000 7fe550d3a640 1 --2- 172.21.15.16:0/2336084145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe54c102230 0x7fe54c19ed80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe544037040 tx=0x7fe544032f40 comp rx=0 tx=0).stop 2024-03-28T22:08:21.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.810+0000 7fe550d3a640 1 -- 172.21.15.16:0/2336084145 shutdown_connections 2024-03-28T22:08:21.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.810+0000 7fe550d3a640 1 --2- 172.21.15.16:0/2336084145 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fe5380a1d80 0x7fe5380a4240 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:21.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.810+0000 7fe550d3a640 1 --2- 172.21.15.16:0/2336084145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe54c102230 0x7fe54c19ed80 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:21.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.810+0000 7fe550d3a640 1 --2- 172.21.15.16:0/2336084145 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe54c101870 0x7fe54c19c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:21.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.810+0000 7fe550d3a640 1 --2- 172.21.15.16:0/2336084145 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe54c100680 0x7fe54c0695b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:21.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.810+0000 7fe550d3a640 1 -- 172.21.15.16:0/2336084145 >> 172.21.15.16:0/2336084145 conn(0x7fe54c0fbbf0 msgr2=0x7fe54c104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:21.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.810+0000 7fe550d3a640 1 -- 172.21.15.16:0/2336084145 shutdown_connections 2024-03-28T22:08:21.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:21.810+0000 7fe550d3a640 1 -- 172.21.15.16:0/2336084145 wait complete. 2024-03-28T22:08:21.995 INFO:journalctl@ceph.mds.a.smithi016.stdout:Mar 28 22:08:21 smithi016 podman[72060]: 2024-03-28 22:08:21.624117153 +0000 UTC m=+0.017985878 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:08:21.995 INFO:journalctl@ceph.mds.a.smithi016.stdout:Mar 28 22:08:21 smithi016 podman[72060]: 2024-03-28 22:08:21.749255941 +0000 UTC m=+0.143124641 container create 6cf1ec98898ed670c6cf70c2fd5120bd85450d6888d22a3d301ef74b2dd211b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-323d23d, GIT_BRANCH=HEAD) 2024-03-28T22:08:21.995 INFO:journalctl@ceph.mds.a.smithi016.stdout:Mar 28 22:08:21 smithi016 podman[72060]: 2024-03-28 22:08:21.968194342 +0000 UTC m=+0.362063049 container init 6cf1ec98898ed670c6cf70c2fd5120bd85450d6888d22a3d301ef74b2dd211b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-a, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-323d23d, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS) 2024-03-28T22:08:21.996 INFO:journalctl@ceph.mds.a.smithi016.stdout:Mar 28 22:08:21 smithi016 podman[72060]: 2024-03-28 22:08:21.97058823 +0000 UTC m=+0.364456928 container start 6cf1ec98898ed670c6cf70c2fd5120bd85450d6888d22a3d301ef74b2dd211b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-a, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-323d23d, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-28T22:08:22.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:22 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2336084145' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:22.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:22 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:22.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:22 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:22.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:22 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:22.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:22 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:08:22.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:22 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:08:22.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:22 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:08:22.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:22 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2336084145' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:22.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:22 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:22.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:22 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:22.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:22 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:22.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:22 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:08:22.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:22 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:08:22.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:22 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:08:22.437 INFO:journalctl@ceph.mds.a.smithi016.stdout:Mar 28 22:08:21 smithi016 ceph-mds[72082]: -- 172.21.15.16:0/1665353460 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a5f15eba40 con 0x55a5f15be800 2024-03-28T22:08:22.437 INFO:journalctl@ceph.mds.a.smithi016.stdout:Mar 28 22:08:21 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-a[72078]: starting mds.a at 2024-03-28T22:08:22.437 INFO:journalctl@ceph.mds.a.smithi016.stdout:Mar 28 22:08:22 smithi016 bash[72060]: 6cf1ec98898ed670c6cf70c2fd5120bd85450d6888d22a3d301ef74b2dd211b8 2024-03-28T22:08:22.437 INFO:journalctl@ceph.mds.a.smithi016.stdout:Mar 28 22:08:22 smithi016 systemd[1]: Started Ceph mds.a for 798d9884-ed4e-11ee-95d0-87774f69a715. 2024-03-28T22:08:22.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:22 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2336084145' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:22.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:22 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:22.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:22 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:22.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:22 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:22.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:22 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:08:22.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:22 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:08:22.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:22 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:08:23.271 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:23 smithi053 ceph-mon[35979]: Deploying daemon mds.b on smithi053 2024-03-28T22:08:23.271 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:23 smithi053 ceph-mon[35979]: mds.? [v2:172.21.15.16:6840/2593062669,v1:172.21.15.16:6841/2593062669] up:boot 2024-03-28T22:08:23.271 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:23 smithi053 ceph-mon[35979]: fsmap 11 up:standby 2024-03-28T22:08:23.271 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:23 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T22:08:23.271 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:23 smithi053 ceph-mon[35979]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:23.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:23 smithi063 ceph-mon[36249]: Deploying daemon mds.b on smithi053 2024-03-28T22:08:23.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:23 smithi063 ceph-mon[36249]: mds.? [v2:172.21.15.16:6840/2593062669,v1:172.21.15.16:6841/2593062669] up:boot 2024-03-28T22:08:23.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:23 smithi063 ceph-mon[36249]: fsmap 11 up:standby 2024-03-28T22:08:23.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:23 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T22:08:23.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:23 smithi063 ceph-mon[36249]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:23.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:23 smithi016 ceph-mon[32374]: Deploying daemon mds.b on smithi053 2024-03-28T22:08:23.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:23 smithi016 ceph-mon[32374]: mds.? [v2:172.21.15.16:6840/2593062669,v1:172.21.15.16:6841/2593062669] up:boot 2024-03-28T22:08:23.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:23 smithi016 ceph-mon[32374]: fsmap 11 up:standby 2024-03-28T22:08:23.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:23 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T22:08:23.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:23 smithi016 ceph-mon[32374]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:23.825 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:08:23.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.938+0000 7fe7b7aa4640 1 -- 172.21.15.16:0/1117353073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7b0104b30 msgr2=0x7fe7b0106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:23.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.938+0000 7fe7b7aa4640 1 --2- 172.21.15.16:0/1117353073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7b0104b30 0x7fe7b0106f20 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe7a0009900 tx=0x7fe7a002bc30 comp rx=0 tx=0).stop 2024-03-28T22:08:23.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.938+0000 7fe7b7aa4640 1 -- 172.21.15.16:0/1117353073 shutdown_connections 2024-03-28T22:08:23.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.938+0000 7fe7b7aa4640 1 --2- 172.21.15.16:0/1117353073 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7b0107460 0x7fe7b01098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:23.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.938+0000 7fe7b7aa4640 1 --2- 172.21.15.16:0/1117353073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7b0104b30 0x7fe7b0106f20 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:23.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.938+0000 7fe7b7aa4640 1 --2- 172.21.15.16:0/1117353073 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe7b0102200 0x7fe7b01045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:23.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.938+0000 7fe7b7aa4640 1 -- 172.21.15.16:0/1117353073 >> 172.21.15.16:0/1117353073 conn(0x7fe7b00fbc10 msgr2=0x7fe7b00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:23.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.938+0000 7fe7b7aa4640 1 -- 172.21.15.16:0/1117353073 shutdown_connections 2024-03-28T22:08:23.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.938+0000 7fe7b7aa4640 1 -- 172.21.15.16:0/1117353073 wait complete. 2024-03-28T22:08:23.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.939+0000 7fe7b7aa4640 1 Processor -- start 2024-03-28T22:08:23.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.939+0000 7fe7b7aa4640 1 -- start start 2024-03-28T22:08:23.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.939+0000 7fe7b7aa4640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7b0102200 0x7fe7b0197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:23.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.939+0000 7fe7b7aa4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe7b0104b30 0x7fe7b0198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:23.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.939+0000 7fe7b7aa4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7b0107460 0x7fe7b019f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:23.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.939+0000 7fe7b7aa4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe7b010d410 con 0x7fe7b0107460 2024-03-28T22:08:23.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.939+0000 7fe7b7aa4640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe7b010d290 con 0x7fe7b0102200 2024-03-28T22:08:23.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.939+0000 7fe7b7aa4640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe7b010d590 con 0x7fe7b0104b30 2024-03-28T22:08:23.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.939+0000 7fe7b6041640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7b0107460 0x7fe7b019f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:23.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.939+0000 7fe7b6041640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7b0107460 0x7fe7b019f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53196/0 (socket says 172.21.15.16:53196) 2024-03-28T22:08:23.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.939+0000 7fe7b6041640 1 -- 172.21.15.16:0/2568698712 learned_addr learned my addr 172.21.15.16:0/2568698712 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:23.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.939+0000 7fe7b5840640 1 --2- 172.21.15.16:0/2568698712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7b0102200 0x7fe7b0197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:23.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.939+0000 7fe7b503f640 1 --2- 172.21.15.16:0/2568698712 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe7b0104b30 0x7fe7b0198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:23.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.939+0000 7fe7b6041640 1 -- 172.21.15.16:0/2568698712 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe7b0104b30 msgr2=0x7fe7b0198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:23.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.940+0000 7fe7b6041640 1 --2- 172.21.15.16:0/2568698712 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe7b0104b30 0x7fe7b0198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:23.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.940+0000 7fe7b6041640 1 -- 172.21.15.16:0/2568698712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7b0102200 msgr2=0x7fe7b0197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:23.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.940+0000 7fe7b6041640 1 --2- 172.21.15.16:0/2568698712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7b0102200 0x7fe7b0197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:23.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.940+0000 7fe7b6041640 1 -- 172.21.15.16:0/2568698712 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7b019f6e0 con 0x7fe7b0107460 2024-03-28T22:08:23.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.940+0000 7fe7b503f640 1 --2- 172.21.15.16:0/2568698712 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe7b0104b30 0x7fe7b0198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:08:23.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.940+0000 7fe7b6041640 1 --2- 172.21.15.16:0/2568698712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7b0107460 0x7fe7b019f070 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe7a800eaa0 tx=0x7fe7a800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:23.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.940+0000 7fe7a6ffd640 1 -- 172.21.15.16:0/2568698712 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a800cdb0 con 0x7fe7b0107460 2024-03-28T22:08:23.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.940+0000 7fe7a6ffd640 1 -- 172.21.15.16:0/2568698712 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7a8005020 con 0x7fe7b0107460 2024-03-28T22:08:23.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.940+0000 7fe7b7aa4640 1 -- 172.21.15.16:0/2568698712 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7b019f9d0 con 0x7fe7b0107460 2024-03-28T22:08:23.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.940+0000 7fe7a6ffd640 1 -- 172.21.15.16:0/2568698712 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a80116b0 con 0x7fe7b0107460 2024-03-28T22:08:23.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.940+0000 7fe7b7aa4640 1 -- 172.21.15.16:0/2568698712 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7b019fea0 con 0x7fe7b0107460 2024-03-28T22:08:23.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.941+0000 7fe7b7aa4640 1 -- 172.21.15.16:0/2568698712 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7b00fd700 con 0x7fe7b0107460 2024-03-28T22:08:23.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.942+0000 7fe7a6ffd640 1 -- 172.21.15.16:0/2568698712 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fe7a8005560 con 0x7fe7b0107460 2024-03-28T22:08:23.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.942+0000 7fe7a6ffd640 1 --2- 172.21.15.16:0/2568698712 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fe79c0a1d80 0x7fe79c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:23.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.943+0000 7fe7b5840640 1 --2- 172.21.15.16:0/2568698712 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fe79c0a1d80 0x7fe79c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:23.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.943+0000 7fe7a6ffd640 1 -- 172.21.15.16:0/2568698712 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fe7a80cdb50 con 0x7fe7b0107460 2024-03-28T22:08:23.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.943+0000 7fe7b5840640 1 --2- 172.21.15.16:0/2568698712 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fe79c0a1d80 0x7fe79c0a4240 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac005fd0 tx=0x7fe7ac00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:23.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:23.947+0000 7fe7a6ffd640 1 -- 172.21.15.16:0/2568698712 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe7a8016030 con 0x7fe7b0107460 2024-03-28T22:08:24.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:24.166+0000 7fe7b7aa4640 1 -- 172.21.15.16:0/2568698712 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe7b0062350 con 0x7fe7b0107460 2024-03-28T22:08:24.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:24.167+0000 7fe7a6ffd640 1 -- 172.21.15.16:0/2568698712 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9638 (secure 0 0 0) 0x7fe7a809b420 con 0x7fe7b0107460 2024-03-28T22:08:24.169 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:08:24.170 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/1644581227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1644581227},{"type":"v1","addr":"172.21.15.16:6839","nonce":1644581227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/2593062669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2593062669},{"type":"v1","addr":"172.21.15.16:6841","nonce":2593062669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/247249857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":247249857},{"type":"v1","addr":"172.21.15.63:6835","nonce":247249857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/2324231032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":2324231032},{"type":"v1","addr":"172.21.15.63:6837","nonce":2324231032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24493,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/1177197077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":1177197077},{"type":"v1","addr":"172.21.15.63:6839","nonce":1177197077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/350727634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":350727634},{"type":"v1","addr":"172.21.15.53:6835","nonce":350727634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/349877573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":349877573},{"type":"v1","addr":"172.21.15.53:6837","nonce":349877573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-03-28T22:08:24.170 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 12 2024-03-28T22:08:24.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:24.169+0000 7fe7b7aa4640 1 -- 172.21.15.16:0/2568698712 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fe79c0a1d80 msgr2=0x7fe79c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:24.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:24.169+0000 7fe7b7aa4640 1 --2- 172.21.15.16:0/2568698712 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fe79c0a1d80 0x7fe79c0a4240 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac005fd0 tx=0x7fe7ac00b390 comp rx=0 tx=0).stop 2024-03-28T22:08:24.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:24.169+0000 7fe7b7aa4640 1 -- 172.21.15.16:0/2568698712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7b0107460 msgr2=0x7fe7b019f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:24.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:24.169+0000 7fe7b7aa4640 1 --2- 172.21.15.16:0/2568698712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7b0107460 0x7fe7b019f070 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe7a800eaa0 tx=0x7fe7a800ef60 comp rx=0 tx=0).stop 2024-03-28T22:08:24.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:24.169+0000 7fe7b7aa4640 1 -- 172.21.15.16:0/2568698712 shutdown_connections 2024-03-28T22:08:24.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:24.169+0000 7fe7b7aa4640 1 --2- 172.21.15.16:0/2568698712 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fe79c0a1d80 0x7fe79c0a4240 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:24.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:24.169+0000 7fe7b7aa4640 1 --2- 172.21.15.16:0/2568698712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7b0107460 0x7fe7b019f070 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:24.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:24.169+0000 7fe7b7aa4640 1 --2- 172.21.15.16:0/2568698712 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe7b0104b30 0x7fe7b0198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:24.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:24.169+0000 7fe7b7aa4640 1 --2- 172.21.15.16:0/2568698712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7b0102200 0x7fe7b0197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:24.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:24.169+0000 7fe7b7aa4640 1 -- 172.21.15.16:0/2568698712 >> 172.21.15.16:0/2568698712 conn(0x7fe7b00fbc10 msgr2=0x7fe7b0102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:24.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:24.169+0000 7fe7b7aa4640 1 -- 172.21.15.16:0/2568698712 shutdown_connections 2024-03-28T22:08:24.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:24.169+0000 7fe7b7aa4640 1 -- 172.21.15.16:0/2568698712 wait complete. 2024-03-28T22:08:24.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:24 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2568698712' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:24.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:24 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2568698712' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:24.548 INFO:journalctl@ceph.mds.b.smithi053.stdout:Mar 28 22:08:24 smithi053 systemd[1]: Starting Ceph mds.b for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:08:24.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:24 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2568698712' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:24.887 INFO:journalctl@ceph.mds.b.smithi053.stdout:Mar 28 22:08:24 smithi053 podman[66440]: 2024-03-28 22:08:24.47161325 +0000 UTC m=+0.018317643 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:08:24.887 INFO:journalctl@ceph.mds.b.smithi053.stdout:Mar 28 22:08:24 smithi053 podman[66440]: 2024-03-28 22:08:24.666894937 +0000 UTC m=+0.213599313 container create 19904610fabe9d4a77a857d8b1f474031e557a481a5f34d9395ec9433588a45a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-323d23d, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-28T22:08:24.887 INFO:journalctl@ceph.mds.b.smithi053.stdout:Mar 28 22:08:24 smithi053 podman[66440]: 2024-03-28 22:08:24.885795958 +0000 UTC m=+0.432500337 container init 19904610fabe9d4a77a857d8b1f474031e557a481a5f34d9395ec9433588a45a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-323d23d, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-28T22:08:25.297 INFO:journalctl@ceph.mds.b.smithi053.stdout:Mar 28 22:08:24 smithi053 podman[66440]: 2024-03-28 22:08:24.888701509 +0000 UTC m=+0.435405880 container start 19904610fabe9d4a77a857d8b1f474031e557a481a5f34d9395ec9433588a45a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-323d23d, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-28T22:08:25.298 INFO:journalctl@ceph.mds.b.smithi053.stdout:Mar 28 22:08:24 smithi053 ceph-mds[66459]: -- 172.21.15.53:0/2578821616 <== mon.2 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ea9341ba40 con 0x55ea933ee800 2024-03-28T22:08:25.298 INFO:journalctl@ceph.mds.b.smithi053.stdout:Mar 28 22:08:24 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-b[66455]: starting mds.b at 2024-03-28T22:08:25.298 INFO:journalctl@ceph.mds.b.smithi053.stdout:Mar 28 22:08:24 smithi053 bash[66440]: 19904610fabe9d4a77a857d8b1f474031e557a481a5f34d9395ec9433588a45a 2024-03-28T22:08:25.298 INFO:journalctl@ceph.mds.b.smithi053.stdout:Mar 28 22:08:24 smithi053 systemd[1]: Started Ceph mds.b for 798d9884-ed4e-11ee-95d0-87774f69a715. 2024-03-28T22:08:26.183 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:08:26.272 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:26 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:26.272 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:26 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:26.272 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:26 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:26.272 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:26 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:26.273 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:26 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:26.273 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:26 smithi063 ceph-mon[36249]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:26.273 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:26 smithi063 ceph-mon[36249]: mds.? [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] up:boot 2024-03-28T22:08:26.273 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:26 smithi063 ceph-mon[36249]: fsmap 12 up:standby 2024-03-28T22:08:26.273 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:26 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T22:08:26.273 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:26 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:08:26.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.286+0000 7fd669055640 1 -- 172.21.15.16:0/3882031154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6641022a0 msgr2=0x7fd664108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:26.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.286+0000 7fd669055640 1 --2- 172.21.15.16:0/3882031154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6641022a0 0x7fd664108b30 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd654009ca0 tx=0x7fd65402c0f0 comp rx=0 tx=0).stop 2024-03-28T22:08:26.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.286+0000 7fd669055640 1 -- 172.21.15.16:0/3882031154 shutdown_connections 2024-03-28T22:08:26.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.286+0000 7fd669055640 1 --2- 172.21.15.16:0/3882031154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6641022a0 0x7fd664108b30 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:26.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.286+0000 7fd669055640 1 --2- 172.21.15.16:0/3882031154 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd6641018e0 0x7fd664101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:26.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.286+0000 7fd669055640 1 --2- 172.21.15.16:0/3882031154 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6641006f0 0x7fd664100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:26.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.286+0000 7fd669055640 1 -- 172.21.15.16:0/3882031154 >> 172.21.15.16:0/3882031154 conn(0x7fd6640fbc80 msgr2=0x7fd6640fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:26.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.286+0000 7fd669055640 1 -- 172.21.15.16:0/3882031154 shutdown_connections 2024-03-28T22:08:26.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.286+0000 7fd669055640 1 -- 172.21.15.16:0/3882031154 wait complete. 2024-03-28T22:08:26.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.287+0000 7fd669055640 1 Processor -- start 2024-03-28T22:08:26.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.287+0000 7fd669055640 1 -- start start 2024-03-28T22:08:26.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.287+0000 7fd669055640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd6641006f0 0x7fd6641979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:26.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.287+0000 7fd669055640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6641018e0 0x7fd664197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:26.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.287+0000 7fd669055640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6641022a0 0x7fd66419ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:26.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.287+0000 7fd669055640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd66410d420 con 0x7fd6641018e0 2024-03-28T22:08:26.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.287+0000 7fd669055640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd66410d2a0 con 0x7fd6641022a0 2024-03-28T22:08:26.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.287+0000 7fd669055640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd66410d5a0 con 0x7fd6641006f0 2024-03-28T22:08:26.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.287+0000 7fd66259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6641018e0 0x7fd664197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:26.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.287+0000 7fd66259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6641018e0 0x7fd664197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53206/0 (socket says 172.21.15.16:53206) 2024-03-28T22:08:26.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.287+0000 7fd66259c640 1 -- 172.21.15.16:0/3912850048 learned_addr learned my addr 172.21.15.16:0/3912850048 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:26.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.288+0000 7fd662d9d640 1 --2- 172.21.15.16:0/3912850048 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd6641006f0 0x7fd6641979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:26.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.288+0000 7fd66259c640 1 -- 172.21.15.16:0/3912850048 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd6641006f0 msgr2=0x7fd6641979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:26.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.288+0000 7fd66359e640 1 --2- 172.21.15.16:0/3912850048 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6641022a0 0x7fd66419ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:26.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.288+0000 7fd66259c640 1 --2- 172.21.15.16:0/3912850048 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd6641006f0 0x7fd6641979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:26.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.288+0000 7fd66259c640 1 -- 172.21.15.16:0/3912850048 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6641022a0 msgr2=0x7fd66419ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:26.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.288+0000 7fd66259c640 1 --2- 172.21.15.16:0/3912850048 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6641022a0 0x7fd66419ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:26.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.288+0000 7fd66259c640 1 -- 172.21.15.16:0/3912850048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd66419f600 con 0x7fd6641018e0 2024-03-28T22:08:26.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.288+0000 7fd66259c640 1 --2- 172.21.15.16:0/3912850048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6641018e0 0x7fd664197f20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd65800e9f0 tx=0x7fd65800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:26.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.288+0000 7fd64bfff640 1 -- 172.21.15.16:0/3912850048 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65800cd80 con 0x7fd6641018e0 2024-03-28T22:08:26.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.288+0000 7fd64bfff640 1 -- 172.21.15.16:0/3912850048 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd658005500 con 0x7fd6641018e0 2024-03-28T22:08:26.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.288+0000 7fd669055640 1 -- 172.21.15.16:0/3912850048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd66419f8f0 con 0x7fd6641018e0 2024-03-28T22:08:26.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.288+0000 7fd64bfff640 1 -- 172.21.15.16:0/3912850048 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd658010690 con 0x7fd6641018e0 2024-03-28T22:08:26.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.288+0000 7fd669055640 1 -- 172.21.15.16:0/3912850048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd66419fdf0 con 0x7fd6641018e0 2024-03-28T22:08:26.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.289+0000 7fd669055640 1 -- 172.21.15.16:0/3912850048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd664107a10 con 0x7fd6641018e0 2024-03-28T22:08:26.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.290+0000 7fd64bfff640 1 -- 172.21.15.16:0/3912850048 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fd6580046c0 con 0x7fd6641018e0 2024-03-28T22:08:26.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.291+0000 7fd64bfff640 1 --2- 172.21.15.16:0/3912850048 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fd64c0a1d80 0x7fd64c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:26.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.291+0000 7fd662d9d640 1 --2- 172.21.15.16:0/3912850048 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fd64c0a1d80 0x7fd64c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:26.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.291+0000 7fd64bfff640 1 -- 172.21.15.16:0/3912850048 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fd6580cdf10 con 0x7fd6641018e0 2024-03-28T22:08:26.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.291+0000 7fd662d9d640 1 --2- 172.21.15.16:0/3912850048 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fd64c0a1d80 0x7fd64c0a4240 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd664101740 tx=0x7fd65c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:26.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:26 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:26.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:26 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:26.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:26 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:26.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:26 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:26.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:26 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:26.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:26 smithi053 ceph-mon[35979]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:26.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:26 smithi053 ceph-mon[35979]: mds.? [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] up:boot 2024-03-28T22:08:26.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:26 smithi053 ceph-mon[35979]: fsmap 12 up:standby 2024-03-28T22:08:26.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:26 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T22:08:26.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:26 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:08:26.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.295+0000 7fd64bfff640 1 -- 172.21.15.16:0/3912850048 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd65809b740 con 0x7fd6641018e0 2024-03-28T22:08:26.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:26 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:26.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:26 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:26.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:26 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:26.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:26 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:26.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:26 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:26.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:26 smithi016 ceph-mon[32374]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:26.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:26 smithi016 ceph-mon[32374]: mds.? [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] up:boot 2024-03-28T22:08:26.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:26 smithi016 ceph-mon[32374]: fsmap 12 up:standby 2024-03-28T22:08:26.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:26 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T22:08:26.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:26 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:08:26.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.536+0000 7fd669055640 1 -- 172.21.15.16:0/3912850048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd66402d170 con 0x7fd6641018e0 2024-03-28T22:08:26.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.537+0000 7fd64bfff640 1 -- 172.21.15.16:0/3912850048 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10464 (secure 0 0 0) 0x7fd65809b040 con 0x7fd6641018e0 2024-03-28T22:08:26.540 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:08:26.540 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/1644581227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1644581227},{"type":"v1","addr":"172.21.15.16:6839","nonce":1644581227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/2593062669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2593062669},{"type":"v1","addr":"172.21.15.16:6841","nonce":2593062669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/247249857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":247249857},{"type":"v1","addr":"172.21.15.63:6835","nonce":247249857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/2324231032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":2324231032},{"type":"v1","addr":"172.21.15.63:6837","nonce":2324231032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24493,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/1177197077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":1177197077},{"type":"v1","addr":"172.21.15.63:6839","nonce":1177197077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/350727634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":350727634},{"type":"v1","addr":"172.21.15.53:6835","nonce":350727634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/349877573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":349877573},{"type":"v1","addr":"172.21.15.53:6837","nonce":349877573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24566,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6839/3714153956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":3714153956},{"type":"v1","addr":"172.21.15.53:6839","nonce":3714153956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-03-28T22:08:26.540 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 13 2024-03-28T22:08:26.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.540+0000 7fd669055640 1 -- 172.21.15.16:0/3912850048 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fd64c0a1d80 msgr2=0x7fd64c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:26.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.540+0000 7fd669055640 1 --2- 172.21.15.16:0/3912850048 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fd64c0a1d80 0x7fd64c0a4240 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd664101740 tx=0x7fd65c005f90 comp rx=0 tx=0).stop 2024-03-28T22:08:26.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.540+0000 7fd669055640 1 -- 172.21.15.16:0/3912850048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6641018e0 msgr2=0x7fd664197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:26.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.540+0000 7fd669055640 1 --2- 172.21.15.16:0/3912850048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6641018e0 0x7fd664197f20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd65800e9f0 tx=0x7fd65800edf0 comp rx=0 tx=0).stop 2024-03-28T22:08:26.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.540+0000 7fd669055640 1 -- 172.21.15.16:0/3912850048 shutdown_connections 2024-03-28T22:08:26.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.540+0000 7fd669055640 1 --2- 172.21.15.16:0/3912850048 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fd64c0a1d80 0x7fd64c0a4240 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:26.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.540+0000 7fd669055640 1 --2- 172.21.15.16:0/3912850048 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6641022a0 0x7fd66419ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:26.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.540+0000 7fd669055640 1 --2- 172.21.15.16:0/3912850048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6641018e0 0x7fd664197f20 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:26.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.540+0000 7fd669055640 1 --2- 172.21.15.16:0/3912850048 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd6641006f0 0x7fd6641979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:26.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.540+0000 7fd669055640 1 -- 172.21.15.16:0/3912850048 >> 172.21.15.16:0/3912850048 conn(0x7fd6640fbc80 msgr2=0x7fd664104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:26.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.540+0000 7fd669055640 1 -- 172.21.15.16:0/3912850048 shutdown_connections 2024-03-28T22:08:26.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.540+0000 7fd669055640 1 -- 172.21.15.16:0/3912850048 wait complete. 2024-03-28T22:08:26.555 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:08:26.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.666+0000 7fed286ae640 1 -- 172.21.15.16:0/975389835 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed20100680 msgr2=0x7fed20100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:26.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.666+0000 7fed286ae640 1 --2- 172.21.15.16:0/975389835 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed20100680 0x7fed20100aa0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fed1c009900 tx=0x7fed1c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:08:26.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.666+0000 7fed286ae640 1 -- 172.21.15.16:0/975389835 shutdown_connections 2024-03-28T22:08:26.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.666+0000 7fed286ae640 1 --2- 172.21.15.16:0/975389835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed20102230 0x7fed20108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:26.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.666+0000 7fed286ae640 1 --2- 172.21.15.16:0/975389835 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed20101870 0x7fed20101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:26.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.666+0000 7fed286ae640 1 --2- 172.21.15.16:0/975389835 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed20100680 0x7fed20100aa0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:26.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.666+0000 7fed286ae640 1 -- 172.21.15.16:0/975389835 >> 172.21.15.16:0/975389835 conn(0x7fed200fbbf0 msgr2=0x7fed200fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:26.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.666+0000 7fed286ae640 1 -- 172.21.15.16:0/975389835 shutdown_connections 2024-03-28T22:08:26.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.666+0000 7fed286ae640 1 -- 172.21.15.16:0/975389835 wait complete. 2024-03-28T22:08:26.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.667+0000 7fed286ae640 1 Processor -- start 2024-03-28T22:08:26.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.667+0000 7fed286ae640 1 -- start start 2024-03-28T22:08:26.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.667+0000 7fed286ae640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed20100680 0x7fed201969d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:26.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.667+0000 7fed286ae640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed20101870 0x7fed20196f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:26.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.667+0000 7fed286ae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed20102230 0x7fed2019df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:26.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.667+0000 7fed286ae640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fed2010c300 con 0x7fed20102230 2024-03-28T22:08:26.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.667+0000 7fed286ae640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fed2010c180 con 0x7fed20101870 2024-03-28T22:08:26.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.667+0000 7fed286ae640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fed2010c480 con 0x7fed20100680 2024-03-28T22:08:26.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.667+0000 7fed26c4b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed20102230 0x7fed2019df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:26.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.667+0000 7fed26c4b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed20102230 0x7fed2019df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53224/0 (socket says 172.21.15.16:53224) 2024-03-28T22:08:26.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.667+0000 7fed26c4b640 1 -- 172.21.15.16:0/2115018154 learned_addr learned my addr 172.21.15.16:0/2115018154 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:26.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.667+0000 7fed2644a640 1 --2- 172.21.15.16:0/2115018154 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed20100680 0x7fed201969d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:26.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.667+0000 7fed25c49640 1 --2- 172.21.15.16:0/2115018154 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed20101870 0x7fed20196f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:26.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.667+0000 7fed26c4b640 1 -- 172.21.15.16:0/2115018154 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed20100680 msgr2=0x7fed201969d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:26.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.667+0000 7fed26c4b640 1 --2- 172.21.15.16:0/2115018154 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed20100680 0x7fed201969d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:26.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.667+0000 7fed26c4b640 1 -- 172.21.15.16:0/2115018154 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed20101870 msgr2=0x7fed20196f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:26.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.667+0000 7fed26c4b640 1 --2- 172.21.15.16:0/2115018154 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed20101870 0x7fed20196f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:26.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.667+0000 7fed26c4b640 1 -- 172.21.15.16:0/2115018154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed2019e5a0 con 0x7fed20102230 2024-03-28T22:08:26.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.667+0000 7fed26c4b640 1 --2- 172.21.15.16:0/2115018154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed20102230 0x7fed2019df30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fed18009f20 tx=0x7fed1800b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:26.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.668+0000 7fed177fe640 1 -- 172.21.15.16:0/2115018154 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed18003970 con 0x7fed20102230 2024-03-28T22:08:26.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.668+0000 7fed177fe640 1 -- 172.21.15.16:0/2115018154 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed18004020 con 0x7fed20102230 2024-03-28T22:08:26.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.668+0000 7fed286ae640 1 -- 172.21.15.16:0/2115018154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed20068a80 con 0x7fed20102230 2024-03-28T22:08:26.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.668+0000 7fed177fe640 1 -- 172.21.15.16:0/2115018154 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed180178d0 con 0x7fed20102230 2024-03-28T22:08:26.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.668+0000 7fed286ae640 1 -- 172.21.15.16:0/2115018154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed20068f80 con 0x7fed20102230 2024-03-28T22:08:26.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.669+0000 7fed286ae640 1 -- 172.21.15.16:0/2115018154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed20107410 con 0x7fed20102230 2024-03-28T22:08:26.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.670+0000 7fed177fe640 1 -- 172.21.15.16:0/2115018154 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fed180041d0 con 0x7fed20102230 2024-03-28T22:08:26.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.670+0000 7fed177fe640 1 --2- 172.21.15.16:0/2115018154 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fed0c0a1d80 0x7fed0c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:26.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.671+0000 7fed177fe640 1 -- 172.21.15.16:0/2115018154 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fed180cf5d0 con 0x7fed20102230 2024-03-28T22:08:26.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.671+0000 7fed2644a640 1 --2- 172.21.15.16:0/2115018154 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fed0c0a1d80 0x7fed0c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:26.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.671+0000 7fed2644a640 1 --2- 172.21.15.16:0/2115018154 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fed0c0a1d80 0x7fed0c0a4240 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fed1c000c00 tx=0x7fed1c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:26.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.675+0000 7fed177fe640 1 -- 172.21.15.16:0/2115018154 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fed1809c850 con 0x7fed20102230 2024-03-28T22:08:26.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.892+0000 7fed286ae640 1 -- 172.21.15.16:0/2115018154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fed2019e770 con 0x7fed20102230 2024-03-28T22:08:26.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.893+0000 7fed177fe640 1 -- 172.21.15.16:0/2115018154 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18348 (secure 0 0 0) 0x7fed180a08f0 con 0x7fed20102230 2024-03-28T22:08:26.896 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:08:26.896 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":82,"fsid":"798d9884-ed4e-11ee-95d0-87774f69a715","created":"2024-03-28T22:00:27.216099+0000","modified":"2024-03-28T22:07:41.393002+0000","last_up_change":"2024-03-28T22:07:40.251141+0000","last_in_change":"2024-03-28T22:07:19.456451+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:03:59.327832+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"46e7c8f8-ecc8-43d6-ae53-0655728f3240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6803","nonce":1589692483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6805","nonce":1589692483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6809","nonce":1589692483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6807","nonce":1589692483}]},"public_addr":"172.21.15.16:6803/1589692483","cluster_addr":"172.21.15.16:6805/1589692483","heartbeat_back_addr":"172.21.15.16:6809/1589692483","heartbeat_front_addr":"172.21.15.16:6807/1589692483","state":["exists","up"]},{"osd":1,"uuid":"af6d9c22-2830-4ce2-a547-011df0c6cbad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6801","nonce":795531234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6803","nonce":795531234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6807","nonce":795531234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6805","nonce":795531234}]},"public_addr":"172.21.15.53:6801/795531234","cluster_addr":"172.21.15.53:6803/795531234","heartbeat_back_addr":"172.21.15.53:6807/795531234","heartbeat_front_addr":"172.21.15.53:6805/795531234","state":["exists","up"]},{"osd":2,"uuid":"b10adca9-110c-4f87-85e0-b7542709d31a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6801","nonce":2214131086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6803","nonce":2214131086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6807","nonce":2214131086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6805","nonce":2214131086}]},"public_addr":"172.21.15.63:6801/2214131086","cluster_addr":"172.21.15.63:6803/2214131086","heartbeat_back_addr":"172.21.15.63:6807/2214131086","heartbeat_front_addr":"172.21.15.63:6805/2214131086","state":["exists","up"]},{"osd":3,"uuid":"c83cf4bb-395a-4786-abf7-74ca93a33eb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6811","nonce":13176789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6813","nonce":13176789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6817","nonce":13176789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6815","nonce":13176789}]},"public_addr":"172.21.15.16:6811/13176789","cluster_addr":"172.21.15.16:6813/13176789","heartbeat_back_addr":"172.21.15.16:6817/13176789","heartbeat_front_addr":"172.21.15.16:6815/13176789","state":["exists","up"]},{"osd":4,"uuid":"941b3e54-47d6-4e85-8069-e730c3fa8eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6809","nonce":772161849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6811","nonce":772161849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6815","nonce":772161849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6813","nonce":772161849}]},"public_addr":"172.21.15.53:6809/772161849","cluster_addr":"172.21.15.53:6811/772161849","heartbeat_back_addr":"172.21.15.53:6815/772161849","heartbeat_front_addr":"172.21.15.53:6813/772161849","state":["exists","up"]},{"osd":5,"uuid":"51d89826-1d60-40fc-8fd8-09bfe1ab4125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6809","nonce":1602554852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6811","nonce":1602554852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6815","nonce":1602554852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6813","nonce":1602554852}]},"public_addr":"172.21.15.63:6809/1602554852","cluster_addr":"172.21.15.63:6811/1602554852","heartbeat_back_addr":"172.21.15.63:6815/1602554852","heartbeat_front_addr":"172.21.15.63:6813/1602554852","state":["exists","up"]},{"osd":6,"uuid":"978efe5d-2c55-4ff8-8a1e-9f891388945e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6819","nonce":4290459224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6821","nonce":4290459224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6825","nonce":4290459224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6823","nonce":4290459224}]},"public_addr":"172.21.15.16:6819/4290459224","cluster_addr":"172.21.15.16:6821/4290459224","heartbeat_back_addr":"172.21.15.16:6825/4290459224","heartbeat_front_addr":"172.21.15.16:6823/4290459224","state":["exists","up"]},{"osd":7,"uuid":"50d489cb-dbb8-48df-b868-b29edf12e04d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6817","nonce":1056467525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6819","nonce":1056467525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6823","nonce":1056467525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6821","nonce":1056467525}]},"public_addr":"172.21.15.53:6817/1056467525","cluster_addr":"172.21.15.53:6819/1056467525","heartbeat_back_addr":"172.21.15.53:6823/1056467525","heartbeat_front_addr":"172.21.15.53:6821/1056467525","state":["exists","up"]},{"osd":8,"uuid":"49b39414-cc9c-4057-ae3f-cc109cd979da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6817","nonce":2918516794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6819","nonce":2918516794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6823","nonce":2918516794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6821","nonce":2918516794}]},"public_addr":"172.21.15.63:6817/2918516794","cluster_addr":"172.21.15.63:6819/2918516794","heartbeat_back_addr":"172.21.15.63:6823/2918516794","heartbeat_front_addr":"172.21.15.63:6821/2918516794","state":["exists","up"]},{"osd":9,"uuid":"4c873e51-b783-4eb6-83a3-fee31722d60b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6827","nonce":1585624367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6829","nonce":1585624367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6833","nonce":1585624367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6831","nonce":1585624367}]},"public_addr":"172.21.15.16:6827/1585624367","cluster_addr":"172.21.15.16:6829/1585624367","heartbeat_back_addr":"172.21.15.16:6833/1585624367","heartbeat_front_addr":"172.21.15.16:6831/1585624367","state":["exists","up"]},{"osd":10,"uuid":"085a78bf-e2ad-481a-85a3-71d1fc549a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6825","nonce":3357446484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6827","nonce":3357446484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6831","nonce":3357446484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6829","nonce":3357446484}]},"public_addr":"172.21.15.53:6825/3357446484","cluster_addr":"172.21.15.53:6827/3357446484","heartbeat_back_addr":"172.21.15.53:6831/3357446484","heartbeat_front_addr":"172.21.15.53:6829/3357446484","state":["exists","up"]},{"osd":11,"uuid":"13302aa3-08b9-4639-9042-27d850b4d5f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6825","nonce":2167866570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6827","nonce":2167866570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6831","nonce":2167866570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6829","nonce":2167866570}]},"public_addr":"172.21.15.63:6825/2167866570","cluster_addr":"172.21.15.63:6827/2167866570","heartbeat_back_addr":"172.21.15.63:6831/2167866570","heartbeat_front_addr":"172.21.15.63:6829/2167866570","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:07.450323+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:31.363172+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:55.295021+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:19.794760+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:44.063492+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:08.195064+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:33.655328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:57.639414+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:22.402394+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:47.462050+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:12.226307+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:37.495625+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3821408150":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6801/1832422779":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/996598130":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/1062264551":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/317557779":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6801/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/3631746518":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6800/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/3116652786":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6800/1832422779":"2024-03-29T22:01:28.993434+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:08:26.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.895+0000 7fed286ae640 1 -- 172.21.15.16:0/2115018154 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fed0c0a1d80 msgr2=0x7fed0c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:26.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.895+0000 7fed286ae640 1 --2- 172.21.15.16:0/2115018154 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fed0c0a1d80 0x7fed0c0a4240 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fed1c000c00 tx=0x7fed1c005fd0 comp rx=0 tx=0).stop 2024-03-28T22:08:26.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.895+0000 7fed286ae640 1 -- 172.21.15.16:0/2115018154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed20102230 msgr2=0x7fed2019df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:26.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.895+0000 7fed286ae640 1 --2- 172.21.15.16:0/2115018154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed20102230 0x7fed2019df30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fed18009f20 tx=0x7fed1800b690 comp rx=0 tx=0).stop 2024-03-28T22:08:26.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.895+0000 7fed286ae640 1 -- 172.21.15.16:0/2115018154 shutdown_connections 2024-03-28T22:08:26.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.895+0000 7fed286ae640 1 --2- 172.21.15.16:0/2115018154 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fed0c0a1d80 0x7fed0c0a4240 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:26.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.895+0000 7fed286ae640 1 --2- 172.21.15.16:0/2115018154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed20102230 0x7fed2019df30 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:26.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.895+0000 7fed286ae640 1 --2- 172.21.15.16:0/2115018154 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed20101870 0x7fed20196f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:26.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.895+0000 7fed286ae640 1 --2- 172.21.15.16:0/2115018154 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed20100680 0x7fed201969d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:26.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.895+0000 7fed286ae640 1 -- 172.21.15.16:0/2115018154 >> 172.21.15.16:0/2115018154 conn(0x7fed200fbbf0 msgr2=0x7fed20104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:26.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.896+0000 7fed286ae640 1 -- 172.21.15.16:0/2115018154 shutdown_connections 2024-03-28T22:08:26.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:26.896+0000 7fed286ae640 1 -- 172.21.15.16:0/2115018154 wait complete. 2024-03-28T22:08:26.909 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T22:03:59.327832+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T22:08:26.909 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T22:08:27.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.017+0000 7f0b9c249640 1 -- 172.21.15.16:0/4105687842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b94100650 msgr2=0x7f0b94100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:27.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.017+0000 7f0b9c249640 1 --2- 172.21.15.16:0/4105687842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b94100650 0x7f0b94100a70 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c009900 tx=0x7f0b8c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:08:27.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.017+0000 7f0b9c249640 1 -- 172.21.15.16:0/4105687842 shutdown_connections 2024-03-28T22:08:27.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.017+0000 7f0b9c249640 1 --2- 172.21.15.16:0/4105687842 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0b94102200 0x7f0b94108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:27.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.017+0000 7f0b9c249640 1 --2- 172.21.15.16:0/4105687842 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b94101840 0x7f0b94101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:27.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.017+0000 7f0b9c249640 1 --2- 172.21.15.16:0/4105687842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b94100650 0x7f0b94100a70 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:27.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.017+0000 7f0b9c249640 1 -- 172.21.15.16:0/4105687842 >> 172.21.15.16:0/4105687842 conn(0x7f0b940fbb80 msgr2=0x7f0b940fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:27.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.018+0000 7f0b9c249640 1 -- 172.21.15.16:0/4105687842 shutdown_connections 2024-03-28T22:08:27.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.018+0000 7f0b9c249640 1 -- 172.21.15.16:0/4105687842 wait complete. 2024-03-28T22:08:27.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.018+0000 7f0b9c249640 1 Processor -- start 2024-03-28T22:08:27.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.018+0000 7f0b9c249640 1 -- start start 2024-03-28T22:08:27.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.018+0000 7f0b9c249640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b94100650 0x7f0b94199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:27.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.018+0000 7f0b9c249640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0b94101840 0x7f0b9419a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:27.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.018+0000 7f0b9c249640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b94102200 0x7f0b941a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:27.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.018+0000 7f0b9c249640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0b9410c2d0 con 0x7f0b94100650 2024-03-28T22:08:27.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.018+0000 7f0b9c249640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0b9410c150 con 0x7f0b94101840 2024-03-28T22:08:27.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.018+0000 7f0b9c249640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0b9410c450 con 0x7f0b94102200 2024-03-28T22:08:27.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.018+0000 7f0b99fe5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b94100650 0x7f0b94199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:27.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.018+0000 7f0b99fe5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b94100650 0x7f0b94199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53242/0 (socket says 172.21.15.16:53242) 2024-03-28T22:08:27.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.018+0000 7f0b99fe5640 1 -- 172.21.15.16:0/1281952224 learned_addr learned my addr 172.21.15.16:0/1281952224 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:27.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.018+0000 7f0b9a7e6640 1 --2- 172.21.15.16:0/1281952224 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b94102200 0x7f0b941a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:27.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.018+0000 7f0b997e4640 1 --2- 172.21.15.16:0/1281952224 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0b94101840 0x7f0b9419a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:27.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.019+0000 7f0b99fe5640 1 -- 172.21.15.16:0/1281952224 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b94102200 msgr2=0x7f0b941a1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:27.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.019+0000 7f0b99fe5640 1 --2- 172.21.15.16:0/1281952224 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b94102200 0x7f0b941a1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:27.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.019+0000 7f0b99fe5640 1 -- 172.21.15.16:0/1281952224 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0b94101840 msgr2=0x7f0b9419a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:27.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.019+0000 7f0b99fe5640 1 --2- 172.21.15.16:0/1281952224 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0b94101840 0x7f0b9419a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:27.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.019+0000 7f0b99fe5640 1 -- 172.21.15.16:0/1281952224 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b941a18b0 con 0x7f0b94100650 2024-03-28T22:08:27.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.019+0000 7f0b99fe5640 1 --2- 172.21.15.16:0/1281952224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b94100650 0x7f0b94199c70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c000c00 tx=0x7f0b8c005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:27.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.019+0000 7f0b8affd640 1 -- 172.21.15.16:0/1281952224 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c041070 con 0x7f0b94100650 2024-03-28T22:08:27.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.019+0000 7f0b8affd640 1 -- 172.21.15.16:0/1281952224 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b8c0044d0 con 0x7f0b94100650 2024-03-28T22:08:27.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.019+0000 7f0b9c249640 1 -- 172.21.15.16:0/1281952224 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b941a1b40 con 0x7f0b94100650 2024-03-28T22:08:27.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.019+0000 7f0b8affd640 1 -- 172.21.15.16:0/1281952224 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c034440 con 0x7f0b94100650 2024-03-28T22:08:27.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.019+0000 7f0b9c249640 1 -- 172.21.15.16:0/1281952224 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b941a2080 con 0x7f0b94100650 2024-03-28T22:08:27.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.020+0000 7f0b9c249640 1 -- 172.21.15.16:0/1281952224 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b941a2310 con 0x7f0b94100650 2024-03-28T22:08:27.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.021+0000 7f0b8affd640 1 -- 172.21.15.16:0/1281952224 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f0b8c044070 con 0x7f0b94100650 2024-03-28T22:08:27.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.022+0000 7f0b8affd640 1 --2- 172.21.15.16:0/1281952224 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0b800a1e00 0x7f0b800a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:27.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.022+0000 7f0b8affd640 1 -- 172.21.15.16:0/1281952224 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f0b8c0ef560 con 0x7f0b94100650 2024-03-28T22:08:27.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.022+0000 7f0b997e4640 1 --2- 172.21.15.16:0/1281952224 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0b800a1e00 0x7f0b800a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:27.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.022+0000 7f0b997e4640 1 --2- 172.21.15.16:0/1281952224 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0b800a1e00 0x7f0b800a42c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0b9419b170 tx=0x7f0b84005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:27.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.025+0000 7f0b8affd640 1 -- 172.21.15.16:0/1281952224 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0b8c0bc7e0 con 0x7f0b94100650 2024-03-28T22:08:27.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:27 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3912850048' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:27.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:27 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2115018154' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:08:27.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:27 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:27.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:27 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:27.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:27 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:27.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:27 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:27.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:27 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:27.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:27 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:27.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:27 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:08:27.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:27 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:08:27.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:27 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:27.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.214+0000 7f0b9c249640 1 -- 172.21.15.16:0/1281952224 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0b941a24e0 con 0x7f0b94100650 2024-03-28T22:08:27.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.214+0000 7f0b8affd640 1 -- 172.21.15.16:0/1281952224 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f0b8c0c0880 con 0x7f0b94100650 2024-03-28T22:08:27.217 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-28T22:08:27.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.217+0000 7f0b9c249640 1 -- 172.21.15.16:0/1281952224 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0b800a1e00 msgr2=0x7f0b800a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:27.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.217+0000 7f0b9c249640 1 --2- 172.21.15.16:0/1281952224 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0b800a1e00 0x7f0b800a42c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0b9419b170 tx=0x7f0b84005f90 comp rx=0 tx=0).stop 2024-03-28T22:08:27.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.217+0000 7f0b9c249640 1 -- 172.21.15.16:0/1281952224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b94100650 msgr2=0x7f0b94199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:27.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.217+0000 7f0b9c249640 1 --2- 172.21.15.16:0/1281952224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b94100650 0x7f0b94199c70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c000c00 tx=0x7f0b8c005d50 comp rx=0 tx=0).stop 2024-03-28T22:08:27.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.217+0000 7f0b9c249640 1 -- 172.21.15.16:0/1281952224 shutdown_connections 2024-03-28T22:08:27.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.217+0000 7f0b9c249640 1 --2- 172.21.15.16:0/1281952224 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0b800a1e00 0x7f0b800a42c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:27.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.217+0000 7f0b9c249640 1 --2- 172.21.15.16:0/1281952224 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b94102200 0x7f0b941a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:27.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.217+0000 7f0b9c249640 1 --2- 172.21.15.16:0/1281952224 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0b94101840 0x7f0b9419a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:27.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.217+0000 7f0b9c249640 1 --2- 172.21.15.16:0/1281952224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b94100650 0x7f0b94199c70 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:27.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.217+0000 7f0b9c249640 1 -- 172.21.15.16:0/1281952224 >> 172.21.15.16:0/1281952224 conn(0x7f0b940fbb80 msgr2=0x7f0b940fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:27.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.217+0000 7f0b9c249640 1 -- 172.21.15.16:0/1281952224 shutdown_connections 2024-03-28T22:08:27.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.217+0000 7f0b9c249640 1 -- 172.21.15.16:0/1281952224 wait complete. 2024-03-28T22:08:27.231 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-28T22:08:27.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.334+0000 7faea0725640 1 -- 172.21.15.16:0/1976321754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae981021f0 msgr2=0x7fae98108a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:27.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.334+0000 7faea0725640 1 --2- 172.21.15.16:0/1976321754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae981021f0 0x7fae98108a80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fae90009ca0 tx=0x7fae9002c0f0 comp rx=0 tx=0).stop 2024-03-28T22:08:27.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.334+0000 7faea0725640 1 -- 172.21.15.16:0/1976321754 shutdown_connections 2024-03-28T22:08:27.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.334+0000 7faea0725640 1 --2- 172.21.15.16:0/1976321754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae981021f0 0x7fae98108a80 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:27.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.334+0000 7faea0725640 1 --2- 172.21.15.16:0/1976321754 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae98101830 0x7fae98101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:27.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.334+0000 7faea0725640 1 --2- 172.21.15.16:0/1976321754 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae98100640 0x7fae98100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:27.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.334+0000 7faea0725640 1 -- 172.21.15.16:0/1976321754 >> 172.21.15.16:0/1976321754 conn(0x7fae980fbbb0 msgr2=0x7fae980fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:27.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.334+0000 7faea0725640 1 -- 172.21.15.16:0/1976321754 shutdown_connections 2024-03-28T22:08:27.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.334+0000 7faea0725640 1 -- 172.21.15.16:0/1976321754 wait complete. 2024-03-28T22:08:27.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.335+0000 7faea0725640 1 Processor -- start 2024-03-28T22:08:27.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.335+0000 7faea0725640 1 -- start start 2024-03-28T22:08:27.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.335+0000 7faea0725640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae98100640 0x7fae981045d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:27.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.335+0000 7faea0725640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae98101830 0x7fae98102c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:27.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.335+0000 7faea0725640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae981021f0 0x7fae98103150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:27.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.335+0000 7faea0725640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fae9810c2a0 con 0x7fae981021f0 2024-03-28T22:08:27.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.335+0000 7faea0725640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fae9810c120 con 0x7fae98101830 2024-03-28T22:08:27.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.335+0000 7faea0725640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fae9810c420 con 0x7fae98100640 2024-03-28T22:08:27.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.335+0000 7fae9ecc2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae981021f0 0x7fae98103150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:27.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.335+0000 7fae9ecc2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae981021f0 0x7fae98103150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53258/0 (socket says 172.21.15.16:53258) 2024-03-28T22:08:27.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.335+0000 7fae9ecc2640 1 -- 172.21.15.16:0/2439906447 learned_addr learned my addr 172.21.15.16:0/2439906447 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:27.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.335+0000 7fae9e4c1640 1 --2- 172.21.15.16:0/2439906447 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae98100640 0x7fae981045d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:27.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.335+0000 7fae9dcc0640 1 --2- 172.21.15.16:0/2439906447 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae98101830 0x7fae98102c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:27.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.335+0000 7fae9ecc2640 1 -- 172.21.15.16:0/2439906447 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae98100640 msgr2=0x7fae981045d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:27.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.335+0000 7fae9ecc2640 1 --2- 172.21.15.16:0/2439906447 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae98100640 0x7fae981045d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:27.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.335+0000 7fae9ecc2640 1 -- 172.21.15.16:0/2439906447 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae98101830 msgr2=0x7fae98102c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:27.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.335+0000 7fae9ecc2640 1 --2- 172.21.15.16:0/2439906447 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae98101830 0x7fae98102c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:27.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.335+0000 7fae9ecc2640 1 -- 172.21.15.16:0/2439906447 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae981039e0 con 0x7fae981021f0 2024-03-28T22:08:27.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.335+0000 7fae9ecc2640 1 --2- 172.21.15.16:0/2439906447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae981021f0 0x7fae98103150 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fae90037040 tx=0x7fae90032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:27.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.336+0000 7fae8b7fe640 1 -- 172.21.15.16:0/2439906447 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae90041020 con 0x7fae981021f0 2024-03-28T22:08:27.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.336+0000 7fae8b7fe640 1 -- 172.21.15.16:0/2439906447 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae90034d90 con 0x7fae981021f0 2024-03-28T22:08:27.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.336+0000 7faea0725640 1 -- 172.21.15.16:0/2439906447 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae981a1a90 con 0x7fae981021f0 2024-03-28T22:08:27.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.336+0000 7fae8b7fe640 1 -- 172.21.15.16:0/2439906447 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae90039040 con 0x7fae981021f0 2024-03-28T22:08:27.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.336+0000 7faea0725640 1 -- 172.21.15.16:0/2439906447 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae981a1fe0 con 0x7fae981021f0 2024-03-28T22:08:27.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.337+0000 7faea0725640 1 -- 172.21.15.16:0/2439906447 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae981a2210 con 0x7fae981021f0 2024-03-28T22:08:27.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.338+0000 7fae8b7fe640 1 -- 172.21.15.16:0/2439906447 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fae90044020 con 0x7fae981021f0 2024-03-28T22:08:27.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.339+0000 7fae8b7fe640 1 --2- 172.21.15.16:0/2439906447 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fae840a1d80 0x7fae840a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:27.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.339+0000 7fae9e4c1640 1 --2- 172.21.15.16:0/2439906447 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fae840a1d80 0x7fae840a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:27.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.339+0000 7fae8b7fe640 1 -- 172.21.15.16:0/2439906447 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fae900ef7d0 con 0x7fae981021f0 2024-03-28T22:08:27.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.339+0000 7fae9e4c1640 1 --2- 172.21.15.16:0/2439906447 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fae840a1d80 0x7fae840a4240 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fae98101690 tx=0x7fae94005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:27.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.343+0000 7fae8b7fe640 1 -- 172.21.15.16:0/2439906447 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fae9004b020 con 0x7fae981021f0 2024-03-28T22:08:27.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:27 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/3912850048' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:27.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:27 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2115018154' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:08:27.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:27 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:27.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:27 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:27.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:27 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:27.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:27 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:27.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:27 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:27.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:27 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:27.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:27 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:08:27.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:27 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:08:27.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:27 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:27.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.528+0000 7faea0725640 1 -- 172.21.15.16:0/2439906447 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fae981a28b0 con 0x7fae981021f0 2024-03-28T22:08:27.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.528+0000 7fae8b7fe640 1 -- 172.21.15.16:0/2439906447 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v82) ==== 84+0+65 (secure 0 0 0) 0x7fae900bd0a0 con 0x7fae981021f0 2024-03-28T22:08:27.531 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:08:27.531 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2024-03-28T22:08:27.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:08:27.531 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2024-03-28T22:08:27.531 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2024-03-28T22:08:27.531 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:08:27.531 INFO:teuthology.orchestra.run.smithi016.stdout:] 2024-03-28T22:08:27.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.530+0000 7faea0725640 1 -- 172.21.15.16:0/2439906447 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fae840a1d80 msgr2=0x7fae840a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:27.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.530+0000 7faea0725640 1 --2- 172.21.15.16:0/2439906447 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fae840a1d80 0x7fae840a4240 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fae98101690 tx=0x7fae94005f90 comp rx=0 tx=0).stop 2024-03-28T22:08:27.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.530+0000 7faea0725640 1 -- 172.21.15.16:0/2439906447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae981021f0 msgr2=0x7fae98103150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:27.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.530+0000 7faea0725640 1 --2- 172.21.15.16:0/2439906447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae981021f0 0x7fae98103150 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fae90037040 tx=0x7fae90032f20 comp rx=0 tx=0).stop 2024-03-28T22:08:27.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.530+0000 7faea0725640 1 -- 172.21.15.16:0/2439906447 shutdown_connections 2024-03-28T22:08:27.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.530+0000 7faea0725640 1 --2- 172.21.15.16:0/2439906447 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fae840a1d80 0x7fae840a4240 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:27.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.530+0000 7faea0725640 1 --2- 172.21.15.16:0/2439906447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae981021f0 0x7fae98103150 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:27.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.530+0000 7faea0725640 1 --2- 172.21.15.16:0/2439906447 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae98101830 0x7fae98102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:27.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.530+0000 7faea0725640 1 --2- 172.21.15.16:0/2439906447 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae98100640 0x7fae981045d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:27.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.530+0000 7faea0725640 1 -- 172.21.15.16:0/2439906447 >> 172.21.15.16:0/2439906447 conn(0x7fae980fbbb0 msgr2=0x7fae980fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:27.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.530+0000 7faea0725640 1 -- 172.21.15.16:0/2439906447 shutdown_connections 2024-03-28T22:08:27.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.530+0000 7faea0725640 1 -- 172.21.15.16:0/2439906447 wait complete. 2024-03-28T22:08:27.546 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-28T22:08:27.546 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-28T22:08:27.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:27 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3912850048' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:27.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:27 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2115018154' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:08:27.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:27 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:27.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:27 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:27.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:27 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:27.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:27 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:27.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:27 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:27.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:27 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:27.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:27 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:08:27.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:27 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:08:27.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:27 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:27.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.666+0000 7fb744ea4640 1 -- 172.21.15.16:0/2466105905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb740102950 msgr2=0x7fb740102dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:27.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.666+0000 7fb744ea4640 1 --2- 172.21.15.16:0/2466105905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb740102950 0x7fb740102dd0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb734009900 tx=0x7fb73402bc30 comp rx=0 tx=0).stop 2024-03-28T22:08:27.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.666+0000 7fb744ea4640 1 -- 172.21.15.16:0/2466105905 shutdown_connections 2024-03-28T22:08:27.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.666+0000 7fb744ea4640 1 --2- 172.21.15.16:0/2466105905 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb740103310 0x7fb740109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:27.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.666+0000 7fb744ea4640 1 --2- 172.21.15.16:0/2466105905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb740102950 0x7fb740102dd0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:27.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.666+0000 7fb744ea4640 1 --2- 172.21.15.16:0/2466105905 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb740101760 0x7fb740101b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:27.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.666+0000 7fb744ea4640 1 -- 172.21.15.16:0/2466105905 >> 172.21.15.16:0/2466105905 conn(0x7fb7400fccd0 msgr2=0x7fb7400ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:27.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.666+0000 7fb744ea4640 1 -- 172.21.15.16:0/2466105905 shutdown_connections 2024-03-28T22:08:27.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.666+0000 7fb744ea4640 1 -- 172.21.15.16:0/2466105905 wait complete. 2024-03-28T22:08:27.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.667+0000 7fb744ea4640 1 Processor -- start 2024-03-28T22:08:27.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.667+0000 7fb744ea4640 1 -- start start 2024-03-28T22:08:27.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.667+0000 7fb744ea4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb740101760 0x7fb740075500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:27.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.667+0000 7fb744ea4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb740102950 0x7fb740073b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:27.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.667+0000 7fb744ea4640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb740103310 0x7fb740074080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:27.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.667+0000 7fb744ea4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb74010d430 con 0x7fb740101760 2024-03-28T22:08:27.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.667+0000 7fb744ea4640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb74010d2b0 con 0x7fb740103310 2024-03-28T22:08:27.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.667+0000 7fb744ea4640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb74010d5b0 con 0x7fb740102950 2024-03-28T22:08:27.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.667+0000 7fb73e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb740101760 0x7fb740075500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:27.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.667+0000 7fb73e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb740101760 0x7fb740075500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53276/0 (socket says 172.21.15.16:53276) 2024-03-28T22:08:27.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.667+0000 7fb73e59c640 1 -- 172.21.15.16:0/1246655427 learned_addr learned my addr 172.21.15.16:0/1246655427 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:27.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.667+0000 7fb73dd9b640 1 --2- 172.21.15.16:0/1246655427 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb740102950 0x7fb740073b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:27.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.667+0000 7fb73ed9d640 1 --2- 172.21.15.16:0/1246655427 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb740103310 0x7fb740074080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:27.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.667+0000 7fb73e59c640 1 -- 172.21.15.16:0/1246655427 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb740102950 msgr2=0x7fb740073b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:27.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.667+0000 7fb73e59c640 1 --2- 172.21.15.16:0/1246655427 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb740102950 0x7fb740073b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:27.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.667+0000 7fb73e59c640 1 -- 172.21.15.16:0/1246655427 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb740103310 msgr2=0x7fb740074080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:27.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.667+0000 7fb73e59c640 1 --2- 172.21.15.16:0/1246655427 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb740103310 0x7fb740074080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:27.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.667+0000 7fb73e59c640 1 -- 172.21.15.16:0/1246655427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb740074910 con 0x7fb740101760 2024-03-28T22:08:27.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.667+0000 7fb73dd9b640 1 --2- 172.21.15.16:0/1246655427 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb740102950 0x7fb740073b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:08:27.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.668+0000 7fb73e59c640 1 --2- 172.21.15.16:0/1246655427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb740101760 0x7fb740075500 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb73000e9c0 tx=0x7fb73000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:27.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.668+0000 7fb72f7fe640 1 -- 172.21.15.16:0/1246655427 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb73000cd80 con 0x7fb740101760 2024-03-28T22:08:27.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.668+0000 7fb72f7fe640 1 -- 172.21.15.16:0/1246655427 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7300035f0 con 0x7fb740101760 2024-03-28T22:08:27.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.668+0000 7fb744ea4640 1 -- 172.21.15.16:0/1246655427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7401a1bc0 con 0x7fb740101760 2024-03-28T22:08:27.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.668+0000 7fb72f7fe640 1 -- 172.21.15.16:0/1246655427 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb730010690 con 0x7fb740101760 2024-03-28T22:08:27.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.668+0000 7fb744ea4640 1 -- 172.21.15.16:0/1246655427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7401a2130 con 0x7fb740101760 2024-03-28T22:08:27.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.669+0000 7fb744ea4640 1 -- 172.21.15.16:0/1246655427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb740108a80 con 0x7fb740101760 2024-03-28T22:08:27.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.670+0000 7fb72f7fe640 1 -- 172.21.15.16:0/1246655427 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fb7300037a0 con 0x7fb740101760 2024-03-28T22:08:27.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.671+0000 7fb72f7fe640 1 --2- 172.21.15.16:0/1246655427 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb7280a1e00 0x7fb7280a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:27.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.671+0000 7fb73dd9b640 1 --2- 172.21.15.16:0/1246655427 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb7280a1e00 0x7fb7280a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:27.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.671+0000 7fb72f7fe640 1 -- 172.21.15.16:0/1246655427 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fb7300cdca0 con 0x7fb740101760 2024-03-28T22:08:27.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.671+0000 7fb73dd9b640 1 --2- 172.21.15.16:0/1246655427 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb7280a1e00 0x7fb7280a42c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb73402c240 tx=0x7fb73402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:27.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.676+0000 7fb72f7fe640 1 -- 172.21.15.16:0/1246655427 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb73009b570 con 0x7fb740101760 2024-03-28T22:08:27.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:27.875+0000 7fb744ea4640 1 -- 172.21.15.16:0/1246655427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fb74002d170 con 0x7fb740101760 2024-03-28T22:08:28.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:28 smithi063 ceph-mon[36249]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:28.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:28 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:08:28.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:28 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/1281952224' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:08:28.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:28 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2439906447' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-28T22:08:28.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:28 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/1246655427' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T22:08:28.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:28 smithi016 ceph-mon[32374]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:28.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:28 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:08:28.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:28 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/1281952224' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:08:28.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:28 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2439906447' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-28T22:08:28.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:28 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/1246655427' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T22:08:28.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:28 smithi053 ceph-mon[35979]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:28.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:28 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:08:28.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:28 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/1281952224' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:08:28.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:28 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2439906447' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-28T22:08:28.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:28 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/1246655427' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T22:08:28.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.304+0000 7fb72f7fe640 1 -- 172.21.15.16:0/1246655427 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v83) ==== 138+0+0 (secure 0 0 0) 0x7fb73009ae70 con 0x7fb740101760 2024-03-28T22:08:28.307 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2024-03-28T22:08:28.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.306+0000 7fb744ea4640 1 -- 172.21.15.16:0/1246655427 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb7280a1e00 msgr2=0x7fb7280a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:28.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.306+0000 7fb744ea4640 1 --2- 172.21.15.16:0/1246655427 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb7280a1e00 0x7fb7280a42c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb73402c240 tx=0x7fb73402df90 comp rx=0 tx=0).stop 2024-03-28T22:08:28.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.307+0000 7fb744ea4640 1 -- 172.21.15.16:0/1246655427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb740101760 msgr2=0x7fb740075500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:28.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.307+0000 7fb744ea4640 1 --2- 172.21.15.16:0/1246655427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb740101760 0x7fb740075500 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb73000e9c0 tx=0x7fb73000ee80 comp rx=0 tx=0).stop 2024-03-28T22:08:28.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.307+0000 7fb744ea4640 1 -- 172.21.15.16:0/1246655427 shutdown_connections 2024-03-28T22:08:28.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.307+0000 7fb744ea4640 1 --2- 172.21.15.16:0/1246655427 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb7280a1e00 0x7fb7280a42c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:28.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.307+0000 7fb744ea4640 1 --2- 172.21.15.16:0/1246655427 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb740103310 0x7fb740074080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:28.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.307+0000 7fb744ea4640 1 --2- 172.21.15.16:0/1246655427 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb740102950 0x7fb740073b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:28.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.307+0000 7fb744ea4640 1 --2- 172.21.15.16:0/1246655427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb740101760 0x7fb740075500 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:28.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.307+0000 7fb744ea4640 1 -- 172.21.15.16:0/1246655427 >> 172.21.15.16:0/1246655427 conn(0x7fb7400fccd0 msgr2=0x7fb740105c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:28.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.308+0000 7fb744ea4640 1 -- 172.21.15.16:0/1246655427 shutdown_connections 2024-03-28T22:08:28.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.308+0000 7fb744ea4640 1 -- 172.21.15.16:0/1246655427 wait complete. 2024-03-28T22:08:28.327 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-28T22:08:28.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.457+0000 7f7bb2055640 1 -- 172.21.15.16:0/3369134538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bac102a00 msgr2=0x7f7bac102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:28.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.457+0000 7f7bb2055640 1 --2- 172.21.15.16:0/3369134538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bac102a00 0x7f7bac102e80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0009930 tx=0x7f7ba002bc30 comp rx=0 tx=0).stop 2024-03-28T22:08:28.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.457+0000 7f7bb2055640 1 -- 172.21.15.16:0/3369134538 shutdown_connections 2024-03-28T22:08:28.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.457+0000 7f7bb2055640 1 --2- 172.21.15.16:0/3369134538 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7bac1033c0 0x7f7bac109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:28.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.457+0000 7f7bb2055640 1 --2- 172.21.15.16:0/3369134538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bac102a00 0x7f7bac102e80 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:28.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.457+0000 7f7bb2055640 1 --2- 172.21.15.16:0/3369134538 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7bac0ffa30 0x7f7bac0ffe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:28.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.457+0000 7f7bb2055640 1 -- 172.21.15.16:0/3369134538 >> 172.21.15.16:0/3369134538 conn(0x7f7bac0fb360 msgr2=0x7f7bac0fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:28.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.457+0000 7f7bb2055640 1 -- 172.21.15.16:0/3369134538 shutdown_connections 2024-03-28T22:08:28.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.457+0000 7f7bb2055640 1 -- 172.21.15.16:0/3369134538 wait complete. 2024-03-28T22:08:28.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.458+0000 7f7bb2055640 1 Processor -- start 2024-03-28T22:08:28.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.458+0000 7f7bb2055640 1 -- start start 2024-03-28T22:08:28.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.458+0000 7f7bb2055640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7bac0ffa30 0x7f7bac197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:28.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.458+0000 7f7bb2055640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bac102a00 0x7f7bac197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:28.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.458+0000 7f7bb2055640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7bac1033c0 0x7f7bac19f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:28.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.458+0000 7f7bb2055640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7bac10d3f0 con 0x7f7bac102a00 2024-03-28T22:08:28.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.458+0000 7f7bb2055640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7bac10d270 con 0x7f7bac0ffa30 2024-03-28T22:08:28.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.458+0000 7f7bb2055640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7bac10d570 con 0x7f7bac1033c0 2024-03-28T22:08:28.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.458+0000 7f7baaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bac102a00 0x7f7bac197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:28.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.459+0000 7f7baaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bac102a00 0x7f7bac197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53294/0 (socket says 172.21.15.16:53294) 2024-03-28T22:08:28.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.459+0000 7f7baaffd640 1 -- 172.21.15.16:0/174983689 learned_addr learned my addr 172.21.15.16:0/174983689 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:28.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.459+0000 7f7bab7fe640 1 --2- 172.21.15.16:0/174983689 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7bac0ffa30 0x7f7bac197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:28.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.459+0000 7f7babfff640 1 --2- 172.21.15.16:0/174983689 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7bac1033c0 0x7f7bac19f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:28.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.459+0000 7f7baaffd640 1 -- 172.21.15.16:0/174983689 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7bac1033c0 msgr2=0x7f7bac19f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:28.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.459+0000 7f7baaffd640 1 --2- 172.21.15.16:0/174983689 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7bac1033c0 0x7f7bac19f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:28.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.459+0000 7f7baaffd640 1 -- 172.21.15.16:0/174983689 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7bac0ffa30 msgr2=0x7f7bac197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:28.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.459+0000 7f7baaffd640 1 --2- 172.21.15.16:0/174983689 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7bac0ffa30 0x7f7bac197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:28.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.459+0000 7f7baaffd640 1 -- 172.21.15.16:0/174983689 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bac19f6d0 con 0x7f7bac102a00 2024-03-28T22:08:28.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.459+0000 7f7baaffd640 1 --2- 172.21.15.16:0/174983689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bac102a00 0x7f7bac197ff0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0000c00 tx=0x7f7ba002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:28.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.459+0000 7f7ba8ff9640 1 -- 172.21.15.16:0/174983689 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba0041070 con 0x7f7bac102a00 2024-03-28T22:08:28.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.459+0000 7f7ba8ff9640 1 -- 172.21.15.16:0/174983689 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ba0004450 con 0x7f7bac102a00 2024-03-28T22:08:28.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.459+0000 7f7bb2055640 1 -- 172.21.15.16:0/174983689 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bac19f960 con 0x7f7bac102a00 2024-03-28T22:08:28.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.459+0000 7f7ba8ff9640 1 -- 172.21.15.16:0/174983689 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba0034440 con 0x7f7bac102a00 2024-03-28T22:08:28.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.459+0000 7f7bb2055640 1 -- 172.21.15.16:0/174983689 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bac19fe50 con 0x7f7bac102a00 2024-03-28T22:08:28.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.460+0000 7f7bb2055640 1 -- 172.21.15.16:0/174983689 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bac108c00 con 0x7f7bac102a00 2024-03-28T22:08:28.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.462+0000 7f7ba8ff9640 1 -- 172.21.15.16:0/174983689 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f7ba0044070 con 0x7f7bac102a00 2024-03-28T22:08:28.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.462+0000 7f7ba8ff9640 1 --2- 172.21.15.16:0/174983689 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f7b940a1d80 0x7f7b940a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:28.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.462+0000 7f7ba8ff9640 1 -- 172.21.15.16:0/174983689 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7497+0+0 (secure 0 0 0) 0x7f7ba00ef890 con 0x7f7bac102a00 2024-03-28T22:08:28.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.462+0000 7f7bab7fe640 1 --2- 172.21.15.16:0/174983689 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f7b940a1d80 0x7f7b940a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:28.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.463+0000 7f7bab7fe640 1 --2- 172.21.15.16:0/174983689 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f7b940a1d80 0x7f7b940a4240 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7bac068820 tx=0x7f7b9c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:28.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.468+0000 7f7ba8ff9640 1 -- 172.21.15.16:0/174983689 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7ba00bc9a0 con 0x7f7bac102a00 2024-03-28T22:08:28.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:28.669+0000 7f7bb2055640 1 -- 172.21.15.16:0/174983689 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f7bac02d290 con 0x7f7bac102a00 2024-03-28T22:08:29.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.309+0000 7f7ba8ff9640 1 -- 172.21.15.16:0/174983689 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v84) ==== 170+0+0 (secure 0 0 0) 0x7f7ba00c0a40 con 0x7f7bac102a00 2024-03-28T22:08:29.312 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2024-03-28T22:08:29.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.311+0000 7f7bb2055640 1 -- 172.21.15.16:0/174983689 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f7b940a1d80 msgr2=0x7f7b940a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:29.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.311+0000 7f7bb2055640 1 --2- 172.21.15.16:0/174983689 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f7b940a1d80 0x7f7b940a4240 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7bac068820 tx=0x7f7b9c005f90 comp rx=0 tx=0).stop 2024-03-28T22:08:29.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.311+0000 7f7bb2055640 1 -- 172.21.15.16:0/174983689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bac102a00 msgr2=0x7f7bac197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:29.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.311+0000 7f7bb2055640 1 --2- 172.21.15.16:0/174983689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bac102a00 0x7f7bac197ff0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0000c00 tx=0x7f7ba002c650 comp rx=0 tx=0).stop 2024-03-28T22:08:29.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.312+0000 7f7bb2055640 1 -- 172.21.15.16:0/174983689 shutdown_connections 2024-03-28T22:08:29.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.312+0000 7f7bb2055640 1 --2- 172.21.15.16:0/174983689 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f7b940a1d80 0x7f7b940a4240 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:29.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.312+0000 7f7bb2055640 1 --2- 172.21.15.16:0/174983689 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7bac1033c0 0x7f7bac19f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:29.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.312+0000 7f7bb2055640 1 --2- 172.21.15.16:0/174983689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bac102a00 0x7f7bac197ff0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:29.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.312+0000 7f7bb2055640 1 --2- 172.21.15.16:0/174983689 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7bac0ffa30 0x7f7bac197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:29.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.312+0000 7f7bb2055640 1 -- 172.21.15.16:0/174983689 >> 172.21.15.16:0/174983689 conn(0x7f7bac0fb360 msgr2=0x7f7bac105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:29.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.312+0000 7f7bb2055640 1 -- 172.21.15.16:0/174983689 shutdown_connections 2024-03-28T22:08:29.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.312+0000 7f7bb2055640 1 -- 172.21.15.16:0/174983689 wait complete. 2024-03-28T22:08:29.335 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-28T22:08:29.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.439+0000 7f3efce15640 1 -- 172.21.15.16:0/1348712084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ef8100660 msgr2=0x7f3ef8100a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:29.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.439+0000 7f3efce15640 1 --2- 172.21.15.16:0/1348712084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ef8100660 0x7f3ef8100a80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3ee8009900 tx=0x7f3ee802bc30 comp rx=0 tx=0).stop 2024-03-28T22:08:29.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.439+0000 7f3efce15640 1 -- 172.21.15.16:0/1348712084 shutdown_connections 2024-03-28T22:08:29.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.439+0000 7f3efce15640 1 --2- 172.21.15.16:0/1348712084 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ef8102210 0x7f3ef8108aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:29.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.439+0000 7f3efce15640 1 --2- 172.21.15.16:0/1348712084 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ef8101850 0x7f3ef8101cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:29.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.439+0000 7f3efce15640 1 --2- 172.21.15.16:0/1348712084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ef8100660 0x7f3ef8100a80 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:29.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.439+0000 7f3efce15640 1 -- 172.21.15.16:0/1348712084 >> 172.21.15.16:0/1348712084 conn(0x7f3ef80fbbb0 msgr2=0x7f3ef80fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:29.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.439+0000 7f3efce15640 1 -- 172.21.15.16:0/1348712084 shutdown_connections 2024-03-28T22:08:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.439+0000 7f3efce15640 1 -- 172.21.15.16:0/1348712084 wait complete. 2024-03-28T22:08:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.440+0000 7f3efce15640 1 Processor -- start 2024-03-28T22:08:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.440+0000 7f3efce15640 1 -- start start 2024-03-28T22:08:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.440+0000 7f3efce15640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ef8100660 0x7f3ef8195890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.440+0000 7f3efce15640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ef8101850 0x7f3ef8195dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.440+0000 7f3efce15640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ef8102210 0x7f3ef819ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.440+0000 7f3efce15640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef810b1e0 con 0x7f3ef8100660 2024-03-28T22:08:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.440+0000 7f3efce15640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef810b060 con 0x7f3ef8101850 2024-03-28T22:08:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.440+0000 7f3efce15640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef810b360 con 0x7f3ef8102210 2024-03-28T22:08:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.440+0000 7f3ef659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ef8100660 0x7f3ef8195890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.440+0000 7f3ef659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ef8100660 0x7f3ef8195890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53312/0 (socket says 172.21.15.16:53312) 2024-03-28T22:08:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.440+0000 7f3ef659c640 1 -- 172.21.15.16:0/1943845534 learned_addr learned my addr 172.21.15.16:0/1943845534 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:29.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.440+0000 7f3ef5d9b640 1 --2- 172.21.15.16:0/1943845534 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ef8101850 0x7f3ef8195dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:29.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.440+0000 7f3ef6d9d640 1 --2- 172.21.15.16:0/1943845534 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ef8102210 0x7f3ef819ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:29.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.440+0000 7f3ef659c640 1 -- 172.21.15.16:0/1943845534 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ef8102210 msgr2=0x7f3ef819ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:29.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.440+0000 7f3ef659c640 1 --2- 172.21.15.16:0/1943845534 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ef8102210 0x7f3ef819ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:29.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.440+0000 7f3ef659c640 1 -- 172.21.15.16:0/1943845534 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ef8101850 msgr2=0x7f3ef8195dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:29.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.440+0000 7f3ef659c640 1 --2- 172.21.15.16:0/1943845534 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ef8101850 0x7f3ef8195dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:29.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.440+0000 7f3ef659c640 1 -- 172.21.15.16:0/1943845534 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ef819d4b0 con 0x7f3ef8100660 2024-03-28T22:08:29.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.440+0000 7f3ef6d9d640 1 --2- 172.21.15.16:0/1943845534 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ef8102210 0x7f3ef819ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:08:29.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.441+0000 7f3ef659c640 1 --2- 172.21.15.16:0/1943845534 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ef8100660 0x7f3ef8195890 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3ee8000c00 tx=0x7f3ee802c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:29.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.441+0000 7f3ee77fe640 1 -- 172.21.15.16:0/1943845534 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee8041070 con 0x7f3ef8100660 2024-03-28T22:08:29.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.441+0000 7f3ee77fe640 1 -- 172.21.15.16:0/1943845534 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ee8004450 con 0x7f3ef8100660 2024-03-28T22:08:29.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.441+0000 7f3efce15640 1 -- 172.21.15.16:0/1943845534 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ef819d740 con 0x7f3ef8100660 2024-03-28T22:08:29.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.441+0000 7f3ee77fe640 1 -- 172.21.15.16:0/1943845534 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee8034440 con 0x7f3ef8100660 2024-03-28T22:08:29.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.441+0000 7f3efce15640 1 -- 172.21.15.16:0/1943845534 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ef819dc30 con 0x7f3ef8100660 2024-03-28T22:08:29.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.442+0000 7f3efce15640 1 -- 172.21.15.16:0/1943845534 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ef8107a00 con 0x7f3ef8100660 2024-03-28T22:08:29.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.443+0000 7f3ee77fe640 1 -- 172.21.15.16:0/1943845534 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f3ee8044070 con 0x7f3ef8100660 2024-03-28T22:08:29.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.443+0000 7f3ee77fe640 1 --2- 172.21.15.16:0/1943845534 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f3ee00a1d80 0x7f3ee00a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:29.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.444+0000 7f3ef5d9b640 1 --2- 172.21.15.16:0/1943845534 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f3ee00a1d80 0x7f3ee00a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:29.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.444+0000 7f3ee77fe640 1 -- 172.21.15.16:0/1943845534 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7879+0+0 (secure 0 0 0) 0x7f3ee80ef7d0 con 0x7f3ef8100660 2024-03-28T22:08:29.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.444+0000 7f3ef5d9b640 1 --2- 172.21.15.16:0/1943845534 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f3ee00a1d80 0x7f3ee00a4240 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3ef8196da0 tx=0x7f3eec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:29.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.448+0000 7f3ee77fe640 1 -- 172.21.15.16:0/1943845534 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3ee80bce40 con 0x7f3ef8100660 2024-03-28T22:08:29.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:29.670+0000 7f3efce15640 1 -- 172.21.15.16:0/1943845534 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f3ef802d170 con 0x7f3ef8100660 2024-03-28T22:08:29.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:29 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/1246655427' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-28T22:08:29.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:29 smithi063 ceph-mon[36249]: osdmap e83: 12 total, 12 up, 12 in 2024-03-28T22:08:29.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:29 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:29.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:29 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:29.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:29 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/174983689' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T22:08:29.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:29 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:29.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:29 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:29.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:29 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:29.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:29 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:29.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:29 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:08:29.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:29 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:08:29.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:29 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:29.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:29 smithi063 ceph-mon[36249]: pgmap v283: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:29.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:29 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:29.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:29 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/1246655427' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-28T22:08:29.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:29 smithi016 ceph-mon[32374]: osdmap e83: 12 total, 12 up, 12 in 2024-03-28T22:08:29.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:29 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:29.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:29 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:29.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:29 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/174983689' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T22:08:29.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:29 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:29.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:29 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:29.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:29 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:29.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:29 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:29.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:29 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:08:29.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:29 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:08:29.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:29 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:29.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:29 smithi016 ceph-mon[32374]: pgmap v283: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:29.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:29 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:29.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:29 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/1246655427' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-28T22:08:29.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:29 smithi053 ceph-mon[35979]: osdmap e83: 12 total, 12 up, 12 in 2024-03-28T22:08:29.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:29 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:29.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:29 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:29.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:29 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/174983689' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T22:08:29.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:29 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:29.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:29 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:29.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:29 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:29.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:29 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:29.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:29 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:08:29.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:29 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:08:29.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:29 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:29.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:29 smithi053 ceph-mon[35979]: pgmap v283: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:08:29.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:29 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:08:30.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.311+0000 7f3ee77fe640 1 -- 172.21.15.16:0/1943845534 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f3ee80dae20 con 0x7f3ef8100660 2024-03-28T22:08:30.314 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 2 and data pool 3 2024-03-28T22:08:30.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.314+0000 7f3efce15640 1 -- 172.21.15.16:0/1943845534 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f3ee00a1d80 msgr2=0x7f3ee00a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:30.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.314+0000 7f3efce15640 1 --2- 172.21.15.16:0/1943845534 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f3ee00a1d80 0x7f3ee00a4240 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3ef8196da0 tx=0x7f3eec005f90 comp rx=0 tx=0).stop 2024-03-28T22:08:30.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.314+0000 7f3efce15640 1 -- 172.21.15.16:0/1943845534 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ef8100660 msgr2=0x7f3ef8195890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:30.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.314+0000 7f3efce15640 1 --2- 172.21.15.16:0/1943845534 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ef8100660 0x7f3ef8195890 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3ee8000c00 tx=0x7f3ee802c640 comp rx=0 tx=0).stop 2024-03-28T22:08:30.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.314+0000 7f3efce15640 1 -- 172.21.15.16:0/1943845534 shutdown_connections 2024-03-28T22:08:30.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.314+0000 7f3efce15640 1 --2- 172.21.15.16:0/1943845534 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f3ee00a1d80 0x7f3ee00a4240 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:30.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.314+0000 7f3efce15640 1 --2- 172.21.15.16:0/1943845534 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ef8102210 0x7f3ef819ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:30.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.314+0000 7f3efce15640 1 --2- 172.21.15.16:0/1943845534 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ef8101850 0x7f3ef8195dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:30.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.314+0000 7f3efce15640 1 --2- 172.21.15.16:0/1943845534 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ef8100660 0x7f3ef8195890 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:30.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.314+0000 7f3efce15640 1 -- 172.21.15.16:0/1943845534 >> 172.21.15.16:0/1943845534 conn(0x7f3ef80fbbb0 msgr2=0x7f3ef8104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:30.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.314+0000 7f3efce15640 1 -- 172.21.15.16:0/1943845534 shutdown_connections 2024-03-28T22:08:30.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.314+0000 7f3efce15640 1 -- 172.21.15.16:0/1943845534 wait complete. 2024-03-28T22:08:30.334 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:08:30.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.444+0000 7fc54190d640 1 -- 172.21.15.16:0/1262183875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc53c102230 msgr2=0x7fc53c108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:30.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.444+0000 7fc54190d640 1 --2- 172.21.15.16:0/1262183875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc53c102230 0x7fc53c108ac0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc52c009ca0 tx=0x7fc52c02c0f0 comp rx=0 tx=0).stop 2024-03-28T22:08:30.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.444+0000 7fc54190d640 1 -- 172.21.15.16:0/1262183875 shutdown_connections 2024-03-28T22:08:30.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.444+0000 7fc54190d640 1 --2- 172.21.15.16:0/1262183875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc53c102230 0x7fc53c108ac0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:30.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.444+0000 7fc54190d640 1 --2- 172.21.15.16:0/1262183875 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc53c101870 0x7fc53c101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:30.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.444+0000 7fc54190d640 1 --2- 172.21.15.16:0/1262183875 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc53c100680 0x7fc53c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:30.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.444+0000 7fc54190d640 1 -- 172.21.15.16:0/1262183875 >> 172.21.15.16:0/1262183875 conn(0x7fc53c0fbbf0 msgr2=0x7fc53c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:30.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.444+0000 7fc54190d640 1 -- 172.21.15.16:0/1262183875 shutdown_connections 2024-03-28T22:08:30.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.444+0000 7fc54190d640 1 -- 172.21.15.16:0/1262183875 wait complete. 2024-03-28T22:08:30.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.445+0000 7fc54190d640 1 Processor -- start 2024-03-28T22:08:30.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.445+0000 7fc54190d640 1 -- start start 2024-03-28T22:08:30.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.445+0000 7fc54190d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc53c100680 0x7fc53c104610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:30.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.445+0000 7fc54190d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc53c101870 0x7fc53c102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:30.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.445+0000 7fc54190d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc53c102230 0x7fc53c103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:30.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.445+0000 7fc54190d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc53c10c300 con 0x7fc53c102230 2024-03-28T22:08:30.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.445+0000 7fc54190d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc53c10c180 con 0x7fc53c100680 2024-03-28T22:08:30.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.445+0000 7fc54190d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc53c10c480 con 0x7fc53c101870 2024-03-28T22:08:30.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.445+0000 7fc53b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc53c102230 0x7fc53c103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:30.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.445+0000 7fc53b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc53c102230 0x7fc53c103190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53330/0 (socket says 172.21.15.16:53330) 2024-03-28T22:08:30.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.445+0000 7fc53b7fe640 1 -- 172.21.15.16:0/3829275212 learned_addr learned my addr 172.21.15.16:0/3829275212 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:30.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.446+0000 7fc53a7fc640 1 --2- 172.21.15.16:0/3829275212 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc53c101870 0x7fc53c102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:30.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.446+0000 7fc53b7fe640 1 -- 172.21.15.16:0/3829275212 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc53c101870 msgr2=0x7fc53c102c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:30.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.446+0000 7fc53affd640 1 --2- 172.21.15.16:0/3829275212 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc53c100680 0x7fc53c104610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:30.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.446+0000 7fc53b7fe640 1 --2- 172.21.15.16:0/3829275212 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc53c101870 0x7fc53c102c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:30.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.446+0000 7fc53b7fe640 1 -- 172.21.15.16:0/3829275212 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc53c100680 msgr2=0x7fc53c104610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:30.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.446+0000 7fc53b7fe640 1 --2- 172.21.15.16:0/3829275212 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc53c100680 0x7fc53c104610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:30.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.446+0000 7fc53b7fe640 1 -- 172.21.15.16:0/3829275212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc53c103a20 con 0x7fc53c102230 2024-03-28T22:08:30.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.446+0000 7fc53b7fe640 1 --2- 172.21.15.16:0/3829275212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc53c102230 0x7fc53c103190 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc52c02c700 tx=0x7fc52c032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:30.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.446+0000 7fc54090b640 1 -- 172.21.15.16:0/3829275212 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc52c039040 con 0x7fc53c102230 2024-03-28T22:08:30.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.446+0000 7fc54090b640 1 -- 172.21.15.16:0/3829275212 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc52c044070 con 0x7fc53c102230 2024-03-28T22:08:30.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.446+0000 7fc54190d640 1 -- 172.21.15.16:0/3829275212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc53c1a1b20 con 0x7fc53c102230 2024-03-28T22:08:30.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.446+0000 7fc54090b640 1 -- 172.21.15.16:0/3829275212 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc52c0042a0 con 0x7fc53c102230 2024-03-28T22:08:30.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.446+0000 7fc54190d640 1 -- 172.21.15.16:0/3829275212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc53c1a20c0 con 0x7fc53c102230 2024-03-28T22:08:30.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.447+0000 7fc54190d640 1 -- 172.21.15.16:0/3829275212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc53c1a2290 con 0x7fc53c102230 2024-03-28T22:08:30.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.448+0000 7fc54090b640 1 -- 172.21.15.16:0/3829275212 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fc52c003970 con 0x7fc53c102230 2024-03-28T22:08:30.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.449+0000 7fc54090b640 1 --2- 172.21.15.16:0/3829275212 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc5280a1e00 0x7fc5280a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:30.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.449+0000 7fc53affd640 1 --2- 172.21.15.16:0/3829275212 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc5280a1e00 0x7fc5280a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:30.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.449+0000 7fc54090b640 1 -- 172.21.15.16:0/3829275212 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fc52c0f0190 con 0x7fc53c102230 2024-03-28T22:08:30.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.449+0000 7fc53affd640 1 --2- 172.21.15.16:0/3829275212 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc5280a1e00 0x7fc5280a42c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc534005fd0 tx=0x7fc534005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:30.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.453+0000 7fc54090b640 1 -- 172.21.15.16:0/3829275212 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc52c0333a0 con 0x7fc53c102230 2024-03-28T22:08:30.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.636+0000 7fc54190d640 1 -- 172.21.15.16:0/3829275212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc53c1a2580 con 0x7fc53c102230 2024-03-28T22:08:30.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.638+0000 7fc54090b640 1 -- 172.21.15.16:0/3829275212 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22124 (secure 0 0 0) 0x7fc52c0bd7a0 con 0x7fc53c102230 2024-03-28T22:08:30.641 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:08:30.641 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":85,"fsid":"798d9884-ed4e-11ee-95d0-87774f69a715","created":"2024-03-28T22:00:27.216099+0000","modified":"2024-03-28T22:08:29.672623+0000","last_up_change":"2024-03-28T22:07:40.251141+0000","last_in_change":"2024-03-28T22:07:19.456451+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:03:59.327832+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:08:28.123529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:08:28.919037+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"46e7c8f8-ecc8-43d6-ae53-0655728f3240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6803","nonce":1589692483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6805","nonce":1589692483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6809","nonce":1589692483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6807","nonce":1589692483}]},"public_addr":"172.21.15.16:6803/1589692483","cluster_addr":"172.21.15.16:6805/1589692483","heartbeat_back_addr":"172.21.15.16:6809/1589692483","heartbeat_front_addr":"172.21.15.16:6807/1589692483","state":["exists","up"]},{"osd":1,"uuid":"af6d9c22-2830-4ce2-a547-011df0c6cbad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6801","nonce":795531234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6803","nonce":795531234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6807","nonce":795531234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6805","nonce":795531234}]},"public_addr":"172.21.15.53:6801/795531234","cluster_addr":"172.21.15.53:6803/795531234","heartbeat_back_addr":"172.21.15.53:6807/795531234","heartbeat_front_addr":"172.21.15.53:6805/795531234","state":["exists","up"]},{"osd":2,"uuid":"b10adca9-110c-4f87-85e0-b7542709d31a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6801","nonce":2214131086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6803","nonce":2214131086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6807","nonce":2214131086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6805","nonce":2214131086}]},"public_addr":"172.21.15.63:6801/2214131086","cluster_addr":"172.21.15.63:6803/2214131086","heartbeat_back_addr":"172.21.15.63:6807/2214131086","heartbeat_front_addr":"172.21.15.63:6805/2214131086","state":["exists","up"]},{"osd":3,"uuid":"c83cf4bb-395a-4786-abf7-74ca93a33eb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6811","nonce":13176789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6813","nonce":13176789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6817","nonce":13176789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6815","nonce":13176789}]},"public_addr":"172.21.15.16:6811/13176789","cluster_addr":"172.21.15.16:6813/13176789","heartbeat_back_addr":"172.21.15.16:6817/13176789","heartbeat_front_addr":"172.21.15.16:6815/13176789","state":["exists","up"]},{"osd":4,"uuid":"941b3e54-47d6-4e85-8069-e730c3fa8eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6809","nonce":772161849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6811","nonce":772161849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6815","nonce":772161849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6813","nonce":772161849}]},"public_addr":"172.21.15.53:6809/772161849","cluster_addr":"172.21.15.53:6811/772161849","heartbeat_back_addr":"172.21.15.53:6815/772161849","heartbeat_front_addr":"172.21.15.53:6813/772161849","state":["exists","up"]},{"osd":5,"uuid":"51d89826-1d60-40fc-8fd8-09bfe1ab4125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6809","nonce":1602554852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6811","nonce":1602554852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6815","nonce":1602554852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6813","nonce":1602554852}]},"public_addr":"172.21.15.63:6809/1602554852","cluster_addr":"172.21.15.63:6811/1602554852","heartbeat_back_addr":"172.21.15.63:6815/1602554852","heartbeat_front_addr":"172.21.15.63:6813/1602554852","state":["exists","up"]},{"osd":6,"uuid":"978efe5d-2c55-4ff8-8a1e-9f891388945e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6819","nonce":4290459224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6821","nonce":4290459224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6825","nonce":4290459224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6823","nonce":4290459224}]},"public_addr":"172.21.15.16:6819/4290459224","cluster_addr":"172.21.15.16:6821/4290459224","heartbeat_back_addr":"172.21.15.16:6825/4290459224","heartbeat_front_addr":"172.21.15.16:6823/4290459224","state":["exists","up"]},{"osd":7,"uuid":"50d489cb-dbb8-48df-b868-b29edf12e04d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6817","nonce":1056467525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6819","nonce":1056467525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6823","nonce":1056467525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6821","nonce":1056467525}]},"public_addr":"172.21.15.53:6817/1056467525","cluster_addr":"172.21.15.53:6819/1056467525","heartbeat_back_addr":"172.21.15.53:6823/1056467525","heartbeat_front_addr":"172.21.15.53:6821/1056467525","state":["exists","up"]},{"osd":8,"uuid":"49b39414-cc9c-4057-ae3f-cc109cd979da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6817","nonce":2918516794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6819","nonce":2918516794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6823","nonce":2918516794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6821","nonce":2918516794}]},"public_addr":"172.21.15.63:6817/2918516794","cluster_addr":"172.21.15.63:6819/2918516794","heartbeat_back_addr":"172.21.15.63:6823/2918516794","heartbeat_front_addr":"172.21.15.63:6821/2918516794","state":["exists","up"]},{"osd":9,"uuid":"4c873e51-b783-4eb6-83a3-fee31722d60b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6827","nonce":1585624367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6829","nonce":1585624367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6833","nonce":1585624367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6831","nonce":1585624367}]},"public_addr":"172.21.15.16:6827/1585624367","cluster_addr":"172.21.15.16:6829/1585624367","heartbeat_back_addr":"172.21.15.16:6833/1585624367","heartbeat_front_addr":"172.21.15.16:6831/1585624367","state":["exists","up"]},{"osd":10,"uuid":"085a78bf-e2ad-481a-85a3-71d1fc549a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6825","nonce":3357446484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6827","nonce":3357446484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6831","nonce":3357446484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6829","nonce":3357446484}]},"public_addr":"172.21.15.53:6825/3357446484","cluster_addr":"172.21.15.53:6827/3357446484","heartbeat_back_addr":"172.21.15.53:6831/3357446484","heartbeat_front_addr":"172.21.15.53:6829/3357446484","state":["exists","up"]},{"osd":11,"uuid":"13302aa3-08b9-4639-9042-27d850b4d5f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6825","nonce":2167866570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6827","nonce":2167866570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6831","nonce":2167866570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6829","nonce":2167866570}]},"public_addr":"172.21.15.63:6825/2167866570","cluster_addr":"172.21.15.63:6827/2167866570","heartbeat_back_addr":"172.21.15.63:6831/2167866570","heartbeat_front_addr":"172.21.15.63:6829/2167866570","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:07.450323+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:31.363172+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:55.295021+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:19.794760+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:44.063492+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:08.195064+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:33.655328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:57.639414+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:22.402394+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:47.462050+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:12.226307+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:37.495625+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3821408150":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6801/1832422779":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/996598130":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/1062264551":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/317557779":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6801/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/3631746518":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6800/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/3116652786":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6800/1832422779":"2024-03-29T22:01:28.993434+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:08:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.640+0000 7fc54190d640 1 -- 172.21.15.16:0/3829275212 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc5280a1e00 msgr2=0x7fc5280a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.640+0000 7fc54190d640 1 --2- 172.21.15.16:0/3829275212 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc5280a1e00 0x7fc5280a42c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc534005fd0 tx=0x7fc534005c70 comp rx=0 tx=0).stop 2024-03-28T22:08:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.640+0000 7fc54190d640 1 -- 172.21.15.16:0/3829275212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc53c102230 msgr2=0x7fc53c103190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.640+0000 7fc54190d640 1 --2- 172.21.15.16:0/3829275212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc53c102230 0x7fc53c103190 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc52c02c700 tx=0x7fc52c032f20 comp rx=0 tx=0).stop 2024-03-28T22:08:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.640+0000 7fc54190d640 1 -- 172.21.15.16:0/3829275212 shutdown_connections 2024-03-28T22:08:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.640+0000 7fc54190d640 1 --2- 172.21.15.16:0/3829275212 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc5280a1e00 0x7fc5280a42c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.640+0000 7fc54190d640 1 --2- 172.21.15.16:0/3829275212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc53c102230 0x7fc53c103190 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.640+0000 7fc54190d640 1 --2- 172.21.15.16:0/3829275212 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc53c101870 0x7fc53c102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.640+0000 7fc54190d640 1 --2- 172.21.15.16:0/3829275212 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc53c100680 0x7fc53c104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.640+0000 7fc54190d640 1 -- 172.21.15.16:0/3829275212 >> 172.21.15.16:0/3829275212 conn(0x7fc53c0fbbf0 msgr2=0x7fc53c0fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.640+0000 7fc54190d640 1 -- 172.21.15.16:0/3829275212 shutdown_connections 2024-03-28T22:08:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.640+0000 7fc54190d640 1 -- 172.21.15.16:0/3829275212 wait complete. 2024-03-28T22:08:30.652 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:30 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/174983689' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-28T22:08:30.652 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:30 smithi016 ceph-mon[32374]: osdmap e84: 12 total, 12 up, 12 in 2024-03-28T22:08:30.652 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:30 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/1943845534' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T22:08:30.652 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:30 smithi016 ceph-mon[32374]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-28T22:08:30.653 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:30 smithi016 ceph-mon[32374]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T22:08:30.653 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:30 smithi016 ceph-mon[32374]: osdmap e85: 12 total, 12 up, 12 in 2024-03-28T22:08:30.654 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:08:30.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:30 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/174983689' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-28T22:08:30.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:30 smithi063 ceph-mon[36249]: osdmap e84: 12 total, 12 up, 12 in 2024-03-28T22:08:30.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:30 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/1943845534' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T22:08:30.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:30 smithi063 ceph-mon[36249]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-28T22:08:30.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:30 smithi063 ceph-mon[36249]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T22:08:30.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:30 smithi063 ceph-mon[36249]: osdmap e85: 12 total, 12 up, 12 in 2024-03-28T22:08:30.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.754+0000 7f8efc69a640 1 -- 172.21.15.16:0/1849250219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ef4103330 msgr2=0x7f8ef4109bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:30.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.754+0000 7f8efc69a640 1 --2- 172.21.15.16:0/1849250219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ef4103330 0x7f8ef4109bc0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8ef0009ca0 tx=0x7f8ef002c0f0 comp rx=0 tx=0).stop 2024-03-28T22:08:30.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.754+0000 7f8efc69a640 1 -- 172.21.15.16:0/1849250219 shutdown_connections 2024-03-28T22:08:30.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.754+0000 7f8efc69a640 1 --2- 172.21.15.16:0/1849250219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ef4103330 0x7f8ef4109bc0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:30.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.754+0000 7f8efc69a640 1 --2- 172.21.15.16:0/1849250219 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8ef4102970 0x7f8ef4102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:30.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.754+0000 7f8efc69a640 1 --2- 172.21.15.16:0/1849250219 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ef4101780 0x7f8ef4101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:30.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.754+0000 7f8efc69a640 1 -- 172.21.15.16:0/1849250219 >> 172.21.15.16:0/1849250219 conn(0x7f8ef40fcd30 msgr2=0x7f8ef40ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:30.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.754+0000 7f8efc69a640 1 -- 172.21.15.16:0/1849250219 shutdown_connections 2024-03-28T22:08:30.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.754+0000 7f8efc69a640 1 -- 172.21.15.16:0/1849250219 wait complete. 2024-03-28T22:08:30.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.754+0000 7f8efc69a640 1 Processor -- start 2024-03-28T22:08:30.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.754+0000 7f8efc69a640 1 -- start start 2024-03-28T22:08:30.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.755+0000 7f8efc69a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8ef4101780 0x7f8ef4075500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:30.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.755+0000 7f8efc69a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ef4102970 0x7f8ef4073b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:30.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.755+0000 7f8efc69a640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ef4103330 0x7f8ef4074080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:30.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.755+0000 7f8efc69a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef410d400 con 0x7f8ef4102970 2024-03-28T22:08:30.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.755+0000 7f8efc69a640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef410d280 con 0x7f8ef4103330 2024-03-28T22:08:30.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.755+0000 7f8efc69a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef410d580 con 0x7f8ef4101780 2024-03-28T22:08:30.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.755+0000 7f8ef9c35640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ef4102970 0x7f8ef4073b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:30.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.755+0000 7f8ef9c35640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ef4102970 0x7f8ef4073b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57440/0 (socket says 172.21.15.16:57440) 2024-03-28T22:08:30.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.755+0000 7f8ef9c35640 1 -- 172.21.15.16:0/2748596090 learned_addr learned my addr 172.21.15.16:0/2748596090 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:30.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.755+0000 7f8efa436640 1 --2- 172.21.15.16:0/2748596090 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8ef4101780 0x7f8ef4075500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:30.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.755+0000 7f8efac37640 1 --2- 172.21.15.16:0/2748596090 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ef4103330 0x7f8ef4074080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:30.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.755+0000 7f8ef9c35640 1 -- 172.21.15.16:0/2748596090 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8ef4101780 msgr2=0x7f8ef4075500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:30.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.755+0000 7f8ef9c35640 1 --2- 172.21.15.16:0/2748596090 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8ef4101780 0x7f8ef4075500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:30.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.755+0000 7f8ef9c35640 1 -- 172.21.15.16:0/2748596090 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ef4103330 msgr2=0x7f8ef4074080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:30.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.755+0000 7f8ef9c35640 1 --2- 172.21.15.16:0/2748596090 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ef4103330 0x7f8ef4074080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:30.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.755+0000 7f8ef9c35640 1 -- 172.21.15.16:0/2748596090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ef4074910 con 0x7f8ef4102970 2024-03-28T22:08:30.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.755+0000 7f8ef9c35640 1 --2- 172.21.15.16:0/2748596090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ef4102970 0x7f8ef4073b40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8ee400e9f0 tx=0x7f8ee400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:30.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.755+0000 7f8eeb7fe640 1 -- 172.21.15.16:0/2748596090 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee400cd80 con 0x7f8ef4102970 2024-03-28T22:08:30.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.755+0000 7f8eeb7fe640 1 -- 172.21.15.16:0/2748596090 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ee4005500 con 0x7f8ef4102970 2024-03-28T22:08:30.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.755+0000 7f8efc69a640 1 -- 172.21.15.16:0/2748596090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ef41a1b30 con 0x7f8ef4102970 2024-03-28T22:08:30.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.755+0000 7f8eeb7fe640 1 -- 172.21.15.16:0/2748596090 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee4010690 con 0x7f8ef4102970 2024-03-28T22:08:30.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.755+0000 7f8efc69a640 1 -- 172.21.15.16:0/2748596090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ef41a2080 con 0x7f8ef4102970 2024-03-28T22:08:30.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.756+0000 7f8efc69a640 1 -- 172.21.15.16:0/2748596090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ef4108a40 con 0x7f8ef4102970 2024-03-28T22:08:30.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.758+0000 7f8eeb7fe640 1 -- 172.21.15.16:0/2748596090 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f8ee40046c0 con 0x7f8ef4102970 2024-03-28T22:08:30.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.758+0000 7f8eeb7fe640 1 --2- 172.21.15.16:0/2748596090 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f8ee00a1e00 0x7f8ee00a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:30.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.758+0000 7f8efa436640 1 --2- 172.21.15.16:0/2748596090 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f8ee00a1e00 0x7f8ee00a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:30.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.758+0000 7f8eeb7fe640 1 -- 172.21.15.16:0/2748596090 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f8ee40ce210 con 0x7f8ef4102970 2024-03-28T22:08:30.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.759+0000 7f8efa436640 1 --2- 172.21.15.16:0/2748596090 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f8ee00a1e00 0x7f8ee00a42c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8ef41027d0 tx=0x7f8eec00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:30.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.762+0000 7f8eeb7fe640 1 -- 172.21.15.16:0/2748596090 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8ee4053280 con 0x7f8ef4102970 2024-03-28T22:08:30.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:30 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/174983689' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-28T22:08:30.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:30 smithi053 ceph-mon[35979]: osdmap e84: 12 total, 12 up, 12 in 2024-03-28T22:08:30.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:30 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/1943845534' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T22:08:30.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:30 smithi053 ceph-mon[35979]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-28T22:08:30.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:30 smithi053 ceph-mon[35979]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T22:08:30.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:30 smithi053 ceph-mon[35979]: osdmap e85: 12 total, 12 up, 12 in 2024-03-28T22:08:30.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.948+0000 7f8efc69a640 1 -- 172.21.15.16:0/2748596090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8ef402d340 con 0x7f8ef4102970 2024-03-28T22:08:30.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.950+0000 7f8eeb7fe640 1 -- 172.21.15.16:0/2748596090 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22111 (secure 0 0 0) 0x7f8ee409b790 con 0x7f8ef4102970 2024-03-28T22:08:30.953 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:08:30.953 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":86,"fsid":"798d9884-ed4e-11ee-95d0-87774f69a715","created":"2024-03-28T22:00:27.216099+0000","modified":"2024-03-28T22:08:30.939084+0000","last_up_change":"2024-03-28T22:07:40.251141+0000","last_in_change":"2024-03-28T22:07:19.456451+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:03:59.327832+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:08:28.123529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:08:28.919037+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"46e7c8f8-ecc8-43d6-ae53-0655728f3240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6803","nonce":1589692483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6805","nonce":1589692483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6809","nonce":1589692483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6807","nonce":1589692483}]},"public_addr":"172.21.15.16:6803/1589692483","cluster_addr":"172.21.15.16:6805/1589692483","heartbeat_back_addr":"172.21.15.16:6809/1589692483","heartbeat_front_addr":"172.21.15.16:6807/1589692483","state":["exists","up"]},{"osd":1,"uuid":"af6d9c22-2830-4ce2-a547-011df0c6cbad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6801","nonce":795531234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6803","nonce":795531234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6807","nonce":795531234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6805","nonce":795531234}]},"public_addr":"172.21.15.53:6801/795531234","cluster_addr":"172.21.15.53:6803/795531234","heartbeat_back_addr":"172.21.15.53:6807/795531234","heartbeat_front_addr":"172.21.15.53:6805/795531234","state":["exists","up"]},{"osd":2,"uuid":"b10adca9-110c-4f87-85e0-b7542709d31a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6801","nonce":2214131086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6803","nonce":2214131086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6807","nonce":2214131086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6805","nonce":2214131086}]},"public_addr":"172.21.15.63:6801/2214131086","cluster_addr":"172.21.15.63:6803/2214131086","heartbeat_back_addr":"172.21.15.63:6807/2214131086","heartbeat_front_addr":"172.21.15.63:6805/2214131086","state":["exists","up"]},{"osd":3,"uuid":"c83cf4bb-395a-4786-abf7-74ca93a33eb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6811","nonce":13176789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6813","nonce":13176789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6817","nonce":13176789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6815","nonce":13176789}]},"public_addr":"172.21.15.16:6811/13176789","cluster_addr":"172.21.15.16:6813/13176789","heartbeat_back_addr":"172.21.15.16:6817/13176789","heartbeat_front_addr":"172.21.15.16:6815/13176789","state":["exists","up"]},{"osd":4,"uuid":"941b3e54-47d6-4e85-8069-e730c3fa8eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6809","nonce":772161849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6811","nonce":772161849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6815","nonce":772161849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6813","nonce":772161849}]},"public_addr":"172.21.15.53:6809/772161849","cluster_addr":"172.21.15.53:6811/772161849","heartbeat_back_addr":"172.21.15.53:6815/772161849","heartbeat_front_addr":"172.21.15.53:6813/772161849","state":["exists","up"]},{"osd":5,"uuid":"51d89826-1d60-40fc-8fd8-09bfe1ab4125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6809","nonce":1602554852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6811","nonce":1602554852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6815","nonce":1602554852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6813","nonce":1602554852}]},"public_addr":"172.21.15.63:6809/1602554852","cluster_addr":"172.21.15.63:6811/1602554852","heartbeat_back_addr":"172.21.15.63:6815/1602554852","heartbeat_front_addr":"172.21.15.63:6813/1602554852","state":["exists","up"]},{"osd":6,"uuid":"978efe5d-2c55-4ff8-8a1e-9f891388945e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6819","nonce":4290459224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6821","nonce":4290459224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6825","nonce":4290459224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6823","nonce":4290459224}]},"public_addr":"172.21.15.16:6819/4290459224","cluster_addr":"172.21.15.16:6821/4290459224","heartbeat_back_addr":"172.21.15.16:6825/4290459224","heartbeat_front_addr":"172.21.15.16:6823/4290459224","state":["exists","up"]},{"osd":7,"uuid":"50d489cb-dbb8-48df-b868-b29edf12e04d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6817","nonce":1056467525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6819","nonce":1056467525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6823","nonce":1056467525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6821","nonce":1056467525}]},"public_addr":"172.21.15.53:6817/1056467525","cluster_addr":"172.21.15.53:6819/1056467525","heartbeat_back_addr":"172.21.15.53:6823/1056467525","heartbeat_front_addr":"172.21.15.53:6821/1056467525","state":["exists","up"]},{"osd":8,"uuid":"49b39414-cc9c-4057-ae3f-cc109cd979da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6817","nonce":2918516794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6819","nonce":2918516794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6823","nonce":2918516794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6821","nonce":2918516794}]},"public_addr":"172.21.15.63:6817/2918516794","cluster_addr":"172.21.15.63:6819/2918516794","heartbeat_back_addr":"172.21.15.63:6823/2918516794","heartbeat_front_addr":"172.21.15.63:6821/2918516794","state":["exists","up"]},{"osd":9,"uuid":"4c873e51-b783-4eb6-83a3-fee31722d60b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6827","nonce":1585624367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6829","nonce":1585624367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6833","nonce":1585624367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6831","nonce":1585624367}]},"public_addr":"172.21.15.16:6827/1585624367","cluster_addr":"172.21.15.16:6829/1585624367","heartbeat_back_addr":"172.21.15.16:6833/1585624367","heartbeat_front_addr":"172.21.15.16:6831/1585624367","state":["exists","up"]},{"osd":10,"uuid":"085a78bf-e2ad-481a-85a3-71d1fc549a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6825","nonce":3357446484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6827","nonce":3357446484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6831","nonce":3357446484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6829","nonce":3357446484}]},"public_addr":"172.21.15.53:6825/3357446484","cluster_addr":"172.21.15.53:6827/3357446484","heartbeat_back_addr":"172.21.15.53:6831/3357446484","heartbeat_front_addr":"172.21.15.53:6829/3357446484","state":["exists","up"]},{"osd":11,"uuid":"13302aa3-08b9-4639-9042-27d850b4d5f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6825","nonce":2167866570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6827","nonce":2167866570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6831","nonce":2167866570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6829","nonce":2167866570}]},"public_addr":"172.21.15.63:6825/2167866570","cluster_addr":"172.21.15.63:6827/2167866570","heartbeat_back_addr":"172.21.15.63:6831/2167866570","heartbeat_front_addr":"172.21.15.63:6829/2167866570","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:07.450323+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:31.363172+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:55.295021+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:19.794760+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:44.063492+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:08.195064+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:33.655328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:57.639414+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:22.402394+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:47.462050+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:12.226307+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:37.495625+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3821408150":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6801/1832422779":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/996598130":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/1062264551":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/317557779":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6801/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/3631746518":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6800/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/3116652786":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6800/1832422779":"2024-03-29T22:01:28.993434+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:08:30.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.952+0000 7f8efc69a640 1 -- 172.21.15.16:0/2748596090 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f8ee00a1e00 msgr2=0x7f8ee00a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:30.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.952+0000 7f8efc69a640 1 --2- 172.21.15.16:0/2748596090 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f8ee00a1e00 0x7f8ee00a42c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8ef41027d0 tx=0x7f8eec00b390 comp rx=0 tx=0).stop 2024-03-28T22:08:30.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.952+0000 7f8efc69a640 1 -- 172.21.15.16:0/2748596090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ef4102970 msgr2=0x7f8ef4073b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:30.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.952+0000 7f8efc69a640 1 --2- 172.21.15.16:0/2748596090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ef4102970 0x7f8ef4073b40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8ee400e9f0 tx=0x7f8ee400edf0 comp rx=0 tx=0).stop 2024-03-28T22:08:30.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.952+0000 7f8efc69a640 1 -- 172.21.15.16:0/2748596090 shutdown_connections 2024-03-28T22:08:30.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.952+0000 7f8efc69a640 1 --2- 172.21.15.16:0/2748596090 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f8ee00a1e00 0x7f8ee00a42c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:30.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.952+0000 7f8efc69a640 1 --2- 172.21.15.16:0/2748596090 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ef4103330 0x7f8ef4074080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:30.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.952+0000 7f8efc69a640 1 --2- 172.21.15.16:0/2748596090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ef4102970 0x7f8ef4073b40 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:30.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.952+0000 7f8efc69a640 1 --2- 172.21.15.16:0/2748596090 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8ef4101780 0x7f8ef4075500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:30.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.952+0000 7f8efc69a640 1 -- 172.21.15.16:0/2748596090 >> 172.21.15.16:0/2748596090 conn(0x7f8ef40fcd30 msgr2=0x7f8ef4105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:30.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.952+0000 7f8efc69a640 1 -- 172.21.15.16:0/2748596090 shutdown_connections 2024-03-28T22:08:30.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:30.952+0000 7f8efc69a640 1 -- 172.21.15.16:0/2748596090 wait complete. 2024-03-28T22:08:30.966 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-28T22:08:31.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.068+0000 7f0074a8f640 1 -- 172.21.15.16:0/3977204894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0070102220 msgr2=0x7f0070108ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:31.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.068+0000 7f0074a8f640 1 --2- 172.21.15.16:0/3977204894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0070102220 0x7f0070108ab0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0060009ca0 tx=0x7f006002c0f0 comp rx=0 tx=0).stop 2024-03-28T22:08:31.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.069+0000 7f0074a8f640 1 -- 172.21.15.16:0/3977204894 shutdown_connections 2024-03-28T22:08:31.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.069+0000 7f0074a8f640 1 --2- 172.21.15.16:0/3977204894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0070102220 0x7f0070108ab0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:31.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.069+0000 7f0074a8f640 1 --2- 172.21.15.16:0/3977204894 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0070101860 0x7f0070101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:31.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.069+0000 7f0074a8f640 1 --2- 172.21.15.16:0/3977204894 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0070100670 0x7f0070100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:31.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.069+0000 7f0074a8f640 1 -- 172.21.15.16:0/3977204894 >> 172.21.15.16:0/3977204894 conn(0x7f00700fbbe0 msgr2=0x7f00700fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:31.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.069+0000 7f0074a8f640 1 -- 172.21.15.16:0/3977204894 shutdown_connections 2024-03-28T22:08:31.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.069+0000 7f0074a8f640 1 -- 172.21.15.16:0/3977204894 wait complete. 2024-03-28T22:08:31.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.069+0000 7f0074a8f640 1 Processor -- start 2024-03-28T22:08:31.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.069+0000 7f0074a8f640 1 -- start start 2024-03-28T22:08:31.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.069+0000 7f0074a8f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0070100670 0x7f0070196960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:31.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.069+0000 7f0074a8f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0070101860 0x7f0070196ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:31.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.069+0000 7f0074a8f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0070102220 0x7f007019df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:31.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.069+0000 7f0074a8f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f007010c2f0 con 0x7f0070102220 2024-03-28T22:08:31.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.069+0000 7f0074a8f640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f007010c170 con 0x7f0070100670 2024-03-28T22:08:31.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.069+0000 7f0074a8f640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f007010c470 con 0x7f0070101860 2024-03-28T22:08:31.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.070+0000 7f006ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0070102220 0x7f007019df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:31.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.070+0000 7f006ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0070102220 0x7f007019df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57460/0 (socket says 172.21.15.16:57460) 2024-03-28T22:08:31.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.070+0000 7f006ed9d640 1 -- 172.21.15.16:0/2790159171 learned_addr learned my addr 172.21.15.16:0/2790159171 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:31.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.070+0000 7f006e59c640 1 --2- 172.21.15.16:0/2790159171 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0070100670 0x7f0070196960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:31.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.070+0000 7f006dd9b640 1 --2- 172.21.15.16:0/2790159171 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0070101860 0x7f0070196ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:31.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.070+0000 7f006ed9d640 1 -- 172.21.15.16:0/2790159171 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0070101860 msgr2=0x7f0070196ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:31.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.070+0000 7f006ed9d640 1 --2- 172.21.15.16:0/2790159171 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0070101860 0x7f0070196ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:31.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.070+0000 7f006ed9d640 1 -- 172.21.15.16:0/2790159171 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0070100670 msgr2=0x7f0070196960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:31.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.070+0000 7f006ed9d640 1 --2- 172.21.15.16:0/2790159171 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0070100670 0x7f0070196960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:31.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.070+0000 7f006ed9d640 1 -- 172.21.15.16:0/2790159171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f007019e580 con 0x7f0070102220 2024-03-28T22:08:31.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.070+0000 7f006ed9d640 1 --2- 172.21.15.16:0/2790159171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0070102220 0x7f007019df10 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0060037040 tx=0x7f0060032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:31.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.070+0000 7f00677fe640 1 -- 172.21.15.16:0/2790159171 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0060034ad0 con 0x7f0070102220 2024-03-28T22:08:31.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.070+0000 7f00677fe640 1 -- 172.21.15.16:0/2790159171 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0060034c70 con 0x7f0070102220 2024-03-28T22:08:31.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.070+0000 7f0074a8f640 1 -- 172.21.15.16:0/2790159171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0070068a20 con 0x7f0070102220 2024-03-28T22:08:31.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.070+0000 7f00677fe640 1 -- 172.21.15.16:0/2790159171 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00600042a0 con 0x7f0070102220 2024-03-28T22:08:31.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.070+0000 7f0074a8f640 1 -- 172.21.15.16:0/2790159171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0070068f10 con 0x7f0070102220 2024-03-28T22:08:31.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.071+0000 7f0074a8f640 1 -- 172.21.15.16:0/2790159171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00700685d0 con 0x7f0070102220 2024-03-28T22:08:31.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.072+0000 7f00677fe640 1 -- 172.21.15.16:0/2790159171 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f0060032690 con 0x7f0070102220 2024-03-28T22:08:31.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.073+0000 7f00677fe640 1 --2- 172.21.15.16:0/2790159171 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f005c0a1d80 0x7f005c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:31.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.073+0000 7f006e59c640 1 --2- 172.21.15.16:0/2790159171 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f005c0a1d80 0x7f005c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:31.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.073+0000 7f00677fe640 1 -- 172.21.15.16:0/2790159171 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f00600f1bb0 con 0x7f0070102220 2024-03-28T22:08:31.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.073+0000 7f006e59c640 1 --2- 172.21.15.16:0/2790159171 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f005c0a1d80 0x7f005c0a4240 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0068005fd0 tx=0x7f006800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:31.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.077+0000 7f00677fe640 1 -- 172.21.15.16:0/2790159171 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f00600bf090 con 0x7f0070102220 2024-03-28T22:08:31.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.297+0000 7f0074a8f640 1 -- 172.21.15.16:0/2790159171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f0070107400 con 0x7f0070102220 2024-03-28T22:08:31.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.317+0000 7f00677fe640 1 -- 172.21.15.16:0/2790159171 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) ==== 118+0+0 (secure 0 0 0) 0x7f00600be990 con 0x7f0070102220 2024-03-28T22:08:31.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.319+0000 7f0074a8f640 1 -- 172.21.15.16:0/2790159171 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f005c0a1d80 msgr2=0x7f005c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:31.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.319+0000 7f0074a8f640 1 --2- 172.21.15.16:0/2790159171 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f005c0a1d80 0x7f005c0a4240 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0068005fd0 tx=0x7f006800b390 comp rx=0 tx=0).stop 2024-03-28T22:08:31.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.319+0000 7f0074a8f640 1 -- 172.21.15.16:0/2790159171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0070102220 msgr2=0x7f007019df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:31.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.319+0000 7f0074a8f640 1 --2- 172.21.15.16:0/2790159171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0070102220 0x7f007019df10 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0060037040 tx=0x7f0060032f40 comp rx=0 tx=0).stop 2024-03-28T22:08:31.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.319+0000 7f0074a8f640 1 -- 172.21.15.16:0/2790159171 shutdown_connections 2024-03-28T22:08:31.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.319+0000 7f0074a8f640 1 --2- 172.21.15.16:0/2790159171 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f005c0a1d80 0x7f005c0a4240 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:31.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.319+0000 7f0074a8f640 1 --2- 172.21.15.16:0/2790159171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0070102220 0x7f007019df10 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:31.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.319+0000 7f0074a8f640 1 --2- 172.21.15.16:0/2790159171 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0070101860 0x7f0070196ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:31.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.319+0000 7f0074a8f640 1 --2- 172.21.15.16:0/2790159171 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0070100670 0x7f0070196960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:31.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.319+0000 7f0074a8f640 1 -- 172.21.15.16:0/2790159171 >> 172.21.15.16:0/2790159171 conn(0x7f00700fbbe0 msgr2=0x7f0070104b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:31.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.319+0000 7f0074a8f640 1 -- 172.21.15.16:0/2790159171 shutdown_connections 2024-03-28T22:08:31.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.319+0000 7f0074a8f640 1 -- 172.21.15.16:0/2790159171 wait complete. 2024-03-28T22:08:31.334 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-03-28T22:08:31.334 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-03-28T22:08:31.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.437+0000 7f85a908f640 1 -- 172.21.15.16:0/3400978750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85a4101760 msgr2=0x7f85a4101b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:31.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.437+0000 7f85a908f640 1 --2- 172.21.15.16:0/3400978750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85a4101760 0x7f85a4101b80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8594009ca0 tx=0x7f859402c0f0 comp rx=0 tx=0).stop 2024-03-28T22:08:31.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.437+0000 7f85a908f640 1 -- 172.21.15.16:0/3400978750 shutdown_connections 2024-03-28T22:08:31.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.437+0000 7f85a908f640 1 --2- 172.21.15.16:0/3400978750 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f85a4103310 0x7f85a4109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:31.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.437+0000 7f85a908f640 1 --2- 172.21.15.16:0/3400978750 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85a4102950 0x7f85a4102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:31.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.437+0000 7f85a908f640 1 --2- 172.21.15.16:0/3400978750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85a4101760 0x7f85a4101b80 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:31.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.437+0000 7f85a908f640 1 -- 172.21.15.16:0/3400978750 >> 172.21.15.16:0/3400978750 conn(0x7f85a40fccd0 msgr2=0x7f85a40ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:31.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.437+0000 7f85a908f640 1 -- 172.21.15.16:0/3400978750 shutdown_connections 2024-03-28T22:08:31.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.437+0000 7f85a908f640 1 -- 172.21.15.16:0/3400978750 wait complete. 2024-03-28T22:08:31.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.438+0000 7f85a908f640 1 Processor -- start 2024-03-28T22:08:31.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.438+0000 7f85a908f640 1 -- start start 2024-03-28T22:08:31.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.438+0000 7f85a908f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f85a4101760 0x7f85a4197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:31.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.438+0000 7f85a908f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85a4102950 0x7f85a4197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:31.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.438+0000 7f85a908f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85a4103310 0x7f85a419f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:31.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.438+0000 7f85a908f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f85a410d3e0 con 0x7f85a4103310 2024-03-28T22:08:31.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.438+0000 7f85a908f640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f85a410d260 con 0x7f85a4102950 2024-03-28T22:08:31.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.438+0000 7f85a908f640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f85a410d560 con 0x7f85a4101760 2024-03-28T22:08:31.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.438+0000 7f85a359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85a4103310 0x7f85a419f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:31.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.438+0000 7f85a359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85a4103310 0x7f85a419f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57474/0 (socket says 172.21.15.16:57474) 2024-03-28T22:08:31.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.438+0000 7f85a359e640 1 -- 172.21.15.16:0/2372856687 learned_addr learned my addr 172.21.15.16:0/2372856687 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:31.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.438+0000 7f85a259c640 1 --2- 172.21.15.16:0/2372856687 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85a4102950 0x7f85a4197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:31.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.439+0000 7f85a2d9d640 1 --2- 172.21.15.16:0/2372856687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f85a4101760 0x7f85a4197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:31.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.439+0000 7f85a359e640 1 -- 172.21.15.16:0/2372856687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f85a4101760 msgr2=0x7f85a4197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:31.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.439+0000 7f85a359e640 1 --2- 172.21.15.16:0/2372856687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f85a4101760 0x7f85a4197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:31.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.439+0000 7f85a359e640 1 -- 172.21.15.16:0/2372856687 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85a4102950 msgr2=0x7f85a4197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:31.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.439+0000 7f85a359e640 1 --2- 172.21.15.16:0/2372856687 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85a4102950 0x7f85a4197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:31.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.439+0000 7f85a359e640 1 -- 172.21.15.16:0/2372856687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85a419f6d0 con 0x7f85a4103310 2024-03-28T22:08:31.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.439+0000 7f85a359e640 1 --2- 172.21.15.16:0/2372856687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85a4103310 0x7f85a419f060 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f859c00e970 tx=0x7f859c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:31.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.439+0000 7f859bfff640 1 -- 172.21.15.16:0/2372856687 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f859c00ccc0 con 0x7f85a4103310 2024-03-28T22:08:31.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.439+0000 7f85a908f640 1 -- 172.21.15.16:0/2372856687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85a419f9c0 con 0x7f85a4103310 2024-03-28T22:08:31.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.439+0000 7f859bfff640 1 -- 172.21.15.16:0/2372856687 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f859c00ce60 con 0x7f85a4103310 2024-03-28T22:08:31.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.439+0000 7f859bfff640 1 -- 172.21.15.16:0/2372856687 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f859c010450 con 0x7f85a4103310 2024-03-28T22:08:31.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.439+0000 7f85a908f640 1 -- 172.21.15.16:0/2372856687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85a419ff60 con 0x7f85a4103310 2024-03-28T22:08:31.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.440+0000 7f85a908f640 1 -- 172.21.15.16:0/2372856687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85a4108a80 con 0x7f85a4103310 2024-03-28T22:08:31.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.441+0000 7f859bfff640 1 -- 172.21.15.16:0/2372856687 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f859c005020 con 0x7f85a4103310 2024-03-28T22:08:31.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.442+0000 7f859bfff640 1 --2- 172.21.15.16:0/2372856687 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f85900a1d80 0x7f85900a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:31.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.442+0000 7f85a2d9d640 1 --2- 172.21.15.16:0/2372856687 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f85900a1d80 0x7f85900a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:31.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.442+0000 7f859bfff640 1 -- 172.21.15.16:0/2372856687 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f859c0ce110 con 0x7f85a4103310 2024-03-28T22:08:31.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.442+0000 7f85a2d9d640 1 --2- 172.21.15.16:0/2372856687 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f85900a1d80 0x7f85900a4240 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f859402c700 tx=0x7f8594009360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:31.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.446+0000 7f859bfff640 1 -- 172.21.15.16:0/2372856687 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f859c09b690 con 0x7f85a4103310 2024-03-28T22:08:31.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:31.668+0000 7f85a908f640 1 -- 172.21.15.16:0/2372856687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7f85a4062350 con 0x7f85a4103310 2024-03-28T22:08:31.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:31 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/1943845534' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-28T22:08:31.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:31 smithi063 ceph-mon[36249]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-28T22:08:31.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:31 smithi063 ceph-mon[36249]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-28T22:08:31.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:31 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/3829275212' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:08:31.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:31 smithi063 ceph-mon[36249]: osdmap e86: 12 total, 12 up, 12 in 2024-03-28T22:08:31.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:31 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2748596090' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:08:31.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:31 smithi063 ceph-mon[36249]: pgmap v287: 97 pgs: 11 creating+peering, 45 active+clean, 41 unknown; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-03-28T22:08:31.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:31 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2790159171' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T22:08:31.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:31 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/1943845534' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-28T22:08:31.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:31 smithi016 ceph-mon[32374]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-28T22:08:31.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:31 smithi016 ceph-mon[32374]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-28T22:08:31.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:31 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3829275212' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:08:31.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:31 smithi016 ceph-mon[32374]: osdmap e86: 12 total, 12 up, 12 in 2024-03-28T22:08:31.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:31 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2748596090' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:08:31.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:31 smithi016 ceph-mon[32374]: pgmap v287: 97 pgs: 11 creating+peering, 45 active+clean, 41 unknown; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-03-28T22:08:31.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:31 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2790159171' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T22:08:31.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:31 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/1943845534' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-28T22:08:31.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:31 smithi053 ceph-mon[35979]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-28T22:08:31.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:31 smithi053 ceph-mon[35979]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-28T22:08:31.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:31 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3829275212' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:08:31.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:31 smithi053 ceph-mon[35979]: osdmap e86: 12 total, 12 up, 12 in 2024-03-28T22:08:31.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:31 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2748596090' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:08:31.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:31 smithi053 ceph-mon[35979]: pgmap v287: 97 pgs: 11 creating+peering, 45 active+clean, 41 unknown; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-03-28T22:08:31.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:31 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2790159171' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T22:08:32.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.306+0000 7f859bfff640 1 -- 172.21.15.16:0/2372856687 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v16) ==== 105+0+0 (secure 0 0 0) 0x7f859c09af90 con 0x7f85a4103310 2024-03-28T22:08:32.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.309+0000 7f85a908f640 1 -- 172.21.15.16:0/2372856687 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f85900a1d80 msgr2=0x7f85900a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:32.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.309+0000 7f85a908f640 1 --2- 172.21.15.16:0/2372856687 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f85900a1d80 0x7f85900a4240 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f859402c700 tx=0x7f8594009360 comp rx=0 tx=0).stop 2024-03-28T22:08:32.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.309+0000 7f85a908f640 1 -- 172.21.15.16:0/2372856687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85a4103310 msgr2=0x7f85a419f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:32.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.309+0000 7f85a908f640 1 --2- 172.21.15.16:0/2372856687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85a4103310 0x7f85a419f060 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f859c00e970 tx=0x7f859c00ed70 comp rx=0 tx=0).stop 2024-03-28T22:08:32.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.309+0000 7f85a908f640 1 -- 172.21.15.16:0/2372856687 shutdown_connections 2024-03-28T22:08:32.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.309+0000 7f85a908f640 1 --2- 172.21.15.16:0/2372856687 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f85900a1d80 0x7f85900a4240 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:32.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.309+0000 7f85a908f640 1 --2- 172.21.15.16:0/2372856687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85a4103310 0x7f85a419f060 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:32.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.309+0000 7f85a908f640 1 --2- 172.21.15.16:0/2372856687 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85a4102950 0x7f85a4197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:32.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.309+0000 7f85a908f640 1 --2- 172.21.15.16:0/2372856687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f85a4101760 0x7f85a4197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:32.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.309+0000 7f85a908f640 1 -- 172.21.15.16:0/2372856687 >> 172.21.15.16:0/2372856687 conn(0x7f85a40fccd0 msgr2=0x7f85a4105c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:32.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.309+0000 7f85a908f640 1 -- 172.21.15.16:0/2372856687 shutdown_connections 2024-03-28T22:08:32.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.309+0000 7f85a908f640 1 -- 172.21.15.16:0/2372856687 wait complete. 2024-03-28T22:08:32.331 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-28T22:08:32.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.439+0000 7f08ae4b1640 1 -- 172.21.15.16:0/1108931962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08a81021f0 msgr2=0x7f08a81045e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:32.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.439+0000 7f08ae4b1640 1 --2- 172.21.15.16:0/1108931962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08a81021f0 0x7f08a81045e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f089c009900 tx=0x7f089c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:08:32.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.439+0000 7f08ae4b1640 1 -- 172.21.15.16:0/1108931962 shutdown_connections 2024-03-28T22:08:32.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.439+0000 7f08ae4b1640 1 --2- 172.21.15.16:0/1108931962 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08a8107450 0x7f08a81098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:32.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.439+0000 7f08ae4b1640 1 --2- 172.21.15.16:0/1108931962 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08a8104b20 0x7f08a8106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:32.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.439+0000 7f08ae4b1640 1 --2- 172.21.15.16:0/1108931962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08a81021f0 0x7f08a81045e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:32.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.439+0000 7f08ae4b1640 1 -- 172.21.15.16:0/1108931962 >> 172.21.15.16:0/1108931962 conn(0x7f08a80fbbe0 msgr2=0x7f08a80fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:32.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.439+0000 7f08ae4b1640 1 -- 172.21.15.16:0/1108931962 shutdown_connections 2024-03-28T22:08:32.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.439+0000 7f08ae4b1640 1 -- 172.21.15.16:0/1108931962 wait complete. 2024-03-28T22:08:32.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.440+0000 7f08ae4b1640 1 Processor -- start 2024-03-28T22:08:32.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.440+0000 7f08ae4b1640 1 -- start start 2024-03-28T22:08:32.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.440+0000 7f08ae4b1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08a81021f0 0x7f08a8101b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:32.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.440+0000 7f08ae4b1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08a8104b20 0x7f08a81001a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:32.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.440+0000 7f08ae4b1640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08a8107450 0x7f08a81006e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:32.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.440+0000 7f08ae4b1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f08a810d3f0 con 0x7f08a8104b20 2024-03-28T22:08:32.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.440+0000 7f08ae4b1640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f08a810d270 con 0x7f08a8107450 2024-03-28T22:08:32.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.440+0000 7f08ae4b1640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f08a810d570 con 0x7f08a81021f0 2024-03-28T22:08:32.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.440+0000 7f08a77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08a8104b20 0x7f08a81001a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:32.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.440+0000 7f08a7fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08a81021f0 0x7f08a8101b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:32.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.440+0000 7f08a7fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08a81021f0 0x7f08a8101b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.16:57290/0 (socket says 172.21.15.16:57290) 2024-03-28T22:08:32.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.440+0000 7f08a77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08a8104b20 0x7f08a81001a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57486/0 (socket says 172.21.15.16:57486) 2024-03-28T22:08:32.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.440+0000 7f08a7fff640 1 -- 172.21.15.16:0/4140712913 learned_addr learned my addr 172.21.15.16:0/4140712913 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:32.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.440+0000 7f08aca4e640 1 --2- 172.21.15.16:0/4140712913 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08a8107450 0x7f08a81006e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:32.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.441+0000 7f08a77fe640 1 -- 172.21.15.16:0/4140712913 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08a81021f0 msgr2=0x7f08a8101b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:32.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.441+0000 7f08a77fe640 1 --2- 172.21.15.16:0/4140712913 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08a81021f0 0x7f08a8101b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:32.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.441+0000 7f08a77fe640 1 -- 172.21.15.16:0/4140712913 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08a8107450 msgr2=0x7f08a81006e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:32.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.441+0000 7f08a77fe640 1 --2- 172.21.15.16:0/4140712913 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08a8107450 0x7f08a81006e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:32.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.441+0000 7f08a77fe640 1 -- 172.21.15.16:0/4140712913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08a8100f70 con 0x7f08a8104b20 2024-03-28T22:08:32.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.441+0000 7f08aca4e640 1 --2- 172.21.15.16:0/4140712913 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08a8107450 0x7f08a81006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:08:32.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.441+0000 7f08a77fe640 1 --2- 172.21.15.16:0/4140712913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08a8104b20 0x7f08a81001a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f089800d6a0 tx=0x7f089800db60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:32.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.441+0000 7f08a57fa640 1 -- 172.21.15.16:0/4140712913 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0898017070 con 0x7f08a8104b20 2024-03-28T22:08:32.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.441+0000 7f08a57fa640 1 -- 172.21.15.16:0/4140712913 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0898009820 con 0x7f08a8104b20 2024-03-28T22:08:32.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.441+0000 7f08ae4b1640 1 -- 172.21.15.16:0/4140712913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08a81a1be0 con 0x7f08a8104b20 2024-03-28T22:08:32.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.441+0000 7f08a57fa640 1 -- 172.21.15.16:0/4140712913 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0898004d00 con 0x7f08a8104b20 2024-03-28T22:08:32.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.441+0000 7f08ae4b1640 1 -- 172.21.15.16:0/4140712913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08a81a2130 con 0x7f08a8104b20 2024-03-28T22:08:32.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.442+0000 7f08ae4b1640 1 -- 172.21.15.16:0/4140712913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08a80fd6d0 con 0x7f08a8104b20 2024-03-28T22:08:32.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.443+0000 7f08a57fa640 1 -- 172.21.15.16:0/4140712913 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f08980099d0 con 0x7f08a8104b20 2024-03-28T22:08:32.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.443+0000 7f08a57fa640 1 --2- 172.21.15.16:0/4140712913 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f08900a1d80 0x7f08900a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:32.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.444+0000 7f08a7fff640 1 --2- 172.21.15.16:0/4140712913 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f08900a1d80 0x7f08900a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:32.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.444+0000 7f08a57fa640 1 -- 172.21.15.16:0/4140712913 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f08980cd590 con 0x7f08a8104b20 2024-03-28T22:08:32.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.444+0000 7f08a7fff640 1 --2- 172.21.15.16:0/4140712913 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f08900a1d80 0x7f08900a4240 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f089c000c00 tx=0x7f089c02c880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:32.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.448+0000 7f08a57fa640 1 -- 172.21.15.16:0/4140712913 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f089809a420 con 0x7f08a8104b20 2024-03-28T22:08:32.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:32.669+0000 7f08ae4b1640 1 -- 172.21.15.16:0/4140712913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f08a8062350 con 0x7f08a8104b20 2024-03-28T22:08:32.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:32 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2790159171' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-28T22:08:32.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:32 smithi063 ceph-mon[36249]: mds.? [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] up:active 2024-03-28T22:08:32.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:32 smithi063 ceph-mon[36249]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T22:08:32.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:32 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2372856687' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-28T22:08:32.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:32 smithi063 ceph-mon[36249]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T22:08:32.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:32 smithi063 ceph-mon[36249]: Cluster is now healthy 2024-03-28T22:08:32.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:32 smithi063 ceph-mon[36249]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-28T22:08:32.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:32 smithi063 ceph-mon[36249]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-28T22:08:32.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:32 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2372856687' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-28T22:08:32.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:32 smithi063 ceph-mon[36249]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-03-28T22:08:32.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:32 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2790159171' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-28T22:08:32.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:32 smithi016 ceph-mon[32374]: mds.? [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] up:active 2024-03-28T22:08:32.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:32 smithi016 ceph-mon[32374]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T22:08:32.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:32 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2372856687' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-28T22:08:32.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:32 smithi016 ceph-mon[32374]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T22:08:32.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:32 smithi016 ceph-mon[32374]: Cluster is now healthy 2024-03-28T22:08:32.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:32 smithi016 ceph-mon[32374]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-28T22:08:32.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:32 smithi016 ceph-mon[32374]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-28T22:08:32.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:32 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2372856687' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-28T22:08:32.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:32 smithi016 ceph-mon[32374]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-03-28T22:08:32.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:32 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2790159171' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-28T22:08:32.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:32 smithi053 ceph-mon[35979]: mds.? [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] up:active 2024-03-28T22:08:32.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:32 smithi053 ceph-mon[35979]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T22:08:32.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:32 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2372856687' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-28T22:08:32.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:32 smithi053 ceph-mon[35979]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T22:08:32.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:32 smithi053 ceph-mon[35979]: Cluster is now healthy 2024-03-28T22:08:32.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:32 smithi053 ceph-mon[35979]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-28T22:08:32.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:32 smithi053 ceph-mon[35979]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-28T22:08:32.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:32 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2372856687' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-28T22:08:32.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:32 smithi053 ceph-mon[35979]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-03-28T22:08:33.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.328+0000 7f08a57fa640 1 -- 172.21.15.16:0/4140712913 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v17) ==== 121+0+0 (secure 0 0 0) 0x7f089809e4c0 con 0x7f08a8104b20 2024-03-28T22:08:33.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.332+0000 7f08ae4b1640 1 -- 172.21.15.16:0/4140712913 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f08900a1d80 msgr2=0x7f08900a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:33.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.332+0000 7f08ae4b1640 1 --2- 172.21.15.16:0/4140712913 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f08900a1d80 0x7f08900a4240 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f089c000c00 tx=0x7f089c02c880 comp rx=0 tx=0).stop 2024-03-28T22:08:33.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.332+0000 7f08ae4b1640 1 -- 172.21.15.16:0/4140712913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08a8104b20 msgr2=0x7f08a81001a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:33.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.332+0000 7f08ae4b1640 1 --2- 172.21.15.16:0/4140712913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08a8104b20 0x7f08a81001a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f089800d6a0 tx=0x7f089800db60 comp rx=0 tx=0).stop 2024-03-28T22:08:33.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.332+0000 7f08ae4b1640 1 -- 172.21.15.16:0/4140712913 shutdown_connections 2024-03-28T22:08:33.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.332+0000 7f08ae4b1640 1 --2- 172.21.15.16:0/4140712913 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f08900a1d80 0x7f08900a4240 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:33.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.332+0000 7f08ae4b1640 1 --2- 172.21.15.16:0/4140712913 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08a8107450 0x7f08a81006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:33.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.332+0000 7f08ae4b1640 1 --2- 172.21.15.16:0/4140712913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08a8104b20 0x7f08a81001a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:33.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.332+0000 7f08ae4b1640 1 --2- 172.21.15.16:0/4140712913 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08a81021f0 0x7f08a8101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:33.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.332+0000 7f08ae4b1640 1 -- 172.21.15.16:0/4140712913 >> 172.21.15.16:0/4140712913 conn(0x7f08a80fbbe0 msgr2=0x7f08a8102c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:33.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.332+0000 7f08ae4b1640 1 -- 172.21.15.16:0/4140712913 shutdown_connections 2024-03-28T22:08:33.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.333+0000 7f08ae4b1640 1 -- 172.21.15.16:0/4140712913 wait complete. 2024-03-28T22:08:33.353 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-28T22:08:33.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.462+0000 7f664feeb640 1 -- 172.21.15.16:0/3679773269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6648103350 msgr2=0x7f6648109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:33.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.462+0000 7f664feeb640 1 --2- 172.21.15.16:0/3679773269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6648103350 0x7f6648109be0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6640009ca0 tx=0x7f664002c0f0 comp rx=0 tx=0).stop 2024-03-28T22:08:33.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.462+0000 7f664feeb640 1 -- 172.21.15.16:0/3679773269 shutdown_connections 2024-03-28T22:08:33.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.462+0000 7f664feeb640 1 --2- 172.21.15.16:0/3679773269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6648103350 0x7f6648109be0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:33.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.462+0000 7f664feeb640 1 --2- 172.21.15.16:0/3679773269 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6648102990 0x7f6648102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:33.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.462+0000 7f664feeb640 1 --2- 172.21.15.16:0/3679773269 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66481017a0 0x7f6648101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:33.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.462+0000 7f664feeb640 1 -- 172.21.15.16:0/3679773269 >> 172.21.15.16:0/3679773269 conn(0x7f66480fcd10 msgr2=0x7f66480ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:33.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.462+0000 7f664feeb640 1 -- 172.21.15.16:0/3679773269 shutdown_connections 2024-03-28T22:08:33.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.462+0000 7f664feeb640 1 -- 172.21.15.16:0/3679773269 wait complete. 2024-03-28T22:08:33.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.463+0000 7f664feeb640 1 Processor -- start 2024-03-28T22:08:33.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.463+0000 7f664feeb640 1 -- start start 2024-03-28T22:08:33.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.463+0000 7f664feeb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66481017a0 0x7f6648075500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:33.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.463+0000 7f664feeb640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6648102990 0x7f6648073b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:33.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.463+0000 7f664feeb640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6648103350 0x7f6648074080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:33.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.463+0000 7f664feeb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f664810d420 con 0x7f66481017a0 2024-03-28T22:08:33.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.463+0000 7f664feeb640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f664810d2a0 con 0x7f6648103350 2024-03-28T22:08:33.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.463+0000 7f664feeb640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f664810d5a0 con 0x7f6648102990 2024-03-28T22:08:33.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.463+0000 7f664dc87640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66481017a0 0x7f6648075500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:33.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.463+0000 7f664dc87640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66481017a0 0x7f6648075500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57506/0 (socket says 172.21.15.16:57506) 2024-03-28T22:08:33.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.463+0000 7f664dc87640 1 -- 172.21.15.16:0/988168491 learned_addr learned my addr 172.21.15.16:0/988168491 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:33.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.463+0000 7f664e488640 1 --2- 172.21.15.16:0/988168491 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6648103350 0x7f6648074080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:33.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.463+0000 7f664dc87640 1 -- 172.21.15.16:0/988168491 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6648102990 msgr2=0x7f6648073b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:33.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.463+0000 7f664d486640 1 --2- 172.21.15.16:0/988168491 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6648102990 0x7f6648073b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:33.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.463+0000 7f664dc87640 1 --2- 172.21.15.16:0/988168491 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6648102990 0x7f6648073b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:33.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.463+0000 7f664dc87640 1 -- 172.21.15.16:0/988168491 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6648103350 msgr2=0x7f6648074080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:33.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.463+0000 7f664dc87640 1 --2- 172.21.15.16:0/988168491 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6648103350 0x7f6648074080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:33.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.463+0000 7f664dc87640 1 -- 172.21.15.16:0/988168491 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6648074910 con 0x7f66481017a0 2024-03-28T22:08:33.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.463+0000 7f664d486640 1 --2- 172.21.15.16:0/988168491 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6648102990 0x7f6648073b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:08:33.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.463+0000 7f664dc87640 1 --2- 172.21.15.16:0/988168491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66481017a0 0x7f6648075500 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f664400c970 tx=0x7f664400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:33.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.464+0000 7f663effd640 1 -- 172.21.15.16:0/988168491 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6644004220 con 0x7f66481017a0 2024-03-28T22:08:33.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.464+0000 7f663effd640 1 -- 172.21.15.16:0/988168491 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66440043c0 con 0x7f66481017a0 2024-03-28T22:08:33.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.464+0000 7f663effd640 1 -- 172.21.15.16:0/988168491 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6644004a00 con 0x7f66481017a0 2024-03-28T22:08:33.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.464+0000 7f664feeb640 1 -- 172.21.15.16:0/988168491 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66481a1b50 con 0x7f66481017a0 2024-03-28T22:08:33.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.464+0000 7f664feeb640 1 -- 172.21.15.16:0/988168491 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66481a20a0 con 0x7f66481017a0 2024-03-28T22:08:33.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.465+0000 7f664feeb640 1 -- 172.21.15.16:0/988168491 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6648108a60 con 0x7f66481017a0 2024-03-28T22:08:33.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.466+0000 7f663effd640 1 -- 172.21.15.16:0/988168491 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f66440079b0 con 0x7f66481017a0 2024-03-28T22:08:33.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.466+0000 7f663effd640 1 --2- 172.21.15.16:0/988168491 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f66340a1e00 0x7f66340a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:33.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.467+0000 7f663effd640 1 -- 172.21.15.16:0/988168491 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f66440cd9d0 con 0x7f66481017a0 2024-03-28T22:08:33.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.467+0000 7f664d486640 1 --2- 172.21.15.16:0/988168491 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f66340a1e00 0x7f66340a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:33.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.467+0000 7f664d486640 1 --2- 172.21.15.16:0/988168491 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f66340a1e00 0x7f66340a42c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6648075270 tx=0x7f6638005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:33.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.471+0000 7f663effd640 1 -- 172.21.15.16:0/988168491 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6644014030 con 0x7f66481017a0 2024-03-28T22:08:33.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:33 smithi063 ceph-mon[36249]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-03-28T22:08:33.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:33 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/4140712913' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T22:08:33.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:33 smithi063 ceph-mon[36249]: pgmap v288: 97 pgs: 11 creating+peering, 76 active+clean, 10 unknown; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 3 op/s 2024-03-28T22:08:33.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:33 smithi016 ceph-mon[32374]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-03-28T22:08:33.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:33 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/4140712913' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T22:08:33.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:33 smithi016 ceph-mon[32374]: pgmap v288: 97 pgs: 11 creating+peering, 76 active+clean, 10 unknown; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 3 op/s 2024-03-28T22:08:33.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:33.692+0000 7f664feeb640 1 -- 172.21.15.16:0/988168491 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f664802d3d0 con 0x7f66481017a0 2024-03-28T22:08:33.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:33 smithi053 ceph-mon[35979]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-03-28T22:08:33.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:33 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/4140712913' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T22:08:33.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:33 smithi053 ceph-mon[35979]: pgmap v288: 97 pgs: 11 creating+peering, 76 active+clean, 10 unknown; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 3 op/s 2024-03-28T22:08:34.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.336+0000 7f663effd640 1 -- 172.21.15.16:0/988168491 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v19) ==== 115+0+0 (secure 0 0 0) 0x7f664409af50 con 0x7f66481017a0 2024-03-28T22:08:34.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.339+0000 7f664feeb640 1 -- 172.21.15.16:0/988168491 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f66340a1e00 msgr2=0x7f66340a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:34.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.339+0000 7f664feeb640 1 --2- 172.21.15.16:0/988168491 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f66340a1e00 0x7f66340a42c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6648075270 tx=0x7f6638005f90 comp rx=0 tx=0).stop 2024-03-28T22:08:34.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.339+0000 7f664feeb640 1 -- 172.21.15.16:0/988168491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66481017a0 msgr2=0x7f6648075500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:34.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.339+0000 7f664feeb640 1 --2- 172.21.15.16:0/988168491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66481017a0 0x7f6648075500 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f664400c970 tx=0x7f664400ce30 comp rx=0 tx=0).stop 2024-03-28T22:08:34.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.339+0000 7f664feeb640 1 -- 172.21.15.16:0/988168491 shutdown_connections 2024-03-28T22:08:34.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.339+0000 7f664feeb640 1 --2- 172.21.15.16:0/988168491 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f66340a1e00 0x7f66340a42c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:34.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.339+0000 7f664feeb640 1 --2- 172.21.15.16:0/988168491 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6648103350 0x7f6648074080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:34.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.339+0000 7f664feeb640 1 --2- 172.21.15.16:0/988168491 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6648102990 0x7f6648073b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:34.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.339+0000 7f664feeb640 1 --2- 172.21.15.16:0/988168491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66481017a0 0x7f6648075500 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:34.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.339+0000 7f664feeb640 1 -- 172.21.15.16:0/988168491 >> 172.21.15.16:0/988168491 conn(0x7f66480fcd10 msgr2=0x7f6648105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:34.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.339+0000 7f664feeb640 1 -- 172.21.15.16:0/988168491 shutdown_connections 2024-03-28T22:08:34.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.339+0000 7f664feeb640 1 -- 172.21.15.16:0/988168491 wait complete. 2024-03-28T22:08:34.356 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated'} subvols for filesystem 'cephfs' 2024-03-28T22:08:34.357 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-28T22:08:34.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.466+0000 7f6ebb0c9640 1 -- 172.21.15.16:0/3128983110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eb4101780 msgr2=0x7f6eb4101ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:34.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.466+0000 7f6ebb0c9640 1 --2- 172.21.15.16:0/3128983110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eb4101780 0x7f6eb4101ba0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6ea8009900 tx=0x7f6ea802bc30 comp rx=0 tx=0).stop 2024-03-28T22:08:34.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.466+0000 7f6ebb0c9640 1 -- 172.21.15.16:0/3128983110 shutdown_connections 2024-03-28T22:08:34.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.466+0000 7f6ebb0c9640 1 --2- 172.21.15.16:0/3128983110 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6eb4103330 0x7f6eb4109bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:34.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.466+0000 7f6ebb0c9640 1 --2- 172.21.15.16:0/3128983110 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6eb4102970 0x7f6eb4102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:34.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.466+0000 7f6ebb0c9640 1 --2- 172.21.15.16:0/3128983110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eb4101780 0x7f6eb4101ba0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:34.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.466+0000 7f6ebb0c9640 1 -- 172.21.15.16:0/3128983110 >> 172.21.15.16:0/3128983110 conn(0x7f6eb40fccb0 msgr2=0x7f6eb40ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:34.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.466+0000 7f6ebb0c9640 1 -- 172.21.15.16:0/3128983110 shutdown_connections 2024-03-28T22:08:34.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.466+0000 7f6ebb0c9640 1 -- 172.21.15.16:0/3128983110 wait complete. 2024-03-28T22:08:34.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.466+0000 7f6ebb0c9640 1 Processor -- start 2024-03-28T22:08:34.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.466+0000 7f6ebb0c9640 1 -- start start 2024-03-28T22:08:34.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.467+0000 7f6ebb0c9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eb4101780 0x7f6eb4197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:34.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.467+0000 7f6ebb0c9640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6eb4102970 0x7f6eb4197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:34.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.467+0000 7f6ebb0c9640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6eb4103330 0x7f6eb419f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:34.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.467+0000 7f6ebb0c9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb410d450 con 0x7f6eb4101780 2024-03-28T22:08:34.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.467+0000 7f6ebb0c9640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb410d2d0 con 0x7f6eb4102970 2024-03-28T22:08:34.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.467+0000 7f6ebb0c9640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb410d5d0 con 0x7f6eb4103330 2024-03-28T22:08:34.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.467+0000 7f6eb8e65640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eb4101780 0x7f6eb4197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:34.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.467+0000 7f6eb8e65640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eb4101780 0x7f6eb4197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57536/0 (socket says 172.21.15.16:57536) 2024-03-28T22:08:34.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.467+0000 7f6eb8e65640 1 -- 172.21.15.16:0/1471305127 learned_addr learned my addr 172.21.15.16:0/1471305127 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:34.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.467+0000 7f6eb3fff640 1 --2- 172.21.15.16:0/1471305127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6eb4102970 0x7f6eb4197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:34.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.467+0000 7f6eb9666640 1 --2- 172.21.15.16:0/1471305127 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6eb4103330 0x7f6eb419f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:34.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.467+0000 7f6eb8e65640 1 -- 172.21.15.16:0/1471305127 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6eb4103330 msgr2=0x7f6eb419f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:34.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.467+0000 7f6eb8e65640 1 --2- 172.21.15.16:0/1471305127 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6eb4103330 0x7f6eb419f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:34.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.467+0000 7f6eb8e65640 1 -- 172.21.15.16:0/1471305127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6eb4102970 msgr2=0x7f6eb4197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:34.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.467+0000 7f6eb8e65640 1 --2- 172.21.15.16:0/1471305127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6eb4102970 0x7f6eb4197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:34.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.467+0000 7f6eb8e65640 1 -- 172.21.15.16:0/1471305127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eb419f680 con 0x7f6eb4101780 2024-03-28T22:08:34.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.467+0000 7f6eb8e65640 1 --2- 172.21.15.16:0/1471305127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eb4101780 0x7f6eb4197a60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6ea8000c00 tx=0x7f6ea802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:34.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.467+0000 7f6eb1ffb640 1 -- 172.21.15.16:0/1471305127 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea8043020 con 0x7f6eb4101780 2024-03-28T22:08:34.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.467+0000 7f6eb1ffb640 1 -- 172.21.15.16:0/1471305127 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ea80054e0 con 0x7f6eb4101780 2024-03-28T22:08:34.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.467+0000 7f6ebb0c9640 1 -- 172.21.15.16:0/1471305127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6eb419f910 con 0x7f6eb4101780 2024-03-28T22:08:34.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.467+0000 7f6eb1ffb640 1 -- 172.21.15.16:0/1471305127 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea8003c40 con 0x7f6eb4101780 2024-03-28T22:08:34.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.468+0000 7f6ebb0c9640 1 -- 172.21.15.16:0/1471305127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6eb419fe60 con 0x7f6eb4101780 2024-03-28T22:08:34.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.469+0000 7f6ebb0c9640 1 -- 172.21.15.16:0/1471305127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6eb4108a40 con 0x7f6eb4101780 2024-03-28T22:08:34.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.470+0000 7f6eb1ffb640 1 -- 172.21.15.16:0/1471305127 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f6ea8046020 con 0x7f6eb4101780 2024-03-28T22:08:34.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.470+0000 7f6eb1ffb640 1 --2- 172.21.15.16:0/1471305127 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f6ea40a1d80 0x7f6ea40a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:34.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.471+0000 7f6eb3fff640 1 --2- 172.21.15.16:0/1471305127 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f6ea40a1d80 0x7f6ea40a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:34.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.471+0000 7f6eb1ffb640 1 -- 172.21.15.16:0/1471305127 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f6ea80efd30 con 0x7f6eb4101780 2024-03-28T22:08:34.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.471+0000 7f6eb3fff640 1 --2- 172.21.15.16:0/1471305127 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f6ea40a1d80 0x7f6ea40a4240 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6eb4198f70 tx=0x7f6ea0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:34.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.475+0000 7f6eb1ffb640 1 -- 172.21.15.16:0/1471305127 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6ea804d020 con 0x7f6eb4101780 2024-03-28T22:08:34.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:34 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/4140712913' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-28T22:08:34.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:34 smithi063 ceph-mon[36249]: mds.? [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] up:active 2024-03-28T22:08:34.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:34 smithi063 ceph-mon[36249]: daemon mds.e assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-28T22:08:34.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:34 smithi063 ceph-mon[36249]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-28T22:08:34.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:34 smithi063 ceph-mon[36249]: Cluster is now healthy 2024-03-28T22:08:34.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:34 smithi063 ceph-mon[36249]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-03-28T22:08:34.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:34 smithi063 ceph-mon[36249]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:creating} 9 up:standby 2024-03-28T22:08:34.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:34 smithi063 ceph-mon[36249]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-03-28T22:08:34.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:34 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/988168491' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T22:08:34.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:34 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/4140712913' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-28T22:08:34.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:34 smithi016 ceph-mon[32374]: mds.? [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] up:active 2024-03-28T22:08:34.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:34 smithi016 ceph-mon[32374]: daemon mds.e assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-28T22:08:34.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:34 smithi016 ceph-mon[32374]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-28T22:08:34.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:34 smithi016 ceph-mon[32374]: Cluster is now healthy 2024-03-28T22:08:34.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:34 smithi016 ceph-mon[32374]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-03-28T22:08:34.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:34 smithi016 ceph-mon[32374]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:creating} 9 up:standby 2024-03-28T22:08:34.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:34 smithi016 ceph-mon[32374]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-03-28T22:08:34.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:34 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/988168491' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T22:08:34.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.684+0000 7f6ebb0c9640 1 -- 172.21.15.16:0/1471305127 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f6eb4061b90 con 0x7f6ea40a1d80 2024-03-28T22:08:34.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.710+0000 7f6eb1ffb640 1 -- 172.21.15.16:0/1471305127 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f6eb4061b90 con 0x7f6ea40a1d80 2024-03-28T22:08:34.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.712+0000 7f6ebb0c9640 1 -- 172.21.15.16:0/1471305127 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f6ea40a1d80 msgr2=0x7f6ea40a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:34.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.712+0000 7f6ebb0c9640 1 --2- 172.21.15.16:0/1471305127 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f6ea40a1d80 0x7f6ea40a4240 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6eb4198f70 tx=0x7f6ea0005f90 comp rx=0 tx=0).stop 2024-03-28T22:08:34.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.712+0000 7f6ebb0c9640 1 -- 172.21.15.16:0/1471305127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eb4101780 msgr2=0x7f6eb4197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:34.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.712+0000 7f6ebb0c9640 1 --2- 172.21.15.16:0/1471305127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eb4101780 0x7f6eb4197a60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6ea8000c00 tx=0x7f6ea802c650 comp rx=0 tx=0).stop 2024-03-28T22:08:34.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.712+0000 7f6ebb0c9640 1 -- 172.21.15.16:0/1471305127 shutdown_connections 2024-03-28T22:08:34.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.712+0000 7f6ebb0c9640 1 --2- 172.21.15.16:0/1471305127 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f6ea40a1d80 0x7f6ea40a4240 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:34.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.712+0000 7f6ebb0c9640 1 --2- 172.21.15.16:0/1471305127 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6eb4103330 0x7f6eb419f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:34.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.712+0000 7f6ebb0c9640 1 --2- 172.21.15.16:0/1471305127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6eb4102970 0x7f6eb4197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:34.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.712+0000 7f6ebb0c9640 1 --2- 172.21.15.16:0/1471305127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eb4101780 0x7f6eb4197a60 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:34.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.712+0000 7f6ebb0c9640 1 -- 172.21.15.16:0/1471305127 >> 172.21.15.16:0/1471305127 conn(0x7f6eb40fccb0 msgr2=0x7f6eb4105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:34.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.712+0000 7f6ebb0c9640 1 -- 172.21.15.16:0/1471305127 shutdown_connections 2024-03-28T22:08:34.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.712+0000 7f6ebb0c9640 1 -- 172.21.15.16:0/1471305127 wait complete. 2024-03-28T22:08:34.726 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-28T22:08:34.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:34 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/4140712913' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-28T22:08:34.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:34 smithi053 ceph-mon[35979]: mds.? [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] up:active 2024-03-28T22:08:34.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:34 smithi053 ceph-mon[35979]: daemon mds.e assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-28T22:08:34.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:34 smithi053 ceph-mon[35979]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-28T22:08:34.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:34 smithi053 ceph-mon[35979]: Cluster is now healthy 2024-03-28T22:08:34.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:34 smithi053 ceph-mon[35979]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-03-28T22:08:34.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:34 smithi053 ceph-mon[35979]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:creating} 9 up:standby 2024-03-28T22:08:34.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:34 smithi053 ceph-mon[35979]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-03-28T22:08:34.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:34 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/988168491' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T22:08:34.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.831+0000 7ff22ac5c640 1 -- 172.21.15.16:0/2750863783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff224101760 msgr2=0x7ff224101b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:34.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.831+0000 7ff22ac5c640 1 --2- 172.21.15.16:0/2750863783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff224101760 0x7ff224101b80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff21c009ca0 tx=0x7ff21c02c0f0 comp rx=0 tx=0).stop 2024-03-28T22:08:34.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.831+0000 7ff22ac5c640 1 -- 172.21.15.16:0/2750863783 shutdown_connections 2024-03-28T22:08:34.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.831+0000 7ff22ac5c640 1 --2- 172.21.15.16:0/2750863783 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff224103310 0x7ff224109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:34.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.831+0000 7ff22ac5c640 1 --2- 172.21.15.16:0/2750863783 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff224102950 0x7ff224102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:34.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.831+0000 7ff22ac5c640 1 --2- 172.21.15.16:0/2750863783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff224101760 0x7ff224101b80 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:34.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.831+0000 7ff22ac5c640 1 -- 172.21.15.16:0/2750863783 >> 172.21.15.16:0/2750863783 conn(0x7ff2240fccd0 msgr2=0x7ff2240ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:34.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.831+0000 7ff22ac5c640 1 -- 172.21.15.16:0/2750863783 shutdown_connections 2024-03-28T22:08:34.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.831+0000 7ff22ac5c640 1 -- 172.21.15.16:0/2750863783 wait complete. 2024-03-28T22:08:34.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.832+0000 7ff22ac5c640 1 Processor -- start 2024-03-28T22:08:34.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.832+0000 7ff22ac5c640 1 -- start start 2024-03-28T22:08:34.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.832+0000 7ff22ac5c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff224101760 0x7ff224197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:34.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.832+0000 7ff22ac5c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff224102950 0x7ff224197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:34.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.832+0000 7ff22ac5c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff224103310 0x7ff22419f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:34.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.832+0000 7ff22ac5c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff22410d3b0 con 0x7ff224103310 2024-03-28T22:08:34.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.832+0000 7ff22ac5c640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff22410d230 con 0x7ff224101760 2024-03-28T22:08:34.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.832+0000 7ff22ac5c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff22410d530 con 0x7ff224102950 2024-03-28T22:08:34.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.832+0000 7ff2291f9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff224103310 0x7ff22419f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:34.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.832+0000 7ff2291f9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff224103310 0x7ff22419f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57584/0 (socket says 172.21.15.16:57584) 2024-03-28T22:08:34.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.832+0000 7ff2291f9640 1 -- 172.21.15.16:0/3666522601 learned_addr learned my addr 172.21.15.16:0/3666522601 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:34.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.832+0000 7ff2289f8640 1 --2- 172.21.15.16:0/3666522601 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff224101760 0x7ff224197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:34.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.832+0000 7ff223fff640 1 --2- 172.21.15.16:0/3666522601 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff224102950 0x7ff224197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:34.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.833+0000 7ff2291f9640 1 -- 172.21.15.16:0/3666522601 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff224102950 msgr2=0x7ff224197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:34.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.833+0000 7ff2291f9640 1 --2- 172.21.15.16:0/3666522601 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff224102950 0x7ff224197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:34.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.833+0000 7ff2291f9640 1 -- 172.21.15.16:0/3666522601 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff224101760 msgr2=0x7ff224197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:34.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.833+0000 7ff2291f9640 1 --2- 172.21.15.16:0/3666522601 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff224101760 0x7ff224197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:34.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.833+0000 7ff2291f9640 1 -- 172.21.15.16:0/3666522601 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff22419f6b0 con 0x7ff224103310 2024-03-28T22:08:34.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.833+0000 7ff2291f9640 1 --2- 172.21.15.16:0/3666522601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff224103310 0x7ff22419f040 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff20c00d750 tx=0x7ff20c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:34.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.833+0000 7ff221ffb640 1 -- 172.21.15.16:0/3666522601 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff20c017070 con 0x7ff224103310 2024-03-28T22:08:34.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.833+0000 7ff221ffb640 1 -- 172.21.15.16:0/3666522601 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff20c005500 con 0x7ff224103310 2024-03-28T22:08:34.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.833+0000 7ff22ac5c640 1 -- 172.21.15.16:0/3666522601 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff22419f9a0 con 0x7ff224103310 2024-03-28T22:08:34.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.833+0000 7ff221ffb640 1 -- 172.21.15.16:0/3666522601 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff20c004d90 con 0x7ff224103310 2024-03-28T22:08:34.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.833+0000 7ff22ac5c640 1 -- 172.21.15.16:0/3666522601 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff22419fea0 con 0x7ff224103310 2024-03-28T22:08:34.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.834+0000 7ff22ac5c640 1 -- 172.21.15.16:0/3666522601 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff224108aa0 con 0x7ff224103310 2024-03-28T22:08:34.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.835+0000 7ff221ffb640 1 -- 172.21.15.16:0/3666522601 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7ff20c004f30 con 0x7ff224103310 2024-03-28T22:08:34.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.836+0000 7ff221ffb640 1 --2- 172.21.15.16:0/3666522601 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff2100a1d80 0x7ff2100a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:34.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.836+0000 7ff2289f8640 1 --2- 172.21.15.16:0/3666522601 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff2100a1d80 0x7ff2100a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:34.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.836+0000 7ff221ffb640 1 -- 172.21.15.16:0/3666522601 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7ff20c0cd630 con 0x7ff224103310 2024-03-28T22:08:34.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.836+0000 7ff2289f8640 1 --2- 172.21.15.16:0/3666522601 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff2100a1d80 0x7ff2100a4240 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff21c02c630 tx=0x7ff21c009360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:34.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:34.840+0000 7ff221ffb640 1 -- 172.21.15.16:0/3666522601 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff20c09a4c0 con 0x7ff224103310 2024-03-28T22:08:35.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.050+0000 7ff22ac5c640 1 -- 172.21.15.16:0/3666522601 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7ff224061c20 con 0x7ff2100a1d80 2024-03-28T22:08:35.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.076+0000 7ff221ffb640 1 -- 172.21.15.16:0/3666522601 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff224061c20 con 0x7ff2100a1d80 2024-03-28T22:08:35.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.078+0000 7ff22ac5c640 1 -- 172.21.15.16:0/3666522601 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff2100a1d80 msgr2=0x7ff2100a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:35.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.078+0000 7ff22ac5c640 1 --2- 172.21.15.16:0/3666522601 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff2100a1d80 0x7ff2100a4240 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff21c02c630 tx=0x7ff21c009360 comp rx=0 tx=0).stop 2024-03-28T22:08:35.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.078+0000 7ff22ac5c640 1 -- 172.21.15.16:0/3666522601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff224103310 msgr2=0x7ff22419f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:35.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.078+0000 7ff22ac5c640 1 --2- 172.21.15.16:0/3666522601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff224103310 0x7ff22419f040 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff20c00d750 tx=0x7ff20c00dc10 comp rx=0 tx=0).stop 2024-03-28T22:08:35.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.078+0000 7ff22ac5c640 1 -- 172.21.15.16:0/3666522601 shutdown_connections 2024-03-28T22:08:35.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.078+0000 7ff22ac5c640 1 --2- 172.21.15.16:0/3666522601 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff2100a1d80 0x7ff2100a4240 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:35.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.078+0000 7ff22ac5c640 1 --2- 172.21.15.16:0/3666522601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff224103310 0x7ff22419f040 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:35.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.078+0000 7ff22ac5c640 1 --2- 172.21.15.16:0/3666522601 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff224102950 0x7ff224197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:35.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.078+0000 7ff22ac5c640 1 --2- 172.21.15.16:0/3666522601 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff224101760 0x7ff224197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:35.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.078+0000 7ff22ac5c640 1 -- 172.21.15.16:0/3666522601 >> 172.21.15.16:0/3666522601 conn(0x7ff2240fccd0 msgr2=0x7ff224105c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:35.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.078+0000 7ff22ac5c640 1 -- 172.21.15.16:0/3666522601 shutdown_connections 2024-03-28T22:08:35.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.078+0000 7ff22ac5c640 1 -- 172.21.15.16:0/3666522601 wait complete. 2024-03-28T22:08:35.090 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:08:34 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:08:34.689+0000 7ff185e6f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:08:35.091 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:08:34 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:08:34.689+0000 7ff185e6f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:08:35.091 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:08:34 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:08:34.689+0000 7ff185e6f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:08:35.091 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:08:34 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:08:34.689+0000 7ff185e6f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:08:35.091 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:08:34 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:08:34.689+0000 7ff185e6f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:08:35.092 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-28T22:08:35.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.197+0000 7f106482a640 1 -- 172.21.15.16:0/4197905439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1060102220 msgr2=0x7f1060108ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:35.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.197+0000 7f106482a640 1 --2- 172.21.15.16:0/4197905439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1060102220 0x7f1060108ab0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1058009ca0 tx=0x7f105802c0f0 comp rx=0 tx=0).stop 2024-03-28T22:08:35.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.197+0000 7f106482a640 1 -- 172.21.15.16:0/4197905439 shutdown_connections 2024-03-28T22:08:35.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.197+0000 7f106482a640 1 --2- 172.21.15.16:0/4197905439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1060102220 0x7f1060108ab0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:35.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.197+0000 7f106482a640 1 --2- 172.21.15.16:0/4197905439 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1060101860 0x7f1060101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:35.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.197+0000 7f106482a640 1 --2- 172.21.15.16:0/4197905439 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1060100670 0x7f1060100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:35.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.197+0000 7f106482a640 1 -- 172.21.15.16:0/4197905439 >> 172.21.15.16:0/4197905439 conn(0x7f10600fbbe0 msgr2=0x7f10600fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:35.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.197+0000 7f106482a640 1 -- 172.21.15.16:0/4197905439 shutdown_connections 2024-03-28T22:08:35.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.197+0000 7f106482a640 1 -- 172.21.15.16:0/4197905439 wait complete. 2024-03-28T22:08:35.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.198+0000 7f106482a640 1 Processor -- start 2024-03-28T22:08:35.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.198+0000 7f106482a640 1 -- start start 2024-03-28T22:08:35.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.198+0000 7f106482a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1060100670 0x7f1060197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:35.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.198+0000 7f106482a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1060101860 0x7f1060197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:35.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.198+0000 7f106482a640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1060102220 0x7f106019eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:35.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.198+0000 7f106482a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f106010c2d0 con 0x7f1060100670 2024-03-28T22:08:35.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.198+0000 7f106482a640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f106010c150 con 0x7f1060102220 2024-03-28T22:08:35.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.198+0000 7f106482a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f106010c450 con 0x7f1060101860 2024-03-28T22:08:35.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.198+0000 7f105e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1060100670 0x7f1060197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:35.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.198+0000 7f105e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1060100670 0x7f1060197a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57604/0 (socket says 172.21.15.16:57604) 2024-03-28T22:08:35.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.198+0000 7f105e59c640 1 -- 172.21.15.16:0/957342997 learned_addr learned my addr 172.21.15.16:0/957342997 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:35.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.198+0000 7f105ed9d640 1 --2- 172.21.15.16:0/957342997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1060102220 0x7f106019eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:35.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.199+0000 7f1055d9b640 1 --2- 172.21.15.16:0/957342997 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1060101860 0x7f1060197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:35.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.199+0000 7f105e59c640 1 -- 172.21.15.16:0/957342997 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1060101860 msgr2=0x7f1060197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:35.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.199+0000 7f105e59c640 1 --2- 172.21.15.16:0/957342997 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1060101860 0x7f1060197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:35.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.199+0000 7f105e59c640 1 -- 172.21.15.16:0/957342997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1060102220 msgr2=0x7f106019eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:35.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.199+0000 7f105e59c640 1 --2- 172.21.15.16:0/957342997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1060102220 0x7f106019eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:35.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.199+0000 7f105e59c640 1 -- 172.21.15.16:0/957342997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f106019f660 con 0x7f1060100670 2024-03-28T22:08:35.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.199+0000 7f105e59c640 1 --2- 172.21.15.16:0/957342997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1060100670 0x7f1060197a40 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f105000d770 tx=0x7f105000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:35.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.199+0000 7f1057fff640 1 -- 172.21.15.16:0/957342997 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1050017070 con 0x7f1060100670 2024-03-28T22:08:35.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.199+0000 7f1057fff640 1 -- 172.21.15.16:0/957342997 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1050009c30 con 0x7f1060100670 2024-03-28T22:08:35.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.199+0000 7f106482a640 1 -- 172.21.15.16:0/957342997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f106019f950 con 0x7f1060100670 2024-03-28T22:08:35.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.199+0000 7f1057fff640 1 -- 172.21.15.16:0/957342997 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1050003dd0 con 0x7f1060100670 2024-03-28T22:08:35.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.199+0000 7f106482a640 1 -- 172.21.15.16:0/957342997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f106019fea0 con 0x7f1060100670 2024-03-28T22:08:35.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.200+0000 7f106482a640 1 -- 172.21.15.16:0/957342997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10600685d0 con 0x7f1060100670 2024-03-28T22:08:35.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.201+0000 7f1057fff640 1 -- 172.21.15.16:0/957342997 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f1050003fb0 con 0x7f1060100670 2024-03-28T22:08:35.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.201+0000 7f1057fff640 1 --2- 172.21.15.16:0/957342997 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f104c0a1d80 0x7f104c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:35.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.202+0000 7f1055d9b640 1 --2- 172.21.15.16:0/957342997 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f104c0a1d80 0x7f104c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:35.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.202+0000 7f1057fff640 1 -- 172.21.15.16:0/957342997 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f10500cd5c0 con 0x7f1060100670 2024-03-28T22:08:35.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.202+0000 7f1055d9b640 1 --2- 172.21.15.16:0/957342997 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f104c0a1d80 0x7f104c0a4240 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1060198f50 tx=0x7f1048005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:35.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.206+0000 7f1057fff640 1 -- 172.21.15.16:0/957342997 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f105009ab40 con 0x7f1060100670 2024-03-28T22:08:35.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.356+0000 7f1057fff640 1 -- 172.21.15.16:0/957342997 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f1050096ab0 con 0x7f1060100670 2024-03-28T22:08:35.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.427+0000 7f106482a640 1 -- 172.21.15.16:0/957342997 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f10600fdd90 con 0x7f104c0a1d80 2024-03-28T22:08:35.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.432+0000 7f1057fff640 1 -- 172.21.15.16:0/957342997 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f10600fdd90 con 0x7f104c0a1d80 2024-03-28T22:08:35.435 INFO:teuthology.orchestra.run.smithi016.stdout:/volumes/qa/sv_0/63908082-7eb1-4bf6-a9b6-645788b838d9 2024-03-28T22:08:35.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:35 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/988168491' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-28T22:08:35.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:35 smithi016 ceph-mon[32374]: mds.? [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] up:active 2024-03-28T22:08:35.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:35 smithi016 ceph-mon[32374]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 9 up:standby 2024-03-28T22:08:35.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:35 smithi016 ceph-mon[32374]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T22:08:35.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:35 smithi016 ceph-mon[32374]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:08:35.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:35 smithi016 ceph-mon[32374]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:08:35.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:35 smithi016 ceph-mon[32374]: pgmap v289: 97 pgs: 97 active+clean; 586 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 5.3 KiB/s wr, 15 op/s 2024-03-28T22:08:35.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.434+0000 7f106482a640 1 -- 172.21.15.16:0/957342997 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f104c0a1d80 msgr2=0x7f104c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:35.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.434+0000 7f106482a640 1 --2- 172.21.15.16:0/957342997 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f104c0a1d80 0x7f104c0a4240 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1060198f50 tx=0x7f1048005f90 comp rx=0 tx=0).stop 2024-03-28T22:08:35.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.434+0000 7f106482a640 1 -- 172.21.15.16:0/957342997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1060100670 msgr2=0x7f1060197a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:35.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.434+0000 7f106482a640 1 --2- 172.21.15.16:0/957342997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1060100670 0x7f1060197a40 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f105000d770 tx=0x7f105000dc30 comp rx=0 tx=0).stop 2024-03-28T22:08:35.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.434+0000 7f106482a640 1 -- 172.21.15.16:0/957342997 shutdown_connections 2024-03-28T22:08:35.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.434+0000 7f106482a640 1 --2- 172.21.15.16:0/957342997 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f104c0a1d80 0x7f104c0a4240 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:35.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.434+0000 7f106482a640 1 --2- 172.21.15.16:0/957342997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1060102220 0x7f106019eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:35.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.434+0000 7f106482a640 1 --2- 172.21.15.16:0/957342997 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1060101860 0x7f1060197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:35.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.434+0000 7f106482a640 1 --2- 172.21.15.16:0/957342997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1060100670 0x7f1060197a40 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:35.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.434+0000 7f106482a640 1 -- 172.21.15.16:0/957342997 >> 172.21.15.16:0/957342997 conn(0x7f10600fbbe0 msgr2=0x7f10600fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:35.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.434+0000 7f106482a640 1 -- 172.21.15.16:0/957342997 shutdown_connections 2024-03-28T22:08:35.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.434+0000 7f106482a640 1 -- 172.21.15.16:0/957342997 wait complete. 2024-03-28T22:08:35.448 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-28T22:08:35.547 INFO:journalctl@ceph.mds.b.smithi053.stdout:Mar 28 22:08:35 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-b[66455]: 2024-03-28T22:08:35.227+0000 7f7b61151700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T22:08:35.548 INFO:journalctl@ceph.mds.b.smithi053.stdout:Mar 28 22:08:35 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-b[66455]: 2024-03-28T22:08:35.227+0000 7f7b61151700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T22:08:35.548 INFO:journalctl@ceph.mds.b.smithi053.stdout:Mar 28 22:08:35 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-b[66455]: 2024-03-28T22:08:35.227+0000 7f7b61151700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-28T22:08:35.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:35 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/988168491' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-28T22:08:35.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:35 smithi053 ceph-mon[35979]: mds.? [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] up:active 2024-03-28T22:08:35.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:35 smithi053 ceph-mon[35979]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 9 up:standby 2024-03-28T22:08:35.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:35 smithi053 ceph-mon[35979]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T22:08:35.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:35 smithi053 ceph-mon[35979]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:08:35.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:35 smithi053 ceph-mon[35979]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:08:35.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:35 smithi053 ceph-mon[35979]: pgmap v289: 97 pgs: 97 active+clean; 586 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 5.3 KiB/s wr, 15 op/s 2024-03-28T22:08:35.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.552+0000 7f0544165640 1 -- 172.21.15.16:0/3302381025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f053c100660 msgr2=0x7f053c100a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:35.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.552+0000 7f0544165640 1 --2- 172.21.15.16:0/3302381025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f053c100660 0x7f053c100a80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0534009ca0 tx=0x7f053402c0f0 comp rx=0 tx=0).stop 2024-03-28T22:08:35.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.552+0000 7f0544165640 1 -- 172.21.15.16:0/3302381025 shutdown_connections 2024-03-28T22:08:35.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.552+0000 7f0544165640 1 --2- 172.21.15.16:0/3302381025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f053c102210 0x7f053c108aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:35.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.552+0000 7f0544165640 1 --2- 172.21.15.16:0/3302381025 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f053c101850 0x7f053c101cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:35.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.552+0000 7f0544165640 1 --2- 172.21.15.16:0/3302381025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f053c100660 0x7f053c100a80 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:35.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.552+0000 7f0544165640 1 -- 172.21.15.16:0/3302381025 >> 172.21.15.16:0/3302381025 conn(0x7f053c0fbbd0 msgr2=0x7f053c0fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:35.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.552+0000 7f0544165640 1 -- 172.21.15.16:0/3302381025 shutdown_connections 2024-03-28T22:08:35.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.552+0000 7f0544165640 1 -- 172.21.15.16:0/3302381025 wait complete. 2024-03-28T22:08:35.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.552+0000 7f0544165640 1 Processor -- start 2024-03-28T22:08:35.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.553+0000 7f0544165640 1 -- start start 2024-03-28T22:08:35.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.553+0000 7f0544165640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f053c100660 0x7f053c06a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:35.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.553+0000 7f0544165640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f053c101850 0x7f053c19c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:35.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.553+0000 7f0544165640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f053c102210 0x7f053c19ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:35.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.553+0000 7f0544165640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f053c10c2e0 con 0x7f053c100660 2024-03-28T22:08:35.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.553+0000 7f0544165640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f053c10c160 con 0x7f053c101850 2024-03-28T22:08:35.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.553+0000 7f0544165640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f053c10c460 con 0x7f053c102210 2024-03-28T22:08:35.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.553+0000 7f0541f01640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f053c100660 0x7f053c06a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:35.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.553+0000 7f0541f01640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f053c100660 0x7f053c06a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57618/0 (socket says 172.21.15.16:57618) 2024-03-28T22:08:35.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.553+0000 7f0541f01640 1 -- 172.21.15.16:0/2446140891 learned_addr learned my addr 172.21.15.16:0/2446140891 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:35.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.553+0000 7f0541700640 1 --2- 172.21.15.16:0/2446140891 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f053c101850 0x7f053c19c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:35.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.553+0000 7f0542702640 1 --2- 172.21.15.16:0/2446140891 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f053c102210 0x7f053c19ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:35.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.553+0000 7f0541f01640 1 -- 172.21.15.16:0/2446140891 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f053c102210 msgr2=0x7f053c19ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:35.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.553+0000 7f0541f01640 1 --2- 172.21.15.16:0/2446140891 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f053c102210 0x7f053c19ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:35.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.553+0000 7f0541f01640 1 -- 172.21.15.16:0/2446140891 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f053c101850 msgr2=0x7f053c19c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:35.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.553+0000 7f0541f01640 1 --2- 172.21.15.16:0/2446140891 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f053c101850 0x7f053c19c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:35.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.553+0000 7f0541f01640 1 -- 172.21.15.16:0/2446140891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f053c19f760 con 0x7f053c100660 2024-03-28T22:08:35.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.553+0000 7f0541700640 1 --2- 172.21.15.16:0/2446140891 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f053c101850 0x7f053c19c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:08:35.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.553+0000 7f0541f01640 1 --2- 172.21.15.16:0/2446140891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f053c100660 0x7f053c06a9e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f05340324c0 tx=0x7f0534032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:35.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.553+0000 7f053affd640 1 -- 172.21.15.16:0/2446140891 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0534041020 con 0x7f053c100660 2024-03-28T22:08:35.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.553+0000 7f053affd640 1 -- 172.21.15.16:0/2446140891 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0534034cd0 con 0x7f053c100660 2024-03-28T22:08:35.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.553+0000 7f0544165640 1 -- 172.21.15.16:0/2446140891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f053c19f990 con 0x7f053c100660 2024-03-28T22:08:35.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.553+0000 7f053affd640 1 -- 172.21.15.16:0/2446140891 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0534039040 con 0x7f053c100660 2024-03-28T22:08:35.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.554+0000 7f0544165640 1 -- 172.21.15.16:0/2446140891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f053c19fed0 con 0x7f053c100660 2024-03-28T22:08:35.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.555+0000 7f0544165640 1 -- 172.21.15.16:0/2446140891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f053c107a00 con 0x7f053c100660 2024-03-28T22:08:35.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.556+0000 7f053affd640 1 -- 172.21.15.16:0/2446140891 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f0534044020 con 0x7f053c100660 2024-03-28T22:08:35.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.556+0000 7f053affd640 1 --2- 172.21.15.16:0/2446140891 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f05240a1e90 0x7f05240a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:35.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.556+0000 7f0541700640 1 --2- 172.21.15.16:0/2446140891 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f05240a1e90 0x7f05240a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:35.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.556+0000 7f053affd640 1 -- 172.21.15.16:0/2446140891 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f05340efa20 con 0x7f053c100660 2024-03-28T22:08:35.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.557+0000 7f0541700640 1 --2- 172.21.15.16:0/2446140891 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f05240a1e90 0x7f05240a4350 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f053c06a750 tx=0x7f053000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:35.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.561+0000 7f053affd640 1 -- 172.21.15.16:0/2446140891 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f053404b020 con 0x7f053c100660 2024-03-28T22:08:35.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:35 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/988168491' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-28T22:08:35.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:35 smithi063 ceph-mon[36249]: mds.? [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] up:active 2024-03-28T22:08:35.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:35 smithi063 ceph-mon[36249]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 9 up:standby 2024-03-28T22:08:35.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:35 smithi063 ceph-mon[36249]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T22:08:35.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:35 smithi063 ceph-mon[36249]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:08:35.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:35 smithi063 ceph-mon[36249]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:08:35.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:35 smithi063 ceph-mon[36249]: pgmap v289: 97 pgs: 97 active+clean; 586 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 5.3 KiB/s wr, 15 op/s 2024-03-28T22:08:35.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.770+0000 7f0544165640 1 -- 172.21.15.16:0/2446140891 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f053c061c20 con 0x7f05240a1e90 2024-03-28T22:08:35.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.801+0000 7f053affd640 1 -- 172.21.15.16:0/2446140891 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f053c061c20 con 0x7f05240a1e90 2024-03-28T22:08:35.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.803+0000 7f0544165640 1 -- 172.21.15.16:0/2446140891 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f05240a1e90 msgr2=0x7f05240a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:35.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.803+0000 7f0544165640 1 --2- 172.21.15.16:0/2446140891 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f05240a1e90 0x7f05240a4350 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f053c06a750 tx=0x7f053000b390 comp rx=0 tx=0).stop 2024-03-28T22:08:35.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.803+0000 7f0544165640 1 -- 172.21.15.16:0/2446140891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f053c100660 msgr2=0x7f053c06a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:35.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.803+0000 7f0544165640 1 --2- 172.21.15.16:0/2446140891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f053c100660 0x7f053c06a9e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f05340324c0 tx=0x7f0534032f40 comp rx=0 tx=0).stop 2024-03-28T22:08:35.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.803+0000 7f0544165640 1 -- 172.21.15.16:0/2446140891 shutdown_connections 2024-03-28T22:08:35.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.803+0000 7f0544165640 1 --2- 172.21.15.16:0/2446140891 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f05240a1e90 0x7f05240a4350 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:35.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.803+0000 7f0544165640 1 --2- 172.21.15.16:0/2446140891 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f053c102210 0x7f053c19ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:35.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.803+0000 7f0544165640 1 --2- 172.21.15.16:0/2446140891 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f053c101850 0x7f053c19c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:35.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.803+0000 7f0544165640 1 --2- 172.21.15.16:0/2446140891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f053c100660 0x7f053c06a9e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:35.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.803+0000 7f0544165640 1 -- 172.21.15.16:0/2446140891 >> 172.21.15.16:0/2446140891 conn(0x7f053c0fbbd0 msgr2=0x7f053c104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:35.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.803+0000 7f0544165640 1 -- 172.21.15.16:0/2446140891 shutdown_connections 2024-03-28T22:08:35.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.803+0000 7f0544165640 1 -- 172.21.15.16:0/2446140891 wait complete. 2024-03-28T22:08:35.817 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-28T22:08:35.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.921+0000 7f22af3e9640 1 -- 172.21.15.16:0/2157941662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22a81021f0 msgr2=0x7f22a81045e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:35.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.921+0000 7f22af3e9640 1 --2- 172.21.15.16:0/2157941662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22a81021f0 0x7f22a81045e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f22a4009900 tx=0x7f22a402bc30 comp rx=0 tx=0).stop 2024-03-28T22:08:35.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.921+0000 7f22af3e9640 1 -- 172.21.15.16:0/2157941662 shutdown_connections 2024-03-28T22:08:35.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.921+0000 7f22af3e9640 1 --2- 172.21.15.16:0/2157941662 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22a8107450 0x7f22a81098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:35.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.921+0000 7f22af3e9640 1 --2- 172.21.15.16:0/2157941662 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f22a8104b20 0x7f22a8106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:35.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.921+0000 7f22af3e9640 1 --2- 172.21.15.16:0/2157941662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22a81021f0 0x7f22a81045e0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:35.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.921+0000 7f22af3e9640 1 -- 172.21.15.16:0/2157941662 >> 172.21.15.16:0/2157941662 conn(0x7f22a80fbbe0 msgr2=0x7f22a80fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:35.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.922+0000 7f22af3e9640 1 -- 172.21.15.16:0/2157941662 shutdown_connections 2024-03-28T22:08:35.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.922+0000 7f22af3e9640 1 -- 172.21.15.16:0/2157941662 wait complete. 2024-03-28T22:08:35.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.922+0000 7f22af3e9640 1 Processor -- start 2024-03-28T22:08:35.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.922+0000 7f22af3e9640 1 -- start start 2024-03-28T22:08:35.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.922+0000 7f22af3e9640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22a81021f0 0x7f22a8197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:35.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.922+0000 7f22af3e9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22a8104b20 0x7f22a8198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:35.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.922+0000 7f22af3e9640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f22a8107450 0x7f22a819f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:35.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.922+0000 7f22af3e9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f22a810d420 con 0x7f22a8104b20 2024-03-28T22:08:35.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.922+0000 7f22af3e9640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f22a810d2a0 con 0x7f22a81021f0 2024-03-28T22:08:35.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.922+0000 7f22af3e9640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f22a810d5a0 con 0x7f22a8107450 2024-03-28T22:08:35.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.922+0000 7f22ac984640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22a8104b20 0x7f22a8198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:35.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.922+0000 7f22ac984640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22a8104b20 0x7f22a8198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57624/0 (socket says 172.21.15.16:57624) 2024-03-28T22:08:35.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.922+0000 7f22ac984640 1 -- 172.21.15.16:0/510118140 learned_addr learned my addr 172.21.15.16:0/510118140 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:35.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.922+0000 7f22ad185640 1 --2- 172.21.15.16:0/510118140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22a81021f0 0x7f22a8197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:35.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.922+0000 7f22ad986640 1 --2- 172.21.15.16:0/510118140 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f22a8107450 0x7f22a819f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:35.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.923+0000 7f22ac984640 1 -- 172.21.15.16:0/510118140 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f22a8107450 msgr2=0x7f22a819f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:35.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.923+0000 7f22ac984640 1 --2- 172.21.15.16:0/510118140 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f22a8107450 0x7f22a819f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:35.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.923+0000 7f22ac984640 1 -- 172.21.15.16:0/510118140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22a81021f0 msgr2=0x7f22a8197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:35.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.923+0000 7f22ac984640 1 --2- 172.21.15.16:0/510118140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22a81021f0 0x7f22a8197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:35.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.923+0000 7f22ac984640 1 -- 172.21.15.16:0/510118140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22a819f710 con 0x7f22a8104b20 2024-03-28T22:08:35.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.923+0000 7f22ad986640 1 --2- 172.21.15.16:0/510118140 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f22a8107450 0x7f22a819f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:08:35.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.923+0000 7f22ac984640 1 --2- 172.21.15.16:0/510118140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22a8104b20 0x7f22a8198030 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f229c00e9f0 tx=0x7f229c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:35.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.923+0000 7f229a7fc640 1 -- 172.21.15.16:0/510118140 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f229c00ccc0 con 0x7f22a8104b20 2024-03-28T22:08:35.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.923+0000 7f229a7fc640 1 -- 172.21.15.16:0/510118140 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f229c005500 con 0x7f22a8104b20 2024-03-28T22:08:35.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.923+0000 7f22af3e9640 1 -- 172.21.15.16:0/510118140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22a819fa00 con 0x7f22a8104b20 2024-03-28T22:08:35.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.923+0000 7f229a7fc640 1 -- 172.21.15.16:0/510118140 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f229c010690 con 0x7f22a8104b20 2024-03-28T22:08:35.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.923+0000 7f22af3e9640 1 -- 172.21.15.16:0/510118140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22a819ff00 con 0x7f22a8104b20 2024-03-28T22:08:35.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.924+0000 7f22af3e9640 1 -- 172.21.15.16:0/510118140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22a80fd6d0 con 0x7f22a8104b20 2024-03-28T22:08:35.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.925+0000 7f229a7fc640 1 -- 172.21.15.16:0/510118140 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f229c0046c0 con 0x7f22a8104b20 2024-03-28T22:08:35.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.926+0000 7f229a7fc640 1 --2- 172.21.15.16:0/510118140 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f22940a1e10 0x7f22940a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:35.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.926+0000 7f22ad185640 1 --2- 172.21.15.16:0/510118140 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f22940a1e10 0x7f22940a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:35.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.926+0000 7f229a7fc640 1 -- 172.21.15.16:0/510118140 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f229c0ce280 con 0x7f22a8104b20 2024-03-28T22:08:35.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.926+0000 7f22ad185640 1 --2- 172.21.15.16:0/510118140 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f22940a1e10 0x7f22940a42d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f22a402c7b0 tx=0x7f22a4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:35.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:35.930+0000 7f229a7fc640 1 -- 172.21.15.16:0/510118140 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f229c09b600 con 0x7f22a8104b20 2024-03-28T22:08:36.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.143+0000 7f22af3e9640 1 -- 172.21.15.16:0/510118140 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f22a8061b90 con 0x7f22940a1e10 2024-03-28T22:08:36.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.147+0000 7f229a7fc640 1 -- 172.21.15.16:0/510118140 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f22a8061b90 con 0x7f22940a1e10 2024-03-28T22:08:36.150 INFO:teuthology.orchestra.run.smithi016.stdout:/volumes/qa/sv_1/ea9205a6-f61d-41bc-b8e4-5f55e6a92aca 2024-03-28T22:08:36.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.149+0000 7f22af3e9640 1 -- 172.21.15.16:0/510118140 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f22940a1e10 msgr2=0x7f22940a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:36.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.149+0000 7f22af3e9640 1 --2- 172.21.15.16:0/510118140 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f22940a1e10 0x7f22940a42d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f22a402c7b0 tx=0x7f22a4005fd0 comp rx=0 tx=0).stop 2024-03-28T22:08:36.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.149+0000 7f22af3e9640 1 -- 172.21.15.16:0/510118140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22a8104b20 msgr2=0x7f22a8198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:36.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.149+0000 7f22af3e9640 1 --2- 172.21.15.16:0/510118140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22a8104b20 0x7f22a8198030 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f229c00e9f0 tx=0x7f229c00eeb0 comp rx=0 tx=0).stop 2024-03-28T22:08:36.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.149+0000 7f22af3e9640 1 -- 172.21.15.16:0/510118140 shutdown_connections 2024-03-28T22:08:36.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.149+0000 7f22af3e9640 1 --2- 172.21.15.16:0/510118140 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f22940a1e10 0x7f22940a42d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:36.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.149+0000 7f22af3e9640 1 --2- 172.21.15.16:0/510118140 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f22a8107450 0x7f22a819f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:36.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.149+0000 7f22af3e9640 1 --2- 172.21.15.16:0/510118140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22a8104b20 0x7f22a8198030 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:36.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.149+0000 7f22af3e9640 1 --2- 172.21.15.16:0/510118140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22a81021f0 0x7f22a8197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:36.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.149+0000 7f22af3e9640 1 -- 172.21.15.16:0/510118140 >> 172.21.15.16:0/510118140 conn(0x7f22a80fbbe0 msgr2=0x7f22a8102c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:36.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.150+0000 7f22af3e9640 1 -- 172.21.15.16:0/510118140 shutdown_connections 2024-03-28T22:08:36.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.150+0000 7f22af3e9640 1 -- 172.21.15.16:0/510118140 wait complete. 2024-03-28T22:08:36.163 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:08:36.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.262+0000 7fa32a014640 1 -- 172.21.15.16:0/1851642409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3241006f0 msgr2=0x7fa324100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:36.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.262+0000 7fa32a014640 1 --2- 172.21.15.16:0/1851642409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3241006f0 0x7fa324100b10 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa314009900 tx=0x7fa31402bc30 comp rx=0 tx=0).stop 2024-03-28T22:08:36.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.262+0000 7fa32a014640 1 -- 172.21.15.16:0/1851642409 shutdown_connections 2024-03-28T22:08:36.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.262+0000 7fa32a014640 1 --2- 172.21.15.16:0/1851642409 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3241022a0 0x7fa324108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:36.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.262+0000 7fa32a014640 1 --2- 172.21.15.16:0/1851642409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa3241018e0 0x7fa324101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:36.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.262+0000 7fa32a014640 1 --2- 172.21.15.16:0/1851642409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3241006f0 0x7fa324100b10 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:36.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.262+0000 7fa32a014640 1 -- 172.21.15.16:0/1851642409 >> 172.21.15.16:0/1851642409 conn(0x7fa3240fbc80 msgr2=0x7fa3240fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:36.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.262+0000 7fa32a014640 1 -- 172.21.15.16:0/1851642409 shutdown_connections 2024-03-28T22:08:36.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.262+0000 7fa32a014640 1 -- 172.21.15.16:0/1851642409 wait complete. 2024-03-28T22:08:36.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.263+0000 7fa32a014640 1 Processor -- start 2024-03-28T22:08:36.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.263+0000 7fa32a014640 1 -- start start 2024-03-28T22:08:36.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.263+0000 7fa32a014640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3241006f0 0x7fa324197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:36.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.263+0000 7fa32a014640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3241018e0 0x7fa324198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:36.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.263+0000 7fa32a014640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa3241022a0 0x7fa32419f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:36.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.263+0000 7fa32a014640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa32410c300 con 0x7fa3241018e0 2024-03-28T22:08:36.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.263+0000 7fa32a014640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fa32410c180 con 0x7fa3241006f0 2024-03-28T22:08:36.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.263+0000 7fa32a014640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa32410c480 con 0x7fa3241022a0 2024-03-28T22:08:36.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.263+0000 7fa322ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3241018e0 0x7fa324198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:36.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.263+0000 7fa322ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3241018e0 0x7fa324198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57642/0 (socket says 172.21.15.16:57642) 2024-03-28T22:08:36.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.263+0000 7fa322ffd640 1 -- 172.21.15.16:0/3668659403 learned_addr learned my addr 172.21.15.16:0/3668659403 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:36.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.263+0000 7fa3237fe640 1 --2- 172.21.15.16:0/3668659403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3241006f0 0x7fa324197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:36.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.264+0000 7fa323fff640 1 --2- 172.21.15.16:0/3668659403 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa3241022a0 0x7fa32419f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:36.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.264+0000 7fa322ffd640 1 -- 172.21.15.16:0/3668659403 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa3241022a0 msgr2=0x7fa32419f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:36.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.264+0000 7fa322ffd640 1 --2- 172.21.15.16:0/3668659403 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa3241022a0 0x7fa32419f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:36.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.264+0000 7fa322ffd640 1 -- 172.21.15.16:0/3668659403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3241006f0 msgr2=0x7fa324197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:36.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.264+0000 7fa322ffd640 1 --2- 172.21.15.16:0/3668659403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3241006f0 0x7fa324197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:36.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.264+0000 7fa322ffd640 1 -- 172.21.15.16:0/3668659403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa32419f740 con 0x7fa3241018e0 2024-03-28T22:08:36.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.264+0000 7fa323fff640 1 --2- 172.21.15.16:0/3668659403 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa3241022a0 0x7fa32419f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:08:36.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.264+0000 7fa322ffd640 1 --2- 172.21.15.16:0/3668659403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3241018e0 0x7fa324198060 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa31800d770 tx=0x7fa31800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:36.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.264+0000 7fa320ff9640 1 -- 172.21.15.16:0/3668659403 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa318017070 con 0x7fa3241018e0 2024-03-28T22:08:36.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.264+0000 7fa32a014640 1 -- 172.21.15.16:0/3668659403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa32419fa30 con 0x7fa3241018e0 2024-03-28T22:08:36.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.264+0000 7fa320ff9640 1 -- 172.21.15.16:0/3668659403 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa318005500 con 0x7fa3241018e0 2024-03-28T22:08:36.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.264+0000 7fa320ff9640 1 -- 172.21.15.16:0/3668659403 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa318004e00 con 0x7fa3241018e0 2024-03-28T22:08:36.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.264+0000 7fa32a014640 1 -- 172.21.15.16:0/3668659403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa32419ff30 con 0x7fa3241018e0 2024-03-28T22:08:36.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.265+0000 7fa32a014640 1 -- 172.21.15.16:0/3668659403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3241a0160 con 0x7fa3241018e0 2024-03-28T22:08:36.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.266+0000 7fa320ff9640 1 -- 172.21.15.16:0/3668659403 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fa318004fa0 con 0x7fa3241018e0 2024-03-28T22:08:36.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.266+0000 7fa320ff9640 1 --2- 172.21.15.16:0/3668659403 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fa30c0a1e10 0x7fa30c0a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:36.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.267+0000 7fa3237fe640 1 --2- 172.21.15.16:0/3668659403 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fa30c0a1e10 0x7fa30c0a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:36.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.267+0000 7fa320ff9640 1 -- 172.21.15.16:0/3668659403 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fa3180cd7c0 con 0x7fa3241018e0 2024-03-28T22:08:36.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.267+0000 7fa3237fe640 1 --2- 172.21.15.16:0/3668659403 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fa30c0a1e10 0x7fa30c0a42d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa314000c00 tx=0x7fa314005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:36.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.271+0000 7fa320ff9640 1 -- 172.21.15.16:0/3668659403 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa31809ac40 con 0x7fa3241018e0 2024-03-28T22:08:36.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:36 smithi016 ceph-mon[32374]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-28T22:08:36.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:36 smithi016 ceph-mon[32374]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:08:36.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:36 smithi016 ceph-mon[32374]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:08:36.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.491+0000 7fa32a014640 1 -- 172.21.15.16:0/3668659403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa3241a0330 con 0x7fa3241018e0 2024-03-28T22:08:36.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.492+0000 7fa320ff9640 1 -- 172.21.15.16:0/3668659403 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11851 (secure 0 0 0) 0x7fa31809a540 con 0x7fa3241018e0 2024-03-28T22:08:36.495 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:08:36.495 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/1644581227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1644581227},{"type":"v1","addr":"172.21.15.16:6839","nonce":1644581227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/2593062669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2593062669},{"type":"v1","addr":"172.21.15.16:6841","nonce":2593062669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:08:29.672976+0000","modified":"2024-03-28T22:08:34.341708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24566,"mds_1":24524,"mds_2":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24466":{"gid":24466,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/247249857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":247249857},{"type":"v1","addr":"172.21.15.63:6835","nonce":247249857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6837/2324231032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":2324231032},{"type":"v1","addr":"172.21.15.63:6837","nonce":2324231032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6839/1177197077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":1177197077},{"type":"v1","addr":"172.21.15.63:6839","nonce":1177197077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"e","rank":2,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.53:6835/350727634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":350727634},{"type":"v1","addr":"172.21.15.53:6835","nonce":350727634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/349877573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":349877573},{"type":"v1","addr":"172.21.15.53:6837","nonce":349877573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6839/3714153956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":3714153956},{"type":"v1","addr":"172.21.15.53:6839","nonce":3714153956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24566,"qdb_cluster":[24500,24566,24524]},"id":1}]} 2024-03-28T22:08:36.495 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 20 2024-03-28T22:08:36.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.494+0000 7fa32a014640 1 -- 172.21.15.16:0/3668659403 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fa30c0a1e10 msgr2=0x7fa30c0a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:36.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.494+0000 7fa32a014640 1 --2- 172.21.15.16:0/3668659403 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fa30c0a1e10 0x7fa30c0a42d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa314000c00 tx=0x7fa314005fd0 comp rx=0 tx=0).stop 2024-03-28T22:08:36.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.494+0000 7fa32a014640 1 -- 172.21.15.16:0/3668659403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3241018e0 msgr2=0x7fa324198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:36.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.494+0000 7fa32a014640 1 --2- 172.21.15.16:0/3668659403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3241018e0 0x7fa324198060 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa31800d770 tx=0x7fa31800dc30 comp rx=0 tx=0).stop 2024-03-28T22:08:36.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.494+0000 7fa32a014640 1 -- 172.21.15.16:0/3668659403 shutdown_connections 2024-03-28T22:08:36.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.494+0000 7fa32a014640 1 --2- 172.21.15.16:0/3668659403 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fa30c0a1e10 0x7fa30c0a42d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:36.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.494+0000 7fa32a014640 1 --2- 172.21.15.16:0/3668659403 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa3241022a0 0x7fa32419f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:36.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.494+0000 7fa32a014640 1 --2- 172.21.15.16:0/3668659403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3241018e0 0x7fa324198060 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:36.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.494+0000 7fa32a014640 1 --2- 172.21.15.16:0/3668659403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3241006f0 0x7fa324197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:36.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.494+0000 7fa32a014640 1 -- 172.21.15.16:0/3668659403 >> 172.21.15.16:0/3668659403 conn(0x7fa3240fbc80 msgr2=0x7fa3240fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:36.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.494+0000 7fa32a014640 1 -- 172.21.15.16:0/3668659403 shutdown_connections 2024-03-28T22:08:36.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.494+0000 7fa32a014640 1 -- 172.21.15.16:0/3668659403 wait complete. 2024-03-28T22:08:36.508 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:08:36.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.608+0000 7f0600c30640 1 -- 172.21.15.16:0/4263179506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05fc100680 msgr2=0x7f05fc100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:36.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.608+0000 7f0600c30640 1 --2- 172.21.15.16:0/4263179506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05fc100680 0x7f05fc100aa0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f05ec009900 tx=0x7f05ec02bc30 comp rx=0 tx=0).stop 2024-03-28T22:08:36.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.609+0000 7f0600c30640 1 -- 172.21.15.16:0/4263179506 shutdown_connections 2024-03-28T22:08:36.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.609+0000 7f0600c30640 1 --2- 172.21.15.16:0/4263179506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f05fc102230 0x7f05fc108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:36.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.609+0000 7f0600c30640 1 --2- 172.21.15.16:0/4263179506 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05fc101870 0x7f05fc101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:36.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.609+0000 7f0600c30640 1 --2- 172.21.15.16:0/4263179506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05fc100680 0x7f05fc100aa0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:36.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.609+0000 7f0600c30640 1 -- 172.21.15.16:0/4263179506 >> 172.21.15.16:0/4263179506 conn(0x7f05fc0fbbf0 msgr2=0x7f05fc0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:36.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.609+0000 7f0600c30640 1 -- 172.21.15.16:0/4263179506 shutdown_connections 2024-03-28T22:08:36.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.609+0000 7f0600c30640 1 -- 172.21.15.16:0/4263179506 wait complete. 2024-03-28T22:08:36.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.609+0000 7f0600c30640 1 Processor -- start 2024-03-28T22:08:36.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.609+0000 7f0600c30640 1 -- start start 2024-03-28T22:08:36.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.609+0000 7f0600c30640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05fc100680 0x7f05fc197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:36.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.609+0000 7f0600c30640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f05fc101870 0x7f05fc198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:36.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.609+0000 7f0600c30640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05fc102230 0x7f05fc19f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:36.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.609+0000 7f0600c30640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f05fc10c300 con 0x7f05fc100680 2024-03-28T22:08:36.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.609+0000 7f0600c30640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f05fc10c180 con 0x7f05fc102230 2024-03-28T22:08:36.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.609+0000 7f0600c30640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f05fc10c480 con 0x7f05fc101870 2024-03-28T22:08:36.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.609+0000 7f05fa59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05fc100680 0x7f05fc197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:36.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.610+0000 7f05fa59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05fc100680 0x7f05fc197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57658/0 (socket says 172.21.15.16:57658) 2024-03-28T22:08:36.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.610+0000 7f05fa59c640 1 -- 172.21.15.16:0/2736808225 learned_addr learned my addr 172.21.15.16:0/2736808225 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:36.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.610+0000 7f05f9d9b640 1 --2- 172.21.15.16:0/2736808225 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f05fc101870 0x7f05fc198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:36.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.610+0000 7f05fad9d640 1 --2- 172.21.15.16:0/2736808225 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05fc102230 0x7f05fc19f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:36.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.610+0000 7f05fa59c640 1 -- 172.21.15.16:0/2736808225 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f05fc101870 msgr2=0x7f05fc198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:36.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.610+0000 7f05fa59c640 1 --2- 172.21.15.16:0/2736808225 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f05fc101870 0x7f05fc198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:36.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.610+0000 7f05fa59c640 1 -- 172.21.15.16:0/2736808225 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05fc102230 msgr2=0x7f05fc19f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:36.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.610+0000 7f05fa59c640 1 --2- 172.21.15.16:0/2736808225 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05fc102230 0x7f05fc19f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:36.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.610+0000 7f05fa59c640 1 -- 172.21.15.16:0/2736808225 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05fc19f6f0 con 0x7f05fc100680 2024-03-28T22:08:36.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.610+0000 7f05fa59c640 1 --2- 172.21.15.16:0/2736808225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05fc100680 0x7f05fc197ad0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f05ec000c00 tx=0x7f05ec02c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:36.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.610+0000 7f05eb7fe640 1 -- 172.21.15.16:0/2736808225 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05ec041070 con 0x7f05fc100680 2024-03-28T22:08:36.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.610+0000 7f0600c30640 1 -- 172.21.15.16:0/2736808225 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05fc19f980 con 0x7f05fc100680 2024-03-28T22:08:36.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.610+0000 7f05eb7fe640 1 -- 172.21.15.16:0/2736808225 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05ec004450 con 0x7f05fc100680 2024-03-28T22:08:36.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.610+0000 7f05eb7fe640 1 -- 172.21.15.16:0/2736808225 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05ec034440 con 0x7f05fc100680 2024-03-28T22:08:36.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.610+0000 7f0600c30640 1 -- 172.21.15.16:0/2736808225 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05fc19fe70 con 0x7f05fc100680 2024-03-28T22:08:36.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.611+0000 7f0600c30640 1 -- 172.21.15.16:0/2736808225 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05fc1a0040 con 0x7f05fc100680 2024-03-28T22:08:36.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.612+0000 7f05eb7fe640 1 -- 172.21.15.16:0/2736808225 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f05ec044070 con 0x7f05fc100680 2024-03-28T22:08:36.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.613+0000 7f05eb7fe640 1 --2- 172.21.15.16:0/2736808225 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f05e40a1e10 0x7f05e40a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:36.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.613+0000 7f05f9d9b640 1 --2- 172.21.15.16:0/2736808225 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f05e40a1e10 0x7f05e40a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:36.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.613+0000 7f05eb7fe640 1 -- 172.21.15.16:0/2736808225 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f05ec0ef6f0 con 0x7f05fc100680 2024-03-28T22:08:36.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.613+0000 7f05f9d9b640 1 --2- 172.21.15.16:0/2736808225 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f05e40a1e10 0x7f05e40a42d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f05fc198fe0 tx=0x7f05f0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:36.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.617+0000 7f05eb7fe640 1 -- 172.21.15.16:0/2736808225 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f05ec0bc6c0 con 0x7f05fc100680 2024-03-28T22:08:36.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:36 smithi063 ceph-mon[36249]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-28T22:08:36.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:36 smithi063 ceph-mon[36249]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:08:36.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:36 smithi063 ceph-mon[36249]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:08:36.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:36 smithi053 ceph-mon[35979]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-28T22:08:36.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:36 smithi053 ceph-mon[35979]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:08:36.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:36 smithi053 ceph-mon[35979]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:08:36.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.801+0000 7f0600c30640 1 -- 172.21.15.16:0/2736808225 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f05fc1a03d0 con 0x7f05fc100680 2024-03-28T22:08:36.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.802+0000 7f05eb7fe640 1 -- 172.21.15.16:0/2736808225 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22111 (secure 0 0 0) 0x7f05ec0c0760 con 0x7f05fc100680 2024-03-28T22:08:36.805 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:08:36.805 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":86,"fsid":"798d9884-ed4e-11ee-95d0-87774f69a715","created":"2024-03-28T22:00:27.216099+0000","modified":"2024-03-28T22:08:30.939084+0000","last_up_change":"2024-03-28T22:07:40.251141+0000","last_in_change":"2024-03-28T22:07:19.456451+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:03:59.327832+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:08:28.123529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:08:28.919037+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"46e7c8f8-ecc8-43d6-ae53-0655728f3240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6803","nonce":1589692483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6805","nonce":1589692483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6809","nonce":1589692483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6807","nonce":1589692483}]},"public_addr":"172.21.15.16:6803/1589692483","cluster_addr":"172.21.15.16:6805/1589692483","heartbeat_back_addr":"172.21.15.16:6809/1589692483","heartbeat_front_addr":"172.21.15.16:6807/1589692483","state":["exists","up"]},{"osd":1,"uuid":"af6d9c22-2830-4ce2-a547-011df0c6cbad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6801","nonce":795531234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6803","nonce":795531234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6807","nonce":795531234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6805","nonce":795531234}]},"public_addr":"172.21.15.53:6801/795531234","cluster_addr":"172.21.15.53:6803/795531234","heartbeat_back_addr":"172.21.15.53:6807/795531234","heartbeat_front_addr":"172.21.15.53:6805/795531234","state":["exists","up"]},{"osd":2,"uuid":"b10adca9-110c-4f87-85e0-b7542709d31a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6801","nonce":2214131086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6803","nonce":2214131086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6807","nonce":2214131086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6805","nonce":2214131086}]},"public_addr":"172.21.15.63:6801/2214131086","cluster_addr":"172.21.15.63:6803/2214131086","heartbeat_back_addr":"172.21.15.63:6807/2214131086","heartbeat_front_addr":"172.21.15.63:6805/2214131086","state":["exists","up"]},{"osd":3,"uuid":"c83cf4bb-395a-4786-abf7-74ca93a33eb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6811","nonce":13176789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6813","nonce":13176789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6817","nonce":13176789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6815","nonce":13176789}]},"public_addr":"172.21.15.16:6811/13176789","cluster_addr":"172.21.15.16:6813/13176789","heartbeat_back_addr":"172.21.15.16:6817/13176789","heartbeat_front_addr":"172.21.15.16:6815/13176789","state":["exists","up"]},{"osd":4,"uuid":"941b3e54-47d6-4e85-8069-e730c3fa8eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6809","nonce":772161849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6811","nonce":772161849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6815","nonce":772161849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6813","nonce":772161849}]},"public_addr":"172.21.15.53:6809/772161849","cluster_addr":"172.21.15.53:6811/772161849","heartbeat_back_addr":"172.21.15.53:6815/772161849","heartbeat_front_addr":"172.21.15.53:6813/772161849","state":["exists","up"]},{"osd":5,"uuid":"51d89826-1d60-40fc-8fd8-09bfe1ab4125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6809","nonce":1602554852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6811","nonce":1602554852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6815","nonce":1602554852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6813","nonce":1602554852}]},"public_addr":"172.21.15.63:6809/1602554852","cluster_addr":"172.21.15.63:6811/1602554852","heartbeat_back_addr":"172.21.15.63:6815/1602554852","heartbeat_front_addr":"172.21.15.63:6813/1602554852","state":["exists","up"]},{"osd":6,"uuid":"978efe5d-2c55-4ff8-8a1e-9f891388945e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6819","nonce":4290459224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6821","nonce":4290459224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6825","nonce":4290459224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6823","nonce":4290459224}]},"public_addr":"172.21.15.16:6819/4290459224","cluster_addr":"172.21.15.16:6821/4290459224","heartbeat_back_addr":"172.21.15.16:6825/4290459224","heartbeat_front_addr":"172.21.15.16:6823/4290459224","state":["exists","up"]},{"osd":7,"uuid":"50d489cb-dbb8-48df-b868-b29edf12e04d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6817","nonce":1056467525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6819","nonce":1056467525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6823","nonce":1056467525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6821","nonce":1056467525}]},"public_addr":"172.21.15.53:6817/1056467525","cluster_addr":"172.21.15.53:6819/1056467525","heartbeat_back_addr":"172.21.15.53:6823/1056467525","heartbeat_front_addr":"172.21.15.53:6821/1056467525","state":["exists","up"]},{"osd":8,"uuid":"49b39414-cc9c-4057-ae3f-cc109cd979da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6817","nonce":2918516794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6819","nonce":2918516794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6823","nonce":2918516794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6821","nonce":2918516794}]},"public_addr":"172.21.15.63:6817/2918516794","cluster_addr":"172.21.15.63:6819/2918516794","heartbeat_back_addr":"172.21.15.63:6823/2918516794","heartbeat_front_addr":"172.21.15.63:6821/2918516794","state":["exists","up"]},{"osd":9,"uuid":"4c873e51-b783-4eb6-83a3-fee31722d60b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6827","nonce":1585624367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6829","nonce":1585624367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6833","nonce":1585624367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6831","nonce":1585624367}]},"public_addr":"172.21.15.16:6827/1585624367","cluster_addr":"172.21.15.16:6829/1585624367","heartbeat_back_addr":"172.21.15.16:6833/1585624367","heartbeat_front_addr":"172.21.15.16:6831/1585624367","state":["exists","up"]},{"osd":10,"uuid":"085a78bf-e2ad-481a-85a3-71d1fc549a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6825","nonce":3357446484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6827","nonce":3357446484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6831","nonce":3357446484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6829","nonce":3357446484}]},"public_addr":"172.21.15.53:6825/3357446484","cluster_addr":"172.21.15.53:6827/3357446484","heartbeat_back_addr":"172.21.15.53:6831/3357446484","heartbeat_front_addr":"172.21.15.53:6829/3357446484","state":["exists","up"]},{"osd":11,"uuid":"13302aa3-08b9-4639-9042-27d850b4d5f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6825","nonce":2167866570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6827","nonce":2167866570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6831","nonce":2167866570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6829","nonce":2167866570}]},"public_addr":"172.21.15.63:6825/2167866570","cluster_addr":"172.21.15.63:6827/2167866570","heartbeat_back_addr":"172.21.15.63:6831/2167866570","heartbeat_front_addr":"172.21.15.63:6829/2167866570","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:07.450323+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:31.363172+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:55.295021+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:19.794760+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:44.063492+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:08.195064+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:33.655328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:57.639414+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:22.402394+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:47.462050+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:12.226307+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:37.495625+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3821408150":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6801/1832422779":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/996598130":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/1062264551":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/317557779":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6801/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/3631746518":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6800/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/3116652786":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6800/1832422779":"2024-03-29T22:01:28.993434+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:08:36.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.804+0000 7f0600c30640 1 -- 172.21.15.16:0/2736808225 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f05e40a1e10 msgr2=0x7f05e40a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:36.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.804+0000 7f0600c30640 1 --2- 172.21.15.16:0/2736808225 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f05e40a1e10 0x7f05e40a42d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f05fc198fe0 tx=0x7f05f0005f90 comp rx=0 tx=0).stop 2024-03-28T22:08:36.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.804+0000 7f0600c30640 1 -- 172.21.15.16:0/2736808225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05fc100680 msgr2=0x7f05fc197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:36.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.804+0000 7f0600c30640 1 --2- 172.21.15.16:0/2736808225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05fc100680 0x7f05fc197ad0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f05ec000c00 tx=0x7f05ec02c640 comp rx=0 tx=0).stop 2024-03-28T22:08:36.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.804+0000 7f0600c30640 1 -- 172.21.15.16:0/2736808225 shutdown_connections 2024-03-28T22:08:36.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.804+0000 7f0600c30640 1 --2- 172.21.15.16:0/2736808225 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f05e40a1e10 0x7f05e40a42d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:36.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.804+0000 7f0600c30640 1 --2- 172.21.15.16:0/2736808225 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05fc102230 0x7f05fc19f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:36.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.804+0000 7f0600c30640 1 --2- 172.21.15.16:0/2736808225 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f05fc101870 0x7f05fc198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:36.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.804+0000 7f0600c30640 1 --2- 172.21.15.16:0/2736808225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05fc100680 0x7f05fc197ad0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:36.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.804+0000 7f0600c30640 1 -- 172.21.15.16:0/2736808225 >> 172.21.15.16:0/2736808225 conn(0x7f05fc0fbbf0 msgr2=0x7f05fc068a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:36.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.804+0000 7f0600c30640 1 -- 172.21.15.16:0/2736808225 shutdown_connections 2024-03-28T22:08:36.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.804+0000 7f0600c30640 1 -- 172.21.15.16:0/2736808225 wait complete. 2024-03-28T22:08:36.819 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-28T22:08:36.819 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T22:08:36.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.919+0000 7f7dea61c640 1 -- 172.21.15.16:0/2328705148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7de4107460 msgr2=0x7f7de41098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:36.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.919+0000 7f7dea61c640 1 --2- 172.21.15.16:0/2328705148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7de4107460 0x7f7de41098b0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc009ca0 tx=0x7f7ddc02c0f0 comp rx=0 tx=0).stop 2024-03-28T22:08:36.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.919+0000 7f7dea61c640 1 -- 172.21.15.16:0/2328705148 shutdown_connections 2024-03-28T22:08:36.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.919+0000 7f7dea61c640 1 --2- 172.21.15.16:0/2328705148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7de4107460 0x7f7de41098b0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:36.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.919+0000 7f7dea61c640 1 --2- 172.21.15.16:0/2328705148 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7de4104b30 0x7f7de4106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:36.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.919+0000 7f7dea61c640 1 --2- 172.21.15.16:0/2328705148 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7de4102200 0x7f7de41045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:36.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.919+0000 7f7dea61c640 1 -- 172.21.15.16:0/2328705148 >> 172.21.15.16:0/2328705148 conn(0x7f7de40fbc10 msgr2=0x7f7de40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:36.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.919+0000 7f7dea61c640 1 -- 172.21.15.16:0/2328705148 shutdown_connections 2024-03-28T22:08:36.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.919+0000 7f7dea61c640 1 -- 172.21.15.16:0/2328705148 wait complete. 2024-03-28T22:08:36.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.919+0000 7f7dea61c640 1 Processor -- start 2024-03-28T22:08:36.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.920+0000 7f7dea61c640 1 -- start start 2024-03-28T22:08:36.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.920+0000 7f7dea61c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7de4102200 0x7f7de4197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:36.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.920+0000 7f7dea61c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7de4104b30 0x7f7de4197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:36.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.920+0000 7f7dea61c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7de4107460 0x7f7de419f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:36.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.920+0000 7f7dea61c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7de410d420 con 0x7f7de4102200 2024-03-28T22:08:36.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.920+0000 7f7dea61c640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7de410d2a0 con 0x7f7de4104b30 2024-03-28T22:08:36.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.920+0000 7f7dea61c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7de410d5a0 con 0x7f7de4107460 2024-03-28T22:08:36.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.920+0000 7f7de3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7de4102200 0x7f7de4197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:36.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.920+0000 7f7de3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7de4102200 0x7f7de4197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57664/0 (socket says 172.21.15.16:57664) 2024-03-28T22:08:36.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.920+0000 7f7de3fff640 1 -- 172.21.15.16:0/2304889969 learned_addr learned my addr 172.21.15.16:0/2304889969 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:36.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.920+0000 7f7de37fe640 1 --2- 172.21.15.16:0/2304889969 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7de4104b30 0x7f7de4197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:36.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.920+0000 7f7de8bb9640 1 --2- 172.21.15.16:0/2304889969 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7de4107460 0x7f7de419f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:36.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.920+0000 7f7de3fff640 1 -- 172.21.15.16:0/2304889969 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7de4107460 msgr2=0x7f7de419f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:36.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.920+0000 7f7de3fff640 1 --2- 172.21.15.16:0/2304889969 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7de4107460 0x7f7de419f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:36.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.920+0000 7f7de3fff640 1 -- 172.21.15.16:0/2304889969 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7de4104b30 msgr2=0x7f7de4197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:36.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.920+0000 7f7de3fff640 1 --2- 172.21.15.16:0/2304889969 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7de4104b30 0x7f7de4197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:36.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.920+0000 7f7de3fff640 1 -- 172.21.15.16:0/2304889969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7de419f680 con 0x7f7de4102200 2024-03-28T22:08:36.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.920+0000 7f7de3fff640 1 --2- 172.21.15.16:0/2304889969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7de4102200 0x7f7de4197a60 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc00d770 tx=0x7f7dcc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:36.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.921+0000 7f7de17fa640 1 -- 172.21.15.16:0/2304889969 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc017070 con 0x7f7de4102200 2024-03-28T22:08:36.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.921+0000 7f7de17fa640 1 -- 172.21.15.16:0/2304889969 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7dcc005500 con 0x7f7de4102200 2024-03-28T22:08:36.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.921+0000 7f7dea61c640 1 -- 172.21.15.16:0/2304889969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7de419f970 con 0x7f7de4102200 2024-03-28T22:08:36.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.921+0000 7f7de17fa640 1 -- 172.21.15.16:0/2304889969 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc004e00 con 0x7f7de4102200 2024-03-28T22:08:36.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.921+0000 7f7dea61c640 1 -- 172.21.15.16:0/2304889969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7de419fe70 con 0x7f7de4102200 2024-03-28T22:08:36.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.922+0000 7f7dea61c640 1 -- 172.21.15.16:0/2304889969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7de40fd700 con 0x7f7de4102200 2024-03-28T22:08:36.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.923+0000 7f7de17fa640 1 -- 172.21.15.16:0/2304889969 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f7dcc004fa0 con 0x7f7de4102200 2024-03-28T22:08:36.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.923+0000 7f7de17fa640 1 --2- 172.21.15.16:0/2304889969 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f7dd00a1e10 0x7f7dd00a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:36.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.923+0000 7f7de37fe640 1 --2- 172.21.15.16:0/2304889969 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f7dd00a1e10 0x7f7dd00a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:36.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.923+0000 7f7de17fa640 1 -- 172.21.15.16:0/2304889969 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f7dcc0cd710 con 0x7f7de4102200 2024-03-28T22:08:36.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.924+0000 7f7de37fe640 1 --2- 172.21.15.16:0/2304889969 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f7dd00a1e10 0x7f7dd00a42d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7de4198f70 tx=0x7f7dd4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:36.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:36.928+0000 7f7de17fa640 1 -- 172.21.15.16:0/2304889969 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7dcc09a540 con 0x7f7de4102200 2024-03-28T22:08:37.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.107+0000 7f7dea61c640 1 -- 172.21.15.16:0/2304889969 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7de4061b90 con 0x7f7dd00a1e10 2024-03-28T22:08:37.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.114+0000 7f7de17fa640 1 -- 172.21.15.16:0/2304889969 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258725 (secure 0 0 0) 0x7f7de4061b90 con 0x7f7dd00a1e10 2024-03-28T22:08:37.120 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:08:37.122 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-03-28T22:08:37.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.117+0000 7f7dea61c640 1 -- 172.21.15.16:0/2304889969 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f7dd00a1e10 msgr2=0x7f7dd00a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:37.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.117+0000 7f7dea61c640 1 --2- 172.21.15.16:0/2304889969 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f7dd00a1e10 0x7f7dd00a42d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7de4198f70 tx=0x7f7dd4005f90 comp rx=0 tx=0).stop 2024-03-28T22:08:37.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.117+0000 7f7dea61c640 1 -- 172.21.15.16:0/2304889969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7de4102200 msgr2=0x7f7de4197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:37.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.117+0000 7f7dea61c640 1 --2- 172.21.15.16:0/2304889969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7de4102200 0x7f7de4197a60 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc00d770 tx=0x7f7dcc00dc30 comp rx=0 tx=0).stop 2024-03-28T22:08:37.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.117+0000 7f7dea61c640 1 -- 172.21.15.16:0/2304889969 shutdown_connections 2024-03-28T22:08:37.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.117+0000 7f7dea61c640 1 --2- 172.21.15.16:0/2304889969 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f7dd00a1e10 0x7f7dd00a42d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:37.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.117+0000 7f7dea61c640 1 --2- 172.21.15.16:0/2304889969 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7de4107460 0x7f7de419f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:37.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.117+0000 7f7dea61c640 1 --2- 172.21.15.16:0/2304889969 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7de4104b30 0x7f7de4197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:37.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.117+0000 7f7dea61c640 1 --2- 172.21.15.16:0/2304889969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7de4102200 0x7f7de4197a60 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:37.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.117+0000 7f7dea61c640 1 -- 172.21.15.16:0/2304889969 >> 172.21.15.16:0/2304889969 conn(0x7f7de40fbc10 msgr2=0x7f7de4102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:37.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.117+0000 7f7dea61c640 1 -- 172.21.15.16:0/2304889969 shutdown_connections 2024-03-28T22:08:37.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.117+0000 7f7dea61c640 1 -- 172.21.15.16:0/2304889969 wait complete. 2024-03-28T22:08:37.131 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2024-03-28T22:08:37.073809+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":602390,"num_objects":57,"num_object_clones":0,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":161,"num_read_kb":138,"num_write":302,"num_write_kb":2985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":244,"ondisk_log_size":244,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378580,"kb_used_data":12776,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563244,"statfs":{"total":1151940427776,"available":1151552761856,"internally_reserved":0,"allocated":13082624,"data_stored":7085956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":374453562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12022,"num_objects":55,"num_object_clones":0,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":121,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.762427"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.320849+0000","last_change":"2024-03-28T22:08:29.962664+0000","last_active":"2024-03-28T22:08:32.320849+0000","last_peered":"2024-03-28T22:08:32.320849+0000","last_clean":"2024-03-28T22:08:32.320849+0000","last_became_active":"2024-03-28T22:08:29.962029+0000","last_became_peered":"2024-03-28T22:08:29.962029+0000","last_unstale":"2024-03-28T22:08:32.320849+0000","last_undegraded":"2024-03-28T22:08:32.320849+0000","last_fullsized":"2024-03-28T22:08:32.320849+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:20:42.832304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.159434+0000","last_change":"2024-03-28T22:08:29.965617+0000","last_active":"2024-03-28T22:08:31.159434+0000","last_peered":"2024-03-28T22:08:31.159434+0000","last_clean":"2024-03-28T22:08:31.159434+0000","last_became_active":"2024-03-28T22:08:29.965248+0000","last_became_peered":"2024-03-28T22:08:29.965248+0000","last_unstale":"2024-03-28T22:08:31.159434+0000","last_undegraded":"2024-03-28T22:08:31.159434+0000","last_fullsized":"2024-03-28T22:08:31.159434+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:37:46.703768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.159381+0000","last_change":"2024-03-28T22:08:29.958452+0000","last_active":"2024-03-28T22:08:31.159381+0000","last_peered":"2024-03-28T22:08:31.159381+0000","last_clean":"2024-03-28T22:08:31.159381+0000","last_became_active":"2024-03-28T22:08:29.958089+0000","last_became_peered":"2024-03-28T22:08:29.958089+0000","last_unstale":"2024-03-28T22:08:31.159381+0000","last_undegraded":"2024-03-28T22:08:31.159381+0000","last_fullsized":"2024-03-28T22:08:31.159381+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:10:44.084925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.315906+0000","last_change":"2024-03-28T22:08:29.964812+0000","last_active":"2024-03-28T22:08:32.315906+0000","last_peered":"2024-03-28T22:08:32.315906+0000","last_clean":"2024-03-28T22:08:32.315906+0000","last_became_active":"2024-03-28T22:08:29.963872+0000","last_became_peered":"2024-03-28T22:08:29.963872+0000","last_unstale":"2024-03-28T22:08:32.315906+0000","last_undegraded":"2024-03-28T22:08:32.315906+0000","last_fullsized":"2024-03-28T22:08:32.315906+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:08.491078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.316698+0000","last_change":"2024-03-28T22:08:29.965564+0000","last_active":"2024-03-28T22:08:32.316698+0000","last_peered":"2024-03-28T22:08:32.316698+0000","last_clean":"2024-03-28T22:08:32.316698+0000","last_became_active":"2024-03-28T22:08:29.962636+0000","last_became_peered":"2024-03-28T22:08:29.962636+0000","last_unstale":"2024-03-28T22:08:32.316698+0000","last_undegraded":"2024-03-28T22:08:32.316698+0000","last_fullsized":"2024-03-28T22:08:32.316698+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:03:50.903231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.316632+0000","last_change":"2024-03-28T22:08:29.962091+0000","last_active":"2024-03-28T22:08:32.316632+0000","last_peered":"2024-03-28T22:08:32.316632+0000","last_clean":"2024-03-28T22:08:32.316632+0000","last_became_active":"2024-03-28T22:08:29.961735+0000","last_became_peered":"2024-03-28T22:08:29.961735+0000","last_unstale":"2024-03-28T22:08:32.316632+0000","last_undegraded":"2024-03-28T22:08:32.316632+0000","last_fullsized":"2024-03-28T22:08:32.316632+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:40:09.853265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.383889+0000","last_change":"2024-03-28T22:08:29.961089+0000","last_active":"2024-03-28T22:08:31.383889+0000","last_peered":"2024-03-28T22:08:31.383889+0000","last_clean":"2024-03-28T22:08:31.383889+0000","last_became_active":"2024-03-28T22:08:29.960711+0000","last_became_peered":"2024-03-28T22:08:29.960711+0000","last_unstale":"2024-03-28T22:08:31.383889+0000","last_undegraded":"2024-03-28T22:08:31.383889+0000","last_fullsized":"2024-03-28T22:08:31.383889+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:05:14.617718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.941889+0000","last_change":"2024-03-28T22:08:29.965142+0000","last_active":"2024-03-28T22:08:30.941889+0000","last_peered":"2024-03-28T22:08:30.941889+0000","last_clean":"2024-03-28T22:08:30.941889+0000","last_became_active":"2024-03-28T22:08:29.964730+0000","last_became_peered":"2024-03-28T22:08:29.964730+0000","last_unstale":"2024-03-28T22:08:30.941889+0000","last_undegraded":"2024-03-28T22:08:30.941889+0000","last_fullsized":"2024-03-28T22:08:30.941889+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:23:43.229681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.943626+0000","last_change":"2024-03-28T22:08:29.958984+0000","last_active":"2024-03-28T22:08:30.943626+0000","last_peered":"2024-03-28T22:08:30.943626+0000","last_clean":"2024-03-28T22:08:30.943626+0000","last_became_active":"2024-03-28T22:08:29.958583+0000","last_became_peered":"2024-03-28T22:08:29.958583+0000","last_unstale":"2024-03-28T22:08:30.943626+0000","last_undegraded":"2024-03-28T22:08:30.943626+0000","last_fullsized":"2024-03-28T22:08:30.943626+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:05:11.361719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.317794+0000","last_change":"2024-03-28T22:08:29.962774+0000","last_active":"2024-03-28T22:08:32.317794+0000","last_peered":"2024-03-28T22:08:32.317794+0000","last_clean":"2024-03-28T22:08:32.317794+0000","last_became_active":"2024-03-28T22:08:29.961222+0000","last_became_peered":"2024-03-28T22:08:29.961222+0000","last_unstale":"2024-03-28T22:08:32.317794+0000","last_undegraded":"2024-03-28T22:08:32.317794+0000","last_fullsized":"2024-03-28T22:08:32.317794+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:47:05.310483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.384331+0000","last_change":"2024-03-28T22:08:29.964041+0000","last_active":"2024-03-28T22:08:31.384331+0000","last_peered":"2024-03-28T22:08:31.384331+0000","last_clean":"2024-03-28T22:08:31.384331+0000","last_became_active":"2024-03-28T22:08:29.963360+0000","last_became_peered":"2024-03-28T22:08:29.963360+0000","last_unstale":"2024-03-28T22:08:31.384331+0000","last_undegraded":"2024-03-28T22:08:31.384331+0000","last_fullsized":"2024-03-28T22:08:31.384331+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:32.921916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.315875+0000","last_change":"2024-03-28T22:08:29.966449+0000","last_active":"2024-03-28T22:08:32.315875+0000","last_peered":"2024-03-28T22:08:32.315875+0000","last_clean":"2024-03-28T22:08:32.315875+0000","last_became_active":"2024-03-28T22:08:29.966036+0000","last_became_peered":"2024-03-28T22:08:29.966036+0000","last_unstale":"2024-03-28T22:08:32.315875+0000","last_undegraded":"2024-03-28T22:08:32.315875+0000","last_fullsized":"2024-03-28T22:08:32.315875+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:12:15.057713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.316451+0000","last_change":"2024-03-28T22:08:29.958924+0000","last_active":"2024-03-28T22:08:31.316451+0000","last_peered":"2024-03-28T22:08:31.316451+0000","last_clean":"2024-03-28T22:08:31.316451+0000","last_became_active":"2024-03-28T22:08:29.958601+0000","last_became_peered":"2024-03-28T22:08:29.958601+0000","last_unstale":"2024-03-28T22:08:31.316451+0000","last_undegraded":"2024-03-28T22:08:31.316451+0000","last_fullsized":"2024-03-28T22:08:31.316451+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:53:37.070276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.159166+0000","last_change":"2024-03-28T22:08:29.965045+0000","last_active":"2024-03-28T22:08:31.159166+0000","last_peered":"2024-03-28T22:08:31.159166+0000","last_clean":"2024-03-28T22:08:31.159166+0000","last_became_active":"2024-03-28T22:08:29.964710+0000","last_became_peered":"2024-03-28T22:08:29.964710+0000","last_unstale":"2024-03-28T22:08:31.159166+0000","last_undegraded":"2024-03-28T22:08:31.159166+0000","last_fullsized":"2024-03-28T22:08:31.159166+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:03:11.491213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.159242+0000","last_change":"2024-03-28T22:08:29.957940+0000","last_active":"2024-03-28T22:08:31.159242+0000","last_peered":"2024-03-28T22:08:31.159242+0000","last_clean":"2024-03-28T22:08:31.159242+0000","last_became_active":"2024-03-28T22:08:29.957587+0000","last_became_peered":"2024-03-28T22:08:29.957587+0000","last_unstale":"2024-03-28T22:08:31.159242+0000","last_undegraded":"2024-03-28T22:08:31.159242+0000","last_fullsized":"2024-03-28T22:08:31.159242+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:03:58.557806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.943285+0000","last_change":"2024-03-28T22:08:29.957397+0000","last_active":"2024-03-28T22:08:30.943285+0000","last_peered":"2024-03-28T22:08:30.943285+0000","last_clean":"2024-03-28T22:08:30.943285+0000","last_became_active":"2024-03-28T22:08:29.957054+0000","last_became_peered":"2024-03-28T22:08:29.957054+0000","last_unstale":"2024-03-28T22:08:30.943285+0000","last_undegraded":"2024-03-28T22:08:30.943285+0000","last_fullsized":"2024-03-28T22:08:30.943285+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:35:28.500561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.320879+0000","last_change":"2024-03-28T22:08:29.963343+0000","last_active":"2024-03-28T22:08:32.320879+0000","last_peered":"2024-03-28T22:08:32.320879+0000","last_clean":"2024-03-28T22:08:32.320879+0000","last_became_active":"2024-03-28T22:08:29.962926+0000","last_became_peered":"2024-03-28T22:08:29.962926+0000","last_unstale":"2024-03-28T22:08:32.320879+0000","last_undegraded":"2024-03-28T22:08:32.320879+0000","last_fullsized":"2024-03-28T22:08:32.320879+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:15:39.133930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.960657+0000","last_change":"2024-03-28T22:08:29.955903+0000","last_active":"2024-03-28T22:08:31.960657+0000","last_peered":"2024-03-28T22:08:31.960657+0000","last_clean":"2024-03-28T22:08:31.960657+0000","last_became_active":"2024-03-28T22:08:29.955460+0000","last_became_peered":"2024-03-28T22:08:29.955460+0000","last_unstale":"2024-03-28T22:08:31.960657+0000","last_undegraded":"2024-03-28T22:08:31.960657+0000","last_fullsized":"2024-03-28T22:08:31.960657+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:36:12.961475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.654730+0000","last_change":"2024-03-28T22:08:29.954279+0000","last_active":"2024-03-28T22:08:31.654730+0000","last_peered":"2024-03-28T22:08:31.654730+0000","last_clean":"2024-03-28T22:08:31.654730+0000","last_became_active":"2024-03-28T22:08:29.953869+0000","last_became_peered":"2024-03-28T22:08:29.953869+0000","last_unstale":"2024-03-28T22:08:31.654730+0000","last_undegraded":"2024-03-28T22:08:31.654730+0000","last_fullsized":"2024-03-28T22:08:31.654730+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:05:52.247391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:33.348241+0000","last_change":"2024-03-28T22:08:29.327644+0000","last_active":"2024-03-28T22:08:33.348241+0000","last_peered":"2024-03-28T22:08:33.348241+0000","last_clean":"2024-03-28T22:08:33.348241+0000","last_became_active":"2024-03-28T22:08:29.327325+0000","last_became_peered":"2024-03-28T22:08:29.327325+0000","last_unstale":"2024-03-28T22:08:33.348241+0000","last_undegraded":"2024-03-28T22:08:33.348241+0000","last_fullsized":"2024-03-28T22:08:33.348241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:26:54.782703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.942406+0000","last_change":"2024-03-28T22:08:29.962905+0000","last_active":"2024-03-28T22:08:30.942406+0000","last_peered":"2024-03-28T22:08:30.942406+0000","last_clean":"2024-03-28T22:08:30.942406+0000","last_became_active":"2024-03-28T22:08:29.962559+0000","last_became_peered":"2024-03-28T22:08:29.962559+0000","last_unstale":"2024-03-28T22:08:30.942406+0000","last_undegraded":"2024-03-28T22:08:30.942406+0000","last_fullsized":"2024-03-28T22:08:30.942406+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:02.347584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.158323+0000","last_change":"2024-03-28T22:08:29.334871+0000","last_active":"2024-03-28T22:08:31.158323+0000","last_peered":"2024-03-28T22:08:31.158323+0000","last_clean":"2024-03-28T22:08:31.158323+0000","last_became_active":"2024-03-28T22:08:29.334565+0000","last_became_peered":"2024-03-28T22:08:29.334565+0000","last_unstale":"2024-03-28T22:08:31.158323+0000","last_undegraded":"2024-03-28T22:08:31.158323+0000","last_fullsized":"2024-03-28T22:08:31.158323+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:31:19.333496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.159562+0000","last_change":"2024-03-28T22:08:29.957622+0000","last_active":"2024-03-28T22:08:31.159562+0000","last_peered":"2024-03-28T22:08:31.159562+0000","last_clean":"2024-03-28T22:08:31.159562+0000","last_became_active":"2024-03-28T22:08:29.957208+0000","last_became_peered":"2024-03-28T22:08:29.957208+0000","last_unstale":"2024-03-28T22:08:31.159562+0000","last_undegraded":"2024-03-28T22:08:31.159562+0000","last_fullsized":"2024-03-28T22:08:31.159562+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:49:44.684203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:33.345708+0000","last_change":"2024-03-28T22:08:29.330291+0000","last_active":"2024-03-28T22:08:33.345708+0000","last_peered":"2024-03-28T22:08:33.345708+0000","last_clean":"2024-03-28T22:08:33.345708+0000","last_became_active":"2024-03-28T22:08:29.328955+0000","last_became_peered":"2024-03-28T22:08:29.328955+0000","last_unstale":"2024-03-28T22:08:33.345708+0000","last_undegraded":"2024-03-28T22:08:33.345708+0000","last_fullsized":"2024-03-28T22:08:33.345708+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:10.838059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.943129+0000","last_change":"2024-03-28T22:08:29.966681+0000","last_active":"2024-03-28T22:08:30.943129+0000","last_peered":"2024-03-28T22:08:30.943129+0000","last_clean":"2024-03-28T22:08:30.943129+0000","last_became_active":"2024-03-28T22:08:29.966023+0000","last_became_peered":"2024-03-28T22:08:29.966023+0000","last_unstale":"2024-03-28T22:08:30.943129+0000","last_undegraded":"2024-03-28T22:08:30.943129+0000","last_fullsized":"2024-03-28T22:08:30.943129+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:04:43.281259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.316909+0000","last_change":"2024-03-28T22:08:29.329026+0000","last_active":"2024-03-28T22:08:32.316909+0000","last_peered":"2024-03-28T22:08:32.316909+0000","last_clean":"2024-03-28T22:08:32.316909+0000","last_became_active":"2024-03-28T22:08:29.327570+0000","last_became_peered":"2024-03-28T22:08:29.327570+0000","last_unstale":"2024-03-28T22:08:32.316909+0000","last_undegraded":"2024-03-28T22:08:32.316909+0000","last_fullsized":"2024-03-28T22:08:32.316909+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:53:54.779073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.946065+0000","last_change":"2024-03-28T22:08:29.966455+0000","last_active":"2024-03-28T22:08:30.946065+0000","last_peered":"2024-03-28T22:08:30.946065+0000","last_clean":"2024-03-28T22:08:30.946065+0000","last_became_active":"2024-03-28T22:08:29.964451+0000","last_became_peered":"2024-03-28T22:08:29.964451+0000","last_unstale":"2024-03-28T22:08:30.946065+0000","last_undegraded":"2024-03-28T22:08:30.946065+0000","last_fullsized":"2024-03-28T22:08:30.946065+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:46:22.811146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'6","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:35.354234+0000","last_change":"2024-03-28T22:08:29.327397+0000","last_active":"2024-03-28T22:08:35.354234+0000","last_peered":"2024-03-28T22:08:35.354234+0000","last_clean":"2024-03-28T22:08:35.354234+0000","last_became_active":"2024-03-28T22:08:29.327009+0000","last_became_peered":"2024-03-28T22:08:29.327009+0000","last_unstale":"2024-03-28T22:08:35.354234+0000","last_undegraded":"2024-03-28T22:08:35.354234+0000","last_fullsized":"2024-03-28T22:08:35.354234+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:38:51.951934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.316406+0000","last_change":"2024-03-28T22:08:29.969821+0000","last_active":"2024-03-28T22:08:32.316406+0000","last_peered":"2024-03-28T22:08:32.316406+0000","last_clean":"2024-03-28T22:08:32.316406+0000","last_became_active":"2024-03-28T22:08:29.969386+0000","last_became_peered":"2024-03-28T22:08:29.969386+0000","last_unstale":"2024-03-28T22:08:32.316406+0000","last_undegraded":"2024-03-28T22:08:32.316406+0000","last_fullsized":"2024-03-28T22:08:32.316406+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:44:19.753083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:33.345509+0000","last_change":"2024-03-28T22:08:29.332385+0000","last_active":"2024-03-28T22:08:33.345509+0000","last_peered":"2024-03-28T22:08:33.345509+0000","last_clean":"2024-03-28T22:08:33.345509+0000","last_became_active":"2024-03-28T22:08:29.331956+0000","last_became_peered":"2024-03-28T22:08:29.331956+0000","last_unstale":"2024-03-28T22:08:33.345509+0000","last_undegraded":"2024-03-28T22:08:33.345509+0000","last_fullsized":"2024-03-28T22:08:33.345509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:10:04.428034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.945895+0000","last_change":"2024-03-28T22:08:29.968586+0000","last_active":"2024-03-28T22:08:30.945895+0000","last_peered":"2024-03-28T22:08:30.945895+0000","last_clean":"2024-03-28T22:08:30.945895+0000","last_became_active":"2024-03-28T22:08:29.968088+0000","last_became_peered":"2024-03-28T22:08:29.968088+0000","last_unstale":"2024-03-28T22:08:30.945895+0000","last_undegraded":"2024-03-28T22:08:30.945895+0000","last_fullsized":"2024-03-28T22:08:30.945895+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:21:03.051733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:33.347174+0000","last_change":"2024-03-28T22:08:29.327180+0000","last_active":"2024-03-28T22:08:33.347174+0000","last_peered":"2024-03-28T22:08:33.347174+0000","last_clean":"2024-03-28T22:08:33.347174+0000","last_became_active":"2024-03-28T22:08:29.326825+0000","last_became_peered":"2024-03-28T22:08:29.326825+0000","last_unstale":"2024-03-28T22:08:33.347174+0000","last_undegraded":"2024-03-28T22:08:33.347174+0000","last_fullsized":"2024-03-28T22:08:33.347174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:11:09.979970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.941917+0000","last_change":"2024-03-28T22:08:29.955299+0000","last_active":"2024-03-28T22:08:30.941917+0000","last_peered":"2024-03-28T22:08:30.941917+0000","last_clean":"2024-03-28T22:08:30.941917+0000","last_became_active":"2024-03-28T22:08:29.954949+0000","last_became_peered":"2024-03-28T22:08:29.954949+0000","last_unstale":"2024-03-28T22:08:30.941917+0000","last_undegraded":"2024-03-28T22:08:30.941917+0000","last_fullsized":"2024-03-28T22:08:30.941917+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:01:48.375251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.945248+0000","last_change":"2024-03-28T22:08:29.327261+0000","last_active":"2024-03-28T22:08:30.945248+0000","last_peered":"2024-03-28T22:08:30.945248+0000","last_clean":"2024-03-28T22:08:30.945248+0000","last_became_active":"2024-03-28T22:08:29.326910+0000","last_became_peered":"2024-03-28T22:08:29.326910+0000","last_unstale":"2024-03-28T22:08:30.945248+0000","last_undegraded":"2024-03-28T22:08:30.945248+0000","last_fullsized":"2024-03-28T22:08:30.945248+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:46:41.933372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.960634+0000","last_change":"2024-03-28T22:08:29.954891+0000","last_active":"2024-03-28T22:08:31.960634+0000","last_peered":"2024-03-28T22:08:31.960634+0000","last_clean":"2024-03-28T22:08:31.960634+0000","last_became_active":"2024-03-28T22:08:29.954519+0000","last_became_peered":"2024-03-28T22:08:29.954519+0000","last_unstale":"2024-03-28T22:08:31.960634+0000","last_undegraded":"2024-03-28T22:08:31.960634+0000","last_fullsized":"2024-03-28T22:08:31.960634+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:07:42.621189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.322199+0000","last_change":"2024-03-28T22:08:29.330180+0000","last_active":"2024-03-28T22:08:32.322199+0000","last_peered":"2024-03-28T22:08:32.322199+0000","last_clean":"2024-03-28T22:08:32.322199+0000","last_became_active":"2024-03-28T22:08:29.329835+0000","last_became_peered":"2024-03-28T22:08:29.329835+0000","last_unstale":"2024-03-28T22:08:32.322199+0000","last_undegraded":"2024-03-28T22:08:32.322199+0000","last_fullsized":"2024-03-28T22:08:32.322199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:48:39.906359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.943559+0000","last_change":"2024-03-28T22:08:29.967074+0000","last_active":"2024-03-28T22:08:30.943559+0000","last_peered":"2024-03-28T22:08:30.943559+0000","last_clean":"2024-03-28T22:08:30.943559+0000","last_became_active":"2024-03-28T22:08:29.966258+0000","last_became_peered":"2024-03-28T22:08:29.966258+0000","last_unstale":"2024-03-28T22:08:30.943559+0000","last_undegraded":"2024-03-28T22:08:30.943559+0000","last_fullsized":"2024-03-28T22:08:30.943559+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:25:44.986179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.318422+0000","last_change":"2024-03-28T22:08:29.326388+0000","last_active":"2024-03-28T22:08:32.318422+0000","last_peered":"2024-03-28T22:08:32.318422+0000","last_clean":"2024-03-28T22:08:32.318422+0000","last_became_active":"2024-03-28T22:08:29.326001+0000","last_became_peered":"2024-03-28T22:08:29.326001+0000","last_unstale":"2024-03-28T22:08:32.318422+0000","last_undegraded":"2024-03-28T22:08:32.318422+0000","last_fullsized":"2024-03-28T22:08:32.318422+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:31:56.227038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.159507+0000","last_change":"2024-03-28T22:08:29.959299+0000","last_active":"2024-03-28T22:08:31.159507+0000","last_peered":"2024-03-28T22:08:31.159507+0000","last_clean":"2024-03-28T22:08:31.159507+0000","last_became_active":"2024-03-28T22:08:29.958955+0000","last_became_peered":"2024-03-28T22:08:29.958955+0000","last_unstale":"2024-03-28T22:08:31.159507+0000","last_undegraded":"2024-03-28T22:08:31.159507+0000","last_fullsized":"2024-03-28T22:08:31.159507+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:11:06.775920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.321904+0000","last_change":"2024-03-28T22:08:29.326089+0000","last_active":"2024-03-28T22:08:32.321904+0000","last_peered":"2024-03-28T22:08:32.321904+0000","last_clean":"2024-03-28T22:08:32.321904+0000","last_became_active":"2024-03-28T22:08:29.325463+0000","last_became_peered":"2024-03-28T22:08:29.325463+0000","last_unstale":"2024-03-28T22:08:32.321904+0000","last_undegraded":"2024-03-28T22:08:32.321904+0000","last_fullsized":"2024-03-28T22:08:32.321904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:18:31.049729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.943352+0000","last_change":"2024-03-28T22:08:29.964248+0000","last_active":"2024-03-28T22:08:30.943352+0000","last_peered":"2024-03-28T22:08:30.943352+0000","last_clean":"2024-03-28T22:08:30.943352+0000","last_became_active":"2024-03-28T22:08:29.963897+0000","last_became_peered":"2024-03-28T22:08:29.963897+0000","last_unstale":"2024-03-28T22:08:30.943352+0000","last_undegraded":"2024-03-28T22:08:30.943352+0000","last_fullsized":"2024-03-28T22:08:30.943352+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:41:39.645360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'11","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:34.704050+0000","last_change":"2024-03-28T22:08:29.330732+0000","last_active":"2024-03-28T22:08:34.704050+0000","last_peered":"2024-03-28T22:08:34.704050+0000","last_clean":"2024-03-28T22:08:34.704050+0000","last_became_active":"2024-03-28T22:08:29.329231+0000","last_became_peered":"2024-03-28T22:08:29.329231+0000","last_unstale":"2024-03-28T22:08:34.704050+0000","last_undegraded":"2024-03-28T22:08:34.704050+0000","last_fullsized":"2024-03-28T22:08:34.704050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:16:39.260429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8212,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":12,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.316836+0000","last_change":"2024-03-28T22:08:29.965791+0000","last_active":"2024-03-28T22:08:32.316836+0000","last_peered":"2024-03-28T22:08:32.316836+0000","last_clean":"2024-03-28T22:08:32.316836+0000","last_became_active":"2024-03-28T22:08:29.965390+0000","last_became_peered":"2024-03-28T22:08:29.965390+0000","last_unstale":"2024-03-28T22:08:32.316836+0000","last_undegraded":"2024-03-28T22:08:32.316836+0000","last_fullsized":"2024-03-28T22:08:32.316836+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:43:53.325906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.317396+0000","last_change":"2024-03-28T22:08:29.325851+0000","last_active":"2024-03-28T22:08:32.317396+0000","last_peered":"2024-03-28T22:08:32.317396+0000","last_clean":"2024-03-28T22:08:32.317396+0000","last_became_active":"2024-03-28T22:08:29.325500+0000","last_became_peered":"2024-03-28T22:08:29.325500+0000","last_unstale":"2024-03-28T22:08:32.317396+0000","last_undegraded":"2024-03-28T22:08:32.317396+0000","last_fullsized":"2024-03-28T22:08:32.317396+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:04:25.151908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.316372+0000","last_change":"2024-03-28T22:08:29.953857+0000","last_active":"2024-03-28T22:08:31.316372+0000","last_peered":"2024-03-28T22:08:31.316372+0000","last_clean":"2024-03-28T22:08:31.316372+0000","last_became_active":"2024-03-28T22:08:29.953538+0000","last_became_peered":"2024-03-28T22:08:29.953538+0000","last_unstale":"2024-03-28T22:08:31.316372+0000","last_undegraded":"2024-03-28T22:08:31.316372+0000","last_fullsized":"2024-03-28T22:08:31.316372+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:40:32.404760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.321807+0000","last_change":"2024-03-28T22:08:29.327286+0000","last_active":"2024-03-28T22:08:32.321807+0000","last_peered":"2024-03-28T22:08:32.321807+0000","last_clean":"2024-03-28T22:08:32.321807+0000","last_became_active":"2024-03-28T22:08:29.326902+0000","last_became_peered":"2024-03-28T22:08:29.326902+0000","last_unstale":"2024-03-28T22:08:32.321807+0000","last_undegraded":"2024-03-28T22:08:32.321807+0000","last_fullsized":"2024-03-28T22:08:32.321807+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:14:58.669534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.384118+0000","last_change":"2024-03-28T22:08:29.959917+0000","last_active":"2024-03-28T22:08:31.384118+0000","last_peered":"2024-03-28T22:08:31.384118+0000","last_clean":"2024-03-28T22:08:31.384118+0000","last_became_active":"2024-03-28T22:08:29.959535+0000","last_became_peered":"2024-03-28T22:08:29.959535+0000","last_unstale":"2024-03-28T22:08:31.384118+0000","last_undegraded":"2024-03-28T22:08:31.384118+0000","last_fullsized":"2024-03-28T22:08:31.384118+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:35:04.725585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'4","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:35.356359+0000","last_change":"2024-03-28T22:08:29.327740+0000","last_active":"2024-03-28T22:08:35.356359+0000","last_peered":"2024-03-28T22:08:35.356359+0000","last_clean":"2024-03-28T22:08:35.356359+0000","last_became_active":"2024-03-28T22:08:29.326664+0000","last_became_peered":"2024-03-28T22:08:29.326664+0000","last_unstale":"2024-03-28T22:08:35.356359+0000","last_undegraded":"2024-03-28T22:08:35.356359+0000","last_fullsized":"2024-03-28T22:08:35.356359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:58:17.681836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.383773+0000","last_change":"2024-03-28T22:08:29.961979+0000","last_active":"2024-03-28T22:08:31.383773+0000","last_peered":"2024-03-28T22:08:31.383773+0000","last_clean":"2024-03-28T22:08:31.383773+0000","last_became_active":"2024-03-28T22:08:29.961615+0000","last_became_peered":"2024-03-28T22:08:29.961615+0000","last_unstale":"2024-03-28T22:08:31.383773+0000","last_undegraded":"2024-03-28T22:08:31.383773+0000","last_fullsized":"2024-03-28T22:08:31.383773+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:50:37.227632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:33.347083+0000","last_change":"2024-03-28T22:08:29.329017+0000","last_active":"2024-03-28T22:08:33.347083+0000","last_peered":"2024-03-28T22:08:33.347083+0000","last_clean":"2024-03-28T22:08:33.347083+0000","last_became_active":"2024-03-28T22:08:29.328686+0000","last_became_peered":"2024-03-28T22:08:29.328686+0000","last_unstale":"2024-03-28T22:08:33.347083+0000","last_undegraded":"2024-03-28T22:08:33.347083+0000","last_fullsized":"2024-03-28T22:08:33.347083+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:25:28.986165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.945487+0000","last_change":"2024-03-28T22:08:29.960784+0000","last_active":"2024-03-28T22:08:30.945487+0000","last_peered":"2024-03-28T22:08:30.945487+0000","last_clean":"2024-03-28T22:08:30.945487+0000","last_became_active":"2024-03-28T22:08:29.960180+0000","last_became_peered":"2024-03-28T22:08:29.960180+0000","last_unstale":"2024-03-28T22:08:30.945487+0000","last_undegraded":"2024-03-28T22:08:30.945487+0000","last_fullsized":"2024-03-28T22:08:30.945487+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:35:22.504812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.316248+0000","last_change":"2024-03-28T22:08:29.323274+0000","last_active":"2024-03-28T22:08:32.316248+0000","last_peered":"2024-03-28T22:08:32.316248+0000","last_clean":"2024-03-28T22:08:32.316248+0000","last_became_active":"2024-03-28T22:08:29.322911+0000","last_became_peered":"2024-03-28T22:08:29.322911+0000","last_unstale":"2024-03-28T22:08:32.316248+0000","last_undegraded":"2024-03-28T22:08:32.316248+0000","last_fullsized":"2024-03-28T22:08:32.316248+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:44:14.439411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.320384+0000","last_change":"2024-03-28T22:08:29.961101+0000","last_active":"2024-03-28T22:08:32.320384+0000","last_peered":"2024-03-28T22:08:32.320384+0000","last_clean":"2024-03-28T22:08:32.320384+0000","last_became_active":"2024-03-28T22:08:29.960767+0000","last_became_peered":"2024-03-28T22:08:29.960767+0000","last_unstale":"2024-03-28T22:08:32.320384+0000","last_undegraded":"2024-03-28T22:08:32.320384+0000","last_fullsized":"2024-03-28T22:08:32.320384+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:33:45.236691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.960562+0000","last_change":"2024-03-28T22:08:29.327160+0000","last_active":"2024-03-28T22:08:31.960562+0000","last_peered":"2024-03-28T22:08:31.960562+0000","last_clean":"2024-03-28T22:08:31.960562+0000","last_became_active":"2024-03-28T22:08:29.326174+0000","last_became_peered":"2024-03-28T22:08:29.326174+0000","last_unstale":"2024-03-28T22:08:31.960562+0000","last_undegraded":"2024-03-28T22:08:31.960562+0000","last_fullsized":"2024-03-28T22:08:31.960562+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:43:12.264966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.316816+0000","last_change":"2024-03-28T22:08:29.954093+0000","last_active":"2024-03-28T22:08:32.316816+0000","last_peered":"2024-03-28T22:08:32.316816+0000","last_clean":"2024-03-28T22:08:32.316816+0000","last_became_active":"2024-03-28T22:08:29.953688+0000","last_became_peered":"2024-03-28T22:08:29.953688+0000","last_unstale":"2024-03-28T22:08:32.316816+0000","last_undegraded":"2024-03-28T22:08:32.316816+0000","last_fullsized":"2024-03-28T22:08:32.316816+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:57:51.026565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.384019+0000","last_change":"2024-03-28T22:08:29.331104+0000","last_active":"2024-03-28T22:08:31.384019+0000","last_peered":"2024-03-28T22:08:31.384019+0000","last_clean":"2024-03-28T22:08:31.384019+0000","last_became_active":"2024-03-28T22:08:29.330505+0000","last_became_peered":"2024-03-28T22:08:29.330505+0000","last_unstale":"2024-03-28T22:08:31.384019+0000","last_undegraded":"2024-03-28T22:08:31.384019+0000","last_fullsized":"2024-03-28T22:08:31.384019+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:36:59.944587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.317054+0000","last_change":"2024-03-28T22:08:29.963540+0000","last_active":"2024-03-28T22:08:32.317054+0000","last_peered":"2024-03-28T22:08:32.317054+0000","last_clean":"2024-03-28T22:08:32.317054+0000","last_became_active":"2024-03-28T22:08:29.963206+0000","last_became_peered":"2024-03-28T22:08:29.963206+0000","last_unstale":"2024-03-28T22:08:32.317054+0000","last_undegraded":"2024-03-28T22:08:32.317054+0000","last_fullsized":"2024-03-28T22:08:32.317054+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:55:53.292554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.945457+0000","last_change":"2024-03-28T22:08:29.962822+0000","last_active":"2024-03-28T22:08:30.945457+0000","last_peered":"2024-03-28T22:08:30.945457+0000","last_clean":"2024-03-28T22:08:30.945457+0000","last_became_active":"2024-03-28T22:08:29.962343+0000","last_became_peered":"2024-03-28T22:08:29.962343+0000","last_unstale":"2024-03-28T22:08:30.945457+0000","last_undegraded":"2024-03-28T22:08:30.945457+0000","last_fullsized":"2024-03-28T22:08:30.945457+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:36.608543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:34.350282+0000","last_change":"2024-03-28T22:08:29.328239+0000","last_active":"2024-03-28T22:08:34.350282+0000","last_peered":"2024-03-28T22:08:34.350282+0000","last_clean":"2024-03-28T22:08:34.350282+0000","last_became_active":"2024-03-28T22:08:29.327903+0000","last_became_peered":"2024-03-28T22:08:29.327903+0000","last_unstale":"2024-03-28T22:08:34.350282+0000","last_undegraded":"2024-03-28T22:08:34.350282+0000","last_fullsized":"2024-03-28T22:08:34.350282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:14:17.558472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.942099+0000","last_change":"2024-03-28T22:08:29.962006+0000","last_active":"2024-03-28T22:08:30.942099+0000","last_peered":"2024-03-28T22:08:30.942099+0000","last_clean":"2024-03-28T22:08:30.942099+0000","last_became_active":"2024-03-28T22:08:29.961676+0000","last_became_peered":"2024-03-28T22:08:29.961676+0000","last_unstale":"2024-03-28T22:08:30.942099+0000","last_undegraded":"2024-03-28T22:08:30.942099+0000","last_fullsized":"2024-03-28T22:08:30.942099+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:00:48.067789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:34.348160+0000","last_change":"2024-03-28T22:08:29.323050+0000","last_active":"2024-03-28T22:08:34.348160+0000","last_peered":"2024-03-28T22:08:34.348160+0000","last_clean":"2024-03-28T22:08:34.348160+0000","last_became_active":"2024-03-28T22:08:29.322684+0000","last_became_peered":"2024-03-28T22:08:29.322684+0000","last_unstale":"2024-03-28T22:08:34.348160+0000","last_undegraded":"2024-03-28T22:08:34.348160+0000","last_fullsized":"2024-03-28T22:08:34.348160+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:14:43.811122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":349,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.383801+0000","last_change":"2024-03-28T22:07:17.785516+0000","last_active":"2024-03-28T22:08:31.383801+0000","last_peered":"2024-03-28T22:08:31.383801+0000","last_clean":"2024-03-28T22:08:31.383801+0000","last_became_active":"2024-03-28T22:07:17.785123+0000","last_became_peered":"2024-03-28T22:07:17.785123+0000","last_unstale":"2024-03-28T22:08:31.383801+0000","last_undegraded":"2024-03-28T22:08:31.383801+0000","last_fullsized":"2024-03-28T22:08:31.383801+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:03:59.840513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:03:59.840513+0000","last_clean_scrub_stamp":"2024-03-28T22:03:59.840513+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:31:06.253632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.316218+0000","last_change":"2024-03-28T22:08:29.961489+0000","last_active":"2024-03-28T22:08:31.316218+0000","last_peered":"2024-03-28T22:08:31.316218+0000","last_clean":"2024-03-28T22:08:31.316218+0000","last_became_active":"2024-03-28T22:08:29.960881+0000","last_became_peered":"2024-03-28T22:08:29.960881+0000","last_unstale":"2024-03-28T22:08:31.316218+0000","last_undegraded":"2024-03-28T22:08:31.316218+0000","last_fullsized":"2024-03-28T22:08:31.316218+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:48.870834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:33.345504+0000","last_change":"2024-03-28T22:08:29.331994+0000","last_active":"2024-03-28T22:08:33.345504+0000","last_peered":"2024-03-28T22:08:33.345504+0000","last_clean":"2024-03-28T22:08:33.345504+0000","last_became_active":"2024-03-28T22:08:29.331630+0000","last_became_peered":"2024-03-28T22:08:29.331630+0000","last_unstale":"2024-03-28T22:08:33.345504+0000","last_undegraded":"2024-03-28T22:08:33.345504+0000","last_fullsized":"2024-03-28T22:08:33.345504+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:11:37.100405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.316954+0000","last_change":"2024-03-28T22:08:29.330353+0000","last_active":"2024-03-28T22:08:32.316954+0000","last_peered":"2024-03-28T22:08:32.316954+0000","last_clean":"2024-03-28T22:08:32.316954+0000","last_became_active":"2024-03-28T22:08:29.329421+0000","last_became_peered":"2024-03-28T22:08:29.329421+0000","last_unstale":"2024-03-28T22:08:32.316954+0000","last_undegraded":"2024-03-28T22:08:32.316954+0000","last_fullsized":"2024-03-28T22:08:32.316954+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:56.166506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.316530+0000","last_change":"2024-03-28T22:08:29.948595+0000","last_active":"2024-03-28T22:08:31.316530+0000","last_peered":"2024-03-28T22:08:31.316530+0000","last_clean":"2024-03-28T22:08:31.316530+0000","last_became_active":"2024-03-28T22:08:29.946986+0000","last_became_peered":"2024-03-28T22:08:29.946986+0000","last_unstale":"2024-03-28T22:08:31.316530+0000","last_undegraded":"2024-03-28T22:08:31.316530+0000","last_fullsized":"2024-03-28T22:08:31.316530+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:56:12.351731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:33.346999+0000","last_change":"2024-03-28T22:08:29.333375+0000","last_active":"2024-03-28T22:08:33.346999+0000","last_peered":"2024-03-28T22:08:33.346999+0000","last_clean":"2024-03-28T22:08:33.346999+0000","last_became_active":"2024-03-28T22:08:29.331594+0000","last_became_peered":"2024-03-28T22:08:29.331594+0000","last_unstale":"2024-03-28T22:08:33.346999+0000","last_undegraded":"2024-03-28T22:08:33.346999+0000","last_fullsized":"2024-03-28T22:08:33.346999+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:07:43.936807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.942409+0000","last_change":"2024-03-28T22:08:29.959061+0000","last_active":"2024-03-28T22:08:30.942409+0000","last_peered":"2024-03-28T22:08:30.942409+0000","last_clean":"2024-03-28T22:08:30.942409+0000","last_became_active":"2024-03-28T22:08:29.958723+0000","last_became_peered":"2024-03-28T22:08:29.958723+0000","last_unstale":"2024-03-28T22:08:30.942409+0000","last_undegraded":"2024-03-28T22:08:30.942409+0000","last_fullsized":"2024-03-28T22:08:30.942409+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:32:23.016606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.318796+0000","last_change":"2024-03-28T22:08:29.329886+0000","last_active":"2024-03-28T22:08:32.318796+0000","last_peered":"2024-03-28T22:08:32.318796+0000","last_clean":"2024-03-28T22:08:32.318796+0000","last_became_active":"2024-03-28T22:08:29.329503+0000","last_became_peered":"2024-03-28T22:08:29.329503+0000","last_unstale":"2024-03-28T22:08:32.318796+0000","last_undegraded":"2024-03-28T22:08:32.318796+0000","last_fullsized":"2024-03-28T22:08:32.318796+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:50:05.686647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.159378+0000","last_change":"2024-03-28T22:08:29.957038+0000","last_active":"2024-03-28T22:08:31.159378+0000","last_peered":"2024-03-28T22:08:31.159378+0000","last_clean":"2024-03-28T22:08:31.159378+0000","last_became_active":"2024-03-28T22:08:29.956693+0000","last_became_peered":"2024-03-28T22:08:29.956693+0000","last_unstale":"2024-03-28T22:08:31.159378+0000","last_undegraded":"2024-03-28T22:08:31.159378+0000","last_fullsized":"2024-03-28T22:08:31.159378+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:19:09.881431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.320080+0000","last_change":"2024-03-28T22:08:29.327356+0000","last_active":"2024-03-28T22:08:32.320080+0000","last_peered":"2024-03-28T22:08:32.320080+0000","last_clean":"2024-03-28T22:08:32.320080+0000","last_became_active":"2024-03-28T22:08:29.326947+0000","last_became_peered":"2024-03-28T22:08:29.326947+0000","last_unstale":"2024-03-28T22:08:32.320080+0000","last_undegraded":"2024-03-28T22:08:32.320080+0000","last_fullsized":"2024-03-28T22:08:32.320080+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:47:29.417465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.319207+0000","last_change":"2024-03-28T22:08:29.966505+0000","last_active":"2024-03-28T22:08:32.319207+0000","last_peered":"2024-03-28T22:08:32.319207+0000","last_clean":"2024-03-28T22:08:32.319207+0000","last_became_active":"2024-03-28T22:08:29.966057+0000","last_became_peered":"2024-03-28T22:08:29.966057+0000","last_unstale":"2024-03-28T22:08:32.319207+0000","last_undegraded":"2024-03-28T22:08:32.319207+0000","last_fullsized":"2024-03-28T22:08:32.319207+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:17:45.196585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.318339+0000","last_change":"2024-03-28T22:08:29.328897+0000","last_active":"2024-03-28T22:08:32.318339+0000","last_peered":"2024-03-28T22:08:32.318339+0000","last_clean":"2024-03-28T22:08:32.318339+0000","last_became_active":"2024-03-28T22:08:29.328390+0000","last_became_peered":"2024-03-28T22:08:29.328390+0000","last_unstale":"2024-03-28T22:08:32.318339+0000","last_undegraded":"2024-03-28T22:08:32.318339+0000","last_fullsized":"2024-03-28T22:08:32.318339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:21:12.699503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.320231+0000","last_change":"2024-03-28T22:08:29.957212+0000","last_active":"2024-03-28T22:08:32.320231+0000","last_peered":"2024-03-28T22:08:32.320231+0000","last_clean":"2024-03-28T22:08:32.320231+0000","last_became_active":"2024-03-28T22:08:29.956215+0000","last_became_peered":"2024-03-28T22:08:29.956215+0000","last_unstale":"2024-03-28T22:08:32.320231+0000","last_undegraded":"2024-03-28T22:08:32.320231+0000","last_fullsized":"2024-03-28T22:08:32.320231+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:57:25.199993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.654491+0000","last_change":"2024-03-28T22:08:29.332300+0000","last_active":"2024-03-28T22:08:31.654491+0000","last_peered":"2024-03-28T22:08:31.654491+0000","last_clean":"2024-03-28T22:08:31.654491+0000","last_became_active":"2024-03-28T22:08:29.331924+0000","last_became_peered":"2024-03-28T22:08:29.331924+0000","last_unstale":"2024-03-28T22:08:31.654491+0000","last_undegraded":"2024-03-28T22:08:31.654491+0000","last_fullsized":"2024-03-28T22:08:31.654491+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:49:28.453925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.942339+0000","last_change":"2024-03-28T22:08:29.955023+0000","last_active":"2024-03-28T22:08:30.942339+0000","last_peered":"2024-03-28T22:08:30.942339+0000","last_clean":"2024-03-28T22:08:30.942339+0000","last_became_active":"2024-03-28T22:08:29.954675+0000","last_became_peered":"2024-03-28T22:08:29.954675+0000","last_unstale":"2024-03-28T22:08:30.942339+0000","last_undegraded":"2024-03-28T22:08:30.942339+0000","last_fullsized":"2024-03-28T22:08:30.942339+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:31.246452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.321497+0000","last_change":"2024-03-28T22:08:29.329280+0000","last_active":"2024-03-28T22:08:32.321497+0000","last_peered":"2024-03-28T22:08:32.321497+0000","last_clean":"2024-03-28T22:08:32.321497+0000","last_became_active":"2024-03-28T22:08:29.328860+0000","last_became_peered":"2024-03-28T22:08:29.328860+0000","last_unstale":"2024-03-28T22:08:32.321497+0000","last_undegraded":"2024-03-28T22:08:32.321497+0000","last_fullsized":"2024-03-28T22:08:32.321497+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:15:49.610026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.384235+0000","last_change":"2024-03-28T22:08:29.966224+0000","last_active":"2024-03-28T22:08:31.384235+0000","last_peered":"2024-03-28T22:08:31.384235+0000","last_clean":"2024-03-28T22:08:31.384235+0000","last_became_active":"2024-03-28T22:08:29.965636+0000","last_became_peered":"2024-03-28T22:08:29.965636+0000","last_unstale":"2024-03-28T22:08:31.384235+0000","last_undegraded":"2024-03-28T22:08:31.384235+0000","last_fullsized":"2024-03-28T22:08:31.384235+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:23:31.230980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.654759+0000","last_change":"2024-03-28T22:08:29.335228+0000","last_active":"2024-03-28T22:08:31.654759+0000","last_peered":"2024-03-28T22:08:31.654759+0000","last_clean":"2024-03-28T22:08:31.654759+0000","last_became_active":"2024-03-28T22:08:29.334686+0000","last_became_peered":"2024-03-28T22:08:29.334686+0000","last_unstale":"2024-03-28T22:08:31.654759+0000","last_undegraded":"2024-03-28T22:08:31.654759+0000","last_fullsized":"2024-03-28T22:08:31.654759+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:17:40.571206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.384342+0000","last_change":"2024-03-28T22:08:29.963912+0000","last_active":"2024-03-28T22:08:31.384342+0000","last_peered":"2024-03-28T22:08:31.384342+0000","last_clean":"2024-03-28T22:08:31.384342+0000","last_became_active":"2024-03-28T22:08:29.962740+0000","last_became_peered":"2024-03-28T22:08:29.962740+0000","last_unstale":"2024-03-28T22:08:31.384342+0000","last_undegraded":"2024-03-28T22:08:31.384342+0000","last_fullsized":"2024-03-28T22:08:31.384342+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:50:25.756923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.945926+0000","last_change":"2024-03-28T22:08:29.325767+0000","last_active":"2024-03-28T22:08:30.945926+0000","last_peered":"2024-03-28T22:08:30.945926+0000","last_clean":"2024-03-28T22:08:30.945926+0000","last_became_active":"2024-03-28T22:08:29.325430+0000","last_became_peered":"2024-03-28T22:08:29.325430+0000","last_unstale":"2024-03-28T22:08:30.945926+0000","last_undegraded":"2024-03-28T22:08:30.945926+0000","last_fullsized":"2024-03-28T22:08:30.945926+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:01:13.663163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.384049+0000","last_change":"2024-03-28T22:08:29.960158+0000","last_active":"2024-03-28T22:08:31.384049+0000","last_peered":"2024-03-28T22:08:31.384049+0000","last_clean":"2024-03-28T22:08:31.384049+0000","last_became_active":"2024-03-28T22:08:29.959792+0000","last_became_peered":"2024-03-28T22:08:29.959792+0000","last_unstale":"2024-03-28T22:08:31.384049+0000","last_undegraded":"2024-03-28T22:08:31.384049+0000","last_fullsized":"2024-03-28T22:08:31.384049+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:28:49.135551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.316366+0000","last_change":"2024-03-28T22:08:29.328547+0000","last_active":"2024-03-28T22:08:31.316366+0000","last_peered":"2024-03-28T22:08:31.316366+0000","last_clean":"2024-03-28T22:08:31.316366+0000","last_became_active":"2024-03-28T22:08:29.327910+0000","last_became_peered":"2024-03-28T22:08:29.327910+0000","last_unstale":"2024-03-28T22:08:31.316366+0000","last_undegraded":"2024-03-28T22:08:31.316366+0000","last_fullsized":"2024-03-28T22:08:31.316366+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:06:13.980071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.384413+0000","last_change":"2024-03-28T22:08:29.963904+0000","last_active":"2024-03-28T22:08:31.384413+0000","last_peered":"2024-03-28T22:08:31.384413+0000","last_clean":"2024-03-28T22:08:31.384413+0000","last_became_active":"2024-03-28T22:08:29.963536+0000","last_became_peered":"2024-03-28T22:08:29.963536+0000","last_unstale":"2024-03-28T22:08:31.384413+0000","last_undegraded":"2024-03-28T22:08:31.384413+0000","last_fullsized":"2024-03-28T22:08:31.384413+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:59:43.376792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.320293+0000","last_change":"2024-03-28T22:08:29.962357+0000","last_active":"2024-03-28T22:08:32.320293+0000","last_peered":"2024-03-28T22:08:32.320293+0000","last_clean":"2024-03-28T22:08:32.320293+0000","last_became_active":"2024-03-28T22:08:29.961998+0000","last_became_peered":"2024-03-28T22:08:29.961998+0000","last_unstale":"2024-03-28T22:08:32.320293+0000","last_undegraded":"2024-03-28T22:08:32.320293+0000","last_fullsized":"2024-03-28T22:08:32.320293+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:40:47.465889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.654689+0000","last_change":"2024-03-28T22:08:29.961326+0000","last_active":"2024-03-28T22:08:31.654689+0000","last_peered":"2024-03-28T22:08:31.654689+0000","last_clean":"2024-03-28T22:08:31.654689+0000","last_became_active":"2024-03-28T22:08:29.960316+0000","last_became_peered":"2024-03-28T22:08:29.960316+0000","last_unstale":"2024-03-28T22:08:31.654689+0000","last_undegraded":"2024-03-28T22:08:31.654689+0000","last_fullsized":"2024-03-28T22:08:31.654689+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:37:55.773686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.654465+0000","last_change":"2024-03-28T22:08:29.956272+0000","last_active":"2024-03-28T22:08:31.654465+0000","last_peered":"2024-03-28T22:08:31.654465+0000","last_clean":"2024-03-28T22:08:31.654465+0000","last_became_active":"2024-03-28T22:08:29.955875+0000","last_became_peered":"2024-03-28T22:08:29.955875+0000","last_unstale":"2024-03-28T22:08:31.654465+0000","last_undegraded":"2024-03-28T22:08:31.654465+0000","last_fullsized":"2024-03-28T22:08:31.654465+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:17:33.062615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.159507+0000","last_change":"2024-03-28T22:08:29.958467+0000","last_active":"2024-03-28T22:08:31.159507+0000","last_peered":"2024-03-28T22:08:31.159507+0000","last_clean":"2024-03-28T22:08:31.159507+0000","last_became_active":"2024-03-28T22:08:29.958131+0000","last_became_peered":"2024-03-28T22:08:29.958131+0000","last_unstale":"2024-03-28T22:08:31.159507+0000","last_undegraded":"2024-03-28T22:08:31.159507+0000","last_fullsized":"2024-03-28T22:08:31.159507+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:38:01.817845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.316509+0000","last_change":"2024-03-28T22:08:29.960464+0000","last_active":"2024-03-28T22:08:31.316509+0000","last_peered":"2024-03-28T22:08:31.316509+0000","last_clean":"2024-03-28T22:08:31.316509+0000","last_became_active":"2024-03-28T22:08:29.960111+0000","last_became_peered":"2024-03-28T22:08:29.960111+0000","last_unstale":"2024-03-28T22:08:31.316509+0000","last_undegraded":"2024-03-28T22:08:31.316509+0000","last_fullsized":"2024-03-28T22:08:31.316509+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:32:33.533199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.320717+0000","last_change":"2024-03-28T22:08:29.960045+0000","last_active":"2024-03-28T22:08:32.320717+0000","last_peered":"2024-03-28T22:08:32.320717+0000","last_clean":"2024-03-28T22:08:32.320717+0000","last_became_active":"2024-03-28T22:08:29.959645+0000","last_became_peered":"2024-03-28T22:08:29.959645+0000","last_unstale":"2024-03-28T22:08:32.320717+0000","last_undegraded":"2024-03-28T22:08:32.320717+0000","last_fullsized":"2024-03-28T22:08:32.320717+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:34:56.202257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.654973+0000","last_change":"2024-03-28T22:08:29.958736+0000","last_active":"2024-03-28T22:08:31.654973+0000","last_peered":"2024-03-28T22:08:31.654973+0000","last_clean":"2024-03-28T22:08:31.654973+0000","last_became_active":"2024-03-28T22:08:29.958387+0000","last_became_peered":"2024-03-28T22:08:29.958387+0000","last_unstale":"2024-03-28T22:08:31.654973+0000","last_undegraded":"2024-03-28T22:08:31.654973+0000","last_fullsized":"2024-03-28T22:08:31.654973+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:07:54.829166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.942401+0000","last_change":"2024-03-28T22:08:29.961373+0000","last_active":"2024-03-28T22:08:30.942401+0000","last_peered":"2024-03-28T22:08:30.942401+0000","last_clean":"2024-03-28T22:08:30.942401+0000","last_became_active":"2024-03-28T22:08:29.959751+0000","last_became_peered":"2024-03-28T22:08:29.959751+0000","last_unstale":"2024-03-28T22:08:30.942401+0000","last_undegraded":"2024-03-28T22:08:30.942401+0000","last_fullsized":"2024-03-28T22:08:30.942401+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:34:18.680552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.945356+0000","last_change":"2024-03-28T22:08:29.962721+0000","last_active":"2024-03-28T22:08:30.945356+0000","last_peered":"2024-03-28T22:08:30.945356+0000","last_clean":"2024-03-28T22:08:30.945356+0000","last_became_active":"2024-03-28T22:08:29.961349+0000","last_became_peered":"2024-03-28T22:08:29.961349+0000","last_unstale":"2024-03-28T22:08:30.945356+0000","last_undegraded":"2024-03-28T22:08:30.945356+0000","last_fullsized":"2024-03-28T22:08:30.945356+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:35:56.535055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.941972+0000","last_change":"2024-03-28T22:08:29.960068+0000","last_active":"2024-03-28T22:08:30.941972+0000","last_peered":"2024-03-28T22:08:30.941972+0000","last_clean":"2024-03-28T22:08:30.941972+0000","last_became_active":"2024-03-28T22:08:29.959716+0000","last_became_peered":"2024-03-28T22:08:29.959716+0000","last_unstale":"2024-03-28T22:08:30.941972+0000","last_undegraded":"2024-03-28T22:08:30.941972+0000","last_fullsized":"2024-03-28T22:08:30.941972+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:18:38.055322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.943409+0000","last_change":"2024-03-28T22:08:29.957920+0000","last_active":"2024-03-28T22:08:30.943409+0000","last_peered":"2024-03-28T22:08:30.943409+0000","last_clean":"2024-03-28T22:08:30.943409+0000","last_became_active":"2024-03-28T22:08:29.957541+0000","last_became_peered":"2024-03-28T22:08:29.957541+0000","last_unstale":"2024-03-28T22:08:30.943409+0000","last_undegraded":"2024-03-28T22:08:30.943409+0000","last_fullsized":"2024-03-28T22:08:30.943409+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:45:05.658745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.316586+0000","last_change":"2024-03-28T22:08:29.953589+0000","last_active":"2024-03-28T22:08:31.316586+0000","last_peered":"2024-03-28T22:08:31.316586+0000","last_clean":"2024-03-28T22:08:31.316586+0000","last_became_active":"2024-03-28T22:08:29.953253+0000","last_became_peered":"2024-03-28T22:08:29.953253+0000","last_unstale":"2024-03-28T22:08:31.316586+0000","last_undegraded":"2024-03-28T22:08:31.316586+0000","last_fullsized":"2024-03-28T22:08:31.316586+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:35.720708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.320612+0000","last_change":"2024-03-28T22:08:29.966235+0000","last_active":"2024-03-28T22:08:32.320612+0000","last_peered":"2024-03-28T22:08:32.320612+0000","last_clean":"2024-03-28T22:08:32.320612+0000","last_became_active":"2024-03-28T22:08:29.965830+0000","last_became_peered":"2024-03-28T22:08:29.965830+0000","last_unstale":"2024-03-28T22:08:32.320612+0000","last_undegraded":"2024-03-28T22:08:32.320612+0000","last_fullsized":"2024-03-28T22:08:32.320612+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:04:27.094743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":12022,"num_objects":55,"num_object_clones":0,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":121,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":23527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107,"ondisk_log_size":107,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350990,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":925696,"data_stored":442154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612626,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28540,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716612,"statfs":{"total":95995035648,"available":95965810688,"internally_reserved":0,"allocated":1560576,"data_stored":1035130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776151,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":942080,"data_stored":442390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037788,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":942080,"data_stored":442310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299426,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36832,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708320,"statfs":{"total":95995035648,"available":95957319680,"internally_reserved":0,"allocated":1531904,"data_stored":1033732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462950,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36244,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708908,"statfs":{"total":95995035648,"available":95957921792,"internally_reserved":0,"allocated":929792,"data_stored":441779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":933888,"data_stored":443184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738435,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":443050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411386,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36264,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708888,"statfs":{"total":95995035648,"available":95957901312,"internally_reserved":0,"allocated":946176,"data_stored":443840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149749,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":441973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888113,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40408,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704744,"statfs":{"total":95995035648,"available":95953657856,"internally_reserved":0,"allocated":933888,"data_stored":442607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724588,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28524,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716628,"statfs":{"total":95995035648,"available":95965827072,"internally_reserved":0,"allocated":1544192,"data_stored":1033807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":10181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T22:08:37.137 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T22:08:37.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.271+0000 7fab213b6640 1 -- 172.21.15.16:0/3562325169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab1c102200 msgr2=0x7fab1c1045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:37.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.271+0000 7fab213b6640 1 --2- 172.21.15.16:0/3562325169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab1c102200 0x7fab1c1045f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fab0c009900 tx=0x7fab0c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:08:37.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.271+0000 7fab213b6640 1 -- 172.21.15.16:0/3562325169 shutdown_connections 2024-03-28T22:08:37.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.271+0000 7fab213b6640 1 --2- 172.21.15.16:0/3562325169 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab1c107460 0x7fab1c1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:37.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.271+0000 7fab213b6640 1 --2- 172.21.15.16:0/3562325169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fab1c104b30 0x7fab1c106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:37.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.271+0000 7fab213b6640 1 --2- 172.21.15.16:0/3562325169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab1c102200 0x7fab1c1045f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:37.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.271+0000 7fab213b6640 1 -- 172.21.15.16:0/3562325169 >> 172.21.15.16:0/3562325169 conn(0x7fab1c0fbbf0 msgr2=0x7fab1c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:37.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.271+0000 7fab213b6640 1 -- 172.21.15.16:0/3562325169 shutdown_connections 2024-03-28T22:08:37.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.271+0000 7fab213b6640 1 -- 172.21.15.16:0/3562325169 wait complete. 2024-03-28T22:08:37.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.272+0000 7fab213b6640 1 Processor -- start 2024-03-28T22:08:37.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.272+0000 7fab213b6640 1 -- start start 2024-03-28T22:08:37.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.272+0000 7fab213b6640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab1c102200 0x7fab1c197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:37.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.272+0000 7fab213b6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab1c104b30 0x7fab1c197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:37.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.272+0000 7fab213b6640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fab1c107460 0x7fab1c19f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:37.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.272+0000 7fab213b6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fab1c10d430 con 0x7fab1c104b30 2024-03-28T22:08:37.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.272+0000 7fab213b6640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fab1c10d2b0 con 0x7fab1c102200 2024-03-28T22:08:37.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.272+0000 7fab213b6640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fab1c10d5b0 con 0x7fab1c107460 2024-03-28T22:08:37.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.272+0000 7fab127fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab1c104b30 0x7fab1c197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:37.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.272+0000 7fab127fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab1c104b30 0x7fab1c197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57688/0 (socket says 172.21.15.16:57688) 2024-03-28T22:08:37.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.272+0000 7fab127fc640 1 -- 172.21.15.16:0/3236787611 learned_addr learned my addr 172.21.15.16:0/3236787611 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:37.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.272+0000 7fab1affd640 1 --2- 172.21.15.16:0/3236787611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab1c102200 0x7fab1c197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:37.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.272+0000 7fab1b7fe640 1 --2- 172.21.15.16:0/3236787611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fab1c107460 0x7fab1c19f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:37.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.273+0000 7fab127fc640 1 -- 172.21.15.16:0/3236787611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fab1c107460 msgr2=0x7fab1c19f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:37.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.273+0000 7fab127fc640 1 --2- 172.21.15.16:0/3236787611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fab1c107460 0x7fab1c19f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:37.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.273+0000 7fab127fc640 1 -- 172.21.15.16:0/3236787611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab1c102200 msgr2=0x7fab1c197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:37.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.273+0000 7fab127fc640 1 --2- 172.21.15.16:0/3236787611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab1c102200 0x7fab1c197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:37.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.273+0000 7fab127fc640 1 -- 172.21.15.16:0/3236787611 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab1c19f6c0 con 0x7fab1c104b30 2024-03-28T22:08:37.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.273+0000 7fab127fc640 1 --2- 172.21.15.16:0/3236787611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab1c104b30 0x7fab1c197fe0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fab0400d740 tx=0x7fab0400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:37.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.273+0000 7fab18ff9640 1 -- 172.21.15.16:0/3236787611 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab04017070 con 0x7fab1c104b30 2024-03-28T22:08:37.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.273+0000 7fab18ff9640 1 -- 172.21.15.16:0/3236787611 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab0400de50 con 0x7fab1c104b30 2024-03-28T22:08:37.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.273+0000 7fab213b6640 1 -- 172.21.15.16:0/3236787611 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab1c19f9b0 con 0x7fab1c104b30 2024-03-28T22:08:37.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.273+0000 7fab18ff9640 1 -- 172.21.15.16:0/3236787611 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab04004e00 con 0x7fab1c104b30 2024-03-28T22:08:37.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.273+0000 7fab213b6640 1 -- 172.21.15.16:0/3236787611 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab1c19feb0 con 0x7fab1c104b30 2024-03-28T22:08:37.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.274+0000 7fab213b6640 1 -- 172.21.15.16:0/3236787611 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab1c0fd6e0 con 0x7fab1c104b30 2024-03-28T22:08:37.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.275+0000 7fab18ff9640 1 -- 172.21.15.16:0/3236787611 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fab04004fa0 con 0x7fab1c104b30 2024-03-28T22:08:37.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.276+0000 7fab18ff9640 1 --2- 172.21.15.16:0/3236787611 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fab080a1e10 0x7fab080a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:37.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.276+0000 7fab1affd640 1 --2- 172.21.15.16:0/3236787611 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fab080a1e10 0x7fab080a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:37.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.276+0000 7fab18ff9640 1 -- 172.21.15.16:0/3236787611 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fab040cd680 con 0x7fab1c104b30 2024-03-28T22:08:37.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.276+0000 7fab1affd640 1 --2- 172.21.15.16:0/3236787611 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fab080a1e10 0x7fab080a42d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fab0c02c7b0 tx=0x7fab0c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:37.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.280+0000 7fab18ff9640 1 -- 172.21.15.16:0/3236787611 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fab0409a4b0 con 0x7fab1c104b30 2024-03-28T22:08:37.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:37 smithi016 ceph-mon[32374]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:08:37.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:37 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3668659403' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:37.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:37 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2736808225' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:08:37.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:37 smithi016 ceph-mon[32374]: pgmap v290: 97 pgs: 97 active+clean; 588 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6.1 KiB/s wr, 18 op/s 2024-03-28T22:08:37.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.464+0000 7fab213b6640 1 -- 172.21.15.16:0/3236787611 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fab1c061b90 con 0x7fab080a1e10 2024-03-28T22:08:37.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.471+0000 7fab18ff9640 1 -- 172.21.15.16:0/3236787611 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258725 (secure 0 0 0) 0x7fab1c061b90 con 0x7fab080a1e10 2024-03-28T22:08:37.477 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:08:37.478 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-03-28T22:08:37.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.474+0000 7fab213b6640 1 -- 172.21.15.16:0/3236787611 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fab080a1e10 msgr2=0x7fab080a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:37.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.474+0000 7fab213b6640 1 --2- 172.21.15.16:0/3236787611 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fab080a1e10 0x7fab080a42d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fab0c02c7b0 tx=0x7fab0c005fd0 comp rx=0 tx=0).stop 2024-03-28T22:08:37.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.474+0000 7fab213b6640 1 -- 172.21.15.16:0/3236787611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab1c104b30 msgr2=0x7fab1c197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:37.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.474+0000 7fab213b6640 1 --2- 172.21.15.16:0/3236787611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab1c104b30 0x7fab1c197fe0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fab0400d740 tx=0x7fab0400dc00 comp rx=0 tx=0).stop 2024-03-28T22:08:37.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.474+0000 7fab213b6640 1 -- 172.21.15.16:0/3236787611 shutdown_connections 2024-03-28T22:08:37.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.474+0000 7fab213b6640 1 --2- 172.21.15.16:0/3236787611 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fab080a1e10 0x7fab080a42d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:37.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.474+0000 7fab213b6640 1 --2- 172.21.15.16:0/3236787611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fab1c107460 0x7fab1c19f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:37.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.474+0000 7fab213b6640 1 --2- 172.21.15.16:0/3236787611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab1c104b30 0x7fab1c197fe0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:37.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.474+0000 7fab213b6640 1 --2- 172.21.15.16:0/3236787611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab1c102200 0x7fab1c197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:37.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.475+0000 7fab213b6640 1 -- 172.21.15.16:0/3236787611 >> 172.21.15.16:0/3236787611 conn(0x7fab1c0fbbf0 msgr2=0x7fab1c102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:37.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.475+0000 7fab213b6640 1 -- 172.21.15.16:0/3236787611 shutdown_connections 2024-03-28T22:08:37.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:37.475+0000 7fab213b6640 1 -- 172.21.15.16:0/3236787611 wait complete. 2024-03-28T22:08:37.488 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2024-03-28T22:08:37.073809+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":602390,"num_objects":57,"num_object_clones":0,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":161,"num_read_kb":138,"num_write":302,"num_write_kb":2985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":244,"ondisk_log_size":244,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378580,"kb_used_data":12776,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563244,"statfs":{"total":1151940427776,"available":1151552761856,"internally_reserved":0,"allocated":13082624,"data_stored":7085956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":374453562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12022,"num_objects":55,"num_object_clones":0,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":121,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.762427"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.320849+0000","last_change":"2024-03-28T22:08:29.962664+0000","last_active":"2024-03-28T22:08:32.320849+0000","last_peered":"2024-03-28T22:08:32.320849+0000","last_clean":"2024-03-28T22:08:32.320849+0000","last_became_active":"2024-03-28T22:08:29.962029+0000","last_became_peered":"2024-03-28T22:08:29.962029+0000","last_unstale":"2024-03-28T22:08:32.320849+0000","last_undegraded":"2024-03-28T22:08:32.320849+0000","last_fullsized":"2024-03-28T22:08:32.320849+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:20:42.832304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.159434+0000","last_change":"2024-03-28T22:08:29.965617+0000","last_active":"2024-03-28T22:08:31.159434+0000","last_peered":"2024-03-28T22:08:31.159434+0000","last_clean":"2024-03-28T22:08:31.159434+0000","last_became_active":"2024-03-28T22:08:29.965248+0000","last_became_peered":"2024-03-28T22:08:29.965248+0000","last_unstale":"2024-03-28T22:08:31.159434+0000","last_undegraded":"2024-03-28T22:08:31.159434+0000","last_fullsized":"2024-03-28T22:08:31.159434+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:37:46.703768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.159381+0000","last_change":"2024-03-28T22:08:29.958452+0000","last_active":"2024-03-28T22:08:31.159381+0000","last_peered":"2024-03-28T22:08:31.159381+0000","last_clean":"2024-03-28T22:08:31.159381+0000","last_became_active":"2024-03-28T22:08:29.958089+0000","last_became_peered":"2024-03-28T22:08:29.958089+0000","last_unstale":"2024-03-28T22:08:31.159381+0000","last_undegraded":"2024-03-28T22:08:31.159381+0000","last_fullsized":"2024-03-28T22:08:31.159381+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:10:44.084925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.315906+0000","last_change":"2024-03-28T22:08:29.964812+0000","last_active":"2024-03-28T22:08:32.315906+0000","last_peered":"2024-03-28T22:08:32.315906+0000","last_clean":"2024-03-28T22:08:32.315906+0000","last_became_active":"2024-03-28T22:08:29.963872+0000","last_became_peered":"2024-03-28T22:08:29.963872+0000","last_unstale":"2024-03-28T22:08:32.315906+0000","last_undegraded":"2024-03-28T22:08:32.315906+0000","last_fullsized":"2024-03-28T22:08:32.315906+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:08.491078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.316698+0000","last_change":"2024-03-28T22:08:29.965564+0000","last_active":"2024-03-28T22:08:32.316698+0000","last_peered":"2024-03-28T22:08:32.316698+0000","last_clean":"2024-03-28T22:08:32.316698+0000","last_became_active":"2024-03-28T22:08:29.962636+0000","last_became_peered":"2024-03-28T22:08:29.962636+0000","last_unstale":"2024-03-28T22:08:32.316698+0000","last_undegraded":"2024-03-28T22:08:32.316698+0000","last_fullsized":"2024-03-28T22:08:32.316698+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:03:50.903231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.316632+0000","last_change":"2024-03-28T22:08:29.962091+0000","last_active":"2024-03-28T22:08:32.316632+0000","last_peered":"2024-03-28T22:08:32.316632+0000","last_clean":"2024-03-28T22:08:32.316632+0000","last_became_active":"2024-03-28T22:08:29.961735+0000","last_became_peered":"2024-03-28T22:08:29.961735+0000","last_unstale":"2024-03-28T22:08:32.316632+0000","last_undegraded":"2024-03-28T22:08:32.316632+0000","last_fullsized":"2024-03-28T22:08:32.316632+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:40:09.853265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.383889+0000","last_change":"2024-03-28T22:08:29.961089+0000","last_active":"2024-03-28T22:08:31.383889+0000","last_peered":"2024-03-28T22:08:31.383889+0000","last_clean":"2024-03-28T22:08:31.383889+0000","last_became_active":"2024-03-28T22:08:29.960711+0000","last_became_peered":"2024-03-28T22:08:29.960711+0000","last_unstale":"2024-03-28T22:08:31.383889+0000","last_undegraded":"2024-03-28T22:08:31.383889+0000","last_fullsized":"2024-03-28T22:08:31.383889+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:05:14.617718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.941889+0000","last_change":"2024-03-28T22:08:29.965142+0000","last_active":"2024-03-28T22:08:30.941889+0000","last_peered":"2024-03-28T22:08:30.941889+0000","last_clean":"2024-03-28T22:08:30.941889+0000","last_became_active":"2024-03-28T22:08:29.964730+0000","last_became_peered":"2024-03-28T22:08:29.964730+0000","last_unstale":"2024-03-28T22:08:30.941889+0000","last_undegraded":"2024-03-28T22:08:30.941889+0000","last_fullsized":"2024-03-28T22:08:30.941889+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:23:43.229681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.943626+0000","last_change":"2024-03-28T22:08:29.958984+0000","last_active":"2024-03-28T22:08:30.943626+0000","last_peered":"2024-03-28T22:08:30.943626+0000","last_clean":"2024-03-28T22:08:30.943626+0000","last_became_active":"2024-03-28T22:08:29.958583+0000","last_became_peered":"2024-03-28T22:08:29.958583+0000","last_unstale":"2024-03-28T22:08:30.943626+0000","last_undegraded":"2024-03-28T22:08:30.943626+0000","last_fullsized":"2024-03-28T22:08:30.943626+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:05:11.361719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.317794+0000","last_change":"2024-03-28T22:08:29.962774+0000","last_active":"2024-03-28T22:08:32.317794+0000","last_peered":"2024-03-28T22:08:32.317794+0000","last_clean":"2024-03-28T22:08:32.317794+0000","last_became_active":"2024-03-28T22:08:29.961222+0000","last_became_peered":"2024-03-28T22:08:29.961222+0000","last_unstale":"2024-03-28T22:08:32.317794+0000","last_undegraded":"2024-03-28T22:08:32.317794+0000","last_fullsized":"2024-03-28T22:08:32.317794+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:47:05.310483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.384331+0000","last_change":"2024-03-28T22:08:29.964041+0000","last_active":"2024-03-28T22:08:31.384331+0000","last_peered":"2024-03-28T22:08:31.384331+0000","last_clean":"2024-03-28T22:08:31.384331+0000","last_became_active":"2024-03-28T22:08:29.963360+0000","last_became_peered":"2024-03-28T22:08:29.963360+0000","last_unstale":"2024-03-28T22:08:31.384331+0000","last_undegraded":"2024-03-28T22:08:31.384331+0000","last_fullsized":"2024-03-28T22:08:31.384331+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:32.921916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.315875+0000","last_change":"2024-03-28T22:08:29.966449+0000","last_active":"2024-03-28T22:08:32.315875+0000","last_peered":"2024-03-28T22:08:32.315875+0000","last_clean":"2024-03-28T22:08:32.315875+0000","last_became_active":"2024-03-28T22:08:29.966036+0000","last_became_peered":"2024-03-28T22:08:29.966036+0000","last_unstale":"2024-03-28T22:08:32.315875+0000","last_undegraded":"2024-03-28T22:08:32.315875+0000","last_fullsized":"2024-03-28T22:08:32.315875+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:12:15.057713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.316451+0000","last_change":"2024-03-28T22:08:29.958924+0000","last_active":"2024-03-28T22:08:31.316451+0000","last_peered":"2024-03-28T22:08:31.316451+0000","last_clean":"2024-03-28T22:08:31.316451+0000","last_became_active":"2024-03-28T22:08:29.958601+0000","last_became_peered":"2024-03-28T22:08:29.958601+0000","last_unstale":"2024-03-28T22:08:31.316451+0000","last_undegraded":"2024-03-28T22:08:31.316451+0000","last_fullsized":"2024-03-28T22:08:31.316451+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:53:37.070276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.159166+0000","last_change":"2024-03-28T22:08:29.965045+0000","last_active":"2024-03-28T22:08:31.159166+0000","last_peered":"2024-03-28T22:08:31.159166+0000","last_clean":"2024-03-28T22:08:31.159166+0000","last_became_active":"2024-03-28T22:08:29.964710+0000","last_became_peered":"2024-03-28T22:08:29.964710+0000","last_unstale":"2024-03-28T22:08:31.159166+0000","last_undegraded":"2024-03-28T22:08:31.159166+0000","last_fullsized":"2024-03-28T22:08:31.159166+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:03:11.491213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.159242+0000","last_change":"2024-03-28T22:08:29.957940+0000","last_active":"2024-03-28T22:08:31.159242+0000","last_peered":"2024-03-28T22:08:31.159242+0000","last_clean":"2024-03-28T22:08:31.159242+0000","last_became_active":"2024-03-28T22:08:29.957587+0000","last_became_peered":"2024-03-28T22:08:29.957587+0000","last_unstale":"2024-03-28T22:08:31.159242+0000","last_undegraded":"2024-03-28T22:08:31.159242+0000","last_fullsized":"2024-03-28T22:08:31.159242+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:03:58.557806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.943285+0000","last_change":"2024-03-28T22:08:29.957397+0000","last_active":"2024-03-28T22:08:30.943285+0000","last_peered":"2024-03-28T22:08:30.943285+0000","last_clean":"2024-03-28T22:08:30.943285+0000","last_became_active":"2024-03-28T22:08:29.957054+0000","last_became_peered":"2024-03-28T22:08:29.957054+0000","last_unstale":"2024-03-28T22:08:30.943285+0000","last_undegraded":"2024-03-28T22:08:30.943285+0000","last_fullsized":"2024-03-28T22:08:30.943285+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:35:28.500561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.320879+0000","last_change":"2024-03-28T22:08:29.963343+0000","last_active":"2024-03-28T22:08:32.320879+0000","last_peered":"2024-03-28T22:08:32.320879+0000","last_clean":"2024-03-28T22:08:32.320879+0000","last_became_active":"2024-03-28T22:08:29.962926+0000","last_became_peered":"2024-03-28T22:08:29.962926+0000","last_unstale":"2024-03-28T22:08:32.320879+0000","last_undegraded":"2024-03-28T22:08:32.320879+0000","last_fullsized":"2024-03-28T22:08:32.320879+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:15:39.133930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.960657+0000","last_change":"2024-03-28T22:08:29.955903+0000","last_active":"2024-03-28T22:08:31.960657+0000","last_peered":"2024-03-28T22:08:31.960657+0000","last_clean":"2024-03-28T22:08:31.960657+0000","last_became_active":"2024-03-28T22:08:29.955460+0000","last_became_peered":"2024-03-28T22:08:29.955460+0000","last_unstale":"2024-03-28T22:08:31.960657+0000","last_undegraded":"2024-03-28T22:08:31.960657+0000","last_fullsized":"2024-03-28T22:08:31.960657+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:36:12.961475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.654730+0000","last_change":"2024-03-28T22:08:29.954279+0000","last_active":"2024-03-28T22:08:31.654730+0000","last_peered":"2024-03-28T22:08:31.654730+0000","last_clean":"2024-03-28T22:08:31.654730+0000","last_became_active":"2024-03-28T22:08:29.953869+0000","last_became_peered":"2024-03-28T22:08:29.953869+0000","last_unstale":"2024-03-28T22:08:31.654730+0000","last_undegraded":"2024-03-28T22:08:31.654730+0000","last_fullsized":"2024-03-28T22:08:31.654730+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:05:52.247391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:33.348241+0000","last_change":"2024-03-28T22:08:29.327644+0000","last_active":"2024-03-28T22:08:33.348241+0000","last_peered":"2024-03-28T22:08:33.348241+0000","last_clean":"2024-03-28T22:08:33.348241+0000","last_became_active":"2024-03-28T22:08:29.327325+0000","last_became_peered":"2024-03-28T22:08:29.327325+0000","last_unstale":"2024-03-28T22:08:33.348241+0000","last_undegraded":"2024-03-28T22:08:33.348241+0000","last_fullsized":"2024-03-28T22:08:33.348241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:26:54.782703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.942406+0000","last_change":"2024-03-28T22:08:29.962905+0000","last_active":"2024-03-28T22:08:30.942406+0000","last_peered":"2024-03-28T22:08:30.942406+0000","last_clean":"2024-03-28T22:08:30.942406+0000","last_became_active":"2024-03-28T22:08:29.962559+0000","last_became_peered":"2024-03-28T22:08:29.962559+0000","last_unstale":"2024-03-28T22:08:30.942406+0000","last_undegraded":"2024-03-28T22:08:30.942406+0000","last_fullsized":"2024-03-28T22:08:30.942406+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:02.347584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.158323+0000","last_change":"2024-03-28T22:08:29.334871+0000","last_active":"2024-03-28T22:08:31.158323+0000","last_peered":"2024-03-28T22:08:31.158323+0000","last_clean":"2024-03-28T22:08:31.158323+0000","last_became_active":"2024-03-28T22:08:29.334565+0000","last_became_peered":"2024-03-28T22:08:29.334565+0000","last_unstale":"2024-03-28T22:08:31.158323+0000","last_undegraded":"2024-03-28T22:08:31.158323+0000","last_fullsized":"2024-03-28T22:08:31.158323+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:31:19.333496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.159562+0000","last_change":"2024-03-28T22:08:29.957622+0000","last_active":"2024-03-28T22:08:31.159562+0000","last_peered":"2024-03-28T22:08:31.159562+0000","last_clean":"2024-03-28T22:08:31.159562+0000","last_became_active":"2024-03-28T22:08:29.957208+0000","last_became_peered":"2024-03-28T22:08:29.957208+0000","last_unstale":"2024-03-28T22:08:31.159562+0000","last_undegraded":"2024-03-28T22:08:31.159562+0000","last_fullsized":"2024-03-28T22:08:31.159562+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:49:44.684203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:33.345708+0000","last_change":"2024-03-28T22:08:29.330291+0000","last_active":"2024-03-28T22:08:33.345708+0000","last_peered":"2024-03-28T22:08:33.345708+0000","last_clean":"2024-03-28T22:08:33.345708+0000","last_became_active":"2024-03-28T22:08:29.328955+0000","last_became_peered":"2024-03-28T22:08:29.328955+0000","last_unstale":"2024-03-28T22:08:33.345708+0000","last_undegraded":"2024-03-28T22:08:33.345708+0000","last_fullsized":"2024-03-28T22:08:33.345708+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:10.838059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.943129+0000","last_change":"2024-03-28T22:08:29.966681+0000","last_active":"2024-03-28T22:08:30.943129+0000","last_peered":"2024-03-28T22:08:30.943129+0000","last_clean":"2024-03-28T22:08:30.943129+0000","last_became_active":"2024-03-28T22:08:29.966023+0000","last_became_peered":"2024-03-28T22:08:29.966023+0000","last_unstale":"2024-03-28T22:08:30.943129+0000","last_undegraded":"2024-03-28T22:08:30.943129+0000","last_fullsized":"2024-03-28T22:08:30.943129+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:04:43.281259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.316909+0000","last_change":"2024-03-28T22:08:29.329026+0000","last_active":"2024-03-28T22:08:32.316909+0000","last_peered":"2024-03-28T22:08:32.316909+0000","last_clean":"2024-03-28T22:08:32.316909+0000","last_became_active":"2024-03-28T22:08:29.327570+0000","last_became_peered":"2024-03-28T22:08:29.327570+0000","last_unstale":"2024-03-28T22:08:32.316909+0000","last_undegraded":"2024-03-28T22:08:32.316909+0000","last_fullsized":"2024-03-28T22:08:32.316909+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:53:54.779073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.946065+0000","last_change":"2024-03-28T22:08:29.966455+0000","last_active":"2024-03-28T22:08:30.946065+0000","last_peered":"2024-03-28T22:08:30.946065+0000","last_clean":"2024-03-28T22:08:30.946065+0000","last_became_active":"2024-03-28T22:08:29.964451+0000","last_became_peered":"2024-03-28T22:08:29.964451+0000","last_unstale":"2024-03-28T22:08:30.946065+0000","last_undegraded":"2024-03-28T22:08:30.946065+0000","last_fullsized":"2024-03-28T22:08:30.946065+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:46:22.811146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'6","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:35.354234+0000","last_change":"2024-03-28T22:08:29.327397+0000","last_active":"2024-03-28T22:08:35.354234+0000","last_peered":"2024-03-28T22:08:35.354234+0000","last_clean":"2024-03-28T22:08:35.354234+0000","last_became_active":"2024-03-28T22:08:29.327009+0000","last_became_peered":"2024-03-28T22:08:29.327009+0000","last_unstale":"2024-03-28T22:08:35.354234+0000","last_undegraded":"2024-03-28T22:08:35.354234+0000","last_fullsized":"2024-03-28T22:08:35.354234+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:38:51.951934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.316406+0000","last_change":"2024-03-28T22:08:29.969821+0000","last_active":"2024-03-28T22:08:32.316406+0000","last_peered":"2024-03-28T22:08:32.316406+0000","last_clean":"2024-03-28T22:08:32.316406+0000","last_became_active":"2024-03-28T22:08:29.969386+0000","last_became_peered":"2024-03-28T22:08:29.969386+0000","last_unstale":"2024-03-28T22:08:32.316406+0000","last_undegraded":"2024-03-28T22:08:32.316406+0000","last_fullsized":"2024-03-28T22:08:32.316406+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:44:19.753083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:33.345509+0000","last_change":"2024-03-28T22:08:29.332385+0000","last_active":"2024-03-28T22:08:33.345509+0000","last_peered":"2024-03-28T22:08:33.345509+0000","last_clean":"2024-03-28T22:08:33.345509+0000","last_became_active":"2024-03-28T22:08:29.331956+0000","last_became_peered":"2024-03-28T22:08:29.331956+0000","last_unstale":"2024-03-28T22:08:33.345509+0000","last_undegraded":"2024-03-28T22:08:33.345509+0000","last_fullsized":"2024-03-28T22:08:33.345509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:10:04.428034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.945895+0000","last_change":"2024-03-28T22:08:29.968586+0000","last_active":"2024-03-28T22:08:30.945895+0000","last_peered":"2024-03-28T22:08:30.945895+0000","last_clean":"2024-03-28T22:08:30.945895+0000","last_became_active":"2024-03-28T22:08:29.968088+0000","last_became_peered":"2024-03-28T22:08:29.968088+0000","last_unstale":"2024-03-28T22:08:30.945895+0000","last_undegraded":"2024-03-28T22:08:30.945895+0000","last_fullsized":"2024-03-28T22:08:30.945895+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:21:03.051733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:33.347174+0000","last_change":"2024-03-28T22:08:29.327180+0000","last_active":"2024-03-28T22:08:33.347174+0000","last_peered":"2024-03-28T22:08:33.347174+0000","last_clean":"2024-03-28T22:08:33.347174+0000","last_became_active":"2024-03-28T22:08:29.326825+0000","last_became_peered":"2024-03-28T22:08:29.326825+0000","last_unstale":"2024-03-28T22:08:33.347174+0000","last_undegraded":"2024-03-28T22:08:33.347174+0000","last_fullsized":"2024-03-28T22:08:33.347174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:11:09.979970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.941917+0000","last_change":"2024-03-28T22:08:29.955299+0000","last_active":"2024-03-28T22:08:30.941917+0000","last_peered":"2024-03-28T22:08:30.941917+0000","last_clean":"2024-03-28T22:08:30.941917+0000","last_became_active":"2024-03-28T22:08:29.954949+0000","last_became_peered":"2024-03-28T22:08:29.954949+0000","last_unstale":"2024-03-28T22:08:30.941917+0000","last_undegraded":"2024-03-28T22:08:30.941917+0000","last_fullsized":"2024-03-28T22:08:30.941917+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:01:48.375251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.945248+0000","last_change":"2024-03-28T22:08:29.327261+0000","last_active":"2024-03-28T22:08:30.945248+0000","last_peered":"2024-03-28T22:08:30.945248+0000","last_clean":"2024-03-28T22:08:30.945248+0000","last_became_active":"2024-03-28T22:08:29.326910+0000","last_became_peered":"2024-03-28T22:08:29.326910+0000","last_unstale":"2024-03-28T22:08:30.945248+0000","last_undegraded":"2024-03-28T22:08:30.945248+0000","last_fullsized":"2024-03-28T22:08:30.945248+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:46:41.933372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.960634+0000","last_change":"2024-03-28T22:08:29.954891+0000","last_active":"2024-03-28T22:08:31.960634+0000","last_peered":"2024-03-28T22:08:31.960634+0000","last_clean":"2024-03-28T22:08:31.960634+0000","last_became_active":"2024-03-28T22:08:29.954519+0000","last_became_peered":"2024-03-28T22:08:29.954519+0000","last_unstale":"2024-03-28T22:08:31.960634+0000","last_undegraded":"2024-03-28T22:08:31.960634+0000","last_fullsized":"2024-03-28T22:08:31.960634+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:07:42.621189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.322199+0000","last_change":"2024-03-28T22:08:29.330180+0000","last_active":"2024-03-28T22:08:32.322199+0000","last_peered":"2024-03-28T22:08:32.322199+0000","last_clean":"2024-03-28T22:08:32.322199+0000","last_became_active":"2024-03-28T22:08:29.329835+0000","last_became_peered":"2024-03-28T22:08:29.329835+0000","last_unstale":"2024-03-28T22:08:32.322199+0000","last_undegraded":"2024-03-28T22:08:32.322199+0000","last_fullsized":"2024-03-28T22:08:32.322199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:48:39.906359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.943559+0000","last_change":"2024-03-28T22:08:29.967074+0000","last_active":"2024-03-28T22:08:30.943559+0000","last_peered":"2024-03-28T22:08:30.943559+0000","last_clean":"2024-03-28T22:08:30.943559+0000","last_became_active":"2024-03-28T22:08:29.966258+0000","last_became_peered":"2024-03-28T22:08:29.966258+0000","last_unstale":"2024-03-28T22:08:30.943559+0000","last_undegraded":"2024-03-28T22:08:30.943559+0000","last_fullsized":"2024-03-28T22:08:30.943559+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:25:44.986179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.318422+0000","last_change":"2024-03-28T22:08:29.326388+0000","last_active":"2024-03-28T22:08:32.318422+0000","last_peered":"2024-03-28T22:08:32.318422+0000","last_clean":"2024-03-28T22:08:32.318422+0000","last_became_active":"2024-03-28T22:08:29.326001+0000","last_became_peered":"2024-03-28T22:08:29.326001+0000","last_unstale":"2024-03-28T22:08:32.318422+0000","last_undegraded":"2024-03-28T22:08:32.318422+0000","last_fullsized":"2024-03-28T22:08:32.318422+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:31:56.227038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.159507+0000","last_change":"2024-03-28T22:08:29.959299+0000","last_active":"2024-03-28T22:08:31.159507+0000","last_peered":"2024-03-28T22:08:31.159507+0000","last_clean":"2024-03-28T22:08:31.159507+0000","last_became_active":"2024-03-28T22:08:29.958955+0000","last_became_peered":"2024-03-28T22:08:29.958955+0000","last_unstale":"2024-03-28T22:08:31.159507+0000","last_undegraded":"2024-03-28T22:08:31.159507+0000","last_fullsized":"2024-03-28T22:08:31.159507+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:11:06.775920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.321904+0000","last_change":"2024-03-28T22:08:29.326089+0000","last_active":"2024-03-28T22:08:32.321904+0000","last_peered":"2024-03-28T22:08:32.321904+0000","last_clean":"2024-03-28T22:08:32.321904+0000","last_became_active":"2024-03-28T22:08:29.325463+0000","last_became_peered":"2024-03-28T22:08:29.325463+0000","last_unstale":"2024-03-28T22:08:32.321904+0000","last_undegraded":"2024-03-28T22:08:32.321904+0000","last_fullsized":"2024-03-28T22:08:32.321904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:18:31.049729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.943352+0000","last_change":"2024-03-28T22:08:29.964248+0000","last_active":"2024-03-28T22:08:30.943352+0000","last_peered":"2024-03-28T22:08:30.943352+0000","last_clean":"2024-03-28T22:08:30.943352+0000","last_became_active":"2024-03-28T22:08:29.963897+0000","last_became_peered":"2024-03-28T22:08:29.963897+0000","last_unstale":"2024-03-28T22:08:30.943352+0000","last_undegraded":"2024-03-28T22:08:30.943352+0000","last_fullsized":"2024-03-28T22:08:30.943352+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:41:39.645360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'11","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:34.704050+0000","last_change":"2024-03-28T22:08:29.330732+0000","last_active":"2024-03-28T22:08:34.704050+0000","last_peered":"2024-03-28T22:08:34.704050+0000","last_clean":"2024-03-28T22:08:34.704050+0000","last_became_active":"2024-03-28T22:08:29.329231+0000","last_became_peered":"2024-03-28T22:08:29.329231+0000","last_unstale":"2024-03-28T22:08:34.704050+0000","last_undegraded":"2024-03-28T22:08:34.704050+0000","last_fullsized":"2024-03-28T22:08:34.704050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:16:39.260429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8212,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":12,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.316836+0000","last_change":"2024-03-28T22:08:29.965791+0000","last_active":"2024-03-28T22:08:32.316836+0000","last_peered":"2024-03-28T22:08:32.316836+0000","last_clean":"2024-03-28T22:08:32.316836+0000","last_became_active":"2024-03-28T22:08:29.965390+0000","last_became_peered":"2024-03-28T22:08:29.965390+0000","last_unstale":"2024-03-28T22:08:32.316836+0000","last_undegraded":"2024-03-28T22:08:32.316836+0000","last_fullsized":"2024-03-28T22:08:32.316836+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:43:53.325906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.317396+0000","last_change":"2024-03-28T22:08:29.325851+0000","last_active":"2024-03-28T22:08:32.317396+0000","last_peered":"2024-03-28T22:08:32.317396+0000","last_clean":"2024-03-28T22:08:32.317396+0000","last_became_active":"2024-03-28T22:08:29.325500+0000","last_became_peered":"2024-03-28T22:08:29.325500+0000","last_unstale":"2024-03-28T22:08:32.317396+0000","last_undegraded":"2024-03-28T22:08:32.317396+0000","last_fullsized":"2024-03-28T22:08:32.317396+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:04:25.151908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.316372+0000","last_change":"2024-03-28T22:08:29.953857+0000","last_active":"2024-03-28T22:08:31.316372+0000","last_peered":"2024-03-28T22:08:31.316372+0000","last_clean":"2024-03-28T22:08:31.316372+0000","last_became_active":"2024-03-28T22:08:29.953538+0000","last_became_peered":"2024-03-28T22:08:29.953538+0000","last_unstale":"2024-03-28T22:08:31.316372+0000","last_undegraded":"2024-03-28T22:08:31.316372+0000","last_fullsized":"2024-03-28T22:08:31.316372+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:40:32.404760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.321807+0000","last_change":"2024-03-28T22:08:29.327286+0000","last_active":"2024-03-28T22:08:32.321807+0000","last_peered":"2024-03-28T22:08:32.321807+0000","last_clean":"2024-03-28T22:08:32.321807+0000","last_became_active":"2024-03-28T22:08:29.326902+0000","last_became_peered":"2024-03-28T22:08:29.326902+0000","last_unstale":"2024-03-28T22:08:32.321807+0000","last_undegraded":"2024-03-28T22:08:32.321807+0000","last_fullsized":"2024-03-28T22:08:32.321807+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:14:58.669534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.384118+0000","last_change":"2024-03-28T22:08:29.959917+0000","last_active":"2024-03-28T22:08:31.384118+0000","last_peered":"2024-03-28T22:08:31.384118+0000","last_clean":"2024-03-28T22:08:31.384118+0000","last_became_active":"2024-03-28T22:08:29.959535+0000","last_became_peered":"2024-03-28T22:08:29.959535+0000","last_unstale":"2024-03-28T22:08:31.384118+0000","last_undegraded":"2024-03-28T22:08:31.384118+0000","last_fullsized":"2024-03-28T22:08:31.384118+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:35:04.725585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'4","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:35.356359+0000","last_change":"2024-03-28T22:08:29.327740+0000","last_active":"2024-03-28T22:08:35.356359+0000","last_peered":"2024-03-28T22:08:35.356359+0000","last_clean":"2024-03-28T22:08:35.356359+0000","last_became_active":"2024-03-28T22:08:29.326664+0000","last_became_peered":"2024-03-28T22:08:29.326664+0000","last_unstale":"2024-03-28T22:08:35.356359+0000","last_undegraded":"2024-03-28T22:08:35.356359+0000","last_fullsized":"2024-03-28T22:08:35.356359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:58:17.681836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.383773+0000","last_change":"2024-03-28T22:08:29.961979+0000","last_active":"2024-03-28T22:08:31.383773+0000","last_peered":"2024-03-28T22:08:31.383773+0000","last_clean":"2024-03-28T22:08:31.383773+0000","last_became_active":"2024-03-28T22:08:29.961615+0000","last_became_peered":"2024-03-28T22:08:29.961615+0000","last_unstale":"2024-03-28T22:08:31.383773+0000","last_undegraded":"2024-03-28T22:08:31.383773+0000","last_fullsized":"2024-03-28T22:08:31.383773+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:50:37.227632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:33.347083+0000","last_change":"2024-03-28T22:08:29.329017+0000","last_active":"2024-03-28T22:08:33.347083+0000","last_peered":"2024-03-28T22:08:33.347083+0000","last_clean":"2024-03-28T22:08:33.347083+0000","last_became_active":"2024-03-28T22:08:29.328686+0000","last_became_peered":"2024-03-28T22:08:29.328686+0000","last_unstale":"2024-03-28T22:08:33.347083+0000","last_undegraded":"2024-03-28T22:08:33.347083+0000","last_fullsized":"2024-03-28T22:08:33.347083+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:25:28.986165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.945487+0000","last_change":"2024-03-28T22:08:29.960784+0000","last_active":"2024-03-28T22:08:30.945487+0000","last_peered":"2024-03-28T22:08:30.945487+0000","last_clean":"2024-03-28T22:08:30.945487+0000","last_became_active":"2024-03-28T22:08:29.960180+0000","last_became_peered":"2024-03-28T22:08:29.960180+0000","last_unstale":"2024-03-28T22:08:30.945487+0000","last_undegraded":"2024-03-28T22:08:30.945487+0000","last_fullsized":"2024-03-28T22:08:30.945487+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:35:22.504812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.316248+0000","last_change":"2024-03-28T22:08:29.323274+0000","last_active":"2024-03-28T22:08:32.316248+0000","last_peered":"2024-03-28T22:08:32.316248+0000","last_clean":"2024-03-28T22:08:32.316248+0000","last_became_active":"2024-03-28T22:08:29.322911+0000","last_became_peered":"2024-03-28T22:08:29.322911+0000","last_unstale":"2024-03-28T22:08:32.316248+0000","last_undegraded":"2024-03-28T22:08:32.316248+0000","last_fullsized":"2024-03-28T22:08:32.316248+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:44:14.439411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.320384+0000","last_change":"2024-03-28T22:08:29.961101+0000","last_active":"2024-03-28T22:08:32.320384+0000","last_peered":"2024-03-28T22:08:32.320384+0000","last_clean":"2024-03-28T22:08:32.320384+0000","last_became_active":"2024-03-28T22:08:29.960767+0000","last_became_peered":"2024-03-28T22:08:29.960767+0000","last_unstale":"2024-03-28T22:08:32.320384+0000","last_undegraded":"2024-03-28T22:08:32.320384+0000","last_fullsized":"2024-03-28T22:08:32.320384+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:33:45.236691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.960562+0000","last_change":"2024-03-28T22:08:29.327160+0000","last_active":"2024-03-28T22:08:31.960562+0000","last_peered":"2024-03-28T22:08:31.960562+0000","last_clean":"2024-03-28T22:08:31.960562+0000","last_became_active":"2024-03-28T22:08:29.326174+0000","last_became_peered":"2024-03-28T22:08:29.326174+0000","last_unstale":"2024-03-28T22:08:31.960562+0000","last_undegraded":"2024-03-28T22:08:31.960562+0000","last_fullsized":"2024-03-28T22:08:31.960562+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:43:12.264966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.316816+0000","last_change":"2024-03-28T22:08:29.954093+0000","last_active":"2024-03-28T22:08:32.316816+0000","last_peered":"2024-03-28T22:08:32.316816+0000","last_clean":"2024-03-28T22:08:32.316816+0000","last_became_active":"2024-03-28T22:08:29.953688+0000","last_became_peered":"2024-03-28T22:08:29.953688+0000","last_unstale":"2024-03-28T22:08:32.316816+0000","last_undegraded":"2024-03-28T22:08:32.316816+0000","last_fullsized":"2024-03-28T22:08:32.316816+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:57:51.026565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.384019+0000","last_change":"2024-03-28T22:08:29.331104+0000","last_active":"2024-03-28T22:08:31.384019+0000","last_peered":"2024-03-28T22:08:31.384019+0000","last_clean":"2024-03-28T22:08:31.384019+0000","last_became_active":"2024-03-28T22:08:29.330505+0000","last_became_peered":"2024-03-28T22:08:29.330505+0000","last_unstale":"2024-03-28T22:08:31.384019+0000","last_undegraded":"2024-03-28T22:08:31.384019+0000","last_fullsized":"2024-03-28T22:08:31.384019+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:36:59.944587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.317054+0000","last_change":"2024-03-28T22:08:29.963540+0000","last_active":"2024-03-28T22:08:32.317054+0000","last_peered":"2024-03-28T22:08:32.317054+0000","last_clean":"2024-03-28T22:08:32.317054+0000","last_became_active":"2024-03-28T22:08:29.963206+0000","last_became_peered":"2024-03-28T22:08:29.963206+0000","last_unstale":"2024-03-28T22:08:32.317054+0000","last_undegraded":"2024-03-28T22:08:32.317054+0000","last_fullsized":"2024-03-28T22:08:32.317054+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:55:53.292554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.945457+0000","last_change":"2024-03-28T22:08:29.962822+0000","last_active":"2024-03-28T22:08:30.945457+0000","last_peered":"2024-03-28T22:08:30.945457+0000","last_clean":"2024-03-28T22:08:30.945457+0000","last_became_active":"2024-03-28T22:08:29.962343+0000","last_became_peered":"2024-03-28T22:08:29.962343+0000","last_unstale":"2024-03-28T22:08:30.945457+0000","last_undegraded":"2024-03-28T22:08:30.945457+0000","last_fullsized":"2024-03-28T22:08:30.945457+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:36.608543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:34.350282+0000","last_change":"2024-03-28T22:08:29.328239+0000","last_active":"2024-03-28T22:08:34.350282+0000","last_peered":"2024-03-28T22:08:34.350282+0000","last_clean":"2024-03-28T22:08:34.350282+0000","last_became_active":"2024-03-28T22:08:29.327903+0000","last_became_peered":"2024-03-28T22:08:29.327903+0000","last_unstale":"2024-03-28T22:08:34.350282+0000","last_undegraded":"2024-03-28T22:08:34.350282+0000","last_fullsized":"2024-03-28T22:08:34.350282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:14:17.558472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.942099+0000","last_change":"2024-03-28T22:08:29.962006+0000","last_active":"2024-03-28T22:08:30.942099+0000","last_peered":"2024-03-28T22:08:30.942099+0000","last_clean":"2024-03-28T22:08:30.942099+0000","last_became_active":"2024-03-28T22:08:29.961676+0000","last_became_peered":"2024-03-28T22:08:29.961676+0000","last_unstale":"2024-03-28T22:08:30.942099+0000","last_undegraded":"2024-03-28T22:08:30.942099+0000","last_fullsized":"2024-03-28T22:08:30.942099+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:00:48.067789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:34.348160+0000","last_change":"2024-03-28T22:08:29.323050+0000","last_active":"2024-03-28T22:08:34.348160+0000","last_peered":"2024-03-28T22:08:34.348160+0000","last_clean":"2024-03-28T22:08:34.348160+0000","last_became_active":"2024-03-28T22:08:29.322684+0000","last_became_peered":"2024-03-28T22:08:29.322684+0000","last_unstale":"2024-03-28T22:08:34.348160+0000","last_undegraded":"2024-03-28T22:08:34.348160+0000","last_fullsized":"2024-03-28T22:08:34.348160+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:14:43.811122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":349,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.383801+0000","last_change":"2024-03-28T22:07:17.785516+0000","last_active":"2024-03-28T22:08:31.383801+0000","last_peered":"2024-03-28T22:08:31.383801+0000","last_clean":"2024-03-28T22:08:31.383801+0000","last_became_active":"2024-03-28T22:07:17.785123+0000","last_became_peered":"2024-03-28T22:07:17.785123+0000","last_unstale":"2024-03-28T22:08:31.383801+0000","last_undegraded":"2024-03-28T22:08:31.383801+0000","last_fullsized":"2024-03-28T22:08:31.383801+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:03:59.840513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:03:59.840513+0000","last_clean_scrub_stamp":"2024-03-28T22:03:59.840513+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:31:06.253632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.316218+0000","last_change":"2024-03-28T22:08:29.961489+0000","last_active":"2024-03-28T22:08:31.316218+0000","last_peered":"2024-03-28T22:08:31.316218+0000","last_clean":"2024-03-28T22:08:31.316218+0000","last_became_active":"2024-03-28T22:08:29.960881+0000","last_became_peered":"2024-03-28T22:08:29.960881+0000","last_unstale":"2024-03-28T22:08:31.316218+0000","last_undegraded":"2024-03-28T22:08:31.316218+0000","last_fullsized":"2024-03-28T22:08:31.316218+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:48.870834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:33.345504+0000","last_change":"2024-03-28T22:08:29.331994+0000","last_active":"2024-03-28T22:08:33.345504+0000","last_peered":"2024-03-28T22:08:33.345504+0000","last_clean":"2024-03-28T22:08:33.345504+0000","last_became_active":"2024-03-28T22:08:29.331630+0000","last_became_peered":"2024-03-28T22:08:29.331630+0000","last_unstale":"2024-03-28T22:08:33.345504+0000","last_undegraded":"2024-03-28T22:08:33.345504+0000","last_fullsized":"2024-03-28T22:08:33.345504+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:11:37.100405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.316954+0000","last_change":"2024-03-28T22:08:29.330353+0000","last_active":"2024-03-28T22:08:32.316954+0000","last_peered":"2024-03-28T22:08:32.316954+0000","last_clean":"2024-03-28T22:08:32.316954+0000","last_became_active":"2024-03-28T22:08:29.329421+0000","last_became_peered":"2024-03-28T22:08:29.329421+0000","last_unstale":"2024-03-28T22:08:32.316954+0000","last_undegraded":"2024-03-28T22:08:32.316954+0000","last_fullsized":"2024-03-28T22:08:32.316954+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:56.166506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.316530+0000","last_change":"2024-03-28T22:08:29.948595+0000","last_active":"2024-03-28T22:08:31.316530+0000","last_peered":"2024-03-28T22:08:31.316530+0000","last_clean":"2024-03-28T22:08:31.316530+0000","last_became_active":"2024-03-28T22:08:29.946986+0000","last_became_peered":"2024-03-28T22:08:29.946986+0000","last_unstale":"2024-03-28T22:08:31.316530+0000","last_undegraded":"2024-03-28T22:08:31.316530+0000","last_fullsized":"2024-03-28T22:08:31.316530+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:56:12.351731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:33.346999+0000","last_change":"2024-03-28T22:08:29.333375+0000","last_active":"2024-03-28T22:08:33.346999+0000","last_peered":"2024-03-28T22:08:33.346999+0000","last_clean":"2024-03-28T22:08:33.346999+0000","last_became_active":"2024-03-28T22:08:29.331594+0000","last_became_peered":"2024-03-28T22:08:29.331594+0000","last_unstale":"2024-03-28T22:08:33.346999+0000","last_undegraded":"2024-03-28T22:08:33.346999+0000","last_fullsized":"2024-03-28T22:08:33.346999+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:07:43.936807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.942409+0000","last_change":"2024-03-28T22:08:29.959061+0000","last_active":"2024-03-28T22:08:30.942409+0000","last_peered":"2024-03-28T22:08:30.942409+0000","last_clean":"2024-03-28T22:08:30.942409+0000","last_became_active":"2024-03-28T22:08:29.958723+0000","last_became_peered":"2024-03-28T22:08:29.958723+0000","last_unstale":"2024-03-28T22:08:30.942409+0000","last_undegraded":"2024-03-28T22:08:30.942409+0000","last_fullsized":"2024-03-28T22:08:30.942409+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:32:23.016606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.318796+0000","last_change":"2024-03-28T22:08:29.329886+0000","last_active":"2024-03-28T22:08:32.318796+0000","last_peered":"2024-03-28T22:08:32.318796+0000","last_clean":"2024-03-28T22:08:32.318796+0000","last_became_active":"2024-03-28T22:08:29.329503+0000","last_became_peered":"2024-03-28T22:08:29.329503+0000","last_unstale":"2024-03-28T22:08:32.318796+0000","last_undegraded":"2024-03-28T22:08:32.318796+0000","last_fullsized":"2024-03-28T22:08:32.318796+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:50:05.686647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.159378+0000","last_change":"2024-03-28T22:08:29.957038+0000","last_active":"2024-03-28T22:08:31.159378+0000","last_peered":"2024-03-28T22:08:31.159378+0000","last_clean":"2024-03-28T22:08:31.159378+0000","last_became_active":"2024-03-28T22:08:29.956693+0000","last_became_peered":"2024-03-28T22:08:29.956693+0000","last_unstale":"2024-03-28T22:08:31.159378+0000","last_undegraded":"2024-03-28T22:08:31.159378+0000","last_fullsized":"2024-03-28T22:08:31.159378+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:19:09.881431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.320080+0000","last_change":"2024-03-28T22:08:29.327356+0000","last_active":"2024-03-28T22:08:32.320080+0000","last_peered":"2024-03-28T22:08:32.320080+0000","last_clean":"2024-03-28T22:08:32.320080+0000","last_became_active":"2024-03-28T22:08:29.326947+0000","last_became_peered":"2024-03-28T22:08:29.326947+0000","last_unstale":"2024-03-28T22:08:32.320080+0000","last_undegraded":"2024-03-28T22:08:32.320080+0000","last_fullsized":"2024-03-28T22:08:32.320080+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:47:29.417465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.319207+0000","last_change":"2024-03-28T22:08:29.966505+0000","last_active":"2024-03-28T22:08:32.319207+0000","last_peered":"2024-03-28T22:08:32.319207+0000","last_clean":"2024-03-28T22:08:32.319207+0000","last_became_active":"2024-03-28T22:08:29.966057+0000","last_became_peered":"2024-03-28T22:08:29.966057+0000","last_unstale":"2024-03-28T22:08:32.319207+0000","last_undegraded":"2024-03-28T22:08:32.319207+0000","last_fullsized":"2024-03-28T22:08:32.319207+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:17:45.196585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.318339+0000","last_change":"2024-03-28T22:08:29.328897+0000","last_active":"2024-03-28T22:08:32.318339+0000","last_peered":"2024-03-28T22:08:32.318339+0000","last_clean":"2024-03-28T22:08:32.318339+0000","last_became_active":"2024-03-28T22:08:29.328390+0000","last_became_peered":"2024-03-28T22:08:29.328390+0000","last_unstale":"2024-03-28T22:08:32.318339+0000","last_undegraded":"2024-03-28T22:08:32.318339+0000","last_fullsized":"2024-03-28T22:08:32.318339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:21:12.699503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.320231+0000","last_change":"2024-03-28T22:08:29.957212+0000","last_active":"2024-03-28T22:08:32.320231+0000","last_peered":"2024-03-28T22:08:32.320231+0000","last_clean":"2024-03-28T22:08:32.320231+0000","last_became_active":"2024-03-28T22:08:29.956215+0000","last_became_peered":"2024-03-28T22:08:29.956215+0000","last_unstale":"2024-03-28T22:08:32.320231+0000","last_undegraded":"2024-03-28T22:08:32.320231+0000","last_fullsized":"2024-03-28T22:08:32.320231+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:57:25.199993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.654491+0000","last_change":"2024-03-28T22:08:29.332300+0000","last_active":"2024-03-28T22:08:31.654491+0000","last_peered":"2024-03-28T22:08:31.654491+0000","last_clean":"2024-03-28T22:08:31.654491+0000","last_became_active":"2024-03-28T22:08:29.331924+0000","last_became_peered":"2024-03-28T22:08:29.331924+0000","last_unstale":"2024-03-28T22:08:31.654491+0000","last_undegraded":"2024-03-28T22:08:31.654491+0000","last_fullsized":"2024-03-28T22:08:31.654491+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:49:28.453925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.942339+0000","last_change":"2024-03-28T22:08:29.955023+0000","last_active":"2024-03-28T22:08:30.942339+0000","last_peered":"2024-03-28T22:08:30.942339+0000","last_clean":"2024-03-28T22:08:30.942339+0000","last_became_active":"2024-03-28T22:08:29.954675+0000","last_became_peered":"2024-03-28T22:08:29.954675+0000","last_unstale":"2024-03-28T22:08:30.942339+0000","last_undegraded":"2024-03-28T22:08:30.942339+0000","last_fullsized":"2024-03-28T22:08:30.942339+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:31.246452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.321497+0000","last_change":"2024-03-28T22:08:29.329280+0000","last_active":"2024-03-28T22:08:32.321497+0000","last_peered":"2024-03-28T22:08:32.321497+0000","last_clean":"2024-03-28T22:08:32.321497+0000","last_became_active":"2024-03-28T22:08:29.328860+0000","last_became_peered":"2024-03-28T22:08:29.328860+0000","last_unstale":"2024-03-28T22:08:32.321497+0000","last_undegraded":"2024-03-28T22:08:32.321497+0000","last_fullsized":"2024-03-28T22:08:32.321497+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:15:49.610026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.384235+0000","last_change":"2024-03-28T22:08:29.966224+0000","last_active":"2024-03-28T22:08:31.384235+0000","last_peered":"2024-03-28T22:08:31.384235+0000","last_clean":"2024-03-28T22:08:31.384235+0000","last_became_active":"2024-03-28T22:08:29.965636+0000","last_became_peered":"2024-03-28T22:08:29.965636+0000","last_unstale":"2024-03-28T22:08:31.384235+0000","last_undegraded":"2024-03-28T22:08:31.384235+0000","last_fullsized":"2024-03-28T22:08:31.384235+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:23:31.230980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.654759+0000","last_change":"2024-03-28T22:08:29.335228+0000","last_active":"2024-03-28T22:08:31.654759+0000","last_peered":"2024-03-28T22:08:31.654759+0000","last_clean":"2024-03-28T22:08:31.654759+0000","last_became_active":"2024-03-28T22:08:29.334686+0000","last_became_peered":"2024-03-28T22:08:29.334686+0000","last_unstale":"2024-03-28T22:08:31.654759+0000","last_undegraded":"2024-03-28T22:08:31.654759+0000","last_fullsized":"2024-03-28T22:08:31.654759+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:17:40.571206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.384342+0000","last_change":"2024-03-28T22:08:29.963912+0000","last_active":"2024-03-28T22:08:31.384342+0000","last_peered":"2024-03-28T22:08:31.384342+0000","last_clean":"2024-03-28T22:08:31.384342+0000","last_became_active":"2024-03-28T22:08:29.962740+0000","last_became_peered":"2024-03-28T22:08:29.962740+0000","last_unstale":"2024-03-28T22:08:31.384342+0000","last_undegraded":"2024-03-28T22:08:31.384342+0000","last_fullsized":"2024-03-28T22:08:31.384342+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:50:25.756923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.945926+0000","last_change":"2024-03-28T22:08:29.325767+0000","last_active":"2024-03-28T22:08:30.945926+0000","last_peered":"2024-03-28T22:08:30.945926+0000","last_clean":"2024-03-28T22:08:30.945926+0000","last_became_active":"2024-03-28T22:08:29.325430+0000","last_became_peered":"2024-03-28T22:08:29.325430+0000","last_unstale":"2024-03-28T22:08:30.945926+0000","last_undegraded":"2024-03-28T22:08:30.945926+0000","last_fullsized":"2024-03-28T22:08:30.945926+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:01:13.663163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.384049+0000","last_change":"2024-03-28T22:08:29.960158+0000","last_active":"2024-03-28T22:08:31.384049+0000","last_peered":"2024-03-28T22:08:31.384049+0000","last_clean":"2024-03-28T22:08:31.384049+0000","last_became_active":"2024-03-28T22:08:29.959792+0000","last_became_peered":"2024-03-28T22:08:29.959792+0000","last_unstale":"2024-03-28T22:08:31.384049+0000","last_undegraded":"2024-03-28T22:08:31.384049+0000","last_fullsized":"2024-03-28T22:08:31.384049+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:28:49.135551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.316366+0000","last_change":"2024-03-28T22:08:29.328547+0000","last_active":"2024-03-28T22:08:31.316366+0000","last_peered":"2024-03-28T22:08:31.316366+0000","last_clean":"2024-03-28T22:08:31.316366+0000","last_became_active":"2024-03-28T22:08:29.327910+0000","last_became_peered":"2024-03-28T22:08:29.327910+0000","last_unstale":"2024-03-28T22:08:31.316366+0000","last_undegraded":"2024-03-28T22:08:31.316366+0000","last_fullsized":"2024-03-28T22:08:31.316366+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:06:13.980071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.384413+0000","last_change":"2024-03-28T22:08:29.963904+0000","last_active":"2024-03-28T22:08:31.384413+0000","last_peered":"2024-03-28T22:08:31.384413+0000","last_clean":"2024-03-28T22:08:31.384413+0000","last_became_active":"2024-03-28T22:08:29.963536+0000","last_became_peered":"2024-03-28T22:08:29.963536+0000","last_unstale":"2024-03-28T22:08:31.384413+0000","last_undegraded":"2024-03-28T22:08:31.384413+0000","last_fullsized":"2024-03-28T22:08:31.384413+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:59:43.376792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.320293+0000","last_change":"2024-03-28T22:08:29.962357+0000","last_active":"2024-03-28T22:08:32.320293+0000","last_peered":"2024-03-28T22:08:32.320293+0000","last_clean":"2024-03-28T22:08:32.320293+0000","last_became_active":"2024-03-28T22:08:29.961998+0000","last_became_peered":"2024-03-28T22:08:29.961998+0000","last_unstale":"2024-03-28T22:08:32.320293+0000","last_undegraded":"2024-03-28T22:08:32.320293+0000","last_fullsized":"2024-03-28T22:08:32.320293+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:40:47.465889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.654689+0000","last_change":"2024-03-28T22:08:29.961326+0000","last_active":"2024-03-28T22:08:31.654689+0000","last_peered":"2024-03-28T22:08:31.654689+0000","last_clean":"2024-03-28T22:08:31.654689+0000","last_became_active":"2024-03-28T22:08:29.960316+0000","last_became_peered":"2024-03-28T22:08:29.960316+0000","last_unstale":"2024-03-28T22:08:31.654689+0000","last_undegraded":"2024-03-28T22:08:31.654689+0000","last_fullsized":"2024-03-28T22:08:31.654689+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:37:55.773686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.654465+0000","last_change":"2024-03-28T22:08:29.956272+0000","last_active":"2024-03-28T22:08:31.654465+0000","last_peered":"2024-03-28T22:08:31.654465+0000","last_clean":"2024-03-28T22:08:31.654465+0000","last_became_active":"2024-03-28T22:08:29.955875+0000","last_became_peered":"2024-03-28T22:08:29.955875+0000","last_unstale":"2024-03-28T22:08:31.654465+0000","last_undegraded":"2024-03-28T22:08:31.654465+0000","last_fullsized":"2024-03-28T22:08:31.654465+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:17:33.062615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.159507+0000","last_change":"2024-03-28T22:08:29.958467+0000","last_active":"2024-03-28T22:08:31.159507+0000","last_peered":"2024-03-28T22:08:31.159507+0000","last_clean":"2024-03-28T22:08:31.159507+0000","last_became_active":"2024-03-28T22:08:29.958131+0000","last_became_peered":"2024-03-28T22:08:29.958131+0000","last_unstale":"2024-03-28T22:08:31.159507+0000","last_undegraded":"2024-03-28T22:08:31.159507+0000","last_fullsized":"2024-03-28T22:08:31.159507+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:38:01.817845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.316509+0000","last_change":"2024-03-28T22:08:29.960464+0000","last_active":"2024-03-28T22:08:31.316509+0000","last_peered":"2024-03-28T22:08:31.316509+0000","last_clean":"2024-03-28T22:08:31.316509+0000","last_became_active":"2024-03-28T22:08:29.960111+0000","last_became_peered":"2024-03-28T22:08:29.960111+0000","last_unstale":"2024-03-28T22:08:31.316509+0000","last_undegraded":"2024-03-28T22:08:31.316509+0000","last_fullsized":"2024-03-28T22:08:31.316509+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:32:33.533199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.320717+0000","last_change":"2024-03-28T22:08:29.960045+0000","last_active":"2024-03-28T22:08:32.320717+0000","last_peered":"2024-03-28T22:08:32.320717+0000","last_clean":"2024-03-28T22:08:32.320717+0000","last_became_active":"2024-03-28T22:08:29.959645+0000","last_became_peered":"2024-03-28T22:08:29.959645+0000","last_unstale":"2024-03-28T22:08:32.320717+0000","last_undegraded":"2024-03-28T22:08:32.320717+0000","last_fullsized":"2024-03-28T22:08:32.320717+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:34:56.202257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.654973+0000","last_change":"2024-03-28T22:08:29.958736+0000","last_active":"2024-03-28T22:08:31.654973+0000","last_peered":"2024-03-28T22:08:31.654973+0000","last_clean":"2024-03-28T22:08:31.654973+0000","last_became_active":"2024-03-28T22:08:29.958387+0000","last_became_peered":"2024-03-28T22:08:29.958387+0000","last_unstale":"2024-03-28T22:08:31.654973+0000","last_undegraded":"2024-03-28T22:08:31.654973+0000","last_fullsized":"2024-03-28T22:08:31.654973+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:07:54.829166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.942401+0000","last_change":"2024-03-28T22:08:29.961373+0000","last_active":"2024-03-28T22:08:30.942401+0000","last_peered":"2024-03-28T22:08:30.942401+0000","last_clean":"2024-03-28T22:08:30.942401+0000","last_became_active":"2024-03-28T22:08:29.959751+0000","last_became_peered":"2024-03-28T22:08:29.959751+0000","last_unstale":"2024-03-28T22:08:30.942401+0000","last_undegraded":"2024-03-28T22:08:30.942401+0000","last_fullsized":"2024-03-28T22:08:30.942401+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:34:18.680552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.945356+0000","last_change":"2024-03-28T22:08:29.962721+0000","last_active":"2024-03-28T22:08:30.945356+0000","last_peered":"2024-03-28T22:08:30.945356+0000","last_clean":"2024-03-28T22:08:30.945356+0000","last_became_active":"2024-03-28T22:08:29.961349+0000","last_became_peered":"2024-03-28T22:08:29.961349+0000","last_unstale":"2024-03-28T22:08:30.945356+0000","last_undegraded":"2024-03-28T22:08:30.945356+0000","last_fullsized":"2024-03-28T22:08:30.945356+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:35:56.535055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.941972+0000","last_change":"2024-03-28T22:08:29.960068+0000","last_active":"2024-03-28T22:08:30.941972+0000","last_peered":"2024-03-28T22:08:30.941972+0000","last_clean":"2024-03-28T22:08:30.941972+0000","last_became_active":"2024-03-28T22:08:29.959716+0000","last_became_peered":"2024-03-28T22:08:29.959716+0000","last_unstale":"2024-03-28T22:08:30.941972+0000","last_undegraded":"2024-03-28T22:08:30.941972+0000","last_fullsized":"2024-03-28T22:08:30.941972+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:18:38.055322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:30.943409+0000","last_change":"2024-03-28T22:08:29.957920+0000","last_active":"2024-03-28T22:08:30.943409+0000","last_peered":"2024-03-28T22:08:30.943409+0000","last_clean":"2024-03-28T22:08:30.943409+0000","last_became_active":"2024-03-28T22:08:29.957541+0000","last_became_peered":"2024-03-28T22:08:29.957541+0000","last_unstale":"2024-03-28T22:08:30.943409+0000","last_undegraded":"2024-03-28T22:08:30.943409+0000","last_fullsized":"2024-03-28T22:08:30.943409+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:45:05.658745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:31.316586+0000","last_change":"2024-03-28T22:08:29.953589+0000","last_active":"2024-03-28T22:08:31.316586+0000","last_peered":"2024-03-28T22:08:31.316586+0000","last_clean":"2024-03-28T22:08:31.316586+0000","last_became_active":"2024-03-28T22:08:29.953253+0000","last_became_peered":"2024-03-28T22:08:29.953253+0000","last_unstale":"2024-03-28T22:08:31.316586+0000","last_undegraded":"2024-03-28T22:08:31.316586+0000","last_fullsized":"2024-03-28T22:08:31.316586+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:35.720708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T22:08:32.320612+0000","last_change":"2024-03-28T22:08:29.966235+0000","last_active":"2024-03-28T22:08:32.320612+0000","last_peered":"2024-03-28T22:08:32.320612+0000","last_clean":"2024-03-28T22:08:32.320612+0000","last_became_active":"2024-03-28T22:08:29.965830+0000","last_became_peered":"2024-03-28T22:08:29.965830+0000","last_unstale":"2024-03-28T22:08:32.320612+0000","last_undegraded":"2024-03-28T22:08:32.320612+0000","last_fullsized":"2024-03-28T22:08:32.320612+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:04:27.094743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":12022,"num_objects":55,"num_object_clones":0,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":121,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":23527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107,"ondisk_log_size":107,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350990,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":925696,"data_stored":442154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612626,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28540,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716612,"statfs":{"total":95995035648,"available":95965810688,"internally_reserved":0,"allocated":1560576,"data_stored":1035130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776151,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":942080,"data_stored":442390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037788,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":942080,"data_stored":442310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299426,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36832,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708320,"statfs":{"total":95995035648,"available":95957319680,"internally_reserved":0,"allocated":1531904,"data_stored":1033732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462950,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36244,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708908,"statfs":{"total":95995035648,"available":95957921792,"internally_reserved":0,"allocated":929792,"data_stored":441779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":933888,"data_stored":443184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738435,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":443050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411386,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36264,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708888,"statfs":{"total":95995035648,"available":95957901312,"internally_reserved":0,"allocated":946176,"data_stored":443840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149749,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":441973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888113,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40408,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704744,"statfs":{"total":95995035648,"available":95953657856,"internally_reserved":0,"allocated":933888,"data_stored":442607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724588,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28524,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716628,"statfs":{"total":95995035648,"available":95965827072,"internally_reserved":0,"allocated":1544192,"data_stored":1033807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":10181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T22:08:37.493 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-28T22:08:37.493 INFO:tasks.cephadm:Setting up client nodes... 2024-03-28T22:08:37.494 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-28T22:08:37.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:37 smithi063 ceph-mon[36249]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:08:37.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:37 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/3668659403' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:37.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:37 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2736808225' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:08:37.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:37 smithi063 ceph-mon[36249]: pgmap v290: 97 pgs: 97 active+clean; 588 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6.1 KiB/s wr, 18 op/s 2024-03-28T22:08:37.770 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:08:37.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:37 smithi053 ceph-mon[35979]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:08:37.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:37 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3668659403' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:08:37.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:37 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2736808225' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:08:37.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:37 smithi053 ceph-mon[35979]: pgmap v290: 97 pgs: 97 active+clean; 588 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6.1 KiB/s wr, 18 op/s 2024-03-28T22:08:38.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:38 smithi063 ceph-mon[36249]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:08:38.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:38 smithi063 ceph-mon[36249]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:08:38.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:38 smithi016 ceph-mon[32374]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:08:38.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:38 smithi016 ceph-mon[32374]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:08:38.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:38 smithi053 ceph-mon[35979]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:08:38.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:38 smithi053 ceph-mon[35979]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:08:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.366+0000 7fb8abe7f700 1 -- 172.21.15.16:0/783898819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8a4103e50 msgr2=0x7fb8a410a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.366+0000 7fb8abe7f700 1 --2- 172.21.15.16:0/783898819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8a4103e50 0x7fb8a410a700 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb89c00b0d0 tx=0x7fb89c00b3e0 comp rx=0 tx=0).stop 2024-03-28T22:08:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.366+0000 7fb8abe7f700 1 -- 172.21.15.16:0/783898819 shutdown_connections 2024-03-28T22:08:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.366+0000 7fb8abe7f700 1 --2- 172.21.15.16:0/783898819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8a4103e50 0x7fb8a410a700 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:39.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.366+0000 7fb8abe7f700 1 --2- 172.21.15.16:0/783898819 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8a41022b0 0x7fb8a41026d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:39.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.366+0000 7fb8abe7f700 1 --2- 172.21.15.16:0/783898819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8a41034b0 0x7fb8a4103910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:39.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.366+0000 7fb8abe7f700 1 -- 172.21.15.16:0/783898819 >> 172.21.15.16:0/783898819 conn(0x7fb8a40fd850 msgr2=0x7fb8a40ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:39.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.366+0000 7fb8abe7f700 1 -- 172.21.15.16:0/783898819 shutdown_connections 2024-03-28T22:08:39.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.366+0000 7fb8abe7f700 1 -- 172.21.15.16:0/783898819 wait complete. 2024-03-28T22:08:39.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.366+0000 7fb8abe7f700 1 Processor -- start 2024-03-28T22:08:39.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.366+0000 7fb8abe7f700 1 -- start start 2024-03-28T22:08:39.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.367+0000 7fb8abe7f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8a41022b0 0x7fb8a41982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:39.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.367+0000 7fb8abe7f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8a41034b0 0x7fb8a4198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:39.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.367+0000 7fb8abe7f700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8a4103e50 0x7fb8a419f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:39.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.367+0000 7fb8abe7f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a410de70 con 0x7fb8a41034b0 2024-03-28T22:08:39.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.367+0000 7fb8abe7f700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a410dcf0 con 0x7fb8a41022b0 2024-03-28T22:08:39.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.367+0000 7fb8abe7f700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a410dff0 con 0x7fb8a4103e50 2024-03-28T22:08:39.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.367+0000 7fb8a9c1b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8a41022b0 0x7fb8a41982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:39.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.367+0000 7fb8a9c1b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8a41022b0 0x7fb8a41982d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:47402/0 (socket says 172.21.15.16:47402) 2024-03-28T22:08:39.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.367+0000 7fb8a9c1b700 1 -- 172.21.15.16:0/306548595 learned_addr learned my addr 172.21.15.16:0/306548595 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:39.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.367+0000 7fb8a941a700 1 --2- 172.21.15.16:0/306548595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8a41034b0 0x7fb8a4198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:39.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.367+0000 7fb8aa41c700 1 --2- 172.21.15.16:0/306548595 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8a4103e50 0x7fb8a419f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:39.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.367+0000 7fb8a9c1b700 1 -- 172.21.15.16:0/306548595 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8a4103e50 msgr2=0x7fb8a419f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:39.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.367+0000 7fb8a9c1b700 1 --2- 172.21.15.16:0/306548595 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8a4103e50 0x7fb8a419f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:39.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.367+0000 7fb8a9c1b700 1 -- 172.21.15.16:0/306548595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8a41034b0 msgr2=0x7fb8a4198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:39.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.367+0000 7fb8a9c1b700 1 --2- 172.21.15.16:0/306548595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8a41034b0 0x7fb8a4198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:39.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.367+0000 7fb8a9c1b700 1 -- 172.21.15.16:0/306548595 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8a419ff30 con 0x7fb8a41022b0 2024-03-28T22:08:39.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.367+0000 7fb8a9c1b700 1 --2- 172.21.15.16:0/306548595 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8a41022b0 0x7fb8a41982d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb8a000d980 tx=0x7fb8a000dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:39.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.367+0000 7fb89affd700 1 -- 172.21.15.16:0/306548595 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a0017030 con 0x7fb8a41022b0 2024-03-28T22:08:39.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.367+0000 7fb89affd700 1 -- 172.21.15.16:0/306548595 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8a0004500 con 0x7fb8a41022b0 2024-03-28T22:08:39.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.367+0000 7fb8abe7f700 1 -- 172.21.15.16:0/306548595 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8a41a0220 con 0x7fb8a41022b0 2024-03-28T22:08:39.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.368+0000 7fb89affd700 1 -- 172.21.15.16:0/306548595 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a00039e0 con 0x7fb8a41022b0 2024-03-28T22:08:39.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.368+0000 7fb8abe7f700 1 -- 172.21.15.16:0/306548595 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8a41a07b0 con 0x7fb8a41022b0 2024-03-28T22:08:39.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.368+0000 7fb8abe7f700 1 -- 172.21.15.16:0/306548595 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8a4050750 con 0x7fb8a41022b0 2024-03-28T22:08:39.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.369+0000 7fb89affd700 1 -- 172.21.15.16:0/306548595 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fb8a001a070 con 0x7fb8a41022b0 2024-03-28T22:08:39.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.370+0000 7fb89affd700 1 --2- 172.21.15.16:0/306548595 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb8900a1fd0 0x7fb8900a4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:39.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.370+0000 7fb8a941a700 1 --2- 172.21.15.16:0/306548595 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb8900a1fd0 0x7fb8900a4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:39.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.370+0000 7fb89affd700 1 -- 172.21.15.16:0/306548595 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fb8a00cd6b0 con 0x7fb8a41022b0 2024-03-28T22:08:39.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.370+0000 7fb8a941a700 1 --2- 172.21.15.16:0/306548595 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb8900a1fd0 0x7fb8900a4490 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb894009c60 tx=0x7fb894009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:39.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.375+0000 7fb89affd700 1 -- 172.21.15.16:0/306548595 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb8a009aa30 con 0x7fb8a41022b0 2024-03-28T22:08:39.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:39 smithi016 ceph-mon[32374]: pgmap v291: 97 pgs: 97 active+clean; 589 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5.3 KiB/s wr, 17 op/s 2024-03-28T22:08:39.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.660+0000 7fb8abe7f700 1 -- 172.21.15.16:0/306548595 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fb8a402dc10 con 0x7fb8a41022b0 2024-03-28T22:08:39.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.665+0000 7fb89affd700 1 -- 172.21.15.16:0/306548595 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7fb8a009a330 con 0x7fb8a41022b0 2024-03-28T22:08:39.668 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2024-03-28T22:08:39.668 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQBn6gVm4nOCJxAA+m/8qBDVp/UhXYNPXSbSjw== 2024-03-28T22:08:39.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.667+0000 7fb8abe7f700 1 -- 172.21.15.16:0/306548595 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb8900a1fd0 msgr2=0x7fb8900a4490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:39.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.667+0000 7fb8abe7f700 1 --2- 172.21.15.16:0/306548595 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb8900a1fd0 0x7fb8900a4490 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb894009c60 tx=0x7fb894009260 comp rx=0 tx=0).stop 2024-03-28T22:08:39.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.667+0000 7fb8abe7f700 1 -- 172.21.15.16:0/306548595 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8a41022b0 msgr2=0x7fb8a41982d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:39.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.667+0000 7fb8abe7f700 1 --2- 172.21.15.16:0/306548595 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8a41022b0 0x7fb8a41982d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb8a000d980 tx=0x7fb8a000dc90 comp rx=0 tx=0).stop 2024-03-28T22:08:39.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.667+0000 7fb8abe7f700 1 -- 172.21.15.16:0/306548595 shutdown_connections 2024-03-28T22:08:39.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.667+0000 7fb8abe7f700 1 --2- 172.21.15.16:0/306548595 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb8900a1fd0 0x7fb8900a4490 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:39.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.667+0000 7fb8abe7f700 1 --2- 172.21.15.16:0/306548595 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8a4103e50 0x7fb8a419f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:39.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.667+0000 7fb8abe7f700 1 --2- 172.21.15.16:0/306548595 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8a41022b0 0x7fb8a41982d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:39.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.667+0000 7fb8abe7f700 1 --2- 172.21.15.16:0/306548595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8a41034b0 0x7fb8a4198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:39.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.667+0000 7fb8abe7f700 1 -- 172.21.15.16:0/306548595 >> 172.21.15.16:0/306548595 conn(0x7fb8a40fd850 msgr2=0x7fb8a41066e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:39.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.667+0000 7fb8abe7f700 1 -- 172.21.15.16:0/306548595 shutdown_connections 2024-03-28T22:08:39.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:39.667+0000 7fb8abe7f700 1 -- 172.21.15.16:0/306548595 wait complete. 2024-03-28T22:08:39.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:39 smithi063 ceph-mon[36249]: pgmap v291: 97 pgs: 97 active+clean; 589 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5.3 KiB/s wr, 17 op/s 2024-03-28T22:08:39.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:39 smithi053 ceph-mon[35979]: pgmap v291: 97 pgs: 97 active+clean; 589 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5.3 KiB/s wr, 17 op/s 2024-03-28T22:08:40.284 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-28T22:08:40.284 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-28T22:08:40.284 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-28T22:08:40.327 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-28T22:08:40.328 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-28T22:08:40.328 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph mgr dump --format=json 2024-03-28T22:08:40.564 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:40 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/306548595' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T22:08:40.564 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:40 smithi016 ceph-mon[32374]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T22:08:40.564 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:40 smithi016 ceph-mon[32374]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-28T22:08:40.615 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:08:40.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:40 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/306548595' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T22:08:40.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:40 smithi063 ceph-mon[36249]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T22:08:40.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:40 smithi063 ceph-mon[36249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-28T22:08:40.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:40 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/306548595' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T22:08:40.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:40 smithi053 ceph-mon[35979]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T22:08:40.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:40 smithi053 ceph-mon[35979]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-28T22:08:41.673 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:41 smithi016 ceph-mon[32374]: pgmap v292: 97 pgs: 97 active+clean; 618 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 7.9 KiB/s wr, 19 op/s 2024-03-28T22:08:41.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:41 smithi063 ceph-mon[36249]: pgmap v292: 97 pgs: 97 active+clean; 618 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 7.9 KiB/s wr, 19 op/s 2024-03-28T22:08:41.698 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:41 smithi053 ceph-mon[35979]: pgmap v292: 97 pgs: 97 active+clean; 618 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 7.9 KiB/s wr, 19 op/s 2024-03-28T22:08:42.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.185+0000 7f0e8037e700 1 -- 172.21.15.16:0/398883035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e78101d80 msgr2=0x7f0e78105d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:42.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.185+0000 7f0e8037e700 1 --2- 172.21.15.16:0/398883035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e78101d80 0x7f0e78105d70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c009990 tx=0x7f0e6c009ca0 comp rx=0 tx=0).stop 2024-03-28T22:08:42.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.185+0000 7f0e8037e700 1 -- 172.21.15.16:0/398883035 shutdown_connections 2024-03-28T22:08:42.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.185+0000 7f0e8037e700 1 --2- 172.21.15.16:0/398883035 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e78106450 0x7f0e7810e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:42.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.185+0000 7f0e8037e700 1 --2- 172.21.15.16:0/398883035 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e781013d0 0x7f0e781017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:42.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.185+0000 7f0e8037e700 1 --2- 172.21.15.16:0/398883035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e78101d80 0x7f0e78105d70 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:42.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.185+0000 7f0e8037e700 1 -- 172.21.15.16:0/398883035 >> 172.21.15.16:0/398883035 conn(0x7f0e78077180 msgr2=0x7f0e78077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:42.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.185+0000 7f0e8037e700 1 -- 172.21.15.16:0/398883035 shutdown_connections 2024-03-28T22:08:42.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.185+0000 7f0e8037e700 1 -- 172.21.15.16:0/398883035 wait complete. 2024-03-28T22:08:42.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.185+0000 7f0e8037e700 1 Processor -- start 2024-03-28T22:08:42.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.186+0000 7f0e8037e700 1 -- start start 2024-03-28T22:08:42.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.186+0000 7f0e8037e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e781013d0 0x7f0e7819e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:42.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.186+0000 7f0e8037e700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e78101d80 0x7f0e7819ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:42.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.186+0000 7f0e8037e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e78106450 0x7f0e78198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:42.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.186+0000 7f0e8037e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0e78111410 con 0x7f0e78106450 2024-03-28T22:08:42.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.186+0000 7f0e8037e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0e78111290 con 0x7f0e781013d0 2024-03-28T22:08:42.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.186+0000 7f0e8037e700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0e78111590 con 0x7f0e78101d80 2024-03-28T22:08:42.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.186+0000 7f0e7e91b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e78106450 0x7f0e78198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:42.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.186+0000 7f0e7e91b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e78106450 0x7f0e78198810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33392/0 (socket says 172.21.15.16:33392) 2024-03-28T22:08:42.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.186+0000 7f0e7e91b700 1 -- 172.21.15.16:0/2912929566 learned_addr learned my addr 172.21.15.16:0/2912929566 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:42.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.186+0000 7f0e7e11a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e781013d0 0x7f0e7819e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:42.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.186+0000 7f0e7d919700 1 --2- 172.21.15.16:0/2912929566 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e78101d80 0x7f0e7819ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:42.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.186+0000 7f0e7e91b700 1 -- 172.21.15.16:0/2912929566 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e78101d80 msgr2=0x7f0e7819ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:42.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.186+0000 7f0e7e91b700 1 --2- 172.21.15.16:0/2912929566 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e78101d80 0x7f0e7819ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:42.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.186+0000 7f0e7e91b700 1 -- 172.21.15.16:0/2912929566 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e781013d0 msgr2=0x7f0e7819e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:42.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.186+0000 7f0e7e91b700 1 --2- 172.21.15.16:0/2912929566 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e781013d0 0x7f0e7819e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:42.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.186+0000 7f0e7e91b700 1 -- 172.21.15.16:0/2912929566 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e78198ff0 con 0x7f0e78106450 2024-03-28T22:08:42.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.186+0000 7f0e7e11a700 1 --2- 172.21.15.16:0/2912929566 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e781013d0 0x7f0e7819e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:08:42.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.186+0000 7f0e7e91b700 1 --2- 172.21.15.16:0/2912929566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e78106450 0x7f0e78198810 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0e7000ef30 tx=0x7f0e7000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:42.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.187+0000 7f0e6b7fe700 1 -- 172.21.15.16:0/2912929566 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e700092e0 con 0x7f0e78106450 2024-03-28T22:08:42.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.187+0000 7f0e8037e700 1 -- 172.21.15.16:0/2912929566 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e78199280 con 0x7f0e78106450 2024-03-28T22:08:42.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.187+0000 7f0e6b7fe700 1 -- 172.21.15.16:0/2912929566 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e70004bb0 con 0x7f0e78106450 2024-03-28T22:08:42.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.187+0000 7f0e6b7fe700 1 -- 172.21.15.16:0/2912929566 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e70004280 con 0x7f0e78106450 2024-03-28T22:08:42.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.187+0000 7f0e8037e700 1 -- 172.21.15.16:0/2912929566 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e781a54f0 con 0x7f0e78106450 2024-03-28T22:08:42.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.188+0000 7f0e8037e700 1 -- 172.21.15.16:0/2912929566 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e78102850 con 0x7f0e78106450 2024-03-28T22:08:42.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.189+0000 7f0e6b7fe700 1 -- 172.21.15.16:0/2912929566 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f0e7001b030 con 0x7f0e78106450 2024-03-28T22:08:42.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.189+0000 7f0e6b7fe700 1 --2- 172.21.15.16:0/2912929566 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0e640a1ea0 0x7f0e640a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:42.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.189+0000 7f0e7e11a700 1 --2- 172.21.15.16:0/2912929566 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0e640a1ea0 0x7f0e640a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:42.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.189+0000 7f0e6b7fe700 1 -- 172.21.15.16:0/2912929566 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f0e700cd850 con 0x7f0e78106450 2024-03-28T22:08:42.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.190+0000 7f0e7e11a700 1 --2- 172.21.15.16:0/2912929566 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0e640a1ea0 0x7f0e640a4360 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0e740104e0 tx=0x7f0e7400b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:42.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.194+0000 7f0e6b7fe700 1 -- 172.21.15.16:0/2912929566 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0e70022020 con 0x7f0e78106450 2024-03-28T22:08:42.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.455+0000 7f0e8037e700 1 -- 172.21.15.16:0/2912929566 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f0e7802db30 con 0x7f0e78106450 2024-03-28T22:08:42.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.460+0000 7f0e6b7fe700 1 -- 172.21.15.16:0/2912929566 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+261464 (secure 0 0 0) 0x7f0e70010040 con 0x7f0e78106450 2024-03-28T22:08:42.467 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:08:42.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.463+0000 7f0e8037e700 1 -- 172.21.15.16:0/2912929566 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0e640a1ea0 msgr2=0x7f0e640a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:42.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.463+0000 7f0e8037e700 1 --2- 172.21.15.16:0/2912929566 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0e640a1ea0 0x7f0e640a4360 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0e740104e0 tx=0x7f0e7400b3c0 comp rx=0 tx=0).stop 2024-03-28T22:08:42.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.463+0000 7f0e8037e700 1 -- 172.21.15.16:0/2912929566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e78106450 msgr2=0x7f0e78198810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:42.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.463+0000 7f0e8037e700 1 --2- 172.21.15.16:0/2912929566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e78106450 0x7f0e78198810 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0e7000ef30 tx=0x7f0e7000c3a0 comp rx=0 tx=0).stop 2024-03-28T22:08:42.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.463+0000 7f0e8037e700 1 -- 172.21.15.16:0/2912929566 shutdown_connections 2024-03-28T22:08:42.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.463+0000 7f0e8037e700 1 --2- 172.21.15.16:0/2912929566 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0e640a1ea0 0x7f0e640a4360 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:42.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.463+0000 7f0e8037e700 1 --2- 172.21.15.16:0/2912929566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e78106450 0x7f0e78198810 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:42.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.463+0000 7f0e8037e700 1 --2- 172.21.15.16:0/2912929566 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e781013d0 0x7f0e7819e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:42.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.463+0000 7f0e8037e700 1 --2- 172.21.15.16:0/2912929566 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e78101d80 0x7f0e7819ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:42.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.463+0000 7f0e8037e700 1 -- 172.21.15.16:0/2912929566 >> 172.21.15.16:0/2912929566 conn(0x7f0e78077180 msgr2=0x7f0e7806ad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:42.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.463+0000 7f0e8037e700 1 -- 172.21.15.16:0/2912929566 shutdown_connections 2024-03-28T22:08:42.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:42.463+0000 7f0e8037e700 1 -- 172.21.15.16:0/2912929566 wait complete. 2024-03-28T22:08:42.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:42 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2912929566' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-28T22:08:42.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:42 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2912929566' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-28T22:08:42.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:42 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2912929566' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-28T22:08:43.330 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":21986004},{"type":"v1","addr":"172.21.15.16:6801","nonce":21986004}]},"active_addr":"172.21.15.16:6801/21986004","active_change":"2024-03-28T22:01:28.993600+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24115,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24121,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.16:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":2787995271}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":4029580607}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":2311413235}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":2117914406}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":2033853748}]}]} 2024-03-28T22:08:43.337 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-28T22:08:43.337 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-28T22:08:43.337 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-28T22:08:43.590 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:08:43.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:43 smithi016 ceph-mon[32374]: pgmap v293: 97 pgs: 97 active+clean; 618 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 6.7 KiB/s wr, 16 op/s 2024-03-28T22:08:43.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:43 smithi053 ceph-mon[35979]: pgmap v293: 97 pgs: 97 active+clean; 618 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 6.7 KiB/s wr, 16 op/s 2024-03-28T22:08:43.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:43 smithi063 ceph-mon[36249]: pgmap v293: 97 pgs: 97 active+clean; 618 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 6.7 KiB/s wr, 16 op/s 2024-03-28T22:08:45.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.115+0000 7f3e260f9700 1 -- 172.21.15.16:0/2595801924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e201011f0 msgr2=0x7f3e2010f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:45.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.115+0000 7f3e260f9700 1 --2- 172.21.15.16:0/2595801924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e201011f0 0x7f3e2010f150 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f3e1800b0a0 tx=0x7f3e1800b3b0 comp rx=0 tx=0).stop 2024-03-28T22:08:45.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.115+0000 7f3e260f9700 1 -- 172.21.15.16:0/2595801924 shutdown_connections 2024-03-28T22:08:45.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.115+0000 7f3e260f9700 1 --2- 172.21.15.16:0/2595801924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e201011f0 0x7f3e2010f150 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:45.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.115+0000 7f3e260f9700 1 --2- 172.21.15.16:0/2595801924 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e201072b0 0x7f3e20107690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:45.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.115+0000 7f3e260f9700 1 --2- 172.21.15.16:0/2595801924 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e20100830 0x7f3e20100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:45.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.115+0000 7f3e260f9700 1 -- 172.21.15.16:0/2595801924 >> 172.21.15.16:0/2595801924 conn(0x7f3e20077740 msgr2=0x7f3e200feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:45.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.115+0000 7f3e260f9700 1 -- 172.21.15.16:0/2595801924 shutdown_connections 2024-03-28T22:08:45.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.115+0000 7f3e260f9700 1 -- 172.21.15.16:0/2595801924 wait complete. 2024-03-28T22:08:45.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.116+0000 7f3e260f9700 1 Processor -- start 2024-03-28T22:08:45.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.116+0000 7f3e260f9700 1 -- start start 2024-03-28T22:08:45.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.116+0000 7f3e260f9700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e20100830 0x7f3e2019e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:45.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.116+0000 7f3e260f9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e201011f0 0x7f3e2019ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:45.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.116+0000 7f3e260f9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e201072b0 0x7f3e20198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:45.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.116+0000 7f3e260f9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3e201126f0 con 0x7f3e201011f0 2024-03-28T22:08:45.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.116+0000 7f3e260f9700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3e20112570 con 0x7f3e201072b0 2024-03-28T22:08:45.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.116+0000 7f3e260f9700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3e20112870 con 0x7f3e20100830 2024-03-28T22:08:45.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.117+0000 7f3e1effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e201011f0 0x7f3e2019ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:45.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.117+0000 7f3e1effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e201011f0 0x7f3e2019ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33408/0 (socket says 172.21.15.16:33408) 2024-03-28T22:08:45.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.117+0000 7f3e1effd700 1 -- 172.21.15.16:0/1823793223 learned_addr learned my addr 172.21.15.16:0/1823793223 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:45.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.117+0000 7f3e1ffff700 1 --2- 172.21.15.16:0/1823793223 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e201072b0 0x7f3e20198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:45.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.117+0000 7f3e1f7fe700 1 --2- 172.21.15.16:0/1823793223 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e20100830 0x7f3e2019e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:45.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.117+0000 7f3e1effd700 1 -- 172.21.15.16:0/1823793223 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e20100830 msgr2=0x7f3e2019e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:45.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.117+0000 7f3e1effd700 1 --2- 172.21.15.16:0/1823793223 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e20100830 0x7f3e2019e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:45.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.117+0000 7f3e1effd700 1 -- 172.21.15.16:0/1823793223 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e201072b0 msgr2=0x7f3e20198930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:45.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.117+0000 7f3e1effd700 1 --2- 172.21.15.16:0/1823793223 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e201072b0 0x7f3e20198930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:45.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.117+0000 7f3e1effd700 1 -- 172.21.15.16:0/1823793223 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e20199090 con 0x7f3e201011f0 2024-03-28T22:08:45.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.117+0000 7f3e1f7fe700 1 --2- 172.21.15.16:0/1823793223 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e20100830 0x7f3e2019e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:08:45.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.117+0000 7f3e1effd700 1 --2- 172.21.15.16:0/1823793223 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e201011f0 0x7f3e2019ecb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f3e1400b530 tx=0x7f3e1400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:45.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.117+0000 7f3e1cff9700 1 -- 172.21.15.16:0/1823793223 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e14015020 con 0x7f3e201011f0 2024-03-28T22:08:45.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.117+0000 7f3e260f9700 1 -- 172.21.15.16:0/1823793223 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e20199380 con 0x7f3e201011f0 2024-03-28T22:08:45.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.117+0000 7f3e1cff9700 1 -- 172.21.15.16:0/1823793223 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e14011d10 con 0x7f3e201011f0 2024-03-28T22:08:45.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.117+0000 7f3e1cff9700 1 -- 172.21.15.16:0/1823793223 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e1400f2e0 con 0x7f3e201011f0 2024-03-28T22:08:45.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.117+0000 7f3e260f9700 1 -- 172.21.15.16:0/1823793223 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e201a5640 con 0x7f3e201011f0 2024-03-28T22:08:45.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.118+0000 7f3e260f9700 1 -- 172.21.15.16:0/1823793223 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e20103b50 con 0x7f3e201011f0 2024-03-28T22:08:45.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.119+0000 7f3e1cff9700 1 -- 172.21.15.16:0/1823793223 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f3e14018020 con 0x7f3e201011f0 2024-03-28T22:08:45.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.120+0000 7f3e1cff9700 1 --2- 172.21.15.16:0/1823793223 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f3e080a1ea0 0x7f3e080a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:45.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.120+0000 7f3e1f7fe700 1 --2- 172.21.15.16:0/1823793223 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f3e080a1ea0 0x7f3e080a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:45.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.120+0000 7f3e1cff9700 1 -- 172.21.15.16:0/1823793223 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f3e140cd550 con 0x7f3e201011f0 2024-03-28T22:08:45.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.120+0000 7f3e1f7fe700 1 --2- 172.21.15.16:0/1823793223 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f3e080a1ea0 0x7f3e080a4360 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3e10005fd0 tx=0x7f3e10005f00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:45.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.124+0000 7f3e1cff9700 1 -- 172.21.15.16:0/1823793223 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3e1401f020 con 0x7f3e201011f0 2024-03-28T22:08:45.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.329+0000 7f3e260f9700 1 -- 172.21.15.16:0/1823793223 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3e201a58d0 con 0x7f3e201011f0 2024-03-28T22:08:45.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.331+0000 7f3e1cff9700 1 -- 172.21.15.16:0/1823793223 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22111 (secure 0 0 0) 0x7f3e1409aae0 con 0x7f3e201011f0 2024-03-28T22:08:45.333 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:08:45.333 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":86,"fsid":"798d9884-ed4e-11ee-95d0-87774f69a715","created":"2024-03-28T22:00:27.216099+0000","modified":"2024-03-28T22:08:30.939084+0000","last_up_change":"2024-03-28T22:07:40.251141+0000","last_in_change":"2024-03-28T22:07:19.456451+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:03:59.327832+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:08:28.123529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:08:28.919037+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"46e7c8f8-ecc8-43d6-ae53-0655728f3240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6803","nonce":1589692483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6805","nonce":1589692483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6809","nonce":1589692483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6807","nonce":1589692483}]},"public_addr":"172.21.15.16:6803/1589692483","cluster_addr":"172.21.15.16:6805/1589692483","heartbeat_back_addr":"172.21.15.16:6809/1589692483","heartbeat_front_addr":"172.21.15.16:6807/1589692483","state":["exists","up"]},{"osd":1,"uuid":"af6d9c22-2830-4ce2-a547-011df0c6cbad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6801","nonce":795531234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6803","nonce":795531234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6807","nonce":795531234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6805","nonce":795531234}]},"public_addr":"172.21.15.53:6801/795531234","cluster_addr":"172.21.15.53:6803/795531234","heartbeat_back_addr":"172.21.15.53:6807/795531234","heartbeat_front_addr":"172.21.15.53:6805/795531234","state":["exists","up"]},{"osd":2,"uuid":"b10adca9-110c-4f87-85e0-b7542709d31a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6801","nonce":2214131086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6803","nonce":2214131086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6807","nonce":2214131086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6805","nonce":2214131086}]},"public_addr":"172.21.15.63:6801/2214131086","cluster_addr":"172.21.15.63:6803/2214131086","heartbeat_back_addr":"172.21.15.63:6807/2214131086","heartbeat_front_addr":"172.21.15.63:6805/2214131086","state":["exists","up"]},{"osd":3,"uuid":"c83cf4bb-395a-4786-abf7-74ca93a33eb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6811","nonce":13176789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6813","nonce":13176789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6817","nonce":13176789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6815","nonce":13176789}]},"public_addr":"172.21.15.16:6811/13176789","cluster_addr":"172.21.15.16:6813/13176789","heartbeat_back_addr":"172.21.15.16:6817/13176789","heartbeat_front_addr":"172.21.15.16:6815/13176789","state":["exists","up"]},{"osd":4,"uuid":"941b3e54-47d6-4e85-8069-e730c3fa8eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6809","nonce":772161849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6811","nonce":772161849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6815","nonce":772161849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6813","nonce":772161849}]},"public_addr":"172.21.15.53:6809/772161849","cluster_addr":"172.21.15.53:6811/772161849","heartbeat_back_addr":"172.21.15.53:6815/772161849","heartbeat_front_addr":"172.21.15.53:6813/772161849","state":["exists","up"]},{"osd":5,"uuid":"51d89826-1d60-40fc-8fd8-09bfe1ab4125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6809","nonce":1602554852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6811","nonce":1602554852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6815","nonce":1602554852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6813","nonce":1602554852}]},"public_addr":"172.21.15.63:6809/1602554852","cluster_addr":"172.21.15.63:6811/1602554852","heartbeat_back_addr":"172.21.15.63:6815/1602554852","heartbeat_front_addr":"172.21.15.63:6813/1602554852","state":["exists","up"]},{"osd":6,"uuid":"978efe5d-2c55-4ff8-8a1e-9f891388945e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6819","nonce":4290459224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6821","nonce":4290459224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6825","nonce":4290459224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6823","nonce":4290459224}]},"public_addr":"172.21.15.16:6819/4290459224","cluster_addr":"172.21.15.16:6821/4290459224","heartbeat_back_addr":"172.21.15.16:6825/4290459224","heartbeat_front_addr":"172.21.15.16:6823/4290459224","state":["exists","up"]},{"osd":7,"uuid":"50d489cb-dbb8-48df-b868-b29edf12e04d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6817","nonce":1056467525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6819","nonce":1056467525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6823","nonce":1056467525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6821","nonce":1056467525}]},"public_addr":"172.21.15.53:6817/1056467525","cluster_addr":"172.21.15.53:6819/1056467525","heartbeat_back_addr":"172.21.15.53:6823/1056467525","heartbeat_front_addr":"172.21.15.53:6821/1056467525","state":["exists","up"]},{"osd":8,"uuid":"49b39414-cc9c-4057-ae3f-cc109cd979da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6817","nonce":2918516794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6819","nonce":2918516794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6823","nonce":2918516794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6821","nonce":2918516794}]},"public_addr":"172.21.15.63:6817/2918516794","cluster_addr":"172.21.15.63:6819/2918516794","heartbeat_back_addr":"172.21.15.63:6823/2918516794","heartbeat_front_addr":"172.21.15.63:6821/2918516794","state":["exists","up"]},{"osd":9,"uuid":"4c873e51-b783-4eb6-83a3-fee31722d60b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6827","nonce":1585624367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6829","nonce":1585624367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6833","nonce":1585624367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6831","nonce":1585624367}]},"public_addr":"172.21.15.16:6827/1585624367","cluster_addr":"172.21.15.16:6829/1585624367","heartbeat_back_addr":"172.21.15.16:6833/1585624367","heartbeat_front_addr":"172.21.15.16:6831/1585624367","state":["exists","up"]},{"osd":10,"uuid":"085a78bf-e2ad-481a-85a3-71d1fc549a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6825","nonce":3357446484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6827","nonce":3357446484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6831","nonce":3357446484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6829","nonce":3357446484}]},"public_addr":"172.21.15.53:6825/3357446484","cluster_addr":"172.21.15.53:6827/3357446484","heartbeat_back_addr":"172.21.15.53:6831/3357446484","heartbeat_front_addr":"172.21.15.53:6829/3357446484","state":["exists","up"]},{"osd":11,"uuid":"13302aa3-08b9-4639-9042-27d850b4d5f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6825","nonce":2167866570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6827","nonce":2167866570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6831","nonce":2167866570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6829","nonce":2167866570}]},"public_addr":"172.21.15.63:6825/2167866570","cluster_addr":"172.21.15.63:6827/2167866570","heartbeat_back_addr":"172.21.15.63:6831/2167866570","heartbeat_front_addr":"172.21.15.63:6829/2167866570","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:07.450323+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:31.363172+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:55.295021+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:19.794760+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:44.063492+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:08.195064+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:33.655328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:57.639414+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:22.402394+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:47.462050+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:12.226307+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:37.495625+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3821408150":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6801/1832422779":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/996598130":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/1062264551":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/317557779":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6801/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/3631746518":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6800/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/3116652786":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6800/1832422779":"2024-03-29T22:01:28.993434+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:08:45.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.333+0000 7f3e260f9700 1 -- 172.21.15.16:0/1823793223 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f3e080a1ea0 msgr2=0x7f3e080a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:45.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.333+0000 7f3e260f9700 1 --2- 172.21.15.16:0/1823793223 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f3e080a1ea0 0x7f3e080a4360 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3e10005fd0 tx=0x7f3e10005f00 comp rx=0 tx=0).stop 2024-03-28T22:08:45.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.333+0000 7f3e260f9700 1 -- 172.21.15.16:0/1823793223 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e201011f0 msgr2=0x7f3e2019ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:45.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.333+0000 7f3e260f9700 1 --2- 172.21.15.16:0/1823793223 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e201011f0 0x7f3e2019ecb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f3e1400b530 tx=0x7f3e1400b840 comp rx=0 tx=0).stop 2024-03-28T22:08:45.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.333+0000 7f3e260f9700 1 -- 172.21.15.16:0/1823793223 shutdown_connections 2024-03-28T22:08:45.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.333+0000 7f3e260f9700 1 --2- 172.21.15.16:0/1823793223 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f3e080a1ea0 0x7f3e080a4360 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:45.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.333+0000 7f3e260f9700 1 --2- 172.21.15.16:0/1823793223 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e201072b0 0x7f3e20198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:45.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.333+0000 7f3e260f9700 1 --2- 172.21.15.16:0/1823793223 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e20100830 0x7f3e2019e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:45.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.333+0000 7f3e260f9700 1 --2- 172.21.15.16:0/1823793223 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e201011f0 0x7f3e2019ecb0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:45.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.333+0000 7f3e260f9700 1 -- 172.21.15.16:0/1823793223 >> 172.21.15.16:0/1823793223 conn(0x7f3e20077740 msgr2=0x7f3e20105160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:45.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.333+0000 7f3e260f9700 1 -- 172.21.15.16:0/1823793223 shutdown_connections 2024-03-28T22:08:45.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:45.334+0000 7f3e260f9700 1 -- 172.21.15.16:0/1823793223 wait complete. 2024-03-28T22:08:45.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:45 smithi063 ceph-mon[36249]: pgmap v294: 97 pgs: 97 active+clean; 625 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 6.8 KiB/s wr, 17 op/s 2024-03-28T22:08:45.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:45 smithi016 ceph-mon[32374]: pgmap v294: 97 pgs: 97 active+clean; 625 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 6.8 KiB/s wr, 17 op/s 2024-03-28T22:08:45.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:45 smithi053 ceph-mon[35979]: pgmap v294: 97 pgs: 97 active+clean; 625 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 6.8 KiB/s wr, 17 op/s 2024-03-28T22:08:45.886 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-28T22:08:45.887 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-28T22:08:46.131 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:08:46.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:46 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/1823793223' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:08:46.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:46 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/1823793223' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:08:46.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:46 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/1823793223' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:08:47.241 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:47 smithi016 ceph-mon[32374]: pgmap v295: 97 pgs: 97 active+clean; 625 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 4.9 KiB/s wr, 12 op/s 2024-03-28T22:08:47.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:47 smithi063 ceph-mon[36249]: pgmap v295: 97 pgs: 97 active+clean; 625 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 4.9 KiB/s wr, 12 op/s 2024-03-28T22:08:47.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:47 smithi053 ceph-mon[35979]: pgmap v295: 97 pgs: 97 active+clean; 625 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 4.9 KiB/s wr, 12 op/s 2024-03-28T22:08:47.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.720+0000 7fc1dc4e3700 1 -- 172.21.15.16:0/4076237118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1d406b590 msgr2=0x7fc1d4106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:47.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.720+0000 7fc1dc4e3700 1 --2- 172.21.15.16:0/4076237118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1d406b590 0x7fc1d4106f50 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc1d0009990 tx=0x7fc1d0009ca0 comp rx=0 tx=0).stop 2024-03-28T22:08:47.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.721+0000 7fc1dc4e3700 1 -- 172.21.15.16:0/4076237118 shutdown_connections 2024-03-28T22:08:47.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.721+0000 7fc1dc4e3700 1 --2- 172.21.15.16:0/4076237118 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1d4107680 0x7fc1d410f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:47.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.721+0000 7fc1dc4e3700 1 --2- 172.21.15.16:0/4076237118 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc1d406abe0 0x7fc1d406afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:47.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.721+0000 7fc1dc4e3700 1 --2- 172.21.15.16:0/4076237118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1d406b590 0x7fc1d4106f50 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:47.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.721+0000 7fc1dc4e3700 1 -- 172.21.15.16:0/4076237118 >> 172.21.15.16:0/4076237118 conn(0x7fc1d4077fa0 msgr2=0x7fc1d40783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:47.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.721+0000 7fc1dc4e3700 1 -- 172.21.15.16:0/4076237118 shutdown_connections 2024-03-28T22:08:47.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.721+0000 7fc1dc4e3700 1 -- 172.21.15.16:0/4076237118 wait complete. 2024-03-28T22:08:47.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.721+0000 7fc1dc4e3700 1 Processor -- start 2024-03-28T22:08:47.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.721+0000 7fc1dc4e3700 1 -- start start 2024-03-28T22:08:47.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.722+0000 7fc1dc4e3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1d406abe0 0x7fc1d419e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:47.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.722+0000 7fc1dc4e3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1d406b590 0x7fc1d419ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:47.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.722+0000 7fc1dc4e3700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc1d4107680 0x7fc1d4198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:47.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.722+0000 7fc1dc4e3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d41125b0 con 0x7fc1d406abe0 2024-03-28T22:08:47.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.722+0000 7fc1dc4e3700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d4112430 con 0x7fc1d406b590 2024-03-28T22:08:47.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.722+0000 7fc1dc4e3700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d4112730 con 0x7fc1d4107680 2024-03-28T22:08:47.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.722+0000 7fc1da27f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1d406abe0 0x7fc1d419e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:47.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.722+0000 7fc1da27f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1d406abe0 0x7fc1d419e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33416/0 (socket says 172.21.15.16:33416) 2024-03-28T22:08:47.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.722+0000 7fc1da27f700 1 -- 172.21.15.16:0/1193279026 learned_addr learned my addr 172.21.15.16:0/1193279026 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:47.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.722+0000 7fc1d9a7e700 1 --2- 172.21.15.16:0/1193279026 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1d406b590 0x7fc1d419ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:47.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.722+0000 7fc1daa80700 1 --2- 172.21.15.16:0/1193279026 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc1d4107680 0x7fc1d4198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:47.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.722+0000 7fc1da27f700 1 -- 172.21.15.16:0/1193279026 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc1d4107680 msgr2=0x7fc1d4198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:47.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.722+0000 7fc1da27f700 1 --2- 172.21.15.16:0/1193279026 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc1d4107680 0x7fc1d4198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:47.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.722+0000 7fc1da27f700 1 -- 172.21.15.16:0/1193279026 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1d406b590 msgr2=0x7fc1d419ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:47.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.722+0000 7fc1da27f700 1 --2- 172.21.15.16:0/1193279026 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1d406b590 0x7fc1d419ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:47.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.722+0000 7fc1da27f700 1 -- 172.21.15.16:0/1193279026 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1d4199070 con 0x7fc1d406abe0 2024-03-28T22:08:47.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.722+0000 7fc1d9a7e700 1 --2- 172.21.15.16:0/1193279026 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1d406b590 0x7fc1d419ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:08:47.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.722+0000 7fc1da27f700 1 --2- 172.21.15.16:0/1193279026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1d406abe0 0x7fc1d419e700 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc1c400d980 tx=0x7fc1c400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:47.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.722+0000 7fc1cb7fe700 1 -- 172.21.15.16:0/1193279026 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c4017070 con 0x7fc1d406abe0 2024-03-28T22:08:47.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.723+0000 7fc1dc4e3700 1 -- 172.21.15.16:0/1193279026 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1d4199360 con 0x7fc1d406abe0 2024-03-28T22:08:47.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.723+0000 7fc1cb7fe700 1 -- 172.21.15.16:0/1193279026 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1c4005020 con 0x7fc1d406abe0 2024-03-28T22:08:47.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.723+0000 7fc1cb7fe700 1 -- 172.21.15.16:0/1193279026 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c4009bc0 con 0x7fc1d406abe0 2024-03-28T22:08:47.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.723+0000 7fc1dc4e3700 1 -- 172.21.15.16:0/1193279026 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1d41a55d0 con 0x7fc1d406abe0 2024-03-28T22:08:47.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.724+0000 7fc1dc4e3700 1 -- 172.21.15.16:0/1193279026 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1d4103a00 con 0x7fc1d406abe0 2024-03-28T22:08:47.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.725+0000 7fc1cb7fe700 1 -- 172.21.15.16:0/1193279026 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fc1c40051d0 con 0x7fc1d406abe0 2024-03-28T22:08:47.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.725+0000 7fc1cb7fe700 1 --2- 172.21.15.16:0/1193279026 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc1c00a1ea0 0x7fc1c00a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:47.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.725+0000 7fc1d9a7e700 1 --2- 172.21.15.16:0/1193279026 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc1c00a1ea0 0x7fc1c00a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:47.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.725+0000 7fc1cb7fe700 1 -- 172.21.15.16:0/1193279026 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fc1c40cdce0 con 0x7fc1d406abe0 2024-03-28T22:08:47.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.726+0000 7fc1d9a7e700 1 --2- 172.21.15.16:0/1193279026 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc1c00a1ea0 0x7fc1c00a4360 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc1d0006010 tx=0x7fc1d0005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:47.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.730+0000 7fc1cb7fe700 1 -- 172.21.15.16:0/1193279026 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc1c4020020 con 0x7fc1d406abe0 2024-03-28T22:08:47.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.936+0000 7fc1dc4e3700 1 -- 172.21.15.16:0/1193279026 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc1d402db30 con 0x7fc1d406abe0 2024-03-28T22:08:47.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.938+0000 7fc1cb7fe700 1 -- 172.21.15.16:0/1193279026 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22111 (secure 0 0 0) 0x7fc1c409b160 con 0x7fc1d406abe0 2024-03-28T22:08:47.940 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:08:47.940 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":86,"fsid":"798d9884-ed4e-11ee-95d0-87774f69a715","created":"2024-03-28T22:00:27.216099+0000","modified":"2024-03-28T22:08:30.939084+0000","last_up_change":"2024-03-28T22:07:40.251141+0000","last_in_change":"2024-03-28T22:07:19.456451+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:03:59.327832+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:08:28.123529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:08:28.919037+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"46e7c8f8-ecc8-43d6-ae53-0655728f3240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6803","nonce":1589692483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6805","nonce":1589692483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6809","nonce":1589692483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6807","nonce":1589692483}]},"public_addr":"172.21.15.16:6803/1589692483","cluster_addr":"172.21.15.16:6805/1589692483","heartbeat_back_addr":"172.21.15.16:6809/1589692483","heartbeat_front_addr":"172.21.15.16:6807/1589692483","state":["exists","up"]},{"osd":1,"uuid":"af6d9c22-2830-4ce2-a547-011df0c6cbad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6801","nonce":795531234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6803","nonce":795531234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6807","nonce":795531234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6805","nonce":795531234}]},"public_addr":"172.21.15.53:6801/795531234","cluster_addr":"172.21.15.53:6803/795531234","heartbeat_back_addr":"172.21.15.53:6807/795531234","heartbeat_front_addr":"172.21.15.53:6805/795531234","state":["exists","up"]},{"osd":2,"uuid":"b10adca9-110c-4f87-85e0-b7542709d31a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6801","nonce":2214131086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6803","nonce":2214131086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6807","nonce":2214131086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6805","nonce":2214131086}]},"public_addr":"172.21.15.63:6801/2214131086","cluster_addr":"172.21.15.63:6803/2214131086","heartbeat_back_addr":"172.21.15.63:6807/2214131086","heartbeat_front_addr":"172.21.15.63:6805/2214131086","state":["exists","up"]},{"osd":3,"uuid":"c83cf4bb-395a-4786-abf7-74ca93a33eb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6811","nonce":13176789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6813","nonce":13176789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6817","nonce":13176789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6815","nonce":13176789}]},"public_addr":"172.21.15.16:6811/13176789","cluster_addr":"172.21.15.16:6813/13176789","heartbeat_back_addr":"172.21.15.16:6817/13176789","heartbeat_front_addr":"172.21.15.16:6815/13176789","state":["exists","up"]},{"osd":4,"uuid":"941b3e54-47d6-4e85-8069-e730c3fa8eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6809","nonce":772161849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6811","nonce":772161849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6815","nonce":772161849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6813","nonce":772161849}]},"public_addr":"172.21.15.53:6809/772161849","cluster_addr":"172.21.15.53:6811/772161849","heartbeat_back_addr":"172.21.15.53:6815/772161849","heartbeat_front_addr":"172.21.15.53:6813/772161849","state":["exists","up"]},{"osd":5,"uuid":"51d89826-1d60-40fc-8fd8-09bfe1ab4125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6809","nonce":1602554852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6811","nonce":1602554852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6815","nonce":1602554852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6813","nonce":1602554852}]},"public_addr":"172.21.15.63:6809/1602554852","cluster_addr":"172.21.15.63:6811/1602554852","heartbeat_back_addr":"172.21.15.63:6815/1602554852","heartbeat_front_addr":"172.21.15.63:6813/1602554852","state":["exists","up"]},{"osd":6,"uuid":"978efe5d-2c55-4ff8-8a1e-9f891388945e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6819","nonce":4290459224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6821","nonce":4290459224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6825","nonce":4290459224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6823","nonce":4290459224}]},"public_addr":"172.21.15.16:6819/4290459224","cluster_addr":"172.21.15.16:6821/4290459224","heartbeat_back_addr":"172.21.15.16:6825/4290459224","heartbeat_front_addr":"172.21.15.16:6823/4290459224","state":["exists","up"]},{"osd":7,"uuid":"50d489cb-dbb8-48df-b868-b29edf12e04d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6817","nonce":1056467525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6819","nonce":1056467525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6823","nonce":1056467525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6821","nonce":1056467525}]},"public_addr":"172.21.15.53:6817/1056467525","cluster_addr":"172.21.15.53:6819/1056467525","heartbeat_back_addr":"172.21.15.53:6823/1056467525","heartbeat_front_addr":"172.21.15.53:6821/1056467525","state":["exists","up"]},{"osd":8,"uuid":"49b39414-cc9c-4057-ae3f-cc109cd979da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6817","nonce":2918516794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6819","nonce":2918516794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6823","nonce":2918516794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6821","nonce":2918516794}]},"public_addr":"172.21.15.63:6817/2918516794","cluster_addr":"172.21.15.63:6819/2918516794","heartbeat_back_addr":"172.21.15.63:6823/2918516794","heartbeat_front_addr":"172.21.15.63:6821/2918516794","state":["exists","up"]},{"osd":9,"uuid":"4c873e51-b783-4eb6-83a3-fee31722d60b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6827","nonce":1585624367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6829","nonce":1585624367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6833","nonce":1585624367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6831","nonce":1585624367}]},"public_addr":"172.21.15.16:6827/1585624367","cluster_addr":"172.21.15.16:6829/1585624367","heartbeat_back_addr":"172.21.15.16:6833/1585624367","heartbeat_front_addr":"172.21.15.16:6831/1585624367","state":["exists","up"]},{"osd":10,"uuid":"085a78bf-e2ad-481a-85a3-71d1fc549a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6825","nonce":3357446484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6827","nonce":3357446484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6831","nonce":3357446484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6829","nonce":3357446484}]},"public_addr":"172.21.15.53:6825/3357446484","cluster_addr":"172.21.15.53:6827/3357446484","heartbeat_back_addr":"172.21.15.53:6831/3357446484","heartbeat_front_addr":"172.21.15.53:6829/3357446484","state":["exists","up"]},{"osd":11,"uuid":"13302aa3-08b9-4639-9042-27d850b4d5f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6825","nonce":2167866570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6827","nonce":2167866570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6831","nonce":2167866570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6829","nonce":2167866570}]},"public_addr":"172.21.15.63:6825/2167866570","cluster_addr":"172.21.15.63:6827/2167866570","heartbeat_back_addr":"172.21.15.63:6831/2167866570","heartbeat_front_addr":"172.21.15.63:6829/2167866570","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:07.450323+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:31.363172+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:55.295021+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:19.794760+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:44.063492+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:08.195064+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:33.655328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:57.639414+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:22.402394+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:47.462050+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:12.226307+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:37.495625+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3821408150":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6801/1832422779":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/996598130":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/1062264551":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/317557779":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6801/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/3631746518":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6800/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/3116652786":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6800/1832422779":"2024-03-29T22:01:28.993434+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:08:47.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.940+0000 7fc1dc4e3700 1 -- 172.21.15.16:0/1193279026 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc1c00a1ea0 msgr2=0x7fc1c00a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:47.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.940+0000 7fc1dc4e3700 1 --2- 172.21.15.16:0/1193279026 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc1c00a1ea0 0x7fc1c00a4360 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc1d0006010 tx=0x7fc1d0005960 comp rx=0 tx=0).stop 2024-03-28T22:08:47.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.940+0000 7fc1dc4e3700 1 -- 172.21.15.16:0/1193279026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1d406abe0 msgr2=0x7fc1d419e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:47.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.940+0000 7fc1dc4e3700 1 --2- 172.21.15.16:0/1193279026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1d406abe0 0x7fc1d419e700 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc1c400d980 tx=0x7fc1c400dc90 comp rx=0 tx=0).stop 2024-03-28T22:08:47.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.940+0000 7fc1dc4e3700 1 -- 172.21.15.16:0/1193279026 shutdown_connections 2024-03-28T22:08:47.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.940+0000 7fc1dc4e3700 1 --2- 172.21.15.16:0/1193279026 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc1c00a1ea0 0x7fc1c00a4360 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:47.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.940+0000 7fc1dc4e3700 1 --2- 172.21.15.16:0/1193279026 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc1d4107680 0x7fc1d4198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:47.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.940+0000 7fc1dc4e3700 1 --2- 172.21.15.16:0/1193279026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1d406abe0 0x7fc1d419e700 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:47.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.940+0000 7fc1dc4e3700 1 --2- 172.21.15.16:0/1193279026 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1d406b590 0x7fc1d419ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:47.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.940+0000 7fc1dc4e3700 1 -- 172.21.15.16:0/1193279026 >> 172.21.15.16:0/1193279026 conn(0x7fc1d4077fa0 msgr2=0x7fc1d41087d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:47.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.940+0000 7fc1dc4e3700 1 -- 172.21.15.16:0/1193279026 shutdown_connections 2024-03-28T22:08:47.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:47.940+0000 7fc1dc4e3700 1 -- 172.21.15.16:0/1193279026 wait complete. 2024-03-28T22:08:48.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:48 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/1193279026' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:08:48.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:48 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/1193279026' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:08:48.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:48 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/1193279026' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:08:48.672 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-28T22:08:48.679 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-28T22:08:48.679 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-28T22:08:48.680 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-28T22:08:48.680 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-28T22:08:48.681 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-28T22:08:48.681 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-28T22:08:48.681 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-28T22:08:48.682 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-28T22:08:48.683 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-28T22:08:48.684 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-28T22:08:48.684 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-28T22:08:49.352 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:08:49.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:49 smithi063 ceph-mon[36249]: pgmap v296: 97 pgs: 97 active+clean; 625 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 3.7 KiB/s wr, 8 op/s 2024-03-28T22:08:49.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:49 smithi016 ceph-mon[32374]: pgmap v296: 97 pgs: 97 active+clean; 625 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 3.7 KiB/s wr, 8 op/s 2024-03-28T22:08:49.508 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:08:49.509 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:08:49.510 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:08:49.510 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:08:49.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:49 smithi053 ceph-mon[35979]: pgmap v296: 97 pgs: 97 active+clean; 625 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 3.7 KiB/s wr, 8 op/s 2024-03-28T22:08:49.614 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:08:49.723 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:08:50.446 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:08:50.447 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:08:50.447 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:08:50.448 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:08:50.448 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:08:51.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:51 smithi063 ceph-mon[36249]: pgmap v297: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 4.7 KiB/s wr, 9 op/s 2024-03-28T22:08:51.274 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:51 smithi016 ceph-mon[32374]: pgmap v297: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 4.7 KiB/s wr, 9 op/s 2024-03-28T22:08:51.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:51 smithi053 ceph-mon[35979]: pgmap v297: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 4.7 KiB/s wr, 9 op/s 2024-03-28T22:08:53.242 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:53 smithi016 ceph-mon[32374]: pgmap v298: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-03-28T22:08:53.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:53 smithi063 ceph-mon[36249]: pgmap v298: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-03-28T22:08:53.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:53 smithi053 ceph-mon[35979]: pgmap v298: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-03-28T22:08:55.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:55 smithi063 ceph-mon[36249]: pgmap v299: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2024-03-28T22:08:55.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:55 smithi016 ceph-mon[32374]: pgmap v299: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2024-03-28T22:08:55.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:55 smithi053 ceph-mon[35979]: pgmap v299: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2024-03-28T22:08:57.278 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:57 smithi016 ceph-mon[32374]: pgmap v300: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-28T22:08:57.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:57 smithi063 ceph-mon[36249]: pgmap v300: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-28T22:08:57.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:57 smithi053 ceph-mon[35979]: pgmap v300: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-28T22:08:58.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.791+0000 7f795a297700 1 -- 172.21.15.16:0/1964803474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7954101ae0 msgr2=0x7f795410dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:58.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.791+0000 7f795a297700 1 --2- 172.21.15.16:0/1964803474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7954101ae0 0x7f795410dfc0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f794c00b0a0 tx=0x7f794c00b3b0 comp rx=0 tx=0).stop 2024-03-28T22:08:58.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.791+0000 7f795a297700 1 -- 172.21.15.16:0/1964803474 shutdown_connections 2024-03-28T22:08:58.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.791+0000 7f795a297700 1 --2- 172.21.15.16:0/1964803474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7954101ae0 0x7f795410dfc0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:58.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.791+0000 7f795a297700 1 --2- 172.21.15.16:0/1964803474 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7954107180 0x7f7954107560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:58.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.791+0000 7f795a297700 1 --2- 172.21.15.16:0/1964803474 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7954101140 0x7f79541015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:58.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.791+0000 7f795a297700 1 -- 172.21.15.16:0/1964803474 >> 172.21.15.16:0/1964803474 conn(0x7f7954078070 msgr2=0x7f7954078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:58.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.792+0000 7f795a297700 1 -- 172.21.15.16:0/1964803474 shutdown_connections 2024-03-28T22:08:58.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.792+0000 7f795a297700 1 -- 172.21.15.16:0/1964803474 wait complete. 2024-03-28T22:08:58.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.792+0000 7f795a297700 1 Processor -- start 2024-03-28T22:08:58.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.792+0000 7f795a297700 1 -- start start 2024-03-28T22:08:58.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.793+0000 7f795a297700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7954101140 0x7f7954199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:58.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.793+0000 7f795a297700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7954101ae0 0x7f7954199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:58.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.793+0000 7f795a297700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7954107180 0x7f79541a48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:58.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.793+0000 7f795a297700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f79541113d0 con 0x7f7954101ae0 2024-03-28T22:08:58.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.793+0000 7f795a297700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7954111250 con 0x7f7954107180 2024-03-28T22:08:58.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.793+0000 7f795a297700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7954111550 con 0x7f7954101140 2024-03-28T22:08:58.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.793+0000 7f79537fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7954101ae0 0x7f7954199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:58.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.793+0000 7f79537fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7954101ae0 0x7f7954199db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46562/0 (socket says 172.21.15.16:46562) 2024-03-28T22:08:58.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.793+0000 7f79537fe700 1 -- 172.21.15.16:0/3720415209 learned_addr learned my addr 172.21.15.16:0/3720415209 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:58.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.793+0000 7f7958834700 1 --2- 172.21.15.16:0/3720415209 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7954107180 0x7f79541a48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:58.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.793+0000 7f7953fff700 1 --2- 172.21.15.16:0/3720415209 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7954101140 0x7f7954199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:58.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.793+0000 7f79537fe700 1 -- 172.21.15.16:0/3720415209 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7954101140 msgr2=0x7f7954199870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:58.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.793+0000 7f79537fe700 1 --2- 172.21.15.16:0/3720415209 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7954101140 0x7f7954199870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:58.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.793+0000 7f79537fe700 1 -- 172.21.15.16:0/3720415209 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7954107180 msgr2=0x7f79541a48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:58.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.793+0000 7f79537fe700 1 --2- 172.21.15.16:0/3720415209 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7954107180 0x7f79541a48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:58.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.793+0000 7f79537fe700 1 -- 172.21.15.16:0/3720415209 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79541a4e30 con 0x7f7954101ae0 2024-03-28T22:08:58.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.793+0000 7f7953fff700 1 --2- 172.21.15.16:0/3720415209 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7954101140 0x7f7954199870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:08:58.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.793+0000 7f79537fe700 1 --2- 172.21.15.16:0/3720415209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7954101ae0 0x7f7954199db0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f794800d950 tx=0x7f794800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:58.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.793+0000 7f79517fa700 1 -- 172.21.15.16:0/3720415209 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7948016070 con 0x7f7954101ae0 2024-03-28T22:08:58.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.793+0000 7f79517fa700 1 -- 172.21.15.16:0/3720415209 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7948009e20 con 0x7f7954101ae0 2024-03-28T22:08:58.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.793+0000 7f795a297700 1 -- 172.21.15.16:0/3720415209 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79541a5000 con 0x7f7954101ae0 2024-03-28T22:08:58.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.793+0000 7f79517fa700 1 -- 172.21.15.16:0/3720415209 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f794800f620 con 0x7f7954101ae0 2024-03-28T22:08:58.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.793+0000 7f795a297700 1 -- 172.21.15.16:0/3720415209 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79541a5510 con 0x7f7954101ae0 2024-03-28T22:08:58.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.794+0000 7f795a297700 1 -- 172.21.15.16:0/3720415209 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7954062910 con 0x7f7954101ae0 2024-03-28T22:08:58.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.795+0000 7f79517fa700 1 -- 172.21.15.16:0/3720415209 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f7948019070 con 0x7f7954101ae0 2024-03-28T22:08:58.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.796+0000 7f79517fa700 1 --2- 172.21.15.16:0/3720415209 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f793c0a1ea0 0x7f793c0a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:58.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.796+0000 7f79517fa700 1 -- 172.21.15.16:0/3720415209 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f79480cde80 con 0x7f7954101ae0 2024-03-28T22:08:58.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.796+0000 7f7953fff700 1 --2- 172.21.15.16:0/3720415209 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f793c0a1ea0 0x7f793c0a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:58.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.796+0000 7f79517fa700 1 --2- 172.21.15.16:0/3720415209 >> [v2:172.21.15.63:6816/2918516794,v1:172.21.15.63:6817/2918516794] conn(0x7f793c0ab680 0x7f793c0adaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:58.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.796+0000 7f79517fa700 1 -- 172.21.15.16:0/3720415209 --> [v2:172.21.15.63:6816/2918516794,v1:172.21.15.63:6817/2918516794] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f794801ee50 con 0x7f793c0ab680 2024-03-28T22:08:58.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.796+0000 7f79517fa700 1 -- 172.21.15.16:0/3720415209 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f79480ce360 con 0x7f7954101ae0 2024-03-28T22:08:58.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.796+0000 7f7953fff700 1 --2- 172.21.15.16:0/3720415209 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f793c0a1ea0 0x7f793c0a4360 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7944009bc0 tx=0x7f7944009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:58.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.797+0000 7f7958834700 1 --2- 172.21.15.16:0/3720415209 >> [v2:172.21.15.63:6816/2918516794,v1:172.21.15.63:6817/2918516794] conn(0x7f793c0ab680 0x7f793c0adaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:58.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.797+0000 7f7958834700 1 --2- 172.21.15.16:0/3720415209 >> [v2:172.21.15.63:6816/2918516794,v1:172.21.15.63:6817/2918516794] conn(0x7f793c0ab680 0x7f793c0adaa0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:58.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.798+0000 7f79517fa700 1 -- 172.21.15.16:0/3720415209 <== osd.8 v2:172.21.15.63:6816/2918516794 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f794801ee50 con 0x7f793c0ab680 2024-03-28T22:08:58.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.814+0000 7f795a297700 1 -- 172.21.15.16:0/3720415209 --> [v2:172.21.15.63:6816/2918516794,v1:172.21.15.63:6817/2918516794] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7954062d40 con 0x7f793c0ab680 2024-03-28T22:08:58.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.815+0000 7f79517fa700 1 -- 172.21.15.16:0/3720415209 <== osd.8 v2:172.21.15.63:6816/2918516794 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7954062d40 con 0x7f793c0ab680 2024-03-28T22:08:58.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.815+0000 7f795a297700 1 -- 172.21.15.16:0/3720415209 >> [v2:172.21.15.63:6816/2918516794,v1:172.21.15.63:6817/2918516794] conn(0x7f793c0ab680 msgr2=0x7f793c0adaa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:58.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.815+0000 7f795a297700 1 --2- 172.21.15.16:0/3720415209 >> [v2:172.21.15.63:6816/2918516794,v1:172.21.15.63:6817/2918516794] conn(0x7f793c0ab680 0x7f793c0adaa0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:58.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.815+0000 7f795a297700 1 -- 172.21.15.16:0/3720415209 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f793c0a1ea0 msgr2=0x7f793c0a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:58.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.815+0000 7f795a297700 1 --2- 172.21.15.16:0/3720415209 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f793c0a1ea0 0x7f793c0a4360 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7944009bc0 tx=0x7f7944009260 comp rx=0 tx=0).stop 2024-03-28T22:08:58.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.815+0000 7f795a297700 1 -- 172.21.15.16:0/3720415209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7954101ae0 msgr2=0x7f7954199db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:58.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.815+0000 7f795a297700 1 --2- 172.21.15.16:0/3720415209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7954101ae0 0x7f7954199db0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f794800d950 tx=0x7f794800dc60 comp rx=0 tx=0).stop 2024-03-28T22:08:58.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.815+0000 7f7958834700 1 -- 172.21.15.16:0/3720415209 reap_dead start 2024-03-28T22:08:58.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.815+0000 7f795a297700 1 -- 172.21.15.16:0/3720415209 shutdown_connections 2024-03-28T22:08:58.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.815+0000 7f795a297700 1 -- 172.21.15.16:0/3720415209 >> 172.21.15.16:0/3720415209 conn(0x7f7954078070 msgr2=0x7f79540ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:58.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.815+0000 7f795a297700 1 -- 172.21.15.16:0/3720415209 shutdown_connections 2024-03-28T22:08:58.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:58.815+0000 7f795a297700 1 -- 172.21.15.16:0/3720415209 wait complete. 2024-03-28T22:08:59.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.111+0000 7f0309dbc700 1 -- 172.21.15.16:0/2678096706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03041072b0 msgr2=0x7f0304107690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:59.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.111+0000 7f0309dbc700 1 --2- 172.21.15.16:0/2678096706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03041072b0 0x7f0304107690 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f02f4009990 tx=0x7f02f4009ca0 comp rx=0 tx=0).stop 2024-03-28T22:08:59.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.111+0000 7f0309dbc700 1 -- 172.21.15.16:0/2678096706 shutdown_connections 2024-03-28T22:08:59.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.111+0000 7f0309dbc700 1 --2- 172.21.15.16:0/2678096706 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f03041011f0 0x7f030410f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:59.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.111+0000 7f0309dbc700 1 --2- 172.21.15.16:0/2678096706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03041072b0 0x7f0304107690 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:59.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.111+0000 7f0309dbc700 1 --2- 172.21.15.16:0/2678096706 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0304100830 0x7f0304100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:59.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.111+0000 7f0309dbc700 1 -- 172.21.15.16:0/2678096706 >> 172.21.15.16:0/2678096706 conn(0x7f0304077740 msgr2=0x7f03040feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:59.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.111+0000 7f0309dbc700 1 -- 172.21.15.16:0/2678096706 shutdown_connections 2024-03-28T22:08:59.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.111+0000 7f0309dbc700 1 -- 172.21.15.16:0/2678096706 wait complete. 2024-03-28T22:08:59.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.112+0000 7f0309dbc700 1 Processor -- start 2024-03-28T22:08:59.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.112+0000 7f0309dbc700 1 -- start start 2024-03-28T22:08:59.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.112+0000 7f0309dbc700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0304100830 0x7f030419e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:59.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.112+0000 7f0309dbc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03041011f0 0x7f030419ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:59.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.112+0000 7f0309dbc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f03041072b0 0x7f0304198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:59.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.112+0000 7f0309dbc700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0304112670 con 0x7f03041011f0 2024-03-28T22:08:59.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.112+0000 7f0309dbc700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f03041124f0 con 0x7f03041072b0 2024-03-28T22:08:59.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.112+0000 7f0309dbc700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f03041127f0 con 0x7f0304100830 2024-03-28T22:08:59.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.112+0000 7f0302ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03041011f0 0x7f030419ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:59.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.112+0000 7f0302ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03041011f0 0x7f030419ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46590/0 (socket says 172.21.15.16:46590) 2024-03-28T22:08:59.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.112+0000 7f0302ffd700 1 -- 172.21.15.16:0/1922806465 learned_addr learned my addr 172.21.15.16:0/1922806465 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:08:59.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.112+0000 7f03037fe700 1 --2- 172.21.15.16:0/1922806465 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0304100830 0x7f030419e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:59.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.112+0000 7f0303fff700 1 --2- 172.21.15.16:0/1922806465 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f03041072b0 0x7f0304198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:59.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.112+0000 7f0302ffd700 1 -- 172.21.15.16:0/1922806465 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0304100830 msgr2=0x7f030419e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:59.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.112+0000 7f0302ffd700 1 --2- 172.21.15.16:0/1922806465 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0304100830 0x7f030419e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:59.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.112+0000 7f0302ffd700 1 -- 172.21.15.16:0/1922806465 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f03041072b0 msgr2=0x7f0304198930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:59.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.112+0000 7f0302ffd700 1 --2- 172.21.15.16:0/1922806465 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f03041072b0 0x7f0304198930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:59.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.112+0000 7f0302ffd700 1 -- 172.21.15.16:0/1922806465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0304199090 con 0x7f03041011f0 2024-03-28T22:08:59.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.113+0000 7f0302ffd700 1 --2- 172.21.15.16:0/1922806465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03041011f0 0x7f030419ec60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f02f800e9a0 tx=0x7f02f800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:59.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.113+0000 7f0300ff9700 1 -- 172.21.15.16:0/1922806465 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f800ca10 con 0x7f03041011f0 2024-03-28T22:08:59.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.113+0000 7f0300ff9700 1 -- 172.21.15.16:0/1922806465 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02f800cbb0 con 0x7f03041011f0 2024-03-28T22:08:59.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.113+0000 7f0309dbc700 1 -- 172.21.15.16:0/1922806465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0304199380 con 0x7f03041011f0 2024-03-28T22:08:59.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.113+0000 7f0300ff9700 1 -- 172.21.15.16:0/1922806465 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f8018680 con 0x7f03041011f0 2024-03-28T22:08:59.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.113+0000 7f0309dbc700 1 -- 172.21.15.16:0/1922806465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03041a55f0 con 0x7f03041011f0 2024-03-28T22:08:59.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.114+0000 7f0309dbc700 1 -- 172.21.15.16:0/1922806465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0304062950 con 0x7f03041011f0 2024-03-28T22:08:59.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.115+0000 7f0300ff9700 1 -- 172.21.15.16:0/1922806465 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f02f8010430 con 0x7f03041011f0 2024-03-28T22:08:59.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.115+0000 7f0300ff9700 1 --2- 172.21.15.16:0/1922806465 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f02ec0a1f20 0x7f02ec0a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:59.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.115+0000 7f03037fe700 1 --2- 172.21.15.16:0/1922806465 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f02ec0a1f20 0x7f02ec0a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:59.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.115+0000 7f0300ff9700 1 -- 172.21.15.16:0/1922806465 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f02f80ce8d0 con 0x7f03041011f0 2024-03-28T22:08:59.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.116+0000 7f0300ff9700 1 --2- 172.21.15.16:0/1922806465 >> [v2:172.21.15.16:6802/1589692483,v1:172.21.15.16:6803/1589692483] conn(0x7f02ec0ab700 0x7f02ec0adb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:08:59.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.116+0000 7f0300ff9700 1 -- 172.21.15.16:0/1922806465 --> [v2:172.21.15.16:6802/1589692483,v1:172.21.15.16:6803/1589692483] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f02f801bec0 con 0x7f02ec0ab700 2024-03-28T22:08:59.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.116+0000 7f0300ff9700 1 -- 172.21.15.16:0/1922806465 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f02f80cecc0 con 0x7f03041011f0 2024-03-28T22:08:59.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.116+0000 7f0303fff700 1 --2- 172.21.15.16:0/1922806465 >> [v2:172.21.15.16:6802/1589692483,v1:172.21.15.16:6803/1589692483] conn(0x7f02ec0ab700 0x7f02ec0adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:08:59.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.116+0000 7f03037fe700 1 --2- 172.21.15.16:0/1922806465 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f02ec0a1f20 0x7f02ec0a43e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f02f400b2e0 tx=0x7f02f401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:59.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.116+0000 7f0303fff700 1 --2- 172.21.15.16:0/1922806465 >> [v2:172.21.15.16:6802/1589692483,v1:172.21.15.16:6803/1589692483] conn(0x7f02ec0ab700 0x7f02ec0adb20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:08:59.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.117+0000 7f0300ff9700 1 -- 172.21.15.16:0/1922806465 <== osd.0 v2:172.21.15.16:6802/1589692483 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f02f801bec0 con 0x7f02ec0ab700 2024-03-28T22:08:59.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.134+0000 7f0309dbc700 1 -- 172.21.15.16:0/1922806465 --> [v2:172.21.15.16:6802/1589692483,v1:172.21.15.16:6803/1589692483] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0304062d50 con 0x7f02ec0ab700 2024-03-28T22:08:59.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.134+0000 7f0300ff9700 1 -- 172.21.15.16:0/1922806465 <== osd.0 v2:172.21.15.16:6802/1589692483 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0304062d50 con 0x7f02ec0ab700 2024-03-28T22:08:59.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.135+0000 7f0309dbc700 1 -- 172.21.15.16:0/1922806465 >> [v2:172.21.15.16:6802/1589692483,v1:172.21.15.16:6803/1589692483] conn(0x7f02ec0ab700 msgr2=0x7f02ec0adb20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:59.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.135+0000 7f0309dbc700 1 --2- 172.21.15.16:0/1922806465 >> [v2:172.21.15.16:6802/1589692483,v1:172.21.15.16:6803/1589692483] conn(0x7f02ec0ab700 0x7f02ec0adb20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:08:59.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.135+0000 7f0309dbc700 1 -- 172.21.15.16:0/1922806465 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f02ec0a1f20 msgr2=0x7f02ec0a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:59.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.135+0000 7f0309dbc700 1 --2- 172.21.15.16:0/1922806465 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f02ec0a1f20 0x7f02ec0a43e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f02f400b2e0 tx=0x7f02f401a040 comp rx=0 tx=0).stop 2024-03-28T22:08:59.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.135+0000 7f0309dbc700 1 -- 172.21.15.16:0/1922806465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03041011f0 msgr2=0x7f030419ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:08:59.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.135+0000 7f0309dbc700 1 --2- 172.21.15.16:0/1922806465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03041011f0 0x7f030419ec60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f02f800e9a0 tx=0x7f02f800ecb0 comp rx=0 tx=0).stop 2024-03-28T22:08:59.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.135+0000 7f0303fff700 1 -- 172.21.15.16:0/1922806465 reap_dead start 2024-03-28T22:08:59.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.135+0000 7f0309dbc700 1 -- 172.21.15.16:0/1922806465 shutdown_connections 2024-03-28T22:08:59.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.135+0000 7f0309dbc700 1 -- 172.21.15.16:0/1922806465 >> 172.21.15.16:0/1922806465 conn(0x7f0304077740 msgr2=0x7f0304105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:08:59.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.135+0000 7f0309dbc700 1 -- 172.21.15.16:0/1922806465 shutdown_connections 2024-03-28T22:08:59.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:08:59.135+0000 7f0309dbc700 1 -- 172.21.15.16:0/1922806465 wait complete. 2024-03-28T22:08:59.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:08:59 smithi063 ceph-mon[36249]: pgmap v301: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-03-28T22:08:59.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:08:59 smithi016 ceph-mon[32374]: pgmap v301: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-03-28T22:08:59.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:08:59 smithi053 ceph-mon[35979]: pgmap v301: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-03-28T22:09:01.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:01 smithi016 ceph-mon[32374]: pgmap v302: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-28T22:09:01.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:01 smithi063 ceph-mon[36249]: pgmap v302: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-28T22:09:01.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:01 smithi053 ceph-mon[35979]: pgmap v302: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-28T22:09:01.739 INFO:teuthology.orchestra.run.smithi016.stdout:34359738440 2024-03-28T22:09:01.739 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-28T22:09:02.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.051+0000 7fe672811700 1 -- 172.21.15.16:0/1112252130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe66c102c50 msgr2=0x7fe66c10f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:02.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.051+0000 7fe672811700 1 --2- 172.21.15.16:0/1112252130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe66c102c50 0x7fe66c10f090 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe66400b0d0 tx=0x7fe66400b3e0 comp rx=0 tx=0).stop 2024-03-28T22:09:02.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.051+0000 7fe672811700 1 -- 172.21.15.16:0/1112252130 shutdown_connections 2024-03-28T22:09:02.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.051+0000 7fe672811700 1 --2- 172.21.15.16:0/1112252130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe66c102c50 0x7fe66c10f090 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:02.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.051+0000 7fe672811700 1 --2- 172.21.15.16:0/1112252130 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe66c108270 0x7fe66c108650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:02.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.051+0000 7fe672811700 1 --2- 172.21.15.16:0/1112252130 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe66c102290 0x7fe66c102710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:02.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.051+0000 7fe672811700 1 -- 172.21.15.16:0/1112252130 >> 172.21.15.16:0/1112252130 conn(0x7fe66c079370 msgr2=0x7fe66c079780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:02.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.051+0000 7fe672811700 1 -- 172.21.15.16:0/1112252130 shutdown_connections 2024-03-28T22:09:02.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.051+0000 7fe672811700 1 -- 172.21.15.16:0/1112252130 wait complete. 2024-03-28T22:09:02.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.051+0000 7fe672811700 1 Processor -- start 2024-03-28T22:09:02.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.051+0000 7fe672811700 1 -- start start 2024-03-28T22:09:02.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.052+0000 7fe672811700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe66c102290 0x7fe66c19e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:02.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.052+0000 7fe672811700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe66c102c50 0x7fe66c19ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:02.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.052+0000 7fe672811700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe66c108270 0x7fe66c198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:02.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.052+0000 7fe672811700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe66c112520 con 0x7fe66c108270 2024-03-28T22:09:02.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.052+0000 7fe672811700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe66c1123a0 con 0x7fe66c102290 2024-03-28T22:09:02.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.052+0000 7fe672811700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe66c1126a0 con 0x7fe66c102c50 2024-03-28T22:09:02.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.052+0000 7fe670dae700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe66c108270 0x7fe66c198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:02.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.052+0000 7fe670dae700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe66c108270 0x7fe66c198860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33360/0 (socket says 172.21.15.16:33360) 2024-03-28T22:09:02.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.052+0000 7fe66bfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe66c102290 0x7fe66c19e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:02.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.052+0000 7fe670dae700 1 -- 172.21.15.16:0/1460131730 learned_addr learned my addr 172.21.15.16:0/1460131730 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:02.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.052+0000 7fe66b7fe700 1 --2- 172.21.15.16:0/1460131730 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe66c102c50 0x7fe66c19ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:02.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.052+0000 7fe670dae700 1 -- 172.21.15.16:0/1460131730 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe66c102c50 msgr2=0x7fe66c19ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:02.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.052+0000 7fe670dae700 1 --2- 172.21.15.16:0/1460131730 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe66c102c50 0x7fe66c19ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:02.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.052+0000 7fe670dae700 1 -- 172.21.15.16:0/1460131730 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe66c102290 msgr2=0x7fe66c19e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:02.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.052+0000 7fe670dae700 1 --2- 172.21.15.16:0/1460131730 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe66c102290 0x7fe66c19e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:02.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.052+0000 7fe670dae700 1 -- 172.21.15.16:0/1460131730 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe66c199040 con 0x7fe66c108270 2024-03-28T22:09:02.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.052+0000 7fe66b7fe700 1 --2- 172.21.15.16:0/1460131730 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe66c102c50 0x7fe66c19ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:09:02.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.052+0000 7fe66bfff700 1 --2- 172.21.15.16:0/1460131730 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe66c102290 0x7fe66c19e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:09:02.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.052+0000 7fe670dae700 1 --2- 172.21.15.16:0/1460131730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe66c108270 0x7fe66c198860 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe664006070 tx=0x7fe664007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:02.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.053+0000 7fe6697fa700 1 -- 172.21.15.16:0/1460131730 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe664003c00 con 0x7fe66c108270 2024-03-28T22:09:02.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.053+0000 7fe6697fa700 1 -- 172.21.15.16:0/1460131730 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe664022070 con 0x7fe66c108270 2024-03-28T22:09:02.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.053+0000 7fe672811700 1 -- 172.21.15.16:0/1460131730 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe66c1992d0 con 0x7fe66c108270 2024-03-28T22:09:02.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.053+0000 7fe6697fa700 1 -- 172.21.15.16:0/1460131730 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe66401b6c0 con 0x7fe66c108270 2024-03-28T22:09:02.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.053+0000 7fe672811700 1 -- 172.21.15.16:0/1460131730 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe66c1a54f0 con 0x7fe66c108270 2024-03-28T22:09:02.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.054+0000 7fe672811700 1 -- 172.21.15.16:0/1460131730 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe66c062910 con 0x7fe66c108270 2024-03-28T22:09:02.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.055+0000 7fe6697fa700 1 -- 172.21.15.16:0/1460131730 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fe66401d470 con 0x7fe66c108270 2024-03-28T22:09:02.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.055+0000 7fe6697fa700 1 --2- 172.21.15.16:0/1460131730 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fe6540a1f20 0x7fe6540a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:02.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.056+0000 7fe66bfff700 1 --2- 172.21.15.16:0/1460131730 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fe6540a1f20 0x7fe6540a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:02.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.056+0000 7fe6697fa700 1 -- 172.21.15.16:0/1460131730 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fe66400f020 con 0x7fe66c108270 2024-03-28T22:09:02.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.056+0000 7fe6697fa700 1 --2- 172.21.15.16:0/1460131730 >> [v2:172.21.15.53:6808/772161849,v1:172.21.15.53:6809/772161849] conn(0x7fe6540ab700 0x7fe6540adb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:02.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.056+0000 7fe6697fa700 1 -- 172.21.15.16:0/1460131730 --> [v2:172.21.15.53:6808/772161849,v1:172.21.15.53:6809/772161849] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe66400ce90 con 0x7fe6540ab700 2024-03-28T22:09:02.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.056+0000 7fe6697fa700 1 -- 172.21.15.16:0/1460131730 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7fe6640cf050 con 0x7fe66c108270 2024-03-28T22:09:02.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.056+0000 7fe66bfff700 1 --2- 172.21.15.16:0/1460131730 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fe6540a1f20 0x7fe6540a43e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe65c009680 tx=0x7fe65c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:02.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.056+0000 7fe66b7fe700 1 --2- 172.21.15.16:0/1460131730 >> [v2:172.21.15.53:6808/772161849,v1:172.21.15.53:6809/772161849] conn(0x7fe6540ab700 0x7fe6540adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:02.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.056+0000 7fe66b7fe700 1 --2- 172.21.15.16:0/1460131730 >> [v2:172.21.15.53:6808/772161849,v1:172.21.15.53:6809/772161849] conn(0x7fe6540ab700 0x7fe6540adb20 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:02.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.057+0000 7fe6697fa700 1 -- 172.21.15.16:0/1460131730 <== osd.4 v2:172.21.15.53:6808/772161849 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe66400ce90 con 0x7fe6540ab700 2024-03-28T22:09:02.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.074+0000 7fe672811700 1 -- 172.21.15.16:0/1460131730 --> [v2:172.21.15.53:6808/772161849,v1:172.21.15.53:6809/772161849] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe66c062d40 con 0x7fe6540ab700 2024-03-28T22:09:02.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.074+0000 7fe6697fa700 1 -- 172.21.15.16:0/1460131730 <== osd.4 v2:172.21.15.53:6808/772161849 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fe66c062d40 con 0x7fe6540ab700 2024-03-28T22:09:02.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.074+0000 7fe672811700 1 -- 172.21.15.16:0/1460131730 >> [v2:172.21.15.53:6808/772161849,v1:172.21.15.53:6809/772161849] conn(0x7fe6540ab700 msgr2=0x7fe6540adb20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:02.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.074+0000 7fe672811700 1 --2- 172.21.15.16:0/1460131730 >> [v2:172.21.15.53:6808/772161849,v1:172.21.15.53:6809/772161849] conn(0x7fe6540ab700 0x7fe6540adb20 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:02.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.074+0000 7fe672811700 1 -- 172.21.15.16:0/1460131730 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fe6540a1f20 msgr2=0x7fe6540a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:02.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.074+0000 7fe672811700 1 --2- 172.21.15.16:0/1460131730 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fe6540a1f20 0x7fe6540a43e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe65c009680 tx=0x7fe65c009260 comp rx=0 tx=0).stop 2024-03-28T22:09:02.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.074+0000 7fe672811700 1 -- 172.21.15.16:0/1460131730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe66c108270 msgr2=0x7fe66c198860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:02.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.074+0000 7fe672811700 1 --2- 172.21.15.16:0/1460131730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe66c108270 0x7fe66c198860 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe664006070 tx=0x7fe664007a20 comp rx=0 tx=0).stop 2024-03-28T22:09:02.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.074+0000 7fe670dae700 1 -- 172.21.15.16:0/1460131730 reap_dead start 2024-03-28T22:09:02.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.075+0000 7fe672811700 1 -- 172.21.15.16:0/1460131730 shutdown_connections 2024-03-28T22:09:02.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.075+0000 7fe672811700 1 -- 172.21.15.16:0/1460131730 >> 172.21.15.16:0/1460131730 conn(0x7fe66c079370 msgr2=0x7fe66c100ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:02.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.075+0000 7fe672811700 1 -- 172.21.15.16:0/1460131730 shutdown_connections 2024-03-28T22:09:02.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:02.075+0000 7fe672811700 1 -- 172.21.15.16:0/1460131730 wait complete. 2024-03-28T22:09:03.426 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:03 smithi016 ceph-mon[32374]: pgmap v303: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-28T22:09:03.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:03 smithi063 ceph-mon[36249]: pgmap v303: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-28T22:09:03.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:03 smithi053 ceph-mon[35979]: pgmap v303: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-28T22:09:03.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.908+0000 7fedaa7b9700 1 -- 172.21.15.16:0/1047010007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feda4102290 msgr2=0x7feda4102710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:03.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.908+0000 7fedaa7b9700 1 --2- 172.21.15.16:0/1047010007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feda4102290 0x7feda4102710 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fed8c0099c0 tx=0x7fed8c009cd0 comp rx=0 tx=0).stop 2024-03-28T22:09:03.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.908+0000 7fedaa7b9700 1 -- 172.21.15.16:0/1047010007 shutdown_connections 2024-03-28T22:09:03.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.908+0000 7fedaa7b9700 1 --2- 172.21.15.16:0/1047010007 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feda4102c50 0x7feda410f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:03.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.908+0000 7fedaa7b9700 1 --2- 172.21.15.16:0/1047010007 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feda4108270 0x7feda4108650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:03.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.908+0000 7fedaa7b9700 1 --2- 172.21.15.16:0/1047010007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feda4102290 0x7feda4102710 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:03.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.908+0000 7fedaa7b9700 1 -- 172.21.15.16:0/1047010007 >> 172.21.15.16:0/1047010007 conn(0x7feda4079370 msgr2=0x7feda4079780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:03.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.908+0000 7fedaa7b9700 1 -- 172.21.15.16:0/1047010007 shutdown_connections 2024-03-28T22:09:03.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.908+0000 7fedaa7b9700 1 -- 172.21.15.16:0/1047010007 wait complete. 2024-03-28T22:09:03.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.909+0000 7fedaa7b9700 1 Processor -- start 2024-03-28T22:09:03.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.909+0000 7fedaa7b9700 1 -- start start 2024-03-28T22:09:03.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.909+0000 7fedaa7b9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feda4102290 0x7feda41a07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:03.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.909+0000 7fedaa7b9700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feda4102c50 0x7feda41a0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:03.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.909+0000 7fedaa7b9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feda4108270 0x7feda419a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:03.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.909+0000 7fedaa7b9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7feda41124f0 con 0x7feda4108270 2024-03-28T22:09:03.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.909+0000 7fedaa7b9700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7feda4112370 con 0x7feda4102290 2024-03-28T22:09:03.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.909+0000 7fedaa7b9700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7feda4112670 con 0x7feda4102c50 2024-03-28T22:09:03.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.909+0000 7feda8d56700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feda4108270 0x7feda419a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:03.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.909+0000 7feda8d56700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feda4108270 0x7feda419a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33370/0 (socket says 172.21.15.16:33370) 2024-03-28T22:09:03.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.909+0000 7feda3fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feda4102290 0x7feda41a07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:03.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.909+0000 7feda8d56700 1 -- 172.21.15.16:0/3989573235 learned_addr learned my addr 172.21.15.16:0/3989573235 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:03.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.909+0000 7feda37fe700 1 --2- 172.21.15.16:0/3989573235 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feda4102c50 0x7feda41a0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:03.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.910+0000 7feda3fff700 1 -- 172.21.15.16:0/3989573235 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feda4102c50 msgr2=0x7feda41a0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:03.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.910+0000 7feda3fff700 1 --2- 172.21.15.16:0/3989573235 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feda4102c50 0x7feda41a0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:03.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.910+0000 7feda3fff700 1 -- 172.21.15.16:0/3989573235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feda4108270 msgr2=0x7feda419a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:03.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.910+0000 7feda3fff700 1 --2- 172.21.15.16:0/3989573235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feda4108270 0x7feda419a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:03.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.910+0000 7feda3fff700 1 -- 172.21.15.16:0/3989573235 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feda419b0e0 con 0x7feda4102290 2024-03-28T22:09:03.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.910+0000 7feda8d56700 1 --2- 172.21.15.16:0/3989573235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feda4108270 0x7feda419a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:09:03.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.910+0000 7feda3fff700 1 --2- 172.21.15.16:0/3989573235 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feda4102290 0x7feda41a07c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fed9400e9f0 tx=0x7fed9400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:03.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.910+0000 7feda17fa700 1 -- 172.21.15.16:0/3989573235 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed9400c9c0 con 0x7feda4102290 2024-03-28T22:09:03.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.910+0000 7fedaa7b9700 1 -- 172.21.15.16:0/3989573235 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feda419b3d0 con 0x7feda4102290 2024-03-28T22:09:03.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.910+0000 7feda17fa700 1 -- 172.21.15.16:0/3989573235 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed9400cb60 con 0x7feda4102290 2024-03-28T22:09:03.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.910+0000 7feda17fa700 1 -- 172.21.15.16:0/3989573235 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed94018680 con 0x7feda4102290 2024-03-28T22:09:03.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.910+0000 7fedaa7b9700 1 -- 172.21.15.16:0/3989573235 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feda41a7690 con 0x7feda4102290 2024-03-28T22:09:03.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.911+0000 7fedaa7b9700 1 -- 172.21.15.16:0/3989573235 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7feda410edf0 con 0x7feda4102290 2024-03-28T22:09:03.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.913+0000 7feda17fa700 1 -- 172.21.15.16:0/3989573235 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fed94010430 con 0x7feda4102290 2024-03-28T22:09:03.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.913+0000 7feda17fa700 1 --2- 172.21.15.16:0/3989573235 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fed900a1ea0 0x7fed900a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:03.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.913+0000 7feda37fe700 1 --2- 172.21.15.16:0/3989573235 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fed900a1ea0 0x7fed900a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:03.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.913+0000 7feda17fa700 1 -- 172.21.15.16:0/3989573235 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fed940ce470 con 0x7feda4102290 2024-03-28T22:09:03.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.913+0000 7feda17fa700 1 --2- 172.21.15.16:0/3989573235 >> [v2:172.21.15.63:6800/2214131086,v1:172.21.15.63:6801/2214131086] conn(0x7fed900ab680 0x7fed900adaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:03.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.913+0000 7feda17fa700 1 -- 172.21.15.16:0/3989573235 --> [v2:172.21.15.63:6800/2214131086,v1:172.21.15.63:6801/2214131086] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fed9401bec0 con 0x7fed900ab680 2024-03-28T22:09:03.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.913+0000 7feda17fa700 1 -- 172.21.15.16:0/3989573235 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7fed940ce820 con 0x7feda4102290 2024-03-28T22:09:03.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.914+0000 7feda37fe700 1 --2- 172.21.15.16:0/3989573235 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fed900a1ea0 0x7fed900a4360 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7feda40765b0 tx=0x7fed8c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:03.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.914+0000 7feda8d56700 1 --2- 172.21.15.16:0/3989573235 >> [v2:172.21.15.63:6800/2214131086,v1:172.21.15.63:6801/2214131086] conn(0x7fed900ab680 0x7fed900adaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:03.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.914+0000 7feda8d56700 1 --2- 172.21.15.16:0/3989573235 >> [v2:172.21.15.63:6800/2214131086,v1:172.21.15.63:6801/2214131086] conn(0x7fed900ab680 0x7fed900adaa0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:03.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.915+0000 7feda17fa700 1 -- 172.21.15.16:0/3989573235 <== osd.2 v2:172.21.15.63:6800/2214131086 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fed9401bec0 con 0x7fed900ab680 2024-03-28T22:09:03.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.932+0000 7fedaa7b9700 1 -- 172.21.15.16:0/3989573235 --> [v2:172.21.15.63:6800/2214131086,v1:172.21.15.63:6801/2214131086] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7feda4103920 con 0x7fed900ab680 2024-03-28T22:09:03.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.932+0000 7feda17fa700 1 -- 172.21.15.16:0/3989573235 <== osd.2 v2:172.21.15.63:6800/2214131086 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7feda4103920 con 0x7fed900ab680 2024-03-28T22:09:03.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.933+0000 7fedaa7b9700 1 -- 172.21.15.16:0/3989573235 >> [v2:172.21.15.63:6800/2214131086,v1:172.21.15.63:6801/2214131086] conn(0x7fed900ab680 msgr2=0x7fed900adaa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:03.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.933+0000 7fedaa7b9700 1 --2- 172.21.15.16:0/3989573235 >> [v2:172.21.15.63:6800/2214131086,v1:172.21.15.63:6801/2214131086] conn(0x7fed900ab680 0x7fed900adaa0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:03.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.933+0000 7fedaa7b9700 1 -- 172.21.15.16:0/3989573235 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fed900a1ea0 msgr2=0x7fed900a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:03.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.933+0000 7fedaa7b9700 1 --2- 172.21.15.16:0/3989573235 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fed900a1ea0 0x7fed900a4360 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7feda40765b0 tx=0x7fed8c005fb0 comp rx=0 tx=0).stop 2024-03-28T22:09:03.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.933+0000 7fedaa7b9700 1 -- 172.21.15.16:0/3989573235 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feda4102290 msgr2=0x7feda41a07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:03.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.933+0000 7fedaa7b9700 1 --2- 172.21.15.16:0/3989573235 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feda4102290 0x7feda41a07c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fed9400e9f0 tx=0x7fed9400ed00 comp rx=0 tx=0).stop 2024-03-28T22:09:03.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.933+0000 7feda8d56700 1 -- 172.21.15.16:0/3989573235 reap_dead start 2024-03-28T22:09:03.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.933+0000 7fedaa7b9700 1 -- 172.21.15.16:0/3989573235 shutdown_connections 2024-03-28T22:09:03.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.933+0000 7fedaa7b9700 1 -- 172.21.15.16:0/3989573235 >> 172.21.15.16:0/3989573235 conn(0x7feda4079370 msgr2=0x7feda410d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:03.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.933+0000 7fedaa7b9700 1 -- 172.21.15.16:0/3989573235 shutdown_connections 2024-03-28T22:09:03.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:03.933+0000 7fedaa7b9700 1 -- 172.21.15.16:0/3989573235 wait complete. 2024-03-28T22:09:04.573 INFO:teuthology.orchestra.run.smithi016.stdout:257698037793 2024-03-28T22:09:04.573 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-28T22:09:04.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.435+0000 7fc36a7d6700 1 -- 172.21.15.16:0/2238059195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc364101140 msgr2=0x7fc3641015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:04.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.435+0000 7fc36a7d6700 1 --2- 172.21.15.16:0/2238059195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc364101140 0x7fc3641015a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc358009990 tx=0x7fc358009ca0 comp rx=0 tx=0).stop 2024-03-28T22:09:04.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.435+0000 7fc36a7d6700 1 -- 172.21.15.16:0/2238059195 shutdown_connections 2024-03-28T22:09:04.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.435+0000 7fc36a7d6700 1 --2- 172.21.15.16:0/2238059195 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc364101ae0 0x7fc36410dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:04.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.435+0000 7fc36a7d6700 1 --2- 172.21.15.16:0/2238059195 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc364107180 0x7fc364107560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:04.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.435+0000 7fc36a7d6700 1 --2- 172.21.15.16:0/2238059195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc364101140 0x7fc3641015a0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:04.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.435+0000 7fc36a7d6700 1 -- 172.21.15.16:0/2238059195 >> 172.21.15.16:0/2238059195 conn(0x7fc364078070 msgr2=0x7fc364078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:04.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.435+0000 7fc36a7d6700 1 -- 172.21.15.16:0/2238059195 shutdown_connections 2024-03-28T22:09:04.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.435+0000 7fc36a7d6700 1 -- 172.21.15.16:0/2238059195 wait complete. 2024-03-28T22:09:04.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.436+0000 7fc36a7d6700 1 Processor -- start 2024-03-28T22:09:04.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.436+0000 7fc36a7d6700 1 -- start start 2024-03-28T22:09:04.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.436+0000 7fc36a7d6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc364101140 0x7fc36419d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:04.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.436+0000 7fc36a7d6700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc364101ae0 0x7fc36419db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:04.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.436+0000 7fc36a7d6700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc364107180 0x7fc364197720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:04.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.436+0000 7fc36a7d6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc3641113e0 con 0x7fc364101140 2024-03-28T22:09:04.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.436+0000 7fc36a7d6700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc364111260 con 0x7fc364101ae0 2024-03-28T22:09:04.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.436+0000 7fc36a7d6700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc364111560 con 0x7fc364107180 2024-03-28T22:09:04.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.436+0000 7fc363fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc364101140 0x7fc36419d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:04.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.436+0000 7fc363fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc364101140 0x7fc36419d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33350/0 (socket says 172.21.15.16:33350) 2024-03-28T22:09:04.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.436+0000 7fc363fff700 1 -- 172.21.15.16:0/4000151586 learned_addr learned my addr 172.21.15.16:0/4000151586 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:04.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.436+0000 7fc3637fe700 1 --2- 172.21.15.16:0/4000151586 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc364101ae0 0x7fc36419db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:04.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.437+0000 7fc368d73700 1 --2- 172.21.15.16:0/4000151586 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc364107180 0x7fc364197720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:04.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.437+0000 7fc363fff700 1 -- 172.21.15.16:0/4000151586 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc364107180 msgr2=0x7fc364197720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:04.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.437+0000 7fc363fff700 1 --2- 172.21.15.16:0/4000151586 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc364107180 0x7fc364197720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:04.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.437+0000 7fc363fff700 1 -- 172.21.15.16:0/4000151586 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc364101ae0 msgr2=0x7fc36419db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:04.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.437+0000 7fc363fff700 1 --2- 172.21.15.16:0/4000151586 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc364101ae0 0x7fc36419db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:04.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.437+0000 7fc363fff700 1 -- 172.21.15.16:0/4000151586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc364198000 con 0x7fc364101140 2024-03-28T22:09:04.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.437+0000 7fc363fff700 1 --2- 172.21.15.16:0/4000151586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc364101140 0x7fc36419d600 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc35400d980 tx=0x7fc35400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:04.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.437+0000 7fc3617fa700 1 -- 172.21.15.16:0/4000151586 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc354017070 con 0x7fc364101140 2024-03-28T22:09:04.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.437+0000 7fc3617fa700 1 -- 172.21.15.16:0/4000151586 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc354005020 con 0x7fc364101140 2024-03-28T22:09:04.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.437+0000 7fc36a7d6700 1 -- 172.21.15.16:0/4000151586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3641982f0 con 0x7fc364101140 2024-03-28T22:09:04.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.437+0000 7fc3617fa700 1 -- 172.21.15.16:0/4000151586 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc354009bc0 con 0x7fc364101140 2024-03-28T22:09:04.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.437+0000 7fc36a7d6700 1 -- 172.21.15.16:0/4000151586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3641a4580 con 0x7fc364101140 2024-03-28T22:09:04.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.438+0000 7fc36a7d6700 1 -- 172.21.15.16:0/4000151586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc3640629d0 con 0x7fc364101140 2024-03-28T22:09:04.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.439+0000 7fc3617fa700 1 -- 172.21.15.16:0/4000151586 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fc3540051d0 con 0x7fc364101140 2024-03-28T22:09:04.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.440+0000 7fc3617fa700 1 --2- 172.21.15.16:0/4000151586 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc34c0a1f20 0x7fc34c0a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:04.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.440+0000 7fc3637fe700 1 --2- 172.21.15.16:0/4000151586 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc34c0a1f20 0x7fc34c0a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:04.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.440+0000 7fc3617fa700 1 -- 172.21.15.16:0/4000151586 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fc3540ce1a0 con 0x7fc364101140 2024-03-28T22:09:04.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.440+0000 7fc3617fa700 1 --2- 172.21.15.16:0/4000151586 >> [v2:172.21.15.63:6808/1602554852,v1:172.21.15.63:6809/1602554852] conn(0x7fc34c0ab700 0x7fc34c0adb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:04.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.440+0000 7fc3617fa700 1 -- 172.21.15.16:0/4000151586 --> [v2:172.21.15.63:6808/1602554852,v1:172.21.15.63:6809/1602554852] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc35401de70 con 0x7fc34c0ab700 2024-03-28T22:09:04.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.440+0000 7fc3617fa700 1 -- 172.21.15.16:0/4000151586 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7fc3540ce680 con 0x7fc364101140 2024-03-28T22:09:04.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.440+0000 7fc3637fe700 1 --2- 172.21.15.16:0/4000151586 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc34c0a1f20 0x7fc34c0a43e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc35800b2e0 tx=0x7fc358005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:04.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.440+0000 7fc368d73700 1 --2- 172.21.15.16:0/4000151586 >> [v2:172.21.15.63:6808/1602554852,v1:172.21.15.63:6809/1602554852] conn(0x7fc34c0ab700 0x7fc34c0adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:04.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.441+0000 7fc368d73700 1 --2- 172.21.15.16:0/4000151586 >> [v2:172.21.15.63:6808/1602554852,v1:172.21.15.63:6809/1602554852] conn(0x7fc34c0ab700 0x7fc34c0adb20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:04.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.442+0000 7fc3617fa700 1 -- 172.21.15.16:0/4000151586 <== osd.5 v2:172.21.15.63:6808/1602554852 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc35401de70 con 0x7fc34c0ab700 2024-03-28T22:09:04.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.458+0000 7fc36a7d6700 1 -- 172.21.15.16:0/4000151586 --> [v2:172.21.15.63:6808/1602554852,v1:172.21.15.63:6809/1602554852] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc364062e50 con 0x7fc34c0ab700 2024-03-28T22:09:04.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.459+0000 7fc3617fa700 1 -- 172.21.15.16:0/4000151586 <== osd.5 v2:172.21.15.63:6808/1602554852 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fc364062e50 con 0x7fc34c0ab700 2024-03-28T22:09:04.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.459+0000 7fc36a7d6700 1 -- 172.21.15.16:0/4000151586 >> [v2:172.21.15.63:6808/1602554852,v1:172.21.15.63:6809/1602554852] conn(0x7fc34c0ab700 msgr2=0x7fc34c0adb20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:04.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.459+0000 7fc36a7d6700 1 --2- 172.21.15.16:0/4000151586 >> [v2:172.21.15.63:6808/1602554852,v1:172.21.15.63:6809/1602554852] conn(0x7fc34c0ab700 0x7fc34c0adb20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:04.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.459+0000 7fc36a7d6700 1 -- 172.21.15.16:0/4000151586 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc34c0a1f20 msgr2=0x7fc34c0a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:04.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.459+0000 7fc36a7d6700 1 --2- 172.21.15.16:0/4000151586 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc34c0a1f20 0x7fc34c0a43e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc35800b2e0 tx=0x7fc358005960 comp rx=0 tx=0).stop 2024-03-28T22:09:04.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.459+0000 7fc36a7d6700 1 -- 172.21.15.16:0/4000151586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc364101140 msgr2=0x7fc36419d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:04.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.459+0000 7fc36a7d6700 1 --2- 172.21.15.16:0/4000151586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc364101140 0x7fc36419d600 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc35400d980 tx=0x7fc35400dc90 comp rx=0 tx=0).stop 2024-03-28T22:09:04.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.459+0000 7fc368d73700 1 -- 172.21.15.16:0/4000151586 reap_dead start 2024-03-28T22:09:04.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.459+0000 7fc36a7d6700 1 -- 172.21.15.16:0/4000151586 shutdown_connections 2024-03-28T22:09:04.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.459+0000 7fc36a7d6700 1 -- 172.21.15.16:0/4000151586 >> 172.21.15.16:0/4000151586 conn(0x7fc364078070 msgr2=0x7fc3640ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:04.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.459+0000 7fc36a7d6700 1 -- 172.21.15.16:0/4000151586 shutdown_connections 2024-03-28T22:09:04.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:01.459+0000 7fc36a7d6700 1 -- 172.21.15.16:0/4000151586 wait complete. 2024-03-28T22:09:05.227 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:05 smithi016 ceph-mon[32374]: pgmap v304: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-28T22:09:05.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:05 smithi063 ceph-mon[36249]: pgmap v304: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-28T22:09:05.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:05 smithi053 ceph-mon[35979]: pgmap v304: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-28T22:09:05.874 INFO:teuthology.orchestra.run.smithi016.stdout:167503724594 2024-03-28T22:09:05.874 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-28T22:09:06.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.155+0000 7f6621dc6700 1 -- 172.21.15.16:0/1844145699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f661c069ef0 msgr2=0x7f661c06a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:06.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.155+0000 7f6621dc6700 1 --2- 172.21.15.16:0/1844145699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f661c069ef0 0x7f661c06a350 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f66100099c0 tx=0x7f6610009cd0 comp rx=0 tx=0).stop 2024-03-28T22:09:06.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.155+0000 7f6621dc6700 1 -- 172.21.15.16:0/1844145699 shutdown_connections 2024-03-28T22:09:06.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.155+0000 7f6621dc6700 1 --2- 172.21.15.16:0/1844145699 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f661c06a890 0x7f661c10f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:06.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.155+0000 7f6621dc6700 1 --2- 172.21.15.16:0/1844145699 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f661c108290 0x7f661c108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:06.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.155+0000 7f6621dc6700 1 --2- 172.21.15.16:0/1844145699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f661c069ef0 0x7f661c06a350 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:06.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.155+0000 7f6621dc6700 1 -- 172.21.15.16:0/1844145699 >> 172.21.15.16:0/1844145699 conn(0x7f661c078070 msgr2=0x7f661c078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:06.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.155+0000 7f6621dc6700 1 -- 172.21.15.16:0/1844145699 shutdown_connections 2024-03-28T22:09:06.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.155+0000 7f6621dc6700 1 -- 172.21.15.16:0/1844145699 wait complete. 2024-03-28T22:09:06.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.156+0000 7f6621dc6700 1 Processor -- start 2024-03-28T22:09:06.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.156+0000 7f6621dc6700 1 -- start start 2024-03-28T22:09:06.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.156+0000 7f6621dc6700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f661c069ef0 0x7f661c19e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:06.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.156+0000 7f6621dc6700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f661c06a890 0x7f661c19ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:06.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.156+0000 7f6621dc6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f661c108290 0x7f661c198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:06.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.156+0000 7f6621dc6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f661c112510 con 0x7f661c108290 2024-03-28T22:09:06.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.156+0000 7f6621dc6700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f661c112390 con 0x7f661c069ef0 2024-03-28T22:09:06.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.156+0000 7f6621dc6700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f661c112690 con 0x7f661c06a890 2024-03-28T22:09:06.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.156+0000 7f661bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f661c108290 0x7f661c198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:06.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.156+0000 7f661bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f661c108290 0x7f661c198870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33382/0 (socket says 172.21.15.16:33382) 2024-03-28T22:09:06.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.156+0000 7f661affd700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f661c06a890 0x7f661c19ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:06.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.156+0000 7f661bfff700 1 -- 172.21.15.16:0/3245763787 learned_addr learned my addr 172.21.15.16:0/3245763787 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:06.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.156+0000 7f661b7fe700 1 --2- 172.21.15.16:0/3245763787 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f661c069ef0 0x7f661c19e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:06.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.157+0000 7f661bfff700 1 -- 172.21.15.16:0/3245763787 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f661c06a890 msgr2=0x7f661c19ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:06.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.157+0000 7f661bfff700 1 --2- 172.21.15.16:0/3245763787 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f661c06a890 0x7f661c19ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:06.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.157+0000 7f661bfff700 1 -- 172.21.15.16:0/3245763787 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f661c069ef0 msgr2=0x7f661c19e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:06.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.157+0000 7f661bfff700 1 --2- 172.21.15.16:0/3245763787 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f661c069ef0 0x7f661c19e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:06.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.157+0000 7f661bfff700 1 -- 172.21.15.16:0/3245763787 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f661c199050 con 0x7f661c108290 2024-03-28T22:09:06.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.157+0000 7f661bfff700 1 --2- 172.21.15.16:0/3245763787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f661c108290 0x7f661c198870 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f6614007ee0 tx=0x7f661400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:06.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.157+0000 7f6618ff9700 1 -- 172.21.15.16:0/3245763787 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6614004bb0 con 0x7f661c108290 2024-03-28T22:09:06.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.157+0000 7f6621dc6700 1 -- 172.21.15.16:0/3245763787 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f661c199340 con 0x7f661c108290 2024-03-28T22:09:06.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.157+0000 7f6618ff9700 1 -- 172.21.15.16:0/3245763787 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6614004d50 con 0x7f661c108290 2024-03-28T22:09:06.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.157+0000 7f6618ff9700 1 -- 172.21.15.16:0/3245763787 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6614005960 con 0x7f661c108290 2024-03-28T22:09:06.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.157+0000 7f6621dc6700 1 -- 172.21.15.16:0/3245763787 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f661c1a5500 con 0x7f661c108290 2024-03-28T22:09:06.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.158+0000 7f6621dc6700 1 -- 172.21.15.16:0/3245763787 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f661c062950 con 0x7f661c108290 2024-03-28T22:09:06.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.159+0000 7f6618ff9700 1 -- 172.21.15.16:0/3245763787 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f661401a020 con 0x7f661c108290 2024-03-28T22:09:06.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.160+0000 7f6618ff9700 1 --2- 172.21.15.16:0/3245763787 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f66040a1ea0 0x7f66040a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:06.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.160+0000 7f661b7fe700 1 --2- 172.21.15.16:0/3245763787 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f66040a1ea0 0x7f66040a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:06.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.160+0000 7f6618ff9700 1 -- 172.21.15.16:0/3245763787 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f66140cdc30 con 0x7f661c108290 2024-03-28T22:09:06.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.160+0000 7f6618ff9700 1 --2- 172.21.15.16:0/3245763787 >> [v2:172.21.15.16:6826/1585624367,v1:172.21.15.16:6827/1585624367] conn(0x7f66040ab680 0x7f66040adaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:06.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.160+0000 7f6618ff9700 1 -- 172.21.15.16:0/3245763787 --> [v2:172.21.15.16:6826/1585624367,v1:172.21.15.16:6827/1585624367] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f66040ae150 con 0x7f66040ab680 2024-03-28T22:09:06.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.160+0000 7f6618ff9700 1 -- 172.21.15.16:0/3245763787 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f66140cdfe0 con 0x7f661c108290 2024-03-28T22:09:06.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.160+0000 7f661affd700 1 --2- 172.21.15.16:0/3245763787 >> [v2:172.21.15.16:6826/1585624367,v1:172.21.15.16:6827/1585624367] conn(0x7f66040ab680 0x7f66040adaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:06.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.160+0000 7f661b7fe700 1 --2- 172.21.15.16:0/3245763787 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f66040a1ea0 0x7f66040a4360 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f660c007b60 tx=0x7f660c0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:06.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.160+0000 7f661affd700 1 --2- 172.21.15.16:0/3245763787 >> [v2:172.21.15.16:6826/1585624367,v1:172.21.15.16:6827/1585624367] conn(0x7f66040ab680 0x7f66040adaa0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:06.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.161+0000 7f6618ff9700 1 -- 172.21.15.16:0/3245763787 <== osd.9 v2:172.21.15.16:6826/1585624367 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f66040ae150 con 0x7f66040ab680 2024-03-28T22:09:06.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.177+0000 7f6621dc6700 1 -- 172.21.15.16:0/3245763787 --> [v2:172.21.15.16:6826/1585624367,v1:172.21.15.16:6827/1585624367] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f661c062dd0 con 0x7f66040ab680 2024-03-28T22:09:06.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.178+0000 7f6618ff9700 1 -- 172.21.15.16:0/3245763787 <== osd.9 v2:172.21.15.16:6826/1585624367 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f661c062dd0 con 0x7f66040ab680 2024-03-28T22:09:06.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.178+0000 7f6621dc6700 1 -- 172.21.15.16:0/3245763787 >> [v2:172.21.15.16:6826/1585624367,v1:172.21.15.16:6827/1585624367] conn(0x7f66040ab680 msgr2=0x7f66040adaa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:06.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.178+0000 7f6621dc6700 1 --2- 172.21.15.16:0/3245763787 >> [v2:172.21.15.16:6826/1585624367,v1:172.21.15.16:6827/1585624367] conn(0x7f66040ab680 0x7f66040adaa0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:06.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.178+0000 7f6621dc6700 1 -- 172.21.15.16:0/3245763787 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f66040a1ea0 msgr2=0x7f66040a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:06.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.178+0000 7f6621dc6700 1 --2- 172.21.15.16:0/3245763787 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f66040a1ea0 0x7f66040a4360 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f660c007b60 tx=0x7f660c0073d0 comp rx=0 tx=0).stop 2024-03-28T22:09:06.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.178+0000 7f6621dc6700 1 -- 172.21.15.16:0/3245763787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f661c108290 msgr2=0x7f661c198870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:06.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.178+0000 7f6621dc6700 1 --2- 172.21.15.16:0/3245763787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f661c108290 0x7f661c198870 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f6614007ee0 tx=0x7f661400d3b0 comp rx=0 tx=0).stop 2024-03-28T22:09:06.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.178+0000 7f661bfff700 1 -- 172.21.15.16:0/3245763787 reap_dead start 2024-03-28T22:09:06.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.178+0000 7f6621dc6700 1 -- 172.21.15.16:0/3245763787 shutdown_connections 2024-03-28T22:09:06.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.178+0000 7f6621dc6700 1 -- 172.21.15.16:0/3245763787 >> 172.21.15.16:0/3245763787 conn(0x7f661c078070 msgr2=0x7f661c0ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:06.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.178+0000 7f6621dc6700 1 -- 172.21.15.16:0/3245763787 shutdown_connections 2024-03-28T22:09:06.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.178+0000 7f6621dc6700 1 -- 172.21.15.16:0/3245763787 wait complete. 2024-03-28T22:09:06.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.875+0000 7f7fe3e46700 1 -- 172.21.15.16:0/2937210594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fdc102c80 msgr2=0x7f7fdc10f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:06.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.875+0000 7f7fe3e46700 1 --2- 172.21.15.16:0/2937210594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fdc102c80 0x7f7fdc10f0e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7fd400b0a0 tx=0x7f7fd400b3b0 comp rx=0 tx=0).stop 2024-03-28T22:09:06.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.876+0000 7f7fe3e46700 1 -- 172.21.15.16:0/2937210594 shutdown_connections 2024-03-28T22:09:06.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.876+0000 7f7fe3e46700 1 --2- 172.21.15.16:0/2937210594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fdc102c80 0x7f7fdc10f0e0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:06.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.876+0000 7f7fe3e46700 1 --2- 172.21.15.16:0/2937210594 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fdc108310 0x7f7fdc1086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:06.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.876+0000 7f7fe3e46700 1 --2- 172.21.15.16:0/2937210594 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7fdc1022c0 0x7f7fdc102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:06.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.876+0000 7f7fe3e46700 1 -- 172.21.15.16:0/2937210594 >> 172.21.15.16:0/2937210594 conn(0x7f7fdc079300 msgr2=0x7f7fdc079710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:06.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.876+0000 7f7fe3e46700 1 -- 172.21.15.16:0/2937210594 shutdown_connections 2024-03-28T22:09:06.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.876+0000 7f7fe3e46700 1 -- 172.21.15.16:0/2937210594 wait complete. 2024-03-28T22:09:06.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.876+0000 7f7fe3e46700 1 Processor -- start 2024-03-28T22:09:06.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.876+0000 7f7fe3e46700 1 -- start start 2024-03-28T22:09:06.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.876+0000 7f7fe3e46700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fdc1022c0 0x7f7fdc19e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:06.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.877+0000 7f7fe3e46700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fdc102c80 0x7f7fdc19eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:06.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.877+0000 7f7fe3e46700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7fdc108310 0x7f7fdc1987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:06.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.877+0000 7f7fe3e46700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7fdc112510 con 0x7f7fdc102c80 2024-03-28T22:09:06.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.877+0000 7f7fe3e46700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7fdc112390 con 0x7f7fdc1022c0 2024-03-28T22:09:06.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.877+0000 7f7fe3e46700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7fdc112690 con 0x7f7fdc108310 2024-03-28T22:09:06.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.877+0000 7f7fe13e1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fdc102c80 0x7f7fdc19eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:06.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.877+0000 7f7fe13e1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fdc102c80 0x7f7fdc19eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33404/0 (socket says 172.21.15.16:33404) 2024-03-28T22:09:06.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.877+0000 7f7fe13e1700 1 -- 172.21.15.16:0/1407805292 learned_addr learned my addr 172.21.15.16:0/1407805292 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:06.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.877+0000 7f7fe1be2700 1 --2- 172.21.15.16:0/1407805292 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fdc1022c0 0x7f7fdc19e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:06.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.877+0000 7f7fe23e3700 1 --2- 172.21.15.16:0/1407805292 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7fdc108310 0x7f7fdc1987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:06.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.877+0000 7f7fe13e1700 1 -- 172.21.15.16:0/1407805292 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7fdc108310 msgr2=0x7f7fdc1987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:06.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.877+0000 7f7fe13e1700 1 --2- 172.21.15.16:0/1407805292 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7fdc108310 0x7f7fdc1987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:06.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.877+0000 7f7fe13e1700 1 -- 172.21.15.16:0/1407805292 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fdc1022c0 msgr2=0x7f7fdc19e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:06.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.877+0000 7f7fe13e1700 1 --2- 172.21.15.16:0/1407805292 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fdc1022c0 0x7f7fdc19e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:06.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.877+0000 7f7fe13e1700 1 -- 172.21.15.16:0/1407805292 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fdc198fb0 con 0x7f7fdc102c80 2024-03-28T22:09:06.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.877+0000 7f7fe13e1700 1 --2- 172.21.15.16:0/1407805292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fdc102c80 0x7f7fdc19eb90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7fd800d980 tx=0x7f7fd800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:06.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.877+0000 7f7fd2ffd700 1 -- 172.21.15.16:0/1407805292 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd8017070 con 0x7f7fdc102c80 2024-03-28T22:09:06.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.877+0000 7f7fd2ffd700 1 -- 172.21.15.16:0/1407805292 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fd8005020 con 0x7f7fdc102c80 2024-03-28T22:09:06.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.877+0000 7f7fe3e46700 1 -- 172.21.15.16:0/1407805292 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fdc1992a0 con 0x7f7fdc102c80 2024-03-28T22:09:06.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.877+0000 7f7fd2ffd700 1 -- 172.21.15.16:0/1407805292 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd800fbf0 con 0x7f7fdc102c80 2024-03-28T22:09:06.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.877+0000 7f7fe3e46700 1 -- 172.21.15.16:0/1407805292 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fdc1a54b0 con 0x7f7fdc102c80 2024-03-28T22:09:06.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.878+0000 7f7fe3e46700 1 -- 172.21.15.16:0/1407805292 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7fdc062950 con 0x7f7fdc102c80 2024-03-28T22:09:06.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.879+0000 7f7fd2ffd700 1 -- 172.21.15.16:0/1407805292 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f7fd80051d0 con 0x7f7fdc102c80 2024-03-28T22:09:06.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.880+0000 7f7fd2ffd700 1 --2- 172.21.15.16:0/1407805292 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f7fc80a1ea0 0x7f7fc80a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:06.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.880+0000 7f7fe1be2700 1 --2- 172.21.15.16:0/1407805292 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f7fc80a1ea0 0x7f7fc80a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:06.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.880+0000 7f7fd2ffd700 1 -- 172.21.15.16:0/1407805292 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f7fd80ce110 con 0x7f7fdc102c80 2024-03-28T22:09:06.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.880+0000 7f7fd2ffd700 1 --2- 172.21.15.16:0/1407805292 >> [v2:172.21.15.63:6824/2167866570,v1:172.21.15.63:6825/2167866570] conn(0x7f7fc80ab680 0x7f7fc80adaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:06.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.880+0000 7f7fd2ffd700 1 -- 172.21.15.16:0/1407805292 --> [v2:172.21.15.63:6824/2167866570,v1:172.21.15.63:6825/2167866570] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7fc80ae150 con 0x7f7fc80ab680 2024-03-28T22:09:06.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.880+0000 7f7fd2ffd700 1 -- 172.21.15.16:0/1407805292 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f7fd80ce5f0 con 0x7f7fdc102c80 2024-03-28T22:09:06.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.880+0000 7f7fe1be2700 1 --2- 172.21.15.16:0/1407805292 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f7fc80a1ea0 0x7f7fc80a4360 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc009be0 tx=0x7f7fcc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:06.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.880+0000 7f7fe23e3700 1 --2- 172.21.15.16:0/1407805292 >> [v2:172.21.15.63:6824/2167866570,v1:172.21.15.63:6825/2167866570] conn(0x7f7fc80ab680 0x7f7fc80adaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:06.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.881+0000 7f7fe23e3700 1 --2- 172.21.15.16:0/1407805292 >> [v2:172.21.15.63:6824/2167866570,v1:172.21.15.63:6825/2167866570] conn(0x7f7fc80ab680 0x7f7fc80adaa0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:06.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.882+0000 7f7fd2ffd700 1 -- 172.21.15.16:0/1407805292 <== osd.11 v2:172.21.15.63:6824/2167866570 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7fc80ae150 con 0x7f7fc80ab680 2024-03-28T22:09:06.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.899+0000 7f7fe3e46700 1 -- 172.21.15.16:0/1407805292 --> [v2:172.21.15.63:6824/2167866570,v1:172.21.15.63:6825/2167866570] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7fdc062dd0 con 0x7f7fc80ab680 2024-03-28T22:09:06.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.899+0000 7f7fd2ffd700 1 -- 172.21.15.16:0/1407805292 <== osd.11 v2:172.21.15.63:6824/2167866570 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7fdc062dd0 con 0x7f7fc80ab680 2024-03-28T22:09:06.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.900+0000 7f7fe3e46700 1 -- 172.21.15.16:0/1407805292 >> [v2:172.21.15.63:6824/2167866570,v1:172.21.15.63:6825/2167866570] conn(0x7f7fc80ab680 msgr2=0x7f7fc80adaa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:06.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.900+0000 7f7fe3e46700 1 --2- 172.21.15.16:0/1407805292 >> [v2:172.21.15.63:6824/2167866570,v1:172.21.15.63:6825/2167866570] conn(0x7f7fc80ab680 0x7f7fc80adaa0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:06.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.900+0000 7f7fe3e46700 1 -- 172.21.15.16:0/1407805292 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f7fc80a1ea0 msgr2=0x7f7fc80a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:06.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.900+0000 7f7fe3e46700 1 --2- 172.21.15.16:0/1407805292 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f7fc80a1ea0 0x7f7fc80a4360 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc009be0 tx=0x7f7fcc009260 comp rx=0 tx=0).stop 2024-03-28T22:09:06.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.900+0000 7f7fe3e46700 1 -- 172.21.15.16:0/1407805292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fdc102c80 msgr2=0x7f7fdc19eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:06.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.900+0000 7f7fe3e46700 1 --2- 172.21.15.16:0/1407805292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fdc102c80 0x7f7fdc19eb90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7fd800d980 tx=0x7f7fd800dc90 comp rx=0 tx=0).stop 2024-03-28T22:09:06.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.900+0000 7f7fe23e3700 1 -- 172.21.15.16:0/1407805292 reap_dead start 2024-03-28T22:09:06.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.900+0000 7f7fe3e46700 1 -- 172.21.15.16:0/1407805292 shutdown_connections 2024-03-28T22:09:06.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.900+0000 7f7fe3e46700 1 -- 172.21.15.16:0/1407805292 >> 172.21.15.16:0/1407805292 conn(0x7f7fdc079300 msgr2=0x7f7fdc100f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:06.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.900+0000 7f7fe3e46700 1 -- 172.21.15.16:0/1407805292 shutdown_connections 2024-03-28T22:09:06.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.900+0000 7f7fe3e46700 1 -- 172.21.15.16:0/1407805292 wait complete. 2024-03-28T22:09:07.397 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:07 smithi016 ceph-mon[32374]: pgmap v305: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-28T22:09:07.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:07 smithi063 ceph-mon[36249]: pgmap v305: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-28T22:09:07.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:07 smithi053 ceph-mon[35979]: pgmap v305: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-28T22:09:08.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.087+0000 7f8099371700 1 -- 172.21.15.16:0/1508349077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8094108390 msgr2=0x7f8094108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:08.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.087+0000 7f8099371700 1 --2- 172.21.15.16:0/1508349077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8094108390 0x7f8094108770 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f808c0099c0 tx=0x7f808c009cd0 comp rx=0 tx=0).stop 2024-03-28T22:09:08.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.087+0000 7f8099371700 1 -- 172.21.15.16:0/1508349077 shutdown_connections 2024-03-28T22:09:08.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.087+0000 7f8099371700 1 --2- 172.21.15.16:0/1508349077 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8094101150 0x7f809410f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:08.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.087+0000 7f8099371700 1 --2- 172.21.15.16:0/1508349077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8094108390 0x7f8094108770 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:08.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.087+0000 7f8099371700 1 --2- 172.21.15.16:0/1508349077 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8094100790 0x7f8094100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:08.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.087+0000 7f8099371700 1 -- 172.21.15.16:0/1508349077 >> 172.21.15.16:0/1508349077 conn(0x7f8094077710 msgr2=0x7f80940feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:08.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.087+0000 7f8099371700 1 -- 172.21.15.16:0/1508349077 shutdown_connections 2024-03-28T22:09:08.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.087+0000 7f8099371700 1 -- 172.21.15.16:0/1508349077 wait complete. 2024-03-28T22:09:08.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.088+0000 7f8099371700 1 Processor -- start 2024-03-28T22:09:08.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.088+0000 7f8099371700 1 -- start start 2024-03-28T22:09:08.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.088+0000 7f8099371700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8094100790 0x7f809419e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:08.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.088+0000 7f8099371700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8094101150 0x7f809419ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:08.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.088+0000 7f8099371700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8094108390 0x7f80941988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:08.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.088+0000 7f8099371700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8094112620 con 0x7f8094100790 2024-03-28T22:09:08.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.088+0000 7f8099371700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f80941124a0 con 0x7f8094101150 2024-03-28T22:09:08.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.088+0000 7f8099371700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f80941127a0 con 0x7f8094108390 2024-03-28T22:09:08.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.088+0000 7f8092ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8094100790 0x7f809419e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:08.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.088+0000 7f8092ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8094100790 0x7f809419e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33420/0 (socket says 172.21.15.16:33420) 2024-03-28T22:09:08.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.088+0000 7f8092ffd700 1 -- 172.21.15.16:0/4179101334 learned_addr learned my addr 172.21.15.16:0/4179101334 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:08.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.088+0000 7f80927fc700 1 --2- 172.21.15.16:0/4179101334 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8094101150 0x7f809419ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:08.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.088+0000 7f80937fe700 1 --2- 172.21.15.16:0/4179101334 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8094108390 0x7f80941988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:08.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.088+0000 7f8092ffd700 1 -- 172.21.15.16:0/4179101334 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8094108390 msgr2=0x7f80941988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:08.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.088+0000 7f8092ffd700 1 --2- 172.21.15.16:0/4179101334 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8094108390 0x7f80941988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:08.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.088+0000 7f8092ffd700 1 -- 172.21.15.16:0/4179101334 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8094101150 msgr2=0x7f809419ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:08.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.088+0000 7f8092ffd700 1 --2- 172.21.15.16:0/4179101334 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8094101150 0x7f809419ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:08.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.088+0000 7f8092ffd700 1 -- 172.21.15.16:0/4179101334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8094199040 con 0x7f8094100790 2024-03-28T22:09:08.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.089+0000 7f8092ffd700 1 --2- 172.21.15.16:0/4179101334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8094100790 0x7f809419e720 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f808c006010 tx=0x7f808c003e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:08.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.089+0000 7f808bfff700 1 -- 172.21.15.16:0/4179101334 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f808c009e60 con 0x7f8094100790 2024-03-28T22:09:08.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.089+0000 7f8099371700 1 -- 172.21.15.16:0/4179101334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80941992d0 con 0x7f8094100790 2024-03-28T22:09:08.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.089+0000 7f808bfff700 1 -- 172.21.15.16:0/4179101334 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f808c003680 con 0x7f8094100790 2024-03-28T22:09:08.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.089+0000 7f8099371700 1 -- 172.21.15.16:0/4179101334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80941a55f0 con 0x7f8094100790 2024-03-28T22:09:08.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.089+0000 7f808bfff700 1 -- 172.21.15.16:0/4179101334 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f808c0173f0 con 0x7f8094100790 2024-03-28T22:09:08.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.090+0000 7f8099371700 1 -- 172.21.15.16:0/4179101334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8094062950 con 0x7f8094100790 2024-03-28T22:09:08.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.091+0000 7f808bfff700 1 -- 172.21.15.16:0/4179101334 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f808c00f660 con 0x7f8094100790 2024-03-28T22:09:08.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.091+0000 7f808bfff700 1 --2- 172.21.15.16:0/4179101334 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f80800a1ea0 0x7f80800a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:08.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.091+0000 7f80927fc700 1 --2- 172.21.15.16:0/4179101334 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f80800a1ea0 0x7f80800a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:08.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.092+0000 7f808bfff700 1 -- 172.21.15.16:0/4179101334 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f808c0cecf0 con 0x7f8094100790 2024-03-28T22:09:08.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.092+0000 7f808bfff700 1 --2- 172.21.15.16:0/4179101334 >> [v2:172.21.15.53:6824/3357446484,v1:172.21.15.53:6825/3357446484] conn(0x7f80800ab680 0x7f80800adaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:08.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.092+0000 7f808bfff700 1 -- 172.21.15.16:0/4179101334 --> [v2:172.21.15.53:6824/3357446484,v1:172.21.15.53:6825/3357446484] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f808c023e50 con 0x7f80800ab680 2024-03-28T22:09:08.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.092+0000 7f808bfff700 1 -- 172.21.15.16:0/4179101334 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f808c0cf1d0 con 0x7f8094100790 2024-03-28T22:09:08.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.092+0000 7f80937fe700 1 --2- 172.21.15.16:0/4179101334 >> [v2:172.21.15.53:6824/3357446484,v1:172.21.15.53:6825/3357446484] conn(0x7f80800ab680 0x7f80800adaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:08.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.092+0000 7f80927fc700 1 --2- 172.21.15.16:0/4179101334 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f80800a1ea0 0x7f80800a4360 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f80940ffd70 tx=0x7f807c00a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:08.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.092+0000 7f80937fe700 1 --2- 172.21.15.16:0/4179101334 >> [v2:172.21.15.53:6824/3357446484,v1:172.21.15.53:6825/3357446484] conn(0x7f80800ab680 0x7f80800adaa0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:08.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.093+0000 7f808bfff700 1 -- 172.21.15.16:0/4179101334 <== osd.10 v2:172.21.15.53:6824/3357446484 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f808c023e50 con 0x7f80800ab680 2024-03-28T22:09:08.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.109+0000 7f8099371700 1 -- 172.21.15.16:0/4179101334 --> [v2:172.21.15.53:6824/3357446484,v1:172.21.15.53:6825/3357446484] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8094062d50 con 0x7f80800ab680 2024-03-28T22:09:08.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.110+0000 7f808bfff700 1 -- 172.21.15.16:0/4179101334 <== osd.10 v2:172.21.15.53:6824/3357446484 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8094062d50 con 0x7f80800ab680 2024-03-28T22:09:08.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.110+0000 7f8099371700 1 -- 172.21.15.16:0/4179101334 >> [v2:172.21.15.53:6824/3357446484,v1:172.21.15.53:6825/3357446484] conn(0x7f80800ab680 msgr2=0x7f80800adaa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:08.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.110+0000 7f8099371700 1 --2- 172.21.15.16:0/4179101334 >> [v2:172.21.15.53:6824/3357446484,v1:172.21.15.53:6825/3357446484] conn(0x7f80800ab680 0x7f80800adaa0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:08.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.110+0000 7f8099371700 1 -- 172.21.15.16:0/4179101334 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f80800a1ea0 msgr2=0x7f80800a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:08.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.110+0000 7f8099371700 1 --2- 172.21.15.16:0/4179101334 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f80800a1ea0 0x7f80800a4360 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f80940ffd70 tx=0x7f807c00a300 comp rx=0 tx=0).stop 2024-03-28T22:09:08.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.110+0000 7f8099371700 1 -- 172.21.15.16:0/4179101334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8094100790 msgr2=0x7f809419e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:08.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.110+0000 7f8099371700 1 --2- 172.21.15.16:0/4179101334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8094100790 0x7f809419e720 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f808c006010 tx=0x7f808c003e00 comp rx=0 tx=0).stop 2024-03-28T22:09:08.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.110+0000 7f80937fe700 1 -- 172.21.15.16:0/4179101334 reap_dead start 2024-03-28T22:09:08.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.110+0000 7f8099371700 1 -- 172.21.15.16:0/4179101334 shutdown_connections 2024-03-28T22:09:08.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.110+0000 7f8099371700 1 -- 172.21.15.16:0/4179101334 >> 172.21.15.16:0/4179101334 conn(0x7f8094077710 msgr2=0x7f80941051d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:08.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.110+0000 7f8099371700 1 -- 172.21.15.16:0/4179101334 shutdown_connections 2024-03-28T22:09:08.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.110+0000 7f8099371700 1 -- 172.21.15.16:0/4179101334 wait complete. 2024-03-28T22:09:08.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.268+0000 7f5cab9dc700 1 -- 172.21.15.16:0/2329973650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ca4101140 msgr2=0x7f5ca41015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:08.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.268+0000 7f5cab9dc700 1 --2- 172.21.15.16:0/2329973650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ca4101140 0x7f5ca41015a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f5c94009990 tx=0x7f5c94009ca0 comp rx=0 tx=0).stop 2024-03-28T22:09:08.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.268+0000 7f5cab9dc700 1 -- 172.21.15.16:0/2329973650 shutdown_connections 2024-03-28T22:09:08.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.268+0000 7f5cab9dc700 1 --2- 172.21.15.16:0/2329973650 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ca4101ae0 0x7f5ca410dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:08.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.268+0000 7f5cab9dc700 1 --2- 172.21.15.16:0/2329973650 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5ca4107180 0x7f5ca4107560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:08.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.268+0000 7f5cab9dc700 1 --2- 172.21.15.16:0/2329973650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ca4101140 0x7f5ca41015a0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:08.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.268+0000 7f5cab9dc700 1 -- 172.21.15.16:0/2329973650 >> 172.21.15.16:0/2329973650 conn(0x7f5ca4078070 msgr2=0x7f5ca4078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:08.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.268+0000 7f5cab9dc700 1 -- 172.21.15.16:0/2329973650 shutdown_connections 2024-03-28T22:09:08.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.268+0000 7f5cab9dc700 1 -- 172.21.15.16:0/2329973650 wait complete. 2024-03-28T22:09:08.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.269+0000 7f5cab9dc700 1 Processor -- start 2024-03-28T22:09:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.269+0000 7f5cab9dc700 1 -- start start 2024-03-28T22:09:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.269+0000 7f5cab9dc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ca4101140 0x7f5ca4199780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.269+0000 7f5cab9dc700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5ca4101ae0 0x7f5ca4199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.269+0000 7f5cab9dc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ca4107180 0x7f5ca41a4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.269+0000 7f5cab9dc700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5ca41113e0 con 0x7f5ca4107180 2024-03-28T22:09:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.269+0000 7f5cab9dc700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5ca4111260 con 0x7f5ca4101140 2024-03-28T22:09:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.269+0000 7f5cab9dc700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5ca4111560 con 0x7f5ca4101ae0 2024-03-28T22:09:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.270+0000 7f5ca9778700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ca4101140 0x7f5ca4199780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.270+0000 7f5ca9f79700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ca4107180 0x7f5ca41a4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.270+0000 7f5ca9778700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ca4101140 0x7f5ca4199780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:39554/0 (socket says 172.21.15.16:39554) 2024-03-28T22:09:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.270+0000 7f5ca9f79700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ca4107180 0x7f5ca41a4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33438/0 (socket says 172.21.15.16:33438) 2024-03-28T22:09:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.270+0000 7f5ca9778700 1 -- 172.21.15.16:0/52136205 learned_addr learned my addr 172.21.15.16:0/52136205 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.270+0000 7f5ca8f77700 1 --2- 172.21.15.16:0/52136205 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5ca4101ae0 0x7f5ca4199cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.270+0000 7f5ca9f79700 1 -- 172.21.15.16:0/52136205 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5ca4101ae0 msgr2=0x7f5ca4199cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:08.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.270+0000 7f5ca9f79700 1 --2- 172.21.15.16:0/52136205 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5ca4101ae0 0x7f5ca4199cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:08.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.270+0000 7f5ca9f79700 1 -- 172.21.15.16:0/52136205 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ca4101140 msgr2=0x7f5ca4199780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:08.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.270+0000 7f5ca9f79700 1 --2- 172.21.15.16:0/52136205 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ca4101140 0x7f5ca4199780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:08.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.270+0000 7f5ca9f79700 1 -- 172.21.15.16:0/52136205 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ca41a4d40 con 0x7f5ca4107180 2024-03-28T22:09:08.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.270+0000 7f5ca9778700 1 --2- 172.21.15.16:0/52136205 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ca4101140 0x7f5ca4199780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:09:08.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.270+0000 7f5ca9f79700 1 --2- 172.21.15.16:0/52136205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ca4107180 0x7f5ca41a4800 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5c9c00efc0 tx=0x7f5c9c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:08.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.270+0000 7f5c9a7fc700 1 -- 172.21.15.16:0/52136205 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c9c0092e0 con 0x7f5ca4107180 2024-03-28T22:09:08.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.270+0000 7f5cab9dc700 1 -- 172.21.15.16:0/52136205 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ca41a4f70 con 0x7f5ca4107180 2024-03-28T22:09:08.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.270+0000 7f5c9a7fc700 1 -- 172.21.15.16:0/52136205 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c9c004bb0 con 0x7f5ca4107180 2024-03-28T22:09:08.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.270+0000 7f5c9a7fc700 1 -- 172.21.15.16:0/52136205 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c9c004280 con 0x7f5ca4107180 2024-03-28T22:09:08.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.271+0000 7f5cab9dc700 1 -- 172.21.15.16:0/52136205 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ca41a53d0 con 0x7f5ca4107180 2024-03-28T22:09:08.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.271+0000 7f5cab9dc700 1 -- 172.21.15.16:0/52136205 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5ca4062950 con 0x7f5ca4107180 2024-03-28T22:09:08.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.273+0000 7f5c9a7fc700 1 -- 172.21.15.16:0/52136205 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f5c9c01b030 con 0x7f5ca4107180 2024-03-28T22:09:08.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.273+0000 7f5c9a7fc700 1 --2- 172.21.15.16:0/52136205 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f5c900a1ea0 0x7f5c900a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:08.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.273+0000 7f5ca9778700 1 --2- 172.21.15.16:0/52136205 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f5c900a1ea0 0x7f5c900a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:08.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.273+0000 7f5c9a7fc700 1 -- 172.21.15.16:0/52136205 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f5c9c0ce1c0 con 0x7f5ca4107180 2024-03-28T22:09:08.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.274+0000 7f5c9a7fc700 1 --2- 172.21.15.16:0/52136205 >> [v2:172.21.15.16:6818/4290459224,v1:172.21.15.16:6819/4290459224] conn(0x7f5c900ab680 0x7f5c900adaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:08.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.274+0000 7f5c9a7fc700 1 -- 172.21.15.16:0/52136205 --> [v2:172.21.15.16:6818/4290459224,v1:172.21.15.16:6819/4290459224] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5c9c011e50 con 0x7f5c900ab680 2024-03-28T22:09:08.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.274+0000 7f5c9a7fc700 1 -- 172.21.15.16:0/52136205 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f5c9c0ce6a0 con 0x7f5ca4107180 2024-03-28T22:09:08.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.274+0000 7f5ca8f77700 1 --2- 172.21.15.16:0/52136205 >> [v2:172.21.15.16:6818/4290459224,v1:172.21.15.16:6819/4290459224] conn(0x7f5c900ab680 0x7f5c900adaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:08.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.274+0000 7f5ca9778700 1 --2- 172.21.15.16:0/52136205 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f5c900a1ea0 0x7f5c900a4360 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5ca000f4d0 tx=0x7f5ca0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:08.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.274+0000 7f5ca8f77700 1 --2- 172.21.15.16:0/52136205 >> [v2:172.21.15.16:6818/4290459224,v1:172.21.15.16:6819/4290459224] conn(0x7f5c900ab680 0x7f5c900adaa0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:08.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.275+0000 7f5c9a7fc700 1 -- 172.21.15.16:0/52136205 <== osd.6 v2:172.21.15.16:6818/4290459224 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5c9c011e50 con 0x7f5c900ab680 2024-03-28T22:09:08.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.292+0000 7f5cab9dc700 1 -- 172.21.15.16:0/52136205 --> [v2:172.21.15.16:6818/4290459224,v1:172.21.15.16:6819/4290459224] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5ca4062d50 con 0x7f5c900ab680 2024-03-28T22:09:08.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.292+0000 7f5c9a7fc700 1 -- 172.21.15.16:0/52136205 <== osd.6 v2:172.21.15.16:6818/4290459224 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5ca4062d50 con 0x7f5c900ab680 2024-03-28T22:09:08.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.293+0000 7f5cab9dc700 1 -- 172.21.15.16:0/52136205 >> [v2:172.21.15.16:6818/4290459224,v1:172.21.15.16:6819/4290459224] conn(0x7f5c900ab680 msgr2=0x7f5c900adaa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:08.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.293+0000 7f5cab9dc700 1 --2- 172.21.15.16:0/52136205 >> [v2:172.21.15.16:6818/4290459224,v1:172.21.15.16:6819/4290459224] conn(0x7f5c900ab680 0x7f5c900adaa0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:08.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.293+0000 7f5cab9dc700 1 -- 172.21.15.16:0/52136205 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f5c900a1ea0 msgr2=0x7f5c900a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:08.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.293+0000 7f5cab9dc700 1 --2- 172.21.15.16:0/52136205 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f5c900a1ea0 0x7f5c900a4360 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5ca000f4d0 tx=0x7f5ca0005f90 comp rx=0 tx=0).stop 2024-03-28T22:09:08.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.293+0000 7f5cab9dc700 1 -- 172.21.15.16:0/52136205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ca4107180 msgr2=0x7f5ca41a4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:08.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.293+0000 7f5cab9dc700 1 --2- 172.21.15.16:0/52136205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ca4107180 0x7f5ca41a4800 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5c9c00efc0 tx=0x7f5c9c00c3a0 comp rx=0 tx=0).stop 2024-03-28T22:09:08.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.293+0000 7f5ca9f79700 1 -- 172.21.15.16:0/52136205 reap_dead start 2024-03-28T22:09:08.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.293+0000 7f5cab9dc700 1 -- 172.21.15.16:0/52136205 shutdown_connections 2024-03-28T22:09:08.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.293+0000 7f5cab9dc700 1 -- 172.21.15.16:0/52136205 >> 172.21.15.16:0/52136205 conn(0x7f5ca4078070 msgr2=0x7f5ca40ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:08.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.293+0000 7f5cab9dc700 1 -- 172.21.15.16:0/52136205 shutdown_connections 2024-03-28T22:09:08.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:08.293+0000 7f5cab9dc700 1 -- 172.21.15.16:0/52136205 wait complete. 2024-03-28T22:09:08.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.785+0000 7f9b994f5700 1 -- 172.21.15.16:0/2152359827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b94108270 msgr2=0x7f9b94108650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:08.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.785+0000 7f9b994f5700 1 --2- 172.21.15.16:0/2152359827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b94108270 0x7f9b94108650 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9b840099c0 tx=0x7f9b84009cd0 comp rx=0 tx=0).stop 2024-03-28T22:09:08.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.785+0000 7f9b994f5700 1 -- 172.21.15.16:0/2152359827 shutdown_connections 2024-03-28T22:09:08.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.785+0000 7f9b994f5700 1 --2- 172.21.15.16:0/2152359827 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b94102c50 0x7f9b9410f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:08.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.785+0000 7f9b994f5700 1 --2- 172.21.15.16:0/2152359827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b94108270 0x7f9b94108650 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:08.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.785+0000 7f9b994f5700 1 --2- 172.21.15.16:0/2152359827 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b94102290 0x7f9b94102710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:08.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.785+0000 7f9b994f5700 1 -- 172.21.15.16:0/2152359827 >> 172.21.15.16:0/2152359827 conn(0x7f9b94079370 msgr2=0x7f9b94079780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:08.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.785+0000 7f9b994f5700 1 -- 172.21.15.16:0/2152359827 shutdown_connections 2024-03-28T22:09:08.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.785+0000 7f9b994f5700 1 -- 172.21.15.16:0/2152359827 wait complete. 2024-03-28T22:09:08.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.786+0000 7f9b994f5700 1 Processor -- start 2024-03-28T22:09:08.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.786+0000 7f9b994f5700 1 -- start start 2024-03-28T22:09:08.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.786+0000 7f9b994f5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b94102290 0x7f9b9419e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:08.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.786+0000 7f9b994f5700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b94102c50 0x7f9b9419ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:08.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.786+0000 7f9b994f5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b94108270 0x7f9b94198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:08.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.786+0000 7f9b994f5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9b94112520 con 0x7f9b94102290 2024-03-28T22:09:08.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.786+0000 7f9b994f5700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9b941123a0 con 0x7f9b94108270 2024-03-28T22:09:08.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.786+0000 7f9b994f5700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9b941126a0 con 0x7f9b94102c50 2024-03-28T22:09:08.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.786+0000 7f9b92ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b94102290 0x7f9b9419e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:08.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.786+0000 7f9b92ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b94102290 0x7f9b9419e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33390/0 (socket says 172.21.15.16:33390) 2024-03-28T22:09:08.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.786+0000 7f9b92ffd700 1 -- 172.21.15.16:0/2204067798 learned_addr learned my addr 172.21.15.16:0/2204067798 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:08.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.787+0000 7f9b937fe700 1 --2- 172.21.15.16:0/2204067798 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b94108270 0x7f9b94198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:08.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.787+0000 7f9b927fc700 1 --2- 172.21.15.16:0/2204067798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b94102c50 0x7f9b9419ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:08.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.787+0000 7f9b92ffd700 1 -- 172.21.15.16:0/2204067798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b94102c50 msgr2=0x7f9b9419ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:08.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.787+0000 7f9b92ffd700 1 --2- 172.21.15.16:0/2204067798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b94102c50 0x7f9b9419ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:08.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.787+0000 7f9b92ffd700 1 -- 172.21.15.16:0/2204067798 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b94108270 msgr2=0x7f9b94198800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:08.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.787+0000 7f9b92ffd700 1 --2- 172.21.15.16:0/2204067798 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b94108270 0x7f9b94198800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:08.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.787+0000 7f9b92ffd700 1 -- 172.21.15.16:0/2204067798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b94198fe0 con 0x7f9b94102290 2024-03-28T22:09:08.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.787+0000 7f9b937fe700 1 --2- 172.21.15.16:0/2204067798 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b94108270 0x7f9b94198800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:09:08.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.787+0000 7f9b92ffd700 1 --2- 172.21.15.16:0/2204067798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b94102290 0x7f9b9419e680 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9b840059d0 tx=0x7f9b8400fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:08.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.787+0000 7f9b8bfff700 1 -- 172.21.15.16:0/2204067798 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b84027070 con 0x7f9b94102290 2024-03-28T22:09:08.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.787+0000 7f9b994f5700 1 -- 172.21.15.16:0/2204067798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b94199270 con 0x7f9b94102290 2024-03-28T22:09:08.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.787+0000 7f9b8bfff700 1 -- 172.21.15.16:0/2204067798 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b84003680 con 0x7f9b94102290 2024-03-28T22:09:08.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.787+0000 7f9b8bfff700 1 -- 172.21.15.16:0/2204067798 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b840173f0 con 0x7f9b94102290 2024-03-28T22:09:08.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.787+0000 7f9b994f5700 1 -- 172.21.15.16:0/2204067798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b941a54e0 con 0x7f9b94102290 2024-03-28T22:09:08.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.788+0000 7f9b994f5700 1 -- 172.21.15.16:0/2204067798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9b94062990 con 0x7f9b94102290 2024-03-28T22:09:08.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.789+0000 7f9b8bfff700 1 -- 172.21.15.16:0/2204067798 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f9b8402a070 con 0x7f9b94102290 2024-03-28T22:09:08.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.790+0000 7f9b8bfff700 1 --2- 172.21.15.16:0/2204067798 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f9b800a1ea0 0x7f9b800a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:08.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.790+0000 7f9b8bfff700 1 -- 172.21.15.16:0/2204067798 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f9b840cec90 con 0x7f9b94102290 2024-03-28T22:09:08.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.790+0000 7f9b927fc700 1 --2- 172.21.15.16:0/2204067798 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f9b800a1ea0 0x7f9b800a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:08.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.790+0000 7f9b8bfff700 1 --2- 172.21.15.16:0/2204067798 >> [v2:172.21.15.16:6810/13176789,v1:172.21.15.16:6811/13176789] conn(0x7f9b800ab680 0x7f9b800adaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:08.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.790+0000 7f9b8bfff700 1 -- 172.21.15.16:0/2204067798 --> [v2:172.21.15.16:6810/13176789,v1:172.21.15.16:6811/13176789] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9b84001d30 con 0x7f9b800ab680 2024-03-28T22:09:08.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.790+0000 7f9b8bfff700 1 -- 172.21.15.16:0/2204067798 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f9b840cf040 con 0x7f9b94102290 2024-03-28T22:09:08.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.790+0000 7f9b937fe700 1 --2- 172.21.15.16:0/2204067798 >> [v2:172.21.15.16:6810/13176789,v1:172.21.15.16:6811/13176789] conn(0x7f9b800ab680 0x7f9b800adaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:08.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.790+0000 7f9b927fc700 1 --2- 172.21.15.16:0/2204067798 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f9b800a1ea0 0x7f9b800a4360 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9b94079950 tx=0x7f9b7c00b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:08.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.791+0000 7f9b937fe700 1 --2- 172.21.15.16:0/2204067798 >> [v2:172.21.15.16:6810/13176789,v1:172.21.15.16:6811/13176789] conn(0x7f9b800ab680 0x7f9b800adaa0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:08.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.791+0000 7f9b8bfff700 1 -- 172.21.15.16:0/2204067798 <== osd.3 v2:172.21.15.16:6810/13176789 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9b84001d30 con 0x7f9b800ab680 2024-03-28T22:09:08.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.808+0000 7f9b994f5700 1 -- 172.21.15.16:0/2204067798 --> [v2:172.21.15.16:6810/13176789,v1:172.21.15.16:6811/13176789] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9b94062dc0 con 0x7f9b800ab680 2024-03-28T22:09:08.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.808+0000 7f9b8bfff700 1 -- 172.21.15.16:0/2204067798 <== osd.3 v2:172.21.15.16:6810/13176789 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9b94062dc0 con 0x7f9b800ab680 2024-03-28T22:09:08.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.808+0000 7f9b994f5700 1 -- 172.21.15.16:0/2204067798 >> [v2:172.21.15.16:6810/13176789,v1:172.21.15.16:6811/13176789] conn(0x7f9b800ab680 msgr2=0x7f9b800adaa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:08.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.809+0000 7f9b994f5700 1 --2- 172.21.15.16:0/2204067798 >> [v2:172.21.15.16:6810/13176789,v1:172.21.15.16:6811/13176789] conn(0x7f9b800ab680 0x7f9b800adaa0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:08.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.809+0000 7f9b994f5700 1 -- 172.21.15.16:0/2204067798 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f9b800a1ea0 msgr2=0x7f9b800a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:08.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.809+0000 7f9b994f5700 1 --2- 172.21.15.16:0/2204067798 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f9b800a1ea0 0x7f9b800a4360 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9b94079950 tx=0x7f9b7c00b3c0 comp rx=0 tx=0).stop 2024-03-28T22:09:08.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.809+0000 7f9b994f5700 1 -- 172.21.15.16:0/2204067798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b94102290 msgr2=0x7f9b9419e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:08.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.809+0000 7f9b994f5700 1 --2- 172.21.15.16:0/2204067798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b94102290 0x7f9b9419e680 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9b840059d0 tx=0x7f9b8400fa00 comp rx=0 tx=0).stop 2024-03-28T22:09:08.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.809+0000 7f9b937fe700 1 -- 172.21.15.16:0/2204067798 reap_dead start 2024-03-28T22:09:08.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.809+0000 7f9b994f5700 1 -- 172.21.15.16:0/2204067798 shutdown_connections 2024-03-28T22:09:08.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.809+0000 7f9b994f5700 1 -- 172.21.15.16:0/2204067798 >> 172.21.15.16:0/2204067798 conn(0x7f9b94079370 msgr2=0x7f9b94100f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:08.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.809+0000 7f9b994f5700 1 -- 172.21.15.16:0/2204067798 shutdown_connections 2024-03-28T22:09:08.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:06.809+0000 7f9b994f5700 1 -- 172.21.15.16:0/2204067798 wait complete. 2024-03-28T22:09:09.237 INFO:teuthology.orchestra.run.smithi016.stdout:111669149756 2024-03-28T22:09:09.238 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-28T22:09:09.426 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:09 smithi016 ceph-mon[32374]: pgmap v306: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-28T22:09:09.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:09 smithi063 ceph-mon[36249]: pgmap v306: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-28T22:09:09.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:09 smithi053 ceph-mon[35979]: pgmap v306: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-28T22:09:09.771 INFO:teuthology.orchestra.run.smithi016.stdout:77309411393 2024-03-28T22:09:09.772 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-28T22:09:09.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.424+0000 7f865fb80700 1 -- 172.21.15.16:0/1434684192 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8658102290 msgr2=0x7f8658102710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:09.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.424+0000 7f865fb80700 1 --2- 172.21.15.16:0/1434684192 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8658102290 0x7f8658102710 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f86480098f0 tx=0x7f8648009c00 comp rx=0 tx=0).stop 2024-03-28T22:09:09.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.424+0000 7f865fb80700 1 -- 172.21.15.16:0/1434684192 shutdown_connections 2024-03-28T22:09:09.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.424+0000 7f865fb80700 1 --2- 172.21.15.16:0/1434684192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8658102c50 0x7f865810f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:09.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.424+0000 7f865fb80700 1 --2- 172.21.15.16:0/1434684192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8658108270 0x7f8658108650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:09.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.424+0000 7f865fb80700 1 --2- 172.21.15.16:0/1434684192 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8658102290 0x7f8658102710 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:09.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.424+0000 7f865fb80700 1 -- 172.21.15.16:0/1434684192 >> 172.21.15.16:0/1434684192 conn(0x7f8658079370 msgr2=0x7f8658079780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:09.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.424+0000 7f865fb80700 1 -- 172.21.15.16:0/1434684192 shutdown_connections 2024-03-28T22:09:09.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.424+0000 7f865fb80700 1 -- 172.21.15.16:0/1434684192 wait complete. 2024-03-28T22:09:09.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.425+0000 7f865fb80700 1 Processor -- start 2024-03-28T22:09:09.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.425+0000 7f865fb80700 1 -- start start 2024-03-28T22:09:09.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.425+0000 7f865fb80700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8658102290 0x7f865819f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:09.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.425+0000 7f865fb80700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8658102c50 0x7f865819fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:09.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.426+0000 7f865fb80700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8658108270 0x7f865810e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:09.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.426+0000 7f865fb80700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8658112510 con 0x7f8658108270 2024-03-28T22:09:09.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.426+0000 7f865fb80700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8658112390 con 0x7f8658102c50 2024-03-28T22:09:09.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.426+0000 7f865fb80700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8658112690 con 0x7f8658102290 2024-03-28T22:09:09.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.426+0000 7f865e11d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8658108270 0x7f865810e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:09.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.426+0000 7f865e11d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8658108270 0x7f865810e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33460/0 (socket says 172.21.15.16:33460) 2024-03-28T22:09:09.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.426+0000 7f865e11d700 1 -- 172.21.15.16:0/4031825280 learned_addr learned my addr 172.21.15.16:0/4031825280 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:09.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.426+0000 7f865d11b700 1 --2- 172.21.15.16:0/4031825280 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8658102c50 0x7f865819fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:09.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.426+0000 7f865d91c700 1 --2- 172.21.15.16:0/4031825280 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8658102290 0x7f865819f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:09.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.426+0000 7f865e11d700 1 -- 172.21.15.16:0/4031825280 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8658102290 msgr2=0x7f865819f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:09.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.426+0000 7f865e11d700 1 --2- 172.21.15.16:0/4031825280 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8658102290 0x7f865819f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:09.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.426+0000 7f865e11d700 1 -- 172.21.15.16:0/4031825280 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8658102c50 msgr2=0x7f865819fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:09.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.426+0000 7f865e11d700 1 --2- 172.21.15.16:0/4031825280 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8658102c50 0x7f865819fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:09.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.426+0000 7f865e11d700 1 -- 172.21.15.16:0/4031825280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f865810eb80 con 0x7f8658108270 2024-03-28T22:09:09.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.426+0000 7f865d91c700 1 --2- 172.21.15.16:0/4031825280 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8658102290 0x7f865819f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:09:09.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.426+0000 7f865e11d700 1 --2- 172.21.15.16:0/4031825280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8658108270 0x7f865810e280 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f865000e3c0 tx=0x7f865000e780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:09.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.427+0000 7f864effd700 1 -- 172.21.15.16:0/4031825280 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86500092d0 con 0x7f8658108270 2024-03-28T22:09:09.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.427+0000 7f864effd700 1 -- 172.21.15.16:0/4031825280 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86500075c0 con 0x7f8658108270 2024-03-28T22:09:09.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.427+0000 7f864effd700 1 -- 172.21.15.16:0/4031825280 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8650004290 con 0x7f8658108270 2024-03-28T22:09:09.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.427+0000 7f865fb80700 1 -- 172.21.15.16:0/4031825280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f865810ee70 con 0x7f8658108270 2024-03-28T22:09:09.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.427+0000 7f865fb80700 1 -- 172.21.15.16:0/4031825280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f865819b2c0 con 0x7f8658108270 2024-03-28T22:09:09.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.428+0000 7f865fb80700 1 -- 172.21.15.16:0/4031825280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f86580760f0 con 0x7f8658108270 2024-03-28T22:09:09.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.429+0000 7f864effd700 1 -- 172.21.15.16:0/4031825280 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f8650012050 con 0x7f8658108270 2024-03-28T22:09:09.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.430+0000 7f864effd700 1 --2- 172.21.15.16:0/4031825280 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f86440a1fa0 0x7f86440a4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:09.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.430+0000 7f864effd700 1 -- 172.21.15.16:0/4031825280 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f86500cd9f0 con 0x7f8658108270 2024-03-28T22:09:09.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.430+0000 7f865d91c700 1 --2- 172.21.15.16:0/4031825280 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f86440a1fa0 0x7f86440a4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:09.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.430+0000 7f864effd700 1 --2- 172.21.15.16:0/4031825280 >> [v2:172.21.15.53:6800/795531234,v1:172.21.15.53:6801/795531234] conn(0x7f86440ab780 0x7f86440adba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:09.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.430+0000 7f864effd700 1 -- 172.21.15.16:0/4031825280 --> [v2:172.21.15.53:6800/795531234,v1:172.21.15.53:6801/795531234] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f86440ae250 con 0x7f86440ab780 2024-03-28T22:09:09.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.430+0000 7f864effd700 1 -- 172.21.15.16:0/4031825280 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f86500d22a0 con 0x7f8658108270 2024-03-28T22:09:09.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.430+0000 7f865d11b700 1 --2- 172.21.15.16:0/4031825280 >> [v2:172.21.15.53:6800/795531234,v1:172.21.15.53:6801/795531234] conn(0x7f86440ab780 0x7f86440adba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:09.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.431+0000 7f865d91c700 1 --2- 172.21.15.16:0/4031825280 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f86440a1fa0 0x7f86440a4460 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8654009cc0 tx=0x7f8654009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:09.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.431+0000 7f865d11b700 1 --2- 172.21.15.16:0/4031825280 >> [v2:172.21.15.53:6800/795531234,v1:172.21.15.53:6801/795531234] conn(0x7f86440ab780 0x7f86440adba0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:09.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.431+0000 7f864effd700 1 -- 172.21.15.16:0/4031825280 <== osd.1 v2:172.21.15.53:6800/795531234 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f86440ae250 con 0x7f86440ab780 2024-03-28T22:09:09.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.448+0000 7f865fb80700 1 -- 172.21.15.16:0/4031825280 --> [v2:172.21.15.53:6800/795531234,v1:172.21.15.53:6801/795531234] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8658103920 con 0x7f86440ab780 2024-03-28T22:09:09.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.449+0000 7f864effd700 1 -- 172.21.15.16:0/4031825280 <== osd.1 v2:172.21.15.53:6800/795531234 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f8658103920 con 0x7f86440ab780 2024-03-28T22:09:09.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.449+0000 7f865fb80700 1 -- 172.21.15.16:0/4031825280 >> [v2:172.21.15.53:6800/795531234,v1:172.21.15.53:6801/795531234] conn(0x7f86440ab780 msgr2=0x7f86440adba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:09.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.449+0000 7f865fb80700 1 --2- 172.21.15.16:0/4031825280 >> [v2:172.21.15.53:6800/795531234,v1:172.21.15.53:6801/795531234] conn(0x7f86440ab780 0x7f86440adba0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:09.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.449+0000 7f865fb80700 1 -- 172.21.15.16:0/4031825280 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f86440a1fa0 msgr2=0x7f86440a4460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:09.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.449+0000 7f865fb80700 1 --2- 172.21.15.16:0/4031825280 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f86440a1fa0 0x7f86440a4460 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8654009cc0 tx=0x7f8654009260 comp rx=0 tx=0).stop 2024-03-28T22:09:09.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.449+0000 7f865fb80700 1 -- 172.21.15.16:0/4031825280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8658108270 msgr2=0x7f865810e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:09.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.449+0000 7f865fb80700 1 --2- 172.21.15.16:0/4031825280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8658108270 0x7f865810e280 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f865000e3c0 tx=0x7f865000e780 comp rx=0 tx=0).stop 2024-03-28T22:09:09.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.449+0000 7f865e11d700 1 -- 172.21.15.16:0/4031825280 reap_dead start 2024-03-28T22:09:09.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.449+0000 7f865fb80700 1 -- 172.21.15.16:0/4031825280 shutdown_connections 2024-03-28T22:09:09.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.449+0000 7f865fb80700 1 -- 172.21.15.16:0/4031825280 >> 172.21.15.16:0/4031825280 conn(0x7f8658079370 msgr2=0x7f8658074700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:09.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.450+0000 7f865fb80700 1 -- 172.21.15.16:0/4031825280 shutdown_connections 2024-03-28T22:09:09.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:09.450+0000 7f865fb80700 1 -- 172.21.15.16:0/4031825280 wait complete. 2024-03-28T22:09:10.776 INFO:teuthology.orchestra.run.smithi016.stdout:313532612633 2024-03-28T22:09:10.776 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-28T22:09:11.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:11 smithi063 ceph-mon[36249]: pgmap v307: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 341 B/s wr, 7 op/s 2024-03-28T22:09:11.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:11 smithi016 ceph-mon[32374]: pgmap v307: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 341 B/s wr, 7 op/s 2024-03-28T22:09:11.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:11 smithi053 ceph-mon[35979]: pgmap v307: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 341 B/s wr, 7 op/s 2024-03-28T22:09:11.647 INFO:teuthology.orchestra.run.smithi016.stdout:146028888119 2024-03-28T22:09:11.648 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-28T22:09:12.201 INFO:teuthology.orchestra.run.smithi016.stdout:292057776158 2024-03-28T22:09:12.202 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-28T22:09:12.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.416+0000 7f44063f7700 1 -- 172.21.15.16:0/2805360149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44001013d0 msgr2=0x7f44001017b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:12.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.416+0000 7f44063f7700 1 --2- 172.21.15.16:0/2805360149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44001013d0 0x7f44001017b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f43f00099c0 tx=0x7f43f0009cd0 comp rx=0 tx=0).stop 2024-03-28T22:09:12.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.416+0000 7f44063f7700 1 -- 172.21.15.16:0/2805360149 shutdown_connections 2024-03-28T22:09:12.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.416+0000 7f44063f7700 1 --2- 172.21.15.16:0/2805360149 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4400106420 0x7f440010dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:12.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.416+0000 7f44063f7700 1 --2- 172.21.15.16:0/2805360149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44001013d0 0x7f44001017b0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:12.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.416+0000 7f44063f7700 1 --2- 172.21.15.16:0/2805360149 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4400101d80 0x7f4400105d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:12.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.416+0000 7f44063f7700 1 -- 172.21.15.16:0/2805360149 >> 172.21.15.16:0/2805360149 conn(0x7f4400077190 msgr2=0x7f44000775a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:12.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.416+0000 7f44063f7700 1 -- 172.21.15.16:0/2805360149 shutdown_connections 2024-03-28T22:09:12.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.416+0000 7f44063f7700 1 -- 172.21.15.16:0/2805360149 wait complete. 2024-03-28T22:09:12.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.417+0000 7f44063f7700 1 Processor -- start 2024-03-28T22:09:12.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.417+0000 7f44063f7700 1 -- start start 2024-03-28T22:09:12.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.417+0000 7f44063f7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44001013d0 0x7f440019e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:12.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.417+0000 7f44063f7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4400101d80 0x7f440019ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:12.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.417+0000 7f44063f7700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4400106420 0x7f44001988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:12.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.417+0000 7f44063f7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4400111400 con 0x7f44001013d0 2024-03-28T22:09:12.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.417+0000 7f44063f7700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4400111280 con 0x7f4400101d80 2024-03-28T22:09:12.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.417+0000 7f44063f7700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4400111580 con 0x7f4400106420 2024-03-28T22:09:12.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.418+0000 7f43fffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44001013d0 0x7f440019e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:12.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.418+0000 7f43fffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44001013d0 0x7f440019e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35822/0 (socket says 172.21.15.16:35822) 2024-03-28T22:09:12.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.418+0000 7f43fffff700 1 -- 172.21.15.16:0/1717022183 learned_addr learned my addr 172.21.15.16:0/1717022183 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:12.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.418+0000 7f43ff7fe700 1 --2- 172.21.15.16:0/1717022183 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4400101d80 0x7f440019ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:12.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.418+0000 7f4404994700 1 --2- 172.21.15.16:0/1717022183 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4400106420 0x7f44001988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:12.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.418+0000 7f43fffff700 1 -- 172.21.15.16:0/1717022183 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4400106420 msgr2=0x7f44001988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:12.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.418+0000 7f43fffff700 1 --2- 172.21.15.16:0/1717022183 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4400106420 0x7f44001988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:12.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.418+0000 7f43fffff700 1 -- 172.21.15.16:0/1717022183 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4400101d80 msgr2=0x7f440019ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:12.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.418+0000 7f43fffff700 1 --2- 172.21.15.16:0/1717022183 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4400101d80 0x7f440019ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:12.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.418+0000 7f43fffff700 1 -- 172.21.15.16:0/1717022183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4400199050 con 0x7f44001013d0 2024-03-28T22:09:12.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.418+0000 7f43fffff700 1 --2- 172.21.15.16:0/1717022183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44001013d0 0x7f440019e6e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f43f0006010 tx=0x7f43f0009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:12.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.418+0000 7f43fd7fa700 1 -- 172.21.15.16:0/1717022183 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f0027070 con 0x7f44001013d0 2024-03-28T22:09:12.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.418+0000 7f44063f7700 1 -- 172.21.15.16:0/1717022183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44001992e0 con 0x7f44001013d0 2024-03-28T22:09:12.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.418+0000 7f44063f7700 1 -- 172.21.15.16:0/1717022183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44001a55b0 con 0x7f44001013d0 2024-03-28T22:09:12.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.418+0000 7f43fd7fa700 1 -- 172.21.15.16:0/1717022183 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43f0003780 con 0x7f44001013d0 2024-03-28T22:09:12.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.418+0000 7f43fd7fa700 1 -- 172.21.15.16:0/1717022183 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f0022da0 con 0x7f44001013d0 2024-03-28T22:09:12.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.419+0000 7f44063f7700 1 -- 172.21.15.16:0/1717022183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4400062950 con 0x7f44001013d0 2024-03-28T22:09:12.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.420+0000 7f43fd7fa700 1 -- 172.21.15.16:0/1717022183 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f43f002a070 con 0x7f44001013d0 2024-03-28T22:09:12.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.421+0000 7f43fd7fa700 1 --2- 172.21.15.16:0/1717022183 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f43e80a1f20 0x7f43e80a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:12.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.421+0000 7f43ff7fe700 1 --2- 172.21.15.16:0/1717022183 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f43e80a1f20 0x7f43e80a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:12.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.421+0000 7f43fd7fa700 1 -- 172.21.15.16:0/1717022183 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f43f00ceff0 con 0x7f44001013d0 2024-03-28T22:09:12.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.421+0000 7f43fd7fa700 1 --2- 172.21.15.16:0/1717022183 >> [v2:172.21.15.53:6816/1056467525,v1:172.21.15.53:6817/1056467525] conn(0x7f43e80ab700 0x7f43e80adb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:12.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.421+0000 7f43fd7fa700 1 -- 172.21.15.16:0/1717022183 --> [v2:172.21.15.53:6816/1056467525,v1:172.21.15.53:6817/1056467525] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f43f00043d0 con 0x7f43e80ab700 2024-03-28T22:09:12.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.421+0000 7f43fd7fa700 1 -- 172.21.15.16:0/1717022183 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f43f00cf3a0 con 0x7f44001013d0 2024-03-28T22:09:12.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.422+0000 7f4404994700 1 --2- 172.21.15.16:0/1717022183 >> [v2:172.21.15.53:6816/1056467525,v1:172.21.15.53:6817/1056467525] conn(0x7f43e80ab700 0x7f43e80adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:12.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.422+0000 7f43ff7fe700 1 --2- 172.21.15.16:0/1717022183 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f43e80a1f20 0x7f43e80a43e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4400107a10 tx=0x7f43f400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:12.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.422+0000 7f4404994700 1 --2- 172.21.15.16:0/1717022183 >> [v2:172.21.15.53:6816/1056467525,v1:172.21.15.53:6817/1056467525] conn(0x7f43e80ab700 0x7f43e80adb20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:12.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.423+0000 7f43fd7fa700 1 -- 172.21.15.16:0/1717022183 <== osd.7 v2:172.21.15.53:6816/1056467525 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f43f00043d0 con 0x7f43e80ab700 2024-03-28T22:09:12.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.440+0000 7f44063f7700 1 -- 172.21.15.16:0/1717022183 --> [v2:172.21.15.53:6816/1056467525,v1:172.21.15.53:6817/1056467525] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4400062d80 con 0x7f43e80ab700 2024-03-28T22:09:12.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.440+0000 7f43fd7fa700 1 -- 172.21.15.16:0/1717022183 <== osd.7 v2:172.21.15.53:6816/1056467525 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4400062d80 con 0x7f43e80ab700 2024-03-28T22:09:12.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.441+0000 7f44063f7700 1 -- 172.21.15.16:0/1717022183 >> [v2:172.21.15.53:6816/1056467525,v1:172.21.15.53:6817/1056467525] conn(0x7f43e80ab700 msgr2=0x7f43e80adb20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:12.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.441+0000 7f44063f7700 1 --2- 172.21.15.16:0/1717022183 >> [v2:172.21.15.53:6816/1056467525,v1:172.21.15.53:6817/1056467525] conn(0x7f43e80ab700 0x7f43e80adb20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:12.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.441+0000 7f44063f7700 1 -- 172.21.15.16:0/1717022183 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f43e80a1f20 msgr2=0x7f43e80a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:12.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.441+0000 7f44063f7700 1 --2- 172.21.15.16:0/1717022183 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f43e80a1f20 0x7f43e80a43e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4400107a10 tx=0x7f43f400b410 comp rx=0 tx=0).stop 2024-03-28T22:09:12.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.441+0000 7f44063f7700 1 -- 172.21.15.16:0/1717022183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44001013d0 msgr2=0x7f440019e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:12.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.441+0000 7f44063f7700 1 --2- 172.21.15.16:0/1717022183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44001013d0 0x7f440019e6e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f43f0006010 tx=0x7f43f0009f80 comp rx=0 tx=0).stop 2024-03-28T22:09:12.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.441+0000 7f4404994700 1 -- 172.21.15.16:0/1717022183 reap_dead start 2024-03-28T22:09:12.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.441+0000 7f44063f7700 1 -- 172.21.15.16:0/1717022183 shutdown_connections 2024-03-28T22:09:12.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.441+0000 7f44063f7700 1 -- 172.21.15.16:0/1717022183 >> 172.21.15.16:0/1717022183 conn(0x7f4400077190 msgr2=0x7f440006ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:12.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.441+0000 7f44063f7700 1 -- 172.21.15.16:0/1717022183 shutdown_connections 2024-03-28T22:09:12.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:12.441+0000 7f44063f7700 1 -- 172.21.15.16:0/1717022183 wait complete. 2024-03-28T22:09:12.478 INFO:teuthology.orchestra.run.smithi016.stdout:201863462957 2024-03-28T22:09:12.479 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-28T22:09:13.067 INFO:teuthology.orchestra.run.smithi016.stdout:223338299434 2024-03-28T22:09:13.067 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-28T22:09:13.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:13 smithi063 ceph-mon[36249]: pgmap v308: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T22:09:13.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:13 smithi016 ceph-mon[32374]: pgmap v308: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T22:09:13.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:13 smithi053 ceph-mon[35979]: pgmap v308: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T22:09:13.630 INFO:teuthology.orchestra.run.smithi016.stdout:55834574918 2024-03-28T22:09:13.630 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-28T22:09:13.646 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:09:14.121 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:09:14.153 INFO:teuthology.orchestra.run.smithi016.stdout:347892350997 2024-03-28T22:09:14.153 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-28T22:09:14.161 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:09:15.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:15 smithi063 ceph-mon[36249]: pgmap v309: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T22:09:15.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:15 smithi016 ceph-mon[32374]: pgmap v309: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T22:09:15.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:15 smithi053 ceph-mon[35979]: pgmap v309: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T22:09:17.227 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:17 smithi016 ceph-mon[32374]: pgmap v310: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T22:09:17.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:17 smithi063 ceph-mon[36249]: pgmap v310: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T22:09:17.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.334+0000 7fe110ace700 1 -- 172.21.15.16:0/4025261351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe10c06b4f0 msgr2=0x7fe10c106ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:17.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.334+0000 7fe110ace700 1 --2- 172.21.15.16:0/4025261351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe10c06b4f0 0x7fe10c106ec0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe100009990 tx=0x7fe100009ca0 comp rx=0 tx=0).stop 2024-03-28T22:09:17.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.334+0000 7fe110ace700 1 -- 172.21.15.16:0/4025261351 shutdown_connections 2024-03-28T22:09:17.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.334+0000 7fe110ace700 1 --2- 172.21.15.16:0/4025261351 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe10c1075a0 0x7fe10c10f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:17.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.334+0000 7fe110ace700 1 --2- 172.21.15.16:0/4025261351 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe10c06ab40 0x7fe10c06af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:17.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.334+0000 7fe110ace700 1 --2- 172.21.15.16:0/4025261351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe10c06b4f0 0x7fe10c106ec0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:17.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.334+0000 7fe110ace700 1 -- 172.21.15.16:0/4025261351 >> 172.21.15.16:0/4025261351 conn(0x7fe10c078000 msgr2=0x7fe10c078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:17.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.334+0000 7fe110ace700 1 -- 172.21.15.16:0/4025261351 shutdown_connections 2024-03-28T22:09:17.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.334+0000 7fe110ace700 1 -- 172.21.15.16:0/4025261351 wait complete. 2024-03-28T22:09:17.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.335+0000 7fe110ace700 1 Processor -- start 2024-03-28T22:09:17.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.335+0000 7fe110ace700 1 -- start start 2024-03-28T22:09:17.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.336+0000 7fe110ace700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe10c06ab40 0x7fe10c19e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:17.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.336+0000 7fe110ace700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe10c06b4f0 0x7fe10c19eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:17.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.336+0000 7fe110ace700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe10c1075a0 0x7fe10c1987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:17.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.336+0000 7fe110ace700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe10c112520 con 0x7fe10c06ab40 2024-03-28T22:09:17.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.336+0000 7fe110ace700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe10c1123a0 con 0x7fe10c06b4f0 2024-03-28T22:09:17.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.336+0000 7fe110ace700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe10c1126a0 con 0x7fe10c1075a0 2024-03-28T22:09:17.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.336+0000 7fe10a59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe10c06ab40 0x7fe10c19e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:17.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.336+0000 7fe10a59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe10c06ab40 0x7fe10c19e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35850/0 (socket says 172.21.15.16:35850) 2024-03-28T22:09:17.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.336+0000 7fe10a59c700 1 -- 172.21.15.16:0/1519322993 learned_addr learned my addr 172.21.15.16:0/1519322993 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:17.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.336+0000 7fe109d9b700 1 --2- 172.21.15.16:0/1519322993 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe10c06b4f0 0x7fe10c19eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:17.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.336+0000 7fe10a59c700 1 -- 172.21.15.16:0/1519322993 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe10c1075a0 msgr2=0x7fe10c1987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:17.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.336+0000 7fe10a59c700 1 --2- 172.21.15.16:0/1519322993 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe10c1075a0 0x7fe10c1987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:17.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.336+0000 7fe10a59c700 1 -- 172.21.15.16:0/1519322993 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe10c06b4f0 msgr2=0x7fe10c19eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:17.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.336+0000 7fe10a59c700 1 --2- 172.21.15.16:0/1519322993 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe10c06b4f0 0x7fe10c19eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:17.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.336+0000 7fe10a59c700 1 -- 172.21.15.16:0/1519322993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe10c198f80 con 0x7fe10c06ab40 2024-03-28T22:09:17.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.336+0000 7fe109d9b700 1 --2- 172.21.15.16:0/1519322993 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe10c06b4f0 0x7fe10c19eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:09:17.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.336+0000 7fe10a59c700 1 --2- 172.21.15.16:0/1519322993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe10c06ab40 0x7fe10c19e620 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc00e920 tx=0x7fe0fc00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:17.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.337+0000 7fe0fb7fe700 1 -- 172.21.15.16:0/1519322993 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc00cbd0 con 0x7fe10c06ab40 2024-03-28T22:09:17.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.337+0000 7fe110ace700 1 -- 172.21.15.16:0/1519322993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe10c199270 con 0x7fe10c06ab40 2024-03-28T22:09:17.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.337+0000 7fe0fb7fe700 1 -- 172.21.15.16:0/1519322993 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0fc00ee70 con 0x7fe10c06ab40 2024-03-28T22:09:17.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.337+0000 7fe0fb7fe700 1 -- 172.21.15.16:0/1519322993 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc018680 con 0x7fe10c06ab40 2024-03-28T22:09:17.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.337+0000 7fe110ace700 1 -- 172.21.15.16:0/1519322993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe10c1a5480 con 0x7fe10c06ab40 2024-03-28T22:09:17.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.338+0000 7fe110ace700 1 -- 172.21.15.16:0/1519322993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe10c1039a0 con 0x7fe10c06ab40 2024-03-28T22:09:17.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.340+0000 7fe0fb7fe700 1 -- 172.21.15.16:0/1519322993 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fe0fc010430 con 0x7fe10c06ab40 2024-03-28T22:09:17.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.340+0000 7fe0fb7fe700 1 --2- 172.21.15.16:0/1519322993 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fe0f40a1ea0 0x7fe0f40a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:17.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.340+0000 7fe0fb7fe700 1 -- 172.21.15.16:0/1519322993 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fe0fc0ce4a0 con 0x7fe10c06ab40 2024-03-28T22:09:17.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.340+0000 7fe109d9b700 1 --2- 172.21.15.16:0/1519322993 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fe0f40a1ea0 0x7fe0f40a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:17.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.341+0000 7fe109d9b700 1 --2- 172.21.15.16:0/1519322993 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fe0f40a1ea0 0x7fe0f40a4360 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe100006010 tx=0x7fe100005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:17.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.344+0000 7fe0fb7fe700 1 -- 172.21.15.16:0/1519322993 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe0fc09b920 con 0x7fe10c06ab40 2024-03-28T22:09:17.454 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:09:17.457 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:09:17.458 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:09:17.458 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:09:17.462 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:09:17.462 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:09:17.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:17 smithi053 ceph-mon[35979]: pgmap v310: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T22:09:17.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.584+0000 7fe110ace700 1 -- 172.21.15.16:0/1519322993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fe10c02db30 con 0x7fe10c06ab40 2024-03-28T22:09:17.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.584+0000 7fe0fb7fe700 1 -- 172.21.15.16:0/1519322993 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fe0fc00cd70 con 0x7fe10c06ab40 2024-03-28T22:09:17.587 INFO:teuthology.orchestra.run.smithi016.stdout:201863462959 2024-03-28T22:09:17.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.586+0000 7fe110ace700 1 -- 172.21.15.16:0/1519322993 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fe0f40a1ea0 msgr2=0x7fe0f40a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:17.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.586+0000 7fe110ace700 1 --2- 172.21.15.16:0/1519322993 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fe0f40a1ea0 0x7fe0f40a4360 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe100006010 tx=0x7fe100005960 comp rx=0 tx=0).stop 2024-03-28T22:09:17.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.586+0000 7fe110ace700 1 -- 172.21.15.16:0/1519322993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe10c06ab40 msgr2=0x7fe10c19e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:17.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.586+0000 7fe110ace700 1 --2- 172.21.15.16:0/1519322993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe10c06ab40 0x7fe10c19e620 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc00e920 tx=0x7fe0fc00ec30 comp rx=0 tx=0).stop 2024-03-28T22:09:17.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.586+0000 7fe110ace700 1 -- 172.21.15.16:0/1519322993 shutdown_connections 2024-03-28T22:09:17.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.586+0000 7fe110ace700 1 --2- 172.21.15.16:0/1519322993 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fe0f40a1ea0 0x7fe0f40a4360 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:17.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.586+0000 7fe110ace700 1 --2- 172.21.15.16:0/1519322993 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe10c1075a0 0x7fe10c1987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:17.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.586+0000 7fe110ace700 1 --2- 172.21.15.16:0/1519322993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe10c06ab40 0x7fe10c19e620 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:17.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.586+0000 7fe110ace700 1 --2- 172.21.15.16:0/1519322993 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe10c06b4f0 0x7fe10c19eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:17.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.586+0000 7fe110ace700 1 -- 172.21.15.16:0/1519322993 >> 172.21.15.16:0/1519322993 conn(0x7fe10c078000 msgr2=0x7fe10c0ff3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:17.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.586+0000 7fe110ace700 1 -- 172.21.15.16:0/1519322993 shutdown_connections 2024-03-28T22:09:17.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:17.586+0000 7fe110ace700 1 -- 172.21.15.16:0/1519322993 wait complete. 2024-03-28T22:09:18.247 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:09:18.247 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:09:18.343 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:18 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/1519322993' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-28T22:09:18.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:18 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/1519322993' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-28T22:09:18.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:18 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/1519322993' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-28T22:09:18.961 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462957 got 201863462959 for osd.6 2024-03-28T22:09:18.961 DEBUG:teuthology.parallel:result is None 2024-03-28T22:09:19.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:19 smithi016 ceph-mon[32374]: pgmap v311: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:09:19.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:19 smithi063 ceph-mon[36249]: pgmap v311: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:09:19.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:19 smithi053 ceph-mon[35979]: pgmap v311: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:09:20.582 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:09:20.775 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:20 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/415569952' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-28T22:09:20.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:20 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/415569952' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-28T22:09:20.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.235+0000 7f0f96ad3700 1 -- 172.21.15.16:0/4065722657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f90102e50 msgr2=0x7f0f90106ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:20.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.235+0000 7f0f96ad3700 1 --2- 172.21.15.16:0/4065722657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f90102e50 0x7f0f90106ea0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0f840099c0 tx=0x7f0f84009cd0 comp rx=0 tx=0).stop 2024-03-28T22:09:20.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.235+0000 7f0f96ad3700 1 -- 172.21.15.16:0/4065722657 shutdown_connections 2024-03-28T22:09:20.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.235+0000 7f0f96ad3700 1 --2- 172.21.15.16:0/4065722657 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f90107580 0x7f0f9010f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:20.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.235+0000 7f0f96ad3700 1 --2- 172.21.15.16:0/4065722657 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f901024a0 0x7f0f90102880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:20.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.235+0000 7f0f96ad3700 1 --2- 172.21.15.16:0/4065722657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f90102e50 0x7f0f90106ea0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:20.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.235+0000 7f0f96ad3700 1 -- 172.21.15.16:0/4065722657 >> 172.21.15.16:0/4065722657 conn(0x7f0f900fdd10 msgr2=0x7f0f90100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:20.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.235+0000 7f0f96ad3700 1 -- 172.21.15.16:0/4065722657 shutdown_connections 2024-03-28T22:09:20.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.235+0000 7f0f96ad3700 1 -- 172.21.15.16:0/4065722657 wait complete. 2024-03-28T22:09:20.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.235+0000 7f0f96ad3700 1 Processor -- start 2024-03-28T22:09:20.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.236+0000 7f0f96ad3700 1 -- start start 2024-03-28T22:09:20.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.236+0000 7f0f96ad3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f901024a0 0x7f0f9019e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:20.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.236+0000 7f0f96ad3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f90102e50 0x7f0f9019ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:20.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.236+0000 7f0f96ad3700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f90107580 0x7f0f901988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:20.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.236+0000 7f0f96ad3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0f901125c0 con 0x7f0f901024a0 2024-03-28T22:09:20.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.236+0000 7f0f96ad3700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0f90112440 con 0x7f0f90102e50 2024-03-28T22:09:20.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.236+0000 7f0f96ad3700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0f90112740 con 0x7f0f90107580 2024-03-28T22:09:20.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.236+0000 7f0f9486f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f901024a0 0x7f0f9019e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:20.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.236+0000 7f0f9486f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f901024a0 0x7f0f9019e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35870/0 (socket says 172.21.15.16:35870) 2024-03-28T22:09:20.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.236+0000 7f0f9486f700 1 -- 172.21.15.16:0/415569952 learned_addr learned my addr 172.21.15.16:0/415569952 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:20.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.236+0000 7f0f8ffff700 1 --2- 172.21.15.16:0/415569952 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f90102e50 0x7f0f9019ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:20.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.236+0000 7f0f95070700 1 --2- 172.21.15.16:0/415569952 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f90107580 0x7f0f901988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:20.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.236+0000 7f0f9486f700 1 -- 172.21.15.16:0/415569952 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f90107580 msgr2=0x7f0f901988a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:20.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.236+0000 7f0f9486f700 1 --2- 172.21.15.16:0/415569952 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f90107580 0x7f0f901988a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:20.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.236+0000 7f0f9486f700 1 -- 172.21.15.16:0/415569952 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f90102e50 msgr2=0x7f0f9019ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:20.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.236+0000 7f0f9486f700 1 --2- 172.21.15.16:0/415569952 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f90102e50 0x7f0f9019ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:20.847 INFO:teuthology.orchestra.run.smithi016.stdout:34359738444 2024-03-28T22:09:20.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.236+0000 7f0f9486f700 1 -- 172.21.15.16:0/415569952 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f90199000 con 0x7f0f901024a0 2024-03-28T22:09:20.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.236+0000 7f0f9486f700 1 --2- 172.21.15.16:0/415569952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f901024a0 0x7f0f9019e6e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0f8000b530 tx=0x7f0f8000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:20.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.236+0000 7f0f8dffb700 1 -- 172.21.15.16:0/415569952 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f80015020 con 0x7f0f901024a0 2024-03-28T22:09:20.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.236+0000 7f0f8dffb700 1 -- 172.21.15.16:0/415569952 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f80011d10 con 0x7f0f901024a0 2024-03-28T22:09:20.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.236+0000 7f0f96ad3700 1 -- 172.21.15.16:0/415569952 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f901992f0 con 0x7f0f901024a0 2024-03-28T22:09:20.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.236+0000 7f0f8dffb700 1 -- 172.21.15.16:0/415569952 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f8000f2e0 con 0x7f0f901024a0 2024-03-28T22:09:20.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.237+0000 7f0f96ad3700 1 -- 172.21.15.16:0/415569952 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f901a55b0 con 0x7f0f901024a0 2024-03-28T22:09:20.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.238+0000 7f0f96ad3700 1 -- 172.21.15.16:0/415569952 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f90068f00 con 0x7f0f901024a0 2024-03-28T22:09:20.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.239+0000 7f0f8dffb700 1 -- 172.21.15.16:0/415569952 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f0f80018020 con 0x7f0f901024a0 2024-03-28T22:09:20.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.239+0000 7f0f8dffb700 1 --2- 172.21.15.16:0/415569952 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0f780a1ea0 0x7f0f780a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:20.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.239+0000 7f0f8ffff700 1 --2- 172.21.15.16:0/415569952 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0f780a1ea0 0x7f0f780a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:20.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.239+0000 7f0f8dffb700 1 -- 172.21.15.16:0/415569952 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f0f800cd550 con 0x7f0f901024a0 2024-03-28T22:09:20.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.240+0000 7f0f8ffff700 1 --2- 172.21.15.16:0/415569952 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0f780a1ea0 0x7f0f780a4360 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0f84000c00 tx=0x7f0f84005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:20.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.244+0000 7f0f8dffb700 1 -- 172.21.15.16:0/415569952 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0f8001f020 con 0x7f0f901024a0 2024-03-28T22:09:20.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.453+0000 7f0f96ad3700 1 -- 172.21.15.16:0/415569952 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f0f90199e00 con 0x7f0f901024a0 2024-03-28T22:09:20.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.453+0000 7f0f8dffb700 1 -- 172.21.15.16:0/415569952 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0f90199e00 con 0x7f0f901024a0 2024-03-28T22:09:20.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.455+0000 7f0f96ad3700 1 -- 172.21.15.16:0/415569952 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0f780a1ea0 msgr2=0x7f0f780a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:20.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.455+0000 7f0f96ad3700 1 --2- 172.21.15.16:0/415569952 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0f780a1ea0 0x7f0f780a4360 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0f84000c00 tx=0x7f0f84005fb0 comp rx=0 tx=0).stop 2024-03-28T22:09:20.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.455+0000 7f0f96ad3700 1 -- 172.21.15.16:0/415569952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f901024a0 msgr2=0x7f0f9019e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:20.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.455+0000 7f0f96ad3700 1 --2- 172.21.15.16:0/415569952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f901024a0 0x7f0f9019e6e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0f8000b530 tx=0x7f0f8000b840 comp rx=0 tx=0).stop 2024-03-28T22:09:20.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.455+0000 7f0f96ad3700 1 -- 172.21.15.16:0/415569952 shutdown_connections 2024-03-28T22:09:20.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.455+0000 7f0f96ad3700 1 --2- 172.21.15.16:0/415569952 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0f780a1ea0 0x7f0f780a4360 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:20.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.455+0000 7f0f96ad3700 1 --2- 172.21.15.16:0/415569952 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f90107580 0x7f0f901988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:20.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.455+0000 7f0f96ad3700 1 --2- 172.21.15.16:0/415569952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f901024a0 0x7f0f9019e6e0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:20.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.455+0000 7f0f96ad3700 1 --2- 172.21.15.16:0/415569952 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f90102e50 0x7f0f9019ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:20.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.455+0000 7f0f96ad3700 1 -- 172.21.15.16:0/415569952 >> 172.21.15.16:0/415569952 conn(0x7f0f900fdd10 msgr2=0x7f0f90100020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:20.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.455+0000 7f0f96ad3700 1 -- 172.21.15.16:0/415569952 shutdown_connections 2024-03-28T22:09:20.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:20.455+0000 7f0f96ad3700 1 -- 172.21.15.16:0/415569952 wait complete. 2024-03-28T22:09:20.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:20 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/415569952' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-28T22:09:21.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:21 smithi053 ceph-mon[35979]: pgmap v312: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:09:21.925 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:21 smithi016 ceph-mon[32374]: pgmap v312: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:09:21.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:21 smithi063 ceph-mon[36249]: pgmap v312: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:09:23.395 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:23 smithi016 ceph-mon[32374]: pgmap v313: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:09:23.432 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738440 got 34359738444 for osd.0 2024-03-28T22:09:23.432 DEBUG:teuthology.parallel:result is None 2024-03-28T22:09:23.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:23 smithi063 ceph-mon[36249]: pgmap v313: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:09:23.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:23 smithi053 ceph-mon[35979]: pgmap v313: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:09:25.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:25 smithi016 ceph-mon[32374]: pgmap v314: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T22:09:25.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:25 smithi063 ceph-mon[36249]: pgmap v314: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T22:09:25.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:25 smithi053 ceph-mon[35979]: pgmap v314: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T22:09:26.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.759+0000 7f616bf29700 1 -- 172.21.15.16:0/151042760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61641011f0 msgr2=0x7f616410f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:26.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.759+0000 7f616bf29700 1 --2- 172.21.15.16:0/151042760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61641011f0 0x7f616410f150 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f615c009990 tx=0x7f615c009ca0 comp rx=0 tx=0).stop 2024-03-28T22:09:26.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.759+0000 7f616bf29700 1 -- 172.21.15.16:0/151042760 shutdown_connections 2024-03-28T22:09:26.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.759+0000 7f616bf29700 1 --2- 172.21.15.16:0/151042760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61641011f0 0x7f616410f150 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:26.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.759+0000 7f616bf29700 1 --2- 172.21.15.16:0/151042760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f61641072b0 0x7f6164107690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:26.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.759+0000 7f616bf29700 1 --2- 172.21.15.16:0/151042760 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6164100830 0x7f6164100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:26.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.759+0000 7f616bf29700 1 -- 172.21.15.16:0/151042760 >> 172.21.15.16:0/151042760 conn(0x7f6164077740 msgr2=0x7f61640feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:26.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.760+0000 7f616bf29700 1 -- 172.21.15.16:0/151042760 shutdown_connections 2024-03-28T22:09:26.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.760+0000 7f616bf29700 1 -- 172.21.15.16:0/151042760 wait complete. 2024-03-28T22:09:26.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.760+0000 7f616bf29700 1 Processor -- start 2024-03-28T22:09:26.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.760+0000 7f616bf29700 1 -- start start 2024-03-28T22:09:26.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.760+0000 7f616bf29700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6164100830 0x7f6164101a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:26.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.760+0000 7f616bf29700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61641011f0 0x7f6164101f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:26.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.761+0000 7f616bf29700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61641072b0 0x7f6164102490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:26.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.761+0000 7f616bf29700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6164112670 con 0x7f61641011f0 2024-03-28T22:09:26.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.761+0000 7f616bf29700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f61641124f0 con 0x7f6164100830 2024-03-28T22:09:26.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.761+0000 7f616bf29700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f61641127f0 con 0x7f61641072b0 2024-03-28T22:09:26.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.761+0000 7f61694c4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61641011f0 0x7f6164101f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:26.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.761+0000 7f6169cc5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6164100830 0x7f6164101a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:26.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.761+0000 7f6169cc5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6164100830 0x7f6164101a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:60448/0 (socket says 172.21.15.16:60448) 2024-03-28T22:09:26.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.761+0000 7f61694c4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61641011f0 0x7f6164101f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53318/0 (socket says 172.21.15.16:53318) 2024-03-28T22:09:26.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.761+0000 7f6169cc5700 1 -- 172.21.15.16:0/1099850703 learned_addr learned my addr 172.21.15.16:0/1099850703 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:26.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.761+0000 7f616a4c6700 1 --2- 172.21.15.16:0/1099850703 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61641072b0 0x7f6164102490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:26.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.761+0000 7f6169cc5700 1 -- 172.21.15.16:0/1099850703 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61641072b0 msgr2=0x7f6164102490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:26.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.761+0000 7f6169cc5700 1 --2- 172.21.15.16:0/1099850703 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61641072b0 0x7f6164102490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:26.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.761+0000 7f6169cc5700 1 -- 172.21.15.16:0/1099850703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61641011f0 msgr2=0x7f6164101f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:26.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.761+0000 7f6169cc5700 1 --2- 172.21.15.16:0/1099850703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61641011f0 0x7f6164101f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:26.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.761+0000 7f6169cc5700 1 -- 172.21.15.16:0/1099850703 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61641a7020 con 0x7f6164100830 2024-03-28T22:09:26.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.761+0000 7f616a4c6700 1 --2- 172.21.15.16:0/1099850703 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61641072b0 0x7f6164102490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:09:26.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.761+0000 7f61694c4700 1 --2- 172.21.15.16:0/1099850703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61641011f0 0x7f6164101f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:09:26.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.761+0000 7f6169cc5700 1 --2- 172.21.15.16:0/1099850703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6164100830 0x7f6164101a10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f615400ef10 tx=0x7f615400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:26.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.761+0000 7f6162ffd700 1 -- 172.21.15.16:0/1099850703 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f615400ca70 con 0x7f6164100830 2024-03-28T22:09:26.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.762+0000 7f616bf29700 1 -- 172.21.15.16:0/1099850703 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61641a72b0 con 0x7f6164100830 2024-03-28T22:09:26.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.762+0000 7f6162ffd700 1 -- 172.21.15.16:0/1099850703 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6154019070 con 0x7f6164100830 2024-03-28T22:09:26.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.762+0000 7f6162ffd700 1 -- 172.21.15.16:0/1099850703 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61540146e0 con 0x7f6164100830 2024-03-28T22:09:26.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.762+0000 7f616bf29700 1 -- 172.21.15.16:0/1099850703 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61641a77f0 con 0x7f6164100830 2024-03-28T22:09:26.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.762+0000 7f616bf29700 1 -- 172.21.15.16:0/1099850703 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6164068f00 con 0x7f6164100830 2024-03-28T22:09:26.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.764+0000 7f6162ffd700 1 -- 172.21.15.16:0/1099850703 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f6154012050 con 0x7f6164100830 2024-03-28T22:09:26.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.765+0000 7f6162ffd700 1 --2- 172.21.15.16:0/1099850703 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f614c0a1ea0 0x7f614c0a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:26.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.765+0000 7f6162ffd700 1 -- 172.21.15.16:0/1099850703 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f61540cdd70 con 0x7f6164100830 2024-03-28T22:09:26.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.765+0000 7f61694c4700 1 --2- 172.21.15.16:0/1099850703 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f614c0a1ea0 0x7f614c0a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:26.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.765+0000 7f61694c4700 1 --2- 172.21.15.16:0/1099850703 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f614c0a1ea0 0x7f614c0a4360 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6164106b10 tx=0x7f615800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:26.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.768+0000 7f6162ffd700 1 -- 172.21.15.16:0/1099850703 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6154010040 con 0x7f6164100830 2024-03-28T22:09:26.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.983+0000 7f616bf29700 1 -- 172.21.15.16:0/1099850703 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f6164103b30 con 0x7f6164100830 2024-03-28T22:09:26.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.983+0000 7f6162ffd700 1 -- 172.21.15.16:0/1099850703 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f615401b310 con 0x7f6164100830 2024-03-28T22:09:26.987 INFO:teuthology.orchestra.run.smithi016.stdout:111669149759 2024-03-28T22:09:26.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.986+0000 7f616bf29700 1 -- 172.21.15.16:0/1099850703 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f614c0a1ea0 msgr2=0x7f614c0a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:26.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.986+0000 7f616bf29700 1 --2- 172.21.15.16:0/1099850703 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f614c0a1ea0 0x7f614c0a4360 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6164106b10 tx=0x7f615800a300 comp rx=0 tx=0).stop 2024-03-28T22:09:26.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.986+0000 7f616bf29700 1 -- 172.21.15.16:0/1099850703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6164100830 msgr2=0x7f6164101a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:26.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.986+0000 7f616bf29700 1 --2- 172.21.15.16:0/1099850703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6164100830 0x7f6164101a10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f615400ef10 tx=0x7f615400c3a0 comp rx=0 tx=0).stop 2024-03-28T22:09:26.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.986+0000 7f616bf29700 1 -- 172.21.15.16:0/1099850703 shutdown_connections 2024-03-28T22:09:26.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.986+0000 7f616bf29700 1 --2- 172.21.15.16:0/1099850703 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f614c0a1ea0 0x7f614c0a4360 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:26.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.986+0000 7f616bf29700 1 --2- 172.21.15.16:0/1099850703 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61641072b0 0x7f6164102490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:26.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.986+0000 7f616bf29700 1 --2- 172.21.15.16:0/1099850703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6164100830 0x7f6164101a10 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:26.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.986+0000 7f616bf29700 1 --2- 172.21.15.16:0/1099850703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61641011f0 0x7f6164101f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:26.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.986+0000 7f616bf29700 1 -- 172.21.15.16:0/1099850703 >> 172.21.15.16:0/1099850703 conn(0x7f6164077740 msgr2=0x7f61641051c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:26.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.986+0000 7f616bf29700 1 -- 172.21.15.16:0/1099850703 shutdown_connections 2024-03-28T22:09:26.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:26.986+0000 7f616bf29700 1 -- 172.21.15.16:0/1099850703 wait complete. 2024-03-28T22:09:27.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:27 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/1099850703' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-28T22:09:27.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:27 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/1099850703' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-28T22:09:27.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:27 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/1099850703' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-28T22:09:28.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.008+0000 7ff268751700 1 -- 172.21.15.16:0/3393660827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff260101ae0 msgr2=0x7ff26010df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:28.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.008+0000 7ff268751700 1 --2- 172.21.15.16:0/3393660827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff260101ae0 0x7ff26010df80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff25800b0a0 tx=0x7ff25800b3b0 comp rx=0 tx=0).stop 2024-03-28T22:09:28.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.009+0000 7ff268751700 1 -- 172.21.15.16:0/3393660827 shutdown_connections 2024-03-28T22:09:28.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.009+0000 7ff268751700 1 --2- 172.21.15.16:0/3393660827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff260101ae0 0x7ff26010df80 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:28.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.009+0000 7ff268751700 1 --2- 172.21.15.16:0/3393660827 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2601071d0 0x7ff2601075b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:28.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.009+0000 7ff268751700 1 --2- 172.21.15.16:0/3393660827 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff260101120 0x7ff2601015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:28.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.009+0000 7ff268751700 1 -- 172.21.15.16:0/3393660827 >> 172.21.15.16:0/3393660827 conn(0x7ff260078150 msgr2=0x7ff260078560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:28.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.009+0000 7ff268751700 1 -- 172.21.15.16:0/3393660827 shutdown_connections 2024-03-28T22:09:28.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.009+0000 7ff268751700 1 -- 172.21.15.16:0/3393660827 wait complete. 2024-03-28T22:09:28.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.009+0000 7ff268751700 1 Processor -- start 2024-03-28T22:09:28.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.009+0000 7ff268751700 1 -- start start 2024-03-28T22:09:28.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.009+0000 7ff268751700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff260101120 0x7ff260199740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:28.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.009+0000 7ff268751700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff260101ae0 0x7ff260199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:28.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.010+0000 7ff268751700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2601071d0 0x7ff2601a47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:28.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.010+0000 7ff268751700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff2601113e0 con 0x7ff2601071d0 2024-03-28T22:09:28.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.010+0000 7ff268751700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff260111260 con 0x7ff260101ae0 2024-03-28T22:09:28.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.010+0000 7ff268751700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff260111560 con 0x7ff260101120 2024-03-28T22:09:28.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.010+0000 7ff266cee700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2601071d0 0x7ff2601a47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:28.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.010+0000 7ff266cee700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2601071d0 0x7ff2601a47c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53342/0 (socket says 172.21.15.16:53342) 2024-03-28T22:09:28.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.010+0000 7ff266cee700 1 -- 172.21.15.16:0/2731423290 learned_addr learned my addr 172.21.15.16:0/2731423290 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:28.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.010+0000 7ff265cec700 1 --2- 172.21.15.16:0/2731423290 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff260101ae0 0x7ff260199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:28.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.010+0000 7ff2664ed700 1 --2- 172.21.15.16:0/2731423290 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff260101120 0x7ff260199740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:28.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.010+0000 7ff266cee700 1 -- 172.21.15.16:0/2731423290 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff260101120 msgr2=0x7ff260199740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:28.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.010+0000 7ff266cee700 1 --2- 172.21.15.16:0/2731423290 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff260101120 0x7ff260199740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:28.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.010+0000 7ff266cee700 1 -- 172.21.15.16:0/2731423290 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff260101ae0 msgr2=0x7ff260199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:28.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.010+0000 7ff266cee700 1 --2- 172.21.15.16:0/2731423290 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff260101ae0 0x7ff260199c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:28.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.010+0000 7ff266cee700 1 -- 172.21.15.16:0/2731423290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2601a4d00 con 0x7ff2601071d0 2024-03-28T22:09:28.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.010+0000 7ff266cee700 1 --2- 172.21.15.16:0/2731423290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2601071d0 0x7ff2601a47c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff25800b780 tx=0x7ff258004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:28.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.010+0000 7ff2537fe700 1 -- 172.21.15.16:0/2731423290 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff258021070 con 0x7ff2601071d0 2024-03-28T22:09:28.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.010+0000 7ff2537fe700 1 -- 172.21.15.16:0/2731423290 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff258007b80 con 0x7ff2601071d0 2024-03-28T22:09:28.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.010+0000 7ff268751700 1 -- 172.21.15.16:0/2731423290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2601a4ed0 con 0x7ff2601071d0 2024-03-28T22:09:28.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.010+0000 7ff2537fe700 1 -- 172.21.15.16:0/2731423290 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff258005370 con 0x7ff2601071d0 2024-03-28T22:09:28.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.010+0000 7ff268751700 1 -- 172.21.15.16:0/2731423290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2601a53e0 con 0x7ff2601071d0 2024-03-28T22:09:28.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.011+0000 7ff268751700 1 -- 172.21.15.16:0/2731423290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff260109520 con 0x7ff2601071d0 2024-03-28T22:09:28.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.013+0000 7ff2537fe700 1 -- 172.21.15.16:0/2731423290 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7ff258024070 con 0x7ff2601071d0 2024-03-28T22:09:28.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.013+0000 7ff2537fe700 1 --2- 172.21.15.16:0/2731423290 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff24c0a1f20 0x7ff24c0a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:28.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.013+0000 7ff2664ed700 1 --2- 172.21.15.16:0/2731423290 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff24c0a1f20 0x7ff24c0a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:28.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.014+0000 7ff2537fe700 1 -- 172.21.15.16:0/2731423290 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7ff2580ced90 con 0x7ff2601071d0 2024-03-28T22:09:28.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.014+0000 7ff2664ed700 1 --2- 172.21.15.16:0/2731423290 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff24c0a1f20 0x7ff24c0a43e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff25c005fd0 tx=0x7ff25c00af20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:28.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.018+0000 7ff2537fe700 1 -- 172.21.15.16:0/2731423290 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff25809c2a0 con 0x7ff2601071d0 2024-03-28T22:09:28.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:28 smithi016 ceph-mon[32374]: pgmap v315: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T22:09:28.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.224+0000 7ff268751700 1 -- 172.21.15.16:0/2731423290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7ff260108f00 con 0x7ff2601071d0 2024-03-28T22:09:28.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.224+0000 7ff2537fe700 1 -- 172.21.15.16:0/2731423290 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff25809bba0 con 0x7ff2601071d0 2024-03-28T22:09:28.227 INFO:teuthology.orchestra.run.smithi016.stdout:257698037799 2024-03-28T22:09:28.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.226+0000 7ff268751700 1 -- 172.21.15.16:0/2731423290 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff24c0a1f20 msgr2=0x7ff24c0a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:28.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.226+0000 7ff268751700 1 --2- 172.21.15.16:0/2731423290 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff24c0a1f20 0x7ff24c0a43e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff25c005fd0 tx=0x7ff25c00af20 comp rx=0 tx=0).stop 2024-03-28T22:09:28.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.226+0000 7ff268751700 1 -- 172.21.15.16:0/2731423290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2601071d0 msgr2=0x7ff2601a47c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:28.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.226+0000 7ff268751700 1 --2- 172.21.15.16:0/2731423290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2601071d0 0x7ff2601a47c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff25800b780 tx=0x7ff258004d00 comp rx=0 tx=0).stop 2024-03-28T22:09:28.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.226+0000 7ff268751700 1 -- 172.21.15.16:0/2731423290 shutdown_connections 2024-03-28T22:09:28.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.226+0000 7ff268751700 1 --2- 172.21.15.16:0/2731423290 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff24c0a1f20 0x7ff24c0a43e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:28.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.226+0000 7ff268751700 1 --2- 172.21.15.16:0/2731423290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2601071d0 0x7ff2601a47c0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:28.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.226+0000 7ff268751700 1 --2- 172.21.15.16:0/2731423290 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff260101120 0x7ff260199740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:28.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.226+0000 7ff268751700 1 --2- 172.21.15.16:0/2731423290 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff260101ae0 0x7ff260199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:28.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.226+0000 7ff268751700 1 -- 172.21.15.16:0/2731423290 >> 172.21.15.16:0/2731423290 conn(0x7ff260078150 msgr2=0x7ff2600ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:28.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.226+0000 7ff268751700 1 -- 172.21.15.16:0/2731423290 shutdown_connections 2024-03-28T22:09:28.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:28.226+0000 7ff268751700 1 -- 172.21.15.16:0/2731423290 wait complete. 2024-03-28T22:09:28.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:28 smithi053 ceph-mon[35979]: pgmap v315: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T22:09:28.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:28 smithi063 ceph-mon[36249]: pgmap v315: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T22:09:29.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.081+0000 7f3efb5cc700 1 -- 172.21.15.16:0/1890452592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ef41013d0 msgr2=0x7f3ef41017b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:29.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.081+0000 7f3efb5cc700 1 --2- 172.21.15.16:0/1890452592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ef41013d0 0x7f3ef41017b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f3ef000b0a0 tx=0x7f3ef000b3b0 comp rx=0 tx=0).stop 2024-03-28T22:09:29.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.081+0000 7f3efb5cc700 1 -- 172.21.15.16:0/1890452592 shutdown_connections 2024-03-28T22:09:29.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.081+0000 7f3efb5cc700 1 --2- 172.21.15.16:0/1890452592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ef4106460 0x7f3ef410e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:29.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.081+0000 7f3efb5cc700 1 --2- 172.21.15.16:0/1890452592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ef41013d0 0x7f3ef41017b0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:29.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.081+0000 7f3efb5cc700 1 --2- 172.21.15.16:0/1890452592 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ef4101d80 0x7f3ef4105d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:29.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.081+0000 7f3efb5cc700 1 -- 172.21.15.16:0/1890452592 >> 172.21.15.16:0/1890452592 conn(0x7f3ef4077160 msgr2=0x7f3ef4077570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:29.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.081+0000 7f3efb5cc700 1 -- 172.21.15.16:0/1890452592 shutdown_connections 2024-03-28T22:09:29.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.081+0000 7f3efb5cc700 1 -- 172.21.15.16:0/1890452592 wait complete. 2024-03-28T22:09:29.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.082+0000 7f3efb5cc700 1 Processor -- start 2024-03-28T22:09:29.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.082+0000 7f3efb5cc700 1 -- start start 2024-03-28T22:09:29.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.082+0000 7f3efb5cc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ef41013d0 0x7f3ef4199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:29.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.082+0000 7f3efb5cc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ef4101d80 0x7f3ef4199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:29.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.082+0000 7f3efb5cc700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ef4106460 0x7f3ef41a4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:29.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.082+0000 7f3efb5cc700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef4111440 con 0x7f3ef41013d0 2024-03-28T22:09:29.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.082+0000 7f3efb5cc700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef41112c0 con 0x7f3ef4101d80 2024-03-28T22:09:29.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.082+0000 7f3efb5cc700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef41115c0 con 0x7f3ef4106460 2024-03-28T22:09:29.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.082+0000 7f3ef9368700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ef41013d0 0x7f3ef4199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:29.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.082+0000 7f3ef9368700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ef41013d0 0x7f3ef4199880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53352/0 (socket says 172.21.15.16:53352) 2024-03-28T22:09:29.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.083+0000 7f3ef9368700 1 -- 172.21.15.16:0/3400236368 learned_addr learned my addr 172.21.15.16:0/3400236368 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:29.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.083+0000 7f3ef8b67700 1 --2- 172.21.15.16:0/3400236368 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ef4101d80 0x7f3ef4199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:29.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.083+0000 7f3ef9b69700 1 --2- 172.21.15.16:0/3400236368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ef4106460 0x7f3ef41a4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:29.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.083+0000 7f3ef9368700 1 -- 172.21.15.16:0/3400236368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ef4106460 msgr2=0x7f3ef41a4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:29.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.083+0000 7f3ef9368700 1 --2- 172.21.15.16:0/3400236368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ef4106460 0x7f3ef41a4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:29.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.083+0000 7f3ef9368700 1 -- 172.21.15.16:0/3400236368 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ef4101d80 msgr2=0x7f3ef4199dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:29.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.083+0000 7f3ef9368700 1 --2- 172.21.15.16:0/3400236368 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ef4101d80 0x7f3ef4199dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:29.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.083+0000 7f3ef9368700 1 -- 172.21.15.16:0/3400236368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ef41a4e40 con 0x7f3ef41013d0 2024-03-28T22:09:29.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.083+0000 7f3ef9368700 1 --2- 172.21.15.16:0/3400236368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ef41013d0 0x7f3ef4199880 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3ef000b070 tx=0x7f3ef0004e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:29.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.083+0000 7f3eea7fc700 1 -- 172.21.15.16:0/3400236368 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef0021070 con 0x7f3ef41013d0 2024-03-28T22:09:29.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.083+0000 7f3eea7fc700 1 -- 172.21.15.16:0/3400236368 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ef0007c80 con 0x7f3ef41013d0 2024-03-28T22:09:29.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.083+0000 7f3efb5cc700 1 -- 172.21.15.16:0/3400236368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ef41a5010 con 0x7f3ef41013d0 2024-03-28T22:09:29.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.083+0000 7f3eea7fc700 1 -- 172.21.15.16:0/3400236368 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef0005390 con 0x7f3ef41013d0 2024-03-28T22:09:29.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.083+0000 7f3efb5cc700 1 -- 172.21.15.16:0/3400236368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ef41a5510 con 0x7f3ef41013d0 2024-03-28T22:09:29.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.084+0000 7f3efb5cc700 1 -- 172.21.15.16:0/3400236368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ef4102860 con 0x7f3ef41013d0 2024-03-28T22:09:29.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.085+0000 7f3eea7fc700 1 -- 172.21.15.16:0/3400236368 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f3ef0024070 con 0x7f3ef41013d0 2024-03-28T22:09:29.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.086+0000 7f3eea7fc700 1 --2- 172.21.15.16:0/3400236368 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f3ee00a1f20 0x7f3ee00a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:29.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.086+0000 7f3ef8b67700 1 --2- 172.21.15.16:0/3400236368 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f3ee00a1f20 0x7f3ee00a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:29.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.086+0000 7f3eea7fc700 1 -- 172.21.15.16:0/3400236368 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7f3ef000f020 con 0x7f3ef41013d0 2024-03-28T22:09:29.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.086+0000 7f3ef8b67700 1 --2- 172.21.15.16:0/3400236368 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f3ee00a1f20 0x7f3ee00a43e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3ef4105ee0 tx=0x7f3ee400b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:29.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.091+0000 7f3eea7fc700 1 -- 172.21.15.16:0/3400236368 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3ef002b020 con 0x7f3ef41013d0 2024-03-28T22:09:29.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:29 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2731423290' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-28T22:09:29.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:29 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-03-28T22:09:29.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:29 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-28T22:09:29.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:29 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-03-28T22:09:29.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:29 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2024-03-28T22:09:29.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:29 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-03-28T22:09:29.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:29 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2024-03-28T22:09:29.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:29 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]: dispatch 2024-03-28T22:09:29.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.310+0000 7f3efb5cc700 1 -- 172.21.15.16:0/3400236368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f3ef402db30 con 0x7f3ef41013d0 2024-03-28T22:09:29.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.311+0000 7f3eea7fc700 1 -- 172.21.15.16:0/3400236368 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3ef009c940 con 0x7f3ef41013d0 2024-03-28T22:09:29.315 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:29 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2731423290' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-28T22:09:29.315 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:29 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-03-28T22:09:29.315 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:29 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-28T22:09:29.315 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:29 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-03-28T22:09:29.316 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:29 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2024-03-28T22:09:29.316 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:29 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-03-28T22:09:29.316 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:29 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2024-03-28T22:09:29.316 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:29 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]: dispatch 2024-03-28T22:09:29.317 INFO:teuthology.orchestra.run.smithi016.stdout:77309411397 2024-03-28T22:09:29.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.314+0000 7f3efb5cc700 1 -- 172.21.15.16:0/3400236368 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f3ee00a1f20 msgr2=0x7f3ee00a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:29.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.314+0000 7f3efb5cc700 1 --2- 172.21.15.16:0/3400236368 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f3ee00a1f20 0x7f3ee00a43e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3ef4105ee0 tx=0x7f3ee400b3c0 comp rx=0 tx=0).stop 2024-03-28T22:09:29.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.314+0000 7f3efb5cc700 1 -- 172.21.15.16:0/3400236368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ef41013d0 msgr2=0x7f3ef4199880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:29.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.314+0000 7f3efb5cc700 1 --2- 172.21.15.16:0/3400236368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ef41013d0 0x7f3ef4199880 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3ef000b070 tx=0x7f3ef0004e40 comp rx=0 tx=0).stop 2024-03-28T22:09:29.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.314+0000 7f3efb5cc700 1 -- 172.21.15.16:0/3400236368 shutdown_connections 2024-03-28T22:09:29.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.314+0000 7f3efb5cc700 1 --2- 172.21.15.16:0/3400236368 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f3ee00a1f20 0x7f3ee00a43e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:29.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.314+0000 7f3efb5cc700 1 --2- 172.21.15.16:0/3400236368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ef4106460 0x7f3ef41a4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:29.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.314+0000 7f3efb5cc700 1 --2- 172.21.15.16:0/3400236368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ef41013d0 0x7f3ef4199880 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:29.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.314+0000 7f3efb5cc700 1 --2- 172.21.15.16:0/3400236368 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ef4101d80 0x7f3ef4199dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:29.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.314+0000 7f3efb5cc700 1 -- 172.21.15.16:0/3400236368 >> 172.21.15.16:0/3400236368 conn(0x7f3ef4077160 msgr2=0x7f3ef4100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:29.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.314+0000 7f3efb5cc700 1 -- 172.21.15.16:0/3400236368 shutdown_connections 2024-03-28T22:09:29.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:29.314+0000 7f3efb5cc700 1 -- 172.21.15.16:0/3400236368 wait complete. 2024-03-28T22:09:29.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:29 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2731423290' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-28T22:09:29.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:29 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-03-28T22:09:29.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:29 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-28T22:09:29.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:29 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-03-28T22:09:29.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:29 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2024-03-28T22:09:29.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:29 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-03-28T22:09:29.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:29 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2024-03-28T22:09:29.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:29 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]: dispatch 2024-03-28T22:09:30.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:30 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-03-28T22:09:30.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:30 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-28T22:09:30.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:30 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-03-28T22:09:30.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:30 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2024-03-28T22:09:30.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:30 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-03-28T22:09:30.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:30 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]': finished 2024-03-28T22:09:30.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:30 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]': finished 2024-03-28T22:09:30.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:30 smithi053 ceph-mon[35979]: osdmap e87: 12 total, 12 up, 12 in 2024-03-28T22:09:30.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:30 smithi053 ceph-mon[35979]: pgmap v317: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T22:09:30.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:30 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:09:30.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:30 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3400236368' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-28T22:09:30.347 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:30 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-03-28T22:09:30.347 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:30 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-28T22:09:30.347 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:30 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-03-28T22:09:30.347 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:30 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2024-03-28T22:09:30.347 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:30 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-03-28T22:09:30.347 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:30 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]': finished 2024-03-28T22:09:30.347 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:30 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]': finished 2024-03-28T22:09:30.347 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:30 smithi016 ceph-mon[32374]: osdmap e87: 12 total, 12 up, 12 in 2024-03-28T22:09:30.347 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:30 smithi016 ceph-mon[32374]: pgmap v317: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T22:09:30.347 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:30 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:09:30.347 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:30 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3400236368' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-28T22:09:30.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:30 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-03-28T22:09:30.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:30 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-28T22:09:30.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:30 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-03-28T22:09:30.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:30 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2024-03-28T22:09:30.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:30 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-03-28T22:09:30.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:30 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]': finished 2024-03-28T22:09:30.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:30 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]': finished 2024-03-28T22:09:30.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:30 smithi063 ceph-mon[36249]: osdmap e87: 12 total, 12 up, 12 in 2024-03-28T22:09:30.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:30 smithi063 ceph-mon[36249]: pgmap v317: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T22:09:30.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:30 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:09:30.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:30 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/3400236368' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-28T22:09:30.738 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037793 got 257698037799 for osd.8 2024-03-28T22:09:30.738 DEBUG:teuthology.parallel:result is None 2024-03-28T22:09:30.963 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149756 got 111669149759 for osd.3 2024-03-28T22:09:30.963 DEBUG:teuthology.parallel:result is None 2024-03-28T22:09:31.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.129+0000 7f537e0a6700 1 -- 172.21.15.16:0/565207017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5378102ed0 msgr2=0x7f5378106ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:31.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.129+0000 7f537e0a6700 1 --2- 172.21.15.16:0/565207017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5378102ed0 0x7f5378106ea0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f536c009990 tx=0x7f536c009ca0 comp rx=0 tx=0).stop 2024-03-28T22:09:31.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.129+0000 7f537e0a6700 1 -- 172.21.15.16:0/565207017 shutdown_connections 2024-03-28T22:09:31.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.129+0000 7f537e0a6700 1 --2- 172.21.15.16:0/565207017 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f53781075d0 0x7f537810f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:31.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.129+0000 7f537e0a6700 1 --2- 172.21.15.16:0/565207017 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5378102520 0x7f5378102900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:31.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.129+0000 7f537e0a6700 1 --2- 172.21.15.16:0/565207017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5378102ed0 0x7f5378106ea0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:31.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.129+0000 7f537e0a6700 1 -- 172.21.15.16:0/565207017 >> 172.21.15.16:0/565207017 conn(0x7f5378079310 msgr2=0x7f5378079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:31.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.129+0000 7f537e0a6700 1 -- 172.21.15.16:0/565207017 shutdown_connections 2024-03-28T22:09:31.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.129+0000 7f537e0a6700 1 -- 172.21.15.16:0/565207017 wait complete. 2024-03-28T22:09:31.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.130+0000 7f537e0a6700 1 Processor -- start 2024-03-28T22:09:31.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.130+0000 7f537e0a6700 1 -- start start 2024-03-28T22:09:31.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.130+0000 7f537e0a6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5378102520 0x7f537819e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:31.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.130+0000 7f537e0a6700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5378102ed0 0x7f537819eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:31.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.130+0000 7f537e0a6700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f53781075d0 0x7f53781987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:31.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.130+0000 7f537e0a6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5378112560 con 0x7f5378102520 2024-03-28T22:09:31.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.130+0000 7f537e0a6700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f53781123e0 con 0x7f5378102ed0 2024-03-28T22:09:31.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.130+0000 7f537e0a6700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f53781126e0 con 0x7f53781075d0 2024-03-28T22:09:31.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.130+0000 7f53777fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5378102520 0x7f537819e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:31.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.130+0000 7f53777fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5378102520 0x7f537819e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53372/0 (socket says 172.21.15.16:53372) 2024-03-28T22:09:31.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.130+0000 7f53777fe700 1 -- 172.21.15.16:0/1355331482 learned_addr learned my addr 172.21.15.16:0/1355331482 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:31.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.130+0000 7f5376ffd700 1 --2- 172.21.15.16:0/1355331482 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5378102ed0 0x7f537819eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:31.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.130+0000 7f5377fff700 1 --2- 172.21.15.16:0/1355331482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f53781075d0 0x7f53781987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:31.249 INFO:teuthology.orchestra.run.smithi016.stdout:313532612637 2024-03-28T22:09:31.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.131+0000 7f53777fe700 1 -- 172.21.15.16:0/1355331482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f53781075d0 msgr2=0x7f53781987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:31.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.131+0000 7f53777fe700 1 --2- 172.21.15.16:0/1355331482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f53781075d0 0x7f53781987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:31.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.131+0000 7f53777fe700 1 -- 172.21.15.16:0/1355331482 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5378102ed0 msgr2=0x7f537819eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:31.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.131+0000 7f53777fe700 1 --2- 172.21.15.16:0/1355331482 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5378102ed0 0x7f537819eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:31.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.131+0000 7f53777fe700 1 -- 172.21.15.16:0/1355331482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5378198f80 con 0x7f5378102520 2024-03-28T22:09:31.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.131+0000 7f5377fff700 1 --2- 172.21.15.16:0/1355331482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f53781075d0 0x7f53781987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:09:31.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.131+0000 7f53777fe700 1 --2- 172.21.15.16:0/1355331482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5378102520 0x7f537819e620 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f536800d980 tx=0x7f536800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:31.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.131+0000 7f5374ff9700 1 -- 172.21.15.16:0/1355331482 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5368016070 con 0x7f5378102520 2024-03-28T22:09:31.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.131+0000 7f5374ff9700 1 -- 172.21.15.16:0/1355331482 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5368004d10 con 0x7f5378102520 2024-03-28T22:09:31.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.131+0000 7f537e0a6700 1 -- 172.21.15.16:0/1355331482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5378199270 con 0x7f5378102520 2024-03-28T22:09:31.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.131+0000 7f5374ff9700 1 -- 172.21.15.16:0/1355331482 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53680055f0 con 0x7f5378102520 2024-03-28T22:09:31.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.131+0000 7f537e0a6700 1 -- 172.21.15.16:0/1355331482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53781a5480 con 0x7f5378102520 2024-03-28T22:09:31.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.132+0000 7f537e0a6700 1 -- 172.21.15.16:0/1355331482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5378103980 con 0x7f5378102520 2024-03-28T22:09:31.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.133+0000 7f5374ff9700 1 -- 172.21.15.16:0/1355331482 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f5368004750 con 0x7f5378102520 2024-03-28T22:09:31.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.134+0000 7f5374ff9700 1 --2- 172.21.15.16:0/1355331482 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f53600a1ea0 0x7f53600a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:31.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.134+0000 7f5376ffd700 1 --2- 172.21.15.16:0/1355331482 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f53600a1ea0 0x7f53600a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:31.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.134+0000 7f5374ff9700 1 -- 172.21.15.16:0/1355331482 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f53680ce340 con 0x7f5378102520 2024-03-28T22:09:31.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.134+0000 7f5376ffd700 1 --2- 172.21.15.16:0/1355331482 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f53600a1ea0 0x7f53600a4360 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5378108940 tx=0x7f536c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:31.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.137+0000 7f5374ff9700 1 -- 172.21.15.16:0/1355331482 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f536801f020 con 0x7f5378102520 2024-03-28T22:09:31.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.343+0000 7f537e0a6700 1 -- 172.21.15.16:0/1355331482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f537802db30 con 0x7f5378102520 2024-03-28T22:09:31.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.344+0000 7f5374ff9700 1 -- 172.21.15.16:0/1355331482 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f536809b5e0 con 0x7f5378102520 2024-03-28T22:09:31.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.346+0000 7f537e0a6700 1 -- 172.21.15.16:0/1355331482 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f53600a1ea0 msgr2=0x7f53600a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:31.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.346+0000 7f537e0a6700 1 --2- 172.21.15.16:0/1355331482 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f53600a1ea0 0x7f53600a4360 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5378108940 tx=0x7f536c005fd0 comp rx=0 tx=0).stop 2024-03-28T22:09:31.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.346+0000 7f537e0a6700 1 -- 172.21.15.16:0/1355331482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5378102520 msgr2=0x7f537819e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:31.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.346+0000 7f537e0a6700 1 --2- 172.21.15.16:0/1355331482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5378102520 0x7f537819e620 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f536800d980 tx=0x7f536800dc90 comp rx=0 tx=0).stop 2024-03-28T22:09:31.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.346+0000 7f537e0a6700 1 -- 172.21.15.16:0/1355331482 shutdown_connections 2024-03-28T22:09:31.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.346+0000 7f537e0a6700 1 --2- 172.21.15.16:0/1355331482 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f53600a1ea0 0x7f53600a4360 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:31.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.346+0000 7f537e0a6700 1 --2- 172.21.15.16:0/1355331482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f53781075d0 0x7f53781987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:31.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.346+0000 7f537e0a6700 1 --2- 172.21.15.16:0/1355331482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5378102520 0x7f537819e620 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:31.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.346+0000 7f537e0a6700 1 --2- 172.21.15.16:0/1355331482 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5378102ed0 0x7f537819eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:31.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.346+0000 7f537e0a6700 1 -- 172.21.15.16:0/1355331482 >> 172.21.15.16:0/1355331482 conn(0x7f5378079310 msgr2=0x7f53781002d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:31.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.346+0000 7f537e0a6700 1 -- 172.21.15.16:0/1355331482 shutdown_connections 2024-03-28T22:09:31.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:30.346+0000 7f537e0a6700 1 -- 172.21.15.16:0/1355331482 wait complete. 2024-03-28T22:09:31.252 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:31 smithi016 ceph-mon[32374]: osdmap e88: 12 total, 12 up, 12 in 2024-03-28T22:09:31.253 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:31 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/1355331482' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-28T22:09:31.253 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:31 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:09:31.253 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:31 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:09:31.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:31 smithi063 ceph-mon[36249]: osdmap e88: 12 total, 12 up, 12 in 2024-03-28T22:09:31.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:31 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/1355331482' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-28T22:09:31.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:31 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:09:31.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:31 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:09:31.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:31 smithi053 ceph-mon[35979]: osdmap e88: 12 total, 12 up, 12 in 2024-03-28T22:09:31.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:31 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/1355331482' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-28T22:09:31.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:31 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:09:31.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:31 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:09:31.796 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532612633 got 313532612637 for osd.10 2024-03-28T22:09:31.796 DEBUG:teuthology.parallel:result is None 2024-03-28T22:09:32.389 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:32 smithi016 ceph-mon[32374]: pgmap v319: 97 pgs: 5 activating, 92 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-03-28T22:09:32.389 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:32 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:09:32.389 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:32 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:09:32.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:32 smithi063 ceph-mon[36249]: pgmap v319: 97 pgs: 5 activating, 92 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-03-28T22:09:32.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:32 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:09:32.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:32 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:09:32.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:32 smithi053 ceph-mon[35979]: pgmap v319: 97 pgs: 5 activating, 92 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-03-28T22:09:32.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:32 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:09:32.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:32 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:09:32.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.576+0000 7ff1a0054700 1 -- 172.21.15.16:0/2015722121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff198074d90 msgr2=0x7ff19810e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:32.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.576+0000 7ff1a0054700 1 --2- 172.21.15.16:0/2015722121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff198074d90 0x7ff19810e700 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff188009990 tx=0x7ff188009ca0 comp rx=0 tx=0).stop 2024-03-28T22:09:32.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.576+0000 7ff1a0054700 1 -- 172.21.15.16:0/2015722121 shutdown_connections 2024-03-28T22:09:32.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.576+0000 7ff1a0054700 1 --2- 172.21.15.16:0/2015722121 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff19810ec40 0x7ff198111030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:32.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.576+0000 7ff1a0054700 1 --2- 172.21.15.16:0/2015722121 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff198074470 0x7ff198074850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:32.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.576+0000 7ff1a0054700 1 --2- 172.21.15.16:0/2015722121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff198074d90 0x7ff19810e700 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:32.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.576+0000 7ff1a0054700 1 -- 172.21.15.16:0/2015722121 >> 172.21.15.16:0/2015722121 conn(0x7ff198079370 msgr2=0x7ff198079780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:32.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.576+0000 7ff1a0054700 1 -- 172.21.15.16:0/2015722121 shutdown_connections 2024-03-28T22:09:32.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.576+0000 7ff1a0054700 1 -- 172.21.15.16:0/2015722121 wait complete. 2024-03-28T22:09:32.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.576+0000 7ff1a0054700 1 Processor -- start 2024-03-28T22:09:32.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.576+0000 7ff1a0054700 1 -- start start 2024-03-28T22:09:32.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.577+0000 7ff1a0054700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff198074470 0x7ff1981a0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:32.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.577+0000 7ff1a0054700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff198074d90 0x7ff1981a0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:32.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.577+0000 7ff1a0054700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff19810ec40 0x7ff19819aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:32.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.577+0000 7ff1a0054700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff1981147a0 con 0x7ff198074d90 2024-03-28T22:09:32.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.577+0000 7ff1a0054700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff198114620 con 0x7ff19810ec40 2024-03-28T22:09:32.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.577+0000 7ff1a0054700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff198114920 con 0x7ff198074470 2024-03-28T22:09:32.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.577+0000 7ff19d5ef700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff198074d90 0x7ff1981a0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:32.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.577+0000 7ff19d5ef700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff198074d90 0x7ff1981a0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60014/0 (socket says 172.21.15.16:60014) 2024-03-28T22:09:32.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.577+0000 7ff19d5ef700 1 -- 172.21.15.16:0/2491572419 learned_addr learned my addr 172.21.15.16:0/2491572419 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:32.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.577+0000 7ff19e5f1700 1 --2- 172.21.15.16:0/2491572419 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff19810ec40 0x7ff19819aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:32.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.577+0000 7ff19ddf0700 1 --2- 172.21.15.16:0/2491572419 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff198074470 0x7ff1981a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:32.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.577+0000 7ff19d5ef700 1 -- 172.21.15.16:0/2491572419 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff198074470 msgr2=0x7ff1981a0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:32.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.577+0000 7ff19d5ef700 1 --2- 172.21.15.16:0/2491572419 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff198074470 0x7ff1981a0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:32.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.577+0000 7ff19d5ef700 1 -- 172.21.15.16:0/2491572419 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff19810ec40 msgr2=0x7ff19819aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:32.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.577+0000 7ff19d5ef700 1 --2- 172.21.15.16:0/2491572419 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff19810ec40 0x7ff19819aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:32.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.577+0000 7ff19d5ef700 1 -- 172.21.15.16:0/2491572419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff19819b1a0 con 0x7ff198074d90 2024-03-28T22:09:32.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.577+0000 7ff19d5ef700 1 --2- 172.21.15.16:0/2491572419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff198074d90 0x7ff1981a0dc0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff188005910 tx=0x7ff18800bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:32.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.577+0000 7ff18effd700 1 -- 172.21.15.16:0/2491572419 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff188027070 con 0x7ff198074d90 2024-03-28T22:09:32.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.577+0000 7ff1a0054700 1 -- 172.21.15.16:0/2491572419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff19819b430 con 0x7ff198074d90 2024-03-28T22:09:32.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.577+0000 7ff18effd700 1 -- 172.21.15.16:0/2491572419 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff188003d70 con 0x7ff198074d90 2024-03-28T22:09:32.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.577+0000 7ff18effd700 1 -- 172.21.15.16:0/2491572419 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff18800f460 con 0x7ff198074d90 2024-03-28T22:09:32.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.578+0000 7ff1a0054700 1 -- 172.21.15.16:0/2491572419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1981a7750 con 0x7ff198074d90 2024-03-28T22:09:32.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.578+0000 7ff1a0054700 1 -- 172.21.15.16:0/2491572419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff198068f00 con 0x7ff198074d90 2024-03-28T22:09:32.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.581+0000 7ff18effd700 1 -- 172.21.15.16:0/2491572419 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7ff18802a070 con 0x7ff198074d90 2024-03-28T22:09:32.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.581+0000 7ff18effd700 1 --2- 172.21.15.16:0/2491572419 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff1840a1ea0 0x7ff1840a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:32.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.581+0000 7ff19ddf0700 1 --2- 172.21.15.16:0/2491572419 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff1840a1ea0 0x7ff1840a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:32.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.581+0000 7ff18effd700 1 -- 172.21.15.16:0/2491572419 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7ff1880cf190 con 0x7ff198074d90 2024-03-28T22:09:32.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.582+0000 7ff19ddf0700 1 --2- 172.21.15.16:0/2491572419 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff1840a1ea0 0x7ff1840a4360 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff19819bb80 tx=0x7ff194009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:32.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.584+0000 7ff18effd700 1 -- 172.21.15.16:0/2491572419 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff188013070 con 0x7ff198074d90 2024-03-28T22:09:32.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.791+0000 7ff1a0054700 1 -- 172.21.15.16:0/2491572419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7ff1981a7920 con 0x7ff198074d90 2024-03-28T22:09:32.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.792+0000 7ff18effd700 1 -- 172.21.15.16:0/2491572419 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff18809c520 con 0x7ff198074d90 2024-03-28T22:09:32.795 INFO:teuthology.orchestra.run.smithi016.stdout:223338299438 2024-03-28T22:09:32.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.795+0000 7ff1a0054700 1 -- 172.21.15.16:0/2491572419 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff1840a1ea0 msgr2=0x7ff1840a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:32.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.795+0000 7ff1a0054700 1 --2- 172.21.15.16:0/2491572419 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff1840a1ea0 0x7ff1840a4360 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff19819bb80 tx=0x7ff194009260 comp rx=0 tx=0).stop 2024-03-28T22:09:32.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.795+0000 7ff1a0054700 1 -- 172.21.15.16:0/2491572419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff198074d90 msgr2=0x7ff1981a0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:32.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.795+0000 7ff1a0054700 1 --2- 172.21.15.16:0/2491572419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff198074d90 0x7ff1981a0dc0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff188005910 tx=0x7ff18800bab0 comp rx=0 tx=0).stop 2024-03-28T22:09:32.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.796+0000 7ff1a0054700 1 -- 172.21.15.16:0/2491572419 shutdown_connections 2024-03-28T22:09:32.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.796+0000 7ff1a0054700 1 --2- 172.21.15.16:0/2491572419 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff1840a1ea0 0x7ff1840a4360 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:32.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.796+0000 7ff1a0054700 1 --2- 172.21.15.16:0/2491572419 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff19810ec40 0x7ff19819aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:32.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.796+0000 7ff1a0054700 1 --2- 172.21.15.16:0/2491572419 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff198074470 0x7ff1981a0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:32.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.796+0000 7ff1a0054700 1 --2- 172.21.15.16:0/2491572419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff198074d90 0x7ff1981a0dc0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:32.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.796+0000 7ff1a0054700 1 -- 172.21.15.16:0/2491572419 >> 172.21.15.16:0/2491572419 conn(0x7ff198079370 msgr2=0x7ff1980ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:32.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.796+0000 7ff1a0054700 1 -- 172.21.15.16:0/2491572419 shutdown_connections 2024-03-28T22:09:32.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:32.796+0000 7ff1a0054700 1 -- 172.21.15.16:0/2491572419 wait complete. 2024-03-28T22:09:33.198 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411393 got 77309411397 for osd.2 2024-03-28T22:09:33.198 DEBUG:teuthology.parallel:result is None 2024-03-28T22:09:33.395 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:33 smithi016 ceph-mon[32374]: Health check failed: Reduced data availability: 2 pgs inactive (PG_AVAILABILITY) 2024-03-28T22:09:33.396 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:33 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2491572419' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-28T22:09:33.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:33 smithi063 ceph-mon[36249]: Health check failed: Reduced data availability: 2 pgs inactive (PG_AVAILABILITY) 2024-03-28T22:09:33.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:33 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2491572419' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-28T22:09:33.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:33 smithi053 ceph-mon[35979]: Health check failed: Reduced data availability: 2 pgs inactive (PG_AVAILABILITY) 2024-03-28T22:09:33.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:33 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2491572419' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-28T22:09:33.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.613+0000 7f205da89700 1 -- 172.21.15.16:0/4049690152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2058108270 msgr2=0x7f2058108650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:33.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.613+0000 7f205da89700 1 --2- 172.21.15.16:0/4049690152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2058108270 0x7f2058108650 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f20480099c0 tx=0x7f2048009cd0 comp rx=0 tx=0).stop 2024-03-28T22:09:33.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.614+0000 7f205da89700 1 -- 172.21.15.16:0/4049690152 shutdown_connections 2024-03-28T22:09:33.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.614+0000 7f205da89700 1 --2- 172.21.15.16:0/4049690152 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f205806b350 0x7f205810f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:33.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.614+0000 7f205da89700 1 --2- 172.21.15.16:0/4049690152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2058108270 0x7f2058108650 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:33.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.614+0000 7f205da89700 1 --2- 172.21.15.16:0/4049690152 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f205806a990 0x7f205806ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:33.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.614+0000 7f205da89700 1 -- 172.21.15.16:0/4049690152 >> 172.21.15.16:0/4049690152 conn(0x7f2058078150 msgr2=0x7f2058078560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:33.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.614+0000 7f205da89700 1 -- 172.21.15.16:0/4049690152 shutdown_connections 2024-03-28T22:09:33.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.614+0000 7f205da89700 1 -- 172.21.15.16:0/4049690152 wait complete. 2024-03-28T22:09:33.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.614+0000 7f205da89700 1 Processor -- start 2024-03-28T22:09:33.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.614+0000 7f205da89700 1 -- start start 2024-03-28T22:09:33.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.615+0000 7f205da89700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f205806a990 0x7f205819e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:33.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.615+0000 7f205da89700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f205806b350 0x7f205819ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:33.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.615+0000 7f205da89700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2058108270 0x7f20581987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:33.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.615+0000 7f205da89700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2058112570 con 0x7f205806b350 2024-03-28T22:09:33.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.615+0000 7f205da89700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f20581123f0 con 0x7f2058108270 2024-03-28T22:09:33.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.615+0000 7f205da89700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f20581126f0 con 0x7f205806a990 2024-03-28T22:09:33.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.615+0000 7f2056ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f205806b350 0x7f205819ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:33.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.615+0000 7f2056ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f205806b350 0x7f205819ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60044/0 (socket says 172.21.15.16:60044) 2024-03-28T22:09:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.615+0000 7f2056ffd700 1 -- 172.21.15.16:0/2152482849 learned_addr learned my addr 172.21.15.16:0/2152482849 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.615+0000 7f2057fff700 1 --2- 172.21.15.16:0/2152482849 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2058108270 0x7f20581987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.615+0000 7f20577fe700 1 --2- 172.21.15.16:0/2152482849 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f205806a990 0x7f205819e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.615+0000 7f2056ffd700 1 -- 172.21.15.16:0/2152482849 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f205806a990 msgr2=0x7f205819e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.615+0000 7f2056ffd700 1 --2- 172.21.15.16:0/2152482849 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f205806a990 0x7f205819e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.615+0000 7f2056ffd700 1 -- 172.21.15.16:0/2152482849 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2058108270 msgr2=0x7f20581987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.615+0000 7f2056ffd700 1 --2- 172.21.15.16:0/2152482849 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2058108270 0x7f20581987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.615+0000 7f2056ffd700 1 -- 172.21.15.16:0/2152482849 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2058198fd0 con 0x7f205806b350 2024-03-28T22:09:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.615+0000 7f2057fff700 1 --2- 172.21.15.16:0/2152482849 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2058108270 0x7f20581987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:09:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.615+0000 7f2056ffd700 1 --2- 172.21.15.16:0/2152482849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f205806b350 0x7f205819ebb0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f204c00e9a0 tx=0x7f204c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.615+0000 7f2054ff9700 1 -- 172.21.15.16:0/2152482849 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f204c00ca10 con 0x7f205806b350 2024-03-28T22:09:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.615+0000 7f205da89700 1 -- 172.21.15.16:0/2152482849 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20581992c0 con 0x7f205806b350 2024-03-28T22:09:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.615+0000 7f2054ff9700 1 -- 172.21.15.16:0/2152482849 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f204c00cbb0 con 0x7f205806b350 2024-03-28T22:09:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.615+0000 7f2054ff9700 1 -- 172.21.15.16:0/2152482849 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f204c018680 con 0x7f205806b350 2024-03-28T22:09:33.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.616+0000 7f205da89700 1 -- 172.21.15.16:0/2152482849 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20581a54d0 con 0x7f205806b350 2024-03-28T22:09:33.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.617+0000 7f205da89700 1 -- 172.21.15.16:0/2152482849 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2058103920 con 0x7f205806b350 2024-03-28T22:09:33.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.618+0000 7f2054ff9700 1 -- 172.21.15.16:0/2152482849 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f204c010430 con 0x7f205806b350 2024-03-28T22:09:33.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.618+0000 7f2054ff9700 1 --2- 172.21.15.16:0/2152482849 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f20400a1ea0 0x7f20400a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:33.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.618+0000 7f20577fe700 1 --2- 172.21.15.16:0/2152482849 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f20400a1ea0 0x7f20400a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:33.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.618+0000 7f2054ff9700 1 -- 172.21.15.16:0/2152482849 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f204c0ce5e0 con 0x7f205806b350 2024-03-28T22:09:33.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.619+0000 7f20577fe700 1 --2- 172.21.15.16:0/2152482849 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f20400a1ea0 0x7f20400a4360 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2048000c00 tx=0x7f2048005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:33.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.622+0000 7f2054ff9700 1 -- 172.21.15.16:0/2152482849 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f204c09b8e0 con 0x7f205806b350 2024-03-28T22:09:33.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.832+0000 7f205da89700 1 -- 172.21.15.16:0/2152482849 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f205802db30 con 0x7f205806b350 2024-03-28T22:09:33.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.832+0000 7f2054ff9700 1 -- 172.21.15.16:0/2152482849 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f204c09b1e0 con 0x7f205806b350 2024-03-28T22:09:33.835 INFO:teuthology.orchestra.run.smithi016.stdout:55834574922 2024-03-28T22:09:33.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.834+0000 7f205da89700 1 -- 172.21.15.16:0/2152482849 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f20400a1ea0 msgr2=0x7f20400a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:33.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.834+0000 7f205da89700 1 --2- 172.21.15.16:0/2152482849 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f20400a1ea0 0x7f20400a4360 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2048000c00 tx=0x7f2048005fb0 comp rx=0 tx=0).stop 2024-03-28T22:09:33.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.834+0000 7f205da89700 1 -- 172.21.15.16:0/2152482849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f205806b350 msgr2=0x7f205819ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:33.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.834+0000 7f205da89700 1 --2- 172.21.15.16:0/2152482849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f205806b350 0x7f205819ebb0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f204c00e9a0 tx=0x7f204c00ecb0 comp rx=0 tx=0).stop 2024-03-28T22:09:33.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.834+0000 7f205da89700 1 -- 172.21.15.16:0/2152482849 shutdown_connections 2024-03-28T22:09:33.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.834+0000 7f205da89700 1 --2- 172.21.15.16:0/2152482849 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f20400a1ea0 0x7f20400a4360 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:33.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.834+0000 7f205da89700 1 --2- 172.21.15.16:0/2152482849 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2058108270 0x7f20581987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:33.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.834+0000 7f205da89700 1 --2- 172.21.15.16:0/2152482849 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f205806a990 0x7f205819e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:33.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.834+0000 7f205da89700 1 --2- 172.21.15.16:0/2152482849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f205806b350 0x7f205819ebb0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:33.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.834+0000 7f205da89700 1 -- 172.21.15.16:0/2152482849 >> 172.21.15.16:0/2152482849 conn(0x7f2058078150 msgr2=0x7f20580ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:33.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.834+0000 7f205da89700 1 -- 172.21.15.16:0/2152482849 shutdown_connections 2024-03-28T22:09:33.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:33.835+0000 7f205da89700 1 -- 172.21.15.16:0/2152482849 wait complete. 2024-03-28T22:09:34.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:34 smithi063 ceph-mon[36249]: pgmap v320: 97 pgs: 5 activating, 92 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-28T22:09:34.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:34 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2152482849' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-28T22:09:34.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:34 smithi016 ceph-mon[32374]: pgmap v320: 97 pgs: 5 activating, 92 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-28T22:09:34.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:34 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2152482849' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-28T22:09:34.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:34 smithi053 ceph-mon[35979]: pgmap v320: 97 pgs: 5 activating, 92 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-28T22:09:34.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:34 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2152482849' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-28T22:09:34.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.774+0000 7fd0629c2700 1 -- 172.21.15.16:0/2076404978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd05c101140 msgr2=0x7fd05c1015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:34.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.774+0000 7fd0629c2700 1 --2- 172.21.15.16:0/2076404978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd05c101140 0x7fd05c1015a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd050009990 tx=0x7fd050009ca0 comp rx=0 tx=0).stop 2024-03-28T22:09:34.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.774+0000 7fd0629c2700 1 -- 172.21.15.16:0/2076404978 shutdown_connections 2024-03-28T22:09:34.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.774+0000 7fd0629c2700 1 --2- 172.21.15.16:0/2076404978 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd05c101ae0 0x7fd05c10dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:34.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.774+0000 7fd0629c2700 1 --2- 172.21.15.16:0/2076404978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd05c107180 0x7fd05c107560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:34.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.774+0000 7fd0629c2700 1 --2- 172.21.15.16:0/2076404978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd05c101140 0x7fd05c1015a0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:34.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.774+0000 7fd0629c2700 1 -- 172.21.15.16:0/2076404978 >> 172.21.15.16:0/2076404978 conn(0x7fd05c078070 msgr2=0x7fd05c078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:34.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.774+0000 7fd0629c2700 1 -- 172.21.15.16:0/2076404978 shutdown_connections 2024-03-28T22:09:34.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.774+0000 7fd0629c2700 1 -- 172.21.15.16:0/2076404978 wait complete. 2024-03-28T22:09:34.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.775+0000 7fd0629c2700 1 Processor -- start 2024-03-28T22:09:34.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.775+0000 7fd0629c2700 1 -- start start 2024-03-28T22:09:34.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.775+0000 7fd0629c2700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd05c101140 0x7fd05c102df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:34.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.775+0000 7fd0629c2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd05c101ae0 0x7fd05c103330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:34.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.775+0000 7fd0629c2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd05c107180 0x7fd05c103870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:34.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.775+0000 7fd0629c2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd05c1113e0 con 0x7fd05c107180 2024-03-28T22:09:34.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.775+0000 7fd0629c2700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd05c111260 con 0x7fd05c101ae0 2024-03-28T22:09:34.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.775+0000 7fd0629c2700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd05c111560 con 0x7fd05c101140 2024-03-28T22:09:34.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.775+0000 7fd060f5f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd05c107180 0x7fd05c103870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:34.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.775+0000 7fd060f5f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd05c107180 0x7fd05c103870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60062/0 (socket says 172.21.15.16:60062) 2024-03-28T22:09:34.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.775+0000 7fd060f5f700 1 -- 172.21.15.16:0/305468630 learned_addr learned my addr 172.21.15.16:0/305468630 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:34.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.776+0000 7fd05b7fe700 1 --2- 172.21.15.16:0/305468630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd05c101ae0 0x7fd05c103330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:34.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.776+0000 7fd05bfff700 1 --2- 172.21.15.16:0/305468630 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd05c101140 0x7fd05c102df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:34.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.776+0000 7fd060f5f700 1 -- 172.21.15.16:0/305468630 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd05c101140 msgr2=0x7fd05c102df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:34.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.776+0000 7fd060f5f700 1 --2- 172.21.15.16:0/305468630 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd05c101140 0x7fd05c102df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:34.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.776+0000 7fd060f5f700 1 -- 172.21.15.16:0/305468630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd05c101ae0 msgr2=0x7fd05c103330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:34.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.776+0000 7fd060f5f700 1 --2- 172.21.15.16:0/305468630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd05c101ae0 0x7fd05c103330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:34.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.776+0000 7fd060f5f700 1 -- 172.21.15.16:0/305468630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd05c1a6f10 con 0x7fd05c107180 2024-03-28T22:09:34.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.776+0000 7fd05b7fe700 1 --2- 172.21.15.16:0/305468630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd05c101ae0 0x7fd05c103330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:09:34.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.776+0000 7fd060f5f700 1 --2- 172.21.15.16:0/305468630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd05c107180 0x7fd05c103870 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fd05400ef30 tx=0x7fd05400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:34.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.776+0000 7fd0597fa700 1 -- 172.21.15.16:0/305468630 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0540092e0 con 0x7fd05c107180 2024-03-28T22:09:34.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.776+0000 7fd0629c2700 1 -- 172.21.15.16:0/305468630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd05c1a71a0 con 0x7fd05c107180 2024-03-28T22:09:34.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.776+0000 7fd0597fa700 1 -- 172.21.15.16:0/305468630 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd054004bb0 con 0x7fd05c107180 2024-03-28T22:09:34.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.776+0000 7fd0597fa700 1 -- 172.21.15.16:0/305468630 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd054004280 con 0x7fd05c107180 2024-03-28T22:09:34.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.776+0000 7fd0629c2700 1 -- 172.21.15.16:0/305468630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd05c1a76b0 con 0x7fd05c107180 2024-03-28T22:09:34.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.777+0000 7fd0629c2700 1 -- 172.21.15.16:0/305468630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd05c068f00 con 0x7fd05c107180 2024-03-28T22:09:34.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.779+0000 7fd0597fa700 1 -- 172.21.15.16:0/305468630 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fd05401b030 con 0x7fd05c107180 2024-03-28T22:09:34.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.779+0000 7fd0597fa700 1 --2- 172.21.15.16:0/305468630 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fd0440a1ea0 0x7fd0440a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:34.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.780+0000 7fd05bfff700 1 --2- 172.21.15.16:0/305468630 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fd0440a1ea0 0x7fd0440a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:34.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.780+0000 7fd0597fa700 1 -- 172.21.15.16:0/305468630 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fd0540ce3c0 con 0x7fd05c107180 2024-03-28T22:09:34.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.780+0000 7fd05bfff700 1 --2- 172.21.15.16:0/305468630 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fd0440a1ea0 0x7fd0440a4360 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd04c007b60 tx=0x7fd04c0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:34.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.783+0000 7fd0597fa700 1 -- 172.21.15.16:0/305468630 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd054022020 con 0x7fd05c107180 2024-03-28T22:09:34.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.989+0000 7fd0629c2700 1 -- 172.21.15.16:0/305468630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fd05c1a7940 con 0x7fd05c107180 2024-03-28T22:09:34.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.989+0000 7fd0597fa700 1 -- 172.21.15.16:0/305468630 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd054010040 con 0x7fd05c107180 2024-03-28T22:09:34.992 INFO:teuthology.orchestra.run.smithi016.stdout:146028888125 2024-03-28T22:09:34.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.992+0000 7fd0629c2700 1 -- 172.21.15.16:0/305468630 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fd0440a1ea0 msgr2=0x7fd0440a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:34.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.992+0000 7fd0629c2700 1 --2- 172.21.15.16:0/305468630 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fd0440a1ea0 0x7fd0440a4360 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd04c007b60 tx=0x7fd04c0073d0 comp rx=0 tx=0).stop 2024-03-28T22:09:34.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.992+0000 7fd0629c2700 1 -- 172.21.15.16:0/305468630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd05c107180 msgr2=0x7fd05c103870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:34.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.992+0000 7fd0629c2700 1 --2- 172.21.15.16:0/305468630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd05c107180 0x7fd05c103870 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fd05400ef30 tx=0x7fd05400c3a0 comp rx=0 tx=0).stop 2024-03-28T22:09:34.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.993+0000 7fd0629c2700 1 -- 172.21.15.16:0/305468630 shutdown_connections 2024-03-28T22:09:34.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.993+0000 7fd0629c2700 1 --2- 172.21.15.16:0/305468630 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fd0440a1ea0 0x7fd0440a4360 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:34.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.993+0000 7fd0629c2700 1 --2- 172.21.15.16:0/305468630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd05c107180 0x7fd05c103870 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:34.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.993+0000 7fd0629c2700 1 --2- 172.21.15.16:0/305468630 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd05c101140 0x7fd05c102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:34.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.993+0000 7fd0629c2700 1 --2- 172.21.15.16:0/305468630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd05c101ae0 0x7fd05c103330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:34.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.993+0000 7fd0629c2700 1 -- 172.21.15.16:0/305468630 >> 172.21.15.16:0/305468630 conn(0x7fd05c078070 msgr2=0x7fd05c100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:34.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.993+0000 7fd0629c2700 1 -- 172.21.15.16:0/305468630 shutdown_connections 2024-03-28T22:09:34.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:34.993+0000 7fd0629c2700 1 -- 172.21.15.16:0/305468630 wait complete. 2024-03-28T22:09:35.396 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:35 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/305468630' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-28T22:09:35.423 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299434 got 223338299438 for osd.7 2024-03-28T22:09:35.423 DEBUG:teuthology.parallel:result is None 2024-03-28T22:09:35.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:35 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/305468630' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-28T22:09:35.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:35 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/305468630' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-28T22:09:35.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.766+0000 7f8ccc027700 1 -- 172.21.15.16:0/2750495396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cc4108270 msgr2=0x7f8cc4108650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:35.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.766+0000 7f8ccc027700 1 --2- 172.21.15.16:0/2750495396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cc4108270 0x7f8cc4108650 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f8cc000b0d0 tx=0x7f8cc000b3e0 comp rx=0 tx=0).stop 2024-03-28T22:09:35.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.766+0000 7f8ccc027700 1 -- 172.21.15.16:0/2750495396 shutdown_connections 2024-03-28T22:09:35.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.766+0000 7f8ccc027700 1 --2- 172.21.15.16:0/2750495396 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8cc4102c50 0x7f8cc410f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:35.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.766+0000 7f8ccc027700 1 --2- 172.21.15.16:0/2750495396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cc4108270 0x7f8cc4108650 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:35.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.766+0000 7f8ccc027700 1 --2- 172.21.15.16:0/2750495396 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8cc4102290 0x7f8cc4102710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:35.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.766+0000 7f8ccc027700 1 -- 172.21.15.16:0/2750495396 >> 172.21.15.16:0/2750495396 conn(0x7f8cc4079370 msgr2=0x7f8cc4079780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:35.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.766+0000 7f8ccc027700 1 -- 172.21.15.16:0/2750495396 shutdown_connections 2024-03-28T22:09:35.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.766+0000 7f8ccc027700 1 -- 172.21.15.16:0/2750495396 wait complete. 2024-03-28T22:09:35.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.766+0000 7f8ccc027700 1 Processor -- start 2024-03-28T22:09:35.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.766+0000 7f8ccc027700 1 -- start start 2024-03-28T22:09:35.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.767+0000 7f8ccc027700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cc4102290 0x7f8cc419e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:35.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.767+0000 7f8ccc027700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8cc4102c50 0x7f8cc419ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:35.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.767+0000 7f8ccc027700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8cc4108270 0x7f8cc41988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:35.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.767+0000 7f8ccc027700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc41125c0 con 0x7f8cc4102290 2024-03-28T22:09:35.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.767+0000 7f8ccc027700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc4112440 con 0x7f8cc4102c50 2024-03-28T22:09:35.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.767+0000 7f8ccc027700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc4112740 con 0x7f8cc4108270 2024-03-28T22:09:35.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.767+0000 7f8cc9dc3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cc4102290 0x7f8cc419e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:35.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.767+0000 7f8cc9dc3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cc4102290 0x7f8cc419e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60086/0 (socket says 172.21.15.16:60086) 2024-03-28T22:09:35.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.767+0000 7f8cc9dc3700 1 -- 172.21.15.16:0/2817629544 learned_addr learned my addr 172.21.15.16:0/2817629544 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:35.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.767+0000 7f8cc95c2700 1 --2- 172.21.15.16:0/2817629544 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8cc4102c50 0x7f8cc419ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:35.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.767+0000 7f8cca5c4700 1 --2- 172.21.15.16:0/2817629544 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8cc4108270 0x7f8cc41988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:35.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.767+0000 7f8cc9dc3700 1 -- 172.21.15.16:0/2817629544 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8cc4108270 msgr2=0x7f8cc41988a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:35.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.767+0000 7f8cc9dc3700 1 --2- 172.21.15.16:0/2817629544 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8cc4108270 0x7f8cc41988a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:35.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.767+0000 7f8cc9dc3700 1 -- 172.21.15.16:0/2817629544 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8cc4102c50 msgr2=0x7f8cc419ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:35.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.767+0000 7f8cc9dc3700 1 --2- 172.21.15.16:0/2817629544 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8cc4102c50 0x7f8cc419ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:35.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.767+0000 7f8cc9dc3700 1 -- 172.21.15.16:0/2817629544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cc4199000 con 0x7f8cc4102290 2024-03-28T22:09:35.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.767+0000 7f8cc9dc3700 1 --2- 172.21.15.16:0/2817629544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cc4102290 0x7f8cc419e6e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8cc0009850 tx=0x7f8cc0007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:35.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.767+0000 7f8cbaffd700 1 -- 172.21.15.16:0/2817629544 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc0021070 con 0x7f8cc4102290 2024-03-28T22:09:35.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.767+0000 7f8cbaffd700 1 -- 172.21.15.16:0/2817629544 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8cc001db30 con 0x7f8cc4102290 2024-03-28T22:09:35.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.768+0000 7f8ccc027700 1 -- 172.21.15.16:0/2817629544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cc4199290 con 0x7f8cc4102290 2024-03-28T22:09:35.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.768+0000 7f8cbaffd700 1 -- 172.21.15.16:0/2817629544 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc001bbe0 con 0x7f8cc4102290 2024-03-28T22:09:35.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.768+0000 7f8ccc027700 1 -- 172.21.15.16:0/2817629544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cc41a55b0 con 0x7f8cc4102290 2024-03-28T22:09:35.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.769+0000 7f8ccc027700 1 -- 172.21.15.16:0/2817629544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cc4068f00 con 0x7f8cc4102290 2024-03-28T22:09:35.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.770+0000 7f8cbaffd700 1 -- 172.21.15.16:0/2817629544 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f8cc0024070 con 0x7f8cc4102290 2024-03-28T22:09:35.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.770+0000 7f8cbaffd700 1 --2- 172.21.15.16:0/2817629544 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f8cb00a1ea0 0x7f8cb00a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:35.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.770+0000 7f8cbaffd700 1 -- 172.21.15.16:0/2817629544 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f8cc000f020 con 0x7f8cc4102290 2024-03-28T22:09:35.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.770+0000 7f8cc95c2700 1 --2- 172.21.15.16:0/2817629544 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f8cb00a1ea0 0x7f8cb00a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:35.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.771+0000 7f8cc95c2700 1 --2- 172.21.15.16:0/2817629544 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f8cb00a1ea0 0x7f8cb00a4360 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8cc4079950 tx=0x7f8cb4009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:35.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.774+0000 7f8cbaffd700 1 -- 172.21.15.16:0/2817629544 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8cc002b030 con 0x7f8cc4102290 2024-03-28T22:09:35.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.978+0000 7f8ccc027700 1 -- 172.21.15.16:0/2817629544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f8cc4199dd0 con 0x7f8cc4102290 2024-03-28T22:09:35.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.979+0000 7f8cbaffd700 1 -- 172.21.15.16:0/2817629544 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f8cc009cf50 con 0x7f8cc4102290 2024-03-28T22:09:35.982 INFO:teuthology.orchestra.run.smithi016.stdout:292057776163 2024-03-28T22:09:35.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.981+0000 7f8ccc027700 1 -- 172.21.15.16:0/2817629544 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f8cb00a1ea0 msgr2=0x7f8cb00a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:35.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.981+0000 7f8ccc027700 1 --2- 172.21.15.16:0/2817629544 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f8cb00a1ea0 0x7f8cb00a4360 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8cc4079950 tx=0x7f8cb4009190 comp rx=0 tx=0).stop 2024-03-28T22:09:35.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.981+0000 7f8ccc027700 1 -- 172.21.15.16:0/2817629544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cc4102290 msgr2=0x7f8cc419e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:35.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.981+0000 7f8ccc027700 1 --2- 172.21.15.16:0/2817629544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cc4102290 0x7f8cc419e6e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8cc0009850 tx=0x7f8cc0007a20 comp rx=0 tx=0).stop 2024-03-28T22:09:35.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.981+0000 7f8ccc027700 1 -- 172.21.15.16:0/2817629544 shutdown_connections 2024-03-28T22:09:35.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.981+0000 7f8ccc027700 1 --2- 172.21.15.16:0/2817629544 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f8cb00a1ea0 0x7f8cb00a4360 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:35.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.981+0000 7f8ccc027700 1 --2- 172.21.15.16:0/2817629544 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8cc4108270 0x7f8cc41988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:35.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.981+0000 7f8ccc027700 1 --2- 172.21.15.16:0/2817629544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cc4102290 0x7f8cc419e6e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:35.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.981+0000 7f8ccc027700 1 --2- 172.21.15.16:0/2817629544 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8cc4102c50 0x7f8cc419ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:35.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.981+0000 7f8ccc027700 1 -- 172.21.15.16:0/2817629544 >> 172.21.15.16:0/2817629544 conn(0x7f8cc4079370 msgr2=0x7f8cc4100f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:35.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.981+0000 7f8ccc027700 1 -- 172.21.15.16:0/2817629544 shutdown_connections 2024-03-28T22:09:35.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:35.981+0000 7f8ccc027700 1 -- 172.21.15.16:0/2817629544 wait complete. 2024-03-28T22:09:36.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.001+0000 7fb116298700 1 -- 172.21.15.16:0/801754447 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb110102e30 msgr2=0x7fb110106e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:36.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.001+0000 7fb116298700 1 --2- 172.21.15.16:0/801754447 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb110102e30 0x7fb110106e80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb1040098f0 tx=0x7fb104009c00 comp rx=0 tx=0).stop 2024-03-28T22:09:36.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.001+0000 7fb116298700 1 -- 172.21.15.16:0/801754447 shutdown_connections 2024-03-28T22:09:36.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.001+0000 7fb116298700 1 --2- 172.21.15.16:0/801754447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb110107560 0x7fb11010f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:36.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.001+0000 7fb116298700 1 --2- 172.21.15.16:0/801754447 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb110102480 0x7fb110102860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:36.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.001+0000 7fb116298700 1 --2- 172.21.15.16:0/801754447 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb110102e30 0x7fb110106e80 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:36.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.001+0000 7fb116298700 1 -- 172.21.15.16:0/801754447 >> 172.21.15.16:0/801754447 conn(0x7fb1100fdcf0 msgr2=0x7fb110100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:36.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.001+0000 7fb116298700 1 -- 172.21.15.16:0/801754447 shutdown_connections 2024-03-28T22:09:36.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.001+0000 7fb116298700 1 -- 172.21.15.16:0/801754447 wait complete. 2024-03-28T22:09:36.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.002+0000 7fb116298700 1 Processor -- start 2024-03-28T22:09:36.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.002+0000 7fb116298700 1 -- start start 2024-03-28T22:09:36.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.002+0000 7fb116298700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb110102480 0x7fb110074540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:36.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.002+0000 7fb116298700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb110102e30 0x7fb110074a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:36.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.002+0000 7fb116298700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb110107560 0x7fb110074fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:36.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.002+0000 7fb116298700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb110112520 con 0x7fb110102480 2024-03-28T22:09:36.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.002+0000 7fb116298700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb1101123a0 con 0x7fb110102e30 2024-03-28T22:09:36.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.002+0000 7fb116298700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb1101126a0 con 0x7fb110107560 2024-03-28T22:09:36.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.003+0000 7fb10ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb110102480 0x7fb110074540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:36.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.003+0000 7fb10ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb110102480 0x7fb110074540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60116/0 (socket says 172.21.15.16:60116) 2024-03-28T22:09:36.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.003+0000 7fb10ffff700 1 -- 172.21.15.16:0/3582563619 learned_addr learned my addr 172.21.15.16:0/3582563619 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:36.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.003+0000 7fb10f7fe700 1 --2- 172.21.15.16:0/3582563619 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb110102e30 0x7fb110074a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:36.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.003+0000 7fb114835700 1 --2- 172.21.15.16:0/3582563619 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb110107560 0x7fb110074fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:36.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.003+0000 7fb10ffff700 1 -- 172.21.15.16:0/3582563619 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb110107560 msgr2=0x7fb110074fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:36.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.003+0000 7fb10ffff700 1 --2- 172.21.15.16:0/3582563619 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb110107560 0x7fb110074fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:36.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.003+0000 7fb10ffff700 1 -- 172.21.15.16:0/3582563619 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb110102e30 msgr2=0x7fb110074a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:36.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.003+0000 7fb10ffff700 1 --2- 172.21.15.16:0/3582563619 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb110102e30 0x7fb110074a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:36.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.003+0000 7fb10ffff700 1 -- 172.21.15.16:0/3582563619 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1101a6f70 con 0x7fb110102480 2024-03-28T22:09:36.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.003+0000 7fb114835700 1 --2- 172.21.15.16:0/3582563619 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb110107560 0x7fb110074fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:09:36.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.003+0000 7fb10f7fe700 1 --2- 172.21.15.16:0/3582563619 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb110102e30 0x7fb110074a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:09:36.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.003+0000 7fb10ffff700 1 --2- 172.21.15.16:0/3582563619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb110102480 0x7fb110074540 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb10000d950 tx=0x7fb10000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:36.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.003+0000 7fb10d7fa700 1 -- 172.21.15.16:0/3582563619 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb100016070 con 0x7fb110102480 2024-03-28T22:09:36.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.003+0000 7fb116298700 1 -- 172.21.15.16:0/3582563619 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1101a7260 con 0x7fb110102480 2024-03-28T22:09:36.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.003+0000 7fb10d7fa700 1 -- 172.21.15.16:0/3582563619 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb100009e20 con 0x7fb110102480 2024-03-28T22:09:36.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.003+0000 7fb10d7fa700 1 -- 172.21.15.16:0/3582563619 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb10000f620 con 0x7fb110102480 2024-03-28T22:09:36.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.003+0000 7fb116298700 1 -- 172.21.15.16:0/3582563619 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1101a77c0 con 0x7fb110102480 2024-03-28T22:09:36.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.004+0000 7fb116298700 1 -- 172.21.15.16:0/3582563619 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb110068f00 con 0x7fb110102480 2024-03-28T22:09:36.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.006+0000 7fb10d7fa700 1 -- 172.21.15.16:0/3582563619 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fb100019070 con 0x7fb110102480 2024-03-28T22:09:36.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.007+0000 7fb10d7fa700 1 --2- 172.21.15.16:0/3582563619 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb0f80a1ea0 0x7fb0f80a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:36.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.007+0000 7fb10d7fa700 1 -- 172.21.15.16:0/3582563619 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fb1000cde90 con 0x7fb110102480 2024-03-28T22:09:36.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.007+0000 7fb10f7fe700 1 --2- 172.21.15.16:0/3582563619 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb0f80a1ea0 0x7fb0f80a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:36.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.007+0000 7fb10f7fe700 1 --2- 172.21.15.16:0/3582563619 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb0f80a1ea0 0x7fb0f80a4360 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb10400c010 tx=0x7fb10401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:36.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.014+0000 7fb10d7fa700 1 -- 172.21.15.16:0/3582563619 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb10009b190 con 0x7fb110102480 2024-03-28T22:09:36.144 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574918 got 55834574922 for osd.1 2024-03-28T22:09:36.144 DEBUG:teuthology.parallel:result is None 2024-03-28T22:09:36.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.233+0000 7fb116298700 1 -- 172.21.15.16:0/3582563619 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fb11002db30 con 0x7fb110102480 2024-03-28T22:09:36.236 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:36 smithi016 ceph-mon[32374]: pgmap v321: 97 pgs: 2 activating, 95 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-28T22:09:36.236 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:36 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2817629544' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-28T22:09:36.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.233+0000 7fb10d7fa700 1 -- 172.21.15.16:0/3582563619 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fb10009aa90 con 0x7fb110102480 2024-03-28T22:09:36.237 INFO:teuthology.orchestra.run.smithi016.stdout:167503724600 2024-03-28T22:09:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.237+0000 7fb116298700 1 -- 172.21.15.16:0/3582563619 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb0f80a1ea0 msgr2=0x7fb0f80a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.237+0000 7fb116298700 1 --2- 172.21.15.16:0/3582563619 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb0f80a1ea0 0x7fb0f80a4360 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb10400c010 tx=0x7fb10401a040 comp rx=0 tx=0).stop 2024-03-28T22:09:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.237+0000 7fb116298700 1 -- 172.21.15.16:0/3582563619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb110102480 msgr2=0x7fb110074540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.237+0000 7fb116298700 1 --2- 172.21.15.16:0/3582563619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb110102480 0x7fb110074540 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb10000d950 tx=0x7fb10000dc60 comp rx=0 tx=0).stop 2024-03-28T22:09:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.237+0000 7fb116298700 1 -- 172.21.15.16:0/3582563619 shutdown_connections 2024-03-28T22:09:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.237+0000 7fb116298700 1 --2- 172.21.15.16:0/3582563619 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb0f80a1ea0 0x7fb0f80a4360 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.237+0000 7fb116298700 1 --2- 172.21.15.16:0/3582563619 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb110107560 0x7fb110074fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.237+0000 7fb116298700 1 --2- 172.21.15.16:0/3582563619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb110102480 0x7fb110074540 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.237+0000 7fb116298700 1 --2- 172.21.15.16:0/3582563619 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb110102e30 0x7fb110074a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.237+0000 7fb116298700 1 -- 172.21.15.16:0/3582563619 >> 172.21.15.16:0/3582563619 conn(0x7fb1100fdcf0 msgr2=0x7fb110100070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.237+0000 7fb116298700 1 -- 172.21.15.16:0/3582563619 shutdown_connections 2024-03-28T22:09:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.237+0000 7fb116298700 1 -- 172.21.15.16:0/3582563619 wait complete. 2024-03-28T22:09:36.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.336+0000 7fa6e94fe700 1 -- 172.21.15.16:0/1990388186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6e4108270 msgr2=0x7fa6e4108650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:36.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.336+0000 7fa6e94fe700 1 --2- 172.21.15.16:0/1990388186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6e4108270 0x7fa6e4108650 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa6d40099c0 tx=0x7fa6d4009cd0 comp rx=0 tx=0).stop 2024-03-28T22:09:36.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.336+0000 7fa6e94fe700 1 -- 172.21.15.16:0/1990388186 shutdown_connections 2024-03-28T22:09:36.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.336+0000 7fa6e94fe700 1 --2- 172.21.15.16:0/1990388186 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6e4102c50 0x7fa6e410f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:36.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.336+0000 7fa6e94fe700 1 --2- 172.21.15.16:0/1990388186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6e4108270 0x7fa6e4108650 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:36.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.336+0000 7fa6e94fe700 1 --2- 172.21.15.16:0/1990388186 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6e4102290 0x7fa6e4102710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:36.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.336+0000 7fa6e94fe700 1 -- 172.21.15.16:0/1990388186 >> 172.21.15.16:0/1990388186 conn(0x7fa6e4079370 msgr2=0x7fa6e4079780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:36.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.336+0000 7fa6e94fe700 1 -- 172.21.15.16:0/1990388186 shutdown_connections 2024-03-28T22:09:36.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.336+0000 7fa6e94fe700 1 -- 172.21.15.16:0/1990388186 wait complete. 2024-03-28T22:09:36.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.337+0000 7fa6e94fe700 1 Processor -- start 2024-03-28T22:09:36.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.337+0000 7fa6e94fe700 1 -- start start 2024-03-28T22:09:36.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.337+0000 7fa6e94fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6e4102290 0x7fa6e419e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:36.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.337+0000 7fa6e94fe700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6e4102c50 0x7fa6e419eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:36.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.337+0000 7fa6e94fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6e4108270 0x7fa6e4198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:36.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.337+0000 7fa6e94fe700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e41124f0 con 0x7fa6e4108270 2024-03-28T22:09:36.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.337+0000 7fa6e94fe700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e4112370 con 0x7fa6e4102290 2024-03-28T22:09:36.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.337+0000 7fa6e94fe700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e4112670 con 0x7fa6e4102c50 2024-03-28T22:09:36.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.338+0000 7fa6e37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6e4108270 0x7fa6e4198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:36.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.338+0000 7fa6e37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6e4108270 0x7fa6e4198790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60132/0 (socket says 172.21.15.16:60132) 2024-03-28T22:09:36.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.338+0000 7fa6e37fe700 1 -- 172.21.15.16:0/518913433 learned_addr learned my addr 172.21.15.16:0/518913433 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:36.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.338+0000 7fa6e2ffd700 1 --2- 172.21.15.16:0/518913433 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6e4102290 0x7fa6e419e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:36.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.338+0000 7fa6e27fc700 1 --2- 172.21.15.16:0/518913433 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6e4102c50 0x7fa6e419eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:36.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.338+0000 7fa6e37fe700 1 -- 172.21.15.16:0/518913433 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6e4102c50 msgr2=0x7fa6e419eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:36.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.338+0000 7fa6e37fe700 1 --2- 172.21.15.16:0/518913433 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6e4102c50 0x7fa6e419eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:36.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.338+0000 7fa6e37fe700 1 -- 172.21.15.16:0/518913433 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6e4102290 msgr2=0x7fa6e419e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:36.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.338+0000 7fa6e37fe700 1 --2- 172.21.15.16:0/518913433 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6e4102290 0x7fa6e419e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:36.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.338+0000 7fa6e37fe700 1 -- 172.21.15.16:0/518913433 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6e4198f70 con 0x7fa6e4108270 2024-03-28T22:09:36.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.338+0000 7fa6e27fc700 1 --2- 172.21.15.16:0/518913433 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6e4102c50 0x7fa6e419eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:09:36.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.338+0000 7fa6e37fe700 1 --2- 172.21.15.16:0/518913433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6e4108270 0x7fa6e4198790 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc007d70 tx=0x7fa6dc00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:36.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.338+0000 7fa6cbfff700 1 -- 172.21.15.16:0/518913433 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc016020 con 0x7fa6e4108270 2024-03-28T22:09:36.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.338+0000 7fa6e94fe700 1 -- 172.21.15.16:0/518913433 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6e4199260 con 0x7fa6e4108270 2024-03-28T22:09:36.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.338+0000 7fa6cbfff700 1 -- 172.21.15.16:0/518913433 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6dc00ad40 con 0x7fa6e4108270 2024-03-28T22:09:36.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.338+0000 7fa6cbfff700 1 -- 172.21.15.16:0/518913433 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc011600 con 0x7fa6e4108270 2024-03-28T22:09:36.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.338+0000 7fa6e94fe700 1 -- 172.21.15.16:0/518913433 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6e41a5470 con 0x7fa6e4108270 2024-03-28T22:09:36.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.339+0000 7fa6c9ffb700 1 -- 172.21.15.16:0/518913433 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6c4005330 con 0x7fa6e4108270 2024-03-28T22:09:36.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.340+0000 7fa6cbfff700 1 -- 172.21.15.16:0/518913433 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fa6dc019020 con 0x7fa6e4108270 2024-03-28T22:09:36.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.341+0000 7fa6cbfff700 1 --2- 172.21.15.16:0/518913433 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fa6cc0a1ea0 0x7fa6cc0a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:36.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.341+0000 7fa6e2ffd700 1 --2- 172.21.15.16:0/518913433 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fa6cc0a1ea0 0x7fa6cc0a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:36.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.341+0000 7fa6cbfff700 1 -- 172.21.15.16:0/518913433 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fa6dc0cdd00 con 0x7fa6e4108270 2024-03-28T22:09:36.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.341+0000 7fa6e2ffd700 1 --2- 172.21.15.16:0/518913433 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fa6cc0a1ea0 0x7fa6cc0a4360 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4000c00 tx=0x7fa6d4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:36.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.345+0000 7fa6cbfff700 1 -- 172.21.15.16:0/518913433 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa6dc020020 con 0x7fa6e4108270 2024-03-28T22:09:36.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:36 smithi063 ceph-mon[36249]: pgmap v321: 97 pgs: 2 activating, 95 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-28T22:09:36.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:36 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2817629544' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-28T22:09:36.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:36 smithi053 ceph-mon[35979]: pgmap v321: 97 pgs: 2 activating, 95 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-28T22:09:36.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:36 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2817629544' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-28T22:09:36.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.552+0000 7fa6c9ffb700 1 -- 172.21.15.16:0/518913433 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fa6c40058f0 con 0x7fa6e4108270 2024-03-28T22:09:36.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.553+0000 7fa6cbfff700 1 -- 172.21.15.16:0/518913433 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fa6dc09b110 con 0x7fa6e4108270 2024-03-28T22:09:36.555 INFO:teuthology.orchestra.run.smithi016.stdout:347892351002 2024-03-28T22:09:36.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.555+0000 7fa6c9ffb700 1 -- 172.21.15.16:0/518913433 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fa6cc0a1ea0 msgr2=0x7fa6cc0a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:36.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.555+0000 7fa6c9ffb700 1 --2- 172.21.15.16:0/518913433 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fa6cc0a1ea0 0x7fa6cc0a4360 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4000c00 tx=0x7fa6d4005fb0 comp rx=0 tx=0).stop 2024-03-28T22:09:36.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.555+0000 7fa6c9ffb700 1 -- 172.21.15.16:0/518913433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6e4108270 msgr2=0x7fa6e4198790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:36.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.555+0000 7fa6c9ffb700 1 --2- 172.21.15.16:0/518913433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6e4108270 0x7fa6e4198790 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc007d70 tx=0x7fa6dc00a340 comp rx=0 tx=0).stop 2024-03-28T22:09:36.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.555+0000 7fa6c9ffb700 1 -- 172.21.15.16:0/518913433 shutdown_connections 2024-03-28T22:09:36.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.555+0000 7fa6c9ffb700 1 --2- 172.21.15.16:0/518913433 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fa6cc0a1ea0 0x7fa6cc0a4360 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:36.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.555+0000 7fa6c9ffb700 1 --2- 172.21.15.16:0/518913433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6e4108270 0x7fa6e4198790 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:36.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.555+0000 7fa6c9ffb700 1 --2- 172.21.15.16:0/518913433 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6e4102290 0x7fa6e419e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:36.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.555+0000 7fa6c9ffb700 1 --2- 172.21.15.16:0/518913433 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6e4102c50 0x7fa6e419eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:36.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.555+0000 7fa6c9ffb700 1 -- 172.21.15.16:0/518913433 >> 172.21.15.16:0/518913433 conn(0x7fa6e4079370 msgr2=0x7fa6e4100eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:36.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.555+0000 7fa6c9ffb700 1 -- 172.21.15.16:0/518913433 shutdown_connections 2024-03-28T22:09:36.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:36.555+0000 7fa6c9ffb700 1 -- 172.21.15.16:0/518913433 wait complete. 2024-03-28T22:09:36.950 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724594 got 167503724600 for osd.5 2024-03-28T22:09:36.950 DEBUG:teuthology.parallel:result is None 2024-03-28T22:09:37.173 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:37 smithi016 ceph-mon[32374]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive) 2024-03-28T22:09:37.173 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:37 smithi016 ceph-mon[32374]: Cluster is now healthy 2024-03-28T22:09:37.173 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:37 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3582563619' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-28T22:09:37.173 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:37 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/518913433' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-28T22:09:37.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:37 smithi063 ceph-mon[36249]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive) 2024-03-28T22:09:37.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:37 smithi063 ceph-mon[36249]: Cluster is now healthy 2024-03-28T22:09:37.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:37 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/3582563619' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-28T22:09:37.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:37 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/518913433' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-28T22:09:37.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:37 smithi053 ceph-mon[35979]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive) 2024-03-28T22:09:37.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:37 smithi053 ceph-mon[35979]: Cluster is now healthy 2024-03-28T22:09:37.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:37 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3582563619' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-28T22:09:37.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:37 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/518913433' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-28T22:09:37.830 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892350997 got 347892351002 for osd.11 2024-03-28T22:09:37.830 DEBUG:teuthology.parallel:result is None 2024-03-28T22:09:38.117 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776158 got 292057776163 for osd.9 2024-03-28T22:09:38.117 DEBUG:teuthology.parallel:result is None 2024-03-28T22:09:38.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:38 smithi063 ceph-mon[36249]: pgmap v322: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 0 B/s, 0 objects/s recovering 2024-03-28T22:09:38.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:38 smithi016 ceph-mon[32374]: pgmap v322: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 0 B/s, 0 objects/s recovering 2024-03-28T22:09:38.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:38 smithi053 ceph-mon[35979]: pgmap v322: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 0 B/s, 0 objects/s recovering 2024-03-28T22:09:38.805 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888119 got 146028888125 for osd.4 2024-03-28T22:09:38.805 DEBUG:teuthology.parallel:result is None 2024-03-28T22:09:38.805 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-28T22:09:38.805 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-28T22:09:39.419 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:39 smithi016 ceph-mon[32374]: pgmap v323: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-28T22:09:39.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:39 smithi063 ceph-mon[36249]: pgmap v323: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-28T22:09:39.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:39 smithi053 ceph-mon[35979]: pgmap v323: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-28T22:09:39.570 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:09:41.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.074+0000 7f838f292700 1 -- 172.21.15.16:0/2278141032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8388101d70 msgr2=0x7f8388105dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:41.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.074+0000 7f838f292700 1 --2- 172.21.15.16:0/2278141032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8388101d70 0x7f8388105dc0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f83780099c0 tx=0x7f8378009cd0 comp rx=0 tx=0).stop 2024-03-28T22:09:41.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.074+0000 7f838f292700 1 -- 172.21.15.16:0/2278141032 shutdown_connections 2024-03-28T22:09:41.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.074+0000 7f838f292700 1 --2- 172.21.15.16:0/2278141032 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f83881064a0 0x7f838810e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:41.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.074+0000 7f838f292700 1 --2- 172.21.15.16:0/2278141032 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83881013c0 0x7f83881017a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:41.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.074+0000 7f838f292700 1 --2- 172.21.15.16:0/2278141032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8388101d70 0x7f8388105dc0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:41.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.074+0000 7f838f292700 1 -- 172.21.15.16:0/2278141032 >> 172.21.15.16:0/2278141032 conn(0x7f83880781a0 msgr2=0x7f83880785b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:41.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.074+0000 7f838f292700 1 -- 172.21.15.16:0/2278141032 shutdown_connections 2024-03-28T22:09:41.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.074+0000 7f838f292700 1 -- 172.21.15.16:0/2278141032 wait complete. 2024-03-28T22:09:41.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.075+0000 7f838f292700 1 Processor -- start 2024-03-28T22:09:41.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.075+0000 7f838f292700 1 -- start start 2024-03-28T22:09:41.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.075+0000 7f838f292700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83881013c0 0x7f83881998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:41.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.075+0000 7f838f292700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8388101d70 0x7f8388199e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:41.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.075+0000 7f838f292700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83881064a0 0x7f83881a4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:41.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.075+0000 7f838f292700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8388111480 con 0x7f83881013c0 2024-03-28T22:09:41.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.075+0000 7f838f292700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8388111300 con 0x7f83881064a0 2024-03-28T22:09:41.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.075+0000 7f838f292700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8388111600 con 0x7f8388101d70 2024-03-28T22:09:41.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.075+0000 7f838d82f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83881064a0 0x7f83881a4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:41.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.075+0000 7f838d02e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83881013c0 0x7f83881998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:41.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.075+0000 7f838c82d700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8388101d70 0x7f8388199e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:41.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.075+0000 7f838d82f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83881064a0 0x7f83881a4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:39042/0 (socket says 172.21.15.16:39042) 2024-03-28T22:09:41.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.075+0000 7f838d82f700 1 -- 172.21.15.16:0/4270085968 learned_addr learned my addr 172.21.15.16:0/4270085968 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:41.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.075+0000 7f838d02e700 1 -- 172.21.15.16:0/4270085968 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8388101d70 msgr2=0x7f8388199e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:41.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.075+0000 7f838d02e700 1 --2- 172.21.15.16:0/4270085968 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8388101d70 0x7f8388199e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:41.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.075+0000 7f838d02e700 1 -- 172.21.15.16:0/4270085968 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83881064a0 msgr2=0x7f83881a4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:41.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.075+0000 7f838d02e700 1 --2- 172.21.15.16:0/4270085968 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83881064a0 0x7f83881a4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:41.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.076+0000 7f838d02e700 1 -- 172.21.15.16:0/4270085968 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83881a4e80 con 0x7f83881013c0 2024-03-28T22:09:41.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.076+0000 7f838d82f700 1 --2- 172.21.15.16:0/4270085968 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83881064a0 0x7f83881a4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:09:41.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.076+0000 7f838c82d700 1 --2- 172.21.15.16:0/4270085968 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8388101d70 0x7f8388199e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:09:41.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.076+0000 7f838d02e700 1 --2- 172.21.15.16:0/4270085968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83881013c0 0x7f83881998c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f838400e9f0 tx=0x7f838400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:41.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.076+0000 7f837e7fc700 1 -- 172.21.15.16:0/4270085968 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838400c9c0 con 0x7f83881013c0 2024-03-28T22:09:41.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.076+0000 7f838f292700 1 -- 172.21.15.16:0/4270085968 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83881a50b0 con 0x7f83881013c0 2024-03-28T22:09:41.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.076+0000 7f837e7fc700 1 -- 172.21.15.16:0/4270085968 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f838400cb60 con 0x7f83881013c0 2024-03-28T22:09:41.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.076+0000 7f837e7fc700 1 -- 172.21.15.16:0/4270085968 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8384018680 con 0x7f83881013c0 2024-03-28T22:09:41.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.076+0000 7f838f292700 1 -- 172.21.15.16:0/4270085968 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83881a5610 con 0x7f83881013c0 2024-03-28T22:09:41.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.077+0000 7f838f292700 1 -- 172.21.15.16:0/4270085968 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8388068f00 con 0x7f83881013c0 2024-03-28T22:09:41.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.078+0000 7f837e7fc700 1 -- 172.21.15.16:0/4270085968 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f8384010430 con 0x7f83881013c0 2024-03-28T22:09:41.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.079+0000 7f837e7fc700 1 --2- 172.21.15.16:0/4270085968 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f83740a1ea0 0x7f83740a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:41.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.079+0000 7f838c82d700 1 --2- 172.21.15.16:0/4270085968 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f83740a1ea0 0x7f83740a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:41.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.079+0000 7f837e7fc700 1 -- 172.21.15.16:0/4270085968 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f83840cdf70 con 0x7f83881013c0 2024-03-28T22:09:41.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.079+0000 7f838c82d700 1 --2- 172.21.15.16:0/4270085968 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f83740a1ea0 0x7f83740a4360 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8378000c00 tx=0x7f8378005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:41.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.084+0000 7f837e7fc700 1 -- 172.21.15.16:0/4270085968 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f838409b310 con 0x7f83881013c0 2024-03-28T22:09:41.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.320+0000 7f838f292700 1 -- 172.21.15.16:0/4270085968 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8388062680 con 0x7f83740a1ea0 2024-03-28T22:09:41.324 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:41 smithi016 ceph-mon[32374]: pgmap v324: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-28T22:09:41.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.329+0000 7f837e7fc700 1 -- 172.21.15.16:0/4270085968 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258702 (secure 0 0 0) 0x7f8388062680 con 0x7f83740a1ea0 2024-03-28T22:09:41.335 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:09:41.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.332+0000 7f838f292700 1 -- 172.21.15.16:0/4270085968 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f83740a1ea0 msgr2=0x7f83740a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:41.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.332+0000 7f838f292700 1 --2- 172.21.15.16:0/4270085968 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f83740a1ea0 0x7f83740a4360 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8378000c00 tx=0x7f8378005fb0 comp rx=0 tx=0).stop 2024-03-28T22:09:41.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.332+0000 7f838f292700 1 -- 172.21.15.16:0/4270085968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83881013c0 msgr2=0x7f83881998c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:41.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.332+0000 7f838f292700 1 --2- 172.21.15.16:0/4270085968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83881013c0 0x7f83881998c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f838400e9f0 tx=0x7f838400ed00 comp rx=0 tx=0).stop 2024-03-28T22:09:41.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.332+0000 7f838f292700 1 -- 172.21.15.16:0/4270085968 shutdown_connections 2024-03-28T22:09:41.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.332+0000 7f838f292700 1 --2- 172.21.15.16:0/4270085968 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f83740a1ea0 0x7f83740a4360 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:41.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.332+0000 7f838f292700 1 --2- 172.21.15.16:0/4270085968 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83881064a0 0x7f83881a4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:41.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.332+0000 7f838f292700 1 --2- 172.21.15.16:0/4270085968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83881013c0 0x7f83881998c0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:41.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.332+0000 7f838f292700 1 --2- 172.21.15.16:0/4270085968 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8388101d70 0x7f8388199e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:41.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.332+0000 7f838f292700 1 -- 172.21.15.16:0/4270085968 >> 172.21.15.16:0/4270085968 conn(0x7f83880781a0 msgr2=0x7f83880ff170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:41.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.332+0000 7f838f292700 1 -- 172.21.15.16:0/4270085968 shutdown_connections 2024-03-28T22:09:41.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:41.332+0000 7f838f292700 1 -- 172.21.15.16:0/4270085968 wait complete. 2024-03-28T22:09:41.338 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-03-28T22:09:41.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:41 smithi063 ceph-mon[36249]: pgmap v324: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-28T22:09:41.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:41 smithi053 ceph-mon[35979]: pgmap v324: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-28T22:09:41.988 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":324,"stamp":"2024-03-28T22:09:41.090364+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653906,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":553,"num_read_kb":393,"num_write":338,"num_write_kb":3046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":273,"ondisk_log_size":273,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":379132,"kb_used_data":13260,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124562692,"statfs":{"total":1151940427776,"available":1151552196608,"internally_reserved":0,"allocated":13578240,"data_stored":7464666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":374453562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":65,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.037452"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.056315+0000","last_change":"2024-03-28T22:08:29.962664+0000","last_active":"2024-03-28T22:09:30.056315+0000","last_peered":"2024-03-28T22:09:30.056315+0000","last_clean":"2024-03-28T22:09:30.056315+0000","last_became_active":"2024-03-28T22:08:29.962029+0000","last_became_peered":"2024-03-28T22:08:29.962029+0000","last_unstale":"2024-03-28T22:09:30.056315+0000","last_undegraded":"2024-03-28T22:09:30.056315+0000","last_fullsized":"2024-03-28T22:09:30.056315+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:20:42.832304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.059447+0000","last_change":"2024-03-28T22:08:29.965617+0000","last_active":"2024-03-28T22:09:30.059447+0000","last_peered":"2024-03-28T22:09:30.059447+0000","last_clean":"2024-03-28T22:09:30.059447+0000","last_became_active":"2024-03-28T22:08:29.965248+0000","last_became_peered":"2024-03-28T22:08:29.965248+0000","last_unstale":"2024-03-28T22:09:30.059447+0000","last_undegraded":"2024-03-28T22:09:30.059447+0000","last_fullsized":"2024-03-28T22:09:30.059447+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:37:46.703768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.059158+0000","last_change":"2024-03-28T22:08:29.958452+0000","last_active":"2024-03-28T22:09:30.059158+0000","last_peered":"2024-03-28T22:09:30.059158+0000","last_clean":"2024-03-28T22:09:30.059158+0000","last_became_active":"2024-03-28T22:08:29.958089+0000","last_became_peered":"2024-03-28T22:08:29.958089+0000","last_unstale":"2024-03-28T22:09:30.059158+0000","last_undegraded":"2024-03-28T22:09:30.059158+0000","last_fullsized":"2024-03-28T22:09:30.059158+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:10:44.084925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.079508+0000","last_change":"2024-03-28T22:08:29.964812+0000","last_active":"2024-03-28T22:09:30.079508+0000","last_peered":"2024-03-28T22:09:30.079508+0000","last_clean":"2024-03-28T22:09:30.079508+0000","last_became_active":"2024-03-28T22:08:29.963872+0000","last_became_peered":"2024-03-28T22:08:29.963872+0000","last_unstale":"2024-03-28T22:09:30.079508+0000","last_undegraded":"2024-03-28T22:09:30.079508+0000","last_fullsized":"2024-03-28T22:09:30.079508+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:08.491078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.057748+0000","last_change":"2024-03-28T22:08:29.965564+0000","last_active":"2024-03-28T22:09:30.057748+0000","last_peered":"2024-03-28T22:09:30.057748+0000","last_clean":"2024-03-28T22:09:30.057748+0000","last_became_active":"2024-03-28T22:08:29.962636+0000","last_became_peered":"2024-03-28T22:08:29.962636+0000","last_unstale":"2024-03-28T22:09:30.057748+0000","last_undegraded":"2024-03-28T22:09:30.057748+0000","last_fullsized":"2024-03-28T22:09:30.057748+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:03:50.903231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.425116+0000","last_change":"2024-03-28T22:08:29.962091+0000","last_active":"2024-03-28T22:09:30.425116+0000","last_peered":"2024-03-28T22:09:30.425116+0000","last_clean":"2024-03-28T22:09:30.425116+0000","last_became_active":"2024-03-28T22:08:29.961735+0000","last_became_peered":"2024-03-28T22:08:29.961735+0000","last_unstale":"2024-03-28T22:09:30.425116+0000","last_undegraded":"2024-03-28T22:09:30.425116+0000","last_fullsized":"2024-03-28T22:09:30.425116+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:40:09.853265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.054864+0000","last_change":"2024-03-28T22:08:29.961089+0000","last_active":"2024-03-28T22:09:30.054864+0000","last_peered":"2024-03-28T22:09:30.054864+0000","last_clean":"2024-03-28T22:09:30.054864+0000","last_became_active":"2024-03-28T22:08:29.960711+0000","last_became_peered":"2024-03-28T22:08:29.960711+0000","last_unstale":"2024-03-28T22:09:30.054864+0000","last_undegraded":"2024-03-28T22:09:30.054864+0000","last_fullsized":"2024-03-28T22:09:30.054864+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:05:14.617718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.056921+0000","last_change":"2024-03-28T22:08:29.965142+0000","last_active":"2024-03-28T22:09:30.056921+0000","last_peered":"2024-03-28T22:09:30.056921+0000","last_clean":"2024-03-28T22:09:30.056921+0000","last_became_active":"2024-03-28T22:08:29.964730+0000","last_became_peered":"2024-03-28T22:08:29.964730+0000","last_unstale":"2024-03-28T22:09:30.056921+0000","last_undegraded":"2024-03-28T22:09:30.056921+0000","last_fullsized":"2024-03-28T22:09:30.056921+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:23:43.229681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.425637+0000","last_change":"2024-03-28T22:08:29.958984+0000","last_active":"2024-03-28T22:09:30.425637+0000","last_peered":"2024-03-28T22:09:30.425637+0000","last_clean":"2024-03-28T22:09:30.425637+0000","last_became_active":"2024-03-28T22:08:29.958583+0000","last_became_peered":"2024-03-28T22:08:29.958583+0000","last_unstale":"2024-03-28T22:09:30.425637+0000","last_undegraded":"2024-03-28T22:09:30.425637+0000","last_fullsized":"2024-03-28T22:09:30.425637+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:05:11.361719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.425101+0000","last_change":"2024-03-28T22:08:29.962774+0000","last_active":"2024-03-28T22:09:30.425101+0000","last_peered":"2024-03-28T22:09:30.425101+0000","last_clean":"2024-03-28T22:09:30.425101+0000","last_became_active":"2024-03-28T22:08:29.961222+0000","last_became_peered":"2024-03-28T22:08:29.961222+0000","last_unstale":"2024-03-28T22:09:30.425101+0000","last_undegraded":"2024-03-28T22:09:30.425101+0000","last_fullsized":"2024-03-28T22:09:30.425101+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:47:05.310483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.055610+0000","last_change":"2024-03-28T22:08:29.964041+0000","last_active":"2024-03-28T22:09:30.055610+0000","last_peered":"2024-03-28T22:09:30.055610+0000","last_clean":"2024-03-28T22:09:30.055610+0000","last_became_active":"2024-03-28T22:08:29.963360+0000","last_became_peered":"2024-03-28T22:08:29.963360+0000","last_unstale":"2024-03-28T22:09:30.055610+0000","last_undegraded":"2024-03-28T22:09:30.055610+0000","last_fullsized":"2024-03-28T22:09:30.055610+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:32.921916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.079414+0000","last_change":"2024-03-28T22:08:29.966449+0000","last_active":"2024-03-28T22:09:30.079414+0000","last_peered":"2024-03-28T22:09:30.079414+0000","last_clean":"2024-03-28T22:09:30.079414+0000","last_became_active":"2024-03-28T22:08:29.966036+0000","last_became_peered":"2024-03-28T22:08:29.966036+0000","last_unstale":"2024-03-28T22:09:30.079414+0000","last_undegraded":"2024-03-28T22:09:30.079414+0000","last_fullsized":"2024-03-28T22:09:30.079414+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:12:15.057713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.293856+0000","last_change":"2024-03-28T22:08:29.958924+0000","last_active":"2024-03-28T22:09:30.293856+0000","last_peered":"2024-03-28T22:09:30.293856+0000","last_clean":"2024-03-28T22:09:30.293856+0000","last_became_active":"2024-03-28T22:08:29.958601+0000","last_became_peered":"2024-03-28T22:08:29.958601+0000","last_unstale":"2024-03-28T22:09:30.293856+0000","last_undegraded":"2024-03-28T22:09:30.293856+0000","last_fullsized":"2024-03-28T22:09:30.293856+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:53:37.070276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058918+0000","last_change":"2024-03-28T22:08:29.965045+0000","last_active":"2024-03-28T22:09:30.058918+0000","last_peered":"2024-03-28T22:09:30.058918+0000","last_clean":"2024-03-28T22:09:30.058918+0000","last_became_active":"2024-03-28T22:08:29.964710+0000","last_became_peered":"2024-03-28T22:08:29.964710+0000","last_unstale":"2024-03-28T22:09:30.058918+0000","last_undegraded":"2024-03-28T22:09:30.058918+0000","last_fullsized":"2024-03-28T22:09:30.058918+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:03:11.491213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058745+0000","last_change":"2024-03-28T22:08:29.957940+0000","last_active":"2024-03-28T22:09:30.058745+0000","last_peered":"2024-03-28T22:09:30.058745+0000","last_clean":"2024-03-28T22:09:30.058745+0000","last_became_active":"2024-03-28T22:08:29.957587+0000","last_became_peered":"2024-03-28T22:08:29.957587+0000","last_unstale":"2024-03-28T22:09:30.058745+0000","last_undegraded":"2024-03-28T22:09:30.058745+0000","last_fullsized":"2024-03-28T22:09:30.058745+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:03:58.557806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.424536+0000","last_change":"2024-03-28T22:08:29.957397+0000","last_active":"2024-03-28T22:09:30.424536+0000","last_peered":"2024-03-28T22:09:30.424536+0000","last_clean":"2024-03-28T22:09:30.424536+0000","last_became_active":"2024-03-28T22:08:29.957054+0000","last_became_peered":"2024-03-28T22:08:29.957054+0000","last_unstale":"2024-03-28T22:09:30.424536+0000","last_undegraded":"2024-03-28T22:09:30.424536+0000","last_fullsized":"2024-03-28T22:09:30.424536+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:35:28.500561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.056537+0000","last_change":"2024-03-28T22:08:29.963343+0000","last_active":"2024-03-28T22:09:30.056537+0000","last_peered":"2024-03-28T22:09:30.056537+0000","last_clean":"2024-03-28T22:09:30.056537+0000","last_became_active":"2024-03-28T22:08:29.962926+0000","last_became_peered":"2024-03-28T22:08:29.962926+0000","last_unstale":"2024-03-28T22:09:30.056537+0000","last_undegraded":"2024-03-28T22:09:30.056537+0000","last_fullsized":"2024-03-28T22:09:30.056537+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:15:39.133930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.054505+0000","last_change":"2024-03-28T22:08:29.955903+0000","last_active":"2024-03-28T22:09:30.054505+0000","last_peered":"2024-03-28T22:09:30.054505+0000","last_clean":"2024-03-28T22:09:30.054505+0000","last_became_active":"2024-03-28T22:08:29.955460+0000","last_became_peered":"2024-03-28T22:08:29.955460+0000","last_unstale":"2024-03-28T22:09:30.054505+0000","last_undegraded":"2024-03-28T22:09:30.054505+0000","last_fullsized":"2024-03-28T22:09:30.054505+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:36:12.961475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058082+0000","last_change":"2024-03-28T22:08:29.954279+0000","last_active":"2024-03-28T22:09:30.058082+0000","last_peered":"2024-03-28T22:09:30.058082+0000","last_clean":"2024-03-28T22:09:30.058082+0000","last_became_active":"2024-03-28T22:08:29.953869+0000","last_became_peered":"2024-03-28T22:08:29.953869+0000","last_unstale":"2024-03-28T22:09:30.058082+0000","last_undegraded":"2024-03-28T22:09:30.058082+0000","last_fullsized":"2024-03-28T22:09:30.058082+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:05:52.247391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.054735+0000","last_change":"2024-03-28T22:08:29.327644+0000","last_active":"2024-03-28T22:09:30.054735+0000","last_peered":"2024-03-28T22:09:30.054735+0000","last_clean":"2024-03-28T22:09:30.054735+0000","last_became_active":"2024-03-28T22:08:29.327325+0000","last_became_peered":"2024-03-28T22:08:29.327325+0000","last_unstale":"2024-03-28T22:09:30.054735+0000","last_undegraded":"2024-03-28T22:09:30.054735+0000","last_fullsized":"2024-03-28T22:09:30.054735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:26:54.782703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058093+0000","last_change":"2024-03-28T22:08:29.962905+0000","last_active":"2024-03-28T22:09:30.058093+0000","last_peered":"2024-03-28T22:09:30.058093+0000","last_clean":"2024-03-28T22:09:30.058093+0000","last_became_active":"2024-03-28T22:08:29.962559+0000","last_became_peered":"2024-03-28T22:08:29.962559+0000","last_unstale":"2024-03-28T22:09:30.058093+0000","last_undegraded":"2024-03-28T22:09:30.058093+0000","last_fullsized":"2024-03-28T22:09:30.058093+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:02.347584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058126+0000","last_change":"2024-03-28T22:08:29.334871+0000","last_active":"2024-03-28T22:09:30.058126+0000","last_peered":"2024-03-28T22:09:30.058126+0000","last_clean":"2024-03-28T22:09:30.058126+0000","last_became_active":"2024-03-28T22:08:29.334565+0000","last_became_peered":"2024-03-28T22:08:29.334565+0000","last_unstale":"2024-03-28T22:09:30.058126+0000","last_undegraded":"2024-03-28T22:09:30.058126+0000","last_fullsized":"2024-03-28T22:09:30.058126+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:31:19.333496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.059794+0000","last_change":"2024-03-28T22:08:29.957622+0000","last_active":"2024-03-28T22:09:30.059794+0000","last_peered":"2024-03-28T22:09:30.059794+0000","last_clean":"2024-03-28T22:09:30.059794+0000","last_became_active":"2024-03-28T22:08:29.957208+0000","last_became_peered":"2024-03-28T22:08:29.957208+0000","last_unstale":"2024-03-28T22:09:30.059794+0000","last_undegraded":"2024-03-28T22:09:30.059794+0000","last_fullsized":"2024-03-28T22:09:30.059794+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:49:44.684203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058576+0000","last_change":"2024-03-28T22:08:29.330291+0000","last_active":"2024-03-28T22:09:30.058576+0000","last_peered":"2024-03-28T22:09:30.058576+0000","last_clean":"2024-03-28T22:09:30.058576+0000","last_became_active":"2024-03-28T22:08:29.328955+0000","last_became_peered":"2024-03-28T22:08:29.328955+0000","last_unstale":"2024-03-28T22:09:30.058576+0000","last_undegraded":"2024-03-28T22:09:30.058576+0000","last_fullsized":"2024-03-28T22:09:30.058576+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:10.838059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.424640+0000","last_change":"2024-03-28T22:08:29.966681+0000","last_active":"2024-03-28T22:09:30.424640+0000","last_peered":"2024-03-28T22:09:30.424640+0000","last_clean":"2024-03-28T22:09:30.424640+0000","last_became_active":"2024-03-28T22:08:29.966023+0000","last_became_peered":"2024-03-28T22:08:29.966023+0000","last_unstale":"2024-03-28T22:09:30.424640+0000","last_undegraded":"2024-03-28T22:09:30.424640+0000","last_fullsized":"2024-03-28T22:09:30.424640+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:04:43.281259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058443+0000","last_change":"2024-03-28T22:08:29.329026+0000","last_active":"2024-03-28T22:09:30.058443+0000","last_peered":"2024-03-28T22:09:30.058443+0000","last_clean":"2024-03-28T22:09:30.058443+0000","last_became_active":"2024-03-28T22:08:29.327570+0000","last_became_peered":"2024-03-28T22:08:29.327570+0000","last_unstale":"2024-03-28T22:09:30.058443+0000","last_undegraded":"2024-03-28T22:09:30.058443+0000","last_fullsized":"2024-03-28T22:09:30.058443+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:53:54.779073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.060453+0000","last_change":"2024-03-28T22:08:29.966455+0000","last_active":"2024-03-28T22:09:30.060453+0000","last_peered":"2024-03-28T22:09:30.060453+0000","last_clean":"2024-03-28T22:09:30.060453+0000","last_became_active":"2024-03-28T22:08:29.964451+0000","last_became_peered":"2024-03-28T22:08:29.964451+0000","last_unstale":"2024-03-28T22:09:30.060453+0000","last_undegraded":"2024-03-28T22:09:30.060453+0000","last_fullsized":"2024-03-28T22:09:30.060453+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:46:22.811146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'6","reported_seq":95,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:39.441567+0000","last_change":"2024-03-28T22:08:29.327397+0000","last_active":"2024-03-28T22:09:39.441567+0000","last_peered":"2024-03-28T22:09:39.441567+0000","last_clean":"2024-03-28T22:09:39.441567+0000","last_became_active":"2024-03-28T22:08:29.327009+0000","last_became_peered":"2024-03-28T22:08:29.327009+0000","last_unstale":"2024-03-28T22:09:39.441567+0000","last_undegraded":"2024-03-28T22:09:39.441567+0000","last_fullsized":"2024-03-28T22:09:39.441567+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:38:51.951934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058110+0000","last_change":"2024-03-28T22:08:29.969821+0000","last_active":"2024-03-28T22:09:30.058110+0000","last_peered":"2024-03-28T22:09:30.058110+0000","last_clean":"2024-03-28T22:09:30.058110+0000","last_became_active":"2024-03-28T22:08:29.969386+0000","last_became_peered":"2024-03-28T22:08:29.969386+0000","last_unstale":"2024-03-28T22:09:30.058110+0000","last_undegraded":"2024-03-28T22:09:30.058110+0000","last_fullsized":"2024-03-28T22:09:30.058110+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:44:19.753083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.424985+0000","last_change":"2024-03-28T22:08:29.332385+0000","last_active":"2024-03-28T22:09:30.424985+0000","last_peered":"2024-03-28T22:09:30.424985+0000","last_clean":"2024-03-28T22:09:30.424985+0000","last_became_active":"2024-03-28T22:08:29.331956+0000","last_became_peered":"2024-03-28T22:08:29.331956+0000","last_unstale":"2024-03-28T22:09:30.424985+0000","last_undegraded":"2024-03-28T22:09:30.424985+0000","last_fullsized":"2024-03-28T22:09:30.424985+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:10:04.428034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.080320+0000","last_change":"2024-03-28T22:09:30.080320+0000","last_active":"2024-03-28T22:09:30.080320+0000","last_peered":"2024-03-28T22:09:30.080320+0000","last_clean":"2024-03-28T22:09:30.080320+0000","last_became_active":"2024-03-28T22:09:30.079928+0000","last_became_peered":"2024-03-28T22:09:30.079928+0000","last_unstale":"2024-03-28T22:09:30.080320+0000","last_undegraded":"2024-03-28T22:09:30.080320+0000","last_fullsized":"2024-03-28T22:09:30.080320+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:12:53.385167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.293420+0000","last_change":"2024-03-28T22:08:29.327180+0000","last_active":"2024-03-28T22:09:30.293420+0000","last_peered":"2024-03-28T22:09:30.293420+0000","last_clean":"2024-03-28T22:09:30.293420+0000","last_became_active":"2024-03-28T22:08:29.326825+0000","last_became_peered":"2024-03-28T22:08:29.326825+0000","last_unstale":"2024-03-28T22:09:30.293420+0000","last_undegraded":"2024-03-28T22:09:30.293420+0000","last_fullsized":"2024-03-28T22:09:30.293420+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:11:09.979970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.057215+0000","last_change":"2024-03-28T22:08:29.955299+0000","last_active":"2024-03-28T22:09:30.057215+0000","last_peered":"2024-03-28T22:09:30.057215+0000","last_clean":"2024-03-28T22:09:30.057215+0000","last_became_active":"2024-03-28T22:08:29.954949+0000","last_became_peered":"2024-03-28T22:08:29.954949+0000","last_unstale":"2024-03-28T22:09:30.057215+0000","last_undegraded":"2024-03-28T22:09:30.057215+0000","last_fullsized":"2024-03-28T22:09:30.057215+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:01:48.375251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.059530+0000","last_change":"2024-03-28T22:08:29.327261+0000","last_active":"2024-03-28T22:09:30.059530+0000","last_peered":"2024-03-28T22:09:30.059530+0000","last_clean":"2024-03-28T22:09:30.059530+0000","last_became_active":"2024-03-28T22:08:29.326910+0000","last_became_peered":"2024-03-28T22:08:29.326910+0000","last_unstale":"2024-03-28T22:09:30.059530+0000","last_undegraded":"2024-03-28T22:09:30.059530+0000","last_fullsized":"2024-03-28T22:09:30.059530+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:46:41.933372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.054198+0000","last_change":"2024-03-28T22:08:29.954891+0000","last_active":"2024-03-28T22:09:30.054198+0000","last_peered":"2024-03-28T22:09:30.054198+0000","last_clean":"2024-03-28T22:09:30.054198+0000","last_became_active":"2024-03-28T22:08:29.954519+0000","last_became_peered":"2024-03-28T22:08:29.954519+0000","last_unstale":"2024-03-28T22:09:30.054198+0000","last_undegraded":"2024-03-28T22:09:30.054198+0000","last_fullsized":"2024-03-28T22:09:30.054198+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:07:42.621189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":88,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:38.440128+0000","last_change":"2024-03-28T22:08:29.330180+0000","last_active":"2024-03-28T22:09:38.440128+0000","last_peered":"2024-03-28T22:09:38.440128+0000","last_clean":"2024-03-28T22:09:38.440128+0000","last_became_active":"2024-03-28T22:08:29.329835+0000","last_became_peered":"2024-03-28T22:08:29.329835+0000","last_unstale":"2024-03-28T22:09:38.440128+0000","last_undegraded":"2024-03-28T22:09:38.440128+0000","last_fullsized":"2024-03-28T22:09:38.440128+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:48:39.906359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":66,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.425037+0000","last_change":"2024-03-28T22:08:29.967074+0000","last_active":"2024-03-28T22:09:30.425037+0000","last_peered":"2024-03-28T22:09:30.425037+0000","last_clean":"2024-03-28T22:09:30.425037+0000","last_became_active":"2024-03-28T22:08:29.966258+0000","last_became_peered":"2024-03-28T22:08:29.966258+0000","last_unstale":"2024-03-28T22:09:30.425037+0000","last_undegraded":"2024-03-28T22:09:30.425037+0000","last_fullsized":"2024-03-28T22:09:30.425037+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:25:44.986179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.055474+0000","last_change":"2024-03-28T22:08:29.326388+0000","last_active":"2024-03-28T22:09:30.055474+0000","last_peered":"2024-03-28T22:09:30.055474+0000","last_clean":"2024-03-28T22:09:30.055474+0000","last_became_active":"2024-03-28T22:08:29.326001+0000","last_became_peered":"2024-03-28T22:08:29.326001+0000","last_unstale":"2024-03-28T22:09:30.055474+0000","last_undegraded":"2024-03-28T22:09:30.055474+0000","last_fullsized":"2024-03-28T22:09:30.055474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:31:56.227038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.059682+0000","last_change":"2024-03-28T22:08:29.959299+0000","last_active":"2024-03-28T22:09:30.059682+0000","last_peered":"2024-03-28T22:09:30.059682+0000","last_clean":"2024-03-28T22:09:30.059682+0000","last_became_active":"2024-03-28T22:08:29.958955+0000","last_became_peered":"2024-03-28T22:08:29.958955+0000","last_unstale":"2024-03-28T22:09:30.059682+0000","last_undegraded":"2024-03-28T22:09:30.059682+0000","last_fullsized":"2024-03-28T22:09:30.059682+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:11:06.775920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.060567+0000","last_change":"2024-03-28T22:08:29.326089+0000","last_active":"2024-03-28T22:09:30.060567+0000","last_peered":"2024-03-28T22:09:30.060567+0000","last_clean":"2024-03-28T22:09:30.060567+0000","last_became_active":"2024-03-28T22:08:29.325463+0000","last_became_peered":"2024-03-28T22:08:29.325463+0000","last_unstale":"2024-03-28T22:09:30.060567+0000","last_undegraded":"2024-03-28T22:09:30.060567+0000","last_fullsized":"2024-03-28T22:09:30.060567+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:18:31.049729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.425439+0000","last_change":"2024-03-28T22:08:29.964248+0000","last_active":"2024-03-28T22:09:30.425439+0000","last_peered":"2024-03-28T22:09:30.425439+0000","last_clean":"2024-03-28T22:09:30.425439+0000","last_became_active":"2024-03-28T22:08:29.963897+0000","last_became_peered":"2024-03-28T22:08:29.963897+0000","last_unstale":"2024-03-28T22:09:30.425439+0000","last_undegraded":"2024-03-28T22:09:30.425439+0000","last_fullsized":"2024-03-28T22:09:30.425439+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:41:39.645360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'26","reported_seq":184,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:39.441151+0000","last_change":"2024-03-28T22:08:29.330732+0000","last_active":"2024-03-28T22:09:39.441151+0000","last_peered":"2024-03-28T22:09:39.441151+0000","last_clean":"2024-03-28T22:09:39.441151+0000","last_became_active":"2024-03-28T22:08:29.329231+0000","last_became_peered":"2024-03-28T22:08:29.329231+0000","last_unstale":"2024-03-28T22:09:39.441151+0000","last_undegraded":"2024-03-28T22:09:39.441151+0000","last_fullsized":"2024-03-28T22:09:39.441151+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:16:39.260429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58912,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":139,"num_read_kb":126,"num_write":27,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058333+0000","last_change":"2024-03-28T22:08:29.965791+0000","last_active":"2024-03-28T22:09:30.058333+0000","last_peered":"2024-03-28T22:09:30.058333+0000","last_clean":"2024-03-28T22:09:30.058333+0000","last_became_active":"2024-03-28T22:08:29.965390+0000","last_became_peered":"2024-03-28T22:08:29.965390+0000","last_unstale":"2024-03-28T22:09:30.058333+0000","last_undegraded":"2024-03-28T22:09:30.058333+0000","last_fullsized":"2024-03-28T22:09:30.058333+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:43:53.325906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.057898+0000","last_change":"2024-03-28T22:08:29.325851+0000","last_active":"2024-03-28T22:09:30.057898+0000","last_peered":"2024-03-28T22:09:30.057898+0000","last_clean":"2024-03-28T22:09:30.057898+0000","last_became_active":"2024-03-28T22:08:29.325500+0000","last_became_peered":"2024-03-28T22:08:29.325500+0000","last_unstale":"2024-03-28T22:09:30.057898+0000","last_undegraded":"2024-03-28T22:09:30.057898+0000","last_fullsized":"2024-03-28T22:09:30.057898+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:04:25.151908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.293686+0000","last_change":"2024-03-28T22:08:29.953857+0000","last_active":"2024-03-28T22:09:30.293686+0000","last_peered":"2024-03-28T22:09:30.293686+0000","last_clean":"2024-03-28T22:09:30.293686+0000","last_became_active":"2024-03-28T22:08:29.953538+0000","last_became_peered":"2024-03-28T22:08:29.953538+0000","last_unstale":"2024-03-28T22:09:30.293686+0000","last_undegraded":"2024-03-28T22:09:30.293686+0000","last_fullsized":"2024-03-28T22:09:30.293686+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:40:32.404760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.060057+0000","last_change":"2024-03-28T22:08:29.327286+0000","last_active":"2024-03-28T22:09:30.060057+0000","last_peered":"2024-03-28T22:09:30.060057+0000","last_clean":"2024-03-28T22:09:30.060057+0000","last_became_active":"2024-03-28T22:08:29.326902+0000","last_became_peered":"2024-03-28T22:08:29.326902+0000","last_unstale":"2024-03-28T22:09:30.060057+0000","last_undegraded":"2024-03-28T22:09:30.060057+0000","last_fullsized":"2024-03-28T22:09:30.060057+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:14:58.669534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.294946+0000","last_change":"2024-03-28T22:09:30.294946+0000","last_active":"2024-03-28T22:09:30.294946+0000","last_peered":"2024-03-28T22:09:30.294946+0000","last_clean":"2024-03-28T22:09:30.294946+0000","last_became_active":"2024-03-28T22:09:30.294496+0000","last_became_peered":"2024-03-28T22:09:30.294496+0000","last_unstale":"2024-03-28T22:09:30.294946+0000","last_undegraded":"2024-03-28T22:09:30.294946+0000","last_fullsized":"2024-03-28T22:09:30.294946+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:26:40.062384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'4","reported_seq":89,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:35.436529+0000","last_change":"2024-03-28T22:08:29.327740+0000","last_active":"2024-03-28T22:09:35.436529+0000","last_peered":"2024-03-28T22:09:35.436529+0000","last_clean":"2024-03-28T22:09:35.436529+0000","last_became_active":"2024-03-28T22:08:29.326664+0000","last_became_peered":"2024-03-28T22:08:29.326664+0000","last_unstale":"2024-03-28T22:09:35.436529+0000","last_undegraded":"2024-03-28T22:09:35.436529+0000","last_fullsized":"2024-03-28T22:09:35.436529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:58:17.681836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":67,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.054778+0000","last_change":"2024-03-28T22:08:29.961979+0000","last_active":"2024-03-28T22:09:30.054778+0000","last_peered":"2024-03-28T22:09:30.054778+0000","last_clean":"2024-03-28T22:09:30.054778+0000","last_became_active":"2024-03-28T22:08:29.961615+0000","last_became_peered":"2024-03-28T22:08:29.961615+0000","last_unstale":"2024-03-28T22:09:30.054778+0000","last_undegraded":"2024-03-28T22:09:30.054778+0000","last_fullsized":"2024-03-28T22:09:30.054778+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:50:37.227632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.057515+0000","last_change":"2024-03-28T22:08:29.329017+0000","last_active":"2024-03-28T22:09:30.057515+0000","last_peered":"2024-03-28T22:09:30.057515+0000","last_clean":"2024-03-28T22:09:30.057515+0000","last_became_active":"2024-03-28T22:08:29.328686+0000","last_became_peered":"2024-03-28T22:08:29.328686+0000","last_unstale":"2024-03-28T22:09:30.057515+0000","last_undegraded":"2024-03-28T22:09:30.057515+0000","last_fullsized":"2024-03-28T22:09:30.057515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:25:28.986165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.426320+0000","last_change":"2024-03-28T22:09:30.426320+0000","last_active":"2024-03-28T22:09:30.426320+0000","last_peered":"2024-03-28T22:09:30.426320+0000","last_clean":"2024-03-28T22:09:30.426320+0000","last_became_active":"2024-03-28T22:09:30.425862+0000","last_became_peered":"2024-03-28T22:09:30.425862+0000","last_unstale":"2024-03-28T22:09:30.426320+0000","last_undegraded":"2024-03-28T22:09:30.426320+0000","last_fullsized":"2024-03-28T22:09:30.426320+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:44.240551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058066+0000","last_change":"2024-03-28T22:08:29.323274+0000","last_active":"2024-03-28T22:09:30.058066+0000","last_peered":"2024-03-28T22:09:30.058066+0000","last_clean":"2024-03-28T22:09:30.058066+0000","last_became_active":"2024-03-28T22:08:29.322911+0000","last_became_peered":"2024-03-28T22:08:29.322911+0000","last_unstale":"2024-03-28T22:09:30.058066+0000","last_undegraded":"2024-03-28T22:09:30.058066+0000","last_fullsized":"2024-03-28T22:09:30.058066+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:44:14.439411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.056385+0000","last_change":"2024-03-28T22:08:29.961101+0000","last_active":"2024-03-28T22:09:30.056385+0000","last_peered":"2024-03-28T22:09:30.056385+0000","last_clean":"2024-03-28T22:09:30.056385+0000","last_became_active":"2024-03-28T22:08:29.960767+0000","last_became_peered":"2024-03-28T22:08:29.960767+0000","last_unstale":"2024-03-28T22:09:30.056385+0000","last_undegraded":"2024-03-28T22:09:30.056385+0000","last_fullsized":"2024-03-28T22:09:30.056385+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:33:45.236691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.060215+0000","last_change":"2024-03-28T22:09:30.060215+0000","last_active":"2024-03-28T22:09:30.060215+0000","last_peered":"2024-03-28T22:09:30.060215+0000","last_clean":"2024-03-28T22:09:30.060215+0000","last_became_active":"2024-03-28T22:09:30.059828+0000","last_became_peered":"2024-03-28T22:09:30.059828+0000","last_unstale":"2024-03-28T22:09:30.060215+0000","last_undegraded":"2024-03-28T22:09:30.060215+0000","last_fullsized":"2024-03-28T22:09:30.060215+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:42:05.087899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.425494+0000","last_change":"2024-03-28T22:08:29.954093+0000","last_active":"2024-03-28T22:09:30.425494+0000","last_peered":"2024-03-28T22:09:30.425494+0000","last_clean":"2024-03-28T22:09:30.425494+0000","last_became_active":"2024-03-28T22:08:29.953688+0000","last_became_peered":"2024-03-28T22:08:29.953688+0000","last_unstale":"2024-03-28T22:09:30.425494+0000","last_undegraded":"2024-03-28T22:09:30.425494+0000","last_fullsized":"2024-03-28T22:09:30.425494+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:57:51.026565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.055115+0000","last_change":"2024-03-28T22:08:29.331104+0000","last_active":"2024-03-28T22:09:30.055115+0000","last_peered":"2024-03-28T22:09:30.055115+0000","last_clean":"2024-03-28T22:09:30.055115+0000","last_became_active":"2024-03-28T22:08:29.330505+0000","last_became_peered":"2024-03-28T22:08:29.330505+0000","last_unstale":"2024-03-28T22:09:30.055115+0000","last_undegraded":"2024-03-28T22:09:30.055115+0000","last_fullsized":"2024-03-28T22:09:30.055115+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:36:59.944587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.425499+0000","last_change":"2024-03-28T22:08:29.963540+0000","last_active":"2024-03-28T22:09:30.425499+0000","last_peered":"2024-03-28T22:09:30.425499+0000","last_clean":"2024-03-28T22:09:30.425499+0000","last_became_active":"2024-03-28T22:08:29.963206+0000","last_became_peered":"2024-03-28T22:08:29.963206+0000","last_unstale":"2024-03-28T22:09:30.425499+0000","last_undegraded":"2024-03-28T22:09:30.425499+0000","last_fullsized":"2024-03-28T22:09:30.425499+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:55:53.292554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.059727+0000","last_change":"2024-03-28T22:08:29.962822+0000","last_active":"2024-03-28T22:09:30.059727+0000","last_peered":"2024-03-28T22:09:30.059727+0000","last_clean":"2024-03-28T22:09:30.059727+0000","last_became_active":"2024-03-28T22:08:29.962343+0000","last_became_peered":"2024-03-28T22:08:29.962343+0000","last_unstale":"2024-03-28T22:09:30.059727+0000","last_undegraded":"2024-03-28T22:09:30.059727+0000","last_fullsized":"2024-03-28T22:09:30.059727+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:36.608543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":85,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:34.434924+0000","last_change":"2024-03-28T22:08:29.328239+0000","last_active":"2024-03-28T22:09:34.434924+0000","last_peered":"2024-03-28T22:09:34.434924+0000","last_clean":"2024-03-28T22:09:34.434924+0000","last_became_active":"2024-03-28T22:08:29.327903+0000","last_became_peered":"2024-03-28T22:08:29.327903+0000","last_unstale":"2024-03-28T22:09:34.434924+0000","last_undegraded":"2024-03-28T22:09:34.434924+0000","last_fullsized":"2024-03-28T22:09:34.434924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:14:17.558472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":62,"num_read_kb":62,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.426683+0000","last_change":"2024-03-28T22:09:30.426683+0000","last_active":"2024-03-28T22:09:30.426683+0000","last_peered":"2024-03-28T22:09:30.426683+0000","last_clean":"2024-03-28T22:09:30.426683+0000","last_became_active":"2024-03-28T22:09:30.426255+0000","last_became_peered":"2024-03-28T22:09:30.426255+0000","last_unstale":"2024-03-28T22:09:30.426683+0000","last_undegraded":"2024-03-28T22:09:30.426683+0000","last_fullsized":"2024-03-28T22:09:30.426683+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:36.755978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.079359+0000","last_change":"2024-03-28T22:08:29.323050+0000","last_active":"2024-03-28T22:09:30.079359+0000","last_peered":"2024-03-28T22:09:30.079359+0000","last_clean":"2024-03-28T22:09:30.079359+0000","last_became_active":"2024-03-28T22:08:29.322684+0000","last_became_peered":"2024-03-28T22:08:29.322684+0000","last_unstale":"2024-03-28T22:09:30.079359+0000","last_undegraded":"2024-03-28T22:09:30.079359+0000","last_fullsized":"2024-03-28T22:09:30.079359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:14:43.811122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":353,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.054871+0000","last_change":"2024-03-28T22:07:17.785516+0000","last_active":"2024-03-28T22:09:30.054871+0000","last_peered":"2024-03-28T22:09:30.054871+0000","last_clean":"2024-03-28T22:09:30.054871+0000","last_became_active":"2024-03-28T22:07:17.785123+0000","last_became_peered":"2024-03-28T22:07:17.785123+0000","last_unstale":"2024-03-28T22:09:30.054871+0000","last_undegraded":"2024-03-28T22:09:30.054871+0000","last_fullsized":"2024-03-28T22:09:30.054871+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:03:59.840513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:03:59.840513+0000","last_clean_scrub_stamp":"2024-03-28T22:03:59.840513+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:31:06.253632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.293289+0000","last_change":"2024-03-28T22:08:29.961489+0000","last_active":"2024-03-28T22:09:30.293289+0000","last_peered":"2024-03-28T22:09:30.293289+0000","last_clean":"2024-03-28T22:09:30.293289+0000","last_became_active":"2024-03-28T22:08:29.960881+0000","last_became_peered":"2024-03-28T22:08:29.960881+0000","last_unstale":"2024-03-28T22:09:30.293289+0000","last_undegraded":"2024-03-28T22:09:30.293289+0000","last_fullsized":"2024-03-28T22:09:30.293289+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:48.870834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.054972+0000","last_change":"2024-03-28T22:08:29.331994+0000","last_active":"2024-03-28T22:09:30.054972+0000","last_peered":"2024-03-28T22:09:30.054972+0000","last_clean":"2024-03-28T22:09:30.054972+0000","last_became_active":"2024-03-28T22:08:29.331630+0000","last_became_peered":"2024-03-28T22:08:29.331630+0000","last_unstale":"2024-03-28T22:09:30.054972+0000","last_undegraded":"2024-03-28T22:09:30.054972+0000","last_fullsized":"2024-03-28T22:09:30.054972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:11:37.100405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058466+0000","last_change":"2024-03-28T22:08:29.330353+0000","last_active":"2024-03-28T22:09:30.058466+0000","last_peered":"2024-03-28T22:09:30.058466+0000","last_clean":"2024-03-28T22:09:30.058466+0000","last_became_active":"2024-03-28T22:08:29.329421+0000","last_became_peered":"2024-03-28T22:08:29.329421+0000","last_unstale":"2024-03-28T22:09:30.058466+0000","last_undegraded":"2024-03-28T22:09:30.058466+0000","last_fullsized":"2024-03-28T22:09:30.058466+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:56.166506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.293863+0000","last_change":"2024-03-28T22:08:29.948595+0000","last_active":"2024-03-28T22:09:30.293863+0000","last_peered":"2024-03-28T22:09:30.293863+0000","last_clean":"2024-03-28T22:09:30.293863+0000","last_became_active":"2024-03-28T22:08:29.946986+0000","last_became_peered":"2024-03-28T22:08:29.946986+0000","last_unstale":"2024-03-28T22:09:30.293863+0000","last_undegraded":"2024-03-28T22:09:30.293863+0000","last_fullsized":"2024-03-28T22:09:30.293863+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:56:12.351731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.054613+0000","last_change":"2024-03-28T22:08:29.333375+0000","last_active":"2024-03-28T22:09:30.054613+0000","last_peered":"2024-03-28T22:09:30.054613+0000","last_clean":"2024-03-28T22:09:30.054613+0000","last_became_active":"2024-03-28T22:08:29.331594+0000","last_became_peered":"2024-03-28T22:08:29.331594+0000","last_unstale":"2024-03-28T22:09:30.054613+0000","last_undegraded":"2024-03-28T22:09:30.054613+0000","last_fullsized":"2024-03-28T22:09:30.054613+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:07:43.936807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.057907+0000","last_change":"2024-03-28T22:08:29.959061+0000","last_active":"2024-03-28T22:09:30.057907+0000","last_peered":"2024-03-28T22:09:30.057907+0000","last_clean":"2024-03-28T22:09:30.057907+0000","last_became_active":"2024-03-28T22:08:29.958723+0000","last_became_peered":"2024-03-28T22:08:29.958723+0000","last_unstale":"2024-03-28T22:09:30.057907+0000","last_undegraded":"2024-03-28T22:09:30.057907+0000","last_fullsized":"2024-03-28T22:09:30.057907+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:32:23.016606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.079461+0000","last_change":"2024-03-28T22:08:29.329886+0000","last_active":"2024-03-28T22:09:30.079461+0000","last_peered":"2024-03-28T22:09:30.079461+0000","last_clean":"2024-03-28T22:09:30.079461+0000","last_became_active":"2024-03-28T22:08:29.329503+0000","last_became_peered":"2024-03-28T22:08:29.329503+0000","last_unstale":"2024-03-28T22:09:30.079461+0000","last_undegraded":"2024-03-28T22:09:30.079461+0000","last_fullsized":"2024-03-28T22:09:30.079461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:50:05.686647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.059336+0000","last_change":"2024-03-28T22:08:29.957038+0000","last_active":"2024-03-28T22:09:30.059336+0000","last_peered":"2024-03-28T22:09:30.059336+0000","last_clean":"2024-03-28T22:09:30.059336+0000","last_became_active":"2024-03-28T22:08:29.956693+0000","last_became_peered":"2024-03-28T22:08:29.956693+0000","last_unstale":"2024-03-28T22:09:30.059336+0000","last_undegraded":"2024-03-28T22:09:30.059336+0000","last_fullsized":"2024-03-28T22:09:30.059336+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:19:09.881431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.424647+0000","last_change":"2024-03-28T22:08:29.327356+0000","last_active":"2024-03-28T22:09:30.424647+0000","last_peered":"2024-03-28T22:09:30.424647+0000","last_clean":"2024-03-28T22:09:30.424647+0000","last_became_active":"2024-03-28T22:08:29.326947+0000","last_became_peered":"2024-03-28T22:08:29.326947+0000","last_unstale":"2024-03-28T22:09:30.424647+0000","last_undegraded":"2024-03-28T22:09:30.424647+0000","last_fullsized":"2024-03-28T22:09:30.424647+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:47:29.417465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.057637+0000","last_change":"2024-03-28T22:08:29.966505+0000","last_active":"2024-03-28T22:09:30.057637+0000","last_peered":"2024-03-28T22:09:30.057637+0000","last_clean":"2024-03-28T22:09:30.057637+0000","last_became_active":"2024-03-28T22:08:29.966057+0000","last_became_peered":"2024-03-28T22:08:29.966057+0000","last_unstale":"2024-03-28T22:09:30.057637+0000","last_undegraded":"2024-03-28T22:09:30.057637+0000","last_fullsized":"2024-03-28T22:09:30.057637+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:17:45.196585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.057477+0000","last_change":"2024-03-28T22:08:29.328897+0000","last_active":"2024-03-28T22:09:30.057477+0000","last_peered":"2024-03-28T22:09:30.057477+0000","last_clean":"2024-03-28T22:09:30.057477+0000","last_became_active":"2024-03-28T22:08:29.328390+0000","last_became_peered":"2024-03-28T22:08:29.328390+0000","last_unstale":"2024-03-28T22:09:30.057477+0000","last_undegraded":"2024-03-28T22:09:30.057477+0000","last_fullsized":"2024-03-28T22:09:30.057477+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:21:12.699503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.055247+0000","last_change":"2024-03-28T22:08:29.957212+0000","last_active":"2024-03-28T22:09:30.055247+0000","last_peered":"2024-03-28T22:09:30.055247+0000","last_clean":"2024-03-28T22:09:30.055247+0000","last_became_active":"2024-03-28T22:08:29.956215+0000","last_became_peered":"2024-03-28T22:08:29.956215+0000","last_unstale":"2024-03-28T22:09:30.055247+0000","last_undegraded":"2024-03-28T22:09:30.055247+0000","last_fullsized":"2024-03-28T22:09:30.055247+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:57:25.199993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.063151+0000","last_change":"2024-03-28T22:09:30.062742+0000","last_active":"2024-03-28T22:09:30.063151+0000","last_peered":"2024-03-28T22:09:30.063151+0000","last_clean":"2024-03-28T22:09:30.063151+0000","last_became_active":"2024-03-28T22:09:30.059356+0000","last_became_peered":"2024-03-28T22:09:30.059356+0000","last_unstale":"2024-03-28T22:09:30.063151+0000","last_undegraded":"2024-03-28T22:09:30.063151+0000","last_fullsized":"2024-03-28T22:09:30.063151+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:52:39.412153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058322+0000","last_change":"2024-03-28T22:08:29.955023+0000","last_active":"2024-03-28T22:09:30.058322+0000","last_peered":"2024-03-28T22:09:30.058322+0000","last_clean":"2024-03-28T22:09:30.058322+0000","last_became_active":"2024-03-28T22:08:29.954675+0000","last_became_peered":"2024-03-28T22:08:29.954675+0000","last_unstale":"2024-03-28T22:09:30.058322+0000","last_undegraded":"2024-03-28T22:09:30.058322+0000","last_fullsized":"2024-03-28T22:09:30.058322+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:31.246452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058628+0000","last_change":"2024-03-28T22:08:29.329280+0000","last_active":"2024-03-28T22:09:30.058628+0000","last_peered":"2024-03-28T22:09:30.058628+0000","last_clean":"2024-03-28T22:09:30.058628+0000","last_became_active":"2024-03-28T22:08:29.328860+0000","last_became_peered":"2024-03-28T22:08:29.328860+0000","last_unstale":"2024-03-28T22:09:30.058628+0000","last_undegraded":"2024-03-28T22:09:30.058628+0000","last_fullsized":"2024-03-28T22:09:30.058628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:15:49.610026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.055286+0000","last_change":"2024-03-28T22:08:29.966224+0000","last_active":"2024-03-28T22:09:30.055286+0000","last_peered":"2024-03-28T22:09:30.055286+0000","last_clean":"2024-03-28T22:09:30.055286+0000","last_became_active":"2024-03-28T22:08:29.965636+0000","last_became_peered":"2024-03-28T22:08:29.965636+0000","last_unstale":"2024-03-28T22:09:30.055286+0000","last_undegraded":"2024-03-28T22:09:30.055286+0000","last_fullsized":"2024-03-28T22:09:30.055286+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:23:31.230980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058666+0000","last_change":"2024-03-28T22:08:29.335228+0000","last_active":"2024-03-28T22:09:30.058666+0000","last_peered":"2024-03-28T22:09:30.058666+0000","last_clean":"2024-03-28T22:09:30.058666+0000","last_became_active":"2024-03-28T22:08:29.334686+0000","last_became_peered":"2024-03-28T22:08:29.334686+0000","last_unstale":"2024-03-28T22:09:30.058666+0000","last_undegraded":"2024-03-28T22:09:30.058666+0000","last_fullsized":"2024-03-28T22:09:30.058666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:17:40.571206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.055703+0000","last_change":"2024-03-28T22:08:29.963912+0000","last_active":"2024-03-28T22:09:30.055703+0000","last_peered":"2024-03-28T22:09:30.055703+0000","last_clean":"2024-03-28T22:09:30.055703+0000","last_became_active":"2024-03-28T22:08:29.962740+0000","last_became_peered":"2024-03-28T22:08:29.962740+0000","last_unstale":"2024-03-28T22:09:30.055703+0000","last_undegraded":"2024-03-28T22:09:30.055703+0000","last_fullsized":"2024-03-28T22:09:30.055703+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:50:25.756923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.060333+0000","last_change":"2024-03-28T22:08:29.325767+0000","last_active":"2024-03-28T22:09:30.060333+0000","last_peered":"2024-03-28T22:09:30.060333+0000","last_clean":"2024-03-28T22:09:30.060333+0000","last_became_active":"2024-03-28T22:08:29.325430+0000","last_became_peered":"2024-03-28T22:08:29.325430+0000","last_unstale":"2024-03-28T22:09:30.060333+0000","last_undegraded":"2024-03-28T22:09:30.060333+0000","last_fullsized":"2024-03-28T22:09:30.060333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:01:13.663163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.055431+0000","last_change":"2024-03-28T22:08:29.960158+0000","last_active":"2024-03-28T22:09:30.055431+0000","last_peered":"2024-03-28T22:09:30.055431+0000","last_clean":"2024-03-28T22:09:30.055431+0000","last_became_active":"2024-03-28T22:08:29.959792+0000","last_became_peered":"2024-03-28T22:08:29.959792+0000","last_unstale":"2024-03-28T22:09:30.055431+0000","last_undegraded":"2024-03-28T22:09:30.055431+0000","last_fullsized":"2024-03-28T22:09:30.055431+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:28:49.135551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.293763+0000","last_change":"2024-03-28T22:08:29.328547+0000","last_active":"2024-03-28T22:09:30.293763+0000","last_peered":"2024-03-28T22:09:30.293763+0000","last_clean":"2024-03-28T22:09:30.293763+0000","last_became_active":"2024-03-28T22:08:29.327910+0000","last_became_peered":"2024-03-28T22:08:29.327910+0000","last_unstale":"2024-03-28T22:09:30.293763+0000","last_undegraded":"2024-03-28T22:09:30.293763+0000","last_fullsized":"2024-03-28T22:09:30.293763+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:06:13.980071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.055841+0000","last_change":"2024-03-28T22:08:29.963904+0000","last_active":"2024-03-28T22:09:30.055841+0000","last_peered":"2024-03-28T22:09:30.055841+0000","last_clean":"2024-03-28T22:09:30.055841+0000","last_became_active":"2024-03-28T22:08:29.963536+0000","last_became_peered":"2024-03-28T22:08:29.963536+0000","last_unstale":"2024-03-28T22:09:30.055841+0000","last_undegraded":"2024-03-28T22:09:30.055841+0000","last_fullsized":"2024-03-28T22:09:30.055841+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:59:43.376792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.055319+0000","last_change":"2024-03-28T22:08:29.962357+0000","last_active":"2024-03-28T22:09:30.055319+0000","last_peered":"2024-03-28T22:09:30.055319+0000","last_clean":"2024-03-28T22:09:30.055319+0000","last_became_active":"2024-03-28T22:08:29.961998+0000","last_became_peered":"2024-03-28T22:08:29.961998+0000","last_unstale":"2024-03-28T22:09:30.055319+0000","last_undegraded":"2024-03-28T22:09:30.055319+0000","last_fullsized":"2024-03-28T22:09:30.055319+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:40:47.465889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.057636+0000","last_change":"2024-03-28T22:08:29.961326+0000","last_active":"2024-03-28T22:09:30.057636+0000","last_peered":"2024-03-28T22:09:30.057636+0000","last_clean":"2024-03-28T22:09:30.057636+0000","last_became_active":"2024-03-28T22:08:29.960316+0000","last_became_peered":"2024-03-28T22:08:29.960316+0000","last_unstale":"2024-03-28T22:09:30.057636+0000","last_undegraded":"2024-03-28T22:09:30.057636+0000","last_fullsized":"2024-03-28T22:09:30.057636+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:37:55.773686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.057451+0000","last_change":"2024-03-28T22:08:29.956272+0000","last_active":"2024-03-28T22:09:30.057451+0000","last_peered":"2024-03-28T22:09:30.057451+0000","last_clean":"2024-03-28T22:09:30.057451+0000","last_became_active":"2024-03-28T22:08:29.955875+0000","last_became_peered":"2024-03-28T22:08:29.955875+0000","last_unstale":"2024-03-28T22:09:30.057451+0000","last_undegraded":"2024-03-28T22:09:30.057451+0000","last_fullsized":"2024-03-28T22:09:30.057451+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:17:33.062615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.294754+0000","last_change":"2024-03-28T22:09:30.294754+0000","last_active":"2024-03-28T22:09:30.294754+0000","last_peered":"2024-03-28T22:09:30.294754+0000","last_clean":"2024-03-28T22:09:30.294754+0000","last_became_active":"2024-03-28T22:09:30.294341+0000","last_became_peered":"2024-03-28T22:09:30.294341+0000","last_unstale":"2024-03-28T22:09:30.294754+0000","last_undegraded":"2024-03-28T22:09:30.294754+0000","last_fullsized":"2024-03-28T22:09:30.294754+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:25:46.017373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.293843+0000","last_change":"2024-03-28T22:08:29.960464+0000","last_active":"2024-03-28T22:09:30.293843+0000","last_peered":"2024-03-28T22:09:30.293843+0000","last_clean":"2024-03-28T22:09:30.293843+0000","last_became_active":"2024-03-28T22:08:29.960111+0000","last_became_peered":"2024-03-28T22:08:29.960111+0000","last_unstale":"2024-03-28T22:09:30.293843+0000","last_undegraded":"2024-03-28T22:09:30.293843+0000","last_fullsized":"2024-03-28T22:09:30.293843+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:32:33.533199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.056104+0000","last_change":"2024-03-28T22:08:29.960045+0000","last_active":"2024-03-28T22:09:30.056104+0000","last_peered":"2024-03-28T22:09:30.056104+0000","last_clean":"2024-03-28T22:09:30.056104+0000","last_became_active":"2024-03-28T22:08:29.959645+0000","last_became_peered":"2024-03-28T22:08:29.959645+0000","last_unstale":"2024-03-28T22:09:30.056104+0000","last_undegraded":"2024-03-28T22:09:30.056104+0000","last_fullsized":"2024-03-28T22:09:30.056104+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:34:56.202257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058450+0000","last_change":"2024-03-28T22:08:29.958736+0000","last_active":"2024-03-28T22:09:30.058450+0000","last_peered":"2024-03-28T22:09:30.058450+0000","last_clean":"2024-03-28T22:09:30.058450+0000","last_became_active":"2024-03-28T22:08:29.958387+0000","last_became_peered":"2024-03-28T22:08:29.958387+0000","last_unstale":"2024-03-28T22:09:30.058450+0000","last_undegraded":"2024-03-28T22:09:30.058450+0000","last_fullsized":"2024-03-28T22:09:30.058450+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:07:54.829166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.057847+0000","last_change":"2024-03-28T22:08:29.961373+0000","last_active":"2024-03-28T22:09:30.057847+0000","last_peered":"2024-03-28T22:09:30.057847+0000","last_clean":"2024-03-28T22:09:30.057847+0000","last_became_active":"2024-03-28T22:08:29.959751+0000","last_became_peered":"2024-03-28T22:08:29.959751+0000","last_unstale":"2024-03-28T22:09:30.057847+0000","last_undegraded":"2024-03-28T22:09:30.057847+0000","last_fullsized":"2024-03-28T22:09:30.057847+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:34:18.680552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.059658+0000","last_change":"2024-03-28T22:08:29.962721+0000","last_active":"2024-03-28T22:09:30.059658+0000","last_peered":"2024-03-28T22:09:30.059658+0000","last_clean":"2024-03-28T22:09:30.059658+0000","last_became_active":"2024-03-28T22:08:29.961349+0000","last_became_peered":"2024-03-28T22:08:29.961349+0000","last_unstale":"2024-03-28T22:09:30.059658+0000","last_undegraded":"2024-03-28T22:09:30.059658+0000","last_fullsized":"2024-03-28T22:09:30.059658+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:35:56.535055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.057325+0000","last_change":"2024-03-28T22:08:29.960068+0000","last_active":"2024-03-28T22:09:30.057325+0000","last_peered":"2024-03-28T22:09:30.057325+0000","last_clean":"2024-03-28T22:09:30.057325+0000","last_became_active":"2024-03-28T22:08:29.959716+0000","last_became_peered":"2024-03-28T22:08:29.959716+0000","last_unstale":"2024-03-28T22:09:30.057325+0000","last_undegraded":"2024-03-28T22:09:30.057325+0000","last_fullsized":"2024-03-28T22:09:30.057325+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:18:38.055322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.424859+0000","last_change":"2024-03-28T22:08:29.957920+0000","last_active":"2024-03-28T22:09:30.424859+0000","last_peered":"2024-03-28T22:09:30.424859+0000","last_clean":"2024-03-28T22:09:30.424859+0000","last_became_active":"2024-03-28T22:08:29.957541+0000","last_became_peered":"2024-03-28T22:08:29.957541+0000","last_unstale":"2024-03-28T22:09:30.424859+0000","last_undegraded":"2024-03-28T22:09:30.424859+0000","last_fullsized":"2024-03-28T22:09:30.424859+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:45:05.658745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.293859+0000","last_change":"2024-03-28T22:08:29.953589+0000","last_active":"2024-03-28T22:09:30.293859+0000","last_peered":"2024-03-28T22:09:30.293859+0000","last_clean":"2024-03-28T22:09:30.293859+0000","last_became_active":"2024-03-28T22:08:29.953253+0000","last_became_peered":"2024-03-28T22:08:29.953253+0000","last_unstale":"2024-03-28T22:09:30.293859+0000","last_undegraded":"2024-03-28T22:09:30.293859+0000","last_fullsized":"2024-03-28T22:09:30.293859+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:35.720708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.055833+0000","last_change":"2024-03-28T22:08:29.966235+0000","last_active":"2024-03-28T22:09:30.055833+0000","last_peered":"2024-03-28T22:09:30.055833+0000","last_clean":"2024-03-28T22:09:30.055833+0000","last_became_active":"2024-03-28T22:08:29.965830+0000","last_became_peered":"2024-03-28T22:08:29.965830+0000","last_unstale":"2024-03-28T22:09:30.055833+0000","last_undegraded":"2024-03-28T22:09:30.055833+0000","last_fullsized":"2024-03-28T22:09:30.055833+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:04:27.094743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63304,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":417,"num_read_kb":274,"num_write":155,"num_write_kb":106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":189912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134,"ondisk_log_size":134,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351004,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32108,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713044,"statfs":{"total":95995035648,"available":95962157056,"internally_reserved":0,"allocated":950272,"data_stored":459161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612640,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28628,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716524,"statfs":{"total":95995035648,"available":95965720576,"internally_reserved":0,"allocated":1646592,"data_stored":1109945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776165,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":966656,"data_stored":459397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037802,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":966656,"data_stored":459317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299440,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36864,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708288,"statfs":{"total":95995035648,"available":95957286912,"internally_reserved":0,"allocated":1560576,"data_stored":1051638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462963,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36272,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708880,"statfs":{"total":95995035648,"available":95957893120,"internally_reserved":0,"allocated":954368,"data_stored":458786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574924,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":517491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738449,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":460057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36360,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708792,"statfs":{"total":95995035648,"available":95957803008,"internally_reserved":0,"allocated":1040384,"data_stored":519232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149762,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":970752,"data_stored":459097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888126,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40440,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704712,"statfs":{"total":95995035648,"available":95953625088,"internally_reserved":0,"allocated":958464,"data_stored":459614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724602,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28560,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716592,"statfs":{"total":95995035648,"available":95965790208,"internally_reserved":0,"allocated":1572864,"data_stored":1050931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":58912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":60653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":60881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T22:09:41.995 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-28T22:09:42.210 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:42 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:09:42.210 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:42 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:09:42.210 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:42 smithi016 ceph-mon[32374]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:09:42.211 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:42 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:09:42.244 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:09:42.471 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:42 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:09:42.472 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:42 smithi016 ceph-mon[32374]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:09:42.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:42 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:09:42.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:42 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:09:42.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:42 smithi053 ceph-mon[35979]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:09:42.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:42 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:09:42.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:42 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:09:42.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:42 smithi053 ceph-mon[35979]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:09:42.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:42 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:09:42.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:42 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:09:42.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:42 smithi063 ceph-mon[36249]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:09:42.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:42 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:09:42.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:42 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:09:42.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:42 smithi063 ceph-mon[36249]: from='mgr.14152 172.21.15.16:0/956354196' entity='mgr.x' 2024-03-28T22:09:43.325 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:43 smithi016 ceph-mon[32374]: pgmap v325: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T22:09:43.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:43 smithi053 ceph-mon[35979]: pgmap v325: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T22:09:43.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:43 smithi063 ceph-mon[36249]: pgmap v325: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T22:09:43.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.853+0000 7fa0917f7700 1 -- 172.21.15.16:0/1729068111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa08c102c50 msgr2=0x7fa08c10f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:43.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.853+0000 7fa0917f7700 1 --2- 172.21.15.16:0/1729068111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa08c102c50 0x7fa08c10f090 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa0840099c0 tx=0x7fa084009cd0 comp rx=0 tx=0).stop 2024-03-28T22:09:43.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.853+0000 7fa0917f7700 1 -- 172.21.15.16:0/1729068111 shutdown_connections 2024-03-28T22:09:43.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.853+0000 7fa0917f7700 1 --2- 172.21.15.16:0/1729068111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa08c102c50 0x7fa08c10f090 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:43.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.853+0000 7fa0917f7700 1 --2- 172.21.15.16:0/1729068111 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa08c108270 0x7fa08c108650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:43.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.853+0000 7fa0917f7700 1 --2- 172.21.15.16:0/1729068111 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa08c102290 0x7fa08c102710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:43.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.853+0000 7fa0917f7700 1 -- 172.21.15.16:0/1729068111 >> 172.21.15.16:0/1729068111 conn(0x7fa08c079370 msgr2=0x7fa08c079780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:43.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.853+0000 7fa0917f7700 1 -- 172.21.15.16:0/1729068111 shutdown_connections 2024-03-28T22:09:43.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.853+0000 7fa0917f7700 1 -- 172.21.15.16:0/1729068111 wait complete. 2024-03-28T22:09:43.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.854+0000 7fa0917f7700 1 Processor -- start 2024-03-28T22:09:43.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.854+0000 7fa0917f7700 1 -- start start 2024-03-28T22:09:43.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.854+0000 7fa0917f7700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa08c102290 0x7fa08c074540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:43.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.854+0000 7fa0917f7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa08c102c50 0x7fa08c074a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:43.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.854+0000 7fa0917f7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa08c108270 0x7fa08c074fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:43.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.854+0000 7fa0917f7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa08c1124f0 con 0x7fa08c108270 2024-03-28T22:09:43.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.854+0000 7fa0917f7700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fa08c112370 con 0x7fa08c102c50 2024-03-28T22:09:43.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.854+0000 7fa0917f7700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa08c112670 con 0x7fa08c102290 2024-03-28T22:09:43.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.854+0000 7fa08b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa08c108270 0x7fa08c074fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:43.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.854+0000 7fa08a7fc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa08c102c50 0x7fa08c074a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:43.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.854+0000 7fa08b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa08c108270 0x7fa08c074fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56738/0 (socket says 172.21.15.16:56738) 2024-03-28T22:09:43.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.854+0000 7fa08b7fe700 1 -- 172.21.15.16:0/3383748208 learned_addr learned my addr 172.21.15.16:0/3383748208 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:43.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.854+0000 7fa08affd700 1 --2- 172.21.15.16:0/3383748208 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa08c102290 0x7fa08c074540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:43.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.855+0000 7fa08b7fe700 1 -- 172.21.15.16:0/3383748208 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa08c102290 msgr2=0x7fa08c074540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:43.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.855+0000 7fa08b7fe700 1 --2- 172.21.15.16:0/3383748208 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa08c102290 0x7fa08c074540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:43.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.855+0000 7fa08b7fe700 1 -- 172.21.15.16:0/3383748208 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa08c102c50 msgr2=0x7fa08c074a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:43.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.855+0000 7fa08b7fe700 1 --2- 172.21.15.16:0/3383748208 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa08c102c50 0x7fa08c074a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:43.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.855+0000 7fa08b7fe700 1 -- 172.21.15.16:0/3383748208 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa08c1a6dd0 con 0x7fa08c108270 2024-03-28T22:09:43.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.855+0000 7fa08a7fc700 1 --2- 172.21.15.16:0/3383748208 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa08c102c50 0x7fa08c074a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:09:43.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.855+0000 7fa08b7fe700 1 --2- 172.21.15.16:0/3383748208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa08c108270 0x7fa08c074fc0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa084006010 tx=0x7fa08400fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:43.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.855+0000 7fa083fff700 1 -- 172.21.15.16:0/3383748208 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa084027070 con 0x7fa08c108270 2024-03-28T22:09:43.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.855+0000 7fa083fff700 1 -- 172.21.15.16:0/3383748208 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa084003680 con 0x7fa08c108270 2024-03-28T22:09:43.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.855+0000 7fa0917f7700 1 -- 172.21.15.16:0/3383748208 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa08c1a7060 con 0x7fa08c108270 2024-03-28T22:09:43.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.855+0000 7fa083fff700 1 -- 172.21.15.16:0/3383748208 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0840173f0 con 0x7fa08c108270 2024-03-28T22:09:43.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.855+0000 7fa0917f7700 1 -- 172.21.15.16:0/3383748208 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa08c1a74b0 con 0x7fa08c108270 2024-03-28T22:09:43.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.856+0000 7fa0917f7700 1 -- 172.21.15.16:0/3383748208 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa08c068f00 con 0x7fa08c108270 2024-03-28T22:09:43.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.857+0000 7fa083fff700 1 -- 172.21.15.16:0/3383748208 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fa08402a070 con 0x7fa08c108270 2024-03-28T22:09:43.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.858+0000 7fa083fff700 1 --2- 172.21.15.16:0/3383748208 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fa0780a1ea0 0x7fa0780a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:43.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.858+0000 7fa08affd700 1 --2- 172.21.15.16:0/3383748208 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fa0780a1ea0 0x7fa0780a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:43.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.858+0000 7fa083fff700 1 -- 172.21.15.16:0/3383748208 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fa0840ceb40 con 0x7fa08c108270 2024-03-28T22:09:43.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.858+0000 7fa08affd700 1 --2- 172.21.15.16:0/3383748208 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fa0780a1ea0 0x7fa0780a4360 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa07c00f540 tx=0x7fa07c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:43.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:43.863+0000 7fa083fff700 1 -- 172.21.15.16:0/3383748208 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa084013070 con 0x7fa08c108270 2024-03-28T22:09:44.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:44.061+0000 7fa0917f7700 1 -- 172.21.15.16:0/3383748208 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa08c101410 con 0x7fa0780a1ea0 2024-03-28T22:09:44.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:44.068+0000 7fa083fff700 1 -- 172.21.15.16:0/3383748208 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258702 (secure 0 0 0) 0x7fa08c101410 con 0x7fa0780a1ea0 2024-03-28T22:09:44.075 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:09:44.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:44.071+0000 7fa0917f7700 1 -- 172.21.15.16:0/3383748208 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fa0780a1ea0 msgr2=0x7fa0780a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:44.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:44.071+0000 7fa0917f7700 1 --2- 172.21.15.16:0/3383748208 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fa0780a1ea0 0x7fa0780a4360 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa07c00f540 tx=0x7fa07c00b040 comp rx=0 tx=0).stop 2024-03-28T22:09:44.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:44.071+0000 7fa0917f7700 1 -- 172.21.15.16:0/3383748208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa08c108270 msgr2=0x7fa08c074fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:44.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:44.071+0000 7fa0917f7700 1 --2- 172.21.15.16:0/3383748208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa08c108270 0x7fa08c074fc0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa084006010 tx=0x7fa08400fa00 comp rx=0 tx=0).stop 2024-03-28T22:09:44.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:44.071+0000 7fa0917f7700 1 -- 172.21.15.16:0/3383748208 shutdown_connections 2024-03-28T22:09:44.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:44.071+0000 7fa0917f7700 1 --2- 172.21.15.16:0/3383748208 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fa0780a1ea0 0x7fa0780a4360 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:44.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:44.071+0000 7fa0917f7700 1 --2- 172.21.15.16:0/3383748208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa08c108270 0x7fa08c074fc0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:44.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:44.071+0000 7fa0917f7700 1 --2- 172.21.15.16:0/3383748208 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa08c102290 0x7fa08c074540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:44.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:44.071+0000 7fa0917f7700 1 --2- 172.21.15.16:0/3383748208 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa08c102c50 0x7fa08c074a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:44.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:44.071+0000 7fa0917f7700 1 -- 172.21.15.16:0/3383748208 >> 172.21.15.16:0/3383748208 conn(0x7fa08c079370 msgr2=0x7fa08c100d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:44.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:44.071+0000 7fa0917f7700 1 -- 172.21.15.16:0/3383748208 shutdown_connections 2024-03-28T22:09:44.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:44.071+0000 7fa0917f7700 1 -- 172.21.15.16:0/3383748208 wait complete. 2024-03-28T22:09:44.077 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-03-28T22:09:44.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:44 smithi016 ceph-mon[32374]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:09:44.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:44 smithi053 ceph-mon[35979]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:09:44.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:44 smithi063 ceph-mon[36249]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:09:44.836 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-03-28T22:09:43.091073+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653906,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":563,"num_read_kb":398,"num_write":338,"num_write_kb":3046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":273,"ondisk_log_size":273,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":379132,"kb_used_data":13260,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124562692,"statfs":{"total":1151940427776,"available":1151552196608,"internally_reserved":0,"allocated":13578240,"data_stored":7464666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":374453562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":35,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003258"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.056315+0000","last_change":"2024-03-28T22:08:29.962664+0000","last_active":"2024-03-28T22:09:30.056315+0000","last_peered":"2024-03-28T22:09:30.056315+0000","last_clean":"2024-03-28T22:09:30.056315+0000","last_became_active":"2024-03-28T22:08:29.962029+0000","last_became_peered":"2024-03-28T22:08:29.962029+0000","last_unstale":"2024-03-28T22:09:30.056315+0000","last_undegraded":"2024-03-28T22:09:30.056315+0000","last_fullsized":"2024-03-28T22:09:30.056315+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:20:42.832304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.059447+0000","last_change":"2024-03-28T22:08:29.965617+0000","last_active":"2024-03-28T22:09:30.059447+0000","last_peered":"2024-03-28T22:09:30.059447+0000","last_clean":"2024-03-28T22:09:30.059447+0000","last_became_active":"2024-03-28T22:08:29.965248+0000","last_became_peered":"2024-03-28T22:08:29.965248+0000","last_unstale":"2024-03-28T22:09:30.059447+0000","last_undegraded":"2024-03-28T22:09:30.059447+0000","last_fullsized":"2024-03-28T22:09:30.059447+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:37:46.703768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.059158+0000","last_change":"2024-03-28T22:08:29.958452+0000","last_active":"2024-03-28T22:09:30.059158+0000","last_peered":"2024-03-28T22:09:30.059158+0000","last_clean":"2024-03-28T22:09:30.059158+0000","last_became_active":"2024-03-28T22:08:29.958089+0000","last_became_peered":"2024-03-28T22:08:29.958089+0000","last_unstale":"2024-03-28T22:09:30.059158+0000","last_undegraded":"2024-03-28T22:09:30.059158+0000","last_fullsized":"2024-03-28T22:09:30.059158+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:10:44.084925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.079508+0000","last_change":"2024-03-28T22:08:29.964812+0000","last_active":"2024-03-28T22:09:30.079508+0000","last_peered":"2024-03-28T22:09:30.079508+0000","last_clean":"2024-03-28T22:09:30.079508+0000","last_became_active":"2024-03-28T22:08:29.963872+0000","last_became_peered":"2024-03-28T22:08:29.963872+0000","last_unstale":"2024-03-28T22:09:30.079508+0000","last_undegraded":"2024-03-28T22:09:30.079508+0000","last_fullsized":"2024-03-28T22:09:30.079508+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:08.491078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.057748+0000","last_change":"2024-03-28T22:08:29.965564+0000","last_active":"2024-03-28T22:09:30.057748+0000","last_peered":"2024-03-28T22:09:30.057748+0000","last_clean":"2024-03-28T22:09:30.057748+0000","last_became_active":"2024-03-28T22:08:29.962636+0000","last_became_peered":"2024-03-28T22:08:29.962636+0000","last_unstale":"2024-03-28T22:09:30.057748+0000","last_undegraded":"2024-03-28T22:09:30.057748+0000","last_fullsized":"2024-03-28T22:09:30.057748+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:03:50.903231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.425116+0000","last_change":"2024-03-28T22:08:29.962091+0000","last_active":"2024-03-28T22:09:30.425116+0000","last_peered":"2024-03-28T22:09:30.425116+0000","last_clean":"2024-03-28T22:09:30.425116+0000","last_became_active":"2024-03-28T22:08:29.961735+0000","last_became_peered":"2024-03-28T22:08:29.961735+0000","last_unstale":"2024-03-28T22:09:30.425116+0000","last_undegraded":"2024-03-28T22:09:30.425116+0000","last_fullsized":"2024-03-28T22:09:30.425116+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:40:09.853265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.054864+0000","last_change":"2024-03-28T22:08:29.961089+0000","last_active":"2024-03-28T22:09:30.054864+0000","last_peered":"2024-03-28T22:09:30.054864+0000","last_clean":"2024-03-28T22:09:30.054864+0000","last_became_active":"2024-03-28T22:08:29.960711+0000","last_became_peered":"2024-03-28T22:08:29.960711+0000","last_unstale":"2024-03-28T22:09:30.054864+0000","last_undegraded":"2024-03-28T22:09:30.054864+0000","last_fullsized":"2024-03-28T22:09:30.054864+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:05:14.617718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.056921+0000","last_change":"2024-03-28T22:08:29.965142+0000","last_active":"2024-03-28T22:09:30.056921+0000","last_peered":"2024-03-28T22:09:30.056921+0000","last_clean":"2024-03-28T22:09:30.056921+0000","last_became_active":"2024-03-28T22:08:29.964730+0000","last_became_peered":"2024-03-28T22:08:29.964730+0000","last_unstale":"2024-03-28T22:09:30.056921+0000","last_undegraded":"2024-03-28T22:09:30.056921+0000","last_fullsized":"2024-03-28T22:09:30.056921+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:23:43.229681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.425637+0000","last_change":"2024-03-28T22:08:29.958984+0000","last_active":"2024-03-28T22:09:30.425637+0000","last_peered":"2024-03-28T22:09:30.425637+0000","last_clean":"2024-03-28T22:09:30.425637+0000","last_became_active":"2024-03-28T22:08:29.958583+0000","last_became_peered":"2024-03-28T22:08:29.958583+0000","last_unstale":"2024-03-28T22:09:30.425637+0000","last_undegraded":"2024-03-28T22:09:30.425637+0000","last_fullsized":"2024-03-28T22:09:30.425637+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:05:11.361719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.425101+0000","last_change":"2024-03-28T22:08:29.962774+0000","last_active":"2024-03-28T22:09:30.425101+0000","last_peered":"2024-03-28T22:09:30.425101+0000","last_clean":"2024-03-28T22:09:30.425101+0000","last_became_active":"2024-03-28T22:08:29.961222+0000","last_became_peered":"2024-03-28T22:08:29.961222+0000","last_unstale":"2024-03-28T22:09:30.425101+0000","last_undegraded":"2024-03-28T22:09:30.425101+0000","last_fullsized":"2024-03-28T22:09:30.425101+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:47:05.310483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.055610+0000","last_change":"2024-03-28T22:08:29.964041+0000","last_active":"2024-03-28T22:09:30.055610+0000","last_peered":"2024-03-28T22:09:30.055610+0000","last_clean":"2024-03-28T22:09:30.055610+0000","last_became_active":"2024-03-28T22:08:29.963360+0000","last_became_peered":"2024-03-28T22:08:29.963360+0000","last_unstale":"2024-03-28T22:09:30.055610+0000","last_undegraded":"2024-03-28T22:09:30.055610+0000","last_fullsized":"2024-03-28T22:09:30.055610+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:32.921916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.079414+0000","last_change":"2024-03-28T22:08:29.966449+0000","last_active":"2024-03-28T22:09:30.079414+0000","last_peered":"2024-03-28T22:09:30.079414+0000","last_clean":"2024-03-28T22:09:30.079414+0000","last_became_active":"2024-03-28T22:08:29.966036+0000","last_became_peered":"2024-03-28T22:08:29.966036+0000","last_unstale":"2024-03-28T22:09:30.079414+0000","last_undegraded":"2024-03-28T22:09:30.079414+0000","last_fullsized":"2024-03-28T22:09:30.079414+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:12:15.057713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.293856+0000","last_change":"2024-03-28T22:08:29.958924+0000","last_active":"2024-03-28T22:09:30.293856+0000","last_peered":"2024-03-28T22:09:30.293856+0000","last_clean":"2024-03-28T22:09:30.293856+0000","last_became_active":"2024-03-28T22:08:29.958601+0000","last_became_peered":"2024-03-28T22:08:29.958601+0000","last_unstale":"2024-03-28T22:09:30.293856+0000","last_undegraded":"2024-03-28T22:09:30.293856+0000","last_fullsized":"2024-03-28T22:09:30.293856+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:53:37.070276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058918+0000","last_change":"2024-03-28T22:08:29.965045+0000","last_active":"2024-03-28T22:09:30.058918+0000","last_peered":"2024-03-28T22:09:30.058918+0000","last_clean":"2024-03-28T22:09:30.058918+0000","last_became_active":"2024-03-28T22:08:29.964710+0000","last_became_peered":"2024-03-28T22:08:29.964710+0000","last_unstale":"2024-03-28T22:09:30.058918+0000","last_undegraded":"2024-03-28T22:09:30.058918+0000","last_fullsized":"2024-03-28T22:09:30.058918+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:03:11.491213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058745+0000","last_change":"2024-03-28T22:08:29.957940+0000","last_active":"2024-03-28T22:09:30.058745+0000","last_peered":"2024-03-28T22:09:30.058745+0000","last_clean":"2024-03-28T22:09:30.058745+0000","last_became_active":"2024-03-28T22:08:29.957587+0000","last_became_peered":"2024-03-28T22:08:29.957587+0000","last_unstale":"2024-03-28T22:09:30.058745+0000","last_undegraded":"2024-03-28T22:09:30.058745+0000","last_fullsized":"2024-03-28T22:09:30.058745+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:03:58.557806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.424536+0000","last_change":"2024-03-28T22:08:29.957397+0000","last_active":"2024-03-28T22:09:30.424536+0000","last_peered":"2024-03-28T22:09:30.424536+0000","last_clean":"2024-03-28T22:09:30.424536+0000","last_became_active":"2024-03-28T22:08:29.957054+0000","last_became_peered":"2024-03-28T22:08:29.957054+0000","last_unstale":"2024-03-28T22:09:30.424536+0000","last_undegraded":"2024-03-28T22:09:30.424536+0000","last_fullsized":"2024-03-28T22:09:30.424536+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:35:28.500561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.056537+0000","last_change":"2024-03-28T22:08:29.963343+0000","last_active":"2024-03-28T22:09:30.056537+0000","last_peered":"2024-03-28T22:09:30.056537+0000","last_clean":"2024-03-28T22:09:30.056537+0000","last_became_active":"2024-03-28T22:08:29.962926+0000","last_became_peered":"2024-03-28T22:08:29.962926+0000","last_unstale":"2024-03-28T22:09:30.056537+0000","last_undegraded":"2024-03-28T22:09:30.056537+0000","last_fullsized":"2024-03-28T22:09:30.056537+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:15:39.133930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.054505+0000","last_change":"2024-03-28T22:08:29.955903+0000","last_active":"2024-03-28T22:09:30.054505+0000","last_peered":"2024-03-28T22:09:30.054505+0000","last_clean":"2024-03-28T22:09:30.054505+0000","last_became_active":"2024-03-28T22:08:29.955460+0000","last_became_peered":"2024-03-28T22:08:29.955460+0000","last_unstale":"2024-03-28T22:09:30.054505+0000","last_undegraded":"2024-03-28T22:09:30.054505+0000","last_fullsized":"2024-03-28T22:09:30.054505+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:36:12.961475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058082+0000","last_change":"2024-03-28T22:08:29.954279+0000","last_active":"2024-03-28T22:09:30.058082+0000","last_peered":"2024-03-28T22:09:30.058082+0000","last_clean":"2024-03-28T22:09:30.058082+0000","last_became_active":"2024-03-28T22:08:29.953869+0000","last_became_peered":"2024-03-28T22:08:29.953869+0000","last_unstale":"2024-03-28T22:09:30.058082+0000","last_undegraded":"2024-03-28T22:09:30.058082+0000","last_fullsized":"2024-03-28T22:09:30.058082+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:05:52.247391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.054735+0000","last_change":"2024-03-28T22:08:29.327644+0000","last_active":"2024-03-28T22:09:30.054735+0000","last_peered":"2024-03-28T22:09:30.054735+0000","last_clean":"2024-03-28T22:09:30.054735+0000","last_became_active":"2024-03-28T22:08:29.327325+0000","last_became_peered":"2024-03-28T22:08:29.327325+0000","last_unstale":"2024-03-28T22:09:30.054735+0000","last_undegraded":"2024-03-28T22:09:30.054735+0000","last_fullsized":"2024-03-28T22:09:30.054735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:26:54.782703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058093+0000","last_change":"2024-03-28T22:08:29.962905+0000","last_active":"2024-03-28T22:09:30.058093+0000","last_peered":"2024-03-28T22:09:30.058093+0000","last_clean":"2024-03-28T22:09:30.058093+0000","last_became_active":"2024-03-28T22:08:29.962559+0000","last_became_peered":"2024-03-28T22:08:29.962559+0000","last_unstale":"2024-03-28T22:09:30.058093+0000","last_undegraded":"2024-03-28T22:09:30.058093+0000","last_fullsized":"2024-03-28T22:09:30.058093+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:02.347584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058126+0000","last_change":"2024-03-28T22:08:29.334871+0000","last_active":"2024-03-28T22:09:30.058126+0000","last_peered":"2024-03-28T22:09:30.058126+0000","last_clean":"2024-03-28T22:09:30.058126+0000","last_became_active":"2024-03-28T22:08:29.334565+0000","last_became_peered":"2024-03-28T22:08:29.334565+0000","last_unstale":"2024-03-28T22:09:30.058126+0000","last_undegraded":"2024-03-28T22:09:30.058126+0000","last_fullsized":"2024-03-28T22:09:30.058126+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:31:19.333496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.059794+0000","last_change":"2024-03-28T22:08:29.957622+0000","last_active":"2024-03-28T22:09:30.059794+0000","last_peered":"2024-03-28T22:09:30.059794+0000","last_clean":"2024-03-28T22:09:30.059794+0000","last_became_active":"2024-03-28T22:08:29.957208+0000","last_became_peered":"2024-03-28T22:08:29.957208+0000","last_unstale":"2024-03-28T22:09:30.059794+0000","last_undegraded":"2024-03-28T22:09:30.059794+0000","last_fullsized":"2024-03-28T22:09:30.059794+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:49:44.684203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058576+0000","last_change":"2024-03-28T22:08:29.330291+0000","last_active":"2024-03-28T22:09:30.058576+0000","last_peered":"2024-03-28T22:09:30.058576+0000","last_clean":"2024-03-28T22:09:30.058576+0000","last_became_active":"2024-03-28T22:08:29.328955+0000","last_became_peered":"2024-03-28T22:08:29.328955+0000","last_unstale":"2024-03-28T22:09:30.058576+0000","last_undegraded":"2024-03-28T22:09:30.058576+0000","last_fullsized":"2024-03-28T22:09:30.058576+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:10.838059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.424640+0000","last_change":"2024-03-28T22:08:29.966681+0000","last_active":"2024-03-28T22:09:30.424640+0000","last_peered":"2024-03-28T22:09:30.424640+0000","last_clean":"2024-03-28T22:09:30.424640+0000","last_became_active":"2024-03-28T22:08:29.966023+0000","last_became_peered":"2024-03-28T22:08:29.966023+0000","last_unstale":"2024-03-28T22:09:30.424640+0000","last_undegraded":"2024-03-28T22:09:30.424640+0000","last_fullsized":"2024-03-28T22:09:30.424640+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:04:43.281259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058443+0000","last_change":"2024-03-28T22:08:29.329026+0000","last_active":"2024-03-28T22:09:30.058443+0000","last_peered":"2024-03-28T22:09:30.058443+0000","last_clean":"2024-03-28T22:09:30.058443+0000","last_became_active":"2024-03-28T22:08:29.327570+0000","last_became_peered":"2024-03-28T22:08:29.327570+0000","last_unstale":"2024-03-28T22:09:30.058443+0000","last_undegraded":"2024-03-28T22:09:30.058443+0000","last_fullsized":"2024-03-28T22:09:30.058443+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:53:54.779073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.060453+0000","last_change":"2024-03-28T22:08:29.966455+0000","last_active":"2024-03-28T22:09:30.060453+0000","last_peered":"2024-03-28T22:09:30.060453+0000","last_clean":"2024-03-28T22:09:30.060453+0000","last_became_active":"2024-03-28T22:08:29.964451+0000","last_became_peered":"2024-03-28T22:08:29.964451+0000","last_unstale":"2024-03-28T22:09:30.060453+0000","last_undegraded":"2024-03-28T22:09:30.060453+0000","last_fullsized":"2024-03-28T22:09:30.060453+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:46:22.811146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'6","reported_seq":95,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:39.441567+0000","last_change":"2024-03-28T22:08:29.327397+0000","last_active":"2024-03-28T22:09:39.441567+0000","last_peered":"2024-03-28T22:09:39.441567+0000","last_clean":"2024-03-28T22:09:39.441567+0000","last_became_active":"2024-03-28T22:08:29.327009+0000","last_became_peered":"2024-03-28T22:08:29.327009+0000","last_unstale":"2024-03-28T22:09:39.441567+0000","last_undegraded":"2024-03-28T22:09:39.441567+0000","last_fullsized":"2024-03-28T22:09:39.441567+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:38:51.951934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058110+0000","last_change":"2024-03-28T22:08:29.969821+0000","last_active":"2024-03-28T22:09:30.058110+0000","last_peered":"2024-03-28T22:09:30.058110+0000","last_clean":"2024-03-28T22:09:30.058110+0000","last_became_active":"2024-03-28T22:08:29.969386+0000","last_became_peered":"2024-03-28T22:08:29.969386+0000","last_unstale":"2024-03-28T22:09:30.058110+0000","last_undegraded":"2024-03-28T22:09:30.058110+0000","last_fullsized":"2024-03-28T22:09:30.058110+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:44:19.753083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.424985+0000","last_change":"2024-03-28T22:08:29.332385+0000","last_active":"2024-03-28T22:09:30.424985+0000","last_peered":"2024-03-28T22:09:30.424985+0000","last_clean":"2024-03-28T22:09:30.424985+0000","last_became_active":"2024-03-28T22:08:29.331956+0000","last_became_peered":"2024-03-28T22:08:29.331956+0000","last_unstale":"2024-03-28T22:09:30.424985+0000","last_undegraded":"2024-03-28T22:09:30.424985+0000","last_fullsized":"2024-03-28T22:09:30.424985+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:10:04.428034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.080320+0000","last_change":"2024-03-28T22:09:30.080320+0000","last_active":"2024-03-28T22:09:30.080320+0000","last_peered":"2024-03-28T22:09:30.080320+0000","last_clean":"2024-03-28T22:09:30.080320+0000","last_became_active":"2024-03-28T22:09:30.079928+0000","last_became_peered":"2024-03-28T22:09:30.079928+0000","last_unstale":"2024-03-28T22:09:30.080320+0000","last_undegraded":"2024-03-28T22:09:30.080320+0000","last_fullsized":"2024-03-28T22:09:30.080320+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:12:53.385167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.293420+0000","last_change":"2024-03-28T22:08:29.327180+0000","last_active":"2024-03-28T22:09:30.293420+0000","last_peered":"2024-03-28T22:09:30.293420+0000","last_clean":"2024-03-28T22:09:30.293420+0000","last_became_active":"2024-03-28T22:08:29.326825+0000","last_became_peered":"2024-03-28T22:08:29.326825+0000","last_unstale":"2024-03-28T22:09:30.293420+0000","last_undegraded":"2024-03-28T22:09:30.293420+0000","last_fullsized":"2024-03-28T22:09:30.293420+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:11:09.979970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.057215+0000","last_change":"2024-03-28T22:08:29.955299+0000","last_active":"2024-03-28T22:09:30.057215+0000","last_peered":"2024-03-28T22:09:30.057215+0000","last_clean":"2024-03-28T22:09:30.057215+0000","last_became_active":"2024-03-28T22:08:29.954949+0000","last_became_peered":"2024-03-28T22:08:29.954949+0000","last_unstale":"2024-03-28T22:09:30.057215+0000","last_undegraded":"2024-03-28T22:09:30.057215+0000","last_fullsized":"2024-03-28T22:09:30.057215+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:01:48.375251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.059530+0000","last_change":"2024-03-28T22:08:29.327261+0000","last_active":"2024-03-28T22:09:30.059530+0000","last_peered":"2024-03-28T22:09:30.059530+0000","last_clean":"2024-03-28T22:09:30.059530+0000","last_became_active":"2024-03-28T22:08:29.326910+0000","last_became_peered":"2024-03-28T22:08:29.326910+0000","last_unstale":"2024-03-28T22:09:30.059530+0000","last_undegraded":"2024-03-28T22:09:30.059530+0000","last_fullsized":"2024-03-28T22:09:30.059530+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:46:41.933372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.054198+0000","last_change":"2024-03-28T22:08:29.954891+0000","last_active":"2024-03-28T22:09:30.054198+0000","last_peered":"2024-03-28T22:09:30.054198+0000","last_clean":"2024-03-28T22:09:30.054198+0000","last_became_active":"2024-03-28T22:08:29.954519+0000","last_became_peered":"2024-03-28T22:08:29.954519+0000","last_unstale":"2024-03-28T22:09:30.054198+0000","last_undegraded":"2024-03-28T22:09:30.054198+0000","last_fullsized":"2024-03-28T22:09:30.054198+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:07:42.621189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":88,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:38.440128+0000","last_change":"2024-03-28T22:08:29.330180+0000","last_active":"2024-03-28T22:09:38.440128+0000","last_peered":"2024-03-28T22:09:38.440128+0000","last_clean":"2024-03-28T22:09:38.440128+0000","last_became_active":"2024-03-28T22:08:29.329835+0000","last_became_peered":"2024-03-28T22:08:29.329835+0000","last_unstale":"2024-03-28T22:09:38.440128+0000","last_undegraded":"2024-03-28T22:09:38.440128+0000","last_fullsized":"2024-03-28T22:09:38.440128+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:48:39.906359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":66,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.425037+0000","last_change":"2024-03-28T22:08:29.967074+0000","last_active":"2024-03-28T22:09:30.425037+0000","last_peered":"2024-03-28T22:09:30.425037+0000","last_clean":"2024-03-28T22:09:30.425037+0000","last_became_active":"2024-03-28T22:08:29.966258+0000","last_became_peered":"2024-03-28T22:08:29.966258+0000","last_unstale":"2024-03-28T22:09:30.425037+0000","last_undegraded":"2024-03-28T22:09:30.425037+0000","last_fullsized":"2024-03-28T22:09:30.425037+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:25:44.986179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.055474+0000","last_change":"2024-03-28T22:08:29.326388+0000","last_active":"2024-03-28T22:09:30.055474+0000","last_peered":"2024-03-28T22:09:30.055474+0000","last_clean":"2024-03-28T22:09:30.055474+0000","last_became_active":"2024-03-28T22:08:29.326001+0000","last_became_peered":"2024-03-28T22:08:29.326001+0000","last_unstale":"2024-03-28T22:09:30.055474+0000","last_undegraded":"2024-03-28T22:09:30.055474+0000","last_fullsized":"2024-03-28T22:09:30.055474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:31:56.227038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.059682+0000","last_change":"2024-03-28T22:08:29.959299+0000","last_active":"2024-03-28T22:09:30.059682+0000","last_peered":"2024-03-28T22:09:30.059682+0000","last_clean":"2024-03-28T22:09:30.059682+0000","last_became_active":"2024-03-28T22:08:29.958955+0000","last_became_peered":"2024-03-28T22:08:29.958955+0000","last_unstale":"2024-03-28T22:09:30.059682+0000","last_undegraded":"2024-03-28T22:09:30.059682+0000","last_fullsized":"2024-03-28T22:09:30.059682+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:11:06.775920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.060567+0000","last_change":"2024-03-28T22:08:29.326089+0000","last_active":"2024-03-28T22:09:30.060567+0000","last_peered":"2024-03-28T22:09:30.060567+0000","last_clean":"2024-03-28T22:09:30.060567+0000","last_became_active":"2024-03-28T22:08:29.325463+0000","last_became_peered":"2024-03-28T22:08:29.325463+0000","last_unstale":"2024-03-28T22:09:30.060567+0000","last_undegraded":"2024-03-28T22:09:30.060567+0000","last_fullsized":"2024-03-28T22:09:30.060567+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:18:31.049729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.425439+0000","last_change":"2024-03-28T22:08:29.964248+0000","last_active":"2024-03-28T22:09:30.425439+0000","last_peered":"2024-03-28T22:09:30.425439+0000","last_clean":"2024-03-28T22:09:30.425439+0000","last_became_active":"2024-03-28T22:08:29.963897+0000","last_became_peered":"2024-03-28T22:08:29.963897+0000","last_unstale":"2024-03-28T22:09:30.425439+0000","last_undegraded":"2024-03-28T22:09:30.425439+0000","last_fullsized":"2024-03-28T22:09:30.425439+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:41:39.645360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'26","reported_seq":184,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:39.441151+0000","last_change":"2024-03-28T22:08:29.330732+0000","last_active":"2024-03-28T22:09:39.441151+0000","last_peered":"2024-03-28T22:09:39.441151+0000","last_clean":"2024-03-28T22:09:39.441151+0000","last_became_active":"2024-03-28T22:08:29.329231+0000","last_became_peered":"2024-03-28T22:08:29.329231+0000","last_unstale":"2024-03-28T22:09:39.441151+0000","last_undegraded":"2024-03-28T22:09:39.441151+0000","last_fullsized":"2024-03-28T22:09:39.441151+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:16:39.260429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58912,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":139,"num_read_kb":126,"num_write":27,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058333+0000","last_change":"2024-03-28T22:08:29.965791+0000","last_active":"2024-03-28T22:09:30.058333+0000","last_peered":"2024-03-28T22:09:30.058333+0000","last_clean":"2024-03-28T22:09:30.058333+0000","last_became_active":"2024-03-28T22:08:29.965390+0000","last_became_peered":"2024-03-28T22:08:29.965390+0000","last_unstale":"2024-03-28T22:09:30.058333+0000","last_undegraded":"2024-03-28T22:09:30.058333+0000","last_fullsized":"2024-03-28T22:09:30.058333+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:43:53.325906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.057898+0000","last_change":"2024-03-28T22:08:29.325851+0000","last_active":"2024-03-28T22:09:30.057898+0000","last_peered":"2024-03-28T22:09:30.057898+0000","last_clean":"2024-03-28T22:09:30.057898+0000","last_became_active":"2024-03-28T22:08:29.325500+0000","last_became_peered":"2024-03-28T22:08:29.325500+0000","last_unstale":"2024-03-28T22:09:30.057898+0000","last_undegraded":"2024-03-28T22:09:30.057898+0000","last_fullsized":"2024-03-28T22:09:30.057898+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:04:25.151908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.293686+0000","last_change":"2024-03-28T22:08:29.953857+0000","last_active":"2024-03-28T22:09:30.293686+0000","last_peered":"2024-03-28T22:09:30.293686+0000","last_clean":"2024-03-28T22:09:30.293686+0000","last_became_active":"2024-03-28T22:08:29.953538+0000","last_became_peered":"2024-03-28T22:08:29.953538+0000","last_unstale":"2024-03-28T22:09:30.293686+0000","last_undegraded":"2024-03-28T22:09:30.293686+0000","last_fullsized":"2024-03-28T22:09:30.293686+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:40:32.404760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.060057+0000","last_change":"2024-03-28T22:08:29.327286+0000","last_active":"2024-03-28T22:09:30.060057+0000","last_peered":"2024-03-28T22:09:30.060057+0000","last_clean":"2024-03-28T22:09:30.060057+0000","last_became_active":"2024-03-28T22:08:29.326902+0000","last_became_peered":"2024-03-28T22:08:29.326902+0000","last_unstale":"2024-03-28T22:09:30.060057+0000","last_undegraded":"2024-03-28T22:09:30.060057+0000","last_fullsized":"2024-03-28T22:09:30.060057+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:14:58.669534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.294946+0000","last_change":"2024-03-28T22:09:30.294946+0000","last_active":"2024-03-28T22:09:30.294946+0000","last_peered":"2024-03-28T22:09:30.294946+0000","last_clean":"2024-03-28T22:09:30.294946+0000","last_became_active":"2024-03-28T22:09:30.294496+0000","last_became_peered":"2024-03-28T22:09:30.294496+0000","last_unstale":"2024-03-28T22:09:30.294946+0000","last_undegraded":"2024-03-28T22:09:30.294946+0000","last_fullsized":"2024-03-28T22:09:30.294946+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:26:40.062384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'4","reported_seq":94,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:40.443177+0000","last_change":"2024-03-28T22:08:29.327740+0000","last_active":"2024-03-28T22:09:40.443177+0000","last_peered":"2024-03-28T22:09:40.443177+0000","last_clean":"2024-03-28T22:09:40.443177+0000","last_became_active":"2024-03-28T22:08:29.326664+0000","last_became_peered":"2024-03-28T22:08:29.326664+0000","last_unstale":"2024-03-28T22:09:40.443177+0000","last_undegraded":"2024-03-28T22:09:40.443177+0000","last_fullsized":"2024-03-28T22:09:40.443177+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:58:17.681836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.054778+0000","last_change":"2024-03-28T22:08:29.961979+0000","last_active":"2024-03-28T22:09:30.054778+0000","last_peered":"2024-03-28T22:09:30.054778+0000","last_clean":"2024-03-28T22:09:30.054778+0000","last_became_active":"2024-03-28T22:08:29.961615+0000","last_became_peered":"2024-03-28T22:08:29.961615+0000","last_unstale":"2024-03-28T22:09:30.054778+0000","last_undegraded":"2024-03-28T22:09:30.054778+0000","last_fullsized":"2024-03-28T22:09:30.054778+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:50:37.227632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.057515+0000","last_change":"2024-03-28T22:08:29.329017+0000","last_active":"2024-03-28T22:09:30.057515+0000","last_peered":"2024-03-28T22:09:30.057515+0000","last_clean":"2024-03-28T22:09:30.057515+0000","last_became_active":"2024-03-28T22:08:29.328686+0000","last_became_peered":"2024-03-28T22:08:29.328686+0000","last_unstale":"2024-03-28T22:09:30.057515+0000","last_undegraded":"2024-03-28T22:09:30.057515+0000","last_fullsized":"2024-03-28T22:09:30.057515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:25:28.986165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.426320+0000","last_change":"2024-03-28T22:09:30.426320+0000","last_active":"2024-03-28T22:09:30.426320+0000","last_peered":"2024-03-28T22:09:30.426320+0000","last_clean":"2024-03-28T22:09:30.426320+0000","last_became_active":"2024-03-28T22:09:30.425862+0000","last_became_peered":"2024-03-28T22:09:30.425862+0000","last_unstale":"2024-03-28T22:09:30.426320+0000","last_undegraded":"2024-03-28T22:09:30.426320+0000","last_fullsized":"2024-03-28T22:09:30.426320+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:44.240551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058066+0000","last_change":"2024-03-28T22:08:29.323274+0000","last_active":"2024-03-28T22:09:30.058066+0000","last_peered":"2024-03-28T22:09:30.058066+0000","last_clean":"2024-03-28T22:09:30.058066+0000","last_became_active":"2024-03-28T22:08:29.322911+0000","last_became_peered":"2024-03-28T22:08:29.322911+0000","last_unstale":"2024-03-28T22:09:30.058066+0000","last_undegraded":"2024-03-28T22:09:30.058066+0000","last_fullsized":"2024-03-28T22:09:30.058066+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:44:14.439411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.056385+0000","last_change":"2024-03-28T22:08:29.961101+0000","last_active":"2024-03-28T22:09:30.056385+0000","last_peered":"2024-03-28T22:09:30.056385+0000","last_clean":"2024-03-28T22:09:30.056385+0000","last_became_active":"2024-03-28T22:08:29.960767+0000","last_became_peered":"2024-03-28T22:08:29.960767+0000","last_unstale":"2024-03-28T22:09:30.056385+0000","last_undegraded":"2024-03-28T22:09:30.056385+0000","last_fullsized":"2024-03-28T22:09:30.056385+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:33:45.236691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.060215+0000","last_change":"2024-03-28T22:09:30.060215+0000","last_active":"2024-03-28T22:09:30.060215+0000","last_peered":"2024-03-28T22:09:30.060215+0000","last_clean":"2024-03-28T22:09:30.060215+0000","last_became_active":"2024-03-28T22:09:30.059828+0000","last_became_peered":"2024-03-28T22:09:30.059828+0000","last_unstale":"2024-03-28T22:09:30.060215+0000","last_undegraded":"2024-03-28T22:09:30.060215+0000","last_fullsized":"2024-03-28T22:09:30.060215+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:42:05.087899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.425494+0000","last_change":"2024-03-28T22:08:29.954093+0000","last_active":"2024-03-28T22:09:30.425494+0000","last_peered":"2024-03-28T22:09:30.425494+0000","last_clean":"2024-03-28T22:09:30.425494+0000","last_became_active":"2024-03-28T22:08:29.953688+0000","last_became_peered":"2024-03-28T22:08:29.953688+0000","last_unstale":"2024-03-28T22:09:30.425494+0000","last_undegraded":"2024-03-28T22:09:30.425494+0000","last_fullsized":"2024-03-28T22:09:30.425494+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:57:51.026565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.055115+0000","last_change":"2024-03-28T22:08:29.331104+0000","last_active":"2024-03-28T22:09:30.055115+0000","last_peered":"2024-03-28T22:09:30.055115+0000","last_clean":"2024-03-28T22:09:30.055115+0000","last_became_active":"2024-03-28T22:08:29.330505+0000","last_became_peered":"2024-03-28T22:08:29.330505+0000","last_unstale":"2024-03-28T22:09:30.055115+0000","last_undegraded":"2024-03-28T22:09:30.055115+0000","last_fullsized":"2024-03-28T22:09:30.055115+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:36:59.944587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.425499+0000","last_change":"2024-03-28T22:08:29.963540+0000","last_active":"2024-03-28T22:09:30.425499+0000","last_peered":"2024-03-28T22:09:30.425499+0000","last_clean":"2024-03-28T22:09:30.425499+0000","last_became_active":"2024-03-28T22:08:29.963206+0000","last_became_peered":"2024-03-28T22:08:29.963206+0000","last_unstale":"2024-03-28T22:09:30.425499+0000","last_undegraded":"2024-03-28T22:09:30.425499+0000","last_fullsized":"2024-03-28T22:09:30.425499+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:55:53.292554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.059727+0000","last_change":"2024-03-28T22:08:29.962822+0000","last_active":"2024-03-28T22:09:30.059727+0000","last_peered":"2024-03-28T22:09:30.059727+0000","last_clean":"2024-03-28T22:09:30.059727+0000","last_became_active":"2024-03-28T22:08:29.962343+0000","last_became_peered":"2024-03-28T22:08:29.962343+0000","last_unstale":"2024-03-28T22:09:30.059727+0000","last_undegraded":"2024-03-28T22:09:30.059727+0000","last_fullsized":"2024-03-28T22:09:30.059727+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:36.608543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":90,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:39.441298+0000","last_change":"2024-03-28T22:08:29.328239+0000","last_active":"2024-03-28T22:09:39.441298+0000","last_peered":"2024-03-28T22:09:39.441298+0000","last_clean":"2024-03-28T22:09:39.441298+0000","last_became_active":"2024-03-28T22:08:29.327903+0000","last_became_peered":"2024-03-28T22:08:29.327903+0000","last_unstale":"2024-03-28T22:09:39.441298+0000","last_undegraded":"2024-03-28T22:09:39.441298+0000","last_fullsized":"2024-03-28T22:09:39.441298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:14:17.558472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":67,"num_read_kb":67,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.426683+0000","last_change":"2024-03-28T22:09:30.426683+0000","last_active":"2024-03-28T22:09:30.426683+0000","last_peered":"2024-03-28T22:09:30.426683+0000","last_clean":"2024-03-28T22:09:30.426683+0000","last_became_active":"2024-03-28T22:09:30.426255+0000","last_became_peered":"2024-03-28T22:09:30.426255+0000","last_unstale":"2024-03-28T22:09:30.426683+0000","last_undegraded":"2024-03-28T22:09:30.426683+0000","last_fullsized":"2024-03-28T22:09:30.426683+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:36.755978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.079359+0000","last_change":"2024-03-28T22:08:29.323050+0000","last_active":"2024-03-28T22:09:30.079359+0000","last_peered":"2024-03-28T22:09:30.079359+0000","last_clean":"2024-03-28T22:09:30.079359+0000","last_became_active":"2024-03-28T22:08:29.322684+0000","last_became_peered":"2024-03-28T22:08:29.322684+0000","last_unstale":"2024-03-28T22:09:30.079359+0000","last_undegraded":"2024-03-28T22:09:30.079359+0000","last_fullsized":"2024-03-28T22:09:30.079359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:14:43.811122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":353,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.054871+0000","last_change":"2024-03-28T22:07:17.785516+0000","last_active":"2024-03-28T22:09:30.054871+0000","last_peered":"2024-03-28T22:09:30.054871+0000","last_clean":"2024-03-28T22:09:30.054871+0000","last_became_active":"2024-03-28T22:07:17.785123+0000","last_became_peered":"2024-03-28T22:07:17.785123+0000","last_unstale":"2024-03-28T22:09:30.054871+0000","last_undegraded":"2024-03-28T22:09:30.054871+0000","last_fullsized":"2024-03-28T22:09:30.054871+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:03:59.840513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:03:59.840513+0000","last_clean_scrub_stamp":"2024-03-28T22:03:59.840513+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:31:06.253632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.293289+0000","last_change":"2024-03-28T22:08:29.961489+0000","last_active":"2024-03-28T22:09:30.293289+0000","last_peered":"2024-03-28T22:09:30.293289+0000","last_clean":"2024-03-28T22:09:30.293289+0000","last_became_active":"2024-03-28T22:08:29.960881+0000","last_became_peered":"2024-03-28T22:08:29.960881+0000","last_unstale":"2024-03-28T22:09:30.293289+0000","last_undegraded":"2024-03-28T22:09:30.293289+0000","last_fullsized":"2024-03-28T22:09:30.293289+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:48.870834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.054972+0000","last_change":"2024-03-28T22:08:29.331994+0000","last_active":"2024-03-28T22:09:30.054972+0000","last_peered":"2024-03-28T22:09:30.054972+0000","last_clean":"2024-03-28T22:09:30.054972+0000","last_became_active":"2024-03-28T22:08:29.331630+0000","last_became_peered":"2024-03-28T22:08:29.331630+0000","last_unstale":"2024-03-28T22:09:30.054972+0000","last_undegraded":"2024-03-28T22:09:30.054972+0000","last_fullsized":"2024-03-28T22:09:30.054972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:11:37.100405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058466+0000","last_change":"2024-03-28T22:08:29.330353+0000","last_active":"2024-03-28T22:09:30.058466+0000","last_peered":"2024-03-28T22:09:30.058466+0000","last_clean":"2024-03-28T22:09:30.058466+0000","last_became_active":"2024-03-28T22:08:29.329421+0000","last_became_peered":"2024-03-28T22:08:29.329421+0000","last_unstale":"2024-03-28T22:09:30.058466+0000","last_undegraded":"2024-03-28T22:09:30.058466+0000","last_fullsized":"2024-03-28T22:09:30.058466+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:56.166506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.293863+0000","last_change":"2024-03-28T22:08:29.948595+0000","last_active":"2024-03-28T22:09:30.293863+0000","last_peered":"2024-03-28T22:09:30.293863+0000","last_clean":"2024-03-28T22:09:30.293863+0000","last_became_active":"2024-03-28T22:08:29.946986+0000","last_became_peered":"2024-03-28T22:08:29.946986+0000","last_unstale":"2024-03-28T22:09:30.293863+0000","last_undegraded":"2024-03-28T22:09:30.293863+0000","last_fullsized":"2024-03-28T22:09:30.293863+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:56:12.351731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.054613+0000","last_change":"2024-03-28T22:08:29.333375+0000","last_active":"2024-03-28T22:09:30.054613+0000","last_peered":"2024-03-28T22:09:30.054613+0000","last_clean":"2024-03-28T22:09:30.054613+0000","last_became_active":"2024-03-28T22:08:29.331594+0000","last_became_peered":"2024-03-28T22:08:29.331594+0000","last_unstale":"2024-03-28T22:09:30.054613+0000","last_undegraded":"2024-03-28T22:09:30.054613+0000","last_fullsized":"2024-03-28T22:09:30.054613+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:07:43.936807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.057907+0000","last_change":"2024-03-28T22:08:29.959061+0000","last_active":"2024-03-28T22:09:30.057907+0000","last_peered":"2024-03-28T22:09:30.057907+0000","last_clean":"2024-03-28T22:09:30.057907+0000","last_became_active":"2024-03-28T22:08:29.958723+0000","last_became_peered":"2024-03-28T22:08:29.958723+0000","last_unstale":"2024-03-28T22:09:30.057907+0000","last_undegraded":"2024-03-28T22:09:30.057907+0000","last_fullsized":"2024-03-28T22:09:30.057907+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:32:23.016606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.079461+0000","last_change":"2024-03-28T22:08:29.329886+0000","last_active":"2024-03-28T22:09:30.079461+0000","last_peered":"2024-03-28T22:09:30.079461+0000","last_clean":"2024-03-28T22:09:30.079461+0000","last_became_active":"2024-03-28T22:08:29.329503+0000","last_became_peered":"2024-03-28T22:08:29.329503+0000","last_unstale":"2024-03-28T22:09:30.079461+0000","last_undegraded":"2024-03-28T22:09:30.079461+0000","last_fullsized":"2024-03-28T22:09:30.079461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:50:05.686647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.059336+0000","last_change":"2024-03-28T22:08:29.957038+0000","last_active":"2024-03-28T22:09:30.059336+0000","last_peered":"2024-03-28T22:09:30.059336+0000","last_clean":"2024-03-28T22:09:30.059336+0000","last_became_active":"2024-03-28T22:08:29.956693+0000","last_became_peered":"2024-03-28T22:08:29.956693+0000","last_unstale":"2024-03-28T22:09:30.059336+0000","last_undegraded":"2024-03-28T22:09:30.059336+0000","last_fullsized":"2024-03-28T22:09:30.059336+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:19:09.881431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.424647+0000","last_change":"2024-03-28T22:08:29.327356+0000","last_active":"2024-03-28T22:09:30.424647+0000","last_peered":"2024-03-28T22:09:30.424647+0000","last_clean":"2024-03-28T22:09:30.424647+0000","last_became_active":"2024-03-28T22:08:29.326947+0000","last_became_peered":"2024-03-28T22:08:29.326947+0000","last_unstale":"2024-03-28T22:09:30.424647+0000","last_undegraded":"2024-03-28T22:09:30.424647+0000","last_fullsized":"2024-03-28T22:09:30.424647+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:47:29.417465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.057637+0000","last_change":"2024-03-28T22:08:29.966505+0000","last_active":"2024-03-28T22:09:30.057637+0000","last_peered":"2024-03-28T22:09:30.057637+0000","last_clean":"2024-03-28T22:09:30.057637+0000","last_became_active":"2024-03-28T22:08:29.966057+0000","last_became_peered":"2024-03-28T22:08:29.966057+0000","last_unstale":"2024-03-28T22:09:30.057637+0000","last_undegraded":"2024-03-28T22:09:30.057637+0000","last_fullsized":"2024-03-28T22:09:30.057637+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:17:45.196585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.057477+0000","last_change":"2024-03-28T22:08:29.328897+0000","last_active":"2024-03-28T22:09:30.057477+0000","last_peered":"2024-03-28T22:09:30.057477+0000","last_clean":"2024-03-28T22:09:30.057477+0000","last_became_active":"2024-03-28T22:08:29.328390+0000","last_became_peered":"2024-03-28T22:08:29.328390+0000","last_unstale":"2024-03-28T22:09:30.057477+0000","last_undegraded":"2024-03-28T22:09:30.057477+0000","last_fullsized":"2024-03-28T22:09:30.057477+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:21:12.699503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.055247+0000","last_change":"2024-03-28T22:08:29.957212+0000","last_active":"2024-03-28T22:09:30.055247+0000","last_peered":"2024-03-28T22:09:30.055247+0000","last_clean":"2024-03-28T22:09:30.055247+0000","last_became_active":"2024-03-28T22:08:29.956215+0000","last_became_peered":"2024-03-28T22:08:29.956215+0000","last_unstale":"2024-03-28T22:09:30.055247+0000","last_undegraded":"2024-03-28T22:09:30.055247+0000","last_fullsized":"2024-03-28T22:09:30.055247+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:57:25.199993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.063151+0000","last_change":"2024-03-28T22:09:30.062742+0000","last_active":"2024-03-28T22:09:30.063151+0000","last_peered":"2024-03-28T22:09:30.063151+0000","last_clean":"2024-03-28T22:09:30.063151+0000","last_became_active":"2024-03-28T22:09:30.059356+0000","last_became_peered":"2024-03-28T22:09:30.059356+0000","last_unstale":"2024-03-28T22:09:30.063151+0000","last_undegraded":"2024-03-28T22:09:30.063151+0000","last_fullsized":"2024-03-28T22:09:30.063151+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:52:39.412153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058322+0000","last_change":"2024-03-28T22:08:29.955023+0000","last_active":"2024-03-28T22:09:30.058322+0000","last_peered":"2024-03-28T22:09:30.058322+0000","last_clean":"2024-03-28T22:09:30.058322+0000","last_became_active":"2024-03-28T22:08:29.954675+0000","last_became_peered":"2024-03-28T22:08:29.954675+0000","last_unstale":"2024-03-28T22:09:30.058322+0000","last_undegraded":"2024-03-28T22:09:30.058322+0000","last_fullsized":"2024-03-28T22:09:30.058322+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:31.246452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058628+0000","last_change":"2024-03-28T22:08:29.329280+0000","last_active":"2024-03-28T22:09:30.058628+0000","last_peered":"2024-03-28T22:09:30.058628+0000","last_clean":"2024-03-28T22:09:30.058628+0000","last_became_active":"2024-03-28T22:08:29.328860+0000","last_became_peered":"2024-03-28T22:08:29.328860+0000","last_unstale":"2024-03-28T22:09:30.058628+0000","last_undegraded":"2024-03-28T22:09:30.058628+0000","last_fullsized":"2024-03-28T22:09:30.058628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:15:49.610026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.055286+0000","last_change":"2024-03-28T22:08:29.966224+0000","last_active":"2024-03-28T22:09:30.055286+0000","last_peered":"2024-03-28T22:09:30.055286+0000","last_clean":"2024-03-28T22:09:30.055286+0000","last_became_active":"2024-03-28T22:08:29.965636+0000","last_became_peered":"2024-03-28T22:08:29.965636+0000","last_unstale":"2024-03-28T22:09:30.055286+0000","last_undegraded":"2024-03-28T22:09:30.055286+0000","last_fullsized":"2024-03-28T22:09:30.055286+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:23:31.230980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058666+0000","last_change":"2024-03-28T22:08:29.335228+0000","last_active":"2024-03-28T22:09:30.058666+0000","last_peered":"2024-03-28T22:09:30.058666+0000","last_clean":"2024-03-28T22:09:30.058666+0000","last_became_active":"2024-03-28T22:08:29.334686+0000","last_became_peered":"2024-03-28T22:08:29.334686+0000","last_unstale":"2024-03-28T22:09:30.058666+0000","last_undegraded":"2024-03-28T22:09:30.058666+0000","last_fullsized":"2024-03-28T22:09:30.058666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:17:40.571206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.055703+0000","last_change":"2024-03-28T22:08:29.963912+0000","last_active":"2024-03-28T22:09:30.055703+0000","last_peered":"2024-03-28T22:09:30.055703+0000","last_clean":"2024-03-28T22:09:30.055703+0000","last_became_active":"2024-03-28T22:08:29.962740+0000","last_became_peered":"2024-03-28T22:08:29.962740+0000","last_unstale":"2024-03-28T22:09:30.055703+0000","last_undegraded":"2024-03-28T22:09:30.055703+0000","last_fullsized":"2024-03-28T22:09:30.055703+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:50:25.756923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.060333+0000","last_change":"2024-03-28T22:08:29.325767+0000","last_active":"2024-03-28T22:09:30.060333+0000","last_peered":"2024-03-28T22:09:30.060333+0000","last_clean":"2024-03-28T22:09:30.060333+0000","last_became_active":"2024-03-28T22:08:29.325430+0000","last_became_peered":"2024-03-28T22:08:29.325430+0000","last_unstale":"2024-03-28T22:09:30.060333+0000","last_undegraded":"2024-03-28T22:09:30.060333+0000","last_fullsized":"2024-03-28T22:09:30.060333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:01:13.663163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.055431+0000","last_change":"2024-03-28T22:08:29.960158+0000","last_active":"2024-03-28T22:09:30.055431+0000","last_peered":"2024-03-28T22:09:30.055431+0000","last_clean":"2024-03-28T22:09:30.055431+0000","last_became_active":"2024-03-28T22:08:29.959792+0000","last_became_peered":"2024-03-28T22:08:29.959792+0000","last_unstale":"2024-03-28T22:09:30.055431+0000","last_undegraded":"2024-03-28T22:09:30.055431+0000","last_fullsized":"2024-03-28T22:09:30.055431+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:28:49.135551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.293763+0000","last_change":"2024-03-28T22:08:29.328547+0000","last_active":"2024-03-28T22:09:30.293763+0000","last_peered":"2024-03-28T22:09:30.293763+0000","last_clean":"2024-03-28T22:09:30.293763+0000","last_became_active":"2024-03-28T22:08:29.327910+0000","last_became_peered":"2024-03-28T22:08:29.327910+0000","last_unstale":"2024-03-28T22:09:30.293763+0000","last_undegraded":"2024-03-28T22:09:30.293763+0000","last_fullsized":"2024-03-28T22:09:30.293763+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:06:13.980071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.055841+0000","last_change":"2024-03-28T22:08:29.963904+0000","last_active":"2024-03-28T22:09:30.055841+0000","last_peered":"2024-03-28T22:09:30.055841+0000","last_clean":"2024-03-28T22:09:30.055841+0000","last_became_active":"2024-03-28T22:08:29.963536+0000","last_became_peered":"2024-03-28T22:08:29.963536+0000","last_unstale":"2024-03-28T22:09:30.055841+0000","last_undegraded":"2024-03-28T22:09:30.055841+0000","last_fullsized":"2024-03-28T22:09:30.055841+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:59:43.376792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.055319+0000","last_change":"2024-03-28T22:08:29.962357+0000","last_active":"2024-03-28T22:09:30.055319+0000","last_peered":"2024-03-28T22:09:30.055319+0000","last_clean":"2024-03-28T22:09:30.055319+0000","last_became_active":"2024-03-28T22:08:29.961998+0000","last_became_peered":"2024-03-28T22:08:29.961998+0000","last_unstale":"2024-03-28T22:09:30.055319+0000","last_undegraded":"2024-03-28T22:09:30.055319+0000","last_fullsized":"2024-03-28T22:09:30.055319+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:40:47.465889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.057636+0000","last_change":"2024-03-28T22:08:29.961326+0000","last_active":"2024-03-28T22:09:30.057636+0000","last_peered":"2024-03-28T22:09:30.057636+0000","last_clean":"2024-03-28T22:09:30.057636+0000","last_became_active":"2024-03-28T22:08:29.960316+0000","last_became_peered":"2024-03-28T22:08:29.960316+0000","last_unstale":"2024-03-28T22:09:30.057636+0000","last_undegraded":"2024-03-28T22:09:30.057636+0000","last_fullsized":"2024-03-28T22:09:30.057636+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:37:55.773686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.057451+0000","last_change":"2024-03-28T22:08:29.956272+0000","last_active":"2024-03-28T22:09:30.057451+0000","last_peered":"2024-03-28T22:09:30.057451+0000","last_clean":"2024-03-28T22:09:30.057451+0000","last_became_active":"2024-03-28T22:08:29.955875+0000","last_became_peered":"2024-03-28T22:08:29.955875+0000","last_unstale":"2024-03-28T22:09:30.057451+0000","last_undegraded":"2024-03-28T22:09:30.057451+0000","last_fullsized":"2024-03-28T22:09:30.057451+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:17:33.062615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.294754+0000","last_change":"2024-03-28T22:09:30.294754+0000","last_active":"2024-03-28T22:09:30.294754+0000","last_peered":"2024-03-28T22:09:30.294754+0000","last_clean":"2024-03-28T22:09:30.294754+0000","last_became_active":"2024-03-28T22:09:30.294341+0000","last_became_peered":"2024-03-28T22:09:30.294341+0000","last_unstale":"2024-03-28T22:09:30.294754+0000","last_undegraded":"2024-03-28T22:09:30.294754+0000","last_fullsized":"2024-03-28T22:09:30.294754+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:25:46.017373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.293843+0000","last_change":"2024-03-28T22:08:29.960464+0000","last_active":"2024-03-28T22:09:30.293843+0000","last_peered":"2024-03-28T22:09:30.293843+0000","last_clean":"2024-03-28T22:09:30.293843+0000","last_became_active":"2024-03-28T22:08:29.960111+0000","last_became_peered":"2024-03-28T22:08:29.960111+0000","last_unstale":"2024-03-28T22:09:30.293843+0000","last_undegraded":"2024-03-28T22:09:30.293843+0000","last_fullsized":"2024-03-28T22:09:30.293843+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:32:33.533199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.056104+0000","last_change":"2024-03-28T22:08:29.960045+0000","last_active":"2024-03-28T22:09:30.056104+0000","last_peered":"2024-03-28T22:09:30.056104+0000","last_clean":"2024-03-28T22:09:30.056104+0000","last_became_active":"2024-03-28T22:08:29.959645+0000","last_became_peered":"2024-03-28T22:08:29.959645+0000","last_unstale":"2024-03-28T22:09:30.056104+0000","last_undegraded":"2024-03-28T22:09:30.056104+0000","last_fullsized":"2024-03-28T22:09:30.056104+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:34:56.202257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058450+0000","last_change":"2024-03-28T22:08:29.958736+0000","last_active":"2024-03-28T22:09:30.058450+0000","last_peered":"2024-03-28T22:09:30.058450+0000","last_clean":"2024-03-28T22:09:30.058450+0000","last_became_active":"2024-03-28T22:08:29.958387+0000","last_became_peered":"2024-03-28T22:08:29.958387+0000","last_unstale":"2024-03-28T22:09:30.058450+0000","last_undegraded":"2024-03-28T22:09:30.058450+0000","last_fullsized":"2024-03-28T22:09:30.058450+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:07:54.829166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.057847+0000","last_change":"2024-03-28T22:08:29.961373+0000","last_active":"2024-03-28T22:09:30.057847+0000","last_peered":"2024-03-28T22:09:30.057847+0000","last_clean":"2024-03-28T22:09:30.057847+0000","last_became_active":"2024-03-28T22:08:29.959751+0000","last_became_peered":"2024-03-28T22:08:29.959751+0000","last_unstale":"2024-03-28T22:09:30.057847+0000","last_undegraded":"2024-03-28T22:09:30.057847+0000","last_fullsized":"2024-03-28T22:09:30.057847+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:34:18.680552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.059658+0000","last_change":"2024-03-28T22:08:29.962721+0000","last_active":"2024-03-28T22:09:30.059658+0000","last_peered":"2024-03-28T22:09:30.059658+0000","last_clean":"2024-03-28T22:09:30.059658+0000","last_became_active":"2024-03-28T22:08:29.961349+0000","last_became_peered":"2024-03-28T22:08:29.961349+0000","last_unstale":"2024-03-28T22:09:30.059658+0000","last_undegraded":"2024-03-28T22:09:30.059658+0000","last_fullsized":"2024-03-28T22:09:30.059658+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:35:56.535055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.057325+0000","last_change":"2024-03-28T22:08:29.960068+0000","last_active":"2024-03-28T22:09:30.057325+0000","last_peered":"2024-03-28T22:09:30.057325+0000","last_clean":"2024-03-28T22:09:30.057325+0000","last_became_active":"2024-03-28T22:08:29.959716+0000","last_became_peered":"2024-03-28T22:08:29.959716+0000","last_unstale":"2024-03-28T22:09:30.057325+0000","last_undegraded":"2024-03-28T22:09:30.057325+0000","last_fullsized":"2024-03-28T22:09:30.057325+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:18:38.055322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.424859+0000","last_change":"2024-03-28T22:08:29.957920+0000","last_active":"2024-03-28T22:09:30.424859+0000","last_peered":"2024-03-28T22:09:30.424859+0000","last_clean":"2024-03-28T22:09:30.424859+0000","last_became_active":"2024-03-28T22:08:29.957541+0000","last_became_peered":"2024-03-28T22:08:29.957541+0000","last_unstale":"2024-03-28T22:09:30.424859+0000","last_undegraded":"2024-03-28T22:09:30.424859+0000","last_fullsized":"2024-03-28T22:09:30.424859+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:45:05.658745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.293859+0000","last_change":"2024-03-28T22:08:29.953589+0000","last_active":"2024-03-28T22:09:30.293859+0000","last_peered":"2024-03-28T22:09:30.293859+0000","last_clean":"2024-03-28T22:09:30.293859+0000","last_became_active":"2024-03-28T22:08:29.953253+0000","last_became_peered":"2024-03-28T22:08:29.953253+0000","last_unstale":"2024-03-28T22:09:30.293859+0000","last_undegraded":"2024-03-28T22:09:30.293859+0000","last_fullsized":"2024-03-28T22:09:30.293859+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:35.720708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.055833+0000","last_change":"2024-03-28T22:08:29.966235+0000","last_active":"2024-03-28T22:09:30.055833+0000","last_peered":"2024-03-28T22:09:30.055833+0000","last_clean":"2024-03-28T22:09:30.055833+0000","last_became_active":"2024-03-28T22:08:29.965830+0000","last_became_peered":"2024-03-28T22:08:29.965830+0000","last_unstale":"2024-03-28T22:09:30.055833+0000","last_undegraded":"2024-03-28T22:09:30.055833+0000","last_fullsized":"2024-03-28T22:09:30.055833+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:04:27.094743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63304,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":427,"num_read_kb":279,"num_write":155,"num_write_kb":106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":189912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134,"ondisk_log_size":134,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351004,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32108,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713044,"statfs":{"total":95995035648,"available":95962157056,"internally_reserved":0,"allocated":950272,"data_stored":459161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612640,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28628,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716524,"statfs":{"total":95995035648,"available":95965720576,"internally_reserved":0,"allocated":1646592,"data_stored":1109945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776165,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":966656,"data_stored":459397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037802,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":966656,"data_stored":459317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299440,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36864,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708288,"statfs":{"total":95995035648,"available":95957286912,"internally_reserved":0,"allocated":1560576,"data_stored":1051638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462964,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36272,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708880,"statfs":{"total":95995035648,"available":95957893120,"internally_reserved":0,"allocated":954368,"data_stored":458786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574924,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":517491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738449,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":460057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411401,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36360,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708792,"statfs":{"total":95995035648,"available":95957803008,"internally_reserved":0,"allocated":1040384,"data_stored":519232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149763,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":970752,"data_stored":459097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888127,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40440,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704712,"statfs":{"total":95995035648,"available":95953625088,"internally_reserved":0,"allocated":958464,"data_stored":459614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724602,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28560,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716592,"statfs":{"total":95995035648,"available":95965790208,"internally_reserved":0,"allocated":1572864,"data_stored":1050931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":58912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":60653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":60881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T22:09:44.841 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-28T22:09:44.841 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-28T22:09:44.841 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-28T22:09:44.841 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph health --format=json 2024-03-28T22:09:45.089 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:09:45.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:45 smithi053 ceph-mon[35979]: pgmap v326: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T22:09:45.551 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:45 smithi016 ceph-mon[32374]: pgmap v326: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T22:09:45.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:45 smithi063 ceph-mon[36249]: pgmap v326: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T22:09:46.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.689+0000 7f38076ac700 1 -- 172.21.15.16:0/4134748288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3800102c50 msgr2=0x7f380010f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:46.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.689+0000 7f38076ac700 1 --2- 172.21.15.16:0/4134748288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3800102c50 0x7f380010f090 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f37f80099c0 tx=0x7f37f8009cd0 comp rx=0 tx=0).stop 2024-03-28T22:09:46.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.689+0000 7f38076ac700 1 -- 172.21.15.16:0/4134748288 shutdown_connections 2024-03-28T22:09:46.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.689+0000 7f38076ac700 1 --2- 172.21.15.16:0/4134748288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3800102c50 0x7f380010f090 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:46.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.689+0000 7f38076ac700 1 --2- 172.21.15.16:0/4134748288 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3800108270 0x7f3800108650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:46.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.689+0000 7f38076ac700 1 --2- 172.21.15.16:0/4134748288 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3800102290 0x7f3800102710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:46.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.689+0000 7f38076ac700 1 -- 172.21.15.16:0/4134748288 >> 172.21.15.16:0/4134748288 conn(0x7f3800079370 msgr2=0x7f3800079780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:46.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.690+0000 7f38076ac700 1 -- 172.21.15.16:0/4134748288 shutdown_connections 2024-03-28T22:09:46.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.690+0000 7f38076ac700 1 -- 172.21.15.16:0/4134748288 wait complete. 2024-03-28T22:09:46.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.690+0000 7f38076ac700 1 Processor -- start 2024-03-28T22:09:46.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.690+0000 7f38076ac700 1 -- start start 2024-03-28T22:09:46.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.690+0000 7f38076ac700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3800102290 0x7f3800074540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:46.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.690+0000 7f38076ac700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3800102c50 0x7f3800074a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:46.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.690+0000 7f38076ac700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3800108270 0x7f3800074fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:46.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.690+0000 7f38076ac700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f38001124f0 con 0x7f3800102c50 2024-03-28T22:09:46.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.690+0000 7f38076ac700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3800112370 con 0x7f3800102290 2024-03-28T22:09:46.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.690+0000 7f38076ac700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3800112670 con 0x7f3800108270 2024-03-28T22:09:46.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.690+0000 7f3804c47700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3800102c50 0x7f3800074a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:46.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.691+0000 7f3804c47700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3800102c50 0x7f3800074a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56764/0 (socket says 172.21.15.16:56764) 2024-03-28T22:09:46.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.691+0000 7f3804c47700 1 -- 172.21.15.16:0/857381530 learned_addr learned my addr 172.21.15.16:0/857381530 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:46.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.691+0000 7f3805448700 1 --2- 172.21.15.16:0/857381530 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3800102290 0x7f3800074540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:46.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.691+0000 7f3805c49700 1 --2- 172.21.15.16:0/857381530 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3800108270 0x7f3800074fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:46.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.691+0000 7f3804c47700 1 -- 172.21.15.16:0/857381530 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3800108270 msgr2=0x7f3800074fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:46.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.691+0000 7f3804c47700 1 --2- 172.21.15.16:0/857381530 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3800108270 0x7f3800074fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:46.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.691+0000 7f3804c47700 1 -- 172.21.15.16:0/857381530 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3800102290 msgr2=0x7f3800074540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:46.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.691+0000 7f3804c47700 1 --2- 172.21.15.16:0/857381530 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3800102290 0x7f3800074540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:46.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.691+0000 7f3804c47700 1 -- 172.21.15.16:0/857381530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38001a6e00 con 0x7f3800102c50 2024-03-28T22:09:46.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.691+0000 7f3805c49700 1 --2- 172.21.15.16:0/857381530 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3800108270 0x7f3800074fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:09:46.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.691+0000 7f3804c47700 1 --2- 172.21.15.16:0/857381530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3800102c50 0x7f3800074a80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f37f400e9a0 tx=0x7f37f400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:46.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.691+0000 7f37fe7fc700 1 -- 172.21.15.16:0/857381530 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f400cbd0 con 0x7f3800102c50 2024-03-28T22:09:46.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.691+0000 7f38076ac700 1 -- 172.21.15.16:0/857381530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38001a70f0 con 0x7f3800102c50 2024-03-28T22:09:46.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.691+0000 7f37fe7fc700 1 -- 172.21.15.16:0/857381530 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37f400cd70 con 0x7f3800102c50 2024-03-28T22:09:46.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.691+0000 7f37fe7fc700 1 -- 172.21.15.16:0/857381530 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f4018750 con 0x7f3800102c50 2024-03-28T22:09:46.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.691+0000 7f38076ac700 1 -- 172.21.15.16:0/857381530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38001a7680 con 0x7f3800102c50 2024-03-28T22:09:46.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.692+0000 7f38076ac700 1 -- 172.21.15.16:0/857381530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3800068f00 con 0x7f3800102c50 2024-03-28T22:09:46.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.693+0000 7f37fe7fc700 1 -- 172.21.15.16:0/857381530 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f37f4010430 con 0x7f3800102c50 2024-03-28T22:09:46.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.694+0000 7f37fe7fc700 1 --2- 172.21.15.16:0/857381530 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f37e80a1ea0 0x7f37e80a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:46.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.694+0000 7f3805448700 1 --2- 172.21.15.16:0/857381530 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f37e80a1ea0 0x7f37e80a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:46.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.694+0000 7f37fe7fc700 1 -- 172.21.15.16:0/857381530 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f37f40ce6b0 con 0x7f3800102c50 2024-03-28T22:09:46.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.694+0000 7f3805448700 1 --2- 172.21.15.16:0/857381530 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f37e80a1ea0 0x7f37e80a4360 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f37f0009550 tx=0x7f37f00091b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:46.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.697+0000 7f37fe7fc700 1 -- 172.21.15.16:0/857381530 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f37f409b9b0 con 0x7f3800102c50 2024-03-28T22:09:46.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.946+0000 7f38076ac700 1 -- 172.21.15.16:0/857381530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f380002db30 con 0x7f3800102c50 2024-03-28T22:09:46.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.946+0000 7f37fe7fc700 1 -- 172.21.15.16:0/857381530 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f37f4095e20 con 0x7f3800102c50 2024-03-28T22:09:46.948 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:09:46.948 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-28T22:09:46.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.948+0000 7f38076ac700 1 -- 172.21.15.16:0/857381530 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f37e80a1ea0 msgr2=0x7f37e80a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:46.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.948+0000 7f38076ac700 1 --2- 172.21.15.16:0/857381530 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f37e80a1ea0 0x7f37e80a4360 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f37f0009550 tx=0x7f37f00091b0 comp rx=0 tx=0).stop 2024-03-28T22:09:46.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.948+0000 7f38076ac700 1 -- 172.21.15.16:0/857381530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3800102c50 msgr2=0x7f3800074a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:46.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.948+0000 7f38076ac700 1 --2- 172.21.15.16:0/857381530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3800102c50 0x7f3800074a80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f37f400e9a0 tx=0x7f37f400ecb0 comp rx=0 tx=0).stop 2024-03-28T22:09:46.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.948+0000 7f38076ac700 1 -- 172.21.15.16:0/857381530 shutdown_connections 2024-03-28T22:09:46.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.948+0000 7f38076ac700 1 --2- 172.21.15.16:0/857381530 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f37e80a1ea0 0x7f37e80a4360 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:46.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.948+0000 7f38076ac700 1 --2- 172.21.15.16:0/857381530 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3800108270 0x7f3800074fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:46.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.948+0000 7f38076ac700 1 --2- 172.21.15.16:0/857381530 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3800102290 0x7f3800074540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:46.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.948+0000 7f38076ac700 1 --2- 172.21.15.16:0/857381530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3800102c50 0x7f3800074a80 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:46.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.948+0000 7f38076ac700 1 -- 172.21.15.16:0/857381530 >> 172.21.15.16:0/857381530 conn(0x7f3800079370 msgr2=0x7f3800100e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:46.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.949+0000 7f38076ac700 1 -- 172.21.15.16:0/857381530 shutdown_connections 2024-03-28T22:09:46.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:46.949+0000 7f38076ac700 1 -- 172.21.15.16:0/857381530 wait complete. 2024-03-28T22:09:47.244 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:46 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/857381530' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-28T22:09:47.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:46 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/857381530' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-28T22:09:47.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:46 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/857381530' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-28T22:09:47.480 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-28T22:09:47.481 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:09:47.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.585+0000 7f0019c5c640 1 -- 172.21.15.16:0/3736787729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0014102980 msgr2=0x7f0014102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:47.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.585+0000 7f0019c5c640 1 --2- 172.21.15.16:0/3736787729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0014102980 0x7f0014102e00 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f0008009900 tx=0x7f000802bc30 comp rx=0 tx=0).stop 2024-03-28T22:09:47.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.585+0000 7f0019c5c640 1 -- 172.21.15.16:0/3736787729 shutdown_connections 2024-03-28T22:09:47.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.585+0000 7f0019c5c640 1 --2- 172.21.15.16:0/3736787729 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0014103340 0x7f0014109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:47.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.585+0000 7f0019c5c640 1 --2- 172.21.15.16:0/3736787729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0014102980 0x7f0014102e00 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:47.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.585+0000 7f0019c5c640 1 --2- 172.21.15.16:0/3736787729 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0014101790 0x7f0014101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:47.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.585+0000 7f0019c5c640 1 -- 172.21.15.16:0/3736787729 >> 172.21.15.16:0/3736787729 conn(0x7f00140fcce0 msgr2=0x7f00140ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:47.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.585+0000 7f0019c5c640 1 -- 172.21.15.16:0/3736787729 shutdown_connections 2024-03-28T22:09:47.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.585+0000 7f0019c5c640 1 -- 172.21.15.16:0/3736787729 wait complete. 2024-03-28T22:09:47.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.586+0000 7f0019c5c640 1 Processor -- start 2024-03-28T22:09:47.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.586+0000 7f0019c5c640 1 -- start start 2024-03-28T22:09:47.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.586+0000 7f0019c5c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0014101790 0x7f0014197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:47.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.586+0000 7f0019c5c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0014102980 0x7f0014198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:47.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.586+0000 7f0019c5c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0014103340 0x7f001419eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:47.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.586+0000 7f0019c5c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f001410d410 con 0x7f0014102980 2024-03-28T22:09:47.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.586+0000 7f0019c5c640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f001410d290 con 0x7f0014101790 2024-03-28T22:09:47.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.586+0000 7f0019c5c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f001410d590 con 0x7f0014103340 2024-03-28T22:09:47.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.586+0000 7f0012ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0014102980 0x7f0014198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:47.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.586+0000 7f00137fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0014101790 0x7f0014197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:47.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.586+0000 7f0012ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0014102980 0x7f0014198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56782/0 (socket says 172.21.15.16:56782) 2024-03-28T22:09:47.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.586+0000 7f0012ffd640 1 -- 172.21.15.16:0/3080094703 learned_addr learned my addr 172.21.15.16:0/3080094703 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:47.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.586+0000 7f0013fff640 1 --2- 172.21.15.16:0/3080094703 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0014103340 0x7f001419eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:47.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.586+0000 7f0012ffd640 1 -- 172.21.15.16:0/3080094703 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0014103340 msgr2=0x7f001419eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:47.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.586+0000 7f0012ffd640 1 --2- 172.21.15.16:0/3080094703 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0014103340 0x7f001419eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:47.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.586+0000 7f0012ffd640 1 -- 172.21.15.16:0/3080094703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0014101790 msgr2=0x7f0014197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:47.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.586+0000 7f0012ffd640 1 --2- 172.21.15.16:0/3080094703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0014101790 0x7f0014197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:47.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.586+0000 7f0012ffd640 1 -- 172.21.15.16:0/3080094703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f001419f660 con 0x7f0014102980 2024-03-28T22:09:47.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.586+0000 7f00137fe640 1 --2- 172.21.15.16:0/3080094703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0014101790 0x7f0014197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:09:47.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.587+0000 7f0012ffd640 1 --2- 172.21.15.16:0/3080094703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0014102980 0x7f0014198020 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f0008009230 tx=0x7f0008005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:47.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.587+0000 7f0010ff9640 1 -- 172.21.15.16:0/3080094703 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0008041070 con 0x7f0014102980 2024-03-28T22:09:47.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.587+0000 7f0019c5c640 1 -- 172.21.15.16:0/3080094703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f001419f8f0 con 0x7f0014102980 2024-03-28T22:09:47.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.587+0000 7f0010ff9640 1 -- 172.21.15.16:0/3080094703 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00080044d0 con 0x7f0014102980 2024-03-28T22:09:47.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.587+0000 7f0010ff9640 1 -- 172.21.15.16:0/3080094703 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0008034440 con 0x7f0014102980 2024-03-28T22:09:47.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.587+0000 7f0019c5c640 1 -- 172.21.15.16:0/3080094703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f001419fe30 con 0x7f0014102980 2024-03-28T22:09:47.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.588+0000 7f0019c5c640 1 -- 172.21.15.16:0/3080094703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0014108ab0 con 0x7f0014102980 2024-03-28T22:09:47.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.590+0000 7f0010ff9640 1 -- 172.21.15.16:0/3080094703 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f0008044070 con 0x7f0014102980 2024-03-28T22:09:47.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.590+0000 7f0010ff9640 1 --2- 172.21.15.16:0/3080094703 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7efffc0a1e10 0x7efffc0a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:47.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.590+0000 7f00137fe640 1 --2- 172.21.15.16:0/3080094703 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7efffc0a1e10 0x7efffc0a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:47.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.590+0000 7f0010ff9640 1 -- 172.21.15.16:0/3080094703 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f00080efcf0 con 0x7f0014102980 2024-03-28T22:09:47.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.591+0000 7f00137fe640 1 --2- 172.21.15.16:0/3080094703 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7efffc0a1e10 0x7efffc0a42d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0004005fd0 tx=0x7f000400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:47.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.594+0000 7f0010ff9640 1 -- 172.21.15.16:0/3080094703 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f00080bcab0 con 0x7f0014102980 2024-03-28T22:09:47.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.777+0000 7f0019c5c640 1 -- 172.21.15.16:0/3080094703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f001402d370 con 0x7f0014102980 2024-03-28T22:09:47.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.779+0000 7f0010ff9640 1 -- 172.21.15.16:0/3080094703 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22441 (secure 0 0 0) 0x7f00080c0b50 con 0x7f0014102980 2024-03-28T22:09:47.781 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:09:47.781 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":88,"fsid":"798d9884-ed4e-11ee-95d0-87774f69a715","created":"2024-03-28T22:00:27.216099+0000","modified":"2024-03-28T22:09:30.047847+0000","last_up_change":"2024-03-28T22:07:40.251141+0000","last_in_change":"2024-03-28T22:07:19.456451+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:03:59.327832+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:08:28.123529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:08:28.919037+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"46e7c8f8-ecc8-43d6-ae53-0655728f3240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6803","nonce":1589692483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6805","nonce":1589692483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6809","nonce":1589692483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6807","nonce":1589692483}]},"public_addr":"172.21.15.16:6803/1589692483","cluster_addr":"172.21.15.16:6805/1589692483","heartbeat_back_addr":"172.21.15.16:6809/1589692483","heartbeat_front_addr":"172.21.15.16:6807/1589692483","state":["exists","up"]},{"osd":1,"uuid":"af6d9c22-2830-4ce2-a547-011df0c6cbad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6801","nonce":795531234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6803","nonce":795531234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6807","nonce":795531234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6805","nonce":795531234}]},"public_addr":"172.21.15.53:6801/795531234","cluster_addr":"172.21.15.53:6803/795531234","heartbeat_back_addr":"172.21.15.53:6807/795531234","heartbeat_front_addr":"172.21.15.53:6805/795531234","state":["exists","up"]},{"osd":2,"uuid":"b10adca9-110c-4f87-85e0-b7542709d31a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6801","nonce":2214131086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6803","nonce":2214131086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6807","nonce":2214131086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6805","nonce":2214131086}]},"public_addr":"172.21.15.63:6801/2214131086","cluster_addr":"172.21.15.63:6803/2214131086","heartbeat_back_addr":"172.21.15.63:6807/2214131086","heartbeat_front_addr":"172.21.15.63:6805/2214131086","state":["exists","up"]},{"osd":3,"uuid":"c83cf4bb-395a-4786-abf7-74ca93a33eb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6811","nonce":13176789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6813","nonce":13176789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6817","nonce":13176789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6815","nonce":13176789}]},"public_addr":"172.21.15.16:6811/13176789","cluster_addr":"172.21.15.16:6813/13176789","heartbeat_back_addr":"172.21.15.16:6817/13176789","heartbeat_front_addr":"172.21.15.16:6815/13176789","state":["exists","up"]},{"osd":4,"uuid":"941b3e54-47d6-4e85-8069-e730c3fa8eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6809","nonce":772161849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6811","nonce":772161849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6815","nonce":772161849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6813","nonce":772161849}]},"public_addr":"172.21.15.53:6809/772161849","cluster_addr":"172.21.15.53:6811/772161849","heartbeat_back_addr":"172.21.15.53:6815/772161849","heartbeat_front_addr":"172.21.15.53:6813/772161849","state":["exists","up"]},{"osd":5,"uuid":"51d89826-1d60-40fc-8fd8-09bfe1ab4125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6809","nonce":1602554852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6811","nonce":1602554852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6815","nonce":1602554852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6813","nonce":1602554852}]},"public_addr":"172.21.15.63:6809/1602554852","cluster_addr":"172.21.15.63:6811/1602554852","heartbeat_back_addr":"172.21.15.63:6815/1602554852","heartbeat_front_addr":"172.21.15.63:6813/1602554852","state":["exists","up"]},{"osd":6,"uuid":"978efe5d-2c55-4ff8-8a1e-9f891388945e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6819","nonce":4290459224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6821","nonce":4290459224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6825","nonce":4290459224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6823","nonce":4290459224}]},"public_addr":"172.21.15.16:6819/4290459224","cluster_addr":"172.21.15.16:6821/4290459224","heartbeat_back_addr":"172.21.15.16:6825/4290459224","heartbeat_front_addr":"172.21.15.16:6823/4290459224","state":["exists","up"]},{"osd":7,"uuid":"50d489cb-dbb8-48df-b868-b29edf12e04d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6817","nonce":1056467525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6819","nonce":1056467525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6823","nonce":1056467525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6821","nonce":1056467525}]},"public_addr":"172.21.15.53:6817/1056467525","cluster_addr":"172.21.15.53:6819/1056467525","heartbeat_back_addr":"172.21.15.53:6823/1056467525","heartbeat_front_addr":"172.21.15.53:6821/1056467525","state":["exists","up"]},{"osd":8,"uuid":"49b39414-cc9c-4057-ae3f-cc109cd979da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6817","nonce":2918516794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6819","nonce":2918516794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6823","nonce":2918516794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6821","nonce":2918516794}]},"public_addr":"172.21.15.63:6817/2918516794","cluster_addr":"172.21.15.63:6819/2918516794","heartbeat_back_addr":"172.21.15.63:6823/2918516794","heartbeat_front_addr":"172.21.15.63:6821/2918516794","state":["exists","up"]},{"osd":9,"uuid":"4c873e51-b783-4eb6-83a3-fee31722d60b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6827","nonce":1585624367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6829","nonce":1585624367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6833","nonce":1585624367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6831","nonce":1585624367}]},"public_addr":"172.21.15.16:6827/1585624367","cluster_addr":"172.21.15.16:6829/1585624367","heartbeat_back_addr":"172.21.15.16:6833/1585624367","heartbeat_front_addr":"172.21.15.16:6831/1585624367","state":["exists","up"]},{"osd":10,"uuid":"085a78bf-e2ad-481a-85a3-71d1fc549a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6825","nonce":3357446484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6827","nonce":3357446484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6831","nonce":3357446484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6829","nonce":3357446484}]},"public_addr":"172.21.15.53:6825/3357446484","cluster_addr":"172.21.15.53:6827/3357446484","heartbeat_back_addr":"172.21.15.53:6831/3357446484","heartbeat_front_addr":"172.21.15.53:6829/3357446484","state":["exists","up"]},{"osd":11,"uuid":"13302aa3-08b9-4639-9042-27d850b4d5f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6825","nonce":2167866570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6827","nonce":2167866570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6831","nonce":2167866570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6829","nonce":2167866570}]},"public_addr":"172.21.15.63:6825/2167866570","cluster_addr":"172.21.15.63:6827/2167866570","heartbeat_back_addr":"172.21.15.63:6831/2167866570","heartbeat_front_addr":"172.21.15.63:6829/2167866570","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:07.450323+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:31.363172+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:55.295021+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:19.794760+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:44.063492+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:08.195064+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:33.655328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:57.639414+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:22.402394+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:47.462050+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:12.226307+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:37.495625+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3821408150":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6801/1832422779":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/996598130":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/1062264551":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/317557779":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6801/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/3631746518":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6800/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/3116652786":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6800/1832422779":"2024-03-29T22:01:28.993434+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:09:47.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.781+0000 7f0019c5c640 1 -- 172.21.15.16:0/3080094703 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7efffc0a1e10 msgr2=0x7efffc0a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:47.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.781+0000 7f0019c5c640 1 --2- 172.21.15.16:0/3080094703 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7efffc0a1e10 0x7efffc0a42d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0004005fd0 tx=0x7f000400b390 comp rx=0 tx=0).stop 2024-03-28T22:09:47.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.781+0000 7f0019c5c640 1 -- 172.21.15.16:0/3080094703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0014102980 msgr2=0x7f0014198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:47.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.781+0000 7f0019c5c640 1 --2- 172.21.15.16:0/3080094703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0014102980 0x7f0014198020 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f0008009230 tx=0x7f0008005d50 comp rx=0 tx=0).stop 2024-03-28T22:09:47.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.781+0000 7f0019c5c640 1 -- 172.21.15.16:0/3080094703 shutdown_connections 2024-03-28T22:09:47.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.781+0000 7f0019c5c640 1 --2- 172.21.15.16:0/3080094703 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7efffc0a1e10 0x7efffc0a42d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:47.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.781+0000 7f0019c5c640 1 --2- 172.21.15.16:0/3080094703 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0014103340 0x7f001419eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:47.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.781+0000 7f0019c5c640 1 --2- 172.21.15.16:0/3080094703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0014102980 0x7f0014198020 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:47.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.781+0000 7f0019c5c640 1 --2- 172.21.15.16:0/3080094703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0014101790 0x7f0014197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:47.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.781+0000 7f0019c5c640 1 -- 172.21.15.16:0/3080094703 >> 172.21.15.16:0/3080094703 conn(0x7f00140fcce0 msgr2=0x7f0014105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:47.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.781+0000 7f0019c5c640 1 -- 172.21.15.16:0/3080094703 shutdown_connections 2024-03-28T22:09:47.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.781+0000 7f0019c5c640 1 -- 172.21.15.16:0/3080094703 wait complete. 2024-03-28T22:09:47.794 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T22:03:59.327832+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T22:08:28.123529+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T22:08:28.919037+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T22:09:47.795 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T22:09:47.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.898+0000 7f1d1a23b640 1 -- 172.21.15.16:0/2463910269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d14102940 msgr2=0x7f1d14102dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:47.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.898+0000 7f1d1a23b640 1 --2- 172.21.15.16:0/2463910269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d14102940 0x7f1d14102dc0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f1d08009900 tx=0x7f1d0802bc30 comp rx=0 tx=0).stop 2024-03-28T22:09:47.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.899+0000 7f1d1a23b640 1 -- 172.21.15.16:0/2463910269 shutdown_connections 2024-03-28T22:09:47.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.899+0000 7f1d1a23b640 1 --2- 172.21.15.16:0/2463910269 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1d14103300 0x7f1d14109b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:47.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.899+0000 7f1d1a23b640 1 --2- 172.21.15.16:0/2463910269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d14102940 0x7f1d14102dc0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:47.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.899+0000 7f1d1a23b640 1 --2- 172.21.15.16:0/2463910269 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d14101750 0x7f1d14101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:47.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.899+0000 7f1d1a23b640 1 -- 172.21.15.16:0/2463910269 >> 172.21.15.16:0/2463910269 conn(0x7f1d140fcc80 msgr2=0x7f1d140ff100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:47.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.899+0000 7f1d1a23b640 1 -- 172.21.15.16:0/2463910269 shutdown_connections 2024-03-28T22:09:47.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.899+0000 7f1d1a23b640 1 -- 172.21.15.16:0/2463910269 wait complete. 2024-03-28T22:09:47.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.899+0000 7f1d1a23b640 1 Processor -- start 2024-03-28T22:09:47.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.899+0000 7f1d1a23b640 1 -- start start 2024-03-28T22:09:47.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.899+0000 7f1d1a23b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1d14101750 0x7f1d14197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:47.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.899+0000 7f1d1a23b640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d14102940 0x7f1d14197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:47.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.899+0000 7f1d1a23b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d14103300 0x7f1d1419efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:47.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.899+0000 7f1d1a23b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1d1410d3d0 con 0x7f1d14103300 2024-03-28T22:09:47.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.899+0000 7f1d1a23b640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f1d1410d250 con 0x7f1d14102940 2024-03-28T22:09:47.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.899+0000 7f1d1a23b640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1d1410d550 con 0x7f1d14101750 2024-03-28T22:09:47.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.900+0000 7f1d13fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d14103300 0x7f1d1419efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:47.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.900+0000 7f1d13fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d14103300 0x7f1d1419efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56796/0 (socket says 172.21.15.16:56796) 2024-03-28T22:09:47.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.900+0000 7f1d13fff640 1 -- 172.21.15.16:0/1827299054 learned_addr learned my addr 172.21.15.16:0/1827299054 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:47.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.900+0000 7f1d12ffd640 1 --2- 172.21.15.16:0/1827299054 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d14102940 0x7f1d14197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:47.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.900+0000 7f1d137fe640 1 --2- 172.21.15.16:0/1827299054 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1d14101750 0x7f1d14197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:47.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.900+0000 7f1d13fff640 1 -- 172.21.15.16:0/1827299054 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1d14101750 msgr2=0x7f1d14197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:47.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.900+0000 7f1d13fff640 1 --2- 172.21.15.16:0/1827299054 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1d14101750 0x7f1d14197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:47.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.900+0000 7f1d13fff640 1 -- 172.21.15.16:0/1827299054 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d14102940 msgr2=0x7f1d14197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:47.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.900+0000 7f1d13fff640 1 --2- 172.21.15.16:0/1827299054 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d14102940 0x7f1d14197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:47.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.900+0000 7f1d13fff640 1 -- 172.21.15.16:0/1827299054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d1419f650 con 0x7f1d14103300 2024-03-28T22:09:47.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.900+0000 7f1d13fff640 1 --2- 172.21.15.16:0/1827299054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d14103300 0x7f1d1419efe0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c00d750 tx=0x7f1d0c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:47.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.900+0000 7f1d10ff9640 1 -- 172.21.15.16:0/1827299054 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d0c017070 con 0x7f1d14103300 2024-03-28T22:09:47.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.900+0000 7f1d10ff9640 1 -- 172.21.15.16:0/1827299054 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d0c005500 con 0x7f1d14103300 2024-03-28T22:09:47.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.900+0000 7f1d10ff9640 1 -- 172.21.15.16:0/1827299054 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d0c004d90 con 0x7f1d14103300 2024-03-28T22:09:47.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.900+0000 7f1d1a23b640 1 -- 172.21.15.16:0/1827299054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d1419f940 con 0x7f1d14103300 2024-03-28T22:09:47.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.900+0000 7f1d1a23b640 1 -- 172.21.15.16:0/1827299054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d1419fe10 con 0x7f1d14103300 2024-03-28T22:09:47.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.901+0000 7f1d1a23b640 1 -- 172.21.15.16:0/1827299054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d14108af0 con 0x7f1d14103300 2024-03-28T22:09:47.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.902+0000 7f1d10ff9640 1 -- 172.21.15.16:0/1827299054 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f1d0c004f30 con 0x7f1d14103300 2024-03-28T22:09:47.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.903+0000 7f1d10ff9640 1 --2- 172.21.15.16:0/1827299054 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f1cfc0a1e10 0x7f1cfc0a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:47.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.903+0000 7f1d137fe640 1 --2- 172.21.15.16:0/1827299054 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f1cfc0a1e10 0x7f1cfc0a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:47.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.903+0000 7f1d10ff9640 1 -- 172.21.15.16:0/1827299054 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f1d0c0cd930 con 0x7f1d14103300 2024-03-28T22:09:47.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.903+0000 7f1d137fe640 1 --2- 172.21.15.16:0/1827299054 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f1cfc0a1e10 0x7f1cfc0a42d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1d141027a0 tx=0x7f1d04009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:47.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:47.907+0000 7f1d10ff9640 1 -- 172.21.15.16:0/1827299054 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1d0c09abd0 con 0x7f1d14103300 2024-03-28T22:09:48.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.090+0000 7f1d1a23b640 1 -- 172.21.15.16:0/1827299054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1d1402d390 con 0x7f1d14103300 2024-03-28T22:09:48.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.091+0000 7f1d10ff9640 1 -- 172.21.15.16:0/1827299054 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f1d0c09a4d0 con 0x7f1d14103300 2024-03-28T22:09:48.093 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-28T22:09:48.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.093+0000 7f1d1a23b640 1 -- 172.21.15.16:0/1827299054 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f1cfc0a1e10 msgr2=0x7f1cfc0a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:48.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.093+0000 7f1d1a23b640 1 --2- 172.21.15.16:0/1827299054 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f1cfc0a1e10 0x7f1cfc0a42d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1d141027a0 tx=0x7f1d04009500 comp rx=0 tx=0).stop 2024-03-28T22:09:48.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.093+0000 7f1d1a23b640 1 -- 172.21.15.16:0/1827299054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d14103300 msgr2=0x7f1d1419efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:48.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.093+0000 7f1d1a23b640 1 --2- 172.21.15.16:0/1827299054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d14103300 0x7f1d1419efe0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c00d750 tx=0x7f1d0c00dc10 comp rx=0 tx=0).stop 2024-03-28T22:09:48.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.093+0000 7f1d1a23b640 1 -- 172.21.15.16:0/1827299054 shutdown_connections 2024-03-28T22:09:48.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.093+0000 7f1d1a23b640 1 --2- 172.21.15.16:0/1827299054 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f1cfc0a1e10 0x7f1cfc0a42d0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:48.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.093+0000 7f1d1a23b640 1 --2- 172.21.15.16:0/1827299054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d14103300 0x7f1d1419efe0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:48.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.093+0000 7f1d1a23b640 1 --2- 172.21.15.16:0/1827299054 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d14102940 0x7f1d14197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:48.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.093+0000 7f1d1a23b640 1 --2- 172.21.15.16:0/1827299054 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1d14101750 0x7f1d14197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:48.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.093+0000 7f1d1a23b640 1 -- 172.21.15.16:0/1827299054 >> 172.21.15.16:0/1827299054 conn(0x7f1d140fcc80 msgr2=0x7f1d14105cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:48.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.093+0000 7f1d1a23b640 1 -- 172.21.15.16:0/1827299054 shutdown_connections 2024-03-28T22:09:48.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.093+0000 7f1d1a23b640 1 -- 172.21.15.16:0/1827299054 wait complete. 2024-03-28T22:09:48.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:47 smithi016 ceph-mon[32374]: pgmap v327: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T22:09:48.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:48 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3080094703' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:09:48.106 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T22:09:48.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.209+0000 7fc6f8919640 1 -- 172.21.15.16:0/409639091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6f41021f0 msgr2=0x7fc6f41045e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:48.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.209+0000 7fc6f8919640 1 --2- 172.21.15.16:0/409639091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6f41021f0 0x7fc6f41045e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec009900 tx=0x7fc6ec02bc30 comp rx=0 tx=0).stop 2024-03-28T22:09:48.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.210+0000 7fc6f8919640 1 -- 172.21.15.16:0/409639091 shutdown_connections 2024-03-28T22:09:48.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.210+0000 7fc6f8919640 1 --2- 172.21.15.16:0/409639091 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6f4107450 0x7fc6f41098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:48.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.210+0000 7fc6f8919640 1 --2- 172.21.15.16:0/409639091 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6f4104b20 0x7fc6f4106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:48.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.210+0000 7fc6f8919640 1 --2- 172.21.15.16:0/409639091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6f41021f0 0x7fc6f41045e0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:48.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.210+0000 7fc6f8919640 1 -- 172.21.15.16:0/409639091 >> 172.21.15.16:0/409639091 conn(0x7fc6f40fbbe0 msgr2=0x7fc6f40fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:48.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.210+0000 7fc6f8919640 1 -- 172.21.15.16:0/409639091 shutdown_connections 2024-03-28T22:09:48.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.210+0000 7fc6f8919640 1 -- 172.21.15.16:0/409639091 wait complete. 2024-03-28T22:09:48.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.210+0000 7fc6f8919640 1 Processor -- start 2024-03-28T22:09:48.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.210+0000 7fc6f8919640 1 -- start start 2024-03-28T22:09:48.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.210+0000 7fc6f8919640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6f41021f0 0x7fc6f4197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:48.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.210+0000 7fc6f8919640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6f4104b20 0x7fc6f4197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:48.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.210+0000 7fc6f8919640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6f4107450 0x7fc6f419f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:48.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.210+0000 7fc6f8919640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc6f410d410 con 0x7fc6f4107450 2024-03-28T22:09:48.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.210+0000 7fc6f8919640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc6f410d290 con 0x7fc6f41021f0 2024-03-28T22:09:48.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.211+0000 7fc6f8919640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc6f410d590 con 0x7fc6f4104b20 2024-03-28T22:09:48.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.211+0000 7fc6f2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6f4107450 0x7fc6f419f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:48.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.211+0000 7fc6f259c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6f41021f0 0x7fc6f4197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:48.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.211+0000 7fc6f2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6f4107450 0x7fc6f419f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56816/0 (socket says 172.21.15.16:56816) 2024-03-28T22:09:48.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.211+0000 7fc6f2d9d640 1 -- 172.21.15.16:0/3780131769 learned_addr learned my addr 172.21.15.16:0/3780131769 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:48.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.211+0000 7fc6f1d9b640 1 --2- 172.21.15.16:0/3780131769 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6f4104b20 0x7fc6f4197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:48.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.211+0000 7fc6f2d9d640 1 -- 172.21.15.16:0/3780131769 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6f4104b20 msgr2=0x7fc6f4197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:48.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.211+0000 7fc6f2d9d640 1 --2- 172.21.15.16:0/3780131769 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6f4104b20 0x7fc6f4197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:48.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.211+0000 7fc6f2d9d640 1 -- 172.21.15.16:0/3780131769 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6f41021f0 msgr2=0x7fc6f4197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:48.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.211+0000 7fc6f2d9d640 1 --2- 172.21.15.16:0/3780131769 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6f41021f0 0x7fc6f4197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:48.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.211+0000 7fc6f2d9d640 1 -- 172.21.15.16:0/3780131769 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6f419f6b0 con 0x7fc6f4107450 2024-03-28T22:09:48.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.211+0000 7fc6f2d9d640 1 --2- 172.21.15.16:0/3780131769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6f4107450 0x7fc6f419f040 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc6e400eaa0 tx=0x7fc6e400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:48.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.211+0000 7fc6e37fe640 1 -- 172.21.15.16:0/3780131769 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e400cdb0 con 0x7fc6f4107450 2024-03-28T22:09:48.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.211+0000 7fc6f8919640 1 -- 172.21.15.16:0/3780131769 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6f419f9a0 con 0x7fc6f4107450 2024-03-28T22:09:48.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.211+0000 7fc6e37fe640 1 -- 172.21.15.16:0/3780131769 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6e4005020 con 0x7fc6f4107450 2024-03-28T22:09:48.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.211+0000 7fc6e37fe640 1 -- 172.21.15.16:0/3780131769 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e40116d0 con 0x7fc6f4107450 2024-03-28T22:09:48.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.211+0000 7fc6f8919640 1 -- 172.21.15.16:0/3780131769 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6f419fef0 con 0x7fc6f4107450 2024-03-28T22:09:48.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.212+0000 7fc6f8919640 1 -- 172.21.15.16:0/3780131769 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6f40fd6d0 con 0x7fc6f4107450 2024-03-28T22:09:48.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.213+0000 7fc6e37fe640 1 -- 172.21.15.16:0/3780131769 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fc6e40054d0 con 0x7fc6f4107450 2024-03-28T22:09:48.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.214+0000 7fc6e37fe640 1 --2- 172.21.15.16:0/3780131769 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc6dc0a1e10 0x7fc6dc0a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:48.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.214+0000 7fc6f259c640 1 --2- 172.21.15.16:0/3780131769 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc6dc0a1e10 0x7fc6dc0a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:48.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.214+0000 7fc6e37fe640 1 -- 172.21.15.16:0/3780131769 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fc6e40ce070 con 0x7fc6f4107450 2024-03-28T22:09:48.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.214+0000 7fc6f259c640 1 --2- 172.21.15.16:0/3780131769 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc6dc0a1e10 0x7fc6dc0a42d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec02c7b0 tx=0x7fc6ec005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:48.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.218+0000 7fc6e37fe640 1 -- 172.21.15.16:0/3780131769 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc6e4016030 con 0x7fc6f4107450 2024-03-28T22:09:48.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:48 smithi053 ceph-mon[35979]: pgmap v327: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T22:09:48.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:48 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3080094703' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:09:48.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.399+0000 7fc6f8919640 1 -- 172.21.15.16:0/3780131769 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc6f402d3c0 con 0x7fc6f4107450 2024-03-28T22:09:48.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.400+0000 7fc6e37fe640 1 -- 172.21.15.16:0/3780131769 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7fc6e409ad20 con 0x7fc6f4107450 2024-03-28T22:09:48.402 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-28T22:09:48.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.402+0000 7fc6f8919640 1 -- 172.21.15.16:0/3780131769 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc6dc0a1e10 msgr2=0x7fc6dc0a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:48.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.402+0000 7fc6f8919640 1 --2- 172.21.15.16:0/3780131769 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc6dc0a1e10 0x7fc6dc0a42d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec02c7b0 tx=0x7fc6ec005fd0 comp rx=0 tx=0).stop 2024-03-28T22:09:48.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.402+0000 7fc6f8919640 1 -- 172.21.15.16:0/3780131769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6f4107450 msgr2=0x7fc6f419f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:48.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.402+0000 7fc6f8919640 1 --2- 172.21.15.16:0/3780131769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6f4107450 0x7fc6f419f040 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc6e400eaa0 tx=0x7fc6e400ef60 comp rx=0 tx=0).stop 2024-03-28T22:09:48.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.402+0000 7fc6f8919640 1 -- 172.21.15.16:0/3780131769 shutdown_connections 2024-03-28T22:09:48.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.402+0000 7fc6f8919640 1 --2- 172.21.15.16:0/3780131769 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc6dc0a1e10 0x7fc6dc0a42d0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:48.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.402+0000 7fc6f8919640 1 --2- 172.21.15.16:0/3780131769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6f4107450 0x7fc6f419f040 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:48.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.402+0000 7fc6f8919640 1 --2- 172.21.15.16:0/3780131769 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6f4104b20 0x7fc6f4197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:48.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.402+0000 7fc6f8919640 1 --2- 172.21.15.16:0/3780131769 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6f41021f0 0x7fc6f4197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:48.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.402+0000 7fc6f8919640 1 -- 172.21.15.16:0/3780131769 >> 172.21.15.16:0/3780131769 conn(0x7fc6f40fbbe0 msgr2=0x7fc6f4102c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:48.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.402+0000 7fc6f8919640 1 -- 172.21.15.16:0/3780131769 shutdown_connections 2024-03-28T22:09:48.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.402+0000 7fc6f8919640 1 -- 172.21.15.16:0/3780131769 wait complete. 2024-03-28T22:09:48.415 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T22:09:48.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:48 smithi063 ceph-mon[36249]: pgmap v327: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T22:09:48.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:48 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/3080094703' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:09:48.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.518+0000 7fdfb656c640 1 -- 172.21.15.16:0/1092910856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb0100650 msgr2=0x7fdfb0100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:48.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.518+0000 7fdfb656c640 1 --2- 172.21.15.16:0/1092910856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb0100650 0x7fdfb0100a70 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fdfa0009900 tx=0x7fdfa002bc30 comp rx=0 tx=0).stop 2024-03-28T22:09:48.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.519+0000 7fdfb656c640 1 -- 172.21.15.16:0/1092910856 shutdown_connections 2024-03-28T22:09:48.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.519+0000 7fdfb656c640 1 --2- 172.21.15.16:0/1092910856 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfb0102200 0x7fdfb0108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:48.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.519+0000 7fdfb656c640 1 --2- 172.21.15.16:0/1092910856 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdfb0101840 0x7fdfb0101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:48.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.519+0000 7fdfb656c640 1 --2- 172.21.15.16:0/1092910856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb0100650 0x7fdfb0100a70 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:48.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.519+0000 7fdfb656c640 1 -- 172.21.15.16:0/1092910856 >> 172.21.15.16:0/1092910856 conn(0x7fdfb00fbb80 msgr2=0x7fdfb00fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:48.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.519+0000 7fdfb656c640 1 -- 172.21.15.16:0/1092910856 shutdown_connections 2024-03-28T22:09:48.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.519+0000 7fdfb656c640 1 -- 172.21.15.16:0/1092910856 wait complete. 2024-03-28T22:09:48.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.519+0000 7fdfb656c640 1 Processor -- start 2024-03-28T22:09:48.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.519+0000 7fdfb656c640 1 -- start start 2024-03-28T22:09:48.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.519+0000 7fdfb656c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb0100650 0x7fdfb0197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:48.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.519+0000 7fdfb656c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdfb0101840 0x7fdfb0198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:48.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.519+0000 7fdfb656c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfb0102200 0x7fdfb019f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:48.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.519+0000 7fdfb656c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb010c330 con 0x7fdfb0100650 2024-03-28T22:09:48.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.519+0000 7fdfb656c640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb010c1b0 con 0x7fdfb0102200 2024-03-28T22:09:48.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.519+0000 7fdfb656c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb010c4b0 con 0x7fdfb0101840 2024-03-28T22:09:48.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.520+0000 7fdfaffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb0100650 0x7fdfb0197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:48.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.520+0000 7fdfaffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb0100650 0x7fdfb0197b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56830/0 (socket says 172.21.15.16:56830) 2024-03-28T22:09:48.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.520+0000 7fdfaffff640 1 -- 172.21.15.16:0/2720749165 learned_addr learned my addr 172.21.15.16:0/2720749165 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:48.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.520+0000 7fdfb4b09640 1 --2- 172.21.15.16:0/2720749165 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfb0102200 0x7fdfb019f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:48.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.520+0000 7fdfaf7fe640 1 --2- 172.21.15.16:0/2720749165 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdfb0101840 0x7fdfb0198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:48.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.520+0000 7fdfaffff640 1 -- 172.21.15.16:0/2720749165 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdfb0101840 msgr2=0x7fdfb0198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:48.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.520+0000 7fdfaffff640 1 --2- 172.21.15.16:0/2720749165 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdfb0101840 0x7fdfb0198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:48.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.520+0000 7fdfaffff640 1 -- 172.21.15.16:0/2720749165 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfb0102200 msgr2=0x7fdfb019f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:48.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.520+0000 7fdfaffff640 1 --2- 172.21.15.16:0/2720749165 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfb0102200 0x7fdfb019f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:48.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.520+0000 7fdfaffff640 1 -- 172.21.15.16:0/2720749165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfb019f6e0 con 0x7fdfb0100650 2024-03-28T22:09:48.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.520+0000 7fdfb4b09640 1 --2- 172.21.15.16:0/2720749165 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfb0102200 0x7fdfb019f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:09:48.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.520+0000 7fdfaffff640 1 --2- 172.21.15.16:0/2720749165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb0100650 0x7fdfb0197b30 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fdfa0000c00 tx=0x7fdfa0005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:48.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.520+0000 7fdfad7fa640 1 -- 172.21.15.16:0/2720749165 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa0041070 con 0x7fdfb0100650 2024-03-28T22:09:48.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.520+0000 7fdfad7fa640 1 -- 172.21.15.16:0/2720749165 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfa00044d0 con 0x7fdfb0100650 2024-03-28T22:09:48.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.520+0000 7fdfb656c640 1 -- 172.21.15.16:0/2720749165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfb019f970 con 0x7fdfb0100650 2024-03-28T22:09:48.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.520+0000 7fdfad7fa640 1 -- 172.21.15.16:0/2720749165 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa0034440 con 0x7fdfb0100650 2024-03-28T22:09:48.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.520+0000 7fdfb656c640 1 -- 172.21.15.16:0/2720749165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfb019fe60 con 0x7fdfb0100650 2024-03-28T22:09:48.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.521+0000 7fdfb656c640 1 -- 172.21.15.16:0/2720749165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfb01a0030 con 0x7fdfb0100650 2024-03-28T22:09:48.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.522+0000 7fdfad7fa640 1 -- 172.21.15.16:0/2720749165 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fdfa0044070 con 0x7fdfb0100650 2024-03-28T22:09:48.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.523+0000 7fdfad7fa640 1 --2- 172.21.15.16:0/2720749165 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdf980a1e10 0x7fdf980a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:48.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.523+0000 7fdfaf7fe640 1 --2- 172.21.15.16:0/2720749165 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdf980a1e10 0x7fdf980a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:48.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.523+0000 7fdfad7fa640 1 -- 172.21.15.16:0/2720749165 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fdfa00ef9c0 con 0x7fdfb0100650 2024-03-28T22:09:48.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.523+0000 7fdfaf7fe640 1 --2- 172.21.15.16:0/2720749165 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdf980a1e10 0x7fdf980a42d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdfb0198fd0 tx=0x7fdfa4009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:48.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.527+0000 7fdfad7fa640 1 -- 172.21.15.16:0/2720749165 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fdfa00bcd70 con 0x7fdfb0100650 2024-03-28T22:09:48.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.709+0000 7fdfb656c640 1 -- 172.21.15.16:0/2720749165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fdfb01a0200 con 0x7fdfb0100650 2024-03-28T22:09:48.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.710+0000 7fdfad7fa640 1 -- 172.21.15.16:0/2720749165 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7fdfa00bc670 con 0x7fdfb0100650 2024-03-28T22:09:48.712 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-28T22:09:48.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.712+0000 7fdfb656c640 1 -- 172.21.15.16:0/2720749165 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdf980a1e10 msgr2=0x7fdf980a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:48.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.712+0000 7fdfb656c640 1 --2- 172.21.15.16:0/2720749165 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdf980a1e10 0x7fdf980a42d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdfb0198fd0 tx=0x7fdfa4009110 comp rx=0 tx=0).stop 2024-03-28T22:09:48.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.712+0000 7fdfb656c640 1 -- 172.21.15.16:0/2720749165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb0100650 msgr2=0x7fdfb0197b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:48.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.712+0000 7fdfb656c640 1 --2- 172.21.15.16:0/2720749165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb0100650 0x7fdfb0197b30 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fdfa0000c00 tx=0x7fdfa0005d50 comp rx=0 tx=0).stop 2024-03-28T22:09:48.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.712+0000 7fdfb656c640 1 -- 172.21.15.16:0/2720749165 shutdown_connections 2024-03-28T22:09:48.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.712+0000 7fdfb656c640 1 --2- 172.21.15.16:0/2720749165 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdf980a1e10 0x7fdf980a42d0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:48.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.712+0000 7fdfb656c640 1 --2- 172.21.15.16:0/2720749165 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfb0102200 0x7fdfb019f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:48.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.712+0000 7fdfb656c640 1 --2- 172.21.15.16:0/2720749165 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdfb0101840 0x7fdfb0198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:48.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.712+0000 7fdfb656c640 1 --2- 172.21.15.16:0/2720749165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb0100650 0x7fdfb0197b30 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:48.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.712+0000 7fdfb656c640 1 -- 172.21.15.16:0/2720749165 >> 172.21.15.16:0/2720749165 conn(0x7fdfb00fbb80 msgr2=0x7fdfb00fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:48.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.712+0000 7fdfb656c640 1 -- 172.21.15.16:0/2720749165 shutdown_connections 2024-03-28T22:09:48.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.712+0000 7fdfb656c640 1 -- 172.21.15.16:0/2720749165 wait complete. 2024-03-28T22:09:48.725 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:09:48.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.825+0000 7f6f1bf65640 1 -- 172.21.15.16:0/110354027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f14103350 msgr2=0x7f6f14109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:48.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.825+0000 7f6f1bf65640 1 --2- 172.21.15.16:0/110354027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f14103350 0x7f6f14109be0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c009ca0 tx=0x7f6f0c02c0f0 comp rx=0 tx=0).stop 2024-03-28T22:09:48.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.825+0000 7f6f1bf65640 1 -- 172.21.15.16:0/110354027 shutdown_connections 2024-03-28T22:09:48.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.825+0000 7f6f1bf65640 1 --2- 172.21.15.16:0/110354027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f14103350 0x7f6f14109be0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:48.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.825+0000 7f6f1bf65640 1 --2- 172.21.15.16:0/110354027 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6f14102990 0x7f6f14102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:48.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.825+0000 7f6f1bf65640 1 --2- 172.21.15.16:0/110354027 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f141017a0 0x7f6f14101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:48.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.825+0000 7f6f1bf65640 1 -- 172.21.15.16:0/110354027 >> 172.21.15.16:0/110354027 conn(0x7f6f140fcd10 msgr2=0x7f6f140ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:48.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.825+0000 7f6f1bf65640 1 -- 172.21.15.16:0/110354027 shutdown_connections 2024-03-28T22:09:48.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.825+0000 7f6f1bf65640 1 -- 172.21.15.16:0/110354027 wait complete. 2024-03-28T22:09:48.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.826+0000 7f6f1bf65640 1 Processor -- start 2024-03-28T22:09:48.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.826+0000 7f6f1bf65640 1 -- start start 2024-03-28T22:09:48.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.826+0000 7f6f1bf65640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f141017a0 0x7f6f14197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:48.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.826+0000 7f6f1bf65640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6f14102990 0x7f6f14197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:48.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.826+0000 7f6f1bf65640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f14103350 0x7f6f1419efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:48.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.826+0000 7f6f1bf65640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6f1410d420 con 0x7f6f14103350 2024-03-28T22:09:48.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.826+0000 7f6f1bf65640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f6f1410d2a0 con 0x7f6f141017a0 2024-03-28T22:09:48.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.826+0000 7f6f1bf65640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6f1410d5a0 con 0x7f6f14102990 2024-03-28T22:09:48.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.826+0000 7f6f1a502640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f14103350 0x7f6f1419efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:48.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.826+0000 7f6f1a502640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f14103350 0x7f6f1419efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56846/0 (socket says 172.21.15.16:56846) 2024-03-28T22:09:48.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.826+0000 7f6f1a502640 1 -- 172.21.15.16:0/749101977 learned_addr learned my addr 172.21.15.16:0/749101977 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:48.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.826+0000 7f6f19d01640 1 --2- 172.21.15.16:0/749101977 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f141017a0 0x7f6f14197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:48.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.826+0000 7f6f19500640 1 --2- 172.21.15.16:0/749101977 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6f14102990 0x7f6f14197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:48.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.826+0000 7f6f1a502640 1 -- 172.21.15.16:0/749101977 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6f14102990 msgr2=0x7f6f14197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:48.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.826+0000 7f6f1a502640 1 --2- 172.21.15.16:0/749101977 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6f14102990 0x7f6f14197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:48.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.827+0000 7f6f1a502640 1 -- 172.21.15.16:0/749101977 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f141017a0 msgr2=0x7f6f14197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:48.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.827+0000 7f6f1a502640 1 --2- 172.21.15.16:0/749101977 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f141017a0 0x7f6f14197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:48.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.827+0000 7f6f1a502640 1 -- 172.21.15.16:0/749101977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f1419f650 con 0x7f6f14103350 2024-03-28T22:09:48.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.827+0000 7f6f1a502640 1 --2- 172.21.15.16:0/749101977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f14103350 0x7f6f1419efe0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c037040 tx=0x7f6f0c032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:48.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.827+0000 7f6f0affd640 1 -- 172.21.15.16:0/749101977 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c041020 con 0x7f6f14103350 2024-03-28T22:09:48.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.827+0000 7f6f0affd640 1 -- 172.21.15.16:0/749101977 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f0c034d90 con 0x7f6f14103350 2024-03-28T22:09:48.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.827+0000 7f6f1bf65640 1 -- 172.21.15.16:0/749101977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f1419f8e0 con 0x7f6f14103350 2024-03-28T22:09:48.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.827+0000 7f6f0affd640 1 -- 172.21.15.16:0/749101977 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c039040 con 0x7f6f14103350 2024-03-28T22:09:48.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.827+0000 7f6f1bf65640 1 -- 172.21.15.16:0/749101977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f1419fe20 con 0x7f6f14103350 2024-03-28T22:09:48.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.828+0000 7f6f1bf65640 1 -- 172.21.15.16:0/749101977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f14108a60 con 0x7f6f14103350 2024-03-28T22:09:48.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.829+0000 7f6f0affd640 1 -- 172.21.15.16:0/749101977 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f6f0c044020 con 0x7f6f14103350 2024-03-28T22:09:48.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.829+0000 7f6f0affd640 1 --2- 172.21.15.16:0/749101977 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f6f000a1e90 0x7f6f000a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:48.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.829+0000 7f6f19d01640 1 --2- 172.21.15.16:0/749101977 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f6f000a1e90 0x7f6f000a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:48.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.829+0000 7f6f0affd640 1 -- 172.21.15.16:0/749101977 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f6f0c0efc50 con 0x7f6f14103350 2024-03-28T22:09:48.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.830+0000 7f6f19d01640 1 --2- 172.21.15.16:0/749101977 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f6f000a1e90 0x7f6f000a4350 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6f141027f0 tx=0x7f6f10005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:48.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:48.834+0000 7f6f0affd640 1 -- 172.21.15.16:0/749101977 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6f0c04b020 con 0x7f6f14103350 2024-03-28T22:09:49.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.054+0000 7f6f1bf65640 1 -- 172.21.15.16:0/749101977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6f1402d340 con 0x7f6f14103350 2024-03-28T22:09:49.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.054+0000 7f6f0affd640 1 -- 172.21.15.16:0/749101977 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11851 (secure 0 0 0) 0x7f6f0c0bd060 con 0x7f6f14103350 2024-03-28T22:09:49.056 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:09:49.056 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/1644581227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1644581227},{"type":"v1","addr":"172.21.15.16:6839","nonce":1644581227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/2593062669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2593062669},{"type":"v1","addr":"172.21.15.16:6841","nonce":2593062669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:08:29.672976+0000","modified":"2024-03-28T22:08:34.341708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24566,"mds_1":24524,"mds_2":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24466":{"gid":24466,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/247249857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":247249857},{"type":"v1","addr":"172.21.15.63:6835","nonce":247249857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6837/2324231032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":2324231032},{"type":"v1","addr":"172.21.15.63:6837","nonce":2324231032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6839/1177197077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":1177197077},{"type":"v1","addr":"172.21.15.63:6839","nonce":1177197077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"e","rank":2,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.53:6835/350727634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":350727634},{"type":"v1","addr":"172.21.15.53:6835","nonce":350727634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/349877573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":349877573},{"type":"v1","addr":"172.21.15.53:6837","nonce":349877573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6839/3714153956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":3714153956},{"type":"v1","addr":"172.21.15.53:6839","nonce":3714153956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24566,"qdb_cluster":[24500,24566,24524]},"id":1}]} 2024-03-28T22:09:49.057 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 20 2024-03-28T22:09:49.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.057+0000 7f6f1bf65640 1 -- 172.21.15.16:0/749101977 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f6f000a1e90 msgr2=0x7f6f000a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:49.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.057+0000 7f6f1bf65640 1 --2- 172.21.15.16:0/749101977 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f6f000a1e90 0x7f6f000a4350 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6f141027f0 tx=0x7f6f10005f90 comp rx=0 tx=0).stop 2024-03-28T22:09:49.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.057+0000 7f6f1bf65640 1 -- 172.21.15.16:0/749101977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f14103350 msgr2=0x7f6f1419efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:49.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.057+0000 7f6f1bf65640 1 --2- 172.21.15.16:0/749101977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f14103350 0x7f6f1419efe0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c037040 tx=0x7f6f0c032f20 comp rx=0 tx=0).stop 2024-03-28T22:09:49.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.057+0000 7f6f1bf65640 1 -- 172.21.15.16:0/749101977 shutdown_connections 2024-03-28T22:09:49.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.057+0000 7f6f1bf65640 1 --2- 172.21.15.16:0/749101977 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f6f000a1e90 0x7f6f000a4350 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:49.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.057+0000 7f6f1bf65640 1 --2- 172.21.15.16:0/749101977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f14103350 0x7f6f1419efe0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:49.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.057+0000 7f6f1bf65640 1 --2- 172.21.15.16:0/749101977 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6f14102990 0x7f6f14197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:49.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.057+0000 7f6f1bf65640 1 --2- 172.21.15.16:0/749101977 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f141017a0 0x7f6f14197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:49.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.057+0000 7f6f1bf65640 1 -- 172.21.15.16:0/749101977 >> 172.21.15.16:0/749101977 conn(0x7f6f140fcd10 msgr2=0x7f6f14105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:49.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.057+0000 7f6f1bf65640 1 -- 172.21.15.16:0/749101977 shutdown_connections 2024-03-28T22:09:49.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.057+0000 7f6f1bf65640 1 -- 172.21.15.16:0/749101977 wait complete. 2024-03-28T22:09:49.070 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:09:49.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.171+0000 7f4cbe884640 1 -- 172.21.15.16:0/3459006531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cb8103390 msgr2=0x7f4cb8109c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:49.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.171+0000 7f4cbe884640 1 --2- 172.21.15.16:0/3459006531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cb8103390 0x7f4cb8109c20 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f4cb0009ca0 tx=0x7f4cb002c0f0 comp rx=0 tx=0).stop 2024-03-28T22:09:49.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.171+0000 7f4cbe884640 1 -- 172.21.15.16:0/3459006531 shutdown_connections 2024-03-28T22:09:49.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.171+0000 7f4cbe884640 1 --2- 172.21.15.16:0/3459006531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cb8103390 0x7f4cb8109c20 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:49.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.171+0000 7f4cbe884640 1 --2- 172.21.15.16:0/3459006531 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cb81029d0 0x7f4cb8102e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:49.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.171+0000 7f4cbe884640 1 --2- 172.21.15.16:0/3459006531 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4cb81017e0 0x7f4cb8101c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:49.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.171+0000 7f4cbe884640 1 -- 172.21.15.16:0/3459006531 >> 172.21.15.16:0/3459006531 conn(0x7f4cb80fcd70 msgr2=0x7f4cb80ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:49.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.171+0000 7f4cbe884640 1 -- 172.21.15.16:0/3459006531 shutdown_connections 2024-03-28T22:09:49.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.171+0000 7f4cbe884640 1 -- 172.21.15.16:0/3459006531 wait complete. 2024-03-28T22:09:49.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.172+0000 7f4cbe884640 1 Processor -- start 2024-03-28T22:09:49.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.172+0000 7f4cbe884640 1 -- start start 2024-03-28T22:09:49.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.172+0000 7f4cbe884640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cb81017e0 0x7f4cb8075500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:49.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.172+0000 7f4cbe884640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4cb81029d0 0x7f4cb8073b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:49.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.172+0000 7f4cbe884640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cb8103390 0x7f4cb8074080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:49.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.172+0000 7f4cbe884640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb810d3d0 con 0x7f4cb81017e0 2024-03-28T22:09:49.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.172+0000 7f4cbe884640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb810d250 con 0x7f4cb8103390 2024-03-28T22:09:49.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.172+0000 7f4cbe884640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb810d550 con 0x7f4cb81029d0 2024-03-28T22:09:49.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.172+0000 7f4cb7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cb81017e0 0x7f4cb8075500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:49.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.172+0000 7f4cb7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cb81017e0 0x7f4cb8075500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56876/0 (socket says 172.21.15.16:56876) 2024-03-28T22:09:49.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.172+0000 7f4cb7fff640 1 -- 172.21.15.16:0/683574167 learned_addr learned my addr 172.21.15.16:0/683574167 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:49.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.172+0000 7f4cbce21640 1 --2- 172.21.15.16:0/683574167 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cb8103390 0x7f4cb8074080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:49.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.172+0000 7f4cb77fe640 1 --2- 172.21.15.16:0/683574167 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4cb81029d0 0x7f4cb8073b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:49.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.172+0000 7f4cb7fff640 1 -- 172.21.15.16:0/683574167 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4cb81029d0 msgr2=0x7f4cb8073b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:49.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.173+0000 7f4cb7fff640 1 --2- 172.21.15.16:0/683574167 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4cb81029d0 0x7f4cb8073b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:49.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.173+0000 7f4cb7fff640 1 -- 172.21.15.16:0/683574167 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cb8103390 msgr2=0x7f4cb8074080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:49.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.173+0000 7f4cb7fff640 1 --2- 172.21.15.16:0/683574167 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cb8103390 0x7f4cb8074080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:49.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.173+0000 7f4cb7fff640 1 -- 172.21.15.16:0/683574167 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cb8074910 con 0x7f4cb81017e0 2024-03-28T22:09:49.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.173+0000 7f4cb7fff640 1 --2- 172.21.15.16:0/683574167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cb81017e0 0x7f4cb8075500 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4ca800d950 tx=0x7f4ca800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:49.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.173+0000 7f4cb57fa640 1 -- 172.21.15.16:0/683574167 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca8017020 con 0x7f4cb81017e0 2024-03-28T22:09:49.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.173+0000 7f4cb57fa640 1 -- 172.21.15.16:0/683574167 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ca8009c30 con 0x7f4cb81017e0 2024-03-28T22:09:49.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.173+0000 7f4cbe884640 1 -- 172.21.15.16:0/683574167 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cb81a1b50 con 0x7f4cb81017e0 2024-03-28T22:09:49.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.173+0000 7f4cb57fa640 1 -- 172.21.15.16:0/683574167 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca800f740 con 0x7f4cb81017e0 2024-03-28T22:09:49.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.173+0000 7f4cbe884640 1 -- 172.21.15.16:0/683574167 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cb81a20a0 con 0x7f4cb81017e0 2024-03-28T22:09:49.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.174+0000 7f4cbe884640 1 -- 172.21.15.16:0/683574167 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cb8108aa0 con 0x7f4cb81017e0 2024-03-28T22:09:49.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.175+0000 7f4cb57fa640 1 -- 172.21.15.16:0/683574167 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f4ca801a020 con 0x7f4cb81017e0 2024-03-28T22:09:49.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.176+0000 7f4cb57fa640 1 --2- 172.21.15.16:0/683574167 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4ca00a1e10 0x7f4ca00a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:49.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.176+0000 7f4cb77fe640 1 --2- 172.21.15.16:0/683574167 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4ca00a1e10 0x7f4ca00a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:49.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.176+0000 7f4cb57fa640 1 -- 172.21.15.16:0/683574167 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f4ca80cdc30 con 0x7f4cb81017e0 2024-03-28T22:09:49.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.176+0000 7f4cb77fe640 1 --2- 172.21.15.16:0/683574167 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4ca00a1e10 0x7f4ca00a42d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4cb8075270 tx=0x7f4cac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:49.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.180+0000 7f4cb57fa640 1 -- 172.21.15.16:0/683574167 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4ca8021070 con 0x7f4cb81017e0 2024-03-28T22:09:49.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:49 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/1827299054' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:09:49.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:49 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3780131769' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:09:49.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:49 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2720749165' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:09:49.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.365+0000 7f4cbe884640 1 -- 172.21.15.16:0/683574167 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4cb802d340 con 0x7f4cb81017e0 2024-03-28T22:09:49.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.366+0000 7f4cb57fa640 1 -- 172.21.15.16:0/683574167 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22441 (secure 0 0 0) 0x7f4ca8009de0 con 0x7f4cb81017e0 2024-03-28T22:09:49.368 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:09:49.369 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":88,"fsid":"798d9884-ed4e-11ee-95d0-87774f69a715","created":"2024-03-28T22:00:27.216099+0000","modified":"2024-03-28T22:09:30.047847+0000","last_up_change":"2024-03-28T22:07:40.251141+0000","last_in_change":"2024-03-28T22:07:19.456451+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:03:59.327832+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:08:28.123529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:08:28.919037+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"46e7c8f8-ecc8-43d6-ae53-0655728f3240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6803","nonce":1589692483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6805","nonce":1589692483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6809","nonce":1589692483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6807","nonce":1589692483}]},"public_addr":"172.21.15.16:6803/1589692483","cluster_addr":"172.21.15.16:6805/1589692483","heartbeat_back_addr":"172.21.15.16:6809/1589692483","heartbeat_front_addr":"172.21.15.16:6807/1589692483","state":["exists","up"]},{"osd":1,"uuid":"af6d9c22-2830-4ce2-a547-011df0c6cbad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6801","nonce":795531234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6803","nonce":795531234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6807","nonce":795531234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6805","nonce":795531234}]},"public_addr":"172.21.15.53:6801/795531234","cluster_addr":"172.21.15.53:6803/795531234","heartbeat_back_addr":"172.21.15.53:6807/795531234","heartbeat_front_addr":"172.21.15.53:6805/795531234","state":["exists","up"]},{"osd":2,"uuid":"b10adca9-110c-4f87-85e0-b7542709d31a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6801","nonce":2214131086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6803","nonce":2214131086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6807","nonce":2214131086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6805","nonce":2214131086}]},"public_addr":"172.21.15.63:6801/2214131086","cluster_addr":"172.21.15.63:6803/2214131086","heartbeat_back_addr":"172.21.15.63:6807/2214131086","heartbeat_front_addr":"172.21.15.63:6805/2214131086","state":["exists","up"]},{"osd":3,"uuid":"c83cf4bb-395a-4786-abf7-74ca93a33eb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6811","nonce":13176789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6813","nonce":13176789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6817","nonce":13176789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6815","nonce":13176789}]},"public_addr":"172.21.15.16:6811/13176789","cluster_addr":"172.21.15.16:6813/13176789","heartbeat_back_addr":"172.21.15.16:6817/13176789","heartbeat_front_addr":"172.21.15.16:6815/13176789","state":["exists","up"]},{"osd":4,"uuid":"941b3e54-47d6-4e85-8069-e730c3fa8eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6809","nonce":772161849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6811","nonce":772161849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6815","nonce":772161849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6813","nonce":772161849}]},"public_addr":"172.21.15.53:6809/772161849","cluster_addr":"172.21.15.53:6811/772161849","heartbeat_back_addr":"172.21.15.53:6815/772161849","heartbeat_front_addr":"172.21.15.53:6813/772161849","state":["exists","up"]},{"osd":5,"uuid":"51d89826-1d60-40fc-8fd8-09bfe1ab4125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6809","nonce":1602554852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6811","nonce":1602554852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6815","nonce":1602554852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6813","nonce":1602554852}]},"public_addr":"172.21.15.63:6809/1602554852","cluster_addr":"172.21.15.63:6811/1602554852","heartbeat_back_addr":"172.21.15.63:6815/1602554852","heartbeat_front_addr":"172.21.15.63:6813/1602554852","state":["exists","up"]},{"osd":6,"uuid":"978efe5d-2c55-4ff8-8a1e-9f891388945e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6819","nonce":4290459224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6821","nonce":4290459224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6825","nonce":4290459224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6823","nonce":4290459224}]},"public_addr":"172.21.15.16:6819/4290459224","cluster_addr":"172.21.15.16:6821/4290459224","heartbeat_back_addr":"172.21.15.16:6825/4290459224","heartbeat_front_addr":"172.21.15.16:6823/4290459224","state":["exists","up"]},{"osd":7,"uuid":"50d489cb-dbb8-48df-b868-b29edf12e04d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6817","nonce":1056467525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6819","nonce":1056467525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6823","nonce":1056467525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6821","nonce":1056467525}]},"public_addr":"172.21.15.53:6817/1056467525","cluster_addr":"172.21.15.53:6819/1056467525","heartbeat_back_addr":"172.21.15.53:6823/1056467525","heartbeat_front_addr":"172.21.15.53:6821/1056467525","state":["exists","up"]},{"osd":8,"uuid":"49b39414-cc9c-4057-ae3f-cc109cd979da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6817","nonce":2918516794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6819","nonce":2918516794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6823","nonce":2918516794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6821","nonce":2918516794}]},"public_addr":"172.21.15.63:6817/2918516794","cluster_addr":"172.21.15.63:6819/2918516794","heartbeat_back_addr":"172.21.15.63:6823/2918516794","heartbeat_front_addr":"172.21.15.63:6821/2918516794","state":["exists","up"]},{"osd":9,"uuid":"4c873e51-b783-4eb6-83a3-fee31722d60b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6827","nonce":1585624367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6829","nonce":1585624367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6833","nonce":1585624367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6831","nonce":1585624367}]},"public_addr":"172.21.15.16:6827/1585624367","cluster_addr":"172.21.15.16:6829/1585624367","heartbeat_back_addr":"172.21.15.16:6833/1585624367","heartbeat_front_addr":"172.21.15.16:6831/1585624367","state":["exists","up"]},{"osd":10,"uuid":"085a78bf-e2ad-481a-85a3-71d1fc549a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6825","nonce":3357446484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6827","nonce":3357446484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6831","nonce":3357446484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6829","nonce":3357446484}]},"public_addr":"172.21.15.53:6825/3357446484","cluster_addr":"172.21.15.53:6827/3357446484","heartbeat_back_addr":"172.21.15.53:6831/3357446484","heartbeat_front_addr":"172.21.15.53:6829/3357446484","state":["exists","up"]},{"osd":11,"uuid":"13302aa3-08b9-4639-9042-27d850b4d5f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6825","nonce":2167866570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6827","nonce":2167866570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6831","nonce":2167866570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6829","nonce":2167866570}]},"public_addr":"172.21.15.63:6825/2167866570","cluster_addr":"172.21.15.63:6827/2167866570","heartbeat_back_addr":"172.21.15.63:6831/2167866570","heartbeat_front_addr":"172.21.15.63:6829/2167866570","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:07.450323+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:31.363172+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:55.295021+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:19.794760+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:44.063492+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:08.195064+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:33.655328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:57.639414+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:22.402394+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:47.462050+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:12.226307+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:37.495625+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3821408150":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6801/1832422779":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/996598130":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/1062264551":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/317557779":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6801/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/3631746518":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6800/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/3116652786":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6800/1832422779":"2024-03-29T22:01:28.993434+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:09:49.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.368+0000 7f4cbe884640 1 -- 172.21.15.16:0/683574167 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4ca00a1e10 msgr2=0x7f4ca00a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:49.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.368+0000 7f4cbe884640 1 --2- 172.21.15.16:0/683574167 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4ca00a1e10 0x7f4ca00a42d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4cb8075270 tx=0x7f4cac005f90 comp rx=0 tx=0).stop 2024-03-28T22:09:49.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.368+0000 7f4cbe884640 1 -- 172.21.15.16:0/683574167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cb81017e0 msgr2=0x7f4cb8075500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:49.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.368+0000 7f4cbe884640 1 --2- 172.21.15.16:0/683574167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cb81017e0 0x7f4cb8075500 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4ca800d950 tx=0x7f4ca800de10 comp rx=0 tx=0).stop 2024-03-28T22:09:49.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.369+0000 7f4cbe884640 1 -- 172.21.15.16:0/683574167 shutdown_connections 2024-03-28T22:09:49.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.369+0000 7f4cbe884640 1 --2- 172.21.15.16:0/683574167 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4ca00a1e10 0x7f4ca00a42d0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:49.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.369+0000 7f4cbe884640 1 --2- 172.21.15.16:0/683574167 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cb8103390 0x7f4cb8074080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:49.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.369+0000 7f4cbe884640 1 --2- 172.21.15.16:0/683574167 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4cb81029d0 0x7f4cb8073b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:49.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.369+0000 7f4cbe884640 1 --2- 172.21.15.16:0/683574167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cb81017e0 0x7f4cb8075500 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:49.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.369+0000 7f4cbe884640 1 -- 172.21.15.16:0/683574167 >> 172.21.15.16:0/683574167 conn(0x7f4cb80fcd70 msgr2=0x7f4cb8105c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:49.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.369+0000 7f4cbe884640 1 -- 172.21.15.16:0/683574167 shutdown_connections 2024-03-28T22:09:49.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.369+0000 7f4cbe884640 1 -- 172.21.15.16:0/683574167 wait complete. 2024-03-28T22:09:49.380 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:49 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/1827299054' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:09:49.380 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:49 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3780131769' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:09:49.381 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:49 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2720749165' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:09:49.382 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T22:03:59.327832+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T22:08:28.123529+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T22:08:28.919037+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T22:09:49.382 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T22:09:49.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:49 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/1827299054' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:09:49.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:49 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/3780131769' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:09:49.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:49 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2720749165' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:09:49.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.487+0000 7f4b3e1a0640 1 -- 172.21.15.16:0/1821881900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b38103100 msgr2=0x7f4b38109990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:49.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.487+0000 7f4b3e1a0640 1 --2- 172.21.15.16:0/1821881900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b38103100 0x7f4b38109990 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f4b30009ca0 tx=0x7f4b3002c0f0 comp rx=0 tx=0).stop 2024-03-28T22:09:49.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.487+0000 7f4b3e1a0640 1 -- 172.21.15.16:0/1821881900 shutdown_connections 2024-03-28T22:09:49.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.487+0000 7f4b3e1a0640 1 --2- 172.21.15.16:0/1821881900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b38103100 0x7f4b38109990 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:49.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.487+0000 7f4b3e1a0640 1 --2- 172.21.15.16:0/1821881900 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b38102740 0x7f4b38102bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:49.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.487+0000 7f4b3e1a0640 1 --2- 172.21.15.16:0/1821881900 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b38101550 0x7f4b38101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:49.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.487+0000 7f4b3e1a0640 1 -- 172.21.15.16:0/1821881900 >> 172.21.15.16:0/1821881900 conn(0x7f4b380fcae0 msgr2=0x7f4b380fef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:49.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.487+0000 7f4b3e1a0640 1 -- 172.21.15.16:0/1821881900 shutdown_connections 2024-03-28T22:09:49.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.487+0000 7f4b3e1a0640 1 -- 172.21.15.16:0/1821881900 wait complete. 2024-03-28T22:09:49.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.488+0000 7f4b3e1a0640 1 Processor -- start 2024-03-28T22:09:49.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.488+0000 7f4b3e1a0640 1 -- start start 2024-03-28T22:09:49.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.488+0000 7f4b3e1a0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b38101550 0x7f4b381977e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:49.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.488+0000 7f4b3e1a0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b38102740 0x7f4b38197d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:49.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.488+0000 7f4b3e1a0640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b38103100 0x7f4b3819ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:49.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.488+0000 7f4b3e1a0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4b3810d1e0 con 0x7f4b38102740 2024-03-28T22:09:49.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.488+0000 7f4b3e1a0640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4b3810d060 con 0x7f4b38103100 2024-03-28T22:09:49.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.488+0000 7f4b3e1a0640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4b3810d360 con 0x7f4b38101550 2024-03-28T22:09:49.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.488+0000 7f4b3c99d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b38102740 0x7f4b38197d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:49.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.488+0000 7f4b3c99d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b38102740 0x7f4b38197d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56886/0 (socket says 172.21.15.16:56886) 2024-03-28T22:09:49.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.488+0000 7f4b3c99d640 1 -- 172.21.15.16:0/1587292714 learned_addr learned my addr 172.21.15.16:0/1587292714 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:49.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.488+0000 7f4b3d99f640 1 --2- 172.21.15.16:0/1587292714 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b38103100 0x7f4b3819ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:49.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.488+0000 7f4b3d19e640 1 --2- 172.21.15.16:0/1587292714 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b38101550 0x7f4b381977e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:49.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.489+0000 7f4b3c99d640 1 -- 172.21.15.16:0/1587292714 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b38101550 msgr2=0x7f4b381977e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:49.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.489+0000 7f4b3c99d640 1 --2- 172.21.15.16:0/1587292714 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b38101550 0x7f4b381977e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:49.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.489+0000 7f4b3c99d640 1 -- 172.21.15.16:0/1587292714 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b38103100 msgr2=0x7f4b3819ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:49.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.489+0000 7f4b3c99d640 1 --2- 172.21.15.16:0/1587292714 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b38103100 0x7f4b3819ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:49.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.489+0000 7f4b3c99d640 1 -- 172.21.15.16:0/1587292714 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b3819f400 con 0x7f4b38102740 2024-03-28T22:09:49.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.489+0000 7f4b3c99d640 1 --2- 172.21.15.16:0/1587292714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b38102740 0x7f4b38197d20 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4b3400e9f0 tx=0x7f4b3400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:49.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.489+0000 7f4b2e7fc640 1 -- 172.21.15.16:0/1587292714 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b3400cd80 con 0x7f4b38102740 2024-03-28T22:09:49.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.489+0000 7f4b2e7fc640 1 -- 172.21.15.16:0/1587292714 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b34010910 con 0x7f4b38102740 2024-03-28T22:09:49.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.489+0000 7f4b3e1a0640 1 -- 172.21.15.16:0/1587292714 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b3819f6f0 con 0x7f4b38102740 2024-03-28T22:09:49.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.489+0000 7f4b2e7fc640 1 -- 172.21.15.16:0/1587292714 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b34018990 con 0x7f4b38102740 2024-03-28T22:09:49.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.489+0000 7f4b3e1a0640 1 -- 172.21.15.16:0/1587292714 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b3819fc40 con 0x7f4b38102740 2024-03-28T22:09:49.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.490+0000 7f4b3e1a0640 1 -- 172.21.15.16:0/1587292714 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b38108870 con 0x7f4b38102740 2024-03-28T22:09:49.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.491+0000 7f4b2e7fc640 1 -- 172.21.15.16:0/1587292714 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f4b34010ac0 con 0x7f4b38102740 2024-03-28T22:09:49.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.491+0000 7f4b2e7fc640 1 --2- 172.21.15.16:0/1587292714 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4b240a1e00 0x7f4b240a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:49.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.492+0000 7f4b3d19e640 1 --2- 172.21.15.16:0/1587292714 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4b240a1e00 0x7f4b240a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:49.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.492+0000 7f4b2e7fc640 1 -- 172.21.15.16:0/1587292714 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f4b340ce670 con 0x7f4b38102740 2024-03-28T22:09:49.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.492+0000 7f4b3d19e640 1 --2- 172.21.15.16:0/1587292714 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4b240a1e00 0x7f4b240a42c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4b28005fd0 tx=0x7f4b2800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:49.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.495+0000 7f4b2e7fc640 1 -- 172.21.15.16:0/1587292714 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4b3409b2c0 con 0x7f4b38102740 2024-03-28T22:09:49.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.677+0000 7f4b3e1a0640 1 -- 172.21.15.16:0/1587292714 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4b3802d300 con 0x7f4b38102740 2024-03-28T22:09:49.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.677+0000 7f4b2e7fc640 1 -- 172.21.15.16:0/1587292714 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f4b3409f360 con 0x7f4b38102740 2024-03-28T22:09:49.679 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-28T22:09:49.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.679+0000 7f4b3e1a0640 1 -- 172.21.15.16:0/1587292714 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4b240a1e00 msgr2=0x7f4b240a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:49.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.680+0000 7f4b3e1a0640 1 --2- 172.21.15.16:0/1587292714 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4b240a1e00 0x7f4b240a42c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4b28005fd0 tx=0x7f4b2800b390 comp rx=0 tx=0).stop 2024-03-28T22:09:49.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.680+0000 7f4b3e1a0640 1 -- 172.21.15.16:0/1587292714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b38102740 msgr2=0x7f4b38197d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:49.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.680+0000 7f4b3e1a0640 1 --2- 172.21.15.16:0/1587292714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b38102740 0x7f4b38197d20 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4b3400e9f0 tx=0x7f4b3400eeb0 comp rx=0 tx=0).stop 2024-03-28T22:09:49.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.680+0000 7f4b3e1a0640 1 -- 172.21.15.16:0/1587292714 shutdown_connections 2024-03-28T22:09:49.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.680+0000 7f4b3e1a0640 1 --2- 172.21.15.16:0/1587292714 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4b240a1e00 0x7f4b240a42c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:49.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.680+0000 7f4b3e1a0640 1 --2- 172.21.15.16:0/1587292714 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b38103100 0x7f4b3819ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:49.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.680+0000 7f4b3e1a0640 1 --2- 172.21.15.16:0/1587292714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b38102740 0x7f4b38197d20 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:49.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.680+0000 7f4b3e1a0640 1 --2- 172.21.15.16:0/1587292714 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b38101550 0x7f4b381977e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:49.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.680+0000 7f4b3e1a0640 1 -- 172.21.15.16:0/1587292714 >> 172.21.15.16:0/1587292714 conn(0x7f4b380fcae0 msgr2=0x7f4b38105a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:49.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.680+0000 7f4b3e1a0640 1 -- 172.21.15.16:0/1587292714 shutdown_connections 2024-03-28T22:09:49.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.680+0000 7f4b3e1a0640 1 -- 172.21.15.16:0/1587292714 wait complete. 2024-03-28T22:09:49.693 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T22:09:49.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.796+0000 7f566661c640 1 -- 172.21.15.16:0/151742172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5660101780 msgr2=0x7f5660101ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:49.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.796+0000 7f566661c640 1 --2- 172.21.15.16:0/151742172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5660101780 0x7f5660101ba0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5648009900 tx=0x7f564802bc30 comp rx=0 tx=0).stop 2024-03-28T22:09:49.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.796+0000 7f566661c640 1 -- 172.21.15.16:0/151742172 shutdown_connections 2024-03-28T22:09:49.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.796+0000 7f566661c640 1 --2- 172.21.15.16:0/151742172 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5660103330 0x7f5660109bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:49.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.796+0000 7f566661c640 1 --2- 172.21.15.16:0/151742172 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5660102970 0x7f5660102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:49.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.796+0000 7f566661c640 1 --2- 172.21.15.16:0/151742172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5660101780 0x7f5660101ba0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:49.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.796+0000 7f566661c640 1 -- 172.21.15.16:0/151742172 >> 172.21.15.16:0/151742172 conn(0x7f56600fcd10 msgr2=0x7f56600ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:49.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.796+0000 7f566661c640 1 -- 172.21.15.16:0/151742172 shutdown_connections 2024-03-28T22:09:49.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.796+0000 7f566661c640 1 -- 172.21.15.16:0/151742172 wait complete. 2024-03-28T22:09:49.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.797+0000 7f566661c640 1 Processor -- start 2024-03-28T22:09:49.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.797+0000 7f566661c640 1 -- start start 2024-03-28T22:09:49.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.797+0000 7f566661c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5660101780 0x7f5660197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:49.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.797+0000 7f566661c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5660102970 0x7f5660198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:49.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.797+0000 7f566661c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5660103330 0x7f566019f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:49.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.797+0000 7f566661c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f566010d420 con 0x7f5660101780 2024-03-28T22:09:49.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.797+0000 7f566661c640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f566010d2a0 con 0x7f5660103330 2024-03-28T22:09:49.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.797+0000 7f566661c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f566010d5a0 con 0x7f5660102970 2024-03-28T22:09:49.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.797+0000 7f565ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5660101780 0x7f5660197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:49.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.797+0000 7f565ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5660101780 0x7f5660197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56904/0 (socket says 172.21.15.16:56904) 2024-03-28T22:09:49.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.797+0000 7f565ffff640 1 -- 172.21.15.16:0/544222847 learned_addr learned my addr 172.21.15.16:0/544222847 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:49.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.797+0000 7f5664bb9640 1 --2- 172.21.15.16:0/544222847 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5660103330 0x7f566019f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:49.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.797+0000 7f565f7fe640 1 --2- 172.21.15.16:0/544222847 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5660102970 0x7f5660198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:49.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.797+0000 7f565ffff640 1 -- 172.21.15.16:0/544222847 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5660102970 msgr2=0x7f5660198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:49.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.797+0000 7f565ffff640 1 --2- 172.21.15.16:0/544222847 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5660102970 0x7f5660198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:49.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.797+0000 7f565ffff640 1 -- 172.21.15.16:0/544222847 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5660103330 msgr2=0x7f566019f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:49.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.797+0000 7f565ffff640 1 --2- 172.21.15.16:0/544222847 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5660103330 0x7f566019f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:49.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.797+0000 7f565ffff640 1 -- 172.21.15.16:0/544222847 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f566019f6e0 con 0x7f5660101780 2024-03-28T22:09:49.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.798+0000 7f565ffff640 1 --2- 172.21.15.16:0/544222847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5660101780 0x7f5660197ac0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5648000c00 tx=0x7f564802c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:49.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.798+0000 7f565d7fa640 1 -- 172.21.15.16:0/544222847 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5648043020 con 0x7f5660101780 2024-03-28T22:09:49.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.798+0000 7f565d7fa640 1 -- 172.21.15.16:0/544222847 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56480054e0 con 0x7f5660101780 2024-03-28T22:09:49.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.798+0000 7f566661c640 1 -- 172.21.15.16:0/544222847 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f566019f970 con 0x7f5660101780 2024-03-28T22:09:49.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.798+0000 7f565d7fa640 1 -- 172.21.15.16:0/544222847 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5648003ce0 con 0x7f5660101780 2024-03-28T22:09:49.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.798+0000 7f566661c640 1 -- 172.21.15.16:0/544222847 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f566019fe90 con 0x7f5660101780 2024-03-28T22:09:49.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.799+0000 7f566661c640 1 -- 172.21.15.16:0/544222847 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5660068a20 con 0x7f5660101780 2024-03-28T22:09:49.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.800+0000 7f565d7fa640 1 -- 172.21.15.16:0/544222847 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f5648046020 con 0x7f5660101780 2024-03-28T22:09:49.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.800+0000 7f565d7fa640 1 --2- 172.21.15.16:0/544222847 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f564c0a1e10 0x7f564c0a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:49.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.800+0000 7f565f7fe640 1 --2- 172.21.15.16:0/544222847 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f564c0a1e10 0x7f564c0a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:49.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.801+0000 7f565d7fa640 1 -- 172.21.15.16:0/544222847 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f56480eff70 con 0x7f5660101780 2024-03-28T22:09:49.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.801+0000 7f565f7fe640 1 --2- 172.21.15.16:0/544222847 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f564c0a1e10 0x7f564c0a42d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5660198fd0 tx=0x7f5650005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:49.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.805+0000 7f565d7fa640 1 -- 172.21.15.16:0/544222847 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f564804d020 con 0x7f5660101780 2024-03-28T22:09:49.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.989+0000 7f566661c640 1 -- 172.21.15.16:0/544222847 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f566002d170 con 0x7f5660101780 2024-03-28T22:09:49.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.989+0000 7f565d7fa640 1 -- 172.21.15.16:0/544222847 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f56480bd380 con 0x7f5660101780 2024-03-28T22:09:49.992 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-28T22:09:49.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.991+0000 7f566661c640 1 -- 172.21.15.16:0/544222847 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f564c0a1e10 msgr2=0x7f564c0a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:49.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.991+0000 7f566661c640 1 --2- 172.21.15.16:0/544222847 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f564c0a1e10 0x7f564c0a42d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5660198fd0 tx=0x7f5650005f90 comp rx=0 tx=0).stop 2024-03-28T22:09:49.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.991+0000 7f566661c640 1 -- 172.21.15.16:0/544222847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5660101780 msgr2=0x7f5660197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:49.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.991+0000 7f566661c640 1 --2- 172.21.15.16:0/544222847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5660101780 0x7f5660197ac0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5648000c00 tx=0x7f564802c6c0 comp rx=0 tx=0).stop 2024-03-28T22:09:49.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.991+0000 7f566661c640 1 -- 172.21.15.16:0/544222847 shutdown_connections 2024-03-28T22:09:49.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.991+0000 7f566661c640 1 --2- 172.21.15.16:0/544222847 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f564c0a1e10 0x7f564c0a42d0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:49.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.991+0000 7f566661c640 1 --2- 172.21.15.16:0/544222847 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5660103330 0x7f566019f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:49.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.991+0000 7f566661c640 1 --2- 172.21.15.16:0/544222847 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5660102970 0x7f5660198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:49.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.991+0000 7f566661c640 1 --2- 172.21.15.16:0/544222847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5660101780 0x7f5660197ac0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:49.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.991+0000 7f566661c640 1 -- 172.21.15.16:0/544222847 >> 172.21.15.16:0/544222847 conn(0x7f56600fcd10 msgr2=0x7f5660105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:49.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.992+0000 7f566661c640 1 -- 172.21.15.16:0/544222847 shutdown_connections 2024-03-28T22:09:49.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:49.992+0000 7f566661c640 1 -- 172.21.15.16:0/544222847 wait complete. 2024-03-28T22:09:50.004 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T22:09:50.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.108+0000 7fdd5e12c640 1 -- 172.21.15.16:0/2378694758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd58101880 msgr2=0x7fdd58101d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:50.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.108+0000 7fdd5e12c640 1 --2- 172.21.15.16:0/2378694758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd58101880 0x7fdd58101d00 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c009900 tx=0x7fdd4c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:09:50.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.108+0000 7fdd5e12c640 1 -- 172.21.15.16:0/2378694758 shutdown_connections 2024-03-28T22:09:50.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.108+0000 7fdd5e12c640 1 --2- 172.21.15.16:0/2378694758 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd58102240 0x7fdd58108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:50.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.108+0000 7fdd5e12c640 1 --2- 172.21.15.16:0/2378694758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd58101880 0x7fdd58101d00 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:50.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.108+0000 7fdd5e12c640 1 --2- 172.21.15.16:0/2378694758 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd58100690 0x7fdd58100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:50.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.108+0000 7fdd5e12c640 1 -- 172.21.15.16:0/2378694758 >> 172.21.15.16:0/2378694758 conn(0x7fdd580fbc20 msgr2=0x7fdd580fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:50.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.108+0000 7fdd5e12c640 1 -- 172.21.15.16:0/2378694758 shutdown_connections 2024-03-28T22:09:50.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.108+0000 7fdd5e12c640 1 -- 172.21.15.16:0/2378694758 wait complete. 2024-03-28T22:09:50.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.109+0000 7fdd5e12c640 1 Processor -- start 2024-03-28T22:09:50.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.109+0000 7fdd5e12c640 1 -- start start 2024-03-28T22:09:50.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.109+0000 7fdd5e12c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd58100690 0x7fdd58104690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:50.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.109+0000 7fdd5e12c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd58101880 0x7fdd58102cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:50.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.109+0000 7fdd5e12c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd58102240 0x7fdd58103210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:50.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.109+0000 7fdd5e12c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdd5810c360 con 0x7fdd58100690 2024-03-28T22:09:50.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.109+0000 7fdd5e12c640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fdd5810c1e0 con 0x7fdd58102240 2024-03-28T22:09:50.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.109+0000 7fdd5e12c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fdd5810c4e0 con 0x7fdd58101880 2024-03-28T22:09:50.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.109+0000 7fdd577fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd58100690 0x7fdd58104690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:50.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.109+0000 7fdd577fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd58100690 0x7fdd58104690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56924/0 (socket says 172.21.15.16:56924) 2024-03-28T22:09:50.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.109+0000 7fdd577fe640 1 -- 172.21.15.16:0/3867476192 learned_addr learned my addr 172.21.15.16:0/3867476192 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:50.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.109+0000 7fdd57fff640 1 --2- 172.21.15.16:0/3867476192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd58102240 0x7fdd58103210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:50.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.109+0000 7fdd56ffd640 1 --2- 172.21.15.16:0/3867476192 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd58101880 0x7fdd58102cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:50.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.109+0000 7fdd577fe640 1 -- 172.21.15.16:0/3867476192 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd58101880 msgr2=0x7fdd58102cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:50.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.109+0000 7fdd577fe640 1 --2- 172.21.15.16:0/3867476192 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd58101880 0x7fdd58102cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:50.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.109+0000 7fdd577fe640 1 -- 172.21.15.16:0/3867476192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd58102240 msgr2=0x7fdd58103210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:50.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.110+0000 7fdd577fe640 1 --2- 172.21.15.16:0/3867476192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd58102240 0x7fdd58103210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:50.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.110+0000 7fdd577fe640 1 -- 172.21.15.16:0/3867476192 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd58103aa0 con 0x7fdd58100690 2024-03-28T22:09:50.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.110+0000 7fdd57fff640 1 --2- 172.21.15.16:0/3867476192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd58102240 0x7fdd58103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:09:50.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.110+0000 7fdd577fe640 1 --2- 172.21.15.16:0/3867476192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd58100690 0x7fdd58104690 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fdd4800e9c0 tx=0x7fdd4800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:50.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.110+0000 7fdd54ff9640 1 -- 172.21.15.16:0/3867476192 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd4800cd80 con 0x7fdd58100690 2024-03-28T22:09:50.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.110+0000 7fdd5e12c640 1 -- 172.21.15.16:0/3867476192 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd581a1be0 con 0x7fdd58100690 2024-03-28T22:09:50.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.110+0000 7fdd54ff9640 1 -- 172.21.15.16:0/3867476192 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd480035f0 con 0x7fdd58100690 2024-03-28T22:09:50.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.110+0000 7fdd54ff9640 1 -- 172.21.15.16:0/3867476192 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd48010690 con 0x7fdd58100690 2024-03-28T22:09:50.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.110+0000 7fdd5e12c640 1 -- 172.21.15.16:0/3867476192 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd581a2100 con 0x7fdd58100690 2024-03-28T22:09:50.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.111+0000 7fdd5e12c640 1 -- 172.21.15.16:0/3867476192 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd58068a20 con 0x7fdd58100690 2024-03-28T22:09:50.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.112+0000 7fdd54ff9640 1 -- 172.21.15.16:0/3867476192 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fdd480037a0 con 0x7fdd58100690 2024-03-28T22:09:50.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.113+0000 7fdd54ff9640 1 --2- 172.21.15.16:0/3867476192 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdd400a1e10 0x7fdd400a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:50.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.113+0000 7fdd56ffd640 1 --2- 172.21.15.16:0/3867476192 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdd400a1e10 0x7fdd400a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:50.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.113+0000 7fdd54ff9640 1 -- 172.21.15.16:0/3867476192 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fdd480ce4d0 con 0x7fdd58100690 2024-03-28T22:09:50.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.113+0000 7fdd56ffd640 1 --2- 172.21.15.16:0/3867476192 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdd400a1e10 0x7fdd400a42d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c006010 tx=0x7fdd4c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:50.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.116+0000 7fdd54ff9640 1 -- 172.21.15.16:0/3867476192 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fdd4809b860 con 0x7fdd58100690 2024-03-28T22:09:50.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:50 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/749101977' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:09:50.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:50 smithi053 ceph-mon[35979]: pgmap v328: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:09:50.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:50 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/683574167' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:09:50.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:50 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/1587292714' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:09:50.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:50 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/544222847' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:09:50.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.298+0000 7fdd5e12c640 1 -- 172.21.15.16:0/3867476192 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fdd581a2590 con 0x7fdd58100690 2024-03-28T22:09:50.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.299+0000 7fdd54ff9640 1 -- 172.21.15.16:0/3867476192 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7fdd4809b160 con 0x7fdd58100690 2024-03-28T22:09:50.300 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-28T22:09:50.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.301+0000 7fdd5e12c640 1 -- 172.21.15.16:0/3867476192 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdd400a1e10 msgr2=0x7fdd400a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:50.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.301+0000 7fdd5e12c640 1 --2- 172.21.15.16:0/3867476192 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdd400a1e10 0x7fdd400a42d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c006010 tx=0x7fdd4c02df90 comp rx=0 tx=0).stop 2024-03-28T22:09:50.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.301+0000 7fdd5e12c640 1 -- 172.21.15.16:0/3867476192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd58100690 msgr2=0x7fdd58104690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:50.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.301+0000 7fdd5e12c640 1 --2- 172.21.15.16:0/3867476192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd58100690 0x7fdd58104690 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fdd4800e9c0 tx=0x7fdd4800ee80 comp rx=0 tx=0).stop 2024-03-28T22:09:50.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.301+0000 7fdd5e12c640 1 -- 172.21.15.16:0/3867476192 shutdown_connections 2024-03-28T22:09:50.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.301+0000 7fdd5e12c640 1 --2- 172.21.15.16:0/3867476192 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdd400a1e10 0x7fdd400a42d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:50.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.301+0000 7fdd5e12c640 1 --2- 172.21.15.16:0/3867476192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd58102240 0x7fdd58103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:50.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.301+0000 7fdd5e12c640 1 --2- 172.21.15.16:0/3867476192 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd58101880 0x7fdd58102cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:50.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.301+0000 7fdd5e12c640 1 --2- 172.21.15.16:0/3867476192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd58100690 0x7fdd58104690 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:50.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.301+0000 7fdd5e12c640 1 -- 172.21.15.16:0/3867476192 >> 172.21.15.16:0/3867476192 conn(0x7fdd580fbc20 msgr2=0x7fdd58069e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:50.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.301+0000 7fdd5e12c640 1 -- 172.21.15.16:0/3867476192 shutdown_connections 2024-03-28T22:09:50.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.301+0000 7fdd5e12c640 1 -- 172.21.15.16:0/3867476192 wait complete. 2024-03-28T22:09:50.312 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:50 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/749101977' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:09:50.313 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:50 smithi016 ceph-mon[32374]: pgmap v328: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:09:50.313 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:50 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/683574167' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:09:50.313 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:50 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/1587292714' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:09:50.313 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:50 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/544222847' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:09:50.314 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:09:50.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.415+0000 7f426f7c7640 1 -- 172.21.15.16:0/3133661104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4268103330 msgr2=0x7f4268109bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:50.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.415+0000 7f426f7c7640 1 --2- 172.21.15.16:0/3133661104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4268103330 0x7f4268109bc0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4264009900 tx=0x7f426402bc30 comp rx=0 tx=0).stop 2024-03-28T22:09:50.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.415+0000 7f426f7c7640 1 -- 172.21.15.16:0/3133661104 shutdown_connections 2024-03-28T22:09:50.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.415+0000 7f426f7c7640 1 --2- 172.21.15.16:0/3133661104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4268103330 0x7f4268109bc0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:50.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.415+0000 7f426f7c7640 1 --2- 172.21.15.16:0/3133661104 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4268102970 0x7f4268102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:50.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.415+0000 7f426f7c7640 1 --2- 172.21.15.16:0/3133661104 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4268101780 0x7f4268101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:50.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.415+0000 7f426f7c7640 1 -- 172.21.15.16:0/3133661104 >> 172.21.15.16:0/3133661104 conn(0x7f42680fcd30 msgr2=0x7f42680ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:50.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.415+0000 7f426f7c7640 1 -- 172.21.15.16:0/3133661104 shutdown_connections 2024-03-28T22:09:50.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.415+0000 7f426f7c7640 1 -- 172.21.15.16:0/3133661104 wait complete. 2024-03-28T22:09:50.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.416+0000 7f426f7c7640 1 Processor -- start 2024-03-28T22:09:50.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.416+0000 7f426f7c7640 1 -- start start 2024-03-28T22:09:50.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.416+0000 7f426f7c7640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4268101780 0x7f4268197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:50.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.416+0000 7f426f7c7640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4268102970 0x7f4268197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:50.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.416+0000 7f426f7c7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4268103330 0x7f426819eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:50.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.416+0000 7f426f7c7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f426810d400 con 0x7f4268103330 2024-03-28T22:09:50.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.416+0000 7f426f7c7640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f426810d280 con 0x7f4268101780 2024-03-28T22:09:50.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.416+0000 7f426f7c7640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f426810d580 con 0x7f4268102970 2024-03-28T22:09:50.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.416+0000 7f426dd64640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4268103330 0x7f426819eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:50.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.416+0000 7f426dd64640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4268103330 0x7f426819eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56946/0 (socket says 172.21.15.16:56946) 2024-03-28T22:09:50.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.416+0000 7f426dd64640 1 -- 172.21.15.16:0/103677021 learned_addr learned my addr 172.21.15.16:0/103677021 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:50.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.416+0000 7f426d563640 1 --2- 172.21.15.16:0/103677021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4268101780 0x7f4268197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:50.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.416+0000 7f426dd64640 1 -- 172.21.15.16:0/103677021 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4268102970 msgr2=0x7f4268197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:50.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.416+0000 7f426dd64640 1 --2- 172.21.15.16:0/103677021 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4268102970 0x7f4268197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:50.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.416+0000 7f426dd64640 1 -- 172.21.15.16:0/103677021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4268101780 msgr2=0x7f4268197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:50.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.416+0000 7f426dd64640 1 --2- 172.21.15.16:0/103677021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4268101780 0x7f4268197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:50.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.416+0000 7f426dd64640 1 -- 172.21.15.16:0/103677021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f426819f660 con 0x7f4268103330 2024-03-28T22:09:50.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.417+0000 7f426dd64640 1 --2- 172.21.15.16:0/103677021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4268103330 0x7f426819eff0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f426402c7b0 tx=0x7f4264005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:50.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.417+0000 7f425e7fc640 1 -- 172.21.15.16:0/103677021 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4264041070 con 0x7f4268103330 2024-03-28T22:09:50.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.417+0000 7f425e7fc640 1 -- 172.21.15.16:0/103677021 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42640054e0 con 0x7f4268103330 2024-03-28T22:09:50.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.417+0000 7f426f7c7640 1 -- 172.21.15.16:0/103677021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f426819f8f0 con 0x7f4268103330 2024-03-28T22:09:50.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.417+0000 7f425e7fc640 1 -- 172.21.15.16:0/103677021 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4264034440 con 0x7f4268103330 2024-03-28T22:09:50.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.417+0000 7f426f7c7640 1 -- 172.21.15.16:0/103677021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f426819fde0 con 0x7f4268103330 2024-03-28T22:09:50.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.418+0000 7f426f7c7640 1 -- 172.21.15.16:0/103677021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4268108a40 con 0x7f4268103330 2024-03-28T22:09:50.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.419+0000 7f425e7fc640 1 -- 172.21.15.16:0/103677021 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f4264044070 con 0x7f4268103330 2024-03-28T22:09:50.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.419+0000 7f425e7fc640 1 --2- 172.21.15.16:0/103677021 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f42540a1e10 0x7f42540a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:50.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.419+0000 7f425e7fc640 1 -- 172.21.15.16:0/103677021 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f42640efb10 con 0x7f4268103330 2024-03-28T22:09:50.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.419+0000 7f426d563640 1 --2- 172.21.15.16:0/103677021 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f42540a1e10 0x7f42540a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:50.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.420+0000 7f426d563640 1 --2- 172.21.15.16:0/103677021 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f42540a1e10 0x7f42540a42d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f42681027d0 tx=0x7f4260009680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:50.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.424+0000 7f425e7fc640 1 -- 172.21.15.16:0/103677021 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f42640bcf20 con 0x7f4268103330 2024-03-28T22:09:50.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:50 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/749101977' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:09:50.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:50 smithi063 ceph-mon[36249]: pgmap v328: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:09:50.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:50 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/683574167' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:09:50.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:50 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/1587292714' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:09:50.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:50 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/544222847' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:09:50.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.671+0000 7f426f7c7640 1 -- 172.21.15.16:0/103677021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4268062350 con 0x7f4268103330 2024-03-28T22:09:50.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.672+0000 7f425e7fc640 1 -- 172.21.15.16:0/103677021 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11851 (secure 0 0 0) 0x7f42640bc820 con 0x7f4268103330 2024-03-28T22:09:50.674 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:09:50.674 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/1644581227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1644581227},{"type":"v1","addr":"172.21.15.16:6839","nonce":1644581227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/2593062669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2593062669},{"type":"v1","addr":"172.21.15.16:6841","nonce":2593062669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:08:29.672976+0000","modified":"2024-03-28T22:08:34.341708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24566,"mds_1":24524,"mds_2":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24466":{"gid":24466,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/247249857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":247249857},{"type":"v1","addr":"172.21.15.63:6835","nonce":247249857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6837/2324231032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":2324231032},{"type":"v1","addr":"172.21.15.63:6837","nonce":2324231032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6839/1177197077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":1177197077},{"type":"v1","addr":"172.21.15.63:6839","nonce":1177197077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"e","rank":2,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.53:6835/350727634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":350727634},{"type":"v1","addr":"172.21.15.53:6835","nonce":350727634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/349877573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":349877573},{"type":"v1","addr":"172.21.15.53:6837","nonce":349877573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6839/3714153956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":3714153956},{"type":"v1","addr":"172.21.15.53:6839","nonce":3714153956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24566,"qdb_cluster":[24500,24566,24524]},"id":1}]} 2024-03-28T22:09:50.674 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 20 2024-03-28T22:09:50.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.674+0000 7f426f7c7640 1 -- 172.21.15.16:0/103677021 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f42540a1e10 msgr2=0x7f42540a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:50.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.674+0000 7f426f7c7640 1 --2- 172.21.15.16:0/103677021 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f42540a1e10 0x7f42540a42d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f42681027d0 tx=0x7f4260009680 comp rx=0 tx=0).stop 2024-03-28T22:09:50.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.674+0000 7f426f7c7640 1 -- 172.21.15.16:0/103677021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4268103330 msgr2=0x7f426819eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:50.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.674+0000 7f426f7c7640 1 --2- 172.21.15.16:0/103677021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4268103330 0x7f426819eff0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f426402c7b0 tx=0x7f4264005dd0 comp rx=0 tx=0).stop 2024-03-28T22:09:50.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.674+0000 7f426f7c7640 1 -- 172.21.15.16:0/103677021 shutdown_connections 2024-03-28T22:09:50.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.674+0000 7f426f7c7640 1 --2- 172.21.15.16:0/103677021 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f42540a1e10 0x7f42540a42d0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:50.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.674+0000 7f426f7c7640 1 --2- 172.21.15.16:0/103677021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4268103330 0x7f426819eff0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:50.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.674+0000 7f426f7c7640 1 --2- 172.21.15.16:0/103677021 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4268102970 0x7f4268197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:50.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.674+0000 7f426f7c7640 1 --2- 172.21.15.16:0/103677021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4268101780 0x7f4268197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:50.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.674+0000 7f426f7c7640 1 -- 172.21.15.16:0/103677021 >> 172.21.15.16:0/103677021 conn(0x7f42680fcd30 msgr2=0x7f4268105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:50.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.674+0000 7f426f7c7640 1 -- 172.21.15.16:0/103677021 shutdown_connections 2024-03-28T22:09:50.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.674+0000 7f426f7c7640 1 -- 172.21.15.16:0/103677021 wait complete. 2024-03-28T22:09:50.687 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:09:50.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.788+0000 7fddc31fd640 1 -- 172.21.15.16:0/3938711033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddbc100680 msgr2=0x7fddbc100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:50.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.788+0000 7fddc31fd640 1 --2- 172.21.15.16:0/3938711033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddbc100680 0x7fddbc100aa0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fddac009ca0 tx=0x7fddac02c0f0 comp rx=0 tx=0).stop 2024-03-28T22:09:50.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.788+0000 7fddc31fd640 1 -- 172.21.15.16:0/3938711033 shutdown_connections 2024-03-28T22:09:50.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.788+0000 7fddc31fd640 1 --2- 172.21.15.16:0/3938711033 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddbc102230 0x7fddbc108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:50.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.788+0000 7fddc31fd640 1 --2- 172.21.15.16:0/3938711033 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddbc101870 0x7fddbc101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:50.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.788+0000 7fddc31fd640 1 --2- 172.21.15.16:0/3938711033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddbc100680 0x7fddbc100aa0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:50.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.788+0000 7fddc31fd640 1 -- 172.21.15.16:0/3938711033 >> 172.21.15.16:0/3938711033 conn(0x7fddbc0fbbf0 msgr2=0x7fddbc0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:50.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.788+0000 7fddc31fd640 1 -- 172.21.15.16:0/3938711033 shutdown_connections 2024-03-28T22:09:50.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.789+0000 7fddc31fd640 1 -- 172.21.15.16:0/3938711033 wait complete. 2024-03-28T22:09:50.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.789+0000 7fddc31fd640 1 Processor -- start 2024-03-28T22:09:50.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.789+0000 7fddc31fd640 1 -- start start 2024-03-28T22:09:50.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.789+0000 7fddc31fd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddbc100680 0x7fddbc0695a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:50.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.789+0000 7fddc31fd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddbc101870 0x7fddbc19c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:50.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.789+0000 7fddc31fd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddbc102230 0x7fddbc19ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:50.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.789+0000 7fddc31fd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fddbc10c300 con 0x7fddbc102230 2024-03-28T22:09:50.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.789+0000 7fddc31fd640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fddbc10c180 con 0x7fddbc101870 2024-03-28T22:09:50.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.789+0000 7fddc31fd640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fddbc10c480 con 0x7fddbc100680 2024-03-28T22:09:50.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.789+0000 7fddc179a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddbc102230 0x7fddbc19ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:50.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.789+0000 7fddc179a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddbc102230 0x7fddbc19ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41512/0 (socket says 172.21.15.16:41512) 2024-03-28T22:09:50.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.789+0000 7fddbbfff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddbc101870 0x7fddbc19c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:50.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.789+0000 7fddc179a640 1 -- 172.21.15.16:0/1790700502 learned_addr learned my addr 172.21.15.16:0/1790700502 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:50.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.789+0000 7fddc0f99640 1 --2- 172.21.15.16:0/1790700502 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddbc100680 0x7fddbc0695a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:50.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.790+0000 7fddc179a640 1 -- 172.21.15.16:0/1790700502 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddbc100680 msgr2=0x7fddbc0695a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:50.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.790+0000 7fddc179a640 1 --2- 172.21.15.16:0/1790700502 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddbc100680 0x7fddbc0695a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:50.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.790+0000 7fddc179a640 1 -- 172.21.15.16:0/1790700502 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddbc101870 msgr2=0x7fddbc19c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:50.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.790+0000 7fddc179a640 1 --2- 172.21.15.16:0/1790700502 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddbc101870 0x7fddbc19c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:50.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.790+0000 7fddc179a640 1 -- 172.21.15.16:0/1790700502 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddbc19f6e0 con 0x7fddbc102230 2024-03-28T22:09:50.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.790+0000 7fddbbfff640 1 --2- 172.21.15.16:0/1790700502 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddbc101870 0x7fddbc19c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:09:50.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.790+0000 7fddc179a640 1 --2- 172.21.15.16:0/1790700502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddbc102230 0x7fddbc19ee20 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fddb400e9b0 tx=0x7fddb400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:50.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.790+0000 7fddb9ffb640 1 -- 172.21.15.16:0/1790700502 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb400cd80 con 0x7fddbc102230 2024-03-28T22:09:50.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.790+0000 7fddb9ffb640 1 -- 172.21.15.16:0/1790700502 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fddb40035f0 con 0x7fddbc102230 2024-03-28T22:09:50.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.790+0000 7fddb9ffb640 1 -- 172.21.15.16:0/1790700502 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb4010670 con 0x7fddbc102230 2024-03-28T22:09:50.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.790+0000 7fddc31fd640 1 -- 172.21.15.16:0/1790700502 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddbc19f9d0 con 0x7fddbc102230 2024-03-28T22:09:50.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.790+0000 7fddc31fd640 1 -- 172.21.15.16:0/1790700502 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddbc19ff20 con 0x7fddbc102230 2024-03-28T22:09:50.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.791+0000 7fddc31fd640 1 -- 172.21.15.16:0/1790700502 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddbc1079a0 con 0x7fddbc102230 2024-03-28T22:09:50.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.792+0000 7fddb9ffb640 1 -- 172.21.15.16:0/1790700502 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fddb40037a0 con 0x7fddbc102230 2024-03-28T22:09:50.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.793+0000 7fddb9ffb640 1 --2- 172.21.15.16:0/1790700502 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdda80a1e90 0x7fdda80a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:50.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.793+0000 7fddc0f99640 1 --2- 172.21.15.16:0/1790700502 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdda80a1e90 0x7fdda80a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:50.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.793+0000 7fddb9ffb640 1 -- 172.21.15.16:0/1790700502 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fddb40ce180 con 0x7fddbc102230 2024-03-28T22:09:50.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.793+0000 7fddc0f99640 1 --2- 172.21.15.16:0/1790700502 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdda80a1e90 0x7fdda80a4350 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fddac02c700 tx=0x7fddac032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:50.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.797+0000 7fddb9ffb640 1 -- 172.21.15.16:0/1790700502 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fddb409b480 con 0x7fddbc102230 2024-03-28T22:09:50.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.980+0000 7fddc31fd640 1 -- 172.21.15.16:0/1790700502 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fddbc02d380 con 0x7fddbc102230 2024-03-28T22:09:50.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.982+0000 7fddb9ffb640 1 -- 172.21.15.16:0/1790700502 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22441 (secure 0 0 0) 0x7fddb409ad80 con 0x7fddbc102230 2024-03-28T22:09:50.984 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:09:50.984 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":88,"fsid":"798d9884-ed4e-11ee-95d0-87774f69a715","created":"2024-03-28T22:00:27.216099+0000","modified":"2024-03-28T22:09:30.047847+0000","last_up_change":"2024-03-28T22:07:40.251141+0000","last_in_change":"2024-03-28T22:07:19.456451+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:03:59.327832+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:08:28.123529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:08:28.919037+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"46e7c8f8-ecc8-43d6-ae53-0655728f3240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6803","nonce":1589692483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6805","nonce":1589692483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6809","nonce":1589692483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6807","nonce":1589692483}]},"public_addr":"172.21.15.16:6803/1589692483","cluster_addr":"172.21.15.16:6805/1589692483","heartbeat_back_addr":"172.21.15.16:6809/1589692483","heartbeat_front_addr":"172.21.15.16:6807/1589692483","state":["exists","up"]},{"osd":1,"uuid":"af6d9c22-2830-4ce2-a547-011df0c6cbad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6801","nonce":795531234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6803","nonce":795531234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6807","nonce":795531234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6805","nonce":795531234}]},"public_addr":"172.21.15.53:6801/795531234","cluster_addr":"172.21.15.53:6803/795531234","heartbeat_back_addr":"172.21.15.53:6807/795531234","heartbeat_front_addr":"172.21.15.53:6805/795531234","state":["exists","up"]},{"osd":2,"uuid":"b10adca9-110c-4f87-85e0-b7542709d31a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6801","nonce":2214131086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6803","nonce":2214131086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6807","nonce":2214131086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6805","nonce":2214131086}]},"public_addr":"172.21.15.63:6801/2214131086","cluster_addr":"172.21.15.63:6803/2214131086","heartbeat_back_addr":"172.21.15.63:6807/2214131086","heartbeat_front_addr":"172.21.15.63:6805/2214131086","state":["exists","up"]},{"osd":3,"uuid":"c83cf4bb-395a-4786-abf7-74ca93a33eb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6811","nonce":13176789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6813","nonce":13176789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6817","nonce":13176789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6815","nonce":13176789}]},"public_addr":"172.21.15.16:6811/13176789","cluster_addr":"172.21.15.16:6813/13176789","heartbeat_back_addr":"172.21.15.16:6817/13176789","heartbeat_front_addr":"172.21.15.16:6815/13176789","state":["exists","up"]},{"osd":4,"uuid":"941b3e54-47d6-4e85-8069-e730c3fa8eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6809","nonce":772161849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6811","nonce":772161849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6815","nonce":772161849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6813","nonce":772161849}]},"public_addr":"172.21.15.53:6809/772161849","cluster_addr":"172.21.15.53:6811/772161849","heartbeat_back_addr":"172.21.15.53:6815/772161849","heartbeat_front_addr":"172.21.15.53:6813/772161849","state":["exists","up"]},{"osd":5,"uuid":"51d89826-1d60-40fc-8fd8-09bfe1ab4125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6809","nonce":1602554852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6811","nonce":1602554852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6815","nonce":1602554852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6813","nonce":1602554852}]},"public_addr":"172.21.15.63:6809/1602554852","cluster_addr":"172.21.15.63:6811/1602554852","heartbeat_back_addr":"172.21.15.63:6815/1602554852","heartbeat_front_addr":"172.21.15.63:6813/1602554852","state":["exists","up"]},{"osd":6,"uuid":"978efe5d-2c55-4ff8-8a1e-9f891388945e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6819","nonce":4290459224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6821","nonce":4290459224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6825","nonce":4290459224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6823","nonce":4290459224}]},"public_addr":"172.21.15.16:6819/4290459224","cluster_addr":"172.21.15.16:6821/4290459224","heartbeat_back_addr":"172.21.15.16:6825/4290459224","heartbeat_front_addr":"172.21.15.16:6823/4290459224","state":["exists","up"]},{"osd":7,"uuid":"50d489cb-dbb8-48df-b868-b29edf12e04d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6817","nonce":1056467525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6819","nonce":1056467525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6823","nonce":1056467525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6821","nonce":1056467525}]},"public_addr":"172.21.15.53:6817/1056467525","cluster_addr":"172.21.15.53:6819/1056467525","heartbeat_back_addr":"172.21.15.53:6823/1056467525","heartbeat_front_addr":"172.21.15.53:6821/1056467525","state":["exists","up"]},{"osd":8,"uuid":"49b39414-cc9c-4057-ae3f-cc109cd979da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6817","nonce":2918516794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6819","nonce":2918516794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6823","nonce":2918516794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6821","nonce":2918516794}]},"public_addr":"172.21.15.63:6817/2918516794","cluster_addr":"172.21.15.63:6819/2918516794","heartbeat_back_addr":"172.21.15.63:6823/2918516794","heartbeat_front_addr":"172.21.15.63:6821/2918516794","state":["exists","up"]},{"osd":9,"uuid":"4c873e51-b783-4eb6-83a3-fee31722d60b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6827","nonce":1585624367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6829","nonce":1585624367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6833","nonce":1585624367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6831","nonce":1585624367}]},"public_addr":"172.21.15.16:6827/1585624367","cluster_addr":"172.21.15.16:6829/1585624367","heartbeat_back_addr":"172.21.15.16:6833/1585624367","heartbeat_front_addr":"172.21.15.16:6831/1585624367","state":["exists","up"]},{"osd":10,"uuid":"085a78bf-e2ad-481a-85a3-71d1fc549a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6825","nonce":3357446484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6827","nonce":3357446484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6831","nonce":3357446484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6829","nonce":3357446484}]},"public_addr":"172.21.15.53:6825/3357446484","cluster_addr":"172.21.15.53:6827/3357446484","heartbeat_back_addr":"172.21.15.53:6831/3357446484","heartbeat_front_addr":"172.21.15.53:6829/3357446484","state":["exists","up"]},{"osd":11,"uuid":"13302aa3-08b9-4639-9042-27d850b4d5f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6825","nonce":2167866570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6827","nonce":2167866570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6831","nonce":2167866570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6829","nonce":2167866570}]},"public_addr":"172.21.15.63:6825/2167866570","cluster_addr":"172.21.15.63:6827/2167866570","heartbeat_back_addr":"172.21.15.63:6831/2167866570","heartbeat_front_addr":"172.21.15.63:6829/2167866570","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:07.450323+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:31.363172+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:55.295021+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:19.794760+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:44.063492+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:08.195064+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:33.655328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:57.639414+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:22.402394+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:47.462050+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:12.226307+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:37.495625+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3821408150":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6801/1832422779":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/996598130":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/1062264551":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/317557779":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6801/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/3631746518":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6800/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/3116652786":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6800/1832422779":"2024-03-29T22:01:28.993434+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:09:50.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.984+0000 7fddc31fd640 1 -- 172.21.15.16:0/1790700502 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdda80a1e90 msgr2=0x7fdda80a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:50.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.984+0000 7fddc31fd640 1 --2- 172.21.15.16:0/1790700502 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdda80a1e90 0x7fdda80a4350 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fddac02c700 tx=0x7fddac032590 comp rx=0 tx=0).stop 2024-03-28T22:09:50.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.984+0000 7fddc31fd640 1 -- 172.21.15.16:0/1790700502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddbc102230 msgr2=0x7fddbc19ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:50.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.984+0000 7fddc31fd640 1 --2- 172.21.15.16:0/1790700502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddbc102230 0x7fddbc19ee20 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fddb400e9b0 tx=0x7fddb400ee70 comp rx=0 tx=0).stop 2024-03-28T22:09:50.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.984+0000 7fddc31fd640 1 -- 172.21.15.16:0/1790700502 shutdown_connections 2024-03-28T22:09:50.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.984+0000 7fddc31fd640 1 --2- 172.21.15.16:0/1790700502 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdda80a1e90 0x7fdda80a4350 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:50.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.984+0000 7fddc31fd640 1 --2- 172.21.15.16:0/1790700502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddbc102230 0x7fddbc19ee20 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:50.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.984+0000 7fddc31fd640 1 --2- 172.21.15.16:0/1790700502 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddbc101870 0x7fddbc19c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:50.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.984+0000 7fddc31fd640 1 --2- 172.21.15.16:0/1790700502 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddbc100680 0x7fddbc0695a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:50.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.984+0000 7fddc31fd640 1 -- 172.21.15.16:0/1790700502 >> 172.21.15.16:0/1790700502 conn(0x7fddbc0fbbf0 msgr2=0x7fddbc104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:50.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.984+0000 7fddc31fd640 1 -- 172.21.15.16:0/1790700502 shutdown_connections 2024-03-28T22:09:50.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:50.984+0000 7fddc31fd640 1 -- 172.21.15.16:0/1790700502 wait complete. 2024-03-28T22:09:50.997 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:09:51.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.099+0000 7fef11bc9640 1 -- 172.21.15.16:0/624188712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef0c1006f0 msgr2=0x7fef0c100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:51.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.099+0000 7fef11bc9640 1 --2- 172.21.15.16:0/624188712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef0c1006f0 0x7fef0c100b10 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7feefc009900 tx=0x7feefc02bc30 comp rx=0 tx=0).stop 2024-03-28T22:09:51.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.099+0000 7fef11bc9640 1 -- 172.21.15.16:0/624188712 shutdown_connections 2024-03-28T22:09:51.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.099+0000 7fef11bc9640 1 --2- 172.21.15.16:0/624188712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef0c1022a0 0x7fef0c108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.099+0000 7fef11bc9640 1 --2- 172.21.15.16:0/624188712 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef0c1018e0 0x7fef0c101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.099+0000 7fef11bc9640 1 --2- 172.21.15.16:0/624188712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef0c1006f0 0x7fef0c100b10 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.099+0000 7fef11bc9640 1 -- 172.21.15.16:0/624188712 >> 172.21.15.16:0/624188712 conn(0x7fef0c0fbc80 msgr2=0x7fef0c0fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:51.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.099+0000 7fef11bc9640 1 -- 172.21.15.16:0/624188712 shutdown_connections 2024-03-28T22:09:51.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.099+0000 7fef11bc9640 1 -- 172.21.15.16:0/624188712 wait complete. 2024-03-28T22:09:51.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.099+0000 7fef11bc9640 1 Processor -- start 2024-03-28T22:09:51.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.099+0000 7fef11bc9640 1 -- start start 2024-03-28T22:09:51.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.100+0000 7fef11bc9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef0c1006f0 0x7fef0c196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.100+0000 7fef11bc9640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef0c1018e0 0x7fef0c196f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.100+0000 7fef11bc9640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef0c1022a0 0x7fef0c19dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.100+0000 7fef11bc9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fef0c10c300 con 0x7fef0c1006f0 2024-03-28T22:09:51.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.100+0000 7fef11bc9640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fef0c10c180 con 0x7fef0c1018e0 2024-03-28T22:09:51.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.100+0000 7fef11bc9640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fef0c10c480 con 0x7fef0c1022a0 2024-03-28T22:09:51.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.100+0000 7fef0b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef0c1006f0 0x7fef0c196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.100+0000 7fef0b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef0c1006f0 0x7fef0c196a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41530/0 (socket says 172.21.15.16:41530) 2024-03-28T22:09:51.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.100+0000 7fef0b7fe640 1 -- 172.21.15.16:0/1115610658 learned_addr learned my addr 172.21.15.16:0/1115610658 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:51.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.100+0000 7fef0affd640 1 --2- 172.21.15.16:0/1115610658 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef0c1018e0 0x7fef0c196f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.100+0000 7fef0bfff640 1 --2- 172.21.15.16:0/1115610658 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef0c1022a0 0x7fef0c19dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.100+0000 7fef0b7fe640 1 -- 172.21.15.16:0/1115610658 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef0c1022a0 msgr2=0x7fef0c19dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:51.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.100+0000 7fef0b7fe640 1 --2- 172.21.15.16:0/1115610658 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef0c1022a0 0x7fef0c19dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.100+0000 7fef0b7fe640 1 -- 172.21.15.16:0/1115610658 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef0c1018e0 msgr2=0x7fef0c196f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:51.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.100+0000 7fef0b7fe640 1 --2- 172.21.15.16:0/1115610658 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef0c1018e0 0x7fef0c196f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.100+0000 7fef0b7fe640 1 -- 172.21.15.16:0/1115610658 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef0c19e620 con 0x7fef0c1006f0 2024-03-28T22:09:51.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.100+0000 7fef0affd640 1 --2- 172.21.15.16:0/1115610658 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef0c1018e0 0x7fef0c196f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:09:51.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.100+0000 7fef0b7fe640 1 --2- 172.21.15.16:0/1115610658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef0c1006f0 0x7fef0c196a00 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7feefc005ab0 tx=0x7feefc005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:51.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.100+0000 7fef08ff9640 1 -- 172.21.15.16:0/1115610658 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feefc005000 con 0x7fef0c1006f0 2024-03-28T22:09:51.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.100+0000 7fef11bc9640 1 -- 172.21.15.16:0/1115610658 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef0c069e30 con 0x7fef0c1006f0 2024-03-28T22:09:51.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.100+0000 7fef08ff9640 1 -- 172.21.15.16:0/1115610658 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feefc0035f0 con 0x7fef0c1006f0 2024-03-28T22:09:51.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.100+0000 7fef08ff9640 1 -- 172.21.15.16:0/1115610658 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feefc03b6a0 con 0x7fef0c1006f0 2024-03-28T22:09:51.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.100+0000 7fef11bc9640 1 -- 172.21.15.16:0/1115610658 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef0c06a320 con 0x7fef0c1006f0 2024-03-28T22:09:51.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.101+0000 7fef11bc9640 1 -- 172.21.15.16:0/1115610658 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef0c068a20 con 0x7fef0c1006f0 2024-03-28T22:09:51.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.103+0000 7fef08ff9640 1 -- 172.21.15.16:0/1115610658 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7feefc003b30 con 0x7fef0c1006f0 2024-03-28T22:09:51.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.103+0000 7fef08ff9640 1 --2- 172.21.15.16:0/1115610658 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7feef40a1e10 0x7feef40a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.103+0000 7fef0affd640 1 --2- 172.21.15.16:0/1115610658 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7feef40a1e10 0x7feef40a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.103+0000 7fef08ff9640 1 -- 172.21.15.16:0/1115610658 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7feefc0f1e10 con 0x7fef0c1006f0 2024-03-28T22:09:51.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.103+0000 7fef0affd640 1 --2- 172.21.15.16:0/1115610658 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7feef40a1e10 0x7feef40a42d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fef000058c0 tx=0x7fef00005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:51.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.107+0000 7fef08ff9640 1 -- 172.21.15.16:0/1115610658 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7feefc0bf1c0 con 0x7fef0c1006f0 2024-03-28T22:09:51.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:51 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3867476192' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:09:51.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:51 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/103677021' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:09:51.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:51 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/1790700502' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:09:51.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.328+0000 7fef11bc9640 1 -- 172.21.15.16:0/1115610658 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fef0c107720 con 0x7fef0c1006f0 2024-03-28T22:09:51.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.329+0000 7fef08ff9640 1 -- 172.21.15.16:0/1115610658 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11851 (secure 0 0 0) 0x7feefc0beac0 con 0x7fef0c1006f0 2024-03-28T22:09:51.331 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:09:51.331 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/1644581227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1644581227},{"type":"v1","addr":"172.21.15.16:6839","nonce":1644581227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/2593062669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2593062669},{"type":"v1","addr":"172.21.15.16:6841","nonce":2593062669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:08:29.672976+0000","modified":"2024-03-28T22:08:34.341708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24566,"mds_1":24524,"mds_2":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24466":{"gid":24466,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/247249857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":247249857},{"type":"v1","addr":"172.21.15.63:6835","nonce":247249857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6837/2324231032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":2324231032},{"type":"v1","addr":"172.21.15.63:6837","nonce":2324231032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6839/1177197077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":1177197077},{"type":"v1","addr":"172.21.15.63:6839","nonce":1177197077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"e","rank":2,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.53:6835/350727634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":350727634},{"type":"v1","addr":"172.21.15.53:6835","nonce":350727634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/349877573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":349877573},{"type":"v1","addr":"172.21.15.53:6837","nonce":349877573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6839/3714153956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":3714153956},{"type":"v1","addr":"172.21.15.53:6839","nonce":3714153956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24566,"qdb_cluster":[24500,24566,24524]},"id":1}]} 2024-03-28T22:09:51.331 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 20 2024-03-28T22:09:51.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.331+0000 7fef11bc9640 1 -- 172.21.15.16:0/1115610658 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7feef40a1e10 msgr2=0x7feef40a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:51.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.331+0000 7fef11bc9640 1 --2- 172.21.15.16:0/1115610658 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7feef40a1e10 0x7feef40a42d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fef000058c0 tx=0x7fef00005850 comp rx=0 tx=0).stop 2024-03-28T22:09:51.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.331+0000 7fef11bc9640 1 -- 172.21.15.16:0/1115610658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef0c1006f0 msgr2=0x7fef0c196a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:51.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.331+0000 7fef11bc9640 1 --2- 172.21.15.16:0/1115610658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef0c1006f0 0x7fef0c196a00 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7feefc005ab0 tx=0x7feefc005dc0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.331+0000 7fef11bc9640 1 -- 172.21.15.16:0/1115610658 shutdown_connections 2024-03-28T22:09:51.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.331+0000 7fef11bc9640 1 --2- 172.21.15.16:0/1115610658 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7feef40a1e10 0x7feef40a42d0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.331+0000 7fef11bc9640 1 --2- 172.21.15.16:0/1115610658 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef0c1022a0 0x7fef0c19dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.331+0000 7fef11bc9640 1 --2- 172.21.15.16:0/1115610658 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef0c1018e0 0x7fef0c196f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.331+0000 7fef11bc9640 1 --2- 172.21.15.16:0/1115610658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef0c1006f0 0x7fef0c196a00 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.331+0000 7fef11bc9640 1 -- 172.21.15.16:0/1115610658 >> 172.21.15.16:0/1115610658 conn(0x7fef0c0fbc80 msgr2=0x7fef0c104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:51.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.331+0000 7fef11bc9640 1 -- 172.21.15.16:0/1115610658 shutdown_connections 2024-03-28T22:09:51.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.331+0000 7fef11bc9640 1 -- 172.21.15.16:0/1115610658 wait complete. 2024-03-28T22:09:51.343 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:51 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3867476192' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:09:51.343 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:51 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/103677021' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:09:51.343 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:51 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/1790700502' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:09:51.345 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-28T22:08:29.672976+0000', 'modified': '2024-03-28T22:08:34.341708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24566, 'mds_1': 24524, 'mds_2': 24500}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24466': {'gid': 24466, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.63:6835/247249857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.63:6834', 'nonce': 247249857}, {'type': 'v1', 'addr': '172.21.15.63:6835', 'nonce': 247249857}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24475': {'gid': 24475, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.63:6837/2324231032', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.63:6836', 'nonce': 2324231032}, {'type': 'v1', 'addr': '172.21.15.63:6837', 'nonce': 2324231032}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24493': {'gid': 24493, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.63:6839/1177197077', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.63:6838', 'nonce': 1177197077}, {'type': 'v1', 'addr': '172.21.15.63:6839', 'nonce': 1177197077}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24500': {'gid': 24500, 'name': 'e', 'rank': 2, 'incarnation': 18, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.53:6835/350727634', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6834', 'nonce': 350727634}, {'type': 'v1', 'addr': '172.21.15.53:6835', 'nonce': 350727634}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24524': {'gid': 24524, 'name': 'h', 'rank': 1, 'incarnation': 16, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.53:6837/349877573', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6836', 'nonce': 349877573}, {'type': 'v1', 'addr': '172.21.15.53:6837', 'nonce': 349877573}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24566': {'gid': 24566, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.53:6839/3714153956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6838', 'nonce': 3714153956}, {'type': 'v1', 'addr': '172.21.15.53:6839', 'nonce': 3714153956}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24566, 'qdb_cluster': [24500, 24566, 24524]} 2024-03-28T22:09:51.345 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-28T22:09:51.345 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2024-03-28T22:09:51.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:51 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/3867476192' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:09:51.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:51 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/103677021' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:09:51.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:51 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/1790700502' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:09:51.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.447+0000 7ff686bb5640 1 -- 172.21.15.16:0/3875345477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff680107450 msgr2=0x7ff6801098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:51.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.447+0000 7ff686bb5640 1 --2- 172.21.15.16:0/3875345477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff680107450 0x7ff6801098a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff678009ca0 tx=0x7ff67802c0f0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.447+0000 7ff686bb5640 1 -- 172.21.15.16:0/3875345477 shutdown_connections 2024-03-28T22:09:51.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.447+0000 7ff686bb5640 1 --2- 172.21.15.16:0/3875345477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff680107450 0x7ff6801098a0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.447+0000 7ff686bb5640 1 --2- 172.21.15.16:0/3875345477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff680104b20 0x7ff680106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.447+0000 7ff686bb5640 1 --2- 172.21.15.16:0/3875345477 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6801021f0 0x7ff6801045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.447+0000 7ff686bb5640 1 -- 172.21.15.16:0/3875345477 >> 172.21.15.16:0/3875345477 conn(0x7ff6800fbbe0 msgr2=0x7ff6800fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:51.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.447+0000 7ff686bb5640 1 -- 172.21.15.16:0/3875345477 shutdown_connections 2024-03-28T22:09:51.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.447+0000 7ff686bb5640 1 -- 172.21.15.16:0/3875345477 wait complete. 2024-03-28T22:09:51.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.448+0000 7ff686bb5640 1 Processor -- start 2024-03-28T22:09:51.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.448+0000 7ff686bb5640 1 -- start start 2024-03-28T22:09:51.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.448+0000 7ff686bb5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6801021f0 0x7ff680197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.448+0000 7ff686bb5640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff680104b20 0x7ff680197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.448+0000 7ff686bb5640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff680107450 0x7ff68019f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.448+0000 7ff686bb5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff68010d410 con 0x7ff6801021f0 2024-03-28T22:09:51.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.448+0000 7ff686bb5640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff68010d290 con 0x7ff680107450 2024-03-28T22:09:51.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.448+0000 7ff686bb5640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff68010d590 con 0x7ff680104b20 2024-03-28T22:09:51.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.448+0000 7ff684951640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6801021f0 0x7ff680197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.448+0000 7ff684951640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6801021f0 0x7ff680197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41544/0 (socket says 172.21.15.16:41544) 2024-03-28T22:09:51.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.448+0000 7ff684951640 1 -- 172.21.15.16:0/3647255881 learned_addr learned my addr 172.21.15.16:0/3647255881 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:51.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.448+0000 7ff685152640 1 --2- 172.21.15.16:0/3647255881 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff680107450 0x7ff68019f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.448+0000 7ff684951640 1 -- 172.21.15.16:0/3647255881 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff680104b20 msgr2=0x7ff680197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:51.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.448+0000 7ff684951640 1 --2- 172.21.15.16:0/3647255881 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff680104b20 0x7ff680197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.448+0000 7ff684951640 1 -- 172.21.15.16:0/3647255881 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff680107450 msgr2=0x7ff68019f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:51.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.448+0000 7ff684951640 1 --2- 172.21.15.16:0/3647255881 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff680107450 0x7ff68019f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.448+0000 7ff684951640 1 -- 172.21.15.16:0/3647255881 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff68019f680 con 0x7ff6801021f0 2024-03-28T22:09:51.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.448+0000 7ff684951640 1 --2- 172.21.15.16:0/3647255881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6801021f0 0x7ff680197a60 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff67400c970 tx=0x7ff67400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:51.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.448+0000 7ff67dffb640 1 -- 172.21.15.16:0/3647255881 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff674007bb0 con 0x7ff6801021f0 2024-03-28T22:09:51.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.448+0000 7ff67dffb640 1 -- 172.21.15.16:0/3647255881 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff674007d50 con 0x7ff6801021f0 2024-03-28T22:09:51.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.448+0000 7ff686bb5640 1 -- 172.21.15.16:0/3647255881 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff68019f970 con 0x7ff6801021f0 2024-03-28T22:09:51.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.448+0000 7ff67dffb640 1 -- 172.21.15.16:0/3647255881 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff67400f450 con 0x7ff6801021f0 2024-03-28T22:09:51.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.449+0000 7ff686bb5640 1 -- 172.21.15.16:0/3647255881 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff68019fe70 con 0x7ff6801021f0 2024-03-28T22:09:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.450+0000 7ff67dffb640 1 -- 172.21.15.16:0/3647255881 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7ff674005020 con 0x7ff6801021f0 2024-03-28T22:09:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.451+0000 7ff689798740 1 Processor -- start 2024-03-28T22:09:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.451+0000 7ff689798740 1 -- start start 2024-03-28T22:09:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.451+0000 7ff67dffb640 1 --2- 172.21.15.16:0/3647255881 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff6700a1e10 0x7ff6700a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.451+0000 7ff689798740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624782fd610 0x5624782fda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.451+0000 7ff689798740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624782fe7e0 0x5624782fec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.451+0000 7ff685152640 1 --2- 172.21.15.16:0/3647255881 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff6700a1e10 0x7ff6700a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.451+0000 7ff689798740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624782ff1a0 0x562478305a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.451+0000 7ff689798740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562478261af0 con 0x5624782ff1a0 2024-03-28T22:09:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.451+0000 7ff689798740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x56247820a9d0 con 0x5624782fe7e0 2024-03-28T22:09:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.451+0000 7ff689798740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56247825db90 con 0x5624782fd610 2024-03-28T22:09:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.452+0000 7ff67dffb640 1 -- 172.21.15.16:0/3647255881 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7ff6740cdd20 con 0x7ff6801021f0 2024-03-28T22:09:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.452+0000 7ff685152640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624782ff1a0 0x562478305a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.452+0000 7ff67ffff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624782fe7e0 0x5624782fec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.452+0000 7ff684951640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624782fd610 0x5624782fda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.452+0000 7ff67ffff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624782fe7e0 0x5624782fec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:44162/0 (socket says 172.21.15.16:44162) 2024-03-28T22:09:51.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.452+0000 7ff685152640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624782ff1a0 0x562478305a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41558/0 (socket says 172.21.15.16:41558) 2024-03-28T22:09:51.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.452+0000 7ff67ffff640 1 -- 172.21.15.16:0/172478416 learned_addr learned my addr 172.21.15.16:0/172478416 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:51.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.452+0000 7ff67ffff640 1 -- 172.21.15.16:0/172478416 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624782fd610 msgr2=0x5624782fda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:09:51.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.452+0000 7ff67ffff640 1 --2- 172.21.15.16:0/172478416 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624782fd610 0x5624782fda30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.452+0000 7ff67ffff640 1 -- 172.21.15.16:0/172478416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624782ff1a0 msgr2=0x562478305a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:09:51.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.452+0000 7ff67ffff640 1 --2- 172.21.15.16:0/172478416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624782ff1a0 0x562478305a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.452+0000 7ff67ffff640 1 -- 172.21.15.16:0/172478416 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562477f762d0 con 0x5624782fe7e0 2024-03-28T22:09:51.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.452+0000 7ff685152640 1 --2- 172.21.15.16:0/3647255881 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff6700a1e10 0x7ff6700a42d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff678005e30 tx=0x7ff67802cf00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:51.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.452+0000 7ff67ffff640 1 --2- 172.21.15.16:0/172478416 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624782fe7e0 0x5624782fec60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff66c009bb0 tx=0x7ff66c00e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3978efca7faaae49 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:51.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.452+0000 7ff66affd640 1 -- 172.21.15.16:0/172478416 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff66c005020 con 0x5624782fe7e0 2024-03-28T22:09:51.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.452+0000 7ff66affd640 1 -- 172.21.15.16:0/172478416 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff66c0047a0 con 0x5624782fe7e0 2024-03-28T22:09:51.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.452+0000 7ff66affd640 1 -- 172.21.15.16:0/172478416 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff66c010690 con 0x5624782fe7e0 2024-03-28T22:09:51.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.453+0000 7ff689798740 1 -- 172.21.15.16:0/172478416 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624782fe7e0 msgr2=0x5624782fec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:51.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.453+0000 7ff689798740 1 --2- 172.21.15.16:0/172478416 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624782fe7e0 0x5624782fec60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff66c009bb0 tx=0x7ff66c00e720 comp rx=0 tx=0).stop 2024-03-28T22:09:51.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.453+0000 7ff689798740 1 -- 172.21.15.16:0/172478416 shutdown_connections 2024-03-28T22:09:51.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.453+0000 7ff689798740 1 --2- 172.21.15.16:0/172478416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624782ff1a0 0x562478305a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.453+0000 7ff689798740 1 --2- 172.21.15.16:0/172478416 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624782fe7e0 0x5624782fec60 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.453+0000 7ff689798740 1 --2- 172.21.15.16:0/172478416 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624782fd610 0x5624782fda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.453+0000 7ff689798740 1 -- 172.21.15.16:0/172478416 >> 172.21.15.16:0/172478416 conn(0x5624782f8a00 msgr2=0x5624782f8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:51.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.453+0000 7ff689798740 1 -- 172.21.15.16:0/172478416 shutdown_connections 2024-03-28T22:09:51.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.453+0000 7ff689798740 1 -- 172.21.15.16:0/172478416 wait complete. 2024-03-28T22:09:51.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.453+0000 7ff689798740 1 -- start start 2024-03-28T22:09:51.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.453+0000 7ff689798740 1 Processor -- start 2024-03-28T22:09:51.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.454+0000 7ff689798740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624782fd610 0x5624783078a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.454+0000 7ff689798740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624782fe7e0 0x562478307de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.454+0000 7ff689798740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624782ff1a0 0x56247830ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.454+0000 7ff689798740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5624783061b0 con 0x5624782ff1a0 2024-03-28T22:09:51.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.454+0000 7ff689798740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x56247821df70 con 0x5624782fd610 2024-03-28T22:09:51.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.454+0000 7ff689798740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x562478306330 con 0x5624782fe7e0 2024-03-28T22:09:51.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.454+0000 7ff689798740 10 client.0 _ll_register_callbacks cb 0x562477f75ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:09:51.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.454+0000 7ff685152640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624782ff1a0 0x56247830ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.454+0000 7ff685152640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624782ff1a0 0x56247830ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41564/0 (socket says 172.21.15.16:41564) 2024-03-28T22:09:51.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.454+0000 7ff685152640 1 -- 172.21.15.16:0/2824676231 learned_addr learned my addr 172.21.15.16:0/2824676231 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:51.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.454+0000 7ff684951640 1 --2- 172.21.15.16:0/2824676231 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624782fd610 0x5624783078a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.454+0000 7ff67ffff640 1 --2- 172.21.15.16:0/2824676231 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624782fe7e0 0x562478307de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.454+0000 7ff685152640 1 -- 172.21.15.16:0/2824676231 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624782fe7e0 msgr2=0x562478307de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:09:51.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.454+0000 7ff685152640 1 --2- 172.21.15.16:0/2824676231 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624782fe7e0 0x562478307de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.454+0000 7ff685152640 1 -- 172.21.15.16:0/2824676231 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624782fd610 msgr2=0x5624783078a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:09:51.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.454+0000 7ff685152640 1 --2- 172.21.15.16:0/2824676231 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624782fd610 0x5624783078a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.454+0000 7ff685152640 1 -- 172.21.15.16:0/2824676231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624780406b0 con 0x5624782ff1a0 2024-03-28T22:09:51.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.454+0000 7ff685152640 1 --2- 172.21.15.16:0/2824676231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624782ff1a0 0x56247830ee20 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff67802dfd0 tx=0x7ff67802c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e87404c78fff38be server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:51.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.454+0000 7ff65ffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:09:51.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.455+0000 7ff65ffff640 1 -- 172.21.15.16:0/2824676231 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff678051070 con 0x5624782ff1a0 2024-03-28T22:09:51.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.455+0000 7ff65ffff640 1 -- 172.21.15.16:0/2824676231 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7ff678004240 con 0x5624782ff1a0 2024-03-28T22:09:51.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.455+0000 7ff689798740 1 -- 172.21.15.16:0/2824676231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff66c0047a0 con 0x5624782ff1a0 2024-03-28T22:09:51.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.455+0000 7ff65ffff640 1 -- 172.21.15.16:0/2824676231 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7ff67805d050 con 0x5624782ff1a0 2024-03-28T22:09:51.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.455+0000 7ff689798740 10 client.15045 fetch_fsmap learned FSMap version 20 2024-03-28T22:09:51.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.455+0000 7ff689798740 1 -- 172.21.15.16:0/2824676231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x562478311e70 con 0x5624782ff1a0 2024-03-28T22:09:51.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.455+0000 7ff65ffff640 1 -- 172.21.15.16:0/2824676231 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff66c0047a0 con 0x5624782ff1a0 2024-03-28T22:09:51.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.455+0000 7ff65ffff640 1 -- 172.21.15.16:0/2824676231 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7ff67803f070 con 0x5624782ff1a0 2024-03-28T22:09:51.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.455+0000 7ff689798740 10 client.15045 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T22:09:51.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.455+0000 7ff689798740 10 client.15045 resolve_mds: resolved name 'e' to daemon mds.e 2024-03-28T22:09:51.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.455+0000 7ff689798740 20 client.15045 populate_metadata read hostname 'smithi016' 2024-03-28T22:09:51.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.455+0000 7ff689798740 1 --2- 172.21.15.16:0/2824676231 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x562478314540 0x5624783159b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.455+0000 7ff689798740 4 client.15045 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:09:51.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.455+0000 7ff689798740 1 -- 172.21.15.16:0/2824676231 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56247826f030 con 0x562478314540 2024-03-28T22:09:51.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.455+0000 7ff684951640 1 --2- 172.21.15.16:0/2824676231 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x562478314540 0x5624783159b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.456+0000 7ff65ffff640 10 client.15045 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:09:51.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.456+0000 7ff684951640 1 --2- 172.21.15.16:0/2824676231 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x562478314540 0x5624783159b0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=75f702517f98214e server_cookie=42f0936ef4fdb659 in_seq=0 out_seq=0 2024-03-28T22:09:51.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.457+0000 7ff65ffff640 1 -- 172.21.15.16:0/2824676231 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56247826f030 con 0x562478314540 2024-03-28T22:09:51.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.457+0000 7ff65ffff640 10 client.15045 handle_command_reply: tid=0 2024-03-28T22:09:51.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.457+0000 7ff689798740 1 client.15045 shutdown 2024-03-28T22:09:51.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.457+0000 7ff689798740 1 -- 172.21.15.16:0/2824676231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624782ff1a0 msgr2=0x56247830ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:51.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.457+0000 7ff689798740 1 --2- 172.21.15.16:0/2824676231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624782ff1a0 0x56247830ee20 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff67802dfd0 tx=0x7ff67802c0f0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.457+0000 7ff689798740 1 -- 172.21.15.16:0/2824676231 shutdown_connections 2024-03-28T22:09:51.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.457+0000 7ff689798740 1 --2- 172.21.15.16:0/2824676231 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x562478314540 0x5624783159b0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.457+0000 7ff689798740 1 --2- 172.21.15.16:0/2824676231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624782ff1a0 0x56247830ee20 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.457+0000 7ff689798740 1 --2- 172.21.15.16:0/2824676231 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624782fe7e0 0x562478307de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.457+0000 7ff65ffff640 0 client.15045 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:09:51.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.457+0000 7ff689798740 1 --2- 172.21.15.16:0/2824676231 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624782fd610 0x5624783078a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.457+0000 7ff689798740 1 -- 172.21.15.16:0/2824676231 >> 172.21.15.16:0/2824676231 conn(0x5624782f8a00 msgr2=0x5624782f7b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:51.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.457+0000 7ff689798740 1 -- 172.21.15.16:0/2824676231 shutdown_connections 2024-03-28T22:09:51.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.457+0000 7ff689798740 1 -- 172.21.15.16:0/2824676231 wait complete. 2024-03-28T22:09:51.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.457+0000 7ff689798740 20 client.15045 trim_cache size 0 max 16384 2024-03-28T22:09:51.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.475+0000 7ff689798740 1 Processor -- start 2024-03-28T22:09:51.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.475+0000 7ff689798740 1 -- start start 2024-03-28T22:09:51.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.475+0000 7ff689798740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624782fd610 0x5624782f8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.475+0000 7ff689798740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624782fe7e0 0x56247830ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.475+0000 7ff689798740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624782ff1a0 0x562478302140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.475+0000 7ff689798740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562478313cf0 con 0x5624782fe7e0 2024-03-28T22:09:51.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.475+0000 7ff689798740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5624783143c0 con 0x5624782ff1a0 2024-03-28T22:09:51.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.475+0000 7ff689798740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x562478314a90 con 0x5624782fd610 2024-03-28T22:09:51.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.475+0000 7ff67ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624782fe7e0 0x56247830ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.475+0000 7ff67ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624782fe7e0 0x56247830ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41572/0 (socket says 172.21.15.16:41572) 2024-03-28T22:09:51.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.475+0000 7ff67ffff640 1 -- 172.21.15.16:0/3289347843 learned_addr learned my addr 172.21.15.16:0/3289347843 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:51.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.476+0000 7ff685152640 1 --2- 172.21.15.16:0/3289347843 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624782ff1a0 0x562478302140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.476+0000 7ff684951640 1 --2- 172.21.15.16:0/3289347843 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624782fd610 0x5624782f8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.476+0000 7ff67ffff640 1 -- 172.21.15.16:0/3289347843 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624782fd610 msgr2=0x5624782f8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:09:51.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.476+0000 7ff67ffff640 1 --2- 172.21.15.16:0/3289347843 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624782fd610 0x5624782f8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.476+0000 7ff67ffff640 1 -- 172.21.15.16:0/3289347843 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624782ff1a0 msgr2=0x562478302140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:09:51.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.476+0000 7ff67ffff640 1 --2- 172.21.15.16:0/3289347843 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624782ff1a0 0x562478302140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.476+0000 7ff67ffff640 1 -- 172.21.15.16:0/3289347843 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624782f85a0 con 0x5624782fe7e0 2024-03-28T22:09:51.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.476+0000 7ff684951640 1 --2- 172.21.15.16:0/3289347843 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624782fd610 0x5624782f8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:09:51.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.476+0000 7ff67ffff640 1 --2- 172.21.15.16:0/3289347843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624782fe7e0 0x56247830ede0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ff66c009b80 tx=0x7ff66c0095a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b79ab5f13dd73e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:51.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.476+0000 7ff66a7fc640 1 -- 172.21.15.16:0/3289347843 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff66c026020 con 0x5624782fe7e0 2024-03-28T22:09:51.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.476+0000 7ff66a7fc640 1 -- 172.21.15.16:0/3289347843 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff66c004520 con 0x5624782fe7e0 2024-03-28T22:09:51.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.476+0000 7ff66a7fc640 1 -- 172.21.15.16:0/3289347843 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff66c003b90 con 0x5624782fe7e0 2024-03-28T22:09:51.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.477+0000 7ff689798740 1 -- 172.21.15.16:0/3289347843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624782fe7e0 msgr2=0x56247830ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:51.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.477+0000 7ff689798740 1 --2- 172.21.15.16:0/3289347843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624782fe7e0 0x56247830ede0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ff66c009b80 tx=0x7ff66c0095a0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.477+0000 7ff689798740 1 -- 172.21.15.16:0/3289347843 shutdown_connections 2024-03-28T22:09:51.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.477+0000 7ff689798740 1 --2- 172.21.15.16:0/3289347843 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624782ff1a0 0x562478302140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.477+0000 7ff689798740 1 --2- 172.21.15.16:0/3289347843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624782fe7e0 0x56247830ede0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.477+0000 7ff689798740 1 --2- 172.21.15.16:0/3289347843 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624782fd610 0x5624782f8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.477+0000 7ff689798740 1 -- 172.21.15.16:0/3289347843 >> 172.21.15.16:0/3289347843 conn(0x5624782f8a00 msgr2=0x562478305710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:51.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.477+0000 7ff689798740 1 -- 172.21.15.16:0/3289347843 shutdown_connections 2024-03-28T22:09:51.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.477+0000 7ff689798740 1 -- 172.21.15.16:0/3289347843 wait complete. 2024-03-28T22:09:51.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.477+0000 7ff689798740 1 -- start start 2024-03-28T22:09:51.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.477+0000 7ff689798740 1 Processor -- start 2024-03-28T22:09:51.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.477+0000 7ff689798740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624782fd610 0x56247830af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.477+0000 7ff689798740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624782fe7e0 0x56247830b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.477+0000 7ff689798740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624782ff1a0 0x56247830dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.477+0000 7ff689798740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562478315830 con 0x5624782fd610 2024-03-28T22:09:51.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.477+0000 7ff689798740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x562478315160 con 0x5624782ff1a0 2024-03-28T22:09:51.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.477+0000 7ff689798740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x562478315f00 con 0x5624782fe7e0 2024-03-28T22:09:51.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.477+0000 7ff689798740 10 client.0 _ll_register_callbacks cb 0x7ff678009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:09:51.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.477+0000 7ff684951640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624782fd610 0x56247830af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.478+0000 7ff684951640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624782fd610 0x56247830af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41584/0 (socket says 172.21.15.16:41584) 2024-03-28T22:09:51.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.478+0000 7ff684951640 1 -- 172.21.15.16:0/1453223658 learned_addr learned my addr 172.21.15.16:0/1453223658 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:51.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.478+0000 7ff685152640 1 --2- 172.21.15.16:0/1453223658 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624782ff1a0 0x56247830dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.478+0000 7ff67ffff640 1 --2- 172.21.15.16:0/1453223658 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624782fe7e0 0x56247830b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.478+0000 7ff684951640 1 -- 172.21.15.16:0/1453223658 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624782fe7e0 msgr2=0x56247830b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:09:51.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.478+0000 7ff684951640 1 --2- 172.21.15.16:0/1453223658 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624782fe7e0 0x56247830b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.478+0000 7ff684951640 1 -- 172.21.15.16:0/1453223658 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624782ff1a0 msgr2=0x56247830dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:09:51.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.478+0000 7ff684951640 1 --2- 172.21.15.16:0/1453223658 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624782ff1a0 0x56247830dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.478+0000 7ff684951640 1 -- 172.21.15.16:0/1453223658 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56247830e520 con 0x5624782fd610 2024-03-28T22:09:51.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.478+0000 7ff684951640 1 --2- 172.21.15.16:0/1453223658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624782fd610 0x56247830af70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff674096d20 tx=0x7ff674096c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57624e061058fff7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:51.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.478+0000 7ff65ffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:09:51.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.478+0000 7ff65ffff640 1 -- 172.21.15.16:0/1453223658 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff67401d040 con 0x5624782fd610 2024-03-28T22:09:51.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.478+0000 7ff65ffff640 1 -- 172.21.15.16:0/1453223658 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7ff6740b3260 con 0x5624782fd610 2024-03-28T22:09:51.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.478+0000 7ff689798740 1 -- 172.21.15.16:0/1453223658 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff66c004520 con 0x5624782fd610 2024-03-28T22:09:51.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.478+0000 7ff65ffff640 1 -- 172.21.15.16:0/1453223658 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7ff6740a3060 con 0x5624782fd610 2024-03-28T22:09:51.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.478+0000 7ff689798740 10 client.15051 fetch_fsmap learned FSMap version 20 2024-03-28T22:09:51.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.478+0000 7ff689798740 1 -- 172.21.15.16:0/1453223658 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x56247837a760 con 0x5624782fd610 2024-03-28T22:09:51.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.478+0000 7ff65ffff640 1 -- 172.21.15.16:0/1453223658 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff66c004520 con 0x5624782fd610 2024-03-28T22:09:51.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.479+0000 7ff65ffff640 1 -- 172.21.15.16:0/1453223658 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7ff6740cb8f0 con 0x5624782fd610 2024-03-28T22:09:51.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.479+0000 7ff689798740 10 client.15051 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T22:09:51.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.479+0000 7ff689798740 10 client.15051 resolve_mds: resolved name 'e' to daemon mds.e 2024-03-28T22:09:51.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.479+0000 7ff689798740 20 client.15051 populate_metadata read hostname 'smithi016' 2024-03-28T22:09:51.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.479+0000 7ff689798740 1 --2- 172.21.15.16:0/1453223658 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x56247837f330 0x56247837b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.479+0000 7ff689798740 4 client.15051 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T22:09:51.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.479+0000 7ff689798740 1 -- 172.21.15.16:0/1453223658 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "status"}) -- 0x562478269380 con 0x56247837f330 2024-03-28T22:09:51.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.479+0000 7ff67ffff640 1 --2- 172.21.15.16:0/1453223658 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x56247837f330 0x56247837b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.479+0000 7ff65ffff640 10 client.15051 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:09:51.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.479+0000 7ff67ffff640 1 --2- 172.21.15.16:0/1453223658 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x56247837f330 0x56247837b770 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9336a5e884b45141 server_cookie=995863b10e92cd00 in_seq=0 out_seq=0 2024-03-28T22:09:51.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.479+0000 7ff65ffff640 1 -- 172.21.15.16:0/1453223658 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x562478269380 con 0x56247837f330 2024-03-28T22:09:51.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.479+0000 7ff65ffff640 10 client.15051 handle_command_reply: tid=0 2024-03-28T22:09:51.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.479+0000 7ff689798740 1 client.15051 shutdown 2024-03-28T22:09:51.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.479+0000 7ff689798740 1 -- 172.21.15.16:0/1453223658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624782fd610 msgr2=0x56247830af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:51.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.479+0000 7ff689798740 1 --2- 172.21.15.16:0/1453223658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624782fd610 0x56247830af70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff674096d20 tx=0x7ff674096c90 comp rx=0 tx=0).stop 2024-03-28T22:09:51.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.480+0000 7ff689798740 1 -- 172.21.15.16:0/1453223658 shutdown_connections 2024-03-28T22:09:51.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.480+0000 7ff689798740 1 --2- 172.21.15.16:0/1453223658 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x56247837f330 0x56247837b770 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.480+0000 7ff689798740 1 --2- 172.21.15.16:0/1453223658 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5624782ff1a0 0x56247830dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.480+0000 7ff689798740 1 --2- 172.21.15.16:0/1453223658 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624782fe7e0 0x56247830b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.480+0000 7ff689798740 1 --2- 172.21.15.16:0/1453223658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624782fd610 0x56247830af70 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.480+0000 7ff65ffff640 0 client.15051 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:09:51.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.480+0000 7ff689798740 1 -- 172.21.15.16:0/1453223658 >> 172.21.15.16:0/1453223658 conn(0x5624782f8a00 msgr2=0x5624782f8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:51.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.480+0000 7ff689798740 1 -- 172.21.15.16:0/1453223658 shutdown_connections 2024-03-28T22:09:51.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.480+0000 7ff689798740 1 -- 172.21.15.16:0/1453223658 wait complete. 2024-03-28T22:09:51.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.480+0000 7ff689798740 20 client.15051 trim_cache size 0 max 16384 2024-03-28T22:09:51.489 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:09:51.489 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "798d9884-ed4e-11ee-95d0-87774f69a715", 2024-03-28T22:09:51.489 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 2, 2024-03-28T22:09:51.489 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24500, 2024-03-28T22:09:51.490 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-03-28T22:09:51.490 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-03-28T22:09:51.490 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-03-28T22:09:51.490 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 78.142856065999993, 2024-03-28T22:09:51.490 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 20, 2024-03-28T22:09:51.490 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 88, 2024-03-28T22:09:51.490 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 86, 2024-03-28T22:09:51.490 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 105.812976807 2024-03-28T22:09:51.490 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:09:51.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.480+0000 7ff686bb5640 1 -- 172.21.15.16:0/3647255881 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff6700a1e10 msgr2=0x7ff6700a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:51.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.480+0000 7ff686bb5640 1 --2- 172.21.15.16:0/3647255881 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff6700a1e10 0x7ff6700a42d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff678005e30 tx=0x7ff67802cf00 comp rx=0 tx=0).stop 2024-03-28T22:09:51.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.480+0000 7ff686bb5640 1 -- 172.21.15.16:0/3647255881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6801021f0 msgr2=0x7ff680197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:51.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.480+0000 7ff686bb5640 1 --2- 172.21.15.16:0/3647255881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6801021f0 0x7ff680197a60 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff67400c970 tx=0x7ff67400ce30 comp rx=0 tx=0).stop 2024-03-28T22:09:51.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.480+0000 7ff686bb5640 1 -- 172.21.15.16:0/3647255881 shutdown_connections 2024-03-28T22:09:51.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.480+0000 7ff686bb5640 1 --2- 172.21.15.16:0/3647255881 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff6700a1e10 0x7ff6700a42d0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.480+0000 7ff686bb5640 1 --2- 172.21.15.16:0/3647255881 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff680107450 0x7ff68019f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.480+0000 7ff686bb5640 1 --2- 172.21.15.16:0/3647255881 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff680104b20 0x7ff680197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.480+0000 7ff686bb5640 1 --2- 172.21.15.16:0/3647255881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6801021f0 0x7ff680197a60 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.480+0000 7ff686bb5640 1 -- 172.21.15.16:0/3647255881 >> 172.21.15.16:0/3647255881 conn(0x7ff6800fbbe0 msgr2=0x7ff680102b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:51.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.480+0000 7ff686bb5640 1 -- 172.21.15.16:0/3647255881 shutdown_connections 2024-03-28T22:09:51.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.480+0000 7ff686bb5640 1 -- 172.21.15.16:0/3647255881 wait complete. 2024-03-28T22:09:51.493 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-28T22:09:51.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.594+0000 7f4ee4af4640 1 -- 172.21.15.16:0/1383636578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ee0100ff0 msgr2=0x7f4ee0101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:51.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.594+0000 7f4ee4af4640 1 --2- 172.21.15.16:0/1383636578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ee0100ff0 0x7f4ee0101470 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4ed4009900 tx=0x7f4ed402bc50 comp rx=0 tx=0).stop 2024-03-28T22:09:51.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.594+0000 7f4ee4af4640 1 -- 172.21.15.16:0/1383636578 shutdown_connections 2024-03-28T22:09:51.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.594+0000 7f4ee4af4640 1 --2- 172.21.15.16:0/1383636578 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4ee01019b0 0x7f4ee0108240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.594+0000 7f4ee4af4640 1 --2- 172.21.15.16:0/1383636578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ee0100ff0 0x7f4ee0101470 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.594+0000 7f4ee4af4640 1 --2- 172.21.15.16:0/1383636578 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ee00ffe00 0x7f4ee0100220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.594+0000 7f4ee4af4640 1 -- 172.21.15.16:0/1383636578 >> 172.21.15.16:0/1383636578 conn(0x7f4ee00fb370 msgr2=0x7f4ee00fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:51.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.594+0000 7f4ee4af4640 1 -- 172.21.15.16:0/1383636578 shutdown_connections 2024-03-28T22:09:51.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.594+0000 7f4ee4af4640 1 -- 172.21.15.16:0/1383636578 wait complete. 2024-03-28T22:09:51.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.595+0000 7f4ee4af4640 1 Processor -- start 2024-03-28T22:09:51.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.595+0000 7f4ee4af4640 1 -- start start 2024-03-28T22:09:51.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.595+0000 7f4ee4af4640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ee00ffe00 0x7f4ee0197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.595+0000 7f4ee4af4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ee0100ff0 0x7f4ee0197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.595+0000 7f4ee4af4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4ee01019b0 0x7f4ee019efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.595+0000 7f4ee4af4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4ee010d480 con 0x7f4ee0100ff0 2024-03-28T22:09:51.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.595+0000 7f4ee4af4640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4ee010d300 con 0x7f4ee00ffe00 2024-03-28T22:09:51.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.595+0000 7f4ee4af4640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4ee010d600 con 0x7f4ee01019b0 2024-03-28T22:09:51.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.595+0000 7f4eddd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ee0100ff0 0x7f4ee0197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.595+0000 7f4eddd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ee0100ff0 0x7f4ee0197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41600/0 (socket says 172.21.15.16:41600) 2024-03-28T22:09:51.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.595+0000 7f4eddd9b640 1 -- 172.21.15.16:0/3837319835 learned_addr learned my addr 172.21.15.16:0/3837319835 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:51.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.595+0000 7f4ede59c640 1 --2- 172.21.15.16:0/3837319835 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ee00ffe00 0x7f4ee0197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.595+0000 7f4eded9d640 1 --2- 172.21.15.16:0/3837319835 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4ee01019b0 0x7f4ee019efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.595+0000 7f4eddd9b640 1 -- 172.21.15.16:0/3837319835 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4ee01019b0 msgr2=0x7f4ee019efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:51.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.595+0000 7f4eddd9b640 1 --2- 172.21.15.16:0/3837319835 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4ee01019b0 0x7f4ee019efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.595+0000 7f4eddd9b640 1 -- 172.21.15.16:0/3837319835 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ee00ffe00 msgr2=0x7f4ee0197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:51.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.595+0000 7f4eddd9b640 1 --2- 172.21.15.16:0/3837319835 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ee00ffe00 0x7f4ee0197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.595+0000 7f4eddd9b640 1 -- 172.21.15.16:0/3837319835 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ee019f630 con 0x7f4ee0100ff0 2024-03-28T22:09:51.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.595+0000 7f4eded9d640 1 --2- 172.21.15.16:0/3837319835 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4ee01019b0 0x7f4ee019efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:09:51.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.596+0000 7f4eddd9b640 1 --2- 172.21.15.16:0/3837319835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ee0100ff0 0x7f4ee0197f50 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f4ed4005ab0 tx=0x7f4ed402c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:51.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.596+0000 7f4ecf7fe640 1 -- 172.21.15.16:0/3837319835 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed4041070 con 0x7f4ee0100ff0 2024-03-28T22:09:51.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.596+0000 7f4ecf7fe640 1 -- 172.21.15.16:0/3837319835 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ed4004450 con 0x7f4ee0100ff0 2024-03-28T22:09:51.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.596+0000 7f4ee4af4640 1 -- 172.21.15.16:0/3837319835 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ee019f8c0 con 0x7f4ee0100ff0 2024-03-28T22:09:51.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.596+0000 7f4ecf7fe640 1 -- 172.21.15.16:0/3837319835 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed4034440 con 0x7f4ee0100ff0 2024-03-28T22:09:51.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.596+0000 7f4ee4af4640 1 -- 172.21.15.16:0/3837319835 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ee019fe60 con 0x7f4ee0100ff0 2024-03-28T22:09:51.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.598+0000 7f4ecf7fe640 1 -- 172.21.15.16:0/3837319835 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f4ed4044070 con 0x7f4ee0100ff0 2024-03-28T22:09:51.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.598+0000 7f4ee77b5740 1 Processor -- start 2024-03-28T22:09:51.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.598+0000 7f4ecf7fe640 1 --2- 172.21.15.16:0/3837319835 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4ec80a1e90 0x7f4ec80a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.599+0000 7f4eded9d640 1 --2- 172.21.15.16:0/3837319835 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4ec80a1e90 0x7f4ec80a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.599+0000 7f4ee77b5740 1 -- start start 2024-03-28T22:09:51.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.599+0000 7f4ecf7fe640 1 -- 172.21.15.16:0/3837319835 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f4ed40efa50 con 0x7f4ee0100ff0 2024-03-28T22:09:51.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.599+0000 7f4ee77b5740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55579932a610 0x55579932aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.599+0000 7f4ee77b5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55579932b7e0 0x55579932bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.599+0000 7f4ee77b5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55579932c1a0 0x555799332a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.599+0000 7f4ee77b5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55579928eaf0 con 0x55579932c1a0 2024-03-28T22:09:51.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.599+0000 7f4ee77b5740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5557992379d0 con 0x55579932a610 2024-03-28T22:09:51.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.599+0000 7f4ee77b5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55579928ab90 con 0x55579932b7e0 2024-03-28T22:09:51.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.599+0000 7f4ede59c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55579932a610 0x55579932aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.599+0000 7f4ede59c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55579932a610 0x55579932aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:44212/0 (socket says 172.21.15.16:44212) 2024-03-28T22:09:51.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.599+0000 7f4ede59c640 1 -- 172.21.15.16:0/2130835381 learned_addr learned my addr 172.21.15.16:0/2130835381 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:51.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.599+0000 7f4eded9d640 1 --2- 172.21.15.16:0/2130835381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55579932c1a0 0x555799332a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.599+0000 7f4eded9d640 1 --2- 172.21.15.16:0/3837319835 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4ec80a1e90 0x7f4ec80a4350 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4ed8005fd0 tx=0x7f4ed80082b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:51.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.599+0000 7f4eddd9b640 1 --2- 172.21.15.16:0/2130835381 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55579932b7e0 0x55579932bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.599+0000 7f4ede59c640 1 -- 172.21.15.16:0/2130835381 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55579932b7e0 msgr2=0x55579932bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:09:51.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.599+0000 7f4ede59c640 1 --2- 172.21.15.16:0/2130835381 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55579932b7e0 0x55579932bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.599+0000 7f4ede59c640 1 -- 172.21.15.16:0/2130835381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55579932c1a0 msgr2=0x555799332a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:09:51.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.599+0000 7f4ede59c640 1 --2- 172.21.15.16:0/2130835381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55579932c1a0 0x555799332a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.599+0000 7f4ede59c640 1 -- 172.21.15.16:0/2130835381 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555798fa32d0 con 0x55579932a610 2024-03-28T22:09:51.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.599+0000 7f4eded9d640 1 --2- 172.21.15.16:0/2130835381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55579932c1a0 0x555799332a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:09:51.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.599+0000 7f4eddd9b640 1 --2- 172.21.15.16:0/2130835381 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55579932b7e0 0x55579932bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:09:51.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.599+0000 7f4ede59c640 1 --2- 172.21.15.16:0/2130835381 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55579932a610 0x55579932aa30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0009c80 tx=0x7f4ed0011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a18bc1f3b6a85dc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:51.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.600+0000 7f4eccff9640 1 -- 172.21.15.16:0/2130835381 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed001e070 con 0x55579932a610 2024-03-28T22:09:51.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.600+0000 7f4eccff9640 1 -- 172.21.15.16:0/2130835381 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ed0009e30 con 0x55579932a610 2024-03-28T22:09:51.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.600+0000 7f4eccff9640 1 -- 172.21.15.16:0/2130835381 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed000ea80 con 0x55579932a610 2024-03-28T22:09:51.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.600+0000 7f4ee77b5740 1 -- 172.21.15.16:0/2130835381 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55579932a610 msgr2=0x55579932aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:51.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.600+0000 7f4ee77b5740 1 --2- 172.21.15.16:0/2130835381 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55579932a610 0x55579932aa30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0009c80 tx=0x7f4ed0011810 comp rx=0 tx=0).stop 2024-03-28T22:09:51.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.600+0000 7f4ee77b5740 1 -- 172.21.15.16:0/2130835381 shutdown_connections 2024-03-28T22:09:51.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.600+0000 7f4ee77b5740 1 --2- 172.21.15.16:0/2130835381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55579932c1a0 0x555799332a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.600+0000 7f4ee77b5740 1 --2- 172.21.15.16:0/2130835381 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55579932b7e0 0x55579932bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.600+0000 7f4ee77b5740 1 --2- 172.21.15.16:0/2130835381 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55579932a610 0x55579932aa30 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.600+0000 7f4ee77b5740 1 -- 172.21.15.16:0/2130835381 >> 172.21.15.16:0/2130835381 conn(0x555799325a00 msgr2=0x555799325e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:51.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.601+0000 7f4ee77b5740 1 -- 172.21.15.16:0/2130835381 shutdown_connections 2024-03-28T22:09:51.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.601+0000 7f4ee77b5740 1 -- 172.21.15.16:0/2130835381 wait complete. 2024-03-28T22:09:51.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.601+0000 7f4ee77b5740 1 -- start start 2024-03-28T22:09:51.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.601+0000 7f4ee77b5740 1 Processor -- start 2024-03-28T22:09:51.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.601+0000 7f4ee77b5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55579932a610 0x555799334970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.601+0000 7f4ee77b5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55579932b7e0 0x555799334eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.601+0000 7f4ee77b5740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55579932c1a0 0x55579933bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.601+0000 7f4ee77b5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5557993331b0 con 0x55579932a610 2024-03-28T22:09:51.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.601+0000 7f4ee77b5740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55579924af70 con 0x55579932c1a0 2024-03-28T22:09:51.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.601+0000 7f4ee77b5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x555799333330 con 0x55579932b7e0 2024-03-28T22:09:51.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.601+0000 7f4ee77b5740 10 client.0 _ll_register_callbacks cb 0x555798fa2ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:09:51.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.601+0000 7f4ede59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55579932a610 0x555799334970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.601+0000 7f4ede59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55579932a610 0x555799334970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41624/0 (socket says 172.21.15.16:41624) 2024-03-28T22:09:51.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.601+0000 7f4ede59c640 1 -- 172.21.15.16:0/3987309628 learned_addr learned my addr 172.21.15.16:0/3987309628 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:51.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.601+0000 7f4eded9d640 1 --2- 172.21.15.16:0/3987309628 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55579932c1a0 0x55579933bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.601+0000 7f4eddd9b640 1 --2- 172.21.15.16:0/3987309628 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55579932b7e0 0x555799334eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.601+0000 7f4ede59c640 1 -- 172.21.15.16:0/3987309628 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55579932b7e0 msgr2=0x555799334eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:09:51.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.601+0000 7f4ede59c640 1 --2- 172.21.15.16:0/3987309628 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55579932b7e0 0x555799334eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.601+0000 7f4ede59c640 1 -- 172.21.15.16:0/3987309628 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55579932c1a0 msgr2=0x55579933bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:09:51.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.601+0000 7f4ede59c640 1 --2- 172.21.15.16:0/3987309628 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55579932c1a0 0x55579933bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.601+0000 7f4ede59c640 1 -- 172.21.15.16:0/3987309628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55579906d6b0 con 0x55579932a610 2024-03-28T22:09:51.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.602+0000 7f4ede59c640 1 --2- 172.21.15.16:0/3987309628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55579932a610 0x555799334970 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0000c00 tx=0x7f4ed0005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b98885707dc1ef85 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:51.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.602+0000 7f4ec5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:09:51.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.602+0000 7f4ec5ffb640 1 -- 172.21.15.16:0/3987309628 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed0029020 con 0x55579932a610 2024-03-28T22:09:51.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.602+0000 7f4ec5ffb640 1 -- 172.21.15.16:0/3987309628 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f4ed000ea00 con 0x55579932a610 2024-03-28T22:09:51.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.602+0000 7f4ee77b5740 1 -- 172.21.15.16:0/3987309628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ed0009e30 con 0x55579932a610 2024-03-28T22:09:51.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.602+0000 7f4ec5ffb640 1 -- 172.21.15.16:0/3987309628 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f4ed001e040 con 0x55579932a610 2024-03-28T22:09:51.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.602+0000 7f4ee77b5740 10 client.15063 fetch_fsmap learned FSMap version 20 2024-03-28T22:09:51.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.602+0000 7f4ee77b5740 1 -- 172.21.15.16:0/3987309628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55579933ef20 con 0x55579932a610 2024-03-28T22:09:51.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.602+0000 7f4ec5ffb640 1 -- 172.21.15.16:0/3987309628 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ed0009e30 con 0x55579932a610 2024-03-28T22:09:51.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.602+0000 7f4ec5ffb640 1 -- 172.21.15.16:0/3987309628 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f4ed0024470 con 0x55579932a610 2024-03-28T22:09:51.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.602+0000 7f4ee77b5740 10 client.15063 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T22:09:51.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.602+0000 7f4ee77b5740 10 client.15063 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-28T22:09:51.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.602+0000 7f4ee77b5740 20 client.15063 populate_metadata read hostname 'smithi016' 2024-03-28T22:09:51.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.602+0000 7f4ee77b5740 1 --2- 172.21.15.16:0/3987309628 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x555799341300 0x555799342770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.602+0000 7f4ee77b5740 4 client.15063 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:09:51.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.602+0000 7f4ee77b5740 1 -- 172.21.15.16:0/3987309628 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55579929c030 con 0x555799341300 2024-03-28T22:09:51.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.603+0000 7f4eddd9b640 1 --2- 172.21.15.16:0/3987309628 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x555799341300 0x555799342770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.603+0000 7f4ec5ffb640 10 client.15063 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:09:51.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.603+0000 7f4eddd9b640 1 --2- 172.21.15.16:0/3987309628 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x555799341300 0x555799342770 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c621e50c9e04f6d6 server_cookie=1f56ed300a80c352 in_seq=0 out_seq=0 2024-03-28T22:09:51.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.604+0000 7f4ec5ffb640 1 -- 172.21.15.16:0/3987309628 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55579929c030 con 0x555799341300 2024-03-28T22:09:51.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.604+0000 7f4ec5ffb640 10 client.15063 handle_command_reply: tid=0 2024-03-28T22:09:51.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.604+0000 7f4ee77b5740 1 client.15063 shutdown 2024-03-28T22:09:51.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.604+0000 7f4ee77b5740 1 -- 172.21.15.16:0/3987309628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55579932a610 msgr2=0x555799334970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:51.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.604+0000 7f4ee77b5740 1 --2- 172.21.15.16:0/3987309628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55579932a610 0x555799334970 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0000c00 tx=0x7f4ed0005720 comp rx=0 tx=0).stop 2024-03-28T22:09:51.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.604+0000 7f4ee77b5740 1 -- 172.21.15.16:0/3987309628 shutdown_connections 2024-03-28T22:09:51.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.604+0000 7f4ee77b5740 1 --2- 172.21.15.16:0/3987309628 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x555799341300 0x555799342770 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.604+0000 7f4ee77b5740 1 --2- 172.21.15.16:0/3987309628 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55579932c1a0 0x55579933bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.604+0000 7f4ee77b5740 1 --2- 172.21.15.16:0/3987309628 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55579932b7e0 0x555799334eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.604+0000 7f4ec5ffb640 0 client.15063 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:09:51.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.604+0000 7f4ee77b5740 1 --2- 172.21.15.16:0/3987309628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55579932a610 0x555799334970 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.604+0000 7f4ee77b5740 1 -- 172.21.15.16:0/3987309628 >> 172.21.15.16:0/3987309628 conn(0x555799325a00 msgr2=0x555799324b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:51.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.604+0000 7f4ee77b5740 1 -- 172.21.15.16:0/3987309628 shutdown_connections 2024-03-28T22:09:51.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.604+0000 7f4ee77b5740 1 -- 172.21.15.16:0/3987309628 wait complete. 2024-03-28T22:09:51.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.604+0000 7f4ee77b5740 20 client.15063 trim_cache size 0 max 16384 2024-03-28T22:09:51.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.622+0000 7f4ee77b5740 1 Processor -- start 2024-03-28T22:09:51.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.622+0000 7f4ee77b5740 1 -- start start 2024-03-28T22:09:51.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.622+0000 7f4ee77b5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55579932a610 0x555799333090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.622+0000 7f4ee77b5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55579932b7e0 0x55579933be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.622+0000 7f4ee77b5740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55579932c1a0 0x5557993402d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.622+0000 7f4ee77b5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555799340ab0 con 0x55579932a610 2024-03-28T22:09:51.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.622+0000 7f4ee77b5740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x555799341180 con 0x55579932c1a0 2024-03-28T22:09:51.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.622+0000 7f4ee77b5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x555799341850 con 0x55579932b7e0 2024-03-28T22:09:51.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.622+0000 7f4ede59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55579932a610 0x555799333090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.622+0000 7f4ede59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55579932a610 0x555799333090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41628/0 (socket says 172.21.15.16:41628) 2024-03-28T22:09:51.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.622+0000 7f4ede59c640 1 -- 172.21.15.16:0/239348879 learned_addr learned my addr 172.21.15.16:0/239348879 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:51.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.622+0000 7f4eded9d640 1 --2- 172.21.15.16:0/239348879 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55579932c1a0 0x5557993402d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.622+0000 7f4ede59c640 1 -- 172.21.15.16:0/239348879 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55579932b7e0 msgr2=0x55579933be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:09:51.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.622+0000 7f4ede59c640 1 --2- 172.21.15.16:0/239348879 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55579932b7e0 0x55579933be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.622+0000 7f4ede59c640 1 -- 172.21.15.16:0/239348879 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55579932c1a0 msgr2=0x5557993402d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:09:51.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.622+0000 7f4ede59c640 1 --2- 172.21.15.16:0/239348879 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55579932c1a0 0x5557993402d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.622+0000 7f4ede59c640 1 -- 172.21.15.16:0/239348879 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555799324040 con 0x55579932a610 2024-03-28T22:09:51.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.623+0000 7f4ede59c640 1 --2- 172.21.15.16:0/239348879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55579932a610 0x555799333090 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4ed00059f0 tx=0x7f4ed0024fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fba844b2d9cf4904 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:51.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.623+0000 7f4ec7fff640 1 -- 172.21.15.16:0/239348879 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed0029070 con 0x55579932a610 2024-03-28T22:09:51.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.623+0000 7f4ec7fff640 1 -- 172.21.15.16:0/239348879 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ed0024b00 con 0x55579932a610 2024-03-28T22:09:51.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.623+0000 7f4ec7fff640 1 -- 172.21.15.16:0/239348879 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed0019430 con 0x55579932a610 2024-03-28T22:09:51.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.623+0000 7f4ee77b5740 1 -- 172.21.15.16:0/239348879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55579932a610 msgr2=0x555799333090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:51.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.623+0000 7f4ee77b5740 1 --2- 172.21.15.16:0/239348879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55579932a610 0x555799333090 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4ed00059f0 tx=0x7f4ed0024fa0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.623+0000 7f4ee77b5740 1 -- 172.21.15.16:0/239348879 shutdown_connections 2024-03-28T22:09:51.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.623+0000 7f4ee77b5740 1 --2- 172.21.15.16:0/239348879 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55579932c1a0 0x5557993402d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.623+0000 7f4ee77b5740 1 --2- 172.21.15.16:0/239348879 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55579932b7e0 0x55579933be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.623+0000 7f4ee77b5740 1 --2- 172.21.15.16:0/239348879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55579932a610 0x555799333090 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.623+0000 7f4ee77b5740 1 -- 172.21.15.16:0/239348879 >> 172.21.15.16:0/239348879 conn(0x555799325a00 msgr2=0x5557993294f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:51.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.623+0000 7f4ee77b5740 1 -- 172.21.15.16:0/239348879 shutdown_connections 2024-03-28T22:09:51.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.623+0000 7f4ee77b5740 1 -- 172.21.15.16:0/239348879 wait complete. 2024-03-28T22:09:51.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.624+0000 7f4ee77b5740 1 -- start start 2024-03-28T22:09:51.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.624+0000 7f4ee77b5740 1 Processor -- start 2024-03-28T22:09:51.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.624+0000 7f4ee77b5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55579932a610 0x5557993377a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.624+0000 7f4ee77b5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55579932b7e0 0x555799337ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.624+0000 7f4ee77b5740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55579932c1a0 0x55579933ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.624+0000 7f4ee77b5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5557993425f0 con 0x55579932b7e0 2024-03-28T22:09:51.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.624+0000 7f4ee77b5740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x555799341f20 con 0x55579932c1a0 2024-03-28T22:09:51.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.624+0000 7f4ee77b5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x555799342cc0 con 0x55579932a610 2024-03-28T22:09:51.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.624+0000 7f4ee77b5740 10 client.0 _ll_register_callbacks cb 0x7f4ed001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:09:51.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.624+0000 7f4eddd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55579932b7e0 0x555799337ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.624+0000 7f4eddd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55579932b7e0 0x555799337ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41630/0 (socket says 172.21.15.16:41630) 2024-03-28T22:09:51.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.624+0000 7f4eddd9b640 1 -- 172.21.15.16:0/2219139004 learned_addr learned my addr 172.21.15.16:0/2219139004 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:51.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.624+0000 7f4eded9d640 1 --2- 172.21.15.16:0/2219139004 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55579932c1a0 0x55579933ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.624+0000 7f4eddd9b640 1 -- 172.21.15.16:0/2219139004 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55579932a610 msgr2=0x5557993377a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:09:51.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.624+0000 7f4ede59c640 1 --2- 172.21.15.16:0/2219139004 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55579932a610 0x5557993377a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.624+0000 7f4eddd9b640 1 --2- 172.21.15.16:0/2219139004 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55579932a610 0x5557993377a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.624+0000 7f4eddd9b640 1 -- 172.21.15.16:0/2219139004 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55579932c1a0 msgr2=0x55579933ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:09:51.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.624+0000 7f4eddd9b640 1 --2- 172.21.15.16:0/2219139004 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55579932c1a0 0x55579933ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.624+0000 7f4eddd9b640 1 -- 172.21.15.16:0/2219139004 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55579933b280 con 0x55579932b7e0 2024-03-28T22:09:51.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.624+0000 7f4eddd9b640 1 --2- 172.21.15.16:0/2219139004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55579932b7e0 0x555799337ce0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4ed40b90d0 tx=0x7f4ed40035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96d7b2dc7ae2ef82 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:51.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.624+0000 7f4ec5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:09:51.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.625+0000 7f4ec5ffb640 1 -- 172.21.15.16:0/2219139004 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed40d5020 con 0x55579932b7e0 2024-03-28T22:09:51.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.625+0000 7f4ec5ffb640 1 -- 172.21.15.16:0/2219139004 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f4ed40baa00 con 0x55579932b7e0 2024-03-28T22:09:51.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.625+0000 7f4ee77b5740 1 -- 172.21.15.16:0/2219139004 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ed0024b00 con 0x55579932b7e0 2024-03-28T22:09:51.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.625+0000 7f4ec5ffb640 1 -- 172.21.15.16:0/2219139004 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f4ed404b040 con 0x55579932b7e0 2024-03-28T22:09:51.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.625+0000 7f4ee77b5740 10 client.15069 fetch_fsmap learned FSMap version 20 2024-03-28T22:09:51.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.625+0000 7f4ee77b5740 1 -- 172.21.15.16:0/2219139004 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5557993a72a0 con 0x55579932b7e0 2024-03-28T22:09:51.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.625+0000 7f4ec5ffb640 1 -- 172.21.15.16:0/2219139004 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ed0024b00 con 0x55579932b7e0 2024-03-28T22:09:51.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.625+0000 7f4ec5ffb640 1 -- 172.21.15.16:0/2219139004 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f4ed4003680 con 0x55579932b7e0 2024-03-28T22:09:51.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.625+0000 7f4ee77b5740 10 client.15069 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T22:09:51.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.625+0000 7f4ee77b5740 10 client.15069 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-28T22:09:51.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.625+0000 7f4ee77b5740 20 client.15069 populate_metadata read hostname 'smithi016' 2024-03-28T22:09:51.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.625+0000 7f4ee77b5740 1 --2- 172.21.15.16:0/2219139004 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5557993abf00 0x5557993aac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.625+0000 7f4ee77b5740 4 client.15069 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T22:09:51.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.625+0000 7f4ee77b5740 1 -- 172.21.15.16:0/2219139004 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "status"}) -- 0x555799296380 con 0x5557993abf00 2024-03-28T22:09:51.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.625+0000 7f4ede59c640 1 --2- 172.21.15.16:0/2219139004 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5557993abf00 0x5557993aac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.626+0000 7f4ec5ffb640 10 client.15069 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:09:51.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.626+0000 7f4ede59c640 1 --2- 172.21.15.16:0/2219139004 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5557993abf00 0x5557993aac10 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=add90851d0011d38 server_cookie=7dc125032fc746c0 in_seq=0 out_seq=0 2024-03-28T22:09:51.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.626+0000 7f4ec5ffb640 1 -- 172.21.15.16:0/2219139004 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x555799296380 con 0x5557993abf00 2024-03-28T22:09:51.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.626+0000 7f4ec5ffb640 10 client.15069 handle_command_reply: tid=0 2024-03-28T22:09:51.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.626+0000 7f4ee77b5740 1 client.15069 shutdown 2024-03-28T22:09:51.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.626+0000 7f4ee77b5740 1 -- 172.21.15.16:0/2219139004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55579932b7e0 msgr2=0x555799337ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:51.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.626+0000 7f4ee77b5740 1 --2- 172.21.15.16:0/2219139004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55579932b7e0 0x555799337ce0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4ed40b90d0 tx=0x7f4ed40035f0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.626+0000 7f4ee77b5740 1 -- 172.21.15.16:0/2219139004 shutdown_connections 2024-03-28T22:09:51.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.626+0000 7f4ee77b5740 1 --2- 172.21.15.16:0/2219139004 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5557993abf00 0x5557993aac10 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.626+0000 7f4ee77b5740 1 --2- 172.21.15.16:0/2219139004 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55579932c1a0 0x55579933ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.626+0000 7f4ec5ffb640 0 client.15069 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:09:51.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.626+0000 7f4ee77b5740 1 --2- 172.21.15.16:0/2219139004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55579932b7e0 0x555799337ce0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.626+0000 7f4ee77b5740 1 --2- 172.21.15.16:0/2219139004 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55579932a610 0x5557993377a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.626+0000 7f4ee77b5740 1 -- 172.21.15.16:0/2219139004 >> 172.21.15.16:0/2219139004 conn(0x555799325a00 msgr2=0x555799320760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:51.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.626+0000 7f4ee77b5740 1 -- 172.21.15.16:0/2219139004 shutdown_connections 2024-03-28T22:09:51.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.626+0000 7f4ee77b5740 1 -- 172.21.15.16:0/2219139004 wait complete. 2024-03-28T22:09:51.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.626+0000 7f4ee77b5740 20 client.15069 trim_cache size 0 max 16384 2024-03-28T22:09:51.635 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:09:51.635 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "798d9884-ed4e-11ee-95d0-87774f69a715", 2024-03-28T22:09:51.635 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 1, 2024-03-28T22:09:51.635 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24524, 2024-03-28T22:09:51.635 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-03-28T22:09:51.635 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-03-28T22:09:51.636 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-03-28T22:09:51.636 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 79.318355229999995, 2024-03-28T22:09:51.636 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 20, 2024-03-28T22:09:51.636 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 88, 2024-03-28T22:09:51.636 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 86, 2024-03-28T22:09:51.636 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 97.789576299000004 2024-03-28T22:09:51.636 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:09:51.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.626+0000 7f4ee4af4640 1 -- 172.21.15.16:0/3837319835 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4ec80a1e90 msgr2=0x7f4ec80a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:51.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.627+0000 7f4ee4af4640 1 --2- 172.21.15.16:0/3837319835 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4ec80a1e90 0x7f4ec80a4350 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4ed8005fd0 tx=0x7f4ed80082b0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.627+0000 7f4ee4af4640 1 -- 172.21.15.16:0/3837319835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ee0100ff0 msgr2=0x7f4ee0197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:51.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.627+0000 7f4ee4af4640 1 --2- 172.21.15.16:0/3837319835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ee0100ff0 0x7f4ee0197f50 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f4ed4005ab0 tx=0x7f4ed402c6e0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.627+0000 7f4ee4af4640 1 -- 172.21.15.16:0/3837319835 shutdown_connections 2024-03-28T22:09:51.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.627+0000 7f4ee4af4640 1 --2- 172.21.15.16:0/3837319835 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4ec80a1e90 0x7f4ec80a4350 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.627+0000 7f4ee4af4640 1 --2- 172.21.15.16:0/3837319835 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4ee01019b0 0x7f4ee019efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.627+0000 7f4ee4af4640 1 --2- 172.21.15.16:0/3837319835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ee0100ff0 0x7f4ee0197f50 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.627+0000 7f4ee4af4640 1 --2- 172.21.15.16:0/3837319835 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ee00ffe00 0x7f4ee0197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.627+0000 7f4ee4af4640 1 -- 172.21.15.16:0/3837319835 >> 172.21.15.16:0/3837319835 conn(0x7f4ee00fb370 msgr2=0x7f4ee0104290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:51.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.627+0000 7f4ee4af4640 1 -- 172.21.15.16:0/3837319835 shutdown_connections 2024-03-28T22:09:51.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.627+0000 7f4ee4af4640 1 -- 172.21.15.16:0/3837319835 wait complete. 2024-03-28T22:09:51.639 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T22:09:51.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.740+0000 7ff347fc9640 1 -- 172.21.15.16:0/3710356318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff340101860 msgr2=0x7ff340101ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:51.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.740+0000 7ff347fc9640 1 --2- 172.21.15.16:0/3710356318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff340101860 0x7ff340101ce0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff334009900 tx=0x7ff33402bc30 comp rx=0 tx=0).stop 2024-03-28T22:09:51.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.740+0000 7ff347fc9640 1 -- 172.21.15.16:0/3710356318 shutdown_connections 2024-03-28T22:09:51.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.741+0000 7ff347fc9640 1 --2- 172.21.15.16:0/3710356318 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff340102220 0x7ff340108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.741+0000 7ff347fc9640 1 --2- 172.21.15.16:0/3710356318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff340101860 0x7ff340101ce0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.741+0000 7ff347fc9640 1 --2- 172.21.15.16:0/3710356318 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff340100670 0x7ff340100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.741+0000 7ff347fc9640 1 -- 172.21.15.16:0/3710356318 >> 172.21.15.16:0/3710356318 conn(0x7ff3400fbbe0 msgr2=0x7ff3400fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:51.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.741+0000 7ff347fc9640 1 -- 172.21.15.16:0/3710356318 shutdown_connections 2024-03-28T22:09:51.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.741+0000 7ff347fc9640 1 -- 172.21.15.16:0/3710356318 wait complete. 2024-03-28T22:09:51.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.741+0000 7ff347fc9640 1 Processor -- start 2024-03-28T22:09:51.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.741+0000 7ff347fc9640 1 -- start start 2024-03-28T22:09:51.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.741+0000 7ff347fc9640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff340100670 0x7ff340104600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.741+0000 7ff347fc9640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff340101860 0x7ff340102c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.741+0000 7ff347fc9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff340102220 0x7ff340103180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.741+0000 7ff347fc9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff34010c300 con 0x7ff340102220 2024-03-28T22:09:51.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.741+0000 7ff347fc9640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff34010c180 con 0x7ff340100670 2024-03-28T22:09:51.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.741+0000 7ff347fc9640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff34010c480 con 0x7ff340101860 2024-03-28T22:09:51.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.741+0000 7ff346566640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff340102220 0x7ff340103180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.741+0000 7ff346566640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff340102220 0x7ff340103180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41642/0 (socket says 172.21.15.16:41642) 2024-03-28T22:09:51.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.741+0000 7ff346566640 1 -- 172.21.15.16:0/1344920433 learned_addr learned my addr 172.21.15.16:0/1344920433 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:51.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.742+0000 7ff345d65640 1 --2- 172.21.15.16:0/1344920433 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff340100670 0x7ff340104600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.742+0000 7ff345564640 1 --2- 172.21.15.16:0/1344920433 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff340101860 0x7ff340102c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.742+0000 7ff346566640 1 -- 172.21.15.16:0/1344920433 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff340101860 msgr2=0x7ff340102c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:51.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.742+0000 7ff346566640 1 --2- 172.21.15.16:0/1344920433 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff340101860 0x7ff340102c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.742+0000 7ff346566640 1 -- 172.21.15.16:0/1344920433 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff340100670 msgr2=0x7ff340104600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:51.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.742+0000 7ff346566640 1 --2- 172.21.15.16:0/1344920433 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff340100670 0x7ff340104600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.742+0000 7ff346566640 1 -- 172.21.15.16:0/1344920433 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff340103a10 con 0x7ff340102220 2024-03-28T22:09:51.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.742+0000 7ff346566640 1 --2- 172.21.15.16:0/1344920433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff340102220 0x7ff340103180 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff33000eb10 tx=0x7ff33000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:51.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.742+0000 7ff33effd640 1 -- 172.21.15.16:0/1344920433 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff330004b20 con 0x7ff340102220 2024-03-28T22:09:51.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.742+0000 7ff347fc9640 1 -- 172.21.15.16:0/1344920433 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3401a1c40 con 0x7ff340102220 2024-03-28T22:09:51.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.742+0000 7ff33effd640 1 -- 172.21.15.16:0/1344920433 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff330004cc0 con 0x7ff340102220 2024-03-28T22:09:51.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.742+0000 7ff33effd640 1 -- 172.21.15.16:0/1344920433 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff330010760 con 0x7ff340102220 2024-03-28T22:09:51.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.742+0000 7ff347fc9640 1 -- 172.21.15.16:0/1344920433 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3401a2140 con 0x7ff340102220 2024-03-28T22:09:51.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.744+0000 7ff33effd640 1 -- 172.21.15.16:0/1344920433 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7ff330005020 con 0x7ff340102220 2024-03-28T22:09:51.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.744+0000 7ff34abaa740 1 Processor -- start 2024-03-28T22:09:51.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.745+0000 7ff34abaa740 1 -- start start 2024-03-28T22:09:51.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.745+0000 7ff33effd640 1 --2- 172.21.15.16:0/1344920433 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff3280a1e10 0x7ff3280a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.745+0000 7ff34abaa740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c884fe610 0x560c884fea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.745+0000 7ff34abaa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c884ff7e0 0x560c884ffc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.745+0000 7ff34abaa740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c885001a0 0x560c88506a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.745+0000 7ff34abaa740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560c88462af0 con 0x560c884ff7e0 2024-03-28T22:09:51.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.745+0000 7ff34abaa740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x560c8840b9d0 con 0x560c884fe610 2024-03-28T22:09:51.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.745+0000 7ff34abaa740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x560c8845eb90 con 0x560c885001a0 2024-03-28T22:09:51.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.745+0000 7ff345564640 1 --2- 172.21.15.16:0/1344920433 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff3280a1e10 0x7ff3280a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.745+0000 7ff345d65640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c884fe610 0x560c884fea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.745+0000 7ff33effd640 1 -- 172.21.15.16:0/1344920433 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7ff3300ce630 con 0x7ff340102220 2024-03-28T22:09:51.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.745+0000 7ff345564640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c884ff7e0 0x560c884ffc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.745+0000 7ff345d65640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c884fe610 0x560c884fea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:44290/0 (socket says 172.21.15.16:44290) 2024-03-28T22:09:51.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.745+0000 7ff345d65640 1 -- 172.21.15.16:0/1368456735 learned_addr learned my addr 172.21.15.16:0/1368456735 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:51.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.745+0000 7ff346566640 1 --2- 172.21.15.16:0/1368456735 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c885001a0 0x560c88506a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.745+0000 7ff345d65640 1 -- 172.21.15.16:0/1368456735 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c885001a0 msgr2=0x560c88506a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:09:51.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.745+0000 7ff345d65640 1 --2- 172.21.15.16:0/1368456735 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c885001a0 0x560c88506a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.745+0000 7ff345d65640 1 -- 172.21.15.16:0/1368456735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c884ff7e0 msgr2=0x560c884ffc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:09:51.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.745+0000 7ff345d65640 1 --2- 172.21.15.16:0/1368456735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c884ff7e0 0x560c884ffc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.745+0000 7ff345d65640 1 -- 172.21.15.16:0/1368456735 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c881772d0 con 0x560c884fe610 2024-03-28T22:09:51.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.745+0000 7ff345564640 1 --2- 172.21.15.16:0/1344920433 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff3280a1e10 0x7ff3280a42d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff33402c8e0 tx=0x7ff3340095e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:51.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.746+0000 7ff345d65640 1 --2- 172.21.15.16:0/1368456735 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c884fe610 0x560c884fea30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff33800bcb0 tx=0x7ff3380107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6330dc4cacdc819a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:51.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.746+0000 7ff327fff640 1 -- 172.21.15.16:0/1368456735 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff338005020 con 0x560c884fe610 2024-03-28T22:09:51.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.746+0000 7ff327fff640 1 -- 172.21.15.16:0/1368456735 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3380047a0 con 0x560c884fe610 2024-03-28T22:09:51.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.746+0000 7ff327fff640 1 -- 172.21.15.16:0/1368456735 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff338011750 con 0x560c884fe610 2024-03-28T22:09:51.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.746+0000 7ff34abaa740 1 -- 172.21.15.16:0/1368456735 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c884fe610 msgr2=0x560c884fea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:51.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.746+0000 7ff34abaa740 1 --2- 172.21.15.16:0/1368456735 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c884fe610 0x560c884fea30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff33800bcb0 tx=0x7ff3380107a0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.746+0000 7ff34abaa740 1 -- 172.21.15.16:0/1368456735 shutdown_connections 2024-03-28T22:09:51.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.746+0000 7ff34abaa740 1 --2- 172.21.15.16:0/1368456735 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c885001a0 0x560c88506a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.746+0000 7ff34abaa740 1 --2- 172.21.15.16:0/1368456735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c884ff7e0 0x560c884ffc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.746+0000 7ff34abaa740 1 --2- 172.21.15.16:0/1368456735 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c884fe610 0x560c884fea30 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.746+0000 7ff34abaa740 1 -- 172.21.15.16:0/1368456735 >> 172.21.15.16:0/1368456735 conn(0x560c884f9a00 msgr2=0x560c884f9e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:51.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.747+0000 7ff34abaa740 1 -- 172.21.15.16:0/1368456735 shutdown_connections 2024-03-28T22:09:51.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.747+0000 7ff34abaa740 1 -- 172.21.15.16:0/1368456735 wait complete. 2024-03-28T22:09:51.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.747+0000 7ff34abaa740 1 -- start start 2024-03-28T22:09:51.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.747+0000 7ff34abaa740 1 Processor -- start 2024-03-28T22:09:51.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.747+0000 7ff34abaa740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c884fe610 0x560c88508920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.747+0000 7ff34abaa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c884ff7e0 0x560c88508e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.747+0000 7ff34abaa740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c885001a0 0x560c8850fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.747+0000 7ff34abaa740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560c885071b0 con 0x560c884ff7e0 2024-03-28T22:09:51.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.747+0000 7ff34abaa740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x560c8841ef70 con 0x560c884fe610 2024-03-28T22:09:51.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.747+0000 7ff34abaa740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x560c88507330 con 0x560c885001a0 2024-03-28T22:09:51.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.747+0000 7ff34abaa740 10 client.0 _ll_register_callbacks cb 0x560c88176ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:09:51.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.747+0000 7ff345564640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c884ff7e0 0x560c88508e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.747+0000 7ff345564640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c884ff7e0 0x560c88508e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41654/0 (socket says 172.21.15.16:41654) 2024-03-28T22:09:51.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.747+0000 7ff345564640 1 -- 172.21.15.16:0/931487324 learned_addr learned my addr 172.21.15.16:0/931487324 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:51.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.748+0000 7ff345d65640 1 --2- 172.21.15.16:0/931487324 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c884fe610 0x560c88508920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.748+0000 7ff346566640 1 --2- 172.21.15.16:0/931487324 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c885001a0 0x560c8850fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.748+0000 7ff345564640 1 -- 172.21.15.16:0/931487324 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c885001a0 msgr2=0x560c8850fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:09:51.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.748+0000 7ff345564640 1 --2- 172.21.15.16:0/931487324 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c885001a0 0x560c8850fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.748+0000 7ff345564640 1 -- 172.21.15.16:0/931487324 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c884fe610 msgr2=0x560c88508920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:09:51.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.748+0000 7ff345564640 1 --2- 172.21.15.16:0/931487324 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c884fe610 0x560c88508920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.748+0000 7ff345564640 1 -- 172.21.15.16:0/931487324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c882416b0 con 0x560c884ff7e0 2024-03-28T22:09:51.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.748+0000 7ff346566640 1 --2- 172.21.15.16:0/931487324 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c885001a0 0x560c8850fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:09:51.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.748+0000 7ff345564640 1 --2- 172.21.15.16:0/931487324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c884ff7e0 0x560c88508e60 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff334035000 tx=0x7ff334005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e01d19b72c95cbeb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:51.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.748+0000 7ff3257fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:09:51.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.748+0000 7ff3257fa640 1 -- 172.21.15.16:0/931487324 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff334054060 con 0x560c884ff7e0 2024-03-28T22:09:51.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.748+0000 7ff3257fa640 1 -- 172.21.15.16:0/931487324 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7ff334003a70 con 0x560c884ff7e0 2024-03-28T22:09:51.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.748+0000 7ff34abaa740 1 -- 172.21.15.16:0/931487324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3380047a0 con 0x560c884ff7e0 2024-03-28T22:09:51.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.748+0000 7ff3257fa640 1 -- 172.21.15.16:0/931487324 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7ff334003c70 con 0x560c884ff7e0 2024-03-28T22:09:51.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.748+0000 7ff34abaa740 10 client.15081 fetch_fsmap learned FSMap version 20 2024-03-28T22:09:51.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.748+0000 7ff34abaa740 1 -- 172.21.15.16:0/931487324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x560c88512ef0 con 0x560c884ff7e0 2024-03-28T22:09:51.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.748+0000 7ff3257fa640 1 -- 172.21.15.16:0/931487324 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3380047a0 con 0x560c884ff7e0 2024-03-28T22:09:51.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.749+0000 7ff3257fa640 1 -- 172.21.15.16:0/931487324 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7ff334043450 con 0x560c884ff7e0 2024-03-28T22:09:51.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.749+0000 7ff34abaa740 10 client.15081 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T22:09:51.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.749+0000 7ff34abaa740 10 client.15081 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T22:09:51.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.749+0000 7ff34abaa740 20 client.15081 populate_metadata read hostname 'smithi016' 2024-03-28T22:09:51.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.749+0000 7ff34abaa740 1 --2- 172.21.15.16:0/931487324 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x560c88515600 0x560c88516a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.749+0000 7ff34abaa740 4 client.15081 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:09:51.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.749+0000 7ff34abaa740 1 -- 172.21.15.16:0/931487324 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560c88470030 con 0x560c88515600 2024-03-28T22:09:51.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.749+0000 7ff345d65640 1 --2- 172.21.15.16:0/931487324 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x560c88515600 0x560c88516a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.749+0000 7ff345d65640 1 --2- 172.21.15.16:0/931487324 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x560c88515600 0x560c88516a70 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd5963b679e10c43 server_cookie=b7595c67ba22c49 in_seq=0 out_seq=0 2024-03-28T22:09:51.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.749+0000 7ff3257fa640 10 client.15081 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:09:51.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.750+0000 7ff3257fa640 1 -- 172.21.15.16:0/931487324 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560c88470030 con 0x560c88515600 2024-03-28T22:09:51.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.750+0000 7ff3257fa640 10 client.15081 handle_command_reply: tid=0 2024-03-28T22:09:51.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.750+0000 7ff34abaa740 1 client.15081 shutdown 2024-03-28T22:09:51.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.750+0000 7ff34abaa740 1 -- 172.21.15.16:0/931487324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c884ff7e0 msgr2=0x560c88508e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:51.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.750+0000 7ff34abaa740 1 --2- 172.21.15.16:0/931487324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c884ff7e0 0x560c88508e60 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff334035000 tx=0x7ff334005fd0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.750+0000 7ff34abaa740 1 -- 172.21.15.16:0/931487324 shutdown_connections 2024-03-28T22:09:51.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.750+0000 7ff34abaa740 1 --2- 172.21.15.16:0/931487324 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x560c88515600 0x560c88516a70 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.750+0000 7ff34abaa740 1 --2- 172.21.15.16:0/931487324 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c885001a0 0x560c8850fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.750+0000 7ff34abaa740 1 --2- 172.21.15.16:0/931487324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c884ff7e0 0x560c88508e60 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.750+0000 7ff3257fa640 0 client.15081 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:09:51.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.750+0000 7ff34abaa740 1 --2- 172.21.15.16:0/931487324 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c884fe610 0x560c88508920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.750+0000 7ff34abaa740 1 -- 172.21.15.16:0/931487324 >> 172.21.15.16:0/931487324 conn(0x560c884f9a00 msgr2=0x560c884f8b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:51.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.750+0000 7ff34abaa740 1 -- 172.21.15.16:0/931487324 shutdown_connections 2024-03-28T22:09:51.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.750+0000 7ff34abaa740 1 -- 172.21.15.16:0/931487324 wait complete. 2024-03-28T22:09:51.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.750+0000 7ff34abaa740 20 client.15081 trim_cache size 0 max 16384 2024-03-28T22:09:51.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.768+0000 7ff34abaa740 1 Processor -- start 2024-03-28T22:09:51.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.768+0000 7ff34abaa740 1 -- start start 2024-03-28T22:09:51.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.768+0000 7ff34abaa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c884fe610 0x560c884f9de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.768+0000 7ff34abaa740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c884ff7e0 0x560c8850fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.768+0000 7ff34abaa740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c885001a0 0x560c88503140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.768+0000 7ff34abaa740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560c88514db0 con 0x560c884fe610 2024-03-28T22:09:51.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.768+0000 7ff34abaa740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x560c88515480 con 0x560c885001a0 2024-03-28T22:09:51.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.768+0000 7ff34abaa740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x560c88515b50 con 0x560c884ff7e0 2024-03-28T22:09:51.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.768+0000 7ff345d65640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c884fe610 0x560c884f9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.768+0000 7ff345d65640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c884fe610 0x560c884f9de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41658/0 (socket says 172.21.15.16:41658) 2024-03-28T22:09:51.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.768+0000 7ff345d65640 1 -- 172.21.15.16:0/2705723674 learned_addr learned my addr 172.21.15.16:0/2705723674 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:51.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.769+0000 7ff346566640 1 --2- 172.21.15.16:0/2705723674 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c885001a0 0x560c88503140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.769+0000 7ff345564640 1 --2- 172.21.15.16:0/2705723674 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c884ff7e0 0x560c8850fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.769+0000 7ff345d65640 1 -- 172.21.15.16:0/2705723674 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c884ff7e0 msgr2=0x560c8850fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:09:51.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.769+0000 7ff345d65640 1 --2- 172.21.15.16:0/2705723674 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c884ff7e0 0x560c8850fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.769+0000 7ff345d65640 1 -- 172.21.15.16:0/2705723674 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c885001a0 msgr2=0x560c88503140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:09:51.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.769+0000 7ff345d65640 1 --2- 172.21.15.16:0/2705723674 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c885001a0 0x560c88503140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.769+0000 7ff345d65640 1 -- 172.21.15.16:0/2705723674 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c884ff260 con 0x560c884fe610 2024-03-28T22:09:51.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.769+0000 7ff346566640 1 --2- 172.21.15.16:0/2705723674 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c885001a0 0x560c88503140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:09:51.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.769+0000 7ff345d65640 1 --2- 172.21.15.16:0/2705723674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c884fe610 0x560c884f9de0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff338010f90 tx=0x7ff338013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaa521e55416b76c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:51.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.769+0000 7ff3277fe640 1 -- 172.21.15.16:0/2705723674 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff33802d070 con 0x560c884fe610 2024-03-28T22:09:51.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.769+0000 7ff3277fe640 1 -- 172.21.15.16:0/2705723674 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff338028da0 con 0x560c884fe610 2024-03-28T22:09:51.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.769+0000 7ff3277fe640 1 -- 172.21.15.16:0/2705723674 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff338028600 con 0x560c884fe610 2024-03-28T22:09:51.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.769+0000 7ff34abaa740 1 -- 172.21.15.16:0/2705723674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c884fe610 msgr2=0x560c884f9de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:51.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.769+0000 7ff34abaa740 1 --2- 172.21.15.16:0/2705723674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c884fe610 0x560c884f9de0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff338010f90 tx=0x7ff338013040 comp rx=0 tx=0).stop 2024-03-28T22:09:51.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.770+0000 7ff34abaa740 1 -- 172.21.15.16:0/2705723674 shutdown_connections 2024-03-28T22:09:51.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.770+0000 7ff34abaa740 1 --2- 172.21.15.16:0/2705723674 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c885001a0 0x560c88503140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.770+0000 7ff34abaa740 1 --2- 172.21.15.16:0/2705723674 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c884ff7e0 0x560c8850fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.770+0000 7ff34abaa740 1 --2- 172.21.15.16:0/2705723674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c884fe610 0x560c884f9de0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.770+0000 7ff34abaa740 1 -- 172.21.15.16:0/2705723674 >> 172.21.15.16:0/2705723674 conn(0x560c884f9a00 msgr2=0x560c88506710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:51.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.770+0000 7ff34abaa740 1 -- 172.21.15.16:0/2705723674 shutdown_connections 2024-03-28T22:09:51.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.770+0000 7ff34abaa740 1 -- 172.21.15.16:0/2705723674 wait complete. 2024-03-28T22:09:51.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.770+0000 7ff34abaa740 1 -- start start 2024-03-28T22:09:51.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.770+0000 7ff34abaa740 1 Processor -- start 2024-03-28T22:09:51.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.770+0000 7ff34abaa740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c884fe610 0x560c8850c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.770+0000 7ff34abaa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c884ff7e0 0x560c8850c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.770+0000 7ff34abaa740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c885001a0 0x560c8850eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.770+0000 7ff34abaa740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560c885168f0 con 0x560c884ff7e0 2024-03-28T22:09:51.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.770+0000 7ff34abaa740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x560c88516220 con 0x560c885001a0 2024-03-28T22:09:51.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.770+0000 7ff34abaa740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x560c88516fc0 con 0x560c884fe610 2024-03-28T22:09:51.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.770+0000 7ff34abaa740 10 client.0 _ll_register_callbacks cb 0x7ff334009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:09:51.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.770+0000 7ff345564640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c884ff7e0 0x560c8850c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.770+0000 7ff345564640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c884ff7e0 0x560c8850c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41670/0 (socket says 172.21.15.16:41670) 2024-03-28T22:09:51.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.770+0000 7ff345564640 1 -- 172.21.15.16:0/277925654 learned_addr learned my addr 172.21.15.16:0/277925654 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:51.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.770+0000 7ff345d65640 1 --2- 172.21.15.16:0/277925654 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c884fe610 0x560c8850c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.770+0000 7ff346566640 1 --2- 172.21.15.16:0/277925654 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c885001a0 0x560c8850eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.771+0000 7ff345564640 1 -- 172.21.15.16:0/277925654 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c884fe610 msgr2=0x560c8850c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:09:51.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.771+0000 7ff345564640 1 --2- 172.21.15.16:0/277925654 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c884fe610 0x560c8850c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.771+0000 7ff345564640 1 -- 172.21.15.16:0/277925654 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c885001a0 msgr2=0x560c8850eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:09:51.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.771+0000 7ff345564640 1 --2- 172.21.15.16:0/277925654 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c885001a0 0x560c8850eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.771+0000 7ff345564640 1 -- 172.21.15.16:0/277925654 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c8850f490 con 0x560c884ff7e0 2024-03-28T22:09:51.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.771+0000 7ff345564640 1 --2- 172.21.15.16:0/277925654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c884ff7e0 0x560c8850c5d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff334009850 tx=0x7ff334004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d38a3c5ae2db6cc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:51.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.771+0000 7ff3257fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:09:51.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.771+0000 7ff3257fa640 1 -- 172.21.15.16:0/277925654 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff33405f030 con 0x560c884ff7e0 2024-03-28T22:09:51.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.771+0000 7ff3257fa640 1 -- 172.21.15.16:0/277925654 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7ff334043450 con 0x560c884ff7e0 2024-03-28T22:09:51.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.771+0000 7ff34abaa740 1 -- 172.21.15.16:0/277925654 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff338028da0 con 0x560c884ff7e0 2024-03-28T22:09:51.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.771+0000 7ff3257fa640 1 -- 172.21.15.16:0/277925654 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7ff334054040 con 0x560c884ff7e0 2024-03-28T22:09:51.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.771+0000 7ff34abaa740 10 client.15087 fetch_fsmap learned FSMap version 20 2024-03-28T22:09:51.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.771+0000 7ff34abaa740 1 -- 172.21.15.16:0/277925654 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x560c8857b710 con 0x560c884ff7e0 2024-03-28T22:09:51.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.771+0000 7ff3257fa640 1 -- 172.21.15.16:0/277925654 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff338028da0 con 0x560c884ff7e0 2024-03-28T22:09:51.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.771+0000 7ff3257fa640 1 -- 172.21.15.16:0/277925654 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7ff334034440 con 0x560c884ff7e0 2024-03-28T22:09:51.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.772+0000 7ff34abaa740 10 client.15087 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T22:09:51.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.772+0000 7ff34abaa740 10 client.15087 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T22:09:51.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.772+0000 7ff34abaa740 20 client.15087 populate_metadata read hostname 'smithi016' 2024-03-28T22:09:51.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.772+0000 7ff34abaa740 1 --2- 172.21.15.16:0/277925654 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x560c88580330 0x560c8857ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:51.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.772+0000 7ff34abaa740 4 client.15087 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T22:09:51.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.772+0000 7ff34abaa740 1 -- 172.21.15.16:0/277925654 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "status"}) -- 0x560c8846a380 con 0x560c88580330 2024-03-28T22:09:51.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.772+0000 7ff345d65640 1 --2- 172.21.15.16:0/277925654 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x560c88580330 0x560c8857ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:51.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.772+0000 7ff345d65640 1 --2- 172.21.15.16:0/277925654 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x560c88580330 0x560c8857ce40 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dcb62ad6049756be server_cookie=71374b9685ab9961 in_seq=0 out_seq=0 2024-03-28T22:09:51.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.772+0000 7ff3257fa640 10 client.15087 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:09:51.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.772+0000 7ff3257fa640 1 -- 172.21.15.16:0/277925654 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x560c8846a380 con 0x560c88580330 2024-03-28T22:09:51.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.772+0000 7ff3257fa640 10 client.15087 handle_command_reply: tid=0 2024-03-28T22:09:51.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.772+0000 7ff34abaa740 1 client.15087 shutdown 2024-03-28T22:09:51.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.772+0000 7ff34abaa740 1 -- 172.21.15.16:0/277925654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c884ff7e0 msgr2=0x560c8850c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:51.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.772+0000 7ff34abaa740 1 --2- 172.21.15.16:0/277925654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c884ff7e0 0x560c8850c5d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff334009850 tx=0x7ff334004040 comp rx=0 tx=0).stop 2024-03-28T22:09:51.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.772+0000 7ff34abaa740 1 -- 172.21.15.16:0/277925654 shutdown_connections 2024-03-28T22:09:51.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.772+0000 7ff34abaa740 1 --2- 172.21.15.16:0/277925654 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x560c88580330 0x560c8857ce40 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.772+0000 7ff34abaa740 1 --2- 172.21.15.16:0/277925654 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c885001a0 0x560c8850eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.772+0000 7ff34abaa740 1 --2- 172.21.15.16:0/277925654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c884ff7e0 0x560c8850c5d0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.772+0000 7ff3257fa640 0 client.15087 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:09:51.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.772+0000 7ff34abaa740 1 --2- 172.21.15.16:0/277925654 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c884fe610 0x560c8850c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.773+0000 7ff34abaa740 1 -- 172.21.15.16:0/277925654 >> 172.21.15.16:0/277925654 conn(0x560c884f9a00 msgr2=0x560c885143b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:51.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.773+0000 7ff34abaa740 1 -- 172.21.15.16:0/277925654 shutdown_connections 2024-03-28T22:09:51.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.773+0000 7ff34abaa740 1 -- 172.21.15.16:0/277925654 wait complete. 2024-03-28T22:09:51.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.773+0000 7ff34abaa740 20 client.15087 trim_cache size 0 max 16384 2024-03-28T22:09:51.781 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:09:51.781 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "798d9884-ed4e-11ee-95d0-87774f69a715", 2024-03-28T22:09:51.781 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2024-03-28T22:09:51.781 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24566, 2024-03-28T22:09:51.782 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-03-28T22:09:51.782 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-03-28T22:09:51.782 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-03-28T22:09:51.782 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 81.458645751999995, 2024-03-28T22:09:51.782 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 20, 2024-03-28T22:09:51.782 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 88, 2024-03-28T22:09:51.782 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 86, 2024-03-28T22:09:51.782 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 86.857863593000005 2024-03-28T22:09:51.782 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:09:51.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.773+0000 7ff347fc9640 1 -- 172.21.15.16:0/1344920433 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff3280a1e10 msgr2=0x7ff3280a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:51.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.773+0000 7ff347fc9640 1 --2- 172.21.15.16:0/1344920433 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff3280a1e10 0x7ff3280a42d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff33402c8e0 tx=0x7ff3340095e0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.773+0000 7ff347fc9640 1 -- 172.21.15.16:0/1344920433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff340102220 msgr2=0x7ff340103180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:51.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.773+0000 7ff347fc9640 1 --2- 172.21.15.16:0/1344920433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff340102220 0x7ff340103180 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff33000eb10 tx=0x7ff33000c2d0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.773+0000 7ff347fc9640 1 -- 172.21.15.16:0/1344920433 shutdown_connections 2024-03-28T22:09:51.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.773+0000 7ff347fc9640 1 --2- 172.21.15.16:0/1344920433 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff3280a1e10 0x7ff3280a42d0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.773+0000 7ff347fc9640 1 --2- 172.21.15.16:0/1344920433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff340102220 0x7ff340103180 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.773+0000 7ff347fc9640 1 --2- 172.21.15.16:0/1344920433 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff340101860 0x7ff340102c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.773+0000 7ff347fc9640 1 --2- 172.21.15.16:0/1344920433 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff340100670 0x7ff340104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:51.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.773+0000 7ff347fc9640 1 -- 172.21.15.16:0/1344920433 >> 172.21.15.16:0/1344920433 conn(0x7ff3400fbbe0 msgr2=0x7ff3400fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:51.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.773+0000 7ff347fc9640 1 -- 172.21.15.16:0/1344920433 shutdown_connections 2024-03-28T22:09:51.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:51.773+0000 7ff347fc9640 1 -- 172.21.15.16:0/1344920433 wait complete. 2024-03-28T22:09:51.786 INFO:tasks.cephadm:Setup complete, yielding 2024-03-28T22:09:51.786 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-28T22:09:51.795 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-03-28T22:09:51.796 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- bash -c 'ceph orch status' 2024-03-28T22:09:52.037 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:09:52.281 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:52 smithi016 ceph-mon[32374]: pgmap v329: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T22:09:52.281 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:52 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/1115610658' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:09:52.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:52 smithi053 ceph-mon[35979]: pgmap v329: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T22:09:52.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:52 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/1115610658' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:09:52.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:52 smithi063 ceph-mon[36249]: pgmap v329: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T22:09:52.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:52 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/1115610658' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:09:53.359 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:53 smithi016 ceph-mon[32374]: pgmap v330: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T22:09:53.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:53 smithi063 ceph-mon[36249]: pgmap v330: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T22:09:53.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:53 smithi053 ceph-mon[35979]: pgmap v330: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-28T22:09:53.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.575+0000 7f75eacd6700 1 -- 172.21.15.16:0/4004054724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75e4102d10 msgr2=0x7f75e41095c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:53.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.575+0000 7f75eacd6700 1 --2- 172.21.15.16:0/4004054724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75e4102d10 0x7f75e41095c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f75dc00b0d0 tx=0x7f75dc00b3e0 comp rx=0 tx=0).stop 2024-03-28T22:09:53.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.575+0000 7f75eacd6700 1 -- 172.21.15.16:0/4004054724 shutdown_connections 2024-03-28T22:09:53.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.575+0000 7f75eacd6700 1 --2- 172.21.15.16:0/4004054724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75e4102d10 0x7f75e41095c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:53.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.575+0000 7f75eacd6700 1 --2- 172.21.15.16:0/4004054724 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75e4101170 0x7f75e4101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:53.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.575+0000 7f75eacd6700 1 --2- 172.21.15.16:0/4004054724 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75e4102370 0x7f75e41027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:53.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.575+0000 7f75eacd6700 1 -- 172.21.15.16:0/4004054724 >> 172.21.15.16:0/4004054724 conn(0x7f75e40fc710 msgr2=0x7f75e40feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:53.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.575+0000 7f75eacd6700 1 -- 172.21.15.16:0/4004054724 shutdown_connections 2024-03-28T22:09:53.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.575+0000 7f75eacd6700 1 -- 172.21.15.16:0/4004054724 wait complete. 2024-03-28T22:09:53.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.576+0000 7f75eacd6700 1 Processor -- start 2024-03-28T22:09:53.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.576+0000 7f75eacd6700 1 -- start start 2024-03-28T22:09:53.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.576+0000 7f75eacd6700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75e4101170 0x7f75e4197380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:53.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.576+0000 7f75eacd6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75e4102370 0x7f75e41978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:53.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.576+0000 7f75eacd6700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75e4102d10 0x7f75e419e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:53.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.576+0000 7f75eacd6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f75e410cda0 con 0x7f75e4102370 2024-03-28T22:09:53.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.576+0000 7f75eacd6700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f75e410cc20 con 0x7f75e4102d10 2024-03-28T22:09:53.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.576+0000 7f75eacd6700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f75e410cf20 con 0x7f75e4101170 2024-03-28T22:09:53.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.576+0000 7f75e3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75e4102370 0x7f75e41978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:53.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.576+0000 7f75e3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75e4102370 0x7f75e41978c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41692/0 (socket says 172.21.15.16:41692) 2024-03-28T22:09:53.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.576+0000 7f75e3fff700 1 -- 172.21.15.16:0/4063438699 learned_addr learned my addr 172.21.15.16:0/4063438699 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:53.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.576+0000 7f75e9273700 1 --2- 172.21.15.16:0/4063438699 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75e4102d10 0x7f75e419e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:53.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.576+0000 7f75e8a72700 1 --2- 172.21.15.16:0/4063438699 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75e4101170 0x7f75e4197380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:53.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.577+0000 7f75e3fff700 1 -- 172.21.15.16:0/4063438699 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75e4101170 msgr2=0x7f75e4197380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:53.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.577+0000 7f75e3fff700 1 --2- 172.21.15.16:0/4063438699 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75e4101170 0x7f75e4197380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:53.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.577+0000 7f75e3fff700 1 -- 172.21.15.16:0/4063438699 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75e4102d10 msgr2=0x7f75e419e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:53.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.577+0000 7f75e3fff700 1 --2- 172.21.15.16:0/4063438699 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75e4102d10 0x7f75e419e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:53.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.577+0000 7f75e3fff700 1 -- 172.21.15.16:0/4063438699 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75e419efe0 con 0x7f75e4102370 2024-03-28T22:09:53.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.577+0000 7f75e8a72700 1 --2- 172.21.15.16:0/4063438699 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75e4101170 0x7f75e4197380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:09:53.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.577+0000 7f75e9273700 1 --2- 172.21.15.16:0/4063438699 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75e4102d10 0x7f75e419e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:09:53.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.577+0000 7f75e3fff700 1 --2- 172.21.15.16:0/4063438699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75e4102370 0x7f75e41978c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f75d000c950 tx=0x7f75d000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:53.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.577+0000 7f75e1ffb700 1 -- 172.21.15.16:0/4063438699 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d0007ac0 con 0x7f75e4102370 2024-03-28T22:09:53.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.577+0000 7f75e1ffb700 1 -- 172.21.15.16:0/4063438699 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75d0004d10 con 0x7f75e4102370 2024-03-28T22:09:53.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.577+0000 7f75eacd6700 1 -- 172.21.15.16:0/4063438699 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75e419f2d0 con 0x7f75e4102370 2024-03-28T22:09:53.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.577+0000 7f75e1ffb700 1 -- 172.21.15.16:0/4063438699 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d000f450 con 0x7f75e4102370 2024-03-28T22:09:53.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.577+0000 7f75eacd6700 1 -- 172.21.15.16:0/4063438699 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75e419f810 con 0x7f75e4102370 2024-03-28T22:09:53.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.578+0000 7f75eacd6700 1 -- 172.21.15.16:0/4063438699 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75e4068f00 con 0x7f75e4102370 2024-03-28T22:09:53.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.579+0000 7f75e1ffb700 1 -- 172.21.15.16:0/4063438699 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f75d0004750 con 0x7f75e4102370 2024-03-28T22:09:53.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.580+0000 7f75e1ffb700 1 --2- 172.21.15.16:0/4063438699 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f75d40a1ea0 0x7f75d40a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:53.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.580+0000 7f75e8a72700 1 --2- 172.21.15.16:0/4063438699 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f75d40a1ea0 0x7f75d40a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:53.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.580+0000 7f75e1ffb700 1 -- 172.21.15.16:0/4063438699 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f75d00cddf0 con 0x7f75e4102370 2024-03-28T22:09:53.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.580+0000 7f75e8a72700 1 --2- 172.21.15.16:0/4063438699 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f75d40a1ea0 0x7f75d40a4360 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f75e41021d0 tx=0x7f75d8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:53.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.585+0000 7f75e1ffb700 1 -- 172.21.15.16:0/4063438699 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f75d0014030 con 0x7f75e4102370 2024-03-28T22:09:53.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.808+0000 7f75eacd6700 1 -- 172.21.15.16:0/4063438699 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f75e4062680 con 0x7f75d40a1ea0 2024-03-28T22:09:53.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.809+0000 7f75e1ffb700 1 -- 172.21.15.16:0/4063438699 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f75e4062680 con 0x7f75d40a1ea0 2024-03-28T22:09:53.811 INFO:teuthology.orchestra.run.smithi016.stdout:Backend: cephadm 2024-03-28T22:09:53.811 INFO:teuthology.orchestra.run.smithi016.stdout:Available: Yes 2024-03-28T22:09:53.811 INFO:teuthology.orchestra.run.smithi016.stdout:Paused: No 2024-03-28T22:09:53.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.811+0000 7f75eacd6700 1 -- 172.21.15.16:0/4063438699 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f75d40a1ea0 msgr2=0x7f75d40a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:53.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.811+0000 7f75eacd6700 1 --2- 172.21.15.16:0/4063438699 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f75d40a1ea0 0x7f75d40a4360 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f75e41021d0 tx=0x7f75d8009260 comp rx=0 tx=0).stop 2024-03-28T22:09:53.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.811+0000 7f75eacd6700 1 -- 172.21.15.16:0/4063438699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75e4102370 msgr2=0x7f75e41978c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:53.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.811+0000 7f75eacd6700 1 --2- 172.21.15.16:0/4063438699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75e4102370 0x7f75e41978c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f75d000c950 tx=0x7f75d000cc60 comp rx=0 tx=0).stop 2024-03-28T22:09:53.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.812+0000 7f75eacd6700 1 -- 172.21.15.16:0/4063438699 shutdown_connections 2024-03-28T22:09:53.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.812+0000 7f75eacd6700 1 --2- 172.21.15.16:0/4063438699 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f75d40a1ea0 0x7f75d40a4360 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:53.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.812+0000 7f75eacd6700 1 --2- 172.21.15.16:0/4063438699 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75e4102d10 0x7f75e419e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:53.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.812+0000 7f75eacd6700 1 --2- 172.21.15.16:0/4063438699 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75e4101170 0x7f75e4197380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:53.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.812+0000 7f75eacd6700 1 --2- 172.21.15.16:0/4063438699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75e4102370 0x7f75e41978c0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:53.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.812+0000 7f75eacd6700 1 -- 172.21.15.16:0/4063438699 >> 172.21.15.16:0/4063438699 conn(0x7f75e40fc710 msgr2=0x7f75e4105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:53.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.812+0000 7f75eacd6700 1 -- 172.21.15.16:0/4063438699 shutdown_connections 2024-03-28T22:09:53.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:53.812+0000 7f75eacd6700 1 -- 172.21.15.16:0/4063438699 wait complete. 2024-03-28T22:09:54.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:54 smithi063 ceph-mon[36249]: from='client.15093 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:09:54.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:54 smithi016 ceph-mon[32374]: from='client.15093 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:09:54.489 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-28T22:09:54.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:54 smithi053 ceph-mon[35979]: from='client.15093 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:09:54.738 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:09:55.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:55 smithi063 ceph-mon[36249]: pgmap v331: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T22:09:55.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:55 smithi016 ceph-mon[32374]: pgmap v331: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T22:09:55.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:55 smithi053 ceph-mon[35979]: pgmap v331: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T22:09:56.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.423+0000 7ff02ce26700 1 -- 172.21.15.16:0/4029432049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff028102d00 msgr2=0x7ff0281050f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:56.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.423+0000 7ff02ce26700 1 --2- 172.21.15.16:0/4029432049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff028102d00 0x7ff0281050f0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff01800b0a0 tx=0x7ff01800b3b0 comp rx=0 tx=0).stop 2024-03-28T22:09:56.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.423+0000 7ff02ce26700 1 -- 172.21.15.16:0/4029432049 shutdown_connections 2024-03-28T22:09:56.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.423+0000 7ff02ce26700 1 --2- 172.21.15.16:0/4029432049 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff028107f60 0x7ff02810a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:56.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.423+0000 7ff02ce26700 1 --2- 172.21.15.16:0/4029432049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff028102d00 0x7ff0281050f0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:56.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.423+0000 7ff02ce26700 1 --2- 172.21.15.16:0/4029432049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff028105630 0x7ff028107a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:56.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.423+0000 7ff02ce26700 1 -- 172.21.15.16:0/4029432049 >> 172.21.15.16:0/4029432049 conn(0x7ff0280fc710 msgr2=0x7ff0280feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:56.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.423+0000 7ff02ce26700 1 -- 172.21.15.16:0/4029432049 shutdown_connections 2024-03-28T22:09:56.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.423+0000 7ff02ce26700 1 -- 172.21.15.16:0/4029432049 wait complete. 2024-03-28T22:09:56.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.424+0000 7ff02ce26700 1 Processor -- start 2024-03-28T22:09:56.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.424+0000 7ff02ce26700 1 -- start start 2024-03-28T22:09:56.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.424+0000 7ff02ce26700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff028102d00 0x7ff028198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:56.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.424+0000 7ff02ce26700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff028105630 0x7ff028198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:56.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.424+0000 7ff02ce26700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff028107f60 0x7ff02819fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:56.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.424+0000 7ff02ce26700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff02810de50 con 0x7ff028102d00 2024-03-28T22:09:56.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.424+0000 7ff02ce26700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff02810dcd0 con 0x7ff028107f60 2024-03-28T22:09:56.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.424+0000 7ff02ce26700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff02810dfd0 con 0x7ff028105630 2024-03-28T22:09:56.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.424+0000 7ff02659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff028102d00 0x7ff028198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:56.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.424+0000 7ff02659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff028102d00 0x7ff028198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41710/0 (socket says 172.21.15.16:41710) 2024-03-28T22:09:56.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.424+0000 7ff02659c700 1 -- 172.21.15.16:0/769659563 learned_addr learned my addr 172.21.15.16:0/769659563 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:56.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.424+0000 7ff026d9d700 1 --2- 172.21.15.16:0/769659563 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff028107f60 0x7ff02819fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:56.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.424+0000 7ff025d9b700 1 --2- 172.21.15.16:0/769659563 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff028105630 0x7ff028198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:56.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.424+0000 7ff02659c700 1 -- 172.21.15.16:0/769659563 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff028105630 msgr2=0x7ff028198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:56.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.424+0000 7ff02659c700 1 --2- 172.21.15.16:0/769659563 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff028105630 0x7ff028198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:56.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.424+0000 7ff02659c700 1 -- 172.21.15.16:0/769659563 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff028107f60 msgr2=0x7ff02819fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:56.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.424+0000 7ff02659c700 1 --2- 172.21.15.16:0/769659563 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff028107f60 0x7ff02819fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:56.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.425+0000 7ff02659c700 1 -- 172.21.15.16:0/769659563 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0281a0180 con 0x7ff028102d00 2024-03-28T22:09:56.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.425+0000 7ff02659c700 1 --2- 172.21.15.16:0/769659563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff028102d00 0x7ff028198520 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff018004d20 tx=0x7ff018004d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:56.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.425+0000 7ff01f7fe700 1 -- 172.21.15.16:0/769659563 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff018021070 con 0x7ff028102d00 2024-03-28T22:09:56.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.425+0000 7ff02ce26700 1 -- 172.21.15.16:0/769659563 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0281a0410 con 0x7ff028102d00 2024-03-28T22:09:56.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.425+0000 7ff01f7fe700 1 -- 172.21.15.16:0/769659563 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff018007cc0 con 0x7ff028102d00 2024-03-28T22:09:56.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.425+0000 7ff01f7fe700 1 -- 172.21.15.16:0/769659563 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0180054a0 con 0x7ff028102d00 2024-03-28T22:09:56.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.425+0000 7ff02ce26700 1 -- 172.21.15.16:0/769659563 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0281a0910 con 0x7ff028102d00 2024-03-28T22:09:56.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.426+0000 7ff02ce26700 1 -- 172.21.15.16:0/769659563 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff028068f00 con 0x7ff028102d00 2024-03-28T22:09:56.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.427+0000 7ff01f7fe700 1 -- 172.21.15.16:0/769659563 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7ff018024070 con 0x7ff028102d00 2024-03-28T22:09:56.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.427+0000 7ff01f7fe700 1 --2- 172.21.15.16:0/769659563 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff0140a1f20 0x7ff0140a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:56.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.428+0000 7ff025d9b700 1 --2- 172.21.15.16:0/769659563 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff0140a1f20 0x7ff0140a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:56.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.428+0000 7ff01f7fe700 1 -- 172.21.15.16:0/769659563 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7ff01800f020 con 0x7ff028102d00 2024-03-28T22:09:56.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.428+0000 7ff025d9b700 1 --2- 172.21.15.16:0/769659563 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff0140a1f20 0x7ff0140a43e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff010009bf0 tx=0x7ff010009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:56.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.431+0000 7ff01f7fe700 1 -- 172.21.15.16:0/769659563 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff01802b020 con 0x7ff028102d00 2024-03-28T22:09:56.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.645+0000 7ff02ce26700 1 -- 172.21.15.16:0/769659563 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7ff028062680 con 0x7ff0140a1f20 2024-03-28T22:09:56.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.659+0000 7ff01f7fe700 1 -- 172.21.15.16:0/769659563 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7ff028062680 con 0x7ff0140a1f20 2024-03-28T22:09:56.661 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-28T22:09:56.661 INFO:teuthology.orchestra.run.smithi016.stdout:mds.a smithi016 running (95s) 15s ago 94s 16.1M - 19.0.0-1899-g323d23dc bed8e21b47c6 6cf1ec98898e 2024-03-28T22:09:56.661 INFO:teuthology.orchestra.run.smithi016.stdout:mds.b smithi053 running (92s) 25s ago 91s 28.4M - 19.0.0-1899-g323d23dc bed8e21b47c6 19904610fabe 2024-03-28T22:09:56.661 INFO:teuthology.orchestra.run.smithi016.stdout:mds.c smithi063 running (106s) 25s ago 105s 16.9M - 19.0.0-1899-g323d23dc bed8e21b47c6 60896cd5fa5f 2024-03-28T22:09:56.661 INFO:teuthology.orchestra.run.smithi016.stdout:mds.d smithi016 running (114s) 15s ago 113s 14.0M - 19.0.0-1899-g323d23dc bed8e21b47c6 ef92b8d3de0c 2024-03-28T22:09:56.662 INFO:teuthology.orchestra.run.smithi016.stdout:mds.e smithi053 running (111s) 25s ago 110s 19.2M - 19.0.0-1899-g323d23dc bed8e21b47c6 71e30f2bb06c 2024-03-28T22:09:56.662 INFO:teuthology.orchestra.run.smithi016.stdout:mds.f smithi063 running (108s) 25s ago 108s 16.8M - 19.0.0-1899-g323d23dc bed8e21b47c6 ce5a887a2420 2024-03-28T22:09:56.662 INFO:teuthology.orchestra.run.smithi016.stdout:mds.g smithi016 running (97s) 15s ago 97s 16.2M - 19.0.0-1899-g323d23dc bed8e21b47c6 ed462ec3a52b 2024-03-28T22:09:56.662 INFO:teuthology.orchestra.run.smithi016.stdout:mds.h smithi053 running (103s) 25s ago 102s 18.0M - 19.0.0-1899-g323d23dc bed8e21b47c6 e38cdb5da585 2024-03-28T22:09:56.662 INFO:teuthology.orchestra.run.smithi016.stdout:mds.i smithi063 running (100s) 25s ago 99s 19.2M - 19.0.0-1899-g323d23dc bed8e21b47c6 4092192cc725 2024-03-28T22:09:56.662 INFO:teuthology.orchestra.run.smithi016.stdout:mds.j smithi016 running (116s) 15s ago 116s 13.6M - 19.0.0-1899-g323d23dc bed8e21b47c6 83061c52f5f6 2024-03-28T22:09:56.662 INFO:teuthology.orchestra.run.smithi016.stdout:mds.k smithi053 running (119s) 25s ago 119s 13.9M - 19.0.0-1899-g323d23dc bed8e21b47c6 321408088b92 2024-03-28T22:09:56.662 INFO:teuthology.orchestra.run.smithi016.stdout:mds.l smithi063 running (2m) 25s ago 2m 15.3M - 19.0.0-1899-g323d23dc bed8e21b47c6 b1246fcb93bf 2024-03-28T22:09:56.662 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi016 *:9283,8765 running (9m) 15s ago 9m 512M - 19.0.0-1899-g323d23dc bed8e21b47c6 ebef8c7b1dbd 2024-03-28T22:09:56.662 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi053 *:8443,8765 running (7m) 25s ago 7m 450M - 19.0.0-1899-g323d23dc bed8e21b47c6 6153d3a6250b 2024-03-28T22:09:56.662 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.z smithi063 *:8443,8765 running (7m) 25s ago 7m 451M - 19.0.0-1899-g323d23dc bed8e21b47c6 7a7c1ac7b208 2024-03-28T22:09:56.662 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (9m) 15s ago 9m 52.3M 2048M 19.0.0-1899-g323d23dc bed8e21b47c6 89528bfea732 2024-03-28T22:09:56.663 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi053 running (7m) 25s ago 7m 37.9M 2048M 19.0.0-1899-g323d23dc bed8e21b47c6 1b0c77331568 2024-03-28T22:09:56.663 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi063 running (7m) 25s ago 7m 39.0M 2048M 19.0.0-1899-g323d23dc bed8e21b47c6 1283d1c3415a 2024-03-28T22:09:56.663 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (6m) 15s ago 6m 47.7M 4273M 19.0.0-1899-g323d23dc bed8e21b47c6 a44558911320 2024-03-28T22:09:56.663 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi053 running (6m) 25s ago 6m 48.6M 4271M 19.0.0-1899-g323d23dc bed8e21b47c6 5250da249d5e 2024-03-28T22:09:56.663 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi063 running (6m) 25s ago 6m 58.3M 4270M 19.0.0-1899-g323d23dc bed8e21b47c6 7ab4716535cb 2024-03-28T22:09:56.663 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (5m) 15s ago 5m 46.8M 4273M 19.0.0-1899-g323d23dc bed8e21b47c6 c4470d6e66af 2024-03-28T22:09:56.663 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi053 running (5m) 25s ago 5m 51.2M 4271M 19.0.0-1899-g323d23dc bed8e21b47c6 be327efe30f3 2024-03-28T22:09:56.663 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi063 running (4m) 25s ago 4m 46.7M 4270M 19.0.0-1899-g323d23dc bed8e21b47c6 9835696fdd40 2024-03-28T22:09:56.663 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi016 running (4m) 15s ago 4m 54.3M 4273M 19.0.0-1899-g323d23dc bed8e21b47c6 905373d56540 2024-03-28T22:09:56.663 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi053 running (4m) 25s ago 4m 54.2M 4271M 19.0.0-1899-g323d23dc bed8e21b47c6 0d27146c6f12 2024-03-28T22:09:56.663 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8 smithi063 running (3m) 25s ago 3m 49.1M 4270M 19.0.0-1899-g323d23dc bed8e21b47c6 ef945d9ba655 2024-03-28T22:09:56.663 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9 smithi016 running (3m) 15s ago 3m 44.7M 4273M 19.0.0-1899-g323d23dc bed8e21b47c6 10ce830a8821 2024-03-28T22:09:56.663 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10 smithi053 running (2m) 25s ago 2m 45.8M 4271M 19.0.0-1899-g323d23dc bed8e21b47c6 95a527461e98 2024-03-28T22:09:56.663 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11 smithi063 running (2m) 25s ago 2m 49.7M 4270M 19.0.0-1899-g323d23dc bed8e21b47c6 166707260b95 2024-03-28T22:09:56.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.662+0000 7ff02ce26700 1 -- 172.21.15.16:0/769659563 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff0140a1f20 msgr2=0x7ff0140a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:56.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.662+0000 7ff02ce26700 1 --2- 172.21.15.16:0/769659563 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff0140a1f20 0x7ff0140a43e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff010009bf0 tx=0x7ff010009260 comp rx=0 tx=0).stop 2024-03-28T22:09:56.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.662+0000 7ff02ce26700 1 -- 172.21.15.16:0/769659563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff028102d00 msgr2=0x7ff028198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:56.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.662+0000 7ff02ce26700 1 --2- 172.21.15.16:0/769659563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff028102d00 0x7ff028198520 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff018004d20 tx=0x7ff018004d50 comp rx=0 tx=0).stop 2024-03-28T22:09:56.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.662+0000 7ff02ce26700 1 -- 172.21.15.16:0/769659563 shutdown_connections 2024-03-28T22:09:56.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.662+0000 7ff02ce26700 1 --2- 172.21.15.16:0/769659563 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7ff0140a1f20 0x7ff0140a43e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:56.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.662+0000 7ff02ce26700 1 --2- 172.21.15.16:0/769659563 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff028107f60 0x7ff02819fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:56.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.662+0000 7ff02ce26700 1 --2- 172.21.15.16:0/769659563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff028102d00 0x7ff028198520 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:56.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.662+0000 7ff02ce26700 1 --2- 172.21.15.16:0/769659563 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff028105630 0x7ff028198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:56.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.662+0000 7ff02ce26700 1 -- 172.21.15.16:0/769659563 >> 172.21.15.16:0/769659563 conn(0x7ff0280fc710 msgr2=0x7ff0281035a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:56.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.662+0000 7ff02ce26700 1 -- 172.21.15.16:0/769659563 shutdown_connections 2024-03-28T22:09:56.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:56.662+0000 7ff02ce26700 1 -- 172.21.15.16:0/769659563 wait complete. 2024-03-28T22:09:57.298 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:57 smithi016 ceph-mon[32374]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:09:57.299 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:57 smithi016 ceph-mon[32374]: pgmap v332: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T22:09:57.332 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-28T22:09:57.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:57 smithi063 ceph-mon[36249]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:09:57.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:57 smithi063 ceph-mon[36249]: pgmap v332: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T22:09:57.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:57 smithi053 ceph-mon[35979]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:09:57.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:57 smithi053 ceph-mon[35979]: pgmap v332: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T22:09:57.587 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:09:59.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.104+0000 7f3a7d89f700 1 -- 172.21.15.16:0/1541832224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a78101c10 msgr2=0x7f3a781064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:59.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.104+0000 7f3a7d89f700 1 --2- 172.21.15.16:0/1541832224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a78101c10 0x7f3a781064b0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3a7000b0a0 tx=0x7f3a7000b3b0 comp rx=0 tx=0).stop 2024-03-28T22:09:59.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.104+0000 7f3a7d89f700 1 -- 172.21.15.16:0/1541832224 shutdown_connections 2024-03-28T22:09:59.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.104+0000 7f3a7d89f700 1 --2- 172.21.15.16:0/1541832224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a78101c10 0x7f3a781064b0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:59.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.104+0000 7f3a7d89f700 1 --2- 172.21.15.16:0/1541832224 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a78100110 0x7f3a78100530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:59.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.104+0000 7f3a7d89f700 1 --2- 172.21.15.16:0/1541832224 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a78101250 0x7f3a781016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:59.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.104+0000 7f3a7d89f700 1 -- 172.21.15.16:0/1541832224 >> 172.21.15.16:0/1541832224 conn(0x7f3a780fba40 msgr2=0x7f3a780fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:59.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.104+0000 7f3a7d89f700 1 -- 172.21.15.16:0/1541832224 shutdown_connections 2024-03-28T22:09:59.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.104+0000 7f3a7d89f700 1 -- 172.21.15.16:0/1541832224 wait complete. 2024-03-28T22:09:59.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.105+0000 7f3a7d89f700 1 Processor -- start 2024-03-28T22:09:59.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.105+0000 7f3a7d89f700 1 -- start start 2024-03-28T22:09:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.105+0000 7f3a7d89f700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a78100110 0x7f3a7819e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.105+0000 7f3a7d89f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a78101250 0x7f3a7819eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.105+0000 7f3a7d89f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a78101c10 0x7f3a781a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.105+0000 7f3a7d89f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3a781144d0 con 0x7f3a78101250 2024-03-28T22:09:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.105+0000 7f3a7d89f700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3a78114350 con 0x7f3a78101c10 2024-03-28T22:09:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.105+0000 7f3a7d89f700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3a78114650 con 0x7f3a78100110 2024-03-28T22:09:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.105+0000 7f3a767fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a78101250 0x7f3a7819eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.105+0000 7f3a76ffd700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a78100110 0x7f3a7819e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.106+0000 7f3a767fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a78101250 0x7f3a7819eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41726/0 (socket says 172.21.15.16:41726) 2024-03-28T22:09:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.106+0000 7f3a767fc700 1 -- 172.21.15.16:0/2307570176 learned_addr learned my addr 172.21.15.16:0/2307570176 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:09:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.106+0000 7f3a767fc700 1 -- 172.21.15.16:0/2307570176 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a78100110 msgr2=0x7f3a7819e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.106+0000 7f3a767fc700 1 --2- 172.21.15.16:0/2307570176 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a78100110 0x7f3a7819e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.106+0000 7f3a767fc700 1 -- 172.21.15.16:0/2307570176 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a78101c10 msgr2=0x7f3a781a5f20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T22:09:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.106+0000 7f3a767fc700 1 --2- 172.21.15.16:0/2307570176 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a78101c10 0x7f3a781a5f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.106+0000 7f3a767fc700 1 -- 172.21.15.16:0/2307570176 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a781a65c0 con 0x7f3a78101250 2024-03-28T22:09:59.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.106+0000 7f3a767fc700 1 --2- 172.21.15.16:0/2307570176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a78101250 0x7f3a7819eea0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c00d980 tx=0x7f3a6c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:59.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.107+0000 7f3a7c89d700 1 -- 172.21.15.16:0/2307570176 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c016070 con 0x7f3a78101250 2024-03-28T22:09:59.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.107+0000 7f3a7c89d700 1 -- 172.21.15.16:0/2307570176 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a6c00de70 con 0x7f3a78101250 2024-03-28T22:09:59.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.107+0000 7f3a7c89d700 1 -- 172.21.15.16:0/2307570176 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c00f3c0 con 0x7f3a78101250 2024-03-28T22:09:59.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.107+0000 7f3a7d89f700 1 -- 172.21.15.16:0/2307570176 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a781a68b0 con 0x7f3a78101250 2024-03-28T22:09:59.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.107+0000 7f3a7d89f700 1 -- 172.21.15.16:0/2307570176 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a781a6da0 con 0x7f3a78101250 2024-03-28T22:09:59.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.108+0000 7f3a7d89f700 1 -- 172.21.15.16:0/2307570176 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a78068f00 con 0x7f3a78101250 2024-03-28T22:09:59.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.110+0000 7f3a7c89d700 1 -- 172.21.15.16:0/2307570176 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f3a6c019070 con 0x7f3a78101250 2024-03-28T22:09:59.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.111+0000 7f3a7c89d700 1 --2- 172.21.15.16:0/2307570176 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f3a600a1ff0 0x7f3a600a44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:09:59.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.111+0000 7f3a7c89d700 1 -- 172.21.15.16:0/2307570176 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f3a6c0cd910 con 0x7f3a78101250 2024-03-28T22:09:59.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.111+0000 7f3a76ffd700 1 --2- 172.21.15.16:0/2307570176 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f3a600a1ff0 0x7f3a600a44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:09:59.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.112+0000 7f3a76ffd700 1 --2- 172.21.15.16:0/2307570176 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f3a600a1ff0 0x7f3a600a44b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3a6800ac50 tx=0x7f3a6800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:09:59.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.116+0000 7f3a7c89d700 1 -- 172.21.15.16:0/2307570176 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3a6c09ac90 con 0x7f3a78101250 2024-03-28T22:09:59.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.332+0000 7f3a7d89f700 1 -- 172.21.15.16:0/2307570176 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f3a78062740 con 0x7f3a600a1ff0 2024-03-28T22:09:59.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.338+0000 7f3a7c89d700 1 -- 172.21.15.16:0/2307570176 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f3a78062740 con 0x7f3a600a1ff0 2024-03-28T22:09:59.339 INFO:teuthology.orchestra.run.smithi016.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-28T22:09:59.339 INFO:teuthology.orchestra.run.smithi016.stdout:mds.all 12/12 28s ago 2m smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi053=b;smithi053=e;smithi053=h;smithi053=k;smithi063=c;smithi063=f;smithi063=i;smithi063=l;count:12 2024-03-28T22:09:59.339 INFO:teuthology.orchestra.run.smithi016.stdout:mgr 3/3 28s ago 7m smithi016=x;smithi053=y;smithi063=z;count:3 2024-03-28T22:09:59.339 INFO:teuthology.orchestra.run.smithi016.stdout:mon 3/3 28s ago 7m smithi016:172.21.15.16=a;smithi053:172.21.15.53=b;smithi063:172.21.15.63=c;count:3 2024-03-28T22:09:59.340 INFO:teuthology.orchestra.run.smithi016.stdout:osd 12 28s ago - 2024-03-28T22:09:59.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.340+0000 7f3a7d89f700 1 -- 172.21.15.16:0/2307570176 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f3a600a1ff0 msgr2=0x7f3a600a44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:59.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.340+0000 7f3a7d89f700 1 --2- 172.21.15.16:0/2307570176 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f3a600a1ff0 0x7f3a600a44b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3a6800ac50 tx=0x7f3a6800a380 comp rx=0 tx=0).stop 2024-03-28T22:09:59.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.340+0000 7f3a7d89f700 1 -- 172.21.15.16:0/2307570176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a78101250 msgr2=0x7f3a7819eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:09:59.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.340+0000 7f3a7d89f700 1 --2- 172.21.15.16:0/2307570176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a78101250 0x7f3a7819eea0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c00d980 tx=0x7f3a6c00dc90 comp rx=0 tx=0).stop 2024-03-28T22:09:59.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.340+0000 7f3a7d89f700 1 -- 172.21.15.16:0/2307570176 shutdown_connections 2024-03-28T22:09:59.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.340+0000 7f3a7d89f700 1 --2- 172.21.15.16:0/2307570176 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f3a600a1ff0 0x7f3a600a44b0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:59.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.340+0000 7f3a7d89f700 1 --2- 172.21.15.16:0/2307570176 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a78101c10 0x7f3a781a5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:59.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.340+0000 7f3a7d89f700 1 --2- 172.21.15.16:0/2307570176 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a78100110 0x7f3a7819e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:59.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.340+0000 7f3a7d89f700 1 --2- 172.21.15.16:0/2307570176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a78101250 0x7f3a7819eea0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:09:59.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.340+0000 7f3a7d89f700 1 -- 172.21.15.16:0/2307570176 >> 172.21.15.16:0/2307570176 conn(0x7f3a780fba40 msgr2=0x7f3a78102470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:09:59.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.340+0000 7f3a7d89f700 1 -- 172.21.15.16:0/2307570176 shutdown_connections 2024-03-28T22:09:59.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:09:59.340+0000 7f3a7d89f700 1 -- 172.21.15.16:0/2307570176 wait complete. 2024-03-28T22:09:59.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:09:59 smithi063 ceph-mon[36249]: pgmap v333: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:09:59.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:09:59 smithi016 ceph-mon[32374]: pgmap v333: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:09:59.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:09:59 smithi053 ceph-mon[35979]: pgmap v333: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:09:59.898 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-28T22:10:00.144 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:10:00.420 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:00 smithi016 ceph-mon[32374]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:10:00.420 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:00 smithi016 ceph-mon[32374]: overall HEALTH_OK 2024-03-28T22:10:00.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:00 smithi063 ceph-mon[36249]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:10:00.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:00 smithi063 ceph-mon[36249]: overall HEALTH_OK 2024-03-28T22:10:00.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:00 smithi053 ceph-mon[35979]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:10:00.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:00 smithi053 ceph-mon[35979]: overall HEALTH_OK 2024-03-28T22:10:01.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:01 smithi063 ceph-mon[36249]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-28T22:10:01.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:01 smithi063 ceph-mon[36249]: pgmap v334: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-28T22:10:01.432 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:01 smithi016 ceph-mon[32374]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-28T22:10:01.433 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:01 smithi016 ceph-mon[32374]: pgmap v334: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-28T22:10:01.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:01 smithi053 ceph-mon[35979]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-28T22:10:01.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:01 smithi053 ceph-mon[35979]: pgmap v334: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-28T22:10:01.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.627+0000 7fc31849c700 1 -- 172.21.15.16:0/3474274929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc310103e30 msgr2=0x7fc31010a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:01.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.627+0000 7fc31849c700 1 --2- 172.21.15.16:0/3474274929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc310103e30 0x7fc31010a6e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc30800b0d0 tx=0x7fc30800b3e0 comp rx=0 tx=0).stop 2024-03-28T22:10:01.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.627+0000 7fc31849c700 1 -- 172.21.15.16:0/3474274929 shutdown_connections 2024-03-28T22:10:01.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.627+0000 7fc31849c700 1 --2- 172.21.15.16:0/3474274929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc310103e30 0x7fc31010a6e0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:01.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.627+0000 7fc31849c700 1 --2- 172.21.15.16:0/3474274929 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc310102290 0x7fc3101026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:01.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.627+0000 7fc31849c700 1 --2- 172.21.15.16:0/3474274929 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc310103490 0x7fc3101038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:01.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.627+0000 7fc31849c700 1 -- 172.21.15.16:0/3474274929 >> 172.21.15.16:0/3474274929 conn(0x7fc3100fd7f0 msgr2=0x7fc3100ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:01.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.627+0000 7fc31849c700 1 -- 172.21.15.16:0/3474274929 shutdown_connections 2024-03-28T22:10:01.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.627+0000 7fc31849c700 1 -- 172.21.15.16:0/3474274929 wait complete. 2024-03-28T22:10:01.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.628+0000 7fc31849c700 1 Processor -- start 2024-03-28T22:10:01.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.628+0000 7fc31849c700 1 -- start start 2024-03-28T22:10:01.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.628+0000 7fc31849c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc310102290 0x7fc310075fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:01.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.628+0000 7fc31849c700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc310103490 0x7fc310074630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:01.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.628+0000 7fc31849c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc310103e30 0x7fc310074b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:01.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.628+0000 7fc31849c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc31010de50 con 0x7fc310103e30 2024-03-28T22:10:01.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.628+0000 7fc31849c700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc31010dcd0 con 0x7fc310102290 2024-03-28T22:10:01.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.628+0000 7fc31849c700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc31010dfd0 con 0x7fc310103490 2024-03-28T22:10:01.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.628+0000 7fc316a39700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc310103e30 0x7fc310074b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:01.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.628+0000 7fc316238700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc310102290 0x7fc310075fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:01.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.628+0000 7fc316a39700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc310103e30 0x7fc310074b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38804/0 (socket says 172.21.15.16:38804) 2024-03-28T22:10:01.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.628+0000 7fc316a39700 1 -- 172.21.15.16:0/3982197520 learned_addr learned my addr 172.21.15.16:0/3982197520 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:01.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.628+0000 7fc315a37700 1 --2- 172.21.15.16:0/3982197520 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc310103490 0x7fc310074630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:01.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.628+0000 7fc316a39700 1 -- 172.21.15.16:0/3982197520 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc310103490 msgr2=0x7fc310074630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:01.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.628+0000 7fc316a39700 1 --2- 172.21.15.16:0/3982197520 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc310103490 0x7fc310074630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:01.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.628+0000 7fc316a39700 1 -- 172.21.15.16:0/3982197520 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc310102290 msgr2=0x7fc310075fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:01.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.628+0000 7fc316a39700 1 --2- 172.21.15.16:0/3982197520 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc310102290 0x7fc310075fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:01.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.628+0000 7fc316a39700 1 -- 172.21.15.16:0/3982197520 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc310075420 con 0x7fc310103e30 2024-03-28T22:10:01.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.628+0000 7fc315a37700 1 --2- 172.21.15.16:0/3982197520 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc310103490 0x7fc310074630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:10:01.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.629+0000 7fc316a39700 1 --2- 172.21.15.16:0/3982197520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc310103e30 0x7fc310074b70 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc308006070 tx=0x7fc308007d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:01.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.629+0000 7fc3077fe700 1 -- 172.21.15.16:0/3982197520 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc308021020 con 0x7fc310103e30 2024-03-28T22:10:01.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.629+0000 7fc31849c700 1 -- 172.21.15.16:0/3982197520 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3101a24d0 con 0x7fc310103e30 2024-03-28T22:10:01.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.629+0000 7fc3077fe700 1 -- 172.21.15.16:0/3982197520 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc30801d950 con 0x7fc310103e30 2024-03-28T22:10:01.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.629+0000 7fc3077fe700 1 -- 172.21.15.16:0/3982197520 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc308005390 con 0x7fc310103e30 2024-03-28T22:10:01.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.629+0000 7fc31849c700 1 -- 172.21.15.16:0/3982197520 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3101a29e0 con 0x7fc310103e30 2024-03-28T22:10:01.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.630+0000 7fc31849c700 1 -- 172.21.15.16:0/3982197520 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc310068f00 con 0x7fc310103e30 2024-03-28T22:10:01.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.631+0000 7fc3077fe700 1 -- 172.21.15.16:0/3982197520 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7fc308024020 con 0x7fc310103e30 2024-03-28T22:10:01.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.631+0000 7fc3077fe700 1 --2- 172.21.15.16:0/3982197520 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc2fc0a1e10 0x7fc2fc0a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:01.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.631+0000 7fc3077fe700 1 -- 172.21.15.16:0/3982197520 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fc30800f020 con 0x7fc310103e30 2024-03-28T22:10:01.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.631+0000 7fc316238700 1 --2- 172.21.15.16:0/3982197520 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc2fc0a1e10 0x7fc2fc0a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:01.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.632+0000 7fc316238700 1 --2- 172.21.15.16:0/3982197520 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc2fc0a1e10 0x7fc2fc0a42d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc3101032f0 tx=0x7fc30c007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:01.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.635+0000 7fc3077fe700 1 -- 172.21.15.16:0/3982197520 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc30809cc50 con 0x7fc310103e30 2024-03-28T22:10:01.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.846+0000 7fc31849c700 1 -- 172.21.15.16:0/3982197520 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fc310062680 con 0x7fc2fc0a1e10 2024-03-28T22:10:01.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.848+0000 7fc3077fe700 1 -- 172.21.15.16:0/3982197520 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+187 (secure 0 0 0) 0x7fc310062680 con 0x7fc2fc0a1e10 2024-03-28T22:10:01.849 INFO:teuthology.orchestra.run.smithi016.stdout:HOST ADDR LABELS STATUS 2024-03-28T22:10:01.850 INFO:teuthology.orchestra.run.smithi016.stdout:smithi016 172.21.15.16 2024-03-28T22:10:01.850 INFO:teuthology.orchestra.run.smithi016.stdout:smithi053 172.21.15.53 2024-03-28T22:10:01.850 INFO:teuthology.orchestra.run.smithi016.stdout:smithi063 172.21.15.63 2024-03-28T22:10:01.850 INFO:teuthology.orchestra.run.smithi016.stdout:3 hosts in cluster 2024-03-28T22:10:01.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.850+0000 7fc31849c700 1 -- 172.21.15.16:0/3982197520 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc2fc0a1e10 msgr2=0x7fc2fc0a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:01.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.850+0000 7fc31849c700 1 --2- 172.21.15.16:0/3982197520 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc2fc0a1e10 0x7fc2fc0a42d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc3101032f0 tx=0x7fc30c007400 comp rx=0 tx=0).stop 2024-03-28T22:10:01.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.850+0000 7fc31849c700 1 -- 172.21.15.16:0/3982197520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc310103e30 msgr2=0x7fc310074b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:01.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.850+0000 7fc31849c700 1 --2- 172.21.15.16:0/3982197520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc310103e30 0x7fc310074b70 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc308006070 tx=0x7fc308007d10 comp rx=0 tx=0).stop 2024-03-28T22:10:01.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.850+0000 7fc31849c700 1 -- 172.21.15.16:0/3982197520 shutdown_connections 2024-03-28T22:10:01.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.850+0000 7fc31849c700 1 --2- 172.21.15.16:0/3982197520 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc2fc0a1e10 0x7fc2fc0a42d0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:01.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.850+0000 7fc31849c700 1 --2- 172.21.15.16:0/3982197520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc310103e30 0x7fc310074b70 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:01.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.850+0000 7fc31849c700 1 --2- 172.21.15.16:0/3982197520 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc310102290 0x7fc310075fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:01.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.850+0000 7fc31849c700 1 --2- 172.21.15.16:0/3982197520 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc310103490 0x7fc310074630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:01.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.850+0000 7fc31849c700 1 -- 172.21.15.16:0/3982197520 >> 172.21.15.16:0/3982197520 conn(0x7fc3100fd7f0 msgr2=0x7fc3101066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:01.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.850+0000 7fc31849c700 1 -- 172.21.15.16:0/3982197520 shutdown_connections 2024-03-28T22:10:01.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:01.850+0000 7fc31849c700 1 -- 172.21.15.16:0/3982197520 wait complete. 2024-03-28T22:10:02.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:02 smithi063 ceph-mon[36249]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:10:02.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:02 smithi016 ceph-mon[32374]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:10:02.536 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-28T22:10:02.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:02 smithi053 ceph-mon[35979]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:10:02.781 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:10:03.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:03 smithi063 ceph-mon[36249]: pgmap v335: 97 pgs: 97 active+clean; 639 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-28T22:10:03.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:03 smithi016 ceph-mon[32374]: pgmap v335: 97 pgs: 97 active+clean; 639 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-28T22:10:03.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:03 smithi053 ceph-mon[35979]: pgmap v335: 97 pgs: 97 active+clean; 639 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-28T22:10:04.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.417+0000 7f52d2d34700 1 -- 172.21.15.16:0/3150311213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52cc102370 msgr2=0x7f52cc1027d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:04.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.417+0000 7f52d2d34700 1 --2- 172.21.15.16:0/3150311213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52cc102370 0x7f52cc1027d0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f52b80099c0 tx=0x7f52b8009cd0 comp rx=0 tx=0).stop 2024-03-28T22:10:04.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.417+0000 7f52d2d34700 1 -- 172.21.15.16:0/3150311213 shutdown_connections 2024-03-28T22:10:04.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.417+0000 7f52d2d34700 1 --2- 172.21.15.16:0/3150311213 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f52cc102d10 0x7f52cc1095c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:04.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.417+0000 7f52d2d34700 1 --2- 172.21.15.16:0/3150311213 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52cc101170 0x7f52cc101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:04.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.417+0000 7f52d2d34700 1 --2- 172.21.15.16:0/3150311213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52cc102370 0x7f52cc1027d0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:04.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.417+0000 7f52d2d34700 1 -- 172.21.15.16:0/3150311213 >> 172.21.15.16:0/3150311213 conn(0x7f52cc0fc710 msgr2=0x7f52cc0feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:04.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.417+0000 7f52d2d34700 1 -- 172.21.15.16:0/3150311213 shutdown_connections 2024-03-28T22:10:04.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.417+0000 7f52d2d34700 1 -- 172.21.15.16:0/3150311213 wait complete. 2024-03-28T22:10:04.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.418+0000 7f52d2d34700 1 Processor -- start 2024-03-28T22:10:04.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.418+0000 7f52d2d34700 1 -- start start 2024-03-28T22:10:04.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.418+0000 7f52d2d34700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52cc101170 0x7f52cc198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:04.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.418+0000 7f52d2d34700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f52cc102370 0x7f52cc1989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:04.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.418+0000 7f52d2d34700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52cc102d10 0x7f52cc19fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:04.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.418+0000 7f52d2d34700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f52cc10cd60 con 0x7f52cc101170 2024-03-28T22:10:04.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.418+0000 7f52d2d34700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f52cc10cbe0 con 0x7f52cc102d10 2024-03-28T22:10:04.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.418+0000 7f52d2d34700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f52cc10cee0 con 0x7f52cc102370 2024-03-28T22:10:04.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.418+0000 7f52d0ad0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52cc101170 0x7f52cc198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:04.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.418+0000 7f52d0ad0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52cc101170 0x7f52cc198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38822/0 (socket says 172.21.15.16:38822) 2024-03-28T22:10:04.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.418+0000 7f52d0ad0700 1 -- 172.21.15.16:0/167826253 learned_addr learned my addr 172.21.15.16:0/167826253 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:04.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.418+0000 7f52d12d1700 1 --2- 172.21.15.16:0/167826253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52cc102d10 0x7f52cc19fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:04.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.418+0000 7f52cbfff700 1 --2- 172.21.15.16:0/167826253 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f52cc102370 0x7f52cc1989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:04.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.418+0000 7f52d0ad0700 1 -- 172.21.15.16:0/167826253 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f52cc102370 msgr2=0x7f52cc1989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:04.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.418+0000 7f52d0ad0700 1 --2- 172.21.15.16:0/167826253 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f52cc102370 0x7f52cc1989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:04.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.419+0000 7f52d0ad0700 1 -- 172.21.15.16:0/167826253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52cc102d10 msgr2=0x7f52cc19fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:04.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.419+0000 7f52d0ad0700 1 --2- 172.21.15.16:0/167826253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52cc102d10 0x7f52cc19fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:04.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.419+0000 7f52d0ad0700 1 -- 172.21.15.16:0/167826253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52cc1a00f0 con 0x7f52cc101170 2024-03-28T22:10:04.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.419+0000 7f52d12d1700 1 --2- 172.21.15.16:0/167826253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52cc102d10 0x7f52cc19fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:10:04.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.419+0000 7f52d0ad0700 1 --2- 172.21.15.16:0/167826253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52cc101170 0x7f52cc198490 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f52c000e9a0 tx=0x7f52c000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:04.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.419+0000 7f52c9ffb700 1 -- 172.21.15.16:0/167826253 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c0004220 con 0x7f52cc101170 2024-03-28T22:10:04.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.419+0000 7f52d2d34700 1 -- 172.21.15.16:0/167826253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52cc1a03e0 con 0x7f52cc101170 2024-03-28T22:10:04.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.419+0000 7f52c9ffb700 1 -- 172.21.15.16:0/167826253 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52c00043c0 con 0x7f52cc101170 2024-03-28T22:10:04.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.419+0000 7f52c9ffb700 1 -- 172.21.15.16:0/167826253 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c0003df0 con 0x7f52cc101170 2024-03-28T22:10:04.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.419+0000 7f52d2d34700 1 -- 172.21.15.16:0/167826253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52cc1a0970 con 0x7f52cc101170 2024-03-28T22:10:04.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.420+0000 7f52d2d34700 1 -- 172.21.15.16:0/167826253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52cc068f00 con 0x7f52cc101170 2024-03-28T22:10:04.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.421+0000 7f52c9ffb700 1 -- 172.21.15.16:0/167826253 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f52c000c810 con 0x7f52cc101170 2024-03-28T22:10:04.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.422+0000 7f52c9ffb700 1 --2- 172.21.15.16:0/167826253 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f52bc0a1e10 0x7f52bc0a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:04.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.422+0000 7f52cbfff700 1 --2- 172.21.15.16:0/167826253 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f52bc0a1e10 0x7f52bc0a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:04.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.422+0000 7f52c9ffb700 1 -- 172.21.15.16:0/167826253 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f52c00ce060 con 0x7f52cc101170 2024-03-28T22:10:04.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.423+0000 7f52cbfff700 1 --2- 172.21.15.16:0/167826253 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f52bc0a1e10 0x7f52bc0a42d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f52b8000c00 tx=0x7f52b8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:04.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.426+0000 7f52c9ffb700 1 -- 172.21.15.16:0/167826253 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f52c009b4f0 con 0x7f52cc101170 2024-03-28T22:10:04.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.639+0000 7f52d2d34700 1 -- 172.21.15.16:0/167826253 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f52cc0fe8c0 con 0x7f52bc0a1e10 2024-03-28T22:10:04.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.640+0000 7f52c9ffb700 1 -- 172.21.15.16:0/167826253 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f52cc0fe8c0 con 0x7f52bc0a1e10 2024-03-28T22:10:04.643 INFO:teuthology.orchestra.run.smithi016.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-28T22:10:04.643 INFO:teuthology.orchestra.run.smithi016.stdout:smithi016 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7513002C480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-03-28T22:10:04.643 INFO:teuthology.orchestra.run.smithi016.stdout:smithi053 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB751300G8480DGN 447G No 2m ago Has a FileSystem, LVM detected 2024-03-28T22:10:04.643 INFO:teuthology.orchestra.run.smithi016.stdout:smithi063 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5344005B400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-28T22:10:04.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.642+0000 7f52d2d34700 1 -- 172.21.15.16:0/167826253 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f52bc0a1e10 msgr2=0x7f52bc0a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:04.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.642+0000 7f52d2d34700 1 --2- 172.21.15.16:0/167826253 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f52bc0a1e10 0x7f52bc0a42d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f52b8000c00 tx=0x7f52b8005fb0 comp rx=0 tx=0).stop 2024-03-28T22:10:04.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.642+0000 7f52d2d34700 1 -- 172.21.15.16:0/167826253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52cc101170 msgr2=0x7f52cc198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:04.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.642+0000 7f52d2d34700 1 --2- 172.21.15.16:0/167826253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52cc101170 0x7f52cc198490 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f52c000e9a0 tx=0x7f52c000ecb0 comp rx=0 tx=0).stop 2024-03-28T22:10:04.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.642+0000 7f52d2d34700 1 -- 172.21.15.16:0/167826253 shutdown_connections 2024-03-28T22:10:04.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.642+0000 7f52d2d34700 1 --2- 172.21.15.16:0/167826253 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f52bc0a1e10 0x7f52bc0a42d0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:04.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.642+0000 7f52d2d34700 1 --2- 172.21.15.16:0/167826253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52cc102d10 0x7f52cc19fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:04.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.642+0000 7f52d2d34700 1 --2- 172.21.15.16:0/167826253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52cc101170 0x7f52cc198490 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:04.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.642+0000 7f52d2d34700 1 --2- 172.21.15.16:0/167826253 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f52cc102370 0x7f52cc1989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:04.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.642+0000 7f52d2d34700 1 -- 172.21.15.16:0/167826253 >> 172.21.15.16:0/167826253 conn(0x7f52cc0fc710 msgr2=0x7f52cc0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:04.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.643+0000 7f52d2d34700 1 -- 172.21.15.16:0/167826253 shutdown_connections 2024-03-28T22:10:04.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:04.643+0000 7f52d2d34700 1 -- 172.21.15.16:0/167826253 wait complete. 2024-03-28T22:10:05.390 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:05 smithi016 ceph-mon[32374]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:10:05.390 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:05 smithi016 ceph-mon[32374]: pgmap v336: 97 pgs: 97 active+clean; 639 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-28T22:10:05.418 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-28T22:10:05.428 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-03-28T22:10:05.428 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-28T22:10:05.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:05 smithi063 ceph-mon[36249]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:10:05.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:05 smithi063 ceph-mon[36249]: pgmap v336: 97 pgs: 97 active+clean; 639 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-28T22:10:05.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:05 smithi053 ceph-mon[35979]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:10:05.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:05 smithi053 ceph-mon[35979]: pgmap v336: 97 pgs: 97 active+clean; 639 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-28T22:10:05.672 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:10:07.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.232+0000 7f8958d14700 1 -- 172.21.15.16:0/821607227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8954103e10 msgr2=0x7f895410a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:07.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.232+0000 7f8958d14700 1 --2- 172.21.15.16:0/821607227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8954103e10 0x7f895410a6c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f894c00b0d0 tx=0x7f894c00b3e0 comp rx=0 tx=0).stop 2024-03-28T22:10:07.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.232+0000 7f8958d14700 1 -- 172.21.15.16:0/821607227 shutdown_connections 2024-03-28T22:10:07.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.232+0000 7f8958d14700 1 --2- 172.21.15.16:0/821607227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8954103e10 0x7f895410a6c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:07.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.232+0000 7f8958d14700 1 --2- 172.21.15.16:0/821607227 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8954102270 0x7f8954102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:07.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.232+0000 7f8958d14700 1 --2- 172.21.15.16:0/821607227 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8954103470 0x7f89541038d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:07.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.232+0000 7f8958d14700 1 -- 172.21.15.16:0/821607227 >> 172.21.15.16:0/821607227 conn(0x7f89540fd850 msgr2=0x7f89540ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:07.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.232+0000 7f8958d14700 1 -- 172.21.15.16:0/821607227 shutdown_connections 2024-03-28T22:10:07.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.232+0000 7f8958d14700 1 -- 172.21.15.16:0/821607227 wait complete. 2024-03-28T22:10:07.236 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:07 smithi016 ceph-mon[32374]: pgmap v337: 97 pgs: 97 active+clean; 639 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-28T22:10:07.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.233+0000 7f8958d14700 1 Processor -- start 2024-03-28T22:10:07.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.233+0000 7f8958d14700 1 -- start start 2024-03-28T22:10:07.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.233+0000 7f8958d14700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8954102270 0x7f895419a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:07.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.233+0000 7f8958d14700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8954103470 0x7f895419ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:07.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.233+0000 7f8958d14700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8954103e10 0x7f89541a1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:07.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.233+0000 7f8958d14700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f895410de50 con 0x7f8954103e10 2024-03-28T22:10:07.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.233+0000 7f8958d14700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f895410dcd0 con 0x7f8954102270 2024-03-28T22:10:07.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.233+0000 7f8958d14700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f895410dfd0 con 0x7f8954103470 2024-03-28T22:10:07.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.234+0000 7f8952d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8954103e10 0x7f89541a1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:07.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.234+0000 7f895259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8954102270 0x7f895419a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:07.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.234+0000 7f8952d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8954103e10 0x7f89541a1ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38838/0 (socket says 172.21.15.16:38838) 2024-03-28T22:10:07.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.234+0000 7f8952d9d700 1 -- 172.21.15.16:0/4217426306 learned_addr learned my addr 172.21.15.16:0/4217426306 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:07.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.234+0000 7f8951d9b700 1 --2- 172.21.15.16:0/4217426306 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8954103470 0x7f895419ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:07.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.234+0000 7f8952d9d700 1 -- 172.21.15.16:0/4217426306 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8954103470 msgr2=0x7f895419ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:07.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.234+0000 7f8952d9d700 1 --2- 172.21.15.16:0/4217426306 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8954103470 0x7f895419ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:07.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.234+0000 7f8952d9d700 1 -- 172.21.15.16:0/4217426306 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8954102270 msgr2=0x7f895419a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:07.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.234+0000 7f8952d9d700 1 --2- 172.21.15.16:0/4217426306 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8954102270 0x7f895419a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:07.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.234+0000 7f8952d9d700 1 -- 172.21.15.16:0/4217426306 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89541a2240 con 0x7f8954103e10 2024-03-28T22:10:07.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.234+0000 7f8951d9b700 1 --2- 172.21.15.16:0/4217426306 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8954103470 0x7f895419ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:10:07.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.234+0000 7f8952d9d700 1 --2- 172.21.15.16:0/4217426306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8954103e10 0x7f89541a1ba0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f894c00b770 tx=0x7f894c007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:07.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.234+0000 7f894b7fe700 1 -- 172.21.15.16:0/4217426306 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894c003e10 con 0x7f8954103e10 2024-03-28T22:10:07.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.234+0000 7f8958d14700 1 -- 172.21.15.16:0/4217426306 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89541a24d0 con 0x7f8954103e10 2024-03-28T22:10:07.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.234+0000 7f894b7fe700 1 -- 172.21.15.16:0/4217426306 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f894c007cc0 con 0x7f8954103e10 2024-03-28T22:10:07.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.234+0000 7f894b7fe700 1 -- 172.21.15.16:0/4217426306 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894c01ba90 con 0x7f8954103e10 2024-03-28T22:10:07.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.234+0000 7f8958d14700 1 -- 172.21.15.16:0/4217426306 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89541a28d0 con 0x7f8954103e10 2024-03-28T22:10:07.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.235+0000 7f8958d14700 1 -- 172.21.15.16:0/4217426306 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8954068f00 con 0x7f8954103e10 2024-03-28T22:10:07.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.236+0000 7f894b7fe700 1 -- 172.21.15.16:0/4217426306 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f894c026400 con 0x7f8954103e10 2024-03-28T22:10:07.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.237+0000 7f894b7fe700 1 --2- 172.21.15.16:0/4217426306 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f89400a1e10 0x7f89400a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:07.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.237+0000 7f895259c700 1 --2- 172.21.15.16:0/4217426306 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f89400a1e10 0x7f89400a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:07.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.237+0000 7f894b7fe700 1 -- 172.21.15.16:0/4217426306 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f894c0cf3f0 con 0x7f8954103e10 2024-03-28T22:10:07.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.237+0000 7f895259c700 1 --2- 172.21.15.16:0/4217426306 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f89400a1e10 0x7f89400a42d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f89541032d0 tx=0x7f8944009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:07.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.241+0000 7f894b7fe700 1 -- 172.21.15.16:0/4217426306 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f894c09c8e0 con 0x7f8954103e10 2024-03-28T22:10:07.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:07 smithi063 ceph-mon[36249]: pgmap v337: 97 pgs: 97 active+clean; 639 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-28T22:10:07.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.490+0000 7f8958d14700 1 -- 172.21.15.16:0/4217426306 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f89541a3210 con 0x7f8954103e10 2024-03-28T22:10:07.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.490+0000 7f894b7fe700 1 -- 172.21.15.16:0/4217426306 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 20 v20) ==== 76+0+3036 (secure 0 0 0) 0x7f894c09c1e0 con 0x7f8954103e10 2024-03-28T22:10:07.492 INFO:teuthology.orchestra.run.smithi016.stdout:e20 2024-03-28T22:10:07.492 INFO:teuthology.orchestra.run.smithi016.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-28T22:10:07.492 INFO:teuthology.orchestra.run.smithi016.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-28T22:10:07.492 INFO:teuthology.orchestra.run.smithi016.stdout:legacy client fscid: 1 2024-03-28T22:10:07.493 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:10:07.493 INFO:teuthology.orchestra.run.smithi016.stdout:Filesystem 'cephfs' (1) 2024-03-28T22:10:07.493 INFO:teuthology.orchestra.run.smithi016.stdout:fs_name cephfs 2024-03-28T22:10:07.493 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 20 2024-03-28T22:10:07.493 INFO:teuthology.orchestra.run.smithi016.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-28T22:10:07.493 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-03-28T22:08:29.672976+0000 2024-03-28T22:10:07.493 INFO:teuthology.orchestra.run.smithi016.stdout:modified 2024-03-28T22:08:34.341708+0000 2024-03-28T22:10:07.493 INFO:teuthology.orchestra.run.smithi016.stdout:tableserver 0 2024-03-28T22:10:07.493 INFO:teuthology.orchestra.run.smithi016.stdout:root 0 2024-03-28T22:10:07.493 INFO:teuthology.orchestra.run.smithi016.stdout:session_timeout 300 2024-03-28T22:10:07.493 INFO:teuthology.orchestra.run.smithi016.stdout:session_autoclose 300 2024-03-28T22:10:07.493 INFO:teuthology.orchestra.run.smithi016.stdout:max_file_size 1099511627776 2024-03-28T22:10:07.493 INFO:teuthology.orchestra.run.smithi016.stdout:max_xattr_size 65536 2024-03-28T22:10:07.494 INFO:teuthology.orchestra.run.smithi016.stdout:required_client_features {} 2024-03-28T22:10:07.494 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure 0 2024-03-28T22:10:07.494 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure_osd_epoch 0 2024-03-28T22:10:07.494 INFO:teuthology.orchestra.run.smithi016.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-28T22:10:07.494 INFO:teuthology.orchestra.run.smithi016.stdout:max_mds 3 2024-03-28T22:10:07.494 INFO:teuthology.orchestra.run.smithi016.stdout:in 0,1,2 2024-03-28T22:10:07.494 INFO:teuthology.orchestra.run.smithi016.stdout:up {0=24566,1=24524,2=24500} 2024-03-28T22:10:07.494 INFO:teuthology.orchestra.run.smithi016.stdout:failed 2024-03-28T22:10:07.494 INFO:teuthology.orchestra.run.smithi016.stdout:damaged 2024-03-28T22:10:07.494 INFO:teuthology.orchestra.run.smithi016.stdout:stopped 2024-03-28T22:10:07.494 INFO:teuthology.orchestra.run.smithi016.stdout:data_pools [3] 2024-03-28T22:10:07.494 INFO:teuthology.orchestra.run.smithi016.stdout:metadata_pool 2 2024-03-28T22:10:07.494 INFO:teuthology.orchestra.run.smithi016.stdout:inline_data disabled 2024-03-28T22:10:07.495 INFO:teuthology.orchestra.run.smithi016.stdout:balancer 2024-03-28T22:10:07.495 INFO:teuthology.orchestra.run.smithi016.stdout:bal_rank_mask -1 2024-03-28T22:10:07.495 INFO:teuthology.orchestra.run.smithi016.stdout:standby_count_wanted 0 2024-03-28T22:10:07.495 INFO:teuthology.orchestra.run.smithi016.stdout:qdb_cluster leader: 24566 members: 24500,24566,24524 2024-03-28T22:10:07.495 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.b{0:24566} state up:active seq 3 addr [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] compat {c=[1],r=[1],i=[1fff]}] 2024-03-28T22:10:07.495 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.i{0:24493} state up:standby-replay seq 1 addr [v2:172.21.15.63:6838/1177197077,v1:172.21.15.63:6839/1177197077] compat {c=[1],r=[1],i=[1fff]}] 2024-03-28T22:10:07.495 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.h{1:24524} state up:active seq 6 addr [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] compat {c=[1],r=[1],i=[1fff]}] 2024-03-28T22:10:07.495 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.c{1:24475} state up:standby-replay seq 1 addr [v2:172.21.15.63:6836/2324231032,v1:172.21.15.63:6837/2324231032] compat {c=[1],r=[1],i=[1fff]}] 2024-03-28T22:10:07.495 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.e{2:24500} state up:active seq 8 addr [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] compat {c=[1],r=[1],i=[1fff]}] 2024-03-28T22:10:07.495 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.f{2:24466} state up:standby-replay seq 1 addr [v2:172.21.15.63:6834/247249857,v1:172.21.15.63:6835/247249857] compat {c=[1],r=[1],i=[1fff]}] 2024-03-28T22:10:07.495 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:10:07.495 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:10:07.495 INFO:teuthology.orchestra.run.smithi016.stdout:Standby daemons: 2024-03-28T22:10:07.495 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:10:07.496 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.j{-1:14541} state up:standby seq 1 addr [v2:172.21.15.16:6834/2478458611,v1:172.21.15.16:6835/2478458611] compat {c=[1],r=[1],i=[1fff]}] 2024-03-28T22:10:07.496 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.d{-1:14553} state up:standby seq 1 addr [v2:172.21.15.16:6836/4019085314,v1:172.21.15.16:6837/4019085314] compat {c=[1],r=[1],i=[1fff]}] 2024-03-28T22:10:07.496 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.g{-1:14619} state up:standby seq 1 addr [v2:172.21.15.16:6838/1644581227,v1:172.21.15.16:6839/1644581227] compat {c=[1],r=[1],i=[1fff]}] 2024-03-28T22:10:07.496 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.a{-1:14631} state up:standby seq 1 addr [v2:172.21.15.16:6840/2593062669,v1:172.21.15.16:6841/2593062669] compat {c=[1],r=[1],i=[1fff]}] 2024-03-28T22:10:07.496 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.l{-1:24409} state up:standby seq 1 addr [v2:172.21.15.63:6832/3228503882,v1:172.21.15.63:6833/3228503882] compat {c=[1],r=[1],i=[1fff]}] 2024-03-28T22:10:07.496 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.k{-1:24458} state up:standby seq 1 addr [v2:172.21.15.53:6832/3748831330,v1:172.21.15.53:6833/3748831330] compat {c=[1],r=[1],i=[1fff]}] 2024-03-28T22:10:07.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.492+0000 7f8958d14700 1 -- 172.21.15.16:0/4217426306 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f89400a1e10 msgr2=0x7f89400a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:07.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.492+0000 7f8958d14700 1 --2- 172.21.15.16:0/4217426306 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f89400a1e10 0x7f89400a42d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f89541032d0 tx=0x7f8944009260 comp rx=0 tx=0).stop 2024-03-28T22:10:07.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.492+0000 7f8958d14700 1 -- 172.21.15.16:0/4217426306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8954103e10 msgr2=0x7f89541a1ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:07.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.492+0000 7f8958d14700 1 --2- 172.21.15.16:0/4217426306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8954103e10 0x7f89541a1ba0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f894c00b770 tx=0x7f894c007a20 comp rx=0 tx=0).stop 2024-03-28T22:10:07.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.492+0000 7f8958d14700 1 -- 172.21.15.16:0/4217426306 shutdown_connections 2024-03-28T22:10:07.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.492+0000 7f8958d14700 1 --2- 172.21.15.16:0/4217426306 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f89400a1e10 0x7f89400a42d0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:07.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.492+0000 7f8958d14700 1 --2- 172.21.15.16:0/4217426306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8954103e10 0x7f89541a1ba0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:07.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.492+0000 7f8958d14700 1 --2- 172.21.15.16:0/4217426306 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8954102270 0x7f895419a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:07.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.492+0000 7f8958d14700 1 --2- 172.21.15.16:0/4217426306 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8954103470 0x7f895419ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:07.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.492+0000 7f8958d14700 1 -- 172.21.15.16:0/4217426306 >> 172.21.15.16:0/4217426306 conn(0x7f89540fd850 msgr2=0x7f8954074470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:07.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.492+0000 7f8958d14700 1 -- 172.21.15.16:0/4217426306 shutdown_connections 2024-03-28T22:10:07.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:07.492+0000 7f8958d14700 1 -- 172.21.15.16:0/4217426306 wait complete. 2024-03-28T22:10:07.498 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 20 2024-03-28T22:10:07.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:07 smithi053 ceph-mon[35979]: pgmap v337: 97 pgs: 97 active+clean; 639 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-28T22:10:08.086 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-28T22:10:08.329 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:10:08.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:08 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/4217426306' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-28T22:10:08.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:08 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/4217426306' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-28T22:10:08.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:08 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/4217426306' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-28T22:10:09.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:09 smithi063 ceph-mon[36249]: pgmap v338: 97 pgs: 97 active+clean; 639 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-03-28T22:10:09.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:09 smithi016 ceph-mon[32374]: pgmap v338: 97 pgs: 97 active+clean; 639 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-03-28T22:10:09.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:09 smithi053 ceph-mon[35979]: pgmap v338: 97 pgs: 97 active+clean; 639 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-03-28T22:10:09.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.974+0000 7f2f825b9700 1 -- 172.21.15.16:0/2605894491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f7c101170 msgr2=0x7f2f7c101590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:09.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.974+0000 7f2f825b9700 1 --2- 172.21.15.16:0/2605894491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f7c101170 0x7f2f7c101590 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c0099c0 tx=0x7f2f6c009cd0 comp rx=0 tx=0).stop 2024-03-28T22:10:09.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.974+0000 7f2f825b9700 1 -- 172.21.15.16:0/2605894491 shutdown_connections 2024-03-28T22:10:09.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.974+0000 7f2f825b9700 1 --2- 172.21.15.16:0/2605894491 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f7c102d10 0x7f2f7c1095c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:09.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.974+0000 7f2f825b9700 1 --2- 172.21.15.16:0/2605894491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f7c101170 0x7f2f7c101590 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:09.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.974+0000 7f2f825b9700 1 --2- 172.21.15.16:0/2605894491 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f7c102370 0x7f2f7c1027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:09.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.974+0000 7f2f825b9700 1 -- 172.21.15.16:0/2605894491 >> 172.21.15.16:0/2605894491 conn(0x7f2f7c0fc710 msgr2=0x7f2f7c0feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:09.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.974+0000 7f2f825b9700 1 -- 172.21.15.16:0/2605894491 shutdown_connections 2024-03-28T22:10:09.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.974+0000 7f2f825b9700 1 -- 172.21.15.16:0/2605894491 wait complete. 2024-03-28T22:10:09.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.975+0000 7f2f825b9700 1 Processor -- start 2024-03-28T22:10:09.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.975+0000 7f2f825b9700 1 -- start start 2024-03-28T22:10:09.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.975+0000 7f2f825b9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f7c101170 0x7f2f7c197430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:09.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.975+0000 7f2f825b9700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f7c102370 0x7f2f7c197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:09.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.975+0000 7f2f825b9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f7c102d10 0x7f2f7c19e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:09.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.975+0000 7f2f825b9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2f7c10cd30 con 0x7f2f7c101170 2024-03-28T22:10:09.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.975+0000 7f2f825b9700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f2f7c10cbb0 con 0x7f2f7c102d10 2024-03-28T22:10:09.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.975+0000 7f2f825b9700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2f7c10ceb0 con 0x7f2f7c102370 2024-03-28T22:10:09.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.975+0000 7f2f7bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f7c101170 0x7f2f7c197430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:09.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.975+0000 7f2f7bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f7c101170 0x7f2f7c197430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38862/0 (socket says 172.21.15.16:38862) 2024-03-28T22:10:09.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.975+0000 7f2f7bfff700 1 -- 172.21.15.16:0/4079880930 learned_addr learned my addr 172.21.15.16:0/4079880930 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:09.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.975+0000 7f2f80b56700 1 --2- 172.21.15.16:0/4079880930 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f7c102d10 0x7f2f7c19e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:09.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.975+0000 7f2f7b7fe700 1 --2- 172.21.15.16:0/4079880930 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f7c102370 0x7f2f7c197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:09.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.976+0000 7f2f7bfff700 1 -- 172.21.15.16:0/4079880930 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f7c102370 msgr2=0x7f2f7c197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:09.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.976+0000 7f2f7bfff700 1 --2- 172.21.15.16:0/4079880930 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f7c102370 0x7f2f7c197970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:09.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.976+0000 7f2f7bfff700 1 -- 172.21.15.16:0/4079880930 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f7c102d10 msgr2=0x7f2f7c19e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:09.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.976+0000 7f2f7bfff700 1 --2- 172.21.15.16:0/4079880930 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f7c102d10 0x7f2f7c19e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:09.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.976+0000 7f2f7bfff700 1 -- 172.21.15.16:0/4079880930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f7c19f090 con 0x7f2f7c101170 2024-03-28T22:10:09.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.976+0000 7f2f7bfff700 1 --2- 172.21.15.16:0/4079880930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f7c101170 0x7f2f7c197430 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c009f90 tx=0x7f2f6c009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:09.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.976+0000 7f2f797fa700 1 -- 172.21.15.16:0/4079880930 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c027070 con 0x7f2f7c101170 2024-03-28T22:10:09.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.976+0000 7f2f797fa700 1 -- 172.21.15.16:0/4079880930 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f6c003f40 con 0x7f2f7c101170 2024-03-28T22:10:09.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.976+0000 7f2f825b9700 1 -- 172.21.15.16:0/4079880930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f7c19f320 con 0x7f2f7c101170 2024-03-28T22:10:09.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.976+0000 7f2f797fa700 1 -- 172.21.15.16:0/4079880930 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c022ca0 con 0x7f2f7c101170 2024-03-28T22:10:09.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.976+0000 7f2f825b9700 1 -- 172.21.15.16:0/4079880930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f7c19f800 con 0x7f2f7c101170 2024-03-28T22:10:09.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.977+0000 7f2f825b9700 1 -- 172.21.15.16:0/4079880930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f7c068f00 con 0x7f2f7c101170 2024-03-28T22:10:09.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.978+0000 7f2f797fa700 1 -- 172.21.15.16:0/4079880930 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f2f6c02a070 con 0x7f2f7c101170 2024-03-28T22:10:09.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.978+0000 7f2f797fa700 1 --2- 172.21.15.16:0/4079880930 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f2f680a1e10 0x7f2f680a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:09.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.979+0000 7f2f7b7fe700 1 --2- 172.21.15.16:0/4079880930 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f2f680a1e10 0x7f2f680a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:09.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.979+0000 7f2f797fa700 1 -- 172.21.15.16:0/4079880930 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f2f6c0cec70 con 0x7f2f7c101170 2024-03-28T22:10:09.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.979+0000 7f2f7b7fe700 1 --2- 172.21.15.16:0/4079880930 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f2f680a1e10 0x7f2f680a42d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2f64007c60 tx=0x7f2f640073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:09.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:09.983+0000 7f2f797fa700 1 -- 172.21.15.16:0/4079880930 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2f6c05e280 con 0x7f2f7c101170 2024-03-28T22:10:10.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:10.188+0000 7f2f825b9700 1 -- 172.21.15.16:0/4079880930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f2f7c062e50 con 0x7f2f7c101170 2024-03-28T22:10:10.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:10.189+0000 7f2f797fa700 1 -- 172.21.15.16:0/4079880930 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v88) ==== 56+0+5221 (secure 0 0 0) 0x7f2f6c013070 con 0x7f2f7c101170 2024-03-28T22:10:10.190 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 88 2024-03-28T22:10:10.191 INFO:teuthology.orchestra.run.smithi016.stdout:fsid 798d9884-ed4e-11ee-95d0-87774f69a715 2024-03-28T22:10:10.191 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-03-28T22:00:27.216099+0000 2024-03-28T22:10:10.191 INFO:teuthology.orchestra.run.smithi016.stdout:modified 2024-03-28T22:09:30.047847+0000 2024-03-28T22:10:10.191 INFO:teuthology.orchestra.run.smithi016.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-28T22:10:10.191 INFO:teuthology.orchestra.run.smithi016.stdout:crush_version 26 2024-03-28T22:10:10.191 INFO:teuthology.orchestra.run.smithi016.stdout:full_ratio 0.95 2024-03-28T22:10:10.191 INFO:teuthology.orchestra.run.smithi016.stdout:backfillfull_ratio 0.9 2024-03-28T22:10:10.191 INFO:teuthology.orchestra.run.smithi016.stdout:nearfull_ratio 0.85 2024-03-28T22:10:10.191 INFO:teuthology.orchestra.run.smithi016.stdout:require_min_compat_client luminous 2024-03-28T22:10:10.191 INFO:teuthology.orchestra.run.smithi016.stdout:min_compat_client luminous 2024-03-28T22:10:10.191 INFO:teuthology.orchestra.run.smithi016.stdout:require_osd_release squid 2024-03-28T22:10:10.191 INFO:teuthology.orchestra.run.smithi016.stdout:stretch_mode_enabled false 2024-03-28T22:10:10.192 INFO:teuthology.orchestra.run.smithi016.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-28T22:10:10.192 INFO:teuthology.orchestra.run.smithi016.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.25 2024-03-28T22:10:10.192 INFO:teuthology.orchestra.run.smithi016.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-03-28T22:10:10.192 INFO:teuthology.orchestra.run.smithi016.stdout:max_osd 12 2024-03-28T22:10:10.192 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 up in weight 1 up_from 8 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6802/1589692483,v1:172.21.15.16:6803/1589692483] [v2:172.21.15.16:6804/1589692483,v1:172.21.15.16:6805/1589692483] exists,up 46e7c8f8-ecc8-43d6-ae53-0655728f3240 2024-03-28T22:10:10.192 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 up in weight 1 up_from 13 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.53:6800/795531234,v1:172.21.15.53:6801/795531234] [v2:172.21.15.53:6802/795531234,v1:172.21.15.53:6803/795531234] exists,up af6d9c22-2830-4ce2-a547-011df0c6cbad 2024-03-28T22:10:10.192 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 up in weight 1 up_from 18 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.63:6800/2214131086,v1:172.21.15.63:6801/2214131086] [v2:172.21.15.63:6802/2214131086,v1:172.21.15.63:6803/2214131086] exists,up b10adca9-110c-4f87-85e0-b7542709d31a 2024-03-28T22:10:10.192 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 up in weight 1 up_from 26 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6810/13176789,v1:172.21.15.16:6811/13176789] [v2:172.21.15.16:6812/13176789,v1:172.21.15.16:6813/13176789] exists,up c83cf4bb-395a-4786-abf7-74ca93a33eb3 2024-03-28T22:10:10.192 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 up in weight 1 up_from 34 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.53:6808/772161849,v1:172.21.15.53:6809/772161849] [v2:172.21.15.53:6810/772161849,v1:172.21.15.53:6811/772161849] exists,up 941b3e54-47d6-4e85-8069-e730c3fa8eec 2024-03-28T22:10:10.192 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 up in weight 1 up_from 39 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.63:6808/1602554852,v1:172.21.15.63:6809/1602554852] [v2:172.21.15.63:6810/1602554852,v1:172.21.15.63:6811/1602554852] exists,up 51d89826-1d60-40fc-8fd8-09bfe1ab4125 2024-03-28T22:10:10.192 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 up in weight 1 up_from 47 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6818/4290459224,v1:172.21.15.16:6819/4290459224] [v2:172.21.15.16:6820/4290459224,v1:172.21.15.16:6821/4290459224] exists,up 978efe5d-2c55-4ff8-8a1e-9f891388945e 2024-03-28T22:10:10.192 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 up in weight 1 up_from 52 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.53:6816/1056467525,v1:172.21.15.53:6817/1056467525] [v2:172.21.15.53:6818/1056467525,v1:172.21.15.53:6819/1056467525] exists,up 50d489cb-dbb8-48df-b868-b29edf12e04d 2024-03-28T22:10:10.192 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8 up in weight 1 up_from 60 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.63:6816/2918516794,v1:172.21.15.63:6817/2918516794] [v2:172.21.15.63:6818/2918516794,v1:172.21.15.63:6819/2918516794] exists,up 49b39414-cc9c-4057-ae3f-cc109cd979da 2024-03-28T22:10:10.193 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9 up in weight 1 up_from 68 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6826/1585624367,v1:172.21.15.16:6827/1585624367] [v2:172.21.15.16:6828/1585624367,v1:172.21.15.16:6829/1585624367] exists,up 4c873e51-b783-4eb6-83a3-fee31722d60b 2024-03-28T22:10:10.193 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10 up in weight 1 up_from 73 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.53:6824/3357446484,v1:172.21.15.53:6825/3357446484] [v2:172.21.15.53:6826/3357446484,v1:172.21.15.53:6827/3357446484] exists,up 085a78bf-e2ad-481a-85a3-71d1fc549a0c 2024-03-28T22:10:10.193 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11 up in weight 1 up_from 81 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.63:6824/2167866570,v1:172.21.15.63:6825/2167866570] [v2:172.21.15.63:6826/2167866570,v1:172.21.15.63:6827/2167866570] exists,up 13302aa3-08b9-4639-9042-27d850b4d5f1 2024-03-28T22:10:10.193 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 2.4 [2,11] 2024-03-28T22:10:10.193 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 2.a [2,11] 2024-03-28T22:10:10.193 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 3.2 [3,9] 2024-03-28T22:10:10.193 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 3.11 [4,8] 2024-03-28T22:10:10.193 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 3.13 [4,9] 2024-03-28T22:10:10.193 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 3.1b [5,9] 2024-03-28T22:10:10.193 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 3.23 [4,7] 2024-03-28T22:10:10.193 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/3821408150 expires 2024-03-29T22:01:28.993434+0000 2024-03-28T22:10:10.194 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:6801/1832422779 expires 2024-03-29T22:01:28.993434+0000 2024-03-28T22:10:10.194 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/996598130 expires 2024-03-29T22:01:02.448898+0000 2024-03-28T22:10:10.194 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/1062264551 expires 2024-03-29T22:01:28.993434+0000 2024-03-28T22:10:10.194 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/317557779 expires 2024-03-29T22:01:02.448898+0000 2024-03-28T22:10:10.194 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:6801/2859950725 expires 2024-03-29T22:01:02.448898+0000 2024-03-28T22:10:10.194 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/3631746518 expires 2024-03-29T22:01:28.993434+0000 2024-03-28T22:10:10.194 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:6800/2859950725 expires 2024-03-29T22:01:02.448898+0000 2024-03-28T22:10:10.194 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/3116652786 expires 2024-03-29T22:01:02.448898+0000 2024-03-28T22:10:10.195 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:6800/1832422779 expires 2024-03-29T22:01:28.993434+0000 2024-03-28T22:10:10.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:10.191+0000 7f2f825b9700 1 -- 172.21.15.16:0/4079880930 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f2f680a1e10 msgr2=0x7f2f680a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:10.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:10.191+0000 7f2f825b9700 1 --2- 172.21.15.16:0/4079880930 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f2f680a1e10 0x7f2f680a42d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2f64007c60 tx=0x7f2f640073d0 comp rx=0 tx=0).stop 2024-03-28T22:10:10.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:10.191+0000 7f2f825b9700 1 -- 172.21.15.16:0/4079880930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f7c101170 msgr2=0x7f2f7c197430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:10.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:10.191+0000 7f2f825b9700 1 --2- 172.21.15.16:0/4079880930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f7c101170 0x7f2f7c197430 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c009f90 tx=0x7f2f6c009580 comp rx=0 tx=0).stop 2024-03-28T22:10:10.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:10.191+0000 7f2f825b9700 1 -- 172.21.15.16:0/4079880930 shutdown_connections 2024-03-28T22:10:10.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:10.191+0000 7f2f825b9700 1 --2- 172.21.15.16:0/4079880930 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f2f680a1e10 0x7f2f680a42d0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:10.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:10.191+0000 7f2f825b9700 1 --2- 172.21.15.16:0/4079880930 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f7c102d10 0x7f2f7c19e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:10.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:10.191+0000 7f2f825b9700 1 --2- 172.21.15.16:0/4079880930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f7c101170 0x7f2f7c197430 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:10.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:10.191+0000 7f2f825b9700 1 --2- 172.21.15.16:0/4079880930 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f7c102370 0x7f2f7c197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:10.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:10.191+0000 7f2f825b9700 1 -- 172.21.15.16:0/4079880930 >> 172.21.15.16:0/4079880930 conn(0x7f2f7c0fc710 msgr2=0x7f2f7c1056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:10.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:10.191+0000 7f2f825b9700 1 -- 172.21.15.16:0/4079880930 shutdown_connections 2024-03-28T22:10:10.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:10.191+0000 7f2f825b9700 1 -- 172.21.15.16:0/4079880930 wait complete. 2024-03-28T22:10:10.484 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:10 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/4079880930' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-28T22:10:10.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:10 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/4079880930' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-28T22:10:10.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:10 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/4079880930' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-28T22:10:10.889 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-28T22:10:10.904 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:10:11.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.009+0000 7faa9e847640 1 -- 172.21.15.16:0/1512889112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa98103990 msgr2=0x7faa98103db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:11.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.009+0000 7faa9e847640 1 --2- 172.21.15.16:0/1512889112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa98103990 0x7faa98103db0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7faa88009900 tx=0x7faa8802bc30 comp rx=0 tx=0).stop 2024-03-28T22:10:11.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.009+0000 7faa9e847640 1 -- 172.21.15.16:0/1512889112 shutdown_connections 2024-03-28T22:10:11.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.009+0000 7faa9e847640 1 --2- 172.21.15.16:0/1512889112 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa98105540 0x7faa9810bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:11.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.009+0000 7faa9e847640 1 --2- 172.21.15.16:0/1512889112 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa98104b80 0x7faa98105000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:11.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.009+0000 7faa9e847640 1 --2- 172.21.15.16:0/1512889112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa98103990 0x7faa98103db0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:11.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.009+0000 7faa9e847640 1 -- 172.21.15.16:0/1512889112 >> 172.21.15.16:0/1512889112 conn(0x7faa980fef00 msgr2=0x7faa98101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:11.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.009+0000 7faa9e847640 1 -- 172.21.15.16:0/1512889112 shutdown_connections 2024-03-28T22:10:11.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.009+0000 7faa9e847640 1 -- 172.21.15.16:0/1512889112 wait complete. 2024-03-28T22:10:11.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.009+0000 7faa9e847640 1 Processor -- start 2024-03-28T22:10:11.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.009+0000 7faa9e847640 1 -- start start 2024-03-28T22:10:11.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.010+0000 7faa9e847640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa98103990 0x7faa98199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:11.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.010+0000 7faa9e847640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa98104b80 0x7faa9819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:11.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.010+0000 7faa9e847640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa98105540 0x7faa981a1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:11.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.010+0000 7faa9e847640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faa9810f630 con 0x7faa98105540 2024-03-28T22:10:11.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.010+0000 7faa9e847640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7faa9810f4b0 con 0x7faa98104b80 2024-03-28T22:10:11.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.010+0000 7faa9e847640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7faa9810f7b0 con 0x7faa98103990 2024-03-28T22:10:11.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.010+0000 7faa9cde4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa98105540 0x7faa981a1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:11.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.010+0000 7faa977fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa98104b80 0x7faa9819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:11.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.010+0000 7faa9cde4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa98105540 0x7faa981a1220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43500/0 (socket says 172.21.15.16:43500) 2024-03-28T22:10:11.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.010+0000 7faa977fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa98104b80 0x7faa9819a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:53916/0 (socket says 172.21.15.16:53916) 2024-03-28T22:10:11.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.010+0000 7faa9cde4640 1 -- 172.21.15.16:0/729378226 learned_addr learned my addr 172.21.15.16:0/729378226 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:11.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.010+0000 7faa97fff640 1 --2- 172.21.15.16:0/729378226 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa98103990 0x7faa98199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:11.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.010+0000 7faa9cde4640 1 -- 172.21.15.16:0/729378226 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa98103990 msgr2=0x7faa98199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:11.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.010+0000 7faa9cde4640 1 --2- 172.21.15.16:0/729378226 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa98103990 0x7faa98199c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:11.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.010+0000 7faa9cde4640 1 -- 172.21.15.16:0/729378226 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa98104b80 msgr2=0x7faa9819a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:11.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.010+0000 7faa9cde4640 1 --2- 172.21.15.16:0/729378226 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa98104b80 0x7faa9819a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:11.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.010+0000 7faa9cde4640 1 -- 172.21.15.16:0/729378226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa981a18c0 con 0x7faa98105540 2024-03-28T22:10:11.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.010+0000 7faa9cde4640 1 --2- 172.21.15.16:0/729378226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa98105540 0x7faa981a1220 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7faa9000cad0 tx=0x7faa9000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:11.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.011+0000 7faa957fa640 1 -- 172.21.15.16:0/729378226 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa9000ae20 con 0x7faa98105540 2024-03-28T22:10:11.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.011+0000 7faa957fa640 1 -- 172.21.15.16:0/729378226 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa90005020 con 0x7faa98105540 2024-03-28T22:10:11.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.011+0000 7faa957fa640 1 -- 172.21.15.16:0/729378226 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa9000f6f0 con 0x7faa98105540 2024-03-28T22:10:11.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.011+0000 7faa9e847640 1 -- 172.21.15.16:0/729378226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa981a1bb0 con 0x7faa98105540 2024-03-28T22:10:11.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.011+0000 7faa9e847640 1 -- 172.21.15.16:0/729378226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa981a20d0 con 0x7faa98105540 2024-03-28T22:10:11.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.012+0000 7faa9e847640 1 -- 172.21.15.16:0/729378226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa9810acb0 con 0x7faa98105540 2024-03-28T22:10:11.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.013+0000 7faa957fa640 1 -- 172.21.15.16:0/729378226 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7faa900051d0 con 0x7faa98105540 2024-03-28T22:10:11.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.014+0000 7faa957fa640 1 --2- 172.21.15.16:0/729378226 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7faa800a1e00 0x7faa800a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:11.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.014+0000 7faa97fff640 1 --2- 172.21.15.16:0/729378226 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7faa800a1e00 0x7faa800a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:11.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.014+0000 7faa957fa640 1 -- 172.21.15.16:0/729378226 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7faa900cdfa0 con 0x7faa98105540 2024-03-28T22:10:11.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.014+0000 7faa97fff640 1 --2- 172.21.15.16:0/729378226 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7faa800a1e00 0x7faa800a42c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7faa8802c170 tx=0x7faa8802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:11.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.018+0000 7faa957fa640 1 -- 172.21.15.16:0/729378226 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7faa90014030 con 0x7faa98105540 2024-03-28T22:10:11.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.204+0000 7faa9e847640 1 -- 172.21.15.16:0/729378226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faa9802d3c0 con 0x7faa98105540 2024-03-28T22:10:11.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.206+0000 7faa957fa640 1 -- 172.21.15.16:0/729378226 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22441 (secure 0 0 0) 0x7faa9009aed0 con 0x7faa98105540 2024-03-28T22:10:11.208 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:10:11.208 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":88,"fsid":"798d9884-ed4e-11ee-95d0-87774f69a715","created":"2024-03-28T22:00:27.216099+0000","modified":"2024-03-28T22:09:30.047847+0000","last_up_change":"2024-03-28T22:07:40.251141+0000","last_in_change":"2024-03-28T22:07:19.456451+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:03:59.327832+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:08:28.123529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:08:28.919037+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"46e7c8f8-ecc8-43d6-ae53-0655728f3240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6803","nonce":1589692483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6805","nonce":1589692483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6809","nonce":1589692483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6807","nonce":1589692483}]},"public_addr":"172.21.15.16:6803/1589692483","cluster_addr":"172.21.15.16:6805/1589692483","heartbeat_back_addr":"172.21.15.16:6809/1589692483","heartbeat_front_addr":"172.21.15.16:6807/1589692483","state":["exists","up"]},{"osd":1,"uuid":"af6d9c22-2830-4ce2-a547-011df0c6cbad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6801","nonce":795531234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6803","nonce":795531234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6807","nonce":795531234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6805","nonce":795531234}]},"public_addr":"172.21.15.53:6801/795531234","cluster_addr":"172.21.15.53:6803/795531234","heartbeat_back_addr":"172.21.15.53:6807/795531234","heartbeat_front_addr":"172.21.15.53:6805/795531234","state":["exists","up"]},{"osd":2,"uuid":"b10adca9-110c-4f87-85e0-b7542709d31a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6801","nonce":2214131086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6803","nonce":2214131086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6807","nonce":2214131086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6805","nonce":2214131086}]},"public_addr":"172.21.15.63:6801/2214131086","cluster_addr":"172.21.15.63:6803/2214131086","heartbeat_back_addr":"172.21.15.63:6807/2214131086","heartbeat_front_addr":"172.21.15.63:6805/2214131086","state":["exists","up"]},{"osd":3,"uuid":"c83cf4bb-395a-4786-abf7-74ca93a33eb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6811","nonce":13176789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6813","nonce":13176789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6817","nonce":13176789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6815","nonce":13176789}]},"public_addr":"172.21.15.16:6811/13176789","cluster_addr":"172.21.15.16:6813/13176789","heartbeat_back_addr":"172.21.15.16:6817/13176789","heartbeat_front_addr":"172.21.15.16:6815/13176789","state":["exists","up"]},{"osd":4,"uuid":"941b3e54-47d6-4e85-8069-e730c3fa8eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6809","nonce":772161849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6811","nonce":772161849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6815","nonce":772161849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6813","nonce":772161849}]},"public_addr":"172.21.15.53:6809/772161849","cluster_addr":"172.21.15.53:6811/772161849","heartbeat_back_addr":"172.21.15.53:6815/772161849","heartbeat_front_addr":"172.21.15.53:6813/772161849","state":["exists","up"]},{"osd":5,"uuid":"51d89826-1d60-40fc-8fd8-09bfe1ab4125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6809","nonce":1602554852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6811","nonce":1602554852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6815","nonce":1602554852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6813","nonce":1602554852}]},"public_addr":"172.21.15.63:6809/1602554852","cluster_addr":"172.21.15.63:6811/1602554852","heartbeat_back_addr":"172.21.15.63:6815/1602554852","heartbeat_front_addr":"172.21.15.63:6813/1602554852","state":["exists","up"]},{"osd":6,"uuid":"978efe5d-2c55-4ff8-8a1e-9f891388945e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6819","nonce":4290459224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6821","nonce":4290459224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6825","nonce":4290459224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6823","nonce":4290459224}]},"public_addr":"172.21.15.16:6819/4290459224","cluster_addr":"172.21.15.16:6821/4290459224","heartbeat_back_addr":"172.21.15.16:6825/4290459224","heartbeat_front_addr":"172.21.15.16:6823/4290459224","state":["exists","up"]},{"osd":7,"uuid":"50d489cb-dbb8-48df-b868-b29edf12e04d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6817","nonce":1056467525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6819","nonce":1056467525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6823","nonce":1056467525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6821","nonce":1056467525}]},"public_addr":"172.21.15.53:6817/1056467525","cluster_addr":"172.21.15.53:6819/1056467525","heartbeat_back_addr":"172.21.15.53:6823/1056467525","heartbeat_front_addr":"172.21.15.53:6821/1056467525","state":["exists","up"]},{"osd":8,"uuid":"49b39414-cc9c-4057-ae3f-cc109cd979da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6817","nonce":2918516794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6819","nonce":2918516794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6823","nonce":2918516794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6821","nonce":2918516794}]},"public_addr":"172.21.15.63:6817/2918516794","cluster_addr":"172.21.15.63:6819/2918516794","heartbeat_back_addr":"172.21.15.63:6823/2918516794","heartbeat_front_addr":"172.21.15.63:6821/2918516794","state":["exists","up"]},{"osd":9,"uuid":"4c873e51-b783-4eb6-83a3-fee31722d60b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6827","nonce":1585624367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6829","nonce":1585624367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6833","nonce":1585624367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6831","nonce":1585624367}]},"public_addr":"172.21.15.16:6827/1585624367","cluster_addr":"172.21.15.16:6829/1585624367","heartbeat_back_addr":"172.21.15.16:6833/1585624367","heartbeat_front_addr":"172.21.15.16:6831/1585624367","state":["exists","up"]},{"osd":10,"uuid":"085a78bf-e2ad-481a-85a3-71d1fc549a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6825","nonce":3357446484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6827","nonce":3357446484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6831","nonce":3357446484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6829","nonce":3357446484}]},"public_addr":"172.21.15.53:6825/3357446484","cluster_addr":"172.21.15.53:6827/3357446484","heartbeat_back_addr":"172.21.15.53:6831/3357446484","heartbeat_front_addr":"172.21.15.53:6829/3357446484","state":["exists","up"]},{"osd":11,"uuid":"13302aa3-08b9-4639-9042-27d850b4d5f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6825","nonce":2167866570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6827","nonce":2167866570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6831","nonce":2167866570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6829","nonce":2167866570}]},"public_addr":"172.21.15.63:6825/2167866570","cluster_addr":"172.21.15.63:6827/2167866570","heartbeat_back_addr":"172.21.15.63:6831/2167866570","heartbeat_front_addr":"172.21.15.63:6829/2167866570","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:07.450323+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:31.363172+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:55.295021+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:19.794760+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:44.063492+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:08.195064+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:33.655328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:57.639414+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:22.402394+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:47.462050+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:12.226307+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:37.495625+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3821408150":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6801/1832422779":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/996598130":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/1062264551":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/317557779":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6801/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/3631746518":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6800/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/3116652786":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6800/1832422779":"2024-03-29T22:01:28.993434+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:10:11.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.208+0000 7faa9e847640 1 -- 172.21.15.16:0/729378226 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7faa800a1e00 msgr2=0x7faa800a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:11.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.208+0000 7faa9e847640 1 --2- 172.21.15.16:0/729378226 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7faa800a1e00 0x7faa800a42c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7faa8802c170 tx=0x7faa8802df90 comp rx=0 tx=0).stop 2024-03-28T22:10:11.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.208+0000 7faa9e847640 1 -- 172.21.15.16:0/729378226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa98105540 msgr2=0x7faa981a1220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:11.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.208+0000 7faa9e847640 1 --2- 172.21.15.16:0/729378226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa98105540 0x7faa981a1220 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7faa9000cad0 tx=0x7faa9000cf90 comp rx=0 tx=0).stop 2024-03-28T22:10:11.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.208+0000 7faa9e847640 1 -- 172.21.15.16:0/729378226 shutdown_connections 2024-03-28T22:10:11.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.208+0000 7faa9e847640 1 --2- 172.21.15.16:0/729378226 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7faa800a1e00 0x7faa800a42c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:11.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.208+0000 7faa9e847640 1 --2- 172.21.15.16:0/729378226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa98105540 0x7faa981a1220 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:11.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.208+0000 7faa9e847640 1 --2- 172.21.15.16:0/729378226 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa98104b80 0x7faa9819a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:11.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.208+0000 7faa9e847640 1 --2- 172.21.15.16:0/729378226 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa98103990 0x7faa98199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:11.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.208+0000 7faa9e847640 1 -- 172.21.15.16:0/729378226 >> 172.21.15.16:0/729378226 conn(0x7faa980fef00 msgr2=0x7faa98107e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:11.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.208+0000 7faa9e847640 1 -- 172.21.15.16:0/729378226 shutdown_connections 2024-03-28T22:10:11.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.208+0000 7faa9e847640 1 -- 172.21.15.16:0/729378226 wait complete. 2024-03-28T22:10:11.221 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T22:03:59.327832+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T22:08:28.123529+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T22:08:28.919037+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T22:10:11.221 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T22:10:11.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:11 smithi053 ceph-mon[35979]: pgmap v339: 97 pgs: 97 active+clean; 639 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-28T22:10:11.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:11 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/729378226' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:10:11.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.326+0000 7f6972423640 1 -- 172.21.15.16:0/1432331643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f696c1021d0 msgr2=0x7f696c1045c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:11.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.326+0000 7f6972423640 1 --2- 172.21.15.16:0/1432331643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f696c1021d0 0x7f696c1045c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f695c009900 tx=0x7f695c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:10:11.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.326+0000 7f6972423640 1 -- 172.21.15.16:0/1432331643 shutdown_connections 2024-03-28T22:10:11.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.326+0000 7f6972423640 1 --2- 172.21.15.16:0/1432331643 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f696c107430 0x7f696c109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:11.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.326+0000 7f6972423640 1 --2- 172.21.15.16:0/1432331643 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f696c104b00 0x7f696c106ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:11.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.326+0000 7f6972423640 1 --2- 172.21.15.16:0/1432331643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f696c1021d0 0x7f696c1045c0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:11.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.326+0000 7f6972423640 1 -- 172.21.15.16:0/1432331643 >> 172.21.15.16:0/1432331643 conn(0x7f696c0fbb80 msgr2=0x7f696c0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:11.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.326+0000 7f6972423640 1 -- 172.21.15.16:0/1432331643 shutdown_connections 2024-03-28T22:10:11.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.326+0000 7f6972423640 1 -- 172.21.15.16:0/1432331643 wait complete. 2024-03-28T22:10:11.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.326+0000 7f6972423640 1 Processor -- start 2024-03-28T22:10:11.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.326+0000 7f6972423640 1 -- start start 2024-03-28T22:10:11.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.327+0000 7f6972423640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f696c1021d0 0x7f696c101b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:11.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.327+0000 7f6972423640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f696c104b00 0x7f696c100180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:11.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.327+0000 7f6972423640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f696c107430 0x7f696c1006c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:11.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.327+0000 7f6972423640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f696c10d400 con 0x7f696c107430 2024-03-28T22:10:11.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.327+0000 7f6972423640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f696c10d280 con 0x7f696c1021d0 2024-03-28T22:10:11.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.327+0000 7f6972423640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f696c10d580 con 0x7f696c104b00 2024-03-28T22:10:11.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.327+0000 7f69709c0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f696c107430 0x7f696c1006c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:11.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.327+0000 7f696bfff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f696c1021d0 0x7f696c101b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:11.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.327+0000 7f69709c0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f696c107430 0x7f696c1006c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43520/0 (socket says 172.21.15.16:43520) 2024-03-28T22:10:11.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.327+0000 7f69709c0640 1 -- 172.21.15.16:0/2191897473 learned_addr learned my addr 172.21.15.16:0/2191897473 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:11.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.327+0000 7f696b7fe640 1 --2- 172.21.15.16:0/2191897473 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f696c104b00 0x7f696c100180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:11.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.327+0000 7f696bfff640 1 -- 172.21.15.16:0/2191897473 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f696c104b00 msgr2=0x7f696c100180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:11.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.327+0000 7f696bfff640 1 --2- 172.21.15.16:0/2191897473 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f696c104b00 0x7f696c100180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:11.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.327+0000 7f696bfff640 1 -- 172.21.15.16:0/2191897473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f696c107430 msgr2=0x7f696c1006c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:11.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.327+0000 7f696bfff640 1 --2- 172.21.15.16:0/2191897473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f696c107430 0x7f696c1006c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:11.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.327+0000 7f696bfff640 1 -- 172.21.15.16:0/2191897473 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f696c100f50 con 0x7f696c1021d0 2024-03-28T22:10:11.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.327+0000 7f69709c0640 1 --2- 172.21.15.16:0/2191897473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f696c107430 0x7f696c1006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:10:11.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.327+0000 7f696bfff640 1 --2- 172.21.15.16:0/2191897473 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f696c1021d0 0x7f696c101b40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f695c009230 tx=0x7f695c02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:11.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.327+0000 7f69697fa640 1 -- 172.21.15.16:0/2191897473 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f695c041070 con 0x7f696c1021d0 2024-03-28T22:10:11.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.328+0000 7f6972423640 1 -- 172.21.15.16:0/2191897473 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f696c1a1b30 con 0x7f696c1021d0 2024-03-28T22:10:11.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.328+0000 7f69697fa640 1 -- 172.21.15.16:0/2191897473 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f695c004450 con 0x7f696c1021d0 2024-03-28T22:10:11.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.328+0000 7f69697fa640 1 -- 172.21.15.16:0/2191897473 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f695c003df0 con 0x7f696c1021d0 2024-03-28T22:10:11.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.328+0000 7f6972423640 1 -- 172.21.15.16:0/2191897473 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f696c1a2080 con 0x7f696c1021d0 2024-03-28T22:10:11.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.328+0000 7f6972423640 1 -- 172.21.15.16:0/2191897473 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f696c0fd670 con 0x7f696c1021d0 2024-03-28T22:10:11.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.330+0000 7f69697fa640 1 -- 172.21.15.16:0/2191897473 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f695c044070 con 0x7f696c1021d0 2024-03-28T22:10:11.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.330+0000 7f69697fa640 1 --2- 172.21.15.16:0/2191897473 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f69540a1d80 0x7f69540a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:11.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.330+0000 7f696b7fe640 1 --2- 172.21.15.16:0/2191897473 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f69540a1d80 0x7f69540a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:11.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.330+0000 7f69697fa640 1 -- 172.21.15.16:0/2191897473 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f695c0ef7c0 con 0x7f696c1021d0 2024-03-28T22:10:11.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.331+0000 7f696b7fe640 1 --2- 172.21.15.16:0/2191897473 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f69540a1d80 0x7f69540a4240 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f696c1018b0 tx=0x7f6960005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:11.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.334+0000 7f69697fa640 1 -- 172.21.15.16:0/2191897473 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f695c0bc570 con 0x7f696c1021d0 2024-03-28T22:10:11.473 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:11 smithi016 ceph-mon[32374]: pgmap v339: 97 pgs: 97 active+clean; 639 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-28T22:10:11.474 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:11 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/729378226' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:10:11.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.515+0000 7f6972423640 1 -- 172.21.15.16:0/2191897473 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f696c02d300 con 0x7f696c1021d0 2024-03-28T22:10:11.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.516+0000 7f69697fa640 1 -- 172.21.15.16:0/2191897473 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f695c0c0610 con 0x7f696c1021d0 2024-03-28T22:10:11.518 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-28T22:10:11.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.518+0000 7f6972423640 1 -- 172.21.15.16:0/2191897473 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f69540a1d80 msgr2=0x7f69540a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:11.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.518+0000 7f6972423640 1 --2- 172.21.15.16:0/2191897473 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f69540a1d80 0x7f69540a4240 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f696c1018b0 tx=0x7f6960005f90 comp rx=0 tx=0).stop 2024-03-28T22:10:11.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.518+0000 7f6972423640 1 -- 172.21.15.16:0/2191897473 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f696c1021d0 msgr2=0x7f696c101b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:11.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.518+0000 7f6972423640 1 --2- 172.21.15.16:0/2191897473 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f696c1021d0 0x7f696c101b40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f695c009230 tx=0x7f695c02c650 comp rx=0 tx=0).stop 2024-03-28T22:10:11.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.518+0000 7f6972423640 1 -- 172.21.15.16:0/2191897473 shutdown_connections 2024-03-28T22:10:11.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.518+0000 7f6972423640 1 --2- 172.21.15.16:0/2191897473 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f69540a1d80 0x7f69540a4240 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:11.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.518+0000 7f6972423640 1 --2- 172.21.15.16:0/2191897473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f696c107430 0x7f696c1006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:11.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.518+0000 7f6972423640 1 --2- 172.21.15.16:0/2191897473 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f696c104b00 0x7f696c100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:11.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.518+0000 7f6972423640 1 --2- 172.21.15.16:0/2191897473 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f696c1021d0 0x7f696c101b40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:11.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.518+0000 7f6972423640 1 -- 172.21.15.16:0/2191897473 >> 172.21.15.16:0/2191897473 conn(0x7f696c0fbb80 msgr2=0x7f696c102bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:11.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.518+0000 7f6972423640 1 -- 172.21.15.16:0/2191897473 shutdown_connections 2024-03-28T22:10:11.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.518+0000 7f6972423640 1 -- 172.21.15.16:0/2191897473 wait complete. 2024-03-28T22:10:11.531 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T22:10:11.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.634+0000 7fc0c0e8f640 1 -- 172.21.15.16:0/196576978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0bc104b20 msgr2=0x7fc0bc106f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:11.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.634+0000 7fc0c0e8f640 1 --2- 172.21.15.16:0/196576978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0bc104b20 0x7fc0bc106f10 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fc0a4009900 tx=0x7fc0a402bc30 comp rx=0 tx=0).stop 2024-03-28T22:10:11.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.634+0000 7fc0c0e8f640 1 -- 172.21.15.16:0/196576978 shutdown_connections 2024-03-28T22:10:11.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.634+0000 7fc0c0e8f640 1 --2- 172.21.15.16:0/196576978 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0bc107450 0x7fc0bc1098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:11.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.634+0000 7fc0c0e8f640 1 --2- 172.21.15.16:0/196576978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0bc104b20 0x7fc0bc106f10 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:11.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.634+0000 7fc0c0e8f640 1 --2- 172.21.15.16:0/196576978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc0bc1021f0 0x7fc0bc1045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:11.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.634+0000 7fc0c0e8f640 1 -- 172.21.15.16:0/196576978 >> 172.21.15.16:0/196576978 conn(0x7fc0bc0fbbe0 msgr2=0x7fc0bc0fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:11.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.634+0000 7fc0c0e8f640 1 -- 172.21.15.16:0/196576978 shutdown_connections 2024-03-28T22:10:11.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.634+0000 7fc0c0e8f640 1 -- 172.21.15.16:0/196576978 wait complete. 2024-03-28T22:10:11.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.635+0000 7fc0c0e8f640 1 Processor -- start 2024-03-28T22:10:11.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.635+0000 7fc0c0e8f640 1 -- start start 2024-03-28T22:10:11.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.635+0000 7fc0c0e8f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc0bc1021f0 0x7fc0bc197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:11.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.635+0000 7fc0c0e8f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0bc104b20 0x7fc0bc198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:11.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.635+0000 7fc0c0e8f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0bc107450 0x7fc0bc19f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:11.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.635+0000 7fc0c0e8f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc0bc10d400 con 0x7fc0bc107450 2024-03-28T22:10:11.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.635+0000 7fc0c0e8f640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc0bc10d280 con 0x7fc0bc104b20 2024-03-28T22:10:11.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.635+0000 7fc0c0e8f640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc0bc10d580 con 0x7fc0bc1021f0 2024-03-28T22:10:11.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.635+0000 7fc0bad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0bc107450 0x7fc0bc19f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:11.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.635+0000 7fc0bad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0bc107450 0x7fc0bc19f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43544/0 (socket says 172.21.15.16:43544) 2024-03-28T22:10:11.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.635+0000 7fc0bad9d640 1 -- 172.21.15.16:0/2762963886 learned_addr learned my addr 172.21.15.16:0/2762963886 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:11.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.635+0000 7fc0b3fff640 1 --2- 172.21.15.16:0/2762963886 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0bc104b20 0x7fc0bc198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:11.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.635+0000 7fc0ba59c640 1 --2- 172.21.15.16:0/2762963886 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc0bc1021f0 0x7fc0bc197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:11.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.635+0000 7fc0bad9d640 1 -- 172.21.15.16:0/2762963886 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc0bc1021f0 msgr2=0x7fc0bc197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:11.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.635+0000 7fc0bad9d640 1 --2- 172.21.15.16:0/2762963886 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc0bc1021f0 0x7fc0bc197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:11.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.635+0000 7fc0bad9d640 1 -- 172.21.15.16:0/2762963886 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0bc104b20 msgr2=0x7fc0bc198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:11.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.635+0000 7fc0bad9d640 1 --2- 172.21.15.16:0/2762963886 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0bc104b20 0x7fc0bc198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:11.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.635+0000 7fc0bad9d640 1 -- 172.21.15.16:0/2762963886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0bc19f6f0 con 0x7fc0bc107450 2024-03-28T22:10:11.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.635+0000 7fc0bad9d640 1 --2- 172.21.15.16:0/2762963886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0bc107450 0x7fc0bc19f080 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fc0b400eb10 tx=0x7fc0b400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:11.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.636+0000 7fc0b37fe640 1 -- 172.21.15.16:0/2762963886 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b4004b20 con 0x7fc0bc107450 2024-03-28T22:10:11.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.636+0000 7fc0b37fe640 1 -- 172.21.15.16:0/2762963886 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0b4004cc0 con 0x7fc0bc107450 2024-03-28T22:10:11.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.636+0000 7fc0c0e8f640 1 -- 172.21.15.16:0/2762963886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0bc19f9e0 con 0x7fc0bc107450 2024-03-28T22:10:11.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.636+0000 7fc0b37fe640 1 -- 172.21.15.16:0/2762963886 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b4010760 con 0x7fc0bc107450 2024-03-28T22:10:11.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.636+0000 7fc0c0e8f640 1 -- 172.21.15.16:0/2762963886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0bc19feb0 con 0x7fc0bc107450 2024-03-28T22:10:11.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.637+0000 7fc0c0e8f640 1 -- 172.21.15.16:0/2762963886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0bc0fd6d0 con 0x7fc0bc107450 2024-03-28T22:10:11.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.638+0000 7fc0b37fe640 1 -- 172.21.15.16:0/2762963886 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7fc0b4010900 con 0x7fc0bc107450 2024-03-28T22:10:11.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.638+0000 7fc0b37fe640 1 --2- 172.21.15.16:0/2762963886 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc0a80a1d80 0x7fc0a80a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:11.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.639+0000 7fc0ba59c640 1 --2- 172.21.15.16:0/2762963886 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc0a80a1d80 0x7fc0a80a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:11.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.639+0000 7fc0b37fe640 1 -- 172.21.15.16:0/2762963886 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fc0b40ce5f0 con 0x7fc0bc107450 2024-03-28T22:10:11.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.639+0000 7fc0ba59c640 1 --2- 172.21.15.16:0/2762963886 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc0a80a1d80 0x7fc0a80a4240 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac005fd0 tx=0x7fc0ac005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:11.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.642+0000 7fc0b37fe640 1 -- 172.21.15.16:0/2762963886 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc0b4053280 con 0x7fc0bc107450 2024-03-28T22:10:11.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:11 smithi063 ceph-mon[36249]: pgmap v339: 97 pgs: 97 active+clean; 639 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-28T22:10:11.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:11 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/729378226' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:10:11.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.823+0000 7fc0c0e8f640 1 -- 172.21.15.16:0/2762963886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc0bc02d170 con 0x7fc0bc107450 2024-03-28T22:10:11.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.824+0000 7fc0b37fe640 1 -- 172.21.15.16:0/2762963886 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7fc0b409b9f0 con 0x7fc0bc107450 2024-03-28T22:10:11.826 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-28T22:10:11.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.826+0000 7fc0c0e8f640 1 -- 172.21.15.16:0/2762963886 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc0a80a1d80 msgr2=0x7fc0a80a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:11.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.826+0000 7fc0c0e8f640 1 --2- 172.21.15.16:0/2762963886 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc0a80a1d80 0x7fc0a80a4240 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac005fd0 tx=0x7fc0ac005850 comp rx=0 tx=0).stop 2024-03-28T22:10:11.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.826+0000 7fc0c0e8f640 1 -- 172.21.15.16:0/2762963886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0bc107450 msgr2=0x7fc0bc19f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:11.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.826+0000 7fc0c0e8f640 1 --2- 172.21.15.16:0/2762963886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0bc107450 0x7fc0bc19f080 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fc0b400eb10 tx=0x7fc0b400c2d0 comp rx=0 tx=0).stop 2024-03-28T22:10:11.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.826+0000 7fc0c0e8f640 1 -- 172.21.15.16:0/2762963886 shutdown_connections 2024-03-28T22:10:11.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.826+0000 7fc0c0e8f640 1 --2- 172.21.15.16:0/2762963886 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc0a80a1d80 0x7fc0a80a4240 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:11.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.826+0000 7fc0c0e8f640 1 --2- 172.21.15.16:0/2762963886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0bc107450 0x7fc0bc19f080 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:11.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.826+0000 7fc0c0e8f640 1 --2- 172.21.15.16:0/2762963886 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0bc104b20 0x7fc0bc198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:11.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.826+0000 7fc0c0e8f640 1 --2- 172.21.15.16:0/2762963886 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc0bc1021f0 0x7fc0bc197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:11.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.826+0000 7fc0c0e8f640 1 -- 172.21.15.16:0/2762963886 >> 172.21.15.16:0/2762963886 conn(0x7fc0bc0fbbe0 msgr2=0x7fc0bc102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:11.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.826+0000 7fc0c0e8f640 1 -- 172.21.15.16:0/2762963886 shutdown_connections 2024-03-28T22:10:11.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.826+0000 7fc0c0e8f640 1 -- 172.21.15.16:0/2762963886 wait complete. 2024-03-28T22:10:11.843 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T22:10:11.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.947+0000 7f505653e640 1 -- 172.21.15.16:0/772859782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5050101860 msgr2=0x7f5050101ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:11.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.947+0000 7f505653e640 1 --2- 172.21.15.16:0/772859782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5050101860 0x7f5050101ce0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f5044009900 tx=0x7f504402bc30 comp rx=0 tx=0).stop 2024-03-28T22:10:11.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.947+0000 7f505653e640 1 -- 172.21.15.16:0/772859782 shutdown_connections 2024-03-28T22:10:11.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.947+0000 7f505653e640 1 --2- 172.21.15.16:0/772859782 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5050102220 0x7f5050108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:11.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.947+0000 7f505653e640 1 --2- 172.21.15.16:0/772859782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5050101860 0x7f5050101ce0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:11.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.947+0000 7f505653e640 1 --2- 172.21.15.16:0/772859782 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5050100670 0x7f5050100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:11.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.947+0000 7f505653e640 1 -- 172.21.15.16:0/772859782 >> 172.21.15.16:0/772859782 conn(0x7f50500fbbc0 msgr2=0x7f50500fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:11.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.947+0000 7f505653e640 1 -- 172.21.15.16:0/772859782 shutdown_connections 2024-03-28T22:10:11.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.947+0000 7f505653e640 1 -- 172.21.15.16:0/772859782 wait complete. 2024-03-28T22:10:11.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.947+0000 7f505653e640 1 Processor -- start 2024-03-28T22:10:11.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.948+0000 7f505653e640 1 -- start start 2024-03-28T22:10:11.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.948+0000 7f505653e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5050100670 0x7f50501969d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:11.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.948+0000 7f505653e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5050101860 0x7f5050196f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:11.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.948+0000 7f505653e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5050102220 0x7f505019df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:11.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.948+0000 7f505653e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f505010c300 con 0x7f5050102220 2024-03-28T22:10:11.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.948+0000 7f505653e640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f505010c180 con 0x7f5050100670 2024-03-28T22:10:11.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.948+0000 7f505653e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f505010c480 con 0x7f5050101860 2024-03-28T22:10:11.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.948+0000 7f5054adb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5050102220 0x7f505019df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:11.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.948+0000 7f5054adb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5050102220 0x7f505019df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43558/0 (socket says 172.21.15.16:43558) 2024-03-28T22:10:11.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.948+0000 7f5054adb640 1 -- 172.21.15.16:0/645224104 learned_addr learned my addr 172.21.15.16:0/645224104 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:11.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.948+0000 7f504ffff640 1 --2- 172.21.15.16:0/645224104 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5050100670 0x7f50501969d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:11.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.948+0000 7f504f7fe640 1 --2- 172.21.15.16:0/645224104 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5050101860 0x7f5050196f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:11.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.948+0000 7f5054adb640 1 -- 172.21.15.16:0/645224104 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5050101860 msgr2=0x7f5050196f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:11.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.948+0000 7f5054adb640 1 --2- 172.21.15.16:0/645224104 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5050101860 0x7f5050196f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:11.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.948+0000 7f5054adb640 1 -- 172.21.15.16:0/645224104 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5050100670 msgr2=0x7f50501969d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:11.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.948+0000 7f5054adb640 1 --2- 172.21.15.16:0/645224104 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5050100670 0x7f50501969d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:11.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.948+0000 7f5054adb640 1 -- 172.21.15.16:0/645224104 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f505019e5f0 con 0x7f5050102220 2024-03-28T22:10:11.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.948+0000 7f5054adb640 1 --2- 172.21.15.16:0/645224104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5050102220 0x7f505019df80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f504800efd0 tx=0x7f504800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:11.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.949+0000 7f504d7fa640 1 -- 172.21.15.16:0/645224104 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5048003b70 con 0x7f5050102220 2024-03-28T22:10:11.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.949+0000 7f504d7fa640 1 -- 172.21.15.16:0/645224104 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5048003d10 con 0x7f5050102220 2024-03-28T22:10:11.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.949+0000 7f505653e640 1 -- 172.21.15.16:0/645224104 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f505019e8e0 con 0x7f5050102220 2024-03-28T22:10:11.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.949+0000 7f504d7fa640 1 -- 172.21.15.16:0/645224104 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50480109f0 con 0x7f5050102220 2024-03-28T22:10:11.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.949+0000 7f505653e640 1 -- 172.21.15.16:0/645224104 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f505019edb0 con 0x7f5050102220 2024-03-28T22:10:11.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.950+0000 7f505653e640 1 -- 172.21.15.16:0/645224104 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5050107a10 con 0x7f5050102220 2024-03-28T22:10:11.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.951+0000 7f504d7fa640 1 -- 172.21.15.16:0/645224104 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f5048004020 con 0x7f5050102220 2024-03-28T22:10:11.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.951+0000 7f504d7fa640 1 --2- 172.21.15.16:0/645224104 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f50380a1d80 0x7f50380a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:11.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.951+0000 7f504ffff640 1 --2- 172.21.15.16:0/645224104 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f50380a1d80 0x7f50380a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:11.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.951+0000 7f504d7fa640 1 -- 172.21.15.16:0/645224104 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f50480ce1b0 con 0x7f5050102220 2024-03-28T22:10:11.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.952+0000 7f504ffff640 1 --2- 172.21.15.16:0/645224104 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f50380a1d80 0x7f50380a4240 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f50501016c0 tx=0x7f5040005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:11.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:11.956+0000 7f504d7fa640 1 -- 172.21.15.16:0/645224104 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f504809b610 con 0x7f5050102220 2024-03-28T22:10:12.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.135+0000 7f505653e640 1 -- 172.21.15.16:0/645224104 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5050062350 con 0x7f5050102220 2024-03-28T22:10:12.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.136+0000 7f504d7fa640 1 -- 172.21.15.16:0/645224104 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f504809af10 con 0x7f5050102220 2024-03-28T22:10:12.138 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-28T22:10:12.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.138+0000 7f505653e640 1 -- 172.21.15.16:0/645224104 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f50380a1d80 msgr2=0x7f50380a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:12.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.138+0000 7f505653e640 1 --2- 172.21.15.16:0/645224104 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f50380a1d80 0x7f50380a4240 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f50501016c0 tx=0x7f5040005f90 comp rx=0 tx=0).stop 2024-03-28T22:10:12.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.138+0000 7f505653e640 1 -- 172.21.15.16:0/645224104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5050102220 msgr2=0x7f505019df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:12.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.138+0000 7f505653e640 1 --2- 172.21.15.16:0/645224104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5050102220 0x7f505019df80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f504800efd0 tx=0x7f504800c6a0 comp rx=0 tx=0).stop 2024-03-28T22:10:12.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.138+0000 7f505653e640 1 -- 172.21.15.16:0/645224104 shutdown_connections 2024-03-28T22:10:12.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.138+0000 7f505653e640 1 --2- 172.21.15.16:0/645224104 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f50380a1d80 0x7f50380a4240 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:12.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.138+0000 7f505653e640 1 --2- 172.21.15.16:0/645224104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5050102220 0x7f505019df80 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:12.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.138+0000 7f505653e640 1 --2- 172.21.15.16:0/645224104 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5050101860 0x7f5050196f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:12.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.138+0000 7f505653e640 1 --2- 172.21.15.16:0/645224104 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5050100670 0x7f50501969d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:12.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.138+0000 7f505653e640 1 -- 172.21.15.16:0/645224104 >> 172.21.15.16:0/645224104 conn(0x7f50500fbbc0 msgr2=0x7f5050104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:12.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.138+0000 7f505653e640 1 -- 172.21.15.16:0/645224104 shutdown_connections 2024-03-28T22:10:12.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.138+0000 7f505653e640 1 -- 172.21.15.16:0/645224104 wait complete. 2024-03-28T22:10:12.151 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:10:12.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.252+0000 7fdb9628d640 1 -- 172.21.15.16:0/4289726237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb90102990 msgr2=0x7fdb90102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:12.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.252+0000 7fdb9628d640 1 --2- 172.21.15.16:0/4289726237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb90102990 0x7fdb90102e10 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fdb84009900 tx=0x7fdb8402bc30 comp rx=0 tx=0).stop 2024-03-28T22:10:12.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.252+0000 7fdb9628d640 1 -- 172.21.15.16:0/4289726237 shutdown_connections 2024-03-28T22:10:12.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.252+0000 7fdb9628d640 1 --2- 172.21.15.16:0/4289726237 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb90103350 0x7fdb90109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:12.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.252+0000 7fdb9628d640 1 --2- 172.21.15.16:0/4289726237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb90102990 0x7fdb90102e10 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:12.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.252+0000 7fdb9628d640 1 --2- 172.21.15.16:0/4289726237 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb901017a0 0x7fdb90101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:12.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.252+0000 7fdb9628d640 1 -- 172.21.15.16:0/4289726237 >> 172.21.15.16:0/4289726237 conn(0x7fdb900fcd10 msgr2=0x7fdb900ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:12.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.252+0000 7fdb9628d640 1 -- 172.21.15.16:0/4289726237 shutdown_connections 2024-03-28T22:10:12.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.252+0000 7fdb9628d640 1 -- 172.21.15.16:0/4289726237 wait complete. 2024-03-28T22:10:12.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.252+0000 7fdb9628d640 1 Processor -- start 2024-03-28T22:10:12.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.252+0000 7fdb9628d640 1 -- start start 2024-03-28T22:10:12.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.252+0000 7fdb9628d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb901017a0 0x7fdb90197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:12.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.252+0000 7fdb9628d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb90102990 0x7fdb90197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:12.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.252+0000 7fdb9628d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb90103350 0x7fdb9019f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:12.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.252+0000 7fdb9628d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdb9010d420 con 0x7fdb90103350 2024-03-28T22:10:12.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.252+0000 7fdb9628d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fdb9010d2a0 con 0x7fdb90102990 2024-03-28T22:10:12.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.252+0000 7fdb9628d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fdb9010d5a0 con 0x7fdb901017a0 2024-03-28T22:10:12.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.253+0000 7fdb9482a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb90103350 0x7fdb9019f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:12.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.253+0000 7fdb9482a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb90103350 0x7fdb9019f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43566/0 (socket says 172.21.15.16:43566) 2024-03-28T22:10:12.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.253+0000 7fdb9482a640 1 -- 172.21.15.16:0/220745499 learned_addr learned my addr 172.21.15.16:0/220745499 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:12.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.253+0000 7fdb8f7fe640 1 --2- 172.21.15.16:0/220745499 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb90102990 0x7fdb90197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:12.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.253+0000 7fdb8ffff640 1 --2- 172.21.15.16:0/220745499 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb901017a0 0x7fdb90197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:12.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.253+0000 7fdb9482a640 1 -- 172.21.15.16:0/220745499 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb901017a0 msgr2=0x7fdb90197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:12.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.253+0000 7fdb9482a640 1 --2- 172.21.15.16:0/220745499 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb901017a0 0x7fdb90197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:12.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.253+0000 7fdb9482a640 1 -- 172.21.15.16:0/220745499 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb90102990 msgr2=0x7fdb90197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:12.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.253+0000 7fdb9482a640 1 --2- 172.21.15.16:0/220745499 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb90102990 0x7fdb90197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:12.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.253+0000 7fdb9482a640 1 -- 172.21.15.16:0/220745499 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb9019f680 con 0x7fdb90103350 2024-03-28T22:10:12.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.253+0000 7fdb9482a640 1 --2- 172.21.15.16:0/220745499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb90103350 0x7fdb9019f010 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fdb8800eb10 tx=0x7fdb8800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:12.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.253+0000 7fdb8d7fa640 1 -- 172.21.15.16:0/220745499 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb88004b20 con 0x7fdb90103350 2024-03-28T22:10:12.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.253+0000 7fdb8d7fa640 1 -- 172.21.15.16:0/220745499 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb88004cc0 con 0x7fdb90103350 2024-03-28T22:10:12.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.253+0000 7fdb9628d640 1 -- 172.21.15.16:0/220745499 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb9019f970 con 0x7fdb90103350 2024-03-28T22:10:12.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.253+0000 7fdb8d7fa640 1 -- 172.21.15.16:0/220745499 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb88010760 con 0x7fdb90103350 2024-03-28T22:10:12.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.253+0000 7fdb9628d640 1 -- 172.21.15.16:0/220745499 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb9019fee0 con 0x7fdb90103350 2024-03-28T22:10:12.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.254+0000 7fdb9628d640 1 -- 172.21.15.16:0/220745499 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb90108a60 con 0x7fdb90103350 2024-03-28T22:10:12.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.255+0000 7fdb8d7fa640 1 -- 172.21.15.16:0/220745499 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7fdb88005000 con 0x7fdb90103350 2024-03-28T22:10:12.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.256+0000 7fdb8d7fa640 1 --2- 172.21.15.16:0/220745499 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdb780a1e00 0x7fdb780a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:12.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.256+0000 7fdb8ffff640 1 --2- 172.21.15.16:0/220745499 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdb780a1e00 0x7fdb780a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:12.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.256+0000 7fdb8d7fa640 1 -- 172.21.15.16:0/220745499 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fdb880ce780 con 0x7fdb90103350 2024-03-28T22:10:12.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.256+0000 7fdb8ffff640 1 --2- 172.21.15.16:0/220745499 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdb780a1e00 0x7fdb780a42c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fdb901027f0 tx=0x7fdb80005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:12.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.259+0000 7fdb8d7fa640 1 -- 172.21.15.16:0/220745499 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fdb8809bb80 con 0x7fdb90103350 2024-03-28T22:10:12.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:12 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2191897473' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:10:12.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:12 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2762963886' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:10:12.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:12 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/645224104' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:10:12.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.480+0000 7fdb9628d640 1 -- 172.21.15.16:0/220745499 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdb9002d170 con 0x7fdb90103350 2024-03-28T22:10:12.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.481+0000 7fdb8d7fa640 1 -- 172.21.15.16:0/220745499 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11851 (secure 0 0 0) 0x7fdb8809b480 con 0x7fdb90103350 2024-03-28T22:10:12.483 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:10:12.483 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/1644581227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1644581227},{"type":"v1","addr":"172.21.15.16:6839","nonce":1644581227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/2593062669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2593062669},{"type":"v1","addr":"172.21.15.16:6841","nonce":2593062669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:08:29.672976+0000","modified":"2024-03-28T22:08:34.341708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24566,"mds_1":24524,"mds_2":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24466":{"gid":24466,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/247249857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":247249857},{"type":"v1","addr":"172.21.15.63:6835","nonce":247249857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6837/2324231032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":2324231032},{"type":"v1","addr":"172.21.15.63:6837","nonce":2324231032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6839/1177197077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":1177197077},{"type":"v1","addr":"172.21.15.63:6839","nonce":1177197077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"e","rank":2,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.53:6835/350727634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":350727634},{"type":"v1","addr":"172.21.15.53:6835","nonce":350727634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/349877573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":349877573},{"type":"v1","addr":"172.21.15.53:6837","nonce":349877573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6839/3714153956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":3714153956},{"type":"v1","addr":"172.21.15.53:6839","nonce":3714153956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24566,"qdb_cluster":[24500,24566,24524]},"id":1}]} 2024-03-28T22:10:12.483 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 20 2024-03-28T22:10:12.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.483+0000 7fdb9628d640 1 -- 172.21.15.16:0/220745499 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdb780a1e00 msgr2=0x7fdb780a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:12.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.483+0000 7fdb9628d640 1 --2- 172.21.15.16:0/220745499 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdb780a1e00 0x7fdb780a42c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fdb901027f0 tx=0x7fdb80005f90 comp rx=0 tx=0).stop 2024-03-28T22:10:12.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.483+0000 7fdb9628d640 1 -- 172.21.15.16:0/220745499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb90103350 msgr2=0x7fdb9019f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:12.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.483+0000 7fdb9628d640 1 --2- 172.21.15.16:0/220745499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb90103350 0x7fdb9019f010 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fdb8800eb10 tx=0x7fdb8800c2d0 comp rx=0 tx=0).stop 2024-03-28T22:10:12.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.483+0000 7fdb9628d640 1 -- 172.21.15.16:0/220745499 shutdown_connections 2024-03-28T22:10:12.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.483+0000 7fdb9628d640 1 --2- 172.21.15.16:0/220745499 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdb780a1e00 0x7fdb780a42c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:12.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.483+0000 7fdb9628d640 1 --2- 172.21.15.16:0/220745499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb90103350 0x7fdb9019f010 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:12.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.483+0000 7fdb9628d640 1 --2- 172.21.15.16:0/220745499 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb90102990 0x7fdb90197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:12.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.483+0000 7fdb9628d640 1 --2- 172.21.15.16:0/220745499 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb901017a0 0x7fdb90197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:12.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.483+0000 7fdb9628d640 1 -- 172.21.15.16:0/220745499 >> 172.21.15.16:0/220745499 conn(0x7fdb900fcd10 msgr2=0x7fdb90105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:12.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.483+0000 7fdb9628d640 1 -- 172.21.15.16:0/220745499 shutdown_connections 2024-03-28T22:10:12.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.483+0000 7fdb9628d640 1 -- 172.21.15.16:0/220745499 wait complete. 2024-03-28T22:10:12.496 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:10:12.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:12 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2191897473' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:10:12.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:12 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2762963886' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:10:12.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:12 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/645224104' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:10:12.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.596+0000 7f74c8ec0640 1 -- 172.21.15.16:0/722137523 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f74c4101780 msgr2=0x7f74c4101ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:12.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.596+0000 7f74c8ec0640 1 --2- 172.21.15.16:0/722137523 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f74c4101780 0x7f74c4101ba0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f74bc009860 tx=0x7f74bc02bb70 comp rx=0 tx=0).stop 2024-03-28T22:10:12.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.596+0000 7f74c8ec0640 1 -- 172.21.15.16:0/722137523 shutdown_connections 2024-03-28T22:10:12.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.596+0000 7f74c8ec0640 1 --2- 172.21.15.16:0/722137523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74c4103330 0x7f74c4109bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:12.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.596+0000 7f74c8ec0640 1 --2- 172.21.15.16:0/722137523 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74c4102970 0x7f74c4102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:12.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.596+0000 7f74c8ec0640 1 --2- 172.21.15.16:0/722137523 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f74c4101780 0x7f74c4101ba0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:12.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.596+0000 7f74c8ec0640 1 -- 172.21.15.16:0/722137523 >> 172.21.15.16:0/722137523 conn(0x7f74c40fcd30 msgr2=0x7f74c40ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:12.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.596+0000 7f74c8ec0640 1 -- 172.21.15.16:0/722137523 shutdown_connections 2024-03-28T22:10:12.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.597+0000 7f74c8ec0640 1 -- 172.21.15.16:0/722137523 wait complete. 2024-03-28T22:10:12.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.597+0000 7f74c8ec0640 1 Processor -- start 2024-03-28T22:10:12.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.597+0000 7f74c8ec0640 1 -- start start 2024-03-28T22:10:12.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.597+0000 7f74c8ec0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74c4101780 0x7f74c4197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:12.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.597+0000 7f74c8ec0640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f74c4102970 0x7f74c4198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:12.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.597+0000 7f74c8ec0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74c4103330 0x7f74c419f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:12.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.597+0000 7f74c8ec0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f74c410d3f0 con 0x7f74c4103330 2024-03-28T22:10:12.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.597+0000 7f74c8ec0640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f74c410d270 con 0x7f74c4102970 2024-03-28T22:10:12.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.597+0000 7f74c8ec0640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f74c410d570 con 0x7f74c4101780 2024-03-28T22:10:12.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.597+0000 7f74c2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74c4103330 0x7f74c419f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:12.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.597+0000 7f74c2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74c4103330 0x7f74c419f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43584/0 (socket says 172.21.15.16:43584) 2024-03-28T22:10:12.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.597+0000 7f74c2d9d640 1 -- 172.21.15.16:0/2484775395 learned_addr learned my addr 172.21.15.16:0/2484775395 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:12.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.598+0000 7f74c259c640 1 --2- 172.21.15.16:0/2484775395 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74c4101780 0x7f74c4197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:12.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.598+0000 7f74c1d9b640 1 --2- 172.21.15.16:0/2484775395 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f74c4102970 0x7f74c4198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:12.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.598+0000 7f74c2d9d640 1 -- 172.21.15.16:0/2484775395 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74c4101780 msgr2=0x7f74c4197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:12.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.598+0000 7f74c2d9d640 1 --2- 172.21.15.16:0/2484775395 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74c4101780 0x7f74c4197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:12.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.598+0000 7f74c2d9d640 1 -- 172.21.15.16:0/2484775395 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f74c4102970 msgr2=0x7f74c4198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:12.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.598+0000 7f74c2d9d640 1 --2- 172.21.15.16:0/2484775395 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f74c4102970 0x7f74c4198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:12.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.598+0000 7f74c2d9d640 1 -- 172.21.15.16:0/2484775395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74c419f6e0 con 0x7f74c4103330 2024-03-28T22:10:12.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.598+0000 7f74c2d9d640 1 --2- 172.21.15.16:0/2484775395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74c4103330 0x7f74c419f070 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f74b4009fd0 tx=0x7f74b400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:12.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.598+0000 7f74bb7fe640 1 -- 172.21.15.16:0/2484775395 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b4004b20 con 0x7f74c4103330 2024-03-28T22:10:12.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.598+0000 7f74c8ec0640 1 -- 172.21.15.16:0/2484775395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74c419f9d0 con 0x7f74c4103330 2024-03-28T22:10:12.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.598+0000 7f74bb7fe640 1 -- 172.21.15.16:0/2484775395 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74b4003e50 con 0x7f74c4103330 2024-03-28T22:10:12.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.598+0000 7f74bb7fe640 1 -- 172.21.15.16:0/2484775395 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b400b630 con 0x7f74c4103330 2024-03-28T22:10:12.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.598+0000 7f74c8ec0640 1 -- 172.21.15.16:0/2484775395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74c419fed0 con 0x7f74c4103330 2024-03-28T22:10:12.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.599+0000 7f74c8ec0640 1 -- 172.21.15.16:0/2484775395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74c4108b20 con 0x7f74c4103330 2024-03-28T22:10:12.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.600+0000 7f74bb7fe640 1 -- 172.21.15.16:0/2484775395 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f74b401d020 con 0x7f74c4103330 2024-03-28T22:10:12.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.601+0000 7f74bb7fe640 1 --2- 172.21.15.16:0/2484775395 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f74b00a1d80 0x7f74b00a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:12.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.601+0000 7f74c259c640 1 --2- 172.21.15.16:0/2484775395 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f74b00a1d80 0x7f74b00a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:12.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.601+0000 7f74bb7fe640 1 -- 172.21.15.16:0/2484775395 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f74b40ce6d0 con 0x7f74c4103330 2024-03-28T22:10:12.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.601+0000 7f74c259c640 1 --2- 172.21.15.16:0/2484775395 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f74b00a1d80 0x7f74b00a4240 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f74bc009830 tx=0x7f74bc034ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:12.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.604+0000 7f74bb7fe640 1 -- 172.21.15.16:0/2484775395 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f74b4024030 con 0x7f74c4103330 2024-03-28T22:10:12.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:12 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2191897473' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:10:12.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:12 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2762963886' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:10:12.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:12 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/645224104' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:10:12.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.789+0000 7f74c8ec0640 1 -- 172.21.15.16:0/2484775395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f74c4062350 con 0x7f74c4103330 2024-03-28T22:10:12.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.790+0000 7f74bb7fe640 1 -- 172.21.15.16:0/2484775395 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22441 (secure 0 0 0) 0x7f74b409bb30 con 0x7f74c4103330 2024-03-28T22:10:12.792 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:10:12.792 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":88,"fsid":"798d9884-ed4e-11ee-95d0-87774f69a715","created":"2024-03-28T22:00:27.216099+0000","modified":"2024-03-28T22:09:30.047847+0000","last_up_change":"2024-03-28T22:07:40.251141+0000","last_in_change":"2024-03-28T22:07:19.456451+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:03:59.327832+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:08:28.123529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:08:28.919037+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"46e7c8f8-ecc8-43d6-ae53-0655728f3240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6803","nonce":1589692483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6805","nonce":1589692483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6809","nonce":1589692483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6807","nonce":1589692483}]},"public_addr":"172.21.15.16:6803/1589692483","cluster_addr":"172.21.15.16:6805/1589692483","heartbeat_back_addr":"172.21.15.16:6809/1589692483","heartbeat_front_addr":"172.21.15.16:6807/1589692483","state":["exists","up"]},{"osd":1,"uuid":"af6d9c22-2830-4ce2-a547-011df0c6cbad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6801","nonce":795531234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6803","nonce":795531234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6807","nonce":795531234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6805","nonce":795531234}]},"public_addr":"172.21.15.53:6801/795531234","cluster_addr":"172.21.15.53:6803/795531234","heartbeat_back_addr":"172.21.15.53:6807/795531234","heartbeat_front_addr":"172.21.15.53:6805/795531234","state":["exists","up"]},{"osd":2,"uuid":"b10adca9-110c-4f87-85e0-b7542709d31a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6801","nonce":2214131086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6803","nonce":2214131086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6807","nonce":2214131086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6805","nonce":2214131086}]},"public_addr":"172.21.15.63:6801/2214131086","cluster_addr":"172.21.15.63:6803/2214131086","heartbeat_back_addr":"172.21.15.63:6807/2214131086","heartbeat_front_addr":"172.21.15.63:6805/2214131086","state":["exists","up"]},{"osd":3,"uuid":"c83cf4bb-395a-4786-abf7-74ca93a33eb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6811","nonce":13176789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6813","nonce":13176789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6817","nonce":13176789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6815","nonce":13176789}]},"public_addr":"172.21.15.16:6811/13176789","cluster_addr":"172.21.15.16:6813/13176789","heartbeat_back_addr":"172.21.15.16:6817/13176789","heartbeat_front_addr":"172.21.15.16:6815/13176789","state":["exists","up"]},{"osd":4,"uuid":"941b3e54-47d6-4e85-8069-e730c3fa8eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6809","nonce":772161849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6811","nonce":772161849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6815","nonce":772161849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6813","nonce":772161849}]},"public_addr":"172.21.15.53:6809/772161849","cluster_addr":"172.21.15.53:6811/772161849","heartbeat_back_addr":"172.21.15.53:6815/772161849","heartbeat_front_addr":"172.21.15.53:6813/772161849","state":["exists","up"]},{"osd":5,"uuid":"51d89826-1d60-40fc-8fd8-09bfe1ab4125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6809","nonce":1602554852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6811","nonce":1602554852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6815","nonce":1602554852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6813","nonce":1602554852}]},"public_addr":"172.21.15.63:6809/1602554852","cluster_addr":"172.21.15.63:6811/1602554852","heartbeat_back_addr":"172.21.15.63:6815/1602554852","heartbeat_front_addr":"172.21.15.63:6813/1602554852","state":["exists","up"]},{"osd":6,"uuid":"978efe5d-2c55-4ff8-8a1e-9f891388945e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6819","nonce":4290459224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6821","nonce":4290459224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6825","nonce":4290459224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6823","nonce":4290459224}]},"public_addr":"172.21.15.16:6819/4290459224","cluster_addr":"172.21.15.16:6821/4290459224","heartbeat_back_addr":"172.21.15.16:6825/4290459224","heartbeat_front_addr":"172.21.15.16:6823/4290459224","state":["exists","up"]},{"osd":7,"uuid":"50d489cb-dbb8-48df-b868-b29edf12e04d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6817","nonce":1056467525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6819","nonce":1056467525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6823","nonce":1056467525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6821","nonce":1056467525}]},"public_addr":"172.21.15.53:6817/1056467525","cluster_addr":"172.21.15.53:6819/1056467525","heartbeat_back_addr":"172.21.15.53:6823/1056467525","heartbeat_front_addr":"172.21.15.53:6821/1056467525","state":["exists","up"]},{"osd":8,"uuid":"49b39414-cc9c-4057-ae3f-cc109cd979da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6817","nonce":2918516794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6819","nonce":2918516794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6823","nonce":2918516794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6821","nonce":2918516794}]},"public_addr":"172.21.15.63:6817/2918516794","cluster_addr":"172.21.15.63:6819/2918516794","heartbeat_back_addr":"172.21.15.63:6823/2918516794","heartbeat_front_addr":"172.21.15.63:6821/2918516794","state":["exists","up"]},{"osd":9,"uuid":"4c873e51-b783-4eb6-83a3-fee31722d60b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6827","nonce":1585624367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6829","nonce":1585624367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6833","nonce":1585624367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6831","nonce":1585624367}]},"public_addr":"172.21.15.16:6827/1585624367","cluster_addr":"172.21.15.16:6829/1585624367","heartbeat_back_addr":"172.21.15.16:6833/1585624367","heartbeat_front_addr":"172.21.15.16:6831/1585624367","state":["exists","up"]},{"osd":10,"uuid":"085a78bf-e2ad-481a-85a3-71d1fc549a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6825","nonce":3357446484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6827","nonce":3357446484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6831","nonce":3357446484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6829","nonce":3357446484}]},"public_addr":"172.21.15.53:6825/3357446484","cluster_addr":"172.21.15.53:6827/3357446484","heartbeat_back_addr":"172.21.15.53:6831/3357446484","heartbeat_front_addr":"172.21.15.53:6829/3357446484","state":["exists","up"]},{"osd":11,"uuid":"13302aa3-08b9-4639-9042-27d850b4d5f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6825","nonce":2167866570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6827","nonce":2167866570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6831","nonce":2167866570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6829","nonce":2167866570}]},"public_addr":"172.21.15.63:6825/2167866570","cluster_addr":"172.21.15.63:6827/2167866570","heartbeat_back_addr":"172.21.15.63:6831/2167866570","heartbeat_front_addr":"172.21.15.63:6829/2167866570","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:07.450323+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:31.363172+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:55.295021+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:19.794760+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:44.063492+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:08.195064+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:33.655328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:57.639414+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:22.402394+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:47.462050+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:12.226307+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:37.495625+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3821408150":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6801/1832422779":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/996598130":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/1062264551":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/317557779":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6801/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/3631746518":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6800/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/3116652786":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6800/1832422779":"2024-03-29T22:01:28.993434+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:10:12.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.792+0000 7f74c8ec0640 1 -- 172.21.15.16:0/2484775395 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f74b00a1d80 msgr2=0x7f74b00a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:12.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.792+0000 7f74c8ec0640 1 --2- 172.21.15.16:0/2484775395 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f74b00a1d80 0x7f74b00a4240 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f74bc009830 tx=0x7f74bc034ad0 comp rx=0 tx=0).stop 2024-03-28T22:10:12.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.792+0000 7f74c8ec0640 1 -- 172.21.15.16:0/2484775395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74c4103330 msgr2=0x7f74c419f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:12.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.792+0000 7f74c8ec0640 1 --2- 172.21.15.16:0/2484775395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74c4103330 0x7f74c419f070 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f74b4009fd0 tx=0x7f74b400c800 comp rx=0 tx=0).stop 2024-03-28T22:10:12.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.792+0000 7f74c8ec0640 1 -- 172.21.15.16:0/2484775395 shutdown_connections 2024-03-28T22:10:12.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.792+0000 7f74c8ec0640 1 --2- 172.21.15.16:0/2484775395 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f74b00a1d80 0x7f74b00a4240 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:12.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.792+0000 7f74c8ec0640 1 --2- 172.21.15.16:0/2484775395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74c4103330 0x7f74c419f070 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:12.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.792+0000 7f74c8ec0640 1 --2- 172.21.15.16:0/2484775395 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f74c4102970 0x7f74c4198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:12.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.792+0000 7f74c8ec0640 1 --2- 172.21.15.16:0/2484775395 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74c4101780 0x7f74c4197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:12.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.792+0000 7f74c8ec0640 1 -- 172.21.15.16:0/2484775395 >> 172.21.15.16:0/2484775395 conn(0x7f74c40fcd30 msgr2=0x7f74c4105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:12.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.792+0000 7f74c8ec0640 1 -- 172.21.15.16:0/2484775395 shutdown_connections 2024-03-28T22:10:12.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.792+0000 7f74c8ec0640 1 -- 172.21.15.16:0/2484775395 wait complete. 2024-03-28T22:10:12.805 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T22:03:59.327832+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T22:08:28.123529+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T22:08:28.919037+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T22:10:12.806 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T22:10:12.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.911+0000 7f01beb3e640 1 -- 172.21.15.16:0/1022223799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01b8102220 msgr2=0x7f01b8108ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:12.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.911+0000 7f01beb3e640 1 --2- 172.21.15.16:0/1022223799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01b8102220 0x7f01b8108ab0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f01b0009ca0 tx=0x7f01b002c0f0 comp rx=0 tx=0).stop 2024-03-28T22:10:12.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.911+0000 7f01beb3e640 1 -- 172.21.15.16:0/1022223799 shutdown_connections 2024-03-28T22:10:12.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.911+0000 7f01beb3e640 1 --2- 172.21.15.16:0/1022223799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01b8102220 0x7f01b8108ab0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:12.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.911+0000 7f01beb3e640 1 --2- 172.21.15.16:0/1022223799 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01b8101860 0x7f01b8101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:12.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.911+0000 7f01beb3e640 1 --2- 172.21.15.16:0/1022223799 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f01b8100670 0x7f01b8100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:12.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.911+0000 7f01beb3e640 1 -- 172.21.15.16:0/1022223799 >> 172.21.15.16:0/1022223799 conn(0x7f01b80fbbc0 msgr2=0x7f01b80fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:12.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.911+0000 7f01beb3e640 1 -- 172.21.15.16:0/1022223799 shutdown_connections 2024-03-28T22:10:12.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.911+0000 7f01beb3e640 1 -- 172.21.15.16:0/1022223799 wait complete. 2024-03-28T22:10:12.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.911+0000 7f01beb3e640 1 Processor -- start 2024-03-28T22:10:12.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.911+0000 7f01beb3e640 1 -- start start 2024-03-28T22:10:12.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.912+0000 7f01beb3e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f01b8100670 0x7f01b81979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:12.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.912+0000 7f01beb3e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01b8101860 0x7f01b8197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:12.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.912+0000 7f01beb3e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01b8102220 0x7f01b819ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:12.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.912+0000 7f01beb3e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f01b810c2d0 con 0x7f01b8102220 2024-03-28T22:10:12.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.912+0000 7f01beb3e640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f01b810c150 con 0x7f01b8101860 2024-03-28T22:10:12.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.912+0000 7f01beb3e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f01b810c450 con 0x7f01b8100670 2024-03-28T22:10:12.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.912+0000 7f01bd0db640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01b8102220 0x7f01b819ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:12.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.912+0000 7f01bd0db640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01b8102220 0x7f01b819ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43600/0 (socket says 172.21.15.16:43600) 2024-03-28T22:10:12.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.912+0000 7f01bd0db640 1 -- 172.21.15.16:0/3750293217 learned_addr learned my addr 172.21.15.16:0/3750293217 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:12.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.912+0000 7f01affff640 1 --2- 172.21.15.16:0/3750293217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01b8101860 0x7f01b8197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:12.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.912+0000 7f01bc8da640 1 --2- 172.21.15.16:0/3750293217 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f01b8100670 0x7f01b81979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:12.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.912+0000 7f01bd0db640 1 -- 172.21.15.16:0/3750293217 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f01b8100670 msgr2=0x7f01b81979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:12.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.912+0000 7f01bd0db640 1 --2- 172.21.15.16:0/3750293217 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f01b8100670 0x7f01b81979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:12.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.912+0000 7f01bd0db640 1 -- 172.21.15.16:0/3750293217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01b8101860 msgr2=0x7f01b8197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:12.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.912+0000 7f01bd0db640 1 --2- 172.21.15.16:0/3750293217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01b8101860 0x7f01b8197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:12.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.912+0000 7f01bd0db640 1 -- 172.21.15.16:0/3750293217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01b819f5e0 con 0x7f01b8102220 2024-03-28T22:10:12.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.912+0000 7f01bd0db640 1 --2- 172.21.15.16:0/3750293217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01b8102220 0x7f01b819ef70 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f01b0037040 tx=0x7f01b0032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:12.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.912+0000 7f01adffb640 1 -- 172.21.15.16:0/3750293217 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01b0041020 con 0x7f01b8102220 2024-03-28T22:10:12.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.912+0000 7f01adffb640 1 -- 172.21.15.16:0/3750293217 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01b0034d90 con 0x7f01b8102220 2024-03-28T22:10:12.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.912+0000 7f01beb3e640 1 -- 172.21.15.16:0/3750293217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01b819f870 con 0x7f01b8102220 2024-03-28T22:10:12.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.912+0000 7f01adffb640 1 -- 172.21.15.16:0/3750293217 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01b0039040 con 0x7f01b8102220 2024-03-28T22:10:12.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.913+0000 7f01beb3e640 1 -- 172.21.15.16:0/3750293217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01b819fd60 con 0x7f01b8102220 2024-03-28T22:10:12.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.913+0000 7f01beb3e640 1 -- 172.21.15.16:0/3750293217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01b819ffc0 con 0x7f01b8102220 2024-03-28T22:10:12.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.915+0000 7f01adffb640 1 -- 172.21.15.16:0/3750293217 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f01b0044020 con 0x7f01b8102220 2024-03-28T22:10:12.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.915+0000 7f01adffb640 1 --2- 172.21.15.16:0/3750293217 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f01a00a1d80 0x7f01a00a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:12.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.915+0000 7f01adffb640 1 -- 172.21.15.16:0/3750293217 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f01b00efd10 con 0x7f01b8102220 2024-03-28T22:10:12.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.915+0000 7f01bc8da640 1 --2- 172.21.15.16:0/3750293217 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f01a00a1d80 0x7f01a00a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:12.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.916+0000 7f01bc8da640 1 --2- 172.21.15.16:0/3750293217 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f01a00a1d80 0x7f01a00a4240 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f01b4005fd0 tx=0x7f01b4005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:12.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:12.920+0000 7f01adffb640 1 -- 172.21.15.16:0/3750293217 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f01b004b020 con 0x7f01b8102220 2024-03-28T22:10:13.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.101+0000 7f01beb3e640 1 -- 172.21.15.16:0/3750293217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f01b81a0350 con 0x7f01b8102220 2024-03-28T22:10:13.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.102+0000 7f01adffb640 1 -- 172.21.15.16:0/3750293217 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f01b00bd170 con 0x7f01b8102220 2024-03-28T22:10:13.104 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-28T22:10:13.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.104+0000 7f01beb3e640 1 -- 172.21.15.16:0/3750293217 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f01a00a1d80 msgr2=0x7f01a00a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:13.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.104+0000 7f01beb3e640 1 --2- 172.21.15.16:0/3750293217 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f01a00a1d80 0x7f01a00a4240 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f01b4005fd0 tx=0x7f01b4005850 comp rx=0 tx=0).stop 2024-03-28T22:10:13.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.104+0000 7f01beb3e640 1 -- 172.21.15.16:0/3750293217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01b8102220 msgr2=0x7f01b819ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:13.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.104+0000 7f01beb3e640 1 --2- 172.21.15.16:0/3750293217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01b8102220 0x7f01b819ef70 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f01b0037040 tx=0x7f01b0032f20 comp rx=0 tx=0).stop 2024-03-28T22:10:13.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.104+0000 7f01beb3e640 1 -- 172.21.15.16:0/3750293217 shutdown_connections 2024-03-28T22:10:13.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.104+0000 7f01beb3e640 1 --2- 172.21.15.16:0/3750293217 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f01a00a1d80 0x7f01a00a4240 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:13.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.104+0000 7f01beb3e640 1 --2- 172.21.15.16:0/3750293217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01b8102220 0x7f01b819ef70 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:13.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.104+0000 7f01beb3e640 1 --2- 172.21.15.16:0/3750293217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01b8101860 0x7f01b8197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:13.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.104+0000 7f01beb3e640 1 --2- 172.21.15.16:0/3750293217 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f01b8100670 0x7f01b81979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:13.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.104+0000 7f01beb3e640 1 -- 172.21.15.16:0/3750293217 >> 172.21.15.16:0/3750293217 conn(0x7f01b80fbbc0 msgr2=0x7f01b8069e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:13.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.104+0000 7f01beb3e640 1 -- 172.21.15.16:0/3750293217 shutdown_connections 2024-03-28T22:10:13.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.104+0000 7f01beb3e640 1 -- 172.21.15.16:0/3750293217 wait complete. 2024-03-28T22:10:13.117 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T22:10:13.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.220+0000 7f18c223b640 1 -- 172.21.15.16:0/3935024761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18bc101860 msgr2=0x7f18bc101ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:13.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.220+0000 7f18c223b640 1 --2- 172.21.15.16:0/3935024761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18bc101860 0x7f18bc101ce0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f18b0009900 tx=0x7f18b002bc30 comp rx=0 tx=0).stop 2024-03-28T22:10:13.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.220+0000 7f18c223b640 1 -- 172.21.15.16:0/3935024761 shutdown_connections 2024-03-28T22:10:13.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.220+0000 7f18c223b640 1 --2- 172.21.15.16:0/3935024761 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18bc102220 0x7f18bc108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:13.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.220+0000 7f18c223b640 1 --2- 172.21.15.16:0/3935024761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18bc101860 0x7f18bc101ce0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:13.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.220+0000 7f18c223b640 1 --2- 172.21.15.16:0/3935024761 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18bc100670 0x7f18bc100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:13.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.220+0000 7f18c223b640 1 -- 172.21.15.16:0/3935024761 >> 172.21.15.16:0/3935024761 conn(0x7f18bc0fbbe0 msgr2=0x7f18bc0fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:13.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.220+0000 7f18c223b640 1 -- 172.21.15.16:0/3935024761 shutdown_connections 2024-03-28T22:10:13.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.220+0000 7f18c223b640 1 -- 172.21.15.16:0/3935024761 wait complete. 2024-03-28T22:10:13.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.221+0000 7f18c223b640 1 Processor -- start 2024-03-28T22:10:13.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.221+0000 7f18c223b640 1 -- start start 2024-03-28T22:10:13.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.221+0000 7f18c223b640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18bc100670 0x7f18bc197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:13.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.221+0000 7f18c223b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18bc101860 0x7f18bc197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:13.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.221+0000 7f18c223b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18bc102220 0x7f18bc19f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:13.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.221+0000 7f18c223b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f18bc10d410 con 0x7f18bc101860 2024-03-28T22:10:13.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.221+0000 7f18c223b640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f18bc10d290 con 0x7f18bc100670 2024-03-28T22:10:13.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.221+0000 7f18c223b640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f18bc10d590 con 0x7f18bc102220 2024-03-28T22:10:13.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.221+0000 7f18baffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18bc101860 0x7f18bc197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:13.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.221+0000 7f18baffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18bc101860 0x7f18bc197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43612/0 (socket says 172.21.15.16:43612) 2024-03-28T22:10:13.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.221+0000 7f18baffd640 1 -- 172.21.15.16:0/196712438 learned_addr learned my addr 172.21.15.16:0/196712438 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:13.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.221+0000 7f18bb7fe640 1 --2- 172.21.15.16:0/196712438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18bc100670 0x7f18bc197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:13.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.222+0000 7f18bbfff640 1 --2- 172.21.15.16:0/196712438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18bc102220 0x7f18bc19f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:13.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.222+0000 7f18baffd640 1 -- 172.21.15.16:0/196712438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18bc102220 msgr2=0x7f18bc19f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:13.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.222+0000 7f18baffd640 1 --2- 172.21.15.16:0/196712438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18bc102220 0x7f18bc19f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:13.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.222+0000 7f18baffd640 1 -- 172.21.15.16:0/196712438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18bc100670 msgr2=0x7f18bc197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:13.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.222+0000 7f18baffd640 1 --2- 172.21.15.16:0/196712438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18bc100670 0x7f18bc197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:13.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.222+0000 7f18baffd640 1 -- 172.21.15.16:0/196712438 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18bc19f690 con 0x7f18bc101860 2024-03-28T22:10:13.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.222+0000 7f18bbfff640 1 --2- 172.21.15.16:0/196712438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18bc102220 0x7f18bc19f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:10:13.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.222+0000 7f18baffd640 1 --2- 172.21.15.16:0/196712438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18bc101860 0x7f18bc197fb0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f18b0009230 tx=0x7f18b0005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:13.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.222+0000 7f18b8ff9640 1 -- 172.21.15.16:0/196712438 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b0041070 con 0x7f18bc101860 2024-03-28T22:10:13.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.222+0000 7f18b8ff9640 1 -- 172.21.15.16:0/196712438 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18b00044d0 con 0x7f18bc101860 2024-03-28T22:10:13.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.222+0000 7f18c223b640 1 -- 172.21.15.16:0/196712438 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18bc19f920 con 0x7f18bc101860 2024-03-28T22:10:13.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.222+0000 7f18b8ff9640 1 -- 172.21.15.16:0/196712438 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b0034440 con 0x7f18bc101860 2024-03-28T22:10:13.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.222+0000 7f18c223b640 1 -- 172.21.15.16:0/196712438 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18bc19fe10 con 0x7f18bc101860 2024-03-28T22:10:13.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.223+0000 7f18c223b640 1 -- 172.21.15.16:0/196712438 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18bc107990 con 0x7f18bc101860 2024-03-28T22:10:13.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.224+0000 7f18b8ff9640 1 -- 172.21.15.16:0/196712438 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f18b0044070 con 0x7f18bc101860 2024-03-28T22:10:13.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.225+0000 7f18b8ff9640 1 --2- 172.21.15.16:0/196712438 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f18a40a1d80 0x7f18a40a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:13.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.225+0000 7f18bb7fe640 1 --2- 172.21.15.16:0/196712438 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f18a40a1d80 0x7f18a40a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:13.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.225+0000 7f18b8ff9640 1 -- 172.21.15.16:0/196712438 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f18b00efce0 con 0x7f18bc101860 2024-03-28T22:10:13.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.225+0000 7f18bb7fe640 1 --2- 172.21.15.16:0/196712438 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f18a40a1d80 0x7f18a40a4240 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f18bc1016c0 tx=0x7f18ac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:13.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.228+0000 7f18b8ff9640 1 -- 172.21.15.16:0/196712438 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f18b00bd140 con 0x7f18bc101860 2024-03-28T22:10:13.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.412+0000 7f18c223b640 1 -- 172.21.15.16:0/196712438 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f18bc02d3c0 con 0x7f18bc101860 2024-03-28T22:10:13.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.413+0000 7f18b8ff9640 1 -- 172.21.15.16:0/196712438 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f18b00dc070 con 0x7f18bc101860 2024-03-28T22:10:13.414 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-28T22:10:13.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.415+0000 7f18c223b640 1 -- 172.21.15.16:0/196712438 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f18a40a1d80 msgr2=0x7f18a40a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:13.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.415+0000 7f18c223b640 1 --2- 172.21.15.16:0/196712438 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f18a40a1d80 0x7f18a40a4240 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f18bc1016c0 tx=0x7f18ac005f90 comp rx=0 tx=0).stop 2024-03-28T22:10:13.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.415+0000 7f18c223b640 1 -- 172.21.15.16:0/196712438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18bc101860 msgr2=0x7f18bc197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:13.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.415+0000 7f18c223b640 1 --2- 172.21.15.16:0/196712438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18bc101860 0x7f18bc197fb0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f18b0009230 tx=0x7f18b0005d50 comp rx=0 tx=0).stop 2024-03-28T22:10:13.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.415+0000 7f18c223b640 1 -- 172.21.15.16:0/196712438 shutdown_connections 2024-03-28T22:10:13.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.415+0000 7f18c223b640 1 --2- 172.21.15.16:0/196712438 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f18a40a1d80 0x7f18a40a4240 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:13.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.415+0000 7f18c223b640 1 --2- 172.21.15.16:0/196712438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18bc102220 0x7f18bc19f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:13.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.415+0000 7f18c223b640 1 --2- 172.21.15.16:0/196712438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18bc101860 0x7f18bc197fb0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:13.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.415+0000 7f18c223b640 1 --2- 172.21.15.16:0/196712438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18bc100670 0x7f18bc197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:13.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.415+0000 7f18c223b640 1 -- 172.21.15.16:0/196712438 >> 172.21.15.16:0/196712438 conn(0x7f18bc0fbbe0 msgr2=0x7f18bc104b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:13.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.415+0000 7f18c223b640 1 -- 172.21.15.16:0/196712438 shutdown_connections 2024-03-28T22:10:13.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.415+0000 7f18c223b640 1 -- 172.21.15.16:0/196712438 wait complete. 2024-03-28T22:10:13.426 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:13 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/220745499' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:10:13.427 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:13 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2484775395' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:10:13.427 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:13 smithi016 ceph-mon[32374]: pgmap v340: 97 pgs: 97 active+clean; 639 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-28T22:10:13.427 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:13 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3750293217' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:10:13.428 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T22:10:13.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.531+0000 7f24cfcfb640 1 -- 172.21.15.16:0/3682111600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24c80ff800 msgr2=0x7f24c80ffc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:13.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.531+0000 7f24cfcfb640 1 --2- 172.21.15.16:0/3682111600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24c80ff800 0x7f24c80ffc80 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f24b8009900 tx=0x7f24b802bc30 comp rx=0 tx=0).stop 2024-03-28T22:10:13.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.531+0000 7f24cfcfb640 1 -- 172.21.15.16:0/3682111600 shutdown_connections 2024-03-28T22:10:13.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.531+0000 7f24cfcfb640 1 --2- 172.21.15.16:0/3682111600 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f24c81001c0 0x7f24c8109d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:13.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.531+0000 7f24cfcfb640 1 --2- 172.21.15.16:0/3682111600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24c80ff800 0x7f24c80ffc80 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:13.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.531+0000 7f24cfcfb640 1 --2- 172.21.15.16:0/3682111600 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24c8102d90 0x7f24c81031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:13.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.531+0000 7f24cfcfb640 1 -- 172.21.15.16:0/3682111600 >> 172.21.15.16:0/3682111600 conn(0x7f24c80fb380 msgr2=0x7f24c80fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:13.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.531+0000 7f24cfcfb640 1 -- 172.21.15.16:0/3682111600 shutdown_connections 2024-03-28T22:10:13.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.531+0000 7f24cfcfb640 1 -- 172.21.15.16:0/3682111600 wait complete. 2024-03-28T22:10:13.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.532+0000 7f24cfcfb640 1 Processor -- start 2024-03-28T22:10:13.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.532+0000 7f24cfcfb640 1 -- start start 2024-03-28T22:10:13.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.532+0000 7f24cfcfb640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f24c80ff800 0x7f24c8197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:13.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.532+0000 7f24cfcfb640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24c81001c0 0x7f24c8197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:13.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.532+0000 7f24cfcfb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24c8102d90 0x7f24c819f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:13.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.532+0000 7f24cfcfb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f24c810d3f0 con 0x7f24c8102d90 2024-03-28T22:10:13.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.532+0000 7f24cfcfb640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f24c810d270 con 0x7f24c81001c0 2024-03-28T22:10:13.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.532+0000 7f24cfcfb640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f24c810d570 con 0x7f24c80ff800 2024-03-28T22:10:13.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.532+0000 7f24ce298640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24c8102d90 0x7f24c819f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:13.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.532+0000 7f24ce298640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24c8102d90 0x7f24c819f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43630/0 (socket says 172.21.15.16:43630) 2024-03-28T22:10:13.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.532+0000 7f24ce298640 1 -- 172.21.15.16:0/783528021 learned_addr learned my addr 172.21.15.16:0/783528021 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:13.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.532+0000 7f24cd296640 1 --2- 172.21.15.16:0/783528021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24c81001c0 0x7f24c8197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:13.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.532+0000 7f24cda97640 1 --2- 172.21.15.16:0/783528021 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f24c80ff800 0x7f24c8197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:13.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.532+0000 7f24ce298640 1 -- 172.21.15.16:0/783528021 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f24c80ff800 msgr2=0x7f24c8197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:13.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.532+0000 7f24ce298640 1 --2- 172.21.15.16:0/783528021 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f24c80ff800 0x7f24c8197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:13.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.532+0000 7f24ce298640 1 -- 172.21.15.16:0/783528021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24c81001c0 msgr2=0x7f24c8197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:13.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.532+0000 7f24ce298640 1 --2- 172.21.15.16:0/783528021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24c81001c0 0x7f24c8197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:13.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.532+0000 7f24ce298640 1 -- 172.21.15.16:0/783528021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24c819f680 con 0x7f24c8102d90 2024-03-28T22:10:13.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.533+0000 7f24cd296640 1 --2- 172.21.15.16:0/783528021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24c81001c0 0x7f24c8197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:10:13.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.533+0000 7f24ce298640 1 --2- 172.21.15.16:0/783528021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24c8102d90 0x7f24c819f010 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f24c000eaa0 tx=0x7f24c000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:13.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.533+0000 7f24beffd640 1 -- 172.21.15.16:0/783528021 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c000cdb0 con 0x7f24c8102d90 2024-03-28T22:10:13.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.533+0000 7f24beffd640 1 -- 172.21.15.16:0/783528021 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24c0005020 con 0x7f24c8102d90 2024-03-28T22:10:13.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.533+0000 7f24cfcfb640 1 -- 172.21.15.16:0/783528021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24c819f970 con 0x7f24c8102d90 2024-03-28T22:10:13.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.533+0000 7f24beffd640 1 -- 172.21.15.16:0/783528021 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c00116b0 con 0x7f24c8102d90 2024-03-28T22:10:13.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.533+0000 7f24cfcfb640 1 -- 172.21.15.16:0/783528021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24c819fe40 con 0x7f24c8102d90 2024-03-28T22:10:13.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.534+0000 7f24cfcfb640 1 -- 172.21.15.16:0/783528021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24c8108c50 con 0x7f24c8102d90 2024-03-28T22:10:13.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.535+0000 7f24beffd640 1 -- 172.21.15.16:0/783528021 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f24c0005560 con 0x7f24c8102d90 2024-03-28T22:10:13.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.535+0000 7f24beffd640 1 --2- 172.21.15.16:0/783528021 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f24b40a1d80 0x7f24b40a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:13.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.536+0000 7f24beffd640 1 -- 172.21.15.16:0/783528021 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f24c00cdf50 con 0x7f24c8102d90 2024-03-28T22:10:13.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.536+0000 7f24cda97640 1 --2- 172.21.15.16:0/783528021 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f24b40a1d80 0x7f24b40a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:13.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.536+0000 7f24cda97640 1 --2- 172.21.15.16:0/783528021 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f24b40a1d80 0x7f24b40a4240 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f24c8069b50 tx=0x7f24c4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:13.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.540+0000 7f24beffd640 1 -- 172.21.15.16:0/783528021 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f24c0016030 con 0x7f24c8102d90 2024-03-28T22:10:13.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:13 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/220745499' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:10:13.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:13 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2484775395' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:10:13.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:13 smithi053 ceph-mon[35979]: pgmap v340: 97 pgs: 97 active+clean; 639 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-28T22:10:13.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:13 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3750293217' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:10:13.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:13 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/220745499' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:10:13.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:13 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2484775395' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:10:13.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:13 smithi063 ceph-mon[36249]: pgmap v340: 97 pgs: 97 active+clean; 639 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-28T22:10:13.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:13 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/3750293217' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:10:13.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.720+0000 7f24cfcfb640 1 -- 172.21.15.16:0/783528021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f24c8062350 con 0x7f24c8102d90 2024-03-28T22:10:13.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.720+0000 7f24beffd640 1 -- 172.21.15.16:0/783528021 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f24c009ad60 con 0x7f24c8102d90 2024-03-28T22:10:13.722 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-28T22:10:13.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.722+0000 7f24cfcfb640 1 -- 172.21.15.16:0/783528021 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f24b40a1d80 msgr2=0x7f24b40a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:13.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.722+0000 7f24cfcfb640 1 --2- 172.21.15.16:0/783528021 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f24b40a1d80 0x7f24b40a4240 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f24c8069b50 tx=0x7f24c4005f90 comp rx=0 tx=0).stop 2024-03-28T22:10:13.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.722+0000 7f24cfcfb640 1 -- 172.21.15.16:0/783528021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24c8102d90 msgr2=0x7f24c819f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:13.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.722+0000 7f24cfcfb640 1 --2- 172.21.15.16:0/783528021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24c8102d90 0x7f24c819f010 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f24c000eaa0 tx=0x7f24c000ef60 comp rx=0 tx=0).stop 2024-03-28T22:10:13.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.723+0000 7f24cfcfb640 1 -- 172.21.15.16:0/783528021 shutdown_connections 2024-03-28T22:10:13.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.723+0000 7f24cfcfb640 1 --2- 172.21.15.16:0/783528021 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f24b40a1d80 0x7f24b40a4240 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:13.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.723+0000 7f24cfcfb640 1 --2- 172.21.15.16:0/783528021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24c8102d90 0x7f24c819f010 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:13.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.723+0000 7f24cfcfb640 1 --2- 172.21.15.16:0/783528021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24c81001c0 0x7f24c8197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:13.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.723+0000 7f24cfcfb640 1 --2- 172.21.15.16:0/783528021 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f24c80ff800 0x7f24c8197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:13.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.723+0000 7f24cfcfb640 1 -- 172.21.15.16:0/783528021 >> 172.21.15.16:0/783528021 conn(0x7f24c80fb380 msgr2=0x7f24c8105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:13.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.723+0000 7f24cfcfb640 1 -- 172.21.15.16:0/783528021 shutdown_connections 2024-03-28T22:10:13.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.723+0000 7f24cfcfb640 1 -- 172.21.15.16:0/783528021 wait complete. 2024-03-28T22:10:13.736 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:10:13.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.836+0000 7f12c9ec2640 1 -- 172.21.15.16:0/2089405770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12c4102230 msgr2=0x7f12c4108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:13.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.836+0000 7f12c9ec2640 1 --2- 172.21.15.16:0/2089405770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12c4102230 0x7f12c4108ac0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f12bc009ca0 tx=0x7f12bc02c0f0 comp rx=0 tx=0).stop 2024-03-28T22:10:13.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.836+0000 7f12c9ec2640 1 -- 172.21.15.16:0/2089405770 shutdown_connections 2024-03-28T22:10:13.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.836+0000 7f12c9ec2640 1 --2- 172.21.15.16:0/2089405770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12c4102230 0x7f12c4108ac0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:13.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.836+0000 7f12c9ec2640 1 --2- 172.21.15.16:0/2089405770 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12c4101870 0x7f12c4101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:13.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.836+0000 7f12c9ec2640 1 --2- 172.21.15.16:0/2089405770 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12c4100680 0x7f12c4100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:13.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.836+0000 7f12c9ec2640 1 -- 172.21.15.16:0/2089405770 >> 172.21.15.16:0/2089405770 conn(0x7f12c40fbc10 msgr2=0x7f12c40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:13.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.836+0000 7f12c9ec2640 1 -- 172.21.15.16:0/2089405770 shutdown_connections 2024-03-28T22:10:13.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.836+0000 7f12c9ec2640 1 -- 172.21.15.16:0/2089405770 wait complete. 2024-03-28T22:10:13.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.836+0000 7f12c9ec2640 1 Processor -- start 2024-03-28T22:10:13.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.837+0000 7f12c9ec2640 1 -- start start 2024-03-28T22:10:13.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.837+0000 7f12c9ec2640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12c4100680 0x7f12c406aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:13.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.837+0000 7f12c9ec2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12c4101870 0x7f12c419c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:13.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.837+0000 7f12c9ec2640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12c4102230 0x7f12c419ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:13.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.837+0000 7f12c9ec2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f12c410c2e0 con 0x7f12c4101870 2024-03-28T22:10:13.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.837+0000 7f12c9ec2640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f12c410c160 con 0x7f12c4102230 2024-03-28T22:10:13.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.837+0000 7f12c9ec2640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f12c410c460 con 0x7f12c4100680 2024-03-28T22:10:13.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.837+0000 7f12c2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12c4101870 0x7f12c419c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:13.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.837+0000 7f12c2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12c4101870 0x7f12c419c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43644/0 (socket says 172.21.15.16:43644) 2024-03-28T22:10:13.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.837+0000 7f12c2ffd640 1 -- 172.21.15.16:0/984700675 learned_addr learned my addr 172.21.15.16:0/984700675 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:13.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.837+0000 7f12c3fff640 1 --2- 172.21.15.16:0/984700675 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12c4102230 0x7f12c419ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:13.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.837+0000 7f12c2ffd640 1 -- 172.21.15.16:0/984700675 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12c4100680 msgr2=0x7f12c406aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:13.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.837+0000 7f12c37fe640 1 --2- 172.21.15.16:0/984700675 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12c4100680 0x7f12c406aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:13.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.837+0000 7f12c2ffd640 1 --2- 172.21.15.16:0/984700675 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12c4100680 0x7f12c406aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:13.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.837+0000 7f12c2ffd640 1 -- 172.21.15.16:0/984700675 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12c4102230 msgr2=0x7f12c419ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:13.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.837+0000 7f12c2ffd640 1 --2- 172.21.15.16:0/984700675 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12c4102230 0x7f12c419ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:13.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.837+0000 7f12c2ffd640 1 -- 172.21.15.16:0/984700675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12c419f590 con 0x7f12c4101870 2024-03-28T22:10:13.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.837+0000 7f12c2ffd640 1 --2- 172.21.15.16:0/984700675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12c4101870 0x7f12c419c660 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f12b800c920 tx=0x7f12b800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:13.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.838+0000 7f12c0ff9640 1 -- 172.21.15.16:0/984700675 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b8007c70 con 0x7f12c4101870 2024-03-28T22:10:13.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.838+0000 7f12c0ff9640 1 -- 172.21.15.16:0/984700675 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12b8007e10 con 0x7f12c4101870 2024-03-28T22:10:13.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.838+0000 7f12c9ec2640 1 -- 172.21.15.16:0/984700675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12c419f880 con 0x7f12c4101870 2024-03-28T22:10:13.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.838+0000 7f12c0ff9640 1 -- 172.21.15.16:0/984700675 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b8018890 con 0x7f12c4101870 2024-03-28T22:10:13.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.838+0000 7f12c9ec2640 1 -- 172.21.15.16:0/984700675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12c419fe20 con 0x7f12c4101870 2024-03-28T22:10:13.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.839+0000 7f12c9ec2640 1 -- 172.21.15.16:0/984700675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12c41078d0 con 0x7f12c4101870 2024-03-28T22:10:13.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.840+0000 7f12c0ff9640 1 -- 172.21.15.16:0/984700675 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f12b8018a70 con 0x7f12c4101870 2024-03-28T22:10:13.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.840+0000 7f12c0ff9640 1 --2- 172.21.15.16:0/984700675 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f12ac0a1e00 0x7f12ac0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:13.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.840+0000 7f12c37fe640 1 --2- 172.21.15.16:0/984700675 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f12ac0a1e00 0x7f12ac0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:13.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.841+0000 7f12c0ff9640 1 -- 172.21.15.16:0/984700675 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f12b80cde70 con 0x7f12c4101870 2024-03-28T22:10:13.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.841+0000 7f12c37fe640 1 --2- 172.21.15.16:0/984700675 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f12ac0a1e00 0x7f12ac0a42c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f12c41016d0 tx=0x7f12b4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:13.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:13.845+0000 7f12c0ff9640 1 -- 172.21.15.16:0/984700675 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f12b8014030 con 0x7f12c4101870 2024-03-28T22:10:14.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.067+0000 7f12c9ec2640 1 -- 172.21.15.16:0/984700675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f12c402d170 con 0x7f12c4101870 2024-03-28T22:10:14.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.068+0000 7f12c0ff9640 1 -- 172.21.15.16:0/984700675 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11851 (secure 0 0 0) 0x7f12b809b2d0 con 0x7f12c4101870 2024-03-28T22:10:14.069 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:10:14.070 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/1644581227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1644581227},{"type":"v1","addr":"172.21.15.16:6839","nonce":1644581227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/2593062669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2593062669},{"type":"v1","addr":"172.21.15.16:6841","nonce":2593062669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:08:29.672976+0000","modified":"2024-03-28T22:08:34.341708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24566,"mds_1":24524,"mds_2":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24466":{"gid":24466,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/247249857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":247249857},{"type":"v1","addr":"172.21.15.63:6835","nonce":247249857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6837/2324231032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":2324231032},{"type":"v1","addr":"172.21.15.63:6837","nonce":2324231032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6839/1177197077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":1177197077},{"type":"v1","addr":"172.21.15.63:6839","nonce":1177197077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"e","rank":2,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.53:6835/350727634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":350727634},{"type":"v1","addr":"172.21.15.53:6835","nonce":350727634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/349877573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":349877573},{"type":"v1","addr":"172.21.15.53:6837","nonce":349877573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6839/3714153956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":3714153956},{"type":"v1","addr":"172.21.15.53:6839","nonce":3714153956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24566,"qdb_cluster":[24500,24566,24524]},"id":1}]} 2024-03-28T22:10:14.070 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 20 2024-03-28T22:10:14.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.070+0000 7f12c9ec2640 1 -- 172.21.15.16:0/984700675 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f12ac0a1e00 msgr2=0x7f12ac0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:14.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.070+0000 7f12c9ec2640 1 --2- 172.21.15.16:0/984700675 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f12ac0a1e00 0x7f12ac0a42c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f12c41016d0 tx=0x7f12b4005f90 comp rx=0 tx=0).stop 2024-03-28T22:10:14.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.070+0000 7f12c9ec2640 1 -- 172.21.15.16:0/984700675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12c4101870 msgr2=0x7f12c419c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:14.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.070+0000 7f12c9ec2640 1 --2- 172.21.15.16:0/984700675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12c4101870 0x7f12c419c660 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f12b800c920 tx=0x7f12b800cde0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.070+0000 7f12c9ec2640 1 -- 172.21.15.16:0/984700675 shutdown_connections 2024-03-28T22:10:14.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.070+0000 7f12c9ec2640 1 --2- 172.21.15.16:0/984700675 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f12ac0a1e00 0x7f12ac0a42c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.070+0000 7f12c9ec2640 1 --2- 172.21.15.16:0/984700675 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12c4102230 0x7f12c419ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.070+0000 7f12c9ec2640 1 --2- 172.21.15.16:0/984700675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12c4101870 0x7f12c419c660 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.070+0000 7f12c9ec2640 1 --2- 172.21.15.16:0/984700675 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12c4100680 0x7f12c406aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.070+0000 7f12c9ec2640 1 -- 172.21.15.16:0/984700675 >> 172.21.15.16:0/984700675 conn(0x7f12c40fbc10 msgr2=0x7f12c4104aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:14.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.070+0000 7f12c9ec2640 1 -- 172.21.15.16:0/984700675 shutdown_connections 2024-03-28T22:10:14.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.070+0000 7f12c9ec2640 1 -- 172.21.15.16:0/984700675 wait complete. 2024-03-28T22:10:14.083 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:10:14.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.184+0000 7febc08c9640 1 -- 172.21.15.16:0/1478933656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febbc101870 msgr2=0x7febbc101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:14.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.184+0000 7febc08c9640 1 --2- 172.21.15.16:0/1478933656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febbc101870 0x7febbc101cf0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7feba4009900 tx=0x7feba402bc30 comp rx=0 tx=0).stop 2024-03-28T22:10:14.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.184+0000 7febc08c9640 1 -- 172.21.15.16:0/1478933656 shutdown_connections 2024-03-28T22:10:14.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.184+0000 7febc08c9640 1 --2- 172.21.15.16:0/1478933656 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7febbc102230 0x7febbc108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.184+0000 7febc08c9640 1 --2- 172.21.15.16:0/1478933656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febbc101870 0x7febbc101cf0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.184+0000 7febc08c9640 1 --2- 172.21.15.16:0/1478933656 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febbc100680 0x7febbc100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.184+0000 7febc08c9640 1 -- 172.21.15.16:0/1478933656 >> 172.21.15.16:0/1478933656 conn(0x7febbc0fbc10 msgr2=0x7febbc0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:14.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.184+0000 7febc08c9640 1 -- 172.21.15.16:0/1478933656 shutdown_connections 2024-03-28T22:10:14.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.184+0000 7febc08c9640 1 -- 172.21.15.16:0/1478933656 wait complete. 2024-03-28T22:10:14.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.185+0000 7febc08c9640 1 Processor -- start 2024-03-28T22:10:14.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.185+0000 7febc08c9640 1 -- start start 2024-03-28T22:10:14.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.185+0000 7febc08c9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febbc100680 0x7febbc10b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.185+0000 7febc08c9640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7febbc101870 0x7febbc109640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.185+0000 7febc08c9640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febbc102230 0x7febbc109b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.185+0000 7febc08c9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7febbc10d480 con 0x7febbc100680 2024-03-28T22:10:14.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.185+0000 7febc08c9640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7febbc10d300 con 0x7febbc102230 2024-03-28T22:10:14.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.185+0000 7febc08c9640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7febbc10d600 con 0x7febbc101870 2024-03-28T22:10:14.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.185+0000 7febba59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febbc100680 0x7febbc10b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.185+0000 7febba59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febbc100680 0x7febbc10b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43656/0 (socket says 172.21.15.16:43656) 2024-03-28T22:10:14.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.185+0000 7febba59c640 1 -- 172.21.15.16:0/291516165 learned_addr learned my addr 172.21.15.16:0/291516165 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:14.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.185+0000 7febbad9d640 1 --2- 172.21.15.16:0/291516165 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febbc102230 0x7febbc109b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.185+0000 7febb3fff640 1 --2- 172.21.15.16:0/291516165 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7febbc101870 0x7febbc109640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.185+0000 7febba59c640 1 -- 172.21.15.16:0/291516165 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7febbc101870 msgr2=0x7febbc109640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:14.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.185+0000 7febba59c640 1 --2- 172.21.15.16:0/291516165 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7febbc101870 0x7febbc109640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.185+0000 7febba59c640 1 -- 172.21.15.16:0/291516165 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febbc102230 msgr2=0x7febbc109b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:14.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.185+0000 7febba59c640 1 --2- 172.21.15.16:0/291516165 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febbc102230 0x7febbc109b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.185+0000 7febba59c640 1 -- 172.21.15.16:0/291516165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febbc10a410 con 0x7febbc100680 2024-03-28T22:10:14.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.185+0000 7febb3fff640 1 --2- 172.21.15.16:0/291516165 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7febbc101870 0x7febbc109640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:10:14.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.185+0000 7febba59c640 1 --2- 172.21.15.16:0/291516165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febbc100680 0x7febbc10b000 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7febac00e9c0 tx=0x7febac00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:14.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.186+0000 7febb37fe640 1 -- 172.21.15.16:0/291516165 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febac00cd80 con 0x7febbc100680 2024-03-28T22:10:14.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.186+0000 7febc08c9640 1 -- 172.21.15.16:0/291516165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febbc1a1b70 con 0x7febbc100680 2024-03-28T22:10:14.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.186+0000 7febb37fe640 1 -- 172.21.15.16:0/291516165 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febac0035f0 con 0x7febbc100680 2024-03-28T22:10:14.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.186+0000 7febb37fe640 1 -- 172.21.15.16:0/291516165 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febac010690 con 0x7febbc100680 2024-03-28T22:10:14.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.186+0000 7febc08c9640 1 -- 172.21.15.16:0/291516165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febbc1a2110 con 0x7febbc100680 2024-03-28T22:10:14.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.187+0000 7febc08c9640 1 -- 172.21.15.16:0/291516165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febbc107940 con 0x7febbc100680 2024-03-28T22:10:14.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.188+0000 7febb37fe640 1 -- 172.21.15.16:0/291516165 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7febac0037a0 con 0x7febbc100680 2024-03-28T22:10:14.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.188+0000 7febb37fe640 1 --2- 172.21.15.16:0/291516165 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7feba80a1d80 0x7feba80a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.188+0000 7febb37fe640 1 -- 172.21.15.16:0/291516165 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7febac0ce230 con 0x7febbc100680 2024-03-28T22:10:14.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.188+0000 7febb3fff640 1 --2- 172.21.15.16:0/291516165 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7feba80a1d80 0x7feba80a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.189+0000 7febb3fff640 1 --2- 172.21.15.16:0/291516165 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7feba80a1d80 0x7feba80a4240 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7feba40096c0 tx=0x7feba402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:14.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.192+0000 7febb37fe640 1 -- 172.21.15.16:0/291516165 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7febac09b690 con 0x7febbc100680 2024-03-28T22:10:14.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.378+0000 7febc08c9640 1 -- 172.21.15.16:0/291516165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7febbc0623f0 con 0x7febbc100680 2024-03-28T22:10:14.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.379+0000 7febb37fe640 1 -- 172.21.15.16:0/291516165 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22441 (secure 0 0 0) 0x7febac09af90 con 0x7febbc100680 2024-03-28T22:10:14.381 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:10:14.381 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":88,"fsid":"798d9884-ed4e-11ee-95d0-87774f69a715","created":"2024-03-28T22:00:27.216099+0000","modified":"2024-03-28T22:09:30.047847+0000","last_up_change":"2024-03-28T22:07:40.251141+0000","last_in_change":"2024-03-28T22:07:19.456451+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:03:59.327832+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:08:28.123529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:08:28.919037+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"46e7c8f8-ecc8-43d6-ae53-0655728f3240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6803","nonce":1589692483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6805","nonce":1589692483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6809","nonce":1589692483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6807","nonce":1589692483}]},"public_addr":"172.21.15.16:6803/1589692483","cluster_addr":"172.21.15.16:6805/1589692483","heartbeat_back_addr":"172.21.15.16:6809/1589692483","heartbeat_front_addr":"172.21.15.16:6807/1589692483","state":["exists","up"]},{"osd":1,"uuid":"af6d9c22-2830-4ce2-a547-011df0c6cbad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6801","nonce":795531234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6803","nonce":795531234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6807","nonce":795531234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6805","nonce":795531234}]},"public_addr":"172.21.15.53:6801/795531234","cluster_addr":"172.21.15.53:6803/795531234","heartbeat_back_addr":"172.21.15.53:6807/795531234","heartbeat_front_addr":"172.21.15.53:6805/795531234","state":["exists","up"]},{"osd":2,"uuid":"b10adca9-110c-4f87-85e0-b7542709d31a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6801","nonce":2214131086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6803","nonce":2214131086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6807","nonce":2214131086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6805","nonce":2214131086}]},"public_addr":"172.21.15.63:6801/2214131086","cluster_addr":"172.21.15.63:6803/2214131086","heartbeat_back_addr":"172.21.15.63:6807/2214131086","heartbeat_front_addr":"172.21.15.63:6805/2214131086","state":["exists","up"]},{"osd":3,"uuid":"c83cf4bb-395a-4786-abf7-74ca93a33eb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6811","nonce":13176789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6813","nonce":13176789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6817","nonce":13176789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6815","nonce":13176789}]},"public_addr":"172.21.15.16:6811/13176789","cluster_addr":"172.21.15.16:6813/13176789","heartbeat_back_addr":"172.21.15.16:6817/13176789","heartbeat_front_addr":"172.21.15.16:6815/13176789","state":["exists","up"]},{"osd":4,"uuid":"941b3e54-47d6-4e85-8069-e730c3fa8eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6809","nonce":772161849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6811","nonce":772161849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6815","nonce":772161849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6813","nonce":772161849}]},"public_addr":"172.21.15.53:6809/772161849","cluster_addr":"172.21.15.53:6811/772161849","heartbeat_back_addr":"172.21.15.53:6815/772161849","heartbeat_front_addr":"172.21.15.53:6813/772161849","state":["exists","up"]},{"osd":5,"uuid":"51d89826-1d60-40fc-8fd8-09bfe1ab4125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6809","nonce":1602554852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6811","nonce":1602554852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6815","nonce":1602554852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6813","nonce":1602554852}]},"public_addr":"172.21.15.63:6809/1602554852","cluster_addr":"172.21.15.63:6811/1602554852","heartbeat_back_addr":"172.21.15.63:6815/1602554852","heartbeat_front_addr":"172.21.15.63:6813/1602554852","state":["exists","up"]},{"osd":6,"uuid":"978efe5d-2c55-4ff8-8a1e-9f891388945e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6819","nonce":4290459224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6821","nonce":4290459224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6825","nonce":4290459224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6823","nonce":4290459224}]},"public_addr":"172.21.15.16:6819/4290459224","cluster_addr":"172.21.15.16:6821/4290459224","heartbeat_back_addr":"172.21.15.16:6825/4290459224","heartbeat_front_addr":"172.21.15.16:6823/4290459224","state":["exists","up"]},{"osd":7,"uuid":"50d489cb-dbb8-48df-b868-b29edf12e04d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6817","nonce":1056467525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6819","nonce":1056467525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6823","nonce":1056467525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6821","nonce":1056467525}]},"public_addr":"172.21.15.53:6817/1056467525","cluster_addr":"172.21.15.53:6819/1056467525","heartbeat_back_addr":"172.21.15.53:6823/1056467525","heartbeat_front_addr":"172.21.15.53:6821/1056467525","state":["exists","up"]},{"osd":8,"uuid":"49b39414-cc9c-4057-ae3f-cc109cd979da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6817","nonce":2918516794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6819","nonce":2918516794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6823","nonce":2918516794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6821","nonce":2918516794}]},"public_addr":"172.21.15.63:6817/2918516794","cluster_addr":"172.21.15.63:6819/2918516794","heartbeat_back_addr":"172.21.15.63:6823/2918516794","heartbeat_front_addr":"172.21.15.63:6821/2918516794","state":["exists","up"]},{"osd":9,"uuid":"4c873e51-b783-4eb6-83a3-fee31722d60b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6827","nonce":1585624367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6829","nonce":1585624367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6833","nonce":1585624367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6831","nonce":1585624367}]},"public_addr":"172.21.15.16:6827/1585624367","cluster_addr":"172.21.15.16:6829/1585624367","heartbeat_back_addr":"172.21.15.16:6833/1585624367","heartbeat_front_addr":"172.21.15.16:6831/1585624367","state":["exists","up"]},{"osd":10,"uuid":"085a78bf-e2ad-481a-85a3-71d1fc549a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6825","nonce":3357446484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6827","nonce":3357446484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6831","nonce":3357446484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6829","nonce":3357446484}]},"public_addr":"172.21.15.53:6825/3357446484","cluster_addr":"172.21.15.53:6827/3357446484","heartbeat_back_addr":"172.21.15.53:6831/3357446484","heartbeat_front_addr":"172.21.15.53:6829/3357446484","state":["exists","up"]},{"osd":11,"uuid":"13302aa3-08b9-4639-9042-27d850b4d5f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6825","nonce":2167866570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6827","nonce":2167866570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6831","nonce":2167866570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6829","nonce":2167866570}]},"public_addr":"172.21.15.63:6825/2167866570","cluster_addr":"172.21.15.63:6827/2167866570","heartbeat_back_addr":"172.21.15.63:6831/2167866570","heartbeat_front_addr":"172.21.15.63:6829/2167866570","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:07.450323+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:31.363172+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:55.295021+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:19.794760+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:44.063492+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:08.195064+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:33.655328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:57.639414+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:22.402394+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:47.462050+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:12.226307+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:37.495625+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3821408150":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6801/1832422779":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/996598130":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/1062264551":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/317557779":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6801/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/3631746518":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6800/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/3116652786":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6800/1832422779":"2024-03-29T22:01:28.993434+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:10:14.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.381+0000 7febc08c9640 1 -- 172.21.15.16:0/291516165 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7feba80a1d80 msgr2=0x7feba80a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:14.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.381+0000 7febc08c9640 1 --2- 172.21.15.16:0/291516165 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7feba80a1d80 0x7feba80a4240 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7feba40096c0 tx=0x7feba402df90 comp rx=0 tx=0).stop 2024-03-28T22:10:14.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.381+0000 7febc08c9640 1 -- 172.21.15.16:0/291516165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febbc100680 msgr2=0x7febbc10b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:14.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.381+0000 7febc08c9640 1 --2- 172.21.15.16:0/291516165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febbc100680 0x7febbc10b000 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7febac00e9c0 tx=0x7febac00ee80 comp rx=0 tx=0).stop 2024-03-28T22:10:14.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.381+0000 7febc08c9640 1 -- 172.21.15.16:0/291516165 shutdown_connections 2024-03-28T22:10:14.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.381+0000 7febc08c9640 1 --2- 172.21.15.16:0/291516165 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7feba80a1d80 0x7feba80a4240 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.381+0000 7febc08c9640 1 --2- 172.21.15.16:0/291516165 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febbc102230 0x7febbc109b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.381+0000 7febc08c9640 1 --2- 172.21.15.16:0/291516165 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7febbc101870 0x7febbc109640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.381+0000 7febc08c9640 1 --2- 172.21.15.16:0/291516165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febbc100680 0x7febbc10b000 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.381+0000 7febc08c9640 1 -- 172.21.15.16:0/291516165 >> 172.21.15.16:0/291516165 conn(0x7febbc0fbc10 msgr2=0x7febbc104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:14.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.381+0000 7febc08c9640 1 -- 172.21.15.16:0/291516165 shutdown_connections 2024-03-28T22:10:14.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.381+0000 7febc08c9640 1 -- 172.21.15.16:0/291516165 wait complete. 2024-03-28T22:10:14.393 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:14 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/196712438' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:10:14.393 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:14 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/783528021' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:10:14.393 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:14 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/984700675' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:10:14.395 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-28T22:08:29.672976+0000', 'modified': '2024-03-28T22:08:34.341708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24566, 'mds_1': 24524, 'mds_2': 24500}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24466': {'gid': 24466, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.63:6835/247249857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.63:6834', 'nonce': 247249857}, {'type': 'v1', 'addr': '172.21.15.63:6835', 'nonce': 247249857}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24475': {'gid': 24475, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.63:6837/2324231032', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.63:6836', 'nonce': 2324231032}, {'type': 'v1', 'addr': '172.21.15.63:6837', 'nonce': 2324231032}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24493': {'gid': 24493, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.63:6839/1177197077', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.63:6838', 'nonce': 1177197077}, {'type': 'v1', 'addr': '172.21.15.63:6839', 'nonce': 1177197077}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24500': {'gid': 24500, 'name': 'e', 'rank': 2, 'incarnation': 18, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.53:6835/350727634', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6834', 'nonce': 350727634}, {'type': 'v1', 'addr': '172.21.15.53:6835', 'nonce': 350727634}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24524': {'gid': 24524, 'name': 'h', 'rank': 1, 'incarnation': 16, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.53:6837/349877573', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6836', 'nonce': 349877573}, {'type': 'v1', 'addr': '172.21.15.53:6837', 'nonce': 349877573}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24566': {'gid': 24566, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.53:6839/3714153956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6838', 'nonce': 3714153956}, {'type': 'v1', 'addr': '172.21.15.53:6839', 'nonce': 3714153956}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24566, 'qdb_cluster': [24500, 24566, 24524]} 2024-03-28T22:10:14.395 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-28T22:10:14.395 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2024-03-28T22:10:14.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.496+0000 7f1ca3e9d640 1 -- 172.21.15.16:0/3299239640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9c102950 msgr2=0x7f1c9c102dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:14.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.496+0000 7f1ca3e9d640 1 --2- 172.21.15.16:0/3299239640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9c102950 0x7f1c9c102dd0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c009900 tx=0x7f1c8c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:10:14.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.496+0000 7f1ca3e9d640 1 -- 172.21.15.16:0/3299239640 shutdown_connections 2024-03-28T22:10:14.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.496+0000 7f1ca3e9d640 1 --2- 172.21.15.16:0/3299239640 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c9c103310 0x7f1c9c109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.496+0000 7f1ca3e9d640 1 --2- 172.21.15.16:0/3299239640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9c102950 0x7f1c9c102dd0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.496+0000 7f1ca3e9d640 1 --2- 172.21.15.16:0/3299239640 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c9c101760 0x7f1c9c101b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.496+0000 7f1ca3e9d640 1 -- 172.21.15.16:0/3299239640 >> 172.21.15.16:0/3299239640 conn(0x7f1c9c0fccd0 msgr2=0x7f1c9c0ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:14.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.496+0000 7f1ca3e9d640 1 -- 172.21.15.16:0/3299239640 shutdown_connections 2024-03-28T22:10:14.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.496+0000 7f1ca3e9d640 1 -- 172.21.15.16:0/3299239640 wait complete. 2024-03-28T22:10:14.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.497+0000 7f1ca3e9d640 1 Processor -- start 2024-03-28T22:10:14.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.497+0000 7f1ca3e9d640 1 -- start start 2024-03-28T22:10:14.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.497+0000 7f1ca3e9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9c101760 0x7f1c9c197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.497+0000 7f1ca3e9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c9c102950 0x7f1c9c197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.497+0000 7f1ca3e9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c9c103310 0x7f1c9c19f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.497+0000 7f1ca3e9d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c10d3e0 con 0x7f1c9c101760 2024-03-28T22:10:14.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.497+0000 7f1ca3e9d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c10d260 con 0x7f1c9c103310 2024-03-28T22:10:14.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.497+0000 7f1ca3e9d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c10d560 con 0x7f1c9c102950 2024-03-28T22:10:14.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.497+0000 7f1ca1c39640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9c101760 0x7f1c9c197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.497+0000 7f1ca1c39640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9c101760 0x7f1c9c197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43674/0 (socket says 172.21.15.16:43674) 2024-03-28T22:10:14.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.497+0000 7f1ca1c39640 1 -- 172.21.15.16:0/2391355260 learned_addr learned my addr 172.21.15.16:0/2391355260 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:14.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.497+0000 7f1ca243a640 1 --2- 172.21.15.16:0/2391355260 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c9c103310 0x7f1c9c19f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.497+0000 7f1ca1438640 1 --2- 172.21.15.16:0/2391355260 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c9c102950 0x7f1c9c197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.497+0000 7f1ca1c39640 1 -- 172.21.15.16:0/2391355260 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c9c102950 msgr2=0x7f1c9c197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:14.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.497+0000 7f1ca1c39640 1 --2- 172.21.15.16:0/2391355260 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c9c102950 0x7f1c9c197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.497+0000 7f1ca1c39640 1 -- 172.21.15.16:0/2391355260 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c9c103310 msgr2=0x7f1c9c19f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:14.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.497+0000 7f1ca1c39640 1 --2- 172.21.15.16:0/2391355260 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c9c103310 0x7f1c9c19f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.497+0000 7f1ca1c39640 1 -- 172.21.15.16:0/2391355260 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c9c19f6a0 con 0x7f1c9c101760 2024-03-28T22:10:14.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.497+0000 7f1ca1438640 1 --2- 172.21.15.16:0/2391355260 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c9c102950 0x7f1c9c197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:10:14.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.497+0000 7f1ca1c39640 1 --2- 172.21.15.16:0/2391355260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9c101760 0x7f1c9c197a80 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f1c9800e9c0 tx=0x7f1c9800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:14.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.497+0000 7f1c92ffd640 1 -- 172.21.15.16:0/2391355260 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c9800cd80 con 0x7f1c9c101760 2024-03-28T22:10:14.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.497+0000 7f1c92ffd640 1 -- 172.21.15.16:0/2391355260 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c980035f0 con 0x7f1c9c101760 2024-03-28T22:10:14.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.497+0000 7f1ca3e9d640 1 -- 172.21.15.16:0/2391355260 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c9c19f990 con 0x7f1c9c101760 2024-03-28T22:10:14.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.498+0000 7f1c92ffd640 1 -- 172.21.15.16:0/2391355260 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c98010690 con 0x7f1c9c101760 2024-03-28T22:10:14.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.498+0000 7f1ca3e9d640 1 -- 172.21.15.16:0/2391355260 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c9c19ff00 con 0x7f1c9c101760 2024-03-28T22:10:14.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.500+0000 7f1c92ffd640 1 -- 172.21.15.16:0/2391355260 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f1c980037a0 con 0x7f1c9c101760 2024-03-28T22:10:14.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.500+0000 7f1ca6a80740 1 Processor -- start 2024-03-28T22:10:14.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.500+0000 7f1ca6a80740 1 -- start start 2024-03-28T22:10:14.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.500+0000 7f1ca6a80740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x563cd3f89610 0x563cd3f89a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.500+0000 7f1ca6a80740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563cd3f8a7e0 0x563cd3f8ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.500+0000 7f1ca6a80740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563cd3f8b1a0 0x563cd3f91a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.500+0000 7f1ca6a80740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563cd3eedaf0 con 0x563cd3f8b1a0 2024-03-28T22:10:14.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.500+0000 7f1ca6a80740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x563cd3e969d0 con 0x563cd3f89610 2024-03-28T22:10:14.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.500+0000 7f1ca6a80740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x563cd3ee9b90 con 0x563cd3f8a7e0 2024-03-28T22:10:14.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.500+0000 7f1c92ffd640 1 --2- 172.21.15.16:0/2391355260 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f1c880a1e00 0x7f1c880a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.500+0000 7f1ca1438640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563cd3f8b1a0 0x563cd3f91a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.500+0000 7f1ca243a640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x563cd3f89610 0x563cd3f89a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.500+0000 7f1ca1438640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563cd3f8b1a0 0x563cd3f91a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43686/0 (socket says 172.21.15.16:43686) 2024-03-28T22:10:14.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.500+0000 7f1ca1438640 1 -- 172.21.15.16:0/629301925 learned_addr learned my addr 172.21.15.16:0/629301925 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:14.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.500+0000 7f1ca243a640 1 --2- 172.21.15.16:0/2391355260 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f1c880a1e00 0x7f1c880a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.500+0000 7f1ca1c39640 1 --2- 172.21.15.16:0/629301925 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563cd3f8a7e0 0x563cd3f8ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.501+0000 7f1c92ffd640 1 -- 172.21.15.16:0/2391355260 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f1c980ce130 con 0x7f1c9c101760 2024-03-28T22:10:14.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.501+0000 7f1ca1438640 1 -- 172.21.15.16:0/629301925 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563cd3f8a7e0 msgr2=0x563cd3f8ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:14.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.501+0000 7f1ca1438640 1 --2- 172.21.15.16:0/629301925 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563cd3f8a7e0 0x563cd3f8ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.501+0000 7f1ca1438640 1 -- 172.21.15.16:0/629301925 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x563cd3f89610 msgr2=0x563cd3f89a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:14.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.501+0000 7f1ca1438640 1 --2- 172.21.15.16:0/629301925 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x563cd3f89610 0x563cd3f89a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.501+0000 7f1ca1438640 1 -- 172.21.15.16:0/629301925 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563cd3c022d0 con 0x563cd3f8b1a0 2024-03-28T22:10:14.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.501+0000 7f1ca243a640 1 --2- 172.21.15.16:0/629301925 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x563cd3f89610 0x563cd3f89a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:10:14.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.501+0000 7f1ca1c39640 1 --2- 172.21.15.16:0/629301925 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563cd3f8a7e0 0x563cd3f8ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:10:14.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.501+0000 7f1ca1438640 1 --2- 172.21.15.16:0/629301925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563cd3f8b1a0 0x563cd3f91a30 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c036040 tx=0x7f1c8c009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22a577f4b038b95 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:14.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.501+0000 7f1ca243a640 1 --2- 172.21.15.16:0/2391355260 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f1c880a1e00 0x7f1c880a42c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x563cd3f8a6e0 tx=0x7f1c94009600 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:14.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.501+0000 7f1c87fff640 1 -- 172.21.15.16:0/629301925 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c02cb80 con 0x563cd3f8b1a0 2024-03-28T22:10:14.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.501+0000 7f1c87fff640 1 -- 172.21.15.16:0/629301925 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c8c005040 con 0x563cd3f8b1a0 2024-03-28T22:10:14.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.501+0000 7f1c87fff640 1 -- 172.21.15.16:0/629301925 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c0049b0 con 0x563cd3f8b1a0 2024-03-28T22:10:14.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.502+0000 7f1ca6a80740 1 -- 172.21.15.16:0/629301925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563cd3f8b1a0 msgr2=0x563cd3f91a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:14.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.502+0000 7f1ca6a80740 1 --2- 172.21.15.16:0/629301925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563cd3f8b1a0 0x563cd3f91a30 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c036040 tx=0x7f1c8c009560 comp rx=0 tx=0).stop 2024-03-28T22:10:14.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.502+0000 7f1ca6a80740 1 -- 172.21.15.16:0/629301925 shutdown_connections 2024-03-28T22:10:14.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.502+0000 7f1ca6a80740 1 --2- 172.21.15.16:0/629301925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563cd3f8b1a0 0x563cd3f91a30 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.502+0000 7f1ca6a80740 1 --2- 172.21.15.16:0/629301925 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563cd3f8a7e0 0x563cd3f8ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.502+0000 7f1ca6a80740 1 --2- 172.21.15.16:0/629301925 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x563cd3f89610 0x563cd3f89a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.502+0000 7f1ca6a80740 1 -- 172.21.15.16:0/629301925 >> 172.21.15.16:0/629301925 conn(0x563cd3f84a00 msgr2=0x563cd3f84e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:14.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.502+0000 7f1ca6a80740 1 -- 172.21.15.16:0/629301925 shutdown_connections 2024-03-28T22:10:14.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.502+0000 7f1ca6a80740 1 -- 172.21.15.16:0/629301925 wait complete. 2024-03-28T22:10:14.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.502+0000 7f1ca6a80740 1 -- start start 2024-03-28T22:10:14.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.502+0000 7f1ca6a80740 1 Processor -- start 2024-03-28T22:10:14.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.503+0000 7f1ca6a80740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563cd3f89610 0x563cd3f93880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.503+0000 7f1ca6a80740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563cd3f8a7e0 0x563cd3f93dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.503+0000 7f1ca6a80740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x563cd3f8b1a0 0x563cd3f9ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.503+0000 7f1ca6a80740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563cd3f92150 con 0x563cd3f8a7e0 2024-03-28T22:10:14.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.503+0000 7f1ca6a80740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x563cd3ea9f70 con 0x563cd3f8b1a0 2024-03-28T22:10:14.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.503+0000 7f1ca6a80740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x563cd3f922d0 con 0x563cd3f89610 2024-03-28T22:10:14.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.503+0000 7f1ca1438640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563cd3f8a7e0 0x563cd3f93dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.503+0000 7f1ca6a80740 10 client.0 _ll_register_callbacks cb 0x563cd3c01ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:10:14.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.503+0000 7f1ca1438640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563cd3f8a7e0 0x563cd3f93dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43690/0 (socket says 172.21.15.16:43690) 2024-03-28T22:10:14.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.503+0000 7f1ca1438640 1 -- 172.21.15.16:0/2666157680 learned_addr learned my addr 172.21.15.16:0/2666157680 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:14.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.503+0000 7f1ca243a640 1 --2- 172.21.15.16:0/2666157680 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x563cd3f8b1a0 0x563cd3f9ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.503+0000 7f1ca1c39640 1 --2- 172.21.15.16:0/2666157680 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563cd3f89610 0x563cd3f93880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.503+0000 7f1ca1438640 1 -- 172.21.15.16:0/2666157680 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563cd3f89610 msgr2=0x563cd3f93880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:14.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.503+0000 7f1ca1438640 1 --2- 172.21.15.16:0/2666157680 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563cd3f89610 0x563cd3f93880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.503+0000 7f1ca1438640 1 -- 172.21.15.16:0/2666157680 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x563cd3f8b1a0 msgr2=0x563cd3f9ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:14.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.503+0000 7f1ca1438640 1 --2- 172.21.15.16:0/2666157680 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x563cd3f8b1a0 0x563cd3f9ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.503+0000 7f1ca1438640 1 -- 172.21.15.16:0/2666157680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563cd3ccc6b0 con 0x563cd3f8a7e0 2024-03-28T22:10:14.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.503+0000 7f1ca1438640 1 --2- 172.21.15.16:0/2666157680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563cd3f8a7e0 0x563cd3f93dc0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c009900 tx=0x7f1c8c0054a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e9c1eb4fcd842d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:14.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.503+0000 7f1c857fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:10:14.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.503+0000 7f1c857fa640 1 -- 172.21.15.16:0/2666157680 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c004b40 con 0x563cd3f8a7e0 2024-03-28T22:10:14.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.504+0000 7f1c857fa640 1 -- 172.21.15.16:0/2666157680 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f1c8c004ce0 con 0x563cd3f8a7e0 2024-03-28T22:10:14.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.504+0000 7f1ca6a80740 1 -- 172.21.15.16:0/2666157680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c8c005040 con 0x563cd3f8a7e0 2024-03-28T22:10:14.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.504+0000 7f1c857fa640 1 -- 172.21.15.16:0/2666157680 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f1c8c034440 con 0x563cd3f8a7e0 2024-03-28T22:10:14.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.504+0000 7f1ca6a80740 10 client.15207 fetch_fsmap learned FSMap version 20 2024-03-28T22:10:14.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.504+0000 7f1ca6a80740 1 -- 172.21.15.16:0/2666157680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x563cd3f9dd50 con 0x563cd3f8a7e0 2024-03-28T22:10:14.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.504+0000 7f1c857fa640 1 -- 172.21.15.16:0/2666157680 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c8c005040 con 0x563cd3f8a7e0 2024-03-28T22:10:14.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.504+0000 7f1c857fa640 1 -- 172.21.15.16:0/2666157680 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f1c8c059790 con 0x563cd3f8a7e0 2024-03-28T22:10:14.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.504+0000 7f1ca6a80740 10 client.15207 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T22:10:14.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.504+0000 7f1ca6a80740 10 client.15207 resolve_mds: resolved name 'e' to daemon mds.e 2024-03-28T22:10:14.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.504+0000 7f1ca6a80740 20 client.15207 populate_metadata read hostname 'smithi016' 2024-03-28T22:10:14.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.504+0000 7f1ca6a80740 1 --2- 172.21.15.16:0/2666157680 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x563cd3f9fef0 0x563cd3fa1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.504+0000 7f1ca6a80740 4 client.15207 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:10:14.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.504+0000 7f1ca6a80740 1 -- 172.21.15.16:0/2666157680 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563cd3efb030 con 0x563cd3f9fef0 2024-03-28T22:10:14.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.504+0000 7f1ca1c39640 1 --2- 172.21.15.16:0/2666157680 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x563cd3f9fef0 0x563cd3fa1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.505+0000 7f1c857fa640 10 client.15207 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:10:14.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.505+0000 7f1ca1c39640 1 --2- 172.21.15.16:0/2666157680 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x563cd3f9fef0 0x563cd3fa1360 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d90b60af8fc8fbf server_cookie=b713804a664bfe04 in_seq=0 out_seq=0 2024-03-28T22:10:14.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.506+0000 7f1c857fa640 1 -- 172.21.15.16:0/2666157680 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563cd3efb030 con 0x563cd3f9fef0 2024-03-28T22:10:14.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.506+0000 7f1c857fa640 10 client.15207 handle_command_reply: tid=0 2024-03-28T22:10:14.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.506+0000 7f1ca6a80740 1 client.15207 shutdown 2024-03-28T22:10:14.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.506+0000 7f1ca6a80740 1 -- 172.21.15.16:0/2666157680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563cd3f8a7e0 msgr2=0x563cd3f93dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:14.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.506+0000 7f1ca6a80740 1 --2- 172.21.15.16:0/2666157680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563cd3f8a7e0 0x563cd3f93dc0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c009900 tx=0x7f1c8c0054a0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.506+0000 7f1ca6a80740 1 -- 172.21.15.16:0/2666157680 shutdown_connections 2024-03-28T22:10:14.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.506+0000 7f1ca6a80740 1 --2- 172.21.15.16:0/2666157680 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x563cd3f9fef0 0x563cd3fa1360 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.506+0000 7f1ca6a80740 1 --2- 172.21.15.16:0/2666157680 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x563cd3f8b1a0 0x563cd3f9ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.506+0000 7f1ca6a80740 1 --2- 172.21.15.16:0/2666157680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563cd3f8a7e0 0x563cd3f93dc0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.506+0000 7f1ca6a80740 1 --2- 172.21.15.16:0/2666157680 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563cd3f89610 0x563cd3f93880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.506+0000 7f1c857fa640 0 client.15207 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:10:14.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.506+0000 7f1ca6a80740 1 -- 172.21.15.16:0/2666157680 >> 172.21.15.16:0/2666157680 conn(0x563cd3f84a00 msgr2=0x563cd3f83b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:14.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.506+0000 7f1ca6a80740 1 -- 172.21.15.16:0/2666157680 shutdown_connections 2024-03-28T22:10:14.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.506+0000 7f1ca6a80740 1 -- 172.21.15.16:0/2666157680 wait complete. 2024-03-28T22:10:14.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.506+0000 7f1ca6a80740 20 client.15207 trim_cache size 0 max 16384 2024-03-28T22:10:14.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.524+0000 7f1ca6a80740 1 Processor -- start 2024-03-28T22:10:14.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.524+0000 7f1ca6a80740 1 -- start start 2024-03-28T22:10:14.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.524+0000 7f1ca6a80740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563cd3f89610 0x563cd3f8abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.524+0000 7f1ca6a80740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563cd3f8a7e0 0x563cd3f9adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.524+0000 7f1ca6a80740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x563cd3f8b1a0 0x563cd3f84de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.524+0000 7f1ca6a80740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563cd3f9f6a0 con 0x563cd3f89610 2024-03-28T22:10:14.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.524+0000 7f1ca6a80740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x563cd3f9fd70 con 0x563cd3f8b1a0 2024-03-28T22:10:14.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.524+0000 7f1ca6a80740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x563cd3fa0440 con 0x563cd3f8a7e0 2024-03-28T22:10:14.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.524+0000 7f1ca1c39640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563cd3f89610 0x563cd3f8abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.524+0000 7f1ca1c39640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563cd3f89610 0x563cd3f8abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43704/0 (socket says 172.21.15.16:43704) 2024-03-28T22:10:14.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.524+0000 7f1ca1c39640 1 -- 172.21.15.16:0/2901606722 learned_addr learned my addr 172.21.15.16:0/2901606722 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:14.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.524+0000 7f1ca243a640 1 --2- 172.21.15.16:0/2901606722 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x563cd3f8b1a0 0x563cd3f84de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.525+0000 7f1ca1c39640 1 -- 172.21.15.16:0/2901606722 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563cd3f8a7e0 msgr2=0x563cd3f9adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:14.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.525+0000 7f1ca1c39640 1 --2- 172.21.15.16:0/2901606722 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563cd3f8a7e0 0x563cd3f9adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.525+0000 7f1ca1c39640 1 -- 172.21.15.16:0/2901606722 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x563cd3f8b1a0 msgr2=0x563cd3f84de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:14.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.525+0000 7f1ca1c39640 1 --2- 172.21.15.16:0/2901606722 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x563cd3f8b1a0 0x563cd3f84de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.525+0000 7f1ca1c39640 1 -- 172.21.15.16:0/2901606722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563cd3f947e0 con 0x563cd3f89610 2024-03-28T22:10:14.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.525+0000 7f1ca243a640 1 --2- 172.21.15.16:0/2901606722 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x563cd3f8b1a0 0x563cd3f84de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:10:14.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.525+0000 7f1ca1c39640 1 --2- 172.21.15.16:0/2901606722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563cd3f89610 0x563cd3f8abc0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f1c9809ab90 tx=0x7f1c9806cea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7a072cf8f26668 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:14.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.525+0000 7f1c877fe640 1 -- 172.21.15.16:0/2901606722 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c9809efe0 con 0x563cd3f89610 2024-03-28T22:10:14.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.525+0000 7f1c877fe640 1 -- 172.21.15.16:0/2901606722 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c980a3630 con 0x563cd3f89610 2024-03-28T22:10:14.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.525+0000 7f1c877fe640 1 -- 172.21.15.16:0/2901606722 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c9809a2e0 con 0x563cd3f89610 2024-03-28T22:10:14.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.525+0000 7f1ca6a80740 1 -- 172.21.15.16:0/2901606722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563cd3f89610 msgr2=0x563cd3f8abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:14.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.525+0000 7f1ca6a80740 1 --2- 172.21.15.16:0/2901606722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563cd3f89610 0x563cd3f8abc0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f1c9809ab90 tx=0x7f1c9806cea0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.525+0000 7f1ca6a80740 1 -- 172.21.15.16:0/2901606722 shutdown_connections 2024-03-28T22:10:14.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.525+0000 7f1ca6a80740 1 --2- 172.21.15.16:0/2901606722 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x563cd3f8b1a0 0x563cd3f84de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.525+0000 7f1ca6a80740 1 --2- 172.21.15.16:0/2901606722 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563cd3f8a7e0 0x563cd3f9adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.525+0000 7f1ca6a80740 1 --2- 172.21.15.16:0/2901606722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563cd3f89610 0x563cd3f8abc0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.525+0000 7f1ca6a80740 1 -- 172.21.15.16:0/2901606722 >> 172.21.15.16:0/2901606722 conn(0x563cd3f84a00 msgr2=0x563cd3f884f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:14.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.526+0000 7f1ca6a80740 1 -- 172.21.15.16:0/2901606722 shutdown_connections 2024-03-28T22:10:14.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.526+0000 7f1ca6a80740 1 -- 172.21.15.16:0/2901606722 wait complete. 2024-03-28T22:10:14.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.526+0000 7f1ca6a80740 1 -- start start 2024-03-28T22:10:14.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.526+0000 7f1ca6a80740 1 Processor -- start 2024-03-28T22:10:14.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.526+0000 7f1ca6a80740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563cd3f89610 0x563cd3f96b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.526+0000 7f1ca6a80740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x563cd3f8a7e0 0x563cd3f970a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.526+0000 7f1ca6a80740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563cd3f8b1a0 0x563cd3f9a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.526+0000 7f1ca6a80740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563cd3fa11e0 con 0x563cd3f8b1a0 2024-03-28T22:10:14.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.526+0000 7f1ca6a80740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x563cd3fa0b10 con 0x563cd3f8a7e0 2024-03-28T22:10:14.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.526+0000 7f1ca6a80740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x563cd3fa18b0 con 0x563cd3f89610 2024-03-28T22:10:14.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.526+0000 7f1ca6a80740 10 client.0 _ll_register_callbacks cb 0x7f1c94009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:10:14.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.526+0000 7f1ca243a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563cd3f8b1a0 0x563cd3f9a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.526+0000 7f1ca243a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563cd3f8b1a0 0x563cd3f9a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43720/0 (socket says 172.21.15.16:43720) 2024-03-28T22:10:14.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.526+0000 7f1ca1438640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x563cd3f8a7e0 0x563cd3f970a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.526+0000 7f1ca243a640 1 -- 172.21.15.16:0/1655381225 learned_addr learned my addr 172.21.15.16:0/1655381225 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:14.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.526+0000 7f1ca1c39640 1 --2- 172.21.15.16:0/1655381225 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563cd3f89610 0x563cd3f96b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.527+0000 7f1ca243a640 1 -- 172.21.15.16:0/1655381225 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563cd3f89610 msgr2=0x563cd3f96b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:14.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.527+0000 7f1ca243a640 1 --2- 172.21.15.16:0/1655381225 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563cd3f89610 0x563cd3f96b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.527+0000 7f1ca243a640 1 -- 172.21.15.16:0/1655381225 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x563cd3f8a7e0 msgr2=0x563cd3f970a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:14.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.527+0000 7f1ca243a640 1 --2- 172.21.15.16:0/1655381225 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x563cd3f8a7e0 0x563cd3f970a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.527+0000 7f1ca243a640 1 -- 172.21.15.16:0/1655381225 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563cd3f9a660 con 0x563cd3f8b1a0 2024-03-28T22:10:14.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.527+0000 7f1ca1c39640 1 --2- 172.21.15.16:0/1655381225 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563cd3f89610 0x563cd3f96b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:10:14.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.527+0000 7f1ca243a640 1 --2- 172.21.15.16:0/1655381225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563cd3f8b1a0 0x563cd3f9a070 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f1c94005c10 tx=0x7f1c94004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=998fe79c07203027 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:14.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.527+0000 7f1c857fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:10:14.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.527+0000 7f1c857fa640 1 -- 172.21.15.16:0/1655381225 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c94004d20 con 0x563cd3f8b1a0 2024-03-28T22:10:14.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.527+0000 7f1c857fa640 1 -- 172.21.15.16:0/1655381225 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f1c94005020 con 0x563cd3f8b1a0 2024-03-28T22:10:14.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.527+0000 7f1ca6a80740 1 -- 172.21.15.16:0/1655381225 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c980a3630 con 0x563cd3f8b1a0 2024-03-28T22:10:14.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.527+0000 7f1c857fa640 1 -- 172.21.15.16:0/1655381225 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f1c94005220 con 0x563cd3f8b1a0 2024-03-28T22:10:14.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.527+0000 7f1ca6a80740 10 client.15213 fetch_fsmap learned FSMap version 20 2024-03-28T22:10:14.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.527+0000 7f1ca6a80740 1 -- 172.21.15.16:0/1655381225 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x563cd4006240 con 0x563cd3f8b1a0 2024-03-28T22:10:14.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.527+0000 7f1c857fa640 1 -- 172.21.15.16:0/1655381225 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c980a3630 con 0x563cd3f8b1a0 2024-03-28T22:10:14.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.527+0000 7f1c857fa640 1 -- 172.21.15.16:0/1655381225 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f1c9402d070 con 0x563cd3f8b1a0 2024-03-28T22:10:14.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.527+0000 7f1ca6a80740 10 client.15213 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T22:10:14.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.527+0000 7f1ca6a80740 10 client.15213 resolve_mds: resolved name 'e' to daemon mds.e 2024-03-28T22:10:14.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.527+0000 7f1ca6a80740 20 client.15213 populate_metadata read hostname 'smithi016' 2024-03-28T22:10:14.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.527+0000 7f1ca6a80740 1 --2- 172.21.15.16:0/1655381225 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x563cd400aef0 0x563cd4009c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.527+0000 7f1ca6a80740 4 client.15213 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T22:10:14.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.527+0000 7f1ca6a80740 1 -- 172.21.15.16:0/1655381225 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "status"}) -- 0x563cd3ef5380 con 0x563cd400aef0 2024-03-28T22:10:14.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.528+0000 7f1ca1c39640 1 --2- 172.21.15.16:0/1655381225 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x563cd400aef0 0x563cd4009c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.528+0000 7f1c857fa640 10 client.15213 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:10:14.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.528+0000 7f1ca1c39640 1 --2- 172.21.15.16:0/1655381225 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x563cd400aef0 0x563cd4009c00 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=97b3b58acfdb0255 server_cookie=3c2f55de9ea0b07f in_seq=0 out_seq=0 2024-03-28T22:10:14.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.528+0000 7f1c857fa640 1 -- 172.21.15.16:0/1655381225 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x563cd3ef5380 con 0x563cd400aef0 2024-03-28T22:10:14.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.528+0000 7f1c857fa640 10 client.15213 handle_command_reply: tid=0 2024-03-28T22:10:14.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.528+0000 7f1ca6a80740 1 client.15213 shutdown 2024-03-28T22:10:14.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.528+0000 7f1ca6a80740 1 -- 172.21.15.16:0/1655381225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563cd3f8b1a0 msgr2=0x563cd3f9a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:14.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.528+0000 7f1ca6a80740 1 --2- 172.21.15.16:0/1655381225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563cd3f8b1a0 0x563cd3f9a070 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f1c94005c10 tx=0x7f1c94004ab0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.528+0000 7f1ca6a80740 1 -- 172.21.15.16:0/1655381225 shutdown_connections 2024-03-28T22:10:14.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.528+0000 7f1ca6a80740 1 --2- 172.21.15.16:0/1655381225 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x563cd400aef0 0x563cd4009c00 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.528+0000 7f1ca6a80740 1 --2- 172.21.15.16:0/1655381225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563cd3f8b1a0 0x563cd3f9a070 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.528+0000 7f1ca6a80740 1 --2- 172.21.15.16:0/1655381225 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x563cd3f8a7e0 0x563cd3f970a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.528+0000 7f1c857fa640 0 client.15213 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:10:14.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.528+0000 7f1ca6a80740 1 --2- 172.21.15.16:0/1655381225 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563cd3f89610 0x563cd3f96b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.528+0000 7f1ca6a80740 1 -- 172.21.15.16:0/1655381225 >> 172.21.15.16:0/1655381225 conn(0x563cd3f84a00 msgr2=0x563cd3f87c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:14.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.528+0000 7f1ca6a80740 1 -- 172.21.15.16:0/1655381225 shutdown_connections 2024-03-28T22:10:14.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.528+0000 7f1ca6a80740 1 -- 172.21.15.16:0/1655381225 wait complete. 2024-03-28T22:10:14.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.529+0000 7f1ca6a80740 20 client.15213 trim_cache size 0 max 16384 2024-03-28T22:10:14.537 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:10:14.537 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "798d9884-ed4e-11ee-95d0-87774f69a715", 2024-03-28T22:10:14.537 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 2, 2024-03-28T22:10:14.537 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24500, 2024-03-28T22:10:14.537 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-03-28T22:10:14.537 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-03-28T22:10:14.538 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-03-28T22:10:14.538 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 101.191592832, 2024-03-28T22:10:14.538 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 20, 2024-03-28T22:10:14.538 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 88, 2024-03-28T22:10:14.538 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 86, 2024-03-28T22:10:14.538 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 128.86171194299999 2024-03-28T22:10:14.538 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:10:14.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.529+0000 7f1ca3e9d640 1 -- 172.21.15.16:0/2391355260 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f1c880a1e00 msgr2=0x7f1c880a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:14.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.529+0000 7f1ca3e9d640 1 --2- 172.21.15.16:0/2391355260 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f1c880a1e00 0x7f1c880a42c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x563cd3f8a6e0 tx=0x7f1c94009600 comp rx=0 tx=0).stop 2024-03-28T22:10:14.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.529+0000 7f1ca3e9d640 1 -- 172.21.15.16:0/2391355260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9c101760 msgr2=0x7f1c9c197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:14.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.529+0000 7f1ca3e9d640 1 --2- 172.21.15.16:0/2391355260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9c101760 0x7f1c9c197a80 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f1c9800e9c0 tx=0x7f1c9800ee80 comp rx=0 tx=0).stop 2024-03-28T22:10:14.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.529+0000 7f1ca3e9d640 1 -- 172.21.15.16:0/2391355260 shutdown_connections 2024-03-28T22:10:14.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.529+0000 7f1ca3e9d640 1 --2- 172.21.15.16:0/2391355260 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f1c880a1e00 0x7f1c880a42c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.529+0000 7f1ca3e9d640 1 --2- 172.21.15.16:0/2391355260 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c9c103310 0x7f1c9c19f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.529+0000 7f1ca3e9d640 1 --2- 172.21.15.16:0/2391355260 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c9c102950 0x7f1c9c197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.529+0000 7f1ca3e9d640 1 --2- 172.21.15.16:0/2391355260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9c101760 0x7f1c9c197a80 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.529+0000 7f1ca3e9d640 1 -- 172.21.15.16:0/2391355260 >> 172.21.15.16:0/2391355260 conn(0x7f1c9c0fccd0 msgr2=0x7f1c9c105c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:14.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.529+0000 7f1ca3e9d640 1 -- 172.21.15.16:0/2391355260 shutdown_connections 2024-03-28T22:10:14.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.529+0000 7f1ca3e9d640 1 -- 172.21.15.16:0/2391355260 wait complete. 2024-03-28T22:10:14.542 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-28T22:10:14.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:14 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/196712438' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:10:14.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:14 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/783528021' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:10:14.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:14 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/984700675' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:10:14.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.644+0000 7f0b7e664640 1 -- 172.21.15.16:0/2907135350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b78101760 msgr2=0x7f0b78101b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:14.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.644+0000 7f0b7e664640 1 --2- 172.21.15.16:0/2907135350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b78101760 0x7f0b78101b80 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f0b68009900 tx=0x7f0b6802bc30 comp rx=0 tx=0).stop 2024-03-28T22:10:14.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.644+0000 7f0b7e664640 1 -- 172.21.15.16:0/2907135350 shutdown_connections 2024-03-28T22:10:14.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.644+0000 7f0b7e664640 1 --2- 172.21.15.16:0/2907135350 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b78103310 0x7f0b78109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.644+0000 7f0b7e664640 1 --2- 172.21.15.16:0/2907135350 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0b78102950 0x7f0b78102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.644+0000 7f0b7e664640 1 --2- 172.21.15.16:0/2907135350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b78101760 0x7f0b78101b80 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.644+0000 7f0b7e664640 1 -- 172.21.15.16:0/2907135350 >> 172.21.15.16:0/2907135350 conn(0x7f0b780fccd0 msgr2=0x7f0b780ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:14.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.644+0000 7f0b7e664640 1 -- 172.21.15.16:0/2907135350 shutdown_connections 2024-03-28T22:10:14.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.644+0000 7f0b7e664640 1 -- 172.21.15.16:0/2907135350 wait complete. 2024-03-28T22:10:14.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.644+0000 7f0b7e664640 1 Processor -- start 2024-03-28T22:10:14.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.644+0000 7f0b7e664640 1 -- start start 2024-03-28T22:10:14.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.645+0000 7f0b7e664640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b78101760 0x7f0b78197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.645+0000 7f0b7e664640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b78102950 0x7f0b78197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.645+0000 7f0b7e664640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0b78103310 0x7f0b7819f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.645+0000 7f0b7e664640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0b7810d3e0 con 0x7f0b78102950 2024-03-28T22:10:14.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.645+0000 7f0b7e664640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0b7810d260 con 0x7f0b78103310 2024-03-28T22:10:14.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.645+0000 7f0b7e664640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0b7810d560 con 0x7f0b78101760 2024-03-28T22:10:14.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.645+0000 7f0b777fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b78102950 0x7f0b78197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.645+0000 7f0b777fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b78102950 0x7f0b78197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43740/0 (socket says 172.21.15.16:43740) 2024-03-28T22:10:14.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.645+0000 7f0b777fe640 1 -- 172.21.15.16:0/1147840156 learned_addr learned my addr 172.21.15.16:0/1147840156 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:14.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.645+0000 7f0b7cc01640 1 --2- 172.21.15.16:0/1147840156 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0b78103310 0x7f0b7819f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.645+0000 7f0b77fff640 1 --2- 172.21.15.16:0/1147840156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b78101760 0x7f0b78197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.645+0000 7f0b777fe640 1 -- 172.21.15.16:0/1147840156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b78101760 msgr2=0x7f0b78197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:14.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.645+0000 7f0b777fe640 1 --2- 172.21.15.16:0/1147840156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b78101760 0x7f0b78197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.645+0000 7f0b777fe640 1 -- 172.21.15.16:0/1147840156 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0b78103310 msgr2=0x7f0b7819f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:14.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.645+0000 7f0b777fe640 1 --2- 172.21.15.16:0/1147840156 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0b78103310 0x7f0b7819f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.645+0000 7f0b777fe640 1 -- 172.21.15.16:0/1147840156 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b7819f6d0 con 0x7f0b78102950 2024-03-28T22:10:14.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.645+0000 7f0b777fe640 1 --2- 172.21.15.16:0/1147840156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b78102950 0x7f0b78197ff0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f0b6000ea60 tx=0x7f0b6000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:14.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.645+0000 7f0b757fa640 1 -- 172.21.15.16:0/1147840156 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b6000cde0 con 0x7f0b78102950 2024-03-28T22:10:14.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.645+0000 7f0b757fa640 1 -- 172.21.15.16:0/1147840156 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b600035f0 con 0x7f0b78102950 2024-03-28T22:10:14.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.645+0000 7f0b7e664640 1 -- 172.21.15.16:0/1147840156 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b7819f9c0 con 0x7f0b78102950 2024-03-28T22:10:14.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.645+0000 7f0b757fa640 1 -- 172.21.15.16:0/1147840156 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b60010670 con 0x7f0b78102950 2024-03-28T22:10:14.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.645+0000 7f0b7e664640 1 -- 172.21.15.16:0/1147840156 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b7819ff60 con 0x7f0b78102950 2024-03-28T22:10:14.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.647+0000 7f0b757fa640 1 -- 172.21.15.16:0/1147840156 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f0b600037a0 con 0x7f0b78102950 2024-03-28T22:10:14.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.648+0000 7f0b757fa640 1 --2- 172.21.15.16:0/1147840156 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0b640a1e00 0x7f0b640a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.648+0000 7f0b81243740 1 Processor -- start 2024-03-28T22:10:14.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.648+0000 7f0b81243740 1 -- start start 2024-03-28T22:10:14.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.648+0000 7f0b77fff640 1 --2- 172.21.15.16:0/1147840156 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0b640a1e00 0x7f0b640a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.648+0000 7f0b757fa640 1 -- 172.21.15.16:0/1147840156 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f0b600cdfd0 con 0x7f0b78102950 2024-03-28T22:10:14.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.648+0000 7f0b81243740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc22526610 0x55dc22526a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.648+0000 7f0b81243740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dc225277e0 0x55dc22527c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.648+0000 7f0b81243740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dc225281a0 0x55dc2252ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.648+0000 7f0b81243740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55dc2248aaf0 con 0x55dc22526610 2024-03-28T22:10:14.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.648+0000 7f0b81243740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55dc224339d0 con 0x55dc225277e0 2024-03-28T22:10:14.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.648+0000 7f0b81243740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55dc22486b90 con 0x55dc225281a0 2024-03-28T22:10:14.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.648+0000 7f0b77fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc22526610 0x55dc22526a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.648+0000 7f0b777fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dc225277e0 0x55dc22527c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.648+0000 7f0b77fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc22526610 0x55dc22526a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43756/0 (socket says 172.21.15.16:43756) 2024-03-28T22:10:14.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.648+0000 7f0b77fff640 1 -- 172.21.15.16:0/3056637568 learned_addr learned my addr 172.21.15.16:0/3056637568 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:14.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.648+0000 7f0b7cc01640 1 --2- 172.21.15.16:0/3056637568 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dc225281a0 0x55dc2252ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.649+0000 7f0b77fff640 1 --2- 172.21.15.16:0/1147840156 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0b640a1e00 0x7f0b640a42c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0b68000c00 tx=0x7f0b68005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:14.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.649+0000 7f0b77fff640 1 -- 172.21.15.16:0/3056637568 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dc225281a0 msgr2=0x55dc2252ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:14.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.649+0000 7f0b77fff640 1 --2- 172.21.15.16:0/3056637568 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dc225281a0 0x55dc2252ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.649+0000 7f0b77fff640 1 -- 172.21.15.16:0/3056637568 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dc225277e0 msgr2=0x55dc22527c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:14.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.649+0000 7f0b77fff640 1 --2- 172.21.15.16:0/3056637568 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dc225277e0 0x55dc22527c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.649+0000 7f0b77fff640 1 -- 172.21.15.16:0/3056637568 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc2219f2d0 con 0x55dc22526610 2024-03-28T22:10:14.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.649+0000 7f0b777fe640 1 --2- 172.21.15.16:0/3056637568 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dc225277e0 0x55dc22527c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:10:14.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.649+0000 7f0b77fff640 1 --2- 172.21.15.16:0/3056637568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc22526610 0x55dc22526a30 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f0b6802c710 tx=0x7f0b68046000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d4a5bf4a48d63ee server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:14.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.649+0000 7f0b6e7fc640 1 -- 172.21.15.16:0/3056637568 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b6804d070 con 0x55dc22526610 2024-03-28T22:10:14.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.649+0000 7f0b6e7fc640 1 -- 172.21.15.16:0/3056637568 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b68005520 con 0x55dc22526610 2024-03-28T22:10:14.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.649+0000 7f0b6e7fc640 1 -- 172.21.15.16:0/3056637568 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b68004d20 con 0x55dc22526610 2024-03-28T22:10:14.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.650+0000 7f0b81243740 1 -- 172.21.15.16:0/3056637568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc22526610 msgr2=0x55dc22526a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:14.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.650+0000 7f0b81243740 1 --2- 172.21.15.16:0/3056637568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc22526610 0x55dc22526a30 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f0b6802c710 tx=0x7f0b68046000 comp rx=0 tx=0).stop 2024-03-28T22:10:14.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.650+0000 7f0b81243740 1 -- 172.21.15.16:0/3056637568 shutdown_connections 2024-03-28T22:10:14.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.650+0000 7f0b81243740 1 --2- 172.21.15.16:0/3056637568 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dc225281a0 0x55dc2252ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.650+0000 7f0b81243740 1 --2- 172.21.15.16:0/3056637568 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dc225277e0 0x55dc22527c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.650+0000 7f0b81243740 1 --2- 172.21.15.16:0/3056637568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc22526610 0x55dc22526a30 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.650+0000 7f0b81243740 1 -- 172.21.15.16:0/3056637568 >> 172.21.15.16:0/3056637568 conn(0x55dc22521a00 msgr2=0x55dc22521e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:14.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.650+0000 7f0b81243740 1 -- 172.21.15.16:0/3056637568 shutdown_connections 2024-03-28T22:10:14.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.650+0000 7f0b81243740 1 -- 172.21.15.16:0/3056637568 wait complete. 2024-03-28T22:10:14.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.650+0000 7f0b81243740 1 -- start start 2024-03-28T22:10:14.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.650+0000 7f0b81243740 1 Processor -- start 2024-03-28T22:10:14.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.650+0000 7f0b81243740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dc22526610 0x55dc225308c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.650+0000 7f0b81243740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc225277e0 0x55dc22530e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.650+0000 7f0b81243740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dc225281a0 0x55dc22537e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.651+0000 7f0b81243740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55dc2252f150 con 0x55dc225277e0 2024-03-28T22:10:14.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.651+0000 7f0b81243740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55dc22446f70 con 0x55dc22526610 2024-03-28T22:10:14.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.651+0000 7f0b81243740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55dc2252f2d0 con 0x55dc225281a0 2024-03-28T22:10:14.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.651+0000 7f0b81243740 10 client.0 _ll_register_callbacks cb 0x55dc2219eac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:10:14.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.651+0000 7f0b777fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc225277e0 0x55dc22530e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.651+0000 7f0b777fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc225277e0 0x55dc22530e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43768/0 (socket says 172.21.15.16:43768) 2024-03-28T22:10:14.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.651+0000 7f0b777fe640 1 -- 172.21.15.16:0/290483295 learned_addr learned my addr 172.21.15.16:0/290483295 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:14.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.651+0000 7f0b77fff640 1 --2- 172.21.15.16:0/290483295 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dc22526610 0x55dc225308c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.651+0000 7f0b7cc01640 1 --2- 172.21.15.16:0/290483295 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dc225281a0 0x55dc22537e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.651+0000 7f0b777fe640 1 -- 172.21.15.16:0/290483295 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dc225281a0 msgr2=0x55dc22537e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:14.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.651+0000 7f0b777fe640 1 --2- 172.21.15.16:0/290483295 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dc225281a0 0x55dc22537e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.651+0000 7f0b777fe640 1 -- 172.21.15.16:0/290483295 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dc22526610 msgr2=0x55dc225308c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:14.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.651+0000 7f0b777fe640 1 --2- 172.21.15.16:0/290483295 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dc22526610 0x55dc225308c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.651+0000 7f0b777fe640 1 -- 172.21.15.16:0/290483295 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc222696b0 con 0x55dc225277e0 2024-03-28T22:10:14.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.651+0000 7f0b777fe640 1 --2- 172.21.15.16:0/290483295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc225277e0 0x55dc22530e00 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f0b6000c760 tx=0x7f0b6009b570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddf34b0b27be07fb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:14.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.651+0000 7f0b577fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:10:14.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.651+0000 7f0b577fe640 1 -- 172.21.15.16:0/290483295 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b600b2070 con 0x55dc225277e0 2024-03-28T22:10:14.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.651+0000 7f0b577fe640 1 -- 172.21.15.16:0/290483295 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f0b600b91c0 con 0x55dc225277e0 2024-03-28T22:10:14.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.651+0000 7f0b81243740 1 -- 172.21.15.16:0/290483295 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b68005520 con 0x55dc225277e0 2024-03-28T22:10:14.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.651+0000 7f0b577fe640 1 -- 172.21.15.16:0/290483295 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f0b600a34d0 con 0x55dc225277e0 2024-03-28T22:10:14.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.651+0000 7f0b81243740 10 client.15225 fetch_fsmap learned FSMap version 20 2024-03-28T22:10:14.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.651+0000 7f0b81243740 1 -- 172.21.15.16:0/290483295 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55dc2253ae90 con 0x55dc225277e0 2024-03-28T22:10:14.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.651+0000 7f0b577fe640 1 -- 172.21.15.16:0/290483295 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b68005520 con 0x55dc225277e0 2024-03-28T22:10:14.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.652+0000 7f0b577fe640 1 -- 172.21.15.16:0/290483295 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f0b60096a80 con 0x55dc225277e0 2024-03-28T22:10:14.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.652+0000 7f0b81243740 10 client.15225 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T22:10:14.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.652+0000 7f0b81243740 10 client.15225 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-28T22:10:14.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.652+0000 7f0b81243740 20 client.15225 populate_metadata read hostname 'smithi016' 2024-03-28T22:10:14.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.652+0000 7f0b81243740 1 --2- 172.21.15.16:0/290483295 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55dc2253d4b0 0x55dc2253e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.652+0000 7f0b81243740 4 client.15225 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:10:14.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.652+0000 7f0b81243740 1 -- 172.21.15.16:0/290483295 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dc22498030 con 0x55dc2253d4b0 2024-03-28T22:10:14.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.652+0000 7f0b77fff640 1 --2- 172.21.15.16:0/290483295 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55dc2253d4b0 0x55dc2253e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.652+0000 7f0b577fe640 10 client.15225 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:10:14.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.652+0000 7f0b77fff640 1 --2- 172.21.15.16:0/290483295 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55dc2253d4b0 0x55dc2253e920 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=275d5f6ea8d3cd1e server_cookie=f4856e7c761f7271 in_seq=0 out_seq=0 2024-03-28T22:10:14.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.653+0000 7f0b577fe640 1 -- 172.21.15.16:0/290483295 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dc22498030 con 0x55dc2253d4b0 2024-03-28T22:10:14.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.653+0000 7f0b577fe640 10 client.15225 handle_command_reply: tid=0 2024-03-28T22:10:14.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.653+0000 7f0b81243740 1 client.15225 shutdown 2024-03-28T22:10:14.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.654+0000 7f0b81243740 1 -- 172.21.15.16:0/290483295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc225277e0 msgr2=0x55dc22530e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:14.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.654+0000 7f0b81243740 1 --2- 172.21.15.16:0/290483295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc225277e0 0x55dc22530e00 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f0b6000c760 tx=0x7f0b6009b570 comp rx=0 tx=0).stop 2024-03-28T22:10:14.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.654+0000 7f0b81243740 1 -- 172.21.15.16:0/290483295 shutdown_connections 2024-03-28T22:10:14.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.654+0000 7f0b81243740 1 --2- 172.21.15.16:0/290483295 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55dc2253d4b0 0x55dc2253e920 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.654+0000 7f0b81243740 1 --2- 172.21.15.16:0/290483295 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dc225281a0 0x55dc22537e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.654+0000 7f0b577fe640 0 client.15225 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:10:14.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.654+0000 7f0b81243740 1 --2- 172.21.15.16:0/290483295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc225277e0 0x55dc22530e00 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.654+0000 7f0b81243740 1 --2- 172.21.15.16:0/290483295 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dc22526610 0x55dc225308c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.654+0000 7f0b81243740 1 -- 172.21.15.16:0/290483295 >> 172.21.15.16:0/290483295 conn(0x55dc22521a00 msgr2=0x55dc22520b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:14.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.654+0000 7f0b81243740 1 -- 172.21.15.16:0/290483295 shutdown_connections 2024-03-28T22:10:14.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.654+0000 7f0b81243740 1 -- 172.21.15.16:0/290483295 wait complete. 2024-03-28T22:10:14.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.654+0000 7f0b81243740 20 client.15225 trim_cache size 0 max 16384 2024-03-28T22:10:14.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.672+0000 7f0b81243740 1 Processor -- start 2024-03-28T22:10:14.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.672+0000 7f0b81243740 1 -- start start 2024-03-28T22:10:14.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.672+0000 7f0b81243740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dc22526610 0x55dc2252b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.672+0000 7f0b81243740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dc225277e0 0x55dc2252b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.672+0000 7f0b81243740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc225281a0 0x55dc225304d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.672+0000 7f0b81243740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55dc2253cc60 con 0x55dc225281a0 2024-03-28T22:10:14.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.672+0000 7f0b81243740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55dc2253d330 con 0x55dc225277e0 2024-03-28T22:10:14.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.672+0000 7f0b81243740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55dc2253da00 con 0x55dc22526610 2024-03-28T22:10:14.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.672+0000 7f0b7cc01640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc225281a0 0x55dc225304d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.672+0000 7f0b7cc01640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc225281a0 0x55dc225304d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43780/0 (socket says 172.21.15.16:43780) 2024-03-28T22:10:14.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.672+0000 7f0b7cc01640 1 -- 172.21.15.16:0/2431542856 learned_addr learned my addr 172.21.15.16:0/2431542856 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:14.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.672+0000 7f0b777fe640 1 --2- 172.21.15.16:0/2431542856 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dc225277e0 0x55dc2252b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.672+0000 7f0b77fff640 1 --2- 172.21.15.16:0/2431542856 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dc22526610 0x55dc2252b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.672+0000 7f0b7cc01640 1 -- 172.21.15.16:0/2431542856 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dc22526610 msgr2=0x55dc2252b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:14.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.672+0000 7f0b7cc01640 1 --2- 172.21.15.16:0/2431542856 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dc22526610 0x55dc2252b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.672+0000 7f0b7cc01640 1 -- 172.21.15.16:0/2431542856 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dc225277e0 msgr2=0x55dc2252b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:14.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.672+0000 7f0b7cc01640 1 --2- 172.21.15.16:0/2431542856 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dc225277e0 0x55dc2252b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.672+0000 7f0b7cc01640 1 -- 172.21.15.16:0/2431542856 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc22522770 con 0x55dc225281a0 2024-03-28T22:10:14.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.672+0000 7f0b777fe640 1 --2- 172.21.15.16:0/2431542856 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dc225277e0 0x55dc2252b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:10:14.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.672+0000 7f0b7cc01640 1 --2- 172.21.15.16:0/2431542856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc225281a0 0x55dc225304d0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f0b700137b0 tx=0x7f0b70013c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bec7b6df58520bfd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:14.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.672+0000 7f0b6dffb640 1 -- 172.21.15.16:0/2431542856 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b70003970 con 0x55dc225281a0 2024-03-28T22:10:14.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.673+0000 7f0b6dffb640 1 -- 172.21.15.16:0/2431542856 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b700118c0 con 0x55dc225281a0 2024-03-28T22:10:14.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.673+0000 7f0b6dffb640 1 -- 172.21.15.16:0/2431542856 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b70004650 con 0x55dc225281a0 2024-03-28T22:10:14.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.673+0000 7f0b81243740 1 -- 172.21.15.16:0/2431542856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc225281a0 msgr2=0x55dc225304d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:14.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.673+0000 7f0b81243740 1 --2- 172.21.15.16:0/2431542856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc225281a0 0x55dc225304d0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f0b700137b0 tx=0x7f0b70013c70 comp rx=0 tx=0).stop 2024-03-28T22:10:14.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.673+0000 7f0b81243740 1 -- 172.21.15.16:0/2431542856 shutdown_connections 2024-03-28T22:10:14.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.673+0000 7f0b81243740 1 --2- 172.21.15.16:0/2431542856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc225281a0 0x55dc225304d0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.673+0000 7f0b81243740 1 --2- 172.21.15.16:0/2431542856 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dc225277e0 0x55dc2252b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.673+0000 7f0b81243740 1 --2- 172.21.15.16:0/2431542856 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dc22526610 0x55dc2252b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.673+0000 7f0b81243740 1 -- 172.21.15.16:0/2431542856 >> 172.21.15.16:0/2431542856 conn(0x55dc22521a00 msgr2=0x55dc22521de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:14.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.673+0000 7f0b81243740 1 -- 172.21.15.16:0/2431542856 shutdown_connections 2024-03-28T22:10:14.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.673+0000 7f0b81243740 1 -- 172.21.15.16:0/2431542856 wait complete. 2024-03-28T22:10:14.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.673+0000 7f0b81243740 1 -- start start 2024-03-28T22:10:14.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.674+0000 7f0b81243740 1 Processor -- start 2024-03-28T22:10:14.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.674+0000 7f0b81243740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc22526610 0x55dc225a1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.674+0000 7f0b81243740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dc225277e0 0x55dc225a3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.674+0000 7f0b81243740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dc225281a0 0x55dc225a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.674+0000 7f0b81243740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55dc2253e7a0 con 0x55dc22526610 2024-03-28T22:10:14.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.674+0000 7f0b81243740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55dc2253e0d0 con 0x55dc225281a0 2024-03-28T22:10:14.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.674+0000 7f0b81243740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55dc2253ee70 con 0x55dc225277e0 2024-03-28T22:10:14.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.674+0000 7f0b81243740 10 client.0 _ll_register_callbacks cb 0x7f0b60097060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:10:14.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.674+0000 7f0b77fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc22526610 0x55dc225a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.674+0000 7f0b77fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc22526610 0x55dc225a1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43794/0 (socket says 172.21.15.16:43794) 2024-03-28T22:10:14.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.674+0000 7f0b77fff640 1 -- 172.21.15.16:0/667964777 learned_addr learned my addr 172.21.15.16:0/667964777 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:14.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.674+0000 7f0b777fe640 1 --2- 172.21.15.16:0/667964777 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dc225277e0 0x55dc225a3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.674+0000 7f0b7cc01640 1 --2- 172.21.15.16:0/667964777 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dc225281a0 0x55dc225a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.674+0000 7f0b77fff640 1 -- 172.21.15.16:0/667964777 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dc225277e0 msgr2=0x55dc225a3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:14.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.674+0000 7f0b77fff640 1 --2- 172.21.15.16:0/667964777 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dc225277e0 0x55dc225a3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.674+0000 7f0b77fff640 1 -- 172.21.15.16:0/667964777 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dc225281a0 msgr2=0x55dc225a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:14.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.674+0000 7f0b77fff640 1 --2- 172.21.15.16:0/667964777 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dc225281a0 0x55dc225a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.674+0000 7f0b77fff640 1 -- 172.21.15.16:0/667964777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc225a6540 con 0x55dc22526610 2024-03-28T22:10:14.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.674+0000 7f0b77fff640 1 --2- 172.21.15.16:0/667964777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc22526610 0x55dc225a1210 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f0b68005e30 tx=0x7f0b6802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efd7a8d8afc266e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:14.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.674+0000 7f0b577fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:10:14.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.674+0000 7f0b577fe640 1 -- 172.21.15.16:0/667964777 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b68061070 con 0x55dc22526610 2024-03-28T22:10:14.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.675+0000 7f0b81243740 1 -- 172.21.15.16:0/667964777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b700118c0 con 0x55dc22526610 2024-03-28T22:10:14.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.675+0000 7f0b577fe640 1 -- 172.21.15.16:0/667964777 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f0b6805d620 con 0x55dc22526610 2024-03-28T22:10:14.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.675+0000 7f0b577fe640 1 -- 172.21.15.16:0/667964777 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f0b6804d040 con 0x55dc22526610 2024-03-28T22:10:14.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.675+0000 7f0b81243740 10 client.15231 fetch_fsmap learned FSMap version 20 2024-03-28T22:10:14.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.675+0000 7f0b81243740 1 -- 172.21.15.16:0/667964777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55dc225a7840 con 0x55dc22526610 2024-03-28T22:10:14.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.675+0000 7f0b577fe640 1 -- 172.21.15.16:0/667964777 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b700118c0 con 0x55dc22526610 2024-03-28T22:10:14.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.675+0000 7f0b577fe640 1 -- 172.21.15.16:0/667964777 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f0b68004c50 con 0x55dc22526610 2024-03-28T22:10:14.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.675+0000 7f0b81243740 10 client.15231 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T22:10:14.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.675+0000 7f0b81243740 10 client.15231 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-28T22:10:14.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.675+0000 7f0b81243740 20 client.15231 populate_metadata read hostname 'smithi016' 2024-03-28T22:10:14.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.675+0000 7f0b81243740 1 --2- 172.21.15.16:0/667964777 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55dc225abd10 0x55dc225aaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.675+0000 7f0b81243740 4 client.15231 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T22:10:14.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.675+0000 7f0b81243740 1 -- 172.21.15.16:0/667964777 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "status"}) -- 0x55dc22492380 con 0x55dc225abd10 2024-03-28T22:10:14.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.675+0000 7f0b777fe640 1 --2- 172.21.15.16:0/667964777 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55dc225abd10 0x55dc225aaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.676+0000 7f0b577fe640 10 client.15231 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:10:14.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.676+0000 7f0b777fe640 1 --2- 172.21.15.16:0/667964777 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55dc225abd10 0x55dc225aaa20 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ae0faeecd70e973e server_cookie=630af807be193681 in_seq=0 out_seq=0 2024-03-28T22:10:14.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.676+0000 7f0b577fe640 1 -- 172.21.15.16:0/667964777 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55dc22492380 con 0x55dc225abd10 2024-03-28T22:10:14.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.676+0000 7f0b577fe640 10 client.15231 handle_command_reply: tid=0 2024-03-28T22:10:14.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.676+0000 7f0b81243740 1 client.15231 shutdown 2024-03-28T22:10:14.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.676+0000 7f0b81243740 1 -- 172.21.15.16:0/667964777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc22526610 msgr2=0x55dc225a1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:14.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.676+0000 7f0b81243740 1 --2- 172.21.15.16:0/667964777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc22526610 0x55dc225a1210 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f0b68005e30 tx=0x7f0b6802bc30 comp rx=0 tx=0).stop 2024-03-28T22:10:14.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.676+0000 7f0b81243740 1 -- 172.21.15.16:0/667964777 shutdown_connections 2024-03-28T22:10:14.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.676+0000 7f0b81243740 1 --2- 172.21.15.16:0/667964777 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55dc225abd10 0x55dc225aaa20 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.676+0000 7f0b81243740 1 --2- 172.21.15.16:0/667964777 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dc225281a0 0x55dc225a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.676+0000 7f0b81243740 1 --2- 172.21.15.16:0/667964777 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dc225277e0 0x55dc225a3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.676+0000 7f0b577fe640 0 client.15231 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:10:14.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.676+0000 7f0b81243740 1 --2- 172.21.15.16:0/667964777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc22526610 0x55dc225a1210 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.676+0000 7f0b81243740 1 -- 172.21.15.16:0/667964777 >> 172.21.15.16:0/667964777 conn(0x55dc22521a00 msgr2=0x55dc2252b140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:14.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.676+0000 7f0b81243740 1 -- 172.21.15.16:0/667964777 shutdown_connections 2024-03-28T22:10:14.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.676+0000 7f0b81243740 1 -- 172.21.15.16:0/667964777 wait complete. 2024-03-28T22:10:14.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.676+0000 7f0b81243740 20 client.15231 trim_cache size 0 max 16384 2024-03-28T22:10:14.685 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:10:14.685 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "798d9884-ed4e-11ee-95d0-87774f69a715", 2024-03-28T22:10:14.685 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 1, 2024-03-28T22:10:14.685 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24524, 2024-03-28T22:10:14.685 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-03-28T22:10:14.685 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-03-28T22:10:14.685 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-03-28T22:10:14.685 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 102.368335853, 2024-03-28T22:10:14.685 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 20, 2024-03-28T22:10:14.685 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 88, 2024-03-28T22:10:14.686 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 86, 2024-03-28T22:10:14.686 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 120.839552982 2024-03-28T22:10:14.686 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:10:14.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.677+0000 7f0b7e664640 1 -- 172.21.15.16:0/1147840156 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0b640a1e00 msgr2=0x7f0b640a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:14.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.677+0000 7f0b7e664640 1 --2- 172.21.15.16:0/1147840156 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0b640a1e00 0x7f0b640a42c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0b68000c00 tx=0x7f0b68005d50 comp rx=0 tx=0).stop 2024-03-28T22:10:14.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.677+0000 7f0b7e664640 1 -- 172.21.15.16:0/1147840156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b78102950 msgr2=0x7f0b78197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:14.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.677+0000 7f0b7e664640 1 --2- 172.21.15.16:0/1147840156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b78102950 0x7f0b78197ff0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f0b6000ea60 tx=0x7f0b6000ef20 comp rx=0 tx=0).stop 2024-03-28T22:10:14.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.677+0000 7f0b7e664640 1 -- 172.21.15.16:0/1147840156 shutdown_connections 2024-03-28T22:10:14.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.677+0000 7f0b7e664640 1 --2- 172.21.15.16:0/1147840156 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0b640a1e00 0x7f0b640a42c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.677+0000 7f0b7e664640 1 --2- 172.21.15.16:0/1147840156 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0b78103310 0x7f0b7819f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.677+0000 7f0b7e664640 1 --2- 172.21.15.16:0/1147840156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b78102950 0x7f0b78197ff0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.677+0000 7f0b7e664640 1 --2- 172.21.15.16:0/1147840156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b78101760 0x7f0b78197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.677+0000 7f0b7e664640 1 -- 172.21.15.16:0/1147840156 >> 172.21.15.16:0/1147840156 conn(0x7f0b780fccd0 msgr2=0x7f0b78105c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:14.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.677+0000 7f0b7e664640 1 -- 172.21.15.16:0/1147840156 shutdown_connections 2024-03-28T22:10:14.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.677+0000 7f0b7e664640 1 -- 172.21.15.16:0/1147840156 wait complete. 2024-03-28T22:10:14.688 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:14 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/196712438' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:10:14.688 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:14 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/783528021' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:10:14.688 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:14 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/984700675' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:10:14.689 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T22:10:14.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.790+0000 7f29e2325640 1 -- 172.21.15.16:0/1929615736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29dc100670 msgr2=0x7f29dc100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:14.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.790+0000 7f29e2325640 1 --2- 172.21.15.16:0/1929615736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29dc100670 0x7f29dc100a90 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f29cc009900 tx=0x7f29cc02bc30 comp rx=0 tx=0).stop 2024-03-28T22:10:14.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.790+0000 7f29e2325640 1 -- 172.21.15.16:0/1929615736 shutdown_connections 2024-03-28T22:10:14.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.791+0000 7f29e2325640 1 --2- 172.21.15.16:0/1929615736 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29dc102220 0x7f29dc108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.791+0000 7f29e2325640 1 --2- 172.21.15.16:0/1929615736 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29dc101860 0x7f29dc101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.791+0000 7f29e2325640 1 --2- 172.21.15.16:0/1929615736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29dc100670 0x7f29dc100a90 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.791+0000 7f29e2325640 1 -- 172.21.15.16:0/1929615736 >> 172.21.15.16:0/1929615736 conn(0x7f29dc0fbbe0 msgr2=0x7f29dc0fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:14.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.791+0000 7f29e2325640 1 -- 172.21.15.16:0/1929615736 shutdown_connections 2024-03-28T22:10:14.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.791+0000 7f29e2325640 1 -- 172.21.15.16:0/1929615736 wait complete. 2024-03-28T22:10:14.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.791+0000 7f29e2325640 1 Processor -- start 2024-03-28T22:10:14.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.791+0000 7f29e2325640 1 -- start start 2024-03-28T22:10:14.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.791+0000 7f29e2325640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29dc100670 0x7f29dc197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.791+0000 7f29e2325640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29dc101860 0x7f29dc1980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.791+0000 7f29e2325640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29dc102220 0x7f29dc19f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.791+0000 7f29e2325640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f29dc10c2f0 con 0x7f29dc101860 2024-03-28T22:10:14.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.791+0000 7f29e2325640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f29dc10c170 con 0x7f29dc100670 2024-03-28T22:10:14.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.791+0000 7f29e2325640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f29dc10c470 con 0x7f29dc102220 2024-03-28T22:10:14.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.792+0000 7f29db7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29dc101860 0x7f29dc1980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.792+0000 7f29db7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29dc101860 0x7f29dc1980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43816/0 (socket says 172.21.15.16:43816) 2024-03-28T22:10:14.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.792+0000 7f29db7fe640 1 -- 172.21.15.16:0/1587347380 learned_addr learned my addr 172.21.15.16:0/1587347380 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:14.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.792+0000 7f29dbfff640 1 --2- 172.21.15.16:0/1587347380 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29dc100670 0x7f29dc197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.792+0000 7f29db7fe640 1 -- 172.21.15.16:0/1587347380 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29dc102220 msgr2=0x7f29dc19f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:14.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.792+0000 7f29e08c2640 1 --2- 172.21.15.16:0/1587347380 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29dc102220 0x7f29dc19f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.792+0000 7f29db7fe640 1 --2- 172.21.15.16:0/1587347380 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29dc102220 0x7f29dc19f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.792+0000 7f29db7fe640 1 -- 172.21.15.16:0/1587347380 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29dc100670 msgr2=0x7f29dc197ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:14.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.792+0000 7f29db7fe640 1 --2- 172.21.15.16:0/1587347380 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29dc100670 0x7f29dc197ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.792+0000 7f29db7fe640 1 -- 172.21.15.16:0/1587347380 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29dc19f770 con 0x7f29dc101860 2024-03-28T22:10:14.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.792+0000 7f29db7fe640 1 --2- 172.21.15.16:0/1587347380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29dc101860 0x7f29dc1980e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f29d000e9f0 tx=0x7f29d000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:14.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.792+0000 7f29d97fa640 1 -- 172.21.15.16:0/1587347380 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d000cd80 con 0x7f29dc101860 2024-03-28T22:10:14.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.792+0000 7f29d97fa640 1 -- 172.21.15.16:0/1587347380 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29d0005500 con 0x7f29dc101860 2024-03-28T22:10:14.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.792+0000 7f29e2325640 1 -- 172.21.15.16:0/1587347380 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29dc19fa60 con 0x7f29dc101860 2024-03-28T22:10:14.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.792+0000 7f29d97fa640 1 -- 172.21.15.16:0/1587347380 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d0010690 con 0x7f29dc101860 2024-03-28T22:10:14.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.792+0000 7f29e2325640 1 -- 172.21.15.16:0/1587347380 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29dc19ffb0 con 0x7f29dc101860 2024-03-28T22:10:14.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.794+0000 7f29d97fa640 1 -- 172.21.15.16:0/1587347380 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f29d00046c0 con 0x7f29dc101860 2024-03-28T22:10:14.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.795+0000 7f29e4fe4740 1 Processor -- start 2024-03-28T22:10:14.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.795+0000 7f29e4fe4740 1 -- start start 2024-03-28T22:10:14.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.795+0000 7f29d97fa640 1 --2- 172.21.15.16:0/1587347380 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f29c40a1e00 0x7f29c40a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.795+0000 7f29e4fe4740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d20e35c610 0x55d20e35ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.795+0000 7f29e4fe4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d20e35d7e0 0x55d20e35dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.795+0000 7f29e4fe4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d20e35e1a0 0x55d20e364a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.795+0000 7f29e4fe4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d20e2c0af0 con 0x55d20e35e1a0 2024-03-28T22:10:14.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.795+0000 7f29e4fe4740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55d20e2699d0 con 0x55d20e35c610 2024-03-28T22:10:14.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.795+0000 7f29e4fe4740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d20e2bcb90 con 0x55d20e35d7e0 2024-03-28T22:10:14.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.795+0000 7f29e08c2640 1 --2- 172.21.15.16:0/1587347380 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f29c40a1e00 0x7f29c40a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.795+0000 7f29dbfff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d20e35c610 0x55d20e35ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.795+0000 7f29dbfff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d20e35c610 0x55d20e35ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:54252/0 (socket says 172.21.15.16:54252) 2024-03-28T22:10:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.795+0000 7f29dbfff640 1 -- 172.21.15.16:0/3664542484 learned_addr learned my addr 172.21.15.16:0/3664542484 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.795+0000 7f29e08c2640 1 --2- 172.21.15.16:0/3664542484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d20e35e1a0 0x55d20e364a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.795+0000 7f29d97fa640 1 -- 172.21.15.16:0/1587347380 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f29d00ce1c0 con 0x7f29dc101860 2024-03-28T22:10:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.795+0000 7f29db7fe640 1 --2- 172.21.15.16:0/3664542484 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d20e35d7e0 0x55d20e35dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.795+0000 7f29dbfff640 1 -- 172.21.15.16:0/3664542484 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d20e35d7e0 msgr2=0x55d20e35dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.795+0000 7f29dbfff640 1 --2- 172.21.15.16:0/3664542484 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d20e35d7e0 0x55d20e35dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.795+0000 7f29dbfff640 1 -- 172.21.15.16:0/3664542484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d20e35e1a0 msgr2=0x55d20e364a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.795+0000 7f29dbfff640 1 --2- 172.21.15.16:0/3664542484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d20e35e1a0 0x55d20e364a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.795+0000 7f29dbfff640 1 -- 172.21.15.16:0/3664542484 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d20dfd52d0 con 0x55d20e35c610 2024-03-28T22:10:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.795+0000 7f29e08c2640 1 --2- 172.21.15.16:0/1587347380 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f29c40a1e00 0x7f29c40a42c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f29d4005fd0 tx=0x7f29d4009db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.795+0000 7f29dbfff640 1 --2- 172.21.15.16:0/3664542484 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d20e35c610 0x55d20e35ca30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f29cc009900 tx=0x7f29cc009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49580ef8431e8d84 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.796+0000 7f29c27fc640 1 -- 172.21.15.16:0/3664542484 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29cc02cb80 con 0x55d20e35c610 2024-03-28T22:10:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.796+0000 7f29c27fc640 1 -- 172.21.15.16:0/3664542484 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29cc005040 con 0x55d20e35c610 2024-03-28T22:10:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.796+0000 7f29c27fc640 1 -- 172.21.15.16:0/3664542484 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29cc004920 con 0x55d20e35c610 2024-03-28T22:10:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.796+0000 7f29e4fe4740 1 -- 172.21.15.16:0/3664542484 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d20e35c610 msgr2=0x55d20e35ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.796+0000 7f29e4fe4740 1 --2- 172.21.15.16:0/3664542484 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d20e35c610 0x55d20e35ca30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f29cc009900 tx=0x7f29cc009560 comp rx=0 tx=0).stop 2024-03-28T22:10:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.796+0000 7f29e4fe4740 1 -- 172.21.15.16:0/3664542484 shutdown_connections 2024-03-28T22:10:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.796+0000 7f29e4fe4740 1 --2- 172.21.15.16:0/3664542484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d20e35e1a0 0x55d20e364a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.796+0000 7f29e4fe4740 1 --2- 172.21.15.16:0/3664542484 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d20e35d7e0 0x55d20e35dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.796+0000 7f29e4fe4740 1 --2- 172.21.15.16:0/3664542484 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d20e35c610 0x55d20e35ca30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.796+0000 7f29e4fe4740 1 -- 172.21.15.16:0/3664542484 >> 172.21.15.16:0/3664542484 conn(0x55d20e357a00 msgr2=0x55d20e357e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.796+0000 7f29e4fe4740 1 -- 172.21.15.16:0/3664542484 shutdown_connections 2024-03-28T22:10:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.796+0000 7f29e4fe4740 1 -- 172.21.15.16:0/3664542484 wait complete. 2024-03-28T22:10:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.797+0000 7f29e4fe4740 1 -- start start 2024-03-28T22:10:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.797+0000 7f29e4fe4740 1 Processor -- start 2024-03-28T22:10:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.797+0000 7f29e4fe4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d20e35c610 0x55d20e366890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.797+0000 7f29e4fe4740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d20e35d7e0 0x55d20e366dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.797+0000 7f29e4fe4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d20e35e1a0 0x55d20e36de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.797+0000 7f29e4fe4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d20e365150 con 0x55d20e35e1a0 2024-03-28T22:10:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.797+0000 7f29e4fe4740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55d20e27cf70 con 0x55d20e35d7e0 2024-03-28T22:10:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.797+0000 7f29e4fe4740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d20e3652d0 con 0x55d20e35c610 2024-03-28T22:10:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.797+0000 7f29e4fe4740 10 client.0 _ll_register_callbacks cb 0x55d20dfd4ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:10:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.797+0000 7f29e08c2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d20e35e1a0 0x55d20e36de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.797+0000 7f29e08c2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d20e35e1a0 0x55d20e36de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43844/0 (socket says 172.21.15.16:43844) 2024-03-28T22:10:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.797+0000 7f29e08c2640 1 -- 172.21.15.16:0/18358522 learned_addr learned my addr 172.21.15.16:0/18358522 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.797+0000 7f29db7fe640 1 --2- 172.21.15.16:0/18358522 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d20e35d7e0 0x55d20e366dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.797+0000 7f29dbfff640 1 --2- 172.21.15.16:0/18358522 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d20e35c610 0x55d20e366890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.797+0000 7f29e08c2640 1 -- 172.21.15.16:0/18358522 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d20e35c610 msgr2=0x55d20e366890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.797+0000 7f29e08c2640 1 --2- 172.21.15.16:0/18358522 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d20e35c610 0x55d20e366890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.797+0000 7f29e08c2640 1 -- 172.21.15.16:0/18358522 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d20e35d7e0 msgr2=0x55d20e366dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.797+0000 7f29e08c2640 1 --2- 172.21.15.16:0/18358522 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d20e35d7e0 0x55d20e366dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.797+0000 7f29e08c2640 1 -- 172.21.15.16:0/18358522 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d20e09f6b0 con 0x55d20e35e1a0 2024-03-28T22:10:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.797+0000 7f29db7fe640 1 --2- 172.21.15.16:0/18358522 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d20e35d7e0 0x55d20e366dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:10:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.798+0000 7f29e08c2640 1 --2- 172.21.15.16:0/18358522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d20e35e1a0 0x55d20e36de10 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f29d40092f0 tx=0x7f29d4009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7da3aaef11ccd4b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.798+0000 7f29bb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:10:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.798+0000 7f29bb7fe640 1 -- 172.21.15.16:0/18358522 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d4027070 con 0x55d20e35e1a0 2024-03-28T22:10:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.798+0000 7f29bb7fe640 1 -- 172.21.15.16:0/18358522 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f29d4004b20 con 0x55d20e35e1a0 2024-03-28T22:10:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.798+0000 7f29e4fe4740 1 -- 172.21.15.16:0/18358522 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29cc005040 con 0x55d20e35e1a0 2024-03-28T22:10:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.798+0000 7f29bb7fe640 1 -- 172.21.15.16:0/18358522 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f29d4032050 con 0x55d20e35e1a0 2024-03-28T22:10:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.798+0000 7f29e4fe4740 10 client.15243 fetch_fsmap learned FSMap version 20 2024-03-28T22:10:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.798+0000 7f29e4fe4740 1 -- 172.21.15.16:0/18358522 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55d20e370eb0 con 0x55d20e35e1a0 2024-03-28T22:10:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.798+0000 7f29bb7fe640 1 -- 172.21.15.16:0/18358522 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29cc005040 con 0x55d20e35e1a0 2024-03-28T22:10:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.798+0000 7f29bb7fe640 1 -- 172.21.15.16:0/18358522 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f29d4021480 con 0x55d20e35e1a0 2024-03-28T22:10:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.798+0000 7f29e4fe4740 10 client.15243 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T22:10:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.798+0000 7f29e4fe4740 10 client.15243 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T22:10:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.798+0000 7f29e4fe4740 20 client.15243 populate_metadata read hostname 'smithi016' 2024-03-28T22:10:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.798+0000 7f29e4fe4740 1 --2- 172.21.15.16:0/18358522 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d20e3735b0 0x55d20e374a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.798+0000 7f29e4fe4740 4 client.15243 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:10:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.798+0000 7f29e4fe4740 1 -- 172.21.15.16:0/18358522 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d20e2ce030 con 0x55d20e3735b0 2024-03-28T22:10:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.799+0000 7f29dbfff640 1 --2- 172.21.15.16:0/18358522 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d20e3735b0 0x55d20e374a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.799+0000 7f29bb7fe640 10 client.15243 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:10:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.799+0000 7f29dbfff640 1 --2- 172.21.15.16:0/18358522 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d20e3735b0 0x55d20e374a20 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b77c80f9c021d95 server_cookie=de94cf4dfd1c4b59 in_seq=0 out_seq=0 2024-03-28T22:10:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.800+0000 7f29bb7fe640 1 -- 172.21.15.16:0/18358522 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d20e2ce030 con 0x55d20e3735b0 2024-03-28T22:10:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.800+0000 7f29bb7fe640 10 client.15243 handle_command_reply: tid=0 2024-03-28T22:10:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.800+0000 7f29e4fe4740 1 client.15243 shutdown 2024-03-28T22:10:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.800+0000 7f29e4fe4740 1 -- 172.21.15.16:0/18358522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d20e35e1a0 msgr2=0x55d20e36de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.800+0000 7f29e4fe4740 1 --2- 172.21.15.16:0/18358522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d20e35e1a0 0x55d20e36de10 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f29d40092f0 tx=0x7f29d4009500 comp rx=0 tx=0).stop 2024-03-28T22:10:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.800+0000 7f29e4fe4740 1 -- 172.21.15.16:0/18358522 shutdown_connections 2024-03-28T22:10:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.800+0000 7f29e4fe4740 1 --2- 172.21.15.16:0/18358522 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d20e3735b0 0x55d20e374a20 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.800+0000 7f29e4fe4740 1 --2- 172.21.15.16:0/18358522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d20e35e1a0 0x55d20e36de10 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.800+0000 7f29bb7fe640 0 client.15243 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:10:14.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.800+0000 7f29e4fe4740 1 --2- 172.21.15.16:0/18358522 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d20e35d7e0 0x55d20e366dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.800+0000 7f29e4fe4740 1 --2- 172.21.15.16:0/18358522 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d20e35c610 0x55d20e366890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.800+0000 7f29e4fe4740 1 -- 172.21.15.16:0/18358522 >> 172.21.15.16:0/18358522 conn(0x55d20e357a00 msgr2=0x55d20e356b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:14.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.800+0000 7f29e4fe4740 1 -- 172.21.15.16:0/18358522 shutdown_connections 2024-03-28T22:10:14.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.800+0000 7f29e4fe4740 1 -- 172.21.15.16:0/18358522 wait complete. 2024-03-28T22:10:14.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.800+0000 7f29e4fe4740 20 client.15243 trim_cache size 0 max 16384 2024-03-28T22:10:14.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.818+0000 7f29e4fe4740 1 Processor -- start 2024-03-28T22:10:14.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.818+0000 7f29e4fe4740 1 -- start start 2024-03-28T22:10:14.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.818+0000 7f29e4fe4740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d20e35c610 0x55d20e361140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.818+0000 7f29e4fe4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d20e35d7e0 0x55d20e361680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.818+0000 7f29e4fe4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d20e35e1a0 0x55d20e36dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.818+0000 7f29e4fe4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d20e372d60 con 0x55d20e35e1a0 2024-03-28T22:10:14.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.818+0000 7f29e4fe4740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55d20e373430 con 0x55d20e35c610 2024-03-28T22:10:14.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.818+0000 7f29e4fe4740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d20e373b00 con 0x55d20e35d7e0 2024-03-28T22:10:14.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.819+0000 7f29e08c2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d20e35e1a0 0x55d20e36dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.819+0000 7f29e08c2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d20e35e1a0 0x55d20e36dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43848/0 (socket says 172.21.15.16:43848) 2024-03-28T22:10:14.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.819+0000 7f29e08c2640 1 -- 172.21.15.16:0/328152673 learned_addr learned my addr 172.21.15.16:0/328152673 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:14.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.819+0000 7f29dbfff640 1 --2- 172.21.15.16:0/328152673 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d20e35c610 0x55d20e361140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.819+0000 7f29db7fe640 1 --2- 172.21.15.16:0/328152673 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d20e35d7e0 0x55d20e361680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.819+0000 7f29e08c2640 1 -- 172.21.15.16:0/328152673 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d20e35d7e0 msgr2=0x55d20e361680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:14.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.819+0000 7f29e08c2640 1 --2- 172.21.15.16:0/328152673 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d20e35d7e0 0x55d20e361680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.819+0000 7f29e08c2640 1 -- 172.21.15.16:0/328152673 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d20e35c610 msgr2=0x55d20e361140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:14.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.819+0000 7f29e08c2640 1 --2- 172.21.15.16:0/328152673 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d20e35c610 0x55d20e361140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.819+0000 7f29e08c2640 1 -- 172.21.15.16:0/328152673 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d20e356040 con 0x55d20e35e1a0 2024-03-28T22:10:14.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.819+0000 7f29dbfff640 1 --2- 172.21.15.16:0/328152673 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d20e35c610 0x55d20e361140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:10:14.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.819+0000 7f29e08c2640 1 --2- 172.21.15.16:0/328152673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d20e35e1a0 0x55d20e36dd30 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f29d4009740 tx=0x7f29d4005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c4c7494adc40be1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:14.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.819+0000 7f29c1ffb640 1 -- 172.21.15.16:0/328152673 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d403b070 con 0x55d20e35e1a0 2024-03-28T22:10:14.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.819+0000 7f29c1ffb640 1 -- 172.21.15.16:0/328152673 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29d4014840 con 0x55d20e35e1a0 2024-03-28T22:10:14.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.819+0000 7f29c1ffb640 1 -- 172.21.15.16:0/328152673 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d4027050 con 0x55d20e35e1a0 2024-03-28T22:10:14.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.820+0000 7f29e4fe4740 1 -- 172.21.15.16:0/328152673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d20e35e1a0 msgr2=0x55d20e36dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:14.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.820+0000 7f29e4fe4740 1 --2- 172.21.15.16:0/328152673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d20e35e1a0 0x55d20e36dd30 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f29d4009740 tx=0x7f29d4005f10 comp rx=0 tx=0).stop 2024-03-28T22:10:14.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.820+0000 7f29e4fe4740 1 -- 172.21.15.16:0/328152673 shutdown_connections 2024-03-28T22:10:14.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.820+0000 7f29e4fe4740 1 --2- 172.21.15.16:0/328152673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d20e35e1a0 0x55d20e36dd30 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.820+0000 7f29e4fe4740 1 --2- 172.21.15.16:0/328152673 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d20e35d7e0 0x55d20e361680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.820+0000 7f29e4fe4740 1 --2- 172.21.15.16:0/328152673 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d20e35c610 0x55d20e361140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.820+0000 7f29e4fe4740 1 -- 172.21.15.16:0/328152673 >> 172.21.15.16:0/328152673 conn(0x55d20e357a00 msgr2=0x55d20e357de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:14.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.820+0000 7f29e4fe4740 1 -- 172.21.15.16:0/328152673 shutdown_connections 2024-03-28T22:10:14.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.820+0000 7f29e4fe4740 1 -- 172.21.15.16:0/328152673 wait complete. 2024-03-28T22:10:14.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.820+0000 7f29e4fe4740 1 -- start start 2024-03-28T22:10:14.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.820+0000 7f29e4fe4740 1 Processor -- start 2024-03-28T22:10:14.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.820+0000 7f29e4fe4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d20e35c610 0x55d20e369f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.820+0000 7f29e4fe4740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d20e35d7e0 0x55d20e36a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.820+0000 7f29e4fe4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d20e35e1a0 0x55d20e36ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.820+0000 7f29e4fe4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d20e3748a0 con 0x55d20e35c610 2024-03-28T22:10:14.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.820+0000 7f29e4fe4740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55d20e3741d0 con 0x55d20e35d7e0 2024-03-28T22:10:14.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.820+0000 7f29e4fe4740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d20e374f70 con 0x55d20e35e1a0 2024-03-28T22:10:14.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.820+0000 7f29e4fe4740 10 client.0 _ll_register_callbacks cb 0x7f29d40097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:10:14.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.821+0000 7f29dbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d20e35c610 0x55d20e369f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.821+0000 7f29dbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d20e35c610 0x55d20e369f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43858/0 (socket says 172.21.15.16:43858) 2024-03-28T22:10:14.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.821+0000 7f29dbfff640 1 -- 172.21.15.16:0/4175804327 learned_addr learned my addr 172.21.15.16:0/4175804327 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:14.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.821+0000 7f29db7fe640 1 --2- 172.21.15.16:0/4175804327 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d20e35d7e0 0x55d20e36a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.821+0000 7f29e08c2640 1 --2- 172.21.15.16:0/4175804327 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d20e35e1a0 0x55d20e36ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.821+0000 7f29dbfff640 1 -- 172.21.15.16:0/4175804327 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d20e35e1a0 msgr2=0x55d20e36ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:14.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.821+0000 7f29dbfff640 1 --2- 172.21.15.16:0/4175804327 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d20e35e1a0 0x55d20e36ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.821+0000 7f29dbfff640 1 -- 172.21.15.16:0/4175804327 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d20e35d7e0 msgr2=0x55d20e36a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:14.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.821+0000 7f29dbfff640 1 --2- 172.21.15.16:0/4175804327 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d20e35d7e0 0x55d20e36a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.821+0000 7f29dbfff640 1 -- 172.21.15.16:0/4175804327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d20e36d4a0 con 0x55d20e35c610 2024-03-28T22:10:14.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.821+0000 7f29dbfff640 1 --2- 172.21.15.16:0/4175804327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d20e35c610 0x55d20e369f60 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f29cc02c710 tx=0x7f29cc005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68db4d4807f1e29d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:14.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.821+0000 7f29bb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:10:14.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.821+0000 7f29bb7fe640 1 -- 172.21.15.16:0/4175804327 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29cc056060 con 0x55d20e35c610 2024-03-28T22:10:14.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.821+0000 7f29e4fe4740 1 -- 172.21.15.16:0/4175804327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29d4014840 con 0x55d20e35c610 2024-03-28T22:10:14.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.821+0000 7f29bb7fe640 1 -- 172.21.15.16:0/4175804327 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f29cc005440 con 0x55d20e35c610 2024-03-28T22:10:14.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.821+0000 7f29bb7fe640 1 -- 172.21.15.16:0/4175804327 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f29cc005640 con 0x55d20e35c610 2024-03-28T22:10:14.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.821+0000 7f29e4fe4740 10 client.15249 fetch_fsmap learned FSMap version 20 2024-03-28T22:10:14.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.821+0000 7f29e4fe4740 1 -- 172.21.15.16:0/4175804327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55d20e3d9660 con 0x55d20e35c610 2024-03-28T22:10:14.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.821+0000 7f29bb7fe640 1 -- 172.21.15.16:0/4175804327 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29d4014840 con 0x55d20e35c610 2024-03-28T22:10:14.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.822+0000 7f29bb7fe640 1 -- 172.21.15.16:0/4175804327 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f29cc004bc0 con 0x55d20e35c610 2024-03-28T22:10:14.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.822+0000 7f29e4fe4740 10 client.15249 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T22:10:14.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.822+0000 7f29e4fe4740 10 client.15249 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T22:10:14.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.822+0000 7f29e4fe4740 20 client.15249 populate_metadata read hostname 'smithi016' 2024-03-28T22:10:14.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.822+0000 7f29e4fe4740 1 --2- 172.21.15.16:0/4175804327 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d20e3dde90 0x55d20e3da9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:14.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.822+0000 7f29e4fe4740 4 client.15249 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T22:10:14.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.822+0000 7f29e4fe4740 1 -- 172.21.15.16:0/4175804327 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "status"}) -- 0x55d20e2c8380 con 0x55d20e3dde90 2024-03-28T22:10:14.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.822+0000 7f29db7fe640 1 --2- 172.21.15.16:0/4175804327 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d20e3dde90 0x55d20e3da9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:14.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.822+0000 7f29bb7fe640 10 client.15249 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:10:14.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.822+0000 7f29db7fe640 1 --2- 172.21.15.16:0/4175804327 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d20e3dde90 0x55d20e3da9a0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebd55bc1d9a4901f server_cookie=a3f38227000049a3 in_seq=0 out_seq=0 2024-03-28T22:10:14.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.822+0000 7f29bb7fe640 1 -- 172.21.15.16:0/4175804327 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55d20e2c8380 con 0x55d20e3dde90 2024-03-28T22:10:14.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.822+0000 7f29bb7fe640 10 client.15249 handle_command_reply: tid=0 2024-03-28T22:10:14.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.822+0000 7f29e4fe4740 1 client.15249 shutdown 2024-03-28T22:10:14.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.823+0000 7f29e4fe4740 1 -- 172.21.15.16:0/4175804327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d20e35c610 msgr2=0x55d20e369f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:14.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.823+0000 7f29e4fe4740 1 --2- 172.21.15.16:0/4175804327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d20e35c610 0x55d20e369f60 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f29cc02c710 tx=0x7f29cc005850 comp rx=0 tx=0).stop 2024-03-28T22:10:14.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.823+0000 7f29e4fe4740 1 -- 172.21.15.16:0/4175804327 shutdown_connections 2024-03-28T22:10:14.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.823+0000 7f29e4fe4740 1 --2- 172.21.15.16:0/4175804327 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d20e3dde90 0x55d20e3da9a0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.823+0000 7f29e4fe4740 1 --2- 172.21.15.16:0/4175804327 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d20e35e1a0 0x55d20e36ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.823+0000 7f29e4fe4740 1 --2- 172.21.15.16:0/4175804327 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d20e35d7e0 0x55d20e36a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.823+0000 7f29e4fe4740 1 --2- 172.21.15.16:0/4175804327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d20e35c610 0x55d20e369f60 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.823+0000 7f29bb7fe640 0 client.15249 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:10:14.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.823+0000 7f29e4fe4740 1 -- 172.21.15.16:0/4175804327 >> 172.21.15.16:0/4175804327 conn(0x55d20e357a00 msgr2=0x55d20e361140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:14.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.823+0000 7f29e4fe4740 1 -- 172.21.15.16:0/4175804327 shutdown_connections 2024-03-28T22:10:14.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.823+0000 7f29e4fe4740 1 -- 172.21.15.16:0/4175804327 wait complete. 2024-03-28T22:10:14.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.823+0000 7f29e4fe4740 20 client.15249 trim_cache size 0 max 16384 2024-03-28T22:10:14.831 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:10:14.831 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "798d9884-ed4e-11ee-95d0-87774f69a715", 2024-03-28T22:10:14.831 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2024-03-28T22:10:14.831 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24566, 2024-03-28T22:10:14.831 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-03-28T22:10:14.831 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-03-28T22:10:14.831 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-03-28T22:10:14.832 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 104.508752297, 2024-03-28T22:10:14.832 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 20, 2024-03-28T22:10:14.832 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 88, 2024-03-28T22:10:14.832 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 86, 2024-03-28T22:10:14.832 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 109.907973885 2024-03-28T22:10:14.832 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:10:14.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.823+0000 7f29e2325640 1 -- 172.21.15.16:0/1587347380 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f29c40a1e00 msgr2=0x7f29c40a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:14.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.823+0000 7f29e2325640 1 --2- 172.21.15.16:0/1587347380 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f29c40a1e00 0x7f29c40a42c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f29d4005fd0 tx=0x7f29d4009db0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.823+0000 7f29e2325640 1 -- 172.21.15.16:0/1587347380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29dc101860 msgr2=0x7f29dc1980e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:14.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.823+0000 7f29e2325640 1 --2- 172.21.15.16:0/1587347380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29dc101860 0x7f29dc1980e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f29d000e9f0 tx=0x7f29d000edf0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.823+0000 7f29e2325640 1 -- 172.21.15.16:0/1587347380 shutdown_connections 2024-03-28T22:10:14.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.823+0000 7f29e2325640 1 --2- 172.21.15.16:0/1587347380 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f29c40a1e00 0x7f29c40a42c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.823+0000 7f29e2325640 1 --2- 172.21.15.16:0/1587347380 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29dc102220 0x7f29dc19f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.823+0000 7f29e2325640 1 --2- 172.21.15.16:0/1587347380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29dc101860 0x7f29dc1980e0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.823+0000 7f29e2325640 1 --2- 172.21.15.16:0/1587347380 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29dc100670 0x7f29dc197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:14.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.823+0000 7f29e2325640 1 -- 172.21.15.16:0/1587347380 >> 172.21.15.16:0/1587347380 conn(0x7f29dc0fbbe0 msgr2=0x7f29dc0fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:14.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.823+0000 7f29e2325640 1 -- 172.21.15.16:0/1587347380 shutdown_connections 2024-03-28T22:10:14.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:14.823+0000 7f29e2325640 1 -- 172.21.15.16:0/1587347380 wait complete. 2024-03-28T22:10:14.836 INFO:teuthology.run_tasks:Running task kclient... 2024-03-28T22:10:14.859 INFO:tasks.kclient:Mounting kernel clients... 2024-03-28T22:10:14.859 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-28T22:10:14.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:10:14.860 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-03-28T22:10:14.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:10:14.875 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link delete ceph-brx 2024-03-28T22:10:14.943 INFO:teuthology.orchestra.run.smithi016.stderr:Cannot find device "ceph-brx" 2024-03-28T22:10:14.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T22:10:14.945 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'wsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-03-28T22:10:14.945 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-28T22:10:14.945 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-28T22:10:14.945 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph fs ls 2024-03-28T22:10:15.227 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:10:15.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:15 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/291516165' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:10:15.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:15 smithi053 ceph-mon[35979]: pgmap v341: 97 pgs: 97 active+clean; 639 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T22:10:15.580 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:15 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/291516165' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:10:15.580 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:15 smithi016 ceph-mon[32374]: pgmap v341: 97 pgs: 97 active+clean; 639 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T22:10:15.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:15 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/291516165' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:10:15.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:15 smithi063 ceph-mon[36249]: pgmap v341: 97 pgs: 97 active+clean; 639 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-28T22:10:16.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.992+0000 7f36192dc700 1 -- 172.21.15.16:0/828928297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3614108270 msgr2=0x7f3614108650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:16.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.992+0000 7f36192dc700 1 --2- 172.21.15.16:0/828928297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3614108270 0x7f3614108650 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f36040099c0 tx=0x7f3604009cd0 comp rx=0 tx=0).stop 2024-03-28T22:10:16.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.992+0000 7f36192dc700 1 -- 172.21.15.16:0/828928297 shutdown_connections 2024-03-28T22:10:16.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.992+0000 7f36192dc700 1 --2- 172.21.15.16:0/828928297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3614102c50 0x7f361410f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:16.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.992+0000 7f36192dc700 1 --2- 172.21.15.16:0/828928297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3614108270 0x7f3614108650 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:16.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.992+0000 7f36192dc700 1 --2- 172.21.15.16:0/828928297 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3614102290 0x7f3614102710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:16.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.992+0000 7f36192dc700 1 -- 172.21.15.16:0/828928297 >> 172.21.15.16:0/828928297 conn(0x7f3614079370 msgr2=0x7f3614079780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:16.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.992+0000 7f36192dc700 1 -- 172.21.15.16:0/828928297 shutdown_connections 2024-03-28T22:10:16.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.992+0000 7f36192dc700 1 -- 172.21.15.16:0/828928297 wait complete. 2024-03-28T22:10:16.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.992+0000 7f36192dc700 1 Processor -- start 2024-03-28T22:10:16.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.992+0000 7f36192dc700 1 -- start start 2024-03-28T22:10:16.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.993+0000 7f36192dc700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3614102290 0x7f36141a06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:16.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.993+0000 7f36192dc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3614102c50 0x7f36141a0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:16.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.993+0000 7f36192dc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3614108270 0x7f361419a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:16.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.993+0000 7f36192dc700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f36141124f0 con 0x7f3614102c50 2024-03-28T22:10:16.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.993+0000 7f36192dc700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3614112370 con 0x7f3614108270 2024-03-28T22:10:16.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.993+0000 7f36192dc700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3614112670 con 0x7f3614102290 2024-03-28T22:10:16.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.993+0000 7f36127fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3614102c50 0x7f36141a0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:17.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.993+0000 7f36127fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3614102c50 0x7f36141a0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43864/0 (socket says 172.21.15.16:43864) 2024-03-28T22:10:17.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.993+0000 7f36127fc700 1 -- 172.21.15.16:0/3725231601 learned_addr learned my addr 172.21.15.16:0/3725231601 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:17.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.993+0000 7f36137fe700 1 --2- 172.21.15.16:0/3725231601 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3614108270 0x7f361419a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:17.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.993+0000 7f3612ffd700 1 --2- 172.21.15.16:0/3725231601 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3614102290 0x7f36141a06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:17.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.993+0000 7f36127fc700 1 -- 172.21.15.16:0/3725231601 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3614102290 msgr2=0x7f36141a06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:17.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.993+0000 7f36127fc700 1 --2- 172.21.15.16:0/3725231601 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3614102290 0x7f36141a06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:17.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.993+0000 7f36127fc700 1 -- 172.21.15.16:0/3725231601 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3614108270 msgr2=0x7f361419a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:17.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.993+0000 7f36127fc700 1 --2- 172.21.15.16:0/3725231601 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3614108270 0x7f361419a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:17.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.993+0000 7f36127fc700 1 -- 172.21.15.16:0/3725231601 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f361419b010 con 0x7f3614102c50 2024-03-28T22:10:17.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.993+0000 7f3612ffd700 1 --2- 172.21.15.16:0/3725231601 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3614102290 0x7f36141a06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:10:17.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.993+0000 7f36127fc700 1 --2- 172.21.15.16:0/3725231601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3614102c50 0x7f36141a0c30 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f360800d950 tx=0x7f360800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:17.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.993+0000 7f35fbfff700 1 -- 172.21.15.16:0/3725231601 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3608016070 con 0x7f3614102c50 2024-03-28T22:10:17.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.993+0000 7f36192dc700 1 -- 172.21.15.16:0/3725231601 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f361419b300 con 0x7f3614102c50 2024-03-28T22:10:17.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.993+0000 7f35fbfff700 1 -- 172.21.15.16:0/3725231601 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3608009e20 con 0x7f3614102c50 2024-03-28T22:10:17.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.993+0000 7f35fbfff700 1 -- 172.21.15.16:0/3725231601 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360800f620 con 0x7f3614102c50 2024-03-28T22:10:17.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.993+0000 7f36192dc700 1 -- 172.21.15.16:0/3725231601 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36141a75c0 con 0x7f3614102c50 2024-03-28T22:10:17.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.994+0000 7f36192dc700 1 -- 172.21.15.16:0/3725231601 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3614068f00 con 0x7f3614102c50 2024-03-28T22:10:17.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.996+0000 7f35fbfff700 1 -- 172.21.15.16:0/3725231601 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f3608019070 con 0x7f3614102c50 2024-03-28T22:10:17.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.997+0000 7f35fbfff700 1 --2- 172.21.15.16:0/3725231601 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f35fc0a1e10 0x7f35fc0a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:17.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.997+0000 7f35fbfff700 1 -- 172.21.15.16:0/3725231601 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f36080cd950 con 0x7f3614102c50 2024-03-28T22:10:17.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.997+0000 7f3612ffd700 1 --2- 172.21.15.16:0/3725231601 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f35fc0a1e10 0x7f35fc0a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:17.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:16.997+0000 7f3612ffd700 1 --2- 172.21.15.16:0/3725231601 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f35fc0a1e10 0x7f35fc0a42d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3604000c00 tx=0x7f3604005940 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:17.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:17.001+0000 7f35fbfff700 1 -- 172.21.15.16:0/3725231601 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f360809adb0 con 0x7f3614102c50 2024-03-28T22:10:17.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:17.252+0000 7f36192dc700 1 -- 172.21.15.16:0/3725231601 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f361419bde0 con 0x7f3614102c50 2024-03-28T22:10:17.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:17.253+0000 7f35fbfff700 1 -- 172.21.15.16:0/3725231601 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v20) ==== 53+0+73 (secure 0 0 0) 0x7f361419bde0 con 0x7f3614102c50 2024-03-28T22:10:17.255 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-28T22:10:17.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:17.255+0000 7f36192dc700 1 -- 172.21.15.16:0/3725231601 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f35fc0a1e10 msgr2=0x7f35fc0a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:17.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:17.255+0000 7f36192dc700 1 --2- 172.21.15.16:0/3725231601 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f35fc0a1e10 0x7f35fc0a42d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3604000c00 tx=0x7f3604005940 comp rx=0 tx=0).stop 2024-03-28T22:10:17.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:17.255+0000 7f36192dc700 1 -- 172.21.15.16:0/3725231601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3614102c50 msgr2=0x7f36141a0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:17.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:17.255+0000 7f36192dc700 1 --2- 172.21.15.16:0/3725231601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3614102c50 0x7f36141a0c30 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f360800d950 tx=0x7f360800dc60 comp rx=0 tx=0).stop 2024-03-28T22:10:17.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:17.255+0000 7f36192dc700 1 -- 172.21.15.16:0/3725231601 shutdown_connections 2024-03-28T22:10:17.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:17.255+0000 7f36192dc700 1 --2- 172.21.15.16:0/3725231601 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f35fc0a1e10 0x7f35fc0a42d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:17.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:17.255+0000 7f36192dc700 1 --2- 172.21.15.16:0/3725231601 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3614108270 0x7f361419a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:17.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:17.255+0000 7f36192dc700 1 --2- 172.21.15.16:0/3725231601 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3614102290 0x7f36141a06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:17.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:17.255+0000 7f36192dc700 1 --2- 172.21.15.16:0/3725231601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3614102c50 0x7f36141a0c30 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:17.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:17.255+0000 7f36192dc700 1 -- 172.21.15.16:0/3725231601 >> 172.21.15.16:0/3725231601 conn(0x7f3614079370 msgr2=0x7f3614074700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:17.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:17.255+0000 7f36192dc700 1 -- 172.21.15.16:0/3725231601 shutdown_connections 2024-03-28T22:10:17.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:17.255+0000 7f36192dc700 1 -- 172.21.15.16:0/3725231601 wait complete. 2024-03-28T22:10:17.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:17 smithi063 ceph-mon[36249]: pgmap v342: 97 pgs: 97 active+clean; 639 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T22:10:17.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:17 smithi016 ceph-mon[32374]: pgmap v342: 97 pgs: 97 active+clean; 639 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T22:10:17.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:17 smithi053 ceph-mon[35979]: pgmap v342: 97 pgs: 97 active+clean; 639 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-28T22:10:17.861 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-28T22:10:17.861 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-28T22:10:17.861 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2024-03-28T22:10:17.861 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-28T22:10:17.861 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-28T22:10:17.862 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-28T22:10:17.862 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-28T22:10:17.862 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-28T22:10:17.862 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-28T22:10:17.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:10:17.862 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2024-03-28T22:10:17.879 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-28T22:10:17.879 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-28T22:10:17.879 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2024-03-28T22:10:17.879 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-28T22:10:17.879 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2024-03-28T22:10:17.879 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-28T22:10:17.880 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T22:10:17.880 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2024-03-28T22:10:17.880 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2024-03-28T22:10:17.880 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T22:10:17.880 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2024-03-28T22:10:17.880 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2024-03-28T22:10:17.880 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-28T22:10:17.880 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2024-03-28T22:10:17.880 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2024-03-28T22:10:17.880 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-28T22:10:17.880 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 42110sec preferred_lft 42110sec 2024-03-28T22:10:17.880 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link noprefixroute 2024-03-28T22:10:17.880 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-03-28T22:10:17.880 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T22:10:17.881 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2024-03-28T22:10:17.881 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2024-03-28T22:10:17.881 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-28T22:10:17.881 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T22:10:17.881 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-03-28T22:10:17.881 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add name ceph-brx type bridge 2024-03-28T22:10:17.881 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip addr flush dev ceph-brx 2024-03-28T22:10:17.881 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set ceph-brx up 2024-03-28T22:10:17.882 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-28T22:10:17.882 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-03-28T22:10:17.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T22:10:18.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T22:10:18.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:10:18.100 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-28T22:10:18.127 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-03-28T22:10:18.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:10:18.128 DEBUG:teuthology.orchestra.run.smithi016:> route 2024-03-28T22:10:18.203 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2024-03-28T22:10:18.203 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-28T22:10:18.203 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-28T22:10:18.203 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-28T22:10:18.203 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-28T22:10:18.204 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T22:10:18.204 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-03-28T22:10:18.204 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-28T22:10:18.204 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-28T22:10:18.204 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-28T22:10:18.204 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-03-28T22:10:18.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T22:10:18.396 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:18 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3725231601' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T22:10:18.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:18 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/3725231601' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T22:10:18.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T22:10:18.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:10:18.496 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-03-28T22:10:18.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:10:18.511 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list-id 2024-03-28T22:10:18.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:18 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3725231601' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T22:10:18.567 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T22:10:18.567 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-03-28T22:10:18.567 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T22:10:18.567 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-28T22:10:18.567 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-03-28T22:10:18.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T22:10:18.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T22:10:18.688 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-28T22:10:18.689 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T22:10:18.689 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-03-28T22:10:18.689 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-28T22:10:18.689 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-28T22:10:18.689 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-28T22:10:18.689 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-28T22:10:18.689 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-28T22:10:18.689 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-03-28T22:10:18.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T22:10:18.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T22:10:18.828 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T22:10:18.829 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-03-28T22:10:18.829 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2024-03-28T22:10:18.829 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set dev brx.0 master ceph-brx 2024-03-28T22:10:18.829 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-03-28T22:10:18.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T22:10:18.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T22:10:18.911 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T22:10:18.911 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-28T22:10:18.933 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-28T22:10:18.933 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T22:10:18.933 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-28T22:10:18.989 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-28T22:10:18.989 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-28T22:10:18.989 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/ea9205a6-f61d-41bc-b8e4-5f55e6a92aca 2024-03-28T22:10:18.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:10:18.990 DEBUG:teuthology.orchestra.run.smithi016:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/ea9205a6-f61d-41bc-b8e4-5f55e6a92aca /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync 2024-03-28T22:10:19.072 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync 2024-03-28T22:10:19.072 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync". 2024-03-28T22:10:19.072 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2024-03-28T22:10:19.196 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync 2024-03-28T22:10:19.196 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync". 2024-03-28T22:10:19.196 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2024-03-28T22:10:19.196 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:6789,172.21.15.53:6789,172.21.15.63:6789" 2024-03-28T22:10:19.196 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.16:6789,172.21.15.53:6789,172.21.15.63:6789:/volumes/qa/sv_1/ea9205a6-f61d-41bc-b8e4-5f55e6a92aca 2024-03-28T22:10:19.197 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync,key=0,fsid=798d9884-ed4e-11ee-95d0-87774f69a715" will pass to kernel 2024-03-28T22:10:19.199 INFO:teuthology.orchestra.run.smithi016.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-28T22:10:19.199 INFO:teuthology.orchestra.run.smithi016.stdout: You just mounted a file system that supports labels which does not 2024-03-28T22:10:19.199 INFO:teuthology.orchestra.run.smithi016.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-28T22:10:19.199 INFO:teuthology.orchestra.run.smithi016.stdout: applications will generate AVC messages and not be allowed access to 2024-03-28T22:10:19.199 INFO:teuthology.orchestra.run.smithi016.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-28T22:10:19.199 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-28T22:10:19.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:10:19.200 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-28T22:10:19.274 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-28T22:10:19.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-28T22:10:19.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T22:10:19.371 INFO:teuthology.orchestra.run.smithi016.stdout:client24988 2024-03-28T22:10:19.371 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-28T22:10:19.371 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T22:10:19.388 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-28T22:10:19.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-28T22:10:19.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T22:10:19.485 INFO:teuthology.orchestra.run.smithi016.stdout:client24988 2024-03-28T22:10:19.485 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-28T22:10:19.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-28T22:10:19.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T22:10:19.542 INFO:teuthology.orchestra.run.smithi016.stdout:client24988 2024-03-28T22:10:19.542 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/798d9884-ed4e-11ee-95d0-87774f69a715.client24988/status') 2024-03-28T22:10:19.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T22:10:19.603 INFO:teuthology.orchestra.run.smithi016.stdout:instance: client.24988 (3)172.21.15.16:0/965615515 2024-03-28T22:10:19.603 INFO:teuthology.orchestra.run.smithi016.stdout:blocklisted: false 2024-03-28T22:10:19.603 INFO:teuthology.orchestra.run.smithi016.stderr:0+1 records in 2024-03-28T22:10:19.604 INFO:teuthology.orchestra.run.smithi016.stderr:0+1 records out 2024-03-28T22:10:19.604 INFO:teuthology.orchestra.run.smithi016.stderr:70 bytes copied, 1.9629e-05 s, 3.6 MB/s 2024-03-28T22:10:19.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T22:10:19.610 INFO:teuthology.run_tasks:Running task exec... 2024-03-28T22:10:19.621 INFO:teuthology.task.exec:Executing custom commands... 2024-03-28T22:10:19.621 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-03-28T22:10:19.621 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-03-28T22:10:19.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.723+0000 7f0fc909a640 1 -- 172.21.15.16:0/3680207620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fc4075710 msgr2=0x7f0fc4073b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:19.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.723+0000 7f0fc909a640 1 --2- 172.21.15.16:0/3680207620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fc4075710 0x7f0fc4073b60 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4009900 tx=0x7f0fb402bcd0 comp rx=0 tx=0).stop 2024-03-28T22:10:19.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.723+0000 7f0fc909a640 1 -- 172.21.15.16:0/3680207620 shutdown_connections 2024-03-28T22:10:19.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.723+0000 7f0fc909a640 1 --2- 172.21.15.16:0/3680207620 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc4109650 0x7f0fc410bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:19.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.723+0000 7f0fc909a640 1 --2- 172.21.15.16:0/3680207620 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0fc4074130 0x7f0fc4074590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:19.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.723+0000 7f0fc909a640 1 --2- 172.21.15.16:0/3680207620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fc4075710 0x7f0fc4073b60 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:19.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.723+0000 7f0fc909a640 1 -- 172.21.15.16:0/3680207620 >> 172.21.15.16:0/3680207620 conn(0x7f0fc40fcd10 msgr2=0x7f0fc40ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:19.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.723+0000 7f0fc909a640 1 -- 172.21.15.16:0/3680207620 shutdown_connections 2024-03-28T22:10:19.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.723+0000 7f0fc909a640 1 -- 172.21.15.16:0/3680207620 wait complete. 2024-03-28T22:10:19.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.724+0000 7f0fc909a640 1 Processor -- start 2024-03-28T22:10:19.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.724+0000 7f0fc909a640 1 -- start start 2024-03-28T22:10:19.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.725+0000 7f0fc909a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fc4074130 0x7f0fc4199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:19.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.725+0000 7f0fc909a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0fc4075710 0x7f0fc419a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:19.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.725+0000 7f0fc909a640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc4109650 0x7f0fc41a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:19.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.725+0000 7f0fc909a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc410f670 con 0x7f0fc4074130 2024-03-28T22:10:19.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.725+0000 7f0fc909a640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc410f4f0 con 0x7f0fc4109650 2024-03-28T22:10:19.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.725+0000 7f0fc909a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc410f7f0 con 0x7f0fc4075710 2024-03-28T22:10:19.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.725+0000 7f0fc2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fc4074130 0x7f0fc4199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:19.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.725+0000 7f0fc2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fc4074130 0x7f0fc4199c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43876/0 (socket says 172.21.15.16:43876) 2024-03-28T22:10:19.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.725+0000 7f0fc2d9d640 1 -- 172.21.15.16:0/3606796073 learned_addr learned my addr 172.21.15.16:0/3606796073 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:19.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.725+0000 7f0fc359e640 1 --2- 172.21.15.16:0/3606796073 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc4109650 0x7f0fc41a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:19.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.725+0000 7f0fc2d9d640 1 -- 172.21.15.16:0/3606796073 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0fc4075710 msgr2=0x7f0fc419a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:19.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.725+0000 7f0fc2d9d640 1 --2- 172.21.15.16:0/3606796073 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0fc4075710 0x7f0fc419a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:19.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.725+0000 7f0fc2d9d640 1 -- 172.21.15.16:0/3606796073 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc4109650 msgr2=0x7f0fc41a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:19.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.725+0000 7f0fc2d9d640 1 --2- 172.21.15.16:0/3606796073 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc4109650 0x7f0fc41a1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:19.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.725+0000 7f0fc2d9d640 1 -- 172.21.15.16:0/3606796073 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fc41a18d0 con 0x7f0fc4074130 2024-03-28T22:10:19.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.725+0000 7f0fc2d9d640 1 --2- 172.21.15.16:0/3606796073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fc4074130 0x7f0fc4199c90 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4005ab0 tx=0x7f0fb402c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:19.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.725+0000 7f0fc0d99640 1 -- 172.21.15.16:0/3606796073 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb4043020 con 0x7f0fc4074130 2024-03-28T22:10:19.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.725+0000 7f0fc909a640 1 -- 172.21.15.16:0/3606796073 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fc41a1b60 con 0x7f0fc4074130 2024-03-28T22:10:19.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.725+0000 7f0fc0d99640 1 -- 172.21.15.16:0/3606796073 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0fb40054e0 con 0x7f0fc4074130 2024-03-28T22:10:19.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.725+0000 7f0fc0d99640 1 -- 172.21.15.16:0/3606796073 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb4003ca0 con 0x7f0fc4074130 2024-03-28T22:10:19.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.725+0000 7f0fc909a640 1 -- 172.21.15.16:0/3606796073 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fc41a2100 con 0x7f0fc4074130 2024-03-28T22:10:19.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.726+0000 7f0fc909a640 1 -- 172.21.15.16:0/3606796073 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fc4108410 con 0x7f0fc4074130 2024-03-28T22:10:19.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.727+0000 7f0fc0d99640 1 -- 172.21.15.16:0/3606796073 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f0fb4046020 con 0x7f0fc4074130 2024-03-28T22:10:19.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.728+0000 7f0fc0d99640 1 --2- 172.21.15.16:0/3606796073 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0fb00a1d70 0x7f0fb00a4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:19.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.728+0000 7f0fbbfff640 1 --2- 172.21.15.16:0/3606796073 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0fb00a1d70 0x7f0fb00a4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:19.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.728+0000 7f0fc0d99640 1 -- 172.21.15.16:0/3606796073 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f0fb40eff00 con 0x7f0fc4074130 2024-03-28T22:10:19.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.729+0000 7f0fbbfff640 1 --2- 172.21.15.16:0/3606796073 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0fb00a1d70 0x7f0fb00a4230 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0fac005fd0 tx=0x7f0fac005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:19.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.732+0000 7f0fc0d99640 1 -- 172.21.15.16:0/3606796073 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0fb404d020 con 0x7f0fc4074130 2024-03-28T22:10:19.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:19.954+0000 7f0fc909a640 1 -- 172.21.15.16:0/3606796073 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7f0fc4062350 con 0x7f0fc4074130 2024-03-28T22:10:20.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.163+0000 7f0fc0d99640 1 -- 172.21.15.16:0/3606796073 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v21) ==== 118+0+0 (secure 0 0 0) 0x7f0fb40bd360 con 0x7f0fc4074130 2024-03-28T22:10:20.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.165+0000 7f0fc909a640 1 -- 172.21.15.16:0/3606796073 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0fb00a1d70 msgr2=0x7f0fb00a4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:20.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.165+0000 7f0fc909a640 1 --2- 172.21.15.16:0/3606796073 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0fb00a1d70 0x7f0fb00a4230 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0fac005fd0 tx=0x7f0fac005850 comp rx=0 tx=0).stop 2024-03-28T22:10:20.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.165+0000 7f0fc909a640 1 -- 172.21.15.16:0/3606796073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fc4074130 msgr2=0x7f0fc4199c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:20.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.165+0000 7f0fc909a640 1 --2- 172.21.15.16:0/3606796073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fc4074130 0x7f0fc4199c90 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4005ab0 tx=0x7f0fb402c760 comp rx=0 tx=0).stop 2024-03-28T22:10:20.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.165+0000 7f0fc909a640 1 -- 172.21.15.16:0/3606796073 shutdown_connections 2024-03-28T22:10:20.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.165+0000 7f0fc909a640 1 --2- 172.21.15.16:0/3606796073 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f0fb00a1d70 0x7f0fb00a4230 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:20.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.165+0000 7f0fc909a640 1 --2- 172.21.15.16:0/3606796073 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc4109650 0x7f0fc41a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:20.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.165+0000 7f0fc909a640 1 --2- 172.21.15.16:0/3606796073 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0fc4075710 0x7f0fc419a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:20.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.165+0000 7f0fc909a640 1 --2- 172.21.15.16:0/3606796073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fc4074130 0x7f0fc4199c90 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:20.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.165+0000 7f0fc909a640 1 -- 172.21.15.16:0/3606796073 >> 172.21.15.16:0/3606796073 conn(0x7f0fc40fcd10 msgr2=0x7f0fc41055e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:20.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.165+0000 7f0fc909a640 1 -- 172.21.15.16:0/3606796073 shutdown_connections 2024-03-28T22:10:20.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.165+0000 7f0fc909a640 1 -- 172.21.15.16:0/3606796073 wait complete. 2024-03-28T22:10:20.179 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-03-28T22:10:20.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.284+0000 7fe08b95e640 1 -- 172.21.15.16:0/1975427921 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe084101870 msgr2=0x7fe084101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:20.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.284+0000 7fe08b95e640 1 --2- 172.21.15.16:0/1975427921 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe084101870 0x7fe084101cf0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fe078009900 tx=0x7fe07802bc30 comp rx=0 tx=0).stop 2024-03-28T22:10:20.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.284+0000 7fe08b95e640 1 -- 172.21.15.16:0/1975427921 shutdown_connections 2024-03-28T22:10:20.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.284+0000 7fe08b95e640 1 --2- 172.21.15.16:0/1975427921 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe084102230 0x7fe084108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:20.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.284+0000 7fe08b95e640 1 --2- 172.21.15.16:0/1975427921 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe084101870 0x7fe084101cf0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:20.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.284+0000 7fe08b95e640 1 --2- 172.21.15.16:0/1975427921 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe084100680 0x7fe084100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:20.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.284+0000 7fe08b95e640 1 -- 172.21.15.16:0/1975427921 >> 172.21.15.16:0/1975427921 conn(0x7fe0840fbbf0 msgr2=0x7fe0840fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:20.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.284+0000 7fe08b95e640 1 -- 172.21.15.16:0/1975427921 shutdown_connections 2024-03-28T22:10:20.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.284+0000 7fe08b95e640 1 -- 172.21.15.16:0/1975427921 wait complete. 2024-03-28T22:10:20.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.285+0000 7fe08b95e640 1 Processor -- start 2024-03-28T22:10:20.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.285+0000 7fe08b95e640 1 -- start start 2024-03-28T22:10:20.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.285+0000 7fe08b95e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe084100680 0x7fe084104610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:20.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.285+0000 7fe08b95e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe084101870 0x7fe084102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:20.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.285+0000 7fe08b95e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe084102230 0x7fe084103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:20.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.285+0000 7fe08b95e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe08410c370 con 0x7fe084100680 2024-03-28T22:10:20.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.285+0000 7fe08b95e640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe08410c1f0 con 0x7fe084101870 2024-03-28T22:10:20.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.285+0000 7fe08b95e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe08410c4f0 con 0x7fe084102230 2024-03-28T22:10:20.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.285+0000 7fe0896fa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe084100680 0x7fe084104610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:20.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.285+0000 7fe0896fa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe084100680 0x7fe084104610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43890/0 (socket says 172.21.15.16:43890) 2024-03-28T22:10:20.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.285+0000 7fe0896fa640 1 -- 172.21.15.16:0/2451986389 learned_addr learned my addr 172.21.15.16:0/2451986389 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:20.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.285+0000 7fe088ef9640 1 --2- 172.21.15.16:0/2451986389 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe084101870 0x7fe084102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:20.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.285+0000 7fe0896fa640 1 -- 172.21.15.16:0/2451986389 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe084102230 msgr2=0x7fe084103190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:20.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.285+0000 7fe0896fa640 1 --2- 172.21.15.16:0/2451986389 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe084102230 0x7fe084103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:20.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.285+0000 7fe0896fa640 1 -- 172.21.15.16:0/2451986389 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe084101870 msgr2=0x7fe084102c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:20.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.285+0000 7fe0896fa640 1 --2- 172.21.15.16:0/2451986389 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe084101870 0x7fe084102c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:20.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.285+0000 7fe0896fa640 1 -- 172.21.15.16:0/2451986389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe084103a20 con 0x7fe084100680 2024-03-28T22:10:20.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.285+0000 7fe0896fa640 1 --2- 172.21.15.16:0/2451986389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe084100680 0x7fe084104610 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fe07c00b450 tx=0x7fe07c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:20.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.285+0000 7fe0827fc640 1 -- 172.21.15.16:0/2451986389 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe07c015020 con 0x7fe084100680 2024-03-28T22:10:20.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.286+0000 7fe08b95e640 1 -- 172.21.15.16:0/2451986389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0841a1ce0 con 0x7fe084100680 2024-03-28T22:10:20.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.286+0000 7fe0827fc640 1 -- 172.21.15.16:0/2451986389 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe07c011d70 con 0x7fe084100680 2024-03-28T22:10:20.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.286+0000 7fe0827fc640 1 -- 172.21.15.16:0/2451986389 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe07c00f340 con 0x7fe084100680 2024-03-28T22:10:20.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.286+0000 7fe08b95e640 1 -- 172.21.15.16:0/2451986389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0841a21b0 con 0x7fe084100680 2024-03-28T22:10:20.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.286+0000 7fe08b95e640 1 -- 172.21.15.16:0/2451986389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0841a2380 con 0x7fe084100680 2024-03-28T22:10:20.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.287+0000 7fe0827fc640 1 -- 172.21.15.16:0/2451986389 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7fe07c018020 con 0x7fe084100680 2024-03-28T22:10:20.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.288+0000 7fe0827fc640 1 --2- 172.21.15.16:0/2451986389 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fe06c0a1d70 0x7fe06c0a4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:20.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.288+0000 7fe0827fc640 1 -- 172.21.15.16:0/2451986389 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fe07c0cd8d0 con 0x7fe084100680 2024-03-28T22:10:20.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.288+0000 7fe088ef9640 1 --2- 172.21.15.16:0/2451986389 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fe06c0a1d70 0x7fe06c0a4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:20.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.289+0000 7fe088ef9640 1 --2- 172.21.15.16:0/2451986389 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fe06c0a1d70 0x7fe06c0a4230 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe07802c7b0 tx=0x7fe078005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:20.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.292+0000 7fe0827fc640 1 -- 172.21.15.16:0/2451986389 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe07c01f020 con 0x7fe084100680 2024-03-28T22:10:20.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:20 smithi063 ceph-mon[36249]: pgmap v343: 97 pgs: 97 active+clean; 639 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:10:20.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:20 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/3606796073' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-28T22:10:20.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:20 smithi016 ceph-mon[32374]: pgmap v343: 97 pgs: 97 active+clean; 639 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:10:20.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:20 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3606796073' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-28T22:10:20.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.504+0000 7fe08b95e640 1 -- 172.21.15.16:0/2451986389 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7fe0840fde30 con 0x7fe06c0a1d70 2024-03-28T22:10:20.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.524+0000 7fe0827fc640 1 -- 172.21.15.16:0/2451986389 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7fe0840fde30 con 0x7fe06c0a1d70 2024-03-28T22:10:20.526 INFO:teuthology.orchestra.run.smithi016.stdout:{} 2024-03-28T22:10:20.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.526+0000 7fe08b95e640 1 -- 172.21.15.16:0/2451986389 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fe06c0a1d70 msgr2=0x7fe06c0a4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:20.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.526+0000 7fe08b95e640 1 --2- 172.21.15.16:0/2451986389 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fe06c0a1d70 0x7fe06c0a4230 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe07802c7b0 tx=0x7fe078005fd0 comp rx=0 tx=0).stop 2024-03-28T22:10:20.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.526+0000 7fe08b95e640 1 -- 172.21.15.16:0/2451986389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe084100680 msgr2=0x7fe084104610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:20.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.526+0000 7fe08b95e640 1 --2- 172.21.15.16:0/2451986389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe084100680 0x7fe084104610 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fe07c00b450 tx=0x7fe07c00b910 comp rx=0 tx=0).stop 2024-03-28T22:10:20.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.526+0000 7fe08b95e640 1 -- 172.21.15.16:0/2451986389 shutdown_connections 2024-03-28T22:10:20.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.526+0000 7fe08b95e640 1 --2- 172.21.15.16:0/2451986389 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fe06c0a1d70 0x7fe06c0a4230 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:20.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.526+0000 7fe08b95e640 1 --2- 172.21.15.16:0/2451986389 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe084102230 0x7fe084103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:20.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.526+0000 7fe08b95e640 1 --2- 172.21.15.16:0/2451986389 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe084101870 0x7fe084102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:20.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.526+0000 7fe08b95e640 1 --2- 172.21.15.16:0/2451986389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe084100680 0x7fe084104610 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:20.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.526+0000 7fe08b95e640 1 -- 172.21.15.16:0/2451986389 >> 172.21.15.16:0/2451986389 conn(0x7fe0840fbbf0 msgr2=0x7fe0840fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:20.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.526+0000 7fe08b95e640 1 -- 172.21.15.16:0/2451986389 shutdown_connections 2024-03-28T22:10:20.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.526+0000 7fe08b95e640 1 -- 172.21.15.16:0/2451986389 wait complete. 2024-03-28T22:10:20.539 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-28T22:10:20.550 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-03-28T22:10:20.562 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:10:20.563 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:20 smithi053 ceph-mon[35979]: pgmap v343: 97 pgs: 97 active+clean; 639 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:10:20.563 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:20 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3606796073' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-28T22:10:20.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.664+0000 7f42072ae640 1 -- 172.21.15.16:0/1432403361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4200101870 msgr2=0x7f4200101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:20.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.664+0000 7f42072ae640 1 --2- 172.21.15.16:0/1432403361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4200101870 0x7f4200101cf0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f41f4009900 tx=0x7f41f402bc30 comp rx=0 tx=0).stop 2024-03-28T22:10:20.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.664+0000 7f42072ae640 1 -- 172.21.15.16:0/1432403361 shutdown_connections 2024-03-28T22:10:20.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.664+0000 7f42072ae640 1 --2- 172.21.15.16:0/1432403361 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4200102230 0x7f4200108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:20.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.664+0000 7f42072ae640 1 --2- 172.21.15.16:0/1432403361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4200101870 0x7f4200101cf0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:20.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.664+0000 7f42072ae640 1 --2- 172.21.15.16:0/1432403361 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4200100680 0x7f4200100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:20.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.664+0000 7f42072ae640 1 -- 172.21.15.16:0/1432403361 >> 172.21.15.16:0/1432403361 conn(0x7f42000fbbf0 msgr2=0x7f42000fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:20.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.665+0000 7f42072ae640 1 -- 172.21.15.16:0/1432403361 shutdown_connections 2024-03-28T22:10:20.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.665+0000 7f42072ae640 1 -- 172.21.15.16:0/1432403361 wait complete. 2024-03-28T22:10:20.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.665+0000 7f42072ae640 1 Processor -- start 2024-03-28T22:10:20.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.665+0000 7f42072ae640 1 -- start start 2024-03-28T22:10:20.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.665+0000 7f42072ae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4200100680 0x7f4200197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:20.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.665+0000 7f42072ae640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4200101870 0x7f4200197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:20.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.665+0000 7f42072ae640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4200102230 0x7f420019f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:20.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.665+0000 7f42072ae640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f420010c330 con 0x7f4200100680 2024-03-28T22:10:20.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.665+0000 7f42072ae640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f420010c1b0 con 0x7f4200101870 2024-03-28T22:10:20.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.665+0000 7f42072ae640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f420010c4b0 con 0x7f4200102230 2024-03-28T22:10:20.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.665+0000 7f420504a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4200100680 0x7f4200197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:20.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.665+0000 7f420504a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4200100680 0x7f4200197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44290/0 (socket says 172.21.15.16:44290) 2024-03-28T22:10:20.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.665+0000 7f420504a640 1 -- 172.21.15.16:0/3415450018 learned_addr learned my addr 172.21.15.16:0/3415450018 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:20.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.665+0000 7f4204849640 1 --2- 172.21.15.16:0/3415450018 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4200101870 0x7f4200197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:20.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.665+0000 7f420584b640 1 --2- 172.21.15.16:0/3415450018 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4200102230 0x7f420019f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:20.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.666+0000 7f420504a640 1 -- 172.21.15.16:0/3415450018 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4200102230 msgr2=0x7f420019f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:20.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.666+0000 7f420504a640 1 --2- 172.21.15.16:0/3415450018 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4200102230 0x7f420019f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:20.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.666+0000 7f420504a640 1 -- 172.21.15.16:0/3415450018 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4200101870 msgr2=0x7f4200197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:20.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.666+0000 7f420504a640 1 --2- 172.21.15.16:0/3415450018 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4200101870 0x7f4200197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:20.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.666+0000 7f420504a640 1 -- 172.21.15.16:0/3415450018 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f420019f680 con 0x7f4200100680 2024-03-28T22:10:20.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.666+0000 7f420504a640 1 --2- 172.21.15.16:0/3415450018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4200100680 0x7f4200197a60 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f41f800d770 tx=0x7f41f800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:20.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.666+0000 7f41fe7fc640 1 -- 172.21.15.16:0/3415450018 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f8017070 con 0x7f4200100680 2024-03-28T22:10:20.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.666+0000 7f41fe7fc640 1 -- 172.21.15.16:0/3415450018 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41f8009c30 con 0x7f4200100680 2024-03-28T22:10:20.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.666+0000 7f42072ae640 1 -- 172.21.15.16:0/3415450018 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f420019f970 con 0x7f4200100680 2024-03-28T22:10:20.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.666+0000 7f41fe7fc640 1 -- 172.21.15.16:0/3415450018 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f8003dd0 con 0x7f4200100680 2024-03-28T22:10:20.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.666+0000 7f42072ae640 1 -- 172.21.15.16:0/3415450018 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f420019fe90 con 0x7f4200100680 2024-03-28T22:10:20.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.667+0000 7f42072ae640 1 -- 172.21.15.16:0/3415450018 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42001a00c0 con 0x7f4200100680 2024-03-28T22:10:20.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.668+0000 7f41fe7fc640 1 -- 172.21.15.16:0/3415450018 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f41f8003fb0 con 0x7f4200100680 2024-03-28T22:10:20.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.669+0000 7f41fe7fc640 1 --2- 172.21.15.16:0/3415450018 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f41e80a1d80 0x7f41e80a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:20.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.669+0000 7f4204849640 1 --2- 172.21.15.16:0/3415450018 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f41e80a1d80 0x7f41e80a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:20.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.669+0000 7f41fe7fc640 1 -- 172.21.15.16:0/3415450018 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f41f80cd690 con 0x7f4200100680 2024-03-28T22:10:20.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.669+0000 7f4204849640 1 --2- 172.21.15.16:0/3415450018 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f41e80a1d80 0x7f41e80a4240 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f41f402c240 tx=0x7f41f402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:20.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.673+0000 7f41fe7fc640 1 -- 172.21.15.16:0/3415450018 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f41f8009de0 con 0x7f4200100680 2024-03-28T22:10:20.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.857+0000 7f42072ae640 1 -- 172.21.15.16:0/3415450018 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f42001a0450 con 0x7f4200100680 2024-03-28T22:10:20.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.859+0000 7f41fe7fc640 1 -- 172.21.15.16:0/3415450018 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22441 (secure 0 0 0) 0x7f41f809a440 con 0x7f4200100680 2024-03-28T22:10:20.877 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:10:20.877 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":88,"fsid":"798d9884-ed4e-11ee-95d0-87774f69a715","created":"2024-03-28T22:00:27.216099+0000","modified":"2024-03-28T22:09:30.047847+0000","last_up_change":"2024-03-28T22:07:40.251141+0000","last_in_change":"2024-03-28T22:07:19.456451+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:03:59.327832+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:08:28.123529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:08:28.919037+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"46e7c8f8-ecc8-43d6-ae53-0655728f3240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6803","nonce":1589692483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6805","nonce":1589692483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6809","nonce":1589692483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6807","nonce":1589692483}]},"public_addr":"172.21.15.16:6803/1589692483","cluster_addr":"172.21.15.16:6805/1589692483","heartbeat_back_addr":"172.21.15.16:6809/1589692483","heartbeat_front_addr":"172.21.15.16:6807/1589692483","state":["exists","up"]},{"osd":1,"uuid":"af6d9c22-2830-4ce2-a547-011df0c6cbad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6801","nonce":795531234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6803","nonce":795531234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6807","nonce":795531234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6805","nonce":795531234}]},"public_addr":"172.21.15.53:6801/795531234","cluster_addr":"172.21.15.53:6803/795531234","heartbeat_back_addr":"172.21.15.53:6807/795531234","heartbeat_front_addr":"172.21.15.53:6805/795531234","state":["exists","up"]},{"osd":2,"uuid":"b10adca9-110c-4f87-85e0-b7542709d31a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6801","nonce":2214131086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6803","nonce":2214131086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6807","nonce":2214131086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6805","nonce":2214131086}]},"public_addr":"172.21.15.63:6801/2214131086","cluster_addr":"172.21.15.63:6803/2214131086","heartbeat_back_addr":"172.21.15.63:6807/2214131086","heartbeat_front_addr":"172.21.15.63:6805/2214131086","state":["exists","up"]},{"osd":3,"uuid":"c83cf4bb-395a-4786-abf7-74ca93a33eb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6811","nonce":13176789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6813","nonce":13176789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6817","nonce":13176789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6815","nonce":13176789}]},"public_addr":"172.21.15.16:6811/13176789","cluster_addr":"172.21.15.16:6813/13176789","heartbeat_back_addr":"172.21.15.16:6817/13176789","heartbeat_front_addr":"172.21.15.16:6815/13176789","state":["exists","up"]},{"osd":4,"uuid":"941b3e54-47d6-4e85-8069-e730c3fa8eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6809","nonce":772161849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6811","nonce":772161849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6815","nonce":772161849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6813","nonce":772161849}]},"public_addr":"172.21.15.53:6809/772161849","cluster_addr":"172.21.15.53:6811/772161849","heartbeat_back_addr":"172.21.15.53:6815/772161849","heartbeat_front_addr":"172.21.15.53:6813/772161849","state":["exists","up"]},{"osd":5,"uuid":"51d89826-1d60-40fc-8fd8-09bfe1ab4125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6809","nonce":1602554852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6811","nonce":1602554852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6815","nonce":1602554852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6813","nonce":1602554852}]},"public_addr":"172.21.15.63:6809/1602554852","cluster_addr":"172.21.15.63:6811/1602554852","heartbeat_back_addr":"172.21.15.63:6815/1602554852","heartbeat_front_addr":"172.21.15.63:6813/1602554852","state":["exists","up"]},{"osd":6,"uuid":"978efe5d-2c55-4ff8-8a1e-9f891388945e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6819","nonce":4290459224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6821","nonce":4290459224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6825","nonce":4290459224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6823","nonce":4290459224}]},"public_addr":"172.21.15.16:6819/4290459224","cluster_addr":"172.21.15.16:6821/4290459224","heartbeat_back_addr":"172.21.15.16:6825/4290459224","heartbeat_front_addr":"172.21.15.16:6823/4290459224","state":["exists","up"]},{"osd":7,"uuid":"50d489cb-dbb8-48df-b868-b29edf12e04d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6817","nonce":1056467525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6819","nonce":1056467525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6823","nonce":1056467525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6821","nonce":1056467525}]},"public_addr":"172.21.15.53:6817/1056467525","cluster_addr":"172.21.15.53:6819/1056467525","heartbeat_back_addr":"172.21.15.53:6823/1056467525","heartbeat_front_addr":"172.21.15.53:6821/1056467525","state":["exists","up"]},{"osd":8,"uuid":"49b39414-cc9c-4057-ae3f-cc109cd979da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6817","nonce":2918516794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6819","nonce":2918516794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6823","nonce":2918516794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6821","nonce":2918516794}]},"public_addr":"172.21.15.63:6817/2918516794","cluster_addr":"172.21.15.63:6819/2918516794","heartbeat_back_addr":"172.21.15.63:6823/2918516794","heartbeat_front_addr":"172.21.15.63:6821/2918516794","state":["exists","up"]},{"osd":9,"uuid":"4c873e51-b783-4eb6-83a3-fee31722d60b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6827","nonce":1585624367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6829","nonce":1585624367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6833","nonce":1585624367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6831","nonce":1585624367}]},"public_addr":"172.21.15.16:6827/1585624367","cluster_addr":"172.21.15.16:6829/1585624367","heartbeat_back_addr":"172.21.15.16:6833/1585624367","heartbeat_front_addr":"172.21.15.16:6831/1585624367","state":["exists","up"]},{"osd":10,"uuid":"085a78bf-e2ad-481a-85a3-71d1fc549a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6825","nonce":3357446484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6827","nonce":3357446484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6831","nonce":3357446484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6829","nonce":3357446484}]},"public_addr":"172.21.15.53:6825/3357446484","cluster_addr":"172.21.15.53:6827/3357446484","heartbeat_back_addr":"172.21.15.53:6831/3357446484","heartbeat_front_addr":"172.21.15.53:6829/3357446484","state":["exists","up"]},{"osd":11,"uuid":"13302aa3-08b9-4639-9042-27d850b4d5f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6825","nonce":2167866570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6827","nonce":2167866570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6831","nonce":2167866570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6829","nonce":2167866570}]},"public_addr":"172.21.15.63:6825/2167866570","cluster_addr":"172.21.15.63:6827/2167866570","heartbeat_back_addr":"172.21.15.63:6831/2167866570","heartbeat_front_addr":"172.21.15.63:6829/2167866570","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:07.450323+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:31.363172+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:55.295021+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:19.794760+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:44.063492+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:08.195064+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:33.655328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:57.639414+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:22.402394+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:47.462050+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:12.226307+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:37.495625+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3821408150":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6801/1832422779":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/996598130":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/1062264551":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/317557779":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6801/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/3631746518":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6800/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/3116652786":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6800/1832422779":"2024-03-29T22:01:28.993434+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:10:20.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.861+0000 7f42072ae640 1 -- 172.21.15.16:0/3415450018 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f41e80a1d80 msgr2=0x7f41e80a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:20.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.861+0000 7f42072ae640 1 --2- 172.21.15.16:0/3415450018 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f41e80a1d80 0x7f41e80a4240 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f41f402c240 tx=0x7f41f402df90 comp rx=0 tx=0).stop 2024-03-28T22:10:20.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.861+0000 7f42072ae640 1 -- 172.21.15.16:0/3415450018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4200100680 msgr2=0x7f4200197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:20.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.861+0000 7f42072ae640 1 --2- 172.21.15.16:0/3415450018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4200100680 0x7f4200197a60 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f41f800d770 tx=0x7f41f800dc30 comp rx=0 tx=0).stop 2024-03-28T22:10:20.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.861+0000 7f42072ae640 1 -- 172.21.15.16:0/3415450018 shutdown_connections 2024-03-28T22:10:20.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.861+0000 7f42072ae640 1 --2- 172.21.15.16:0/3415450018 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f41e80a1d80 0x7f41e80a4240 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:20.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.861+0000 7f42072ae640 1 --2- 172.21.15.16:0/3415450018 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4200102230 0x7f420019f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:20.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.861+0000 7f42072ae640 1 --2- 172.21.15.16:0/3415450018 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4200101870 0x7f4200197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:20.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.861+0000 7f42072ae640 1 --2- 172.21.15.16:0/3415450018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4200100680 0x7f4200197a60 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:20.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.861+0000 7f42072ae640 1 -- 172.21.15.16:0/3415450018 >> 172.21.15.16:0/3415450018 conn(0x7f42000fbbf0 msgr2=0x7f42000fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:20.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.861+0000 7f42072ae640 1 -- 172.21.15.16:0/3415450018 shutdown_connections 2024-03-28T22:10:20.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.861+0000 7f42072ae640 1 -- 172.21.15.16:0/3415450018 wait complete. 2024-03-28T22:10:20.880 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:20 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:20.509+0000 7ff185e6f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:10:20.880 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:20 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:20.509+0000 7ff185e6f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:10:20.880 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:20 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:20.509+0000 7ff185e6f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:10:20.880 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:20 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:20.509+0000 7ff185e6f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:10:20.880 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:20 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:20.509+0000 7ff185e6f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:10:20.881 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T22:03:59.327832+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T22:08:28.123529+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T22:08:28.919037+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T22:10:20.882 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T22:10:20.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.988+0000 7fc62c4ed640 1 -- 172.21.15.16:0/1219305348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6241033a0 msgr2=0x7fc624109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:20.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.988+0000 7fc62c4ed640 1 --2- 172.21.15.16:0/1219305348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6241033a0 0x7fc624109c30 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fc61c009cd0 tx=0x7fc61c02c0f0 comp rx=0 tx=0).stop 2024-03-28T22:10:20.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.988+0000 7fc62c4ed640 1 -- 172.21.15.16:0/1219305348 shutdown_connections 2024-03-28T22:10:20.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.988+0000 7fc62c4ed640 1 --2- 172.21.15.16:0/1219305348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6241033a0 0x7fc624109c30 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:20.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.988+0000 7fc62c4ed640 1 --2- 172.21.15.16:0/1219305348 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6241029e0 0x7fc624102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:20.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.988+0000 7fc62c4ed640 1 --2- 172.21.15.16:0/1219305348 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6240ffa10 0x7fc6240ffe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:20.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.988+0000 7fc62c4ed640 1 -- 172.21.15.16:0/1219305348 >> 172.21.15.16:0/1219305348 conn(0x7fc6240fb300 msgr2=0x7fc6240fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:20.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.988+0000 7fc62c4ed640 1 -- 172.21.15.16:0/1219305348 shutdown_connections 2024-03-28T22:10:20.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.988+0000 7fc62c4ed640 1 -- 172.21.15.16:0/1219305348 wait complete. 2024-03-28T22:10:20.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.989+0000 7fc62c4ed640 1 Processor -- start 2024-03-28T22:10:20.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.989+0000 7fc62c4ed640 1 -- start start 2024-03-28T22:10:20.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.989+0000 7fc62c4ed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6240ffa10 0x7fc624197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:20.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.989+0000 7fc62c4ed640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6241029e0 0x7fc624197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:20.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.989+0000 7fc62c4ed640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6241033a0 0x7fc62419ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:20.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.989+0000 7fc62c4ed640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc62410d3c0 con 0x7fc6240ffa10 2024-03-28T22:10:20.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.989+0000 7fc62c4ed640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc62410d240 con 0x7fc6241033a0 2024-03-28T22:10:20.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.989+0000 7fc62c4ed640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc62410d540 con 0x7fc6241029e0 2024-03-28T22:10:20.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.989+0000 7fc62a289640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6240ffa10 0x7fc624197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:20.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.989+0000 7fc62a289640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6240ffa10 0x7fc624197950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44312/0 (socket says 172.21.15.16:44312) 2024-03-28T22:10:20.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.989+0000 7fc62a289640 1 -- 172.21.15.16:0/86822463 learned_addr learned my addr 172.21.15.16:0/86822463 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:20.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.989+0000 7fc62aa8a640 1 --2- 172.21.15.16:0/86822463 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6241033a0 0x7fc62419ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:20.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.989+0000 7fc629a88640 1 --2- 172.21.15.16:0/86822463 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6241029e0 0x7fc624197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:20.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.990+0000 7fc62a289640 1 -- 172.21.15.16:0/86822463 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6241029e0 msgr2=0x7fc624197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:20.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.990+0000 7fc62a289640 1 --2- 172.21.15.16:0/86822463 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6241029e0 0x7fc624197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:20.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.990+0000 7fc62a289640 1 -- 172.21.15.16:0/86822463 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6241033a0 msgr2=0x7fc62419ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:20.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.990+0000 7fc62a289640 1 --2- 172.21.15.16:0/86822463 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6241033a0 0x7fc62419ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:20.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.990+0000 7fc62a289640 1 -- 172.21.15.16:0/86822463 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc62419f570 con 0x7fc6240ffa10 2024-03-28T22:10:20.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.990+0000 7fc629a88640 1 --2- 172.21.15.16:0/86822463 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6241029e0 0x7fc624197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:10:20.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.990+0000 7fc62a289640 1 --2- 172.21.15.16:0/86822463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6240ffa10 0x7fc624197950 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fc62000e9c0 tx=0x7fc62000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:20.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.990+0000 7fc61b7fe640 1 -- 172.21.15.16:0/86822463 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62000cd80 con 0x7fc6240ffa10 2024-03-28T22:10:20.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.990+0000 7fc61b7fe640 1 -- 172.21.15.16:0/86822463 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6200035f0 con 0x7fc6240ffa10 2024-03-28T22:10:20.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.990+0000 7fc62c4ed640 1 -- 172.21.15.16:0/86822463 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc62419f860 con 0x7fc6240ffa10 2024-03-28T22:10:20.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.990+0000 7fc61b7fe640 1 -- 172.21.15.16:0/86822463 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc620010690 con 0x7fc6240ffa10 2024-03-28T22:10:20.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.990+0000 7fc62c4ed640 1 -- 172.21.15.16:0/86822463 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc62419fe00 con 0x7fc6240ffa10 2024-03-28T22:10:20.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.991+0000 7fc62c4ed640 1 -- 172.21.15.16:0/86822463 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc624068a20 con 0x7fc6240ffa10 2024-03-28T22:10:20.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.993+0000 7fc61b7fe640 1 -- 172.21.15.16:0/86822463 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7fc6200037a0 con 0x7fc6240ffa10 2024-03-28T22:10:20.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.993+0000 7fc61b7fe640 1 --2- 172.21.15.16:0/86822463 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc6100a1e00 0x7fc6100a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:20.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.993+0000 7fc629a88640 1 --2- 172.21.15.16:0/86822463 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc6100a1e00 0x7fc6100a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:20.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.993+0000 7fc61b7fe640 1 -- 172.21.15.16:0/86822463 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fc6200ce200 con 0x7fc6240ffa10 2024-03-28T22:10:20.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.994+0000 7fc629a88640 1 --2- 172.21.15.16:0/86822463 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc6100a1e00 0x7fc6100a42c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc624198e60 tx=0x7fc61400a5c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:20.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:20.997+0000 7fc61b7fe640 1 -- 172.21.15.16:0/86822463 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc62009b6f0 con 0x7fc6240ffa10 2024-03-28T22:10:21.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.179+0000 7fc61b7fe640 1 -- 172.21.15.16:0/86822463 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7fc620097660 con 0x7fc6240ffa10 2024-03-28T22:10:21.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.184+0000 7fc62c4ed640 1 -- 172.21.15.16:0/86822463 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc62402d1f0 con 0x7fc6240ffa10 2024-03-28T22:10:21.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.184+0000 7fc61b7fe640 1 -- 172.21.15.16:0/86822463 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7fc6200c4b70 con 0x7fc6240ffa10 2024-03-28T22:10:21.186 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-28T22:10:21.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.186+0000 7fc62c4ed640 1 -- 172.21.15.16:0/86822463 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc6100a1e00 msgr2=0x7fc6100a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:21.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.186+0000 7fc62c4ed640 1 --2- 172.21.15.16:0/86822463 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc6100a1e00 0x7fc6100a42c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc624198e60 tx=0x7fc61400a5c0 comp rx=0 tx=0).stop 2024-03-28T22:10:21.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.186+0000 7fc62c4ed640 1 -- 172.21.15.16:0/86822463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6240ffa10 msgr2=0x7fc624197950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:21.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.186+0000 7fc62c4ed640 1 --2- 172.21.15.16:0/86822463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6240ffa10 0x7fc624197950 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fc62000e9c0 tx=0x7fc62000ee80 comp rx=0 tx=0).stop 2024-03-28T22:10:21.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.186+0000 7fc62c4ed640 1 -- 172.21.15.16:0/86822463 shutdown_connections 2024-03-28T22:10:21.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.186+0000 7fc62c4ed640 1 --2- 172.21.15.16:0/86822463 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc6100a1e00 0x7fc6100a42c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:21.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.186+0000 7fc62c4ed640 1 --2- 172.21.15.16:0/86822463 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6241033a0 0x7fc62419ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:21.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.186+0000 7fc62c4ed640 1 --2- 172.21.15.16:0/86822463 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6241029e0 0x7fc624197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:21.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.186+0000 7fc62c4ed640 1 --2- 172.21.15.16:0/86822463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6240ffa10 0x7fc624197950 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:21.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.186+0000 7fc62c4ed640 1 -- 172.21.15.16:0/86822463 >> 172.21.15.16:0/86822463 conn(0x7fc6240fb300 msgr2=0x7fc624105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:21.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.186+0000 7fc62c4ed640 1 -- 172.21.15.16:0/86822463 shutdown_connections 2024-03-28T22:10:21.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.186+0000 7fc62c4ed640 1 -- 172.21.15.16:0/86822463 wait complete. 2024-03-28T22:10:21.199 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T22:10:21.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.303+0000 7f664435e640 1 -- 172.21.15.16:0/207103284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f663c102950 msgr2=0x7f663c102dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:21.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.303+0000 7f664435e640 1 --2- 172.21.15.16:0/207103284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f663c102950 0x7f663c102dd0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f662c009900 tx=0x7f662c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:10:21.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.303+0000 7f664435e640 1 -- 172.21.15.16:0/207103284 shutdown_connections 2024-03-28T22:10:21.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.303+0000 7f664435e640 1 --2- 172.21.15.16:0/207103284 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f663c103310 0x7f663c109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:21.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.303+0000 7f664435e640 1 --2- 172.21.15.16:0/207103284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f663c102950 0x7f663c102dd0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:21.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.303+0000 7f664435e640 1 --2- 172.21.15.16:0/207103284 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f663c101760 0x7f663c101b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:21.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.303+0000 7f664435e640 1 -- 172.21.15.16:0/207103284 >> 172.21.15.16:0/207103284 conn(0x7f663c0fccd0 msgr2=0x7f663c0ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:21.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.303+0000 7f664435e640 1 -- 172.21.15.16:0/207103284 shutdown_connections 2024-03-28T22:10:21.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.303+0000 7f664435e640 1 -- 172.21.15.16:0/207103284 wait complete. 2024-03-28T22:10:21.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.304+0000 7f664435e640 1 Processor -- start 2024-03-28T22:10:21.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.304+0000 7f664435e640 1 -- start start 2024-03-28T22:10:21.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.304+0000 7f664435e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f663c101760 0x7f663c197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:21.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.304+0000 7f664435e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f663c102950 0x7f663c197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:21.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.304+0000 7f664435e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f663c103310 0x7f663c19f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:21.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.304+0000 7f664435e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f663c10d430 con 0x7f663c103310 2024-03-28T22:10:21.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.304+0000 7f664435e640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f663c10d2b0 con 0x7f663c101760 2024-03-28T22:10:21.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.304+0000 7f664435e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f663c10d5b0 con 0x7f663c102950 2024-03-28T22:10:21.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.304+0000 7f66428fb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f663c103310 0x7f663c19f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:21.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.304+0000 7f66420fa640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f663c101760 0x7f663c197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:21.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.304+0000 7f66428fb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f663c103310 0x7f663c19f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44328/0 (socket says 172.21.15.16:44328) 2024-03-28T22:10:21.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.304+0000 7f66428fb640 1 -- 172.21.15.16:0/2267731908 learned_addr learned my addr 172.21.15.16:0/2267731908 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:21.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.304+0000 7f66418f9640 1 --2- 172.21.15.16:0/2267731908 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f663c102950 0x7f663c197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:21.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.304+0000 7f66428fb640 1 -- 172.21.15.16:0/2267731908 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f663c102950 msgr2=0x7f663c197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:21.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.304+0000 7f66428fb640 1 --2- 172.21.15.16:0/2267731908 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f663c102950 0x7f663c197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:21.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.305+0000 7f66428fb640 1 -- 172.21.15.16:0/2267731908 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f663c101760 msgr2=0x7f663c197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:21.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.305+0000 7f66428fb640 1 --2- 172.21.15.16:0/2267731908 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f663c101760 0x7f663c197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:21.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.305+0000 7f66428fb640 1 -- 172.21.15.16:0/2267731908 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f663c19f6c0 con 0x7f663c103310 2024-03-28T22:10:21.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.305+0000 7f66418f9640 1 --2- 172.21.15.16:0/2267731908 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f663c102950 0x7f663c197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:10:21.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.305+0000 7f66428fb640 1 --2- 172.21.15.16:0/2267731908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f663c103310 0x7f663c19f050 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f663400eaa0 tx=0x7f663400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:21.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.305+0000 7f66337fe640 1 -- 172.21.15.16:0/2267731908 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f663400cdb0 con 0x7f663c103310 2024-03-28T22:10:21.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.305+0000 7f66337fe640 1 -- 172.21.15.16:0/2267731908 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6634005020 con 0x7f663c103310 2024-03-28T22:10:21.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.305+0000 7f664435e640 1 -- 172.21.15.16:0/2267731908 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f663c19f9b0 con 0x7f663c103310 2024-03-28T22:10:21.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.305+0000 7f66337fe640 1 -- 172.21.15.16:0/2267731908 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66340116b0 con 0x7f663c103310 2024-03-28T22:10:21.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.305+0000 7f664435e640 1 -- 172.21.15.16:0/2267731908 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f663c19ff20 con 0x7f663c103310 2024-03-28T22:10:21.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.306+0000 7f664435e640 1 -- 172.21.15.16:0/2267731908 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f663c108a80 con 0x7f663c103310 2024-03-28T22:10:21.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.307+0000 7f66337fe640 1 -- 172.21.15.16:0/2267731908 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7f6634005560 con 0x7f663c103310 2024-03-28T22:10:21.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.307+0000 7f66337fe640 1 --2- 172.21.15.16:0/2267731908 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f66280a1e90 0x7f66280a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:21.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.307+0000 7f66420fa640 1 --2- 172.21.15.16:0/2267731908 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f66280a1e90 0x7f66280a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:21.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.308+0000 7f66337fe640 1 -- 172.21.15.16:0/2267731908 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f66340ce100 con 0x7f663c103310 2024-03-28T22:10:21.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.308+0000 7f66420fa640 1 --2- 172.21.15.16:0/2267731908 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f66280a1e90 0x7f66280a4350 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f663c1027b0 tx=0x7f6638005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:21.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.312+0000 7f66337fe640 1 -- 172.21.15.16:0/2267731908 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6634016030 con 0x7f663c103310 2024-03-28T22:10:21.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:21 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/3606796073' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-28T22:10:21.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:21 smithi063 ceph-mon[36249]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T22:10:21.434 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:21 smithi063 ceph-mon[36249]: from='client.15267 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:10:21.434 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:21 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/3415450018' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:10:21.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:21 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3606796073' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-28T22:10:21.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:21 smithi016 ceph-mon[32374]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T22:10:21.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:21 smithi016 ceph-mon[32374]: from='client.15267 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:10:21.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:21 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3415450018' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:10:21.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.497+0000 7f664435e640 1 -- 172.21.15.16:0/2267731908 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f663c062350 con 0x7f663c103310 2024-03-28T22:10:21.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.497+0000 7f66337fe640 1 -- 172.21.15.16:0/2267731908 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f663409b3a0 con 0x7f663c103310 2024-03-28T22:10:21.500 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-28T22:10:21.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.499+0000 7f664435e640 1 -- 172.21.15.16:0/2267731908 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f66280a1e90 msgr2=0x7f66280a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:21.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.499+0000 7f664435e640 1 --2- 172.21.15.16:0/2267731908 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f66280a1e90 0x7f66280a4350 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f663c1027b0 tx=0x7f6638005f90 comp rx=0 tx=0).stop 2024-03-28T22:10:21.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.499+0000 7f664435e640 1 -- 172.21.15.16:0/2267731908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f663c103310 msgr2=0x7f663c19f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:21.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.499+0000 7f664435e640 1 --2- 172.21.15.16:0/2267731908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f663c103310 0x7f663c19f050 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f663400eaa0 tx=0x7f663400ef60 comp rx=0 tx=0).stop 2024-03-28T22:10:21.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.499+0000 7f664435e640 1 -- 172.21.15.16:0/2267731908 shutdown_connections 2024-03-28T22:10:21.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.499+0000 7f664435e640 1 --2- 172.21.15.16:0/2267731908 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f66280a1e90 0x7f66280a4350 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:21.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.499+0000 7f664435e640 1 --2- 172.21.15.16:0/2267731908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f663c103310 0x7f663c19f050 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:21.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.499+0000 7f664435e640 1 --2- 172.21.15.16:0/2267731908 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f663c102950 0x7f663c197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:21.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.499+0000 7f664435e640 1 --2- 172.21.15.16:0/2267731908 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f663c101760 0x7f663c197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:21.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.499+0000 7f664435e640 1 -- 172.21.15.16:0/2267731908 >> 172.21.15.16:0/2267731908 conn(0x7f663c0fccd0 msgr2=0x7f663c105c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:21.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.500+0000 7f664435e640 1 -- 172.21.15.16:0/2267731908 shutdown_connections 2024-03-28T22:10:21.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.500+0000 7f664435e640 1 -- 172.21.15.16:0/2267731908 wait complete. 2024-03-28T22:10:21.514 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T22:10:21.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:21 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3606796073' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-28T22:10:21.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:21 smithi053 ceph-mon[35979]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T22:10:21.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:21 smithi053 ceph-mon[35979]: from='client.15267 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:10:21.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:21 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3415450018' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:10:21.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.618+0000 7f876ff8c640 1 -- 172.21.15.16:0/2116459669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8768101750 msgr2=0x7f8768101b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:21.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.618+0000 7f876ff8c640 1 --2- 172.21.15.16:0/2116459669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8768101750 0x7f8768101b70 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f8764009900 tx=0x7f876402bc30 comp rx=0 tx=0).stop 2024-03-28T22:10:21.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.618+0000 7f876ff8c640 1 -- 172.21.15.16:0/2116459669 shutdown_connections 2024-03-28T22:10:21.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.618+0000 7f876ff8c640 1 --2- 172.21.15.16:0/2116459669 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8768103300 0x7f8768109b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:21.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.618+0000 7f876ff8c640 1 --2- 172.21.15.16:0/2116459669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8768102940 0x7f8768102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:21.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.618+0000 7f876ff8c640 1 --2- 172.21.15.16:0/2116459669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8768101750 0x7f8768101b70 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:21.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.618+0000 7f876ff8c640 1 -- 172.21.15.16:0/2116459669 >> 172.21.15.16:0/2116459669 conn(0x7f87680fcc80 msgr2=0x7f87680ff100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:21.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.618+0000 7f876ff8c640 1 -- 172.21.15.16:0/2116459669 shutdown_connections 2024-03-28T22:10:21.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.618+0000 7f876ff8c640 1 -- 172.21.15.16:0/2116459669 wait complete. 2024-03-28T22:10:21.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.619+0000 7f876ff8c640 1 Processor -- start 2024-03-28T22:10:21.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.619+0000 7f876ff8c640 1 -- start start 2024-03-28T22:10:21.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.619+0000 7f876ff8c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8768101750 0x7f8768197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:21.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.619+0000 7f876ff8c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8768102940 0x7f8768197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:21.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.619+0000 7f876ff8c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8768103300 0x7f876819f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:21.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.619+0000 7f876ff8c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f876810d470 con 0x7f8768103300 2024-03-28T22:10:21.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.619+0000 7f876ff8c640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f876810d2f0 con 0x7f8768101750 2024-03-28T22:10:21.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.619+0000 7f876ff8c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f876810d5f0 con 0x7f8768102940 2024-03-28T22:10:21.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.619+0000 7f876e529640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8768103300 0x7f876819f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:21.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.619+0000 7f876e529640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8768103300 0x7f876819f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44354/0 (socket says 172.21.15.16:44354) 2024-03-28T22:10:21.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.619+0000 7f876e529640 1 -- 172.21.15.16:0/1805437640 learned_addr learned my addr 172.21.15.16:0/1805437640 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:21.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.619+0000 7f876dd28640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8768101750 0x7f8768197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:21.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.619+0000 7f876d527640 1 --2- 172.21.15.16:0/1805437640 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8768102940 0x7f8768197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:21.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.619+0000 7f876e529640 1 -- 172.21.15.16:0/1805437640 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8768102940 msgr2=0x7f8768197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:21.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.619+0000 7f876e529640 1 --2- 172.21.15.16:0/1805437640 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8768102940 0x7f8768197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:21.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.619+0000 7f876e529640 1 -- 172.21.15.16:0/1805437640 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8768101750 msgr2=0x7f8768197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:21.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.619+0000 7f876e529640 1 --2- 172.21.15.16:0/1805437640 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8768101750 0x7f8768197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:21.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.619+0000 7f876e529640 1 -- 172.21.15.16:0/1805437640 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f876819f6d0 con 0x7f8768103300 2024-03-28T22:10:21.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.619+0000 7f876e529640 1 --2- 172.21.15.16:0/1805437640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8768103300 0x7f876819f060 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f876000b970 tx=0x7f876000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:21.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.619+0000 7f875effd640 1 -- 172.21.15.16:0/1805437640 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8760019070 con 0x7f8768103300 2024-03-28T22:10:21.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.620+0000 7f875effd640 1 -- 172.21.15.16:0/1805437640 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f876000cb40 con 0x7f8768103300 2024-03-28T22:10:21.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.620+0000 7f876ff8c640 1 -- 172.21.15.16:0/1805437640 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f876819f9c0 con 0x7f8768103300 2024-03-28T22:10:21.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.620+0000 7f875effd640 1 -- 172.21.15.16:0/1805437640 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8760007650 con 0x7f8768103300 2024-03-28T22:10:21.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.620+0000 7f876ff8c640 1 -- 172.21.15.16:0/1805437640 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f876819ff10 con 0x7f8768103300 2024-03-28T22:10:21.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.621+0000 7f876ff8c640 1 -- 172.21.15.16:0/1805437640 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8768108b40 con 0x7f8768103300 2024-03-28T22:10:21.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.622+0000 7f875effd640 1 -- 172.21.15.16:0/1805437640 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7f876001c070 con 0x7f8768103300 2024-03-28T22:10:21.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.622+0000 7f875effd640 1 --2- 172.21.15.16:0/1805437640 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f87540a1e90 0x7f87540a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:21.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.623+0000 7f876dd28640 1 --2- 172.21.15.16:0/1805437640 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f87540a1e90 0x7f87540a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:21.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.623+0000 7f875effd640 1 -- 172.21.15.16:0/1805437640 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f87600cd360 con 0x7f8768103300 2024-03-28T22:10:21.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.623+0000 7f876dd28640 1 --2- 172.21.15.16:0/1805437640 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f87540a1e90 0x7f87540a4350 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f876402c170 tx=0x7f876402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:21.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.627+0000 7f875effd640 1 -- 172.21.15.16:0/1805437640 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f876009a120 con 0x7f8768103300 2024-03-28T22:10:21.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.809+0000 7f876ff8c640 1 -- 172.21.15.16:0/1805437640 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f876802d380 con 0x7f8768103300 2024-03-28T22:10:21.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.809+0000 7f875effd640 1 -- 172.21.15.16:0/1805437640 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f876009e1c0 con 0x7f8768103300 2024-03-28T22:10:21.812 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-28T22:10:21.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.811+0000 7f876ff8c640 1 -- 172.21.15.16:0/1805437640 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f87540a1e90 msgr2=0x7f87540a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:21.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.811+0000 7f876ff8c640 1 --2- 172.21.15.16:0/1805437640 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f87540a1e90 0x7f87540a4350 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f876402c170 tx=0x7f876402df90 comp rx=0 tx=0).stop 2024-03-28T22:10:21.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.812+0000 7f876ff8c640 1 -- 172.21.15.16:0/1805437640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8768103300 msgr2=0x7f876819f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:21.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.812+0000 7f876ff8c640 1 --2- 172.21.15.16:0/1805437640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8768103300 0x7f876819f060 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f876000b970 tx=0x7f876000be30 comp rx=0 tx=0).stop 2024-03-28T22:10:21.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.812+0000 7f876ff8c640 1 -- 172.21.15.16:0/1805437640 shutdown_connections 2024-03-28T22:10:21.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.812+0000 7f876ff8c640 1 --2- 172.21.15.16:0/1805437640 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f87540a1e90 0x7f87540a4350 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:21.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.812+0000 7f876ff8c640 1 --2- 172.21.15.16:0/1805437640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8768103300 0x7f876819f060 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:21.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.812+0000 7f876ff8c640 1 --2- 172.21.15.16:0/1805437640 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8768102940 0x7f8768197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:21.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.812+0000 7f876ff8c640 1 --2- 172.21.15.16:0/1805437640 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8768101750 0x7f8768197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:21.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.812+0000 7f876ff8c640 1 -- 172.21.15.16:0/1805437640 >> 172.21.15.16:0/1805437640 conn(0x7f87680fcc80 msgr2=0x7f8768105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:21.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.812+0000 7f876ff8c640 1 -- 172.21.15.16:0/1805437640 shutdown_connections 2024-03-28T22:10:21.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.812+0000 7f876ff8c640 1 -- 172.21.15.16:0/1805437640 wait complete. 2024-03-28T22:10:21.826 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:10:21.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.926+0000 7f2ab1640640 1 -- 172.21.15.16:0/981439275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2aac107530 msgr2=0x7f2aac109920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:21.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.926+0000 7f2ab1640640 1 --2- 172.21.15.16:0/981439275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2aac107530 0x7f2aac109920 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4009ca0 tx=0x7f2aa402c0f0 comp rx=0 tx=0).stop 2024-03-28T22:10:21.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.926+0000 7f2ab1640640 1 -- 172.21.15.16:0/981439275 shutdown_connections 2024-03-28T22:10:21.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.926+0000 7f2ab1640640 1 --2- 172.21.15.16:0/981439275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2aac107530 0x7f2aac109920 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:21.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.926+0000 7f2ab1640640 1 --2- 172.21.15.16:0/981439275 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2aac104c00 0x7f2aac106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:21.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.926+0000 7f2ab1640640 1 --2- 172.21.15.16:0/981439275 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2aac1022d0 0x7f2aac1046c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:21.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.926+0000 7f2ab1640640 1 -- 172.21.15.16:0/981439275 >> 172.21.15.16:0/981439275 conn(0x7f2aac0fbc80 msgr2=0x7f2aac0fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:21.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.926+0000 7f2ab1640640 1 -- 172.21.15.16:0/981439275 shutdown_connections 2024-03-28T22:10:21.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.926+0000 7f2ab1640640 1 -- 172.21.15.16:0/981439275 wait complete. 2024-03-28T22:10:21.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.927+0000 7f2ab1640640 1 Processor -- start 2024-03-28T22:10:21.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.927+0000 7f2ab1640640 1 -- start start 2024-03-28T22:10:21.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.927+0000 7f2ab1640640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2aac1022d0 0x7f2aac197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:21.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.927+0000 7f2ab1640640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2aac104c00 0x7f2aac197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:21.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.927+0000 7f2ab1640640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2aac107530 0x7f2aac19efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:21.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.927+0000 7f2ab1640640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2aac10d400 con 0x7f2aac104c00 2024-03-28T22:10:21.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.927+0000 7f2ab1640640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f2aac10d280 con 0x7f2aac1022d0 2024-03-28T22:10:21.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.927+0000 7f2ab1640640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2aac10d580 con 0x7f2aac107530 2024-03-28T22:10:21.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.927+0000 7f2aaa7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2aac104c00 0x7f2aac197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:21.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.927+0000 7f2aaa7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2aac104c00 0x7f2aac197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44370/0 (socket says 172.21.15.16:44370) 2024-03-28T22:10:21.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.927+0000 7f2aaa7fc640 1 -- 172.21.15.16:0/1671527466 learned_addr learned my addr 172.21.15.16:0/1671527466 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:21.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.927+0000 7f2aaaffd640 1 --2- 172.21.15.16:0/1671527466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2aac1022d0 0x7f2aac197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:21.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.927+0000 7f2aaa7fc640 1 -- 172.21.15.16:0/1671527466 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2aac107530 msgr2=0x7f2aac19efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:21.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.927+0000 7f2aaa7fc640 1 --2- 172.21.15.16:0/1671527466 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2aac107530 0x7f2aac19efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:21.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.927+0000 7f2aaa7fc640 1 -- 172.21.15.16:0/1671527466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2aac1022d0 msgr2=0x7f2aac197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:21.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.927+0000 7f2aaa7fc640 1 --2- 172.21.15.16:0/1671527466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2aac1022d0 0x7f2aac197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:21.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.927+0000 7f2aaa7fc640 1 -- 172.21.15.16:0/1671527466 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2aac19f640 con 0x7f2aac104c00 2024-03-28T22:10:21.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.927+0000 7f2aaa7fc640 1 --2- 172.21.15.16:0/1671527466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2aac104c00 0x7f2aac197f60 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f2aa000c970 tx=0x7f2aa000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:21.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.928+0000 7f2a93fff640 1 -- 172.21.15.16:0/1671527466 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa0007bb0 con 0x7f2aac104c00 2024-03-28T22:10:21.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.928+0000 7f2a93fff640 1 -- 172.21.15.16:0/1671527466 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2aa0007d50 con 0x7f2aac104c00 2024-03-28T22:10:21.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.928+0000 7f2ab1640640 1 -- 172.21.15.16:0/1671527466 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2aac19f930 con 0x7f2aac104c00 2024-03-28T22:10:21.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.928+0000 7f2a93fff640 1 -- 172.21.15.16:0/1671527466 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa000f450 con 0x7f2aac104c00 2024-03-28T22:10:21.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.928+0000 7f2ab1640640 1 -- 172.21.15.16:0/1671527466 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2aac19fed0 con 0x7f2aac104c00 2024-03-28T22:10:21.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.929+0000 7f2ab1640640 1 -- 172.21.15.16:0/1671527466 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2aac0fd770 con 0x7f2aac104c00 2024-03-28T22:10:21.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.930+0000 7f2a93fff640 1 -- 172.21.15.16:0/1671527466 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7f2aa00046c0 con 0x7f2aac104c00 2024-03-28T22:10:21.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.930+0000 7f2a93fff640 1 --2- 172.21.15.16:0/1671527466 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f2a940a1e90 0x7f2a940a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:21.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.930+0000 7f2aaaffd640 1 --2- 172.21.15.16:0/1671527466 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f2a940a1e90 0x7f2a940a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:21.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.930+0000 7f2a93fff640 1 -- 172.21.15.16:0/1671527466 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f2aa00cdeb0 con 0x7f2aac104c00 2024-03-28T22:10:21.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.931+0000 7f2aaaffd640 1 --2- 172.21.15.16:0/1671527466 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f2a940a1e90 0x7f2a940a4350 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c0095f0 tx=0x7f2a9c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:21.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:21.934+0000 7f2a93fff640 1 -- 172.21.15.16:0/1671527466 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2aa0014030 con 0x7f2aac104c00 2024-03-28T22:10:22.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.119+0000 7f2ab1640640 1 -- 172.21.15.16:0/1671527466 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2aac02d170 con 0x7f2aac104c00 2024-03-28T22:10:22.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.120+0000 7f2a93fff640 1 -- 172.21.15.16:0/1671527466 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22441 (secure 0 0 0) 0x7f2aa009b230 con 0x7f2aac104c00 2024-03-28T22:10:22.123 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:10:22.123 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":88,"fsid":"798d9884-ed4e-11ee-95d0-87774f69a715","created":"2024-03-28T22:00:27.216099+0000","modified":"2024-03-28T22:09:30.047847+0000","last_up_change":"2024-03-28T22:07:40.251141+0000","last_in_change":"2024-03-28T22:07:19.456451+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:03:59.327832+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:08:28.123529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:08:28.919037+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"46e7c8f8-ecc8-43d6-ae53-0655728f3240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6803","nonce":1589692483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6805","nonce":1589692483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6809","nonce":1589692483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6807","nonce":1589692483}]},"public_addr":"172.21.15.16:6803/1589692483","cluster_addr":"172.21.15.16:6805/1589692483","heartbeat_back_addr":"172.21.15.16:6809/1589692483","heartbeat_front_addr":"172.21.15.16:6807/1589692483","state":["exists","up"]},{"osd":1,"uuid":"af6d9c22-2830-4ce2-a547-011df0c6cbad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6801","nonce":795531234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6803","nonce":795531234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6807","nonce":795531234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6805","nonce":795531234}]},"public_addr":"172.21.15.53:6801/795531234","cluster_addr":"172.21.15.53:6803/795531234","heartbeat_back_addr":"172.21.15.53:6807/795531234","heartbeat_front_addr":"172.21.15.53:6805/795531234","state":["exists","up"]},{"osd":2,"uuid":"b10adca9-110c-4f87-85e0-b7542709d31a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6801","nonce":2214131086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6803","nonce":2214131086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6807","nonce":2214131086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6805","nonce":2214131086}]},"public_addr":"172.21.15.63:6801/2214131086","cluster_addr":"172.21.15.63:6803/2214131086","heartbeat_back_addr":"172.21.15.63:6807/2214131086","heartbeat_front_addr":"172.21.15.63:6805/2214131086","state":["exists","up"]},{"osd":3,"uuid":"c83cf4bb-395a-4786-abf7-74ca93a33eb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6811","nonce":13176789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6813","nonce":13176789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6817","nonce":13176789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6815","nonce":13176789}]},"public_addr":"172.21.15.16:6811/13176789","cluster_addr":"172.21.15.16:6813/13176789","heartbeat_back_addr":"172.21.15.16:6817/13176789","heartbeat_front_addr":"172.21.15.16:6815/13176789","state":["exists","up"]},{"osd":4,"uuid":"941b3e54-47d6-4e85-8069-e730c3fa8eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6809","nonce":772161849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6811","nonce":772161849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6815","nonce":772161849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6813","nonce":772161849}]},"public_addr":"172.21.15.53:6809/772161849","cluster_addr":"172.21.15.53:6811/772161849","heartbeat_back_addr":"172.21.15.53:6815/772161849","heartbeat_front_addr":"172.21.15.53:6813/772161849","state":["exists","up"]},{"osd":5,"uuid":"51d89826-1d60-40fc-8fd8-09bfe1ab4125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6809","nonce":1602554852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6811","nonce":1602554852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6815","nonce":1602554852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6813","nonce":1602554852}]},"public_addr":"172.21.15.63:6809/1602554852","cluster_addr":"172.21.15.63:6811/1602554852","heartbeat_back_addr":"172.21.15.63:6815/1602554852","heartbeat_front_addr":"172.21.15.63:6813/1602554852","state":["exists","up"]},{"osd":6,"uuid":"978efe5d-2c55-4ff8-8a1e-9f891388945e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6819","nonce":4290459224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6821","nonce":4290459224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6825","nonce":4290459224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6823","nonce":4290459224}]},"public_addr":"172.21.15.16:6819/4290459224","cluster_addr":"172.21.15.16:6821/4290459224","heartbeat_back_addr":"172.21.15.16:6825/4290459224","heartbeat_front_addr":"172.21.15.16:6823/4290459224","state":["exists","up"]},{"osd":7,"uuid":"50d489cb-dbb8-48df-b868-b29edf12e04d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6817","nonce":1056467525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6819","nonce":1056467525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6823","nonce":1056467525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6821","nonce":1056467525}]},"public_addr":"172.21.15.53:6817/1056467525","cluster_addr":"172.21.15.53:6819/1056467525","heartbeat_back_addr":"172.21.15.53:6823/1056467525","heartbeat_front_addr":"172.21.15.53:6821/1056467525","state":["exists","up"]},{"osd":8,"uuid":"49b39414-cc9c-4057-ae3f-cc109cd979da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6817","nonce":2918516794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6819","nonce":2918516794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6823","nonce":2918516794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6821","nonce":2918516794}]},"public_addr":"172.21.15.63:6817/2918516794","cluster_addr":"172.21.15.63:6819/2918516794","heartbeat_back_addr":"172.21.15.63:6823/2918516794","heartbeat_front_addr":"172.21.15.63:6821/2918516794","state":["exists","up"]},{"osd":9,"uuid":"4c873e51-b783-4eb6-83a3-fee31722d60b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6827","nonce":1585624367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6829","nonce":1585624367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6833","nonce":1585624367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6831","nonce":1585624367}]},"public_addr":"172.21.15.16:6827/1585624367","cluster_addr":"172.21.15.16:6829/1585624367","heartbeat_back_addr":"172.21.15.16:6833/1585624367","heartbeat_front_addr":"172.21.15.16:6831/1585624367","state":["exists","up"]},{"osd":10,"uuid":"085a78bf-e2ad-481a-85a3-71d1fc549a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6825","nonce":3357446484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6827","nonce":3357446484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6831","nonce":3357446484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6829","nonce":3357446484}]},"public_addr":"172.21.15.53:6825/3357446484","cluster_addr":"172.21.15.53:6827/3357446484","heartbeat_back_addr":"172.21.15.53:6831/3357446484","heartbeat_front_addr":"172.21.15.53:6829/3357446484","state":["exists","up"]},{"osd":11,"uuid":"13302aa3-08b9-4639-9042-27d850b4d5f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6825","nonce":2167866570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6827","nonce":2167866570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6831","nonce":2167866570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6829","nonce":2167866570}]},"public_addr":"172.21.15.63:6825/2167866570","cluster_addr":"172.21.15.63:6827/2167866570","heartbeat_back_addr":"172.21.15.63:6831/2167866570","heartbeat_front_addr":"172.21.15.63:6829/2167866570","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:07.450323+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:31.363172+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:55.295021+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:19.794760+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:44.063492+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:08.195064+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:33.655328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:57.639414+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:22.402394+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:47.462050+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:12.226307+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:37.495625+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3821408150":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6801/1832422779":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/996598130":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/1062264551":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/317557779":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6801/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/3631746518":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6800/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/3116652786":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6800/1832422779":"2024-03-29T22:01:28.993434+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:10:22.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.122+0000 7f2ab1640640 1 -- 172.21.15.16:0/1671527466 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f2a940a1e90 msgr2=0x7f2a940a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:22.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.122+0000 7f2ab1640640 1 --2- 172.21.15.16:0/1671527466 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f2a940a1e90 0x7f2a940a4350 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c0095f0 tx=0x7f2a9c005f90 comp rx=0 tx=0).stop 2024-03-28T22:10:22.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.122+0000 7f2ab1640640 1 -- 172.21.15.16:0/1671527466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2aac104c00 msgr2=0x7f2aac197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:22.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.122+0000 7f2ab1640640 1 --2- 172.21.15.16:0/1671527466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2aac104c00 0x7f2aac197f60 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f2aa000c970 tx=0x7f2aa000ce30 comp rx=0 tx=0).stop 2024-03-28T22:10:22.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.122+0000 7f2ab1640640 1 -- 172.21.15.16:0/1671527466 shutdown_connections 2024-03-28T22:10:22.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.122+0000 7f2ab1640640 1 --2- 172.21.15.16:0/1671527466 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f2a940a1e90 0x7f2a940a4350 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:22.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.122+0000 7f2ab1640640 1 --2- 172.21.15.16:0/1671527466 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2aac107530 0x7f2aac19efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:22.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.122+0000 7f2ab1640640 1 --2- 172.21.15.16:0/1671527466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2aac104c00 0x7f2aac197f60 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:22.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.122+0000 7f2ab1640640 1 --2- 172.21.15.16:0/1671527466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2aac1022d0 0x7f2aac197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:22.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.122+0000 7f2ab1640640 1 -- 172.21.15.16:0/1671527466 >> 172.21.15.16:0/1671527466 conn(0x7f2aac0fbc80 msgr2=0x7f2aac102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:22.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.122+0000 7f2ab1640640 1 -- 172.21.15.16:0/1671527466 shutdown_connections 2024-03-28T22:10:22.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.122+0000 7f2ab1640640 1 -- 172.21.15.16:0/1671527466 wait complete. 2024-03-28T22:10:22.137 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T22:03:59.327832+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T22:08:28.123529+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T22:08:28.919037+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T22:10:22.137 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T22:10:22.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.241+0000 7f1738b81640 1 -- 172.21.15.16:0/3190227251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17340ffa50 msgr2=0x7f17340ffe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:22.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.241+0000 7f1738b81640 1 --2- 172.21.15.16:0/3190227251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17340ffa50 0x7f17340ffe70 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1724009930 tx=0x7f172402bc30 comp rx=0 tx=0).stop 2024-03-28T22:10:22.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.241+0000 7f1738b81640 1 -- 172.21.15.16:0/3190227251 shutdown_connections 2024-03-28T22:10:22.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.241+0000 7f1738b81640 1 --2- 172.21.15.16:0/3190227251 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f17341033e0 0x7f1734109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:22.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.241+0000 7f1738b81640 1 --2- 172.21.15.16:0/3190227251 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1734102a20 0x7f1734102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:22.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.241+0000 7f1738b81640 1 --2- 172.21.15.16:0/3190227251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17340ffa50 0x7f17340ffe70 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:22.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.241+0000 7f1738b81640 1 -- 172.21.15.16:0/3190227251 >> 172.21.15.16:0/3190227251 conn(0x7f17340fb380 msgr2=0x7f17340fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:22.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.241+0000 7f1738b81640 1 -- 172.21.15.16:0/3190227251 shutdown_connections 2024-03-28T22:10:22.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.241+0000 7f1738b81640 1 -- 172.21.15.16:0/3190227251 wait complete. 2024-03-28T22:10:22.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.242+0000 7f1738b81640 1 Processor -- start 2024-03-28T22:10:22.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.242+0000 7f1738b81640 1 -- start start 2024-03-28T22:10:22.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.242+0000 7f1738b81640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f17340ffa50 0x7f1734197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:22.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.242+0000 7f1738b81640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1734102a20 0x7f1734197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:22.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.242+0000 7f1738b81640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17341033e0 0x7f173419f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:22.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.242+0000 7f1738b81640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f173410d400 con 0x7f17341033e0 2024-03-28T22:10:22.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.242+0000 7f1738b81640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f173410d280 con 0x7f1734102a20 2024-03-28T22:10:22.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.242+0000 7f1738b81640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f173410d580 con 0x7f17340ffa50 2024-03-28T22:10:22.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.242+0000 7f1732d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17341033e0 0x7f173419f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:22.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.242+0000 7f1732d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17341033e0 0x7f173419f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44380/0 (socket says 172.21.15.16:44380) 2024-03-28T22:10:22.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.242+0000 7f1732d9d640 1 -- 172.21.15.16:0/2815731001 learned_addr learned my addr 172.21.15.16:0/2815731001 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:22.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.242+0000 7f1731d9b640 1 --2- 172.21.15.16:0/2815731001 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1734102a20 0x7f1734197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:22.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.242+0000 7f173259c640 1 --2- 172.21.15.16:0/2815731001 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f17340ffa50 0x7f1734197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:22.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.242+0000 7f1732d9d640 1 -- 172.21.15.16:0/2815731001 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f17340ffa50 msgr2=0x7f1734197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:22.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.242+0000 7f1732d9d640 1 --2- 172.21.15.16:0/2815731001 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f17340ffa50 0x7f1734197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:22.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.242+0000 7f1732d9d640 1 -- 172.21.15.16:0/2815731001 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1734102a20 msgr2=0x7f1734197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:22.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.242+0000 7f1732d9d640 1 --2- 172.21.15.16:0/2815731001 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1734102a20 0x7f1734197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:22.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.242+0000 7f1732d9d640 1 -- 172.21.15.16:0/2815731001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f173419f6a0 con 0x7f17341033e0 2024-03-28T22:10:22.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.242+0000 7f173259c640 1 --2- 172.21.15.16:0/2815731001 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f17340ffa50 0x7f1734197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:10:22.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.243+0000 7f1732d9d640 1 --2- 172.21.15.16:0/2815731001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17341033e0 0x7f173419f030 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f172c00eaa0 tx=0x7f172c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:22.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.243+0000 7f17237fe640 1 -- 172.21.15.16:0/2815731001 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f172c00cdb0 con 0x7f17341033e0 2024-03-28T22:10:22.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.243+0000 7f1738b81640 1 -- 172.21.15.16:0/2815731001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f173419f990 con 0x7f17341033e0 2024-03-28T22:10:22.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.243+0000 7f17237fe640 1 -- 172.21.15.16:0/2815731001 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f172c005020 con 0x7f17341033e0 2024-03-28T22:10:22.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.243+0000 7f17237fe640 1 -- 172.21.15.16:0/2815731001 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f172c0116b0 con 0x7f17341033e0 2024-03-28T22:10:22.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.243+0000 7f1738b81640 1 -- 172.21.15.16:0/2815731001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f173419fe90 con 0x7f17341033e0 2024-03-28T22:10:22.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.244+0000 7f1738b81640 1 -- 172.21.15.16:0/2815731001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1734108c20 con 0x7f17341033e0 2024-03-28T22:10:22.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.245+0000 7f17237fe640 1 -- 172.21.15.16:0/2815731001 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7f172c005560 con 0x7f17341033e0 2024-03-28T22:10:22.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.245+0000 7f17237fe640 1 --2- 172.21.15.16:0/2815731001 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f171c0a1e10 0x7f171c0a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:22.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.246+0000 7f17237fe640 1 -- 172.21.15.16:0/2815731001 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f172c0ce500 con 0x7f17341033e0 2024-03-28T22:10:22.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.246+0000 7f173259c640 1 --2- 172.21.15.16:0/2815731001 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f171c0a1e10 0x7f171c0a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:22.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.246+0000 7f173259c640 1 --2- 172.21.15.16:0/2815731001 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f171c0a1e10 0x7f171c0a42d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f172402c7b0 tx=0x7f1724005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:22.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.249+0000 7f17237fe640 1 -- 172.21.15.16:0/2815731001 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f172c016030 con 0x7f17341033e0 2024-03-28T22:10:22.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:22 smithi063 ceph-mon[36249]: pgmap v344: 97 pgs: 97 active+clean; 640 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 7 op/s 2024-03-28T22:10:22.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:22 smithi063 ceph-mon[36249]: mgrmap e19: x(active, since 8m), standbys: z, y 2024-03-28T22:10:22.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:22 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/86822463' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:10:22.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:22 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2267731908' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:10:22.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:22 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/1805437640' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:10:22.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:22 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/1671527466' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:10:22.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.430+0000 7f1738b81640 1 -- 172.21.15.16:0/2815731001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f173402d3c0 con 0x7f17341033e0 2024-03-28T22:10:22.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.430+0000 7f17237fe640 1 -- 172.21.15.16:0/2815731001 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f172c09b800 con 0x7f17341033e0 2024-03-28T22:10:22.433 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-28T22:10:22.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.432+0000 7f1738b81640 1 -- 172.21.15.16:0/2815731001 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f171c0a1e10 msgr2=0x7f171c0a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:22.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.432+0000 7f1738b81640 1 --2- 172.21.15.16:0/2815731001 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f171c0a1e10 0x7f171c0a42d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f172402c7b0 tx=0x7f1724005fd0 comp rx=0 tx=0).stop 2024-03-28T22:10:22.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.432+0000 7f1738b81640 1 -- 172.21.15.16:0/2815731001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17341033e0 msgr2=0x7f173419f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:22.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.432+0000 7f1738b81640 1 --2- 172.21.15.16:0/2815731001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17341033e0 0x7f173419f030 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f172c00eaa0 tx=0x7f172c00ef60 comp rx=0 tx=0).stop 2024-03-28T22:10:22.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.432+0000 7f1738b81640 1 -- 172.21.15.16:0/2815731001 shutdown_connections 2024-03-28T22:10:22.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.432+0000 7f1738b81640 1 --2- 172.21.15.16:0/2815731001 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f171c0a1e10 0x7f171c0a42d0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:22.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.432+0000 7f1738b81640 1 --2- 172.21.15.16:0/2815731001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17341033e0 0x7f173419f030 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:22.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.432+0000 7f1738b81640 1 --2- 172.21.15.16:0/2815731001 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1734102a20 0x7f1734197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:22.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.432+0000 7f1738b81640 1 --2- 172.21.15.16:0/2815731001 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f17340ffa50 0x7f1734197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:22.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.432+0000 7f1738b81640 1 -- 172.21.15.16:0/2815731001 >> 172.21.15.16:0/2815731001 conn(0x7f17340fb380 msgr2=0x7f1734105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:22.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.432+0000 7f1738b81640 1 -- 172.21.15.16:0/2815731001 shutdown_connections 2024-03-28T22:10:22.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.432+0000 7f1738b81640 1 -- 172.21.15.16:0/2815731001 wait complete. 2024-03-28T22:10:22.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:22 smithi016 ceph-mon[32374]: pgmap v344: 97 pgs: 97 active+clean; 640 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 7 op/s 2024-03-28T22:10:22.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:22 smithi016 ceph-mon[32374]: mgrmap e19: x(active, since 8m), standbys: z, y 2024-03-28T22:10:22.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:22 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/86822463' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:10:22.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:22 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2267731908' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:10:22.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:22 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/1805437640' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:10:22.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:22 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/1671527466' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:10:22.447 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T22:10:22.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:22 smithi053 ceph-mon[35979]: pgmap v344: 97 pgs: 97 active+clean; 640 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 7 op/s 2024-03-28T22:10:22.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:22 smithi053 ceph-mon[35979]: mgrmap e19: x(active, since 8m), standbys: z, y 2024-03-28T22:10:22.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:22 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/86822463' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:10:22.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:22 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2267731908' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:10:22.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:22 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/1805437640' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:10:22.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:22 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/1671527466' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:10:22.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.551+0000 7fdef7ada640 1 -- 172.21.15.16:0/3529753307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdef0101c50 msgr2=0x7fdef0102070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:22.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.551+0000 7fdef7ada640 1 --2- 172.21.15.16:0/3529753307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdef0101c50 0x7fdef0102070 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fdeec009900 tx=0x7fdeec02bc30 comp rx=0 tx=0).stop 2024-03-28T22:10:22.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.551+0000 7fdef7ada640 1 -- 172.21.15.16:0/3529753307 shutdown_connections 2024-03-28T22:10:22.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.551+0000 7fdef7ada640 1 --2- 172.21.15.16:0/3529753307 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdef0101260 0x7fdef0108b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:22.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.551+0000 7fdef7ada640 1 --2- 172.21.15.16:0/3529753307 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdef01008a0 0x7fdef0100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:22.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.551+0000 7fdef7ada640 1 --2- 172.21.15.16:0/3529753307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdef0101c50 0x7fdef0102070 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:22.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.551+0000 7fdef7ada640 1 -- 172.21.15.16:0/3529753307 >> 172.21.15.16:0/3529753307 conn(0x7fdef00fb370 msgr2=0x7fdef00fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:22.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.551+0000 7fdef7ada640 1 -- 172.21.15.16:0/3529753307 shutdown_connections 2024-03-28T22:10:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.551+0000 7fdef7ada640 1 -- 172.21.15.16:0/3529753307 wait complete. 2024-03-28T22:10:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.552+0000 7fdef7ada640 1 Processor -- start 2024-03-28T22:10:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.552+0000 7fdef7ada640 1 -- start start 2024-03-28T22:10:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.552+0000 7fdef7ada640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdef01008a0 0x7fdef0197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.552+0000 7fdef7ada640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdef0101260 0x7fdef0197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.552+0000 7fdef7ada640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdef0101c50 0x7fdef019f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.552+0000 7fdef7ada640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdef010c310 con 0x7fdef01008a0 2024-03-28T22:10:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.552+0000 7fdef7ada640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fdef010c190 con 0x7fdef0101c50 2024-03-28T22:10:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.552+0000 7fdef7ada640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fdef010c490 con 0x7fdef0101260 2024-03-28T22:10:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.552+0000 7fdef5876640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdef01008a0 0x7fdef0197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.552+0000 7fdef5876640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdef01008a0 0x7fdef0197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44396/0 (socket says 172.21.15.16:44396) 2024-03-28T22:10:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.552+0000 7fdef5876640 1 -- 172.21.15.16:0/485856424 learned_addr learned my addr 172.21.15.16:0/485856424 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.552+0000 7fdef6077640 1 --2- 172.21.15.16:0/485856424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdef0101c50 0x7fdef019f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.552+0000 7fdef5075640 1 --2- 172.21.15.16:0/485856424 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdef0101260 0x7fdef0197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:22.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.552+0000 7fdef5876640 1 -- 172.21.15.16:0/485856424 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdef0101260 msgr2=0x7fdef0197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:22.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.552+0000 7fdef5876640 1 --2- 172.21.15.16:0/485856424 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdef0101260 0x7fdef0197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:22.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.552+0000 7fdef5876640 1 -- 172.21.15.16:0/485856424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdef0101c50 msgr2=0x7fdef019f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:22.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.552+0000 7fdef5876640 1 --2- 172.21.15.16:0/485856424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdef0101c50 0x7fdef019f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:22.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.552+0000 7fdef5876640 1 -- 172.21.15.16:0/485856424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdef019f6d0 con 0x7fdef01008a0 2024-03-28T22:10:22.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.552+0000 7fdef5075640 1 --2- 172.21.15.16:0/485856424 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdef0101260 0x7fdef0197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:10:22.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.553+0000 7fdef5876640 1 --2- 172.21.15.16:0/485856424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdef01008a0 0x7fdef0197ab0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fdeec000c00 tx=0x7fdeec005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:22.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.553+0000 7fdee6ffd640 1 -- 172.21.15.16:0/485856424 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeec041070 con 0x7fdef01008a0 2024-03-28T22:10:22.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.553+0000 7fdef7ada640 1 -- 172.21.15.16:0/485856424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdef019f960 con 0x7fdef01008a0 2024-03-28T22:10:22.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.553+0000 7fdee6ffd640 1 -- 172.21.15.16:0/485856424 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdeec0044d0 con 0x7fdef01008a0 2024-03-28T22:10:22.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.553+0000 7fdee6ffd640 1 -- 172.21.15.16:0/485856424 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeec034440 con 0x7fdef01008a0 2024-03-28T22:10:22.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.553+0000 7fdef7ada640 1 -- 172.21.15.16:0/485856424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdef019fea0 con 0x7fdef01008a0 2024-03-28T22:10:22.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.554+0000 7fdef7ada640 1 -- 172.21.15.16:0/485856424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdef01a0070 con 0x7fdef01008a0 2024-03-28T22:10:22.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.555+0000 7fdee6ffd640 1 -- 172.21.15.16:0/485856424 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7fdeec044070 con 0x7fdef01008a0 2024-03-28T22:10:22.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.555+0000 7fdee6ffd640 1 --2- 172.21.15.16:0/485856424 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdedc0a1e10 0x7fdedc0a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:22.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.556+0000 7fdef5075640 1 --2- 172.21.15.16:0/485856424 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdedc0a1e10 0x7fdedc0a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:22.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.556+0000 7fdee6ffd640 1 -- 172.21.15.16:0/485856424 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fdeec0efa20 con 0x7fdef01008a0 2024-03-28T22:10:22.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.556+0000 7fdef5075640 1 --2- 172.21.15.16:0/485856424 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdedc0a1e10 0x7fdedc0a42d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fdef0198fc0 tx=0x7fdee0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:22.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.560+0000 7fdee6ffd640 1 -- 172.21.15.16:0/485856424 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fdeec0bcdd0 con 0x7fdef01008a0 2024-03-28T22:10:22.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.742+0000 7fdef7ada640 1 -- 172.21.15.16:0/485856424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdef01a0400 con 0x7fdef01008a0 2024-03-28T22:10:22.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.742+0000 7fdee6ffd640 1 -- 172.21.15.16:0/485856424 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7fdeec0bc6d0 con 0x7fdef01008a0 2024-03-28T22:10:22.745 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-28T22:10:22.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.744+0000 7fdef7ada640 1 -- 172.21.15.16:0/485856424 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdedc0a1e10 msgr2=0x7fdedc0a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:22.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.744+0000 7fdef7ada640 1 --2- 172.21.15.16:0/485856424 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdedc0a1e10 0x7fdedc0a42d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fdef0198fc0 tx=0x7fdee0005f90 comp rx=0 tx=0).stop 2024-03-28T22:10:22.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.744+0000 7fdef7ada640 1 -- 172.21.15.16:0/485856424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdef01008a0 msgr2=0x7fdef0197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:22.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.744+0000 7fdef7ada640 1 --2- 172.21.15.16:0/485856424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdef01008a0 0x7fdef0197ab0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fdeec000c00 tx=0x7fdeec005d50 comp rx=0 tx=0).stop 2024-03-28T22:10:22.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.744+0000 7fdef7ada640 1 -- 172.21.15.16:0/485856424 shutdown_connections 2024-03-28T22:10:22.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.744+0000 7fdef7ada640 1 --2- 172.21.15.16:0/485856424 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fdedc0a1e10 0x7fdedc0a42d0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:22.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.744+0000 7fdef7ada640 1 --2- 172.21.15.16:0/485856424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdef0101c50 0x7fdef019f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:22.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.744+0000 7fdef7ada640 1 --2- 172.21.15.16:0/485856424 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdef0101260 0x7fdef0197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:22.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.744+0000 7fdef7ada640 1 --2- 172.21.15.16:0/485856424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdef01008a0 0x7fdef0197ab0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:22.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.744+0000 7fdef7ada640 1 -- 172.21.15.16:0/485856424 >> 172.21.15.16:0/485856424 conn(0x7fdef00fb370 msgr2=0x7fdef00fce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:22.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.744+0000 7fdef7ada640 1 -- 172.21.15.16:0/485856424 shutdown_connections 2024-03-28T22:10:22.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.744+0000 7fdef7ada640 1 -- 172.21.15.16:0/485856424 wait complete. 2024-03-28T22:10:22.758 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T22:10:22.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.861+0000 7fedc7847640 1 -- 172.21.15.16:0/1213369092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedc0100660 msgr2=0x7fedc0100a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:22.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.861+0000 7fedc7847640 1 --2- 172.21.15.16:0/1213369092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedc0100660 0x7fedc0100a80 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fedbc009900 tx=0x7fedbc02bc30 comp rx=0 tx=0).stop 2024-03-28T22:10:22.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.861+0000 7fedc7847640 1 -- 172.21.15.16:0/1213369092 shutdown_connections 2024-03-28T22:10:22.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.861+0000 7fedc7847640 1 --2- 172.21.15.16:0/1213369092 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fedc0102210 0x7fedc0108aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:22.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.861+0000 7fedc7847640 1 --2- 172.21.15.16:0/1213369092 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fedc0101850 0x7fedc0101cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:22.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.861+0000 7fedc7847640 1 --2- 172.21.15.16:0/1213369092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedc0100660 0x7fedc0100a80 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:22.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.861+0000 7fedc7847640 1 -- 172.21.15.16:0/1213369092 >> 172.21.15.16:0/1213369092 conn(0x7fedc00fbbb0 msgr2=0x7fedc00fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:22.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.861+0000 7fedc7847640 1 -- 172.21.15.16:0/1213369092 shutdown_connections 2024-03-28T22:10:22.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.861+0000 7fedc7847640 1 -- 172.21.15.16:0/1213369092 wait complete. 2024-03-28T22:10:22.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.862+0000 7fedc7847640 1 Processor -- start 2024-03-28T22:10:22.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.862+0000 7fedc7847640 1 -- start start 2024-03-28T22:10:22.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.862+0000 7fedc7847640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fedc0100660 0x7fedc006a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:22.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.862+0000 7fedc7847640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedc0101850 0x7fedc019c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:22.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.862+0000 7fedc7847640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fedc0102210 0x7fedc019ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:22.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.862+0000 7fedc7847640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fedc010c300 con 0x7fedc0101850 2024-03-28T22:10:22.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.862+0000 7fedc7847640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fedc010c180 con 0x7fedc0102210 2024-03-28T22:10:22.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.862+0000 7fedc7847640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fedc010c480 con 0x7fedc0100660 2024-03-28T22:10:22.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.862+0000 7fedc4de2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedc0101850 0x7fedc019c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:22.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.862+0000 7fedc4de2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedc0101850 0x7fedc019c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44410/0 (socket says 172.21.15.16:44410) 2024-03-28T22:10:22.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.862+0000 7fedc4de2640 1 -- 172.21.15.16:0/361900355 learned_addr learned my addr 172.21.15.16:0/361900355 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:22.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.862+0000 7fedc5de4640 1 --2- 172.21.15.16:0/361900355 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fedc0102210 0x7fedc019ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:22.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.862+0000 7fedc55e3640 1 --2- 172.21.15.16:0/361900355 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fedc0100660 0x7fedc006a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:22.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.863+0000 7fedc4de2640 1 -- 172.21.15.16:0/361900355 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fedc0100660 msgr2=0x7fedc006a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:22.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.863+0000 7fedc4de2640 1 --2- 172.21.15.16:0/361900355 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fedc0100660 0x7fedc006a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:22.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.863+0000 7fedc4de2640 1 -- 172.21.15.16:0/361900355 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fedc0102210 msgr2=0x7fedc019ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:22.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.863+0000 7fedc4de2640 1 --2- 172.21.15.16:0/361900355 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fedc0102210 0x7fedc019ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:22.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.863+0000 7fedc4de2640 1 -- 172.21.15.16:0/361900355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedc019f630 con 0x7fedc0101850 2024-03-28T22:10:22.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.863+0000 7fedc5de4640 1 --2- 172.21.15.16:0/361900355 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fedc0102210 0x7fedc019ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:10:22.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.863+0000 7fedc4de2640 1 --2- 172.21.15.16:0/361900355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedc0101850 0x7fedc019c700 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fedb000c920 tx=0x7fedb000cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:22.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.863+0000 7fedb67fc640 1 -- 172.21.15.16:0/361900355 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb0007bb0 con 0x7fedc0101850 2024-03-28T22:10:22.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.863+0000 7fedb67fc640 1 -- 172.21.15.16:0/361900355 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedb0007d50 con 0x7fedc0101850 2024-03-28T22:10:22.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.863+0000 7fedc7847640 1 -- 172.21.15.16:0/361900355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedc019f920 con 0x7fedc0101850 2024-03-28T22:10:22.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.863+0000 7fedb67fc640 1 -- 172.21.15.16:0/361900355 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb00187d0 con 0x7fedc0101850 2024-03-28T22:10:22.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.863+0000 7fedc7847640 1 -- 172.21.15.16:0/361900355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedc019fec0 con 0x7fedc0101850 2024-03-28T22:10:22.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.864+0000 7fedc7847640 1 -- 172.21.15.16:0/361900355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedc0107920 con 0x7fedc0101850 2024-03-28T22:10:22.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.865+0000 7fedb67fc640 1 -- 172.21.15.16:0/361900355 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7fedb000f450 con 0x7fedc0101850 2024-03-28T22:10:22.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.866+0000 7fedb67fc640 1 --2- 172.21.15.16:0/361900355 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fedac0a1e90 0x7fedac0a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:22.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.866+0000 7fedc55e3640 1 --2- 172.21.15.16:0/361900355 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fedac0a1e90 0x7fedac0a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:22.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.866+0000 7fedb67fc640 1 -- 172.21.15.16:0/361900355 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fedb00ce1a0 con 0x7fedc0101850 2024-03-28T22:10:22.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.866+0000 7fedc55e3640 1 --2- 172.21.15.16:0/361900355 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fedac0a1e90 0x7fedac0a4350 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fedbc02c170 tx=0x7fedbc02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:22.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:22.869+0000 7fedb67fc640 1 -- 172.21.15.16:0/361900355 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fedb0014030 con 0x7fedc0101850 2024-03-28T22:10:23.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.052+0000 7fedc7847640 1 -- 172.21.15.16:0/361900355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fedc002d3c0 con 0x7fedc0101850 2024-03-28T22:10:23.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.052+0000 7fedb67fc640 1 -- 172.21.15.16:0/361900355 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7fedb009b5b0 con 0x7fedc0101850 2024-03-28T22:10:23.055 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-28T22:10:23.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.054+0000 7fedc7847640 1 -- 172.21.15.16:0/361900355 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fedac0a1e90 msgr2=0x7fedac0a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:23.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.054+0000 7fedc7847640 1 --2- 172.21.15.16:0/361900355 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fedac0a1e90 0x7fedac0a4350 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fedbc02c170 tx=0x7fedbc02df90 comp rx=0 tx=0).stop 2024-03-28T22:10:23.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.055+0000 7fedc7847640 1 -- 172.21.15.16:0/361900355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedc0101850 msgr2=0x7fedc019c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:23.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.055+0000 7fedc7847640 1 --2- 172.21.15.16:0/361900355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedc0101850 0x7fedc019c700 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fedb000c920 tx=0x7fedb000cd20 comp rx=0 tx=0).stop 2024-03-28T22:10:23.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.055+0000 7fedc7847640 1 -- 172.21.15.16:0/361900355 shutdown_connections 2024-03-28T22:10:23.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.055+0000 7fedc7847640 1 --2- 172.21.15.16:0/361900355 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fedac0a1e90 0x7fedac0a4350 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:23.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.055+0000 7fedc7847640 1 --2- 172.21.15.16:0/361900355 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fedc0102210 0x7fedc019ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:23.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.055+0000 7fedc7847640 1 --2- 172.21.15.16:0/361900355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedc0101850 0x7fedc019c700 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:23.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.055+0000 7fedc7847640 1 --2- 172.21.15.16:0/361900355 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fedc0100660 0x7fedc006a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:23.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.055+0000 7fedc7847640 1 -- 172.21.15.16:0/361900355 >> 172.21.15.16:0/361900355 conn(0x7fedc00fbbb0 msgr2=0x7fedc0104af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:23.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.055+0000 7fedc7847640 1 -- 172.21.15.16:0/361900355 shutdown_connections 2024-03-28T22:10:23.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.055+0000 7fedc7847640 1 -- 172.21.15.16:0/361900355 wait complete. 2024-03-28T22:10:23.068 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-03-28T22:10:23.069 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:10:23.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.169+0000 7fb82aa5c640 1 -- 172.21.15.16:0/2903613417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb824101790 msgr2=0x7fb824101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:23.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.169+0000 7fb82aa5c640 1 --2- 172.21.15.16:0/2903613417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb824101790 0x7fb824101bb0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fb814009900 tx=0x7fb81402bc30 comp rx=0 tx=0).stop 2024-03-28T22:10:23.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.169+0000 7fb82aa5c640 1 -- 172.21.15.16:0/2903613417 shutdown_connections 2024-03-28T22:10:23.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.169+0000 7fb82aa5c640 1 --2- 172.21.15.16:0/2903613417 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb824103340 0x7fb824109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:23.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.169+0000 7fb82aa5c640 1 --2- 172.21.15.16:0/2903613417 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb824102980 0x7fb824102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:23.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.169+0000 7fb82aa5c640 1 --2- 172.21.15.16:0/2903613417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb824101790 0x7fb824101bb0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:23.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.169+0000 7fb82aa5c640 1 -- 172.21.15.16:0/2903613417 >> 172.21.15.16:0/2903613417 conn(0x7fb8240fcce0 msgr2=0x7fb8240ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:23.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.169+0000 7fb82aa5c640 1 -- 172.21.15.16:0/2903613417 shutdown_connections 2024-03-28T22:10:23.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.169+0000 7fb82aa5c640 1 -- 172.21.15.16:0/2903613417 wait complete. 2024-03-28T22:10:23.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.169+0000 7fb82aa5c640 1 Processor -- start 2024-03-28T22:10:23.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.169+0000 7fb82aa5c640 1 -- start start 2024-03-28T22:10:23.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.169+0000 7fb82aa5c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb824101790 0x7fb824197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:23.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.169+0000 7fb82aa5c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb824102980 0x7fb824198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:23.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.170+0000 7fb82aa5c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb824103340 0x7fb82419f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:23.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.170+0000 7fb82aa5c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb82410d430 con 0x7fb824102980 2024-03-28T22:10:23.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.170+0000 7fb82aa5c640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb82410d2b0 con 0x7fb824103340 2024-03-28T22:10:23.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.170+0000 7fb82aa5c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb82410d5b0 con 0x7fb824101790 2024-03-28T22:10:23.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.170+0000 7fb8237fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb824102980 0x7fb824198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:23.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.170+0000 7fb8237fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb824102980 0x7fb824198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44434/0 (socket says 172.21.15.16:44434) 2024-03-28T22:10:23.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.170+0000 7fb8237fe640 1 -- 172.21.15.16:0/462332625 learned_addr learned my addr 172.21.15.16:0/462332625 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:23.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.170+0000 7fb828ff9640 1 --2- 172.21.15.16:0/462332625 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb824103340 0x7fb82419f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:23.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.170+0000 7fb823fff640 1 --2- 172.21.15.16:0/462332625 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb824101790 0x7fb824197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:23.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.170+0000 7fb8237fe640 1 -- 172.21.15.16:0/462332625 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb824101790 msgr2=0x7fb824197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:23.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.170+0000 7fb8237fe640 1 --2- 172.21.15.16:0/462332625 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb824101790 0x7fb824197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:23.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.170+0000 7fb8237fe640 1 -- 172.21.15.16:0/462332625 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb824103340 msgr2=0x7fb82419f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:23.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.170+0000 7fb8237fe640 1 --2- 172.21.15.16:0/462332625 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb824103340 0x7fb82419f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:23.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.170+0000 7fb8237fe640 1 -- 172.21.15.16:0/462332625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb82419f760 con 0x7fb824102980 2024-03-28T22:10:23.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.170+0000 7fb8237fe640 1 --2- 172.21.15.16:0/462332625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb824102980 0x7fb824198080 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fb80c00d950 tx=0x7fb80c00dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:23.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.170+0000 7fb8217fa640 1 -- 172.21.15.16:0/462332625 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb80c017070 con 0x7fb824102980 2024-03-28T22:10:23.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.170+0000 7fb82aa5c640 1 -- 172.21.15.16:0/462332625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb82419fa50 con 0x7fb824102980 2024-03-28T22:10:23.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.170+0000 7fb8217fa640 1 -- 172.21.15.16:0/462332625 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb80c00de80 con 0x7fb824102980 2024-03-28T22:10:23.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.170+0000 7fb8217fa640 1 -- 172.21.15.16:0/462332625 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb80c00f620 con 0x7fb824102980 2024-03-28T22:10:23.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.170+0000 7fb82aa5c640 1 -- 172.21.15.16:0/462332625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb82419fff0 con 0x7fb824102980 2024-03-28T22:10:23.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.171+0000 7fb82aa5c640 1 -- 172.21.15.16:0/462332625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb824108a50 con 0x7fb824102980 2024-03-28T22:10:23.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.173+0000 7fb8217fa640 1 -- 172.21.15.16:0/462332625 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7fb80c009c30 con 0x7fb824102980 2024-03-28T22:10:23.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.173+0000 7fb8217fa640 1 --2- 172.21.15.16:0/462332625 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb8100a1e10 0x7fb8100a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:23.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.174+0000 7fb823fff640 1 --2- 172.21.15.16:0/462332625 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb8100a1e10 0x7fb8100a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:23.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.174+0000 7fb8217fa640 1 -- 172.21.15.16:0/462332625 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fb80c098830 con 0x7fb824102980 2024-03-28T22:10:23.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.174+0000 7fb823fff640 1 --2- 172.21.15.16:0/462332625 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb8100a1e10 0x7fb8100a42d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb81402c240 tx=0x7fb81402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:23.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.179+0000 7fb8217fa640 1 -- 172.21.15.16:0/462332625 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb80c09ae90 con 0x7fb824102980 2024-03-28T22:10:23.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.400+0000 7fb82aa5c640 1 -- 172.21.15.16:0/462332625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb82402d170 con 0x7fb824102980 2024-03-28T22:10:23.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.401+0000 7fb8217fa640 1 -- 172.21.15.16:0/462332625 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11851 (secure 0 0 0) 0x7fb80c09a790 con 0x7fb824102980 2024-03-28T22:10:23.404 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:10:23.404 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/1644581227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1644581227},{"type":"v1","addr":"172.21.15.16:6839","nonce":1644581227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/2593062669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2593062669},{"type":"v1","addr":"172.21.15.16:6841","nonce":2593062669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:08:29.672976+0000","modified":"2024-03-28T22:10:19.955098+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24566,"mds_1":24524,"mds_2":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24466":{"gid":24466,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/247249857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":247249857},{"type":"v1","addr":"172.21.15.63:6835","nonce":247249857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6837/2324231032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":2324231032},{"type":"v1","addr":"172.21.15.63:6837","nonce":2324231032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6839/1177197077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":1177197077},{"type":"v1","addr":"172.21.15.63:6839","nonce":1177197077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"e","rank":2,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.53:6835/350727634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":350727634},{"type":"v1","addr":"172.21.15.53:6835","nonce":350727634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/349877573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":349877573},{"type":"v1","addr":"172.21.15.53:6837","nonce":349877573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6839/3714153956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":3714153956},{"type":"v1","addr":"172.21.15.53:6839","nonce":3714153956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24566,"qdb_cluster":[24500,24524,24566]},"id":1}]} 2024-03-28T22:10:23.404 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 21 2024-03-28T22:10:23.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.403+0000 7fb82aa5c640 1 -- 172.21.15.16:0/462332625 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb8100a1e10 msgr2=0x7fb8100a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:23.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.403+0000 7fb82aa5c640 1 --2- 172.21.15.16:0/462332625 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb8100a1e10 0x7fb8100a42d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb81402c240 tx=0x7fb81402df90 comp rx=0 tx=0).stop 2024-03-28T22:10:23.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.403+0000 7fb82aa5c640 1 -- 172.21.15.16:0/462332625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb824102980 msgr2=0x7fb824198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:23.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.403+0000 7fb82aa5c640 1 --2- 172.21.15.16:0/462332625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb824102980 0x7fb824198080 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fb80c00d950 tx=0x7fb80c00dd50 comp rx=0 tx=0).stop 2024-03-28T22:10:23.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.403+0000 7fb82aa5c640 1 -- 172.21.15.16:0/462332625 shutdown_connections 2024-03-28T22:10:23.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.403+0000 7fb82aa5c640 1 --2- 172.21.15.16:0/462332625 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fb8100a1e10 0x7fb8100a42d0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:23.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.403+0000 7fb82aa5c640 1 --2- 172.21.15.16:0/462332625 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb824103340 0x7fb82419f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:23.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.403+0000 7fb82aa5c640 1 --2- 172.21.15.16:0/462332625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb824102980 0x7fb824198080 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:23.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.403+0000 7fb82aa5c640 1 --2- 172.21.15.16:0/462332625 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb824101790 0x7fb824197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:23.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.403+0000 7fb82aa5c640 1 -- 172.21.15.16:0/462332625 >> 172.21.15.16:0/462332625 conn(0x7fb8240fcce0 msgr2=0x7fb824105c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:23.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.403+0000 7fb82aa5c640 1 -- 172.21.15.16:0/462332625 shutdown_connections 2024-03-28T22:10:23.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.403+0000 7fb82aa5c640 1 -- 172.21.15.16:0/462332625 wait complete. 2024-03-28T22:10:23.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:23 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2815731001' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:10:23.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:23 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/485856424' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:10:23.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:23 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/361900355' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:10:23.417 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-03-28T22:10:23.417 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-03-28T22:10:23.417 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T22:10:23.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.518+0000 7fc1a0bd0640 1 -- 172.21.15.16:0/2052293216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc19c1029d0 msgr2=0x7fc19c102e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:23.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.518+0000 7fc1a0bd0640 1 --2- 172.21.15.16:0/2052293216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc19c1029d0 0x7fc19c102e50 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fc184009900 tx=0x7fc18402bc30 comp rx=0 tx=0).stop 2024-03-28T22:10:23.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.518+0000 7fc1a0bd0640 1 -- 172.21.15.16:0/2052293216 shutdown_connections 2024-03-28T22:10:23.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.518+0000 7fc1a0bd0640 1 --2- 172.21.15.16:0/2052293216 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc19c103390 0x7fc19c109c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:23.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.518+0000 7fc1a0bd0640 1 --2- 172.21.15.16:0/2052293216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc19c1029d0 0x7fc19c102e50 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:23.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.518+0000 7fc1a0bd0640 1 --2- 172.21.15.16:0/2052293216 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc19c1017e0 0x7fc19c101c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:23.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.518+0000 7fc1a0bd0640 1 -- 172.21.15.16:0/2052293216 >> 172.21.15.16:0/2052293216 conn(0x7fc19c0fcd70 msgr2=0x7fc19c0ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:23.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.518+0000 7fc1a0bd0640 1 -- 172.21.15.16:0/2052293216 shutdown_connections 2024-03-28T22:10:23.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.518+0000 7fc1a0bd0640 1 -- 172.21.15.16:0/2052293216 wait complete. 2024-03-28T22:10:23.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.519+0000 7fc1a0bd0640 1 Processor -- start 2024-03-28T22:10:23.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.519+0000 7fc1a0bd0640 1 -- start start 2024-03-28T22:10:23.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.519+0000 7fc1a0bd0640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc19c1017e0 0x7fc19c197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:23.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.519+0000 7fc1a0bd0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc19c1029d0 0x7fc19c197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:23.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.519+0000 7fc1a0bd0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc19c103390 0x7fc19c19eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:23.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.519+0000 7fc1a0bd0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc19c10d3d0 con 0x7fc19c1029d0 2024-03-28T22:10:23.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.519+0000 7fc1a0bd0640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc19c10d250 con 0x7fc19c1017e0 2024-03-28T22:10:23.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.519+0000 7fc1a0bd0640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc19c10d550 con 0x7fc19c103390 2024-03-28T22:10:23.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.519+0000 7fc191d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc19c1029d0 0x7fc19c197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:23.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.519+0000 7fc191d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc19c1029d0 0x7fc19c197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44462/0 (socket says 172.21.15.16:44462) 2024-03-28T22:10:23.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.519+0000 7fc19a59c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc19c1017e0 0x7fc19c197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:23.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.519+0000 7fc191d9b640 1 -- 172.21.15.16:0/2583745546 learned_addr learned my addr 172.21.15.16:0/2583745546 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:23.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.519+0000 7fc19ad9d640 1 --2- 172.21.15.16:0/2583745546 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc19c103390 0x7fc19c19eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:23.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.519+0000 7fc191d9b640 1 -- 172.21.15.16:0/2583745546 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc19c103390 msgr2=0x7fc19c19eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:23.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.519+0000 7fc191d9b640 1 --2- 172.21.15.16:0/2583745546 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc19c103390 0x7fc19c19eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:23.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.519+0000 7fc191d9b640 1 -- 172.21.15.16:0/2583745546 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc19c1017e0 msgr2=0x7fc19c197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:23.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.519+0000 7fc191d9b640 1 --2- 172.21.15.16:0/2583745546 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc19c1017e0 0x7fc19c197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:23.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.519+0000 7fc191d9b640 1 -- 172.21.15.16:0/2583745546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc19c19f660 con 0x7fc19c1029d0 2024-03-28T22:10:23.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.519+0000 7fc19ad9d640 1 --2- 172.21.15.16:0/2583745546 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc19c103390 0x7fc19c19eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:10:23.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.520+0000 7fc191d9b640 1 --2- 172.21.15.16:0/2583745546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc19c1029d0 0x7fc19c197f80 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fc184009230 tx=0x7fc184005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:23.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.520+0000 7fc193fff640 1 -- 172.21.15.16:0/2583745546 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc184041070 con 0x7fc19c1029d0 2024-03-28T22:10:23.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.520+0000 7fc1a0bd0640 1 -- 172.21.15.16:0/2583745546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc19c19f8f0 con 0x7fc19c1029d0 2024-03-28T22:10:23.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.520+0000 7fc193fff640 1 -- 172.21.15.16:0/2583745546 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1840044d0 con 0x7fc19c1029d0 2024-03-28T22:10:23.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.520+0000 7fc193fff640 1 -- 172.21.15.16:0/2583745546 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc184034440 con 0x7fc19c1029d0 2024-03-28T22:10:23.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.520+0000 7fc1a0bd0640 1 -- 172.21.15.16:0/2583745546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc19c19fde0 con 0x7fc19c1029d0 2024-03-28T22:10:23.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.521+0000 7fc1a0bd0640 1 -- 172.21.15.16:0/2583745546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc19c108b00 con 0x7fc19c1029d0 2024-03-28T22:10:23.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.522+0000 7fc193fff640 1 -- 172.21.15.16:0/2583745546 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7fc184044070 con 0x7fc19c1029d0 2024-03-28T22:10:23.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.522+0000 7fc193fff640 1 --2- 172.21.15.16:0/2583745546 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc1880a1e10 0x7fc1880a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:23.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.523+0000 7fc19a59c640 1 --2- 172.21.15.16:0/2583745546 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc1880a1e10 0x7fc1880a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:23.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.523+0000 7fc193fff640 1 -- 172.21.15.16:0/2583745546 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fc1840efa70 con 0x7fc19c1029d0 2024-03-28T22:10:23.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.523+0000 7fc19a59c640 1 --2- 172.21.15.16:0/2583745546 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc1880a1e10 0x7fc1880a42d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc19c102830 tx=0x7fc18c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:23.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.527+0000 7fc193fff640 1 -- 172.21.15.16:0/2583745546 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc1840bce80 con 0x7fc19c1029d0 2024-03-28T22:10:23.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:23 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2815731001' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:10:23.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:23 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/485856424' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:10:23.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:23 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/361900355' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:10:23.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:23 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2815731001' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:10:23.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:23 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/485856424' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:10:23.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:23 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/361900355' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:10:23.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.705+0000 7fc1a0bd0640 1 -- 172.21.15.16:0/2583745546 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc19c061b90 con 0x7fc1880a1e10 2024-03-28T22:10:23.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.712+0000 7fc193fff640 1 -- 172.21.15.16:0/2583745546 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258824 (secure 0 0 0) 0x7fc19c061b90 con 0x7fc1880a1e10 2024-03-28T22:10:23.719 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:10:23.720 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-03-28T22:10:23.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.715+0000 7fc1a0bd0640 1 -- 172.21.15.16:0/2583745546 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc1880a1e10 msgr2=0x7fc1880a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:23.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.715+0000 7fc1a0bd0640 1 --2- 172.21.15.16:0/2583745546 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc1880a1e10 0x7fc1880a42d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc19c102830 tx=0x7fc18c00b390 comp rx=0 tx=0).stop 2024-03-28T22:10:23.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.715+0000 7fc1a0bd0640 1 -- 172.21.15.16:0/2583745546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc19c1029d0 msgr2=0x7fc19c197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:23.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.715+0000 7fc1a0bd0640 1 --2- 172.21.15.16:0/2583745546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc19c1029d0 0x7fc19c197f80 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fc184009230 tx=0x7fc184005d50 comp rx=0 tx=0).stop 2024-03-28T22:10:23.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.715+0000 7fc1a0bd0640 1 -- 172.21.15.16:0/2583745546 shutdown_connections 2024-03-28T22:10:23.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.715+0000 7fc1a0bd0640 1 --2- 172.21.15.16:0/2583745546 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc1880a1e10 0x7fc1880a42d0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:23.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.715+0000 7fc1a0bd0640 1 --2- 172.21.15.16:0/2583745546 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc19c103390 0x7fc19c19eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:23.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.715+0000 7fc1a0bd0640 1 --2- 172.21.15.16:0/2583745546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc19c1029d0 0x7fc19c197f80 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:23.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.715+0000 7fc1a0bd0640 1 --2- 172.21.15.16:0/2583745546 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc19c1017e0 0x7fc19c197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:23.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.715+0000 7fc1a0bd0640 1 -- 172.21.15.16:0/2583745546 >> 172.21.15.16:0/2583745546 conn(0x7fc19c0fcd70 msgr2=0x7fc19c105cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:23.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.716+0000 7fc1a0bd0640 1 -- 172.21.15.16:0/2583745546 shutdown_connections 2024-03-28T22:10:23.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.716+0000 7fc1a0bd0640 1 -- 172.21.15.16:0/2583745546 wait complete. 2024-03-28T22:10:23.729 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-03-28T22:10:23.100856+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":655527,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":808,"num_read_kb":523,"num_write":354,"num_write_kb":3059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":787,"ondisk_log_size":787,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":383216,"kb_used_data":13264,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124558608,"statfs":{"total":1151940427776,"available":1151548014592,"internally_reserved":0,"allocated":13582336,"data_stored":7473113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1090,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":72,"num_read_kb":37,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002694"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.119037+0000","last_change":"2024-03-28T22:08:29.962664+0000","last_active":"2024-03-28T22:09:59.119037+0000","last_peered":"2024-03-28T22:09:59.119037+0000","last_clean":"2024-03-28T22:09:59.119037+0000","last_became_active":"2024-03-28T22:08:29.962029+0000","last_became_peered":"2024-03-28T22:08:29.962029+0000","last_unstale":"2024-03-28T22:09:59.119037+0000","last_undegraded":"2024-03-28T22:09:59.119037+0000","last_fullsized":"2024-03-28T22:09:59.119037+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:20:42.832304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.099587+0000","last_change":"2024-03-28T22:08:29.965617+0000","last_active":"2024-03-28T22:09:59.099587+0000","last_peered":"2024-03-28T22:09:59.099587+0000","last_clean":"2024-03-28T22:09:59.099587+0000","last_became_active":"2024-03-28T22:08:29.965248+0000","last_became_peered":"2024-03-28T22:08:29.965248+0000","last_unstale":"2024-03-28T22:09:59.099587+0000","last_undegraded":"2024-03-28T22:09:59.099587+0000","last_fullsized":"2024-03-28T22:09:59.099587+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:37:46.703768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.119356+0000","last_change":"2024-03-28T22:08:29.958452+0000","last_active":"2024-03-28T22:09:59.119356+0000","last_peered":"2024-03-28T22:09:59.119356+0000","last_clean":"2024-03-28T22:09:59.119356+0000","last_became_active":"2024-03-28T22:08:29.958089+0000","last_became_peered":"2024-03-28T22:08:29.958089+0000","last_unstale":"2024-03-28T22:09:59.119356+0000","last_undegraded":"2024-03-28T22:09:59.119356+0000","last_fullsized":"2024-03-28T22:09:59.119356+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:10:44.084925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.104325+0000","last_change":"2024-03-28T22:08:29.964812+0000","last_active":"2024-03-28T22:09:59.104325+0000","last_peered":"2024-03-28T22:09:59.104325+0000","last_clean":"2024-03-28T22:09:59.104325+0000","last_became_active":"2024-03-28T22:08:29.963872+0000","last_became_peered":"2024-03-28T22:08:29.963872+0000","last_unstale":"2024-03-28T22:09:59.104325+0000","last_undegraded":"2024-03-28T22:09:59.104325+0000","last_fullsized":"2024-03-28T22:09:59.104325+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:08.491078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.080889+0000","last_change":"2024-03-28T22:08:29.965564+0000","last_active":"2024-03-28T22:09:59.080889+0000","last_peered":"2024-03-28T22:09:59.080889+0000","last_clean":"2024-03-28T22:09:59.080889+0000","last_became_active":"2024-03-28T22:08:29.962636+0000","last_became_peered":"2024-03-28T22:08:29.962636+0000","last_unstale":"2024-03-28T22:09:59.080889+0000","last_undegraded":"2024-03-28T22:09:59.080889+0000","last_fullsized":"2024-03-28T22:09:59.080889+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:03:50.903231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.111439+0000","last_change":"2024-03-28T22:08:29.962091+0000","last_active":"2024-03-28T22:09:59.111439+0000","last_peered":"2024-03-28T22:09:59.111439+0000","last_clean":"2024-03-28T22:09:59.111439+0000","last_became_active":"2024-03-28T22:08:29.961735+0000","last_became_peered":"2024-03-28T22:08:29.961735+0000","last_unstale":"2024-03-28T22:09:59.111439+0000","last_undegraded":"2024-03-28T22:09:59.111439+0000","last_fullsized":"2024-03-28T22:09:59.111439+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:40:09.853265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.108285+0000","last_change":"2024-03-28T22:08:29.961089+0000","last_active":"2024-03-28T22:09:59.108285+0000","last_peered":"2024-03-28T22:09:59.108285+0000","last_clean":"2024-03-28T22:09:59.108285+0000","last_became_active":"2024-03-28T22:08:29.960711+0000","last_became_peered":"2024-03-28T22:08:29.960711+0000","last_unstale":"2024-03-28T22:09:59.108285+0000","last_undegraded":"2024-03-28T22:09:59.108285+0000","last_fullsized":"2024-03-28T22:09:59.108285+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:05:14.617718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.110495+0000","last_change":"2024-03-28T22:08:29.965142+0000","last_active":"2024-03-28T22:09:59.110495+0000","last_peered":"2024-03-28T22:09:59.110495+0000","last_clean":"2024-03-28T22:09:59.110495+0000","last_became_active":"2024-03-28T22:08:29.964730+0000","last_became_peered":"2024-03-28T22:08:29.964730+0000","last_unstale":"2024-03-28T22:09:59.110495+0000","last_undegraded":"2024-03-28T22:09:59.110495+0000","last_fullsized":"2024-03-28T22:09:59.110495+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:23:43.229681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.115178+0000","last_change":"2024-03-28T22:08:29.958984+0000","last_active":"2024-03-28T22:09:59.115178+0000","last_peered":"2024-03-28T22:09:59.115178+0000","last_clean":"2024-03-28T22:09:59.115178+0000","last_became_active":"2024-03-28T22:08:29.958583+0000","last_became_peered":"2024-03-28T22:08:29.958583+0000","last_unstale":"2024-03-28T22:09:59.115178+0000","last_undegraded":"2024-03-28T22:09:59.115178+0000","last_fullsized":"2024-03-28T22:09:59.115178+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:05:11.361719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.113771+0000","last_change":"2024-03-28T22:08:29.962774+0000","last_active":"2024-03-28T22:09:59.113771+0000","last_peered":"2024-03-28T22:09:59.113771+0000","last_clean":"2024-03-28T22:09:59.113771+0000","last_became_active":"2024-03-28T22:08:29.961222+0000","last_became_peered":"2024-03-28T22:08:29.961222+0000","last_unstale":"2024-03-28T22:09:59.113771+0000","last_undegraded":"2024-03-28T22:09:59.113771+0000","last_fullsized":"2024-03-28T22:09:59.113771+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:47:05.310483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.119577+0000","last_change":"2024-03-28T22:08:29.964041+0000","last_active":"2024-03-28T22:09:59.119577+0000","last_peered":"2024-03-28T22:09:59.119577+0000","last_clean":"2024-03-28T22:09:59.119577+0000","last_became_active":"2024-03-28T22:08:29.963360+0000","last_became_peered":"2024-03-28T22:08:29.963360+0000","last_unstale":"2024-03-28T22:09:59.119577+0000","last_undegraded":"2024-03-28T22:09:59.119577+0000","last_fullsized":"2024-03-28T22:09:59.119577+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:32.921916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.095082+0000","last_change":"2024-03-28T22:08:29.966449+0000","last_active":"2024-03-28T22:09:59.095082+0000","last_peered":"2024-03-28T22:09:59.095082+0000","last_clean":"2024-03-28T22:09:59.095082+0000","last_became_active":"2024-03-28T22:08:29.966036+0000","last_became_peered":"2024-03-28T22:08:29.966036+0000","last_unstale":"2024-03-28T22:09:59.095082+0000","last_undegraded":"2024-03-28T22:09:59.095082+0000","last_fullsized":"2024-03-28T22:09:59.095082+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:12:15.057713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.113074+0000","last_change":"2024-03-28T22:08:29.958924+0000","last_active":"2024-03-28T22:09:59.113074+0000","last_peered":"2024-03-28T22:09:59.113074+0000","last_clean":"2024-03-28T22:09:59.113074+0000","last_became_active":"2024-03-28T22:08:29.958601+0000","last_became_peered":"2024-03-28T22:08:29.958601+0000","last_unstale":"2024-03-28T22:09:59.113074+0000","last_undegraded":"2024-03-28T22:09:59.113074+0000","last_fullsized":"2024-03-28T22:09:59.113074+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:53:37.070276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.098999+0000","last_change":"2024-03-28T22:08:29.965045+0000","last_active":"2024-03-28T22:09:59.098999+0000","last_peered":"2024-03-28T22:09:59.098999+0000","last_clean":"2024-03-28T22:09:59.098999+0000","last_became_active":"2024-03-28T22:08:29.964710+0000","last_became_peered":"2024-03-28T22:08:29.964710+0000","last_unstale":"2024-03-28T22:09:59.098999+0000","last_undegraded":"2024-03-28T22:09:59.098999+0000","last_fullsized":"2024-03-28T22:09:59.098999+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:03:11.491213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.104261+0000","last_change":"2024-03-28T22:08:29.957940+0000","last_active":"2024-03-28T22:09:59.104261+0000","last_peered":"2024-03-28T22:09:59.104261+0000","last_clean":"2024-03-28T22:09:59.104261+0000","last_became_active":"2024-03-28T22:08:29.957587+0000","last_became_peered":"2024-03-28T22:08:29.957587+0000","last_unstale":"2024-03-28T22:09:59.104261+0000","last_undegraded":"2024-03-28T22:09:59.104261+0000","last_fullsized":"2024-03-28T22:09:59.104261+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:03:58.557806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.120996+0000","last_change":"2024-03-28T22:08:29.957397+0000","last_active":"2024-03-28T22:09:59.120996+0000","last_peered":"2024-03-28T22:09:59.120996+0000","last_clean":"2024-03-28T22:09:59.120996+0000","last_became_active":"2024-03-28T22:08:29.957054+0000","last_became_peered":"2024-03-28T22:08:29.957054+0000","last_unstale":"2024-03-28T22:09:59.120996+0000","last_undegraded":"2024-03-28T22:09:59.120996+0000","last_fullsized":"2024-03-28T22:09:59.120996+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:35:28.500561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.119267+0000","last_change":"2024-03-28T22:08:29.963343+0000","last_active":"2024-03-28T22:09:59.119267+0000","last_peered":"2024-03-28T22:09:59.119267+0000","last_clean":"2024-03-28T22:09:59.119267+0000","last_became_active":"2024-03-28T22:08:29.962926+0000","last_became_peered":"2024-03-28T22:08:29.962926+0000","last_unstale":"2024-03-28T22:09:59.119267+0000","last_undegraded":"2024-03-28T22:09:59.119267+0000","last_fullsized":"2024-03-28T22:09:59.119267+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:15:39.133930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.094891+0000","last_change":"2024-03-28T22:08:29.955903+0000","last_active":"2024-03-28T22:09:59.094891+0000","last_peered":"2024-03-28T22:09:59.094891+0000","last_clean":"2024-03-28T22:09:59.094891+0000","last_became_active":"2024-03-28T22:08:29.955460+0000","last_became_peered":"2024-03-28T22:08:29.955460+0000","last_unstale":"2024-03-28T22:09:59.094891+0000","last_undegraded":"2024-03-28T22:09:59.094891+0000","last_fullsized":"2024-03-28T22:09:59.094891+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:36:12.961475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.110397+0000","last_change":"2024-03-28T22:08:29.954279+0000","last_active":"2024-03-28T22:09:59.110397+0000","last_peered":"2024-03-28T22:09:59.110397+0000","last_clean":"2024-03-28T22:09:59.110397+0000","last_became_active":"2024-03-28T22:08:29.953869+0000","last_became_peered":"2024-03-28T22:08:29.953869+0000","last_unstale":"2024-03-28T22:09:59.110397+0000","last_undegraded":"2024-03-28T22:09:59.110397+0000","last_fullsized":"2024-03-28T22:09:59.110397+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:05:52.247391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.054735+0000","last_change":"2024-03-28T22:08:29.327644+0000","last_active":"2024-03-28T22:09:30.054735+0000","last_peered":"2024-03-28T22:09:30.054735+0000","last_clean":"2024-03-28T22:09:30.054735+0000","last_became_active":"2024-03-28T22:08:29.327325+0000","last_became_peered":"2024-03-28T22:08:29.327325+0000","last_unstale":"2024-03-28T22:09:30.054735+0000","last_undegraded":"2024-03-28T22:09:30.054735+0000","last_fullsized":"2024-03-28T22:09:30.054735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:26:54.782703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.102346+0000","last_change":"2024-03-28T22:08:29.962905+0000","last_active":"2024-03-28T22:09:59.102346+0000","last_peered":"2024-03-28T22:09:59.102346+0000","last_clean":"2024-03-28T22:09:59.102346+0000","last_became_active":"2024-03-28T22:08:29.962559+0000","last_became_peered":"2024-03-28T22:08:29.962559+0000","last_unstale":"2024-03-28T22:09:59.102346+0000","last_undegraded":"2024-03-28T22:09:59.102346+0000","last_fullsized":"2024-03-28T22:09:59.102346+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:02.347584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058126+0000","last_change":"2024-03-28T22:08:29.334871+0000","last_active":"2024-03-28T22:09:30.058126+0000","last_peered":"2024-03-28T22:09:30.058126+0000","last_clean":"2024-03-28T22:09:30.058126+0000","last_became_active":"2024-03-28T22:08:29.334565+0000","last_became_peered":"2024-03-28T22:08:29.334565+0000","last_unstale":"2024-03-28T22:09:30.058126+0000","last_undegraded":"2024-03-28T22:09:30.058126+0000","last_fullsized":"2024-03-28T22:09:30.058126+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:31:19.333496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.119365+0000","last_change":"2024-03-28T22:08:29.957622+0000","last_active":"2024-03-28T22:09:59.119365+0000","last_peered":"2024-03-28T22:09:59.119365+0000","last_clean":"2024-03-28T22:09:59.119365+0000","last_became_active":"2024-03-28T22:08:29.957208+0000","last_became_peered":"2024-03-28T22:08:29.957208+0000","last_unstale":"2024-03-28T22:09:59.119365+0000","last_undegraded":"2024-03-28T22:09:59.119365+0000","last_fullsized":"2024-03-28T22:09:59.119365+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:49:44.684203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058576+0000","last_change":"2024-03-28T22:08:29.330291+0000","last_active":"2024-03-28T22:09:30.058576+0000","last_peered":"2024-03-28T22:09:30.058576+0000","last_clean":"2024-03-28T22:09:30.058576+0000","last_became_active":"2024-03-28T22:08:29.328955+0000","last_became_peered":"2024-03-28T22:08:29.328955+0000","last_unstale":"2024-03-28T22:09:30.058576+0000","last_undegraded":"2024-03-28T22:09:30.058576+0000","last_fullsized":"2024-03-28T22:09:30.058576+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:10.838059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.116812+0000","last_change":"2024-03-28T22:08:29.966681+0000","last_active":"2024-03-28T22:09:59.116812+0000","last_peered":"2024-03-28T22:09:59.116812+0000","last_clean":"2024-03-28T22:09:59.116812+0000","last_became_active":"2024-03-28T22:08:29.966023+0000","last_became_peered":"2024-03-28T22:08:29.966023+0000","last_unstale":"2024-03-28T22:09:59.116812+0000","last_undegraded":"2024-03-28T22:09:59.116812+0000","last_fullsized":"2024-03-28T22:09:59.116812+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:04:43.281259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058443+0000","last_change":"2024-03-28T22:08:29.329026+0000","last_active":"2024-03-28T22:09:30.058443+0000","last_peered":"2024-03-28T22:09:30.058443+0000","last_clean":"2024-03-28T22:09:30.058443+0000","last_became_active":"2024-03-28T22:08:29.327570+0000","last_became_peered":"2024-03-28T22:08:29.327570+0000","last_unstale":"2024-03-28T22:09:30.058443+0000","last_undegraded":"2024-03-28T22:09:30.058443+0000","last_fullsized":"2024-03-28T22:09:30.058443+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:53:54.779073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.106272+0000","last_change":"2024-03-28T22:08:29.966455+0000","last_active":"2024-03-28T22:09:59.106272+0000","last_peered":"2024-03-28T22:09:59.106272+0000","last_clean":"2024-03-28T22:09:59.106272+0000","last_became_active":"2024-03-28T22:08:29.964451+0000","last_became_peered":"2024-03-28T22:08:29.964451+0000","last_unstale":"2024-03-28T22:09:59.106272+0000","last_undegraded":"2024-03-28T22:09:59.106272+0000","last_fullsized":"2024-03-28T22:09:59.106272+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:46:22.811146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'8","reported_seq":138,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:10:20.499412+0000","last_change":"2024-03-28T22:08:29.327397+0000","last_active":"2024-03-28T22:10:20.499412+0000","last_peered":"2024-03-28T22:10:20.499412+0000","last_clean":"2024-03-28T22:10:20.499412+0000","last_became_active":"2024-03-28T22:08:29.327009+0000","last_became_peered":"2024-03-28T22:08:29.327009+0000","last_unstale":"2024-03-28T22:10:20.499412+0000","last_undegraded":"2024-03-28T22:10:20.499412+0000","last_fullsized":"2024-03-28T22:10:20.499412+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:38:51.951934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1545,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":111,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.096251+0000","last_change":"2024-03-28T22:08:29.969821+0000","last_active":"2024-03-28T22:09:59.096251+0000","last_peered":"2024-03-28T22:09:59.096251+0000","last_clean":"2024-03-28T22:09:59.096251+0000","last_became_active":"2024-03-28T22:08:29.969386+0000","last_became_peered":"2024-03-28T22:08:29.969386+0000","last_unstale":"2024-03-28T22:09:59.096251+0000","last_undegraded":"2024-03-28T22:09:59.096251+0000","last_fullsized":"2024-03-28T22:09:59.096251+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:44:19.753083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.424985+0000","last_change":"2024-03-28T22:08:29.332385+0000","last_active":"2024-03-28T22:09:30.424985+0000","last_peered":"2024-03-28T22:09:30.424985+0000","last_clean":"2024-03-28T22:09:30.424985+0000","last_became_active":"2024-03-28T22:08:29.331956+0000","last_became_peered":"2024-03-28T22:08:29.331956+0000","last_unstale":"2024-03-28T22:09:30.424985+0000","last_undegraded":"2024-03-28T22:09:30.424985+0000","last_fullsized":"2024-03-28T22:09:30.424985+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:10:04.428034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.092555+0000","last_change":"2024-03-28T22:09:30.080320+0000","last_active":"2024-03-28T22:09:59.092555+0000","last_peered":"2024-03-28T22:09:59.092555+0000","last_clean":"2024-03-28T22:09:59.092555+0000","last_became_active":"2024-03-28T22:09:30.079928+0000","last_became_peered":"2024-03-28T22:09:30.079928+0000","last_unstale":"2024-03-28T22:09:59.092555+0000","last_undegraded":"2024-03-28T22:09:59.092555+0000","last_fullsized":"2024-03-28T22:09:59.092555+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:12:53.385167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.293420+0000","last_change":"2024-03-28T22:08:29.327180+0000","last_active":"2024-03-28T22:09:30.293420+0000","last_peered":"2024-03-28T22:09:30.293420+0000","last_clean":"2024-03-28T22:09:30.293420+0000","last_became_active":"2024-03-28T22:08:29.326825+0000","last_became_peered":"2024-03-28T22:08:29.326825+0000","last_unstale":"2024-03-28T22:09:30.293420+0000","last_undegraded":"2024-03-28T22:09:30.293420+0000","last_fullsized":"2024-03-28T22:09:30.293420+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:11:09.979970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.098656+0000","last_change":"2024-03-28T22:08:29.955299+0000","last_active":"2024-03-28T22:09:59.098656+0000","last_peered":"2024-03-28T22:09:59.098656+0000","last_clean":"2024-03-28T22:09:59.098656+0000","last_became_active":"2024-03-28T22:08:29.954949+0000","last_became_peered":"2024-03-28T22:08:29.954949+0000","last_unstale":"2024-03-28T22:09:59.098656+0000","last_undegraded":"2024-03-28T22:09:59.098656+0000","last_fullsized":"2024-03-28T22:09:59.098656+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:01:48.375251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.059530+0000","last_change":"2024-03-28T22:08:29.327261+0000","last_active":"2024-03-28T22:09:30.059530+0000","last_peered":"2024-03-28T22:09:30.059530+0000","last_clean":"2024-03-28T22:09:30.059530+0000","last_became_active":"2024-03-28T22:08:29.326910+0000","last_became_peered":"2024-03-28T22:08:29.326910+0000","last_unstale":"2024-03-28T22:09:30.059530+0000","last_undegraded":"2024-03-28T22:09:30.059530+0000","last_fullsized":"2024-03-28T22:09:30.059530+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:46:41.933372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.093770+0000","last_change":"2024-03-28T22:08:29.954891+0000","last_active":"2024-03-28T22:09:59.093770+0000","last_peered":"2024-03-28T22:09:59.093770+0000","last_clean":"2024-03-28T22:09:59.093770+0000","last_became_active":"2024-03-28T22:08:29.954519+0000","last_became_peered":"2024-03-28T22:08:29.954519+0000","last_unstale":"2024-03-28T22:09:59.093770+0000","last_undegraded":"2024-03-28T22:09:59.093770+0000","last_fullsized":"2024-03-28T22:09:59.093770+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:07:42.621189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":129,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:10:18.495749+0000","last_change":"2024-03-28T22:08:29.330180+0000","last_active":"2024-03-28T22:10:18.495749+0000","last_peered":"2024-03-28T22:10:18.495749+0000","last_clean":"2024-03-28T22:10:18.495749+0000","last_became_active":"2024-03-28T22:08:29.329835+0000","last_became_peered":"2024-03-28T22:08:29.329835+0000","last_unstale":"2024-03-28T22:10:18.495749+0000","last_undegraded":"2024-03-28T22:10:18.495749+0000","last_fullsized":"2024-03-28T22:10:18.495749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:48:39.906359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":106,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.118105+0000","last_change":"2024-03-28T22:08:29.967074+0000","last_active":"2024-03-28T22:09:59.118105+0000","last_peered":"2024-03-28T22:09:59.118105+0000","last_clean":"2024-03-28T22:09:59.118105+0000","last_became_active":"2024-03-28T22:08:29.966258+0000","last_became_peered":"2024-03-28T22:08:29.966258+0000","last_unstale":"2024-03-28T22:09:59.118105+0000","last_undegraded":"2024-03-28T22:09:59.118105+0000","last_fullsized":"2024-03-28T22:09:59.118105+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:25:44.986179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.055474+0000","last_change":"2024-03-28T22:08:29.326388+0000","last_active":"2024-03-28T22:09:30.055474+0000","last_peered":"2024-03-28T22:09:30.055474+0000","last_clean":"2024-03-28T22:09:30.055474+0000","last_became_active":"2024-03-28T22:08:29.326001+0000","last_became_peered":"2024-03-28T22:08:29.326001+0000","last_unstale":"2024-03-28T22:09:30.055474+0000","last_undegraded":"2024-03-28T22:09:30.055474+0000","last_fullsized":"2024-03-28T22:09:30.055474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:31:56.227038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.104748+0000","last_change":"2024-03-28T22:08:29.959299+0000","last_active":"2024-03-28T22:09:59.104748+0000","last_peered":"2024-03-28T22:09:59.104748+0000","last_clean":"2024-03-28T22:09:59.104748+0000","last_became_active":"2024-03-28T22:08:29.958955+0000","last_became_peered":"2024-03-28T22:08:29.958955+0000","last_unstale":"2024-03-28T22:09:59.104748+0000","last_undegraded":"2024-03-28T22:09:59.104748+0000","last_fullsized":"2024-03-28T22:09:59.104748+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:11:06.775920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.060567+0000","last_change":"2024-03-28T22:08:29.326089+0000","last_active":"2024-03-28T22:09:30.060567+0000","last_peered":"2024-03-28T22:09:30.060567+0000","last_clean":"2024-03-28T22:09:30.060567+0000","last_became_active":"2024-03-28T22:08:29.325463+0000","last_became_peered":"2024-03-28T22:08:29.325463+0000","last_unstale":"2024-03-28T22:09:30.060567+0000","last_undegraded":"2024-03-28T22:09:30.060567+0000","last_fullsized":"2024-03-28T22:09:30.060567+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:18:31.049729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.118631+0000","last_change":"2024-03-28T22:08:29.964248+0000","last_active":"2024-03-28T22:09:59.118631+0000","last_peered":"2024-03-28T22:09:59.118631+0000","last_clean":"2024-03-28T22:09:59.118631+0000","last_became_active":"2024-03-28T22:08:29.963897+0000","last_became_peered":"2024-03-28T22:08:29.963897+0000","last_unstale":"2024-03-28T22:09:59.118631+0000","last_undegraded":"2024-03-28T22:09:59.118631+0000","last_fullsized":"2024-03-28T22:09:59.118631+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:41:39.645360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'31","reported_seq":271,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:10:19.497584+0000","last_change":"2024-03-28T22:08:29.330732+0000","last_active":"2024-03-28T22:10:19.497584+0000","last_peered":"2024-03-28T22:10:19.497584+0000","last_clean":"2024-03-28T22:10:19.497584+0000","last_became_active":"2024-03-28T22:08:29.329231+0000","last_became_peered":"2024-03-28T22:08:29.329231+0000","last_unstale":"2024-03-28T22:10:19.497584+0000","last_undegraded":"2024-03-28T22:10:19.497584+0000","last_fullsized":"2024-03-28T22:10:19.497584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:16:39.260429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59469,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":221,"num_read_kb":168,"num_write":32,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.109344+0000","last_change":"2024-03-28T22:08:29.965791+0000","last_active":"2024-03-28T22:09:59.109344+0000","last_peered":"2024-03-28T22:09:59.109344+0000","last_clean":"2024-03-28T22:09:59.109344+0000","last_became_active":"2024-03-28T22:08:29.965390+0000","last_became_peered":"2024-03-28T22:08:29.965390+0000","last_unstale":"2024-03-28T22:09:59.109344+0000","last_undegraded":"2024-03-28T22:09:59.109344+0000","last_fullsized":"2024-03-28T22:09:59.109344+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:43:53.325906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.057898+0000","last_change":"2024-03-28T22:08:29.325851+0000","last_active":"2024-03-28T22:09:30.057898+0000","last_peered":"2024-03-28T22:09:30.057898+0000","last_clean":"2024-03-28T22:09:30.057898+0000","last_became_active":"2024-03-28T22:08:29.325500+0000","last_became_peered":"2024-03-28T22:08:29.325500+0000","last_unstale":"2024-03-28T22:09:30.057898+0000","last_undegraded":"2024-03-28T22:09:30.057898+0000","last_fullsized":"2024-03-28T22:09:30.057898+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:04:25.151908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.115301+0000","last_change":"2024-03-28T22:08:29.953857+0000","last_active":"2024-03-28T22:09:59.115301+0000","last_peered":"2024-03-28T22:09:59.115301+0000","last_clean":"2024-03-28T22:09:59.115301+0000","last_became_active":"2024-03-28T22:08:29.953538+0000","last_became_peered":"2024-03-28T22:08:29.953538+0000","last_unstale":"2024-03-28T22:09:59.115301+0000","last_undegraded":"2024-03-28T22:09:59.115301+0000","last_fullsized":"2024-03-28T22:09:59.115301+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:40:32.404760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.060057+0000","last_change":"2024-03-28T22:08:29.327286+0000","last_active":"2024-03-28T22:09:30.060057+0000","last_peered":"2024-03-28T22:09:30.060057+0000","last_clean":"2024-03-28T22:09:30.060057+0000","last_became_active":"2024-03-28T22:08:29.326902+0000","last_became_peered":"2024-03-28T22:08:29.326902+0000","last_unstale":"2024-03-28T22:09:30.060057+0000","last_undegraded":"2024-03-28T22:09:30.060057+0000","last_fullsized":"2024-03-28T22:09:30.060057+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:14:58.669534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.116496+0000","last_change":"2024-03-28T22:09:30.294946+0000","last_active":"2024-03-28T22:09:59.116496+0000","last_peered":"2024-03-28T22:09:59.116496+0000","last_clean":"2024-03-28T22:09:59.116496+0000","last_became_active":"2024-03-28T22:09:30.294496+0000","last_became_peered":"2024-03-28T22:09:30.294496+0000","last_unstale":"2024-03-28T22:09:59.116496+0000","last_undegraded":"2024-03-28T22:09:59.116496+0000","last_fullsized":"2024-03-28T22:09:59.116496+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:26:40.062384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'6","reported_seq":138,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:10:21.500594+0000","last_change":"2024-03-28T22:08:29.327740+0000","last_active":"2024-03-28T22:10:21.500594+0000","last_peered":"2024-03-28T22:10:21.500594+0000","last_clean":"2024-03-28T22:10:21.500594+0000","last_became_active":"2024-03-28T22:08:29.326664+0000","last_became_peered":"2024-03-28T22:08:29.326664+0000","last_unstale":"2024-03-28T22:10:21.500594+0000","last_undegraded":"2024-03-28T22:10:21.500594+0000","last_fullsized":"2024-03-28T22:10:21.500594+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:58:17.681836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1511,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":114,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.104158+0000","last_change":"2024-03-28T22:08:29.961979+0000","last_active":"2024-03-28T22:09:59.104158+0000","last_peered":"2024-03-28T22:09:59.104158+0000","last_clean":"2024-03-28T22:09:59.104158+0000","last_became_active":"2024-03-28T22:08:29.961615+0000","last_became_peered":"2024-03-28T22:08:29.961615+0000","last_unstale":"2024-03-28T22:09:59.104158+0000","last_undegraded":"2024-03-28T22:09:59.104158+0000","last_fullsized":"2024-03-28T22:09:59.104158+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:50:37.227632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.057515+0000","last_change":"2024-03-28T22:08:29.329017+0000","last_active":"2024-03-28T22:09:30.057515+0000","last_peered":"2024-03-28T22:09:30.057515+0000","last_clean":"2024-03-28T22:09:30.057515+0000","last_became_active":"2024-03-28T22:08:29.328686+0000","last_became_peered":"2024-03-28T22:08:29.328686+0000","last_unstale":"2024-03-28T22:09:30.057515+0000","last_undegraded":"2024-03-28T22:09:30.057515+0000","last_fullsized":"2024-03-28T22:09:30.057515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:25:28.986165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.102273+0000","last_change":"2024-03-28T22:09:30.426320+0000","last_active":"2024-03-28T22:09:59.102273+0000","last_peered":"2024-03-28T22:09:59.102273+0000","last_clean":"2024-03-28T22:09:59.102273+0000","last_became_active":"2024-03-28T22:09:30.425862+0000","last_became_peered":"2024-03-28T22:09:30.425862+0000","last_unstale":"2024-03-28T22:09:59.102273+0000","last_undegraded":"2024-03-28T22:09:59.102273+0000","last_fullsized":"2024-03-28T22:09:59.102273+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:44.240551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058066+0000","last_change":"2024-03-28T22:08:29.323274+0000","last_active":"2024-03-28T22:09:30.058066+0000","last_peered":"2024-03-28T22:09:30.058066+0000","last_clean":"2024-03-28T22:09:30.058066+0000","last_became_active":"2024-03-28T22:08:29.322911+0000","last_became_peered":"2024-03-28T22:08:29.322911+0000","last_unstale":"2024-03-28T22:09:30.058066+0000","last_undegraded":"2024-03-28T22:09:30.058066+0000","last_fullsized":"2024-03-28T22:09:30.058066+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:44:14.439411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.107898+0000","last_change":"2024-03-28T22:08:29.961101+0000","last_active":"2024-03-28T22:09:59.107898+0000","last_peered":"2024-03-28T22:09:59.107898+0000","last_clean":"2024-03-28T22:09:59.107898+0000","last_became_active":"2024-03-28T22:08:29.960767+0000","last_became_peered":"2024-03-28T22:08:29.960767+0000","last_unstale":"2024-03-28T22:09:59.107898+0000","last_undegraded":"2024-03-28T22:09:59.107898+0000","last_fullsized":"2024-03-28T22:09:59.107898+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:33:45.236691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.060215+0000","last_change":"2024-03-28T22:09:30.060215+0000","last_active":"2024-03-28T22:09:30.060215+0000","last_peered":"2024-03-28T22:09:30.060215+0000","last_clean":"2024-03-28T22:09:30.060215+0000","last_became_active":"2024-03-28T22:09:30.059828+0000","last_became_peered":"2024-03-28T22:09:30.059828+0000","last_unstale":"2024-03-28T22:09:30.060215+0000","last_undegraded":"2024-03-28T22:09:30.060215+0000","last_fullsized":"2024-03-28T22:09:30.060215+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:42:05.087899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.112511+0000","last_change":"2024-03-28T22:08:29.954093+0000","last_active":"2024-03-28T22:09:59.112511+0000","last_peered":"2024-03-28T22:09:59.112511+0000","last_clean":"2024-03-28T22:09:59.112511+0000","last_became_active":"2024-03-28T22:08:29.953688+0000","last_became_peered":"2024-03-28T22:08:29.953688+0000","last_unstale":"2024-03-28T22:09:59.112511+0000","last_undegraded":"2024-03-28T22:09:59.112511+0000","last_fullsized":"2024-03-28T22:09:59.112511+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:57:51.026565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.055115+0000","last_change":"2024-03-28T22:08:29.331104+0000","last_active":"2024-03-28T22:09:30.055115+0000","last_peered":"2024-03-28T22:09:30.055115+0000","last_clean":"2024-03-28T22:09:30.055115+0000","last_became_active":"2024-03-28T22:08:29.330505+0000","last_became_peered":"2024-03-28T22:08:29.330505+0000","last_unstale":"2024-03-28T22:09:30.055115+0000","last_undegraded":"2024-03-28T22:09:30.055115+0000","last_fullsized":"2024-03-28T22:09:30.055115+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:36:59.944587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.110591+0000","last_change":"2024-03-28T22:08:29.963540+0000","last_active":"2024-03-28T22:09:59.110591+0000","last_peered":"2024-03-28T22:09:59.110591+0000","last_clean":"2024-03-28T22:09:59.110591+0000","last_became_active":"2024-03-28T22:08:29.963206+0000","last_became_peered":"2024-03-28T22:08:29.963206+0000","last_unstale":"2024-03-28T22:09:59.110591+0000","last_undegraded":"2024-03-28T22:09:59.110591+0000","last_fullsized":"2024-03-28T22:09:59.110591+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:55:53.292554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.098046+0000","last_change":"2024-03-28T22:08:29.962822+0000","last_active":"2024-03-28T22:09:59.098046+0000","last_peered":"2024-03-28T22:09:59.098046+0000","last_clean":"2024-03-28T22:09:59.098046+0000","last_became_active":"2024-03-28T22:08:29.962343+0000","last_became_peered":"2024-03-28T22:08:29.962343+0000","last_unstale":"2024-03-28T22:09:59.098046+0000","last_undegraded":"2024-03-28T22:09:59.098046+0000","last_fullsized":"2024-03-28T22:09:59.098046+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:36.608543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'6","reported_seq":132,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:10:20.498618+0000","last_change":"2024-03-28T22:08:29.328239+0000","last_active":"2024-03-28T22:10:20.498618+0000","last_peered":"2024-03-28T22:10:20.498618+0000","last_clean":"2024-03-28T22:10:20.498618+0000","last_became_active":"2024-03-28T22:08:29.327903+0000","last_became_peered":"2024-03-28T22:08:29.327903+0000","last_unstale":"2024-03-28T22:10:20.498618+0000","last_undegraded":"2024-03-28T22:10:20.498618+0000","last_fullsized":"2024-03-28T22:10:20.498618+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:14:17.558472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":107,"num_read_kb":107,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.085982+0000","last_change":"2024-03-28T22:09:30.426683+0000","last_active":"2024-03-28T22:09:59.085982+0000","last_peered":"2024-03-28T22:09:59.085982+0000","last_clean":"2024-03-28T22:09:59.085982+0000","last_became_active":"2024-03-28T22:09:30.426255+0000","last_became_peered":"2024-03-28T22:09:30.426255+0000","last_unstale":"2024-03-28T22:09:59.085982+0000","last_undegraded":"2024-03-28T22:09:59.085982+0000","last_fullsized":"2024-03-28T22:09:59.085982+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:36.755978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.079359+0000","last_change":"2024-03-28T22:08:29.323050+0000","last_active":"2024-03-28T22:09:30.079359+0000","last_peered":"2024-03-28T22:09:30.079359+0000","last_clean":"2024-03-28T22:09:30.079359+0000","last_became_active":"2024-03-28T22:08:29.322684+0000","last_became_peered":"2024-03-28T22:08:29.322684+0000","last_unstale":"2024-03-28T22:09:30.079359+0000","last_undegraded":"2024-03-28T22:09:30.079359+0000","last_fullsized":"2024-03-28T22:09:30.079359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:14:43.811122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":354,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.054922+0000","last_change":"2024-03-28T22:07:17.785516+0000","last_active":"2024-03-28T22:09:30.054922+0000","last_peered":"2024-03-28T22:09:30.054922+0000","last_clean":"2024-03-28T22:09:30.054922+0000","last_became_active":"2024-03-28T22:07:17.785123+0000","last_became_peered":"2024-03-28T22:07:17.785123+0000","last_unstale":"2024-03-28T22:09:30.054922+0000","last_undegraded":"2024-03-28T22:09:30.054922+0000","last_fullsized":"2024-03-28T22:09:30.054922+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:03:59.840513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:03:59.840513+0000","last_clean_scrub_stamp":"2024-03-28T22:03:59.840513+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:31:06.253632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.108076+0000","last_change":"2024-03-28T22:08:29.961489+0000","last_active":"2024-03-28T22:09:59.108076+0000","last_peered":"2024-03-28T22:09:59.108076+0000","last_clean":"2024-03-28T22:09:59.108076+0000","last_became_active":"2024-03-28T22:08:29.960881+0000","last_became_peered":"2024-03-28T22:08:29.960881+0000","last_unstale":"2024-03-28T22:09:59.108076+0000","last_undegraded":"2024-03-28T22:09:59.108076+0000","last_fullsized":"2024-03-28T22:09:59.108076+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:48.870834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.054972+0000","last_change":"2024-03-28T22:08:29.331994+0000","last_active":"2024-03-28T22:09:30.054972+0000","last_peered":"2024-03-28T22:09:30.054972+0000","last_clean":"2024-03-28T22:09:30.054972+0000","last_became_active":"2024-03-28T22:08:29.331630+0000","last_became_peered":"2024-03-28T22:08:29.331630+0000","last_unstale":"2024-03-28T22:09:30.054972+0000","last_undegraded":"2024-03-28T22:09:30.054972+0000","last_fullsized":"2024-03-28T22:09:30.054972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:11:37.100405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058466+0000","last_change":"2024-03-28T22:08:29.330353+0000","last_active":"2024-03-28T22:09:30.058466+0000","last_peered":"2024-03-28T22:09:30.058466+0000","last_clean":"2024-03-28T22:09:30.058466+0000","last_became_active":"2024-03-28T22:08:29.329421+0000","last_became_peered":"2024-03-28T22:08:29.329421+0000","last_unstale":"2024-03-28T22:09:30.058466+0000","last_undegraded":"2024-03-28T22:09:30.058466+0000","last_fullsized":"2024-03-28T22:09:30.058466+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:56.166506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.108762+0000","last_change":"2024-03-28T22:08:29.948595+0000","last_active":"2024-03-28T22:09:59.108762+0000","last_peered":"2024-03-28T22:09:59.108762+0000","last_clean":"2024-03-28T22:09:59.108762+0000","last_became_active":"2024-03-28T22:08:29.946986+0000","last_became_peered":"2024-03-28T22:08:29.946986+0000","last_unstale":"2024-03-28T22:09:59.108762+0000","last_undegraded":"2024-03-28T22:09:59.108762+0000","last_fullsized":"2024-03-28T22:09:59.108762+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:56:12.351731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.328434+0000","last_change":"2024-03-28T22:08:29.333375+0000","last_active":"2024-03-28T22:09:59.328434+0000","last_peered":"2024-03-28T22:09:59.328434+0000","last_clean":"2024-03-28T22:09:59.328434+0000","last_became_active":"2024-03-28T22:08:29.331594+0000","last_became_peered":"2024-03-28T22:08:29.331594+0000","last_unstale":"2024-03-28T22:09:59.328434+0000","last_undegraded":"2024-03-28T22:09:59.328434+0000","last_fullsized":"2024-03-28T22:09:59.328434+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:07:43.936807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.109527+0000","last_change":"2024-03-28T22:08:29.959061+0000","last_active":"2024-03-28T22:09:59.109527+0000","last_peered":"2024-03-28T22:09:59.109527+0000","last_clean":"2024-03-28T22:09:59.109527+0000","last_became_active":"2024-03-28T22:08:29.958723+0000","last_became_peered":"2024-03-28T22:08:29.958723+0000","last_unstale":"2024-03-28T22:09:59.109527+0000","last_undegraded":"2024-03-28T22:09:59.109527+0000","last_fullsized":"2024-03-28T22:09:59.109527+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:32:23.016606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.079461+0000","last_change":"2024-03-28T22:08:29.329886+0000","last_active":"2024-03-28T22:09:30.079461+0000","last_peered":"2024-03-28T22:09:30.079461+0000","last_clean":"2024-03-28T22:09:30.079461+0000","last_became_active":"2024-03-28T22:08:29.329503+0000","last_became_peered":"2024-03-28T22:08:29.329503+0000","last_unstale":"2024-03-28T22:09:30.079461+0000","last_undegraded":"2024-03-28T22:09:30.079461+0000","last_fullsized":"2024-03-28T22:09:30.079461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:50:05.686647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.102695+0000","last_change":"2024-03-28T22:08:29.957038+0000","last_active":"2024-03-28T22:09:59.102695+0000","last_peered":"2024-03-28T22:09:59.102695+0000","last_clean":"2024-03-28T22:09:59.102695+0000","last_became_active":"2024-03-28T22:08:29.956693+0000","last_became_peered":"2024-03-28T22:08:29.956693+0000","last_unstale":"2024-03-28T22:09:59.102695+0000","last_undegraded":"2024-03-28T22:09:59.102695+0000","last_fullsized":"2024-03-28T22:09:59.102695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:19:09.881431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.424647+0000","last_change":"2024-03-28T22:08:29.327356+0000","last_active":"2024-03-28T22:09:30.424647+0000","last_peered":"2024-03-28T22:09:30.424647+0000","last_clean":"2024-03-28T22:09:30.424647+0000","last_became_active":"2024-03-28T22:08:29.326947+0000","last_became_peered":"2024-03-28T22:08:29.326947+0000","last_unstale":"2024-03-28T22:09:30.424647+0000","last_undegraded":"2024-03-28T22:09:30.424647+0000","last_fullsized":"2024-03-28T22:09:30.424647+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:47:29.417465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.064033+0000","last_change":"2024-03-28T22:08:29.966505+0000","last_active":"2024-03-28T22:09:59.064033+0000","last_peered":"2024-03-28T22:09:59.064033+0000","last_clean":"2024-03-28T22:09:59.064033+0000","last_became_active":"2024-03-28T22:08:29.966057+0000","last_became_peered":"2024-03-28T22:08:29.966057+0000","last_unstale":"2024-03-28T22:09:59.064033+0000","last_undegraded":"2024-03-28T22:09:59.064033+0000","last_fullsized":"2024-03-28T22:09:59.064033+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:17:45.196585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.057477+0000","last_change":"2024-03-28T22:08:29.328897+0000","last_active":"2024-03-28T22:09:30.057477+0000","last_peered":"2024-03-28T22:09:30.057477+0000","last_clean":"2024-03-28T22:09:30.057477+0000","last_became_active":"2024-03-28T22:08:29.328390+0000","last_became_peered":"2024-03-28T22:08:29.328390+0000","last_unstale":"2024-03-28T22:09:30.057477+0000","last_undegraded":"2024-03-28T22:09:30.057477+0000","last_fullsized":"2024-03-28T22:09:30.057477+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:21:12.699503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.094866+0000","last_change":"2024-03-28T22:08:29.957212+0000","last_active":"2024-03-28T22:09:59.094866+0000","last_peered":"2024-03-28T22:09:59.094866+0000","last_clean":"2024-03-28T22:09:59.094866+0000","last_became_active":"2024-03-28T22:08:29.956215+0000","last_became_peered":"2024-03-28T22:08:29.956215+0000","last_unstale":"2024-03-28T22:09:59.094866+0000","last_undegraded":"2024-03-28T22:09:59.094866+0000","last_fullsized":"2024-03-28T22:09:59.094866+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:57:25.199993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.063151+0000","last_change":"2024-03-28T22:09:30.062742+0000","last_active":"2024-03-28T22:09:30.063151+0000","last_peered":"2024-03-28T22:09:30.063151+0000","last_clean":"2024-03-28T22:09:30.063151+0000","last_became_active":"2024-03-28T22:09:30.059356+0000","last_became_peered":"2024-03-28T22:09:30.059356+0000","last_unstale":"2024-03-28T22:09:30.063151+0000","last_undegraded":"2024-03-28T22:09:30.063151+0000","last_fullsized":"2024-03-28T22:09:30.063151+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:52:39.412153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.110714+0000","last_change":"2024-03-28T22:08:29.955023+0000","last_active":"2024-03-28T22:09:59.110714+0000","last_peered":"2024-03-28T22:09:59.110714+0000","last_clean":"2024-03-28T22:09:59.110714+0000","last_became_active":"2024-03-28T22:08:29.954675+0000","last_became_peered":"2024-03-28T22:08:29.954675+0000","last_unstale":"2024-03-28T22:09:59.110714+0000","last_undegraded":"2024-03-28T22:09:59.110714+0000","last_fullsized":"2024-03-28T22:09:59.110714+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:31.246452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058628+0000","last_change":"2024-03-28T22:08:29.329280+0000","last_active":"2024-03-28T22:09:30.058628+0000","last_peered":"2024-03-28T22:09:30.058628+0000","last_clean":"2024-03-28T22:09:30.058628+0000","last_became_active":"2024-03-28T22:08:29.328860+0000","last_became_peered":"2024-03-28T22:08:29.328860+0000","last_unstale":"2024-03-28T22:09:30.058628+0000","last_undegraded":"2024-03-28T22:09:30.058628+0000","last_fullsized":"2024-03-28T22:09:30.058628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:15:49.610026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.118448+0000","last_change":"2024-03-28T22:08:29.966224+0000","last_active":"2024-03-28T22:09:59.118448+0000","last_peered":"2024-03-28T22:09:59.118448+0000","last_clean":"2024-03-28T22:09:59.118448+0000","last_became_active":"2024-03-28T22:08:29.965636+0000","last_became_peered":"2024-03-28T22:08:29.965636+0000","last_unstale":"2024-03-28T22:09:59.118448+0000","last_undegraded":"2024-03-28T22:09:59.118448+0000","last_fullsized":"2024-03-28T22:09:59.118448+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:23:31.230980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058666+0000","last_change":"2024-03-28T22:08:29.335228+0000","last_active":"2024-03-28T22:09:30.058666+0000","last_peered":"2024-03-28T22:09:30.058666+0000","last_clean":"2024-03-28T22:09:30.058666+0000","last_became_active":"2024-03-28T22:08:29.334686+0000","last_became_peered":"2024-03-28T22:08:29.334686+0000","last_unstale":"2024-03-28T22:09:30.058666+0000","last_undegraded":"2024-03-28T22:09:30.058666+0000","last_fullsized":"2024-03-28T22:09:30.058666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:17:40.571206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.119960+0000","last_change":"2024-03-28T22:08:29.963912+0000","last_active":"2024-03-28T22:09:59.119960+0000","last_peered":"2024-03-28T22:09:59.119960+0000","last_clean":"2024-03-28T22:09:59.119960+0000","last_became_active":"2024-03-28T22:08:29.962740+0000","last_became_peered":"2024-03-28T22:08:29.962740+0000","last_unstale":"2024-03-28T22:09:59.119960+0000","last_undegraded":"2024-03-28T22:09:59.119960+0000","last_fullsized":"2024-03-28T22:09:59.119960+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:50:25.756923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.060333+0000","last_change":"2024-03-28T22:08:29.325767+0000","last_active":"2024-03-28T22:09:30.060333+0000","last_peered":"2024-03-28T22:09:30.060333+0000","last_clean":"2024-03-28T22:09:30.060333+0000","last_became_active":"2024-03-28T22:08:29.325430+0000","last_became_peered":"2024-03-28T22:08:29.325430+0000","last_unstale":"2024-03-28T22:09:30.060333+0000","last_undegraded":"2024-03-28T22:09:30.060333+0000","last_fullsized":"2024-03-28T22:09:30.060333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:01:13.663163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.114614+0000","last_change":"2024-03-28T22:08:29.960158+0000","last_active":"2024-03-28T22:09:59.114614+0000","last_peered":"2024-03-28T22:09:59.114614+0000","last_clean":"2024-03-28T22:09:59.114614+0000","last_became_active":"2024-03-28T22:08:29.959792+0000","last_became_peered":"2024-03-28T22:08:29.959792+0000","last_unstale":"2024-03-28T22:09:59.114614+0000","last_undegraded":"2024-03-28T22:09:59.114614+0000","last_fullsized":"2024-03-28T22:09:59.114614+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:28:49.135551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.293763+0000","last_change":"2024-03-28T22:08:29.328547+0000","last_active":"2024-03-28T22:09:30.293763+0000","last_peered":"2024-03-28T22:09:30.293763+0000","last_clean":"2024-03-28T22:09:30.293763+0000","last_became_active":"2024-03-28T22:08:29.327910+0000","last_became_peered":"2024-03-28T22:08:29.327910+0000","last_unstale":"2024-03-28T22:09:30.293763+0000","last_undegraded":"2024-03-28T22:09:30.293763+0000","last_fullsized":"2024-03-28T22:09:30.293763+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:06:13.980071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.105047+0000","last_change":"2024-03-28T22:08:29.963904+0000","last_active":"2024-03-28T22:09:59.105047+0000","last_peered":"2024-03-28T22:09:59.105047+0000","last_clean":"2024-03-28T22:09:59.105047+0000","last_became_active":"2024-03-28T22:08:29.963536+0000","last_became_peered":"2024-03-28T22:08:29.963536+0000","last_unstale":"2024-03-28T22:09:59.105047+0000","last_undegraded":"2024-03-28T22:09:59.105047+0000","last_fullsized":"2024-03-28T22:09:59.105047+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:59:43.376792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.100207+0000","last_change":"2024-03-28T22:08:29.962357+0000","last_active":"2024-03-28T22:09:59.100207+0000","last_peered":"2024-03-28T22:09:59.100207+0000","last_clean":"2024-03-28T22:09:59.100207+0000","last_became_active":"2024-03-28T22:08:29.961998+0000","last_became_peered":"2024-03-28T22:08:29.961998+0000","last_unstale":"2024-03-28T22:09:59.100207+0000","last_undegraded":"2024-03-28T22:09:59.100207+0000","last_fullsized":"2024-03-28T22:09:59.100207+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:40:47.465889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.108275+0000","last_change":"2024-03-28T22:08:29.961326+0000","last_active":"2024-03-28T22:09:59.108275+0000","last_peered":"2024-03-28T22:09:59.108275+0000","last_clean":"2024-03-28T22:09:59.108275+0000","last_became_active":"2024-03-28T22:08:29.960316+0000","last_became_peered":"2024-03-28T22:08:29.960316+0000","last_unstale":"2024-03-28T22:09:59.108275+0000","last_undegraded":"2024-03-28T22:09:59.108275+0000","last_fullsized":"2024-03-28T22:09:59.108275+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:37:55.773686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.099312+0000","last_change":"2024-03-28T22:08:29.956272+0000","last_active":"2024-03-28T22:09:59.099312+0000","last_peered":"2024-03-28T22:09:59.099312+0000","last_clean":"2024-03-28T22:09:59.099312+0000","last_became_active":"2024-03-28T22:08:29.955875+0000","last_became_peered":"2024-03-28T22:08:29.955875+0000","last_unstale":"2024-03-28T22:09:59.099312+0000","last_undegraded":"2024-03-28T22:09:59.099312+0000","last_fullsized":"2024-03-28T22:09:59.099312+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:17:33.062615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.103567+0000","last_change":"2024-03-28T22:09:30.294754+0000","last_active":"2024-03-28T22:09:59.103567+0000","last_peered":"2024-03-28T22:09:59.103567+0000","last_clean":"2024-03-28T22:09:59.103567+0000","last_became_active":"2024-03-28T22:09:30.294341+0000","last_became_peered":"2024-03-28T22:09:30.294341+0000","last_unstale":"2024-03-28T22:09:59.103567+0000","last_undegraded":"2024-03-28T22:09:59.103567+0000","last_fullsized":"2024-03-28T22:09:59.103567+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:25:46.017373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.105840+0000","last_change":"2024-03-28T22:08:29.960464+0000","last_active":"2024-03-28T22:09:59.105840+0000","last_peered":"2024-03-28T22:09:59.105840+0000","last_clean":"2024-03-28T22:09:59.105840+0000","last_became_active":"2024-03-28T22:08:29.960111+0000","last_became_peered":"2024-03-28T22:08:29.960111+0000","last_unstale":"2024-03-28T22:09:59.105840+0000","last_undegraded":"2024-03-28T22:09:59.105840+0000","last_fullsized":"2024-03-28T22:09:59.105840+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:32:33.533199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.100620+0000","last_change":"2024-03-28T22:08:29.960045+0000","last_active":"2024-03-28T22:09:59.100620+0000","last_peered":"2024-03-28T22:09:59.100620+0000","last_clean":"2024-03-28T22:09:59.100620+0000","last_became_active":"2024-03-28T22:08:29.959645+0000","last_became_peered":"2024-03-28T22:08:29.959645+0000","last_unstale":"2024-03-28T22:09:59.100620+0000","last_undegraded":"2024-03-28T22:09:59.100620+0000","last_fullsized":"2024-03-28T22:09:59.100620+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:34:56.202257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.114146+0000","last_change":"2024-03-28T22:08:29.958736+0000","last_active":"2024-03-28T22:09:59.114146+0000","last_peered":"2024-03-28T22:09:59.114146+0000","last_clean":"2024-03-28T22:09:59.114146+0000","last_became_active":"2024-03-28T22:08:29.958387+0000","last_became_peered":"2024-03-28T22:08:29.958387+0000","last_unstale":"2024-03-28T22:09:59.114146+0000","last_undegraded":"2024-03-28T22:09:59.114146+0000","last_fullsized":"2024-03-28T22:09:59.114146+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:07:54.829166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.101041+0000","last_change":"2024-03-28T22:08:29.961373+0000","last_active":"2024-03-28T22:09:59.101041+0000","last_peered":"2024-03-28T22:09:59.101041+0000","last_clean":"2024-03-28T22:09:59.101041+0000","last_became_active":"2024-03-28T22:08:29.959751+0000","last_became_peered":"2024-03-28T22:08:29.959751+0000","last_unstale":"2024-03-28T22:09:59.101041+0000","last_undegraded":"2024-03-28T22:09:59.101041+0000","last_fullsized":"2024-03-28T22:09:59.101041+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:34:18.680552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.098710+0000","last_change":"2024-03-28T22:08:29.962721+0000","last_active":"2024-03-28T22:09:59.098710+0000","last_peered":"2024-03-28T22:09:59.098710+0000","last_clean":"2024-03-28T22:09:59.098710+0000","last_became_active":"2024-03-28T22:08:29.961349+0000","last_became_peered":"2024-03-28T22:08:29.961349+0000","last_unstale":"2024-03-28T22:09:59.098710+0000","last_undegraded":"2024-03-28T22:09:59.098710+0000","last_fullsized":"2024-03-28T22:09:59.098710+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:35:56.535055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.097778+0000","last_change":"2024-03-28T22:08:29.960068+0000","last_active":"2024-03-28T22:09:59.097778+0000","last_peered":"2024-03-28T22:09:59.097778+0000","last_clean":"2024-03-28T22:09:59.097778+0000","last_became_active":"2024-03-28T22:08:29.959716+0000","last_became_peered":"2024-03-28T22:08:29.959716+0000","last_unstale":"2024-03-28T22:09:59.097778+0000","last_undegraded":"2024-03-28T22:09:59.097778+0000","last_fullsized":"2024-03-28T22:09:59.097778+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:18:38.055322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.121200+0000","last_change":"2024-03-28T22:08:29.957920+0000","last_active":"2024-03-28T22:09:59.121200+0000","last_peered":"2024-03-28T22:09:59.121200+0000","last_clean":"2024-03-28T22:09:59.121200+0000","last_became_active":"2024-03-28T22:08:29.957541+0000","last_became_peered":"2024-03-28T22:08:29.957541+0000","last_unstale":"2024-03-28T22:09:59.121200+0000","last_undegraded":"2024-03-28T22:09:59.121200+0000","last_fullsized":"2024-03-28T22:09:59.121200+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:45:05.658745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.115831+0000","last_change":"2024-03-28T22:08:29.953589+0000","last_active":"2024-03-28T22:09:59.115831+0000","last_peered":"2024-03-28T22:09:59.115831+0000","last_clean":"2024-03-28T22:09:59.115831+0000","last_became_active":"2024-03-28T22:08:29.953253+0000","last_became_peered":"2024-03-28T22:08:29.953253+0000","last_unstale":"2024-03-28T22:09:59.115831+0000","last_undegraded":"2024-03-28T22:09:59.115831+0000","last_fullsized":"2024-03-28T22:09:59.115831+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:35.720708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.109906+0000","last_change":"2024-03-28T22:08:29.966235+0000","last_active":"2024-03-28T22:09:59.109906+0000","last_peered":"2024-03-28T22:09:59.109906+0000","last_clean":"2024-03-28T22:09:59.109906+0000","last_became_active":"2024-03-28T22:08:29.965830+0000","last_became_peered":"2024-03-28T22:08:29.965830+0000","last_unstale":"2024-03-28T22:09:59.109906+0000","last_undegraded":"2024-03-28T22:09:59.109906+0000","last_fullsized":"2024-03-28T22:09:59.109906+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:04:27.094743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":64925,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":672,"num_read_kb":404,"num_write":171,"num_write_kb":119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":421888,"data_stored":199366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351012,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":950272,"data_stored":459161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612648,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28624,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716528,"statfs":{"total":95995035648,"available":95965724672,"internally_reserved":0,"allocated":1646592,"data_stored":1110335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776173,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":459397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037810,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":459317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299448,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36860,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708292,"statfs":{"total":95995035648,"available":95957291008,"internally_reserved":0,"allocated":1560576,"data_stored":1051920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462972,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40428,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704724,"statfs":{"total":95995035648,"available":95953637376,"internally_reserved":0,"allocated":954368,"data_stored":458786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":517740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":460057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411409,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1044480,"data_stored":526085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149771,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":459097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888135,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40432,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704720,"statfs":{"total":95995035648,"available":95953633280,"internally_reserved":0,"allocated":958464,"data_stored":460146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724610,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28552,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716600,"statfs":{"total":95995035648,"available":95965798400,"internally_reserved":0,"allocated":1572864,"data_stored":1051072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":59469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":67506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":61579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T22:10:23.735 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T22:10:23.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.870+0000 7f04f8de2640 1 -- 172.21.15.16:0/2181504554 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04f4102230 msgr2=0x7f04f4108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:23.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.870+0000 7f04f8de2640 1 --2- 172.21.15.16:0/2181504554 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04f4102230 0x7f04f4108ac0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f04ec009be0 tx=0x7f04ec02bef0 comp rx=0 tx=0).stop 2024-03-28T22:10:23.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.870+0000 7f04f8de2640 1 -- 172.21.15.16:0/2181504554 shutdown_connections 2024-03-28T22:10:23.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.870+0000 7f04f8de2640 1 --2- 172.21.15.16:0/2181504554 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04f4102230 0x7f04f4108ac0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:23.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.870+0000 7f04f8de2640 1 --2- 172.21.15.16:0/2181504554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04f4101870 0x7f04f4101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:23.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.870+0000 7f04f8de2640 1 --2- 172.21.15.16:0/2181504554 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f04f4100680 0x7f04f4100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:23.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.870+0000 7f04f8de2640 1 -- 172.21.15.16:0/2181504554 >> 172.21.15.16:0/2181504554 conn(0x7f04f40fbbf0 msgr2=0x7f04f40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:23.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.870+0000 7f04f8de2640 1 -- 172.21.15.16:0/2181504554 shutdown_connections 2024-03-28T22:10:23.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.870+0000 7f04f8de2640 1 -- 172.21.15.16:0/2181504554 wait complete. 2024-03-28T22:10:23.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.871+0000 7f04f8de2640 1 Processor -- start 2024-03-28T22:10:23.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.871+0000 7f04f8de2640 1 -- start start 2024-03-28T22:10:23.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.871+0000 7f04f8de2640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04f4100680 0x7f04f40695b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:23.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.871+0000 7f04f8de2640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f04f4101870 0x7f04f419c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:23.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.871+0000 7f04f8de2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04f4102230 0x7f04f419ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:23.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.871+0000 7f04f8de2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f04f410c2e0 con 0x7f04f4102230 2024-03-28T22:10:23.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.871+0000 7f04f8de2640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f04f410c160 con 0x7f04f4100680 2024-03-28T22:10:23.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.871+0000 7f04f8de2640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f04f410c460 con 0x7f04f4101870 2024-03-28T22:10:23.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.871+0000 7f04f2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04f4102230 0x7f04f419ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:23.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.871+0000 7f04f2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04f4102230 0x7f04f419ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44494/0 (socket says 172.21.15.16:44494) 2024-03-28T22:10:23.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.871+0000 7f04f2d9d640 1 -- 172.21.15.16:0/3901178389 learned_addr learned my addr 172.21.15.16:0/3901178389 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:23.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.871+0000 7f04f259c640 1 --2- 172.21.15.16:0/3901178389 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04f4100680 0x7f04f40695b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:23.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.871+0000 7f04ebfff640 1 --2- 172.21.15.16:0/3901178389 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f04f4101870 0x7f04f419c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:23.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.871+0000 7f04f2d9d640 1 -- 172.21.15.16:0/3901178389 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f04f4101870 msgr2=0x7f04f419c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:23.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.872+0000 7f04f2d9d640 1 --2- 172.21.15.16:0/3901178389 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f04f4101870 0x7f04f419c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:23.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.872+0000 7f04f2d9d640 1 -- 172.21.15.16:0/3901178389 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04f4100680 msgr2=0x7f04f40695b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:23.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.872+0000 7f04f2d9d640 1 --2- 172.21.15.16:0/3901178389 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04f4100680 0x7f04f40695b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:23.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.872+0000 7f04f2d9d640 1 -- 172.21.15.16:0/3901178389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04f419f5f0 con 0x7f04f4102230 2024-03-28T22:10:23.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.872+0000 7f04f2d9d640 1 --2- 172.21.15.16:0/3901178389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04f4102230 0x7f04f419ed30 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f04ec036040 tx=0x7f04ec0337b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:23.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.872+0000 7f04eb7fe640 1 -- 172.21.15.16:0/3901178389 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04ec043020 con 0x7f04f4102230 2024-03-28T22:10:23.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.872+0000 7f04f8de2640 1 -- 172.21.15.16:0/3901178389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04f419f880 con 0x7f04f4102230 2024-03-28T22:10:23.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.872+0000 7f04eb7fe640 1 -- 172.21.15.16:0/3901178389 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04ec02ca60 con 0x7f04f4102230 2024-03-28T22:10:23.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.872+0000 7f04f8de2640 1 -- 172.21.15.16:0/3901178389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04f419fe20 con 0x7f04f4102230 2024-03-28T22:10:23.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.872+0000 7f04eb7fe640 1 -- 172.21.15.16:0/3901178389 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04ec034630 con 0x7f04f4102230 2024-03-28T22:10:23.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.873+0000 7f04f8de2640 1 -- 172.21.15.16:0/3901178389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04f4107940 con 0x7f04f4102230 2024-03-28T22:10:23.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.874+0000 7f04eb7fe640 1 -- 172.21.15.16:0/3901178389 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7f04ec046020 con 0x7f04f4102230 2024-03-28T22:10:23.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.874+0000 7f04eb7fe640 1 --2- 172.21.15.16:0/3901178389 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f04e00a1e10 0x7f04e00a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:23.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.875+0000 7f04f259c640 1 --2- 172.21.15.16:0/3901178389 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f04e00a1e10 0x7f04e00a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:23.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.875+0000 7f04eb7fe640 1 -- 172.21.15.16:0/3901178389 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f04ec0f04d0 con 0x7f04f4102230 2024-03-28T22:10:23.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.875+0000 7f04f259c640 1 --2- 172.21.15.16:0/3901178389 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f04e00a1e10 0x7f04e00a42d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f04f41016d0 tx=0x7f04e4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:23.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:23.879+0000 7f04eb7fe640 1 -- 172.21.15.16:0/3901178389 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f04ec04d020 con 0x7f04f4102230 2024-03-28T22:10:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.059+0000 7f04f8de2640 1 -- 172.21.15.16:0/3901178389 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f04f4061b90 con 0x7f04e00a1e10 2024-03-28T22:10:24.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.067+0000 7f04eb7fe640 1 -- 172.21.15.16:0/3901178389 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258824 (secure 0 0 0) 0x7f04f4061b90 con 0x7f04e00a1e10 2024-03-28T22:10:24.072 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:10:24.074 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-03-28T22:10:24.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.070+0000 7f04f8de2640 1 -- 172.21.15.16:0/3901178389 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f04e00a1e10 msgr2=0x7f04e00a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:24.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.070+0000 7f04f8de2640 1 --2- 172.21.15.16:0/3901178389 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f04e00a1e10 0x7f04e00a42d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f04f41016d0 tx=0x7f04e4005f90 comp rx=0 tx=0).stop 2024-03-28T22:10:24.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.070+0000 7f04f8de2640 1 -- 172.21.15.16:0/3901178389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04f4102230 msgr2=0x7f04f419ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:24.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.070+0000 7f04f8de2640 1 --2- 172.21.15.16:0/3901178389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04f4102230 0x7f04f419ed30 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f04ec036040 tx=0x7f04ec0337b0 comp rx=0 tx=0).stop 2024-03-28T22:10:24.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.070+0000 7f04f8de2640 1 -- 172.21.15.16:0/3901178389 shutdown_connections 2024-03-28T22:10:24.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.070+0000 7f04f8de2640 1 --2- 172.21.15.16:0/3901178389 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f04e00a1e10 0x7f04e00a42d0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:24.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.070+0000 7f04f8de2640 1 --2- 172.21.15.16:0/3901178389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04f4102230 0x7f04f419ed30 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:24.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.070+0000 7f04f8de2640 1 --2- 172.21.15.16:0/3901178389 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f04f4101870 0x7f04f419c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:24.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.070+0000 7f04f8de2640 1 --2- 172.21.15.16:0/3901178389 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04f4100680 0x7f04f40695b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:24.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.070+0000 7f04f8de2640 1 -- 172.21.15.16:0/3901178389 >> 172.21.15.16:0/3901178389 conn(0x7f04f40fbbf0 msgr2=0x7f04f4104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:24.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.070+0000 7f04f8de2640 1 -- 172.21.15.16:0/3901178389 shutdown_connections 2024-03-28T22:10:24.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.070+0000 7f04f8de2640 1 -- 172.21.15.16:0/3901178389 wait complete. 2024-03-28T22:10:24.085 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-03-28T22:10:23.100856+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":655527,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":808,"num_read_kb":523,"num_write":354,"num_write_kb":3059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":787,"ondisk_log_size":787,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":383216,"kb_used_data":13264,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124558608,"statfs":{"total":1151940427776,"available":1151548014592,"internally_reserved":0,"allocated":13582336,"data_stored":7473113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1090,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":72,"num_read_kb":37,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002694"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.119037+0000","last_change":"2024-03-28T22:08:29.962664+0000","last_active":"2024-03-28T22:09:59.119037+0000","last_peered":"2024-03-28T22:09:59.119037+0000","last_clean":"2024-03-28T22:09:59.119037+0000","last_became_active":"2024-03-28T22:08:29.962029+0000","last_became_peered":"2024-03-28T22:08:29.962029+0000","last_unstale":"2024-03-28T22:09:59.119037+0000","last_undegraded":"2024-03-28T22:09:59.119037+0000","last_fullsized":"2024-03-28T22:09:59.119037+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:20:42.832304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.099587+0000","last_change":"2024-03-28T22:08:29.965617+0000","last_active":"2024-03-28T22:09:59.099587+0000","last_peered":"2024-03-28T22:09:59.099587+0000","last_clean":"2024-03-28T22:09:59.099587+0000","last_became_active":"2024-03-28T22:08:29.965248+0000","last_became_peered":"2024-03-28T22:08:29.965248+0000","last_unstale":"2024-03-28T22:09:59.099587+0000","last_undegraded":"2024-03-28T22:09:59.099587+0000","last_fullsized":"2024-03-28T22:09:59.099587+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:37:46.703768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.119356+0000","last_change":"2024-03-28T22:08:29.958452+0000","last_active":"2024-03-28T22:09:59.119356+0000","last_peered":"2024-03-28T22:09:59.119356+0000","last_clean":"2024-03-28T22:09:59.119356+0000","last_became_active":"2024-03-28T22:08:29.958089+0000","last_became_peered":"2024-03-28T22:08:29.958089+0000","last_unstale":"2024-03-28T22:09:59.119356+0000","last_undegraded":"2024-03-28T22:09:59.119356+0000","last_fullsized":"2024-03-28T22:09:59.119356+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:10:44.084925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.104325+0000","last_change":"2024-03-28T22:08:29.964812+0000","last_active":"2024-03-28T22:09:59.104325+0000","last_peered":"2024-03-28T22:09:59.104325+0000","last_clean":"2024-03-28T22:09:59.104325+0000","last_became_active":"2024-03-28T22:08:29.963872+0000","last_became_peered":"2024-03-28T22:08:29.963872+0000","last_unstale":"2024-03-28T22:09:59.104325+0000","last_undegraded":"2024-03-28T22:09:59.104325+0000","last_fullsized":"2024-03-28T22:09:59.104325+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:08.491078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.080889+0000","last_change":"2024-03-28T22:08:29.965564+0000","last_active":"2024-03-28T22:09:59.080889+0000","last_peered":"2024-03-28T22:09:59.080889+0000","last_clean":"2024-03-28T22:09:59.080889+0000","last_became_active":"2024-03-28T22:08:29.962636+0000","last_became_peered":"2024-03-28T22:08:29.962636+0000","last_unstale":"2024-03-28T22:09:59.080889+0000","last_undegraded":"2024-03-28T22:09:59.080889+0000","last_fullsized":"2024-03-28T22:09:59.080889+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:03:50.903231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.111439+0000","last_change":"2024-03-28T22:08:29.962091+0000","last_active":"2024-03-28T22:09:59.111439+0000","last_peered":"2024-03-28T22:09:59.111439+0000","last_clean":"2024-03-28T22:09:59.111439+0000","last_became_active":"2024-03-28T22:08:29.961735+0000","last_became_peered":"2024-03-28T22:08:29.961735+0000","last_unstale":"2024-03-28T22:09:59.111439+0000","last_undegraded":"2024-03-28T22:09:59.111439+0000","last_fullsized":"2024-03-28T22:09:59.111439+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:40:09.853265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.108285+0000","last_change":"2024-03-28T22:08:29.961089+0000","last_active":"2024-03-28T22:09:59.108285+0000","last_peered":"2024-03-28T22:09:59.108285+0000","last_clean":"2024-03-28T22:09:59.108285+0000","last_became_active":"2024-03-28T22:08:29.960711+0000","last_became_peered":"2024-03-28T22:08:29.960711+0000","last_unstale":"2024-03-28T22:09:59.108285+0000","last_undegraded":"2024-03-28T22:09:59.108285+0000","last_fullsized":"2024-03-28T22:09:59.108285+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:05:14.617718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.110495+0000","last_change":"2024-03-28T22:08:29.965142+0000","last_active":"2024-03-28T22:09:59.110495+0000","last_peered":"2024-03-28T22:09:59.110495+0000","last_clean":"2024-03-28T22:09:59.110495+0000","last_became_active":"2024-03-28T22:08:29.964730+0000","last_became_peered":"2024-03-28T22:08:29.964730+0000","last_unstale":"2024-03-28T22:09:59.110495+0000","last_undegraded":"2024-03-28T22:09:59.110495+0000","last_fullsized":"2024-03-28T22:09:59.110495+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:23:43.229681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.115178+0000","last_change":"2024-03-28T22:08:29.958984+0000","last_active":"2024-03-28T22:09:59.115178+0000","last_peered":"2024-03-28T22:09:59.115178+0000","last_clean":"2024-03-28T22:09:59.115178+0000","last_became_active":"2024-03-28T22:08:29.958583+0000","last_became_peered":"2024-03-28T22:08:29.958583+0000","last_unstale":"2024-03-28T22:09:59.115178+0000","last_undegraded":"2024-03-28T22:09:59.115178+0000","last_fullsized":"2024-03-28T22:09:59.115178+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:05:11.361719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.113771+0000","last_change":"2024-03-28T22:08:29.962774+0000","last_active":"2024-03-28T22:09:59.113771+0000","last_peered":"2024-03-28T22:09:59.113771+0000","last_clean":"2024-03-28T22:09:59.113771+0000","last_became_active":"2024-03-28T22:08:29.961222+0000","last_became_peered":"2024-03-28T22:08:29.961222+0000","last_unstale":"2024-03-28T22:09:59.113771+0000","last_undegraded":"2024-03-28T22:09:59.113771+0000","last_fullsized":"2024-03-28T22:09:59.113771+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:47:05.310483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.119577+0000","last_change":"2024-03-28T22:08:29.964041+0000","last_active":"2024-03-28T22:09:59.119577+0000","last_peered":"2024-03-28T22:09:59.119577+0000","last_clean":"2024-03-28T22:09:59.119577+0000","last_became_active":"2024-03-28T22:08:29.963360+0000","last_became_peered":"2024-03-28T22:08:29.963360+0000","last_unstale":"2024-03-28T22:09:59.119577+0000","last_undegraded":"2024-03-28T22:09:59.119577+0000","last_fullsized":"2024-03-28T22:09:59.119577+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:32.921916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.095082+0000","last_change":"2024-03-28T22:08:29.966449+0000","last_active":"2024-03-28T22:09:59.095082+0000","last_peered":"2024-03-28T22:09:59.095082+0000","last_clean":"2024-03-28T22:09:59.095082+0000","last_became_active":"2024-03-28T22:08:29.966036+0000","last_became_peered":"2024-03-28T22:08:29.966036+0000","last_unstale":"2024-03-28T22:09:59.095082+0000","last_undegraded":"2024-03-28T22:09:59.095082+0000","last_fullsized":"2024-03-28T22:09:59.095082+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:12:15.057713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.113074+0000","last_change":"2024-03-28T22:08:29.958924+0000","last_active":"2024-03-28T22:09:59.113074+0000","last_peered":"2024-03-28T22:09:59.113074+0000","last_clean":"2024-03-28T22:09:59.113074+0000","last_became_active":"2024-03-28T22:08:29.958601+0000","last_became_peered":"2024-03-28T22:08:29.958601+0000","last_unstale":"2024-03-28T22:09:59.113074+0000","last_undegraded":"2024-03-28T22:09:59.113074+0000","last_fullsized":"2024-03-28T22:09:59.113074+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:53:37.070276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.098999+0000","last_change":"2024-03-28T22:08:29.965045+0000","last_active":"2024-03-28T22:09:59.098999+0000","last_peered":"2024-03-28T22:09:59.098999+0000","last_clean":"2024-03-28T22:09:59.098999+0000","last_became_active":"2024-03-28T22:08:29.964710+0000","last_became_peered":"2024-03-28T22:08:29.964710+0000","last_unstale":"2024-03-28T22:09:59.098999+0000","last_undegraded":"2024-03-28T22:09:59.098999+0000","last_fullsized":"2024-03-28T22:09:59.098999+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:03:11.491213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.104261+0000","last_change":"2024-03-28T22:08:29.957940+0000","last_active":"2024-03-28T22:09:59.104261+0000","last_peered":"2024-03-28T22:09:59.104261+0000","last_clean":"2024-03-28T22:09:59.104261+0000","last_became_active":"2024-03-28T22:08:29.957587+0000","last_became_peered":"2024-03-28T22:08:29.957587+0000","last_unstale":"2024-03-28T22:09:59.104261+0000","last_undegraded":"2024-03-28T22:09:59.104261+0000","last_fullsized":"2024-03-28T22:09:59.104261+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:03:58.557806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.120996+0000","last_change":"2024-03-28T22:08:29.957397+0000","last_active":"2024-03-28T22:09:59.120996+0000","last_peered":"2024-03-28T22:09:59.120996+0000","last_clean":"2024-03-28T22:09:59.120996+0000","last_became_active":"2024-03-28T22:08:29.957054+0000","last_became_peered":"2024-03-28T22:08:29.957054+0000","last_unstale":"2024-03-28T22:09:59.120996+0000","last_undegraded":"2024-03-28T22:09:59.120996+0000","last_fullsized":"2024-03-28T22:09:59.120996+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:35:28.500561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.119267+0000","last_change":"2024-03-28T22:08:29.963343+0000","last_active":"2024-03-28T22:09:59.119267+0000","last_peered":"2024-03-28T22:09:59.119267+0000","last_clean":"2024-03-28T22:09:59.119267+0000","last_became_active":"2024-03-28T22:08:29.962926+0000","last_became_peered":"2024-03-28T22:08:29.962926+0000","last_unstale":"2024-03-28T22:09:59.119267+0000","last_undegraded":"2024-03-28T22:09:59.119267+0000","last_fullsized":"2024-03-28T22:09:59.119267+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:15:39.133930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.094891+0000","last_change":"2024-03-28T22:08:29.955903+0000","last_active":"2024-03-28T22:09:59.094891+0000","last_peered":"2024-03-28T22:09:59.094891+0000","last_clean":"2024-03-28T22:09:59.094891+0000","last_became_active":"2024-03-28T22:08:29.955460+0000","last_became_peered":"2024-03-28T22:08:29.955460+0000","last_unstale":"2024-03-28T22:09:59.094891+0000","last_undegraded":"2024-03-28T22:09:59.094891+0000","last_fullsized":"2024-03-28T22:09:59.094891+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:36:12.961475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.110397+0000","last_change":"2024-03-28T22:08:29.954279+0000","last_active":"2024-03-28T22:09:59.110397+0000","last_peered":"2024-03-28T22:09:59.110397+0000","last_clean":"2024-03-28T22:09:59.110397+0000","last_became_active":"2024-03-28T22:08:29.953869+0000","last_became_peered":"2024-03-28T22:08:29.953869+0000","last_unstale":"2024-03-28T22:09:59.110397+0000","last_undegraded":"2024-03-28T22:09:59.110397+0000","last_fullsized":"2024-03-28T22:09:59.110397+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:05:52.247391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.054735+0000","last_change":"2024-03-28T22:08:29.327644+0000","last_active":"2024-03-28T22:09:30.054735+0000","last_peered":"2024-03-28T22:09:30.054735+0000","last_clean":"2024-03-28T22:09:30.054735+0000","last_became_active":"2024-03-28T22:08:29.327325+0000","last_became_peered":"2024-03-28T22:08:29.327325+0000","last_unstale":"2024-03-28T22:09:30.054735+0000","last_undegraded":"2024-03-28T22:09:30.054735+0000","last_fullsized":"2024-03-28T22:09:30.054735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:26:54.782703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.102346+0000","last_change":"2024-03-28T22:08:29.962905+0000","last_active":"2024-03-28T22:09:59.102346+0000","last_peered":"2024-03-28T22:09:59.102346+0000","last_clean":"2024-03-28T22:09:59.102346+0000","last_became_active":"2024-03-28T22:08:29.962559+0000","last_became_peered":"2024-03-28T22:08:29.962559+0000","last_unstale":"2024-03-28T22:09:59.102346+0000","last_undegraded":"2024-03-28T22:09:59.102346+0000","last_fullsized":"2024-03-28T22:09:59.102346+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:02.347584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058126+0000","last_change":"2024-03-28T22:08:29.334871+0000","last_active":"2024-03-28T22:09:30.058126+0000","last_peered":"2024-03-28T22:09:30.058126+0000","last_clean":"2024-03-28T22:09:30.058126+0000","last_became_active":"2024-03-28T22:08:29.334565+0000","last_became_peered":"2024-03-28T22:08:29.334565+0000","last_unstale":"2024-03-28T22:09:30.058126+0000","last_undegraded":"2024-03-28T22:09:30.058126+0000","last_fullsized":"2024-03-28T22:09:30.058126+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:31:19.333496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.119365+0000","last_change":"2024-03-28T22:08:29.957622+0000","last_active":"2024-03-28T22:09:59.119365+0000","last_peered":"2024-03-28T22:09:59.119365+0000","last_clean":"2024-03-28T22:09:59.119365+0000","last_became_active":"2024-03-28T22:08:29.957208+0000","last_became_peered":"2024-03-28T22:08:29.957208+0000","last_unstale":"2024-03-28T22:09:59.119365+0000","last_undegraded":"2024-03-28T22:09:59.119365+0000","last_fullsized":"2024-03-28T22:09:59.119365+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:49:44.684203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058576+0000","last_change":"2024-03-28T22:08:29.330291+0000","last_active":"2024-03-28T22:09:30.058576+0000","last_peered":"2024-03-28T22:09:30.058576+0000","last_clean":"2024-03-28T22:09:30.058576+0000","last_became_active":"2024-03-28T22:08:29.328955+0000","last_became_peered":"2024-03-28T22:08:29.328955+0000","last_unstale":"2024-03-28T22:09:30.058576+0000","last_undegraded":"2024-03-28T22:09:30.058576+0000","last_fullsized":"2024-03-28T22:09:30.058576+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:10.838059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.116812+0000","last_change":"2024-03-28T22:08:29.966681+0000","last_active":"2024-03-28T22:09:59.116812+0000","last_peered":"2024-03-28T22:09:59.116812+0000","last_clean":"2024-03-28T22:09:59.116812+0000","last_became_active":"2024-03-28T22:08:29.966023+0000","last_became_peered":"2024-03-28T22:08:29.966023+0000","last_unstale":"2024-03-28T22:09:59.116812+0000","last_undegraded":"2024-03-28T22:09:59.116812+0000","last_fullsized":"2024-03-28T22:09:59.116812+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:04:43.281259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058443+0000","last_change":"2024-03-28T22:08:29.329026+0000","last_active":"2024-03-28T22:09:30.058443+0000","last_peered":"2024-03-28T22:09:30.058443+0000","last_clean":"2024-03-28T22:09:30.058443+0000","last_became_active":"2024-03-28T22:08:29.327570+0000","last_became_peered":"2024-03-28T22:08:29.327570+0000","last_unstale":"2024-03-28T22:09:30.058443+0000","last_undegraded":"2024-03-28T22:09:30.058443+0000","last_fullsized":"2024-03-28T22:09:30.058443+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:53:54.779073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.106272+0000","last_change":"2024-03-28T22:08:29.966455+0000","last_active":"2024-03-28T22:09:59.106272+0000","last_peered":"2024-03-28T22:09:59.106272+0000","last_clean":"2024-03-28T22:09:59.106272+0000","last_became_active":"2024-03-28T22:08:29.964451+0000","last_became_peered":"2024-03-28T22:08:29.964451+0000","last_unstale":"2024-03-28T22:09:59.106272+0000","last_undegraded":"2024-03-28T22:09:59.106272+0000","last_fullsized":"2024-03-28T22:09:59.106272+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:46:22.811146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'8","reported_seq":138,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:10:20.499412+0000","last_change":"2024-03-28T22:08:29.327397+0000","last_active":"2024-03-28T22:10:20.499412+0000","last_peered":"2024-03-28T22:10:20.499412+0000","last_clean":"2024-03-28T22:10:20.499412+0000","last_became_active":"2024-03-28T22:08:29.327009+0000","last_became_peered":"2024-03-28T22:08:29.327009+0000","last_unstale":"2024-03-28T22:10:20.499412+0000","last_undegraded":"2024-03-28T22:10:20.499412+0000","last_fullsized":"2024-03-28T22:10:20.499412+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:38:51.951934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1545,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":111,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.096251+0000","last_change":"2024-03-28T22:08:29.969821+0000","last_active":"2024-03-28T22:09:59.096251+0000","last_peered":"2024-03-28T22:09:59.096251+0000","last_clean":"2024-03-28T22:09:59.096251+0000","last_became_active":"2024-03-28T22:08:29.969386+0000","last_became_peered":"2024-03-28T22:08:29.969386+0000","last_unstale":"2024-03-28T22:09:59.096251+0000","last_undegraded":"2024-03-28T22:09:59.096251+0000","last_fullsized":"2024-03-28T22:09:59.096251+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:44:19.753083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.424985+0000","last_change":"2024-03-28T22:08:29.332385+0000","last_active":"2024-03-28T22:09:30.424985+0000","last_peered":"2024-03-28T22:09:30.424985+0000","last_clean":"2024-03-28T22:09:30.424985+0000","last_became_active":"2024-03-28T22:08:29.331956+0000","last_became_peered":"2024-03-28T22:08:29.331956+0000","last_unstale":"2024-03-28T22:09:30.424985+0000","last_undegraded":"2024-03-28T22:09:30.424985+0000","last_fullsized":"2024-03-28T22:09:30.424985+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:10:04.428034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.092555+0000","last_change":"2024-03-28T22:09:30.080320+0000","last_active":"2024-03-28T22:09:59.092555+0000","last_peered":"2024-03-28T22:09:59.092555+0000","last_clean":"2024-03-28T22:09:59.092555+0000","last_became_active":"2024-03-28T22:09:30.079928+0000","last_became_peered":"2024-03-28T22:09:30.079928+0000","last_unstale":"2024-03-28T22:09:59.092555+0000","last_undegraded":"2024-03-28T22:09:59.092555+0000","last_fullsized":"2024-03-28T22:09:59.092555+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:12:53.385167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.293420+0000","last_change":"2024-03-28T22:08:29.327180+0000","last_active":"2024-03-28T22:09:30.293420+0000","last_peered":"2024-03-28T22:09:30.293420+0000","last_clean":"2024-03-28T22:09:30.293420+0000","last_became_active":"2024-03-28T22:08:29.326825+0000","last_became_peered":"2024-03-28T22:08:29.326825+0000","last_unstale":"2024-03-28T22:09:30.293420+0000","last_undegraded":"2024-03-28T22:09:30.293420+0000","last_fullsized":"2024-03-28T22:09:30.293420+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:11:09.979970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.098656+0000","last_change":"2024-03-28T22:08:29.955299+0000","last_active":"2024-03-28T22:09:59.098656+0000","last_peered":"2024-03-28T22:09:59.098656+0000","last_clean":"2024-03-28T22:09:59.098656+0000","last_became_active":"2024-03-28T22:08:29.954949+0000","last_became_peered":"2024-03-28T22:08:29.954949+0000","last_unstale":"2024-03-28T22:09:59.098656+0000","last_undegraded":"2024-03-28T22:09:59.098656+0000","last_fullsized":"2024-03-28T22:09:59.098656+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:01:48.375251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.059530+0000","last_change":"2024-03-28T22:08:29.327261+0000","last_active":"2024-03-28T22:09:30.059530+0000","last_peered":"2024-03-28T22:09:30.059530+0000","last_clean":"2024-03-28T22:09:30.059530+0000","last_became_active":"2024-03-28T22:08:29.326910+0000","last_became_peered":"2024-03-28T22:08:29.326910+0000","last_unstale":"2024-03-28T22:09:30.059530+0000","last_undegraded":"2024-03-28T22:09:30.059530+0000","last_fullsized":"2024-03-28T22:09:30.059530+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:46:41.933372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.093770+0000","last_change":"2024-03-28T22:08:29.954891+0000","last_active":"2024-03-28T22:09:59.093770+0000","last_peered":"2024-03-28T22:09:59.093770+0000","last_clean":"2024-03-28T22:09:59.093770+0000","last_became_active":"2024-03-28T22:08:29.954519+0000","last_became_peered":"2024-03-28T22:08:29.954519+0000","last_unstale":"2024-03-28T22:09:59.093770+0000","last_undegraded":"2024-03-28T22:09:59.093770+0000","last_fullsized":"2024-03-28T22:09:59.093770+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:07:42.621189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":129,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:10:18.495749+0000","last_change":"2024-03-28T22:08:29.330180+0000","last_active":"2024-03-28T22:10:18.495749+0000","last_peered":"2024-03-28T22:10:18.495749+0000","last_clean":"2024-03-28T22:10:18.495749+0000","last_became_active":"2024-03-28T22:08:29.329835+0000","last_became_peered":"2024-03-28T22:08:29.329835+0000","last_unstale":"2024-03-28T22:10:18.495749+0000","last_undegraded":"2024-03-28T22:10:18.495749+0000","last_fullsized":"2024-03-28T22:10:18.495749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:48:39.906359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":106,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.118105+0000","last_change":"2024-03-28T22:08:29.967074+0000","last_active":"2024-03-28T22:09:59.118105+0000","last_peered":"2024-03-28T22:09:59.118105+0000","last_clean":"2024-03-28T22:09:59.118105+0000","last_became_active":"2024-03-28T22:08:29.966258+0000","last_became_peered":"2024-03-28T22:08:29.966258+0000","last_unstale":"2024-03-28T22:09:59.118105+0000","last_undegraded":"2024-03-28T22:09:59.118105+0000","last_fullsized":"2024-03-28T22:09:59.118105+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:25:44.986179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.055474+0000","last_change":"2024-03-28T22:08:29.326388+0000","last_active":"2024-03-28T22:09:30.055474+0000","last_peered":"2024-03-28T22:09:30.055474+0000","last_clean":"2024-03-28T22:09:30.055474+0000","last_became_active":"2024-03-28T22:08:29.326001+0000","last_became_peered":"2024-03-28T22:08:29.326001+0000","last_unstale":"2024-03-28T22:09:30.055474+0000","last_undegraded":"2024-03-28T22:09:30.055474+0000","last_fullsized":"2024-03-28T22:09:30.055474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:31:56.227038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.104748+0000","last_change":"2024-03-28T22:08:29.959299+0000","last_active":"2024-03-28T22:09:59.104748+0000","last_peered":"2024-03-28T22:09:59.104748+0000","last_clean":"2024-03-28T22:09:59.104748+0000","last_became_active":"2024-03-28T22:08:29.958955+0000","last_became_peered":"2024-03-28T22:08:29.958955+0000","last_unstale":"2024-03-28T22:09:59.104748+0000","last_undegraded":"2024-03-28T22:09:59.104748+0000","last_fullsized":"2024-03-28T22:09:59.104748+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:11:06.775920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.060567+0000","last_change":"2024-03-28T22:08:29.326089+0000","last_active":"2024-03-28T22:09:30.060567+0000","last_peered":"2024-03-28T22:09:30.060567+0000","last_clean":"2024-03-28T22:09:30.060567+0000","last_became_active":"2024-03-28T22:08:29.325463+0000","last_became_peered":"2024-03-28T22:08:29.325463+0000","last_unstale":"2024-03-28T22:09:30.060567+0000","last_undegraded":"2024-03-28T22:09:30.060567+0000","last_fullsized":"2024-03-28T22:09:30.060567+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:18:31.049729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.118631+0000","last_change":"2024-03-28T22:08:29.964248+0000","last_active":"2024-03-28T22:09:59.118631+0000","last_peered":"2024-03-28T22:09:59.118631+0000","last_clean":"2024-03-28T22:09:59.118631+0000","last_became_active":"2024-03-28T22:08:29.963897+0000","last_became_peered":"2024-03-28T22:08:29.963897+0000","last_unstale":"2024-03-28T22:09:59.118631+0000","last_undegraded":"2024-03-28T22:09:59.118631+0000","last_fullsized":"2024-03-28T22:09:59.118631+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:41:39.645360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'31","reported_seq":271,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:10:19.497584+0000","last_change":"2024-03-28T22:08:29.330732+0000","last_active":"2024-03-28T22:10:19.497584+0000","last_peered":"2024-03-28T22:10:19.497584+0000","last_clean":"2024-03-28T22:10:19.497584+0000","last_became_active":"2024-03-28T22:08:29.329231+0000","last_became_peered":"2024-03-28T22:08:29.329231+0000","last_unstale":"2024-03-28T22:10:19.497584+0000","last_undegraded":"2024-03-28T22:10:19.497584+0000","last_fullsized":"2024-03-28T22:10:19.497584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:16:39.260429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59469,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":221,"num_read_kb":168,"num_write":32,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.109344+0000","last_change":"2024-03-28T22:08:29.965791+0000","last_active":"2024-03-28T22:09:59.109344+0000","last_peered":"2024-03-28T22:09:59.109344+0000","last_clean":"2024-03-28T22:09:59.109344+0000","last_became_active":"2024-03-28T22:08:29.965390+0000","last_became_peered":"2024-03-28T22:08:29.965390+0000","last_unstale":"2024-03-28T22:09:59.109344+0000","last_undegraded":"2024-03-28T22:09:59.109344+0000","last_fullsized":"2024-03-28T22:09:59.109344+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:43:53.325906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.057898+0000","last_change":"2024-03-28T22:08:29.325851+0000","last_active":"2024-03-28T22:09:30.057898+0000","last_peered":"2024-03-28T22:09:30.057898+0000","last_clean":"2024-03-28T22:09:30.057898+0000","last_became_active":"2024-03-28T22:08:29.325500+0000","last_became_peered":"2024-03-28T22:08:29.325500+0000","last_unstale":"2024-03-28T22:09:30.057898+0000","last_undegraded":"2024-03-28T22:09:30.057898+0000","last_fullsized":"2024-03-28T22:09:30.057898+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:04:25.151908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.115301+0000","last_change":"2024-03-28T22:08:29.953857+0000","last_active":"2024-03-28T22:09:59.115301+0000","last_peered":"2024-03-28T22:09:59.115301+0000","last_clean":"2024-03-28T22:09:59.115301+0000","last_became_active":"2024-03-28T22:08:29.953538+0000","last_became_peered":"2024-03-28T22:08:29.953538+0000","last_unstale":"2024-03-28T22:09:59.115301+0000","last_undegraded":"2024-03-28T22:09:59.115301+0000","last_fullsized":"2024-03-28T22:09:59.115301+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:40:32.404760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.060057+0000","last_change":"2024-03-28T22:08:29.327286+0000","last_active":"2024-03-28T22:09:30.060057+0000","last_peered":"2024-03-28T22:09:30.060057+0000","last_clean":"2024-03-28T22:09:30.060057+0000","last_became_active":"2024-03-28T22:08:29.326902+0000","last_became_peered":"2024-03-28T22:08:29.326902+0000","last_unstale":"2024-03-28T22:09:30.060057+0000","last_undegraded":"2024-03-28T22:09:30.060057+0000","last_fullsized":"2024-03-28T22:09:30.060057+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:14:58.669534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.116496+0000","last_change":"2024-03-28T22:09:30.294946+0000","last_active":"2024-03-28T22:09:59.116496+0000","last_peered":"2024-03-28T22:09:59.116496+0000","last_clean":"2024-03-28T22:09:59.116496+0000","last_became_active":"2024-03-28T22:09:30.294496+0000","last_became_peered":"2024-03-28T22:09:30.294496+0000","last_unstale":"2024-03-28T22:09:59.116496+0000","last_undegraded":"2024-03-28T22:09:59.116496+0000","last_fullsized":"2024-03-28T22:09:59.116496+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:26:40.062384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'6","reported_seq":138,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:10:21.500594+0000","last_change":"2024-03-28T22:08:29.327740+0000","last_active":"2024-03-28T22:10:21.500594+0000","last_peered":"2024-03-28T22:10:21.500594+0000","last_clean":"2024-03-28T22:10:21.500594+0000","last_became_active":"2024-03-28T22:08:29.326664+0000","last_became_peered":"2024-03-28T22:08:29.326664+0000","last_unstale":"2024-03-28T22:10:21.500594+0000","last_undegraded":"2024-03-28T22:10:21.500594+0000","last_fullsized":"2024-03-28T22:10:21.500594+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:58:17.681836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1511,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":114,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.104158+0000","last_change":"2024-03-28T22:08:29.961979+0000","last_active":"2024-03-28T22:09:59.104158+0000","last_peered":"2024-03-28T22:09:59.104158+0000","last_clean":"2024-03-28T22:09:59.104158+0000","last_became_active":"2024-03-28T22:08:29.961615+0000","last_became_peered":"2024-03-28T22:08:29.961615+0000","last_unstale":"2024-03-28T22:09:59.104158+0000","last_undegraded":"2024-03-28T22:09:59.104158+0000","last_fullsized":"2024-03-28T22:09:59.104158+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:50:37.227632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.057515+0000","last_change":"2024-03-28T22:08:29.329017+0000","last_active":"2024-03-28T22:09:30.057515+0000","last_peered":"2024-03-28T22:09:30.057515+0000","last_clean":"2024-03-28T22:09:30.057515+0000","last_became_active":"2024-03-28T22:08:29.328686+0000","last_became_peered":"2024-03-28T22:08:29.328686+0000","last_unstale":"2024-03-28T22:09:30.057515+0000","last_undegraded":"2024-03-28T22:09:30.057515+0000","last_fullsized":"2024-03-28T22:09:30.057515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:25:28.986165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.102273+0000","last_change":"2024-03-28T22:09:30.426320+0000","last_active":"2024-03-28T22:09:59.102273+0000","last_peered":"2024-03-28T22:09:59.102273+0000","last_clean":"2024-03-28T22:09:59.102273+0000","last_became_active":"2024-03-28T22:09:30.425862+0000","last_became_peered":"2024-03-28T22:09:30.425862+0000","last_unstale":"2024-03-28T22:09:59.102273+0000","last_undegraded":"2024-03-28T22:09:59.102273+0000","last_fullsized":"2024-03-28T22:09:59.102273+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:44.240551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058066+0000","last_change":"2024-03-28T22:08:29.323274+0000","last_active":"2024-03-28T22:09:30.058066+0000","last_peered":"2024-03-28T22:09:30.058066+0000","last_clean":"2024-03-28T22:09:30.058066+0000","last_became_active":"2024-03-28T22:08:29.322911+0000","last_became_peered":"2024-03-28T22:08:29.322911+0000","last_unstale":"2024-03-28T22:09:30.058066+0000","last_undegraded":"2024-03-28T22:09:30.058066+0000","last_fullsized":"2024-03-28T22:09:30.058066+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:44:14.439411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.107898+0000","last_change":"2024-03-28T22:08:29.961101+0000","last_active":"2024-03-28T22:09:59.107898+0000","last_peered":"2024-03-28T22:09:59.107898+0000","last_clean":"2024-03-28T22:09:59.107898+0000","last_became_active":"2024-03-28T22:08:29.960767+0000","last_became_peered":"2024-03-28T22:08:29.960767+0000","last_unstale":"2024-03-28T22:09:59.107898+0000","last_undegraded":"2024-03-28T22:09:59.107898+0000","last_fullsized":"2024-03-28T22:09:59.107898+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:33:45.236691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.060215+0000","last_change":"2024-03-28T22:09:30.060215+0000","last_active":"2024-03-28T22:09:30.060215+0000","last_peered":"2024-03-28T22:09:30.060215+0000","last_clean":"2024-03-28T22:09:30.060215+0000","last_became_active":"2024-03-28T22:09:30.059828+0000","last_became_peered":"2024-03-28T22:09:30.059828+0000","last_unstale":"2024-03-28T22:09:30.060215+0000","last_undegraded":"2024-03-28T22:09:30.060215+0000","last_fullsized":"2024-03-28T22:09:30.060215+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:42:05.087899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.112511+0000","last_change":"2024-03-28T22:08:29.954093+0000","last_active":"2024-03-28T22:09:59.112511+0000","last_peered":"2024-03-28T22:09:59.112511+0000","last_clean":"2024-03-28T22:09:59.112511+0000","last_became_active":"2024-03-28T22:08:29.953688+0000","last_became_peered":"2024-03-28T22:08:29.953688+0000","last_unstale":"2024-03-28T22:09:59.112511+0000","last_undegraded":"2024-03-28T22:09:59.112511+0000","last_fullsized":"2024-03-28T22:09:59.112511+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:57:51.026565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.055115+0000","last_change":"2024-03-28T22:08:29.331104+0000","last_active":"2024-03-28T22:09:30.055115+0000","last_peered":"2024-03-28T22:09:30.055115+0000","last_clean":"2024-03-28T22:09:30.055115+0000","last_became_active":"2024-03-28T22:08:29.330505+0000","last_became_peered":"2024-03-28T22:08:29.330505+0000","last_unstale":"2024-03-28T22:09:30.055115+0000","last_undegraded":"2024-03-28T22:09:30.055115+0000","last_fullsized":"2024-03-28T22:09:30.055115+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:36:59.944587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.110591+0000","last_change":"2024-03-28T22:08:29.963540+0000","last_active":"2024-03-28T22:09:59.110591+0000","last_peered":"2024-03-28T22:09:59.110591+0000","last_clean":"2024-03-28T22:09:59.110591+0000","last_became_active":"2024-03-28T22:08:29.963206+0000","last_became_peered":"2024-03-28T22:08:29.963206+0000","last_unstale":"2024-03-28T22:09:59.110591+0000","last_undegraded":"2024-03-28T22:09:59.110591+0000","last_fullsized":"2024-03-28T22:09:59.110591+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:55:53.292554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.098046+0000","last_change":"2024-03-28T22:08:29.962822+0000","last_active":"2024-03-28T22:09:59.098046+0000","last_peered":"2024-03-28T22:09:59.098046+0000","last_clean":"2024-03-28T22:09:59.098046+0000","last_became_active":"2024-03-28T22:08:29.962343+0000","last_became_peered":"2024-03-28T22:08:29.962343+0000","last_unstale":"2024-03-28T22:09:59.098046+0000","last_undegraded":"2024-03-28T22:09:59.098046+0000","last_fullsized":"2024-03-28T22:09:59.098046+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:36.608543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'6","reported_seq":132,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:10:20.498618+0000","last_change":"2024-03-28T22:08:29.328239+0000","last_active":"2024-03-28T22:10:20.498618+0000","last_peered":"2024-03-28T22:10:20.498618+0000","last_clean":"2024-03-28T22:10:20.498618+0000","last_became_active":"2024-03-28T22:08:29.327903+0000","last_became_peered":"2024-03-28T22:08:29.327903+0000","last_unstale":"2024-03-28T22:10:20.498618+0000","last_undegraded":"2024-03-28T22:10:20.498618+0000","last_fullsized":"2024-03-28T22:10:20.498618+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:14:17.558472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":107,"num_read_kb":107,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.085982+0000","last_change":"2024-03-28T22:09:30.426683+0000","last_active":"2024-03-28T22:09:59.085982+0000","last_peered":"2024-03-28T22:09:59.085982+0000","last_clean":"2024-03-28T22:09:59.085982+0000","last_became_active":"2024-03-28T22:09:30.426255+0000","last_became_peered":"2024-03-28T22:09:30.426255+0000","last_unstale":"2024-03-28T22:09:59.085982+0000","last_undegraded":"2024-03-28T22:09:59.085982+0000","last_fullsized":"2024-03-28T22:09:59.085982+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:36.755978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.079359+0000","last_change":"2024-03-28T22:08:29.323050+0000","last_active":"2024-03-28T22:09:30.079359+0000","last_peered":"2024-03-28T22:09:30.079359+0000","last_clean":"2024-03-28T22:09:30.079359+0000","last_became_active":"2024-03-28T22:08:29.322684+0000","last_became_peered":"2024-03-28T22:08:29.322684+0000","last_unstale":"2024-03-28T22:09:30.079359+0000","last_undegraded":"2024-03-28T22:09:30.079359+0000","last_fullsized":"2024-03-28T22:09:30.079359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:14:43.811122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":354,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.054922+0000","last_change":"2024-03-28T22:07:17.785516+0000","last_active":"2024-03-28T22:09:30.054922+0000","last_peered":"2024-03-28T22:09:30.054922+0000","last_clean":"2024-03-28T22:09:30.054922+0000","last_became_active":"2024-03-28T22:07:17.785123+0000","last_became_peered":"2024-03-28T22:07:17.785123+0000","last_unstale":"2024-03-28T22:09:30.054922+0000","last_undegraded":"2024-03-28T22:09:30.054922+0000","last_fullsized":"2024-03-28T22:09:30.054922+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:03:59.840513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:03:59.840513+0000","last_clean_scrub_stamp":"2024-03-28T22:03:59.840513+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:31:06.253632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.108076+0000","last_change":"2024-03-28T22:08:29.961489+0000","last_active":"2024-03-28T22:09:59.108076+0000","last_peered":"2024-03-28T22:09:59.108076+0000","last_clean":"2024-03-28T22:09:59.108076+0000","last_became_active":"2024-03-28T22:08:29.960881+0000","last_became_peered":"2024-03-28T22:08:29.960881+0000","last_unstale":"2024-03-28T22:09:59.108076+0000","last_undegraded":"2024-03-28T22:09:59.108076+0000","last_fullsized":"2024-03-28T22:09:59.108076+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:48.870834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.054972+0000","last_change":"2024-03-28T22:08:29.331994+0000","last_active":"2024-03-28T22:09:30.054972+0000","last_peered":"2024-03-28T22:09:30.054972+0000","last_clean":"2024-03-28T22:09:30.054972+0000","last_became_active":"2024-03-28T22:08:29.331630+0000","last_became_peered":"2024-03-28T22:08:29.331630+0000","last_unstale":"2024-03-28T22:09:30.054972+0000","last_undegraded":"2024-03-28T22:09:30.054972+0000","last_fullsized":"2024-03-28T22:09:30.054972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:11:37.100405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058466+0000","last_change":"2024-03-28T22:08:29.330353+0000","last_active":"2024-03-28T22:09:30.058466+0000","last_peered":"2024-03-28T22:09:30.058466+0000","last_clean":"2024-03-28T22:09:30.058466+0000","last_became_active":"2024-03-28T22:08:29.329421+0000","last_became_peered":"2024-03-28T22:08:29.329421+0000","last_unstale":"2024-03-28T22:09:30.058466+0000","last_undegraded":"2024-03-28T22:09:30.058466+0000","last_fullsized":"2024-03-28T22:09:30.058466+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:56.166506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.108762+0000","last_change":"2024-03-28T22:08:29.948595+0000","last_active":"2024-03-28T22:09:59.108762+0000","last_peered":"2024-03-28T22:09:59.108762+0000","last_clean":"2024-03-28T22:09:59.108762+0000","last_became_active":"2024-03-28T22:08:29.946986+0000","last_became_peered":"2024-03-28T22:08:29.946986+0000","last_unstale":"2024-03-28T22:09:59.108762+0000","last_undegraded":"2024-03-28T22:09:59.108762+0000","last_fullsized":"2024-03-28T22:09:59.108762+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:56:12.351731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.328434+0000","last_change":"2024-03-28T22:08:29.333375+0000","last_active":"2024-03-28T22:09:59.328434+0000","last_peered":"2024-03-28T22:09:59.328434+0000","last_clean":"2024-03-28T22:09:59.328434+0000","last_became_active":"2024-03-28T22:08:29.331594+0000","last_became_peered":"2024-03-28T22:08:29.331594+0000","last_unstale":"2024-03-28T22:09:59.328434+0000","last_undegraded":"2024-03-28T22:09:59.328434+0000","last_fullsized":"2024-03-28T22:09:59.328434+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:07:43.936807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.109527+0000","last_change":"2024-03-28T22:08:29.959061+0000","last_active":"2024-03-28T22:09:59.109527+0000","last_peered":"2024-03-28T22:09:59.109527+0000","last_clean":"2024-03-28T22:09:59.109527+0000","last_became_active":"2024-03-28T22:08:29.958723+0000","last_became_peered":"2024-03-28T22:08:29.958723+0000","last_unstale":"2024-03-28T22:09:59.109527+0000","last_undegraded":"2024-03-28T22:09:59.109527+0000","last_fullsized":"2024-03-28T22:09:59.109527+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:32:23.016606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.079461+0000","last_change":"2024-03-28T22:08:29.329886+0000","last_active":"2024-03-28T22:09:30.079461+0000","last_peered":"2024-03-28T22:09:30.079461+0000","last_clean":"2024-03-28T22:09:30.079461+0000","last_became_active":"2024-03-28T22:08:29.329503+0000","last_became_peered":"2024-03-28T22:08:29.329503+0000","last_unstale":"2024-03-28T22:09:30.079461+0000","last_undegraded":"2024-03-28T22:09:30.079461+0000","last_fullsized":"2024-03-28T22:09:30.079461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:50:05.686647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.102695+0000","last_change":"2024-03-28T22:08:29.957038+0000","last_active":"2024-03-28T22:09:59.102695+0000","last_peered":"2024-03-28T22:09:59.102695+0000","last_clean":"2024-03-28T22:09:59.102695+0000","last_became_active":"2024-03-28T22:08:29.956693+0000","last_became_peered":"2024-03-28T22:08:29.956693+0000","last_unstale":"2024-03-28T22:09:59.102695+0000","last_undegraded":"2024-03-28T22:09:59.102695+0000","last_fullsized":"2024-03-28T22:09:59.102695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:19:09.881431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.424647+0000","last_change":"2024-03-28T22:08:29.327356+0000","last_active":"2024-03-28T22:09:30.424647+0000","last_peered":"2024-03-28T22:09:30.424647+0000","last_clean":"2024-03-28T22:09:30.424647+0000","last_became_active":"2024-03-28T22:08:29.326947+0000","last_became_peered":"2024-03-28T22:08:29.326947+0000","last_unstale":"2024-03-28T22:09:30.424647+0000","last_undegraded":"2024-03-28T22:09:30.424647+0000","last_fullsized":"2024-03-28T22:09:30.424647+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:47:29.417465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.064033+0000","last_change":"2024-03-28T22:08:29.966505+0000","last_active":"2024-03-28T22:09:59.064033+0000","last_peered":"2024-03-28T22:09:59.064033+0000","last_clean":"2024-03-28T22:09:59.064033+0000","last_became_active":"2024-03-28T22:08:29.966057+0000","last_became_peered":"2024-03-28T22:08:29.966057+0000","last_unstale":"2024-03-28T22:09:59.064033+0000","last_undegraded":"2024-03-28T22:09:59.064033+0000","last_fullsized":"2024-03-28T22:09:59.064033+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:17:45.196585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.057477+0000","last_change":"2024-03-28T22:08:29.328897+0000","last_active":"2024-03-28T22:09:30.057477+0000","last_peered":"2024-03-28T22:09:30.057477+0000","last_clean":"2024-03-28T22:09:30.057477+0000","last_became_active":"2024-03-28T22:08:29.328390+0000","last_became_peered":"2024-03-28T22:08:29.328390+0000","last_unstale":"2024-03-28T22:09:30.057477+0000","last_undegraded":"2024-03-28T22:09:30.057477+0000","last_fullsized":"2024-03-28T22:09:30.057477+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:21:12.699503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.094866+0000","last_change":"2024-03-28T22:08:29.957212+0000","last_active":"2024-03-28T22:09:59.094866+0000","last_peered":"2024-03-28T22:09:59.094866+0000","last_clean":"2024-03-28T22:09:59.094866+0000","last_became_active":"2024-03-28T22:08:29.956215+0000","last_became_peered":"2024-03-28T22:08:29.956215+0000","last_unstale":"2024-03-28T22:09:59.094866+0000","last_undegraded":"2024-03-28T22:09:59.094866+0000","last_fullsized":"2024-03-28T22:09:59.094866+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:57:25.199993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.063151+0000","last_change":"2024-03-28T22:09:30.062742+0000","last_active":"2024-03-28T22:09:30.063151+0000","last_peered":"2024-03-28T22:09:30.063151+0000","last_clean":"2024-03-28T22:09:30.063151+0000","last_became_active":"2024-03-28T22:09:30.059356+0000","last_became_peered":"2024-03-28T22:09:30.059356+0000","last_unstale":"2024-03-28T22:09:30.063151+0000","last_undegraded":"2024-03-28T22:09:30.063151+0000","last_fullsized":"2024-03-28T22:09:30.063151+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:52:39.412153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.110714+0000","last_change":"2024-03-28T22:08:29.955023+0000","last_active":"2024-03-28T22:09:59.110714+0000","last_peered":"2024-03-28T22:09:59.110714+0000","last_clean":"2024-03-28T22:09:59.110714+0000","last_became_active":"2024-03-28T22:08:29.954675+0000","last_became_peered":"2024-03-28T22:08:29.954675+0000","last_unstale":"2024-03-28T22:09:59.110714+0000","last_undegraded":"2024-03-28T22:09:59.110714+0000","last_fullsized":"2024-03-28T22:09:59.110714+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:31.246452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058628+0000","last_change":"2024-03-28T22:08:29.329280+0000","last_active":"2024-03-28T22:09:30.058628+0000","last_peered":"2024-03-28T22:09:30.058628+0000","last_clean":"2024-03-28T22:09:30.058628+0000","last_became_active":"2024-03-28T22:08:29.328860+0000","last_became_peered":"2024-03-28T22:08:29.328860+0000","last_unstale":"2024-03-28T22:09:30.058628+0000","last_undegraded":"2024-03-28T22:09:30.058628+0000","last_fullsized":"2024-03-28T22:09:30.058628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:15:49.610026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.118448+0000","last_change":"2024-03-28T22:08:29.966224+0000","last_active":"2024-03-28T22:09:59.118448+0000","last_peered":"2024-03-28T22:09:59.118448+0000","last_clean":"2024-03-28T22:09:59.118448+0000","last_became_active":"2024-03-28T22:08:29.965636+0000","last_became_peered":"2024-03-28T22:08:29.965636+0000","last_unstale":"2024-03-28T22:09:59.118448+0000","last_undegraded":"2024-03-28T22:09:59.118448+0000","last_fullsized":"2024-03-28T22:09:59.118448+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:23:31.230980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058666+0000","last_change":"2024-03-28T22:08:29.335228+0000","last_active":"2024-03-28T22:09:30.058666+0000","last_peered":"2024-03-28T22:09:30.058666+0000","last_clean":"2024-03-28T22:09:30.058666+0000","last_became_active":"2024-03-28T22:08:29.334686+0000","last_became_peered":"2024-03-28T22:08:29.334686+0000","last_unstale":"2024-03-28T22:09:30.058666+0000","last_undegraded":"2024-03-28T22:09:30.058666+0000","last_fullsized":"2024-03-28T22:09:30.058666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:17:40.571206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.119960+0000","last_change":"2024-03-28T22:08:29.963912+0000","last_active":"2024-03-28T22:09:59.119960+0000","last_peered":"2024-03-28T22:09:59.119960+0000","last_clean":"2024-03-28T22:09:59.119960+0000","last_became_active":"2024-03-28T22:08:29.962740+0000","last_became_peered":"2024-03-28T22:08:29.962740+0000","last_unstale":"2024-03-28T22:09:59.119960+0000","last_undegraded":"2024-03-28T22:09:59.119960+0000","last_fullsized":"2024-03-28T22:09:59.119960+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:50:25.756923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.060333+0000","last_change":"2024-03-28T22:08:29.325767+0000","last_active":"2024-03-28T22:09:30.060333+0000","last_peered":"2024-03-28T22:09:30.060333+0000","last_clean":"2024-03-28T22:09:30.060333+0000","last_became_active":"2024-03-28T22:08:29.325430+0000","last_became_peered":"2024-03-28T22:08:29.325430+0000","last_unstale":"2024-03-28T22:09:30.060333+0000","last_undegraded":"2024-03-28T22:09:30.060333+0000","last_fullsized":"2024-03-28T22:09:30.060333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:01:13.663163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.114614+0000","last_change":"2024-03-28T22:08:29.960158+0000","last_active":"2024-03-28T22:09:59.114614+0000","last_peered":"2024-03-28T22:09:59.114614+0000","last_clean":"2024-03-28T22:09:59.114614+0000","last_became_active":"2024-03-28T22:08:29.959792+0000","last_became_peered":"2024-03-28T22:08:29.959792+0000","last_unstale":"2024-03-28T22:09:59.114614+0000","last_undegraded":"2024-03-28T22:09:59.114614+0000","last_fullsized":"2024-03-28T22:09:59.114614+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:28:49.135551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.293763+0000","last_change":"2024-03-28T22:08:29.328547+0000","last_active":"2024-03-28T22:09:30.293763+0000","last_peered":"2024-03-28T22:09:30.293763+0000","last_clean":"2024-03-28T22:09:30.293763+0000","last_became_active":"2024-03-28T22:08:29.327910+0000","last_became_peered":"2024-03-28T22:08:29.327910+0000","last_unstale":"2024-03-28T22:09:30.293763+0000","last_undegraded":"2024-03-28T22:09:30.293763+0000","last_fullsized":"2024-03-28T22:09:30.293763+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:06:13.980071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.105047+0000","last_change":"2024-03-28T22:08:29.963904+0000","last_active":"2024-03-28T22:09:59.105047+0000","last_peered":"2024-03-28T22:09:59.105047+0000","last_clean":"2024-03-28T22:09:59.105047+0000","last_became_active":"2024-03-28T22:08:29.963536+0000","last_became_peered":"2024-03-28T22:08:29.963536+0000","last_unstale":"2024-03-28T22:09:59.105047+0000","last_undegraded":"2024-03-28T22:09:59.105047+0000","last_fullsized":"2024-03-28T22:09:59.105047+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:59:43.376792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.100207+0000","last_change":"2024-03-28T22:08:29.962357+0000","last_active":"2024-03-28T22:09:59.100207+0000","last_peered":"2024-03-28T22:09:59.100207+0000","last_clean":"2024-03-28T22:09:59.100207+0000","last_became_active":"2024-03-28T22:08:29.961998+0000","last_became_peered":"2024-03-28T22:08:29.961998+0000","last_unstale":"2024-03-28T22:09:59.100207+0000","last_undegraded":"2024-03-28T22:09:59.100207+0000","last_fullsized":"2024-03-28T22:09:59.100207+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:40:47.465889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.108275+0000","last_change":"2024-03-28T22:08:29.961326+0000","last_active":"2024-03-28T22:09:59.108275+0000","last_peered":"2024-03-28T22:09:59.108275+0000","last_clean":"2024-03-28T22:09:59.108275+0000","last_became_active":"2024-03-28T22:08:29.960316+0000","last_became_peered":"2024-03-28T22:08:29.960316+0000","last_unstale":"2024-03-28T22:09:59.108275+0000","last_undegraded":"2024-03-28T22:09:59.108275+0000","last_fullsized":"2024-03-28T22:09:59.108275+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:37:55.773686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.099312+0000","last_change":"2024-03-28T22:08:29.956272+0000","last_active":"2024-03-28T22:09:59.099312+0000","last_peered":"2024-03-28T22:09:59.099312+0000","last_clean":"2024-03-28T22:09:59.099312+0000","last_became_active":"2024-03-28T22:08:29.955875+0000","last_became_peered":"2024-03-28T22:08:29.955875+0000","last_unstale":"2024-03-28T22:09:59.099312+0000","last_undegraded":"2024-03-28T22:09:59.099312+0000","last_fullsized":"2024-03-28T22:09:59.099312+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:17:33.062615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.103567+0000","last_change":"2024-03-28T22:09:30.294754+0000","last_active":"2024-03-28T22:09:59.103567+0000","last_peered":"2024-03-28T22:09:59.103567+0000","last_clean":"2024-03-28T22:09:59.103567+0000","last_became_active":"2024-03-28T22:09:30.294341+0000","last_became_peered":"2024-03-28T22:09:30.294341+0000","last_unstale":"2024-03-28T22:09:59.103567+0000","last_undegraded":"2024-03-28T22:09:59.103567+0000","last_fullsized":"2024-03-28T22:09:59.103567+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:25:46.017373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.105840+0000","last_change":"2024-03-28T22:08:29.960464+0000","last_active":"2024-03-28T22:09:59.105840+0000","last_peered":"2024-03-28T22:09:59.105840+0000","last_clean":"2024-03-28T22:09:59.105840+0000","last_became_active":"2024-03-28T22:08:29.960111+0000","last_became_peered":"2024-03-28T22:08:29.960111+0000","last_unstale":"2024-03-28T22:09:59.105840+0000","last_undegraded":"2024-03-28T22:09:59.105840+0000","last_fullsized":"2024-03-28T22:09:59.105840+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:32:33.533199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.100620+0000","last_change":"2024-03-28T22:08:29.960045+0000","last_active":"2024-03-28T22:09:59.100620+0000","last_peered":"2024-03-28T22:09:59.100620+0000","last_clean":"2024-03-28T22:09:59.100620+0000","last_became_active":"2024-03-28T22:08:29.959645+0000","last_became_peered":"2024-03-28T22:08:29.959645+0000","last_unstale":"2024-03-28T22:09:59.100620+0000","last_undegraded":"2024-03-28T22:09:59.100620+0000","last_fullsized":"2024-03-28T22:09:59.100620+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:34:56.202257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.114146+0000","last_change":"2024-03-28T22:08:29.958736+0000","last_active":"2024-03-28T22:09:59.114146+0000","last_peered":"2024-03-28T22:09:59.114146+0000","last_clean":"2024-03-28T22:09:59.114146+0000","last_became_active":"2024-03-28T22:08:29.958387+0000","last_became_peered":"2024-03-28T22:08:29.958387+0000","last_unstale":"2024-03-28T22:09:59.114146+0000","last_undegraded":"2024-03-28T22:09:59.114146+0000","last_fullsized":"2024-03-28T22:09:59.114146+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:07:54.829166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.101041+0000","last_change":"2024-03-28T22:08:29.961373+0000","last_active":"2024-03-28T22:09:59.101041+0000","last_peered":"2024-03-28T22:09:59.101041+0000","last_clean":"2024-03-28T22:09:59.101041+0000","last_became_active":"2024-03-28T22:08:29.959751+0000","last_became_peered":"2024-03-28T22:08:29.959751+0000","last_unstale":"2024-03-28T22:09:59.101041+0000","last_undegraded":"2024-03-28T22:09:59.101041+0000","last_fullsized":"2024-03-28T22:09:59.101041+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:34:18.680552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.098710+0000","last_change":"2024-03-28T22:08:29.962721+0000","last_active":"2024-03-28T22:09:59.098710+0000","last_peered":"2024-03-28T22:09:59.098710+0000","last_clean":"2024-03-28T22:09:59.098710+0000","last_became_active":"2024-03-28T22:08:29.961349+0000","last_became_peered":"2024-03-28T22:08:29.961349+0000","last_unstale":"2024-03-28T22:09:59.098710+0000","last_undegraded":"2024-03-28T22:09:59.098710+0000","last_fullsized":"2024-03-28T22:09:59.098710+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:35:56.535055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.097778+0000","last_change":"2024-03-28T22:08:29.960068+0000","last_active":"2024-03-28T22:09:59.097778+0000","last_peered":"2024-03-28T22:09:59.097778+0000","last_clean":"2024-03-28T22:09:59.097778+0000","last_became_active":"2024-03-28T22:08:29.959716+0000","last_became_peered":"2024-03-28T22:08:29.959716+0000","last_unstale":"2024-03-28T22:09:59.097778+0000","last_undegraded":"2024-03-28T22:09:59.097778+0000","last_fullsized":"2024-03-28T22:09:59.097778+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:18:38.055322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.121200+0000","last_change":"2024-03-28T22:08:29.957920+0000","last_active":"2024-03-28T22:09:59.121200+0000","last_peered":"2024-03-28T22:09:59.121200+0000","last_clean":"2024-03-28T22:09:59.121200+0000","last_became_active":"2024-03-28T22:08:29.957541+0000","last_became_peered":"2024-03-28T22:08:29.957541+0000","last_unstale":"2024-03-28T22:09:59.121200+0000","last_undegraded":"2024-03-28T22:09:59.121200+0000","last_fullsized":"2024-03-28T22:09:59.121200+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:45:05.658745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.115831+0000","last_change":"2024-03-28T22:08:29.953589+0000","last_active":"2024-03-28T22:09:59.115831+0000","last_peered":"2024-03-28T22:09:59.115831+0000","last_clean":"2024-03-28T22:09:59.115831+0000","last_became_active":"2024-03-28T22:08:29.953253+0000","last_became_peered":"2024-03-28T22:08:29.953253+0000","last_unstale":"2024-03-28T22:09:59.115831+0000","last_undegraded":"2024-03-28T22:09:59.115831+0000","last_fullsized":"2024-03-28T22:09:59.115831+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:35.720708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.109906+0000","last_change":"2024-03-28T22:08:29.966235+0000","last_active":"2024-03-28T22:09:59.109906+0000","last_peered":"2024-03-28T22:09:59.109906+0000","last_clean":"2024-03-28T22:09:59.109906+0000","last_became_active":"2024-03-28T22:08:29.965830+0000","last_became_peered":"2024-03-28T22:08:29.965830+0000","last_unstale":"2024-03-28T22:09:59.109906+0000","last_undegraded":"2024-03-28T22:09:59.109906+0000","last_fullsized":"2024-03-28T22:09:59.109906+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:04:27.094743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":64925,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":672,"num_read_kb":404,"num_write":171,"num_write_kb":119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":421888,"data_stored":199366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351012,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":950272,"data_stored":459161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612648,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28624,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716528,"statfs":{"total":95995035648,"available":95965724672,"internally_reserved":0,"allocated":1646592,"data_stored":1110335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776173,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":459397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037810,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":459317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299448,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36860,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708292,"statfs":{"total":95995035648,"available":95957291008,"internally_reserved":0,"allocated":1560576,"data_stored":1051920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462972,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40428,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704724,"statfs":{"total":95995035648,"available":95953637376,"internally_reserved":0,"allocated":954368,"data_stored":458786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":517740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":460057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411409,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1044480,"data_stored":526085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149771,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":459097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888135,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40432,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704720,"statfs":{"total":95995035648,"available":95953633280,"internally_reserved":0,"allocated":958464,"data_stored":460146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724610,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28552,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716600,"statfs":{"total":95995035648,"available":95965798400,"internally_reserved":0,"allocated":1572864,"data_stored":1051072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":59469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":67506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":61579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T22:10:24.088 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-03-28T22:10:24.089 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T22:10:24.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.227+0000 7f6bcb1e2640 1 -- 172.21.15.16:0/1126006731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bc41022a0 msgr2=0x7f6bc4108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:24.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.227+0000 7f6bcb1e2640 1 --2- 172.21.15.16:0/1126006731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bc41022a0 0x7f6bc4108b30 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc009ca0 tx=0x7f6bbc02c0f0 comp rx=0 tx=0).stop 2024-03-28T22:10:24.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.227+0000 7f6bcb1e2640 1 -- 172.21.15.16:0/1126006731 shutdown_connections 2024-03-28T22:10:24.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.227+0000 7f6bcb1e2640 1 --2- 172.21.15.16:0/1126006731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bc41022a0 0x7f6bc4108b30 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:24.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.227+0000 7f6bcb1e2640 1 --2- 172.21.15.16:0/1126006731 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6bc41018e0 0x7f6bc4101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:24.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.227+0000 7f6bcb1e2640 1 --2- 172.21.15.16:0/1126006731 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6bc41006f0 0x7f6bc4100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:24.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.227+0000 7f6bcb1e2640 1 -- 172.21.15.16:0/1126006731 >> 172.21.15.16:0/1126006731 conn(0x7f6bc40fbc80 msgr2=0x7f6bc40fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:24.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.227+0000 7f6bcb1e2640 1 -- 172.21.15.16:0/1126006731 shutdown_connections 2024-03-28T22:10:24.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.227+0000 7f6bcb1e2640 1 -- 172.21.15.16:0/1126006731 wait complete. 2024-03-28T22:10:24.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.227+0000 7f6bcb1e2640 1 Processor -- start 2024-03-28T22:10:24.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.228+0000 7f6bcb1e2640 1 -- start start 2024-03-28T22:10:24.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.228+0000 7f6bcb1e2640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6bc41006f0 0x7f6bc406a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:24.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.228+0000 7f6bcb1e2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bc41018e0 0x7f6bc419c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:24.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.228+0000 7f6bcb1e2640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6bc41022a0 0x7f6bc419ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:24.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.228+0000 7f6bcb1e2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6bc410c360 con 0x7f6bc41018e0 2024-03-28T22:10:24.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.228+0000 7f6bcb1e2640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f6bc410c1e0 con 0x7f6bc41006f0 2024-03-28T22:10:24.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.228+0000 7f6bcb1e2640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6bc410c4e0 con 0x7f6bc41022a0 2024-03-28T22:10:24.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.228+0000 7f6bc3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bc41018e0 0x7f6bc419c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:24.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.228+0000 7f6bc3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bc41018e0 0x7f6bc419c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44504/0 (socket says 172.21.15.16:44504) 2024-03-28T22:10:24.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.228+0000 7f6bc3fff640 1 -- 172.21.15.16:0/413512151 learned_addr learned my addr 172.21.15.16:0/413512151 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:24.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.228+0000 7f6bc8f7e640 1 --2- 172.21.15.16:0/413512151 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6bc41006f0 0x7f6bc406a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:24.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.228+0000 7f6bc977f640 1 --2- 172.21.15.16:0/413512151 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6bc41022a0 0x7f6bc419ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:24.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.228+0000 7f6bc3fff640 1 -- 172.21.15.16:0/413512151 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6bc41022a0 msgr2=0x7f6bc419ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:24.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.228+0000 7f6bc3fff640 1 --2- 172.21.15.16:0/413512151 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6bc41022a0 0x7f6bc419ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:24.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.228+0000 7f6bc3fff640 1 -- 172.21.15.16:0/413512151 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6bc41006f0 msgr2=0x7f6bc406a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:24.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.228+0000 7f6bc3fff640 1 --2- 172.21.15.16:0/413512151 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6bc41006f0 0x7f6bc406a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:24.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.228+0000 7f6bc3fff640 1 -- 172.21.15.16:0/413512151 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bc419f720 con 0x7f6bc41018e0 2024-03-28T22:10:24.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.228+0000 7f6bc977f640 1 --2- 172.21.15.16:0/413512151 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6bc41022a0 0x7f6bc419ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:10:24.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.228+0000 7f6bc3fff640 1 --2- 172.21.15.16:0/413512151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bc41018e0 0x7f6bc419c7f0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f6bb000b4a0 tx=0x7f6bb000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:24.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.228+0000 7f6bc1ffb640 1 -- 172.21.15.16:0/413512151 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bb0014070 con 0x7f6bc41018e0 2024-03-28T22:10:24.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.228+0000 7f6bc1ffb640 1 -- 172.21.15.16:0/413512151 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6bb0010af0 con 0x7f6bc41018e0 2024-03-28T22:10:24.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.228+0000 7f6bcb1e2640 1 -- 172.21.15.16:0/413512151 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bc419f9b0 con 0x7f6bc41018e0 2024-03-28T22:10:24.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.228+0000 7f6bc1ffb640 1 -- 172.21.15.16:0/413512151 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bb0017d10 con 0x7f6bc41018e0 2024-03-28T22:10:24.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.229+0000 7f6bcb1e2640 1 -- 172.21.15.16:0/413512151 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bc419ff00 con 0x7f6bc41018e0 2024-03-28T22:10:24.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.230+0000 7f6bcb1e2640 1 -- 172.21.15.16:0/413512151 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bc41079b0 con 0x7f6bc41018e0 2024-03-28T22:10:24.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.231+0000 7f6bc1ffb640 1 -- 172.21.15.16:0/413512151 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7f6bb0017070 con 0x7f6bc41018e0 2024-03-28T22:10:24.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.231+0000 7f6bc1ffb640 1 --2- 172.21.15.16:0/413512151 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f6bb40a1e90 0x7f6bb40a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:24.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.232+0000 7f6bc1ffb640 1 -- 172.21.15.16:0/413512151 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f6bb00cd110 con 0x7f6bc41018e0 2024-03-28T22:10:24.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.232+0000 7f6bc8f7e640 1 --2- 172.21.15.16:0/413512151 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f6bb40a1e90 0x7f6bb40a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:24.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.232+0000 7f6bc8f7e640 1 --2- 172.21.15.16:0/413512151 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f6bb40a1e90 0x7f6bb40a4350 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6bb80095f0 tx=0x7f6bb8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:24.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.236+0000 7f6bc1ffb640 1 -- 172.21.15.16:0/413512151 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6bb009a520 con 0x7f6bc41018e0 2024-03-28T22:10:24.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.417+0000 7f6bcb1e2640 1 -- 172.21.15.16:0/413512151 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6bc4061b90 con 0x7f6bb40a1e90 2024-03-28T22:10:24.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.424+0000 7f6bc1ffb640 1 -- 172.21.15.16:0/413512151 <== mgr.14152 v2:172.21.15.16:6800/21986004 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258824 (secure 0 0 0) 0x7f6bc4061b90 con 0x7f6bb40a1e90 2024-03-28T22:10:24.430 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:10:24.431 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-03-28T22:10:24.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.427+0000 7f6bcb1e2640 1 -- 172.21.15.16:0/413512151 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f6bb40a1e90 msgr2=0x7f6bb40a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:24.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.427+0000 7f6bcb1e2640 1 --2- 172.21.15.16:0/413512151 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f6bb40a1e90 0x7f6bb40a4350 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6bb80095f0 tx=0x7f6bb8005f90 comp rx=0 tx=0).stop 2024-03-28T22:10:24.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.427+0000 7f6bcb1e2640 1 -- 172.21.15.16:0/413512151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bc41018e0 msgr2=0x7f6bc419c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:24.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.427+0000 7f6bcb1e2640 1 --2- 172.21.15.16:0/413512151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bc41018e0 0x7f6bc419c7f0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f6bb000b4a0 tx=0x7f6bb000b960 comp rx=0 tx=0).stop 2024-03-28T22:10:24.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.427+0000 7f6bcb1e2640 1 -- 172.21.15.16:0/413512151 shutdown_connections 2024-03-28T22:10:24.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.427+0000 7f6bcb1e2640 1 --2- 172.21.15.16:0/413512151 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f6bb40a1e90 0x7f6bb40a4350 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:24.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.427+0000 7f6bcb1e2640 1 --2- 172.21.15.16:0/413512151 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6bc41022a0 0x7f6bc419ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:24.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.427+0000 7f6bcb1e2640 1 --2- 172.21.15.16:0/413512151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bc41018e0 0x7f6bc419c7f0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:24.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.427+0000 7f6bcb1e2640 1 --2- 172.21.15.16:0/413512151 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6bc41006f0 0x7f6bc406a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:24.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.427+0000 7f6bcb1e2640 1 -- 172.21.15.16:0/413512151 >> 172.21.15.16:0/413512151 conn(0x7f6bc40fbc80 msgr2=0x7f6bc4104b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:24.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.427+0000 7f6bcb1e2640 1 -- 172.21.15.16:0/413512151 shutdown_connections 2024-03-28T22:10:24.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.427+0000 7f6bcb1e2640 1 -- 172.21.15.16:0/413512151 wait complete. 2024-03-28T22:10:24.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:24 smithi016 ceph-mon[32374]: pgmap v345: 97 pgs: 97 active+clean; 640 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-28T22:10:24.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:24 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/462332625' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:10:24.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:24 smithi016 ceph-mon[32374]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:10:24.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:24 smithi016 ceph-mon[32374]: from='client.15339 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:10:24.441 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-03-28T22:10:23.100856+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":655527,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":808,"num_read_kb":523,"num_write":354,"num_write_kb":3059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":787,"ondisk_log_size":787,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":383216,"kb_used_data":13264,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124558608,"statfs":{"total":1151940427776,"available":1151548014592,"internally_reserved":0,"allocated":13582336,"data_stored":7473113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1090,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":72,"num_read_kb":37,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002694"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.119037+0000","last_change":"2024-03-28T22:08:29.962664+0000","last_active":"2024-03-28T22:09:59.119037+0000","last_peered":"2024-03-28T22:09:59.119037+0000","last_clean":"2024-03-28T22:09:59.119037+0000","last_became_active":"2024-03-28T22:08:29.962029+0000","last_became_peered":"2024-03-28T22:08:29.962029+0000","last_unstale":"2024-03-28T22:09:59.119037+0000","last_undegraded":"2024-03-28T22:09:59.119037+0000","last_fullsized":"2024-03-28T22:09:59.119037+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:20:42.832304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.099587+0000","last_change":"2024-03-28T22:08:29.965617+0000","last_active":"2024-03-28T22:09:59.099587+0000","last_peered":"2024-03-28T22:09:59.099587+0000","last_clean":"2024-03-28T22:09:59.099587+0000","last_became_active":"2024-03-28T22:08:29.965248+0000","last_became_peered":"2024-03-28T22:08:29.965248+0000","last_unstale":"2024-03-28T22:09:59.099587+0000","last_undegraded":"2024-03-28T22:09:59.099587+0000","last_fullsized":"2024-03-28T22:09:59.099587+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:37:46.703768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.119356+0000","last_change":"2024-03-28T22:08:29.958452+0000","last_active":"2024-03-28T22:09:59.119356+0000","last_peered":"2024-03-28T22:09:59.119356+0000","last_clean":"2024-03-28T22:09:59.119356+0000","last_became_active":"2024-03-28T22:08:29.958089+0000","last_became_peered":"2024-03-28T22:08:29.958089+0000","last_unstale":"2024-03-28T22:09:59.119356+0000","last_undegraded":"2024-03-28T22:09:59.119356+0000","last_fullsized":"2024-03-28T22:09:59.119356+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:10:44.084925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.104325+0000","last_change":"2024-03-28T22:08:29.964812+0000","last_active":"2024-03-28T22:09:59.104325+0000","last_peered":"2024-03-28T22:09:59.104325+0000","last_clean":"2024-03-28T22:09:59.104325+0000","last_became_active":"2024-03-28T22:08:29.963872+0000","last_became_peered":"2024-03-28T22:08:29.963872+0000","last_unstale":"2024-03-28T22:09:59.104325+0000","last_undegraded":"2024-03-28T22:09:59.104325+0000","last_fullsized":"2024-03-28T22:09:59.104325+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:08.491078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.080889+0000","last_change":"2024-03-28T22:08:29.965564+0000","last_active":"2024-03-28T22:09:59.080889+0000","last_peered":"2024-03-28T22:09:59.080889+0000","last_clean":"2024-03-28T22:09:59.080889+0000","last_became_active":"2024-03-28T22:08:29.962636+0000","last_became_peered":"2024-03-28T22:08:29.962636+0000","last_unstale":"2024-03-28T22:09:59.080889+0000","last_undegraded":"2024-03-28T22:09:59.080889+0000","last_fullsized":"2024-03-28T22:09:59.080889+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:03:50.903231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.111439+0000","last_change":"2024-03-28T22:08:29.962091+0000","last_active":"2024-03-28T22:09:59.111439+0000","last_peered":"2024-03-28T22:09:59.111439+0000","last_clean":"2024-03-28T22:09:59.111439+0000","last_became_active":"2024-03-28T22:08:29.961735+0000","last_became_peered":"2024-03-28T22:08:29.961735+0000","last_unstale":"2024-03-28T22:09:59.111439+0000","last_undegraded":"2024-03-28T22:09:59.111439+0000","last_fullsized":"2024-03-28T22:09:59.111439+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:40:09.853265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.108285+0000","last_change":"2024-03-28T22:08:29.961089+0000","last_active":"2024-03-28T22:09:59.108285+0000","last_peered":"2024-03-28T22:09:59.108285+0000","last_clean":"2024-03-28T22:09:59.108285+0000","last_became_active":"2024-03-28T22:08:29.960711+0000","last_became_peered":"2024-03-28T22:08:29.960711+0000","last_unstale":"2024-03-28T22:09:59.108285+0000","last_undegraded":"2024-03-28T22:09:59.108285+0000","last_fullsized":"2024-03-28T22:09:59.108285+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:05:14.617718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.110495+0000","last_change":"2024-03-28T22:08:29.965142+0000","last_active":"2024-03-28T22:09:59.110495+0000","last_peered":"2024-03-28T22:09:59.110495+0000","last_clean":"2024-03-28T22:09:59.110495+0000","last_became_active":"2024-03-28T22:08:29.964730+0000","last_became_peered":"2024-03-28T22:08:29.964730+0000","last_unstale":"2024-03-28T22:09:59.110495+0000","last_undegraded":"2024-03-28T22:09:59.110495+0000","last_fullsized":"2024-03-28T22:09:59.110495+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:23:43.229681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.115178+0000","last_change":"2024-03-28T22:08:29.958984+0000","last_active":"2024-03-28T22:09:59.115178+0000","last_peered":"2024-03-28T22:09:59.115178+0000","last_clean":"2024-03-28T22:09:59.115178+0000","last_became_active":"2024-03-28T22:08:29.958583+0000","last_became_peered":"2024-03-28T22:08:29.958583+0000","last_unstale":"2024-03-28T22:09:59.115178+0000","last_undegraded":"2024-03-28T22:09:59.115178+0000","last_fullsized":"2024-03-28T22:09:59.115178+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:05:11.361719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.113771+0000","last_change":"2024-03-28T22:08:29.962774+0000","last_active":"2024-03-28T22:09:59.113771+0000","last_peered":"2024-03-28T22:09:59.113771+0000","last_clean":"2024-03-28T22:09:59.113771+0000","last_became_active":"2024-03-28T22:08:29.961222+0000","last_became_peered":"2024-03-28T22:08:29.961222+0000","last_unstale":"2024-03-28T22:09:59.113771+0000","last_undegraded":"2024-03-28T22:09:59.113771+0000","last_fullsized":"2024-03-28T22:09:59.113771+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:47:05.310483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.119577+0000","last_change":"2024-03-28T22:08:29.964041+0000","last_active":"2024-03-28T22:09:59.119577+0000","last_peered":"2024-03-28T22:09:59.119577+0000","last_clean":"2024-03-28T22:09:59.119577+0000","last_became_active":"2024-03-28T22:08:29.963360+0000","last_became_peered":"2024-03-28T22:08:29.963360+0000","last_unstale":"2024-03-28T22:09:59.119577+0000","last_undegraded":"2024-03-28T22:09:59.119577+0000","last_fullsized":"2024-03-28T22:09:59.119577+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:32.921916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.095082+0000","last_change":"2024-03-28T22:08:29.966449+0000","last_active":"2024-03-28T22:09:59.095082+0000","last_peered":"2024-03-28T22:09:59.095082+0000","last_clean":"2024-03-28T22:09:59.095082+0000","last_became_active":"2024-03-28T22:08:29.966036+0000","last_became_peered":"2024-03-28T22:08:29.966036+0000","last_unstale":"2024-03-28T22:09:59.095082+0000","last_undegraded":"2024-03-28T22:09:59.095082+0000","last_fullsized":"2024-03-28T22:09:59.095082+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:12:15.057713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.113074+0000","last_change":"2024-03-28T22:08:29.958924+0000","last_active":"2024-03-28T22:09:59.113074+0000","last_peered":"2024-03-28T22:09:59.113074+0000","last_clean":"2024-03-28T22:09:59.113074+0000","last_became_active":"2024-03-28T22:08:29.958601+0000","last_became_peered":"2024-03-28T22:08:29.958601+0000","last_unstale":"2024-03-28T22:09:59.113074+0000","last_undegraded":"2024-03-28T22:09:59.113074+0000","last_fullsized":"2024-03-28T22:09:59.113074+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:53:37.070276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.098999+0000","last_change":"2024-03-28T22:08:29.965045+0000","last_active":"2024-03-28T22:09:59.098999+0000","last_peered":"2024-03-28T22:09:59.098999+0000","last_clean":"2024-03-28T22:09:59.098999+0000","last_became_active":"2024-03-28T22:08:29.964710+0000","last_became_peered":"2024-03-28T22:08:29.964710+0000","last_unstale":"2024-03-28T22:09:59.098999+0000","last_undegraded":"2024-03-28T22:09:59.098999+0000","last_fullsized":"2024-03-28T22:09:59.098999+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:03:11.491213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.104261+0000","last_change":"2024-03-28T22:08:29.957940+0000","last_active":"2024-03-28T22:09:59.104261+0000","last_peered":"2024-03-28T22:09:59.104261+0000","last_clean":"2024-03-28T22:09:59.104261+0000","last_became_active":"2024-03-28T22:08:29.957587+0000","last_became_peered":"2024-03-28T22:08:29.957587+0000","last_unstale":"2024-03-28T22:09:59.104261+0000","last_undegraded":"2024-03-28T22:09:59.104261+0000","last_fullsized":"2024-03-28T22:09:59.104261+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:03:58.557806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.120996+0000","last_change":"2024-03-28T22:08:29.957397+0000","last_active":"2024-03-28T22:09:59.120996+0000","last_peered":"2024-03-28T22:09:59.120996+0000","last_clean":"2024-03-28T22:09:59.120996+0000","last_became_active":"2024-03-28T22:08:29.957054+0000","last_became_peered":"2024-03-28T22:08:29.957054+0000","last_unstale":"2024-03-28T22:09:59.120996+0000","last_undegraded":"2024-03-28T22:09:59.120996+0000","last_fullsized":"2024-03-28T22:09:59.120996+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:35:28.500561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.119267+0000","last_change":"2024-03-28T22:08:29.963343+0000","last_active":"2024-03-28T22:09:59.119267+0000","last_peered":"2024-03-28T22:09:59.119267+0000","last_clean":"2024-03-28T22:09:59.119267+0000","last_became_active":"2024-03-28T22:08:29.962926+0000","last_became_peered":"2024-03-28T22:08:29.962926+0000","last_unstale":"2024-03-28T22:09:59.119267+0000","last_undegraded":"2024-03-28T22:09:59.119267+0000","last_fullsized":"2024-03-28T22:09:59.119267+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:15:39.133930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.094891+0000","last_change":"2024-03-28T22:08:29.955903+0000","last_active":"2024-03-28T22:09:59.094891+0000","last_peered":"2024-03-28T22:09:59.094891+0000","last_clean":"2024-03-28T22:09:59.094891+0000","last_became_active":"2024-03-28T22:08:29.955460+0000","last_became_peered":"2024-03-28T22:08:29.955460+0000","last_unstale":"2024-03-28T22:09:59.094891+0000","last_undegraded":"2024-03-28T22:09:59.094891+0000","last_fullsized":"2024-03-28T22:09:59.094891+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:36:12.961475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.110397+0000","last_change":"2024-03-28T22:08:29.954279+0000","last_active":"2024-03-28T22:09:59.110397+0000","last_peered":"2024-03-28T22:09:59.110397+0000","last_clean":"2024-03-28T22:09:59.110397+0000","last_became_active":"2024-03-28T22:08:29.953869+0000","last_became_peered":"2024-03-28T22:08:29.953869+0000","last_unstale":"2024-03-28T22:09:59.110397+0000","last_undegraded":"2024-03-28T22:09:59.110397+0000","last_fullsized":"2024-03-28T22:09:59.110397+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:05:52.247391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.054735+0000","last_change":"2024-03-28T22:08:29.327644+0000","last_active":"2024-03-28T22:09:30.054735+0000","last_peered":"2024-03-28T22:09:30.054735+0000","last_clean":"2024-03-28T22:09:30.054735+0000","last_became_active":"2024-03-28T22:08:29.327325+0000","last_became_peered":"2024-03-28T22:08:29.327325+0000","last_unstale":"2024-03-28T22:09:30.054735+0000","last_undegraded":"2024-03-28T22:09:30.054735+0000","last_fullsized":"2024-03-28T22:09:30.054735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:26:54.782703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.102346+0000","last_change":"2024-03-28T22:08:29.962905+0000","last_active":"2024-03-28T22:09:59.102346+0000","last_peered":"2024-03-28T22:09:59.102346+0000","last_clean":"2024-03-28T22:09:59.102346+0000","last_became_active":"2024-03-28T22:08:29.962559+0000","last_became_peered":"2024-03-28T22:08:29.962559+0000","last_unstale":"2024-03-28T22:09:59.102346+0000","last_undegraded":"2024-03-28T22:09:59.102346+0000","last_fullsized":"2024-03-28T22:09:59.102346+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:02.347584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058126+0000","last_change":"2024-03-28T22:08:29.334871+0000","last_active":"2024-03-28T22:09:30.058126+0000","last_peered":"2024-03-28T22:09:30.058126+0000","last_clean":"2024-03-28T22:09:30.058126+0000","last_became_active":"2024-03-28T22:08:29.334565+0000","last_became_peered":"2024-03-28T22:08:29.334565+0000","last_unstale":"2024-03-28T22:09:30.058126+0000","last_undegraded":"2024-03-28T22:09:30.058126+0000","last_fullsized":"2024-03-28T22:09:30.058126+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:31:19.333496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.119365+0000","last_change":"2024-03-28T22:08:29.957622+0000","last_active":"2024-03-28T22:09:59.119365+0000","last_peered":"2024-03-28T22:09:59.119365+0000","last_clean":"2024-03-28T22:09:59.119365+0000","last_became_active":"2024-03-28T22:08:29.957208+0000","last_became_peered":"2024-03-28T22:08:29.957208+0000","last_unstale":"2024-03-28T22:09:59.119365+0000","last_undegraded":"2024-03-28T22:09:59.119365+0000","last_fullsized":"2024-03-28T22:09:59.119365+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:49:44.684203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058576+0000","last_change":"2024-03-28T22:08:29.330291+0000","last_active":"2024-03-28T22:09:30.058576+0000","last_peered":"2024-03-28T22:09:30.058576+0000","last_clean":"2024-03-28T22:09:30.058576+0000","last_became_active":"2024-03-28T22:08:29.328955+0000","last_became_peered":"2024-03-28T22:08:29.328955+0000","last_unstale":"2024-03-28T22:09:30.058576+0000","last_undegraded":"2024-03-28T22:09:30.058576+0000","last_fullsized":"2024-03-28T22:09:30.058576+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:10.838059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.116812+0000","last_change":"2024-03-28T22:08:29.966681+0000","last_active":"2024-03-28T22:09:59.116812+0000","last_peered":"2024-03-28T22:09:59.116812+0000","last_clean":"2024-03-28T22:09:59.116812+0000","last_became_active":"2024-03-28T22:08:29.966023+0000","last_became_peered":"2024-03-28T22:08:29.966023+0000","last_unstale":"2024-03-28T22:09:59.116812+0000","last_undegraded":"2024-03-28T22:09:59.116812+0000","last_fullsized":"2024-03-28T22:09:59.116812+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:04:43.281259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058443+0000","last_change":"2024-03-28T22:08:29.329026+0000","last_active":"2024-03-28T22:09:30.058443+0000","last_peered":"2024-03-28T22:09:30.058443+0000","last_clean":"2024-03-28T22:09:30.058443+0000","last_became_active":"2024-03-28T22:08:29.327570+0000","last_became_peered":"2024-03-28T22:08:29.327570+0000","last_unstale":"2024-03-28T22:09:30.058443+0000","last_undegraded":"2024-03-28T22:09:30.058443+0000","last_fullsized":"2024-03-28T22:09:30.058443+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:53:54.779073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.106272+0000","last_change":"2024-03-28T22:08:29.966455+0000","last_active":"2024-03-28T22:09:59.106272+0000","last_peered":"2024-03-28T22:09:59.106272+0000","last_clean":"2024-03-28T22:09:59.106272+0000","last_became_active":"2024-03-28T22:08:29.964451+0000","last_became_peered":"2024-03-28T22:08:29.964451+0000","last_unstale":"2024-03-28T22:09:59.106272+0000","last_undegraded":"2024-03-28T22:09:59.106272+0000","last_fullsized":"2024-03-28T22:09:59.106272+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:46:22.811146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'8","reported_seq":138,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:10:20.499412+0000","last_change":"2024-03-28T22:08:29.327397+0000","last_active":"2024-03-28T22:10:20.499412+0000","last_peered":"2024-03-28T22:10:20.499412+0000","last_clean":"2024-03-28T22:10:20.499412+0000","last_became_active":"2024-03-28T22:08:29.327009+0000","last_became_peered":"2024-03-28T22:08:29.327009+0000","last_unstale":"2024-03-28T22:10:20.499412+0000","last_undegraded":"2024-03-28T22:10:20.499412+0000","last_fullsized":"2024-03-28T22:10:20.499412+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:38:51.951934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1545,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":111,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.096251+0000","last_change":"2024-03-28T22:08:29.969821+0000","last_active":"2024-03-28T22:09:59.096251+0000","last_peered":"2024-03-28T22:09:59.096251+0000","last_clean":"2024-03-28T22:09:59.096251+0000","last_became_active":"2024-03-28T22:08:29.969386+0000","last_became_peered":"2024-03-28T22:08:29.969386+0000","last_unstale":"2024-03-28T22:09:59.096251+0000","last_undegraded":"2024-03-28T22:09:59.096251+0000","last_fullsized":"2024-03-28T22:09:59.096251+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:44:19.753083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.424985+0000","last_change":"2024-03-28T22:08:29.332385+0000","last_active":"2024-03-28T22:09:30.424985+0000","last_peered":"2024-03-28T22:09:30.424985+0000","last_clean":"2024-03-28T22:09:30.424985+0000","last_became_active":"2024-03-28T22:08:29.331956+0000","last_became_peered":"2024-03-28T22:08:29.331956+0000","last_unstale":"2024-03-28T22:09:30.424985+0000","last_undegraded":"2024-03-28T22:09:30.424985+0000","last_fullsized":"2024-03-28T22:09:30.424985+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:10:04.428034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.092555+0000","last_change":"2024-03-28T22:09:30.080320+0000","last_active":"2024-03-28T22:09:59.092555+0000","last_peered":"2024-03-28T22:09:59.092555+0000","last_clean":"2024-03-28T22:09:59.092555+0000","last_became_active":"2024-03-28T22:09:30.079928+0000","last_became_peered":"2024-03-28T22:09:30.079928+0000","last_unstale":"2024-03-28T22:09:59.092555+0000","last_undegraded":"2024-03-28T22:09:59.092555+0000","last_fullsized":"2024-03-28T22:09:59.092555+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:12:53.385167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.293420+0000","last_change":"2024-03-28T22:08:29.327180+0000","last_active":"2024-03-28T22:09:30.293420+0000","last_peered":"2024-03-28T22:09:30.293420+0000","last_clean":"2024-03-28T22:09:30.293420+0000","last_became_active":"2024-03-28T22:08:29.326825+0000","last_became_peered":"2024-03-28T22:08:29.326825+0000","last_unstale":"2024-03-28T22:09:30.293420+0000","last_undegraded":"2024-03-28T22:09:30.293420+0000","last_fullsized":"2024-03-28T22:09:30.293420+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:11:09.979970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.098656+0000","last_change":"2024-03-28T22:08:29.955299+0000","last_active":"2024-03-28T22:09:59.098656+0000","last_peered":"2024-03-28T22:09:59.098656+0000","last_clean":"2024-03-28T22:09:59.098656+0000","last_became_active":"2024-03-28T22:08:29.954949+0000","last_became_peered":"2024-03-28T22:08:29.954949+0000","last_unstale":"2024-03-28T22:09:59.098656+0000","last_undegraded":"2024-03-28T22:09:59.098656+0000","last_fullsized":"2024-03-28T22:09:59.098656+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:01:48.375251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.059530+0000","last_change":"2024-03-28T22:08:29.327261+0000","last_active":"2024-03-28T22:09:30.059530+0000","last_peered":"2024-03-28T22:09:30.059530+0000","last_clean":"2024-03-28T22:09:30.059530+0000","last_became_active":"2024-03-28T22:08:29.326910+0000","last_became_peered":"2024-03-28T22:08:29.326910+0000","last_unstale":"2024-03-28T22:09:30.059530+0000","last_undegraded":"2024-03-28T22:09:30.059530+0000","last_fullsized":"2024-03-28T22:09:30.059530+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:46:41.933372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.093770+0000","last_change":"2024-03-28T22:08:29.954891+0000","last_active":"2024-03-28T22:09:59.093770+0000","last_peered":"2024-03-28T22:09:59.093770+0000","last_clean":"2024-03-28T22:09:59.093770+0000","last_became_active":"2024-03-28T22:08:29.954519+0000","last_became_peered":"2024-03-28T22:08:29.954519+0000","last_unstale":"2024-03-28T22:09:59.093770+0000","last_undegraded":"2024-03-28T22:09:59.093770+0000","last_fullsized":"2024-03-28T22:09:59.093770+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:07:42.621189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":129,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:10:18.495749+0000","last_change":"2024-03-28T22:08:29.330180+0000","last_active":"2024-03-28T22:10:18.495749+0000","last_peered":"2024-03-28T22:10:18.495749+0000","last_clean":"2024-03-28T22:10:18.495749+0000","last_became_active":"2024-03-28T22:08:29.329835+0000","last_became_peered":"2024-03-28T22:08:29.329835+0000","last_unstale":"2024-03-28T22:10:18.495749+0000","last_undegraded":"2024-03-28T22:10:18.495749+0000","last_fullsized":"2024-03-28T22:10:18.495749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:48:39.906359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":106,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.118105+0000","last_change":"2024-03-28T22:08:29.967074+0000","last_active":"2024-03-28T22:09:59.118105+0000","last_peered":"2024-03-28T22:09:59.118105+0000","last_clean":"2024-03-28T22:09:59.118105+0000","last_became_active":"2024-03-28T22:08:29.966258+0000","last_became_peered":"2024-03-28T22:08:29.966258+0000","last_unstale":"2024-03-28T22:09:59.118105+0000","last_undegraded":"2024-03-28T22:09:59.118105+0000","last_fullsized":"2024-03-28T22:09:59.118105+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:25:44.986179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.055474+0000","last_change":"2024-03-28T22:08:29.326388+0000","last_active":"2024-03-28T22:09:30.055474+0000","last_peered":"2024-03-28T22:09:30.055474+0000","last_clean":"2024-03-28T22:09:30.055474+0000","last_became_active":"2024-03-28T22:08:29.326001+0000","last_became_peered":"2024-03-28T22:08:29.326001+0000","last_unstale":"2024-03-28T22:09:30.055474+0000","last_undegraded":"2024-03-28T22:09:30.055474+0000","last_fullsized":"2024-03-28T22:09:30.055474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:31:56.227038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.104748+0000","last_change":"2024-03-28T22:08:29.959299+0000","last_active":"2024-03-28T22:09:59.104748+0000","last_peered":"2024-03-28T22:09:59.104748+0000","last_clean":"2024-03-28T22:09:59.104748+0000","last_became_active":"2024-03-28T22:08:29.958955+0000","last_became_peered":"2024-03-28T22:08:29.958955+0000","last_unstale":"2024-03-28T22:09:59.104748+0000","last_undegraded":"2024-03-28T22:09:59.104748+0000","last_fullsized":"2024-03-28T22:09:59.104748+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:11:06.775920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.060567+0000","last_change":"2024-03-28T22:08:29.326089+0000","last_active":"2024-03-28T22:09:30.060567+0000","last_peered":"2024-03-28T22:09:30.060567+0000","last_clean":"2024-03-28T22:09:30.060567+0000","last_became_active":"2024-03-28T22:08:29.325463+0000","last_became_peered":"2024-03-28T22:08:29.325463+0000","last_unstale":"2024-03-28T22:09:30.060567+0000","last_undegraded":"2024-03-28T22:09:30.060567+0000","last_fullsized":"2024-03-28T22:09:30.060567+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:18:31.049729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.118631+0000","last_change":"2024-03-28T22:08:29.964248+0000","last_active":"2024-03-28T22:09:59.118631+0000","last_peered":"2024-03-28T22:09:59.118631+0000","last_clean":"2024-03-28T22:09:59.118631+0000","last_became_active":"2024-03-28T22:08:29.963897+0000","last_became_peered":"2024-03-28T22:08:29.963897+0000","last_unstale":"2024-03-28T22:09:59.118631+0000","last_undegraded":"2024-03-28T22:09:59.118631+0000","last_fullsized":"2024-03-28T22:09:59.118631+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:41:39.645360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'31","reported_seq":271,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:10:19.497584+0000","last_change":"2024-03-28T22:08:29.330732+0000","last_active":"2024-03-28T22:10:19.497584+0000","last_peered":"2024-03-28T22:10:19.497584+0000","last_clean":"2024-03-28T22:10:19.497584+0000","last_became_active":"2024-03-28T22:08:29.329231+0000","last_became_peered":"2024-03-28T22:08:29.329231+0000","last_unstale":"2024-03-28T22:10:19.497584+0000","last_undegraded":"2024-03-28T22:10:19.497584+0000","last_fullsized":"2024-03-28T22:10:19.497584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:16:39.260429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59469,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":221,"num_read_kb":168,"num_write":32,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.109344+0000","last_change":"2024-03-28T22:08:29.965791+0000","last_active":"2024-03-28T22:09:59.109344+0000","last_peered":"2024-03-28T22:09:59.109344+0000","last_clean":"2024-03-28T22:09:59.109344+0000","last_became_active":"2024-03-28T22:08:29.965390+0000","last_became_peered":"2024-03-28T22:08:29.965390+0000","last_unstale":"2024-03-28T22:09:59.109344+0000","last_undegraded":"2024-03-28T22:09:59.109344+0000","last_fullsized":"2024-03-28T22:09:59.109344+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:43:53.325906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.057898+0000","last_change":"2024-03-28T22:08:29.325851+0000","last_active":"2024-03-28T22:09:30.057898+0000","last_peered":"2024-03-28T22:09:30.057898+0000","last_clean":"2024-03-28T22:09:30.057898+0000","last_became_active":"2024-03-28T22:08:29.325500+0000","last_became_peered":"2024-03-28T22:08:29.325500+0000","last_unstale":"2024-03-28T22:09:30.057898+0000","last_undegraded":"2024-03-28T22:09:30.057898+0000","last_fullsized":"2024-03-28T22:09:30.057898+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:04:25.151908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.115301+0000","last_change":"2024-03-28T22:08:29.953857+0000","last_active":"2024-03-28T22:09:59.115301+0000","last_peered":"2024-03-28T22:09:59.115301+0000","last_clean":"2024-03-28T22:09:59.115301+0000","last_became_active":"2024-03-28T22:08:29.953538+0000","last_became_peered":"2024-03-28T22:08:29.953538+0000","last_unstale":"2024-03-28T22:09:59.115301+0000","last_undegraded":"2024-03-28T22:09:59.115301+0000","last_fullsized":"2024-03-28T22:09:59.115301+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:40:32.404760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.060057+0000","last_change":"2024-03-28T22:08:29.327286+0000","last_active":"2024-03-28T22:09:30.060057+0000","last_peered":"2024-03-28T22:09:30.060057+0000","last_clean":"2024-03-28T22:09:30.060057+0000","last_became_active":"2024-03-28T22:08:29.326902+0000","last_became_peered":"2024-03-28T22:08:29.326902+0000","last_unstale":"2024-03-28T22:09:30.060057+0000","last_undegraded":"2024-03-28T22:09:30.060057+0000","last_fullsized":"2024-03-28T22:09:30.060057+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:14:58.669534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.116496+0000","last_change":"2024-03-28T22:09:30.294946+0000","last_active":"2024-03-28T22:09:59.116496+0000","last_peered":"2024-03-28T22:09:59.116496+0000","last_clean":"2024-03-28T22:09:59.116496+0000","last_became_active":"2024-03-28T22:09:30.294496+0000","last_became_peered":"2024-03-28T22:09:30.294496+0000","last_unstale":"2024-03-28T22:09:59.116496+0000","last_undegraded":"2024-03-28T22:09:59.116496+0000","last_fullsized":"2024-03-28T22:09:59.116496+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:26:40.062384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'6","reported_seq":138,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:10:21.500594+0000","last_change":"2024-03-28T22:08:29.327740+0000","last_active":"2024-03-28T22:10:21.500594+0000","last_peered":"2024-03-28T22:10:21.500594+0000","last_clean":"2024-03-28T22:10:21.500594+0000","last_became_active":"2024-03-28T22:08:29.326664+0000","last_became_peered":"2024-03-28T22:08:29.326664+0000","last_unstale":"2024-03-28T22:10:21.500594+0000","last_undegraded":"2024-03-28T22:10:21.500594+0000","last_fullsized":"2024-03-28T22:10:21.500594+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:58:17.681836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1511,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":114,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.104158+0000","last_change":"2024-03-28T22:08:29.961979+0000","last_active":"2024-03-28T22:09:59.104158+0000","last_peered":"2024-03-28T22:09:59.104158+0000","last_clean":"2024-03-28T22:09:59.104158+0000","last_became_active":"2024-03-28T22:08:29.961615+0000","last_became_peered":"2024-03-28T22:08:29.961615+0000","last_unstale":"2024-03-28T22:09:59.104158+0000","last_undegraded":"2024-03-28T22:09:59.104158+0000","last_fullsized":"2024-03-28T22:09:59.104158+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:50:37.227632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.057515+0000","last_change":"2024-03-28T22:08:29.329017+0000","last_active":"2024-03-28T22:09:30.057515+0000","last_peered":"2024-03-28T22:09:30.057515+0000","last_clean":"2024-03-28T22:09:30.057515+0000","last_became_active":"2024-03-28T22:08:29.328686+0000","last_became_peered":"2024-03-28T22:08:29.328686+0000","last_unstale":"2024-03-28T22:09:30.057515+0000","last_undegraded":"2024-03-28T22:09:30.057515+0000","last_fullsized":"2024-03-28T22:09:30.057515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:25:28.986165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.102273+0000","last_change":"2024-03-28T22:09:30.426320+0000","last_active":"2024-03-28T22:09:59.102273+0000","last_peered":"2024-03-28T22:09:59.102273+0000","last_clean":"2024-03-28T22:09:59.102273+0000","last_became_active":"2024-03-28T22:09:30.425862+0000","last_became_peered":"2024-03-28T22:09:30.425862+0000","last_unstale":"2024-03-28T22:09:59.102273+0000","last_undegraded":"2024-03-28T22:09:59.102273+0000","last_fullsized":"2024-03-28T22:09:59.102273+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:39:44.240551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058066+0000","last_change":"2024-03-28T22:08:29.323274+0000","last_active":"2024-03-28T22:09:30.058066+0000","last_peered":"2024-03-28T22:09:30.058066+0000","last_clean":"2024-03-28T22:09:30.058066+0000","last_became_active":"2024-03-28T22:08:29.322911+0000","last_became_peered":"2024-03-28T22:08:29.322911+0000","last_unstale":"2024-03-28T22:09:30.058066+0000","last_undegraded":"2024-03-28T22:09:30.058066+0000","last_fullsized":"2024-03-28T22:09:30.058066+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:44:14.439411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.107898+0000","last_change":"2024-03-28T22:08:29.961101+0000","last_active":"2024-03-28T22:09:59.107898+0000","last_peered":"2024-03-28T22:09:59.107898+0000","last_clean":"2024-03-28T22:09:59.107898+0000","last_became_active":"2024-03-28T22:08:29.960767+0000","last_became_peered":"2024-03-28T22:08:29.960767+0000","last_unstale":"2024-03-28T22:09:59.107898+0000","last_undegraded":"2024-03-28T22:09:59.107898+0000","last_fullsized":"2024-03-28T22:09:59.107898+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:33:45.236691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.060215+0000","last_change":"2024-03-28T22:09:30.060215+0000","last_active":"2024-03-28T22:09:30.060215+0000","last_peered":"2024-03-28T22:09:30.060215+0000","last_clean":"2024-03-28T22:09:30.060215+0000","last_became_active":"2024-03-28T22:09:30.059828+0000","last_became_peered":"2024-03-28T22:09:30.059828+0000","last_unstale":"2024-03-28T22:09:30.060215+0000","last_undegraded":"2024-03-28T22:09:30.060215+0000","last_fullsized":"2024-03-28T22:09:30.060215+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:42:05.087899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.112511+0000","last_change":"2024-03-28T22:08:29.954093+0000","last_active":"2024-03-28T22:09:59.112511+0000","last_peered":"2024-03-28T22:09:59.112511+0000","last_clean":"2024-03-28T22:09:59.112511+0000","last_became_active":"2024-03-28T22:08:29.953688+0000","last_became_peered":"2024-03-28T22:08:29.953688+0000","last_unstale":"2024-03-28T22:09:59.112511+0000","last_undegraded":"2024-03-28T22:09:59.112511+0000","last_fullsized":"2024-03-28T22:09:59.112511+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:57:51.026565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.055115+0000","last_change":"2024-03-28T22:08:29.331104+0000","last_active":"2024-03-28T22:09:30.055115+0000","last_peered":"2024-03-28T22:09:30.055115+0000","last_clean":"2024-03-28T22:09:30.055115+0000","last_became_active":"2024-03-28T22:08:29.330505+0000","last_became_peered":"2024-03-28T22:08:29.330505+0000","last_unstale":"2024-03-28T22:09:30.055115+0000","last_undegraded":"2024-03-28T22:09:30.055115+0000","last_fullsized":"2024-03-28T22:09:30.055115+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:36:59.944587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.110591+0000","last_change":"2024-03-28T22:08:29.963540+0000","last_active":"2024-03-28T22:09:59.110591+0000","last_peered":"2024-03-28T22:09:59.110591+0000","last_clean":"2024-03-28T22:09:59.110591+0000","last_became_active":"2024-03-28T22:08:29.963206+0000","last_became_peered":"2024-03-28T22:08:29.963206+0000","last_unstale":"2024-03-28T22:09:59.110591+0000","last_undegraded":"2024-03-28T22:09:59.110591+0000","last_fullsized":"2024-03-28T22:09:59.110591+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:55:53.292554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.098046+0000","last_change":"2024-03-28T22:08:29.962822+0000","last_active":"2024-03-28T22:09:59.098046+0000","last_peered":"2024-03-28T22:09:59.098046+0000","last_clean":"2024-03-28T22:09:59.098046+0000","last_became_active":"2024-03-28T22:08:29.962343+0000","last_became_peered":"2024-03-28T22:08:29.962343+0000","last_unstale":"2024-03-28T22:09:59.098046+0000","last_undegraded":"2024-03-28T22:09:59.098046+0000","last_fullsized":"2024-03-28T22:09:59.098046+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:36.608543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'6","reported_seq":132,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:10:20.498618+0000","last_change":"2024-03-28T22:08:29.328239+0000","last_active":"2024-03-28T22:10:20.498618+0000","last_peered":"2024-03-28T22:10:20.498618+0000","last_clean":"2024-03-28T22:10:20.498618+0000","last_became_active":"2024-03-28T22:08:29.327903+0000","last_became_peered":"2024-03-28T22:08:29.327903+0000","last_unstale":"2024-03-28T22:10:20.498618+0000","last_undegraded":"2024-03-28T22:10:20.498618+0000","last_fullsized":"2024-03-28T22:10:20.498618+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:14:17.558472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":107,"num_read_kb":107,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.085982+0000","last_change":"2024-03-28T22:09:30.426683+0000","last_active":"2024-03-28T22:09:59.085982+0000","last_peered":"2024-03-28T22:09:59.085982+0000","last_clean":"2024-03-28T22:09:59.085982+0000","last_became_active":"2024-03-28T22:09:30.426255+0000","last_became_peered":"2024-03-28T22:09:30.426255+0000","last_unstale":"2024-03-28T22:09:59.085982+0000","last_undegraded":"2024-03-28T22:09:59.085982+0000","last_fullsized":"2024-03-28T22:09:59.085982+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:36.755978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.079359+0000","last_change":"2024-03-28T22:08:29.323050+0000","last_active":"2024-03-28T22:09:30.079359+0000","last_peered":"2024-03-28T22:09:30.079359+0000","last_clean":"2024-03-28T22:09:30.079359+0000","last_became_active":"2024-03-28T22:08:29.322684+0000","last_became_peered":"2024-03-28T22:08:29.322684+0000","last_unstale":"2024-03-28T22:09:30.079359+0000","last_undegraded":"2024-03-28T22:09:30.079359+0000","last_fullsized":"2024-03-28T22:09:30.079359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:14:43.811122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":354,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.054922+0000","last_change":"2024-03-28T22:07:17.785516+0000","last_active":"2024-03-28T22:09:30.054922+0000","last_peered":"2024-03-28T22:09:30.054922+0000","last_clean":"2024-03-28T22:09:30.054922+0000","last_became_active":"2024-03-28T22:07:17.785123+0000","last_became_peered":"2024-03-28T22:07:17.785123+0000","last_unstale":"2024-03-28T22:09:30.054922+0000","last_undegraded":"2024-03-28T22:09:30.054922+0000","last_fullsized":"2024-03-28T22:09:30.054922+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:03:59.840513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:03:59.840513+0000","last_clean_scrub_stamp":"2024-03-28T22:03:59.840513+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:31:06.253632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.108076+0000","last_change":"2024-03-28T22:08:29.961489+0000","last_active":"2024-03-28T22:09:59.108076+0000","last_peered":"2024-03-28T22:09:59.108076+0000","last_clean":"2024-03-28T22:09:59.108076+0000","last_became_active":"2024-03-28T22:08:29.960881+0000","last_became_peered":"2024-03-28T22:08:29.960881+0000","last_unstale":"2024-03-28T22:09:59.108076+0000","last_undegraded":"2024-03-28T22:09:59.108076+0000","last_fullsized":"2024-03-28T22:09:59.108076+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:48.870834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.054972+0000","last_change":"2024-03-28T22:08:29.331994+0000","last_active":"2024-03-28T22:09:30.054972+0000","last_peered":"2024-03-28T22:09:30.054972+0000","last_clean":"2024-03-28T22:09:30.054972+0000","last_became_active":"2024-03-28T22:08:29.331630+0000","last_became_peered":"2024-03-28T22:08:29.331630+0000","last_unstale":"2024-03-28T22:09:30.054972+0000","last_undegraded":"2024-03-28T22:09:30.054972+0000","last_fullsized":"2024-03-28T22:09:30.054972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:11:37.100405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058466+0000","last_change":"2024-03-28T22:08:29.330353+0000","last_active":"2024-03-28T22:09:30.058466+0000","last_peered":"2024-03-28T22:09:30.058466+0000","last_clean":"2024-03-28T22:09:30.058466+0000","last_became_active":"2024-03-28T22:08:29.329421+0000","last_became_peered":"2024-03-28T22:08:29.329421+0000","last_unstale":"2024-03-28T22:09:30.058466+0000","last_undegraded":"2024-03-28T22:09:30.058466+0000","last_fullsized":"2024-03-28T22:09:30.058466+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:35:56.166506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.108762+0000","last_change":"2024-03-28T22:08:29.948595+0000","last_active":"2024-03-28T22:09:59.108762+0000","last_peered":"2024-03-28T22:09:59.108762+0000","last_clean":"2024-03-28T22:09:59.108762+0000","last_became_active":"2024-03-28T22:08:29.946986+0000","last_became_peered":"2024-03-28T22:08:29.946986+0000","last_unstale":"2024-03-28T22:09:59.108762+0000","last_undegraded":"2024-03-28T22:09:59.108762+0000","last_fullsized":"2024-03-28T22:09:59.108762+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:56:12.351731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.328434+0000","last_change":"2024-03-28T22:08:29.333375+0000","last_active":"2024-03-28T22:09:59.328434+0000","last_peered":"2024-03-28T22:09:59.328434+0000","last_clean":"2024-03-28T22:09:59.328434+0000","last_became_active":"2024-03-28T22:08:29.331594+0000","last_became_peered":"2024-03-28T22:08:29.331594+0000","last_unstale":"2024-03-28T22:09:59.328434+0000","last_undegraded":"2024-03-28T22:09:59.328434+0000","last_fullsized":"2024-03-28T22:09:59.328434+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:07:43.936807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.109527+0000","last_change":"2024-03-28T22:08:29.959061+0000","last_active":"2024-03-28T22:09:59.109527+0000","last_peered":"2024-03-28T22:09:59.109527+0000","last_clean":"2024-03-28T22:09:59.109527+0000","last_became_active":"2024-03-28T22:08:29.958723+0000","last_became_peered":"2024-03-28T22:08:29.958723+0000","last_unstale":"2024-03-28T22:09:59.109527+0000","last_undegraded":"2024-03-28T22:09:59.109527+0000","last_fullsized":"2024-03-28T22:09:59.109527+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:32:23.016606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.079461+0000","last_change":"2024-03-28T22:08:29.329886+0000","last_active":"2024-03-28T22:09:30.079461+0000","last_peered":"2024-03-28T22:09:30.079461+0000","last_clean":"2024-03-28T22:09:30.079461+0000","last_became_active":"2024-03-28T22:08:29.329503+0000","last_became_peered":"2024-03-28T22:08:29.329503+0000","last_unstale":"2024-03-28T22:09:30.079461+0000","last_undegraded":"2024-03-28T22:09:30.079461+0000","last_fullsized":"2024-03-28T22:09:30.079461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:50:05.686647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.102695+0000","last_change":"2024-03-28T22:08:29.957038+0000","last_active":"2024-03-28T22:09:59.102695+0000","last_peered":"2024-03-28T22:09:59.102695+0000","last_clean":"2024-03-28T22:09:59.102695+0000","last_became_active":"2024-03-28T22:08:29.956693+0000","last_became_peered":"2024-03-28T22:08:29.956693+0000","last_unstale":"2024-03-28T22:09:59.102695+0000","last_undegraded":"2024-03-28T22:09:59.102695+0000","last_fullsized":"2024-03-28T22:09:59.102695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:19:09.881431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.424647+0000","last_change":"2024-03-28T22:08:29.327356+0000","last_active":"2024-03-28T22:09:30.424647+0000","last_peered":"2024-03-28T22:09:30.424647+0000","last_clean":"2024-03-28T22:09:30.424647+0000","last_became_active":"2024-03-28T22:08:29.326947+0000","last_became_peered":"2024-03-28T22:08:29.326947+0000","last_unstale":"2024-03-28T22:09:30.424647+0000","last_undegraded":"2024-03-28T22:09:30.424647+0000","last_fullsized":"2024-03-28T22:09:30.424647+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:47:29.417465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.064033+0000","last_change":"2024-03-28T22:08:29.966505+0000","last_active":"2024-03-28T22:09:59.064033+0000","last_peered":"2024-03-28T22:09:59.064033+0000","last_clean":"2024-03-28T22:09:59.064033+0000","last_became_active":"2024-03-28T22:08:29.966057+0000","last_became_peered":"2024-03-28T22:08:29.966057+0000","last_unstale":"2024-03-28T22:09:59.064033+0000","last_undegraded":"2024-03-28T22:09:59.064033+0000","last_fullsized":"2024-03-28T22:09:59.064033+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:17:45.196585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.057477+0000","last_change":"2024-03-28T22:08:29.328897+0000","last_active":"2024-03-28T22:09:30.057477+0000","last_peered":"2024-03-28T22:09:30.057477+0000","last_clean":"2024-03-28T22:09:30.057477+0000","last_became_active":"2024-03-28T22:08:29.328390+0000","last_became_peered":"2024-03-28T22:08:29.328390+0000","last_unstale":"2024-03-28T22:09:30.057477+0000","last_undegraded":"2024-03-28T22:09:30.057477+0000","last_fullsized":"2024-03-28T22:09:30.057477+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:21:12.699503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.094866+0000","last_change":"2024-03-28T22:08:29.957212+0000","last_active":"2024-03-28T22:09:59.094866+0000","last_peered":"2024-03-28T22:09:59.094866+0000","last_clean":"2024-03-28T22:09:59.094866+0000","last_became_active":"2024-03-28T22:08:29.956215+0000","last_became_peered":"2024-03-28T22:08:29.956215+0000","last_unstale":"2024-03-28T22:09:59.094866+0000","last_undegraded":"2024-03-28T22:09:59.094866+0000","last_fullsized":"2024-03-28T22:09:59.094866+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:57:25.199993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.063151+0000","last_change":"2024-03-28T22:09:30.062742+0000","last_active":"2024-03-28T22:09:30.063151+0000","last_peered":"2024-03-28T22:09:30.063151+0000","last_clean":"2024-03-28T22:09:30.063151+0000","last_became_active":"2024-03-28T22:09:30.059356+0000","last_became_peered":"2024-03-28T22:09:30.059356+0000","last_unstale":"2024-03-28T22:09:30.063151+0000","last_undegraded":"2024-03-28T22:09:30.063151+0000","last_fullsized":"2024-03-28T22:09:30.063151+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:52:39.412153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.110714+0000","last_change":"2024-03-28T22:08:29.955023+0000","last_active":"2024-03-28T22:09:59.110714+0000","last_peered":"2024-03-28T22:09:59.110714+0000","last_clean":"2024-03-28T22:09:59.110714+0000","last_became_active":"2024-03-28T22:08:29.954675+0000","last_became_peered":"2024-03-28T22:08:29.954675+0000","last_unstale":"2024-03-28T22:09:59.110714+0000","last_undegraded":"2024-03-28T22:09:59.110714+0000","last_fullsized":"2024-03-28T22:09:59.110714+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:31.246452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058628+0000","last_change":"2024-03-28T22:08:29.329280+0000","last_active":"2024-03-28T22:09:30.058628+0000","last_peered":"2024-03-28T22:09:30.058628+0000","last_clean":"2024-03-28T22:09:30.058628+0000","last_became_active":"2024-03-28T22:08:29.328860+0000","last_became_peered":"2024-03-28T22:08:29.328860+0000","last_unstale":"2024-03-28T22:09:30.058628+0000","last_undegraded":"2024-03-28T22:09:30.058628+0000","last_fullsized":"2024-03-28T22:09:30.058628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:15:49.610026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.118448+0000","last_change":"2024-03-28T22:08:29.966224+0000","last_active":"2024-03-28T22:09:59.118448+0000","last_peered":"2024-03-28T22:09:59.118448+0000","last_clean":"2024-03-28T22:09:59.118448+0000","last_became_active":"2024-03-28T22:08:29.965636+0000","last_became_peered":"2024-03-28T22:08:29.965636+0000","last_unstale":"2024-03-28T22:09:59.118448+0000","last_undegraded":"2024-03-28T22:09:59.118448+0000","last_fullsized":"2024-03-28T22:09:59.118448+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:23:31.230980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.058666+0000","last_change":"2024-03-28T22:08:29.335228+0000","last_active":"2024-03-28T22:09:30.058666+0000","last_peered":"2024-03-28T22:09:30.058666+0000","last_clean":"2024-03-28T22:09:30.058666+0000","last_became_active":"2024-03-28T22:08:29.334686+0000","last_became_peered":"2024-03-28T22:08:29.334686+0000","last_unstale":"2024-03-28T22:09:30.058666+0000","last_undegraded":"2024-03-28T22:09:30.058666+0000","last_fullsized":"2024-03-28T22:09:30.058666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:17:40.571206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.119960+0000","last_change":"2024-03-28T22:08:29.963912+0000","last_active":"2024-03-28T22:09:59.119960+0000","last_peered":"2024-03-28T22:09:59.119960+0000","last_clean":"2024-03-28T22:09:59.119960+0000","last_became_active":"2024-03-28T22:08:29.962740+0000","last_became_peered":"2024-03-28T22:08:29.962740+0000","last_unstale":"2024-03-28T22:09:59.119960+0000","last_undegraded":"2024-03-28T22:09:59.119960+0000","last_fullsized":"2024-03-28T22:09:59.119960+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:50:25.756923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.060333+0000","last_change":"2024-03-28T22:08:29.325767+0000","last_active":"2024-03-28T22:09:30.060333+0000","last_peered":"2024-03-28T22:09:30.060333+0000","last_clean":"2024-03-28T22:09:30.060333+0000","last_became_active":"2024-03-28T22:08:29.325430+0000","last_became_peered":"2024-03-28T22:08:29.325430+0000","last_unstale":"2024-03-28T22:09:30.060333+0000","last_undegraded":"2024-03-28T22:09:30.060333+0000","last_fullsized":"2024-03-28T22:09:30.060333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:01:13.663163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.114614+0000","last_change":"2024-03-28T22:08:29.960158+0000","last_active":"2024-03-28T22:09:59.114614+0000","last_peered":"2024-03-28T22:09:59.114614+0000","last_clean":"2024-03-28T22:09:59.114614+0000","last_became_active":"2024-03-28T22:08:29.959792+0000","last_became_peered":"2024-03-28T22:08:29.959792+0000","last_unstale":"2024-03-28T22:09:59.114614+0000","last_undegraded":"2024-03-28T22:09:59.114614+0000","last_fullsized":"2024-03-28T22:09:59.114614+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:28:49.135551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:30.293763+0000","last_change":"2024-03-28T22:08:29.328547+0000","last_active":"2024-03-28T22:09:30.293763+0000","last_peered":"2024-03-28T22:09:30.293763+0000","last_clean":"2024-03-28T22:09:30.293763+0000","last_became_active":"2024-03-28T22:08:29.327910+0000","last_became_peered":"2024-03-28T22:08:29.327910+0000","last_unstale":"2024-03-28T22:09:30.293763+0000","last_undegraded":"2024-03-28T22:09:30.293763+0000","last_fullsized":"2024-03-28T22:09:30.293763+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:28.304200+0000","last_clean_scrub_stamp":"2024-03-28T22:08:28.304200+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:06:13.980071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.105047+0000","last_change":"2024-03-28T22:08:29.963904+0000","last_active":"2024-03-28T22:09:59.105047+0000","last_peered":"2024-03-28T22:09:59.105047+0000","last_clean":"2024-03-28T22:09:59.105047+0000","last_became_active":"2024-03-28T22:08:29.963536+0000","last_became_peered":"2024-03-28T22:08:29.963536+0000","last_unstale":"2024-03-28T22:09:59.105047+0000","last_undegraded":"2024-03-28T22:09:59.105047+0000","last_fullsized":"2024-03-28T22:09:59.105047+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:59:43.376792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.100207+0000","last_change":"2024-03-28T22:08:29.962357+0000","last_active":"2024-03-28T22:09:59.100207+0000","last_peered":"2024-03-28T22:09:59.100207+0000","last_clean":"2024-03-28T22:09:59.100207+0000","last_became_active":"2024-03-28T22:08:29.961998+0000","last_became_peered":"2024-03-28T22:08:29.961998+0000","last_unstale":"2024-03-28T22:09:59.100207+0000","last_undegraded":"2024-03-28T22:09:59.100207+0000","last_fullsized":"2024-03-28T22:09:59.100207+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:40:47.465889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.108275+0000","last_change":"2024-03-28T22:08:29.961326+0000","last_active":"2024-03-28T22:09:59.108275+0000","last_peered":"2024-03-28T22:09:59.108275+0000","last_clean":"2024-03-28T22:09:59.108275+0000","last_became_active":"2024-03-28T22:08:29.960316+0000","last_became_peered":"2024-03-28T22:08:29.960316+0000","last_unstale":"2024-03-28T22:09:59.108275+0000","last_undegraded":"2024-03-28T22:09:59.108275+0000","last_fullsized":"2024-03-28T22:09:59.108275+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:37:55.773686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.099312+0000","last_change":"2024-03-28T22:08:29.956272+0000","last_active":"2024-03-28T22:09:59.099312+0000","last_peered":"2024-03-28T22:09:59.099312+0000","last_clean":"2024-03-28T22:09:59.099312+0000","last_became_active":"2024-03-28T22:08:29.955875+0000","last_became_peered":"2024-03-28T22:08:29.955875+0000","last_unstale":"2024-03-28T22:09:59.099312+0000","last_undegraded":"2024-03-28T22:09:59.099312+0000","last_fullsized":"2024-03-28T22:09:59.099312+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:17:33.062615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.103567+0000","last_change":"2024-03-28T22:09:30.294754+0000","last_active":"2024-03-28T22:09:59.103567+0000","last_peered":"2024-03-28T22:09:59.103567+0000","last_clean":"2024-03-28T22:09:59.103567+0000","last_became_active":"2024-03-28T22:09:30.294341+0000","last_became_peered":"2024-03-28T22:09:30.294341+0000","last_unstale":"2024-03-28T22:09:59.103567+0000","last_undegraded":"2024-03-28T22:09:59.103567+0000","last_fullsized":"2024-03-28T22:09:59.103567+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:25:46.017373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.105840+0000","last_change":"2024-03-28T22:08:29.960464+0000","last_active":"2024-03-28T22:09:59.105840+0000","last_peered":"2024-03-28T22:09:59.105840+0000","last_clean":"2024-03-28T22:09:59.105840+0000","last_became_active":"2024-03-28T22:08:29.960111+0000","last_became_peered":"2024-03-28T22:08:29.960111+0000","last_unstale":"2024-03-28T22:09:59.105840+0000","last_undegraded":"2024-03-28T22:09:59.105840+0000","last_fullsized":"2024-03-28T22:09:59.105840+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:32:33.533199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.100620+0000","last_change":"2024-03-28T22:08:29.960045+0000","last_active":"2024-03-28T22:09:59.100620+0000","last_peered":"2024-03-28T22:09:59.100620+0000","last_clean":"2024-03-28T22:09:59.100620+0000","last_became_active":"2024-03-28T22:08:29.959645+0000","last_became_peered":"2024-03-28T22:08:29.959645+0000","last_unstale":"2024-03-28T22:09:59.100620+0000","last_undegraded":"2024-03-28T22:09:59.100620+0000","last_fullsized":"2024-03-28T22:09:59.100620+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:34:56.202257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.114146+0000","last_change":"2024-03-28T22:08:29.958736+0000","last_active":"2024-03-28T22:09:59.114146+0000","last_peered":"2024-03-28T22:09:59.114146+0000","last_clean":"2024-03-28T22:09:59.114146+0000","last_became_active":"2024-03-28T22:08:29.958387+0000","last_became_peered":"2024-03-28T22:08:29.958387+0000","last_unstale":"2024-03-28T22:09:59.114146+0000","last_undegraded":"2024-03-28T22:09:59.114146+0000","last_fullsized":"2024-03-28T22:09:59.114146+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:07:54.829166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.101041+0000","last_change":"2024-03-28T22:08:29.961373+0000","last_active":"2024-03-28T22:09:59.101041+0000","last_peered":"2024-03-28T22:09:59.101041+0000","last_clean":"2024-03-28T22:09:59.101041+0000","last_became_active":"2024-03-28T22:08:29.959751+0000","last_became_peered":"2024-03-28T22:08:29.959751+0000","last_unstale":"2024-03-28T22:09:59.101041+0000","last_undegraded":"2024-03-28T22:09:59.101041+0000","last_fullsized":"2024-03-28T22:09:59.101041+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:34:18.680552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.098710+0000","last_change":"2024-03-28T22:08:29.962721+0000","last_active":"2024-03-28T22:09:59.098710+0000","last_peered":"2024-03-28T22:09:59.098710+0000","last_clean":"2024-03-28T22:09:59.098710+0000","last_became_active":"2024-03-28T22:08:29.961349+0000","last_became_peered":"2024-03-28T22:08:29.961349+0000","last_unstale":"2024-03-28T22:09:59.098710+0000","last_undegraded":"2024-03-28T22:09:59.098710+0000","last_fullsized":"2024-03-28T22:09:59.098710+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:35:56.535055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.097778+0000","last_change":"2024-03-28T22:08:29.960068+0000","last_active":"2024-03-28T22:09:59.097778+0000","last_peered":"2024-03-28T22:09:59.097778+0000","last_clean":"2024-03-28T22:09:59.097778+0000","last_became_active":"2024-03-28T22:08:29.959716+0000","last_became_peered":"2024-03-28T22:08:29.959716+0000","last_unstale":"2024-03-28T22:09:59.097778+0000","last_undegraded":"2024-03-28T22:09:59.097778+0000","last_fullsized":"2024-03-28T22:09:59.097778+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:18:38.055322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.121200+0000","last_change":"2024-03-28T22:08:29.957920+0000","last_active":"2024-03-28T22:09:59.121200+0000","last_peered":"2024-03-28T22:09:59.121200+0000","last_clean":"2024-03-28T22:09:59.121200+0000","last_became_active":"2024-03-28T22:08:29.957541+0000","last_became_peered":"2024-03-28T22:08:29.957541+0000","last_unstale":"2024-03-28T22:09:59.121200+0000","last_undegraded":"2024-03-28T22:09:59.121200+0000","last_fullsized":"2024-03-28T22:09:59.121200+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:45:05.658745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.115831+0000","last_change":"2024-03-28T22:08:29.953589+0000","last_active":"2024-03-28T22:09:59.115831+0000","last_peered":"2024-03-28T22:09:59.115831+0000","last_clean":"2024-03-28T22:09:59.115831+0000","last_became_active":"2024-03-28T22:08:29.953253+0000","last_became_peered":"2024-03-28T22:08:29.953253+0000","last_unstale":"2024-03-28T22:09:59.115831+0000","last_undegraded":"2024-03-28T22:09:59.115831+0000","last_fullsized":"2024-03-28T22:09:59.115831+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:41:35.720708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T22:09:59.109906+0000","last_change":"2024-03-28T22:08:29.966235+0000","last_active":"2024-03-28T22:09:59.109906+0000","last_peered":"2024-03-28T22:09:59.109906+0000","last_clean":"2024-03-28T22:09:59.109906+0000","last_became_active":"2024-03-28T22:08:29.965830+0000","last_became_peered":"2024-03-28T22:08:29.965830+0000","last_unstale":"2024-03-28T22:09:59.109906+0000","last_undegraded":"2024-03-28T22:09:59.109906+0000","last_fullsized":"2024-03-28T22:09:59.109906+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:08:29.306116+0000","last_clean_scrub_stamp":"2024-03-28T22:08:29.306116+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:04:27.094743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":64925,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":672,"num_read_kb":404,"num_write":171,"num_write_kb":119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":421888,"data_stored":199366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351012,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":950272,"data_stored":459161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612648,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28624,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716528,"statfs":{"total":95995035648,"available":95965724672,"internally_reserved":0,"allocated":1646592,"data_stored":1110335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776173,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":459397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037810,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":459317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299448,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36860,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708292,"statfs":{"total":95995035648,"available":95957291008,"internally_reserved":0,"allocated":1560576,"data_stored":1051920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462972,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40428,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704724,"statfs":{"total":95995035648,"available":95953637376,"internally_reserved":0,"allocated":954368,"data_stored":458786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":517740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":460057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411409,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1044480,"data_stored":526085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149771,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":459097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888135,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40432,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704720,"statfs":{"total":95995035648,"available":95953633280,"internally_reserved":0,"allocated":958464,"data_stored":460146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724610,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28552,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716600,"statfs":{"total":95995035648,"available":95965798400,"internally_reserved":0,"allocated":1572864,"data_stored":1051072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":59469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":67506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":61579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T22:10:24.445 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:10:24.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:24 smithi053 ceph-mon[35979]: pgmap v345: 97 pgs: 97 active+clean; 640 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-28T22:10:24.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:24 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/462332625' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:10:24.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:24 smithi053 ceph-mon[35979]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:10:24.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:24 smithi053 ceph-mon[35979]: from='client.15339 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:10:24.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.582+0000 7f942fa15640 1 -- 172.21.15.16:0/361726455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9428101870 msgr2=0x7f9428101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:24.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.582+0000 7f942fa15640 1 --2- 172.21.15.16:0/361726455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9428101870 0x7f9428101cf0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f941c009900 tx=0x7f941c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:10:24.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.582+0000 7f942fa15640 1 -- 172.21.15.16:0/361726455 shutdown_connections 2024-03-28T22:10:24.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.582+0000 7f942fa15640 1 --2- 172.21.15.16:0/361726455 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9428102230 0x7f9428108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:24.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.582+0000 7f942fa15640 1 --2- 172.21.15.16:0/361726455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9428101870 0x7f9428101cf0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:24.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.582+0000 7f942fa15640 1 --2- 172.21.15.16:0/361726455 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9428100680 0x7f9428100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:24.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.582+0000 7f942fa15640 1 -- 172.21.15.16:0/361726455 >> 172.21.15.16:0/361726455 conn(0x7f94280fbbf0 msgr2=0x7f94280fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:24.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.583+0000 7f942fa15640 1 -- 172.21.15.16:0/361726455 shutdown_connections 2024-03-28T22:10:24.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.583+0000 7f942fa15640 1 -- 172.21.15.16:0/361726455 wait complete. 2024-03-28T22:10:24.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.583+0000 7f942fa15640 1 Processor -- start 2024-03-28T22:10:24.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.583+0000 7f942fa15640 1 -- start start 2024-03-28T22:10:24.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.583+0000 7f942fa15640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9428100680 0x7f94280695b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:24.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.583+0000 7f942fa15640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9428101870 0x7f942819c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:24.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.583+0000 7f942fa15640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9428102230 0x7f942819edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:24.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.583+0000 7f942fa15640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f942810c350 con 0x7f9428102230 2024-03-28T22:10:24.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.583+0000 7f942fa15640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f942810c1d0 con 0x7f9428101870 2024-03-28T22:10:24.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.583+0000 7f942fa15640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f942810c4d0 con 0x7f9428100680 2024-03-28T22:10:24.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.583+0000 7f942dfb2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9428102230 0x7f942819edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:24.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.583+0000 7f942dfb2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9428102230 0x7f942819edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44520/0 (socket says 172.21.15.16:44520) 2024-03-28T22:10:24.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.584+0000 7f942dfb2640 1 -- 172.21.15.16:0/2143763526 learned_addr learned my addr 172.21.15.16:0/2143763526 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:24.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.584+0000 7f942cfb0640 1 --2- 172.21.15.16:0/2143763526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9428101870 0x7f942819c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:24.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.584+0000 7f942d7b1640 1 --2- 172.21.15.16:0/2143763526 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9428100680 0x7f94280695b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:24.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.584+0000 7f942dfb2640 1 -- 172.21.15.16:0/2143763526 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9428100680 msgr2=0x7f94280695b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:24.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.584+0000 7f942dfb2640 1 --2- 172.21.15.16:0/2143763526 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9428100680 0x7f94280695b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:24.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.584+0000 7f942dfb2640 1 -- 172.21.15.16:0/2143763526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9428101870 msgr2=0x7f942819c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:24.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.584+0000 7f942dfb2640 1 --2- 172.21.15.16:0/2143763526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9428101870 0x7f942819c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:24.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.584+0000 7f942dfb2640 1 -- 172.21.15.16:0/2143763526 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f942819f690 con 0x7f9428102230 2024-03-28T22:10:24.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.584+0000 7f942dfb2640 1 --2- 172.21.15.16:0/2143763526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9428102230 0x7f942819edd0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f941800cad0 tx=0x7f941800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:24.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.584+0000 7f94267fc640 1 -- 172.21.15.16:0/2143763526 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f941800ae20 con 0x7f9428102230 2024-03-28T22:10:24.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.584+0000 7f94267fc640 1 -- 172.21.15.16:0/2143763526 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9418005020 con 0x7f9428102230 2024-03-28T22:10:24.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.584+0000 7f942fa15640 1 -- 172.21.15.16:0/2143763526 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f942819f980 con 0x7f9428102230 2024-03-28T22:10:24.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.584+0000 7f94267fc640 1 -- 172.21.15.16:0/2143763526 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f941800f6f0 con 0x7f9428102230 2024-03-28T22:10:24.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.584+0000 7f942fa15640 1 -- 172.21.15.16:0/2143763526 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f942819fea0 con 0x7f9428102230 2024-03-28T22:10:24.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.585+0000 7f942fa15640 1 -- 172.21.15.16:0/2143763526 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9428107940 con 0x7f9428102230 2024-03-28T22:10:24.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.586+0000 7f94267fc640 1 -- 172.21.15.16:0/2143763526 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7f94180051d0 con 0x7f9428102230 2024-03-28T22:10:24.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.587+0000 7f94267fc640 1 --2- 172.21.15.16:0/2143763526 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f94100a1e10 0x7f94100a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:24.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.587+0000 7f942d7b1640 1 --2- 172.21.15.16:0/2143763526 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f94100a1e10 0x7f94100a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:24.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.587+0000 7f94267fc640 1 -- 172.21.15.16:0/2143763526 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f94180ce220 con 0x7f9428102230 2024-03-28T22:10:24.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.587+0000 7f942d7b1640 1 --2- 172.21.15.16:0/2143763526 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f94100a1e10 0x7f94100a42d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f94200095f0 tx=0x7f9420005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:24.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.591+0000 7f94267fc640 1 -- 172.21.15.16:0/2143763526 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9418014030 con 0x7f9428102230 2024-03-28T22:10:24.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:24 smithi063 ceph-mon[36249]: pgmap v345: 97 pgs: 97 active+clean; 640 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-28T22:10:24.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:24 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/462332625' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:10:24.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:24 smithi063 ceph-mon[36249]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:10:24.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:24 smithi063 ceph-mon[36249]: from='client.15339 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:10:24.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.774+0000 7f942fa15640 1 -- 172.21.15.16:0/2143763526 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f942802d350 con 0x7f9428102230 2024-03-28T22:10:24.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.776+0000 7f94267fc640 1 -- 172.21.15.16:0/2143763526 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22441 (secure 0 0 0) 0x7f941809ae70 con 0x7f9428102230 2024-03-28T22:10:24.778 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:10:24.779 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":88,"fsid":"798d9884-ed4e-11ee-95d0-87774f69a715","created":"2024-03-28T22:00:27.216099+0000","modified":"2024-03-28T22:09:30.047847+0000","last_up_change":"2024-03-28T22:07:40.251141+0000","last_in_change":"2024-03-28T22:07:19.456451+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:03:59.327832+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:08:28.123529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:08:28.919037+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"46e7c8f8-ecc8-43d6-ae53-0655728f3240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6803","nonce":1589692483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6805","nonce":1589692483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6809","nonce":1589692483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6807","nonce":1589692483}]},"public_addr":"172.21.15.16:6803/1589692483","cluster_addr":"172.21.15.16:6805/1589692483","heartbeat_back_addr":"172.21.15.16:6809/1589692483","heartbeat_front_addr":"172.21.15.16:6807/1589692483","state":["exists","up"]},{"osd":1,"uuid":"af6d9c22-2830-4ce2-a547-011df0c6cbad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6801","nonce":795531234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6803","nonce":795531234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6807","nonce":795531234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6805","nonce":795531234}]},"public_addr":"172.21.15.53:6801/795531234","cluster_addr":"172.21.15.53:6803/795531234","heartbeat_back_addr":"172.21.15.53:6807/795531234","heartbeat_front_addr":"172.21.15.53:6805/795531234","state":["exists","up"]},{"osd":2,"uuid":"b10adca9-110c-4f87-85e0-b7542709d31a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6801","nonce":2214131086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6803","nonce":2214131086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6807","nonce":2214131086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6805","nonce":2214131086}]},"public_addr":"172.21.15.63:6801/2214131086","cluster_addr":"172.21.15.63:6803/2214131086","heartbeat_back_addr":"172.21.15.63:6807/2214131086","heartbeat_front_addr":"172.21.15.63:6805/2214131086","state":["exists","up"]},{"osd":3,"uuid":"c83cf4bb-395a-4786-abf7-74ca93a33eb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6811","nonce":13176789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6813","nonce":13176789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6817","nonce":13176789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6815","nonce":13176789}]},"public_addr":"172.21.15.16:6811/13176789","cluster_addr":"172.21.15.16:6813/13176789","heartbeat_back_addr":"172.21.15.16:6817/13176789","heartbeat_front_addr":"172.21.15.16:6815/13176789","state":["exists","up"]},{"osd":4,"uuid":"941b3e54-47d6-4e85-8069-e730c3fa8eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6809","nonce":772161849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6811","nonce":772161849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6815","nonce":772161849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6813","nonce":772161849}]},"public_addr":"172.21.15.53:6809/772161849","cluster_addr":"172.21.15.53:6811/772161849","heartbeat_back_addr":"172.21.15.53:6815/772161849","heartbeat_front_addr":"172.21.15.53:6813/772161849","state":["exists","up"]},{"osd":5,"uuid":"51d89826-1d60-40fc-8fd8-09bfe1ab4125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6809","nonce":1602554852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6811","nonce":1602554852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6815","nonce":1602554852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6813","nonce":1602554852}]},"public_addr":"172.21.15.63:6809/1602554852","cluster_addr":"172.21.15.63:6811/1602554852","heartbeat_back_addr":"172.21.15.63:6815/1602554852","heartbeat_front_addr":"172.21.15.63:6813/1602554852","state":["exists","up"]},{"osd":6,"uuid":"978efe5d-2c55-4ff8-8a1e-9f891388945e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6819","nonce":4290459224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6821","nonce":4290459224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6825","nonce":4290459224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6823","nonce":4290459224}]},"public_addr":"172.21.15.16:6819/4290459224","cluster_addr":"172.21.15.16:6821/4290459224","heartbeat_back_addr":"172.21.15.16:6825/4290459224","heartbeat_front_addr":"172.21.15.16:6823/4290459224","state":["exists","up"]},{"osd":7,"uuid":"50d489cb-dbb8-48df-b868-b29edf12e04d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6817","nonce":1056467525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6819","nonce":1056467525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6823","nonce":1056467525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6821","nonce":1056467525}]},"public_addr":"172.21.15.53:6817/1056467525","cluster_addr":"172.21.15.53:6819/1056467525","heartbeat_back_addr":"172.21.15.53:6823/1056467525","heartbeat_front_addr":"172.21.15.53:6821/1056467525","state":["exists","up"]},{"osd":8,"uuid":"49b39414-cc9c-4057-ae3f-cc109cd979da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6817","nonce":2918516794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6819","nonce":2918516794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6823","nonce":2918516794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6821","nonce":2918516794}]},"public_addr":"172.21.15.63:6817/2918516794","cluster_addr":"172.21.15.63:6819/2918516794","heartbeat_back_addr":"172.21.15.63:6823/2918516794","heartbeat_front_addr":"172.21.15.63:6821/2918516794","state":["exists","up"]},{"osd":9,"uuid":"4c873e51-b783-4eb6-83a3-fee31722d60b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6827","nonce":1585624367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6829","nonce":1585624367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6833","nonce":1585624367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6831","nonce":1585624367}]},"public_addr":"172.21.15.16:6827/1585624367","cluster_addr":"172.21.15.16:6829/1585624367","heartbeat_back_addr":"172.21.15.16:6833/1585624367","heartbeat_front_addr":"172.21.15.16:6831/1585624367","state":["exists","up"]},{"osd":10,"uuid":"085a78bf-e2ad-481a-85a3-71d1fc549a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6825","nonce":3357446484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6827","nonce":3357446484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6831","nonce":3357446484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6829","nonce":3357446484}]},"public_addr":"172.21.15.53:6825/3357446484","cluster_addr":"172.21.15.53:6827/3357446484","heartbeat_back_addr":"172.21.15.53:6831/3357446484","heartbeat_front_addr":"172.21.15.53:6829/3357446484","state":["exists","up"]},{"osd":11,"uuid":"13302aa3-08b9-4639-9042-27d850b4d5f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6825","nonce":2167866570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6827","nonce":2167866570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6831","nonce":2167866570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6829","nonce":2167866570}]},"public_addr":"172.21.15.63:6825/2167866570","cluster_addr":"172.21.15.63:6827/2167866570","heartbeat_back_addr":"172.21.15.63:6831/2167866570","heartbeat_front_addr":"172.21.15.63:6829/2167866570","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:07.450323+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:31.363172+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:55.295021+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:19.794760+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:44.063492+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:08.195064+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:33.655328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:57.639414+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:22.402394+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:47.462050+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:12.226307+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:37.495625+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3821408150":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6801/1832422779":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/996598130":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/1062264551":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/317557779":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6801/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/3631746518":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6800/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/3116652786":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6800/1832422779":"2024-03-29T22:01:28.993434+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:10:24.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.777+0000 7f942fa15640 1 -- 172.21.15.16:0/2143763526 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f94100a1e10 msgr2=0x7f94100a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:24.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.777+0000 7f942fa15640 1 --2- 172.21.15.16:0/2143763526 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f94100a1e10 0x7f94100a42d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f94200095f0 tx=0x7f9420005f90 comp rx=0 tx=0).stop 2024-03-28T22:10:24.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.777+0000 7f942fa15640 1 -- 172.21.15.16:0/2143763526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9428102230 msgr2=0x7f942819edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:24.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.777+0000 7f942fa15640 1 --2- 172.21.15.16:0/2143763526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9428102230 0x7f942819edd0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f941800cad0 tx=0x7f941800cf90 comp rx=0 tx=0).stop 2024-03-28T22:10:24.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.778+0000 7f942fa15640 1 -- 172.21.15.16:0/2143763526 shutdown_connections 2024-03-28T22:10:24.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.778+0000 7f942fa15640 1 --2- 172.21.15.16:0/2143763526 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f94100a1e10 0x7f94100a42d0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:24.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.778+0000 7f942fa15640 1 --2- 172.21.15.16:0/2143763526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9428102230 0x7f942819edd0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:24.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.778+0000 7f942fa15640 1 --2- 172.21.15.16:0/2143763526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9428101870 0x7f942819c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:24.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.778+0000 7f942fa15640 1 --2- 172.21.15.16:0/2143763526 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9428100680 0x7f94280695b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:24.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.778+0000 7f942fa15640 1 -- 172.21.15.16:0/2143763526 >> 172.21.15.16:0/2143763526 conn(0x7f94280fbbf0 msgr2=0x7f9428104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:24.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.778+0000 7f942fa15640 1 -- 172.21.15.16:0/2143763526 shutdown_connections 2024-03-28T22:10:24.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.778+0000 7f942fa15640 1 -- 172.21.15.16:0/2143763526 wait complete. 2024-03-28T22:10:24.792 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T22:03:59.327832+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T22:08:28.123529+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T22:08:28.919037+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T22:10:24.793 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T22:10:24.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.895+0000 7f7fa9b9a640 1 -- 172.21.15.16:0/4231014024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fa4101880 msgr2=0x7f7fa4101d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:24.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.895+0000 7f7fa9b9a640 1 --2- 172.21.15.16:0/4231014024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fa4101880 0x7f7fa4101d00 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f7f94009900 tx=0x7f7f9402bc30 comp rx=0 tx=0).stop 2024-03-28T22:10:24.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.895+0000 7f7fa9b9a640 1 -- 172.21.15.16:0/4231014024 shutdown_connections 2024-03-28T22:10:24.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.895+0000 7f7fa9b9a640 1 --2- 172.21.15.16:0/4231014024 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7fa4102240 0x7f7fa4108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:24.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.895+0000 7f7fa9b9a640 1 --2- 172.21.15.16:0/4231014024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fa4101880 0x7f7fa4101d00 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:24.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.895+0000 7f7fa9b9a640 1 --2- 172.21.15.16:0/4231014024 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fa4100690 0x7f7fa4100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:24.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.895+0000 7f7fa9b9a640 1 -- 172.21.15.16:0/4231014024 >> 172.21.15.16:0/4231014024 conn(0x7f7fa40fbc00 msgr2=0x7f7fa40fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:24.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.895+0000 7f7fa9b9a640 1 -- 172.21.15.16:0/4231014024 shutdown_connections 2024-03-28T22:10:24.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.895+0000 7f7fa9b9a640 1 -- 172.21.15.16:0/4231014024 wait complete. 2024-03-28T22:10:24.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.896+0000 7f7fa9b9a640 1 Processor -- start 2024-03-28T22:10:24.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.896+0000 7f7fa9b9a640 1 -- start start 2024-03-28T22:10:24.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.896+0000 7f7fa9b9a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fa4100690 0x7f7fa41969b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:24.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.896+0000 7f7fa9b9a640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fa4101880 0x7f7fa4196ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:24.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.896+0000 7f7fa9b9a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7fa4102240 0x7f7fa419df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:24.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.896+0000 7f7fa9b9a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa410c370 con 0x7f7fa4100690 2024-03-28T22:10:24.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.896+0000 7f7fa9b9a640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa410c1f0 con 0x7f7fa4101880 2024-03-28T22:10:24.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.896+0000 7f7fa9b9a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa410c4f0 con 0x7f7fa4102240 2024-03-28T22:10:24.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.896+0000 7f7fa37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fa4100690 0x7f7fa41969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:24.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.896+0000 7f7fa37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fa4100690 0x7f7fa41969b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44526/0 (socket says 172.21.15.16:44526) 2024-03-28T22:10:24.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.896+0000 7f7fa37fe640 1 -- 172.21.15.16:0/3182880473 learned_addr learned my addr 172.21.15.16:0/3182880473 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:24.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.896+0000 7f7fa2ffd640 1 --2- 172.21.15.16:0/3182880473 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fa4101880 0x7f7fa4196ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.896+0000 7f7fa3fff640 1 --2- 172.21.15.16:0/3182880473 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7fa4102240 0x7f7fa419df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.897+0000 7f7fa37fe640 1 -- 172.21.15.16:0/3182880473 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7fa4102240 msgr2=0x7f7fa419df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.897+0000 7f7fa37fe640 1 --2- 172.21.15.16:0/3182880473 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7fa4102240 0x7f7fa419df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.897+0000 7f7fa37fe640 1 -- 172.21.15.16:0/3182880473 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fa4101880 msgr2=0x7f7fa4196ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.897+0000 7f7fa37fe640 1 --2- 172.21.15.16:0/3182880473 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fa4101880 0x7f7fa4196ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.897+0000 7f7fa37fe640 1 -- 172.21.15.16:0/3182880473 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fa419e5d0 con 0x7f7fa4100690 2024-03-28T22:10:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.897+0000 7f7fa2ffd640 1 --2- 172.21.15.16:0/3182880473 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fa4101880 0x7f7fa4196ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:10:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.897+0000 7f7fa37fe640 1 --2- 172.21.15.16:0/3182880473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fa4100690 0x7f7fa41969b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f7f9800e9c0 tx=0x7f7f9800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.897+0000 7f7fa0ff9640 1 -- 172.21.15.16:0/3182880473 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f9800cd80 con 0x7f7fa4100690 2024-03-28T22:10:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.897+0000 7f7fa9b9a640 1 -- 172.21.15.16:0/3182880473 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fa419e8c0 con 0x7f7fa4100690 2024-03-28T22:10:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.897+0000 7f7fa0ff9640 1 -- 172.21.15.16:0/3182880473 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f980035f0 con 0x7f7fa4100690 2024-03-28T22:10:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.897+0000 7f7fa0ff9640 1 -- 172.21.15.16:0/3182880473 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f98010690 con 0x7f7fa4100690 2024-03-28T22:10:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.897+0000 7f7fa9b9a640 1 -- 172.21.15.16:0/3182880473 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fa419ede0 con 0x7f7fa4100690 2024-03-28T22:10:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.898+0000 7f7fa9b9a640 1 -- 172.21.15.16:0/3182880473 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fa41079b0 con 0x7f7fa4100690 2024-03-28T22:10:24.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.899+0000 7f7fa0ff9640 1 -- 172.21.15.16:0/3182880473 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7f7f980037a0 con 0x7f7fa4100690 2024-03-28T22:10:24.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.899+0000 7f7fa0ff9640 1 --2- 172.21.15.16:0/3182880473 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f7f8c0a1e10 0x7f7f8c0a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:24.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.900+0000 7f7fa2ffd640 1 --2- 172.21.15.16:0/3182880473 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f7f8c0a1e10 0x7f7f8c0a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:24.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.900+0000 7f7fa0ff9640 1 -- 172.21.15.16:0/3182880473 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f7f980ce300 con 0x7f7fa4100690 2024-03-28T22:10:24.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.900+0000 7f7fa2ffd640 1 --2- 172.21.15.16:0/3182880473 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f7f8c0a1e10 0x7f7f8c0a42d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7f9402c7b0 tx=0x7f7f94005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:24.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:24.904+0000 7f7fa0ff9640 1 -- 172.21.15.16:0/3182880473 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7f9809b5a0 con 0x7f7fa4100690 2024-03-28T22:10:25.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.086+0000 7f7fa9b9a640 1 -- 172.21.15.16:0/3182880473 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7fa4062350 con 0x7f7fa4100690 2024-03-28T22:10:25.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.087+0000 7f7fa0ff9640 1 -- 172.21.15.16:0/3182880473 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f7f9809aea0 con 0x7f7fa4100690 2024-03-28T22:10:25.089 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-28T22:10:25.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.088+0000 7f7fa9b9a640 1 -- 172.21.15.16:0/3182880473 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f7f8c0a1e10 msgr2=0x7f7f8c0a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:25.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.088+0000 7f7fa9b9a640 1 --2- 172.21.15.16:0/3182880473 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f7f8c0a1e10 0x7f7f8c0a42d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7f9402c7b0 tx=0x7f7f94005fd0 comp rx=0 tx=0).stop 2024-03-28T22:10:25.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.088+0000 7f7fa9b9a640 1 -- 172.21.15.16:0/3182880473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fa4100690 msgr2=0x7f7fa41969b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:25.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.088+0000 7f7fa9b9a640 1 --2- 172.21.15.16:0/3182880473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fa4100690 0x7f7fa41969b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f7f9800e9c0 tx=0x7f7f9800ee80 comp rx=0 tx=0).stop 2024-03-28T22:10:25.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.088+0000 7f7fa9b9a640 1 -- 172.21.15.16:0/3182880473 shutdown_connections 2024-03-28T22:10:25.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.088+0000 7f7fa9b9a640 1 --2- 172.21.15.16:0/3182880473 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f7f8c0a1e10 0x7f7f8c0a42d0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:25.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.088+0000 7f7fa9b9a640 1 --2- 172.21.15.16:0/3182880473 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7fa4102240 0x7f7fa419df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:25.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.088+0000 7f7fa9b9a640 1 --2- 172.21.15.16:0/3182880473 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fa4101880 0x7f7fa4196ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:25.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.088+0000 7f7fa9b9a640 1 --2- 172.21.15.16:0/3182880473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fa4100690 0x7f7fa41969b0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:25.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.088+0000 7f7fa9b9a640 1 -- 172.21.15.16:0/3182880473 >> 172.21.15.16:0/3182880473 conn(0x7f7fa40fbc00 msgr2=0x7f7fa4104b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:25.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.089+0000 7f7fa9b9a640 1 -- 172.21.15.16:0/3182880473 shutdown_connections 2024-03-28T22:10:25.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.089+0000 7f7fa9b9a640 1 -- 172.21.15.16:0/3182880473 wait complete. 2024-03-28T22:10:25.103 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T22:10:25.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.209+0000 7f20469a1640 1 -- 172.21.15.16:0/3069629792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20401074f0 msgr2=0x7f20401098e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:25.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.209+0000 7f20469a1640 1 --2- 172.21.15.16:0/3069629792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20401074f0 0x7f20401098e0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f2038009ca0 tx=0x7f203802c0f0 comp rx=0 tx=0).stop 2024-03-28T22:10:25.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.209+0000 7f20469a1640 1 -- 172.21.15.16:0/3069629792 shutdown_connections 2024-03-28T22:10:25.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.209+0000 7f20469a1640 1 --2- 172.21.15.16:0/3069629792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20401074f0 0x7f20401098e0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:25.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.209+0000 7f20469a1640 1 --2- 172.21.15.16:0/3069629792 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2040104bc0 0x7f2040106fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:25.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.209+0000 7f20469a1640 1 --2- 172.21.15.16:0/3069629792 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2040102290 0x7f2040104680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:25.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.209+0000 7f20469a1640 1 -- 172.21.15.16:0/3069629792 >> 172.21.15.16:0/3069629792 conn(0x7f20400fbbe0 msgr2=0x7f20400fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:25.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.209+0000 7f20469a1640 1 -- 172.21.15.16:0/3069629792 shutdown_connections 2024-03-28T22:10:25.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.209+0000 7f20469a1640 1 -- 172.21.15.16:0/3069629792 wait complete. 2024-03-28T22:10:25.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.210+0000 7f20469a1640 1 Processor -- start 2024-03-28T22:10:25.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.210+0000 7f20469a1640 1 -- start start 2024-03-28T22:10:25.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.210+0000 7f20469a1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2040102290 0x7f2040197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:25.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.210+0000 7f20469a1640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2040104bc0 0x7f2040197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:25.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.210+0000 7f20469a1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20401074f0 0x7f204019f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:25.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.210+0000 7f20469a1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f204010d410 con 0x7f2040102290 2024-03-28T22:10:25.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.210+0000 7f20469a1640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f204010d290 con 0x7f2040104bc0 2024-03-28T22:10:25.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.210+0000 7f20469a1640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f204010d590 con 0x7f20401074f0 2024-03-28T22:10:25.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.210+0000 7f203ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2040102290 0x7f2040197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:25.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.210+0000 7f203ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2040102290 0x7f2040197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44540/0 (socket says 172.21.15.16:44540) 2024-03-28T22:10:25.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.210+0000 7f203ffff640 1 -- 172.21.15.16:0/3729785811 learned_addr learned my addr 172.21.15.16:0/3729785811 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:25.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.210+0000 7f203f7fe640 1 --2- 172.21.15.16:0/3729785811 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2040104bc0 0x7f2040197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:25.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.210+0000 7f2044f3e640 1 --2- 172.21.15.16:0/3729785811 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20401074f0 0x7f204019f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:25.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.210+0000 7f203ffff640 1 -- 172.21.15.16:0/3729785811 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20401074f0 msgr2=0x7f204019f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:25.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.210+0000 7f203ffff640 1 --2- 172.21.15.16:0/3729785811 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20401074f0 0x7f204019f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:25.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.210+0000 7f203ffff640 1 -- 172.21.15.16:0/3729785811 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2040104bc0 msgr2=0x7f2040197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:25.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.210+0000 7f203ffff640 1 --2- 172.21.15.16:0/3729785811 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2040104bc0 0x7f2040197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:25.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.210+0000 7f203ffff640 1 -- 172.21.15.16:0/3729785811 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f204019f6b0 con 0x7f2040102290 2024-03-28T22:10:25.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.210+0000 7f2044f3e640 1 --2- 172.21.15.16:0/3729785811 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20401074f0 0x7f204019f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:10:25.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.211+0000 7f203ffff640 1 --2- 172.21.15.16:0/3729785811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2040102290 0x7f2040197a90 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f203000d770 tx=0x7f203000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:25.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.211+0000 7f203d7fa640 1 -- 172.21.15.16:0/3729785811 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2030017070 con 0x7f2040102290 2024-03-28T22:10:25.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.211+0000 7f203d7fa640 1 -- 172.21.15.16:0/3729785811 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2030005500 con 0x7f2040102290 2024-03-28T22:10:25.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.211+0000 7f20469a1640 1 -- 172.21.15.16:0/3729785811 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f204019f9a0 con 0x7f2040102290 2024-03-28T22:10:25.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.211+0000 7f203d7fa640 1 -- 172.21.15.16:0/3729785811 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2030004e00 con 0x7f2040102290 2024-03-28T22:10:25.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.211+0000 7f20469a1640 1 -- 172.21.15.16:0/3729785811 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f204019fea0 con 0x7f2040102290 2024-03-28T22:10:25.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.212+0000 7f20469a1640 1 -- 172.21.15.16:0/3729785811 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20400fd6d0 con 0x7f2040102290 2024-03-28T22:10:25.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.213+0000 7f203d7fa640 1 -- 172.21.15.16:0/3729785811 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7f2030004fa0 con 0x7f2040102290 2024-03-28T22:10:25.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.213+0000 7f203d7fa640 1 --2- 172.21.15.16:0/3729785811 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f20280a1e10 0x7f20280a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:25.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.214+0000 7f203f7fe640 1 --2- 172.21.15.16:0/3729785811 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f20280a1e10 0x7f20280a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:25.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.214+0000 7f203d7fa640 1 -- 172.21.15.16:0/3729785811 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f20300cd940 con 0x7f2040102290 2024-03-28T22:10:25.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.214+0000 7f203f7fe640 1 --2- 172.21.15.16:0/3729785811 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f20280a1e10 0x7f20280a42d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2040198fa0 tx=0x7f2034005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:25.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.218+0000 7f203d7fa640 1 -- 172.21.15.16:0/3729785811 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f203009ac40 con 0x7f2040102290 2024-03-28T22:10:25.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.400+0000 7f20469a1640 1 -- 172.21.15.16:0/3729785811 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f204002d170 con 0x7f2040102290 2024-03-28T22:10:25.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.401+0000 7f203d7fa640 1 -- 172.21.15.16:0/3729785811 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f204002d170 con 0x7f2040102290 2024-03-28T22:10:25.403 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-28T22:10:25.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.402+0000 7f20469a1640 1 -- 172.21.15.16:0/3729785811 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f20280a1e10 msgr2=0x7f20280a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:25.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.402+0000 7f20469a1640 1 --2- 172.21.15.16:0/3729785811 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f20280a1e10 0x7f20280a42d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2040198fa0 tx=0x7f2034005f90 comp rx=0 tx=0).stop 2024-03-28T22:10:25.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.402+0000 7f20469a1640 1 -- 172.21.15.16:0/3729785811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2040102290 msgr2=0x7f2040197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:25.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.402+0000 7f20469a1640 1 --2- 172.21.15.16:0/3729785811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2040102290 0x7f2040197a90 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f203000d770 tx=0x7f203000dc30 comp rx=0 tx=0).stop 2024-03-28T22:10:25.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.403+0000 7f20469a1640 1 -- 172.21.15.16:0/3729785811 shutdown_connections 2024-03-28T22:10:25.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.403+0000 7f20469a1640 1 --2- 172.21.15.16:0/3729785811 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f20280a1e10 0x7f20280a42d0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:25.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.403+0000 7f20469a1640 1 --2- 172.21.15.16:0/3729785811 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20401074f0 0x7f204019f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:25.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.403+0000 7f20469a1640 1 --2- 172.21.15.16:0/3729785811 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2040104bc0 0x7f2040197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:25.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.403+0000 7f20469a1640 1 --2- 172.21.15.16:0/3729785811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2040102290 0x7f2040197a90 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:25.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.403+0000 7f20469a1640 1 -- 172.21.15.16:0/3729785811 >> 172.21.15.16:0/3729785811 conn(0x7f20400fbbe0 msgr2=0x7f2040102b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:25.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.403+0000 7f20469a1640 1 -- 172.21.15.16:0/3729785811 shutdown_connections 2024-03-28T22:10:25.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.403+0000 7f20469a1640 1 -- 172.21.15.16:0/3729785811 wait complete. 2024-03-28T22:10:25.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:25 smithi016 ceph-mon[32374]: from='client.15345 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:10:25.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:25 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2143763526' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:10:25.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:25 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3182880473' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:10:25.417 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T22:10:25.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.520+0000 7f6bf26ed640 1 -- 172.21.15.16:0/570337175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bec103470 msgr2=0x7f6bec109d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:25.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.520+0000 7f6bf26ed640 1 --2- 172.21.15.16:0/570337175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bec103470 0x7f6bec109d00 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f6be4009cd0 tx=0x7f6be402c0f0 comp rx=0 tx=0).stop 2024-03-28T22:10:25.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.520+0000 7f6bf26ed640 1 -- 172.21.15.16:0/570337175 shutdown_connections 2024-03-28T22:10:25.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.520+0000 7f6bf26ed640 1 --2- 172.21.15.16:0/570337175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bec103470 0x7f6bec109d00 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:25.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.520+0000 7f6bf26ed640 1 --2- 172.21.15.16:0/570337175 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6bec102ab0 0x7f6bec102f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:25.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.520+0000 7f6bf26ed640 1 --2- 172.21.15.16:0/570337175 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6bec0ffa40 0x7f6bec0ffe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:25.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.520+0000 7f6bf26ed640 1 -- 172.21.15.16:0/570337175 >> 172.21.15.16:0/570337175 conn(0x7f6bec0fb330 msgr2=0x7f6bec0fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:25.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.520+0000 7f6bf26ed640 1 -- 172.21.15.16:0/570337175 shutdown_connections 2024-03-28T22:10:25.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.520+0000 7f6bf26ed640 1 -- 172.21.15.16:0/570337175 wait complete. 2024-03-28T22:10:25.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.520+0000 7f6bf26ed640 1 Processor -- start 2024-03-28T22:10:25.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.520+0000 7f6bf26ed640 1 -- start start 2024-03-28T22:10:25.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.521+0000 7f6bf26ed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bec0ffa40 0x7f6bec197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:25.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.521+0000 7f6bf26ed640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6bec102ab0 0x7f6bec197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:25.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.521+0000 7f6bf26ed640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6bec103470 0x7f6bec19efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:25.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.521+0000 7f6bf26ed640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6bec10d3d0 con 0x7f6bec0ffa40 2024-03-28T22:10:25.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.521+0000 7f6bf26ed640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f6bec10d250 con 0x7f6bec102ab0 2024-03-28T22:10:25.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.521+0000 7f6bf26ed640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6bec10d550 con 0x7f6bec103470 2024-03-28T22:10:25.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.521+0000 7f6bebfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bec0ffa40 0x7f6bec197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:25.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.521+0000 7f6bebfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bec0ffa40 0x7f6bec197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44554/0 (socket says 172.21.15.16:44554) 2024-03-28T22:10:25.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.521+0000 7f6bebfff640 1 -- 172.21.15.16:0/272091219 learned_addr learned my addr 172.21.15.16:0/272091219 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:25.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.521+0000 7f6beb7fe640 1 --2- 172.21.15.16:0/272091219 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6bec102ab0 0x7f6bec197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:25.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.521+0000 7f6bf0c8a640 1 --2- 172.21.15.16:0/272091219 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6bec103470 0x7f6bec19efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:25.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.521+0000 7f6bebfff640 1 -- 172.21.15.16:0/272091219 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6bec103470 msgr2=0x7f6bec19efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:25.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.521+0000 7f6bebfff640 1 --2- 172.21.15.16:0/272091219 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6bec103470 0x7f6bec19efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:25.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.521+0000 7f6bebfff640 1 -- 172.21.15.16:0/272091219 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6bec102ab0 msgr2=0x7f6bec197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:25.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.521+0000 7f6bebfff640 1 --2- 172.21.15.16:0/272091219 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6bec102ab0 0x7f6bec197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:25.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.521+0000 7f6bebfff640 1 -- 172.21.15.16:0/272091219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bec19f640 con 0x7f6bec0ffa40 2024-03-28T22:10:25.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.521+0000 7f6bf0c8a640 1 --2- 172.21.15.16:0/272091219 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6bec103470 0x7f6bec19efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:10:25.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.521+0000 7f6bebfff640 1 --2- 172.21.15.16:0/272091219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bec0ffa40 0x7f6bec197a20 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc00e9c0 tx=0x7f6bdc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:25.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.521+0000 7f6be97fa640 1 -- 172.21.15.16:0/272091219 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bdc00cd80 con 0x7f6bec0ffa40 2024-03-28T22:10:25.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.521+0000 7f6be97fa640 1 -- 172.21.15.16:0/272091219 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6bdc0035f0 con 0x7f6bec0ffa40 2024-03-28T22:10:25.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.521+0000 7f6bf26ed640 1 -- 172.21.15.16:0/272091219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bec19f930 con 0x7f6bec0ffa40 2024-03-28T22:10:25.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.521+0000 7f6be97fa640 1 -- 172.21.15.16:0/272091219 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bdc010690 con 0x7f6bec0ffa40 2024-03-28T22:10:25.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.522+0000 7f6bf26ed640 1 -- 172.21.15.16:0/272091219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bec19fed0 con 0x7f6bec0ffa40 2024-03-28T22:10:25.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.523+0000 7f6bf26ed640 1 -- 172.21.15.16:0/272091219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bec108be0 con 0x7f6bec0ffa40 2024-03-28T22:10:25.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.524+0000 7f6be97fa640 1 -- 172.21.15.16:0/272091219 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7f6bdc0037a0 con 0x7f6bec0ffa40 2024-03-28T22:10:25.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.524+0000 7f6be97fa640 1 --2- 172.21.15.16:0/272091219 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f6bd80a1e90 0x7f6bd80a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:25.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.524+0000 7f6beb7fe640 1 --2- 172.21.15.16:0/272091219 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f6bd80a1e90 0x7f6bd80a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:25.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.524+0000 7f6be97fa640 1 -- 172.21.15.16:0/272091219 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f6bdc0ce2c0 con 0x7f6bec0ffa40 2024-03-28T22:10:25.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.525+0000 7f6beb7fe640 1 --2- 172.21.15.16:0/272091219 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f6bd80a1e90 0x7f6bd80a4350 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f6bec198f30 tx=0x7f6bd400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:25.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.528+0000 7f6be97fa640 1 -- 172.21.15.16:0/272091219 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6bdc09b5c0 con 0x7f6bec0ffa40 2024-03-28T22:10:25.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:25 smithi053 ceph-mon[35979]: from='client.15345 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:10:25.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:25 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2143763526' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:10:25.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:25 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3182880473' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:10:25.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:25 smithi063 ceph-mon[36249]: from='client.15345 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:10:25.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:25 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2143763526' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:10:25.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:25 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/3182880473' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:10:25.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.709+0000 7f6bf26ed640 1 -- 172.21.15.16:0/272091219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6bec02d170 con 0x7f6bec0ffa40 2024-03-28T22:10:25.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.709+0000 7f6be97fa640 1 -- 172.21.15.16:0/272091219 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f6bdc09aec0 con 0x7f6bec0ffa40 2024-03-28T22:10:25.712 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-28T22:10:25.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.712+0000 7f6bf26ed640 1 -- 172.21.15.16:0/272091219 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f6bd80a1e90 msgr2=0x7f6bd80a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:25.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.712+0000 7f6bf26ed640 1 --2- 172.21.15.16:0/272091219 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f6bd80a1e90 0x7f6bd80a4350 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f6bec198f30 tx=0x7f6bd400b390 comp rx=0 tx=0).stop 2024-03-28T22:10:25.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.712+0000 7f6bf26ed640 1 -- 172.21.15.16:0/272091219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bec0ffa40 msgr2=0x7f6bec197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:25.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.712+0000 7f6bf26ed640 1 --2- 172.21.15.16:0/272091219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bec0ffa40 0x7f6bec197a20 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc00e9c0 tx=0x7f6bdc00ee80 comp rx=0 tx=0).stop 2024-03-28T22:10:25.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.712+0000 7f6bf26ed640 1 -- 172.21.15.16:0/272091219 shutdown_connections 2024-03-28T22:10:25.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.712+0000 7f6bf26ed640 1 --2- 172.21.15.16:0/272091219 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f6bd80a1e90 0x7f6bd80a4350 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:25.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.712+0000 7f6bf26ed640 1 --2- 172.21.15.16:0/272091219 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6bec103470 0x7f6bec19efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:25.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.712+0000 7f6bf26ed640 1 --2- 172.21.15.16:0/272091219 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6bec102ab0 0x7f6bec197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:25.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.712+0000 7f6bf26ed640 1 --2- 172.21.15.16:0/272091219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bec0ffa40 0x7f6bec197a20 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:25.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.712+0000 7f6bf26ed640 1 -- 172.21.15.16:0/272091219 >> 172.21.15.16:0/272091219 conn(0x7f6bec0fb330 msgr2=0x7f6bec105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:25.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.712+0000 7f6bf26ed640 1 -- 172.21.15.16:0/272091219 shutdown_connections 2024-03-28T22:10:25.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.712+0000 7f6bf26ed640 1 -- 172.21.15.16:0/272091219 wait complete. 2024-03-28T22:10:25.726 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:10:25.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.827+0000 7fd88bf8c640 1 -- 172.21.15.16:0/548339596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd884102230 msgr2=0x7fd884108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:25.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.827+0000 7fd88bf8c640 1 --2- 172.21.15.16:0/548339596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd884102230 0x7fd884108ac0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd87c009ca0 tx=0x7fd87c02c0f0 comp rx=0 tx=0).stop 2024-03-28T22:10:25.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.827+0000 7fd88bf8c640 1 -- 172.21.15.16:0/548339596 shutdown_connections 2024-03-28T22:10:25.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.827+0000 7fd88bf8c640 1 --2- 172.21.15.16:0/548339596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd884102230 0x7fd884108ac0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:25.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.827+0000 7fd88bf8c640 1 --2- 172.21.15.16:0/548339596 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd884101870 0x7fd884101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:25.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.827+0000 7fd88bf8c640 1 --2- 172.21.15.16:0/548339596 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd884100680 0x7fd884100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:25.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.827+0000 7fd88bf8c640 1 -- 172.21.15.16:0/548339596 >> 172.21.15.16:0/548339596 conn(0x7fd8840fbbf0 msgr2=0x7fd8840fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:25.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.827+0000 7fd88bf8c640 1 -- 172.21.15.16:0/548339596 shutdown_connections 2024-03-28T22:10:25.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.827+0000 7fd88bf8c640 1 -- 172.21.15.16:0/548339596 wait complete. 2024-03-28T22:10:25.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.827+0000 7fd88bf8c640 1 Processor -- start 2024-03-28T22:10:25.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.827+0000 7fd88bf8c640 1 -- start start 2024-03-28T22:10:25.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.828+0000 7fd88bf8c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd884100680 0x7fd884104610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:25.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.828+0000 7fd88bf8c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd884101870 0x7fd884102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:25.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.828+0000 7fd88bf8c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd884102230 0x7fd884103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:25.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.828+0000 7fd88bf8c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd88410c300 con 0x7fd884102230 2024-03-28T22:10:25.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.828+0000 7fd88bf8c640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd88410c180 con 0x7fd884101870 2024-03-28T22:10:25.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.828+0000 7fd88bf8c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd88410c480 con 0x7fd884100680 2024-03-28T22:10:25.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.828+0000 7fd88a529640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd884102230 0x7fd884103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:25.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.828+0000 7fd88a529640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd884102230 0x7fd884103190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44568/0 (socket says 172.21.15.16:44568) 2024-03-28T22:10:25.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.828+0000 7fd88a529640 1 -- 172.21.15.16:0/3332510513 learned_addr learned my addr 172.21.15.16:0/3332510513 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:25.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.828+0000 7fd889527640 1 --2- 172.21.15.16:0/3332510513 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd884101870 0x7fd884102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:25.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.828+0000 7fd889d28640 1 --2- 172.21.15.16:0/3332510513 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd884100680 0x7fd884104610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:25.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.828+0000 7fd88a529640 1 -- 172.21.15.16:0/3332510513 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd884100680 msgr2=0x7fd884104610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:25.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.828+0000 7fd88a529640 1 --2- 172.21.15.16:0/3332510513 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd884100680 0x7fd884104610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:25.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.828+0000 7fd88a529640 1 -- 172.21.15.16:0/3332510513 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd884101870 msgr2=0x7fd884102c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:25.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.828+0000 7fd88a529640 1 --2- 172.21.15.16:0/3332510513 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd884101870 0x7fd884102c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:25.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.828+0000 7fd88a529640 1 -- 172.21.15.16:0/3332510513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd884103a20 con 0x7fd884102230 2024-03-28T22:10:25.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.828+0000 7fd889527640 1 --2- 172.21.15.16:0/3332510513 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd884101870 0x7fd884102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:10:25.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.828+0000 7fd88a529640 1 --2- 172.21.15.16:0/3332510513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd884102230 0x7fd884103190 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fd87c032530 tx=0x7fd87c032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:25.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.828+0000 7fd87affd640 1 -- 172.21.15.16:0/3332510513 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd87c041020 con 0x7fd884102230 2024-03-28T22:10:25.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.828+0000 7fd87affd640 1 -- 172.21.15.16:0/3332510513 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd87c034cd0 con 0x7fd884102230 2024-03-28T22:10:25.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.828+0000 7fd88bf8c640 1 -- 172.21.15.16:0/3332510513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8841a1b60 con 0x7fd884102230 2024-03-28T22:10:25.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.828+0000 7fd87affd640 1 -- 172.21.15.16:0/3332510513 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd87c039040 con 0x7fd884102230 2024-03-28T22:10:25.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.828+0000 7fd88bf8c640 1 -- 172.21.15.16:0/3332510513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8841a20d0 con 0x7fd884102230 2024-03-28T22:10:25.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.829+0000 7fd88bf8c640 1 -- 172.21.15.16:0/3332510513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8841a22a0 con 0x7fd884102230 2024-03-28T22:10:25.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.831+0000 7fd87affd640 1 -- 172.21.15.16:0/3332510513 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7fd87c044020 con 0x7fd884102230 2024-03-28T22:10:25.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.831+0000 7fd87affd640 1 --2- 172.21.15.16:0/3332510513 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fd8700a1e90 0x7fd8700a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:25.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.831+0000 7fd889d28640 1 --2- 172.21.15.16:0/3332510513 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fd8700a1e90 0x7fd8700a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:25.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.831+0000 7fd87affd640 1 -- 172.21.15.16:0/3332510513 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fd87c0eff90 con 0x7fd884102230 2024-03-28T22:10:25.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.832+0000 7fd889d28640 1 --2- 172.21.15.16:0/3332510513 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fd8700a1e90 0x7fd8700a4350 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd8740095f0 tx=0x7fd874005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:25.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:25.835+0000 7fd87affd640 1 -- 172.21.15.16:0/3332510513 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd87c04b020 con 0x7fd884102230 2024-03-28T22:10:26.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.068+0000 7fd88bf8c640 1 -- 172.21.15.16:0/3332510513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd8841a2630 con 0x7fd884102230 2024-03-28T22:10:26.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.069+0000 7fd87affd640 1 -- 172.21.15.16:0/3332510513 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11851 (secure 0 0 0) 0x7fd87c0bd3a0 con 0x7fd884102230 2024-03-28T22:10:26.072 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:10:26.072 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/1644581227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1644581227},{"type":"v1","addr":"172.21.15.16:6839","nonce":1644581227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/2593062669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2593062669},{"type":"v1","addr":"172.21.15.16:6841","nonce":2593062669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:08:29.672976+0000","modified":"2024-03-28T22:10:19.955098+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24566,"mds_1":24524,"mds_2":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24466":{"gid":24466,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/247249857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":247249857},{"type":"v1","addr":"172.21.15.63:6835","nonce":247249857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6837/2324231032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":2324231032},{"type":"v1","addr":"172.21.15.63:6837","nonce":2324231032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6839/1177197077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":1177197077},{"type":"v1","addr":"172.21.15.63:6839","nonce":1177197077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"e","rank":2,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.53:6835/350727634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":350727634},{"type":"v1","addr":"172.21.15.53:6835","nonce":350727634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/349877573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":349877573},{"type":"v1","addr":"172.21.15.53:6837","nonce":349877573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6839/3714153956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":3714153956},{"type":"v1","addr":"172.21.15.53:6839","nonce":3714153956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24566,"qdb_cluster":[24500,24524,24566]},"id":1}]} 2024-03-28T22:10:26.072 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 21 2024-03-28T22:10:26.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.071+0000 7fd88bf8c640 1 -- 172.21.15.16:0/3332510513 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fd8700a1e90 msgr2=0x7fd8700a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:26.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.071+0000 7fd88bf8c640 1 --2- 172.21.15.16:0/3332510513 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fd8700a1e90 0x7fd8700a4350 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd8740095f0 tx=0x7fd874005f90 comp rx=0 tx=0).stop 2024-03-28T22:10:26.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.071+0000 7fd88bf8c640 1 -- 172.21.15.16:0/3332510513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd884102230 msgr2=0x7fd884103190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:26.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.071+0000 7fd88bf8c640 1 --2- 172.21.15.16:0/3332510513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd884102230 0x7fd884103190 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fd87c032530 tx=0x7fd87c032f20 comp rx=0 tx=0).stop 2024-03-28T22:10:26.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.071+0000 7fd88bf8c640 1 -- 172.21.15.16:0/3332510513 shutdown_connections 2024-03-28T22:10:26.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.071+0000 7fd88bf8c640 1 --2- 172.21.15.16:0/3332510513 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fd8700a1e90 0x7fd8700a4350 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.071+0000 7fd88bf8c640 1 --2- 172.21.15.16:0/3332510513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd884102230 0x7fd884103190 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.071+0000 7fd88bf8c640 1 --2- 172.21.15.16:0/3332510513 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd884101870 0x7fd884102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.071+0000 7fd88bf8c640 1 --2- 172.21.15.16:0/3332510513 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd884100680 0x7fd884104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.071+0000 7fd88bf8c640 1 -- 172.21.15.16:0/3332510513 >> 172.21.15.16:0/3332510513 conn(0x7fd8840fbbf0 msgr2=0x7fd8840fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:26.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.071+0000 7fd88bf8c640 1 -- 172.21.15.16:0/3332510513 shutdown_connections 2024-03-28T22:10:26.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.071+0000 7fd88bf8c640 1 -- 172.21.15.16:0/3332510513 wait complete. 2024-03-28T22:10:26.085 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:10:26.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.185+0000 7fce82eb5640 1 -- 172.21.15.16:0/3045007433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce7c102230 msgr2=0x7fce7c108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:26.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.185+0000 7fce82eb5640 1 --2- 172.21.15.16:0/3045007433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce7c102230 0x7fce7c108ac0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fce64009900 tx=0x7fce6402bc30 comp rx=0 tx=0).stop 2024-03-28T22:10:26.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.185+0000 7fce82eb5640 1 -- 172.21.15.16:0/3045007433 shutdown_connections 2024-03-28T22:10:26.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.185+0000 7fce82eb5640 1 --2- 172.21.15.16:0/3045007433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce7c102230 0x7fce7c108ac0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.185+0000 7fce82eb5640 1 --2- 172.21.15.16:0/3045007433 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce7c101870 0x7fce7c101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.185+0000 7fce82eb5640 1 --2- 172.21.15.16:0/3045007433 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce7c100680 0x7fce7c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.185+0000 7fce82eb5640 1 -- 172.21.15.16:0/3045007433 >> 172.21.15.16:0/3045007433 conn(0x7fce7c0fbbf0 msgr2=0x7fce7c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:26.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.185+0000 7fce82eb5640 1 -- 172.21.15.16:0/3045007433 shutdown_connections 2024-03-28T22:10:26.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.185+0000 7fce82eb5640 1 -- 172.21.15.16:0/3045007433 wait complete. 2024-03-28T22:10:26.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.186+0000 7fce82eb5640 1 Processor -- start 2024-03-28T22:10:26.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.186+0000 7fce82eb5640 1 -- start start 2024-03-28T22:10:26.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.186+0000 7fce82eb5640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce7c100680 0x7fce7c197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.186+0000 7fce82eb5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce7c101870 0x7fce7c197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.186+0000 7fce82eb5640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce7c102230 0x7fce7c19ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.186+0000 7fce82eb5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fce7c10c300 con 0x7fce7c101870 2024-03-28T22:10:26.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.186+0000 7fce82eb5640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fce7c10c180 con 0x7fce7c100680 2024-03-28T22:10:26.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.186+0000 7fce82eb5640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fce7c10c480 con 0x7fce7c102230 2024-03-28T22:10:26.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.186+0000 7fce7bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce7c101870 0x7fce7c197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.186+0000 7fce7bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce7c101870 0x7fce7c197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44574/0 (socket says 172.21.15.16:44574) 2024-03-28T22:10:26.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.186+0000 7fce7bfff640 1 -- 172.21.15.16:0/2500006459 learned_addr learned my addr 172.21.15.16:0/2500006459 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:26.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.186+0000 7fce80c51640 1 --2- 172.21.15.16:0/2500006459 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce7c100680 0x7fce7c197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.186+0000 7fce81452640 1 --2- 172.21.15.16:0/2500006459 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce7c102230 0x7fce7c19ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.186+0000 7fce7bfff640 1 -- 172.21.15.16:0/2500006459 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce7c102230 msgr2=0x7fce7c19ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:26.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.186+0000 7fce7bfff640 1 --2- 172.21.15.16:0/2500006459 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce7c102230 0x7fce7c19ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.186+0000 7fce7bfff640 1 -- 172.21.15.16:0/2500006459 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce7c100680 msgr2=0x7fce7c197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:26.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.186+0000 7fce7bfff640 1 --2- 172.21.15.16:0/2500006459 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce7c100680 0x7fce7c197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.186+0000 7fce7bfff640 1 -- 172.21.15.16:0/2500006459 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce7c19f600 con 0x7fce7c101870 2024-03-28T22:10:26.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.186+0000 7fce80c51640 1 --2- 172.21.15.16:0/2500006459 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce7c100680 0x7fce7c197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:10:26.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.186+0000 7fce7bfff640 1 --2- 172.21.15.16:0/2500006459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce7c101870 0x7fce7c197f70 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fce6c00ebc0 tx=0x7fce6c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:26.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.186+0000 7fce79ffb640 1 -- 172.21.15.16:0/2500006459 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce6c010630 con 0x7fce7c101870 2024-03-28T22:10:26.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.187+0000 7fce79ffb640 1 -- 172.21.15.16:0/2500006459 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce6c010cb0 con 0x7fce7c101870 2024-03-28T22:10:26.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.187+0000 7fce82eb5640 1 -- 172.21.15.16:0/2500006459 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce7c19f8f0 con 0x7fce7c101870 2024-03-28T22:10:26.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.187+0000 7fce79ffb640 1 -- 172.21.15.16:0/2500006459 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce6c018bd0 con 0x7fce7c101870 2024-03-28T22:10:26.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.187+0000 7fce82eb5640 1 -- 172.21.15.16:0/2500006459 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce7c19fe90 con 0x7fce7c101870 2024-03-28T22:10:26.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.188+0000 7fce82eb5640 1 -- 172.21.15.16:0/2500006459 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce7c1a0060 con 0x7fce7c101870 2024-03-28T22:10:26.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.192+0000 7fce79ffb640 1 -- 172.21.15.16:0/2500006459 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7fce6c0107d0 con 0x7fce7c101870 2024-03-28T22:10:26.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.192+0000 7fce79ffb640 1 --2- 172.21.15.16:0/2500006459 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fce680a1f10 0x7fce680a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.192+0000 7fce80c51640 1 --2- 172.21.15.16:0/2500006459 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fce680a1f10 0x7fce680a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.192+0000 7fce79ffb640 1 -- 172.21.15.16:0/2500006459 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fce6c0ce7a0 con 0x7fce7c101870 2024-03-28T22:10:26.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.193+0000 7fce80c51640 1 --2- 172.21.15.16:0/2500006459 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fce680a1f10 0x7fce680a43d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fce74005fd0 tx=0x7fce7400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:26.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.195+0000 7fce79ffb640 1 -- 172.21.15.16:0/2500006459 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fce6c09ba40 con 0x7fce7c101870 2024-03-28T22:10:26.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.378+0000 7fce82eb5640 1 -- 172.21.15.16:0/2500006459 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fce7c1a03f0 con 0x7fce7c101870 2024-03-28T22:10:26.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.379+0000 7fce79ffb640 1 -- 172.21.15.16:0/2500006459 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22441 (secure 0 0 0) 0x7fce6c09b340 con 0x7fce7c101870 2024-03-28T22:10:26.382 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:10:26.383 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":88,"fsid":"798d9884-ed4e-11ee-95d0-87774f69a715","created":"2024-03-28T22:00:27.216099+0000","modified":"2024-03-28T22:09:30.047847+0000","last_up_change":"2024-03-28T22:07:40.251141+0000","last_in_change":"2024-03-28T22:07:19.456451+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:03:59.327832+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:08:28.123529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:08:28.919037+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"46e7c8f8-ecc8-43d6-ae53-0655728f3240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6803","nonce":1589692483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6805","nonce":1589692483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6809","nonce":1589692483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6807","nonce":1589692483}]},"public_addr":"172.21.15.16:6803/1589692483","cluster_addr":"172.21.15.16:6805/1589692483","heartbeat_back_addr":"172.21.15.16:6809/1589692483","heartbeat_front_addr":"172.21.15.16:6807/1589692483","state":["exists","up"]},{"osd":1,"uuid":"af6d9c22-2830-4ce2-a547-011df0c6cbad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6801","nonce":795531234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6803","nonce":795531234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6807","nonce":795531234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6805","nonce":795531234}]},"public_addr":"172.21.15.53:6801/795531234","cluster_addr":"172.21.15.53:6803/795531234","heartbeat_back_addr":"172.21.15.53:6807/795531234","heartbeat_front_addr":"172.21.15.53:6805/795531234","state":["exists","up"]},{"osd":2,"uuid":"b10adca9-110c-4f87-85e0-b7542709d31a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6801","nonce":2214131086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6803","nonce":2214131086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6807","nonce":2214131086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6805","nonce":2214131086}]},"public_addr":"172.21.15.63:6801/2214131086","cluster_addr":"172.21.15.63:6803/2214131086","heartbeat_back_addr":"172.21.15.63:6807/2214131086","heartbeat_front_addr":"172.21.15.63:6805/2214131086","state":["exists","up"]},{"osd":3,"uuid":"c83cf4bb-395a-4786-abf7-74ca93a33eb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6811","nonce":13176789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6813","nonce":13176789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6817","nonce":13176789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6815","nonce":13176789}]},"public_addr":"172.21.15.16:6811/13176789","cluster_addr":"172.21.15.16:6813/13176789","heartbeat_back_addr":"172.21.15.16:6817/13176789","heartbeat_front_addr":"172.21.15.16:6815/13176789","state":["exists","up"]},{"osd":4,"uuid":"941b3e54-47d6-4e85-8069-e730c3fa8eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6809","nonce":772161849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6811","nonce":772161849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6815","nonce":772161849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6813","nonce":772161849}]},"public_addr":"172.21.15.53:6809/772161849","cluster_addr":"172.21.15.53:6811/772161849","heartbeat_back_addr":"172.21.15.53:6815/772161849","heartbeat_front_addr":"172.21.15.53:6813/772161849","state":["exists","up"]},{"osd":5,"uuid":"51d89826-1d60-40fc-8fd8-09bfe1ab4125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6809","nonce":1602554852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6811","nonce":1602554852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6815","nonce":1602554852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6813","nonce":1602554852}]},"public_addr":"172.21.15.63:6809/1602554852","cluster_addr":"172.21.15.63:6811/1602554852","heartbeat_back_addr":"172.21.15.63:6815/1602554852","heartbeat_front_addr":"172.21.15.63:6813/1602554852","state":["exists","up"]},{"osd":6,"uuid":"978efe5d-2c55-4ff8-8a1e-9f891388945e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6819","nonce":4290459224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6821","nonce":4290459224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6825","nonce":4290459224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6823","nonce":4290459224}]},"public_addr":"172.21.15.16:6819/4290459224","cluster_addr":"172.21.15.16:6821/4290459224","heartbeat_back_addr":"172.21.15.16:6825/4290459224","heartbeat_front_addr":"172.21.15.16:6823/4290459224","state":["exists","up"]},{"osd":7,"uuid":"50d489cb-dbb8-48df-b868-b29edf12e04d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6817","nonce":1056467525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6819","nonce":1056467525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6823","nonce":1056467525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6821","nonce":1056467525}]},"public_addr":"172.21.15.53:6817/1056467525","cluster_addr":"172.21.15.53:6819/1056467525","heartbeat_back_addr":"172.21.15.53:6823/1056467525","heartbeat_front_addr":"172.21.15.53:6821/1056467525","state":["exists","up"]},{"osd":8,"uuid":"49b39414-cc9c-4057-ae3f-cc109cd979da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6817","nonce":2918516794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6819","nonce":2918516794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6823","nonce":2918516794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6821","nonce":2918516794}]},"public_addr":"172.21.15.63:6817/2918516794","cluster_addr":"172.21.15.63:6819/2918516794","heartbeat_back_addr":"172.21.15.63:6823/2918516794","heartbeat_front_addr":"172.21.15.63:6821/2918516794","state":["exists","up"]},{"osd":9,"uuid":"4c873e51-b783-4eb6-83a3-fee31722d60b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6827","nonce":1585624367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6829","nonce":1585624367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6833","nonce":1585624367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6831","nonce":1585624367}]},"public_addr":"172.21.15.16:6827/1585624367","cluster_addr":"172.21.15.16:6829/1585624367","heartbeat_back_addr":"172.21.15.16:6833/1585624367","heartbeat_front_addr":"172.21.15.16:6831/1585624367","state":["exists","up"]},{"osd":10,"uuid":"085a78bf-e2ad-481a-85a3-71d1fc549a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6825","nonce":3357446484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6827","nonce":3357446484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6831","nonce":3357446484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6829","nonce":3357446484}]},"public_addr":"172.21.15.53:6825/3357446484","cluster_addr":"172.21.15.53:6827/3357446484","heartbeat_back_addr":"172.21.15.53:6831/3357446484","heartbeat_front_addr":"172.21.15.53:6829/3357446484","state":["exists","up"]},{"osd":11,"uuid":"13302aa3-08b9-4639-9042-27d850b4d5f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6825","nonce":2167866570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6827","nonce":2167866570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6831","nonce":2167866570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6829","nonce":2167866570}]},"public_addr":"172.21.15.63:6825/2167866570","cluster_addr":"172.21.15.63:6827/2167866570","heartbeat_back_addr":"172.21.15.63:6831/2167866570","heartbeat_front_addr":"172.21.15.63:6829/2167866570","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:07.450323+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:31.363172+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:55.295021+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:19.794760+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:44.063492+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:08.195064+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:33.655328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:57.639414+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:22.402394+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:47.462050+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:12.226307+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:37.495625+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3821408150":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6801/1832422779":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/996598130":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/1062264551":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/317557779":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6801/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/3631746518":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6800/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/3116652786":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6800/1832422779":"2024-03-29T22:01:28.993434+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:10:26.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.381+0000 7fce82eb5640 1 -- 172.21.15.16:0/2500006459 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fce680a1f10 msgr2=0x7fce680a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:26.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.381+0000 7fce82eb5640 1 --2- 172.21.15.16:0/2500006459 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fce680a1f10 0x7fce680a43d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fce74005fd0 tx=0x7fce7400b390 comp rx=0 tx=0).stop 2024-03-28T22:10:26.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.381+0000 7fce82eb5640 1 -- 172.21.15.16:0/2500006459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce7c101870 msgr2=0x7fce7c197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:26.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.381+0000 7fce82eb5640 1 --2- 172.21.15.16:0/2500006459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce7c101870 0x7fce7c197f70 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fce6c00ebc0 tx=0x7fce6c00c6a0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.381+0000 7fce82eb5640 1 -- 172.21.15.16:0/2500006459 shutdown_connections 2024-03-28T22:10:26.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.381+0000 7fce82eb5640 1 --2- 172.21.15.16:0/2500006459 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fce680a1f10 0x7fce680a43d0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.381+0000 7fce82eb5640 1 --2- 172.21.15.16:0/2500006459 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce7c102230 0x7fce7c19ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.381+0000 7fce82eb5640 1 --2- 172.21.15.16:0/2500006459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce7c101870 0x7fce7c197f70 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.381+0000 7fce82eb5640 1 --2- 172.21.15.16:0/2500006459 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce7c100680 0x7fce7c197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.381+0000 7fce82eb5640 1 -- 172.21.15.16:0/2500006459 >> 172.21.15.16:0/2500006459 conn(0x7fce7c0fbbf0 msgr2=0x7fce7c0fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:26.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.382+0000 7fce82eb5640 1 -- 172.21.15.16:0/2500006459 shutdown_connections 2024-03-28T22:10:26.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.382+0000 7fce82eb5640 1 -- 172.21.15.16:0/2500006459 wait complete. 2024-03-28T22:10:26.394 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:26 smithi016 ceph-mon[32374]: pgmap v346: 97 pgs: 97 active+clean; 646 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-28T22:10:26.394 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:26 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3729785811' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:10:26.394 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:26 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/272091219' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:10:26.395 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:26 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3332510513' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:10:26.396 DEBUG:tasks.fwd_scrub:Yielding 2024-03-28T22:10:26.396 INFO:teuthology.run_tasks:Running task exec... 2024-03-28T22:10:26.409 INFO:teuthology.task.exec:Executing custom commands... 2024-03-28T22:10:26.409 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-03-28T22:10:26.410 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-03-28T22:10:26.410 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-03-28T22:10:26.410 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:10:26.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:10:26.410 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:10:26.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.511+0000 7f80e98c2640 1 -- 172.21.15.16:0/1979084088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80e4102990 msgr2=0x7f80e4102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:26.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.511+0000 7f80e98c2640 1 --2- 172.21.15.16:0/1979084088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80e4102990 0x7f80e4102e10 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f80d4009900 tx=0x7f80d402bc30 comp rx=0 tx=0).stop 2024-03-28T22:10:26.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.511+0000 7f80e98c2640 1 -- 172.21.15.16:0/1979084088 shutdown_connections 2024-03-28T22:10:26.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.511+0000 7f80e98c2640 1 --2- 172.21.15.16:0/1979084088 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80e4103350 0x7f80e4109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.511+0000 7f80e98c2640 1 --2- 172.21.15.16:0/1979084088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80e4102990 0x7f80e4102e10 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.511+0000 7f80e98c2640 1 --2- 172.21.15.16:0/1979084088 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f80e41017a0 0x7f80e4101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.511+0000 7f80e98c2640 1 -- 172.21.15.16:0/1979084088 >> 172.21.15.16:0/1979084088 conn(0x7f80e40fcd10 msgr2=0x7f80e40ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:26.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.511+0000 7f80e98c2640 1 -- 172.21.15.16:0/1979084088 shutdown_connections 2024-03-28T22:10:26.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.512+0000 7f80e98c2640 1 -- 172.21.15.16:0/1979084088 wait complete. 2024-03-28T22:10:26.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.512+0000 7f80e98c2640 1 Processor -- start 2024-03-28T22:10:26.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.512+0000 7f80e98c2640 1 -- start start 2024-03-28T22:10:26.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.512+0000 7f80e98c2640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80e41017a0 0x7f80e4075500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.512+0000 7f80e98c2640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f80e4102990 0x7f80e4073b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.512+0000 7f80e98c2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80e4103350 0x7f80e4074080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.512+0000 7f80e98c2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f80e410d470 con 0x7f80e4103350 2024-03-28T22:10:26.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.512+0000 7f80e98c2640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f80e410d2f0 con 0x7f80e41017a0 2024-03-28T22:10:26.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.512+0000 7f80e98c2640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f80e410d5f0 con 0x7f80e4102990 2024-03-28T22:10:26.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.512+0000 7f80e37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80e4103350 0x7f80e4074080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.513+0000 7f80e2ffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80e41017a0 0x7f80e4075500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.513+0000 7f80e37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80e4103350 0x7f80e4074080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44602/0 (socket says 172.21.15.16:44602) 2024-03-28T22:10:26.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.513+0000 7f80e37fe640 1 -- 172.21.15.16:0/828561439 learned_addr learned my addr 172.21.15.16:0/828561439 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:26.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.513+0000 7f80e27fc640 1 --2- 172.21.15.16:0/828561439 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f80e4102990 0x7f80e4073b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.513+0000 7f80e37fe640 1 -- 172.21.15.16:0/828561439 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f80e4102990 msgr2=0x7f80e4073b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:26.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.513+0000 7f80e37fe640 1 --2- 172.21.15.16:0/828561439 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f80e4102990 0x7f80e4073b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.513+0000 7f80e37fe640 1 -- 172.21.15.16:0/828561439 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80e41017a0 msgr2=0x7f80e4075500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:26.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.513+0000 7f80e37fe640 1 --2- 172.21.15.16:0/828561439 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80e41017a0 0x7f80e4075500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.513+0000 7f80e37fe640 1 -- 172.21.15.16:0/828561439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80e4074910 con 0x7f80e4103350 2024-03-28T22:10:26.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.513+0000 7f80e2ffd640 1 --2- 172.21.15.16:0/828561439 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80e41017a0 0x7f80e4075500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:10:26.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.513+0000 7f80e37fe640 1 --2- 172.21.15.16:0/828561439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80e4103350 0x7f80e4074080 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f80dc00eb10 tx=0x7f80dc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:26.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.513+0000 7f80e88c0640 1 -- 172.21.15.16:0/828561439 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80dc004b20 con 0x7f80e4103350 2024-03-28T22:10:26.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.513+0000 7f80e98c2640 1 -- 172.21.15.16:0/828561439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80e41a1be0 con 0x7f80e4103350 2024-03-28T22:10:26.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.513+0000 7f80e88c0640 1 -- 172.21.15.16:0/828561439 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80dc004cc0 con 0x7f80e4103350 2024-03-28T22:10:26.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.513+0000 7f80e98c2640 1 -- 172.21.15.16:0/828561439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80e41a2100 con 0x7f80e4103350 2024-03-28T22:10:26.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.513+0000 7f80e88c0640 1 -- 172.21.15.16:0/828561439 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80dc010760 con 0x7f80e4103350 2024-03-28T22:10:26.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.515+0000 7f80e98c2640 1 -- 172.21.15.16:0/828561439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80e404fd00 con 0x7f80e4103350 2024-03-28T22:10:26.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.515+0000 7fc01c52e640 1 -- 172.21.15.16:0/1400862781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc014100670 msgr2=0x7fc014100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:26.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.515+0000 7fc01c52e640 1 --2- 172.21.15.16:0/1400862781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc014100670 0x7fc014100a90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc010009860 tx=0x7fc01002bb70 comp rx=0 tx=0).stop 2024-03-28T22:10:26.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.516+0000 7fc01c52e640 1 -- 172.21.15.16:0/1400862781 shutdown_connections 2024-03-28T22:10:26.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.516+0000 7fc01c52e640 1 --2- 172.21.15.16:0/1400862781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc014102220 0x7fc014108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.516+0000 7fc01c52e640 1 --2- 172.21.15.16:0/1400862781 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc014101860 0x7fc014101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.516+0000 7fc01c52e640 1 --2- 172.21.15.16:0/1400862781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc014100670 0x7fc014100a90 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.516+0000 7fc01c52e640 1 -- 172.21.15.16:0/1400862781 >> 172.21.15.16:0/1400862781 conn(0x7fc0140fbbe0 msgr2=0x7fc0140fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:26.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.516+0000 7fc01c52e640 1 -- 172.21.15.16:0/1400862781 shutdown_connections 2024-03-28T22:10:26.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.516+0000 7fc01c52e640 1 -- 172.21.15.16:0/1400862781 wait complete. 2024-03-28T22:10:26.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.516+0000 7f80e88c0640 1 -- 172.21.15.16:0/828561439 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7f80dc010900 con 0x7f80e4103350 2024-03-28T22:10:26.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.516+0000 7fc01c52e640 1 Processor -- start 2024-03-28T22:10:26.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.516+0000 7fc01c52e640 1 -- start start 2024-03-28T22:10:26.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.516+0000 7f80e88c0640 1 --2- 172.21.15.16:0/828561439 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f80cc0a1e80 0x7f80cc0a4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.516+0000 7fc01c52e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc014100670 0x7fc014197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.516+0000 7fc01c52e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc014101860 0x7fc0141980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.516+0000 7f80e2ffd640 1 --2- 172.21.15.16:0/828561439 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f80cc0a1e80 0x7f80cc0a4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.517+0000 7f80e88c0640 1 -- 172.21.15.16:0/828561439 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f80dc0ce660 con 0x7f80e4103350 2024-03-28T22:10:26.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.516+0000 7fc01c52e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc014102220 0x7fc01419f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.516+0000 7fc01c52e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc01410c2e0 con 0x7fc014101860 2024-03-28T22:10:26.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.516+0000 7fc01c52e640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc01410c160 con 0x7fc014102220 2024-03-28T22:10:26.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.516+0000 7fc01c52e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc01410c460 con 0x7fc014100670 2024-03-28T22:10:26.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.517+0000 7fc019ac9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc014101860 0x7fc0141980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.517+0000 7fc019ac9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc014101860 0x7fc0141980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44610/0 (socket says 172.21.15.16:44610) 2024-03-28T22:10:26.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.517+0000 7fc019ac9640 1 -- 172.21.15.16:0/1025046832 learned_addr learned my addr 172.21.15.16:0/1025046832 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:26.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.517+0000 7fc01a2ca640 1 --2- 172.21.15.16:0/1025046832 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc014100670 0x7fc014197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.517+0000 7fc01aacb640 1 --2- 172.21.15.16:0/1025046832 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc014102220 0x7fc01419f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.517+0000 7fc019ac9640 1 -- 172.21.15.16:0/1025046832 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc014100670 msgr2=0x7fc014197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:26.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.517+0000 7fc019ac9640 1 --2- 172.21.15.16:0/1025046832 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc014100670 0x7fc014197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.517+0000 7fc019ac9640 1 -- 172.21.15.16:0/1025046832 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc014102220 msgr2=0x7fc01419f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:26.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.517+0000 7fc019ac9640 1 --2- 172.21.15.16:0/1025046832 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc014102220 0x7fc01419f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.517+0000 7fc019ac9640 1 -- 172.21.15.16:0/1025046832 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc014102a80 con 0x7fc014101860 2024-03-28T22:10:26.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.517+0000 7f80e2ffd640 1 --2- 172.21.15.16:0/828561439 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f80cc0a1e80 0x7f80cc0a4340 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f80d8005fd0 tx=0x7f80d800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:26.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.517+0000 7fc019ac9640 1 --2- 172.21.15.16:0/1025046832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc014101860 0x7fc0141980a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fc00400e9f0 tx=0x7fc00400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:26.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.520+0000 7fc00b7fe640 1 -- 172.21.15.16:0/1025046832 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc00400cd80 con 0x7fc014101860 2024-03-28T22:10:26.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.520+0000 7fc00b7fe640 1 -- 172.21.15.16:0/1025046832 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc004010910 con 0x7fc014101860 2024-03-28T22:10:26.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.520+0000 7fc01c52e640 1 -- 172.21.15.16:0/1025046832 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc014102d70 con 0x7fc014101860 2024-03-28T22:10:26.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.520+0000 7fc00b7fe640 1 -- 172.21.15.16:0/1025046832 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc004018990 con 0x7fc014101860 2024-03-28T22:10:26.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.520+0000 7fc01c52e640 1 -- 172.21.15.16:0/1025046832 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc014103310 con 0x7fc014101860 2024-03-28T22:10:26.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.520+0000 7f80e88c0640 1 -- 172.21.15.16:0/828561439 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f80dc053280 con 0x7f80e4103350 2024-03-28T22:10:26.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.522+0000 7fc00b7fe640 1 -- 172.21.15.16:0/1025046832 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7fc004010ac0 con 0x7fc014101860 2024-03-28T22:10:26.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.523+0000 7fc01f1ea740 1 Processor -- start 2024-03-28T22:10:26.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.523+0000 7fc01f1ea740 1 -- start start 2024-03-28T22:10:26.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.523+0000 7fc00b7fe640 1 --2- 172.21.15.16:0/1025046832 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc0000a1e90 0x7fc0000a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.523+0000 7fc01f1ea740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bc5fa0da10 0x55bc5fa0de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.523+0000 7fc01f1ea740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bc5fa0e400 0x55bc5fa12f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.523+0000 7fc01f1ea740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc5fa13570 0x55bc5fa15a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.523+0000 7fc01f1ea740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55bc5f90f0b0 con 0x55bc5fa13570 2024-03-28T22:10:26.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.523+0000 7fc01f1ea740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55bc5f971ea0 con 0x55bc5fa0da10 2024-03-28T22:10:26.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.523+0000 7fc01f1ea740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55bc5f91ac00 con 0x55bc5fa0e400 2024-03-28T22:10:26.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.524+0000 7fc01aacb640 1 --2- 172.21.15.16:0/1025046832 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc0000a1e90 0x7fc0000a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.524+0000 7fc01a2ca640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bc5fa0da10 0x55bc5fa0de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.524+0000 7fc00b7fe640 1 -- 172.21.15.16:0/1025046832 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fc0040ce210 con 0x7fc014101860 2024-03-28T22:10:26.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.524+0000 7fc01aacb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc5fa13570 0x55bc5fa15a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.524+0000 7fc01a2ca640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bc5fa0da10 0x55bc5fa0de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:33888/0 (socket says 172.21.15.16:33888) 2024-03-28T22:10:26.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.524+0000 7fc01a2ca640 1 -- 172.21.15.16:0/318818118 learned_addr learned my addr 172.21.15.16:0/318818118 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:26.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.524+0000 7fc019ac9640 1 --2- 172.21.15.16:0/318818118 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bc5fa0e400 0x55bc5fa12f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.524+0000 7fc01a2ca640 1 -- 172.21.15.16:0/318818118 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bc5fa0e400 msgr2=0x55bc5fa12f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:26.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.524+0000 7fc01a2ca640 1 --2- 172.21.15.16:0/318818118 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bc5fa0e400 0x55bc5fa12f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.524+0000 7fc01a2ca640 1 -- 172.21.15.16:0/318818118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc5fa13570 msgr2=0x55bc5fa15a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:26.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.524+0000 7fc01a2ca640 1 --2- 172.21.15.16:0/318818118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc5fa13570 0x55bc5fa15a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.524+0000 7fc01a2ca640 1 -- 172.21.15.16:0/318818118 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc5f74e7e0 con 0x55bc5fa0da10 2024-03-28T22:10:26.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.524+0000 7fc01aacb640 1 --2- 172.21.15.16:0/1025046832 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc0000a1e90 0x7fc0000a4350 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc00c005fd0 tx=0x7fc00c010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:26.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.524+0000 7fc01a2ca640 1 --2- 172.21.15.16:0/318818118 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bc5fa0da10 0x55bc5fa0de30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc0100057c0 tx=0x7fc010009490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f5cbe9986584857 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:26.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.524+0000 7fc008bfe640 1 -- 172.21.15.16:0/318818118 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc010043020 con 0x55bc5fa0da10 2024-03-28T22:10:26.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.524+0000 7fc008bfe640 1 -- 172.21.15.16:0/318818118 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc010034db0 con 0x55bc5fa0da10 2024-03-28T22:10:26.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.524+0000 7fc008bfe640 1 -- 172.21.15.16:0/318818118 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc010039040 con 0x55bc5fa0da10 2024-03-28T22:10:26.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.525+0000 7fc01f1ea740 1 -- 172.21.15.16:0/318818118 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bc5fa0da10 msgr2=0x55bc5fa0de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:26.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.525+0000 7fc01f1ea740 1 --2- 172.21.15.16:0/318818118 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bc5fa0da10 0x55bc5fa0de30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc0100057c0 tx=0x7fc010009490 comp rx=0 tx=0).stop 2024-03-28T22:10:26.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.525+0000 7fc01f1ea740 1 -- 172.21.15.16:0/318818118 shutdown_connections 2024-03-28T22:10:26.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.525+0000 7fc01f1ea740 1 --2- 172.21.15.16:0/318818118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc5fa13570 0x55bc5fa15a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.525+0000 7fc01f1ea740 1 --2- 172.21.15.16:0/318818118 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bc5fa0e400 0x55bc5fa12f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.525+0000 7fc01f1ea740 1 --2- 172.21.15.16:0/318818118 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bc5fa0da10 0x55bc5fa0de30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.525+0000 7fc01f1ea740 1 -- 172.21.15.16:0/318818118 >> 172.21.15.16:0/318818118 conn(0x55bc5fa08f80 msgr2=0x55bc5fa09380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:26.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.525+0000 7fc01f1ea740 1 -- 172.21.15.16:0/318818118 shutdown_connections 2024-03-28T22:10:26.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.525+0000 7fc01f1ea740 1 -- 172.21.15.16:0/318818118 wait complete. 2024-03-28T22:10:26.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.526+0000 7fc01f1ea740 1 -- start start 2024-03-28T22:10:26.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.526+0000 7fc01f1ea740 1 Processor -- start 2024-03-28T22:10:26.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.526+0000 7fc01f1ea740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc5fa0da10 0x55bc5fa17cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.526+0000 7fc01f1ea740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bc5fa0e400 0x55bc5fa18200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.526+0000 7fc01f1ea740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bc5fa13570 0x55bc5fa1f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.526+0000 7fc01f1ea740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55bc5f9103d0 con 0x55bc5fa0da10 2024-03-28T22:10:26.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.526+0000 7fc01f1ea740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55bc5f96df40 con 0x55bc5fa13570 2024-03-28T22:10:26.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.526+0000 7fc01f1ea740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55bc5fa164c0 con 0x55bc5fa0e400 2024-03-28T22:10:26.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.526+0000 7fc01f1ea740 10 client.0 _ll_register_callbacks cb 0x55bc5f92db70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:10:26.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.526+0000 7fc01a2ca640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc5fa0da10 0x55bc5fa17cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.526+0000 7fc01a2ca640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc5fa0da10 0x55bc5fa17cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44638/0 (socket says 172.21.15.16:44638) 2024-03-28T22:10:26.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.526+0000 7fc01a2ca640 1 -- 172.21.15.16:0/2613382169 learned_addr learned my addr 172.21.15.16:0/2613382169 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:26.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.526+0000 7fc01aacb640 1 --2- 172.21.15.16:0/2613382169 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bc5fa13570 0x55bc5fa1f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.526+0000 7fc019ac9640 1 --2- 172.21.15.16:0/2613382169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bc5fa0e400 0x55bc5fa18200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.526+0000 7fc01a2ca640 1 -- 172.21.15.16:0/2613382169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bc5fa0e400 msgr2=0x55bc5fa18200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:26.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.526+0000 7fc01a2ca640 1 --2- 172.21.15.16:0/2613382169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bc5fa0e400 0x55bc5fa18200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.526+0000 7fc01a2ca640 1 -- 172.21.15.16:0/2613382169 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bc5fa13570 msgr2=0x55bc5fa1f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:26.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.526+0000 7fc01a2ca640 1 --2- 172.21.15.16:0/2613382169 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bc5fa13570 0x55bc5fa1f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.526+0000 7fc01a2ca640 1 -- 172.21.15.16:0/2613382169 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc5f7aef70 con 0x55bc5fa0da10 2024-03-28T22:10:26.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.526+0000 7fc01aacb640 1 --2- 172.21.15.16:0/2613382169 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bc5fa13570 0x55bc5fa1f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:10:26.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.526+0000 7fc01a2ca640 1 --2- 172.21.15.16:0/2613382169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc5fa0da10 0x55bc5fa17cc0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fc01002c0b0 tx=0x7fc01002bb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f41715685ebb075 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:26.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.526+0000 7fbffdffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:10:26.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.527+0000 7fbffdffb640 1 -- 172.21.15.16:0/2613382169 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc010043050 con 0x55bc5fa0da10 2024-03-28T22:10:26.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.527+0000 7fbffdffb640 1 -- 172.21.15.16:0/2613382169 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fc010004250 con 0x55bc5fa0da10 2024-03-28T22:10:26.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.527+0000 7fc01f1ea740 1 -- 172.21.15.16:0/2613382169 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc010034db0 con 0x55bc5fa0da10 2024-03-28T22:10:26.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.527+0000 7fbffdffb640 1 -- 172.21.15.16:0/2613382169 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fc010059070 con 0x55bc5fa0da10 2024-03-28T22:10:26.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.527+0000 7fc01f1ea740 10 client.15399 fetch_fsmap learned FSMap version 21 2024-03-28T22:10:26.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.527+0000 7fc01f1ea740 1 -- 172.21.15.16:0/2613382169 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55bc5f646170 con 0x55bc5fa0da10 2024-03-28T22:10:26.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.527+0000 7fbffdffb640 1 -- 172.21.15.16:0/2613382169 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc010034db0 con 0x55bc5fa0da10 2024-03-28T22:10:26.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.527+0000 7fbffdffb640 1 -- 172.21.15.16:0/2613382169 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fc010032b20 con 0x55bc5fa0da10 2024-03-28T22:10:26.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.527+0000 7fc01f1ea740 10 client.15399 fetch_fsmap finished waiting for FSMap version 21 2024-03-28T22:10:26.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.527+0000 7fc01f1ea740 10 client.15399 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:10:26.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.527+0000 7fc01f1ea740 20 client.15399 populate_metadata read hostname 'smithi016' 2024-03-28T22:10:26.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.527+0000 7fc01f1ea740 1 --2- 172.21.15.16:0/2613382169 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55bc5fa23390 0x55bc5fa25780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.527+0000 7fc01f1ea740 4 client.15399 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:10:26.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.527+0000 7fc01f1ea740 1 -- 172.21.15.16:0/2613382169 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bc5f97f3e0 con 0x55bc5fa23390 2024-03-28T22:10:26.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.527+0000 7fc019ac9640 1 --2- 172.21.15.16:0/2613382169 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55bc5fa23390 0x55bc5fa25780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.528+0000 7fc019ac9640 1 --2- 172.21.15.16:0/2613382169 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55bc5fa23390 0x55bc5fa25780 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8578778539fdefe5 server_cookie=9c57f92b35f1b4fd in_seq=0 out_seq=0 2024-03-28T22:10:26.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.528+0000 7fbffdffb640 10 client.15399 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:10:26.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.529+0000 7fbffdffb640 1 -- 172.21.15.16:0/2613382169 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bc5f97f3e0 con 0x55bc5fa23390 2024-03-28T22:10:26.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.529+0000 7fbffdffb640 10 client.15399 handle_command_reply: tid=0 2024-03-28T22:10:26.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.529+0000 7fc01f1ea740 1 client.15399 shutdown 2024-03-28T22:10:26.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.529+0000 7fc01f1ea740 1 -- 172.21.15.16:0/2613382169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc5fa0da10 msgr2=0x55bc5fa17cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:26.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.529+0000 7fc01f1ea740 1 --2- 172.21.15.16:0/2613382169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc5fa0da10 0x55bc5fa17cc0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fc01002c0b0 tx=0x7fc01002bb70 comp rx=0 tx=0).stop 2024-03-28T22:10:26.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.529+0000 7fc01f1ea740 1 -- 172.21.15.16:0/2613382169 shutdown_connections 2024-03-28T22:10:26.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.529+0000 7fc01f1ea740 1 --2- 172.21.15.16:0/2613382169 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55bc5fa23390 0x55bc5fa25780 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.529+0000 7fc01f1ea740 1 --2- 172.21.15.16:0/2613382169 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bc5fa13570 0x55bc5fa1f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.529+0000 7fc01f1ea740 1 --2- 172.21.15.16:0/2613382169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bc5fa0e400 0x55bc5fa18200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.529+0000 7fc01f1ea740 1 --2- 172.21.15.16:0/2613382169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc5fa0da10 0x55bc5fa17cc0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.529+0000 7fbffdffb640 0 client.15399 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:10:26.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.529+0000 7fc01f1ea740 1 -- 172.21.15.16:0/2613382169 >> 172.21.15.16:0/2613382169 conn(0x55bc5fa08f80 msgr2=0x55bc5fa08180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:26.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.529+0000 7fc01f1ea740 1 -- 172.21.15.16:0/2613382169 shutdown_connections 2024-03-28T22:10:26.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.529+0000 7fc01f1ea740 1 -- 172.21.15.16:0/2613382169 wait complete. 2024-03-28T22:10:26.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.529+0000 7fc01f1ea740 20 client.15399 trim_cache size 0 max 16384 2024-03-28T22:10:26.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:26 smithi053 ceph-mon[35979]: pgmap v346: 97 pgs: 97 active+clean; 646 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-28T22:10:26.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:26 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3729785811' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:10:26.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:26 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/272091219' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:10:26.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:26 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3332510513' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:10:26.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.547+0000 7fc01f1ea740 1 Processor -- start 2024-03-28T22:10:26.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.547+0000 7fc01f1ea740 1 -- start start 2024-03-28T22:10:26.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.547+0000 7fc01f1ea740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bc5fa0da10 0x55bc5fa08000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.547+0000 7fc01f1ea740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc5fa0e400 0x55bc5fa08540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.547+0000 7fc01f1ea740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bc5fa13570 0x55bc5fa1f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.547+0000 7fc01f1ea740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc0141ad630 con 0x55bc5fa0e400 2024-03-28T22:10:26.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.547+0000 7fc01f1ea740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc0141aaf00 con 0x55bc5fa13570 2024-03-28T22:10:26.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.547+0000 7fc01f1ea740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc0141a8d50 con 0x55bc5fa0da10 2024-03-28T22:10:26.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.547+0000 7fc019ac9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc5fa0e400 0x55bc5fa08540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.547+0000 7fc019ac9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc5fa0e400 0x55bc5fa08540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44642/0 (socket says 172.21.15.16:44642) 2024-03-28T22:10:26.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.547+0000 7fc019ac9640 1 -- 172.21.15.16:0/3478736966 learned_addr learned my addr 172.21.15.16:0/3478736966 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:26.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.547+0000 7fc01aacb640 1 --2- 172.21.15.16:0/3478736966 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bc5fa13570 0x55bc5fa1f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.547+0000 7fc01a2ca640 1 --2- 172.21.15.16:0/3478736966 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bc5fa0da10 0x55bc5fa08000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.547+0000 7fc019ac9640 1 -- 172.21.15.16:0/3478736966 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bc5fa0da10 msgr2=0x55bc5fa08000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:26.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.547+0000 7fc019ac9640 1 --2- 172.21.15.16:0/3478736966 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bc5fa0da10 0x55bc5fa08000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.547+0000 7fc019ac9640 1 -- 172.21.15.16:0/3478736966 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bc5fa13570 msgr2=0x55bc5fa1f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:26.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.547+0000 7fc019ac9640 1 --2- 172.21.15.16:0/3478736966 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bc5fa13570 0x55bc5fa1f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.547+0000 7fc019ac9640 1 -- 172.21.15.16:0/3478736966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc5fa25d10 con 0x55bc5fa0e400 2024-03-28T22:10:26.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.547+0000 7fc01aacb640 1 --2- 172.21.15.16:0/3478736966 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bc5fa13570 0x55bc5fa1f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:10:26.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.548+0000 7fc019ac9640 1 --2- 172.21.15.16:0/3478736966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc5fa0e400 0x55bc5fa08540 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fc00400e9c0 tx=0x7fc004000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a975578d07bb30c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:26.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.548+0000 7fbffffff640 1 -- 172.21.15.16:0/3478736966 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0040a3550 con 0x55bc5fa0e400 2024-03-28T22:10:26.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.548+0000 7fbffffff640 1 -- 172.21.15.16:0/3478736966 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc004099160 con 0x55bc5fa0e400 2024-03-28T22:10:26.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.548+0000 7fbffffff640 1 -- 172.21.15.16:0/3478736966 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0040b67b0 con 0x55bc5fa0e400 2024-03-28T22:10:26.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.548+0000 7fc01f1ea740 1 -- 172.21.15.16:0/3478736966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc5fa0e400 msgr2=0x55bc5fa08540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:26.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.548+0000 7fc01f1ea740 1 --2- 172.21.15.16:0/3478736966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc5fa0e400 0x55bc5fa08540 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fc00400e9c0 tx=0x7fc004000bc0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.548+0000 7fc01f1ea740 1 -- 172.21.15.16:0/3478736966 shutdown_connections 2024-03-28T22:10:26.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.548+0000 7fc01f1ea740 1 --2- 172.21.15.16:0/3478736966 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bc5fa13570 0x55bc5fa1f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.548+0000 7fc01f1ea740 1 --2- 172.21.15.16:0/3478736966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc5fa0e400 0x55bc5fa08540 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.548+0000 7fc01f1ea740 1 --2- 172.21.15.16:0/3478736966 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bc5fa0da10 0x55bc5fa08000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.548+0000 7fc01f1ea740 1 -- 172.21.15.16:0/3478736966 >> 172.21.15.16:0/3478736966 conn(0x55bc5fa08f80 msgr2=0x55bc5fa078f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:26.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.548+0000 7fc01f1ea740 1 -- 172.21.15.16:0/3478736966 shutdown_connections 2024-03-28T22:10:26.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.548+0000 7fc01f1ea740 1 -- 172.21.15.16:0/3478736966 wait complete. 2024-03-28T22:10:26.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.549+0000 7fc01f1ea740 1 -- start start 2024-03-28T22:10:26.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.549+0000 7fc01f1ea740 1 Processor -- start 2024-03-28T22:10:26.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.549+0000 7fc01f1ea740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bc5fa0da10 0x55bc5fa1eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.549+0000 7fc01f1ea740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc5fa0e400 0x55bc5fa1d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.549+0000 7fc01f1ea740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bc5fa13570 0x55bc5fa1d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.549+0000 7fc01f1ea740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc0141a93c0 con 0x55bc5fa0e400 2024-03-28T22:10:26.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.549+0000 7fc01f1ea740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc0141aa830 con 0x55bc5fa13570 2024-03-28T22:10:26.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.549+0000 7fc01f1ea740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc0141a9a90 con 0x55bc5fa0da10 2024-03-28T22:10:26.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.549+0000 7fc01f1ea740 10 client.0 _ll_register_callbacks cb 0x7fc004005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:10:26.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.549+0000 7fc019ac9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc5fa0e400 0x55bc5fa1d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.549+0000 7fc019ac9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc5fa0e400 0x55bc5fa1d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44658/0 (socket says 172.21.15.16:44658) 2024-03-28T22:10:26.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.549+0000 7fc019ac9640 1 -- 172.21.15.16:0/3778253496 learned_addr learned my addr 172.21.15.16:0/3778253496 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:26.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.549+0000 7fc01aacb640 1 --2- 172.21.15.16:0/3778253496 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bc5fa13570 0x55bc5fa1d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.549+0000 7fc01a2ca640 1 --2- 172.21.15.16:0/3778253496 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bc5fa0da10 0x55bc5fa1eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.549+0000 7fc019ac9640 1 -- 172.21.15.16:0/3778253496 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bc5fa0da10 msgr2=0x55bc5fa1eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:26.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.549+0000 7fc019ac9640 1 --2- 172.21.15.16:0/3778253496 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bc5fa0da10 0x55bc5fa1eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.549+0000 7fc019ac9640 1 -- 172.21.15.16:0/3778253496 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bc5fa13570 msgr2=0x55bc5fa1d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:26.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.549+0000 7fc019ac9640 1 --2- 172.21.15.16:0/3778253496 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bc5fa13570 0x55bc5fa1d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.549+0000 7fc019ac9640 1 -- 172.21.15.16:0/3778253496 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc5fa1dda0 con 0x55bc5fa0e400 2024-03-28T22:10:26.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.550+0000 7fc019ac9640 1 --2- 172.21.15.16:0/3778253496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc5fa0e400 0x55bc5fa1d320 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fc004097830 tx=0x7fc004018420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebf1d60974727f67 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:26.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.550+0000 7fbffdffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:10:26.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.550+0000 7fbffdffb640 1 -- 172.21.15.16:0/3778253496 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0040a3550 con 0x55bc5fa0e400 2024-03-28T22:10:26.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.550+0000 7fbffdffb640 1 -- 172.21.15.16:0/3778253496 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fc0040b6300 con 0x55bc5fa0e400 2024-03-28T22:10:26.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.550+0000 7fc01f1ea740 1 -- 172.21.15.16:0/3778253496 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc004099160 con 0x55bc5fa0e400 2024-03-28T22:10:26.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.550+0000 7fbffdffb640 1 -- 172.21.15.16:0/3778253496 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fc0040cbde0 con 0x55bc5fa0e400 2024-03-28T22:10:26.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.550+0000 7fc01f1ea740 10 client.15405 fetch_fsmap learned FSMap version 21 2024-03-28T22:10:26.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.550+0000 7fc01f1ea740 1 -- 172.21.15.16:0/3778253496 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55bc5fa8a2a0 con 0x55bc5fa0e400 2024-03-28T22:10:26.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.550+0000 7fbffdffb640 1 -- 172.21.15.16:0/3778253496 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc004099160 con 0x55bc5fa0e400 2024-03-28T22:10:26.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.550+0000 7fbffdffb640 1 -- 172.21.15.16:0/3778253496 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fc004096bb0 con 0x55bc5fa0e400 2024-03-28T22:10:26.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.550+0000 7fc01f1ea740 10 client.15405 fetch_fsmap finished waiting for FSMap version 21 2024-03-28T22:10:26.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.550+0000 7fc01f1ea740 10 client.15405 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:10:26.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.550+0000 7fc01f1ea740 20 client.15405 populate_metadata read hostname 'smithi016' 2024-03-28T22:10:26.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.550+0000 7fc01f1ea740 1 --2- 172.21.15.16:0/3778253496 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55bc5fa8ac30 0x55bc5fa8d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.550+0000 7fc01f1ea740 4 client.15405 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:10:26.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.550+0000 7fc01f1ea740 1 -- 172.21.15.16:0/3778253496 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55bc5f979730 con 0x55bc5fa8ac30 2024-03-28T22:10:26.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.550+0000 7fc01a2ca640 1 --2- 172.21.15.16:0/3778253496 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55bc5fa8ac30 0x55bc5fa8d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.551+0000 7fc01a2ca640 1 --2- 172.21.15.16:0/3778253496 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55bc5fa8ac30 0x55bc5fa8d050 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de97c22e803f61c1 server_cookie=d8b92099e135a94f in_seq=0 out_seq=0 2024-03-28T22:10:26.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.551+0000 7fbffdffb640 10 client.15405 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:10:26.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.552+0000 7fbffdffb640 1 -- 172.21.15.16:0/3778253496 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55bc5f979730 con 0x55bc5fa8ac30 2024-03-28T22:10:26.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.552+0000 7fbffdffb640 10 client.15405 handle_command_reply: tid=0 2024-03-28T22:10:26.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.552+0000 7fc01f1ea740 1 client.15405 shutdown 2024-03-28T22:10:26.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.552+0000 7fc01f1ea740 1 -- 172.21.15.16:0/3778253496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc5fa0e400 msgr2=0x55bc5fa1d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:26.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.552+0000 7fc01f1ea740 1 --2- 172.21.15.16:0/3778253496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc5fa0e400 0x55bc5fa1d320 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fc004097830 tx=0x7fc004018420 comp rx=0 tx=0).stop 2024-03-28T22:10:26.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.552+0000 7fc01f1ea740 1 -- 172.21.15.16:0/3778253496 shutdown_connections 2024-03-28T22:10:26.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.552+0000 7fc01f1ea740 1 --2- 172.21.15.16:0/3778253496 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55bc5fa8ac30 0x55bc5fa8d050 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.552+0000 7fc01f1ea740 1 --2- 172.21.15.16:0/3778253496 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bc5fa13570 0x55bc5fa1d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.552+0000 7fc01f1ea740 1 --2- 172.21.15.16:0/3778253496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc5fa0e400 0x55bc5fa1d320 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.552+0000 7fbffdffb640 0 client.15405 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:10:26.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.552+0000 7fc01f1ea740 1 --2- 172.21.15.16:0/3778253496 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bc5fa0da10 0x55bc5fa1eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.552+0000 7fc01f1ea740 1 -- 172.21.15.16:0/3778253496 >> 172.21.15.16:0/3778253496 conn(0x55bc5fa08f80 msgr2=0x55bc5fa08180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:26.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.553+0000 7fc01f1ea740 1 -- 172.21.15.16:0/3778253496 shutdown_connections 2024-03-28T22:10:26.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.553+0000 7fc01f1ea740 1 -- 172.21.15.16:0/3778253496 wait complete. 2024-03-28T22:10:26.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.553+0000 7fc01f1ea740 20 client.15405 trim_cache size 0 max 16384 2024-03-28T22:10:26.563 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:10:26.563 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-03-28T22:10:26.563 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "ec95a5dc-1296-4853-8f53-230f7de5bd2b", 2024-03-28T22:10:26.563 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-03-28T22:10:26.563 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:10:26.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.553+0000 7fc01c52e640 1 -- 172.21.15.16:0/1025046832 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc0000a1e90 msgr2=0x7fc0000a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:26.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.553+0000 7fc01c52e640 1 --2- 172.21.15.16:0/1025046832 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc0000a1e90 0x7fc0000a4350 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc00c005fd0 tx=0x7fc00c010040 comp rx=0 tx=0).stop 2024-03-28T22:10:26.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.553+0000 7fc01c52e640 1 -- 172.21.15.16:0/1025046832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc014101860 msgr2=0x7fc0141980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:26.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.553+0000 7fc01c52e640 1 --2- 172.21.15.16:0/1025046832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc014101860 0x7fc0141980a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fc00400e9f0 tx=0x7fc00400eeb0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.553+0000 7fc01c52e640 1 -- 172.21.15.16:0/1025046832 shutdown_connections 2024-03-28T22:10:26.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.553+0000 7fc01c52e640 1 --2- 172.21.15.16:0/1025046832 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fc0000a1e90 0x7fc0000a4350 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.553+0000 7fc01c52e640 1 --2- 172.21.15.16:0/1025046832 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc014102220 0x7fc01419f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.553+0000 7fc01c52e640 1 --2- 172.21.15.16:0/1025046832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc014101860 0x7fc0141980a0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.553+0000 7fc01c52e640 1 --2- 172.21.15.16:0/1025046832 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc014100670 0x7fc014197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.553+0000 7fc01c52e640 1 -- 172.21.15.16:0/1025046832 >> 172.21.15.16:0/1025046832 conn(0x7fc0140fbbe0 msgr2=0x7fc0140fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:26.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.553+0000 7fc01c52e640 1 -- 172.21.15.16:0/1025046832 shutdown_connections 2024-03-28T22:10:26.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.553+0000 7fc01c52e640 1 -- 172.21.15.16:0/1025046832 wait complete. 2024-03-28T22:10:26.567 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:10:26.567 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:10:26.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.670+0000 7efd0b6f4640 1 -- 172.21.15.16:0/2860118937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd04102220 msgr2=0x7efd04108ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:26.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.670+0000 7efd0b6f4640 1 --2- 172.21.15.16:0/2860118937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd04102220 0x7efd04108ab0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7efcfc009ca0 tx=0x7efcfc02c0f0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.670+0000 7efd0b6f4640 1 -- 172.21.15.16:0/2860118937 shutdown_connections 2024-03-28T22:10:26.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.670+0000 7efd0b6f4640 1 --2- 172.21.15.16:0/2860118937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd04102220 0x7efd04108ab0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.670+0000 7efd0b6f4640 1 --2- 172.21.15.16:0/2860118937 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efd04101860 0x7efd04101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.670+0000 7efd0b6f4640 1 --2- 172.21.15.16:0/2860118937 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd04100670 0x7efd04100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.670+0000 7efd0b6f4640 1 -- 172.21.15.16:0/2860118937 >> 172.21.15.16:0/2860118937 conn(0x7efd040fbbe0 msgr2=0x7efd040fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:26.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.670+0000 7efd0b6f4640 1 -- 172.21.15.16:0/2860118937 shutdown_connections 2024-03-28T22:10:26.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.670+0000 7efd0b6f4640 1 -- 172.21.15.16:0/2860118937 wait complete. 2024-03-28T22:10:26.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.671+0000 7efd0b6f4640 1 Processor -- start 2024-03-28T22:10:26.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.671+0000 7efd0b6f4640 1 -- start start 2024-03-28T22:10:26.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.671+0000 7efd0b6f4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efd04100670 0x7efd04197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.671+0000 7efd0b6f4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd04101860 0x7efd04197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.671+0000 7efd0b6f4640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd04102220 0x7efd0419f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.671+0000 7efd0b6f4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efd0410c300 con 0x7efd04101860 2024-03-28T22:10:26.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.671+0000 7efd0b6f4640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7efd0410c180 con 0x7efd04102220 2024-03-28T22:10:26.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.671+0000 7efd0b6f4640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7efd0410c480 con 0x7efd04100670 2024-03-28T22:10:26.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.671+0000 7efd08c8f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd04101860 0x7efd04197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.671+0000 7efd09c91640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd04102220 0x7efd0419f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.671+0000 7efd08c8f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd04101860 0x7efd04197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44682/0 (socket says 172.21.15.16:44682) 2024-03-28T22:10:26.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.671+0000 7efd08c8f640 1 -- 172.21.15.16:0/1280635551 learned_addr learned my addr 172.21.15.16:0/1280635551 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:26.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.671+0000 7efd09490640 1 --2- 172.21.15.16:0/1280635551 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efd04100670 0x7efd04197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.671+0000 7efd08c8f640 1 -- 172.21.15.16:0/1280635551 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efd04100670 msgr2=0x7efd04197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:26.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.671+0000 7efd08c8f640 1 --2- 172.21.15.16:0/1280635551 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efd04100670 0x7efd04197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.671+0000 7efd08c8f640 1 -- 172.21.15.16:0/1280635551 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd04102220 msgr2=0x7efd0419f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:26.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.671+0000 7efd08c8f640 1 --2- 172.21.15.16:0/1280635551 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd04102220 0x7efd0419f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.671+0000 7efd08c8f640 1 -- 172.21.15.16:0/1280635551 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd0419f6b0 con 0x7efd04101860 2024-03-28T22:10:26.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.671+0000 7efd09c91640 1 --2- 172.21.15.16:0/1280635551 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd04102220 0x7efd0419f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:10:26.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.672+0000 7efd08c8f640 1 --2- 172.21.15.16:0/1280635551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd04101860 0x7efd04197fd0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7efcf400c970 tx=0x7efcf400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:26.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.672+0000 7efcfa7fc640 1 -- 172.21.15.16:0/1280635551 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf4007bb0 con 0x7efd04101860 2024-03-28T22:10:26.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.672+0000 7efcfa7fc640 1 -- 172.21.15.16:0/1280635551 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efcf4007d50 con 0x7efd04101860 2024-03-28T22:10:26.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.672+0000 7efcfa7fc640 1 -- 172.21.15.16:0/1280635551 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf400f450 con 0x7efd04101860 2024-03-28T22:10:26.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.672+0000 7efd0b6f4640 1 -- 172.21.15.16:0/1280635551 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd0419f9a0 con 0x7efd04101860 2024-03-28T22:10:26.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.672+0000 7efd0b6f4640 1 -- 172.21.15.16:0/1280635551 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd0419fef0 con 0x7efd04101860 2024-03-28T22:10:26.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:26 smithi063 ceph-mon[36249]: pgmap v346: 97 pgs: 97 active+clean; 646 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-28T22:10:26.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:26 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/3729785811' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:10:26.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:26 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/272091219' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:10:26.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:26 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/3332510513' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:10:26.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.674+0000 7efcfa7fc640 1 -- 172.21.15.16:0/1280635551 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7efcf4005020 con 0x7efd04101860 2024-03-28T22:10:26.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.674+0000 7efd0e3b2740 1 Processor -- start 2024-03-28T22:10:26.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.674+0000 7efd0e3b2740 1 -- start start 2024-03-28T22:10:26.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.675+0000 7efd0e3b2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605e0ec72b0 0x5605e0ec76d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.675+0000 7efd0e3b2740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5605e0ec7ca0 0x5605e0ecc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.675+0000 7efd0e3b2740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605e0ecce10 0x5605e0ecf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.675+0000 7efd0e3b2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5605e0e2c180 con 0x5605e0ec72b0 2024-03-28T22:10:26.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.675+0000 7efd0e3b2740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5605e0dd4b70 con 0x5605e0ec7ca0 2024-03-28T22:10:26.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.675+0000 7efd0e3b2740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5605e0e28220 con 0x5605e0ecce10 2024-03-28T22:10:26.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.675+0000 7efcfa7fc640 1 --2- 172.21.15.16:0/1280635551 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7efcf00a1e90 0x7efcf00a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.675+0000 7efd09c91640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605e0ec72b0 0x5605e0ec76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.675+0000 7efd09c91640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605e0ec72b0 0x5605e0ec76d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44688/0 (socket says 172.21.15.16:44688) 2024-03-28T22:10:26.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.675+0000 7efd09c91640 1 -- 172.21.15.16:0/3767993140 learned_addr learned my addr 172.21.15.16:0/3767993140 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:26.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.675+0000 7efd09490640 1 --2- 172.21.15.16:0/3767993140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5605e0ec7ca0 0x5605e0ecc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.675+0000 7efcfa7fc640 1 -- 172.21.15.16:0/1280635551 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7efcf40cdc40 con 0x7efd04101860 2024-03-28T22:10:26.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.675+0000 7efd09c91640 1 --2- 172.21.15.16:0/1280635551 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7efcf00a1e90 0x7efcf00a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.675+0000 7efd08c8f640 1 --2- 172.21.15.16:0/3767993140 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605e0ecce10 0x5605e0ecf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.675+0000 7efd09490640 1 -- 172.21.15.16:0/3767993140 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605e0ecce10 msgr2=0x5605e0ecf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:26.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.675+0000 7efd09490640 1 --2- 172.21.15.16:0/3767993140 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605e0ecce10 0x5605e0ecf2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.675+0000 7efd09490640 1 -- 172.21.15.16:0/3767993140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605e0ec72b0 msgr2=0x5605e0ec76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:26.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.675+0000 7efd09490640 1 --2- 172.21.15.16:0/3767993140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605e0ec72b0 0x5605e0ec76d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.675+0000 7efd09490640 1 -- 172.21.15.16:0/3767993140 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605e0de87c0 con 0x5605e0ec7ca0 2024-03-28T22:10:26.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.675+0000 7efd09c91640 1 --2- 172.21.15.16:0/3767993140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605e0ec72b0 0x5605e0ec76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:10:26.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.675+0000 7efd09c91640 1 --2- 172.21.15.16:0/1280635551 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7efcf00a1e90 0x7efcf00a4350 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x5605e0ec8500 tx=0x7efcfc032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:26.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.675+0000 7efd09490640 1 --2- 172.21.15.16:0/3767993140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5605e0ec7ca0 0x5605e0ecc7a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7efd00009b40 tx=0x7efd000113e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=710bc5c4b39c7480 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:26.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.676+0000 7efcef7fe640 1 -- 172.21.15.16:0/3767993140 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd00011e10 con 0x5605e0ec7ca0 2024-03-28T22:10:26.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.676+0000 7efcef7fe640 1 -- 172.21.15.16:0/3767993140 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd00012930 con 0x5605e0ec7ca0 2024-03-28T22:10:26.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.676+0000 7efcef7fe640 1 -- 172.21.15.16:0/3767993140 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd0000fa50 con 0x5605e0ec7ca0 2024-03-28T22:10:26.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.676+0000 7efd0e3b2740 1 -- 172.21.15.16:0/3767993140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5605e0ec7ca0 msgr2=0x5605e0ecc7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:26.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.676+0000 7efd0e3b2740 1 --2- 172.21.15.16:0/3767993140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5605e0ec7ca0 0x5605e0ecc7a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7efd00009b40 tx=0x7efd000113e0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.676+0000 7efd0e3b2740 1 -- 172.21.15.16:0/3767993140 shutdown_connections 2024-03-28T22:10:26.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.676+0000 7efd0e3b2740 1 --2- 172.21.15.16:0/3767993140 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605e0ecce10 0x5605e0ecf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.676+0000 7efd0e3b2740 1 --2- 172.21.15.16:0/3767993140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5605e0ec7ca0 0x5605e0ecc7a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.676+0000 7efd0e3b2740 1 --2- 172.21.15.16:0/3767993140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605e0ec72b0 0x5605e0ec76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.676+0000 7efd0e3b2740 1 -- 172.21.15.16:0/3767993140 >> 172.21.15.16:0/3767993140 conn(0x5605e0ec2820 msgr2=0x5605e0ec2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:26.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.676+0000 7efd0e3b2740 1 -- 172.21.15.16:0/3767993140 shutdown_connections 2024-03-28T22:10:26.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.676+0000 7efd0e3b2740 1 -- 172.21.15.16:0/3767993140 wait complete. 2024-03-28T22:10:26.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.677+0000 7efd0e3b2740 1 -- start start 2024-03-28T22:10:26.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.677+0000 7efd0e3b2740 1 Processor -- start 2024-03-28T22:10:26.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.677+0000 7efd0e3b2740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5605e0ec72b0 0x5605e0ed17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.677+0000 7efd0e3b2740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605e0ec7ca0 0x5605e0ed1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.677+0000 7efd0e3b2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605e0ecce10 0x5605e0ed8d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.677+0000 7efd0e3b2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5605e0ecfdc0 con 0x5605e0ecce10 2024-03-28T22:10:26.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.677+0000 7efd0e3b2740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5605e0dca4a0 con 0x5605e0ec72b0 2024-03-28T22:10:26.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.677+0000 7efd0e3b2740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5605e0ecff40 con 0x5605e0ec7ca0 2024-03-28T22:10:26.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.677+0000 7efd0e3b2740 10 client.0 _ll_register_callbacks cb 0x5605e0dcdd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:10:26.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.677+0000 7efd09c91640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605e0ecce10 0x5605e0ed8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.677+0000 7efd09490640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5605e0ec72b0 0x5605e0ed17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.677+0000 7efd09490640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5605e0ec72b0 0x5605e0ed17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:33936/0 (socket says 172.21.15.16:33936) 2024-03-28T22:10:26.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.677+0000 7efd09490640 1 -- 172.21.15.16:0/572769117 learned_addr learned my addr 172.21.15.16:0/572769117 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:26.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.677+0000 7efd09c91640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605e0ecce10 0x5605e0ed8d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44696/0 (socket says 172.21.15.16:44696) 2024-03-28T22:10:26.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.677+0000 7efd08c8f640 1 --2- 172.21.15.16:0/572769117 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605e0ec7ca0 0x5605e0ed1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.678+0000 7efd09c91640 1 -- 172.21.15.16:0/572769117 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605e0ec7ca0 msgr2=0x5605e0ed1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:26.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.678+0000 7efd09c91640 1 --2- 172.21.15.16:0/572769117 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605e0ec7ca0 0x5605e0ed1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.678+0000 7efd09c91640 1 -- 172.21.15.16:0/572769117 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5605e0ec72b0 msgr2=0x5605e0ed17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:26.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.678+0000 7efd09c91640 1 --2- 172.21.15.16:0/572769117 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5605e0ec72b0 0x5605e0ed17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.678+0000 7efd09c91640 1 -- 172.21.15.16:0/572769117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605e0c0a7d0 con 0x5605e0ecce10 2024-03-28T22:10:26.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.678+0000 7efd09490640 1 --2- 172.21.15.16:0/572769117 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5605e0ec72b0 0x5605e0ed17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:10:26.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.678+0000 7efd09c91640 1 --2- 172.21.15.16:0/572769117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605e0ecce10 0x5605e0ed8d40 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7efcfc02cfd0 tx=0x7efcfc0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5c179c7cb41f790 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:26.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.678+0000 7efcecff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:10:26.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.678+0000 7efcecff9640 1 -- 172.21.15.16:0/572769117 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc053060 con 0x5605e0ecce10 2024-03-28T22:10:26.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.678+0000 7efcecff9640 1 -- 172.21.15.16:0/572769117 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7efcfc034a00 con 0x5605e0ecce10 2024-03-28T22:10:26.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.678+0000 7efd0e3b2740 1 -- 172.21.15.16:0/572769117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd00012930 con 0x5605e0ecce10 2024-03-28T22:10:26.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.678+0000 7efcecff9640 1 -- 172.21.15.16:0/572769117 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7efcfc039040 con 0x5605e0ecce10 2024-03-28T22:10:26.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.678+0000 7efd0e3b2740 10 client.15417 fetch_fsmap learned FSMap version 21 2024-03-28T22:10:26.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.678+0000 7efd0e3b2740 1 -- 172.21.15.16:0/572769117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5605e0b403f0 con 0x5605e0ecce10 2024-03-28T22:10:26.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.678+0000 7efcecff9640 1 -- 172.21.15.16:0/572769117 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd00012930 con 0x5605e0ecce10 2024-03-28T22:10:26.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.679+0000 7efcecff9640 1 -- 172.21.15.16:0/572769117 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7efcfc03f070 con 0x5605e0ecce10 2024-03-28T22:10:26.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.679+0000 7efd0e3b2740 10 client.15417 fetch_fsmap finished waiting for FSMap version 21 2024-03-28T22:10:26.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.679+0000 7efd0e3b2740 10 client.15417 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:10:26.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.679+0000 7efd0e3b2740 20 client.15417 populate_metadata read hostname 'smithi016' 2024-03-28T22:10:26.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.679+0000 7efd0e3b2740 1 --2- 172.21.15.16:0/572769117 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5605e0edd5c0 0x5605e0edfa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.679+0000 7efd0e3b2740 4 client.15417 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:10:26.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.679+0000 7efd0e3b2740 1 -- 172.21.15.16:0/572769117 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5605e0e396c0 con 0x5605e0edd5c0 2024-03-28T22:10:26.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.679+0000 7efd09490640 1 --2- 172.21.15.16:0/572769117 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5605e0edd5c0 0x5605e0edfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.679+0000 7efcecff9640 10 client.15417 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:10:26.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.679+0000 7efd09490640 1 --2- 172.21.15.16:0/572769117 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5605e0edd5c0 0x5605e0edfa80 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17089a810f8d0591 server_cookie=2c4250933a673430 in_seq=0 out_seq=0 2024-03-28T22:10:26.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.680+0000 7efcecff9640 1 -- 172.21.15.16:0/572769117 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5605e0e396c0 con 0x5605e0edd5c0 2024-03-28T22:10:26.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.680+0000 7efcecff9640 10 client.15417 handle_command_reply: tid=0 2024-03-28T22:10:26.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.680+0000 7efd0e3b2740 1 client.15417 shutdown 2024-03-28T22:10:26.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.680+0000 7efd0e3b2740 1 -- 172.21.15.16:0/572769117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605e0ecce10 msgr2=0x5605e0ed8d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:26.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.680+0000 7efd0e3b2740 1 --2- 172.21.15.16:0/572769117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605e0ecce10 0x5605e0ed8d40 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7efcfc02cfd0 tx=0x7efcfc0095c0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.680+0000 7efd0e3b2740 1 -- 172.21.15.16:0/572769117 shutdown_connections 2024-03-28T22:10:26.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.680+0000 7efd0e3b2740 1 --2- 172.21.15.16:0/572769117 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5605e0edd5c0 0x5605e0edfa80 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.680+0000 7efd0e3b2740 1 --2- 172.21.15.16:0/572769117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605e0ecce10 0x5605e0ed8d40 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.680+0000 7efd0e3b2740 1 --2- 172.21.15.16:0/572769117 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605e0ec7ca0 0x5605e0ed1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.680+0000 7efcecff9640 0 client.15417 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:10:26.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.680+0000 7efd0e3b2740 1 --2- 172.21.15.16:0/572769117 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5605e0ec72b0 0x5605e0ed17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.680+0000 7efd0e3b2740 1 -- 172.21.15.16:0/572769117 >> 172.21.15.16:0/572769117 conn(0x5605e0ec2820 msgr2=0x5605e0ec1c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:26.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.680+0000 7efd0e3b2740 1 -- 172.21.15.16:0/572769117 shutdown_connections 2024-03-28T22:10:26.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.680+0000 7efd0e3b2740 1 -- 172.21.15.16:0/572769117 wait complete. 2024-03-28T22:10:26.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.680+0000 7efd0e3b2740 20 client.15417 trim_cache size 0 max 16384 2024-03-28T22:10:26.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.697+0000 7efd0e3b2740 1 Processor -- start 2024-03-28T22:10:26.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.698+0000 7efd0e3b2740 1 -- start start 2024-03-28T22:10:26.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.698+0000 7efd0e3b2740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605e0ec72b0 0x5605e0ec1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.698+0000 7efd0e3b2740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5605e0ec7ca0 0x5605e0ed8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.698+0000 7efd0e3b2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605e0ecce10 0x5605e0ec18d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.698+0000 7efd0e3b2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efd041ab470 con 0x5605e0ecce10 2024-03-28T22:10:26.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.698+0000 7efd0e3b2740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7efd041a8d40 con 0x5605e0ec7ca0 2024-03-28T22:10:26.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.698+0000 7efd0e3b2740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7efd041a6b90 con 0x5605e0ec72b0 2024-03-28T22:10:26.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.698+0000 7efd09c91640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605e0ecce10 0x5605e0ec18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.698+0000 7efd08c8f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5605e0ec7ca0 0x5605e0ed8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.698+0000 7efd09c91640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605e0ecce10 0x5605e0ec18d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44712/0 (socket says 172.21.15.16:44712) 2024-03-28T22:10:26.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.698+0000 7efd08c8f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5605e0ec7ca0 0x5605e0ed8d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:33942/0 (socket says 172.21.15.16:33942) 2024-03-28T22:10:26.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.698+0000 7efd09c91640 1 -- 172.21.15.16:0/3608537477 learned_addr learned my addr 172.21.15.16:0/3608537477 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:26.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.698+0000 7efd09490640 1 --2- 172.21.15.16:0/3608537477 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605e0ec72b0 0x5605e0ec1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.698+0000 7efd09c91640 1 -- 172.21.15.16:0/3608537477 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605e0ec72b0 msgr2=0x5605e0ec1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:26.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.698+0000 7efd09c91640 1 --2- 172.21.15.16:0/3608537477 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605e0ec72b0 0x5605e0ec1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.698+0000 7efd09c91640 1 -- 172.21.15.16:0/3608537477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5605e0ec7ca0 msgr2=0x5605e0ed8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:26.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.698+0000 7efd09c91640 1 --2- 172.21.15.16:0/3608537477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5605e0ec7ca0 0x5605e0ed8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.698+0000 7efd09c91640 1 -- 172.21.15.16:0/3608537477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605e0b000f0 con 0x5605e0ecce10 2024-03-28T22:10:26.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.698+0000 7efd08c8f640 1 --2- 172.21.15.16:0/3608537477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5605e0ec7ca0 0x5605e0ed8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:10:26.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.698+0000 7efd09c91640 1 --2- 172.21.15.16:0/3608537477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605e0ecce10 0x5605e0ec18d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7efcfc0324c0 tx=0x7efcfc0099f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=767b7f36f1cdacad server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:26.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.699+0000 7efceeffd640 1 -- 172.21.15.16:0/3608537477 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc053040 con 0x5605e0ecce10 2024-03-28T22:10:26.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.699+0000 7efceeffd640 1 -- 172.21.15.16:0/3608537477 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efcfc05f020 con 0x5605e0ecce10 2024-03-28T22:10:26.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.699+0000 7efceeffd640 1 -- 172.21.15.16:0/3608537477 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc0043a0 con 0x5605e0ecce10 2024-03-28T22:10:26.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.699+0000 7efd0e3b2740 1 -- 172.21.15.16:0/3608537477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605e0ecce10 msgr2=0x5605e0ec18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:26.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.699+0000 7efd0e3b2740 1 --2- 172.21.15.16:0/3608537477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605e0ecce10 0x5605e0ec18d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7efcfc0324c0 tx=0x7efcfc0099f0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.699+0000 7efd0e3b2740 1 -- 172.21.15.16:0/3608537477 shutdown_connections 2024-03-28T22:10:26.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.699+0000 7efd0e3b2740 1 --2- 172.21.15.16:0/3608537477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605e0ecce10 0x5605e0ec18d0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.699+0000 7efd0e3b2740 1 --2- 172.21.15.16:0/3608537477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5605e0ec7ca0 0x5605e0ed8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.699+0000 7efd0e3b2740 1 --2- 172.21.15.16:0/3608537477 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605e0ec72b0 0x5605e0ec1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.699+0000 7efd0e3b2740 1 -- 172.21.15.16:0/3608537477 >> 172.21.15.16:0/3608537477 conn(0x5605e0ec2820 msgr2=0x5605e0ec2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:26.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.699+0000 7efd0e3b2740 1 -- 172.21.15.16:0/3608537477 shutdown_connections 2024-03-28T22:10:26.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.699+0000 7efd0e3b2740 1 -- 172.21.15.16:0/3608537477 wait complete. 2024-03-28T22:10:26.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.700+0000 7efd0e3b2740 1 -- start start 2024-03-28T22:10:26.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.700+0000 7efd0e3b2740 1 Processor -- start 2024-03-28T22:10:26.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.700+0000 7efd0e3b2740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605e0ec72b0 0x5605e0ed6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.700+0000 7efd0e3b2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605e0ec7ca0 0x5605e0ed6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.700+0000 7efd0e3b2740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5605e0ecce10 0x5605e0ed7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.700+0000 7efd0e3b2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efd041a7200 con 0x5605e0ec7ca0 2024-03-28T22:10:26.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.700+0000 7efd0e3b2740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7efd041a8670 con 0x5605e0ecce10 2024-03-28T22:10:26.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.700+0000 7efd0e3b2740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7efd041a78d0 con 0x5605e0ec72b0 2024-03-28T22:10:26.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.700+0000 7efd0e3b2740 10 client.0 _ll_register_callbacks cb 0x7efcfc009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:10:26.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.700+0000 7efd08c8f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605e0ec7ca0 0x5605e0ed6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.700+0000 7efd08c8f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605e0ec7ca0 0x5605e0ed6ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44728/0 (socket says 172.21.15.16:44728) 2024-03-28T22:10:26.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.700+0000 7efd08c8f640 1 -- 172.21.15.16:0/3248877749 learned_addr learned my addr 172.21.15.16:0/3248877749 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:26.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.700+0000 7efd09490640 1 --2- 172.21.15.16:0/3248877749 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605e0ec72b0 0x5605e0ed6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.700+0000 7efd09c91640 1 --2- 172.21.15.16:0/3248877749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5605e0ecce10 0x5605e0ed7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.700+0000 7efd08c8f640 1 -- 172.21.15.16:0/3248877749 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605e0ec72b0 msgr2=0x5605e0ed6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:26.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.700+0000 7efd08c8f640 1 --2- 172.21.15.16:0/3248877749 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605e0ec72b0 0x5605e0ed6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.700+0000 7efd08c8f640 1 -- 172.21.15.16:0/3248877749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5605e0ecce10 msgr2=0x5605e0ed7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:26.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.700+0000 7efd08c8f640 1 --2- 172.21.15.16:0/3248877749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5605e0ecce10 0x5605e0ed7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.700+0000 7efd08c8f640 1 -- 172.21.15.16:0/3248877749 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605e0ed84a0 con 0x5605e0ec7ca0 2024-03-28T22:10:26.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.700+0000 7efd09c91640 1 --2- 172.21.15.16:0/3248877749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5605e0ecce10 0x5605e0ed7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:10:26.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.701+0000 7efd08c8f640 1 --2- 172.21.15.16:0/3248877749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605e0ec7ca0 0x5605e0ed6ea0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7efcf409a540 tx=0x7efcf4085040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33cb545d84f2f3f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:26.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.701+0000 7efcecff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:10:26.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.701+0000 7efcecff9640 1 -- 172.21.15.16:0/3248877749 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf401d040 con 0x5605e0ec7ca0 2024-03-28T22:10:26.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.701+0000 7efcecff9640 1 -- 172.21.15.16:0/3248877749 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7efcf40cb870 con 0x5605e0ec7ca0 2024-03-28T22:10:26.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.701+0000 7efd0e3b2740 1 -- 172.21.15.16:0/3248877749 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efcfc05f020 con 0x5605e0ec7ca0 2024-03-28T22:10:26.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.701+0000 7efcecff9640 1 -- 172.21.15.16:0/3248877749 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7efcf409a840 con 0x5605e0ec7ca0 2024-03-28T22:10:26.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.701+0000 7efd0e3b2740 10 client.15423 fetch_fsmap learned FSMap version 21 2024-03-28T22:10:26.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.701+0000 7efd0e3b2740 1 -- 172.21.15.16:0/3248877749 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5605e0f43f70 con 0x5605e0ec7ca0 2024-03-28T22:10:26.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.701+0000 7efcecff9640 1 -- 172.21.15.16:0/3248877749 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efcfc05f020 con 0x5605e0ec7ca0 2024-03-28T22:10:26.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.701+0000 7efcecff9640 1 -- 172.21.15.16:0/3248877749 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7efcf4005310 con 0x5605e0ec7ca0 2024-03-28T22:10:26.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.701+0000 7efd0e3b2740 10 client.15423 fetch_fsmap finished waiting for FSMap version 21 2024-03-28T22:10:26.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.701+0000 7efd0e3b2740 10 client.15423 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:10:26.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.701+0000 7efd0e3b2740 20 client.15423 populate_metadata read hostname 'smithi016' 2024-03-28T22:10:26.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.701+0000 7efd0e3b2740 1 --2- 172.21.15.16:0/3248877749 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5605e0f44960 0x5605e0f46d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:26.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.701+0000 7efd0e3b2740 4 client.15423 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:10:26.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.701+0000 7efd0e3b2740 1 -- 172.21.15.16:0/3248877749 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5605e0e33a10 con 0x5605e0f44960 2024-03-28T22:10:26.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.701+0000 7efd09490640 1 --2- 172.21.15.16:0/3248877749 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5605e0f44960 0x5605e0f46d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:26.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.702+0000 7efd09490640 1 --2- 172.21.15.16:0/3248877749 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5605e0f44960 0x5605e0f46d50 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5304103513b00ae server_cookie=4d96fc7430d5abb7 in_seq=0 out_seq=0 2024-03-28T22:10:26.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.702+0000 7efcecff9640 10 client.15423 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:10:26.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.702+0000 7efcecff9640 1 -- 172.21.15.16:0/3248877749 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5605e0e33a10 con 0x5605e0f44960 2024-03-28T22:10:26.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.702+0000 7efcecff9640 10 client.15423 handle_command_reply: tid=0 2024-03-28T22:10:26.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.702+0000 7efd0e3b2740 1 client.15423 shutdown 2024-03-28T22:10:26.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.702+0000 7efd0e3b2740 1 -- 172.21.15.16:0/3248877749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605e0ec7ca0 msgr2=0x5605e0ed6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:26.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.702+0000 7efd0e3b2740 1 --2- 172.21.15.16:0/3248877749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605e0ec7ca0 0x5605e0ed6ea0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7efcf409a540 tx=0x7efcf4085040 comp rx=0 tx=0).stop 2024-03-28T22:10:26.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.702+0000 7efd0e3b2740 1 -- 172.21.15.16:0/3248877749 shutdown_connections 2024-03-28T22:10:26.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.702+0000 7efd0e3b2740 1 --2- 172.21.15.16:0/3248877749 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5605e0f44960 0x5605e0f46d50 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.702+0000 7efd0e3b2740 1 --2- 172.21.15.16:0/3248877749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5605e0ecce10 0x5605e0ed7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.702+0000 7efd0e3b2740 1 --2- 172.21.15.16:0/3248877749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5605e0ec7ca0 0x5605e0ed6ea0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.702+0000 7efd0e3b2740 1 --2- 172.21.15.16:0/3248877749 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605e0ec72b0 0x5605e0ed6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.702+0000 7efcecff9640 0 client.15423 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:10:26.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.702+0000 7efd0e3b2740 1 -- 172.21.15.16:0/3248877749 >> 172.21.15.16:0/3248877749 conn(0x5605e0ec2820 msgr2=0x5605e0ec5a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:26.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.702+0000 7efd0e3b2740 1 -- 172.21.15.16:0/3248877749 shutdown_connections 2024-03-28T22:10:26.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.702+0000 7efd0e3b2740 1 -- 172.21.15.16:0/3248877749 wait complete. 2024-03-28T22:10:26.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.702+0000 7efd0e3b2740 20 client.15423 trim_cache size 0 max 16384 2024-03-28T22:10:26.712 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:10:26.712 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T22:10:26.712 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-03-28T22:10:26.712 INFO:teuthology.orchestra.run.smithi016.stdout: "ec95a5dc-1296-4853-8f53-230f7de5bd2b": { 2024-03-28T22:10:26.712 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-03-28T22:10:26.712 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "ec95a5dc-1296-4853-8f53-230f7de5bd2b", 2024-03-28T22:10:26.712 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-03-28T22:10:26.713 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:10:26.713 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:10:26.713 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:10:26.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.703+0000 7efd0b6f4640 1 -- 172.21.15.16:0/1280635551 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7efcf00a1e90 msgr2=0x7efcf00a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:26.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.703+0000 7efd0b6f4640 1 --2- 172.21.15.16:0/1280635551 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7efcf00a1e90 0x7efcf00a4350 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x5605e0ec8500 tx=0x7efcfc032590 comp rx=0 tx=0).stop 2024-03-28T22:10:26.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.703+0000 7efd0b6f4640 1 -- 172.21.15.16:0/1280635551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd04101860 msgr2=0x7efd04197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:26.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.703+0000 7efd0b6f4640 1 --2- 172.21.15.16:0/1280635551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd04101860 0x7efd04197fd0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7efcf400c970 tx=0x7efcf400ce30 comp rx=0 tx=0).stop 2024-03-28T22:10:26.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.703+0000 7efd0b6f4640 1 -- 172.21.15.16:0/1280635551 shutdown_connections 2024-03-28T22:10:26.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.703+0000 7efd0b6f4640 1 --2- 172.21.15.16:0/1280635551 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7efcf00a1e90 0x7efcf00a4350 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.703+0000 7efd0b6f4640 1 --2- 172.21.15.16:0/1280635551 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd04102220 0x7efd0419f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.703+0000 7efd0b6f4640 1 --2- 172.21.15.16:0/1280635551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd04101860 0x7efd04197fd0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.703+0000 7efd0b6f4640 1 --2- 172.21.15.16:0/1280635551 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efd04100670 0x7efd04197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:26.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.703+0000 7efd0b6f4640 1 -- 172.21.15.16:0/1280635551 >> 172.21.15.16:0/1280635551 conn(0x7efd040fbbe0 msgr2=0x7efd04068a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:26.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.703+0000 7efd0b6f4640 1 -- 172.21.15.16:0/1280635551 shutdown_connections 2024-03-28T22:10:26.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.703+0000 7efd0b6f4640 1 -- 172.21.15.16:0/1280635551 wait complete. 2024-03-28T22:10:26.716 INFO:tasks.cephfs.filesystem:scrub status for tag:ec95a5dc-1296-4853-8f53-230f7de5bd2b - {'path': '/', 'tag': 'ec95a5dc-1296-4853-8f53-230f7de5bd2b', 'options': 'recursive,force'} 2024-03-28T22:10:26.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:26.761+0000 7f80e98c2640 1 -- 172.21.15.16:0/828561439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f80e4108ac0 con 0x7f80e4103350 2024-03-28T22:10:27.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.194+0000 7f80e88c0640 1 -- 172.21.15.16:0/828561439 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v20) ==== 92+0+0 (secure 0 0 0) 0x7f80dc09b960 con 0x7f80e4103350 2024-03-28T22:10:27.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.196+0000 7f80e98c2640 1 -- 172.21.15.16:0/828561439 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f80cc0a1e80 msgr2=0x7f80cc0a4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:27.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.196+0000 7f80e98c2640 1 --2- 172.21.15.16:0/828561439 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f80cc0a1e80 0x7f80cc0a4340 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f80d8005fd0 tx=0x7f80d800b390 comp rx=0 tx=0).stop 2024-03-28T22:10:27.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.196+0000 7f80e98c2640 1 -- 172.21.15.16:0/828561439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80e4103350 msgr2=0x7f80e4074080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:27.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.197+0000 7f80e98c2640 1 --2- 172.21.15.16:0/828561439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80e4103350 0x7f80e4074080 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f80dc00eb10 tx=0x7f80dc00c2d0 comp rx=0 tx=0).stop 2024-03-28T22:10:27.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.198+0000 7f80e98c2640 1 -- 172.21.15.16:0/828561439 shutdown_connections 2024-03-28T22:10:27.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.199+0000 7f80e98c2640 1 --2- 172.21.15.16:0/828561439 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f80cc0a1e80 0x7f80cc0a4340 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:27.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.199+0000 7f80e98c2640 1 --2- 172.21.15.16:0/828561439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80e4103350 0x7f80e4074080 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:27.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.199+0000 7f80e98c2640 1 --2- 172.21.15.16:0/828561439 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f80e4102990 0x7f80e4073b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:27.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.199+0000 7f80e98c2640 1 --2- 172.21.15.16:0/828561439 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80e41017a0 0x7f80e4075500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:27.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.199+0000 7f80e98c2640 1 -- 172.21.15.16:0/828561439 >> 172.21.15.16:0/828561439 conn(0x7f80e40fcd10 msgr2=0x7f80e4105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:27.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.199+0000 7f80e98c2640 1 -- 172.21.15.16:0/828561439 shutdown_connections 2024-03-28T22:10:27.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.199+0000 7f80e98c2640 1 -- 172.21.15.16:0/828561439 wait complete. 2024-03-28T22:10:27.218 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-03-28T22:10:27.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.365+0000 7fa52ccf4640 1 -- 172.21.15.16:0/743850787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa528107460 msgr2=0x7fa5281098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:27.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.365+0000 7fa52ccf4640 1 --2- 172.21.15.16:0/743850787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa528107460 0x7fa5281098b0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fa518009ca0 tx=0x7fa51802c0f0 comp rx=0 tx=0).stop 2024-03-28T22:10:27.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.365+0000 7fa52ccf4640 1 -- 172.21.15.16:0/743850787 shutdown_connections 2024-03-28T22:10:27.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.365+0000 7fa52ccf4640 1 --2- 172.21.15.16:0/743850787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa528107460 0x7fa5281098b0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:27.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.365+0000 7fa52ccf4640 1 --2- 172.21.15.16:0/743850787 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa528104b30 0x7fa528106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:27.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.365+0000 7fa52ccf4640 1 --2- 172.21.15.16:0/743850787 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa528102200 0x7fa5281045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:27.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.365+0000 7fa52ccf4640 1 -- 172.21.15.16:0/743850787 >> 172.21.15.16:0/743850787 conn(0x7fa5280fbbf0 msgr2=0x7fa5280fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:27.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.365+0000 7fa52ccf4640 1 -- 172.21.15.16:0/743850787 shutdown_connections 2024-03-28T22:10:27.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.365+0000 7fa52ccf4640 1 -- 172.21.15.16:0/743850787 wait complete. 2024-03-28T22:10:27.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.366+0000 7fa52ccf4640 1 Processor -- start 2024-03-28T22:10:27.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.366+0000 7fa52ccf4640 1 -- start start 2024-03-28T22:10:27.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.366+0000 7fa52ccf4640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa528102200 0x7fa528197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:27.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.367+0000 7fa52ccf4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa528104b30 0x7fa528197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:27.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.367+0000 7fa52ccf4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa528107460 0x7fa52819eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:27.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.367+0000 7fa52ccf4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa52810d420 con 0x7fa528107460 2024-03-28T22:10:27.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.367+0000 7fa52ccf4640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fa52810d2a0 con 0x7fa528102200 2024-03-28T22:10:27.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.367+0000 7fa52ccf4640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa52810d5a0 con 0x7fa528104b30 2024-03-28T22:10:27.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.367+0000 7fa526d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa528107460 0x7fa52819eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:27.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.367+0000 7fa52659c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa528102200 0x7fa528197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:27.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.367+0000 7fa526d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa528107460 0x7fa52819eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44754/0 (socket says 172.21.15.16:44754) 2024-03-28T22:10:27.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.367+0000 7fa526d9d640 1 -- 172.21.15.16:0/3240951283 learned_addr learned my addr 172.21.15.16:0/3240951283 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:27.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.367+0000 7fa525d9b640 1 --2- 172.21.15.16:0/3240951283 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa528104b30 0x7fa528197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:27.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.367+0000 7fa526d9d640 1 -- 172.21.15.16:0/3240951283 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa528104b30 msgr2=0x7fa528197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:27.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.367+0000 7fa526d9d640 1 --2- 172.21.15.16:0/3240951283 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa528104b30 0x7fa528197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:27.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.367+0000 7fa526d9d640 1 -- 172.21.15.16:0/3240951283 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa528102200 msgr2=0x7fa528197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:27.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.367+0000 7fa526d9d640 1 --2- 172.21.15.16:0/3240951283 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa528102200 0x7fa528197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:27.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.367+0000 7fa526d9d640 1 -- 172.21.15.16:0/3240951283 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa52819f660 con 0x7fa528107460 2024-03-28T22:10:27.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.367+0000 7fa526d9d640 1 --2- 172.21.15.16:0/3240951283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa528107460 0x7fa52819eff0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fa518037040 tx=0x7fa518032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:27.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.367+0000 7fa51f7fe640 1 -- 172.21.15.16:0/3240951283 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa518041020 con 0x7fa528107460 2024-03-28T22:10:27.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.367+0000 7fa51f7fe640 1 -- 172.21.15.16:0/3240951283 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa518034d90 con 0x7fa528107460 2024-03-28T22:10:27.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.367+0000 7fa51f7fe640 1 -- 172.21.15.16:0/3240951283 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa518039040 con 0x7fa528107460 2024-03-28T22:10:27.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.367+0000 7fa52ccf4640 1 -- 172.21.15.16:0/3240951283 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa52819f8f0 con 0x7fa528107460 2024-03-28T22:10:27.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.368+0000 7fa52ccf4640 1 -- 172.21.15.16:0/3240951283 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa52819fe30 con 0x7fa528107460 2024-03-28T22:10:27.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.368+0000 7fa52ccf4640 1 -- 172.21.15.16:0/3240951283 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5280fd6e0 con 0x7fa528107460 2024-03-28T22:10:27.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.370+0000 7fa51f7fe640 1 -- 172.21.15.16:0/3240951283 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 136824+0+0 (secure 0 0 0) 0x7fa518044020 con 0x7fa528107460 2024-03-28T22:10:27.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.370+0000 7fa51f7fe640 1 --2- 172.21.15.16:0/3240951283 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fa5140a1ed0 0x7fa5140a4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:27.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.371+0000 7fa52659c640 1 -- 172.21.15.16:0/3240951283 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fa5140a1ed0 msgr2=0x7fa5140a4390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/21986004 2024-03-28T22:10:27.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.371+0000 7fa51f7fe640 1 -- 172.21.15.16:0/3240951283 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fa5180f0130 con 0x7fa528107460 2024-03-28T22:10:27.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.371+0000 7fa52659c640 1 --2- 172.21.15.16:0/3240951283 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fa5140a1ed0 0x7fa5140a4390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:10:27.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.374+0000 7fa51f7fe640 1 -- 172.21.15.16:0/3240951283 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa51804b020 con 0x7fa528107460 2024-03-28T22:10:27.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:27 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2500006459' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:10:27.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:27 smithi053 ceph-mon[35979]: scrub queued for path: / 2024-03-28T22:10:27.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:27 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:10:27.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:27 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:10:27.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:27 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:10:27.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:27 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/828561439' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T22:10:27.548 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:10:27 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: ignoring --setuser ceph since I am not root 2024-03-28T22:10:27.548 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:10:27 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: ignoring --setgroup ceph since I am not root 2024-03-28T22:10:27.548 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:10:27 smithi053 ceph-mgr[37381]: -- 172.21.15.53:0/1956858241 <== mon.2 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555f7d5745a0 con 0x555f7d549000 2024-03-28T22:10:27.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.557+0000 7fa52ccf4640 1 -- 172.21.15.16:0/3240951283 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fa528062350 con 0x7fa528107460 2024-03-28T22:10:27.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.561+0000 7fa51f7fe640 1 -- 172.21.15.16:0/3240951283 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v28)=0 v28) ==== 138+0+0 (secure 0 0 0) 0x7fa5180bd4e0 con 0x7fa528107460 2024-03-28T22:10:27.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.563+0000 7fa52ccf4640 1 -- 172.21.15.16:0/3240951283 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fa5140a1ed0 msgr2=0x7fa5140a4390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T22:10:27.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.563+0000 7fa52ccf4640 1 --2- 172.21.15.16:0/3240951283 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fa5140a1ed0 0x7fa5140a4390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:27.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.563+0000 7fa52ccf4640 1 -- 172.21.15.16:0/3240951283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa528107460 msgr2=0x7fa52819eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:27.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.563+0000 7fa52ccf4640 1 --2- 172.21.15.16:0/3240951283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa528107460 0x7fa52819eff0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fa518037040 tx=0x7fa518032f20 comp rx=0 tx=0).stop 2024-03-28T22:10:27.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.564+0000 7fa52ccf4640 1 -- 172.21.15.16:0/3240951283 shutdown_connections 2024-03-28T22:10:27.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.564+0000 7fa52ccf4640 1 --2- 172.21.15.16:0/3240951283 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7fa5140a1ed0 0x7fa5140a4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:27.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.564+0000 7fa52ccf4640 1 --2- 172.21.15.16:0/3240951283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa528107460 0x7fa52819eff0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:27.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.564+0000 7fa52ccf4640 1 --2- 172.21.15.16:0/3240951283 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa528104b30 0x7fa528197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:27.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.564+0000 7fa52ccf4640 1 --2- 172.21.15.16:0/3240951283 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa528102200 0x7fa528197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:27.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.564+0000 7fa52ccf4640 1 -- 172.21.15.16:0/3240951283 >> 172.21.15.16:0/3240951283 conn(0x7fa5280fbbf0 msgr2=0x7fa528102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:27.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.564+0000 7fa52ccf4640 1 -- 172.21.15.16:0/3240951283 shutdown_connections 2024-03-28T22:10:27.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.564+0000 7fa52ccf4640 1 -- 172.21.15.16:0/3240951283 wait complete. 2024-03-28T22:10:27.580 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:27 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: ignoring --setuser ceph since I am not root 2024-03-28T22:10:27.580 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:27 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: ignoring --setgroup ceph since I am not root 2024-03-28T22:10:27.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:27 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2500006459' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:10:27.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:27 smithi016 ceph-mon[32374]: scrub queued for path: / 2024-03-28T22:10:27.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:27 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:10:27.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:27 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:10:27.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:27 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:10:27.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:27 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/828561439' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T22:10:27.582 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-03-28T22:10:27.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:27 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2500006459' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:10:27.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:27 smithi063 ceph-mon[36249]: scrub queued for path: / 2024-03-28T22:10:27.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:27 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:10:27.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:27 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:10:27.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:27 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:10:27.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:27 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/828561439' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T22:10:27.682 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:10:27 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: ignoring --setuser ceph since I am not root 2024-03-28T22:10:27.683 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:10:27 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: ignoring --setgroup ceph since I am not root 2024-03-28T22:10:27.683 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:10:27 smithi063 ceph-mgr[38109]: -- 172.21.15.63:0/2267939869 <== mon.1 v2:172.21.15.63:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557ed441a5a0 con 0x557ed43eec00 2024-03-28T22:10:27.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.684+0000 7f33dcc8f640 1 -- 172.21.15.16:0/782921270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33d8102260 msgr2=0x7f33d8104650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:27.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.684+0000 7f33dcc8f640 1 --2- 172.21.15.16:0/782921270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33d8102260 0x7f33d8104650 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f33c8009900 tx=0x7f33c802bc30 comp rx=0 tx=0).stop 2024-03-28T22:10:27.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.684+0000 7f33dcc8f640 1 -- 172.21.15.16:0/782921270 shutdown_connections 2024-03-28T22:10:27.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.684+0000 7f33dcc8f640 1 --2- 172.21.15.16:0/782921270 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33d81074c0 0x7f33d81098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:27.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.684+0000 7f33dcc8f640 1 --2- 172.21.15.16:0/782921270 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33d8104b90 0x7f33d8106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:27.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.684+0000 7f33dcc8f640 1 --2- 172.21.15.16:0/782921270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33d8102260 0x7f33d8104650 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:27.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.684+0000 7f33dcc8f640 1 -- 172.21.15.16:0/782921270 >> 172.21.15.16:0/782921270 conn(0x7f33d80fbbf0 msgr2=0x7f33d80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:27.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.684+0000 7f33dcc8f640 1 -- 172.21.15.16:0/782921270 shutdown_connections 2024-03-28T22:10:27.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.684+0000 7f33dcc8f640 1 -- 172.21.15.16:0/782921270 wait complete. 2024-03-28T22:10:27.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.685+0000 7f33dcc8f640 1 Processor -- start 2024-03-28T22:10:27.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.685+0000 7f33dcc8f640 1 -- start start 2024-03-28T22:10:27.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.685+0000 7f33dcc8f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33d8102260 0x7f33d8101bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:27.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.685+0000 7f33dcc8f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33d8104b90 0x7f33d8100210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:27.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.685+0000 7f33dcc8f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33d81074c0 0x7f33d8100750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:27.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.685+0000 7f33dcc8f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f33d810d420 con 0x7f33d81074c0 2024-03-28T22:10:27.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.685+0000 7f33dcc8f640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f33d810d2a0 con 0x7f33d8104b90 2024-03-28T22:10:27.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.685+0000 7f33dcc8f640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f33d810d5a0 con 0x7f33d8102260 2024-03-28T22:10:27.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.685+0000 7f33d6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33d81074c0 0x7f33d8100750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:27.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.685+0000 7f33d6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33d81074c0 0x7f33d8100750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44774/0 (socket says 172.21.15.16:44774) 2024-03-28T22:10:27.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.685+0000 7f33d6d9d640 1 -- 172.21.15.16:0/4113808544 learned_addr learned my addr 172.21.15.16:0/4113808544 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:27.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.685+0000 7f33d659c640 1 --2- 172.21.15.16:0/4113808544 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33d8102260 0x7f33d8101bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:27.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.685+0000 7f33d5d9b640 1 --2- 172.21.15.16:0/4113808544 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33d8104b90 0x7f33d8100210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:27.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.685+0000 7f33d6d9d640 1 -- 172.21.15.16:0/4113808544 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33d8102260 msgr2=0x7f33d8101bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:27.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.685+0000 7f33d6d9d640 1 --2- 172.21.15.16:0/4113808544 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33d8102260 0x7f33d8101bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:27.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.685+0000 7f33d6d9d640 1 -- 172.21.15.16:0/4113808544 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33d8104b90 msgr2=0x7f33d8100210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:27.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.685+0000 7f33d6d9d640 1 --2- 172.21.15.16:0/4113808544 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33d8104b90 0x7f33d8100210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:27.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.685+0000 7f33d6d9d640 1 -- 172.21.15.16:0/4113808544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33d8100fe0 con 0x7f33d81074c0 2024-03-28T22:10:27.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.685+0000 7f33d6d9d640 1 --2- 172.21.15.16:0/4113808544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33d81074c0 0x7f33d8100750 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f33d000eb10 tx=0x7f33d000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:27.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.686+0000 7f33cf7fe640 1 -- 172.21.15.16:0/4113808544 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d0004b20 con 0x7f33d81074c0 2024-03-28T22:10:27.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.686+0000 7f33dcc8f640 1 -- 172.21.15.16:0/4113808544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33d81a1c10 con 0x7f33d81074c0 2024-03-28T22:10:27.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.686+0000 7f33cf7fe640 1 -- 172.21.15.16:0/4113808544 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33d0004cc0 con 0x7f33d81074c0 2024-03-28T22:10:27.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.686+0000 7f33cf7fe640 1 -- 172.21.15.16:0/4113808544 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d0010760 con 0x7f33d81074c0 2024-03-28T22:10:27.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.686+0000 7f33dcc8f640 1 -- 172.21.15.16:0/4113808544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33d81a21b0 con 0x7f33d81074c0 2024-03-28T22:10:27.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.687+0000 7f33dcc8f640 1 -- 172.21.15.16:0/4113808544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33d80fd6e0 con 0x7f33d81074c0 2024-03-28T22:10:27.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.688+0000 7f33cf7fe640 1 -- 172.21.15.16:0/4113808544 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 136824+0+0 (secure 0 0 0) 0x7f33d0005020 con 0x7f33d81074c0 2024-03-28T22:10:27.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.688+0000 7f33cf7fe640 1 --2- 172.21.15.16:0/4113808544 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f33c40a1e50 0x7f33c40a4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:27.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.689+0000 7f33d659c640 1 -- 172.21.15.16:0/4113808544 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f33c40a1e50 msgr2=0x7f33c40a4310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/21986004 2024-03-28T22:10:27.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.689+0000 7f33cf7fe640 1 -- 172.21.15.16:0/4113808544 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f33d00ce6d0 con 0x7f33d81074c0 2024-03-28T22:10:27.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.689+0000 7f33d659c640 1 --2- 172.21.15.16:0/4113808544 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f33c40a1e50 0x7f33c40a4310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:10:27.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.694+0000 7f33cf7fe640 1 -- 172.21.15.16:0/4113808544 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f33d009b970 con 0x7f33d81074c0 2024-03-28T22:10:27.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.879+0000 7f33dcc8f640 1 -- 172.21.15.16:0/4113808544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f33d80623c0 con 0x7f33d81074c0 2024-03-28T22:10:27.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.882+0000 7f33cf7fe640 1 -- 172.21.15.16:0/4113808544 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v29)=0 v29) ==== 133+0+0 (secure 0 0 0) 0x7f33d009b270 con 0x7f33d81074c0 2024-03-28T22:10:27.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.885+0000 7f33dcc8f640 1 -- 172.21.15.16:0/4113808544 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f33c40a1e50 msgr2=0x7f33c40a4310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T22:10:27.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.885+0000 7f33dcc8f640 1 --2- 172.21.15.16:0/4113808544 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f33c40a1e50 0x7f33c40a4310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:27.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.885+0000 7f33dcc8f640 1 -- 172.21.15.16:0/4113808544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33d81074c0 msgr2=0x7f33d8100750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:27.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.885+0000 7f33dcc8f640 1 --2- 172.21.15.16:0/4113808544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33d81074c0 0x7f33d8100750 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f33d000eb10 tx=0x7f33d000c2d0 comp rx=0 tx=0).stop 2024-03-28T22:10:27.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.885+0000 7f33dcc8f640 1 -- 172.21.15.16:0/4113808544 shutdown_connections 2024-03-28T22:10:27.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.885+0000 7f33dcc8f640 1 --2- 172.21.15.16:0/4113808544 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f33c40a1e50 0x7f33c40a4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:27.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.885+0000 7f33dcc8f640 1 --2- 172.21.15.16:0/4113808544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33d81074c0 0x7f33d8100750 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:27.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.885+0000 7f33dcc8f640 1 --2- 172.21.15.16:0/4113808544 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33d8104b90 0x7f33d8100210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:27.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.885+0000 7f33dcc8f640 1 --2- 172.21.15.16:0/4113808544 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33d8102260 0x7f33d8101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:27.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.885+0000 7f33dcc8f640 1 -- 172.21.15.16:0/4113808544 >> 172.21.15.16:0/4113808544 conn(0x7f33d80fbbf0 msgr2=0x7f33d8102c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:27.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.885+0000 7f33dcc8f640 1 -- 172.21.15.16:0/4113808544 shutdown_connections 2024-03-28T22:10:27.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:27.885+0000 7f33dcc8f640 1 -- 172.21.15.16:0/4113808544 wait complete. 2024-03-28T22:10:27.899 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-03-28T22:10:28.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.002+0000 7f4a42081640 1 -- 172.21.15.16:0/3995077830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a3c102200 msgr2=0x7f4a3c1045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:28.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.002+0000 7f4a42081640 1 --2- 172.21.15.16:0/3995077830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a3c102200 0x7f4a3c1045f0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f4a24009900 tx=0x7f4a2402bc30 comp rx=0 tx=0).stop 2024-03-28T22:10:28.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.002+0000 7f4a42081640 1 -- 172.21.15.16:0/3995077830 shutdown_connections 2024-03-28T22:10:28.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.002+0000 7f4a42081640 1 --2- 172.21.15.16:0/3995077830 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a3c107460 0x7f4a3c1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:28.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.002+0000 7f4a42081640 1 --2- 172.21.15.16:0/3995077830 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a3c104b30 0x7f4a3c106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:28.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.002+0000 7f4a42081640 1 --2- 172.21.15.16:0/3995077830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a3c102200 0x7f4a3c1045f0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:28.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.002+0000 7f4a42081640 1 -- 172.21.15.16:0/3995077830 >> 172.21.15.16:0/3995077830 conn(0x7f4a3c0fbbf0 msgr2=0x7f4a3c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:28.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.002+0000 7f4a42081640 1 -- 172.21.15.16:0/3995077830 shutdown_connections 2024-03-28T22:10:28.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.002+0000 7f4a42081640 1 -- 172.21.15.16:0/3995077830 wait complete. 2024-03-28T22:10:28.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.003+0000 7f4a42081640 1 Processor -- start 2024-03-28T22:10:28.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.003+0000 7f4a42081640 1 -- start start 2024-03-28T22:10:28.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.003+0000 7f4a42081640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a3c102200 0x7f4a3c197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:28.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.003+0000 7f4a42081640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a3c104b30 0x7f4a3c198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:28.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.003+0000 7f4a42081640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a3c107460 0x7f4a3c19f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:28.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.003+0000 7f4a42081640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c10d400 con 0x7f4a3c104b30 2024-03-28T22:10:28.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.003+0000 7f4a42081640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c10d280 con 0x7f4a3c107460 2024-03-28T22:10:28.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.003+0000 7f4a42081640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c10d580 con 0x7f4a3c102200 2024-03-28T22:10:28.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.003+0000 7f4a3affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a3c104b30 0x7f4a3c198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:28.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.003+0000 7f4a3affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a3c104b30 0x7f4a3c198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44790/0 (socket says 172.21.15.16:44790) 2024-03-28T22:10:28.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.003+0000 7f4a3affd640 1 -- 172.21.15.16:0/1379000031 learned_addr learned my addr 172.21.15.16:0/1379000031 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:28.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.003+0000 7f4a3b7fe640 1 --2- 172.21.15.16:0/1379000031 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a3c102200 0x7f4a3c197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:28.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.003+0000 7f4a3bfff640 1 --2- 172.21.15.16:0/1379000031 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a3c107460 0x7f4a3c19f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:28.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.004+0000 7f4a3affd640 1 -- 172.21.15.16:0/1379000031 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a3c102200 msgr2=0x7f4a3c197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:28.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.004+0000 7f4a3affd640 1 --2- 172.21.15.16:0/1379000031 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a3c102200 0x7f4a3c197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:28.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.004+0000 7f4a3affd640 1 -- 172.21.15.16:0/1379000031 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a3c107460 msgr2=0x7f4a3c19f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:28.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.004+0000 7f4a3affd640 1 --2- 172.21.15.16:0/1379000031 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a3c107460 0x7f4a3c19f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:28.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.004+0000 7f4a3affd640 1 -- 172.21.15.16:0/1379000031 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a3c19f6f0 con 0x7f4a3c104b30 2024-03-28T22:10:28.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.004+0000 7f4a3b7fe640 1 --2- 172.21.15.16:0/1379000031 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a3c102200 0x7f4a3c197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:10:28.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.004+0000 7f4a3affd640 1 --2- 172.21.15.16:0/1379000031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a3c104b30 0x7f4a3c198010 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c00e9c0 tx=0x7f4a2c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:28.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.004+0000 7f4a38ff9640 1 -- 172.21.15.16:0/1379000031 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c00cd80 con 0x7f4a3c104b30 2024-03-28T22:10:28.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.004+0000 7f4a42081640 1 -- 172.21.15.16:0/1379000031 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a3c19f9e0 con 0x7f4a3c104b30 2024-03-28T22:10:28.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.004+0000 7f4a38ff9640 1 -- 172.21.15.16:0/1379000031 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a2c005500 con 0x7f4a3c104b30 2024-03-28T22:10:28.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.004+0000 7f4a38ff9640 1 -- 172.21.15.16:0/1379000031 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c010690 con 0x7f4a3c104b30 2024-03-28T22:10:28.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.004+0000 7f4a42081640 1 -- 172.21.15.16:0/1379000031 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a3c19ff30 con 0x7f4a3c104b30 2024-03-28T22:10:28.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.005+0000 7f4a42081640 1 -- 172.21.15.16:0/1379000031 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a3c0fd6e0 con 0x7f4a3c104b30 2024-03-28T22:10:28.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.006+0000 7f4a38ff9640 1 -- 172.21.15.16:0/1379000031 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 136824+0+0 (secure 0 0 0) 0x7f4a2c0046c0 con 0x7f4a3c104b30 2024-03-28T22:10:28.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.007+0000 7f4a38ff9640 1 --2- 172.21.15.16:0/1379000031 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4a280a1e50 0x7f4a280a4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:28.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.007+0000 7f4a3b7fe640 1 -- 172.21.15.16:0/1379000031 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4a280a1e50 msgr2=0x7f4a280a4310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/21986004 2024-03-28T22:10:28.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.007+0000 7f4a38ff9640 1 -- 172.21.15.16:0/1379000031 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f4a2c0ce2e0 con 0x7f4a3c104b30 2024-03-28T22:10:28.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.007+0000 7f4a3b7fe640 1 --2- 172.21.15.16:0/1379000031 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4a280a1e50 0x7f4a280a4310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:10:28.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.012+0000 7f4a38ff9640 1 -- 172.21.15.16:0/1379000031 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4a2c09b5e0 con 0x7f4a3c104b30 2024-03-28T22:10:28.134 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:27 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:27.837+0000 7f8ae104d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T22:10:28.134 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:27 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:27.936+0000 7f8ae104d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T22:10:28.182 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:10:27 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:10:27.792+0000 7fc206156200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T22:10:28.182 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:10:27 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:10:27.889+0000 7fc206156200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T22:10:28.198 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:10:27 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:10:27.901+0000 7f94a9e33200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T22:10:28.198 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:10:28 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:10:28.032+0000 7f94a9e33200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T22:10:28.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.208+0000 7f4a3b7fe640 1 -- 172.21.15.16:0/1379000031 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4a280a1e50 msgr2=0x7f4a280a4310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/21986004 2024-03-28T22:10:28.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.208+0000 7f4a3b7fe640 1 --2- 172.21.15.16:0/1379000031 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4a280a1e50 0x7f4a280a4310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:10:28.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.218+0000 7f4a42081640 1 -- 172.21.15.16:0/1379000031 --> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f4a3c061b90 con 0x7f4a280a1e50 2024-03-28T22:10:28.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:28 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/828561439' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-28T22:10:28.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:28 smithi016 ceph-mon[32374]: mgrmap e20: x(active, since 8m), standbys: z, y 2024-03-28T22:10:28.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:28 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3240951283' entity='client.admin' 2024-03-28T22:10:28.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:28 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/4113808544' entity='client.admin' 2024-03-28T22:10:28.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:28 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/828561439' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-28T22:10:28.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:28 smithi053 ceph-mon[35979]: mgrmap e20: x(active, since 8m), standbys: z, y 2024-03-28T22:10:28.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:28 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3240951283' entity='client.admin' 2024-03-28T22:10:28.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:28 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/4113808544' entity='client.admin' 2024-03-28T22:10:28.556 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:28 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/828561439' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-28T22:10:28.556 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:28 smithi063 ceph-mon[36249]: mgrmap e20: x(active, since 8m), standbys: z, y 2024-03-28T22:10:28.556 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:28 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/3240951283' entity='client.admin' 2024-03-28T22:10:28.556 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:28 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/4113808544' entity='client.admin' 2024-03-28T22:10:28.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.609+0000 7f4a3b7fe640 1 -- 172.21.15.16:0/1379000031 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4a280a1e50 msgr2=0x7f4a280a4310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/21986004 2024-03-28T22:10:28.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:28.609+0000 7f4a3b7fe640 1 --2- 172.21.15.16:0/1379000031 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4a280a1e50 0x7f4a280a4310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:10:28.895 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:10:28 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:10:28.553+0000 7fc206156200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T22:10:28.895 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:10:28 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:10:28.649+0000 7fc206156200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T22:10:28.895 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:10:28 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:10:28.794+0000 7fc206156200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T22:10:28.895 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:10:28 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:10:28.892+0000 7fc206156200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T22:10:28.936 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:28 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:28.603+0000 7f8ae104d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T22:10:28.936 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:28 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:28.699+0000 7f8ae104d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T22:10:28.936 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:28 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:28.842+0000 7f8ae104d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T22:10:29.182 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:10:28 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:10:28.978+0000 7fc206156200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T22:10:29.195 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:10:28 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:10:28.904+0000 7f94a9e33200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T22:10:29.196 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:10:29 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:10:29.027+0000 7f94a9e33200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T22:10:29.196 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:10:29 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:10:29.193+0000 7f94a9e33200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T22:10:29.298 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:28 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:28.940+0000 7f8ae104d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T22:10:29.299 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:29 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:29.026+0000 7f8ae104d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T22:10:29.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:29.410+0000 7f4a3b7fe640 1 -- 172.21.15.16:0/1379000031 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4a280a1e50 msgr2=0x7f4a280a4310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/21986004 2024-03-28T22:10:29.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:29.410+0000 7f4a3b7fe640 1 --2- 172.21.15.16:0/1379000031 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4a280a1e50 0x7f4a280a4310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:10:29.451 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:10:29 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:10:29.342+0000 7f94a9e33200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T22:10:29.553 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:10:29 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:10:29.249+0000 7fc206156200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T22:10:29.553 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:10:29 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:10:29.364+0000 7fc206156200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T22:10:29.554 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:10:29 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:10:29.453+0000 7fc206156200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T22:10:29.554 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:10:29 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:10:29.551+0000 7fc206156200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T22:10:29.599 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:29 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:29.296+0000 7f8ae104d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T22:10:29.599 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:29 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:29.410+0000 7f8ae104d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T22:10:29.599 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:29 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:29.499+0000 7f8ae104d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T22:10:29.776 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:10:29 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:10:29.448+0000 7f94a9e33200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T22:10:29.932 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:10:29 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:10:29.732+0000 7fc206156200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T22:10:29.936 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:29 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:29.596+0000 7f8ae104d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T22:10:29.936 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:29 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:29.775+0000 7f8ae104d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T22:10:30.040 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:29 smithi053 ceph-mon[35979]: scrub summary: idle 2024-03-28T22:10:30.041 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:10:29 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:10:29.773+0000 7f94a9e33200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T22:10:30.041 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:10:29 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:10:29.903+0000 7f94a9e33200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T22:10:30.297 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:10:30 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:10:30.038+0000 7f94a9e33200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T22:10:30.298 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:10:30 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:10:30.137+0000 7f94a9e33200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T22:10:30.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:29 smithi063 ceph-mon[36249]: scrub summary: idle 2024-03-28T22:10:30.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:29 smithi016 ceph-mon[32374]: scrub summary: idle 2024-03-28T22:10:30.797 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:10:30 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:10:30.355+0000 7f94a9e33200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T22:10:31.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:31.011+0000 7f4a3b7fe640 1 -- 172.21.15.16:0/1379000031 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4a280a1e50 msgr2=0x7f4a280a4310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/21986004 2024-03-28T22:10:31.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:31.011+0000 7f4a3b7fe640 1 --2- 172.21.15.16:0/1379000031 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4a280a1e50 0x7f4a280a4310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-28T22:10:31.182 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:10:30 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:10:30.742+0000 7fc206156200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T22:10:31.182 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:10:30 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:10:30.877+0000 7fc206156200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T22:10:31.186 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:30 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:30.767+0000 7f8ae104d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T22:10:31.186 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:30 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:30.902+0000 7f8ae104d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T22:10:31.517 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:10:31 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:10:31.221+0000 7fc206156200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T22:10:31.518 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:10:31 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:10:31.313+0000 7fc206156200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T22:10:31.544 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:31 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:31.244+0000 7f8ae104d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T22:10:31.544 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:31 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:31.337+0000 7f8ae104d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T22:10:31.932 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:10:31 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:10:31.515+0000 7fc206156200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T22:10:31.932 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:10:31 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:10:31.705+0000 7fc206156200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T22:10:31.936 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:31 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:31.541+0000 7f8ae104d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T22:10:31.936 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:31 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:31.735+0000 7f8ae104d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T22:10:32.047 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:10:31 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:10:31.607+0000 7f94a9e33200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T22:10:32.047 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:10:31 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:10:31.744+0000 7f94a9e33200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T22:10:32.266 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:10:31 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:10:31.996+0000 7fc206156200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T22:10:32.297 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:32 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:32.031+0000 7f8ae104d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T22:10:32.465 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:10:32 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:10:32.140+0000 7f94a9e33200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T22:10:32.465 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:10:32 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:10:32.232+0000 7f94a9e33200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T22:10:32.682 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:10:32 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:10:32.263+0000 7fc206156200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T22:10:32.682 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:10:32 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:10:32.506+0000 7fc206156200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T22:10:32.686 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:32 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:32.294+0000 7f8ae104d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T22:10:32.686 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:32 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:32.540+0000 7f8ae104d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T22:10:32.721 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:10:32 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:10:32.463+0000 7f94a9e33200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T22:10:33.047 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:10:32 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:10:32.719+0000 7f94a9e33200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T22:10:33.397 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:10:33 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:10:33.097+0000 7f94a9e33200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T22:10:33.397 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:10:33 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:10:33.394+0000 7f94a9e33200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T22:10:33.682 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:10:33 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:10:33.339+0000 7fc206156200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T22:10:33.686 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:33 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:33.379+0000 7f8ae104d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T22:10:34.047 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:10:33 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:10:33.722+0000 7f94a9e33200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T22:10:34.182 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:10:33 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:10:33.928+0000 7fc206156200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T22:10:34.182 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:10:34 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:10:34.028+0000 7fc206156200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T22:10:34.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:34.215+0000 7f4a3b7fe640 1 -- 172.21.15.16:0/1379000031 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4a280a1e50 msgr2=0x7f4a280a4310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/21986004 2024-03-28T22:10:34.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:34.215+0000 7f4a3b7fe640 1 --2- 172.21.15.16:0/1379000031 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4a280a1e50 0x7f4a280a4310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-28T22:10:34.301 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:33 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:33.965+0000 7f8ae104d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T22:10:34.301 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:34 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:34.069+0000 7f8ae104d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T22:10:34.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:34.311+0000 7f4a38ff9640 1 -- 172.21.15.16:0/1379000031 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 21) ==== 136441+0+0 (secure 0 0 0) 0x7f4a2c097550 con 0x7f4a3c104b30 2024-03-28T22:10:34.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:34.311+0000 7f4a38ff9640 1 -- 172.21.15.16:0/1379000031 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4a280a1e50 msgr2=0x7f4a280a4310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T22:10:34.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:34.311+0000 7f4a38ff9640 1 --2- 172.21.15.16:0/1379000031 >> [v2:172.21.15.16:6800/21986004,v1:172.21.15.16:6801/21986004] conn(0x7f4a280a1e50 0x7f4a280a4310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:34.564 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:10:34 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z[38105]: 2024-03-28T22:10:34.250+0000 7fc206156200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T22:10:34.564 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:34 smithi063 ceph-mon[36249]: Standby manager daemon z restarted 2024-03-28T22:10:34.564 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:34 smithi063 ceph-mon[36249]: Standby manager daemon z started 2024-03-28T22:10:34.564 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:34 smithi063 ceph-mon[36249]: from='mgr.? 172.21.15.63:0/2361740598' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T22:10:34.564 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:34 smithi063 ceph-mon[36249]: from='mgr.? 172.21.15.63:0/2361740598' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T22:10:34.564 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:34 smithi063 ceph-mon[36249]: from='mgr.? 172.21.15.63:0/2361740598' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T22:10:34.564 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:34 smithi063 ceph-mon[36249]: from='mgr.? 172.21.15.63:0/2361740598' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T22:10:34.565 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:34 smithi063 ceph-mon[36249]: Active manager daemon x restarted 2024-03-28T22:10:34.565 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:34 smithi063 ceph-mon[36249]: Activating manager daemon x 2024-03-28T22:10:34.565 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:34 smithi063 ceph-mon[36249]: osdmap e89: 12 total, 12 up, 12 in 2024-03-28T22:10:34.565 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:34 smithi063 ceph-mon[36249]: mgrmap e21: x(active, starting, since 0.0118871s), standbys: y, z 2024-03-28T22:10:34.565 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:34 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:10:34.565 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:34 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:10:34.565 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:34 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:10:34.580 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:34 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:34.298+0000 7f8ae104d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T22:10:34.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:34 smithi016 ceph-mon[32374]: Standby manager daemon z restarted 2024-03-28T22:10:34.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:34 smithi016 ceph-mon[32374]: Standby manager daemon z started 2024-03-28T22:10:34.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:34 smithi016 ceph-mon[32374]: from='mgr.? 172.21.15.63:0/2361740598' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T22:10:34.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:34 smithi016 ceph-mon[32374]: from='mgr.? 172.21.15.63:0/2361740598' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T22:10:34.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:34 smithi016 ceph-mon[32374]: from='mgr.? 172.21.15.63:0/2361740598' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T22:10:34.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:34 smithi016 ceph-mon[32374]: from='mgr.? 172.21.15.63:0/2361740598' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T22:10:34.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:34 smithi016 ceph-mon[32374]: Active manager daemon x restarted 2024-03-28T22:10:34.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:34 smithi016 ceph-mon[32374]: Activating manager daemon x 2024-03-28T22:10:34.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:34 smithi016 ceph-mon[32374]: osdmap e89: 12 total, 12 up, 12 in 2024-03-28T22:10:34.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:34 smithi016 ceph-mon[32374]: mgrmap e21: x(active, starting, since 0.0118871s), standbys: y, z 2024-03-28T22:10:34.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:34 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:10:34.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:34 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:10:34.581 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:34 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:10:34.621 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:34 smithi053 ceph-mon[35979]: Standby manager daemon z restarted 2024-03-28T22:10:34.621 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:34 smithi053 ceph-mon[35979]: Standby manager daemon z started 2024-03-28T22:10:34.621 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:34 smithi053 ceph-mon[35979]: from='mgr.? 172.21.15.63:0/2361740598' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T22:10:34.621 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:34 smithi053 ceph-mon[35979]: from='mgr.? 172.21.15.63:0/2361740598' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T22:10:34.621 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:34 smithi053 ceph-mon[35979]: from='mgr.? 172.21.15.63:0/2361740598' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T22:10:34.621 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:34 smithi053 ceph-mon[35979]: from='mgr.? 172.21.15.63:0/2361740598' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T22:10:34.621 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:34 smithi053 ceph-mon[35979]: Active manager daemon x restarted 2024-03-28T22:10:34.622 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:34 smithi053 ceph-mon[35979]: Activating manager daemon x 2024-03-28T22:10:34.622 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:34 smithi053 ceph-mon[35979]: osdmap e89: 12 total, 12 up, 12 in 2024-03-28T22:10:34.622 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:34 smithi053 ceph-mon[35979]: mgrmap e21: x(active, starting, since 0.0118871s), standbys: y, z 2024-03-28T22:10:34.622 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:34 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:10:34.622 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:34 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:10:34.622 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:34 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:10:34.936 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:34 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:34.577+0000 7f8a43a2d700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:10:34.936 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:34 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:34.577+0000 7f8a43a2d700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:10:34.936 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:34 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:34.577+0000 7f8a43a2d700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:10:34.937 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:34 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:34.578+0000 7f8a43a2d700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:10:34.937 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:34 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:34.578+0000 7f8a43a2d700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:10:34.937 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:34 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:34.578+0000 7f8a40a27700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:10:34.937 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:34 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:34.578+0000 7f8a40a27700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:10:34.937 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:34 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:34.578+0000 7f8a40a27700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:10:34.937 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:34 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:34.578+0000 7f8a40a27700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:10:34.937 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:10:34 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:10:34.578+0000 7f8a40a27700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:10:35.047 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:10:34 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:10:34.747+0000 7f94a9e33200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T22:10:35.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.343+0000 7f4a38ff9640 1 -- 172.21.15.16:0/1379000031 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 22) ==== 136568+0+0 (secure 0 0 0) 0x7f4a2c06b7d0 con 0x7f4a3c104b30 2024-03-28T22:10:35.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.344+0000 7f4a38ff9640 1 --2- 172.21.15.16:0/1379000031 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4a28051510 0x7f4a280518f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:35.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.344+0000 7f4a38ff9640 1 -- 172.21.15.16:0/1379000031 --> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f4a2803c6a0 con 0x7f4a28051510 2024-03-28T22:10:35.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.345+0000 7f4a3b7fe640 1 --2- 172.21.15.16:0/1379000031 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4a28051510 0x7f4a280518f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:35.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.346+0000 7f4a3b7fe640 1 --2- 172.21.15.16:0/1379000031 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4a28051510 0x7f4a280518f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4a2402c7b0 tx=0x7f4a24005fd0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:35.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.413+0000 7f4a38ff9640 1 -- 172.21.15.16:0/1379000031 <== mgr.15441 v2:172.21.15.16:6800/1819900910 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f4a2803c6a0 con 0x7f4a28051510 2024-03-28T22:10:35.416 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path / 2024-03-28T22:10:35.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.415+0000 7f4a42081640 1 -- 172.21.15.16:0/1379000031 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4a28051510 msgr2=0x7f4a280518f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:35.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.415+0000 7f4a42081640 1 --2- 172.21.15.16:0/1379000031 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4a28051510 0x7f4a280518f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4a2402c7b0 tx=0x7f4a24005fd0 comp rx=0 tx=0).stop 2024-03-28T22:10:35.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.415+0000 7f4a42081640 1 -- 172.21.15.16:0/1379000031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a3c104b30 msgr2=0x7f4a3c198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:35.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.415+0000 7f4a42081640 1 --2- 172.21.15.16:0/1379000031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a3c104b30 0x7f4a3c198010 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c00e9c0 tx=0x7f4a2c00ee80 comp rx=0 tx=0).stop 2024-03-28T22:10:35.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.415+0000 7f4a3bfff640 1 -- 172.21.15.16:0/1379000031 reap_dead start 2024-03-28T22:10:35.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.415+0000 7f4a42081640 1 -- 172.21.15.16:0/1379000031 shutdown_connections 2024-03-28T22:10:35.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.415+0000 7f4a42081640 1 -- 172.21.15.16:0/1379000031 >> 172.21.15.16:0/1379000031 conn(0x7f4a3c0fbbf0 msgr2=0x7f4a3c102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:35.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.415+0000 7f4a42081640 1 -- 172.21.15.16:0/1379000031 shutdown_connections 2024-03-28T22:10:35.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.415+0000 7f4a42081640 1 -- 172.21.15.16:0/1379000031 wait complete. 2024-03-28T22:10:35.428 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-03-28T22:10:35.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.533+0000 7fb054313640 1 -- 172.21.15.16:0/3700546181 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb04c102230 msgr2=0x7fb04c108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:35.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.533+0000 7fb054313640 1 --2- 172.21.15.16:0/3700546181 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb04c102230 0x7fb04c108ac0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fb03c009900 tx=0x7fb03c02bc50 comp rx=0 tx=0).stop 2024-03-28T22:10:35.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.533+0000 7fb054313640 1 -- 172.21.15.16:0/3700546181 shutdown_connections 2024-03-28T22:10:35.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.533+0000 7fb054313640 1 --2- 172.21.15.16:0/3700546181 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb04c102230 0x7fb04c108ac0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:35.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.533+0000 7fb054313640 1 --2- 172.21.15.16:0/3700546181 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb04c101870 0x7fb04c101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:35.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.533+0000 7fb054313640 1 --2- 172.21.15.16:0/3700546181 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb04c100680 0x7fb04c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:35.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.533+0000 7fb054313640 1 -- 172.21.15.16:0/3700546181 >> 172.21.15.16:0/3700546181 conn(0x7fb04c0fbbf0 msgr2=0x7fb04c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:35.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.533+0000 7fb054313640 1 -- 172.21.15.16:0/3700546181 shutdown_connections 2024-03-28T22:10:35.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.533+0000 7fb054313640 1 -- 172.21.15.16:0/3700546181 wait complete. 2024-03-28T22:10:35.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.533+0000 7fb054313640 1 Processor -- start 2024-03-28T22:10:35.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.533+0000 7fb054313640 1 -- start start 2024-03-28T22:10:35.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.533+0000 7fb054313640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb04c100680 0x7fb04c069ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:35.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.533+0000 7fb054313640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb04c101870 0x7fb04c19c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:35.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.533+0000 7fb054313640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb04c102230 0x7fb04c19ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:35.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.533+0000 7fb054313640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb04c10c2e0 con 0x7fb04c101870 2024-03-28T22:10:35.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.533+0000 7fb054313640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb04c10c160 con 0x7fb04c102230 2024-03-28T22:10:35.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.533+0000 7fb054313640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb04c10c460 con 0x7fb04c100680 2024-03-28T22:10:35.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.533+0000 7fb0518ae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb04c101870 0x7fb04c19c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:35.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.534+0000 7fb0518ae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb04c101870 0x7fb04c19c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59774/0 (socket says 172.21.15.16:59774) 2024-03-28T22:10:35.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.534+0000 7fb0518ae640 1 -- 172.21.15.16:0/172756912 learned_addr learned my addr 172.21.15.16:0/172756912 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:35.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.534+0000 7fb0528b0640 1 --2- 172.21.15.16:0/172756912 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb04c102230 0x7fb04c19ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:35.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.534+0000 7fb0520af640 1 --2- 172.21.15.16:0/172756912 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb04c100680 0x7fb04c069ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:35.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.534+0000 7fb0518ae640 1 -- 172.21.15.16:0/172756912 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb04c100680 msgr2=0x7fb04c069ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:35.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.534+0000 7fb0518ae640 1 --2- 172.21.15.16:0/172756912 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb04c100680 0x7fb04c069ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:35.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.534+0000 7fb0518ae640 1 -- 172.21.15.16:0/172756912 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb04c102230 msgr2=0x7fb04c19ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:35.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.534+0000 7fb0518ae640 1 --2- 172.21.15.16:0/172756912 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb04c102230 0x7fb04c19ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:35.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.534+0000 7fb0518ae640 1 -- 172.21.15.16:0/172756912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb04c19f600 con 0x7fb04c101870 2024-03-28T22:10:35.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.534+0000 7fb0518ae640 1 --2- 172.21.15.16:0/172756912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb04c101870 0x7fb04c19c700 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fb04000c920 tx=0x7fb04000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:35.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.534+0000 7fb04b7fe640 1 -- 172.21.15.16:0/172756912 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb040007c70 con 0x7fb04c101870 2024-03-28T22:10:35.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.534+0000 7fb04b7fe640 1 -- 172.21.15.16:0/172756912 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb040007e10 con 0x7fb04c101870 2024-03-28T22:10:35.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.534+0000 7fb054313640 1 -- 172.21.15.16:0/172756912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb04c19f8f0 con 0x7fb04c101870 2024-03-28T22:10:35.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.534+0000 7fb04b7fe640 1 -- 172.21.15.16:0/172756912 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb040018890 con 0x7fb04c101870 2024-03-28T22:10:35.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.534+0000 7fb054313640 1 -- 172.21.15.16:0/172756912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb04c19fe40 con 0x7fb04c101870 2024-03-28T22:10:35.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.535+0000 7fb054313640 1 -- 172.21.15.16:0/172756912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb04c1078d0 con 0x7fb04c101870 2024-03-28T22:10:35.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.536+0000 7fb04b7fe640 1 -- 172.21.15.16:0/172756912 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) ==== 136568+0+0 (secure 0 0 0) 0x7fb040018a70 con 0x7fb04c101870 2024-03-28T22:10:35.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.537+0000 7fb04b7fe640 1 --2- 172.21.15.16:0/172756912 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fb0340a1b60 0x7fb0340a4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:35.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.537+0000 7fb04b7fe640 1 -- 172.21.15.16:0/172756912 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fb0400cdbf0 con 0x7fb04c101870 2024-03-28T22:10:35.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.537+0000 7fb0520af640 1 --2- 172.21.15.16:0/172756912 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fb0340a1b60 0x7fb0340a4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:35.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.537+0000 7fb0520af640 1 --2- 172.21.15.16:0/172756912 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fb0340a1b60 0x7fb0340a4020 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb044005c40 tx=0x7fb044005bd0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:35.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.541+0000 7fb04b7fe640 1 -- 172.21.15.16:0/172756912 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb040014030 con 0x7fb04c101870 2024-03-28T22:10:35.657 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T22:10:35.658 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T22:10:35.658 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T22:10:35.658 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T22:10:35.658 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T22:10:35.658 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T22:10:35.658 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T22:10:35.658 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T22:10:35.658 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T22:10:35.658 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T22:10:35.658 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T22:10:35.658 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T22:10:35.658 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T22:10:35.658 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T22:10:35.659 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T22:10:35.659 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T22:10:35.659 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:10:35.659 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:10:35.659 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T22:10:35.659 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:10:35.659 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:10:35.659 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:10:35.659 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:10:35.659 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:10:35.659 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T22:10:35.659 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T22:10:35.659 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:10:35.659 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T22:10:35.660 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T22:10:35.660 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T22:10:35.660 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: Manager daemon x is now available 2024-03-28T22:10:35.660 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:10:35.660 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:10:35.660 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T22:10:35.660 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T22:10:35.660 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: [28/Mar/2024:22:10:34] ENGINE Bus STARTING 2024-03-28T22:10:35.660 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: [28/Mar/2024:22:10:34] ENGINE Serving on http://172.21.15.16:8765 2024-03-28T22:10:35.660 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: [28/Mar/2024:22:10:35] ENGINE Serving on https://172.21.15.16:7150 2024-03-28T22:10:35.660 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:35 smithi063 ceph-mon[36249]: [28/Mar/2024:22:10:35] ENGINE Bus STARTED 2024-03-28T22:10:35.664 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:10:35.554+0000 7f94a9e33200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T22:10:35.664 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:10:35.661+0000 7f94a9e33200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T22:10:35.664 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T22:10:35.664 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T22:10:35.664 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T22:10:35.664 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T22:10:35.665 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T22:10:35.665 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T22:10:35.665 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T22:10:35.665 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T22:10:35.665 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T22:10:35.665 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T22:10:35.665 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T22:10:35.665 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T22:10:35.665 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T22:10:35.665 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T22:10:35.665 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T22:10:35.665 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T22:10:35.665 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:10:35.665 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:10:35.666 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T22:10:35.666 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:10:35.666 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:10:35.666 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:10:35.666 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:10:35.666 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:10:35.666 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T22:10:35.667 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T22:10:35.667 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:10:35.667 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T22:10:35.667 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T22:10:35.667 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T22:10:35.667 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: Manager daemon x is now available 2024-03-28T22:10:35.667 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:10:35.667 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:10:35.667 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T22:10:35.668 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T22:10:35.668 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: [28/Mar/2024:22:10:34] ENGINE Bus STARTING 2024-03-28T22:10:35.668 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: [28/Mar/2024:22:10:34] ENGINE Serving on http://172.21.15.16:8765 2024-03-28T22:10:35.668 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: [28/Mar/2024:22:10:35] ENGINE Serving on https://172.21.15.16:7150 2024-03-28T22:10:35.668 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-mon[35979]: [28/Mar/2024:22:10:35] ENGINE Bus STARTED 2024-03-28T22:10:35.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T22:10:35.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T22:10:35.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T22:10:35.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T22:10:35.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T22:10:35.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T22:10:35.688 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T22:10:35.688 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T22:10:35.688 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T22:10:35.688 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T22:10:35.688 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T22:10:35.688 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T22:10:35.688 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T22:10:35.688 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T22:10:35.688 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T22:10:35.688 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T22:10:35.688 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:10:35.688 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:10:35.688 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T22:10:35.688 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:10:35.689 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:10:35.689 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:10:35.689 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:10:35.689 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:10:35.689 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T22:10:35.689 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T22:10:35.689 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:10:35.689 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T22:10:35.689 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T22:10:35.689 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T22:10:35.689 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: Manager daemon x is now available 2024-03-28T22:10:35.689 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:10:35.689 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:10:35.689 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T22:10:35.690 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T22:10:35.690 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: [28/Mar/2024:22:10:34] ENGINE Bus STARTING 2024-03-28T22:10:35.690 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: [28/Mar/2024:22:10:34] ENGINE Serving on http://172.21.15.16:8765 2024-03-28T22:10:35.690 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: [28/Mar/2024:22:10:35] ENGINE Serving on https://172.21.15.16:7150 2024-03-28T22:10:35.690 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:35 smithi016 ceph-mon[32374]: [28/Mar/2024:22:10:35] ENGINE Bus STARTED 2024-03-28T22:10:35.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.754+0000 7fb054313640 1 -- 172.21.15.16:0/172756912 --> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7fb04c061c30 con 0x7fb0340a1b60 2024-03-28T22:10:35.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.810+0000 7fb04b7fe640 1 -- 172.21.15.16:0/172756912 <== mgr.15441 v2:172.21.15.16:6800/1819900910 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7fb04c061c30 con 0x7fb0340a1b60 2024-03-28T22:10:35.813 INFO:teuthology.orchestra.run.smithi016.stdout:Retention added to path / 2024-03-28T22:10:35.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.812+0000 7fb054313640 1 -- 172.21.15.16:0/172756912 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fb0340a1b60 msgr2=0x7fb0340a4020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:35.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.812+0000 7fb054313640 1 --2- 172.21.15.16:0/172756912 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fb0340a1b60 0x7fb0340a4020 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb044005c40 tx=0x7fb044005bd0 comp rx=0 tx=0).stop 2024-03-28T22:10:35.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.812+0000 7fb054313640 1 -- 172.21.15.16:0/172756912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb04c101870 msgr2=0x7fb04c19c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:35.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.812+0000 7fb054313640 1 --2- 172.21.15.16:0/172756912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb04c101870 0x7fb04c19c700 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fb04000c920 tx=0x7fb04000cde0 comp rx=0 tx=0).stop 2024-03-28T22:10:35.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.812+0000 7fb054313640 1 -- 172.21.15.16:0/172756912 shutdown_connections 2024-03-28T22:10:35.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.812+0000 7fb054313640 1 --2- 172.21.15.16:0/172756912 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fb0340a1b60 0x7fb0340a4020 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:35.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.812+0000 7fb054313640 1 --2- 172.21.15.16:0/172756912 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb04c102230 0x7fb04c19ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:35.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.812+0000 7fb054313640 1 --2- 172.21.15.16:0/172756912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb04c101870 0x7fb04c19c700 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:35.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.812+0000 7fb054313640 1 --2- 172.21.15.16:0/172756912 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb04c100680 0x7fb04c069ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:35.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.812+0000 7fb054313640 1 -- 172.21.15.16:0/172756912 >> 172.21.15.16:0/172756912 conn(0x7fb04c0fbbf0 msgr2=0x7fb04c104aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:35.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.813+0000 7fb054313640 1 -- 172.21.15.16:0/172756912 shutdown_connections 2024-03-28T22:10:35.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.813+0000 7fb054313640 1 -- 172.21.15.16:0/172756912 wait complete. 2024-03-28T22:10:35.826 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-28T22:10:35.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.929+0000 7f630f42a640 1 -- 172.21.15.16:0/2757705380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63081029a0 msgr2=0x7f6308104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:35.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.929+0000 7f630f42a640 1 --2- 172.21.15.16:0/2757705380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63081029a0 0x7f6308104e10 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f62fc009860 tx=0x7f62fc02bb70 comp rx=0 tx=0).stop 2024-03-28T22:10:35.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.929+0000 7f630f42a640 1 -- 172.21.15.16:0/2757705380 shutdown_connections 2024-03-28T22:10:35.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.929+0000 7f630f42a640 1 --2- 172.21.15.16:0/2757705380 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6308105350 0x7f63081057d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:35.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.929+0000 7f630f42a640 1 --2- 172.21.15.16:0/2757705380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63081029a0 0x7f6308104e10 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:35.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.929+0000 7f630f42a640 1 --2- 172.21.15.16:0/2757705380 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6308100030 0x7f6308102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:35.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.929+0000 7f630f42a640 1 -- 172.21.15.16:0/2757705380 >> 172.21.15.16:0/2757705380 conn(0x7f63080fbc30 msgr2=0x7f63080fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:35.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.929+0000 7f630f42a640 1 -- 172.21.15.16:0/2757705380 shutdown_connections 2024-03-28T22:10:35.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.929+0000 7f630f42a640 1 -- 172.21.15.16:0/2757705380 wait complete. 2024-03-28T22:10:35.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.930+0000 7f630f42a640 1 Processor -- start 2024-03-28T22:10:35.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.930+0000 7f630f42a640 1 -- start start 2024-03-28T22:10:35.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.930+0000 7f630f42a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6308100030 0x7f630819e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:35.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.930+0000 7f630f42a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f63081029a0 0x7f630819e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:35.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.930+0000 7f630f42a640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6308105350 0x7f63081a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:35.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.930+0000 7f630f42a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6308113a30 con 0x7f6308100030 2024-03-28T22:10:35.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.930+0000 7f630f42a640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f63081138b0 con 0x7f6308105350 2024-03-28T22:10:35.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.930+0000 7f630f42a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6308113bb0 con 0x7f63081029a0 2024-03-28T22:10:35.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.930+0000 7f630d1c6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6308100030 0x7f630819e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:35.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.930+0000 7f630d1c6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6308100030 0x7f630819e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59792/0 (socket says 172.21.15.16:59792) 2024-03-28T22:10:35.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.930+0000 7f630d1c6640 1 -- 172.21.15.16:0/1947897763 learned_addr learned my addr 172.21.15.16:0/1947897763 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:35.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.930+0000 7f630d9c7640 1 --2- 172.21.15.16:0/1947897763 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6308105350 0x7f63081a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:35.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.930+0000 7f630c9c5640 1 --2- 172.21.15.16:0/1947897763 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f63081029a0 0x7f630819e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:35.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.930+0000 7f630d1c6640 1 -- 172.21.15.16:0/1947897763 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f63081029a0 msgr2=0x7f630819e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:35.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.930+0000 7f630d1c6640 1 --2- 172.21.15.16:0/1947897763 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f63081029a0 0x7f630819e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:35.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.930+0000 7f630d1c6640 1 -- 172.21.15.16:0/1947897763 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6308105350 msgr2=0x7f63081a5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:35.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.930+0000 7f630d1c6640 1 --2- 172.21.15.16:0/1947897763 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6308105350 0x7f63081a5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:35.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.930+0000 7f630d1c6640 1 -- 172.21.15.16:0/1947897763 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63081a5c70 con 0x7f6308100030 2024-03-28T22:10:35.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.931+0000 7f630d1c6640 1 --2- 172.21.15.16:0/1947897763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6308100030 0x7f630819e050 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f630000c420 tx=0x7f6300009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:35.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.931+0000 7f63067fc640 1 -- 172.21.15.16:0/1947897763 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6300003970 con 0x7f6308100030 2024-03-28T22:10:35.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.931+0000 7f63067fc640 1 -- 172.21.15.16:0/1947897763 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6300004020 con 0x7f6308100030 2024-03-28T22:10:35.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.931+0000 7f630f42a640 1 -- 172.21.15.16:0/1947897763 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63081a5f60 con 0x7f6308100030 2024-03-28T22:10:35.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.931+0000 7f63067fc640 1 -- 172.21.15.16:0/1947897763 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f630000bad0 con 0x7f6308100030 2024-03-28T22:10:35.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.931+0000 7f630f42a640 1 -- 172.21.15.16:0/1947897763 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63081a63c0 con 0x7f6308100030 2024-03-28T22:10:35.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.932+0000 7f630f42a640 1 -- 172.21.15.16:0/1947897763 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63080fd720 con 0x7f6308100030 2024-03-28T22:10:35.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.933+0000 7f63067fc640 1 -- 172.21.15.16:0/1947897763 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) ==== 136568+0+0 (secure 0 0 0) 0x7f630001d020 con 0x7f6308100030 2024-03-28T22:10:35.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.933+0000 7f63067fc640 1 --2- 172.21.15.16:0/1947897763 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f62f00a1b60 0x7f62f00a4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:35.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.934+0000 7f630c9c5640 1 --2- 172.21.15.16:0/1947897763 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f62f00a1b60 0x7f62f00a4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:35.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.934+0000 7f63067fc640 1 -- 172.21.15.16:0/1947897763 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f63000cdf40 con 0x7f6308100030 2024-03-28T22:10:35.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.934+0000 7f630c9c5640 1 --2- 172.21.15.16:0/1947897763 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f62f00a1b60 0x7f62f00a4020 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f62fc02c180 tx=0x7f62fc034ad0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:35.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:35.937+0000 7f63067fc640 1 -- 172.21.15.16:0/1947897763 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f630009b520 con 0x7f6308100030 2024-03-28T22:10:36.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.154+0000 7f630f42a640 1 -- 172.21.15.16:0/1947897763 --> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f6308061b90 con 0x7f62f00a1b60 2024-03-28T22:10:36.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.162+0000 7f63067fc640 1 -- 172.21.15.16:0/1947897763 <== mgr.15441 v2:172.21.15.16:6800/1819900910 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f6308061b90 con 0x7f62f00a1b60 2024-03-28T22:10:36.164 INFO:teuthology.orchestra.run.smithi016.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-28T00:00:00", "created": "2024-03-28T22:10:35", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-03-28T22:10:36.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.164+0000 7f630f42a640 1 -- 172.21.15.16:0/1947897763 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f62f00a1b60 msgr2=0x7f62f00a4020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:36.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.164+0000 7f630f42a640 1 --2- 172.21.15.16:0/1947897763 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f62f00a1b60 0x7f62f00a4020 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f62fc02c180 tx=0x7f62fc034ad0 comp rx=0 tx=0).stop 2024-03-28T22:10:36.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.164+0000 7f630f42a640 1 -- 172.21.15.16:0/1947897763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6308100030 msgr2=0x7f630819e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:36.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.164+0000 7f630f42a640 1 --2- 172.21.15.16:0/1947897763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6308100030 0x7f630819e050 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f630000c420 tx=0x7f6300009f90 comp rx=0 tx=0).stop 2024-03-28T22:10:36.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.164+0000 7f630f42a640 1 -- 172.21.15.16:0/1947897763 shutdown_connections 2024-03-28T22:10:36.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.164+0000 7f630f42a640 1 --2- 172.21.15.16:0/1947897763 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f62f00a1b60 0x7f62f00a4020 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:36.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.164+0000 7f630f42a640 1 --2- 172.21.15.16:0/1947897763 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6308105350 0x7f63081a5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:36.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.164+0000 7f630f42a640 1 --2- 172.21.15.16:0/1947897763 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f63081029a0 0x7f630819e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:36.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.164+0000 7f630f42a640 1 --2- 172.21.15.16:0/1947897763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6308100030 0x7f630819e050 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:36.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.164+0000 7f630f42a640 1 -- 172.21.15.16:0/1947897763 >> 172.21.15.16:0/1947897763 conn(0x7f63080fbc30 msgr2=0x7f63081009b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:36.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.164+0000 7f630f42a640 1 -- 172.21.15.16:0/1947897763 shutdown_connections 2024-03-28T22:10:36.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.164+0000 7f630f42a640 1 -- 172.21.15.16:0/1947897763 wait complete. 2024-03-28T22:10:36.179 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-28T22:10:36.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.290+0000 7f19997c7640 1 -- 172.21.15.16:0/345557727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1994102260 msgr2=0x7f1994108af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:36.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.290+0000 7f19997c7640 1 --2- 172.21.15.16:0/345557727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1994102260 0x7f1994108af0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f198c009ca0 tx=0x7f198c02c0f0 comp rx=0 tx=0).stop 2024-03-28T22:10:36.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.290+0000 7f19997c7640 1 -- 172.21.15.16:0/345557727 shutdown_connections 2024-03-28T22:10:36.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.290+0000 7f19997c7640 1 --2- 172.21.15.16:0/345557727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1994102260 0x7f1994108af0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:36.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.290+0000 7f19997c7640 1 --2- 172.21.15.16:0/345557727 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19941018a0 0x7f1994101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:36.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.290+0000 7f19997c7640 1 --2- 172.21.15.16:0/345557727 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f19941006b0 0x7f1994100ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:36.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.290+0000 7f19997c7640 1 -- 172.21.15.16:0/345557727 >> 172.21.15.16:0/345557727 conn(0x7f19940fbc60 msgr2=0x7f19940fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:36.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.290+0000 7f19997c7640 1 -- 172.21.15.16:0/345557727 shutdown_connections 2024-03-28T22:10:36.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.290+0000 7f19997c7640 1 -- 172.21.15.16:0/345557727 wait complete. 2024-03-28T22:10:36.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.291+0000 7f19997c7640 1 Processor -- start 2024-03-28T22:10:36.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.291+0000 7f19997c7640 1 -- start start 2024-03-28T22:10:36.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.291+0000 7f19997c7640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f19941006b0 0x7f1994197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:36.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.291+0000 7f19997c7640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19941018a0 0x7f1994197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:36.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.291+0000 7f19997c7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1994102260 0x7f199419f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:36.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.291+0000 7f19997c7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f199410c330 con 0x7f1994102260 2024-03-28T22:10:36.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.291+0000 7f19997c7640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f199410c1b0 con 0x7f19941018a0 2024-03-28T22:10:36.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.291+0000 7f19997c7640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f199410c4b0 con 0x7f19941006b0 2024-03-28T22:10:36.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.291+0000 7f19937fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1994102260 0x7f199419f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:36.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.291+0000 7f1992ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f19941006b0 0x7f1994197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:36.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.291+0000 7f19937fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1994102260 0x7f199419f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59808/0 (socket says 172.21.15.16:59808) 2024-03-28T22:10:36.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.291+0000 7f19937fe640 1 -- 172.21.15.16:0/2819084471 learned_addr learned my addr 172.21.15.16:0/2819084471 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:36.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.291+0000 7f19927fc640 1 --2- 172.21.15.16:0/2819084471 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19941018a0 0x7f1994197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:36.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.292+0000 7f19937fe640 1 -- 172.21.15.16:0/2819084471 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f19941006b0 msgr2=0x7f1994197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:36.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.292+0000 7f19937fe640 1 --2- 172.21.15.16:0/2819084471 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f19941006b0 0x7f1994197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:36.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.292+0000 7f19937fe640 1 -- 172.21.15.16:0/2819084471 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19941018a0 msgr2=0x7f1994197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:36.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.292+0000 7f19937fe640 1 --2- 172.21.15.16:0/2819084471 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19941018a0 0x7f1994197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:36.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.292+0000 7f19937fe640 1 -- 172.21.15.16:0/2819084471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f199419f6c0 con 0x7f1994102260 2024-03-28T22:10:36.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.292+0000 7f1992ffd640 1 --2- 172.21.15.16:0/2819084471 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f19941006b0 0x7f1994197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:10:36.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.292+0000 7f19937fe640 1 --2- 172.21.15.16:0/2819084471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1994102260 0x7f199419f050 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f198c037040 tx=0x7f198c032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:36.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.292+0000 7f198bfff640 1 -- 172.21.15.16:0/2819084471 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f198c041020 con 0x7f1994102260 2024-03-28T22:10:36.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.292+0000 7f198bfff640 1 -- 172.21.15.16:0/2819084471 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f198c034cd0 con 0x7f1994102260 2024-03-28T22:10:36.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.292+0000 7f198bfff640 1 -- 172.21.15.16:0/2819084471 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f198c039040 con 0x7f1994102260 2024-03-28T22:10:36.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.292+0000 7f19997c7640 1 -- 172.21.15.16:0/2819084471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f199419f950 con 0x7f1994102260 2024-03-28T22:10:36.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.293+0000 7f19997c7640 1 -- 172.21.15.16:0/2819084471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f199419fe90 con 0x7f1994102260 2024-03-28T22:10:36.298 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:10:35 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T22:10:35.921+0000 7f94a9e33200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T22:10:36.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.294+0000 7f19997c7640 1 -- 172.21.15.16:0/2819084471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19941a0180 con 0x7f1994102260 2024-03-28T22:10:36.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.295+0000 7f198bfff640 1 -- 172.21.15.16:0/2819084471 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) ==== 136568+0+0 (secure 0 0 0) 0x7f198c044020 con 0x7f1994102260 2024-03-28T22:10:36.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.295+0000 7f198bfff640 1 --2- 172.21.15.16:0/2819084471 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f19800a1be0 0x7f19800a40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:36.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.295+0000 7f1992ffd640 1 --2- 172.21.15.16:0/2819084471 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f19800a1be0 0x7f19800a40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:36.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.295+0000 7f198bfff640 1 -- 172.21.15.16:0/2819084471 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f198c0ef830 con 0x7f1994102260 2024-03-28T22:10:36.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.296+0000 7f1992ffd640 1 --2- 172.21.15.16:0/2819084471 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f19800a1be0 0x7f19800a40a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1994101700 tx=0x7f1984005f90 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:36.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.300+0000 7f198bfff640 1 -- 172.21.15.16:0/2819084471 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f198c04b020 con 0x7f1994102260 2024-03-28T22:10:36.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.348+0000 7f198bfff640 1 -- 172.21.15.16:0/2819084471 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 23) ==== 136876+0+0 (secure 0 0 0) 0x7f198c0b8e50 con 0x7f1994102260 2024-03-28T22:10:36.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:36 smithi016 ceph-mon[32374]: mgrmap e22: x(active, since 1.04261s), standbys: y, z 2024-03-28T22:10:36.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:36 smithi016 ceph-mon[32374]: from='client.15459 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:10:36.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:36 smithi016 ceph-mon[32374]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:10:36.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:36 smithi016 ceph-mon[32374]: from='client.15501 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:10:36.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:36 smithi016 ceph-mon[32374]: Standby manager daemon y restarted 2024-03-28T22:10:36.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:36 smithi016 ceph-mon[32374]: Standby manager daemon y started 2024-03-28T22:10:36.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:36 smithi016 ceph-mon[32374]: from='mgr.? 172.21.15.53:0/2940232469' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T22:10:36.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:36 smithi016 ceph-mon[32374]: from='mgr.? 172.21.15.53:0/2940232469' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T22:10:36.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:36 smithi016 ceph-mon[32374]: from='mgr.? 172.21.15.53:0/2940232469' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T22:10:36.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:36 smithi016 ceph-mon[32374]: from='mgr.? 172.21.15.53:0/2940232469' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T22:10:36.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:36 smithi016 ceph-mon[32374]: from='client.15507 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:10:36.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.530+0000 7f19997c7640 1 -- 172.21.15.16:0/2819084471 --> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f19940fdda0 con 0x7f19800a1be0 2024-03-28T22:10:36.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.537+0000 7f198bfff640 1 -- 172.21.15.16:0/2819084471 <== mgr.15441 v2:172.21.15.16:6800/1819900910 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f19940fdda0 con 0x7f19800a1be0 2024-03-28T22:10:36.540 INFO:teuthology.orchestra.run.smithi016.stdout:/ 1m 6m3h 2024-03-28T22:10:36.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.540+0000 7f19997c7640 1 -- 172.21.15.16:0/2819084471 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f19800a1be0 msgr2=0x7f19800a40a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:36.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.540+0000 7f19997c7640 1 --2- 172.21.15.16:0/2819084471 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f19800a1be0 0x7f19800a40a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1994101700 tx=0x7f1984005f90 comp rx=0 tx=0).stop 2024-03-28T22:10:36.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.540+0000 7f19997c7640 1 -- 172.21.15.16:0/2819084471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1994102260 msgr2=0x7f199419f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:36.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.540+0000 7f19997c7640 1 --2- 172.21.15.16:0/2819084471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1994102260 0x7f199419f050 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f198c037040 tx=0x7f198c032f40 comp rx=0 tx=0).stop 2024-03-28T22:10:36.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.540+0000 7f19997c7640 1 -- 172.21.15.16:0/2819084471 shutdown_connections 2024-03-28T22:10:36.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.540+0000 7f19997c7640 1 --2- 172.21.15.16:0/2819084471 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f19800a1be0 0x7f19800a40a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:36.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.540+0000 7f19997c7640 1 --2- 172.21.15.16:0/2819084471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1994102260 0x7f199419f050 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:36.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.540+0000 7f19997c7640 1 --2- 172.21.15.16:0/2819084471 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19941018a0 0x7f1994197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:36.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.540+0000 7f19997c7640 1 --2- 172.21.15.16:0/2819084471 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f19941006b0 0x7f1994197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:36.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.540+0000 7f19997c7640 1 -- 172.21.15.16:0/2819084471 >> 172.21.15.16:0/2819084471 conn(0x7f19940fbc60 msgr2=0x7f19940fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:36.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.540+0000 7f19997c7640 1 -- 172.21.15.16:0/2819084471 shutdown_connections 2024-03-28T22:10:36.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:36.540+0000 7f19997c7640 1 -- 172.21.15.16:0/2819084471 wait complete. 2024-03-28T22:10:36.554 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-03-28T22:10:36.582 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-03-28T22:10:36.594 INFO:teuthology.run_tasks:Running task workunit... 2024-03-28T22:10:36.611 INFO:tasks.workunit:Pulling workunits from ref 55f5138942aa4866e49329c9103468889cec3f73 2024-03-28T22:10:36.611 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-28T22:10:36.612 INFO:tasks.workunit:timeout=3h 2024-03-28T22:10:36.612 INFO:tasks.workunit:cleanup=True 2024-03-28T22:10:36.612 DEBUG:teuthology.orchestra.run.smithi016:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-28T22:10:36.637 INFO:teuthology.orchestra.run.smithi016.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-28T22:10:36.637 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-28T22:10:36.638 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 77h/119d Inode: 1099511628283 Links: 2 2024-03-28T22:10:36.638 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T22:10:36.638 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-28T22:10:36.638 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-03-28 22:08:35.776548732 +0000 2024-03-28T22:10:36.638 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-03-28 22:08:35.776548732 +0000 2024-03-28T22:10:36.638 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-03-28 22:10:19.270032293 +0000 2024-03-28T22:10:36.638 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-03-28 22:08:35.776548732 +0000 2024-03-28T22:10:36.638 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-28T22:10:36.638 DEBUG:teuthology.orchestra.run.smithi016:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-28T22:10:36.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:36 smithi063 ceph-mon[36249]: mgrmap e22: x(active, since 1.04261s), standbys: y, z 2024-03-28T22:10:36.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:36 smithi063 ceph-mon[36249]: from='client.15459 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:10:36.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:36 smithi063 ceph-mon[36249]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:10:36.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:36 smithi063 ceph-mon[36249]: from='client.15501 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:10:36.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:36 smithi063 ceph-mon[36249]: Standby manager daemon y restarted 2024-03-28T22:10:36.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:36 smithi063 ceph-mon[36249]: Standby manager daemon y started 2024-03-28T22:10:36.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:36 smithi063 ceph-mon[36249]: from='mgr.? 172.21.15.53:0/2940232469' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T22:10:36.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:36 smithi063 ceph-mon[36249]: from='mgr.? 172.21.15.53:0/2940232469' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T22:10:36.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:36 smithi063 ceph-mon[36249]: from='mgr.? 172.21.15.53:0/2940232469' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T22:10:36.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:36 smithi063 ceph-mon[36249]: from='mgr.? 172.21.15.53:0/2940232469' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T22:10:36.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:36 smithi063 ceph-mon[36249]: from='client.15507 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:10:36.713 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 55f5138942aa4866e49329c9103468889cec3f73 2024-03-28T22:10:36.767 INFO:tasks.workunit.client.0.smithi016.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-28T22:10:36.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:36 smithi053 ceph-mon[35979]: mgrmap e22: x(active, since 1.04261s), standbys: y, z 2024-03-28T22:10:36.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:36 smithi053 ceph-mon[35979]: from='client.15459 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:10:36.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:36 smithi053 ceph-mon[35979]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:10:36.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:36 smithi053 ceph-mon[35979]: from='client.15501 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:10:36.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:36 smithi053 ceph-mon[35979]: Standby manager daemon y restarted 2024-03-28T22:10:36.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:36 smithi053 ceph-mon[35979]: Standby manager daemon y started 2024-03-28T22:10:36.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:36 smithi053 ceph-mon[35979]: from='mgr.? 172.21.15.53:0/2940232469' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T22:10:36.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:36 smithi053 ceph-mon[35979]: from='mgr.? 172.21.15.53:0/2940232469' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T22:10:36.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:36 smithi053 ceph-mon[35979]: from='mgr.? 172.21.15.53:0/2940232469' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T22:10:36.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:36 smithi053 ceph-mon[35979]: from='mgr.? 172.21.15.53:0/2940232469' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T22:10:36.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:36 smithi053 ceph-mon[35979]: from='client.15507 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:10:37.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:37 smithi063 ceph-mon[36249]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:10:37.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:37 smithi063 ceph-mon[36249]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-03-28T22:10:37.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:37 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:37.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:37 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:37.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:37 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:37.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:37 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:37.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:37 smithi063 ceph-mon[36249]: from='client.15513 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:10:37.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:37 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:37.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:37 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:37.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:37 smithi016 ceph-mon[32374]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:10:37.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:37 smithi016 ceph-mon[32374]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-03-28T22:10:37.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:37 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:37.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:37 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:37.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:37 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:37.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:37 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:37.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:37 smithi016 ceph-mon[32374]: from='client.15513 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:10:37.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:37 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:37.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:37 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:37.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:37 smithi053 ceph-mon[35979]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:10:37.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:37 smithi053 ceph-mon[35979]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-03-28T22:10:37.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:37 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:37.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:37 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:37.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:37 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:37.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:37 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:37.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:37 smithi053 ceph-mon[35979]: from='client.15513 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:10:37.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:37 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:37.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:37 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:38.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:38 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:38.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:38 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:38.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:38 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:38.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:38 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:10:38.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:38 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:38.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:38 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:10:38.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:38 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:10:38.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:38 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:10:38.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:38 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:10:38.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:38 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:10:38.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:38 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:10:38.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:38 smithi063 ceph-mon[36249]: Adjusting osd_memory_target on smithi063 to 174.1M 2024-03-28T22:10:38.934 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:38 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:10:38.934 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:38 smithi063 ceph-mon[36249]: Unable to set osd_memory_target on smithi063 to 182642073: error parsing value: Value '182642073' is below minimum 939524096 2024-03-28T22:10:38.934 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:38 smithi063 ceph-mon[36249]: Adjusting osd_memory_target on smithi016 to 177.2M 2024-03-28T22:10:38.934 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:38 smithi063 ceph-mon[36249]: Unable to set osd_memory_target on smithi016 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-03-28T22:10:38.934 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:38 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:38.934 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:38 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:38.934 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:38 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:10:38.934 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:38 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:10:38.934 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:38 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:10:38.934 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:38 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:10:38.934 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:38 smithi063 ceph-mon[36249]: Adjusting osd_memory_target on smithi053 to 175.2M 2024-03-28T22:10:38.934 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:38 smithi063 ceph-mon[36249]: Unable to set osd_memory_target on smithi053 to 183743795: error parsing value: Value '183743795' is below minimum 939524096 2024-03-28T22:10:38.934 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:38 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:10:38.935 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:38 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:10:38.935 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:38 smithi063 ceph-mon[36249]: Updating smithi016:/etc/ceph/ceph.conf 2024-03-28T22:10:38.935 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:38 smithi063 ceph-mon[36249]: Updating smithi053:/etc/ceph/ceph.conf 2024-03-28T22:10:38.935 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:38 smithi063 ceph-mon[36249]: Updating smithi063:/etc/ceph/ceph.conf 2024-03-28T22:10:38.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:38 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:38.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:38 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:38.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:38 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:38.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:38 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:10:38.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:38 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:38.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:38 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:10:38.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:38 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:10:38.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:38 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:10:38.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:38 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:10:38.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:38 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:10:38.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:38 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:10:38.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:38 smithi016 ceph-mon[32374]: Adjusting osd_memory_target on smithi063 to 174.1M 2024-03-28T22:10:38.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:38 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:10:38.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:38 smithi016 ceph-mon[32374]: Unable to set osd_memory_target on smithi063 to 182642073: error parsing value: Value '182642073' is below minimum 939524096 2024-03-28T22:10:38.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:38 smithi016 ceph-mon[32374]: Adjusting osd_memory_target on smithi016 to 177.2M 2024-03-28T22:10:38.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:38 smithi016 ceph-mon[32374]: Unable to set osd_memory_target on smithi016 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-03-28T22:10:38.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:38 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:38.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:38 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:38.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:38 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:10:38.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:38 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:10:38.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:38 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:10:38.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:38 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:10:38.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:38 smithi016 ceph-mon[32374]: Adjusting osd_memory_target on smithi053 to 175.2M 2024-03-28T22:10:38.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:38 smithi016 ceph-mon[32374]: Unable to set osd_memory_target on smithi053 to 183743795: error parsing value: Value '183743795' is below minimum 939524096 2024-03-28T22:10:38.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:38 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:10:38.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:38 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:10:38.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:38 smithi016 ceph-mon[32374]: Updating smithi016:/etc/ceph/ceph.conf 2024-03-28T22:10:38.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:38 smithi016 ceph-mon[32374]: Updating smithi053:/etc/ceph/ceph.conf 2024-03-28T22:10:38.939 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:38 smithi016 ceph-mon[32374]: Updating smithi063:/etc/ceph/ceph.conf 2024-03-28T22:10:39.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:38 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:39.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:38 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:39.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:38 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:39.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:38 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:10:39.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:38 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:39.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:38 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:10:39.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:38 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:10:39.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:38 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:10:39.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:38 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:10:39.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:38 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:10:39.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:38 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:10:39.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:38 smithi053 ceph-mon[35979]: Adjusting osd_memory_target on smithi063 to 174.1M 2024-03-28T22:10:39.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:38 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:10:39.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:38 smithi053 ceph-mon[35979]: Unable to set osd_memory_target on smithi063 to 182642073: error parsing value: Value '182642073' is below minimum 939524096 2024-03-28T22:10:39.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:38 smithi053 ceph-mon[35979]: Adjusting osd_memory_target on smithi016 to 177.2M 2024-03-28T22:10:39.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:38 smithi053 ceph-mon[35979]: Unable to set osd_memory_target on smithi016 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-03-28T22:10:39.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:38 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:39.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:38 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:39.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:38 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:10:39.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:38 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:10:39.050 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:38 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:10:39.050 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:38 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:10:39.050 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:38 smithi053 ceph-mon[35979]: Adjusting osd_memory_target on smithi053 to 175.2M 2024-03-28T22:10:39.050 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:38 smithi053 ceph-mon[35979]: Unable to set osd_memory_target on smithi053 to 183743795: error parsing value: Value '183743795' is below minimum 939524096 2024-03-28T22:10:39.050 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:38 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:10:39.050 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:38 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:10:39.050 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:38 smithi053 ceph-mon[35979]: Updating smithi016:/etc/ceph/ceph.conf 2024-03-28T22:10:39.050 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:38 smithi053 ceph-mon[35979]: Updating smithi053:/etc/ceph/ceph.conf 2024-03-28T22:10:39.050 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:38 smithi053 ceph-mon[35979]: Updating smithi063:/etc/ceph/ceph.conf 2024-03-28T22:10:40.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:40 smithi016 ceph-mon[32374]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:10:40.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:40 smithi016 ceph-mon[32374]: Updating smithi016:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:10:40.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:40 smithi016 ceph-mon[32374]: Updating smithi063:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:10:40.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:40 smithi016 ceph-mon[32374]: Updating smithi053:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:10:40.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:40 smithi016 ceph-mon[32374]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-03-28T22:10:40.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:40 smithi016 ceph-mon[32374]: Updating smithi063:/etc/ceph/ceph.client.admin.keyring 2024-03-28T22:10:40.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:40 smithi016 ceph-mon[32374]: Updating smithi053:/etc/ceph/ceph.client.admin.keyring 2024-03-28T22:10:40.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:40 smithi016 ceph-mon[32374]: Updating smithi016:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T22:10:40.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:40 smithi016 ceph-mon[32374]: Updating smithi063:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T22:10:40.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:40 smithi016 ceph-mon[32374]: Updating smithi053:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T22:10:40.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:40 smithi016 ceph-mon[32374]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-03-28T22:10:40.438 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:40 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:40.438 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:40 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:40.438 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:40 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:40.438 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:40 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:40.438 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:40 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:40.438 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:40 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:40.438 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:40 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:40.438 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:40 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:10:40.438 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:40 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:10:40.438 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:40 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:10:40.438 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:40 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:40.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:40 smithi053 ceph-mon[35979]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:10:40.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:40 smithi053 ceph-mon[35979]: Updating smithi016:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:10:40.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:40 smithi053 ceph-mon[35979]: Updating smithi063:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:10:40.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:40 smithi053 ceph-mon[35979]: Updating smithi053:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:10:40.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:40 smithi053 ceph-mon[35979]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-03-28T22:10:40.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:40 smithi053 ceph-mon[35979]: Updating smithi063:/etc/ceph/ceph.client.admin.keyring 2024-03-28T22:10:40.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:40 smithi053 ceph-mon[35979]: Updating smithi053:/etc/ceph/ceph.client.admin.keyring 2024-03-28T22:10:40.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:40 smithi053 ceph-mon[35979]: Updating smithi016:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T22:10:40.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:40 smithi053 ceph-mon[35979]: Updating smithi063:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T22:10:40.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:40 smithi053 ceph-mon[35979]: Updating smithi053:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T22:10:40.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:40 smithi053 ceph-mon[35979]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-03-28T22:10:40.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:40 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:40.549 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:40 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:40.549 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:40 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:40.549 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:40 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:40.549 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:40 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:40.549 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:40 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:40.549 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:40 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:40.549 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:40 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:10:40.549 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:40 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:10:40.549 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:40 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:10:40.549 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:40 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:40.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:40 smithi063 ceph-mon[36249]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:10:40.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:40 smithi063 ceph-mon[36249]: Updating smithi016:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:10:40.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:40 smithi063 ceph-mon[36249]: Updating smithi063:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:10:40.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:40 smithi063 ceph-mon[36249]: Updating smithi053:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:10:40.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:40 smithi063 ceph-mon[36249]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-03-28T22:10:40.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:40 smithi063 ceph-mon[36249]: Updating smithi063:/etc/ceph/ceph.client.admin.keyring 2024-03-28T22:10:40.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:40 smithi063 ceph-mon[36249]: Updating smithi053:/etc/ceph/ceph.client.admin.keyring 2024-03-28T22:10:40.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:40 smithi063 ceph-mon[36249]: Updating smithi016:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T22:10:40.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:40 smithi063 ceph-mon[36249]: Updating smithi063:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T22:10:40.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:40 smithi063 ceph-mon[36249]: Updating smithi053:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T22:10:40.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:40 smithi063 ceph-mon[36249]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-03-28T22:10:40.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:40 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:40.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:40 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:40.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:40 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:40.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:40 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:40.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:40 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:40.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:40 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:40.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:40 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:40.684 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:40 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:10:40.684 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:40 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:10:40.684 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:40 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:10:40.684 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:40 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:10:42.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:42 smithi053 ceph-mon[35979]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:10:42.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:42 smithi063 ceph-mon[36249]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:10:42.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:42 smithi016 ceph-mon[32374]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:10:44.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:44 smithi053 ceph-mon[35979]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 133 KiB/s wr, 77 op/s 2024-03-28T22:10:44.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:44 smithi063 ceph-mon[36249]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 133 KiB/s wr, 77 op/s 2024-03-28T22:10:44.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:44 smithi016 ceph-mon[32374]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 133 KiB/s wr, 77 op/s 2024-03-28T22:10:46.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:46 smithi053 ceph-mon[35979]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 103 KiB/s wr, 60 op/s 2024-03-28T22:10:46.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:46 smithi063 ceph-mon[36249]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 103 KiB/s wr, 60 op/s 2024-03-28T22:10:46.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:46 smithi016 ceph-mon[32374]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 103 KiB/s wr, 60 op/s 2024-03-28T22:10:48.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:48 smithi053 ceph-mon[35979]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 52 op/s 2024-03-28T22:10:48.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:48 smithi063 ceph-mon[36249]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 52 op/s 2024-03-28T22:10:48.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:48 smithi016 ceph-mon[32374]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 52 op/s 2024-03-28T22:10:50.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:50 smithi053 ceph-mon[35979]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-28T22:10:50.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:50 smithi063 ceph-mon[36249]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-28T22:10:50.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:50 smithi016 ceph-mon[32374]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-28T22:10:52.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:52 smithi063 ceph-mon[36249]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-28T22:10:52.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:52 smithi016 ceph-mon[32374]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-28T22:10:52.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:52 smithi053 ceph-mon[35979]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-28T22:10:54.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:54 smithi053 ceph-mon[35979]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-03-28T22:10:54.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:54 smithi063 ceph-mon[36249]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-03-28T22:10:54.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:54 smithi016 ceph-mon[32374]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-03-28T22:10:56.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:56 smithi053 ceph-mon[35979]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:10:56.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:56 smithi063 ceph-mon[36249]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:10:56.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:56 smithi016 ceph-mon[32374]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:10:56.719 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:10:56.719 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:10:56.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.838+0000 7f4a66348640 1 -- 172.21.15.16:0/2202127281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a6010bd40 msgr2=0x7f4a6010e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:56.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.838+0000 7f4a66348640 1 --2- 172.21.15.16:0/2202127281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a6010bd40 0x7f4a6010e170 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f4a58009ca0 tx=0x7f4a5802c180 comp rx=0 tx=0).stop 2024-03-28T22:10:56.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.838+0000 7f4a66348640 1 -- 172.21.15.16:0/2202127281 shutdown_connections 2024-03-28T22:10:56.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.838+0000 7f4a66348640 1 --2- 172.21.15.16:0/2202127281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a6010bd40 0x7f4a6010e170 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:56.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.838+0000 7f4a66348640 1 --2- 172.21.15.16:0/2202127281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a60103290 0x7f4a6010b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:56.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.838+0000 7f4a66348640 1 --2- 172.21.15.16:0/2202127281 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a60102970 0x7f4a60102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:56.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.838+0000 7f4a66348640 1 -- 172.21.15.16:0/2202127281 >> 172.21.15.16:0/2202127281 conn(0x7f4a60077770 msgr2=0x7f4a60077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:56.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.838+0000 7f4a66348640 1 -- 172.21.15.16:0/2202127281 shutdown_connections 2024-03-28T22:10:56.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.838+0000 7f4a66348640 1 -- 172.21.15.16:0/2202127281 wait complete. 2024-03-28T22:10:56.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.838+0000 7f4a66348640 1 Processor -- start 2024-03-28T22:10:56.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.838+0000 7f4a66348640 1 -- start start 2024-03-28T22:10:56.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.839+0000 7f4a66348640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a60102970 0x7f4a60198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:56.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.839+0000 7f4a66348640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a60103290 0x7f4a60198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:56.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.839+0000 7f4a66348640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a6010bd40 0x7f4a6019c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:56.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.839+0000 7f4a66348640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4a60111b10 con 0x7f4a60102970 2024-03-28T22:10:56.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.839+0000 7f4a66348640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4a60111990 con 0x7f4a6010bd40 2024-03-28T22:10:56.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.839+0000 7f4a66348640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4a60111c90 con 0x7f4a60103290 2024-03-28T22:10:56.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.839+0000 7f4a5ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a60102970 0x7f4a60198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:56.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.839+0000 7f4a5ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a60102970 0x7f4a60198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37504/0 (socket says 172.21.15.16:37504) 2024-03-28T22:10:56.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.839+0000 7f4a5ffff640 1 -- 172.21.15.16:0/302463495 learned_addr learned my addr 172.21.15.16:0/302463495 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:56.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.839+0000 7f4a648e5640 1 --2- 172.21.15.16:0/302463495 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a6010bd40 0x7f4a6019c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:56.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.839+0000 7f4a5ffff640 1 -- 172.21.15.16:0/302463495 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a60103290 msgr2=0x7f4a60198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:56.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.839+0000 7f4a5ffff640 1 --2- 172.21.15.16:0/302463495 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a60103290 0x7f4a60198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:56.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.839+0000 7f4a5ffff640 1 -- 172.21.15.16:0/302463495 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a6010bd40 msgr2=0x7f4a6019c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:56.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.839+0000 7f4a5ffff640 1 --2- 172.21.15.16:0/302463495 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a6010bd40 0x7f4a6019c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:56.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.839+0000 7f4a5ffff640 1 -- 172.21.15.16:0/302463495 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a6019d020 con 0x7f4a60102970 2024-03-28T22:10:56.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.839+0000 7f4a5ffff640 1 --2- 172.21.15.16:0/302463495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a60102970 0x7f4a60198030 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4a4800b4a0 tx=0x7f4a4800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:56.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.839+0000 7f4a5d7fa640 1 -- 172.21.15.16:0/302463495 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a48014070 con 0x7f4a60102970 2024-03-28T22:10:56.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.840+0000 7f4a66348640 1 -- 172.21.15.16:0/302463495 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a6019d310 con 0x7f4a60102970 2024-03-28T22:10:56.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.840+0000 7f4a5d7fa640 1 -- 172.21.15.16:0/302463495 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a48010af0 con 0x7f4a60102970 2024-03-28T22:10:56.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.840+0000 7f4a5d7fa640 1 -- 172.21.15.16:0/302463495 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a48017d10 con 0x7f4a60102970 2024-03-28T22:10:56.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.840+0000 7f4a66348640 1 -- 172.21.15.16:0/302463495 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a601a4ba0 con 0x7f4a60102970 2024-03-28T22:10:56.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.842+0000 7f4a5d7fa640 1 -- 172.21.15.16:0/302463495 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 136932+0+0 (secure 0 0 0) 0x7f4a48017070 con 0x7f4a60102970 2024-03-28T22:10:56.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.842+0000 7f4a69009740 1 Processor -- start 2024-03-28T22:10:56.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.842+0000 7f4a69009740 1 -- start start 2024-03-28T22:10:56.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.842+0000 7f4a69009740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c5dca33290 0x55c5dca33670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:56.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.842+0000 7f4a69009740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5dca33bb0 0x55c5dca3c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:56.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.842+0000 7f4a69009740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c5dca3c790 0x55c5dca3eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:56.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.842+0000 7f4a69009740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c5dc997180 con 0x55c5dca33bb0 2024-03-28T22:10:56.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.842+0000 7f4a69009740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55c5dc93fb70 con 0x55c5dca33290 2024-03-28T22:10:56.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.842+0000 7f4a69009740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55c5dc993220 con 0x55c5dca3c790 2024-03-28T22:10:56.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.842+0000 7f4a5ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5dca33bb0 0x55c5dca3c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:56.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.842+0000 7f4a648e5640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c5dca33290 0x55c5dca33670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:56.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.843+0000 7f4a648e5640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c5dca33290 0x55c5dca33670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:47960/0 (socket says 172.21.15.16:47960) 2024-03-28T22:10:56.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.843+0000 7f4a5ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5dca33bb0 0x55c5dca3c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37516/0 (socket says 172.21.15.16:37516) 2024-03-28T22:10:56.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.843+0000 7f4a648e5640 1 -- 172.21.15.16:0/85278536 learned_addr learned my addr 172.21.15.16:0/85278536 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:56.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.843+0000 7f4a5d7fa640 1 --2- 172.21.15.16:0/302463495 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4a4c0a1f80 0x7f4a4c0a4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:56.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.843+0000 7f4a5f7fe640 1 --2- 172.21.15.16:0/85278536 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c5dca3c790 0x55c5dca3eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:56.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.843+0000 7f4a648e5640 1 -- 172.21.15.16:0/85278536 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c5dca3c790 msgr2=0x55c5dca3eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:56.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.843+0000 7f4a648e5640 1 --2- 172.21.15.16:0/85278536 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c5dca3c790 0x55c5dca3eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:56.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.843+0000 7f4a648e5640 1 -- 172.21.15.16:0/85278536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5dca33bb0 msgr2=0x55c5dca3c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:56.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.843+0000 7f4a648e5640 1 --2- 172.21.15.16:0/85278536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5dca33bb0 0x55c5dca3c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:56.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.843+0000 7f4a648e5640 1 -- 172.21.15.16:0/85278536 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5dc9537c0 con 0x55c5dca33290 2024-03-28T22:10:56.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.843+0000 7f4a5d7fa640 1 -- 172.21.15.16:0/302463495 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f4a480cd630 con 0x7f4a60102970 2024-03-28T22:10:56.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.843+0000 7f4a648e5640 1 --2- 172.21.15.16:0/302463495 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4a4c0a1f80 0x7f4a4c0a4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:56.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.843+0000 7f4a648e5640 1 --2- 172.21.15.16:0/85278536 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c5dca33290 0x55c5dca33670 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4a6006aed0 tx=0x7f4a58009980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d59b4d523778fcf2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:56.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.843+0000 7f4a567fc640 1 -- 172.21.15.16:0/85278536 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a58039040 con 0x55c5dca33290 2024-03-28T22:10:56.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.843+0000 7f4a648e5640 1 --2- 172.21.15.16:0/302463495 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4a4c0a1f80 0x7f4a4c0a4440 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4a5804c000 tx=0x7f4a58045000 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:56.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.843+0000 7f4a567fc640 1 -- 172.21.15.16:0/85278536 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a58032670 con 0x55c5dca33290 2024-03-28T22:10:56.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.843+0000 7f4a567fc640 1 -- 172.21.15.16:0/85278536 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a5804a640 con 0x55c5dca33290 2024-03-28T22:10:56.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.844+0000 7f4a69009740 1 -- 172.21.15.16:0/85278536 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c5dca33290 msgr2=0x55c5dca33670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:56.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.844+0000 7f4a69009740 1 --2- 172.21.15.16:0/85278536 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c5dca33290 0x55c5dca33670 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4a6006aed0 tx=0x7f4a58009980 comp rx=0 tx=0).stop 2024-03-28T22:10:56.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.844+0000 7f4a69009740 1 -- 172.21.15.16:0/85278536 shutdown_connections 2024-03-28T22:10:56.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.844+0000 7f4a69009740 1 --2- 172.21.15.16:0/85278536 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c5dca3c790 0x55c5dca3eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:56.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.844+0000 7f4a69009740 1 --2- 172.21.15.16:0/85278536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5dca33bb0 0x55c5dca3c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:56.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.844+0000 7f4a69009740 1 --2- 172.21.15.16:0/85278536 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c5dca33290 0x55c5dca33670 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:56.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.844+0000 7f4a69009740 1 -- 172.21.15.16:0/85278536 >> 172.21.15.16:0/85278536 conn(0x55c5dca2d9c0 msgr2=0x55c5dca2dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:56.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.844+0000 7f4a69009740 1 -- 172.21.15.16:0/85278536 shutdown_connections 2024-03-28T22:10:56.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.844+0000 7f4a69009740 1 -- 172.21.15.16:0/85278536 wait complete. 2024-03-28T22:10:56.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.844+0000 7f4a69009740 1 -- start start 2024-03-28T22:10:56.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.845+0000 7f4a69009740 1 Processor -- start 2024-03-28T22:10:56.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.845+0000 7f4a69009740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c5dca33290 0x55c5dca39790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:56.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.845+0000 7f4a69009740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5dca33bb0 0x55c5dca39cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:56.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.845+0000 7f4a69009740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c5dca3c790 0x55c5dca3a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:56.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.845+0000 7f4a69009740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c5dca3f580 con 0x55c5dca33bb0 2024-03-28T22:10:56.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.845+0000 7f4a69009740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55c5dc9354a0 con 0x55c5dca33290 2024-03-28T22:10:56.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.845+0000 7f4a69009740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55c5dca3f700 con 0x55c5dca3c790 2024-03-28T22:10:56.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.845+0000 7f4a69009740 10 client.0 _ll_register_callbacks cb 0x55c5dc938d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:10:56.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.845+0000 7f4a5f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5dca33bb0 0x55c5dca39cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:56.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.845+0000 7f4a5ffff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c5dca33290 0x55c5dca39790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:56.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.845+0000 7f4a5f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5dca33bb0 0x55c5dca39cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37528/0 (socket says 172.21.15.16:37528) 2024-03-28T22:10:56.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.845+0000 7f4a5ffff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c5dca33290 0x55c5dca39790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:47972/0 (socket says 172.21.15.16:47972) 2024-03-28T22:10:56.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.845+0000 7f4a5f7fe640 1 -- 172.21.15.16:0/245988927 learned_addr learned my addr 172.21.15.16:0/245988927 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:56.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.845+0000 7f4a648e5640 1 --2- 172.21.15.16:0/245988927 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c5dca3c790 0x55c5dca3a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:56.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.845+0000 7f4a5f7fe640 1 -- 172.21.15.16:0/245988927 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c5dca3c790 msgr2=0x55c5dca3a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:56.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.845+0000 7f4a5f7fe640 1 --2- 172.21.15.16:0/245988927 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c5dca3c790 0x55c5dca3a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:56.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.845+0000 7f4a5f7fe640 1 -- 172.21.15.16:0/245988927 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c5dca33290 msgr2=0x55c5dca39790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:56.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.845+0000 7f4a5f7fe640 1 --2- 172.21.15.16:0/245988927 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c5dca33290 0x55c5dca39790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:56.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.845+0000 7f4a5f7fe640 1 -- 172.21.15.16:0/245988927 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5dc7757d0 con 0x55c5dca33bb0 2024-03-28T22:10:56.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.845+0000 7f4a5ffff640 1 --2- 172.21.15.16:0/245988927 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c5dca33290 0x55c5dca39790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:10:56.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.846+0000 7f4a5f7fe640 1 --2- 172.21.15.16:0/245988927 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5dca33bb0 0x55c5dca39cd0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f4a50011480 tx=0x7f4a50011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e469f7f0408ca7a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:56.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.846+0000 7f4a3f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:10:56.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.846+0000 7f4a3f7fe640 1 -- 172.21.15.16:0/245988927 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a50012980 con 0x55c5dca33bb0 2024-03-28T22:10:56.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.846+0000 7f4a3f7fe640 1 -- 172.21.15.16:0/245988927 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f4a50012b20 con 0x55c5dca33bb0 2024-03-28T22:10:56.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.846+0000 7f4a69009740 1 -- 172.21.15.16:0/245988927 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a58032670 con 0x55c5dca33bb0 2024-03-28T22:10:56.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.846+0000 7f4a3f7fe640 1 -- 172.21.15.16:0/245988927 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f4a50012d20 con 0x55c5dca33bb0 2024-03-28T22:10:56.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.846+0000 7f4a69009740 10 client.15531 fetch_fsmap learned FSMap version 21 2024-03-28T22:10:56.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.846+0000 7f4a69009740 1 -- 172.21.15.16:0/245988927 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55c5dc6ab3f0 con 0x55c5dca33bb0 2024-03-28T22:10:56.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.846+0000 7f4a3f7fe640 1 -- 172.21.15.16:0/245988927 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a58032670 con 0x55c5dca33bb0 2024-03-28T22:10:56.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.846+0000 7f4a3f7fe640 1 -- 172.21.15.16:0/245988927 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f4a50021020 con 0x55c5dca33bb0 2024-03-28T22:10:56.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.846+0000 7f4a69009740 10 client.15531 fetch_fsmap finished waiting for FSMap version 21 2024-03-28T22:10:56.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.846+0000 7f4a69009740 10 client.15531 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:10:56.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.846+0000 7f4a69009740 20 client.15531 populate_metadata read hostname 'smithi016' 2024-03-28T22:10:56.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.846+0000 7f4a69009740 1 --2- 172.21.15.16:0/245988927 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55c5dca4cb30 0x55c5dca4efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:56.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.846+0000 7f4a69009740 4 client.15531 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:10:56.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.846+0000 7f4a69009740 1 -- 172.21.15.16:0/245988927 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c5dc9a46c0 con 0x55c5dca4cb30 2024-03-28T22:10:56.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.847+0000 7f4a5ffff640 1 --2- 172.21.15.16:0/245988927 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55c5dca4cb30 0x55c5dca4efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:56.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.847+0000 7f4a3f7fe640 10 client.15531 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:10:56.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.847+0000 7f4a5ffff640 1 --2- 172.21.15.16:0/245988927 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55c5dca4cb30 0x55c5dca4efd0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d37a3f032f982947 server_cookie=17fe5bb61874fedf in_seq=0 out_seq=0 2024-03-28T22:10:56.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.848+0000 7f4a3f7fe640 1 -- 172.21.15.16:0/245988927 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c5dc9a46c0 con 0x55c5dca4cb30 2024-03-28T22:10:56.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.848+0000 7f4a3f7fe640 10 client.15531 handle_command_reply: tid=0 2024-03-28T22:10:56.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.848+0000 7f4a69009740 1 client.15531 shutdown 2024-03-28T22:10:56.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.848+0000 7f4a69009740 1 -- 172.21.15.16:0/245988927 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5dca33bb0 msgr2=0x55c5dca39cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:56.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.848+0000 7f4a69009740 1 --2- 172.21.15.16:0/245988927 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5dca33bb0 0x55c5dca39cd0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f4a50011480 tx=0x7f4a50011940 comp rx=0 tx=0).stop 2024-03-28T22:10:56.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.848+0000 7f4a69009740 1 -- 172.21.15.16:0/245988927 shutdown_connections 2024-03-28T22:10:56.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.848+0000 7f4a69009740 1 --2- 172.21.15.16:0/245988927 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55c5dca4cb30 0x55c5dca4efd0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:56.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.848+0000 7f4a69009740 1 --2- 172.21.15.16:0/245988927 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c5dca3c790 0x55c5dca3a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:56.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.848+0000 7f4a69009740 1 --2- 172.21.15.16:0/245988927 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5dca33bb0 0x55c5dca39cd0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:56.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.848+0000 7f4a69009740 1 --2- 172.21.15.16:0/245988927 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c5dca33290 0x55c5dca39790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:56.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.848+0000 7f4a3f7fe640 0 client.15531 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:10:56.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.848+0000 7f4a69009740 1 -- 172.21.15.16:0/245988927 >> 172.21.15.16:0/245988927 conn(0x55c5dca2d9c0 msgr2=0x55c5dca2f710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:56.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.848+0000 7f4a69009740 1 -- 172.21.15.16:0/245988927 shutdown_connections 2024-03-28T22:10:56.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.848+0000 7f4a69009740 1 -- 172.21.15.16:0/245988927 wait complete. 2024-03-28T22:10:56.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.848+0000 7f4a69009740 20 client.15531 trim_cache size 0 max 16384 2024-03-28T22:10:56.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.865+0000 7f4a69009740 1 Processor -- start 2024-03-28T22:10:56.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.865+0000 7f4a69009740 1 -- start start 2024-03-28T22:10:56.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.866+0000 7f4a69009740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c5dca33290 0x55c5dca48320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:56.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.866+0000 7f4a69009740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c5dca33bb0 0x55c5dca39e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:56.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.866+0000 7f4a69009740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5dca3c790 0x55c5dca548a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:56.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.866+0000 7f4a69009740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4a601afcb0 con 0x55c5dca3c790 2024-03-28T22:10:56.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.866+0000 7f4a69009740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4a601ad580 con 0x55c5dca33290 2024-03-28T22:10:56.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.866+0000 7f4a69009740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4a601ab3d0 con 0x55c5dca33bb0 2024-03-28T22:10:56.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.866+0000 7f4a648e5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5dca3c790 0x55c5dca548a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:56.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.866+0000 7f4a648e5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5dca3c790 0x55c5dca548a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37534/0 (socket says 172.21.15.16:37534) 2024-03-28T22:10:56.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.866+0000 7f4a648e5640 1 -- 172.21.15.16:0/3023522927 learned_addr learned my addr 172.21.15.16:0/3023522927 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:56.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.866+0000 7f4a5ffff640 1 --2- 172.21.15.16:0/3023522927 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c5dca33290 0x55c5dca48320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:56.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.866+0000 7f4a648e5640 1 -- 172.21.15.16:0/3023522927 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c5dca33bb0 msgr2=0x55c5dca39e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:56.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.866+0000 7f4a5f7fe640 1 --2- 172.21.15.16:0/3023522927 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c5dca33bb0 0x55c5dca39e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:56.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.866+0000 7f4a648e5640 1 --2- 172.21.15.16:0/3023522927 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c5dca33bb0 0x55c5dca39e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:56.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.866+0000 7f4a648e5640 1 -- 172.21.15.16:0/3023522927 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c5dca33290 msgr2=0x55c5dca48320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:56.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.866+0000 7f4a648e5640 1 --2- 172.21.15.16:0/3023522927 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c5dca33290 0x55c5dca48320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:56.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.866+0000 7f4a648e5640 1 -- 172.21.15.16:0/3023522927 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5dca339e0 con 0x55c5dca3c790 2024-03-28T22:10:56.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.866+0000 7f4a648e5640 1 --2- 172.21.15.16:0/3023522927 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5dca3c790 0x55c5dca548a0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f4a58005de0 tx=0x7f4a58005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a7e0b16281c983 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:56.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.866+0000 7f4a55ffb640 1 -- 172.21.15.16:0/3023522927 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a5805b070 con 0x55c5dca3c790 2024-03-28T22:10:56.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.866+0000 7f4a55ffb640 1 -- 172.21.15.16:0/3023522927 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a5804a640 con 0x55c5dca3c790 2024-03-28T22:10:56.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.866+0000 7f4a55ffb640 1 -- 172.21.15.16:0/3023522927 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a58039050 con 0x55c5dca3c790 2024-03-28T22:10:56.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.867+0000 7f4a69009740 1 -- 172.21.15.16:0/3023522927 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5dca3c790 msgr2=0x55c5dca548a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:56.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.867+0000 7f4a69009740 1 --2- 172.21.15.16:0/3023522927 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5dca3c790 0x55c5dca548a0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f4a58005de0 tx=0x7f4a58005f10 comp rx=0 tx=0).stop 2024-03-28T22:10:56.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.867+0000 7f4a69009740 1 -- 172.21.15.16:0/3023522927 shutdown_connections 2024-03-28T22:10:56.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.867+0000 7f4a69009740 1 --2- 172.21.15.16:0/3023522927 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5dca3c790 0x55c5dca548a0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:56.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.867+0000 7f4a69009740 1 --2- 172.21.15.16:0/3023522927 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c5dca33bb0 0x55c5dca39e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:56.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.867+0000 7f4a69009740 1 --2- 172.21.15.16:0/3023522927 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c5dca33290 0x55c5dca48320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:56.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.867+0000 7f4a69009740 1 -- 172.21.15.16:0/3023522927 >> 172.21.15.16:0/3023522927 conn(0x55c5dca2d9c0 msgr2=0x55c5dca39750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:56.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.867+0000 7f4a69009740 1 -- 172.21.15.16:0/3023522927 shutdown_connections 2024-03-28T22:10:56.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.867+0000 7f4a69009740 1 -- 172.21.15.16:0/3023522927 wait complete. 2024-03-28T22:10:56.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.867+0000 7f4a69009740 1 -- start start 2024-03-28T22:10:56.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.868+0000 7f4a69009740 1 Processor -- start 2024-03-28T22:10:56.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.868+0000 7f4a69009740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c5dca33290 0x55c5dca44930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:56.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.868+0000 7f4a69009740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5dca33bb0 0x55c5dca40cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:56.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.868+0000 7f4a69009740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c5dca3c790 0x55c5dca411f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:56.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.868+0000 7f4a69009740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4a601aba40 con 0x55c5dca33bb0 2024-03-28T22:10:56.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.868+0000 7f4a69009740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4a601aceb0 con 0x55c5dca33290 2024-03-28T22:10:56.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.868+0000 7f4a69009740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4a601ac110 con 0x55c5dca3c790 2024-03-28T22:10:56.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.868+0000 7f4a69009740 10 client.0 _ll_register_callbacks cb 0x7f4a50009550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:10:56.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.868+0000 7f4a5f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5dca33bb0 0x55c5dca40cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:56.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.868+0000 7f4a5f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5dca33bb0 0x55c5dca40cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37536/0 (socket says 172.21.15.16:37536) 2024-03-28T22:10:56.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.868+0000 7f4a5ffff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c5dca33290 0x55c5dca44930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:56.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.868+0000 7f4a5f7fe640 1 -- 172.21.15.16:0/1903256765 learned_addr learned my addr 172.21.15.16:0/1903256765 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:56.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.868+0000 7f4a648e5640 1 --2- 172.21.15.16:0/1903256765 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c5dca3c790 0x55c5dca411f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:56.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.868+0000 7f4a5f7fe640 1 -- 172.21.15.16:0/1903256765 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c5dca3c790 msgr2=0x55c5dca411f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:56.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.868+0000 7f4a5f7fe640 1 --2- 172.21.15.16:0/1903256765 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c5dca3c790 0x55c5dca411f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:56.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.868+0000 7f4a5f7fe640 1 -- 172.21.15.16:0/1903256765 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c5dca33290 msgr2=0x55c5dca44930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:56.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.868+0000 7f4a5f7fe640 1 --2- 172.21.15.16:0/1903256765 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c5dca33290 0x55c5dca44930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:56.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.868+0000 7f4a5f7fe640 1 -- 172.21.15.16:0/1903256765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5dc66b0f0 con 0x55c5dca33bb0 2024-03-28T22:10:56.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.868+0000 7f4a5f7fe640 1 --2- 172.21.15.16:0/1903256765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5dca33bb0 0x55c5dca40cb0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f4a5001a5d0 tx=0x7f4a50004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9221894766f3a69b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:56.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.868+0000 7f4a3f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:10:56.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.869+0000 7f4a3f7fe640 1 -- 172.21.15.16:0/1903256765 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a50004230 con 0x55c5dca33bb0 2024-03-28T22:10:56.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.869+0000 7f4a3f7fe640 1 -- 172.21.15.16:0/1903256765 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f4a500043d0 con 0x55c5dca33bb0 2024-03-28T22:10:56.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.869+0000 7f4a69009740 1 -- 172.21.15.16:0/1903256765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a5804a640 con 0x55c5dca33bb0 2024-03-28T22:10:56.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.869+0000 7f4a3f7fe640 1 -- 172.21.15.16:0/1903256765 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f4a50004e50 con 0x55c5dca33bb0 2024-03-28T22:10:56.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.869+0000 7f4a69009740 10 client.15537 fetch_fsmap learned FSMap version 21 2024-03-28T22:10:56.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.869+0000 7f4a69009740 1 -- 172.21.15.16:0/1903256765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55c5dcab3a50 con 0x55c5dca33bb0 2024-03-28T22:10:56.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.869+0000 7f4a3f7fe640 1 -- 172.21.15.16:0/1903256765 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a5804a640 con 0x55c5dca33bb0 2024-03-28T22:10:56.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.869+0000 7f4a3f7fe640 1 -- 172.21.15.16:0/1903256765 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f4a50033070 con 0x55c5dca33bb0 2024-03-28T22:10:56.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.869+0000 7f4a69009740 10 client.15537 fetch_fsmap finished waiting for FSMap version 21 2024-03-28T22:10:56.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.869+0000 7f4a69009740 10 client.15537 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:10:56.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.869+0000 7f4a69009740 20 client.15537 populate_metadata read hostname 'smithi016' 2024-03-28T22:10:56.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.869+0000 7f4a69009740 1 --2- 172.21.15.16:0/1903256765 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55c5dcab4220 0x55c5dcab6640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:56.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.869+0000 7f4a69009740 4 client.15537 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:10:56.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.869+0000 7f4a69009740 1 -- 172.21.15.16:0/1903256765 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c5dc99ea10 con 0x55c5dcab4220 2024-03-28T22:10:56.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.869+0000 7f4a5ffff640 1 --2- 172.21.15.16:0/1903256765 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55c5dcab4220 0x55c5dcab6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:56.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.870+0000 7f4a5ffff640 1 --2- 172.21.15.16:0/1903256765 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55c5dcab4220 0x55c5dcab6640 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ed0e19acbd3e911 server_cookie=4dd10e8678de5ccc in_seq=0 out_seq=0 2024-03-28T22:10:56.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.870+0000 7f4a3f7fe640 10 client.15537 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:10:56.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.870+0000 7f4a3f7fe640 1 -- 172.21.15.16:0/1903256765 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c5dc99ea10 con 0x55c5dcab4220 2024-03-28T22:10:56.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.870+0000 7f4a3f7fe640 10 client.15537 handle_command_reply: tid=0 2024-03-28T22:10:56.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.870+0000 7f4a69009740 1 client.15537 shutdown 2024-03-28T22:10:56.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.870+0000 7f4a69009740 1 -- 172.21.15.16:0/1903256765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5dca33bb0 msgr2=0x55c5dca40cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:56.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.870+0000 7f4a69009740 1 --2- 172.21.15.16:0/1903256765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5dca33bb0 0x55c5dca40cb0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f4a5001a5d0 tx=0x7f4a50004a50 comp rx=0 tx=0).stop 2024-03-28T22:10:56.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.870+0000 7f4a69009740 1 -- 172.21.15.16:0/1903256765 shutdown_connections 2024-03-28T22:10:56.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.870+0000 7f4a69009740 1 --2- 172.21.15.16:0/1903256765 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55c5dcab4220 0x55c5dcab6640 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:56.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.870+0000 7f4a69009740 1 --2- 172.21.15.16:0/1903256765 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c5dca3c790 0x55c5dca411f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:56.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.870+0000 7f4a69009740 1 --2- 172.21.15.16:0/1903256765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c5dca33bb0 0x55c5dca40cb0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:56.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.870+0000 7f4a3f7fe640 0 client.15537 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:10:56.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.870+0000 7f4a69009740 1 --2- 172.21.15.16:0/1903256765 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c5dca33290 0x55c5dca44930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:56.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.870+0000 7f4a69009740 1 -- 172.21.15.16:0/1903256765 >> 172.21.15.16:0/1903256765 conn(0x55c5dca2d9c0 msgr2=0x55c5dca2f680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:56.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.870+0000 7f4a69009740 1 -- 172.21.15.16:0/1903256765 shutdown_connections 2024-03-28T22:10:56.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.870+0000 7f4a69009740 1 -- 172.21.15.16:0/1903256765 wait complete. 2024-03-28T22:10:56.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.870+0000 7f4a69009740 20 client.15537 trim_cache size 0 max 16384 2024-03-28T22:10:56.879 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:10:56.879 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-03-28T22:10:56.879 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-03-28T22:10:56.879 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:10:56.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.871+0000 7f4a66348640 1 -- 172.21.15.16:0/302463495 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4a4c0a1f80 msgr2=0x7f4a4c0a4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:56.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.871+0000 7f4a66348640 1 --2- 172.21.15.16:0/302463495 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4a4c0a1f80 0x7f4a4c0a4440 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4a5804c000 tx=0x7f4a58045000 comp rx=0 tx=0).stop 2024-03-28T22:10:56.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.871+0000 7f4a66348640 1 -- 172.21.15.16:0/302463495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a60102970 msgr2=0x7f4a60198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:56.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.871+0000 7f4a66348640 1 --2- 172.21.15.16:0/302463495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a60102970 0x7f4a60198030 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4a4800b4a0 tx=0x7f4a4800b960 comp rx=0 tx=0).stop 2024-03-28T22:10:56.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.871+0000 7f4a66348640 1 -- 172.21.15.16:0/302463495 shutdown_connections 2024-03-28T22:10:56.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.871+0000 7f4a66348640 1 --2- 172.21.15.16:0/302463495 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4a4c0a1f80 0x7f4a4c0a4440 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:56.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.871+0000 7f4a66348640 1 --2- 172.21.15.16:0/302463495 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a6010bd40 0x7f4a6019c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:56.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.871+0000 7f4a66348640 1 --2- 172.21.15.16:0/302463495 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a60103290 0x7f4a60198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:56.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.871+0000 7f4a66348640 1 --2- 172.21.15.16:0/302463495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a60102970 0x7f4a60198030 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:56.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.871+0000 7f4a66348640 1 -- 172.21.15.16:0/302463495 >> 172.21.15.16:0/302463495 conn(0x7f4a60077770 msgr2=0x7f4a600fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:56.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.871+0000 7f4a66348640 1 -- 172.21.15.16:0/302463495 shutdown_connections 2024-03-28T22:10:56.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.871+0000 7f4a66348640 1 -- 172.21.15.16:0/302463495 wait complete. 2024-03-28T22:10:56.884 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:10:56.884 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:10:56.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.985+0000 7f4b843d0640 1 -- 172.21.15.16:0/3067889834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b7c102970 msgr2=0x7f4b7c102d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:56.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.985+0000 7f4b843d0640 1 --2- 172.21.15.16:0/3067889834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b7c102970 0x7f4b7c102d50 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f4b78009900 tx=0x7f4b7802bcd0 comp rx=0 tx=0).stop 2024-03-28T22:10:56.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.985+0000 7f4b843d0640 1 -- 172.21.15.16:0/3067889834 shutdown_connections 2024-03-28T22:10:56.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.985+0000 7f4b843d0640 1 --2- 172.21.15.16:0/3067889834 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b7c10bd40 0x7f4b7c10e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:56.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.985+0000 7f4b843d0640 1 --2- 172.21.15.16:0/3067889834 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b7c103290 0x7f4b7c10b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:56.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.985+0000 7f4b843d0640 1 --2- 172.21.15.16:0/3067889834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b7c102970 0x7f4b7c102d50 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:56.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.985+0000 7f4b843d0640 1 -- 172.21.15.16:0/3067889834 >> 172.21.15.16:0/3067889834 conn(0x7f4b7c0777d0 msgr2=0x7f4b7c077be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:56.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.985+0000 7f4b843d0640 1 -- 172.21.15.16:0/3067889834 shutdown_connections 2024-03-28T22:10:56.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.985+0000 7f4b843d0640 1 -- 172.21.15.16:0/3067889834 wait complete. 2024-03-28T22:10:56.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.986+0000 7f4b843d0640 1 Processor -- start 2024-03-28T22:10:56.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.986+0000 7f4b843d0640 1 -- start start 2024-03-28T22:10:56.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.986+0000 7f4b843d0640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b7c102970 0x7f4b7c1980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:56.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.986+0000 7f4b843d0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b7c103290 0x7f4b7c1985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:56.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.986+0000 7f4b843d0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b7c10bd40 0x7f4b7c19c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:56.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.986+0000 7f4b843d0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4b7c111b70 con 0x7f4b7c10bd40 2024-03-28T22:10:56.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.986+0000 7f4b843d0640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4b7c1119f0 con 0x7f4b7c102970 2024-03-28T22:10:56.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.986+0000 7f4b843d0640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4b7c111cf0 con 0x7f4b7c103290 2024-03-28T22:10:56.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.986+0000 7f4b8296d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b7c10bd40 0x7f4b7c19c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:56.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.986+0000 7f4b8216c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b7c102970 0x7f4b7c1980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:56.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.986+0000 7f4b8296d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b7c10bd40 0x7f4b7c19c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37552/0 (socket says 172.21.15.16:37552) 2024-03-28T22:10:56.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.986+0000 7f4b8296d640 1 -- 172.21.15.16:0/2077102799 learned_addr learned my addr 172.21.15.16:0/2077102799 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:56.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.986+0000 7f4b8196b640 1 --2- 172.21.15.16:0/2077102799 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b7c103290 0x7f4b7c1985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:56.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.986+0000 7f4b8296d640 1 -- 172.21.15.16:0/2077102799 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b7c103290 msgr2=0x7f4b7c1985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:56.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.986+0000 7f4b8296d640 1 --2- 172.21.15.16:0/2077102799 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b7c103290 0x7f4b7c1985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:56.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.986+0000 7f4b8296d640 1 -- 172.21.15.16:0/2077102799 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b7c102970 msgr2=0x7f4b7c1980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:56.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.986+0000 7f4b8296d640 1 --2- 172.21.15.16:0/2077102799 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b7c102970 0x7f4b7c1980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:56.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.986+0000 7f4b8296d640 1 -- 172.21.15.16:0/2077102799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b7c19d050 con 0x7f4b7c10bd40 2024-03-28T22:10:56.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.986+0000 7f4b8216c640 1 --2- 172.21.15.16:0/2077102799 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b7c102970 0x7f4b7c1980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:10:56.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.986+0000 7f4b8296d640 1 --2- 172.21.15.16:0/2077102799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b7c10bd40 0x7f4b7c19c920 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f4b7400eb10 tx=0x7f4b7400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:56.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.987+0000 7f4b737fe640 1 -- 172.21.15.16:0/2077102799 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b74004b20 con 0x7f4b7c10bd40 2024-03-28T22:10:56.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.987+0000 7f4b737fe640 1 -- 172.21.15.16:0/2077102799 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b74004cc0 con 0x7f4b7c10bd40 2024-03-28T22:10:56.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.987+0000 7f4b843d0640 1 -- 172.21.15.16:0/2077102799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b7c19d340 con 0x7f4b7c10bd40 2024-03-28T22:10:56.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.987+0000 7f4b737fe640 1 -- 172.21.15.16:0/2077102799 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b74010760 con 0x7f4b7c10bd40 2024-03-28T22:10:56.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.987+0000 7f4b843d0640 1 -- 172.21.15.16:0/2077102799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b7c1a4bd0 con 0x7f4b7c10bd40 2024-03-28T22:10:56.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.988+0000 7f4b843d0640 1 -- 172.21.15.16:0/2077102799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b7c068a20 con 0x7f4b7c10bd40 2024-03-28T22:10:56.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.989+0000 7f4b737fe640 1 -- 172.21.15.16:0/2077102799 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 136932+0+0 (secure 0 0 0) 0x7f4b74010900 con 0x7f4b7c10bd40 2024-03-28T22:10:56.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.989+0000 7f4b737fe640 1 --2- 172.21.15.16:0/2077102799 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4b680a2000 0x7f4b680a44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:56.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.990+0000 7f4b8216c640 1 --2- 172.21.15.16:0/2077102799 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4b680a2000 0x7f4b680a44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:56.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.990+0000 7f4b737fe640 1 -- 172.21.15.16:0/2077102799 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f4b740ceb40 con 0x7f4b7c10bd40 2024-03-28T22:10:56.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.990+0000 7f4b8216c640 1 --2- 172.21.15.16:0/2077102799 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4b680a2000 0x7f4b680a44c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4b78005ab0 tx=0x7f4b78005a20 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:56.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:56.993+0000 7f4b737fe640 1 -- 172.21.15.16:0/2077102799 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4b74053280 con 0x7f4b7c10bd40 2024-03-28T22:10:57.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.214+0000 7f4b843d0640 1 -- 172.21.15.16:0/2077102799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b7c02d170 con 0x7f4b7c10bd40 2024-03-28T22:10:57.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.215+0000 7f4b737fe640 1 -- 172.21.15.16:0/2077102799 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11851 (secure 0 0 0) 0x7f4b7409ba80 con 0x7f4b7c10bd40 2024-03-28T22:10:57.217 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:10:57.217 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/1644581227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1644581227},{"type":"v1","addr":"172.21.15.16:6839","nonce":1644581227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/2593062669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2593062669},{"type":"v1","addr":"172.21.15.16:6841","nonce":2593062669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:08:29.672976+0000","modified":"2024-03-28T22:10:19.955098+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24566,"mds_1":24524,"mds_2":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24466":{"gid":24466,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/247249857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":247249857},{"type":"v1","addr":"172.21.15.63:6835","nonce":247249857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6837/2324231032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":2324231032},{"type":"v1","addr":"172.21.15.63:6837","nonce":2324231032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6839/1177197077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":1177197077},{"type":"v1","addr":"172.21.15.63:6839","nonce":1177197077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"e","rank":2,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.53:6835/350727634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":350727634},{"type":"v1","addr":"172.21.15.53:6835","nonce":350727634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/349877573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":349877573},{"type":"v1","addr":"172.21.15.53:6837","nonce":349877573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6839/3714153956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":3714153956},{"type":"v1","addr":"172.21.15.53:6839","nonce":3714153956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24566,"qdb_cluster":[24500,24524,24566]},"id":1}]} 2024-03-28T22:10:57.217 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 21 2024-03-28T22:10:57.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.217+0000 7f4b843d0640 1 -- 172.21.15.16:0/2077102799 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4b680a2000 msgr2=0x7f4b680a44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:57.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.217+0000 7f4b843d0640 1 --2- 172.21.15.16:0/2077102799 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4b680a2000 0x7f4b680a44c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4b78005ab0 tx=0x7f4b78005a20 comp rx=0 tx=0).stop 2024-03-28T22:10:57.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.217+0000 7f4b843d0640 1 -- 172.21.15.16:0/2077102799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b7c10bd40 msgr2=0x7f4b7c19c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:57.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.217+0000 7f4b843d0640 1 --2- 172.21.15.16:0/2077102799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b7c10bd40 0x7f4b7c19c920 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f4b7400eb10 tx=0x7f4b7400c2d0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.217+0000 7f4b843d0640 1 -- 172.21.15.16:0/2077102799 shutdown_connections 2024-03-28T22:10:57.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.217+0000 7f4b843d0640 1 --2- 172.21.15.16:0/2077102799 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4b680a2000 0x7f4b680a44c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.217+0000 7f4b843d0640 1 --2- 172.21.15.16:0/2077102799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b7c10bd40 0x7f4b7c19c920 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.217+0000 7f4b843d0640 1 --2- 172.21.15.16:0/2077102799 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b7c103290 0x7f4b7c1985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.217+0000 7f4b843d0640 1 --2- 172.21.15.16:0/2077102799 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b7c102970 0x7f4b7c1980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.217+0000 7f4b843d0640 1 -- 172.21.15.16:0/2077102799 >> 172.21.15.16:0/2077102799 conn(0x7f4b7c0777d0 msgr2=0x7f4b7c0fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:57.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.217+0000 7f4b843d0640 1 -- 172.21.15.16:0/2077102799 shutdown_connections 2024-03-28T22:10:57.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.217+0000 7f4b843d0640 1 -- 172.21.15.16:0/2077102799 wait complete. 2024-03-28T22:10:57.230 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:10:57.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.334+0000 7f7db791e640 1 -- 172.21.15.16:0/240212122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7db01020b0 msgr2=0x7f7db010e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:57.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.334+0000 7f7db791e640 1 --2- 172.21.15.16:0/240212122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7db01020b0 0x7f7db010e5c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f7da0009930 tx=0x7f7da002bc30 comp rx=0 tx=0).stop 2024-03-28T22:10:57.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.334+0000 7f7db791e640 1 -- 172.21.15.16:0/240212122 shutdown_connections 2024-03-28T22:10:57.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.334+0000 7f7db791e640 1 --2- 172.21.15.16:0/240212122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7db01020b0 0x7f7db010e5c0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.334+0000 7f7db791e640 1 --2- 172.21.15.16:0/240212122 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7db01016f0 0x7f7db0101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.334+0000 7f7db791e640 1 --2- 172.21.15.16:0/240212122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7db0107700 0x7f7db0107ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.334+0000 7f7db791e640 1 -- 172.21.15.16:0/240212122 >> 172.21.15.16:0/240212122 conn(0x7f7db00fd1e0 msgr2=0x7f7db00ff600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:57.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.334+0000 7f7db791e640 1 -- 172.21.15.16:0/240212122 shutdown_connections 2024-03-28T22:10:57.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.334+0000 7f7db791e640 1 -- 172.21.15.16:0/240212122 wait complete. 2024-03-28T22:10:57.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.334+0000 7f7db791e640 1 Processor -- start 2024-03-28T22:10:57.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.334+0000 7f7db791e640 1 -- start start 2024-03-28T22:10:57.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.335+0000 7f7db791e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7db01016f0 0x7f7db0073a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.335+0000 7f7db791e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7db01020b0 0x7f7db0073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.335+0000 7f7db791e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7db0107700 0x7f7db00744d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.335+0000 7f7db791e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7db0111b10 con 0x7f7db01020b0 2024-03-28T22:10:57.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.335+0000 7f7db791e640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7db0111990 con 0x7f7db01016f0 2024-03-28T22:10:57.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.335+0000 7f7db791e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7db0111c90 con 0x7f7db0107700 2024-03-28T22:10:57.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.335+0000 7f7db4eb9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7db01020b0 0x7f7db0073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.335+0000 7f7db4eb9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7db01020b0 0x7f7db0073f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37564/0 (socket says 172.21.15.16:37564) 2024-03-28T22:10:57.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.335+0000 7f7db4eb9640 1 -- 172.21.15.16:0/2517094870 learned_addr learned my addr 172.21.15.16:0/2517094870 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:57.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.335+0000 7f7db56ba640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7db01016f0 0x7f7db0073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.335+0000 7f7db5ebb640 1 --2- 172.21.15.16:0/2517094870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7db0107700 0x7f7db00744d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.335+0000 7f7db4eb9640 1 -- 172.21.15.16:0/2517094870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7db0107700 msgr2=0x7f7db00744d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:57.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.335+0000 7f7db4eb9640 1 --2- 172.21.15.16:0/2517094870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7db0107700 0x7f7db00744d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.335+0000 7f7db4eb9640 1 -- 172.21.15.16:0/2517094870 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7db01016f0 msgr2=0x7f7db0073a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:57.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.335+0000 7f7db4eb9640 1 --2- 172.21.15.16:0/2517094870 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7db01016f0 0x7f7db0073a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.335+0000 7f7db4eb9640 1 -- 172.21.15.16:0/2517094870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7db01a6430 con 0x7f7db01020b0 2024-03-28T22:10:57.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.335+0000 7f7db5ebb640 1 --2- 172.21.15.16:0/2517094870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7db0107700 0x7f7db00744d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:10:57.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.335+0000 7f7db56ba640 1 --2- 172.21.15.16:0/2517094870 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7db01016f0 0x7f7db0073a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:10:57.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.335+0000 7f7db4eb9640 1 --2- 172.21.15.16:0/2517094870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7db01020b0 0x7f7db0073f90 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f7da400ea60 tx=0x7f7da400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:57.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.335+0000 7f7dae7fc640 1 -- 172.21.15.16:0/2517094870 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da400cde0 con 0x7f7db01020b0 2024-03-28T22:10:57.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.335+0000 7f7dae7fc640 1 -- 172.21.15.16:0/2517094870 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7da40035f0 con 0x7f7db01020b0 2024-03-28T22:10:57.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.335+0000 7f7db791e640 1 -- 172.21.15.16:0/2517094870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7db01a6720 con 0x7f7db01020b0 2024-03-28T22:10:57.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.335+0000 7f7dae7fc640 1 -- 172.21.15.16:0/2517094870 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da4010670 con 0x7f7db01020b0 2024-03-28T22:10:57.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.335+0000 7f7db791e640 1 -- 172.21.15.16:0/2517094870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7db01a6c70 con 0x7f7db01020b0 2024-03-28T22:10:57.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.338+0000 7f7dae7fc640 1 -- 172.21.15.16:0/2517094870 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 136932+0+0 (secure 0 0 0) 0x7f7da40037a0 con 0x7f7db01020b0 2024-03-28T22:10:57.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.338+0000 7f7dba5da740 1 Processor -- start 2024-03-28T22:10:57.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.338+0000 7f7dba5da740 1 -- start start 2024-03-28T22:10:57.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.338+0000 7f7dba5da740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c4dfd470 0x55b4c4dfd850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.338+0000 7f7dba5da740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4c4dfdd90 0x55b4c4e06430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.338+0000 7f7dba5da740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4c4e06970 0x55b4c4e08d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.338+0000 7f7dba5da740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b4c4d611b0 con 0x55b4c4dfd470 2024-03-28T22:10:57.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.338+0000 7f7dba5da740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55b4c4d09c00 con 0x55b4c4dfdd90 2024-03-28T22:10:57.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.338+0000 7f7dba5da740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55b4c4d5d250 con 0x55b4c4e06970 2024-03-28T22:10:57.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.338+0000 7f7dae7fc640 1 --2- 172.21.15.16:0/2517094870 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f7d980a1fb0 0x7f7d980a4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.339+0000 7f7db5ebb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c4dfd470 0x55b4c4dfd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.339+0000 7f7db56ba640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4c4dfdd90 0x55b4c4e06430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.339+0000 7f7db56ba640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4c4dfdd90 0x55b4c4e06430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:48024/0 (socket says 172.21.15.16:48024) 2024-03-28T22:10:57.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.339+0000 7f7db56ba640 1 -- 172.21.15.16:0/2393976784 learned_addr learned my addr 172.21.15.16:0/2393976784 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:57.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.339+0000 7f7dae7fc640 1 -- 172.21.15.16:0/2517094870 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f7da40ce690 con 0x7f7db01020b0 2024-03-28T22:10:57.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.339+0000 7f7db5ebb640 1 --2- 172.21.15.16:0/2517094870 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f7d980a1fb0 0x7f7d980a4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.339+0000 7f7db4eb9640 1 --2- 172.21.15.16:0/2393976784 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4c4e06970 0x55b4c4e08d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.339+0000 7f7db56ba640 1 -- 172.21.15.16:0/2393976784 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4c4e06970 msgr2=0x55b4c4e08d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:57.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.339+0000 7f7db56ba640 1 --2- 172.21.15.16:0/2393976784 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4c4e06970 0x55b4c4e08d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.339+0000 7f7db56ba640 1 -- 172.21.15.16:0/2393976784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c4dfd470 msgr2=0x55b4c4dfd850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:57.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.339+0000 7f7db56ba640 1 --2- 172.21.15.16:0/2393976784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c4dfd470 0x55b4c4dfd850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.339+0000 7f7db56ba640 1 -- 172.21.15.16:0/2393976784 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4c4d293d0 con 0x55b4c4dfdd90 2024-03-28T22:10:57.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.339+0000 7f7db4eb9640 1 --2- 172.21.15.16:0/2393976784 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4c4e06970 0x55b4c4e08d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:10:57.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.339+0000 7f7db5ebb640 1 --2- 172.21.15.16:0/2393976784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c4dfd470 0x55b4c4dfd850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:10:57.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.339+0000 7f7db5ebb640 1 --2- 172.21.15.16:0/2517094870 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f7d980a1fb0 0x7f7d980a4470 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7da0036040 tx=0x7f7da0005fd0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:57.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.339+0000 7f7db56ba640 1 --2- 172.21.15.16:0/2393976784 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4c4dfdd90 0x55b4c4e06430 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7da8010760 tx=0x7f7da8010c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1202ff5f3d8c043c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:57.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.339+0000 7f7d977fe640 1 -- 172.21.15.16:0/2393976784 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da800ead0 con 0x55b4c4dfdd90 2024-03-28T22:10:57.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.339+0000 7f7d977fe640 1 -- 172.21.15.16:0/2393976784 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7da8010e70 con 0x55b4c4dfdd90 2024-03-28T22:10:57.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.339+0000 7f7d977fe640 1 -- 172.21.15.16:0/2393976784 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da801c670 con 0x55b4c4dfdd90 2024-03-28T22:10:57.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.340+0000 7f7dba5da740 1 -- 172.21.15.16:0/2393976784 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4c4dfdd90 msgr2=0x55b4c4e06430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:57.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.340+0000 7f7dba5da740 1 --2- 172.21.15.16:0/2393976784 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4c4dfdd90 0x55b4c4e06430 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7da8010760 tx=0x7f7da8010c20 comp rx=0 tx=0).stop 2024-03-28T22:10:57.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.340+0000 7f7dba5da740 1 -- 172.21.15.16:0/2393976784 shutdown_connections 2024-03-28T22:10:57.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.340+0000 7f7dba5da740 1 --2- 172.21.15.16:0/2393976784 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4c4e06970 0x55b4c4e08d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.340+0000 7f7dba5da740 1 --2- 172.21.15.16:0/2393976784 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4c4dfdd90 0x55b4c4e06430 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.340+0000 7f7dba5da740 1 --2- 172.21.15.16:0/2393976784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c4dfd470 0x55b4c4dfd850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.340+0000 7f7dba5da740 1 -- 172.21.15.16:0/2393976784 >> 172.21.15.16:0/2393976784 conn(0x55b4c4df7ba0 msgr2=0x55b4c4df7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:57.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.340+0000 7f7dba5da740 1 -- 172.21.15.16:0/2393976784 shutdown_connections 2024-03-28T22:10:57.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.340+0000 7f7dba5da740 1 -- 172.21.15.16:0/2393976784 wait complete. 2024-03-28T22:10:57.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.340+0000 7f7dba5da740 1 -- start start 2024-03-28T22:10:57.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.340+0000 7f7dba5da740 1 Processor -- start 2024-03-28T22:10:57.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.341+0000 7f7dba5da740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4c4dfd470 0x55b4c4e03a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.341+0000 7f7dba5da740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c4dfdd90 0x55b4c4e03f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.341+0000 7f7dba5da740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4c4e06970 0x55b4c4e04490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.341+0000 7f7dba5da740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b4c4e09760 con 0x55b4c4dfdd90 2024-03-28T22:10:57.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.341+0000 7f7dba5da740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55b4c4cff470 con 0x55b4c4e06970 2024-03-28T22:10:57.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.341+0000 7f7dba5da740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55b4c4e098e0 con 0x55b4c4dfd470 2024-03-28T22:10:57.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.341+0000 7f7dba5da740 10 client.0 _ll_register_callbacks cb 0x55b4c4d1ccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:10:57.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.341+0000 7f7db4eb9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c4dfdd90 0x55b4c4e03f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.341+0000 7f7db4eb9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c4dfdd90 0x55b4c4e03f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37588/0 (socket says 172.21.15.16:37588) 2024-03-28T22:10:57.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.341+0000 7f7db4eb9640 1 -- 172.21.15.16:0/3247379624 learned_addr learned my addr 172.21.15.16:0/3247379624 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:57.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.341+0000 7f7db5ebb640 1 --2- 172.21.15.16:0/3247379624 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4c4e06970 0x55b4c4e04490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.341+0000 7f7db4eb9640 1 -- 172.21.15.16:0/3247379624 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4c4dfd470 msgr2=0x55b4c4e03a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:57.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.341+0000 7f7db4eb9640 1 --2- 172.21.15.16:0/3247379624 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4c4dfd470 0x55b4c4e03a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.341+0000 7f7db4eb9640 1 -- 172.21.15.16:0/3247379624 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4c4e06970 msgr2=0x55b4c4e04490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:57.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.341+0000 7f7db4eb9640 1 --2- 172.21.15.16:0/3247379624 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4c4e06970 0x55b4c4e04490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.341+0000 7f7db4eb9640 1 -- 172.21.15.16:0/3247379624 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4c4a753f0 con 0x55b4c4dfdd90 2024-03-28T22:10:57.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.341+0000 7f7db4eb9640 1 --2- 172.21.15.16:0/3247379624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c4dfdd90 0x55b4c4e03f50 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f7da409ac30 tx=0x7f7da409a950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99a6d895cb11c683 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:57.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.341+0000 7f7d94ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:10:57.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.342+0000 7f7d94ff9640 1 -- 172.21.15.16:0/3247379624 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da40ae0e0 con 0x55b4c4dfdd90 2024-03-28T22:10:57.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.342+0000 7f7d94ff9640 1 -- 172.21.15.16:0/3247379624 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f7da40cbff0 con 0x55b4c4dfdd90 2024-03-28T22:10:57.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.342+0000 7f7dba5da740 1 -- 172.21.15.16:0/3247379624 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7da8010e70 con 0x55b4c4dfdd90 2024-03-28T22:10:57.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.342+0000 7f7d94ff9640 1 -- 172.21.15.16:0/3247379624 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7da40b8050 con 0x55b4c4dfdd90 2024-03-28T22:10:57.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.342+0000 7f7dba5da740 10 client.15555 fetch_fsmap learned FSMap version 21 2024-03-28T22:10:57.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.342+0000 7f7dba5da740 1 -- 172.21.15.16:0/3247379624 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b4c4b3f7d0 con 0x55b4c4dfdd90 2024-03-28T22:10:57.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.342+0000 7f7d94ff9640 1 -- 172.21.15.16:0/3247379624 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7da8010e70 con 0x55b4c4dfdd90 2024-03-28T22:10:57.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.342+0000 7f7d94ff9640 1 -- 172.21.15.16:0/3247379624 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f7da409a380 con 0x55b4c4dfdd90 2024-03-28T22:10:57.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.342+0000 7f7dba5da740 10 client.15555 fetch_fsmap finished waiting for FSMap version 21 2024-03-28T22:10:57.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.342+0000 7f7dba5da740 10 client.15555 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-28T22:10:57.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.342+0000 7f7dba5da740 20 client.15555 populate_metadata read hostname 'smithi016' 2024-03-28T22:10:57.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.342+0000 7f7dba5da740 1 --2- 172.21.15.16:0/3247379624 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55b4c4e16920 0x55b4c4e18d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.342+0000 7f7dba5da740 4 client.15555 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:10:57.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.342+0000 7f7dba5da740 1 -- 172.21.15.16:0/3247379624 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b4c4d6e6f0 con 0x55b4c4e16920 2024-03-28T22:10:57.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.342+0000 7f7db56ba640 1 --2- 172.21.15.16:0/3247379624 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55b4c4e16920 0x55b4c4e18d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.343+0000 7f7d94ff9640 10 client.15555 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:10:57.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.343+0000 7f7db56ba640 1 --2- 172.21.15.16:0/3247379624 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55b4c4e16920 0x55b4c4e18d90 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a35b107ed1c634d1 server_cookie=c1d73089260d79ec in_seq=0 out_seq=0 2024-03-28T22:10:57.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.343+0000 7f7d94ff9640 1 -- 172.21.15.16:0/3247379624 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b4c4d6e6f0 con 0x55b4c4e16920 2024-03-28T22:10:57.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.343+0000 7f7d94ff9640 10 client.15555 handle_command_reply: tid=0 2024-03-28T22:10:57.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.343+0000 7f7dba5da740 1 client.15555 shutdown 2024-03-28T22:10:57.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.344+0000 7f7dba5da740 1 -- 172.21.15.16:0/3247379624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c4dfdd90 msgr2=0x55b4c4e03f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:57.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.344+0000 7f7dba5da740 1 --2- 172.21.15.16:0/3247379624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c4dfdd90 0x55b4c4e03f50 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f7da409ac30 tx=0x7f7da409a950 comp rx=0 tx=0).stop 2024-03-28T22:10:57.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.344+0000 7f7dba5da740 1 -- 172.21.15.16:0/3247379624 shutdown_connections 2024-03-28T22:10:57.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.344+0000 7f7dba5da740 1 --2- 172.21.15.16:0/3247379624 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55b4c4e16920 0x55b4c4e18d90 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.344+0000 7f7dba5da740 1 --2- 172.21.15.16:0/3247379624 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4c4e06970 0x55b4c4e04490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.344+0000 7f7dba5da740 1 --2- 172.21.15.16:0/3247379624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c4dfdd90 0x55b4c4e03f50 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.344+0000 7f7d94ff9640 0 client.15555 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:10:57.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.344+0000 7f7dba5da740 1 --2- 172.21.15.16:0/3247379624 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4c4dfd470 0x55b4c4e03a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.344+0000 7f7dba5da740 1 -- 172.21.15.16:0/3247379624 >> 172.21.15.16:0/3247379624 conn(0x55b4c4df7ba0 msgr2=0x55b4c4df98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:57.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.344+0000 7f7dba5da740 1 -- 172.21.15.16:0/3247379624 shutdown_connections 2024-03-28T22:10:57.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.344+0000 7f7dba5da740 1 -- 172.21.15.16:0/3247379624 wait complete. 2024-03-28T22:10:57.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.344+0000 7f7dba5da740 20 client.15555 trim_cache size 0 max 16384 2024-03-28T22:10:57.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.357+0000 7f7dba5da740 1 Processor -- start 2024-03-28T22:10:57.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.357+0000 7f7dba5da740 1 -- start start 2024-03-28T22:10:57.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.357+0000 7f7dba5da740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4c4dfd470 0x55b4c4e12500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.357+0000 7f7dba5da740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c4dfdd90 0x55b4c4e1e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.357+0000 7f7dba5da740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4c4e06970 0x55b4c4e0eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.357+0000 7f7dba5da740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7db01b1e30 con 0x55b4c4dfdd90 2024-03-28T22:10:57.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.357+0000 7f7dba5da740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7db01af700 con 0x55b4c4dfd470 2024-03-28T22:10:57.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.357+0000 7f7dba5da740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7db01ad550 con 0x55b4c4e06970 2024-03-28T22:10:57.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.357+0000 7f7db4eb9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c4dfdd90 0x55b4c4e1e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.357+0000 7f7db4eb9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c4dfdd90 0x55b4c4e1e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37592/0 (socket says 172.21.15.16:37592) 2024-03-28T22:10:57.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.357+0000 7f7db4eb9640 1 -- 172.21.15.16:0/1654523018 learned_addr learned my addr 172.21.15.16:0/1654523018 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:57.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.357+0000 7f7db56ba640 1 --2- 172.21.15.16:0/1654523018 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4c4dfd470 0x55b4c4e12500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.358+0000 7f7db5ebb640 1 --2- 172.21.15.16:0/1654523018 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4c4e06970 0x55b4c4e0eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.358+0000 7f7db4eb9640 1 -- 172.21.15.16:0/1654523018 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4c4e06970 msgr2=0x55b4c4e0eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:57.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.358+0000 7f7db4eb9640 1 --2- 172.21.15.16:0/1654523018 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4c4e06970 0x55b4c4e0eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.358+0000 7f7db4eb9640 1 -- 172.21.15.16:0/1654523018 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4c4dfd470 msgr2=0x55b4c4e12500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:57.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.358+0000 7f7db4eb9640 1 --2- 172.21.15.16:0/1654523018 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4c4dfd470 0x55b4c4e12500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.358+0000 7f7db4eb9640 1 -- 172.21.15.16:0/1654523018 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4c4df76d0 con 0x55b4c4dfdd90 2024-03-28T22:10:57.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.358+0000 7f7db4eb9640 1 --2- 172.21.15.16:0/1654523018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c4dfdd90 0x55b4c4e1e660 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f7da4097390 tx=0x7f7da4003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e09a8bd20252656 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:57.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.358+0000 7f7d96ffd640 1 -- 172.21.15.16:0/1654523018 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da409f080 con 0x55b4c4dfdd90 2024-03-28T22:10:57.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.358+0000 7f7d96ffd640 1 -- 172.21.15.16:0/1654523018 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7da40a36d0 con 0x55b4c4dfdd90 2024-03-28T22:10:57.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.358+0000 7f7d96ffd640 1 -- 172.21.15.16:0/1654523018 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da40b9240 con 0x55b4c4dfdd90 2024-03-28T22:10:57.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.358+0000 7f7dba5da740 1 -- 172.21.15.16:0/1654523018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c4dfdd90 msgr2=0x55b4c4e1e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:57.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.358+0000 7f7dba5da740 1 --2- 172.21.15.16:0/1654523018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c4dfdd90 0x55b4c4e1e660 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f7da4097390 tx=0x7f7da4003d40 comp rx=0 tx=0).stop 2024-03-28T22:10:57.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.358+0000 7f7dba5da740 1 -- 172.21.15.16:0/1654523018 shutdown_connections 2024-03-28T22:10:57.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.358+0000 7f7dba5da740 1 --2- 172.21.15.16:0/1654523018 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4c4e06970 0x55b4c4e0eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.358+0000 7f7dba5da740 1 --2- 172.21.15.16:0/1654523018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c4dfdd90 0x55b4c4e1e660 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.358+0000 7f7dba5da740 1 --2- 172.21.15.16:0/1654523018 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4c4dfd470 0x55b4c4e12500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.358+0000 7f7dba5da740 1 -- 172.21.15.16:0/1654523018 >> 172.21.15.16:0/1654523018 conn(0x55b4c4df7ba0 msgr2=0x55b4c4df64b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:57.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.358+0000 7f7dba5da740 1 -- 172.21.15.16:0/1654523018 shutdown_connections 2024-03-28T22:10:57.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.359+0000 7f7dba5da740 1 -- 172.21.15.16:0/1654523018 wait complete. 2024-03-28T22:10:57.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.359+0000 7f7dba5da740 1 -- start start 2024-03-28T22:10:57.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.359+0000 7f7dba5da740 1 Processor -- start 2024-03-28T22:10:57.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.359+0000 7f7dba5da740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c4dfd470 0x55b4c4e0aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.359+0000 7f7dba5da740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4c4dfdd90 0x55b4c4e0b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.359+0000 7f7dba5da740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4c4e06970 0x55b4c4e7bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.359+0000 7f7dba5da740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7db01adbc0 con 0x55b4c4dfd470 2024-03-28T22:10:57.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.359+0000 7f7dba5da740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7db01af030 con 0x55b4c4dfdd90 2024-03-28T22:10:57.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.359+0000 7f7dba5da740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7db01ae290 con 0x55b4c4e06970 2024-03-28T22:10:57.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.359+0000 7f7dba5da740 10 client.0 _ll_register_callbacks cb 0x7f7da409aa70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:10:57.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.359+0000 7f7db56ba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c4dfd470 0x55b4c4e0aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.359+0000 7f7db56ba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c4dfd470 0x55b4c4e0aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37604/0 (socket says 172.21.15.16:37604) 2024-03-28T22:10:57.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.359+0000 7f7db56ba640 1 -- 172.21.15.16:0/3313988540 learned_addr learned my addr 172.21.15.16:0/3313988540 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:57.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.359+0000 7f7db4eb9640 1 --2- 172.21.15.16:0/3313988540 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4c4dfdd90 0x55b4c4e0b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.359+0000 7f7db5ebb640 1 --2- 172.21.15.16:0/3313988540 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4c4e06970 0x55b4c4e7bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.360+0000 7f7db56ba640 1 -- 172.21.15.16:0/3313988540 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4c4e06970 msgr2=0x55b4c4e7bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:57.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.360+0000 7f7db56ba640 1 --2- 172.21.15.16:0/3313988540 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4c4e06970 0x55b4c4e7bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.360+0000 7f7db56ba640 1 -- 172.21.15.16:0/3313988540 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4c4dfdd90 msgr2=0x55b4c4e0b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:57.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.360+0000 7f7db56ba640 1 --2- 172.21.15.16:0/3313988540 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4c4dfdd90 0x55b4c4e0b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.360+0000 7f7db56ba640 1 -- 172.21.15.16:0/3313988540 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4c4e7c1d0 con 0x55b4c4dfd470 2024-03-28T22:10:57.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.360+0000 7f7db56ba640 1 --2- 172.21.15.16:0/3313988540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c4dfd470 0x55b4c4e0aad0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f7da8015040 tx=0x7f7da8012d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b43bfc2dcb65355d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:57.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.360+0000 7f7d94ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:10:57.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.360+0000 7f7d94ff9640 1 -- 172.21.15.16:0/3313988540 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da802c070 con 0x55b4c4dfd470 2024-03-28T22:10:57.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.360+0000 7f7d94ff9640 1 -- 172.21.15.16:0/3313988540 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f7da8003970 con 0x55b4c4dfd470 2024-03-28T22:10:57.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.360+0000 7f7dba5da740 1 -- 172.21.15.16:0/3313988540 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7da40a36d0 con 0x55b4c4dfd470 2024-03-28T22:10:57.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.360+0000 7f7d94ff9640 1 -- 172.21.15.16:0/3313988540 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7da8038050 con 0x55b4c4dfd470 2024-03-28T22:10:57.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.360+0000 7f7dba5da740 10 client.15561 fetch_fsmap learned FSMap version 21 2024-03-28T22:10:57.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.360+0000 7f7dba5da740 1 -- 172.21.15.16:0/3313988540 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b4c4e7d440 con 0x55b4c4dfd470 2024-03-28T22:10:57.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.360+0000 7f7d94ff9640 1 -- 172.21.15.16:0/3313988540 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7da40a36d0 con 0x55b4c4dfd470 2024-03-28T22:10:57.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.360+0000 7f7d94ff9640 1 -- 172.21.15.16:0/3313988540 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f7da800f040 con 0x55b4c4dfd470 2024-03-28T22:10:57.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.360+0000 7f7dba5da740 10 client.15561 fetch_fsmap finished waiting for FSMap version 21 2024-03-28T22:10:57.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.360+0000 7f7dba5da740 10 client.15561 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-28T22:10:57.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.360+0000 7f7dba5da740 20 client.15561 populate_metadata read hostname 'smithi016' 2024-03-28T22:10:57.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.360+0000 7f7dba5da740 1 --2- 172.21.15.16:0/3313988540 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55b4c4e7d830 0x55b4c4e7fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.360+0000 7f7dba5da740 4 client.15561 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:10:57.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.360+0000 7f7dba5da740 1 -- 172.21.15.16:0/3313988540 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b4c4d68a40 con 0x55b4c4e7d830 2024-03-28T22:10:57.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.361+0000 7f7db4eb9640 1 --2- 172.21.15.16:0/3313988540 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55b4c4e7d830 0x55b4c4e7fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.361+0000 7f7d94ff9640 10 client.15561 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:10:57.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.361+0000 7f7db4eb9640 1 --2- 172.21.15.16:0/3313988540 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55b4c4e7d830 0x55b4c4e7fc20 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2c530eb3d0890ccc server_cookie=2c011c31c9c02b3e in_seq=0 out_seq=0 2024-03-28T22:10:57.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.361+0000 7f7d94ff9640 1 -- 172.21.15.16:0/3313988540 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b4c4d68a40 con 0x55b4c4e7d830 2024-03-28T22:10:57.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.361+0000 7f7d94ff9640 10 client.15561 handle_command_reply: tid=0 2024-03-28T22:10:57.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.361+0000 7f7dba5da740 1 client.15561 shutdown 2024-03-28T22:10:57.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.361+0000 7f7dba5da740 1 -- 172.21.15.16:0/3313988540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c4dfd470 msgr2=0x55b4c4e0aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:57.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.361+0000 7f7dba5da740 1 --2- 172.21.15.16:0/3313988540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c4dfd470 0x55b4c4e0aad0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f7da8015040 tx=0x7f7da8012d10 comp rx=0 tx=0).stop 2024-03-28T22:10:57.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.361+0000 7f7dba5da740 1 -- 172.21.15.16:0/3313988540 shutdown_connections 2024-03-28T22:10:57.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.361+0000 7f7dba5da740 1 --2- 172.21.15.16:0/3313988540 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55b4c4e7d830 0x55b4c4e7fc20 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.361+0000 7f7dba5da740 1 --2- 172.21.15.16:0/3313988540 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4c4e06970 0x55b4c4e7bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.361+0000 7f7dba5da740 1 --2- 172.21.15.16:0/3313988540 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4c4dfdd90 0x55b4c4e0b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.361+0000 7f7dba5da740 1 --2- 172.21.15.16:0/3313988540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c4dfd470 0x55b4c4e0aad0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.361+0000 7f7dba5da740 1 -- 172.21.15.16:0/3313988540 >> 172.21.15.16:0/3313988540 conn(0x55b4c4df7ba0 msgr2=0x55b4c4e04240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:57.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.361+0000 7f7d94ff9640 0 client.15561 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:10:57.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.361+0000 7f7dba5da740 1 -- 172.21.15.16:0/3313988540 shutdown_connections 2024-03-28T22:10:57.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.361+0000 7f7dba5da740 1 -- 172.21.15.16:0/3313988540 wait complete. 2024-03-28T22:10:57.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.362+0000 7f7dba5da740 20 client.15561 trim_cache size 0 max 16384 2024-03-28T22:10:57.376 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:10:57.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.362+0000 7f7db791e640 1 -- 172.21.15.16:0/2517094870 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f7d980a1fb0 msgr2=0x7f7d980a4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:57.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.362+0000 7f7db791e640 1 --2- 172.21.15.16:0/2517094870 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f7d980a1fb0 0x7f7d980a4470 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7da0036040 tx=0x7f7da0005fd0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.362+0000 7f7db791e640 1 -- 172.21.15.16:0/2517094870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7db01020b0 msgr2=0x7f7db0073f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:57.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.362+0000 7f7db791e640 1 --2- 172.21.15.16:0/2517094870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7db01020b0 0x7f7db0073f90 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f7da400ea60 tx=0x7f7da400ef20 comp rx=0 tx=0).stop 2024-03-28T22:10:57.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.362+0000 7f7db791e640 1 -- 172.21.15.16:0/2517094870 shutdown_connections 2024-03-28T22:10:57.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.362+0000 7f7db791e640 1 --2- 172.21.15.16:0/2517094870 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f7d980a1fb0 0x7f7d980a4470 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.362+0000 7f7db791e640 1 --2- 172.21.15.16:0/2517094870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7db0107700 0x7f7db00744d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.362+0000 7f7db791e640 1 --2- 172.21.15.16:0/2517094870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7db01020b0 0x7f7db0073f90 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.362+0000 7f7db791e640 1 --2- 172.21.15.16:0/2517094870 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7db01016f0 0x7f7db0073a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.362+0000 7f7db791e640 1 -- 172.21.15.16:0/2517094870 >> 172.21.15.16:0/2517094870 conn(0x7f7db00fd1e0 msgr2=0x7f7db00fd9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:57.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.362+0000 7f7db791e640 1 -- 172.21.15.16:0/2517094870 shutdown_connections 2024-03-28T22:10:57.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.362+0000 7f7db791e640 1 -- 172.21.15.16:0/2517094870 wait complete. 2024-03-28T22:10:57.379 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:10:57.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.483+0000 7fbc1437a640 1 -- 172.21.15.16:0/2131428908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc0c1076f0 msgr2=0x7fbc0c107ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:57.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.483+0000 7fbc1437a640 1 --2- 172.21.15.16:0/2131428908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc0c1076f0 0x7fbc0c107ad0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fbc08009930 tx=0x7fbc0802bc30 comp rx=0 tx=0).stop 2024-03-28T22:10:57.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.483+0000 7fbc1437a640 1 -- 172.21.15.16:0/2131428908 shutdown_connections 2024-03-28T22:10:57.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.483+0000 7fbc1437a640 1 --2- 172.21.15.16:0/2131428908 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc0c1020a0 0x7fbc0c10e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.483+0000 7fbc1437a640 1 --2- 172.21.15.16:0/2131428908 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc0c1016e0 0x7fbc0c101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.483+0000 7fbc1437a640 1 --2- 172.21.15.16:0/2131428908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc0c1076f0 0x7fbc0c107ad0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.483+0000 7fbc1437a640 1 -- 172.21.15.16:0/2131428908 >> 172.21.15.16:0/2131428908 conn(0x7fbc0c0fd1f0 msgr2=0x7fbc0c0ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:57.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.483+0000 7fbc1437a640 1 -- 172.21.15.16:0/2131428908 shutdown_connections 2024-03-28T22:10:57.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.483+0000 7fbc1437a640 1 -- 172.21.15.16:0/2131428908 wait complete. 2024-03-28T22:10:57.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.483+0000 7fbc1437a640 1 Processor -- start 2024-03-28T22:10:57.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.483+0000 7fbc1437a640 1 -- start start 2024-03-28T22:10:57.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.484+0000 7fbc1437a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc0c1016e0 0x7fbc0c1980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.484+0000 7fbc1437a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc0c1020a0 0x7fbc0c1985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.484+0000 7fbc1437a640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc0c1076f0 0x7fbc0c19c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.484+0000 7fbc1437a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0c111b00 con 0x7fbc0c1020a0 2024-03-28T22:10:57.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.484+0000 7fbc1437a640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0c111980 con 0x7fbc0c1076f0 2024-03-28T22:10:57.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.484+0000 7fbc1437a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0c111c80 con 0x7fbc0c1016e0 2024-03-28T22:10:57.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.484+0000 7fbc11915640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc0c1020a0 0x7fbc0c1985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.484+0000 7fbc11915640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc0c1020a0 0x7fbc0c1985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37626/0 (socket says 172.21.15.16:37626) 2024-03-28T22:10:57.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.484+0000 7fbc11915640 1 -- 172.21.15.16:0/1070159542 learned_addr learned my addr 172.21.15.16:0/1070159542 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:57.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.484+0000 7fbc12917640 1 --2- 172.21.15.16:0/1070159542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc0c1076f0 0x7fbc0c19c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.484+0000 7fbc11915640 1 -- 172.21.15.16:0/1070159542 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc0c1016e0 msgr2=0x7fbc0c1980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:57.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.484+0000 7fbc11915640 1 --2- 172.21.15.16:0/1070159542 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc0c1016e0 0x7fbc0c1980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.484+0000 7fbc11915640 1 -- 172.21.15.16:0/1070159542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc0c1076f0 msgr2=0x7fbc0c19c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:57.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.484+0000 7fbc11915640 1 --2- 172.21.15.16:0/1070159542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc0c1076f0 0x7fbc0c19c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.484+0000 7fbc11915640 1 -- 172.21.15.16:0/1070159542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc0c19d090 con 0x7fbc0c1020a0 2024-03-28T22:10:57.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.484+0000 7fbc11915640 1 --2- 172.21.15.16:0/1070159542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc0c1020a0 0x7fbc0c1985e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc00b4a0 tx=0x7fbbfc00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:57.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.484+0000 7fbc037fe640 1 -- 172.21.15.16:0/1070159542 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc015020 con 0x7fbc0c1020a0 2024-03-28T22:10:57.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.484+0000 7fbc037fe640 1 -- 172.21.15.16:0/1070159542 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbfc011d10 con 0x7fbc0c1020a0 2024-03-28T22:10:57.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.484+0000 7fbc1437a640 1 -- 172.21.15.16:0/1070159542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc0c19d380 con 0x7fbc0c1020a0 2024-03-28T22:10:57.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.484+0000 7fbc037fe640 1 -- 172.21.15.16:0/1070159542 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc00f2e0 con 0x7fbc0c1020a0 2024-03-28T22:10:57.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.484+0000 7fbc1437a640 1 -- 172.21.15.16:0/1070159542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc0c1a4c10 con 0x7fbc0c1020a0 2024-03-28T22:10:57.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.487+0000 7fbc037fe640 1 -- 172.21.15.16:0/1070159542 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 136932+0+0 (secure 0 0 0) 0x7fbbfc018020 con 0x7fbc0c1020a0 2024-03-28T22:10:57.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.487+0000 7fbc1703a740 1 Processor -- start 2024-03-28T22:10:57.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.487+0000 7fbc1703a740 1 -- start start 2024-03-28T22:10:57.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.487+0000 7fbc1703a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565e3060470 0x5565e3060850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.487+0000 7fbc1703a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3060d90 0x5565e3069430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.487+0000 7fbc037fe640 1 --2- 172.21.15.16:0/1070159542 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fbbf80a1f80 0x7fbbf80a4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.487+0000 7fbc1703a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e3069970 0x5565e306bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.487+0000 7fbc1703a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5565e2fc41b0 con 0x5565e3069970 2024-03-28T22:10:57.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.487+0000 7fbc1703a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5565e2f6cc00 con 0x5565e3060d90 2024-03-28T22:10:57.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.487+0000 7fbc1703a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5565e2fc0250 con 0x5565e3060470 2024-03-28T22:10:57.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.488+0000 7fbc12116640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3060d90 0x5565e3069430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.488+0000 7fbc12917640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e3069970 0x5565e306bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.488+0000 7fbc12116640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3060d90 0x5565e3069430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:48096/0 (socket says 172.21.15.16:48096) 2024-03-28T22:10:57.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.488+0000 7fbc11915640 1 --2- 172.21.15.16:0/1070159542 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fbbf80a1f80 0x7fbbf80a4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.488+0000 7fbc12116640 1 -- 172.21.15.16:0/2286553872 learned_addr learned my addr 172.21.15.16:0/2286553872 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:57.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.488+0000 7fbc037fe640 1 -- 172.21.15.16:0/1070159542 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fbbfc0cdb60 con 0x7fbc0c1020a0 2024-03-28T22:10:57.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.488+0000 7fbc12917640 1 --2- 172.21.15.16:0/2286553872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565e3060470 0x5565e3060850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.488+0000 7fbc12116640 1 -- 172.21.15.16:0/2286553872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565e3060470 msgr2=0x5565e3060850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:57.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.488+0000 7fbc12116640 1 --2- 172.21.15.16:0/2286553872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565e3060470 0x5565e3060850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.488+0000 7fbc12116640 1 -- 172.21.15.16:0/2286553872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e3069970 msgr2=0x5565e306bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:57.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.488+0000 7fbc12116640 1 --2- 172.21.15.16:0/2286553872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e3069970 0x5565e306bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.488+0000 7fbc12116640 1 -- 172.21.15.16:0/2286553872 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565e2f8c3d0 con 0x5565e3060d90 2024-03-28T22:10:57.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.488+0000 7fbc12917640 1 --2- 172.21.15.16:0/2286553872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e3069970 0x5565e306bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:10:57.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.488+0000 7fbc11915640 1 --2- 172.21.15.16:0/1070159542 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fbbf80a1f80 0x7fbbf80a4440 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc0cb380 tx=0x7fbbfc016f00 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:57.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.488+0000 7fbc12116640 1 --2- 172.21.15.16:0/2286553872 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3060d90 0x5565e3069430 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbc0802cfd0 tx=0x7fbc080095e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9cb1d9944df38b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:57.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.488+0000 7fbc00bfe640 1 -- 172.21.15.16:0/2286553872 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc08041030 con 0x5565e3060d90 2024-03-28T22:10:57.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.488+0000 7fbc00bfe640 1 -- 172.21.15.16:0/2286553872 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc0802c5d0 con 0x5565e3060d90 2024-03-28T22:10:57.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.488+0000 7fbc00bfe640 1 -- 172.21.15.16:0/2286553872 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc08005280 con 0x5565e3060d90 2024-03-28T22:10:57.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.489+0000 7fbc1703a740 1 -- 172.21.15.16:0/2286553872 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3060d90 msgr2=0x5565e3069430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:57.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.489+0000 7fbc1703a740 1 --2- 172.21.15.16:0/2286553872 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3060d90 0x5565e3069430 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbc0802cfd0 tx=0x7fbc080095e0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.489+0000 7fbc1703a740 1 -- 172.21.15.16:0/2286553872 shutdown_connections 2024-03-28T22:10:57.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.489+0000 7fbc1703a740 1 --2- 172.21.15.16:0/2286553872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e3069970 0x5565e306bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.489+0000 7fbc1703a740 1 --2- 172.21.15.16:0/2286553872 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3060d90 0x5565e3069430 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.489+0000 7fbc1703a740 1 --2- 172.21.15.16:0/2286553872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565e3060470 0x5565e3060850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.489+0000 7fbc1703a740 1 -- 172.21.15.16:0/2286553872 >> 172.21.15.16:0/2286553872 conn(0x5565e305aba0 msgr2=0x5565e305af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:57.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.489+0000 7fbc1703a740 1 -- 172.21.15.16:0/2286553872 shutdown_connections 2024-03-28T22:10:57.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.489+0000 7fbc1703a740 1 -- 172.21.15.16:0/2286553872 wait complete. 2024-03-28T22:10:57.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.489+0000 7fbc1703a740 1 -- start start 2024-03-28T22:10:57.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.490+0000 7fbc1703a740 1 Processor -- start 2024-03-28T22:10:57.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.490+0000 7fbc1703a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e3060470 0x5565e30669b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.490+0000 7fbc1703a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565e3060d90 0x5565e3066ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.490+0000 7fbc1703a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3069970 0x5565e3067430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.490+0000 7fbc1703a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5565e306c700 con 0x5565e3060470 2024-03-28T22:10:57.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.490+0000 7fbc1703a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5565e2f62470 con 0x5565e3069970 2024-03-28T22:10:57.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.490+0000 7fbc1703a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5565e306c880 con 0x5565e3060d90 2024-03-28T22:10:57.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.490+0000 7fbc1703a740 10 client.0 _ll_register_callbacks cb 0x5565e2f7fcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:10:57.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.490+0000 7fbc12917640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e3060470 0x5565e30669b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.490+0000 7fbc12917640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e3060470 0x5565e30669b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37642/0 (socket says 172.21.15.16:37642) 2024-03-28T22:10:57.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.490+0000 7fbc12917640 1 -- 172.21.15.16:0/3973205814 learned_addr learned my addr 172.21.15.16:0/3973205814 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:57.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.490+0000 7fbc12917640 1 --2- 172.21.15.16:0/3973205814 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3069970 0x5565e3067430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.490+0000 7fbc12116640 1 --2- 172.21.15.16:0/3973205814 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565e3060d90 0x5565e3066ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.490+0000 7fbc12917640 1 -- 172.21.15.16:0/3973205814 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565e3060d90 msgr2=0x5565e3066ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:57.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.490+0000 7fbc12917640 1 --2- 172.21.15.16:0/3973205814 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565e3060d90 0x5565e3066ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.490+0000 7fbc12917640 1 -- 172.21.15.16:0/3973205814 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3069970 msgr2=0x5565e3067430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:57.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.490+0000 7fbc12917640 1 --2- 172.21.15.16:0/3973205814 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3069970 0x5565e3067430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.490+0000 7fbc12917640 1 -- 172.21.15.16:0/3973205814 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565e2cd83f0 con 0x5565e3060470 2024-03-28T22:10:57.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.490+0000 7fbc12116640 1 --2- 172.21.15.16:0/3973205814 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565e3060d90 0x5565e3066ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:10:57.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.490+0000 7fbc12917640 1 --2- 172.21.15.16:0/3973205814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e3060470 0x5565e30669b0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fbc0400e780 tx=0x7fbc04009d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8a00c73a4a56e0b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:57.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.490+0000 7fbbf5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:10:57.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.491+0000 7fbbf5ffb640 1 -- 172.21.15.16:0/3973205814 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc04013170 con 0x5565e3060470 2024-03-28T22:10:57.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.491+0000 7fbbf5ffb640 1 -- 172.21.15.16:0/3973205814 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fbc04004b20 con 0x5565e3060470 2024-03-28T22:10:57.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.491+0000 7fbc1703a740 1 -- 172.21.15.16:0/3973205814 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc0802c5d0 con 0x5565e3060470 2024-03-28T22:10:57.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.491+0000 7fbbf5ffb640 1 -- 172.21.15.16:0/3973205814 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fbc0402a050 con 0x5565e3060470 2024-03-28T22:10:57.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.491+0000 7fbc1703a740 10 client.15573 fetch_fsmap learned FSMap version 21 2024-03-28T22:10:57.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.491+0000 7fbc1703a740 1 -- 172.21.15.16:0/3973205814 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5565e2da27d0 con 0x5565e3060470 2024-03-28T22:10:57.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.491+0000 7fbbf5ffb640 1 -- 172.21.15.16:0/3973205814 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc0802c5d0 con 0x5565e3060470 2024-03-28T22:10:57.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.491+0000 7fbbf5ffb640 1 -- 172.21.15.16:0/3973205814 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fbc04010730 con 0x5565e3060470 2024-03-28T22:10:57.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.491+0000 7fbc1703a740 10 client.15573 fetch_fsmap finished waiting for FSMap version 21 2024-03-28T22:10:57.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.491+0000 7fbc1703a740 10 client.15573 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T22:10:57.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.491+0000 7fbc1703a740 20 client.15573 populate_metadata read hostname 'smithi016' 2024-03-28T22:10:57.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.491+0000 7fbc1703a740 1 --2- 172.21.15.16:0/3973205814 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5565e3079950 0x5565e307bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.491+0000 7fbc1703a740 4 client.15573 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:10:57.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.491+0000 7fbc1703a740 1 -- 172.21.15.16:0/3973205814 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5565e2fd16f0 con 0x5565e3079950 2024-03-28T22:10:57.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.491+0000 7fbc12116640 1 --2- 172.21.15.16:0/3973205814 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5565e3079950 0x5565e307bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.492+0000 7fbbf5ffb640 10 client.15573 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:10:57.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.492+0000 7fbc12116640 1 --2- 172.21.15.16:0/3973205814 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5565e3079950 0x5565e307bdd0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3f62e9d2f03f544f server_cookie=c5cd78ec5249139a in_seq=0 out_seq=0 2024-03-28T22:10:57.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.493+0000 7fbbf5ffb640 1 -- 172.21.15.16:0/3973205814 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5565e2fd16f0 con 0x5565e3079950 2024-03-28T22:10:57.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.493+0000 7fbbf5ffb640 10 client.15573 handle_command_reply: tid=0 2024-03-28T22:10:57.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.493+0000 7fbc1703a740 1 client.15573 shutdown 2024-03-28T22:10:57.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.493+0000 7fbc1703a740 1 -- 172.21.15.16:0/3973205814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e3060470 msgr2=0x5565e30669b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:57.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.493+0000 7fbc1703a740 1 --2- 172.21.15.16:0/3973205814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e3060470 0x5565e30669b0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fbc0400e780 tx=0x7fbc04009d10 comp rx=0 tx=0).stop 2024-03-28T22:10:57.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.493+0000 7fbc1703a740 1 -- 172.21.15.16:0/3973205814 shutdown_connections 2024-03-28T22:10:57.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.493+0000 7fbc1703a740 1 --2- 172.21.15.16:0/3973205814 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5565e3079950 0x5565e307bdd0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.493+0000 7fbc1703a740 1 --2- 172.21.15.16:0/3973205814 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3069970 0x5565e3067430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.493+0000 7fbbf5ffb640 0 client.15573 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:10:57.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.493+0000 7fbc1703a740 1 --2- 172.21.15.16:0/3973205814 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565e3060d90 0x5565e3066ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.493+0000 7fbc1703a740 1 --2- 172.21.15.16:0/3973205814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e3060470 0x5565e30669b0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.493+0000 7fbc1703a740 1 -- 172.21.15.16:0/3973205814 >> 172.21.15.16:0/3973205814 conn(0x5565e305aba0 msgr2=0x5565e305c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:57.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.493+0000 7fbc1703a740 1 -- 172.21.15.16:0/3973205814 shutdown_connections 2024-03-28T22:10:57.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.493+0000 7fbc1703a740 1 -- 172.21.15.16:0/3973205814 wait complete. 2024-03-28T22:10:57.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.493+0000 7fbc1703a740 20 client.15573 trim_cache size 0 max 16384 2024-03-28T22:10:57.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.506+0000 7fbc1703a740 1 Processor -- start 2024-03-28T22:10:57.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.506+0000 7fbc1703a740 1 -- start start 2024-03-28T22:10:57.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.506+0000 7fbc1703a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565e3060470 0x5565e3059f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.506+0000 7fbc1703a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3060d90 0x5565e3071eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.506+0000 7fbc1703a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e3069970 0x5565e30754a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.506+0000 7fbc1703a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0c1afd30 con 0x5565e3069970 2024-03-28T22:10:57.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.506+0000 7fbc1703a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0c1ad600 con 0x5565e3060d90 2024-03-28T22:10:57.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.506+0000 7fbc1703a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0c1ab450 con 0x5565e3060470 2024-03-28T22:10:57.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.507+0000 7fbc12917640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e3069970 0x5565e30754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.507+0000 7fbc12917640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e3069970 0x5565e30754a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37654/0 (socket says 172.21.15.16:37654) 2024-03-28T22:10:57.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.507+0000 7fbc12917640 1 -- 172.21.15.16:0/2506941057 learned_addr learned my addr 172.21.15.16:0/2506941057 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:57.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.507+0000 7fbc12116640 1 --2- 172.21.15.16:0/2506941057 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3060d90 0x5565e3071eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.507+0000 7fbc12917640 1 --2- 172.21.15.16:0/2506941057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565e3060470 0x5565e3059f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.507+0000 7fbc12917640 1 -- 172.21.15.16:0/2506941057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565e3060470 msgr2=0x5565e3059f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:57.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.507+0000 7fbc12917640 1 --2- 172.21.15.16:0/2506941057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565e3060470 0x5565e3059f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.507+0000 7fbc12917640 1 -- 172.21.15.16:0/2506941057 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3060d90 msgr2=0x5565e3071eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:57.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.507+0000 7fbc12917640 1 --2- 172.21.15.16:0/2506941057 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3060d90 0x5565e3071eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.507+0000 7fbc12917640 1 -- 172.21.15.16:0/2506941057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565e306bd60 con 0x5565e3069970 2024-03-28T22:10:57.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.507+0000 7fbc12116640 1 --2- 172.21.15.16:0/2506941057 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3060d90 0x5565e3071eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:10:57.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.507+0000 7fbc12917640 1 --2- 172.21.15.16:0/2506941057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e3069970 0x5565e30754a0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fbc04013ee0 tx=0x7fbc0400efd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad9b55ed2be1dcc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:57.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.507+0000 7fbbf7fff640 1 -- 172.21.15.16:0/2506941057 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc04004b20 con 0x5565e3069970 2024-03-28T22:10:57.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.507+0000 7fbbf7fff640 1 -- 172.21.15.16:0/2506941057 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc04009480 con 0x5565e3069970 2024-03-28T22:10:57.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.507+0000 7fbbf7fff640 1 -- 172.21.15.16:0/2506941057 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc04010650 con 0x5565e3069970 2024-03-28T22:10:57.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.508+0000 7fbc1703a740 1 -- 172.21.15.16:0/2506941057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e3069970 msgr2=0x5565e30754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:57.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.508+0000 7fbc1703a740 1 --2- 172.21.15.16:0/2506941057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e3069970 0x5565e30754a0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fbc04013ee0 tx=0x7fbc0400efd0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.508+0000 7fbc1703a740 1 -- 172.21.15.16:0/2506941057 shutdown_connections 2024-03-28T22:10:57.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.508+0000 7fbc1703a740 1 --2- 172.21.15.16:0/2506941057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e3069970 0x5565e30754a0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.508+0000 7fbc1703a740 1 --2- 172.21.15.16:0/2506941057 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3060d90 0x5565e3071eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.508+0000 7fbc1703a740 1 --2- 172.21.15.16:0/2506941057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565e3060470 0x5565e3059f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.508+0000 7fbc1703a740 1 -- 172.21.15.16:0/2506941057 >> 172.21.15.16:0/2506941057 conn(0x5565e305aba0 msgr2=0x5565e3067200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:57.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.508+0000 7fbc1703a740 1 -- 172.21.15.16:0/2506941057 shutdown_connections 2024-03-28T22:10:57.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.508+0000 7fbc1703a740 1 -- 172.21.15.16:0/2506941057 wait complete. 2024-03-28T22:10:57.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.508+0000 7fbc1703a740 1 -- start start 2024-03-28T22:10:57.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.508+0000 7fbc1703a740 1 Processor -- start 2024-03-28T22:10:57.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.508+0000 7fbc1703a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3060470 0x5565e30d7c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.508+0000 7fbc1703a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565e3060d90 0x5565e30d81d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.508+0000 7fbc1703a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e3069970 0x5565e30d8710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.508+0000 7fbc1703a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0c1abac0 con 0x5565e3069970 2024-03-28T22:10:57.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.508+0000 7fbc1703a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0c1acf30 con 0x5565e3060470 2024-03-28T22:10:57.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.508+0000 7fbc1703a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0c1ac190 con 0x5565e3060d90 2024-03-28T22:10:57.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.508+0000 7fbc1703a740 10 client.0 _ll_register_callbacks cb 0x7fbc0400e3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:10:57.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.509+0000 7fbc12917640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3060470 0x5565e30d7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.509+0000 7fbc12917640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3060470 0x5565e30d7c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:48122/0 (socket says 172.21.15.16:48122) 2024-03-28T22:10:57.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.509+0000 7fbc12917640 1 -- 172.21.15.16:0/3264105682 learned_addr learned my addr 172.21.15.16:0/3264105682 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:57.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.509+0000 7fbc12917640 1 --2- 172.21.15.16:0/3264105682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e3069970 0x5565e30d8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.509+0000 7fbc12116640 1 --2- 172.21.15.16:0/3264105682 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565e3060d90 0x5565e30d81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.509+0000 7fbc12917640 1 -- 172.21.15.16:0/3264105682 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565e3060d90 msgr2=0x5565e30d81d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:57.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.509+0000 7fbc12917640 1 --2- 172.21.15.16:0/3264105682 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565e3060d90 0x5565e30d81d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.509+0000 7fbc12917640 1 -- 172.21.15.16:0/3264105682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e3069970 msgr2=0x5565e30d8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:57.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.509+0000 7fbc12917640 1 --2- 172.21.15.16:0/3264105682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e3069970 0x5565e30d8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.509+0000 7fbc12917640 1 -- 172.21.15.16:0/3264105682 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565e30e2f10 con 0x5565e3060470 2024-03-28T22:10:57.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.509+0000 7fbc12917640 1 --2- 172.21.15.16:0/3264105682 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3060470 0x5565e30d7c90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbc04009db0 tx=0x7fbc04031040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=341fefcf68f3393f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:57.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.509+0000 7fbbf5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-03-28T22:10:57.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.509+0000 7fbbf5ffb640 1 -- 172.21.15.16:0/3264105682 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc04013370 con 0x5565e3060470 2024-03-28T22:10:57.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.509+0000 7fbc1703a740 1 -- 172.21.15.16:0/3264105682 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc04009480 con 0x5565e3060470 2024-03-28T22:10:57.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.509+0000 7fbbf5ffb640 1 -- 172.21.15.16:0/3264105682 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fbc040105d0 con 0x5565e3060470 2024-03-28T22:10:57.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.509+0000 7fbbf5ffb640 1 -- 172.21.15.16:0/3264105682 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fbc040107f0 con 0x5565e3060470 2024-03-28T22:10:57.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.509+0000 7fbc1703a740 10 client.25487 fetch_fsmap learned FSMap version 21 2024-03-28T22:10:57.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.509+0000 7fbc1703a740 1 -- 172.21.15.16:0/3264105682 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5565e30e41f0 con 0x5565e3060470 2024-03-28T22:10:57.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.510+0000 7fbbf5ffb640 1 -- 172.21.15.16:0/3264105682 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc04009480 con 0x5565e3060470 2024-03-28T22:10:57.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.510+0000 7fbbf5ffb640 1 -- 172.21.15.16:0/3264105682 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fbc0403d930 con 0x5565e3060470 2024-03-28T22:10:57.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.510+0000 7fbc1703a740 10 client.25487 fetch_fsmap finished waiting for FSMap version 21 2024-03-28T22:10:57.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.510+0000 7fbc1703a740 10 client.25487 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T22:10:57.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.510+0000 7fbc1703a740 20 client.25487 populate_metadata read hostname 'smithi016' 2024-03-28T22:10:57.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.510+0000 7fbc1703a740 1 --2- 172.21.15.16:0/3264105682 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5565e30e4860 0x5565e30e6c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.510+0000 7fbc1703a740 4 client.25487 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:10:57.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.510+0000 7fbc1703a740 1 -- 172.21.15.16:0/3264105682 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5565e2fcba40 con 0x5565e30e4860 2024-03-28T22:10:57.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.510+0000 7fbc12116640 1 --2- 172.21.15.16:0/3264105682 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5565e30e4860 0x5565e30e6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.510+0000 7fbbf5ffb640 10 client.25487 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:10:57.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.510+0000 7fbc12116640 1 --2- 172.21.15.16:0/3264105682 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5565e30e4860 0x5565e30e6c50 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d71c00bf827da2fe server_cookie=222a61dc105483f0 in_seq=0 out_seq=0 2024-03-28T22:10:57.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.510+0000 7fbbf5ffb640 1 -- 172.21.15.16:0/3264105682 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5565e2fcba40 con 0x5565e30e4860 2024-03-28T22:10:57.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.510+0000 7fbbf5ffb640 10 client.25487 handle_command_reply: tid=0 2024-03-28T22:10:57.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.510+0000 7fbc1703a740 1 client.25487 shutdown 2024-03-28T22:10:57.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.511+0000 7fbc1703a740 1 -- 172.21.15.16:0/3264105682 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3060470 msgr2=0x5565e30d7c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:57.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.511+0000 7fbc1703a740 1 --2- 172.21.15.16:0/3264105682 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3060470 0x5565e30d7c90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbc04009db0 tx=0x7fbc04031040 comp rx=0 tx=0).stop 2024-03-28T22:10:57.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.511+0000 7fbc1703a740 1 -- 172.21.15.16:0/3264105682 shutdown_connections 2024-03-28T22:10:57.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.511+0000 7fbc1703a740 1 --2- 172.21.15.16:0/3264105682 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5565e30e4860 0x5565e30e6c50 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.511+0000 7fbc1703a740 1 --2- 172.21.15.16:0/3264105682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565e3069970 0x5565e30d8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.511+0000 7fbbf5ffb640 0 client.25487 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:10:57.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.511+0000 7fbc1703a740 1 --2- 172.21.15.16:0/3264105682 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565e3060d90 0x5565e30d81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.511+0000 7fbc1703a740 1 --2- 172.21.15.16:0/3264105682 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3060470 0x5565e30d7c90 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.511+0000 7fbc1703a740 1 -- 172.21.15.16:0/3264105682 >> 172.21.15.16:0/3264105682 conn(0x5565e305aba0 msgr2=0x5565e305bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:57.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.511+0000 7fbc1703a740 1 -- 172.21.15.16:0/3264105682 shutdown_connections 2024-03-28T22:10:57.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.511+0000 7fbc1703a740 1 -- 172.21.15.16:0/3264105682 wait complete. 2024-03-28T22:10:57.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.511+0000 7fbc1703a740 20 client.25487 trim_cache size 0 max 16384 2024-03-28T22:10:57.520 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:10:57.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.511+0000 7fbc1437a640 1 -- 172.21.15.16:0/1070159542 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fbbf80a1f80 msgr2=0x7fbbf80a4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:57.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.511+0000 7fbc1437a640 1 --2- 172.21.15.16:0/1070159542 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fbbf80a1f80 0x7fbbf80a4440 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc0cb380 tx=0x7fbbfc016f00 comp rx=0 tx=0).stop 2024-03-28T22:10:57.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.511+0000 7fbc1437a640 1 -- 172.21.15.16:0/1070159542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc0c1020a0 msgr2=0x7fbc0c1985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:57.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.511+0000 7fbc1437a640 1 --2- 172.21.15.16:0/1070159542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc0c1020a0 0x7fbc0c1985e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc00b4a0 tx=0x7fbbfc00b8a0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.511+0000 7fbc1437a640 1 -- 172.21.15.16:0/1070159542 shutdown_connections 2024-03-28T22:10:57.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.511+0000 7fbc1437a640 1 --2- 172.21.15.16:0/1070159542 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fbbf80a1f80 0x7fbbf80a4440 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.511+0000 7fbc1437a640 1 --2- 172.21.15.16:0/1070159542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc0c1076f0 0x7fbc0c19c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.511+0000 7fbc1437a640 1 --2- 172.21.15.16:0/1070159542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc0c1020a0 0x7fbc0c1985e0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.511+0000 7fbc1437a640 1 --2- 172.21.15.16:0/1070159542 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc0c1016e0 0x7fbc0c1980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.511+0000 7fbc1437a640 1 -- 172.21.15.16:0/1070159542 >> 172.21.15.16:0/1070159542 conn(0x7fbc0c0fd1f0 msgr2=0x7fbc0c0fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:57.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.511+0000 7fbc1437a640 1 -- 172.21.15.16:0/1070159542 shutdown_connections 2024-03-28T22:10:57.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.511+0000 7fbc1437a640 1 -- 172.21.15.16:0/1070159542 wait complete. 2024-03-28T22:10:57.523 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:10:57.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.627+0000 7f7676910640 1 -- 172.21.15.16:0/4019021106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76701067c0 msgr2=0x7f7670106ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:57.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.627+0000 7f7676910640 1 --2- 172.21.15.16:0/4019021106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76701067c0 0x7f7670106ba0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f7660009930 tx=0x7f766002bc30 comp rx=0 tx=0).stop 2024-03-28T22:10:57.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.627+0000 7f7676910640 1 -- 172.21.15.16:0/4019021106 shutdown_connections 2024-03-28T22:10:57.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.627+0000 7f7676910640 1 --2- 172.21.15.16:0/4019021106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7670100fd0 0x7f767010d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.627+0000 7f7676910640 1 --2- 172.21.15.16:0/4019021106 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7670100610 0x7f7670100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.627+0000 7f7676910640 1 --2- 172.21.15.16:0/4019021106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76701067c0 0x7f7670106ba0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.627+0000 7f7676910640 1 -- 172.21.15.16:0/4019021106 >> 172.21.15.16:0/4019021106 conn(0x7f76700777c0 msgr2=0x7f7670077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:57.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.627+0000 7f7676910640 1 -- 172.21.15.16:0/4019021106 shutdown_connections 2024-03-28T22:10:57.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.627+0000 7f7676910640 1 -- 172.21.15.16:0/4019021106 wait complete. 2024-03-28T22:10:57.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.627+0000 7f7676910640 1 Processor -- start 2024-03-28T22:10:57.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.627+0000 7f7676910640 1 -- start start 2024-03-28T22:10:57.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.628+0000 7f7676910640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7670100610 0x7f7670198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.628+0000 7f7676910640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7670100fd0 0x7f7670198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.628+0000 7f7676910640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76701067c0 0x7f767019c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.628+0000 7f7676910640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7670111c40 con 0x7f7670100fd0 2024-03-28T22:10:57.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.628+0000 7f7676910640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7670111ac0 con 0x7f76701067c0 2024-03-28T22:10:57.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.628+0000 7f7676910640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7670111dc0 con 0x7f7670100610 2024-03-28T22:10:57.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.628+0000 7f766f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7670100fd0 0x7f7670198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.628+0000 7f7674ead640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76701067c0 0x7f767019c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.628+0000 7f766f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7670100fd0 0x7f7670198660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37676/0 (socket says 172.21.15.16:37676) 2024-03-28T22:10:57.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.628+0000 7f766f7fe640 1 -- 172.21.15.16:0/3094422624 learned_addr learned my addr 172.21.15.16:0/3094422624 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:57.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.628+0000 7f766ffff640 1 --2- 172.21.15.16:0/3094422624 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7670100610 0x7f7670198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.628+0000 7f766f7fe640 1 -- 172.21.15.16:0/3094422624 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7670100610 msgr2=0x7f7670198120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:57.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.628+0000 7f766f7fe640 1 --2- 172.21.15.16:0/3094422624 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7670100610 0x7f7670198120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.628+0000 7f766f7fe640 1 -- 172.21.15.16:0/3094422624 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76701067c0 msgr2=0x7f767019c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:57.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.628+0000 7f766f7fe640 1 --2- 172.21.15.16:0/3094422624 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76701067c0 0x7f767019c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.628+0000 7f766f7fe640 1 -- 172.21.15.16:0/3094422624 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f767019d0c0 con 0x7f7670100fd0 2024-03-28T22:10:57.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.628+0000 7f766f7fe640 1 --2- 172.21.15.16:0/3094422624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7670100fd0 0x7f7670198660 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f766400c970 tx=0x7f766400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:57.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.629+0000 7f766d7fa640 1 -- 172.21.15.16:0/3094422624 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7664007bb0 con 0x7f7670100fd0 2024-03-28T22:10:57.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.629+0000 7f7676910640 1 -- 172.21.15.16:0/3094422624 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f767019d3b0 con 0x7f7670100fd0 2024-03-28T22:10:57.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.629+0000 7f766d7fa640 1 -- 172.21.15.16:0/3094422624 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7664007d50 con 0x7f7670100fd0 2024-03-28T22:10:57.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.629+0000 7f766d7fa640 1 -- 172.21.15.16:0/3094422624 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7664018800 con 0x7f7670100fd0 2024-03-28T22:10:57.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.629+0000 7f7676910640 1 -- 172.21.15.16:0/3094422624 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76701a4c40 con 0x7f7670100fd0 2024-03-28T22:10:57.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.631+0000 7f766d7fa640 1 -- 172.21.15.16:0/3094422624 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 136932+0+0 (secure 0 0 0) 0x7f76640189e0 con 0x7f7670100fd0 2024-03-28T22:10:57.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.631+0000 7f76795cd740 1 Processor -- start 2024-03-28T22:10:57.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.631+0000 7f76795cd740 1 -- start start 2024-03-28T22:10:57.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.631+0000 7f76795cd740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561138962470 0x561138962850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.631+0000 7f76795cd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561138962d90 0x56113896b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.631+0000 7f76795cd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56113896b970 0x56113896dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.631+0000 7f76795cd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5611388c61b0 con 0x56113896b970 2024-03-28T22:10:57.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.631+0000 7f76795cd740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x56113886ec00 con 0x561138962470 2024-03-28T22:10:57.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.631+0000 7f76795cd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5611388c2250 con 0x561138962d90 2024-03-28T22:10:57.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.631+0000 7f766d7fa640 1 --2- 172.21.15.16:0/3094422624 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f76580a1f80 0x7f76580a4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.631+0000 7f766f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56113896b970 0x56113896dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.631+0000 7f7674ead640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561138962470 0x561138962850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.632+0000 7f7674ead640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561138962470 0x561138962850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:48154/0 (socket says 172.21.15.16:48154) 2024-03-28T22:10:57.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.632+0000 7f7674ead640 1 -- 172.21.15.16:0/2295883309 learned_addr learned my addr 172.21.15.16:0/2295883309 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:57.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.632+0000 7f766ffff640 1 --2- 172.21.15.16:0/2295883309 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561138962d90 0x56113896b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.632+0000 7f7674ead640 1 --2- 172.21.15.16:0/3094422624 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f76580a1f80 0x7f76580a4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.632+0000 7f766d7fa640 1 -- 172.21.15.16:0/3094422624 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f76640ce450 con 0x7f7670100fd0 2024-03-28T22:10:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.632+0000 7f7674ead640 1 -- 172.21.15.16:0/2295883309 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561138962d90 msgr2=0x56113896b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.632+0000 7f7674ead640 1 --2- 172.21.15.16:0/2295883309 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561138962d90 0x56113896b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.632+0000 7f7674ead640 1 -- 172.21.15.16:0/2295883309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56113896b970 msgr2=0x56113896dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.632+0000 7f7674ead640 1 --2- 172.21.15.16:0/2295883309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56113896b970 0x56113896dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.632+0000 7f7674ead640 1 -- 172.21.15.16:0/2295883309 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56113888e3d0 con 0x561138962470 2024-03-28T22:10:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.632+0000 7f7674ead640 1 --2- 172.21.15.16:0/2295883309 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561138962470 0x561138962850 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7668011e00 tx=0x7f766800f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b9403f256ceeb99 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.632+0000 7f7674ead640 1 --2- 172.21.15.16:0/3094422624 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f76580a1f80 0x7f76580a4440 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f766800fc90 tx=0x7f766801e000 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.632+0000 7f76567fc640 1 -- 172.21.15.16:0/2295883309 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f766801dcb0 con 0x561138962470 2024-03-28T22:10:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.632+0000 7f76567fc640 1 -- 172.21.15.16:0/2295883309 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f766801de50 con 0x561138962470 2024-03-28T22:10:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.632+0000 7f76567fc640 1 -- 172.21.15.16:0/2295883309 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7668004300 con 0x561138962470 2024-03-28T22:10:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.633+0000 7f76795cd740 1 -- 172.21.15.16:0/2295883309 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561138962470 msgr2=0x561138962850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.633+0000 7f76795cd740 1 --2- 172.21.15.16:0/2295883309 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561138962470 0x561138962850 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7668011e00 tx=0x7f766800f790 comp rx=0 tx=0).stop 2024-03-28T22:10:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.633+0000 7f76795cd740 1 -- 172.21.15.16:0/2295883309 shutdown_connections 2024-03-28T22:10:57.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.633+0000 7f76795cd740 1 --2- 172.21.15.16:0/2295883309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56113896b970 0x56113896dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.633+0000 7f76795cd740 1 --2- 172.21.15.16:0/2295883309 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561138962d90 0x56113896b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.633+0000 7f76795cd740 1 --2- 172.21.15.16:0/2295883309 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561138962470 0x561138962850 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.633+0000 7f76795cd740 1 -- 172.21.15.16:0/2295883309 >> 172.21.15.16:0/2295883309 conn(0x56113895cba0 msgr2=0x56113895cf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:57.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.633+0000 7f76795cd740 1 -- 172.21.15.16:0/2295883309 shutdown_connections 2024-03-28T22:10:57.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.633+0000 7f76795cd740 1 -- 172.21.15.16:0/2295883309 wait complete. 2024-03-28T22:10:57.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.633+0000 7f76795cd740 1 -- start start 2024-03-28T22:10:57.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.633+0000 7f76795cd740 1 Processor -- start 2024-03-28T22:10:57.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.634+0000 7f76795cd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561138962470 0x561138968ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.634+0000 7f76795cd740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561138962d90 0x561138968ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.634+0000 7f76795cd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56113896b970 0x561138969530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.634+0000 7f76795cd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56113896e6a0 con 0x56113896b970 2024-03-28T22:10:57.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.634+0000 7f76795cd740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x561138864470 con 0x561138962d90 2024-03-28T22:10:57.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.634+0000 7f76795cd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56113896e820 con 0x561138962470 2024-03-28T22:10:57.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.634+0000 7f76795cd740 10 client.0 _ll_register_callbacks cb 0x561138881cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:10:57.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.634+0000 7f7674ead640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56113896b970 0x561138969530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.634+0000 7f7674ead640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56113896b970 0x561138969530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37686/0 (socket says 172.21.15.16:37686) 2024-03-28T22:10:57.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.634+0000 7f7674ead640 1 -- 172.21.15.16:0/2426456322 learned_addr learned my addr 172.21.15.16:0/2426456322 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:57.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.634+0000 7f766f7fe640 1 --2- 172.21.15.16:0/2426456322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561138962d90 0x561138968ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.634+0000 7f766ffff640 1 --2- 172.21.15.16:0/2426456322 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561138962470 0x561138968ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.634+0000 7f7674ead640 1 -- 172.21.15.16:0/2426456322 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561138962470 msgr2=0x561138968ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:57.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.634+0000 7f7674ead640 1 --2- 172.21.15.16:0/2426456322 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561138962470 0x561138968ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.634+0000 7f7674ead640 1 -- 172.21.15.16:0/2426456322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561138962d90 msgr2=0x561138968ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:57.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.634+0000 7f7674ead640 1 --2- 172.21.15.16:0/2426456322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561138962d90 0x561138968ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.634+0000 7f7674ead640 1 -- 172.21.15.16:0/2426456322 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611385da3f0 con 0x56113896b970 2024-03-28T22:10:57.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.634+0000 7f766ffff640 1 --2- 172.21.15.16:0/2426456322 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561138962470 0x561138968ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:10:57.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.634+0000 7f7674ead640 1 --2- 172.21.15.16:0/2426456322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56113896b970 0x561138969530 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f7668003ec0 tx=0x7f7668005620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e46667efeccd5625 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.634+0000 7f764f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:10:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.634+0000 7f764f7fe640 1 -- 172.21.15.16:0/2426456322 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7668033030 con 0x56113896b970 2024-03-28T22:10:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.635+0000 7f764f7fe640 1 -- 172.21.15.16:0/2426456322 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f766802b040 con 0x56113896b970 2024-03-28T22:10:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.635+0000 7f76795cd740 1 -- 172.21.15.16:0/2426456322 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f766801de50 con 0x56113896b970 2024-03-28T22:10:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.635+0000 7f764f7fe640 1 -- 172.21.15.16:0/2426456322 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f766803d050 con 0x56113896b970 2024-03-28T22:10:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.635+0000 7f76795cd740 10 client.15591 fetch_fsmap learned FSMap version 21 2024-03-28T22:10:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.635+0000 7f76795cd740 1 -- 172.21.15.16:0/2426456322 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5611386a47d0 con 0x56113896b970 2024-03-28T22:10:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.635+0000 7f764f7fe640 1 -- 172.21.15.16:0/2426456322 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f766801de50 con 0x56113896b970 2024-03-28T22:10:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.635+0000 7f764f7fe640 1 -- 172.21.15.16:0/2426456322 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f7668005810 con 0x56113896b970 2024-03-28T22:10:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.635+0000 7f76795cd740 10 client.15591 fetch_fsmap finished waiting for FSMap version 21 2024-03-28T22:10:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.635+0000 7f76795cd740 10 client.15591 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:10:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.635+0000 7f76795cd740 20 client.15591 populate_metadata read hostname 'smithi016' 2024-03-28T22:10:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.635+0000 7f76795cd740 1 --2- 172.21.15.16:0/2426456322 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x56113897b9b0 0x56113897dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.635+0000 7f76795cd740 4 client.15591 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:10:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.635+0000 7f76795cd740 1 -- 172.21.15.16:0/2426456322 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5611388d36f0 con 0x56113897b9b0 2024-03-28T22:10:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.635+0000 7f766ffff640 1 --2- 172.21.15.16:0/2426456322 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x56113897b9b0 0x56113897dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.636+0000 7f764f7fe640 10 client.15591 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:10:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.636+0000 7f766ffff640 1 --2- 172.21.15.16:0/2426456322 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x56113897b9b0 0x56113897dda0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f0dfcc02262f93f server_cookie=7a58fc8f66f835b9 in_seq=0 out_seq=0 2024-03-28T22:10:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.637+0000 7f764f7fe640 1 -- 172.21.15.16:0/2426456322 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5611388d36f0 con 0x56113897b9b0 2024-03-28T22:10:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.637+0000 7f764f7fe640 10 client.15591 handle_command_reply: tid=0 2024-03-28T22:10:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.637+0000 7f76795cd740 1 client.15591 shutdown 2024-03-28T22:10:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.637+0000 7f76795cd740 1 -- 172.21.15.16:0/2426456322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56113896b970 msgr2=0x561138969530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.637+0000 7f76795cd740 1 --2- 172.21.15.16:0/2426456322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56113896b970 0x561138969530 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f7668003ec0 tx=0x7f7668005620 comp rx=0 tx=0).stop 2024-03-28T22:10:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.637+0000 7f76795cd740 1 -- 172.21.15.16:0/2426456322 shutdown_connections 2024-03-28T22:10:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.637+0000 7f76795cd740 1 --2- 172.21.15.16:0/2426456322 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x56113897b9b0 0x56113897dda0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.637+0000 7f76795cd740 1 --2- 172.21.15.16:0/2426456322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56113896b970 0x561138969530 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.637+0000 7f76795cd740 1 --2- 172.21.15.16:0/2426456322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561138962d90 0x561138968ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.637+0000 7f764f7fe640 0 client.15591 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:10:57.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.637+0000 7f76795cd740 1 --2- 172.21.15.16:0/2426456322 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561138962470 0x561138968ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.637+0000 7f76795cd740 1 -- 172.21.15.16:0/2426456322 >> 172.21.15.16:0/2426456322 conn(0x56113895cba0 msgr2=0x56113895e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.637+0000 7f76795cd740 1 -- 172.21.15.16:0/2426456322 shutdown_connections 2024-03-28T22:10:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.637+0000 7f76795cd740 1 -- 172.21.15.16:0/2426456322 wait complete. 2024-03-28T22:10:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.637+0000 7f76795cd740 20 client.15591 trim_cache size 0 max 16384 2024-03-28T22:10:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.650+0000 7f76795cd740 1 Processor -- start 2024-03-28T22:10:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.650+0000 7f76795cd740 1 -- start start 2024-03-28T22:10:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.650+0000 7f76795cd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561138962470 0x56113896b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.650+0000 7f76795cd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561138962d90 0x561138977440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.650+0000 7f76795cd740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56113896b970 0x56113895bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.650+0000 7f76795cd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f76701afbf0 con 0x561138962d90 2024-03-28T22:10:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.650+0000 7f76795cd740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f76701ad4c0 con 0x56113896b970 2024-03-28T22:10:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.650+0000 7f76795cd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f76701ab310 con 0x561138962470 2024-03-28T22:10:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.651+0000 7f766f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561138962d90 0x561138977440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.651+0000 7f766f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561138962d90 0x561138977440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37696/0 (socket says 172.21.15.16:37696) 2024-03-28T22:10:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.651+0000 7f766f7fe640 1 -- 172.21.15.16:0/1525077013 learned_addr learned my addr 172.21.15.16:0/1525077013 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.651+0000 7f7674ead640 1 --2- 172.21.15.16:0/1525077013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56113896b970 0x56113895bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.651+0000 7f766ffff640 1 --2- 172.21.15.16:0/1525077013 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561138962470 0x56113896b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.651+0000 7f766f7fe640 1 -- 172.21.15.16:0/1525077013 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561138962470 msgr2=0x56113896b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:57.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.651+0000 7f766f7fe640 1 --2- 172.21.15.16:0/1525077013 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561138962470 0x56113896b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.651+0000 7f766f7fe640 1 -- 172.21.15.16:0/1525077013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56113896b970 msgr2=0x56113895bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:57.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.651+0000 7f766f7fe640 1 --2- 172.21.15.16:0/1525077013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56113896b970 0x56113895bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.651+0000 7f766f7fe640 1 -- 172.21.15.16:0/1525077013 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56113897e110 con 0x561138962d90 2024-03-28T22:10:57.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.651+0000 7f766f7fe640 1 --2- 172.21.15.16:0/1525077013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561138962d90 0x561138977440 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f7664097660 tx=0x7f766400fea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89f0e8c8a6c43c7e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:57.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.651+0000 7f7655ffb640 1 -- 172.21.15.16:0/1525077013 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76640b3030 con 0x561138962d90 2024-03-28T22:10:57.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.651+0000 7f7655ffb640 1 -- 172.21.15.16:0/1525077013 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f766409ede0 con 0x561138962d90 2024-03-28T22:10:57.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.651+0000 7f7655ffb640 1 -- 172.21.15.16:0/1525077013 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76640cbd50 con 0x561138962d90 2024-03-28T22:10:57.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.652+0000 7f76795cd740 1 -- 172.21.15.16:0/1525077013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561138962d90 msgr2=0x561138977440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:57.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.652+0000 7f76795cd740 1 --2- 172.21.15.16:0/1525077013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561138962d90 0x561138977440 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f7664097660 tx=0x7f766400fea0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.652+0000 7f76795cd740 1 -- 172.21.15.16:0/1525077013 shutdown_connections 2024-03-28T22:10:57.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.652+0000 7f76795cd740 1 --2- 172.21.15.16:0/1525077013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56113896b970 0x56113895bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.652+0000 7f76795cd740 1 --2- 172.21.15.16:0/1525077013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561138962d90 0x561138977440 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.652+0000 7f76795cd740 1 --2- 172.21.15.16:0/1525077013 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561138962470 0x56113896b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.652+0000 7f76795cd740 1 -- 172.21.15.16:0/1525077013 >> 172.21.15.16:0/1525077013 conn(0x56113895cba0 msgr2=0x56113895b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:57.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.652+0000 7f76795cd740 1 -- 172.21.15.16:0/1525077013 shutdown_connections 2024-03-28T22:10:57.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.652+0000 7f76795cd740 1 -- 172.21.15.16:0/1525077013 wait complete. 2024-03-28T22:10:57.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.652+0000 7f76795cd740 1 -- start start 2024-03-28T22:10:57.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.652+0000 7f76795cd740 1 Processor -- start 2024-03-28T22:10:57.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.653+0000 7f76795cd740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561138962470 0x561138973d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.653+0000 7f76795cd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561138962d90 0x5611389653d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.653+0000 7f76795cd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56113896b970 0x561138965990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.653+0000 7f76795cd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f76701ab980 con 0x561138962d90 2024-03-28T22:10:57.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.653+0000 7f76795cd740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f76701acdf0 con 0x561138962470 2024-03-28T22:10:57.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.653+0000 7f76795cd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f76701ac050 con 0x56113896b970 2024-03-28T22:10:57.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.653+0000 7f76795cd740 10 client.0 _ll_register_callbacks cb 0x7f7668011c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:10:57.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.653+0000 7f766f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561138962d90 0x5611389653d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.653+0000 7f766ffff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561138962470 0x561138973d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.653+0000 7f766f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561138962d90 0x5611389653d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37700/0 (socket says 172.21.15.16:37700) 2024-03-28T22:10:57.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.653+0000 7f766f7fe640 1 -- 172.21.15.16:0/2724927480 learned_addr learned my addr 172.21.15.16:0/2724927480 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:57.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.653+0000 7f7674ead640 1 --2- 172.21.15.16:0/2724927480 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56113896b970 0x561138965990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.653+0000 7f766f7fe640 1 -- 172.21.15.16:0/2724927480 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56113896b970 msgr2=0x561138965990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:57.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.653+0000 7f766f7fe640 1 --2- 172.21.15.16:0/2724927480 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56113896b970 0x561138965990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.653+0000 7f766f7fe640 1 -- 172.21.15.16:0/2724927480 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561138962470 msgr2=0x561138973d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:57.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.653+0000 7f766f7fe640 1 --2- 172.21.15.16:0/2724927480 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561138962470 0x561138973d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.653+0000 7f766f7fe640 1 -- 172.21.15.16:0/2724927480 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611389661c0 con 0x561138962d90 2024-03-28T22:10:57.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.653+0000 7f7674ead640 1 --2- 172.21.15.16:0/2724927480 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56113896b970 0x561138965990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:10:57.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.653+0000 7f766ffff640 1 --2- 172.21.15.16:0/2724927480 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561138962470 0x561138973d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:10:57.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.653+0000 7f766f7fe640 1 --2- 172.21.15.16:0/2724927480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561138962d90 0x5611389653d0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f766409a990 tx=0x7f766409aae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6fad5d07ff0351 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:57.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.653+0000 7f764f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:10:57.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.653+0000 7f764f7fe640 1 -- 172.21.15.16:0/2724927480 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7664065060 con 0x561138962d90 2024-03-28T22:10:57.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.653+0000 7f764f7fe640 1 -- 172.21.15.16:0/2724927480 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f7664097300 con 0x561138962d90 2024-03-28T22:10:57.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.653+0000 7f76795cd740 1 -- 172.21.15.16:0/2724927480 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f766409ede0 con 0x561138962d90 2024-03-28T22:10:57.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.653+0000 7f764f7fe640 1 -- 172.21.15.16:0/2724927480 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f76640b3070 con 0x561138962d90 2024-03-28T22:10:57.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.653+0000 7f76795cd740 10 client.15597 fetch_fsmap learned FSMap version 21 2024-03-28T22:10:57.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.653+0000 7f76795cd740 1 -- 172.21.15.16:0/2724927480 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5611389e1ca0 con 0x561138962d90 2024-03-28T22:10:57.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.654+0000 7f764f7fe640 1 -- 172.21.15.16:0/2724927480 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f766409ede0 con 0x561138962d90 2024-03-28T22:10:57.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.654+0000 7f764f7fe640 1 -- 172.21.15.16:0/2724927480 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f76640aece0 con 0x561138962d90 2024-03-28T22:10:57.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.654+0000 7f76795cd740 10 client.15597 fetch_fsmap finished waiting for FSMap version 21 2024-03-28T22:10:57.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.654+0000 7f76795cd740 10 client.15597 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:10:57.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.654+0000 7f76795cd740 20 client.15597 populate_metadata read hostname 'smithi016' 2024-03-28T22:10:57.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.654+0000 7f76795cd740 1 --2- 172.21.15.16:0/2724927480 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5611389e23b0 0x5611389e47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:57.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.654+0000 7f76795cd740 4 client.15597 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:10:57.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.654+0000 7f76795cd740 1 -- 172.21.15.16:0/2724927480 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5611388cda40 con 0x5611389e23b0 2024-03-28T22:10:57.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.654+0000 7f766ffff640 1 --2- 172.21.15.16:0/2724927480 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5611389e23b0 0x5611389e47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:57.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.654+0000 7f764f7fe640 10 client.15597 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:10:57.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.654+0000 7f766ffff640 1 --2- 172.21.15.16:0/2724927480 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5611389e23b0 0x5611389e47d0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=338a3f5da463e1b1 server_cookie=e7f07d991bcd5946 in_seq=0 out_seq=0 2024-03-28T22:10:57.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.654+0000 7f764f7fe640 1 -- 172.21.15.16:0/2724927480 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5611388cda40 con 0x5611389e23b0 2024-03-28T22:10:57.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.654+0000 7f764f7fe640 10 client.15597 handle_command_reply: tid=0 2024-03-28T22:10:57.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.655+0000 7f76795cd740 1 client.15597 shutdown 2024-03-28T22:10:57.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.655+0000 7f76795cd740 1 -- 172.21.15.16:0/2724927480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561138962d90 msgr2=0x5611389653d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:57.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.655+0000 7f76795cd740 1 --2- 172.21.15.16:0/2724927480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561138962d90 0x5611389653d0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f766409a990 tx=0x7f766409aae0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.655+0000 7f76795cd740 1 -- 172.21.15.16:0/2724927480 shutdown_connections 2024-03-28T22:10:57.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.655+0000 7f76795cd740 1 --2- 172.21.15.16:0/2724927480 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5611389e23b0 0x5611389e47d0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.655+0000 7f76795cd740 1 --2- 172.21.15.16:0/2724927480 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56113896b970 0x561138965990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.655+0000 7f76795cd740 1 --2- 172.21.15.16:0/2724927480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561138962d90 0x5611389653d0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.655+0000 7f764f7fe640 0 client.15597 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:10:57.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.655+0000 7f76795cd740 1 --2- 172.21.15.16:0/2724927480 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561138962470 0x561138973d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.655+0000 7f76795cd740 1 -- 172.21.15.16:0/2724927480 >> 172.21.15.16:0/2724927480 conn(0x56113895cba0 msgr2=0x561138967b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:57.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.655+0000 7f76795cd740 1 -- 172.21.15.16:0/2724927480 shutdown_connections 2024-03-28T22:10:57.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.655+0000 7f76795cd740 1 -- 172.21.15.16:0/2724927480 wait complete. 2024-03-28T22:10:57.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.655+0000 7f76795cd740 20 client.15597 trim_cache size 0 max 16384 2024-03-28T22:10:57.667 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:10:57.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.655+0000 7f7676910640 1 -- 172.21.15.16:0/3094422624 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f76580a1f80 msgr2=0x7f76580a4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:57.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.655+0000 7f7676910640 1 --2- 172.21.15.16:0/3094422624 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f76580a1f80 0x7f76580a4440 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f766800fc90 tx=0x7f766801e000 comp rx=0 tx=0).stop 2024-03-28T22:10:57.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.655+0000 7f7676910640 1 -- 172.21.15.16:0/3094422624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7670100fd0 msgr2=0x7f7670198660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:57.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.655+0000 7f7676910640 1 --2- 172.21.15.16:0/3094422624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7670100fd0 0x7f7670198660 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f766400c970 tx=0x7f766400ce30 comp rx=0 tx=0).stop 2024-03-28T22:10:57.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.655+0000 7f7676910640 1 -- 172.21.15.16:0/3094422624 shutdown_connections 2024-03-28T22:10:57.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.655+0000 7f7676910640 1 --2- 172.21.15.16:0/3094422624 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f76580a1f80 0x7f76580a4440 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.655+0000 7f7676910640 1 --2- 172.21.15.16:0/3094422624 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76701067c0 0x7f767019c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.655+0000 7f7676910640 1 --2- 172.21.15.16:0/3094422624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7670100fd0 0x7f7670198660 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.655+0000 7f7676910640 1 --2- 172.21.15.16:0/3094422624 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7670100610 0x7f7670198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:57.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.655+0000 7f7676910640 1 -- 172.21.15.16:0/3094422624 >> 172.21.15.16:0/3094422624 conn(0x7f76700777c0 msgr2=0x7f76700ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:57.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.656+0000 7f7676910640 1 -- 172.21.15.16:0/3094422624 shutdown_connections 2024-03-28T22:10:57.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:57.656+0000 7f7676910640 1 -- 172.21.15.16:0/3094422624 wait complete. 2024-03-28T22:10:58.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:58 smithi053 ceph-mon[35979]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:10:58.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:58 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2077102799' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:10:58.670 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:10:58.670 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:10:58.670 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:10:58.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:58 smithi063 ceph-mon[36249]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:10:58.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:58 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2077102799' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:10:58.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:58 smithi016 ceph-mon[32374]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:10:58.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:58 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2077102799' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:10:58.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.788+0000 7f9981e49640 1 -- 172.21.15.16:0/1387093417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f997c1020a0 msgr2=0x7f997c10e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:58.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.788+0000 7f9981e49640 1 --2- 172.21.15.16:0/1387093417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f997c1020a0 0x7f997c10e5b0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f9974009930 tx=0x7f997402bc30 comp rx=0 tx=0).stop 2024-03-28T22:10:58.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.788+0000 7f9981e49640 1 -- 172.21.15.16:0/1387093417 shutdown_connections 2024-03-28T22:10:58.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.788+0000 7f9981e49640 1 --2- 172.21.15.16:0/1387093417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f997c1020a0 0x7f997c10e5b0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:58.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.788+0000 7f9981e49640 1 --2- 172.21.15.16:0/1387093417 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f997c1016e0 0x7f997c101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:58.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.788+0000 7f9981e49640 1 --2- 172.21.15.16:0/1387093417 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f997c1076f0 0x7f997c107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:58.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.788+0000 7f9981e49640 1 -- 172.21.15.16:0/1387093417 >> 172.21.15.16:0/1387093417 conn(0x7f997c0fd1f0 msgr2=0x7f997c0ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:58.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.788+0000 7f9981e49640 1 -- 172.21.15.16:0/1387093417 shutdown_connections 2024-03-28T22:10:58.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.788+0000 7f9981e49640 1 -- 172.21.15.16:0/1387093417 wait complete. 2024-03-28T22:10:58.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.789+0000 7f9981e49640 1 Processor -- start 2024-03-28T22:10:58.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.789+0000 7f9981e49640 1 -- start start 2024-03-28T22:10:58.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.789+0000 7f9981e49640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f997c1016e0 0x7f997c197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:58.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.789+0000 7f9981e49640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f997c1020a0 0x7f997c198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:58.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.789+0000 7f9981e49640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f997c1076f0 0x7f997c19c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:58.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.789+0000 7f9981e49640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f997c111b00 con 0x7f997c1020a0 2024-03-28T22:10:58.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.789+0000 7f9981e49640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f997c111980 con 0x7f997c1016e0 2024-03-28T22:10:58.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.789+0000 7f9981e49640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f997c111c80 con 0x7f997c1076f0 2024-03-28T22:10:58.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.789+0000 7f997affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f997c1020a0 0x7f997c198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:58.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.789+0000 7f997b7fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f997c1016e0 0x7f997c197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:58.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.789+0000 7f997affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f997c1020a0 0x7f997c198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37720/0 (socket says 172.21.15.16:37720) 2024-03-28T22:10:58.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.789+0000 7f997affd640 1 -- 172.21.15.16:0/1526520201 learned_addr learned my addr 172.21.15.16:0/1526520201 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:58.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.789+0000 7f997b7fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f997c1016e0 0x7f997c197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:48198/0 (socket says 172.21.15.16:48198) 2024-03-28T22:10:58.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.789+0000 7f997bfff640 1 --2- 172.21.15.16:0/1526520201 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f997c1076f0 0x7f997c19c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:58.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.790+0000 7f997affd640 1 -- 172.21.15.16:0/1526520201 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f997c1076f0 msgr2=0x7f997c19c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:58.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.790+0000 7f997affd640 1 --2- 172.21.15.16:0/1526520201 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f997c1076f0 0x7f997c19c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:58.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.790+0000 7f997affd640 1 -- 172.21.15.16:0/1526520201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f997c1016e0 msgr2=0x7f997c197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:58.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.790+0000 7f997affd640 1 --2- 172.21.15.16:0/1526520201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f997c1016e0 0x7f997c197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:58.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.790+0000 7f997affd640 1 -- 172.21.15.16:0/1526520201 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f997c19cfb0 con 0x7f997c1020a0 2024-03-28T22:10:58.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.790+0000 7f997bfff640 1 --2- 172.21.15.16:0/1526520201 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f997c1076f0 0x7f997c19c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:10:58.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.790+0000 7f997b7fe640 1 --2- 172.21.15.16:0/1526520201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f997c1016e0 0x7f997c197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:10:58.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.790+0000 7f997affd640 1 --2- 172.21.15.16:0/1526520201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f997c1020a0 0x7f997c198500 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f996400ea60 tx=0x7f996400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:58.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.790+0000 7f9978ff9640 1 -- 172.21.15.16:0/1526520201 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f996400cde0 con 0x7f997c1020a0 2024-03-28T22:10:58.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.790+0000 7f9978ff9640 1 -- 172.21.15.16:0/1526520201 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99640035f0 con 0x7f997c1020a0 2024-03-28T22:10:58.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.790+0000 7f9981e49640 1 -- 172.21.15.16:0/1526520201 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f997c19d2a0 con 0x7f997c1020a0 2024-03-28T22:10:58.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.790+0000 7f9978ff9640 1 -- 172.21.15.16:0/1526520201 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9964010670 con 0x7f997c1020a0 2024-03-28T22:10:58.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.790+0000 7f9981e49640 1 -- 172.21.15.16:0/1526520201 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f997c1a4b30 con 0x7f997c1020a0 2024-03-28T22:10:58.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.792+0000 7f9978ff9640 1 -- 172.21.15.16:0/1526520201 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 136932+0+0 (secure 0 0 0) 0x7f99640037a0 con 0x7f997c1020a0 2024-03-28T22:10:58.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.793+0000 7f9984a20740 1 Processor -- start 2024-03-28T22:10:58.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.793+0000 7f9984a20740 1 -- start start 2024-03-28T22:10:58.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.793+0000 7f9978ff9640 1 --2- 172.21.15.16:0/1526520201 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f99680a2000 0x7f99680a44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:58.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.793+0000 7f9984a20740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3e40e76a0 0x55a3e40e7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:58.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.793+0000 7f9984a20740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3e40e7fc0 0x55a3e40f4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:58.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.793+0000 7f9984a20740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3e40f5300 0x55a3e40f76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:58.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.793+0000 7f9984a20740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a3e3fe80b0 con 0x55a3e40e7fc0 2024-03-28T22:10:58.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.793+0000 7f9984a20740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55a3e404aea0 con 0x55a3e40e76a0 2024-03-28T22:10:58.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.793+0000 7f9984a20740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55a3e3ff3c00 con 0x55a3e40f5300 2024-03-28T22:10:58.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.793+0000 7f997bfff640 1 --2- 172.21.15.16:0/1526520201 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f99680a2000 0x7f99680a44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:58.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.793+0000 7f9978ff9640 1 -- 172.21.15.16:0/1526520201 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f99640ce6a0 con 0x7f997c1020a0 2024-03-28T22:10:58.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.793+0000 7f997b7fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3e40e76a0 0x55a3e40e7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:58.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.793+0000 7f997b7fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3e40e76a0 0x55a3e40e7a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:48206/0 (socket says 172.21.15.16:48206) 2024-03-28T22:10:58.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.793+0000 7f997b7fe640 1 -- 172.21.15.16:0/4146457559 learned_addr learned my addr 172.21.15.16:0/4146457559 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:58.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.793+0000 7f997affd640 1 --2- 172.21.15.16:0/4146457559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3e40e7fc0 0x55a3e40f4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:58.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.793+0000 7f997b7fe640 1 -- 172.21.15.16:0/4146457559 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3e40f5300 msgr2=0x55a3e40f76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:58.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.793+0000 7f997b7fe640 1 --2- 172.21.15.16:0/4146457559 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3e40f5300 0x55a3e40f76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:58.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.793+0000 7f997b7fe640 1 -- 172.21.15.16:0/4146457559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3e40e7fc0 msgr2=0x55a3e40f4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:58.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.793+0000 7f997b7fe640 1 --2- 172.21.15.16:0/4146457559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3e40e7fc0 0x55a3e40f4dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:58.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.793+0000 7f997b7fe640 1 -- 172.21.15.16:0/4146457559 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3e3e277e0 con 0x55a3e40e76a0 2024-03-28T22:10:58.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.794+0000 7f997b7fe640 1 --2- 172.21.15.16:0/4146457559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3e40e76a0 0x55a3e40e7a80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f996c010760 tx=0x7f996c010c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23184918926d51e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:58.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.794+0000 7f997bfff640 1 --2- 172.21.15.16:0/1526520201 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f99680a2000 0x7f99680a44c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9974036040 tx=0x7f997402c650 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:58.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.794+0000 7f9971ffb640 1 -- 172.21.15.16:0/4146457559 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f996c00ead0 con 0x55a3e40e76a0 2024-03-28T22:10:58.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.794+0000 7f9971ffb640 1 -- 172.21.15.16:0/4146457559 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f996c010e70 con 0x55a3e40e76a0 2024-03-28T22:10:58.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.794+0000 7f9971ffb640 1 -- 172.21.15.16:0/4146457559 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f996c01c670 con 0x55a3e40e76a0 2024-03-28T22:10:58.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.794+0000 7f9984a20740 1 -- 172.21.15.16:0/4146457559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3e40e76a0 msgr2=0x55a3e40e7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:58.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.794+0000 7f9984a20740 1 --2- 172.21.15.16:0/4146457559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3e40e76a0 0x55a3e40e7a80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f996c010760 tx=0x7f996c010c20 comp rx=0 tx=0).stop 2024-03-28T22:10:58.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.795+0000 7f9984a20740 1 -- 172.21.15.16:0/4146457559 shutdown_connections 2024-03-28T22:10:58.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.795+0000 7f9984a20740 1 --2- 172.21.15.16:0/4146457559 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3e40f5300 0x55a3e40f76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:58.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.795+0000 7f9984a20740 1 --2- 172.21.15.16:0/4146457559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3e40e7fc0 0x55a3e40f4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:58.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.795+0000 7f9984a20740 1 --2- 172.21.15.16:0/4146457559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3e40e76a0 0x55a3e40e7a80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:58.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.795+0000 7f9984a20740 1 -- 172.21.15.16:0/4146457559 >> 172.21.15.16:0/4146457559 conn(0x55a3e40e2120 msgr2=0x55a3e40e2500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:58.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.795+0000 7f9984a20740 1 -- 172.21.15.16:0/4146457559 shutdown_connections 2024-03-28T22:10:58.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.795+0000 7f9984a20740 1 -- 172.21.15.16:0/4146457559 wait complete. 2024-03-28T22:10:58.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.795+0000 7f9984a20740 1 -- start start 2024-03-28T22:10:58.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.795+0000 7f9984a20740 1 Processor -- start 2024-03-28T22:10:58.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.795+0000 7f9984a20740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3e40e76a0 0x55a3e40f22a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:58.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.795+0000 7f9984a20740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3e40e7fc0 0x55a3e40f27e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:58.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.795+0000 7f9984a20740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3e40f5300 0x55a3e40f2d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:58.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.795+0000 7f9984a20740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a3e3fe93d0 con 0x55a3e40f5300 2024-03-28T22:10:58.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.795+0000 7f9984a20740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55a3e4046f40 con 0x55a3e40e76a0 2024-03-28T22:10:58.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.795+0000 7f9984a20740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55a3e40f8140 con 0x55a3e40e7fc0 2024-03-28T22:10:58.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.795+0000 7f9984a20740 10 client.0 _ll_register_callbacks cb 0x55a3e4006b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:10:58.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.796+0000 7f997bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3e40f5300 0x55a3e40f2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:58.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.796+0000 7f997b7fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3e40e76a0 0x55a3e40f22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:58.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.796+0000 7f997bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3e40f5300 0x55a3e40f2d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37728/0 (socket says 172.21.15.16:37728) 2024-03-28T22:10:58.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.796+0000 7f997b7fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3e40e76a0 0x55a3e40f22a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:48214/0 (socket says 172.21.15.16:48214) 2024-03-28T22:10:58.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.796+0000 7f997bfff640 1 -- 172.21.15.16:0/1174649392 learned_addr learned my addr 172.21.15.16:0/1174649392 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:58.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.796+0000 7f997affd640 1 --2- 172.21.15.16:0/1174649392 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3e40e7fc0 0x55a3e40f27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:58.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.796+0000 7f997bfff640 1 -- 172.21.15.16:0/1174649392 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3e40e7fc0 msgr2=0x55a3e40f27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:58.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.796+0000 7f997bfff640 1 --2- 172.21.15.16:0/1174649392 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3e40e7fc0 0x55a3e40f27e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:58.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.796+0000 7f997bfff640 1 -- 172.21.15.16:0/1174649392 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3e40e76a0 msgr2=0x55a3e40f22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:58.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.796+0000 7f997bfff640 1 --2- 172.21.15.16:0/1174649392 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3e40e76a0 0x55a3e40f22a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:58.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.796+0000 7f997bfff640 1 -- 172.21.15.16:0/1174649392 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3e3e87f70 con 0x55a3e40f5300 2024-03-28T22:10:58.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.796+0000 7f997bfff640 1 --2- 172.21.15.16:0/1174649392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3e40f5300 0x55a3e40f2d20 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f9974004270 tx=0x7f9974009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aa13b0993d346fd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:58.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.796+0000 7f995affd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:10:58.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.796+0000 7f995affd640 1 -- 172.21.15.16:0/1174649392 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99740054d0 con 0x55a3e40f5300 2024-03-28T22:10:58.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.796+0000 7f995affd640 1 -- 172.21.15.16:0/1174649392 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f99740046c0 con 0x55a3e40f5300 2024-03-28T22:10:58.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.796+0000 7f9984a20740 1 -- 172.21.15.16:0/1174649392 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f996c010e70 con 0x55a3e40f5300 2024-03-28T22:10:58.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.796+0000 7f995affd640 1 -- 172.21.15.16:0/1174649392 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f9974058050 con 0x55a3e40f5300 2024-03-28T22:10:58.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.796+0000 7f9984a20740 10 client.15609 fetch_fsmap learned FSMap version 21 2024-03-28T22:10:58.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.796+0000 7f9984a20740 1 -- 172.21.15.16:0/1174649392 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55a3e3d1f170 con 0x55a3e40f5300 2024-03-28T22:10:58.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.797+0000 7f995affd640 1 -- 172.21.15.16:0/1174649392 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f996c010e70 con 0x55a3e40f5300 2024-03-28T22:10:58.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.797+0000 7f995affd640 1 -- 172.21.15.16:0/1174649392 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f9974003b50 con 0x55a3e40f5300 2024-03-28T22:10:58.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.797+0000 7f9984a20740 10 client.15609 fetch_fsmap finished waiting for FSMap version 21 2024-03-28T22:10:58.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.797+0000 7f9984a20740 10 client.15609 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:10:58.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.797+0000 7f9984a20740 20 client.15609 populate_metadata read hostname 'smithi016' 2024-03-28T22:10:58.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.797+0000 7f9984a20740 1 --2- 172.21.15.16:0/1174649392 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55a3e4105290 0x55a3e41076e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:58.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.797+0000 7f9984a20740 4 client.15609 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:10:58.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.797+0000 7f9984a20740 1 -- 172.21.15.16:0/1174649392 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a3e40583e0 con 0x55a3e4105290 2024-03-28T22:10:58.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.797+0000 7f997b7fe640 1 --2- 172.21.15.16:0/1174649392 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55a3e4105290 0x55a3e41076e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:58.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.797+0000 7f997b7fe640 1 --2- 172.21.15.16:0/1174649392 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55a3e4105290 0x55a3e41076e0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21810408345be7fa server_cookie=9acfaa9604547bb3 in_seq=0 out_seq=0 2024-03-28T22:10:58.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.797+0000 7f995affd640 10 client.15609 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:10:58.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.798+0000 7f995affd640 1 -- 172.21.15.16:0/1174649392 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a3e40583e0 con 0x55a3e4105290 2024-03-28T22:10:58.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.798+0000 7f995affd640 10 client.15609 handle_command_reply: tid=0 2024-03-28T22:10:58.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.798+0000 7f9984a20740 1 client.15609 shutdown 2024-03-28T22:10:58.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.798+0000 7f9984a20740 1 -- 172.21.15.16:0/1174649392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3e40f5300 msgr2=0x55a3e40f2d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:58.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.798+0000 7f9984a20740 1 --2- 172.21.15.16:0/1174649392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3e40f5300 0x55a3e40f2d20 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f9974004270 tx=0x7f9974009260 comp rx=0 tx=0).stop 2024-03-28T22:10:58.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.798+0000 7f9984a20740 1 -- 172.21.15.16:0/1174649392 shutdown_connections 2024-03-28T22:10:58.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.798+0000 7f9984a20740 1 --2- 172.21.15.16:0/1174649392 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55a3e4105290 0x55a3e41076e0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:58.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.798+0000 7f9984a20740 1 --2- 172.21.15.16:0/1174649392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3e40f5300 0x55a3e40f2d20 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:58.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.798+0000 7f9984a20740 1 --2- 172.21.15.16:0/1174649392 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3e40e7fc0 0x55a3e40f27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:58.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.798+0000 7f995affd640 0 client.15609 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:10:58.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.798+0000 7f9984a20740 1 --2- 172.21.15.16:0/1174649392 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3e40e76a0 0x55a3e40f22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:58.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.798+0000 7f9984a20740 1 -- 172.21.15.16:0/1174649392 >> 172.21.15.16:0/1174649392 conn(0x55a3e40e2120 msgr2=0x55a3e40e3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:58.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.798+0000 7f9984a20740 1 -- 172.21.15.16:0/1174649392 shutdown_connections 2024-03-28T22:10:58.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.798+0000 7f9984a20740 1 -- 172.21.15.16:0/1174649392 wait complete. 2024-03-28T22:10:58.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.799+0000 7f9984a20740 20 client.15609 trim_cache size 0 max 16384 2024-03-28T22:10:58.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.816+0000 7f9984a20740 1 Processor -- start 2024-03-28T22:10:58.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.816+0000 7f9984a20740 1 -- start start 2024-03-28T22:10:58.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.816+0000 7f9984a20740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3e40e76a0 0x55a3e40f49d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:58.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.816+0000 7f9984a20740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3e40e7fc0 0x55a3e4100e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:58.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.816+0000 7f9984a20740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3e40f5300 0x55a3e40e3c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:58.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.816+0000 7f9984a20740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f997c1ada00 con 0x55a3e40f5300 2024-03-28T22:10:58.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.816+0000 7f9984a20740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f997c1ab2d0 con 0x55a3e40e7fc0 2024-03-28T22:10:58.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.816+0000 7f9984a20740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f997c1a9120 con 0x55a3e40e76a0 2024-03-28T22:10:58.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.816+0000 7f997bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3e40f5300 0x55a3e40e3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:58.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.816+0000 7f997bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3e40f5300 0x55a3e40e3c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37744/0 (socket says 172.21.15.16:37744) 2024-03-28T22:10:58.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.816+0000 7f997bfff640 1 -- 172.21.15.16:0/1567416918 learned_addr learned my addr 172.21.15.16:0/1567416918 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:58.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.816+0000 7f997affd640 1 --2- 172.21.15.16:0/1567416918 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3e40e7fc0 0x55a3e4100e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:58.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.817+0000 7f997bfff640 1 -- 172.21.15.16:0/1567416918 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3e40e76a0 msgr2=0x55a3e40f49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:58.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.817+0000 7f997b7fe640 1 --2- 172.21.15.16:0/1567416918 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3e40e76a0 0x55a3e40f49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:58.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.817+0000 7f997bfff640 1 --2- 172.21.15.16:0/1567416918 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3e40e76a0 0x55a3e40f49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:58.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.817+0000 7f997bfff640 1 -- 172.21.15.16:0/1567416918 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3e40e7fc0 msgr2=0x55a3e4100e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:58.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.817+0000 7f997bfff640 1 --2- 172.21.15.16:0/1567416918 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3e40e7fc0 0x55a3e4100e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:58.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.817+0000 7f997bfff640 1 -- 172.21.15.16:0/1567416918 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3e40e8730 con 0x55a3e40f5300 2024-03-28T22:10:58.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.817+0000 7f997b7fe640 1 --2- 172.21.15.16:0/1567416918 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3e40e76a0 0x55a3e40f49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:10:58.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.817+0000 7f997affd640 1 --2- 172.21.15.16:0/1567416918 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3e40e7fc0 0x55a3e4100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:10:58.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.817+0000 7f997bfff640 1 --2- 172.21.15.16:0/1567416918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3e40f5300 0x55a3e40e3c80 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f997402c170 tx=0x7f997402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6e7e58ea983a335 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:58.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.817+0000 7f99717fa640 1 -- 172.21.15.16:0/1567416918 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9974062020 con 0x55a3e40f5300 2024-03-28T22:10:58.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.817+0000 7f99717fa640 1 -- 172.21.15.16:0/1567416918 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f997404f070 con 0x55a3e40f5300 2024-03-28T22:10:58.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.817+0000 7f99717fa640 1 -- 172.21.15.16:0/1567416918 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99740415d0 con 0x55a3e40f5300 2024-03-28T22:10:58.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.818+0000 7f9984a20740 1 -- 172.21.15.16:0/1567416918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3e40f5300 msgr2=0x55a3e40e3c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:58.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.818+0000 7f9984a20740 1 --2- 172.21.15.16:0/1567416918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3e40f5300 0x55a3e40e3c80 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f997402c170 tx=0x7f997402bc30 comp rx=0 tx=0).stop 2024-03-28T22:10:58.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.818+0000 7f9984a20740 1 -- 172.21.15.16:0/1567416918 shutdown_connections 2024-03-28T22:10:58.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.818+0000 7f9984a20740 1 --2- 172.21.15.16:0/1567416918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3e40f5300 0x55a3e40e3c80 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:58.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.818+0000 7f9984a20740 1 --2- 172.21.15.16:0/1567416918 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3e40e7fc0 0x55a3e4100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:58.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.818+0000 7f9984a20740 1 --2- 172.21.15.16:0/1567416918 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3e40e76a0 0x55a3e40f49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:58.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.818+0000 7f9984a20740 1 -- 172.21.15.16:0/1567416918 >> 172.21.15.16:0/1567416918 conn(0x55a3e40e2120 msgr2=0x55a3e40e3570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:58.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.818+0000 7f9984a20740 1 -- 172.21.15.16:0/1567416918 shutdown_connections 2024-03-28T22:10:58.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.818+0000 7f9984a20740 1 -- 172.21.15.16:0/1567416918 wait complete. 2024-03-28T22:10:58.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.818+0000 7f9984a20740 1 -- start start 2024-03-28T22:10:58.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.818+0000 7f9984a20740 1 Processor -- start 2024-03-28T22:10:58.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.818+0000 7f9984a20740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3e40e76a0 0x55a3e40f9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:58.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.818+0000 7f9984a20740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3e40e7fc0 0x55a3e40f95c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:58.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.818+0000 7f9984a20740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3e40f5300 0x55a3e416a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:58.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.818+0000 7f9984a20740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f997c1a9790 con 0x55a3e40e7fc0 2024-03-28T22:10:58.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.818+0000 7f9984a20740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f997c1aac00 con 0x55a3e40e76a0 2024-03-28T22:10:58.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.818+0000 7f9984a20740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f997c1a9e60 con 0x55a3e40f5300 2024-03-28T22:10:58.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.818+0000 7f9984a20740 10 client.0 _ll_register_callbacks cb 0x7f997402cd40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:10:58.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.818+0000 7f997affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3e40e7fc0 0x55a3e40f95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:58.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.819+0000 7f997affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3e40e7fc0 0x55a3e40f95c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37754/0 (socket says 172.21.15.16:37754) 2024-03-28T22:10:58.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.819+0000 7f997affd640 1 -- 172.21.15.16:0/59436842 learned_addr learned my addr 172.21.15.16:0/59436842 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:58.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.819+0000 7f997b7fe640 1 --2- 172.21.15.16:0/59436842 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3e40e76a0 0x55a3e40f9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:58.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.819+0000 7f997bfff640 1 --2- 172.21.15.16:0/59436842 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3e40f5300 0x55a3e416a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:58.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.819+0000 7f997affd640 1 -- 172.21.15.16:0/59436842 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3e40f5300 msgr2=0x55a3e416a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:58.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.819+0000 7f997affd640 1 --2- 172.21.15.16:0/59436842 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3e40f5300 0x55a3e416a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:58.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.819+0000 7f997affd640 1 -- 172.21.15.16:0/59436842 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3e40e76a0 msgr2=0x55a3e40f9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:58.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.819+0000 7f997affd640 1 --2- 172.21.15.16:0/59436842 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3e40e76a0 0x55a3e40f9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:58.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.819+0000 7f997affd640 1 -- 172.21.15.16:0/59436842 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3e40f9e90 con 0x55a3e40e7fc0 2024-03-28T22:10:58.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.819+0000 7f997bfff640 1 --2- 172.21.15.16:0/59436842 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3e40f5300 0x55a3e416a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:10:58.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.819+0000 7f997affd640 1 --2- 172.21.15.16:0/59436842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3e40e7fc0 0x55a3e40f95c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f996401f000 tx=0x7f99640993f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a439887569780676 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:58.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.819+0000 7f995affd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:10:58.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.819+0000 7f995affd640 1 -- 172.21.15.16:0/59436842 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99640b8070 con 0x55a3e40e7fc0 2024-03-28T22:10:58.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.819+0000 7f995affd640 1 -- 172.21.15.16:0/59436842 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f99640b9270 con 0x55a3e40e7fc0 2024-03-28T22:10:58.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.819+0000 7f9984a20740 1 -- 172.21.15.16:0/59436842 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f997404f070 con 0x55a3e40e7fc0 2024-03-28T22:10:58.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.819+0000 7f995affd640 1 -- 172.21.15.16:0/59436842 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f99640a3680 con 0x55a3e40e7fc0 2024-03-28T22:10:58.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.819+0000 7f9984a20740 10 client.15615 fetch_fsmap learned FSMap version 21 2024-03-28T22:10:58.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.819+0000 7f9984a20740 1 -- 172.21.15.16:0/59436842 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55a3e416bc00 con 0x55a3e40e7fc0 2024-03-28T22:10:58.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.819+0000 7f995affd640 1 -- 172.21.15.16:0/59436842 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f997404f070 con 0x55a3e40e7fc0 2024-03-28T22:10:58.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.820+0000 7f995affd640 1 -- 172.21.15.16:0/59436842 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f996409a330 con 0x55a3e40e7fc0 2024-03-28T22:10:58.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.820+0000 7f9984a20740 10 client.15615 fetch_fsmap finished waiting for FSMap version 21 2024-03-28T22:10:58.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.820+0000 7f9984a20740 10 client.15615 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:10:58.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.820+0000 7f9984a20740 20 client.15615 populate_metadata read hostname 'smithi016' 2024-03-28T22:10:58.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.820+0000 7f9984a20740 1 --2- 172.21.15.16:0/59436842 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55a3e416c360 0x55a3e416e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:58.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.820+0000 7f9984a20740 4 client.15615 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:10:58.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.820+0000 7f9984a20740 1 -- 172.21.15.16:0/59436842 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55a3e4052730 con 0x55a3e416c360 2024-03-28T22:10:58.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.820+0000 7f997b7fe640 1 --2- 172.21.15.16:0/59436842 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55a3e416c360 0x55a3e416e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:58.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.820+0000 7f997b7fe640 1 --2- 172.21.15.16:0/59436842 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55a3e416c360 0x55a3e416e780 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbf33add4b712a82 server_cookie=795b240137c6f1a9 in_seq=0 out_seq=0 2024-03-28T22:10:58.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.820+0000 7f995affd640 10 client.15615 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:10:58.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.821+0000 7f995affd640 1 -- 172.21.15.16:0/59436842 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a3e4052730 con 0x55a3e416c360 2024-03-28T22:10:58.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.821+0000 7f995affd640 10 client.15615 handle_command_reply: tid=0 2024-03-28T22:10:58.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.821+0000 7f9984a20740 1 client.15615 shutdown 2024-03-28T22:10:58.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.821+0000 7f9984a20740 1 -- 172.21.15.16:0/59436842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3e40e7fc0 msgr2=0x55a3e40f95c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:58.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.821+0000 7f9984a20740 1 --2- 172.21.15.16:0/59436842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3e40e7fc0 0x55a3e40f95c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f996401f000 tx=0x7f99640993f0 comp rx=0 tx=0).stop 2024-03-28T22:10:58.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.821+0000 7f9984a20740 1 -- 172.21.15.16:0/59436842 shutdown_connections 2024-03-28T22:10:58.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.821+0000 7f9984a20740 1 --2- 172.21.15.16:0/59436842 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55a3e416c360 0x55a3e416e780 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:58.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.821+0000 7f9984a20740 1 --2- 172.21.15.16:0/59436842 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3e40f5300 0x55a3e416a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:58.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.821+0000 7f9984a20740 1 --2- 172.21.15.16:0/59436842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3e40e7fc0 0x55a3e40f95c0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:58.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.821+0000 7f9984a20740 1 --2- 172.21.15.16:0/59436842 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3e40e76a0 0x55a3e40f9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:58.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.821+0000 7f9984a20740 1 -- 172.21.15.16:0/59436842 >> 172.21.15.16:0/59436842 conn(0x55a3e40e2120 msgr2=0x55a3e40e1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:58.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.822+0000 7f995affd640 0 client.15615 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:10:58.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.822+0000 7f9984a20740 1 -- 172.21.15.16:0/59436842 shutdown_connections 2024-03-28T22:10:58.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.822+0000 7f9984a20740 1 -- 172.21.15.16:0/59436842 wait complete. 2024-03-28T22:10:58.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.822+0000 7f9984a20740 20 client.15615 trim_cache size 0 max 16384 2024-03-28T22:10:58.841 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:10:58.841 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-03-28T22:10:58.841 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "ce5c9ef5-6a21-491b-a2b8-59292917968e", 2024-03-28T22:10:58.841 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-03-28T22:10:58.842 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:10:58.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.822+0000 7f9981e49640 1 -- 172.21.15.16:0/1526520201 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f99680a2000 msgr2=0x7f99680a44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:58.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.822+0000 7f9981e49640 1 --2- 172.21.15.16:0/1526520201 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f99680a2000 0x7f99680a44c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9974036040 tx=0x7f997402c650 comp rx=0 tx=0).stop 2024-03-28T22:10:58.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.822+0000 7f9981e49640 1 -- 172.21.15.16:0/1526520201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f997c1020a0 msgr2=0x7f997c198500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:58.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.822+0000 7f9981e49640 1 --2- 172.21.15.16:0/1526520201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f997c1020a0 0x7f997c198500 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f996400ea60 tx=0x7f996400ef20 comp rx=0 tx=0).stop 2024-03-28T22:10:58.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.822+0000 7f9981e49640 1 -- 172.21.15.16:0/1526520201 shutdown_connections 2024-03-28T22:10:58.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.822+0000 7f9981e49640 1 --2- 172.21.15.16:0/1526520201 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f99680a2000 0x7f99680a44c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:58.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.822+0000 7f9981e49640 1 --2- 172.21.15.16:0/1526520201 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f997c1076f0 0x7f997c19c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:58.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.822+0000 7f9981e49640 1 --2- 172.21.15.16:0/1526520201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f997c1020a0 0x7f997c198500 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:58.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.822+0000 7f9981e49640 1 --2- 172.21.15.16:0/1526520201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f997c1016e0 0x7f997c197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:58.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.822+0000 7f9981e49640 1 -- 172.21.15.16:0/1526520201 >> 172.21.15.16:0/1526520201 conn(0x7f997c0fd1f0 msgr2=0x7f997c0fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:58.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.822+0000 7f9981e49640 1 -- 172.21.15.16:0/1526520201 shutdown_connections 2024-03-28T22:10:58.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.822+0000 7f9981e49640 1 -- 172.21.15.16:0/1526520201 wait complete. 2024-03-28T22:10:58.844 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:10:58.845 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:10:58.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.987+0000 7f814c82a640 1 -- 172.21.15.16:0/3257046412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81481066a0 msgr2=0x7f8148106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:58.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.987+0000 7f814c82a640 1 --2- 172.21.15.16:0/3257046412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81481066a0 0x7f8148106a80 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f8138009930 tx=0x7f813802bc30 comp rx=0 tx=0).stop 2024-03-28T22:10:58.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.987+0000 7f814c82a640 1 -- 172.21.15.16:0/3257046412 shutdown_connections 2024-03-28T22:10:58.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.987+0000 7f814c82a640 1 --2- 172.21.15.16:0/3257046412 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8148101010 0x7f814810d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:58.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.987+0000 7f814c82a640 1 --2- 172.21.15.16:0/3257046412 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8148100650 0x7f8148100ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:58.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.987+0000 7f814c82a640 1 --2- 172.21.15.16:0/3257046412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81481066a0 0x7f8148106a80 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:58.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.987+0000 7f814c82a640 1 -- 172.21.15.16:0/3257046412 >> 172.21.15.16:0/3257046412 conn(0x7f8148077820 msgr2=0x7f8148077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:58.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.987+0000 7f814c82a640 1 -- 172.21.15.16:0/3257046412 shutdown_connections 2024-03-28T22:10:58.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.987+0000 7f814c82a640 1 -- 172.21.15.16:0/3257046412 wait complete. 2024-03-28T22:10:58.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.988+0000 7f814c82a640 1 Processor -- start 2024-03-28T22:10:58.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.988+0000 7f814c82a640 1 -- start start 2024-03-28T22:10:58.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.988+0000 7f814c82a640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8148100650 0x7f814819a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:58.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.988+0000 7f814c82a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8148101010 0x7f814819a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:58.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.988+0000 7f814c82a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81481066a0 0x7f814819ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:58.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.988+0000 7f814c82a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f81481109e0 con 0x7f81481066a0 2024-03-28T22:10:58.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.988+0000 7f814c82a640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8148110860 con 0x7f8148100650 2024-03-28T22:10:58.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.988+0000 7f814c82a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8148110b60 con 0x7f8148101010 2024-03-28T22:10:58.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.988+0000 7f8146d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81481066a0 0x7f814819ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:58.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.988+0000 7f8146d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81481066a0 0x7f814819ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37782/0 (socket says 172.21.15.16:37782) 2024-03-28T22:10:58.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.988+0000 7f8146d9d640 1 -- 172.21.15.16:0/1022573889 learned_addr learned my addr 172.21.15.16:0/1022573889 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:58.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.988+0000 7f814659c640 1 --2- 172.21.15.16:0/1022573889 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8148100650 0x7f814819a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:58.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.988+0000 7f813ffff640 1 --2- 172.21.15.16:0/1022573889 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8148101010 0x7f814819a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:58.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.989+0000 7f814659c640 1 -- 172.21.15.16:0/1022573889 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8148101010 msgr2=0x7f814819a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:58.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.989+0000 7f814659c640 1 --2- 172.21.15.16:0/1022573889 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8148101010 0x7f814819a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:58.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.989+0000 7f814659c640 1 -- 172.21.15.16:0/1022573889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81481066a0 msgr2=0x7f814819ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:58.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.989+0000 7f814659c640 1 --2- 172.21.15.16:0/1022573889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81481066a0 0x7f814819ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:58.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.989+0000 7f814659c640 1 -- 172.21.15.16:0/1022573889 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f814819f170 con 0x7f8148100650 2024-03-28T22:10:58.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.989+0000 7f8146d9d640 1 --2- 172.21.15.16:0/1022573889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81481066a0 0x7f814819ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:10:58.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.989+0000 7f814659c640 1 --2- 172.21.15.16:0/1022573889 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8148100650 0x7f814819a180 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f813802c170 tx=0x7f813802cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:58.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.989+0000 7f813f7fe640 1 -- 172.21.15.16:0/1022573889 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8138041070 con 0x7f8148100650 2024-03-28T22:10:58.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.989+0000 7f814c82a640 1 -- 172.21.15.16:0/1022573889 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f814819f400 con 0x7f8148100650 2024-03-28T22:10:58.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.989+0000 7f813f7fe640 1 -- 172.21.15.16:0/1022573889 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8138004450 con 0x7f8148100650 2024-03-28T22:10:58.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.989+0000 7f813f7fe640 1 -- 172.21.15.16:0/1022573889 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8138003e30 con 0x7f8148100650 2024-03-28T22:10:58.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.989+0000 7f814c82a640 1 -- 172.21.15.16:0/1022573889 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81481a6cf0 con 0x7f8148100650 2024-03-28T22:10:59.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.991+0000 7f813f7fe640 1 -- 172.21.15.16:0/1022573889 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 24) ==== 136932+0+0 (secure 0 0 0) 0x7f8138044070 con 0x7f8148100650 2024-03-28T22:10:59.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.992+0000 7f814f40d740 1 Processor -- start 2024-03-28T22:10:59.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.992+0000 7f814f40d740 1 -- start start 2024-03-28T22:10:59.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.992+0000 7f813f7fe640 1 --2- 172.21.15.16:0/1022573889 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f81340a1f80 0x7f81340a4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:59.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.992+0000 7f814f40d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b50fe290 0x55f6b50fe670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:59.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.992+0000 7f814f40d740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b50febb0 0x55f6b5107250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:59.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.992+0000 7f814f40d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b5107790 0x55f6b5109b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:59.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.992+0000 7f814f40d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f6b5062180 con 0x55f6b50fe290 2024-03-28T22:10:59.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.992+0000 7f814f40d740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f6b500ab70 con 0x55f6b50febb0 2024-03-28T22:10:59.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.992+0000 7f814f40d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f6b505e220 con 0x55f6b5107790 2024-03-28T22:10:59.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.992+0000 7f8146d9d640 1 --2- 172.21.15.16:0/1022573889 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f81340a1f80 0x7f81340a4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:59.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.992+0000 7f813f7fe640 1 -- 172.21.15.16:0/1022573889 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f81380efb20 con 0x7f8148100650 2024-03-28T22:10:59.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.992+0000 7f813ffff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b50febb0 0x55f6b5107250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:59.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.992+0000 7f814659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b50fe290 0x55f6b50fe670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:59.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.992+0000 7f813ffff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b50febb0 0x55f6b5107250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:48266/0 (socket says 172.21.15.16:48266) 2024-03-28T22:10:59.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.992+0000 7f814659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b50fe290 0x55f6b50fe670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37784/0 (socket says 172.21.15.16:37784) 2024-03-28T22:10:59.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.992+0000 7f813ffff640 1 -- 172.21.15.16:0/682094866 learned_addr learned my addr 172.21.15.16:0/682094866 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:59.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.992+0000 7f8146d9d640 1 --2- 172.21.15.16:0/682094866 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b5107790 0x55f6b5109b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:59.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.992+0000 7f813ffff640 1 -- 172.21.15.16:0/682094866 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b5107790 msgr2=0x55f6b5109b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:59.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.992+0000 7f813ffff640 1 --2- 172.21.15.16:0/682094866 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b5107790 0x55f6b5109b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:59.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.992+0000 7f813ffff640 1 -- 172.21.15.16:0/682094866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b50fe290 msgr2=0x55f6b50fe670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:59.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.992+0000 7f813ffff640 1 --2- 172.21.15.16:0/682094866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b50fe290 0x55f6b50fe670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:59.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.992+0000 7f813ffff640 1 -- 172.21.15.16:0/682094866 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6b501e7c0 con 0x55f6b50febb0 2024-03-28T22:10:59.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.992+0000 7f814659c640 1 --2- 172.21.15.16:0/682094866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b50fe290 0x55f6b50fe670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:10:59.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.993+0000 7f8146d9d640 1 --2- 172.21.15.16:0/1022573889 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f81340a1f80 0x7f81340a4440 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f81400094f0 tx=0x7f814000fef0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:59.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.993+0000 7f813ffff640 1 --2- 172.21.15.16:0/682094866 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b50febb0 0x55f6b5107250 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8130009c90 tx=0x7f8130011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f746704885f737e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:59.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.993+0000 7f813cff9640 1 -- 172.21.15.16:0/682094866 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f813001e070 con 0x55f6b50febb0 2024-03-28T22:10:59.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.993+0000 7f813cff9640 1 -- 172.21.15.16:0/682094866 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8130009e40 con 0x55f6b50febb0 2024-03-28T22:10:59.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.993+0000 7f813cff9640 1 -- 172.21.15.16:0/682094866 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81300188b0 con 0x55f6b50febb0 2024-03-28T22:10:59.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.993+0000 7f814f40d740 1 -- 172.21.15.16:0/682094866 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b50febb0 msgr2=0x55f6b5107250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:59.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.993+0000 7f814f40d740 1 --2- 172.21.15.16:0/682094866 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b50febb0 0x55f6b5107250 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8130009c90 tx=0x7f8130011810 comp rx=0 tx=0).stop 2024-03-28T22:10:59.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.994+0000 7f814f40d740 1 -- 172.21.15.16:0/682094866 shutdown_connections 2024-03-28T22:10:59.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.994+0000 7f814f40d740 1 --2- 172.21.15.16:0/682094866 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b5107790 0x55f6b5109b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:59.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.994+0000 7f814f40d740 1 --2- 172.21.15.16:0/682094866 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b50febb0 0x55f6b5107250 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:59.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.994+0000 7f814f40d740 1 --2- 172.21.15.16:0/682094866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b50fe290 0x55f6b50fe670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:59.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.994+0000 7f814f40d740 1 -- 172.21.15.16:0/682094866 >> 172.21.15.16:0/682094866 conn(0x55f6b50f89c0 msgr2=0x55f6b50f8da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:59.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.994+0000 7f814f40d740 1 -- 172.21.15.16:0/682094866 shutdown_connections 2024-03-28T22:10:59.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.994+0000 7f814f40d740 1 -- 172.21.15.16:0/682094866 wait complete. 2024-03-28T22:10:59.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.994+0000 7f814f40d740 1 -- start start 2024-03-28T22:10:59.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.994+0000 7f814f40d740 1 Processor -- start 2024-03-28T22:10:59.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.994+0000 7f814f40d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b50fe290 0x55f6b5104850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:59.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.994+0000 7f814f40d740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b50febb0 0x55f6b5104d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:59.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.994+0000 7f814f40d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b5107790 0x55f6b51052d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:59.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.994+0000 7f814f40d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f6b510a580 con 0x55f6b5107790 2024-03-28T22:10:59.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.994+0000 7f814f40d740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f6b50004a0 con 0x55f6b50febb0 2024-03-28T22:10:59.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.994+0000 7f814f40d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f6b510a700 con 0x55f6b50fe290 2024-03-28T22:10:59.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.994+0000 7f814f40d740 10 client.0 _ll_register_callbacks cb 0x55f6b5003d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:10:59.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.994+0000 7f8146d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b5107790 0x55f6b51052d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:59.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.994+0000 7f8146d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b5107790 0x55f6b51052d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37800/0 (socket says 172.21.15.16:37800) 2024-03-28T22:10:59.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.994+0000 7f8146d9d640 1 -- 172.21.15.16:0/506597176 learned_addr learned my addr 172.21.15.16:0/506597176 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:59.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.994+0000 7f813ffff640 1 --2- 172.21.15.16:0/506597176 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b50febb0 0x55f6b5104d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:59.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.994+0000 7f814659c640 1 --2- 172.21.15.16:0/506597176 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b50fe290 0x55f6b5104850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:59.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.995+0000 7f8146d9d640 1 -- 172.21.15.16:0/506597176 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b50fe290 msgr2=0x55f6b5104850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:59.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.995+0000 7f8146d9d640 1 --2- 172.21.15.16:0/506597176 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b50fe290 0x55f6b5104850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:59.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.995+0000 7f8146d9d640 1 -- 172.21.15.16:0/506597176 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b50febb0 msgr2=0x55f6b5104d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:59.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.995+0000 7f8146d9d640 1 --2- 172.21.15.16:0/506597176 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b50febb0 0x55f6b5104d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:59.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.995+0000 7f8146d9d640 1 -- 172.21.15.16:0/506597176 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6b4e407d0 con 0x55f6b5107790 2024-03-28T22:10:59.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.995+0000 7f8146d9d640 1 --2- 172.21.15.16:0/506597176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b5107790 0x55f6b51052d0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f8140010630 tx=0x7f8140004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a2ea20fee696522 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:59.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.995+0000 7f812dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:10:59.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.995+0000 7f812dffb640 1 -- 172.21.15.16:0/506597176 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8140020800 con 0x55f6b5107790 2024-03-28T22:10:59.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.995+0000 7f814f40d740 1 -- 172.21.15.16:0/506597176 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f6b506f6c0 con 0x55f6b5107790 2024-03-28T22:10:59.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.995+0000 7f812dffb640 1 -- 172.21.15.16:0/506597176 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f81400209a0 con 0x55f6b5107790 2024-03-28T22:10:59.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.995+0000 7f812dffb640 1 -- 172.21.15.16:0/506597176 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8140020ba0 con 0x55f6b5107790 2024-03-28T22:10:59.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.995+0000 7f814f40d740 10 client.15627 fetch_fsmap learned FSMap version 21 2024-03-28T22:10:59.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.995+0000 7f814f40d740 1 -- 172.21.15.16:0/506597176 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55f6b4d763f0 con 0x55f6b5107790 2024-03-28T22:10:59.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.995+0000 7f812dffb640 1 -- 172.21.15.16:0/506597176 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f6b506f6c0 con 0x55f6b5107790 2024-03-28T22:10:59.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.995+0000 7f812dffb640 1 -- 172.21.15.16:0/506597176 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f814002f020 con 0x55f6b5107790 2024-03-28T22:10:59.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.996+0000 7f814f40d740 10 client.15627 fetch_fsmap finished waiting for FSMap version 21 2024-03-28T22:10:59.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.996+0000 7f814f40d740 10 client.15627 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:10:59.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.996+0000 7f814f40d740 20 client.15627 populate_metadata read hostname 'smithi016' 2024-03-28T22:10:59.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.996+0000 7f814f40d740 1 --2- 172.21.15.16:0/506597176 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f6b5117af0 0x55f6b5119f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:59.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.996+0000 7f814f40d740 4 client.15627 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:10:59.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.996+0000 7f814f40d740 1 -- 172.21.15.16:0/506597176 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f6b5069a10 con 0x55f6b5117af0 2024-03-28T22:10:59.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.996+0000 7f814659c640 1 --2- 172.21.15.16:0/506597176 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f6b5117af0 0x55f6b5119f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:59.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.996+0000 7f814659c640 1 --2- 172.21.15.16:0/506597176 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f6b5117af0 0x55f6b5119f20 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f0fa647fc9f5783 server_cookie=7502b691b6373c79 in_seq=0 out_seq=0 2024-03-28T22:10:59.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.996+0000 7f812dffb640 10 client.15627 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:10:59.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.997+0000 7f812dffb640 1 -- 172.21.15.16:0/506597176 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f6b5069a10 con 0x55f6b5117af0 2024-03-28T22:10:59.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.997+0000 7f812dffb640 10 client.15627 handle_command_reply: tid=0 2024-03-28T22:10:59.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.997+0000 7f814f40d740 1 client.15627 shutdown 2024-03-28T22:10:59.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.997+0000 7f814f40d740 1 -- 172.21.15.16:0/506597176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b5107790 msgr2=0x55f6b51052d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:59.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.997+0000 7f814f40d740 1 --2- 172.21.15.16:0/506597176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b5107790 0x55f6b51052d0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f8140010630 tx=0x7f8140004ab0 comp rx=0 tx=0).stop 2024-03-28T22:10:59.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.997+0000 7f814f40d740 1 -- 172.21.15.16:0/506597176 shutdown_connections 2024-03-28T22:10:59.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.997+0000 7f814f40d740 1 --2- 172.21.15.16:0/506597176 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f6b5117af0 0x55f6b5119f20 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:59.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.997+0000 7f814f40d740 1 --2- 172.21.15.16:0/506597176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b5107790 0x55f6b51052d0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:59.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.997+0000 7f812dffb640 0 client.15627 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:10:59.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.997+0000 7f814f40d740 1 --2- 172.21.15.16:0/506597176 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b50febb0 0x55f6b5104d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:59.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.997+0000 7f814f40d740 1 --2- 172.21.15.16:0/506597176 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b50fe290 0x55f6b5104850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:59.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.997+0000 7f814f40d740 1 -- 172.21.15.16:0/506597176 >> 172.21.15.16:0/506597176 conn(0x55f6b50f89c0 msgr2=0x55f6b50f7c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:59.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.997+0000 7f814f40d740 1 -- 172.21.15.16:0/506597176 shutdown_connections 2024-03-28T22:10:59.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.997+0000 7f814f40d740 1 -- 172.21.15.16:0/506597176 wait complete. 2024-03-28T22:10:59.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:58.997+0000 7f814f40d740 20 client.15627 trim_cache size 0 max 16384 2024-03-28T22:10:59.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.015+0000 7f814f40d740 1 Processor -- start 2024-03-28T22:10:59.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.015+0000 7f814f40d740 1 -- start start 2024-03-28T22:10:59.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.015+0000 7f814f40d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b50fe290 0x55f6b5106f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:59.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.015+0000 7f814f40d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b50febb0 0x55f6b5113320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:59.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.015+0000 7f814f40d740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b5107790 0x55f6b50f7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:59.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.015+0000 7f814f40d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f81481b1d50 con 0x55f6b50fe290 2024-03-28T22:10:59.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.015+0000 7f814f40d740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f81481af620 con 0x55f6b5107790 2024-03-28T22:10:59.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.015+0000 7f814f40d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f81481ad470 con 0x55f6b50febb0 2024-03-28T22:10:59.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.015+0000 7f814659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b50fe290 0x55f6b5106f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:59.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.015+0000 7f814659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b50fe290 0x55f6b5106f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37806/0 (socket says 172.21.15.16:37806) 2024-03-28T22:10:59.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.015+0000 7f814659c640 1 -- 172.21.15.16:0/3519658345 learned_addr learned my addr 172.21.15.16:0/3519658345 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:59.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.015+0000 7f8146d9d640 1 --2- 172.21.15.16:0/3519658345 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b5107790 0x55f6b50f7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:59.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.015+0000 7f814659c640 1 -- 172.21.15.16:0/3519658345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b50febb0 msgr2=0x55f6b5113320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:59.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.015+0000 7f814659c640 1 --2- 172.21.15.16:0/3519658345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b50febb0 0x55f6b5113320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:59.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.015+0000 7f814659c640 1 -- 172.21.15.16:0/3519658345 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b5107790 msgr2=0x55f6b50f7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:59.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.015+0000 7f814659c640 1 --2- 172.21.15.16:0/3519658345 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b5107790 0x55f6b50f7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:59.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.015+0000 7f814659c640 1 -- 172.21.15.16:0/3519658345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6b50fe9e0 con 0x55f6b50fe290 2024-03-28T22:10:59.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.016+0000 7f814659c640 1 --2- 172.21.15.16:0/3519658345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b50fe290 0x55f6b5106f80 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f81380bc280 tx=0x7f81380056e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e26f26f0d5120d6b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:59.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.016+0000 7f812ffff640 1 -- 172.21.15.16:0/3519658345 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f813804b040 con 0x55f6b50fe290 2024-03-28T22:10:59.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.016+0000 7f812ffff640 1 -- 172.21.15.16:0/3519658345 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81380dc0c0 con 0x55f6b50fe290 2024-03-28T22:10:59.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.016+0000 7f812ffff640 1 -- 172.21.15.16:0/3519658345 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81380ba620 con 0x55f6b50fe290 2024-03-28T22:10:59.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.016+0000 7f814f40d740 1 -- 172.21.15.16:0/3519658345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b50fe290 msgr2=0x55f6b5106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:59.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.016+0000 7f814f40d740 1 --2- 172.21.15.16:0/3519658345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b50fe290 0x55f6b5106f80 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f81380bc280 tx=0x7f81380056e0 comp rx=0 tx=0).stop 2024-03-28T22:10:59.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.016+0000 7f814f40d740 1 -- 172.21.15.16:0/3519658345 shutdown_connections 2024-03-28T22:10:59.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.016+0000 7f814f40d740 1 --2- 172.21.15.16:0/3519658345 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b5107790 0x55f6b50f7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:59.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.016+0000 7f814f40d740 1 --2- 172.21.15.16:0/3519658345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b50febb0 0x55f6b5113320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:59.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.016+0000 7f814f40d740 1 --2- 172.21.15.16:0/3519658345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b50fe290 0x55f6b5106f80 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:59.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.016+0000 7f814f40d740 1 -- 172.21.15.16:0/3519658345 >> 172.21.15.16:0/3519658345 conn(0x55f6b50f89c0 msgr2=0x55f6b50f7390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:59.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.016+0000 7f814f40d740 1 -- 172.21.15.16:0/3519658345 shutdown_connections 2024-03-28T22:10:59.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.016+0000 7f814f40d740 1 -- 172.21.15.16:0/3519658345 wait complete. 2024-03-28T22:10:59.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.017+0000 7f814f40d740 1 -- start start 2024-03-28T22:10:59.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.017+0000 7f814f40d740 1 Processor -- start 2024-03-28T22:10:59.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.017+0000 7f814f40d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b50fe290 0x55f6b510fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:59.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.017+0000 7f814f40d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b50febb0 0x55f6b51010c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:59.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.017+0000 7f814f40d740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b5107790 0x55f6b5101600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:59.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.017+0000 7f814f40d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f81481adae0 con 0x55f6b50fe290 2024-03-28T22:10:59.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.017+0000 7f814f40d740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f81481aef50 con 0x55f6b5107790 2024-03-28T22:10:59.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.017+0000 7f814f40d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f81481ae1b0 con 0x55f6b50febb0 2024-03-28T22:10:59.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.017+0000 7f814f40d740 10 client.0 _ll_register_callbacks cb 0x7f8140013bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:10:59.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.017+0000 7f814659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b50fe290 0x55f6b510fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:59.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.017+0000 7f814659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b50fe290 0x55f6b510fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37814/0 (socket says 172.21.15.16:37814) 2024-03-28T22:10:59.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.017+0000 7f814659c640 1 -- 172.21.15.16:0/502745679 learned_addr learned my addr 172.21.15.16:0/502745679 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:10:59.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.017+0000 7f8146d9d640 1 --2- 172.21.15.16:0/502745679 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b5107790 0x55f6b5101600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:59.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.017+0000 7f813ffff640 1 --2- 172.21.15.16:0/502745679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b50febb0 0x55f6b51010c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:59.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.017+0000 7f814659c640 1 -- 172.21.15.16:0/502745679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b50febb0 msgr2=0x55f6b51010c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:59.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.017+0000 7f814659c640 1 --2- 172.21.15.16:0/502745679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b50febb0 0x55f6b51010c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:59.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.017+0000 7f814659c640 1 -- 172.21.15.16:0/502745679 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b5107790 msgr2=0x55f6b5101600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:10:59.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.017+0000 7f814659c640 1 --2- 172.21.15.16:0/502745679 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b5107790 0x55f6b5101600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:59.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.017+0000 7f814659c640 1 -- 172.21.15.16:0/502745679 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6b4d360f0 con 0x55f6b50fe290 2024-03-28T22:10:59.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.017+0000 7f813ffff640 1 --2- 172.21.15.16:0/502745679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b50febb0 0x55f6b51010c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:10:59.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.017+0000 7f814659c640 1 --2- 172.21.15.16:0/502745679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b50fe290 0x55f6b510fbb0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f81380ed3c0 tx=0x7f81380c4c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a6d28ca1f6dd735 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:10:59.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.017+0000 7f812dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:10:59.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.017+0000 7f812dffb640 1 -- 172.21.15.16:0/502745679 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f813804b070 con 0x55f6b50fe290 2024-03-28T22:10:59.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.018+0000 7f812dffb640 1 -- 172.21.15.16:0/502745679 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f81380b8bf0 con 0x55f6b50fe290 2024-03-28T22:10:59.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.018+0000 7f814f40d740 1 -- 172.21.15.16:0/502745679 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f81380dc0c0 con 0x55f6b50fe290 2024-03-28T22:10:59.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.018+0000 7f812dffb640 1 -- 172.21.15.16:0/502745679 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8138037120 con 0x55f6b50fe290 2024-03-28T22:10:59.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.018+0000 7f814f40d740 10 client.15633 fetch_fsmap learned FSMap version 21 2024-03-28T22:10:59.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.018+0000 7f814f40d740 1 -- 172.21.15.16:0/502745679 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55f6b517dc30 con 0x55f6b50fe290 2024-03-28T22:10:59.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.018+0000 7f812dffb640 1 -- 172.21.15.16:0/502745679 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81380dc0c0 con 0x55f6b50fe290 2024-03-28T22:10:59.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.018+0000 7f812dffb640 1 -- 172.21.15.16:0/502745679 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f8138000ed0 con 0x55f6b50fe290 2024-03-28T22:10:59.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.018+0000 7f814f40d740 10 client.15633 fetch_fsmap finished waiting for FSMap version 21 2024-03-28T22:10:59.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.018+0000 7f814f40d740 10 client.15633 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:10:59.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.018+0000 7f814f40d740 20 client.15633 populate_metadata read hostname 'smithi016' 2024-03-28T22:10:59.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.018+0000 7f814f40d740 1 --2- 172.21.15.16:0/502745679 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f6b517e2f0 0x55f6b51806e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:10:59.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.018+0000 7f814f40d740 4 client.15633 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:10:59.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.018+0000 7f814f40d740 1 -- 172.21.15.16:0/502745679 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f6b4fe7540 con 0x55f6b517e2f0 2024-03-28T22:10:59.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.018+0000 7f813ffff640 1 --2- 172.21.15.16:0/502745679 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f6b517e2f0 0x55f6b51806e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:10:59.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.019+0000 7f812dffb640 10 client.15633 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:10:59.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.019+0000 7f813ffff640 1 --2- 172.21.15.16:0/502745679 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f6b517e2f0 0x55f6b51806e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68a86899ee36be1d server_cookie=3ea842b136c4df61 in_seq=0 out_seq=0 2024-03-28T22:10:59.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.019+0000 7f812dffb640 1 -- 172.21.15.16:0/502745679 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55f6b4fe7540 con 0x55f6b517e2f0 2024-03-28T22:10:59.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.019+0000 7f812dffb640 10 client.15633 handle_command_reply: tid=0 2024-03-28T22:10:59.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.019+0000 7f814f40d740 1 client.15633 shutdown 2024-03-28T22:10:59.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.019+0000 7f814f40d740 1 -- 172.21.15.16:0/502745679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b50fe290 msgr2=0x55f6b510fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:59.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.019+0000 7f814f40d740 1 --2- 172.21.15.16:0/502745679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b50fe290 0x55f6b510fbb0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f81380ed3c0 tx=0x7f81380c4c70 comp rx=0 tx=0).stop 2024-03-28T22:10:59.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.019+0000 7f814f40d740 1 -- 172.21.15.16:0/502745679 shutdown_connections 2024-03-28T22:10:59.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.019+0000 7f814f40d740 1 --2- 172.21.15.16:0/502745679 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f6b517e2f0 0x55f6b51806e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:59.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.019+0000 7f814f40d740 1 --2- 172.21.15.16:0/502745679 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b5107790 0x55f6b5101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:59.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.019+0000 7f814f40d740 1 --2- 172.21.15.16:0/502745679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b50febb0 0x55f6b51010c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:59.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.019+0000 7f814f40d740 1 --2- 172.21.15.16:0/502745679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b50fe290 0x55f6b510fbb0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:59.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.019+0000 7f812dffb640 0 client.15633 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:10:59.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.019+0000 7f814f40d740 1 -- 172.21.15.16:0/502745679 >> 172.21.15.16:0/502745679 conn(0x55f6b50f89c0 msgr2=0x55f6b50fa680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:59.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.019+0000 7f814f40d740 1 -- 172.21.15.16:0/502745679 shutdown_connections 2024-03-28T22:10:59.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.019+0000 7f814f40d740 1 -- 172.21.15.16:0/502745679 wait complete. 2024-03-28T22:10:59.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.019+0000 7f814f40d740 20 client.15633 trim_cache size 0 max 16384 2024-03-28T22:10:59.029 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:10:59.029 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T22:10:59.029 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-03-28T22:10:59.029 INFO:teuthology.orchestra.run.smithi016.stdout: "ce5c9ef5-6a21-491b-a2b8-59292917968e": { 2024-03-28T22:10:59.029 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-03-28T22:10:59.029 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "ce5c9ef5-6a21-491b-a2b8-59292917968e", 2024-03-28T22:10:59.029 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-03-28T22:10:59.030 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:10:59.030 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:10:59.030 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:10:59.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.019+0000 7f814c82a640 1 -- 172.21.15.16:0/1022573889 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f81340a1f80 msgr2=0x7f81340a4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:59.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.019+0000 7f814c82a640 1 --2- 172.21.15.16:0/1022573889 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f81340a1f80 0x7f81340a4440 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f81400094f0 tx=0x7f814000fef0 comp rx=0 tx=0).stop 2024-03-28T22:10:59.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.020+0000 7f814c82a640 1 -- 172.21.15.16:0/1022573889 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8148100650 msgr2=0x7f814819a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:10:59.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.020+0000 7f814c82a640 1 --2- 172.21.15.16:0/1022573889 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8148100650 0x7f814819a180 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f813802c170 tx=0x7f813802cbf0 comp rx=0 tx=0).stop 2024-03-28T22:10:59.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.020+0000 7f814c82a640 1 -- 172.21.15.16:0/1022573889 shutdown_connections 2024-03-28T22:10:59.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.020+0000 7f814c82a640 1 --2- 172.21.15.16:0/1022573889 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f81340a1f80 0x7f81340a4440 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:59.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.020+0000 7f814c82a640 1 --2- 172.21.15.16:0/1022573889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81481066a0 0x7f814819ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:59.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.020+0000 7f814c82a640 1 --2- 172.21.15.16:0/1022573889 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8148101010 0x7f814819a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:59.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.020+0000 7f814c82a640 1 --2- 172.21.15.16:0/1022573889 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8148100650 0x7f814819a180 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:10:59.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.020+0000 7f814c82a640 1 -- 172.21.15.16:0/1022573889 >> 172.21.15.16:0/1022573889 conn(0x7f8148077820 msgr2=0x7f8148069e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:10:59.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.020+0000 7f814c82a640 1 -- 172.21.15.16:0/1022573889 shutdown_connections 2024-03-28T22:10:59.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:10:59.020+0000 7f814c82a640 1 -- 172.21.15.16:0/1022573889 wait complete. 2024-03-28T22:10:59.032 INFO:tasks.cephfs.filesystem:scrub status for tag:ce5c9ef5-6a21-491b-a2b8-59292917968e - {'path': '/', 'tag': 'ce5c9ef5-6a21-491b-a2b8-59292917968e', 'options': 'recursive,force'} 2024-03-28T22:10:59.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:59 smithi016 ceph-mon[32374]: scrub queued for path: / 2024-03-28T22:10:59.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:59 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:10:59.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:59 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:10:59.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:10:59 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:10:59.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:59 smithi053 ceph-mon[35979]: scrub queued for path: / 2024-03-28T22:10:59.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:59 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:10:59.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:59 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:10:59.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:10:59 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:10:59.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:59 smithi063 ceph-mon[36249]: scrub queued for path: / 2024-03-28T22:10:59.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:59 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:10:59.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:59 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:10:59.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:10:59 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:11:00.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:00 smithi053 ceph-mon[35979]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:11:00.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:00 smithi053 ceph-mon[35979]: scrub summary: idle 2024-03-28T22:11:00.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:00 smithi063 ceph-mon[36249]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:11:00.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:00 smithi063 ceph-mon[36249]: scrub summary: idle 2024-03-28T22:11:00.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:00 smithi016 ceph-mon[32374]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:11:00.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:00 smithi016 ceph-mon[32374]: scrub summary: idle 2024-03-28T22:11:01.187 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:11:00 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:11:00.823+0000 7f8a21629700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:11:01.187 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:11:00 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:11:00.823+0000 7f8a21629700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:11:01.187 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:11:00 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:11:00.823+0000 7f8a21629700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:11:01.187 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:11:00 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:11:00.823+0000 7f8a21629700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:11:01.187 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:11:00 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:11:00.823+0000 7f8a21629700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:11:02.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:02 smithi053 ceph-mon[35979]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:11:02.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:02 smithi063 ceph-mon[36249]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:11:02.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:02 smithi016 ceph-mon[32374]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:11:04.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:04 smithi053 ceph-mon[35979]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:11:04.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:04 smithi053 ceph-mon[35979]: mgrmap e25: x(active, since 28s), standbys: y, z 2024-03-28T22:11:04.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:04 smithi063 ceph-mon[36249]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:11:04.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:04 smithi063 ceph-mon[36249]: mgrmap e25: x(active, since 28s), standbys: y, z 2024-03-28T22:11:04.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:04 smithi016 ceph-mon[32374]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:11:04.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:04 smithi016 ceph-mon[32374]: mgrmap e25: x(active, since 28s), standbys: y, z 2024-03-28T22:11:06.549 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:06 smithi053 ceph-mon[35979]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:11:06.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:06 smithi063 ceph-mon[36249]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:11:06.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:06 smithi016 ceph-mon[32374]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:11:08.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:08 smithi053 ceph-mon[35979]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-28T22:11:08.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:08 smithi063 ceph-mon[36249]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-28T22:11:08.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:08 smithi016 ceph-mon[32374]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-28T22:11:10.484 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:10 smithi063 ceph-mon[36249]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-28T22:11:10.487 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:10 smithi016 ceph-mon[32374]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-28T22:11:10.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:10 smithi053 ceph-mon[35979]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-28T22:11:12.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:12 smithi053 ceph-mon[35979]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-28T22:11:12.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:12 smithi063 ceph-mon[36249]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-28T22:11:12.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:12 smithi016 ceph-mon[32374]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-28T22:11:14.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:14 smithi053 ceph-mon[35979]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-03-28T22:11:14.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:14 smithi063 ceph-mon[36249]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-03-28T22:11:14.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:14 smithi016 ceph-mon[32374]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-03-28T22:11:16.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:16 smithi053 ceph-mon[35979]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 20 op/s 2024-03-28T22:11:16.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:16 smithi063 ceph-mon[36249]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 20 op/s 2024-03-28T22:11:16.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:16 smithi016 ceph-mon[32374]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 20 op/s 2024-03-28T22:11:18.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:18 smithi053 ceph-mon[35979]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-03-28T22:11:18.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:18 smithi063 ceph-mon[36249]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-03-28T22:11:18.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:18 smithi016 ceph-mon[32374]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-03-28T22:11:20.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:20 smithi053 ceph-mon[35979]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-28T22:11:20.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:20 smithi063 ceph-mon[36249]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-28T22:11:20.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:20 smithi016 ceph-mon[32374]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-28T22:11:22.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:22 smithi053 ceph-mon[35979]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-28T22:11:22.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:22 smithi063 ceph-mon[36249]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-28T22:11:22.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:22 smithi016 ceph-mon[32374]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-28T22:11:24.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:24 smithi053 ceph-mon[35979]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 853 B/s wr, 7 op/s 2024-03-28T22:11:24.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:24 smithi063 ceph-mon[36249]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 853 B/s wr, 7 op/s 2024-03-28T22:11:24.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:24 smithi016 ceph-mon[32374]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 853 B/s wr, 7 op/s 2024-03-28T22:11:26.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:26 smithi053 ceph-mon[35979]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:11:26.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:26 smithi063 ceph-mon[36249]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:11:26.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:26 smithi016 ceph-mon[32374]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:11:28.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:28 smithi053 ceph-mon[35979]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:11:28.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:28 smithi063 ceph-mon[36249]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:11:28.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:28 smithi016 ceph-mon[32374]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:11:29.033 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:11:29.033 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:11:29.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.194+0000 7f2c3d593640 1 -- 172.21.15.16:0/2138767430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c381005f0 msgr2=0x7f2c38100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:29.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.194+0000 7f2c3d593640 1 --2- 172.21.15.16:0/2138767430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c381005f0 0x7f2c38100a70 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c009930 tx=0x7f2c2c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:11:29.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.194+0000 7f2c3d593640 1 -- 172.21.15.16:0/2138767430 shutdown_connections 2024-03-28T22:11:29.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.194+0000 7f2c3d593640 1 --2- 172.21.15.16:0/2138767430 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c38100fb0 0x7f2c3810d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.194+0000 7f2c3d593640 1 --2- 172.21.15.16:0/2138767430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c381005f0 0x7f2c38100a70 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.194+0000 7f2c3d593640 1 --2- 172.21.15.16:0/2138767430 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c38106690 0x7f2c38106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.194+0000 7f2c3d593640 1 -- 172.21.15.16:0/2138767430 >> 172.21.15.16:0/2138767430 conn(0x7f2c38077800 msgr2=0x7f2c38077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:11:29.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.194+0000 7f2c3d593640 1 -- 172.21.15.16:0/2138767430 shutdown_connections 2024-03-28T22:11:29.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.194+0000 7f2c3d593640 1 -- 172.21.15.16:0/2138767430 wait complete. 2024-03-28T22:11:29.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.195+0000 7f2c3d593640 1 Processor -- start 2024-03-28T22:11:29.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.195+0000 7f2c3d593640 1 -- start start 2024-03-28T22:11:29.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.195+0000 7f2c3d593640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c381005f0 0x7f2c38197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:29.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.195+0000 7f2c3d593640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c38100fb0 0x7f2c38198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:29.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.195+0000 7f2c3d593640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c38106690 0x7f2c3819c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:29.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.196+0000 7f2c367fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c38100fb0 0x7f2c38198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:29.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.196+0000 7f2c36ffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c381005f0 0x7f2c38197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:29.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.196+0000 7f2c367fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c38100fb0 0x7f2c38198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57840/0 (socket says 172.21.15.16:57840) 2024-03-28T22:11:29.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.196+0000 7f2c367fc640 1 -- 172.21.15.16:0/1002700204 learned_addr learned my addr 172.21.15.16:0/1002700204 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:11:29.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.196+0000 7f2c3d593640 1 -- 172.21.15.16:0/1002700204 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2c38111b10 con 0x7f2c38100fb0 2024-03-28T22:11:29.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.196+0000 7f2c3d593640 1 -- 172.21.15.16:0/1002700204 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f2c38111990 con 0x7f2c381005f0 2024-03-28T22:11:29.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.196+0000 7f2c3d593640 1 -- 172.21.15.16:0/1002700204 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2c38111c90 con 0x7f2c38106690 2024-03-28T22:11:29.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.196+0000 7f2c377fe640 1 --2- 172.21.15.16:0/1002700204 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c38106690 0x7f2c3819c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:29.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.196+0000 7f2c367fc640 1 -- 172.21.15.16:0/1002700204 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c38106690 msgr2=0x7f2c3819c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:29.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.196+0000 7f2c367fc640 1 --2- 172.21.15.16:0/1002700204 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c38106690 0x7f2c3819c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.196+0000 7f2c367fc640 1 -- 172.21.15.16:0/1002700204 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c381005f0 msgr2=0x7f2c38197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:29.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.196+0000 7f2c367fc640 1 --2- 172.21.15.16:0/1002700204 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c381005f0 0x7f2c38197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.196+0000 7f2c367fc640 1 -- 172.21.15.16:0/1002700204 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c3819cf00 con 0x7f2c38100fb0 2024-03-28T22:11:29.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.196+0000 7f2c367fc640 1 --2- 172.21.15.16:0/1002700204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c38100fb0 0x7f2c38198450 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c009230 tx=0x7f2c2c02cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:11:29.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.197+0000 7f2c1ffff640 1 -- 172.21.15.16:0/1002700204 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c041070 con 0x7f2c38100fb0 2024-03-28T22:11:29.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.197+0000 7f2c1ffff640 1 -- 172.21.15.16:0/1002700204 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c2c004450 con 0x7f2c38100fb0 2024-03-28T22:11:29.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.197+0000 7f2c3d593640 1 -- 172.21.15.16:0/1002700204 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c3819d190 con 0x7f2c38100fb0 2024-03-28T22:11:29.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.197+0000 7f2c1ffff640 1 -- 172.21.15.16:0/1002700204 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c034440 con 0x7f2c38100fb0 2024-03-28T22:11:29.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.197+0000 7f2c3d593640 1 -- 172.21.15.16:0/1002700204 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c381a4a80 con 0x7f2c38100fb0 2024-03-28T22:11:29.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.200+0000 7f2c1ffff640 1 -- 172.21.15.16:0/1002700204 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 25) ==== 136989+0+0 (secure 0 0 0) 0x7f2c2c044070 con 0x7f2c38100fb0 2024-03-28T22:11:29.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.200+0000 7f2c4017b740 1 Processor -- start 2024-03-28T22:11:29.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.200+0000 7f2c4017b740 1 -- start start 2024-03-28T22:11:29.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.201+0000 7f2c4017b740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5642c102e290 0x5642c102e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:29.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.201+0000 7f2c4017b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5642c102ebb0 0x5642c1037250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:29.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.201+0000 7f2c4017b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c1037790 0x5642c1039b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:29.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.201+0000 7f2c4017b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5642c0f92180 con 0x5642c1037790 2024-03-28T22:11:29.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.201+0000 7f2c4017b740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5642c0f3ab70 con 0x5642c102e290 2024-03-28T22:11:29.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.201+0000 7f2c4017b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5642c0f8e220 con 0x5642c102ebb0 2024-03-28T22:11:29.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.201+0000 7f2c367fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c1037790 0x5642c1039b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:29.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.201+0000 7f2c377fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5642c102e290 0x5642c102e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:29.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.201+0000 7f2c377fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5642c102e290 0x5642c102e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:50214/0 (socket says 172.21.15.16:50214) 2024-03-28T22:11:29.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.201+0000 7f2c377fe640 1 -- 172.21.15.16:0/422623616 learned_addr learned my addr 172.21.15.16:0/422623616 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:11:29.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.201+0000 7f2c1ffff640 1 --2- 172.21.15.16:0/1002700204 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f2c200a2010 0x7f2c200a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:29.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.201+0000 7f2c36ffd640 1 --2- 172.21.15.16:0/422623616 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5642c102ebb0 0x5642c1037250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:29.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.201+0000 7f2c1ffff640 1 -- 172.21.15.16:0/1002700204 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f2c2c0efd70 con 0x7f2c38100fb0 2024-03-28T22:11:29.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.201+0000 7f2c377fe640 1 --2- 172.21.15.16:0/1002700204 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f2c200a2010 0x7f2c200a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:29.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.201+0000 7f2c367fc640 1 -- 172.21.15.16:0/422623616 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5642c102ebb0 msgr2=0x5642c1037250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:29.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.201+0000 7f2c367fc640 1 --2- 172.21.15.16:0/422623616 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5642c102ebb0 0x5642c1037250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.201+0000 7f2c367fc640 1 -- 172.21.15.16:0/422623616 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5642c102e290 msgr2=0x5642c102e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:29.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.201+0000 7f2c367fc640 1 --2- 172.21.15.16:0/422623616 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5642c102e290 0x5642c102e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.201+0000 7f2c367fc640 1 -- 172.21.15.16:0/422623616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642c0f4e7c0 con 0x5642c1037790 2024-03-28T22:11:29.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.201+0000 7f2c36ffd640 1 --2- 172.21.15.16:0/422623616 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5642c102ebb0 0x5642c1037250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:11:29.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.202+0000 7f2c367fc640 1 --2- 172.21.15.16:0/422623616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c1037790 0x5642c1039b80 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c0ed470 tx=0x7f2c2c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c39ef328e27c3c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:11:29.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.202+0000 7f2c1d7fa640 1 -- 172.21.15.16:0/422623616 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c04b070 con 0x5642c1037790 2024-03-28T22:11:29.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.202+0000 7f2c1d7fa640 1 -- 172.21.15.16:0/422623616 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c2c004c80 con 0x5642c1037790 2024-03-28T22:11:29.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.202+0000 7f2c1d7fa640 1 -- 172.21.15.16:0/422623616 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c0bc640 con 0x5642c1037790 2024-03-28T22:11:29.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.202+0000 7f2c377fe640 1 --2- 172.21.15.16:0/1002700204 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f2c200a2010 0x7f2c200a44d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2c30007fd0 tx=0x7f2c30004ab0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:11:29.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.203+0000 7f2c4017b740 1 -- 172.21.15.16:0/422623616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c1037790 msgr2=0x5642c1039b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:29.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.203+0000 7f2c4017b740 1 --2- 172.21.15.16:0/422623616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c1037790 0x5642c1039b80 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c0ed470 tx=0x7f2c2c000bc0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.203+0000 7f2c4017b740 1 -- 172.21.15.16:0/422623616 shutdown_connections 2024-03-28T22:11:29.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.203+0000 7f2c4017b740 1 --2- 172.21.15.16:0/422623616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c1037790 0x5642c1039b80 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.203+0000 7f2c4017b740 1 --2- 172.21.15.16:0/422623616 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5642c102ebb0 0x5642c1037250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.203+0000 7f2c4017b740 1 --2- 172.21.15.16:0/422623616 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5642c102e290 0x5642c102e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.203+0000 7f2c4017b740 1 -- 172.21.15.16:0/422623616 >> 172.21.15.16:0/422623616 conn(0x5642c10289c0 msgr2=0x5642c1028da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:11:29.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.203+0000 7f2c4017b740 1 -- 172.21.15.16:0/422623616 shutdown_connections 2024-03-28T22:11:29.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.203+0000 7f2c4017b740 1 -- 172.21.15.16:0/422623616 wait complete. 2024-03-28T22:11:29.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.203+0000 7f2c4017b740 1 -- start start 2024-03-28T22:11:29.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.204+0000 7f2c4017b740 1 Processor -- start 2024-03-28T22:11:29.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.204+0000 7f2c4017b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5642c102e290 0x5642c1034840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:29.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.204+0000 7f2c4017b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c102ebb0 0x5642c1034d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:29.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.204+0000 7f2c4017b740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5642c1037790 0x5642c10352c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:29.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.204+0000 7f2c4017b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5642c0f92180 con 0x5642c102ebb0 2024-03-28T22:11:29.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.204+0000 7f2c367fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c102ebb0 0x5642c1034d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:29.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.204+0000 7f2c4017b740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5642c0f304a0 con 0x5642c1037790 2024-03-28T22:11:29.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.204+0000 7f2c4017b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5642c103e750 con 0x5642c102e290 2024-03-28T22:11:29.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.204+0000 7f2c367fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c102ebb0 0x5642c1034d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57852/0 (socket says 172.21.15.16:57852) 2024-03-28T22:11:29.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.204+0000 7f2c367fc640 1 -- 172.21.15.16:0/1673544056 learned_addr learned my addr 172.21.15.16:0/1673544056 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:11:29.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.204+0000 7f2c377fe640 1 --2- 172.21.15.16:0/1673544056 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5642c1037790 0x5642c10352c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:29.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.204+0000 7f2c36ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5642c102e290 0x5642c1034840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:29.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.205+0000 7f2c4017b740 10 client.0 _ll_register_callbacks cb 0x5642c0f33d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:11:29.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.205+0000 7f2c367fc640 1 -- 172.21.15.16:0/1673544056 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5642c102e290 msgr2=0x5642c1034840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:29.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.205+0000 7f2c367fc640 1 --2- 172.21.15.16:0/1673544056 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5642c102e290 0x5642c1034840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.205+0000 7f2c367fc640 1 -- 172.21.15.16:0/1673544056 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5642c1037790 msgr2=0x5642c10352c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:29.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.205+0000 7f2c367fc640 1 --2- 172.21.15.16:0/1673544056 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5642c1037790 0x5642c10352c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.205+0000 7f2c367fc640 1 -- 172.21.15.16:0/1673544056 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642c0d707d0 con 0x5642c102ebb0 2024-03-28T22:11:29.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.205+0000 7f2c377fe640 1 --2- 172.21.15.16:0/1673544056 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5642c1037790 0x5642c10352c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:11:29.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.205+0000 7f2c4017b740 1 -- 172.21.15.16:0/1673544056 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c2c004c80 con 0x5642c102ebb0 2024-03-28T22:11:29.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.205+0000 7f2c367fc640 1 --2- 172.21.15.16:0/1673544056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c102ebb0 0x5642c1034d80 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c0edae0 tx=0x7f2c2c0dc360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a56756c2b1475293 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:11:29.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.205+0000 7f2c1aff5640 10 client.15651 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:11:29.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.205+0000 7f2c1aff5640 1 -- 172.21.15.16:0/1673544056 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c04b070 con 0x5642c102ebb0 2024-03-28T22:11:29.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.205+0000 7f2c1aff5640 1 -- 172.21.15.16:0/1673544056 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f2c2c0035f0 con 0x5642c102ebb0 2024-03-28T22:11:29.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.205+0000 7f2c1aff5640 1 -- 172.21.15.16:0/1673544056 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f2c2c0bc180 con 0x5642c102ebb0 2024-03-28T22:11:29.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.205+0000 7f2c4017b740 10 client.15651 fetch_fsmap learned FSMap version 21 2024-03-28T22:11:29.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.205+0000 7f2c4017b740 1 -- 172.21.15.16:0/1673544056 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({fsmap=21}) -- 0x5642c0ca63f0 con 0x5642c102ebb0 2024-03-28T22:11:29.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.206+0000 7f2c1aff5640 1 -- 172.21.15.16:0/1673544056 <== mon.0 v2:172.21.15.16:3300/0 4 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f2c2c0c4950 con 0x5642c102ebb0 2024-03-28T22:11:29.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.206+0000 7f2c4017b740 10 client.15651 fetch_fsmap finished waiting for FSMap version 21 2024-03-28T22:11:29.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.206+0000 7f2c4017b740 10 client.15651 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:11:29.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.206+0000 7f2c4017b740 20 client.15651 populate_metadata read hostname 'smithi016' 2024-03-28T22:11:29.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.206+0000 7f2c4017b740 1 --2- 172.21.15.16:0/1673544056 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5642c104b8e0 0x5642c104dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:29.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.206+0000 7f2c4017b740 4 client.15651 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:11:29.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.206+0000 7f2c4017b740 1 -- 172.21.15.16:0/1673544056 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f2c2c004c80 con 0x5642c104b8e0 2024-03-28T22:11:29.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.206+0000 7f2c36ffd640 1 --2- 172.21.15.16:0/1673544056 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5642c104b8e0 0x5642c104dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:29.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.207+0000 7f2c1aff5640 10 client.15651 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:11:29.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.207+0000 7f2c36ffd640 1 --2- 172.21.15.16:0/1673544056 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5642c104b8e0 0x5642c104dcd0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7aef96d4a3725eeb server_cookie=56a79c28d15d7e56 in_seq=0 out_seq=0 2024-03-28T22:11:29.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.208+0000 7f2c1aff5640 1 -- 172.21.15.16:0/1673544056 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f2c2c004c80 con 0x5642c104b8e0 2024-03-28T22:11:29.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.208+0000 7f2c1aff5640 10 client.15651 handle_command_reply: tid=0 2024-03-28T22:11:29.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.208+0000 7f2c4017b740 1 client.15651 shutdown 2024-03-28T22:11:29.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.208+0000 7f2c4017b740 1 -- 172.21.15.16:0/1673544056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c102ebb0 msgr2=0x5642c1034d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:29.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.208+0000 7f2c4017b740 1 --2- 172.21.15.16:0/1673544056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c102ebb0 0x5642c1034d80 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c0edae0 tx=0x7f2c2c0dc360 comp rx=0 tx=0).stop 2024-03-28T22:11:29.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.208+0000 7f2c4017b740 1 -- 172.21.15.16:0/1673544056 shutdown_connections 2024-03-28T22:11:29.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.208+0000 7f2c4017b740 1 --2- 172.21.15.16:0/1673544056 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5642c104b8e0 0x5642c104dcd0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.208+0000 7f2c4017b740 1 --2- 172.21.15.16:0/1673544056 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5642c1037790 0x5642c10352c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.208+0000 7f2c4017b740 1 --2- 172.21.15.16:0/1673544056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c102ebb0 0x5642c1034d80 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.208+0000 7f2c4017b740 1 --2- 172.21.15.16:0/1673544056 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5642c102e290 0x5642c1034840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.208+0000 7f2c4017b740 1 -- 172.21.15.16:0/1673544056 >> 172.21.15.16:0/1673544056 conn(0x5642c10289c0 msgr2=0x5642c102a710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:11:29.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.208+0000 7f2c1aff5640 0 client.15651 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:11:29.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.208+0000 7f2c4017b740 1 -- 172.21.15.16:0/1673544056 shutdown_connections 2024-03-28T22:11:29.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.208+0000 7f2c4017b740 1 -- 172.21.15.16:0/1673544056 wait complete. 2024-03-28T22:11:29.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.208+0000 7f2c4017b740 20 client.15651 trim_cache size 0 max 16384 2024-03-28T22:11:29.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.241+0000 7f2c4017b740 1 Processor -- start 2024-03-28T22:11:29.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.241+0000 7f2c4017b740 1 -- start start 2024-03-28T22:11:29.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.242+0000 7f2c4017b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5642c102e290 0x5642c1036f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:29.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.242+0000 7f2c4017b740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5642c102ebb0 0x5642c1033780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:29.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.242+0000 7f2c4017b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c1037790 0x5642c104d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:29.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.242+0000 7f2c367fc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5642c102ebb0 0x5642c1033780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:29.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.242+0000 7f2c377fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c1037790 0x5642c104d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:29.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.242+0000 7f2c367fc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5642c102ebb0 0x5642c1033780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:50226/0 (socket says 172.21.15.16:50226) 2024-03-28T22:11:29.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.242+0000 7f2c367fc640 1 -- 172.21.15.16:0/4001698605 learned_addr learned my addr 172.21.15.16:0/4001698605 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:11:29.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.242+0000 7f2c4017b740 1 -- 172.21.15.16:0/4001698605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2c381ad9c0 con 0x5642c1037790 2024-03-28T22:11:29.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.242+0000 7f2c4017b740 1 -- 172.21.15.16:0/4001698605 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f2c381ab290 con 0x5642c102ebb0 2024-03-28T22:11:29.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.242+0000 7f2c4017b740 1 -- 172.21.15.16:0/4001698605 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2c381a90e0 con 0x5642c102e290 2024-03-28T22:11:29.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.242+0000 7f2c36ffd640 1 --2- 172.21.15.16:0/4001698605 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5642c102e290 0x5642c1036f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:29.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.242+0000 7f2c367fc640 1 -- 172.21.15.16:0/4001698605 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5642c102e290 msgr2=0x5642c1036f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:29.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.242+0000 7f2c367fc640 1 --2- 172.21.15.16:0/4001698605 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5642c102e290 0x5642c1036f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.242+0000 7f2c367fc640 1 -- 172.21.15.16:0/4001698605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c1037790 msgr2=0x5642c104d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:29.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.242+0000 7f2c367fc640 1 --2- 172.21.15.16:0/4001698605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c1037790 0x5642c104d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.242+0000 7f2c367fc640 1 -- 172.21.15.16:0/4001698605 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642c0c660f0 con 0x5642c102ebb0 2024-03-28T22:11:29.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.243+0000 7f2c367fc640 1 --2- 172.21.15.16:0/4001698605 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5642c102ebb0 0x5642c1033780 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c009430 tx=0x7f2c2c0a7000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b7f4a9d15fa87e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:11:29.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.243+0000 7f2c1cff9640 1 -- 172.21.15.16:0/4001698605 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c0bbfa0 con 0x5642c102ebb0 2024-03-28T22:11:29.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.243+0000 7f2c1cff9640 1 -- 172.21.15.16:0/4001698605 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c2c004c80 con 0x5642c102ebb0 2024-03-28T22:11:29.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.243+0000 7f2c1cff9640 1 -- 172.21.15.16:0/4001698605 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c0035f0 con 0x5642c102ebb0 2024-03-28T22:11:29.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.244+0000 7f2c4017b740 1 -- 172.21.15.16:0/4001698605 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5642c102ebb0 msgr2=0x5642c1033780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:29.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.244+0000 7f2c4017b740 1 --2- 172.21.15.16:0/4001698605 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5642c102ebb0 0x5642c1033780 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c009430 tx=0x7f2c2c0a7000 comp rx=0 tx=0).stop 2024-03-28T22:11:29.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.244+0000 7f2c4017b740 1 -- 172.21.15.16:0/4001698605 shutdown_connections 2024-03-28T22:11:29.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.244+0000 7f2c4017b740 1 --2- 172.21.15.16:0/4001698605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c1037790 0x5642c104d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.244+0000 7f2c4017b740 1 --2- 172.21.15.16:0/4001698605 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5642c102ebb0 0x5642c1033780 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.244+0000 7f2c4017b740 1 --2- 172.21.15.16:0/4001698605 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5642c102e290 0x5642c1036f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.244+0000 7f2c4017b740 1 -- 172.21.15.16:0/4001698605 >> 172.21.15.16:0/4001698605 conn(0x5642c10289c0 msgr2=0x5642c1027c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:11:29.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.244+0000 7f2c4017b740 1 -- 172.21.15.16:0/4001698605 shutdown_connections 2024-03-28T22:11:29.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.244+0000 7f2c4017b740 1 -- 172.21.15.16:0/4001698605 wait complete. 2024-03-28T22:11:29.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.244+0000 7f2c4017b740 1 -- start start 2024-03-28T22:11:29.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.245+0000 7f2c4017b740 1 Processor -- start 2024-03-28T22:11:29.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.245+0000 7f2c4017b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5642c102e290 0x5642c1043c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:29.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.245+0000 7f2c4017b740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5642c102ebb0 0x5642c1031190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:29.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.245+0000 7f2c4017b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c1037790 0x5642c1031730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:29.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.245+0000 7f2c367fc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5642c102ebb0 0x5642c1031190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:29.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.245+0000 7f2c367fc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5642c102ebb0 0x5642c1031190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:50232/0 (socket says 172.21.15.16:50232) 2024-03-28T22:11:29.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.245+0000 7f2c377fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c1037790 0x5642c1031730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:29.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.245+0000 7f2c36ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5642c102e290 0x5642c1043c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:29.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.245+0000 7f2c36ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5642c102e290 0x5642c1043c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.16:54648/0 (socket says 172.21.15.16:54648) 2024-03-28T22:11:29.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.245+0000 7f2c377fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c1037790 0x5642c1031730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57874/0 (socket says 172.21.15.16:57874) 2024-03-28T22:11:29.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.245+0000 7f2c367fc640 1 -- 172.21.15.16:0/1484289462 learned_addr learned my addr 172.21.15.16:0/1484289462 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:11:29.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.245+0000 7f2c4017b740 1 -- 172.21.15.16:0/1484289462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2c381a9750 con 0x5642c1037790 2024-03-28T22:11:29.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.245+0000 7f2c4017b740 1 -- 172.21.15.16:0/1484289462 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f2c381aabc0 con 0x5642c102ebb0 2024-03-28T22:11:29.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.245+0000 7f2c4017b740 1 -- 172.21.15.16:0/1484289462 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2c381a9e20 con 0x5642c102e290 2024-03-28T22:11:29.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.246+0000 7f2c4017b740 10 client.0 _ll_register_callbacks cb 0x7f2c3806afe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:11:29.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.246+0000 7f2c367fc640 1 -- 172.21.15.16:0/1484289462 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5642c102e290 msgr2=0x5642c1043c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:29.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.246+0000 7f2c367fc640 1 --2- 172.21.15.16:0/1484289462 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5642c102e290 0x5642c1043c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.246+0000 7f2c367fc640 1 -- 172.21.15.16:0/1484289462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c1037790 msgr2=0x5642c1031730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:29.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.246+0000 7f2c367fc640 1 --2- 172.21.15.16:0/1484289462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c1037790 0x5642c1031730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.246+0000 7f2c367fc640 1 -- 172.21.15.16:0/1484289462 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642c1031f60 con 0x5642c102ebb0 2024-03-28T22:11:29.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.246+0000 7f2c4017b740 1 -- 172.21.15.16:0/1484289462 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c2c004c80 con 0x5642c102ebb0 2024-03-28T22:11:29.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.246+0000 7f2c367fc640 1 --2- 172.21.15.16:0/1484289462 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5642c102ebb0 0x5642c1031190 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c0edae0 tx=0x7f2c2c005be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b5f7ce846857270 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:11:29.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.246+0000 7f2c1aff5640 10 client.25565 ms_handle_connect on v2:172.21.15.53:3300/0 2024-03-28T22:11:29.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.246+0000 7f2c1aff5640 1 -- 172.21.15.16:0/1484289462 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c0da070 con 0x5642c102ebb0 2024-03-28T22:11:29.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.246+0000 7f2c1aff5640 1 -- 172.21.15.16:0/1484289462 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f2c2c0c4950 con 0x5642c102ebb0 2024-03-28T22:11:29.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.246+0000 7f2c1aff5640 1 -- 172.21.15.16:0/1484289462 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f2c2c0bbfa0 con 0x5642c102ebb0 2024-03-28T22:11:29.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.246+0000 7f2c4017b740 10 client.25565 fetch_fsmap learned FSMap version 21 2024-03-28T22:11:29.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.247+0000 7f2c4017b740 1 -- 172.21.15.16:0/1484289462 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({fsmap=21}) -- 0x5642c1031f60 con 0x5642c102ebb0 2024-03-28T22:11:29.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.247+0000 7f2c1aff5640 1 -- 172.21.15.16:0/1484289462 <== mon.2 v2:172.21.15.53:3300/0 4 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f2c2c0a5b80 con 0x5642c102ebb0 2024-03-28T22:11:29.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.247+0000 7f2c4017b740 10 client.25565 fetch_fsmap finished waiting for FSMap version 21 2024-03-28T22:11:29.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.247+0000 7f2c4017b740 10 client.25565 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:11:29.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.247+0000 7f2c4017b740 20 client.25565 populate_metadata read hostname 'smithi016' 2024-03-28T22:11:29.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.247+0000 7f2c4017b740 1 --2- 172.21.15.16:0/1484289462 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5642c10b1ca0 0x5642c10b4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:29.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.247+0000 7f2c4017b740 4 client.25565 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:11:29.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.247+0000 7f2c4017b740 1 -- 172.21.15.16:0/1484289462 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5642c0f9f6c0 con 0x5642c10b1ca0 2024-03-28T22:11:29.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.247+0000 7f2c36ffd640 1 --2- 172.21.15.16:0/1484289462 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5642c10b1ca0 0x5642c10b4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:29.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.248+0000 7f2c1aff5640 10 client.25565 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:11:29.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.248+0000 7f2c36ffd640 1 --2- 172.21.15.16:0/1484289462 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5642c10b1ca0 0x5642c10b4090 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5077147a31e7b266 server_cookie=5537523671110543 in_seq=0 out_seq=0 2024-03-28T22:11:29.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.248+0000 7f2c1aff5640 1 -- 172.21.15.16:0/1484289462 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5642c0f9f6c0 con 0x5642c10b1ca0 2024-03-28T22:11:29.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.248+0000 7f2c1aff5640 10 client.25565 handle_command_reply: tid=0 2024-03-28T22:11:29.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.248+0000 7f2c4017b740 1 client.25565 shutdown 2024-03-28T22:11:29.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.249+0000 7f2c4017b740 1 -- 172.21.15.16:0/1484289462 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5642c102ebb0 msgr2=0x5642c1031190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:29.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.249+0000 7f2c4017b740 1 --2- 172.21.15.16:0/1484289462 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5642c102ebb0 0x5642c1031190 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c0edae0 tx=0x7f2c2c005be0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.249+0000 7f2c4017b740 1 -- 172.21.15.16:0/1484289462 shutdown_connections 2024-03-28T22:11:29.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.249+0000 7f2c4017b740 1 --2- 172.21.15.16:0/1484289462 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5642c10b1ca0 0x5642c10b4090 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.249+0000 7f2c4017b740 1 --2- 172.21.15.16:0/1484289462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c1037790 0x5642c1031730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.249+0000 7f2c1aff5640 0 client.25565 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:11:29.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.249+0000 7f2c4017b740 1 --2- 172.21.15.16:0/1484289462 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5642c102ebb0 0x5642c1031190 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.249+0000 7f2c4017b740 1 --2- 172.21.15.16:0/1484289462 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5642c102e290 0x5642c1043c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.249+0000 7f2c4017b740 1 -- 172.21.15.16:0/1484289462 >> 172.21.15.16:0/1484289462 conn(0x5642c10289c0 msgr2=0x5642c1027390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:11:29.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.249+0000 7f2c4017b740 1 -- 172.21.15.16:0/1484289462 shutdown_connections 2024-03-28T22:11:29.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.249+0000 7f2c4017b740 1 -- 172.21.15.16:0/1484289462 wait complete. 2024-03-28T22:11:29.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.249+0000 7f2c4017b740 20 client.25565 trim_cache size 0 max 16384 2024-03-28T22:11:29.257 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:11:29.257 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-03-28T22:11:29.257 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-03-28T22:11:29.257 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:11:29.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.250+0000 7f2c3d593640 1 -- 172.21.15.16:0/1002700204 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f2c200a2010 msgr2=0x7f2c200a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:29.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.250+0000 7f2c3d593640 1 --2- 172.21.15.16:0/1002700204 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f2c200a2010 0x7f2c200a44d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2c30007fd0 tx=0x7f2c30004ab0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.250+0000 7f2c3d593640 1 -- 172.21.15.16:0/1002700204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c38100fb0 msgr2=0x7f2c38198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:29.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.250+0000 7f2c3d593640 1 --2- 172.21.15.16:0/1002700204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c38100fb0 0x7f2c38198450 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c009230 tx=0x7f2c2c02cba0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.250+0000 7f2c3d593640 1 -- 172.21.15.16:0/1002700204 shutdown_connections 2024-03-28T22:11:29.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.250+0000 7f2c3d593640 1 --2- 172.21.15.16:0/1002700204 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f2c200a2010 0x7f2c200a44d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.250+0000 7f2c3d593640 1 --2- 172.21.15.16:0/1002700204 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c38106690 0x7f2c3819c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.250+0000 7f2c3d593640 1 --2- 172.21.15.16:0/1002700204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c38100fb0 0x7f2c38198450 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.250+0000 7f2c3d593640 1 --2- 172.21.15.16:0/1002700204 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c381005f0 0x7f2c38197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.250+0000 7f2c3d593640 1 -- 172.21.15.16:0/1002700204 >> 172.21.15.16:0/1002700204 conn(0x7f2c38077800 msgr2=0x7f2c380ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:11:29.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.251+0000 7f2c3d593640 1 -- 172.21.15.16:0/1002700204 shutdown_connections 2024-03-28T22:11:29.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.251+0000 7f2c3d593640 1 -- 172.21.15.16:0/1002700204 wait complete. 2024-03-28T22:11:29.270 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:11:29.270 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:11:29.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.422+0000 7fd04159a640 1 -- 172.21.15.16:0/742241443 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03802cd20 con 0x7fd044103290 2024-03-28T22:11:29.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.423+0000 7fd048f56640 1 -- 172.21.15.16:0/742241443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd044103290 msgr2=0x7fd04410b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:29.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.423+0000 7fd048f56640 1 --2- 172.21.15.16:0/742241443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd044103290 0x7fd04410b770 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fd038009900 tx=0x7fd03802bcd0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.423+0000 7fd048f56640 1 -- 172.21.15.16:0/742241443 shutdown_connections 2024-03-28T22:11:29.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.423+0000 7fd048f56640 1 --2- 172.21.15.16:0/742241443 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd04410bd40 0x7fd04410e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.423+0000 7fd048f56640 1 --2- 172.21.15.16:0/742241443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd044103290 0x7fd04410b770 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.423+0000 7fd048f56640 1 --2- 172.21.15.16:0/742241443 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd044102970 0x7fd044102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.423+0000 7fd048f56640 1 -- 172.21.15.16:0/742241443 >> 172.21.15.16:0/742241443 conn(0x7fd0440777d0 msgr2=0x7fd044077be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:11:29.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.423+0000 7fd048f56640 1 -- 172.21.15.16:0/742241443 shutdown_connections 2024-03-28T22:11:29.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.423+0000 7fd048f56640 1 -- 172.21.15.16:0/742241443 wait complete. 2024-03-28T22:11:29.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.424+0000 7fd048f56640 1 Processor -- start 2024-03-28T22:11:29.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.424+0000 7fd048f56640 1 -- start start 2024-03-28T22:11:29.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.424+0000 7fd048f56640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd044102970 0x7fd04419a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:29.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.424+0000 7fd048f56640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd044103290 0x7fd04419a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:29.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.424+0000 7fd048f56640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd04410bd40 0x7fd04419ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:29.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.424+0000 7fd048f56640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd044101020 con 0x7fd044102970 2024-03-28T22:11:29.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.424+0000 7fd048f56640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd044100ea0 con 0x7fd04410bd40 2024-03-28T22:11:29.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.424+0000 7fd048f56640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd0441011a0 con 0x7fd044103290 2024-03-28T22:11:29.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.424+0000 7fd042d9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd04410bd40 0x7fd04419ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:29.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.424+0000 7fd04259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd044102970 0x7fd04419a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:29.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.424+0000 7fd042d9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd04410bd40 0x7fd04419ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:50264/0 (socket says 172.21.15.16:50264) 2024-03-28T22:11:29.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.424+0000 7fd042d9d640 1 -- 172.21.15.16:0/2471516290 learned_addr learned my addr 172.21.15.16:0/2471516290 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:11:29.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.424+0000 7fd041d9b640 1 --2- 172.21.15.16:0/2471516290 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd044103290 0x7fd04419a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:29.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.424+0000 7fd042d9d640 1 -- 172.21.15.16:0/2471516290 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd044103290 msgr2=0x7fd04419a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:29.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.424+0000 7fd042d9d640 1 --2- 172.21.15.16:0/2471516290 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd044103290 0x7fd04419a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.424+0000 7fd042d9d640 1 -- 172.21.15.16:0/2471516290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd044102970 msgr2=0x7fd04419a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:29.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.424+0000 7fd042d9d640 1 --2- 172.21.15.16:0/2471516290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd044102970 0x7fd04419a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.424+0000 7fd042d9d640 1 -- 172.21.15.16:0/2471516290 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd04419f180 con 0x7fd04410bd40 2024-03-28T22:11:29.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.424+0000 7fd041d9b640 1 --2- 172.21.15.16:0/2471516290 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd044103290 0x7fd04419a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:11:29.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.424+0000 7fd04259c640 1 --2- 172.21.15.16:0/2471516290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd044102970 0x7fd04419a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:11:29.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.425+0000 7fd042d9d640 1 --2- 172.21.15.16:0/2471516290 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd04410bd40 0x7fd04419ea50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd03c00efd0 tx=0x7fd03c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:11:29.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.425+0000 7fd0337fe640 1 -- 172.21.15.16:0/2471516290 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03c003b70 con 0x7fd04410bd40 2024-03-28T22:11:29.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.425+0000 7fd0337fe640 1 -- 172.21.15.16:0/2471516290 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd03c003d10 con 0x7fd04410bd40 2024-03-28T22:11:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.425+0000 7fd048f56640 1 -- 172.21.15.16:0/2471516290 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd04419f470 con 0x7fd04410bd40 2024-03-28T22:11:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.425+0000 7fd0337fe640 1 -- 172.21.15.16:0/2471516290 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03c0108d0 con 0x7fd04410bd40 2024-03-28T22:11:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.426+0000 7fd048f56640 1 -- 172.21.15.16:0/2471516290 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0441a6d00 con 0x7fd04410bd40 2024-03-28T22:11:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.427+0000 7fd048f56640 1 -- 172.21.15.16:0/2471516290 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0440685d0 con 0x7fd04410bd40 2024-03-28T22:11:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.427+0000 7fd0337fe640 1 -- 172.21.15.16:0/2471516290 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 25) ==== 136989+0+0 (secure 0 0 0) 0x7fd03c004020 con 0x7fd04410bd40 2024-03-28T22:11:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.428+0000 7fd0337fe640 1 --2- 172.21.15.16:0/2471516290 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fd02c0a2010 0x7fd02c0a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.428+0000 7fd0337fe640 1 -- 172.21.15.16:0/2471516290 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fd03c0ce7c0 con 0x7fd04410bd40 2024-03-28T22:11:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.428+0000 7fd04259c640 1 --2- 172.21.15.16:0/2471516290 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fd02c0a2010 0x7fd02c0a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.429+0000 7fd04259c640 1 --2- 172.21.15.16:0/2471516290 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fd02c0a2010 0x7fd02c0a44d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd03400f3c0 tx=0x7fd03400a2d0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:11:29.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.433+0000 7fd0337fe640 1 -- 172.21.15.16:0/2471516290 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd03c09b680 con 0x7fd04410bd40 2024-03-28T22:11:29.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.885+0000 7fd048f56640 1 -- 172.21.15.16:0/2471516290 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd044105e90 con 0x7fd04410bd40 2024-03-28T22:11:29.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.886+0000 7fd0337fe640 1 -- 172.21.15.16:0/2471516290 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11851 (secure 0 0 0) 0x7fd03c09af80 con 0x7fd04410bd40 2024-03-28T22:11:29.889 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:11:29.889 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/1644581227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1644581227},{"type":"v1","addr":"172.21.15.16:6839","nonce":1644581227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/2593062669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2593062669},{"type":"v1","addr":"172.21.15.16:6841","nonce":2593062669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:08:29.672976+0000","modified":"2024-03-28T22:10:19.955098+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24566,"mds_1":24524,"mds_2":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24466":{"gid":24466,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/247249857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":247249857},{"type":"v1","addr":"172.21.15.63:6835","nonce":247249857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6837/2324231032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":2324231032},{"type":"v1","addr":"172.21.15.63:6837","nonce":2324231032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6839/1177197077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":1177197077},{"type":"v1","addr":"172.21.15.63:6839","nonce":1177197077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"e","rank":2,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.53:6835/350727634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":350727634},{"type":"v1","addr":"172.21.15.53:6835","nonce":350727634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/349877573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":349877573},{"type":"v1","addr":"172.21.15.53:6837","nonce":349877573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6839/3714153956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":3714153956},{"type":"v1","addr":"172.21.15.53:6839","nonce":3714153956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24566,"qdb_cluster":[24500,24524,24566]},"id":1}]} 2024-03-28T22:11:29.890 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 21 2024-03-28T22:11:29.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.889+0000 7fd048f56640 1 -- 172.21.15.16:0/2471516290 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fd02c0a2010 msgr2=0x7fd02c0a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:29.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.889+0000 7fd048f56640 1 --2- 172.21.15.16:0/2471516290 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fd02c0a2010 0x7fd02c0a44d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd03400f3c0 tx=0x7fd03400a2d0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.889+0000 7fd048f56640 1 -- 172.21.15.16:0/2471516290 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd04410bd40 msgr2=0x7fd04419ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:29.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.889+0000 7fd048f56640 1 --2- 172.21.15.16:0/2471516290 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd04410bd40 0x7fd04419ea50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd03c00efd0 tx=0x7fd03c00c6a0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.889+0000 7fd048f56640 1 -- 172.21.15.16:0/2471516290 shutdown_connections 2024-03-28T22:11:29.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.889+0000 7fd048f56640 1 --2- 172.21.15.16:0/2471516290 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fd02c0a2010 0x7fd02c0a44d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.889+0000 7fd048f56640 1 --2- 172.21.15.16:0/2471516290 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd04410bd40 0x7fd04419ea50 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.889+0000 7fd048f56640 1 --2- 172.21.15.16:0/2471516290 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd044103290 0x7fd04419a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.889+0000 7fd048f56640 1 --2- 172.21.15.16:0/2471516290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd044102970 0x7fd04419a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:29.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.889+0000 7fd048f56640 1 -- 172.21.15.16:0/2471516290 >> 172.21.15.16:0/2471516290 conn(0x7fd0440777d0 msgr2=0x7fd04410d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:11:29.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.889+0000 7fd048f56640 1 -- 172.21.15.16:0/2471516290 shutdown_connections 2024-03-28T22:11:29.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:29.889+0000 7fd048f56640 1 -- 172.21.15.16:0/2471516290 wait complete. 2024-03-28T22:11:29.907 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:11:30.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.065+0000 7ff9ba7fc640 1 -- 172.21.15.16:0/3568313492 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a402cca0 con 0x7ff9bc0fffd0 2024-03-28T22:11:30.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.066+0000 7ff9c19d0640 1 -- 172.21.15.16:0/3568313492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9bc0fffd0 msgr2=0x7ff9bc100450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:30.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.066+0000 7ff9c19d0640 1 --2- 172.21.15.16:0/3568313492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9bc0fffd0 0x7ff9bc100450 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4009900 tx=0x7ff9a402bc80 comp rx=0 tx=0).stop 2024-03-28T22:11:30.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.067+0000 7ff9c19d0640 1 -- 172.21.15.16:0/3568313492 shutdown_connections 2024-03-28T22:11:30.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.067+0000 7ff9c19d0640 1 --2- 172.21.15.16:0/3568313492 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9bc100990 0x7ff9bc10e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.067+0000 7ff9c19d0640 1 --2- 172.21.15.16:0/3568313492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9bc0fffd0 0x7ff9bc100450 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.067+0000 7ff9c19d0640 1 --2- 172.21.15.16:0/3568313492 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9bc107860 0x7ff9bc107c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.067+0000 7ff9c19d0640 1 -- 172.21.15.16:0/3568313492 >> 172.21.15.16:0/3568313492 conn(0x7ff9bc0758b0 msgr2=0x7ff9bc075cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:11:30.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.067+0000 7ff9c19d0640 1 -- 172.21.15.16:0/3568313492 shutdown_connections 2024-03-28T22:11:30.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.067+0000 7ff9c19d0640 1 -- 172.21.15.16:0/3568313492 wait complete. 2024-03-28T22:11:30.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.068+0000 7ff9c19d0640 1 Processor -- start 2024-03-28T22:11:30.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.068+0000 7ff9c19d0640 1 -- start start 2024-03-28T22:11:30.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.069+0000 7ff9c19d0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9bc0fffd0 0x7ff9bc198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.069+0000 7ff9c19d0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9bc100990 0x7ff9bc198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.069+0000 7ff9baffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9bc0fffd0 0x7ff9bc198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.069+0000 7ff9baffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9bc0fffd0 0x7ff9bc198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57906/0 (socket says 172.21.15.16:57906) 2024-03-28T22:11:30.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.069+0000 7ff9c19d0640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9bc107860 0x7ff9bc19c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.069+0000 7ff9baffd640 1 -- 172.21.15.16:0/1082086044 learned_addr learned my addr 172.21.15.16:0/1082086044 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:11:30.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.069+0000 7ff9c19d0640 1 -- 172.21.15.16:0/1082086044 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff9bc111b90 con 0x7ff9bc0fffd0 2024-03-28T22:11:30.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.069+0000 7ff9b3fff640 1 --2- 172.21.15.16:0/1082086044 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9bc100990 0x7ff9bc198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.069+0000 7ff9bb7fe640 1 --2- 172.21.15.16:0/1082086044 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9bc107860 0x7ff9bc19c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.069+0000 7ff9c19d0640 1 -- 172.21.15.16:0/1082086044 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff9bc111a10 con 0x7ff9bc107860 2024-03-28T22:11:30.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.069+0000 7ff9c19d0640 1 -- 172.21.15.16:0/1082086044 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff9bc111d10 con 0x7ff9bc100990 2024-03-28T22:11:30.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.070+0000 7ff9baffd640 1 -- 172.21.15.16:0/1082086044 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9bc100990 msgr2=0x7ff9bc198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:30.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.070+0000 7ff9baffd640 1 --2- 172.21.15.16:0/1082086044 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9bc100990 0x7ff9bc198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.070+0000 7ff9baffd640 1 -- 172.21.15.16:0/1082086044 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9bc107860 msgr2=0x7ff9bc19c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:30.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.070+0000 7ff9baffd640 1 --2- 172.21.15.16:0/1082086044 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9bc107860 0x7ff9bc19c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.070+0000 7ff9baffd640 1 -- 172.21.15.16:0/1082086044 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9bc19cff0 con 0x7ff9bc0fffd0 2024-03-28T22:11:30.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.070+0000 7ff9bb7fe640 1 --2- 172.21.15.16:0/1082086044 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9bc107860 0x7ff9bc19c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:11:30.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.070+0000 7ff9b3fff640 1 --2- 172.21.15.16:0/1082086044 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9bc100990 0x7ff9bc198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:11:30.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.070+0000 7ff9baffd640 1 --2- 172.21.15.16:0/1082086044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9bc0fffd0 0x7ff9bc198000 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac00e9c0 tx=0x7ff9ac00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:11:30.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.070+0000 7ff9c09ce640 1 -- 172.21.15.16:0/1082086044 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ac00cd80 con 0x7ff9bc0fffd0 2024-03-28T22:11:30.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.070+0000 7ff9c09ce640 1 -- 172.21.15.16:0/1082086044 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9ac0035f0 con 0x7ff9bc0fffd0 2024-03-28T22:11:30.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.070+0000 7ff9c19d0640 1 -- 172.21.15.16:0/1082086044 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9bc19d2e0 con 0x7ff9bc0fffd0 2024-03-28T22:11:30.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.071+0000 7ff9c09ce640 1 -- 172.21.15.16:0/1082086044 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ac010690 con 0x7ff9bc0fffd0 2024-03-28T22:11:30.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.071+0000 7ff9c19d0640 1 -- 172.21.15.16:0/1082086044 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9bc1a4b70 con 0x7ff9bc0fffd0 2024-03-28T22:11:30.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.074+0000 7ff9c09ce640 1 -- 172.21.15.16:0/1082086044 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 25) ==== 136989+0+0 (secure 0 0 0) 0x7ff9ac0037a0 con 0x7ff9bc0fffd0 2024-03-28T22:11:30.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.075+0000 7ff9c09ce640 1 --2- 172.21.15.16:0/1082086044 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff9a80a2090 0x7ff9a80a4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.075+0000 7ff9c45ba740 1 Processor -- start 2024-03-28T22:11:30.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.075+0000 7ff9c45ba740 1 -- start start 2024-03-28T22:11:30.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.075+0000 7ff9c09ce640 1 -- 172.21.15.16:0/1082086044 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7ff9ac0ce860 con 0x7ff9bc0fffd0 2024-03-28T22:11:30.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.075+0000 7ff9b3fff640 1 --2- 172.21.15.16:0/1082086044 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff9a80a2090 0x7ff9a80a4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.075+0000 7ff9c45ba740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a18f4b470 0x557a18f4b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.075+0000 7ff9c45ba740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a18f4bd90 0x557a18f54430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.075+0000 7ff9c45ba740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a18f54970 0x557a18f56d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.075+0000 7ff9c45ba740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557a18eaf1b0 con 0x557a18f4b470 2024-03-28T22:11:30.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.075+0000 7ff9c45ba740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x557a18e57c00 con 0x557a18f54970 2024-03-28T22:11:30.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.075+0000 7ff9c45ba740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x557a18eab250 con 0x557a18f4bd90 2024-03-28T22:11:30.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.075+0000 7ff9bb7fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a18f54970 0x557a18f56d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.075+0000 7ff9bb7fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a18f54970 0x557a18f56d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:50296/0 (socket says 172.21.15.16:50296) 2024-03-28T22:11:30.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.075+0000 7ff9bb7fe640 1 -- 172.21.15.16:0/2824933801 learned_addr learned my addr 172.21.15.16:0/2824933801 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:11:30.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.076+0000 7ff9baffd640 1 --2- 172.21.15.16:0/2824933801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a18f4b470 0x557a18f4b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.076+0000 7ff9bb7fe640 1 -- 172.21.15.16:0/2824933801 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a18f4bd90 msgr2=0x557a18f54430 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T22:11:30.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.076+0000 7ff9bb7fe640 1 --2- 172.21.15.16:0/2824933801 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a18f4bd90 0x557a18f54430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.076+0000 7ff9bb7fe640 1 -- 172.21.15.16:0/2824933801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a18f4b470 msgr2=0x557a18f4b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:30.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.076+0000 7ff9bb7fe640 1 --2- 172.21.15.16:0/2824933801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a18f4b470 0x557a18f4b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.076+0000 7ff9bb7fe640 1 -- 172.21.15.16:0/2824933801 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a18e773d0 con 0x557a18f54970 2024-03-28T22:11:30.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.076+0000 7ff9bb7fe640 1 --2- 172.21.15.16:0/2824933801 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a18f54970 0x557a18f56d60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff9b400af40 tx=0x7ff9b400e7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=256240c3958b6ca8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:11:30.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.076+0000 7ff9b3fff640 1 --2- 172.21.15.16:0/1082086044 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff9a80a2090 0x7ff9a80a4550 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc106eb0 tx=0x7ff9a4005a20 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:11:30.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.076+0000 7ff9b1ffb640 1 -- 172.21.15.16:0/2824933801 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b4003b70 con 0x557a18f54970 2024-03-28T22:11:30.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.076+0000 7ff9b1ffb640 1 -- 172.21.15.16:0/2824933801 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9b4003d10 con 0x557a18f54970 2024-03-28T22:11:30.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.076+0000 7ff9b1ffb640 1 -- 172.21.15.16:0/2824933801 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b4010be0 con 0x557a18f54970 2024-03-28T22:11:30.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.077+0000 7ff9c45ba740 1 -- 172.21.15.16:0/2824933801 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a18f54970 msgr2=0x557a18f56d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:30.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.077+0000 7ff9c45ba740 1 --2- 172.21.15.16:0/2824933801 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a18f54970 0x557a18f56d60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff9b400af40 tx=0x7ff9b400e7e0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.077+0000 7ff9c45ba740 1 -- 172.21.15.16:0/2824933801 shutdown_connections 2024-03-28T22:11:30.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.077+0000 7ff9c45ba740 1 --2- 172.21.15.16:0/2824933801 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a18f54970 0x557a18f56d60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.077+0000 7ff9c45ba740 1 --2- 172.21.15.16:0/2824933801 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a18f4bd90 0x557a18f54430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.077+0000 7ff9c45ba740 1 --2- 172.21.15.16:0/2824933801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a18f4b470 0x557a18f4b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.077+0000 7ff9c45ba740 1 -- 172.21.15.16:0/2824933801 >> 172.21.15.16:0/2824933801 conn(0x557a18f45ba0 msgr2=0x557a18f45f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:11:30.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.077+0000 7ff9c45ba740 1 -- 172.21.15.16:0/2824933801 shutdown_connections 2024-03-28T22:11:30.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.077+0000 7ff9c45ba740 1 -- 172.21.15.16:0/2824933801 wait complete. 2024-03-28T22:11:30.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.078+0000 7ff9c45ba740 1 -- start start 2024-03-28T22:11:30.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.078+0000 7ff9c45ba740 1 Processor -- start 2024-03-28T22:11:30.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.078+0000 7ff9c45ba740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a18f4b470 0x557a18f51950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.078+0000 7ff9c45ba740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a18f4bd90 0x557a18f51e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.078+0000 7ff9c45ba740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a18f54970 0x557a18f523d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.078+0000 7ff9c45ba740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557a18f57760 con 0x557a18f4bd90 2024-03-28T22:11:30.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.078+0000 7ff9c45ba740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x557a18e4d470 con 0x557a18f54970 2024-03-28T22:11:30.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.078+0000 7ff9c45ba740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x557a18f578e0 con 0x557a18f4b470 2024-03-28T22:11:30.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.078+0000 7ff9c45ba740 10 client.0 _ll_register_callbacks cb 0x557a18e6acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:11:30.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.078+0000 7ff9b3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a18f4bd90 0x557a18f51e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.079+0000 7ff9b3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a18f4bd90 0x557a18f51e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57918/0 (socket says 172.21.15.16:57918) 2024-03-28T22:11:30.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.079+0000 7ff9b3fff640 1 -- 172.21.15.16:0/801084931 learned_addr learned my addr 172.21.15.16:0/801084931 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:11:30.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.079+0000 7ff9bb7fe640 1 --2- 172.21.15.16:0/801084931 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a18f54970 0x557a18f523d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.079+0000 7ff9baffd640 1 --2- 172.21.15.16:0/801084931 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a18f4b470 0x557a18f51950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.079+0000 7ff9b3fff640 1 -- 172.21.15.16:0/801084931 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a18f4b470 msgr2=0x557a18f51950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:30.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.079+0000 7ff9b3fff640 1 --2- 172.21.15.16:0/801084931 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a18f4b470 0x557a18f51950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.079+0000 7ff9b3fff640 1 -- 172.21.15.16:0/801084931 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a18f54970 msgr2=0x557a18f523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:30.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.079+0000 7ff9b3fff640 1 --2- 172.21.15.16:0/801084931 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a18f54970 0x557a18f523d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.079+0000 7ff9b3fff640 1 -- 172.21.15.16:0/801084931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a18bc33f0 con 0x557a18f4bd90 2024-03-28T22:11:30.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.079+0000 7ff9b3fff640 1 --2- 172.21.15.16:0/801084931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a18f4bd90 0x557a18f51e90 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4035000 tx=0x7ff9a4009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d40b48ed91980725 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:11:30.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.079+0000 7ff99affd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:11:30.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.079+0000 7ff99affd640 1 -- 172.21.15.16:0/801084931 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a404e070 con 0x557a18f4bd90 2024-03-28T22:11:30.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.079+0000 7ff99affd640 1 -- 172.21.15.16:0/801084931 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7ff9a40046c0 con 0x557a18f4bd90 2024-03-28T22:11:30.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.079+0000 7ff9c45ba740 1 -- 172.21.15.16:0/801084931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9b4003d10 con 0x557a18f4bd90 2024-03-28T22:11:30.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.080+0000 7ff99affd640 1 -- 172.21.15.16:0/801084931 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7ff9a405a050 con 0x557a18f4bd90 2024-03-28T22:11:30.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.080+0000 7ff9c45ba740 10 client.15675 fetch_fsmap learned FSMap version 21 2024-03-28T22:11:30.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.080+0000 7ff9c45ba740 1 -- 172.21.15.16:0/801084931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x557a18c8d7d0 con 0x557a18f4bd90 2024-03-28T22:11:30.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.080+0000 7ff99affd640 1 -- 172.21.15.16:0/801084931 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9a4044070 con 0x557a18f4bd90 2024-03-28T22:11:30.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.080+0000 7ff99affd640 1 -- 172.21.15.16:0/801084931 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7ff9a4049070 con 0x557a18f4bd90 2024-03-28T22:11:30.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.080+0000 7ff9c45ba740 10 client.15675 fetch_fsmap finished waiting for FSMap version 21 2024-03-28T22:11:30.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.080+0000 7ff9c45ba740 10 client.15675 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-28T22:11:30.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.080+0000 7ff9c45ba740 20 client.15675 populate_metadata read hostname 'smithi016' 2024-03-28T22:11:30.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.080+0000 7ff9c45ba740 1 --2- 172.21.15.16:0/801084931 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x557a18f648c0 0x557a18f66cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.080+0000 7ff9c45ba740 4 client.15675 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:11:30.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.080+0000 7ff9c45ba740 1 -- 172.21.15.16:0/801084931 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff9b4003d10 con 0x557a18f648c0 2024-03-28T22:11:30.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.081+0000 7ff9baffd640 1 --2- 172.21.15.16:0/801084931 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x557a18f648c0 0x557a18f66cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.081+0000 7ff99affd640 10 client.15675 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:11:30.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.081+0000 7ff9baffd640 1 --2- 172.21.15.16:0/801084931 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x557a18f648c0 0x557a18f66cf0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4c0ba43d706a46e1 server_cookie=b35574dd1183a406 in_seq=0 out_seq=0 2024-03-28T22:11:30.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.082+0000 7ff99affd640 1 -- 172.21.15.16:0/801084931 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7ff9b4003d10 con 0x557a18f648c0 2024-03-28T22:11:30.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.082+0000 7ff99affd640 10 client.15675 handle_command_reply: tid=0 2024-03-28T22:11:30.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.082+0000 7ff9c45ba740 1 client.15675 shutdown 2024-03-28T22:11:30.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.082+0000 7ff9c45ba740 1 -- 172.21.15.16:0/801084931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a18f4bd90 msgr2=0x557a18f51e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:30.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.082+0000 7ff9c45ba740 1 --2- 172.21.15.16:0/801084931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a18f4bd90 0x557a18f51e90 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4035000 tx=0x7ff9a4009260 comp rx=0 tx=0).stop 2024-03-28T22:11:30.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.082+0000 7ff9c45ba740 1 -- 172.21.15.16:0/801084931 shutdown_connections 2024-03-28T22:11:30.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.082+0000 7ff9c45ba740 1 --2- 172.21.15.16:0/801084931 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x557a18f648c0 0x557a18f66cf0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.082+0000 7ff99affd640 0 client.15675 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:11:30.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.082+0000 7ff9c45ba740 1 --2- 172.21.15.16:0/801084931 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a18f54970 0x557a18f523d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.082+0000 7ff9c45ba740 1 --2- 172.21.15.16:0/801084931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a18f4bd90 0x557a18f51e90 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.082+0000 7ff9c45ba740 1 --2- 172.21.15.16:0/801084931 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a18f4b470 0x557a18f51950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.082+0000 7ff9c45ba740 1 -- 172.21.15.16:0/801084931 >> 172.21.15.16:0/801084931 conn(0x557a18f45ba0 msgr2=0x557a18f478f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:11:30.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.082+0000 7ff9c45ba740 1 -- 172.21.15.16:0/801084931 shutdown_connections 2024-03-28T22:11:30.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.082+0000 7ff9c45ba740 1 -- 172.21.15.16:0/801084931 wait complete. 2024-03-28T22:11:30.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.082+0000 7ff9c45ba740 20 client.15675 trim_cache size 0 max 16384 2024-03-28T22:11:30.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.109+0000 7ff9c45ba740 1 Processor -- start 2024-03-28T22:11:30.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.109+0000 7ff9c45ba740 1 -- start start 2024-03-28T22:11:30.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.109+0000 7ff9c45ba740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a18f4b470 0x557a18f60560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.109+0000 7ff9c45ba740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a18f4bd90 0x557a18f521f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.109+0000 7ff9c45ba740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a18f54970 0x557a18f6c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.109+0000 7ff9c45ba740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff9bc1ada90 con 0x557a18f4b470 2024-03-28T22:11:30.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.109+0000 7ff9c45ba740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff9bc1ab360 con 0x557a18f4bd90 2024-03-28T22:11:30.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.109+0000 7ff9baffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a18f4b470 0x557a18f60560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.109+0000 7ff9c45ba740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff9bc1a91b0 con 0x557a18f54970 2024-03-28T22:11:30.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.109+0000 7ff9b3fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a18f4bd90 0x557a18f521f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.109+0000 7ff9baffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a18f4b470 0x557a18f60560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57924/0 (socket says 172.21.15.16:57924) 2024-03-28T22:11:30.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.109+0000 7ff9baffd640 1 -- 172.21.15.16:0/2275819708 learned_addr learned my addr 172.21.15.16:0/2275819708 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:11:30.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.109+0000 7ff9bb7fe640 1 --2- 172.21.15.16:0/2275819708 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a18f54970 0x557a18f6c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.110+0000 7ff9b3fff640 1 -- 172.21.15.16:0/2275819708 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a18f54970 msgr2=0x557a18f6c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:30.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.110+0000 7ff9b3fff640 1 --2- 172.21.15.16:0/2275819708 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a18f54970 0x557a18f6c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.110+0000 7ff9b3fff640 1 -- 172.21.15.16:0/2275819708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a18f4b470 msgr2=0x557a18f60560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:30.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.110+0000 7ff9b3fff640 1 --2- 172.21.15.16:0/2275819708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a18f4b470 0x557a18f60560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.110+0000 7ff9b3fff640 1 -- 172.21.15.16:0/2275819708 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a18f4bbc0 con 0x557a18f4bd90 2024-03-28T22:11:30.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.110+0000 7ff9baffd640 1 --2- 172.21.15.16:0/2275819708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a18f4b470 0x557a18f60560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:11:30.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.110+0000 7ff9b3fff640 1 --2- 172.21.15.16:0/2275819708 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a18f4bd90 0x557a18f521f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4003f30 tx=0x7ff9a402bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f0c515be73352bf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:11:30.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.110+0000 7ff9b17fa640 1 -- 172.21.15.16:0/2275819708 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a4057020 con 0x557a18f4bd90 2024-03-28T22:11:30.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.110+0000 7ff9b17fa640 1 -- 172.21.15.16:0/2275819708 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9a4044040 con 0x557a18f4bd90 2024-03-28T22:11:30.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.110+0000 7ff9b17fa640 1 -- 172.21.15.16:0/2275819708 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a404e070 con 0x557a18f4bd90 2024-03-28T22:11:30.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.111+0000 7ff9c45ba740 1 -- 172.21.15.16:0/2275819708 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a18f4bd90 msgr2=0x557a18f521f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:30.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.111+0000 7ff9c45ba740 1 --2- 172.21.15.16:0/2275819708 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a18f4bd90 0x557a18f521f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4003f30 tx=0x7ff9a402bc80 comp rx=0 tx=0).stop 2024-03-28T22:11:30.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.111+0000 7ff9c45ba740 1 -- 172.21.15.16:0/2275819708 shutdown_connections 2024-03-28T22:11:30.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.111+0000 7ff9c45ba740 1 --2- 172.21.15.16:0/2275819708 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a18f54970 0x557a18f6c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.111+0000 7ff9c45ba740 1 --2- 172.21.15.16:0/2275819708 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a18f4bd90 0x557a18f521f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.111+0000 7ff9c45ba740 1 --2- 172.21.15.16:0/2275819708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a18f4b470 0x557a18f60560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.111+0000 7ff9c45ba740 1 -- 172.21.15.16:0/2275819708 >> 172.21.15.16:0/2275819708 conn(0x557a18f45ba0 msgr2=0x557a18f44d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:11:30.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.111+0000 7ff9c45ba740 1 -- 172.21.15.16:0/2275819708 shutdown_connections 2024-03-28T22:11:30.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.111+0000 7ff9c45ba740 1 -- 172.21.15.16:0/2275819708 wait complete. 2024-03-28T22:11:30.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.112+0000 7ff9c45ba740 1 -- start start 2024-03-28T22:11:30.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.112+0000 7ff9c45ba740 1 Processor -- start 2024-03-28T22:11:30.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.112+0000 7ff9c45ba740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a18f4b470 0x557a18f4ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.112+0000 7ff9c45ba740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a18f4bd90 0x557a18f4f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.112+0000 7ff9c45ba740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a18f54970 0x557a18fc9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.112+0000 7ff9baffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a18f4b470 0x557a18f4ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.112+0000 7ff9baffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a18f4b470 0x557a18f4ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:50320/0 (socket says 172.21.15.16:50320) 2024-03-28T22:11:30.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.113+0000 7ff9b3fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a18f4bd90 0x557a18f4f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.113+0000 7ff9b3fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a18f4bd90 0x557a18f4f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.16:54728/0 (socket says 172.21.15.16:54728) 2024-03-28T22:11:30.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.113+0000 7ff9bb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a18f54970 0x557a18fc9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.113+0000 7ff9bb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a18f54970 0x557a18fc9bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57928/0 (socket says 172.21.15.16:57928) 2024-03-28T22:11:30.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.113+0000 7ff9baffd640 1 -- 172.21.15.16:0/831219218 learned_addr learned my addr 172.21.15.16:0/831219218 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:11:30.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.113+0000 7ff9c45ba740 1 -- 172.21.15.16:0/831219218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff9bc1a9820 con 0x557a18f54970 2024-03-28T22:11:30.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.113+0000 7ff9c45ba740 1 -- 172.21.15.16:0/831219218 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff9bc1aac90 con 0x557a18f4b470 2024-03-28T22:11:30.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.113+0000 7ff9c45ba740 1 -- 172.21.15.16:0/831219218 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff9bc1a9ef0 con 0x557a18f4bd90 2024-03-28T22:11:30.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.113+0000 7ff9baffd640 1 -- 172.21.15.16:0/831219218 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a18f4bd90 msgr2=0x557a18f4f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:30.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.113+0000 7ff9baffd640 1 --2- 172.21.15.16:0/831219218 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a18f4bd90 0x557a18f4f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.113+0000 7ff9baffd640 1 -- 172.21.15.16:0/831219218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a18f54970 msgr2=0x557a18fc9bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:30.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.113+0000 7ff9baffd640 1 --2- 172.21.15.16:0/831219218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a18f54970 0x557a18fc9bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.113+0000 7ff9c45ba740 10 client.0 _ll_register_callbacks cb 0x7ff9a40045d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:11:30.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.113+0000 7ff9baffd640 1 -- 172.21.15.16:0/831219218 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a18fca1e0 con 0x557a18f4b470 2024-03-28T22:11:30.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.113+0000 7ff9bb7fe640 1 --2- 172.21.15.16:0/831219218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a18f54970 0x557a18fc9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:11:30.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.113+0000 7ff9c45ba740 1 -- 172.21.15.16:0/831219218 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9a4044040 con 0x557a18f4b470 2024-03-28T22:11:30.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.113+0000 7ff9baffd640 1 --2- 172.21.15.16:0/831219218 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a18f4b470 0x557a18f4ecf0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac0a3290 tx=0x7ff9ac0973f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b2271a3b81d8bf5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:11:30.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.113+0000 7ff99affd640 10 client.25583 ms_handle_connect on v2:172.21.15.53:3300/0 2024-03-28T22:11:30.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.113+0000 7ff99affd640 1 -- 172.21.15.16:0/831219218 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ac07e070 con 0x557a18f4b470 2024-03-28T22:11:30.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.114+0000 7ff99affd640 1 -- 172.21.15.16:0/831219218 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7ff9ac003b30 con 0x557a18f4b470 2024-03-28T22:11:30.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.114+0000 7ff99affd640 1 -- 172.21.15.16:0/831219218 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7ff9ac097690 con 0x557a18f4b470 2024-03-28T22:11:30.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.114+0000 7ff9c45ba740 10 client.25583 fetch_fsmap learned FSMap version 21 2024-03-28T22:11:30.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.114+0000 7ff9c45ba740 1 -- 172.21.15.16:0/831219218 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({fsmap=21}) -- 0x557a18fcb360 con 0x557a18f4b470 2024-03-28T22:11:30.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.114+0000 7ff99affd640 1 -- 172.21.15.16:0/831219218 <== mon.2 v2:172.21.15.53:3300/0 4 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7ff9ac0b92c0 con 0x557a18f4b470 2024-03-28T22:11:30.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.114+0000 7ff9c45ba740 10 client.25583 fetch_fsmap finished waiting for FSMap version 21 2024-03-28T22:11:30.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.114+0000 7ff9c45ba740 10 client.25583 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-28T22:11:30.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.114+0000 7ff9c45ba740 20 client.25583 populate_metadata read hostname 'smithi016' 2024-03-28T22:11:30.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.114+0000 7ff9c45ba740 1 --2- 172.21.15.16:0/831219218 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x557a18fcba70 0x557a18fcde90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.114+0000 7ff9c45ba740 4 client.25583 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:11:30.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.114+0000 7ff9c45ba740 1 -- 172.21.15.16:0/831219218 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557a18ebc6f0 con 0x557a18fcba70 2024-03-28T22:11:30.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.114+0000 7ff9b3fff640 1 --2- 172.21.15.16:0/831219218 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x557a18fcba70 0x557a18fcde90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.115+0000 7ff99affd640 10 client.25583 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:11:30.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.115+0000 7ff9b3fff640 1 --2- 172.21.15.16:0/831219218 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x557a18fcba70 0x557a18fcde90 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9571aefe5f6b35e7 server_cookie=224facb9b06b809d in_seq=0 out_seq=0 2024-03-28T22:11:30.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.115+0000 7ff99affd640 1 -- 172.21.15.16:0/831219218 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557a18ebc6f0 con 0x557a18fcba70 2024-03-28T22:11:30.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.115+0000 7ff99affd640 10 client.25583 handle_command_reply: tid=0 2024-03-28T22:11:30.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.115+0000 7ff9c45ba740 1 client.25583 shutdown 2024-03-28T22:11:30.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.115+0000 7ff9c45ba740 1 -- 172.21.15.16:0/831219218 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a18f4b470 msgr2=0x557a18f4ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:30.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.115+0000 7ff9c45ba740 1 --2- 172.21.15.16:0/831219218 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a18f4b470 0x557a18f4ecf0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac0a3290 tx=0x7ff9ac0973f0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.115+0000 7ff9c45ba740 1 -- 172.21.15.16:0/831219218 shutdown_connections 2024-03-28T22:11:30.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.115+0000 7ff9c45ba740 1 --2- 172.21.15.16:0/831219218 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x557a18fcba70 0x557a18fcde90 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.115+0000 7ff9c45ba740 1 --2- 172.21.15.16:0/831219218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a18f54970 0x557a18fc9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.115+0000 7ff9c45ba740 1 --2- 172.21.15.16:0/831219218 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a18f4bd90 0x557a18f4f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.115+0000 7ff99affd640 0 client.25583 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:11:30.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.115+0000 7ff9c45ba740 1 --2- 172.21.15.16:0/831219218 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557a18f4b470 0x557a18f4ecf0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.115+0000 7ff9c45ba740 1 -- 172.21.15.16:0/831219218 >> 172.21.15.16:0/831219218 conn(0x557a18f45ba0 msgr2=0x557a18f46f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:11:30.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.115+0000 7ff9c45ba740 1 -- 172.21.15.16:0/831219218 shutdown_connections 2024-03-28T22:11:30.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.115+0000 7ff9c45ba740 1 -- 172.21.15.16:0/831219218 wait complete. 2024-03-28T22:11:30.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.115+0000 7ff9c45ba740 20 client.25583 trim_cache size 0 max 16384 2024-03-28T22:11:30.124 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:11:30.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.116+0000 7ff9c19d0640 1 -- 172.21.15.16:0/1082086044 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff9a80a2090 msgr2=0x7ff9a80a4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:30.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.116+0000 7ff9c19d0640 1 --2- 172.21.15.16:0/1082086044 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff9a80a2090 0x7ff9a80a4550 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc106eb0 tx=0x7ff9a4005a20 comp rx=0 tx=0).stop 2024-03-28T22:11:30.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.116+0000 7ff9c19d0640 1 -- 172.21.15.16:0/1082086044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9bc0fffd0 msgr2=0x7ff9bc198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:30.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.116+0000 7ff9c19d0640 1 --2- 172.21.15.16:0/1082086044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9bc0fffd0 0x7ff9bc198000 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac00e9c0 tx=0x7ff9ac00ee80 comp rx=0 tx=0).stop 2024-03-28T22:11:30.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.116+0000 7ff9c19d0640 1 -- 172.21.15.16:0/1082086044 shutdown_connections 2024-03-28T22:11:30.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.116+0000 7ff9c19d0640 1 --2- 172.21.15.16:0/1082086044 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff9a80a2090 0x7ff9a80a4550 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.116+0000 7ff9c19d0640 1 --2- 172.21.15.16:0/1082086044 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9bc107860 0x7ff9bc19c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.116+0000 7ff9c19d0640 1 --2- 172.21.15.16:0/1082086044 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9bc100990 0x7ff9bc198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.116+0000 7ff9c19d0640 1 --2- 172.21.15.16:0/1082086044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9bc0fffd0 0x7ff9bc198000 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.116+0000 7ff9c19d0640 1 -- 172.21.15.16:0/1082086044 >> 172.21.15.16:0/1082086044 conn(0x7ff9bc0758b0 msgr2=0x7ff9bc0febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:11:30.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.116+0000 7ff9c19d0640 1 -- 172.21.15.16:0/1082086044 shutdown_connections 2024-03-28T22:11:30.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.116+0000 7ff9c19d0640 1 -- 172.21.15.16:0/1082086044 wait complete. 2024-03-28T22:11:30.133 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:11:30.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.237+0000 7f6b6dbd0640 1 -- 172.21.15.16:0/2738011522 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b68103290 msgr2=0x7f6b6810b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:30.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.238+0000 7f6b6dbd0640 1 --2- 172.21.15.16:0/2738011522 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b68103290 0x7f6b6810b770 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c009900 tx=0x7f6b5c02bcd0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.238+0000 7f6b6dbd0640 1 -- 172.21.15.16:0/2738011522 shutdown_connections 2024-03-28T22:11:30.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.238+0000 7f6b6dbd0640 1 --2- 172.21.15.16:0/2738011522 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6b6810bd40 0x7f6b6810e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.238+0000 7f6b6dbd0640 1 --2- 172.21.15.16:0/2738011522 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b68103290 0x7f6b6810b770 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.238+0000 7f6b6dbd0640 1 --2- 172.21.15.16:0/2738011522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b68102970 0x7f6b68102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.238+0000 7f6b6dbd0640 1 -- 172.21.15.16:0/2738011522 >> 172.21.15.16:0/2738011522 conn(0x7f6b68077800 msgr2=0x7f6b68077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:11:30.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.238+0000 7f6b6dbd0640 1 -- 172.21.15.16:0/2738011522 shutdown_connections 2024-03-28T22:11:30.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.238+0000 7f6b6dbd0640 1 -- 172.21.15.16:0/2738011522 wait complete. 2024-03-28T22:11:30.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.238+0000 7f6b6dbd0640 1 Processor -- start 2024-03-28T22:11:30.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.238+0000 7f6b6dbd0640 1 -- start start 2024-03-28T22:11:30.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.239+0000 7f6b6dbd0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6b68102970 0x7f6b681005c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.239+0000 7f6b6dbd0640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b68103290 0x7f6b68100b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.239+0000 7f6b6dbd0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b6810bd40 0x7f6b68101040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.239+0000 7f6b6dbd0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6b68111ad0 con 0x7f6b6810bd40 2024-03-28T22:11:30.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.239+0000 7f6b6dbd0640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f6b68111950 con 0x7f6b68103290 2024-03-28T22:11:30.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.239+0000 7f6b6dbd0640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6b68111c50 con 0x7f6b68102970 2024-03-28T22:11:30.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.239+0000 7f6b66ffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b68103290 0x7f6b68100b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.239+0000 7f6b66ffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b68103290 0x7f6b68100b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:50332/0 (socket says 172.21.15.16:50332) 2024-03-28T22:11:30.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.239+0000 7f6b66ffd640 1 -- 172.21.15.16:0/3599287394 learned_addr learned my addr 172.21.15.16:0/3599287394 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:11:30.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.239+0000 7f6b67fff640 1 --2- 172.21.15.16:0/3599287394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b6810bd40 0x7f6b68101040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.239+0000 7f6b677fe640 1 --2- 172.21.15.16:0/3599287394 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6b68102970 0x7f6b681005c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.239+0000 7f6b66ffd640 1 -- 172.21.15.16:0/3599287394 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6b68102970 msgr2=0x7f6b681005c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:30.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.239+0000 7f6b66ffd640 1 --2- 172.21.15.16:0/3599287394 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6b68102970 0x7f6b681005c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.239+0000 7f6b66ffd640 1 -- 172.21.15.16:0/3599287394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b6810bd40 msgr2=0x7f6b68101040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:30.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.239+0000 7f6b66ffd640 1 --2- 172.21.15.16:0/3599287394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b6810bd40 0x7f6b68101040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.239+0000 7f6b66ffd640 1 -- 172.21.15.16:0/3599287394 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b681a64f0 con 0x7f6b68103290 2024-03-28T22:11:30.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.240+0000 7f6b66ffd640 1 --2- 172.21.15.16:0/3599287394 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b68103290 0x7f6b68100b00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c005940 tx=0x7f6b5c02cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:11:30.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.240+0000 7f6b64ff9640 1 -- 172.21.15.16:0/3599287394 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b5c041070 con 0x7f6b68103290 2024-03-28T22:11:30.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.240+0000 7f6b64ff9640 1 -- 172.21.15.16:0/3599287394 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b5c004450 con 0x7f6b68103290 2024-03-28T22:11:30.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.241+0000 7f6b64ff9640 1 -- 172.21.15.16:0/3599287394 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b5c003df0 con 0x7f6b68103290 2024-03-28T22:11:30.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.241+0000 7f6b6dbd0640 1 -- 172.21.15.16:0/3599287394 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b681a6780 con 0x7f6b68103290 2024-03-28T22:11:30.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.241+0000 7f6b6dbd0640 1 -- 172.21.15.16:0/3599287394 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b681a6cc0 con 0x7f6b68103290 2024-03-28T22:11:30.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.243+0000 7f6b64ff9640 1 -- 172.21.15.16:0/3599287394 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 25) ==== 136989+0+0 (secure 0 0 0) 0x7f6b5c044070 con 0x7f6b68103290 2024-03-28T22:11:30.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.243+0000 7f6b64ff9640 1 --2- 172.21.15.16:0/3599287394 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f6b500a2090 0x7f6b500a4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.243+0000 7f6b64ff9640 1 -- 172.21.15.16:0/3599287394 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f6b5c0ef8e0 con 0x7f6b68103290 2024-03-28T22:11:30.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.243+0000 7f6b677fe640 1 --2- 172.21.15.16:0/3599287394 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f6b500a2090 0x7f6b500a4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.244+0000 7f6b677fe640 1 --2- 172.21.15.16:0/3599287394 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f6b500a2090 0x7f6b500a4550 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6b58009bd0 tx=0x7f6b58005f90 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:11:30.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.244+0000 7f6b707bb740 1 Processor -- start 2024-03-28T22:11:30.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.244+0000 7f6b707bb740 1 -- start start 2024-03-28T22:11:30.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.244+0000 7f6b707bb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a16ff470 0x55a5a16ff850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.244+0000 7f6b707bb740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5a16ffd90 0x55a5a1708430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.244+0000 7f6b707bb740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5a1708970 0x55a5a170ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.244+0000 7f6b707bb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a5a16631b0 con 0x55a5a16ff470 2024-03-28T22:11:30.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.244+0000 7f6b707bb740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55a5a160bc00 con 0x55a5a1708970 2024-03-28T22:11:30.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.244+0000 7f6b707bb740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55a5a165f250 con 0x55a5a16ffd90 2024-03-28T22:11:30.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.244+0000 7f6b677fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a16ff470 0x55a5a16ff850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.244+0000 7f6b677fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a16ff470 0x55a5a16ff850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57958/0 (socket says 172.21.15.16:57958) 2024-03-28T22:11:30.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.244+0000 7f6b677fe640 1 -- 172.21.15.16:0/695070213 learned_addr learned my addr 172.21.15.16:0/695070213 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:11:30.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.245+0000 7f6b67fff640 1 --2- 172.21.15.16:0/695070213 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5a1708970 0x55a5a170ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.245+0000 7f6b66ffd640 1 --2- 172.21.15.16:0/695070213 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5a16ffd90 0x55a5a1708430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.245+0000 7f6b677fe640 1 -- 172.21.15.16:0/695070213 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5a16ffd90 msgr2=0x55a5a1708430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:30.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.245+0000 7f6b677fe640 1 --2- 172.21.15.16:0/695070213 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5a16ffd90 0x55a5a1708430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.245+0000 7f6b677fe640 1 -- 172.21.15.16:0/695070213 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5a1708970 msgr2=0x55a5a170ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:30.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.245+0000 7f6b677fe640 1 --2- 172.21.15.16:0/695070213 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5a1708970 0x55a5a170ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.245+0000 7f6b677fe640 1 -- 172.21.15.16:0/695070213 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5a162b3d0 con 0x55a5a16ff470 2024-03-28T22:11:30.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.245+0000 7f6b67fff640 1 --2- 172.21.15.16:0/695070213 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5a1708970 0x55a5a170ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:11:30.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.245+0000 7f6b677fe640 1 --2- 172.21.15.16:0/695070213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a16ff470 0x55a5a16ff850 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f6b5800e630 tx=0x7f6b58014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7e58d3ddfa4fbde server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:11:30.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.245+0000 7f6b4dffb640 1 -- 172.21.15.16:0/695070213 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b58004020 con 0x55a5a16ff470 2024-03-28T22:11:30.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.245+0000 7f6b4dffb640 1 -- 172.21.15.16:0/695070213 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b580041c0 con 0x55a5a16ff470 2024-03-28T22:11:30.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.245+0000 7f6b4dffb640 1 -- 172.21.15.16:0/695070213 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b5801c7a0 con 0x55a5a16ff470 2024-03-28T22:11:30.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.246+0000 7f6b707bb740 1 -- 172.21.15.16:0/695070213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a16ff470 msgr2=0x55a5a16ff850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:30.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.246+0000 7f6b707bb740 1 --2- 172.21.15.16:0/695070213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a16ff470 0x55a5a16ff850 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f6b5800e630 tx=0x7f6b58014000 comp rx=0 tx=0).stop 2024-03-28T22:11:30.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.246+0000 7f6b707bb740 1 -- 172.21.15.16:0/695070213 shutdown_connections 2024-03-28T22:11:30.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.246+0000 7f6b707bb740 1 --2- 172.21.15.16:0/695070213 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5a1708970 0x55a5a170ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.246+0000 7f6b707bb740 1 --2- 172.21.15.16:0/695070213 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5a16ffd90 0x55a5a1708430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.246+0000 7f6b707bb740 1 --2- 172.21.15.16:0/695070213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a16ff470 0x55a5a16ff850 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.246+0000 7f6b707bb740 1 -- 172.21.15.16:0/695070213 >> 172.21.15.16:0/695070213 conn(0x55a5a16f9ba0 msgr2=0x55a5a16f9f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:11:30.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.246+0000 7f6b707bb740 1 -- 172.21.15.16:0/695070213 shutdown_connections 2024-03-28T22:11:30.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.246+0000 7f6b707bb740 1 -- 172.21.15.16:0/695070213 wait complete. 2024-03-28T22:11:30.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.246+0000 7f6b707bb740 1 -- start start 2024-03-28T22:11:30.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.247+0000 7f6b707bb740 1 Processor -- start 2024-03-28T22:11:30.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.247+0000 7f6b707bb740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5a16ff470 0x55a5a1705a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.247+0000 7f6b707bb740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5a16ffd90 0x55a5a1705fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.247+0000 7f6b707bb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a1708970 0x55a5a17064f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.247+0000 7f6b707bb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a5a170b760 con 0x55a5a1708970 2024-03-28T22:11:30.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.247+0000 7f6b707bb740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55a5a1601470 con 0x55a5a16ff470 2024-03-28T22:11:30.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.247+0000 7f6b707bb740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55a5a170b8e0 con 0x55a5a16ffd90 2024-03-28T22:11:30.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.247+0000 7f6b707bb740 10 client.0 _ll_register_callbacks cb 0x55a5a161ecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:11:30.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.247+0000 7f6b67fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a1708970 0x55a5a17064f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.247+0000 7f6b677fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5a16ff470 0x55a5a1705a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.247+0000 7f6b677fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5a16ff470 0x55a5a1705a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:50338/0 (socket says 172.21.15.16:50338) 2024-03-28T22:11:30.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.247+0000 7f6b67fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a1708970 0x55a5a17064f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57960/0 (socket says 172.21.15.16:57960) 2024-03-28T22:11:30.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.247+0000 7f6b677fe640 1 -- 172.21.15.16:0/1997969032 learned_addr learned my addr 172.21.15.16:0/1997969032 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:11:30.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.247+0000 7f6b66ffd640 1 --2- 172.21.15.16:0/1997969032 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5a16ffd90 0x55a5a1705fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.247+0000 7f6b677fe640 1 -- 172.21.15.16:0/1997969032 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5a16ffd90 msgr2=0x55a5a1705fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:30.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.247+0000 7f6b677fe640 1 --2- 172.21.15.16:0/1997969032 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5a16ffd90 0x55a5a1705fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.247+0000 7f6b677fe640 1 -- 172.21.15.16:0/1997969032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a1708970 msgr2=0x55a5a17064f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:30.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.247+0000 7f6b677fe640 1 --2- 172.21.15.16:0/1997969032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a1708970 0x55a5a17064f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.247+0000 7f6b677fe640 1 -- 172.21.15.16:0/1997969032 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5a13773f0 con 0x55a5a16ff470 2024-03-28T22:11:30.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.247+0000 7f6b67fff640 1 --2- 172.21.15.16:0/1997969032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a1708970 0x55a5a17064f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:11:30.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.248+0000 7f6b677fe640 1 --2- 172.21.15.16:0/1997969032 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5a16ff470 0x55a5a1705a70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6b58009250 tx=0x7f6b58004020 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38793104571e3607 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:11:30.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.248+0000 7f6b46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-03-28T22:11:30.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.248+0000 7f6b46ffd640 1 -- 172.21.15.16:0/1997969032 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b58034060 con 0x55a5a16ff470 2024-03-28T22:11:30.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.248+0000 7f6b46ffd640 1 -- 172.21.15.16:0/1997969032 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f6b580035f0 con 0x55a5a16ff470 2024-03-28T22:11:30.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.248+0000 7f6b707bb740 1 -- 172.21.15.16:0/1997969032 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b580041c0 con 0x55a5a16ff470 2024-03-28T22:11:30.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.248+0000 7f6b46ffd640 1 -- 172.21.15.16:0/1997969032 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f6b580037f0 con 0x55a5a16ff470 2024-03-28T22:11:30.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.248+0000 7f6b707bb740 10 client.25595 fetch_fsmap learned FSMap version 21 2024-03-28T22:11:30.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.248+0000 7f6b707bb740 1 -- 172.21.15.16:0/1997969032 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55a5a14417d0 con 0x55a5a16ff470 2024-03-28T22:11:30.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.248+0000 7f6b46ffd640 1 -- 172.21.15.16:0/1997969032 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b580041c0 con 0x55a5a16ff470 2024-03-28T22:11:30.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.248+0000 7f6b46ffd640 1 -- 172.21.15.16:0/1997969032 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f6b5802e030 con 0x55a5a16ff470 2024-03-28T22:11:30.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.249+0000 7f6b707bb740 10 client.25595 fetch_fsmap finished waiting for FSMap version 21 2024-03-28T22:11:30.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.249+0000 7f6b707bb740 10 client.25595 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T22:11:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.249+0000 7f6b707bb740 20 client.25595 populate_metadata read hostname 'smithi016' 2024-03-28T22:11:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.249+0000 7f6b707bb740 1 --2- 172.21.15.16:0/1997969032 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55a5a1718920 0x55a5a171ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.249+0000 7f6b707bb740 4 client.25595 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:11:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.249+0000 7f6b707bb740 1 -- 172.21.15.16:0/1997969032 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a5a16706f0 con 0x55a5a1718920 2024-03-28T22:11:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.249+0000 7f6b66ffd640 1 --2- 172.21.15.16:0/1997969032 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55a5a1718920 0x55a5a171ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.249+0000 7f6b66ffd640 1 --2- 172.21.15.16:0/1997969032 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55a5a1718920 0x55a5a171ad10 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=aacd152851205f3 server_cookie=33e7b9cd95109671 in_seq=0 out_seq=0 2024-03-28T22:11:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.249+0000 7f6b46ffd640 10 client.25595 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:11:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.250+0000 7f6b46ffd640 1 -- 172.21.15.16:0/1997969032 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a5a16706f0 con 0x55a5a1718920 2024-03-28T22:11:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.250+0000 7f6b46ffd640 10 client.25595 handle_command_reply: tid=0 2024-03-28T22:11:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.250+0000 7f6b707bb740 1 client.25595 shutdown 2024-03-28T22:11:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.250+0000 7f6b707bb740 1 -- 172.21.15.16:0/1997969032 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5a16ff470 msgr2=0x55a5a1705a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.250+0000 7f6b707bb740 1 --2- 172.21.15.16:0/1997969032 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5a16ff470 0x55a5a1705a70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6b58009250 tx=0x7f6b58004020 comp rx=0 tx=0).stop 2024-03-28T22:11:30.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.250+0000 7f6b707bb740 1 -- 172.21.15.16:0/1997969032 shutdown_connections 2024-03-28T22:11:30.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.250+0000 7f6b707bb740 1 --2- 172.21.15.16:0/1997969032 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55a5a1718920 0x55a5a171ad10 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.250+0000 7f6b707bb740 1 --2- 172.21.15.16:0/1997969032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a1708970 0x55a5a17064f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.250+0000 7f6b46ffd640 0 client.25595 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:11:30.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.250+0000 7f6b707bb740 1 --2- 172.21.15.16:0/1997969032 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5a16ffd90 0x55a5a1705fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.250+0000 7f6b707bb740 1 --2- 172.21.15.16:0/1997969032 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5a16ff470 0x55a5a1705a70 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.250+0000 7f6b707bb740 1 -- 172.21.15.16:0/1997969032 >> 172.21.15.16:0/1997969032 conn(0x55a5a16f9ba0 msgr2=0x55a5a16fb8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:11:30.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.251+0000 7f6b707bb740 1 -- 172.21.15.16:0/1997969032 shutdown_connections 2024-03-28T22:11:30.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.251+0000 7f6b707bb740 1 -- 172.21.15.16:0/1997969032 wait complete. 2024-03-28T22:11:30.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.251+0000 7f6b707bb740 20 client.25595 trim_cache size 0 max 16384 2024-03-28T22:11:30.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.266+0000 7f6b707bb740 1 Processor -- start 2024-03-28T22:11:30.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.266+0000 7f6b707bb740 1 -- start start 2024-03-28T22:11:30.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.267+0000 7f6b707bb740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5a16ff470 0x55a5a1708130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.267+0000 7f6b707bb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a16ffd90 0x55a5a1705350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.267+0000 7f6b707bb740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5a1708970 0x55a5a1714500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.267+0000 7f6b707bb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6b681b1e80 con 0x55a5a16ffd90 2024-03-28T22:11:30.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.267+0000 7f6b707bb740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f6b681af750 con 0x55a5a16ff470 2024-03-28T22:11:30.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.267+0000 7f6b707bb740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6b681ad5a0 con 0x55a5a1708970 2024-03-28T22:11:30.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.267+0000 7f6b677fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5a16ff470 0x55a5a1708130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.267+0000 7f6b66ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a16ffd90 0x55a5a1705350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.267+0000 7f6b677fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5a16ff470 0x55a5a1708130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:50344/0 (socket says 172.21.15.16:50344) 2024-03-28T22:11:30.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.267+0000 7f6b677fe640 1 -- 172.21.15.16:0/4017558372 learned_addr learned my addr 172.21.15.16:0/4017558372 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:11:30.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.267+0000 7f6b66ffd640 1 -- 172.21.15.16:0/4017558372 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5a1708970 msgr2=0x55a5a1714500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:30.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.267+0000 7f6b66ffd640 1 --2- 172.21.15.16:0/4017558372 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5a1708970 0x55a5a1714500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.267+0000 7f6b66ffd640 1 -- 172.21.15.16:0/4017558372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5a16ff470 msgr2=0x55a5a1708130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:30.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.267+0000 7f6b66ffd640 1 --2- 172.21.15.16:0/4017558372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5a16ff470 0x55a5a1708130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.267+0000 7f6b66ffd640 1 -- 172.21.15.16:0/4017558372 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5a170b0d0 con 0x55a5a16ffd90 2024-03-28T22:11:30.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.267+0000 7f6b677fe640 1 --2- 172.21.15.16:0/4017558372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5a16ff470 0x55a5a1708130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:11:30.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.267+0000 7f6b66ffd640 1 --2- 172.21.15.16:0/4017558372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a16ffd90 0x55a5a1705350 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c0ecfe0 tx=0x7f6b5c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f8c96f74fe6151f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:11:30.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.268+0000 7f6b4d7fa640 1 -- 172.21.15.16:0/4017558372 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b5c0bc8b0 con 0x55a5a16ffd90 2024-03-28T22:11:30.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.268+0000 7f6b4d7fa640 1 -- 172.21.15.16:0/4017558372 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b5c0bc1b0 con 0x55a5a16ffd90 2024-03-28T22:11:30.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.268+0000 7f6b4d7fa640 1 -- 172.21.15.16:0/4017558372 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b5c0b8820 con 0x55a5a16ffd90 2024-03-28T22:11:30.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.268+0000 7f6b707bb740 1 -- 172.21.15.16:0/4017558372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a16ffd90 msgr2=0x55a5a1705350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:30.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.268+0000 7f6b707bb740 1 --2- 172.21.15.16:0/4017558372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a16ffd90 0x55a5a1705350 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c0ecfe0 tx=0x7f6b5c000bc0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.268+0000 7f6b707bb740 1 -- 172.21.15.16:0/4017558372 shutdown_connections 2024-03-28T22:11:30.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.268+0000 7f6b707bb740 1 --2- 172.21.15.16:0/4017558372 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5a1708970 0x55a5a1714500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.268+0000 7f6b707bb740 1 --2- 172.21.15.16:0/4017558372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a16ffd90 0x55a5a1705350 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.268+0000 7f6b707bb740 1 --2- 172.21.15.16:0/4017558372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5a16ff470 0x55a5a1708130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.268+0000 7f6b707bb740 1 -- 172.21.15.16:0/4017558372 >> 172.21.15.16:0/4017558372 conn(0x55a5a16f9ba0 msgr2=0x55a5a16f8d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:11:30.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.269+0000 7f6b707bb740 1 -- 172.21.15.16:0/4017558372 shutdown_connections 2024-03-28T22:11:30.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.269+0000 7f6b707bb740 1 -- 172.21.15.16:0/4017558372 wait complete. 2024-03-28T22:11:30.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.269+0000 7f6b707bb740 1 -- start start 2024-03-28T22:11:30.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.269+0000 7f6b707bb740 1 Processor -- start 2024-03-28T22:11:30.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.269+0000 7f6b707bb740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5a16ff470 0x55a5a1710e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.270+0000 7f6b707bb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a16ffd90 0x55a5a170c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.270+0000 7f6b707bb740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5a1708970 0x55a5a170c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.270+0000 7f6b707bb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6b681adc10 con 0x55a5a16ffd90 2024-03-28T22:11:30.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.270+0000 7f6b707bb740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f6b681af080 con 0x55a5a1708970 2024-03-28T22:11:30.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.270+0000 7f6b707bb740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6b681ae2e0 con 0x55a5a16ff470 2024-03-28T22:11:30.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.270+0000 7f6b707bb740 10 client.0 _ll_register_callbacks cb 0x7f6b58003cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:11:30.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.270+0000 7f6b66ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a16ffd90 0x55a5a170c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.270+0000 7f6b66ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a16ffd90 0x55a5a170c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57974/0 (socket says 172.21.15.16:57974) 2024-03-28T22:11:30.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.270+0000 7f6b66ffd640 1 -- 172.21.15.16:0/3153142240 learned_addr learned my addr 172.21.15.16:0/3153142240 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:11:30.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.270+0000 7f6b67fff640 1 --2- 172.21.15.16:0/3153142240 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5a1708970 0x55a5a170c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.270+0000 7f6b677fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5a16ff470 0x55a5a1710e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.270+0000 7f6b66ffd640 1 -- 172.21.15.16:0/3153142240 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5a16ff470 msgr2=0x55a5a1710e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:30.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.270+0000 7f6b66ffd640 1 --2- 172.21.15.16:0/3153142240 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5a16ff470 0x55a5a1710e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.270+0000 7f6b66ffd640 1 -- 172.21.15.16:0/3153142240 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5a1708970 msgr2=0x55a5a170c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:30.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.270+0000 7f6b66ffd640 1 --2- 172.21.15.16:0/3153142240 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5a1708970 0x55a5a170c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.270+0000 7f6b66ffd640 1 -- 172.21.15.16:0/3153142240 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5a170ceb0 con 0x55a5a16ffd90 2024-03-28T22:11:30.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.270+0000 7f6b66ffd640 1 --2- 172.21.15.16:0/3153142240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a16ffd90 0x55a5a170c170 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c0c4420 tx=0x7f6b5c0ba690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c306cededbeabf5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:11:30.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.270+0000 7f6b46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:11:30.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.271+0000 7f6b46ffd640 1 -- 172.21.15.16:0/3153142240 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b5c0c4950 con 0x55a5a16ffd90 2024-03-28T22:11:30.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.271+0000 7f6b46ffd640 1 -- 172.21.15.16:0/3153142240 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f6b5c0c44c0 con 0x55a5a16ffd90 2024-03-28T22:11:30.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.271+0000 7f6b707bb740 1 -- 172.21.15.16:0/3153142240 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b5c0bc1b0 con 0x55a5a16ffd90 2024-03-28T22:11:30.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.271+0000 7f6b46ffd640 1 -- 172.21.15.16:0/3153142240 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f6b5c0bc8b0 con 0x55a5a16ffd90 2024-03-28T22:11:30.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.271+0000 7f6b707bb740 10 client.15699 fetch_fsmap learned FSMap version 21 2024-03-28T22:11:30.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.271+0000 7f6b707bb740 1 -- 172.21.15.16:0/3153142240 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55a5a177e900 con 0x55a5a16ffd90 2024-03-28T22:11:30.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.271+0000 7f6b46ffd640 1 -- 172.21.15.16:0/3153142240 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b5c0bc1b0 con 0x55a5a16ffd90 2024-03-28T22:11:30.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.271+0000 7f6b46ffd640 1 -- 172.21.15.16:0/3153142240 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f6b5c0ed240 con 0x55a5a16ffd90 2024-03-28T22:11:30.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.271+0000 7f6b707bb740 10 client.15699 fetch_fsmap finished waiting for FSMap version 21 2024-03-28T22:11:30.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.271+0000 7f6b707bb740 10 client.15699 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T22:11:30.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.271+0000 7f6b707bb740 20 client.15699 populate_metadata read hostname 'smithi016' 2024-03-28T22:11:30.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.271+0000 7f6b707bb740 1 --2- 172.21.15.16:0/3153142240 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55a5a1718920 0x55a5a1780ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.271+0000 7f6b707bb740 4 client.15699 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:11:30.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.271+0000 7f6b707bb740 1 -- 172.21.15.16:0/3153142240 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a5a166aa40 con 0x55a5a1718920 2024-03-28T22:11:30.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.271+0000 7f6b677fe640 1 --2- 172.21.15.16:0/3153142240 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55a5a1718920 0x55a5a1780ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.272+0000 7f6b46ffd640 10 client.15699 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:11:30.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.272+0000 7f6b677fe640 1 --2- 172.21.15.16:0/3153142240 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55a5a1718920 0x55a5a1780ed0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=eca61ecfe9cd1d01 server_cookie=9a1904c3d8ae1644 in_seq=0 out_seq=0 2024-03-28T22:11:30.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.272+0000 7f6b46ffd640 1 -- 172.21.15.16:0/3153142240 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a5a166aa40 con 0x55a5a1718920 2024-03-28T22:11:30.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.272+0000 7f6b46ffd640 10 client.15699 handle_command_reply: tid=0 2024-03-28T22:11:30.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.272+0000 7f6b707bb740 1 client.15699 shutdown 2024-03-28T22:11:30.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.272+0000 7f6b707bb740 1 -- 172.21.15.16:0/3153142240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a16ffd90 msgr2=0x55a5a170c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:30.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.272+0000 7f6b707bb740 1 --2- 172.21.15.16:0/3153142240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a16ffd90 0x55a5a170c170 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c0c4420 tx=0x7f6b5c0ba690 comp rx=0 tx=0).stop 2024-03-28T22:11:30.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.272+0000 7f6b707bb740 1 -- 172.21.15.16:0/3153142240 shutdown_connections 2024-03-28T22:11:30.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.272+0000 7f6b707bb740 1 --2- 172.21.15.16:0/3153142240 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55a5a1718920 0x55a5a1780ed0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.272+0000 7f6b46ffd640 0 client.15699 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:11:30.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.272+0000 7f6b707bb740 1 --2- 172.21.15.16:0/3153142240 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5a1708970 0x55a5a170c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.272+0000 7f6b707bb740 1 --2- 172.21.15.16:0/3153142240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a5a16ffd90 0x55a5a170c170 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.272+0000 7f6b707bb740 1 --2- 172.21.15.16:0/3153142240 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5a16ff470 0x55a5a1710e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.272+0000 7f6b707bb740 1 -- 172.21.15.16:0/3153142240 >> 172.21.15.16:0/3153142240 conn(0x55a5a16f9ba0 msgr2=0x55a5a16faea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:11:30.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.272+0000 7f6b707bb740 1 -- 172.21.15.16:0/3153142240 shutdown_connections 2024-03-28T22:11:30.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.272+0000 7f6b707bb740 1 -- 172.21.15.16:0/3153142240 wait complete. 2024-03-28T22:11:30.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.273+0000 7f6b707bb740 20 client.15699 trim_cache size 0 max 16384 2024-03-28T22:11:30.280 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:11:30.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.273+0000 7f6b6dbd0640 1 -- 172.21.15.16:0/3599287394 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f6b500a2090 msgr2=0x7f6b500a4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:30.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.273+0000 7f6b6dbd0640 1 --2- 172.21.15.16:0/3599287394 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f6b500a2090 0x7f6b500a4550 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6b58009bd0 tx=0x7f6b58005f90 comp rx=0 tx=0).stop 2024-03-28T22:11:30.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.273+0000 7f6b6dbd0640 1 -- 172.21.15.16:0/3599287394 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b68103290 msgr2=0x7f6b68100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:30.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.273+0000 7f6b6dbd0640 1 --2- 172.21.15.16:0/3599287394 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b68103290 0x7f6b68100b00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c005940 tx=0x7f6b5c02cc40 comp rx=0 tx=0).stop 2024-03-28T22:11:30.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.273+0000 7f6b6dbd0640 1 -- 172.21.15.16:0/3599287394 shutdown_connections 2024-03-28T22:11:30.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.273+0000 7f6b6dbd0640 1 --2- 172.21.15.16:0/3599287394 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f6b500a2090 0x7f6b500a4550 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.273+0000 7f6b6dbd0640 1 --2- 172.21.15.16:0/3599287394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b6810bd40 0x7f6b68101040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.273+0000 7f6b6dbd0640 1 --2- 172.21.15.16:0/3599287394 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b68103290 0x7f6b68100b00 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.273+0000 7f6b6dbd0640 1 --2- 172.21.15.16:0/3599287394 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6b68102970 0x7f6b681005c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.273+0000 7f6b6dbd0640 1 -- 172.21.15.16:0/3599287394 >> 172.21.15.16:0/3599287394 conn(0x7f6b68077800 msgr2=0x7f6b680fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:11:30.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.273+0000 7f6b6dbd0640 1 -- 172.21.15.16:0/3599287394 shutdown_connections 2024-03-28T22:11:30.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.273+0000 7f6b6dbd0640 1 -- 172.21.15.16:0/3599287394 wait complete. 2024-03-28T22:11:30.287 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:11:30.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.404+0000 7fde95d56640 1 -- 172.21.15.16:0/2779162195 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde90100fb0 msgr2=0x7fde9010d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:30.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.404+0000 7fde95d56640 1 --2- 172.21.15.16:0/2779162195 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde90100fb0 0x7fde9010d4c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fde88009be0 tx=0x7fde8802bef0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.404+0000 7fde95d56640 1 -- 172.21.15.16:0/2779162195 shutdown_connections 2024-03-28T22:11:30.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.404+0000 7fde95d56640 1 --2- 172.21.15.16:0/2779162195 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde90100fb0 0x7fde9010d4c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.404+0000 7fde95d56640 1 --2- 172.21.15.16:0/2779162195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde901005f0 0x7fde90100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.404+0000 7fde95d56640 1 --2- 172.21.15.16:0/2779162195 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde90106690 0x7fde90106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.404+0000 7fde95d56640 1 -- 172.21.15.16:0/2779162195 >> 172.21.15.16:0/2779162195 conn(0x7fde90077800 msgr2=0x7fde90077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:11:30.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.404+0000 7fde95d56640 1 -- 172.21.15.16:0/2779162195 shutdown_connections 2024-03-28T22:11:30.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.404+0000 7fde95d56640 1 -- 172.21.15.16:0/2779162195 wait complete. 2024-03-28T22:11:30.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.405+0000 7fde95d56640 1 Processor -- start 2024-03-28T22:11:30.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.405+0000 7fde95d56640 1 -- start start 2024-03-28T22:11:30.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.405+0000 7fde95d56640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde901005f0 0x7fde9019b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.405+0000 7fde95d56640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde90100fb0 0x7fde9019bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.405+0000 7fde95d56640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde90106690 0x7fde90195a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.405+0000 7fde95d56640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fde9010f900 con 0x7fde901005f0 2024-03-28T22:11:30.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.405+0000 7fde95d56640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fde9010f780 con 0x7fde90106690 2024-03-28T22:11:30.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.405+0000 7fde95d56640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fde9010fa80 con 0x7fde90100fb0 2024-03-28T22:11:30.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.405+0000 7fde8f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde901005f0 0x7fde9019b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.405+0000 7fde8f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde901005f0 0x7fde9019b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57996/0 (socket says 172.21.15.16:57996) 2024-03-28T22:11:30.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.405+0000 7fde8f7fe640 1 -- 172.21.15.16:0/3116322904 learned_addr learned my addr 172.21.15.16:0/3116322904 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:11:30.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.405+0000 7fde8ffff640 1 --2- 172.21.15.16:0/3116322904 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde90106690 0x7fde90195a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.405+0000 7fde8effd640 1 --2- 172.21.15.16:0/3116322904 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde90100fb0 0x7fde9019bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.406+0000 7fde8f7fe640 1 -- 172.21.15.16:0/3116322904 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde90100fb0 msgr2=0x7fde9019bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:30.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.406+0000 7fde8f7fe640 1 --2- 172.21.15.16:0/3116322904 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde90100fb0 0x7fde9019bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.406+0000 7fde8f7fe640 1 -- 172.21.15.16:0/3116322904 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde90106690 msgr2=0x7fde90195a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:30.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.406+0000 7fde8f7fe640 1 --2- 172.21.15.16:0/3116322904 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde90106690 0x7fde90195a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.406+0000 7fde8f7fe640 1 -- 172.21.15.16:0/3116322904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde901961c0 con 0x7fde901005f0 2024-03-28T22:11:30.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.406+0000 7fde8ffff640 1 --2- 172.21.15.16:0/3116322904 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde90106690 0x7fde90195a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:11:30.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.406+0000 7fde8f7fe640 1 --2- 172.21.15.16:0/3116322904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde901005f0 0x7fde9019b890 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fde8000c970 tx=0x7fde8000cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:11:30.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.406+0000 7fde8cff9640 1 -- 172.21.15.16:0/3116322904 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde80007b50 con 0x7fde901005f0 2024-03-28T22:11:30.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.406+0000 7fde95d56640 1 -- 172.21.15.16:0/3116322904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde901964b0 con 0x7fde901005f0 2024-03-28T22:11:30.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.406+0000 7fde8cff9640 1 -- 172.21.15.16:0/3116322904 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde80007cf0 con 0x7fde901005f0 2024-03-28T22:11:30.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.406+0000 7fde8cff9640 1 -- 172.21.15.16:0/3116322904 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde8000f7d0 con 0x7fde901005f0 2024-03-28T22:11:30.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.406+0000 7fde95d56640 1 -- 172.21.15.16:0/3116322904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde901a2750 con 0x7fde901005f0 2024-03-28T22:11:30.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.408+0000 7fde8cff9640 1 -- 172.21.15.16:0/3116322904 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 25) ==== 136989+0+0 (secure 0 0 0) 0x7fde80018480 con 0x7fde901005f0 2024-03-28T22:11:30.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.409+0000 7fde8cff9640 1 --2- 172.21.15.16:0/3116322904 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fde780a2010 0x7fde780a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.409+0000 7fde8effd640 1 --2- 172.21.15.16:0/3116322904 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fde780a2010 0x7fde780a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.409+0000 7fde8cff9640 1 -- 172.21.15.16:0/3116322904 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fde800ce710 con 0x7fde901005f0 2024-03-28T22:11:30.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.410+0000 7fde98a1a740 1 Processor -- start 2024-03-28T22:11:30.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.410+0000 7fde98a1a740 1 -- start start 2024-03-28T22:11:30.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.410+0000 7fde98a1a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4a5ca3b00 0x55b4a5ca3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.410+0000 7fde98a1a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4a5c91670 0x55b4a5c91af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.410+0000 7fde98a1a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4a5c92030 0x55b4a5cab9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.410+0000 7fde98a1a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b4a5bfb1b0 con 0x55b4a5c91670 2024-03-28T22:11:30.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.410+0000 7fde8effd640 1 --2- 172.21.15.16:0/3116322904 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fde780a2010 0x7fde780a44d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fde8400b3c0 tx=0x7fde8400d040 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:11:30.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.410+0000 7fde98a1a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55b4a5ba3c00 con 0x55b4a5ca3b00 2024-03-28T22:11:30.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.410+0000 7fde98a1a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55b4a5bf7250 con 0x55b4a5c92030 2024-03-28T22:11:30.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.410+0000 7fde8effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4a5c91670 0x55b4a5c91af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.410+0000 7fde8f7fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4a5ca3b00 0x55b4a5ca3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.410+0000 7fde8effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4a5c91670 0x55b4a5c91af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58002/0 (socket says 172.21.15.16:58002) 2024-03-28T22:11:30.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.410+0000 7fde8effd640 1 -- 172.21.15.16:0/3758201281 learned_addr learned my addr 172.21.15.16:0/3758201281 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:11:30.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.410+0000 7fde8f7fe640 1 --2- 172.21.15.16:0/3758201281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4a5ca3b00 0x55b4a5ca3ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:50388/0 (socket says 172.21.15.16:50388) 2024-03-28T22:11:30.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.410+0000 7fde8ffff640 1 --2- 172.21.15.16:0/3758201281 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4a5c92030 0x55b4a5cab9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.410+0000 7fde8effd640 1 -- 172.21.15.16:0/3758201281 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4a5c92030 msgr2=0x55b4a5cab9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:30.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.411+0000 7fde8effd640 1 --2- 172.21.15.16:0/3758201281 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4a5c92030 0x55b4a5cab9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.411+0000 7fde8effd640 1 -- 172.21.15.16:0/3758201281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4a5ca3b00 msgr2=0x55b4a5ca3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:30.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.411+0000 7fde8effd640 1 --2- 172.21.15.16:0/3758201281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4a5ca3b00 0x55b4a5ca3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.411+0000 7fde8effd640 1 -- 172.21.15.16:0/3758201281 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4a5bc33d0 con 0x55b4a5c91670 2024-03-28T22:11:30.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.411+0000 7fde8f7fe640 1 --2- 172.21.15.16:0/3758201281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4a5ca3b00 0x55b4a5ca3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:11:30.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.411+0000 7fde8effd640 1 --2- 172.21.15.16:0/3758201281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4a5c91670 0x55b4a5c91af0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fde84009f60 tx=0x7fde84014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4499e58f94f616f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:11:30.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.411+0000 7fde75ffb640 1 -- 172.21.15.16:0/3758201281 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde84028070 con 0x55b4a5c91670 2024-03-28T22:11:30.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.411+0000 7fde75ffb640 1 -- 172.21.15.16:0/3758201281 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde8401f040 con 0x55b4a5c91670 2024-03-28T22:11:30.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.411+0000 7fde75ffb640 1 -- 172.21.15.16:0/3758201281 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde84023430 con 0x55b4a5c91670 2024-03-28T22:11:30.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.412+0000 7fde98a1a740 1 -- 172.21.15.16:0/3758201281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4a5c91670 msgr2=0x55b4a5c91af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:30.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.412+0000 7fde98a1a740 1 --2- 172.21.15.16:0/3758201281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4a5c91670 0x55b4a5c91af0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fde84009f60 tx=0x7fde84014000 comp rx=0 tx=0).stop 2024-03-28T22:11:30.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.412+0000 7fde98a1a740 1 -- 172.21.15.16:0/3758201281 shutdown_connections 2024-03-28T22:11:30.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.412+0000 7fde98a1a740 1 --2- 172.21.15.16:0/3758201281 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4a5c92030 0x55b4a5cab9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.412+0000 7fde98a1a740 1 --2- 172.21.15.16:0/3758201281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4a5c91670 0x55b4a5c91af0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.412+0000 7fde98a1a740 1 --2- 172.21.15.16:0/3758201281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4a5ca3b00 0x55b4a5ca3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.412+0000 7fde98a1a740 1 -- 172.21.15.16:0/3758201281 >> 172.21.15.16:0/3758201281 conn(0x55b4a5c94130 msgr2=0x55b4a5c94510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:11:30.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.412+0000 7fde98a1a740 1 -- 172.21.15.16:0/3758201281 shutdown_connections 2024-03-28T22:11:30.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.412+0000 7fde98a1a740 1 -- 172.21.15.16:0/3758201281 wait complete. 2024-03-28T22:11:30.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.413+0000 7fde98a1a740 1 -- start start 2024-03-28T22:11:30.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.413+0000 7fde98a1a740 1 Processor -- start 2024-03-28T22:11:30.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.413+0000 7fde98a1a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4a5c91670 0x55b4a5ca8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.413+0000 7fde98a1a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4a5c92030 0x55b4a5ca93f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.413+0000 7fde8f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4a5c91670 0x55b4a5ca8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.414+0000 7fde98a1a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4a5ca3b00 0x55b4a5ca9930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.414+0000 7fde8f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4a5c91670 0x55b4a5ca8eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58004/0 (socket says 172.21.15.16:58004) 2024-03-28T22:11:30.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.414+0000 7fde8effd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4a5c92030 0x55b4a5ca93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.414+0000 7fde8f7fe640 1 -- 172.21.15.16:0/1153146415 learned_addr learned my addr 172.21.15.16:0/1153146415 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:11:30.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.414+0000 7fde98a1a740 1 -- 172.21.15.16:0/1153146415 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b4a5cac030 con 0x55b4a5c91670 2024-03-28T22:11:30.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.414+0000 7fde98a1a740 1 -- 172.21.15.16:0/1153146415 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55b4a5b99470 con 0x55b4a5c92030 2024-03-28T22:11:30.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.414+0000 7fde98a1a740 1 -- 172.21.15.16:0/1153146415 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55b4a5cac1b0 con 0x55b4a5ca3b00 2024-03-28T22:11:30.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.414+0000 7fde98a1a740 10 client.0 _ll_register_callbacks cb 0x55b4a5bb6cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:11:30.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.414+0000 7fde8ffff640 1 --2- 172.21.15.16:0/1153146415 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4a5ca3b00 0x55b4a5ca9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.414+0000 7fde8effd640 1 -- 172.21.15.16:0/1153146415 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4a5ca3b00 msgr2=0x55b4a5ca9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:30.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.414+0000 7fde8effd640 1 --2- 172.21.15.16:0/1153146415 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4a5ca3b00 0x55b4a5ca9930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.414+0000 7fde8effd640 1 -- 172.21.15.16:0/1153146415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4a5c91670 msgr2=0x55b4a5ca8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:30.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.414+0000 7fde8effd640 1 --2- 172.21.15.16:0/1153146415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4a5c91670 0x55b4a5ca8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.414+0000 7fde8effd640 1 -- 172.21.15.16:0/1153146415 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4a590f3f0 con 0x55b4a5c92030 2024-03-28T22:11:30.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.414+0000 7fde8f7fe640 1 --2- 172.21.15.16:0/1153146415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4a5c91670 0x55b4a5ca8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:11:30.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.414+0000 7fde98a1a740 1 -- 172.21.15.16:0/1153146415 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde8401f040 con 0x55b4a5c92030 2024-03-28T22:11:30.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.414+0000 7fde8effd640 1 --2- 172.21.15.16:0/1153146415 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4a5c92030 0x55b4a5ca93f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fde8400be50 tx=0x7fde8400b460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e47d8ac3ca4fe385 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:11:30.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.414+0000 7fde6effd640 10 client.25613 ms_handle_connect on v2:172.21.15.53:3300/0 2024-03-28T22:11:30.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.415+0000 7fde6effd640 1 -- 172.21.15.16:0/1153146415 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde84028050 con 0x55b4a5c92030 2024-03-28T22:11:30.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.415+0000 7fde6effd640 1 -- 172.21.15.16:0/1153146415 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fde84003d80 con 0x55b4a5c92030 2024-03-28T22:11:30.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.415+0000 7fde6effd640 1 -- 172.21.15.16:0/1153146415 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fde84018460 con 0x55b4a5c92030 2024-03-28T22:11:30.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.415+0000 7fde98a1a740 10 client.25613 fetch_fsmap learned FSMap version 21 2024-03-28T22:11:30.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.415+0000 7fde98a1a740 1 -- 172.21.15.16:0/1153146415 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({fsmap=21}) -- 0x55b4a59d97d0 con 0x55b4a5c92030 2024-03-28T22:11:30.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.415+0000 7fde6effd640 1 -- 172.21.15.16:0/1153146415 <== mon.2 v2:172.21.15.53:3300/0 4 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fde84018600 con 0x55b4a5c92030 2024-03-28T22:11:30.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.415+0000 7fde98a1a740 10 client.25613 fetch_fsmap finished waiting for FSMap version 21 2024-03-28T22:11:30.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.415+0000 7fde98a1a740 10 client.25613 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:11:30.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.415+0000 7fde98a1a740 20 client.25613 populate_metadata read hostname 'smithi016' 2024-03-28T22:11:30.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.415+0000 7fde98a1a740 1 --2- 172.21.15.16:0/1153146415 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b4a5cb93f0 0x55b4a5cbb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.415+0000 7fde98a1a740 4 client.25613 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:11:30.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.415+0000 7fde98a1a740 1 -- 172.21.15.16:0/1153146415 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b4a5c086f0 con 0x55b4a5cb93f0 2024-03-28T22:11:30.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.415+0000 7fde8f7fe640 1 --2- 172.21.15.16:0/1153146415 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b4a5cb93f0 0x55b4a5cbb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.416+0000 7fde6effd640 10 client.25613 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:11:30.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.416+0000 7fde8f7fe640 1 --2- 172.21.15.16:0/1153146415 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b4a5cb93f0 0x55b4a5cbb870 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1982e29459c7d83f server_cookie=5de9edb6da44c2d4 in_seq=0 out_seq=0 2024-03-28T22:11:30.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.416+0000 7fde6effd640 1 -- 172.21.15.16:0/1153146415 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b4a5c086f0 con 0x55b4a5cb93f0 2024-03-28T22:11:30.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.416+0000 7fde6effd640 10 client.25613 handle_command_reply: tid=0 2024-03-28T22:11:30.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.416+0000 7fde98a1a740 1 client.25613 shutdown 2024-03-28T22:11:30.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.417+0000 7fde98a1a740 1 -- 172.21.15.16:0/1153146415 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4a5c92030 msgr2=0x55b4a5ca93f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:30.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.417+0000 7fde98a1a740 1 --2- 172.21.15.16:0/1153146415 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4a5c92030 0x55b4a5ca93f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fde8400be50 tx=0x7fde8400b460 comp rx=0 tx=0).stop 2024-03-28T22:11:30.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.417+0000 7fde98a1a740 1 -- 172.21.15.16:0/1153146415 shutdown_connections 2024-03-28T22:11:30.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.417+0000 7fde98a1a740 1 --2- 172.21.15.16:0/1153146415 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b4a5cb93f0 0x55b4a5cbb870 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.417+0000 7fde98a1a740 1 --2- 172.21.15.16:0/1153146415 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4a5ca3b00 0x55b4a5ca9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.417+0000 7fde98a1a740 1 --2- 172.21.15.16:0/1153146415 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4a5c92030 0x55b4a5ca93f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.417+0000 7fde6effd640 0 client.25613 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:11:30.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.417+0000 7fde98a1a740 1 --2- 172.21.15.16:0/1153146415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4a5c91670 0x55b4a5ca8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.417+0000 7fde98a1a740 1 -- 172.21.15.16:0/1153146415 >> 172.21.15.16:0/1153146415 conn(0x55b4a5c94130 msgr2=0x55b4a5ca51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:11:30.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.417+0000 7fde98a1a740 1 -- 172.21.15.16:0/1153146415 shutdown_connections 2024-03-28T22:11:30.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.417+0000 7fde98a1a740 1 -- 172.21.15.16:0/1153146415 wait complete. 2024-03-28T22:11:30.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.417+0000 7fde98a1a740 20 client.25613 trim_cache size 0 max 16384 2024-03-28T22:11:30.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.431+0000 7fde98a1a740 1 Processor -- start 2024-03-28T22:11:30.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.431+0000 7fde98a1a740 1 -- start start 2024-03-28T22:11:30.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.431+0000 7fde98a1a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4a5c91670 0x55b4a5ca1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.431+0000 7fde98a1a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4a5c92030 0x55b4a5cb4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.431+0000 7fde98a1a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4a5ca3b00 0x55b4a5ca9750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.431+0000 7fde98a1a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fde901ad920 con 0x55b4a5ca3b00 2024-03-28T22:11:30.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.431+0000 7fde98a1a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fde901ab1f0 con 0x55b4a5c92030 2024-03-28T22:11:30.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.431+0000 7fde98a1a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fde901a9040 con 0x55b4a5c91670 2024-03-28T22:11:30.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.431+0000 7fde8effd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4a5c92030 0x55b4a5cb4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.431+0000 7fde8effd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4a5c92030 0x55b4a5cb4dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:50398/0 (socket says 172.21.15.16:50398) 2024-03-28T22:11:30.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.431+0000 7fde8effd640 1 -- 172.21.15.16:0/4054597899 learned_addr learned my addr 172.21.15.16:0/4054597899 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:11:30.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.431+0000 7fde8f7fe640 1 --2- 172.21.15.16:0/4054597899 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4a5c91670 0x55b4a5ca1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.431+0000 7fde8ffff640 1 --2- 172.21.15.16:0/4054597899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4a5ca3b00 0x55b4a5ca9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.431+0000 7fde8effd640 1 -- 172.21.15.16:0/4054597899 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4a5c91670 msgr2=0x55b4a5ca1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:30.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.431+0000 7fde8effd640 1 --2- 172.21.15.16:0/4054597899 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4a5c91670 0x55b4a5ca1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.431+0000 7fde8effd640 1 -- 172.21.15.16:0/4054597899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4a5ca3b00 msgr2=0x55b4a5ca9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:30.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.431+0000 7fde8effd640 1 --2- 172.21.15.16:0/4054597899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4a5ca3b00 0x55b4a5ca9750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.431+0000 7fde8effd640 1 -- 172.21.15.16:0/4054597899 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4a5ca1ab0 con 0x55b4a5c92030 2024-03-28T22:11:30.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.431+0000 7fde8f7fe640 1 --2- 172.21.15.16:0/4054597899 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4a5c91670 0x55b4a5ca1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:11:30.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.432+0000 7fde8effd640 1 --2- 172.21.15.16:0/4054597899 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4a5c92030 0x55b4a5cb4dd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fde8400d010 tx=0x7fde8402e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a55af819d9437aac server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:11:30.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.432+0000 7fde757fa640 1 -- 172.21.15.16:0/4054597899 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde84028070 con 0x55b4a5c92030 2024-03-28T22:11:30.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.432+0000 7fde757fa640 1 -- 172.21.15.16:0/4054597899 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde8401f040 con 0x55b4a5c92030 2024-03-28T22:11:30.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.432+0000 7fde757fa640 1 -- 172.21.15.16:0/4054597899 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde84018b90 con 0x55b4a5c92030 2024-03-28T22:11:30.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.433+0000 7fde98a1a740 1 -- 172.21.15.16:0/4054597899 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4a5c92030 msgr2=0x55b4a5cb4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:30.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.433+0000 7fde98a1a740 1 --2- 172.21.15.16:0/4054597899 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4a5c92030 0x55b4a5cb4dd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fde8400d010 tx=0x7fde8402e000 comp rx=0 tx=0).stop 2024-03-28T22:11:30.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.433+0000 7fde98a1a740 1 -- 172.21.15.16:0/4054597899 shutdown_connections 2024-03-28T22:11:30.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.433+0000 7fde98a1a740 1 --2- 172.21.15.16:0/4054597899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4a5ca3b00 0x55b4a5ca9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.433+0000 7fde98a1a740 1 --2- 172.21.15.16:0/4054597899 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4a5c92030 0x55b4a5cb4dd0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.433+0000 7fde98a1a740 1 --2- 172.21.15.16:0/4054597899 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4a5c91670 0x55b4a5ca1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.433+0000 7fde98a1a740 1 -- 172.21.15.16:0/4054597899 >> 172.21.15.16:0/4054597899 conn(0x55b4a5c94130 msgr2=0x55b4a5ca1fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:11:30.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.433+0000 7fde98a1a740 1 -- 172.21.15.16:0/4054597899 shutdown_connections 2024-03-28T22:11:30.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.433+0000 7fde98a1a740 1 -- 172.21.15.16:0/4054597899 wait complete. 2024-03-28T22:11:30.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.433+0000 7fde98a1a740 1 -- start start 2024-03-28T22:11:30.438 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:30 smithi016 ceph-mon[32374]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:11:30.438 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:30 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2471516290' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:11:30.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.434+0000 7fde98a1a740 1 Processor -- start 2024-03-28T22:11:30.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.434+0000 7fde98a1a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4a5c91670 0x55b4a5cb1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.434+0000 7fde98a1a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4a5c92030 0x55b4a5cc0170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.434+0000 7fde98a1a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4a5ca3b00 0x55b4a5cc06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.434+0000 7fde98a1a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fde901a96b0 con 0x55b4a5ca3b00 2024-03-28T22:11:30.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.434+0000 7fde98a1a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fde901aab20 con 0x55b4a5c92030 2024-03-28T22:11:30.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.434+0000 7fde98a1a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fde901a9d80 con 0x55b4a5c91670 2024-03-28T22:11:30.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.434+0000 7fde8effd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4a5c92030 0x55b4a5cc0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.434+0000 7fde8effd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4a5c92030 0x55b4a5cc0170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:50412/0 (socket says 172.21.15.16:50412) 2024-03-28T22:11:30.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.434+0000 7fde8effd640 1 -- 172.21.15.16:0/1574544604 learned_addr learned my addr 172.21.15.16:0/1574544604 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:11:30.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.434+0000 7fde98a1a740 10 client.0 _ll_register_callbacks cb 0x7fde84009e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:11:30.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.434+0000 7fde8f7fe640 1 --2- 172.21.15.16:0/1574544604 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4a5c91670 0x55b4a5cb1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.435+0000 7fde8ffff640 1 --2- 172.21.15.16:0/1574544604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4a5ca3b00 0x55b4a5cc06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.435+0000 7fde8effd640 1 -- 172.21.15.16:0/1574544604 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4a5c91670 msgr2=0x55b4a5cb1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:30.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.435+0000 7fde8effd640 1 --2- 172.21.15.16:0/1574544604 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4a5c91670 0x55b4a5cb1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.435+0000 7fde8effd640 1 -- 172.21.15.16:0/1574544604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4a5ca3b00 msgr2=0x55b4a5cc06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:30.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.435+0000 7fde8effd640 1 --2- 172.21.15.16:0/1574544604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4a5ca3b00 0x55b4a5cc06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.435+0000 7fde8effd640 1 -- 172.21.15.16:0/1574544604 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4a5cc0bf0 con 0x55b4a5c92030 2024-03-28T22:11:30.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.435+0000 7fde8ffff640 1 --2- 172.21.15.16:0/1574544604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4a5ca3b00 0x55b4a5cc06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:11:30.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.435+0000 7fde8effd640 1 --2- 172.21.15.16:0/1574544604 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4a5c92030 0x55b4a5cc0170 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fde840039c0 tx=0x7fde8402ff80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d540b730c2767266 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:11:30.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.435+0000 7fde6effd640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-03-28T22:11:30.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.435+0000 7fde6effd640 1 -- 172.21.15.16:0/1574544604 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde8404e060 con 0x55b4a5c92030 2024-03-28T22:11:30.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.435+0000 7fde6effd640 1 -- 172.21.15.16:0/1574544604 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fde84023430 con 0x55b4a5c92030 2024-03-28T22:11:30.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.435+0000 7fde98a1a740 1 -- 172.21.15.16:0/1574544604 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde8401f040 con 0x55b4a5c92030 2024-03-28T22:11:30.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.435+0000 7fde6effd640 1 -- 172.21.15.16:0/1574544604 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fde84028050 con 0x55b4a5c92030 2024-03-28T22:11:30.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.435+0000 7fde98a1a740 10 client.25619 fetch_fsmap learned FSMap version 21 2024-03-28T22:11:30.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.435+0000 7fde98a1a740 1 -- 172.21.15.16:0/1574544604 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b4a5d1f440 con 0x55b4a5c92030 2024-03-28T22:11:30.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.435+0000 7fde6effd640 1 -- 172.21.15.16:0/1574544604 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde84029040 con 0x55b4a5c92030 2024-03-28T22:11:30.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.436+0000 7fde6effd640 1 -- 172.21.15.16:0/1574544604 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fde84041410 con 0x55b4a5c92030 2024-03-28T22:11:30.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.436+0000 7fde98a1a740 10 client.25619 fetch_fsmap finished waiting for FSMap version 21 2024-03-28T22:11:30.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.436+0000 7fde98a1a740 10 client.25619 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:11:30.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.436+0000 7fde98a1a740 20 client.25619 populate_metadata read hostname 'smithi016' 2024-03-28T22:11:30.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.436+0000 7fde98a1a740 1 --2- 172.21.15.16:0/1574544604 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b4a5d1fa90 0x55b4a5d21eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:30.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.436+0000 7fde98a1a740 4 client.25619 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:11:30.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.436+0000 7fde98a1a740 1 -- 172.21.15.16:0/1574544604 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fde8401f040 con 0x55b4a5d1fa90 2024-03-28T22:11:30.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.436+0000 7fde8f7fe640 1 --2- 172.21.15.16:0/1574544604 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b4a5d1fa90 0x55b4a5d21eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:30.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.436+0000 7fde6effd640 10 client.25619 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:11:30.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.436+0000 7fde8f7fe640 1 --2- 172.21.15.16:0/1574544604 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b4a5d1fa90 0x55b4a5d21eb0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bed653e4d6e54023 server_cookie=2bb76d6b800e55 in_seq=0 out_seq=0 2024-03-28T22:11:30.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.436+0000 7fde6effd640 1 -- 172.21.15.16:0/1574544604 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fde8401f040 con 0x55b4a5d1fa90 2024-03-28T22:11:30.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.436+0000 7fde6effd640 10 client.25619 handle_command_reply: tid=0 2024-03-28T22:11:30.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.436+0000 7fde98a1a740 1 client.25619 shutdown 2024-03-28T22:11:30.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.437+0000 7fde98a1a740 1 -- 172.21.15.16:0/1574544604 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4a5c92030 msgr2=0x55b4a5cc0170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:30.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.437+0000 7fde98a1a740 1 --2- 172.21.15.16:0/1574544604 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4a5c92030 0x55b4a5cc0170 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fde840039c0 tx=0x7fde8402ff80 comp rx=0 tx=0).stop 2024-03-28T22:11:30.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.437+0000 7fde98a1a740 1 -- 172.21.15.16:0/1574544604 shutdown_connections 2024-03-28T22:11:30.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.437+0000 7fde98a1a740 1 --2- 172.21.15.16:0/1574544604 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b4a5d1fa90 0x55b4a5d21eb0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.437+0000 7fde98a1a740 1 --2- 172.21.15.16:0/1574544604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4a5ca3b00 0x55b4a5cc06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.437+0000 7fde6effd640 0 client.25619 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:11:30.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.437+0000 7fde98a1a740 1 --2- 172.21.15.16:0/1574544604 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b4a5c92030 0x55b4a5cc0170 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.437+0000 7fde98a1a740 1 --2- 172.21.15.16:0/1574544604 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b4a5c91670 0x55b4a5cb1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.437+0000 7fde98a1a740 1 -- 172.21.15.16:0/1574544604 >> 172.21.15.16:0/1574544604 conn(0x55b4a5c94130 msgr2=0x55b4a5ca76e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:11:30.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.437+0000 7fde98a1a740 1 -- 172.21.15.16:0/1574544604 shutdown_connections 2024-03-28T22:11:30.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.437+0000 7fde98a1a740 1 -- 172.21.15.16:0/1574544604 wait complete. 2024-03-28T22:11:30.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.437+0000 7fde98a1a740 20 client.25619 trim_cache size 0 max 16384 2024-03-28T22:11:30.445 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:11:30.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.438+0000 7fde95d56640 1 -- 172.21.15.16:0/3116322904 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fde780a2010 msgr2=0x7fde780a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:30.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.438+0000 7fde95d56640 1 --2- 172.21.15.16:0/3116322904 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fde780a2010 0x7fde780a44d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fde8400b3c0 tx=0x7fde8400d040 comp rx=0 tx=0).stop 2024-03-28T22:11:30.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.438+0000 7fde95d56640 1 -- 172.21.15.16:0/3116322904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde901005f0 msgr2=0x7fde9019b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:30.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.438+0000 7fde95d56640 1 --2- 172.21.15.16:0/3116322904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde901005f0 0x7fde9019b890 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fde8000c970 tx=0x7fde8000cd70 comp rx=0 tx=0).stop 2024-03-28T22:11:30.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.438+0000 7fde95d56640 1 -- 172.21.15.16:0/3116322904 shutdown_connections 2024-03-28T22:11:30.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.438+0000 7fde95d56640 1 --2- 172.21.15.16:0/3116322904 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fde780a2010 0x7fde780a44d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.438+0000 7fde95d56640 1 --2- 172.21.15.16:0/3116322904 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde90106690 0x7fde90195a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.438+0000 7fde95d56640 1 --2- 172.21.15.16:0/3116322904 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde90100fb0 0x7fde9019bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.438+0000 7fde95d56640 1 --2- 172.21.15.16:0/3116322904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde901005f0 0x7fde9019b890 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:30.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.438+0000 7fde95d56640 1 -- 172.21.15.16:0/3116322904 >> 172.21.15.16:0/3116322904 conn(0x7fde90077800 msgr2=0x7fde900fef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:11:30.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.438+0000 7fde95d56640 1 -- 172.21.15.16:0/3116322904 shutdown_connections 2024-03-28T22:11:30.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:30.438+0000 7fde95d56640 1 -- 172.21.15.16:0/3116322904 wait complete. 2024-03-28T22:11:30.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:30 smithi053 ceph-mon[35979]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:11:30.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:30 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2471516290' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:11:30.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:30 smithi063 ceph-mon[36249]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:11:30.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:30 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2471516290' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:11:31.455 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:11:31.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:11:31.455 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:11:31.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.572+0000 7f5556b25640 1 -- 172.21.15.16:0/754205477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55501005f0 msgr2=0x7f5550100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:31.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.572+0000 7f5556b25640 1 --2- 172.21.15.16:0/754205477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55501005f0 0x7f5550100a70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f553c009860 tx=0x7f553c02bb70 comp rx=0 tx=0).stop 2024-03-28T22:11:31.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.572+0000 7f5556b25640 1 -- 172.21.15.16:0/754205477 shutdown_connections 2024-03-28T22:11:31.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.572+0000 7f5556b25640 1 --2- 172.21.15.16:0/754205477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5550100fb0 0x7f555010d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.572+0000 7f5556b25640 1 --2- 172.21.15.16:0/754205477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55501005f0 0x7f5550100a70 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.572+0000 7f5556b25640 1 --2- 172.21.15.16:0/754205477 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5550106690 0x7f5550106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.572+0000 7f5556b25640 1 -- 172.21.15.16:0/754205477 >> 172.21.15.16:0/754205477 conn(0x7f5550077800 msgr2=0x7f5550077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:11:31.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.572+0000 7f5556b25640 1 -- 172.21.15.16:0/754205477 shutdown_connections 2024-03-28T22:11:31.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.572+0000 7f5556b25640 1 -- 172.21.15.16:0/754205477 wait complete. 2024-03-28T22:11:31.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.573+0000 7f5556b25640 1 Processor -- start 2024-03-28T22:11:31.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.573+0000 7f5556b25640 1 -- start start 2024-03-28T22:11:31.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.573+0000 7f5556b25640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55501005f0 0x7f5550107560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:31.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.573+0000 7f5556b25640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5550100fb0 0x7f5550107aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:31.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.573+0000 7f5556b25640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5550106690 0x7f5550107fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:31.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.573+0000 7f5556b25640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f55501109d0 con 0x7f5550100fb0 2024-03-28T22:11:31.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.573+0000 7f5556b25640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5550110850 con 0x7f55501005f0 2024-03-28T22:11:31.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.573+0000 7f5556b25640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5550110b50 con 0x7f5550106690 2024-03-28T22:11:31.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.573+0000 7f554ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5550100fb0 0x7f5550107aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:31.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.573+0000 7f554ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5550100fb0 0x7f5550107aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48638/0 (socket says 172.21.15.16:48638) 2024-03-28T22:11:31.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.573+0000 7f554ffff640 1 -- 172.21.15.16:0/1919400485 learned_addr learned my addr 172.21.15.16:0/1919400485 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:11:31.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.573+0000 7f55548c1640 1 --2- 172.21.15.16:0/1919400485 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55501005f0 0x7f5550107560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:31.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.574+0000 7f55550c2640 1 --2- 172.21.15.16:0/1919400485 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5550106690 0x7f5550107fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:31.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.574+0000 7f554ffff640 1 -- 172.21.15.16:0/1919400485 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5550106690 msgr2=0x7f5550107fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:31.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.574+0000 7f554ffff640 1 --2- 172.21.15.16:0/1919400485 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5550106690 0x7f5550107fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.574+0000 7f554ffff640 1 -- 172.21.15.16:0/1919400485 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55501005f0 msgr2=0x7f5550107560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:31.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.574+0000 7f554ffff640 1 --2- 172.21.15.16:0/1919400485 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55501005f0 0x7f5550107560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.574+0000 7f554ffff640 1 -- 172.21.15.16:0/1919400485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55501a64e0 con 0x7f5550100fb0 2024-03-28T22:11:31.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.574+0000 7f55548c1640 1 --2- 172.21.15.16:0/1919400485 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55501005f0 0x7f5550107560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:11:31.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.574+0000 7f554ffff640 1 --2- 172.21.15.16:0/1919400485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5550100fb0 0x7f5550107aa0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f553c037040 tx=0x7f553c0045b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:11:31.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.574+0000 7f554dffb640 1 -- 172.21.15.16:0/1919400485 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f553c041020 con 0x7f5550100fb0 2024-03-28T22:11:31.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.574+0000 7f5556b25640 1 -- 172.21.15.16:0/1919400485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55501a6770 con 0x7f5550100fb0 2024-03-28T22:11:31.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.574+0000 7f554dffb640 1 -- 172.21.15.16:0/1919400485 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f553c02c9f0 con 0x7f5550100fb0 2024-03-28T22:11:31.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.574+0000 7f554dffb640 1 -- 172.21.15.16:0/1919400485 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f553c039040 con 0x7f5550100fb0 2024-03-28T22:11:31.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.574+0000 7f5556b25640 1 -- 172.21.15.16:0/1919400485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55501a6c60 con 0x7f5550100fb0 2024-03-28T22:11:31.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.577+0000 7f554dffb640 1 -- 172.21.15.16:0/1919400485 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 25) ==== 136989+0+0 (secure 0 0 0) 0x7f553c044020 con 0x7f5550100fb0 2024-03-28T22:11:31.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.577+0000 7f554dffb640 1 --2- 172.21.15.16:0/1919400485 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f55400a2010 0x7f55400a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:31.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.578+0000 7f55548c1640 1 --2- 172.21.15.16:0/1919400485 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f55400a2010 0x7f55400a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:31.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.578+0000 7f554dffb640 1 -- 172.21.15.16:0/1919400485 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f553c0ef5a0 con 0x7f5550100fb0 2024-03-28T22:11:31.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.578+0000 7f55548c1640 1 --2- 172.21.15.16:0/1919400485 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f55400a2010 0x7f55400a44d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5550101730 tx=0x7f5544005f90 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:11:31.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.579+0000 7f55597e6740 1 Processor -- start 2024-03-28T22:11:31.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.579+0000 7f55597e6740 1 -- start start 2024-03-28T22:11:31.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.579+0000 7f55597e6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560326c949f0 0x560326c94dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:31.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.580+0000 7f55597e6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560326c95310 0x560326c9d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:31.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.580+0000 7f55597e6740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560326c9def0 0x560326ca02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:31.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.580+0000 7f55548c1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560326c949f0 0x560326c94dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:31.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.580+0000 7f55548c1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560326c949f0 0x560326c94dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48644/0 (socket says 172.21.15.16:48644) 2024-03-28T22:11:31.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.580+0000 7f55548c1640 1 -- 172.21.15.16:0/1200728263 learned_addr learned my addr 172.21.15.16:0/1200728263 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:11:31.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.580+0000 7f55597e6740 1 -- 172.21.15.16:0/1200728263 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560326b950b0 con 0x560326c949f0 2024-03-28T22:11:31.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.580+0000 7f554ffff640 1 --2- 172.21.15.16:0/1200728263 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560326c95310 0x560326c9d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:31.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.580+0000 7f55597e6740 1 -- 172.21.15.16:0/1200728263 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x560326bf7ea0 con 0x560326c9def0 2024-03-28T22:11:31.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.580+0000 7f55550c2640 1 --2- 172.21.15.16:0/1200728263 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560326c9def0 0x560326ca02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:31.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.580+0000 7f55597e6740 1 -- 172.21.15.16:0/1200728263 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x560326ba0c00 con 0x560326c95310 2024-03-28T22:11:31.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.580+0000 7f55548c1640 1 -- 172.21.15.16:0/1200728263 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560326c95310 msgr2=0x560326c9d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:31.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.580+0000 7f55548c1640 1 --2- 172.21.15.16:0/1200728263 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560326c95310 0x560326c9d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.581+0000 7f55548c1640 1 -- 172.21.15.16:0/1200728263 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560326c9def0 msgr2=0x560326ca02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:31.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.581+0000 7f55548c1640 1 --2- 172.21.15.16:0/1200728263 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560326c9def0 0x560326ca02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.581+0000 7f55548c1640 1 -- 172.21.15.16:0/1200728263 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603269d47e0 con 0x560326c949f0 2024-03-28T22:11:31.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.581+0000 7f55550c2640 1 --2- 172.21.15.16:0/1200728263 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560326c9def0 0x560326ca02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:11:31.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.581+0000 7f55548c1640 1 --2- 172.21.15.16:0/1200728263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560326c949f0 0x560326c94dd0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f55440058c0 tx=0x7f5544014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d50de3a7b1863198 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:11:31.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.581+0000 7f553affd640 1 -- 172.21.15.16:0/1200728263 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5544005020 con 0x560326c949f0 2024-03-28T22:11:31.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.581+0000 7f553affd640 1 -- 172.21.15.16:0/1200728263 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55440047a0 con 0x560326c949f0 2024-03-28T22:11:31.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.581+0000 7f553affd640 1 -- 172.21.15.16:0/1200728263 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f554401c7b0 con 0x560326c949f0 2024-03-28T22:11:31.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.582+0000 7f55597e6740 1 -- 172.21.15.16:0/1200728263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560326c949f0 msgr2=0x560326c94dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:31.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.582+0000 7f55597e6740 1 --2- 172.21.15.16:0/1200728263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560326c949f0 0x560326c94dd0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f55440058c0 tx=0x7f5544014000 comp rx=0 tx=0).stop 2024-03-28T22:11:31.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.582+0000 7f55597e6740 1 -- 172.21.15.16:0/1200728263 shutdown_connections 2024-03-28T22:11:31.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.582+0000 7f55597e6740 1 --2- 172.21.15.16:0/1200728263 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560326c9def0 0x560326ca02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.582+0000 7f55597e6740 1 --2- 172.21.15.16:0/1200728263 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560326c95310 0x560326c9d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.582+0000 7f55597e6740 1 --2- 172.21.15.16:0/1200728263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560326c949f0 0x560326c94dd0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.582+0000 7f55597e6740 1 -- 172.21.15.16:0/1200728263 >> 172.21.15.16:0/1200728263 conn(0x560326c8f120 msgr2=0x560326c8f500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:11:31.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.582+0000 7f55597e6740 1 -- 172.21.15.16:0/1200728263 shutdown_connections 2024-03-28T22:11:31.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.582+0000 7f55597e6740 1 -- 172.21.15.16:0/1200728263 wait complete. 2024-03-28T22:11:31.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.582+0000 7f55597e6740 1 -- start start 2024-03-28T22:11:31.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.582+0000 7f55597e6740 1 Processor -- start 2024-03-28T22:11:31.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.583+0000 7f55597e6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560326c95310 0x560326c9b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:31.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.583+0000 7f55597e6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560326c9def0 0x560326c9b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:31.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.583+0000 7f55597e6740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560326c9bb30 0x560326ca9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:31.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.583+0000 7f55597e6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560326b963d0 con 0x560326c95310 2024-03-28T22:11:31.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.583+0000 7f55597e6740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x560326bf3f40 con 0x560326c9bb30 2024-03-28T22:11:31.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.583+0000 7f55597e6740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x560326ca0ce0 con 0x560326c9def0 2024-03-28T22:11:31.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.583+0000 7f55597e6740 10 client.0 _ll_register_callbacks cb 0x560326bb3b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:11:31.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.583+0000 7f55548c1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560326c95310 0x560326c9b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:31.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.583+0000 7f55550c2640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560326c9bb30 0x560326ca9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:31.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.583+0000 7f55548c1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560326c95310 0x560326c9b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48658/0 (socket says 172.21.15.16:48658) 2024-03-28T22:11:31.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.583+0000 7f55548c1640 1 -- 172.21.15.16:0/3696117971 learned_addr learned my addr 172.21.15.16:0/3696117971 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:11:31.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.583+0000 7f554ffff640 1 --2- 172.21.15.16:0/3696117971 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560326c9def0 0x560326c9b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:31.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.583+0000 7f55550c2640 1 -- 172.21.15.16:0/3696117971 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560326c9def0 msgr2=0x560326c9b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:31.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.583+0000 7f55550c2640 1 --2- 172.21.15.16:0/3696117971 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560326c9def0 0x560326c9b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.583+0000 7f55550c2640 1 -- 172.21.15.16:0/3696117971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560326c95310 msgr2=0x560326c9b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:31.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.583+0000 7f55550c2640 1 --2- 172.21.15.16:0/3696117971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560326c95310 0x560326c9b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.583+0000 7f55550c2640 1 -- 172.21.15.16:0/3696117971 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560326a34f70 con 0x560326c9bb30 2024-03-28T22:11:31.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.583+0000 7f55548c1640 1 --2- 172.21.15.16:0/3696117971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560326c95310 0x560326c9b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:11:31.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.584+0000 7f55550c2640 1 --2- 172.21.15.16:0/3696117971 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560326c9bb30 0x560326ca9fd0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5548009a60 tx=0x7f554800b360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=255799429ebffe53 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:11:31.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.584+0000 7f552ffff640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-03-28T22:11:31.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.584+0000 7f552ffff640 1 -- 172.21.15.16:0/3696117971 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f554801d030 con 0x560326c9bb30 2024-03-28T22:11:31.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.584+0000 7f55597e6740 1 -- 172.21.15.16:0/3696117971 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55440047a0 con 0x560326c9bb30 2024-03-28T22:11:31.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.584+0000 7f552ffff640 1 -- 172.21.15.16:0/3696117971 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f554800bc90 con 0x560326c9bb30 2024-03-28T22:11:31.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.584+0000 7f552ffff640 1 -- 172.21.15.16:0/3696117971 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f5548027050 con 0x560326c9bb30 2024-03-28T22:11:31.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.584+0000 7f55597e6740 10 client.25631 fetch_fsmap learned FSMap version 21 2024-03-28T22:11:31.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.584+0000 7f55597e6740 1 -- 172.21.15.16:0/3696117971 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5603268cc170 con 0x560326c9bb30 2024-03-28T22:11:31.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.584+0000 7f552ffff640 1 -- 172.21.15.16:0/3696117971 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55440047a0 con 0x560326c9bb30 2024-03-28T22:11:31.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.584+0000 7f552ffff640 1 -- 172.21.15.16:0/3696117971 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f554800b040 con 0x560326c9bb30 2024-03-28T22:11:31.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.584+0000 7f55597e6740 10 client.25631 fetch_fsmap finished waiting for FSMap version 21 2024-03-28T22:11:31.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.584+0000 7f55597e6740 10 client.25631 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:11:31.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.584+0000 7f55597e6740 20 client.25631 populate_metadata read hostname 'smithi016' 2024-03-28T22:11:31.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.584+0000 7f55597e6740 1 --2- 172.21.15.16:0/3696117971 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x560326cae410 0x560326cb0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:31.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.584+0000 7f55597e6740 4 client.25631 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:11:31.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.584+0000 7f55597e6740 1 -- 172.21.15.16:0/3696117971 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560326c053e0 con 0x560326cae410 2024-03-28T22:11:31.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.585+0000 7f55548c1640 1 --2- 172.21.15.16:0/3696117971 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x560326cae410 0x560326cb0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:31.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.585+0000 7f552ffff640 10 client.25631 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:11:31.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.585+0000 7f55548c1640 1 --2- 172.21.15.16:0/3696117971 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x560326cae410 0x560326cb0870 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=392c8b18d20ebe2c server_cookie=485f3685a46d63ec in_seq=0 out_seq=0 2024-03-28T22:11:31.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.586+0000 7f552ffff640 1 -- 172.21.15.16:0/3696117971 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560326c053e0 con 0x560326cae410 2024-03-28T22:11:31.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.586+0000 7f552ffff640 10 client.25631 handle_command_reply: tid=0 2024-03-28T22:11:31.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.586+0000 7f55597e6740 1 client.25631 shutdown 2024-03-28T22:11:31.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.586+0000 7f55597e6740 1 -- 172.21.15.16:0/3696117971 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560326c9bb30 msgr2=0x560326ca9fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:31.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.586+0000 7f55597e6740 1 --2- 172.21.15.16:0/3696117971 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560326c9bb30 0x560326ca9fd0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5548009a60 tx=0x7f554800b360 comp rx=0 tx=0).stop 2024-03-28T22:11:31.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.586+0000 7f55597e6740 1 -- 172.21.15.16:0/3696117971 shutdown_connections 2024-03-28T22:11:31.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.586+0000 7f55597e6740 1 --2- 172.21.15.16:0/3696117971 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x560326cae410 0x560326cb0870 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.586+0000 7f552ffff640 0 client.25631 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:11:31.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.586+0000 7f55597e6740 1 --2- 172.21.15.16:0/3696117971 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560326c9bb30 0x560326ca9fd0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.586+0000 7f55597e6740 1 --2- 172.21.15.16:0/3696117971 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560326c9def0 0x560326c9b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.586+0000 7f55597e6740 1 --2- 172.21.15.16:0/3696117971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560326c95310 0x560326c9b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.586+0000 7f55597e6740 1 -- 172.21.15.16:0/3696117971 >> 172.21.15.16:0/3696117971 conn(0x560326c8f120 msgr2=0x560326c90e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:11:31.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.586+0000 7f55597e6740 1 -- 172.21.15.16:0/3696117971 shutdown_connections 2024-03-28T22:11:31.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.586+0000 7f55597e6740 1 -- 172.21.15.16:0/3696117971 wait complete. 2024-03-28T22:11:31.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.587+0000 7f55597e6740 20 client.25631 trim_cache size 0 max 16384 2024-03-28T22:11:31.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.605+0000 7f55597e6740 1 Processor -- start 2024-03-28T22:11:31.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.605+0000 7f55597e6740 1 -- start start 2024-03-28T22:11:31.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.605+0000 7f55597e6740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560326c95310 0x560326cb6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:31.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.605+0000 7f55597e6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560326c9bb30 0x560326ca9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:31.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.605+0000 7f55597e6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560326c9def0 0x560326ca9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:31.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.605+0000 7f55597e6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f55501b1e80 con 0x560326c9def0 2024-03-28T22:11:31.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.605+0000 7f55597e6740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f55501af750 con 0x560326c95310 2024-03-28T22:11:31.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.605+0000 7f55597e6740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f55501ad5a0 con 0x560326c9bb30 2024-03-28T22:11:31.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.605+0000 7f55548c1640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560326c95310 0x560326cb6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:31.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.605+0000 7f55548c1640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560326c95310 0x560326cb6140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:53650/0 (socket says 172.21.15.16:53650) 2024-03-28T22:11:31.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.605+0000 7f55548c1640 1 -- 172.21.15.16:0/2274335990 learned_addr learned my addr 172.21.15.16:0/2274335990 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:11:31.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.605+0000 7f55550c2640 1 --2- 172.21.15.16:0/2274335990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560326c9def0 0x560326ca9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:31.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.605+0000 7f554ffff640 1 --2- 172.21.15.16:0/2274335990 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560326c9bb30 0x560326ca9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:31.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.606+0000 7f55548c1640 1 -- 172.21.15.16:0/2274335990 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560326c9bb30 msgr2=0x560326ca9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:31.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.606+0000 7f55548c1640 1 --2- 172.21.15.16:0/2274335990 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560326c9bb30 0x560326ca9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.606+0000 7f55548c1640 1 -- 172.21.15.16:0/2274335990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560326c9def0 msgr2=0x560326ca9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:31.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.606+0000 7f55548c1640 1 --2- 172.21.15.16:0/2274335990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560326c9def0 0x560326ca9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.606+0000 7f55548c1640 1 -- 172.21.15.16:0/2274335990 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560326c99600 con 0x560326c95310 2024-03-28T22:11:31.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.606+0000 7f55548c1640 1 --2- 172.21.15.16:0/2274335990 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560326c95310 0x560326cb6140 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f55440042d0 tx=0x7f55440096c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=619e8d105a3dcd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:11:31.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.606+0000 7f553a7fc640 1 -- 172.21.15.16:0/2274335990 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5544038020 con 0x560326c95310 2024-03-28T22:11:31.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.606+0000 7f553a7fc640 1 -- 172.21.15.16:0/2274335990 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5544004ff0 con 0x560326c95310 2024-03-28T22:11:31.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.606+0000 7f553a7fc640 1 -- 172.21.15.16:0/2274335990 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5544034350 con 0x560326c95310 2024-03-28T22:11:31.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.607+0000 7f55597e6740 1 -- 172.21.15.16:0/2274335990 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560326c95310 msgr2=0x560326cb6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:31.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.607+0000 7f55597e6740 1 --2- 172.21.15.16:0/2274335990 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560326c95310 0x560326cb6140 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f55440042d0 tx=0x7f55440096c0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.607+0000 7f55597e6740 1 -- 172.21.15.16:0/2274335990 shutdown_connections 2024-03-28T22:11:31.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.607+0000 7f55597e6740 1 --2- 172.21.15.16:0/2274335990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560326c9def0 0x560326ca9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.607+0000 7f55597e6740 1 --2- 172.21.15.16:0/2274335990 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560326c9bb30 0x560326ca9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.607+0000 7f55597e6740 1 --2- 172.21.15.16:0/2274335990 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560326c95310 0x560326cb6140 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.607+0000 7f55597e6740 1 -- 172.21.15.16:0/2274335990 >> 172.21.15.16:0/2274335990 conn(0x560326c8f120 msgr2=0x560326c8e180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:11:31.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.607+0000 7f55597e6740 1 -- 172.21.15.16:0/2274335990 shutdown_connections 2024-03-28T22:11:31.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.607+0000 7f55597e6740 1 -- 172.21.15.16:0/2274335990 wait complete. 2024-03-28T22:11:31.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.608+0000 7f55597e6740 1 -- start start 2024-03-28T22:11:31.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.608+0000 7f55597e6740 1 Processor -- start 2024-03-28T22:11:31.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.609+0000 7f55597e6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560326c95310 0x560326c97e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:31.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.609+0000 7f55597e6740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560326c9bb30 0x560326c983a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:31.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.609+0000 7f55597e6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560326c9def0 0x560326c988e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:31.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.609+0000 7f55597e6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f55501adc10 con 0x560326c9def0 2024-03-28T22:11:31.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.609+0000 7f55597e6740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f55501af080 con 0x560326c9bb30 2024-03-28T22:11:31.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.609+0000 7f554ffff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560326c9bb30 0x560326c983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:31.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.609+0000 7f554ffff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560326c9bb30 0x560326c983a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:53656/0 (socket says 172.21.15.16:53656) 2024-03-28T22:11:31.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.609+0000 7f554ffff640 1 -- 172.21.15.16:0/2083226515 learned_addr learned my addr 172.21.15.16:0/2083226515 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:11:31.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.609+0000 7f55597e6740 1 -- 172.21.15.16:0/2083226515 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f55501ae2e0 con 0x560326c95310 2024-03-28T22:11:31.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.609+0000 7f55550c2640 1 --2- 172.21.15.16:0/2083226515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560326c9def0 0x560326c988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:31.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.609+0000 7f55548c1640 1 --2- 172.21.15.16:0/2083226515 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560326c95310 0x560326c97e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:31.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.609+0000 7f55597e6740 10 client.0 _ll_register_callbacks cb 0x7f554800f490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:11:31.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.609+0000 7f554ffff640 1 -- 172.21.15.16:0/2083226515 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560326c95310 msgr2=0x560326c97e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:31.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.609+0000 7f554ffff640 1 --2- 172.21.15.16:0/2083226515 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560326c95310 0x560326c97e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.609+0000 7f554ffff640 1 -- 172.21.15.16:0/2083226515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560326c9def0 msgr2=0x560326c988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:31.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.609+0000 7f554ffff640 1 --2- 172.21.15.16:0/2083226515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560326c9def0 0x560326c988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.609+0000 7f554ffff640 1 -- 172.21.15.16:0/2083226515 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560326d13920 con 0x560326c9bb30 2024-03-28T22:11:31.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.609+0000 7f55548c1640 1 --2- 172.21.15.16:0/2083226515 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560326c95310 0x560326c97e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:11:31.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.610+0000 7f55550c2640 1 --2- 172.21.15.16:0/2083226515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560326c9def0 0x560326c988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:11:31.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.610+0000 7f554ffff640 1 --2- 172.21.15.16:0/2083226515 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560326c9bb30 0x560326c983a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f553c0b82d0 tx=0x7f553c0b8240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bbdf19bd78c61db server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:11:31.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.610+0000 7f552ffff640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-03-28T22:11:31.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.610+0000 7f552ffff640 1 -- 172.21.15.16:0/2083226515 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f553c0bbe70 con 0x560326c9bb30 2024-03-28T22:11:31.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.610+0000 7f552ffff640 1 -- 172.21.15.16:0/2083226515 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f553c032db0 con 0x560326c9bb30 2024-03-28T22:11:31.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.610+0000 7f55597e6740 1 -- 172.21.15.16:0/2083226515 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5544004ff0 con 0x560326c9bb30 2024-03-28T22:11:31.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.610+0000 7f552ffff640 1 -- 172.21.15.16:0/2083226515 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f553c0bffc0 con 0x560326c9bb30 2024-03-28T22:11:31.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.610+0000 7f55597e6740 10 client.25637 fetch_fsmap learned FSMap version 21 2024-03-28T22:11:31.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.610+0000 7f55597e6740 1 -- 172.21.15.16:0/2083226515 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x560326d14c60 con 0x560326c9bb30 2024-03-28T22:11:31.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.610+0000 7f552ffff640 1 -- 172.21.15.16:0/2083226515 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5544004ff0 con 0x560326c9bb30 2024-03-28T22:11:31.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.611+0000 7f552ffff640 1 -- 172.21.15.16:0/2083226515 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f553c0a1020 con 0x560326c9bb30 2024-03-28T22:11:31.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.611+0000 7f55597e6740 10 client.25637 fetch_fsmap finished waiting for FSMap version 21 2024-03-28T22:11:31.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.611+0000 7f55597e6740 10 client.25637 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:11:31.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.611+0000 7f55597e6740 20 client.25637 populate_metadata read hostname 'smithi016' 2024-03-28T22:11:31.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.611+0000 7f55597e6740 1 --2- 172.21.15.16:0/2083226515 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x560326cae410 0x560326d17260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:31.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.612+0000 7f55597e6740 4 client.25637 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:11:31.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.612+0000 7f55597e6740 1 -- 172.21.15.16:0/2083226515 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560326bff730 con 0x560326cae410 2024-03-28T22:11:31.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.612+0000 7f55548c1640 1 --2- 172.21.15.16:0/2083226515 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x560326cae410 0x560326d17260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:31.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.612+0000 7f552ffff640 10 client.25637 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:11:31.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.612+0000 7f55548c1640 1 --2- 172.21.15.16:0/2083226515 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x560326cae410 0x560326d17260 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=deaebd6e4975c259 server_cookie=12e4059ba4c8867a in_seq=0 out_seq=0 2024-03-28T22:11:31.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.613+0000 7f552ffff640 1 -- 172.21.15.16:0/2083226515 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560326bff730 con 0x560326cae410 2024-03-28T22:11:31.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.613+0000 7f552ffff640 10 client.25637 handle_command_reply: tid=0 2024-03-28T22:11:31.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.613+0000 7f55597e6740 1 client.25637 shutdown 2024-03-28T22:11:31.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.614+0000 7f55597e6740 1 -- 172.21.15.16:0/2083226515 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560326c9bb30 msgr2=0x560326c983a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:31.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.614+0000 7f55597e6740 1 --2- 172.21.15.16:0/2083226515 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560326c9bb30 0x560326c983a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f553c0b82d0 tx=0x7f553c0b8240 comp rx=0 tx=0).stop 2024-03-28T22:11:31.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.614+0000 7f55597e6740 1 -- 172.21.15.16:0/2083226515 shutdown_connections 2024-03-28T22:11:31.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.614+0000 7f55597e6740 1 --2- 172.21.15.16:0/2083226515 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x560326cae410 0x560326d17260 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.614+0000 7f552ffff640 0 client.25637 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:11:31.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.614+0000 7f55597e6740 1 --2- 172.21.15.16:0/2083226515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560326c9def0 0x560326c988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.614+0000 7f55597e6740 1 --2- 172.21.15.16:0/2083226515 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560326c9bb30 0x560326c983a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.614+0000 7f55597e6740 1 --2- 172.21.15.16:0/2083226515 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560326c95310 0x560326c97e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.614+0000 7f55597e6740 1 -- 172.21.15.16:0/2083226515 >> 172.21.15.16:0/2083226515 conn(0x560326c8f120 msgr2=0x560326c8d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:11:31.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.614+0000 7f55597e6740 1 -- 172.21.15.16:0/2083226515 shutdown_connections 2024-03-28T22:11:31.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.614+0000 7f55597e6740 1 -- 172.21.15.16:0/2083226515 wait complete. 2024-03-28T22:11:31.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.614+0000 7f55597e6740 20 client.25637 trim_cache size 0 max 16384 2024-03-28T22:11:31.621 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:11:31.621 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-03-28T22:11:31.621 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "b15856aa-cfee-4b46-83b1-ef12cf00fd04", 2024-03-28T22:11:31.621 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-03-28T22:11:31.621 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:11:31.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.615+0000 7f5556b25640 1 -- 172.21.15.16:0/1919400485 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f55400a2010 msgr2=0x7f55400a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:31.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.615+0000 7f5556b25640 1 --2- 172.21.15.16:0/1919400485 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f55400a2010 0x7f55400a44d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5550101730 tx=0x7f5544005f90 comp rx=0 tx=0).stop 2024-03-28T22:11:31.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.615+0000 7f5556b25640 1 -- 172.21.15.16:0/1919400485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5550100fb0 msgr2=0x7f5550107aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:31.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.615+0000 7f5556b25640 1 --2- 172.21.15.16:0/1919400485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5550100fb0 0x7f5550107aa0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f553c037040 tx=0x7f553c0045b0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.615+0000 7f5556b25640 1 -- 172.21.15.16:0/1919400485 shutdown_connections 2024-03-28T22:11:31.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.615+0000 7f5556b25640 1 --2- 172.21.15.16:0/1919400485 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f55400a2010 0x7f55400a44d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.615+0000 7f5556b25640 1 --2- 172.21.15.16:0/1919400485 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5550106690 0x7f5550107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.615+0000 7f5556b25640 1 --2- 172.21.15.16:0/1919400485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5550100fb0 0x7f5550107aa0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.615+0000 7f5556b25640 1 --2- 172.21.15.16:0/1919400485 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55501005f0 0x7f5550107560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.615+0000 7f5556b25640 1 -- 172.21.15.16:0/1919400485 >> 172.21.15.16:0/1919400485 conn(0x7f5550077800 msgr2=0x7f55500fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:11:31.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.615+0000 7f5556b25640 1 -- 172.21.15.16:0/1919400485 shutdown_connections 2024-03-28T22:11:31.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.615+0000 7f5556b25640 1 -- 172.21.15.16:0/1919400485 wait complete. 2024-03-28T22:11:31.629 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:11:31.629 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:11:31.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.733+0000 7f70ddbbc640 1 -- 172.21.15.16:0/3483237800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70d8100fb0 msgr2=0x7f70d810d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:31.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.733+0000 7f70ddbbc640 1 --2- 172.21.15.16:0/3483237800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70d8100fb0 0x7f70d810d4c0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f70d0009cd0 tx=0x7f70d002c0e0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.733+0000 7f70ddbbc640 1 -- 172.21.15.16:0/3483237800 shutdown_connections 2024-03-28T22:11:31.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.733+0000 7f70ddbbc640 1 --2- 172.21.15.16:0/3483237800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70d8100fb0 0x7f70d810d4c0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.733+0000 7f70ddbbc640 1 --2- 172.21.15.16:0/3483237800 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f70d81005f0 0x7f70d8100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.733+0000 7f70ddbbc640 1 --2- 172.21.15.16:0/3483237800 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70d8106690 0x7f70d8106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.733+0000 7f70ddbbc640 1 -- 172.21.15.16:0/3483237800 >> 172.21.15.16:0/3483237800 conn(0x7f70d8077800 msgr2=0x7f70d8077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:11:31.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.734+0000 7f70ddbbc640 1 -- 172.21.15.16:0/3483237800 shutdown_connections 2024-03-28T22:11:31.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.734+0000 7f70ddbbc640 1 -- 172.21.15.16:0/3483237800 wait complete. 2024-03-28T22:11:31.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.734+0000 7f70ddbbc640 1 Processor -- start 2024-03-28T22:11:31.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.735+0000 7f70ddbbc640 1 -- start start 2024-03-28T22:11:31.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.735+0000 7f70ddbbc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70d81005f0 0x7f70d8197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:31.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.735+0000 7f70ddbbc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70d8100fb0 0x7f70d8198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:31.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.735+0000 7f70ddbbc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f70d8106690 0x7f70d819c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:31.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.735+0000 7f70ddbbc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f70d8111b10 con 0x7f70d81005f0 2024-03-28T22:11:31.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.735+0000 7f70ddbbc640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f70d8111990 con 0x7f70d8100fb0 2024-03-28T22:11:31.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.735+0000 7f70ddbbc640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f70d8111c90 con 0x7f70d8106690 2024-03-28T22:11:31.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.735+0000 7f70d77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70d81005f0 0x7f70d8197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:31.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.735+0000 7f70d77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70d81005f0 0x7f70d8197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48702/0 (socket says 172.21.15.16:48702) 2024-03-28T22:11:31.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.735+0000 7f70d77fe640 1 -- 172.21.15.16:0/767382458 learned_addr learned my addr 172.21.15.16:0/767382458 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:11:31.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.735+0000 7f70d6ffd640 1 --2- 172.21.15.16:0/767382458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70d8100fb0 0x7f70d8198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:31.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.735+0000 7f70d7fff640 1 --2- 172.21.15.16:0/767382458 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f70d8106690 0x7f70d819c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:31.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.735+0000 7f70d77fe640 1 -- 172.21.15.16:0/767382458 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f70d8106690 msgr2=0x7f70d819c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:31.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.735+0000 7f70d77fe640 1 --2- 172.21.15.16:0/767382458 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f70d8106690 0x7f70d819c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.735+0000 7f70d77fe640 1 -- 172.21.15.16:0/767382458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70d8100fb0 msgr2=0x7f70d8198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:31.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.735+0000 7f70d77fe640 1 --2- 172.21.15.16:0/767382458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70d8100fb0 0x7f70d8198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.735+0000 7f70d77fe640 1 -- 172.21.15.16:0/767382458 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70d819cfe0 con 0x7f70d81005f0 2024-03-28T22:11:31.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.735+0000 7f70d6ffd640 1 --2- 172.21.15.16:0/767382458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70d8100fb0 0x7f70d8198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:11:31.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.736+0000 7f70d77fe640 1 --2- 172.21.15.16:0/767382458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70d81005f0 0x7f70d8197ff0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f70c800e9c0 tx=0x7f70c800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:11:31.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.736+0000 7f70d4ff9640 1 -- 172.21.15.16:0/767382458 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c800cde0 con 0x7f70d81005f0 2024-03-28T22:11:31.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.736+0000 7f70ddbbc640 1 -- 172.21.15.16:0/767382458 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70d819d2d0 con 0x7f70d81005f0 2024-03-28T22:11:31.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.736+0000 7f70ddbbc640 1 -- 172.21.15.16:0/767382458 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70d81a4b60 con 0x7f70d81005f0 2024-03-28T22:11:31.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.736+0000 7f70d4ff9640 1 -- 172.21.15.16:0/767382458 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70c8010910 con 0x7f70d81005f0 2024-03-28T22:11:31.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.736+0000 7f70d4ff9640 1 -- 172.21.15.16:0/767382458 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c8018a10 con 0x7f70d81005f0 2024-03-28T22:11:31.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.738+0000 7f70e079d740 1 Processor -- start 2024-03-28T22:11:31.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.738+0000 7f70e079d740 1 -- start start 2024-03-28T22:11:31.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.739+0000 7f70e079d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fbe313290 0x559fbe313670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:31.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.739+0000 7f70e079d740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559fbe313bb0 0x559fbe31c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:31.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.739+0000 7f70e079d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fbe31c790 0x559fbe31eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:31.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.739+0000 7f70e079d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559fbe277180 con 0x559fbe313290 2024-03-28T22:11:31.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.739+0000 7f70e079d740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x559fbe21fb70 con 0x559fbe313bb0 2024-03-28T22:11:31.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.739+0000 7f70e079d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x559fbe273220 con 0x559fbe31c790 2024-03-28T22:11:31.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.739+0000 7f70d7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fbe313290 0x559fbe313670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:31.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.739+0000 7f70d7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fbe313290 0x559fbe313670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48714/0 (socket says 172.21.15.16:48714) 2024-03-28T22:11:31.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.739+0000 7f70d7fff640 1 -- 172.21.15.16:0/3386328287 learned_addr learned my addr 172.21.15.16:0/3386328287 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:11:31.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.739+0000 7f70d6ffd640 1 --2- 172.21.15.16:0/3386328287 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fbe31c790 0x559fbe31eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:31.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.739+0000 7f70d7fff640 1 -- 172.21.15.16:0/3386328287 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fbe31c790 msgr2=0x559fbe31eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:31.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.739+0000 7f70d7fff640 1 --2- 172.21.15.16:0/3386328287 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fbe31c790 0x559fbe31eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.739+0000 7f70d7fff640 1 -- 172.21.15.16:0/3386328287 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559fbe313bb0 msgr2=0x559fbe31c250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:11:31.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.739+0000 7f70d7fff640 1 --2- 172.21.15.16:0/3386328287 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559fbe313bb0 0x559fbe31c250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.739+0000 7f70d7fff640 1 -- 172.21.15.16:0/3386328287 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559fbe2337c0 con 0x559fbe313290 2024-03-28T22:11:31.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.739+0000 7f70d7fff640 1 --2- 172.21.15.16:0/3386328287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fbe313290 0x559fbe313670 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f70d002c6f0 tx=0x7f70d00323d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd0d21455dd4fdb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:11:31.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.740+0000 7f70bdffb640 1 -- 172.21.15.16:0/3386328287 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d0043020 con 0x559fbe313290 2024-03-28T22:11:31.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.740+0000 7f70bdffb640 1 -- 172.21.15.16:0/3386328287 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70d0032c70 con 0x559fbe313290 2024-03-28T22:11:31.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.740+0000 7f70d4ff9640 1 -- 172.21.15.16:0/767382458 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 25) ==== 136989+0+0 (secure 0 0 0) 0x7f70c801f020 con 0x7f70d81005f0 2024-03-28T22:11:31.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.740+0000 7f70e079d740 1 -- 172.21.15.16:0/3386328287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fbe313290 msgr2=0x559fbe313670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:31.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.740+0000 7f70e079d740 1 --2- 172.21.15.16:0/3386328287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fbe313290 0x559fbe313670 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f70d002c6f0 tx=0x7f70d00323d0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.741+0000 7f70d4ff9640 1 --2- 172.21.15.16:0/767382458 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f70c00a2010 0x7f70c00a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:31.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.741+0000 7f70d7fff640 1 --2- 172.21.15.16:0/767382458 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f70c00a2010 0x7f70c00a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:31.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.741+0000 7f70d4ff9640 1 -- 172.21.15.16:0/767382458 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f70c80ce660 con 0x7f70d81005f0 2024-03-28T22:11:31.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.741+0000 7f70d7fff640 1 --2- 172.21.15.16:0/767382458 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f70c00a2010 0x7f70c00a44d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f70d0003de0 tx=0x7f70d0032e20 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:11:31.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.741+0000 7f70e079d740 1 -- 172.21.15.16:0/3386328287 shutdown_connections 2024-03-28T22:11:31.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.741+0000 7f70e079d740 1 --2- 172.21.15.16:0/3386328287 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fbe31c790 0x559fbe31eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.741+0000 7f70e079d740 1 --2- 172.21.15.16:0/3386328287 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559fbe313bb0 0x559fbe31c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.741+0000 7f70e079d740 1 --2- 172.21.15.16:0/3386328287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fbe313290 0x559fbe313670 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.741+0000 7f70e079d740 1 -- 172.21.15.16:0/3386328287 >> 172.21.15.16:0/3386328287 conn(0x559fbe30d9c0 msgr2=0x559fbe30dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:11:31.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.741+0000 7f70e079d740 1 -- 172.21.15.16:0/3386328287 shutdown_connections 2024-03-28T22:11:31.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.741+0000 7f70e079d740 1 -- 172.21.15.16:0/3386328287 wait complete. 2024-03-28T22:11:31.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.742+0000 7f70e079d740 1 -- start start 2024-03-28T22:11:31.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.742+0000 7f70e079d740 1 Processor -- start 2024-03-28T22:11:31.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.742+0000 7f70e079d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fbe313290 0x559fbe319700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:31.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.742+0000 7f70e079d740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559fbe313bb0 0x559fbe319c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:31.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.742+0000 7f70e079d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fbe31c790 0x559fbe31a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:31.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.742+0000 7f70e079d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559fbe31f5c0 con 0x559fbe313290 2024-03-28T22:11:31.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.742+0000 7f70e079d740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x559fbe2154a0 con 0x559fbe313bb0 2024-03-28T22:11:31.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.742+0000 7f70e079d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x559fbe31f740 con 0x559fbe31c790 2024-03-28T22:11:31.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.742+0000 7f70d77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fbe313290 0x559fbe319700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:31.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.742+0000 7f70d77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fbe313290 0x559fbe319700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48720/0 (socket says 172.21.15.16:48720) 2024-03-28T22:11:31.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.742+0000 7f70d77fe640 1 -- 172.21.15.16:0/2340701316 learned_addr learned my addr 172.21.15.16:0/2340701316 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:11:31.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.742+0000 7f70e079d740 10 client.0 _ll_register_callbacks cb 0x559fbe218d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:11:31.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.743+0000 7f70d6ffd640 1 --2- 172.21.15.16:0/2340701316 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559fbe313bb0 0x559fbe319c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:31.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.743+0000 7f70d7fff640 1 --2- 172.21.15.16:0/2340701316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fbe31c790 0x559fbe31a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:31.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.743+0000 7f70d77fe640 1 -- 172.21.15.16:0/2340701316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fbe31c790 msgr2=0x559fbe31a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:31.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.743+0000 7f70d77fe640 1 --2- 172.21.15.16:0/2340701316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fbe31c790 0x559fbe31a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.743+0000 7f70d77fe640 1 -- 172.21.15.16:0/2340701316 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559fbe313bb0 msgr2=0x559fbe319c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:31.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.743+0000 7f70d77fe640 1 --2- 172.21.15.16:0/2340701316 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559fbe313bb0 0x559fbe319c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.743+0000 7f70d77fe640 1 -- 172.21.15.16:0/2340701316 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559fbe0557d0 con 0x559fbe313290 2024-03-28T22:11:31.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.743+0000 7f70d77fe640 1 --2- 172.21.15.16:0/2340701316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fbe313290 0x559fbe319700 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f70c809ab20 tx=0x7f70c809a840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=516bc81b308f02eb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:11:31.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.743+0000 7f70b6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:11:31.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.743+0000 7f70b6ffd640 1 -- 172.21.15.16:0/2340701316 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c809ae20 con 0x559fbe313290 2024-03-28T22:11:31.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.743+0000 7f70b6ffd640 1 -- 172.21.15.16:0/2340701316 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f70c80a3130 con 0x559fbe313290 2024-03-28T22:11:31.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.743+0000 7f70e079d740 1 -- 172.21.15.16:0/2340701316 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70d0032c70 con 0x559fbe313290 2024-03-28T22:11:31.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.743+0000 7f70b6ffd640 1 -- 172.21.15.16:0/2340701316 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f70c809ef70 con 0x559fbe313290 2024-03-28T22:11:31.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.743+0000 7f70e079d740 10 client.15744 fetch_fsmap learned FSMap version 21 2024-03-28T22:11:31.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.743+0000 7f70e079d740 1 -- 172.21.15.16:0/2340701316 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x559fbdf8b3f0 con 0x559fbe313290 2024-03-28T22:11:31.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.744+0000 7f70b6ffd640 1 -- 172.21.15.16:0/2340701316 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70d0032c70 con 0x559fbe313290 2024-03-28T22:11:31.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.744+0000 7f70b6ffd640 1 -- 172.21.15.16:0/2340701316 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f70c8018bb0 con 0x559fbe313290 2024-03-28T22:11:31.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.744+0000 7f70e079d740 10 client.15744 fetch_fsmap finished waiting for FSMap version 21 2024-03-28T22:11:31.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.744+0000 7f70e079d740 10 client.15744 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:11:31.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.744+0000 7f70e079d740 20 client.15744 populate_metadata read hostname 'smithi016' 2024-03-28T22:11:31.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.744+0000 7f70e079d740 1 --2- 172.21.15.16:0/2340701316 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x559fbe32cbb0 0x559fbe32efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:31.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.744+0000 7f70e079d740 4 client.15744 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:11:31.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.744+0000 7f70e079d740 1 -- 172.21.15.16:0/2340701316 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559fbe2846c0 con 0x559fbe32cbb0 2024-03-28T22:11:31.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.744+0000 7f70d6ffd640 1 --2- 172.21.15.16:0/2340701316 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x559fbe32cbb0 0x559fbe32efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:31.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.745+0000 7f70b6ffd640 10 client.15744 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:11:31.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.745+0000 7f70d6ffd640 1 --2- 172.21.15.16:0/2340701316 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x559fbe32cbb0 0x559fbe32efa0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15dc2a7ffb642113 server_cookie=87d7b0ef23e5c8f8 in_seq=0 out_seq=0 2024-03-28T22:11:31.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.745+0000 7f70b6ffd640 1 -- 172.21.15.16:0/2340701316 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559fbe2846c0 con 0x559fbe32cbb0 2024-03-28T22:11:31.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.745+0000 7f70b6ffd640 10 client.15744 handle_command_reply: tid=0 2024-03-28T22:11:31.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.745+0000 7f70e079d740 1 client.15744 shutdown 2024-03-28T22:11:31.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.746+0000 7f70e079d740 1 -- 172.21.15.16:0/2340701316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fbe313290 msgr2=0x559fbe319700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:31.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.746+0000 7f70e079d740 1 --2- 172.21.15.16:0/2340701316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fbe313290 0x559fbe319700 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f70c809ab20 tx=0x7f70c809a840 comp rx=0 tx=0).stop 2024-03-28T22:11:31.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.746+0000 7f70e079d740 1 -- 172.21.15.16:0/2340701316 shutdown_connections 2024-03-28T22:11:31.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.746+0000 7f70e079d740 1 --2- 172.21.15.16:0/2340701316 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x559fbe32cbb0 0x559fbe32efa0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.746+0000 7f70e079d740 1 --2- 172.21.15.16:0/2340701316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fbe31c790 0x559fbe31a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.746+0000 7f70e079d740 1 --2- 172.21.15.16:0/2340701316 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559fbe313bb0 0x559fbe319c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.746+0000 7f70b6ffd640 0 client.15744 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:11:31.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.746+0000 7f70e079d740 1 --2- 172.21.15.16:0/2340701316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fbe313290 0x559fbe319700 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.746+0000 7f70e079d740 1 -- 172.21.15.16:0/2340701316 >> 172.21.15.16:0/2340701316 conn(0x559fbe30d9c0 msgr2=0x559fbe30cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:11:31.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.746+0000 7f70e079d740 1 -- 172.21.15.16:0/2340701316 shutdown_connections 2024-03-28T22:11:31.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.746+0000 7f70e079d740 1 -- 172.21.15.16:0/2340701316 wait complete. 2024-03-28T22:11:31.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.746+0000 7f70e079d740 20 client.15744 trim_cache size 0 max 16384 2024-03-28T22:11:31.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.763+0000 7f70e079d740 1 Processor -- start 2024-03-28T22:11:31.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.763+0000 7f70e079d740 1 -- start start 2024-03-28T22:11:31.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.764+0000 7f70e079d740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559fbe30d9c0 0x559fbe31be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:31.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.764+0000 7f70e079d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fbe313bb0 0x559fbe3283e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:31.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.764+0000 7f70e079d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fbe31c790 0x559fbe334800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:31.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.764+0000 7f70e079d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f70d81afba0 con 0x559fbe31c790 2024-03-28T22:11:31.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.764+0000 7f70e079d740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f70d81ad470 con 0x559fbe30d9c0 2024-03-28T22:11:31.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.764+0000 7f70e079d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f70d81ab2e0 con 0x559fbe313bb0 2024-03-28T22:11:31.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.764+0000 7f70d7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fbe31c790 0x559fbe334800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:31.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.764+0000 7f70d77fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559fbe30d9c0 0x559fbe31be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:31.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.764+0000 7f70d7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fbe31c790 0x559fbe334800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48734/0 (socket says 172.21.15.16:48734) 2024-03-28T22:11:31.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.764+0000 7f70d7fff640 1 -- 172.21.15.16:0/1512770067 learned_addr learned my addr 172.21.15.16:0/1512770067 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:11:31.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.764+0000 7f70d6ffd640 1 --2- 172.21.15.16:0/1512770067 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fbe313bb0 0x559fbe3283e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:31.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.764+0000 7f70d77fe640 1 -- 172.21.15.16:0/1512770067 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fbe313bb0 msgr2=0x559fbe3283e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:31.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.764+0000 7f70d77fe640 1 --2- 172.21.15.16:0/1512770067 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fbe313bb0 0x559fbe3283e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.764+0000 7f70d77fe640 1 -- 172.21.15.16:0/1512770067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fbe31c790 msgr2=0x559fbe334800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:31.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.764+0000 7f70d77fe640 1 --2- 172.21.15.16:0/1512770067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fbe31c790 0x559fbe334800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.764+0000 7f70d77fe640 1 -- 172.21.15.16:0/1512770067 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559fbdf4b0f0 con 0x559fbe30d9c0 2024-03-28T22:11:31.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.764+0000 7f70d7fff640 1 --2- 172.21.15.16:0/1512770067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fbe31c790 0x559fbe334800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:11:31.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.764+0000 7f70d6ffd640 1 --2- 172.21.15.16:0/1512770067 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fbe313bb0 0x559fbe3283e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:11:31.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.764+0000 7f70d77fe640 1 --2- 172.21.15.16:0/1512770067 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559fbe30d9c0 0x559fbe31be30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f70c80058c0 tx=0x7f70c809a8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a30ad665acee5d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:11:31.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.765+0000 7f70bd7fa640 1 -- 172.21.15.16:0/1512770067 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c807e070 con 0x559fbe30d9c0 2024-03-28T22:11:31.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.765+0000 7f70bd7fa640 1 -- 172.21.15.16:0/1512770067 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70c80b3070 con 0x559fbe30d9c0 2024-03-28T22:11:31.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.765+0000 7f70bd7fa640 1 -- 172.21.15.16:0/1512770067 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c80662e0 con 0x559fbe30d9c0 2024-03-28T22:11:31.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.765+0000 7f70e079d740 1 -- 172.21.15.16:0/1512770067 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559fbe30d9c0 msgr2=0x559fbe31be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:31.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.765+0000 7f70e079d740 1 --2- 172.21.15.16:0/1512770067 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559fbe30d9c0 0x559fbe31be30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f70c80058c0 tx=0x7f70c809a8b0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.765+0000 7f70e079d740 1 -- 172.21.15.16:0/1512770067 shutdown_connections 2024-03-28T22:11:31.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.765+0000 7f70e079d740 1 --2- 172.21.15.16:0/1512770067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fbe31c790 0x559fbe334800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.765+0000 7f70e079d740 1 --2- 172.21.15.16:0/1512770067 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fbe313bb0 0x559fbe3283e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.765+0000 7f70e079d740 1 --2- 172.21.15.16:0/1512770067 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559fbe30d9c0 0x559fbe31be30 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.765+0000 7f70e079d740 1 -- 172.21.15.16:0/1512770067 >> 172.21.15.16:0/1512770067 conn(0x559fbe318340 msgr2=0x559fbe30cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:11:31.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.765+0000 7f70e079d740 1 -- 172.21.15.16:0/1512770067 shutdown_connections 2024-03-28T22:11:31.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.765+0000 7f70e079d740 1 -- 172.21.15.16:0/1512770067 wait complete. 2024-03-28T22:11:31.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.766+0000 7f70e079d740 1 -- start start 2024-03-28T22:11:31.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.766+0000 7f70e079d740 1 Processor -- start 2024-03-28T22:11:31.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.766+0000 7f70e079d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fbe30d9c0 0x559fbe324a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:31.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.766+0000 7f70e079d740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559fbe313bb0 0x559fbe3219e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:31.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.766+0000 7f70e079d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fbe31c790 0x559fbe321f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:31.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.766+0000 7f70e079d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f70d81ab930 con 0x559fbe30d9c0 2024-03-28T22:11:31.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.766+0000 7f70e079d740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f70d81acda0 con 0x559fbe313bb0 2024-03-28T22:11:31.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.766+0000 7f70e079d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f70d81ac000 con 0x559fbe31c790 2024-03-28T22:11:31.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.766+0000 7f70d77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fbe30d9c0 0x559fbe324a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:31.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.766+0000 7f70e079d740 10 client.0 _ll_register_callbacks cb 0x7f70c8010e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:11:31.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.766+0000 7f70d6ffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559fbe313bb0 0x559fbe3219e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:31.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.766+0000 7f70d77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fbe30d9c0 0x559fbe324a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48750/0 (socket says 172.21.15.16:48750) 2024-03-28T22:11:31.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.766+0000 7f70d77fe640 1 -- 172.21.15.16:0/652969241 learned_addr learned my addr 172.21.15.16:0/652969241 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:11:31.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.766+0000 7f70d6ffd640 1 --2- 172.21.15.16:0/652969241 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559fbe313bb0 0x559fbe3219e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:53720/0 (socket says 172.21.15.16:53720) 2024-03-28T22:11:31.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.766+0000 7f70d7fff640 1 --2- 172.21.15.16:0/652969241 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fbe31c790 0x559fbe321f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:31.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.767+0000 7f70d6ffd640 1 -- 172.21.15.16:0/652969241 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fbe31c790 msgr2=0x559fbe321f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:31.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.767+0000 7f70d6ffd640 1 --2- 172.21.15.16:0/652969241 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fbe31c790 0x559fbe321f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.767+0000 7f70d6ffd640 1 -- 172.21.15.16:0/652969241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fbe30d9c0 msgr2=0x559fbe324a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:11:31.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.767+0000 7f70d6ffd640 1 --2- 172.21.15.16:0/652969241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fbe30d9c0 0x559fbe324a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.767+0000 7f70d6ffd640 1 -- 172.21.15.16:0/652969241 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559fbe322460 con 0x559fbe313bb0 2024-03-28T22:11:31.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.767+0000 7f70d6ffd640 1 --2- 172.21.15.16:0/652969241 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559fbe313bb0 0x559fbe3219e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x559fbe31acd0 tx=0x7f70cc0039c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21fef65e4bc5f757 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:11:31.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.767+0000 7f70b6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-03-28T22:11:31.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.767+0000 7f70b6ffd640 1 -- 172.21.15.16:0/652969241 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70cc020570 con 0x559fbe313bb0 2024-03-28T22:11:31.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.767+0000 7f70e079d740 1 -- 172.21.15.16:0/652969241 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70c80b3070 con 0x559fbe313bb0 2024-03-28T22:11:31.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.767+0000 7f70b6ffd640 1 -- 172.21.15.16:0/652969241 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f70cc020710 con 0x559fbe313bb0 2024-03-28T22:11:31.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.768+0000 7f70b6ffd640 1 -- 172.21.15.16:0/652969241 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f70cc02e050 con 0x559fbe313bb0 2024-03-28T22:11:31.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.768+0000 7f70e079d740 10 client.25652 fetch_fsmap learned FSMap version 21 2024-03-28T22:11:31.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.768+0000 7f70e079d740 1 -- 172.21.15.16:0/652969241 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x559fbe393b50 con 0x559fbe313bb0 2024-03-28T22:11:31.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.768+0000 7f70b6ffd640 1 -- 172.21.15.16:0/652969241 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70c80b3070 con 0x559fbe313bb0 2024-03-28T22:11:31.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.768+0000 7f70b6ffd640 1 -- 172.21.15.16:0/652969241 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f70cc029650 con 0x559fbe313bb0 2024-03-28T22:11:31.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.768+0000 7f70e079d740 10 client.25652 fetch_fsmap finished waiting for FSMap version 21 2024-03-28T22:11:31.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.768+0000 7f70e079d740 10 client.25652 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:11:31.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.768+0000 7f70e079d740 20 client.25652 populate_metadata read hostname 'smithi016' 2024-03-28T22:11:31.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.768+0000 7f70e079d740 1 --2- 172.21.15.16:0/652969241 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x559fbe3942d0 0x559fbe3966f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:11:31.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.768+0000 7f70e079d740 4 client.25652 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:11:31.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.768+0000 7f70e079d740 1 -- 172.21.15.16:0/652969241 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559fbe27ea10 con 0x559fbe3942d0 2024-03-28T22:11:31.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.768+0000 7f70d77fe640 1 --2- 172.21.15.16:0/652969241 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x559fbe3942d0 0x559fbe3966f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:11:31.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.768+0000 7f70b6ffd640 10 client.25652 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:11:31.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.769+0000 7f70d77fe640 1 --2- 172.21.15.16:0/652969241 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x559fbe3942d0 0x559fbe3966f0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=786415a68404395d server_cookie=a4b61e0f4e09137a in_seq=0 out_seq=0 2024-03-28T22:11:31.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.769+0000 7f70b6ffd640 1 -- 172.21.15.16:0/652969241 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x559fbe27ea10 con 0x559fbe3942d0 2024-03-28T22:11:31.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.769+0000 7f70b6ffd640 10 client.25652 handle_command_reply: tid=0 2024-03-28T22:11:31.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.769+0000 7f70e079d740 1 client.25652 shutdown 2024-03-28T22:11:31.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.769+0000 7f70e079d740 1 -- 172.21.15.16:0/652969241 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559fbe313bb0 msgr2=0x559fbe3219e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:31.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.769+0000 7f70e079d740 1 --2- 172.21.15.16:0/652969241 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559fbe313bb0 0x559fbe3219e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x559fbe31acd0 tx=0x7f70cc0039c0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.769+0000 7f70e079d740 1 -- 172.21.15.16:0/652969241 shutdown_connections 2024-03-28T22:11:31.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.769+0000 7f70e079d740 1 --2- 172.21.15.16:0/652969241 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x559fbe3942d0 0x559fbe3966f0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.769+0000 7f70e079d740 1 --2- 172.21.15.16:0/652969241 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fbe31c790 0x559fbe321f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.769+0000 7f70e079d740 1 --2- 172.21.15.16:0/652969241 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559fbe313bb0 0x559fbe3219e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.769+0000 7f70b6ffd640 0 client.25652 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:11:31.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.769+0000 7f70e079d740 1 --2- 172.21.15.16:0/652969241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559fbe30d9c0 0x559fbe324a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.769+0000 7f70e079d740 1 -- 172.21.15.16:0/652969241 >> 172.21.15.16:0/652969241 conn(0x559fbe318340 msgr2=0x559fbe30edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:11:31.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.769+0000 7f70e079d740 1 -- 172.21.15.16:0/652969241 shutdown_connections 2024-03-28T22:11:31.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.769+0000 7f70e079d740 1 -- 172.21.15.16:0/652969241 wait complete. 2024-03-28T22:11:31.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.769+0000 7f70e079d740 20 client.25652 trim_cache size 0 max 16384 2024-03-28T22:11:31.778 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:11:31.778 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T22:11:31.778 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-03-28T22:11:31.778 INFO:teuthology.orchestra.run.smithi016.stdout: "b15856aa-cfee-4b46-83b1-ef12cf00fd04": { 2024-03-28T22:11:31.778 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-03-28T22:11:31.779 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "b15856aa-cfee-4b46-83b1-ef12cf00fd04", 2024-03-28T22:11:31.779 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-03-28T22:11:31.779 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:11:31.779 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:11:31.779 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:11:31.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.770+0000 7f70ddbbc640 1 -- 172.21.15.16:0/767382458 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f70c00a2010 msgr2=0x7f70c00a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:31.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.770+0000 7f70ddbbc640 1 --2- 172.21.15.16:0/767382458 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f70c00a2010 0x7f70c00a44d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f70d0003de0 tx=0x7f70d0032e20 comp rx=0 tx=0).stop 2024-03-28T22:11:31.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.770+0000 7f70ddbbc640 1 -- 172.21.15.16:0/767382458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70d81005f0 msgr2=0x7f70d8197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:11:31.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.770+0000 7f70ddbbc640 1 --2- 172.21.15.16:0/767382458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70d81005f0 0x7f70d8197ff0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f70c800e9c0 tx=0x7f70c800ee80 comp rx=0 tx=0).stop 2024-03-28T22:11:31.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.770+0000 7f70ddbbc640 1 -- 172.21.15.16:0/767382458 shutdown_connections 2024-03-28T22:11:31.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.770+0000 7f70ddbbc640 1 --2- 172.21.15.16:0/767382458 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f70c00a2010 0x7f70c00a44d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.770+0000 7f70ddbbc640 1 --2- 172.21.15.16:0/767382458 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f70d8106690 0x7f70d819c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.770+0000 7f70ddbbc640 1 --2- 172.21.15.16:0/767382458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70d8100fb0 0x7f70d8198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.770+0000 7f70ddbbc640 1 --2- 172.21.15.16:0/767382458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70d81005f0 0x7f70d8197ff0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:11:31.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.770+0000 7f70ddbbc640 1 -- 172.21.15.16:0/767382458 >> 172.21.15.16:0/767382458 conn(0x7f70d8077800 msgr2=0x7f70d80ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:11:31.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.770+0000 7f70ddbbc640 1 -- 172.21.15.16:0/767382458 shutdown_connections 2024-03-28T22:11:31.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:11:31.770+0000 7f70ddbbc640 1 -- 172.21.15.16:0/767382458 wait complete. 2024-03-28T22:11:31.787 INFO:tasks.cephfs.filesystem:scrub status for tag:b15856aa-cfee-4b46-83b1-ef12cf00fd04 - {'path': '/', 'tag': 'b15856aa-cfee-4b46-83b1-ef12cf00fd04', 'options': 'recursive,force'} 2024-03-28T22:11:32.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:32 smithi053 ceph-mon[35979]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:11:32.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:32 smithi053 ceph-mon[35979]: scrub queued for path: / 2024-03-28T22:11:32.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:32 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:11:32.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:32 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:11:32.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:32 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:11:32.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:32 smithi063 ceph-mon[36249]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:11:32.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:32 smithi063 ceph-mon[36249]: scrub queued for path: / 2024-03-28T22:11:32.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:32 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:11:32.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:32 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:11:32.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:32 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:11:32.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:32 smithi016 ceph-mon[32374]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:11:32.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:32 smithi016 ceph-mon[32374]: scrub queued for path: / 2024-03-28T22:11:32.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:32 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:11:32.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:32 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:11:32.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:32 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:11:34.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:34 smithi053 ceph-mon[35979]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:11:34.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:34 smithi063 ceph-mon[36249]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:11:34.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:34 smithi016 ceph-mon[32374]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:11:35.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:35 smithi053 ceph-mon[35979]: scrub summary: idle 2024-03-28T22:11:35.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:35 smithi063 ceph-mon[36249]: scrub summary: idle 2024-03-28T22:11:35.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:35 smithi016 ceph-mon[32374]: scrub summary: idle 2024-03-28T22:11:36.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:36 smithi053 ceph-mon[35979]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:11:36.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:36 smithi063 ceph-mon[36249]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:11:36.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:36 smithi016 ceph-mon[32374]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:11:37.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:37 smithi063 ceph-mon[36249]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:11:37.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:37 smithi016 ceph-mon[32374]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:11:37.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:37 smithi053 ceph-mon[35979]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:11:39.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:39 smithi063 ceph-mon[36249]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:11:39.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:39 smithi016 ceph-mon[32374]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:11:39.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:39 smithi053 ceph-mon[35979]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:11:40.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:40 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:11:40.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:40 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:11:40.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:40 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:11:40.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:40 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:11:40.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:40 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:11:40.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:40 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:11:40.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:40 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:11:40.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:40 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:11:40.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:40 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:11:41.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:41 smithi063 ceph-mon[36249]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:11:41.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:41 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:11:41.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:41 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:11:41.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:41 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:11:41.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:41 smithi016 ceph-mon[32374]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:11:41.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:41 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:11:41.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:41 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:11:41.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:41 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:11:41.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:41 smithi053 ceph-mon[35979]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:11:41.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:41 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:11:41.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:41 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:11:41.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:41 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:11:43.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:43 smithi063 ceph-mon[36249]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:11:43.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:43 smithi016 ceph-mon[32374]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:11:43.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:43 smithi053 ceph-mon[35979]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:11:45.527 INFO:tasks.workunit.client.0.smithi016.stderr:Updating files: 98% (12278/12435) Updating files: 99% (12311/12435) Updating files: 100% (12435/12435) Updating files: 100% (12435/12435), done. 2024-03-28T22:11:45.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:45 smithi063 ceph-mon[36249]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:11:45.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:45 smithi016 ceph-mon[32374]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:11:45.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:45 smithi053 ceph-mon[35979]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:11:46.123 INFO:tasks.workunit.client.0.smithi016.stderr:Note: switching to '55f5138942aa4866e49329c9103468889cec3f73'. 2024-03-28T22:11:46.123 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-03-28T22:11:46.123 INFO:tasks.workunit.client.0.smithi016.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-28T22:11:46.123 INFO:tasks.workunit.client.0.smithi016.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-28T22:11:46.123 INFO:tasks.workunit.client.0.smithi016.stderr:state without impacting any branches by switching back to a branch. 2024-03-28T22:11:46.123 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-03-28T22:11:46.123 INFO:tasks.workunit.client.0.smithi016.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-28T22:11:46.124 INFO:tasks.workunit.client.0.smithi016.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-28T22:11:46.124 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-03-28T22:11:46.124 INFO:tasks.workunit.client.0.smithi016.stderr: git switch -c 2024-03-28T22:11:46.124 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-03-28T22:11:46.124 INFO:tasks.workunit.client.0.smithi016.stderr:Or undo this operation with: 2024-03-28T22:11:46.124 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-03-28T22:11:46.124 INFO:tasks.workunit.client.0.smithi016.stderr: git switch - 2024-03-28T22:11:46.124 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-03-28T22:11:46.124 INFO:tasks.workunit.client.0.smithi016.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-28T22:11:46.124 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-03-28T22:11:46.124 INFO:tasks.workunit.client.0.smithi016.stderr:HEAD is now at 55f5138942a Merge pull request #56314 from adk3798/wip-64621-squid 2024-03-28T22:11:46.127 DEBUG:teuthology.orchestra.run.smithi016:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-28T22:11:46.158 INFO:tasks.workunit.client.0.smithi016.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-28T22:11:46.160 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-28T22:11:46.161 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-28T22:11:46.877 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-28T22:11:46.934 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-28T22:11:46.975 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-28T22:11:46.976 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-28T22:11:46.976 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-28T22:11:47.014 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-28T22:11:47.017 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-28T22:11:47.018 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-28T22:11:47.073 INFO:tasks.workunit:Running workunits matching suites/fsx.sh on client.0... 2024-03-28T22:11:47.074 INFO:tasks.workunit:Running workunit suites/fsx.sh... 2024-03-28T22:11:47.074 DEBUG:teuthology.orchestra.run.smithi016:workunit test suites/fsx.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=55f5138942aa4866e49329c9103468889cec3f73 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh 2024-03-28T22:11:47.134 INFO:tasks.workunit.client.0.smithi016.stderr:+ set -e 2024-03-28T22:11:47.134 INFO:tasks.workunit.client.0.smithi016.stderr:+ git clone https://git.ceph.com/xfstests-dev.git 2024-03-28T22:11:47.136 INFO:tasks.workunit.client.0.smithi016.stderr:Cloning into 'xfstests-dev'... 2024-03-28T22:11:47.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:47 smithi063 ceph-mon[36249]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:11:47.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:47 smithi016 ceph-mon[32374]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:11:47.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:47 smithi053 ceph-mon[35979]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:11:49.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:49 smithi063 ceph-mon[36249]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:11:49.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:49 smithi016 ceph-mon[32374]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:11:49.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:49 smithi053 ceph-mon[35979]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:11:51.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:51 smithi063 ceph-mon[36249]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:11:51.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:51 smithi063 ceph-mon[36249]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T22:11:51.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:51 smithi016 ceph-mon[32374]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:11:51.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:51 smithi016 ceph-mon[32374]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T22:11:51.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:51 smithi053 ceph-mon[35979]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:11:51.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:51 smithi053 ceph-mon[35979]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T22:11:52.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:52 smithi063 ceph-mon[36249]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T22:11:52.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:52 smithi016 ceph-mon[32374]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T22:11:52.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:52 smithi053 ceph-mon[35979]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T22:11:53.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:53 smithi063 ceph-mon[36249]: pgmap v42: 97 pgs: 97 active+clean; 15 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 1.1 MiB/s wr, 12 op/s 2024-03-28T22:11:53.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:53 smithi016 ceph-mon[32374]: pgmap v42: 97 pgs: 97 active+clean; 15 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 1.1 MiB/s wr, 12 op/s 2024-03-28T22:11:53.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:53 smithi053 ceph-mon[35979]: pgmap v42: 97 pgs: 97 active+clean; 15 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 1.1 MiB/s wr, 12 op/s 2024-03-28T22:11:55.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:55 smithi053 ceph-mon[35979]: pgmap v43: 97 pgs: 97 active+clean; 15 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 1.1 MiB/s wr, 10 op/s 2024-03-28T22:11:55.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:55 smithi063 ceph-mon[36249]: pgmap v43: 97 pgs: 97 active+clean; 15 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 1.1 MiB/s wr, 10 op/s 2024-03-28T22:11:55.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:55 smithi016 ceph-mon[32374]: pgmap v43: 97 pgs: 97 active+clean; 15 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 1.1 MiB/s wr, 10 op/s 2024-03-28T22:11:56.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:56 smithi063 ceph-mon[36249]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T22:11:56.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:56 smithi016 ceph-mon[32374]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T22:11:56.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:56 smithi053 ceph-mon[35979]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T22:11:57.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:57 smithi016 ceph-mon[32374]: pgmap v44: 97 pgs: 97 active+clean; 16 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 1.3 MiB/s wr, 21 op/s 2024-03-28T22:11:57.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:57 smithi053 ceph-mon[35979]: pgmap v44: 97 pgs: 97 active+clean; 16 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 1.3 MiB/s wr, 21 op/s 2024-03-28T22:11:57.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:57 smithi063 ceph-mon[36249]: pgmap v44: 97 pgs: 97 active+clean; 16 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 1.3 MiB/s wr, 21 op/s 2024-03-28T22:11:59.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:11:59 smithi016 ceph-mon[32374]: pgmap v45: 97 pgs: 97 active+clean; 16 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 1.3 MiB/s wr, 19 op/s 2024-03-28T22:11:59.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:11:59 smithi053 ceph-mon[35979]: pgmap v45: 97 pgs: 97 active+clean; 16 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 1.3 MiB/s wr, 19 op/s 2024-03-28T22:11:59.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:11:59 smithi063 ceph-mon[36249]: pgmap v45: 97 pgs: 97 active+clean; 16 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 1.3 MiB/s wr, 19 op/s 2024-03-28T22:12:01.789 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:12:01.789 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:12:01.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:01 smithi053 ceph-mon[35979]: pgmap v46: 97 pgs: 97 active+clean; 16 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 1.3 MiB/s wr, 19 op/s 2024-03-28T22:12:01.818 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:01 smithi016 ceph-mon[32374]: pgmap v46: 97 pgs: 97 active+clean; 16 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 1.3 MiB/s wr, 19 op/s 2024-03-28T22:12:01.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.905+0000 7f08c4088640 1 -- 172.21.15.16:0/1633310804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08bc101ab0 msgr2=0x7f08bc101e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:01.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.905+0000 7f08c4088640 1 --2- 172.21.15.16:0/1633310804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08bc101ab0 0x7f08bc101e90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f08b8009930 tx=0x7f08b802bcd0 comp rx=0 tx=0).stop 2024-03-28T22:12:01.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.905+0000 7f08c4088640 1 -- 172.21.15.16:0/1633310804 shutdown_connections 2024-03-28T22:12:01.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.905+0000 7f08c4088640 1 --2- 172.21.15.16:0/1633310804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08bc103b90 0x7f08bc10e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:01.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.905+0000 7f08c4088640 1 --2- 172.21.15.16:0/1633310804 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08bc1031f0 0x7f08bc103650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:01.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.905+0000 7f08c4088640 1 --2- 172.21.15.16:0/1633310804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08bc101ab0 0x7f08bc101e90 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:01.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.905+0000 7f08c4088640 1 -- 172.21.15.16:0/1633310804 >> 172.21.15.16:0/1633310804 conn(0x7f08bc0fd1e0 msgr2=0x7f08bc0ff600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:01.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.906+0000 7f08c4088640 1 -- 172.21.15.16:0/1633310804 shutdown_connections 2024-03-28T22:12:01.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.906+0000 7f08c4088640 1 -- 172.21.15.16:0/1633310804 wait complete. 2024-03-28T22:12:01.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.906+0000 7f08c4088640 1 Processor -- start 2024-03-28T22:12:01.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.906+0000 7f08c4088640 1 -- start start 2024-03-28T22:12:01.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.906+0000 7f08c4088640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08bc101ab0 0x7f08bc198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:01.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.906+0000 7f08c4088640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08bc1031f0 0x7f08bc1985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:01.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.906+0000 7f08c4088640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08bc103b90 0x7f08bc19c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:01.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.906+0000 7f08c4088640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f08bc111b40 con 0x7f08bc1031f0 2024-03-28T22:12:01.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.906+0000 7f08c4088640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f08bc1119c0 con 0x7f08bc101ab0 2024-03-28T22:12:01.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.906+0000 7f08c4088640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f08bc111cc0 con 0x7f08bc103b90 2024-03-28T22:12:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.906+0000 7f08c1623640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08bc1031f0 0x7f08bc1985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.906+0000 7f08c1e24640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08bc101ab0 0x7f08bc198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.906+0000 7f08c1623640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08bc1031f0 0x7f08bc1985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50910/0 (socket says 172.21.15.16:50910) 2024-03-28T22:12:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.906+0000 7f08c1623640 1 -- 172.21.15.16:0/469639226 learned_addr learned my addr 172.21.15.16:0/469639226 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.907+0000 7f08c2625640 1 --2- 172.21.15.16:0/469639226 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08bc103b90 0x7f08bc19c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.907+0000 7f08c1623640 1 -- 172.21.15.16:0/469639226 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08bc103b90 msgr2=0x7f08bc19c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.907+0000 7f08c1623640 1 --2- 172.21.15.16:0/469639226 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08bc103b90 0x7f08bc19c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.907+0000 7f08c1623640 1 -- 172.21.15.16:0/469639226 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08bc101ab0 msgr2=0x7f08bc198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.907+0000 7f08c1623640 1 --2- 172.21.15.16:0/469639226 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08bc101ab0 0x7f08bc198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.907+0000 7f08c1623640 1 -- 172.21.15.16:0/469639226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08bc19d050 con 0x7f08bc1031f0 2024-03-28T22:12:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.907+0000 7f08c1e24640 1 --2- 172.21.15.16:0/469639226 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08bc101ab0 0x7f08bc198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:12:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.907+0000 7f08c1623640 1 --2- 172.21.15.16:0/469639226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08bc1031f0 0x7f08bc1985a0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f08ac00d770 tx=0x7f08ac00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.907+0000 7f08b2ffd640 1 -- 172.21.15.16:0/469639226 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ac017070 con 0x7f08bc1031f0 2024-03-28T22:12:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.907+0000 7f08c4088640 1 -- 172.21.15.16:0/469639226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08bc19d340 con 0x7f08bc1031f0 2024-03-28T22:12:01.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.907+0000 7f08b2ffd640 1 -- 172.21.15.16:0/469639226 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08ac005500 con 0x7f08bc1031f0 2024-03-28T22:12:01.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.907+0000 7f08b2ffd640 1 -- 172.21.15.16:0/469639226 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ac004e00 con 0x7f08bc1031f0 2024-03-28T22:12:01.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.907+0000 7f08c4088640 1 -- 172.21.15.16:0/469639226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08bc1a4bd0 con 0x7f08bc1031f0 2024-03-28T22:12:01.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.909+0000 7f08b2ffd640 1 -- 172.21.15.16:0/469639226 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 25) ==== 136989+0+0 (secure 0 0 0) 0x7f08ac004fa0 con 0x7f08bc1031f0 2024-03-28T22:12:01.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.910+0000 7f08c6c62740 1 Processor -- start 2024-03-28T22:12:01.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.910+0000 7f08c6c62740 1 -- start start 2024-03-28T22:12:01.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.910+0000 7f08b2ffd640 1 --2- 172.21.15.16:0/469639226 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f08a80a2010 0x7f08a80a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:01.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.910+0000 7f08c6c62740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555ee94bcfc0 0x555ee94bd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:01.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.910+0000 7f08c6c62740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ee94bd8e0 0x555ee94ca660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:01.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.910+0000 7f08c6c62740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555ee94caba0 0x555ee94ccf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:01.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.910+0000 7f08c6c62740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555ee9421180 con 0x555ee94bd8e0 2024-03-28T22:12:01.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.910+0000 7f08c6c62740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x555ee93c9b70 con 0x555ee94caba0 2024-03-28T22:12:01.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.910+0000 7f08c6c62740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x555ee941d220 con 0x555ee94bcfc0 2024-03-28T22:12:01.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.910+0000 7f08c1623640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ee94bd8e0 0x555ee94ca660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:01.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.910+0000 7f08b2ffd640 1 -- 172.21.15.16:0/469639226 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f08ac0cdee0 con 0x7f08bc1031f0 2024-03-28T22:12:01.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.910+0000 7f08c2625640 1 --2- 172.21.15.16:0/469639226 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f08a80a2010 0x7f08a80a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:01.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.910+0000 7f08c1623640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ee94bd8e0 0x555ee94ca660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50924/0 (socket says 172.21.15.16:50924) 2024-03-28T22:12:01.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.910+0000 7f08c1623640 1 -- 172.21.15.16:0/573770611 learned_addr learned my addr 172.21.15.16:0/573770611 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:01.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.910+0000 7f08c1e24640 1 --2- 172.21.15.16:0/573770611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555ee94bcfc0 0x555ee94bd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:01.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.910+0000 7f08c2625640 1 --2- 172.21.15.16:0/573770611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555ee94caba0 0x555ee94ccf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:01.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.911+0000 7f08c1623640 1 -- 172.21.15.16:0/573770611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555ee94bcfc0 msgr2=0x555ee94bd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:01.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.911+0000 7f08c1623640 1 --2- 172.21.15.16:0/573770611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555ee94bcfc0 0x555ee94bd3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:01.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.911+0000 7f08c1623640 1 -- 172.21.15.16:0/573770611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555ee94caba0 msgr2=0x555ee94ccf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:01.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.911+0000 7f08c1623640 1 --2- 172.21.15.16:0/573770611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555ee94caba0 0x555ee94ccf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:01.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.911+0000 7f08c1623640 1 -- 172.21.15.16:0/573770611 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ee93dd7c0 con 0x555ee94bd8e0 2024-03-28T22:12:01.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.911+0000 7f08c2625640 1 --2- 172.21.15.16:0/469639226 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f08a80a2010 0x7f08a80a44d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f08b4009fd0 tx=0x7f08b4009dc0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:01.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.911+0000 7f08c1623640 1 --2- 172.21.15.16:0/573770611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ee94bd8e0 0x555ee94ca660 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f08ac096aa0 tx=0x7f08ac099fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7e5cec4413f1c20 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:01.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.911+0000 7f08a7fff640 1 -- 172.21.15.16:0/573770611 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ac020020 con 0x555ee94bd8e0 2024-03-28T22:12:01.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.911+0000 7f08a7fff640 1 -- 172.21.15.16:0/573770611 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08ac09a640 con 0x555ee94bd8e0 2024-03-28T22:12:01.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.911+0000 7f08a7fff640 1 -- 172.21.15.16:0/573770611 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ac009cc0 con 0x555ee94bd8e0 2024-03-28T22:12:01.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.912+0000 7f08c6c62740 1 -- 172.21.15.16:0/573770611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ee94bd8e0 msgr2=0x555ee94ca660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:01.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.912+0000 7f08c6c62740 1 --2- 172.21.15.16:0/573770611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ee94bd8e0 0x555ee94ca660 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f08ac096aa0 tx=0x7f08ac099fa0 comp rx=0 tx=0).stop 2024-03-28T22:12:01.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.912+0000 7f08c6c62740 1 -- 172.21.15.16:0/573770611 shutdown_connections 2024-03-28T22:12:01.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.912+0000 7f08c6c62740 1 --2- 172.21.15.16:0/573770611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555ee94caba0 0x555ee94ccf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:01.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.912+0000 7f08c6c62740 1 --2- 172.21.15.16:0/573770611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ee94bd8e0 0x555ee94ca660 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:01.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.912+0000 7f08c6c62740 1 --2- 172.21.15.16:0/573770611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555ee94bcfc0 0x555ee94bd3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:01.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.912+0000 7f08c6c62740 1 -- 172.21.15.16:0/573770611 >> 172.21.15.16:0/573770611 conn(0x555ee94b79c0 msgr2=0x555ee94b7da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:01.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.912+0000 7f08c6c62740 1 -- 172.21.15.16:0/573770611 shutdown_connections 2024-03-28T22:12:01.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.912+0000 7f08c6c62740 1 -- 172.21.15.16:0/573770611 wait complete. 2024-03-28T22:12:01.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.912+0000 7f08c6c62740 1 -- start start 2024-03-28T22:12:01.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.912+0000 7f08c6c62740 1 Processor -- start 2024-03-28T22:12:01.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.913+0000 7f08c6c62740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555ee94bcfc0 0x555ee94c7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:01.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.913+0000 7f08c6c62740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ee94bd8e0 0x555ee94c8110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:01.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.913+0000 7f08c6c62740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555ee94caba0 0x555ee94c8650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:01.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.913+0000 7f08c6c62740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555ee94cd8a0 con 0x555ee94bd8e0 2024-03-28T22:12:01.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.913+0000 7f08c6c62740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x555ee93bf4a0 con 0x555ee94caba0 2024-03-28T22:12:01.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.913+0000 7f08c6c62740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x555ee94cda20 con 0x555ee94bcfc0 2024-03-28T22:12:01.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.913+0000 7f08c6c62740 10 client.0 _ll_register_callbacks cb 0x555ee93c2d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:12:01.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.913+0000 7f08c1623640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ee94bd8e0 0x555ee94c8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:01.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.913+0000 7f08c1623640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ee94bd8e0 0x555ee94c8110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50940/0 (socket says 172.21.15.16:50940) 2024-03-28T22:12:01.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.913+0000 7f08c1623640 1 -- 172.21.15.16:0/1744232531 learned_addr learned my addr 172.21.15.16:0/1744232531 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:01.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.913+0000 7f08c2625640 1 --2- 172.21.15.16:0/1744232531 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555ee94caba0 0x555ee94c8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:01.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.913+0000 7f08c1e24640 1 --2- 172.21.15.16:0/1744232531 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555ee94bcfc0 0x555ee94c7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:01.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.913+0000 7f08c1623640 1 -- 172.21.15.16:0/1744232531 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555ee94bcfc0 msgr2=0x555ee94c7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:01.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.913+0000 7f08c1623640 1 --2- 172.21.15.16:0/1744232531 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555ee94bcfc0 0x555ee94c7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:01.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.913+0000 7f08c1623640 1 -- 172.21.15.16:0/1744232531 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555ee94caba0 msgr2=0x555ee94c8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:01.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.913+0000 7f08c1623640 1 --2- 172.21.15.16:0/1744232531 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555ee94caba0 0x555ee94c8650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:01.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.913+0000 7f08c1623640 1 -- 172.21.15.16:0/1744232531 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ee91ff7d0 con 0x555ee94bd8e0 2024-03-28T22:12:01.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.913+0000 7f08c1623640 1 --2- 172.21.15.16:0/1744232531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ee94bd8e0 0x555ee94c8110 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f08ac09a5d0 tx=0x7f08ac00df90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c9767f41df84449 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:01.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.913+0000 7f08a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:12:01.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.913+0000 7f08a57fa640 1 -- 172.21.15.16:0/1744232531 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ac020070 con 0x555ee94bd8e0 2024-03-28T22:12:01.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.913+0000 7f08a57fa640 1 -- 172.21.15.16:0/1744232531 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f08ac096cb0 con 0x555ee94bd8e0 2024-03-28T22:12:01.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.913+0000 7f08c6c62740 1 -- 172.21.15.16:0/1744232531 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08ac09a640 con 0x555ee94bd8e0 2024-03-28T22:12:01.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.914+0000 7f08a57fa640 1 -- 172.21.15.16:0/1744232531 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f08ac09e790 con 0x555ee94bd8e0 2024-03-28T22:12:01.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.914+0000 7f08c6c62740 10 client.15762 fetch_fsmap learned FSMap version 24 2024-03-28T22:12:01.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.914+0000 7f08c6c62740 1 -- 172.21.15.16:0/1744232531 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x555ee91353f0 con 0x555ee94bd8e0 2024-03-28T22:12:01.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.914+0000 7f08a57fa640 1 -- 172.21.15.16:0/1744232531 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08ac09a640 con 0x555ee94bd8e0 2024-03-28T22:12:01.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.914+0000 7f08a57fa640 1 -- 172.21.15.16:0/1744232531 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7f08ac083b80 con 0x555ee94bd8e0 2024-03-28T22:12:01.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.914+0000 7f08c6c62740 10 client.15762 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:12:01.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.914+0000 7f08c6c62740 10 client.15762 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:12:01.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.914+0000 7f08c6c62740 20 client.15762 populate_metadata read hostname 'smithi016' 2024-03-28T22:12:01.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.914+0000 7f08c6c62740 1 --2- 172.21.15.16:0/1744232531 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x555ee94dad90 0x555ee94dd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:01.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.914+0000 7f08c6c62740 4 client.15762 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:12:01.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.914+0000 7f08c6c62740 1 -- 172.21.15.16:0/1744232531 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555ee942e6c0 con 0x555ee94dad90 2024-03-28T22:12:01.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.914+0000 7f08c1e24640 1 --2- 172.21.15.16:0/1744232531 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x555ee94dad90 0x555ee94dd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:01.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.915+0000 7f08a57fa640 10 client.15762 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:12:01.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.915+0000 7f08c1e24640 1 --2- 172.21.15.16:0/1744232531 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x555ee94dad90 0x555ee94dd180 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d8c8b3895ded9da server_cookie=f95e94747b52591a in_seq=0 out_seq=0 2024-03-28T22:12:01.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.915+0000 7f08a57fa640 1 -- 172.21.15.16:0/1744232531 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555ee942e6c0 con 0x555ee94dad90 2024-03-28T22:12:01.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.915+0000 7f08a57fa640 10 client.15762 handle_command_reply: tid=0 2024-03-28T22:12:01.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.915+0000 7f08c6c62740 1 client.15762 shutdown 2024-03-28T22:12:01.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.916+0000 7f08c6c62740 1 -- 172.21.15.16:0/1744232531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ee94bd8e0 msgr2=0x555ee94c8110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:01.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.916+0000 7f08c6c62740 1 --2- 172.21.15.16:0/1744232531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ee94bd8e0 0x555ee94c8110 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f08ac09a5d0 tx=0x7f08ac00df90 comp rx=0 tx=0).stop 2024-03-28T22:12:01.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.916+0000 7f08c6c62740 1 -- 172.21.15.16:0/1744232531 shutdown_connections 2024-03-28T22:12:01.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.916+0000 7f08c6c62740 1 --2- 172.21.15.16:0/1744232531 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x555ee94dad90 0x555ee94dd180 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:01.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.916+0000 7f08c6c62740 1 --2- 172.21.15.16:0/1744232531 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555ee94caba0 0x555ee94c8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:01.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.916+0000 7f08c6c62740 1 --2- 172.21.15.16:0/1744232531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ee94bd8e0 0x555ee94c8110 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:01.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.916+0000 7f08a57fa640 0 client.15762 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:12:01.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.916+0000 7f08c6c62740 1 --2- 172.21.15.16:0/1744232531 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555ee94bcfc0 0x555ee94c7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:01.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.916+0000 7f08c6c62740 1 -- 172.21.15.16:0/1744232531 >> 172.21.15.16:0/1744232531 conn(0x555ee94b79c0 msgr2=0x555ee94b9710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:01.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.916+0000 7f08c6c62740 1 -- 172.21.15.16:0/1744232531 shutdown_connections 2024-03-28T22:12:01.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.916+0000 7f08c6c62740 1 -- 172.21.15.16:0/1744232531 wait complete. 2024-03-28T22:12:01.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.916+0000 7f08c6c62740 20 client.15762 trim_cache size 0 max 16384 2024-03-28T22:12:01.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:01 smithi063 ceph-mon[36249]: pgmap v46: 97 pgs: 97 active+clean; 16 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 1.3 MiB/s wr, 19 op/s 2024-03-28T22:12:01.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.934+0000 7f08c6c62740 1 Processor -- start 2024-03-28T22:12:01.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.934+0000 7f08c6c62740 1 -- start start 2024-03-28T22:12:01.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.934+0000 7f08c6c62740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555ee94bcfc0 0x555ee94b6c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:01.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.934+0000 7f08c6c62740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ee94bd8e0 0x555ee94e01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:01.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.934+0000 7f08c6c62740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555ee94caba0 0x555ee94c4e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:01.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.934+0000 7f08c6c62740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f08bc1ad9e0 con 0x555ee94bd8e0 2024-03-28T22:12:01.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.934+0000 7f08c6c62740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f08bc1ab280 con 0x555ee94caba0 2024-03-28T22:12:01.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.934+0000 7f08c6c62740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f08bc1a90d0 con 0x555ee94bcfc0 2024-03-28T22:12:01.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.934+0000 7f08c1623640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ee94bd8e0 0x555ee94e01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:01.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.934+0000 7f08c1623640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ee94bd8e0 0x555ee94e01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50954/0 (socket says 172.21.15.16:50954) 2024-03-28T22:12:01.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.934+0000 7f08c1623640 1 -- 172.21.15.16:0/1889294565 learned_addr learned my addr 172.21.15.16:0/1889294565 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:01.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.934+0000 7f08c2625640 1 --2- 172.21.15.16:0/1889294565 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555ee94caba0 0x555ee94c4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:01.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.934+0000 7f08c1e24640 1 --2- 172.21.15.16:0/1889294565 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555ee94bcfc0 0x555ee94b6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:01.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.934+0000 7f08c1623640 1 -- 172.21.15.16:0/1889294565 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555ee94bcfc0 msgr2=0x555ee94b6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:01.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.934+0000 7f08c1623640 1 --2- 172.21.15.16:0/1889294565 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555ee94bcfc0 0x555ee94b6c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:01.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.934+0000 7f08c1623640 1 -- 172.21.15.16:0/1889294565 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555ee94caba0 msgr2=0x555ee94c4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:01.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.934+0000 7f08c1623640 1 --2- 172.21.15.16:0/1889294565 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555ee94caba0 0x555ee94c4e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:01.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.934+0000 7f08c1623640 1 -- 172.21.15.16:0/1889294565 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ee90f50f0 con 0x555ee94bd8e0 2024-03-28T22:12:01.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.934+0000 7f08c1e24640 1 --2- 172.21.15.16:0/1889294565 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555ee94bcfc0 0x555ee94b6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:12:01.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.935+0000 7f08c1623640 1 --2- 172.21.15.16:0/1889294565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ee94bd8e0 0x555ee94e01a0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f08ac00d740 tx=0x7f08ac0b3240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6567a27dd81ce109 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:01.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.935+0000 7f08a77fe640 1 -- 172.21.15.16:0/1889294565 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ac020030 con 0x555ee94bd8e0 2024-03-28T22:12:01.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.935+0000 7f08a77fe640 1 -- 172.21.15.16:0/1889294565 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08ac09ad40 con 0x555ee94bd8e0 2024-03-28T22:12:01.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.935+0000 7f08a77fe640 1 -- 172.21.15.16:0/1889294565 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ac009cc0 con 0x555ee94bd8e0 2024-03-28T22:12:01.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.935+0000 7f08c6c62740 1 -- 172.21.15.16:0/1889294565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ee94bd8e0 msgr2=0x555ee94e01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:01.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.935+0000 7f08c6c62740 1 --2- 172.21.15.16:0/1889294565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ee94bd8e0 0x555ee94e01a0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f08ac00d740 tx=0x7f08ac0b3240 comp rx=0 tx=0).stop 2024-03-28T22:12:01.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.935+0000 7f08c6c62740 1 -- 172.21.15.16:0/1889294565 shutdown_connections 2024-03-28T22:12:01.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.935+0000 7f08c6c62740 1 --2- 172.21.15.16:0/1889294565 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555ee94caba0 0x555ee94c4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:01.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.935+0000 7f08c6c62740 1 --2- 172.21.15.16:0/1889294565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ee94bd8e0 0x555ee94e01a0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:01.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.935+0000 7f08c6c62740 1 --2- 172.21.15.16:0/1889294565 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555ee94bcfc0 0x555ee94b6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:01.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.935+0000 7f08c6c62740 1 -- 172.21.15.16:0/1889294565 >> 172.21.15.16:0/1889294565 conn(0x555ee94b79c0 msgr2=0x555ee94c6b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:01.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.936+0000 7f08c6c62740 1 -- 172.21.15.16:0/1889294565 shutdown_connections 2024-03-28T22:12:01.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.936+0000 7f08c6c62740 1 -- 172.21.15.16:0/1889294565 wait complete. 2024-03-28T22:12:01.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.936+0000 7f08c6c62740 1 -- start start 2024-03-28T22:12:01.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.936+0000 7f08c6c62740 1 Processor -- start 2024-03-28T22:12:01.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.936+0000 7f08c6c62740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555ee94bcfc0 0x555ee94ddc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:01.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.936+0000 7f08c6c62740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ee94bd8e0 0x555ee94de160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:01.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.936+0000 7f08c6c62740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555ee94caba0 0x555ee94de6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:01.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.936+0000 7f08c6c62740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f08bc1a9740 con 0x555ee94bd8e0 2024-03-28T22:12:01.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.936+0000 7f08c6c62740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f08bc1aabb0 con 0x555ee94caba0 2024-03-28T22:12:01.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.936+0000 7f08c6c62740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f08bc1a9e10 con 0x555ee94bcfc0 2024-03-28T22:12:01.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.936+0000 7f08c6c62740 10 client.0 _ll_register_callbacks cb 0x7f08b4012770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:12:01.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.936+0000 7f08c1623640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ee94bd8e0 0x555ee94de160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:01.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.936+0000 7f08c1623640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ee94bd8e0 0x555ee94de160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50962/0 (socket says 172.21.15.16:50962) 2024-03-28T22:12:01.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.936+0000 7f08c1623640 1 -- 172.21.15.16:0/2044612577 learned_addr learned my addr 172.21.15.16:0/2044612577 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:01.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.937+0000 7f08c2625640 1 --2- 172.21.15.16:0/2044612577 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555ee94caba0 0x555ee94de6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:01.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.937+0000 7f08c1e24640 1 --2- 172.21.15.16:0/2044612577 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555ee94bcfc0 0x555ee94ddc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:01.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.937+0000 7f08c1623640 1 -- 172.21.15.16:0/2044612577 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555ee94bcfc0 msgr2=0x555ee94ddc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:01.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.937+0000 7f08c1623640 1 --2- 172.21.15.16:0/2044612577 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555ee94bcfc0 0x555ee94ddc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:01.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.937+0000 7f08c1623640 1 -- 172.21.15.16:0/2044612577 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555ee94caba0 msgr2=0x555ee94de6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:01.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.937+0000 7f08c1623640 1 --2- 172.21.15.16:0/2044612577 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555ee94caba0 0x555ee94de6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:01.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.937+0000 7f08c1623640 1 -- 172.21.15.16:0/2044612577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ee953fe90 con 0x555ee94bd8e0 2024-03-28T22:12:01.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.937+0000 7f08c2625640 1 --2- 172.21.15.16:0/2044612577 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555ee94caba0 0x555ee94de6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:12:01.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.937+0000 7f08c1623640 1 --2- 172.21.15.16:0/2044612577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ee94bd8e0 0x555ee94de160 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f08ac09a5d0 tx=0x7f08ac0cb750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e8c0db5ff8286c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:01.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.937+0000 7f08a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:12:01.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.937+0000 7f08a57fa640 1 -- 172.21.15.16:0/2044612577 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ac020070 con 0x555ee94bd8e0 2024-03-28T22:12:01.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.937+0000 7f08a57fa640 1 -- 172.21.15.16:0/2044612577 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f08ac083b80 con 0x555ee94bd8e0 2024-03-28T22:12:01.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.937+0000 7f08c6c62740 1 -- 172.21.15.16:0/2044612577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08ac09ad40 con 0x555ee94bd8e0 2024-03-28T22:12:01.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.937+0000 7f08a57fa640 1 -- 172.21.15.16:0/2044612577 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f08ac09a640 con 0x555ee94bd8e0 2024-03-28T22:12:01.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.937+0000 7f08c6c62740 10 client.15768 fetch_fsmap learned FSMap version 24 2024-03-28T22:12:01.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.937+0000 7f08c6c62740 1 -- 172.21.15.16:0/2044612577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x555ee953fe90 con 0x555ee94bd8e0 2024-03-28T22:12:01.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.937+0000 7f08a57fa640 1 -- 172.21.15.16:0/2044612577 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08ac09ad40 con 0x555ee94bd8e0 2024-03-28T22:12:01.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.938+0000 7f08a57fa640 1 -- 172.21.15.16:0/2044612577 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7f08ac099a90 con 0x555ee94bd8e0 2024-03-28T22:12:01.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.938+0000 7f08c6c62740 10 client.15768 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:12:01.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.938+0000 7f08c6c62740 10 client.15768 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:12:01.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.938+0000 7f08c6c62740 20 client.15768 populate_metadata read hostname 'smithi016' 2024-03-28T22:12:01.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.938+0000 7f08c6c62740 1 --2- 172.21.15.16:0/2044612577 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x555ee95414e0 0x555ee95438d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:01.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.938+0000 7f08c6c62740 4 client.15768 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:12:01.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.938+0000 7f08c6c62740 1 -- 172.21.15.16:0/2044612577 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555ee9428a10 con 0x555ee95414e0 2024-03-28T22:12:01.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.938+0000 7f08c1e24640 1 --2- 172.21.15.16:0/2044612577 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x555ee95414e0 0x555ee95438d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:01.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.938+0000 7f08a57fa640 10 client.15768 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:12:01.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.938+0000 7f08c1e24640 1 --2- 172.21.15.16:0/2044612577 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x555ee95414e0 0x555ee95438d0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d2aa8bf42b1329d server_cookie=4c9098ba9ff4ea8b in_seq=0 out_seq=0 2024-03-28T22:12:01.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.938+0000 7f08a57fa640 1 -- 172.21.15.16:0/2044612577 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x555ee9428a10 con 0x555ee95414e0 2024-03-28T22:12:01.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.938+0000 7f08a57fa640 10 client.15768 handle_command_reply: tid=0 2024-03-28T22:12:01.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.938+0000 7f08c6c62740 1 client.15768 shutdown 2024-03-28T22:12:01.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.938+0000 7f08c6c62740 1 -- 172.21.15.16:0/2044612577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ee94bd8e0 msgr2=0x555ee94de160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:01.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.938+0000 7f08c6c62740 1 --2- 172.21.15.16:0/2044612577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ee94bd8e0 0x555ee94de160 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f08ac09a5d0 tx=0x7f08ac0cb750 comp rx=0 tx=0).stop 2024-03-28T22:12:01.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.939+0000 7f08c6c62740 1 -- 172.21.15.16:0/2044612577 shutdown_connections 2024-03-28T22:12:01.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.939+0000 7f08c6c62740 1 --2- 172.21.15.16:0/2044612577 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x555ee95414e0 0x555ee95438d0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:01.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.939+0000 7f08c6c62740 1 --2- 172.21.15.16:0/2044612577 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555ee94caba0 0x555ee94de6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:01.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.939+0000 7f08c6c62740 1 --2- 172.21.15.16:0/2044612577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ee94bd8e0 0x555ee94de160 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:01.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.939+0000 7f08c6c62740 1 --2- 172.21.15.16:0/2044612577 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555ee94bcfc0 0x555ee94ddc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:01.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.939+0000 7f08a57fa640 0 client.15768 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:12:01.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.939+0000 7f08c6c62740 1 -- 172.21.15.16:0/2044612577 >> 172.21.15.16:0/2044612577 conn(0x555ee94b79c0 msgr2=0x555ee94d6540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:01.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.939+0000 7f08c6c62740 1 -- 172.21.15.16:0/2044612577 shutdown_connections 2024-03-28T22:12:01.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.939+0000 7f08c6c62740 1 -- 172.21.15.16:0/2044612577 wait complete. 2024-03-28T22:12:01.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.939+0000 7f08c6c62740 20 client.15768 trim_cache size 0 max 16384 2024-03-28T22:12:01.947 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:12:01.947 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-03-28T22:12:01.947 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-03-28T22:12:01.947 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:12:01.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.939+0000 7f08c4088640 1 -- 172.21.15.16:0/469639226 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f08a80a2010 msgr2=0x7f08a80a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:01.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.939+0000 7f08c4088640 1 --2- 172.21.15.16:0/469639226 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f08a80a2010 0x7f08a80a44d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f08b4009fd0 tx=0x7f08b4009dc0 comp rx=0 tx=0).stop 2024-03-28T22:12:01.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.939+0000 7f08c4088640 1 -- 172.21.15.16:0/469639226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08bc1031f0 msgr2=0x7f08bc1985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:01.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.939+0000 7f08c4088640 1 --2- 172.21.15.16:0/469639226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08bc1031f0 0x7f08bc1985a0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f08ac00d770 tx=0x7f08ac00dc30 comp rx=0 tx=0).stop 2024-03-28T22:12:01.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.939+0000 7f08c4088640 1 -- 172.21.15.16:0/469639226 shutdown_connections 2024-03-28T22:12:01.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.939+0000 7f08c4088640 1 --2- 172.21.15.16:0/469639226 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f08a80a2010 0x7f08a80a44d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:01.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.939+0000 7f08c4088640 1 --2- 172.21.15.16:0/469639226 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08bc103b90 0x7f08bc19c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:01.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.939+0000 7f08c4088640 1 --2- 172.21.15.16:0/469639226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08bc1031f0 0x7f08bc1985a0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:01.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.939+0000 7f08c4088640 1 --2- 172.21.15.16:0/469639226 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08bc101ab0 0x7f08bc198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:01.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.939+0000 7f08c4088640 1 -- 172.21.15.16:0/469639226 >> 172.21.15.16:0/469639226 conn(0x7f08bc0fd1e0 msgr2=0x7f08bc105600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:01.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.939+0000 7f08c4088640 1 -- 172.21.15.16:0/469639226 shutdown_connections 2024-03-28T22:12:01.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:01.939+0000 7f08c4088640 1 -- 172.21.15.16:0/469639226 wait complete. 2024-03-28T22:12:01.952 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:12:01.952 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:12:02.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.094+0000 7f28640c9640 1 -- 172.21.15.16:0/2855166191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f285c103290 msgr2=0x7f285c10b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:02.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.094+0000 7f28640c9640 1 --2- 172.21.15.16:0/2855166191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f285c103290 0x7f285c10b770 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f2850009900 tx=0x7f285002bcd0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.094+0000 7f28640c9640 1 -- 172.21.15.16:0/2855166191 shutdown_connections 2024-03-28T22:12:02.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.094+0000 7f28640c9640 1 --2- 172.21.15.16:0/2855166191 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f285c10bd40 0x7f285c10e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.094+0000 7f28640c9640 1 --2- 172.21.15.16:0/2855166191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f285c103290 0x7f285c10b770 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.094+0000 7f28640c9640 1 --2- 172.21.15.16:0/2855166191 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f285c102970 0x7f285c102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.094+0000 7f28640c9640 1 -- 172.21.15.16:0/2855166191 >> 172.21.15.16:0/2855166191 conn(0x7f285c0777d0 msgr2=0x7f285c077be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.095+0000 7f28640c9640 1 -- 172.21.15.16:0/2855166191 shutdown_connections 2024-03-28T22:12:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.095+0000 7f28640c9640 1 -- 172.21.15.16:0/2855166191 wait complete. 2024-03-28T22:12:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.095+0000 7f28640c9640 1 Processor -- start 2024-03-28T22:12:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.095+0000 7f28640c9640 1 -- start start 2024-03-28T22:12:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.095+0000 7f28640c9640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f285c102970 0x7f285c1005c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.095+0000 7f28640c9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f285c103290 0x7f285c100b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.095+0000 7f28640c9640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f285c10bd40 0x7f285c101040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.095+0000 7f28640c9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f285c111aa0 con 0x7f285c103290 2024-03-28T22:12:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.095+0000 7f28640c9640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f285c111920 con 0x7f285c10bd40 2024-03-28T22:12:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.095+0000 7f28640c9640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f285c111c20 con 0x7f285c102970 2024-03-28T22:12:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.095+0000 7f2861664640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f285c103290 0x7f285c100b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.095+0000 7f2861664640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f285c103290 0x7f285c100b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50984/0 (socket says 172.21.15.16:50984) 2024-03-28T22:12:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.095+0000 7f2861664640 1 -- 172.21.15.16:0/2870844085 learned_addr learned my addr 172.21.15.16:0/2870844085 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.095+0000 7f2862666640 1 --2- 172.21.15.16:0/2870844085 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f285c10bd40 0x7f285c101040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.096+0000 7f2861e65640 1 --2- 172.21.15.16:0/2870844085 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f285c102970 0x7f285c1005c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.096+0000 7f2861664640 1 -- 172.21.15.16:0/2870844085 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f285c102970 msgr2=0x7f285c1005c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.096+0000 7f2861664640 1 --2- 172.21.15.16:0/2870844085 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f285c102970 0x7f285c1005c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.096+0000 7f2861664640 1 -- 172.21.15.16:0/2870844085 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f285c10bd40 msgr2=0x7f285c101040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.096+0000 7f2861664640 1 --2- 172.21.15.16:0/2870844085 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f285c10bd40 0x7f285c101040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.096+0000 7f2861664640 1 -- 172.21.15.16:0/2870844085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f285c1a6440 con 0x7f285c103290 2024-03-28T22:12:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.096+0000 7f2861e65640 1 --2- 172.21.15.16:0/2870844085 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f285c102970 0x7f285c1005c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:12:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.096+0000 7f2862666640 1 --2- 172.21.15.16:0/2870844085 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f285c10bd40 0x7f285c101040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:12:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.096+0000 7f2861664640 1 --2- 172.21.15.16:0/2870844085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f285c103290 0x7f285c100b00 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f2850005940 tx=0x7f285002cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.096+0000 7f285affd640 1 -- 172.21.15.16:0/2870844085 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2850041070 con 0x7f285c103290 2024-03-28T22:12:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.096+0000 7f285affd640 1 -- 172.21.15.16:0/2870844085 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2850004450 con 0x7f285c103290 2024-03-28T22:12:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.096+0000 7f28640c9640 1 -- 172.21.15.16:0/2870844085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f285c1a66d0 con 0x7f285c103290 2024-03-28T22:12:02.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.096+0000 7f285affd640 1 -- 172.21.15.16:0/2870844085 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2850003df0 con 0x7f285c103290 2024-03-28T22:12:02.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.096+0000 7f28640c9640 1 -- 172.21.15.16:0/2870844085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f285c1a6c10 con 0x7f285c103290 2024-03-28T22:12:02.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.097+0000 7f28640c9640 1 -- 172.21.15.16:0/2870844085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f285c0685d0 con 0x7f285c103290 2024-03-28T22:12:02.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.098+0000 7f285affd640 1 -- 172.21.15.16:0/2870844085 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 25) ==== 136989+0+0 (secure 0 0 0) 0x7f2850044070 con 0x7f285c103290 2024-03-28T22:12:02.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.099+0000 7f285affd640 1 --2- 172.21.15.16:0/2870844085 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f28440a2090 0x7f28440a4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.099+0000 7f2861e65640 1 --2- 172.21.15.16:0/2870844085 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f28440a2090 0x7f28440a4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.099+0000 7f285affd640 1 -- 172.21.15.16:0/2870844085 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f28500efe50 con 0x7f285c103290 2024-03-28T22:12:02.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.099+0000 7f2861e65640 1 --2- 172.21.15.16:0/2870844085 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f28440a2090 0x7f28440a4550 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2854009c30 tx=0x7f2854005f90 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:02.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.104+0000 7f285affd640 1 -- 172.21.15.16:0/2870844085 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f28500bce20 con 0x7f285c103290 2024-03-28T22:12:02.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.323+0000 7f28640c9640 1 -- 172.21.15.16:0/2870844085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f285c106e30 con 0x7f285c103290 2024-03-28T22:12:02.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.324+0000 7f285affd640 1 -- 172.21.15.16:0/2870844085 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11855 (secure 0 0 0) 0x7f28500bc720 con 0x7f285c103290 2024-03-28T22:12:02.326 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:12:02.326 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/1644581227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1644581227},{"type":"v1","addr":"172.21.15.16:6839","nonce":1644581227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/2593062669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2593062669},{"type":"v1","addr":"172.21.15.16:6841","nonce":2593062669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:08:29.672976+0000","modified":"2024-03-28T22:11:54.920802+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24566,"mds_1":24524,"mds_2":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24466":{"gid":24466,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/247249857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":247249857},{"type":"v1","addr":"172.21.15.63:6835","nonce":247249857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6837/2324231032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":2324231032},{"type":"v1","addr":"172.21.15.63:6837","nonce":2324231032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6839/1177197077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":1177197077},{"type":"v1","addr":"172.21.15.63:6839","nonce":1177197077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"e","rank":2,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.53:6835/350727634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":350727634},{"type":"v1","addr":"172.21.15.53:6835","nonce":350727634}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/349877573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":349877573},{"type":"v1","addr":"172.21.15.53:6837","nonce":349877573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6839/3714153956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":3714153956},{"type":"v1","addr":"172.21.15.53:6839","nonce":3714153956}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24566,"qdb_cluster":[24500,24566,24524]},"id":1}]} 2024-03-28T22:12:02.326 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 24 2024-03-28T22:12:02.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.326+0000 7f28640c9640 1 -- 172.21.15.16:0/2870844085 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f28440a2090 msgr2=0x7f28440a4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:02.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.326+0000 7f28640c9640 1 --2- 172.21.15.16:0/2870844085 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f28440a2090 0x7f28440a4550 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2854009c30 tx=0x7f2854005f90 comp rx=0 tx=0).stop 2024-03-28T22:12:02.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.326+0000 7f28640c9640 1 -- 172.21.15.16:0/2870844085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f285c103290 msgr2=0x7f285c100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:02.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.326+0000 7f28640c9640 1 --2- 172.21.15.16:0/2870844085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f285c103290 0x7f285c100b00 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f2850005940 tx=0x7f285002cc40 comp rx=0 tx=0).stop 2024-03-28T22:12:02.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.326+0000 7f28640c9640 1 -- 172.21.15.16:0/2870844085 shutdown_connections 2024-03-28T22:12:02.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.326+0000 7f28640c9640 1 --2- 172.21.15.16:0/2870844085 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f28440a2090 0x7f28440a4550 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.326+0000 7f28640c9640 1 --2- 172.21.15.16:0/2870844085 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f285c10bd40 0x7f285c101040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.326+0000 7f28640c9640 1 --2- 172.21.15.16:0/2870844085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f285c103290 0x7f285c100b00 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.326+0000 7f28640c9640 1 --2- 172.21.15.16:0/2870844085 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f285c102970 0x7f285c1005c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.326+0000 7f28640c9640 1 -- 172.21.15.16:0/2870844085 >> 172.21.15.16:0/2870844085 conn(0x7f285c0777d0 msgr2=0x7f285c0fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:02.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.326+0000 7f28640c9640 1 -- 172.21.15.16:0/2870844085 shutdown_connections 2024-03-28T22:12:02.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.326+0000 7f28640c9640 1 -- 172.21.15.16:0/2870844085 wait complete. 2024-03-28T22:12:02.339 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:12:02.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.445+0000 7f3a58a23640 1 -- 172.21.15.16:0/2128763529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a54106680 msgr2=0x7f3a54106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:02.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.445+0000 7f3a58a23640 1 --2- 172.21.15.16:0/2128763529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a54106680 0x7f3a54106a60 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f3a44009930 tx=0x7f3a4402bc30 comp rx=0 tx=0).stop 2024-03-28T22:12:02.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.445+0000 7f3a58a23640 1 -- 172.21.15.16:0/2128763529 shutdown_connections 2024-03-28T22:12:02.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.445+0000 7f3a58a23640 1 --2- 172.21.15.16:0/2128763529 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a54100fa0 0x7f3a5410d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.445+0000 7f3a58a23640 1 --2- 172.21.15.16:0/2128763529 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a541005e0 0x7f3a54100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.445+0000 7f3a58a23640 1 --2- 172.21.15.16:0/2128763529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a54106680 0x7f3a54106a60 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.445+0000 7f3a58a23640 1 -- 172.21.15.16:0/2128763529 >> 172.21.15.16:0/2128763529 conn(0x7f3a54077770 msgr2=0x7f3a54077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:02.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.445+0000 7f3a58a23640 1 -- 172.21.15.16:0/2128763529 shutdown_connections 2024-03-28T22:12:02.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.445+0000 7f3a58a23640 1 -- 172.21.15.16:0/2128763529 wait complete. 2024-03-28T22:12:02.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.446+0000 7f3a58a23640 1 Processor -- start 2024-03-28T22:12:02.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.446+0000 7f3a58a23640 1 -- start start 2024-03-28T22:12:02.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.446+0000 7f3a58a23640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a541005e0 0x7f3a54196fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.446+0000 7f3a58a23640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a54100fa0 0x7f3a54197520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.446+0000 7f3a58a23640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a54106680 0x7f3a5419b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.446+0000 7f3a58a23640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3a54110a00 con 0x7f3a54106680 2024-03-28T22:12:02.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.446+0000 7f3a58a23640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3a54110880 con 0x7f3a541005e0 2024-03-28T22:12:02.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.446+0000 7f3a58a23640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3a54110b80 con 0x7f3a54100fa0 2024-03-28T22:12:02.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.446+0000 7f3a52d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a54106680 0x7f3a5419b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.446+0000 7f3a5259c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a541005e0 0x7f3a54196fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.446+0000 7f3a52d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a54106680 0x7f3a5419b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51006/0 (socket says 172.21.15.16:51006) 2024-03-28T22:12:02.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.446+0000 7f3a5259c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a541005e0 0x7f3a54196fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:52180/0 (socket says 172.21.15.16:52180) 2024-03-28T22:12:02.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.446+0000 7f3a52d9d640 1 -- 172.21.15.16:0/1922905007 learned_addr learned my addr 172.21.15.16:0/1922905007 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:02.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.446+0000 7f3a49d9b640 1 --2- 172.21.15.16:0/1922905007 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a54100fa0 0x7f3a54197520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.447+0000 7f3a5259c640 1 -- 172.21.15.16:0/1922905007 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a54100fa0 msgr2=0x7f3a54197520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:02.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.447+0000 7f3a5259c640 1 --2- 172.21.15.16:0/1922905007 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a54100fa0 0x7f3a54197520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.447+0000 7f3a5259c640 1 -- 172.21.15.16:0/1922905007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a54106680 msgr2=0x7f3a5419b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:02.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.447+0000 7f3a5259c640 1 --2- 172.21.15.16:0/1922905007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a54106680 0x7f3a5419b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.447+0000 7f3a5259c640 1 -- 172.21.15.16:0/1922905007 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a5419bfd0 con 0x7f3a541005e0 2024-03-28T22:12:02.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.447+0000 7f3a52d9d640 1 --2- 172.21.15.16:0/1922905007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a54106680 0x7f3a5419b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:12:02.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.447+0000 7f3a5259c640 1 --2- 172.21.15.16:0/1922905007 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a541005e0 0x7f3a54196fe0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3a4402c170 tx=0x7f3a4402cb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:02.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.447+0000 7f3a4bfff640 1 -- 172.21.15.16:0/1922905007 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a44041070 con 0x7f3a541005e0 2024-03-28T22:12:02.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.447+0000 7f3a58a23640 1 -- 172.21.15.16:0/1922905007 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a5419c260 con 0x7f3a541005e0 2024-03-28T22:12:02.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.447+0000 7f3a4bfff640 1 -- 172.21.15.16:0/1922905007 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a44004450 con 0x7f3a541005e0 2024-03-28T22:12:02.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.447+0000 7f3a4bfff640 1 -- 172.21.15.16:0/1922905007 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a44003de0 con 0x7f3a541005e0 2024-03-28T22:12:02.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.447+0000 7f3a58a23640 1 -- 172.21.15.16:0/1922905007 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a541a3b50 con 0x7f3a541005e0 2024-03-28T22:12:02.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.449+0000 7f3a4bfff640 1 -- 172.21.15.16:0/1922905007 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 25) ==== 136989+0+0 (secure 0 0 0) 0x7f3a44044070 con 0x7f3a541005e0 2024-03-28T22:12:02.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.449+0000 7f3a5b607740 1 Processor -- start 2024-03-28T22:12:02.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.450+0000 7f3a5b607740 1 -- start start 2024-03-28T22:12:02.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.450+0000 7f3a4bfff640 1 --2- 172.21.15.16:0/1922905007 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f3a400a2010 0x7f3a400a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.450+0000 7f3a5b607740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e97c7cdd80 0x55e97c7ce160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.450+0000 7f3a5b607740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e97c7ce6a0 0x55e97c7dba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.450+0000 7f3a5b607740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e97c7dbf70 0x55e97c7de360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.450+0000 7f3a5b607740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e97c7321b0 con 0x55e97c7ce6a0 2024-03-28T22:12:02.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.450+0000 7f3a5b607740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55e97c6dac00 con 0x55e97c7cdd80 2024-03-28T22:12:02.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.450+0000 7f3a5b607740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e97c72e250 con 0x55e97c7dbf70 2024-03-28T22:12:02.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.450+0000 7f3a4bfff640 1 -- 172.21.15.16:0/1922905007 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f3a440efb60 con 0x7f3a541005e0 2024-03-28T22:12:02.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.450+0000 7f3a52d9d640 1 --2- 172.21.15.16:0/1922905007 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f3a400a2010 0x7f3a400a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.450+0000 7f3a49d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e97c7ce6a0 0x55e97c7dba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.450+0000 7f3a5259c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e97c7cdd80 0x55e97c7ce160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.450+0000 7f3a49d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e97c7ce6a0 0x55e97c7dba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51018/0 (socket says 172.21.15.16:51018) 2024-03-28T22:12:02.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.450+0000 7f3a49d9b640 1 -- 172.21.15.16:0/3790654880 learned_addr learned my addr 172.21.15.16:0/3790654880 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:02.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.450+0000 7f3a52d9d640 1 --2- 172.21.15.16:0/3790654880 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e97c7dbf70 0x55e97c7de360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.450+0000 7f3a49d9b640 1 -- 172.21.15.16:0/3790654880 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e97c7dbf70 msgr2=0x55e97c7de360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:02.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.450+0000 7f3a49d9b640 1 --2- 172.21.15.16:0/3790654880 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e97c7dbf70 0x55e97c7de360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.450+0000 7f3a49d9b640 1 -- 172.21.15.16:0/3790654880 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e97c7cdd80 msgr2=0x55e97c7ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:02.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.450+0000 7f3a49d9b640 1 --2- 172.21.15.16:0/3790654880 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e97c7cdd80 0x55e97c7ce160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.450+0000 7f3a49d9b640 1 -- 172.21.15.16:0/3790654880 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e97c6fa3d0 con 0x55e97c7ce6a0 2024-03-28T22:12:02.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.450+0000 7f3a52d9d640 1 --2- 172.21.15.16:0/3790654880 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e97c7dbf70 0x55e97c7de360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:12:02.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.450+0000 7f3a5259c640 1 --2- 172.21.15.16:0/3790654880 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e97c7cdd80 0x55e97c7ce160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:12:02.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.451+0000 7f3a52d9d640 1 --2- 172.21.15.16:0/1922905007 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f3a400a2010 0x7f3a400a44d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c009fd0 tx=0x7f3a4c00ff30 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:02.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.451+0000 7f3a49d9b640 1 --2- 172.21.15.16:0/3790654880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e97c7ce6a0 0x55e97c7dba30 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c009ca0 tx=0x7f3a3c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=558a66b70c92b43d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:02.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.451+0000 7f3a48d99640 1 -- 172.21.15.16:0/3790654880 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c019620 con 0x55e97c7ce6a0 2024-03-28T22:12:02.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.451+0000 7f3a48d99640 1 -- 172.21.15.16:0/3790654880 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a3c009e50 con 0x55e97c7ce6a0 2024-03-28T22:12:02.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.451+0000 7f3a48d99640 1 -- 172.21.15.16:0/3790654880 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c018b10 con 0x55e97c7ce6a0 2024-03-28T22:12:02.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.451+0000 7f3a5b607740 1 -- 172.21.15.16:0/3790654880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e97c7ce6a0 msgr2=0x55e97c7dba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:02.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.451+0000 7f3a5b607740 1 --2- 172.21.15.16:0/3790654880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e97c7ce6a0 0x55e97c7dba30 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c009ca0 tx=0x7f3a3c011810 comp rx=0 tx=0).stop 2024-03-28T22:12:02.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.451+0000 7f3a5b607740 1 -- 172.21.15.16:0/3790654880 shutdown_connections 2024-03-28T22:12:02.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.451+0000 7f3a5b607740 1 --2- 172.21.15.16:0/3790654880 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e97c7dbf70 0x55e97c7de360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.451+0000 7f3a5b607740 1 --2- 172.21.15.16:0/3790654880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e97c7ce6a0 0x55e97c7dba30 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.451+0000 7f3a5b607740 1 --2- 172.21.15.16:0/3790654880 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e97c7cdd80 0x55e97c7ce160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.451+0000 7f3a5b607740 1 -- 172.21.15.16:0/3790654880 >> 172.21.15.16:0/3790654880 conn(0x55e97c7c8ba0 msgr2=0x55e97c7c8f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:02.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.452+0000 7f3a5b607740 1 -- 172.21.15.16:0/3790654880 shutdown_connections 2024-03-28T22:12:02.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.452+0000 7f3a5b607740 1 -- 172.21.15.16:0/3790654880 wait complete. 2024-03-28T22:12:02.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.452+0000 7f3a5b607740 1 -- start start 2024-03-28T22:12:02.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.452+0000 7f3a5b607740 1 Processor -- start 2024-03-28T22:12:02.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.452+0000 7f3a5b607740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e97c7cdd80 0x55e97c7d8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.452+0000 7f3a5b607740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e97c7ce6a0 0x55e97c7d9490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.452+0000 7f3a5b607740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e97c7dbf70 0x55e97c7d99d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.452+0000 7f3a5b607740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e97c7deba0 con 0x55e97c7dbf70 2024-03-28T22:12:02.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.452+0000 7f3a5b607740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55e97c6d0470 con 0x55e97c7ce6a0 2024-03-28T22:12:02.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.452+0000 7f3a5b607740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e97c7ded20 con 0x55e97c7cdd80 2024-03-28T22:12:02.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.452+0000 7f3a5b607740 10 client.0 _ll_register_callbacks cb 0x55e97c6edcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:12:02.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.452+0000 7f3a52d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e97c7dbf70 0x55e97c7d99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.452+0000 7f3a52d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e97c7dbf70 0x55e97c7d99d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51024/0 (socket says 172.21.15.16:51024) 2024-03-28T22:12:02.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.452+0000 7f3a52d9d640 1 -- 172.21.15.16:0/1858241541 learned_addr learned my addr 172.21.15.16:0/1858241541 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:02.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.452+0000 7f3a49d9b640 1 --2- 172.21.15.16:0/1858241541 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e97c7ce6a0 0x55e97c7d9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.452+0000 7f3a5259c640 1 --2- 172.21.15.16:0/1858241541 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e97c7cdd80 0x55e97c7d8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.453+0000 7f3a52d9d640 1 -- 172.21.15.16:0/1858241541 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e97c7cdd80 msgr2=0x55e97c7d8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:02.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.453+0000 7f3a52d9d640 1 --2- 172.21.15.16:0/1858241541 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e97c7cdd80 0x55e97c7d8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.453+0000 7f3a52d9d640 1 -- 172.21.15.16:0/1858241541 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e97c7ce6a0 msgr2=0x55e97c7d9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:02.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.453+0000 7f3a52d9d640 1 --2- 172.21.15.16:0/1858241541 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e97c7ce6a0 0x55e97c7d9490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.453+0000 7f3a52d9d640 1 -- 172.21.15.16:0/1858241541 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e97c4463f0 con 0x55e97c7dbf70 2024-03-28T22:12:02.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.453+0000 7f3a52d9d640 1 --2- 172.21.15.16:0/1858241541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e97c7dbf70 0x55e97c7d99d0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c011720 tx=0x7f3a4c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6fa27b7e6a56c11 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:02.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.453+0000 7f3a39ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:12:02.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.453+0000 7f3a39ffb640 1 -- 172.21.15.16:0/1858241541 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a4c026070 con 0x55e97c7dbf70 2024-03-28T22:12:02.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.453+0000 7f3a5b607740 1 -- 172.21.15.16:0/1858241541 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a3c009e50 con 0x55e97c7dbf70 2024-03-28T22:12:02.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.453+0000 7f3a39ffb640 1 -- 172.21.15.16:0/1858241541 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f3a4c020640 con 0x55e97c7dbf70 2024-03-28T22:12:02.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.453+0000 7f3a39ffb640 1 -- 172.21.15.16:0/1858241541 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3a4c032050 con 0x55e97c7dbf70 2024-03-28T22:12:02.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.453+0000 7f3a5b607740 10 client.15786 fetch_fsmap learned FSMap version 24 2024-03-28T22:12:02.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.453+0000 7f3a5b607740 1 -- 172.21.15.16:0/1858241541 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e97c5107d0 con 0x55e97c7dbf70 2024-03-28T22:12:02.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.453+0000 7f3a39ffb640 1 -- 172.21.15.16:0/1858241541 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a3c009e50 con 0x55e97c7dbf70 2024-03-28T22:12:02.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.454+0000 7f3a39ffb640 1 -- 172.21.15.16:0/1858241541 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7f3a4c01f070 con 0x55e97c7dbf70 2024-03-28T22:12:02.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.454+0000 7f3a5b607740 10 client.15786 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:12:02.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.454+0000 7f3a5b607740 10 client.15786 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-28T22:12:02.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.454+0000 7f3a5b607740 20 client.15786 populate_metadata read hostname 'smithi016' 2024-03-28T22:12:02.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.454+0000 7f3a5b607740 1 --2- 172.21.15.16:0/1858241541 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55e97c7ebd30 0x55e97c7ee1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.454+0000 7f3a5b607740 4 client.15786 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:12:02.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.454+0000 7f3a5b607740 1 -- 172.21.15.16:0/1858241541 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e97c73f6f0 con 0x55e97c7ebd30 2024-03-28T22:12:02.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.454+0000 7f3a5259c640 1 --2- 172.21.15.16:0/1858241541 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55e97c7ebd30 0x55e97c7ee1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.454+0000 7f3a5259c640 1 --2- 172.21.15.16:0/1858241541 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55e97c7ebd30 0x55e97c7ee1b0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=62cfba13dc4571bc server_cookie=c7c4260c1e2e80de in_seq=0 out_seq=0 2024-03-28T22:12:02.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.454+0000 7f3a39ffb640 10 client.15786 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:12:02.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.455+0000 7f3a39ffb640 1 -- 172.21.15.16:0/1858241541 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e97c73f6f0 con 0x55e97c7ebd30 2024-03-28T22:12:02.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.455+0000 7f3a39ffb640 10 client.15786 handle_command_reply: tid=0 2024-03-28T22:12:02.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.455+0000 7f3a5b607740 1 client.15786 shutdown 2024-03-28T22:12:02.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.455+0000 7f3a5b607740 1 -- 172.21.15.16:0/1858241541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e97c7dbf70 msgr2=0x55e97c7d99d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:02.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.455+0000 7f3a5b607740 1 --2- 172.21.15.16:0/1858241541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e97c7dbf70 0x55e97c7d99d0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c011720 tx=0x7f3a4c004ab0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.455+0000 7f3a5b607740 1 -- 172.21.15.16:0/1858241541 shutdown_connections 2024-03-28T22:12:02.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.455+0000 7f3a5b607740 1 --2- 172.21.15.16:0/1858241541 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55e97c7ebd30 0x55e97c7ee1b0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.455+0000 7f3a5b607740 1 --2- 172.21.15.16:0/1858241541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e97c7dbf70 0x55e97c7d99d0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.455+0000 7f3a5b607740 1 --2- 172.21.15.16:0/1858241541 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e97c7ce6a0 0x55e97c7d9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.455+0000 7f3a39ffb640 0 client.15786 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:12:02.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.455+0000 7f3a5b607740 1 --2- 172.21.15.16:0/1858241541 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e97c7cdd80 0x55e97c7d8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.455+0000 7f3a5b607740 1 -- 172.21.15.16:0/1858241541 >> 172.21.15.16:0/1858241541 conn(0x55e97c7c8ba0 msgr2=0x55e97c7ca8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:02.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.456+0000 7f3a5b607740 1 -- 172.21.15.16:0/1858241541 shutdown_connections 2024-03-28T22:12:02.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.456+0000 7f3a5b607740 1 -- 172.21.15.16:0/1858241541 wait complete. 2024-03-28T22:12:02.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.456+0000 7f3a5b607740 20 client.15786 trim_cache size 0 max 16384 2024-03-28T22:12:02.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.469+0000 7f3a5b607740 1 Processor -- start 2024-03-28T22:12:02.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.469+0000 7f3a5b607740 1 -- start start 2024-03-28T22:12:02.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.469+0000 7f3a5b607740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e97c7cdd80 0x55e97c7db680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.469+0000 7f3a5b607740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e97c7ce6a0 0x55e97c7d6a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.469+0000 7f3a5b607740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e97c7dbf70 0x55e97c7e7980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.469+0000 7f3a5b607740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3a541ada30 con 0x55e97c7dbf70 2024-03-28T22:12:02.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.469+0000 7f3a5b607740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3a541ab2d0 con 0x55e97c7cdd80 2024-03-28T22:12:02.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.469+0000 7f3a5b607740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3a541a9140 con 0x55e97c7ce6a0 2024-03-28T22:12:02.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.469+0000 7f3a52d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e97c7dbf70 0x55e97c7e7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.469+0000 7f3a52d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e97c7dbf70 0x55e97c7e7980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51034/0 (socket says 172.21.15.16:51034) 2024-03-28T22:12:02.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.469+0000 7f3a52d9d640 1 -- 172.21.15.16:0/1176594963 learned_addr learned my addr 172.21.15.16:0/1176594963 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:02.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.469+0000 7f3a5259c640 1 --2- 172.21.15.16:0/1176594963 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e97c7cdd80 0x55e97c7db680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.469+0000 7f3a52d9d640 1 -- 172.21.15.16:0/1176594963 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e97c7ce6a0 msgr2=0x55e97c7d6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:02.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.469+0000 7f3a52d9d640 1 --2- 172.21.15.16:0/1176594963 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e97c7ce6a0 0x55e97c7d6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.469+0000 7f3a52d9d640 1 -- 172.21.15.16:0/1176594963 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e97c7cdd80 msgr2=0x55e97c7db680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:02.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.469+0000 7f3a52d9d640 1 --2- 172.21.15.16:0/1176594963 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e97c7cdd80 0x55e97c7db680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.469+0000 7f3a52d9d640 1 -- 172.21.15.16:0/1176594963 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e97c7dc350 con 0x55e97c7dbf70 2024-03-28T22:12:02.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.469+0000 7f3a52d9d640 1 --2- 172.21.15.16:0/1176594963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e97c7dbf70 0x55e97c7e7980 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c0091d0 tx=0x7f3a4c0049d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86fc850abd152a8a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:02.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.470+0000 7f3a3bfff640 1 -- 172.21.15.16:0/1176594963 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a4c03b070 con 0x55e97c7dbf70 2024-03-28T22:12:02.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.470+0000 7f3a3bfff640 1 -- 172.21.15.16:0/1176594963 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a4c00e890 con 0x55e97c7dbf70 2024-03-28T22:12:02.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.470+0000 7f3a3bfff640 1 -- 172.21.15.16:0/1176594963 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a4c026040 con 0x55e97c7dbf70 2024-03-28T22:12:02.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.470+0000 7f3a5b607740 1 -- 172.21.15.16:0/1176594963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e97c7dbf70 msgr2=0x55e97c7e7980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:02.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.470+0000 7f3a5b607740 1 --2- 172.21.15.16:0/1176594963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e97c7dbf70 0x55e97c7e7980 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c0091d0 tx=0x7f3a4c0049d0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.470+0000 7f3a5b607740 1 -- 172.21.15.16:0/1176594963 shutdown_connections 2024-03-28T22:12:02.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.470+0000 7f3a5b607740 1 --2- 172.21.15.16:0/1176594963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e97c7dbf70 0x55e97c7e7980 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.470+0000 7f3a5b607740 1 --2- 172.21.15.16:0/1176594963 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e97c7ce6a0 0x55e97c7d6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.470+0000 7f3a5b607740 1 --2- 172.21.15.16:0/1176594963 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e97c7cdd80 0x55e97c7db680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.470+0000 7f3a5b607740 1 -- 172.21.15.16:0/1176594963 >> 172.21.15.16:0/1176594963 conn(0x55e97c7c8ba0 msgr2=0x55e97c7ca7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:02.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.470+0000 7f3a5b607740 1 -- 172.21.15.16:0/1176594963 shutdown_connections 2024-03-28T22:12:02.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.470+0000 7f3a5b607740 1 -- 172.21.15.16:0/1176594963 wait complete. 2024-03-28T22:12:02.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.470+0000 7f3a5b607740 1 -- start start 2024-03-28T22:12:02.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.470+0000 7f3a5b607740 1 Processor -- start 2024-03-28T22:12:02.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.471+0000 7f3a5b607740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e97c7cdd80 0x55e97c7e3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.471+0000 7f3a5b607740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e97c7ce6a0 0x55e97c7cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.471+0000 7f3a5b607740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e97c7dbf70 0x55e97c7cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.471+0000 7f3a5b607740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3a541a9790 con 0x55e97c7cdd80 2024-03-28T22:12:02.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.471+0000 7f3a5b607740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3a541aac00 con 0x55e97c7dbf70 2024-03-28T22:12:02.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.471+0000 7f3a5b607740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3a541a9e60 con 0x55e97c7ce6a0 2024-03-28T22:12:02.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.471+0000 7f3a5b607740 10 client.0 _ll_register_callbacks cb 0x7f3a4c0121c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:12:02.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.471+0000 7f3a5259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e97c7cdd80 0x55e97c7e3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.471+0000 7f3a5259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e97c7cdd80 0x55e97c7e3fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51036/0 (socket says 172.21.15.16:51036) 2024-03-28T22:12:02.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.471+0000 7f3a5259c640 1 -- 172.21.15.16:0/3434808957 learned_addr learned my addr 172.21.15.16:0/3434808957 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:02.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.471+0000 7f3a52d9d640 1 --2- 172.21.15.16:0/3434808957 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e97c7dbf70 0x55e97c7cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.471+0000 7f3a5259c640 1 -- 172.21.15.16:0/3434808957 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e97c7ce6a0 msgr2=0x55e97c7cf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:02.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.471+0000 7f3a5259c640 1 --2- 172.21.15.16:0/3434808957 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e97c7ce6a0 0x55e97c7cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.471+0000 7f3a5259c640 1 -- 172.21.15.16:0/3434808957 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e97c7dbf70 msgr2=0x55e97c7cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:02.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.471+0000 7f3a5259c640 1 --2- 172.21.15.16:0/3434808957 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e97c7dbf70 0x55e97c7cf8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.471+0000 7f3a5259c640 1 -- 172.21.15.16:0/3434808957 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e97c7cfe30 con 0x55e97c7cdd80 2024-03-28T22:12:02.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.471+0000 7f3a5259c640 1 --2- 172.21.15.16:0/3434808957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e97c7cdd80 0x55e97c7e3fb0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f3a440ed370 tx=0x7f3a4402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b9c83cb92afdaf2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:02.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.471+0000 7f3a39ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:12:02.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.471+0000 7f3a39ffb640 1 -- 172.21.15.16:0/3434808957 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a440bbe40 con 0x55e97c7cdd80 2024-03-28T22:12:02.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.471+0000 7f3a39ffb640 1 -- 172.21.15.16:0/3434808957 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f3a44040d60 con 0x55e97c7cdd80 2024-03-28T22:12:02.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.471+0000 7f3a5b607740 1 -- 172.21.15.16:0/3434808957 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a4c00e890 con 0x55e97c7cdd80 2024-03-28T22:12:02.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.472+0000 7f3a39ffb640 1 -- 172.21.15.16:0/3434808957 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3a4404a170 con 0x55e97c7cdd80 2024-03-28T22:12:02.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.472+0000 7f3a5b607740 10 client.15792 fetch_fsmap learned FSMap version 24 2024-03-28T22:12:02.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.472+0000 7f3a5b607740 1 -- 172.21.15.16:0/3434808957 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e97c8530c0 con 0x55e97c7cdd80 2024-03-28T22:12:02.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.472+0000 7f3a39ffb640 1 -- 172.21.15.16:0/3434808957 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a4c00e890 con 0x55e97c7cdd80 2024-03-28T22:12:02.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.472+0000 7f3a39ffb640 1 -- 172.21.15.16:0/3434808957 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7f3a440b8b50 con 0x55e97c7cdd80 2024-03-28T22:12:02.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.472+0000 7f3a5b607740 10 client.15792 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:12:02.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.472+0000 7f3a5b607740 10 client.15792 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-28T22:12:02.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.472+0000 7f3a5b607740 20 client.15792 populate_metadata read hostname 'smithi016' 2024-03-28T22:12:02.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.472+0000 7f3a5b607740 1 --2- 172.21.15.16:0/3434808957 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55e97c853700 0x55e97c855b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.472+0000 7f3a5b607740 4 client.15792 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:12:02.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.472+0000 7f3a5b607740 1 -- 172.21.15.16:0/3434808957 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e97c739a40 con 0x55e97c853700 2024-03-28T22:12:02.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.472+0000 7f3a49d9b640 1 --2- 172.21.15.16:0/3434808957 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55e97c853700 0x55e97c855b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.472+0000 7f3a49d9b640 1 --2- 172.21.15.16:0/3434808957 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55e97c853700 0x55e97c855b20 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a6aaa5ad801e64cd server_cookie=6080475489968728 in_seq=0 out_seq=0 2024-03-28T22:12:02.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.472+0000 7f3a39ffb640 10 client.15792 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:12:02.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.473+0000 7f3a39ffb640 1 -- 172.21.15.16:0/3434808957 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e97c739a40 con 0x55e97c853700 2024-03-28T22:12:02.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.473+0000 7f3a39ffb640 10 client.15792 handle_command_reply: tid=0 2024-03-28T22:12:02.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.473+0000 7f3a5b607740 1 client.15792 shutdown 2024-03-28T22:12:02.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.473+0000 7f3a5b607740 1 -- 172.21.15.16:0/3434808957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e97c7cdd80 msgr2=0x55e97c7e3fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:02.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.473+0000 7f3a5b607740 1 --2- 172.21.15.16:0/3434808957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e97c7cdd80 0x55e97c7e3fb0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f3a440ed370 tx=0x7f3a4402bc30 comp rx=0 tx=0).stop 2024-03-28T22:12:02.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.473+0000 7f3a5b607740 1 -- 172.21.15.16:0/3434808957 shutdown_connections 2024-03-28T22:12:02.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.473+0000 7f3a5b607740 1 --2- 172.21.15.16:0/3434808957 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55e97c853700 0x55e97c855b20 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.473+0000 7f3a5b607740 1 --2- 172.21.15.16:0/3434808957 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e97c7dbf70 0x55e97c7cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.473+0000 7f3a39ffb640 0 client.15792 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:12:02.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.473+0000 7f3a5b607740 1 --2- 172.21.15.16:0/3434808957 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e97c7ce6a0 0x55e97c7cf3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.473+0000 7f3a5b607740 1 --2- 172.21.15.16:0/3434808957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e97c7cdd80 0x55e97c7e3fb0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.473+0000 7f3a5b607740 1 -- 172.21.15.16:0/3434808957 >> 172.21.15.16:0/3434808957 conn(0x55e97c7c8ba0 msgr2=0x55e97c7c74b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:02.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.473+0000 7f3a5b607740 1 -- 172.21.15.16:0/3434808957 shutdown_connections 2024-03-28T22:12:02.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.473+0000 7f3a5b607740 1 -- 172.21.15.16:0/3434808957 wait complete. 2024-03-28T22:12:02.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.473+0000 7f3a5b607740 20 client.15792 trim_cache size 0 max 16384 2024-03-28T22:12:02.482 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:12:02.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.473+0000 7f3a58a23640 1 -- 172.21.15.16:0/1922905007 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f3a400a2010 msgr2=0x7f3a400a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:02.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.473+0000 7f3a58a23640 1 --2- 172.21.15.16:0/1922905007 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f3a400a2010 0x7f3a400a44d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c009fd0 tx=0x7f3a4c00ff30 comp rx=0 tx=0).stop 2024-03-28T22:12:02.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.473+0000 7f3a58a23640 1 -- 172.21.15.16:0/1922905007 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a541005e0 msgr2=0x7f3a54196fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:02.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.473+0000 7f3a58a23640 1 --2- 172.21.15.16:0/1922905007 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a541005e0 0x7f3a54196fe0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3a4402c170 tx=0x7f3a4402cb20 comp rx=0 tx=0).stop 2024-03-28T22:12:02.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.473+0000 7f3a58a23640 1 -- 172.21.15.16:0/1922905007 shutdown_connections 2024-03-28T22:12:02.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.473+0000 7f3a58a23640 1 --2- 172.21.15.16:0/1922905007 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f3a400a2010 0x7f3a400a44d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.473+0000 7f3a58a23640 1 --2- 172.21.15.16:0/1922905007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a54106680 0x7f3a5419b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.473+0000 7f3a58a23640 1 --2- 172.21.15.16:0/1922905007 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a54100fa0 0x7f3a54197520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.473+0000 7f3a58a23640 1 --2- 172.21.15.16:0/1922905007 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a541005e0 0x7f3a54196fe0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.473+0000 7f3a58a23640 1 -- 172.21.15.16:0/1922905007 >> 172.21.15.16:0/1922905007 conn(0x7f3a54077770 msgr2=0x7f3a540ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:02.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.474+0000 7f3a58a23640 1 -- 172.21.15.16:0/1922905007 shutdown_connections 2024-03-28T22:12:02.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.474+0000 7f3a58a23640 1 -- 172.21.15.16:0/1922905007 wait complete. 2024-03-28T22:12:02.486 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:12:02.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.589+0000 7fbc2fd93640 1 -- 172.21.15.16:0/1457819634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc281020a0 msgr2=0x7fbc2810e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:02.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.589+0000 7fbc2fd93640 1 --2- 172.21.15.16:0/1457819634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc281020a0 0x7fbc2810e5b0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fbc20009cd0 tx=0x7fbc2002c0f0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.589+0000 7fbc2fd93640 1 -- 172.21.15.16:0/1457819634 shutdown_connections 2024-03-28T22:12:02.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.589+0000 7fbc2fd93640 1 --2- 172.21.15.16:0/1457819634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc281020a0 0x7fbc2810e5b0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.589+0000 7fbc2fd93640 1 --2- 172.21.15.16:0/1457819634 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc281016e0 0x7fbc28101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.589+0000 7fbc2fd93640 1 --2- 172.21.15.16:0/1457819634 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc281076f0 0x7fbc28107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.589+0000 7fbc2fd93640 1 -- 172.21.15.16:0/1457819634 >> 172.21.15.16:0/1457819634 conn(0x7fbc280fd1f0 msgr2=0x7fbc280ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:02.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.589+0000 7fbc2fd93640 1 -- 172.21.15.16:0/1457819634 shutdown_connections 2024-03-28T22:12:02.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.589+0000 7fbc2fd93640 1 -- 172.21.15.16:0/1457819634 wait complete. 2024-03-28T22:12:02.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.590+0000 7fbc2fd93640 1 Processor -- start 2024-03-28T22:12:02.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.590+0000 7fbc2fd93640 1 -- start start 2024-03-28T22:12:02.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.590+0000 7fbc2fd93640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc281016e0 0x7fbc28073a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.590+0000 7fbc2fd93640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc281020a0 0x7fbc28073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.590+0000 7fbc2fd93640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc281076f0 0x7fbc280744d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.590+0000 7fbc2fd93640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbc28111ac0 con 0x7fbc281016e0 2024-03-28T22:12:02.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.590+0000 7fbc2fd93640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fbc28111940 con 0x7fbc281076f0 2024-03-28T22:12:02.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.590+0000 7fbc2fd93640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbc28111c40 con 0x7fbc281020a0 2024-03-28T22:12:02.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.590+0000 7fbc2db2f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc281016e0 0x7fbc28073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.590+0000 7fbc2db2f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc281016e0 0x7fbc28073a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51054/0 (socket says 172.21.15.16:51054) 2024-03-28T22:12:02.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.590+0000 7fbc2db2f640 1 -- 172.21.15.16:0/1588289155 learned_addr learned my addr 172.21.15.16:0/1588289155 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:02.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.590+0000 7fbc2e330640 1 --2- 172.21.15.16:0/1588289155 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc281076f0 0x7fbc280744d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.590+0000 7fbc2d32e640 1 --2- 172.21.15.16:0/1588289155 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc281020a0 0x7fbc28073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.590+0000 7fbc2db2f640 1 -- 172.21.15.16:0/1588289155 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc281020a0 msgr2=0x7fbc28073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:02.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.590+0000 7fbc2db2f640 1 --2- 172.21.15.16:0/1588289155 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc281020a0 0x7fbc28073f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.590+0000 7fbc2db2f640 1 -- 172.21.15.16:0/1588289155 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc281076f0 msgr2=0x7fbc280744d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:02.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.590+0000 7fbc2db2f640 1 --2- 172.21.15.16:0/1588289155 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc281076f0 0x7fbc280744d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.590+0000 7fbc2db2f640 1 -- 172.21.15.16:0/1588289155 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc281a6480 con 0x7fbc281016e0 2024-03-28T22:12:02.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.590+0000 7fbc2e330640 1 --2- 172.21.15.16:0/1588289155 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc281076f0 0x7fbc280744d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:12:02.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.591+0000 7fbc2db2f640 1 --2- 172.21.15.16:0/1588289155 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc281016e0 0x7fbc28073a50 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fbc2400ea60 tx=0x7fbc2400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:02.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.591+0000 7fbc1effd640 1 -- 172.21.15.16:0/1588289155 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc2400cde0 con 0x7fbc281016e0 2024-03-28T22:12:02.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.591+0000 7fbc1effd640 1 -- 172.21.15.16:0/1588289155 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc240035f0 con 0x7fbc281016e0 2024-03-28T22:12:02.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.591+0000 7fbc2fd93640 1 -- 172.21.15.16:0/1588289155 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc281a6770 con 0x7fbc281016e0 2024-03-28T22:12:02.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.591+0000 7fbc1effd640 1 -- 172.21.15.16:0/1588289155 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc24010670 con 0x7fbc281016e0 2024-03-28T22:12:02.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.591+0000 7fbc2fd93640 1 -- 172.21.15.16:0/1588289155 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc281a6cc0 con 0x7fbc281016e0 2024-03-28T22:12:02.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.593+0000 7fbc1effd640 1 -- 172.21.15.16:0/1588289155 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 25) ==== 136989+0+0 (secure 0 0 0) 0x7fbc240037a0 con 0x7fbc281016e0 2024-03-28T22:12:02.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.593+0000 7fbc32979740 1 Processor -- start 2024-03-28T22:12:02.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.593+0000 7fbc32979740 1 -- start start 2024-03-28T22:12:02.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.594+0000 7fbc32979740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffb3679470 0x55ffb3679850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.594+0000 7fbc1effd640 1 --2- 172.21.15.16:0/1588289155 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fbc140a2010 0x7fbc140a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.594+0000 7fbc32979740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ffb3679d90 0x55ffb3682430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.594+0000 7fbc32979740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ffb3682970 0x55ffb3684d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.594+0000 7fbc32979740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ffb35dd1b0 con 0x55ffb3679470 2024-03-28T22:12:02.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.594+0000 7fbc32979740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55ffb3585c00 con 0x55ffb3679d90 2024-03-28T22:12:02.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.594+0000 7fbc32979740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55ffb35d9250 con 0x55ffb3682970 2024-03-28T22:12:02.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.594+0000 7fbc2e330640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffb3679470 0x55ffb3679850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.594+0000 7fbc2db2f640 1 --2- 172.21.15.16:0/1588289155 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fbc140a2010 0x7fbc140a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.594+0000 7fbc2e330640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffb3679470 0x55ffb3679850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51070/0 (socket says 172.21.15.16:51070) 2024-03-28T22:12:02.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.594+0000 7fbc2e330640 1 -- 172.21.15.16:0/157810926 learned_addr learned my addr 172.21.15.16:0/157810926 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:02.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.594+0000 7fbc2d32e640 1 --2- 172.21.15.16:0/157810926 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ffb3679d90 0x55ffb3682430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.594+0000 7fbc1effd640 1 -- 172.21.15.16:0/1588289155 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fbc240ce740 con 0x7fbc281016e0 2024-03-28T22:12:02.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.594+0000 7fbc2e330640 1 --2- 172.21.15.16:0/157810926 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ffb3682970 0x55ffb3684d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.594+0000 7fbc2e330640 1 -- 172.21.15.16:0/157810926 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ffb3682970 msgr2=0x55ffb3684d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:02.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.594+0000 7fbc2e330640 1 --2- 172.21.15.16:0/157810926 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ffb3682970 0x55ffb3684d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.594+0000 7fbc2e330640 1 -- 172.21.15.16:0/157810926 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ffb3679d90 msgr2=0x55ffb3682430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:02.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.594+0000 7fbc2e330640 1 --2- 172.21.15.16:0/157810926 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ffb3679d90 0x55ffb3682430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.594+0000 7fbc2e330640 1 -- 172.21.15.16:0/157810926 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffb35a53d0 con 0x55ffb3679470 2024-03-28T22:12:02.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.594+0000 7fbc2d32e640 1 --2- 172.21.15.16:0/157810926 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ffb3679d90 0x55ffb3682430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:12:02.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.594+0000 7fbc2db2f640 1 --2- 172.21.15.16:0/1588289155 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fbc140a2010 0x7fbc140a44d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbc240cbd90 tx=0x7fbc24090390 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:02.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.594+0000 7fbc2e330640 1 --2- 172.21.15.16:0/157810926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffb3679470 0x55ffb3679850 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fbc20009430 tx=0x7fbc20009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ee2731a6e5d8f25 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:02.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.595+0000 7fbc13fff640 1 -- 172.21.15.16:0/157810926 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc20048070 con 0x55ffb3679470 2024-03-28T22:12:02.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.595+0000 7fbc13fff640 1 -- 172.21.15.16:0/157810926 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc20003970 con 0x55ffb3679470 2024-03-28T22:12:02.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.595+0000 7fbc13fff640 1 -- 172.21.15.16:0/157810926 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc20039040 con 0x55ffb3679470 2024-03-28T22:12:02.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.595+0000 7fbc32979740 1 -- 172.21.15.16:0/157810926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffb3679470 msgr2=0x55ffb3679850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:02.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.595+0000 7fbc32979740 1 --2- 172.21.15.16:0/157810926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffb3679470 0x55ffb3679850 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fbc20009430 tx=0x7fbc20009360 comp rx=0 tx=0).stop 2024-03-28T22:12:02.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.595+0000 7fbc32979740 1 -- 172.21.15.16:0/157810926 shutdown_connections 2024-03-28T22:12:02.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.595+0000 7fbc32979740 1 --2- 172.21.15.16:0/157810926 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ffb3682970 0x55ffb3684d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.595+0000 7fbc32979740 1 --2- 172.21.15.16:0/157810926 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ffb3679d90 0x55ffb3682430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.595+0000 7fbc32979740 1 --2- 172.21.15.16:0/157810926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffb3679470 0x55ffb3679850 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.595+0000 7fbc32979740 1 -- 172.21.15.16:0/157810926 >> 172.21.15.16:0/157810926 conn(0x55ffb3673ba0 msgr2=0x55ffb3673f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:02.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.595+0000 7fbc32979740 1 -- 172.21.15.16:0/157810926 shutdown_connections 2024-03-28T22:12:02.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.595+0000 7fbc32979740 1 -- 172.21.15.16:0/157810926 wait complete. 2024-03-28T22:12:02.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.596+0000 7fbc32979740 1 -- start start 2024-03-28T22:12:02.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.596+0000 7fbc32979740 1 Processor -- start 2024-03-28T22:12:02.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.596+0000 7fbc32979740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ffb3679470 0x55ffb367f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.596+0000 7fbc32979740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ffb3679d90 0x55ffb367ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.596+0000 7fbc32979740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffb3682970 0x55ffb3680460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.596+0000 7fbc32979740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ffb3685700 con 0x55ffb3682970 2024-03-28T22:12:02.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.596+0000 7fbc32979740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55ffb357b470 con 0x55ffb3679470 2024-03-28T22:12:02.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.596+0000 7fbc32979740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55ffb3685880 con 0x55ffb3679d90 2024-03-28T22:12:02.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.596+0000 7fbc32979740 10 client.0 _ll_register_callbacks cb 0x55ffb3598cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:12:02.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.596+0000 7fbc2e330640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffb3682970 0x55ffb3680460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.596+0000 7fbc2e330640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffb3682970 0x55ffb3680460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51084/0 (socket says 172.21.15.16:51084) 2024-03-28T22:12:02.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.596+0000 7fbc2e330640 1 -- 172.21.15.16:0/3882865054 learned_addr learned my addr 172.21.15.16:0/3882865054 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:02.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.596+0000 7fbc2e330640 1 --2- 172.21.15.16:0/3882865054 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ffb3679470 0x55ffb367f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.597+0000 7fbc2d32e640 1 --2- 172.21.15.16:0/3882865054 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ffb3679d90 0x55ffb367ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.597+0000 7fbc2e330640 1 -- 172.21.15.16:0/3882865054 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ffb3679d90 msgr2=0x55ffb367ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:02.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.597+0000 7fbc2e330640 1 --2- 172.21.15.16:0/3882865054 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ffb3679d90 0x55ffb367ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.597+0000 7fbc2e330640 1 -- 172.21.15.16:0/3882865054 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ffb3679470 msgr2=0x55ffb367f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:02.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.597+0000 7fbc2e330640 1 --2- 172.21.15.16:0/3882865054 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ffb3679470 0x55ffb367f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.597+0000 7fbc2e330640 1 -- 172.21.15.16:0/3882865054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffb32f13f0 con 0x55ffb3682970 2024-03-28T22:12:02.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.597+0000 7fbc2e330640 1 --2- 172.21.15.16:0/3882865054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffb3682970 0x55ffb3680460 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fbc20046040 tx=0x7fbc200347f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26d1d46498577dab server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:02.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.597+0000 7fbc117fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:12:02.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.597+0000 7fbc117fa640 1 -- 172.21.15.16:0/3882865054 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc20039030 con 0x55ffb3682970 2024-03-28T22:12:02.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.597+0000 7fbc117fa640 1 -- 172.21.15.16:0/3882865054 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fbc20032790 con 0x55ffb3682970 2024-03-28T22:12:02.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.597+0000 7fbc32979740 1 -- 172.21.15.16:0/3882865054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc20003970 con 0x55ffb3682970 2024-03-28T22:12:02.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.597+0000 7fbc117fa640 1 -- 172.21.15.16:0/3882865054 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fbc20048040 con 0x55ffb3682970 2024-03-28T22:12:02.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.597+0000 7fbc32979740 10 client.15804 fetch_fsmap learned FSMap version 24 2024-03-28T22:12:02.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.597+0000 7fbc32979740 1 -- 172.21.15.16:0/3882865054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ffb33bb7d0 con 0x55ffb3682970 2024-03-28T22:12:02.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.597+0000 7fbc117fa640 1 -- 172.21.15.16:0/3882865054 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc20003970 con 0x55ffb3682970 2024-03-28T22:12:02.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.597+0000 7fbc117fa640 1 -- 172.21.15.16:0/3882865054 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7fbc2004b050 con 0x55ffb3682970 2024-03-28T22:12:02.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.597+0000 7fbc32979740 10 client.15804 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:12:02.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.597+0000 7fbc32979740 10 client.15804 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T22:12:02.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.598+0000 7fbc32979740 20 client.15804 populate_metadata read hostname 'smithi016' 2024-03-28T22:12:02.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.598+0000 7fbc32979740 1 --2- 172.21.15.16:0/3882865054 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55ffb36927f0 0x55ffb3694be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.598+0000 7fbc32979740 4 client.15804 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:12:02.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.598+0000 7fbc32979740 1 -- 172.21.15.16:0/3882865054 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ffb35ea6f0 con 0x55ffb36927f0 2024-03-28T22:12:02.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.598+0000 7fbc2d32e640 1 --2- 172.21.15.16:0/3882865054 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55ffb36927f0 0x55ffb3694be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.598+0000 7fbc117fa640 10 client.15804 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:12:02.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.598+0000 7fbc2d32e640 1 --2- 172.21.15.16:0/3882865054 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55ffb36927f0 0x55ffb3694be0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=eb2bf2e6f8042f23 server_cookie=c71bc06c41803ff1 in_seq=0 out_seq=0 2024-03-28T22:12:02.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.599+0000 7fbc117fa640 1 -- 172.21.15.16:0/3882865054 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ffb35ea6f0 con 0x55ffb36927f0 2024-03-28T22:12:02.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.599+0000 7fbc117fa640 10 client.15804 handle_command_reply: tid=0 2024-03-28T22:12:02.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.599+0000 7fbc32979740 1 client.15804 shutdown 2024-03-28T22:12:02.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.599+0000 7fbc32979740 1 -- 172.21.15.16:0/3882865054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffb3682970 msgr2=0x55ffb3680460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:02.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.599+0000 7fbc32979740 1 --2- 172.21.15.16:0/3882865054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffb3682970 0x55ffb3680460 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fbc20046040 tx=0x7fbc200347f0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.599+0000 7fbc32979740 1 -- 172.21.15.16:0/3882865054 shutdown_connections 2024-03-28T22:12:02.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.599+0000 7fbc32979740 1 --2- 172.21.15.16:0/3882865054 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55ffb36927f0 0x55ffb3694be0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.599+0000 7fbc32979740 1 --2- 172.21.15.16:0/3882865054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffb3682970 0x55ffb3680460 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.599+0000 7fbc32979740 1 --2- 172.21.15.16:0/3882865054 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ffb3679d90 0x55ffb367ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.599+0000 7fbc32979740 1 --2- 172.21.15.16:0/3882865054 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ffb3679470 0x55ffb367f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.599+0000 7fbc117fa640 0 client.15804 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:12:02.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.599+0000 7fbc32979740 1 -- 172.21.15.16:0/3882865054 >> 172.21.15.16:0/3882865054 conn(0x55ffb3673ba0 msgr2=0x55ffb36758f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:02.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.599+0000 7fbc32979740 1 -- 172.21.15.16:0/3882865054 shutdown_connections 2024-03-28T22:12:02.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.599+0000 7fbc32979740 1 -- 172.21.15.16:0/3882865054 wait complete. 2024-03-28T22:12:02.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.599+0000 7fbc32979740 20 client.15804 trim_cache size 0 max 16384 2024-03-28T22:12:02.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.612+0000 7fbc32979740 1 Processor -- start 2024-03-28T22:12:02.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.612+0000 7fbc32979740 1 -- start start 2024-03-28T22:12:02.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.612+0000 7fbc32979740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ffb3679470 0x55ffb367f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.612+0000 7fbc32979740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ffb3679d90 0x55ffb367cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.612+0000 7fbc32979740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffb3682970 0x55ffb367d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.613+0000 7fbc32979740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbc281b1e80 con 0x55ffb3682970 2024-03-28T22:12:02.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.613+0000 7fbc32979740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fbc281af720 con 0x55ffb3679470 2024-03-28T22:12:02.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.613+0000 7fbc32979740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbc281ad570 con 0x55ffb3679d90 2024-03-28T22:12:02.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.613+0000 7fbc2e330640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffb3682970 0x55ffb367d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.613+0000 7fbc2e330640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffb3682970 0x55ffb367d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51088/0 (socket says 172.21.15.16:51088) 2024-03-28T22:12:02.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.613+0000 7fbc2e330640 1 -- 172.21.15.16:0/3985400123 learned_addr learned my addr 172.21.15.16:0/3985400123 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:02.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.613+0000 7fbc2e330640 1 --2- 172.21.15.16:0/3985400123 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ffb3679470 0x55ffb367f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.613+0000 7fbc2d32e640 1 --2- 172.21.15.16:0/3985400123 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ffb3679d90 0x55ffb367cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.613+0000 7fbc2e330640 1 -- 172.21.15.16:0/3985400123 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ffb3679d90 msgr2=0x55ffb367cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:02.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.613+0000 7fbc2e330640 1 --2- 172.21.15.16:0/3985400123 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ffb3679d90 0x55ffb367cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.613+0000 7fbc2e330640 1 -- 172.21.15.16:0/3985400123 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ffb3679470 msgr2=0x55ffb367f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:02.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.613+0000 7fbc2e330640 1 --2- 172.21.15.16:0/3985400123 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ffb3679470 0x55ffb367f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.613+0000 7fbc2e330640 1 -- 172.21.15.16:0/3985400123 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffb369a900 con 0x55ffb3682970 2024-03-28T22:12:02.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.613+0000 7fbc2e330640 1 --2- 172.21.15.16:0/3985400123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffb3682970 0x55ffb367d180 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fbc20004aa0 tx=0x7fbc2002cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fce8de7e45425ef server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:02.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.613+0000 7fbc137fe640 1 -- 172.21.15.16:0/3985400123 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc2002cd10 con 0x55ffb3682970 2024-03-28T22:12:02.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.613+0000 7fbc137fe640 1 -- 172.21.15.16:0/3985400123 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc20032c90 con 0x55ffb3682970 2024-03-28T22:12:02.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.613+0000 7fbc137fe640 1 -- 172.21.15.16:0/3985400123 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc20042770 con 0x55ffb3682970 2024-03-28T22:12:02.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.614+0000 7fbc32979740 1 -- 172.21.15.16:0/3985400123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffb3682970 msgr2=0x55ffb367d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:02.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.614+0000 7fbc32979740 1 --2- 172.21.15.16:0/3985400123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffb3682970 0x55ffb367d180 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fbc20004aa0 tx=0x7fbc2002cca0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.614+0000 7fbc32979740 1 -- 172.21.15.16:0/3985400123 shutdown_connections 2024-03-28T22:12:02.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.614+0000 7fbc32979740 1 --2- 172.21.15.16:0/3985400123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffb3682970 0x55ffb367d180 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.614+0000 7fbc32979740 1 --2- 172.21.15.16:0/3985400123 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ffb3679d90 0x55ffb367cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.614+0000 7fbc32979740 1 --2- 172.21.15.16:0/3985400123 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ffb3679470 0x55ffb367f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.614+0000 7fbc32979740 1 -- 172.21.15.16:0/3985400123 >> 172.21.15.16:0/3985400123 conn(0x55ffb3673ba0 msgr2=0x55ffb3681990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:02.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.614+0000 7fbc32979740 1 -- 172.21.15.16:0/3985400123 shutdown_connections 2024-03-28T22:12:02.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.614+0000 7fbc32979740 1 -- 172.21.15.16:0/3985400123 wait complete. 2024-03-28T22:12:02.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.614+0000 7fbc32979740 1 -- start start 2024-03-28T22:12:02.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.614+0000 7fbc32979740 1 Processor -- start 2024-03-28T22:12:02.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.615+0000 7fbc32979740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffb3679470 0x55ffb36954d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.615+0000 7fbc32979740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ffb3679d90 0x55ffb3695a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.615+0000 7fbc32979740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ffb3682970 0x55ffb3695f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.615+0000 7fbc32979740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbc281adbe0 con 0x55ffb3679470 2024-03-28T22:12:02.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.615+0000 7fbc32979740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fbc281af050 con 0x55ffb3679d90 2024-03-28T22:12:02.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.615+0000 7fbc32979740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbc281ae2b0 con 0x55ffb3682970 2024-03-28T22:12:02.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.615+0000 7fbc32979740 10 client.0 _ll_register_callbacks cb 0x7fbc20034950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:12:02.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.615+0000 7fbc2e330640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffb3679470 0x55ffb36954d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.615+0000 7fbc2e330640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffb3679470 0x55ffb36954d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51104/0 (socket says 172.21.15.16:51104) 2024-03-28T22:12:02.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.615+0000 7fbc2e330640 1 -- 172.21.15.16:0/116153298 learned_addr learned my addr 172.21.15.16:0/116153298 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:02.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.615+0000 7fbc2d32e640 1 --2- 172.21.15.16:0/116153298 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ffb3679d90 0x55ffb3695a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.615+0000 7fbc2e330640 1 --2- 172.21.15.16:0/116153298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ffb3682970 0x55ffb3695f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.615+0000 7fbc2e330640 1 -- 172.21.15.16:0/116153298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ffb3682970 msgr2=0x55ffb3695f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:02.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.615+0000 7fbc2e330640 1 --2- 172.21.15.16:0/116153298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ffb3682970 0x55ffb3695f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.615+0000 7fbc2e330640 1 -- 172.21.15.16:0/116153298 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ffb3679d90 msgr2=0x55ffb3695a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:02.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.615+0000 7fbc2e330640 1 --2- 172.21.15.16:0/116153298 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ffb3679d90 0x55ffb3695a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.615+0000 7fbc2e330640 1 -- 172.21.15.16:0/116153298 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffb36f7790 con 0x55ffb3679470 2024-03-28T22:12:02.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.615+0000 7fbc2e330640 1 --2- 172.21.15.16:0/116153298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffb3679470 0x55ffb36954d0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fbc20042ea0 tx=0x7fbc2002c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eae7347e36cf500 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:02.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.615+0000 7fbc117fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:12:02.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.615+0000 7fbc117fa640 1 -- 172.21.15.16:0/116153298 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc20039070 con 0x55ffb3679470 2024-03-28T22:12:02.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.615+0000 7fbc32979740 1 -- 172.21.15.16:0/116153298 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc20032c90 con 0x55ffb3679470 2024-03-28T22:12:02.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.615+0000 7fbc117fa640 1 -- 172.21.15.16:0/116153298 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fbc20004d40 con 0x55ffb3679470 2024-03-28T22:12:02.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.616+0000 7fbc117fa640 1 -- 172.21.15.16:0/116153298 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fbc2002cd10 con 0x55ffb3679470 2024-03-28T22:12:02.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.616+0000 7fbc32979740 10 client.15810 fetch_fsmap learned FSMap version 24 2024-03-28T22:12:02.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.616+0000 7fbc32979740 1 -- 172.21.15.16:0/116153298 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ffb36f8a30 con 0x55ffb3679470 2024-03-28T22:12:02.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.616+0000 7fbc117fa640 1 -- 172.21.15.16:0/116153298 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc20032c90 con 0x55ffb3679470 2024-03-28T22:12:02.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.616+0000 7fbc117fa640 1 -- 172.21.15.16:0/116153298 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7fbc2004b040 con 0x55ffb3679470 2024-03-28T22:12:02.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.616+0000 7fbc32979740 10 client.15810 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:12:02.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.616+0000 7fbc32979740 10 client.15810 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T22:12:02.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.616+0000 7fbc32979740 20 client.15810 populate_metadata read hostname 'smithi016' 2024-03-28T22:12:02.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.616+0000 7fbc32979740 1 --2- 172.21.15.16:0/116153298 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55ffb36f9150 0x55ffb36fb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.616+0000 7fbc32979740 4 client.15810 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:12:02.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.616+0000 7fbc32979740 1 -- 172.21.15.16:0/116153298 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ffb35e4a40 con 0x55ffb36f9150 2024-03-28T22:12:02.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.616+0000 7fbc2d32e640 1 --2- 172.21.15.16:0/116153298 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55ffb36f9150 0x55ffb36fb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.617+0000 7fbc117fa640 10 client.15810 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:12:02.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.617+0000 7fbc2d32e640 1 --2- 172.21.15.16:0/116153298 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55ffb36f9150 0x55ffb36fb570 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=aa3c62e89c144c84 server_cookie=330b92a9ad65d927 in_seq=0 out_seq=0 2024-03-28T22:12:02.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.617+0000 7fbc117fa640 1 -- 172.21.15.16:0/116153298 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ffb35e4a40 con 0x55ffb36f9150 2024-03-28T22:12:02.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.617+0000 7fbc117fa640 10 client.15810 handle_command_reply: tid=0 2024-03-28T22:12:02.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.617+0000 7fbc32979740 1 client.15810 shutdown 2024-03-28T22:12:02.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.617+0000 7fbc32979740 1 -- 172.21.15.16:0/116153298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffb3679470 msgr2=0x55ffb36954d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:02.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.617+0000 7fbc32979740 1 --2- 172.21.15.16:0/116153298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffb3679470 0x55ffb36954d0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fbc20042ea0 tx=0x7fbc2002c0f0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.617+0000 7fbc32979740 1 -- 172.21.15.16:0/116153298 shutdown_connections 2024-03-28T22:12:02.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.617+0000 7fbc32979740 1 --2- 172.21.15.16:0/116153298 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55ffb36f9150 0x55ffb36fb570 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.617+0000 7fbc32979740 1 --2- 172.21.15.16:0/116153298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ffb3682970 0x55ffb3695f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.617+0000 7fbc32979740 1 --2- 172.21.15.16:0/116153298 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ffb3679d90 0x55ffb3695a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.617+0000 7fbc32979740 1 --2- 172.21.15.16:0/116153298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ffb3679470 0x55ffb36954d0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.617+0000 7fbc117fa640 0 client.15810 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:12:02.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.617+0000 7fbc32979740 1 -- 172.21.15.16:0/116153298 >> 172.21.15.16:0/116153298 conn(0x55ffb3673ba0 msgr2=0x55ffb367f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:02.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.617+0000 7fbc32979740 1 -- 172.21.15.16:0/116153298 shutdown_connections 2024-03-28T22:12:02.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.617+0000 7fbc32979740 1 -- 172.21.15.16:0/116153298 wait complete. 2024-03-28T22:12:02.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.617+0000 7fbc32979740 20 client.15810 trim_cache size 0 max 16384 2024-03-28T22:12:02.626 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:12:02.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.618+0000 7fbc2fd93640 1 -- 172.21.15.16:0/1588289155 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fbc140a2010 msgr2=0x7fbc140a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:02.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.618+0000 7fbc2fd93640 1 --2- 172.21.15.16:0/1588289155 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fbc140a2010 0x7fbc140a44d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbc240cbd90 tx=0x7fbc24090390 comp rx=0 tx=0).stop 2024-03-28T22:12:02.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.618+0000 7fbc2fd93640 1 -- 172.21.15.16:0/1588289155 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc281016e0 msgr2=0x7fbc28073a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:02.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.618+0000 7fbc2fd93640 1 --2- 172.21.15.16:0/1588289155 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc281016e0 0x7fbc28073a50 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fbc2400ea60 tx=0x7fbc2400ef20 comp rx=0 tx=0).stop 2024-03-28T22:12:02.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.618+0000 7fbc2fd93640 1 -- 172.21.15.16:0/1588289155 shutdown_connections 2024-03-28T22:12:02.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.618+0000 7fbc2fd93640 1 --2- 172.21.15.16:0/1588289155 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fbc140a2010 0x7fbc140a44d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.618+0000 7fbc2fd93640 1 --2- 172.21.15.16:0/1588289155 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc281076f0 0x7fbc280744d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.618+0000 7fbc2fd93640 1 --2- 172.21.15.16:0/1588289155 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc281020a0 0x7fbc28073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.618+0000 7fbc2fd93640 1 --2- 172.21.15.16:0/1588289155 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc281016e0 0x7fbc28073a50 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.618+0000 7fbc2fd93640 1 -- 172.21.15.16:0/1588289155 >> 172.21.15.16:0/1588289155 conn(0x7fbc280fd1f0 msgr2=0x7fbc280fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:02.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.618+0000 7fbc2fd93640 1 -- 172.21.15.16:0/1588289155 shutdown_connections 2024-03-28T22:12:02.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.618+0000 7fbc2fd93640 1 -- 172.21.15.16:0/1588289155 wait complete. 2024-03-28T22:12:02.629 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:02 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2870844085' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:12:02.630 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:12:02.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.733+0000 7f8b3ed65640 1 -- 172.21.15.16:0/1113612302 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b381016e0 msgr2=0x7f8b38101b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:02.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.733+0000 7f8b3ed65640 1 --2- 172.21.15.16:0/1113612302 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b381016e0 0x7f8b38101b60 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8b24009860 tx=0x7f8b2402bb70 comp rx=0 tx=0).stop 2024-03-28T22:12:02.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.733+0000 7f8b3ed65640 1 -- 172.21.15.16:0/1113612302 shutdown_connections 2024-03-28T22:12:02.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.733+0000 7f8b3ed65640 1 --2- 172.21.15.16:0/1113612302 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8b381020a0 0x7f8b3810e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.733+0000 7f8b3ed65640 1 --2- 172.21.15.16:0/1113612302 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b381016e0 0x7f8b38101b60 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.733+0000 7f8b3ed65640 1 --2- 172.21.15.16:0/1113612302 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b381076f0 0x7f8b38107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.733+0000 7f8b3ed65640 1 -- 172.21.15.16:0/1113612302 >> 172.21.15.16:0/1113612302 conn(0x7f8b380fd1d0 msgr2=0x7f8b380ff5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:02.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.733+0000 7f8b3ed65640 1 -- 172.21.15.16:0/1113612302 shutdown_connections 2024-03-28T22:12:02.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.733+0000 7f8b3ed65640 1 -- 172.21.15.16:0/1113612302 wait complete. 2024-03-28T22:12:02.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.734+0000 7f8b3ed65640 1 Processor -- start 2024-03-28T22:12:02.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.734+0000 7f8b3ed65640 1 -- start start 2024-03-28T22:12:02.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.734+0000 7f8b3ed65640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8b381016e0 0x7f8b38197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.734+0000 7f8b3ed65640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b381020a0 0x7f8b381984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.734+0000 7f8b3ed65640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b381076f0 0x7f8b3819c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.734+0000 7f8b3ed65640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8b38111b00 con 0x7f8b381020a0 2024-03-28T22:12:02.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.734+0000 7f8b3ed65640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8b38111980 con 0x7f8b381076f0 2024-03-28T22:12:02.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.734+0000 7f8b3ed65640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8b38111c80 con 0x7f8b381016e0 2024-03-28T22:12:02.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.734+0000 7f8b37fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b381020a0 0x7f8b381984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.734+0000 7f8b37fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b381020a0 0x7f8b381984e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51126/0 (socket says 172.21.15.16:51126) 2024-03-28T22:12:02.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.734+0000 7f8b37fff640 1 -- 172.21.15.16:0/1496816344 learned_addr learned my addr 172.21.15.16:0/1496816344 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:02.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.734+0000 7f8b3d302640 1 --2- 172.21.15.16:0/1496816344 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b381076f0 0x7f8b3819c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.734+0000 7f8b3cb01640 1 --2- 172.21.15.16:0/1496816344 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8b381016e0 0x7f8b38197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.735+0000 7f8b37fff640 1 -- 172.21.15.16:0/1496816344 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8b381016e0 msgr2=0x7f8b38197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:02.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.735+0000 7f8b37fff640 1 --2- 172.21.15.16:0/1496816344 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8b381016e0 0x7f8b38197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.735+0000 7f8b37fff640 1 -- 172.21.15.16:0/1496816344 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b381076f0 msgr2=0x7f8b3819c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:02.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.735+0000 7f8b37fff640 1 --2- 172.21.15.16:0/1496816344 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b381076f0 0x7f8b3819c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.735+0000 7f8b37fff640 1 -- 172.21.15.16:0/1496816344 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b3819cf90 con 0x7f8b381020a0 2024-03-28T22:12:02.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.735+0000 7f8b37fff640 1 --2- 172.21.15.16:0/1496816344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b381020a0 0x7f8b381984e0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f8b24037040 tx=0x7f8b240045b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:02.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.735+0000 7f8b35ffb640 1 -- 172.21.15.16:0/1496816344 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b24041020 con 0x7f8b381020a0 2024-03-28T22:12:02.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.735+0000 7f8b3ed65640 1 -- 172.21.15.16:0/1496816344 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b3819d220 con 0x7f8b381020a0 2024-03-28T22:12:02.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.735+0000 7f8b35ffb640 1 -- 172.21.15.16:0/1496816344 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b2402c9f0 con 0x7f8b381020a0 2024-03-28T22:12:02.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.735+0000 7f8b35ffb640 1 -- 172.21.15.16:0/1496816344 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b24039040 con 0x7f8b381020a0 2024-03-28T22:12:02.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.735+0000 7f8b3ed65640 1 -- 172.21.15.16:0/1496816344 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b381a4b10 con 0x7f8b381020a0 2024-03-28T22:12:02.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.737+0000 7f8b35ffb640 1 -- 172.21.15.16:0/1496816344 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 25) ==== 136989+0+0 (secure 0 0 0) 0x7f8b24044020 con 0x7f8b381020a0 2024-03-28T22:12:02.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.738+0000 7f8b41a29740 1 Processor -- start 2024-03-28T22:12:02.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.738+0000 7f8b41a29740 1 -- start start 2024-03-28T22:12:02.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.738+0000 7f8b41a29740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56318dacd470 0x56318dacd850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.738+0000 7f8b35ffb640 1 --2- 172.21.15.16:0/1496816344 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f8b280a2010 0x7f8b280a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.738+0000 7f8b41a29740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56318dacdd90 0x56318dad6430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.738+0000 7f8b41a29740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56318dad6970 0x56318dad8d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.738+0000 7f8b41a29740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56318da311b0 con 0x56318dacd470 2024-03-28T22:12:02.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.738+0000 7f8b41a29740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x56318d9d9c00 con 0x56318dacdd90 2024-03-28T22:12:02.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.738+0000 7f8b41a29740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56318da2d250 con 0x56318dad6970 2024-03-28T22:12:02.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.738+0000 7f8b3d302640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56318dacd470 0x56318dacd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.738+0000 7f8b3cb01640 1 --2- 172.21.15.16:0/1496816344 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f8b280a2010 0x7f8b280a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.738+0000 7f8b3d302640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56318dacd470 0x56318dacd850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51134/0 (socket says 172.21.15.16:51134) 2024-03-28T22:12:02.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.738+0000 7f8b3d302640 1 -- 172.21.15.16:0/2959936907 learned_addr learned my addr 172.21.15.16:0/2959936907 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:02.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.738+0000 7f8b35ffb640 1 -- 172.21.15.16:0/1496816344 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f8b240efbc0 con 0x7f8b381020a0 2024-03-28T22:12:02.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.738+0000 7f8b37fff640 1 --2- 172.21.15.16:0/2959936907 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56318dacdd90 0x56318dad6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.738+0000 7f8b3d302640 1 --2- 172.21.15.16:0/2959936907 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56318dad6970 0x56318dad8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.738+0000 7f8b3d302640 1 -- 172.21.15.16:0/2959936907 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56318dad6970 msgr2=0x56318dad8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:02.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.738+0000 7f8b3d302640 1 --2- 172.21.15.16:0/2959936907 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56318dad6970 0x56318dad8d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.738+0000 7f8b3d302640 1 -- 172.21.15.16:0/2959936907 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56318dacdd90 msgr2=0x56318dad6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:02.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.738+0000 7f8b3d302640 1 --2- 172.21.15.16:0/2959936907 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56318dacdd90 0x56318dad6430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.738+0000 7f8b3d302640 1 -- 172.21.15.16:0/2959936907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56318d9f93d0 con 0x56318dacd470 2024-03-28T22:12:02.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.738+0000 7f8b37fff640 1 --2- 172.21.15.16:0/2959936907 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56318dacdd90 0x56318dad6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:12:02.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.739+0000 7f8b3d302640 1 --2- 172.21.15.16:0/2959936907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56318dacd470 0x56318dacd850 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f8b30013960 tx=0x7f8b30009be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=875495f26730c5f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:02.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.739+0000 7f8b3cb01640 1 --2- 172.21.15.16:0/1496816344 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f8b280a2010 0x7f8b280a44d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c005fd0 tx=0x7f8b2c009260 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:02.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.739+0000 7f8b22ffd640 1 -- 172.21.15.16:0/2959936907 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b30003970 con 0x56318dacd470 2024-03-28T22:12:02.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.739+0000 7f8b22ffd640 1 -- 172.21.15.16:0/2959936907 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b3000f8c0 con 0x56318dacd470 2024-03-28T22:12:02.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.739+0000 7f8b22ffd640 1 -- 172.21.15.16:0/2959936907 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b300045b0 con 0x56318dacd470 2024-03-28T22:12:02.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.739+0000 7f8b41a29740 1 -- 172.21.15.16:0/2959936907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56318dacd470 msgr2=0x56318dacd850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:02.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.739+0000 7f8b41a29740 1 --2- 172.21.15.16:0/2959936907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56318dacd470 0x56318dacd850 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f8b30013960 tx=0x7f8b30009be0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.740+0000 7f8b41a29740 1 -- 172.21.15.16:0/2959936907 shutdown_connections 2024-03-28T22:12:02.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.740+0000 7f8b41a29740 1 --2- 172.21.15.16:0/2959936907 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56318dad6970 0x56318dad8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.740+0000 7f8b41a29740 1 --2- 172.21.15.16:0/2959936907 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56318dacdd90 0x56318dad6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.740+0000 7f8b41a29740 1 --2- 172.21.15.16:0/2959936907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56318dacd470 0x56318dacd850 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.740+0000 7f8b41a29740 1 -- 172.21.15.16:0/2959936907 >> 172.21.15.16:0/2959936907 conn(0x56318dac7ba0 msgr2=0x56318dac7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:02.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.740+0000 7f8b41a29740 1 -- 172.21.15.16:0/2959936907 shutdown_connections 2024-03-28T22:12:02.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.740+0000 7f8b41a29740 1 -- 172.21.15.16:0/2959936907 wait complete. 2024-03-28T22:12:02.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.740+0000 7f8b41a29740 1 -- start start 2024-03-28T22:12:02.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.740+0000 7f8b41a29740 1 Processor -- start 2024-03-28T22:12:02.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.740+0000 7f8b41a29740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56318dacd470 0x56318dad3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.740+0000 7f8b41a29740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56318dacdd90 0x56318dad3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.740+0000 7f8b41a29740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56318dad6970 0x56318dad43e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.740+0000 7f8b41a29740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56318dad9760 con 0x56318dacd470 2024-03-28T22:12:02.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.740+0000 7f8b41a29740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x56318d9cf470 con 0x56318dad6970 2024-03-28T22:12:02.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.740+0000 7f8b41a29740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56318dad98e0 con 0x56318dacdd90 2024-03-28T22:12:02.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.740+0000 7f8b41a29740 10 client.0 _ll_register_callbacks cb 0x56318d9eccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:12:02.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.740+0000 7f8b3cb01640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56318dacd470 0x56318dad3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.740+0000 7f8b3cb01640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56318dacd470 0x56318dad3960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51144/0 (socket says 172.21.15.16:51144) 2024-03-28T22:12:02.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.740+0000 7f8b3cb01640 1 -- 172.21.15.16:0/479836412 learned_addr learned my addr 172.21.15.16:0/479836412 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:02.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.741+0000 7f8b3d302640 1 --2- 172.21.15.16:0/479836412 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56318dad6970 0x56318dad43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.741+0000 7f8b37fff640 1 --2- 172.21.15.16:0/479836412 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56318dacdd90 0x56318dad3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.741+0000 7f8b3cb01640 1 -- 172.21.15.16:0/479836412 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56318dacdd90 msgr2=0x56318dad3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:02.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.741+0000 7f8b3cb01640 1 --2- 172.21.15.16:0/479836412 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56318dacdd90 0x56318dad3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.741+0000 7f8b3cb01640 1 -- 172.21.15.16:0/479836412 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56318dad6970 msgr2=0x56318dad43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:02.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.741+0000 7f8b3cb01640 1 --2- 172.21.15.16:0/479836412 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56318dad6970 0x56318dad43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.741+0000 7f8b3cb01640 1 -- 172.21.15.16:0/479836412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56318d7453f0 con 0x56318dacd470 2024-03-28T22:12:02.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.741+0000 7f8b37fff640 1 --2- 172.21.15.16:0/479836412 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56318dacdd90 0x56318dad3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:12:02.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.741+0000 7f8b3cb01640 1 --2- 172.21.15.16:0/479836412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56318dacd470 0x56318dad3960 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c011c30 tx=0x7f8b2c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e9d9ded81609035 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:02.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.741+0000 7f8b17fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:12:02.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.741+0000 7f8b17fff640 1 -- 172.21.15.16:0/479836412 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b2c011e60 con 0x56318dacd470 2024-03-28T22:12:02.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.741+0000 7f8b17fff640 1 -- 172.21.15.16:0/479836412 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f8b2c010c20 con 0x56318dacd470 2024-03-28T22:12:02.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.741+0000 7f8b41a29740 1 -- 172.21.15.16:0/479836412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b3000f8c0 con 0x56318dacd470 2024-03-28T22:12:02.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.741+0000 7f8b17fff640 1 -- 172.21.15.16:0/479836412 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f8b2c010e40 con 0x56318dacd470 2024-03-28T22:12:02.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.741+0000 7f8b41a29740 10 client.15822 fetch_fsmap learned FSMap version 24 2024-03-28T22:12:02.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.741+0000 7f8b41a29740 1 -- 172.21.15.16:0/479836412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56318d80f7d0 con 0x56318dacd470 2024-03-28T22:12:02.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.741+0000 7f8b17fff640 1 -- 172.21.15.16:0/479836412 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b3000f8c0 con 0x56318dacd470 2024-03-28T22:12:02.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.742+0000 7f8b17fff640 1 -- 172.21.15.16:0/479836412 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7f8b2c02d020 con 0x56318dacd470 2024-03-28T22:12:02.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.742+0000 7f8b41a29740 10 client.15822 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:12:02.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.742+0000 7f8b41a29740 10 client.15822 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:12:02.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.742+0000 7f8b41a29740 20 client.15822 populate_metadata read hostname 'smithi016' 2024-03-28T22:12:02.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.742+0000 7f8b41a29740 1 --2- 172.21.15.16:0/479836412 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x56318dae6910 0x56318dae8d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.742+0000 7f8b41a29740 4 client.15822 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:12:02.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.742+0000 7f8b41a29740 1 -- 172.21.15.16:0/479836412 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56318da3e6f0 con 0x56318dae6910 2024-03-28T22:12:02.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.742+0000 7f8b37fff640 1 --2- 172.21.15.16:0/479836412 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x56318dae6910 0x56318dae8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.742+0000 7f8b17fff640 10 client.15822 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:12:02.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.742+0000 7f8b37fff640 1 --2- 172.21.15.16:0/479836412 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x56318dae6910 0x56318dae8d90 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1cc18107d4f6c376 server_cookie=65c9ee71c164b1f6 in_seq=0 out_seq=0 2024-03-28T22:12:02.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.743+0000 7f8b17fff640 1 -- 172.21.15.16:0/479836412 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56318da3e6f0 con 0x56318dae6910 2024-03-28T22:12:02.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.743+0000 7f8b17fff640 10 client.15822 handle_command_reply: tid=0 2024-03-28T22:12:02.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.743+0000 7f8b41a29740 1 client.15822 shutdown 2024-03-28T22:12:02.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.743+0000 7f8b41a29740 1 -- 172.21.15.16:0/479836412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56318dacd470 msgr2=0x56318dad3960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:02.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.743+0000 7f8b41a29740 1 --2- 172.21.15.16:0/479836412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56318dacd470 0x56318dad3960 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c011c30 tx=0x7f8b2c004730 comp rx=0 tx=0).stop 2024-03-28T22:12:02.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.743+0000 7f8b41a29740 1 -- 172.21.15.16:0/479836412 shutdown_connections 2024-03-28T22:12:02.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.743+0000 7f8b41a29740 1 --2- 172.21.15.16:0/479836412 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x56318dae6910 0x56318dae8d90 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.743+0000 7f8b41a29740 1 --2- 172.21.15.16:0/479836412 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56318dad6970 0x56318dad43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.743+0000 7f8b41a29740 1 --2- 172.21.15.16:0/479836412 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56318dacdd90 0x56318dad3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.743+0000 7f8b17fff640 0 client.15822 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:12:02.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.743+0000 7f8b41a29740 1 --2- 172.21.15.16:0/479836412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56318dacd470 0x56318dad3960 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.743+0000 7f8b41a29740 1 -- 172.21.15.16:0/479836412 >> 172.21.15.16:0/479836412 conn(0x56318dac7ba0 msgr2=0x56318dac98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:02.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.744+0000 7f8b41a29740 1 -- 172.21.15.16:0/479836412 shutdown_connections 2024-03-28T22:12:02.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.744+0000 7f8b41a29740 1 -- 172.21.15.16:0/479836412 wait complete. 2024-03-28T22:12:02.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.744+0000 7f8b41a29740 20 client.15822 trim_cache size 0 max 16384 2024-03-28T22:12:02.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.757+0000 7f8b41a29740 1 Processor -- start 2024-03-28T22:12:02.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.757+0000 7f8b41a29740 1 -- start start 2024-03-28T22:12:02.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.757+0000 7f8b41a29740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56318dacd470 0x56318dac9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.757+0000 7f8b41a29740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56318dacdd90 0x56318dae2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.757+0000 7f8b41a29740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56318dad6970 0x56318daee660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.757+0000 7f8b41a29740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8b381ada80 con 0x56318dad6970 2024-03-28T22:12:02.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.757+0000 7f8b41a29740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8b381ab320 con 0x56318dacdd90 2024-03-28T22:12:02.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.757+0000 7f8b41a29740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8b381a9190 con 0x56318dacd470 2024-03-28T22:12:02.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.757+0000 7f8b3d302640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56318dad6970 0x56318daee660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.757+0000 7f8b3d302640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56318dad6970 0x56318daee660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51156/0 (socket says 172.21.15.16:51156) 2024-03-28T22:12:02.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.757+0000 7f8b3d302640 1 -- 172.21.15.16:0/239861468 learned_addr learned my addr 172.21.15.16:0/239861468 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:02.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.757+0000 7f8b37fff640 1 --2- 172.21.15.16:0/239861468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56318dacdd90 0x56318dae2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.757+0000 7f8b3cb01640 1 --2- 172.21.15.16:0/239861468 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56318dacd470 0x56318dac9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.757+0000 7f8b3d302640 1 -- 172.21.15.16:0/239861468 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56318dacd470 msgr2=0x56318dac9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:02.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.757+0000 7f8b3d302640 1 --2- 172.21.15.16:0/239861468 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56318dacd470 0x56318dac9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.757+0000 7f8b3d302640 1 -- 172.21.15.16:0/239861468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56318dacdd90 msgr2=0x56318dae2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:02.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.757+0000 7f8b3d302640 1 --2- 172.21.15.16:0/239861468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56318dacdd90 0x56318dae2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.757+0000 7f8b3d302640 1 -- 172.21.15.16:0/239861468 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56318dac2150 con 0x56318dad6970 2024-03-28T22:12:02.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.757+0000 7f8b3cb01640 1 --2- 172.21.15.16:0/239861468 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56318dacd470 0x56318dac9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:12:02.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.758+0000 7f8b3d302640 1 --2- 172.21.15.16:0/239861468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56318dad6970 0x56318daee660 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f8b30004200 tx=0x7f8b30009d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b1b5aac355093f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:02.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.758+0000 7f8b227fc640 1 -- 172.21.15.16:0/239861468 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b3000f8c0 con 0x56318dad6970 2024-03-28T22:12:02.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.758+0000 7f8b227fc640 1 -- 172.21.15.16:0/239861468 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b30003970 con 0x56318dad6970 2024-03-28T22:12:02.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.758+0000 7f8b227fc640 1 -- 172.21.15.16:0/239861468 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b30005530 con 0x56318dad6970 2024-03-28T22:12:02.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.758+0000 7f8b41a29740 1 -- 172.21.15.16:0/239861468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56318dad6970 msgr2=0x56318daee660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:02.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.758+0000 7f8b41a29740 1 --2- 172.21.15.16:0/239861468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56318dad6970 0x56318daee660 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f8b30004200 tx=0x7f8b30009d40 comp rx=0 tx=0).stop 2024-03-28T22:12:02.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.758+0000 7f8b41a29740 1 -- 172.21.15.16:0/239861468 shutdown_connections 2024-03-28T22:12:02.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.758+0000 7f8b41a29740 1 --2- 172.21.15.16:0/239861468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56318dad6970 0x56318daee660 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.758+0000 7f8b41a29740 1 --2- 172.21.15.16:0/239861468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56318dacdd90 0x56318dae2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.758+0000 7f8b41a29740 1 --2- 172.21.15.16:0/239861468 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56318dacd470 0x56318dac9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.758+0000 7f8b41a29740 1 -- 172.21.15.16:0/239861468 >> 172.21.15.16:0/239861468 conn(0x56318dac7ba0 msgr2=0x56318dac6d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:02.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.758+0000 7f8b41a29740 1 -- 172.21.15.16:0/239861468 shutdown_connections 2024-03-28T22:12:02.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.758+0000 7f8b41a29740 1 -- 172.21.15.16:0/239861468 wait complete. 2024-03-28T22:12:02.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.759+0000 7f8b41a29740 1 -- start start 2024-03-28T22:12:02.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.759+0000 7f8b41a29740 1 Processor -- start 2024-03-28T22:12:02.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.759+0000 7f8b41a29740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56318dacd470 0x56318dadbb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.759+0000 7f8b41a29740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56318dacdd90 0x56318dadc040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.759+0000 7f8b41a29740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56318dad6970 0x56318dadc580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.759+0000 7f8b41a29740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8b381a97e0 con 0x56318dad6970 2024-03-28T22:12:02.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.759+0000 7f8b41a29740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8b381aac50 con 0x56318dacdd90 2024-03-28T22:12:02.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.759+0000 7f8b41a29740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8b381a9eb0 con 0x56318dacd470 2024-03-28T22:12:02.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.759+0000 7f8b41a29740 10 client.0 _ll_register_callbacks cb 0x7f8b2c0118b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:12:02.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.759+0000 7f8b3d302640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56318dad6970 0x56318dadc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.759+0000 7f8b3d302640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56318dad6970 0x56318dadc580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51162/0 (socket says 172.21.15.16:51162) 2024-03-28T22:12:02.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.759+0000 7f8b3d302640 1 -- 172.21.15.16:0/2741653116 learned_addr learned my addr 172.21.15.16:0/2741653116 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:02.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.759+0000 7f8b37fff640 1 --2- 172.21.15.16:0/2741653116 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56318dacdd90 0x56318dadc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.759+0000 7f8b3cb01640 1 --2- 172.21.15.16:0/2741653116 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56318dacd470 0x56318dadbb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.760+0000 7f8b3d302640 1 -- 172.21.15.16:0/2741653116 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56318dacd470 msgr2=0x56318dadbb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:02.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.760+0000 7f8b3d302640 1 --2- 172.21.15.16:0/2741653116 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56318dacd470 0x56318dadbb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.760+0000 7f8b3d302640 1 -- 172.21.15.16:0/2741653116 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56318dacdd90 msgr2=0x56318dadc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:02.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.760+0000 7f8b3d302640 1 --2- 172.21.15.16:0/2741653116 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56318dacdd90 0x56318dadc040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.760+0000 7f8b3d302640 1 -- 172.21.15.16:0/2741653116 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56318db4cca0 con 0x56318dad6970 2024-03-28T22:12:02.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.760+0000 7f8b3d302640 1 --2- 172.21.15.16:0/2741653116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56318dad6970 0x56318dadc580 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f8b30005fb0 tx=0x7f8b300133e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7398b2cf0da55ba7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:02.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.760+0000 7f8b17fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:12:02.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.760+0000 7f8b17fff640 1 -- 172.21.15.16:0/2741653116 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b3000fa50 con 0x56318dad6970 2024-03-28T22:12:02.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.760+0000 7f8b17fff640 1 -- 172.21.15.16:0/2741653116 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f8b3000fbf0 con 0x56318dad6970 2024-03-28T22:12:02.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.760+0000 7f8b41a29740 1 -- 172.21.15.16:0/2741653116 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b30003970 con 0x56318dad6970 2024-03-28T22:12:02.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.760+0000 7f8b17fff640 1 -- 172.21.15.16:0/2741653116 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f8b3003a050 con 0x56318dad6970 2024-03-28T22:12:02.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.760+0000 7f8b41a29740 10 client.15828 fetch_fsmap learned FSMap version 24 2024-03-28T22:12:02.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.760+0000 7f8b41a29740 1 -- 172.21.15.16:0/2741653116 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56318db4dee0 con 0x56318dad6970 2024-03-28T22:12:02.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.760+0000 7f8b17fff640 1 -- 172.21.15.16:0/2741653116 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b30003970 con 0x56318dad6970 2024-03-28T22:12:02.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.760+0000 7f8b17fff640 1 -- 172.21.15.16:0/2741653116 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7f8b300216e0 con 0x56318dad6970 2024-03-28T22:12:02.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.760+0000 7f8b41a29740 10 client.15828 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:12:02.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.760+0000 7f8b41a29740 10 client.15828 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:12:02.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.760+0000 7f8b41a29740 20 client.15828 populate_metadata read hostname 'smithi016' 2024-03-28T22:12:02.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.761+0000 7f8b41a29740 1 --2- 172.21.15.16:0/2741653116 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x56318db4e400 0x56318db50820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:02.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.761+0000 7f8b41a29740 4 client.15828 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:12:02.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.761+0000 7f8b41a29740 1 -- 172.21.15.16:0/2741653116 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56318da38a40 con 0x56318db4e400 2024-03-28T22:12:02.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.761+0000 7f8b3cb01640 1 --2- 172.21.15.16:0/2741653116 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x56318db4e400 0x56318db50820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:02.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.761+0000 7f8b3cb01640 1 --2- 172.21.15.16:0/2741653116 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x56318db4e400 0x56318db50820 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f99ebe98e36f3569 server_cookie=49b36c861e6742ee in_seq=0 out_seq=0 2024-03-28T22:12:02.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.761+0000 7f8b17fff640 10 client.15828 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:12:02.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.761+0000 7f8b17fff640 1 -- 172.21.15.16:0/2741653116 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56318da38a40 con 0x56318db4e400 2024-03-28T22:12:02.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.761+0000 7f8b17fff640 10 client.15828 handle_command_reply: tid=0 2024-03-28T22:12:02.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.761+0000 7f8b41a29740 1 client.15828 shutdown 2024-03-28T22:12:02.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.761+0000 7f8b41a29740 1 -- 172.21.15.16:0/2741653116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56318dad6970 msgr2=0x56318dadc580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:02.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.761+0000 7f8b41a29740 1 --2- 172.21.15.16:0/2741653116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56318dad6970 0x56318dadc580 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f8b30005fb0 tx=0x7f8b300133e0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.761+0000 7f8b41a29740 1 -- 172.21.15.16:0/2741653116 shutdown_connections 2024-03-28T22:12:02.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.761+0000 7f8b41a29740 1 --2- 172.21.15.16:0/2741653116 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x56318db4e400 0x56318db50820 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.761+0000 7f8b41a29740 1 --2- 172.21.15.16:0/2741653116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56318dad6970 0x56318dadc580 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.761+0000 7f8b41a29740 1 --2- 172.21.15.16:0/2741653116 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56318dacdd90 0x56318dadc040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.761+0000 7f8b17fff640 0 client.15828 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:12:02.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.761+0000 7f8b41a29740 1 --2- 172.21.15.16:0/2741653116 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56318dacd470 0x56318dadbb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.761+0000 7f8b41a29740 1 -- 172.21.15.16:0/2741653116 >> 172.21.15.16:0/2741653116 conn(0x56318dac7ba0 msgr2=0x56318dad3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:02.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.761+0000 7f8b41a29740 1 -- 172.21.15.16:0/2741653116 shutdown_connections 2024-03-28T22:12:02.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.761+0000 7f8b41a29740 1 -- 172.21.15.16:0/2741653116 wait complete. 2024-03-28T22:12:02.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.762+0000 7f8b41a29740 20 client.15828 trim_cache size 0 max 16384 2024-03-28T22:12:02.770 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:12:02.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.762+0000 7f8b3ed65640 1 -- 172.21.15.16:0/1496816344 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f8b280a2010 msgr2=0x7f8b280a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:02.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.762+0000 7f8b3ed65640 1 --2- 172.21.15.16:0/1496816344 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f8b280a2010 0x7f8b280a44d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c005fd0 tx=0x7f8b2c009260 comp rx=0 tx=0).stop 2024-03-28T22:12:02.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.762+0000 7f8b3ed65640 1 -- 172.21.15.16:0/1496816344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b381020a0 msgr2=0x7f8b381984e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:02.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.762+0000 7f8b3ed65640 1 --2- 172.21.15.16:0/1496816344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b381020a0 0x7f8b381984e0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f8b24037040 tx=0x7f8b240045b0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.762+0000 7f8b3ed65640 1 -- 172.21.15.16:0/1496816344 shutdown_connections 2024-03-28T22:12:02.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.762+0000 7f8b3ed65640 1 --2- 172.21.15.16:0/1496816344 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f8b280a2010 0x7f8b280a44d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.762+0000 7f8b3ed65640 1 --2- 172.21.15.16:0/1496816344 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b381076f0 0x7f8b3819c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.762+0000 7f8b3ed65640 1 --2- 172.21.15.16:0/1496816344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b381020a0 0x7f8b381984e0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.762+0000 7f8b3ed65640 1 --2- 172.21.15.16:0/1496816344 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8b381016e0 0x7f8b38197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:02.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.762+0000 7f8b3ed65640 1 -- 172.21.15.16:0/1496816344 >> 172.21.15.16:0/1496816344 conn(0x7f8b380fd1d0 msgr2=0x7f8b380fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:02.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.762+0000 7f8b3ed65640 1 -- 172.21.15.16:0/1496816344 shutdown_connections 2024-03-28T22:12:02.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:02.762+0000 7f8b3ed65640 1 -- 172.21.15.16:0/1496816344 wait complete. 2024-03-28T22:12:02.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:02 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2870844085' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:12:02.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:02 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2870844085' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:12:03.775 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:12:03.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:12:03.775 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:12:03.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:03 smithi053 ceph-mon[35979]: pgmap v47: 97 pgs: 97 active+clean; 20 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 655 KiB/s rd, 1.7 MiB/s wr, 40 op/s 2024-03-28T22:12:03.804 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:03 smithi016 ceph-mon[32374]: pgmap v47: 97 pgs: 97 active+clean; 20 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 655 KiB/s rd, 1.7 MiB/s wr, 40 op/s 2024-03-28T22:12:03.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.892+0000 7f883d08f640 1 -- 172.21.15.16:0/4084935711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8838107700 msgr2=0x7f8838107ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:03.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.892+0000 7f883d08f640 1 --2- 172.21.15.16:0/4084935711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8838107700 0x7f8838107ae0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f8828009930 tx=0x7f882802bc30 comp rx=0 tx=0).stop 2024-03-28T22:12:03.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.892+0000 7f883d08f640 1 -- 172.21.15.16:0/4084935711 shutdown_connections 2024-03-28T22:12:03.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.892+0000 7f883d08f640 1 --2- 172.21.15.16:0/4084935711 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88381020b0 0x7f883810e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:03.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.892+0000 7f883d08f640 1 --2- 172.21.15.16:0/4084935711 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88381016f0 0x7f8838101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:03.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.892+0000 7f883d08f640 1 --2- 172.21.15.16:0/4084935711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8838107700 0x7f8838107ae0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:03.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.892+0000 7f883d08f640 1 -- 172.21.15.16:0/4084935711 >> 172.21.15.16:0/4084935711 conn(0x7f88380fd1e0 msgr2=0x7f88380ff600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:03.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.892+0000 7f883d08f640 1 -- 172.21.15.16:0/4084935711 shutdown_connections 2024-03-28T22:12:03.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.892+0000 7f883d08f640 1 -- 172.21.15.16:0/4084935711 wait complete. 2024-03-28T22:12:03.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.893+0000 7f883d08f640 1 Processor -- start 2024-03-28T22:12:03.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.893+0000 7f883d08f640 1 -- start start 2024-03-28T22:12:03.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.893+0000 7f883d08f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88381016f0 0x7f8838198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:03.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.893+0000 7f883d08f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88381020b0 0x7f88381985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:03.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.893+0000 7f883d08f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8838107700 0x7f883819c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:03.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.893+0000 7f883d08f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8838111b10 con 0x7f88381020b0 2024-03-28T22:12:03.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.893+0000 7f883d08f640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8838111990 con 0x7f8838107700 2024-03-28T22:12:03.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.893+0000 7f883d08f640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8838111c90 con 0x7f88381016f0 2024-03-28T22:12:03.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.893+0000 7f883659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88381020b0 0x7f88381985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:03.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.893+0000 7f883659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88381020b0 0x7f88381985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51168/0 (socket says 172.21.15.16:51168) 2024-03-28T22:12:03.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.893+0000 7f883659c640 1 -- 172.21.15.16:0/2123621395 learned_addr learned my addr 172.21.15.16:0/2123621395 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:03.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.893+0000 7f883759e640 1 --2- 172.21.15.16:0/2123621395 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8838107700 0x7f883819c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:03.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.893+0000 7f8836d9d640 1 --2- 172.21.15.16:0/2123621395 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88381016f0 0x7f8838198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:03.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.894+0000 7f883659c640 1 -- 172.21.15.16:0/2123621395 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88381016f0 msgr2=0x7f8838198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:03.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.894+0000 7f883659c640 1 --2- 172.21.15.16:0/2123621395 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88381016f0 0x7f8838198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:03.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.894+0000 7f883659c640 1 -- 172.21.15.16:0/2123621395 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8838107700 msgr2=0x7f883819c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:03.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.894+0000 7f883659c640 1 --2- 172.21.15.16:0/2123621395 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8838107700 0x7f883819c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:03.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.894+0000 7f883659c640 1 -- 172.21.15.16:0/2123621395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f883819d010 con 0x7f88381020b0 2024-03-28T22:12:03.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.894+0000 7f883659c640 1 --2- 172.21.15.16:0/2123621395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88381020b0 0x7f88381985b0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f882c00d770 tx=0x7f882c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:03.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.894+0000 7f881ffff640 1 -- 172.21.15.16:0/2123621395 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f882c017070 con 0x7f88381020b0 2024-03-28T22:12:03.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.894+0000 7f883d08f640 1 -- 172.21.15.16:0/2123621395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f883819d300 con 0x7f88381020b0 2024-03-28T22:12:03.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.894+0000 7f881ffff640 1 -- 172.21.15.16:0/2123621395 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f882c005500 con 0x7f88381020b0 2024-03-28T22:12:03.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.894+0000 7f881ffff640 1 -- 172.21.15.16:0/2123621395 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f882c004e00 con 0x7f88381020b0 2024-03-28T22:12:03.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.894+0000 7f883d08f640 1 -- 172.21.15.16:0/2123621395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88381a4b90 con 0x7f88381020b0 2024-03-28T22:12:03.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.896+0000 7f881ffff640 1 -- 172.21.15.16:0/2123621395 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 25) ==== 136989+0+0 (secure 0 0 0) 0x7f882c004fa0 con 0x7f88381020b0 2024-03-28T22:12:03.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.896+0000 7f883fc76740 1 Processor -- start 2024-03-28T22:12:03.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.897+0000 7f883fc76740 1 -- start start 2024-03-28T22:12:03.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.897+0000 7f883fc76740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55707de9d9f0 0x55707de9ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:03.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.897+0000 7f883fc76740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55707de9e310 0x55707dea69b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:03.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.897+0000 7f883fc76740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55707dea6ef0 0x55707dea92e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:03.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.897+0000 7f883fc76740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55707dd9e0b0 con 0x55707dea6ef0 2024-03-28T22:12:03.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.897+0000 7f883fc76740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55707de00ea0 con 0x55707de9e310 2024-03-28T22:12:03.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.897+0000 7f883fc76740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55707dda9c00 con 0x55707de9d9f0 2024-03-28T22:12:03.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.897+0000 7f881ffff640 1 --2- 172.21.15.16:0/2123621395 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f88200a2010 0x7f88200a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:03.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.897+0000 7f883659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55707dea6ef0 0x55707dea92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:03.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.897+0000 7f8836d9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55707de9e310 0x55707dea69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:03.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.897+0000 7f883759e640 1 --2- 172.21.15.16:0/2123621395 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f88200a2010 0x7f88200a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:03.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.897+0000 7f8836d9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55707de9e310 0x55707dea69b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:52336/0 (socket says 172.21.15.16:52336) 2024-03-28T22:12:03.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.897+0000 7f8836d9d640 1 -- 172.21.15.16:0/3495401818 learned_addr learned my addr 172.21.15.16:0/3495401818 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:03.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.897+0000 7f883759e640 1 --2- 172.21.15.16:0/3495401818 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55707de9d9f0 0x55707de9ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:03.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.897+0000 7f881ffff640 1 -- 172.21.15.16:0/2123621395 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f882c0cddd0 con 0x7f88381020b0 2024-03-28T22:12:03.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.897+0000 7f8836d9d640 1 -- 172.21.15.16:0/3495401818 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55707de9d9f0 msgr2=0x55707de9ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:03.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.897+0000 7f8836d9d640 1 --2- 172.21.15.16:0/3495401818 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55707de9d9f0 0x55707de9ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:03.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.897+0000 7f8836d9d640 1 -- 172.21.15.16:0/3495401818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55707dea6ef0 msgr2=0x55707dea92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:03.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.897+0000 7f8836d9d640 1 --2- 172.21.15.16:0/3495401818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55707dea6ef0 0x55707dea92e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:03.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.897+0000 7f8836d9d640 1 -- 172.21.15.16:0/3495401818 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55707dbdd7e0 con 0x55707de9e310 2024-03-28T22:12:03.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.897+0000 7f883659c640 1 --2- 172.21.15.16:0/3495401818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55707dea6ef0 0x55707dea92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:12:03.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.897+0000 7f883759e640 1 --2- 172.21.15.16:0/2123621395 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f88200a2010 0x7f88200a44d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8830005fd0 tx=0x7f8830009bd0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:03.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.898+0000 7f8836d9d640 1 --2- 172.21.15.16:0/3495401818 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55707de9e310 0x55707dea69b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f882802cfd0 tx=0x7f8828005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e22fe1c7e2659e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:03.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.898+0000 7f881d7fa640 1 -- 172.21.15.16:0/3495401818 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8828042070 con 0x55707de9e310 2024-03-28T22:12:03.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.898+0000 7f881d7fa640 1 -- 172.21.15.16:0/3495401818 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8828004030 con 0x55707de9e310 2024-03-28T22:12:03.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.898+0000 7f881d7fa640 1 -- 172.21.15.16:0/3495401818 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8828004c30 con 0x55707de9e310 2024-03-28T22:12:03.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.898+0000 7f883fc76740 1 -- 172.21.15.16:0/3495401818 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55707de9e310 msgr2=0x55707dea69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:03.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.898+0000 7f883fc76740 1 --2- 172.21.15.16:0/3495401818 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55707de9e310 0x55707dea69b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f882802cfd0 tx=0x7f8828005d50 comp rx=0 tx=0).stop 2024-03-28T22:12:03.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.898+0000 7f883fc76740 1 -- 172.21.15.16:0/3495401818 shutdown_connections 2024-03-28T22:12:03.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.898+0000 7f883fc76740 1 --2- 172.21.15.16:0/3495401818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55707dea6ef0 0x55707dea92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:03.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.898+0000 7f883fc76740 1 --2- 172.21.15.16:0/3495401818 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55707de9e310 0x55707dea69b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:03.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.898+0000 7f883fc76740 1 --2- 172.21.15.16:0/3495401818 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55707de9d9f0 0x55707de9ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:03.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.898+0000 7f883fc76740 1 -- 172.21.15.16:0/3495401818 >> 172.21.15.16:0/3495401818 conn(0x55707de98120 msgr2=0x55707de98500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:03.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.898+0000 7f883fc76740 1 -- 172.21.15.16:0/3495401818 shutdown_connections 2024-03-28T22:12:03.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.898+0000 7f883fc76740 1 -- 172.21.15.16:0/3495401818 wait complete. 2024-03-28T22:12:03.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.899+0000 7f883fc76740 1 -- start start 2024-03-28T22:12:03.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.899+0000 7f883fc76740 1 Processor -- start 2024-03-28T22:12:03.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.899+0000 7f883fc76740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55707de9d9f0 0x55707dea3f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:03.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.899+0000 7f883fc76740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55707de9e310 0x55707dea4490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:03.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.899+0000 7f883fc76740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55707dea6ef0 0x55707dea49d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:03.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.899+0000 7f883fc76740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55707dd9f3d0 con 0x55707de9d9f0 2024-03-28T22:12:03.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.899+0000 7f883fc76740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55707ddfcf40 con 0x55707dea6ef0 2024-03-28T22:12:03.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.899+0000 7f883fc76740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55707dea9c80 con 0x55707de9e310 2024-03-28T22:12:03.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.899+0000 7f883fc76740 10 client.0 _ll_register_callbacks cb 0x55707ddbcb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:12:03.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.899+0000 7f8836d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55707de9d9f0 0x55707dea3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:03.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.899+0000 7f8836d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55707de9d9f0 0x55707dea3f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51176/0 (socket says 172.21.15.16:51176) 2024-03-28T22:12:03.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.899+0000 7f8836d9d640 1 -- 172.21.15.16:0/1337951457 learned_addr learned my addr 172.21.15.16:0/1337951457 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:03.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.899+0000 7f883759e640 1 --2- 172.21.15.16:0/1337951457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55707dea6ef0 0x55707dea49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:03.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.899+0000 7f883659c640 1 --2- 172.21.15.16:0/1337951457 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55707de9e310 0x55707dea4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:03.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.899+0000 7f8836d9d640 1 -- 172.21.15.16:0/1337951457 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55707de9e310 msgr2=0x55707dea4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:03.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.899+0000 7f8836d9d640 1 --2- 172.21.15.16:0/1337951457 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55707de9e310 0x55707dea4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:03.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.899+0000 7f8836d9d640 1 -- 172.21.15.16:0/1337951457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55707dea6ef0 msgr2=0x55707dea49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:03.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.899+0000 7f8836d9d640 1 --2- 172.21.15.16:0/1337951457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55707dea6ef0 0x55707dea49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:03.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.899+0000 7f8836d9d640 1 -- 172.21.15.16:0/1337951457 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55707dc3df70 con 0x55707de9d9f0 2024-03-28T22:12:03.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.900+0000 7f8836d9d640 1 --2- 172.21.15.16:0/1337951457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55707de9d9f0 0x55707dea3f50 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f8828005ab0 tx=0x7f8828005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c90ef342c4ecab2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:03.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.900+0000 7f881aff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:12:03.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.900+0000 7f881aff5640 1 -- 172.21.15.16:0/1337951457 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f882804e060 con 0x55707de9d9f0 2024-03-28T22:12:03.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.900+0000 7f881aff5640 1 -- 172.21.15.16:0/1337951457 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f882802ca90 con 0x55707de9d9f0 2024-03-28T22:12:03.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.900+0000 7f883fc76740 1 -- 172.21.15.16:0/1337951457 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8828004030 con 0x55707de9d9f0 2024-03-28T22:12:03.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.900+0000 7f881aff5640 1 -- 172.21.15.16:0/1337951457 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f8828042070 con 0x55707de9d9f0 2024-03-28T22:12:03.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.900+0000 7f883fc76740 10 client.15840 fetch_fsmap learned FSMap version 24 2024-03-28T22:12:03.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.900+0000 7f883fc76740 1 -- 172.21.15.16:0/1337951457 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55707dad5170 con 0x55707de9d9f0 2024-03-28T22:12:03.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.900+0000 7f881aff5640 1 -- 172.21.15.16:0/1337951457 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8828004030 con 0x55707de9d9f0 2024-03-28T22:12:03.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.900+0000 7f881aff5640 1 -- 172.21.15.16:0/1337951457 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7f8828003850 con 0x55707de9d9f0 2024-03-28T22:12:03.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.900+0000 7f883fc76740 10 client.15840 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:12:03.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.900+0000 7f883fc76740 10 client.15840 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:12:03.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.900+0000 7f883fc76740 20 client.15840 populate_metadata read hostname 'smithi016' 2024-03-28T22:12:03.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.900+0000 7f883fc76740 1 --2- 172.21.15.16:0/1337951457 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55707deb6bc0 0x55707deb8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:03.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.901+0000 7f883fc76740 4 client.15840 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:12:03.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.901+0000 7f883fc76740 1 -- 172.21.15.16:0/1337951457 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55707de0e3e0 con 0x55707deb6bc0 2024-03-28T22:12:03.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.901+0000 7f883659c640 1 --2- 172.21.15.16:0/1337951457 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55707deb6bc0 0x55707deb8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:03.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.901+0000 7f883659c640 1 --2- 172.21.15.16:0/1337951457 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55707deb6bc0 0x55707deb8fb0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7dd9a8a3c23874be server_cookie=12eab2b9114ca988 in_seq=0 out_seq=0 2024-03-28T22:12:03.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.901+0000 7f881aff5640 10 client.15840 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:12:03.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.902+0000 7f881aff5640 1 -- 172.21.15.16:0/1337951457 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55707de0e3e0 con 0x55707deb6bc0 2024-03-28T22:12:03.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.902+0000 7f881aff5640 10 client.15840 handle_command_reply: tid=0 2024-03-28T22:12:03.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.902+0000 7f883fc76740 1 client.15840 shutdown 2024-03-28T22:12:03.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.902+0000 7f883fc76740 1 -- 172.21.15.16:0/1337951457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55707de9d9f0 msgr2=0x55707dea3f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:03.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.902+0000 7f883fc76740 1 --2- 172.21.15.16:0/1337951457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55707de9d9f0 0x55707dea3f50 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f8828005ab0 tx=0x7f8828005c00 comp rx=0 tx=0).stop 2024-03-28T22:12:03.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.902+0000 7f883fc76740 1 -- 172.21.15.16:0/1337951457 shutdown_connections 2024-03-28T22:12:03.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.902+0000 7f883fc76740 1 --2- 172.21.15.16:0/1337951457 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55707deb6bc0 0x55707deb8fb0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:03.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.902+0000 7f883fc76740 1 --2- 172.21.15.16:0/1337951457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55707dea6ef0 0x55707dea49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:03.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.902+0000 7f883fc76740 1 --2- 172.21.15.16:0/1337951457 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55707de9e310 0x55707dea4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:03.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.902+0000 7f883fc76740 1 --2- 172.21.15.16:0/1337951457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55707de9d9f0 0x55707dea3f50 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:03.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.902+0000 7f883fc76740 1 -- 172.21.15.16:0/1337951457 >> 172.21.15.16:0/1337951457 conn(0x55707de98120 msgr2=0x55707de99e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:03.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.902+0000 7f881aff5640 0 client.15840 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:12:03.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.902+0000 7f883fc76740 1 -- 172.21.15.16:0/1337951457 shutdown_connections 2024-03-28T22:12:03.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.902+0000 7f883fc76740 1 -- 172.21.15.16:0/1337951457 wait complete. 2024-03-28T22:12:03.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.902+0000 7f883fc76740 20 client.15840 trim_cache size 0 max 16384 2024-03-28T22:12:03.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.920+0000 7f883fc76740 1 Processor -- start 2024-03-28T22:12:03.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.920+0000 7f883fc76740 1 -- start start 2024-03-28T22:12:03.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.920+0000 7f883fc76740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55707de9d9f0 0x55707dea6680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:03.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.920+0000 7f883fc76740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55707de9e310 0x55707deb28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:03.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.920+0000 7f883fc76740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55707dea6ef0 0x55707debe660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:03.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.920+0000 7f883fc76740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f88381afb30 con 0x55707de9e310 2024-03-28T22:12:03.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.920+0000 7f883fc76740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f88381ad3d0 con 0x55707de9d9f0 2024-03-28T22:12:03.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.920+0000 7f883fc76740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f88381ab240 con 0x55707dea6ef0 2024-03-28T22:12:03.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.920+0000 7f883659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55707de9e310 0x55707deb28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:03.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.920+0000 7f883659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55707de9e310 0x55707deb28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51186/0 (socket says 172.21.15.16:51186) 2024-03-28T22:12:03.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.920+0000 7f883659c640 1 -- 172.21.15.16:0/2016862968 learned_addr learned my addr 172.21.15.16:0/2016862968 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:03.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.920+0000 7f8836d9d640 1 --2- 172.21.15.16:0/2016862968 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55707de9d9f0 0x55707dea6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:03.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.920+0000 7f883659c640 1 -- 172.21.15.16:0/2016862968 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55707dea6ef0 msgr2=0x55707debe660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:03.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.920+0000 7f883659c640 1 --2- 172.21.15.16:0/2016862968 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55707dea6ef0 0x55707debe660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:03.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.920+0000 7f883659c640 1 -- 172.21.15.16:0/2016862968 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55707de9d9f0 msgr2=0x55707dea6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:03.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.920+0000 7f883659c640 1 --2- 172.21.15.16:0/2016862968 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55707de9d9f0 0x55707dea6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:03.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.920+0000 7f883659c640 1 -- 172.21.15.16:0/2016862968 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55707de932c0 con 0x55707de9e310 2024-03-28T22:12:03.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.920+0000 7f883659c640 1 --2- 172.21.15.16:0/2016862968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55707de9e310 0x55707deb28a0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f882c009610 tx=0x7f882c06d6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6431e38e7e09f36 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:03.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.920+0000 7f881cff9640 1 -- 172.21.15.16:0/2016862968 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f882c020070 con 0x55707de9e310 2024-03-28T22:12:03.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.920+0000 7f881cff9640 1 -- 172.21.15.16:0/2016862968 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f882c0a2d30 con 0x55707de9e310 2024-03-28T22:12:03.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.920+0000 7f881cff9640 1 -- 172.21.15.16:0/2016862968 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f882c0790b0 con 0x55707de9e310 2024-03-28T22:12:03.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.921+0000 7f883fc76740 1 -- 172.21.15.16:0/2016862968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55707de9e310 msgr2=0x55707deb28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:03.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.921+0000 7f883fc76740 1 --2- 172.21.15.16:0/2016862968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55707de9e310 0x55707deb28a0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f882c009610 tx=0x7f882c06d6b0 comp rx=0 tx=0).stop 2024-03-28T22:12:03.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.921+0000 7f883fc76740 1 -- 172.21.15.16:0/2016862968 shutdown_connections 2024-03-28T22:12:03.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.921+0000 7f883fc76740 1 --2- 172.21.15.16:0/2016862968 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55707dea6ef0 0x55707debe660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:03.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.921+0000 7f883fc76740 1 --2- 172.21.15.16:0/2016862968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55707de9e310 0x55707deb28a0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:03.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.921+0000 7f883fc76740 1 --2- 172.21.15.16:0/2016862968 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55707de9d9f0 0x55707dea6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:03.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.921+0000 7f883fc76740 1 -- 172.21.15.16:0/2016862968 >> 172.21.15.16:0/2016862968 conn(0x55707de98120 msgr2=0x55707dea3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:03.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.921+0000 7f883fc76740 1 -- 172.21.15.16:0/2016862968 shutdown_connections 2024-03-28T22:12:03.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.921+0000 7f883fc76740 1 -- 172.21.15.16:0/2016862968 wait complete. 2024-03-28T22:12:03.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.921+0000 7f883fc76740 1 -- start start 2024-03-28T22:12:03.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.921+0000 7f883fc76740 1 Processor -- start 2024-03-28T22:12:03.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.922+0000 7f883fc76740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55707de9d9f0 0x55707deaf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:03.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.922+0000 7f883fc76740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55707de9e310 0x55707deabea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:03.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.922+0000 7f883fc76740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55707dea6ef0 0x55707deac3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:03.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.922+0000 7f883fc76740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f88381ab890 con 0x55707de9d9f0 2024-03-28T22:12:03.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.922+0000 7f883fc76740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f88381acd00 con 0x55707dea6ef0 2024-03-28T22:12:03.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.922+0000 7f883fc76740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f88381abf60 con 0x55707de9e310 2024-03-28T22:12:03.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.922+0000 7f883fc76740 10 client.0 _ll_register_callbacks cb 0x7f8828004530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:12:03.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.922+0000 7f8836d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55707de9d9f0 0x55707deaf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:03.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.922+0000 7f8836d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55707de9d9f0 0x55707deaf0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51194/0 (socket says 172.21.15.16:51194) 2024-03-28T22:12:03.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.922+0000 7f8836d9d640 1 -- 172.21.15.16:0/2154496628 learned_addr learned my addr 172.21.15.16:0/2154496628 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:03.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.922+0000 7f883759e640 1 --2- 172.21.15.16:0/2154496628 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55707dea6ef0 0x55707deac3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:03.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.922+0000 7f883659c640 1 --2- 172.21.15.16:0/2154496628 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55707de9e310 0x55707deabea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:03.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.922+0000 7f8836d9d640 1 -- 172.21.15.16:0/2154496628 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55707de9e310 msgr2=0x55707deabea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:03.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.922+0000 7f8836d9d640 1 --2- 172.21.15.16:0/2154496628 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55707de9e310 0x55707deabea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:03.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.922+0000 7f8836d9d640 1 -- 172.21.15.16:0/2154496628 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55707dea6ef0 msgr2=0x55707deac3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:03.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.922+0000 7f8836d9d640 1 --2- 172.21.15.16:0/2154496628 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55707dea6ef0 0x55707deac3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:03.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.922+0000 7f8836d9d640 1 -- 172.21.15.16:0/2154496628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55707deac9a0 con 0x55707de9d9f0 2024-03-28T22:12:03.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.922+0000 7f883759e640 1 --2- 172.21.15.16:0/2154496628 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55707dea6ef0 0x55707deac3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:12:03.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.922+0000 7f883659c640 1 --2- 172.21.15.16:0/2154496628 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55707de9e310 0x55707deabea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:12:03.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.922+0000 7f8836d9d640 1 --2- 172.21.15.16:0/2154496628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55707de9d9f0 0x55707deaf0c0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f8828003bf0 tx=0x7f8828005e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a49249836dea4083 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:03.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.922+0000 7f881aff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:12:03.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.922+0000 7f881aff5640 1 -- 172.21.15.16:0/2154496628 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f882804e030 con 0x55707de9d9f0 2024-03-28T22:12:03.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.922+0000 7f881aff5640 1 -- 172.21.15.16:0/2154496628 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f882802cc90 con 0x55707de9d9f0 2024-03-28T22:12:03.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.922+0000 7f883fc76740 1 -- 172.21.15.16:0/2154496628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f882c0a2d30 con 0x55707de9d9f0 2024-03-28T22:12:03.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.922+0000 7f881aff5640 1 -- 172.21.15.16:0/2154496628 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f88280035f0 con 0x55707de9d9f0 2024-03-28T22:12:03.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.922+0000 7f883fc76740 10 client.15846 fetch_fsmap learned FSMap version 24 2024-03-28T22:12:03.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.922+0000 7f883fc76740 1 -- 172.21.15.16:0/2154496628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55707df1dad0 con 0x55707de9d9f0 2024-03-28T22:12:03.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.923+0000 7f881aff5640 1 -- 172.21.15.16:0/2154496628 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f882c0a2d30 con 0x55707de9d9f0 2024-03-28T22:12:03.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.923+0000 7f881aff5640 1 -- 172.21.15.16:0/2154496628 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7f8828034440 con 0x55707de9d9f0 2024-03-28T22:12:03.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.923+0000 7f883fc76740 10 client.15846 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:12:03.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.923+0000 7f883fc76740 10 client.15846 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:12:03.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.923+0000 7f883fc76740 20 client.15846 populate_metadata read hostname 'smithi016' 2024-03-28T22:12:03.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.923+0000 7f883fc76740 1 --2- 172.21.15.16:0/2154496628 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55707df1e280 0x55707df206a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:03.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.923+0000 7f883fc76740 4 client.15846 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:12:03.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.923+0000 7f883fc76740 1 -- 172.21.15.16:0/2154496628 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55707de08730 con 0x55707df1e280 2024-03-28T22:12:03.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.923+0000 7f883659c640 1 --2- 172.21.15.16:0/2154496628 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55707df1e280 0x55707df206a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:03.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.923+0000 7f881aff5640 10 client.15846 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:12:03.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.923+0000 7f883659c640 1 --2- 172.21.15.16:0/2154496628 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55707df1e280 0x55707df206a0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c27188d6cddaa971 server_cookie=2686e49ba9fa5244 in_seq=0 out_seq=0 2024-03-28T22:12:03.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.924+0000 7f881aff5640 1 -- 172.21.15.16:0/2154496628 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55707de08730 con 0x55707df1e280 2024-03-28T22:12:03.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.924+0000 7f881aff5640 10 client.15846 handle_command_reply: tid=0 2024-03-28T22:12:03.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.924+0000 7f883fc76740 1 client.15846 shutdown 2024-03-28T22:12:03.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.925+0000 7f883fc76740 1 -- 172.21.15.16:0/2154496628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55707de9d9f0 msgr2=0x55707deaf0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:03.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.925+0000 7f883fc76740 1 --2- 172.21.15.16:0/2154496628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55707de9d9f0 0x55707deaf0c0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f8828003bf0 tx=0x7f8828005e20 comp rx=0 tx=0).stop 2024-03-28T22:12:03.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.925+0000 7f883fc76740 1 -- 172.21.15.16:0/2154496628 shutdown_connections 2024-03-28T22:12:03.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.925+0000 7f883fc76740 1 --2- 172.21.15.16:0/2154496628 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55707df1e280 0x55707df206a0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:03.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.925+0000 7f883fc76740 1 --2- 172.21.15.16:0/2154496628 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55707dea6ef0 0x55707deac3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:03.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.925+0000 7f883fc76740 1 --2- 172.21.15.16:0/2154496628 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55707de9e310 0x55707deabea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:03.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.925+0000 7f883fc76740 1 --2- 172.21.15.16:0/2154496628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55707de9d9f0 0x55707deaf0c0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:03.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.925+0000 7f883fc76740 1 -- 172.21.15.16:0/2154496628 >> 172.21.15.16:0/2154496628 conn(0x55707de98120 msgr2=0x55707de97180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:03.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.925+0000 7f881aff5640 0 client.15846 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:12:03.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.925+0000 7f883fc76740 1 -- 172.21.15.16:0/2154496628 shutdown_connections 2024-03-28T22:12:03.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.925+0000 7f883fc76740 1 -- 172.21.15.16:0/2154496628 wait complete. 2024-03-28T22:12:03.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.925+0000 7f883fc76740 20 client.15846 trim_cache size 0 max 16384 2024-03-28T22:12:03.933 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:12:03.933 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-03-28T22:12:03.933 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "3ebe087c-c4c1-412d-b220-8c965bd41dbd", 2024-03-28T22:12:03.933 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-03-28T22:12:03.934 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:12:03.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.925+0000 7f883d08f640 1 -- 172.21.15.16:0/2123621395 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f88200a2010 msgr2=0x7f88200a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:03.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.925+0000 7f883d08f640 1 --2- 172.21.15.16:0/2123621395 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f88200a2010 0x7f88200a44d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8830005fd0 tx=0x7f8830009bd0 comp rx=0 tx=0).stop 2024-03-28T22:12:03.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.925+0000 7f883d08f640 1 -- 172.21.15.16:0/2123621395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88381020b0 msgr2=0x7f88381985b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:03.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.925+0000 7f883d08f640 1 --2- 172.21.15.16:0/2123621395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88381020b0 0x7f88381985b0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f882c00d770 tx=0x7f882c00dc30 comp rx=0 tx=0).stop 2024-03-28T22:12:03.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.925+0000 7f883d08f640 1 -- 172.21.15.16:0/2123621395 shutdown_connections 2024-03-28T22:12:03.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.925+0000 7f883d08f640 1 --2- 172.21.15.16:0/2123621395 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f88200a2010 0x7f88200a44d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:03.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.925+0000 7f883d08f640 1 --2- 172.21.15.16:0/2123621395 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8838107700 0x7f883819c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:03.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.925+0000 7f883d08f640 1 --2- 172.21.15.16:0/2123621395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88381020b0 0x7f88381985b0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:03.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.925+0000 7f883d08f640 1 --2- 172.21.15.16:0/2123621395 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88381016f0 0x7f8838198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:03.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.925+0000 7f883d08f640 1 -- 172.21.15.16:0/2123621395 >> 172.21.15.16:0/2123621395 conn(0x7f88380fd1e0 msgr2=0x7f88380fd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:03.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.926+0000 7f883d08f640 1 -- 172.21.15.16:0/2123621395 shutdown_connections 2024-03-28T22:12:03.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:03.926+0000 7f883d08f640 1 -- 172.21.15.16:0/2123621395 wait complete. 2024-03-28T22:12:03.936 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:03 smithi063 ceph-mon[36249]: pgmap v47: 97 pgs: 97 active+clean; 20 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 655 KiB/s rd, 1.7 MiB/s wr, 40 op/s 2024-03-28T22:12:03.938 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:12:03.938 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:12:04.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.084+0000 7f9196e1c640 1 -- 172.21.15.16:0/1480029191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9190102970 msgr2=0x7f9190102d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:04.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.084+0000 7f9196e1c640 1 --2- 172.21.15.16:0/1480029191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9190102970 0x7f9190102d50 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f9188009ca0 tx=0x7f918802c180 comp rx=0 tx=0).stop 2024-03-28T22:12:04.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.084+0000 7f9196e1c640 1 -- 172.21.15.16:0/1480029191 shutdown_connections 2024-03-28T22:12:04.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.084+0000 7f9196e1c640 1 --2- 172.21.15.16:0/1480029191 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f919010bde0 0x7f919010e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:04.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.084+0000 7f9196e1c640 1 --2- 172.21.15.16:0/1480029191 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9190103290 0x7f919010b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:04.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.084+0000 7f9196e1c640 1 --2- 172.21.15.16:0/1480029191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9190102970 0x7f9190102d50 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:04.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.084+0000 7f9196e1c640 1 -- 172.21.15.16:0/1480029191 >> 172.21.15.16:0/1480029191 conn(0x7f9190077770 msgr2=0x7f9190077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:04.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.084+0000 7f9196e1c640 1 -- 172.21.15.16:0/1480029191 shutdown_connections 2024-03-28T22:12:04.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.084+0000 7f9196e1c640 1 -- 172.21.15.16:0/1480029191 wait complete. 2024-03-28T22:12:04.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.085+0000 7f9196e1c640 1 Processor -- start 2024-03-28T22:12:04.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.085+0000 7f9196e1c640 1 -- start start 2024-03-28T22:12:04.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.085+0000 7f9196e1c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9190102970 0x7f91901005c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:04.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.085+0000 7f9196e1c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9190103290 0x7f9190100b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:04.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.085+0000 7f9196e1c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f919010bde0 0x7f9190101040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:04.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.085+0000 7f9196e1c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9190111be0 con 0x7f919010bde0 2024-03-28T22:12:04.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.085+0000 7f9196e1c640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9190111a60 con 0x7f9190103290 2024-03-28T22:12:04.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.085+0000 7f9196e1c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9190111d60 con 0x7f9190102970 2024-03-28T22:12:04.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.085+0000 7f91953b9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f919010bde0 0x7f9190101040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:04.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.085+0000 7f91953b9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f919010bde0 0x7f9190101040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51216/0 (socket says 172.21.15.16:51216) 2024-03-28T22:12:04.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.085+0000 7f918ffff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9190103290 0x7f9190100b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:04.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.085+0000 7f91953b9640 1 -- 172.21.15.16:0/980956373 learned_addr learned my addr 172.21.15.16:0/980956373 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:04.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.086+0000 7f9194bb8640 1 --2- 172.21.15.16:0/980956373 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9190102970 0x7f91901005c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:04.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.086+0000 7f91953b9640 1 -- 172.21.15.16:0/980956373 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9190102970 msgr2=0x7f91901005c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:04.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.086+0000 7f91953b9640 1 --2- 172.21.15.16:0/980956373 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9190102970 0x7f91901005c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:04.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.086+0000 7f91953b9640 1 -- 172.21.15.16:0/980956373 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9190103290 msgr2=0x7f9190100b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:04.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.086+0000 7f91953b9640 1 --2- 172.21.15.16:0/980956373 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9190103290 0x7f9190100b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:04.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.086+0000 7f91953b9640 1 -- 172.21.15.16:0/980956373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91901a65e0 con 0x7f919010bde0 2024-03-28T22:12:04.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.086+0000 7f918ffff640 1 --2- 172.21.15.16:0/980956373 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9190103290 0x7f9190100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:12:04.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.086+0000 7f91953b9640 1 --2- 172.21.15.16:0/980956373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f919010bde0 0x7f9190101040 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f917800e970 tx=0x7f917800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:04.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.086+0000 7f918dffb640 1 -- 172.21.15.16:0/980956373 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f917800ccc0 con 0x7f919010bde0 2024-03-28T22:12:04.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.086+0000 7f918dffb640 1 -- 172.21.15.16:0/980956373 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f917800ce60 con 0x7f919010bde0 2024-03-28T22:12:04.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.086+0000 7f9196e1c640 1 -- 172.21.15.16:0/980956373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91901a68d0 con 0x7f919010bde0 2024-03-28T22:12:04.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.086+0000 7f918dffb640 1 -- 172.21.15.16:0/980956373 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9178010490 con 0x7f919010bde0 2024-03-28T22:12:04.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.086+0000 7f9196e1c640 1 -- 172.21.15.16:0/980956373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91901a6dd0 con 0x7f919010bde0 2024-03-28T22:12:04.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.088+0000 7f918dffb640 1 -- 172.21.15.16:0/980956373 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 25) ==== 136989+0+0 (secure 0 0 0) 0x7f9178005020 con 0x7f919010bde0 2024-03-28T22:12:04.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.089+0000 7f918dffb640 1 --2- 172.21.15.16:0/980956373 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f917c0a2010 0x7f917c0a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:04.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.089+0000 7f918dffb640 1 -- 172.21.15.16:0/980956373 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f91780ce930 con 0x7f919010bde0 2024-03-28T22:12:04.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.089+0000 7f9194bb8640 1 --2- 172.21.15.16:0/980956373 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f917c0a2010 0x7f917c0a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:04.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.089+0000 7f9199a04740 1 Processor -- start 2024-03-28T22:12:04.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.089+0000 7f9199a04740 1 -- start start 2024-03-28T22:12:04.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.089+0000 7f9199a04740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e03b027290 0x55e03b027670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:04.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.089+0000 7f9199a04740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e03b027bb0 0x55e03b030250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:04.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.089+0000 7f9199a04740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e03b030790 0x55e03b032b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:04.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.089+0000 7f9199a04740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e03af8b180 con 0x55e03b030790 2024-03-28T22:12:04.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.089+0000 7f9199a04740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55e03af33b70 con 0x55e03b027bb0 2024-03-28T22:12:04.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.089+0000 7f9199a04740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e03af87220 con 0x55e03b027290 2024-03-28T22:12:04.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.089+0000 7f9194bb8640 1 --2- 172.21.15.16:0/980956373 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f917c0a2010 0x7f917c0a44d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9188006010 tx=0x7f9188032480 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:04.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.089+0000 7f91953b9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e03b030790 0x55e03b032b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:04.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.090+0000 7f91953b9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e03b030790 0x55e03b032b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51220/0 (socket says 172.21.15.16:51220) 2024-03-28T22:12:04.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.090+0000 7f91953b9640 1 -- 172.21.15.16:0/139245457 learned_addr learned my addr 172.21.15.16:0/139245457 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:04.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.090+0000 7f918ffff640 1 --2- 172.21.15.16:0/139245457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e03b027bb0 0x55e03b030250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:04.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.090+0000 7f9194bb8640 1 --2- 172.21.15.16:0/139245457 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e03b027290 0x55e03b027670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:04.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.090+0000 7f91953b9640 1 -- 172.21.15.16:0/139245457 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e03b027290 msgr2=0x55e03b027670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:04.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.090+0000 7f91953b9640 1 --2- 172.21.15.16:0/139245457 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e03b027290 0x55e03b027670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:04.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.090+0000 7f91953b9640 1 -- 172.21.15.16:0/139245457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e03b027bb0 msgr2=0x55e03b030250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:04.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.090+0000 7f91953b9640 1 --2- 172.21.15.16:0/139245457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e03b027bb0 0x55e03b030250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:04.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.090+0000 7f91953b9640 1 -- 172.21.15.16:0/139245457 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e03af477c0 con 0x55e03b030790 2024-03-28T22:12:04.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.090+0000 7f91953b9640 1 --2- 172.21.15.16:0/139245457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e03b030790 0x55e03b032b80 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f9178004fd0 tx=0x7f917809b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab7750311e88fe3c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:04.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.090+0000 7f9186ffd640 1 -- 172.21.15.16:0/139245457 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91780a3830 con 0x55e03b030790 2024-03-28T22:12:04.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.090+0000 7f9186ffd640 1 -- 172.21.15.16:0/139245457 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9178099390 con 0x55e03b030790 2024-03-28T22:12:04.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.090+0000 7f9186ffd640 1 -- 172.21.15.16:0/139245457 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91780a35a0 con 0x55e03b030790 2024-03-28T22:12:04.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.091+0000 7f9199a04740 1 -- 172.21.15.16:0/139245457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e03b030790 msgr2=0x55e03b032b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:04.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.091+0000 7f9199a04740 1 --2- 172.21.15.16:0/139245457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e03b030790 0x55e03b032b80 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f9178004fd0 tx=0x7f917809b790 comp rx=0 tx=0).stop 2024-03-28T22:12:04.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.091+0000 7f9199a04740 1 -- 172.21.15.16:0/139245457 shutdown_connections 2024-03-28T22:12:04.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.091+0000 7f9199a04740 1 --2- 172.21.15.16:0/139245457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e03b030790 0x55e03b032b80 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:04.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.091+0000 7f9199a04740 1 --2- 172.21.15.16:0/139245457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e03b027bb0 0x55e03b030250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:04.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.091+0000 7f9199a04740 1 --2- 172.21.15.16:0/139245457 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e03b027290 0x55e03b027670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:04.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.091+0000 7f9199a04740 1 -- 172.21.15.16:0/139245457 >> 172.21.15.16:0/139245457 conn(0x55e03b0219c0 msgr2=0x55e03b021da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:04.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.091+0000 7f9199a04740 1 -- 172.21.15.16:0/139245457 shutdown_connections 2024-03-28T22:12:04.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.091+0000 7f9199a04740 1 -- 172.21.15.16:0/139245457 wait complete. 2024-03-28T22:12:04.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.091+0000 7f9199a04740 1 -- start start 2024-03-28T22:12:04.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.091+0000 7f9199a04740 1 Processor -- start 2024-03-28T22:12:04.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.092+0000 7f9199a04740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e03b027290 0x55e03b02d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:04.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.092+0000 7f9199a04740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e03b027bb0 0x55e03b02dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:04.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.092+0000 7f9199a04740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e03b030790 0x55e03b02e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:04.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.092+0000 7f9199a04740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e03b033580 con 0x55e03b027bb0 2024-03-28T22:12:04.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.092+0000 7f9199a04740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55e03af294a0 con 0x55e03b030790 2024-03-28T22:12:04.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.092+0000 7f9199a04740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e03b033700 con 0x55e03b027290 2024-03-28T22:12:04.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.092+0000 7f918ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e03b027bb0 0x55e03b02dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:04.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.092+0000 7f9199a04740 10 client.0 _ll_register_callbacks cb 0x55e03af2cd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:12:04.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.092+0000 7f918ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e03b027bb0 0x55e03b02dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51224/0 (socket says 172.21.15.16:51224) 2024-03-28T22:12:04.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.092+0000 7f918ffff640 1 -- 172.21.15.16:0/1317974218 learned_addr learned my addr 172.21.15.16:0/1317974218 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:04.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.092+0000 7f91953b9640 1 --2- 172.21.15.16:0/1317974218 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e03b030790 0x55e03b02e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:04.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.092+0000 7f9194bb8640 1 --2- 172.21.15.16:0/1317974218 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e03b027290 0x55e03b02d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:04.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.092+0000 7f918ffff640 1 -- 172.21.15.16:0/1317974218 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e03b027290 msgr2=0x55e03b02d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:04.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.092+0000 7f918ffff640 1 --2- 172.21.15.16:0/1317974218 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e03b027290 0x55e03b02d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:04.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.092+0000 7f918ffff640 1 -- 172.21.15.16:0/1317974218 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e03b030790 msgr2=0x55e03b02e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:04.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.092+0000 7f918ffff640 1 --2- 172.21.15.16:0/1317974218 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e03b030790 0x55e03b02e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:04.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.092+0000 7f918ffff640 1 -- 172.21.15.16:0/1317974218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e03ad697d0 con 0x55e03b027bb0 2024-03-28T22:12:04.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.092+0000 7f918ffff640 1 --2- 172.21.15.16:0/1317974218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e03b027bb0 0x55e03b02dd90 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f9180010bd0 tx=0x7f918000f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abb90597515ae1a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:04.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.092+0000 7f916ffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:12:04.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.092+0000 7f916ffff640 1 -- 172.21.15.16:0/1317974218 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9180005020 con 0x55e03b027bb0 2024-03-28T22:12:04.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.092+0000 7f9199a04740 1 -- 172.21.15.16:0/1317974218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9178099390 con 0x55e03b027bb0 2024-03-28T22:12:04.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.092+0000 7f916ffff640 1 -- 172.21.15.16:0/1317974218 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f91800051c0 con 0x55e03b027bb0 2024-03-28T22:12:04.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.093+0000 7f916ffff640 1 -- 172.21.15.16:0/1317974218 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f91800053c0 con 0x55e03b027bb0 2024-03-28T22:12:04.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.093+0000 7f9199a04740 10 client.15858 fetch_fsmap learned FSMap version 24 2024-03-28T22:12:04.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.093+0000 7f9199a04740 1 -- 172.21.15.16:0/1317974218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e03ac9f3f0 con 0x55e03b027bb0 2024-03-28T22:12:04.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.093+0000 7f916ffff640 1 -- 172.21.15.16:0/1317974218 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9178099390 con 0x55e03b027bb0 2024-03-28T22:12:04.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.093+0000 7f916ffff640 1 -- 172.21.15.16:0/1317974218 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7f9180020020 con 0x55e03b027bb0 2024-03-28T22:12:04.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.093+0000 7f9199a04740 10 client.15858 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:12:04.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.093+0000 7f9199a04740 10 client.15858 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:12:04.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.093+0000 7f9199a04740 20 client.15858 populate_metadata read hostname 'smithi016' 2024-03-28T22:12:04.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.093+0000 7f9199a04740 1 --2- 172.21.15.16:0/1317974218 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e03b040ca0 0x55e03b0430e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:04.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.093+0000 7f9199a04740 4 client.15858 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:12:04.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.093+0000 7f9199a04740 1 -- 172.21.15.16:0/1317974218 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e03af986c0 con 0x55e03b040ca0 2024-03-28T22:12:04.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.093+0000 7f9194bb8640 1 --2- 172.21.15.16:0/1317974218 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e03b040ca0 0x55e03b0430e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:04.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.094+0000 7f916ffff640 10 client.15858 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:12:04.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.094+0000 7f9194bb8640 1 --2- 172.21.15.16:0/1317974218 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e03b040ca0 0x55e03b0430e0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22bc0c5744acbd13 server_cookie=66671748dab885bd in_seq=0 out_seq=0 2024-03-28T22:12:04.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.094+0000 7f916ffff640 1 -- 172.21.15.16:0/1317974218 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e03af986c0 con 0x55e03b040ca0 2024-03-28T22:12:04.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.094+0000 7f916ffff640 10 client.15858 handle_command_reply: tid=0 2024-03-28T22:12:04.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.094+0000 7f9199a04740 1 client.15858 shutdown 2024-03-28T22:12:04.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.095+0000 7f9199a04740 1 -- 172.21.15.16:0/1317974218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e03b027bb0 msgr2=0x55e03b02dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:04.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.095+0000 7f9199a04740 1 --2- 172.21.15.16:0/1317974218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e03b027bb0 0x55e03b02dd90 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f9180010bd0 tx=0x7f918000f7f0 comp rx=0 tx=0).stop 2024-03-28T22:12:04.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.095+0000 7f9199a04740 1 -- 172.21.15.16:0/1317974218 shutdown_connections 2024-03-28T22:12:04.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.095+0000 7f9199a04740 1 --2- 172.21.15.16:0/1317974218 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e03b040ca0 0x55e03b0430e0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:04.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.095+0000 7f9199a04740 1 --2- 172.21.15.16:0/1317974218 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e03b030790 0x55e03b02e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:04.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.095+0000 7f9199a04740 1 --2- 172.21.15.16:0/1317974218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e03b027bb0 0x55e03b02dd90 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:04.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.095+0000 7f9199a04740 1 --2- 172.21.15.16:0/1317974218 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e03b027290 0x55e03b02d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:04.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.095+0000 7f916ffff640 0 client.15858 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:12:04.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.095+0000 7f9199a04740 1 -- 172.21.15.16:0/1317974218 >> 172.21.15.16:0/1317974218 conn(0x55e03b0219c0 msgr2=0x55e03b020c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:04.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.095+0000 7f9199a04740 1 -- 172.21.15.16:0/1317974218 shutdown_connections 2024-03-28T22:12:04.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.095+0000 7f9199a04740 1 -- 172.21.15.16:0/1317974218 wait complete. 2024-03-28T22:12:04.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.095+0000 7f9199a04740 20 client.15858 trim_cache size 0 max 16384 2024-03-28T22:12:04.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.112+0000 7f9199a04740 1 Processor -- start 2024-03-28T22:12:04.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.112+0000 7f9199a04740 1 -- start start 2024-03-28T22:12:04.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.112+0000 7f9199a04740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e03b027290 0x55e03b02ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:04.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.112+0000 7f9199a04740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e03b027bb0 0x55e03b021da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:04.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.112+0000 7f9199a04740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e03b030790 0x55e03b03c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:04.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.112+0000 7f9199a04740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f91901b1ee0 con 0x55e03b027bb0 2024-03-28T22:12:04.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.112+0000 7f9199a04740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f91901af780 con 0x55e03b027290 2024-03-28T22:12:04.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.112+0000 7f9199a04740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f91901ad5f0 con 0x55e03b030790 2024-03-28T22:12:04.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.112+0000 7f918ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e03b027bb0 0x55e03b021da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:04.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.112+0000 7f918ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e03b027bb0 0x55e03b021da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51240/0 (socket says 172.21.15.16:51240) 2024-03-28T22:12:04.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.112+0000 7f9194bb8640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e03b027290 0x55e03b02ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:04.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.112+0000 7f918ffff640 1 -- 172.21.15.16:0/221201618 learned_addr learned my addr 172.21.15.16:0/221201618 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:04.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.112+0000 7f91953b9640 1 --2- 172.21.15.16:0/221201618 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e03b030790 0x55e03b03c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:04.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.113+0000 7f918ffff640 1 -- 172.21.15.16:0/221201618 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e03b030790 msgr2=0x55e03b03c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:04.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.113+0000 7f918ffff640 1 --2- 172.21.15.16:0/221201618 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e03b030790 0x55e03b03c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:04.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.113+0000 7f918ffff640 1 -- 172.21.15.16:0/221201618 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e03b027290 msgr2=0x55e03b02ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:04.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.113+0000 7f918ffff640 1 --2- 172.21.15.16:0/221201618 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e03b027290 0x55e03b02ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:04.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.113+0000 7f918ffff640 1 -- 172.21.15.16:0/221201618 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e03ac5f0f0 con 0x55e03b027bb0 2024-03-28T22:12:04.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.113+0000 7f91953b9640 1 --2- 172.21.15.16:0/221201618 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e03b030790 0x55e03b03c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:12:04.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.113+0000 7f918ffff640 1 --2- 172.21.15.16:0/221201618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e03b027bb0 0x55e03b021da0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f91800058c0 tx=0x7f91800104c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58731df7799db15e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:04.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.113+0000 7f91867fc640 1 -- 172.21.15.16:0/221201618 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f918000e680 con 0x55e03b027bb0 2024-03-28T22:12:04.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.113+0000 7f91867fc640 1 -- 172.21.15.16:0/221201618 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f918000ec40 con 0x55e03b027bb0 2024-03-28T22:12:04.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.113+0000 7f91867fc640 1 -- 172.21.15.16:0/221201618 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9180022db0 con 0x55e03b027bb0 2024-03-28T22:12:04.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.113+0000 7f9199a04740 1 -- 172.21.15.16:0/221201618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e03b027bb0 msgr2=0x55e03b021da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:04.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.113+0000 7f9199a04740 1 --2- 172.21.15.16:0/221201618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e03b027bb0 0x55e03b021da0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f91800058c0 tx=0x7f91800104c0 comp rx=0 tx=0).stop 2024-03-28T22:12:04.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.113+0000 7f9199a04740 1 -- 172.21.15.16:0/221201618 shutdown_connections 2024-03-28T22:12:04.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.113+0000 7f9199a04740 1 --2- 172.21.15.16:0/221201618 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e03b030790 0x55e03b03c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:04.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.113+0000 7f9199a04740 1 --2- 172.21.15.16:0/221201618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e03b027bb0 0x55e03b021da0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:04.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.113+0000 7f9199a04740 1 --2- 172.21.15.16:0/221201618 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e03b027290 0x55e03b02ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:04.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.113+0000 7f9199a04740 1 -- 172.21.15.16:0/221201618 >> 172.21.15.16:0/221201618 conn(0x55e03b0219c0 msgr2=0x55e03b020390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:04.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.114+0000 7f9199a04740 1 -- 172.21.15.16:0/221201618 shutdown_connections 2024-03-28T22:12:04.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.114+0000 7f9199a04740 1 -- 172.21.15.16:0/221201618 wait complete. 2024-03-28T22:12:04.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.114+0000 7f9199a04740 1 -- start start 2024-03-28T22:12:04.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.114+0000 7f9199a04740 1 Processor -- start 2024-03-28T22:12:04.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.114+0000 7f9199a04740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e03b027290 0x55e03b0348a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:04.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.114+0000 7f9199a04740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e03b027bb0 0x55e03b034de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:04.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.114+0000 7f9199a04740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e03b030790 0x55e03b0a6020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:04.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.114+0000 7f9199a04740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f91901adc40 con 0x55e03b027290 2024-03-28T22:12:04.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.114+0000 7f9199a04740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f91901af0b0 con 0x55e03b030790 2024-03-28T22:12:04.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.114+0000 7f9199a04740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f91901ae310 con 0x55e03b027bb0 2024-03-28T22:12:04.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.114+0000 7f9199a04740 10 client.0 _ll_register_callbacks cb 0x7f918000ab90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:12:04.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.114+0000 7f9194bb8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e03b027290 0x55e03b0348a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:04.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.114+0000 7f9194bb8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e03b027290 0x55e03b0348a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51248/0 (socket says 172.21.15.16:51248) 2024-03-28T22:12:04.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.114+0000 7f9194bb8640 1 -- 172.21.15.16:0/577943217 learned_addr learned my addr 172.21.15.16:0/577943217 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:04.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.114+0000 7f91953b9640 1 --2- 172.21.15.16:0/577943217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e03b030790 0x55e03b0a6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:04.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.115+0000 7f918ffff640 1 --2- 172.21.15.16:0/577943217 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e03b027bb0 0x55e03b034de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:04.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.115+0000 7f9194bb8640 1 -- 172.21.15.16:0/577943217 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e03b027bb0 msgr2=0x55e03b034de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:04.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.115+0000 7f9194bb8640 1 --2- 172.21.15.16:0/577943217 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e03b027bb0 0x55e03b034de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:04.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.115+0000 7f9194bb8640 1 -- 172.21.15.16:0/577943217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e03b030790 msgr2=0x55e03b0a6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:04.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.115+0000 7f9194bb8640 1 --2- 172.21.15.16:0/577943217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e03b030790 0x55e03b0a6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:04.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.115+0000 7f9194bb8640 1 -- 172.21.15.16:0/577943217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e03b0a6560 con 0x55e03b027290 2024-03-28T22:12:04.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.115+0000 7f9194bb8640 1 --2- 172.21.15.16:0/577943217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e03b027290 0x55e03b0348a0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f9188009a60 tx=0x7f918802e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c9bb92434c1f385 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:04.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.115+0000 7f916ffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:12:04.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.115+0000 7f916ffff640 1 -- 172.21.15.16:0/577943217 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9188046040 con 0x55e03b027290 2024-03-28T22:12:04.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.115+0000 7f916ffff640 1 -- 172.21.15.16:0/577943217 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f9188003f90 con 0x55e03b027290 2024-03-28T22:12:04.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.115+0000 7f9199a04740 1 -- 172.21.15.16:0/577943217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f918000ec40 con 0x55e03b027290 2024-03-28T22:12:04.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.115+0000 7f916ffff640 1 -- 172.21.15.16:0/577943217 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9188039040 con 0x55e03b027290 2024-03-28T22:12:04.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.115+0000 7f9199a04740 10 client.15864 fetch_fsmap learned FSMap version 24 2024-03-28T22:12:04.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.115+0000 7f9199a04740 1 -- 172.21.15.16:0/577943217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e03b0a7710 con 0x55e03b027290 2024-03-28T22:12:04.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.115+0000 7f916ffff640 1 -- 172.21.15.16:0/577943217 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f918000ec40 con 0x55e03b027290 2024-03-28T22:12:04.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.116+0000 7f916ffff640 1 -- 172.21.15.16:0/577943217 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7f9188060030 con 0x55e03b027290 2024-03-28T22:12:04.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.116+0000 7f9199a04740 10 client.15864 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:12:04.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.116+0000 7f9199a04740 10 client.15864 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:12:04.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.116+0000 7f9199a04740 20 client.15864 populate_metadata read hostname 'smithi016' 2024-03-28T22:12:04.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.116+0000 7f9199a04740 1 --2- 172.21.15.16:0/577943217 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e03b0a7e90 0x55e03b0aa2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:04.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.116+0000 7f9199a04740 4 client.15864 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:12:04.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.116+0000 7f9199a04740 1 -- 172.21.15.16:0/577943217 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e03af92a10 con 0x55e03b0a7e90 2024-03-28T22:12:04.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.116+0000 7f918ffff640 1 --2- 172.21.15.16:0/577943217 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e03b0a7e90 0x55e03b0aa2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:04.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.116+0000 7f916ffff640 10 client.15864 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:12:04.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.116+0000 7f918ffff640 1 --2- 172.21.15.16:0/577943217 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e03b0a7e90 0x55e03b0aa2f0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11e7ccfdcae78f71 server_cookie=9e9fbbc1974442a in_seq=0 out_seq=0 2024-03-28T22:12:04.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.117+0000 7f916ffff640 1 -- 172.21.15.16:0/577943217 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55e03af92a10 con 0x55e03b0a7e90 2024-03-28T22:12:04.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.117+0000 7f916ffff640 10 client.15864 handle_command_reply: tid=0 2024-03-28T22:12:04.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.117+0000 7f9199a04740 1 client.15864 shutdown 2024-03-28T22:12:04.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.117+0000 7f9199a04740 1 -- 172.21.15.16:0/577943217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e03b027290 msgr2=0x55e03b0348a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:04.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.117+0000 7f9199a04740 1 --2- 172.21.15.16:0/577943217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e03b027290 0x55e03b0348a0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f9188009a60 tx=0x7f918802e050 comp rx=0 tx=0).stop 2024-03-28T22:12:04.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.117+0000 7f9199a04740 1 -- 172.21.15.16:0/577943217 shutdown_connections 2024-03-28T22:12:04.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.117+0000 7f9199a04740 1 --2- 172.21.15.16:0/577943217 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e03b0a7e90 0x55e03b0aa2f0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:04.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.117+0000 7f9199a04740 1 --2- 172.21.15.16:0/577943217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e03b030790 0x55e03b0a6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:04.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.117+0000 7f916ffff640 0 client.15864 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:12:04.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.117+0000 7f9199a04740 1 --2- 172.21.15.16:0/577943217 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e03b027bb0 0x55e03b034de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:04.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.117+0000 7f9199a04740 1 --2- 172.21.15.16:0/577943217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e03b027290 0x55e03b0348a0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:04.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.117+0000 7f9199a04740 1 -- 172.21.15.16:0/577943217 >> 172.21.15.16:0/577943217 conn(0x55e03b0219c0 msgr2=0x55e03b02e080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:04.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.117+0000 7f9199a04740 1 -- 172.21.15.16:0/577943217 shutdown_connections 2024-03-28T22:12:04.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.117+0000 7f9199a04740 1 -- 172.21.15.16:0/577943217 wait complete. 2024-03-28T22:12:04.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.117+0000 7f9199a04740 20 client.15864 trim_cache size 0 max 16384 2024-03-28T22:12:04.128 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:12:04.128 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T22:12:04.128 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-03-28T22:12:04.128 INFO:teuthology.orchestra.run.smithi016.stdout: "3ebe087c-c4c1-412d-b220-8c965bd41dbd": { 2024-03-28T22:12:04.128 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-03-28T22:12:04.128 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "3ebe087c-c4c1-412d-b220-8c965bd41dbd", 2024-03-28T22:12:04.129 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-03-28T22:12:04.129 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:12:04.129 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:12:04.129 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:12:04.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.118+0000 7f9196e1c640 1 -- 172.21.15.16:0/980956373 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f917c0a2010 msgr2=0x7f917c0a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:04.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.118+0000 7f9196e1c640 1 --2- 172.21.15.16:0/980956373 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f917c0a2010 0x7f917c0a44d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9188006010 tx=0x7f9188032480 comp rx=0 tx=0).stop 2024-03-28T22:12:04.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.118+0000 7f9196e1c640 1 -- 172.21.15.16:0/980956373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f919010bde0 msgr2=0x7f9190101040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:04.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.118+0000 7f9196e1c640 1 --2- 172.21.15.16:0/980956373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f919010bde0 0x7f9190101040 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f917800e970 tx=0x7f917800ed70 comp rx=0 tx=0).stop 2024-03-28T22:12:04.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.118+0000 7f9196e1c640 1 -- 172.21.15.16:0/980956373 shutdown_connections 2024-03-28T22:12:04.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.118+0000 7f9196e1c640 1 --2- 172.21.15.16:0/980956373 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f917c0a2010 0x7f917c0a44d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:04.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.118+0000 7f9196e1c640 1 --2- 172.21.15.16:0/980956373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f919010bde0 0x7f9190101040 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:04.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.118+0000 7f9196e1c640 1 --2- 172.21.15.16:0/980956373 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9190103290 0x7f9190100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:04.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.118+0000 7f9196e1c640 1 --2- 172.21.15.16:0/980956373 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9190102970 0x7f91901005c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:04.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.118+0000 7f9196e1c640 1 -- 172.21.15.16:0/980956373 >> 172.21.15.16:0/980956373 conn(0x7f9190077770 msgr2=0x7f91900fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:04.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.118+0000 7f9196e1c640 1 -- 172.21.15.16:0/980956373 shutdown_connections 2024-03-28T22:12:04.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:04.118+0000 7f9196e1c640 1 -- 172.21.15.16:0/980956373 wait complete. 2024-03-28T22:12:04.131 INFO:tasks.cephfs.filesystem:scrub status for tag:3ebe087c-c4c1-412d-b220-8c965bd41dbd - {'path': '/', 'tag': '3ebe087c-c4c1-412d-b220-8c965bd41dbd', 'options': 'recursive,force'} 2024-03-28T22:12:04.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:04 smithi053 ceph-mon[35979]: scrub queued for path: / 2024-03-28T22:12:04.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:04 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:12:04.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:04 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:12:04.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:04 smithi063 ceph-mon[36249]: scrub queued for path: / 2024-03-28T22:12:04.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:04 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:12:04.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:04 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:12:04.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:04 smithi016 ceph-mon[32374]: scrub queued for path: / 2024-03-28T22:12:04.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:04 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:12:04.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:04 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:12:05.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:05 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:12:05.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:05 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:12:05.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:05 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:12:05.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:05 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:12:05.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:05 smithi053 ceph-mon[35979]: pgmap v48: 97 pgs: 97 active+clean; 20 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 580 KiB/s wr, 32 op/s 2024-03-28T22:12:05.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:05 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:12:05.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:05 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:12:05.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:05 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:12:05.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:05 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:12:05.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:05 smithi063 ceph-mon[36249]: pgmap v48: 97 pgs: 97 active+clean; 20 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 580 KiB/s wr, 32 op/s 2024-03-28T22:12:05.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:05 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:12:05.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:05 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:12:05.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:05 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:12:05.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:05 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:12:05.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:05 smithi016 ceph-mon[32374]: pgmap v48: 97 pgs: 97 active+clean; 20 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 580 KiB/s wr, 32 op/s 2024-03-28T22:12:06.842 INFO:tasks.workunit.client.0.smithi016.stderr:Updating files: 10% (452/4277) Updating files: 11% (471/4277) Updating files: 11% (510/4277) Updating files: 12% (514/4277) Updating files: 13% (557/4277) Updating files: 14% (599/4277) Updating files: 15% (642/4277) Updating files: 16% (685/4277) Updating files: 17% (728/4277) Updating files: 18% (770/4277) Updating files: 19% (813/4277) Updating files: 20% (856/4277) Updating files: 21% (899/4277) Updating files: 22% (941/4277) Updating files: 23% (984/4277) Updating files: 23% (1018/4277) Updating files: 24% (1027/4277) Updating files: 25% (1070/4277) Updating files: 26% (1113/4277) Updating files: 27% (1155/4277) Updating files: 28% (1198/4277) Updating files: 29% (1241/4277) Updating files: 29% (1283/4277) Updating files: 30% (1284/4277) Updating files: 31% (1326/4277) Updating files: 32% (1369/4277) Updating files: 33% (1412/4277) Updating files: 34% (1455/4277) Updating files: 35% (1497/4277) Updating files: 36% (1540/4277) Updating files: 37% (1583/4277) Updating files: 38% (1626/4277) Updating files: 39% (1669/4277) Updating files: 40% (1711/4277) Updating files: 41% (1754/4277) Updating files: 42% (1797/4277) Updating files: 43% (1840/4277) Updating files: 44% (1882/4277) Updating files: 45% (1925/4277) Updating files: 46% (1968/4277) Updating files: 46% (2009/4277) Updating files: 47% (2011/4277) Updating files: 48% (2053/4277) Updating files: 49% (2096/4277) Updating files: 50% (2139/4277) Updating files: 51% (2182/4277) Updating files: 52% (2225/4277) Updating files: 53% (2267/4277) Updating files: 53% (2269/4277) Updating files: 54% (2310/4277) Updating files: 55% (2353/4277) Updating files: 56% (2396/4277) Updating files: 57% (2438/4277) Updating files: 58% (2481/4277) Updating files: 59% (2524/4277) Updating files: 60% (2567/4277) Updating files: 61% (2609/4277) Updating files: 62% (2652/4277) Updating files: 63% (2695/4277) Updating files: 64% (2738/4277) Updating files: 64% (2773/4277) Updating files: 65% (2781/4277) Updating files: 66% (2823/4277) Updating files: 67% (2866/4277) Updating files: 68% (2909/4277) Updating files: 69% (2952/4277) Updating files: 70% (2994/4277) Updating files: 70% (3011/4277) Updating files: 71% (3037/4277) Updating files: 72% (3080/4277) Updating files: 72% (3110/4277) Updating files: 73% (3123/4277) Updating files: 74% (3165/4277) Updating files: 75% (3208/4277) Updating files: 76% (3251/4277) Updating files: 77% (3294/4277) Updating files: 78% (3337/4277) Updating files: 79% (3379/4277) Updating files: 80% (3422/4277) Updating files: 81% (3465/4277) Updating files: 82% (3508/4277) Updating files: 83% (3550/4277) Updating files: 84% (3593/4277) Updating files: 85% (3636/4277) Updating files: 86% (3679/4277) Updating files: 87% (3721/4277) Updating files: 87% (3727/4277) Updating files: 88% (3764/4277) Updating files: 89% (3807/4277) Updating files: 90% (3850/4277) Updating files: 91% (3893/4277) Updating files: 92% (3935/4277) Updating files: 93% (3978/4277) Updating files: 94% (4021/4277) Updating files: 94% (4047/4277) Updating files: 95% (4064/4277) Updating files: 96% (4106/4277) Updating files: 97% (4149/4277) Updating files: 98% (4192/4277) Updating files: 99% (4235/4277) Updating files: 100% (4277/4277) Updating files: 100% (4277/4277), done. 2024-03-28T22:12:06.852 INFO:tasks.workunit.client.0.smithi016.stderr:+ cd xfstests-dev 2024-03-28T22:12:06.853 INFO:tasks.workunit.client.0.smithi016.stderr:+ git checkout 12973fc04fd10d4af086901e10ffa8e48866b735 2024-03-28T22:12:07.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:07 smithi053 ceph-mon[35979]: pgmap v49: 97 pgs: 97 active+clean; 22 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 756 KiB/s wr, 56 op/s 2024-03-28T22:12:07.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:07 smithi053 ceph-mon[35979]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-03-28T22:12:07.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:07 smithi063 ceph-mon[36249]: pgmap v49: 97 pgs: 97 active+clean; 22 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 756 KiB/s wr, 56 op/s 2024-03-28T22:12:07.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:07 smithi063 ceph-mon[36249]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-03-28T22:12:07.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:07 smithi016 ceph-mon[32374]: pgmap v49: 97 pgs: 97 active+clean; 22 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 756 KiB/s wr, 56 op/s 2024-03-28T22:12:07.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:07 smithi016 ceph-mon[32374]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-03-28T22:12:08.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:08 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:12:08.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:08 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:12:08.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:08 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:12:09.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:09 smithi053 ceph-mon[35979]: pgmap v50: 97 pgs: 97 active+clean; 22 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 624 KiB/s wr, 44 op/s 2024-03-28T22:12:09.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:09 smithi063 ceph-mon[36249]: pgmap v50: 97 pgs: 97 active+clean; 22 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 624 KiB/s wr, 44 op/s 2024-03-28T22:12:09.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:09 smithi016 ceph-mon[32374]: pgmap v50: 97 pgs: 97 active+clean; 22 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 624 KiB/s wr, 44 op/s 2024-03-28T22:12:11.734 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:11 smithi063 ceph-mon[36249]: pgmap v51: 97 pgs: 97 active+clean; 22 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 624 KiB/s wr, 44 op/s 2024-03-28T22:12:11.738 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:11 smithi016 ceph-mon[32374]: pgmap v51: 97 pgs: 97 active+clean; 22 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 624 KiB/s wr, 44 op/s 2024-03-28T22:12:11.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:11 smithi053 ceph-mon[35979]: pgmap v51: 97 pgs: 97 active+clean; 22 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 624 KiB/s wr, 44 op/s 2024-03-28T22:12:13.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:13 smithi053 ceph-mon[35979]: pgmap v52: 97 pgs: 97 active+clean; 29 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 62 op/s 2024-03-28T22:12:13.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:13 smithi063 ceph-mon[36249]: pgmap v52: 97 pgs: 97 active+clean; 29 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 62 op/s 2024-03-28T22:12:13.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:13 smithi016 ceph-mon[32374]: pgmap v52: 97 pgs: 97 active+clean; 29 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 62 op/s 2024-03-28T22:12:15.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:15 smithi053 ceph-mon[35979]: pgmap v53: 97 pgs: 97 active+clean; 29 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 889 KiB/s wr, 41 op/s 2024-03-28T22:12:15.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:15 smithi053 ceph-mon[35979]: scrub summary: idle 2024-03-28T22:12:15.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:15 smithi063 ceph-mon[36249]: pgmap v53: 97 pgs: 97 active+clean; 29 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 889 KiB/s wr, 41 op/s 2024-03-28T22:12:15.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:15 smithi063 ceph-mon[36249]: scrub summary: idle 2024-03-28T22:12:15.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:15 smithi016 ceph-mon[32374]: pgmap v53: 97 pgs: 97 active+clean; 29 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 889 KiB/s wr, 41 op/s 2024-03-28T22:12:15.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:15 smithi016 ceph-mon[32374]: scrub summary: idle 2024-03-28T22:12:16.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:16 smithi053 ceph-mon[35979]: scrub summary: idle 2024-03-28T22:12:16.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:16 smithi063 ceph-mon[36249]: scrub summary: idle 2024-03-28T22:12:16.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:16 smithi016 ceph-mon[32374]: scrub summary: idle 2024-03-28T22:12:17.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:17 smithi053 ceph-mon[35979]: pgmap v54: 97 pgs: 97 active+clean; 40 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 47 op/s 2024-03-28T22:12:17.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:17 smithi063 ceph-mon[36249]: pgmap v54: 97 pgs: 97 active+clean; 40 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 47 op/s 2024-03-28T22:12:17.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:17 smithi016 ceph-mon[32374]: pgmap v54: 97 pgs: 97 active+clean; 40 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 47 op/s 2024-03-28T22:12:19.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:19 smithi053 ceph-mon[35979]: pgmap v55: 97 pgs: 97 active+clean; 40 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 23 op/s 2024-03-28T22:12:19.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:19 smithi063 ceph-mon[36249]: pgmap v55: 97 pgs: 97 active+clean; 40 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 23 op/s 2024-03-28T22:12:19.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:19 smithi016 ceph-mon[32374]: pgmap v55: 97 pgs: 97 active+clean; 40 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 23 op/s 2024-03-28T22:12:21.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:21 smithi053 ceph-mon[35979]: pgmap v56: 97 pgs: 97 active+clean; 40 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 23 op/s 2024-03-28T22:12:21.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:21 smithi063 ceph-mon[36249]: pgmap v56: 97 pgs: 97 active+clean; 40 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 23 op/s 2024-03-28T22:12:21.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:21 smithi016 ceph-mon[32374]: pgmap v56: 97 pgs: 97 active+clean; 40 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 23 op/s 2024-03-28T22:12:23.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:23 smithi053 ceph-mon[35979]: pgmap v57: 97 pgs: 97 active+clean; 62 MiB data, 635 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.7 MiB/s wr, 512 op/s 2024-03-28T22:12:23.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:23 smithi063 ceph-mon[36249]: pgmap v57: 97 pgs: 97 active+clean; 62 MiB data, 635 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.7 MiB/s wr, 512 op/s 2024-03-28T22:12:23.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:23 smithi016 ceph-mon[32374]: pgmap v57: 97 pgs: 97 active+clean; 62 MiB data, 635 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.7 MiB/s wr, 512 op/s 2024-03-28T22:12:25.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:25 smithi053 ceph-mon[35979]: pgmap v58: 97 pgs: 97 active+clean; 62 MiB data, 635 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 494 op/s 2024-03-28T22:12:25.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:25 smithi063 ceph-mon[36249]: pgmap v58: 97 pgs: 97 active+clean; 62 MiB data, 635 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 494 op/s 2024-03-28T22:12:25.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:25 smithi016 ceph-mon[32374]: pgmap v58: 97 pgs: 97 active+clean; 62 MiB data, 635 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 494 op/s 2024-03-28T22:12:27.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:27 smithi053 ceph-mon[35979]: pgmap v59: 97 pgs: 97 active+clean; 76 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.2 MiB/s wr, 514 op/s 2024-03-28T22:12:27.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:27 smithi063 ceph-mon[36249]: pgmap v59: 97 pgs: 97 active+clean; 76 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.2 MiB/s wr, 514 op/s 2024-03-28T22:12:27.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:27 smithi016 ceph-mon[32374]: pgmap v59: 97 pgs: 97 active+clean; 76 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.2 MiB/s wr, 514 op/s 2024-03-28T22:12:29.478 INFO:tasks.workunit.client.0.smithi016.stderr:Note: switching to '12973fc04fd10d4af086901e10ffa8e48866b735'. 2024-03-28T22:12:29.478 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-03-28T22:12:29.478 INFO:tasks.workunit.client.0.smithi016.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-28T22:12:29.479 INFO:tasks.workunit.client.0.smithi016.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-28T22:12:29.479 INFO:tasks.workunit.client.0.smithi016.stderr:state without impacting any branches by switching back to a branch. 2024-03-28T22:12:29.479 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-03-28T22:12:29.479 INFO:tasks.workunit.client.0.smithi016.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-28T22:12:29.479 INFO:tasks.workunit.client.0.smithi016.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-28T22:12:29.479 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-03-28T22:12:29.479 INFO:tasks.workunit.client.0.smithi016.stderr: git switch -c 2024-03-28T22:12:29.479 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-03-28T22:12:29.479 INFO:tasks.workunit.client.0.smithi016.stderr:Or undo this operation with: 2024-03-28T22:12:29.479 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-03-28T22:12:29.479 INFO:tasks.workunit.client.0.smithi016.stderr: git switch - 2024-03-28T22:12:29.479 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-03-28T22:12:29.479 INFO:tasks.workunit.client.0.smithi016.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-28T22:12:29.479 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-03-28T22:12:29.479 INFO:tasks.workunit.client.0.smithi016.stderr:HEAD is now at 12973fc0 btrfs/022: Correct seq number in golden output 2024-03-28T22:12:29.480 INFO:tasks.workunit.client.0.smithi016.stderr:+ make -j4 2024-03-28T22:12:29.489 INFO:tasks.workunit.client.0.smithi016.stdout:libtoolize -cfi 2024-03-28T22:12:29.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:29 smithi053 ceph-mon[35979]: pgmap v60: 97 pgs: 97 active+clean; 76 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.3 MiB/s wr, 508 op/s 2024-03-28T22:12:29.885 INFO:tasks.workunit.client.0.smithi016.stdout:libtoolize: putting auxiliary files in '.'. 2024-03-28T22:12:29.885 INFO:tasks.workunit.client.0.smithi016.stdout:libtoolize: copying file './config.guess' 2024-03-28T22:12:29.891 INFO:tasks.workunit.client.0.smithi016.stdout:libtoolize: copying file './config.sub' 2024-03-28T22:12:29.897 INFO:tasks.workunit.client.0.smithi016.stdout:libtoolize: copying file './install-sh' 2024-03-28T22:12:29.902 INFO:tasks.workunit.client.0.smithi016.stdout:libtoolize: copying file './ltmain.sh' 2024-03-28T22:12:29.932 INFO:tasks.workunit.client.0.smithi016.stdout:libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'. 2024-03-28T22:12:29.932 INFO:tasks.workunit.client.0.smithi016.stdout:libtoolize: copying file 'm4/libtool.m4' 2024-03-28T22:12:29.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:29 smithi063 ceph-mon[36249]: pgmap v60: 97 pgs: 97 active+clean; 76 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.3 MiB/s wr, 508 op/s 2024-03-28T22:12:29.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:29 smithi016 ceph-mon[32374]: pgmap v60: 97 pgs: 97 active+clean; 76 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.3 MiB/s wr, 508 op/s 2024-03-28T22:12:29.938 INFO:tasks.workunit.client.0.smithi016.stdout:libtoolize: copying file 'm4/ltoptions.m4' 2024-03-28T22:12:29.944 INFO:tasks.workunit.client.0.smithi016.stdout:libtoolize: copying file 'm4/ltsugar.m4' 2024-03-28T22:12:29.949 INFO:tasks.workunit.client.0.smithi016.stdout:libtoolize: copying file 'm4/ltversion.m4' 2024-03-28T22:12:29.955 INFO:tasks.workunit.client.0.smithi016.stdout:libtoolize: copying file 'm4/lt~obsolete.m4' 2024-03-28T22:12:29.961 INFO:tasks.workunit.client.0.smithi016.stdout:libtoolize: Consider adding '-I m4' to ACLOCAL_AMFLAGS in Makefile.am. 2024-03-28T22:12:29.962 INFO:tasks.workunit.client.0.smithi016.stdout:cp include/install-sh . 2024-03-28T22:12:29.983 INFO:tasks.workunit.client.0.smithi016.stdout:aclocal -I m4 2024-03-28T22:12:31.064 INFO:tasks.workunit.client.0.smithi016.stdout:autoheader 2024-03-28T22:12:31.375 INFO:tasks.workunit.client.0.smithi016.stdout:autoconf 2024-03-28T22:12:31.556 INFO:tasks.workunit.client.0.smithi016.stdout:./configure \ 2024-03-28T22:12:31.556 INFO:tasks.workunit.client.0.smithi016.stdout: --libexecdir=/usr/lib \ 2024-03-28T22:12:31.556 INFO:tasks.workunit.client.0.smithi016.stdout: --exec_prefix=/var/lib 2024-03-28T22:12:31.707 INFO:tasks.workunit.client.0.smithi016.stdout:checking build system type... x86_64-pc-linux-gnu 2024-03-28T22:12:31.707 INFO:tasks.workunit.client.0.smithi016.stdout:checking host system type... x86_64-pc-linux-gnu 2024-03-28T22:12:31.708 INFO:tasks.workunit.client.0.smithi016.stdout:checking how to print strings... printf 2024-03-28T22:12:31.708 INFO:tasks.workunit.client.0.smithi016.stdout:checking for gcc... gcc 2024-03-28T22:12:31.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:31 smithi053 ceph-mon[35979]: pgmap v61: 97 pgs: 97 active+clean; 76 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.3 MiB/s wr, 508 op/s 2024-03-28T22:12:31.800 INFO:tasks.workunit.client.0.smithi016.stdout:checking whether the C compiler works... yes 2024-03-28T22:12:31.801 INFO:tasks.workunit.client.0.smithi016.stdout:checking for C compiler default output file name... a.out 2024-03-28T22:12:31.841 INFO:tasks.workunit.client.0.smithi016.stdout:checking for suffix of executables... 2024-03-28T22:12:31.880 INFO:tasks.workunit.client.0.smithi016.stdout:checking whether we are cross compiling... no 2024-03-28T22:12:31.922 INFO:tasks.workunit.client.0.smithi016.stdout:checking for suffix of object files... o 2024-03-28T22:12:31.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:31 smithi063 ceph-mon[36249]: pgmap v61: 97 pgs: 97 active+clean; 76 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.3 MiB/s wr, 508 op/s 2024-03-28T22:12:31.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:31 smithi016 ceph-mon[32374]: pgmap v61: 97 pgs: 97 active+clean; 76 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.3 MiB/s wr, 508 op/s 2024-03-28T22:12:31.949 INFO:tasks.workunit.client.0.smithi016.stdout:checking whether we are using the GNU C compiler... yes 2024-03-28T22:12:31.973 INFO:tasks.workunit.client.0.smithi016.stdout:checking whether gcc accepts -g... yes 2024-03-28T22:12:32.060 INFO:tasks.workunit.client.0.smithi016.stdout:checking for gcc option to accept ISO C89... none needed 2024-03-28T22:12:32.064 INFO:tasks.workunit.client.0.smithi016.stdout:checking for a sed that does not truncate output... /usr/bin/sed 2024-03-28T22:12:32.068 INFO:tasks.workunit.client.0.smithi016.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-03-28T22:12:32.069 INFO:tasks.workunit.client.0.smithi016.stdout:checking for egrep... /usr/bin/grep -E 2024-03-28T22:12:32.070 INFO:tasks.workunit.client.0.smithi016.stdout:checking for fgrep... /usr/bin/grep -F 2024-03-28T22:12:32.073 INFO:tasks.workunit.client.0.smithi016.stdout:checking for ld used by gcc... /usr/bin/ld 2024-03-28T22:12:32.075 INFO:tasks.workunit.client.0.smithi016.stdout:checking if the linker (/usr/bin/ld) is GNU ld... yes 2024-03-28T22:12:32.089 INFO:tasks.workunit.client.0.smithi016.stdout:checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B 2024-03-28T22:12:33.100 INFO:tasks.workunit.client.0.smithi016.stdout:checking the name lister (/usr/bin/nm -B) interface... BSD nm 2024-03-28T22:12:33.100 INFO:tasks.workunit.client.0.smithi016.stdout:checking whether ln -s works... yes 2024-03-28T22:12:33.112 INFO:tasks.workunit.client.0.smithi016.stdout:checking the maximum length of command line arguments... 1572864 2024-03-28T22:12:33.115 INFO:tasks.workunit.client.0.smithi016.stdout:checking how to convert x86_64-pc-linux-gnu file names to x86_64-pc-linux-gnu format... func_convert_file_noop 2024-03-28T22:12:33.115 INFO:tasks.workunit.client.0.smithi016.stdout:checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop 2024-03-28T22:12:33.115 INFO:tasks.workunit.client.0.smithi016.stdout:checking for /usr/bin/ld option to reload object files... -r 2024-03-28T22:12:33.116 INFO:tasks.workunit.client.0.smithi016.stdout:checking for objdump... objdump 2024-03-28T22:12:33.116 INFO:tasks.workunit.client.0.smithi016.stdout:checking how to recognize dependent libraries... pass_all 2024-03-28T22:12:33.117 INFO:tasks.workunit.client.0.smithi016.stdout:checking for dlltool... no 2024-03-28T22:12:33.117 INFO:tasks.workunit.client.0.smithi016.stdout:checking how to associate runtime and link libraries... printf %s\n 2024-03-28T22:12:33.117 INFO:tasks.workunit.client.0.smithi016.stdout:checking for ar... ar 2024-03-28T22:12:33.340 INFO:tasks.workunit.client.0.smithi016.stdout:checking for archiver @FILE support... @ 2024-03-28T22:12:33.340 INFO:tasks.workunit.client.0.smithi016.stdout:checking for strip... strip 2024-03-28T22:12:33.340 INFO:tasks.workunit.client.0.smithi016.stdout:checking for ranlib... ranlib 2024-03-28T22:12:33.340 INFO:tasks.workunit.client.0.smithi016.stdout:checking for gawk... gawk 2024-03-28T22:12:33.458 INFO:tasks.workunit.client.0.smithi016.stdout:checking command to parse /usr/bin/nm -B output from gcc object... ok 2024-03-28T22:12:33.460 INFO:tasks.workunit.client.0.smithi016.stdout:checking for sysroot... no 2024-03-28T22:12:33.478 INFO:tasks.workunit.client.0.smithi016.stdout:checking for a working dd... /usr/bin/dd 2024-03-28T22:12:33.489 INFO:tasks.workunit.client.0.smithi016.stdout:checking how to truncate binary pipes... /usr/bin/dd bs=4096 count=1 2024-03-28T22:12:33.536 INFO:tasks.workunit.client.0.smithi016.stdout:checking for mt... no 2024-03-28T22:12:33.554 INFO:tasks.workunit.client.0.smithi016.stdout:checking if : is a manifest tool... no 2024-03-28T22:12:33.786 INFO:tasks.workunit.client.0.smithi016.stdout:checking how to run the C preprocessor... gcc -E 2024-03-28T22:12:33.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:33 smithi053 ceph-mon[35979]: pgmap v62: 97 pgs: 97 active+clean; 82 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.0 MiB/s wr, 748 op/s 2024-03-28T22:12:33.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:33 smithi063 ceph-mon[36249]: pgmap v62: 97 pgs: 97 active+clean; 82 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.0 MiB/s wr, 748 op/s 2024-03-28T22:12:33.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:33 smithi016 ceph-mon[32374]: pgmap v62: 97 pgs: 97 active+clean; 82 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.0 MiB/s wr, 748 op/s 2024-03-28T22:12:34.017 INFO:tasks.workunit.client.0.smithi016.stdout:checking for ANSI C header files... yes 2024-03-28T22:12:34.059 INFO:tasks.workunit.client.0.smithi016.stdout:checking for sys/types.h... yes 2024-03-28T22:12:34.106 INFO:tasks.workunit.client.0.smithi016.stdout:checking for sys/stat.h... yes 2024-03-28T22:12:34.132 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:12:34.132 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:12:34.149 INFO:tasks.workunit.client.0.smithi016.stdout:checking for stdlib.h... yes 2024-03-28T22:12:34.198 INFO:tasks.workunit.client.0.smithi016.stdout:checking for string.h... yes 2024-03-28T22:12:34.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.239+0000 7fcae79a7640 1 -- 172.21.15.16:0/411832250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae0106690 msgr2=0x7fcae0106a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:34.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.239+0000 7fcae79a7640 1 --2- 172.21.15.16:0/411832250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae0106690 0x7fcae0106a70 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fcadc009930 tx=0x7fcadc02bc30 comp rx=0 tx=0).stop 2024-03-28T22:12:34.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.239+0000 7fcae79a7640 1 -- 172.21.15.16:0/411832250 shutdown_connections 2024-03-28T22:12:34.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.239+0000 7fcae79a7640 1 --2- 172.21.15.16:0/411832250 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcae0100fb0 0x7fcae010d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.239+0000 7fcae79a7640 1 --2- 172.21.15.16:0/411832250 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcae01005f0 0x7fcae0100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.239+0000 7fcae79a7640 1 --2- 172.21.15.16:0/411832250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae0106690 0x7fcae0106a70 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.239+0000 7fcae79a7640 1 -- 172.21.15.16:0/411832250 >> 172.21.15.16:0/411832250 conn(0x7fcae0077800 msgr2=0x7fcae0077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:34.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.239+0000 7fcae79a7640 1 -- 172.21.15.16:0/411832250 shutdown_connections 2024-03-28T22:12:34.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.239+0000 7fcae79a7640 1 -- 172.21.15.16:0/411832250 wait complete. 2024-03-28T22:12:34.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.239+0000 7fcae79a7640 1 Processor -- start 2024-03-28T22:12:34.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.239+0000 7fcae79a7640 1 -- start start 2024-03-28T22:12:34.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.240+0000 7fcae79a7640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcae01005f0 0x7fcae0198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.240+0000 7fcae79a7640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcae0100fb0 0x7fcae0198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.240+0000 7fcae79a7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae0106690 0x7fcae019c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.240+0000 7fcae79a7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcae0110a20 con 0x7fcae0106690 2024-03-28T22:12:34.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.240+0000 7fcae79a7640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcae01108a0 con 0x7fcae01005f0 2024-03-28T22:12:34.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.240+0000 7fcae79a7640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcae0110ba0 con 0x7fcae0100fb0 2024-03-28T22:12:34.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.240+0000 7fcae5f44640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae0106690 0x7fcae019c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.240+0000 7fcae5743640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcae01005f0 0x7fcae0198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.240+0000 7fcae5f44640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae0106690 0x7fcae019c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57048/0 (socket says 172.21.15.16:57048) 2024-03-28T22:12:34.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.240+0000 7fcae5f44640 1 -- 172.21.15.16:0/3044967622 learned_addr learned my addr 172.21.15.16:0/3044967622 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:34.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.240+0000 7fcae4f42640 1 --2- 172.21.15.16:0/3044967622 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcae0100fb0 0x7fcae0198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.240+0000 7fcae5743640 1 -- 172.21.15.16:0/3044967622 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcae0100fb0 msgr2=0x7fcae0198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:34.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.240+0000 7fcae5743640 1 --2- 172.21.15.16:0/3044967622 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcae0100fb0 0x7fcae0198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.240+0000 7fcae5743640 1 -- 172.21.15.16:0/3044967622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae0106690 msgr2=0x7fcae019c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:34.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.240+0000 7fcae5743640 1 --2- 172.21.15.16:0/3044967622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae0106690 0x7fcae019c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.240+0000 7fcae5743640 1 -- 172.21.15.16:0/3044967622 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcae019d000 con 0x7fcae01005f0 2024-03-28T22:12:34.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.240+0000 7fcae5f44640 1 --2- 172.21.15.16:0/3044967622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae0106690 0x7fcae019c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:12:34.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.240+0000 7fcae5743640 1 --2- 172.21.15.16:0/3044967622 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcae01005f0 0x7fcae0198010 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fcadc009230 tx=0x7fcadc004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:34.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.240+0000 7fcad67fc640 1 -- 172.21.15.16:0/3044967622 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcadc043020 con 0x7fcae01005f0 2024-03-28T22:12:34.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.241+0000 7fcae79a7640 1 -- 172.21.15.16:0/3044967622 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcae019d290 con 0x7fcae01005f0 2024-03-28T22:12:34.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.241+0000 7fcad67fc640 1 -- 172.21.15.16:0/3044967622 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcadc0054e0 con 0x7fcae01005f0 2024-03-28T22:12:34.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.241+0000 7fcad67fc640 1 -- 172.21.15.16:0/3044967622 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcadc003a80 con 0x7fcae01005f0 2024-03-28T22:12:34.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.241+0000 7fcae79a7640 1 -- 172.21.15.16:0/3044967622 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcae01a4b80 con 0x7fcae01005f0 2024-03-28T22:12:34.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.243+0000 7fcad67fc640 1 -- 172.21.15.16:0/3044967622 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7fcadc046020 con 0x7fcae01005f0 2024-03-28T22:12:34.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.243+0000 7fcaea582740 1 Processor -- start 2024-03-28T22:12:34.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.243+0000 7fcaea582740 1 -- start start 2024-03-28T22:12:34.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.243+0000 7fcaea582740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bc9f67290 0x560bc9f67670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.243+0000 7fcaea582740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560bc9f67bb0 0x560bc9f70250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.243+0000 7fcaea582740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bc9f70790 0x560bc9f72b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.243+0000 7fcaea582740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560bc9ecb180 con 0x560bc9f70790 2024-03-28T22:12:34.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.243+0000 7fcaea582740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x560bc9e73b70 con 0x560bc9f67290 2024-03-28T22:12:34.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.243+0000 7fcaea582740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x560bc9ec7220 con 0x560bc9f67bb0 2024-03-28T22:12:34.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.244+0000 7fcae4f42640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bc9f70790 0x560bc9f72b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.244+0000 7fcae5f44640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bc9f67290 0x560bc9f67670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.244+0000 7fcae5f44640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bc9f67290 0x560bc9f67670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:48130/0 (socket says 172.21.15.16:48130) 2024-03-28T22:12:34.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.244+0000 7fcae5f44640 1 -- 172.21.15.16:0/4164964643 learned_addr learned my addr 172.21.15.16:0/4164964643 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:34.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.244+0000 7fcad67fc640 1 --2- 172.21.15.16:0/3044967622 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fcacc0a1ef0 0x7fcacc0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.244+0000 7fcae5743640 1 --2- 172.21.15.16:0/4164964643 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560bc9f67bb0 0x560bc9f70250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.244+0000 7fcae5f44640 1 --2- 172.21.15.16:0/3044967622 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fcacc0a1ef0 0x7fcacc0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.244+0000 7fcae5f44640 1 -- 172.21.15.16:0/4164964643 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560bc9f67bb0 msgr2=0x560bc9f70250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:34.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.244+0000 7fcae5f44640 1 --2- 172.21.15.16:0/4164964643 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560bc9f67bb0 0x560bc9f70250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.244+0000 7fcae5f44640 1 -- 172.21.15.16:0/4164964643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bc9f70790 msgr2=0x560bc9f72b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:34.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.244+0000 7fcae5f44640 1 --2- 172.21.15.16:0/4164964643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bc9f70790 0x560bc9f72b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.244+0000 7fcad67fc640 1 -- 172.21.15.16:0/3044967622 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fcadc0efff0 con 0x7fcae01005f0 2024-03-28T22:12:34.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.244+0000 7fcae5f44640 1 -- 172.21.15.16:0/4164964643 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bc9e877c0 con 0x560bc9f67290 2024-03-28T22:12:34.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.244+0000 7fcae5743640 1 --2- 172.21.15.16:0/4164964643 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560bc9f67bb0 0x560bc9f70250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:12:34.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.244+0000 7fcae5f44640 1 --2- 172.21.15.16:0/4164964643 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bc9f67290 0x560bc9f67670 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcad8009fd0 tx=0x7fcad8013830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64b74eed6a61c70f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:34.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.244+0000 7fcae5f44640 1 --2- 172.21.15.16:0/3044967622 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fcacc0a1ef0 0x7fcacc0a43b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcad801c000 tx=0x7fcad8020000 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:34.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.244+0000 7fcacb7fe640 1 -- 172.21.15.16:0/4164964643 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad800e8e0 con 0x560bc9f67290 2024-03-28T22:12:34.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.244+0000 7fcacb7fe640 1 -- 172.21.15.16:0/4164964643 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcad800fe20 con 0x560bc9f67290 2024-03-28T22:12:34.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.244+0000 7fcacb7fe640 1 -- 172.21.15.16:0/4164964643 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad8010690 con 0x560bc9f67290 2024-03-28T22:12:34.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.245+0000 7fcaea582740 1 -- 172.21.15.16:0/4164964643 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bc9f67290 msgr2=0x560bc9f67670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:34.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.245+0000 7fcaea582740 1 --2- 172.21.15.16:0/4164964643 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bc9f67290 0x560bc9f67670 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcad8009fd0 tx=0x7fcad8013830 comp rx=0 tx=0).stop 2024-03-28T22:12:34.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.245+0000 7fcaea582740 1 -- 172.21.15.16:0/4164964643 shutdown_connections 2024-03-28T22:12:34.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.245+0000 7fcaea582740 1 --2- 172.21.15.16:0/4164964643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bc9f70790 0x560bc9f72b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.245+0000 7fcaea582740 1 --2- 172.21.15.16:0/4164964643 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560bc9f67bb0 0x560bc9f70250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.245+0000 7fcaea582740 1 --2- 172.21.15.16:0/4164964643 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bc9f67290 0x560bc9f67670 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.245+0000 7fcaea582740 1 -- 172.21.15.16:0/4164964643 >> 172.21.15.16:0/4164964643 conn(0x560bc9f619c0 msgr2=0x560bc9f61da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:34.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.245+0000 7fcaea582740 1 -- 172.21.15.16:0/4164964643 shutdown_connections 2024-03-28T22:12:34.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.245+0000 7fcaea582740 1 -- 172.21.15.16:0/4164964643 wait complete. 2024-03-28T22:12:34.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.245+0000 7fcaea582740 1 -- start start 2024-03-28T22:12:34.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.245+0000 7fcaea582740 1 Processor -- start 2024-03-28T22:12:34.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.246+0000 7fcaea582740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bc9f67290 0x560bc9f6d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.246+0000 7fcaea582740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bc9f67bb0 0x560bc9f6dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.246+0000 7fcaea582740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560bc9f70790 0x560bc9f6e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.246+0000 7fcaea582740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560bc9f734c0 con 0x560bc9f67290 2024-03-28T22:12:34.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.246+0000 7fcaea582740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x560bc9e694a0 con 0x560bc9f67bb0 2024-03-28T22:12:34.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.246+0000 7fcaea582740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x560bc9f73640 con 0x560bc9f70790 2024-03-28T22:12:34.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.246+0000 7fcaea582740 10 client.0 _ll_register_callbacks cb 0x560bc9e6cd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:12:34.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.246+0000 7fcae5743640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bc9f67290 0x560bc9f6d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.246+0000 7fcae5743640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bc9f67290 0x560bc9f6d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57062/0 (socket says 172.21.15.16:57062) 2024-03-28T22:12:34.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.246+0000 7fcae4f42640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bc9f67bb0 0x560bc9f6dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.246+0000 7fcae5743640 1 -- 172.21.15.16:0/2517215715 learned_addr learned my addr 172.21.15.16:0/2517215715 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:34.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.246+0000 7fcae5f44640 1 --2- 172.21.15.16:0/2517215715 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560bc9f70790 0x560bc9f6e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.246+0000 7fcae5743640 1 -- 172.21.15.16:0/2517215715 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560bc9f70790 msgr2=0x560bc9f6e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:34.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.246+0000 7fcae5743640 1 --2- 172.21.15.16:0/2517215715 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560bc9f70790 0x560bc9f6e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.246+0000 7fcae5743640 1 -- 172.21.15.16:0/2517215715 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bc9f67bb0 msgr2=0x560bc9f6dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:34.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.246+0000 7fcae5743640 1 --2- 172.21.15.16:0/2517215715 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bc9f67bb0 0x560bc9f6dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.246+0000 7fcae5743640 1 -- 172.21.15.16:0/2517215715 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bc9ca97d0 con 0x560bc9f67290 2024-03-28T22:12:34.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.246+0000 7fcae5f44640 1 --2- 172.21.15.16:0/2517215715 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560bc9f70790 0x560bc9f6e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:12:34.256 INFO:tasks.workunit.client.0.smithi016.stdout:checking for memory.h... yes 2024-03-28T22:12:34.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.246+0000 7fcae5743640 1 --2- 172.21.15.16:0/2517215715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bc9f67290 0x560bc9f6d820 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fcadc006010 tx=0x7fcadc009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9df34d3c28dcb68d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:34.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.246+0000 7fcac8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:12:34.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.247+0000 7fcac8ff9640 1 -- 172.21.15.16:0/2517215715 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcadc0edb80 con 0x560bc9f67290 2024-03-28T22:12:34.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.247+0000 7fcac8ff9640 1 -- 172.21.15.16:0/2517215715 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fcadc0bcd10 con 0x560bc9f67290 2024-03-28T22:12:34.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.247+0000 7fcaea582740 1 -- 172.21.15.16:0/2517215715 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcad800fe20 con 0x560bc9f67290 2024-03-28T22:12:34.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.247+0000 7fcac8ff9640 1 -- 172.21.15.16:0/2517215715 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fcadc0bcab0 con 0x560bc9f67290 2024-03-28T22:12:34.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.247+0000 7fcaea582740 10 client.15876 fetch_fsmap learned FSMap version 24 2024-03-28T22:12:34.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.247+0000 7fcaea582740 1 -- 172.21.15.16:0/2517215715 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x560bc9bdf3f0 con 0x560bc9f67290 2024-03-28T22:12:34.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.247+0000 7fcac8ff9640 1 -- 172.21.15.16:0/2517215715 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcad800fe20 con 0x560bc9f67290 2024-03-28T22:12:34.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.247+0000 7fcac8ff9640 1 -- 172.21.15.16:0/2517215715 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7fcadc0c4dc0 con 0x560bc9f67290 2024-03-28T22:12:34.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.247+0000 7fcaea582740 10 client.15876 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:12:34.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.247+0000 7fcaea582740 10 client.15876 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:12:34.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.247+0000 7fcaea582740 20 client.15876 populate_metadata read hostname 'smithi016' 2024-03-28T22:12:34.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.247+0000 7fcaea582740 1 --2- 172.21.15.16:0/2517215715 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x560bc9f80970 0x560bc9f82e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.247+0000 7fcaea582740 4 client.15876 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:12:34.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.247+0000 7fcaea582740 1 -- 172.21.15.16:0/2517215715 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560bc9ed86c0 con 0x560bc9f80970 2024-03-28T22:12:34.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.247+0000 7fcae4f42640 1 --2- 172.21.15.16:0/2517215715 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x560bc9f80970 0x560bc9f82e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.248+0000 7fcae4f42640 1 --2- 172.21.15.16:0/2517215715 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x560bc9f80970 0x560bc9f82e10 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9dabd3c8010cfd16 server_cookie=47b63f7c7d2c5fa5 in_seq=0 out_seq=0 2024-03-28T22:12:34.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.248+0000 7fcac8ff9640 10 client.15876 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:12:34.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.249+0000 7fcac8ff9640 1 -- 172.21.15.16:0/2517215715 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560bc9ed86c0 con 0x560bc9f80970 2024-03-28T22:12:34.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.249+0000 7fcac8ff9640 10 client.15876 handle_command_reply: tid=0 2024-03-28T22:12:34.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.249+0000 7fcaea582740 1 client.15876 shutdown 2024-03-28T22:12:34.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.249+0000 7fcaea582740 1 -- 172.21.15.16:0/2517215715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bc9f67290 msgr2=0x560bc9f6d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:34.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.249+0000 7fcaea582740 1 --2- 172.21.15.16:0/2517215715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bc9f67290 0x560bc9f6d820 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fcadc006010 tx=0x7fcadc009260 comp rx=0 tx=0).stop 2024-03-28T22:12:34.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.249+0000 7fcaea582740 1 -- 172.21.15.16:0/2517215715 shutdown_connections 2024-03-28T22:12:34.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.249+0000 7fcaea582740 1 --2- 172.21.15.16:0/2517215715 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x560bc9f80970 0x560bc9f82e10 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.249+0000 7fcaea582740 1 --2- 172.21.15.16:0/2517215715 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560bc9f70790 0x560bc9f6e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.249+0000 7fcaea582740 1 --2- 172.21.15.16:0/2517215715 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bc9f67bb0 0x560bc9f6dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.249+0000 7fcac8ff9640 0 client.15876 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:12:34.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.249+0000 7fcaea582740 1 --2- 172.21.15.16:0/2517215715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bc9f67290 0x560bc9f6d820 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.249+0000 7fcaea582740 1 -- 172.21.15.16:0/2517215715 >> 172.21.15.16:0/2517215715 conn(0x560bc9f619c0 msgr2=0x560bc9f60c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:34.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.249+0000 7fcaea582740 1 -- 172.21.15.16:0/2517215715 shutdown_connections 2024-03-28T22:12:34.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.249+0000 7fcaea582740 1 -- 172.21.15.16:0/2517215715 wait complete. 2024-03-28T22:12:34.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.249+0000 7fcaea582740 20 client.15876 trim_cache size 0 max 16384 2024-03-28T22:12:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.267+0000 7fcaea582740 1 Processor -- start 2024-03-28T22:12:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.267+0000 7fcaea582740 1 -- start start 2024-03-28T22:12:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.267+0000 7fcaea582740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560bc9f67290 0x560bc9f6ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.267+0000 7fcaea582740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bc9f67bb0 0x560bc9f60390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.267+0000 7fcaea582740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bc9f70790 0x560bc9f7c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.267+0000 7fcaea582740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcae01afbc0 con 0x560bc9f67bb0 2024-03-28T22:12:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.267+0000 7fcaea582740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcae01ad460 con 0x560bc9f70790 2024-03-28T22:12:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.267+0000 7fcaea582740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcae01ab2b0 con 0x560bc9f67290 2024-03-28T22:12:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.267+0000 7fcae4f42640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bc9f67bb0 0x560bc9f60390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.267+0000 7fcae4f42640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bc9f67bb0 0x560bc9f60390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57068/0 (socket says 172.21.15.16:57068) 2024-03-28T22:12:34.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.267+0000 7fcae5f44640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bc9f70790 0x560bc9f7c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.267+0000 7fcae4f42640 1 -- 172.21.15.16:0/2362575554 learned_addr learned my addr 172.21.15.16:0/2362575554 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:34.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.268+0000 7fcae5743640 1 --2- 172.21.15.16:0/2362575554 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560bc9f67290 0x560bc9f6ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.268+0000 7fcae4f42640 1 -- 172.21.15.16:0/2362575554 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560bc9f67290 msgr2=0x560bc9f6ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:34.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.268+0000 7fcae4f42640 1 --2- 172.21.15.16:0/2362575554 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560bc9f67290 0x560bc9f6ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.268+0000 7fcae4f42640 1 -- 172.21.15.16:0/2362575554 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bc9f70790 msgr2=0x560bc9f7c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:34.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.268+0000 7fcae4f42640 1 --2- 172.21.15.16:0/2362575554 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bc9f70790 0x560bc9f7c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.268+0000 7fcae4f42640 1 -- 172.21.15.16:0/2362575554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bc9f679e0 con 0x560bc9f67bb0 2024-03-28T22:12:34.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.268+0000 7fcae4f42640 1 --2- 172.21.15.16:0/2362575554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bc9f67bb0 0x560bc9f60390 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x560bc9f6d080 tx=0x7fcad0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38015f6ce43b1941 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:34.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.268+0000 7fcacaffd640 1 -- 172.21.15.16:0/2362575554 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad001f050 con 0x560bc9f67bb0 2024-03-28T22:12:34.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.268+0000 7fcacaffd640 1 -- 172.21.15.16:0/2362575554 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcad0023020 con 0x560bc9f67bb0 2024-03-28T22:12:34.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.268+0000 7fcacaffd640 1 -- 172.21.15.16:0/2362575554 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad001ed20 con 0x560bc9f67bb0 2024-03-28T22:12:34.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.269+0000 7fcaea582740 1 -- 172.21.15.16:0/2362575554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bc9f67bb0 msgr2=0x560bc9f60390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:34.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.269+0000 7fcaea582740 1 --2- 172.21.15.16:0/2362575554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bc9f67bb0 0x560bc9f60390 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x560bc9f6d080 tx=0x7fcad0005f90 comp rx=0 tx=0).stop 2024-03-28T22:12:34.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.269+0000 7fcaea582740 1 -- 172.21.15.16:0/2362575554 shutdown_connections 2024-03-28T22:12:34.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.269+0000 7fcaea582740 1 --2- 172.21.15.16:0/2362575554 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bc9f70790 0x560bc9f7c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.269+0000 7fcaea582740 1 --2- 172.21.15.16:0/2362575554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bc9f67bb0 0x560bc9f60390 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.269+0000 7fcaea582740 1 --2- 172.21.15.16:0/2362575554 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560bc9f67290 0x560bc9f6ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.269+0000 7fcaea582740 1 -- 172.21.15.16:0/2362575554 >> 172.21.15.16:0/2362575554 conn(0x560bc9f619c0 msgr2=0x560bc9f63680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:34.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.269+0000 7fcaea582740 1 -- 172.21.15.16:0/2362575554 shutdown_connections 2024-03-28T22:12:34.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.269+0000 7fcaea582740 1 -- 172.21.15.16:0/2362575554 wait complete. 2024-03-28T22:12:34.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.269+0000 7fcaea582740 1 -- start start 2024-03-28T22:12:34.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.269+0000 7fcaea582740 1 Processor -- start 2024-03-28T22:12:34.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.270+0000 7fcaea582740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bc9f67290 0x560bc9f78cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.270+0000 7fcaea582740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bc9f67bb0 0x560bc9f740d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.270+0000 7fcaea582740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560bc9f70790 0x560bc9f74730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.270+0000 7fcaea582740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcae01ab920 con 0x560bc9f67290 2024-03-28T22:12:34.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.270+0000 7fcaea582740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcae01acd90 con 0x560bc9f67bb0 2024-03-28T22:12:34.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.270+0000 7fcaea582740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcae01abff0 con 0x560bc9f70790 2024-03-28T22:12:34.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.270+0000 7fcaea582740 10 client.0 _ll_register_callbacks cb 0x7fcadc0bc950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:12:34.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.270+0000 7fcae5743640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bc9f67290 0x560bc9f78cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.270+0000 7fcae5743640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bc9f67290 0x560bc9f78cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57070/0 (socket says 172.21.15.16:57070) 2024-03-28T22:12:34.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.270+0000 7fcae5743640 1 -- 172.21.15.16:0/240098581 learned_addr learned my addr 172.21.15.16:0/240098581 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:34.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.270+0000 7fcae4f42640 1 --2- 172.21.15.16:0/240098581 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bc9f67bb0 0x560bc9f740d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.270+0000 7fcae5f44640 1 --2- 172.21.15.16:0/240098581 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560bc9f70790 0x560bc9f74730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.270+0000 7fcae5743640 1 -- 172.21.15.16:0/240098581 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560bc9f70790 msgr2=0x560bc9f74730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:34.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.270+0000 7fcae5743640 1 --2- 172.21.15.16:0/240098581 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560bc9f70790 0x560bc9f74730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.270+0000 7fcae5743640 1 -- 172.21.15.16:0/240098581 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bc9f67bb0 msgr2=0x560bc9f740d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:34.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.270+0000 7fcae5743640 1 --2- 172.21.15.16:0/240098581 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bc9f67bb0 0x560bc9f740d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.270+0000 7fcae5743640 1 -- 172.21.15.16:0/240098581 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bc9b9f0f0 con 0x560bc9f67290 2024-03-28T22:12:34.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.270+0000 7fcae5f44640 1 --2- 172.21.15.16:0/240098581 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560bc9f70790 0x560bc9f74730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:12:34.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.270+0000 7fcae5743640 1 --2- 172.21.15.16:0/240098581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bc9f67290 0x560bc9f78cc0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fcadc0bcc70 tx=0x7fcadc02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2005d31b3cb67af3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:34.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.270+0000 7fcac8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:12:34.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.270+0000 7fcac8ff9640 1 -- 172.21.15.16:0/240098581 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcadc0da020 con 0x560bc9f67290 2024-03-28T22:12:34.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.270+0000 7fcac8ff9640 1 -- 172.21.15.16:0/240098581 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fcadc0d0d00 con 0x560bc9f67290 2024-03-28T22:12:34.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.270+0000 7fcaea582740 1 -- 172.21.15.16:0/240098581 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcad0023020 con 0x560bc9f67290 2024-03-28T22:12:34.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.271+0000 7fcac8ff9640 1 -- 172.21.15.16:0/240098581 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fcadc0edb90 con 0x560bc9f67290 2024-03-28T22:12:34.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.271+0000 7fcaea582740 10 client.15882 fetch_fsmap learned FSMap version 24 2024-03-28T22:12:34.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.271+0000 7fcaea582740 1 -- 172.21.15.16:0/240098581 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x560bc9fe6cd0 con 0x560bc9f67290 2024-03-28T22:12:34.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.271+0000 7fcac8ff9640 1 -- 172.21.15.16:0/240098581 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcad0023020 con 0x560bc9f67290 2024-03-28T22:12:34.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.271+0000 7fcac8ff9640 1 -- 172.21.15.16:0/240098581 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7fcadc0bb970 con 0x560bc9f67290 2024-03-28T22:12:34.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.271+0000 7fcaea582740 10 client.15882 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:12:34.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.271+0000 7fcaea582740 10 client.15882 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:12:34.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.271+0000 7fcaea582740 20 client.15882 populate_metadata read hostname 'smithi016' 2024-03-28T22:12:34.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.271+0000 7fcaea582740 1 --2- 172.21.15.16:0/240098581 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x560bc9fe7340 0x560bc9fe9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.271+0000 7fcaea582740 4 client.15882 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:12:34.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.271+0000 7fcaea582740 1 -- 172.21.15.16:0/240098581 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560bc9ed2a10 con 0x560bc9fe7340 2024-03-28T22:12:34.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.271+0000 7fcae4f42640 1 --2- 172.21.15.16:0/240098581 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x560bc9fe7340 0x560bc9fe9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.272+0000 7fcac8ff9640 10 client.15882 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:12:34.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.272+0000 7fcae4f42640 1 --2- 172.21.15.16:0/240098581 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x560bc9fe7340 0x560bc9fe9760 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd31886e49256ed2 server_cookie=45c1e73ce3621b23 in_seq=0 out_seq=0 2024-03-28T22:12:34.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.272+0000 7fcac8ff9640 1 -- 172.21.15.16:0/240098581 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x560bc9ed2a10 con 0x560bc9fe7340 2024-03-28T22:12:34.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.272+0000 7fcac8ff9640 10 client.15882 handle_command_reply: tid=0 2024-03-28T22:12:34.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.272+0000 7fcaea582740 1 client.15882 shutdown 2024-03-28T22:12:34.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.272+0000 7fcaea582740 1 -- 172.21.15.16:0/240098581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bc9f67290 msgr2=0x560bc9f78cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:34.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.272+0000 7fcaea582740 1 --2- 172.21.15.16:0/240098581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bc9f67290 0x560bc9f78cc0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fcadc0bcc70 tx=0x7fcadc02bc30 comp rx=0 tx=0).stop 2024-03-28T22:12:34.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.272+0000 7fcaea582740 1 -- 172.21.15.16:0/240098581 shutdown_connections 2024-03-28T22:12:34.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.272+0000 7fcaea582740 1 --2- 172.21.15.16:0/240098581 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x560bc9fe7340 0x560bc9fe9760 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.272+0000 7fcaea582740 1 --2- 172.21.15.16:0/240098581 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560bc9f70790 0x560bc9f74730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.272+0000 7fcac8ff9640 0 client.15882 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:12:34.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.272+0000 7fcaea582740 1 --2- 172.21.15.16:0/240098581 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560bc9f67bb0 0x560bc9f740d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.272+0000 7fcaea582740 1 --2- 172.21.15.16:0/240098581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560bc9f67290 0x560bc9f78cc0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.272+0000 7fcaea582740 1 -- 172.21.15.16:0/240098581 >> 172.21.15.16:0/240098581 conn(0x560bc9f619c0 msgr2=0x560bc9f6d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:34.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.272+0000 7fcaea582740 1 -- 172.21.15.16:0/240098581 shutdown_connections 2024-03-28T22:12:34.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.272+0000 7fcaea582740 1 -- 172.21.15.16:0/240098581 wait complete. 2024-03-28T22:12:34.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.272+0000 7fcaea582740 20 client.15882 trim_cache size 0 max 16384 2024-03-28T22:12:34.282 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:12:34.283 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-03-28T22:12:34.283 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-03-28T22:12:34.283 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:12:34.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.273+0000 7fcae79a7640 1 -- 172.21.15.16:0/3044967622 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fcacc0a1ef0 msgr2=0x7fcacc0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:34.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.273+0000 7fcae79a7640 1 --2- 172.21.15.16:0/3044967622 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fcacc0a1ef0 0x7fcacc0a43b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcad801c000 tx=0x7fcad8020000 comp rx=0 tx=0).stop 2024-03-28T22:12:34.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.273+0000 7fcae79a7640 1 -- 172.21.15.16:0/3044967622 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcae01005f0 msgr2=0x7fcae0198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:34.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.273+0000 7fcae79a7640 1 --2- 172.21.15.16:0/3044967622 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcae01005f0 0x7fcae0198010 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fcadc009230 tx=0x7fcadc004360 comp rx=0 tx=0).stop 2024-03-28T22:12:34.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.273+0000 7fcae79a7640 1 -- 172.21.15.16:0/3044967622 shutdown_connections 2024-03-28T22:12:34.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.273+0000 7fcae79a7640 1 --2- 172.21.15.16:0/3044967622 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fcacc0a1ef0 0x7fcacc0a43b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.273+0000 7fcae79a7640 1 --2- 172.21.15.16:0/3044967622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae0106690 0x7fcae019c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.273+0000 7fcae79a7640 1 --2- 172.21.15.16:0/3044967622 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcae0100fb0 0x7fcae0198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.273+0000 7fcae79a7640 1 --2- 172.21.15.16:0/3044967622 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcae01005f0 0x7fcae0198010 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.273+0000 7fcae79a7640 1 -- 172.21.15.16:0/3044967622 >> 172.21.15.16:0/3044967622 conn(0x7fcae0077800 msgr2=0x7fcae00ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:34.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.273+0000 7fcae79a7640 1 -- 172.21.15.16:0/3044967622 shutdown_connections 2024-03-28T22:12:34.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.273+0000 7fcae79a7640 1 -- 172.21.15.16:0/3044967622 wait complete. 2024-03-28T22:12:34.287 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:12:34.287 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:12:34.292 INFO:tasks.workunit.client.0.smithi016.stdout:checking for strings.h... yes 2024-03-28T22:12:34.334 INFO:tasks.workunit.client.0.smithi016.stdout:checking for inttypes.h... yes 2024-03-28T22:12:34.380 INFO:tasks.workunit.client.0.smithi016.stdout:checking for stdint.h... yes 2024-03-28T22:12:34.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.395+0000 7f7ac7548640 1 -- 172.21.15.16:0/3971090848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ac0073b80 msgr2=0x7f7ac0073f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:34.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.395+0000 7f7ac7548640 1 --2- 172.21.15.16:0/3971090848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ac0073b80 0x7f7ac0073f60 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f7abc009900 tx=0x7f7abc02bcd0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.395+0000 7f7ac7548640 1 -- 172.21.15.16:0/3971090848 shutdown_connections 2024-03-28T22:12:34.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.395+0000 7f7ac7548640 1 --2- 172.21.15.16:0/3971090848 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ac010df60 0x7f7ac01103b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.395+0000 7f7ac7548640 1 --2- 172.21.15.16:0/3971090848 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ac00744a0 0x7f7ac010da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.395+0000 7f7ac7548640 1 --2- 172.21.15.16:0/3971090848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ac0073b80 0x7f7ac0073f60 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.395+0000 7f7ac7548640 1 -- 172.21.15.16:0/3971090848 >> 172.21.15.16:0/3971090848 conn(0x7f7ac00fd230 msgr2=0x7f7ac00ff650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:34.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.395+0000 7f7ac7548640 1 -- 172.21.15.16:0/3971090848 shutdown_connections 2024-03-28T22:12:34.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.395+0000 7f7ac7548640 1 -- 172.21.15.16:0/3971090848 wait complete. 2024-03-28T22:12:34.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.395+0000 7f7ac7548640 1 Processor -- start 2024-03-28T22:12:34.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.395+0000 7f7ac7548640 1 -- start start 2024-03-28T22:12:34.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.396+0000 7f7ac7548640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ac0073b80 0x7f7ac019a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.396+0000 7f7ac7548640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ac00744a0 0x7f7ac019a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.396+0000 7f7ac7548640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ac010df60 0x7f7ac019eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.396+0000 7f7ac7548640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7ac0113d00 con 0x7f7ac010df60 2024-03-28T22:12:34.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.396+0000 7f7ac7548640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7ac0113b80 con 0x7f7ac00744a0 2024-03-28T22:12:34.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.396+0000 7f7ac7548640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7ac0113e80 con 0x7f7ac0073b80 2024-03-28T22:12:34.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.396+0000 7f7ac5ae5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ac010df60 0x7f7ac019eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.396+0000 7f7ac5ae5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ac010df60 0x7f7ac019eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57096/0 (socket says 172.21.15.16:57096) 2024-03-28T22:12:34.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.396+0000 7f7ac4ae3640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ac00744a0 0x7f7ac019a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.396+0000 7f7ac5ae5640 1 -- 172.21.15.16:0/447753672 learned_addr learned my addr 172.21.15.16:0/447753672 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:34.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.396+0000 7f7ac52e4640 1 --2- 172.21.15.16:0/447753672 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ac0073b80 0x7f7ac019a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.396+0000 7f7ac5ae5640 1 -- 172.21.15.16:0/447753672 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ac0073b80 msgr2=0x7f7ac019a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:34.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.396+0000 7f7ac5ae5640 1 --2- 172.21.15.16:0/447753672 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ac0073b80 0x7f7ac019a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.396+0000 7f7ac5ae5640 1 -- 172.21.15.16:0/447753672 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ac00744a0 msgr2=0x7f7ac019a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:34.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.396+0000 7f7ac5ae5640 1 --2- 172.21.15.16:0/447753672 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ac00744a0 0x7f7ac019a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.396+0000 7f7ac5ae5640 1 -- 172.21.15.16:0/447753672 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ac019f270 con 0x7f7ac010df60 2024-03-28T22:12:34.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.396+0000 7f7ac4ae3640 1 --2- 172.21.15.16:0/447753672 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ac00744a0 0x7f7ac019a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:12:34.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.396+0000 7f7ac5ae5640 1 --2- 172.21.15.16:0/447753672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ac010df60 0x7f7ac019eb40 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f7ab8009fd0 tx=0x7f7ab800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:34.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.396+0000 7f7ab67fc640 1 -- 172.21.15.16:0/447753672 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab8017070 con 0x7f7ac010df60 2024-03-28T22:12:34.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.397+0000 7f7ab67fc640 1 -- 172.21.15.16:0/447753672 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ab8003e50 con 0x7f7ac010df60 2024-03-28T22:12:34.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.397+0000 7f7ac7548640 1 -- 172.21.15.16:0/447753672 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ac019f560 con 0x7f7ac010df60 2024-03-28T22:12:34.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.397+0000 7f7ab67fc640 1 -- 172.21.15.16:0/447753672 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab80106c0 con 0x7f7ac010df60 2024-03-28T22:12:34.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.397+0000 7f7ac7548640 1 -- 172.21.15.16:0/447753672 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ac01a6df0 con 0x7f7ac010df60 2024-03-28T22:12:34.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.398+0000 7f7ac7548640 1 -- 172.21.15.16:0/447753672 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ac00685d0 con 0x7f7ac010df60 2024-03-28T22:12:34.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.399+0000 7f7ab67fc640 1 -- 172.21.15.16:0/447753672 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f7ab8003970 con 0x7f7ac010df60 2024-03-28T22:12:34.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.400+0000 7f7ab67fc640 1 --2- 172.21.15.16:0/447753672 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f7aac0a1ef0 0x7f7aac0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.400+0000 7f7ac52e4640 1 --2- 172.21.15.16:0/447753672 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f7aac0a1ef0 0x7f7aac0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.400+0000 7f7ab67fc640 1 -- 172.21.15.16:0/447753672 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f7ab80ce0e0 con 0x7f7ac010df60 2024-03-28T22:12:34.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.401+0000 7f7ac52e4640 1 --2- 172.21.15.16:0/447753672 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f7aac0a1ef0 0x7f7aac0a43b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7abc02efd0 tx=0x7f7abc02cf90 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:34.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.405+0000 7f7ab67fc640 1 -- 172.21.15.16:0/447753672 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7ab809aba0 con 0x7f7ac010df60 2024-03-28T22:12:34.432 INFO:tasks.workunit.client.0.smithi016.stdout:checking for unistd.h... yes 2024-03-28T22:12:34.485 INFO:tasks.workunit.client.0.smithi016.stdout:checking for dlfcn.h... yes 2024-03-28T22:12:34.491 INFO:tasks.workunit.client.0.smithi016.stdout:checking for objdir... .libs 2024-03-28T22:12:34.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.633+0000 7f7ac7548640 1 -- 172.21.15.16:0/447753672 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ac002d170 con 0x7f7ac010df60 2024-03-28T22:12:34.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.633+0000 7f7ab67fc640 1 -- 172.21.15.16:0/447753672 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11855 (secure 0 0 0) 0x7f7ab809ec40 con 0x7f7ac010df60 2024-03-28T22:12:34.636 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:12:34.636 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/1644581227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1644581227},{"type":"v1","addr":"172.21.15.16:6839","nonce":1644581227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/2593062669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2593062669},{"type":"v1","addr":"172.21.15.16:6841","nonce":2593062669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:08:29.672976+0000","modified":"2024-03-28T22:11:54.920802+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24566,"mds_1":24524,"mds_2":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24466":{"gid":24466,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/247249857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":247249857},{"type":"v1","addr":"172.21.15.63:6835","nonce":247249857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6837/2324231032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":2324231032},{"type":"v1","addr":"172.21.15.63:6837","nonce":2324231032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6839/1177197077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":1177197077},{"type":"v1","addr":"172.21.15.63:6839","nonce":1177197077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"e","rank":2,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.53:6835/350727634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":350727634},{"type":"v1","addr":"172.21.15.53:6835","nonce":350727634}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/349877573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":349877573},{"type":"v1","addr":"172.21.15.53:6837","nonce":349877573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6839/3714153956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":3714153956},{"type":"v1","addr":"172.21.15.53:6839","nonce":3714153956}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24566,"qdb_cluster":[24500,24566,24524]},"id":1}]} 2024-03-28T22:12:34.637 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 24 2024-03-28T22:12:34.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.635+0000 7f7ac7548640 1 -- 172.21.15.16:0/447753672 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f7aac0a1ef0 msgr2=0x7f7aac0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:34.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.635+0000 7f7ac7548640 1 --2- 172.21.15.16:0/447753672 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f7aac0a1ef0 0x7f7aac0a43b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7abc02efd0 tx=0x7f7abc02cf90 comp rx=0 tx=0).stop 2024-03-28T22:12:34.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.635+0000 7f7ac7548640 1 -- 172.21.15.16:0/447753672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ac010df60 msgr2=0x7f7ac019eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:34.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.635+0000 7f7ac7548640 1 --2- 172.21.15.16:0/447753672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ac010df60 0x7f7ac019eb40 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f7ab8009fd0 tx=0x7f7ab800b260 comp rx=0 tx=0).stop 2024-03-28T22:12:34.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.635+0000 7f7ac7548640 1 -- 172.21.15.16:0/447753672 shutdown_connections 2024-03-28T22:12:34.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.635+0000 7f7ac7548640 1 --2- 172.21.15.16:0/447753672 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f7aac0a1ef0 0x7f7aac0a43b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.635+0000 7f7ac7548640 1 --2- 172.21.15.16:0/447753672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ac010df60 0x7f7ac019eb40 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.635+0000 7f7ac7548640 1 --2- 172.21.15.16:0/447753672 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ac00744a0 0x7f7ac019a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.635+0000 7f7ac7548640 1 --2- 172.21.15.16:0/447753672 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ac0073b80 0x7f7ac019a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.635+0000 7f7ac7548640 1 -- 172.21.15.16:0/447753672 >> 172.21.15.16:0/447753672 conn(0x7f7ac00fd230 msgr2=0x7f7ac00feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:34.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.636+0000 7f7ac7548640 1 -- 172.21.15.16:0/447753672 shutdown_connections 2024-03-28T22:12:34.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.636+0000 7f7ac7548640 1 -- 172.21.15.16:0/447753672 wait complete. 2024-03-28T22:12:34.646 INFO:tasks.workunit.client.0.smithi016.stdout:checking if gcc supports -fno-rtti -fno-exceptions... no 2024-03-28T22:12:34.646 INFO:tasks.workunit.client.0.smithi016.stdout:checking for gcc option to produce PIC... -fPIC -DPIC 2024-03-28T22:12:34.649 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:12:34.699 INFO:tasks.workunit.client.0.smithi016.stdout:checking if gcc PIC flag -fPIC -DPIC works... yes 2024-03-28T22:12:34.744 INFO:tasks.workunit.client.0.smithi016.stdout:checking if gcc static flag -static works... no 2024-03-28T22:12:34.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.794+0000 7f3db4801640 1 -- 172.21.15.16:0/1746925856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3db0102780 msgr2=0x7f3db0102b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:34.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.794+0000 7f3db4801640 1 --2- 172.21.15.16:0/1746925856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3db0102780 0x7f3db0102b60 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f3da0009900 tx=0x7f3da002bcd0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.795+0000 7f3db4801640 1 -- 172.21.15.16:0/1746925856 shutdown_connections 2024-03-28T22:12:34.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.795+0000 7f3db4801640 1 --2- 172.21.15.16:0/1746925856 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3db010bb50 0x7f3db010df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.795+0000 7f3db4801640 1 --2- 172.21.15.16:0/1746925856 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3db01030a0 0x7f3db010b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.795+0000 7f3db4801640 1 --2- 172.21.15.16:0/1746925856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3db0102780 0x7f3db0102b60 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.795+0000 7f3db4801640 1 -- 172.21.15.16:0/1746925856 >> 172.21.15.16:0/1746925856 conn(0x7f3db00771e0 msgr2=0x7f3db00fe570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:34.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.795+0000 7f3db4801640 1 -- 172.21.15.16:0/1746925856 shutdown_connections 2024-03-28T22:12:34.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.795+0000 7f3db4801640 1 -- 172.21.15.16:0/1746925856 wait complete. 2024-03-28T22:12:34.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.795+0000 7f3db4801640 1 Processor -- start 2024-03-28T22:12:34.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.795+0000 7f3db4801640 1 -- start start 2024-03-28T22:12:34.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.796+0000 7f3db4801640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3db0102780 0x7f3db0197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.796+0000 7f3db4801640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3db01030a0 0x7f3db01983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.796+0000 7f3db4801640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3db010bb50 0x7f3db019c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.796+0000 7f3db4801640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3db01118d0 con 0x7f3db01030a0 2024-03-28T22:12:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.796+0000 7f3db4801640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3db0111750 con 0x7f3db0102780 2024-03-28T22:12:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.796+0000 7f3db4801640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3db0111a50 con 0x7f3db010bb50 2024-03-28T22:12:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.796+0000 7f3da659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3db01030a0 0x7f3db01983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.796+0000 7f3daed9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3db0102780 0x7f3db0197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.796+0000 7f3da659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3db01030a0 0x7f3db01983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57114/0 (socket says 172.21.15.16:57114) 2024-03-28T22:12:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.796+0000 7f3da659c640 1 -- 172.21.15.16:0/3688648915 learned_addr learned my addr 172.21.15.16:0/3688648915 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.796+0000 7f3daf59e640 1 --2- 172.21.15.16:0/3688648915 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3db010bb50 0x7f3db019c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.796+0000 7f3da659c640 1 -- 172.21.15.16:0/3688648915 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3db010bb50 msgr2=0x7f3db019c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.796+0000 7f3da659c640 1 --2- 172.21.15.16:0/3688648915 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3db010bb50 0x7f3db019c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.796+0000 7f3da659c640 1 -- 172.21.15.16:0/3688648915 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3db0102780 msgr2=0x7f3db0197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.796+0000 7f3da659c640 1 --2- 172.21.15.16:0/3688648915 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3db0102780 0x7f3db0197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.796+0000 7f3da659c640 1 -- 172.21.15.16:0/3688648915 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3db019ce30 con 0x7f3db01030a0 2024-03-28T22:12:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.796+0000 7f3da659c640 1 --2- 172.21.15.16:0/3688648915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3db01030a0 0x7f3db01983d0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f3d9800e9f0 tx=0x7f3d9800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:34.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.796+0000 7f3dacd99640 1 -- 172.21.15.16:0/3688648915 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d9800cd80 con 0x7f3db01030a0 2024-03-28T22:12:34.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.797+0000 7f3db4801640 1 -- 172.21.15.16:0/3688648915 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3db019d120 con 0x7f3db01030a0 2024-03-28T22:12:34.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.797+0000 7f3dacd99640 1 -- 172.21.15.16:0/3688648915 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d98005500 con 0x7f3db01030a0 2024-03-28T22:12:34.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.797+0000 7f3dacd99640 1 -- 172.21.15.16:0/3688648915 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d98010690 con 0x7f3db01030a0 2024-03-28T22:12:34.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.797+0000 7f3db4801640 1 -- 172.21.15.16:0/3688648915 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3db01a49b0 con 0x7f3db01030a0 2024-03-28T22:12:34.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.799+0000 7f3dacd99640 1 -- 172.21.15.16:0/3688648915 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f3d980046c0 con 0x7f3db01030a0 2024-03-28T22:12:34.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.799+0000 7f3dacd99640 1 --2- 172.21.15.16:0/3688648915 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f3d9c0a1ef0 0x7f3d9c0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.799+0000 7f3db7be8740 1 Processor -- start 2024-03-28T22:12:34.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.799+0000 7f3db7be8740 1 -- start start 2024-03-28T22:12:34.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.799+0000 7f3daed9d640 1 --2- 172.21.15.16:0/3688648915 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f3d9c0a1ef0 0x7f3d9c0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.800+0000 7f3dacd99640 1 -- 172.21.15.16:0/3688648915 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f3d980ce570 con 0x7f3db01030a0 2024-03-28T22:12:34.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.800+0000 7f3db7be8740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dbc0371680 0x55dbc0371a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.800+0000 7f3db7be8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbc0371fa0 0x55dbc037a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.800+0000 7f3db7be8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dbc037ab80 0x55dbc037cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.800+0000 7f3db7be8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55dbc02d51b0 con 0x55dbc0371fa0 2024-03-28T22:12:34.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.800+0000 7f3db7be8740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55dbc027dc00 con 0x55dbc0371680 2024-03-28T22:12:34.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.800+0000 7f3db7be8740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55dbc02d1250 con 0x55dbc037ab80 2024-03-28T22:12:34.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.800+0000 7f3da659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbc0371fa0 0x55dbc037a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.800+0000 7f3daed9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dbc0371680 0x55dbc0371a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.800+0000 7f3da659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbc0371fa0 0x55dbc037a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57126/0 (socket says 172.21.15.16:57126) 2024-03-28T22:12:34.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.800+0000 7f3da659c640 1 -- 172.21.15.16:0/1114459908 learned_addr learned my addr 172.21.15.16:0/1114459908 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:34.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.800+0000 7f3daf59e640 1 --2- 172.21.15.16:0/1114459908 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dbc037ab80 0x55dbc037cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.800+0000 7f3daed9d640 1 --2- 172.21.15.16:0/3688648915 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f3d9c0a1ef0 0x7f3d9c0a43b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3da002efd0 tx=0x7f3da002cf90 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:34.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.800+0000 7f3daed9d640 1 -- 172.21.15.16:0/1114459908 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dbc037ab80 msgr2=0x55dbc037cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:34.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.800+0000 7f3daed9d640 1 --2- 172.21.15.16:0/1114459908 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dbc037ab80 0x55dbc037cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.800+0000 7f3daed9d640 1 -- 172.21.15.16:0/1114459908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbc0371fa0 msgr2=0x55dbc037a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:34.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.800+0000 7f3daed9d640 1 --2- 172.21.15.16:0/1114459908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbc0371fa0 0x55dbc037a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.800+0000 7f3daed9d640 1 -- 172.21.15.16:0/1114459908 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbc029d3d0 con 0x55dbc0371680 2024-03-28T22:12:34.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.800+0000 7f3da659c640 1 --2- 172.21.15.16:0/1114459908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbc0371fa0 0x55dbc037a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:12:34.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.801+0000 7f3daed9d640 1 --2- 172.21.15.16:0/1114459908 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dbc0371680 0x55dbc0371a60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3da0000c00 tx=0x7f3da0044000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49365cee7b463a23 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:34.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.801+0000 7f3da559a640 1 -- 172.21.15.16:0/1114459908 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da0004300 con 0x55dbc0371680 2024-03-28T22:12:34.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.801+0000 7f3da559a640 1 -- 172.21.15.16:0/1114459908 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3da0044070 con 0x55dbc0371680 2024-03-28T22:12:34.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.801+0000 7f3da559a640 1 -- 172.21.15.16:0/1114459908 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da00055e0 con 0x55dbc0371680 2024-03-28T22:12:34.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.802+0000 7f3db7be8740 1 -- 172.21.15.16:0/1114459908 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dbc0371680 msgr2=0x55dbc0371a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:34.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.802+0000 7f3db7be8740 1 --2- 172.21.15.16:0/1114459908 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dbc0371680 0x55dbc0371a60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3da0000c00 tx=0x7f3da0044000 comp rx=0 tx=0).stop 2024-03-28T22:12:34.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.802+0000 7f3db7be8740 1 -- 172.21.15.16:0/1114459908 shutdown_connections 2024-03-28T22:12:34.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.802+0000 7f3db7be8740 1 --2- 172.21.15.16:0/1114459908 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dbc037ab80 0x55dbc037cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.802+0000 7f3db7be8740 1 --2- 172.21.15.16:0/1114459908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbc0371fa0 0x55dbc037a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.802+0000 7f3db7be8740 1 --2- 172.21.15.16:0/1114459908 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dbc0371680 0x55dbc0371a60 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.802+0000 7f3db7be8740 1 -- 172.21.15.16:0/1114459908 >> 172.21.15.16:0/1114459908 conn(0x55dbc036bdb0 msgr2=0x55dbc036c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:34.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.802+0000 7f3db7be8740 1 -- 172.21.15.16:0/1114459908 shutdown_connections 2024-03-28T22:12:34.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.802+0000 7f3db7be8740 1 -- 172.21.15.16:0/1114459908 wait complete. 2024-03-28T22:12:34.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.802+0000 7f3db7be8740 1 -- start start 2024-03-28T22:12:34.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.802+0000 7f3db7be8740 1 Processor -- start 2024-03-28T22:12:34.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.802+0000 7f3db7be8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbc0371680 0x55dbc0377bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.802+0000 7f3db7be8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dbc0371fa0 0x55dbc0378100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.802+0000 7f3db7be8740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dbc037ab80 0x55dbc0378640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.802+0000 7f3db7be8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55dbc037d910 con 0x55dbc0371680 2024-03-28T22:12:34.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.803+0000 7f3db7be8740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55dbc0273470 con 0x55dbc037ab80 2024-03-28T22:12:34.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.803+0000 7f3db7be8740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55dbc037da90 con 0x55dbc0371fa0 2024-03-28T22:12:34.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.803+0000 7f3daed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbc0371680 0x55dbc0377bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.803+0000 7f3db7be8740 10 client.0 _ll_register_callbacks cb 0x55dbc0290cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:12:34.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.803+0000 7f3daed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbc0371680 0x55dbc0377bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57140/0 (socket says 172.21.15.16:57140) 2024-03-28T22:12:34.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.803+0000 7f3daf59e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dbc037ab80 0x55dbc0378640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.803+0000 7f3da659c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dbc0371fa0 0x55dbc0378100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.803+0000 7f3daed9d640 1 -- 172.21.15.16:0/2003685884 learned_addr learned my addr 172.21.15.16:0/2003685884 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:34.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.803+0000 7f3daed9d640 1 -- 172.21.15.16:0/2003685884 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dbc0371fa0 msgr2=0x55dbc0378100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:34.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.803+0000 7f3daed9d640 1 --2- 172.21.15.16:0/2003685884 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dbc0371fa0 0x55dbc0378100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.803+0000 7f3daed9d640 1 -- 172.21.15.16:0/2003685884 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dbc037ab80 msgr2=0x55dbc0378640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:34.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.803+0000 7f3daed9d640 1 --2- 172.21.15.16:0/2003685884 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dbc037ab80 0x55dbc0378640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.803+0000 7f3daed9d640 1 -- 172.21.15.16:0/2003685884 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbbffe93f0 con 0x55dbc0371680 2024-03-28T22:12:34.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.803+0000 7f3daed9d640 1 --2- 172.21.15.16:0/2003685884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbc0371680 0x55dbc0377bc0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f3da0009900 tx=0x7f3da002bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad0ab284fe746822 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:34.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.803+0000 7f3d967fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:12:34.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.803+0000 7f3d967fc640 1 -- 172.21.15.16:0/2003685884 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da0004310 con 0x55dbc0371680 2024-03-28T22:12:34.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.803+0000 7f3d967fc640 1 -- 172.21.15.16:0/2003685884 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f3da00046c0 con 0x55dbc0371680 2024-03-28T22:12:34.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.803+0000 7f3db7be8740 1 -- 172.21.15.16:0/2003685884 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3da0044070 con 0x55dbc0371680 2024-03-28T22:12:34.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.803+0000 7f3d967fc640 1 -- 172.21.15.16:0/2003685884 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3da00044b0 con 0x55dbc0371680 2024-03-28T22:12:34.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.803+0000 7f3db7be8740 10 client.15900 fetch_fsmap learned FSMap version 24 2024-03-28T22:12:34.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.803+0000 7f3db7be8740 1 -- 172.21.15.16:0/2003685884 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55dbc00b37d0 con 0x55dbc0371680 2024-03-28T22:12:34.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.804+0000 7f3d967fc640 1 -- 172.21.15.16:0/2003685884 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3da0044070 con 0x55dbc0371680 2024-03-28T22:12:34.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.804+0000 7f3d967fc640 1 -- 172.21.15.16:0/2003685884 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7f3da0063070 con 0x55dbc0371680 2024-03-28T22:12:34.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.804+0000 7f3db7be8740 10 client.15900 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:12:34.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.804+0000 7f3db7be8740 10 client.15900 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-28T22:12:34.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.804+0000 7f3db7be8740 20 client.15900 populate_metadata read hostname 'smithi016' 2024-03-28T22:12:34.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.804+0000 7f3db7be8740 1 --2- 172.21.15.16:0/2003685884 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55dbc038a9c0 0x55dbc038cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.804+0000 7f3db7be8740 4 client.15900 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:12:34.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.804+0000 7f3db7be8740 1 -- 172.21.15.16:0/2003685884 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dbc02e26f0 con 0x55dbc038a9c0 2024-03-28T22:12:34.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.804+0000 7f3da659c640 1 --2- 172.21.15.16:0/2003685884 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55dbc038a9c0 0x55dbc038cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.804+0000 7f3d967fc640 10 client.15900 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:12:34.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.804+0000 7f3da659c640 1 --2- 172.21.15.16:0/2003685884 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55dbc038a9c0 0x55dbc038cdb0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9d9c52b88d8701b1 server_cookie=24c69bf5c3385e7c in_seq=0 out_seq=0 2024-03-28T22:12:34.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.807+0000 7f3d967fc640 1 -- 172.21.15.16:0/2003685884 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dbc02e26f0 con 0x55dbc038a9c0 2024-03-28T22:12:34.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.807+0000 7f3d967fc640 10 client.15900 handle_command_reply: tid=0 2024-03-28T22:12:34.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.807+0000 7f3db7be8740 1 client.15900 shutdown 2024-03-28T22:12:34.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.807+0000 7f3db7be8740 1 -- 172.21.15.16:0/2003685884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbc0371680 msgr2=0x55dbc0377bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:34.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.807+0000 7f3db7be8740 1 --2- 172.21.15.16:0/2003685884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbc0371680 0x55dbc0377bc0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f3da0009900 tx=0x7f3da002bcd0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.807+0000 7f3db7be8740 1 -- 172.21.15.16:0/2003685884 shutdown_connections 2024-03-28T22:12:34.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.807+0000 7f3db7be8740 1 --2- 172.21.15.16:0/2003685884 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55dbc038a9c0 0x55dbc038cdb0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.807+0000 7f3db7be8740 1 --2- 172.21.15.16:0/2003685884 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dbc037ab80 0x55dbc0378640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.807+0000 7f3d967fc640 0 client.15900 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:12:34.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.807+0000 7f3db7be8740 1 --2- 172.21.15.16:0/2003685884 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dbc0371fa0 0x55dbc0378100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.807+0000 7f3db7be8740 1 --2- 172.21.15.16:0/2003685884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbc0371680 0x55dbc0377bc0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.807+0000 7f3db7be8740 1 -- 172.21.15.16:0/2003685884 >> 172.21.15.16:0/2003685884 conn(0x55dbc036bdb0 msgr2=0x55dbc036db00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:34.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.807+0000 7f3db7be8740 1 -- 172.21.15.16:0/2003685884 shutdown_connections 2024-03-28T22:12:34.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.807+0000 7f3db7be8740 1 -- 172.21.15.16:0/2003685884 wait complete. 2024-03-28T22:12:34.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.807+0000 7f3db7be8740 20 client.15900 trim_cache size 0 max 16384 2024-03-28T22:12:34.819 INFO:tasks.workunit.client.0.smithi016.stdout:checking if gcc supports -c -o file.o... yes 2024-03-28T22:12:34.819 INFO:tasks.workunit.client.0.smithi016.stdout:checking if gcc supports -c -o file.o... (cached) yes 2024-03-28T22:12:34.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.820+0000 7f3db7be8740 1 Processor -- start 2024-03-28T22:12:34.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.820+0000 7f3db7be8740 1 -- start start 2024-03-28T22:12:34.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.821+0000 7f3db7be8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dbc0371680 0x55dbc037a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.821+0000 7f3db7be8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbc0371fa0 0x55dbc03866b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.821+0000 7f3db7be8740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dbc037ab80 0x55dbc036ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.821+0000 7f3db7be8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3db01ad930 con 0x55dbc0371fa0 2024-03-28T22:12:34.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.821+0000 7f3db7be8740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3db01ab1d0 con 0x55dbc037ab80 2024-03-28T22:12:34.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.821+0000 7f3db7be8740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3db01a9040 con 0x55dbc0371680 2024-03-28T22:12:34.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.821+0000 7f3da659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbc0371fa0 0x55dbc03866b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.821+0000 7f3daf59e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dbc037ab80 0x55dbc036ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.821+0000 7f3daf59e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dbc037ab80 0x55dbc036ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:48248/0 (socket says 172.21.15.16:48248) 2024-03-28T22:12:34.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.821+0000 7f3da659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbc0371fa0 0x55dbc03866b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57144/0 (socket says 172.21.15.16:57144) 2024-03-28T22:12:34.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.821+0000 7f3daf59e640 1 -- 172.21.15.16:0/3009498091 learned_addr learned my addr 172.21.15.16:0/3009498091 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:34.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.821+0000 7f3daed9d640 1 --2- 172.21.15.16:0/3009498091 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dbc0371680 0x55dbc037a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.821+0000 7f3daf59e640 1 -- 172.21.15.16:0/3009498091 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dbc0371680 msgr2=0x55dbc037a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:34.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.821+0000 7f3daf59e640 1 --2- 172.21.15.16:0/3009498091 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dbc0371680 0x55dbc037a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.821+0000 7f3daf59e640 1 -- 172.21.15.16:0/3009498091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbc0371fa0 msgr2=0x55dbc03866b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:34.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.821+0000 7f3daf59e640 1 --2- 172.21.15.16:0/3009498091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbc0371fa0 0x55dbc03866b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.821+0000 7f3daf59e640 1 -- 172.21.15.16:0/3009498091 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbc038d120 con 0x55dbc037ab80 2024-03-28T22:12:34.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.821+0000 7f3da659c640 1 --2- 172.21.15.16:0/3009498091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbc0371fa0 0x55dbc03866b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:12:34.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.822+0000 7f3daf59e640 1 --2- 172.21.15.16:0/3009498091 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dbc037ab80 0x55dbc036ae10 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3da8013900 tx=0x7f3da8009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe9b85bb2e66eec server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:34.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.822+0000 7f3da4d99640 1 -- 172.21.15.16:0/3009498091 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da8022070 con 0x55dbc037ab80 2024-03-28T22:12:34.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.822+0000 7f3da4d99640 1 -- 172.21.15.16:0/3009498091 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3da800edf0 con 0x55dbc037ab80 2024-03-28T22:12:34.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.822+0000 7f3da4d99640 1 -- 172.21.15.16:0/3009498091 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da8005710 con 0x55dbc037ab80 2024-03-28T22:12:34.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.822+0000 7f3db7be8740 1 -- 172.21.15.16:0/3009498091 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dbc037ab80 msgr2=0x55dbc036ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:34.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.822+0000 7f3db7be8740 1 --2- 172.21.15.16:0/3009498091 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dbc037ab80 0x55dbc036ae10 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3da8013900 tx=0x7f3da8009500 comp rx=0 tx=0).stop 2024-03-28T22:12:34.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.823+0000 7f3db7be8740 1 -- 172.21.15.16:0/3009498091 shutdown_connections 2024-03-28T22:12:34.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.823+0000 7f3db7be8740 1 --2- 172.21.15.16:0/3009498091 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dbc037ab80 0x55dbc036ae10 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.823+0000 7f3db7be8740 1 --2- 172.21.15.16:0/3009498091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbc0371fa0 0x55dbc03866b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.823+0000 7f3db7be8740 1 --2- 172.21.15.16:0/3009498091 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dbc0371680 0x55dbc037a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.823+0000 7f3db7be8740 1 -- 172.21.15.16:0/3009498091 >> 172.21.15.16:0/3009498091 conn(0x55dbc036bdb0 msgr2=0x55dbc036a700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:34.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.823+0000 7f3db7be8740 1 -- 172.21.15.16:0/3009498091 shutdown_connections 2024-03-28T22:12:34.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.823+0000 7f3db7be8740 1 -- 172.21.15.16:0/3009498091 wait complete. 2024-03-28T22:12:34.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.823+0000 7f3db7be8740 1 -- start start 2024-03-28T22:12:34.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.823+0000 7f3db7be8740 1 Processor -- start 2024-03-28T22:12:34.829 INFO:tasks.workunit.client.0.smithi016.stdout:checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes 2024-03-28T22:12:34.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.823+0000 7f3db7be8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbc0371680 0x55dbc0382f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.823+0000 7f3db7be8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dbc0371fa0 0x55dbc03743f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.823+0000 7f3db7be8740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dbc037ab80 0x55dbc0374930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.823+0000 7f3db7be8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3db01a9690 con 0x55dbc0371680 2024-03-28T22:12:34.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.823+0000 7f3db7be8740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3db01aab00 con 0x55dbc037ab80 2024-03-28T22:12:34.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.823+0000 7f3db7be8740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3db01a9d60 con 0x55dbc0371fa0 2024-03-28T22:12:34.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.823+0000 7f3db7be8740 10 client.0 _ll_register_callbacks cb 0x7f3d98005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:12:34.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.823+0000 7f3daed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbc0371680 0x55dbc0382f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.823+0000 7f3daed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbc0371680 0x55dbc0382f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57158/0 (socket says 172.21.15.16:57158) 2024-03-28T22:12:34.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.823+0000 7f3daed9d640 1 -- 172.21.15.16:0/1348895822 learned_addr learned my addr 172.21.15.16:0/1348895822 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:34.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.823+0000 7f3daf59e640 1 --2- 172.21.15.16:0/1348895822 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dbc037ab80 0x55dbc0374930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.824+0000 7f3da659c640 1 --2- 172.21.15.16:0/1348895822 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dbc0371fa0 0x55dbc03743f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.824+0000 7f3daed9d640 1 -- 172.21.15.16:0/1348895822 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dbc0371fa0 msgr2=0x55dbc03743f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:34.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.824+0000 7f3daed9d640 1 --2- 172.21.15.16:0/1348895822 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dbc0371fa0 0x55dbc03743f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.824+0000 7f3daed9d640 1 -- 172.21.15.16:0/1348895822 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dbc037ab80 msgr2=0x55dbc0374930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:34.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.824+0000 7f3daed9d640 1 --2- 172.21.15.16:0/1348895822 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dbc037ab80 0x55dbc0374930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.824+0000 7f3daed9d640 1 -- 172.21.15.16:0/1348895822 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbc0375160 con 0x55dbc0371680 2024-03-28T22:12:34.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.824+0000 7f3da659c640 1 --2- 172.21.15.16:0/1348895822 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dbc0371fa0 0x55dbc03743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:12:34.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.824+0000 7f3daed9d640 1 --2- 172.21.15.16:0/1348895822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbc0371680 0x55dbc0382f40 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f3da0003eb0 tx=0x7f3da0005950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dca2db721d627968 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:34.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.824+0000 7f3d967fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:12:34.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.824+0000 7f3d967fc640 1 -- 172.21.15.16:0/1348895822 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da0069020 con 0x55dbc0371680 2024-03-28T22:12:34.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.824+0000 7f3d967fc640 1 -- 172.21.15.16:0/1348895822 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f3da0034a80 con 0x55dbc0371680 2024-03-28T22:12:34.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.824+0000 7f3db7be8740 1 -- 172.21.15.16:0/1348895822 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3da800edf0 con 0x55dbc0371680 2024-03-28T22:12:34.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.824+0000 7f3d967fc640 1 -- 172.21.15.16:0/1348895822 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3da0034c80 con 0x55dbc0371680 2024-03-28T22:12:34.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.824+0000 7f3db7be8740 10 client.15906 fetch_fsmap learned FSMap version 24 2024-03-28T22:12:34.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.824+0000 7f3db7be8740 1 -- 172.21.15.16:0/1348895822 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55dbc03f0910 con 0x55dbc0371680 2024-03-28T22:12:34.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.824+0000 7f3d967fc640 1 -- 172.21.15.16:0/1348895822 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3da800edf0 con 0x55dbc0371680 2024-03-28T22:12:34.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.825+0000 7f3d967fc640 1 -- 172.21.15.16:0/1348895822 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7f3da0063070 con 0x55dbc0371680 2024-03-28T22:12:34.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.825+0000 7f3db7be8740 10 client.15906 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:12:34.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.825+0000 7f3db7be8740 10 client.15906 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-28T22:12:34.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.825+0000 7f3db7be8740 20 client.15906 populate_metadata read hostname 'smithi016' 2024-03-28T22:12:34.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.825+0000 7f3db7be8740 1 --2- 172.21.15.16:0/1348895822 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55dbc03f0fd0 0x55dbc03f3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.825+0000 7f3db7be8740 4 client.15906 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:12:34.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.825+0000 7f3db7be8740 1 -- 172.21.15.16:0/1348895822 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dbc02dca40 con 0x55dbc03f0fd0 2024-03-28T22:12:34.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.825+0000 7f3da659c640 1 --2- 172.21.15.16:0/1348895822 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55dbc03f0fd0 0x55dbc03f3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.825+0000 7f3d967fc640 10 client.15906 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:12:34.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.825+0000 7f3da659c640 1 --2- 172.21.15.16:0/1348895822 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55dbc03f0fd0 0x55dbc03f3410 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3182cb143dbae783 server_cookie=e6230c80758b1cd8 in_seq=0 out_seq=0 2024-03-28T22:12:34.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.826+0000 7f3d967fc640 1 -- 172.21.15.16:0/1348895822 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dbc02dca40 con 0x55dbc03f0fd0 2024-03-28T22:12:34.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.826+0000 7f3d967fc640 10 client.15906 handle_command_reply: tid=0 2024-03-28T22:12:34.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.826+0000 7f3db7be8740 1 client.15906 shutdown 2024-03-28T22:12:34.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.826+0000 7f3db7be8740 1 -- 172.21.15.16:0/1348895822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbc0371680 msgr2=0x55dbc0382f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:34.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.826+0000 7f3db7be8740 1 --2- 172.21.15.16:0/1348895822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbc0371680 0x55dbc0382f40 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f3da0003eb0 tx=0x7f3da0005950 comp rx=0 tx=0).stop 2024-03-28T22:12:34.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.826+0000 7f3db7be8740 1 -- 172.21.15.16:0/1348895822 shutdown_connections 2024-03-28T22:12:34.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.826+0000 7f3db7be8740 1 --2- 172.21.15.16:0/1348895822 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55dbc03f0fd0 0x55dbc03f3410 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.826+0000 7f3db7be8740 1 --2- 172.21.15.16:0/1348895822 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55dbc037ab80 0x55dbc0374930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.826+0000 7f3d967fc640 0 client.15906 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:12:34.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.826+0000 7f3db7be8740 1 --2- 172.21.15.16:0/1348895822 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dbc0371fa0 0x55dbc03743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.826+0000 7f3db7be8740 1 --2- 172.21.15.16:0/1348895822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dbc0371680 0x55dbc0382f40 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.826+0000 7f3db7be8740 1 -- 172.21.15.16:0/1348895822 >> 172.21.15.16:0/1348895822 conn(0x55dbc036bdb0 msgr2=0x55dbc036af90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:34.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.826+0000 7f3db7be8740 1 -- 172.21.15.16:0/1348895822 shutdown_connections 2024-03-28T22:12:34.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.826+0000 7f3db7be8740 1 -- 172.21.15.16:0/1348895822 wait complete. 2024-03-28T22:12:34.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.826+0000 7f3db7be8740 20 client.15906 trim_cache size 0 max 16384 2024-03-28T22:12:34.837 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:12:34.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.826+0000 7f3db4801640 1 -- 172.21.15.16:0/3688648915 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f3d9c0a1ef0 msgr2=0x7f3d9c0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:34.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.826+0000 7f3db4801640 1 --2- 172.21.15.16:0/3688648915 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f3d9c0a1ef0 0x7f3d9c0a43b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3da002efd0 tx=0x7f3da002cf90 comp rx=0 tx=0).stop 2024-03-28T22:12:34.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.826+0000 7f3db4801640 1 -- 172.21.15.16:0/3688648915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3db01030a0 msgr2=0x7f3db01983d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:34.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.826+0000 7f3db4801640 1 --2- 172.21.15.16:0/3688648915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3db01030a0 0x7f3db01983d0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f3d9800e9f0 tx=0x7f3d9800edf0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.826+0000 7f3db4801640 1 -- 172.21.15.16:0/3688648915 shutdown_connections 2024-03-28T22:12:34.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.826+0000 7f3db4801640 1 --2- 172.21.15.16:0/3688648915 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f3d9c0a1ef0 0x7f3d9c0a43b0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.826+0000 7f3db4801640 1 --2- 172.21.15.16:0/3688648915 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3db010bb50 0x7f3db019c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.826+0000 7f3db4801640 1 --2- 172.21.15.16:0/3688648915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3db01030a0 0x7f3db01983d0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.826+0000 7f3db4801640 1 --2- 172.21.15.16:0/3688648915 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3db0102780 0x7f3db0197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.826+0000 7f3db4801640 1 -- 172.21.15.16:0/3688648915 >> 172.21.15.16:0/3688648915 conn(0x7f3db00771e0 msgr2=0x7f3db00fc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:34.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.827+0000 7f3db4801640 1 -- 172.21.15.16:0/3688648915 shutdown_connections 2024-03-28T22:12:34.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.827+0000 7f3db4801640 1 -- 172.21.15.16:0/3688648915 wait complete. 2024-03-28T22:12:34.841 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:12:34.879 INFO:tasks.workunit.client.0.smithi016.stdout:checking whether -lc should be explicitly linked in... no 2024-03-28T22:12:34.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.981+0000 7f0379c72640 1 -- 172.21.15.16:0/457653234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03741009d0 msgr2=0x7f0374100db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:34.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.981+0000 7f0379c72640 1 --2- 172.21.15.16:0/457653234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03741009d0 0x7f0374100db0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f0368009900 tx=0x7f036802bd10 comp rx=0 tx=0).stop 2024-03-28T22:12:34.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.981+0000 7f0379c72640 1 -- 172.21.15.16:0/457653234 shutdown_connections 2024-03-28T22:12:34.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.981+0000 7f0379c72640 1 --2- 172.21.15.16:0/457653234 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0374102b80 0x7f037410d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.981+0000 7f0379c72640 1 --2- 172.21.15.16:0/457653234 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f03741021e0 0x7f0374102640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.981+0000 7f0379c72640 1 --2- 172.21.15.16:0/457653234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03741009d0 0x7f0374100db0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.981+0000 7f0379c72640 1 -- 172.21.15.16:0/457653234 >> 172.21.15.16:0/457653234 conn(0x7f0374077830 msgr2=0x7f0374077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:34.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.981+0000 7f0379c72640 1 -- 172.21.15.16:0/457653234 shutdown_connections 2024-03-28T22:12:34.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.981+0000 7f0379c72640 1 -- 172.21.15.16:0/457653234 wait complete. 2024-03-28T22:12:34.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.981+0000 7f0379c72640 1 Processor -- start 2024-03-28T22:12:34.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.982+0000 7f0379c72640 1 -- start start 2024-03-28T22:12:34.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.982+0000 7f0379c72640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f03741009d0 0x7f0374198ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.982+0000 7f0379c72640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03741021e0 0x7f03741993e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.982+0000 7f0379c72640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0374102b80 0x7f03741a3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.982+0000 7f0379c72640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0374110b20 con 0x7f03741021e0 2024-03-28T22:12:34.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.982+0000 7f0379c72640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f03741109a0 con 0x7f0374102b80 2024-03-28T22:12:34.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.982+0000 7f0379c72640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0374110ca0 con 0x7f03741009d0 2024-03-28T22:12:34.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.982+0000 7f0372ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03741021e0 0x7f03741993e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.982+0000 7f0372ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03741021e0 0x7f03741993e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57178/0 (socket says 172.21.15.16:57178) 2024-03-28T22:12:34.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.982+0000 7f0372ffd640 1 -- 172.21.15.16:0/1710550196 learned_addr learned my addr 172.21.15.16:0/1710550196 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:34.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.982+0000 7f0373fff640 1 --2- 172.21.15.16:0/1710550196 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0374102b80 0x7f03741a3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.982+0000 7f03737fe640 1 --2- 172.21.15.16:0/1710550196 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f03741009d0 0x7f0374198ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.982+0000 7f0372ffd640 1 -- 172.21.15.16:0/1710550196 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f03741009d0 msgr2=0x7f0374198ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:34.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.982+0000 7f0372ffd640 1 --2- 172.21.15.16:0/1710550196 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f03741009d0 0x7f0374198ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.982+0000 7f0372ffd640 1 -- 172.21.15.16:0/1710550196 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0374102b80 msgr2=0x7f03741a3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:34.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.982+0000 7f0372ffd640 1 --2- 172.21.15.16:0/1710550196 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0374102b80 0x7f03741a3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.982+0000 7f0372ffd640 1 -- 172.21.15.16:0/1710550196 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03741a4460 con 0x7f03741021e0 2024-03-28T22:12:34.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.982+0000 7f03737fe640 1 --2- 172.21.15.16:0/1710550196 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f03741009d0 0x7f0374198ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:12:34.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.982+0000 7f0372ffd640 1 --2- 172.21.15.16:0/1710550196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03741021e0 0x7f03741993e0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f036400d770 tx=0x7f036400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:34.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.983+0000 7f0370ff9640 1 -- 172.21.15.16:0/1710550196 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0364017070 con 0x7f03741021e0 2024-03-28T22:12:34.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.983+0000 7f0370ff9640 1 -- 172.21.15.16:0/1710550196 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0364005500 con 0x7f03741021e0 2024-03-28T22:12:34.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.983+0000 7f0379c72640 1 -- 172.21.15.16:0/1710550196 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03741a4630 con 0x7f03741021e0 2024-03-28T22:12:34.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.983+0000 7f0370ff9640 1 -- 172.21.15.16:0/1710550196 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0364004e00 con 0x7f03741021e0 2024-03-28T22:12:34.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.983+0000 7f0379c72640 1 -- 172.21.15.16:0/1710550196 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03741a4b30 con 0x7f03741021e0 2024-03-28T22:12:34.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.985+0000 7f0370ff9640 1 -- 172.21.15.16:0/1710550196 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f0364004fa0 con 0x7f03741021e0 2024-03-28T22:12:34.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.985+0000 7f037c850740 1 Processor -- start 2024-03-28T22:12:34.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.985+0000 7f037c850740 1 -- start start 2024-03-28T22:12:34.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.985+0000 7f0370ff9640 1 --2- 172.21.15.16:0/1710550196 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f035c0a1ef0 0x7f035c0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.986+0000 7f0370ff9640 1 -- 172.21.15.16:0/1710550196 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f03640cdbd0 con 0x7f03741021e0 2024-03-28T22:12:34.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.986+0000 7f0373fff640 1 --2- 172.21.15.16:0/1710550196 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f035c0a1ef0 0x7f035c0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.986+0000 7f037c850740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55edaab3b470 0x55edaab3b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.986+0000 7f037c850740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55edaab3bd90 0x55edaab44430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.986+0000 7f037c850740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55edaab44970 0x55edaab46d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.986+0000 7f037c850740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55edaaa9f1b0 con 0x55edaab3bd90 2024-03-28T22:12:34.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.986+0000 7f037c850740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55edaaa47c00 con 0x55edaab3b470 2024-03-28T22:12:34.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.986+0000 7f037c850740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55edaaa9b250 con 0x55edaab44970 2024-03-28T22:12:34.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.986+0000 7f0372ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55edaab3bd90 0x55edaab44430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.986+0000 7f0372ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55edaab3bd90 0x55edaab44430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57182/0 (socket says 172.21.15.16:57182) 2024-03-28T22:12:34.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.986+0000 7f0372ffd640 1 -- 172.21.15.16:0/441237853 learned_addr learned my addr 172.21.15.16:0/441237853 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:34.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.986+0000 7f03737fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55edaab3b470 0x55edaab3b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.986+0000 7f0373fff640 1 --2- 172.21.15.16:0/441237853 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55edaab44970 0x55edaab46d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.986+0000 7f0372ffd640 1 -- 172.21.15.16:0/441237853 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55edaab44970 msgr2=0x55edaab46d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:34.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.986+0000 7f0372ffd640 1 --2- 172.21.15.16:0/441237853 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55edaab44970 0x55edaab46d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.986+0000 7f0372ffd640 1 -- 172.21.15.16:0/441237853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55edaab3b470 msgr2=0x55edaab3b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:34.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.986+0000 7f0372ffd640 1 --2- 172.21.15.16:0/441237853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55edaab3b470 0x55edaab3b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.986+0000 7f0372ffd640 1 -- 172.21.15.16:0/441237853 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55edaaa673d0 con 0x55edaab3bd90 2024-03-28T22:12:34.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.986+0000 7f0373fff640 1 --2- 172.21.15.16:0/441237853 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55edaab44970 0x55edaab46d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:12:34.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.986+0000 7f0373fff640 1 --2- 172.21.15.16:0/1710550196 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f035c0a1ef0 0x7f035c0a43b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f036c005fd0 tx=0x7f036c011810 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:34.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.986+0000 7f0372ffd640 1 --2- 172.21.15.16:0/441237853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55edaab3bd90 0x55edaab44430 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f036409a570 tx=0x7f0364096280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2711a75753f9149 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:34.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.986+0000 7f0359ffb640 1 -- 172.21.15.16:0/441237853 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0364020050 con 0x55edaab3bd90 2024-03-28T22:12:34.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.986+0000 7f0359ffb640 1 -- 172.21.15.16:0/441237853 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f036409e730 con 0x55edaab3bd90 2024-03-28T22:12:34.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.986+0000 7f0359ffb640 1 -- 172.21.15.16:0/441237853 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03640cb5c0 con 0x55edaab3bd90 2024-03-28T22:12:34.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.987+0000 7f037c850740 1 -- 172.21.15.16:0/441237853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55edaab3bd90 msgr2=0x55edaab44430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:34.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.987+0000 7f037c850740 1 --2- 172.21.15.16:0/441237853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55edaab3bd90 0x55edaab44430 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f036409a570 tx=0x7f0364096280 comp rx=0 tx=0).stop 2024-03-28T22:12:34.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.987+0000 7f037c850740 1 -- 172.21.15.16:0/441237853 shutdown_connections 2024-03-28T22:12:34.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.987+0000 7f037c850740 1 --2- 172.21.15.16:0/441237853 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55edaab44970 0x55edaab46d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.987+0000 7f037c850740 1 --2- 172.21.15.16:0/441237853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55edaab3bd90 0x55edaab44430 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.987+0000 7f037c850740 1 --2- 172.21.15.16:0/441237853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55edaab3b470 0x55edaab3b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.987+0000 7f037c850740 1 -- 172.21.15.16:0/441237853 >> 172.21.15.16:0/441237853 conn(0x55edaab35ba0 msgr2=0x55edaab35f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:34.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.987+0000 7f037c850740 1 -- 172.21.15.16:0/441237853 shutdown_connections 2024-03-28T22:12:34.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.987+0000 7f037c850740 1 -- 172.21.15.16:0/441237853 wait complete. 2024-03-28T22:12:34.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.987+0000 7f037c850740 1 -- start start 2024-03-28T22:12:34.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.988+0000 7f037c850740 1 Processor -- start 2024-03-28T22:12:34.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.988+0000 7f037c850740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55edaab3b470 0x55edaab41a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.988+0000 7f037c850740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55edaab3bd90 0x55edaab41f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.988+0000 7f037c850740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55edaab44970 0x55edaab424d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:34.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.988+0000 7f037c850740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55edaab47700 con 0x55edaab44970 2024-03-28T22:12:34.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.988+0000 7f037c850740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55edaaa3d470 con 0x55edaab3b470 2024-03-28T22:12:34.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.988+0000 7f037c850740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55edaab47880 con 0x55edaab3bd90 2024-03-28T22:12:34.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.988+0000 7f037c850740 10 client.0 _ll_register_callbacks cb 0x55edaaa5acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:12:34.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.988+0000 7f0373fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55edaab44970 0x55edaab424d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.988+0000 7f03737fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55edaab3b470 0x55edaab41a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.988+0000 7f0373fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55edaab44970 0x55edaab424d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57192/0 (socket says 172.21.15.16:57192) 2024-03-28T22:12:34.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.988+0000 7f03737fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55edaab3b470 0x55edaab41a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:48296/0 (socket says 172.21.15.16:48296) 2024-03-28T22:12:34.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.988+0000 7f0373fff640 1 -- 172.21.15.16:0/1078541876 learned_addr learned my addr 172.21.15.16:0/1078541876 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:34.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.988+0000 7f0372ffd640 1 --2- 172.21.15.16:0/1078541876 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55edaab3bd90 0x55edaab41f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:34.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.988+0000 7f0373fff640 1 -- 172.21.15.16:0/1078541876 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55edaab3bd90 msgr2=0x55edaab41f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:34.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.988+0000 7f0373fff640 1 --2- 172.21.15.16:0/1078541876 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55edaab3bd90 0x55edaab41f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.988+0000 7f0373fff640 1 -- 172.21.15.16:0/1078541876 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55edaab3b470 msgr2=0x55edaab41a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:34.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.988+0000 7f0373fff640 1 --2- 172.21.15.16:0/1078541876 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55edaab3b470 0x55edaab41a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:34.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.988+0000 7f0373fff640 1 -- 172.21.15.16:0/1078541876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55edaa7b33f0 con 0x55edaab44970 2024-03-28T22:12:34.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.988+0000 7f03737fe640 1 --2- 172.21.15.16:0/1078541876 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55edaab3b470 0x55edaab41a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:12:34.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.989+0000 7f0373fff640 1 --2- 172.21.15.16:0/1078541876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55edaab44970 0x55edaab424d0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f036c004070 tx=0x7f036c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d40fc7c15fe8ab server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:34.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.989+0000 7f0352ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:12:34.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.989+0000 7f0352ffd640 1 -- 172.21.15.16:0/1078541876 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f036c024070 con 0x55edaab44970 2024-03-28T22:12:34.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.989+0000 7f0352ffd640 1 -- 172.21.15.16:0/1078541876 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f036c00f790 con 0x55edaab44970 2024-03-28T22:12:34.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.989+0000 7f037c850740 1 -- 172.21.15.16:0/1078541876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f036409e730 con 0x55edaab44970 2024-03-28T22:12:34.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.989+0000 7f0352ffd640 1 -- 172.21.15.16:0/1078541876 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f036c030050 con 0x55edaab44970 2024-03-28T22:12:34.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.989+0000 7f037c850740 10 client.15918 fetch_fsmap learned FSMap version 24 2024-03-28T22:12:34.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.989+0000 7f037c850740 1 -- 172.21.15.16:0/1078541876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55edaa87d7d0 con 0x55edaab44970 2024-03-28T22:12:34.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.989+0000 7f0352ffd640 1 -- 172.21.15.16:0/1078541876 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f036409e730 con 0x55edaab44970 2024-03-28T22:12:35.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.989+0000 7f0352ffd640 1 -- 172.21.15.16:0/1078541876 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7f036c020ca0 con 0x55edaab44970 2024-03-28T22:12:35.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.989+0000 7f037c850740 10 client.15918 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:12:35.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.989+0000 7f037c850740 10 client.15918 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T22:12:35.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.989+0000 7f037c850740 20 client.15918 populate_metadata read hostname 'smithi016' 2024-03-28T22:12:35.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.989+0000 7f037c850740 1 --2- 172.21.15.16:0/1078541876 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55edaab54930 0x55edaab56d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:35.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.989+0000 7f037c850740 4 client.15918 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:12:35.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.989+0000 7f037c850740 1 -- 172.21.15.16:0/1078541876 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55edaaaac6f0 con 0x55edaab54930 2024-03-28T22:12:35.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.990+0000 7f03737fe640 1 --2- 172.21.15.16:0/1078541876 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55edaab54930 0x55edaab56d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:35.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.990+0000 7f0352ffd640 10 client.15918 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:12:35.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.990+0000 7f03737fe640 1 --2- 172.21.15.16:0/1078541876 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55edaab54930 0x55edaab56d80 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d25111d5c7931215 server_cookie=698cd5991404bdbd in_seq=0 out_seq=0 2024-03-28T22:12:35.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.991+0000 7f0352ffd640 1 -- 172.21.15.16:0/1078541876 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55edaaaac6f0 con 0x55edaab54930 2024-03-28T22:12:35.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.991+0000 7f0352ffd640 10 client.15918 handle_command_reply: tid=0 2024-03-28T22:12:35.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.991+0000 7f037c850740 1 client.15918 shutdown 2024-03-28T22:12:35.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.991+0000 7f037c850740 1 -- 172.21.15.16:0/1078541876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55edaab44970 msgr2=0x55edaab424d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:35.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.991+0000 7f037c850740 1 --2- 172.21.15.16:0/1078541876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55edaab44970 0x55edaab424d0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f036c004070 tx=0x7f036c004ab0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.991+0000 7f037c850740 1 -- 172.21.15.16:0/1078541876 shutdown_connections 2024-03-28T22:12:35.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.991+0000 7f037c850740 1 --2- 172.21.15.16:0/1078541876 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55edaab54930 0x55edaab56d80 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.991+0000 7f037c850740 1 --2- 172.21.15.16:0/1078541876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55edaab44970 0x55edaab424d0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.991+0000 7f0352ffd640 0 client.15918 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:12:35.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.991+0000 7f037c850740 1 --2- 172.21.15.16:0/1078541876 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55edaab3bd90 0x55edaab41f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.991+0000 7f037c850740 1 --2- 172.21.15.16:0/1078541876 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55edaab3b470 0x55edaab41a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.991+0000 7f037c850740 1 -- 172.21.15.16:0/1078541876 >> 172.21.15.16:0/1078541876 conn(0x55edaab35ba0 msgr2=0x55edaab378f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:35.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.991+0000 7f037c850740 1 -- 172.21.15.16:0/1078541876 shutdown_connections 2024-03-28T22:12:35.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.991+0000 7f037c850740 1 -- 172.21.15.16:0/1078541876 wait complete. 2024-03-28T22:12:35.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:34.991+0000 7f037c850740 20 client.15918 trim_cache size 0 max 16384 2024-03-28T22:12:35.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.005+0000 7f037c850740 1 Processor -- start 2024-03-28T22:12:35.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.005+0000 7f037c850740 1 -- start start 2024-03-28T22:12:35.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.005+0000 7f037c850740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55edaab3b470 0x55edaab44180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:35.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.005+0000 7f037c850740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55edaab3bd90 0x55edaab504a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:35.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.005+0000 7f037c850740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55edaab44970 0x55edaab5c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:35.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.005+0000 7f037c850740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f03741afc20 con 0x55edaab3b470 2024-03-28T22:12:35.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.005+0000 7f037c850740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f03741ad4c0 con 0x55edaab3bd90 2024-03-28T22:12:35.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.005+0000 7f037c850740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f03741ab330 con 0x55edaab44970 2024-03-28T22:12:35.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.005+0000 7f03737fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55edaab3b470 0x55edaab44180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:35.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.005+0000 7f03737fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55edaab3b470 0x55edaab44180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57200/0 (socket says 172.21.15.16:57200) 2024-03-28T22:12:35.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.005+0000 7f03737fe640 1 -- 172.21.15.16:0/439385744 learned_addr learned my addr 172.21.15.16:0/439385744 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:35.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.005+0000 7f0372ffd640 1 --2- 172.21.15.16:0/439385744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55edaab3bd90 0x55edaab504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:35.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.005+0000 7f03737fe640 1 -- 172.21.15.16:0/439385744 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55edaab44970 msgr2=0x55edaab5c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:35.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.005+0000 7f03737fe640 1 --2- 172.21.15.16:0/439385744 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55edaab44970 0x55edaab5c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.005+0000 7f03737fe640 1 -- 172.21.15.16:0/439385744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55edaab3bd90 msgr2=0x55edaab504a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:35.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.005+0000 7f03737fe640 1 --2- 172.21.15.16:0/439385744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55edaab3bd90 0x55edaab504a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.005+0000 7f03737fe640 1 -- 172.21.15.16:0/439385744 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55edaab31460 con 0x55edaab3b470 2024-03-28T22:12:35.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.005+0000 7f03737fe640 1 --2- 172.21.15.16:0/439385744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55edaab3b470 0x55edaab44180 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f036802c250 tx=0x7f0368005e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25b97046cc0f31df server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:35.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.006+0000 7f03597fa640 1 -- 172.21.15.16:0/439385744 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f036802c930 con 0x55edaab3b470 2024-03-28T22:12:35.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.006+0000 7f03597fa640 1 -- 172.21.15.16:0/439385744 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0368005040 con 0x55edaab3b470 2024-03-28T22:12:35.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.006+0000 7f03597fa640 1 -- 172.21.15.16:0/439385744 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0368004b50 con 0x55edaab3b470 2024-03-28T22:12:35.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.006+0000 7f037c850740 1 -- 172.21.15.16:0/439385744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55edaab3b470 msgr2=0x55edaab44180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:35.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.006+0000 7f037c850740 1 --2- 172.21.15.16:0/439385744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55edaab3b470 0x55edaab44180 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f036802c250 tx=0x7f0368005e30 comp rx=0 tx=0).stop 2024-03-28T22:12:35.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.006+0000 7f037c850740 1 -- 172.21.15.16:0/439385744 shutdown_connections 2024-03-28T22:12:35.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.006+0000 7f037c850740 1 --2- 172.21.15.16:0/439385744 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55edaab44970 0x55edaab5c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.006+0000 7f037c850740 1 --2- 172.21.15.16:0/439385744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55edaab3bd90 0x55edaab504a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.006+0000 7f037c850740 1 --2- 172.21.15.16:0/439385744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55edaab3b470 0x55edaab44180 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.006+0000 7f037c850740 1 -- 172.21.15.16:0/439385744 >> 172.21.15.16:0/439385744 conn(0x55edaab35ba0 msgr2=0x55edaab37860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:35.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.006+0000 7f037c850740 1 -- 172.21.15.16:0/439385744 shutdown_connections 2024-03-28T22:12:35.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.006+0000 7f037c850740 1 -- 172.21.15.16:0/439385744 wait complete. 2024-03-28T22:12:35.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.006+0000 7f037c850740 1 -- start start 2024-03-28T22:12:35.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.007+0000 7f037c850740 1 Processor -- start 2024-03-28T22:12:35.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.007+0000 7f037c850740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55edaab3b470 0x55edaab3e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:35.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.007+0000 7f037c850740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55edaab3bd90 0x55edaab3e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:35.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.007+0000 7f037c850740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55edaab44970 0x55edaab3eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:35.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.007+0000 7f037c850740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f03741ab980 con 0x55edaab44970 2024-03-28T22:12:35.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.007+0000 7f037c850740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f03741acdf0 con 0x55edaab3b470 2024-03-28T22:12:35.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.007+0000 7f037c850740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f03741ac050 con 0x55edaab3bd90 2024-03-28T22:12:35.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.007+0000 7f037c850740 10 client.0 _ll_register_callbacks cb 0x7f036c003e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:12:35.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.007+0000 7f0373fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55edaab44970 0x55edaab3eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:35.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.007+0000 7f0373fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55edaab44970 0x55edaab3eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57216/0 (socket says 172.21.15.16:57216) 2024-03-28T22:12:35.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.007+0000 7f0373fff640 1 -- 172.21.15.16:0/2473501193 learned_addr learned my addr 172.21.15.16:0/2473501193 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:35.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.007+0000 7f03737fe640 1 --2- 172.21.15.16:0/2473501193 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55edaab3b470 0x55edaab3e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:35.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.007+0000 7f0372ffd640 1 --2- 172.21.15.16:0/2473501193 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55edaab3bd90 0x55edaab3e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:35.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.007+0000 7f0373fff640 1 -- 172.21.15.16:0/2473501193 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55edaab3bd90 msgr2=0x55edaab3e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:35.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.007+0000 7f0373fff640 1 --2- 172.21.15.16:0/2473501193 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55edaab3bd90 0x55edaab3e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.007+0000 7f0373fff640 1 -- 172.21.15.16:0/2473501193 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55edaab3b470 msgr2=0x55edaab3e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:35.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.007+0000 7f0373fff640 1 --2- 172.21.15.16:0/2473501193 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55edaab3b470 0x55edaab3e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.007+0000 7f0373fff640 1 -- 172.21.15.16:0/2473501193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55edaabb9c80 con 0x55edaab44970 2024-03-28T22:12:35.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.007+0000 7f0372ffd640 1 --2- 172.21.15.16:0/2473501193 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55edaab3bd90 0x55edaab3e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:12:35.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.007+0000 7f03737fe640 1 --2- 172.21.15.16:0/2473501193 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55edaab3b470 0x55edaab3e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:12:35.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.008+0000 7f0373fff640 1 --2- 172.21.15.16:0/2473501193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55edaab44970 0x55edaab3eef0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f036c0285f0 tx=0x7f036c009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9fb13073d542939 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:35.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.008+0000 7f0352ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:12:35.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.008+0000 7f0352ffd640 1 -- 172.21.15.16:0/2473501193 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f036c024070 con 0x55edaab44970 2024-03-28T22:12:35.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.008+0000 7f037c850740 1 -- 172.21.15.16:0/2473501193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0368005040 con 0x55edaab44970 2024-03-28T22:12:35.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.008+0000 7f0352ffd640 1 -- 172.21.15.16:0/2473501193 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f036c028a80 con 0x55edaab44970 2024-03-28T22:12:35.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.008+0000 7f0352ffd640 1 -- 172.21.15.16:0/2473501193 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f036c017040 con 0x55edaab44970 2024-03-28T22:12:35.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.008+0000 7f037c850740 10 client.15924 fetch_fsmap learned FSMap version 24 2024-03-28T22:12:35.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.008+0000 7f037c850740 1 -- 172.21.15.16:0/2473501193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55edaabbae00 con 0x55edaab44970 2024-03-28T22:12:35.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.008+0000 7f0352ffd640 1 -- 172.21.15.16:0/2473501193 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0368005040 con 0x55edaab44970 2024-03-28T22:12:35.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.008+0000 7f0352ffd640 1 -- 172.21.15.16:0/2473501193 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7f036c043020 con 0x55edaab44970 2024-03-28T22:12:35.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.008+0000 7f037c850740 10 client.15924 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:12:35.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.008+0000 7f037c850740 10 client.15924 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T22:12:35.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.008+0000 7f037c850740 20 client.15924 populate_metadata read hostname 'smithi016' 2024-03-28T22:12:35.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.008+0000 7f037c850740 1 --2- 172.21.15.16:0/2473501193 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55edaabbb4d0 0x55edaabbd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:35.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.008+0000 7f037c850740 4 client.15924 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:12:35.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.008+0000 7f037c850740 1 -- 172.21.15.16:0/2473501193 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55edaaaa6a40 con 0x55edaabbb4d0 2024-03-28T22:12:35.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.008+0000 7f03737fe640 1 --2- 172.21.15.16:0/2473501193 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55edaabbb4d0 0x55edaabbd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:35.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.009+0000 7f0352ffd640 10 client.15924 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:12:35.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.009+0000 7f03737fe640 1 --2- 172.21.15.16:0/2473501193 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55edaabbb4d0 0x55edaabbd8f0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1e6c3096a823b182 server_cookie=a9cf20e113b55f69 in_seq=0 out_seq=0 2024-03-28T22:12:35.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.009+0000 7f0352ffd640 1 -- 172.21.15.16:0/2473501193 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55edaaaa6a40 con 0x55edaabbb4d0 2024-03-28T22:12:35.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.009+0000 7f0352ffd640 10 client.15924 handle_command_reply: tid=0 2024-03-28T22:12:35.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.009+0000 7f037c850740 1 client.15924 shutdown 2024-03-28T22:12:35.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.009+0000 7f037c850740 1 -- 172.21.15.16:0/2473501193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55edaab44970 msgr2=0x55edaab3eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:35.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.009+0000 7f037c850740 1 --2- 172.21.15.16:0/2473501193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55edaab44970 0x55edaab3eef0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f036c0285f0 tx=0x7f036c009bf0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.009+0000 7f037c850740 1 -- 172.21.15.16:0/2473501193 shutdown_connections 2024-03-28T22:12:35.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.009+0000 7f037c850740 1 --2- 172.21.15.16:0/2473501193 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55edaabbb4d0 0x55edaabbd8f0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.009+0000 7f037c850740 1 --2- 172.21.15.16:0/2473501193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55edaab44970 0x55edaab3eef0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.009+0000 7f037c850740 1 --2- 172.21.15.16:0/2473501193 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55edaab3bd90 0x55edaab3e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.009+0000 7f0352ffd640 0 client.15924 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:12:35.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.009+0000 7f037c850740 1 --2- 172.21.15.16:0/2473501193 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55edaab3b470 0x55edaab3e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.009+0000 7f037c850740 1 -- 172.21.15.16:0/2473501193 >> 172.21.15.16:0/2473501193 conn(0x55edaab35ba0 msgr2=0x55edaab344b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:35.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.009+0000 7f037c850740 1 -- 172.21.15.16:0/2473501193 shutdown_connections 2024-03-28T22:12:35.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.009+0000 7f037c850740 1 -- 172.21.15.16:0/2473501193 wait complete. 2024-03-28T22:12:35.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.009+0000 7f037c850740 20 client.15924 trim_cache size 0 max 16384 2024-03-28T22:12:35.022 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:12:35.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.010+0000 7f0379c72640 1 -- 172.21.15.16:0/1710550196 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f035c0a1ef0 msgr2=0x7f035c0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:35.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.010+0000 7f0379c72640 1 --2- 172.21.15.16:0/1710550196 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f035c0a1ef0 0x7f035c0a43b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f036c005fd0 tx=0x7f036c011810 comp rx=0 tx=0).stop 2024-03-28T22:12:35.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.010+0000 7f0379c72640 1 -- 172.21.15.16:0/1710550196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03741021e0 msgr2=0x7f03741993e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:35.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.010+0000 7f0379c72640 1 --2- 172.21.15.16:0/1710550196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03741021e0 0x7f03741993e0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f036400d770 tx=0x7f036400dc30 comp rx=0 tx=0).stop 2024-03-28T22:12:35.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.010+0000 7f0379c72640 1 -- 172.21.15.16:0/1710550196 shutdown_connections 2024-03-28T22:12:35.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.010+0000 7f0379c72640 1 --2- 172.21.15.16:0/1710550196 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f035c0a1ef0 0x7f035c0a43b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.010+0000 7f0379c72640 1 --2- 172.21.15.16:0/1710550196 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0374102b80 0x7f03741a3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.010+0000 7f0379c72640 1 --2- 172.21.15.16:0/1710550196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03741021e0 0x7f03741993e0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.010+0000 7f0379c72640 1 --2- 172.21.15.16:0/1710550196 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f03741009d0 0x7f0374198ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.010+0000 7f0379c72640 1 -- 172.21.15.16:0/1710550196 >> 172.21.15.16:0/1710550196 conn(0x7f0374077830 msgr2=0x7f03741043c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:35.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.010+0000 7f0379c72640 1 -- 172.21.15.16:0/1710550196 shutdown_connections 2024-03-28T22:12:35.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.010+0000 7f0379c72640 1 -- 172.21.15.16:0/1710550196 wait complete. 2024-03-28T22:12:35.024 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:12:35.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.126+0000 7f77ba215640 1 -- 172.21.15.16:0/2666128277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77b41076f0 msgr2=0x7f77b4107ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:35.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.126+0000 7f77ba215640 1 --2- 172.21.15.16:0/2666128277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77b41076f0 0x7f77b4107ad0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f77a4009cd0 tx=0x7f77a402c0f0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.126+0000 7f77ba215640 1 -- 172.21.15.16:0/2666128277 shutdown_connections 2024-03-28T22:12:35.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.126+0000 7f77ba215640 1 --2- 172.21.15.16:0/2666128277 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77b41020a0 0x7f77b410e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.126+0000 7f77ba215640 1 --2- 172.21.15.16:0/2666128277 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f77b41016e0 0x7f77b4101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.126+0000 7f77ba215640 1 --2- 172.21.15.16:0/2666128277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77b41076f0 0x7f77b4107ad0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.126+0000 7f77ba215640 1 -- 172.21.15.16:0/2666128277 >> 172.21.15.16:0/2666128277 conn(0x7f77b40fd1f0 msgr2=0x7f77b40ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:35.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.127+0000 7f77ba215640 1 -- 172.21.15.16:0/2666128277 shutdown_connections 2024-03-28T22:12:35.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.127+0000 7f77ba215640 1 -- 172.21.15.16:0/2666128277 wait complete. 2024-03-28T22:12:35.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.127+0000 7f77ba215640 1 Processor -- start 2024-03-28T22:12:35.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.127+0000 7f77ba215640 1 -- start start 2024-03-28T22:12:35.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.127+0000 7f77ba215640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77b41016e0 0x7f77b4198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:35.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.127+0000 7f77ba215640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f77b41020a0 0x7f77b4198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:35.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.128+0000 7f77ba215640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77b41076f0 0x7f77b419c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:35.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.128+0000 7f77ba215640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f77b4111b00 con 0x7f77b41076f0 2024-03-28T22:12:35.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.128+0000 7f77ba215640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f77b4111980 con 0x7f77b41016e0 2024-03-28T22:12:35.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.128+0000 7f77ba215640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f77b4111c80 con 0x7f77b41020a0 2024-03-28T22:12:35.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.128+0000 7f77b3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77b41076f0 0x7f77b419c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:35.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.128+0000 7f77b3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77b41076f0 0x7f77b419c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57234/0 (socket says 172.21.15.16:57234) 2024-03-28T22:12:35.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.128+0000 7f77b37fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77b41016e0 0x7f77b4198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:35.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.128+0000 7f77b3fff640 1 -- 172.21.15.16:0/3101428643 learned_addr learned my addr 172.21.15.16:0/3101428643 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:35.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.128+0000 7f77b2ffd640 1 --2- 172.21.15.16:0/3101428643 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f77b41020a0 0x7f77b4198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:35.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.128+0000 7f77b3fff640 1 -- 172.21.15.16:0/3101428643 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f77b41020a0 msgr2=0x7f77b4198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:35.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.128+0000 7f77b3fff640 1 --2- 172.21.15.16:0/3101428643 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f77b41020a0 0x7f77b4198580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.128+0000 7f77b3fff640 1 -- 172.21.15.16:0/3101428643 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77b41016e0 msgr2=0x7f77b4198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:35.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.128+0000 7f77b3fff640 1 --2- 172.21.15.16:0/3101428643 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77b41016e0 0x7f77b4198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.128+0000 7f77b3fff640 1 -- 172.21.15.16:0/3101428643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77b419d030 con 0x7f77b41076f0 2024-03-28T22:12:35.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.128+0000 7f77b37fe640 1 --2- 172.21.15.16:0/3101428643 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77b41016e0 0x7f77b4198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:12:35.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.128+0000 7f77b3fff640 1 --2- 172.21.15.16:0/3101428643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77b41076f0 0x7f77b419c900 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f77ac00e9b0 tx=0x7f77ac00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:35.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.128+0000 7f77b0ff9640 1 -- 172.21.15.16:0/3101428643 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77ac00cd80 con 0x7f77b41076f0 2024-03-28T22:12:35.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.128+0000 7f77b0ff9640 1 -- 172.21.15.16:0/3101428643 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77ac0035f0 con 0x7f77b41076f0 2024-03-28T22:12:35.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.128+0000 7f77ba215640 1 -- 172.21.15.16:0/3101428643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77b419d320 con 0x7f77b41076f0 2024-03-28T22:12:35.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.128+0000 7f77b0ff9640 1 -- 172.21.15.16:0/3101428643 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77ac010670 con 0x7f77b41076f0 2024-03-28T22:12:35.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.128+0000 7f77ba215640 1 -- 172.21.15.16:0/3101428643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77b41a4bb0 con 0x7f77b41076f0 2024-03-28T22:12:35.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.131+0000 7f77b0ff9640 1 -- 172.21.15.16:0/3101428643 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f77ac0037a0 con 0x7f77b41076f0 2024-03-28T22:12:35.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.131+0000 7f77bcdfc740 1 Processor -- start 2024-03-28T22:12:35.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.131+0000 7f77bcdfc740 1 -- start start 2024-03-28T22:12:35.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.131+0000 7f77bcdfc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6b33fe470 0x55e6b33fe850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:35.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.131+0000 7f77b0ff9640 1 --2- 172.21.15.16:0/3101428643 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f77a00a1ef0 0x7f77a00a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:35.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.131+0000 7f77bcdfc740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e6b33fed90 0x55e6b3407430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:35.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.131+0000 7f77bcdfc740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6b3407970 0x55e6b3409d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:35.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.131+0000 7f77bcdfc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e6b33621b0 con 0x55e6b33fe470 2024-03-28T22:12:35.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.131+0000 7f77bcdfc740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55e6b330ac00 con 0x55e6b33fed90 2024-03-28T22:12:35.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.131+0000 7f77bcdfc740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e6b335e250 con 0x55e6b3407970 2024-03-28T22:12:35.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.131+0000 7f77b2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6b33fe470 0x55e6b33fe850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:35.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.132+0000 7f77b37fe640 1 --2- 172.21.15.16:0/3101428643 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f77a00a1ef0 0x7f77a00a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:35.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.132+0000 7f77b2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6b33fe470 0x55e6b33fe850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57238/0 (socket says 172.21.15.16:57238) 2024-03-28T22:12:35.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.132+0000 7f77b2ffd640 1 -- 172.21.15.16:0/1229419876 learned_addr learned my addr 172.21.15.16:0/1229419876 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:35.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.132+0000 7f77b2ffd640 1 --2- 172.21.15.16:0/1229419876 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e6b33fed90 0x55e6b3407430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:35.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.132+0000 7f77b3fff640 1 --2- 172.21.15.16:0/1229419876 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6b3407970 0x55e6b3409d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:35.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.132+0000 7f77b0ff9640 1 -- 172.21.15.16:0/3101428643 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f77ac0ce4b0 con 0x7f77b41076f0 2024-03-28T22:12:35.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.132+0000 7f77b2ffd640 1 -- 172.21.15.16:0/1229419876 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6b3407970 msgr2=0x55e6b3409d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:35.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.132+0000 7f77b2ffd640 1 --2- 172.21.15.16:0/1229419876 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6b3407970 0x55e6b3409d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.132+0000 7f77b2ffd640 1 -- 172.21.15.16:0/1229419876 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e6b33fed90 msgr2=0x55e6b3407430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:35.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.132+0000 7f77b2ffd640 1 --2- 172.21.15.16:0/1229419876 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e6b33fed90 0x55e6b3407430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.132+0000 7f77b2ffd640 1 -- 172.21.15.16:0/1229419876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6b332a3d0 con 0x55e6b33fe470 2024-03-28T22:12:35.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.132+0000 7f77b37fe640 1 --2- 172.21.15.16:0/3101428643 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f77a00a1ef0 0x7f77a00a43b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f77a4009ca0 tx=0x7f77a4032340 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:35.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.132+0000 7f77b2ffd640 1 --2- 172.21.15.16:0/1229419876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6b33fe470 0x55e6b33fe850 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f779c0147f0 tx=0x7f779c0095d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c26168c188c6a44f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:35.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.132+0000 7f77a9ffb640 1 -- 172.21.15.16:0/1229419876 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f779c014230 con 0x55e6b33fe470 2024-03-28T22:12:35.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.132+0000 7f77a9ffb640 1 -- 172.21.15.16:0/1229419876 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f779c00e6f0 con 0x55e6b33fe470 2024-03-28T22:12:35.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.132+0000 7f77a9ffb640 1 -- 172.21.15.16:0/1229419876 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f779c003850 con 0x55e6b33fe470 2024-03-28T22:12:35.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.133+0000 7f77bcdfc740 1 -- 172.21.15.16:0/1229419876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6b33fe470 msgr2=0x55e6b33fe850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:35.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.133+0000 7f77bcdfc740 1 --2- 172.21.15.16:0/1229419876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6b33fe470 0x55e6b33fe850 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f779c0147f0 tx=0x7f779c0095d0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.133+0000 7f77bcdfc740 1 -- 172.21.15.16:0/1229419876 shutdown_connections 2024-03-28T22:12:35.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.133+0000 7f77bcdfc740 1 --2- 172.21.15.16:0/1229419876 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6b3407970 0x55e6b3409d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.133+0000 7f77bcdfc740 1 --2- 172.21.15.16:0/1229419876 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e6b33fed90 0x55e6b3407430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.133+0000 7f77bcdfc740 1 --2- 172.21.15.16:0/1229419876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6b33fe470 0x55e6b33fe850 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.133+0000 7f77bcdfc740 1 -- 172.21.15.16:0/1229419876 >> 172.21.15.16:0/1229419876 conn(0x55e6b33f8ba0 msgr2=0x55e6b33f8f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:35.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.133+0000 7f77bcdfc740 1 -- 172.21.15.16:0/1229419876 shutdown_connections 2024-03-28T22:12:35.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.133+0000 7f77bcdfc740 1 -- 172.21.15.16:0/1229419876 wait complete. 2024-03-28T22:12:35.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.133+0000 7f77bcdfc740 1 -- start start 2024-03-28T22:12:35.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.134+0000 7f77bcdfc740 1 Processor -- start 2024-03-28T22:12:35.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.134+0000 7f77bcdfc740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6b33fe470 0x55e6b34049d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:35.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.134+0000 7f77bcdfc740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e6b33fed90 0x55e6b3404f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:35.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.134+0000 7f77bcdfc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6b3407970 0x55e6b3405450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:35.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.134+0000 7f77bcdfc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e6b340a700 con 0x55e6b3407970 2024-03-28T22:12:35.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.134+0000 7f77bcdfc740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55e6b3300470 con 0x55e6b33fed90 2024-03-28T22:12:35.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.134+0000 7f77bcdfc740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e6b340a880 con 0x55e6b33fe470 2024-03-28T22:12:35.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.134+0000 7f77bcdfc740 10 client.0 _ll_register_callbacks cb 0x55e6b331dcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:12:35.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.134+0000 7f77b3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6b3407970 0x55e6b3405450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:35.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.134+0000 7f77b3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6b3407970 0x55e6b3405450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57252/0 (socket says 172.21.15.16:57252) 2024-03-28T22:12:35.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.134+0000 7f77b3fff640 1 -- 172.21.15.16:0/2065674350 learned_addr learned my addr 172.21.15.16:0/2065674350 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:35.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.134+0000 7f77b2ffd640 1 --2- 172.21.15.16:0/2065674350 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e6b33fed90 0x55e6b3404f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:35.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.134+0000 7f77b37fe640 1 --2- 172.21.15.16:0/2065674350 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6b33fe470 0x55e6b34049d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:35.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.134+0000 7f77b3fff640 1 -- 172.21.15.16:0/2065674350 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6b33fe470 msgr2=0x55e6b34049d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:35.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.134+0000 7f77b3fff640 1 --2- 172.21.15.16:0/2065674350 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6b33fe470 0x55e6b34049d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.134+0000 7f77b3fff640 1 -- 172.21.15.16:0/2065674350 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e6b33fed90 msgr2=0x55e6b3404f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:35.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.134+0000 7f77b3fff640 1 --2- 172.21.15.16:0/2065674350 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e6b33fed90 0x55e6b3404f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.134+0000 7f77b3fff640 1 -- 172.21.15.16:0/2065674350 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6b30763f0 con 0x55e6b3407970 2024-03-28T22:12:35.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.134+0000 7f77b2ffd640 1 --2- 172.21.15.16:0/2065674350 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e6b33fed90 0x55e6b3404f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:12:35.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.134+0000 7f77b3fff640 1 --2- 172.21.15.16:0/2065674350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6b3407970 0x55e6b3405450 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f77ac097320 tx=0x7f77ac0b3c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8a53b6ced2fc5b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:35.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.134+0000 7f7792ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:12:35.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.135+0000 7f7792ffd640 1 -- 172.21.15.16:0/2065674350 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77ac0cc0a0 con 0x55e6b3407970 2024-03-28T22:12:35.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.135+0000 7f7792ffd640 1 -- 172.21.15.16:0/2065674350 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f77ac0b9150 con 0x55e6b3407970 2024-03-28T22:12:35.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.135+0000 7f77bcdfc740 1 -- 172.21.15.16:0/2065674350 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f779c00e6f0 con 0x55e6b3407970 2024-03-28T22:12:35.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.135+0000 7f7792ffd640 1 -- 172.21.15.16:0/2065674350 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f77ac09f010 con 0x55e6b3407970 2024-03-28T22:12:35.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.135+0000 7f77bcdfc740 10 client.15936 fetch_fsmap learned FSMap version 24 2024-03-28T22:12:35.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.135+0000 7f77bcdfc740 1 -- 172.21.15.16:0/2065674350 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e6b31407d0 con 0x55e6b3407970 2024-03-28T22:12:35.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.135+0000 7f7792ffd640 1 -- 172.21.15.16:0/2065674350 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f779c00e6f0 con 0x55e6b3407970 2024-03-28T22:12:35.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.135+0000 7f7792ffd640 1 -- 172.21.15.16:0/2065674350 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7f77ac0ce030 con 0x55e6b3407970 2024-03-28T22:12:35.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.135+0000 7f77bcdfc740 10 client.15936 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:12:35.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.135+0000 7f77bcdfc740 10 client.15936 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:12:35.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.135+0000 7f77bcdfc740 20 client.15936 populate_metadata read hostname 'smithi016' 2024-03-28T22:12:35.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.135+0000 7f77bcdfc740 1 --2- 172.21.15.16:0/2065674350 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e6b3417980 0x55e6b3419da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:35.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.135+0000 7f77bcdfc740 4 client.15936 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:12:35.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.135+0000 7f77bcdfc740 1 -- 172.21.15.16:0/2065674350 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e6b336f6f0 con 0x55e6b3417980 2024-03-28T22:12:35.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.135+0000 7f77b37fe640 1 --2- 172.21.15.16:0/2065674350 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e6b3417980 0x55e6b3419da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:35.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.136+0000 7f7792ffd640 10 client.15936 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:12:35.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.136+0000 7f77b37fe640 1 --2- 172.21.15.16:0/2065674350 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e6b3417980 0x55e6b3419da0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5f97ae60404a475 server_cookie=5b1e08d23533a869 in_seq=0 out_seq=0 2024-03-28T22:12:35.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.136+0000 7f7792ffd640 1 -- 172.21.15.16:0/2065674350 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e6b336f6f0 con 0x55e6b3417980 2024-03-28T22:12:35.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.136+0000 7f7792ffd640 10 client.15936 handle_command_reply: tid=0 2024-03-28T22:12:35.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.136+0000 7f77bcdfc740 1 client.15936 shutdown 2024-03-28T22:12:35.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.137+0000 7f77bcdfc740 1 -- 172.21.15.16:0/2065674350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6b3407970 msgr2=0x55e6b3405450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:35.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.137+0000 7f77bcdfc740 1 --2- 172.21.15.16:0/2065674350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6b3407970 0x55e6b3405450 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f77ac097320 tx=0x7f77ac0b3c70 comp rx=0 tx=0).stop 2024-03-28T22:12:35.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.137+0000 7f77bcdfc740 1 -- 172.21.15.16:0/2065674350 shutdown_connections 2024-03-28T22:12:35.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.137+0000 7f77bcdfc740 1 --2- 172.21.15.16:0/2065674350 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e6b3417980 0x55e6b3419da0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.137+0000 7f77bcdfc740 1 --2- 172.21.15.16:0/2065674350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6b3407970 0x55e6b3405450 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.137+0000 7f77bcdfc740 1 --2- 172.21.15.16:0/2065674350 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e6b33fed90 0x55e6b3404f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.137+0000 7f7792ffd640 0 client.15936 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:12:35.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.137+0000 7f77bcdfc740 1 --2- 172.21.15.16:0/2065674350 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6b33fe470 0x55e6b34049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.137+0000 7f77bcdfc740 1 -- 172.21.15.16:0/2065674350 >> 172.21.15.16:0/2065674350 conn(0x55e6b33f8ba0 msgr2=0x55e6b33fa8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:35.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.137+0000 7f77bcdfc740 1 -- 172.21.15.16:0/2065674350 shutdown_connections 2024-03-28T22:12:35.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.137+0000 7f77bcdfc740 1 -- 172.21.15.16:0/2065674350 wait complete. 2024-03-28T22:12:35.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.137+0000 7f77bcdfc740 20 client.15936 trim_cache size 0 max 16384 2024-03-28T22:12:35.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.150+0000 7f77bcdfc740 1 Processor -- start 2024-03-28T22:12:35.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.150+0000 7f77bcdfc740 1 -- start start 2024-03-28T22:12:35.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.150+0000 7f77bcdfc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6b33fe470 0x55e6b3407100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:35.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.150+0000 7f77bcdfc740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e6b33fed90 0x55e6b33ff170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:35.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.151+0000 7f77bcdfc740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6b3407970 0x55e6b34134a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:35.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.151+0000 7f77bcdfc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f77b41afd50 con 0x55e6b33fe470 2024-03-28T22:12:35.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.151+0000 7f77bcdfc740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f77b41ad5f0 con 0x55e6b33fed90 2024-03-28T22:12:35.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.151+0000 7f77bcdfc740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f77b41ab440 con 0x55e6b3407970 2024-03-28T22:12:35.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.151+0000 7f77b37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6b33fe470 0x55e6b3407100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:35.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.151+0000 7f77b37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6b33fe470 0x55e6b3407100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57258/0 (socket says 172.21.15.16:57258) 2024-03-28T22:12:35.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.151+0000 7f77b37fe640 1 -- 172.21.15.16:0/1105846427 learned_addr learned my addr 172.21.15.16:0/1105846427 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:35.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.151+0000 7f77b2ffd640 1 --2- 172.21.15.16:0/1105846427 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e6b33fed90 0x55e6b33ff170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:35.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.151+0000 7f77b3fff640 1 --2- 172.21.15.16:0/1105846427 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6b3407970 0x55e6b34134a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:35.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.151+0000 7f77b37fe640 1 -- 172.21.15.16:0/1105846427 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6b3407970 msgr2=0x55e6b34134a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:35.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.151+0000 7f77b37fe640 1 --2- 172.21.15.16:0/1105846427 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6b3407970 0x55e6b34134a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.151+0000 7f77b37fe640 1 -- 172.21.15.16:0/1105846427 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e6b33fed90 msgr2=0x55e6b33ff170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:35.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.151+0000 7f77b37fe640 1 --2- 172.21.15.16:0/1105846427 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e6b33fed90 0x55e6b33ff170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.151+0000 7f77b37fe640 1 -- 172.21.15.16:0/1105846427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6b3403980 con 0x55e6b33fe470 2024-03-28T22:12:35.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.151+0000 7f77b37fe640 1 --2- 172.21.15.16:0/1105846427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6b33fe470 0x55e6b3407100 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f77a4004af0 tx=0x7f77a403e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1f02de3b6eb618f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:35.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.151+0000 7f77a97fa640 1 -- 172.21.15.16:0/1105846427 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a4039040 con 0x55e6b33fe470 2024-03-28T22:12:35.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.151+0000 7f77a97fa640 1 -- 172.21.15.16:0/1105846427 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77a4003e50 con 0x55e6b33fe470 2024-03-28T22:12:35.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.151+0000 7f77a97fa640 1 -- 172.21.15.16:0/1105846427 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a4034e00 con 0x55e6b33fe470 2024-03-28T22:12:35.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.152+0000 7f77bcdfc740 1 -- 172.21.15.16:0/1105846427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6b33fe470 msgr2=0x55e6b3407100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:35.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.152+0000 7f77bcdfc740 1 --2- 172.21.15.16:0/1105846427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6b33fe470 0x55e6b3407100 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f77a4004af0 tx=0x7f77a403e000 comp rx=0 tx=0).stop 2024-03-28T22:12:35.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.152+0000 7f77bcdfc740 1 -- 172.21.15.16:0/1105846427 shutdown_connections 2024-03-28T22:12:35.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.152+0000 7f77bcdfc740 1 --2- 172.21.15.16:0/1105846427 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6b3407970 0x55e6b34134a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.152+0000 7f77bcdfc740 1 --2- 172.21.15.16:0/1105846427 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e6b33fed90 0x55e6b33ff170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.152+0000 7f77bcdfc740 1 --2- 172.21.15.16:0/1105846427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6b33fe470 0x55e6b3407100 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.152+0000 7f77bcdfc740 1 -- 172.21.15.16:0/1105846427 >> 172.21.15.16:0/1105846427 conn(0x55e6b33f8ba0 msgr2=0x55e6b33f7d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:35.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.152+0000 7f77bcdfc740 1 -- 172.21.15.16:0/1105846427 shutdown_connections 2024-03-28T22:12:35.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.152+0000 7f77bcdfc740 1 -- 172.21.15.16:0/1105846427 wait complete. 2024-03-28T22:12:35.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.152+0000 7f77bcdfc740 1 -- start start 2024-03-28T22:12:35.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.152+0000 7f77bcdfc740 1 Processor -- start 2024-03-28T22:12:35.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.153+0000 7f77bcdfc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6b33fe470 0x55e6b3401ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:35.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.153+0000 7f77bcdfc740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e6b33fed90 0x55e6b34020e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:35.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.153+0000 7f77bcdfc740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6b3407970 0x55e6b347cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:35.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.153+0000 7f77bcdfc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f77b41abab0 con 0x55e6b33fe470 2024-03-28T22:12:35.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.153+0000 7f77bcdfc740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f77b41acf20 con 0x55e6b33fed90 2024-03-28T22:12:35.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.153+0000 7f77bcdfc740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f77b41ac180 con 0x55e6b3407970 2024-03-28T22:12:35.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.153+0000 7f77bcdfc740 10 client.0 _ll_register_callbacks cb 0x7f77ac0b3dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:12:35.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.153+0000 7f77b37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6b33fe470 0x55e6b3401ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:35.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.153+0000 7f77b37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6b33fe470 0x55e6b3401ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57260/0 (socket says 172.21.15.16:57260) 2024-03-28T22:12:35.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.153+0000 7f77b37fe640 1 -- 172.21.15.16:0/510949413 learned_addr learned my addr 172.21.15.16:0/510949413 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:35.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.153+0000 7f77b2ffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e6b33fed90 0x55e6b34020e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:35.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.153+0000 7f77b3fff640 1 --2- 172.21.15.16:0/510949413 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6b3407970 0x55e6b347cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:35.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.153+0000 7f77b37fe640 1 -- 172.21.15.16:0/510949413 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6b3407970 msgr2=0x55e6b347cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:35.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.153+0000 7f77b37fe640 1 --2- 172.21.15.16:0/510949413 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6b3407970 0x55e6b347cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.153+0000 7f77b37fe640 1 -- 172.21.15.16:0/510949413 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e6b33fed90 msgr2=0x55e6b34020e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:35.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.153+0000 7f77b37fe640 1 --2- 172.21.15.16:0/510949413 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e6b33fed90 0x55e6b34020e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.153+0000 7f77b37fe640 1 -- 172.21.15.16:0/510949413 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6b347d1c0 con 0x55e6b33fe470 2024-03-28T22:12:35.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.153+0000 7f77b2ffd640 1 --2- 172.21.15.16:0/510949413 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e6b33fed90 0x55e6b34020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:12:35.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.153+0000 7f77b37fe640 1 --2- 172.21.15.16:0/510949413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6b33fe470 0x55e6b3401ba0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f77a4052040 tx=0x7f77a402cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=139d35f29e0d055e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:35.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.153+0000 7f7792ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:12:35.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.153+0000 7f7792ffd640 1 -- 172.21.15.16:0/510949413 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a405c020 con 0x55e6b33fe470 2024-03-28T22:12:35.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.153+0000 7f7792ffd640 1 -- 172.21.15.16:0/510949413 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f77a4032d10 con 0x55e6b33fe470 2024-03-28T22:12:35.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.153+0000 7f77bcdfc740 1 -- 172.21.15.16:0/510949413 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77a4003e50 con 0x55e6b33fe470 2024-03-28T22:12:35.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.154+0000 7f7792ffd640 1 -- 172.21.15.16:0/510949413 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f77a4039040 con 0x55e6b33fe470 2024-03-28T22:12:35.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.154+0000 7f77bcdfc740 10 client.15942 fetch_fsmap learned FSMap version 24 2024-03-28T22:12:35.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.154+0000 7f77bcdfc740 1 -- 172.21.15.16:0/510949413 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e6b347e3c0 con 0x55e6b33fe470 2024-03-28T22:12:35.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.154+0000 7f7792ffd640 1 -- 172.21.15.16:0/510949413 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77a4003e50 con 0x55e6b33fe470 2024-03-28T22:12:35.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.154+0000 7f7792ffd640 1 -- 172.21.15.16:0/510949413 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7f77a4050100 con 0x55e6b33fe470 2024-03-28T22:12:35.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.154+0000 7f77bcdfc740 10 client.15942 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:12:35.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.154+0000 7f77bcdfc740 10 client.15942 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:12:35.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.154+0000 7f77bcdfc740 20 client.15942 populate_metadata read hostname 'smithi016' 2024-03-28T22:12:35.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.154+0000 7f77bcdfc740 1 --2- 172.21.15.16:0/510949413 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e6b347ead0 0x55e6b3480ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:35.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.154+0000 7f77bcdfc740 4 client.15942 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:12:35.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.154+0000 7f77bcdfc740 1 -- 172.21.15.16:0/510949413 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e6b3369a40 con 0x55e6b347ead0 2024-03-28T22:12:35.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.154+0000 7f77b2ffd640 1 --2- 172.21.15.16:0/510949413 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e6b347ead0 0x55e6b3480ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:35.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.154+0000 7f77b2ffd640 1 --2- 172.21.15.16:0/510949413 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e6b347ead0 0x55e6b3480ec0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=316aa1cfd9a4834 server_cookie=2f2508ba942ca911 in_seq=0 out_seq=0 2024-03-28T22:12:35.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.154+0000 7f7792ffd640 10 client.15942 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:12:35.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.155+0000 7f7792ffd640 1 -- 172.21.15.16:0/510949413 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e6b3369a40 con 0x55e6b347ead0 2024-03-28T22:12:35.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.155+0000 7f7792ffd640 10 client.15942 handle_command_reply: tid=0 2024-03-28T22:12:35.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.155+0000 7f77bcdfc740 1 client.15942 shutdown 2024-03-28T22:12:35.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.155+0000 7f77bcdfc740 1 -- 172.21.15.16:0/510949413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6b33fe470 msgr2=0x55e6b3401ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:35.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.155+0000 7f77bcdfc740 1 --2- 172.21.15.16:0/510949413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6b33fe470 0x55e6b3401ba0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f77a4052040 tx=0x7f77a402cca0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.155+0000 7f77bcdfc740 1 -- 172.21.15.16:0/510949413 shutdown_connections 2024-03-28T22:12:35.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.155+0000 7f77bcdfc740 1 --2- 172.21.15.16:0/510949413 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e6b347ead0 0x55e6b3480ec0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.155+0000 7f77bcdfc740 1 --2- 172.21.15.16:0/510949413 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6b3407970 0x55e6b347cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.155+0000 7f77bcdfc740 1 --2- 172.21.15.16:0/510949413 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e6b33fed90 0x55e6b34020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.155+0000 7f77bcdfc740 1 --2- 172.21.15.16:0/510949413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6b33fe470 0x55e6b3401ba0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.155+0000 7f7792ffd640 0 client.15942 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:12:35.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.155+0000 7f77bcdfc740 1 -- 172.21.15.16:0/510949413 >> 172.21.15.16:0/510949413 conn(0x55e6b33f8ba0 msgr2=0x55e6b33f74b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:35.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.155+0000 7f77bcdfc740 1 -- 172.21.15.16:0/510949413 shutdown_connections 2024-03-28T22:12:35.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.155+0000 7f77bcdfc740 1 -- 172.21.15.16:0/510949413 wait complete. 2024-03-28T22:12:35.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.155+0000 7f77bcdfc740 20 client.15942 trim_cache size 0 max 16384 2024-03-28T22:12:35.166 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:12:35.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.155+0000 7f77ba215640 1 -- 172.21.15.16:0/3101428643 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f77a00a1ef0 msgr2=0x7f77a00a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:35.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.155+0000 7f77ba215640 1 --2- 172.21.15.16:0/3101428643 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f77a00a1ef0 0x7f77a00a43b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f77a4009ca0 tx=0x7f77a4032340 comp rx=0 tx=0).stop 2024-03-28T22:12:35.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.155+0000 7f77ba215640 1 -- 172.21.15.16:0/3101428643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77b41076f0 msgr2=0x7f77b419c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:35.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.155+0000 7f77ba215640 1 --2- 172.21.15.16:0/3101428643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77b41076f0 0x7f77b419c900 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f77ac00e9b0 tx=0x7f77ac00ee70 comp rx=0 tx=0).stop 2024-03-28T22:12:35.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.155+0000 7f77ba215640 1 -- 172.21.15.16:0/3101428643 shutdown_connections 2024-03-28T22:12:35.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.155+0000 7f77ba215640 1 --2- 172.21.15.16:0/3101428643 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f77a00a1ef0 0x7f77a00a43b0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.155+0000 7f77ba215640 1 --2- 172.21.15.16:0/3101428643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77b41076f0 0x7f77b419c900 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.155+0000 7f77ba215640 1 --2- 172.21.15.16:0/3101428643 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f77b41020a0 0x7f77b4198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.155+0000 7f77ba215640 1 --2- 172.21.15.16:0/3101428643 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77b41016e0 0x7f77b4198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:35.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.155+0000 7f77ba215640 1 -- 172.21.15.16:0/3101428643 >> 172.21.15.16:0/3101428643 conn(0x7f77b40fd1f0 msgr2=0x7f77b40fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:35.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.156+0000 7f77ba215640 1 -- 172.21.15.16:0/3101428643 shutdown_connections 2024-03-28T22:12:35.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:35.156+0000 7f77ba215640 1 -- 172.21.15.16:0/3101428643 wait complete. 2024-03-28T22:12:35.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:35 smithi053 ceph-mon[35979]: pgmap v63: 97 pgs: 97 active+clean; 82 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 260 op/s 2024-03-28T22:12:35.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:35 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/447753672' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:12:35.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:35 smithi063 ceph-mon[36249]: pgmap v63: 97 pgs: 97 active+clean; 82 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 260 op/s 2024-03-28T22:12:35.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:35 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/447753672' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:12:35.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:35 smithi016 ceph-mon[32374]: pgmap v63: 97 pgs: 97 active+clean; 82 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 260 op/s 2024-03-28T22:12:35.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:35 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/447753672' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:12:36.170 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:12:36.170 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:12:36.170 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:12:36.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.285+0000 7fe649484640 1 -- 172.21.15.16:0/1769020557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe64410bd40 msgr2=0x7fe64410e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:36.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.285+0000 7fe649484640 1 --2- 172.21.15.16:0/1769020557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe64410bd40 0x7fe64410e170 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fe63c009ca0 tx=0x7fe63c02c180 comp rx=0 tx=0).stop 2024-03-28T22:12:36.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.285+0000 7fe649484640 1 -- 172.21.15.16:0/1769020557 shutdown_connections 2024-03-28T22:12:36.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.285+0000 7fe649484640 1 --2- 172.21.15.16:0/1769020557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe64410bd40 0x7fe64410e170 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.285+0000 7fe649484640 1 --2- 172.21.15.16:0/1769020557 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe644103290 0x7fe64410b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.285+0000 7fe649484640 1 --2- 172.21.15.16:0/1769020557 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe644102970 0x7fe644102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.285+0000 7fe649484640 1 -- 172.21.15.16:0/1769020557 >> 172.21.15.16:0/1769020557 conn(0x7fe644077800 msgr2=0x7fe644077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:36.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.285+0000 7fe649484640 1 -- 172.21.15.16:0/1769020557 shutdown_connections 2024-03-28T22:12:36.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.285+0000 7fe649484640 1 -- 172.21.15.16:0/1769020557 wait complete. 2024-03-28T22:12:36.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.286+0000 7fe649484640 1 Processor -- start 2024-03-28T22:12:36.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.286+0000 7fe649484640 1 -- start start 2024-03-28T22:12:36.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.286+0000 7fe649484640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe644102970 0x7fe64419a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:36.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.286+0000 7fe649484640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe644103290 0x7fe64419a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:36.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.286+0000 7fe649484640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe64410bd40 0x7fe64419ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:36.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.286+0000 7fe649484640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe644111ad0 con 0x7fe644102970 2024-03-28T22:12:36.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.286+0000 7fe649484640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe644111950 con 0x7fe64410bd40 2024-03-28T22:12:36.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.286+0000 7fe649484640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe644111c50 con 0x7fe644103290 2024-03-28T22:12:36.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.286+0000 7fe642ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe644102970 0x7fe64419a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:36.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.286+0000 7fe642ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe644102970 0x7fe64419a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57282/0 (socket says 172.21.15.16:57282) 2024-03-28T22:12:36.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.286+0000 7fe642ffd640 1 -- 172.21.15.16:0/1171365565 learned_addr learned my addr 172.21.15.16:0/1171365565 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:36.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.286+0000 7fe6437fe640 1 --2- 172.21.15.16:0/1171365565 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe64410bd40 0x7fe64419ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:36.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.286+0000 7fe6427fc640 1 --2- 172.21.15.16:0/1171365565 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe644103290 0x7fe64419a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:36.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.286+0000 7fe642ffd640 1 -- 172.21.15.16:0/1171365565 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe644103290 msgr2=0x7fe64419a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:36.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.286+0000 7fe642ffd640 1 --2- 172.21.15.16:0/1171365565 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe644103290 0x7fe64419a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.286+0000 7fe642ffd640 1 -- 172.21.15.16:0/1171365565 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe64410bd40 msgr2=0x7fe64419ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:36.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.286+0000 7fe642ffd640 1 --2- 172.21.15.16:0/1171365565 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe64410bd40 0x7fe64419ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.286+0000 7fe642ffd640 1 -- 172.21.15.16:0/1171365565 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe64419f1c0 con 0x7fe644102970 2024-03-28T22:12:36.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.287+0000 7fe642ffd640 1 --2- 172.21.15.16:0/1171365565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe644102970 0x7fe64419a1d0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fe63400e9f0 tx=0x7fe63400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:36.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.287+0000 7fe62bfff640 1 -- 172.21.15.16:0/1171365565 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe63400cd80 con 0x7fe644102970 2024-03-28T22:12:36.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.287+0000 7fe649484640 1 -- 172.21.15.16:0/1171365565 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe64419f4b0 con 0x7fe644102970 2024-03-28T22:12:36.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.287+0000 7fe62bfff640 1 -- 172.21.15.16:0/1171365565 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe634010910 con 0x7fe644102970 2024-03-28T22:12:36.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.287+0000 7fe62bfff640 1 -- 172.21.15.16:0/1171365565 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6340188f0 con 0x7fe644102970 2024-03-28T22:12:36.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.287+0000 7fe649484640 1 -- 172.21.15.16:0/1171365565 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6441a6d40 con 0x7fe644102970 2024-03-28T22:12:36.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.290+0000 7fe62bfff640 1 -- 172.21.15.16:0/1171365565 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7fe634010ac0 con 0x7fe644102970 2024-03-28T22:12:36.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.290+0000 7fe64c066740 1 Processor -- start 2024-03-28T22:12:36.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.290+0000 7fe64c066740 1 -- start start 2024-03-28T22:12:36.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.290+0000 7fe64c066740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560361cfe9f0 0x560361cfedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:36.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.290+0000 7fe64c066740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560361cff310 0x560361d079b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:36.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.290+0000 7fe64c066740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560361d07ef0 0x560361d0a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:36.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.290+0000 7fe64c066740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560361bff0b0 con 0x560361d07ef0 2024-03-28T22:12:36.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.290+0000 7fe64c066740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x560361c61ea0 con 0x560361cff310 2024-03-28T22:12:36.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.290+0000 7fe64c066740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x560361c0ac00 con 0x560361cfe9f0 2024-03-28T22:12:36.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.290+0000 7fe62bfff640 1 --2- 172.21.15.16:0/1171365565 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fe62c0a1ef0 0x7fe62c0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:36.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.290+0000 7fe642ffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560361cff310 0x560361d079b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:36.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.290+0000 7fe6427fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560361d07ef0 0x560361d0a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:36.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.290+0000 7fe642ffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560361cff310 0x560361d079b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:48390/0 (socket says 172.21.15.16:48390) 2024-03-28T22:12:36.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.290+0000 7fe6427fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560361d07ef0 0x560361d0a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57296/0 (socket says 172.21.15.16:57296) 2024-03-28T22:12:36.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.290+0000 7fe642ffd640 1 -- 172.21.15.16:0/948734595 learned_addr learned my addr 172.21.15.16:0/948734595 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:36.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.290+0000 7fe6437fe640 1 --2- 172.21.15.16:0/948734595 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560361cfe9f0 0x560361cfedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:36.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.290+0000 7fe6437fe640 1 --2- 172.21.15.16:0/1171365565 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fe62c0a1ef0 0x7fe62c0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:36.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.290+0000 7fe62bfff640 1 -- 172.21.15.16:0/1171365565 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fe6340ce5e0 con 0x7fe644102970 2024-03-28T22:12:36.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.291+0000 7fe642ffd640 1 -- 172.21.15.16:0/948734595 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560361cfe9f0 msgr2=0x560361cfedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:36.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.291+0000 7fe642ffd640 1 --2- 172.21.15.16:0/948734595 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560361cfe9f0 0x560361cfedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.291+0000 7fe642ffd640 1 -- 172.21.15.16:0/948734595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560361d07ef0 msgr2=0x560361d0a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:36.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.291+0000 7fe642ffd640 1 --2- 172.21.15.16:0/948734595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560361d07ef0 0x560361d0a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.291+0000 7fe642ffd640 1 -- 172.21.15.16:0/948734595 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560361a3e7e0 con 0x560361cff310 2024-03-28T22:12:36.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.291+0000 7fe642ffd640 1 --2- 172.21.15.16:0/948734595 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560361cff310 0x560361d079b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe6340cbcc0 tx=0x7fe63409aa10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1daf878e671b7acb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:36.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.291+0000 7fe6437fe640 1 --2- 172.21.15.16:0/1171365565 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fe62c0a1ef0 0x7fe62c0a43b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe63c004af0 tx=0x7fe63c005c50 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:36.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.291+0000 7fe6297fa640 1 -- 172.21.15.16:0/948734595 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6340a3790 con 0x560361cff310 2024-03-28T22:12:36.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.291+0000 7fe6297fa640 1 -- 172.21.15.16:0/948734595 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6340cbfd0 con 0x560361cff310 2024-03-28T22:12:36.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.291+0000 7fe6297fa640 1 -- 172.21.15.16:0/948734595 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe634095d80 con 0x560361cff310 2024-03-28T22:12:36.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.292+0000 7fe64c066740 1 -- 172.21.15.16:0/948734595 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560361cff310 msgr2=0x560361d079b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:36.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.292+0000 7fe64c066740 1 --2- 172.21.15.16:0/948734595 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560361cff310 0x560361d079b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe6340cbcc0 tx=0x7fe63409aa10 comp rx=0 tx=0).stop 2024-03-28T22:12:36.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.292+0000 7fe64c066740 1 -- 172.21.15.16:0/948734595 shutdown_connections 2024-03-28T22:12:36.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.292+0000 7fe64c066740 1 --2- 172.21.15.16:0/948734595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560361d07ef0 0x560361d0a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.292+0000 7fe64c066740 1 --2- 172.21.15.16:0/948734595 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560361cff310 0x560361d079b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.292+0000 7fe64c066740 1 --2- 172.21.15.16:0/948734595 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560361cfe9f0 0x560361cfedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.292+0000 7fe64c066740 1 -- 172.21.15.16:0/948734595 >> 172.21.15.16:0/948734595 conn(0x560361cf9120 msgr2=0x560361cf9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:36.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.292+0000 7fe64c066740 1 -- 172.21.15.16:0/948734595 shutdown_connections 2024-03-28T22:12:36.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.292+0000 7fe64c066740 1 -- 172.21.15.16:0/948734595 wait complete. 2024-03-28T22:12:36.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.292+0000 7fe64c066740 1 -- start start 2024-03-28T22:12:36.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.292+0000 7fe64c066740 1 Processor -- start 2024-03-28T22:12:36.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.292+0000 7fe64c066740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560361cfe9f0 0x560361d04e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:36.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.292+0000 7fe64c066740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560361cff310 0x560361d053a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:36.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.292+0000 7fe64c066740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560361d07ef0 0x560361d058e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:36.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.292+0000 7fe64c066740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560361c003d0 con 0x560361cff310 2024-03-28T22:12:36.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.292+0000 7fe64c066740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x560361c5df40 con 0x560361cfe9f0 2024-03-28T22:12:36.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.292+0000 7fe64c066740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x560361d0ace0 con 0x560361d07ef0 2024-03-28T22:12:36.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.292+0000 7fe64c066740 10 client.0 _ll_register_callbacks cb 0x560361c1db70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:12:36.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.292+0000 7fe6427fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560361cff310 0x560361d053a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:36.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.293+0000 7fe6427fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560361cff310 0x560361d053a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57308/0 (socket says 172.21.15.16:57308) 2024-03-28T22:12:36.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.293+0000 7fe6427fc640 1 -- 172.21.15.16:0/4291680636 learned_addr learned my addr 172.21.15.16:0/4291680636 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:36.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.293+0000 7fe642ffd640 1 --2- 172.21.15.16:0/4291680636 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560361cfe9f0 0x560361d04e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:36.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.293+0000 7fe6427fc640 1 -- 172.21.15.16:0/4291680636 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560361d07ef0 msgr2=0x560361d058e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:36.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.293+0000 7fe6437fe640 1 --2- 172.21.15.16:0/4291680636 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560361d07ef0 0x560361d058e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:36.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.293+0000 7fe6427fc640 1 --2- 172.21.15.16:0/4291680636 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560361d07ef0 0x560361d058e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.293+0000 7fe6427fc640 1 -- 172.21.15.16:0/4291680636 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560361cfe9f0 msgr2=0x560361d04e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:36.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.293+0000 7fe6427fc640 1 --2- 172.21.15.16:0/4291680636 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560361cfe9f0 0x560361d04e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.293+0000 7fe6427fc640 1 -- 172.21.15.16:0/4291680636 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560361a9ef70 con 0x560361cff310 2024-03-28T22:12:36.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.293+0000 7fe642ffd640 1 --2- 172.21.15.16:0/4291680636 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560361cfe9f0 0x560361d04e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:12:36.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.293+0000 7fe6427fc640 1 --2- 172.21.15.16:0/4291680636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560361cff310 0x560361d053a0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fe6380116c0 tx=0x7fe638011b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eb0466a12fe4203 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:36.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.293+0000 7fe626ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:12:36.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.293+0000 7fe626ff5640 1 -- 172.21.15.16:0/4291680636 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe63801e060 con 0x560361cff310 2024-03-28T22:12:36.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.293+0000 7fe626ff5640 1 -- 172.21.15.16:0/4291680636 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fe6380193f0 con 0x560361cff310 2024-03-28T22:12:36.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.293+0000 7fe64c066740 1 -- 172.21.15.16:0/4291680636 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6340cbfd0 con 0x560361cff310 2024-03-28T22:12:36.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.293+0000 7fe626ff5640 1 -- 172.21.15.16:0/4291680636 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe63800fe70 con 0x560361cff310 2024-03-28T22:12:36.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.293+0000 7fe64c066740 10 client.15954 fetch_fsmap learned FSMap version 24 2024-03-28T22:12:36.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.293+0000 7fe64c066740 1 -- 172.21.15.16:0/4291680636 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x560361936170 con 0x560361cff310 2024-03-28T22:12:36.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.294+0000 7fe626ff5640 1 -- 172.21.15.16:0/4291680636 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6340cbfd0 con 0x560361cff310 2024-03-28T22:12:36.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.294+0000 7fe626ff5640 1 -- 172.21.15.16:0/4291680636 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7fe638018030 con 0x560361cff310 2024-03-28T22:12:36.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.294+0000 7fe64c066740 10 client.15954 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:12:36.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.294+0000 7fe64c066740 10 client.15954 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:12:36.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.294+0000 7fe64c066740 20 client.15954 populate_metadata read hostname 'smithi016' 2024-03-28T22:12:36.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.294+0000 7fe64c066740 1 --2- 172.21.15.16:0/4291680636 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x560361d17da0 0x560361d1a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:36.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.294+0000 7fe64c066740 4 client.15954 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:12:36.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.294+0000 7fe64c066740 1 -- 172.21.15.16:0/4291680636 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560361c6f3e0 con 0x560361d17da0 2024-03-28T22:12:36.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.294+0000 7fe642ffd640 1 --2- 172.21.15.16:0/4291680636 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x560361d17da0 0x560361d1a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:36.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.294+0000 7fe642ffd640 1 --2- 172.21.15.16:0/4291680636 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x560361d17da0 0x560361d1a210 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0fe94498f7bd033 server_cookie=49ea4e1643b126c6 in_seq=0 out_seq=0 2024-03-28T22:12:36.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.294+0000 7fe626ff5640 10 client.15954 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:12:36.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.295+0000 7fe626ff5640 1 -- 172.21.15.16:0/4291680636 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560361c6f3e0 con 0x560361d17da0 2024-03-28T22:12:36.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.295+0000 7fe626ff5640 10 client.15954 handle_command_reply: tid=0 2024-03-28T22:12:36.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.295+0000 7fe64c066740 1 client.15954 shutdown 2024-03-28T22:12:36.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.296+0000 7fe64c066740 1 -- 172.21.15.16:0/4291680636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560361cff310 msgr2=0x560361d053a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:36.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.296+0000 7fe64c066740 1 --2- 172.21.15.16:0/4291680636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560361cff310 0x560361d053a0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fe6380116c0 tx=0x7fe638011b80 comp rx=0 tx=0).stop 2024-03-28T22:12:36.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.296+0000 7fe64c066740 1 -- 172.21.15.16:0/4291680636 shutdown_connections 2024-03-28T22:12:36.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.296+0000 7fe64c066740 1 --2- 172.21.15.16:0/4291680636 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x560361d17da0 0x560361d1a210 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.296+0000 7fe64c066740 1 --2- 172.21.15.16:0/4291680636 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560361d07ef0 0x560361d058e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.296+0000 7fe64c066740 1 --2- 172.21.15.16:0/4291680636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560361cff310 0x560361d053a0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.296+0000 7fe626ff5640 0 client.15954 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:12:36.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.296+0000 7fe64c066740 1 --2- 172.21.15.16:0/4291680636 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560361cfe9f0 0x560361d04e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.296+0000 7fe64c066740 1 -- 172.21.15.16:0/4291680636 >> 172.21.15.16:0/4291680636 conn(0x560361cf9120 msgr2=0x560361cfae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:36.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.296+0000 7fe64c066740 1 -- 172.21.15.16:0/4291680636 shutdown_connections 2024-03-28T22:12:36.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.296+0000 7fe64c066740 1 -- 172.21.15.16:0/4291680636 wait complete. 2024-03-28T22:12:36.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.296+0000 7fe64c066740 20 client.15954 trim_cache size 0 max 16384 2024-03-28T22:12:36.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.313+0000 7fe64c066740 1 Processor -- start 2024-03-28T22:12:36.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.313+0000 7fe64c066740 1 -- start start 2024-03-28T22:12:36.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.314+0000 7fe64c066740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560361cfe9f0 0x560361d07590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:36.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.314+0000 7fe64c066740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560361cff310 0x560361d13960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:36.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.314+0000 7fe64c066740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560361d07ef0 0x560361cf8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:36.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.314+0000 7fe64c066740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe6441b1e70 con 0x560361cfe9f0 2024-03-28T22:12:36.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.314+0000 7fe64c066740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe6441af710 con 0x560361cff310 2024-03-28T22:12:36.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.314+0000 7fe64c066740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe6441ad560 con 0x560361d07ef0 2024-03-28T22:12:36.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.314+0000 7fe642ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560361cfe9f0 0x560361d07590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:36.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.314+0000 7fe642ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560361cfe9f0 0x560361d07590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57312/0 (socket says 172.21.15.16:57312) 2024-03-28T22:12:36.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.314+0000 7fe642ffd640 1 -- 172.21.15.16:0/3603740713 learned_addr learned my addr 172.21.15.16:0/3603740713 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:36.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.314+0000 7fe6427fc640 1 --2- 172.21.15.16:0/3603740713 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560361cff310 0x560361d13960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:36.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.314+0000 7fe642ffd640 1 -- 172.21.15.16:0/3603740713 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560361d07ef0 msgr2=0x560361cf8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:36.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.314+0000 7fe642ffd640 1 --2- 172.21.15.16:0/3603740713 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560361d07ef0 0x560361cf8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.314+0000 7fe642ffd640 1 -- 172.21.15.16:0/3603740713 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560361cff310 msgr2=0x560361d13960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:36.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.314+0000 7fe642ffd640 1 --2- 172.21.15.16:0/3603740713 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560361cff310 0x560361d13960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.314+0000 7fe642ffd640 1 -- 172.21.15.16:0/3603740713 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560361d14b10 con 0x560361cfe9f0 2024-03-28T22:12:36.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.314+0000 7fe6427fc640 1 --2- 172.21.15.16:0/3603740713 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560361cff310 0x560361d13960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:12:36.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.314+0000 7fe642ffd640 1 --2- 172.21.15.16:0/3603740713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560361cfe9f0 0x560361d07590 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fe63409acf0 tx=0x7fe63409b130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b47bc8ef83c08bf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:36.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.314+0000 7fe628ff9640 1 -- 172.21.15.16:0/3603740713 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe634097660 con 0x560361cfe9f0 2024-03-28T22:12:36.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.314+0000 7fe628ff9640 1 -- 172.21.15.16:0/3603740713 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6340cbfd0 con 0x560361cfe9f0 2024-03-28T22:12:36.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.314+0000 7fe628ff9640 1 -- 172.21.15.16:0/3603740713 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe63409a770 con 0x560361cfe9f0 2024-03-28T22:12:36.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.315+0000 7fe64c066740 1 -- 172.21.15.16:0/3603740713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560361cfe9f0 msgr2=0x560361d07590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:36.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.315+0000 7fe64c066740 1 --2- 172.21.15.16:0/3603740713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560361cfe9f0 0x560361d07590 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fe63409acf0 tx=0x7fe63409b130 comp rx=0 tx=0).stop 2024-03-28T22:12:36.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.315+0000 7fe64c066740 1 -- 172.21.15.16:0/3603740713 shutdown_connections 2024-03-28T22:12:36.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.315+0000 7fe64c066740 1 --2- 172.21.15.16:0/3603740713 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560361d07ef0 0x560361cf8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.315+0000 7fe64c066740 1 --2- 172.21.15.16:0/3603740713 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560361cff310 0x560361d13960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.315+0000 7fe64c066740 1 --2- 172.21.15.16:0/3603740713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560361cfe9f0 0x560361d07590 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.315+0000 7fe64c066740 1 -- 172.21.15.16:0/3603740713 >> 172.21.15.16:0/3603740713 conn(0x560361cf9120 msgr2=0x560361cf78f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:36.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.315+0000 7fe64c066740 1 -- 172.21.15.16:0/3603740713 shutdown_connections 2024-03-28T22:12:36.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.315+0000 7fe64c066740 1 -- 172.21.15.16:0/3603740713 wait complete. 2024-03-28T22:12:36.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.315+0000 7fe64c066740 1 -- start start 2024-03-28T22:12:36.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.315+0000 7fe64c066740 1 Processor -- start 2024-03-28T22:12:36.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.315+0000 7fe64c066740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560361cfe9f0 0x560361d10180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:36.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.315+0000 7fe64c066740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560361cff310 0x560361d0cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:36.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.315+0000 7fe64c066740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560361d07ef0 0x560361d0d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:36.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.315+0000 7fe64c066740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe6441adbd0 con 0x560361cfe9f0 2024-03-28T22:12:36.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.315+0000 7fe64c066740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe6441af040 con 0x560361d07ef0 2024-03-28T22:12:36.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.315+0000 7fe64c066740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe6441ae2a0 con 0x560361cff310 2024-03-28T22:12:36.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.315+0000 7fe64c066740 10 client.0 _ll_register_callbacks cb 0x560361c1db70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:12:36.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.315+0000 7fe642ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560361cfe9f0 0x560361d10180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:36.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.316+0000 7fe642ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560361cfe9f0 0x560361d10180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57326/0 (socket says 172.21.15.16:57326) 2024-03-28T22:12:36.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.316+0000 7fe642ffd640 1 -- 172.21.15.16:0/894360952 learned_addr learned my addr 172.21.15.16:0/894360952 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:36.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.316+0000 7fe6437fe640 1 --2- 172.21.15.16:0/894360952 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560361d07ef0 0x560361d0d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:36.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.316+0000 7fe6427fc640 1 --2- 172.21.15.16:0/894360952 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560361cff310 0x560361d0cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:36.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.316+0000 7fe642ffd640 1 -- 172.21.15.16:0/894360952 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560361cff310 msgr2=0x560361d0cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:36.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.316+0000 7fe642ffd640 1 --2- 172.21.15.16:0/894360952 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560361cff310 0x560361d0cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.316+0000 7fe642ffd640 1 -- 172.21.15.16:0/894360952 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560361d07ef0 msgr2=0x560361d0d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:36.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.316+0000 7fe642ffd640 1 --2- 172.21.15.16:0/894360952 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560361d07ef0 0x560361d0d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.316+0000 7fe642ffd640 1 -- 172.21.15.16:0/894360952 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560361d0db90 con 0x560361cfe9f0 2024-03-28T22:12:36.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.316+0000 7fe6427fc640 1 --2- 172.21.15.16:0/894360952 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560361cff310 0x560361d0cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:12:36.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.316+0000 7fe642ffd640 1 --2- 172.21.15.16:0/894360952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560361cfe9f0 0x560361d10180 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fe6340cc350 tx=0x7fe6340cc210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70447e800089cc67 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:36.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.316+0000 7fe626ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:12:36.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.316+0000 7fe626ff5640 1 -- 172.21.15.16:0/894360952 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6340a3790 con 0x560361cfe9f0 2024-03-28T22:12:36.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.316+0000 7fe626ff5640 1 -- 172.21.15.16:0/894360952 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fe634084030 con 0x560361cfe9f0 2024-03-28T22:12:36.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.316+0000 7fe64c066740 1 -- 172.21.15.16:0/894360952 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6340cbfd0 con 0x560361cfe9f0 2024-03-28T22:12:36.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.316+0000 7fe626ff5640 1 -- 172.21.15.16:0/894360952 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe634084230 con 0x560361cfe9f0 2024-03-28T22:12:36.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.316+0000 7fe64c066740 10 client.15960 fetch_fsmap learned FSMap version 24 2024-03-28T22:12:36.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.316+0000 7fe64c066740 1 -- 172.21.15.16:0/894360952 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x560361d7f110 con 0x560361cfe9f0 2024-03-28T22:12:36.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.316+0000 7fe626ff5640 1 -- 172.21.15.16:0/894360952 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6340cbfd0 con 0x560361cfe9f0 2024-03-28T22:12:36.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.317+0000 7fe626ff5640 1 -- 172.21.15.16:0/894360952 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7fe634095c40 con 0x560361cfe9f0 2024-03-28T22:12:36.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.317+0000 7fe64c066740 10 client.15960 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:12:36.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.317+0000 7fe64c066740 10 client.15960 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:12:36.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.317+0000 7fe64c066740 20 client.15960 populate_metadata read hostname 'smithi016' 2024-03-28T22:12:36.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.317+0000 7fe64c066740 1 --2- 172.21.15.16:0/894360952 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x560361d7f3f0 0x560361d817e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:36.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.317+0000 7fe64c066740 4 client.15960 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:12:36.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.317+0000 7fe64c066740 1 -- 172.21.15.16:0/894360952 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560361c69730 con 0x560361d7f3f0 2024-03-28T22:12:36.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.317+0000 7fe6427fc640 1 --2- 172.21.15.16:0/894360952 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x560361d7f3f0 0x560361d817e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:36.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.317+0000 7fe626ff5640 10 client.15960 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:12:36.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.317+0000 7fe6427fc640 1 --2- 172.21.15.16:0/894360952 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x560361d7f3f0 0x560361d817e0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d82cac27593143 server_cookie=80c7ff52917954fd in_seq=0 out_seq=0 2024-03-28T22:12:36.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.319+0000 7fe626ff5640 1 -- 172.21.15.16:0/894360952 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560361c69730 con 0x560361d7f3f0 2024-03-28T22:12:36.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.319+0000 7fe626ff5640 10 client.15960 handle_command_reply: tid=0 2024-03-28T22:12:36.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.319+0000 7fe64c066740 1 client.15960 shutdown 2024-03-28T22:12:36.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.319+0000 7fe64c066740 1 -- 172.21.15.16:0/894360952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560361cfe9f0 msgr2=0x560361d10180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:36.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.319+0000 7fe64c066740 1 --2- 172.21.15.16:0/894360952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560361cfe9f0 0x560361d10180 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fe6340cc350 tx=0x7fe6340cc210 comp rx=0 tx=0).stop 2024-03-28T22:12:36.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.319+0000 7fe64c066740 1 -- 172.21.15.16:0/894360952 shutdown_connections 2024-03-28T22:12:36.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.319+0000 7fe64c066740 1 --2- 172.21.15.16:0/894360952 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x560361d7f3f0 0x560361d817e0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.319+0000 7fe64c066740 1 --2- 172.21.15.16:0/894360952 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560361d07ef0 0x560361d0d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.319+0000 7fe64c066740 1 --2- 172.21.15.16:0/894360952 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560361cff310 0x560361d0cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.319+0000 7fe64c066740 1 --2- 172.21.15.16:0/894360952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560361cfe9f0 0x560361d10180 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.319+0000 7fe64c066740 1 -- 172.21.15.16:0/894360952 >> 172.21.15.16:0/894360952 conn(0x560361cf9120 msgr2=0x560361cf8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:36.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.319+0000 7fe626ff5640 0 client.15960 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:12:36.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.319+0000 7fe64c066740 1 -- 172.21.15.16:0/894360952 shutdown_connections 2024-03-28T22:12:36.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.319+0000 7fe64c066740 1 -- 172.21.15.16:0/894360952 wait complete. 2024-03-28T22:12:36.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.319+0000 7fe64c066740 20 client.15960 trim_cache size 0 max 16384 2024-03-28T22:12:36.328 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:12:36.328 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-03-28T22:12:36.328 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "f1d3d6a4-0dc0-49e3-b4ef-5909a37808de", 2024-03-28T22:12:36.328 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-03-28T22:12:36.329 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:12:36.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.319+0000 7fe649484640 1 -- 172.21.15.16:0/1171365565 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fe62c0a1ef0 msgr2=0x7fe62c0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:36.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.319+0000 7fe649484640 1 --2- 172.21.15.16:0/1171365565 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fe62c0a1ef0 0x7fe62c0a43b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe63c004af0 tx=0x7fe63c005c50 comp rx=0 tx=0).stop 2024-03-28T22:12:36.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.319+0000 7fe649484640 1 -- 172.21.15.16:0/1171365565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe644102970 msgr2=0x7fe64419a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:36.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.319+0000 7fe649484640 1 --2- 172.21.15.16:0/1171365565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe644102970 0x7fe64419a1d0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fe63400e9f0 tx=0x7fe63400eeb0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.320+0000 7fe649484640 1 -- 172.21.15.16:0/1171365565 shutdown_connections 2024-03-28T22:12:36.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.320+0000 7fe649484640 1 --2- 172.21.15.16:0/1171365565 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fe62c0a1ef0 0x7fe62c0a43b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.320+0000 7fe649484640 1 --2- 172.21.15.16:0/1171365565 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe64410bd40 0x7fe64419ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.320+0000 7fe649484640 1 --2- 172.21.15.16:0/1171365565 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe644103290 0x7fe64419a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.320+0000 7fe649484640 1 --2- 172.21.15.16:0/1171365565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe644102970 0x7fe64419a1d0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.320+0000 7fe649484640 1 -- 172.21.15.16:0/1171365565 >> 172.21.15.16:0/1171365565 conn(0x7fe644077800 msgr2=0x7fe644107b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:36.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.320+0000 7fe649484640 1 -- 172.21.15.16:0/1171365565 shutdown_connections 2024-03-28T22:12:36.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.320+0000 7fe649484640 1 -- 172.21.15.16:0/1171365565 wait complete. 2024-03-28T22:12:36.333 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:12:36.334 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:12:36.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.438+0000 7f5beebd0640 1 -- 172.21.15.16:0/276753811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5be8106680 msgr2=0x7f5be8106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:36.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.438+0000 7f5beebd0640 1 --2- 172.21.15.16:0/276753811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5be8106680 0x7f5be8106a60 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc009930 tx=0x7f5bdc02bc30 comp rx=0 tx=0).stop 2024-03-28T22:12:36.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.438+0000 7f5beebd0640 1 -- 172.21.15.16:0/276753811 shutdown_connections 2024-03-28T22:12:36.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.438+0000 7f5beebd0640 1 --2- 172.21.15.16:0/276753811 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5be8100fa0 0x7f5be810d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.438+0000 7f5beebd0640 1 --2- 172.21.15.16:0/276753811 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5be81005e0 0x7f5be8100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.438+0000 7f5beebd0640 1 --2- 172.21.15.16:0/276753811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5be8106680 0x7f5be8106a60 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.438+0000 7f5beebd0640 1 -- 172.21.15.16:0/276753811 >> 172.21.15.16:0/276753811 conn(0x7f5be8077770 msgr2=0x7f5be8077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:36.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.438+0000 7f5beebd0640 1 -- 172.21.15.16:0/276753811 shutdown_connections 2024-03-28T22:12:36.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.438+0000 7f5beebd0640 1 -- 172.21.15.16:0/276753811 wait complete. 2024-03-28T22:12:36.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.439+0000 7f5beebd0640 1 Processor -- start 2024-03-28T22:12:36.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.439+0000 7f5beebd0640 1 -- start start 2024-03-28T22:12:36.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.439+0000 7f5beebd0640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5be81005e0 0x7f5be8198d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:36.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.439+0000 7f5beebd0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5be8100fa0 0x7f5be8199280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:36.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.439+0000 7f5beebd0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5be8106680 0x7f5be81a3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:36.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.439+0000 7f5beebd0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5be81109c0 con 0x7f5be8100fa0 2024-03-28T22:12:36.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.439+0000 7f5beebd0640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5be8110840 con 0x7f5be81005e0 2024-03-28T22:12:36.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.439+0000 7f5beebd0640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5be8110b40 con 0x7f5be8106680 2024-03-28T22:12:36.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.439+0000 7f5be7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5be8100fa0 0x7f5be8199280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:36.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.439+0000 7f5be7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5be8100fa0 0x7f5be8199280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57352/0 (socket says 172.21.15.16:57352) 2024-03-28T22:12:36.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.439+0000 7f5be7fff640 1 -- 172.21.15.16:0/1517173882 learned_addr learned my addr 172.21.15.16:0/1517173882 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:36.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.439+0000 7f5bec96c640 1 --2- 172.21.15.16:0/1517173882 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5be81005e0 0x7f5be8198d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:36.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.439+0000 7f5bed16d640 1 --2- 172.21.15.16:0/1517173882 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5be8106680 0x7f5be81a3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:36.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.439+0000 7f5be7fff640 1 -- 172.21.15.16:0/1517173882 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5be8106680 msgr2=0x7f5be81a3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:36.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.439+0000 7f5be7fff640 1 --2- 172.21.15.16:0/1517173882 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5be8106680 0x7f5be81a3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.439+0000 7f5be7fff640 1 -- 172.21.15.16:0/1517173882 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5be81005e0 msgr2=0x7f5be8198d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:36.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.439+0000 7f5be7fff640 1 --2- 172.21.15.16:0/1517173882 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5be81005e0 0x7f5be8198d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.439+0000 7f5be7fff640 1 -- 172.21.15.16:0/1517173882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5be81a4300 con 0x7f5be8100fa0 2024-03-28T22:12:36.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.440+0000 7f5be7fff640 1 --2- 172.21.15.16:0/1517173882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5be8100fa0 0x7f5be8199280 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f5bd400e9f0 tx=0x7f5bd400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:36.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.440+0000 7f5be5ffb640 1 -- 172.21.15.16:0/1517173882 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd400cd80 con 0x7f5be8100fa0 2024-03-28T22:12:36.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.440+0000 7f5be5ffb640 1 -- 172.21.15.16:0/1517173882 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bd4010910 con 0x7f5be8100fa0 2024-03-28T22:12:36.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.440+0000 7f5beebd0640 1 -- 172.21.15.16:0/1517173882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5be81a44d0 con 0x7f5be8100fa0 2024-03-28T22:12:36.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.440+0000 7f5be5ffb640 1 -- 172.21.15.16:0/1517173882 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd4018990 con 0x7f5be8100fa0 2024-03-28T22:12:36.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.440+0000 7f5beebd0640 1 -- 172.21.15.16:0/1517173882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5be81a4a20 con 0x7f5be8100fa0 2024-03-28T22:12:36.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.442+0000 7f5be5ffb640 1 -- 172.21.15.16:0/1517173882 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f5bd4010ac0 con 0x7f5be8100fa0 2024-03-28T22:12:36.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.442+0000 7f5bf17b2740 1 Processor -- start 2024-03-28T22:12:36.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.442+0000 7f5bf17b2740 1 -- start start 2024-03-28T22:12:36.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.442+0000 7f5bf17b2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2339a5290 0x55a2339a5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:36.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.443+0000 7f5bf17b2740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a2339a5bb0 0x55a2339ae250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:36.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.443+0000 7f5bf17b2740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a2339ae790 0x55a2339b0b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:36.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.443+0000 7f5bf17b2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a233909180 con 0x55a2339a5290 2024-03-28T22:12:36.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.443+0000 7f5bf17b2740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55a2338b1b70 con 0x55a2339ae790 2024-03-28T22:12:36.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.443+0000 7f5be5ffb640 1 --2- 172.21.15.16:0/1517173882 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f5bd80a1ef0 0x7f5bd80a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:36.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.443+0000 7f5bf17b2740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55a233905220 con 0x55a2339a5bb0 2024-03-28T22:12:36.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.443+0000 7f5bed16d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2339a5290 0x55a2339a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:36.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.443+0000 7f5bed16d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2339a5290 0x55a2339a5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57362/0 (socket says 172.21.15.16:57362) 2024-03-28T22:12:36.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.443+0000 7f5be5ffb640 1 -- 172.21.15.16:0/1517173882 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f5bd40ce4b0 con 0x7f5be8100fa0 2024-03-28T22:12:36.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.443+0000 7f5bed16d640 1 -- 172.21.15.16:0/2036308309 learned_addr learned my addr 172.21.15.16:0/2036308309 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:36.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.443+0000 7f5be7fff640 1 --2- 172.21.15.16:0/2036308309 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a2339ae790 0x55a2339b0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:36.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.443+0000 7f5bed16d640 1 --2- 172.21.15.16:0/1517173882 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f5bd80a1ef0 0x7f5bd80a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:36.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.443+0000 7f5bec96c640 1 --2- 172.21.15.16:0/2036308309 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a2339a5bb0 0x55a2339ae250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:36.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.443+0000 7f5bed16d640 1 -- 172.21.15.16:0/2036308309 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a2339a5bb0 msgr2=0x55a2339ae250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:36.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.443+0000 7f5bed16d640 1 --2- 172.21.15.16:0/2036308309 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a2339a5bb0 0x55a2339ae250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.443+0000 7f5bed16d640 1 -- 172.21.15.16:0/2036308309 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a2339ae790 msgr2=0x55a2339b0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:36.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.443+0000 7f5bed16d640 1 --2- 172.21.15.16:0/2036308309 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a2339ae790 0x55a2339b0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.443+0000 7f5bed16d640 1 -- 172.21.15.16:0/2036308309 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2338c57c0 con 0x55a2339a5290 2024-03-28T22:12:36.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.443+0000 7f5bec96c640 1 --2- 172.21.15.16:0/2036308309 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a2339a5bb0 0x55a2339ae250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:12:36.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.443+0000 7f5be7fff640 1 --2- 172.21.15.16:0/2036308309 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a2339ae790 0x55a2339b0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:12:36.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.443+0000 7f5bed16d640 1 --2- 172.21.15.16:0/2036308309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2339a5290 0x55a2339a5670 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f5be0014e80 tx=0x7f5be00117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=560fad876655de8b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:36.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.443+0000 7f5bed16d640 1 --2- 172.21.15.16:0/1517173882 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f5bd80a1ef0 0x7f5bd80a43b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5be0011cb0 tx=0x7f5be0016000 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:36.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.443+0000 7f5bd2ffd640 1 -- 172.21.15.16:0/2036308309 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be0024070 con 0x55a2339a5290 2024-03-28T22:12:36.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.443+0000 7f5bd2ffd640 1 -- 172.21.15.16:0/2036308309 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5be0007c60 con 0x55a2339a5290 2024-03-28T22:12:36.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.443+0000 7f5bd2ffd640 1 -- 172.21.15.16:0/2036308309 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be00058b0 con 0x55a2339a5290 2024-03-28T22:12:36.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.444+0000 7f5bf17b2740 1 -- 172.21.15.16:0/2036308309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2339a5290 msgr2=0x55a2339a5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:36.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.444+0000 7f5bf17b2740 1 --2- 172.21.15.16:0/2036308309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2339a5290 0x55a2339a5670 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f5be0014e80 tx=0x7f5be00117b0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.444+0000 7f5bf17b2740 1 -- 172.21.15.16:0/2036308309 shutdown_connections 2024-03-28T22:12:36.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.444+0000 7f5bf17b2740 1 --2- 172.21.15.16:0/2036308309 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a2339ae790 0x55a2339b0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.444+0000 7f5bf17b2740 1 --2- 172.21.15.16:0/2036308309 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a2339a5bb0 0x55a2339ae250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.444+0000 7f5bf17b2740 1 --2- 172.21.15.16:0/2036308309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2339a5290 0x55a2339a5670 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.444+0000 7f5bf17b2740 1 -- 172.21.15.16:0/2036308309 >> 172.21.15.16:0/2036308309 conn(0x55a23399f9c0 msgr2=0x55a23399fda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:36.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.444+0000 7f5bf17b2740 1 -- 172.21.15.16:0/2036308309 shutdown_connections 2024-03-28T22:12:36.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.444+0000 7f5bf17b2740 1 -- 172.21.15.16:0/2036308309 wait complete. 2024-03-28T22:12:36.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.444+0000 7f5bf17b2740 1 -- start start 2024-03-28T22:12:36.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.445+0000 7f5bf17b2740 1 Processor -- start 2024-03-28T22:12:36.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.445+0000 7f5bf17b2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2339a5290 0x55a2339ab840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:36.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.445+0000 7f5bf17b2740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a2339a5bb0 0x55a2339abd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:36.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.445+0000 7f5bf17b2740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a2339ae790 0x55a2339ac2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:36.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.445+0000 7f5bf17b2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a2339b1520 con 0x55a2339a5290 2024-03-28T22:12:36.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.445+0000 7f5bf17b2740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55a2338a74a0 con 0x55a2339ae790 2024-03-28T22:12:36.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.445+0000 7f5bf17b2740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55a2339b16a0 con 0x55a2339a5bb0 2024-03-28T22:12:36.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.445+0000 7f5bf17b2740 10 client.0 _ll_register_callbacks cb 0x55a2338aad90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:12:36.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.445+0000 7f5bec96c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2339a5290 0x55a2339ab840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:36.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.445+0000 7f5bec96c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2339a5290 0x55a2339ab840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57366/0 (socket says 172.21.15.16:57366) 2024-03-28T22:12:36.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.445+0000 7f5bec96c640 1 -- 172.21.15.16:0/3525179572 learned_addr learned my addr 172.21.15.16:0/3525179572 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:36.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.445+0000 7f5bed16d640 1 --2- 172.21.15.16:0/3525179572 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a2339ae790 0x55a2339ac2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:36.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.445+0000 7f5be7fff640 1 --2- 172.21.15.16:0/3525179572 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a2339a5bb0 0x55a2339abd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:36.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.445+0000 7f5bec96c640 1 -- 172.21.15.16:0/3525179572 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a2339a5bb0 msgr2=0x55a2339abd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:36.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.445+0000 7f5bec96c640 1 --2- 172.21.15.16:0/3525179572 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a2339a5bb0 0x55a2339abd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.445+0000 7f5bec96c640 1 -- 172.21.15.16:0/3525179572 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a2339ae790 msgr2=0x55a2339ac2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:36.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.445+0000 7f5bec96c640 1 --2- 172.21.15.16:0/3525179572 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a2339ae790 0x55a2339ac2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.445+0000 7f5bec96c640 1 -- 172.21.15.16:0/3525179572 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2336e77d0 con 0x55a2339a5290 2024-03-28T22:12:36.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.446+0000 7f5bec96c640 1 --2- 172.21.15.16:0/3525179572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2339a5290 0x55a2339ab840 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc02cfd0 tx=0x7f5bdc005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c3521845784e9a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:36.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.446+0000 7f5bc7fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:12:36.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.446+0000 7f5bc7fff640 1 -- 172.21.15.16:0/3525179572 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bdc004380 con 0x55a2339a5290 2024-03-28T22:12:36.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.446+0000 7f5bc7fff640 1 -- 172.21.15.16:0/3525179572 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f5bdc0046c0 con 0x55a2339a5290 2024-03-28T22:12:36.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.446+0000 7f5bf17b2740 1 -- 172.21.15.16:0/3525179572 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5be0007c60 con 0x55a2339a5290 2024-03-28T22:12:36.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.446+0000 7f5bc7fff640 1 -- 172.21.15.16:0/3525179572 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5bdc0048c0 con 0x55a2339a5290 2024-03-28T22:12:36.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.446+0000 7f5bf17b2740 10 client.15972 fetch_fsmap learned FSMap version 24 2024-03-28T22:12:36.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.446+0000 7f5bf17b2740 1 -- 172.21.15.16:0/3525179572 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a23361d3f0 con 0x55a2339a5290 2024-03-28T22:12:36.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.446+0000 7f5bc7fff640 1 -- 172.21.15.16:0/3525179572 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5be0007c60 con 0x55a2339a5290 2024-03-28T22:12:36.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.446+0000 7f5bc7fff640 1 -- 172.21.15.16:0/3525179572 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7f5bdc04e070 con 0x55a2339a5290 2024-03-28T22:12:36.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.446+0000 7f5bf17b2740 10 client.15972 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:12:36.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.446+0000 7f5bf17b2740 10 client.15972 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:12:36.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.446+0000 7f5bf17b2740 20 client.15972 populate_metadata read hostname 'smithi016' 2024-03-28T22:12:36.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.446+0000 7f5bf17b2740 1 --2- 172.21.15.16:0/3525179572 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55a2339beb90 0x55a2339c1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:36.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.446+0000 7f5bf17b2740 4 client.15972 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:12:36.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.446+0000 7f5bf17b2740 1 -- 172.21.15.16:0/3525179572 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a2339166c0 con 0x55a2339beb90 2024-03-28T22:12:36.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.446+0000 7f5be7fff640 1 --2- 172.21.15.16:0/3525179572 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55a2339beb90 0x55a2339c1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:36.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.447+0000 7f5be7fff640 1 --2- 172.21.15.16:0/3525179572 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55a2339beb90 0x55a2339c1010 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8b7a53bfea6d5b8 server_cookie=f0de03363bd558d0 in_seq=0 out_seq=0 2024-03-28T22:12:36.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.447+0000 7f5bc7fff640 10 client.15972 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:12:36.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.448+0000 7f5bc7fff640 1 -- 172.21.15.16:0/3525179572 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a2339166c0 con 0x55a2339beb90 2024-03-28T22:12:36.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.448+0000 7f5bc7fff640 10 client.15972 handle_command_reply: tid=0 2024-03-28T22:12:36.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.448+0000 7f5bf17b2740 1 client.15972 shutdown 2024-03-28T22:12:36.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.448+0000 7f5bf17b2740 1 -- 172.21.15.16:0/3525179572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2339a5290 msgr2=0x55a2339ab840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:36.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.448+0000 7f5bf17b2740 1 --2- 172.21.15.16:0/3525179572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2339a5290 0x55a2339ab840 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc02cfd0 tx=0x7f5bdc005d50 comp rx=0 tx=0).stop 2024-03-28T22:12:36.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.448+0000 7f5bf17b2740 1 -- 172.21.15.16:0/3525179572 shutdown_connections 2024-03-28T22:12:36.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.448+0000 7f5bf17b2740 1 --2- 172.21.15.16:0/3525179572 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55a2339beb90 0x55a2339c1010 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.448+0000 7f5bf17b2740 1 --2- 172.21.15.16:0/3525179572 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a2339ae790 0x55a2339ac2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.448+0000 7f5bf17b2740 1 --2- 172.21.15.16:0/3525179572 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a2339a5bb0 0x55a2339abd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.448+0000 7f5bf17b2740 1 --2- 172.21.15.16:0/3525179572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2339a5290 0x55a2339ab840 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.448+0000 7f5bc7fff640 0 client.15972 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:12:36.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.448+0000 7f5bf17b2740 1 -- 172.21.15.16:0/3525179572 >> 172.21.15.16:0/3525179572 conn(0x55a23399f9c0 msgr2=0x55a2339a1710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:36.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.448+0000 7f5bf17b2740 1 -- 172.21.15.16:0/3525179572 shutdown_connections 2024-03-28T22:12:36.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.448+0000 7f5bf17b2740 1 -- 172.21.15.16:0/3525179572 wait complete. 2024-03-28T22:12:36.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.448+0000 7f5bf17b2740 20 client.15972 trim_cache size 0 max 16384 2024-03-28T22:12:36.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.467+0000 7f5bf17b2740 1 Processor -- start 2024-03-28T22:12:36.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.467+0000 7f5bf17b2740 1 -- start start 2024-03-28T22:12:36.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.468+0000 7f5bf17b2740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a2339a5290 0x55a2339adf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:36.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.468+0000 7f5bf17b2740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a2339a5bb0 0x55a2339ba2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:36.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.468+0000 7f5bf17b2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2339ae790 0x55a23399eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:36.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.468+0000 7f5bf17b2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5be81afc10 con 0x55a2339ae790 2024-03-28T22:12:36.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.468+0000 7f5bf17b2740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5be81ad4b0 con 0x55a2339a5bb0 2024-03-28T22:12:36.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.468+0000 7f5bf17b2740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5be81ab320 con 0x55a2339a5290 2024-03-28T22:12:36.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.468+0000 7f5bed16d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2339ae790 0x55a23399eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:36.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.468+0000 7f5bed16d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2339ae790 0x55a23399eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57376/0 (socket says 172.21.15.16:57376) 2024-03-28T22:12:36.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.468+0000 7f5bed16d640 1 -- 172.21.15.16:0/540235507 learned_addr learned my addr 172.21.15.16:0/540235507 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:36.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.468+0000 7f5be7fff640 1 --2- 172.21.15.16:0/540235507 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a2339a5bb0 0x55a2339ba2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:36.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.468+0000 7f5bec96c640 1 --2- 172.21.15.16:0/540235507 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a2339a5290 0x55a2339adf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:36.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.468+0000 7f5bed16d640 1 -- 172.21.15.16:0/540235507 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a2339a5290 msgr2=0x55a2339adf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:36.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.468+0000 7f5bed16d640 1 --2- 172.21.15.16:0/540235507 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a2339a5290 0x55a2339adf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.468+0000 7f5bed16d640 1 -- 172.21.15.16:0/540235507 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a2339a5bb0 msgr2=0x55a2339ba2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:36.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.468+0000 7f5bed16d640 1 --2- 172.21.15.16:0/540235507 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a2339a5bb0 0x55a2339ba2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.468+0000 7f5bed16d640 1 -- 172.21.15.16:0/540235507 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2339a59e0 con 0x55a2339ae790 2024-03-28T22:12:36.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.468+0000 7f5bed16d640 1 --2- 172.21.15.16:0/540235507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2339ae790 0x55a23399eaa0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f5be0014680 tx=0x7f5be0005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bf1339ce12396e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:36.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.468+0000 7f5bd27fc640 1 -- 172.21.15.16:0/540235507 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be0024040 con 0x55a2339ae790 2024-03-28T22:12:36.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.468+0000 7f5bd27fc640 1 -- 172.21.15.16:0/540235507 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5be00050b0 con 0x55a2339ae790 2024-03-28T22:12:36.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.468+0000 7f5bd27fc640 1 -- 172.21.15.16:0/540235507 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be0003bd0 con 0x55a2339ae790 2024-03-28T22:12:36.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.469+0000 7f5bf17b2740 1 -- 172.21.15.16:0/540235507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2339ae790 msgr2=0x55a23399eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:36.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.469+0000 7f5bf17b2740 1 --2- 172.21.15.16:0/540235507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2339ae790 0x55a23399eaa0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f5be0014680 tx=0x7f5be0005040 comp rx=0 tx=0).stop 2024-03-28T22:12:36.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.469+0000 7f5bf17b2740 1 -- 172.21.15.16:0/540235507 shutdown_connections 2024-03-28T22:12:36.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.469+0000 7f5bf17b2740 1 --2- 172.21.15.16:0/540235507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2339ae790 0x55a23399eaa0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.469+0000 7f5bf17b2740 1 --2- 172.21.15.16:0/540235507 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a2339a5bb0 0x55a2339ba2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.469+0000 7f5bf17b2740 1 --2- 172.21.15.16:0/540235507 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a2339a5290 0x55a2339adf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.469+0000 7f5bf17b2740 1 -- 172.21.15.16:0/540235507 >> 172.21.15.16:0/540235507 conn(0x55a23399f9c0 msgr2=0x55a23399e390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:36.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.469+0000 7f5bf17b2740 1 -- 172.21.15.16:0/540235507 shutdown_connections 2024-03-28T22:12:36.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.469+0000 7f5bf17b2740 1 -- 172.21.15.16:0/540235507 wait complete. 2024-03-28T22:12:36.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.470+0000 7f5bf17b2740 1 -- start start 2024-03-28T22:12:36.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.470+0000 7f5bf17b2740 1 Processor -- start 2024-03-28T22:12:36.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.470+0000 7f5bf17b2740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a2339a5290 0x55a2339a89e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:36.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.470+0000 7f5bf17b2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2339a5bb0 0x55a2339a8f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:36.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.470+0000 7f5bf17b2740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a2339ae790 0x55a233a23f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:36.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.470+0000 7f5bf17b2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5be81ab970 con 0x55a2339a5bb0 2024-03-28T22:12:36.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.470+0000 7f5bf17b2740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5be81acde0 con 0x55a2339ae790 2024-03-28T22:12:36.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.470+0000 7f5bf17b2740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5be81ac040 con 0x55a2339a5290 2024-03-28T22:12:36.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.470+0000 7f5bf17b2740 10 client.0 _ll_register_callbacks cb 0x7f5bdc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:12:36.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.470+0000 7f5be7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2339a5bb0 0x55a2339a8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:36.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.470+0000 7f5be7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2339a5bb0 0x55a2339a8f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57380/0 (socket says 172.21.15.16:57380) 2024-03-28T22:12:36.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.470+0000 7f5be7fff640 1 -- 172.21.15.16:0/2134281555 learned_addr learned my addr 172.21.15.16:0/2134281555 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:12:36.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.470+0000 7f5bec96c640 1 --2- 172.21.15.16:0/2134281555 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a2339a5290 0x55a2339a89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:36.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.470+0000 7f5bed16d640 1 --2- 172.21.15.16:0/2134281555 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a2339ae790 0x55a233a23f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:36.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.471+0000 7f5be7fff640 1 -- 172.21.15.16:0/2134281555 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a2339a5290 msgr2=0x55a2339a89e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:36.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.471+0000 7f5be7fff640 1 --2- 172.21.15.16:0/2134281555 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a2339a5290 0x55a2339a89e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.471+0000 7f5be7fff640 1 -- 172.21.15.16:0/2134281555 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a2339ae790 msgr2=0x55a233a23f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:12:36.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.471+0000 7f5be7fff640 1 --2- 172.21.15.16:0/2134281555 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a2339ae790 0x55a233a23f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.471+0000 7f5be7fff640 1 -- 172.21.15.16:0/2134281555 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2335dd0f0 con 0x55a2339a5bb0 2024-03-28T22:12:36.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.471+0000 7f5be7fff640 1 --2- 172.21.15.16:0/2134281555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2339a5bb0 0x55a2339a8f20 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f5bd40973d0 tx=0x7f5bd40ae040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a72e8ca3257a94a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:12:36.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.471+0000 7f5bc7fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:12:36.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.471+0000 7f5bc7fff640 1 -- 172.21.15.16:0/2134281555 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd40cbea0 con 0x55a2339a5bb0 2024-03-28T22:12:36.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.471+0000 7f5bc7fff640 1 -- 172.21.15.16:0/2134281555 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f5bd40b92f0 con 0x55a2339a5bb0 2024-03-28T22:12:36.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.471+0000 7f5bf17b2740 1 -- 172.21.15.16:0/2134281555 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5be00050b0 con 0x55a2339a5bb0 2024-03-28T22:12:36.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.471+0000 7f5bc7fff640 1 -- 172.21.15.16:0/2134281555 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5bd40975e0 con 0x55a2339a5bb0 2024-03-28T22:12:36.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.471+0000 7f5bf17b2740 10 client.15978 fetch_fsmap learned FSMap version 24 2024-03-28T22:12:36.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.471+0000 7f5bf17b2740 1 -- 172.21.15.16:0/2134281555 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a233a25490 con 0x55a2339a5bb0 2024-03-28T22:12:36.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.471+0000 7f5bc7fff640 1 -- 172.21.15.16:0/2134281555 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5be00050b0 con 0x55a2339a5bb0 2024-03-28T22:12:36.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.472+0000 7f5bc7fff640 1 -- 172.21.15.16:0/2134281555 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7f5bd4018b30 con 0x55a2339a5bb0 2024-03-28T22:12:36.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.472+0000 7f5bf17b2740 10 client.15978 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:12:36.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.472+0000 7f5bf17b2740 10 client.15978 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:12:36.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.472+0000 7f5bf17b2740 20 client.15978 populate_metadata read hostname 'smithi016' 2024-03-28T22:12:36.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.472+0000 7f5bf17b2740 1 --2- 172.21.15.16:0/2134281555 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55a233a25b60 0x55a233a27f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:12:36.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.472+0000 7f5bf17b2740 4 client.15978 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:12:36.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.472+0000 7f5bf17b2740 1 -- 172.21.15.16:0/2134281555 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a233910a10 con 0x55a233a25b60 2024-03-28T22:12:36.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.472+0000 7f5bec96c640 1 --2- 172.21.15.16:0/2134281555 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55a233a25b60 0x55a233a27f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:12:36.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.472+0000 7f5bc7fff640 10 client.15978 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:12:36.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.472+0000 7f5bec96c640 1 --2- 172.21.15.16:0/2134281555 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55a233a25b60 0x55a233a27f80 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=710c92ea1cd3bd7 server_cookie=6be55c858ea6c602 in_seq=0 out_seq=0 2024-03-28T22:12:36.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.473+0000 7f5bc7fff640 1 -- 172.21.15.16:0/2134281555 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55a233910a10 con 0x55a233a25b60 2024-03-28T22:12:36.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.473+0000 7f5bc7fff640 10 client.15978 handle_command_reply: tid=0 2024-03-28T22:12:36.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.473+0000 7f5bf17b2740 1 client.15978 shutdown 2024-03-28T22:12:36.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.473+0000 7f5bf17b2740 1 -- 172.21.15.16:0/2134281555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2339a5bb0 msgr2=0x55a2339a8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:36.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.473+0000 7f5bf17b2740 1 --2- 172.21.15.16:0/2134281555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2339a5bb0 0x55a2339a8f20 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f5bd40973d0 tx=0x7f5bd40ae040 comp rx=0 tx=0).stop 2024-03-28T22:12:36.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.473+0000 7f5bf17b2740 1 -- 172.21.15.16:0/2134281555 shutdown_connections 2024-03-28T22:12:36.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.473+0000 7f5bf17b2740 1 --2- 172.21.15.16:0/2134281555 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55a233a25b60 0x55a233a27f80 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.473+0000 7f5bf17b2740 1 --2- 172.21.15.16:0/2134281555 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a2339ae790 0x55a233a23f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.473+0000 7f5bf17b2740 1 --2- 172.21.15.16:0/2134281555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2339a5bb0 0x55a2339a8f20 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.473+0000 7f5bf17b2740 1 --2- 172.21.15.16:0/2134281555 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a2339a5290 0x55a2339a89e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.473+0000 7f5bf17b2740 1 -- 172.21.15.16:0/2134281555 >> 172.21.15.16:0/2134281555 conn(0x55a23399f9c0 msgr2=0x55a23399ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:36.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.473+0000 7f5bc7fff640 0 client.15978 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:12:36.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.473+0000 7f5bf17b2740 1 -- 172.21.15.16:0/2134281555 shutdown_connections 2024-03-28T22:12:36.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.473+0000 7f5bf17b2740 1 -- 172.21.15.16:0/2134281555 wait complete. 2024-03-28T22:12:36.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.473+0000 7f5bf17b2740 20 client.15978 trim_cache size 0 max 16384 2024-03-28T22:12:36.482 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:12:36.482 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T22:12:36.482 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-03-28T22:12:36.482 INFO:teuthology.orchestra.run.smithi016.stdout: "f1d3d6a4-0dc0-49e3-b4ef-5909a37808de": { 2024-03-28T22:12:36.482 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-03-28T22:12:36.482 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "f1d3d6a4-0dc0-49e3-b4ef-5909a37808de", 2024-03-28T22:12:36.482 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-03-28T22:12:36.483 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:12:36.483 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:12:36.483 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:12:36.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.474+0000 7f5beebd0640 1 -- 172.21.15.16:0/1517173882 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f5bd80a1ef0 msgr2=0x7f5bd80a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:36.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.474+0000 7f5beebd0640 1 --2- 172.21.15.16:0/1517173882 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f5bd80a1ef0 0x7f5bd80a43b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5be0011cb0 tx=0x7f5be0016000 comp rx=0 tx=0).stop 2024-03-28T22:12:36.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.474+0000 7f5beebd0640 1 -- 172.21.15.16:0/1517173882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5be8100fa0 msgr2=0x7f5be8199280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:12:36.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.474+0000 7f5beebd0640 1 --2- 172.21.15.16:0/1517173882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5be8100fa0 0x7f5be8199280 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f5bd400e9f0 tx=0x7f5bd400eeb0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.474+0000 7f5beebd0640 1 -- 172.21.15.16:0/1517173882 shutdown_connections 2024-03-28T22:12:36.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.474+0000 7f5beebd0640 1 --2- 172.21.15.16:0/1517173882 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f5bd80a1ef0 0x7f5bd80a43b0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.474+0000 7f5beebd0640 1 --2- 172.21.15.16:0/1517173882 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5be8106680 0x7f5be81a3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.474+0000 7f5beebd0640 1 --2- 172.21.15.16:0/1517173882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5be8100fa0 0x7f5be8199280 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.474+0000 7f5beebd0640 1 --2- 172.21.15.16:0/1517173882 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5be81005e0 0x7f5be8198d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:12:36.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.474+0000 7f5beebd0640 1 -- 172.21.15.16:0/1517173882 >> 172.21.15.16:0/1517173882 conn(0x7f5be8077770 msgr2=0x7f5be80ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:12:36.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.474+0000 7f5beebd0640 1 -- 172.21.15.16:0/1517173882 shutdown_connections 2024-03-28T22:12:36.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:12:36.474+0000 7f5beebd0640 1 -- 172.21.15.16:0/1517173882 wait complete. 2024-03-28T22:12:36.489 INFO:tasks.cephfs.filesystem:scrub status for tag:f1d3d6a4-0dc0-49e3-b4ef-5909a37808de - {'path': '/', 'tag': 'f1d3d6a4-0dc0-49e3-b4ef-5909a37808de', 'options': 'recursive,force'} 2024-03-28T22:12:37.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:37 smithi053 ceph-mon[35979]: scrub queued for path: / 2024-03-28T22:12:37.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:37 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:12:37.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:37 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:12:37.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:37 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:12:37.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:37 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:12:37.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:37 smithi053 ceph-mon[35979]: pgmap v64: 97 pgs: 97 active+clean; 83 MiB data, 680 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 273 op/s 2024-03-28T22:12:37.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:37 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:12:37.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:37 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:12:37.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:37 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:12:37.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:37 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:12:37.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:37 smithi063 ceph-mon[36249]: scrub queued for path: / 2024-03-28T22:12:37.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:37 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:12:37.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:37 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:12:37.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:37 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:12:37.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:37 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:12:37.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:37 smithi063 ceph-mon[36249]: pgmap v64: 97 pgs: 97 active+clean; 83 MiB data, 680 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 273 op/s 2024-03-28T22:12:37.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:37 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:12:37.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:37 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:12:37.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:37 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:12:37.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:37 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:12:37.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:37 smithi016 ceph-mon[32374]: scrub queued for path: / 2024-03-28T22:12:37.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:37 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:12:37.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:37 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:12:37.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:37 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:12:37.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:37 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:12:37.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:37 smithi016 ceph-mon[32374]: pgmap v64: 97 pgs: 97 active+clean; 83 MiB data, 680 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 273 op/s 2024-03-28T22:12:37.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:37 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:12:37.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:37 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:12:37.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:37 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:12:37.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:37 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:12:38.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:38 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:12:38.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:38 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:12:38.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:38 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:12:39.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:39 smithi053 ceph-mon[35979]: pgmap v65: 97 pgs: 97 active+clean; 83 MiB data, 680 MiB used, 1.0 TiB / 1.0 TiB avail; 774 KiB/s rd, 856 KiB/s wr, 254 op/s 2024-03-28T22:12:39.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:39 smithi063 ceph-mon[36249]: pgmap v65: 97 pgs: 97 active+clean; 83 MiB data, 680 MiB used, 1.0 TiB / 1.0 TiB avail; 774 KiB/s rd, 856 KiB/s wr, 254 op/s 2024-03-28T22:12:39.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:39 smithi016 ceph-mon[32374]: pgmap v65: 97 pgs: 97 active+clean; 83 MiB data, 680 MiB used, 1.0 TiB / 1.0 TiB avail; 774 KiB/s rd, 856 KiB/s wr, 254 op/s 2024-03-28T22:12:40.688 INFO:tasks.workunit.client.0.smithi016.stdout:checking dynamic linker characteristics... GNU/Linux ld.so 2024-03-28T22:12:40.688 INFO:tasks.workunit.client.0.smithi016.stdout:checking how to hardcode library paths into programs... immediate 2024-03-28T22:12:40.690 INFO:tasks.workunit.client.0.smithi016.stdout:checking whether stripping libraries is possible... yes 2024-03-28T22:12:40.690 INFO:tasks.workunit.client.0.smithi016.stdout:checking if libtool supports shared libraries... yes 2024-03-28T22:12:40.690 INFO:tasks.workunit.client.0.smithi016.stdout:checking whether to build shared libraries... yes 2024-03-28T22:12:40.690 INFO:tasks.workunit.client.0.smithi016.stdout:checking whether to build static libraries... yes 2024-03-28T22:12:40.698 INFO:tasks.workunit.client.0.smithi016.stdout:checking for gcc... (cached) gcc 2024-03-28T22:12:40.750 INFO:tasks.workunit.client.0.smithi016.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-03-28T22:12:40.750 INFO:tasks.workunit.client.0.smithi016.stdout:checking whether gcc accepts -g... (cached) yes 2024-03-28T22:12:40.750 INFO:tasks.workunit.client.0.smithi016.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-03-28T22:12:40.750 INFO:tasks.workunit.client.0.smithi016.stdout:checking for gmake... /usr/bin/gmake 2024-03-28T22:12:40.753 INFO:tasks.workunit.client.0.smithi016.stderr:./configure: line 11968: test: $(SHELL): binary operator expected 2024-03-28T22:12:40.753 INFO:tasks.workunit.client.0.smithi016.stdout:checking for tar... /bin/tar 2024-03-28T22:12:40.754 INFO:tasks.workunit.client.0.smithi016.stdout:checking for gzip... /bin/gzip 2024-03-28T22:12:40.782 INFO:tasks.workunit.client.0.smithi016.stdout:checking whether gcc -MM is supported... yes 2024-03-28T22:12:40.782 INFO:tasks.workunit.client.0.smithi016.stdout:checking for sort... /bin/sort 2024-03-28T22:12:40.782 INFO:tasks.workunit.client.0.smithi016.stdout:checking whether ln -s works... yes 2024-03-28T22:12:40.782 INFO:tasks.workunit.client.0.smithi016.stdout:checking for rpm... /bin/rpm 2024-03-28T22:12:40.786 INFO:tasks.workunit.client.0.smithi016.stdout:checking for rpmbuild... no 2024-03-28T22:12:40.786 INFO:tasks.workunit.client.0.smithi016.stdout:checking for ANSI C header files... (cached) yes 2024-03-28T22:12:40.849 INFO:tasks.workunit.client.0.smithi016.stdout:checking assert.h usability... yes 2024-03-28T22:12:40.873 INFO:tasks.workunit.client.0.smithi016.stdout:checking assert.h presence... yes 2024-03-28T22:12:40.873 INFO:tasks.workunit.client.0.smithi016.stdout:checking for assert.h... yes 2024-03-28T22:12:40.938 INFO:tasks.workunit.client.0.smithi016.stdout:checking bstring.h usability... no 2024-03-28T22:12:40.979 INFO:tasks.workunit.client.0.smithi016.stdout:checking bstring.h presence... no 2024-03-28T22:12:40.980 INFO:tasks.workunit.client.0.smithi016.stdout:checking for bstring.h... no 2024-03-28T22:12:41.023 INFO:tasks.workunit.client.0.smithi016.stdout:checking libgen.h usability... yes 2024-03-28T22:12:41.040 INFO:tasks.workunit.client.0.smithi016.stdout:checking libgen.h presence... yes 2024-03-28T22:12:41.041 INFO:tasks.workunit.client.0.smithi016.stdout:checking for libgen.h... yes 2024-03-28T22:12:41.100 INFO:tasks.workunit.client.0.smithi016.stdout:checking dirent.h usability... yes 2024-03-28T22:12:41.123 INFO:tasks.workunit.client.0.smithi016.stdout:checking dirent.h presence... yes 2024-03-28T22:12:41.124 INFO:tasks.workunit.client.0.smithi016.stdout:checking for dirent.h... yes 2024-03-28T22:12:41.171 INFO:tasks.workunit.client.0.smithi016.stdout:checking errno.h usability... yes 2024-03-28T22:12:41.187 INFO:tasks.workunit.client.0.smithi016.stdout:checking errno.h presence... yes 2024-03-28T22:12:41.187 INFO:tasks.workunit.client.0.smithi016.stdout:checking for errno.h... yes 2024-03-28T22:12:41.243 INFO:tasks.workunit.client.0.smithi016.stdout:checking malloc.h usability... yes 2024-03-28T22:12:41.261 INFO:tasks.workunit.client.0.smithi016.stdout:checking malloc.h presence... yes 2024-03-28T22:12:41.261 INFO:tasks.workunit.client.0.smithi016.stdout:checking for malloc.h... yes 2024-03-28T22:12:41.320 INFO:tasks.workunit.client.0.smithi016.stdout:checking uuid.h usability... no 2024-03-28T22:12:41.362 INFO:tasks.workunit.client.0.smithi016.stdout:checking uuid.h presence... no 2024-03-28T22:12:41.362 INFO:tasks.workunit.client.0.smithi016.stdout:checking for uuid.h... no 2024-03-28T22:12:41.411 INFO:tasks.workunit.client.0.smithi016.stdout:checking uuid/uuid.h usability... yes 2024-03-28T22:12:41.434 INFO:tasks.workunit.client.0.smithi016.stdout:checking uuid/uuid.h presence... yes 2024-03-28T22:12:41.435 INFO:tasks.workunit.client.0.smithi016.stdout:checking for uuid/uuid.h... yes 2024-03-28T22:12:41.495 INFO:tasks.workunit.client.0.smithi016.stdout:checking sys/uuid.h usability... no 2024-03-28T22:12:41.528 INFO:tasks.workunit.client.0.smithi016.stdout:checking sys/uuid.h presence... no 2024-03-28T22:12:41.529 INFO:tasks.workunit.client.0.smithi016.stdout:checking for sys/uuid.h... no 2024-03-28T22:12:41.576 INFO:tasks.workunit.client.0.smithi016.stdout:checking sys/file.h usability... yes 2024-03-28T22:12:41.599 INFO:tasks.workunit.client.0.smithi016.stdout:checking sys/file.h presence... yes 2024-03-28T22:12:41.599 INFO:tasks.workunit.client.0.smithi016.stdout:checking for sys/file.h... yes 2024-03-28T22:12:41.664 INFO:tasks.workunit.client.0.smithi016.stdout:checking sys/fcntl.h usability... yes 2024-03-28T22:12:41.686 INFO:tasks.workunit.client.0.smithi016.stdout:checking sys/fcntl.h presence... yes 2024-03-28T22:12:41.686 INFO:tasks.workunit.client.0.smithi016.stdout:checking for sys/fcntl.h... yes 2024-03-28T22:12:41.749 INFO:tasks.workunit.client.0.smithi016.stdout:checking sys/syssgi.h usability... no 2024-03-28T22:12:41.779 INFO:tasks.workunit.client.0.smithi016.stdout:checking sys/syssgi.h presence... no 2024-03-28T22:12:41.779 INFO:tasks.workunit.client.0.smithi016.stdout:checking for sys/syssgi.h... no 2024-03-28T22:12:41.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:41 smithi053 ceph-mon[35979]: pgmap v66: 97 pgs: 97 active+clean; 83 MiB data, 680 MiB used, 1.0 TiB / 1.0 TiB avail; 774 KiB/s rd, 856 KiB/s wr, 254 op/s 2024-03-28T22:12:41.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:41 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:12:41.890 INFO:tasks.workunit.client.0.smithi016.stdout:checking sys/param.h usability... yes 2024-03-28T22:12:41.916 INFO:tasks.workunit.client.0.smithi016.stdout:checking sys/param.h presence... yes 2024-03-28T22:12:41.917 INFO:tasks.workunit.client.0.smithi016.stdout:checking for sys/param.h... yes 2024-03-28T22:12:41.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:41 smithi063 ceph-mon[36249]: pgmap v66: 97 pgs: 97 active+clean; 83 MiB data, 680 MiB used, 1.0 TiB / 1.0 TiB avail; 774 KiB/s rd, 856 KiB/s wr, 254 op/s 2024-03-28T22:12:41.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:41 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:12:41.934 INFO:tasks.workunit.client.0.smithi016.stdout:checking for sys/stat.h... (cached) yes 2024-03-28T22:12:41.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:41 smithi016 ceph-mon[32374]: pgmap v66: 97 pgs: 97 active+clean; 83 MiB data, 680 MiB used, 1.0 TiB / 1.0 TiB avail; 774 KiB/s rd, 856 KiB/s wr, 254 op/s 2024-03-28T22:12:41.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:41 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:12:41.990 INFO:tasks.workunit.client.0.smithi016.stdout:checking sys/statvfs.h usability... yes 2024-03-28T22:12:42.014 INFO:tasks.workunit.client.0.smithi016.stdout:checking sys/statvfs.h presence... yes 2024-03-28T22:12:42.014 INFO:tasks.workunit.client.0.smithi016.stdout:checking for sys/statvfs.h... yes 2024-03-28T22:12:42.071 INFO:tasks.workunit.client.0.smithi016.stdout:checking sys/time.h usability... yes 2024-03-28T22:12:42.095 INFO:tasks.workunit.client.0.smithi016.stdout:checking sys/time.h presence... yes 2024-03-28T22:12:42.095 INFO:tasks.workunit.client.0.smithi016.stdout:checking for sys/time.h... yes 2024-03-28T22:12:42.151 INFO:tasks.workunit.client.0.smithi016.stdout:checking sys/ioctl.h usability... yes 2024-03-28T22:12:42.168 INFO:tasks.workunit.client.0.smithi016.stdout:checking sys/ioctl.h presence... yes 2024-03-28T22:12:42.168 INFO:tasks.workunit.client.0.smithi016.stdout:checking for sys/ioctl.h... yes 2024-03-28T22:12:42.226 INFO:tasks.workunit.client.0.smithi016.stdout:checking sys/wait.h usability... yes 2024-03-28T22:12:42.251 INFO:tasks.workunit.client.0.smithi016.stdout:checking sys/wait.h presence... yes 2024-03-28T22:12:42.252 INFO:tasks.workunit.client.0.smithi016.stdout:checking for sys/wait.h... yes 2024-03-28T22:12:42.274 INFO:tasks.workunit.client.0.smithi016.stdout:checking for sys/types.h... (cached) yes 2024-03-28T22:12:42.278 INFO:tasks.workunit.client.0.smithi016.stdout:checking for strings.h... (cached) yes 2024-03-28T22:12:42.319 INFO:tasks.workunit.client.0.smithi016.stdout:checking err.h usability... yes 2024-03-28T22:12:42.335 INFO:tasks.workunit.client.0.smithi016.stdout:checking err.h presence... yes 2024-03-28T22:12:42.335 INFO:tasks.workunit.client.0.smithi016.stdout:checking for err.h... yes 2024-03-28T22:12:42.386 INFO:tasks.workunit.client.0.smithi016.stdout:checking linux/falloc.h usability... yes 2024-03-28T22:12:42.408 INFO:tasks.workunit.client.0.smithi016.stdout:checking linux/falloc.h presence... yes 2024-03-28T22:12:42.408 INFO:tasks.workunit.client.0.smithi016.stdout:checking for linux/falloc.h... yes 2024-03-28T22:12:42.475 INFO:tasks.workunit.client.0.smithi016.stdout:checking sys/fs/xfs_fsops.h usability... no 2024-03-28T22:12:42.515 INFO:tasks.workunit.client.0.smithi016.stdout:checking sys/fs/xfs_fsops.h presence... no 2024-03-28T22:12:42.516 INFO:tasks.workunit.client.0.smithi016.stdout:checking for sys/fs/xfs_fsops.h... no 2024-03-28T22:12:42.567 INFO:tasks.workunit.client.0.smithi016.stdout:checking sys/fs/xfs_itable.h usability... no 2024-03-28T22:12:42.611 INFO:tasks.workunit.client.0.smithi016.stdout:checking sys/fs/xfs_itable.h presence... no 2024-03-28T22:12:42.611 INFO:tasks.workunit.client.0.smithi016.stdout:checking for sys/fs/xfs_itable.h... no 2024-03-28T22:12:42.661 INFO:tasks.workunit.client.0.smithi016.stdout:checking xfs/platform_defs.h usability... no 2024-03-28T22:12:42.696 INFO:tasks.workunit.client.0.smithi016.stdout:checking xfs/platform_defs.h presence... no 2024-03-28T22:12:42.697 INFO:tasks.workunit.client.0.smithi016.stdout:checking for xfs/platform_defs.h... no 2024-03-28T22:12:42.743 INFO:tasks.workunit.client.0.smithi016.stdout:checking btrfs/ioctl.h usability... no 2024-03-28T22:12:42.778 INFO:tasks.workunit.client.0.smithi016.stdout:checking btrfs/ioctl.h presence... no 2024-03-28T22:12:42.778 INFO:tasks.workunit.client.0.smithi016.stdout:checking for btrfs/ioctl.h... no 2024-03-28T22:12:42.830 INFO:tasks.workunit.client.0.smithi016.stdout:checking cifs/ioctl.h usability... no 2024-03-28T22:12:42.867 INFO:tasks.workunit.client.0.smithi016.stdout:checking cifs/ioctl.h presence... no 2024-03-28T22:12:42.867 INFO:tasks.workunit.client.0.smithi016.stdout:checking for cifs/ioctl.h... no 2024-03-28T22:12:42.911 INFO:tasks.workunit.client.0.smithi016.stdout:checking sys/mman.h usability... yes 2024-03-28T22:12:42.928 INFO:tasks.workunit.client.0.smithi016.stdout:checking sys/mman.h presence... yes 2024-03-28T22:12:42.928 INFO:tasks.workunit.client.0.smithi016.stdout:checking for sys/mman.h... yes 2024-03-28T22:12:42.975 INFO:tasks.workunit.client.0.smithi016.stdout:checking for xfs/xfs_log_format.h... no 2024-03-28T22:12:43.039 INFO:tasks.workunit.client.0.smithi016.stdout:checking for uuid_compare... no 2024-03-28T22:12:43.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:42 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:12:43.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:42 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:12:43.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:42 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:12:43.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:42 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:12:43.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:42 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:12:43.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:42 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:12:43.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:42 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:12:43.084 INFO:tasks.workunit.client.0.smithi016.stdout:checking for uuid_compare in -luuid... yes 2024-03-28T22:12:43.157 INFO:tasks.workunit.client.0.smithi016.stdout:checking for xfs/xfs.h... yes 2024-03-28T22:12:43.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:42 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:12:43.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:42 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:12:43.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:42 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:12:43.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:42 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:12:43.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:42 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:12:43.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:42 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:12:43.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:42 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:12:43.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:42 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:12:43.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:42 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:12:43.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:42 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:12:43.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:42 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:12:43.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:42 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:12:43.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:42 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:12:43.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:42 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:12:43.208 INFO:tasks.workunit.client.0.smithi016.stdout:checking for xfs/libxfs.h... no 2024-03-28T22:12:43.247 INFO:tasks.workunit.client.0.smithi016.stdout:checking whether xlog_assign_lsn is declared... no 2024-03-28T22:12:43.284 INFO:tasks.workunit.client.0.smithi016.stdout:checking for xfs/xqm.h... yes 2024-03-28T22:12:43.338 INFO:tasks.workunit.client.0.smithi016.stdout:checking xfsctl from xfs/xfs.h... ok 2024-03-28T22:12:43.378 INFO:tasks.workunit.client.0.smithi016.stdout:checking xfs/handle.h usability... yes 2024-03-28T22:12:43.398 INFO:tasks.workunit.client.0.smithi016.stdout:checking xfs/handle.h presence... yes 2024-03-28T22:12:43.398 INFO:tasks.workunit.client.0.smithi016.stdout:checking for xfs/handle.h... yes 2024-03-28T22:12:43.444 INFO:tasks.workunit.client.0.smithi016.stdout:checking for attr_list_by_handle in -lhandle... yes 2024-03-28T22:12:43.529 INFO:tasks.workunit.client.0.smithi016.stdout:checking attr/attributes.h usability... yes 2024-03-28T22:12:43.546 INFO:tasks.workunit.client.0.smithi016.stdout:checking attr/attributes.h presence... yes 2024-03-28T22:12:43.547 INFO:tasks.workunit.client.0.smithi016.stdout:checking for attr/attributes.h... yes 2024-03-28T22:12:43.606 INFO:tasks.workunit.client.0.smithi016.stdout:checking sys/attributes.h usability... no 2024-03-28T22:12:43.649 INFO:tasks.workunit.client.0.smithi016.stdout:checking sys/attributes.h presence... no 2024-03-28T22:12:43.650 INFO:tasks.workunit.client.0.smithi016.stdout:checking for sys/attributes.h... no 2024-03-28T22:12:43.695 INFO:tasks.workunit.client.0.smithi016.stdout:checking for attr_list in -lattr... yes 2024-03-28T22:12:43.735 INFO:tasks.workunit.client.0.smithi016.stdout:checking for attr_set in -lattr... yes 2024-03-28T22:12:44.137 INFO:tasks.workunit.client.0.smithi016.stdout:checking sys/acl.h usability... yes 2024-03-28T22:12:44.158 INFO:tasks.workunit.client.0.smithi016.stdout:checking sys/acl.h presence... yes 2024-03-28T22:12:44.158 INFO:tasks.workunit.client.0.smithi016.stdout:checking for sys/acl.h... yes 2024-03-28T22:12:44.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:43 smithi016 ceph-mon[32374]: pgmap v67: 97 pgs: 97 active+clean; 83 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 861 KiB/s wr, 258 op/s 2024-03-28T22:12:44.192 INFO:tasks.workunit.client.0.smithi016.stdout:checking acl/libacl.h usability... yes 2024-03-28T22:12:44.214 INFO:tasks.workunit.client.0.smithi016.stdout:checking acl/libacl.h presence... yes 2024-03-28T22:12:44.214 INFO:tasks.workunit.client.0.smithi016.stdout:checking for acl/libacl.h... yes 2024-03-28T22:12:44.261 INFO:tasks.workunit.client.0.smithi016.stdout:checking for acl_init in -lacl... yes 2024-03-28T22:12:44.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:43 smithi053 ceph-mon[35979]: pgmap v67: 97 pgs: 97 active+clean; 83 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 861 KiB/s wr, 258 op/s 2024-03-28T22:12:44.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:43 smithi063 ceph-mon[36249]: pgmap v67: 97 pgs: 97 active+clean; 83 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 861 KiB/s wr, 258 op/s 2024-03-28T22:12:44.476 INFO:tasks.workunit.client.0.smithi016.stdout:checking gdbm-ndbm.h usability... no 2024-03-28T22:12:44.519 INFO:tasks.workunit.client.0.smithi016.stdout:checking gdbm-ndbm.h presence... no 2024-03-28T22:12:44.519 INFO:tasks.workunit.client.0.smithi016.stdout:checking for gdbm-ndbm.h... no 2024-03-28T22:12:44.568 INFO:tasks.workunit.client.0.smithi016.stdout:checking gdbm/ndbm.h usability... no 2024-03-28T22:12:44.624 INFO:tasks.workunit.client.0.smithi016.stdout:checking gdbm/ndbm.h presence... no 2024-03-28T22:12:44.624 INFO:tasks.workunit.client.0.smithi016.stdout:checking for gdbm/ndbm.h... no 2024-03-28T22:12:44.728 INFO:tasks.workunit.client.0.smithi016.stdout:checking ndbm.h usability... no 2024-03-28T22:12:44.766 INFO:tasks.workunit.client.0.smithi016.stdout:checking ndbm.h presence... no 2024-03-28T22:12:44.766 INFO:tasks.workunit.client.0.smithi016.stdout:checking for ndbm.h... no 2024-03-28T22:12:44.799 INFO:tasks.workunit.client.0.smithi016.stdout:checking libaio.h usability... yes 2024-03-28T22:12:44.818 INFO:tasks.workunit.client.0.smithi016.stdout:checking libaio.h presence... yes 2024-03-28T22:12:44.819 INFO:tasks.workunit.client.0.smithi016.stdout:checking for libaio.h... yes 2024-03-28T22:12:44.882 INFO:tasks.workunit.client.0.smithi016.stdout:checking for dm_make_handle in -ldm... no 2024-03-28T22:12:44.882 INFO:tasks.workunit.client.0.smithi016.stdout: 2024-03-28T22:12:44.882 INFO:tasks.workunit.client.0.smithi016.stdout:WARNING: could not find a valid DMAPI base library. 2024-03-28T22:12:44.882 INFO:tasks.workunit.client.0.smithi016.stdout:If you want DMAPI support please install the data migration 2024-03-28T22:12:44.882 INFO:tasks.workunit.client.0.smithi016.stdout:API (dmapi) library package. Alternatively, run "make install" 2024-03-28T22:12:44.882 INFO:tasks.workunit.client.0.smithi016.stdout:from the dmapi source. 2024-03-28T22:12:44.882 INFO:tasks.workunit.client.0.smithi016.stdout: 2024-03-28T22:12:44.918 INFO:tasks.workunit.client.0.smithi016.stdout:checking linux/fiemap.h usability... yes 2024-03-28T22:12:44.939 INFO:tasks.workunit.client.0.smithi016.stdout:checking linux/fiemap.h presence... yes 2024-03-28T22:12:44.939 INFO:tasks.workunit.client.0.smithi016.stdout:checking for linux/fiemap.h... yes 2024-03-28T22:12:44.976 INFO:tasks.workunit.client.0.smithi016.stdout:checking for fallocate... yes 2024-03-28T22:12:45.027 INFO:tasks.workunit.client.0.smithi016.stdout:checking for open_by_handle_at... yes 2024-03-28T22:12:45.082 INFO:tasks.workunit.client.0.smithi016.stdout:checking sys/prctl.h usability... yes 2024-03-28T22:12:45.105 INFO:tasks.workunit.client.0.smithi016.stdout:checking sys/prctl.h presence... yes 2024-03-28T22:12:45.105 INFO:tasks.workunit.client.0.smithi016.stdout:checking for sys/prctl.h... yes 2024-03-28T22:12:45.144 INFO:tasks.workunit.client.0.smithi016.stdout:checking linux/fs.h usability... yes 2024-03-28T22:12:45.166 INFO:tasks.workunit.client.0.smithi016.stdout:checking linux/fs.h presence... yes 2024-03-28T22:12:45.166 INFO:tasks.workunit.client.0.smithi016.stdout:checking for linux/fs.h... yes 2024-03-28T22:12:45.234 INFO:tasks.workunit.client.0.smithi016.stdout:checking btrfsutil.h usability... no 2024-03-28T22:12:45.272 INFO:tasks.workunit.client.0.smithi016.stdout:checking btrfsutil.h presence... no 2024-03-28T22:12:45.272 INFO:tasks.workunit.client.0.smithi016.stdout:checking for btrfsutil.h... no 2024-03-28T22:12:45.381 INFO:tasks.workunit.client.0.smithi016.stdout:checking for copy_file_range... yes 2024-03-28T22:12:45.431 INFO:tasks.workunit.client.0.smithi016.stdout:checking for renameat2... yes 2024-03-28T22:12:45.449 INFO:tasks.workunit.client.0.smithi016.stdout:configure: creating ./config.status 2024-03-28T22:12:45.795 INFO:tasks.workunit.client.0.smithi016.stdout:config.status: creating include/builddefs 2024-03-28T22:12:45.809 INFO:tasks.workunit.client.0.smithi016.stdout:config.status: creating include/config.h 2024-03-28T22:12:45.820 INFO:tasks.workunit.client.0.smithi016.stdout:config.status: executing libtool commands 2024-03-28T22:12:45.915 INFO:tasks.workunit.client.0.smithi016.stdout:Building include 2024-03-28T22:12:45.915 INFO:tasks.workunit.client.0.smithi016.stdout:Building lib 2024-03-28T22:12:45.916 INFO:tasks.workunit.client.0.smithi016.stdout:Building tests 2024-03-28T22:12:45.921 INFO:tasks.workunit.client.0.smithi016.stdout: [LTDEP] 2024-03-28T22:12:46.082 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] dataascii.lo 2024-03-28T22:12:46.083 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] databin.lo 2024-03-28T22:12:46.083 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] datapid.lo 2024-03-28T22:12:46.083 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] file_lock.lo 2024-03-28T22:12:46.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:45 smithi016 ceph-mon[32374]: pgmap v68: 97 pgs: 97 active+clean; 83 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 121 KiB/s wr, 18 op/s 2024-03-28T22:12:46.194 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] forker.lo 2024-03-28T22:12:46.207 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] pattern.lo 2024-03-28T22:12:46.240 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] open_flags.lo 2024-03-28T22:12:46.247 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] random_range.lo 2024-03-28T22:12:46.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:45 smithi053 ceph-mon[35979]: pgmap v68: 97 pgs: 97 active+clean; 83 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 121 KiB/s wr, 18 op/s 2024-03-28T22:12:46.347 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] string_to_tokens.lo 2024-03-28T22:12:46.357 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] str_to_bytes.lo 2024-03-28T22:12:46.395 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] tlibio.lo 2024-03-28T22:12:46.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:45 smithi063 ceph-mon[36249]: pgmap v68: 97 pgs: 97 active+clean; 83 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 121 KiB/s wr, 18 op/s 2024-03-28T22:12:46.463 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] write_log.lo 2024-03-28T22:12:46.493 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] random.lo 2024-03-28T22:12:46.707 INFO:tasks.workunit.client.0.smithi016.stdout: [LD] 2024-03-28T22:12:46.856 INFO:tasks.workunit.client.0.smithi016.stdout:Building ltp 2024-03-28T22:12:46.856 INFO:tasks.workunit.client.0.smithi016.stdout:Building src 2024-03-28T22:12:46.857 INFO:tasks.workunit.client.0.smithi016.stdout:Building m4 2024-03-28T22:12:46.857 INFO:tasks.workunit.client.0.smithi016.stdout:Building common 2024-03-28T22:12:46.864 INFO:tasks.workunit.client.0.smithi016.stdout: [DEP] 2024-03-28T22:12:46.886 INFO:tasks.workunit.client.0.smithi016.stdout: [DEP] 2024-03-28T22:12:47.091 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] doio 2024-03-28T22:12:47.091 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] fsstress 2024-03-28T22:12:47.091 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] fsx 2024-03-28T22:12:47.232 INFO:tasks.workunit.client.0.smithi016.stderr:fsstress.c: In function 'attr_list_path': 2024-03-28T22:12:47.232 INFO:tasks.workunit.client.0.smithi016.stderr:fsstress.c:859:9: warning: 'attr_list' is deprecated: Use listxattr or llistxattr instead [-Wdeprecated-declarations] 2024-03-28T22:12:47.232 INFO:tasks.workunit.client.0.smithi016.stderr: 859 | rval = attr_list(name->path, buffer, buffersize, flags, cursor); 2024-03-28T22:12:47.232 INFO:tasks.workunit.client.0.smithi016.stderr: | ^~~~ 2024-03-28T22:12:47.232 INFO:tasks.workunit.client.0.smithi016.stderr:In file included from ../src/global.h:25, 2024-03-28T22:12:47.232 INFO:tasks.workunit.client.0.smithi016.stderr: from fsstress.c:12: 2024-03-28T22:12:47.232 INFO:tasks.workunit.client.0.smithi016.stderr:/usr/include/attr/attributes.h:165:12: note: declared here 2024-03-28T22:12:47.232 INFO:tasks.workunit.client.0.smithi016.stderr: 165 | extern int attr_list(const char *__path, char *__buffer, const int __buffersize, 2024-03-28T22:12:47.232 INFO:tasks.workunit.client.0.smithi016.stderr: | ^~~~~~~~~ 2024-03-28T22:12:47.232 INFO:tasks.workunit.client.0.smithi016.stderr:fsstress.c: In function 'attr_remove_path': 2024-03-28T22:12:47.232 INFO:tasks.workunit.client.0.smithi016.stderr:fsstress.c:878:9: warning: 'attr_remove' is deprecated: Use removexattr or lremovexattr instead [-Wdeprecated-declarations] 2024-03-28T22:12:47.233 INFO:tasks.workunit.client.0.smithi016.stderr: 878 | rval = attr_remove(name->path, attrname, flags); 2024-03-28T22:12:47.233 INFO:tasks.workunit.client.0.smithi016.stderr: | ^~~~ 2024-03-28T22:12:47.233 INFO:tasks.workunit.client.0.smithi016.stderr:In file included from ../src/global.h:25, 2024-03-28T22:12:47.233 INFO:tasks.workunit.client.0.smithi016.stderr: from fsstress.c:12: 2024-03-28T22:12:47.233 INFO:tasks.workunit.client.0.smithi016.stderr:/usr/include/attr/attributes.h:152:12: note: declared here 2024-03-28T22:12:47.233 INFO:tasks.workunit.client.0.smithi016.stderr: 152 | extern int attr_remove (const char *__path, const char *__attrname, 2024-03-28T22:12:47.233 INFO:tasks.workunit.client.0.smithi016.stderr: | ^~~~~~~~~~~ 2024-03-28T22:12:47.233 INFO:tasks.workunit.client.0.smithi016.stderr:fsstress.c: In function 'attr_set_path': 2024-03-28T22:12:47.233 INFO:tasks.workunit.client.0.smithi016.stderr:fsstress.c:898:9: warning: 'attr_set' is deprecated: Use setxattr or lsetxattr instead [-Wdeprecated-declarations] 2024-03-28T22:12:47.233 INFO:tasks.workunit.client.0.smithi016.stderr: 898 | rval = attr_set(name->path, attrname, attrvalue, valuelength, flags); 2024-03-28T22:12:47.233 INFO:tasks.workunit.client.0.smithi016.stderr: | ^~~~ 2024-03-28T22:12:47.233 INFO:tasks.workunit.client.0.smithi016.stderr:In file included from ../src/global.h:25, 2024-03-28T22:12:47.233 INFO:tasks.workunit.client.0.smithi016.stderr: from fsstress.c:12: 2024-03-28T22:12:47.233 INFO:tasks.workunit.client.0.smithi016.stderr:/usr/include/attr/attributes.h:139:12: note: declared here 2024-03-28T22:12:47.233 INFO:tasks.workunit.client.0.smithi016.stderr: 139 | extern int attr_set (const char *__path, const char *__attrname, 2024-03-28T22:12:47.234 INFO:tasks.workunit.client.0.smithi016.stderr: | ^~~~~~~~ 2024-03-28T22:12:47.242 INFO:tasks.workunit.client.0.smithi016.stderr:fsstress.c: In function 'allocsp_f': 2024-03-28T22:12:47.243 INFO:tasks.workunit.client.0.smithi016.stderr:fsstress.c:2047:32: error: 'XFS_IOC_ALLOCSP64' undeclared (first use in this function); did you mean 'XFS_IOC_RESVSP64'? 2024-03-28T22:12:47.243 INFO:tasks.workunit.client.0.smithi016.stderr: 2047 | e = xfsctl(f.path, fd, XFS_IOC_ALLOCSP64, &fl) < 0 ? errno : 0; 2024-03-28T22:12:47.243 INFO:tasks.workunit.client.0.smithi016.stderr: | ^~~~~~~~~~~~~~~~~ 2024-03-28T22:12:47.243 INFO:tasks.workunit.client.0.smithi016.stderr: | XFS_IOC_RESVSP64 2024-03-28T22:12:47.243 INFO:tasks.workunit.client.0.smithi016.stderr:fsstress.c:2047:32: note: each undeclared identifier is reported only once for each function it appears in 2024-03-28T22:12:47.260 INFO:tasks.workunit.client.0.smithi016.stderr:fsstress.c: In function 'freesp_f': 2024-03-28T22:12:47.261 INFO:tasks.workunit.client.0.smithi016.stderr:fsstress.c:3640:32: error: 'XFS_IOC_FREESP64' undeclared (first use in this function); did you mean 'XFS_IOC_RESVSP64'? 2024-03-28T22:12:47.261 INFO:tasks.workunit.client.0.smithi016.stderr: 3640 | e = xfsctl(f.path, fd, XFS_IOC_FREESP64, &fl) < 0 ? errno : 0; 2024-03-28T22:12:47.261 INFO:tasks.workunit.client.0.smithi016.stderr: | ^~~~~~~~~~~~~~~~ 2024-03-28T22:12:47.261 INFO:tasks.workunit.client.0.smithi016.stderr: | XFS_IOC_RESVSP64 2024-03-28T22:12:47.294 INFO:tasks.workunit.client.0.smithi016.stderr:gmake[3]: *** [Makefile:47: fsstress] Error 1 2024-03-28T22:12:47.294 INFO:tasks.workunit.client.0.smithi016.stderr:gmake[3]: *** Waiting for unfinished jobs.... 2024-03-28T22:12:47.661 INFO:tasks.workunit.client.0.smithi016.stderr:doio.c: In function 'parse_cmdline': 2024-03-28T22:12:47.661 INFO:tasks.workunit.client.0.smithi016.stderr:doio.c:3113:33: warning: '%s' directive argument is null [-Wformat-overflow=] 2024-03-28T22:12:47.661 INFO:tasks.workunit.client.0.smithi016.stderr: 3113 | fprintf(stderr, 2024-03-28T22:12:47.661 INFO:tasks.workunit.client.0.smithi016.stderr: | ^~~~~~~~~~~~~~~ 2024-03-28T22:12:47.662 INFO:tasks.workunit.client.0.smithi016.stderr: 3114 | "%s%s: Illegal -C arg (%s). Must be one of: ", 2024-03-28T22:12:47.662 INFO:tasks.workunit.client.0.smithi016.stderr: | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-03-28T22:12:47.662 INFO:tasks.workunit.client.0.smithi016.stderr: 3115 | Prog, TagName, tok); 2024-03-28T22:12:47.662 INFO:tasks.workunit.client.0.smithi016.stderr: | ~~~~~~~~~~~~~~~~~~~ 2024-03-28T22:12:47.696 INFO:tasks.workunit.client.0.smithi016.stderr:fsx.c: In function 'main': 2024-03-28T22:12:47.696 INFO:tasks.workunit.client.0.smithi016.stderr:fsx.c:2662:33: warning: 'strncpy' specified bound 4096 equals destination size [-Wstringop-truncation] 2024-03-28T22:12:47.696 INFO:tasks.workunit.client.0.smithi016.stderr: 2662 | strncpy(opsfile, optarg, sizeof(opsfile)); 2024-03-28T22:12:47.696 INFO:tasks.workunit.client.0.smithi016.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-03-28T22:12:47.696 INFO:tasks.workunit.client.0.smithi016.stderr:fsx.c:2632:25: warning: 'strncpy' specified bound 1024 equals destination size [-Wstringop-truncation] 2024-03-28T22:12:47.696 INFO:tasks.workunit.client.0.smithi016.stderr: 2632 | strncpy(dname, optarg, sizeof(dname)); 2024-03-28T22:12:47.696 INFO:tasks.workunit.client.0.smithi016.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-03-28T22:12:47.795 INFO:tasks.workunit.client.0.smithi016.stderr:gmake[2]: *** [include/buildrules:30: ltp] Error 2 2024-03-28T22:12:47.795 INFO:tasks.workunit.client.0.smithi016.stderr:gmake[2]: *** Waiting for unfinished jobs.... 2024-03-28T22:12:48.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:47 smithi053 ceph-mon[35979]: pgmap v69: 97 pgs: 97 active+clean; 91 MiB data, 705 MiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 876 KiB/s wr, 107 op/s 2024-03-28T22:12:48.358 INFO:tasks.workunit.client.0.smithi016.stdout: [DEP] 2024-03-28T22:12:48.359 INFO:tasks.workunit.client.0.smithi016.stdout: [DEP] 2024-03-28T22:12:48.360 INFO:tasks.workunit.client.0.smithi016.stdout: [DEP] 2024-03-28T22:12:48.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:47 smithi063 ceph-mon[36249]: pgmap v69: 97 pgs: 97 active+clean; 91 MiB data, 705 MiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 876 KiB/s wr, 107 op/s 2024-03-28T22:12:48.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:47 smithi016 ceph-mon[32374]: pgmap v69: 97 pgs: 97 active+clean; 91 MiB data, 705 MiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 876 KiB/s wr, 107 op/s 2024-03-28T22:12:48.596 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] dirstress 2024-03-28T22:12:48.598 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] fill 2024-03-28T22:12:48.598 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] fill2 2024-03-28T22:12:48.598 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] getpagesize 2024-03-28T22:12:48.731 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] holes 2024-03-28T22:12:48.760 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] lstat64 2024-03-28T22:12:48.831 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] nametest 2024-03-28T22:12:48.835 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] permname 2024-03-28T22:12:48.927 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] randholes 2024-03-28T22:12:48.943 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] runas 2024-03-28T22:12:49.059 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] truncfile 2024-03-28T22:12:49.084 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] usemem 2024-03-28T22:12:49.116 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] mmapcat 2024-03-28T22:12:49.202 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] append_reader 2024-03-28T22:12:49.233 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] append_writer 2024-03-28T22:12:49.239 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] dirperf 2024-03-28T22:12:49.267 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] metaperf 2024-03-28T22:12:49.355 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] devzero 2024-03-28T22:12:49.391 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] feature 2024-03-28T22:12:49.478 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] alloc 2024-03-28T22:12:49.549 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] fault 2024-03-28T22:12:49.562 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] fstest 2024-03-28T22:12:49.593 INFO:tasks.workunit.client.0.smithi016.stdout: [CC] t_access_root 2024-03-28T22:12:49.614 INFO:tasks.workunit.client.0.smithi016.stderr:alloc.c: In function 'main': 2024-03-28T22:12:49.614 INFO:tasks.workunit.client.0.smithi016.stderr:alloc.c:154:1: error: 'bozo' undeclared (first use in this function) 2024-03-28T22:12:49.614 INFO:tasks.workunit.client.0.smithi016.stderr: 154 | bozo! 2024-03-28T22:12:49.614 INFO:tasks.workunit.client.0.smithi016.stderr: | ^~~~ 2024-03-28T22:12:49.614 INFO:tasks.workunit.client.0.smithi016.stderr:alloc.c:154:1: note: each undeclared identifier is reported only once for each function it appears in 2024-03-28T22:12:49.614 INFO:tasks.workunit.client.0.smithi016.stderr:alloc.c:154:5: error: expected ';' before '!' token 2024-03-28T22:12:49.614 INFO:tasks.workunit.client.0.smithi016.stderr: 154 | bozo! 2024-03-28T22:12:49.614 INFO:tasks.workunit.client.0.smithi016.stderr: | ^ 2024-03-28T22:12:49.614 INFO:tasks.workunit.client.0.smithi016.stderr: | ; 2024-03-28T22:12:49.615 INFO:tasks.workunit.client.0.smithi016.stderr:alloc.c:184:25: error: 'rflag' undeclared (first use in this function); did you mean 'nflag'? 2024-03-28T22:12:49.615 INFO:tasks.workunit.client.0.smithi016.stderr: 184 | rflag = 1; 2024-03-28T22:12:49.615 INFO:tasks.workunit.client.0.smithi016.stderr: | ^~~~~ 2024-03-28T22:12:49.615 INFO:tasks.workunit.client.0.smithi016.stderr: | nflag 2024-03-28T22:12:49.615 INFO:tasks.workunit.client.0.smithi016.stderr:alloc.c:320:5: error: expected ';' before '!' token 2024-03-28T22:12:49.615 INFO:tasks.workunit.client.0.smithi016.stderr: 320 | bozo! 2024-03-28T22:12:49.615 INFO:tasks.workunit.client.0.smithi016.stderr: | ^ 2024-03-28T22:12:49.616 INFO:tasks.workunit.client.0.smithi016.stderr: | ; 2024-03-28T22:12:49.616 INFO:tasks.workunit.client.0.smithi016.stderr:alloc.c:128:25: warning: variable 'f' set but not used [-Wunused-but-set-variable] 2024-03-28T22:12:49.616 INFO:tasks.workunit.client.0.smithi016.stderr: 128 | struct flock64 f; 2024-03-28T22:12:49.616 INFO:tasks.workunit.client.0.smithi016.stderr: | ^ 2024-03-28T22:12:49.618 INFO:tasks.workunit.client.0.smithi016.stderr:gmake[3]: *** [Makefile:83: alloc] Error 1 2024-03-28T22:12:49.618 INFO:tasks.workunit.client.0.smithi016.stderr:gmake[3]: *** Waiting for unfinished jobs.... 2024-03-28T22:12:49.710 INFO:tasks.workunit.client.0.smithi016.stderr:fstest.c: In function 'run_child': 2024-03-28T22:12:49.711 INFO:tasks.workunit.client.0.smithi016.stderr:fstest.c:95:27: warning: '/file' directive writing 5 bytes into a region of size between 1 and 1024 [-Wformat-overflow=] 2024-03-28T22:12:49.711 INFO:tasks.workunit.client.0.smithi016.stderr: 95 | sprintf(fname, "%s/file%d", dir, fnum); 2024-03-28T22:12:49.711 INFO:tasks.workunit.client.0.smithi016.stderr: | ^~~~~ 2024-03-28T22:12:49.711 INFO:tasks.workunit.client.0.smithi016.stderr:fstest.c:95:24: note: directive argument in the range [0, 2147483647] 2024-03-28T22:12:49.711 INFO:tasks.workunit.client.0.smithi016.stderr: 95 | sprintf(fname, "%s/file%d", dir, fnum); 2024-03-28T22:12:49.711 INFO:tasks.workunit.client.0.smithi016.stderr: | ^~~~~~~~~~~ 2024-03-28T22:12:49.711 INFO:tasks.workunit.client.0.smithi016.stderr:fstest.c:95:9: note: 'sprintf' output between 7 and 1039 bytes into a destination of size 1024 2024-03-28T22:12:49.711 INFO:tasks.workunit.client.0.smithi016.stderr: 95 | sprintf(fname, "%s/file%d", dir, fnum); 2024-03-28T22:12:49.711 INFO:tasks.workunit.client.0.smithi016.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-03-28T22:12:49.711 INFO:tasks.workunit.client.0.smithi016.stderr:fstest.c:166:27: warning: '/file' directive writing 5 bytes into a region of size between 1 and 1024 [-Wformat-overflow=] 2024-03-28T22:12:49.711 INFO:tasks.workunit.client.0.smithi016.stderr: 166 | sprintf(fname, "%s/file%d", dir, fnum); 2024-03-28T22:12:49.711 INFO:tasks.workunit.client.0.smithi016.stderr: | ^~~~~ 2024-03-28T22:12:49.711 INFO:tasks.workunit.client.0.smithi016.stderr:fstest.c:166:24: note: directive argument in the range [0, 2147483647] 2024-03-28T22:12:49.711 INFO:tasks.workunit.client.0.smithi016.stderr: 166 | sprintf(fname, "%s/file%d", dir, fnum); 2024-03-28T22:12:49.711 INFO:tasks.workunit.client.0.smithi016.stderr: | ^~~~~~~~~~~ 2024-03-28T22:12:49.712 INFO:tasks.workunit.client.0.smithi016.stderr:fstest.c:166:9: note: 'sprintf' output between 7 and 1039 bytes into a destination of size 1024 2024-03-28T22:12:49.712 INFO:tasks.workunit.client.0.smithi016.stderr: 166 | sprintf(fname, "%s/file%d", dir, fnum); 2024-03-28T22:12:49.712 INFO:tasks.workunit.client.0.smithi016.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-03-28T22:12:49.782 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-03-28T22:12:49.782 INFO:tasks.workunit.client.0.smithi016.stderr:gmake[2]: *** [include/buildrules:30: src] Error 2 2024-03-28T22:12:49.782 INFO:tasks.workunit.client.0.smithi016.stderr:make[1]: *** [Makefile:53: default] Error 2 2024-03-28T22:12:49.783 INFO:tasks.workunit.client.0.smithi016.stderr:make: *** [Makefile:51: default] Error 2 2024-03-28T22:12:49.783 INFO:tasks.workunit:Stopping ['suites/fsx.sh'] on client.0... 2024-03-28T22:12:49.783 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-28T22:12:50.077 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:49 smithi016 ceph-mon[32374]: pgmap v70: 97 pgs: 97 active+clean; 91 MiB data, 705 MiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 760 KiB/s wr, 93 op/s 2024-03-28T22:12:50.077 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:49 smithi016 ceph-mon[32374]: scrub summary: idle 2024-03-28T22:12:50.078 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_55f5138942aa4866e49329c9103468889cec3f73/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_55f5138942aa4866e49329c9103468889cec3f73/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_55f5138942aa4866e49329c9103468889cec3f73/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/fsx.sh) on smithi016 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=55f5138942aa4866e49329c9103468889cec3f73 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2024-03-28T22:12:50.328 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=7f9a46a26efd4604b4fc653ab336cdc4 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_55f5138942aa4866e49329c9103468889cec3f73/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_55f5138942aa4866e49329c9103468889cec3f73/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_55f5138942aa4866e49329c9103468889cec3f73/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/fsx.sh) on smithi016 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=55f5138942aa4866e49329c9103468889cec3f73 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2024-03-28T22:12:50.331 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-03-28T22:12:50.343 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-03-28T22:12:50.343 INFO:teuthology.task.exec:Executing custom commands... 2024-03-28T22:12:50.343 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-03-28T22:12:50.343 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-03-28T22:12:50.348 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:49 smithi053 ceph-mon[35979]: pgmap v70: 97 pgs: 97 active+clean; 91 MiB data, 705 MiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 760 KiB/s wr, 93 op/s 2024-03-28T22:12:50.349 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:49 smithi053 ceph-mon[35979]: scrub summary: idle 2024-03-28T22:12:50.376 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-03-28T22:12:50.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:49 smithi063 ceph-mon[36249]: pgmap v70: 97 pgs: 97 active+clean; 91 MiB data, 705 MiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 760 KiB/s wr, 93 op/s 2024-03-28T22:12:50.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:49 smithi063 ceph-mon[36249]: scrub summary: idle 2024-03-28T22:12:51.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:50 smithi053 ceph-mon[35979]: scrub summary: idle 2024-03-28T22:12:51.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:50 smithi063 ceph-mon[36249]: scrub summary: idle 2024-03-28T22:12:51.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:50 smithi016 ceph-mon[32374]: scrub summary: idle 2024-03-28T22:12:52.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:51 smithi053 ceph-mon[35979]: pgmap v71: 97 pgs: 97 active+clean; 91 MiB data, 705 MiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 760 KiB/s wr, 93 op/s 2024-03-28T22:12:52.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:51 smithi063 ceph-mon[36249]: pgmap v71: 97 pgs: 97 active+clean; 91 MiB data, 705 MiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 760 KiB/s wr, 93 op/s 2024-03-28T22:12:52.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:51 smithi016 ceph-mon[32374]: pgmap v71: 97 pgs: 97 active+clean; 91 MiB data, 705 MiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 760 KiB/s wr, 93 op/s 2024-03-28T22:12:54.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:53 smithi053 ceph-mon[35979]: pgmap v72: 97 pgs: 97 active+clean; 91 MiB data, 706 MiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 797 KiB/s wr, 100 op/s 2024-03-28T22:12:54.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:53 smithi063 ceph-mon[36249]: pgmap v72: 97 pgs: 97 active+clean; 91 MiB data, 706 MiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 797 KiB/s wr, 100 op/s 2024-03-28T22:12:54.482 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:53 smithi016 ceph-mon[32374]: pgmap v72: 97 pgs: 97 active+clean; 91 MiB data, 706 MiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 797 KiB/s wr, 100 op/s 2024-03-28T22:12:56.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:55 smithi053 ceph-mon[35979]: pgmap v73: 97 pgs: 97 active+clean; 91 MiB data, 706 MiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 791 KiB/s wr, 96 op/s 2024-03-28T22:12:56.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:55 smithi063 ceph-mon[36249]: pgmap v73: 97 pgs: 97 active+clean; 91 MiB data, 706 MiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 791 KiB/s wr, 96 op/s 2024-03-28T22:12:56.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:55 smithi016 ceph-mon[32374]: pgmap v73: 97 pgs: 97 active+clean; 91 MiB data, 706 MiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 791 KiB/s wr, 96 op/s 2024-03-28T22:12:58.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:57 smithi053 ceph-mon[35979]: pgmap v74: 97 pgs: 97 active+clean; 91 MiB data, 706 MiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 802 KiB/s wr, 100 op/s 2024-03-28T22:12:58.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:57 smithi063 ceph-mon[36249]: pgmap v74: 97 pgs: 97 active+clean; 91 MiB data, 706 MiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 802 KiB/s wr, 100 op/s 2024-03-28T22:12:58.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:57 smithi016 ceph-mon[32374]: pgmap v74: 97 pgs: 97 active+clean; 91 MiB data, 706 MiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 802 KiB/s wr, 100 op/s 2024-03-28T22:13:00.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:12:59 smithi053 ceph-mon[35979]: pgmap v75: 97 pgs: 97 active+clean; 91 MiB data, 706 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 11 op/s 2024-03-28T22:13:00.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:12:59 smithi063 ceph-mon[36249]: pgmap v75: 97 pgs: 97 active+clean; 91 MiB data, 706 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 11 op/s 2024-03-28T22:13:00.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:12:59 smithi016 ceph-mon[32374]: pgmap v75: 97 pgs: 97 active+clean; 91 MiB data, 706 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 11 op/s 2024-03-28T22:13:02.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:01 smithi053 ceph-mon[35979]: pgmap v76: 97 pgs: 97 active+clean; 91 MiB data, 706 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 11 op/s 2024-03-28T22:13:02.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:01 smithi063 ceph-mon[36249]: pgmap v76: 97 pgs: 97 active+clean; 91 MiB data, 706 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 11 op/s 2024-03-28T22:13:02.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:01 smithi016 ceph-mon[32374]: pgmap v76: 97 pgs: 97 active+clean; 91 MiB data, 706 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 11 op/s 2024-03-28T22:13:04.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:03 smithi053 ceph-mon[35979]: pgmap v77: 97 pgs: 97 active+clean; 91 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 71 KiB/s wr, 29 op/s 2024-03-28T22:13:04.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:03 smithi063 ceph-mon[36249]: pgmap v77: 97 pgs: 97 active+clean; 91 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 71 KiB/s wr, 29 op/s 2024-03-28T22:13:04.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:03 smithi016 ceph-mon[32374]: pgmap v77: 97 pgs: 97 active+clean; 91 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 71 KiB/s wr, 29 op/s 2024-03-28T22:13:06.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:05 smithi053 ceph-mon[35979]: pgmap v78: 97 pgs: 97 active+clean; 91 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 34 KiB/s wr, 22 op/s 2024-03-28T22:13:06.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:05 smithi063 ceph-mon[36249]: pgmap v78: 97 pgs: 97 active+clean; 91 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 34 KiB/s wr, 22 op/s 2024-03-28T22:13:06.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:05 smithi016 ceph-mon[32374]: pgmap v78: 97 pgs: 97 active+clean; 91 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 34 KiB/s wr, 22 op/s 2024-03-28T22:13:06.490 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:13:06.491 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:13:06.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.607+0000 7fe3b2581640 1 -- 172.21.15.16:0/3775843911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3ac1020b0 msgr2=0x7fe3ac10e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:06.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.607+0000 7fe3b2581640 1 --2- 172.21.15.16:0/3775843911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3ac1020b0 0x7fe3ac10e5c0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fe3a4009cd0 tx=0x7fe3a402c0f0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.607+0000 7fe3b2581640 1 -- 172.21.15.16:0/3775843911 shutdown_connections 2024-03-28T22:13:06.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.607+0000 7fe3b2581640 1 --2- 172.21.15.16:0/3775843911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3ac1020b0 0x7fe3ac10e5c0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.607+0000 7fe3b2581640 1 --2- 172.21.15.16:0/3775843911 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe3ac1016f0 0x7fe3ac101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.607+0000 7fe3b2581640 1 --2- 172.21.15.16:0/3775843911 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3ac107700 0x7fe3ac107ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.607+0000 7fe3b2581640 1 -- 172.21.15.16:0/3775843911 >> 172.21.15.16:0/3775843911 conn(0x7fe3ac0fd1e0 msgr2=0x7fe3ac0ff600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:06.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.607+0000 7fe3b2581640 1 -- 172.21.15.16:0/3775843911 shutdown_connections 2024-03-28T22:13:06.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.607+0000 7fe3b2581640 1 -- 172.21.15.16:0/3775843911 wait complete. 2024-03-28T22:13:06.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.607+0000 7fe3b2581640 1 Processor -- start 2024-03-28T22:13:06.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.607+0000 7fe3b2581640 1 -- start start 2024-03-28T22:13:06.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.608+0000 7fe3b2581640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3ac1016f0 0x7fe3ac073a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:06.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.608+0000 7fe3b2581640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3ac1020b0 0x7fe3ac073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:06.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.608+0000 7fe3b2581640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe3ac107700 0x7fe3ac0744d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:06.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.608+0000 7fe3b2581640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe3ac111b10 con 0x7fe3ac1016f0 2024-03-28T22:13:06.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.608+0000 7fe3b2581640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe3ac111990 con 0x7fe3ac1020b0 2024-03-28T22:13:06.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.608+0000 7fe3b2581640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe3ac111c90 con 0x7fe3ac107700 2024-03-28T22:13:06.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.608+0000 7fe3abfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3ac1016f0 0x7fe3ac073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:06.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.608+0000 7fe3abfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3ac1016f0 0x7fe3ac073a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55750/0 (socket says 172.21.15.16:55750) 2024-03-28T22:13:06.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.608+0000 7fe3abfff640 1 -- 172.21.15.16:0/4046861621 learned_addr learned my addr 172.21.15.16:0/4046861621 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:06.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.608+0000 7fe3ab7fe640 1 --2- 172.21.15.16:0/4046861621 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3ac1020b0 0x7fe3ac073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:06.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.608+0000 7fe3b0b1e640 1 --2- 172.21.15.16:0/4046861621 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe3ac107700 0x7fe3ac0744d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:06.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.608+0000 7fe3abfff640 1 -- 172.21.15.16:0/4046861621 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe3ac107700 msgr2=0x7fe3ac0744d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:06.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.608+0000 7fe3abfff640 1 --2- 172.21.15.16:0/4046861621 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe3ac107700 0x7fe3ac0744d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.608+0000 7fe3abfff640 1 -- 172.21.15.16:0/4046861621 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3ac1020b0 msgr2=0x7fe3ac073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:06.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.608+0000 7fe3abfff640 1 --2- 172.21.15.16:0/4046861621 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3ac1020b0 0x7fe3ac073f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.608+0000 7fe3abfff640 1 -- 172.21.15.16:0/4046861621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3ac1a6480 con 0x7fe3ac1016f0 2024-03-28T22:13:06.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.609+0000 7fe3abfff640 1 --2- 172.21.15.16:0/4046861621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3ac1016f0 0x7fe3ac073a50 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fe39c00e9f0 tx=0x7fe39c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:06.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.609+0000 7fe3a97fa640 1 -- 172.21.15.16:0/4046861621 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe39c00cd80 con 0x7fe3ac1016f0 2024-03-28T22:13:06.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.609+0000 7fe3a97fa640 1 -- 172.21.15.16:0/4046861621 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe39c010910 con 0x7fe3ac1016f0 2024-03-28T22:13:06.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.609+0000 7fe3b2581640 1 -- 172.21.15.16:0/4046861621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3ac1a6770 con 0x7fe3ac1016f0 2024-03-28T22:13:06.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.609+0000 7fe3a97fa640 1 -- 172.21.15.16:0/4046861621 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe39c018990 con 0x7fe3ac1016f0 2024-03-28T22:13:06.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.609+0000 7fe3b2581640 1 -- 172.21.15.16:0/4046861621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3ac1a6cc0 con 0x7fe3ac1016f0 2024-03-28T22:13:06.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.612+0000 7fe3a97fa640 1 -- 172.21.15.16:0/4046861621 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7fe39c010ac0 con 0x7fe3ac1016f0 2024-03-28T22:13:06.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.612+0000 7fe3b5167740 1 Processor -- start 2024-03-28T22:13:06.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.612+0000 7fe3b5167740 1 -- start start 2024-03-28T22:13:06.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.612+0000 7fe3b5167740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e46a682290 0x55e46a682670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:06.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.612+0000 7fe3b5167740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e46a682bb0 0x55e46a68b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:06.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.612+0000 7fe3b5167740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e46a68b790 0x55e46a68db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:06.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.612+0000 7fe3b5167740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e46a5e6180 con 0x55e46a682290 2024-03-28T22:13:06.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.612+0000 7fe3b5167740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55e46a58eb70 con 0x55e46a682bb0 2024-03-28T22:13:06.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.612+0000 7fe3b5167740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e46a5e2220 con 0x55e46a68b790 2024-03-28T22:13:06.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.612+0000 7fe3b0b1e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e46a682290 0x55e46a682670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:06.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.612+0000 7fe3b0b1e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e46a682290 0x55e46a682670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55758/0 (socket says 172.21.15.16:55758) 2024-03-28T22:13:06.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.612+0000 7fe3b0b1e640 1 -- 172.21.15.16:0/1025049709 learned_addr learned my addr 172.21.15.16:0/1025049709 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:06.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.612+0000 7fe3abfff640 1 --2- 172.21.15.16:0/1025049709 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e46a682bb0 0x55e46a68b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:06.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.612+0000 7fe3a97fa640 1 --2- 172.21.15.16:0/4046861621 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fe3940a1ef0 0x7fe3940a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:06.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.612+0000 7fe3ab7fe640 1 --2- 172.21.15.16:0/1025049709 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e46a68b790 0x55e46a68db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:06.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.612+0000 7fe3b0b1e640 1 --2- 172.21.15.16:0/4046861621 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fe3940a1ef0 0x7fe3940a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:06.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.613+0000 7fe3b0b1e640 1 -- 172.21.15.16:0/1025049709 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e46a68b790 msgr2=0x55e46a68db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:06.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.613+0000 7fe3b0b1e640 1 --2- 172.21.15.16:0/1025049709 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e46a68b790 0x55e46a68db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.613+0000 7fe3b0b1e640 1 -- 172.21.15.16:0/1025049709 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e46a682bb0 msgr2=0x55e46a68b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:06.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.613+0000 7fe3b0b1e640 1 --2- 172.21.15.16:0/1025049709 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e46a682bb0 0x55e46a68b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.613+0000 7fe3b0b1e640 1 -- 172.21.15.16:0/1025049709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e46a5a27c0 con 0x55e46a682290 2024-03-28T22:13:06.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.613+0000 7fe3a97fa640 1 -- 172.21.15.16:0/4046861621 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fe39c0ce420 con 0x7fe3ac1016f0 2024-03-28T22:13:06.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.613+0000 7fe3b0b1e640 1 --2- 172.21.15.16:0/1025049709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e46a682290 0x55e46a682670 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fe3a4005e30 tx=0x7fe3a402ce80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba42e80714157345 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:06.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.613+0000 7fe3927fc640 1 -- 172.21.15.16:0/1025049709 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a4050070 con 0x55e46a682290 2024-03-28T22:13:06.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.613+0000 7fe3b0b1e640 1 --2- 172.21.15.16:0/4046861621 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fe3940a1ef0 0x7fe3940a43b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe3a4047000 tx=0x7fe3a4048000 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:06.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.613+0000 7fe3927fc640 1 -- 172.21.15.16:0/1025049709 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3a4003e50 con 0x55e46a682290 2024-03-28T22:13:06.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.613+0000 7fe3927fc640 1 -- 172.21.15.16:0/1025049709 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a4039040 con 0x55e46a682290 2024-03-28T22:13:06.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.614+0000 7fe3b5167740 1 -- 172.21.15.16:0/1025049709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e46a682290 msgr2=0x55e46a682670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:06.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.614+0000 7fe3b5167740 1 --2- 172.21.15.16:0/1025049709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e46a682290 0x55e46a682670 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fe3a4005e30 tx=0x7fe3a402ce80 comp rx=0 tx=0).stop 2024-03-28T22:13:06.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.614+0000 7fe3b5167740 1 -- 172.21.15.16:0/1025049709 shutdown_connections 2024-03-28T22:13:06.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.614+0000 7fe3b5167740 1 --2- 172.21.15.16:0/1025049709 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e46a68b790 0x55e46a68db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.614+0000 7fe3b5167740 1 --2- 172.21.15.16:0/1025049709 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e46a682bb0 0x55e46a68b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.614+0000 7fe3b5167740 1 --2- 172.21.15.16:0/1025049709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e46a682290 0x55e46a682670 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.614+0000 7fe3b5167740 1 -- 172.21.15.16:0/1025049709 >> 172.21.15.16:0/1025049709 conn(0x55e46a67c9c0 msgr2=0x55e46a67cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:06.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.614+0000 7fe3b5167740 1 -- 172.21.15.16:0/1025049709 shutdown_connections 2024-03-28T22:13:06.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.614+0000 7fe3b5167740 1 -- 172.21.15.16:0/1025049709 wait complete. 2024-03-28T22:13:06.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.614+0000 7fe3b5167740 1 -- start start 2024-03-28T22:13:06.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.614+0000 7fe3b5167740 1 Processor -- start 2024-03-28T22:13:06.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.615+0000 7fe3b5167740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e46a682290 0x55e46a6887c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:06.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.615+0000 7fe3b5167740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e46a682bb0 0x55e46a688d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:06.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.615+0000 7fe3b5167740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e46a68b790 0x55e46a689240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:06.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.615+0000 7fe3b5167740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e46a68e520 con 0x55e46a68b790 2024-03-28T22:13:06.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.615+0000 7fe3b5167740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55e46a5844a0 con 0x55e46a682290 2024-03-28T22:13:06.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.615+0000 7fe3b5167740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e46a68e6a0 con 0x55e46a682bb0 2024-03-28T22:13:06.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.615+0000 7fe3b5167740 10 client.0 _ll_register_callbacks cb 0x55e46a587d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:13:06.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.615+0000 7fe3b0b1e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e46a68b790 0x55e46a689240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:06.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.615+0000 7fe3b0b1e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e46a68b790 0x55e46a689240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55772/0 (socket says 172.21.15.16:55772) 2024-03-28T22:13:06.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.615+0000 7fe3b0b1e640 1 -- 172.21.15.16:0/2889724438 learned_addr learned my addr 172.21.15.16:0/2889724438 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:06.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.615+0000 7fe3abfff640 1 --2- 172.21.15.16:0/2889724438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e46a682290 0x55e46a6887c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:06.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.615+0000 7fe3ab7fe640 1 --2- 172.21.15.16:0/2889724438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e46a682bb0 0x55e46a688d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:06.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.615+0000 7fe3b0b1e640 1 -- 172.21.15.16:0/2889724438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e46a682bb0 msgr2=0x55e46a688d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:06.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.615+0000 7fe3b0b1e640 1 --2- 172.21.15.16:0/2889724438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e46a682bb0 0x55e46a688d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.615+0000 7fe3b0b1e640 1 -- 172.21.15.16:0/2889724438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e46a682290 msgr2=0x55e46a6887c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:06.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.615+0000 7fe3b0b1e640 1 --2- 172.21.15.16:0/2889724438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e46a682290 0x55e46a6887c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.615+0000 7fe3b0b1e640 1 -- 172.21.15.16:0/2889724438 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e46a3c47d0 con 0x55e46a68b790 2024-03-28T22:13:06.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.615+0000 7fe3b0b1e640 1 --2- 172.21.15.16:0/2889724438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e46a68b790 0x55e46a689240 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fe3a4004af0 tx=0x7fe3a402c770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ae942d604a32de4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:06.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.615+0000 7fe38b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:13:06.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.615+0000 7fe38b7fe640 1 -- 172.21.15.16:0/2889724438 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a4050030 con 0x55e46a68b790 2024-03-28T22:13:06.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.615+0000 7fe38b7fe640 1 -- 172.21.15.16:0/2889724438 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fe3a4003970 con 0x55e46a68b790 2024-03-28T22:13:06.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.615+0000 7fe3b5167740 1 -- 172.21.15.16:0/2889724438 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3a4003e50 con 0x55e46a68b790 2024-03-28T22:13:06.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.616+0000 7fe38b7fe640 1 -- 172.21.15.16:0/2889724438 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe3a4061050 con 0x55e46a68b790 2024-03-28T22:13:06.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.616+0000 7fe3b5167740 10 client.15990 fetch_fsmap learned FSMap version 24 2024-03-28T22:13:06.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.616+0000 7fe3b5167740 1 -- 172.21.15.16:0/2889724438 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e46a2fa3f0 con 0x55e46a68b790 2024-03-28T22:13:06.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.616+0000 7fe38b7fe640 1 -- 172.21.15.16:0/2889724438 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3a4003e50 con 0x55e46a68b790 2024-03-28T22:13:06.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.616+0000 7fe38b7fe640 1 -- 172.21.15.16:0/2889724438 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7fe3a403f070 con 0x55e46a68b790 2024-03-28T22:13:06.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.616+0000 7fe3b5167740 10 client.15990 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:13:06.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.616+0000 7fe3b5167740 10 client.15990 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:13:06.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.616+0000 7fe3b5167740 20 client.15990 populate_metadata read hostname 'smithi016' 2024-03-28T22:13:06.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.616+0000 7fe3b5167740 1 --2- 172.21.15.16:0/2889724438 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e46a69b940 0x55e46a69dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:06.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.616+0000 7fe3b5167740 4 client.15990 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:13:06.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.616+0000 7fe3b5167740 1 -- 172.21.15.16:0/2889724438 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e46a5f36c0 con 0x55e46a69b940 2024-03-28T22:13:06.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.616+0000 7fe3abfff640 1 --2- 172.21.15.16:0/2889724438 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e46a69b940 0x55e46a69dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:06.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.617+0000 7fe3abfff640 1 --2- 172.21.15.16:0/2889724438 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e46a69b940 0x55e46a69dd30 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb24c4dd69621446 server_cookie=c655a845a9e19085 in_seq=0 out_seq=0 2024-03-28T22:13:06.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.617+0000 7fe38b7fe640 10 client.15990 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:13:06.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.617+0000 7fe38b7fe640 1 -- 172.21.15.16:0/2889724438 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e46a5f36c0 con 0x55e46a69b940 2024-03-28T22:13:06.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.617+0000 7fe38b7fe640 10 client.15990 handle_command_reply: tid=0 2024-03-28T22:13:06.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.617+0000 7fe3b5167740 1 client.15990 shutdown 2024-03-28T22:13:06.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.617+0000 7fe3b5167740 1 -- 172.21.15.16:0/2889724438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e46a68b790 msgr2=0x55e46a689240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:06.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.617+0000 7fe3b5167740 1 --2- 172.21.15.16:0/2889724438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e46a68b790 0x55e46a689240 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fe3a4004af0 tx=0x7fe3a402c770 comp rx=0 tx=0).stop 2024-03-28T22:13:06.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.617+0000 7fe3b5167740 1 -- 172.21.15.16:0/2889724438 shutdown_connections 2024-03-28T22:13:06.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.617+0000 7fe3b5167740 1 --2- 172.21.15.16:0/2889724438 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e46a69b940 0x55e46a69dd30 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.617+0000 7fe3b5167740 1 --2- 172.21.15.16:0/2889724438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e46a68b790 0x55e46a689240 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.617+0000 7fe3b5167740 1 --2- 172.21.15.16:0/2889724438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e46a682bb0 0x55e46a688d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.617+0000 7fe38b7fe640 0 client.15990 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:13:06.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.617+0000 7fe3b5167740 1 --2- 172.21.15.16:0/2889724438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e46a682290 0x55e46a6887c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.617+0000 7fe3b5167740 1 -- 172.21.15.16:0/2889724438 >> 172.21.15.16:0/2889724438 conn(0x55e46a67c9c0 msgr2=0x55e46a67bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:06.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.618+0000 7fe3b5167740 1 -- 172.21.15.16:0/2889724438 shutdown_connections 2024-03-28T22:13:06.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.618+0000 7fe3b5167740 1 -- 172.21.15.16:0/2889724438 wait complete. 2024-03-28T22:13:06.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.618+0000 7fe3b5167740 20 client.15990 trim_cache size 0 max 16384 2024-03-28T22:13:06.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.635+0000 7fe3b5167740 1 Processor -- start 2024-03-28T22:13:06.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.635+0000 7fe3b5167740 1 -- start start 2024-03-28T22:13:06.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.635+0000 7fe3b5167740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e46a682290 0x55e46a68aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:06.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.635+0000 7fe3b5167740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e46a682bb0 0x55e46a6972c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:06.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.635+0000 7fe3b5167740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e46a68b790 0x55e46a67baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:06.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.635+0000 7fe3b5167740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe3ac1b1e70 con 0x55e46a682bb0 2024-03-28T22:13:06.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.635+0000 7fe3b5167740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe3ac1af710 con 0x55e46a68b790 2024-03-28T22:13:06.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.635+0000 7fe3b5167740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe3ac1ad560 con 0x55e46a682290 2024-03-28T22:13:06.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.636+0000 7fe3ab7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e46a682bb0 0x55e46a6972c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:06.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.636+0000 7fe3ab7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e46a682bb0 0x55e46a6972c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55774/0 (socket says 172.21.15.16:55774) 2024-03-28T22:13:06.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.636+0000 7fe3ab7fe640 1 -- 172.21.15.16:0/3702658480 learned_addr learned my addr 172.21.15.16:0/3702658480 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:06.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.636+0000 7fe3b0b1e640 1 --2- 172.21.15.16:0/3702658480 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e46a68b790 0x55e46a67baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:06.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.636+0000 7fe3ab7fe640 1 -- 172.21.15.16:0/3702658480 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e46a682290 msgr2=0x55e46a68aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:06.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.636+0000 7fe3ab7fe640 1 --2- 172.21.15.16:0/3702658480 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e46a682290 0x55e46a68aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.636+0000 7fe3ab7fe640 1 -- 172.21.15.16:0/3702658480 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e46a68b790 msgr2=0x55e46a67baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:06.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.636+0000 7fe3ab7fe640 1 --2- 172.21.15.16:0/3702658480 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e46a68b790 0x55e46a67baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.636+0000 7fe3ab7fe640 1 -- 172.21.15.16:0/3702658480 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e46a6829e0 con 0x55e46a682bb0 2024-03-28T22:13:06.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.636+0000 7fe3ab7fe640 1 --2- 172.21.15.16:0/3702658480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e46a682bb0 0x55e46a6972c0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fe3a00045d0 tx=0x7fe3a000bb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1353b6ea6ccd34c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:06.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.636+0000 7fe391ffb640 1 -- 172.21.15.16:0/3702658480 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a0005020 con 0x55e46a682bb0 2024-03-28T22:13:06.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.636+0000 7fe391ffb640 1 -- 172.21.15.16:0/3702658480 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3a00037f0 con 0x55e46a682bb0 2024-03-28T22:13:06.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.636+0000 7fe391ffb640 1 -- 172.21.15.16:0/3702658480 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a0004bc0 con 0x55e46a682bb0 2024-03-28T22:13:06.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.637+0000 7fe3b5167740 1 -- 172.21.15.16:0/3702658480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e46a682bb0 msgr2=0x55e46a6972c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:06.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.637+0000 7fe3b5167740 1 --2- 172.21.15.16:0/3702658480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e46a682bb0 0x55e46a6972c0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fe3a00045d0 tx=0x7fe3a000bb00 comp rx=0 tx=0).stop 2024-03-28T22:13:06.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.637+0000 7fe3b5167740 1 -- 172.21.15.16:0/3702658480 shutdown_connections 2024-03-28T22:13:06.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.637+0000 7fe3b5167740 1 --2- 172.21.15.16:0/3702658480 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e46a68b790 0x55e46a67baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.637+0000 7fe3b5167740 1 --2- 172.21.15.16:0/3702658480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e46a682bb0 0x55e46a6972c0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.637+0000 7fe3b5167740 1 --2- 172.21.15.16:0/3702658480 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e46a682290 0x55e46a68aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.637+0000 7fe3b5167740 1 -- 172.21.15.16:0/3702658480 >> 172.21.15.16:0/3702658480 conn(0x55e46a67c9c0 msgr2=0x55e46a67b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:06.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.637+0000 7fe3b5167740 1 -- 172.21.15.16:0/3702658480 shutdown_connections 2024-03-28T22:13:06.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.637+0000 7fe3b5167740 1 -- 172.21.15.16:0/3702658480 wait complete. 2024-03-28T22:13:06.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.637+0000 7fe3b5167740 1 -- start start 2024-03-28T22:13:06.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.637+0000 7fe3b5167740 1 Processor -- start 2024-03-28T22:13:06.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.638+0000 7fe3b5167740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e46a682290 0x55e46a693c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:06.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.638+0000 7fe3b5167740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e46a682bb0 0x55e46a685110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:06.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.638+0000 7fe3b5167740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e46a68b790 0x55e46a685720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:06.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.638+0000 7fe3b5167740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe3ac1adbd0 con 0x55e46a68b790 2024-03-28T22:13:06.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.638+0000 7fe3b5167740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe3ac1af040 con 0x55e46a682290 2024-03-28T22:13:06.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.638+0000 7fe3b5167740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe3ac1ae2a0 con 0x55e46a682bb0 2024-03-28T22:13:06.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.638+0000 7fe3b5167740 10 client.0 _ll_register_callbacks cb 0x7fe3a404e070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:13:06.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.638+0000 7fe3b0b1e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e46a68b790 0x55e46a685720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:06.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.638+0000 7fe3abfff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e46a682290 0x55e46a693c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:06.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.638+0000 7fe3b0b1e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e46a68b790 0x55e46a685720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55790/0 (socket says 172.21.15.16:55790) 2024-03-28T22:13:06.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.638+0000 7fe3abfff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e46a682290 0x55e46a693c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:44688/0 (socket says 172.21.15.16:44688) 2024-03-28T22:13:06.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.638+0000 7fe3b0b1e640 1 -- 172.21.15.16:0/531204452 learned_addr learned my addr 172.21.15.16:0/531204452 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:06.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.638+0000 7fe3ab7fe640 1 --2- 172.21.15.16:0/531204452 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e46a682bb0 0x55e46a685110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:06.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.638+0000 7fe3b0b1e640 1 -- 172.21.15.16:0/531204452 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e46a682bb0 msgr2=0x55e46a685110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:06.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.638+0000 7fe3b0b1e640 1 --2- 172.21.15.16:0/531204452 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e46a682bb0 0x55e46a685110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.638+0000 7fe3b0b1e640 1 -- 172.21.15.16:0/531204452 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e46a682290 msgr2=0x55e46a693c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:06.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.638+0000 7fe3b0b1e640 1 --2- 172.21.15.16:0/531204452 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e46a682290 0x55e46a693c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.638+0000 7fe3b0b1e640 1 -- 172.21.15.16:0/531204452 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e46a2ba0f0 con 0x55e46a68b790 2024-03-28T22:13:06.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.638+0000 7fe3abfff640 1 --2- 172.21.15.16:0/531204452 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e46a682290 0x55e46a693c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:13:06.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.638+0000 7fe3b0b1e640 1 --2- 172.21.15.16:0/531204452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e46a68b790 0x55e46a685720 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fe3a402c630 tx=0x7fe3a402c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12239075f134785e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:06.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.638+0000 7fe38b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:13:06.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.638+0000 7fe38b7fe640 1 -- 172.21.15.16:0/531204452 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a4050040 con 0x55e46a68b790 2024-03-28T22:13:06.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.638+0000 7fe38b7fe640 1 -- 172.21.15.16:0/531204452 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fe3a4003970 con 0x55e46a68b790 2024-03-28T22:13:06.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.638+0000 7fe3b5167740 1 -- 172.21.15.16:0/531204452 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3a00037f0 con 0x55e46a68b790 2024-03-28T22:13:06.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.639+0000 7fe38b7fe640 1 -- 172.21.15.16:0/531204452 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe3a4039040 con 0x55e46a68b790 2024-03-28T22:13:06.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.639+0000 7fe3b5167740 10 client.15996 fetch_fsmap learned FSMap version 24 2024-03-28T22:13:06.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.639+0000 7fe3b5167740 1 -- 172.21.15.16:0/531204452 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e46a701930 con 0x55e46a68b790 2024-03-28T22:13:06.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.639+0000 7fe38b7fe640 1 -- 172.21.15.16:0/531204452 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3a00037f0 con 0x55e46a68b790 2024-03-28T22:13:06.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.639+0000 7fe38b7fe640 1 -- 172.21.15.16:0/531204452 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7fe3a403f040 con 0x55e46a68b790 2024-03-28T22:13:06.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.639+0000 7fe3b5167740 10 client.15996 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:13:06.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.639+0000 7fe3b5167740 10 client.15996 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:13:06.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.639+0000 7fe3b5167740 20 client.15996 populate_metadata read hostname 'smithi016' 2024-03-28T22:13:06.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.639+0000 7fe3b5167740 1 --2- 172.21.15.16:0/531204452 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e46a701fc0 0x55e46a7043e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:06.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.639+0000 7fe3b5167740 4 client.15996 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:13:06.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.639+0000 7fe3b5167740 1 -- 172.21.15.16:0/531204452 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e46a5eda10 con 0x55e46a701fc0 2024-03-28T22:13:06.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.639+0000 7fe3abfff640 1 --2- 172.21.15.16:0/531204452 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e46a701fc0 0x55e46a7043e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:06.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.639+0000 7fe38b7fe640 10 client.15996 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:13:06.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.639+0000 7fe3abfff640 1 --2- 172.21.15.16:0/531204452 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e46a701fc0 0x55e46a7043e0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6cb29d62d22cdf5c server_cookie=7ecc0da591f9ae4b in_seq=0 out_seq=0 2024-03-28T22:13:06.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.640+0000 7fe38b7fe640 1 -- 172.21.15.16:0/531204452 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e46a5eda10 con 0x55e46a701fc0 2024-03-28T22:13:06.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.640+0000 7fe38b7fe640 10 client.15996 handle_command_reply: tid=0 2024-03-28T22:13:06.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.640+0000 7fe3b5167740 1 client.15996 shutdown 2024-03-28T22:13:06.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.640+0000 7fe3b5167740 1 -- 172.21.15.16:0/531204452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e46a68b790 msgr2=0x55e46a685720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:06.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.640+0000 7fe3b5167740 1 --2- 172.21.15.16:0/531204452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e46a68b790 0x55e46a685720 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fe3a402c630 tx=0x7fe3a402c850 comp rx=0 tx=0).stop 2024-03-28T22:13:06.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.640+0000 7fe3b5167740 1 -- 172.21.15.16:0/531204452 shutdown_connections 2024-03-28T22:13:06.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.640+0000 7fe3b5167740 1 --2- 172.21.15.16:0/531204452 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e46a701fc0 0x55e46a7043e0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.640+0000 7fe3b5167740 1 --2- 172.21.15.16:0/531204452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e46a68b790 0x55e46a685720 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.640+0000 7fe3b5167740 1 --2- 172.21.15.16:0/531204452 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e46a682bb0 0x55e46a685110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.640+0000 7fe38b7fe640 0 client.15996 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:13:06.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.640+0000 7fe3b5167740 1 --2- 172.21.15.16:0/531204452 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e46a682290 0x55e46a693c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.640+0000 7fe3b5167740 1 -- 172.21.15.16:0/531204452 >> 172.21.15.16:0/531204452 conn(0x55e46a67c9c0 msgr2=0x55e46a67bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:06.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.640+0000 7fe3b5167740 1 -- 172.21.15.16:0/531204452 shutdown_connections 2024-03-28T22:13:06.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.640+0000 7fe3b5167740 1 -- 172.21.15.16:0/531204452 wait complete. 2024-03-28T22:13:06.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.640+0000 7fe3b5167740 20 client.15996 trim_cache size 0 max 16384 2024-03-28T22:13:06.653 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:13:06.653 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-03-28T22:13:06.654 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-03-28T22:13:06.654 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:13:06.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.641+0000 7fe3b2581640 1 -- 172.21.15.16:0/4046861621 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fe3940a1ef0 msgr2=0x7fe3940a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:06.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.641+0000 7fe3b2581640 1 --2- 172.21.15.16:0/4046861621 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fe3940a1ef0 0x7fe3940a43b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe3a4047000 tx=0x7fe3a4048000 comp rx=0 tx=0).stop 2024-03-28T22:13:06.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.641+0000 7fe3b2581640 1 -- 172.21.15.16:0/4046861621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3ac1016f0 msgr2=0x7fe3ac073a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:06.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.641+0000 7fe3b2581640 1 --2- 172.21.15.16:0/4046861621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3ac1016f0 0x7fe3ac073a50 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fe39c00e9f0 tx=0x7fe39c00eeb0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.641+0000 7fe3b2581640 1 -- 172.21.15.16:0/4046861621 shutdown_connections 2024-03-28T22:13:06.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.641+0000 7fe3b2581640 1 --2- 172.21.15.16:0/4046861621 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fe3940a1ef0 0x7fe3940a43b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.641+0000 7fe3b2581640 1 --2- 172.21.15.16:0/4046861621 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe3ac107700 0x7fe3ac0744d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.641+0000 7fe3b2581640 1 --2- 172.21.15.16:0/4046861621 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3ac1020b0 0x7fe3ac073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.641+0000 7fe3b2581640 1 --2- 172.21.15.16:0/4046861621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3ac1016f0 0x7fe3ac073a50 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.641+0000 7fe3b2581640 1 -- 172.21.15.16:0/4046861621 >> 172.21.15.16:0/4046861621 conn(0x7fe3ac0fd1e0 msgr2=0x7fe3ac0fd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:06.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.641+0000 7fe3b2581640 1 -- 172.21.15.16:0/4046861621 shutdown_connections 2024-03-28T22:13:06.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.641+0000 7fe3b2581640 1 -- 172.21.15.16:0/4046861621 wait complete. 2024-03-28T22:13:06.657 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:13:06.657 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:13:06.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.758+0000 7f63902c2640 1 -- 172.21.15.16:0/3978204997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63881016f0 msgr2=0x7f6388101b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:06.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.758+0000 7f63902c2640 1 --2- 172.21.15.16:0/3978204997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63881016f0 0x7f6388101b70 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f6384009930 tx=0x7f638402bc30 comp rx=0 tx=0).stop 2024-03-28T22:13:06.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.758+0000 7f63902c2640 1 -- 172.21.15.16:0/3978204997 shutdown_connections 2024-03-28T22:13:06.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.758+0000 7f63902c2640 1 --2- 172.21.15.16:0/3978204997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63881020b0 0x7f638810e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.758+0000 7f63902c2640 1 --2- 172.21.15.16:0/3978204997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63881016f0 0x7f6388101b70 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.758+0000 7f63902c2640 1 --2- 172.21.15.16:0/3978204997 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6388107700 0x7f6388107ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.758+0000 7f63902c2640 1 -- 172.21.15.16:0/3978204997 >> 172.21.15.16:0/3978204997 conn(0x7f63880fd200 msgr2=0x7f63880ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:06.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.758+0000 7f63902c2640 1 -- 172.21.15.16:0/3978204997 shutdown_connections 2024-03-28T22:13:06.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.758+0000 7f63902c2640 1 -- 172.21.15.16:0/3978204997 wait complete. 2024-03-28T22:13:06.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.759+0000 7f63902c2640 1 Processor -- start 2024-03-28T22:13:06.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.759+0000 7f63902c2640 1 -- start start 2024-03-28T22:13:06.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.759+0000 7f63902c2640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f63881016f0 0x7f6388197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:06.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.759+0000 7f63902c2640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63881020b0 0x7f63881984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:06.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.759+0000 7f63902c2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6388107700 0x7f638819c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:06.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.759+0000 7f63902c2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6388111b10 con 0x7f6388107700 2024-03-28T22:13:06.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.759+0000 7f63902c2640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f6388111990 con 0x7f63881020b0 2024-03-28T22:13:06.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.759+0000 7f63902c2640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6388111c90 con 0x7f63881016f0 2024-03-28T22:13:06.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.759+0000 7f638e85f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6388107700 0x7f638819c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:06.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.759+0000 7f638e85f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6388107700 0x7f638819c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55800/0 (socket says 172.21.15.16:55800) 2024-03-28T22:13:06.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.759+0000 7f638e85f640 1 -- 172.21.15.16:0/3207599492 learned_addr learned my addr 172.21.15.16:0/3207599492 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:06.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.759+0000 7f638d85d640 1 --2- 172.21.15.16:0/3207599492 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63881020b0 0x7f63881984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:06.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.759+0000 7f638e05e640 1 --2- 172.21.15.16:0/3207599492 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f63881016f0 0x7f6388197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:06.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.759+0000 7f638e85f640 1 -- 172.21.15.16:0/3207599492 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f63881016f0 msgr2=0x7f6388197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:06.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.759+0000 7f638e85f640 1 --2- 172.21.15.16:0/3207599492 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f63881016f0 0x7f6388197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.759+0000 7f638e85f640 1 -- 172.21.15.16:0/3207599492 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63881020b0 msgr2=0x7f63881984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:06.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.759+0000 7f638e85f640 1 --2- 172.21.15.16:0/3207599492 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63881020b0 0x7f63881984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.759+0000 7f638e85f640 1 -- 172.21.15.16:0/3207599492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f638819cf80 con 0x7f6388107700 2024-03-28T22:13:06.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.759+0000 7f638e05e640 1 --2- 172.21.15.16:0/3207599492 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f63881016f0 0x7f6388197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:13:06.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.759+0000 7f638e85f640 1 --2- 172.21.15.16:0/3207599492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6388107700 0x7f638819c850 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f638000efd0 tx=0x7f638000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:06.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.760+0000 7f637f7fe640 1 -- 172.21.15.16:0/3207599492 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6380003b70 con 0x7f6388107700 2024-03-28T22:13:06.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.760+0000 7f637f7fe640 1 -- 172.21.15.16:0/3207599492 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6380003d10 con 0x7f6388107700 2024-03-28T22:13:06.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.760+0000 7f63902c2640 1 -- 172.21.15.16:0/3207599492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f638819d270 con 0x7f6388107700 2024-03-28T22:13:06.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.760+0000 7f637f7fe640 1 -- 172.21.15.16:0/3207599492 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63800109f0 con 0x7f6388107700 2024-03-28T22:13:06.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.760+0000 7f63902c2640 1 -- 172.21.15.16:0/3207599492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63881a4b00 con 0x7f6388107700 2024-03-28T22:13:06.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.761+0000 7f63902c2640 1 -- 172.21.15.16:0/3207599492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6388068a20 con 0x7f6388107700 2024-03-28T22:13:06.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.762+0000 7f637f7fe640 1 -- 172.21.15.16:0/3207599492 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f6380010b90 con 0x7f6388107700 2024-03-28T22:13:06.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.762+0000 7f637f7fe640 1 --2- 172.21.15.16:0/3207599492 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f63740a1ef0 0x7f63740a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:06.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.762+0000 7f638e05e640 1 --2- 172.21.15.16:0/3207599492 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f63740a1ef0 0x7f63740a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:06.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.762+0000 7f637f7fe640 1 -- 172.21.15.16:0/3207599492 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f63800cea10 con 0x7f6388107700 2024-03-28T22:13:06.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.763+0000 7f638e05e640 1 --2- 172.21.15.16:0/3207599492 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f63740a1ef0 0x7f63740a43b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f637800baa0 tx=0x7f6378005d20 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:06.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.766+0000 7f637f7fe640 1 -- 172.21.15.16:0/3207599492 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f638009bb20 con 0x7f6388107700 2024-03-28T22:13:06.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.990+0000 7f63902c2640 1 -- 172.21.15.16:0/3207599492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f638802d170 con 0x7f6388107700 2024-03-28T22:13:06.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.991+0000 7f637f7fe640 1 -- 172.21.15.16:0/3207599492 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11855 (secure 0 0 0) 0x7f638009b420 con 0x7f6388107700 2024-03-28T22:13:06.993 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:13:06.993 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/1644581227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1644581227},{"type":"v1","addr":"172.21.15.16:6839","nonce":1644581227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/2593062669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2593062669},{"type":"v1","addr":"172.21.15.16:6841","nonce":2593062669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:08:29.672976+0000","modified":"2024-03-28T22:11:54.920802+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24566,"mds_1":24524,"mds_2":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24466":{"gid":24466,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/247249857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":247249857},{"type":"v1","addr":"172.21.15.63:6835","nonce":247249857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6837/2324231032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":2324231032},{"type":"v1","addr":"172.21.15.63:6837","nonce":2324231032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6839/1177197077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":1177197077},{"type":"v1","addr":"172.21.15.63:6839","nonce":1177197077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"e","rank":2,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.53:6835/350727634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":350727634},{"type":"v1","addr":"172.21.15.53:6835","nonce":350727634}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/349877573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":349877573},{"type":"v1","addr":"172.21.15.53:6837","nonce":349877573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6839/3714153956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":3714153956},{"type":"v1","addr":"172.21.15.53:6839","nonce":3714153956}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24566,"qdb_cluster":[24500,24566,24524]},"id":1}]} 2024-03-28T22:13:06.994 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 24 2024-03-28T22:13:06.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.993+0000 7f63902c2640 1 -- 172.21.15.16:0/3207599492 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f63740a1ef0 msgr2=0x7f63740a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:06.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.993+0000 7f63902c2640 1 --2- 172.21.15.16:0/3207599492 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f63740a1ef0 0x7f63740a43b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f637800baa0 tx=0x7f6378005d20 comp rx=0 tx=0).stop 2024-03-28T22:13:06.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.993+0000 7f63902c2640 1 -- 172.21.15.16:0/3207599492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6388107700 msgr2=0x7f638819c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:06.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.993+0000 7f63902c2640 1 --2- 172.21.15.16:0/3207599492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6388107700 0x7f638819c850 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f638000efd0 tx=0x7f638000c6a0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.993+0000 7f63902c2640 1 -- 172.21.15.16:0/3207599492 shutdown_connections 2024-03-28T22:13:06.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.993+0000 7f63902c2640 1 --2- 172.21.15.16:0/3207599492 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f63740a1ef0 0x7f63740a43b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.993+0000 7f63902c2640 1 --2- 172.21.15.16:0/3207599492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6388107700 0x7f638819c850 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.993+0000 7f63902c2640 1 --2- 172.21.15.16:0/3207599492 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63881020b0 0x7f63881984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.993+0000 7f63902c2640 1 --2- 172.21.15.16:0/3207599492 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f63881016f0 0x7f6388197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:06.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.993+0000 7f63902c2640 1 -- 172.21.15.16:0/3207599492 >> 172.21.15.16:0/3207599492 conn(0x7f63880fd200 msgr2=0x7f63880fd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:06.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.993+0000 7f63902c2640 1 -- 172.21.15.16:0/3207599492 shutdown_connections 2024-03-28T22:13:06.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:06.993+0000 7f63902c2640 1 -- 172.21.15.16:0/3207599492 wait complete. 2024-03-28T22:13:07.006 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:13:07.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.109+0000 7f943d11e640 1 -- 172.21.15.16:0/1072183814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9438100840 msgr2=0x7f94381062f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:07.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.109+0000 7f943d11e640 1 --2- 172.21.15.16:0/1072183814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9438100840 0x7f94381062f0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f9420009930 tx=0x7f942002bc80 comp rx=0 tx=0).stop 2024-03-28T22:13:07.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.109+0000 7f943d11e640 1 -- 172.21.15.16:0/1072183814 shutdown_connections 2024-03-28T22:13:07.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.109+0000 7f943d11e640 1 --2- 172.21.15.16:0/1072183814 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9438106a20 0x7f943810e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.109+0000 7f943d11e640 1 --2- 172.21.15.16:0/1072183814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9438100840 0x7f94381062f0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.109+0000 7f943d11e640 1 --2- 172.21.15.16:0/1072183814 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94380ffe90 0x7f9438100270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.109+0000 7f943d11e640 1 -- 172.21.15.16:0/1072183814 >> 172.21.15.16:0/1072183814 conn(0x7f9438076b50 msgr2=0x7f94380fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:07.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.109+0000 7f943d11e640 1 -- 172.21.15.16:0/1072183814 shutdown_connections 2024-03-28T22:13:07.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.109+0000 7f943d11e640 1 -- 172.21.15.16:0/1072183814 wait complete. 2024-03-28T22:13:07.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.110+0000 7f943d11e640 1 Processor -- start 2024-03-28T22:13:07.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.110+0000 7f943d11e640 1 -- start start 2024-03-28T22:13:07.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.110+0000 7f943d11e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94380ffe90 0x7f9438100e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.110+0000 7f943d11e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9438106a20 0x7f9438101340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.110+0000 7f943d11e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9438101880 0x7f94381a6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.110+0000 7f943d11e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9438111ae0 con 0x7f94380ffe90 2024-03-28T22:13:07.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.110+0000 7f943d11e640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9438111960 con 0x7f9438101880 2024-03-28T22:13:07.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.110+0000 7f943d11e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9438111c60 con 0x7f9438106a20 2024-03-28T22:13:07.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.110+0000 7f9436d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94380ffe90 0x7f9438100e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.110+0000 7f9436d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94380ffe90 0x7f9438100e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55824/0 (socket says 172.21.15.16:55824) 2024-03-28T22:13:07.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.110+0000 7f9436d9d640 1 -- 172.21.15.16:0/4167261444 learned_addr learned my addr 172.21.15.16:0/4167261444 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:07.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.110+0000 7f943759e640 1 --2- 172.21.15.16:0/4167261444 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9438101880 0x7f94381a6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.111+0000 7f942ffff640 1 --2- 172.21.15.16:0/4167261444 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9438106a20 0x7f9438101340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.111+0000 7f9436d9d640 1 -- 172.21.15.16:0/4167261444 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9438106a20 msgr2=0x7f9438101340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:07.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.111+0000 7f9436d9d640 1 --2- 172.21.15.16:0/4167261444 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9438106a20 0x7f9438101340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.111+0000 7f9436d9d640 1 -- 172.21.15.16:0/4167261444 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9438101880 msgr2=0x7f94381a6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:07.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.111+0000 7f9436d9d640 1 --2- 172.21.15.16:0/4167261444 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9438101880 0x7f94381a6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.111+0000 7f9436d9d640 1 -- 172.21.15.16:0/4167261444 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94381a6850 con 0x7f94380ffe90 2024-03-28T22:13:07.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.111+0000 7f9436d9d640 1 --2- 172.21.15.16:0/4167261444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94380ffe90 0x7f9438100e00 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f942800e9f0 tx=0x7f942800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:07.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.111+0000 7f9434d99640 1 -- 172.21.15.16:0/4167261444 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f942800cd80 con 0x7f94380ffe90 2024-03-28T22:13:07.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.111+0000 7f943d11e640 1 -- 172.21.15.16:0/4167261444 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94381a6b40 con 0x7f94380ffe90 2024-03-28T22:13:07.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.111+0000 7f9434d99640 1 -- 172.21.15.16:0/4167261444 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9428010910 con 0x7f94380ffe90 2024-03-28T22:13:07.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.111+0000 7f9434d99640 1 -- 172.21.15.16:0/4167261444 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94280188f0 con 0x7f94380ffe90 2024-03-28T22:13:07.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.111+0000 7f943d11e640 1 -- 172.21.15.16:0/4167261444 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94381a70b0 con 0x7f94380ffe90 2024-03-28T22:13:07.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.113+0000 7f9434d99640 1 -- 172.21.15.16:0/4167261444 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f9428010ac0 con 0x7f94380ffe90 2024-03-28T22:13:07.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.113+0000 7f943fddd740 1 Processor -- start 2024-03-28T22:13:07.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.114+0000 7f943fddd740 1 -- start start 2024-03-28T22:13:07.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.114+0000 7f943fddd740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f54cb0d470 0x55f54cb0d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.114+0000 7f943fddd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f54cb0dd90 0x55f54cb16430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.114+0000 7f9434d99640 1 --2- 172.21.15.16:0/4167261444 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f94240a1f20 0x7f94240a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.114+0000 7f943fddd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f54cb16970 0x55f54cb18d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.114+0000 7f943fddd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f54ca711b0 con 0x55f54cb16970 2024-03-28T22:13:07.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.114+0000 7f943fddd740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f54ca19c00 con 0x55f54cb0d470 2024-03-28T22:13:07.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.114+0000 7f943fddd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f54ca6d250 con 0x55f54cb0dd90 2024-03-28T22:13:07.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.114+0000 7f943759e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f54cb0d470 0x55f54cb0d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.114+0000 7f942ffff640 1 --2- 172.21.15.16:0/4167261444 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f94240a1f20 0x7f94240a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.114+0000 7f943759e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f54cb0d470 0x55f54cb0d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:44726/0 (socket says 172.21.15.16:44726) 2024-03-28T22:13:07.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.114+0000 7f943759e640 1 -- 172.21.15.16:0/1986680760 learned_addr learned my addr 172.21.15.16:0/1986680760 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:07.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.114+0000 7f943759e640 1 --2- 172.21.15.16:0/1986680760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f54cb16970 0x55f54cb18d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.114+0000 7f9436d9d640 1 --2- 172.21.15.16:0/1986680760 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f54cb0dd90 0x55f54cb16430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.114+0000 7f9434d99640 1 -- 172.21.15.16:0/4167261444 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f94280ce5a0 con 0x7f94380ffe90 2024-03-28T22:13:07.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.114+0000 7f943759e640 1 -- 172.21.15.16:0/1986680760 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f54cb0dd90 msgr2=0x55f54cb16430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:07.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.114+0000 7f943759e640 1 --2- 172.21.15.16:0/1986680760 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f54cb0dd90 0x55f54cb16430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.114+0000 7f943759e640 1 -- 172.21.15.16:0/1986680760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f54cb16970 msgr2=0x55f54cb18d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:07.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.114+0000 7f943759e640 1 --2- 172.21.15.16:0/1986680760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f54cb16970 0x55f54cb18d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.115+0000 7f943759e640 1 -- 172.21.15.16:0/1986680760 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f54ca393d0 con 0x55f54cb0d470 2024-03-28T22:13:07.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.115+0000 7f9436d9d640 1 --2- 172.21.15.16:0/1986680760 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f54cb0dd90 0x55f54cb16430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:13:07.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.115+0000 7f943759e640 1 --2- 172.21.15.16:0/1986680760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f54cb0d470 0x55f54cb0d850 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9430014ba0 tx=0x7f943000a940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87a5c1dc882035ec server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:07.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.115+0000 7f942ffff640 1 --2- 172.21.15.16:0/4167261444 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f94240a1f20 0x7f94240a43e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9420009900 tx=0x7f9420005fd0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:07.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.115+0000 7f942d7fa640 1 -- 172.21.15.16:0/1986680760 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f943000fdf0 con 0x55f54cb0d470 2024-03-28T22:13:07.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.115+0000 7f942d7fa640 1 -- 172.21.15.16:0/1986680760 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94300140d0 con 0x55f54cb0d470 2024-03-28T22:13:07.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.115+0000 7f942d7fa640 1 -- 172.21.15.16:0/1986680760 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9430004020 con 0x55f54cb0d470 2024-03-28T22:13:07.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.116+0000 7f943fddd740 1 -- 172.21.15.16:0/1986680760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f54cb0d470 msgr2=0x55f54cb0d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:07.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.116+0000 7f943fddd740 1 --2- 172.21.15.16:0/1986680760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f54cb0d470 0x55f54cb0d850 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9430014ba0 tx=0x7f943000a940 comp rx=0 tx=0).stop 2024-03-28T22:13:07.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.116+0000 7f943fddd740 1 -- 172.21.15.16:0/1986680760 shutdown_connections 2024-03-28T22:13:07.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.116+0000 7f943fddd740 1 --2- 172.21.15.16:0/1986680760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f54cb16970 0x55f54cb18d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.116+0000 7f943fddd740 1 --2- 172.21.15.16:0/1986680760 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f54cb0dd90 0x55f54cb16430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.116+0000 7f943fddd740 1 --2- 172.21.15.16:0/1986680760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f54cb0d470 0x55f54cb0d850 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.116+0000 7f943fddd740 1 -- 172.21.15.16:0/1986680760 >> 172.21.15.16:0/1986680760 conn(0x55f54cb07ba0 msgr2=0x55f54cb07f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:07.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.116+0000 7f943fddd740 1 -- 172.21.15.16:0/1986680760 shutdown_connections 2024-03-28T22:13:07.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.116+0000 7f943fddd740 1 -- 172.21.15.16:0/1986680760 wait complete. 2024-03-28T22:13:07.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.116+0000 7f943fddd740 1 -- start start 2024-03-28T22:13:07.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.116+0000 7f943fddd740 1 Processor -- start 2024-03-28T22:13:07.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.116+0000 7f943fddd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f54cb0d470 0x55f54cb13980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.116+0000 7f943fddd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f54cb0dd90 0x55f54cb13ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.116+0000 7f943fddd740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f54cb16970 0x55f54cb14400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.116+0000 7f943fddd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f54cb19700 con 0x55f54cb0d470 2024-03-28T22:13:07.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.116+0000 7f943fddd740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f54ca0f470 con 0x55f54cb16970 2024-03-28T22:13:07.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.116+0000 7f943fddd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f54cb19880 con 0x55f54cb0dd90 2024-03-28T22:13:07.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.116+0000 7f943fddd740 10 client.0 _ll_register_callbacks cb 0x55f54ca2ccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:13:07.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.116+0000 7f9436d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f54cb0d470 0x55f54cb13980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.116+0000 7f9436d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f54cb0d470 0x55f54cb13980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55852/0 (socket says 172.21.15.16:55852) 2024-03-28T22:13:07.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.116+0000 7f9436d9d640 1 -- 172.21.15.16:0/3155160776 learned_addr learned my addr 172.21.15.16:0/3155160776 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:07.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.117+0000 7f943759e640 1 --2- 172.21.15.16:0/3155160776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f54cb16970 0x55f54cb14400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.117+0000 7f942ffff640 1 --2- 172.21.15.16:0/3155160776 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f54cb0dd90 0x55f54cb13ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.117+0000 7f9436d9d640 1 -- 172.21.15.16:0/3155160776 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f54cb0dd90 msgr2=0x55f54cb13ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:07.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.117+0000 7f9436d9d640 1 --2- 172.21.15.16:0/3155160776 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f54cb0dd90 0x55f54cb13ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.117+0000 7f9436d9d640 1 -- 172.21.15.16:0/3155160776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f54cb16970 msgr2=0x55f54cb14400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:07.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.117+0000 7f9436d9d640 1 --2- 172.21.15.16:0/3155160776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f54cb16970 0x55f54cb14400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.117+0000 7f9436d9d640 1 -- 172.21.15.16:0/3155160776 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f54c7853f0 con 0x55f54cb0d470 2024-03-28T22:13:07.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.117+0000 7f9436d9d640 1 --2- 172.21.15.16:0/3155160776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f54cb0d470 0x55f54cb13980 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f94280973f0 tx=0x7f942809b750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6710903644970392 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:07.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.117+0000 7f941e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:13:07.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.117+0000 7f941e7fc640 1 -- 172.21.15.16:0/3155160776 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9428026040 con 0x55f54cb0d470 2024-03-28T22:13:07.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.117+0000 7f941e7fc640 1 -- 172.21.15.16:0/3155160776 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f94280cbf70 con 0x55f54cb0d470 2024-03-28T22:13:07.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.117+0000 7f943fddd740 1 -- 172.21.15.16:0/3155160776 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94300140d0 con 0x55f54cb0d470 2024-03-28T22:13:07.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.117+0000 7f941e7fc640 1 -- 172.21.15.16:0/3155160776 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f94280cc170 con 0x55f54cb0d470 2024-03-28T22:13:07.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.117+0000 7f943fddd740 10 client.16014 fetch_fsmap learned FSMap version 24 2024-03-28T22:13:07.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.117+0000 7f943fddd740 1 -- 172.21.15.16:0/3155160776 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f54c84f7d0 con 0x55f54cb0d470 2024-03-28T22:13:07.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.117+0000 7f941e7fc640 1 -- 172.21.15.16:0/3155160776 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94300140d0 con 0x55f54cb0d470 2024-03-28T22:13:07.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.118+0000 7f941e7fc640 1 -- 172.21.15.16:0/3155160776 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7f942809a3e0 con 0x55f54cb0d470 2024-03-28T22:13:07.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.118+0000 7f943fddd740 10 client.16014 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:13:07.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.118+0000 7f943fddd740 10 client.16014 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-28T22:13:07.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.118+0000 7f943fddd740 20 client.16014 populate_metadata read hostname 'smithi016' 2024-03-28T22:13:07.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.118+0000 7f943fddd740 1 --2- 172.21.15.16:0/3155160776 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55f54cb26870 0x55f54cb28cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.118+0000 7f943fddd740 4 client.16014 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:13:07.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.118+0000 7f943fddd740 1 -- 172.21.15.16:0/3155160776 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f54ca7e6f0 con 0x55f54cb26870 2024-03-28T22:13:07.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.118+0000 7f942ffff640 1 --2- 172.21.15.16:0/3155160776 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55f54cb26870 0x55f54cb28cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.118+0000 7f941e7fc640 10 client.16014 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:13:07.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.118+0000 7f942ffff640 1 --2- 172.21.15.16:0/3155160776 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55f54cb26870 0x55f54cb28cc0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d5e1d253e1eb3429 server_cookie=41ce2bda860ac6c2 in_seq=0 out_seq=0 2024-03-28T22:13:07.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.119+0000 7f941e7fc640 1 -- 172.21.15.16:0/3155160776 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f54ca7e6f0 con 0x55f54cb26870 2024-03-28T22:13:07.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.119+0000 7f941e7fc640 10 client.16014 handle_command_reply: tid=0 2024-03-28T22:13:07.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.119+0000 7f943fddd740 1 client.16014 shutdown 2024-03-28T22:13:07.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.119+0000 7f943fddd740 1 -- 172.21.15.16:0/3155160776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f54cb0d470 msgr2=0x55f54cb13980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:07.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.119+0000 7f943fddd740 1 --2- 172.21.15.16:0/3155160776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f54cb0d470 0x55f54cb13980 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f94280973f0 tx=0x7f942809b750 comp rx=0 tx=0).stop 2024-03-28T22:13:07.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.119+0000 7f943fddd740 1 -- 172.21.15.16:0/3155160776 shutdown_connections 2024-03-28T22:13:07.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.119+0000 7f943fddd740 1 --2- 172.21.15.16:0/3155160776 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55f54cb26870 0x55f54cb28cc0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.119+0000 7f943fddd740 1 --2- 172.21.15.16:0/3155160776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f54cb16970 0x55f54cb14400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.119+0000 7f941e7fc640 0 client.16014 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:13:07.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.119+0000 7f943fddd740 1 --2- 172.21.15.16:0/3155160776 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f54cb0dd90 0x55f54cb13ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.119+0000 7f943fddd740 1 --2- 172.21.15.16:0/3155160776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f54cb0d470 0x55f54cb13980 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.119+0000 7f943fddd740 1 -- 172.21.15.16:0/3155160776 >> 172.21.15.16:0/3155160776 conn(0x55f54cb07ba0 msgr2=0x55f54cb098f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:07.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.119+0000 7f943fddd740 1 -- 172.21.15.16:0/3155160776 shutdown_connections 2024-03-28T22:13:07.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.119+0000 7f943fddd740 1 -- 172.21.15.16:0/3155160776 wait complete. 2024-03-28T22:13:07.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.119+0000 7f943fddd740 20 client.16014 trim_cache size 0 max 16384 2024-03-28T22:13:07.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.132+0000 7f943fddd740 1 Processor -- start 2024-03-28T22:13:07.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.132+0000 7f943fddd740 1 -- start start 2024-03-28T22:13:07.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.133+0000 7f943fddd740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f54cb0d470 0x55f54cb160b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.133+0000 7f943fddd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f54cb0dd90 0x55f54cb224d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.133+0000 7f943fddd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f54cb16970 0x55f54cb2e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.133+0000 7f943fddd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f94381b1ee0 con 0x55f54cb16970 2024-03-28T22:13:07.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.133+0000 7f943fddd740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f94381af780 con 0x55f54cb0d470 2024-03-28T22:13:07.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.133+0000 7f943fddd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f94381ad5f0 con 0x55f54cb0dd90 2024-03-28T22:13:07.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.133+0000 7f943759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f54cb16970 0x55f54cb2e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.133+0000 7f943759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f54cb16970 0x55f54cb2e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55858/0 (socket says 172.21.15.16:55858) 2024-03-28T22:13:07.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.133+0000 7f943759e640 1 -- 172.21.15.16:0/366270171 learned_addr learned my addr 172.21.15.16:0/366270171 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:07.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.133+0000 7f9436d9d640 1 --2- 172.21.15.16:0/366270171 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f54cb0d470 0x55f54cb160b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.133+0000 7f942ffff640 1 --2- 172.21.15.16:0/366270171 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f54cb0dd90 0x55f54cb224d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.133+0000 7f943759e640 1 -- 172.21.15.16:0/366270171 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f54cb0dd90 msgr2=0x55f54cb224d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:07.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.133+0000 7f943759e640 1 --2- 172.21.15.16:0/366270171 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f54cb0dd90 0x55f54cb224d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.133+0000 7f943759e640 1 -- 172.21.15.16:0/366270171 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f54cb0d470 msgr2=0x55f54cb160b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:07.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.133+0000 7f943759e640 1 --2- 172.21.15.16:0/366270171 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f54cb0d470 0x55f54cb160b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.133+0000 7f943759e640 1 -- 172.21.15.16:0/366270171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f54cb0dbc0 con 0x55f54cb16970 2024-03-28T22:13:07.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.133+0000 7f942ffff640 1 --2- 172.21.15.16:0/366270171 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f54cb0dd90 0x55f54cb224d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:13:07.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.133+0000 7f943759e640 1 --2- 172.21.15.16:0/366270171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f54cb16970 0x55f54cb2e590 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f943000a750 tx=0x7f9430014f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=755fad216341f0d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:07.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.133+0000 7f942cff9640 1 -- 172.21.15.16:0/366270171 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9430003970 con 0x55f54cb16970 2024-03-28T22:13:07.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.134+0000 7f942cff9640 1 -- 172.21.15.16:0/366270171 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9430003b10 con 0x55f54cb16970 2024-03-28T22:13:07.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.134+0000 7f942cff9640 1 -- 172.21.15.16:0/366270171 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94300052b0 con 0x55f54cb16970 2024-03-28T22:13:07.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.134+0000 7f943fddd740 1 -- 172.21.15.16:0/366270171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f54cb16970 msgr2=0x55f54cb2e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:07.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.134+0000 7f943fddd740 1 --2- 172.21.15.16:0/366270171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f54cb16970 0x55f54cb2e590 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f943000a750 tx=0x7f9430014f60 comp rx=0 tx=0).stop 2024-03-28T22:13:07.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.134+0000 7f943fddd740 1 -- 172.21.15.16:0/366270171 shutdown_connections 2024-03-28T22:13:07.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.134+0000 7f943fddd740 1 --2- 172.21.15.16:0/366270171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f54cb16970 0x55f54cb2e590 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.134+0000 7f943fddd740 1 --2- 172.21.15.16:0/366270171 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f54cb0dd90 0x55f54cb224d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.134+0000 7f943fddd740 1 --2- 172.21.15.16:0/366270171 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f54cb0d470 0x55f54cb160b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.134+0000 7f943fddd740 1 -- 172.21.15.16:0/366270171 >> 172.21.15.16:0/366270171 conn(0x55f54cb07ba0 msgr2=0x55f54cb06d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:07.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.134+0000 7f943fddd740 1 -- 172.21.15.16:0/366270171 shutdown_connections 2024-03-28T22:13:07.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.134+0000 7f943fddd740 1 -- 172.21.15.16:0/366270171 wait complete. 2024-03-28T22:13:07.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.134+0000 7f943fddd740 1 -- start start 2024-03-28T22:13:07.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.135+0000 7f943fddd740 1 Processor -- start 2024-03-28T22:13:07.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.135+0000 7f943fddd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f54cb0d470 0x55f54cb1a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.135+0000 7f943fddd740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f54cb0dd90 0x55f54cb1acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.135+0000 7f943fddd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f54cb16970 0x55f54cb8bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.135+0000 7f943fddd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f94381adc40 con 0x55f54cb0d470 2024-03-28T22:13:07.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.135+0000 7f943fddd740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f94381af0b0 con 0x55f54cb0dd90 2024-03-28T22:13:07.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.135+0000 7f943fddd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f94381ae310 con 0x55f54cb16970 2024-03-28T22:13:07.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.135+0000 7f943fddd740 10 client.0 _ll_register_callbacks cb 0x7f94280b03b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:13:07.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.135+0000 7f9436d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f54cb0d470 0x55f54cb1a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.135+0000 7f9436d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f54cb0d470 0x55f54cb1a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55872/0 (socket says 172.21.15.16:55872) 2024-03-28T22:13:07.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.135+0000 7f9436d9d640 1 -- 172.21.15.16:0/2271338212 learned_addr learned my addr 172.21.15.16:0/2271338212 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:07.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.135+0000 7f942ffff640 1 --2- 172.21.15.16:0/2271338212 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f54cb0dd90 0x55f54cb1acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.135+0000 7f9436d9d640 1 -- 172.21.15.16:0/2271338212 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f54cb16970 msgr2=0x55f54cb8bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:07.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.135+0000 7f9436d9d640 1 --2- 172.21.15.16:0/2271338212 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f54cb16970 0x55f54cb8bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.135+0000 7f9436d9d640 1 -- 172.21.15.16:0/2271338212 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f54cb0dd90 msgr2=0x55f54cb1acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:07.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.135+0000 7f9436d9d640 1 --2- 172.21.15.16:0/2271338212 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f54cb0dd90 0x55f54cb1acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.135+0000 7f9436d9d640 1 -- 172.21.15.16:0/2271338212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f54cb1b5b0 con 0x55f54cb0d470 2024-03-28T22:13:07.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.135+0000 7f9436d9d640 1 --2- 172.21.15.16:0/2271338212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f54cb0d470 0x55f54cb1a770 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f94280a3200 tx=0x7f942809a840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa748eaa6ab538d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:07.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.135+0000 7f941e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:13:07.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.135+0000 7f941e7fc640 1 -- 172.21.15.16:0/2271338212 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f942809ad60 con 0x55f54cb0d470 2024-03-28T22:13:07.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.135+0000 7f941e7fc640 1 -- 172.21.15.16:0/2271338212 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f94280ce2d0 con 0x55f54cb0d470 2024-03-28T22:13:07.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.135+0000 7f943fddd740 1 -- 172.21.15.16:0/2271338212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9430003b10 con 0x55f54cb0d470 2024-03-28T22:13:07.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.136+0000 7f941e7fc640 1 -- 172.21.15.16:0/2271338212 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f94280cc180 con 0x55f54cb0d470 2024-03-28T22:13:07.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.136+0000 7f943fddd740 10 client.16020 fetch_fsmap learned FSMap version 24 2024-03-28T22:13:07.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.136+0000 7f943fddd740 1 -- 172.21.15.16:0/2271338212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f54cb8d1a0 con 0x55f54cb0d470 2024-03-28T22:13:07.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.136+0000 7f941e7fc640 1 -- 172.21.15.16:0/2271338212 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9430003b10 con 0x55f54cb0d470 2024-03-28T22:13:07.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.136+0000 7f941e7fc640 1 -- 172.21.15.16:0/2271338212 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7f9428095c20 con 0x55f54cb0d470 2024-03-28T22:13:07.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.136+0000 7f943fddd740 10 client.16020 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:13:07.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.136+0000 7f943fddd740 10 client.16020 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-28T22:13:07.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.136+0000 7f943fddd740 20 client.16020 populate_metadata read hostname 'smithi016' 2024-03-28T22:13:07.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.136+0000 7f943fddd740 1 --2- 172.21.15.16:0/2271338212 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55f54cb8d7b0 0x55f54cb8fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.136+0000 7f943fddd740 4 client.16020 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:13:07.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.136+0000 7f943fddd740 1 -- 172.21.15.16:0/2271338212 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f54ca78a40 con 0x55f54cb8d7b0 2024-03-28T22:13:07.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.136+0000 7f942ffff640 1 --2- 172.21.15.16:0/2271338212 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55f54cb8d7b0 0x55f54cb8fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.137+0000 7f941e7fc640 10 client.16020 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:13:07.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.137+0000 7f942ffff640 1 --2- 172.21.15.16:0/2271338212 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55f54cb8d7b0 0x55f54cb8fbf0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=be76f9aef02b8a25 server_cookie=8993e268f19d8604 in_seq=0 out_seq=0 2024-03-28T22:13:07.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.137+0000 7f941e7fc640 1 -- 172.21.15.16:0/2271338212 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f54ca78a40 con 0x55f54cb8d7b0 2024-03-28T22:13:07.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.137+0000 7f941e7fc640 10 client.16020 handle_command_reply: tid=0 2024-03-28T22:13:07.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.137+0000 7f943fddd740 1 client.16020 shutdown 2024-03-28T22:13:07.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.137+0000 7f943fddd740 1 -- 172.21.15.16:0/2271338212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f54cb0d470 msgr2=0x55f54cb1a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:07.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.137+0000 7f943fddd740 1 --2- 172.21.15.16:0/2271338212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f54cb0d470 0x55f54cb1a770 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f94280a3200 tx=0x7f942809a840 comp rx=0 tx=0).stop 2024-03-28T22:13:07.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.137+0000 7f943fddd740 1 -- 172.21.15.16:0/2271338212 shutdown_connections 2024-03-28T22:13:07.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.137+0000 7f943fddd740 1 --2- 172.21.15.16:0/2271338212 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55f54cb8d7b0 0x55f54cb8fbf0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.137+0000 7f943fddd740 1 --2- 172.21.15.16:0/2271338212 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f54cb16970 0x55f54cb8bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.137+0000 7f941e7fc640 0 client.16020 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:13:07.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.137+0000 7f943fddd740 1 --2- 172.21.15.16:0/2271338212 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f54cb0dd90 0x55f54cb1acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.137+0000 7f943fddd740 1 --2- 172.21.15.16:0/2271338212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f54cb0d470 0x55f54cb1a770 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.137+0000 7f943fddd740 1 -- 172.21.15.16:0/2271338212 >> 172.21.15.16:0/2271338212 conn(0x55f54cb07ba0 msgr2=0x55f54cb13940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:07.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.137+0000 7f943fddd740 1 -- 172.21.15.16:0/2271338212 shutdown_connections 2024-03-28T22:13:07.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.137+0000 7f943fddd740 1 -- 172.21.15.16:0/2271338212 wait complete. 2024-03-28T22:13:07.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.137+0000 7f943fddd740 20 client.16020 trim_cache size 0 max 16384 2024-03-28T22:13:07.150 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:13:07.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.137+0000 7f943d11e640 1 -- 172.21.15.16:0/4167261444 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f94240a1f20 msgr2=0x7f94240a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:07.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.137+0000 7f943d11e640 1 --2- 172.21.15.16:0/4167261444 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f94240a1f20 0x7f94240a43e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9420009900 tx=0x7f9420005fd0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.137+0000 7f943d11e640 1 -- 172.21.15.16:0/4167261444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94380ffe90 msgr2=0x7f9438100e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:07.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.137+0000 7f943d11e640 1 --2- 172.21.15.16:0/4167261444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94380ffe90 0x7f9438100e00 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f942800e9f0 tx=0x7f942800eeb0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.138+0000 7f943d11e640 1 -- 172.21.15.16:0/4167261444 shutdown_connections 2024-03-28T22:13:07.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.138+0000 7f943d11e640 1 --2- 172.21.15.16:0/4167261444 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f94240a1f20 0x7f94240a43e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.138+0000 7f943d11e640 1 --2- 172.21.15.16:0/4167261444 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9438101880 0x7f94381a6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.138+0000 7f943d11e640 1 --2- 172.21.15.16:0/4167261444 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9438106a20 0x7f9438101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.138+0000 7f943d11e640 1 --2- 172.21.15.16:0/4167261444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94380ffe90 0x7f9438100e00 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.138+0000 7f943d11e640 1 -- 172.21.15.16:0/4167261444 >> 172.21.15.16:0/4167261444 conn(0x7f9438076b50 msgr2=0x7f9438104b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:07.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.138+0000 7f943d11e640 1 -- 172.21.15.16:0/4167261444 shutdown_connections 2024-03-28T22:13:07.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.138+0000 7f943d11e640 1 -- 172.21.15.16:0/4167261444 wait complete. 2024-03-28T22:13:07.153 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:13:07.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.257+0000 7f9e68ceb640 1 -- 172.21.15.16:0/3308607496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e64103290 msgr2=0x7f9e6410b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:07.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.257+0000 7f9e68ceb640 1 --2- 172.21.15.16:0/3308607496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e64103290 0x7f9e6410b770 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c009900 tx=0x7f9e4c02bcd0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.257+0000 7f9e68ceb640 1 -- 172.21.15.16:0/3308607496 shutdown_connections 2024-03-28T22:13:07.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.257+0000 7f9e68ceb640 1 --2- 172.21.15.16:0/3308607496 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e6410bd40 0x7f9e6410e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.257+0000 7f9e68ceb640 1 --2- 172.21.15.16:0/3308607496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e64103290 0x7f9e6410b770 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.257+0000 7f9e68ceb640 1 --2- 172.21.15.16:0/3308607496 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e64102970 0x7f9e64102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.257+0000 7f9e68ceb640 1 -- 172.21.15.16:0/3308607496 >> 172.21.15.16:0/3308607496 conn(0x7f9e64077800 msgr2=0x7f9e64077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:07.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.257+0000 7f9e68ceb640 1 -- 172.21.15.16:0/3308607496 shutdown_connections 2024-03-28T22:13:07.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.257+0000 7f9e68ceb640 1 -- 172.21.15.16:0/3308607496 wait complete. 2024-03-28T22:13:07.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.257+0000 7f9e68ceb640 1 Processor -- start 2024-03-28T22:13:07.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.257+0000 7f9e68ceb640 1 -- start start 2024-03-28T22:13:07.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.258+0000 7f9e68ceb640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e64102970 0x7f9e64198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.258+0000 7f9e68ceb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e64103290 0x7f9e64198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.258+0000 7f9e68ceb640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e6410bd40 0x7f9e6419c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.258+0000 7f9e68ceb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9e64111ad0 con 0x7f9e64103290 2024-03-28T22:13:07.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.258+0000 7f9e68ceb640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9e64111950 con 0x7f9e6410bd40 2024-03-28T22:13:07.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.258+0000 7f9e68ceb640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9e64111c50 con 0x7f9e64102970 2024-03-28T22:13:07.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.258+0000 7f9e61d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e64103290 0x7f9e64198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.258+0000 7f9e61d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e64103290 0x7f9e64198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55904/0 (socket says 172.21.15.16:55904) 2024-03-28T22:13:07.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.258+0000 7f9e61d9b640 1 -- 172.21.15.16:0/1595032437 learned_addr learned my addr 172.21.15.16:0/1595032437 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:07.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.258+0000 7f9e62d9d640 1 --2- 172.21.15.16:0/1595032437 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e6410bd40 0x7f9e6419c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.258+0000 7f9e6259c640 1 --2- 172.21.15.16:0/1595032437 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e64102970 0x7f9e64198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.258+0000 7f9e61d9b640 1 -- 172.21.15.16:0/1595032437 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e64102970 msgr2=0x7f9e64198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:07.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.258+0000 7f9e61d9b640 1 --2- 172.21.15.16:0/1595032437 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e64102970 0x7f9e64198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.258+0000 7f9e61d9b640 1 -- 172.21.15.16:0/1595032437 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e6410bd40 msgr2=0x7f9e6419c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:07.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.258+0000 7f9e61d9b640 1 --2- 172.21.15.16:0/1595032437 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e6410bd40 0x7f9e6419c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.258+0000 7f9e61d9b640 1 -- 172.21.15.16:0/1595032437 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e6419cff0 con 0x7f9e64103290 2024-03-28T22:13:07.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.258+0000 7f9e61d9b640 1 --2- 172.21.15.16:0/1595032437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e64103290 0x7f9e64198590 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c005940 tx=0x7f9e4c02cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:07.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.258+0000 7f9e5b7fe640 1 -- 172.21.15.16:0/1595032437 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e4c041070 con 0x7f9e64103290 2024-03-28T22:13:07.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.258+0000 7f9e5b7fe640 1 -- 172.21.15.16:0/1595032437 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e4c004450 con 0x7f9e64103290 2024-03-28T22:13:07.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.258+0000 7f9e68ceb640 1 -- 172.21.15.16:0/1595032437 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e6419d280 con 0x7f9e64103290 2024-03-28T22:13:07.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.258+0000 7f9e5b7fe640 1 -- 172.21.15.16:0/1595032437 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e4c003df0 con 0x7f9e64103290 2024-03-28T22:13:07.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.259+0000 7f9e68ceb640 1 -- 172.21.15.16:0/1595032437 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e641a4b70 con 0x7f9e64103290 2024-03-28T22:13:07.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.260+0000 7f9e5b7fe640 1 -- 172.21.15.16:0/1595032437 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f9e4c044070 con 0x7f9e64103290 2024-03-28T22:13:07.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.261+0000 7f9e6b9a7740 1 Processor -- start 2024-03-28T22:13:07.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.261+0000 7f9e6b9a7740 1 -- start start 2024-03-28T22:13:07.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.261+0000 7f9e5b7fe640 1 --2- 172.21.15.16:0/1595032437 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f9e500a1ef0 0x7f9e500a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.262+0000 7f9e6b9a7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf8b282470 0x55bf8b282850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.262+0000 7f9e6b9a7740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bf8b282d90 0x55bf8b28b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.262+0000 7f9e6b9a7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bf8b28b970 0x55bf8b28dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.262+0000 7f9e6b9a7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55bf8b1e61b0 con 0x55bf8b282470 2024-03-28T22:13:07.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.262+0000 7f9e6b9a7740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55bf8b18ec00 con 0x55bf8b282d90 2024-03-28T22:13:07.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.262+0000 7f9e6b9a7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55bf8b1e2250 con 0x55bf8b28b970 2024-03-28T22:13:07.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.262+0000 7f9e62d9d640 1 --2- 172.21.15.16:0/1595032437 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f9e500a1ef0 0x7f9e500a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.262+0000 7f9e6259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf8b282470 0x55bf8b282850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.262+0000 7f9e5b7fe640 1 -- 172.21.15.16:0/1595032437 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f9e4c0efbd0 con 0x7f9e64103290 2024-03-28T22:13:07.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.262+0000 7f9e6259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf8b282470 0x55bf8b282850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55906/0 (socket says 172.21.15.16:55906) 2024-03-28T22:13:07.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.262+0000 7f9e6259c640 1 -- 172.21.15.16:0/4252278660 learned_addr learned my addr 172.21.15.16:0/4252278660 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:07.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.262+0000 7f9e61d9b640 1 --2- 172.21.15.16:0/4252278660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bf8b282d90 0x55bf8b28b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.262+0000 7f9e62d9d640 1 --2- 172.21.15.16:0/4252278660 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bf8b28b970 0x55bf8b28dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.262+0000 7f9e6259c640 1 -- 172.21.15.16:0/4252278660 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bf8b28b970 msgr2=0x55bf8b28dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:07.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.262+0000 7f9e6259c640 1 --2- 172.21.15.16:0/4252278660 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bf8b28b970 0x55bf8b28dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.262+0000 7f9e6259c640 1 -- 172.21.15.16:0/4252278660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bf8b282d90 msgr2=0x55bf8b28b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:07.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.262+0000 7f9e6259c640 1 --2- 172.21.15.16:0/4252278660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bf8b282d90 0x55bf8b28b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.262+0000 7f9e6259c640 1 -- 172.21.15.16:0/4252278660 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf8b1ae3d0 con 0x55bf8b282470 2024-03-28T22:13:07.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.262+0000 7f9e62d9d640 1 --2- 172.21.15.16:0/4252278660 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bf8b28b970 0x55bf8b28dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:13:07.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.262+0000 7f9e61d9b640 1 --2- 172.21.15.16:0/4252278660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bf8b282d90 0x55bf8b28b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:13:07.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.262+0000 7f9e6259c640 1 --2- 172.21.15.16:0/4252278660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf8b282470 0x55bf8b282850 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c00bd10 tx=0x7f9e5c0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee616d1ec303ecf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:07.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.262+0000 7f9e62d9d640 1 --2- 172.21.15.16:0/1595032437 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f9e500a1ef0 0x7f9e500a43b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x55bf8b27dd30 tx=0x7f9e54009880 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:07.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.262+0000 7f9e58ff9640 1 -- 172.21.15.16:0/4252278660 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c004030 con 0x55bf8b282470 2024-03-28T22:13:07.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.262+0000 7f9e58ff9640 1 -- 172.21.15.16:0/4252278660 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e5c0041d0 con 0x55bf8b282470 2024-03-28T22:13:07.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.262+0000 7f9e58ff9640 1 -- 172.21.15.16:0/4252278660 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c011820 con 0x55bf8b282470 2024-03-28T22:13:07.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.263+0000 7f9e6b9a7740 1 -- 172.21.15.16:0/4252278660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf8b282470 msgr2=0x55bf8b282850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:07.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.263+0000 7f9e6b9a7740 1 --2- 172.21.15.16:0/4252278660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf8b282470 0x55bf8b282850 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c00bd10 tx=0x7f9e5c0107a0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.263+0000 7f9e6b9a7740 1 -- 172.21.15.16:0/4252278660 shutdown_connections 2024-03-28T22:13:07.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.263+0000 7f9e6b9a7740 1 --2- 172.21.15.16:0/4252278660 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bf8b28b970 0x55bf8b28dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.263+0000 7f9e6b9a7740 1 --2- 172.21.15.16:0/4252278660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bf8b282d90 0x55bf8b28b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.263+0000 7f9e6b9a7740 1 --2- 172.21.15.16:0/4252278660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf8b282470 0x55bf8b282850 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.263+0000 7f9e6b9a7740 1 -- 172.21.15.16:0/4252278660 >> 172.21.15.16:0/4252278660 conn(0x55bf8b27cba0 msgr2=0x55bf8b27cf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:07.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.263+0000 7f9e6b9a7740 1 -- 172.21.15.16:0/4252278660 shutdown_connections 2024-03-28T22:13:07.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.263+0000 7f9e6b9a7740 1 -- 172.21.15.16:0/4252278660 wait complete. 2024-03-28T22:13:07.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.264+0000 7f9e6b9a7740 1 -- start start 2024-03-28T22:13:07.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.264+0000 7f9e6b9a7740 1 Processor -- start 2024-03-28T22:13:07.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.264+0000 7f9e6b9a7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf8b282470 0x55bf8b288990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.264+0000 7f9e6b9a7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bf8b282d90 0x55bf8b288ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.264+0000 7f9e6b9a7740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bf8b28b970 0x55bf8b289410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.264+0000 7f9e6b9a7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55bf8b28e760 con 0x55bf8b282470 2024-03-28T22:13:07.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.264+0000 7f9e6b9a7740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55bf8b184470 con 0x55bf8b28b970 2024-03-28T22:13:07.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.264+0000 7f9e6b9a7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55bf8b28e8e0 con 0x55bf8b282d90 2024-03-28T22:13:07.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.264+0000 7f9e6b9a7740 10 client.0 _ll_register_callbacks cb 0x55bf8b1a1cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:13:07.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.264+0000 7f9e6259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf8b282470 0x55bf8b288990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.264+0000 7f9e6259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf8b282470 0x55bf8b288990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55910/0 (socket says 172.21.15.16:55910) 2024-03-28T22:13:07.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.264+0000 7f9e6259c640 1 -- 172.21.15.16:0/3576741690 learned_addr learned my addr 172.21.15.16:0/3576741690 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:07.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.264+0000 7f9e61d9b640 1 --2- 172.21.15.16:0/3576741690 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bf8b282d90 0x55bf8b288ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.264+0000 7f9e62d9d640 1 --2- 172.21.15.16:0/3576741690 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bf8b28b970 0x55bf8b289410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.264+0000 7f9e6259c640 1 -- 172.21.15.16:0/3576741690 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bf8b282d90 msgr2=0x55bf8b288ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:07.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.264+0000 7f9e6259c640 1 --2- 172.21.15.16:0/3576741690 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bf8b282d90 0x55bf8b288ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.264+0000 7f9e6259c640 1 -- 172.21.15.16:0/3576741690 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bf8b28b970 msgr2=0x55bf8b289410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:07.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.264+0000 7f9e6259c640 1 --2- 172.21.15.16:0/3576741690 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bf8b28b970 0x55bf8b289410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.264+0000 7f9e6259c640 1 -- 172.21.15.16:0/3576741690 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf8aefa3f0 con 0x55bf8b282470 2024-03-28T22:13:07.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.264+0000 7f9e61d9b640 1 --2- 172.21.15.16:0/3576741690 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bf8b282d90 0x55bf8b288ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:13:07.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.264+0000 7f9e62d9d640 1 --2- 172.21.15.16:0/3576741690 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bf8b28b970 0x55bf8b289410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:13:07.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.264+0000 7f9e6259c640 1 --2- 172.21.15.16:0/3576741690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf8b282470 0x55bf8b288990 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c010ca0 tx=0x7f9e5c0058e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a617fffecaf1f2a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:07.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.264+0000 7f9e49ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:13:07.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.265+0000 7f9e49ffb640 1 -- 172.21.15.16:0/3576741690 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c029060 con 0x55bf8b282470 2024-03-28T22:13:07.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.265+0000 7f9e49ffb640 1 -- 172.21.15.16:0/3576741690 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f9e5c0035f0 con 0x55bf8b282470 2024-03-28T22:13:07.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.265+0000 7f9e6b9a7740 1 -- 172.21.15.16:0/3576741690 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e5c0041d0 con 0x55bf8b282470 2024-03-28T22:13:07.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.265+0000 7f9e49ffb640 1 -- 172.21.15.16:0/3576741690 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9e5c00b8e0 con 0x55bf8b282470 2024-03-28T22:13:07.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.265+0000 7f9e6b9a7740 10 client.16032 fetch_fsmap learned FSMap version 24 2024-03-28T22:13:07.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.265+0000 7f9e6b9a7740 1 -- 172.21.15.16:0/3576741690 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55bf8afc47d0 con 0x55bf8b282470 2024-03-28T22:13:07.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.265+0000 7f9e49ffb640 1 -- 172.21.15.16:0/3576741690 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e5c0041d0 con 0x55bf8b282470 2024-03-28T22:13:07.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.265+0000 7f9e49ffb640 1 -- 172.21.15.16:0/3576741690 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7f9e5c004990 con 0x55bf8b282470 2024-03-28T22:13:07.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.265+0000 7f9e6b9a7740 10 client.16032 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:13:07.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.265+0000 7f9e6b9a7740 10 client.16032 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T22:13:07.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.265+0000 7f9e6b9a7740 20 client.16032 populate_metadata read hostname 'smithi016' 2024-03-28T22:13:07.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.265+0000 7f9e6b9a7740 1 --2- 172.21.15.16:0/3576741690 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55bf8b29b850 0x55bf8b29dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.265+0000 7f9e6b9a7740 4 client.16032 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:13:07.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.265+0000 7f9e6b9a7740 1 -- 172.21.15.16:0/3576741690 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bf8b1f36f0 con 0x55bf8b29b850 2024-03-28T22:13:07.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.265+0000 7f9e61d9b640 1 --2- 172.21.15.16:0/3576741690 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55bf8b29b850 0x55bf8b29dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.266+0000 7f9e61d9b640 1 --2- 172.21.15.16:0/3576741690 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55bf8b29b850 0x55bf8b29dc40 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b712fc7422af1e82 server_cookie=feaec2310be74e30 in_seq=0 out_seq=0 2024-03-28T22:13:07.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.266+0000 7f9e49ffb640 10 client.16032 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:13:07.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.267+0000 7f9e49ffb640 1 -- 172.21.15.16:0/3576741690 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bf8b1f36f0 con 0x55bf8b29b850 2024-03-28T22:13:07.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.267+0000 7f9e49ffb640 10 client.16032 handle_command_reply: tid=0 2024-03-28T22:13:07.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.267+0000 7f9e6b9a7740 1 client.16032 shutdown 2024-03-28T22:13:07.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.267+0000 7f9e6b9a7740 1 -- 172.21.15.16:0/3576741690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf8b282470 msgr2=0x55bf8b288990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:07.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.267+0000 7f9e6b9a7740 1 --2- 172.21.15.16:0/3576741690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf8b282470 0x55bf8b288990 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c010ca0 tx=0x7f9e5c0058e0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.267+0000 7f9e6b9a7740 1 -- 172.21.15.16:0/3576741690 shutdown_connections 2024-03-28T22:13:07.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.267+0000 7f9e6b9a7740 1 --2- 172.21.15.16:0/3576741690 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55bf8b29b850 0x55bf8b29dc40 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.267+0000 7f9e6b9a7740 1 --2- 172.21.15.16:0/3576741690 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bf8b28b970 0x55bf8b289410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.267+0000 7f9e49ffb640 0 client.16032 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:13:07.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.267+0000 7f9e6b9a7740 1 --2- 172.21.15.16:0/3576741690 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bf8b282d90 0x55bf8b288ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.267+0000 7f9e6b9a7740 1 --2- 172.21.15.16:0/3576741690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf8b282470 0x55bf8b288990 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.267+0000 7f9e6b9a7740 1 -- 172.21.15.16:0/3576741690 >> 172.21.15.16:0/3576741690 conn(0x55bf8b27cba0 msgr2=0x55bf8b27e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:07.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.267+0000 7f9e6b9a7740 1 -- 172.21.15.16:0/3576741690 shutdown_connections 2024-03-28T22:13:07.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.267+0000 7f9e6b9a7740 1 -- 172.21.15.16:0/3576741690 wait complete. 2024-03-28T22:13:07.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.267+0000 7f9e6b9a7740 20 client.16032 trim_cache size 0 max 16384 2024-03-28T22:13:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.280+0000 7f9e6b9a7740 1 Processor -- start 2024-03-28T22:13:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.280+0000 7f9e6b9a7740 1 -- start start 2024-03-28T22:13:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.280+0000 7f9e6b9a7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf8b282470 0x55bf8b27bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.280+0000 7f9e6b9a7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bf8b282d90 0x55bf8b288270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.280+0000 7f9e6b9a7740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bf8b28b970 0x55bf8b297500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.280+0000 7f9e6b9a7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9e641afcd0 con 0x55bf8b282470 2024-03-28T22:13:07.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.280+0000 7f9e6b9a7740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9e641ad570 con 0x55bf8b28b970 2024-03-28T22:13:07.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.280+0000 7f9e6b9a7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9e641ab3e0 con 0x55bf8b282d90 2024-03-28T22:13:07.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.280+0000 7f9e6259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf8b282470 0x55bf8b27bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.280+0000 7f9e6259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf8b282470 0x55bf8b27bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55916/0 (socket says 172.21.15.16:55916) 2024-03-28T22:13:07.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.281+0000 7f9e6259c640 1 -- 172.21.15.16:0/3895624863 learned_addr learned my addr 172.21.15.16:0/3895624863 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:07.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.281+0000 7f9e62d9d640 1 --2- 172.21.15.16:0/3895624863 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bf8b28b970 0x55bf8b297500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.281+0000 7f9e61d9b640 1 --2- 172.21.15.16:0/3895624863 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bf8b282d90 0x55bf8b288270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.281+0000 7f9e6259c640 1 -- 172.21.15.16:0/3895624863 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bf8b282d90 msgr2=0x55bf8b288270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.281+0000 7f9e6259c640 1 --2- 172.21.15.16:0/3895624863 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bf8b282d90 0x55bf8b288270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.281+0000 7f9e6259c640 1 -- 172.21.15.16:0/3895624863 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bf8b28b970 msgr2=0x55bf8b297500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.281+0000 7f9e6259c640 1 --2- 172.21.15.16:0/3895624863 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bf8b28b970 0x55bf8b297500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.281+0000 7f9e6259c640 1 -- 172.21.15.16:0/3895624863 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf8b283740 con 0x55bf8b282470 2024-03-28T22:13:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.281+0000 7f9e61d9b640 1 --2- 172.21.15.16:0/3895624863 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bf8b282d90 0x55bf8b288270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:13:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.281+0000 7f9e6259c640 1 --2- 172.21.15.16:0/3895624863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf8b282470 0x55bf8b27bf80 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c00c010 tx=0x7f9e5c00b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d2f47f8217f56ec server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.281+0000 7f9e4bfff640 1 -- 172.21.15.16:0/3895624863 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c029030 con 0x55bf8b282470 2024-03-28T22:13:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.281+0000 7f9e4bfff640 1 -- 172.21.15.16:0/3895624863 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e5c003eb0 con 0x55bf8b282470 2024-03-28T22:13:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.281+0000 7f9e4bfff640 1 -- 172.21.15.16:0/3895624863 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c011cd0 con 0x55bf8b282470 2024-03-28T22:13:07.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.282+0000 7f9e6b9a7740 1 -- 172.21.15.16:0/3895624863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf8b282470 msgr2=0x55bf8b27bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:07.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.282+0000 7f9e6b9a7740 1 --2- 172.21.15.16:0/3895624863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf8b282470 0x55bf8b27bf80 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c00c010 tx=0x7f9e5c00b7f0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.282+0000 7f9e6b9a7740 1 -- 172.21.15.16:0/3895624863 shutdown_connections 2024-03-28T22:13:07.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.282+0000 7f9e6b9a7740 1 --2- 172.21.15.16:0/3895624863 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bf8b28b970 0x55bf8b297500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.282+0000 7f9e6b9a7740 1 --2- 172.21.15.16:0/3895624863 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bf8b282d90 0x55bf8b288270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.282+0000 7f9e6b9a7740 1 --2- 172.21.15.16:0/3895624863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf8b282470 0x55bf8b27bf80 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.282+0000 7f9e6b9a7740 1 -- 172.21.15.16:0/3895624863 >> 172.21.15.16:0/3895624863 conn(0x55bf8b27cba0 msgr2=0x55bf8b2891e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:07.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.282+0000 7f9e6b9a7740 1 -- 172.21.15.16:0/3895624863 shutdown_connections 2024-03-28T22:13:07.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.282+0000 7f9e6b9a7740 1 -- 172.21.15.16:0/3895624863 wait complete. 2024-03-28T22:13:07.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.282+0000 7f9e6b9a7740 1 -- start start 2024-03-28T22:13:07.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.282+0000 7f9e6b9a7740 1 Processor -- start 2024-03-28T22:13:07.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.282+0000 7f9e6b9a7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf8b282470 0x55bf8b293f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.282+0000 7f9e6b9a7740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bf8b282d90 0x55bf8b28f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.282+0000 7f9e6b9a7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bf8b28b970 0x55bf8b28f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.282+0000 7f9e6b9a7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9e641aba30 con 0x55bf8b282470 2024-03-28T22:13:07.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.282+0000 7f9e6b9a7740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9e641acea0 con 0x55bf8b282d90 2024-03-28T22:13:07.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.282+0000 7f9e6b9a7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9e641ac100 con 0x55bf8b28b970 2024-03-28T22:13:07.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.282+0000 7f9e6b9a7740 10 client.0 _ll_register_callbacks cb 0x7f9e5c003d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:13:07.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.282+0000 7f9e6259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf8b282470 0x55bf8b293f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.282+0000 7f9e6259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf8b282470 0x55bf8b293f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55920/0 (socket says 172.21.15.16:55920) 2024-03-28T22:13:07.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.282+0000 7f9e6259c640 1 -- 172.21.15.16:0/1055018371 learned_addr learned my addr 172.21.15.16:0/1055018371 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:07.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.282+0000 7f9e61d9b640 1 --2- 172.21.15.16:0/1055018371 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bf8b282d90 0x55bf8b28f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.283+0000 7f9e62d9d640 1 --2- 172.21.15.16:0/1055018371 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bf8b28b970 0x55bf8b28f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.283+0000 7f9e6259c640 1 -- 172.21.15.16:0/1055018371 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bf8b28b970 msgr2=0x55bf8b28f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:07.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.283+0000 7f9e6259c640 1 --2- 172.21.15.16:0/1055018371 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bf8b28b970 0x55bf8b28f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.283+0000 7f9e6259c640 1 -- 172.21.15.16:0/1055018371 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bf8b282d90 msgr2=0x55bf8b28f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:07.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.283+0000 7f9e6259c640 1 --2- 172.21.15.16:0/1055018371 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bf8b282d90 0x55bf8b28f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.283+0000 7f9e6259c640 1 -- 172.21.15.16:0/1055018371 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf8b304c80 con 0x55bf8b282470 2024-03-28T22:13:07.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.283+0000 7f9e6259c640 1 --2- 172.21.15.16:0/1055018371 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf8b282470 0x55bf8b293f30 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c003d30 tx=0x7f9e5c005cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42742c5da849f64a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:07.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.283+0000 7f9e49ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:13:07.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.283+0000 7f9e49ffb640 1 -- 172.21.15.16:0/1055018371 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c029030 con 0x55bf8b282470 2024-03-28T22:13:07.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.283+0000 7f9e49ffb640 1 -- 172.21.15.16:0/1055018371 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f9e5c011dc0 con 0x55bf8b282470 2024-03-28T22:13:07.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.283+0000 7f9e6b9a7740 1 -- 172.21.15.16:0/1055018371 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e5c003eb0 con 0x55bf8b282470 2024-03-28T22:13:07.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.283+0000 7f9e49ffb640 1 -- 172.21.15.16:0/1055018371 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9e5c0035f0 con 0x55bf8b282470 2024-03-28T22:13:07.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.283+0000 7f9e6b9a7740 10 client.16038 fetch_fsmap learned FSMap version 24 2024-03-28T22:13:07.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.283+0000 7f9e6b9a7740 1 -- 172.21.15.16:0/1055018371 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55bf8b305d50 con 0x55bf8b282470 2024-03-28T22:13:07.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.283+0000 7f9e49ffb640 1 -- 172.21.15.16:0/1055018371 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e5c003eb0 con 0x55bf8b282470 2024-03-28T22:13:07.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.283+0000 7f9e49ffb640 1 -- 172.21.15.16:0/1055018371 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7f9e5c005280 con 0x55bf8b282470 2024-03-28T22:13:07.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.283+0000 7f9e6b9a7740 10 client.16038 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:13:07.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.283+0000 7f9e6b9a7740 10 client.16038 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T22:13:07.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.283+0000 7f9e6b9a7740 20 client.16038 populate_metadata read hostname 'smithi016' 2024-03-28T22:13:07.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.284+0000 7f9e6b9a7740 1 --2- 172.21.15.16:0/1055018371 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55bf8b306310 0x55bf8b308730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.284+0000 7f9e6b9a7740 4 client.16038 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:13:07.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.284+0000 7f9e6b9a7740 1 -- 172.21.15.16:0/1055018371 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bf8b1eda40 con 0x55bf8b306310 2024-03-28T22:13:07.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.284+0000 7f9e61d9b640 1 --2- 172.21.15.16:0/1055018371 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55bf8b306310 0x55bf8b308730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.284+0000 7f9e61d9b640 1 --2- 172.21.15.16:0/1055018371 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55bf8b306310 0x55bf8b308730 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a4b962c37b234d2b server_cookie=2611c0de16d487e in_seq=0 out_seq=0 2024-03-28T22:13:07.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.284+0000 7f9e49ffb640 10 client.16038 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:13:07.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.284+0000 7f9e49ffb640 1 -- 172.21.15.16:0/1055018371 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bf8b1eda40 con 0x55bf8b306310 2024-03-28T22:13:07.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.284+0000 7f9e49ffb640 10 client.16038 handle_command_reply: tid=0 2024-03-28T22:13:07.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.284+0000 7f9e6b9a7740 1 client.16038 shutdown 2024-03-28T22:13:07.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.284+0000 7f9e6b9a7740 1 -- 172.21.15.16:0/1055018371 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf8b282470 msgr2=0x55bf8b293f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:07.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.284+0000 7f9e6b9a7740 1 --2- 172.21.15.16:0/1055018371 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf8b282470 0x55bf8b293f30 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c003d30 tx=0x7f9e5c005cc0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.284+0000 7f9e6b9a7740 1 -- 172.21.15.16:0/1055018371 shutdown_connections 2024-03-28T22:13:07.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.284+0000 7f9e6b9a7740 1 --2- 172.21.15.16:0/1055018371 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55bf8b306310 0x55bf8b308730 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.284+0000 7f9e6b9a7740 1 --2- 172.21.15.16:0/1055018371 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bf8b28b970 0x55bf8b28f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.284+0000 7f9e6b9a7740 1 --2- 172.21.15.16:0/1055018371 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bf8b282d90 0x55bf8b28f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.284+0000 7f9e49ffb640 0 client.16038 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:13:07.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.284+0000 7f9e6b9a7740 1 --2- 172.21.15.16:0/1055018371 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bf8b282470 0x55bf8b293f30 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.284+0000 7f9e6b9a7740 1 -- 172.21.15.16:0/1055018371 >> 172.21.15.16:0/1055018371 conn(0x55bf8b27cba0 msgr2=0x55bf8b288910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:07.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.284+0000 7f9e6b9a7740 1 -- 172.21.15.16:0/1055018371 shutdown_connections 2024-03-28T22:13:07.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.285+0000 7f9e6b9a7740 1 -- 172.21.15.16:0/1055018371 wait complete. 2024-03-28T22:13:07.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.285+0000 7f9e6b9a7740 20 client.16038 trim_cache size 0 max 16384 2024-03-28T22:13:07.299 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:13:07.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.285+0000 7f9e68ceb640 1 -- 172.21.15.16:0/1595032437 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f9e500a1ef0 msgr2=0x7f9e500a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:07.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.285+0000 7f9e68ceb640 1 --2- 172.21.15.16:0/1595032437 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f9e500a1ef0 0x7f9e500a43b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x55bf8b27dd30 tx=0x7f9e54009880 comp rx=0 tx=0).stop 2024-03-28T22:13:07.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.285+0000 7f9e68ceb640 1 -- 172.21.15.16:0/1595032437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e64103290 msgr2=0x7f9e64198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:07.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.285+0000 7f9e68ceb640 1 --2- 172.21.15.16:0/1595032437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e64103290 0x7f9e64198590 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c005940 tx=0x7f9e4c02cc40 comp rx=0 tx=0).stop 2024-03-28T22:13:07.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.285+0000 7f9e68ceb640 1 -- 172.21.15.16:0/1595032437 shutdown_connections 2024-03-28T22:13:07.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.285+0000 7f9e68ceb640 1 --2- 172.21.15.16:0/1595032437 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f9e500a1ef0 0x7f9e500a43b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.285+0000 7f9e68ceb640 1 --2- 172.21.15.16:0/1595032437 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e6410bd40 0x7f9e6419c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.285+0000 7f9e68ceb640 1 --2- 172.21.15.16:0/1595032437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e64103290 0x7f9e64198590 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.285+0000 7f9e68ceb640 1 --2- 172.21.15.16:0/1595032437 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e64102970 0x7f9e64198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.285+0000 7f9e68ceb640 1 -- 172.21.15.16:0/1595032437 >> 172.21.15.16:0/1595032437 conn(0x7f9e64077800 msgr2=0x7f9e640fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:07.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.285+0000 7f9e68ceb640 1 -- 172.21.15.16:0/1595032437 shutdown_connections 2024-03-28T22:13:07.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.285+0000 7f9e68ceb640 1 -- 172.21.15.16:0/1595032437 wait complete. 2024-03-28T22:13:07.302 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:13:07.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.405+0000 7fd3eab65640 1 -- 172.21.15.16:0/1919953752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3e41016d0 msgr2=0x7fd3e4101b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:07.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.405+0000 7fd3eab65640 1 --2- 172.21.15.16:0/1919953752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3e41016d0 0x7fd3e4101b50 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0009930 tx=0x7fd3d002bc30 comp rx=0 tx=0).stop 2024-03-28T22:13:07.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.406+0000 7fd3eab65640 1 -- 172.21.15.16:0/1919953752 shutdown_connections 2024-03-28T22:13:07.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.406+0000 7fd3eab65640 1 --2- 172.21.15.16:0/1919953752 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3e4102090 0x7fd3e410e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.406+0000 7fd3eab65640 1 --2- 172.21.15.16:0/1919953752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3e41016d0 0x7fd3e4101b50 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.406+0000 7fd3eab65640 1 --2- 172.21.15.16:0/1919953752 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3e41076e0 0x7fd3e4107ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.406+0000 7fd3eab65640 1 -- 172.21.15.16:0/1919953752 >> 172.21.15.16:0/1919953752 conn(0x7fd3e40fd1c0 msgr2=0x7fd3e40ff5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:07.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.406+0000 7fd3eab65640 1 -- 172.21.15.16:0/1919953752 shutdown_connections 2024-03-28T22:13:07.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.406+0000 7fd3eab65640 1 -- 172.21.15.16:0/1919953752 wait complete. 2024-03-28T22:13:07.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.406+0000 7fd3eab65640 1 Processor -- start 2024-03-28T22:13:07.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.406+0000 7fd3eab65640 1 -- start start 2024-03-28T22:13:07.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.406+0000 7fd3eab65640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3e41016d0 0x7fd3e4198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.406+0000 7fd3eab65640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3e4102090 0x7fd3e4198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.406+0000 7fd3eab65640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3e41076e0 0x7fd3e419c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.406+0000 7fd3eab65640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e4111ab0 con 0x7fd3e4102090 2024-03-28T22:13:07.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.406+0000 7fd3eab65640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e4111930 con 0x7fd3e41076e0 2024-03-28T22:13:07.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.406+0000 7fd3eab65640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e4111c30 con 0x7fd3e41016d0 2024-03-28T22:13:07.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.406+0000 7fd3e3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3e4102090 0x7fd3e4198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.406+0000 7fd3e3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3e4102090 0x7fd3e4198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55932/0 (socket says 172.21.15.16:55932) 2024-03-28T22:13:07.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.406+0000 7fd3e3fff640 1 -- 172.21.15.16:0/3867722022 learned_addr learned my addr 172.21.15.16:0/3867722022 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.407+0000 7fd3e8901640 1 --2- 172.21.15.16:0/3867722022 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3e41016d0 0x7fd3e4198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.407+0000 7fd3e9102640 1 --2- 172.21.15.16:0/3867722022 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3e41076e0 0x7fd3e419c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.407+0000 7fd3e3fff640 1 -- 172.21.15.16:0/3867722022 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3e41016d0 msgr2=0x7fd3e4198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.407+0000 7fd3e3fff640 1 --2- 172.21.15.16:0/3867722022 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3e41016d0 0x7fd3e4198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.407+0000 7fd3e3fff640 1 -- 172.21.15.16:0/3867722022 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3e41076e0 msgr2=0x7fd3e419c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.407+0000 7fd3e3fff640 1 --2- 172.21.15.16:0/3867722022 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3e41076e0 0x7fd3e419c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.407+0000 7fd3e3fff640 1 -- 172.21.15.16:0/3867722022 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3e419d000 con 0x7fd3e4102090 2024-03-28T22:13:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.407+0000 7fd3e3fff640 1 --2- 172.21.15.16:0/3867722022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3e4102090 0x7fd3e4198550 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0009230 tx=0x7fd3d002cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.407+0000 7fd3e1ffb640 1 -- 172.21.15.16:0/3867722022 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d0041070 con 0x7fd3e4102090 2024-03-28T22:13:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.407+0000 7fd3eab65640 1 -- 172.21.15.16:0/3867722022 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3e419d290 con 0x7fd3e4102090 2024-03-28T22:13:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.407+0000 7fd3e1ffb640 1 -- 172.21.15.16:0/3867722022 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3d0004450 con 0x7fd3e4102090 2024-03-28T22:13:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.407+0000 7fd3e1ffb640 1 -- 172.21.15.16:0/3867722022 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d0034440 con 0x7fd3e4102090 2024-03-28T22:13:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.407+0000 7fd3eab65640 1 -- 172.21.15.16:0/3867722022 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3e41a4b80 con 0x7fd3e4102090 2024-03-28T22:13:07.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.410+0000 7fd3e1ffb640 1 -- 172.21.15.16:0/3867722022 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7fd3d0044070 con 0x7fd3e4102090 2024-03-28T22:13:07.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.410+0000 7fd3ed828740 1 Processor -- start 2024-03-28T22:13:07.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.410+0000 7fd3ed828740 1 -- start start 2024-03-28T22:13:07.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.410+0000 7fd3ed828740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571ecaa5470 0x5571ecaa5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.410+0000 7fd3ed828740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5571ecaa5d90 0x5571ecaae430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.410+0000 7fd3ed828740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5571ecaae970 0x5571ecab0d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.410+0000 7fd3ed828740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5571eca091b0 con 0x5571ecaa5d90 2024-03-28T22:13:07.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.410+0000 7fd3ed828740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5571ec9b1c00 con 0x5571ecaae970 2024-03-28T22:13:07.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.411+0000 7fd3ed828740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5571eca05250 con 0x5571ecaa5470 2024-03-28T22:13:07.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.411+0000 7fd3e8901640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5571ecaa5d90 0x5571ecaae430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.411+0000 7fd3e8901640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5571ecaa5d90 0x5571ecaae430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55940/0 (socket says 172.21.15.16:55940) 2024-03-28T22:13:07.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.411+0000 7fd3e8901640 1 -- 172.21.15.16:0/717832550 learned_addr learned my addr 172.21.15.16:0/717832550 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:07.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.411+0000 7fd3e1ffb640 1 --2- 172.21.15.16:0/3867722022 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fd3d40a1ef0 0x7fd3d40a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.411+0000 7fd3e3fff640 1 --2- 172.21.15.16:0/717832550 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5571ecaae970 0x5571ecab0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.411+0000 7fd3e9102640 1 --2- 172.21.15.16:0/3867722022 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fd3d40a1ef0 0x7fd3d40a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.411+0000 7fd3e9102640 1 --2- 172.21.15.16:0/717832550 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571ecaa5470 0x5571ecaa5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.411+0000 7fd3e8901640 1 -- 172.21.15.16:0/717832550 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571ecaa5470 msgr2=0x5571ecaa5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:07.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.411+0000 7fd3e8901640 1 --2- 172.21.15.16:0/717832550 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571ecaa5470 0x5571ecaa5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.411+0000 7fd3e8901640 1 -- 172.21.15.16:0/717832550 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5571ecaae970 msgr2=0x5571ecab0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:07.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.411+0000 7fd3e8901640 1 --2- 172.21.15.16:0/717832550 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5571ecaae970 0x5571ecab0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.411+0000 7fd3e8901640 1 -- 172.21.15.16:0/717832550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571ec9d13d0 con 0x5571ecaa5d90 2024-03-28T22:13:07.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.411+0000 7fd3e1ffb640 1 -- 172.21.15.16:0/3867722022 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fd3d00efa50 con 0x7fd3e4102090 2024-03-28T22:13:07.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.411+0000 7fd3e8901640 1 --2- 172.21.15.16:0/717832550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5571ecaa5d90 0x5571ecaae430 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8009c70 tx=0x7fd3d8011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f1f750337f90f80 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:07.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.411+0000 7fd3e9102640 1 --2- 172.21.15.16:0/3867722022 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fd3d40a1ef0 0x7fd3d40a43b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc00a780 tx=0x7fd3dc005f60 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:07.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.411+0000 7fd3ceffd640 1 -- 172.21.15.16:0/717832550 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d8004230 con 0x5571ecaa5d90 2024-03-28T22:13:07.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.411+0000 7fd3ceffd640 1 -- 172.21.15.16:0/717832550 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3d80043d0 con 0x5571ecaa5d90 2024-03-28T22:13:07.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.411+0000 7fd3ceffd640 1 -- 172.21.15.16:0/717832550 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d80197c0 con 0x5571ecaa5d90 2024-03-28T22:13:07.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.412+0000 7fd3ed828740 1 -- 172.21.15.16:0/717832550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5571ecaa5d90 msgr2=0x5571ecaae430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:07.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.412+0000 7fd3ed828740 1 --2- 172.21.15.16:0/717832550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5571ecaa5d90 0x5571ecaae430 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8009c70 tx=0x7fd3d8011810 comp rx=0 tx=0).stop 2024-03-28T22:13:07.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.412+0000 7fd3ed828740 1 -- 172.21.15.16:0/717832550 shutdown_connections 2024-03-28T22:13:07.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.412+0000 7fd3ed828740 1 --2- 172.21.15.16:0/717832550 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5571ecaae970 0x5571ecab0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.412+0000 7fd3ed828740 1 --2- 172.21.15.16:0/717832550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5571ecaa5d90 0x5571ecaae430 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.412+0000 7fd3ed828740 1 --2- 172.21.15.16:0/717832550 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571ecaa5470 0x5571ecaa5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.412+0000 7fd3ed828740 1 -- 172.21.15.16:0/717832550 >> 172.21.15.16:0/717832550 conn(0x5571eca9fba0 msgr2=0x5571eca9ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:07.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.412+0000 7fd3ed828740 1 -- 172.21.15.16:0/717832550 shutdown_connections 2024-03-28T22:13:07.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.412+0000 7fd3ed828740 1 -- 172.21.15.16:0/717832550 wait complete. 2024-03-28T22:13:07.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.412+0000 7fd3ed828740 1 -- start start 2024-03-28T22:13:07.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.412+0000 7fd3ed828740 1 Processor -- start 2024-03-28T22:13:07.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.413+0000 7fd3ed828740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571ecaa5470 0x5571ecaab970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.413+0000 7fd3ed828740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5571ecaa5d90 0x5571ecaabeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.413+0000 7fd3ed828740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5571ecaae970 0x5571ecaac3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.413+0000 7fd3ed828740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5571ecab1760 con 0x5571ecaa5d90 2024-03-28T22:13:07.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.413+0000 7fd3ed828740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5571ec9a7470 con 0x5571ecaae970 2024-03-28T22:13:07.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.413+0000 7fd3ed828740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5571ecab18e0 con 0x5571ecaa5470 2024-03-28T22:13:07.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.413+0000 7fd3ed828740 10 client.0 _ll_register_callbacks cb 0x5571ec9c4cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:13:07.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.413+0000 7fd3e3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5571ecaa5d90 0x5571ecaabeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.413+0000 7fd3e3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5571ecaa5d90 0x5571ecaabeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55954/0 (socket says 172.21.15.16:55954) 2024-03-28T22:13:07.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.413+0000 7fd3e3fff640 1 -- 172.21.15.16:0/3631825132 learned_addr learned my addr 172.21.15.16:0/3631825132 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:07.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.413+0000 7fd3e9102640 1 --2- 172.21.15.16:0/3631825132 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5571ecaae970 0x5571ecaac3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.413+0000 7fd3e3fff640 1 -- 172.21.15.16:0/3631825132 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571ecaa5470 msgr2=0x5571ecaab970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:07.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.413+0000 7fd3e8901640 1 --2- 172.21.15.16:0/3631825132 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571ecaa5470 0x5571ecaab970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.413+0000 7fd3e3fff640 1 --2- 172.21.15.16:0/3631825132 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571ecaa5470 0x5571ecaab970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.413+0000 7fd3e3fff640 1 -- 172.21.15.16:0/3631825132 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5571ecaae970 msgr2=0x5571ecaac3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:07.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.413+0000 7fd3e3fff640 1 --2- 172.21.15.16:0/3631825132 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5571ecaae970 0x5571ecaac3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.413+0000 7fd3e3fff640 1 -- 172.21.15.16:0/3631825132 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5571ec71d3f0 con 0x5571ecaa5d90 2024-03-28T22:13:07.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.413+0000 7fd3e3fff640 1 --2- 172.21.15.16:0/3631825132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5571ecaa5d90 0x5571ecaabeb0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0009430 tx=0x7fd3d002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a28799ee5ef9fcb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:07.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.413+0000 7fd3c3fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:13:07.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.414+0000 7fd3c3fff640 1 -- 172.21.15.16:0/3631825132 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d004b040 con 0x5571ecaa5d90 2024-03-28T22:13:07.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.414+0000 7fd3c3fff640 1 -- 172.21.15.16:0/3631825132 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fd3d00ed3e0 con 0x5571ecaa5d90 2024-03-28T22:13:07.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.414+0000 7fd3ed828740 1 -- 172.21.15.16:0/3631825132 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3d80043d0 con 0x5571ecaa5d90 2024-03-28T22:13:07.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.414+0000 7fd3c3fff640 1 -- 172.21.15.16:0/3631825132 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd3d00bc100 con 0x5571ecaa5d90 2024-03-28T22:13:07.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.414+0000 7fd3ed828740 10 client.16050 fetch_fsmap learned FSMap version 24 2024-03-28T22:13:07.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.414+0000 7fd3ed828740 1 -- 172.21.15.16:0/3631825132 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5571ec7e77d0 con 0x5571ecaa5d90 2024-03-28T22:13:07.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.414+0000 7fd3c3fff640 1 -- 172.21.15.16:0/3631825132 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3d80043d0 con 0x5571ecaa5d90 2024-03-28T22:13:07.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.414+0000 7fd3c3fff640 1 -- 172.21.15.16:0/3631825132 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7fd3d00c4820 con 0x5571ecaa5d90 2024-03-28T22:13:07.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.414+0000 7fd3ed828740 10 client.16050 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:13:07.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.414+0000 7fd3ed828740 10 client.16050 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:13:07.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.414+0000 7fd3ed828740 20 client.16050 populate_metadata read hostname 'smithi016' 2024-03-28T22:13:07.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.414+0000 7fd3ed828740 1 --2- 172.21.15.16:0/3631825132 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5571ecabe880 0x5571ecac0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.414+0000 7fd3ed828740 4 client.16050 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:13:07.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.414+0000 7fd3ed828740 1 -- 172.21.15.16:0/3631825132 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5571eca166f0 con 0x5571ecabe880 2024-03-28T22:13:07.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.414+0000 7fd3e8901640 1 --2- 172.21.15.16:0/3631825132 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5571ecabe880 0x5571ecac0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.415+0000 7fd3c3fff640 10 client.16050 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:13:07.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.415+0000 7fd3e8901640 1 --2- 172.21.15.16:0/3631825132 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5571ecabe880 0x5571ecac0ce0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5dc016bfe52c3a43 server_cookie=a76747033d33c8c5 in_seq=0 out_seq=0 2024-03-28T22:13:07.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.416+0000 7fd3c3fff640 1 -- 172.21.15.16:0/3631825132 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5571eca166f0 con 0x5571ecabe880 2024-03-28T22:13:07.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.416+0000 7fd3c3fff640 10 client.16050 handle_command_reply: tid=0 2024-03-28T22:13:07.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.416+0000 7fd3ed828740 1 client.16050 shutdown 2024-03-28T22:13:07.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.416+0000 7fd3ed828740 1 -- 172.21.15.16:0/3631825132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5571ecaa5d90 msgr2=0x5571ecaabeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:07.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.416+0000 7fd3ed828740 1 --2- 172.21.15.16:0/3631825132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5571ecaa5d90 0x5571ecaabeb0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0009430 tx=0x7fd3d002bc30 comp rx=0 tx=0).stop 2024-03-28T22:13:07.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.416+0000 7fd3ed828740 1 -- 172.21.15.16:0/3631825132 shutdown_connections 2024-03-28T22:13:07.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.416+0000 7fd3ed828740 1 --2- 172.21.15.16:0/3631825132 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5571ecabe880 0x5571ecac0ce0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.416+0000 7fd3ed828740 1 --2- 172.21.15.16:0/3631825132 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5571ecaae970 0x5571ecaac3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.416+0000 7fd3c3fff640 0 client.16050 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:13:07.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.416+0000 7fd3ed828740 1 --2- 172.21.15.16:0/3631825132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5571ecaa5d90 0x5571ecaabeb0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.416+0000 7fd3ed828740 1 --2- 172.21.15.16:0/3631825132 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571ecaa5470 0x5571ecaab970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.416+0000 7fd3ed828740 1 -- 172.21.15.16:0/3631825132 >> 172.21.15.16:0/3631825132 conn(0x5571eca9fba0 msgr2=0x5571ecaa18f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:07.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.416+0000 7fd3ed828740 1 -- 172.21.15.16:0/3631825132 shutdown_connections 2024-03-28T22:13:07.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.416+0000 7fd3ed828740 1 -- 172.21.15.16:0/3631825132 wait complete. 2024-03-28T22:13:07.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.416+0000 7fd3ed828740 20 client.16050 trim_cache size 0 max 16384 2024-03-28T22:13:07.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.429+0000 7fd3ed828740 1 Processor -- start 2024-03-28T22:13:07.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.429+0000 7fd3ed828740 1 -- start start 2024-03-28T22:13:07.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.429+0000 7fd3ed828740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5571ecaa5470 0x5571ecaba500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.429+0000 7fd3ed828740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571ecaa5d90 0x5571ecac65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.429+0000 7fd3ed828740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5571ecaae970 0x5571ecaa8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.429+0000 7fd3ed828740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e41afd10 con 0x5571ecaae970 2024-03-28T22:13:07.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.429+0000 7fd3ed828740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e41ad5b0 con 0x5571ecaa5470 2024-03-28T22:13:07.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.429+0000 7fd3ed828740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e41ab400 con 0x5571ecaa5d90 2024-03-28T22:13:07.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.429+0000 7fd3e9102640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5571ecaae970 0x5571ecaa8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.430+0000 7fd3e9102640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5571ecaae970 0x5571ecaa8bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55968/0 (socket says 172.21.15.16:55968) 2024-03-28T22:13:07.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.430+0000 7fd3e9102640 1 -- 172.21.15.16:0/3766821868 learned_addr learned my addr 172.21.15.16:0/3766821868 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:07.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.430+0000 7fd3e8901640 1 --2- 172.21.15.16:0/3766821868 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5571ecaa5470 0x5571ecaba500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.430+0000 7fd3e9102640 1 -- 172.21.15.16:0/3766821868 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571ecaa5d90 msgr2=0x5571ecac65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:07.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.430+0000 7fd3e9102640 1 --2- 172.21.15.16:0/3766821868 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571ecaa5d90 0x5571ecac65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.430+0000 7fd3e9102640 1 -- 172.21.15.16:0/3766821868 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5571ecaa5470 msgr2=0x5571ecaba500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:07.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.430+0000 7fd3e9102640 1 --2- 172.21.15.16:0/3766821868 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5571ecaa5470 0x5571ecaba500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.430+0000 7fd3e9102640 1 -- 172.21.15.16:0/3766821868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571ecac1360 con 0x5571ecaae970 2024-03-28T22:13:07.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.430+0000 7fd3e9102640 1 --2- 172.21.15.16:0/3766821868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5571ecaae970 0x5571ecaa8bd0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc00ae00 tx=0x7fd3dc014070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bec6d91f6698b27c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:07.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.430+0000 7fd3ce7fc640 1 -- 172.21.15.16:0/3766821868 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc01d820 con 0x5571ecaae970 2024-03-28T22:13:07.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.430+0000 7fd3ce7fc640 1 -- 172.21.15.16:0/3766821868 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3dc016070 con 0x5571ecaae970 2024-03-28T22:13:07.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.430+0000 7fd3ce7fc640 1 -- 172.21.15.16:0/3766821868 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc0193b0 con 0x5571ecaae970 2024-03-28T22:13:07.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.431+0000 7fd3ed828740 1 -- 172.21.15.16:0/3766821868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5571ecaae970 msgr2=0x5571ecaa8bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:07.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.431+0000 7fd3ed828740 1 --2- 172.21.15.16:0/3766821868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5571ecaae970 0x5571ecaa8bd0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc00ae00 tx=0x7fd3dc014070 comp rx=0 tx=0).stop 2024-03-28T22:13:07.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.431+0000 7fd3ed828740 1 -- 172.21.15.16:0/3766821868 shutdown_connections 2024-03-28T22:13:07.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.431+0000 7fd3ed828740 1 --2- 172.21.15.16:0/3766821868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5571ecaae970 0x5571ecaa8bd0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.431+0000 7fd3ed828740 1 --2- 172.21.15.16:0/3766821868 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571ecaa5d90 0x5571ecac65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.431+0000 7fd3ed828740 1 --2- 172.21.15.16:0/3766821868 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5571ecaa5470 0x5571ecaba500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.431+0000 7fd3ed828740 1 -- 172.21.15.16:0/3766821868 >> 172.21.15.16:0/3766821868 conn(0x5571eca9fba0 msgr2=0x5571ecaad920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:07.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.431+0000 7fd3ed828740 1 -- 172.21.15.16:0/3766821868 shutdown_connections 2024-03-28T22:13:07.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.431+0000 7fd3ed828740 1 -- 172.21.15.16:0/3766821868 wait complete. 2024-03-28T22:13:07.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.431+0000 7fd3ed828740 1 -- start start 2024-03-28T22:13:07.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.431+0000 7fd3ed828740 1 Processor -- start 2024-03-28T22:13:07.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.431+0000 7fd3ed828740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5571ecaa5470 0x5571ecb1ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.431+0000 7fd3ed828740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571ecaa5d90 0x5571ecb1d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.431+0000 7fd3ed828740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5571ecaae970 0x5571ecab2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.431+0000 7fd3ed828740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e41aba70 con 0x5571ecaae970 2024-03-28T22:13:07.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.431+0000 7fd3ed828740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e41acee0 con 0x5571ecaa5470 2024-03-28T22:13:07.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.431+0000 7fd3ed828740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e41ac140 con 0x5571ecaa5d90 2024-03-28T22:13:07.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.431+0000 7fd3ed828740 10 client.0 _ll_register_callbacks cb 0x7fd3d0005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:13:07.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.432+0000 7fd3e9102640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5571ecaae970 0x5571ecab2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.432+0000 7fd3e8901640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5571ecaa5470 0x5571ecb1ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.432+0000 7fd3e9102640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5571ecaae970 0x5571ecab2d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55972/0 (socket says 172.21.15.16:55972) 2024-03-28T22:13:07.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.432+0000 7fd3e9102640 1 -- 172.21.15.16:0/2430418424 learned_addr learned my addr 172.21.15.16:0/2430418424 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:07.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.432+0000 7fd3e8901640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5571ecaa5470 0x5571ecb1ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:44858/0 (socket says 172.21.15.16:44858) 2024-03-28T22:13:07.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.432+0000 7fd3e3fff640 1 --2- 172.21.15.16:0/2430418424 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571ecaa5d90 0x5571ecb1d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.432+0000 7fd3e9102640 1 -- 172.21.15.16:0/2430418424 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571ecaa5d90 msgr2=0x5571ecb1d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:07.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.432+0000 7fd3e9102640 1 --2- 172.21.15.16:0/2430418424 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571ecaa5d90 0x5571ecb1d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.432+0000 7fd3e9102640 1 -- 172.21.15.16:0/2430418424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5571ecaa5470 msgr2=0x5571ecb1ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:07.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.432+0000 7fd3e9102640 1 --2- 172.21.15.16:0/2430418424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5571ecaa5470 0x5571ecb1ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.432+0000 7fd3e9102640 1 -- 172.21.15.16:0/2430418424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5571ecab3460 con 0x5571ecaae970 2024-03-28T22:13:07.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.432+0000 7fd3e3fff640 1 --2- 172.21.15.16:0/2430418424 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571ecaa5d90 0x5571ecb1d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:13:07.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.432+0000 7fd3e8901640 1 --2- 172.21.15.16:0/2430418424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5571ecaa5470 0x5571ecb1ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:13:07.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.432+0000 7fd3e9102640 1 --2- 172.21.15.16:0/2430418424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5571ecaae970 0x5571ecab2d70 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc028dd0 tx=0x7fd3dc011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b232293dad9d2840 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:07.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.432+0000 7fd3c3fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:13:07.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.432+0000 7fd3c3fff640 1 -- 172.21.15.16:0/2430418424 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc004e50 con 0x5571ecaae970 2024-03-28T22:13:07.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.432+0000 7fd3c3fff640 1 -- 172.21.15.16:0/2430418424 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fd3dc013040 con 0x5571ecaae970 2024-03-28T22:13:07.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.432+0000 7fd3ed828740 1 -- 172.21.15.16:0/2430418424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3dc016070 con 0x5571ecaae970 2024-03-28T22:13:07.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.432+0000 7fd3c3fff640 1 -- 172.21.15.16:0/2430418424 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd3dc03e050 con 0x5571ecaae970 2024-03-28T22:13:07.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.432+0000 7fd3ed828740 10 client.16056 fetch_fsmap learned FSMap version 24 2024-03-28T22:13:07.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.432+0000 7fd3ed828740 1 -- 172.21.15.16:0/2430418424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5571ecb257a0 con 0x5571ecaae970 2024-03-28T22:13:07.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.433+0000 7fd3c3fff640 1 -- 172.21.15.16:0/2430418424 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3dc016070 con 0x5571ecaae970 2024-03-28T22:13:07.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.433+0000 7fd3c3fff640 1 -- 172.21.15.16:0/2430418424 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7fd3dc025020 con 0x5571ecaae970 2024-03-28T22:13:07.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.433+0000 7fd3ed828740 10 client.16056 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:13:07.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.433+0000 7fd3ed828740 10 client.16056 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:13:07.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.433+0000 7fd3ed828740 20 client.16056 populate_metadata read hostname 'smithi016' 2024-03-28T22:13:07.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.433+0000 7fd3ed828740 1 --2- 172.21.15.16:0/2430418424 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5571ecb25e90 0x5571ecb282b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:07.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.433+0000 7fd3ed828740 4 client.16056 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:13:07.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.433+0000 7fd3ed828740 1 -- 172.21.15.16:0/2430418424 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5571eca10a40 con 0x5571ecb25e90 2024-03-28T22:13:07.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.433+0000 7fd3e8901640 1 --2- 172.21.15.16:0/2430418424 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5571ecb25e90 0x5571ecb282b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:07.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.433+0000 7fd3e8901640 1 --2- 172.21.15.16:0/2430418424 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5571ecb25e90 0x5571ecb282b0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7230c8aa78cafe29 server_cookie=aec295e616f6c6ef in_seq=0 out_seq=0 2024-03-28T22:13:07.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.433+0000 7fd3c3fff640 10 client.16056 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:13:07.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.433+0000 7fd3c3fff640 1 -- 172.21.15.16:0/2430418424 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5571eca10a40 con 0x5571ecb25e90 2024-03-28T22:13:07.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.433+0000 7fd3c3fff640 10 client.16056 handle_command_reply: tid=0 2024-03-28T22:13:07.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.433+0000 7fd3ed828740 1 client.16056 shutdown 2024-03-28T22:13:07.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.434+0000 7fd3ed828740 1 -- 172.21.15.16:0/2430418424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5571ecaae970 msgr2=0x5571ecab2d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:07.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.434+0000 7fd3ed828740 1 --2- 172.21.15.16:0/2430418424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5571ecaae970 0x5571ecab2d70 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc028dd0 tx=0x7fd3dc011f80 comp rx=0 tx=0).stop 2024-03-28T22:13:07.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.434+0000 7fd3ed828740 1 -- 172.21.15.16:0/2430418424 shutdown_connections 2024-03-28T22:13:07.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.434+0000 7fd3ed828740 1 --2- 172.21.15.16:0/2430418424 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5571ecb25e90 0x5571ecb282b0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.434+0000 7fd3ed828740 1 --2- 172.21.15.16:0/2430418424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5571ecaae970 0x5571ecab2d70 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.434+0000 7fd3ed828740 1 --2- 172.21.15.16:0/2430418424 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571ecaa5d90 0x5571ecb1d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.434+0000 7fd3c3fff640 0 client.16056 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:13:07.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.434+0000 7fd3ed828740 1 --2- 172.21.15.16:0/2430418424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5571ecaa5470 0x5571ecb1ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.434+0000 7fd3ed828740 1 -- 172.21.15.16:0/2430418424 >> 172.21.15.16:0/2430418424 conn(0x5571eca9fba0 msgr2=0x5571ecaad020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:07.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.434+0000 7fd3ed828740 1 -- 172.21.15.16:0/2430418424 shutdown_connections 2024-03-28T22:13:07.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.434+0000 7fd3ed828740 1 -- 172.21.15.16:0/2430418424 wait complete. 2024-03-28T22:13:07.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.434+0000 7fd3ed828740 20 client.16056 trim_cache size 0 max 16384 2024-03-28T22:13:07.447 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:13:07.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.434+0000 7fd3eab65640 1 -- 172.21.15.16:0/3867722022 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fd3d40a1ef0 msgr2=0x7fd3d40a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:07.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.434+0000 7fd3eab65640 1 --2- 172.21.15.16:0/3867722022 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fd3d40a1ef0 0x7fd3d40a43b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc00a780 tx=0x7fd3dc005f60 comp rx=0 tx=0).stop 2024-03-28T22:13:07.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.434+0000 7fd3eab65640 1 -- 172.21.15.16:0/3867722022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3e4102090 msgr2=0x7fd3e4198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:07.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.434+0000 7fd3eab65640 1 --2- 172.21.15.16:0/3867722022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3e4102090 0x7fd3e4198550 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0009230 tx=0x7fd3d002cba0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.434+0000 7fd3eab65640 1 -- 172.21.15.16:0/3867722022 shutdown_connections 2024-03-28T22:13:07.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.434+0000 7fd3eab65640 1 --2- 172.21.15.16:0/3867722022 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fd3d40a1ef0 0x7fd3d40a43b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.434+0000 7fd3eab65640 1 --2- 172.21.15.16:0/3867722022 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3e41076e0 0x7fd3e419c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.434+0000 7fd3eab65640 1 --2- 172.21.15.16:0/3867722022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3e4102090 0x7fd3e4198550 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.434+0000 7fd3eab65640 1 --2- 172.21.15.16:0/3867722022 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3e41016d0 0x7fd3e4198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:07.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.434+0000 7fd3eab65640 1 -- 172.21.15.16:0/3867722022 >> 172.21.15.16:0/3867722022 conn(0x7fd3e40fd1c0 msgr2=0x7fd3e40fd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:07.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.435+0000 7fd3eab65640 1 -- 172.21.15.16:0/3867722022 shutdown_connections 2024-03-28T22:13:07.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:07.435+0000 7fd3eab65640 1 -- 172.21.15.16:0/3867722022 wait complete. 2024-03-28T22:13:08.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:07 smithi053 ceph-mon[35979]: pgmap v79: 97 pgs: 97 active+clean; 94 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 318 KiB/s wr, 44 op/s 2024-03-28T22:13:08.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:07 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3207599492' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:13:08.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:07 smithi063 ceph-mon[36249]: pgmap v79: 97 pgs: 97 active+clean; 94 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 318 KiB/s wr, 44 op/s 2024-03-28T22:13:08.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:07 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/3207599492' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:13:08.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:07 smithi016 ceph-mon[32374]: pgmap v79: 97 pgs: 97 active+clean; 94 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 318 KiB/s wr, 44 op/s 2024-03-28T22:13:08.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:07 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3207599492' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:13:08.451 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:13:08.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:13:08.451 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:13:08.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.569+0000 7fde8deae640 1 -- 172.21.15.16:0/3409895265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde881066a0 msgr2=0x7fde88106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:08.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.569+0000 7fde8deae640 1 --2- 172.21.15.16:0/3409895265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde881066a0 0x7fde88106a80 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fde78009930 tx=0x7fde7802bc30 comp rx=0 tx=0).stop 2024-03-28T22:13:08.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.569+0000 7fde8deae640 1 -- 172.21.15.16:0/3409895265 shutdown_connections 2024-03-28T22:13:08.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.569+0000 7fde8deae640 1 --2- 172.21.15.16:0/3409895265 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde88101010 0x7fde8810d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.569+0000 7fde8deae640 1 --2- 172.21.15.16:0/3409895265 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde88100650 0x7fde88100ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.570+0000 7fde8deae640 1 --2- 172.21.15.16:0/3409895265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde881066a0 0x7fde88106a80 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.570+0000 7fde8deae640 1 -- 172.21.15.16:0/3409895265 >> 172.21.15.16:0/3409895265 conn(0x7fde88077820 msgr2=0x7fde88077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:08.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.570+0000 7fde8deae640 1 -- 172.21.15.16:0/3409895265 shutdown_connections 2024-03-28T22:13:08.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.570+0000 7fde8deae640 1 -- 172.21.15.16:0/3409895265 wait complete. 2024-03-28T22:13:08.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.570+0000 7fde8deae640 1 Processor -- start 2024-03-28T22:13:08.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.570+0000 7fde8deae640 1 -- start start 2024-03-28T22:13:08.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.571+0000 7fde8deae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde88100650 0x7fde8819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:08.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.571+0000 7fde8deae640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde88101010 0x7fde8819a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:08.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.571+0000 7fde8deae640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde881066a0 0x7fde8819ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:08.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.571+0000 7fde8deae640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fde88111b20 con 0x7fde88100650 2024-03-28T22:13:08.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.571+0000 7fde8deae640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fde881119a0 con 0x7fde88101010 2024-03-28T22:13:08.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.571+0000 7fde8deae640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fde88111ca0 con 0x7fde881066a0 2024-03-28T22:13:08.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.571+0000 7fde877fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde88100650 0x7fde8819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:08.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.571+0000 7fde877fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde88100650 0x7fde8819a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55984/0 (socket says 172.21.15.16:55984) 2024-03-28T22:13:08.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.571+0000 7fde877fe640 1 -- 172.21.15.16:0/2011652695 learned_addr learned my addr 172.21.15.16:0/2011652695 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:08.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.571+0000 7fde86ffd640 1 --2- 172.21.15.16:0/2011652695 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde88101010 0x7fde8819a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:08.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.571+0000 7fde87fff640 1 --2- 172.21.15.16:0/2011652695 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde881066a0 0x7fde8819ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:08.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.571+0000 7fde877fe640 1 -- 172.21.15.16:0/2011652695 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde881066a0 msgr2=0x7fde8819ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:08.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.571+0000 7fde877fe640 1 --2- 172.21.15.16:0/2011652695 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde881066a0 0x7fde8819ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.571+0000 7fde877fe640 1 -- 172.21.15.16:0/2011652695 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde88101010 msgr2=0x7fde8819a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:08.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.571+0000 7fde877fe640 1 --2- 172.21.15.16:0/2011652695 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde88101010 0x7fde8819a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.571+0000 7fde877fe640 1 -- 172.21.15.16:0/2011652695 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde8819f1b0 con 0x7fde88100650 2024-03-28T22:13:08.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.571+0000 7fde87fff640 1 --2- 172.21.15.16:0/2011652695 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde881066a0 0x7fde8819ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:13:08.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.571+0000 7fde877fe640 1 --2- 172.21.15.16:0/2011652695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde88100650 0x7fde8819a1c0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fde78009230 tx=0x7fde7802cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:08.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.571+0000 7fde84ff9640 1 -- 172.21.15.16:0/2011652695 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde78041070 con 0x7fde88100650 2024-03-28T22:13:08.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.571+0000 7fde84ff9640 1 -- 172.21.15.16:0/2011652695 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde78004450 con 0x7fde88100650 2024-03-28T22:13:08.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.571+0000 7fde84ff9640 1 -- 172.21.15.16:0/2011652695 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde78034440 con 0x7fde88100650 2024-03-28T22:13:08.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.571+0000 7fde8deae640 1 -- 172.21.15.16:0/2011652695 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde8819f440 con 0x7fde88100650 2024-03-28T22:13:08.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.571+0000 7fde8deae640 1 -- 172.21.15.16:0/2011652695 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde881a6d30 con 0x7fde88100650 2024-03-28T22:13:08.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.574+0000 7fde84ff9640 1 -- 172.21.15.16:0/2011652695 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7fde78044070 con 0x7fde88100650 2024-03-28T22:13:08.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.574+0000 7fde90b6e740 1 Processor -- start 2024-03-28T22:13:08.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.574+0000 7fde90b6e740 1 -- start start 2024-03-28T22:13:08.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.574+0000 7fde90b6e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcaff239f0 0x55fcaff23dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:08.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.574+0000 7fde90b6e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcaff24310 0x55fcaff2c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:08.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.574+0000 7fde90b6e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fcaff2cef0 0x55fcaff2f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:08.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.574+0000 7fde84ff9640 1 --2- 172.21.15.16:0/2011652695 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fde700a1ef0 0x7fde700a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:08.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.574+0000 7fde90b6e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55fcafe240b0 con 0x55fcaff239f0 2024-03-28T22:13:08.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.574+0000 7fde90b6e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55fcafe86ea0 con 0x55fcaff24310 2024-03-28T22:13:08.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.574+0000 7fde90b6e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55fcafe2fc00 con 0x55fcaff2cef0 2024-03-28T22:13:08.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.574+0000 7fde87fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcaff239f0 0x55fcaff23dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:08.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.574+0000 7fde877fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcaff24310 0x55fcaff2c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:08.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.574+0000 7fde87fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcaff239f0 0x55fcaff23dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55986/0 (socket says 172.21.15.16:55986) 2024-03-28T22:13:08.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.574+0000 7fde87fff640 1 -- 172.21.15.16:0/4161729494 learned_addr learned my addr 172.21.15.16:0/4161729494 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:08.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.574+0000 7fde87fff640 1 --2- 172.21.15.16:0/2011652695 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fde700a1ef0 0x7fde700a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:08.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.574+0000 7fde86ffd640 1 --2- 172.21.15.16:0/4161729494 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fcaff2cef0 0x55fcaff2f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:08.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.574+0000 7fde84ff9640 1 -- 172.21.15.16:0/2011652695 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fde780efc10 con 0x7fde88100650 2024-03-28T22:13:08.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.575+0000 7fde877fe640 1 -- 172.21.15.16:0/4161729494 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fcaff2cef0 msgr2=0x55fcaff2f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:08.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.575+0000 7fde877fe640 1 --2- 172.21.15.16:0/4161729494 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fcaff2cef0 0x55fcaff2f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.575+0000 7fde877fe640 1 -- 172.21.15.16:0/4161729494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcaff239f0 msgr2=0x55fcaff23dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:08.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.575+0000 7fde877fe640 1 --2- 172.21.15.16:0/4161729494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcaff239f0 0x55fcaff23dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.575+0000 7fde877fe640 1 -- 172.21.15.16:0/4161729494 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcafc637e0 con 0x55fcaff24310 2024-03-28T22:13:08.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.575+0000 7fde87fff640 1 --2- 172.21.15.16:0/4161729494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcaff239f0 0x55fcaff23dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:13:08.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.575+0000 7fde87fff640 1 --2- 172.21.15.16:0/2011652695 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fde700a1ef0 0x7fde700a43b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fde80012d00 tx=0x7fde80005f20 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:08.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.575+0000 7fde877fe640 1 --2- 172.21.15.16:0/4161729494 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcaff24310 0x55fcaff2c9b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fde780ed490 tx=0x7fde7802bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1938b9c551a3414 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:08.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.575+0000 7fde6dffb640 1 -- 172.21.15.16:0/4161729494 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde7804b050 con 0x55fcaff24310 2024-03-28T22:13:08.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.575+0000 7fde6dffb640 1 -- 172.21.15.16:0/4161729494 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde780bce30 con 0x55fcaff24310 2024-03-28T22:13:08.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.575+0000 7fde6dffb640 1 -- 172.21.15.16:0/4161729494 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde780c4ed0 con 0x55fcaff24310 2024-03-28T22:13:08.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.576+0000 7fde90b6e740 1 -- 172.21.15.16:0/4161729494 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcaff24310 msgr2=0x55fcaff2c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:08.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.576+0000 7fde90b6e740 1 --2- 172.21.15.16:0/4161729494 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcaff24310 0x55fcaff2c9b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fde780ed490 tx=0x7fde7802bc30 comp rx=0 tx=0).stop 2024-03-28T22:13:08.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.576+0000 7fde90b6e740 1 -- 172.21.15.16:0/4161729494 shutdown_connections 2024-03-28T22:13:08.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.576+0000 7fde90b6e740 1 --2- 172.21.15.16:0/4161729494 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fcaff2cef0 0x55fcaff2f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.576+0000 7fde90b6e740 1 --2- 172.21.15.16:0/4161729494 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcaff24310 0x55fcaff2c9b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.576+0000 7fde90b6e740 1 --2- 172.21.15.16:0/4161729494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcaff239f0 0x55fcaff23dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.576+0000 7fde90b6e740 1 -- 172.21.15.16:0/4161729494 >> 172.21.15.16:0/4161729494 conn(0x55fcaff1e120 msgr2=0x55fcaff1e500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:08.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.576+0000 7fde90b6e740 1 -- 172.21.15.16:0/4161729494 shutdown_connections 2024-03-28T22:13:08.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.576+0000 7fde90b6e740 1 -- 172.21.15.16:0/4161729494 wait complete. 2024-03-28T22:13:08.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.576+0000 7fde90b6e740 1 -- start start 2024-03-28T22:13:08.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.576+0000 7fde90b6e740 1 Processor -- start 2024-03-28T22:13:08.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.577+0000 7fde90b6e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcaff239f0 0x55fcaff29eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:08.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.577+0000 7fde90b6e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcaff24310 0x55fcaff2a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:08.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.577+0000 7fde90b6e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fcaff2cef0 0x55fcaff2a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:08.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.577+0000 7fde90b6e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55fcafe253d0 con 0x55fcaff24310 2024-03-28T22:13:08.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.577+0000 7fde90b6e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55fcafe82f40 con 0x55fcaff239f0 2024-03-28T22:13:08.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.577+0000 7fde90b6e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55fcafe86ea0 con 0x55fcaff2cef0 2024-03-28T22:13:08.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.577+0000 7fde90b6e740 10 client.0 _ll_register_callbacks cb 0x55fcafe42b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:13:08.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.577+0000 7fde86ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcaff24310 0x55fcaff2a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:08.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.577+0000 7fde877fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcaff239f0 0x55fcaff29eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:08.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.577+0000 7fde86ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcaff24310 0x55fcaff2a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55990/0 (socket says 172.21.15.16:55990) 2024-03-28T22:13:08.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.577+0000 7fde86ffd640 1 -- 172.21.15.16:0/4156726549 learned_addr learned my addr 172.21.15.16:0/4156726549 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:08.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.577+0000 7fde877fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcaff239f0 0x55fcaff29eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:44908/0 (socket says 172.21.15.16:44908) 2024-03-28T22:13:08.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.577+0000 7fde87fff640 1 --2- 172.21.15.16:0/4156726549 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fcaff2cef0 0x55fcaff2a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:08.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.577+0000 7fde86ffd640 1 -- 172.21.15.16:0/4156726549 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fcaff2cef0 msgr2=0x55fcaff2a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:08.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.577+0000 7fde86ffd640 1 --2- 172.21.15.16:0/4156726549 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fcaff2cef0 0x55fcaff2a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.577+0000 7fde86ffd640 1 -- 172.21.15.16:0/4156726549 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcaff239f0 msgr2=0x55fcaff29eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:08.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.577+0000 7fde86ffd640 1 --2- 172.21.15.16:0/4156726549 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcaff239f0 0x55fcaff29eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.577+0000 7fde86ffd640 1 -- 172.21.15.16:0/4156726549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcafcc3f70 con 0x55fcaff24310 2024-03-28T22:13:08.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.577+0000 7fde877fe640 1 --2- 172.21.15.16:0/4156726549 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcaff239f0 0x55fcaff29eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:13:08.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.577+0000 7fde86ffd640 1 --2- 172.21.15.16:0/4156726549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcaff24310 0x55fcaff2a3f0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fde7c012750 tx=0x7fde7c012c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25f972331f8bc5fe server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:08.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.577+0000 7fde66ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:13:08.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.577+0000 7fde66ffd640 1 -- 172.21.15.16:0/4156726549 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde7c012e60 con 0x55fcaff24310 2024-03-28T22:13:08.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.578+0000 7fde66ffd640 1 -- 172.21.15.16:0/4156726549 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fde7c00f8c0 con 0x55fcaff24310 2024-03-28T22:13:08.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.578+0000 7fde90b6e740 1 -- 172.21.15.16:0/4156726549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde780bce30 con 0x55fcaff24310 2024-03-28T22:13:08.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.578+0000 7fde66ffd640 1 -- 172.21.15.16:0/4156726549 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fde7c00fac0 con 0x55fcaff24310 2024-03-28T22:13:08.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.578+0000 7fde90b6e740 10 client.16068 fetch_fsmap learned FSMap version 24 2024-03-28T22:13:08.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.578+0000 7fde90b6e740 1 -- 172.21.15.16:0/4156726549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55fcafb5b170 con 0x55fcaff24310 2024-03-28T22:13:08.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.578+0000 7fde66ffd640 1 -- 172.21.15.16:0/4156726549 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde780bce30 con 0x55fcaff24310 2024-03-28T22:13:08.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.578+0000 7fde66ffd640 1 -- 172.21.15.16:0/4156726549 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7fde7c023070 con 0x55fcaff24310 2024-03-28T22:13:08.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.578+0000 7fde90b6e740 10 client.16068 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:13:08.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.578+0000 7fde90b6e740 10 client.16068 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:13:08.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.578+0000 7fde90b6e740 20 client.16068 populate_metadata read hostname 'smithi016' 2024-03-28T22:13:08.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.578+0000 7fde90b6e740 1 --2- 172.21.15.16:0/4156726549 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55fcaff3cb30 0x55fcaff3efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:08.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.578+0000 7fde90b6e740 4 client.16068 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:13:08.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.578+0000 7fde90b6e740 1 -- 172.21.15.16:0/4156726549 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fcafe943e0 con 0x55fcaff3cb30 2024-03-28T22:13:08.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.578+0000 7fde877fe640 1 --2- 172.21.15.16:0/4156726549 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55fcaff3cb30 0x55fcaff3efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:08.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.579+0000 7fde66ffd640 10 client.16068 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:13:08.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.579+0000 7fde877fe640 1 --2- 172.21.15.16:0/4156726549 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55fcaff3cb30 0x55fcaff3efd0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ad08b59f799c05c server_cookie=8dfd01f60325d04c in_seq=0 out_seq=0 2024-03-28T22:13:08.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.580+0000 7fde66ffd640 1 -- 172.21.15.16:0/4156726549 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fcafe943e0 con 0x55fcaff3cb30 2024-03-28T22:13:08.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.580+0000 7fde66ffd640 10 client.16068 handle_command_reply: tid=0 2024-03-28T22:13:08.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.580+0000 7fde90b6e740 1 client.16068 shutdown 2024-03-28T22:13:08.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.580+0000 7fde90b6e740 1 -- 172.21.15.16:0/4156726549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcaff24310 msgr2=0x55fcaff2a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:08.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.580+0000 7fde90b6e740 1 --2- 172.21.15.16:0/4156726549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcaff24310 0x55fcaff2a3f0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fde7c012750 tx=0x7fde7c012c10 comp rx=0 tx=0).stop 2024-03-28T22:13:08.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.580+0000 7fde90b6e740 1 -- 172.21.15.16:0/4156726549 shutdown_connections 2024-03-28T22:13:08.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.580+0000 7fde90b6e740 1 --2- 172.21.15.16:0/4156726549 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55fcaff3cb30 0x55fcaff3efd0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.580+0000 7fde90b6e740 1 --2- 172.21.15.16:0/4156726549 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fcaff2cef0 0x55fcaff2a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.580+0000 7fde90b6e740 1 --2- 172.21.15.16:0/4156726549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcaff24310 0x55fcaff2a3f0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.580+0000 7fde66ffd640 0 client.16068 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:13:08.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.580+0000 7fde90b6e740 1 --2- 172.21.15.16:0/4156726549 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcaff239f0 0x55fcaff29eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.580+0000 7fde90b6e740 1 -- 172.21.15.16:0/4156726549 >> 172.21.15.16:0/4156726549 conn(0x55fcaff1e120 msgr2=0x55fcaff1fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:08.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.580+0000 7fde90b6e740 1 -- 172.21.15.16:0/4156726549 shutdown_connections 2024-03-28T22:13:08.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.580+0000 7fde90b6e740 1 -- 172.21.15.16:0/4156726549 wait complete. 2024-03-28T22:13:08.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.580+0000 7fde90b6e740 20 client.16068 trim_cache size 0 max 16384 2024-03-28T22:13:08.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.598+0000 7fde90b6e740 1 Processor -- start 2024-03-28T22:13:08.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.598+0000 7fde90b6e740 1 -- start start 2024-03-28T22:13:08.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.598+0000 7fde90b6e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fcaff239f0 0x55fcaff2c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:08.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.598+0000 7fde90b6e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcaff24310 0x55fcaff28df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:08.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.598+0000 7fde90b6e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcaff2cef0 0x55fcaff38570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:08.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.598+0000 7fde90b6e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fde881b1e60 con 0x55fcaff24310 2024-03-28T22:13:08.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.598+0000 7fde90b6e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fde881af700 con 0x55fcaff2cef0 2024-03-28T22:13:08.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.598+0000 7fde90b6e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fde881ad550 con 0x55fcaff239f0 2024-03-28T22:13:08.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.598+0000 7fde86ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcaff24310 0x55fcaff28df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:08.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.598+0000 7fde87fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcaff2cef0 0x55fcaff38570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:08.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.598+0000 7fde86ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcaff24310 0x55fcaff28df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56004/0 (socket says 172.21.15.16:56004) 2024-03-28T22:13:08.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.598+0000 7fde86ffd640 1 -- 172.21.15.16:0/143833324 learned_addr learned my addr 172.21.15.16:0/143833324 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:08.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.598+0000 7fde877fe640 1 --2- 172.21.15.16:0/143833324 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fcaff239f0 0x55fcaff2c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:08.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.598+0000 7fde86ffd640 1 -- 172.21.15.16:0/143833324 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fcaff239f0 msgr2=0x55fcaff2c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:08.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.598+0000 7fde86ffd640 1 --2- 172.21.15.16:0/143833324 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fcaff239f0 0x55fcaff2c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.598+0000 7fde86ffd640 1 -- 172.21.15.16:0/143833324 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcaff2cef0 msgr2=0x55fcaff38570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:08.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.598+0000 7fde86ffd640 1 --2- 172.21.15.16:0/143833324 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcaff2cef0 0x55fcaff38570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.598+0000 7fde86ffd640 1 -- 172.21.15.16:0/143833324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcaff24140 con 0x55fcaff24310 2024-03-28T22:13:08.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.599+0000 7fde87fff640 1 --2- 172.21.15.16:0/143833324 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcaff2cef0 0x55fcaff38570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:13:08.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.599+0000 7fde86ffd640 1 --2- 172.21.15.16:0/143833324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcaff24310 0x55fcaff28df0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fde7c010f20 tx=0x7fde7c00f8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7baa1a314a94ade7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:08.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.599+0000 7fde6d7fa640 1 -- 172.21.15.16:0/143833324 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde7c029070 con 0x55fcaff24310 2024-03-28T22:13:08.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.599+0000 7fde6d7fa640 1 -- 172.21.15.16:0/143833324 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde7c0055c0 con 0x55fcaff24310 2024-03-28T22:13:08.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.599+0000 7fde6d7fa640 1 -- 172.21.15.16:0/143833324 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde7c012e60 con 0x55fcaff24310 2024-03-28T22:13:08.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.599+0000 7fde90b6e740 1 -- 172.21.15.16:0/143833324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcaff24310 msgr2=0x55fcaff28df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:08.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.599+0000 7fde90b6e740 1 --2- 172.21.15.16:0/143833324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcaff24310 0x55fcaff28df0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fde7c010f20 tx=0x7fde7c00f8c0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.599+0000 7fde90b6e740 1 -- 172.21.15.16:0/143833324 shutdown_connections 2024-03-28T22:13:08.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.599+0000 7fde90b6e740 1 --2- 172.21.15.16:0/143833324 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcaff2cef0 0x55fcaff38570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.599+0000 7fde90b6e740 1 --2- 172.21.15.16:0/143833324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcaff24310 0x55fcaff28df0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.599+0000 7fde90b6e740 1 --2- 172.21.15.16:0/143833324 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fcaff239f0 0x55fcaff2c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.599+0000 7fde90b6e740 1 -- 172.21.15.16:0/143833324 >> 172.21.15.16:0/143833324 conn(0x55fcaff1e120 msgr2=0x55fcaff1d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:08.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.599+0000 7fde90b6e740 1 -- 172.21.15.16:0/143833324 shutdown_connections 2024-03-28T22:13:08.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.599+0000 7fde90b6e740 1 -- 172.21.15.16:0/143833324 wait complete. 2024-03-28T22:13:08.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.600+0000 7fde90b6e740 1 -- start start 2024-03-28T22:13:08.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.600+0000 7fde90b6e740 1 Processor -- start 2024-03-28T22:13:08.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.600+0000 7fde90b6e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fcaff239f0 0x55fcaff30d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:08.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.600+0000 7fde90b6e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcaff24310 0x55fcaff31260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:08.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.600+0000 7fde90b6e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcaff2cef0 0x55fcaff9bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:08.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.600+0000 7fde90b6e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fde881adbc0 con 0x55fcaff24310 2024-03-28T22:13:08.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.600+0000 7fde90b6e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fde881af030 con 0x55fcaff2cef0 2024-03-28T22:13:08.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.600+0000 7fde90b6e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fde881ae290 con 0x55fcaff239f0 2024-03-28T22:13:08.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.600+0000 7fde90b6e740 10 client.0 _ll_register_callbacks cb 0x7fde7c009da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:13:08.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.600+0000 7fde86ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcaff24310 0x55fcaff31260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:08.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.600+0000 7fde86ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcaff24310 0x55fcaff31260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56012/0 (socket says 172.21.15.16:56012) 2024-03-28T22:13:08.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.600+0000 7fde86ffd640 1 -- 172.21.15.16:0/645390600 learned_addr learned my addr 172.21.15.16:0/645390600 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:08.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.600+0000 7fde87fff640 1 --2- 172.21.15.16:0/645390600 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcaff2cef0 0x55fcaff9bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:08.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.600+0000 7fde877fe640 1 --2- 172.21.15.16:0/645390600 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fcaff239f0 0x55fcaff30d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:08.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.601+0000 7fde86ffd640 1 -- 172.21.15.16:0/645390600 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fcaff239f0 msgr2=0x55fcaff30d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:08.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.601+0000 7fde86ffd640 1 --2- 172.21.15.16:0/645390600 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fcaff239f0 0x55fcaff30d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.601+0000 7fde86ffd640 1 -- 172.21.15.16:0/645390600 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcaff2cef0 msgr2=0x55fcaff9bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:08.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.601+0000 7fde86ffd640 1 --2- 172.21.15.16:0/645390600 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcaff2cef0 0x55fcaff9bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.601+0000 7fde86ffd640 1 -- 172.21.15.16:0/645390600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcaff9c550 con 0x55fcaff24310 2024-03-28T22:13:08.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.601+0000 7fde87fff640 1 --2- 172.21.15.16:0/645390600 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcaff2cef0 0x55fcaff9bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:13:08.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.601+0000 7fde86ffd640 1 --2- 172.21.15.16:0/645390600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcaff24310 0x55fcaff31260 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fde7c010a10 tx=0x7fde7c00ff90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea2965265ebdb2f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:08.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.601+0000 7fde66ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:13:08.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.601+0000 7fde66ffd640 1 -- 172.21.15.16:0/645390600 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde7c032070 con 0x55fcaff24310 2024-03-28T22:13:08.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.601+0000 7fde66ffd640 1 -- 172.21.15.16:0/645390600 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fde7c010640 con 0x55fcaff24310 2024-03-28T22:13:08.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.601+0000 7fde90b6e740 1 -- 172.21.15.16:0/645390600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde7c0055c0 con 0x55fcaff24310 2024-03-28T22:13:08.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.601+0000 7fde66ffd640 1 -- 172.21.15.16:0/645390600 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fde7c03b050 con 0x55fcaff24310 2024-03-28T22:13:08.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.601+0000 7fde90b6e740 10 client.16074 fetch_fsmap learned FSMap version 24 2024-03-28T22:13:08.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.601+0000 7fde90b6e740 1 -- 172.21.15.16:0/645390600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55fcaffa3900 con 0x55fcaff24310 2024-03-28T22:13:08.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.601+0000 7fde66ffd640 1 -- 172.21.15.16:0/645390600 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde7c0055c0 con 0x55fcaff24310 2024-03-28T22:13:08.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.602+0000 7fde66ffd640 1 -- 172.21.15.16:0/645390600 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7fde7c035020 con 0x55fcaff24310 2024-03-28T22:13:08.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.602+0000 7fde90b6e740 10 client.16074 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:13:08.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.602+0000 7fde90b6e740 10 client.16074 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:13:08.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.602+0000 7fde90b6e740 20 client.16074 populate_metadata read hostname 'smithi016' 2024-03-28T22:13:08.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.602+0000 7fde90b6e740 1 --2- 172.21.15.16:0/645390600 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55fcaffa4090 0x55fcaffa64b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:08.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.602+0000 7fde90b6e740 4 client.16074 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:13:08.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.602+0000 7fde90b6e740 1 -- 172.21.15.16:0/645390600 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55fcafe8e730 con 0x55fcaffa4090 2024-03-28T22:13:08.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.602+0000 7fde877fe640 1 --2- 172.21.15.16:0/645390600 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55fcaffa4090 0x55fcaffa64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:08.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.602+0000 7fde66ffd640 10 client.16074 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:13:08.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.602+0000 7fde877fe640 1 --2- 172.21.15.16:0/645390600 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55fcaffa4090 0x55fcaffa64b0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29b168eb55f942fe server_cookie=7d6e4f6b3488c07b in_seq=0 out_seq=0 2024-03-28T22:13:08.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.603+0000 7fde66ffd640 1 -- 172.21.15.16:0/645390600 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55fcafe8e730 con 0x55fcaffa4090 2024-03-28T22:13:08.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.603+0000 7fde66ffd640 10 client.16074 handle_command_reply: tid=0 2024-03-28T22:13:08.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.604+0000 7fde90b6e740 1 client.16074 shutdown 2024-03-28T22:13:08.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.604+0000 7fde90b6e740 1 -- 172.21.15.16:0/645390600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcaff24310 msgr2=0x55fcaff31260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:08.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.604+0000 7fde90b6e740 1 --2- 172.21.15.16:0/645390600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcaff24310 0x55fcaff31260 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fde7c010a10 tx=0x7fde7c00ff90 comp rx=0 tx=0).stop 2024-03-28T22:13:08.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.604+0000 7fde90b6e740 1 -- 172.21.15.16:0/645390600 shutdown_connections 2024-03-28T22:13:08.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.604+0000 7fde90b6e740 1 --2- 172.21.15.16:0/645390600 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55fcaffa4090 0x55fcaffa64b0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.604+0000 7fde90b6e740 1 --2- 172.21.15.16:0/645390600 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcaff2cef0 0x55fcaff9bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.604+0000 7fde90b6e740 1 --2- 172.21.15.16:0/645390600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fcaff24310 0x55fcaff31260 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.604+0000 7fde66ffd640 0 client.16074 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:13:08.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.604+0000 7fde90b6e740 1 --2- 172.21.15.16:0/645390600 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fcaff239f0 0x55fcaff30d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.604+0000 7fde90b6e740 1 -- 172.21.15.16:0/645390600 >> 172.21.15.16:0/645390600 conn(0x55fcaff1e120 msgr2=0x55fcaff29e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:08.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.604+0000 7fde90b6e740 1 -- 172.21.15.16:0/645390600 shutdown_connections 2024-03-28T22:13:08.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.604+0000 7fde90b6e740 1 -- 172.21.15.16:0/645390600 wait complete. 2024-03-28T22:13:08.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.604+0000 7fde90b6e740 20 client.16074 trim_cache size 0 max 16384 2024-03-28T22:13:08.616 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:13:08.616 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-03-28T22:13:08.616 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "52b642bd-993b-478c-a923-2cb16b0020ed", 2024-03-28T22:13:08.616 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-03-28T22:13:08.616 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:13:08.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.604+0000 7fde8deae640 1 -- 172.21.15.16:0/2011652695 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fde700a1ef0 msgr2=0x7fde700a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:08.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.605+0000 7fde8deae640 1 --2- 172.21.15.16:0/2011652695 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fde700a1ef0 0x7fde700a43b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fde80012d00 tx=0x7fde80005f20 comp rx=0 tx=0).stop 2024-03-28T22:13:08.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.605+0000 7fde8deae640 1 -- 172.21.15.16:0/2011652695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde88100650 msgr2=0x7fde8819a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:08.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.605+0000 7fde8deae640 1 --2- 172.21.15.16:0/2011652695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde88100650 0x7fde8819a1c0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fde78009230 tx=0x7fde7802cbf0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.605+0000 7fde8deae640 1 -- 172.21.15.16:0/2011652695 shutdown_connections 2024-03-28T22:13:08.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.605+0000 7fde8deae640 1 --2- 172.21.15.16:0/2011652695 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fde700a1ef0 0x7fde700a43b0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.605+0000 7fde8deae640 1 --2- 172.21.15.16:0/2011652695 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde881066a0 0x7fde8819ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.605+0000 7fde8deae640 1 --2- 172.21.15.16:0/2011652695 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde88101010 0x7fde8819a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.605+0000 7fde8deae640 1 --2- 172.21.15.16:0/2011652695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde88100650 0x7fde8819a1c0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.605+0000 7fde8deae640 1 -- 172.21.15.16:0/2011652695 >> 172.21.15.16:0/2011652695 conn(0x7fde88077820 msgr2=0x7fde8810baa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:08.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.605+0000 7fde8deae640 1 -- 172.21.15.16:0/2011652695 shutdown_connections 2024-03-28T22:13:08.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.605+0000 7fde8deae640 1 -- 172.21.15.16:0/2011652695 wait complete. 2024-03-28T22:13:08.620 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:13:08.621 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:13:08.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.727+0000 7f2b9c725640 1 -- 172.21.15.16:0/2654355002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b94100fb0 msgr2=0x7f2b9410d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:08.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.727+0000 7f2b9c725640 1 --2- 172.21.15.16:0/2654355002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b94100fb0 0x7f2b9410d4c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c009cd0 tx=0x7f2b8c02c0f0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.728+0000 7f2b9c725640 1 -- 172.21.15.16:0/2654355002 shutdown_connections 2024-03-28T22:13:08.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.728+0000 7f2b9c725640 1 --2- 172.21.15.16:0/2654355002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b94100fb0 0x7f2b9410d4c0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.728+0000 7f2b9c725640 1 --2- 172.21.15.16:0/2654355002 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b941005f0 0x7f2b94100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.728+0000 7f2b9c725640 1 --2- 172.21.15.16:0/2654355002 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b94106690 0x7f2b94106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.728+0000 7f2b9c725640 1 -- 172.21.15.16:0/2654355002 >> 172.21.15.16:0/2654355002 conn(0x7f2b94077800 msgr2=0x7f2b94077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:08.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.728+0000 7f2b9c725640 1 -- 172.21.15.16:0/2654355002 shutdown_connections 2024-03-28T22:13:08.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.728+0000 7f2b9c725640 1 -- 172.21.15.16:0/2654355002 wait complete. 2024-03-28T22:13:08.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.728+0000 7f2b9c725640 1 Processor -- start 2024-03-28T22:13:08.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.728+0000 7f2b9c725640 1 -- start start 2024-03-28T22:13:08.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.728+0000 7f2b9c725640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b941005f0 0x7f2b94198d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:08.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.728+0000 7f2b9c725640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b94100fb0 0x7f2b94199270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:08.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.729+0000 7f2b9c725640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b94106690 0x7f2b941a3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:08.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.729+0000 7f2b9c725640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2b941109d0 con 0x7f2b94106690 2024-03-28T22:13:08.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.729+0000 7f2b9c725640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f2b94110850 con 0x7f2b941005f0 2024-03-28T22:13:08.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.729+0000 7f2b9c725640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2b94110b50 con 0x7f2b94100fb0 2024-03-28T22:13:08.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.729+0000 7f2b9acc2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b94106690 0x7f2b941a3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:08.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.729+0000 7f2b9acc2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b94106690 0x7f2b941a3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56038/0 (socket says 172.21.15.16:56038) 2024-03-28T22:13:08.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.729+0000 7f2b9acc2640 1 -- 172.21.15.16:0/369642577 learned_addr learned my addr 172.21.15.16:0/369642577 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:08.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.729+0000 7f2b99cc0640 1 --2- 172.21.15.16:0/369642577 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b94100fb0 0x7f2b94199270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:08.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.729+0000 7f2b9a4c1640 1 --2- 172.21.15.16:0/369642577 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b941005f0 0x7f2b94198d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:08.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.729+0000 7f2b9acc2640 1 -- 172.21.15.16:0/369642577 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b94100fb0 msgr2=0x7f2b94199270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:08.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.729+0000 7f2b9acc2640 1 --2- 172.21.15.16:0/369642577 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b94100fb0 0x7f2b94199270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.729+0000 7f2b9acc2640 1 -- 172.21.15.16:0/369642577 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b941005f0 msgr2=0x7f2b94198d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:08.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.729+0000 7f2b9acc2640 1 --2- 172.21.15.16:0/369642577 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b941005f0 0x7f2b94198d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.729+0000 7f2b9acc2640 1 -- 172.21.15.16:0/369642577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b941a42f0 con 0x7f2b94106690 2024-03-28T22:13:08.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.729+0000 7f2b9acc2640 1 --2- 172.21.15.16:0/369642577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b94106690 0x7f2b941a3db0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c0348f0 tx=0x7f2b8c032c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:08.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.729+0000 7f2b8b7fe640 1 -- 172.21.15.16:0/369642577 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c041020 con 0x7f2b94106690 2024-03-28T22:13:08.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.729+0000 7f2b9c725640 1 -- 172.21.15.16:0/369642577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b941a44c0 con 0x7f2b94106690 2024-03-28T22:13:08.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.729+0000 7f2b8b7fe640 1 -- 172.21.15.16:0/369642577 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b8c032e80 con 0x7f2b94106690 2024-03-28T22:13:08.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.729+0000 7f2b8b7fe640 1 -- 172.21.15.16:0/369642577 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c039040 con 0x7f2b94106690 2024-03-28T22:13:08.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.729+0000 7f2b9c725640 1 -- 172.21.15.16:0/369642577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b941a4980 con 0x7f2b94106690 2024-03-28T22:13:08.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.732+0000 7f2b8b7fe640 1 -- 172.21.15.16:0/369642577 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f2b8c044020 con 0x7f2b94106690 2024-03-28T22:13:08.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.732+0000 7f2b9f3e9740 1 Processor -- start 2024-03-28T22:13:08.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.732+0000 7f2b9f3e9740 1 -- start start 2024-03-28T22:13:08.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.732+0000 7f2b8b7fe640 1 --2- 172.21.15.16:0/369642577 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f2b800a1ef0 0x7f2b800a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:08.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.732+0000 7f2b9f3e9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41c017f40 0x55b41c018320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:08.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.732+0000 7f2b9f3e9740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b41c018860 0x55b41c025660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:08.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.732+0000 7f2b9f3e9740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b41c025ba0 0x55b41c027f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:08.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.732+0000 7f2b9f3e9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b41bf7c180 con 0x55b41c017f40 2024-03-28T22:13:08.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.732+0000 7f2b9f3e9740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55b41bf24b70 con 0x55b41c018860 2024-03-28T22:13:08.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.732+0000 7f2b9f3e9740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55b41bf78220 con 0x55b41c025ba0 2024-03-28T22:13:08.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.733+0000 7f2b9a4c1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41c017f40 0x55b41c018320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:08.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.733+0000 7f2b8b7fe640 1 -- 172.21.15.16:0/369642577 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f2b8c0efb90 con 0x7f2b94106690 2024-03-28T22:13:08.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.733+0000 7f2b99cc0640 1 --2- 172.21.15.16:0/369642577 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f2b800a1ef0 0x7f2b800a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:08.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.733+0000 7f2b9a4c1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41c017f40 0x55b41c018320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56042/0 (socket says 172.21.15.16:56042) 2024-03-28T22:13:08.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.733+0000 7f2b9a4c1640 1 -- 172.21.15.16:0/2453736463 learned_addr learned my addr 172.21.15.16:0/2453736463 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:08.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.733+0000 7f2b9acc2640 1 --2- 172.21.15.16:0/2453736463 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b41c025ba0 0x55b41c027f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.733+0000 7f2b99cc0640 1 --2- 172.21.15.16:0/2453736463 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b41c018860 0x55b41c025660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.733+0000 7f2b9a4c1640 1 -- 172.21.15.16:0/2453736463 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b41c025ba0 msgr2=0x55b41c027f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.733+0000 7f2b9a4c1640 1 --2- 172.21.15.16:0/2453736463 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b41c025ba0 0x55b41c027f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.733+0000 7f2b9a4c1640 1 -- 172.21.15.16:0/2453736463 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b41c018860 msgr2=0x55b41c025660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.733+0000 7f2b9a4c1640 1 --2- 172.21.15.16:0/2453736463 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b41c018860 0x55b41c025660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.733+0000 7f2b9a4c1640 1 -- 172.21.15.16:0/2453736463 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b41bf387c0 con 0x55b41c017f40 2024-03-28T22:13:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.733+0000 7f2b99cc0640 1 --2- 172.21.15.16:0/369642577 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f2b800a1ef0 0x7f2b800a43b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2b94069590 tx=0x7f2b84009b60 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.733+0000 7f2b9a4c1640 1 --2- 172.21.15.16:0/2453736463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41c017f40 0x55b41c018320 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f2b90009c20 tx=0x7f2b90011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f0a2f910f4cdb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:08.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.733+0000 7f2b88ff9640 1 -- 172.21.15.16:0/2453736463 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b90004020 con 0x55b41c017f40 2024-03-28T22:13:08.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.733+0000 7f2b88ff9640 1 -- 172.21.15.16:0/2453736463 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b900041c0 con 0x55b41c017f40 2024-03-28T22:13:08.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.733+0000 7f2b88ff9640 1 -- 172.21.15.16:0/2453736463 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b90019670 con 0x55b41c017f40 2024-03-28T22:13:08.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.734+0000 7f2b9f3e9740 1 -- 172.21.15.16:0/2453736463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41c017f40 msgr2=0x55b41c018320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:08.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.734+0000 7f2b9f3e9740 1 --2- 172.21.15.16:0/2453736463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41c017f40 0x55b41c018320 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f2b90009c20 tx=0x7f2b90011810 comp rx=0 tx=0).stop 2024-03-28T22:13:08.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.734+0000 7f2b9f3e9740 1 -- 172.21.15.16:0/2453736463 shutdown_connections 2024-03-28T22:13:08.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.734+0000 7f2b9f3e9740 1 --2- 172.21.15.16:0/2453736463 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b41c025ba0 0x55b41c027f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.734+0000 7f2b9f3e9740 1 --2- 172.21.15.16:0/2453736463 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b41c018860 0x55b41c025660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.734+0000 7f2b9f3e9740 1 --2- 172.21.15.16:0/2453736463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41c017f40 0x55b41c018320 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.734+0000 7f2b9f3e9740 1 -- 172.21.15.16:0/2453736463 >> 172.21.15.16:0/2453736463 conn(0x55b41c0129c0 msgr2=0x55b41c012da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:08.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.734+0000 7f2b9f3e9740 1 -- 172.21.15.16:0/2453736463 shutdown_connections 2024-03-28T22:13:08.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.734+0000 7f2b9f3e9740 1 -- 172.21.15.16:0/2453736463 wait complete. 2024-03-28T22:13:08.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.735+0000 7f2b9f3e9740 1 -- start start 2024-03-28T22:13:08.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.735+0000 7f2b9f3e9740 1 Processor -- start 2024-03-28T22:13:08.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.735+0000 7f2b9f3e9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41c017f40 0x55b41c022b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:08.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.735+0000 7f2b9f3e9740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b41c018860 0x55b41c0230c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:08.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.735+0000 7f2b9f3e9740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b41c025ba0 0x55b41c023600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:08.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.735+0000 7f2b9f3e9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b41c0289e0 con 0x55b41c017f40 2024-03-28T22:13:08.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.735+0000 7f2b9f3e9740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55b41bf1a4a0 con 0x55b41c018860 2024-03-28T22:13:08.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.735+0000 7f2b9f3e9740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55b41c028b60 con 0x55b41c025ba0 2024-03-28T22:13:08.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.735+0000 7f2b9f3e9740 10 client.0 _ll_register_callbacks cb 0x55b41bf1dd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:13:08.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.735+0000 7f2b9a4c1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41c017f40 0x55b41c022b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:08.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.735+0000 7f2b9a4c1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41c017f40 0x55b41c022b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56058/0 (socket says 172.21.15.16:56058) 2024-03-28T22:13:08.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.735+0000 7f2b9a4c1640 1 -- 172.21.15.16:0/800445325 learned_addr learned my addr 172.21.15.16:0/800445325 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:08.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.735+0000 7f2b99cc0640 1 --2- 172.21.15.16:0/800445325 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b41c018860 0x55b41c0230c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:08.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.735+0000 7f2b9acc2640 1 --2- 172.21.15.16:0/800445325 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b41c025ba0 0x55b41c023600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:08.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.736+0000 7f2b9a4c1640 1 -- 172.21.15.16:0/800445325 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b41c025ba0 msgr2=0x55b41c023600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:08.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.736+0000 7f2b9a4c1640 1 --2- 172.21.15.16:0/800445325 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b41c025ba0 0x55b41c023600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.736+0000 7f2b9a4c1640 1 -- 172.21.15.16:0/800445325 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b41c018860 msgr2=0x55b41c0230c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:08.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.736+0000 7f2b9a4c1640 1 --2- 172.21.15.16:0/800445325 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b41c018860 0x55b41c0230c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.736+0000 7f2b9a4c1640 1 -- 172.21.15.16:0/800445325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b41bd5a7d0 con 0x55b41c017f40 2024-03-28T22:13:08.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.736+0000 7f2b9a4c1640 1 --2- 172.21.15.16:0/800445325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41c017f40 0x55b41c022b80 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f2b90009bf0 tx=0x7f2b90009850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23b5b14acc3f06f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:08.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.736+0000 7f2b7dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:13:08.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.736+0000 7f2b7dffb640 1 -- 172.21.15.16:0/800445325 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b90028070 con 0x55b41c017f40 2024-03-28T22:13:08.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.736+0000 7f2b9f3e9740 1 -- 172.21.15.16:0/800445325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b900041c0 con 0x55b41c017f40 2024-03-28T22:13:08.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.736+0000 7f2b7dffb640 1 -- 172.21.15.16:0/800445325 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f2b90003800 con 0x55b41c017f40 2024-03-28T22:13:08.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.736+0000 7f2b7dffb640 1 -- 172.21.15.16:0/800445325 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2b90034050 con 0x55b41c017f40 2024-03-28T22:13:08.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.736+0000 7f2b9f3e9740 10 client.16086 fetch_fsmap learned FSMap version 24 2024-03-28T22:13:08.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.736+0000 7f2b9f3e9740 1 -- 172.21.15.16:0/800445325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b41bc903f0 con 0x55b41c017f40 2024-03-28T22:13:08.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.736+0000 7f2b7dffb640 1 -- 172.21.15.16:0/800445325 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b900041c0 con 0x55b41c017f40 2024-03-28T22:13:08.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.737+0000 7f2b7dffb640 1 -- 172.21.15.16:0/800445325 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7f2b90004a30 con 0x55b41c017f40 2024-03-28T22:13:08.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.737+0000 7f2b9f3e9740 10 client.16086 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:13:08.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.737+0000 7f2b9f3e9740 10 client.16086 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:13:08.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.737+0000 7f2b9f3e9740 20 client.16086 populate_metadata read hostname 'smithi016' 2024-03-28T22:13:08.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.737+0000 7f2b9f3e9740 1 --2- 172.21.15.16:0/800445325 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b41c035ea0 0x55b41c038290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:08.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.737+0000 7f2b9f3e9740 4 client.16086 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:13:08.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.737+0000 7f2b9f3e9740 1 -- 172.21.15.16:0/800445325 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b41bf896c0 con 0x55b41c035ea0 2024-03-28T22:13:08.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.737+0000 7f2b99cc0640 1 --2- 172.21.15.16:0/800445325 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b41c035ea0 0x55b41c038290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:08.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.738+0000 7f2b7dffb640 10 client.16086 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:13:08.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.738+0000 7f2b99cc0640 1 --2- 172.21.15.16:0/800445325 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b41c035ea0 0x55b41c038290 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4529113081ac170f server_cookie=d647f4e0c31f5854 in_seq=0 out_seq=0 2024-03-28T22:13:08.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.739+0000 7f2b7dffb640 1 -- 172.21.15.16:0/800445325 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b41bf896c0 con 0x55b41c035ea0 2024-03-28T22:13:08.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.739+0000 7f2b7dffb640 10 client.16086 handle_command_reply: tid=0 2024-03-28T22:13:08.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.739+0000 7f2b9f3e9740 1 client.16086 shutdown 2024-03-28T22:13:08.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.739+0000 7f2b9f3e9740 1 -- 172.21.15.16:0/800445325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41c017f40 msgr2=0x55b41c022b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:08.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.739+0000 7f2b9f3e9740 1 --2- 172.21.15.16:0/800445325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41c017f40 0x55b41c022b80 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f2b90009bf0 tx=0x7f2b90009850 comp rx=0 tx=0).stop 2024-03-28T22:13:08.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.739+0000 7f2b9f3e9740 1 -- 172.21.15.16:0/800445325 shutdown_connections 2024-03-28T22:13:08.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.739+0000 7f2b9f3e9740 1 --2- 172.21.15.16:0/800445325 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b41c035ea0 0x55b41c038290 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.739+0000 7f2b9f3e9740 1 --2- 172.21.15.16:0/800445325 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b41c025ba0 0x55b41c023600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.739+0000 7f2b7dffb640 0 client.16086 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:13:08.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.739+0000 7f2b9f3e9740 1 --2- 172.21.15.16:0/800445325 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b41c018860 0x55b41c0230c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.739+0000 7f2b9f3e9740 1 --2- 172.21.15.16:0/800445325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41c017f40 0x55b41c022b80 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.739+0000 7f2b9f3e9740 1 -- 172.21.15.16:0/800445325 >> 172.21.15.16:0/800445325 conn(0x55b41c0129c0 msgr2=0x55b41c014710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:08.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.739+0000 7f2b9f3e9740 1 -- 172.21.15.16:0/800445325 shutdown_connections 2024-03-28T22:13:08.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.739+0000 7f2b9f3e9740 1 -- 172.21.15.16:0/800445325 wait complete. 2024-03-28T22:13:08.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.739+0000 7f2b9f3e9740 20 client.16086 trim_cache size 0 max 16384 2024-03-28T22:13:08.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.760+0000 7f2b9f3e9740 1 Processor -- start 2024-03-28T22:13:08.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.760+0000 7f2b9f3e9740 1 -- start start 2024-03-28T22:13:08.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.760+0000 7f2b9f3e9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41c017f40 0x55b41c021ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:08.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.760+0000 7f2b9f3e9740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b41c018860 0x55b41c0317c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:08.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.760+0000 7f2b9f3e9740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b41c025ba0 0x55b41c03db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:08.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.760+0000 7f2b9f3e9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2b941ada90 con 0x55b41c017f40 2024-03-28T22:13:08.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.760+0000 7f2b9f3e9740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f2b941ab330 con 0x55b41c018860 2024-03-28T22:13:08.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.760+0000 7f2b9f3e9740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2b941a9180 con 0x55b41c025ba0 2024-03-28T22:13:08.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.760+0000 7f2b9a4c1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41c017f40 0x55b41c021ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:08.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.760+0000 7f2b9a4c1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41c017f40 0x55b41c021ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56064/0 (socket says 172.21.15.16:56064) 2024-03-28T22:13:08.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.760+0000 7f2b9a4c1640 1 -- 172.21.15.16:0/252669869 learned_addr learned my addr 172.21.15.16:0/252669869 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:08.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.760+0000 7f2b9acc2640 1 --2- 172.21.15.16:0/252669869 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b41c025ba0 0x55b41c03db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:08.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.760+0000 7f2b99cc0640 1 --2- 172.21.15.16:0/252669869 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b41c018860 0x55b41c0317c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:08.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.760+0000 7f2b9acc2640 1 -- 172.21.15.16:0/252669869 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b41c018860 msgr2=0x55b41c0317c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:08.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.760+0000 7f2b9acc2640 1 --2- 172.21.15.16:0/252669869 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b41c018860 0x55b41c0317c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.760+0000 7f2b9acc2640 1 -- 172.21.15.16:0/252669869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41c017f40 msgr2=0x55b41c021ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:08.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.760+0000 7f2b9acc2640 1 --2- 172.21.15.16:0/252669869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41c017f40 0x55b41c021ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.760+0000 7f2b9acc2640 1 -- 172.21.15.16:0/252669869 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b41bc500f0 con 0x55b41c025ba0 2024-03-28T22:13:08.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.760+0000 7f2b9a4c1640 1 --2- 172.21.15.16:0/252669869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41c017f40 0x55b41c021ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:13:08.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.761+0000 7f2b9acc2640 1 --2- 172.21.15.16:0/252669869 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b41c025ba0 0x55b41c03db40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c0ed3e0 tx=0x7f2b8c0b8340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6dd2f2db3141fd33 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:08.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.761+0000 7f2b7ffff640 1 -- 172.21.15.16:0/252669869 <== mon.1 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c0bc2a0 con 0x55b41c025ba0 2024-03-28T22:13:08.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.761+0000 7f2b7ffff640 1 -- 172.21.15.16:0/252669869 <== mon.1 v2:172.21.15.63:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b8c034920 con 0x55b41c025ba0 2024-03-28T22:13:08.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.761+0000 7f2b7ffff640 1 -- 172.21.15.16:0/252669869 <== mon.1 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c0bc0c0 con 0x55b41c025ba0 2024-03-28T22:13:08.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.762+0000 7f2b9f3e9740 1 -- 172.21.15.16:0/252669869 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b41c025ba0 msgr2=0x55b41c03db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:08.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.762+0000 7f2b9f3e9740 1 --2- 172.21.15.16:0/252669869 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b41c025ba0 0x55b41c03db40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c0ed3e0 tx=0x7f2b8c0b8340 comp rx=0 tx=0).stop 2024-03-28T22:13:08.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.762+0000 7f2b9f3e9740 1 -- 172.21.15.16:0/252669869 shutdown_connections 2024-03-28T22:13:08.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.762+0000 7f2b9f3e9740 1 --2- 172.21.15.16:0/252669869 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b41c025ba0 0x55b41c03db40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.762+0000 7f2b9f3e9740 1 --2- 172.21.15.16:0/252669869 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b41c018860 0x55b41c0317c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.762+0000 7f2b9f3e9740 1 --2- 172.21.15.16:0/252669869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41c017f40 0x55b41c021ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.762+0000 7f2b9f3e9740 1 -- 172.21.15.16:0/252669869 >> 172.21.15.16:0/252669869 conn(0x55b41c0129c0 msgr2=0x55b41c011c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:08.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.762+0000 7f2b9f3e9740 1 -- 172.21.15.16:0/252669869 shutdown_connections 2024-03-28T22:13:08.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.762+0000 7f2b9f3e9740 1 -- 172.21.15.16:0/252669869 wait complete. 2024-03-28T22:13:08.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.762+0000 7f2b9f3e9740 1 -- start start 2024-03-28T22:13:08.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.762+0000 7f2b9f3e9740 1 Processor -- start 2024-03-28T22:13:08.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.762+0000 7f2b9f3e9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41c017f40 0x55b41c029610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:08.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.762+0000 7f2b9f3e9740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b41c018860 0x55b41c029b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:08.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.762+0000 7f2b9f3e9740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b41c025ba0 0x55b41c02a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:08.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.762+0000 7f2b9f3e9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2b941a97f0 con 0x55b41c017f40 2024-03-28T22:13:08.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.762+0000 7f2b9f3e9740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f2b941aac60 con 0x55b41c018860 2024-03-28T22:13:08.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.762+0000 7f2b9f3e9740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2b941a9ec0 con 0x55b41c025ba0 2024-03-28T22:13:08.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.762+0000 7f2b9f3e9740 10 client.0 _ll_register_callbacks cb 0x7f2b840114c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:13:08.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.762+0000 7f2b9a4c1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41c017f40 0x55b41c029610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:08.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.762+0000 7f2b9a4c1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41c017f40 0x55b41c029610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56080/0 (socket says 172.21.15.16:56080) 2024-03-28T22:13:08.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.763+0000 7f2b9a4c1640 1 -- 172.21.15.16:0/3050690603 learned_addr learned my addr 172.21.15.16:0/3050690603 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:08.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.763+0000 7f2b99cc0640 1 --2- 172.21.15.16:0/3050690603 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b41c018860 0x55b41c029b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:08.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.763+0000 7f2b9acc2640 1 --2- 172.21.15.16:0/3050690603 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b41c025ba0 0x55b41c02a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:08.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.763+0000 7f2b9a4c1640 1 -- 172.21.15.16:0/3050690603 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b41c025ba0 msgr2=0x55b41c02a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:08.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.763+0000 7f2b9a4c1640 1 --2- 172.21.15.16:0/3050690603 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b41c025ba0 0x55b41c02a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.763+0000 7f2b9a4c1640 1 -- 172.21.15.16:0/3050690603 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b41c018860 msgr2=0x55b41c029b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:08.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.763+0000 7f2b9a4c1640 1 --2- 172.21.15.16:0/3050690603 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b41c018860 0x55b41c029b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.763+0000 7f2b9a4c1640 1 -- 172.21.15.16:0/3050690603 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b41c09b170 con 0x55b41c017f40 2024-03-28T22:13:08.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.763+0000 7f2b9a4c1640 1 --2- 172.21.15.16:0/3050690603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41c017f40 0x55b41c029610 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f2b90004e80 tx=0x7f2b90005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92d4f47aeaabfeb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:08.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.763+0000 7f2b7dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:13:08.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.763+0000 7f2b7dffb640 1 -- 172.21.15.16:0/3050690603 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b9003d060 con 0x55b41c017f40 2024-03-28T22:13:08.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.763+0000 7f2b7dffb640 1 -- 172.21.15.16:0/3050690603 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f2b90003c10 con 0x55b41c017f40 2024-03-28T22:13:08.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.763+0000 7f2b9f3e9740 1 -- 172.21.15.16:0/3050690603 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b8c034920 con 0x55b41c017f40 2024-03-28T22:13:08.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.763+0000 7f2b7dffb640 1 -- 172.21.15.16:0/3050690603 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2b90028040 con 0x55b41c017f40 2024-03-28T22:13:08.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.763+0000 7f2b9f3e9740 10 client.16092 fetch_fsmap learned FSMap version 24 2024-03-28T22:13:08.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.763+0000 7f2b9f3e9740 1 -- 172.21.15.16:0/3050690603 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b41c09c390 con 0x55b41c017f40 2024-03-28T22:13:08.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.764+0000 7f2b7dffb640 1 -- 172.21.15.16:0/3050690603 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b8c034920 con 0x55b41c017f40 2024-03-28T22:13:08.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.764+0000 7f2b7dffb640 1 -- 172.21.15.16:0/3050690603 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7f2b90024870 con 0x55b41c017f40 2024-03-28T22:13:08.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.764+0000 7f2b9f3e9740 10 client.16092 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:13:08.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.764+0000 7f2b9f3e9740 10 client.16092 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:13:08.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.764+0000 7f2b9f3e9740 20 client.16092 populate_metadata read hostname 'smithi016' 2024-03-28T22:13:08.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.764+0000 7f2b9f3e9740 1 --2- 172.21.15.16:0/3050690603 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b41c09cba0 0x55b41c09ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:08.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.764+0000 7f2b9f3e9740 4 client.16092 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:13:08.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.764+0000 7f2b9f3e9740 1 -- 172.21.15.16:0/3050690603 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b41bf83a10 con 0x55b41c09cba0 2024-03-28T22:13:08.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.764+0000 7f2b99cc0640 1 --2- 172.21.15.16:0/3050690603 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b41c09cba0 0x55b41c09ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:08.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.764+0000 7f2b99cc0640 1 --2- 172.21.15.16:0/3050690603 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b41c09cba0 0x55b41c09ef90 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a43388b855de62ce server_cookie=eaca3b9fdafbdf39 in_seq=0 out_seq=0 2024-03-28T22:13:08.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.764+0000 7f2b7dffb640 10 client.16092 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:13:08.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.765+0000 7f2b7dffb640 1 -- 172.21.15.16:0/3050690603 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55b41bf83a10 con 0x55b41c09cba0 2024-03-28T22:13:08.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.765+0000 7f2b7dffb640 10 client.16092 handle_command_reply: tid=0 2024-03-28T22:13:08.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.765+0000 7f2b9f3e9740 1 client.16092 shutdown 2024-03-28T22:13:08.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.765+0000 7f2b9f3e9740 1 -- 172.21.15.16:0/3050690603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41c017f40 msgr2=0x55b41c029610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:08.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.765+0000 7f2b9f3e9740 1 --2- 172.21.15.16:0/3050690603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41c017f40 0x55b41c029610 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f2b90004e80 tx=0x7f2b90005d30 comp rx=0 tx=0).stop 2024-03-28T22:13:08.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.765+0000 7f2b9f3e9740 1 -- 172.21.15.16:0/3050690603 shutdown_connections 2024-03-28T22:13:08.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.765+0000 7f2b9f3e9740 1 --2- 172.21.15.16:0/3050690603 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b41c09cba0 0x55b41c09ef90 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.765+0000 7f2b9f3e9740 1 --2- 172.21.15.16:0/3050690603 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b41c025ba0 0x55b41c02a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.765+0000 7f2b7dffb640 0 client.16092 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:13:08.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.765+0000 7f2b9f3e9740 1 --2- 172.21.15.16:0/3050690603 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b41c018860 0x55b41c029b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.765+0000 7f2b9f3e9740 1 --2- 172.21.15.16:0/3050690603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b41c017f40 0x55b41c029610 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.765+0000 7f2b9f3e9740 1 -- 172.21.15.16:0/3050690603 >> 172.21.15.16:0/3050690603 conn(0x55b41c0129c0 msgr2=0x55b41c022970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:08.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.765+0000 7f2b9f3e9740 1 -- 172.21.15.16:0/3050690603 shutdown_connections 2024-03-28T22:13:08.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.765+0000 7f2b9f3e9740 1 -- 172.21.15.16:0/3050690603 wait complete. 2024-03-28T22:13:08.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.765+0000 7f2b9f3e9740 20 client.16092 trim_cache size 0 max 16384 2024-03-28T22:13:08.778 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:13:08.778 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T22:13:08.778 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-03-28T22:13:08.778 INFO:teuthology.orchestra.run.smithi016.stdout: "52b642bd-993b-478c-a923-2cb16b0020ed": { 2024-03-28T22:13:08.779 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-03-28T22:13:08.779 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "52b642bd-993b-478c-a923-2cb16b0020ed", 2024-03-28T22:13:08.779 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-03-28T22:13:08.779 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:13:08.779 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:13:08.779 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:13:08.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.766+0000 7f2b9c725640 1 -- 172.21.15.16:0/369642577 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f2b800a1ef0 msgr2=0x7f2b800a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:08.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.766+0000 7f2b9c725640 1 --2- 172.21.15.16:0/369642577 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f2b800a1ef0 0x7f2b800a43b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2b94069590 tx=0x7f2b84009b60 comp rx=0 tx=0).stop 2024-03-28T22:13:08.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.766+0000 7f2b9c725640 1 -- 172.21.15.16:0/369642577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b94106690 msgr2=0x7f2b941a3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:08.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.766+0000 7f2b9c725640 1 --2- 172.21.15.16:0/369642577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b94106690 0x7f2b941a3db0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c0348f0 tx=0x7f2b8c032c90 comp rx=0 tx=0).stop 2024-03-28T22:13:08.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.766+0000 7f2b9c725640 1 -- 172.21.15.16:0/369642577 shutdown_connections 2024-03-28T22:13:08.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.766+0000 7f2b9c725640 1 --2- 172.21.15.16:0/369642577 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f2b800a1ef0 0x7f2b800a43b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.766+0000 7f2b9c725640 1 --2- 172.21.15.16:0/369642577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b94106690 0x7f2b941a3db0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.766+0000 7f2b9c725640 1 --2- 172.21.15.16:0/369642577 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b94100fb0 0x7f2b94199270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.766+0000 7f2b9c725640 1 --2- 172.21.15.16:0/369642577 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b941005f0 0x7f2b94198d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:08.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.766+0000 7f2b9c725640 1 -- 172.21.15.16:0/369642577 >> 172.21.15.16:0/369642577 conn(0x7f2b94077800 msgr2=0x7f2b940ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:08.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.766+0000 7f2b9c725640 1 -- 172.21.15.16:0/369642577 shutdown_connections 2024-03-28T22:13:08.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:08.766+0000 7f2b9c725640 1 -- 172.21.15.16:0/369642577 wait complete. 2024-03-28T22:13:08.782 INFO:tasks.cephfs.filesystem:scrub status for tag:52b642bd-993b-478c-a923-2cb16b0020ed - {'path': '/', 'tag': '52b642bd-993b-478c-a923-2cb16b0020ed', 'options': 'recursive,force'} 2024-03-28T22:13:09.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:08 smithi016 ceph-mon[32374]: scrub queued for path: / 2024-03-28T22:13:09.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:08 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:13:09.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:08 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:13:09.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:08 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:13:09.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:08 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:13:09.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:08 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:13:09.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:08 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:13:09.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:08 smithi053 ceph-mon[35979]: scrub queued for path: / 2024-03-28T22:13:09.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:08 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:13:09.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:08 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:13:09.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:08 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:13:09.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:08 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:13:09.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:08 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:13:09.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:08 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:13:09.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:08 smithi063 ceph-mon[36249]: scrub queued for path: / 2024-03-28T22:13:09.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:08 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:13:09.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:08 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:13:09.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:08 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:13:09.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:08 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:13:09.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:08 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:13:09.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:08 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:13:10.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:09 smithi053 ceph-mon[35979]: pgmap v80: 97 pgs: 97 active+clean; 94 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 307 KiB/s wr, 40 op/s 2024-03-28T22:13:10.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:09 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:13:10.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:09 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:13:10.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:09 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:13:10.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:09 smithi063 ceph-mon[36249]: pgmap v80: 97 pgs: 97 active+clean; 94 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 307 KiB/s wr, 40 op/s 2024-03-28T22:13:10.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:09 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:13:10.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:09 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:13:10.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:09 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:13:10.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:09 smithi016 ceph-mon[32374]: pgmap v80: 97 pgs: 97 active+clean; 94 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 307 KiB/s wr, 40 op/s 2024-03-28T22:13:10.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:09 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:13:10.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:09 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:13:10.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:09 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:13:12.233 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:11 smithi063 ceph-mon[36249]: pgmap v81: 97 pgs: 97 active+clean; 94 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 307 KiB/s wr, 40 op/s 2024-03-28T22:13:12.238 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:11 smithi016 ceph-mon[32374]: pgmap v81: 97 pgs: 97 active+clean; 94 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 307 KiB/s wr, 40 op/s 2024-03-28T22:13:12.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:11 smithi053 ceph-mon[35979]: pgmap v81: 97 pgs: 97 active+clean; 94 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 307 KiB/s wr, 40 op/s 2024-03-28T22:13:14.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:13 smithi053 ceph-mon[35979]: pgmap v82: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 313 KiB/s wr, 44 op/s 2024-03-28T22:13:14.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:13 smithi063 ceph-mon[36249]: pgmap v82: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 313 KiB/s wr, 44 op/s 2024-03-28T22:13:14.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:13 smithi016 ceph-mon[32374]: pgmap v82: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 313 KiB/s wr, 44 op/s 2024-03-28T22:13:16.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:15 smithi053 ceph-mon[35979]: pgmap v83: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 290 KiB/s wr, 26 op/s 2024-03-28T22:13:16.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:15 smithi063 ceph-mon[36249]: pgmap v83: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 290 KiB/s wr, 26 op/s 2024-03-28T22:13:16.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:15 smithi016 ceph-mon[32374]: pgmap v83: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 290 KiB/s wr, 26 op/s 2024-03-28T22:13:18.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:17 smithi053 ceph-mon[35979]: pgmap v84: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 291 KiB/s wr, 29 op/s 2024-03-28T22:13:18.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:17 smithi063 ceph-mon[36249]: pgmap v84: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 291 KiB/s wr, 29 op/s 2024-03-28T22:13:18.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:17 smithi016 ceph-mon[32374]: pgmap v84: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 291 KiB/s wr, 29 op/s 2024-03-28T22:13:20.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:19 smithi053 ceph-mon[35979]: pgmap v85: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 7.2 KiB/s wr, 6 op/s 2024-03-28T22:13:20.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:19 smithi053 ceph-mon[35979]: scrub summary: idle 2024-03-28T22:13:20.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:19 smithi063 ceph-mon[36249]: pgmap v85: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 7.2 KiB/s wr, 6 op/s 2024-03-28T22:13:20.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:19 smithi063 ceph-mon[36249]: scrub summary: idle 2024-03-28T22:13:20.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:19 smithi016 ceph-mon[32374]: pgmap v85: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 7.2 KiB/s wr, 6 op/s 2024-03-28T22:13:20.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:19 smithi016 ceph-mon[32374]: scrub summary: idle 2024-03-28T22:13:21.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:20 smithi053 ceph-mon[35979]: scrub summary: idle 2024-03-28T22:13:21.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:20 smithi063 ceph-mon[36249]: scrub summary: idle 2024-03-28T22:13:21.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:20 smithi016 ceph-mon[32374]: scrub summary: idle 2024-03-28T22:13:22.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:21 smithi053 ceph-mon[35979]: pgmap v86: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 7.2 KiB/s wr, 6 op/s 2024-03-28T22:13:22.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:21 smithi063 ceph-mon[36249]: pgmap v86: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 7.2 KiB/s wr, 6 op/s 2024-03-28T22:13:22.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:21 smithi016 ceph-mon[32374]: pgmap v86: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 7.2 KiB/s wr, 6 op/s 2024-03-28T22:13:24.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:23 smithi053 ceph-mon[35979]: pgmap v87: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.4 KiB/s wr, 9 op/s 2024-03-28T22:13:24.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:23 smithi063 ceph-mon[36249]: pgmap v87: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.4 KiB/s wr, 9 op/s 2024-03-28T22:13:24.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:23 smithi016 ceph-mon[32374]: pgmap v87: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.4 KiB/s wr, 9 op/s 2024-03-28T22:13:26.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:25 smithi053 ceph-mon[35979]: pgmap v88: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 2.7 KiB/s wr, 5 op/s 2024-03-28T22:13:26.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:25 smithi063 ceph-mon[36249]: pgmap v88: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 2.7 KiB/s wr, 5 op/s 2024-03-28T22:13:26.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:25 smithi016 ceph-mon[32374]: pgmap v88: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 2.7 KiB/s wr, 5 op/s 2024-03-28T22:13:28.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:28 smithi053 ceph-mon[35979]: pgmap v89: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 3.7 KiB/s wr, 8 op/s 2024-03-28T22:13:28.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:28 smithi063 ceph-mon[36249]: pgmap v89: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 3.7 KiB/s wr, 8 op/s 2024-03-28T22:13:28.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:28 smithi016 ceph-mon[32374]: pgmap v89: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 3.7 KiB/s wr, 8 op/s 2024-03-28T22:13:30.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:30 smithi053 ceph-mon[35979]: pgmap v90: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-03-28T22:13:30.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:30 smithi063 ceph-mon[36249]: pgmap v90: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-03-28T22:13:30.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:30 smithi016 ceph-mon[32374]: pgmap v90: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-03-28T22:13:32.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:32 smithi053 ceph-mon[35979]: pgmap v91: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-03-28T22:13:32.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:32 smithi063 ceph-mon[36249]: pgmap v91: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-03-28T22:13:32.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:32 smithi016 ceph-mon[32374]: pgmap v91: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-03-28T22:13:34.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:34 smithi053 ceph-mon[35979]: pgmap v92: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-03-28T22:13:34.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:34 smithi063 ceph-mon[36249]: pgmap v92: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-03-28T22:13:34.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:34 smithi016 ceph-mon[32374]: pgmap v92: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-03-28T22:13:36.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:36 smithi053 ceph-mon[35979]: pgmap v93: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1023 B/s wr, 5 op/s 2024-03-28T22:13:36.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:36 smithi063 ceph-mon[36249]: pgmap v93: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1023 B/s wr, 5 op/s 2024-03-28T22:13:36.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:36 smithi016 ceph-mon[32374]: pgmap v93: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1023 B/s wr, 5 op/s 2024-03-28T22:13:38.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:38 smithi053 ceph-mon[35979]: pgmap v94: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.9 KiB/s wr, 8 op/s 2024-03-28T22:13:38.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:38 smithi063 ceph-mon[36249]: pgmap v94: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.9 KiB/s wr, 8 op/s 2024-03-28T22:13:38.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:38 smithi016 ceph-mon[32374]: pgmap v94: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.9 KiB/s wr, 8 op/s 2024-03-28T22:13:38.783 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:13:38.784 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:13:38.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.900+0000 7fea1cd10640 1 -- 172.21.15.16:0/3532944937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea181005f0 msgr2=0x7fea18100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:38.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.900+0000 7fea1cd10640 1 --2- 172.21.15.16:0/3532944937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea181005f0 0x7fea18100a70 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fea00009930 tx=0x7fea0002bc30 comp rx=0 tx=0).stop 2024-03-28T22:13:38.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.900+0000 7fea1cd10640 1 -- 172.21.15.16:0/3532944937 shutdown_connections 2024-03-28T22:13:38.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.900+0000 7fea1cd10640 1 --2- 172.21.15.16:0/3532944937 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fea18100fb0 0x7fea1810d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:38.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.900+0000 7fea1cd10640 1 --2- 172.21.15.16:0/3532944937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea181005f0 0x7fea18100a70 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:38.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.900+0000 7fea1cd10640 1 --2- 172.21.15.16:0/3532944937 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea18106690 0x7fea18106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:38.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.900+0000 7fea1cd10640 1 -- 172.21.15.16:0/3532944937 >> 172.21.15.16:0/3532944937 conn(0x7fea18077800 msgr2=0x7fea18077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:38.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.900+0000 7fea1cd10640 1 -- 172.21.15.16:0/3532944937 shutdown_connections 2024-03-28T22:13:38.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.900+0000 7fea1cd10640 1 -- 172.21.15.16:0/3532944937 wait complete. 2024-03-28T22:13:38.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.900+0000 7fea1cd10640 1 Processor -- start 2024-03-28T22:13:38.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.900+0000 7fea1cd10640 1 -- start start 2024-03-28T22:13:38.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.901+0000 7fea1cd10640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea181005f0 0x7fea18196f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:38.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.901+0000 7fea1cd10640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea18100fb0 0x7fea18197470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:38.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.901+0000 7fea1cd10640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fea18106690 0x7fea1819b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:38.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.901+0000 7fea1cd10640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fea18110a10 con 0x7fea181005f0 2024-03-28T22:13:38.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.901+0000 7fea1cd10640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fea18110890 con 0x7fea18100fb0 2024-03-28T22:13:38.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.901+0000 7fea1cd10640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fea18110b90 con 0x7fea18106690 2024-03-28T22:13:38.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.901+0000 7fea1659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea181005f0 0x7fea18196f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:38.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.901+0000 7fea1659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea181005f0 0x7fea18196f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54622/0 (socket says 172.21.15.16:54622) 2024-03-28T22:13:38.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.901+0000 7fea1659c640 1 -- 172.21.15.16:0/2378419372 learned_addr learned my addr 172.21.15.16:0/2378419372 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:38.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.901+0000 7fea0ffff640 1 --2- 172.21.15.16:0/2378419372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea18100fb0 0x7fea18197470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:38.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.901+0000 7fea16d9d640 1 --2- 172.21.15.16:0/2378419372 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fea18106690 0x7fea1819b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:38.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.901+0000 7fea1659c640 1 -- 172.21.15.16:0/2378419372 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fea18106690 msgr2=0x7fea1819b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:38.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.901+0000 7fea1659c640 1 --2- 172.21.15.16:0/2378419372 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fea18106690 0x7fea1819b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:38.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.901+0000 7fea1659c640 1 -- 172.21.15.16:0/2378419372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea18100fb0 msgr2=0x7fea18197470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:38.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.901+0000 7fea1659c640 1 --2- 172.21.15.16:0/2378419372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea18100fb0 0x7fea18197470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:38.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.901+0000 7fea1659c640 1 -- 172.21.15.16:0/2378419372 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea1819bed0 con 0x7fea181005f0 2024-03-28T22:13:38.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.901+0000 7fea1659c640 1 --2- 172.21.15.16:0/2378419372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea181005f0 0x7fea18196f30 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fea0800e9c0 tx=0x7fea0800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:38.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.902+0000 7fea0f7fe640 1 -- 172.21.15.16:0/2378419372 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea0800cd80 con 0x7fea181005f0 2024-03-28T22:13:38.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.902+0000 7fea1cd10640 1 -- 172.21.15.16:0/2378419372 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea1819c1c0 con 0x7fea181005f0 2024-03-28T22:13:38.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.902+0000 7fea0f7fe640 1 -- 172.21.15.16:0/2378419372 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea080035f0 con 0x7fea181005f0 2024-03-28T22:13:38.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.902+0000 7fea0f7fe640 1 -- 172.21.15.16:0/2378419372 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea08010690 con 0x7fea181005f0 2024-03-28T22:13:38.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.902+0000 7fea1cd10640 1 -- 172.21.15.16:0/2378419372 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea181a3a50 con 0x7fea181005f0 2024-03-28T22:13:38.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.904+0000 7fea0f7fe640 1 -- 172.21.15.16:0/2378419372 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7fea080037a0 con 0x7fea181005f0 2024-03-28T22:13:38.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.904+0000 7fea0f7fe640 1 --2- 172.21.15.16:0/2378419372 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fea040a1f70 0x7fea040a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:38.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.904+0000 7fea1f9cd740 1 Processor -- start 2024-03-28T22:13:38.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.905+0000 7fea0ffff640 1 --2- 172.21.15.16:0/2378419372 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fea040a1f70 0x7fea040a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:38.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.905+0000 7fea0f7fe640 1 -- 172.21.15.16:0/2378419372 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fea080ce640 con 0x7fea181005f0 2024-03-28T22:13:38.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.905+0000 7fea1f9cd740 1 -- start start 2024-03-28T22:13:38.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.905+0000 7fea1f9cd740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f9e083af80 0x55f9e083b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:38.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.905+0000 7fea1f9cd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9e083b8a0 0x55f9e0846460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:38.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.905+0000 7fea1f9cd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f9e08469a0 0x55f9e0848d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:38.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.905+0000 7fea1f9cd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f9e079f180 con 0x55f9e083b8a0 2024-03-28T22:13:38.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.905+0000 7fea1f9cd740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f9e0747b70 con 0x55f9e083af80 2024-03-28T22:13:38.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.905+0000 7fea1f9cd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f9e079b220 con 0x55f9e08469a0 2024-03-28T22:13:38.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.905+0000 7fea1659c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f9e083af80 0x55f9e083b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:38.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.905+0000 7fea1659c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f9e083af80 0x55f9e083b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:38194/0 (socket says 172.21.15.16:38194) 2024-03-28T22:13:38.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.905+0000 7fea1659c640 1 -- 172.21.15.16:0/1040639616 learned_addr learned my addr 172.21.15.16:0/1040639616 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:38.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.905+0000 7fea0ffff640 1 --2- 172.21.15.16:0/1040639616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9e083b8a0 0x55f9e0846460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:38.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.905+0000 7fea16d9d640 1 --2- 172.21.15.16:0/1040639616 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f9e08469a0 0x55f9e0848d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:38.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.905+0000 7fea0ffff640 1 --2- 172.21.15.16:0/2378419372 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fea040a1f70 0x7fea040a4430 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fea00005ab0 tx=0x7fea00005fd0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:38.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.905+0000 7fea1659c640 1 -- 172.21.15.16:0/1040639616 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f9e08469a0 msgr2=0x55f9e0848d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:38.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.905+0000 7fea1659c640 1 --2- 172.21.15.16:0/1040639616 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f9e08469a0 0x55f9e0848d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:38.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.905+0000 7fea1659c640 1 -- 172.21.15.16:0/1040639616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9e083b8a0 msgr2=0x55f9e0846460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:38.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.905+0000 7fea1659c640 1 --2- 172.21.15.16:0/1040639616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9e083b8a0 0x55f9e0846460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:38.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.905+0000 7fea1659c640 1 -- 172.21.15.16:0/1040639616 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9e075b7c0 con 0x55f9e083af80 2024-03-28T22:13:38.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.906+0000 7fea1659c640 1 --2- 172.21.15.16:0/1040639616 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f9e083af80 0x55f9e083b360 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fea080a3260 tx=0x7fea0809aa10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=471fcf89d668f8b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:38.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.906+0000 7fea0cff9640 1 -- 172.21.15.16:0/1040639616 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea080a3790 con 0x55f9e083af80 2024-03-28T22:13:38.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.906+0000 7fea0cff9640 1 -- 172.21.15.16:0/1040639616 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea08003b30 con 0x55f9e083af80 2024-03-28T22:13:38.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.906+0000 7fea0cff9640 1 -- 172.21.15.16:0/1040639616 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea0809a6a0 con 0x55f9e083af80 2024-03-28T22:13:38.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.906+0000 7fea1f9cd740 1 -- 172.21.15.16:0/1040639616 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f9e083af80 msgr2=0x55f9e083b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:38.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.906+0000 7fea1f9cd740 1 --2- 172.21.15.16:0/1040639616 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f9e083af80 0x55f9e083b360 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fea080a3260 tx=0x7fea0809aa10 comp rx=0 tx=0).stop 2024-03-28T22:13:38.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.907+0000 7fea1f9cd740 1 -- 172.21.15.16:0/1040639616 shutdown_connections 2024-03-28T22:13:38.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.907+0000 7fea1f9cd740 1 --2- 172.21.15.16:0/1040639616 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f9e08469a0 0x55f9e0848d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:38.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.907+0000 7fea1f9cd740 1 --2- 172.21.15.16:0/1040639616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9e083b8a0 0x55f9e0846460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:38.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.907+0000 7fea1f9cd740 1 --2- 172.21.15.16:0/1040639616 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f9e083af80 0x55f9e083b360 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:38.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.907+0000 7fea1f9cd740 1 -- 172.21.15.16:0/1040639616 >> 172.21.15.16:0/1040639616 conn(0x55f9e08359c0 msgr2=0x55f9e0835da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:38.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.907+0000 7fea1f9cd740 1 -- 172.21.15.16:0/1040639616 shutdown_connections 2024-03-28T22:13:38.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.907+0000 7fea1f9cd740 1 -- 172.21.15.16:0/1040639616 wait complete. 2024-03-28T22:13:38.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.907+0000 7fea1f9cd740 1 -- start start 2024-03-28T22:13:38.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.907+0000 7fea1f9cd740 1 Processor -- start 2024-03-28T22:13:38.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.907+0000 7fea1f9cd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f9e083af80 0x55f9e08439a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:38.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.907+0000 7fea1f9cd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9e083b8a0 0x55f9e0843ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:38.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.907+0000 7fea1f9cd740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f9e08469a0 0x55f9e0844420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:38.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.907+0000 7fea1f9cd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f9e0849790 con 0x55f9e083b8a0 2024-03-28T22:13:38.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.907+0000 7fea1f9cd740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f9e073d4a0 con 0x55f9e08469a0 2024-03-28T22:13:38.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.907+0000 7fea1f9cd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f9e0849910 con 0x55f9e083af80 2024-03-28T22:13:38.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.907+0000 7fea1f9cd740 10 client.0 _ll_register_callbacks cb 0x55f9e0740d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:13:38.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.907+0000 7fea0ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9e083b8a0 0x55f9e0843ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:38.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.908+0000 7fea0ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9e083b8a0 0x55f9e0843ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54632/0 (socket says 172.21.15.16:54632) 2024-03-28T22:13:38.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.908+0000 7fea0ffff640 1 -- 172.21.15.16:0/1628052766 learned_addr learned my addr 172.21.15.16:0/1628052766 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:38.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.908+0000 7fea16d9d640 1 --2- 172.21.15.16:0/1628052766 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f9e08469a0 0x55f9e0844420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:38.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.908+0000 7fea1659c640 1 --2- 172.21.15.16:0/1628052766 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f9e083af80 0x55f9e08439a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:38.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.908+0000 7fea0ffff640 1 -- 172.21.15.16:0/1628052766 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f9e083af80 msgr2=0x55f9e08439a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:38.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.908+0000 7fea0ffff640 1 --2- 172.21.15.16:0/1628052766 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f9e083af80 0x55f9e08439a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:38.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.908+0000 7fea0ffff640 1 -- 172.21.15.16:0/1628052766 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f9e08469a0 msgr2=0x55f9e0844420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:38.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.908+0000 7fea0ffff640 1 --2- 172.21.15.16:0/1628052766 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f9e08469a0 0x55f9e0844420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:38.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.908+0000 7fea0ffff640 1 -- 172.21.15.16:0/1628052766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9e057d7d0 con 0x55f9e083b8a0 2024-03-28T22:13:38.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.908+0000 7fea0ffff640 1 --2- 172.21.15.16:0/1628052766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9e083b8a0 0x55f9e0843ee0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fea00004f50 tx=0x7fea0002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8677cf9eade274ca server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:38.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.908+0000 7fe9fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:13:38.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.908+0000 7fe9fdffb640 1 -- 172.21.15.16:0/1628052766 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea00005650 con 0x55f9e083b8a0 2024-03-28T22:13:38.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.908+0000 7fe9fdffb640 1 -- 172.21.15.16:0/1628052766 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fea000048c0 con 0x55f9e083b8a0 2024-03-28T22:13:38.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.908+0000 7fea1f9cd740 1 -- 172.21.15.16:0/1628052766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea08003b30 con 0x55f9e083b8a0 2024-03-28T22:13:38.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.908+0000 7fe9fdffb640 1 -- 172.21.15.16:0/1628052766 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fea000044d0 con 0x55f9e083b8a0 2024-03-28T22:13:38.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.908+0000 7fea1f9cd740 10 client.16104 fetch_fsmap learned FSMap version 24 2024-03-28T22:13:38.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.908+0000 7fea1f9cd740 1 -- 172.21.15.16:0/1628052766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f9e04b33f0 con 0x55f9e083b8a0 2024-03-28T22:13:38.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.909+0000 7fe9fdffb640 1 -- 172.21.15.16:0/1628052766 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea08003b30 con 0x55f9e083b8a0 2024-03-28T22:13:38.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.909+0000 7fe9fdffb640 1 -- 172.21.15.16:0/1628052766 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7fea0005c020 con 0x55f9e083b8a0 2024-03-28T22:13:38.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.909+0000 7fea1f9cd740 10 client.16104 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:13:38.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.909+0000 7fea1f9cd740 10 client.16104 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:13:38.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.909+0000 7fea1f9cd740 20 client.16104 populate_metadata read hostname 'smithi016' 2024-03-28T22:13:38.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.909+0000 7fea1f9cd740 1 --2- 172.21.15.16:0/1628052766 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f9e0856dc0 0x55f9e08591b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:38.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.909+0000 7fea1f9cd740 4 client.16104 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:13:38.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.909+0000 7fea1f9cd740 1 -- 172.21.15.16:0/1628052766 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f9e07ac6c0 con 0x55f9e0856dc0 2024-03-28T22:13:38.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.909+0000 7fea1659c640 1 --2- 172.21.15.16:0/1628052766 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f9e0856dc0 0x55f9e08591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:38.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.909+0000 7fea1659c640 1 --2- 172.21.15.16:0/1628052766 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f9e0856dc0 0x55f9e08591b0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7548d8cbae00cf85 server_cookie=ca36d82f847e2c24 in_seq=0 out_seq=0 2024-03-28T22:13:38.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.909+0000 7fe9fdffb640 10 client.16104 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:13:38.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.910+0000 7fe9fdffb640 1 -- 172.21.15.16:0/1628052766 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f9e07ac6c0 con 0x55f9e0856dc0 2024-03-28T22:13:38.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.910+0000 7fe9fdffb640 10 client.16104 handle_command_reply: tid=0 2024-03-28T22:13:38.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.910+0000 7fea1f9cd740 1 client.16104 shutdown 2024-03-28T22:13:38.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.910+0000 7fea1f9cd740 1 -- 172.21.15.16:0/1628052766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9e083b8a0 msgr2=0x55f9e0843ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:38.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.910+0000 7fea1f9cd740 1 --2- 172.21.15.16:0/1628052766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9e083b8a0 0x55f9e0843ee0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fea00004f50 tx=0x7fea0002bc30 comp rx=0 tx=0).stop 2024-03-28T22:13:38.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.910+0000 7fea1f9cd740 1 -- 172.21.15.16:0/1628052766 shutdown_connections 2024-03-28T22:13:38.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.910+0000 7fea1f9cd740 1 --2- 172.21.15.16:0/1628052766 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f9e0856dc0 0x55f9e08591b0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:38.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.910+0000 7fea1f9cd740 1 --2- 172.21.15.16:0/1628052766 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f9e08469a0 0x55f9e0844420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:38.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.910+0000 7fea1f9cd740 1 --2- 172.21.15.16:0/1628052766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9e083b8a0 0x55f9e0843ee0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:38.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.910+0000 7fe9fdffb640 0 client.16104 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:13:38.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.910+0000 7fea1f9cd740 1 --2- 172.21.15.16:0/1628052766 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f9e083af80 0x55f9e08439a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:38.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.910+0000 7fea1f9cd740 1 -- 172.21.15.16:0/1628052766 >> 172.21.15.16:0/1628052766 conn(0x55f9e08359c0 msgr2=0x55f9e0837710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:38.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.910+0000 7fea1f9cd740 1 -- 172.21.15.16:0/1628052766 shutdown_connections 2024-03-28T22:13:38.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.910+0000 7fea1f9cd740 1 -- 172.21.15.16:0/1628052766 wait complete. 2024-03-28T22:13:38.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.910+0000 7fea1f9cd740 20 client.16104 trim_cache size 0 max 16384 2024-03-28T22:13:38.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.928+0000 7fea1f9cd740 1 Processor -- start 2024-03-28T22:13:38.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.928+0000 7fea1f9cd740 1 -- start start 2024-03-28T22:13:38.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.928+0000 7fea1f9cd740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f9e083af80 0x55f9e08460d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:38.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.928+0000 7fea1f9cd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f9e083b8a0 0x55f9e0852570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:38.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.928+0000 7fea1f9cd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9e08469a0 0x55f9e085eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:38.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.928+0000 7fea1f9cd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fea181adb20 con 0x55f9e08469a0 2024-03-28T22:13:38.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.928+0000 7fea1f9cd740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fea181ab3c0 con 0x55f9e083af80 2024-03-28T22:13:38.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.928+0000 7fea1f9cd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fea181a9210 con 0x55f9e083b8a0 2024-03-28T22:13:38.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.928+0000 7fea16d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9e08469a0 0x55f9e085eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:38.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.928+0000 7fea16d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9e08469a0 0x55f9e085eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54644/0 (socket says 172.21.15.16:54644) 2024-03-28T22:13:38.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.928+0000 7fea16d9d640 1 -- 172.21.15.16:0/3252810193 learned_addr learned my addr 172.21.15.16:0/3252810193 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:38.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.928+0000 7fea1659c640 1 --2- 172.21.15.16:0/3252810193 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f9e083af80 0x55f9e08460d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:38.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.929+0000 7fea16d9d640 1 -- 172.21.15.16:0/3252810193 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f9e083b8a0 msgr2=0x55f9e0852570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:38.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.929+0000 7fea16d9d640 1 --2- 172.21.15.16:0/3252810193 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f9e083b8a0 0x55f9e0852570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:38.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.929+0000 7fea16d9d640 1 -- 172.21.15.16:0/3252810193 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f9e083af80 msgr2=0x55f9e08460d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:38.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.929+0000 7fea16d9d640 1 --2- 172.21.15.16:0/3252810193 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f9e083af80 0x55f9e08460d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:38.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.929+0000 7fea16d9d640 1 -- 172.21.15.16:0/3252810193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9e083e770 con 0x55f9e08469a0 2024-03-28T22:13:38.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.929+0000 7fea1659c640 1 --2- 172.21.15.16:0/3252810193 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f9e083af80 0x55f9e08460d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:13:38.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.929+0000 7fea16d9d640 1 --2- 172.21.15.16:0/3252810193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9e08469a0 0x55f9e085eac0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fea10007fb0 tx=0x7fea10011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72a664d449b3a0c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:38.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.929+0000 7fe9fffff640 1 -- 172.21.15.16:0/3252810193 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1001e070 con 0x55f9e08469a0 2024-03-28T22:13:38.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.929+0000 7fe9fffff640 1 -- 172.21.15.16:0/3252810193 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea10008040 con 0x55f9e08469a0 2024-03-28T22:13:38.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.929+0000 7fe9fffff640 1 -- 172.21.15.16:0/3252810193 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea10019c90 con 0x55f9e08469a0 2024-03-28T22:13:38.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.929+0000 7fea1f9cd740 1 -- 172.21.15.16:0/3252810193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9e08469a0 msgr2=0x55f9e085eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:38.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.929+0000 7fea1f9cd740 1 --2- 172.21.15.16:0/3252810193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9e08469a0 0x55f9e085eac0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fea10007fb0 tx=0x7fea10011890 comp rx=0 tx=0).stop 2024-03-28T22:13:38.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.929+0000 7fea1f9cd740 1 -- 172.21.15.16:0/3252810193 shutdown_connections 2024-03-28T22:13:38.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.929+0000 7fea1f9cd740 1 --2- 172.21.15.16:0/3252810193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9e08469a0 0x55f9e085eac0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:38.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.929+0000 7fea1f9cd740 1 --2- 172.21.15.16:0/3252810193 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f9e083b8a0 0x55f9e0852570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:38.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.929+0000 7fea1f9cd740 1 --2- 172.21.15.16:0/3252810193 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f9e083af80 0x55f9e08460d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:38.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.929+0000 7fea1f9cd740 1 -- 172.21.15.16:0/3252810193 >> 172.21.15.16:0/3252810193 conn(0x55f9e08359c0 msgr2=0x55f9e0835da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:38.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.929+0000 7fea1f9cd740 1 -- 172.21.15.16:0/3252810193 shutdown_connections 2024-03-28T22:13:38.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.930+0000 7fea1f9cd740 1 -- 172.21.15.16:0/3252810193 wait complete. 2024-03-28T22:13:38.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.930+0000 7fea1f9cd740 1 -- start start 2024-03-28T22:13:38.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.930+0000 7fea1f9cd740 1 Processor -- start 2024-03-28T22:13:38.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.930+0000 7fea1f9cd740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f9e083af80 0x55f9e084a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:38.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.930+0000 7fea1f9cd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9e083b8a0 0x55f9e084abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:38.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.930+0000 7fea1f9cd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f9e08469a0 0x55f9e084b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:38.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.930+0000 7fea1f9cd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fea181a9880 con 0x55f9e083b8a0 2024-03-28T22:13:38.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.930+0000 7fea1f9cd740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fea181aacf0 con 0x55f9e083af80 2024-03-28T22:13:38.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.930+0000 7fea1f9cd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fea181a9f50 con 0x55f9e08469a0 2024-03-28T22:13:38.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.930+0000 7fea1f9cd740 10 client.0 _ll_register_callbacks cb 0x7fea00009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:13:38.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.930+0000 7fea0ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9e083b8a0 0x55f9e084abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:38.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.930+0000 7fea0ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9e083b8a0 0x55f9e084abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54646/0 (socket says 172.21.15.16:54646) 2024-03-28T22:13:38.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.930+0000 7fea0ffff640 1 -- 172.21.15.16:0/880130220 learned_addr learned my addr 172.21.15.16:0/880130220 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:38.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.930+0000 7fea1659c640 1 --2- 172.21.15.16:0/880130220 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f9e083af80 0x55f9e084a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:38.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.930+0000 7fea16d9d640 1 --2- 172.21.15.16:0/880130220 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f9e08469a0 0x55f9e084b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:38.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.930+0000 7fea0ffff640 1 -- 172.21.15.16:0/880130220 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f9e08469a0 msgr2=0x55f9e084b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:38.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.930+0000 7fea0ffff640 1 --2- 172.21.15.16:0/880130220 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f9e08469a0 0x55f9e084b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:38.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.931+0000 7fea0ffff640 1 -- 172.21.15.16:0/880130220 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f9e083af80 msgr2=0x55f9e084a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:38.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.931+0000 7fea0ffff640 1 --2- 172.21.15.16:0/880130220 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f9e083af80 0x55f9e084a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:38.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.931+0000 7fea0ffff640 1 -- 172.21.15.16:0/880130220 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9e04730f0 con 0x55f9e083b8a0 2024-03-28T22:13:38.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.931+0000 7fea16d9d640 1 --2- 172.21.15.16:0/880130220 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f9e08469a0 0x55f9e084b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:13:38.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.931+0000 7fea0ffff640 1 --2- 172.21.15.16:0/880130220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9e083b8a0 0x55f9e084abc0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fea00034540 tx=0x7fea00034d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=245fd1150bb0d7ae server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:38.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.931+0000 7fe9fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:13:38.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.931+0000 7fe9fdffb640 1 -- 172.21.15.16:0/880130220 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea0002cc40 con 0x55f9e083b8a0 2024-03-28T22:13:38.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.931+0000 7fe9fdffb640 1 -- 172.21.15.16:0/880130220 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fea00004ad0 con 0x55f9e083b8a0 2024-03-28T22:13:38.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.931+0000 7fea1f9cd740 1 -- 172.21.15.16:0/880130220 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea10008040 con 0x55f9e083b8a0 2024-03-28T22:13:38.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.931+0000 7fe9fdffb640 1 -- 172.21.15.16:0/880130220 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fea00005d50 con 0x55f9e083b8a0 2024-03-28T22:13:38.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.931+0000 7fea1f9cd740 10 client.16110 fetch_fsmap learned FSMap version 24 2024-03-28T22:13:38.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.931+0000 7fea1f9cd740 1 -- 172.21.15.16:0/880130220 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f9e08bd410 con 0x55f9e083b8a0 2024-03-28T22:13:38.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.931+0000 7fe9fdffb640 1 -- 172.21.15.16:0/880130220 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea10008040 con 0x55f9e083b8a0 2024-03-28T22:13:38.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.931+0000 7fe9fdffb640 1 -- 172.21.15.16:0/880130220 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7fea0006d020 con 0x55f9e083b8a0 2024-03-28T22:13:38.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.931+0000 7fea1f9cd740 10 client.16110 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:13:38.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.931+0000 7fea1f9cd740 10 client.16110 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:13:38.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.931+0000 7fea1f9cd740 20 client.16110 populate_metadata read hostname 'smithi016' 2024-03-28T22:13:38.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.931+0000 7fea1f9cd740 1 --2- 172.21.15.16:0/880130220 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f9e08bdb60 0x55f9e08bff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:38.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.931+0000 7fea1f9cd740 4 client.16110 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:13:38.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.931+0000 7fea1f9cd740 1 -- 172.21.15.16:0/880130220 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f9e07a6a10 con 0x55f9e08bdb60 2024-03-28T22:13:38.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.932+0000 7fea1659c640 1 --2- 172.21.15.16:0/880130220 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f9e08bdb60 0x55f9e08bff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:38.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.932+0000 7fe9fdffb640 10 client.16110 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:13:38.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.932+0000 7fea1659c640 1 --2- 172.21.15.16:0/880130220 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f9e08bdb60 0x55f9e08bff80 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39b672f28b901018 server_cookie=d813581bdbb13a39 in_seq=0 out_seq=0 2024-03-28T22:13:38.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.932+0000 7fe9fdffb640 1 -- 172.21.15.16:0/880130220 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f9e07a6a10 con 0x55f9e08bdb60 2024-03-28T22:13:38.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.932+0000 7fe9fdffb640 10 client.16110 handle_command_reply: tid=0 2024-03-28T22:13:38.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.932+0000 7fea1f9cd740 1 client.16110 shutdown 2024-03-28T22:13:38.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.932+0000 7fea1f9cd740 1 -- 172.21.15.16:0/880130220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9e083b8a0 msgr2=0x55f9e084abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:38.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.932+0000 7fea1f9cd740 1 --2- 172.21.15.16:0/880130220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9e083b8a0 0x55f9e084abc0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fea00034540 tx=0x7fea00034d90 comp rx=0 tx=0).stop 2024-03-28T22:13:38.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.932+0000 7fea1f9cd740 1 -- 172.21.15.16:0/880130220 shutdown_connections 2024-03-28T22:13:38.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.932+0000 7fea1f9cd740 1 --2- 172.21.15.16:0/880130220 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f9e08bdb60 0x55f9e08bff80 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:38.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.932+0000 7fea1f9cd740 1 --2- 172.21.15.16:0/880130220 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f9e08469a0 0x55f9e084b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:38.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.932+0000 7fe9fdffb640 0 client.16110 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:13:38.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.932+0000 7fea1f9cd740 1 --2- 172.21.15.16:0/880130220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9e083b8a0 0x55f9e084abc0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:38.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.932+0000 7fea1f9cd740 1 --2- 172.21.15.16:0/880130220 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f9e083af80 0x55f9e084a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:38.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.932+0000 7fea1f9cd740 1 -- 172.21.15.16:0/880130220 >> 172.21.15.16:0/880130220 conn(0x55f9e08359c0 msgr2=0x55f9e0836e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:38.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.932+0000 7fea1f9cd740 1 -- 172.21.15.16:0/880130220 shutdown_connections 2024-03-28T22:13:38.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.932+0000 7fea1f9cd740 1 -- 172.21.15.16:0/880130220 wait complete. 2024-03-28T22:13:38.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.933+0000 7fea1f9cd740 20 client.16110 trim_cache size 0 max 16384 2024-03-28T22:13:38.949 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:13:38.949 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-03-28T22:13:38.949 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-03-28T22:13:38.949 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:13:38.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.933+0000 7fea1cd10640 1 -- 172.21.15.16:0/2378419372 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fea040a1f70 msgr2=0x7fea040a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:38.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.933+0000 7fea1cd10640 1 --2- 172.21.15.16:0/2378419372 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fea040a1f70 0x7fea040a4430 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fea00005ab0 tx=0x7fea00005fd0 comp rx=0 tx=0).stop 2024-03-28T22:13:38.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.933+0000 7fea1cd10640 1 -- 172.21.15.16:0/2378419372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea181005f0 msgr2=0x7fea18196f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:38.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.933+0000 7fea1cd10640 1 --2- 172.21.15.16:0/2378419372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea181005f0 0x7fea18196f30 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fea0800e9c0 tx=0x7fea0800ee80 comp rx=0 tx=0).stop 2024-03-28T22:13:38.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.933+0000 7fea1cd10640 1 -- 172.21.15.16:0/2378419372 shutdown_connections 2024-03-28T22:13:38.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.933+0000 7fea1cd10640 1 --2- 172.21.15.16:0/2378419372 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fea040a1f70 0x7fea040a4430 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:38.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.933+0000 7fea1cd10640 1 --2- 172.21.15.16:0/2378419372 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fea18106690 0x7fea1819b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:38.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.933+0000 7fea1cd10640 1 --2- 172.21.15.16:0/2378419372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea18100fb0 0x7fea18197470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:38.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.933+0000 7fea1cd10640 1 --2- 172.21.15.16:0/2378419372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea181005f0 0x7fea18196f30 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:38.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.933+0000 7fea1cd10640 1 -- 172.21.15.16:0/2378419372 >> 172.21.15.16:0/2378419372 conn(0x7fea18077800 msgr2=0x7fea180ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:38.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.933+0000 7fea1cd10640 1 -- 172.21.15.16:0/2378419372 shutdown_connections 2024-03-28T22:13:38.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:38.933+0000 7fea1cd10640 1 -- 172.21.15.16:0/2378419372 wait complete. 2024-03-28T22:13:38.952 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:13:38.953 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:13:39.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.054+0000 7fee4361c640 1 -- 172.21.15.16:0/4035537233 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee3c107700 msgr2=0x7fee3c107ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:39.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.054+0000 7fee4361c640 1 --2- 172.21.15.16:0/4035537233 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee3c107700 0x7fee3c107ae0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fee34009930 tx=0x7fee3402bc30 comp rx=0 tx=0).stop 2024-03-28T22:13:39.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.054+0000 7fee4361c640 1 -- 172.21.15.16:0/4035537233 shutdown_connections 2024-03-28T22:13:39.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.054+0000 7fee4361c640 1 --2- 172.21.15.16:0/4035537233 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee3c1020b0 0x7fee3c10e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.054+0000 7fee4361c640 1 --2- 172.21.15.16:0/4035537233 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee3c1016f0 0x7fee3c101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.054+0000 7fee4361c640 1 --2- 172.21.15.16:0/4035537233 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee3c107700 0x7fee3c107ae0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.054+0000 7fee4361c640 1 -- 172.21.15.16:0/4035537233 >> 172.21.15.16:0/4035537233 conn(0x7fee3c0fd200 msgr2=0x7fee3c0ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:39.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.054+0000 7fee4361c640 1 -- 172.21.15.16:0/4035537233 shutdown_connections 2024-03-28T22:13:39.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.054+0000 7fee4361c640 1 -- 172.21.15.16:0/4035537233 wait complete. 2024-03-28T22:13:39.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.054+0000 7fee4361c640 1 Processor -- start 2024-03-28T22:13:39.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.054+0000 7fee4361c640 1 -- start start 2024-03-28T22:13:39.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.055+0000 7fee4361c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee3c1016f0 0x7fee3c198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.055+0000 7fee4361c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee3c1020b0 0x7fee3c198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.055+0000 7fee4361c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee3c107700 0x7fee3c19c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.055+0000 7fee4361c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fee3c111ad0 con 0x7fee3c107700 2024-03-28T22:13:39.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.055+0000 7fee4361c640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fee3c111950 con 0x7fee3c1016f0 2024-03-28T22:13:39.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.055+0000 7fee4361c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fee3c111c50 con 0x7fee3c1020b0 2024-03-28T22:13:39.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.055+0000 7fee413b8640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee3c1016f0 0x7fee3c198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.055+0000 7fee41bb9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee3c107700 0x7fee3c19c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.055+0000 7fee413b8640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee3c1016f0 0x7fee3c198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:38246/0 (socket says 172.21.15.16:38246) 2024-03-28T22:13:39.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.055+0000 7fee413b8640 1 -- 172.21.15.16:0/2379091685 learned_addr learned my addr 172.21.15.16:0/2379091685 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:39.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.055+0000 7fee41bb9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee3c107700 0x7fee3c19c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54670/0 (socket says 172.21.15.16:54670) 2024-03-28T22:13:39.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.055+0000 7fee40bb7640 1 --2- 172.21.15.16:0/2379091685 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee3c1020b0 0x7fee3c198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.055+0000 7fee41bb9640 1 -- 172.21.15.16:0/2379091685 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee3c1020b0 msgr2=0x7fee3c198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:39.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.055+0000 7fee41bb9640 1 --2- 172.21.15.16:0/2379091685 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee3c1020b0 0x7fee3c198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.055+0000 7fee41bb9640 1 -- 172.21.15.16:0/2379091685 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee3c1016f0 msgr2=0x7fee3c198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:39.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.055+0000 7fee41bb9640 1 --2- 172.21.15.16:0/2379091685 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee3c1016f0 0x7fee3c198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.055+0000 7fee41bb9640 1 -- 172.21.15.16:0/2379091685 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee3c19d020 con 0x7fee3c107700 2024-03-28T22:13:39.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.055+0000 7fee41bb9640 1 --2- 172.21.15.16:0/2379091685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee3c107700 0x7fee3c19c8f0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fee2c009fd0 tx=0x7fee2c00ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:39.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.055+0000 7fee3a7fc640 1 -- 172.21.15.16:0/2379091685 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee2c00ce20 con 0x7fee3c107700 2024-03-28T22:13:39.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.055+0000 7fee3a7fc640 1 -- 172.21.15.16:0/2379091685 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee2c005020 con 0x7fee3c107700 2024-03-28T22:13:39.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.055+0000 7fee4361c640 1 -- 172.21.15.16:0/2379091685 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee3c19d310 con 0x7fee3c107700 2024-03-28T22:13:39.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.055+0000 7fee3a7fc640 1 -- 172.21.15.16:0/2379091685 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee2c010720 con 0x7fee3c107700 2024-03-28T22:13:39.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.055+0000 7fee4361c640 1 -- 172.21.15.16:0/2379091685 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee3c1a4ba0 con 0x7fee3c107700 2024-03-28T22:13:39.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.056+0000 7fee4361c640 1 -- 172.21.15.16:0/2379091685 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee3c068a20 con 0x7fee3c107700 2024-03-28T22:13:39.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.058+0000 7fee3a7fc640 1 -- 172.21.15.16:0/2379091685 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7fee2c005540 con 0x7fee3c107700 2024-03-28T22:13:39.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.058+0000 7fee3a7fc640 1 --2- 172.21.15.16:0/2379091685 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fee240a1ef0 0x7fee240a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.058+0000 7fee413b8640 1 --2- 172.21.15.16:0/2379091685 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fee240a1ef0 0x7fee240a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.058+0000 7fee3a7fc640 1 -- 172.21.15.16:0/2379091685 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fee2c0ce830 con 0x7fee3c107700 2024-03-28T22:13:39.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.059+0000 7fee413b8640 1 --2- 172.21.15.16:0/2379091685 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fee240a1ef0 0x7fee240a43b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fee34000c00 tx=0x7fee340058d0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:39.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.062+0000 7fee3a7fc640 1 -- 172.21.15.16:0/2379091685 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fee2c09b940 con 0x7fee3c107700 2024-03-28T22:13:39.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.281+0000 7fee4361c640 1 -- 172.21.15.16:0/2379091685 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fee3c02d170 con 0x7fee3c107700 2024-03-28T22:13:39.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.281+0000 7fee3a7fc640 1 -- 172.21.15.16:0/2379091685 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11855 (secure 0 0 0) 0x7fee2c09b240 con 0x7fee3c107700 2024-03-28T22:13:39.285 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:13:39.285 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/1644581227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1644581227},{"type":"v1","addr":"172.21.15.16:6839","nonce":1644581227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/2593062669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2593062669},{"type":"v1","addr":"172.21.15.16:6841","nonce":2593062669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:08:29.672976+0000","modified":"2024-03-28T22:11:54.920802+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24566,"mds_1":24524,"mds_2":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24466":{"gid":24466,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/247249857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":247249857},{"type":"v1","addr":"172.21.15.63:6835","nonce":247249857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6837/2324231032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":2324231032},{"type":"v1","addr":"172.21.15.63:6837","nonce":2324231032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6839/1177197077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":1177197077},{"type":"v1","addr":"172.21.15.63:6839","nonce":1177197077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"e","rank":2,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.53:6835/350727634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":350727634},{"type":"v1","addr":"172.21.15.53:6835","nonce":350727634}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/349877573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":349877573},{"type":"v1","addr":"172.21.15.53:6837","nonce":349877573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6839/3714153956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":3714153956},{"type":"v1","addr":"172.21.15.53:6839","nonce":3714153956}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24566,"qdb_cluster":[24500,24566,24524]},"id":1}]} 2024-03-28T22:13:39.285 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 24 2024-03-28T22:13:39.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.283+0000 7fee4361c640 1 -- 172.21.15.16:0/2379091685 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fee240a1ef0 msgr2=0x7fee240a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:39.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.283+0000 7fee4361c640 1 --2- 172.21.15.16:0/2379091685 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fee240a1ef0 0x7fee240a43b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fee34000c00 tx=0x7fee340058d0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.283+0000 7fee4361c640 1 -- 172.21.15.16:0/2379091685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee3c107700 msgr2=0x7fee3c19c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:39.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.283+0000 7fee4361c640 1 --2- 172.21.15.16:0/2379091685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee3c107700 0x7fee3c19c8f0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fee2c009fd0 tx=0x7fee2c00ef10 comp rx=0 tx=0).stop 2024-03-28T22:13:39.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.283+0000 7fee4361c640 1 -- 172.21.15.16:0/2379091685 shutdown_connections 2024-03-28T22:13:39.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.283+0000 7fee4361c640 1 --2- 172.21.15.16:0/2379091685 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fee240a1ef0 0x7fee240a43b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.283+0000 7fee4361c640 1 --2- 172.21.15.16:0/2379091685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee3c107700 0x7fee3c19c8f0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.283+0000 7fee4361c640 1 --2- 172.21.15.16:0/2379091685 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee3c1020b0 0x7fee3c198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.283+0000 7fee4361c640 1 --2- 172.21.15.16:0/2379091685 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee3c1016f0 0x7fee3c198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.283+0000 7fee4361c640 1 -- 172.21.15.16:0/2379091685 >> 172.21.15.16:0/2379091685 conn(0x7fee3c0fd200 msgr2=0x7fee3c0fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:39.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.284+0000 7fee4361c640 1 -- 172.21.15.16:0/2379091685 shutdown_connections 2024-03-28T22:13:39.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.284+0000 7fee4361c640 1 -- 172.21.15.16:0/2379091685 wait complete. 2024-03-28T22:13:39.298 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:13:39.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.401+0000 7f1708e0f640 1 -- 172.21.15.16:0/1586980922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1704102970 msgr2=0x7f1704102d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:39.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.401+0000 7f1708e0f640 1 --2- 172.21.15.16:0/1586980922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1704102970 0x7f1704102d50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f16fc009860 tx=0x7f16fc02bc00 comp rx=0 tx=0).stop 2024-03-28T22:13:39.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.401+0000 7f1708e0f640 1 -- 172.21.15.16:0/1586980922 shutdown_connections 2024-03-28T22:13:39.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.401+0000 7f1708e0f640 1 --2- 172.21.15.16:0/1586980922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f170410bd40 0x7f170410e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.401+0000 7f1708e0f640 1 --2- 172.21.15.16:0/1586980922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1704103290 0x7f170410b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.401+0000 7f1708e0f640 1 --2- 172.21.15.16:0/1586980922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1704102970 0x7f1704102d50 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.401+0000 7f1708e0f640 1 -- 172.21.15.16:0/1586980922 >> 172.21.15.16:0/1586980922 conn(0x7f1704077770 msgr2=0x7f1704077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:39.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.401+0000 7f1708e0f640 1 -- 172.21.15.16:0/1586980922 shutdown_connections 2024-03-28T22:13:39.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.401+0000 7f1708e0f640 1 -- 172.21.15.16:0/1586980922 wait complete. 2024-03-28T22:13:39.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.402+0000 7f1708e0f640 1 Processor -- start 2024-03-28T22:13:39.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.402+0000 7f1708e0f640 1 -- start start 2024-03-28T22:13:39.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.402+0000 7f1708e0f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1704102970 0x7f1704198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.402+0000 7f1708e0f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1704103290 0x7f1704198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.402+0000 7f1708e0f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f170410bd40 0x7f170419c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.402+0000 7f1708e0f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1704111ad0 con 0x7f1704102970 2024-03-28T22:13:39.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.402+0000 7f1708e0f640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f1704111950 con 0x7f170410bd40 2024-03-28T22:13:39.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.402+0000 7f1708e0f640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1704111c50 con 0x7f1704103290 2024-03-28T22:13:39.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.402+0000 7f170259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1704102970 0x7f1704198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.402+0000 7f170259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1704102970 0x7f1704198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54692/0 (socket says 172.21.15.16:54692) 2024-03-28T22:13:39.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.402+0000 7f170259c640 1 -- 172.21.15.16:0/1231629875 learned_addr learned my addr 172.21.15.16:0/1231629875 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:39.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.402+0000 7f1702d9d640 1 --2- 172.21.15.16:0/1231629875 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f170410bd40 0x7f170419c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.403+0000 7f1701d9b640 1 --2- 172.21.15.16:0/1231629875 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1704103290 0x7f1704198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.403+0000 7f170259c640 1 -- 172.21.15.16:0/1231629875 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1704103290 msgr2=0x7f1704198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:39.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.403+0000 7f170259c640 1 --2- 172.21.15.16:0/1231629875 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1704103290 0x7f1704198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.403+0000 7f170259c640 1 -- 172.21.15.16:0/1231629875 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f170410bd40 msgr2=0x7f170419c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:39.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.403+0000 7f170259c640 1 --2- 172.21.15.16:0/1231629875 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f170410bd40 0x7f170419c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.403+0000 7f170259c640 1 -- 172.21.15.16:0/1231629875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f170419d000 con 0x7f1704102970 2024-03-28T22:13:39.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.403+0000 7f170259c640 1 --2- 172.21.15.16:0/1231629875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1704102970 0x7f1704198010 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f16fc037040 tx=0x7f16fc0346f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:39.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.403+0000 7f16fb7fe640 1 -- 172.21.15.16:0/1231629875 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16fc041020 con 0x7f1704102970 2024-03-28T22:13:39.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.403+0000 7f16fb7fe640 1 -- 172.21.15.16:0/1231629875 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16fc0044d0 con 0x7f1704102970 2024-03-28T22:13:39.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.403+0000 7f1708e0f640 1 -- 172.21.15.16:0/1231629875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f170419d290 con 0x7f1704102970 2024-03-28T22:13:39.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.403+0000 7f16fb7fe640 1 -- 172.21.15.16:0/1231629875 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16fc039040 con 0x7f1704102970 2024-03-28T22:13:39.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.403+0000 7f1708e0f640 1 -- 172.21.15.16:0/1231629875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17041a4b80 con 0x7f1704102970 2024-03-28T22:13:39.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.405+0000 7f16fb7fe640 1 -- 172.21.15.16:0/1231629875 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f16fc044020 con 0x7f1704102970 2024-03-28T22:13:39.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.406+0000 7f170b9ea740 1 Processor -- start 2024-03-28T22:13:39.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.406+0000 7f16fb7fe640 1 --2- 172.21.15.16:0/1231629875 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f16f00a1ef0 0x7f16f00a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.406+0000 7f16fb7fe640 1 -- 172.21.15.16:0/1231629875 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f16fc0ef820 con 0x7f1704102970 2024-03-28T22:13:39.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.406+0000 7f1702d9d640 1 --2- 172.21.15.16:0/1231629875 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f16f00a1ef0 0x7f16f00a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.406+0000 7f170b9ea740 1 -- start start 2024-03-28T22:13:39.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.406+0000 7f170b9ea740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c8b93e470 0x560c8b93e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.406+0000 7f170b9ea740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b93ed90 0x560c8b947430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.406+0000 7f170b9ea740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c8b947970 0x560c8b949d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.406+0000 7f170b9ea740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560c8b8a21b0 con 0x560c8b93e470 2024-03-28T22:13:39.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.406+0000 7f170b9ea740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x560c8b84ac00 con 0x560c8b947970 2024-03-28T22:13:39.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.406+0000 7f170b9ea740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x560c8b89e250 con 0x560c8b93ed90 2024-03-28T22:13:39.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.406+0000 7f170259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c8b93e470 0x560c8b93e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.406+0000 7f170259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c8b93e470 0x560c8b93e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54696/0 (socket says 172.21.15.16:54696) 2024-03-28T22:13:39.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.406+0000 7f170259c640 1 -- 172.21.15.16:0/1549112193 learned_addr learned my addr 172.21.15.16:0/1549112193 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:39.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.406+0000 7f1701d9b640 1 --2- 172.21.15.16:0/1549112193 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b93ed90 0x560c8b947430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.406+0000 7f1702d9d640 1 --2- 172.21.15.16:0/1231629875 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f16f00a1ef0 0x7f16f00a43b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f16f4009a60 tx=0x7f16f4010040 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:39.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.406+0000 7f1702d9d640 1 --2- 172.21.15.16:0/1549112193 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c8b947970 0x560c8b949d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.407+0000 7f170259c640 1 -- 172.21.15.16:0/1549112193 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b93ed90 msgr2=0x560c8b947430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:39.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.407+0000 7f170259c640 1 --2- 172.21.15.16:0/1549112193 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b93ed90 0x560c8b947430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.407+0000 7f170259c640 1 -- 172.21.15.16:0/1549112193 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c8b947970 msgr2=0x560c8b949d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:39.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.407+0000 7f170259c640 1 --2- 172.21.15.16:0/1549112193 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c8b947970 0x560c8b949d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.407+0000 7f170259c640 1 -- 172.21.15.16:0/1549112193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c8b86a3d0 con 0x560c8b93e470 2024-03-28T22:13:39.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.407+0000 7f170259c640 1 --2- 172.21.15.16:0/1549112193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c8b93e470 0x560c8b93e850 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f16fc005fd0 tx=0x7f16fc003a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a18d91852465d259 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:39.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.407+0000 7f16f8ff9640 1 -- 172.21.15.16:0/1549112193 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16fc0bca40 con 0x560c8b93e470 2024-03-28T22:13:39.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.407+0000 7f16f8ff9640 1 -- 172.21.15.16:0/1549112193 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16fc0bc340 con 0x560c8b93e470 2024-03-28T22:13:39.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.407+0000 7f16f8ff9640 1 -- 172.21.15.16:0/1549112193 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16fc0ed360 con 0x560c8b93e470 2024-03-28T22:13:39.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.408+0000 7f170b9ea740 1 -- 172.21.15.16:0/1549112193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c8b93e470 msgr2=0x560c8b93e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:39.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.408+0000 7f170b9ea740 1 --2- 172.21.15.16:0/1549112193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c8b93e470 0x560c8b93e850 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f16fc005fd0 tx=0x7f16fc003a20 comp rx=0 tx=0).stop 2024-03-28T22:13:39.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.408+0000 7f170b9ea740 1 -- 172.21.15.16:0/1549112193 shutdown_connections 2024-03-28T22:13:39.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.408+0000 7f170b9ea740 1 --2- 172.21.15.16:0/1549112193 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c8b947970 0x560c8b949d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.408+0000 7f170b9ea740 1 --2- 172.21.15.16:0/1549112193 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b93ed90 0x560c8b947430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.408+0000 7f170b9ea740 1 --2- 172.21.15.16:0/1549112193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c8b93e470 0x560c8b93e850 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.408+0000 7f170b9ea740 1 -- 172.21.15.16:0/1549112193 >> 172.21.15.16:0/1549112193 conn(0x560c8b938ba0 msgr2=0x560c8b938f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:39.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.408+0000 7f170b9ea740 1 -- 172.21.15.16:0/1549112193 shutdown_connections 2024-03-28T22:13:39.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.408+0000 7f170b9ea740 1 -- 172.21.15.16:0/1549112193 wait complete. 2024-03-28T22:13:39.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.408+0000 7f170b9ea740 1 -- start start 2024-03-28T22:13:39.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.408+0000 7f170b9ea740 1 Processor -- start 2024-03-28T22:13:39.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.408+0000 7f170b9ea740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c8b93e470 0x560c8b944a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.408+0000 7f170b9ea740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c8b93ed90 0x560c8b944f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.408+0000 7f170b9ea740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b947970 0x560c8b9454c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.408+0000 7f170b9ea740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560c8b94a520 con 0x560c8b93e470 2024-03-28T22:13:39.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.408+0000 7f170b9ea740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x560c8b840470 con 0x560c8b93ed90 2024-03-28T22:13:39.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.408+0000 7f170b9ea740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x560c8b94a6a0 con 0x560c8b947970 2024-03-28T22:13:39.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.408+0000 7f170b9ea740 10 client.0 _ll_register_callbacks cb 0x560c8b85dcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:13:39.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.408+0000 7f170259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c8b93e470 0x560c8b944a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.409+0000 7f170259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c8b93e470 0x560c8b944a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54710/0 (socket says 172.21.15.16:54710) 2024-03-28T22:13:39.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.409+0000 7f170259c640 1 -- 172.21.15.16:0/2950512965 learned_addr learned my addr 172.21.15.16:0/2950512965 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:39.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.409+0000 7f1701d9b640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c8b93ed90 0x560c8b944f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.409+0000 7f1702d9d640 1 --2- 172.21.15.16:0/2950512965 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b947970 0x560c8b9454c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.409+0000 7f170259c640 1 -- 172.21.15.16:0/2950512965 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b947970 msgr2=0x560c8b9454c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:39.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.409+0000 7f170259c640 1 --2- 172.21.15.16:0/2950512965 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b947970 0x560c8b9454c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.409+0000 7f170259c640 1 -- 172.21.15.16:0/2950512965 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c8b93ed90 msgr2=0x560c8b944f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:39.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.409+0000 7f170259c640 1 --2- 172.21.15.16:0/2950512965 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c8b93ed90 0x560c8b944f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.409+0000 7f170259c640 1 -- 172.21.15.16:0/2950512965 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c8b5b63f0 con 0x560c8b93e470 2024-03-28T22:13:39.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.409+0000 7f170259c640 1 --2- 172.21.15.16:0/2950512965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c8b93e470 0x560c8b944a40 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f16fc0c45b0 tx=0x7f16fc02bc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7cd68929de6f1c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:39.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.409+0000 7f16e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:13:39.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.409+0000 7f16e9ffb640 1 -- 172.21.15.16:0/2950512965 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16fc0d8030 con 0x560c8b93e470 2024-03-28T22:13:39.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.409+0000 7f16e9ffb640 1 -- 172.21.15.16:0/2950512965 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f16fc0ba640 con 0x560c8b93e470 2024-03-28T22:13:39.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.409+0000 7f170b9ea740 1 -- 172.21.15.16:0/2950512965 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16fc0bc340 con 0x560c8b93e470 2024-03-28T22:13:39.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.409+0000 7f16e9ffb640 1 -- 172.21.15.16:0/2950512965 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f16fc0c0490 con 0x560c8b93e470 2024-03-28T22:13:39.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.409+0000 7f170b9ea740 10 client.16128 fetch_fsmap learned FSMap version 24 2024-03-28T22:13:39.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.409+0000 7f170b9ea740 1 -- 172.21.15.16:0/2950512965 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x560c8b6807d0 con 0x560c8b93e470 2024-03-28T22:13:39.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.409+0000 7f16e9ffb640 1 -- 172.21.15.16:0/2950512965 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16fc0bc340 con 0x560c8b93e470 2024-03-28T22:13:39.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.410+0000 7f16e9ffb640 1 -- 172.21.15.16:0/2950512965 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7f16fc03c030 con 0x560c8b93e470 2024-03-28T22:13:39.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.410+0000 7f170b9ea740 10 client.16128 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:13:39.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.410+0000 7f170b9ea740 10 client.16128 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-28T22:13:39.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.410+0000 7f170b9ea740 20 client.16128 populate_metadata read hostname 'smithi016' 2024-03-28T22:13:39.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.410+0000 7f170b9ea740 1 --2- 172.21.15.16:0/2950512965 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x560c8b9575b0 0x560c8b9599a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.410+0000 7f170b9ea740 4 client.16128 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:13:39.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.410+0000 7f170b9ea740 1 -- 172.21.15.16:0/2950512965 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560c8b8af6f0 con 0x560c8b9575b0 2024-03-28T22:13:39.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.410+0000 7f1701d9b640 1 --2- 172.21.15.16:0/2950512965 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x560c8b9575b0 0x560c8b9599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.410+0000 7f1701d9b640 1 --2- 172.21.15.16:0/2950512965 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x560c8b9575b0 0x560c8b9599a0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6b30d04d76b59519 server_cookie=adffa8fb1f620939 in_seq=0 out_seq=0 2024-03-28T22:13:39.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.410+0000 7f16e9ffb640 10 client.16128 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:13:39.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.411+0000 7f16e9ffb640 1 -- 172.21.15.16:0/2950512965 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560c8b8af6f0 con 0x560c8b9575b0 2024-03-28T22:13:39.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.411+0000 7f16e9ffb640 10 client.16128 handle_command_reply: tid=0 2024-03-28T22:13:39.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.411+0000 7f170b9ea740 1 client.16128 shutdown 2024-03-28T22:13:39.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.412+0000 7f170b9ea740 1 -- 172.21.15.16:0/2950512965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c8b93e470 msgr2=0x560c8b944a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:39.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.412+0000 7f170b9ea740 1 --2- 172.21.15.16:0/2950512965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c8b93e470 0x560c8b944a40 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f16fc0c45b0 tx=0x7f16fc02bc00 comp rx=0 tx=0).stop 2024-03-28T22:13:39.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.412+0000 7f170b9ea740 1 -- 172.21.15.16:0/2950512965 shutdown_connections 2024-03-28T22:13:39.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.412+0000 7f170b9ea740 1 --2- 172.21.15.16:0/2950512965 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x560c8b9575b0 0x560c8b9599a0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.412+0000 7f170b9ea740 1 --2- 172.21.15.16:0/2950512965 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b947970 0x560c8b9454c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.412+0000 7f170b9ea740 1 --2- 172.21.15.16:0/2950512965 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c8b93ed90 0x560c8b944f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.412+0000 7f170b9ea740 1 --2- 172.21.15.16:0/2950512965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c8b93e470 0x560c8b944a40 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.412+0000 7f16e9ffb640 0 client.16128 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:13:39.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.412+0000 7f170b9ea740 1 -- 172.21.15.16:0/2950512965 >> 172.21.15.16:0/2950512965 conn(0x560c8b938ba0 msgr2=0x560c8b93a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:39.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.412+0000 7f170b9ea740 1 -- 172.21.15.16:0/2950512965 shutdown_connections 2024-03-28T22:13:39.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.412+0000 7f170b9ea740 1 -- 172.21.15.16:0/2950512965 wait complete. 2024-03-28T22:13:39.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.412+0000 7f170b9ea740 20 client.16128 trim_cache size 0 max 16384 2024-03-28T22:13:39.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.425+0000 7f170b9ea740 1 Processor -- start 2024-03-28T22:13:39.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.425+0000 7f170b9ea740 1 -- start start 2024-03-28T22:13:39.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.425+0000 7f170b9ea740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c8b93e470 0x560c8b953100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.425+0000 7f170b9ea740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b93ed90 0x560c8b937bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.425+0000 7f170b9ea740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c8b947970 0x560c8b95f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.426+0000 7f170b9ea740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f17041afcd0 con 0x560c8b947970 2024-03-28T22:13:39.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.426+0000 7f170b9ea740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f17041ad570 con 0x560c8b93e470 2024-03-28T22:13:39.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.426+0000 7f170b9ea740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f17041ab3c0 con 0x560c8b93ed90 2024-03-28T22:13:39.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.426+0000 7f1702d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c8b947970 0x560c8b95f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.426+0000 7f1702d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c8b947970 0x560c8b95f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54722/0 (socket says 172.21.15.16:54722) 2024-03-28T22:13:39.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.426+0000 7f1702d9d640 1 -- 172.21.15.16:0/55439156 learned_addr learned my addr 172.21.15.16:0/55439156 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:39.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.426+0000 7f170259c640 1 --2- 172.21.15.16:0/55439156 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c8b93e470 0x560c8b953100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.426+0000 7f1702d9d640 1 -- 172.21.15.16:0/55439156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b93ed90 msgr2=0x560c8b937bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:39.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.426+0000 7f1702d9d640 1 --2- 172.21.15.16:0/55439156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b93ed90 0x560c8b937bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.426+0000 7f1702d9d640 1 -- 172.21.15.16:0/55439156 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c8b93e470 msgr2=0x560c8b953100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:39.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.426+0000 7f1702d9d640 1 --2- 172.21.15.16:0/55439156 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c8b93e470 0x560c8b953100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.426+0000 7f1702d9d640 1 -- 172.21.15.16:0/55439156 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c8b949d60 con 0x560c8b947970 2024-03-28T22:13:39.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.426+0000 7f1702d9d640 1 --2- 172.21.15.16:0/55439156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c8b947970 0x560c8b95f000 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f16f4005fd0 tx=0x7f16f4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b4095eb2d6398a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:39.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.426+0000 7f16ebfff640 1 -- 172.21.15.16:0/55439156 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f401b4f0 con 0x560c8b947970 2024-03-28T22:13:39.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.426+0000 7f16ebfff640 1 -- 172.21.15.16:0/55439156 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16f401bb70 con 0x560c8b947970 2024-03-28T22:13:39.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.426+0000 7f16ebfff640 1 -- 172.21.15.16:0/55439156 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f4027ba0 con 0x560c8b947970 2024-03-28T22:13:39.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.427+0000 7f170b9ea740 1 -- 172.21.15.16:0/55439156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c8b947970 msgr2=0x560c8b95f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:39.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.427+0000 7f170b9ea740 1 --2- 172.21.15.16:0/55439156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c8b947970 0x560c8b95f000 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f16f4005fd0 tx=0x7f16f4014000 comp rx=0 tx=0).stop 2024-03-28T22:13:39.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.427+0000 7f170b9ea740 1 -- 172.21.15.16:0/55439156 shutdown_connections 2024-03-28T22:13:39.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.427+0000 7f170b9ea740 1 --2- 172.21.15.16:0/55439156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c8b947970 0x560c8b95f000 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.427+0000 7f170b9ea740 1 --2- 172.21.15.16:0/55439156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b93ed90 0x560c8b937bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.427+0000 7f170b9ea740 1 --2- 172.21.15.16:0/55439156 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c8b93e470 0x560c8b953100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.427+0000 7f170b9ea740 1 -- 172.21.15.16:0/55439156 >> 172.21.15.16:0/55439156 conn(0x560c8b938ba0 msgr2=0x560c8b9374b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:39.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.427+0000 7f170b9ea740 1 -- 172.21.15.16:0/55439156 shutdown_connections 2024-03-28T22:13:39.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.427+0000 7f170b9ea740 1 -- 172.21.15.16:0/55439156 wait complete. 2024-03-28T22:13:39.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.427+0000 7f170b9ea740 1 -- start start 2024-03-28T22:13:39.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.427+0000 7f170b9ea740 1 Processor -- start 2024-03-28T22:13:39.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.427+0000 7f170b9ea740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c8b93e470 0x560c8b94b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.427+0000 7f170b9ea740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b93ed90 0x560c8b94b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.427+0000 7f170b9ea740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c8b947970 0x560c8b9bc630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.427+0000 7f170b9ea740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f17041aba30 con 0x560c8b947970 2024-03-28T22:13:39.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.427+0000 7f170b9ea740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f17041acea0 con 0x560c8b93e470 2024-03-28T22:13:39.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.427+0000 7f170b9ea740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f17041ac100 con 0x560c8b93ed90 2024-03-28T22:13:39.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.427+0000 7f170b9ea740 10 client.0 _ll_register_callbacks cb 0x7f16fc0bc110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:13:39.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.428+0000 7f1702d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c8b947970 0x560c8b9bc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.428+0000 7f1702d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c8b947970 0x560c8b9bc630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54728/0 (socket says 172.21.15.16:54728) 2024-03-28T22:13:39.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.428+0000 7f1702d9d640 1 -- 172.21.15.16:0/1617326198 learned_addr learned my addr 172.21.15.16:0/1617326198 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:39.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.428+0000 7f170259c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c8b93e470 0x560c8b94b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.428+0000 7f1701d9b640 1 --2- 172.21.15.16:0/1617326198 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b93ed90 0x560c8b94b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.428+0000 7f1702d9d640 1 -- 172.21.15.16:0/1617326198 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b93ed90 msgr2=0x560c8b94b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:39.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.428+0000 7f1702d9d640 1 --2- 172.21.15.16:0/1617326198 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b93ed90 0x560c8b94b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.428+0000 7f1702d9d640 1 -- 172.21.15.16:0/1617326198 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c8b93e470 msgr2=0x560c8b94b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:39.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.428+0000 7f1702d9d640 1 --2- 172.21.15.16:0/1617326198 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c8b93e470 0x560c8b94b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.428+0000 7f1702d9d640 1 -- 172.21.15.16:0/1617326198 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c8b9bcb70 con 0x560c8b947970 2024-03-28T22:13:39.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.428+0000 7f1701d9b640 1 --2- 172.21.15.16:0/1617326198 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b93ed90 0x560c8b94b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:13:39.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.428+0000 7f170259c640 1 --2- 172.21.15.16:0/1617326198 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c8b93e470 0x560c8b94b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:13:39.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.428+0000 7f1702d9d640 1 --2- 172.21.15.16:0/1617326198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c8b947970 0x560c8b9bc630 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f16f400efd0 tx=0x7f16f4003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8a4d2d7b78c4507 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:39.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.428+0000 7f16e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:13:39.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.428+0000 7f16e9ffb640 1 -- 172.21.15.16:0/1617326198 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f4035020 con 0x560c8b947970 2024-03-28T22:13:39.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.428+0000 7f16e9ffb640 1 -- 172.21.15.16:0/1617326198 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f16f401b900 con 0x560c8b947970 2024-03-28T22:13:39.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.428+0000 7f170b9ea740 1 -- 172.21.15.16:0/1617326198 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16f401bb70 con 0x560c8b947970 2024-03-28T22:13:39.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.428+0000 7f16e9ffb640 1 -- 172.21.15.16:0/1617326198 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f16f4040050 con 0x560c8b947970 2024-03-28T22:13:39.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.428+0000 7f170b9ea740 10 client.16134 fetch_fsmap learned FSMap version 24 2024-03-28T22:13:39.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.428+0000 7f170b9ea740 1 -- 172.21.15.16:0/1617326198 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x560c8b9bdcb0 con 0x560c8b947970 2024-03-28T22:13:39.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.428+0000 7f16e9ffb640 1 -- 172.21.15.16:0/1617326198 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16f401bb70 con 0x560c8b947970 2024-03-28T22:13:39.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.429+0000 7f16e9ffb640 1 -- 172.21.15.16:0/1617326198 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7f16f400b040 con 0x560c8b947970 2024-03-28T22:13:39.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.429+0000 7f170b9ea740 10 client.16134 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:13:39.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.429+0000 7f170b9ea740 10 client.16134 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-28T22:13:39.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.429+0000 7f170b9ea740 20 client.16134 populate_metadata read hostname 'smithi016' 2024-03-28T22:13:39.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.429+0000 7f170b9ea740 1 --2- 172.21.15.16:0/1617326198 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x560c8b9be4b0 0x560c8b9c08d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.429+0000 7f170b9ea740 4 client.16134 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:13:39.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.429+0000 7f170b9ea740 1 -- 172.21.15.16:0/1617326198 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560c8b8a9a40 con 0x560c8b9be4b0 2024-03-28T22:13:39.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.429+0000 7f170259c640 1 --2- 172.21.15.16:0/1617326198 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x560c8b9be4b0 0x560c8b9c08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.429+0000 7f16e9ffb640 10 client.16134 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:13:39.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.429+0000 7f170259c640 1 --2- 172.21.15.16:0/1617326198 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x560c8b9be4b0 0x560c8b9c08d0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c605e7e9c364e19b server_cookie=3504015fdfdd3f6e in_seq=0 out_seq=0 2024-03-28T22:13:39.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.429+0000 7f16e9ffb640 1 -- 172.21.15.16:0/1617326198 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560c8b8a9a40 con 0x560c8b9be4b0 2024-03-28T22:13:39.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.429+0000 7f16e9ffb640 10 client.16134 handle_command_reply: tid=0 2024-03-28T22:13:39.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.429+0000 7f170b9ea740 1 client.16134 shutdown 2024-03-28T22:13:39.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.429+0000 7f170b9ea740 1 -- 172.21.15.16:0/1617326198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c8b947970 msgr2=0x560c8b9bc630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:39.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.429+0000 7f170b9ea740 1 --2- 172.21.15.16:0/1617326198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c8b947970 0x560c8b9bc630 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f16f400efd0 tx=0x7f16f4003970 comp rx=0 tx=0).stop 2024-03-28T22:13:39.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.430+0000 7f170b9ea740 1 -- 172.21.15.16:0/1617326198 shutdown_connections 2024-03-28T22:13:39.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.430+0000 7f170b9ea740 1 --2- 172.21.15.16:0/1617326198 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x560c8b9be4b0 0x560c8b9c08d0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.430+0000 7f170b9ea740 1 --2- 172.21.15.16:0/1617326198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560c8b947970 0x560c8b9bc630 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.430+0000 7f16e9ffb640 0 client.16134 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:13:39.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.430+0000 7f170b9ea740 1 --2- 172.21.15.16:0/1617326198 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b93ed90 0x560c8b94b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.430+0000 7f170b9ea740 1 --2- 172.21.15.16:0/1617326198 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560c8b93e470 0x560c8b94b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.430+0000 7f170b9ea740 1 -- 172.21.15.16:0/1617326198 >> 172.21.15.16:0/1617326198 conn(0x560c8b938ba0 msgr2=0x560c8b942ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:39.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.430+0000 7f170b9ea740 1 -- 172.21.15.16:0/1617326198 shutdown_connections 2024-03-28T22:13:39.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.430+0000 7f170b9ea740 1 -- 172.21.15.16:0/1617326198 wait complete. 2024-03-28T22:13:39.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.430+0000 7f170b9ea740 20 client.16134 trim_cache size 0 max 16384 2024-03-28T22:13:39.445 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:13:39.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.430+0000 7f1708e0f640 1 -- 172.21.15.16:0/1231629875 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f16f00a1ef0 msgr2=0x7f16f00a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:39.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.430+0000 7f1708e0f640 1 --2- 172.21.15.16:0/1231629875 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f16f00a1ef0 0x7f16f00a43b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f16f4009a60 tx=0x7f16f4010040 comp rx=0 tx=0).stop 2024-03-28T22:13:39.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.430+0000 7f1708e0f640 1 -- 172.21.15.16:0/1231629875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1704102970 msgr2=0x7f1704198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:39.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.430+0000 7f1708e0f640 1 --2- 172.21.15.16:0/1231629875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1704102970 0x7f1704198010 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f16fc037040 tx=0x7f16fc0346f0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.430+0000 7f1708e0f640 1 -- 172.21.15.16:0/1231629875 shutdown_connections 2024-03-28T22:13:39.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.430+0000 7f1708e0f640 1 --2- 172.21.15.16:0/1231629875 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f16f00a1ef0 0x7f16f00a43b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.430+0000 7f1708e0f640 1 --2- 172.21.15.16:0/1231629875 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f170410bd40 0x7f170419c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.430+0000 7f1708e0f640 1 --2- 172.21.15.16:0/1231629875 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1704103290 0x7f1704198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.430+0000 7f1708e0f640 1 --2- 172.21.15.16:0/1231629875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1704102970 0x7f1704198010 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.430+0000 7f1708e0f640 1 -- 172.21.15.16:0/1231629875 >> 172.21.15.16:0/1231629875 conn(0x7f1704077770 msgr2=0x7f17040fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:39.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.430+0000 7f1708e0f640 1 -- 172.21.15.16:0/1231629875 shutdown_connections 2024-03-28T22:13:39.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.430+0000 7f1708e0f640 1 -- 172.21.15.16:0/1231629875 wait complete. 2024-03-28T22:13:39.448 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:13:39.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.550+0000 7fee5a0c9640 1 -- 172.21.15.16:0/2581134941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee54107e50 msgr2=0x7fee5410e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:39.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.550+0000 7fee5a0c9640 1 --2- 172.21.15.16:0/2581134941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee54107e50 0x7fee5410e7f0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fee4c009cd0 tx=0x7fee4c02c0f0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.550+0000 7fee5a0c9640 1 -- 172.21.15.16:0/2581134941 shutdown_connections 2024-03-28T22:13:39.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.550+0000 7fee5a0c9640 1 --2- 172.21.15.16:0/2581134941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee54107e50 0x7fee5410e7f0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.550+0000 7fee5a0c9640 1 --2- 172.21.15.16:0/2581134941 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee54103310 0x7fee54107740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.550+0000 7fee5a0c9640 1 --2- 172.21.15.16:0/2581134941 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee54102960 0x7fee54102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.550+0000 7fee5a0c9640 1 -- 172.21.15.16:0/2581134941 >> 172.21.15.16:0/2581134941 conn(0x7fee54077830 msgr2=0x7fee54077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:39.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.550+0000 7fee5a0c9640 1 -- 172.21.15.16:0/2581134941 shutdown_connections 2024-03-28T22:13:39.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.550+0000 7fee5a0c9640 1 -- 172.21.15.16:0/2581134941 wait complete. 2024-03-28T22:13:39.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.550+0000 7fee5a0c9640 1 Processor -- start 2024-03-28T22:13:39.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.551+0000 7fee5a0c9640 1 -- start start 2024-03-28T22:13:39.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.551+0000 7fee5a0c9640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee54102960 0x7fee54198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.551+0000 7fee5a0c9640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee54107e50 0x7fee54198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.551+0000 7fee5a0c9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee5419c9a0 0x7fee5419ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.551+0000 7fee5a0c9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fee54111c50 con 0x7fee5419c9a0 2024-03-28T22:13:39.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.551+0000 7fee5a0c9640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fee54111ad0 con 0x7fee54107e50 2024-03-28T22:13:39.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.551+0000 7fee5a0c9640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fee54111dd0 con 0x7fee54102960 2024-03-28T22:13:39.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.551+0000 7fee53fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee5419c9a0 0x7fee5419ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.551+0000 7fee53fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee5419c9a0 0x7fee5419ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54734/0 (socket says 172.21.15.16:54734) 2024-03-28T22:13:39.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.551+0000 7fee53fff640 1 -- 172.21.15.16:0/1360383489 learned_addr learned my addr 172.21.15.16:0/1360383489 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:39.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.551+0000 7fee52ffd640 1 --2- 172.21.15.16:0/1360383489 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee54107e50 0x7fee54198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.551+0000 7fee537fe640 1 --2- 172.21.15.16:0/1360383489 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee54102960 0x7fee54198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.551+0000 7fee53fff640 1 -- 172.21.15.16:0/1360383489 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee54102960 msgr2=0x7fee54198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:39.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.551+0000 7fee53fff640 1 --2- 172.21.15.16:0/1360383489 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee54102960 0x7fee54198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.551+0000 7fee53fff640 1 -- 172.21.15.16:0/1360383489 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee54107e50 msgr2=0x7fee54198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:39.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.551+0000 7fee53fff640 1 --2- 172.21.15.16:0/1360383489 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee54107e50 0x7fee54198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.551+0000 7fee53fff640 1 -- 172.21.15.16:0/1360383489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee5419d490 con 0x7fee5419c9a0 2024-03-28T22:13:39.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.551+0000 7fee53fff640 1 --2- 172.21.15.16:0/1360383489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee5419c9a0 0x7fee5419ce50 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fee4c0348f0 tx=0x7fee4c032c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:39.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.552+0000 7fee50ff9640 1 -- 172.21.15.16:0/1360383489 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4c041020 con 0x7fee5419c9a0 2024-03-28T22:13:39.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.552+0000 7fee50ff9640 1 -- 172.21.15.16:0/1360383489 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee4c032e80 con 0x7fee5419c9a0 2024-03-28T22:13:39.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.552+0000 7fee5a0c9640 1 -- 172.21.15.16:0/1360383489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee5419d720 con 0x7fee5419c9a0 2024-03-28T22:13:39.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.552+0000 7fee50ff9640 1 -- 172.21.15.16:0/1360383489 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4c039040 con 0x7fee5419c9a0 2024-03-28T22:13:39.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.552+0000 7fee5a0c9640 1 -- 172.21.15.16:0/1360383489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee541a4ee0 con 0x7fee5419c9a0 2024-03-28T22:13:39.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.554+0000 7fee50ff9640 1 -- 172.21.15.16:0/1360383489 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7fee4c044020 con 0x7fee5419c9a0 2024-03-28T22:13:39.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.554+0000 7fee5cd86740 1 Processor -- start 2024-03-28T22:13:39.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.554+0000 7fee5cd86740 1 -- start start 2024-03-28T22:13:39.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.555+0000 7fee50ff9640 1 --2- 172.21.15.16:0/1360383489 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fee3c0a1ef0 0x7fee3c0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.555+0000 7fee5cd86740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fbdf2c7470 0x55fbdf2c7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.555+0000 7fee5cd86740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbdf2c7d90 0x55fbdf2d0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.555+0000 7fee5cd86740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbdf2d0970 0x55fbdf2d2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.555+0000 7fee5cd86740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55fbdf22b1b0 con 0x55fbdf2d0970 2024-03-28T22:13:39.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.555+0000 7fee5cd86740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55fbdf1d3c00 con 0x55fbdf2c7470 2024-03-28T22:13:39.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.555+0000 7fee5cd86740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55fbdf227250 con 0x55fbdf2c7d90 2024-03-28T22:13:39.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.555+0000 7fee52ffd640 1 --2- 172.21.15.16:0/1360383489 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fee3c0a1ef0 0x7fee3c0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.555+0000 7fee537fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fbdf2c7470 0x55fbdf2c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.555+0000 7fee53fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbdf2d0970 0x55fbdf2d2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.555+0000 7fee537fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fbdf2c7470 0x55fbdf2c7850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:38316/0 (socket says 172.21.15.16:38316) 2024-03-28T22:13:39.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.555+0000 7fee537fe640 1 -- 172.21.15.16:0/718881096 learned_addr learned my addr 172.21.15.16:0/718881096 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:39.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.555+0000 7fee50ff9640 1 -- 172.21.15.16:0/1360383489 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fee4c0efca0 con 0x7fee5419c9a0 2024-03-28T22:13:39.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.555+0000 7fee52ffd640 1 --2- 172.21.15.16:0/718881096 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbdf2c7d90 0x55fbdf2d0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.555+0000 7fee537fe640 1 -- 172.21.15.16:0/718881096 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbdf2c7d90 msgr2=0x55fbdf2d0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:39.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.555+0000 7fee537fe640 1 --2- 172.21.15.16:0/718881096 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbdf2c7d90 0x55fbdf2d0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.555+0000 7fee537fe640 1 -- 172.21.15.16:0/718881096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbdf2d0970 msgr2=0x55fbdf2d2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:39.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.555+0000 7fee537fe640 1 --2- 172.21.15.16:0/718881096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbdf2d0970 0x55fbdf2d2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.555+0000 7fee537fe640 1 -- 172.21.15.16:0/718881096 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbdf1f33d0 con 0x55fbdf2c7470 2024-03-28T22:13:39.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.555+0000 7fee52ffd640 1 --2- 172.21.15.16:0/718881096 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbdf2c7d90 0x55fbdf2d0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:13:39.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.555+0000 7fee52ffd640 1 --2- 172.21.15.16:0/1360383489 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fee3c0a1ef0 0x7fee3c0a43b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fee540fede0 tx=0x7fee440097c0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:39.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.555+0000 7fee537fe640 1 --2- 172.21.15.16:0/718881096 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fbdf2c7470 0x55fbdf2c7850 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fee4800b760 tx=0x7fee4800bc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96dea20f894dbdd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:39.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.556+0000 7fee39ffb640 1 -- 172.21.15.16:0/718881096 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee48003dd0 con 0x55fbdf2c7470 2024-03-28T22:13:39.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.556+0000 7fee39ffb640 1 -- 172.21.15.16:0/718881096 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee48015070 con 0x55fbdf2c7470 2024-03-28T22:13:39.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.556+0000 7fee39ffb640 1 -- 172.21.15.16:0/718881096 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee48011440 con 0x55fbdf2c7470 2024-03-28T22:13:39.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.556+0000 7fee5cd86740 1 -- 172.21.15.16:0/718881096 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fbdf2c7470 msgr2=0x55fbdf2c7850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:39.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.556+0000 7fee5cd86740 1 --2- 172.21.15.16:0/718881096 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fbdf2c7470 0x55fbdf2c7850 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fee4800b760 tx=0x7fee4800bc20 comp rx=0 tx=0).stop 2024-03-28T22:13:39.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.556+0000 7fee5cd86740 1 -- 172.21.15.16:0/718881096 shutdown_connections 2024-03-28T22:13:39.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.556+0000 7fee5cd86740 1 --2- 172.21.15.16:0/718881096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbdf2d0970 0x55fbdf2d2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.556+0000 7fee5cd86740 1 --2- 172.21.15.16:0/718881096 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbdf2c7d90 0x55fbdf2d0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.556+0000 7fee5cd86740 1 --2- 172.21.15.16:0/718881096 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fbdf2c7470 0x55fbdf2c7850 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.556+0000 7fee5cd86740 1 -- 172.21.15.16:0/718881096 >> 172.21.15.16:0/718881096 conn(0x55fbdf2c1ba0 msgr2=0x55fbdf2c1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:39.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.556+0000 7fee5cd86740 1 -- 172.21.15.16:0/718881096 shutdown_connections 2024-03-28T22:13:39.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.556+0000 7fee5cd86740 1 -- 172.21.15.16:0/718881096 wait complete. 2024-03-28T22:13:39.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.557+0000 7fee5cd86740 1 -- start start 2024-03-28T22:13:39.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.557+0000 7fee5cd86740 1 Processor -- start 2024-03-28T22:13:39.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.557+0000 7fee5cd86740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fbdf2c7470 0x55fbdf2cd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.557+0000 7fee5cd86740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbdf2c7d90 0x55fbdf2cdf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.557+0000 7fee5cd86740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbdf2d0970 0x55fbdf2ce450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.557+0000 7fee5cd86740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55fbdf2d3760 con 0x55fbdf2d0970 2024-03-28T22:13:39.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.557+0000 7fee5cd86740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55fbdf1c9470 con 0x55fbdf2c7470 2024-03-28T22:13:39.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.557+0000 7fee5cd86740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55fbdf2d38e0 con 0x55fbdf2c7d90 2024-03-28T22:13:39.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.557+0000 7fee5cd86740 10 client.0 _ll_register_callbacks cb 0x55fbdf1e6cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:13:39.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.557+0000 7fee53fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbdf2d0970 0x55fbdf2ce450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.557+0000 7fee537fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fbdf2c7470 0x55fbdf2cd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.557+0000 7fee53fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbdf2d0970 0x55fbdf2ce450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54738/0 (socket says 172.21.15.16:54738) 2024-03-28T22:13:39.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.557+0000 7fee53fff640 1 -- 172.21.15.16:0/1255255963 learned_addr learned my addr 172.21.15.16:0/1255255963 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:39.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.557+0000 7fee52ffd640 1 --2- 172.21.15.16:0/1255255963 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbdf2c7d90 0x55fbdf2cdf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.558+0000 7fee53fff640 1 -- 172.21.15.16:0/1255255963 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbdf2c7d90 msgr2=0x55fbdf2cdf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:39.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.558+0000 7fee53fff640 1 --2- 172.21.15.16:0/1255255963 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbdf2c7d90 0x55fbdf2cdf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.558+0000 7fee53fff640 1 -- 172.21.15.16:0/1255255963 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fbdf2c7470 msgr2=0x55fbdf2cd9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:39.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.558+0000 7fee53fff640 1 --2- 172.21.15.16:0/1255255963 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fbdf2c7470 0x55fbdf2cd9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.558+0000 7fee53fff640 1 -- 172.21.15.16:0/1255255963 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbdef3f3f0 con 0x55fbdf2d0970 2024-03-28T22:13:39.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.558+0000 7fee537fe640 1 --2- 172.21.15.16:0/1255255963 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fbdf2c7470 0x55fbdf2cd9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:13:39.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.558+0000 7fee53fff640 1 --2- 172.21.15.16:0/1255255963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbdf2d0970 0x55fbdf2ce450 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fee4c0bc460 tx=0x7fee4c0bc5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2a56c662a4d421a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:39.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.558+0000 7fee32ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:13:39.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.558+0000 7fee32ffd640 1 -- 172.21.15.16:0/1255255963 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4c034e60 con 0x55fbdf2d0970 2024-03-28T22:13:39.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.558+0000 7fee32ffd640 1 -- 172.21.15.16:0/1255255963 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fee4c0baa60 con 0x55fbdf2d0970 2024-03-28T22:13:39.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.558+0000 7fee5cd86740 1 -- 172.21.15.16:0/1255255963 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee48015070 con 0x55fbdf2d0970 2024-03-28T22:13:39.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.558+0000 7fee32ffd640 1 -- 172.21.15.16:0/1255255963 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fee4c034920 con 0x55fbdf2d0970 2024-03-28T22:13:39.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.558+0000 7fee5cd86740 10 client.16146 fetch_fsmap learned FSMap version 24 2024-03-28T22:13:39.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.558+0000 7fee5cd86740 1 -- 172.21.15.16:0/1255255963 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55fbdf0097d0 con 0x55fbdf2d0970 2024-03-28T22:13:39.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.558+0000 7fee32ffd640 1 -- 172.21.15.16:0/1255255963 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee48015070 con 0x55fbdf2d0970 2024-03-28T22:13:39.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.558+0000 7fee32ffd640 1 -- 172.21.15.16:0/1255255963 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7fee4c03c030 con 0x55fbdf2d0970 2024-03-28T22:13:39.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.558+0000 7fee5cd86740 10 client.16146 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:13:39.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.558+0000 7fee5cd86740 10 client.16146 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T22:13:39.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.558+0000 7fee5cd86740 20 client.16146 populate_metadata read hostname 'smithi016' 2024-03-28T22:13:39.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.558+0000 7fee5cd86740 1 --2- 172.21.15.16:0/1255255963 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55fbdf2e07d0 0x55fbdf2e2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.558+0000 7fee5cd86740 4 client.16146 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:13:39.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.558+0000 7fee5cd86740 1 -- 172.21.15.16:0/1255255963 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fbdf2386f0 con 0x55fbdf2e07d0 2024-03-28T22:13:39.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.559+0000 7fee537fe640 1 --2- 172.21.15.16:0/1255255963 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55fbdf2e07d0 0x55fbdf2e2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.559+0000 7fee32ffd640 10 client.16146 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:13:39.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.559+0000 7fee537fe640 1 --2- 172.21.15.16:0/1255255963 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55fbdf2e07d0 0x55fbdf2e2c40 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6018fda511066b83 server_cookie=52f86ef29c97854d in_seq=0 out_seq=0 2024-03-28T22:13:39.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.560+0000 7fee32ffd640 1 -- 172.21.15.16:0/1255255963 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fbdf2386f0 con 0x55fbdf2e07d0 2024-03-28T22:13:39.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.560+0000 7fee32ffd640 10 client.16146 handle_command_reply: tid=0 2024-03-28T22:13:39.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.560+0000 7fee5cd86740 1 client.16146 shutdown 2024-03-28T22:13:39.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.560+0000 7fee5cd86740 1 -- 172.21.15.16:0/1255255963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbdf2d0970 msgr2=0x55fbdf2ce450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:39.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.560+0000 7fee5cd86740 1 --2- 172.21.15.16:0/1255255963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbdf2d0970 0x55fbdf2ce450 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fee4c0bc460 tx=0x7fee4c0bc5f0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.560+0000 7fee5cd86740 1 -- 172.21.15.16:0/1255255963 shutdown_connections 2024-03-28T22:13:39.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.560+0000 7fee5cd86740 1 --2- 172.21.15.16:0/1255255963 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55fbdf2e07d0 0x55fbdf2e2c40 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.560+0000 7fee5cd86740 1 --2- 172.21.15.16:0/1255255963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbdf2d0970 0x55fbdf2ce450 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.560+0000 7fee32ffd640 0 client.16146 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:13:39.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.560+0000 7fee5cd86740 1 --2- 172.21.15.16:0/1255255963 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbdf2c7d90 0x55fbdf2cdf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.560+0000 7fee5cd86740 1 --2- 172.21.15.16:0/1255255963 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fbdf2c7470 0x55fbdf2cd9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.560+0000 7fee5cd86740 1 -- 172.21.15.16:0/1255255963 >> 172.21.15.16:0/1255255963 conn(0x55fbdf2c1ba0 msgr2=0x55fbdf2c38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:39.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.560+0000 7fee5cd86740 1 -- 172.21.15.16:0/1255255963 shutdown_connections 2024-03-28T22:13:39.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.560+0000 7fee5cd86740 1 -- 172.21.15.16:0/1255255963 wait complete. 2024-03-28T22:13:39.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.560+0000 7fee5cd86740 20 client.16146 trim_cache size 0 max 16384 2024-03-28T22:13:39.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.573+0000 7fee5cd86740 1 Processor -- start 2024-03-28T22:13:39.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.573+0000 7fee5cd86740 1 -- start start 2024-03-28T22:13:39.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.573+0000 7fee5cd86740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fbdf2c7470 0x55fbdf2c0f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.573+0000 7fee5cd86740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbdf2c7d90 0x55fbdf2cd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.573+0000 7fee5cd86740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbdf2d0970 0x55fbdf2dc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.573+0000 7fee5cd86740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fee541afc20 con 0x55fbdf2d0970 2024-03-28T22:13:39.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.573+0000 7fee5cd86740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fee541ad4c0 con 0x55fbdf2c7470 2024-03-28T22:13:39.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.574+0000 7fee5cd86740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fee541ab330 con 0x55fbdf2c7d90 2024-03-28T22:13:39.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.574+0000 7fee53fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbdf2d0970 0x55fbdf2dc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.574+0000 7fee537fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fbdf2c7470 0x55fbdf2c0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.574+0000 7fee53fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbdf2d0970 0x55fbdf2dc500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54750/0 (socket says 172.21.15.16:54750) 2024-03-28T22:13:39.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.574+0000 7fee53fff640 1 -- 172.21.15.16:0/2945607908 learned_addr learned my addr 172.21.15.16:0/2945607908 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:39.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.574+0000 7fee52ffd640 1 --2- 172.21.15.16:0/2945607908 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbdf2c7d90 0x55fbdf2cd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.574+0000 7fee53fff640 1 -- 172.21.15.16:0/2945607908 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbdf2c7d90 msgr2=0x55fbdf2cd2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:39.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.574+0000 7fee53fff640 1 --2- 172.21.15.16:0/2945607908 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbdf2c7d90 0x55fbdf2cd2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.574+0000 7fee53fff640 1 -- 172.21.15.16:0/2945607908 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fbdf2c7470 msgr2=0x55fbdf2c0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:39.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.574+0000 7fee53fff640 1 --2- 172.21.15.16:0/2945607908 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fbdf2c7470 0x55fbdf2c0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.574+0000 7fee53fff640 1 -- 172.21.15.16:0/2945607908 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbdf2c1900 con 0x55fbdf2d0970 2024-03-28T22:13:39.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.574+0000 7fee537fe640 1 --2- 172.21.15.16:0/2945607908 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fbdf2c7470 0x55fbdf2c0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:13:39.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.574+0000 7fee52ffd640 1 --2- 172.21.15.16:0/2945607908 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbdf2c7d90 0x55fbdf2cd2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:13:39.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.574+0000 7fee53fff640 1 --2- 172.21.15.16:0/2945607908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbdf2d0970 0x55fbdf2dc500 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fee4c0b8bc0 tx=0x7fee4c0a03d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adb591d3a01e2102 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:39.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.574+0000 7fee397fa640 1 -- 172.21.15.16:0/2945607908 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4c0bc0c0 con 0x55fbdf2d0970 2024-03-28T22:13:39.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.574+0000 7fee397fa640 1 -- 172.21.15.16:0/2945607908 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee4c034920 con 0x55fbdf2d0970 2024-03-28T22:13:39.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.574+0000 7fee397fa640 1 -- 172.21.15.16:0/2945607908 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4c0ed800 con 0x55fbdf2d0970 2024-03-28T22:13:39.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.575+0000 7fee5cd86740 1 -- 172.21.15.16:0/2945607908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbdf2d0970 msgr2=0x55fbdf2dc500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:39.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.575+0000 7fee5cd86740 1 --2- 172.21.15.16:0/2945607908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbdf2d0970 0x55fbdf2dc500 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fee4c0b8bc0 tx=0x7fee4c0a03d0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.575+0000 7fee5cd86740 1 -- 172.21.15.16:0/2945607908 shutdown_connections 2024-03-28T22:13:39.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.575+0000 7fee5cd86740 1 --2- 172.21.15.16:0/2945607908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbdf2d0970 0x55fbdf2dc500 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.575+0000 7fee5cd86740 1 --2- 172.21.15.16:0/2945607908 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbdf2c7d90 0x55fbdf2cd2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.575+0000 7fee5cd86740 1 --2- 172.21.15.16:0/2945607908 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fbdf2c7470 0x55fbdf2c0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.575+0000 7fee5cd86740 1 -- 172.21.15.16:0/2945607908 >> 172.21.15.16:0/2945607908 conn(0x55fbdf2c1ba0 msgr2=0x55fbdf2cf980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:39.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.575+0000 7fee5cd86740 1 -- 172.21.15.16:0/2945607908 shutdown_connections 2024-03-28T22:13:39.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.575+0000 7fee5cd86740 1 -- 172.21.15.16:0/2945607908 wait complete. 2024-03-28T22:13:39.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.575+0000 7fee5cd86740 1 -- start start 2024-03-28T22:13:39.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.575+0000 7fee5cd86740 1 Processor -- start 2024-03-28T22:13:39.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.575+0000 7fee5cd86740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbdf2c7470 0x55fbdf2cab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.575+0000 7fee5cd86740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fbdf2c7d90 0x55fbdf2cb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.575+0000 7fee5cd86740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbdf2d0970 0x55fbdf345b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.575+0000 7fee5cd86740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fee541ab980 con 0x55fbdf2c7470 2024-03-28T22:13:39.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.575+0000 7fee5cd86740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fee541acdf0 con 0x55fbdf2c7d90 2024-03-28T22:13:39.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.575+0000 7fee5cd86740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fee541ac050 con 0x55fbdf2d0970 2024-03-28T22:13:39.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.576+0000 7fee5cd86740 10 client.0 _ll_register_callbacks cb 0x7fee4c0b8a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:13:39.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.576+0000 7fee537fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbdf2c7470 0x55fbdf2cab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.576+0000 7fee537fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbdf2c7470 0x55fbdf2cab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54762/0 (socket says 172.21.15.16:54762) 2024-03-28T22:13:39.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.576+0000 7fee537fe640 1 -- 172.21.15.16:0/2400375252 learned_addr learned my addr 172.21.15.16:0/2400375252 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:39.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.576+0000 7fee52ffd640 1 --2- 172.21.15.16:0/2400375252 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fbdf2c7d90 0x55fbdf2cb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.576+0000 7fee53fff640 1 --2- 172.21.15.16:0/2400375252 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbdf2d0970 0x55fbdf345b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.576+0000 7fee537fe640 1 -- 172.21.15.16:0/2400375252 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbdf2d0970 msgr2=0x55fbdf345b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:39.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.576+0000 7fee537fe640 1 --2- 172.21.15.16:0/2400375252 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbdf2d0970 0x55fbdf345b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.576+0000 7fee537fe640 1 -- 172.21.15.16:0/2400375252 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fbdf2c7d90 msgr2=0x55fbdf2cb0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:39.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.576+0000 7fee537fe640 1 --2- 172.21.15.16:0/2400375252 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fbdf2c7d90 0x55fbdf2cb0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.576+0000 7fee537fe640 1 -- 172.21.15.16:0/2400375252 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbdf346050 con 0x55fbdf2c7470 2024-03-28T22:13:39.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.576+0000 7fee53fff640 1 --2- 172.21.15.16:0/2400375252 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbdf2d0970 0x55fbdf345b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:13:39.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.576+0000 7fee537fe640 1 --2- 172.21.15.16:0/2400375252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbdf2c7470 0x55fbdf2cab90 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fee480039c0 tx=0x7fee48004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c96200a9dcae8e33 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:39.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.576+0000 7fee32ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:13:39.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.576+0000 7fee32ffd640 1 -- 172.21.15.16:0/2400375252 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4802c020 con 0x55fbdf2c7470 2024-03-28T22:13:39.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.576+0000 7fee32ffd640 1 -- 172.21.15.16:0/2400375252 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fee480280e0 con 0x55fbdf2c7470 2024-03-28T22:13:39.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.576+0000 7fee5cd86740 1 -- 172.21.15.16:0/2400375252 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee4c034920 con 0x55fbdf2c7470 2024-03-28T22:13:39.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.576+0000 7fee32ffd640 1 -- 172.21.15.16:0/2400375252 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fee48037050 con 0x55fbdf2c7470 2024-03-28T22:13:39.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.576+0000 7fee5cd86740 10 client.16152 fetch_fsmap learned FSMap version 24 2024-03-28T22:13:39.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.576+0000 7fee5cd86740 1 -- 172.21.15.16:0/2400375252 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55fbdf3471c0 con 0x55fbdf2c7470 2024-03-28T22:13:39.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.576+0000 7fee32ffd640 1 -- 172.21.15.16:0/2400375252 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee4c034920 con 0x55fbdf2c7470 2024-03-28T22:13:39.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.577+0000 7fee32ffd640 1 -- 172.21.15.16:0/2400375252 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7fee48011930 con 0x55fbdf2c7470 2024-03-28T22:13:39.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.577+0000 7fee5cd86740 10 client.16152 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:13:39.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.577+0000 7fee5cd86740 10 client.16152 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T22:13:39.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.577+0000 7fee5cd86740 20 client.16152 populate_metadata read hostname 'smithi016' 2024-03-28T22:13:39.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.577+0000 7fee5cd86740 1 --2- 172.21.15.16:0/2400375252 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55fbdf347600 0x55fbdf3499f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.577+0000 7fee5cd86740 4 client.16152 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:13:39.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.577+0000 7fee5cd86740 1 -- 172.21.15.16:0/2400375252 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fbdf232a40 con 0x55fbdf347600 2024-03-28T22:13:39.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.577+0000 7fee52ffd640 1 --2- 172.21.15.16:0/2400375252 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55fbdf347600 0x55fbdf3499f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.577+0000 7fee32ffd640 10 client.16152 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:13:39.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.577+0000 7fee52ffd640 1 --2- 172.21.15.16:0/2400375252 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55fbdf347600 0x55fbdf3499f0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2dc432da20bcf342 server_cookie=38d6fcc9f1812dfe in_seq=0 out_seq=0 2024-03-28T22:13:39.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.577+0000 7fee32ffd640 1 -- 172.21.15.16:0/2400375252 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fbdf232a40 con 0x55fbdf347600 2024-03-28T22:13:39.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.577+0000 7fee32ffd640 10 client.16152 handle_command_reply: tid=0 2024-03-28T22:13:39.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.577+0000 7fee5cd86740 1 client.16152 shutdown 2024-03-28T22:13:39.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.578+0000 7fee5cd86740 1 -- 172.21.15.16:0/2400375252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbdf2c7470 msgr2=0x55fbdf2cab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:39.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.578+0000 7fee5cd86740 1 --2- 172.21.15.16:0/2400375252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbdf2c7470 0x55fbdf2cab90 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fee480039c0 tx=0x7fee48004af0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.578+0000 7fee5cd86740 1 -- 172.21.15.16:0/2400375252 shutdown_connections 2024-03-28T22:13:39.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.578+0000 7fee5cd86740 1 --2- 172.21.15.16:0/2400375252 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55fbdf347600 0x55fbdf3499f0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.578+0000 7fee5cd86740 1 --2- 172.21.15.16:0/2400375252 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbdf2d0970 0x55fbdf345b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.578+0000 7fee5cd86740 1 --2- 172.21.15.16:0/2400375252 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fbdf2c7d90 0x55fbdf2cb0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.578+0000 7fee5cd86740 1 --2- 172.21.15.16:0/2400375252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbdf2c7470 0x55fbdf2cab90 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.578+0000 7fee32ffd640 0 client.16152 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:13:39.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.578+0000 7fee5cd86740 1 -- 172.21.15.16:0/2400375252 >> 172.21.15.16:0/2400375252 conn(0x55fbdf2c1ba0 msgr2=0x55fbdf2cf010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:39.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.578+0000 7fee5cd86740 1 -- 172.21.15.16:0/2400375252 shutdown_connections 2024-03-28T22:13:39.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.578+0000 7fee5cd86740 1 -- 172.21.15.16:0/2400375252 wait complete. 2024-03-28T22:13:39.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.578+0000 7fee5cd86740 20 client.16152 trim_cache size 0 max 16384 2024-03-28T22:13:39.600 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:13:39.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.578+0000 7fee5a0c9640 1 -- 172.21.15.16:0/1360383489 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fee3c0a1ef0 msgr2=0x7fee3c0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:39.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.578+0000 7fee5a0c9640 1 --2- 172.21.15.16:0/1360383489 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fee3c0a1ef0 0x7fee3c0a43b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fee540fede0 tx=0x7fee440097c0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.578+0000 7fee5a0c9640 1 -- 172.21.15.16:0/1360383489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee5419c9a0 msgr2=0x7fee5419ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:39.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.578+0000 7fee5a0c9640 1 --2- 172.21.15.16:0/1360383489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee5419c9a0 0x7fee5419ce50 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fee4c0348f0 tx=0x7fee4c032c90 comp rx=0 tx=0).stop 2024-03-28T22:13:39.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.578+0000 7fee5a0c9640 1 -- 172.21.15.16:0/1360383489 shutdown_connections 2024-03-28T22:13:39.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.578+0000 7fee5a0c9640 1 --2- 172.21.15.16:0/1360383489 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fee3c0a1ef0 0x7fee3c0a43b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.578+0000 7fee5a0c9640 1 --2- 172.21.15.16:0/1360383489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee5419c9a0 0x7fee5419ce50 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.578+0000 7fee5a0c9640 1 --2- 172.21.15.16:0/1360383489 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee54107e50 0x7fee54198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.578+0000 7fee5a0c9640 1 --2- 172.21.15.16:0/1360383489 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee54102960 0x7fee54198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.578+0000 7fee5a0c9640 1 -- 172.21.15.16:0/1360383489 >> 172.21.15.16:0/1360383489 conn(0x7fee54077830 msgr2=0x7fee540ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:39.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.578+0000 7fee5a0c9640 1 -- 172.21.15.16:0/1360383489 shutdown_connections 2024-03-28T22:13:39.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.579+0000 7fee5a0c9640 1 -- 172.21.15.16:0/1360383489 wait complete. 2024-03-28T22:13:39.603 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:13:39.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.706+0000 7fe27708f640 1 -- 172.21.15.16:0/2980763741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2701021e0 msgr2=0x7fe270102640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:39.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.706+0000 7fe27708f640 1 --2- 172.21.15.16:0/2980763741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2701021e0 0x7fe270102640 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fe25c009900 tx=0x7fe25c02bd10 comp rx=0 tx=0).stop 2024-03-28T22:13:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.706+0000 7fe27708f640 1 -- 172.21.15.16:0/2980763741 shutdown_connections 2024-03-28T22:13:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.706+0000 7fe27708f640 1 --2- 172.21.15.16:0/2980763741 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe270102b80 0x7fe27010d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.706+0000 7fe27708f640 1 --2- 172.21.15.16:0/2980763741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2701021e0 0x7fe270102640 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.706+0000 7fe27708f640 1 --2- 172.21.15.16:0/2980763741 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2701009d0 0x7fe270100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.706+0000 7fe27708f640 1 -- 172.21.15.16:0/2980763741 >> 172.21.15.16:0/2980763741 conn(0x7fe270077830 msgr2=0x7fe270077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.706+0000 7fe27708f640 1 -- 172.21.15.16:0/2980763741 shutdown_connections 2024-03-28T22:13:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.706+0000 7fe27708f640 1 -- 172.21.15.16:0/2980763741 wait complete. 2024-03-28T22:13:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.706+0000 7fe27708f640 1 Processor -- start 2024-03-28T22:13:39.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.706+0000 7fe27708f640 1 -- start start 2024-03-28T22:13:39.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.706+0000 7fe27708f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2701009d0 0x7fe270198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.706+0000 7fe27708f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2701021e0 0x7fe2701985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.706+0000 7fe27708f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe270102b80 0x7fe27019c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.706+0000 7fe27708f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe270110b70 con 0x7fe2701009d0 2024-03-28T22:13:39.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.706+0000 7fe27708f640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe2701109f0 con 0x7fe270102b80 2024-03-28T22:13:39.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.706+0000 7fe27708f640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe270110cf0 con 0x7fe2701021e0 2024-03-28T22:13:39.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.707+0000 7fe274e2b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2701009d0 0x7fe270198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.707+0000 7fe274e2b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2701009d0 0x7fe270198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54794/0 (socket says 172.21.15.16:54794) 2024-03-28T22:13:39.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.707+0000 7fe274e2b640 1 -- 172.21.15.16:0/1323580760 learned_addr learned my addr 172.21.15.16:0/1323580760 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:39.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.707+0000 7fe27562c640 1 --2- 172.21.15.16:0/1323580760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe270102b80 0x7fe27019c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.707+0000 7fe26ffff640 1 --2- 172.21.15.16:0/1323580760 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2701021e0 0x7fe2701985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.707+0000 7fe274e2b640 1 -- 172.21.15.16:0/1323580760 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2701021e0 msgr2=0x7fe2701985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:39.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.707+0000 7fe274e2b640 1 --2- 172.21.15.16:0/1323580760 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2701021e0 0x7fe2701985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.707+0000 7fe274e2b640 1 -- 172.21.15.16:0/1323580760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe270102b80 msgr2=0x7fe27019c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:39.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.707+0000 7fe274e2b640 1 --2- 172.21.15.16:0/1323580760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe270102b80 0x7fe27019c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.707+0000 7fe274e2b640 1 -- 172.21.15.16:0/1323580760 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe27019d070 con 0x7fe2701009d0 2024-03-28T22:13:39.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.707+0000 7fe27562c640 1 --2- 172.21.15.16:0/1323580760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe270102b80 0x7fe27019c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:13:39.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.707+0000 7fe274e2b640 1 --2- 172.21.15.16:0/1323580760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2701009d0 0x7fe270198080 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fe26400d770 tx=0x7fe26400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:39.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.707+0000 7fe26dffb640 1 -- 172.21.15.16:0/1323580760 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe264017070 con 0x7fe2701009d0 2024-03-28T22:13:39.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.707+0000 7fe27708f640 1 -- 172.21.15.16:0/1323580760 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe27019d360 con 0x7fe2701009d0 2024-03-28T22:13:39.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.707+0000 7fe26dffb640 1 -- 172.21.15.16:0/1323580760 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe264005500 con 0x7fe2701009d0 2024-03-28T22:13:39.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.707+0000 7fe26dffb640 1 -- 172.21.15.16:0/1323580760 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe264004e00 con 0x7fe2701009d0 2024-03-28T22:13:39.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.708+0000 7fe27708f640 1 -- 172.21.15.16:0/1323580760 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2701a4bf0 con 0x7fe2701009d0 2024-03-28T22:13:39.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.709+0000 7fe26dffb640 1 -- 172.21.15.16:0/1323580760 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7fe264004fa0 con 0x7fe2701009d0 2024-03-28T22:13:39.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.710+0000 7fe279c76740 1 Processor -- start 2024-03-28T22:13:39.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.710+0000 7fe279c76740 1 -- start start 2024-03-28T22:13:39.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.710+0000 7fe26dffb640 1 --2- 172.21.15.16:0/1323580760 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fe2600a1ef0 0x7fe2600a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.710+0000 7fe279c76740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56488dea3470 0x56488dea3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.710+0000 7fe279c76740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56488dea3d90 0x56488deac430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.710+0000 7fe279c76740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56488deac970 0x56488deaed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.710+0000 7fe279c76740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56488de071b0 con 0x56488deac970 2024-03-28T22:13:39.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.710+0000 7fe279c76740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x56488ddafc00 con 0x56488dea3470 2024-03-28T22:13:39.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.710+0000 7fe279c76740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56488de03250 con 0x56488dea3d90 2024-03-28T22:13:39.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.710+0000 7fe27562c640 1 --2- 172.21.15.16:0/1323580760 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fe2600a1ef0 0x7fe2600a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.710+0000 7fe26dffb640 1 -- 172.21.15.16:0/1323580760 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fe2640cdb30 con 0x7fe2701009d0 2024-03-28T22:13:39.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.710+0000 7fe274e2b640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56488dea3470 0x56488dea3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.710+0000 7fe274e2b640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56488dea3470 0x56488dea3850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:38378/0 (socket says 172.21.15.16:38378) 2024-03-28T22:13:39.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.710+0000 7fe27562c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56488deac970 0x56488deaed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.710+0000 7fe274e2b640 1 -- 172.21.15.16:0/2623901279 learned_addr learned my addr 172.21.15.16:0/2623901279 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:39.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.710+0000 7fe26ffff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56488dea3d90 0x56488deac430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.711+0000 7fe274e2b640 1 -- 172.21.15.16:0/2623901279 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56488dea3d90 msgr2=0x56488deac430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:39.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.711+0000 7fe274e2b640 1 --2- 172.21.15.16:0/2623901279 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56488dea3d90 0x56488deac430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.711+0000 7fe274e2b640 1 -- 172.21.15.16:0/2623901279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56488deac970 msgr2=0x56488deaed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:39.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.711+0000 7fe274e2b640 1 --2- 172.21.15.16:0/2623901279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56488deac970 0x56488deaed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.711+0000 7fe274e2b640 1 -- 172.21.15.16:0/2623901279 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56488ddcf3d0 con 0x56488dea3470 2024-03-28T22:13:39.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.711+0000 7fe27562c640 1 --2- 172.21.15.16:0/1323580760 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fe2600a1ef0 0x7fe2600a43b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe268005fd0 tx=0x7fe268011690 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:39.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.711+0000 7fe274e2b640 1 --2- 172.21.15.16:0/2623901279 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56488dea3470 0x56488dea3850 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe264009610 tx=0x7fe26409a010 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3452ad206e7e65dd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:39.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.711+0000 7fe25affd640 1 -- 172.21.15.16:0/2623901279 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe264020070 con 0x56488dea3470 2024-03-28T22:13:39.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.711+0000 7fe25affd640 1 -- 172.21.15.16:0/2623901279 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2640c8980 con 0x56488dea3470 2024-03-28T22:13:39.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.711+0000 7fe25affd640 1 -- 172.21.15.16:0/2623901279 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe264009cc0 con 0x56488dea3470 2024-03-28T22:13:39.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.712+0000 7fe279c76740 1 -- 172.21.15.16:0/2623901279 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56488dea3470 msgr2=0x56488dea3850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:39.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.712+0000 7fe279c76740 1 --2- 172.21.15.16:0/2623901279 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56488dea3470 0x56488dea3850 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe264009610 tx=0x7fe26409a010 comp rx=0 tx=0).stop 2024-03-28T22:13:39.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.712+0000 7fe279c76740 1 -- 172.21.15.16:0/2623901279 shutdown_connections 2024-03-28T22:13:39.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.712+0000 7fe279c76740 1 --2- 172.21.15.16:0/2623901279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56488deac970 0x56488deaed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.712+0000 7fe279c76740 1 --2- 172.21.15.16:0/2623901279 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56488dea3d90 0x56488deac430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.712+0000 7fe279c76740 1 --2- 172.21.15.16:0/2623901279 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56488dea3470 0x56488dea3850 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.712+0000 7fe279c76740 1 -- 172.21.15.16:0/2623901279 >> 172.21.15.16:0/2623901279 conn(0x56488de9dba0 msgr2=0x56488de9df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:39.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.712+0000 7fe279c76740 1 -- 172.21.15.16:0/2623901279 shutdown_connections 2024-03-28T22:13:39.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.712+0000 7fe279c76740 1 -- 172.21.15.16:0/2623901279 wait complete. 2024-03-28T22:13:39.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.712+0000 7fe279c76740 1 -- start start 2024-03-28T22:13:39.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.712+0000 7fe279c76740 1 Processor -- start 2024-03-28T22:13:39.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.713+0000 7fe279c76740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56488dea3470 0x56488dea9a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.713+0000 7fe279c76740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56488dea3d90 0x56488dea9f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.713+0000 7fe279c76740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56488deac970 0x56488deaa4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.713+0000 7fe279c76740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56488deaf6a0 con 0x56488deac970 2024-03-28T22:13:39.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.713+0000 7fe279c76740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x56488dda5470 con 0x56488dea3d90 2024-03-28T22:13:39.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.713+0000 7fe279c76740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56488deaf820 con 0x56488dea3470 2024-03-28T22:13:39.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.713+0000 7fe279c76740 10 client.0 _ll_register_callbacks cb 0x56488ddc2cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:13:39.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.713+0000 7fe27562c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56488deac970 0x56488deaa4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.713+0000 7fe26ffff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56488dea3d90 0x56488dea9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.713+0000 7fe27562c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56488deac970 0x56488deaa4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54824/0 (socket says 172.21.15.16:54824) 2024-03-28T22:13:39.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.713+0000 7fe27562c640 1 -- 172.21.15.16:0/1655068985 learned_addr learned my addr 172.21.15.16:0/1655068985 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:39.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.713+0000 7fe274e2b640 1 --2- 172.21.15.16:0/1655068985 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56488dea3470 0x56488dea9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.713+0000 7fe27562c640 1 -- 172.21.15.16:0/1655068985 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56488dea3470 msgr2=0x56488dea9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:39.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.713+0000 7fe27562c640 1 --2- 172.21.15.16:0/1655068985 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56488dea3470 0x56488dea9a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.713+0000 7fe27562c640 1 -- 172.21.15.16:0/1655068985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56488dea3d90 msgr2=0x56488dea9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:39.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.713+0000 7fe27562c640 1 --2- 172.21.15.16:0/1655068985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56488dea3d90 0x56488dea9f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.713+0000 7fe27562c640 1 -- 172.21.15.16:0/1655068985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56488db1b3f0 con 0x56488deac970 2024-03-28T22:13:39.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.713+0000 7fe26ffff640 1 --2- 172.21.15.16:0/1655068985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56488dea3d90 0x56488dea9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:13:39.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.713+0000 7fe27562c640 1 --2- 172.21.15.16:0/1655068985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56488deac970 0x56488deaa4b0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fe26801e8d0 tx=0x7fe26801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45d9da19ab95661e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:39.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.713+0000 7fe24ffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:13:39.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.713+0000 7fe24ffff640 1 -- 172.21.15.16:0/1655068985 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe268024070 con 0x56488deac970 2024-03-28T22:13:39.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.713+0000 7fe24ffff640 1 -- 172.21.15.16:0/1655068985 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fe268004b20 con 0x56488deac970 2024-03-28T22:13:39.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.713+0000 7fe279c76740 1 -- 172.21.15.16:0/1655068985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2640c8980 con 0x56488deac970 2024-03-28T22:13:39.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.714+0000 7fe24ffff640 1 -- 172.21.15.16:0/1655068985 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe268004d20 con 0x56488deac970 2024-03-28T22:13:39.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.714+0000 7fe279c76740 10 client.16164 fetch_fsmap learned FSMap version 24 2024-03-28T22:13:39.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.714+0000 7fe279c76740 1 -- 172.21.15.16:0/1655068985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56488dbe57d0 con 0x56488deac970 2024-03-28T22:13:39.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.714+0000 7fe24ffff640 1 -- 172.21.15.16:0/1655068985 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2640c8980 con 0x56488deac970 2024-03-28T22:13:39.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.714+0000 7fe24ffff640 1 -- 172.21.15.16:0/1655068985 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7fe268031020 con 0x56488deac970 2024-03-28T22:13:39.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.714+0000 7fe279c76740 10 client.16164 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:13:39.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.714+0000 7fe279c76740 10 client.16164 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:13:39.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.714+0000 7fe279c76740 20 client.16164 populate_metadata read hostname 'smithi016' 2024-03-28T22:13:39.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.714+0000 7fe279c76740 1 --2- 172.21.15.16:0/1655068985 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x56488debc8d0 0x56488debed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.714+0000 7fe279c76740 4 client.16164 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:13:39.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.714+0000 7fe279c76740 1 -- 172.21.15.16:0/1655068985 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56488de146f0 con 0x56488debc8d0 2024-03-28T22:13:39.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.714+0000 7fe274e2b640 1 --2- 172.21.15.16:0/1655068985 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x56488debc8d0 0x56488debed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.715+0000 7fe274e2b640 1 --2- 172.21.15.16:0/1655068985 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x56488debc8d0 0x56488debed10 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60e2da17172e7b46 server_cookie=aa77717de59d0478 in_seq=0 out_seq=0 2024-03-28T22:13:39.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.715+0000 7fe24ffff640 10 client.16164 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:13:39.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.716+0000 7fe24ffff640 1 -- 172.21.15.16:0/1655068985 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56488de146f0 con 0x56488debc8d0 2024-03-28T22:13:39.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.716+0000 7fe24ffff640 10 client.16164 handle_command_reply: tid=0 2024-03-28T22:13:39.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.716+0000 7fe279c76740 1 client.16164 shutdown 2024-03-28T22:13:39.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.716+0000 7fe279c76740 1 -- 172.21.15.16:0/1655068985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56488deac970 msgr2=0x56488deaa4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:39.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.716+0000 7fe279c76740 1 --2- 172.21.15.16:0/1655068985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56488deac970 0x56488deaa4b0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fe26801e8d0 tx=0x7fe26801d000 comp rx=0 tx=0).stop 2024-03-28T22:13:39.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.716+0000 7fe279c76740 1 -- 172.21.15.16:0/1655068985 shutdown_connections 2024-03-28T22:13:39.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.716+0000 7fe279c76740 1 --2- 172.21.15.16:0/1655068985 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x56488debc8d0 0x56488debed10 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.716+0000 7fe279c76740 1 --2- 172.21.15.16:0/1655068985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56488deac970 0x56488deaa4b0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.716+0000 7fe279c76740 1 --2- 172.21.15.16:0/1655068985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56488dea3d90 0x56488dea9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.716+0000 7fe24ffff640 0 client.16164 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:13:39.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.716+0000 7fe279c76740 1 --2- 172.21.15.16:0/1655068985 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56488dea3470 0x56488dea9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.716+0000 7fe279c76740 1 -- 172.21.15.16:0/1655068985 >> 172.21.15.16:0/1655068985 conn(0x56488de9dba0 msgr2=0x56488de9f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:39.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.716+0000 7fe279c76740 1 -- 172.21.15.16:0/1655068985 shutdown_connections 2024-03-28T22:13:39.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.716+0000 7fe279c76740 1 -- 172.21.15.16:0/1655068985 wait complete. 2024-03-28T22:13:39.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.716+0000 7fe279c76740 20 client.16164 trim_cache size 0 max 16384 2024-03-28T22:13:39.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.729+0000 7fe279c76740 1 Processor -- start 2024-03-28T22:13:39.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.729+0000 7fe279c76740 1 -- start start 2024-03-28T22:13:39.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.729+0000 7fe279c76740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56488dea3470 0x56488deac160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.729+0000 7fe279c76740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56488dea3d90 0x56488deb8440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.729+0000 7fe279c76740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56488deac970 0x56488dec45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.729+0000 7fe279c76740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe2701afc30 con 0x56488dea3d90 2024-03-28T22:13:39.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.729+0000 7fe279c76740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe2701ad4d0 con 0x56488dea3470 2024-03-28T22:13:39.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.729+0000 7fe279c76740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe2701ab340 con 0x56488deac970 2024-03-28T22:13:39.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.729+0000 7fe26ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56488dea3d90 0x56488deb8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.729+0000 7fe26ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56488dea3d90 0x56488deb8440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54828/0 (socket says 172.21.15.16:54828) 2024-03-28T22:13:39.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.729+0000 7fe26ffff640 1 -- 172.21.15.16:0/1858612128 learned_addr learned my addr 172.21.15.16:0/1858612128 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:39.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.729+0000 7fe274e2b640 1 --2- 172.21.15.16:0/1858612128 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56488dea3470 0x56488deac160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.730+0000 7fe27562c640 1 --2- 172.21.15.16:0/1858612128 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56488deac970 0x56488dec45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.730+0000 7fe26ffff640 1 -- 172.21.15.16:0/1858612128 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56488deac970 msgr2=0x56488dec45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:39.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.730+0000 7fe26ffff640 1 --2- 172.21.15.16:0/1858612128 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56488deac970 0x56488dec45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.730+0000 7fe26ffff640 1 -- 172.21.15.16:0/1858612128 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56488dea3470 msgr2=0x56488deac160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:39.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.730+0000 7fe26ffff640 1 --2- 172.21.15.16:0/1858612128 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56488dea3470 0x56488deac160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.730+0000 7fe26ffff640 1 -- 172.21.15.16:0/1858612128 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56488de9d6d0 con 0x56488dea3d90 2024-03-28T22:13:39.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.730+0000 7fe27562c640 1 --2- 172.21.15.16:0/1858612128 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56488deac970 0x56488dec45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:13:39.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.730+0000 7fe26ffff640 1 --2- 172.21.15.16:0/1858612128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56488dea3d90 0x56488deb8440 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fe25c009230 tx=0x7fe25c005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38744adbd3ecf4cd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:39.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.730+0000 7fe25a7fc640 1 -- 172.21.15.16:0/1858612128 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe25c004350 con 0x56488dea3d90 2024-03-28T22:13:39.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.730+0000 7fe25a7fc640 1 -- 172.21.15.16:0/1858612128 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe25c02ce10 con 0x56488dea3d90 2024-03-28T22:13:39.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.730+0000 7fe25a7fc640 1 -- 172.21.15.16:0/1858612128 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe25c0051d0 con 0x56488dea3d90 2024-03-28T22:13:39.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.730+0000 7fe279c76740 1 -- 172.21.15.16:0/1858612128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56488dea3d90 msgr2=0x56488deb8440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:39.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.730+0000 7fe279c76740 1 --2- 172.21.15.16:0/1858612128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56488dea3d90 0x56488deb8440 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fe25c009230 tx=0x7fe25c005a20 comp rx=0 tx=0).stop 2024-03-28T22:13:39.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.731+0000 7fe279c76740 1 -- 172.21.15.16:0/1858612128 shutdown_connections 2024-03-28T22:13:39.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.731+0000 7fe279c76740 1 --2- 172.21.15.16:0/1858612128 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56488deac970 0x56488dec45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.731+0000 7fe279c76740 1 --2- 172.21.15.16:0/1858612128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56488dea3d90 0x56488deb8440 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.731+0000 7fe279c76740 1 --2- 172.21.15.16:0/1858612128 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56488dea3470 0x56488deac160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.731+0000 7fe279c76740 1 -- 172.21.15.16:0/1858612128 >> 172.21.15.16:0/1858612128 conn(0x56488de9dba0 msgr2=0x56488dea4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:39.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.731+0000 7fe279c76740 1 -- 172.21.15.16:0/1858612128 shutdown_connections 2024-03-28T22:13:39.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.731+0000 7fe279c76740 1 -- 172.21.15.16:0/1858612128 wait complete. 2024-03-28T22:13:39.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.731+0000 7fe279c76740 1 -- start start 2024-03-28T22:13:39.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.731+0000 7fe279c76740 1 Processor -- start 2024-03-28T22:13:39.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.731+0000 7fe279c76740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56488dea3470 0x56488deb03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.731+0000 7fe279c76740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56488dea3d90 0x56488deb0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.731+0000 7fe279c76740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56488deac970 0x56488deb0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.731+0000 7fe279c76740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe2701ab990 con 0x56488dea3470 2024-03-28T22:13:39.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.731+0000 7fe279c76740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe2701ace00 con 0x56488dea3d90 2024-03-28T22:13:39.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.731+0000 7fe279c76740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe2701ac060 con 0x56488deac970 2024-03-28T22:13:39.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.731+0000 7fe279c76740 10 client.0 _ll_register_callbacks cb 0x7fe26801e7e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:13:39.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.731+0000 7fe274e2b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56488dea3470 0x56488deb03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.731+0000 7fe274e2b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56488dea3470 0x56488deb03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54840/0 (socket says 172.21.15.16:54840) 2024-03-28T22:13:39.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.731+0000 7fe274e2b640 1 -- 172.21.15.16:0/2795649402 learned_addr learned my addr 172.21.15.16:0/2795649402 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:39.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.731+0000 7fe26ffff640 1 --2- 172.21.15.16:0/2795649402 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56488dea3d90 0x56488deb0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.732+0000 7fe27562c640 1 --2- 172.21.15.16:0/2795649402 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56488deac970 0x56488deb0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.732+0000 7fe274e2b640 1 -- 172.21.15.16:0/2795649402 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56488deac970 msgr2=0x56488deb0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:39.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.732+0000 7fe274e2b640 1 --2- 172.21.15.16:0/2795649402 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56488deac970 0x56488deb0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.732+0000 7fe274e2b640 1 -- 172.21.15.16:0/2795649402 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56488dea3d90 msgr2=0x56488deb0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:39.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.732+0000 7fe274e2b640 1 --2- 172.21.15.16:0/2795649402 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56488dea3d90 0x56488deb0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.732+0000 7fe274e2b640 1 -- 172.21.15.16:0/2795649402 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56488df21c90 con 0x56488dea3470 2024-03-28T22:13:39.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.732+0000 7fe27562c640 1 --2- 172.21.15.16:0/2795649402 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56488deac970 0x56488deb0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:13:39.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.732+0000 7fe274e2b640 1 --2- 172.21.15.16:0/2795649402 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56488dea3470 0x56488deb03f0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fe26409a580 tx=0x7fe26409a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a6ec0255057181b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:39.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.732+0000 7fe24ffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:13:39.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.732+0000 7fe24ffff640 1 -- 172.21.15.16:0/2795649402 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2640b2020 con 0x56488dea3470 2024-03-28T22:13:39.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.732+0000 7fe24ffff640 1 -- 172.21.15.16:0/2795649402 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fe264009cc0 con 0x56488dea3470 2024-03-28T22:13:39.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.732+0000 7fe279c76740 1 -- 172.21.15.16:0/2795649402 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe25c02ce10 con 0x56488dea3470 2024-03-28T22:13:39.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.732+0000 7fe24ffff640 1 -- 172.21.15.16:0/2795649402 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe264020040 con 0x56488dea3470 2024-03-28T22:13:39.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.732+0000 7fe279c76740 10 client.16170 fetch_fsmap learned FSMap version 24 2024-03-28T22:13:39.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.732+0000 7fe279c76740 1 -- 172.21.15.16:0/2795649402 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56488df22ee0 con 0x56488dea3470 2024-03-28T22:13:39.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.732+0000 7fe24ffff640 1 -- 172.21.15.16:0/2795649402 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe25c02ce10 con 0x56488dea3470 2024-03-28T22:13:39.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.732+0000 7fe24ffff640 1 -- 172.21.15.16:0/2795649402 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7fe264096c60 con 0x56488dea3470 2024-03-28T22:13:39.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.733+0000 7fe279c76740 10 client.16170 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:13:39.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.733+0000 7fe279c76740 10 client.16170 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:13:39.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.733+0000 7fe279c76740 20 client.16170 populate_metadata read hostname 'smithi016' 2024-03-28T22:13:39.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.733+0000 7fe279c76740 1 --2- 172.21.15.16:0/2795649402 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x56488df232e0 0x56488df256d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:39.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.733+0000 7fe279c76740 4 client.16170 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:13:39.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.733+0000 7fe279c76740 1 -- 172.21.15.16:0/2795649402 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56488de0ea40 con 0x56488df232e0 2024-03-28T22:13:39.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.733+0000 7fe26ffff640 1 --2- 172.21.15.16:0/2795649402 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x56488df232e0 0x56488df256d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:39.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.733+0000 7fe24ffff640 10 client.16170 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:13:39.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.733+0000 7fe26ffff640 1 --2- 172.21.15.16:0/2795649402 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x56488df232e0 0x56488df256d0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc2524f324a8bb31 server_cookie=a43ec145170ce29c in_seq=0 out_seq=0 2024-03-28T22:13:39.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.733+0000 7fe24ffff640 1 -- 172.21.15.16:0/2795649402 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56488de0ea40 con 0x56488df232e0 2024-03-28T22:13:39.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.733+0000 7fe24ffff640 10 client.16170 handle_command_reply: tid=0 2024-03-28T22:13:39.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.733+0000 7fe279c76740 1 client.16170 shutdown 2024-03-28T22:13:39.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.733+0000 7fe279c76740 1 -- 172.21.15.16:0/2795649402 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56488dea3470 msgr2=0x56488deb03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:39.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.733+0000 7fe279c76740 1 --2- 172.21.15.16:0/2795649402 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56488dea3470 0x56488deb03f0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fe26409a580 tx=0x7fe26409a080 comp rx=0 tx=0).stop 2024-03-28T22:13:39.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.733+0000 7fe279c76740 1 -- 172.21.15.16:0/2795649402 shutdown_connections 2024-03-28T22:13:39.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.733+0000 7fe279c76740 1 --2- 172.21.15.16:0/2795649402 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x56488df232e0 0x56488df256d0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.733+0000 7fe279c76740 1 --2- 172.21.15.16:0/2795649402 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56488deac970 0x56488deb0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.733+0000 7fe24ffff640 0 client.16170 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:13:39.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.733+0000 7fe279c76740 1 --2- 172.21.15.16:0/2795649402 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56488dea3d90 0x56488deb0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.733+0000 7fe279c76740 1 --2- 172.21.15.16:0/2795649402 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56488dea3470 0x56488deb03f0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.733+0000 7fe279c76740 1 -- 172.21.15.16:0/2795649402 >> 172.21.15.16:0/2795649402 conn(0x56488de9dba0 msgr2=0x56488dea99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:39.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.734+0000 7fe279c76740 1 -- 172.21.15.16:0/2795649402 shutdown_connections 2024-03-28T22:13:39.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.734+0000 7fe279c76740 1 -- 172.21.15.16:0/2795649402 wait complete. 2024-03-28T22:13:39.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.734+0000 7fe279c76740 20 client.16170 trim_cache size 0 max 16384 2024-03-28T22:13:39.749 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:13:39.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.734+0000 7fe27708f640 1 -- 172.21.15.16:0/1323580760 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fe2600a1ef0 msgr2=0x7fe2600a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:39.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.734+0000 7fe27708f640 1 --2- 172.21.15.16:0/1323580760 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fe2600a1ef0 0x7fe2600a43b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe268005fd0 tx=0x7fe268011690 comp rx=0 tx=0).stop 2024-03-28T22:13:39.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.734+0000 7fe27708f640 1 -- 172.21.15.16:0/1323580760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2701009d0 msgr2=0x7fe270198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:39.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.734+0000 7fe27708f640 1 --2- 172.21.15.16:0/1323580760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2701009d0 0x7fe270198080 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fe26400d770 tx=0x7fe26400dc30 comp rx=0 tx=0).stop 2024-03-28T22:13:39.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.734+0000 7fe27708f640 1 -- 172.21.15.16:0/1323580760 shutdown_connections 2024-03-28T22:13:39.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.734+0000 7fe27708f640 1 --2- 172.21.15.16:0/1323580760 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fe2600a1ef0 0x7fe2600a43b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.734+0000 7fe27708f640 1 --2- 172.21.15.16:0/1323580760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe270102b80 0x7fe27019c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.734+0000 7fe27708f640 1 --2- 172.21.15.16:0/1323580760 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2701021e0 0x7fe2701985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.734+0000 7fe27708f640 1 --2- 172.21.15.16:0/1323580760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2701009d0 0x7fe270198080 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:39.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.734+0000 7fe27708f640 1 -- 172.21.15.16:0/1323580760 >> 172.21.15.16:0/1323580760 conn(0x7fe270077830 msgr2=0x7fe27006a0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:39.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.734+0000 7fe27708f640 1 -- 172.21.15.16:0/1323580760 shutdown_connections 2024-03-28T22:13:39.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:39.734+0000 7fe27708f640 1 -- 172.21.15.16:0/1323580760 wait complete. 2024-03-28T22:13:40.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:40 smithi053 ceph-mon[35979]: pgmap v95: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.9 KiB/s wr, 5 op/s 2024-03-28T22:13:40.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:40 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2379091685' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:13:40.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:40 smithi063 ceph-mon[36249]: pgmap v95: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.9 KiB/s wr, 5 op/s 2024-03-28T22:13:40.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:40 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2379091685' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:13:40.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:40 smithi016 ceph-mon[32374]: pgmap v95: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.9 KiB/s wr, 5 op/s 2024-03-28T22:13:40.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:40 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2379091685' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:13:40.753 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:13:40.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:13:40.754 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:13:40.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.871+0000 7fafd01f4640 1 -- 172.21.15.16:0/2203995126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafc8103b90 msgr2=0x7fafc810e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:40.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.871+0000 7fafd01f4640 1 --2- 172.21.15.16:0/2203995126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafc8103b90 0x7fafc810e6c0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fafc0009ca0 tx=0x7fafc002c1d0 comp rx=0 tx=0).stop 2024-03-28T22:13:40.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.871+0000 7fafd01f4640 1 -- 172.21.15.16:0/2203995126 shutdown_connections 2024-03-28T22:13:40.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.871+0000 7fafd01f4640 1 --2- 172.21.15.16:0/2203995126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafc8103b90 0x7fafc810e6c0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:40.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.871+0000 7fafd01f4640 1 --2- 172.21.15.16:0/2203995126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fafc81031f0 0x7fafc8103650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:40.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.871+0000 7fafd01f4640 1 --2- 172.21.15.16:0/2203995126 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafc8101ab0 0x7fafc8101e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:40.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.871+0000 7fafd01f4640 1 -- 172.21.15.16:0/2203995126 >> 172.21.15.16:0/2203995126 conn(0x7fafc80fd1e0 msgr2=0x7fafc80ff600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:40.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.871+0000 7fafd01f4640 1 -- 172.21.15.16:0/2203995126 shutdown_connections 2024-03-28T22:13:40.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.871+0000 7fafd01f4640 1 -- 172.21.15.16:0/2203995126 wait complete. 2024-03-28T22:13:40.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.872+0000 7fafd01f4640 1 Processor -- start 2024-03-28T22:13:40.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.872+0000 7fafd01f4640 1 -- start start 2024-03-28T22:13:40.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.872+0000 7fafd01f4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafc8101ab0 0x7fafc8073a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:40.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.872+0000 7fafd01f4640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafc81031f0 0x7fafc8073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:40.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.872+0000 7fafd01f4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fafc8103b90 0x7fafc80744d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:40.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.872+0000 7fafd01f4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fafc8111c00 con 0x7fafc8101ab0 2024-03-28T22:13:40.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.872+0000 7fafd01f4640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fafc8111a80 con 0x7fafc81031f0 2024-03-28T22:13:40.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.872+0000 7fafd01f4640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fafc8111d80 con 0x7fafc8103b90 2024-03-28T22:13:40.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.872+0000 7fafcdf90640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafc8101ab0 0x7fafc8073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:40.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.872+0000 7fafcdf90640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafc8101ab0 0x7fafc8073a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38872/0 (socket says 172.21.15.16:38872) 2024-03-28T22:13:40.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.872+0000 7fafcdf90640 1 -- 172.21.15.16:0/2213404722 learned_addr learned my addr 172.21.15.16:0/2213404722 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:40.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.872+0000 7fafcd78f640 1 --2- 172.21.15.16:0/2213404722 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafc81031f0 0x7fafc8073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:40.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.872+0000 7fafce791640 1 --2- 172.21.15.16:0/2213404722 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fafc8103b90 0x7fafc80744d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:40.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.872+0000 7fafcdf90640 1 -- 172.21.15.16:0/2213404722 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fafc8103b90 msgr2=0x7fafc80744d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:40.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.872+0000 7fafcdf90640 1 --2- 172.21.15.16:0/2213404722 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fafc8103b90 0x7fafc80744d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:40.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.872+0000 7fafcdf90640 1 -- 172.21.15.16:0/2213404722 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafc81031f0 msgr2=0x7fafc8073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:40.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.872+0000 7fafcdf90640 1 --2- 172.21.15.16:0/2213404722 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafc81031f0 0x7fafc8073f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:40.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.872+0000 7fafcdf90640 1 -- 172.21.15.16:0/2213404722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafc81a64e0 con 0x7fafc8101ab0 2024-03-28T22:13:40.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.872+0000 7fafcd78f640 1 --2- 172.21.15.16:0/2213404722 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafc81031f0 0x7fafc8073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:13:40.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.873+0000 7fafcdf90640 1 --2- 172.21.15.16:0/2213404722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafc8101ab0 0x7fafc8073a50 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fafc400e9f0 tx=0x7fafc400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:40.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.873+0000 7fafbeffd640 1 -- 172.21.15.16:0/2213404722 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc400ccc0 con 0x7fafc8101ab0 2024-03-28T22:13:40.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.873+0000 7fafd01f4640 1 -- 172.21.15.16:0/2213404722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafc81a67d0 con 0x7fafc8101ab0 2024-03-28T22:13:40.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.873+0000 7fafbeffd640 1 -- 172.21.15.16:0/2213404722 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafc4010910 con 0x7fafc8101ab0 2024-03-28T22:13:40.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.873+0000 7fafd01f4640 1 -- 172.21.15.16:0/2213404722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafc81a6d70 con 0x7fafc8101ab0 2024-03-28T22:13:40.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.873+0000 7fafbeffd640 1 -- 172.21.15.16:0/2213404722 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc4018840 con 0x7fafc8101ab0 2024-03-28T22:13:40.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.875+0000 7fafbeffd640 1 -- 172.21.15.16:0/2213404722 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7fafc4010ac0 con 0x7fafc8101ab0 2024-03-28T22:13:40.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.875+0000 7fafd2dd8740 1 Processor -- start 2024-03-28T22:13:40.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.876+0000 7fafd2dd8740 1 -- start start 2024-03-28T22:13:40.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.876+0000 7fafbeffd640 1 --2- 172.21.15.16:0/2213404722 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fafb40a1ef0 0x7fafb40a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:40.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.876+0000 7fafd2dd8740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e8552bf9f0 0x55e8552bfdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:40.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.876+0000 7fafd2dd8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8552c0310 0x55e8552c89b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:40.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.876+0000 7fafd2dd8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e8552c8ef0 0x55e8552cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:40.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.876+0000 7fafd2dd8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e8551c00b0 con 0x55e8552c8ef0 2024-03-28T22:13:40.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.876+0000 7fafd2dd8740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55e855222ea0 con 0x55e8552bf9f0 2024-03-28T22:13:40.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.876+0000 7fafd2dd8740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e8551cbc00 con 0x55e8552c0310 2024-03-28T22:13:40.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.876+0000 7fafbeffd640 1 -- 172.21.15.16:0/2213404722 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fafc40ce570 con 0x7fafc8101ab0 2024-03-28T22:13:40.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.876+0000 7fafce791640 1 --2- 172.21.15.16:0/2213404722 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fafb40a1ef0 0x7fafb40a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:40.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.876+0000 7fafcdf90640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e8552bf9f0 0x55e8552bfdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:40.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.876+0000 7fafcdf90640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e8552bf9f0 0x55e8552bfdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:51688/0 (socket says 172.21.15.16:51688) 2024-03-28T22:13:40.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.876+0000 7fafcdf90640 1 -- 172.21.15.16:0/790012146 learned_addr learned my addr 172.21.15.16:0/790012146 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:40.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.876+0000 7fafce791640 1 --2- 172.21.15.16:0/790012146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e8552c8ef0 0x55e8552cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:40.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.876+0000 7fafcd78f640 1 --2- 172.21.15.16:0/790012146 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8552c0310 0x55e8552c89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:40.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.876+0000 7fafcdf90640 1 -- 172.21.15.16:0/790012146 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8552c0310 msgr2=0x55e8552c89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:40.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.876+0000 7fafcdf90640 1 --2- 172.21.15.16:0/790012146 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8552c0310 0x55e8552c89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:40.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.876+0000 7fafcdf90640 1 -- 172.21.15.16:0/790012146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e8552c8ef0 msgr2=0x55e8552cb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:40.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.876+0000 7fafcdf90640 1 --2- 172.21.15.16:0/790012146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e8552c8ef0 0x55e8552cb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:40.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.876+0000 7fafcdf90640 1 -- 172.21.15.16:0/790012146 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e854fff7e0 con 0x55e8552bf9f0 2024-03-28T22:13:40.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.877+0000 7fafce791640 1 --2- 172.21.15.16:0/2213404722 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fafb40a1ef0 0x7fafb40a43b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fafc0004af0 tx=0x7fafc0032470 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:40.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.877+0000 7fafcdf90640 1 --2- 172.21.15.16:0/790012146 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e8552bf9f0 0x55e8552bfdd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fafc40a31f0 tx=0x7fafc40560c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f0000e8f679ecfd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:40.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.877+0000 7fafb3fff640 1 -- 172.21.15.16:0/790012146 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc409f0d0 con 0x55e8552bf9f0 2024-03-28T22:13:40.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.877+0000 7fafb3fff640 1 -- 172.21.15.16:0/790012146 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafc40a3720 con 0x55e8552bf9f0 2024-03-28T22:13:40.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.877+0000 7fafb3fff640 1 -- 172.21.15.16:0/790012146 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc40ce110 con 0x55e8552bf9f0 2024-03-28T22:13:40.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.877+0000 7fafd2dd8740 1 -- 172.21.15.16:0/790012146 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e8552bf9f0 msgr2=0x55e8552bfdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:40.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.877+0000 7fafd2dd8740 1 --2- 172.21.15.16:0/790012146 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e8552bf9f0 0x55e8552bfdd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fafc40a31f0 tx=0x7fafc40560c0 comp rx=0 tx=0).stop 2024-03-28T22:13:40.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.877+0000 7fafd2dd8740 1 -- 172.21.15.16:0/790012146 shutdown_connections 2024-03-28T22:13:40.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.877+0000 7fafd2dd8740 1 --2- 172.21.15.16:0/790012146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e8552c8ef0 0x55e8552cb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:40.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.877+0000 7fafd2dd8740 1 --2- 172.21.15.16:0/790012146 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8552c0310 0x55e8552c89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:40.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.877+0000 7fafd2dd8740 1 --2- 172.21.15.16:0/790012146 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e8552bf9f0 0x55e8552bfdd0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:40.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.877+0000 7fafd2dd8740 1 -- 172.21.15.16:0/790012146 >> 172.21.15.16:0/790012146 conn(0x55e8552ba120 msgr2=0x55e8552ba500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:40.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.878+0000 7fafd2dd8740 1 -- 172.21.15.16:0/790012146 shutdown_connections 2024-03-28T22:13:40.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.878+0000 7fafd2dd8740 1 -- 172.21.15.16:0/790012146 wait complete. 2024-03-28T22:13:40.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.878+0000 7fafd2dd8740 1 -- start start 2024-03-28T22:13:40.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.878+0000 7fafd2dd8740 1 Processor -- start 2024-03-28T22:13:40.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.878+0000 7fafd2dd8740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e8552bf9f0 0x55e8552c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:40.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.878+0000 7fafd2dd8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e8552c0310 0x55e8552c64b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:40.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.878+0000 7fafd2dd8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8552c8ef0 0x55e8552c69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:40.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.878+0000 7fafd2dd8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e8551c13d0 con 0x55e8552c0310 2024-03-28T22:13:40.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.878+0000 7fafd2dd8740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55e85521ef40 con 0x55e8552bf9f0 2024-03-28T22:13:40.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.878+0000 7fafd2dd8740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e8552cbce0 con 0x55e8552c8ef0 2024-03-28T22:13:40.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.878+0000 7fafd2dd8740 10 client.0 _ll_register_callbacks cb 0x55e8551deb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:13:40.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.878+0000 7fafcd78f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e8552c0310 0x55e8552c64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:40.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.879+0000 7fafcd78f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e8552c0310 0x55e8552c64b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38888/0 (socket says 172.21.15.16:38888) 2024-03-28T22:13:40.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.879+0000 7fafcd78f640 1 -- 172.21.15.16:0/630180039 learned_addr learned my addr 172.21.15.16:0/630180039 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:40.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.879+0000 7fafcdf90640 1 --2- 172.21.15.16:0/630180039 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e8552bf9f0 0x55e8552c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:40.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.879+0000 7fafce791640 1 --2- 172.21.15.16:0/630180039 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8552c8ef0 0x55e8552c69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:40.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.879+0000 7fafcd78f640 1 -- 172.21.15.16:0/630180039 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8552c8ef0 msgr2=0x55e8552c69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:40.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.879+0000 7fafcd78f640 1 --2- 172.21.15.16:0/630180039 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8552c8ef0 0x55e8552c69f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:40.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.879+0000 7fafcd78f640 1 -- 172.21.15.16:0/630180039 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e8552bf9f0 msgr2=0x55e8552c5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:40.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.879+0000 7fafcd78f640 1 --2- 172.21.15.16:0/630180039 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e8552bf9f0 0x55e8552c5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:40.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.879+0000 7fafcd78f640 1 -- 172.21.15.16:0/630180039 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e85505ff70 con 0x55e8552c0310 2024-03-28T22:13:40.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.879+0000 7fafcdf90640 1 --2- 172.21.15.16:0/630180039 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e8552bf9f0 0x55e8552c5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:13:40.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.879+0000 7fafcd78f640 1 --2- 172.21.15.16:0/630180039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e8552c0310 0x55e8552c64b0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fafb8010f10 tx=0x7fafb800f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12bee06c18047819 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:40.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.879+0000 7fafb17fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:13:40.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.879+0000 7fafb17fa640 1 -- 172.21.15.16:0/630180039 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb8005020 con 0x55e8552c0310 2024-03-28T22:13:40.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.879+0000 7fafb17fa640 1 -- 172.21.15.16:0/630180039 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fafb80051c0 con 0x55e8552c0310 2024-03-28T22:13:40.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.879+0000 7fafd2dd8740 1 -- 172.21.15.16:0/630180039 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafc40a3720 con 0x55e8552c0310 2024-03-28T22:13:40.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.879+0000 7fafb17fa640 1 -- 172.21.15.16:0/630180039 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fafb80053c0 con 0x55e8552c0310 2024-03-28T22:13:40.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.879+0000 7fafd2dd8740 10 client.16182 fetch_fsmap learned FSMap version 24 2024-03-28T22:13:40.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.879+0000 7fafd2dd8740 1 -- 172.21.15.16:0/630180039 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e854ef7170 con 0x55e8552c0310 2024-03-28T22:13:40.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.880+0000 7fafb17fa640 1 -- 172.21.15.16:0/630180039 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafc40a3720 con 0x55e8552c0310 2024-03-28T22:13:40.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.880+0000 7fafb17fa640 1 -- 172.21.15.16:0/630180039 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7fafb8020020 con 0x55e8552c0310 2024-03-28T22:13:40.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.880+0000 7fafd2dd8740 10 client.16182 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:13:40.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.880+0000 7fafd2dd8740 10 client.16182 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:13:40.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.880+0000 7fafd2dd8740 20 client.16182 populate_metadata read hostname 'smithi016' 2024-03-28T22:13:40.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.880+0000 7fafd2dd8740 1 --2- 172.21.15.16:0/630180039 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e8552d8eb0 0x55e8552db320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:40.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.880+0000 7fafd2dd8740 4 client.16182 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:13:40.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.880+0000 7fafd2dd8740 1 -- 172.21.15.16:0/630180039 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e8552303e0 con 0x55e8552d8eb0 2024-03-28T22:13:40.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.880+0000 7fafcdf90640 1 --2- 172.21.15.16:0/630180039 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e8552d8eb0 0x55e8552db320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:40.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.880+0000 7fafcdf90640 1 --2- 172.21.15.16:0/630180039 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e8552d8eb0 0x55e8552db320 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f25a1b902a43524 server_cookie=6a58a35899373669 in_seq=0 out_seq=0 2024-03-28T22:13:40.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.880+0000 7fafb17fa640 10 client.16182 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:13:40.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.881+0000 7fafb17fa640 1 -- 172.21.15.16:0/630180039 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e8552303e0 con 0x55e8552d8eb0 2024-03-28T22:13:40.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.881+0000 7fafb17fa640 10 client.16182 handle_command_reply: tid=0 2024-03-28T22:13:40.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.881+0000 7fafd2dd8740 1 client.16182 shutdown 2024-03-28T22:13:40.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.881+0000 7fafd2dd8740 1 -- 172.21.15.16:0/630180039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e8552c0310 msgr2=0x55e8552c64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:40.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.881+0000 7fafd2dd8740 1 --2- 172.21.15.16:0/630180039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e8552c0310 0x55e8552c64b0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fafb8010f10 tx=0x7fafb800f7f0 comp rx=0 tx=0).stop 2024-03-28T22:13:40.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.882+0000 7fafd2dd8740 1 -- 172.21.15.16:0/630180039 shutdown_connections 2024-03-28T22:13:40.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.882+0000 7fafd2dd8740 1 --2- 172.21.15.16:0/630180039 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e8552d8eb0 0x55e8552db320 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:40.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.882+0000 7fafd2dd8740 1 --2- 172.21.15.16:0/630180039 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8552c8ef0 0x55e8552c69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:40.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.882+0000 7fafd2dd8740 1 --2- 172.21.15.16:0/630180039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e8552c0310 0x55e8552c64b0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:40.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.882+0000 7fafd2dd8740 1 --2- 172.21.15.16:0/630180039 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e8552bf9f0 0x55e8552c5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:40.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.882+0000 7fafb17fa640 0 client.16182 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:13:40.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.882+0000 7fafd2dd8740 1 -- 172.21.15.16:0/630180039 >> 172.21.15.16:0/630180039 conn(0x55e8552ba120 msgr2=0x55e8552bbe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:40.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.882+0000 7fafd2dd8740 1 -- 172.21.15.16:0/630180039 shutdown_connections 2024-03-28T22:13:40.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.882+0000 7fafd2dd8740 1 -- 172.21.15.16:0/630180039 wait complete. 2024-03-28T22:13:40.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.882+0000 7fafd2dd8740 20 client.16182 trim_cache size 0 max 16384 2024-03-28T22:13:40.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.899+0000 7fafd2dd8740 1 Processor -- start 2024-03-28T22:13:40.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.899+0000 7fafd2dd8740 1 -- start start 2024-03-28T22:13:40.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.899+0000 7fafd2dd8740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e8552bf9f0 0x55e8552c86a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:40.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.899+0000 7fafd2dd8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e8552c0310 0x55e8552d4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:40.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.899+0000 7fafd2dd8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8552c8ef0 0x55e8552e0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:40.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.899+0000 7fafd2dd8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fafc81b1f40 con 0x55e8552c0310 2024-03-28T22:13:40.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.899+0000 7fafd2dd8740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fafc81af7e0 con 0x55e8552bf9f0 2024-03-28T22:13:40.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.899+0000 7fafd2dd8740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fafc81ad630 con 0x55e8552c8ef0 2024-03-28T22:13:40.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.900+0000 7fafcd78f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e8552c0310 0x55e8552d4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:40.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.900+0000 7fafcd78f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e8552c0310 0x55e8552d4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38902/0 (socket says 172.21.15.16:38902) 2024-03-28T22:13:40.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.900+0000 7fafcd78f640 1 -- 172.21.15.16:0/3932040627 learned_addr learned my addr 172.21.15.16:0/3932040627 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:40.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.900+0000 7fafcdf90640 1 --2- 172.21.15.16:0/3932040627 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e8552bf9f0 0x55e8552c86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:40.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.900+0000 7fafcd78f640 1 -- 172.21.15.16:0/3932040627 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8552c8ef0 msgr2=0x55e8552e0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:40.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.900+0000 7fafcd78f640 1 --2- 172.21.15.16:0/3932040627 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8552c8ef0 0x55e8552e0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:40.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.900+0000 7fafcd78f640 1 -- 172.21.15.16:0/3932040627 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e8552bf9f0 msgr2=0x55e8552c86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:40.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.900+0000 7fafcd78f640 1 --2- 172.21.15.16:0/3932040627 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e8552bf9f0 0x55e8552c86a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:40.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.900+0000 7fafcd78f640 1 -- 172.21.15.16:0/3932040627 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8552beae0 con 0x55e8552c0310 2024-03-28T22:13:40.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.900+0000 7fafcd78f640 1 --2- 172.21.15.16:0/3932040627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e8552c0310 0x55e8552d4900 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fafb80198a0 tx=0x7fafb800aa90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14e17bd7eb8dab82 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:40.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.900+0000 7fafb37fe640 1 -- 172.21.15.16:0/3932040627 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb8019410 con 0x55e8552c0310 2024-03-28T22:13:40.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.900+0000 7fafb37fe640 1 -- 172.21.15.16:0/3932040627 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafb8019920 con 0x55e8552c0310 2024-03-28T22:13:40.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.900+0000 7fafb37fe640 1 -- 172.21.15.16:0/3932040627 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb8018370 con 0x55e8552c0310 2024-03-28T22:13:40.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.901+0000 7fafd2dd8740 1 -- 172.21.15.16:0/3932040627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e8552c0310 msgr2=0x55e8552d4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:40.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.901+0000 7fafd2dd8740 1 --2- 172.21.15.16:0/3932040627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e8552c0310 0x55e8552d4900 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fafb80198a0 tx=0x7fafb800aa90 comp rx=0 tx=0).stop 2024-03-28T22:13:40.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.901+0000 7fafd2dd8740 1 -- 172.21.15.16:0/3932040627 shutdown_connections 2024-03-28T22:13:40.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.901+0000 7fafd2dd8740 1 --2- 172.21.15.16:0/3932040627 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8552c8ef0 0x55e8552e0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:40.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.901+0000 7fafd2dd8740 1 --2- 172.21.15.16:0/3932040627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e8552c0310 0x55e8552d4900 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:40.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.901+0000 7fafd2dd8740 1 --2- 172.21.15.16:0/3932040627 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e8552bf9f0 0x55e8552c86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:40.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.901+0000 7fafd2dd8740 1 -- 172.21.15.16:0/3932040627 >> 172.21.15.16:0/3932040627 conn(0x55e8552ba120 msgr2=0x55e8552c67c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:40.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.901+0000 7fafd2dd8740 1 -- 172.21.15.16:0/3932040627 shutdown_connections 2024-03-28T22:13:40.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.901+0000 7fafd2dd8740 1 -- 172.21.15.16:0/3932040627 wait complete. 2024-03-28T22:13:40.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.901+0000 7fafd2dd8740 1 -- start start 2024-03-28T22:13:40.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.901+0000 7fafd2dd8740 1 Processor -- start 2024-03-28T22:13:40.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.901+0000 7fafd2dd8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8552bf9f0 0x55e8552d0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:40.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.901+0000 7fafd2dd8740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e8552c0310 0x55e8552cd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:40.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.901+0000 7fafd2dd8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e8552c8ef0 0x55e8552cd810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:40.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.901+0000 7fafd2dd8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fafc81adca0 con 0x55e8552c8ef0 2024-03-28T22:13:40.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.901+0000 7fafd2dd8740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fafc81af110 con 0x55e8552c0310 2024-03-28T22:13:40.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.901+0000 7fafd2dd8740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fafc81ae370 con 0x55e8552bf9f0 2024-03-28T22:13:40.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.902+0000 7fafd2dd8740 10 client.0 _ll_register_callbacks cb 0x7fafb8010b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:13:40.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.902+0000 7fafce791640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e8552c8ef0 0x55e8552cd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:40.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.902+0000 7fafce791640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e8552c8ef0 0x55e8552cd810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38914/0 (socket says 172.21.15.16:38914) 2024-03-28T22:13:40.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.902+0000 7fafce791640 1 -- 172.21.15.16:0/1104269666 learned_addr learned my addr 172.21.15.16:0/1104269666 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:40.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.902+0000 7fafcd78f640 1 --2- 172.21.15.16:0/1104269666 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e8552c0310 0x55e8552cd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:40.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.902+0000 7fafcdf90640 1 --2- 172.21.15.16:0/1104269666 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8552bf9f0 0x55e8552d0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:40.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.902+0000 7fafce791640 1 -- 172.21.15.16:0/1104269666 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8552bf9f0 msgr2=0x55e8552d0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:40.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.902+0000 7fafce791640 1 --2- 172.21.15.16:0/1104269666 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8552bf9f0 0x55e8552d0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:40.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.902+0000 7fafce791640 1 -- 172.21.15.16:0/1104269666 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e8552c0310 msgr2=0x55e8552cd2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:40.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.902+0000 7fafce791640 1 --2- 172.21.15.16:0/1104269666 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e8552c0310 0x55e8552cd2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:40.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.902+0000 7fafce791640 1 -- 172.21.15.16:0/1104269666 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8552cdd80 con 0x55e8552c8ef0 2024-03-28T22:13:40.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.902+0000 7fafce791640 1 --2- 172.21.15.16:0/1104269666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e8552c8ef0 0x55e8552cd810 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fafc0009890 tx=0x7fafc00095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5113514e523205f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:40.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.902+0000 7fafb17fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:13:40.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.902+0000 7fafb17fa640 1 -- 172.21.15.16:0/1104269666 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc0032800 con 0x55e8552c8ef0 2024-03-28T22:13:40.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.902+0000 7fafb17fa640 1 -- 172.21.15.16:0/1104269666 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fafc00329a0 con 0x55e8552c8ef0 2024-03-28T22:13:40.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.902+0000 7fafd2dd8740 1 -- 172.21.15.16:0/1104269666 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafb8019920 con 0x55e8552c8ef0 2024-03-28T22:13:40.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.902+0000 7fafb17fa640 1 -- 172.21.15.16:0/1104269666 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fafc00098e0 con 0x55e8552c8ef0 2024-03-28T22:13:40.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.902+0000 7fafd2dd8740 10 client.16188 fetch_fsmap learned FSMap version 24 2024-03-28T22:13:40.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.902+0000 7fafd2dd8740 1 -- 172.21.15.16:0/1104269666 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e855340030 con 0x55e8552c8ef0 2024-03-28T22:13:40.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.903+0000 7fafb17fa640 1 -- 172.21.15.16:0/1104269666 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafb8019920 con 0x55e8552c8ef0 2024-03-28T22:13:40.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.903+0000 7fafb17fa640 1 -- 172.21.15.16:0/1104269666 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7fafc003f040 con 0x55e8552c8ef0 2024-03-28T22:13:40.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.903+0000 7fafd2dd8740 10 client.16188 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:13:40.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.903+0000 7fafd2dd8740 10 client.16188 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:13:40.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.903+0000 7fafd2dd8740 20 client.16188 populate_metadata read hostname 'smithi016' 2024-03-28T22:13:40.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.903+0000 7fafd2dd8740 1 --2- 172.21.15.16:0/1104269666 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e855340860 0x55e855342c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:40.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.903+0000 7fafd2dd8740 4 client.16188 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:13:40.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.903+0000 7fafd2dd8740 1 -- 172.21.15.16:0/1104269666 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55e85522a730 con 0x55e855340860 2024-03-28T22:13:40.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.903+0000 7fafcdf90640 1 --2- 172.21.15.16:0/1104269666 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e855340860 0x55e855342c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:40.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.903+0000 7fafb17fa640 10 client.16188 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:13:40.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.903+0000 7fafcdf90640 1 --2- 172.21.15.16:0/1104269666 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e855340860 0x55e855342c80 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=104e7c3ee0447997 server_cookie=c6dc665dc41acbff in_seq=0 out_seq=0 2024-03-28T22:13:40.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.905+0000 7fafb17fa640 1 -- 172.21.15.16:0/1104269666 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55e85522a730 con 0x55e855340860 2024-03-28T22:13:40.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.905+0000 7fafb17fa640 10 client.16188 handle_command_reply: tid=0 2024-03-28T22:13:40.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.905+0000 7fafd2dd8740 1 client.16188 shutdown 2024-03-28T22:13:40.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.905+0000 7fafd2dd8740 1 -- 172.21.15.16:0/1104269666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e8552c8ef0 msgr2=0x55e8552cd810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:40.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.905+0000 7fafd2dd8740 1 --2- 172.21.15.16:0/1104269666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e8552c8ef0 0x55e8552cd810 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fafc0009890 tx=0x7fafc00095c0 comp rx=0 tx=0).stop 2024-03-28T22:13:40.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.905+0000 7fafd2dd8740 1 -- 172.21.15.16:0/1104269666 shutdown_connections 2024-03-28T22:13:40.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.905+0000 7fafd2dd8740 1 --2- 172.21.15.16:0/1104269666 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e855340860 0x55e855342c80 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:40.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.905+0000 7fafd2dd8740 1 --2- 172.21.15.16:0/1104269666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e8552c8ef0 0x55e8552cd810 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:40.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.905+0000 7fafd2dd8740 1 --2- 172.21.15.16:0/1104269666 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e8552c0310 0x55e8552cd2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:40.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.905+0000 7fafd2dd8740 1 --2- 172.21.15.16:0/1104269666 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8552bf9f0 0x55e8552d0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:40.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.905+0000 7fafb17fa640 0 client.16188 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:13:40.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.905+0000 7fafd2dd8740 1 -- 172.21.15.16:0/1104269666 >> 172.21.15.16:0/1104269666 conn(0x55e8552ba120 msgr2=0x55e8552b88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:40.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.905+0000 7fafd2dd8740 1 -- 172.21.15.16:0/1104269666 shutdown_connections 2024-03-28T22:13:40.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.905+0000 7fafd2dd8740 1 -- 172.21.15.16:0/1104269666 wait complete. 2024-03-28T22:13:40.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.905+0000 7fafd2dd8740 20 client.16188 trim_cache size 0 max 16384 2024-03-28T22:13:40.920 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:13:40.920 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-03-28T22:13:40.920 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "55ca60c4-9da8-40a9-af97-ba2ee7977c3f", 2024-03-28T22:13:40.920 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-03-28T22:13:40.921 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:13:40.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.906+0000 7fafd01f4640 1 -- 172.21.15.16:0/2213404722 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fafb40a1ef0 msgr2=0x7fafb40a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:40.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.906+0000 7fafd01f4640 1 --2- 172.21.15.16:0/2213404722 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fafb40a1ef0 0x7fafb40a43b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fafc0004af0 tx=0x7fafc0032470 comp rx=0 tx=0).stop 2024-03-28T22:13:40.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.906+0000 7fafd01f4640 1 -- 172.21.15.16:0/2213404722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafc8101ab0 msgr2=0x7fafc8073a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:40.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.906+0000 7fafd01f4640 1 --2- 172.21.15.16:0/2213404722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafc8101ab0 0x7fafc8073a50 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fafc400e9f0 tx=0x7fafc400edf0 comp rx=0 tx=0).stop 2024-03-28T22:13:40.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.906+0000 7fafd01f4640 1 -- 172.21.15.16:0/2213404722 shutdown_connections 2024-03-28T22:13:40.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.906+0000 7fafd01f4640 1 --2- 172.21.15.16:0/2213404722 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fafb40a1ef0 0x7fafb40a43b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:40.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.906+0000 7fafd01f4640 1 --2- 172.21.15.16:0/2213404722 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fafc8103b90 0x7fafc80744d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:40.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.906+0000 7fafd01f4640 1 --2- 172.21.15.16:0/2213404722 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafc81031f0 0x7fafc8073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:40.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.906+0000 7fafd01f4640 1 --2- 172.21.15.16:0/2213404722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafc8101ab0 0x7fafc8073a50 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:40.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.906+0000 7fafd01f4640 1 -- 172.21.15.16:0/2213404722 >> 172.21.15.16:0/2213404722 conn(0x7fafc80fd1e0 msgr2=0x7fafc8107590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:40.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.906+0000 7fafd01f4640 1 -- 172.21.15.16:0/2213404722 shutdown_connections 2024-03-28T22:13:40.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:40.906+0000 7fafd01f4640 1 -- 172.21.15.16:0/2213404722 wait complete. 2024-03-28T22:13:40.924 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:13:40.924 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:13:41.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.028+0000 7ffa4fc2e640 1 -- 172.21.15.16:0/2947556313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa48106680 msgr2=0x7ffa48106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:41.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.028+0000 7ffa4fc2e640 1 --2- 172.21.15.16:0/2947556313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa48106680 0x7ffa48106a60 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7ffa44009930 tx=0x7ffa4402bc30 comp rx=0 tx=0).stop 2024-03-28T22:13:41.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.028+0000 7ffa4fc2e640 1 -- 172.21.15.16:0/2947556313 shutdown_connections 2024-03-28T22:13:41.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.028+0000 7ffa4fc2e640 1 --2- 172.21.15.16:0/2947556313 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa48100fa0 0x7ffa4810d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:41.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.028+0000 7ffa4fc2e640 1 --2- 172.21.15.16:0/2947556313 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa481005e0 0x7ffa48100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:41.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.028+0000 7ffa4fc2e640 1 --2- 172.21.15.16:0/2947556313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa48106680 0x7ffa48106a60 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:41.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.028+0000 7ffa4fc2e640 1 -- 172.21.15.16:0/2947556313 >> 172.21.15.16:0/2947556313 conn(0x7ffa480fc0b0 msgr2=0x7ffa480fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:41.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.028+0000 7ffa4fc2e640 1 -- 172.21.15.16:0/2947556313 shutdown_connections 2024-03-28T22:13:41.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.028+0000 7ffa4fc2e640 1 -- 172.21.15.16:0/2947556313 wait complete. 2024-03-28T22:13:41.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.029+0000 7ffa4fc2e640 1 Processor -- start 2024-03-28T22:13:41.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.029+0000 7ffa4fc2e640 1 -- start start 2024-03-28T22:13:41.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.029+0000 7ffa4fc2e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa481005e0 0x7ffa48198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:41.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.029+0000 7ffa4fc2e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa48100fa0 0x7ffa48198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:41.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.029+0000 7ffa4fc2e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa48106680 0x7ffa4819c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:41.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.029+0000 7ffa4fc2e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffa48110a00 con 0x7ffa48100fa0 2024-03-28T22:13:41.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.029+0000 7ffa4fc2e640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ffa48110880 con 0x7ffa481005e0 2024-03-28T22:13:41.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.029+0000 7ffa4fc2e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ffa48110b80 con 0x7ffa48106680 2024-03-28T22:13:41.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.029+0000 7ffa4d1c9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa48100fa0 0x7ffa48198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:41.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.029+0000 7ffa4d1c9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa48100fa0 0x7ffa48198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38936/0 (socket says 172.21.15.16:38936) 2024-03-28T22:13:41.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.029+0000 7ffa4d1c9640 1 -- 172.21.15.16:0/4281856259 learned_addr learned my addr 172.21.15.16:0/4281856259 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:41.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.029+0000 7ffa4d9ca640 1 --2- 172.21.15.16:0/4281856259 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa481005e0 0x7ffa48198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:41.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.029+0000 7ffa4e1cb640 1 --2- 172.21.15.16:0/4281856259 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa48106680 0x7ffa4819c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:41.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.029+0000 7ffa4d1c9640 1 -- 172.21.15.16:0/4281856259 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa48106680 msgr2=0x7ffa4819c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:41.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.029+0000 7ffa4d1c9640 1 --2- 172.21.15.16:0/4281856259 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa48106680 0x7ffa4819c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:41.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.029+0000 7ffa4d1c9640 1 -- 172.21.15.16:0/4281856259 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa481005e0 msgr2=0x7ffa48198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:41.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.029+0000 7ffa4d1c9640 1 --2- 172.21.15.16:0/4281856259 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa481005e0 0x7ffa48198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:41.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.029+0000 7ffa4d1c9640 1 -- 172.21.15.16:0/4281856259 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa4819d040 con 0x7ffa48100fa0 2024-03-28T22:13:41.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.029+0000 7ffa4e1cb640 1 --2- 172.21.15.16:0/4281856259 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa48106680 0x7ffa4819c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:13:41.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.029+0000 7ffa4d1c9640 1 --2- 172.21.15.16:0/4281856259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa48100fa0 0x7ffa48198590 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7ffa3800e9f0 tx=0x7ffa3800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:41.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.030+0000 7ffa3effd640 1 -- 172.21.15.16:0/4281856259 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3800cd80 con 0x7ffa48100fa0 2024-03-28T22:13:41.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.030+0000 7ffa3effd640 1 -- 172.21.15.16:0/4281856259 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa38010910 con 0x7ffa48100fa0 2024-03-28T22:13:41.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.030+0000 7ffa4fc2e640 1 -- 172.21.15.16:0/4281856259 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa4819d330 con 0x7ffa48100fa0 2024-03-28T22:13:41.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.030+0000 7ffa3effd640 1 -- 172.21.15.16:0/4281856259 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa38018990 con 0x7ffa48100fa0 2024-03-28T22:13:41.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.030+0000 7ffa4fc2e640 1 -- 172.21.15.16:0/4281856259 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa481a4bc0 con 0x7ffa48100fa0 2024-03-28T22:13:41.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.032+0000 7ffa3effd640 1 -- 172.21.15.16:0/4281856259 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7ffa38010ac0 con 0x7ffa48100fa0 2024-03-28T22:13:41.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.032+0000 7ffa52814740 1 Processor -- start 2024-03-28T22:13:41.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.033+0000 7ffa52814740 1 -- start start 2024-03-28T22:13:41.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.033+0000 7ffa3effd640 1 --2- 172.21.15.16:0/4281856259 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ffa340a1ef0 0x7ffa340a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:41.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.033+0000 7ffa52814740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f882372290 0x55f882372670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:41.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.033+0000 7ffa52814740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f882372bb0 0x55f88237b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:41.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.033+0000 7ffa52814740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f88237b790 0x55f88237db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:41.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.033+0000 7ffa52814740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f8822d6180 con 0x55f88237b790 2024-03-28T22:13:41.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.033+0000 7ffa52814740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f88227eb70 con 0x55f882372290 2024-03-28T22:13:41.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.033+0000 7ffa52814740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f8822d2220 con 0x55f882372bb0 2024-03-28T22:13:41.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.033+0000 7ffa3effd640 1 -- 172.21.15.16:0/4281856259 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7ffa380ce520 con 0x7ffa48100fa0 2024-03-28T22:13:41.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.033+0000 7ffa4e1cb640 1 --2- 172.21.15.16:0/4281856259 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ffa340a1ef0 0x7ffa340a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:41.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.033+0000 7ffa4d9ca640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f882372290 0x55f882372670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:41.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.033+0000 7ffa4d9ca640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f882372290 0x55f882372670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:51730/0 (socket says 172.21.15.16:51730) 2024-03-28T22:13:41.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.033+0000 7ffa4d9ca640 1 -- 172.21.15.16:0/1863671348 learned_addr learned my addr 172.21.15.16:0/1863671348 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:41.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.033+0000 7ffa4e1cb640 1 --2- 172.21.15.16:0/1863671348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f88237b790 0x55f88237db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:41.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.033+0000 7ffa4d1c9640 1 --2- 172.21.15.16:0/1863671348 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f882372bb0 0x55f88237b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:41.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.033+0000 7ffa4d9ca640 1 -- 172.21.15.16:0/1863671348 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f882372bb0 msgr2=0x55f88237b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:41.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.033+0000 7ffa4d9ca640 1 --2- 172.21.15.16:0/1863671348 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f882372bb0 0x55f88237b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:41.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.033+0000 7ffa4d9ca640 1 -- 172.21.15.16:0/1863671348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f88237b790 msgr2=0x55f88237db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:41.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.033+0000 7ffa4d9ca640 1 --2- 172.21.15.16:0/1863671348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f88237b790 0x55f88237db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:41.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.033+0000 7ffa4d9ca640 1 -- 172.21.15.16:0/1863671348 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8822927c0 con 0x55f882372290 2024-03-28T22:13:41.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.033+0000 7ffa4e1cb640 1 --2- 172.21.15.16:0/4281856259 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ffa340a1ef0 0x7ffa340a43b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ffa4000eba0 tx=0x7ffa40009d70 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:41.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.033+0000 7ffa4d9ca640 1 --2- 172.21.15.16:0/1863671348 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f882372290 0x55f882372670 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ffa4402cfd0 tx=0x7ffa44005e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74bb114b77ee624e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:41.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.034+0000 7ffa33fff640 1 -- 172.21.15.16:0/1863671348 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa44042070 con 0x55f882372290 2024-03-28T22:13:41.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.034+0000 7ffa33fff640 1 -- 172.21.15.16:0/1863671348 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa4402ca80 con 0x55f882372290 2024-03-28T22:13:41.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.034+0000 7ffa33fff640 1 -- 172.21.15.16:0/1863671348 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa44004f50 con 0x55f882372290 2024-03-28T22:13:41.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.034+0000 7ffa52814740 1 -- 172.21.15.16:0/1863671348 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f882372290 msgr2=0x55f882372670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:41.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.034+0000 7ffa52814740 1 --2- 172.21.15.16:0/1863671348 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f882372290 0x55f882372670 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ffa4402cfd0 tx=0x7ffa44005e00 comp rx=0 tx=0).stop 2024-03-28T22:13:41.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.034+0000 7ffa52814740 1 -- 172.21.15.16:0/1863671348 shutdown_connections 2024-03-28T22:13:41.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.034+0000 7ffa52814740 1 --2- 172.21.15.16:0/1863671348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f88237b790 0x55f88237db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:41.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.034+0000 7ffa52814740 1 --2- 172.21.15.16:0/1863671348 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f882372bb0 0x55f88237b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:41.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.034+0000 7ffa52814740 1 --2- 172.21.15.16:0/1863671348 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f882372290 0x55f882372670 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:41.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.035+0000 7ffa52814740 1 -- 172.21.15.16:0/1863671348 >> 172.21.15.16:0/1863671348 conn(0x55f88236c9c0 msgr2=0x55f88236cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:41.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.035+0000 7ffa52814740 1 -- 172.21.15.16:0/1863671348 shutdown_connections 2024-03-28T22:13:41.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.035+0000 7ffa52814740 1 -- 172.21.15.16:0/1863671348 wait complete. 2024-03-28T22:13:41.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.035+0000 7ffa52814740 1 -- start start 2024-03-28T22:13:41.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.035+0000 7ffa52814740 1 Processor -- start 2024-03-28T22:13:41.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.035+0000 7ffa52814740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f882372290 0x55f8823787e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:41.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.035+0000 7ffa52814740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f882372bb0 0x55f882378d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:41.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.035+0000 7ffa52814740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f88237b790 0x55f882379260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:41.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.035+0000 7ffa52814740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f882382930 con 0x55f882372bb0 2024-03-28T22:13:41.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.035+0000 7ffa52814740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f8822744a0 con 0x55f882372290 2024-03-28T22:13:41.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.035+0000 7ffa52814740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f882382ab0 con 0x55f88237b790 2024-03-28T22:13:41.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.035+0000 7ffa52814740 10 client.0 _ll_register_callbacks cb 0x55f882277d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:13:41.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.036+0000 7ffa4d1c9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f882372bb0 0x55f882378d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:41.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.036+0000 7ffa4d1c9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f882372bb0 0x55f882378d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38946/0 (socket says 172.21.15.16:38946) 2024-03-28T22:13:41.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.036+0000 7ffa4d9ca640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f882372290 0x55f8823787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:41.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.036+0000 7ffa4d1c9640 1 -- 172.21.15.16:0/1750496977 learned_addr learned my addr 172.21.15.16:0/1750496977 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:41.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.036+0000 7ffa4e1cb640 1 --2- 172.21.15.16:0/1750496977 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f88237b790 0x55f882379260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:41.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.036+0000 7ffa4d1c9640 1 -- 172.21.15.16:0/1750496977 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f88237b790 msgr2=0x55f882379260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:41.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.036+0000 7ffa4d1c9640 1 --2- 172.21.15.16:0/1750496977 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f88237b790 0x55f882379260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:41.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.036+0000 7ffa4d1c9640 1 -- 172.21.15.16:0/1750496977 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f882372290 msgr2=0x55f8823787e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:41.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.036+0000 7ffa4d1c9640 1 --2- 172.21.15.16:0/1750496977 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f882372290 0x55f8823787e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:41.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.036+0000 7ffa4d1c9640 1 -- 172.21.15.16:0/1750496977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8820b47d0 con 0x55f882372bb0 2024-03-28T22:13:41.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.036+0000 7ffa4d9ca640 1 --2- 172.21.15.16:0/1750496977 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f882372290 0x55f8823787e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:13:41.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.036+0000 7ffa4d1c9640 1 --2- 172.21.15.16:0/1750496977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f882372bb0 0x55f882378d20 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7ffa3809aec0 tx=0x7ffa38005210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76152035f198b2fa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:41.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.036+0000 7ffa317fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:13:41.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.036+0000 7ffa317fa640 1 -- 172.21.15.16:0/1750496977 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3809b630 con 0x55f882372bb0 2024-03-28T22:13:41.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.036+0000 7ffa317fa640 1 -- 172.21.15.16:0/1750496977 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7ffa38019720 con 0x55f882372bb0 2024-03-28T22:13:41.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.036+0000 7ffa52814740 1 -- 172.21.15.16:0/1750496977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa4402ca80 con 0x55f882372bb0 2024-03-28T22:13:41.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.036+0000 7ffa317fa640 1 -- 172.21.15.16:0/1750496977 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ffa380b8050 con 0x55f882372bb0 2024-03-28T22:13:41.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.036+0000 7ffa52814740 10 client.16200 fetch_fsmap learned FSMap version 24 2024-03-28T22:13:41.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.036+0000 7ffa52814740 1 -- 172.21.15.16:0/1750496977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f881fea3f0 con 0x55f882372bb0 2024-03-28T22:13:41.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.036+0000 7ffa317fa640 1 -- 172.21.15.16:0/1750496977 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa4402ca80 con 0x55f882372bb0 2024-03-28T22:13:41.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.037+0000 7ffa317fa640 1 -- 172.21.15.16:0/1750496977 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7ffa380b9240 con 0x55f882372bb0 2024-03-28T22:13:41.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.037+0000 7ffa52814740 10 client.16200 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:13:41.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.037+0000 7ffa52814740 10 client.16200 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:13:41.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.037+0000 7ffa52814740 20 client.16200 populate_metadata read hostname 'smithi016' 2024-03-28T22:13:41.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.037+0000 7ffa52814740 1 --2- 172.21.15.16:0/1750496977 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f88238fe90 0x55f882392350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:41.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.037+0000 7ffa52814740 4 client.16200 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:13:41.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.037+0000 7ffa52814740 1 -- 172.21.15.16:0/1750496977 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f8822e36c0 con 0x55f88238fe90 2024-03-28T22:13:41.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.037+0000 7ffa4d9ca640 1 --2- 172.21.15.16:0/1750496977 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f88238fe90 0x55f882392350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:41.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.037+0000 7ffa317fa640 10 client.16200 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:13:41.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.037+0000 7ffa4d9ca640 1 --2- 172.21.15.16:0/1750496977 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f88238fe90 0x55f882392350 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8451d02c1d2f45b server_cookie=35e93b60eece87a3 in_seq=0 out_seq=0 2024-03-28T22:13:41.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.038+0000 7ffa317fa640 1 -- 172.21.15.16:0/1750496977 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f8822e36c0 con 0x55f88238fe90 2024-03-28T22:13:41.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.038+0000 7ffa317fa640 10 client.16200 handle_command_reply: tid=0 2024-03-28T22:13:41.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.038+0000 7ffa52814740 1 client.16200 shutdown 2024-03-28T22:13:41.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.038+0000 7ffa52814740 1 -- 172.21.15.16:0/1750496977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f882372bb0 msgr2=0x55f882378d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:41.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.038+0000 7ffa52814740 1 --2- 172.21.15.16:0/1750496977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f882372bb0 0x55f882378d20 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7ffa3809aec0 tx=0x7ffa38005210 comp rx=0 tx=0).stop 2024-03-28T22:13:41.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.038+0000 7ffa52814740 1 -- 172.21.15.16:0/1750496977 shutdown_connections 2024-03-28T22:13:41.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.038+0000 7ffa52814740 1 --2- 172.21.15.16:0/1750496977 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f88238fe90 0x55f882392350 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:41.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.038+0000 7ffa52814740 1 --2- 172.21.15.16:0/1750496977 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f88237b790 0x55f882379260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:41.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.038+0000 7ffa52814740 1 --2- 172.21.15.16:0/1750496977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f882372bb0 0x55f882378d20 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:41.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.038+0000 7ffa317fa640 0 client.16200 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:13:41.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.038+0000 7ffa52814740 1 --2- 172.21.15.16:0/1750496977 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f882372290 0x55f8823787e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:41.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.038+0000 7ffa52814740 1 -- 172.21.15.16:0/1750496977 >> 172.21.15.16:0/1750496977 conn(0x55f88236c9c0 msgr2=0x55f88236e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:41.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.038+0000 7ffa52814740 1 -- 172.21.15.16:0/1750496977 shutdown_connections 2024-03-28T22:13:41.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.038+0000 7ffa52814740 1 -- 172.21.15.16:0/1750496977 wait complete. 2024-03-28T22:13:41.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.038+0000 7ffa52814740 20 client.16200 trim_cache size 0 max 16384 2024-03-28T22:13:41.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.056+0000 7ffa52814740 1 Processor -- start 2024-03-28T22:13:41.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.056+0000 7ffa52814740 1 -- start start 2024-03-28T22:13:41.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.056+0000 7ffa52814740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f882372290 0x55f88237af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:41.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.056+0000 7ffa52814740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f882372bb0 0x55f88238b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:41.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.056+0000 7ffa52814740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f88237b790 0x55f882397c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:41.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.056+0000 7ffa52814740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffa481ada00 con 0x55f882372bb0 2024-03-28T22:13:41.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.056+0000 7ffa52814740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ffa481ab2a0 con 0x55f88237b790 2024-03-28T22:13:41.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.056+0000 7ffa52814740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ffa481a90f0 con 0x55f882372290 2024-03-28T22:13:41.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.056+0000 7ffa4d1c9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f882372bb0 0x55f88238b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:41.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.056+0000 7ffa4d1c9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f882372bb0 0x55f88238b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38956/0 (socket says 172.21.15.16:38956) 2024-03-28T22:13:41.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.056+0000 7ffa4d1c9640 1 -- 172.21.15.16:0/1613755478 learned_addr learned my addr 172.21.15.16:0/1613755478 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:41.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.056+0000 7ffa4d9ca640 1 --2- 172.21.15.16:0/1613755478 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f882372290 0x55f88237af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:41.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.057+0000 7ffa4e1cb640 1 --2- 172.21.15.16:0/1613755478 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f88237b790 0x55f882397c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:41.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.057+0000 7ffa4d1c9640 1 -- 172.21.15.16:0/1613755478 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f882372290 msgr2=0x55f88237af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:41.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.057+0000 7ffa4d1c9640 1 --2- 172.21.15.16:0/1613755478 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f882372290 0x55f88237af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:41.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.057+0000 7ffa4d1c9640 1 -- 172.21.15.16:0/1613755478 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f88237b790 msgr2=0x55f882397c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:41.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.057+0000 7ffa4d1c9640 1 --2- 172.21.15.16:0/1613755478 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f88237b790 0x55f882397c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:41.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.057+0000 7ffa4d1c9640 1 -- 172.21.15.16:0/1613755478 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f881faa0f0 con 0x55f882372bb0 2024-03-28T22:13:41.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.057+0000 7ffa4d1c9640 1 --2- 172.21.15.16:0/1613755478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f882372bb0 0x55f88238b6d0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7ffa380a3720 tx=0x7ffa38018420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44cbf136bf63ea6b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:41.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.057+0000 7ffa337fe640 1 -- 172.21.15.16:0/1613755478 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa380a8070 con 0x55f882372bb0 2024-03-28T22:13:41.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.057+0000 7ffa337fe640 1 -- 172.21.15.16:0/1613755478 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa3809f080 con 0x55f882372bb0 2024-03-28T22:13:41.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.057+0000 7ffa337fe640 1 -- 172.21.15.16:0/1613755478 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa380975a0 con 0x55f882372bb0 2024-03-28T22:13:41.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.058+0000 7ffa52814740 1 -- 172.21.15.16:0/1613755478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f882372bb0 msgr2=0x55f88238b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:41.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.058+0000 7ffa52814740 1 --2- 172.21.15.16:0/1613755478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f882372bb0 0x55f88238b6d0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7ffa380a3720 tx=0x7ffa38018420 comp rx=0 tx=0).stop 2024-03-28T22:13:41.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.058+0000 7ffa52814740 1 -- 172.21.15.16:0/1613755478 shutdown_connections 2024-03-28T22:13:41.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.058+0000 7ffa52814740 1 --2- 172.21.15.16:0/1613755478 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f88237b790 0x55f882397c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:41.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.058+0000 7ffa52814740 1 --2- 172.21.15.16:0/1613755478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f882372bb0 0x55f88238b6d0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:41.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.058+0000 7ffa52814740 1 --2- 172.21.15.16:0/1613755478 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f882372290 0x55f88237af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:41.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.058+0000 7ffa52814740 1 -- 172.21.15.16:0/1613755478 >> 172.21.15.16:0/1613755478 conn(0x55f88236c9c0 msgr2=0x55f88236bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:41.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.058+0000 7ffa52814740 1 -- 172.21.15.16:0/1613755478 shutdown_connections 2024-03-28T22:13:41.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.058+0000 7ffa52814740 1 -- 172.21.15.16:0/1613755478 wait complete. 2024-03-28T22:13:41.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.058+0000 7ffa52814740 1 -- start start 2024-03-28T22:13:41.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.058+0000 7ffa52814740 1 Processor -- start 2024-03-28T22:13:41.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.059+0000 7ffa52814740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f882372290 0x55f8823756b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:41.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.059+0000 7ffa52814740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f88237b790 0x55f882375bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:41.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.059+0000 7ffa52814740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f882376180 0x55f8823f5250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:41.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.059+0000 7ffa52814740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffa481a9760 con 0x55f882376180 2024-03-28T22:13:41.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.059+0000 7ffa52814740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ffa481aabd0 con 0x55f88237b790 2024-03-28T22:13:41.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.059+0000 7ffa52814740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ffa481a9e30 con 0x55f882372290 2024-03-28T22:13:41.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.059+0000 7ffa52814740 10 client.0 _ll_register_callbacks cb 0x7ffa3800e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:13:41.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.059+0000 7ffa4e1cb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f882376180 0x55f8823f5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:41.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.059+0000 7ffa4d9ca640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f882372290 0x55f8823756b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:41.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.059+0000 7ffa4e1cb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f882376180 0x55f8823f5250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38964/0 (socket says 172.21.15.16:38964) 2024-03-28T22:13:41.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.059+0000 7ffa4e1cb640 1 -- 172.21.15.16:0/4040074012 learned_addr learned my addr 172.21.15.16:0/4040074012 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:13:41.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.059+0000 7ffa4d1c9640 1 --2- 172.21.15.16:0/4040074012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f88237b790 0x55f882375bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:41.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.059+0000 7ffa4e1cb640 1 -- 172.21.15.16:0/4040074012 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f882372290 msgr2=0x55f8823756b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:41.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.059+0000 7ffa4e1cb640 1 --2- 172.21.15.16:0/4040074012 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f882372290 0x55f8823756b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:41.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.059+0000 7ffa4e1cb640 1 -- 172.21.15.16:0/4040074012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f88237b790 msgr2=0x55f882375bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:13:41.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.059+0000 7ffa4e1cb640 1 --2- 172.21.15.16:0/4040074012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f88237b790 0x55f882375bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:41.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.059+0000 7ffa4e1cb640 1 -- 172.21.15.16:0/4040074012 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8823f5990 con 0x55f882376180 2024-03-28T22:13:41.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.059+0000 7ffa4e1cb640 1 --2- 172.21.15.16:0/4040074012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f882376180 0x55f8823f5250 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7ffa4000cd70 tx=0x7ffa40018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bba93feedaa695c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:13:41.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.059+0000 7ffa317fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:13:41.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.060+0000 7ffa317fa640 1 -- 172.21.15.16:0/4040074012 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa4002c060 con 0x55f882376180 2024-03-28T22:13:41.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.060+0000 7ffa317fa640 1 -- 172.21.15.16:0/4040074012 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7ffa400273f0 con 0x55f882376180 2024-03-28T22:13:41.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.060+0000 7ffa52814740 1 -- 172.21.15.16:0/4040074012 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa3809f080 con 0x55f882376180 2024-03-28T22:13:41.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.060+0000 7ffa317fa640 1 -- 172.21.15.16:0/4040074012 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ffa4000e280 con 0x55f882376180 2024-03-28T22:13:41.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.060+0000 7ffa52814740 10 client.16206 fetch_fsmap learned FSMap version 24 2024-03-28T22:13:41.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.060+0000 7ffa52814740 1 -- 172.21.15.16:0/4040074012 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f8823f6ca0 con 0x55f882376180 2024-03-28T22:13:41.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.060+0000 7ffa317fa640 1 -- 172.21.15.16:0/4040074012 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa3809f080 con 0x55f882376180 2024-03-28T22:13:41.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.060+0000 7ffa317fa640 1 -- 172.21.15.16:0/4040074012 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 24) ==== 8350+0+0 (secure 0 0 0) 0x7ffa40026070 con 0x55f882376180 2024-03-28T22:13:41.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.060+0000 7ffa52814740 10 client.16206 fetch_fsmap finished waiting for FSMap version 24 2024-03-28T22:13:41.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.060+0000 7ffa52814740 10 client.16206 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:13:41.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.060+0000 7ffa52814740 20 client.16206 populate_metadata read hostname 'smithi016' 2024-03-28T22:13:41.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.060+0000 7ffa52814740 1 --2- 172.21.15.16:0/4040074012 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f8823f74b0 0x55f8823f98d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:13:41.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.060+0000 7ffa52814740 4 client.16206 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:13:41.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.060+0000 7ffa52814740 1 -- 172.21.15.16:0/4040074012 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f8822dda10 con 0x55f8823f74b0 2024-03-28T22:13:41.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.061+0000 7ffa4d9ca640 1 --2- 172.21.15.16:0/4040074012 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f8823f74b0 0x55f8823f98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:13:41.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.061+0000 7ffa4d9ca640 1 --2- 172.21.15.16:0/4040074012 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f8823f74b0 0x55f8823f98d0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30cc5313299371d8 server_cookie=826376d71692b070 in_seq=0 out_seq=0 2024-03-28T22:13:41.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.061+0000 7ffa317fa640 10 client.16206 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:13:41.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.062+0000 7ffa317fa640 1 -- 172.21.15.16:0/4040074012 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55f8822dda10 con 0x55f8823f74b0 2024-03-28T22:13:41.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.062+0000 7ffa317fa640 10 client.16206 handle_command_reply: tid=0 2024-03-28T22:13:41.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.062+0000 7ffa52814740 1 client.16206 shutdown 2024-03-28T22:13:41.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.062+0000 7ffa52814740 1 -- 172.21.15.16:0/4040074012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f882376180 msgr2=0x55f8823f5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:41.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.062+0000 7ffa52814740 1 --2- 172.21.15.16:0/4040074012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f882376180 0x55f8823f5250 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7ffa4000cd70 tx=0x7ffa40018000 comp rx=0 tx=0).stop 2024-03-28T22:13:41.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.062+0000 7ffa52814740 1 -- 172.21.15.16:0/4040074012 shutdown_connections 2024-03-28T22:13:41.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.062+0000 7ffa52814740 1 --2- 172.21.15.16:0/4040074012 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f8823f74b0 0x55f8823f98d0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:41.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.062+0000 7ffa52814740 1 --2- 172.21.15.16:0/4040074012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f882376180 0x55f8823f5250 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:41.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.062+0000 7ffa52814740 1 --2- 172.21.15.16:0/4040074012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f88237b790 0x55f882375bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:41.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.062+0000 7ffa52814740 1 --2- 172.21.15.16:0/4040074012 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f882372290 0x55f8823756b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:41.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.062+0000 7ffa317fa640 0 client.16206 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:13:41.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.062+0000 7ffa52814740 1 -- 172.21.15.16:0/4040074012 >> 172.21.15.16:0/4040074012 conn(0x55f88236c9c0 msgr2=0x55f8823787a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:41.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.062+0000 7ffa52814740 1 -- 172.21.15.16:0/4040074012 shutdown_connections 2024-03-28T22:13:41.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.062+0000 7ffa52814740 1 -- 172.21.15.16:0/4040074012 wait complete. 2024-03-28T22:13:41.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.062+0000 7ffa52814740 20 client.16206 trim_cache size 0 max 16384 2024-03-28T22:13:41.076 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:13:41.076 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T22:13:41.076 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-03-28T22:13:41.076 INFO:teuthology.orchestra.run.smithi016.stdout: "55ca60c4-9da8-40a9-af97-ba2ee7977c3f": { 2024-03-28T22:13:41.076 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-03-28T22:13:41.076 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "55ca60c4-9da8-40a9-af97-ba2ee7977c3f", 2024-03-28T22:13:41.076 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-03-28T22:13:41.076 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:13:41.077 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:13:41.077 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:13:41.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.063+0000 7ffa4fc2e640 1 -- 172.21.15.16:0/4281856259 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ffa340a1ef0 msgr2=0x7ffa340a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:41.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.063+0000 7ffa4fc2e640 1 --2- 172.21.15.16:0/4281856259 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ffa340a1ef0 0x7ffa340a43b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ffa4000eba0 tx=0x7ffa40009d70 comp rx=0 tx=0).stop 2024-03-28T22:13:41.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.063+0000 7ffa4fc2e640 1 -- 172.21.15.16:0/4281856259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa48100fa0 msgr2=0x7ffa48198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:13:41.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.063+0000 7ffa4fc2e640 1 --2- 172.21.15.16:0/4281856259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa48100fa0 0x7ffa48198590 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7ffa3800e9f0 tx=0x7ffa3800eeb0 comp rx=0 tx=0).stop 2024-03-28T22:13:41.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.063+0000 7ffa4fc2e640 1 -- 172.21.15.16:0/4281856259 shutdown_connections 2024-03-28T22:13:41.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.063+0000 7ffa4fc2e640 1 --2- 172.21.15.16:0/4281856259 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ffa340a1ef0 0x7ffa340a43b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:41.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.063+0000 7ffa4fc2e640 1 --2- 172.21.15.16:0/4281856259 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa48106680 0x7ffa4819c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:41.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.063+0000 7ffa4fc2e640 1 --2- 172.21.15.16:0/4281856259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa48100fa0 0x7ffa48198590 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:41.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.063+0000 7ffa4fc2e640 1 --2- 172.21.15.16:0/4281856259 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa481005e0 0x7ffa48198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:13:41.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.063+0000 7ffa4fc2e640 1 -- 172.21.15.16:0/4281856259 >> 172.21.15.16:0/4281856259 conn(0x7ffa480fc0b0 msgr2=0x7ffa480fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:13:41.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.063+0000 7ffa4fc2e640 1 -- 172.21.15.16:0/4281856259 shutdown_connections 2024-03-28T22:13:41.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:13:41.063+0000 7ffa4fc2e640 1 -- 172.21.15.16:0/4281856259 wait complete. 2024-03-28T22:13:41.079 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:41 smithi016 ceph-mon[32374]: scrub queued for path: / 2024-03-28T22:13:41.079 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:41 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:13:41.080 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:41 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:13:41.080 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:41 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:13:41.081 INFO:tasks.cephfs.filesystem:scrub status for tag:55ca60c4-9da8-40a9-af97-ba2ee7977c3f - {'path': '/', 'tag': '55ca60c4-9da8-40a9-af97-ba2ee7977c3f', 'options': 'recursive,force'} 2024-03-28T22:13:41.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:41 smithi053 ceph-mon[35979]: scrub queued for path: / 2024-03-28T22:13:41.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:41 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:13:41.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:41 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:13:41.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:41 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:13:41.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:41 smithi063 ceph-mon[36249]: scrub queued for path: / 2024-03-28T22:13:41.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:41 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:13:41.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:41 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:13:41.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:41 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:13:42.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:42 smithi053 ceph-mon[35979]: pgmap v96: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-03-28T22:13:42.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:42 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:13:42.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:42 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:13:42.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:42 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:13:42.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:42 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:13:42.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:42 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:13:42.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:42 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:13:42.333 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:42 smithi063 ceph-mon[36249]: pgmap v96: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-03-28T22:13:42.333 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:42 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:13:42.333 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:42 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:13:42.333 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:42 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:13:42.334 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:42 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:13:42.334 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:42 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:13:42.334 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:42 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:13:42.338 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:42 smithi016 ceph-mon[32374]: pgmap v96: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-03-28T22:13:42.338 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:42 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:13:42.338 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:42 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:13:42.338 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:42 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:13:42.338 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:42 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:13:42.338 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:42 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:13:42.338 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:42 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:13:43.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:43 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:13:43.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:43 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:13:43.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:43 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:13:44.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:44 smithi053 ceph-mon[35979]: pgmap v97: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-03-28T22:13:44.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:44 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:13:44.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:44 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:13:44.298 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:44 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:13:44.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:44 smithi063 ceph-mon[36249]: pgmap v97: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-03-28T22:13:44.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:44 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:13:44.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:44 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:13:44.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:44 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:13:44.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:44 smithi016 ceph-mon[32374]: pgmap v97: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-03-28T22:13:44.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:44 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:13:44.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:44 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:13:44.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:44 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:13:46.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:46 smithi053 ceph-mon[35979]: pgmap v98: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-03-28T22:13:46.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:46 smithi063 ceph-mon[36249]: pgmap v98: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-03-28T22:13:46.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:46 smithi016 ceph-mon[32374]: pgmap v98: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-03-28T22:13:48.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:48 smithi053 ceph-mon[35979]: pgmap v99: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.2 KiB/s wr, 9 op/s 2024-03-28T22:13:48.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:48 smithi063 ceph-mon[36249]: pgmap v99: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.2 KiB/s wr, 9 op/s 2024-03-28T22:13:48.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:48 smithi016 ceph-mon[32374]: pgmap v99: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.2 KiB/s wr, 9 op/s 2024-03-28T22:13:50.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:50 smithi053 ceph-mon[35979]: pgmap v100: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-28T22:13:50.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:50 smithi063 ceph-mon[36249]: pgmap v100: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-28T22:13:50.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:50 smithi016 ceph-mon[32374]: pgmap v100: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-28T22:13:52.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:52 smithi053 ceph-mon[35979]: pgmap v101: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2024-03-28T22:13:52.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:52 smithi063 ceph-mon[36249]: pgmap v101: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2024-03-28T22:13:52.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:52 smithi016 ceph-mon[32374]: pgmap v101: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2024-03-28T22:13:54.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:54 smithi053 ceph-mon[35979]: pgmap v102: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-28T22:13:54.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:54 smithi063 ceph-mon[36249]: pgmap v102: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-28T22:13:54.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:54 smithi016 ceph-mon[32374]: pgmap v102: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-28T22:13:55.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:55 smithi063 ceph-mon[36249]: scrub summary: idle 2024-03-28T22:13:55.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:55 smithi016 ceph-mon[32374]: scrub summary: idle 2024-03-28T22:13:55.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:55 smithi053 ceph-mon[35979]: scrub summary: idle 2024-03-28T22:13:56.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:56 smithi063 ceph-mon[36249]: pgmap v103: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-28T22:13:56.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:56 smithi063 ceph-mon[36249]: scrub summary: idle 2024-03-28T22:13:56.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:56 smithi016 ceph-mon[32374]: pgmap v103: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-28T22:13:56.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:56 smithi016 ceph-mon[32374]: scrub summary: idle 2024-03-28T22:13:56.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:56 smithi053 ceph-mon[35979]: pgmap v103: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-28T22:13:56.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:56 smithi053 ceph-mon[35979]: scrub summary: idle 2024-03-28T22:13:58.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:13:58 smithi063 ceph-mon[36249]: pgmap v104: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.4 KiB/s wr, 9 op/s 2024-03-28T22:13:58.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:13:58 smithi016 ceph-mon[32374]: pgmap v104: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.4 KiB/s wr, 9 op/s 2024-03-28T22:13:58.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:13:58 smithi053 ceph-mon[35979]: pgmap v104: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.4 KiB/s wr, 9 op/s 2024-03-28T22:14:00.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:00 smithi063 ceph-mon[36249]: pgmap v105: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-28T22:14:00.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:00 smithi016 ceph-mon[32374]: pgmap v105: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-28T22:14:00.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:00 smithi053 ceph-mon[35979]: pgmap v105: 97 pgs: 97 active+clean; 94 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-28T22:14:01.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:01 smithi063 ceph-mon[36249]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T22:14:01.438 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:01 smithi016 ceph-mon[32374]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T22:14:01.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:01 smithi053 ceph-mon[35979]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T22:14:02.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:02 smithi063 ceph-mon[36249]: pgmap v106: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 213 KiB/s wr, 24 op/s 2024-03-28T22:14:02.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:02 smithi016 ceph-mon[32374]: pgmap v106: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 213 KiB/s wr, 24 op/s 2024-03-28T22:14:02.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:02 smithi053 ceph-mon[35979]: pgmap v106: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 213 KiB/s wr, 24 op/s 2024-03-28T22:14:04.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:04 smithi063 ceph-mon[36249]: pgmap v107: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 213 KiB/s wr, 21 op/s 2024-03-28T22:14:04.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:04 smithi016 ceph-mon[32374]: pgmap v107: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 213 KiB/s wr, 21 op/s 2024-03-28T22:14:04.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:04 smithi053 ceph-mon[35979]: pgmap v107: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 213 KiB/s wr, 21 op/s 2024-03-28T22:14:06.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:06 smithi063 ceph-mon[36249]: pgmap v108: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 213 KiB/s wr, 20 op/s 2024-03-28T22:14:06.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:06 smithi016 ceph-mon[32374]: pgmap v108: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 213 KiB/s wr, 20 op/s 2024-03-28T22:14:06.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:06 smithi053 ceph-mon[35979]: pgmap v108: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 213 KiB/s wr, 20 op/s 2024-03-28T22:14:08.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:08 smithi063 ceph-mon[36249]: pgmap v109: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 216 KiB/s wr, 24 op/s 2024-03-28T22:14:08.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:08 smithi016 ceph-mon[32374]: pgmap v109: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 216 KiB/s wr, 24 op/s 2024-03-28T22:14:08.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:08 smithi053 ceph-mon[35979]: pgmap v109: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 216 KiB/s wr, 24 op/s 2024-03-28T22:14:10.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:10 smithi063 ceph-mon[36249]: pgmap v110: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 216 KiB/s wr, 20 op/s 2024-03-28T22:14:10.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:10 smithi016 ceph-mon[32374]: pgmap v110: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 216 KiB/s wr, 20 op/s 2024-03-28T22:14:10.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:10 smithi053 ceph-mon[35979]: pgmap v110: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 216 KiB/s wr, 20 op/s 2024-03-28T22:14:11.082 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:14:11.083 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:14:11.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.197+0000 7f50de6eb640 1 -- 172.21.15.16:0/2831529038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d81009d0 msgr2=0x7f50d8100db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:11.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.197+0000 7f50de6eb640 1 --2- 172.21.15.16:0/2831529038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d81009d0 0x7f50d8100db0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f50cc009900 tx=0x7f50cc02bd10 comp rx=0 tx=0).stop 2024-03-28T22:14:11.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.197+0000 7f50de6eb640 1 -- 172.21.15.16:0/2831529038 shutdown_connections 2024-03-28T22:14:11.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.197+0000 7f50de6eb640 1 --2- 172.21.15.16:0/2831529038 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50d8102b80 0x7f50d810d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.197+0000 7f50de6eb640 1 --2- 172.21.15.16:0/2831529038 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50d81021e0 0x7f50d8102640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.197+0000 7f50de6eb640 1 --2- 172.21.15.16:0/2831529038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d81009d0 0x7f50d8100db0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.197+0000 7f50de6eb640 1 -- 172.21.15.16:0/2831529038 >> 172.21.15.16:0/2831529038 conn(0x7f50d8077830 msgr2=0x7f50d8077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:11.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.197+0000 7f50de6eb640 1 -- 172.21.15.16:0/2831529038 shutdown_connections 2024-03-28T22:14:11.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.197+0000 7f50de6eb640 1 -- 172.21.15.16:0/2831529038 wait complete. 2024-03-28T22:14:11.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.198+0000 7f50de6eb640 1 Processor -- start 2024-03-28T22:14:11.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.198+0000 7f50de6eb640 1 -- start start 2024-03-28T22:14:11.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.198+0000 7f50de6eb640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50d81009d0 0x7f50d8198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.198+0000 7f50de6eb640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50d81021e0 0x7f50d8198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.198+0000 7f50de6eb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d8102b80 0x7f50d819c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.198+0000 7f50de6eb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f50d8110b20 con 0x7f50d8102b80 2024-03-28T22:14:11.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.198+0000 7f50de6eb640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f50d81109a0 con 0x7f50d81009d0 2024-03-28T22:14:11.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.198+0000 7f50de6eb640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f50d8110ca0 con 0x7f50d81021e0 2024-03-28T22:14:11.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.199+0000 7f50dcc88640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d8102b80 0x7f50d819c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.199+0000 7f50d7fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50d81009d0 0x7f50d8198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.199+0000 7f50dcc88640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d8102b80 0x7f50d819c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46192/0 (socket says 172.21.15.16:46192) 2024-03-28T22:14:11.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.199+0000 7f50dcc88640 1 -- 172.21.15.16:0/3235246960 learned_addr learned my addr 172.21.15.16:0/3235246960 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:11.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.199+0000 7f50d77fe640 1 --2- 172.21.15.16:0/3235246960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50d81021e0 0x7f50d8198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.199+0000 7f50dcc88640 1 -- 172.21.15.16:0/3235246960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50d81021e0 msgr2=0x7f50d8198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:11.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.199+0000 7f50dcc88640 1 --2- 172.21.15.16:0/3235246960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50d81021e0 0x7f50d8198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.199+0000 7f50dcc88640 1 -- 172.21.15.16:0/3235246960 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50d81009d0 msgr2=0x7f50d8198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:11.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.199+0000 7f50dcc88640 1 --2- 172.21.15.16:0/3235246960 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50d81009d0 0x7f50d8198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.199+0000 7f50dcc88640 1 -- 172.21.15.16:0/3235246960 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50d819d030 con 0x7f50d8102b80 2024-03-28T22:14:11.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.199+0000 7f50d77fe640 1 --2- 172.21.15.16:0/3235246960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50d81021e0 0x7f50d8198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:14:11.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.199+0000 7f50dcc88640 1 --2- 172.21.15.16:0/3235246960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d8102b80 0x7f50d819c900 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f50d000ea60 tx=0x7f50d000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:11.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.199+0000 7f50d57fa640 1 -- 172.21.15.16:0/3235246960 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d000ce40 con 0x7f50d8102b80 2024-03-28T22:14:11.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.199+0000 7f50d57fa640 1 -- 172.21.15.16:0/3235246960 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50d0005020 con 0x7f50d8102b80 2024-03-28T22:14:11.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.199+0000 7f50d57fa640 1 -- 172.21.15.16:0/3235246960 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d0010690 con 0x7f50d8102b80 2024-03-28T22:14:11.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.199+0000 7f50de6eb640 1 -- 172.21.15.16:0/3235246960 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50d819d320 con 0x7f50d8102b80 2024-03-28T22:14:11.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.199+0000 7f50de6eb640 1 -- 172.21.15.16:0/3235246960 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50d81a4bb0 con 0x7f50d8102b80 2024-03-28T22:14:11.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.201+0000 7f50d57fa640 1 -- 172.21.15.16:0/3235246960 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f50d0005520 con 0x7f50d8102b80 2024-03-28T22:14:11.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.202+0000 7f50e13a6740 1 Processor -- start 2024-03-28T22:14:11.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.202+0000 7f50e13a6740 1 -- start start 2024-03-28T22:14:11.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.202+0000 7f50d57fa640 1 --2- 172.21.15.16:0/3235246960 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f50c00a1ef0 0x7f50c00a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.202+0000 7f50e13a6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ed0271290 0x558ed0271670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.202+0000 7f50e13a6740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558ed0271bb0 0x558ed027a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.202+0000 7f50e13a6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558ed027a790 0x558ed027cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.202+0000 7f50e13a6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558ed01d5180 con 0x558ed0271290 2024-03-28T22:14:11.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.202+0000 7f50e13a6740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x558ed017db70 con 0x558ed0271bb0 2024-03-28T22:14:11.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.202+0000 7f50e13a6740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x558ed01d1220 con 0x558ed027a790 2024-03-28T22:14:11.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.202+0000 7f50d57fa640 1 -- 172.21.15.16:0/3235246960 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f50d00ce710 con 0x7f50d8102b80 2024-03-28T22:14:11.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.202+0000 7f50d77fe640 1 --2- 172.21.15.16:0/3235246960 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f50c00a1ef0 0x7f50c00a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.202+0000 7f50d7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ed0271290 0x558ed0271670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.202+0000 7f50d7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ed0271290 0x558ed0271670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46200/0 (socket says 172.21.15.16:46200) 2024-03-28T22:14:11.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.202+0000 7f50d7fff640 1 -- 172.21.15.16:0/3895463906 learned_addr learned my addr 172.21.15.16:0/3895463906 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:11.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.202+0000 7f50d77fe640 1 --2- 172.21.15.16:0/3895463906 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558ed0271bb0 0x558ed027a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.202+0000 7f50dcc88640 1 --2- 172.21.15.16:0/3895463906 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558ed027a790 0x558ed027cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.202+0000 7f50d7fff640 1 -- 172.21.15.16:0/3895463906 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558ed027a790 msgr2=0x558ed027cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:11.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.203+0000 7f50d7fff640 1 --2- 172.21.15.16:0/3895463906 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558ed027a790 0x558ed027cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.203+0000 7f50d7fff640 1 -- 172.21.15.16:0/3895463906 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558ed0271bb0 msgr2=0x558ed027a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:11.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.203+0000 7f50d7fff640 1 --2- 172.21.15.16:0/3895463906 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558ed0271bb0 0x558ed027a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.203+0000 7f50d7fff640 1 -- 172.21.15.16:0/3895463906 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ed01917c0 con 0x558ed0271290 2024-03-28T22:14:11.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.203+0000 7f50dcc88640 1 --2- 172.21.15.16:0/3895463906 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558ed027a790 0x558ed027cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:14:11.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.203+0000 7f50d7fff640 1 --2- 172.21.15.16:0/3895463906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ed0271290 0x558ed0271670 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f50cc009900 tx=0x7f50cc02c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1d288e002eb32d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:11.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.203+0000 7f50d77fe640 1 --2- 172.21.15.16:0/3235246960 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f50c00a1ef0 0x7f50c00a43b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f50c8005fd0 tx=0x7f50c800ab60 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:11.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.203+0000 7f50be7fc640 1 -- 172.21.15.16:0/3895463906 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50cc043070 con 0x558ed0271290 2024-03-28T22:14:11.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.203+0000 7f50be7fc640 1 -- 172.21.15.16:0/3895463906 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50cc02c930 con 0x558ed0271290 2024-03-28T22:14:11.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.203+0000 7f50be7fc640 1 -- 172.21.15.16:0/3895463906 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50cc004c20 con 0x558ed0271290 2024-03-28T22:14:11.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.204+0000 7f50e13a6740 1 -- 172.21.15.16:0/3895463906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ed0271290 msgr2=0x558ed0271670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:11.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.204+0000 7f50e13a6740 1 --2- 172.21.15.16:0/3895463906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ed0271290 0x558ed0271670 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f50cc009900 tx=0x7f50cc02c650 comp rx=0 tx=0).stop 2024-03-28T22:14:11.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.204+0000 7f50e13a6740 1 -- 172.21.15.16:0/3895463906 shutdown_connections 2024-03-28T22:14:11.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.204+0000 7f50e13a6740 1 --2- 172.21.15.16:0/3895463906 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558ed027a790 0x558ed027cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.204+0000 7f50e13a6740 1 --2- 172.21.15.16:0/3895463906 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558ed0271bb0 0x558ed027a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.204+0000 7f50e13a6740 1 --2- 172.21.15.16:0/3895463906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ed0271290 0x558ed0271670 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.204+0000 7f50e13a6740 1 -- 172.21.15.16:0/3895463906 >> 172.21.15.16:0/3895463906 conn(0x558ed026b9c0 msgr2=0x558ed026bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:11.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.204+0000 7f50e13a6740 1 -- 172.21.15.16:0/3895463906 shutdown_connections 2024-03-28T22:14:11.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.204+0000 7f50e13a6740 1 -- 172.21.15.16:0/3895463906 wait complete. 2024-03-28T22:14:11.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.204+0000 7f50e13a6740 1 -- start start 2024-03-28T22:14:11.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.204+0000 7f50e13a6740 1 Processor -- start 2024-03-28T22:14:11.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.204+0000 7f50e13a6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ed0271290 0x558ed0277780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.205+0000 7f50e13a6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558ed0271bb0 0x558ed0277cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.205+0000 7f50e13a6740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558ed027a790 0x558ed0278200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.205+0000 7f50e13a6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558ed027d520 con 0x558ed0271290 2024-03-28T22:14:11.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.205+0000 7f50e13a6740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x558ed01734a0 con 0x558ed027a790 2024-03-28T22:14:11.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.205+0000 7f50e13a6740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x558ed027d6a0 con 0x558ed0271bb0 2024-03-28T22:14:11.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.205+0000 7f50e13a6740 10 client.0 _ll_register_callbacks cb 0x558ed0176d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:14:11.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.205+0000 7f50d7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ed0271290 0x558ed0277780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.205+0000 7f50d7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ed0271290 0x558ed0277780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46208/0 (socket says 172.21.15.16:46208) 2024-03-28T22:14:11.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.205+0000 7f50d7fff640 1 -- 172.21.15.16:0/459085019 learned_addr learned my addr 172.21.15.16:0/459085019 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:11.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.205+0000 7f50dcc88640 1 --2- 172.21.15.16:0/459085019 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558ed027a790 0x558ed0278200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.205+0000 7f50d77fe640 1 --2- 172.21.15.16:0/459085019 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558ed0271bb0 0x558ed0277cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.205+0000 7f50d7fff640 1 -- 172.21.15.16:0/459085019 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558ed0271bb0 msgr2=0x558ed0277cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:11.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.205+0000 7f50d7fff640 1 --2- 172.21.15.16:0/459085019 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558ed0271bb0 0x558ed0277cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.205+0000 7f50d7fff640 1 -- 172.21.15.16:0/459085019 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558ed027a790 msgr2=0x558ed0278200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:11.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.205+0000 7f50d7fff640 1 --2- 172.21.15.16:0/459085019 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558ed027a790 0x558ed0278200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.205+0000 7f50d7fff640 1 -- 172.21.15.16:0/459085019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ecffb37d0 con 0x558ed0271290 2024-03-28T22:14:11.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.205+0000 7f50d7fff640 1 --2- 172.21.15.16:0/459085019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ed0271290 0x558ed0277780 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f50cc006010 tx=0x7f50cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e0a9f66dea5924e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:11.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.205+0000 7f50b77fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:14:11.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.205+0000 7f50b77fe640 1 -- 172.21.15.16:0/459085019 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50cc043070 con 0x558ed0271290 2024-03-28T22:14:11.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.205+0000 7f50b77fe640 1 -- 172.21.15.16:0/459085019 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f50cc004460 con 0x558ed0271290 2024-03-28T22:14:11.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.205+0000 7f50e13a6740 1 -- 172.21.15.16:0/459085019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50cc02c930 con 0x558ed0271290 2024-03-28T22:14:11.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.206+0000 7f50b77fe640 1 -- 172.21.15.16:0/459085019 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f50cc004ec0 con 0x558ed0271290 2024-03-28T22:14:11.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.206+0000 7f50e13a6740 10 client.16218 fetch_fsmap learned FSMap version 25 2024-03-28T22:14:11.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.206+0000 7f50e13a6740 1 -- 172.21.15.16:0/459085019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558ecfee93f0 con 0x558ed0271290 2024-03-28T22:14:11.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.206+0000 7f50b77fe640 1 -- 172.21.15.16:0/459085019 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50cc02c930 con 0x558ed0271290 2024-03-28T22:14:11.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.206+0000 7f50b77fe640 1 -- 172.21.15.16:0/459085019 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f50cc048450 con 0x558ed0271290 2024-03-28T22:14:11.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.206+0000 7f50e13a6740 10 client.16218 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T22:14:11.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.206+0000 7f50e13a6740 10 client.16218 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:14:11.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.206+0000 7f50e13a6740 20 client.16218 populate_metadata read hostname 'smithi016' 2024-03-28T22:14:11.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.206+0000 7f50e13a6740 1 --2- 172.21.15.16:0/459085019 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558ed028a8f0 0x558ed028cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.206+0000 7f50e13a6740 4 client.16218 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:14:11.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.206+0000 7f50e13a6740 1 -- 172.21.15.16:0/459085019 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558ed01e26c0 con 0x558ed028a8f0 2024-03-28T22:14:11.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.206+0000 7f50d77fe640 1 --2- 172.21.15.16:0/459085019 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558ed028a8f0 0x558ed028cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.207+0000 7f50d77fe640 1 --2- 172.21.15.16:0/459085019 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558ed028a8f0 0x558ed028cce0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=545e39ea9ae11a6c server_cookie=a2e1c09c9ad47701 in_seq=0 out_seq=0 2024-03-28T22:14:11.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.207+0000 7f50b77fe640 10 client.16218 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:14:11.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.207+0000 7f50b77fe640 1 -- 172.21.15.16:0/459085019 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558ed01e26c0 con 0x558ed028a8f0 2024-03-28T22:14:11.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.207+0000 7f50b77fe640 10 client.16218 handle_command_reply: tid=0 2024-03-28T22:14:11.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.207+0000 7f50e13a6740 1 client.16218 shutdown 2024-03-28T22:14:11.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.207+0000 7f50e13a6740 1 -- 172.21.15.16:0/459085019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ed0271290 msgr2=0x558ed0277780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:11.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.207+0000 7f50e13a6740 1 --2- 172.21.15.16:0/459085019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ed0271290 0x558ed0277780 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f50cc006010 tx=0x7f50cc0046c0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.207+0000 7f50e13a6740 1 -- 172.21.15.16:0/459085019 shutdown_connections 2024-03-28T22:14:11.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.207+0000 7f50e13a6740 1 --2- 172.21.15.16:0/459085019 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558ed028a8f0 0x558ed028cce0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.207+0000 7f50e13a6740 1 --2- 172.21.15.16:0/459085019 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558ed027a790 0x558ed0278200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.207+0000 7f50e13a6740 1 --2- 172.21.15.16:0/459085019 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558ed0271bb0 0x558ed0277cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.207+0000 7f50b77fe640 0 client.16218 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:14:11.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.207+0000 7f50e13a6740 1 --2- 172.21.15.16:0/459085019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ed0271290 0x558ed0277780 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.207+0000 7f50e13a6740 1 -- 172.21.15.16:0/459085019 >> 172.21.15.16:0/459085019 conn(0x558ed026b9c0 msgr2=0x558ed026ac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:11.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.207+0000 7f50e13a6740 1 -- 172.21.15.16:0/459085019 shutdown_connections 2024-03-28T22:14:11.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.208+0000 7f50e13a6740 1 -- 172.21.15.16:0/459085019 wait complete. 2024-03-28T22:14:11.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.208+0000 7f50e13a6740 20 client.16218 trim_cache size 0 max 16384 2024-03-28T22:14:11.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.225+0000 7f50e13a6740 1 Processor -- start 2024-03-28T22:14:11.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.225+0000 7f50e13a6740 1 -- start start 2024-03-28T22:14:11.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.226+0000 7f50e13a6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ed0271290 0x558ed0279eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.226+0000 7f50e13a6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558ed0271bb0 0x558ed02862c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.226+0000 7f50e13a6740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558ed027a790 0x558ed026aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.226+0000 7f50e13a6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f50d81afbc0 con 0x558ed0271290 2024-03-28T22:14:11.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.226+0000 7f50e13a6740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f50d81ad460 con 0x558ed027a790 2024-03-28T22:14:11.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.226+0000 7f50e13a6740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f50d81ab2b0 con 0x558ed0271bb0 2024-03-28T22:14:11.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.226+0000 7f50d7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ed0271290 0x558ed0279eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.226+0000 7f50d7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ed0271290 0x558ed0279eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46212/0 (socket says 172.21.15.16:46212) 2024-03-28T22:14:11.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.226+0000 7f50d7fff640 1 -- 172.21.15.16:0/349558838 learned_addr learned my addr 172.21.15.16:0/349558838 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:11.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.226+0000 7f50dcc88640 1 --2- 172.21.15.16:0/349558838 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558ed027a790 0x558ed026aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.226+0000 7f50d77fe640 1 --2- 172.21.15.16:0/349558838 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558ed0271bb0 0x558ed02862c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.226+0000 7f50d7fff640 1 -- 172.21.15.16:0/349558838 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558ed0271bb0 msgr2=0x558ed02862c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:11.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.226+0000 7f50d7fff640 1 --2- 172.21.15.16:0/349558838 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558ed0271bb0 0x558ed02862c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.226+0000 7f50d7fff640 1 -- 172.21.15.16:0/349558838 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558ed027a790 msgr2=0x558ed026aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:11.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.226+0000 7f50d7fff640 1 --2- 172.21.15.16:0/349558838 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558ed027a790 0x558ed026aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.226+0000 7f50d7fff640 1 -- 172.21.15.16:0/349558838 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ecfea90f0 con 0x558ed0271290 2024-03-28T22:14:11.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.226+0000 7f50d7fff640 1 --2- 172.21.15.16:0/349558838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ed0271290 0x558ed0279eb0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f50cc048820 tx=0x7f50cc005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e150c1dd74a315d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:11.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.226+0000 7f50bdffb640 1 -- 172.21.15.16:0/349558838 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50cc043070 con 0x558ed0271290 2024-03-28T22:14:11.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.226+0000 7f50bdffb640 1 -- 172.21.15.16:0/349558838 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50cc02c650 con 0x558ed0271290 2024-03-28T22:14:11.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.226+0000 7f50bdffb640 1 -- 172.21.15.16:0/349558838 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50cc0052c0 con 0x558ed0271290 2024-03-28T22:14:11.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.227+0000 7f50e13a6740 1 -- 172.21.15.16:0/349558838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ed0271290 msgr2=0x558ed0279eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:11.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.227+0000 7f50e13a6740 1 --2- 172.21.15.16:0/349558838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ed0271290 0x558ed0279eb0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f50cc048820 tx=0x7f50cc005c00 comp rx=0 tx=0).stop 2024-03-28T22:14:11.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.227+0000 7f50e13a6740 1 -- 172.21.15.16:0/349558838 shutdown_connections 2024-03-28T22:14:11.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.227+0000 7f50e13a6740 1 --2- 172.21.15.16:0/349558838 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558ed027a790 0x558ed026aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.227+0000 7f50e13a6740 1 --2- 172.21.15.16:0/349558838 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558ed0271bb0 0x558ed02862c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.227+0000 7f50e13a6740 1 --2- 172.21.15.16:0/349558838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ed0271290 0x558ed0279eb0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.227+0000 7f50e13a6740 1 -- 172.21.15.16:0/349558838 >> 172.21.15.16:0/349558838 conn(0x558ed026b9c0 msgr2=0x558ed026a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:11.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.227+0000 7f50e13a6740 1 -- 172.21.15.16:0/349558838 shutdown_connections 2024-03-28T22:14:11.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.227+0000 7f50e13a6740 1 -- 172.21.15.16:0/349558838 wait complete. 2024-03-28T22:14:11.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.227+0000 7f50e13a6740 1 -- start start 2024-03-28T22:14:11.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.228+0000 7f50e13a6740 1 Processor -- start 2024-03-28T22:14:11.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.228+0000 7f50e13a6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558ed0271290 0x558ed027e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.228+0000 7f50e13a6740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558ed0271bb0 0x558ed027e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.228+0000 7f50e13a6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ed027a790 0x558ed027ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.228+0000 7f50e13a6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f50d81ab920 con 0x558ed027a790 2024-03-28T22:14:11.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.228+0000 7f50e13a6740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f50d81acd90 con 0x558ed0271bb0 2024-03-28T22:14:11.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.228+0000 7f50e13a6740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f50d81abff0 con 0x558ed0271290 2024-03-28T22:14:11.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.228+0000 7f50e13a6740 10 client.0 _ll_register_callbacks cb 0x7f50c8013440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:14:11.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.228+0000 7f50dcc88640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ed027a790 0x558ed027ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.228+0000 7f50d77fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558ed0271bb0 0x558ed027e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.228+0000 7f50dcc88640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ed027a790 0x558ed027ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46214/0 (socket says 172.21.15.16:46214) 2024-03-28T22:14:11.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.228+0000 7f50dcc88640 1 -- 172.21.15.16:0/2757374375 learned_addr learned my addr 172.21.15.16:0/2757374375 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:11.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.228+0000 7f50d7fff640 1 --2- 172.21.15.16:0/2757374375 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558ed0271290 0x558ed027e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.228+0000 7f50dcc88640 1 -- 172.21.15.16:0/2757374375 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558ed0271290 msgr2=0x558ed027e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:11.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.228+0000 7f50dcc88640 1 --2- 172.21.15.16:0/2757374375 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558ed0271290 0x558ed027e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.228+0000 7f50dcc88640 1 -- 172.21.15.16:0/2757374375 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558ed0271bb0 msgr2=0x558ed027e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:11.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.228+0000 7f50dcc88640 1 --2- 172.21.15.16:0/2757374375 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558ed0271bb0 0x558ed027e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.228+0000 7f50dcc88640 1 -- 172.21.15.16:0/2757374375 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ed02ef9b0 con 0x558ed027a790 2024-03-28T22:14:11.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.228+0000 7f50d7fff640 1 --2- 172.21.15.16:0/2757374375 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558ed0271290 0x558ed027e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:14:11.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.228+0000 7f50dcc88640 1 --2- 172.21.15.16:0/2757374375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ed027a790 0x558ed027ed50 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f50d00cbee0 tx=0x7f50d009b120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64ef307412f10d4f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:11.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.228+0000 7f50b77fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:14:11.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.229+0000 7f50b77fe640 1 -- 172.21.15.16:0/2757374375 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d00a3430 con 0x558ed027a790 2024-03-28T22:14:11.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.229+0000 7f50b77fe640 1 -- 172.21.15.16:0/2757374375 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f50d00a35d0 con 0x558ed027a790 2024-03-28T22:14:11.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.229+0000 7f50e13a6740 1 -- 172.21.15.16:0/2757374375 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50cc02c650 con 0x558ed027a790 2024-03-28T22:14:11.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.229+0000 7f50b77fe640 1 -- 172.21.15.16:0/2757374375 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f50d00b0020 con 0x558ed027a790 2024-03-28T22:14:11.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.229+0000 7f50e13a6740 10 client.16224 fetch_fsmap learned FSMap version 25 2024-03-28T22:14:11.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.229+0000 7f50e13a6740 1 -- 172.21.15.16:0/2757374375 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558ed02f0cc0 con 0x558ed027a790 2024-03-28T22:14:11.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.229+0000 7f50b77fe640 1 -- 172.21.15.16:0/2757374375 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50cc02c650 con 0x558ed027a790 2024-03-28T22:14:11.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.229+0000 7f50b77fe640 1 -- 172.21.15.16:0/2757374375 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f50d0099fe0 con 0x558ed027a790 2024-03-28T22:14:11.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.229+0000 7f50e13a6740 10 client.16224 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T22:14:11.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.229+0000 7f50e13a6740 10 client.16224 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:14:11.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.229+0000 7f50e13a6740 20 client.16224 populate_metadata read hostname 'smithi016' 2024-03-28T22:14:11.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.229+0000 7f50e13a6740 1 --2- 172.21.15.16:0/2757374375 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558ed02f1460 0x558ed02f3880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.229+0000 7f50e13a6740 4 client.16224 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:14:11.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.229+0000 7f50e13a6740 1 -- 172.21.15.16:0/2757374375 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558ed01dca10 con 0x558ed02f1460 2024-03-28T22:14:11.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.229+0000 7f50d7fff640 1 --2- 172.21.15.16:0/2757374375 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558ed02f1460 0x558ed02f3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.230+0000 7f50b77fe640 10 client.16224 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:14:11.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.230+0000 7f50d7fff640 1 --2- 172.21.15.16:0/2757374375 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558ed02f1460 0x558ed02f3880 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e148aafd4e8d971 server_cookie=9c96a935dc25f7c6 in_seq=0 out_seq=0 2024-03-28T22:14:11.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.230+0000 7f50b77fe640 1 -- 172.21.15.16:0/2757374375 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x558ed01dca10 con 0x558ed02f1460 2024-03-28T22:14:11.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.230+0000 7f50b77fe640 10 client.16224 handle_command_reply: tid=0 2024-03-28T22:14:11.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.230+0000 7f50e13a6740 1 client.16224 shutdown 2024-03-28T22:14:11.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.230+0000 7f50e13a6740 1 -- 172.21.15.16:0/2757374375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ed027a790 msgr2=0x558ed027ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:11.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.230+0000 7f50e13a6740 1 --2- 172.21.15.16:0/2757374375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ed027a790 0x558ed027ed50 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f50d00cbee0 tx=0x7f50d009b120 comp rx=0 tx=0).stop 2024-03-28T22:14:11.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.230+0000 7f50e13a6740 1 -- 172.21.15.16:0/2757374375 shutdown_connections 2024-03-28T22:14:11.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.230+0000 7f50e13a6740 1 --2- 172.21.15.16:0/2757374375 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558ed02f1460 0x558ed02f3880 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.230+0000 7f50e13a6740 1 --2- 172.21.15.16:0/2757374375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ed027a790 0x558ed027ed50 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.230+0000 7f50e13a6740 1 --2- 172.21.15.16:0/2757374375 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558ed0271bb0 0x558ed027e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.230+0000 7f50e13a6740 1 --2- 172.21.15.16:0/2757374375 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558ed0271290 0x558ed027e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.230+0000 7f50b77fe640 0 client.16224 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:14:11.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.230+0000 7f50e13a6740 1 -- 172.21.15.16:0/2757374375 >> 172.21.15.16:0/2757374375 conn(0x558ed026b9c0 msgr2=0x558ed026ac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:11.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.230+0000 7f50e13a6740 1 -- 172.21.15.16:0/2757374375 shutdown_connections 2024-03-28T22:14:11.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.230+0000 7f50e13a6740 1 -- 172.21.15.16:0/2757374375 wait complete. 2024-03-28T22:14:11.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.230+0000 7f50e13a6740 20 client.16224 trim_cache size 0 max 16384 2024-03-28T22:14:11.243 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:14:11.243 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-03-28T22:14:11.243 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-03-28T22:14:11.244 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:14:11.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.231+0000 7f50de6eb640 1 -- 172.21.15.16:0/3235246960 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f50c00a1ef0 msgr2=0x7f50c00a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:11.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.231+0000 7f50de6eb640 1 --2- 172.21.15.16:0/3235246960 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f50c00a1ef0 0x7f50c00a43b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f50c8005fd0 tx=0x7f50c800ab60 comp rx=0 tx=0).stop 2024-03-28T22:14:11.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.231+0000 7f50de6eb640 1 -- 172.21.15.16:0/3235246960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d8102b80 msgr2=0x7f50d819c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:11.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.231+0000 7f50de6eb640 1 --2- 172.21.15.16:0/3235246960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d8102b80 0x7f50d819c900 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f50d000ea60 tx=0x7f50d000ef20 comp rx=0 tx=0).stop 2024-03-28T22:14:11.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.231+0000 7f50de6eb640 1 -- 172.21.15.16:0/3235246960 shutdown_connections 2024-03-28T22:14:11.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.231+0000 7f50de6eb640 1 --2- 172.21.15.16:0/3235246960 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f50c00a1ef0 0x7f50c00a43b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.231+0000 7f50de6eb640 1 --2- 172.21.15.16:0/3235246960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d8102b80 0x7f50d819c900 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.231+0000 7f50de6eb640 1 --2- 172.21.15.16:0/3235246960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50d81021e0 0x7f50d8198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.231+0000 7f50de6eb640 1 --2- 172.21.15.16:0/3235246960 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50d81009d0 0x7f50d8198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.231+0000 7f50de6eb640 1 -- 172.21.15.16:0/3235246960 >> 172.21.15.16:0/3235246960 conn(0x7f50d8077830 msgr2=0x7f50d806a0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:11.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.231+0000 7f50de6eb640 1 -- 172.21.15.16:0/3235246960 shutdown_connections 2024-03-28T22:14:11.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.231+0000 7f50de6eb640 1 -- 172.21.15.16:0/3235246960 wait complete. 2024-03-28T22:14:11.247 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:14:11.247 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:14:11.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.348+0000 7fe5cf4cd640 1 -- 172.21.15.16:0/3973994523 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5c8102970 msgr2=0x7fe5c8102d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:11.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.348+0000 7fe5cf4cd640 1 --2- 172.21.15.16:0/3973994523 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5c8102970 0x7fe5c8102d50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe5c4009900 tx=0x7fe5c402bcd0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.348+0000 7fe5cf4cd640 1 -- 172.21.15.16:0/3973994523 shutdown_connections 2024-03-28T22:14:11.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.348+0000 7fe5cf4cd640 1 --2- 172.21.15.16:0/3973994523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5c810bd40 0x7fe5c810e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.348+0000 7fe5cf4cd640 1 --2- 172.21.15.16:0/3973994523 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe5c8103290 0x7fe5c810b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.348+0000 7fe5cf4cd640 1 --2- 172.21.15.16:0/3973994523 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5c8102970 0x7fe5c8102d50 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.348+0000 7fe5cf4cd640 1 -- 172.21.15.16:0/3973994523 >> 172.21.15.16:0/3973994523 conn(0x7fe5c80777d0 msgr2=0x7fe5c8077be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:11.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.348+0000 7fe5cf4cd640 1 -- 172.21.15.16:0/3973994523 shutdown_connections 2024-03-28T22:14:11.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.348+0000 7fe5cf4cd640 1 -- 172.21.15.16:0/3973994523 wait complete. 2024-03-28T22:14:11.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.349+0000 7fe5cf4cd640 1 Processor -- start 2024-03-28T22:14:11.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.349+0000 7fe5cf4cd640 1 -- start start 2024-03-28T22:14:11.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.349+0000 7fe5cf4cd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5c8102970 0x7fe5c81980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.349+0000 7fe5cf4cd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe5c8103290 0x7fe5c81985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.349+0000 7fe5cf4cd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5c810bd40 0x7fe5c819c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.349+0000 7fe5cf4cd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c8111ae0 con 0x7fe5c810bd40 2024-03-28T22:14:11.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.349+0000 7fe5cf4cd640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c8111960 con 0x7fe5c8102970 2024-03-28T22:14:11.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.349+0000 7fe5cf4cd640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c8111c60 con 0x7fe5c8103290 2024-03-28T22:14:11.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.349+0000 7fe5cda6a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5c810bd40 0x7fe5c819c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.349+0000 7fe5cd269640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5c8102970 0x7fe5c81980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.349+0000 7fe5cda6a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5c810bd40 0x7fe5c819c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46234/0 (socket says 172.21.15.16:46234) 2024-03-28T22:14:11.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.349+0000 7fe5cda6a640 1 -- 172.21.15.16:0/1997784913 learned_addr learned my addr 172.21.15.16:0/1997784913 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:11.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.349+0000 7fe5cca68640 1 --2- 172.21.15.16:0/1997784913 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe5c8103290 0x7fe5c81985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.350+0000 7fe5cda6a640 1 -- 172.21.15.16:0/1997784913 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe5c8103290 msgr2=0x7fe5c81985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:11.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.350+0000 7fe5cda6a640 1 --2- 172.21.15.16:0/1997784913 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe5c8103290 0x7fe5c81985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.350+0000 7fe5cda6a640 1 -- 172.21.15.16:0/1997784913 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5c8102970 msgr2=0x7fe5c81980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:11.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.350+0000 7fe5cda6a640 1 --2- 172.21.15.16:0/1997784913 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5c8102970 0x7fe5c81980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.350+0000 7fe5cda6a640 1 -- 172.21.15.16:0/1997784913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5c819d090 con 0x7fe5c810bd40 2024-03-28T22:14:11.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.350+0000 7fe5cda6a640 1 --2- 172.21.15.16:0/1997784913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5c810bd40 0x7fe5c819c960 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fe5c000eb10 tx=0x7fe5c000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:11.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.350+0000 7fe5be7fc640 1 -- 172.21.15.16:0/1997784913 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c0004b20 con 0x7fe5c810bd40 2024-03-28T22:14:11.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.350+0000 7fe5cf4cd640 1 -- 172.21.15.16:0/1997784913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5c819d380 con 0x7fe5c810bd40 2024-03-28T22:14:11.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.350+0000 7fe5be7fc640 1 -- 172.21.15.16:0/1997784913 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5c0004cc0 con 0x7fe5c810bd40 2024-03-28T22:14:11.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.350+0000 7fe5be7fc640 1 -- 172.21.15.16:0/1997784913 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c0010760 con 0x7fe5c810bd40 2024-03-28T22:14:11.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.350+0000 7fe5cf4cd640 1 -- 172.21.15.16:0/1997784913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5c81a4c10 con 0x7fe5c810bd40 2024-03-28T22:14:11.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.351+0000 7fe5cf4cd640 1 -- 172.21.15.16:0/1997784913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5c8068a20 con 0x7fe5c810bd40 2024-03-28T22:14:11.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.352+0000 7fe5be7fc640 1 -- 172.21.15.16:0/1997784913 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7fe5c0005020 con 0x7fe5c810bd40 2024-03-28T22:14:11.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.352+0000 7fe5be7fc640 1 --2- 172.21.15.16:0/1997784913 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fe5b40a1ef0 0x7fe5b40a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.353+0000 7fe5cd269640 1 --2- 172.21.15.16:0/1997784913 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fe5b40a1ef0 0x7fe5b40a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.353+0000 7fe5be7fc640 1 -- 172.21.15.16:0/1997784913 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fe5c00ce7d0 con 0x7fe5c810bd40 2024-03-28T22:14:11.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.353+0000 7fe5cd269640 1 --2- 172.21.15.16:0/1997784913 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fe5b40a1ef0 0x7fe5b40a43b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe5c402efd0 tx=0x7fe5c402cf90 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:11.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.357+0000 7fe5be7fc640 1 -- 172.21.15.16:0/1997784913 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe5c009b880 con 0x7fe5c810bd40 2024-03-28T22:14:11.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.578+0000 7fe5cf4cd640 1 -- 172.21.15.16:0/1997784913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe5c8106e90 con 0x7fe5c810bd40 2024-03-28T22:14:11.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.579+0000 7fe5be7fc640 1 -- 172.21.15.16:0/1997784913 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11853 (secure 0 0 0) 0x7fe5c009b180 con 0x7fe5c810bd40 2024-03-28T22:14:11.581 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:14:11.581 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/1644581227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1644581227},{"type":"v1","addr":"172.21.15.16:6839","nonce":1644581227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/2593062669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2593062669},{"type":"v1","addr":"172.21.15.16:6841","nonce":2593062669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:08:29.672976+0000","modified":"2024-03-28T22:13:59.921432+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24566,"mds_1":24524,"mds_2":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24466":{"gid":24466,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/247249857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":247249857},{"type":"v1","addr":"172.21.15.63:6835","nonce":247249857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6837/2324231032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":2324231032},{"type":"v1","addr":"172.21.15.63:6837","nonce":2324231032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6839/1177197077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":1177197077},{"type":"v1","addr":"172.21.15.63:6839","nonce":1177197077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"e","rank":2,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.53:6835/350727634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":350727634},{"type":"v1","addr":"172.21.15.53:6835","nonce":350727634}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/349877573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":349877573},{"type":"v1","addr":"172.21.15.53:6837","nonce":349877573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6839/3714153956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":3714153956},{"type":"v1","addr":"172.21.15.53:6839","nonce":3714153956}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24566,"qdb_cluster":[24500,24524,24566]},"id":1}]} 2024-03-28T22:14:11.581 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 25 2024-03-28T22:14:11.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.581+0000 7fe5cf4cd640 1 -- 172.21.15.16:0/1997784913 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fe5b40a1ef0 msgr2=0x7fe5b40a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:11.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.581+0000 7fe5cf4cd640 1 --2- 172.21.15.16:0/1997784913 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fe5b40a1ef0 0x7fe5b40a43b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe5c402efd0 tx=0x7fe5c402cf90 comp rx=0 tx=0).stop 2024-03-28T22:14:11.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.581+0000 7fe5cf4cd640 1 -- 172.21.15.16:0/1997784913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5c810bd40 msgr2=0x7fe5c819c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:11.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.581+0000 7fe5cf4cd640 1 --2- 172.21.15.16:0/1997784913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5c810bd40 0x7fe5c819c960 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fe5c000eb10 tx=0x7fe5c000c2d0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.581+0000 7fe5cf4cd640 1 -- 172.21.15.16:0/1997784913 shutdown_connections 2024-03-28T22:14:11.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.581+0000 7fe5cf4cd640 1 --2- 172.21.15.16:0/1997784913 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fe5b40a1ef0 0x7fe5b40a43b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.581+0000 7fe5cf4cd640 1 --2- 172.21.15.16:0/1997784913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5c810bd40 0x7fe5c819c960 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.581+0000 7fe5cf4cd640 1 --2- 172.21.15.16:0/1997784913 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe5c8103290 0x7fe5c81985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.581+0000 7fe5cf4cd640 1 --2- 172.21.15.16:0/1997784913 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5c8102970 0x7fe5c81980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.581+0000 7fe5cf4cd640 1 -- 172.21.15.16:0/1997784913 >> 172.21.15.16:0/1997784913 conn(0x7fe5c80777d0 msgr2=0x7fe5c80feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:11.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.581+0000 7fe5cf4cd640 1 -- 172.21.15.16:0/1997784913 shutdown_connections 2024-03-28T22:14:11.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.581+0000 7fe5cf4cd640 1 -- 172.21.15.16:0/1997784913 wait complete. 2024-03-28T22:14:11.594 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:14:11.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.697+0000 7f03568c0640 1 -- 172.21.15.16:0/400811090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0350103290 msgr2=0x7f035010b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:11.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.697+0000 7f03568c0640 1 --2- 172.21.15.16:0/400811090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0350103290 0x7f035010b770 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f0340009900 tx=0x7f034002bcd0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.697+0000 7f03568c0640 1 -- 172.21.15.16:0/400811090 shutdown_connections 2024-03-28T22:14:11.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.697+0000 7f03568c0640 1 --2- 172.21.15.16:0/400811090 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f035010bd40 0x7f035010e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.697+0000 7f03568c0640 1 --2- 172.21.15.16:0/400811090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0350103290 0x7f035010b770 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.697+0000 7f03568c0640 1 --2- 172.21.15.16:0/400811090 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0350102970 0x7f0350102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.697+0000 7f03568c0640 1 -- 172.21.15.16:0/400811090 >> 172.21.15.16:0/400811090 conn(0x7f0350077800 msgr2=0x7f0350077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:11.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.697+0000 7f03568c0640 1 -- 172.21.15.16:0/400811090 shutdown_connections 2024-03-28T22:14:11.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.697+0000 7f03568c0640 1 -- 172.21.15.16:0/400811090 wait complete. 2024-03-28T22:14:11.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.698+0000 7f03568c0640 1 Processor -- start 2024-03-28T22:14:11.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.698+0000 7f03568c0640 1 -- start start 2024-03-28T22:14:11.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.698+0000 7f03568c0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0350102970 0x7f0350198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.698+0000 7f03568c0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0350103290 0x7f0350198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.698+0000 7f03568c0640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f035010bd40 0x7f035019c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.698+0000 7f03568c0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0350111b20 con 0x7f0350103290 2024-03-28T22:14:11.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.698+0000 7f03568c0640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f03501119a0 con 0x7f035010bd40 2024-03-28T22:14:11.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.698+0000 7f03568c0640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0350111ca0 con 0x7f0350102970 2024-03-28T22:14:11.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.698+0000 7f034f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0350103290 0x7f0350198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.698+0000 7f034f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0350103290 0x7f0350198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46254/0 (socket says 172.21.15.16:46254) 2024-03-28T22:14:11.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.698+0000 7f034f7fe640 1 -- 172.21.15.16:0/2240227327 learned_addr learned my addr 172.21.15.16:0/2240227327 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:11.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.698+0000 7f0354e5d640 1 --2- 172.21.15.16:0/2240227327 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f035010bd40 0x7f035019c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.699+0000 7f034ffff640 1 --2- 172.21.15.16:0/2240227327 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0350102970 0x7f0350198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.699+0000 7f034f7fe640 1 -- 172.21.15.16:0/2240227327 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0350102970 msgr2=0x7f0350198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:11.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.699+0000 7f034f7fe640 1 --2- 172.21.15.16:0/2240227327 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0350102970 0x7f0350198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.699+0000 7f034f7fe640 1 -- 172.21.15.16:0/2240227327 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f035010bd40 msgr2=0x7f035019c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:11.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.699+0000 7f034f7fe640 1 --2- 172.21.15.16:0/2240227327 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f035010bd40 0x7f035019c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.699+0000 7f034f7fe640 1 -- 172.21.15.16:0/2240227327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f035019cfa0 con 0x7f0350103290 2024-03-28T22:14:11.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.699+0000 7f034f7fe640 1 --2- 172.21.15.16:0/2240227327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0350103290 0x7f0350198540 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f0340000c00 tx=0x7f03400043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:11.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.699+0000 7f034d7fa640 1 -- 172.21.15.16:0/2240227327 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0340042020 con 0x7f0350103290 2024-03-28T22:14:11.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.699+0000 7f034d7fa640 1 -- 172.21.15.16:0/2240227327 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03400054e0 con 0x7f0350103290 2024-03-28T22:14:11.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.699+0000 7f034d7fa640 1 -- 172.21.15.16:0/2240227327 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0340003b20 con 0x7f0350103290 2024-03-28T22:14:11.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.699+0000 7f03568c0640 1 -- 172.21.15.16:0/2240227327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f035019d230 con 0x7f0350103290 2024-03-28T22:14:11.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.699+0000 7f03568c0640 1 -- 172.21.15.16:0/2240227327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03501a4b20 con 0x7f0350103290 2024-03-28T22:14:11.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.702+0000 7f034d7fa640 1 -- 172.21.15.16:0/2240227327 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f0340045020 con 0x7f0350103290 2024-03-28T22:14:11.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.702+0000 7f035957a740 1 Processor -- start 2024-03-28T22:14:11.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.702+0000 7f035957a740 1 -- start start 2024-03-28T22:14:11.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.702+0000 7f035957a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55720427b470 0x55720427b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.702+0000 7f035957a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55720427bd90 0x557204284430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.702+0000 7f035957a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557204284970 0x557204286d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.702+0000 7f035957a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5572041df1b0 con 0x55720427bd90 2024-03-28T22:14:11.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.702+0000 7f035957a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x557204187c00 con 0x55720427b470 2024-03-28T22:14:11.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.702+0000 7f035957a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5572041db250 con 0x557204284970 2024-03-28T22:14:11.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.702+0000 7f034ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55720427bd90 0x557204284430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.702+0000 7f0354e5d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55720427b470 0x55720427b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.702+0000 7f0354e5d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55720427b470 0x55720427b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:55898/0 (socket says 172.21.15.16:55898) 2024-03-28T22:14:11.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.702+0000 7f034ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55720427bd90 0x557204284430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46268/0 (socket says 172.21.15.16:46268) 2024-03-28T22:14:11.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.702+0000 7f0354e5d640 1 -- 172.21.15.16:0/570372820 learned_addr learned my addr 172.21.15.16:0/570372820 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:11.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.703+0000 7f034d7fa640 1 --2- 172.21.15.16:0/2240227327 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f03380a1ef0 0x7f03380a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.703+0000 7f034f7fe640 1 --2- 172.21.15.16:0/570372820 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557204284970 0x557204286d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.703+0000 7f034ffff640 1 -- 172.21.15.16:0/570372820 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557204284970 msgr2=0x557204286d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:11.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.703+0000 7f034ffff640 1 --2- 172.21.15.16:0/570372820 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557204284970 0x557204286d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.703+0000 7f034ffff640 1 -- 172.21.15.16:0/570372820 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55720427b470 msgr2=0x55720427b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:11.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.703+0000 7f034ffff640 1 --2- 172.21.15.16:0/570372820 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55720427b470 0x55720427b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.703+0000 7f034ffff640 1 -- 172.21.15.16:0/570372820 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572041a73d0 con 0x55720427bd90 2024-03-28T22:14:11.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.703+0000 7f0354e5d640 1 --2- 172.21.15.16:0/570372820 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55720427b470 0x55720427b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:14:11.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.703+0000 7f0354e5d640 1 --2- 172.21.15.16:0/2240227327 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f03380a1ef0 0x7f03380a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.703+0000 7f034d7fa640 1 -- 172.21.15.16:0/2240227327 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f03400efe30 con 0x7f0350103290 2024-03-28T22:14:11.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.703+0000 7f034ffff640 1 --2- 172.21.15.16:0/570372820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55720427bd90 0x557204284430 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f0344009a90 tx=0x7f0344009f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=803519a7bb7c6ff8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:11.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.703+0000 7f03367fc640 1 -- 172.21.15.16:0/570372820 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0344010d30 con 0x55720427bd90 2024-03-28T22:14:11.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.703+0000 7f03367fc640 1 -- 172.21.15.16:0/570372820 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0344005500 con 0x55720427bd90 2024-03-28T22:14:11.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.703+0000 7f03367fc640 1 -- 172.21.15.16:0/570372820 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0344012650 con 0x55720427bd90 2024-03-28T22:14:11.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.704+0000 7f035957a740 1 -- 172.21.15.16:0/570372820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55720427bd90 msgr2=0x557204284430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:11.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.704+0000 7f0354e5d640 1 --2- 172.21.15.16:0/2240227327 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f03380a1ef0 0x7f03380a43b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0348005f40 tx=0x7f0348004a40 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:11.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.704+0000 7f035957a740 1 --2- 172.21.15.16:0/570372820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55720427bd90 0x557204284430 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f0344009a90 tx=0x7f0344009f50 comp rx=0 tx=0).stop 2024-03-28T22:14:11.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.704+0000 7f035957a740 1 -- 172.21.15.16:0/570372820 shutdown_connections 2024-03-28T22:14:11.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.704+0000 7f035957a740 1 --2- 172.21.15.16:0/570372820 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557204284970 0x557204286d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.704+0000 7f035957a740 1 --2- 172.21.15.16:0/570372820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55720427bd90 0x557204284430 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.704+0000 7f035957a740 1 --2- 172.21.15.16:0/570372820 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55720427b470 0x55720427b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.704+0000 7f035957a740 1 -- 172.21.15.16:0/570372820 >> 172.21.15.16:0/570372820 conn(0x557204275ba0 msgr2=0x557204275f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:11.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.704+0000 7f035957a740 1 -- 172.21.15.16:0/570372820 shutdown_connections 2024-03-28T22:14:11.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.704+0000 7f035957a740 1 -- 172.21.15.16:0/570372820 wait complete. 2024-03-28T22:14:11.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.704+0000 7f035957a740 1 -- start start 2024-03-28T22:14:11.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.704+0000 7f035957a740 1 Processor -- start 2024-03-28T22:14:11.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.705+0000 7f035957a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55720427b470 0x5572042819f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.705+0000 7f035957a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55720427bd90 0x557204281f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.705+0000 7f035957a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557204284970 0x557204282470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.705+0000 7f035957a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557204287760 con 0x55720427b470 2024-03-28T22:14:11.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.705+0000 7f035957a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55720417d470 con 0x55720427bd90 2024-03-28T22:14:11.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.705+0000 7f035957a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5572042878e0 con 0x557204284970 2024-03-28T22:14:11.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.705+0000 7f035957a740 10 client.0 _ll_register_callbacks cb 0x55720419acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:14:11.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.705+0000 7f034ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55720427b470 0x5572042819f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.705+0000 7f034ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55720427b470 0x5572042819f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46278/0 (socket says 172.21.15.16:46278) 2024-03-28T22:14:11.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.705+0000 7f034ffff640 1 -- 172.21.15.16:0/2024115605 learned_addr learned my addr 172.21.15.16:0/2024115605 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:11.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.705+0000 7f034f7fe640 1 --2- 172.21.15.16:0/2024115605 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55720427bd90 0x557204281f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.705+0000 7f034ffff640 1 -- 172.21.15.16:0/2024115605 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557204284970 msgr2=0x557204282470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:11.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.705+0000 7f034ffff640 1 --2- 172.21.15.16:0/2024115605 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557204284970 0x557204282470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.705+0000 7f034ffff640 1 -- 172.21.15.16:0/2024115605 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55720427bd90 msgr2=0x557204281f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:11.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.705+0000 7f034ffff640 1 --2- 172.21.15.16:0/2024115605 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55720427bd90 0x557204281f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.705+0000 7f034ffff640 1 -- 172.21.15.16:0/2024115605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557203ef33f0 con 0x55720427b470 2024-03-28T22:14:11.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.705+0000 7f034ffff640 1 --2- 172.21.15.16:0/2024115605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55720427b470 0x5572042819f0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f0344010800 tx=0x7f03440052d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb6c26cf252285fc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:11.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.705+0000 7f032f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:14:11.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.705+0000 7f032f7fe640 1 -- 172.21.15.16:0/2024115605 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0344012450 con 0x55720427b470 2024-03-28T22:14:11.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.705+0000 7f032f7fe640 1 -- 172.21.15.16:0/2024115605 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f03440125f0 con 0x55720427b470 2024-03-28T22:14:11.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.705+0000 7f035957a740 1 -- 172.21.15.16:0/2024115605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0344005500 con 0x55720427b470 2024-03-28T22:14:11.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.706+0000 7f032f7fe640 1 -- 172.21.15.16:0/2024115605 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f0344012810 con 0x55720427b470 2024-03-28T22:14:11.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.706+0000 7f035957a740 10 client.16242 fetch_fsmap learned FSMap version 25 2024-03-28T22:14:11.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.706+0000 7f035957a740 1 -- 172.21.15.16:0/2024115605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x557203fbd7d0 con 0x55720427b470 2024-03-28T22:14:11.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.706+0000 7f032f7fe640 1 -- 172.21.15.16:0/2024115605 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0344005500 con 0x55720427b470 2024-03-28T22:14:11.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.706+0000 7f032f7fe640 1 -- 172.21.15.16:0/2024115605 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f034402d020 con 0x55720427b470 2024-03-28T22:14:11.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.706+0000 7f035957a740 10 client.16242 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T22:14:11.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.706+0000 7f035957a740 10 client.16242 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-28T22:14:11.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.706+0000 7f035957a740 20 client.16242 populate_metadata read hostname 'smithi016' 2024-03-28T22:14:11.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.706+0000 7f035957a740 1 --2- 172.21.15.16:0/2024115605 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x557204294950 0x557204296d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.706+0000 7f035957a740 4 client.16242 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:14:11.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.706+0000 7f035957a740 1 -- 172.21.15.16:0/2024115605 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5572041ec6f0 con 0x557204294950 2024-03-28T22:14:11.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.706+0000 7f034f7fe640 1 --2- 172.21.15.16:0/2024115605 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x557204294950 0x557204296d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.707+0000 7f032f7fe640 10 client.16242 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:14:11.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.707+0000 7f034f7fe640 1 --2- 172.21.15.16:0/2024115605 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x557204294950 0x557204296d40 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d8613c9cce234ad3 server_cookie=65b86b90465b7835 in_seq=0 out_seq=0 2024-03-28T22:14:11.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.708+0000 7f032f7fe640 1 -- 172.21.15.16:0/2024115605 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5572041ec6f0 con 0x557204294950 2024-03-28T22:14:11.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.708+0000 7f032f7fe640 10 client.16242 handle_command_reply: tid=0 2024-03-28T22:14:11.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.708+0000 7f035957a740 1 client.16242 shutdown 2024-03-28T22:14:11.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.708+0000 7f035957a740 1 -- 172.21.15.16:0/2024115605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55720427b470 msgr2=0x5572042819f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:11.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.708+0000 7f035957a740 1 --2- 172.21.15.16:0/2024115605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55720427b470 0x5572042819f0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f0344010800 tx=0x7f03440052d0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.708+0000 7f035957a740 1 -- 172.21.15.16:0/2024115605 shutdown_connections 2024-03-28T22:14:11.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.708+0000 7f035957a740 1 --2- 172.21.15.16:0/2024115605 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x557204294950 0x557204296d40 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.708+0000 7f035957a740 1 --2- 172.21.15.16:0/2024115605 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557204284970 0x557204282470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.708+0000 7f032f7fe640 0 client.16242 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:14:11.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.708+0000 7f035957a740 1 --2- 172.21.15.16:0/2024115605 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55720427bd90 0x557204281f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.708+0000 7f035957a740 1 --2- 172.21.15.16:0/2024115605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55720427b470 0x5572042819f0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.708+0000 7f035957a740 1 -- 172.21.15.16:0/2024115605 >> 172.21.15.16:0/2024115605 conn(0x557204275ba0 msgr2=0x5572042778f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:11.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.708+0000 7f035957a740 1 -- 172.21.15.16:0/2024115605 shutdown_connections 2024-03-28T22:14:11.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.708+0000 7f035957a740 1 -- 172.21.15.16:0/2024115605 wait complete. 2024-03-28T22:14:11.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.708+0000 7f035957a740 20 client.16242 trim_cache size 0 max 16384 2024-03-28T22:14:11.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.721+0000 7f035957a740 1 Processor -- start 2024-03-28T22:14:11.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.721+0000 7f035957a740 1 -- start start 2024-03-28T22:14:11.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.721+0000 7f035957a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55720427b470 0x557204284120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.721+0000 7f035957a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55720427bd90 0x557204290500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.721+0000 7f035957a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557204284970 0x55720429c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.721+0000 7f035957a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f03501afcd0 con 0x55720427b470 2024-03-28T22:14:11.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.721+0000 7f035957a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f03501ad570 con 0x55720427bd90 2024-03-28T22:14:11.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.721+0000 7f035957a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f03501ab3c0 con 0x557204284970 2024-03-28T22:14:11.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.721+0000 7f034ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55720427b470 0x557204284120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.721+0000 7f034ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55720427b470 0x557204284120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46280/0 (socket says 172.21.15.16:46280) 2024-03-28T22:14:11.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.721+0000 7f034ffff640 1 -- 172.21.15.16:0/3359439806 learned_addr learned my addr 172.21.15.16:0/3359439806 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:11.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.721+0000 7f034f7fe640 1 --2- 172.21.15.16:0/3359439806 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55720427bd90 0x557204290500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.722+0000 7f034ffff640 1 -- 172.21.15.16:0/3359439806 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557204284970 msgr2=0x55720429c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:11.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.722+0000 7f034ffff640 1 --2- 172.21.15.16:0/3359439806 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557204284970 0x55720429c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.722+0000 7f034ffff640 1 -- 172.21.15.16:0/3359439806 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55720427bd90 msgr2=0x557204290500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:11.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.722+0000 7f034ffff640 1 --2- 172.21.15.16:0/3359439806 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55720427bd90 0x557204290500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.722+0000 7f034ffff640 1 -- 172.21.15.16:0/3359439806 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55720427bbc0 con 0x55720427b470 2024-03-28T22:14:11.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.722+0000 7f034ffff640 1 --2- 172.21.15.16:0/3359439806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55720427b470 0x557204284120 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f034402f7d0 tx=0x7f034402f4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b116ae21cbad855 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:11.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.722+0000 7f0335ffb640 1 -- 172.21.15.16:0/3359439806 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034402f8a0 con 0x55720427b470 2024-03-28T22:14:11.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.722+0000 7f0335ffb640 1 -- 172.21.15.16:0/3359439806 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0344009370 con 0x55720427b470 2024-03-28T22:14:11.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.722+0000 7f0335ffb640 1 -- 172.21.15.16:0/3359439806 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034402fde0 con 0x55720427b470 2024-03-28T22:14:11.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.722+0000 7f035957a740 1 -- 172.21.15.16:0/3359439806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55720427b470 msgr2=0x557204284120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:11.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.722+0000 7f035957a740 1 --2- 172.21.15.16:0/3359439806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55720427b470 0x557204284120 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f034402f7d0 tx=0x7f034402f4f0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.722+0000 7f035957a740 1 -- 172.21.15.16:0/3359439806 shutdown_connections 2024-03-28T22:14:11.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.722+0000 7f035957a740 1 --2- 172.21.15.16:0/3359439806 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557204284970 0x55720429c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.722+0000 7f035957a740 1 --2- 172.21.15.16:0/3359439806 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55720427bd90 0x557204290500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.722+0000 7f035957a740 1 --2- 172.21.15.16:0/3359439806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55720427b470 0x557204284120 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.722+0000 7f035957a740 1 -- 172.21.15.16:0/3359439806 >> 172.21.15.16:0/3359439806 conn(0x557204275ba0 msgr2=0x557204274d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:11.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.723+0000 7f035957a740 1 -- 172.21.15.16:0/3359439806 shutdown_connections 2024-03-28T22:14:11.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.723+0000 7f035957a740 1 -- 172.21.15.16:0/3359439806 wait complete. 2024-03-28T22:14:11.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.723+0000 7f035957a740 1 -- start start 2024-03-28T22:14:11.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.723+0000 7f035957a740 1 Processor -- start 2024-03-28T22:14:11.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.723+0000 7f035957a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55720427b470 0x55720428ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.723+0000 7f035957a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55720427bd90 0x55720427e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.723+0000 7f035957a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557204284970 0x55720427e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.723+0000 7f035957a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f03501aba30 con 0x55720427b470 2024-03-28T22:14:11.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.723+0000 7f035957a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f03501acea0 con 0x557204284970 2024-03-28T22:14:11.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.723+0000 7f035957a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f03501ac100 con 0x55720427bd90 2024-03-28T22:14:11.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.723+0000 7f035957a740 10 client.0 _ll_register_callbacks cb 0x7f0344005850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:14:11.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.723+0000 7f034ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55720427b470 0x55720428ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.723+0000 7f034ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55720427b470 0x55720428ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46284/0 (socket says 172.21.15.16:46284) 2024-03-28T22:14:11.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.723+0000 7f034ffff640 1 -- 172.21.15.16:0/3213149377 learned_addr learned my addr 172.21.15.16:0/3213149377 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:11.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.723+0000 7f0354e5d640 1 --2- 172.21.15.16:0/3213149377 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557204284970 0x55720427e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.723+0000 7f034f7fe640 1 --2- 172.21.15.16:0/3213149377 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55720427bd90 0x55720427e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.724+0000 7f034ffff640 1 -- 172.21.15.16:0/3213149377 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55720427bd90 msgr2=0x55720427e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:11.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.724+0000 7f034ffff640 1 --2- 172.21.15.16:0/3213149377 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55720427bd90 0x55720427e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.724+0000 7f034ffff640 1 -- 172.21.15.16:0/3213149377 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557204284970 msgr2=0x55720427e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:11.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.724+0000 7f034ffff640 1 --2- 172.21.15.16:0/3213149377 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557204284970 0x55720427e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.724+0000 7f034ffff640 1 -- 172.21.15.16:0/3213149377 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55720427f0b0 con 0x55720427b470 2024-03-28T22:14:11.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.724+0000 7f034f7fe640 1 --2- 172.21.15.16:0/3213149377 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55720427bd90 0x55720427e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:14:11.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.724+0000 7f034ffff640 1 --2- 172.21.15.16:0/3213149377 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55720427b470 0x55720428ce20 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f03440109e0 tx=0x7f0344009590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=932a050d0abd7f41 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:11.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.724+0000 7f032f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:14:11.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.724+0000 7f032f7fe640 1 -- 172.21.15.16:0/3213149377 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034401e070 con 0x55720427b470 2024-03-28T22:14:11.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.724+0000 7f032f7fe640 1 -- 172.21.15.16:0/3213149377 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f0344004a70 con 0x55720427b470 2024-03-28T22:14:11.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.724+0000 7f035957a740 1 -- 172.21.15.16:0/3213149377 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0344009370 con 0x55720427b470 2024-03-28T22:14:11.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.724+0000 7f032f7fe640 1 -- 172.21.15.16:0/3213149377 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f0344004c70 con 0x55720427b470 2024-03-28T22:14:11.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.724+0000 7f035957a740 10 client.16248 fetch_fsmap learned FSMap version 25 2024-03-28T22:14:11.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.724+0000 7f035957a740 1 -- 172.21.15.16:0/3213149377 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5572042faa80 con 0x55720427b470 2024-03-28T22:14:11.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.724+0000 7f032f7fe640 1 -- 172.21.15.16:0/3213149377 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0344009370 con 0x55720427b470 2024-03-28T22:14:11.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.724+0000 7f032f7fe640 1 -- 172.21.15.16:0/3213149377 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f0344048020 con 0x55720427b470 2024-03-28T22:14:11.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.724+0000 7f035957a740 10 client.16248 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T22:14:11.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.724+0000 7f035957a740 10 client.16248 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-28T22:14:11.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.724+0000 7f035957a740 20 client.16248 populate_metadata read hostname 'smithi016' 2024-03-28T22:14:11.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.724+0000 7f035957a740 1 --2- 172.21.15.16:0/3213149377 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x5572042fae90 0x5572042fd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.724+0000 7f035957a740 4 client.16248 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:14:11.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.724+0000 7f035957a740 1 -- 172.21.15.16:0/3213149377 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5572041e6a40 con 0x5572042fae90 2024-03-28T22:14:11.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.725+0000 7f034f7fe640 1 --2- 172.21.15.16:0/3213149377 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x5572042fae90 0x5572042fd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.725+0000 7f034f7fe640 1 --2- 172.21.15.16:0/3213149377 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x5572042fae90 0x5572042fd2b0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=176a5e53b30499f4 server_cookie=c42f64dc765356f7 in_seq=0 out_seq=0 2024-03-28T22:14:11.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.725+0000 7f032f7fe640 10 client.16248 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:14:11.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.725+0000 7f032f7fe640 1 -- 172.21.15.16:0/3213149377 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5572041e6a40 con 0x5572042fae90 2024-03-28T22:14:11.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.725+0000 7f032f7fe640 10 client.16248 handle_command_reply: tid=0 2024-03-28T22:14:11.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.725+0000 7f035957a740 1 client.16248 shutdown 2024-03-28T22:14:11.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.725+0000 7f035957a740 1 -- 172.21.15.16:0/3213149377 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55720427b470 msgr2=0x55720428ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:11.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.725+0000 7f035957a740 1 --2- 172.21.15.16:0/3213149377 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55720427b470 0x55720428ce20 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f03440109e0 tx=0x7f0344009590 comp rx=0 tx=0).stop 2024-03-28T22:14:11.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.725+0000 7f035957a740 1 -- 172.21.15.16:0/3213149377 shutdown_connections 2024-03-28T22:14:11.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.725+0000 7f035957a740 1 --2- 172.21.15.16:0/3213149377 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x5572042fae90 0x5572042fd2b0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.725+0000 7f035957a740 1 --2- 172.21.15.16:0/3213149377 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557204284970 0x55720427e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.725+0000 7f032f7fe640 0 client.16248 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:14:11.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.725+0000 7f035957a740 1 --2- 172.21.15.16:0/3213149377 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55720427bd90 0x55720427e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.725+0000 7f035957a740 1 --2- 172.21.15.16:0/3213149377 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55720427b470 0x55720428ce20 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.725+0000 7f035957a740 1 -- 172.21.15.16:0/3213149377 >> 172.21.15.16:0/3213149377 conn(0x557204275ba0 msgr2=0x5572042744b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:11.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.725+0000 7f035957a740 1 -- 172.21.15.16:0/3213149377 shutdown_connections 2024-03-28T22:14:11.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.725+0000 7f035957a740 1 -- 172.21.15.16:0/3213149377 wait complete. 2024-03-28T22:14:11.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.725+0000 7f035957a740 20 client.16248 trim_cache size 0 max 16384 2024-03-28T22:14:11.742 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:14:11.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.726+0000 7f03568c0640 1 -- 172.21.15.16:0/2240227327 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f03380a1ef0 msgr2=0x7f03380a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:11.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.726+0000 7f03568c0640 1 --2- 172.21.15.16:0/2240227327 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f03380a1ef0 0x7f03380a43b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0348005f40 tx=0x7f0348004a40 comp rx=0 tx=0).stop 2024-03-28T22:14:11.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.726+0000 7f03568c0640 1 -- 172.21.15.16:0/2240227327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0350103290 msgr2=0x7f0350198540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:11.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.726+0000 7f03568c0640 1 --2- 172.21.15.16:0/2240227327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0350103290 0x7f0350198540 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f0340000c00 tx=0x7f03400043c0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.726+0000 7f03568c0640 1 -- 172.21.15.16:0/2240227327 shutdown_connections 2024-03-28T22:14:11.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.726+0000 7f03568c0640 1 --2- 172.21.15.16:0/2240227327 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f03380a1ef0 0x7f03380a43b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.726+0000 7f03568c0640 1 --2- 172.21.15.16:0/2240227327 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f035010bd40 0x7f035019c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.726+0000 7f03568c0640 1 --2- 172.21.15.16:0/2240227327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0350103290 0x7f0350198540 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.726+0000 7f03568c0640 1 --2- 172.21.15.16:0/2240227327 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0350102970 0x7f0350198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.726+0000 7f03568c0640 1 -- 172.21.15.16:0/2240227327 >> 172.21.15.16:0/2240227327 conn(0x7f0350077800 msgr2=0x7f03500fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:11.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.726+0000 7f03568c0640 1 -- 172.21.15.16:0/2240227327 shutdown_connections 2024-03-28T22:14:11.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.726+0000 7f03568c0640 1 -- 172.21.15.16:0/2240227327 wait complete. 2024-03-28T22:14:11.745 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:14:11.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.848+0000 7f857c08f640 1 -- 172.21.15.16:0/1335711395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f857410df90 msgr2=0x7f85741103a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:11.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.848+0000 7f857c08f640 1 --2- 172.21.15.16:0/1335711395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f857410df90 0x7f85741103a0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f8564009ca0 tx=0x7f856402c180 comp rx=0 tx=0).stop 2024-03-28T22:14:11.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.848+0000 7f857c08f640 1 -- 172.21.15.16:0/1335711395 shutdown_connections 2024-03-28T22:14:11.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.848+0000 7f857c08f640 1 --2- 172.21.15.16:0/1335711395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f857410df90 0x7f85741103a0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.848+0000 7f857c08f640 1 --2- 172.21.15.16:0/1335711395 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8574074440 0x7f857410da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.848+0000 7f857c08f640 1 --2- 172.21.15.16:0/1335711395 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8574073b20 0x7f8574073f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.848+0000 7f857c08f640 1 -- 172.21.15.16:0/1335711395 >> 172.21.15.16:0/1335711395 conn(0x7f85740fd1e0 msgr2=0x7f85740ff600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:11.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.848+0000 7f857c08f640 1 -- 172.21.15.16:0/1335711395 shutdown_connections 2024-03-28T22:14:11.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.848+0000 7f857c08f640 1 -- 172.21.15.16:0/1335711395 wait complete. 2024-03-28T22:14:11.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.849+0000 7f857c08f640 1 Processor -- start 2024-03-28T22:14:11.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.849+0000 7f857c08f640 1 -- start start 2024-03-28T22:14:11.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.849+0000 7f857c08f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8574073b20 0x7f857419a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.849+0000 7f857c08f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8574074440 0x7f857419a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.849+0000 7f857c08f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f857410df90 0x7f857419ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.849+0000 7f857c08f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8574113cf0 con 0x7f857410df90 2024-03-28T22:14:11.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.849+0000 7f857c08f640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8574113b70 con 0x7f8574073b20 2024-03-28T22:14:11.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.849+0000 7f857c08f640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8574113e70 con 0x7f8574074440 2024-03-28T22:14:11.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.849+0000 7f857a62c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f857410df90 0x7f857419ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.849+0000 7f8579e2b640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8574073b20 0x7f857419a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.849+0000 7f857a62c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f857410df90 0x7f857419ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46302/0 (socket says 172.21.15.16:46302) 2024-03-28T22:14:11.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.849+0000 7f8579e2b640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8574073b20 0x7f857419a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:55946/0 (socket says 172.21.15.16:55946) 2024-03-28T22:14:11.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.849+0000 7f857a62c640 1 -- 172.21.15.16:0/3151092200 learned_addr learned my addr 172.21.15.16:0/3151092200 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:11.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.850+0000 7f857962a640 1 --2- 172.21.15.16:0/3151092200 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8574074440 0x7f857419a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.850+0000 7f857a62c640 1 -- 172.21.15.16:0/3151092200 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8574074440 msgr2=0x7f857419a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:11.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.850+0000 7f857a62c640 1 --2- 172.21.15.16:0/3151092200 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8574074440 0x7f857419a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.850+0000 7f857a62c640 1 -- 172.21.15.16:0/3151092200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8574073b20 msgr2=0x7f857419a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:11.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.850+0000 7f857a62c640 1 --2- 172.21.15.16:0/3151092200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8574073b20 0x7f857419a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.850+0000 7f857a62c640 1 -- 172.21.15.16:0/3151092200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f857419f1c0 con 0x7f857410df90 2024-03-28T22:14:11.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.850+0000 7f8579e2b640 1 --2- 172.21.15.16:0/3151092200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8574073b20 0x7f857419a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:14:11.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.850+0000 7f857a62c640 1 --2- 172.21.15.16:0/3151092200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f857410df90 0x7f857419ea90 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f8564005cc0 tx=0x7f8564035ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:11.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.850+0000 7f8572ffd640 1 -- 172.21.15.16:0/3151092200 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8564039040 con 0x7f857410df90 2024-03-28T22:14:11.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.850+0000 7f8572ffd640 1 -- 172.21.15.16:0/3151092200 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8564044070 con 0x7f857410df90 2024-03-28T22:14:11.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.850+0000 7f857c08f640 1 -- 172.21.15.16:0/3151092200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f857419f450 con 0x7f857410df90 2024-03-28T22:14:11.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.850+0000 7f8572ffd640 1 -- 172.21.15.16:0/3151092200 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85640329f0 con 0x7f857410df90 2024-03-28T22:14:11.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.850+0000 7f857c08f640 1 -- 172.21.15.16:0/3151092200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85741a6d40 con 0x7f857410df90 2024-03-28T22:14:11.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.852+0000 7f8572ffd640 1 -- 172.21.15.16:0/3151092200 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f8564032b90 con 0x7f857410df90 2024-03-28T22:14:11.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.852+0000 7f857ec77740 1 Processor -- start 2024-03-28T22:14:11.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.853+0000 7f857ec77740 1 -- start start 2024-03-28T22:14:11.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.853+0000 7f8572ffd640 1 --2- 172.21.15.16:0/3151092200 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f855c0a1ef0 0x7f855c0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.853+0000 7f857ec77740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad1cc98470 0x55ad1cc98850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.853+0000 7f857ec77740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad1cc98d90 0x55ad1cca1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.853+0000 7f857ec77740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ad1cca1970 0x55ad1cca3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.853+0000 7f857ec77740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ad1cbfc1b0 con 0x55ad1cc98470 2024-03-28T22:14:11.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.853+0000 7f857ec77740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55ad1cba4c00 con 0x55ad1cca1970 2024-03-28T22:14:11.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.853+0000 7f857ec77740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55ad1cbf8250 con 0x55ad1cc98d90 2024-03-28T22:14:11.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.853+0000 7f857962a640 1 --2- 172.21.15.16:0/3151092200 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f855c0a1ef0 0x7f855c0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.853+0000 7f8572ffd640 1 -- 172.21.15.16:0/3151092200 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f85640f0650 con 0x7f857410df90 2024-03-28T22:14:11.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.853+0000 7f8579e2b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad1cc98470 0x55ad1cc98850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.853+0000 7f8579e2b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad1cc98470 0x55ad1cc98850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46310/0 (socket says 172.21.15.16:46310) 2024-03-28T22:14:11.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.853+0000 7f8579e2b640 1 -- 172.21.15.16:0/4157293036 learned_addr learned my addr 172.21.15.16:0/4157293036 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:11.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.853+0000 7f857a62c640 1 --2- 172.21.15.16:0/4157293036 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ad1cca1970 0x55ad1cca3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.853+0000 7f857962a640 1 --2- 172.21.15.16:0/4157293036 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad1cc98d90 0x55ad1cca1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.853+0000 7f8579e2b640 1 -- 172.21.15.16:0/4157293036 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad1cc98d90 msgr2=0x55ad1cca1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:11.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.853+0000 7f8579e2b640 1 --2- 172.21.15.16:0/4157293036 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad1cc98d90 0x55ad1cca1430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.853+0000 7f8579e2b640 1 -- 172.21.15.16:0/4157293036 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ad1cca1970 msgr2=0x55ad1cca3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:11.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.853+0000 7f8579e2b640 1 --2- 172.21.15.16:0/4157293036 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ad1cca1970 0x55ad1cca3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.853+0000 7f8579e2b640 1 -- 172.21.15.16:0/4157293036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad1cbc43d0 con 0x55ad1cc98470 2024-03-28T22:14:11.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.854+0000 7f857962a640 1 --2- 172.21.15.16:0/3151092200 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f855c0a1ef0 0x7f855c0a43b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f857410f240 tx=0x7f85680137d0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:11.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.854+0000 7f8579e2b640 1 --2- 172.21.15.16:0/4157293036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad1cc98470 0x55ad1cc98850 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f856c0115f0 tx=0x7f856c011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58299b754695717f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:11.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.854+0000 7f855bfff640 1 -- 172.21.15.16:0/4157293036 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f856c01e070 con 0x55ad1cc98470 2024-03-28T22:14:11.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.854+0000 7f855bfff640 1 -- 172.21.15.16:0/4157293036 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f856c00fb20 con 0x55ad1cc98470 2024-03-28T22:14:11.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.854+0000 7f855bfff640 1 -- 172.21.15.16:0/4157293036 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f856c019c80 con 0x55ad1cc98470 2024-03-28T22:14:11.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.854+0000 7f857ec77740 1 -- 172.21.15.16:0/4157293036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad1cc98470 msgr2=0x55ad1cc98850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:11.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.854+0000 7f857ec77740 1 --2- 172.21.15.16:0/4157293036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad1cc98470 0x55ad1cc98850 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f856c0115f0 tx=0x7f856c011ab0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.854+0000 7f857ec77740 1 -- 172.21.15.16:0/4157293036 shutdown_connections 2024-03-28T22:14:11.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.854+0000 7f857ec77740 1 --2- 172.21.15.16:0/4157293036 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ad1cca1970 0x55ad1cca3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.854+0000 7f857ec77740 1 --2- 172.21.15.16:0/4157293036 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad1cc98d90 0x55ad1cca1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.854+0000 7f857ec77740 1 --2- 172.21.15.16:0/4157293036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad1cc98470 0x55ad1cc98850 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.854+0000 7f857ec77740 1 -- 172.21.15.16:0/4157293036 >> 172.21.15.16:0/4157293036 conn(0x55ad1cc92ba0 msgr2=0x55ad1cc92f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:11.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.855+0000 7f857ec77740 1 -- 172.21.15.16:0/4157293036 shutdown_connections 2024-03-28T22:14:11.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.855+0000 7f857ec77740 1 -- 172.21.15.16:0/4157293036 wait complete. 2024-03-28T22:14:11.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.855+0000 7f857ec77740 1 -- start start 2024-03-28T22:14:11.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.855+0000 7f857ec77740 1 Processor -- start 2024-03-28T22:14:11.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.855+0000 7f857ec77740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad1cc98470 0x55ad1cc9e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.855+0000 7f857ec77740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ad1cc98d90 0x55ad1cc9eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.855+0000 7f857ec77740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad1cca1970 0x55ad1cc9f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.855+0000 7f857ec77740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ad1cca4760 con 0x55ad1cc98470 2024-03-28T22:14:11.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.855+0000 7f857ec77740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55ad1cb9a470 con 0x55ad1cc98d90 2024-03-28T22:14:11.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.855+0000 7f857ec77740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55ad1cca48e0 con 0x55ad1cca1970 2024-03-28T22:14:11.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.855+0000 7f857ec77740 10 client.0 _ll_register_callbacks cb 0x55ad1cbb7cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:14:11.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.855+0000 7f8579e2b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad1cc98470 0x55ad1cc9e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.856+0000 7f857962a640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ad1cc98d90 0x55ad1cc9eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.856+0000 7f8579e2b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad1cc98470 0x55ad1cc9e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46314/0 (socket says 172.21.15.16:46314) 2024-03-28T22:14:11.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.856+0000 7f8579e2b640 1 -- 172.21.15.16:0/1795860508 learned_addr learned my addr 172.21.15.16:0/1795860508 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:11.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.856+0000 7f857a62c640 1 --2- 172.21.15.16:0/1795860508 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad1cca1970 0x55ad1cc9f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.856+0000 7f8579e2b640 1 -- 172.21.15.16:0/1795860508 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad1cca1970 msgr2=0x55ad1cc9f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:11.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.856+0000 7f8579e2b640 1 --2- 172.21.15.16:0/1795860508 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad1cca1970 0x55ad1cc9f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.856+0000 7f8579e2b640 1 -- 172.21.15.16:0/1795860508 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ad1cc98d90 msgr2=0x55ad1cc9eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:11.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.856+0000 7f8579e2b640 1 --2- 172.21.15.16:0/1795860508 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ad1cc98d90 0x55ad1cc9eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.856+0000 7f8579e2b640 1 -- 172.21.15.16:0/1795860508 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad1c9103f0 con 0x55ad1cc98470 2024-03-28T22:14:11.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.856+0000 7f857a62c640 1 --2- 172.21.15.16:0/1795860508 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad1cca1970 0x55ad1cc9f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:14:11.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.856+0000 7f857962a640 1 --2- 172.21.15.16:0/1795860508 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ad1cc98d90 0x55ad1cc9eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:14:11.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.856+0000 7f8579e2b640 1 --2- 172.21.15.16:0/1795860508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad1cc98470 0x55ad1cc9e9b0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f856c00f930 tx=0x7f856c00f960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9000afb3fa54e47 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:11.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.856+0000 7f85597fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:14:11.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.856+0000 7f85597fa640 1 -- 172.21.15.16:0/1795860508 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f856c01e040 con 0x55ad1cc98470 2024-03-28T22:14:11.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.856+0000 7f85597fa640 1 -- 172.21.15.16:0/1795860508 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f856c0199e0 con 0x55ad1cc98470 2024-03-28T22:14:11.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.856+0000 7f857ec77740 1 -- 172.21.15.16:0/1795860508 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f856c00fb20 con 0x55ad1cc98470 2024-03-28T22:14:11.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.856+0000 7f85597fa640 1 -- 172.21.15.16:0/1795860508 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f856c030050 con 0x55ad1cc98470 2024-03-28T22:14:11.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.856+0000 7f857ec77740 10 client.16260 fetch_fsmap learned FSMap version 25 2024-03-28T22:14:11.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.856+0000 7f857ec77740 1 -- 172.21.15.16:0/1795860508 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ad1c9da7d0 con 0x55ad1cc98470 2024-03-28T22:14:11.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.857+0000 7f85597fa640 1 -- 172.21.15.16:0/1795860508 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f856c00fb20 con 0x55ad1cc98470 2024-03-28T22:14:11.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.857+0000 7f85597fa640 1 -- 172.21.15.16:0/1795860508 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f856c021070 con 0x55ad1cc98470 2024-03-28T22:14:11.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.857+0000 7f857ec77740 10 client.16260 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T22:14:11.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.857+0000 7f857ec77740 10 client.16260 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T22:14:11.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.857+0000 7f857ec77740 20 client.16260 populate_metadata read hostname 'smithi016' 2024-03-28T22:14:11.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.857+0000 7f857ec77740 1 --2- 172.21.15.16:0/1795860508 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55ad1ccb1810 0x55ad1ccb3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.857+0000 7f857ec77740 4 client.16260 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:14:11.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.857+0000 7f857ec77740 1 -- 172.21.15.16:0/1795860508 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ad1cc096f0 con 0x55ad1ccb1810 2024-03-28T22:14:11.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.857+0000 7f857962a640 1 --2- 172.21.15.16:0/1795860508 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55ad1ccb1810 0x55ad1ccb3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.857+0000 7f85597fa640 10 client.16260 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:14:11.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.857+0000 7f857962a640 1 --2- 172.21.15.16:0/1795860508 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55ad1ccb1810 0x55ad1ccb3c00 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b57a558c0f2625cf server_cookie=202479c974ddb35e in_seq=0 out_seq=0 2024-03-28T22:14:11.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.858+0000 7f85597fa640 1 -- 172.21.15.16:0/1795860508 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ad1cc096f0 con 0x55ad1ccb1810 2024-03-28T22:14:11.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.858+0000 7f85597fa640 10 client.16260 handle_command_reply: tid=0 2024-03-28T22:14:11.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.859+0000 7f857ec77740 1 client.16260 shutdown 2024-03-28T22:14:11.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.859+0000 7f857ec77740 1 -- 172.21.15.16:0/1795860508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad1cc98470 msgr2=0x55ad1cc9e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:11.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.859+0000 7f857ec77740 1 --2- 172.21.15.16:0/1795860508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad1cc98470 0x55ad1cc9e9b0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f856c00f930 tx=0x7f856c00f960 comp rx=0 tx=0).stop 2024-03-28T22:14:11.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.859+0000 7f857ec77740 1 -- 172.21.15.16:0/1795860508 shutdown_connections 2024-03-28T22:14:11.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.859+0000 7f857ec77740 1 --2- 172.21.15.16:0/1795860508 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55ad1ccb1810 0x55ad1ccb3c00 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.859+0000 7f857ec77740 1 --2- 172.21.15.16:0/1795860508 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad1cca1970 0x55ad1cc9f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.859+0000 7f857ec77740 1 --2- 172.21.15.16:0/1795860508 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ad1cc98d90 0x55ad1cc9eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.859+0000 7f857ec77740 1 --2- 172.21.15.16:0/1795860508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad1cc98470 0x55ad1cc9e9b0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.859+0000 7f85597fa640 0 client.16260 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:14:11.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.859+0000 7f857ec77740 1 -- 172.21.15.16:0/1795860508 >> 172.21.15.16:0/1795860508 conn(0x55ad1cc92ba0 msgr2=0x55ad1cc948f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:11.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.859+0000 7f857ec77740 1 -- 172.21.15.16:0/1795860508 shutdown_connections 2024-03-28T22:14:11.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.859+0000 7f857ec77740 1 -- 172.21.15.16:0/1795860508 wait complete. 2024-03-28T22:14:11.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.859+0000 7f857ec77740 20 client.16260 trim_cache size 0 max 16384 2024-03-28T22:14:11.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.872+0000 7f857ec77740 1 Processor -- start 2024-03-28T22:14:11.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.872+0000 7f857ec77740 1 -- start start 2024-03-28T22:14:11.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.872+0000 7f857ec77740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad1cc98470 0x55ad1cca10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.872+0000 7f857ec77740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ad1cc98d90 0x55ad1ccad500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.872+0000 7f857ec77740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad1cca1970 0x55ad1cc91d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.872+0000 7f857ec77740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f85741b1fa0 con 0x55ad1cca1970 2024-03-28T22:14:11.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.872+0000 7f857ec77740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f85741af840 con 0x55ad1cc98d90 2024-03-28T22:14:11.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.872+0000 7f857ec77740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f85741ad690 con 0x55ad1cc98470 2024-03-28T22:14:11.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.872+0000 7f857a62c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad1cca1970 0x55ad1cc91d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.872+0000 7f857a62c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad1cca1970 0x55ad1cc91d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46324/0 (socket says 172.21.15.16:46324) 2024-03-28T22:14:11.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.872+0000 7f857a62c640 1 -- 172.21.15.16:0/4266297178 learned_addr learned my addr 172.21.15.16:0/4266297178 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:11.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.873+0000 7f857962a640 1 --2- 172.21.15.16:0/4266297178 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ad1cc98d90 0x55ad1ccad500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.873+0000 7f8579e2b640 1 --2- 172.21.15.16:0/4266297178 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad1cc98470 0x55ad1cca10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.873+0000 7f857a62c640 1 -- 172.21.15.16:0/4266297178 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad1cc98470 msgr2=0x55ad1cca10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:11.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.873+0000 7f857a62c640 1 --2- 172.21.15.16:0/4266297178 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad1cc98470 0x55ad1cca10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.873+0000 7f857a62c640 1 -- 172.21.15.16:0/4266297178 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ad1cc98d90 msgr2=0x55ad1ccad500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:11.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.873+0000 7f857a62c640 1 --2- 172.21.15.16:0/4266297178 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ad1cc98d90 0x55ad1ccad500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.873+0000 7f857a62c640 1 -- 172.21.15.16:0/4266297178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad1cc9d960 con 0x55ad1cca1970 2024-03-28T22:14:11.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.873+0000 7f8579e2b640 1 --2- 172.21.15.16:0/4266297178 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad1cc98470 0x55ad1cca10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:14:11.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.873+0000 7f857a62c640 1 --2- 172.21.15.16:0/4266297178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad1cca1970 0x55ad1cc91d40 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f85640bd0a0 tx=0x7f85640b9470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2394f97384dfa3d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:11.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.873+0000 7f855b7fe640 1 -- 172.21.15.16:0/4266297178 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85640bca70 con 0x55ad1cca1970 2024-03-28T22:14:11.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.873+0000 7f855b7fe640 1 -- 172.21.15.16:0/4266297178 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85640bd810 con 0x55ad1cca1970 2024-03-28T22:14:11.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.873+0000 7f855b7fe640 1 -- 172.21.15.16:0/4266297178 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f856402d040 con 0x55ad1cca1970 2024-03-28T22:14:11.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.874+0000 7f857ec77740 1 -- 172.21.15.16:0/4266297178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad1cca1970 msgr2=0x55ad1cc91d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:11.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.874+0000 7f857ec77740 1 --2- 172.21.15.16:0/4266297178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad1cca1970 0x55ad1cc91d40 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f85640bd0a0 tx=0x7f85640b9470 comp rx=0 tx=0).stop 2024-03-28T22:14:11.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.874+0000 7f857ec77740 1 -- 172.21.15.16:0/4266297178 shutdown_connections 2024-03-28T22:14:11.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.874+0000 7f857ec77740 1 --2- 172.21.15.16:0/4266297178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad1cca1970 0x55ad1cc91d40 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.874+0000 7f857ec77740 1 --2- 172.21.15.16:0/4266297178 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ad1cc98d90 0x55ad1ccad500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.874+0000 7f857ec77740 1 --2- 172.21.15.16:0/4266297178 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad1cc98470 0x55ad1cca10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.874+0000 7f857ec77740 1 -- 172.21.15.16:0/4266297178 >> 172.21.15.16:0/4266297178 conn(0x55ad1cc92ba0 msgr2=0x55ad1cc92f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:11.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.874+0000 7f857ec77740 1 -- 172.21.15.16:0/4266297178 shutdown_connections 2024-03-28T22:14:11.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.874+0000 7f857ec77740 1 -- 172.21.15.16:0/4266297178 wait complete. 2024-03-28T22:14:11.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.874+0000 7f857ec77740 1 -- start start 2024-03-28T22:14:11.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.874+0000 7f857ec77740 1 Processor -- start 2024-03-28T22:14:11.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.874+0000 7f857ec77740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ad1cc98470 0x55ad1cca9f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.874+0000 7f857ec77740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad1cc98d90 0x55ad1cca5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.874+0000 7f857ec77740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad1cca1970 0x55ad1cca59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.874+0000 7f857ec77740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f85741add00 con 0x55ad1cca1970 2024-03-28T22:14:11.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.874+0000 7f857ec77740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f85741af170 con 0x55ad1cc98470 2024-03-28T22:14:11.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.874+0000 7f857ec77740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f85741ae3d0 con 0x55ad1cc98d90 2024-03-28T22:14:11.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.874+0000 7f857ec77740 10 client.0 _ll_register_callbacks cb 0x7f856c011440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:14:11.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.875+0000 7f857a62c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad1cca1970 0x55ad1cca59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.875+0000 7f857a62c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad1cca1970 0x55ad1cca59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46330/0 (socket says 172.21.15.16:46330) 2024-03-28T22:14:11.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.875+0000 7f857a62c640 1 -- 172.21.15.16:0/3211851828 learned_addr learned my addr 172.21.15.16:0/3211851828 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:11.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.875+0000 7f8579e2b640 1 --2- 172.21.15.16:0/3211851828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ad1cc98470 0x55ad1cca9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.875+0000 7f857a62c640 1 -- 172.21.15.16:0/3211851828 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad1cc98d90 msgr2=0x55ad1cca5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:11.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.875+0000 7f857a62c640 1 --2- 172.21.15.16:0/3211851828 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad1cc98d90 0x55ad1cca5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.875+0000 7f857a62c640 1 -- 172.21.15.16:0/3211851828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ad1cc98470 msgr2=0x55ad1cca9f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:11.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.875+0000 7f857a62c640 1 --2- 172.21.15.16:0/3211851828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ad1cc98470 0x55ad1cca9f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.875+0000 7f857a62c640 1 -- 172.21.15.16:0/3211851828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad1cd16af0 con 0x55ad1cca1970 2024-03-28T22:14:11.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.875+0000 7f857a62c640 1 --2- 172.21.15.16:0/3211851828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad1cca1970 0x55ad1cca59d0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f8564009a60 tx=0x7f85640bb620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fb76875457a1db7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:11.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.875+0000 7f85597fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:14:11.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.875+0000 7f85597fa640 1 -- 172.21.15.16:0/3211851828 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85640da060 con 0x55ad1cca1970 2024-03-28T22:14:11.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.875+0000 7f85597fa640 1 -- 172.21.15.16:0/3211851828 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f85640bd370 con 0x55ad1cca1970 2024-03-28T22:14:11.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.875+0000 7f857ec77740 1 -- 172.21.15.16:0/3211851828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85640bd810 con 0x55ad1cca1970 2024-03-28T22:14:11.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.875+0000 7f85597fa640 1 -- 172.21.15.16:0/3211851828 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f85640bca70 con 0x55ad1cca1970 2024-03-28T22:14:11.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.875+0000 7f857ec77740 10 client.16266 fetch_fsmap learned FSMap version 25 2024-03-28T22:14:11.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.875+0000 7f857ec77740 1 -- 172.21.15.16:0/3211851828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ad1cd17ba0 con 0x55ad1cca1970 2024-03-28T22:14:11.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.875+0000 7f85597fa640 1 -- 172.21.15.16:0/3211851828 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85640bd810 con 0x55ad1cca1970 2024-03-28T22:14:11.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.876+0000 7f85597fa640 1 -- 172.21.15.16:0/3211851828 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f856403c030 con 0x55ad1cca1970 2024-03-28T22:14:11.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.876+0000 7f857ec77740 10 client.16266 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T22:14:11.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.876+0000 7f857ec77740 10 client.16266 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T22:14:11.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.876+0000 7f857ec77740 20 client.16266 populate_metadata read hostname 'smithi016' 2024-03-28T22:14:11.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.876+0000 7f857ec77740 1 --2- 172.21.15.16:0/3211851828 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55ad1cd18170 0x55ad1cd1a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:11.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.876+0000 7f857ec77740 4 client.16266 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:14:11.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.876+0000 7f857ec77740 1 -- 172.21.15.16:0/3211851828 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ad1cc03a40 con 0x55ad1cd18170 2024-03-28T22:14:11.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.876+0000 7f8579e2b640 1 --2- 172.21.15.16:0/3211851828 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55ad1cd18170 0x55ad1cd1a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:11.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.876+0000 7f85597fa640 10 client.16266 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:14:11.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.876+0000 7f8579e2b640 1 --2- 172.21.15.16:0/3211851828 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55ad1cd18170 0x55ad1cd1a590 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d0eda6ad81005d73 server_cookie=ee5f1b4b92794cf8 in_seq=0 out_seq=0 2024-03-28T22:14:11.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.876+0000 7f85597fa640 1 -- 172.21.15.16:0/3211851828 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ad1cc03a40 con 0x55ad1cd18170 2024-03-28T22:14:11.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.876+0000 7f85597fa640 10 client.16266 handle_command_reply: tid=0 2024-03-28T22:14:11.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.876+0000 7f857ec77740 1 client.16266 shutdown 2024-03-28T22:14:11.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.876+0000 7f857ec77740 1 -- 172.21.15.16:0/3211851828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad1cca1970 msgr2=0x55ad1cca59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:11.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.876+0000 7f857ec77740 1 --2- 172.21.15.16:0/3211851828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad1cca1970 0x55ad1cca59d0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f8564009a60 tx=0x7f85640bb620 comp rx=0 tx=0).stop 2024-03-28T22:14:11.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.876+0000 7f857ec77740 1 -- 172.21.15.16:0/3211851828 shutdown_connections 2024-03-28T22:14:11.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.876+0000 7f857ec77740 1 --2- 172.21.15.16:0/3211851828 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55ad1cd18170 0x55ad1cd1a590 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.876+0000 7f857ec77740 1 --2- 172.21.15.16:0/3211851828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ad1cca1970 0x55ad1cca59d0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.876+0000 7f857ec77740 1 --2- 172.21.15.16:0/3211851828 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad1cc98d90 0x55ad1cca5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.876+0000 7f857ec77740 1 --2- 172.21.15.16:0/3211851828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ad1cc98470 0x55ad1cca9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.876+0000 7f85597fa640 0 client.16266 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:14:11.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.876+0000 7f857ec77740 1 -- 172.21.15.16:0/3211851828 >> 172.21.15.16:0/3211851828 conn(0x55ad1cc92ba0 msgr2=0x55ad1cc9f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:11.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.877+0000 7f857ec77740 1 -- 172.21.15.16:0/3211851828 shutdown_connections 2024-03-28T22:14:11.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.877+0000 7f857ec77740 1 -- 172.21.15.16:0/3211851828 wait complete. 2024-03-28T22:14:11.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.877+0000 7f857ec77740 20 client.16266 trim_cache size 0 max 16384 2024-03-28T22:14:11.898 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:14:11.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.877+0000 7f857c08f640 1 -- 172.21.15.16:0/3151092200 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f855c0a1ef0 msgr2=0x7f855c0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:11.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.877+0000 7f857c08f640 1 --2- 172.21.15.16:0/3151092200 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f855c0a1ef0 0x7f855c0a43b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f857410f240 tx=0x7f85680137d0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.877+0000 7f857c08f640 1 -- 172.21.15.16:0/3151092200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f857410df90 msgr2=0x7f857419ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:11.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.877+0000 7f857c08f640 1 --2- 172.21.15.16:0/3151092200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f857410df90 0x7f857419ea90 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f8564005cc0 tx=0x7f8564035ed0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.877+0000 7f857c08f640 1 -- 172.21.15.16:0/3151092200 shutdown_connections 2024-03-28T22:14:11.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.877+0000 7f857c08f640 1 --2- 172.21.15.16:0/3151092200 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f855c0a1ef0 0x7f855c0a43b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.877+0000 7f857c08f640 1 --2- 172.21.15.16:0/3151092200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f857410df90 0x7f857419ea90 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.877+0000 7f857c08f640 1 --2- 172.21.15.16:0/3151092200 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8574074440 0x7f857419a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.877+0000 7f857c08f640 1 --2- 172.21.15.16:0/3151092200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8574073b20 0x7f857419a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:11.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.877+0000 7f857c08f640 1 -- 172.21.15.16:0/3151092200 >> 172.21.15.16:0/3151092200 conn(0x7f85740fd1e0 msgr2=0x7f85740fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:11.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.877+0000 7f857c08f640 1 -- 172.21.15.16:0/3151092200 shutdown_connections 2024-03-28T22:14:11.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:11.877+0000 7f857c08f640 1 -- 172.21.15.16:0/3151092200 wait complete. 2024-03-28T22:14:11.901 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:14:12.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.004+0000 7faa93447640 1 -- 172.21.15.16:0/1763951729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa8c103310 msgr2=0x7faa8c107740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:12.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.004+0000 7faa93447640 1 --2- 172.21.15.16:0/1763951729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa8c103310 0x7faa8c107740 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7faa7c009930 tx=0x7faa7c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:14:12.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.004+0000 7faa93447640 1 -- 172.21.15.16:0/1763951729 shutdown_connections 2024-03-28T22:14:12.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.004+0000 7faa93447640 1 --2- 172.21.15.16:0/1763951729 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa8c107e50 0x7faa8c10e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:12.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.004+0000 7faa93447640 1 --2- 172.21.15.16:0/1763951729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa8c103310 0x7faa8c107740 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:12.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.004+0000 7faa93447640 1 --2- 172.21.15.16:0/1763951729 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa8c102960 0x7faa8c102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:12.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.004+0000 7faa93447640 1 -- 172.21.15.16:0/1763951729 >> 172.21.15.16:0/1763951729 conn(0x7faa8c077830 msgr2=0x7faa8c077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:12.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.004+0000 7faa93447640 1 -- 172.21.15.16:0/1763951729 shutdown_connections 2024-03-28T22:14:12.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.005+0000 7faa93447640 1 -- 172.21.15.16:0/1763951729 wait complete. 2024-03-28T22:14:12.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.005+0000 7faa93447640 1 Processor -- start 2024-03-28T22:14:12.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.005+0000 7faa93447640 1 -- start start 2024-03-28T22:14:12.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.005+0000 7faa93447640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa8c102960 0x7faa8c19a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:12.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.005+0000 7faa93447640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa8c107e50 0x7faa8c19a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:12.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.005+0000 7faa93447640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa8c19ecd0 0x7faa8c19f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:12.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.005+0000 7faa93447640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faa8c102540 con 0x7faa8c19ecd0 2024-03-28T22:14:12.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.005+0000 7faa93447640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7faa8c1023c0 con 0x7faa8c102960 2024-03-28T22:14:12.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.005+0000 7faa93447640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7faa8c114000 con 0x7faa8c107e50 2024-03-28T22:14:12.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.005+0000 7faa919e4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa8c19ecd0 0x7faa8c19f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:12.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.005+0000 7faa911e3640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa8c102960 0x7faa8c19a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:12.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.005+0000 7faa919e4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa8c19ecd0 0x7faa8c19f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46346/0 (socket says 172.21.15.16:46346) 2024-03-28T22:14:12.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.005+0000 7faa919e4640 1 -- 172.21.15.16:0/1657697634 learned_addr learned my addr 172.21.15.16:0/1657697634 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:12.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.005+0000 7faa909e2640 1 --2- 172.21.15.16:0/1657697634 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa8c107e50 0x7faa8c19a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:12.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.006+0000 7faa911e3640 1 -- 172.21.15.16:0/1657697634 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa8c107e50 msgr2=0x7faa8c19a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:12.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.006+0000 7faa911e3640 1 --2- 172.21.15.16:0/1657697634 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa8c107e50 0x7faa8c19a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:12.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.006+0000 7faa911e3640 1 -- 172.21.15.16:0/1657697634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa8c19ecd0 msgr2=0x7faa8c19f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:12.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.006+0000 7faa911e3640 1 --2- 172.21.15.16:0/1657697634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa8c19ecd0 0x7faa8c19f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:12.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.006+0000 7faa911e3640 1 -- 172.21.15.16:0/1657697634 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa8c19f7c0 con 0x7faa8c102960 2024-03-28T22:14:12.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.006+0000 7faa919e4640 1 --2- 172.21.15.16:0/1657697634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa8c19ecd0 0x7faa8c19f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:14:12.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.006+0000 7faa911e3640 1 --2- 172.21.15.16:0/1657697634 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa8c102960 0x7faa8c19a410 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7faa8800c970 tx=0x7faa8800ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:12.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.006+0000 7faa827fc640 1 -- 172.21.15.16:0/1657697634 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa88007bb0 con 0x7faa8c102960 2024-03-28T22:14:12.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.006+0000 7faa93447640 1 -- 172.21.15.16:0/1657697634 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa8c19fab0 con 0x7faa8c102960 2024-03-28T22:14:12.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.006+0000 7faa827fc640 1 -- 172.21.15.16:0/1657697634 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa88007d50 con 0x7faa8c102960 2024-03-28T22:14:12.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.006+0000 7faa827fc640 1 -- 172.21.15.16:0/1657697634 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa8800f450 con 0x7faa8c102960 2024-03-28T22:14:12.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.006+0000 7faa93447640 1 -- 172.21.15.16:0/1657697634 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa8c1a71c0 con 0x7faa8c102960 2024-03-28T22:14:12.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.008+0000 7faa827fc640 1 -- 172.21.15.16:0/1657697634 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7faa88004f30 con 0x7faa8c102960 2024-03-28T22:14:12.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.009+0000 7faa9602e740 1 Processor -- start 2024-03-28T22:14:12.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.009+0000 7faa9602e740 1 -- start start 2024-03-28T22:14:12.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.009+0000 7faa9602e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55faa2a24470 0x55faa2a24850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:12.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.009+0000 7faa9602e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55faa2a24d90 0x55faa2a2d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:12.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.009+0000 7faa9602e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55faa2a2d970 0x55faa2a2fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:12.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.009+0000 7faa9602e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55faa29881b0 con 0x55faa2a24d90 2024-03-28T22:14:12.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.009+0000 7faa9602e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55faa2930c00 con 0x55faa2a2d970 2024-03-28T22:14:12.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.009+0000 7faa9602e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55faa2984250 con 0x55faa2a24470 2024-03-28T22:14:12.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.009+0000 7faa827fc640 1 --2- 172.21.15.16:0/1657697634 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7faa780a1ef0 0x7faa780a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:12.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.009+0000 7faa911e3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55faa2a24d90 0x55faa2a2d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:12.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.009+0000 7faa919e4640 1 --2- 172.21.15.16:0/1657697634 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7faa780a1ef0 0x7faa780a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:12.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.009+0000 7faa909e2640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55faa2a2d970 0x55faa2a2fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:12.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.009+0000 7faa909e2640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55faa2a2d970 0x55faa2a2fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:56022/0 (socket says 172.21.15.16:56022) 2024-03-28T22:14:12.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.009+0000 7faa909e2640 1 -- 172.21.15.16:0/3453462967 learned_addr learned my addr 172.21.15.16:0/3453462967 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:12.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.009+0000 7faa911e3640 1 --2- 172.21.15.16:0/3453462967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55faa2a24d90 0x55faa2a2d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46362/0 (socket says 172.21.15.16:46362) 2024-03-28T22:14:12.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.009+0000 7faa919e4640 1 --2- 172.21.15.16:0/3453462967 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55faa2a24470 0x55faa2a24850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:12.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.009+0000 7faa827fc640 1 -- 172.21.15.16:0/1657697634 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7faa880cdd90 con 0x7faa8c102960 2024-03-28T22:14:12.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.009+0000 7faa909e2640 1 -- 172.21.15.16:0/3453462967 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55faa2a24470 msgr2=0x55faa2a24850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:12.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.009+0000 7faa909e2640 1 --2- 172.21.15.16:0/3453462967 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55faa2a24470 0x55faa2a24850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:12.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.009+0000 7faa909e2640 1 -- 172.21.15.16:0/3453462967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55faa2a24d90 msgr2=0x55faa2a2d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:12.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.009+0000 7faa909e2640 1 --2- 172.21.15.16:0/3453462967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55faa2a24d90 0x55faa2a2d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:12.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.009+0000 7faa909e2640 1 -- 172.21.15.16:0/3453462967 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55faa29503d0 con 0x55faa2a2d970 2024-03-28T22:14:12.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.009+0000 7faa911e3640 1 --2- 172.21.15.16:0/3453462967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55faa2a24d90 0x55faa2a2d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:14:12.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.009+0000 7faa919e4640 1 --2- 172.21.15.16:0/3453462967 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55faa2a24470 0x55faa2a24850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:14:12.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.010+0000 7faa919e4640 1 --2- 172.21.15.16:0/1657697634 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7faa780a1ef0 0x7faa780a43b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7faa840049d0 tx=0x7faa84011440 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:12.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.010+0000 7faa909e2640 1 --2- 172.21.15.16:0/3453462967 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55faa2a2d970 0x55faa2a2fd60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7faa7c005ab0 tx=0x7faa7c005e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2536abd13bba46e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:12.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.010+0000 7faa777fe640 1 -- 172.21.15.16:0/3453462967 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa7c042070 con 0x55faa2a2d970 2024-03-28T22:14:12.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.010+0000 7faa777fe640 1 -- 172.21.15.16:0/3453462967 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa7c02ca80 con 0x55faa2a2d970 2024-03-28T22:14:12.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.010+0000 7faa777fe640 1 -- 172.21.15.16:0/3453462967 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa7c004f30 con 0x55faa2a2d970 2024-03-28T22:14:12.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.011+0000 7faa9602e740 1 -- 172.21.15.16:0/3453462967 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55faa2a2d970 msgr2=0x55faa2a2fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:12.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.011+0000 7faa9602e740 1 --2- 172.21.15.16:0/3453462967 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55faa2a2d970 0x55faa2a2fd60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7faa7c005ab0 tx=0x7faa7c005e00 comp rx=0 tx=0).stop 2024-03-28T22:14:12.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.011+0000 7faa9602e740 1 -- 172.21.15.16:0/3453462967 shutdown_connections 2024-03-28T22:14:12.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.011+0000 7faa9602e740 1 --2- 172.21.15.16:0/3453462967 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55faa2a2d970 0x55faa2a2fd60 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:12.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.011+0000 7faa9602e740 1 --2- 172.21.15.16:0/3453462967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55faa2a24d90 0x55faa2a2d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:12.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.011+0000 7faa9602e740 1 --2- 172.21.15.16:0/3453462967 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55faa2a24470 0x55faa2a24850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:12.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.011+0000 7faa9602e740 1 -- 172.21.15.16:0/3453462967 >> 172.21.15.16:0/3453462967 conn(0x55faa2a1eba0 msgr2=0x55faa2a1ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:12.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.011+0000 7faa9602e740 1 -- 172.21.15.16:0/3453462967 shutdown_connections 2024-03-28T22:14:12.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.011+0000 7faa9602e740 1 -- 172.21.15.16:0/3453462967 wait complete. 2024-03-28T22:14:12.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.011+0000 7faa9602e740 1 -- start start 2024-03-28T22:14:12.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.011+0000 7faa9602e740 1 Processor -- start 2024-03-28T22:14:12.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.011+0000 7faa9602e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55faa2a24470 0x55faa2a2a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:12.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.011+0000 7faa9602e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55faa2a24d90 0x55faa2a2aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:12.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.011+0000 7faa9602e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55faa2a2d970 0x55faa2a2b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:12.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.011+0000 7faa9602e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55faa2a30700 con 0x55faa2a24470 2024-03-28T22:14:12.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.011+0000 7faa9602e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55faa2926470 con 0x55faa2a24d90 2024-03-28T22:14:12.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.011+0000 7faa9602e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55faa2a30880 con 0x55faa2a2d970 2024-03-28T22:14:12.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.011+0000 7faa9602e740 10 client.0 _ll_register_callbacks cb 0x55faa2943cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:14:12.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.011+0000 7faa911e3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55faa2a24470 0x55faa2a2a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:12.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.011+0000 7faa911e3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55faa2a24470 0x55faa2a2a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46374/0 (socket says 172.21.15.16:46374) 2024-03-28T22:14:12.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.011+0000 7faa911e3640 1 -- 172.21.15.16:0/3343614563 learned_addr learned my addr 172.21.15.16:0/3343614563 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:12.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.012+0000 7faa909e2640 1 --2- 172.21.15.16:0/3343614563 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55faa2a24d90 0x55faa2a2aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:12.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.012+0000 7faa919e4640 1 --2- 172.21.15.16:0/3343614563 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55faa2a2d970 0x55faa2a2b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:12.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.012+0000 7faa911e3640 1 -- 172.21.15.16:0/3343614563 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55faa2a2d970 msgr2=0x55faa2a2b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:12.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.012+0000 7faa911e3640 1 --2- 172.21.15.16:0/3343614563 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55faa2a2d970 0x55faa2a2b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:12.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.012+0000 7faa911e3640 1 -- 172.21.15.16:0/3343614563 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55faa2a24d90 msgr2=0x55faa2a2aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:12.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.012+0000 7faa911e3640 1 --2- 172.21.15.16:0/3343614563 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55faa2a24d90 0x55faa2a2aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:12.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.012+0000 7faa911e3640 1 -- 172.21.15.16:0/3343614563 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55faa269c3f0 con 0x55faa2a24470 2024-03-28T22:14:12.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.012+0000 7faa911e3640 1 --2- 172.21.15.16:0/3343614563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55faa2a24470 0x55faa2a2a9a0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7faa880cb560 tx=0x7faa8806a6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf51afdf1a9f068b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:12.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.012+0000 7faa74ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:14:12.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.012+0000 7faa74ff9640 1 -- 172.21.15.16:0/3343614563 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa8801d040 con 0x55faa2a24470 2024-03-28T22:14:12.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.012+0000 7faa74ff9640 1 -- 172.21.15.16:0/3343614563 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7faa88098b50 con 0x55faa2a24470 2024-03-28T22:14:12.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.012+0000 7faa9602e740 1 -- 172.21.15.16:0/3343614563 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa7c02ca80 con 0x55faa2a24470 2024-03-28T22:14:12.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.012+0000 7faa74ff9640 1 -- 172.21.15.16:0/3343614563 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7faa8809a850 con 0x55faa2a24470 2024-03-28T22:14:12.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.012+0000 7faa9602e740 10 client.16278 fetch_fsmap learned FSMap version 25 2024-03-28T22:14:12.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.012+0000 7faa9602e740 1 -- 172.21.15.16:0/3343614563 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55faa27667d0 con 0x55faa2a24470 2024-03-28T22:14:12.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.012+0000 7faa74ff9640 1 -- 172.21.15.16:0/3343614563 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa7c02ca80 con 0x55faa2a24470 2024-03-28T22:14:12.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.013+0000 7faa74ff9640 1 -- 172.21.15.16:0/3343614563 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7faa880a2ab0 con 0x55faa2a24470 2024-03-28T22:14:12.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.013+0000 7faa9602e740 10 client.16278 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T22:14:12.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.013+0000 7faa9602e740 10 client.16278 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:14:12.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.013+0000 7faa9602e740 20 client.16278 populate_metadata read hostname 'smithi016' 2024-03-28T22:14:12.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.013+0000 7faa9602e740 1 --2- 172.21.15.16:0/3343614563 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55faa2a3d8a0 0x55faa2a3fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:12.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.013+0000 7faa9602e740 4 client.16278 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:14:12.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.013+0000 7faa9602e740 1 -- 172.21.15.16:0/3343614563 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55faa29956f0 con 0x55faa2a3d8a0 2024-03-28T22:14:12.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.013+0000 7faa909e2640 1 --2- 172.21.15.16:0/3343614563 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55faa2a3d8a0 0x55faa2a3fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:12.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.013+0000 7faa74ff9640 10 client.16278 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:14:12.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.013+0000 7faa909e2640 1 --2- 172.21.15.16:0/3343614563 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55faa2a3d8a0 0x55faa2a3fc90 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30662e82c7572fd0 server_cookie=f3d4457c1aee5616 in_seq=0 out_seq=0 2024-03-28T22:14:12.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.014+0000 7faa74ff9640 1 -- 172.21.15.16:0/3343614563 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55faa29956f0 con 0x55faa2a3d8a0 2024-03-28T22:14:12.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.014+0000 7faa74ff9640 10 client.16278 handle_command_reply: tid=0 2024-03-28T22:14:12.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.014+0000 7faa9602e740 1 client.16278 shutdown 2024-03-28T22:14:12.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.014+0000 7faa9602e740 1 -- 172.21.15.16:0/3343614563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55faa2a24470 msgr2=0x55faa2a2a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:12.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.014+0000 7faa9602e740 1 --2- 172.21.15.16:0/3343614563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55faa2a24470 0x55faa2a2a9a0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7faa880cb560 tx=0x7faa8806a6b0 comp rx=0 tx=0).stop 2024-03-28T22:14:12.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.014+0000 7faa9602e740 1 -- 172.21.15.16:0/3343614563 shutdown_connections 2024-03-28T22:14:12.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.014+0000 7faa9602e740 1 --2- 172.21.15.16:0/3343614563 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55faa2a3d8a0 0x55faa2a3fc90 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:12.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.014+0000 7faa9602e740 1 --2- 172.21.15.16:0/3343614563 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55faa2a2d970 0x55faa2a2b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:12.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.014+0000 7faa9602e740 1 --2- 172.21.15.16:0/3343614563 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55faa2a24d90 0x55faa2a2aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:12.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.014+0000 7faa74ff9640 0 client.16278 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:14:12.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.014+0000 7faa9602e740 1 --2- 172.21.15.16:0/3343614563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55faa2a24470 0x55faa2a2a9a0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:12.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.014+0000 7faa9602e740 1 -- 172.21.15.16:0/3343614563 >> 172.21.15.16:0/3343614563 conn(0x55faa2a1eba0 msgr2=0x55faa2a208f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:12.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.014+0000 7faa9602e740 1 -- 172.21.15.16:0/3343614563 shutdown_connections 2024-03-28T22:14:12.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.014+0000 7faa9602e740 1 -- 172.21.15.16:0/3343614563 wait complete. 2024-03-28T22:14:12.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.014+0000 7faa9602e740 20 client.16278 trim_cache size 0 max 16384 2024-03-28T22:14:12.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.027+0000 7faa9602e740 1 Processor -- start 2024-03-28T22:14:12.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.027+0000 7faa9602e740 1 -- start start 2024-03-28T22:14:12.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.028+0000 7faa9602e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55faa2a1eba0 0x55faa2a394a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:12.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.028+0000 7faa9602e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55faa2a24470 0x55faa2a45510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:12.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.028+0000 7faa9602e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55faa2a24d90 0x55faa2a27c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:12.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.028+0000 7faa9602e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faa8c1b1e10 con 0x55faa2a1eba0 2024-03-28T22:14:12.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.028+0000 7faa9602e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7faa8c1af6b0 con 0x55faa2a24d90 2024-03-28T22:14:12.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.028+0000 7faa9602e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7faa8c1ad500 con 0x55faa2a24470 2024-03-28T22:14:12.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.028+0000 7faa911e3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55faa2a1eba0 0x55faa2a394a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:12.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.028+0000 7faa911e3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55faa2a1eba0 0x55faa2a394a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46376/0 (socket says 172.21.15.16:46376) 2024-03-28T22:14:12.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.028+0000 7faa911e3640 1 -- 172.21.15.16:0/1846753660 learned_addr learned my addr 172.21.15.16:0/1846753660 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:12.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.028+0000 7faa919e4640 1 --2- 172.21.15.16:0/1846753660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55faa2a24d90 0x55faa2a27c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:12.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.028+0000 7faa909e2640 1 --2- 172.21.15.16:0/1846753660 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55faa2a24470 0x55faa2a45510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:12.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.028+0000 7faa911e3640 1 -- 172.21.15.16:0/1846753660 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55faa2a24470 msgr2=0x55faa2a45510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:12.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.028+0000 7faa911e3640 1 --2- 172.21.15.16:0/1846753660 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55faa2a24470 0x55faa2a45510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:12.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.028+0000 7faa911e3640 1 -- 172.21.15.16:0/1846753660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55faa2a24d90 msgr2=0x55faa2a27c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:12.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.028+0000 7faa911e3640 1 --2- 172.21.15.16:0/1846753660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55faa2a24d90 0x55faa2a27c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:12.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.028+0000 7faa911e3640 1 -- 172.21.15.16:0/1846753660 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55faa2a2fd60 con 0x55faa2a1eba0 2024-03-28T22:14:12.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.028+0000 7faa919e4640 1 --2- 172.21.15.16:0/1846753660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55faa2a24d90 0x55faa2a27c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:14:12.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.028+0000 7faa911e3640 1 --2- 172.21.15.16:0/1846753660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55faa2a1eba0 0x55faa2a394a0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7faa880cbb00 tx=0x7faa880791b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27ea888a0daac222 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:12.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.028+0000 7faa76ffd640 1 -- 172.21.15.16:0/1846753660 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa88069070 con 0x55faa2a1eba0 2024-03-28T22:14:12.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.028+0000 7faa76ffd640 1 -- 172.21.15.16:0/1846753660 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa880a2f40 con 0x55faa2a1eba0 2024-03-28T22:14:12.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.028+0000 7faa76ffd640 1 -- 172.21.15.16:0/1846753660 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa8801d030 con 0x55faa2a1eba0 2024-03-28T22:14:12.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.029+0000 7faa9602e740 1 -- 172.21.15.16:0/1846753660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55faa2a1eba0 msgr2=0x55faa2a394a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:12.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.029+0000 7faa9602e740 1 --2- 172.21.15.16:0/1846753660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55faa2a1eba0 0x55faa2a394a0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7faa880cbb00 tx=0x7faa880791b0 comp rx=0 tx=0).stop 2024-03-28T22:14:12.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.029+0000 7faa9602e740 1 -- 172.21.15.16:0/1846753660 shutdown_connections 2024-03-28T22:14:12.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.029+0000 7faa9602e740 1 --2- 172.21.15.16:0/1846753660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55faa2a24d90 0x55faa2a27c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:12.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.029+0000 7faa9602e740 1 --2- 172.21.15.16:0/1846753660 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55faa2a24470 0x55faa2a45510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:12.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.029+0000 7faa9602e740 1 --2- 172.21.15.16:0/1846753660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55faa2a1eba0 0x55faa2a394a0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:12.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.029+0000 7faa9602e740 1 -- 172.21.15.16:0/1846753660 >> 172.21.15.16:0/1846753660 conn(0x55faa2a26370 msgr2=0x55faa2a1dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:12.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.029+0000 7faa9602e740 1 -- 172.21.15.16:0/1846753660 shutdown_connections 2024-03-28T22:14:12.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.029+0000 7faa9602e740 1 -- 172.21.15.16:0/1846753660 wait complete. 2024-03-28T22:14:12.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.029+0000 7faa9602e740 1 -- start start 2024-03-28T22:14:12.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.029+0000 7faa9602e740 1 Processor -- start 2024-03-28T22:14:12.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.030+0000 7faa9602e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55faa2a1eba0 0x55faa2a35a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:12.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.030+0000 7faa9602e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55faa2a24470 0x55faa2a31e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:12.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.030+0000 7faa9602e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55faa2a24d90 0x55faa2a32360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:12.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.030+0000 7faa9602e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faa8c1adb70 con 0x55faa2a24d90 2024-03-28T22:14:12.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.030+0000 7faa9602e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7faa8c1aefe0 con 0x55faa2a1eba0 2024-03-28T22:14:12.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.030+0000 7faa9602e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7faa8c1ae240 con 0x55faa2a24470 2024-03-28T22:14:12.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.030+0000 7faa9602e740 10 client.0 _ll_register_callbacks cb 0x7faa8800a3f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:14:12.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.030+0000 7faa919e4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55faa2a24d90 0x55faa2a32360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:12.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.030+0000 7faa919e4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55faa2a24d90 0x55faa2a32360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46390/0 (socket says 172.21.15.16:46390) 2024-03-28T22:14:12.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.030+0000 7faa911e3640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55faa2a1eba0 0x55faa2a35a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:12.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.030+0000 7faa919e4640 1 -- 172.21.15.16:0/1504703867 learned_addr learned my addr 172.21.15.16:0/1504703867 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:12.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.030+0000 7faa909e2640 1 --2- 172.21.15.16:0/1504703867 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55faa2a24470 0x55faa2a31e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:12.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.030+0000 7faa919e4640 1 -- 172.21.15.16:0/1504703867 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55faa2a24470 msgr2=0x55faa2a31e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:12.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.030+0000 7faa919e4640 1 --2- 172.21.15.16:0/1504703867 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55faa2a24470 0x55faa2a31e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:12.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.030+0000 7faa919e4640 1 -- 172.21.15.16:0/1504703867 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55faa2a1eba0 msgr2=0x55faa2a35a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:12.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.030+0000 7faa919e4640 1 --2- 172.21.15.16:0/1504703867 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55faa2a1eba0 0x55faa2a35a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:12.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.030+0000 7faa919e4640 1 -- 172.21.15.16:0/1504703867 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55faa2a328a0 con 0x55faa2a24d90 2024-03-28T22:14:12.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.030+0000 7faa911e3640 1 --2- 172.21.15.16:0/1504703867 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55faa2a1eba0 0x55faa2a35a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:14:12.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.030+0000 7faa919e4640 1 --2- 172.21.15.16:0/1504703867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55faa2a24d90 0x55faa2a32360 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7faa84012780 tx=0x7faa84004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec5e3d640cabaf7f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:12.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.030+0000 7faa74ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:14:12.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.030+0000 7faa74ff9640 1 -- 172.21.15.16:0/1504703867 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa84012e50 con 0x55faa2a24d90 2024-03-28T22:14:12.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.030+0000 7faa74ff9640 1 -- 172.21.15.16:0/1504703867 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7faa84009040 con 0x55faa2a24d90 2024-03-28T22:14:12.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.030+0000 7faa9602e740 1 -- 172.21.15.16:0/1504703867 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa880a2f40 con 0x55faa2a24d90 2024-03-28T22:14:12.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.031+0000 7faa74ff9640 1 -- 172.21.15.16:0/1504703867 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7faa8400e960 con 0x55faa2a24d90 2024-03-28T22:14:12.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.031+0000 7faa9602e740 10 client.16284 fetch_fsmap learned FSMap version 25 2024-03-28T22:14:12.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.031+0000 7faa9602e740 1 -- 172.21.15.16:0/1504703867 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55faa2aa4890 con 0x55faa2a24d90 2024-03-28T22:14:12.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.031+0000 7faa74ff9640 1 -- 172.21.15.16:0/1504703867 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa880a2f40 con 0x55faa2a24d90 2024-03-28T22:14:12.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.031+0000 7faa74ff9640 1 -- 172.21.15.16:0/1504703867 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7faa84031070 con 0x55faa2a24d90 2024-03-28T22:14:12.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.031+0000 7faa9602e740 10 client.16284 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T22:14:12.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.031+0000 7faa9602e740 10 client.16284 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:14:12.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.031+0000 7faa9602e740 20 client.16284 populate_metadata read hostname 'smithi016' 2024-03-28T22:14:12.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.031+0000 7faa9602e740 1 --2- 172.21.15.16:0/1504703867 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55faa2a2d970 0x55faa2aa7100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:12.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.031+0000 7faa9602e740 4 client.16284 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:14:12.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.031+0000 7faa9602e740 1 -- 172.21.15.16:0/1504703867 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55faa298fa40 con 0x55faa2a2d970 2024-03-28T22:14:12.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.031+0000 7faa911e3640 1 --2- 172.21.15.16:0/1504703867 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55faa2a2d970 0x55faa2aa7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:12.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.031+0000 7faa74ff9640 10 client.16284 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:14:12.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.031+0000 7faa911e3640 1 --2- 172.21.15.16:0/1504703867 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55faa2a2d970 0x55faa2aa7100 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57807bac0a6e2412 server_cookie=852bd7ffee36c425 in_seq=0 out_seq=0 2024-03-28T22:14:12.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.032+0000 7faa74ff9640 1 -- 172.21.15.16:0/1504703867 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55faa298fa40 con 0x55faa2a2d970 2024-03-28T22:14:12.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.032+0000 7faa74ff9640 10 client.16284 handle_command_reply: tid=0 2024-03-28T22:14:12.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.032+0000 7faa9602e740 1 client.16284 shutdown 2024-03-28T22:14:12.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.032+0000 7faa9602e740 1 -- 172.21.15.16:0/1504703867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55faa2a24d90 msgr2=0x55faa2a32360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:12.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.032+0000 7faa9602e740 1 --2- 172.21.15.16:0/1504703867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55faa2a24d90 0x55faa2a32360 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7faa84012780 tx=0x7faa84004ab0 comp rx=0 tx=0).stop 2024-03-28T22:14:12.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.032+0000 7faa9602e740 1 -- 172.21.15.16:0/1504703867 shutdown_connections 2024-03-28T22:14:12.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.032+0000 7faa9602e740 1 --2- 172.21.15.16:0/1504703867 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55faa2a2d970 0x55faa2aa7100 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:12.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.032+0000 7faa9602e740 1 --2- 172.21.15.16:0/1504703867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55faa2a24d90 0x55faa2a32360 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:12.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.032+0000 7faa74ff9640 0 client.16284 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:14:12.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.032+0000 7faa9602e740 1 --2- 172.21.15.16:0/1504703867 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55faa2a24470 0x55faa2a31e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:12.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.032+0000 7faa9602e740 1 --2- 172.21.15.16:0/1504703867 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55faa2a1eba0 0x55faa2a35a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:12.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.032+0000 7faa9602e740 1 -- 172.21.15.16:0/1504703867 >> 172.21.15.16:0/1504703867 conn(0x55faa2a26370 msgr2=0x55faa2a2a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:12.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.032+0000 7faa9602e740 1 -- 172.21.15.16:0/1504703867 shutdown_connections 2024-03-28T22:14:12.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.032+0000 7faa9602e740 1 -- 172.21.15.16:0/1504703867 wait complete. 2024-03-28T22:14:12.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.032+0000 7faa9602e740 20 client.16284 trim_cache size 0 max 16384 2024-03-28T22:14:12.046 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:14:12.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.032+0000 7faa93447640 1 -- 172.21.15.16:0/1657697634 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7faa780a1ef0 msgr2=0x7faa780a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:12.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.032+0000 7faa93447640 1 --2- 172.21.15.16:0/1657697634 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7faa780a1ef0 0x7faa780a43b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7faa840049d0 tx=0x7faa84011440 comp rx=0 tx=0).stop 2024-03-28T22:14:12.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.032+0000 7faa93447640 1 -- 172.21.15.16:0/1657697634 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa8c102960 msgr2=0x7faa8c19a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:12.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.032+0000 7faa93447640 1 --2- 172.21.15.16:0/1657697634 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa8c102960 0x7faa8c19a410 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7faa8800c970 tx=0x7faa8800ce30 comp rx=0 tx=0).stop 2024-03-28T22:14:12.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.032+0000 7faa93447640 1 -- 172.21.15.16:0/1657697634 shutdown_connections 2024-03-28T22:14:12.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.032+0000 7faa93447640 1 --2- 172.21.15.16:0/1657697634 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7faa780a1ef0 0x7faa780a43b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:12.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.032+0000 7faa93447640 1 --2- 172.21.15.16:0/1657697634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa8c19ecd0 0x7faa8c19f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:12.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.032+0000 7faa93447640 1 --2- 172.21.15.16:0/1657697634 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa8c107e50 0x7faa8c19a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:12.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.032+0000 7faa93447640 1 --2- 172.21.15.16:0/1657697634 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa8c102960 0x7faa8c19a410 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:12.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.033+0000 7faa93447640 1 -- 172.21.15.16:0/1657697634 >> 172.21.15.16:0/1657697634 conn(0x7faa8c077830 msgr2=0x7faa8c105fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:12.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.033+0000 7faa93447640 1 -- 172.21.15.16:0/1657697634 shutdown_connections 2024-03-28T22:14:12.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:12.033+0000 7faa93447640 1 -- 172.21.15.16:0/1657697634 wait complete. 2024-03-28T22:14:12.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:12 smithi016 ceph-mon[32374]: pgmap v111: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 216 KiB/s wr, 23 op/s 2024-03-28T22:14:12.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:12 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/1997784913' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:14:12.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:12 smithi063 ceph-mon[36249]: pgmap v111: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 216 KiB/s wr, 23 op/s 2024-03-28T22:14:12.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:12 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/1997784913' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:14:12.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:12 smithi053 ceph-mon[35979]: pgmap v111: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 216 KiB/s wr, 23 op/s 2024-03-28T22:14:12.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:12 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/1997784913' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:14:13.049 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:14:13.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:14:13.050 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:14:13.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.168+0000 7f88f3230640 1 -- 172.21.15.16:0/1257969005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88ec1021e0 msgr2=0x7f88ec102640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:13.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.168+0000 7f88f3230640 1 --2- 172.21.15.16:0/1257969005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88ec1021e0 0x7f88ec102640 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f88d8009900 tx=0x7f88d802bd10 comp rx=0 tx=0).stop 2024-03-28T22:14:13.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.168+0000 7f88f3230640 1 -- 172.21.15.16:0/1257969005 shutdown_connections 2024-03-28T22:14:13.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.168+0000 7f88f3230640 1 --2- 172.21.15.16:0/1257969005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88ec102b80 0x7f88ec10d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.168+0000 7f88f3230640 1 --2- 172.21.15.16:0/1257969005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88ec1021e0 0x7f88ec102640 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.168+0000 7f88f3230640 1 --2- 172.21.15.16:0/1257969005 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88ec1009d0 0x7f88ec100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.168+0000 7f88f3230640 1 -- 172.21.15.16:0/1257969005 >> 172.21.15.16:0/1257969005 conn(0x7f88ec077830 msgr2=0x7f88ec077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:13.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.168+0000 7f88f3230640 1 -- 172.21.15.16:0/1257969005 shutdown_connections 2024-03-28T22:14:13.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.168+0000 7f88f3230640 1 -- 172.21.15.16:0/1257969005 wait complete. 2024-03-28T22:14:13.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.168+0000 7f88f3230640 1 Processor -- start 2024-03-28T22:14:13.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.169+0000 7f88f3230640 1 -- start start 2024-03-28T22:14:13.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.169+0000 7f88f3230640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88ec1009d0 0x7f88ec198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:13.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.169+0000 7f88f3230640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88ec1021e0 0x7f88ec198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:13.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.169+0000 7f88f3230640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88ec102b80 0x7f88ec19c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:13.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.169+0000 7f88f3230640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f88ec110b20 con 0x7f88ec102b80 2024-03-28T22:14:13.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.169+0000 7f88f3230640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f88ec1109a0 con 0x7f88ec1009d0 2024-03-28T22:14:13.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.169+0000 7f88f3230640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f88ec110ca0 con 0x7f88ec1021e0 2024-03-28T22:14:13.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.169+0000 7f88f17cd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88ec102b80 0x7f88ec19c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:13.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.169+0000 7f88f17cd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88ec102b80 0x7f88ec19c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46406/0 (socket says 172.21.15.16:46406) 2024-03-28T22:14:13.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.169+0000 7f88f17cd640 1 -- 172.21.15.16:0/145114924 learned_addr learned my addr 172.21.15.16:0/145114924 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:13.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.169+0000 7f88f0fcc640 1 --2- 172.21.15.16:0/145114924 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88ec1009d0 0x7f88ec198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:13.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.169+0000 7f88ebfff640 1 --2- 172.21.15.16:0/145114924 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88ec1021e0 0x7f88ec198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:13.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.169+0000 7f88f17cd640 1 -- 172.21.15.16:0/145114924 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88ec1021e0 msgr2=0x7f88ec198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:13.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.169+0000 7f88f17cd640 1 --2- 172.21.15.16:0/145114924 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88ec1021e0 0x7f88ec198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.169+0000 7f88f17cd640 1 -- 172.21.15.16:0/145114924 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88ec1009d0 msgr2=0x7f88ec198120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:13.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.169+0000 7f88f17cd640 1 --2- 172.21.15.16:0/145114924 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88ec1009d0 0x7f88ec198120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.169+0000 7f88f17cd640 1 -- 172.21.15.16:0/145114924 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88ec19d0c0 con 0x7f88ec102b80 2024-03-28T22:14:13.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.169+0000 7f88ebfff640 1 --2- 172.21.15.16:0/145114924 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88ec1021e0 0x7f88ec198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:14:13.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.169+0000 7f88f0fcc640 1 --2- 172.21.15.16:0/145114924 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88ec1009d0 0x7f88ec198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:14:13.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.169+0000 7f88f17cd640 1 --2- 172.21.15.16:0/145114924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88ec102b80 0x7f88ec19c990 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f88e400ec70 tx=0x7f88e400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:13.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.170+0000 7f88e9ffb640 1 -- 172.21.15.16:0/145114924 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e4003b70 con 0x7f88ec102b80 2024-03-28T22:14:13.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.170+0000 7f88f3230640 1 -- 172.21.15.16:0/145114924 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88ec19d3b0 con 0x7f88ec102b80 2024-03-28T22:14:13.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.170+0000 7f88e9ffb640 1 -- 172.21.15.16:0/145114924 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88e400ee80 con 0x7f88ec102b80 2024-03-28T22:14:13.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.170+0000 7f88e9ffb640 1 -- 172.21.15.16:0/145114924 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e40108a0 con 0x7f88ec102b80 2024-03-28T22:14:13.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.170+0000 7f88f3230640 1 -- 172.21.15.16:0/145114924 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88ec1a4c40 con 0x7f88ec102b80 2024-03-28T22:14:13.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.172+0000 7f88e9ffb640 1 -- 172.21.15.16:0/145114924 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f88e4004020 con 0x7f88ec102b80 2024-03-28T22:14:13.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.173+0000 7f88f5e0c740 1 Processor -- start 2024-03-28T22:14:13.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.173+0000 7f88e9ffb640 1 --2- 172.21.15.16:0/145114924 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f88dc0a1ef0 0x7f88dc0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:13.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.173+0000 7f88f5e0c740 1 -- start start 2024-03-28T22:14:13.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.173+0000 7f88e9ffb640 1 -- 172.21.15.16:0/145114924 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f88e40ce590 con 0x7f88ec102b80 2024-03-28T22:14:13.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.173+0000 7f88ebfff640 1 --2- 172.21.15.16:0/145114924 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f88dc0a1ef0 0x7f88dc0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:13.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.173+0000 7f88f5e0c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5614c30e93d0 0x5614c30e97b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:13.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.173+0000 7f88f5e0c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614c30e9d80 0x5614c30f6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:13.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.173+0000 7f88f5e0c740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614c30f7300 0x5614c30f96f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:13.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.173+0000 7f88f5e0c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5614c2fea0b0 con 0x5614c30e93d0 2024-03-28T22:14:13.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.173+0000 7f88f5e0c740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5614c304cea0 con 0x5614c30f7300 2024-03-28T22:14:13.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.173+0000 7f88f5e0c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5614c2ff5c00 con 0x5614c30e9d80 2024-03-28T22:14:13.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.173+0000 7f88f0fcc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5614c30e93d0 0x5614c30e97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:13.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.173+0000 7f88f0fcc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5614c30e93d0 0x5614c30e97b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46410/0 (socket says 172.21.15.16:46410) 2024-03-28T22:14:13.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.173+0000 7f88f0fcc640 1 -- 172.21.15.16:0/475886539 learned_addr learned my addr 172.21.15.16:0/475886539 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:13.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.173+0000 7f88f17cd640 1 --2- 172.21.15.16:0/475886539 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614c30f7300 0x5614c30f96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:13.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.173+0000 7f88ebfff640 1 --2- 172.21.15.16:0/475886539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614c30e9d80 0x5614c30f6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:13.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.173+0000 7f88f0fcc640 1 -- 172.21.15.16:0/475886539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614c30e9d80 msgr2=0x5614c30f6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:13.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.173+0000 7f88ebfff640 1 --2- 172.21.15.16:0/145114924 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f88dc0a1ef0 0x7f88dc0a43b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f88ec077fe0 tx=0x7f88d802c870 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:13.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.173+0000 7f88f0fcc640 1 --2- 172.21.15.16:0/475886539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614c30e9d80 0x5614c30f6dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.173+0000 7f88f0fcc640 1 -- 172.21.15.16:0/475886539 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614c30f7300 msgr2=0x5614c30f96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:13.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.173+0000 7f88f0fcc640 1 --2- 172.21.15.16:0/475886539 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614c30f7300 0x5614c30f96f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.173+0000 7f88f0fcc640 1 -- 172.21.15.16:0/475886539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614c2e297e0 con 0x5614c30e93d0 2024-03-28T22:14:13.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.173+0000 7f88f17cd640 1 --2- 172.21.15.16:0/475886539 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614c30f7300 0x5614c30f96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:14:13.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.174+0000 7f88f0fcc640 1 --2- 172.21.15.16:0/475886539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5614c30e93d0 0x5614c30e97b0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f88e000f3c0 tx=0x7f88e000f880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddefb5e25160ef1d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:13.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.174+0000 7f88d6ffd640 1 -- 172.21.15.16:0/475886539 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e001d070 con 0x5614c30e93d0 2024-03-28T22:14:13.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.174+0000 7f88d6ffd640 1 -- 172.21.15.16:0/475886539 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88e0010de0 con 0x5614c30e93d0 2024-03-28T22:14:13.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.174+0000 7f88d6ffd640 1 -- 172.21.15.16:0/475886539 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e0016340 con 0x5614c30e93d0 2024-03-28T22:14:13.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.174+0000 7f88f5e0c740 1 -- 172.21.15.16:0/475886539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5614c30e93d0 msgr2=0x5614c30e97b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:13.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.174+0000 7f88f5e0c740 1 --2- 172.21.15.16:0/475886539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5614c30e93d0 0x5614c30e97b0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f88e000f3c0 tx=0x7f88e000f880 comp rx=0 tx=0).stop 2024-03-28T22:14:13.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.174+0000 7f88f5e0c740 1 -- 172.21.15.16:0/475886539 shutdown_connections 2024-03-28T22:14:13.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.174+0000 7f88f5e0c740 1 --2- 172.21.15.16:0/475886539 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614c30f7300 0x5614c30f96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.174+0000 7f88f5e0c740 1 --2- 172.21.15.16:0/475886539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614c30e9d80 0x5614c30f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.174+0000 7f88f5e0c740 1 --2- 172.21.15.16:0/475886539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5614c30e93d0 0x5614c30e97b0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.174+0000 7f88f5e0c740 1 -- 172.21.15.16:0/475886539 >> 172.21.15.16:0/475886539 conn(0x5614c30e4120 msgr2=0x5614c30e4500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:13.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.175+0000 7f88f5e0c740 1 -- 172.21.15.16:0/475886539 shutdown_connections 2024-03-28T22:14:13.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.175+0000 7f88f5e0c740 1 -- 172.21.15.16:0/475886539 wait complete. 2024-03-28T22:14:13.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.175+0000 7f88f5e0c740 1 -- start start 2024-03-28T22:14:13.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.175+0000 7f88f5e0c740 1 Processor -- start 2024-03-28T22:14:13.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.175+0000 7f88f5e0c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614c30e93d0 0x5614c30f4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:13.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.175+0000 7f88f5e0c740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614c30e9d80 0x5614c30f4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:13.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.175+0000 7f88f5e0c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5614c30f7300 0x5614c30f4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:13.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.175+0000 7f88f5e0c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5614c2feb3d0 con 0x5614c30f7300 2024-03-28T22:14:13.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.175+0000 7f88f5e0c740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5614c3048f40 con 0x5614c30e9d80 2024-03-28T22:14:13.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.175+0000 7f88f5e0c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5614c30fa0c0 con 0x5614c30e93d0 2024-03-28T22:14:13.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.175+0000 7f88f5e0c740 10 client.0 _ll_register_callbacks cb 0x5614c3008b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:14:13.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.176+0000 7f88f17cd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5614c30f7300 0x5614c30f4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:13.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.176+0000 7f88ebfff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614c30e9d80 0x5614c30f4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:13.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.176+0000 7f88f17cd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5614c30f7300 0x5614c30f4d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46412/0 (socket says 172.21.15.16:46412) 2024-03-28T22:14:13.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.176+0000 7f88f17cd640 1 -- 172.21.15.16:0/205605051 learned_addr learned my addr 172.21.15.16:0/205605051 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:13.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.176+0000 7f88f0fcc640 1 --2- 172.21.15.16:0/205605051 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614c30e93d0 0x5614c30f4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:13.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.176+0000 7f88f17cd640 1 -- 172.21.15.16:0/205605051 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614c30e93d0 msgr2=0x5614c30f4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:13.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.176+0000 7f88f17cd640 1 --2- 172.21.15.16:0/205605051 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614c30e93d0 0x5614c30f4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.176+0000 7f88f17cd640 1 -- 172.21.15.16:0/205605051 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614c30e9d80 msgr2=0x5614c30f4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:13.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.176+0000 7f88f17cd640 1 --2- 172.21.15.16:0/205605051 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614c30e9d80 0x5614c30f4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.176+0000 7f88f17cd640 1 -- 172.21.15.16:0/205605051 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614c2e89f70 con 0x5614c30f7300 2024-03-28T22:14:13.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.176+0000 7f88ebfff640 1 --2- 172.21.15.16:0/205605051 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614c30e9d80 0x5614c30f4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:14:13.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.176+0000 7f88f17cd640 1 --2- 172.21.15.16:0/205605051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5614c30f7300 0x5614c30f4d80 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f88e4097970 tx=0x7f88e409afa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=886695a76c3e5060 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:13.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.176+0000 7f88cbfff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:14:13.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.176+0000 7f88cbfff640 1 -- 172.21.15.16:0/205605051 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e40b8060 con 0x5614c30f7300 2024-03-28T22:14:13.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.176+0000 7f88cbfff640 1 -- 172.21.15.16:0/205605051 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f88e4004b40 con 0x5614c30f7300 2024-03-28T22:14:13.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.176+0000 7f88f5e0c740 1 -- 172.21.15.16:0/205605051 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88e0010de0 con 0x5614c30f7300 2024-03-28T22:14:13.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.176+0000 7f88cbfff640 1 -- 172.21.15.16:0/205605051 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f88e4004d40 con 0x5614c30f7300 2024-03-28T22:14:13.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.176+0000 7f88f5e0c740 10 client.16296 fetch_fsmap learned FSMap version 25 2024-03-28T22:14:13.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.176+0000 7f88f5e0c740 1 -- 172.21.15.16:0/205605051 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5614c2d21170 con 0x5614c30f7300 2024-03-28T22:14:13.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.177+0000 7f88cbfff640 1 -- 172.21.15.16:0/205605051 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88e0010de0 con 0x5614c30f7300 2024-03-28T22:14:13.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.177+0000 7f88cbfff640 1 -- 172.21.15.16:0/205605051 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f88e401f020 con 0x5614c30f7300 2024-03-28T22:14:13.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.177+0000 7f88f5e0c740 10 client.16296 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T22:14:13.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.177+0000 7f88f5e0c740 10 client.16296 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:14:13.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.177+0000 7f88f5e0c740 20 client.16296 populate_metadata read hostname 'smithi016' 2024-03-28T22:14:13.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.177+0000 7f88f5e0c740 1 --2- 172.21.15.16:0/205605051 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5614c3107170 0x5614c3109560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:13.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.177+0000 7f88f5e0c740 4 client.16296 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:14:13.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.177+0000 7f88f5e0c740 1 -- 172.21.15.16:0/205605051 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5614c305a3e0 con 0x5614c3107170 2024-03-28T22:14:13.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.177+0000 7f88f0fcc640 1 --2- 172.21.15.16:0/205605051 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5614c3107170 0x5614c3109560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:13.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.177+0000 7f88f0fcc640 1 --2- 172.21.15.16:0/205605051 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5614c3107170 0x5614c3109560 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a32923853723ee4 server_cookie=f66cef76fa0aec4e in_seq=0 out_seq=0 2024-03-28T22:14:13.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.177+0000 7f88cbfff640 10 client.16296 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:14:13.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.178+0000 7f88cbfff640 1 -- 172.21.15.16:0/205605051 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5614c305a3e0 con 0x5614c3107170 2024-03-28T22:14:13.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.178+0000 7f88cbfff640 10 client.16296 handle_command_reply: tid=0 2024-03-28T22:14:13.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.178+0000 7f88f5e0c740 1 client.16296 shutdown 2024-03-28T22:14:13.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.178+0000 7f88f5e0c740 1 -- 172.21.15.16:0/205605051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5614c30f7300 msgr2=0x5614c30f4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:13.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.178+0000 7f88f5e0c740 1 --2- 172.21.15.16:0/205605051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5614c30f7300 0x5614c30f4d80 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f88e4097970 tx=0x7f88e409afa0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.178+0000 7f88f5e0c740 1 -- 172.21.15.16:0/205605051 shutdown_connections 2024-03-28T22:14:13.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.178+0000 7f88f5e0c740 1 --2- 172.21.15.16:0/205605051 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5614c3107170 0x5614c3109560 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.178+0000 7f88f5e0c740 1 --2- 172.21.15.16:0/205605051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5614c30f7300 0x5614c30f4d80 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.178+0000 7f88f5e0c740 1 --2- 172.21.15.16:0/205605051 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614c30e9d80 0x5614c30f4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.178+0000 7f88cbfff640 0 client.16296 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:14:13.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.178+0000 7f88f5e0c740 1 --2- 172.21.15.16:0/205605051 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614c30e93d0 0x5614c30f4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.178+0000 7f88f5e0c740 1 -- 172.21.15.16:0/205605051 >> 172.21.15.16:0/205605051 conn(0x5614c30e4120 msgr2=0x5614c30e5e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:13.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.178+0000 7f88f5e0c740 1 -- 172.21.15.16:0/205605051 shutdown_connections 2024-03-28T22:14:13.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.179+0000 7f88f5e0c740 1 -- 172.21.15.16:0/205605051 wait complete. 2024-03-28T22:14:13.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.179+0000 7f88f5e0c740 20 client.16296 trim_cache size 0 max 16384 2024-03-28T22:14:13.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.196+0000 7f88f5e0c740 1 Processor -- start 2024-03-28T22:14:13.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.196+0000 7f88f5e0c740 1 -- start start 2024-03-28T22:14:13.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.196+0000 7f88f5e0c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614c30e93d0 0x5614c3102d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:13.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.196+0000 7f88f5e0c740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614c30e9d80 0x5614c30e3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:13.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.196+0000 7f88f5e0c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5614c30f7300 0x5614c310ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:13.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.196+0000 7f88f5e0c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f88ec1adae0 con 0x5614c30f7300 2024-03-28T22:14:13.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.196+0000 7f88f5e0c740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f88ec1ab380 con 0x5614c30e9d80 2024-03-28T22:14:13.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.196+0000 7f88f5e0c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f88ec1a91d0 con 0x5614c30e93d0 2024-03-28T22:14:13.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.196+0000 7f88f17cd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5614c30f7300 0x5614c310ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:13.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.197+0000 7f88ebfff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614c30e9d80 0x5614c30e3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:13.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.197+0000 7f88f17cd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5614c30f7300 0x5614c310ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46418/0 (socket says 172.21.15.16:46418) 2024-03-28T22:14:13.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.197+0000 7f88f17cd640 1 -- 172.21.15.16:0/2255235617 learned_addr learned my addr 172.21.15.16:0/2255235617 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:13.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.197+0000 7f88f0fcc640 1 --2- 172.21.15.16:0/2255235617 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614c30e93d0 0x5614c3102d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:13.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.197+0000 7f88f17cd640 1 -- 172.21.15.16:0/2255235617 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614c30e93d0 msgr2=0x5614c3102d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:13.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.197+0000 7f88f17cd640 1 --2- 172.21.15.16:0/2255235617 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614c30e93d0 0x5614c3102d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.197+0000 7f88f17cd640 1 -- 172.21.15.16:0/2255235617 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614c30e9d80 msgr2=0x5614c30e3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:13.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.197+0000 7f88f17cd640 1 --2- 172.21.15.16:0/2255235617 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614c30e9d80 0x5614c30e3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.197+0000 7f88f17cd640 1 -- 172.21.15.16:0/2255235617 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614c31098d0 con 0x5614c30f7300 2024-03-28T22:14:13.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.197+0000 7f88f0fcc640 1 --2- 172.21.15.16:0/2255235617 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614c30e93d0 0x5614c3102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:14:13.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.197+0000 7f88ebfff640 1 --2- 172.21.15.16:0/2255235617 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614c30e9d80 0x5614c30e3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:14:13.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.197+0000 7f88f17cd640 1 --2- 172.21.15.16:0/2255235617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5614c30f7300 0x5614c310ee30 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f88e4099250 tx=0x7f88e40994b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e19b81604a2b3f4d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:13.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.197+0000 7f88d67fc640 1 -- 172.21.15.16:0/2255235617 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e40b8070 con 0x5614c30f7300 2024-03-28T22:14:13.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.197+0000 7f88d67fc640 1 -- 172.21.15.16:0/2255235617 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88e4096e60 con 0x5614c30f7300 2024-03-28T22:14:13.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.197+0000 7f88d67fc640 1 -- 172.21.15.16:0/2255235617 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e40b94b0 con 0x5614c30f7300 2024-03-28T22:14:13.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.198+0000 7f88f5e0c740 1 -- 172.21.15.16:0/2255235617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5614c30f7300 msgr2=0x5614c310ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:13.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.198+0000 7f88f5e0c740 1 --2- 172.21.15.16:0/2255235617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5614c30f7300 0x5614c310ee30 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f88e4099250 tx=0x7f88e40994b0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.198+0000 7f88f5e0c740 1 -- 172.21.15.16:0/2255235617 shutdown_connections 2024-03-28T22:14:13.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.198+0000 7f88f5e0c740 1 --2- 172.21.15.16:0/2255235617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5614c30f7300 0x5614c310ee30 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.198+0000 7f88f5e0c740 1 --2- 172.21.15.16:0/2255235617 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614c30e9d80 0x5614c30e3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.198+0000 7f88f5e0c740 1 --2- 172.21.15.16:0/2255235617 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614c30e93d0 0x5614c3102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.198+0000 7f88f5e0c740 1 -- 172.21.15.16:0/2255235617 >> 172.21.15.16:0/2255235617 conn(0x5614c30e4120 msgr2=0x5614c30e28f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:13.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.198+0000 7f88f5e0c740 1 -- 172.21.15.16:0/2255235617 shutdown_connections 2024-03-28T22:14:13.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.198+0000 7f88f5e0c740 1 -- 172.21.15.16:0/2255235617 wait complete. 2024-03-28T22:14:13.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.198+0000 7f88f5e0c740 1 -- start start 2024-03-28T22:14:13.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.198+0000 7f88f5e0c740 1 Processor -- start 2024-03-28T22:14:13.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.198+0000 7f88f5e0c740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614c30e93d0 0x5614c30fb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:13.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.198+0000 7f88f5e0c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614c30e9d80 0x5614c30fb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:13.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.198+0000 7f88f5e0c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5614c30f7300 0x5614c316c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:13.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.199+0000 7f88f5e0c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f88ec1a9840 con 0x5614c30f7300 2024-03-28T22:14:13.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.199+0000 7f88f5e0c740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f88ec1aacb0 con 0x5614c30e93d0 2024-03-28T22:14:13.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.199+0000 7f88f5e0c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f88ec1a9f10 con 0x5614c30e9d80 2024-03-28T22:14:13.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.199+0000 7f88f5e0c740 10 client.0 _ll_register_callbacks cb 0x7f88e40098f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:14:13.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.199+0000 7f88f17cd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5614c30f7300 0x5614c316c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:13.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.199+0000 7f88f17cd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5614c30f7300 0x5614c316c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46422/0 (socket says 172.21.15.16:46422) 2024-03-28T22:14:13.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.199+0000 7f88f17cd640 1 -- 172.21.15.16:0/3445763500 learned_addr learned my addr 172.21.15.16:0/3445763500 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:13.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.199+0000 7f88f0fcc640 1 --2- 172.21.15.16:0/3445763500 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614c30e93d0 0x5614c30fb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:13.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.199+0000 7f88ebfff640 1 --2- 172.21.15.16:0/3445763500 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614c30e9d80 0x5614c30fb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:13.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.199+0000 7f88f17cd640 1 -- 172.21.15.16:0/3445763500 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614c30e9d80 msgr2=0x5614c30fb8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:13.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.199+0000 7f88f17cd640 1 --2- 172.21.15.16:0/3445763500 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614c30e9d80 0x5614c30fb8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.199+0000 7f88f17cd640 1 -- 172.21.15.16:0/3445763500 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614c30e93d0 msgr2=0x5614c30fb3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:13.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.199+0000 7f88f17cd640 1 --2- 172.21.15.16:0/3445763500 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614c30e93d0 0x5614c30fb3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.199+0000 7f88f17cd640 1 -- 172.21.15.16:0/3445763500 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614c316c9a0 con 0x5614c30f7300 2024-03-28T22:14:13.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.199+0000 7f88f0fcc640 1 --2- 172.21.15.16:0/3445763500 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614c30e93d0 0x5614c30fb3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:14:13.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.199+0000 7f88f17cd640 1 --2- 172.21.15.16:0/3445763500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5614c30f7300 0x5614c316c460 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f88e409af30 tx=0x7f88e400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41caa0f634f1b331 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:13.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.199+0000 7f88cbfff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:14:13.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.199+0000 7f88cbfff640 1 -- 172.21.15.16:0/3445763500 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e407e070 con 0x5614c30f7300 2024-03-28T22:14:13.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.199+0000 7f88cbfff640 1 -- 172.21.15.16:0/3445763500 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f88e4097010 con 0x5614c30f7300 2024-03-28T22:14:13.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.199+0000 7f88f5e0c740 1 -- 172.21.15.16:0/3445763500 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88e4096e60 con 0x5614c30f7300 2024-03-28T22:14:13.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.199+0000 7f88cbfff640 1 -- 172.21.15.16:0/3445763500 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f88e40b8070 con 0x5614c30f7300 2024-03-28T22:14:13.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.199+0000 7f88f5e0c740 10 client.16302 fetch_fsmap learned FSMap version 25 2024-03-28T22:14:13.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.199+0000 7f88f5e0c740 1 -- 172.21.15.16:0/3445763500 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5614c316dce0 con 0x5614c30f7300 2024-03-28T22:14:13.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.200+0000 7f88cbfff640 1 -- 172.21.15.16:0/3445763500 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88e4096e60 con 0x5614c30f7300 2024-03-28T22:14:13.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.200+0000 7f88cbfff640 1 -- 172.21.15.16:0/3445763500 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f88e401f070 con 0x5614c30f7300 2024-03-28T22:14:13.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.200+0000 7f88f5e0c740 10 client.16302 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T22:14:13.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.200+0000 7f88f5e0c740 10 client.16302 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:14:13.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.200+0000 7f88f5e0c740 20 client.16302 populate_metadata read hostname 'smithi016' 2024-03-28T22:14:13.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.200+0000 7f88f5e0c740 1 --2- 172.21.15.16:0/3445763500 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5614c316e200 0x5614c31705f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:13.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.200+0000 7f88f5e0c740 4 client.16302 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:14:13.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.200+0000 7f88f5e0c740 1 -- 172.21.15.16:0/3445763500 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5614c3054730 con 0x5614c316e200 2024-03-28T22:14:13.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.200+0000 7f88f0fcc640 1 --2- 172.21.15.16:0/3445763500 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5614c316e200 0x5614c31705f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:13.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.200+0000 7f88cbfff640 10 client.16302 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:14:13.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.200+0000 7f88f0fcc640 1 --2- 172.21.15.16:0/3445763500 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5614c316e200 0x5614c31705f0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c400faa7a88f8475 server_cookie=2f7176a846853a2c in_seq=0 out_seq=0 2024-03-28T22:14:13.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.202+0000 7f88cbfff640 1 -- 172.21.15.16:0/3445763500 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5614c3054730 con 0x5614c316e200 2024-03-28T22:14:13.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.202+0000 7f88cbfff640 10 client.16302 handle_command_reply: tid=0 2024-03-28T22:14:13.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.202+0000 7f88f5e0c740 1 client.16302 shutdown 2024-03-28T22:14:13.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.202+0000 7f88f5e0c740 1 -- 172.21.15.16:0/3445763500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5614c30f7300 msgr2=0x5614c316c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:13.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.202+0000 7f88f5e0c740 1 --2- 172.21.15.16:0/3445763500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5614c30f7300 0x5614c316c460 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f88e409af30 tx=0x7f88e400cf60 comp rx=0 tx=0).stop 2024-03-28T22:14:13.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.202+0000 7f88f5e0c740 1 -- 172.21.15.16:0/3445763500 shutdown_connections 2024-03-28T22:14:13.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.202+0000 7f88f5e0c740 1 --2- 172.21.15.16:0/3445763500 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5614c316e200 0x5614c31705f0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.202+0000 7f88f5e0c740 1 --2- 172.21.15.16:0/3445763500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5614c30f7300 0x5614c316c460 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.202+0000 7f88f5e0c740 1 --2- 172.21.15.16:0/3445763500 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614c30e9d80 0x5614c30fb8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.202+0000 7f88cbfff640 0 client.16302 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:14:13.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.202+0000 7f88f5e0c740 1 --2- 172.21.15.16:0/3445763500 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5614c30e93d0 0x5614c30fb3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.202+0000 7f88f5e0c740 1 -- 172.21.15.16:0/3445763500 >> 172.21.15.16:0/3445763500 conn(0x5614c30e4120 msgr2=0x5614c30e54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:13.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.202+0000 7f88f5e0c740 1 -- 172.21.15.16:0/3445763500 shutdown_connections 2024-03-28T22:14:13.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.202+0000 7f88f5e0c740 1 -- 172.21.15.16:0/3445763500 wait complete. 2024-03-28T22:14:13.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.202+0000 7f88f5e0c740 20 client.16302 trim_cache size 0 max 16384 2024-03-28T22:14:13.216 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:14:13.216 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-03-28T22:14:13.216 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "1590a3c8-388b-4c8e-91dd-31dd85ab4312", 2024-03-28T22:14:13.216 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-03-28T22:14:13.216 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:14:13.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.203+0000 7f88f3230640 1 -- 172.21.15.16:0/145114924 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f88dc0a1ef0 msgr2=0x7f88dc0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:13.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.203+0000 7f88f3230640 1 --2- 172.21.15.16:0/145114924 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f88dc0a1ef0 0x7f88dc0a43b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f88ec077fe0 tx=0x7f88d802c870 comp rx=0 tx=0).stop 2024-03-28T22:14:13.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.203+0000 7f88f3230640 1 -- 172.21.15.16:0/145114924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88ec102b80 msgr2=0x7f88ec19c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:13.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.203+0000 7f88f3230640 1 --2- 172.21.15.16:0/145114924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88ec102b80 0x7f88ec19c990 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f88e400ec70 tx=0x7f88e400c6a0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.203+0000 7f88f3230640 1 -- 172.21.15.16:0/145114924 shutdown_connections 2024-03-28T22:14:13.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.203+0000 7f88f3230640 1 --2- 172.21.15.16:0/145114924 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f88dc0a1ef0 0x7f88dc0a43b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.203+0000 7f88f3230640 1 --2- 172.21.15.16:0/145114924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88ec102b80 0x7f88ec19c990 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.203+0000 7f88f3230640 1 --2- 172.21.15.16:0/145114924 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88ec1021e0 0x7f88ec198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.203+0000 7f88f3230640 1 --2- 172.21.15.16:0/145114924 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88ec1009d0 0x7f88ec198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.203+0000 7f88f3230640 1 -- 172.21.15.16:0/145114924 >> 172.21.15.16:0/145114924 conn(0x7f88ec077830 msgr2=0x7f88ec06a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:13.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.203+0000 7f88f3230640 1 -- 172.21.15.16:0/145114924 shutdown_connections 2024-03-28T22:14:13.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.203+0000 7f88f3230640 1 -- 172.21.15.16:0/145114924 wait complete. 2024-03-28T22:14:13.220 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:14:13.220 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:14:13.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.325+0000 7fabf30d6640 1 -- 172.21.15.16:0/2046613281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabec1005e0 msgr2=0x7fabec100a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:13.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.325+0000 7fabf30d6640 1 --2- 172.21.15.16:0/2046613281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabec1005e0 0x7fabec100a60 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fabd8009930 tx=0x7fabd802bc30 comp rx=0 tx=0).stop 2024-03-28T22:14:13.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.325+0000 7fabf30d6640 1 -- 172.21.15.16:0/2046613281 shutdown_connections 2024-03-28T22:14:13.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.325+0000 7fabf30d6640 1 --2- 172.21.15.16:0/2046613281 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fabec100fa0 0x7fabec10d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.325+0000 7fabf30d6640 1 --2- 172.21.15.16:0/2046613281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabec1005e0 0x7fabec100a60 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.325+0000 7fabf30d6640 1 --2- 172.21.15.16:0/2046613281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabec106680 0x7fabec106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.325+0000 7fabf30d6640 1 -- 172.21.15.16:0/2046613281 >> 172.21.15.16:0/2046613281 conn(0x7fabec0fc0b0 msgr2=0x7fabec0fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:13.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.325+0000 7fabf30d6640 1 -- 172.21.15.16:0/2046613281 shutdown_connections 2024-03-28T22:14:13.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.325+0000 7fabf30d6640 1 -- 172.21.15.16:0/2046613281 wait complete. 2024-03-28T22:14:13.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.326+0000 7fabf30d6640 1 Processor -- start 2024-03-28T22:14:13.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.326+0000 7fabf30d6640 1 -- start start 2024-03-28T22:14:13.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.326+0000 7fabf30d6640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabec1005e0 0x7fabec198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:13.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.326+0000 7fabf30d6640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fabec100fa0 0x7fabec198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:13.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.326+0000 7fabf30d6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabec106680 0x7fabec19c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:13.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.326+0000 7fabf30d6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fabec110a00 con 0x7fabec106680 2024-03-28T22:14:13.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.326+0000 7fabf30d6640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fabec110880 con 0x7fabec1005e0 2024-03-28T22:14:13.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.326+0000 7fabf30d6640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fabec110b80 con 0x7fabec100fa0 2024-03-28T22:14:13.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.326+0000 7fabf1673640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabec106680 0x7fabec19c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:13.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.326+0000 7fabf1673640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabec106680 0x7fabec19c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46446/0 (socket says 172.21.15.16:46446) 2024-03-28T22:14:13.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.326+0000 7fabf1673640 1 -- 172.21.15.16:0/91452433 learned_addr learned my addr 172.21.15.16:0/91452433 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:13.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.326+0000 7fabf0e72640 1 --2- 172.21.15.16:0/91452433 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabec1005e0 0x7fabec198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:13.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.326+0000 7fabebfff640 1 --2- 172.21.15.16:0/91452433 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fabec100fa0 0x7fabec198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:13.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.326+0000 7fabf1673640 1 -- 172.21.15.16:0/91452433 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fabec100fa0 msgr2=0x7fabec198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:13.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.326+0000 7fabf1673640 1 --2- 172.21.15.16:0/91452433 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fabec100fa0 0x7fabec198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.326+0000 7fabf1673640 1 -- 172.21.15.16:0/91452433 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabec1005e0 msgr2=0x7fabec198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:13.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.326+0000 7fabf1673640 1 --2- 172.21.15.16:0/91452433 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabec1005e0 0x7fabec198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.326+0000 7fabf1673640 1 -- 172.21.15.16:0/91452433 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabec19d000 con 0x7fabec106680 2024-03-28T22:14:13.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.326+0000 7fabebfff640 1 --2- 172.21.15.16:0/91452433 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fabec100fa0 0x7fabec198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:14:13.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.326+0000 7fabf1673640 1 --2- 172.21.15.16:0/91452433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabec106680 0x7fabec19c8d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fabe400eb10 tx=0x7fabe400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:13.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.327+0000 7fabe9ffb640 1 -- 172.21.15.16:0/91452433 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe4004b20 con 0x7fabec106680 2024-03-28T22:14:13.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.327+0000 7fabe9ffb640 1 -- 172.21.15.16:0/91452433 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabe4004cc0 con 0x7fabec106680 2024-03-28T22:14:13.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.327+0000 7fabf30d6640 1 -- 172.21.15.16:0/91452433 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabec19d2f0 con 0x7fabec106680 2024-03-28T22:14:13.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.327+0000 7fabe9ffb640 1 -- 172.21.15.16:0/91452433 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe4010760 con 0x7fabec106680 2024-03-28T22:14:13.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.327+0000 7fabf30d6640 1 -- 172.21.15.16:0/91452433 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabec1a4b80 con 0x7fabec106680 2024-03-28T22:14:13.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.329+0000 7fabe9ffb640 1 -- 172.21.15.16:0/91452433 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7fabe4005020 con 0x7fabec106680 2024-03-28T22:14:13.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.330+0000 7fabf5d8a740 1 Processor -- start 2024-03-28T22:14:13.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.330+0000 7fabf5d8a740 1 -- start start 2024-03-28T22:14:13.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.330+0000 7fabf5d8a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e2160c290 0x559e2160c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:13.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.330+0000 7fabf5d8a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e2160cbb0 0x559e21615250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:13.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.330+0000 7fabf5d8a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e21615790 0x559e21617b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:13.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.330+0000 7fabf5d8a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559e21570180 con 0x559e21615790 2024-03-28T22:14:13.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.330+0000 7fabf5d8a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x559e21518b70 con 0x559e2160cbb0 2024-03-28T22:14:13.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.330+0000 7fabf5d8a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x559e2156c220 con 0x559e2160c290 2024-03-28T22:14:13.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.330+0000 7fabe9ffb640 1 --2- 172.21.15.16:0/91452433 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fabdc0a1f70 0x7fabdc0a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:13.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.330+0000 7fabebfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e21615790 0x559e21617b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:13.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.330+0000 7fabebfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e21615790 0x559e21617b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46460/0 (socket says 172.21.15.16:46460) 2024-03-28T22:14:13.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.330+0000 7fabebfff640 1 -- 172.21.15.16:0/302817743 learned_addr learned my addr 172.21.15.16:0/302817743 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:13.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.330+0000 7fabf0e72640 1 --2- 172.21.15.16:0/302817743 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e2160cbb0 0x559e21615250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:13.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.330+0000 7fabebfff640 1 --2- 172.21.15.16:0/302817743 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e2160c290 0x559e2160c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:13.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.330+0000 7fabf1673640 1 --2- 172.21.15.16:0/91452433 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fabdc0a1f70 0x7fabdc0a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:13.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.330+0000 7fabe9ffb640 1 -- 172.21.15.16:0/91452433 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fabe40ce900 con 0x7fabec106680 2024-03-28T22:14:13.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.331+0000 7fabf0e72640 1 -- 172.21.15.16:0/302817743 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e2160c290 msgr2=0x559e2160c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:13.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.331+0000 7fabf0e72640 1 --2- 172.21.15.16:0/302817743 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e2160c290 0x559e2160c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.331+0000 7fabf0e72640 1 -- 172.21.15.16:0/302817743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e21615790 msgr2=0x559e21617b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:13.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.331+0000 7fabf0e72640 1 --2- 172.21.15.16:0/302817743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e21615790 0x559e21617b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.331+0000 7fabf0e72640 1 -- 172.21.15.16:0/302817743 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e2152c7c0 con 0x559e2160cbb0 2024-03-28T22:14:13.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.331+0000 7fabebfff640 1 --2- 172.21.15.16:0/302817743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e21615790 0x559e21617b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:14:13.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.331+0000 7fabf1673640 1 --2- 172.21.15.16:0/91452433 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fabdc0a1f70 0x7fabdc0a4430 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fabe4097710 tx=0x7fabe4005b40 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:13.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.331+0000 7fabf0e72640 1 --2- 172.21.15.16:0/302817743 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e2160cbb0 0x559e21615250 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fabe0009bf0 tx=0x7fabe0011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f6de4af11caad98 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:13.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.331+0000 7fabd6ffd640 1 -- 172.21.15.16:0/302817743 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe0009da0 con 0x559e2160cbb0 2024-03-28T22:14:13.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.331+0000 7fabd6ffd640 1 -- 172.21.15.16:0/302817743 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabe0005020 con 0x559e2160cbb0 2024-03-28T22:14:13.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.331+0000 7fabd6ffd640 1 -- 172.21.15.16:0/302817743 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe0004a00 con 0x559e2160cbb0 2024-03-28T22:14:13.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.332+0000 7fabf5d8a740 1 -- 172.21.15.16:0/302817743 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e2160cbb0 msgr2=0x559e21615250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:13.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.332+0000 7fabf5d8a740 1 --2- 172.21.15.16:0/302817743 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e2160cbb0 0x559e21615250 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fabe0009bf0 tx=0x7fabe0011810 comp rx=0 tx=0).stop 2024-03-28T22:14:13.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.332+0000 7fabf5d8a740 1 -- 172.21.15.16:0/302817743 shutdown_connections 2024-03-28T22:14:13.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.332+0000 7fabf5d8a740 1 --2- 172.21.15.16:0/302817743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e21615790 0x559e21617b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.332+0000 7fabf5d8a740 1 --2- 172.21.15.16:0/302817743 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e2160cbb0 0x559e21615250 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.332+0000 7fabf5d8a740 1 --2- 172.21.15.16:0/302817743 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e2160c290 0x559e2160c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.332+0000 7fabf5d8a740 1 -- 172.21.15.16:0/302817743 >> 172.21.15.16:0/302817743 conn(0x559e216069c0 msgr2=0x559e21606da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:13.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.332+0000 7fabf5d8a740 1 -- 172.21.15.16:0/302817743 shutdown_connections 2024-03-28T22:14:13.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.332+0000 7fabf5d8a740 1 -- 172.21.15.16:0/302817743 wait complete. 2024-03-28T22:14:13.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.332+0000 7fabf5d8a740 1 -- start start 2024-03-28T22:14:13.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.332+0000 7fabf5d8a740 1 Processor -- start 2024-03-28T22:14:13.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.333+0000 7fabf5d8a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e2160c290 0x559e216127b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:13.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.333+0000 7fabf5d8a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e2160cbb0 0x559e21612cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:13.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.333+0000 7fabf5d8a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e21615790 0x559e21613230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:13.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.333+0000 7fabf5d8a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559e21618580 con 0x559e21615790 2024-03-28T22:14:13.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.333+0000 7fabf5d8a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x559e2150e4a0 con 0x559e2160c290 2024-03-28T22:14:13.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.333+0000 7fabf5d8a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x559e21618700 con 0x559e2160cbb0 2024-03-28T22:14:13.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.333+0000 7fabf5d8a740 10 client.0 _ll_register_callbacks cb 0x559e21511d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:14:13.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.333+0000 7fabebfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e21615790 0x559e21613230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:13.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.333+0000 7fabebfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e21615790 0x559e21613230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46462/0 (socket says 172.21.15.16:46462) 2024-03-28T22:14:13.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.333+0000 7fabebfff640 1 -- 172.21.15.16:0/3195157506 learned_addr learned my addr 172.21.15.16:0/3195157506 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:13.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.333+0000 7fabebfff640 1 --2- 172.21.15.16:0/3195157506 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e2160c290 0x559e216127b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:13.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.333+0000 7fabf0e72640 1 --2- 172.21.15.16:0/3195157506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e2160cbb0 0x559e21612cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:13.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.333+0000 7fabebfff640 1 -- 172.21.15.16:0/3195157506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e2160cbb0 msgr2=0x559e21612cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.333+0000 7fabebfff640 1 --2- 172.21.15.16:0/3195157506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e2160cbb0 0x559e21612cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.333+0000 7fabebfff640 1 -- 172.21.15.16:0/3195157506 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e2160c290 msgr2=0x559e216127b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.333+0000 7fabebfff640 1 --2- 172.21.15.16:0/3195157506 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e2160c290 0x559e216127b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.333+0000 7fabebfff640 1 -- 172.21.15.16:0/3195157506 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e2134e7d0 con 0x559e21615790 2024-03-28T22:14:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.333+0000 7fabebfff640 1 --2- 172.21.15.16:0/3195157506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e21615790 0x559e21613230 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fabd8009900 tx=0x7fabd802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7018a5df0402cab0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.333+0000 7fabcbfff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:14:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.333+0000 7fabcbfff640 1 -- 172.21.15.16:0/3195157506 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd8049070 con 0x559e21615790 2024-03-28T22:14:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.333+0000 7fabf5d8a740 1 -- 172.21.15.16:0/3195157506 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fabe0005020 con 0x559e21615790 2024-03-28T22:14:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.333+0000 7fabcbfff640 1 -- 172.21.15.16:0/3195157506 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fabd8004480 con 0x559e21615790 2024-03-28T22:14:13.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.333+0000 7fabcbfff640 1 -- 172.21.15.16:0/3195157506 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fabd80035f0 con 0x559e21615790 2024-03-28T22:14:13.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.334+0000 7fabf5d8a740 10 client.16314 fetch_fsmap learned FSMap version 25 2024-03-28T22:14:13.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.334+0000 7fabf5d8a740 1 -- 172.21.15.16:0/3195157506 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x559e212843f0 con 0x559e21615790 2024-03-28T22:14:13.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.334+0000 7fabcbfff640 1 -- 172.21.15.16:0/3195157506 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabe0005020 con 0x559e21615790 2024-03-28T22:14:13.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.334+0000 7fabcbfff640 1 -- 172.21.15.16:0/3195157506 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fabd804c020 con 0x559e21615790 2024-03-28T22:14:13.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.334+0000 7fabf5d8a740 10 client.16314 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T22:14:13.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.334+0000 7fabf5d8a740 10 client.16314 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:14:13.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.334+0000 7fabf5d8a740 20 client.16314 populate_metadata read hostname 'smithi016' 2024-03-28T22:14:13.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.334+0000 7fabf5d8a740 1 --2- 172.21.15.16:0/3195157506 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x559e21625ac0 0x559e21627ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:13.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.334+0000 7fabf5d8a740 4 client.16314 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:14:13.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.334+0000 7fabf5d8a740 1 -- 172.21.15.16:0/3195157506 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559e2157d6c0 con 0x559e21625ac0 2024-03-28T22:14:13.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.334+0000 7fabf0e72640 1 --2- 172.21.15.16:0/3195157506 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x559e21625ac0 0x559e21627ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:13.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.335+0000 7fabcbfff640 10 client.16314 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:14:13.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.335+0000 7fabf0e72640 1 --2- 172.21.15.16:0/3195157506 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x559e21625ac0 0x559e21627ef0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a9f6678c384ed88 server_cookie=f9eb24ee11e24949 in_seq=0 out_seq=0 2024-03-28T22:14:13.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.336+0000 7fabcbfff640 1 -- 172.21.15.16:0/3195157506 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559e2157d6c0 con 0x559e21625ac0 2024-03-28T22:14:13.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.336+0000 7fabcbfff640 10 client.16314 handle_command_reply: tid=0 2024-03-28T22:14:13.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.336+0000 7fabf5d8a740 1 client.16314 shutdown 2024-03-28T22:14:13.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.336+0000 7fabf5d8a740 1 -- 172.21.15.16:0/3195157506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e21615790 msgr2=0x559e21613230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:13.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.336+0000 7fabf5d8a740 1 --2- 172.21.15.16:0/3195157506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e21615790 0x559e21613230 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fabd8009900 tx=0x7fabd802bc30 comp rx=0 tx=0).stop 2024-03-28T22:14:13.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.336+0000 7fabf5d8a740 1 -- 172.21.15.16:0/3195157506 shutdown_connections 2024-03-28T22:14:13.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.336+0000 7fabf5d8a740 1 --2- 172.21.15.16:0/3195157506 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x559e21625ac0 0x559e21627ef0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.336+0000 7fabf5d8a740 1 --2- 172.21.15.16:0/3195157506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e21615790 0x559e21613230 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.336+0000 7fabf5d8a740 1 --2- 172.21.15.16:0/3195157506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e2160cbb0 0x559e21612cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.336+0000 7fabcbfff640 0 client.16314 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:14:13.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.336+0000 7fabf5d8a740 1 --2- 172.21.15.16:0/3195157506 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e2160c290 0x559e216127b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.336+0000 7fabf5d8a740 1 -- 172.21.15.16:0/3195157506 >> 172.21.15.16:0/3195157506 conn(0x559e216069c0 msgr2=0x559e21605c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:13.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.337+0000 7fabf5d8a740 1 -- 172.21.15.16:0/3195157506 shutdown_connections 2024-03-28T22:14:13.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.337+0000 7fabf5d8a740 1 -- 172.21.15.16:0/3195157506 wait complete. 2024-03-28T22:14:13.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.337+0000 7fabf5d8a740 20 client.16314 trim_cache size 0 max 16384 2024-03-28T22:14:13.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.355+0000 7fabf5d8a740 1 Processor -- start 2024-03-28T22:14:13.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.355+0000 7fabf5d8a740 1 -- start start 2024-03-28T22:14:13.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.355+0000 7fabf5d8a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e2160c290 0x559e21614ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:13.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.355+0000 7fabf5d8a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e2160cbb0 0x559e21621350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:13.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.355+0000 7fabf5d8a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e21615790 0x559e21605aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:13.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.355+0000 7fabf5d8a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fabec1afdb0 con 0x559e2160cbb0 2024-03-28T22:14:13.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.355+0000 7fabf5d8a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fabec1ad650 con 0x559e21615790 2024-03-28T22:14:13.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.355+0000 7fabf5d8a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fabec1ab4a0 con 0x559e2160c290 2024-03-28T22:14:13.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.356+0000 7fabf0e72640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e2160cbb0 0x559e21621350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:13.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.356+0000 7fabf0e72640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e2160cbb0 0x559e21621350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46466/0 (socket says 172.21.15.16:46466) 2024-03-28T22:14:13.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.356+0000 7fabf0e72640 1 -- 172.21.15.16:0/1112810667 learned_addr learned my addr 172.21.15.16:0/1112810667 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:13.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.356+0000 7fabebfff640 1 --2- 172.21.15.16:0/1112810667 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e21615790 0x559e21605aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:13.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.356+0000 7fabf0e72640 1 -- 172.21.15.16:0/1112810667 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e2160c290 msgr2=0x559e21614ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:13.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.356+0000 7fabebfff640 1 --2- 172.21.15.16:0/1112810667 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e2160c290 0x559e21614ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:13.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.356+0000 7fabf0e72640 1 --2- 172.21.15.16:0/1112810667 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e2160c290 0x559e21614ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.356+0000 7fabf0e72640 1 -- 172.21.15.16:0/1112810667 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e21615790 msgr2=0x559e21605aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:13.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.356+0000 7fabf0e72640 1 --2- 172.21.15.16:0/1112810667 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e21615790 0x559e21605aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.356+0000 7fabf0e72640 1 -- 172.21.15.16:0/1112810667 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e212440f0 con 0x559e2160cbb0 2024-03-28T22:14:13.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.356+0000 7fabf0e72640 1 --2- 172.21.15.16:0/1112810667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e2160cbb0 0x559e21621350 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fabe0005270 tx=0x7fabe0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f88686d92de8186f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:13.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.356+0000 7fabd67fc640 1 -- 172.21.15.16:0/1112810667 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe0030070 con 0x559e2160cbb0 2024-03-28T22:14:13.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.356+0000 7fabd67fc640 1 -- 172.21.15.16:0/1112810667 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabe00053a0 con 0x559e2160cbb0 2024-03-28T22:14:13.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.356+0000 7fabd67fc640 1 -- 172.21.15.16:0/1112810667 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe002b6e0 con 0x559e2160cbb0 2024-03-28T22:14:13.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.357+0000 7fabf5d8a740 1 -- 172.21.15.16:0/1112810667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e2160cbb0 msgr2=0x559e21621350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:13.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.357+0000 7fabf5d8a740 1 --2- 172.21.15.16:0/1112810667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e2160cbb0 0x559e21621350 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fabe0005270 tx=0x7fabe0011810 comp rx=0 tx=0).stop 2024-03-28T22:14:13.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.357+0000 7fabf5d8a740 1 -- 172.21.15.16:0/1112810667 shutdown_connections 2024-03-28T22:14:13.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.357+0000 7fabf5d8a740 1 --2- 172.21.15.16:0/1112810667 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e21615790 0x559e21605aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.357+0000 7fabf5d8a740 1 --2- 172.21.15.16:0/1112810667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e2160cbb0 0x559e21621350 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.357+0000 7fabf5d8a740 1 --2- 172.21.15.16:0/1112810667 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e2160c290 0x559e21614ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.357+0000 7fabf5d8a740 1 -- 172.21.15.16:0/1112810667 >> 172.21.15.16:0/1112810667 conn(0x559e216069c0 msgr2=0x559e21605390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:13.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.357+0000 7fabf5d8a740 1 -- 172.21.15.16:0/1112810667 shutdown_connections 2024-03-28T22:14:13.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.357+0000 7fabf5d8a740 1 -- 172.21.15.16:0/1112810667 wait complete. 2024-03-28T22:14:13.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.357+0000 7fabf5d8a740 1 -- start start 2024-03-28T22:14:13.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.357+0000 7fabf5d8a740 1 Processor -- start 2024-03-28T22:14:13.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.358+0000 7fabf5d8a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e2160c290 0x559e2161da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:13.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.358+0000 7fabf5d8a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e2160cbb0 0x559e21618f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:13.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.358+0000 7fabf5d8a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e21615790 0x559e21619480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:13.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.358+0000 7fabf5d8a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fabec1abb10 con 0x559e21615790 2024-03-28T22:14:13.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.358+0000 7fabf5d8a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fabec1acf80 con 0x559e2160c290 2024-03-28T22:14:13.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.358+0000 7fabf5d8a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fabec1ac1e0 con 0x559e2160cbb0 2024-03-28T22:14:13.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.358+0000 7fabf5d8a740 10 client.0 _ll_register_callbacks cb 0x7fabd8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:14:13.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.358+0000 7fabebfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e21615790 0x559e21619480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:13.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.358+0000 7fabf0e72640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e2160cbb0 0x559e21618f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:13.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.358+0000 7fabebfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e21615790 0x559e21619480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46468/0 (socket says 172.21.15.16:46468) 2024-03-28T22:14:13.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.358+0000 7fabebfff640 1 -- 172.21.15.16:0/100873536 learned_addr learned my addr 172.21.15.16:0/100873536 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:13.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.358+0000 7fabebfff640 1 --2- 172.21.15.16:0/100873536 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e2160c290 0x559e2161da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:13.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.358+0000 7fabebfff640 1 -- 172.21.15.16:0/100873536 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e2160cbb0 msgr2=0x559e21618f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:13.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.358+0000 7fabebfff640 1 --2- 172.21.15.16:0/100873536 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e2160cbb0 0x559e21618f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.358+0000 7fabebfff640 1 -- 172.21.15.16:0/100873536 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e2160c290 msgr2=0x559e2161da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:13.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.358+0000 7fabebfff640 1 --2- 172.21.15.16:0/100873536 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e2160c290 0x559e2161da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.358+0000 7fabebfff640 1 -- 172.21.15.16:0/100873536 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e21619ac0 con 0x559e21615790 2024-03-28T22:14:13.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.358+0000 7fabf0e72640 1 --2- 172.21.15.16:0/100873536 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e2160cbb0 0x559e21618f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:14:13.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.358+0000 7fabebfff640 1 --2- 172.21.15.16:0/100873536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e21615790 0x559e21619480 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fabd802c820 tx=0x7fabd8009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f19d8263714902f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:13.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.358+0000 7fabcbfff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:14:13.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.359+0000 7fabcbfff640 1 -- 172.21.15.16:0/100873536 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd8049040 con 0x559e21615790 2024-03-28T22:14:13.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.359+0000 7fabcbfff640 1 -- 172.21.15.16:0/100873536 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fabd8003790 con 0x559e21615790 2024-03-28T22:14:13.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.359+0000 7fabf5d8a740 1 -- 172.21.15.16:0/100873536 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fabe00053a0 con 0x559e21615790 2024-03-28T22:14:13.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.359+0000 7fabcbfff640 1 -- 172.21.15.16:0/100873536 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fabd8003990 con 0x559e21615790 2024-03-28T22:14:13.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.359+0000 7fabf5d8a740 10 client.16320 fetch_fsmap learned FSMap version 25 2024-03-28T22:14:13.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.359+0000 7fabf5d8a740 1 -- 172.21.15.16:0/100873536 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x559e21690080 con 0x559e21615790 2024-03-28T22:14:13.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.359+0000 7fabcbfff640 1 -- 172.21.15.16:0/100873536 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabe00053a0 con 0x559e21615790 2024-03-28T22:14:13.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.359+0000 7fabcbfff640 1 -- 172.21.15.16:0/100873536 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fabd8061070 con 0x559e21615790 2024-03-28T22:14:13.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.359+0000 7fabf5d8a740 10 client.16320 fetch_fsmap finished waiting for FSMap version 25 2024-03-28T22:14:13.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.359+0000 7fabf5d8a740 10 client.16320 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:14:13.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.359+0000 7fabf5d8a740 20 client.16320 populate_metadata read hostname 'smithi016' 2024-03-28T22:14:13.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.359+0000 7fabf5d8a740 1 --2- 172.21.15.16:0/100873536 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x559e216907a0 0x559e21692bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:13.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.359+0000 7fabf5d8a740 4 client.16320 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:14:13.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.359+0000 7fabf5d8a740 1 -- 172.21.15.16:0/100873536 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559e21577a10 con 0x559e216907a0 2024-03-28T22:14:13.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.360+0000 7fabf0e72640 1 --2- 172.21.15.16:0/100873536 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x559e216907a0 0x559e21692bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:13.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.360+0000 7fabcbfff640 10 client.16320 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:14:13.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.360+0000 7fabf0e72640 1 --2- 172.21.15.16:0/100873536 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x559e216907a0 0x559e21692bc0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb9800c9825b2af4 server_cookie=c0d67c957fc6ac3 in_seq=0 out_seq=0 2024-03-28T22:14:13.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.360+0000 7fabcbfff640 1 -- 172.21.15.16:0/100873536 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x559e21577a10 con 0x559e216907a0 2024-03-28T22:14:13.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.360+0000 7fabcbfff640 10 client.16320 handle_command_reply: tid=0 2024-03-28T22:14:13.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.360+0000 7fabf5d8a740 1 client.16320 shutdown 2024-03-28T22:14:13.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.360+0000 7fabf5d8a740 1 -- 172.21.15.16:0/100873536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e21615790 msgr2=0x559e21619480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:13.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.360+0000 7fabf5d8a740 1 --2- 172.21.15.16:0/100873536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e21615790 0x559e21619480 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fabd802c820 tx=0x7fabd8009680 comp rx=0 tx=0).stop 2024-03-28T22:14:13.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.360+0000 7fabf5d8a740 1 -- 172.21.15.16:0/100873536 shutdown_connections 2024-03-28T22:14:13.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.360+0000 7fabf5d8a740 1 --2- 172.21.15.16:0/100873536 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x559e216907a0 0x559e21692bc0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.360+0000 7fabf5d8a740 1 --2- 172.21.15.16:0/100873536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e21615790 0x559e21619480 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.360+0000 7fabf5d8a740 1 --2- 172.21.15.16:0/100873536 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e2160cbb0 0x559e21618f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.360+0000 7fabcbfff640 0 client.16320 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:14:13.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.360+0000 7fabf5d8a740 1 --2- 172.21.15.16:0/100873536 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e2160c290 0x559e2161da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.360+0000 7fabf5d8a740 1 -- 172.21.15.16:0/100873536 >> 172.21.15.16:0/100873536 conn(0x559e216069c0 msgr2=0x559e21605c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:13.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.361+0000 7fabf5d8a740 1 -- 172.21.15.16:0/100873536 shutdown_connections 2024-03-28T22:14:13.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.361+0000 7fabf5d8a740 1 -- 172.21.15.16:0/100873536 wait complete. 2024-03-28T22:14:13.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.361+0000 7fabf5d8a740 20 client.16320 trim_cache size 0 max 16384 2024-03-28T22:14:13.374 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:14:13.375 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T22:14:13.375 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-03-28T22:14:13.375 INFO:teuthology.orchestra.run.smithi016.stdout: "1590a3c8-388b-4c8e-91dd-31dd85ab4312": { 2024-03-28T22:14:13.375 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-03-28T22:14:13.375 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "1590a3c8-388b-4c8e-91dd-31dd85ab4312", 2024-03-28T22:14:13.375 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-03-28T22:14:13.375 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:14:13.375 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:14:13.376 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:14:13.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.361+0000 7fabf30d6640 1 -- 172.21.15.16:0/91452433 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fabdc0a1f70 msgr2=0x7fabdc0a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:13.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.361+0000 7fabf30d6640 1 --2- 172.21.15.16:0/91452433 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fabdc0a1f70 0x7fabdc0a4430 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fabe4097710 tx=0x7fabe4005b40 comp rx=0 tx=0).stop 2024-03-28T22:14:13.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.361+0000 7fabf30d6640 1 -- 172.21.15.16:0/91452433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabec106680 msgr2=0x7fabec19c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:13.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.361+0000 7fabf30d6640 1 --2- 172.21.15.16:0/91452433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabec106680 0x7fabec19c8d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fabe400eb10 tx=0x7fabe400c2d0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.361+0000 7fabf30d6640 1 -- 172.21.15.16:0/91452433 shutdown_connections 2024-03-28T22:14:13.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.361+0000 7fabf30d6640 1 --2- 172.21.15.16:0/91452433 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fabdc0a1f70 0x7fabdc0a4430 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.361+0000 7fabf30d6640 1 --2- 172.21.15.16:0/91452433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabec106680 0x7fabec19c8d0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.361+0000 7fabf30d6640 1 --2- 172.21.15.16:0/91452433 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fabec100fa0 0x7fabec198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.361+0000 7fabf30d6640 1 --2- 172.21.15.16:0/91452433 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabec1005e0 0x7fabec198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:13.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.361+0000 7fabf30d6640 1 -- 172.21.15.16:0/91452433 >> 172.21.15.16:0/91452433 conn(0x7fabec0fc0b0 msgr2=0x7fabec0fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:13.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.361+0000 7fabf30d6640 1 -- 172.21.15.16:0/91452433 shutdown_connections 2024-03-28T22:14:13.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:13.361+0000 7fabf30d6640 1 -- 172.21.15.16:0/91452433 wait complete. 2024-03-28T22:14:13.379 INFO:tasks.cephfs.filesystem:scrub status for tag:1590a3c8-388b-4c8e-91dd-31dd85ab4312 - {'path': '/', 'tag': '1590a3c8-388b-4c8e-91dd-31dd85ab4312', 'options': 'recursive,force'} 2024-03-28T22:14:14.348 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:14 smithi053 ceph-mon[35979]: pgmap v112: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.5 KiB/s wr, 5 op/s 2024-03-28T22:14:14.349 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:14 smithi053 ceph-mon[35979]: scrub queued for path: / 2024-03-28T22:14:14.349 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:14 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:14:14.349 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:14 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:14:14.349 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:14 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:14:14.349 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:14 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:14:14.349 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:14 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:14:14.349 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:14 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:14:14.349 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:14 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:14:14.350 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:14 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:14:14.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:14 smithi063 ceph-mon[36249]: pgmap v112: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.5 KiB/s wr, 5 op/s 2024-03-28T22:14:14.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:14 smithi063 ceph-mon[36249]: scrub queued for path: / 2024-03-28T22:14:14.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:14 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:14:14.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:14 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:14:14.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:14 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:14:14.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:14 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:14:14.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:14 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:14:14.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:14 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:14:14.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:14 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:14:14.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:14 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:14:14.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:14 smithi016 ceph-mon[32374]: pgmap v112: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.5 KiB/s wr, 5 op/s 2024-03-28T22:14:14.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:14 smithi016 ceph-mon[32374]: scrub queued for path: / 2024-03-28T22:14:14.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:14 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:14:14.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:14 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:14:14.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:14 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:14:14.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:14 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:14:14.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:14 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:14:14.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:14 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:14:14.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:14 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:14:14.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:14 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:14:15.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:15 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:14:15.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:15 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:14:15.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:15 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:14:16.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:16 smithi063 ceph-mon[36249]: pgmap v113: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.5 KiB/s wr, 5 op/s 2024-03-28T22:14:16.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:16 smithi016 ceph-mon[32374]: pgmap v113: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.5 KiB/s wr, 5 op/s 2024-03-28T22:14:16.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:16 smithi053 ceph-mon[35979]: pgmap v113: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.5 KiB/s wr, 5 op/s 2024-03-28T22:14:18.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:18 smithi063 ceph-mon[36249]: pgmap v114: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 5.7 KiB/s wr, 9 op/s 2024-03-28T22:14:18.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:18 smithi016 ceph-mon[32374]: pgmap v114: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 5.7 KiB/s wr, 9 op/s 2024-03-28T22:14:18.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:18 smithi053 ceph-mon[35979]: pgmap v114: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 5.7 KiB/s wr, 9 op/s 2024-03-28T22:14:20.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:20 smithi063 ceph-mon[36249]: pgmap v115: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.4 KiB/s wr, 5 op/s 2024-03-28T22:14:20.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:20 smithi016 ceph-mon[32374]: pgmap v115: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.4 KiB/s wr, 5 op/s 2024-03-28T22:14:20.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:20 smithi053 ceph-mon[35979]: pgmap v115: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.4 KiB/s wr, 5 op/s 2024-03-28T22:14:22.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:22 smithi063 ceph-mon[36249]: pgmap v116: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.5 KiB/s wr, 8 op/s 2024-03-28T22:14:22.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:22 smithi016 ceph-mon[32374]: pgmap v116: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.5 KiB/s wr, 8 op/s 2024-03-28T22:14:22.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:22 smithi053 ceph-mon[35979]: pgmap v116: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.5 KiB/s wr, 8 op/s 2024-03-28T22:14:24.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:24 smithi063 ceph-mon[36249]: pgmap v117: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-03-28T22:14:24.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:24 smithi016 ceph-mon[32374]: pgmap v117: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-03-28T22:14:24.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:24 smithi053 ceph-mon[35979]: pgmap v117: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-03-28T22:14:25.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:25 smithi063 ceph-mon[36249]: scrub summary: idle 2024-03-28T22:14:25.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:25 smithi016 ceph-mon[32374]: scrub summary: idle 2024-03-28T22:14:25.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:25 smithi053 ceph-mon[35979]: scrub summary: idle 2024-03-28T22:14:26.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:26 smithi063 ceph-mon[36249]: pgmap v118: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-03-28T22:14:26.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:26 smithi063 ceph-mon[36249]: scrub summary: idle 2024-03-28T22:14:26.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:26 smithi016 ceph-mon[32374]: pgmap v118: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-03-28T22:14:26.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:26 smithi016 ceph-mon[32374]: scrub summary: idle 2024-03-28T22:14:26.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:26 smithi053 ceph-mon[35979]: pgmap v118: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-03-28T22:14:26.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:26 smithi053 ceph-mon[35979]: scrub summary: idle 2024-03-28T22:14:27.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:27 smithi063 ceph-mon[36249]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T22:14:27.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:27 smithi016 ceph-mon[32374]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T22:14:27.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:27 smithi053 ceph-mon[35979]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T22:14:28.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:28 smithi063 ceph-mon[36249]: pgmap v119: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-03-28T22:14:28.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:28 smithi016 ceph-mon[32374]: pgmap v119: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-03-28T22:14:28.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:28 smithi053 ceph-mon[35979]: pgmap v119: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-03-28T22:14:30.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:30 smithi063 ceph-mon[36249]: pgmap v120: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-28T22:14:30.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:30 smithi016 ceph-mon[32374]: pgmap v120: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-28T22:14:30.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:30 smithi053 ceph-mon[35979]: pgmap v120: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-28T22:14:32.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:32 smithi063 ceph-mon[36249]: pgmap v121: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-28T22:14:32.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:32 smithi016 ceph-mon[32374]: pgmap v121: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-28T22:14:32.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:32 smithi053 ceph-mon[35979]: pgmap v121: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-28T22:14:34.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:34 smithi063 ceph-mon[36249]: pgmap v122: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:14:34.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:34 smithi016 ceph-mon[32374]: pgmap v122: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:14:34.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:34 smithi053 ceph-mon[35979]: pgmap v122: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:14:36.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:36 smithi063 ceph-mon[36249]: pgmap v123: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:14:36.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:36 smithi016 ceph-mon[32374]: pgmap v123: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:14:36.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:36 smithi053 ceph-mon[35979]: pgmap v123: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:14:38.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:38 smithi063 ceph-mon[36249]: pgmap v124: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:14:38.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:38 smithi016 ceph-mon[32374]: pgmap v124: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:14:38.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:38 smithi053 ceph-mon[35979]: pgmap v124: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:14:40.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:40 smithi063 ceph-mon[36249]: pgmap v125: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:14:40.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:40 smithi016 ceph-mon[32374]: pgmap v125: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:14:40.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:40 smithi053 ceph-mon[35979]: pgmap v125: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:14:42.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:42 smithi063 ceph-mon[36249]: pgmap v126: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:14:42.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:42 smithi016 ceph-mon[32374]: pgmap v126: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:14:42.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:42 smithi053 ceph-mon[35979]: pgmap v126: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:14:43.379 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:14:43.380 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:14:43.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.495+0000 7f9ed1581640 1 -- 172.21.15.16:0/3017780419 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ecc100fb0 msgr2=0x7f9ecc10d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:43.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.495+0000 7f9ed1581640 1 --2- 172.21.15.16:0/3017780419 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ecc100fb0 0x7f9ecc10d4c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4009900 tx=0x7f9ec402bc30 comp rx=0 tx=0).stop 2024-03-28T22:14:43.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.495+0000 7f9ed1581640 1 -- 172.21.15.16:0/3017780419 shutdown_connections 2024-03-28T22:14:43.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.495+0000 7f9ed1581640 1 --2- 172.21.15.16:0/3017780419 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ecc100fb0 0x7f9ecc10d4c0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.495+0000 7f9ed1581640 1 --2- 172.21.15.16:0/3017780419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ecc1005f0 0x7f9ecc100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.495+0000 7f9ed1581640 1 --2- 172.21.15.16:0/3017780419 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9ecc106690 0x7f9ecc106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.495+0000 7f9ed1581640 1 -- 172.21.15.16:0/3017780419 >> 172.21.15.16:0/3017780419 conn(0x7f9ecc077800 msgr2=0x7f9ecc077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:43.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.495+0000 7f9ed1581640 1 -- 172.21.15.16:0/3017780419 shutdown_connections 2024-03-28T22:14:43.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.495+0000 7f9ed1581640 1 -- 172.21.15.16:0/3017780419 wait complete. 2024-03-28T22:14:43.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.495+0000 7f9ed1581640 1 Processor -- start 2024-03-28T22:14:43.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.496+0000 7f9ed1581640 1 -- start start 2024-03-28T22:14:43.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.496+0000 7f9ed1581640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ecc1005f0 0x7f9ecc102290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:43.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.496+0000 7f9ed1581640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9ecc100fb0 0x7f9ecc1027d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:43.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.496+0000 7f9ed1581640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ecc106690 0x7f9ecc102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:43.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.496+0000 7f9ed1581640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9ecc1109d0 con 0x7f9ecc106690 2024-03-28T22:14:43.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.496+0000 7f9ed1581640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9ecc110850 con 0x7f9ecc1005f0 2024-03-28T22:14:43.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.496+0000 7f9ed1581640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9ecc110b50 con 0x7f9ecc100fb0 2024-03-28T22:14:43.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.496+0000 7f9ecb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ecc106690 0x7f9ecc102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:43.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.496+0000 7f9ecaffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ecc1005f0 0x7f9ecc102290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:43.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.496+0000 7f9ecb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ecc106690 0x7f9ecc102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47462/0 (socket says 172.21.15.16:47462) 2024-03-28T22:14:43.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.496+0000 7f9ecb7fe640 1 -- 172.21.15.16:0/414284651 learned_addr learned my addr 172.21.15.16:0/414284651 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:43.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.496+0000 7f9eca7fc640 1 --2- 172.21.15.16:0/414284651 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9ecc100fb0 0x7f9ecc1027d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:43.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.496+0000 7f9ecb7fe640 1 -- 172.21.15.16:0/414284651 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9ecc100fb0 msgr2=0x7f9ecc1027d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:43.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.496+0000 7f9ecb7fe640 1 --2- 172.21.15.16:0/414284651 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9ecc100fb0 0x7f9ecc1027d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.496+0000 7f9ecb7fe640 1 -- 172.21.15.16:0/414284651 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ecc1005f0 msgr2=0x7f9ecc102290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:43.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.496+0000 7f9ecb7fe640 1 --2- 172.21.15.16:0/414284651 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ecc1005f0 0x7f9ecc102290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.496+0000 7f9ecb7fe640 1 -- 172.21.15.16:0/414284651 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ecc1a6420 con 0x7f9ecc106690 2024-03-28T22:14:43.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.496+0000 7f9ecaffd640 1 --2- 172.21.15.16:0/414284651 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ecc1005f0 0x7f9ecc102290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:14:43.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.496+0000 7f9ecb7fe640 1 --2- 172.21.15.16:0/414284651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ecc106690 0x7f9ecc102d10 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f9ec402efd0 tx=0x7f9ec4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:43.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.497+0000 7f9eb3fff640 1 -- 172.21.15.16:0/414284651 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec4041070 con 0x7f9ecc106690 2024-03-28T22:14:43.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.497+0000 7f9ed1581640 1 -- 172.21.15.16:0/414284651 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ecc1a66b0 con 0x7f9ecc106690 2024-03-28T22:14:43.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.497+0000 7f9eb3fff640 1 -- 172.21.15.16:0/414284651 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ec40054e0 con 0x7f9ecc106690 2024-03-28T22:14:43.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.497+0000 7f9eb3fff640 1 -- 172.21.15.16:0/414284651 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec4003bf0 con 0x7f9ecc106690 2024-03-28T22:14:43.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.497+0000 7f9ed1581640 1 -- 172.21.15.16:0/414284651 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ecc1a6bf0 con 0x7f9ecc106690 2024-03-28T22:14:43.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.499+0000 7f9eb3fff640 1 -- 172.21.15.16:0/414284651 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f9ec4044070 con 0x7f9ecc106690 2024-03-28T22:14:43.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.499+0000 7f9ed4165740 1 Processor -- start 2024-03-28T22:14:43.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.499+0000 7f9eb3fff640 1 --2- 172.21.15.16:0/414284651 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f9eb40a1f20 0x7f9eb40a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:43.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.499+0000 7f9ed4165740 1 -- start start 2024-03-28T22:14:43.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.500+0000 7f9eca7fc640 1 --2- 172.21.15.16:0/414284651 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f9eb40a1f20 0x7f9eb40a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:43.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.500+0000 7f9ed4165740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556cd8ce0290 0x556cd8ce0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:43.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.500+0000 7f9ed4165740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556cd8ce0bb0 0x556cd8ce9250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:43.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.500+0000 7f9eb3fff640 1 -- 172.21.15.16:0/414284651 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f9ec40efae0 con 0x7f9ecc106690 2024-03-28T22:14:43.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.500+0000 7f9ed4165740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556cd8ce9790 0x556cd8cebb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:43.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.500+0000 7f9ed4165740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556cd8c44180 con 0x556cd8ce9790 2024-03-28T22:14:43.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.500+0000 7f9ed4165740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x556cd8becb70 con 0x556cd8ce0bb0 2024-03-28T22:14:43.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.500+0000 7f9ed4165740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x556cd8c40220 con 0x556cd8ce0290 2024-03-28T22:14:43.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.500+0000 7f9ecb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556cd8ce9790 0x556cd8cebb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:43.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.500+0000 7f9eca7fc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556cd8ce0bb0 0x556cd8ce9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:43.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.500+0000 7f9ecb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556cd8ce9790 0x556cd8cebb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47470/0 (socket says 172.21.15.16:47470) 2024-03-28T22:14:43.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.500+0000 7f9eca7fc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556cd8ce0bb0 0x556cd8ce9250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:50934/0 (socket says 172.21.15.16:50934) 2024-03-28T22:14:43.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.500+0000 7f9ecb7fe640 1 -- 172.21.15.16:0/85030060 learned_addr learned my addr 172.21.15.16:0/85030060 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:43.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.500+0000 7f9ecaffd640 1 --2- 172.21.15.16:0/85030060 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556cd8ce0290 0x556cd8ce0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:43.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.500+0000 7f9eca7fc640 1 --2- 172.21.15.16:0/414284651 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f9eb40a1f20 0x7f9eb40a43e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc103cd0 tx=0x7f9ec00158a0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:43.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.500+0000 7f9ecb7fe640 1 -- 172.21.15.16:0/85030060 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556cd8ce0290 msgr2=0x556cd8ce0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:43.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.500+0000 7f9ecb7fe640 1 --2- 172.21.15.16:0/85030060 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556cd8ce0290 0x556cd8ce0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.500+0000 7f9ecb7fe640 1 -- 172.21.15.16:0/85030060 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556cd8ce0bb0 msgr2=0x556cd8ce9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:43.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.500+0000 7f9ecb7fe640 1 --2- 172.21.15.16:0/85030060 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556cd8ce0bb0 0x556cd8ce9250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.500+0000 7f9ecb7fe640 1 -- 172.21.15.16:0/85030060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556cd8c007c0 con 0x556cd8ce9790 2024-03-28T22:14:43.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.500+0000 7f9eca7fc640 1 --2- 172.21.15.16:0/85030060 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556cd8ce0bb0 0x556cd8ce9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:14:43.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.500+0000 7f9ecb7fe640 1 --2- 172.21.15.16:0/85030060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556cd8ce9790 0x556cd8cebb80 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4006010 tx=0x7f9ec40bc440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29029d4bb13fc7b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:43.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.500+0000 7f9eb17fa640 1 -- 172.21.15.16:0/85030060 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec404b070 con 0x556cd8ce9790 2024-03-28T22:14:43.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.501+0000 7f9eb17fa640 1 -- 172.21.15.16:0/85030060 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ec4005690 con 0x556cd8ce9790 2024-03-28T22:14:43.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.501+0000 7f9eb17fa640 1 -- 172.21.15.16:0/85030060 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec402cc40 con 0x556cd8ce9790 2024-03-28T22:14:43.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.501+0000 7f9ed4165740 1 -- 172.21.15.16:0/85030060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556cd8ce9790 msgr2=0x556cd8cebb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:43.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.501+0000 7f9ed4165740 1 --2- 172.21.15.16:0/85030060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556cd8ce9790 0x556cd8cebb80 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4006010 tx=0x7f9ec40bc440 comp rx=0 tx=0).stop 2024-03-28T22:14:43.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.501+0000 7f9ed4165740 1 -- 172.21.15.16:0/85030060 shutdown_connections 2024-03-28T22:14:43.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.501+0000 7f9ed4165740 1 --2- 172.21.15.16:0/85030060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556cd8ce9790 0x556cd8cebb80 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.501+0000 7f9ed4165740 1 --2- 172.21.15.16:0/85030060 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556cd8ce0bb0 0x556cd8ce9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.501+0000 7f9ed4165740 1 --2- 172.21.15.16:0/85030060 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556cd8ce0290 0x556cd8ce0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.501+0000 7f9ed4165740 1 -- 172.21.15.16:0/85030060 >> 172.21.15.16:0/85030060 conn(0x556cd8cda9c0 msgr2=0x556cd8cdada0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:43.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.501+0000 7f9ed4165740 1 -- 172.21.15.16:0/85030060 shutdown_connections 2024-03-28T22:14:43.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.501+0000 7f9ed4165740 1 -- 172.21.15.16:0/85030060 wait complete. 2024-03-28T22:14:43.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.502+0000 7f9ed4165740 1 -- start start 2024-03-28T22:14:43.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.502+0000 7f9ed4165740 1 Processor -- start 2024-03-28T22:14:43.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.502+0000 7f9ed4165740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556cd8ce0290 0x556cd8ce67c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:43.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.502+0000 7f9ed4165740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556cd8ce0bb0 0x556cd8ce6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:43.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.502+0000 7f9ed4165740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556cd8ce9790 0x556cd8ce7240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:43.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.502+0000 7f9ed4165740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556cd8c44180 con 0x556cd8ce9790 2024-03-28T22:14:43.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.502+0000 7f9ed4165740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x556cd8be24a0 con 0x556cd8ce0290 2024-03-28T22:14:43.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.502+0000 7f9ed4165740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x556cd8cec340 con 0x556cd8ce0bb0 2024-03-28T22:14:43.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.502+0000 7f9ed4165740 10 client.0 _ll_register_callbacks cb 0x556cd8be5d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:14:43.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.502+0000 7f9ecb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556cd8ce9790 0x556cd8ce7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:43.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.502+0000 7f9ecaffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556cd8ce0290 0x556cd8ce67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:43.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.502+0000 7f9ecaffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556cd8ce0290 0x556cd8ce67c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:50936/0 (socket says 172.21.15.16:50936) 2024-03-28T22:14:43.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.502+0000 7f9ecaffd640 1 -- 172.21.15.16:0/2800508722 learned_addr learned my addr 172.21.15.16:0/2800508722 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:43.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.502+0000 7f9eca7fc640 1 --2- 172.21.15.16:0/2800508722 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556cd8ce0bb0 0x556cd8ce6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:43.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.502+0000 7f9ecb7fe640 1 -- 172.21.15.16:0/2800508722 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556cd8ce0bb0 msgr2=0x556cd8ce6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:43.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.502+0000 7f9ecb7fe640 1 --2- 172.21.15.16:0/2800508722 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556cd8ce0bb0 0x556cd8ce6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.502+0000 7f9ecb7fe640 1 -- 172.21.15.16:0/2800508722 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556cd8ce0290 msgr2=0x556cd8ce67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:43.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.502+0000 7f9ecb7fe640 1 --2- 172.21.15.16:0/2800508722 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556cd8ce0290 0x556cd8ce67c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.502+0000 7f9ecb7fe640 1 -- 172.21.15.16:0/2800508722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556cd8a227d0 con 0x556cd8ce9790 2024-03-28T22:14:43.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.503+0000 7f9ecb7fe640 1 --2- 172.21.15.16:0/2800508722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556cd8ce9790 0x556cd8ce7240 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4005940 tx=0x7f9ec40bc610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b157cf7532a2a12 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:43.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.503+0000 7f9eaeff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:14:43.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.503+0000 7f9eaeff5640 1 -- 172.21.15.16:0/2800508722 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec40d4060 con 0x556cd8ce9790 2024-03-28T22:14:43.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.503+0000 7f9eaeff5640 1 -- 172.21.15.16:0/2800508722 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f9ec40b8c10 con 0x556cd8ce9790 2024-03-28T22:14:43.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.503+0000 7f9ed4165740 1 -- 172.21.15.16:0/2800508722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556cd8c516c0 con 0x556cd8ce9790 2024-03-28T22:14:43.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.503+0000 7f9eaeff5640 1 -- 172.21.15.16:0/2800508722 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9ec404b070 con 0x556cd8ce9790 2024-03-28T22:14:43.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.503+0000 7f9ed4165740 10 client.16332 fetch_fsmap learned FSMap version 26 2024-03-28T22:14:43.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.503+0000 7f9ed4165740 1 -- 172.21.15.16:0/2800508722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x556cd89583f0 con 0x556cd8ce9790 2024-03-28T22:14:43.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.503+0000 7f9eaeff5640 1 -- 172.21.15.16:0/2800508722 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x556cd8c516c0 con 0x556cd8ce9790 2024-03-28T22:14:43.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.503+0000 7f9eaeff5640 1 -- 172.21.15.16:0/2800508722 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 26) ==== 8342+0+0 (secure 0 0 0) 0x7f9ec40c48b0 con 0x556cd8ce9790 2024-03-28T22:14:43.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.503+0000 7f9ed4165740 10 client.16332 fetch_fsmap finished waiting for FSMap version 26 2024-03-28T22:14:43.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.503+0000 7f9ed4165740 10 client.16332 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:14:43.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.503+0000 7f9ed4165740 20 client.16332 populate_metadata read hostname 'smithi016' 2024-03-28T22:14:43.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.503+0000 7f9ed4165740 1 --2- 172.21.15.16:0/2800508722 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x556cd8cf95e0 0x556cd8cfb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:43.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.503+0000 7f9ed4165740 4 client.16332 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:14:43.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.503+0000 7f9ed4165740 1 -- 172.21.15.16:0/2800508722 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556cd8c4ba10 con 0x556cd8cf95e0 2024-03-28T22:14:43.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.503+0000 7f9ecaffd640 1 --2- 172.21.15.16:0/2800508722 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x556cd8cf95e0 0x556cd8cfb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:43.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.504+0000 7f9ecaffd640 1 --2- 172.21.15.16:0/2800508722 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x556cd8cf95e0 0x556cd8cfb9d0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62bb8191fd82d7b9 server_cookie=339a0b80be6b0e85 in_seq=0 out_seq=0 2024-03-28T22:14:43.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.504+0000 7f9eaeff5640 10 client.16332 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:14:43.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.505+0000 7f9eaeff5640 1 -- 172.21.15.16:0/2800508722 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556cd8c4ba10 con 0x556cd8cf95e0 2024-03-28T22:14:43.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.505+0000 7f9eaeff5640 10 client.16332 handle_command_reply: tid=0 2024-03-28T22:14:43.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.505+0000 7f9ed4165740 1 client.16332 shutdown 2024-03-28T22:14:43.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.505+0000 7f9ed4165740 1 -- 172.21.15.16:0/2800508722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556cd8ce9790 msgr2=0x556cd8ce7240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:43.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.505+0000 7f9ed4165740 1 --2- 172.21.15.16:0/2800508722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556cd8ce9790 0x556cd8ce7240 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4005940 tx=0x7f9ec40bc610 comp rx=0 tx=0).stop 2024-03-28T22:14:43.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.505+0000 7f9ed4165740 1 -- 172.21.15.16:0/2800508722 shutdown_connections 2024-03-28T22:14:43.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.505+0000 7f9ed4165740 1 --2- 172.21.15.16:0/2800508722 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x556cd8cf95e0 0x556cd8cfb9d0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.505+0000 7f9ed4165740 1 --2- 172.21.15.16:0/2800508722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556cd8ce9790 0x556cd8ce7240 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.505+0000 7f9ed4165740 1 --2- 172.21.15.16:0/2800508722 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556cd8ce0bb0 0x556cd8ce6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.505+0000 7f9eaeff5640 0 client.16332 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:14:43.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.505+0000 7f9ed4165740 1 --2- 172.21.15.16:0/2800508722 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556cd8ce0290 0x556cd8ce67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.505+0000 7f9ed4165740 1 -- 172.21.15.16:0/2800508722 >> 172.21.15.16:0/2800508722 conn(0x556cd8cda9c0 msgr2=0x556cd8cd9c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:43.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.505+0000 7f9ed4165740 1 -- 172.21.15.16:0/2800508722 shutdown_connections 2024-03-28T22:14:43.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.505+0000 7f9ed4165740 1 -- 172.21.15.16:0/2800508722 wait complete. 2024-03-28T22:14:43.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.505+0000 7f9ed4165740 20 client.16332 trim_cache size 0 max 16384 2024-03-28T22:14:43.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.523+0000 7f9ed4165740 1 Processor -- start 2024-03-28T22:14:43.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.523+0000 7f9ed4165740 1 -- start start 2024-03-28T22:14:43.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.523+0000 7f9ed4165740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556cd8ce0290 0x556cd8ce8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:43.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.523+0000 7f9ed4165740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556cd8ce0bb0 0x556cd8cf4f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:43.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.523+0000 7f9ed4165740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556cd8ce9790 0x556cd8cd9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:43.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.523+0000 7f9ed4165740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9ecc1b1d40 con 0x556cd8ce0290 2024-03-28T22:14:43.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.523+0000 7f9ed4165740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9ecc1af610 con 0x556cd8ce9790 2024-03-28T22:14:43.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.523+0000 7f9ed4165740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9ecc1ad460 con 0x556cd8ce0bb0 2024-03-28T22:14:43.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.523+0000 7f9ecaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556cd8ce0290 0x556cd8ce8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:43.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.523+0000 7f9ecaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556cd8ce0290 0x556cd8ce8ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47482/0 (socket says 172.21.15.16:47482) 2024-03-28T22:14:43.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.523+0000 7f9ecaffd640 1 -- 172.21.15.16:0/1472315000 learned_addr learned my addr 172.21.15.16:0/1472315000 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:43.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.523+0000 7f9ecb7fe640 1 --2- 172.21.15.16:0/1472315000 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556cd8ce9790 0x556cd8cd9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:43.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.523+0000 7f9eca7fc640 1 --2- 172.21.15.16:0/1472315000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556cd8ce0bb0 0x556cd8cf4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:43.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.523+0000 7f9ecaffd640 1 -- 172.21.15.16:0/1472315000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556cd8ce0bb0 msgr2=0x556cd8cf4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:43.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.523+0000 7f9ecaffd640 1 --2- 172.21.15.16:0/1472315000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556cd8ce0bb0 0x556cd8cf4f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.523+0000 7f9ecaffd640 1 -- 172.21.15.16:0/1472315000 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556cd8ce9790 msgr2=0x556cd8cd9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:43.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.523+0000 7f9ecaffd640 1 --2- 172.21.15.16:0/1472315000 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556cd8ce9790 0x556cd8cd9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.523+0000 7f9ecaffd640 1 -- 172.21.15.16:0/1472315000 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556cd89180f0 con 0x556cd8ce0290 2024-03-28T22:14:43.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.523+0000 7f9eca7fc640 1 --2- 172.21.15.16:0/1472315000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556cd8ce0bb0 0x556cd8cf4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:14:43.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.523+0000 7f9ecb7fe640 1 --2- 172.21.15.16:0/1472315000 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556cd8ce9790 0x556cd8cd9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:14:43.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.523+0000 7f9ecaffd640 1 --2- 172.21.15.16:0/1472315000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556cd8ce0290 0x556cd8ce8ef0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc012d50 tx=0x7f9ebc003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abeefee2ff86942b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:43.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.523+0000 7f9eb0ff9640 1 -- 172.21.15.16:0/1472315000 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ebc021020 con 0x556cd8ce0290 2024-03-28T22:14:43.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.524+0000 7f9eb0ff9640 1 -- 172.21.15.16:0/1472315000 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ebc012e40 con 0x556cd8ce0290 2024-03-28T22:14:43.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.524+0000 7f9eb0ff9640 1 -- 172.21.15.16:0/1472315000 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ebc0116d0 con 0x556cd8ce0290 2024-03-28T22:14:43.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.524+0000 7f9ed4165740 1 -- 172.21.15.16:0/1472315000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556cd8ce0290 msgr2=0x556cd8ce8ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:43.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.524+0000 7f9ed4165740 1 --2- 172.21.15.16:0/1472315000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556cd8ce0290 0x556cd8ce8ef0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc012d50 tx=0x7f9ebc003d40 comp rx=0 tx=0).stop 2024-03-28T22:14:43.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.524+0000 7f9ed4165740 1 -- 172.21.15.16:0/1472315000 shutdown_connections 2024-03-28T22:14:43.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.524+0000 7f9ed4165740 1 --2- 172.21.15.16:0/1472315000 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556cd8ce9790 0x556cd8cd9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.524+0000 7f9ed4165740 1 --2- 172.21.15.16:0/1472315000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556cd8ce0bb0 0x556cd8cf4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.524+0000 7f9ed4165740 1 --2- 172.21.15.16:0/1472315000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556cd8ce0290 0x556cd8ce8ef0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.524+0000 7f9ed4165740 1 -- 172.21.15.16:0/1472315000 >> 172.21.15.16:0/1472315000 conn(0x556cd8cda9c0 msgr2=0x556cd8cd9390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:43.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.524+0000 7f9ed4165740 1 -- 172.21.15.16:0/1472315000 shutdown_connections 2024-03-28T22:14:43.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.524+0000 7f9ed4165740 1 -- 172.21.15.16:0/1472315000 wait complete. 2024-03-28T22:14:43.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.524+0000 7f9ed4165740 1 -- start start 2024-03-28T22:14:43.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.524+0000 7f9ed4165740 1 Processor -- start 2024-03-28T22:14:43.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.525+0000 7f9ed4165740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556cd8ce0290 0x556cd8cec740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:43.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.525+0000 7f9ed4165740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556cd8ce0bb0 0x556cd8cecc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:43.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.525+0000 7f9ed4165740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556cd8ce9790 0x556cd8d5d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:43.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.525+0000 7f9ed4165740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9ecc1adad0 con 0x556cd8ce0bb0 2024-03-28T22:14:43.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.525+0000 7f9ed4165740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9ecc1aef40 con 0x556cd8ce9790 2024-03-28T22:14:43.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.525+0000 7f9ed4165740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9ecc1ae1a0 con 0x556cd8ce0290 2024-03-28T22:14:43.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.525+0000 7f9ed4165740 10 client.0 _ll_register_callbacks cb 0x7f9ec40b88d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:14:43.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.525+0000 7f9eca7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556cd8ce0bb0 0x556cd8cecc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:43.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.525+0000 7f9eca7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556cd8ce0bb0 0x556cd8cecc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47484/0 (socket says 172.21.15.16:47484) 2024-03-28T22:14:43.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.525+0000 7f9eca7fc640 1 -- 172.21.15.16:0/1365631676 learned_addr learned my addr 172.21.15.16:0/1365631676 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:43.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.525+0000 7f9ecb7fe640 1 --2- 172.21.15.16:0/1365631676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556cd8ce9790 0x556cd8d5d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:43.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.525+0000 7f9eca7fc640 1 -- 172.21.15.16:0/1365631676 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556cd8ce0290 msgr2=0x556cd8cec740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:43.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.525+0000 7f9ecaffd640 1 --2- 172.21.15.16:0/1365631676 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556cd8ce0290 0x556cd8cec740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:43.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.525+0000 7f9eca7fc640 1 --2- 172.21.15.16:0/1365631676 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556cd8ce0290 0x556cd8cec740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.525+0000 7f9eca7fc640 1 -- 172.21.15.16:0/1365631676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556cd8ce9790 msgr2=0x556cd8d5d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:43.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.525+0000 7f9eca7fc640 1 --2- 172.21.15.16:0/1365631676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556cd8ce9790 0x556cd8d5d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.525+0000 7f9eca7fc640 1 -- 172.21.15.16:0/1365631676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556cd8d5de10 con 0x556cd8ce0bb0 2024-03-28T22:14:43.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.525+0000 7f9eca7fc640 1 --2- 172.21.15.16:0/1365631676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556cd8ce0bb0 0x556cd8cecc80 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x556cd8cdfd60 tx=0x7f9ec001b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6537ecc4e054ae5e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:43.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.525+0000 7f9eaeff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:14:43.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.525+0000 7f9eaeff5640 1 -- 172.21.15.16:0/1365631676 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec0004220 con 0x556cd8ce0bb0 2024-03-28T22:14:43.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.525+0000 7f9eaeff5640 1 -- 172.21.15.16:0/1365631676 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f9ec00043c0 con 0x556cd8ce0bb0 2024-03-28T22:14:43.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.526+0000 7f9ed4165740 1 -- 172.21.15.16:0/1365631676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556cd8bc9540 con 0x556cd8ce0bb0 2024-03-28T22:14:43.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.526+0000 7f9eaeff5640 1 -- 172.21.15.16:0/1365631676 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9ec00127e0 con 0x556cd8ce0bb0 2024-03-28T22:14:43.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.526+0000 7f9ed4165740 10 client.16338 fetch_fsmap learned FSMap version 26 2024-03-28T22:14:43.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.526+0000 7f9ed4165740 1 -- 172.21.15.16:0/1365631676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x556cd8d5f0a0 con 0x556cd8ce0bb0 2024-03-28T22:14:43.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.526+0000 7f9eaeff5640 1 -- 172.21.15.16:0/1365631676 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x556cd8bc9540 con 0x556cd8ce0bb0 2024-03-28T22:14:43.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.526+0000 7f9eaeff5640 1 -- 172.21.15.16:0/1365631676 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 26) ==== 8342+0+0 (secure 0 0 0) 0x7f9ec002a460 con 0x556cd8ce0bb0 2024-03-28T22:14:43.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.526+0000 7f9ed4165740 10 client.16338 fetch_fsmap finished waiting for FSMap version 26 2024-03-28T22:14:43.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.526+0000 7f9ed4165740 10 client.16338 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:14:43.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.526+0000 7f9ed4165740 20 client.16338 populate_metadata read hostname 'smithi016' 2024-03-28T22:14:43.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.526+0000 7f9ed4165740 1 --2- 172.21.15.16:0/1365631676 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x556cd8cf95e0 0x556cd8d618f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:43.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.526+0000 7f9ed4165740 4 client.16338 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:14:43.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.526+0000 7f9ed4165740 1 -- 172.21.15.16:0/1365631676 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556cd8c15ab0 con 0x556cd8cf95e0 2024-03-28T22:14:43.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.526+0000 7f9ecaffd640 1 --2- 172.21.15.16:0/1365631676 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x556cd8cf95e0 0x556cd8d618f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:43.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.527+0000 7f9ecaffd640 1 --2- 172.21.15.16:0/1365631676 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x556cd8cf95e0 0x556cd8d618f0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9e43cd4e71d5007 server_cookie=1e3aadbdc3ded395 in_seq=0 out_seq=0 2024-03-28T22:14:43.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.527+0000 7f9eaeff5640 10 client.16338 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:14:43.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.527+0000 7f9eaeff5640 1 -- 172.21.15.16:0/1365631676 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x556cd8c15ab0 con 0x556cd8cf95e0 2024-03-28T22:14:43.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.527+0000 7f9eaeff5640 10 client.16338 handle_command_reply: tid=0 2024-03-28T22:14:43.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.527+0000 7f9ed4165740 1 client.16338 shutdown 2024-03-28T22:14:43.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.527+0000 7f9ed4165740 1 -- 172.21.15.16:0/1365631676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556cd8ce0bb0 msgr2=0x556cd8cecc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:43.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.527+0000 7f9ed4165740 1 --2- 172.21.15.16:0/1365631676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556cd8ce0bb0 0x556cd8cecc80 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x556cd8cdfd60 tx=0x7f9ec001b000 comp rx=0 tx=0).stop 2024-03-28T22:14:43.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.527+0000 7f9ed4165740 1 -- 172.21.15.16:0/1365631676 shutdown_connections 2024-03-28T22:14:43.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.527+0000 7f9ed4165740 1 --2- 172.21.15.16:0/1365631676 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x556cd8cf95e0 0x556cd8d618f0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.527+0000 7f9ed4165740 1 --2- 172.21.15.16:0/1365631676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556cd8ce9790 0x556cd8d5d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.527+0000 7f9ed4165740 1 --2- 172.21.15.16:0/1365631676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556cd8ce0bb0 0x556cd8cecc80 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.527+0000 7f9ed4165740 1 --2- 172.21.15.16:0/1365631676 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556cd8ce0290 0x556cd8cec740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.527+0000 7f9eaeff5640 0 client.16338 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:14:43.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.527+0000 7f9ed4165740 1 -- 172.21.15.16:0/1365631676 >> 172.21.15.16:0/1365631676 conn(0x556cd8cda9c0 msgr2=0x556cd8cd9c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:43.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.527+0000 7f9ed4165740 1 -- 172.21.15.16:0/1365631676 shutdown_connections 2024-03-28T22:14:43.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.527+0000 7f9ed4165740 1 -- 172.21.15.16:0/1365631676 wait complete. 2024-03-28T22:14:43.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.527+0000 7f9ed4165740 20 client.16338 trim_cache size 0 max 16384 2024-03-28T22:14:43.545 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:14:43.545 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-03-28T22:14:43.545 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-03-28T22:14:43.545 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:14:43.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.528+0000 7f9ed1581640 1 -- 172.21.15.16:0/414284651 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f9eb40a1f20 msgr2=0x7f9eb40a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:43.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.528+0000 7f9ed1581640 1 --2- 172.21.15.16:0/414284651 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f9eb40a1f20 0x7f9eb40a43e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc103cd0 tx=0x7f9ec00158a0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.528+0000 7f9ed1581640 1 -- 172.21.15.16:0/414284651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ecc106690 msgr2=0x7f9ecc102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:43.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.528+0000 7f9ed1581640 1 --2- 172.21.15.16:0/414284651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ecc106690 0x7f9ecc102d10 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f9ec402efd0 tx=0x7f9ec4004270 comp rx=0 tx=0).stop 2024-03-28T22:14:43.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.528+0000 7f9ed1581640 1 -- 172.21.15.16:0/414284651 shutdown_connections 2024-03-28T22:14:43.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.528+0000 7f9ed1581640 1 --2- 172.21.15.16:0/414284651 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f9eb40a1f20 0x7f9eb40a43e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.528+0000 7f9ed1581640 1 --2- 172.21.15.16:0/414284651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ecc106690 0x7f9ecc102d10 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.528+0000 7f9ed1581640 1 --2- 172.21.15.16:0/414284651 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9ecc100fb0 0x7f9ecc1027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.528+0000 7f9ed1581640 1 --2- 172.21.15.16:0/414284651 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ecc1005f0 0x7f9ecc102290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.528+0000 7f9ed1581640 1 -- 172.21.15.16:0/414284651 >> 172.21.15.16:0/414284651 conn(0x7f9ecc077800 msgr2=0x7f9ecc0ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:43.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.528+0000 7f9ed1581640 1 -- 172.21.15.16:0/414284651 shutdown_connections 2024-03-28T22:14:43.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.528+0000 7f9ed1581640 1 -- 172.21.15.16:0/414284651 wait complete. 2024-03-28T22:14:43.548 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:14:43.549 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:14:43.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.649+0000 7f73f7c0e640 1 -- 172.21.15.16:0/1893014265 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73f010e0a0 msgr2=0x7f73f0110490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:43.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.649+0000 7f73f7c0e640 1 --2- 172.21.15.16:0/1893014265 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73f010e0a0 0x7f73f0110490 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f73ec009ca0 tx=0x7f73ec02c100 comp rx=0 tx=0).stop 2024-03-28T22:14:43.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.649+0000 7f73f7c0e640 1 -- 172.21.15.16:0/1893014265 shutdown_connections 2024-03-28T22:14:43.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.649+0000 7f73f7c0e640 1 --2- 172.21.15.16:0/1893014265 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73f010e0a0 0x7f73f0110490 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.649+0000 7f73f7c0e640 1 --2- 172.21.15.16:0/1893014265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73f0100e10 0x7f73f010db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.649+0000 7f73f7c0e640 1 --2- 172.21.15.16:0/1893014265 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f73f01004f0 0x7f73f01008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.649+0000 7f73f7c0e640 1 -- 172.21.15.16:0/1893014265 >> 172.21.15.16:0/1893014265 conn(0x7f73f00777d0 msgr2=0x7f73f0077be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:43.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.650+0000 7f73f7c0e640 1 -- 172.21.15.16:0/1893014265 shutdown_connections 2024-03-28T22:14:43.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.650+0000 7f73f7c0e640 1 -- 172.21.15.16:0/1893014265 wait complete. 2024-03-28T22:14:43.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.650+0000 7f73f7c0e640 1 Processor -- start 2024-03-28T22:14:43.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.650+0000 7f73f7c0e640 1 -- start start 2024-03-28T22:14:43.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.650+0000 7f73f7c0e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f73f01004f0 0x7f73f019a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:43.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.650+0000 7f73f7c0e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73f0100e10 0x7f73f019a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:43.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.650+0000 7f73f7c0e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73f010e0a0 0x7f73f019eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:43.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.650+0000 7f73f7c0e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f73f0113d00 con 0x7f73f0100e10 2024-03-28T22:14:43.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.650+0000 7f73f7c0e640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f73f0113b80 con 0x7f73f010e0a0 2024-03-28T22:14:43.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.650+0000 7f73f7c0e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f73f0113e80 con 0x7f73f01004f0 2024-03-28T22:14:43.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.650+0000 7f73f51a9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73f0100e10 0x7f73f019a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:43.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.651+0000 7f73f51a9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73f0100e10 0x7f73f019a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47498/0 (socket says 172.21.15.16:47498) 2024-03-28T22:14:43.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.651+0000 7f73f51a9640 1 -- 172.21.15.16:0/721300580 learned_addr learned my addr 172.21.15.16:0/721300580 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:43.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.651+0000 7f73f61ab640 1 --2- 172.21.15.16:0/721300580 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73f010e0a0 0x7f73f019eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:43.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.651+0000 7f73f59aa640 1 --2- 172.21.15.16:0/721300580 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f73f01004f0 0x7f73f019a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:43.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.651+0000 7f73f51a9640 1 -- 172.21.15.16:0/721300580 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f73f01004f0 msgr2=0x7f73f019a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:43.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.651+0000 7f73f51a9640 1 --2- 172.21.15.16:0/721300580 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f73f01004f0 0x7f73f019a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.651+0000 7f73f51a9640 1 -- 172.21.15.16:0/721300580 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73f010e0a0 msgr2=0x7f73f019eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:43.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.651+0000 7f73f51a9640 1 --2- 172.21.15.16:0/721300580 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73f010e0a0 0x7f73f019eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.651+0000 7f73f51a9640 1 -- 172.21.15.16:0/721300580 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73f019f1d0 con 0x7f73f0100e10 2024-03-28T22:14:43.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.651+0000 7f73f61ab640 1 --2- 172.21.15.16:0/721300580 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73f010e0a0 0x7f73f019eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:14:43.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.651+0000 7f73f51a9640 1 --2- 172.21.15.16:0/721300580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73f0100e10 0x7f73f019a720 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f73e000ea60 tx=0x7f73e000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:43.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.651+0000 7f73e6ffd640 1 -- 172.21.15.16:0/721300580 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e000cde0 con 0x7f73f0100e10 2024-03-28T22:14:43.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.651+0000 7f73f7c0e640 1 -- 172.21.15.16:0/721300580 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73f019f4c0 con 0x7f73f0100e10 2024-03-28T22:14:43.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.651+0000 7f73e6ffd640 1 -- 172.21.15.16:0/721300580 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73e00035f0 con 0x7f73f0100e10 2024-03-28T22:14:43.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.651+0000 7f73e6ffd640 1 -- 172.21.15.16:0/721300580 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e0010670 con 0x7f73f0100e10 2024-03-28T22:14:43.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.651+0000 7f73f7c0e640 1 -- 172.21.15.16:0/721300580 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73f01a6d50 con 0x7f73f0100e10 2024-03-28T22:14:43.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.652+0000 7f73f7c0e640 1 -- 172.21.15.16:0/721300580 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73f00685d0 con 0x7f73f0100e10 2024-03-28T22:14:43.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.653+0000 7f73e6ffd640 1 -- 172.21.15.16:0/721300580 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f73e00037a0 con 0x7f73f0100e10 2024-03-28T22:14:43.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.654+0000 7f73e6ffd640 1 --2- 172.21.15.16:0/721300580 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f73dc0a1ef0 0x7f73dc0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:43.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.654+0000 7f73f59aa640 1 --2- 172.21.15.16:0/721300580 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f73dc0a1ef0 0x7f73dc0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:43.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.654+0000 7f73e6ffd640 1 -- 172.21.15.16:0/721300580 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f73e00ce3e0 con 0x7f73f0100e10 2024-03-28T22:14:43.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.654+0000 7f73f59aa640 1 --2- 172.21.15.16:0/721300580 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f73dc0a1ef0 0x7f73dc0a43b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f73e8005fd0 tx=0x7f73e800b390 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:43.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.658+0000 7f73e6ffd640 1 -- 172.21.15.16:0/721300580 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f73e009b4f0 con 0x7f73f0100e10 2024-03-28T22:14:43.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.881+0000 7f73f7c0e640 1 -- 172.21.15.16:0/721300580 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f73f002d170 con 0x7f73f0100e10 2024-03-28T22:14:43.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.881+0000 7f73e6ffd640 1 -- 172.21.15.16:0/721300580 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11852 (secure 0 0 0) 0x7f73e009adf0 con 0x7f73f0100e10 2024-03-28T22:14:43.884 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:14:43.885 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/1644581227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1644581227},{"type":"v1","addr":"172.21.15.16:6839","nonce":1644581227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/2593062669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2593062669},{"type":"v1","addr":"172.21.15.16:6841","nonce":2593062669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:08:29.672976+0000","modified":"2024-03-28T22:14:25.678443+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24566,"mds_1":24524,"mds_2":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24466":{"gid":24466,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/247249857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":247249857},{"type":"v1","addr":"172.21.15.63:6835","nonce":247249857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6837/2324231032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":2324231032},{"type":"v1","addr":"172.21.15.63:6837","nonce":2324231032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6839/1177197077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":1177197077},{"type":"v1","addr":"172.21.15.63:6839","nonce":1177197077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"e","rank":2,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.53:6835/350727634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":350727634},{"type":"v1","addr":"172.21.15.53:6835","nonce":350727634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/349877573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":349877573},{"type":"v1","addr":"172.21.15.53:6837","nonce":349877573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6839/3714153956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":3714153956},{"type":"v1","addr":"172.21.15.53:6839","nonce":3714153956}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24566,"qdb_cluster":[24500,24566,24524]},"id":1}]} 2024-03-28T22:14:43.885 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 26 2024-03-28T22:14:43.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.883+0000 7f73f7c0e640 1 -- 172.21.15.16:0/721300580 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f73dc0a1ef0 msgr2=0x7f73dc0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:43.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.883+0000 7f73f7c0e640 1 --2- 172.21.15.16:0/721300580 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f73dc0a1ef0 0x7f73dc0a43b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f73e8005fd0 tx=0x7f73e800b390 comp rx=0 tx=0).stop 2024-03-28T22:14:43.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.883+0000 7f73f7c0e640 1 -- 172.21.15.16:0/721300580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73f0100e10 msgr2=0x7f73f019a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:43.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.883+0000 7f73f7c0e640 1 --2- 172.21.15.16:0/721300580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73f0100e10 0x7f73f019a720 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f73e000ea60 tx=0x7f73e000ef20 comp rx=0 tx=0).stop 2024-03-28T22:14:43.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.883+0000 7f73f7c0e640 1 -- 172.21.15.16:0/721300580 shutdown_connections 2024-03-28T22:14:43.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.883+0000 7f73f7c0e640 1 --2- 172.21.15.16:0/721300580 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f73dc0a1ef0 0x7f73dc0a43b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.883+0000 7f73f7c0e640 1 --2- 172.21.15.16:0/721300580 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73f010e0a0 0x7f73f019eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.883+0000 7f73f7c0e640 1 --2- 172.21.15.16:0/721300580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73f0100e10 0x7f73f019a720 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.883+0000 7f73f7c0e640 1 --2- 172.21.15.16:0/721300580 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f73f01004f0 0x7f73f019a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:43.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.883+0000 7f73f7c0e640 1 -- 172.21.15.16:0/721300580 >> 172.21.15.16:0/721300580 conn(0x7f73f00777d0 msgr2=0x7f73f010f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:43.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.884+0000 7f73f7c0e640 1 -- 172.21.15.16:0/721300580 shutdown_connections 2024-03-28T22:14:43.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:43.884+0000 7f73f7c0e640 1 -- 172.21.15.16:0/721300580 wait complete. 2024-03-28T22:14:43.898 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:14:44.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.001+0000 7f6b55fe2640 1 -- 172.21.15.16:0/94259978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b50103290 msgr2=0x7f6b5010b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:44.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.001+0000 7f6b55fe2640 1 --2- 172.21.15.16:0/94259978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b50103290 0x7f6b5010b770 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f6b44009900 tx=0x7f6b4402bcd0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.001+0000 7f6b55fe2640 1 -- 172.21.15.16:0/94259978 shutdown_connections 2024-03-28T22:14:44.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.001+0000 7f6b55fe2640 1 --2- 172.21.15.16:0/94259978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6b5010bd40 0x7f6b5010e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.001+0000 7f6b55fe2640 1 --2- 172.21.15.16:0/94259978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b50103290 0x7f6b5010b770 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.001+0000 7f6b55fe2640 1 --2- 172.21.15.16:0/94259978 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b50102970 0x7f6b50102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.001+0000 7f6b55fe2640 1 -- 172.21.15.16:0/94259978 >> 172.21.15.16:0/94259978 conn(0x7f6b50077800 msgr2=0x7f6b50077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:44.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.001+0000 7f6b55fe2640 1 -- 172.21.15.16:0/94259978 shutdown_connections 2024-03-28T22:14:44.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.001+0000 7f6b55fe2640 1 -- 172.21.15.16:0/94259978 wait complete. 2024-03-28T22:14:44.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.001+0000 7f6b55fe2640 1 Processor -- start 2024-03-28T22:14:44.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.002+0000 7f6b55fe2640 1 -- start start 2024-03-28T22:14:44.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.002+0000 7f6b55fe2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b50102970 0x7f6b50197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.002+0000 7f6b55fe2640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b50103290 0x7f6b50198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.002+0000 7f6b55fe2640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6b5010bd40 0x7f6b5019c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.002+0000 7f6b55fe2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6b50111ad0 con 0x7f6b50102970 2024-03-28T22:14:44.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.002+0000 7f6b55fe2640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f6b50111950 con 0x7f6b50103290 2024-03-28T22:14:44.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.002+0000 7f6b55fe2640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6b50111c50 con 0x7f6b5010bd40 2024-03-28T22:14:44.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.002+0000 7f6b4f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b50102970 0x7f6b50197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.002+0000 7f6b4effd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b50103290 0x7f6b50198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.002+0000 7f6b4effd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b50103290 0x7f6b50198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:50992/0 (socket says 172.21.15.16:50992) 2024-03-28T22:14:44.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.002+0000 7f6b4effd640 1 -- 172.21.15.16:0/2676492293 learned_addr learned my addr 172.21.15.16:0/2676492293 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:44.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.002+0000 7f6b4ffff640 1 --2- 172.21.15.16:0/2676492293 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6b5010bd40 0x7f6b5019c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.002+0000 7f6b4f7fe640 1 -- 172.21.15.16:0/2676492293 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6b5010bd40 msgr2=0x7f6b5019c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:44.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.002+0000 7f6b4f7fe640 1 --2- 172.21.15.16:0/2676492293 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6b5010bd40 0x7f6b5019c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.002+0000 7f6b4f7fe640 1 -- 172.21.15.16:0/2676492293 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b50103290 msgr2=0x7f6b50198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:44.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.002+0000 7f6b4f7fe640 1 --2- 172.21.15.16:0/2676492293 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b50103290 0x7f6b50198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.002+0000 7f6b4f7fe640 1 -- 172.21.15.16:0/2676492293 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b5019cf90 con 0x7f6b50102970 2024-03-28T22:14:44.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.002+0000 7f6b4f7fe640 1 --2- 172.21.15.16:0/2676492293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b50102970 0x7f6b50197ff0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f6b4000e9f0 tx=0x7f6b4000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:44.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.003+0000 7f6b4cff9640 1 -- 172.21.15.16:0/2676492293 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b4000ccc0 con 0x7f6b50102970 2024-03-28T22:14:44.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.003+0000 7f6b4cff9640 1 -- 172.21.15.16:0/2676492293 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b40010910 con 0x7f6b50102970 2024-03-28T22:14:44.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.003+0000 7f6b55fe2640 1 -- 172.21.15.16:0/2676492293 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b5019d280 con 0x7f6b50102970 2024-03-28T22:14:44.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.003+0000 7f6b4cff9640 1 -- 172.21.15.16:0/2676492293 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b40018960 con 0x7f6b50102970 2024-03-28T22:14:44.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.003+0000 7f6b55fe2640 1 -- 172.21.15.16:0/2676492293 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b501a4b10 con 0x7f6b50102970 2024-03-28T22:14:44.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.005+0000 7f6b4cff9640 1 -- 172.21.15.16:0/2676492293 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f6b40010ac0 con 0x7f6b50102970 2024-03-28T22:14:44.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.005+0000 7f6b4cff9640 1 --2- 172.21.15.16:0/2676492293 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f6b380a1ef0 0x7f6b380a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.005+0000 7f6b4effd640 1 --2- 172.21.15.16:0/2676492293 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f6b380a1ef0 0x7f6b380a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.005+0000 7f6b4cff9640 1 -- 172.21.15.16:0/2676492293 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f6b400ce500 con 0x7f6b50102970 2024-03-28T22:14:44.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.006+0000 7f6b4effd640 1 --2- 172.21.15.16:0/2676492293 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f6b380a1ef0 0x7f6b380a43b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6b44000c00 tx=0x7f6b4402cf90 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:44.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.006+0000 7f6b58bc2740 1 Processor -- start 2024-03-28T22:14:44.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.006+0000 7f6b58bc2740 1 -- start start 2024-03-28T22:14:44.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.006+0000 7f6b58bc2740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5610e3ae3470 0x5610e3ae3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.006+0000 7f6b58bc2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610e3ae3d90 0x5610e3aec430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.007+0000 7f6b58bc2740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610e3aec970 0x5610e3aeed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.007+0000 7f6b58bc2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5610e3a471b0 con 0x5610e3ae3d90 2024-03-28T22:14:44.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.007+0000 7f6b58bc2740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5610e39efc00 con 0x5610e3ae3470 2024-03-28T22:14:44.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.007+0000 7f6b58bc2740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5610e3a43250 con 0x5610e3aec970 2024-03-28T22:14:44.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.007+0000 7f6b4effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610e3ae3d90 0x5610e3aec430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.007+0000 7f6b4f7fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5610e3ae3470 0x5610e3ae3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.007+0000 7f6b4effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610e3ae3d90 0x5610e3aec430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47522/0 (socket says 172.21.15.16:47522) 2024-03-28T22:14:44.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.007+0000 7f6b4effd640 1 -- 172.21.15.16:0/340143721 learned_addr learned my addr 172.21.15.16:0/340143721 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:44.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.007+0000 7f6b4ffff640 1 --2- 172.21.15.16:0/340143721 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610e3aec970 0x5610e3aeed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.007+0000 7f6b4effd640 1 -- 172.21.15.16:0/340143721 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610e3aec970 msgr2=0x5610e3aeed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:44.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.007+0000 7f6b4effd640 1 --2- 172.21.15.16:0/340143721 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610e3aec970 0x5610e3aeed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.007+0000 7f6b4effd640 1 -- 172.21.15.16:0/340143721 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5610e3ae3470 msgr2=0x5610e3ae3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:44.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.007+0000 7f6b4effd640 1 --2- 172.21.15.16:0/340143721 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5610e3ae3470 0x5610e3ae3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.007+0000 7f6b4effd640 1 -- 172.21.15.16:0/340143721 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610e3a0f3d0 con 0x5610e3ae3d90 2024-03-28T22:14:44.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.007+0000 7f6b4f7fe640 1 --2- 172.21.15.16:0/340143721 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5610e3ae3470 0x5610e3ae3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:14:44.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.007+0000 7f6b4effd640 1 --2- 172.21.15.16:0/340143721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610e3ae3d90 0x5610e3aec430 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f6b44009430 tx=0x7f6b4402e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7759cec3f4c5d353 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:44.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.007+0000 7f6b35ffb640 1 -- 172.21.15.16:0/340143721 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b4404f070 con 0x5610e3ae3d90 2024-03-28T22:14:44.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.007+0000 7f6b35ffb640 1 -- 172.21.15.16:0/340143721 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b44004ba0 con 0x5610e3ae3d90 2024-03-28T22:14:44.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.007+0000 7f6b35ffb640 1 -- 172.21.15.16:0/340143721 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b440035f0 con 0x5610e3ae3d90 2024-03-28T22:14:44.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.008+0000 7f6b58bc2740 1 -- 172.21.15.16:0/340143721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610e3ae3d90 msgr2=0x5610e3aec430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:44.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.008+0000 7f6b58bc2740 1 --2- 172.21.15.16:0/340143721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610e3ae3d90 0x5610e3aec430 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f6b44009430 tx=0x7f6b4402e050 comp rx=0 tx=0).stop 2024-03-28T22:14:44.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.008+0000 7f6b58bc2740 1 -- 172.21.15.16:0/340143721 shutdown_connections 2024-03-28T22:14:44.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.008+0000 7f6b58bc2740 1 --2- 172.21.15.16:0/340143721 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610e3aec970 0x5610e3aeed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.008+0000 7f6b58bc2740 1 --2- 172.21.15.16:0/340143721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610e3ae3d90 0x5610e3aec430 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.008+0000 7f6b58bc2740 1 --2- 172.21.15.16:0/340143721 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5610e3ae3470 0x5610e3ae3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.008+0000 7f6b58bc2740 1 -- 172.21.15.16:0/340143721 >> 172.21.15.16:0/340143721 conn(0x5610e3addba0 msgr2=0x5610e3addf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:44.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.008+0000 7f6b58bc2740 1 -- 172.21.15.16:0/340143721 shutdown_connections 2024-03-28T22:14:44.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.008+0000 7f6b58bc2740 1 -- 172.21.15.16:0/340143721 wait complete. 2024-03-28T22:14:44.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.008+0000 7f6b58bc2740 1 -- start start 2024-03-28T22:14:44.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.009+0000 7f6b58bc2740 1 Processor -- start 2024-03-28T22:14:44.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.009+0000 7f6b58bc2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610e3ae3470 0x5610e3ae99f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.009+0000 7f6b58bc2740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610e3ae3d90 0x5610e3ae9f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.009+0000 7f6b58bc2740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5610e3aec970 0x5610e3aea470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.009+0000 7f6b58bc2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5610e3aef700 con 0x5610e3ae3470 2024-03-28T22:14:44.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.009+0000 7f6b58bc2740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5610e39e5470 con 0x5610e3aec970 2024-03-28T22:14:44.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.009+0000 7f6b58bc2740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5610e3aef880 con 0x5610e3ae3d90 2024-03-28T22:14:44.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.009+0000 7f6b58bc2740 10 client.0 _ll_register_callbacks cb 0x5610e3a02cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:14:44.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.009+0000 7f6b4f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610e3ae3470 0x5610e3ae99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.009+0000 7f6b4f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610e3ae3470 0x5610e3ae99f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47524/0 (socket says 172.21.15.16:47524) 2024-03-28T22:14:44.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.009+0000 7f6b4f7fe640 1 -- 172.21.15.16:0/2931724405 learned_addr learned my addr 172.21.15.16:0/2931724405 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:44.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.009+0000 7f6b4effd640 1 --2- 172.21.15.16:0/2931724405 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610e3ae3d90 0x5610e3ae9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.009+0000 7f6b4ffff640 1 --2- 172.21.15.16:0/2931724405 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5610e3aec970 0x5610e3aea470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.009+0000 7f6b4f7fe640 1 -- 172.21.15.16:0/2931724405 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610e3ae3d90 msgr2=0x5610e3ae9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:44.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.009+0000 7f6b4f7fe640 1 --2- 172.21.15.16:0/2931724405 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610e3ae3d90 0x5610e3ae9f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.009+0000 7f6b4f7fe640 1 -- 172.21.15.16:0/2931724405 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5610e3aec970 msgr2=0x5610e3aea470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:44.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.009+0000 7f6b4f7fe640 1 --2- 172.21.15.16:0/2931724405 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5610e3aec970 0x5610e3aea470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.009+0000 7f6b4f7fe640 1 -- 172.21.15.16:0/2931724405 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610e375b3f0 con 0x5610e3ae3470 2024-03-28T22:14:44.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.009+0000 7f6b4effd640 1 --2- 172.21.15.16:0/2931724405 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610e3ae3d90 0x5610e3ae9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:14:44.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.010+0000 7f6b4f7fe640 1 --2- 172.21.15.16:0/2931724405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610e3ae3470 0x5610e3ae99f0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f6b400cbd20 tx=0x7f6b4009f120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a1838c930114c3b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:44.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.010+0000 7f6b2effd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:14:44.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.010+0000 7f6b2effd640 1 -- 172.21.15.16:0/2931724405 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b400b2020 con 0x5610e3ae3470 2024-03-28T22:14:44.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.010+0000 7f6b2effd640 1 -- 172.21.15.16:0/2931724405 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f6b400ce240 con 0x5610e3ae3470 2024-03-28T22:14:44.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.010+0000 7f6b58bc2740 1 -- 172.21.15.16:0/2931724405 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b44004ba0 con 0x5610e3ae3470 2024-03-28T22:14:44.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.010+0000 7f6b2effd640 1 -- 172.21.15.16:0/2931724405 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6b400b92e0 con 0x5610e3ae3470 2024-03-28T22:14:44.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.010+0000 7f6b58bc2740 10 client.16356 fetch_fsmap learned FSMap version 26 2024-03-28T22:14:44.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.010+0000 7f6b58bc2740 1 -- 172.21.15.16:0/2931724405 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5610e38257d0 con 0x5610e3ae3470 2024-03-28T22:14:44.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.010+0000 7f6b2effd640 1 -- 172.21.15.16:0/2931724405 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b44004ba0 con 0x5610e3ae3470 2024-03-28T22:14:44.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.010+0000 7f6b2effd640 1 -- 172.21.15.16:0/2931724405 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 26) ==== 8342+0+0 (secure 0 0 0) 0x7f6b40018c90 con 0x5610e3ae3470 2024-03-28T22:14:44.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.010+0000 7f6b58bc2740 10 client.16356 fetch_fsmap finished waiting for FSMap version 26 2024-03-28T22:14:44.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.010+0000 7f6b58bc2740 10 client.16356 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-28T22:14:44.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.010+0000 7f6b58bc2740 20 client.16356 populate_metadata read hostname 'smithi016' 2024-03-28T22:14:44.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.010+0000 7f6b58bc2740 1 --2- 172.21.15.16:0/2931724405 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x5610e3afc980 0x5610e3afedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.010+0000 7f6b58bc2740 4 client.16356 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:14:44.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.010+0000 7f6b58bc2740 1 -- 172.21.15.16:0/2931724405 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5610e3a546f0 con 0x5610e3afc980 2024-03-28T22:14:44.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.010+0000 7f6b4effd640 1 --2- 172.21.15.16:0/2931724405 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x5610e3afc980 0x5610e3afedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.011+0000 7f6b2effd640 10 client.16356 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:14:44.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.011+0000 7f6b4effd640 1 --2- 172.21.15.16:0/2931724405 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x5610e3afc980 0x5610e3afedd0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a1079d7ff1ce77b8 server_cookie=dedd4dfc2ae43595 in_seq=0 out_seq=0 2024-03-28T22:14:44.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.012+0000 7f6b2effd640 1 -- 172.21.15.16:0/2931724405 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5610e3a546f0 con 0x5610e3afc980 2024-03-28T22:14:44.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.012+0000 7f6b2effd640 10 client.16356 handle_command_reply: tid=0 2024-03-28T22:14:44.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.012+0000 7f6b58bc2740 1 client.16356 shutdown 2024-03-28T22:14:44.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.012+0000 7f6b58bc2740 1 -- 172.21.15.16:0/2931724405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610e3ae3470 msgr2=0x5610e3ae99f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:44.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.012+0000 7f6b58bc2740 1 --2- 172.21.15.16:0/2931724405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610e3ae3470 0x5610e3ae99f0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f6b400cbd20 tx=0x7f6b4009f120 comp rx=0 tx=0).stop 2024-03-28T22:14:44.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.012+0000 7f6b58bc2740 1 -- 172.21.15.16:0/2931724405 shutdown_connections 2024-03-28T22:14:44.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.012+0000 7f6b58bc2740 1 --2- 172.21.15.16:0/2931724405 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x5610e3afc980 0x5610e3afedd0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.012+0000 7f6b58bc2740 1 --2- 172.21.15.16:0/2931724405 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5610e3aec970 0x5610e3aea470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.012+0000 7f6b58bc2740 1 --2- 172.21.15.16:0/2931724405 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610e3ae3d90 0x5610e3ae9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.012+0000 7f6b2effd640 0 client.16356 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:14:44.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.012+0000 7f6b58bc2740 1 --2- 172.21.15.16:0/2931724405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610e3ae3470 0x5610e3ae99f0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.012+0000 7f6b58bc2740 1 -- 172.21.15.16:0/2931724405 >> 172.21.15.16:0/2931724405 conn(0x5610e3addba0 msgr2=0x5610e3adf8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:44.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.012+0000 7f6b58bc2740 1 -- 172.21.15.16:0/2931724405 shutdown_connections 2024-03-28T22:14:44.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.012+0000 7f6b58bc2740 1 -- 172.21.15.16:0/2931724405 wait complete. 2024-03-28T22:14:44.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.012+0000 7f6b58bc2740 20 client.16356 trim_cache size 0 max 16384 2024-03-28T22:14:44.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.026+0000 7f6b58bc2740 1 Processor -- start 2024-03-28T22:14:44.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.026+0000 7f6b58bc2740 1 -- start start 2024-03-28T22:14:44.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.026+0000 7f6b58bc2740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5610e3ae3470 0x5610e3adfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.026+0000 7f6b58bc2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610e3ae3d90 0x5610e3af84d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.026+0000 7f6b58bc2740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610e3aec970 0x5610e3ae91e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.026+0000 7f6b58bc2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6b501afc60 con 0x5610e3ae3d90 2024-03-28T22:14:44.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.026+0000 7f6b58bc2740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f6b501ad530 con 0x5610e3ae3470 2024-03-28T22:14:44.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.026+0000 7f6b58bc2740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6b501ab380 con 0x5610e3aec970 2024-03-28T22:14:44.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.026+0000 7f6b4effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610e3ae3d90 0x5610e3af84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.026+0000 7f6b4f7fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5610e3ae3470 0x5610e3adfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.026+0000 7f6b4effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610e3ae3d90 0x5610e3af84d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47540/0 (socket says 172.21.15.16:47540) 2024-03-28T22:14:44.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.026+0000 7f6b4effd640 1 -- 172.21.15.16:0/152501113 learned_addr learned my addr 172.21.15.16:0/152501113 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:44.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.026+0000 7f6b4ffff640 1 --2- 172.21.15.16:0/152501113 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610e3aec970 0x5610e3ae91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.026+0000 7f6b4effd640 1 -- 172.21.15.16:0/152501113 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610e3aec970 msgr2=0x5610e3ae91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:44.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.026+0000 7f6b4effd640 1 --2- 172.21.15.16:0/152501113 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610e3aec970 0x5610e3ae91e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.026+0000 7f6b4effd640 1 -- 172.21.15.16:0/152501113 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5610e3ae3470 msgr2=0x5610e3adfaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:44.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.026+0000 7f6b4effd640 1 --2- 172.21.15.16:0/152501113 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5610e3ae3470 0x5610e3adfaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.026+0000 7f6b4effd640 1 -- 172.21.15.16:0/152501113 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610e3ae4810 con 0x5610e3ae3d90 2024-03-28T22:14:44.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.026+0000 7f6b4effd640 1 --2- 172.21.15.16:0/152501113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610e3ae3d90 0x5610e3af84d0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f6b44009430 tx=0x7f6b440053b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=806c53c48e301393 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:44.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.027+0000 7f6b357fa640 1 -- 172.21.15.16:0/152501113 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b44042070 con 0x5610e3ae3d90 2024-03-28T22:14:44.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.027+0000 7f6b357fa640 1 -- 172.21.15.16:0/152501113 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b440054e0 con 0x5610e3ae3d90 2024-03-28T22:14:44.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.027+0000 7f6b357fa640 1 -- 172.21.15.16:0/152501113 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b4404f040 con 0x5610e3ae3d90 2024-03-28T22:14:44.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.027+0000 7f6b58bc2740 1 -- 172.21.15.16:0/152501113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610e3ae3d90 msgr2=0x5610e3af84d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:44.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.027+0000 7f6b58bc2740 1 --2- 172.21.15.16:0/152501113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610e3ae3d90 0x5610e3af84d0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f6b44009430 tx=0x7f6b440053b0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.027+0000 7f6b58bc2740 1 -- 172.21.15.16:0/152501113 shutdown_connections 2024-03-28T22:14:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.027+0000 7f6b58bc2740 1 --2- 172.21.15.16:0/152501113 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610e3aec970 0x5610e3ae91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.027+0000 7f6b58bc2740 1 --2- 172.21.15.16:0/152501113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610e3ae3d90 0x5610e3af84d0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.027+0000 7f6b58bc2740 1 --2- 172.21.15.16:0/152501113 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5610e3ae3470 0x5610e3adfaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.027+0000 7f6b58bc2740 1 -- 172.21.15.16:0/152501113 >> 172.21.15.16:0/152501113 conn(0x5610e3addba0 msgr2=0x5610e3adcd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.027+0000 7f6b58bc2740 1 -- 172.21.15.16:0/152501113 shutdown_connections 2024-03-28T22:14:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.027+0000 7f6b58bc2740 1 -- 172.21.15.16:0/152501113 wait complete. 2024-03-28T22:14:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.028+0000 7f6b58bc2740 1 -- start start 2024-03-28T22:14:44.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.028+0000 7f6b58bc2740 1 Processor -- start 2024-03-28T22:14:44.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.028+0000 7f6b58bc2740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610e3ae3470 0x5610e3ae64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.028+0000 7f6b58bc2740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5610e3ae3d90 0x5610e3ae6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.028+0000 7f6b58bc2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610e3aec970 0x5610e3ae6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.028+0000 7f6b58bc2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6b501ab9f0 con 0x5610e3aec970 2024-03-28T22:14:44.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.028+0000 7f6b58bc2740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f6b501ace60 con 0x5610e3ae3d90 2024-03-28T22:14:44.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.028+0000 7f6b58bc2740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6b501ac0c0 con 0x5610e3ae3470 2024-03-28T22:14:44.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.028+0000 7f6b58bc2740 10 client.0 _ll_register_callbacks cb 0x7f6b4009afc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:14:44.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.028+0000 7f6b4ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610e3aec970 0x5610e3ae6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.028+0000 7f6b4ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610e3aec970 0x5610e3ae6f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47544/0 (socket says 172.21.15.16:47544) 2024-03-28T22:14:44.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.028+0000 7f6b4f7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610e3ae3470 0x5610e3ae64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.028+0000 7f6b4ffff640 1 -- 172.21.15.16:0/2893989067 learned_addr learned my addr 172.21.15.16:0/2893989067 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:44.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.028+0000 7f6b4effd640 1 --2- 172.21.15.16:0/2893989067 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5610e3ae3d90 0x5610e3ae6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.029+0000 7f6b4f7fe640 1 -- 172.21.15.16:0/2893989067 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5610e3ae3d90 msgr2=0x5610e3ae6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:44.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.029+0000 7f6b4f7fe640 1 --2- 172.21.15.16:0/2893989067 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5610e3ae3d90 0x5610e3ae6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.029+0000 7f6b4f7fe640 1 -- 172.21.15.16:0/2893989067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610e3aec970 msgr2=0x5610e3ae6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:44.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.029+0000 7f6b4f7fe640 1 --2- 172.21.15.16:0/2893989067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610e3aec970 0x5610e3ae6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.029+0000 7f6b4f7fe640 1 -- 172.21.15.16:0/2893989067 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610e3b61cd0 con 0x5610e3ae3470 2024-03-28T22:14:44.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.029+0000 7f6b4ffff640 1 --2- 172.21.15.16:0/2893989067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610e3aec970 0x5610e3ae6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:14:44.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.029+0000 7f6b4f7fe640 1 --2- 172.21.15.16:0/2893989067 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610e3ae3470 0x5610e3ae64d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6b40097990 tx=0x7f6b400ce2f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c75ac943154a1be server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:44.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.029+0000 7f6b2effd640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-03-28T22:14:44.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.029+0000 7f6b2effd640 1 -- 172.21.15.16:0/2893989067 <== mon.1 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b400b2040 con 0x5610e3ae3470 2024-03-28T22:14:44.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.029+0000 7f6b58bc2740 1 -- 172.21.15.16:0/2893989067 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b440054e0 con 0x5610e3ae3470 2024-03-28T22:14:44.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.029+0000 7f6b2effd640 1 -- 172.21.15.16:0/2893989067 <== mon.1 v2:172.21.15.63:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f6b40018b00 con 0x5610e3ae3470 2024-03-28T22:14:44.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.029+0000 7f6b2effd640 1 -- 172.21.15.16:0/2893989067 <== mon.1 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6b400a8050 con 0x5610e3ae3470 2024-03-28T22:14:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.029+0000 7f6b58bc2740 10 client.25762 fetch_fsmap learned FSMap version 26 2024-03-28T22:14:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.029+0000 7f6b58bc2740 1 -- 172.21.15.16:0/2893989067 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5610e3b62f30 con 0x5610e3ae3470 2024-03-28T22:14:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.029+0000 7f6b2effd640 1 -- 172.21.15.16:0/2893989067 <== mon.1 v2:172.21.15.63:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b440054e0 con 0x5610e3ae3470 2024-03-28T22:14:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.029+0000 7f6b2effd640 1 -- 172.21.15.16:0/2893989067 <== mon.1 v2:172.21.15.63:3300/0 5 ==== fsmap(e 26) ==== 8342+0+0 (secure 0 0 0) 0x7f6b400840d0 con 0x5610e3ae3470 2024-03-28T22:14:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.030+0000 7f6b58bc2740 10 client.25762 fetch_fsmap finished waiting for FSMap version 26 2024-03-28T22:14:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.030+0000 7f6b58bc2740 10 client.25762 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-28T22:14:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.030+0000 7f6b58bc2740 20 client.25762 populate_metadata read hostname 'smithi016' 2024-03-28T22:14:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.030+0000 7f6b58bc2740 1 --2- 172.21.15.16:0/2893989067 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x5610e3b634c0 0x5610e3b658b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.030+0000 7f6b58bc2740 4 client.25762 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:14:44.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.030+0000 7f6b58bc2740 1 -- 172.21.15.16:0/2893989067 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5610e3a4ea40 con 0x5610e3b634c0 2024-03-28T22:14:44.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.030+0000 7f6b4effd640 1 --2- 172.21.15.16:0/2893989067 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x5610e3b634c0 0x5610e3b658b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.030+0000 7f6b4effd640 1 --2- 172.21.15.16:0/2893989067 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x5610e3b634c0 0x5610e3b658b0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=46c857267a8333d5 server_cookie=9578691bd29a1af5 in_seq=0 out_seq=0 2024-03-28T22:14:44.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.030+0000 7f6b2effd640 10 client.25762 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:14:44.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.030+0000 7f6b2effd640 1 -- 172.21.15.16:0/2893989067 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5610e3a4ea40 con 0x5610e3b634c0 2024-03-28T22:14:44.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.030+0000 7f6b2effd640 10 client.25762 handle_command_reply: tid=0 2024-03-28T22:14:44.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.030+0000 7f6b58bc2740 1 client.25762 shutdown 2024-03-28T22:14:44.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.031+0000 7f6b58bc2740 1 -- 172.21.15.16:0/2893989067 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610e3ae3470 msgr2=0x5610e3ae64d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:44.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.031+0000 7f6b58bc2740 1 --2- 172.21.15.16:0/2893989067 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610e3ae3470 0x5610e3ae64d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6b40097990 tx=0x7f6b400ce2f0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.031+0000 7f6b58bc2740 1 -- 172.21.15.16:0/2893989067 shutdown_connections 2024-03-28T22:14:44.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.031+0000 7f6b58bc2740 1 --2- 172.21.15.16:0/2893989067 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x5610e3b634c0 0x5610e3b658b0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.031+0000 7f6b58bc2740 1 --2- 172.21.15.16:0/2893989067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610e3aec970 0x5610e3ae6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.031+0000 7f6b2effd640 0 client.25762 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:14:44.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.031+0000 7f6b58bc2740 1 --2- 172.21.15.16:0/2893989067 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5610e3ae3d90 0x5610e3ae6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.031+0000 7f6b58bc2740 1 --2- 172.21.15.16:0/2893989067 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610e3ae3470 0x5610e3ae64d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.031+0000 7f6b58bc2740 1 -- 172.21.15.16:0/2893989067 >> 172.21.15.16:0/2893989067 conn(0x5610e3addba0 msgr2=0x5610e3adc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:44.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.031+0000 7f6b58bc2740 1 -- 172.21.15.16:0/2893989067 shutdown_connections 2024-03-28T22:14:44.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.031+0000 7f6b58bc2740 1 -- 172.21.15.16:0/2893989067 wait complete. 2024-03-28T22:14:44.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.031+0000 7f6b58bc2740 20 client.25762 trim_cache size 0 max 16384 2024-03-28T22:14:44.044 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:14:44.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.031+0000 7f6b55fe2640 1 -- 172.21.15.16:0/2676492293 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f6b380a1ef0 msgr2=0x7f6b380a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:44.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.031+0000 7f6b55fe2640 1 --2- 172.21.15.16:0/2676492293 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f6b380a1ef0 0x7f6b380a43b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6b44000c00 tx=0x7f6b4402cf90 comp rx=0 tx=0).stop 2024-03-28T22:14:44.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.031+0000 7f6b55fe2640 1 -- 172.21.15.16:0/2676492293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b50102970 msgr2=0x7f6b50197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:44.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.031+0000 7f6b55fe2640 1 --2- 172.21.15.16:0/2676492293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b50102970 0x7f6b50197ff0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f6b4000e9f0 tx=0x7f6b4000eeb0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.031+0000 7f6b55fe2640 1 -- 172.21.15.16:0/2676492293 shutdown_connections 2024-03-28T22:14:44.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.031+0000 7f6b55fe2640 1 --2- 172.21.15.16:0/2676492293 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f6b380a1ef0 0x7f6b380a43b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.031+0000 7f6b55fe2640 1 --2- 172.21.15.16:0/2676492293 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6b5010bd40 0x7f6b5019c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.031+0000 7f6b55fe2640 1 --2- 172.21.15.16:0/2676492293 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b50103290 0x7f6b50198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.031+0000 7f6b55fe2640 1 --2- 172.21.15.16:0/2676492293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b50102970 0x7f6b50197ff0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.031+0000 7f6b55fe2640 1 -- 172.21.15.16:0/2676492293 >> 172.21.15.16:0/2676492293 conn(0x7f6b50077800 msgr2=0x7f6b500fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.031+0000 7f6b55fe2640 1 -- 172.21.15.16:0/2676492293 shutdown_connections 2024-03-28T22:14:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.031+0000 7f6b55fe2640 1 -- 172.21.15.16:0/2676492293 wait complete. 2024-03-28T22:14:44.047 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:14:44.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.150+0000 7fc4969ac640 1 -- 172.21.15.16:0/1540369894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc490106690 msgr2=0x7fc490106a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:44.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.150+0000 7fc4969ac640 1 --2- 172.21.15.16:0/1540369894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc490106690 0x7fc490106a70 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fc480009930 tx=0x7fc48002bc30 comp rx=0 tx=0).stop 2024-03-28T22:14:44.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.150+0000 7fc4969ac640 1 -- 172.21.15.16:0/1540369894 shutdown_connections 2024-03-28T22:14:44.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.150+0000 7fc4969ac640 1 --2- 172.21.15.16:0/1540369894 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc490100fb0 0x7fc49010d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.150+0000 7fc4969ac640 1 --2- 172.21.15.16:0/1540369894 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc4901005f0 0x7fc490100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.150+0000 7fc4969ac640 1 --2- 172.21.15.16:0/1540369894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc490106690 0x7fc490106a70 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.150+0000 7fc4969ac640 1 -- 172.21.15.16:0/1540369894 >> 172.21.15.16:0/1540369894 conn(0x7fc490077800 msgr2=0x7fc490077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:44.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.150+0000 7fc4969ac640 1 -- 172.21.15.16:0/1540369894 shutdown_connections 2024-03-28T22:14:44.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.150+0000 7fc4969ac640 1 -- 172.21.15.16:0/1540369894 wait complete. 2024-03-28T22:14:44.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.151+0000 7fc4969ac640 1 Processor -- start 2024-03-28T22:14:44.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.151+0000 7fc4969ac640 1 -- start start 2024-03-28T22:14:44.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.151+0000 7fc4969ac640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4901005f0 0x7fc490198df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.151+0000 7fc4969ac640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc490100fb0 0x7fc490199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.151+0000 7fc4969ac640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc490106690 0x7fc4901a3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.151+0000 7fc4969ac640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc4901109d0 con 0x7fc4901005f0 2024-03-28T22:14:44.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.151+0000 7fc4969ac640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc490110850 con 0x7fc490106690 2024-03-28T22:14:44.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.151+0000 7fc4969ac640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc490110b50 con 0x7fc490100fb0 2024-03-28T22:14:44.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.151+0000 7fc48ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4901005f0 0x7fc490198df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.152+0000 7fc494f49640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc490106690 0x7fc4901a3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.152+0000 7fc48ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4901005f0 0x7fc490198df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47560/0 (socket says 172.21.15.16:47560) 2024-03-28T22:14:44.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.152+0000 7fc48ffff640 1 -- 172.21.15.16:0/3840953217 learned_addr learned my addr 172.21.15.16:0/3840953217 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:44.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.152+0000 7fc48f7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc490100fb0 0x7fc490199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.152+0000 7fc494f49640 1 -- 172.21.15.16:0/3840953217 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc490100fb0 msgr2=0x7fc490199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:44.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.152+0000 7fc494f49640 1 --2- 172.21.15.16:0/3840953217 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc490100fb0 0x7fc490199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.152+0000 7fc494f49640 1 -- 172.21.15.16:0/3840953217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4901005f0 msgr2=0x7fc490198df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:44.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.152+0000 7fc494f49640 1 --2- 172.21.15.16:0/3840953217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4901005f0 0x7fc490198df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.152+0000 7fc494f49640 1 -- 172.21.15.16:0/3840953217 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4901a43b0 con 0x7fc490106690 2024-03-28T22:14:44.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.152+0000 7fc48f7fe640 1 --2- 172.21.15.16:0/3840953217 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc490100fb0 0x7fc490199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:14:44.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.152+0000 7fc494f49640 1 --2- 172.21.15.16:0/3840953217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc490106690 0x7fc4901a3e70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc488005fd0 tx=0x7fc48800ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:44.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.152+0000 7fc48d7fa640 1 -- 172.21.15.16:0/3840953217 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc48800cd80 con 0x7fc490106690 2024-03-28T22:14:44.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.152+0000 7fc4969ac640 1 -- 172.21.15.16:0/3840953217 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4901a4580 con 0x7fc490106690 2024-03-28T22:14:44.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.152+0000 7fc48d7fa640 1 -- 172.21.15.16:0/3840953217 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc488005020 con 0x7fc490106690 2024-03-28T22:14:44.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.152+0000 7fc48d7fa640 1 -- 172.21.15.16:0/3840953217 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc488010720 con 0x7fc490106690 2024-03-28T22:14:44.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.152+0000 7fc4969ac640 1 -- 172.21.15.16:0/3840953217 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4901a4ad0 con 0x7fc490106690 2024-03-28T22:14:44.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.155+0000 7fc48d7fa640 1 -- 172.21.15.16:0/3840953217 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7fc4880051d0 con 0x7fc490106690 2024-03-28T22:14:44.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.155+0000 7fc48d7fa640 1 --2- 172.21.15.16:0/3840953217 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fc4780a1ef0 0x7fc4780a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.155+0000 7fc48d7fa640 1 -- 172.21.15.16:0/3840953217 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fc4880ce6e0 con 0x7fc490106690 2024-03-28T22:14:44.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.155+0000 7fc48ffff640 1 --2- 172.21.15.16:0/3840953217 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fc4780a1ef0 0x7fc4780a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.156+0000 7fc49958e740 1 Processor -- start 2024-03-28T22:14:44.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.156+0000 7fc49958e740 1 -- start start 2024-03-28T22:14:44.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.156+0000 7fc48ffff640 1 --2- 172.21.15.16:0/3840953217 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fc4780a1ef0 0x7fc4780a43b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc480005ec0 tx=0x7fc480005dd0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:44.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.156+0000 7fc49958e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c79e88470 0x561c79e88850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.156+0000 7fc49958e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561c79e88d90 0x561c79e91430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.156+0000 7fc49958e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c79e91970 0x561c79e93d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.156+0000 7fc49958e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561c79dec1b0 con 0x561c79e88470 2024-03-28T22:14:44.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.156+0000 7fc49958e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x561c79d94c00 con 0x561c79e91970 2024-03-28T22:14:44.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.156+0000 7fc49958e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x561c79de8250 con 0x561c79e88d90 2024-03-28T22:14:44.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.156+0000 7fc48ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c79e88470 0x561c79e88850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.156+0000 7fc48ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c79e88470 0x561c79e88850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47562/0 (socket says 172.21.15.16:47562) 2024-03-28T22:14:44.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.156+0000 7fc48ffff640 1 -- 172.21.15.16:0/3431245983 learned_addr learned my addr 172.21.15.16:0/3431245983 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:44.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.156+0000 7fc494f49640 1 --2- 172.21.15.16:0/3431245983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c79e91970 0x561c79e93d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.156+0000 7fc48f7fe640 1 --2- 172.21.15.16:0/3431245983 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561c79e88d90 0x561c79e91430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.157+0000 7fc48ffff640 1 -- 172.21.15.16:0/3431245983 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561c79e88d90 msgr2=0x561c79e91430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:44.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.157+0000 7fc48ffff640 1 --2- 172.21.15.16:0/3431245983 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561c79e88d90 0x561c79e91430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.157+0000 7fc48ffff640 1 -- 172.21.15.16:0/3431245983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c79e91970 msgr2=0x561c79e93d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:44.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.157+0000 7fc48ffff640 1 --2- 172.21.15.16:0/3431245983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c79e91970 0x561c79e93d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.157+0000 7fc48ffff640 1 -- 172.21.15.16:0/3431245983 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c79db43d0 con 0x561c79e88470 2024-03-28T22:14:44.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.157+0000 7fc494f49640 1 --2- 172.21.15.16:0/3431245983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c79e91970 0x561c79e93d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:14:44.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.157+0000 7fc48ffff640 1 --2- 172.21.15.16:0/3431245983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c79e88470 0x561c79e88850 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fc48002c820 tx=0x7fc48003f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13aca7eb89d66268 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:44.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.157+0000 7fc4767fc640 1 -- 172.21.15.16:0/3431245983 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc480004400 con 0x561c79e88470 2024-03-28T22:14:44.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.157+0000 7fc4767fc640 1 -- 172.21.15.16:0/3431245983 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc480050070 con 0x561c79e88470 2024-03-28T22:14:44.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.157+0000 7fc4767fc640 1 -- 172.21.15.16:0/3431245983 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4800035f0 con 0x561c79e88470 2024-03-28T22:14:44.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.158+0000 7fc49958e740 1 -- 172.21.15.16:0/3431245983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c79e88470 msgr2=0x561c79e88850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:44.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.158+0000 7fc49958e740 1 --2- 172.21.15.16:0/3431245983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c79e88470 0x561c79e88850 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fc48002c820 tx=0x7fc48003f000 comp rx=0 tx=0).stop 2024-03-28T22:14:44.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.158+0000 7fc49958e740 1 -- 172.21.15.16:0/3431245983 shutdown_connections 2024-03-28T22:14:44.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.158+0000 7fc49958e740 1 --2- 172.21.15.16:0/3431245983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c79e91970 0x561c79e93d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.158+0000 7fc49958e740 1 --2- 172.21.15.16:0/3431245983 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561c79e88d90 0x561c79e91430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.158+0000 7fc49958e740 1 --2- 172.21.15.16:0/3431245983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c79e88470 0x561c79e88850 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.158+0000 7fc49958e740 1 -- 172.21.15.16:0/3431245983 >> 172.21.15.16:0/3431245983 conn(0x561c79e82ba0 msgr2=0x561c79e82f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:44.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.158+0000 7fc49958e740 1 -- 172.21.15.16:0/3431245983 shutdown_connections 2024-03-28T22:14:44.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.158+0000 7fc49958e740 1 -- 172.21.15.16:0/3431245983 wait complete. 2024-03-28T22:14:44.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.158+0000 7fc49958e740 1 -- start start 2024-03-28T22:14:44.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.159+0000 7fc49958e740 1 Processor -- start 2024-03-28T22:14:44.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.159+0000 7fc49958e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c79e88470 0x561c79e8e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.159+0000 7fc49958e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561c79e88d90 0x561c79e8ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.159+0000 7fc49958e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c79e91970 0x561c79e8f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.159+0000 7fc49958e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561c79e94700 con 0x561c79e88470 2024-03-28T22:14:44.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.159+0000 7fc49958e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x561c79d8a470 con 0x561c79e91970 2024-03-28T22:14:44.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.159+0000 7fc49958e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x561c79e94880 con 0x561c79e88d90 2024-03-28T22:14:44.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.159+0000 7fc49958e740 10 client.0 _ll_register_callbacks cb 0x561c79da7cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:14:44.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.159+0000 7fc48ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c79e88470 0x561c79e8e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.159+0000 7fc48ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c79e88470 0x561c79e8e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47564/0 (socket says 172.21.15.16:47564) 2024-03-28T22:14:44.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.159+0000 7fc48f7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561c79e88d90 0x561c79e8ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.159+0000 7fc48ffff640 1 -- 172.21.15.16:0/3109265642 learned_addr learned my addr 172.21.15.16:0/3109265642 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:44.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.159+0000 7fc494f49640 1 --2- 172.21.15.16:0/3109265642 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c79e91970 0x561c79e8f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.159+0000 7fc48ffff640 1 -- 172.21.15.16:0/3109265642 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561c79e88d90 msgr2=0x561c79e8ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:44.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.159+0000 7fc48ffff640 1 --2- 172.21.15.16:0/3109265642 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561c79e88d90 0x561c79e8ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.159+0000 7fc48ffff640 1 -- 172.21.15.16:0/3109265642 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c79e91970 msgr2=0x561c79e8f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:44.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.159+0000 7fc48ffff640 1 --2- 172.21.15.16:0/3109265642 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c79e91970 0x561c79e8f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.159+0000 7fc48ffff640 1 -- 172.21.15.16:0/3109265642 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c79b003f0 con 0x561c79e88470 2024-03-28T22:14:44.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.159+0000 7fc48f7fe640 1 --2- 172.21.15.16:0/3109265642 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561c79e88d90 0x561c79e8ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:14:44.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.159+0000 7fc494f49640 1 --2- 172.21.15.16:0/3109265642 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c79e91970 0x561c79e8f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:14:44.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.160+0000 7fc48ffff640 1 --2- 172.21.15.16:0/3109265642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c79e88470 0x561c79e8e9f0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fc4800096c0 tx=0x7fc4800052e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29b8fad337d32816 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:44.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.160+0000 7fc46f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:14:44.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.160+0000 7fc46f7fe640 1 -- 172.21.15.16:0/3109265642 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc480058070 con 0x561c79e88470 2024-03-28T22:14:44.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.160+0000 7fc46f7fe640 1 -- 172.21.15.16:0/3109265642 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fc480005600 con 0x561c79e88470 2024-03-28T22:14:44.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.160+0000 7fc49958e740 1 -- 172.21.15.16:0/3109265642 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc480050070 con 0x561c79e88470 2024-03-28T22:14:44.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.160+0000 7fc46f7fe640 1 -- 172.21.15.16:0/3109265642 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc480064050 con 0x561c79e88470 2024-03-28T22:14:44.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.160+0000 7fc49958e740 10 client.16374 fetch_fsmap learned FSMap version 26 2024-03-28T22:14:44.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.160+0000 7fc49958e740 1 -- 172.21.15.16:0/3109265642 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561c79bca7d0 con 0x561c79e88470 2024-03-28T22:14:44.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.160+0000 7fc46f7fe640 1 -- 172.21.15.16:0/3109265642 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc480050070 con 0x561c79e88470 2024-03-28T22:14:44.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.160+0000 7fc46f7fe640 1 -- 172.21.15.16:0/3109265642 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 26) ==== 8342+0+0 (secure 0 0 0) 0x7fc480003d10 con 0x561c79e88470 2024-03-28T22:14:44.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.160+0000 7fc49958e740 10 client.16374 fetch_fsmap finished waiting for FSMap version 26 2024-03-28T22:14:44.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.160+0000 7fc49958e740 10 client.16374 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T22:14:44.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.160+0000 7fc49958e740 20 client.16374 populate_metadata read hostname 'smithi016' 2024-03-28T22:14:44.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.160+0000 7fc49958e740 1 --2- 172.21.15.16:0/3109265642 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x561c79ea17e0 0x561c79ea3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.161+0000 7fc49958e740 4 client.16374 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:14:44.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.161+0000 7fc49958e740 1 -- 172.21.15.16:0/3109265642 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561c79df96f0 con 0x561c79ea17e0 2024-03-28T22:14:44.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.161+0000 7fc48f7fe640 1 --2- 172.21.15.16:0/3109265642 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x561c79ea17e0 0x561c79ea3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.161+0000 7fc48f7fe640 1 --2- 172.21.15.16:0/3109265642 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x561c79ea17e0 0x561c79ea3bd0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=69a57e2f396a3b32 server_cookie=c652e2c63e043b27 in_seq=0 out_seq=0 2024-03-28T22:14:44.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.161+0000 7fc46f7fe640 10 client.16374 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:14:44.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.162+0000 7fc46f7fe640 1 -- 172.21.15.16:0/3109265642 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561c79df96f0 con 0x561c79ea17e0 2024-03-28T22:14:44.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.162+0000 7fc46f7fe640 10 client.16374 handle_command_reply: tid=0 2024-03-28T22:14:44.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.162+0000 7fc49958e740 1 client.16374 shutdown 2024-03-28T22:14:44.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.162+0000 7fc49958e740 1 -- 172.21.15.16:0/3109265642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c79e88470 msgr2=0x561c79e8e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:44.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.162+0000 7fc49958e740 1 --2- 172.21.15.16:0/3109265642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c79e88470 0x561c79e8e9f0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fc4800096c0 tx=0x7fc4800052e0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.162+0000 7fc49958e740 1 -- 172.21.15.16:0/3109265642 shutdown_connections 2024-03-28T22:14:44.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.162+0000 7fc49958e740 1 --2- 172.21.15.16:0/3109265642 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x561c79ea17e0 0x561c79ea3bd0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.162+0000 7fc49958e740 1 --2- 172.21.15.16:0/3109265642 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c79e91970 0x561c79e8f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.162+0000 7fc49958e740 1 --2- 172.21.15.16:0/3109265642 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561c79e88d90 0x561c79e8ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.162+0000 7fc46f7fe640 0 client.16374 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:14:44.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.162+0000 7fc49958e740 1 --2- 172.21.15.16:0/3109265642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c79e88470 0x561c79e8e9f0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.162+0000 7fc49958e740 1 -- 172.21.15.16:0/3109265642 >> 172.21.15.16:0/3109265642 conn(0x561c79e82ba0 msgr2=0x561c79e848f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:44.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.162+0000 7fc49958e740 1 -- 172.21.15.16:0/3109265642 shutdown_connections 2024-03-28T22:14:44.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.162+0000 7fc49958e740 1 -- 172.21.15.16:0/3109265642 wait complete. 2024-03-28T22:14:44.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.162+0000 7fc49958e740 20 client.16374 trim_cache size 0 max 16384 2024-03-28T22:14:44.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.176+0000 7fc49958e740 1 Processor -- start 2024-03-28T22:14:44.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.176+0000 7fc49958e740 1 -- start start 2024-03-28T22:14:44.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.176+0000 7fc49958e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561c79e88470 0x561c79e91120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.176+0000 7fc49958e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c79e88d90 0x561c79e9d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.176+0000 7fc49958e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c79e91970 0x561c79ea91a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.176+0000 7fc49958e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc4901afca0 con 0x561c79e91970 2024-03-28T22:14:44.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.176+0000 7fc49958e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc4901ad570 con 0x561c79e88d90 2024-03-28T22:14:44.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.176+0000 7fc49958e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc4901ab3c0 con 0x561c79e88470 2024-03-28T22:14:44.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.176+0000 7fc494f49640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c79e91970 0x561c79ea91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.176+0000 7fc48f7fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c79e88d90 0x561c79e9d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.176+0000 7fc494f49640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c79e91970 0x561c79ea91a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47578/0 (socket says 172.21.15.16:47578) 2024-03-28T22:14:44.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.176+0000 7fc494f49640 1 -- 172.21.15.16:0/4083865988 learned_addr learned my addr 172.21.15.16:0/4083865988 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:44.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.176+0000 7fc48f7fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c79e88d90 0x561c79e9d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:51080/0 (socket says 172.21.15.16:51080) 2024-03-28T22:14:44.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.176+0000 7fc48ffff640 1 --2- 172.21.15.16:0/4083865988 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561c79e88470 0x561c79e91120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.177+0000 7fc48f7fe640 1 -- 172.21.15.16:0/4083865988 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561c79e88470 msgr2=0x561c79e91120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:44.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.177+0000 7fc48f7fe640 1 --2- 172.21.15.16:0/4083865988 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561c79e88470 0x561c79e91120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.177+0000 7fc48f7fe640 1 -- 172.21.15.16:0/4083865988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c79e91970 msgr2=0x561c79ea91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:44.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.177+0000 7fc48f7fe640 1 --2- 172.21.15.16:0/4083865988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c79e91970 0x561c79ea91a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.177+0000 7fc48f7fe640 1 -- 172.21.15.16:0/4083865988 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c79ea3f40 con 0x561c79e88d90 2024-03-28T22:14:44.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.177+0000 7fc494f49640 1 --2- 172.21.15.16:0/4083865988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c79e91970 0x561c79ea91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:14:44.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.177+0000 7fc48f7fe640 1 --2- 172.21.15.16:0/4083865988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c79e88d90 0x561c79e9d4a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x561c79ea4140 tx=0x7fc484005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ea2adb6b9ec53fb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:44.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.177+0000 7fc475ffb640 1 -- 172.21.15.16:0/4083865988 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc484012110 con 0x561c79e88d90 2024-03-28T22:14:44.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.177+0000 7fc475ffb640 1 -- 172.21.15.16:0/4083865988 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4840220f0 con 0x561c79e88d90 2024-03-28T22:14:44.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.177+0000 7fc475ffb640 1 -- 172.21.15.16:0/4083865988 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4840122b0 con 0x561c79e88d90 2024-03-28T22:14:44.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.178+0000 7fc49958e740 1 -- 172.21.15.16:0/4083865988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c79e88d90 msgr2=0x561c79e9d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:44.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.178+0000 7fc49958e740 1 --2- 172.21.15.16:0/4083865988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c79e88d90 0x561c79e9d4a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x561c79ea4140 tx=0x7fc484005f90 comp rx=0 tx=0).stop 2024-03-28T22:14:44.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.178+0000 7fc49958e740 1 -- 172.21.15.16:0/4083865988 shutdown_connections 2024-03-28T22:14:44.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.178+0000 7fc49958e740 1 --2- 172.21.15.16:0/4083865988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c79e91970 0x561c79ea91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.178+0000 7fc49958e740 1 --2- 172.21.15.16:0/4083865988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c79e88d90 0x561c79e9d4a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.178+0000 7fc49958e740 1 --2- 172.21.15.16:0/4083865988 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561c79e88470 0x561c79e91120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.178+0000 7fc49958e740 1 -- 172.21.15.16:0/4083865988 >> 172.21.15.16:0/4083865988 conn(0x561c79e82ba0 msgr2=0x561c79e82f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:44.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.178+0000 7fc49958e740 1 -- 172.21.15.16:0/4083865988 shutdown_connections 2024-03-28T22:14:44.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.178+0000 7fc49958e740 1 -- 172.21.15.16:0/4083865988 wait complete. 2024-03-28T22:14:44.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.178+0000 7fc49958e740 1 -- start start 2024-03-28T22:14:44.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.178+0000 7fc49958e740 1 Processor -- start 2024-03-28T22:14:44.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.179+0000 7fc49958e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c79e88470 0x561c79e99d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.179+0000 7fc49958e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561c79e88d90 0x561c79e950d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.179+0000 7fc49958e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c79e91970 0x561c79e95610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.179+0000 7fc49958e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc4901aba30 con 0x561c79e91970 2024-03-28T22:14:44.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.179+0000 7fc49958e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc4901acea0 con 0x561c79e88470 2024-03-28T22:14:44.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.179+0000 7fc49958e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc4901ac100 con 0x561c79e88d90 2024-03-28T22:14:44.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.179+0000 7fc49958e740 10 client.0 _ll_register_callbacks cb 0x7fc4880093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:14:44.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.179+0000 7fc494f49640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c79e91970 0x561c79e95610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.179+0000 7fc48f7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561c79e88d90 0x561c79e950d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.179+0000 7fc48ffff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c79e88470 0x561c79e99d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.179+0000 7fc48f7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561c79e88d90 0x561c79e950d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.16:54070/0 (socket says 172.21.15.16:54070) 2024-03-28T22:14:44.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.179+0000 7fc48f7fe640 1 -- 172.21.15.16:0/1377946663 learned_addr learned my addr 172.21.15.16:0/1377946663 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:44.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.179+0000 7fc48f7fe640 1 -- 172.21.15.16:0/1377946663 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c79e88470 msgr2=0x561c79e99d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:44.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.179+0000 7fc48f7fe640 1 --2- 172.21.15.16:0/1377946663 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c79e88470 0x561c79e99d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.179+0000 7fc48f7fe640 1 -- 172.21.15.16:0/1377946663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c79e91970 msgr2=0x561c79e95610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:44.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.179+0000 7fc48f7fe640 1 --2- 172.21.15.16:0/1377946663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c79e91970 0x561c79e95610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.179+0000 7fc48f7fe640 1 -- 172.21.15.16:0/1377946663 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c79e95de0 con 0x561c79e88d90 2024-03-28T22:14:44.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.179+0000 7fc48f7fe640 1 --2- 172.21.15.16:0/1377946663 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561c79e88d90 0x561c79e950d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc4840058c0 tx=0x7fc484012b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d96c53fb4862b19 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:44.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.179+0000 7fc46f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-03-28T22:14:44.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.179+0000 7fc46f7fe640 1 -- 172.21.15.16:0/1377946663 <== mon.1 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc48402f070 con 0x561c79e88d90 2024-03-28T22:14:44.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.180+0000 7fc46f7fe640 1 -- 172.21.15.16:0/1377946663 <== mon.1 v2:172.21.15.63:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fc4840224a0 con 0x561c79e88d90 2024-03-28T22:14:44.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.180+0000 7fc49958e740 1 -- 172.21.15.16:0/1377946663 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4840220f0 con 0x561c79e88d90 2024-03-28T22:14:44.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.180+0000 7fc46f7fe640 1 -- 172.21.15.16:0/1377946663 <== mon.1 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc48403b050 con 0x561c79e88d90 2024-03-28T22:14:44.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.180+0000 7fc49958e740 10 client.25780 fetch_fsmap learned FSMap version 26 2024-03-28T22:14:44.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.180+0000 7fc49958e740 1 -- 172.21.15.16:0/1377946663 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561c79f0bb80 con 0x561c79e88d90 2024-03-28T22:14:44.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.180+0000 7fc46f7fe640 1 -- 172.21.15.16:0/1377946663 <== mon.1 v2:172.21.15.63:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4840220f0 con 0x561c79e88d90 2024-03-28T22:14:44.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.180+0000 7fc46f7fe640 1 -- 172.21.15.16:0/1377946663 <== mon.1 v2:172.21.15.63:3300/0 5 ==== fsmap(e 26) ==== 8342+0+0 (secure 0 0 0) 0x7fc4840047e0 con 0x561c79e88d90 2024-03-28T22:14:44.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.180+0000 7fc49958e740 10 client.25780 fetch_fsmap finished waiting for FSMap version 26 2024-03-28T22:14:44.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.180+0000 7fc49958e740 10 client.25780 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T22:14:44.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.180+0000 7fc49958e740 20 client.25780 populate_metadata read hostname 'smithi016' 2024-03-28T22:14:44.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.180+0000 7fc49958e740 1 --2- 172.21.15.16:0/1377946663 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x561c79ea17e0 0x561c79f0e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.180+0000 7fc49958e740 4 client.25780 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:14:44.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.180+0000 7fc49958e740 1 -- 172.21.15.16:0/1377946663 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561c79df3a40 con 0x561c79ea17e0 2024-03-28T22:14:44.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.180+0000 7fc48ffff640 1 --2- 172.21.15.16:0/1377946663 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x561c79ea17e0 0x561c79f0e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.181+0000 7fc48ffff640 1 --2- 172.21.15.16:0/1377946663 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x561c79ea17e0 0x561c79f0e310 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8d0a1e91f293e86b server_cookie=a19a627737060e5a in_seq=0 out_seq=0 2024-03-28T22:14:44.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.181+0000 7fc46f7fe640 10 client.25780 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:14:44.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.181+0000 7fc46f7fe640 1 -- 172.21.15.16:0/1377946663 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561c79df3a40 con 0x561c79ea17e0 2024-03-28T22:14:44.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.181+0000 7fc46f7fe640 10 client.25780 handle_command_reply: tid=0 2024-03-28T22:14:44.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.181+0000 7fc49958e740 1 client.25780 shutdown 2024-03-28T22:14:44.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.181+0000 7fc49958e740 1 -- 172.21.15.16:0/1377946663 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561c79e88d90 msgr2=0x561c79e950d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:44.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.181+0000 7fc49958e740 1 --2- 172.21.15.16:0/1377946663 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561c79e88d90 0x561c79e950d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc4840058c0 tx=0x7fc484012b80 comp rx=0 tx=0).stop 2024-03-28T22:14:44.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.181+0000 7fc49958e740 1 -- 172.21.15.16:0/1377946663 shutdown_connections 2024-03-28T22:14:44.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.181+0000 7fc49958e740 1 --2- 172.21.15.16:0/1377946663 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x561c79ea17e0 0x561c79f0e310 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.181+0000 7fc49958e740 1 --2- 172.21.15.16:0/1377946663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c79e91970 0x561c79e95610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.181+0000 7fc46f7fe640 0 client.25780 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:14:44.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.181+0000 7fc49958e740 1 --2- 172.21.15.16:0/1377946663 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561c79e88d90 0x561c79e950d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.181+0000 7fc49958e740 1 --2- 172.21.15.16:0/1377946663 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c79e88470 0x561c79e99d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.181+0000 7fc49958e740 1 -- 172.21.15.16:0/1377946663 >> 172.21.15.16:0/1377946663 conn(0x561c79e82ba0 msgr2=0x561c79e814b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:44.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.181+0000 7fc49958e740 1 -- 172.21.15.16:0/1377946663 shutdown_connections 2024-03-28T22:14:44.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.181+0000 7fc49958e740 1 -- 172.21.15.16:0/1377946663 wait complete. 2024-03-28T22:14:44.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.181+0000 7fc49958e740 20 client.25780 trim_cache size 0 max 16384 2024-03-28T22:14:44.194 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:14:44.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.182+0000 7fc4969ac640 1 -- 172.21.15.16:0/3840953217 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fc4780a1ef0 msgr2=0x7fc4780a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:44.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.182+0000 7fc4969ac640 1 --2- 172.21.15.16:0/3840953217 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fc4780a1ef0 0x7fc4780a43b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc480005ec0 tx=0x7fc480005dd0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.182+0000 7fc4969ac640 1 -- 172.21.15.16:0/3840953217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc490106690 msgr2=0x7fc4901a3e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:44.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.182+0000 7fc4969ac640 1 --2- 172.21.15.16:0/3840953217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc490106690 0x7fc4901a3e70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc488005fd0 tx=0x7fc48800ee60 comp rx=0 tx=0).stop 2024-03-28T22:14:44.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.182+0000 7fc4969ac640 1 -- 172.21.15.16:0/3840953217 shutdown_connections 2024-03-28T22:14:44.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.182+0000 7fc4969ac640 1 --2- 172.21.15.16:0/3840953217 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fc4780a1ef0 0x7fc4780a43b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.182+0000 7fc4969ac640 1 --2- 172.21.15.16:0/3840953217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc490106690 0x7fc4901a3e70 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.182+0000 7fc4969ac640 1 --2- 172.21.15.16:0/3840953217 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc490100fb0 0x7fc490199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.182+0000 7fc4969ac640 1 --2- 172.21.15.16:0/3840953217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4901005f0 0x7fc490198df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.182+0000 7fc4969ac640 1 -- 172.21.15.16:0/3840953217 >> 172.21.15.16:0/3840953217 conn(0x7fc490077800 msgr2=0x7fc4900ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:44.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.182+0000 7fc4969ac640 1 -- 172.21.15.16:0/3840953217 shutdown_connections 2024-03-28T22:14:44.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.182+0000 7fc4969ac640 1 -- 172.21.15.16:0/3840953217 wait complete. 2024-03-28T22:14:44.197 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:44 smithi016 ceph-mon[32374]: pgmap v127: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:14:44.197 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:44 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:14:44.197 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:44 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/721300580' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:14:44.198 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:14:44.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.303+0000 7fe123064640 1 -- 172.21.15.16:0/712783527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe11c1066a0 msgr2=0x7fe11c106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:44.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.303+0000 7fe123064640 1 --2- 172.21.15.16:0/712783527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe11c1066a0 0x7fe11c106a80 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fe118009930 tx=0x7fe11802bc30 comp rx=0 tx=0).stop 2024-03-28T22:14:44.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.303+0000 7fe123064640 1 -- 172.21.15.16:0/712783527 shutdown_connections 2024-03-28T22:14:44.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.303+0000 7fe123064640 1 --2- 172.21.15.16:0/712783527 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe11c100fc0 0x7fe11c10d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.303+0000 7fe123064640 1 --2- 172.21.15.16:0/712783527 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe11c100600 0x7fe11c100a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.303+0000 7fe123064640 1 --2- 172.21.15.16:0/712783527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe11c1066a0 0x7fe11c106a80 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.303+0000 7fe123064640 1 -- 172.21.15.16:0/712783527 >> 172.21.15.16:0/712783527 conn(0x7fe11c077770 msgr2=0x7fe11c077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:44.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.303+0000 7fe123064640 1 -- 172.21.15.16:0/712783527 shutdown_connections 2024-03-28T22:14:44.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.303+0000 7fe123064640 1 -- 172.21.15.16:0/712783527 wait complete. 2024-03-28T22:14:44.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.304+0000 7fe123064640 1 Processor -- start 2024-03-28T22:14:44.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.304+0000 7fe123064640 1 -- start start 2024-03-28T22:14:44.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.304+0000 7fe123064640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe11c100600 0x7fe11c10df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.304+0000 7fe123064640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe11c100fc0 0x7fe11c10e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.304+0000 7fe123064640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe11c1066a0 0x7fe11c10e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.304+0000 7fe123064640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe11c111b70 con 0x7fe11c100fc0 2024-03-28T22:14:44.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.304+0000 7fe123064640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe11c1119f0 con 0x7fe11c100600 2024-03-28T22:14:44.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.304+0000 7fe123064640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe11c111cf0 con 0x7fe11c1066a0 2024-03-28T22:14:44.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.304+0000 7fe113fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe11c100fc0 0x7fe11c10e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.304+0000 7fe113fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe11c100fc0 0x7fe11c10e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47610/0 (socket says 172.21.15.16:47610) 2024-03-28T22:14:44.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.304+0000 7fe113fff640 1 -- 172.21.15.16:0/1757805228 learned_addr learned my addr 172.21.15.16:0/1757805228 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:44.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.304+0000 7fe120e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe11c100600 0x7fe11c10df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.304+0000 7fe121601640 1 --2- 172.21.15.16:0/1757805228 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe11c1066a0 0x7fe11c10e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.304+0000 7fe113fff640 1 -- 172.21.15.16:0/1757805228 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe11c1066a0 msgr2=0x7fe11c10e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:44.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.304+0000 7fe113fff640 1 --2- 172.21.15.16:0/1757805228 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe11c1066a0 0x7fe11c10e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.304+0000 7fe113fff640 1 -- 172.21.15.16:0/1757805228 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe11c100600 msgr2=0x7fe11c10df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:44.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.304+0000 7fe113fff640 1 --2- 172.21.15.16:0/1757805228 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe11c100600 0x7fe11c10df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.304+0000 7fe113fff640 1 -- 172.21.15.16:0/1757805228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe11c1a64b0 con 0x7fe11c100fc0 2024-03-28T22:14:44.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.304+0000 7fe120e00640 1 --2- 172.21.15.16:0/1757805228 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe11c100600 0x7fe11c10df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:14:44.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.304+0000 7fe113fff640 1 --2- 172.21.15.16:0/1757805228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe11c100fc0 0x7fe11c10e4a0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fe10c00e9f0 tx=0x7fe10c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:44.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.305+0000 7fe111ffb640 1 -- 172.21.15.16:0/1757805228 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe10c00cd80 con 0x7fe11c100fc0 2024-03-28T22:14:44.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.305+0000 7fe111ffb640 1 -- 172.21.15.16:0/1757805228 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe10c010910 con 0x7fe11c100fc0 2024-03-28T22:14:44.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.305+0000 7fe123064640 1 -- 172.21.15.16:0/1757805228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe11c1a67a0 con 0x7fe11c100fc0 2024-03-28T22:14:44.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.305+0000 7fe111ffb640 1 -- 172.21.15.16:0/1757805228 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe10c018990 con 0x7fe11c100fc0 2024-03-28T22:14:44.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.305+0000 7fe123064640 1 -- 172.21.15.16:0/1757805228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe11c1a6cf0 con 0x7fe11c100fc0 2024-03-28T22:14:44.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.307+0000 7fe111ffb640 1 -- 172.21.15.16:0/1757805228 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7fe10c010ac0 con 0x7fe11c100fc0 2024-03-28T22:14:44.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.307+0000 7fe125c44740 1 Processor -- start 2024-03-28T22:14:44.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.307+0000 7fe125c44740 1 -- start start 2024-03-28T22:14:44.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.308+0000 7fe125c44740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d5f0f26470 0x55d5f0f26850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.308+0000 7fe111ffb640 1 --2- 172.21.15.16:0/1757805228 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fe1040a1ef0 0x7fe1040a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.308+0000 7fe125c44740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5f0f26d90 0x55d5f0f2f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.308+0000 7fe125c44740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5f0f2f970 0x55d5f0f31d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.308+0000 7fe125c44740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d5f0e8a1b0 con 0x55d5f0f2f970 2024-03-28T22:14:44.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.308+0000 7fe125c44740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55d5f0e32c00 con 0x55d5f0f26470 2024-03-28T22:14:44.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.308+0000 7fe125c44740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d5f0e86250 con 0x55d5f0f26d90 2024-03-28T22:14:44.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.308+0000 7fe111ffb640 1 -- 172.21.15.16:0/1757805228 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fe10c0ce560 con 0x7fe11c100fc0 2024-03-28T22:14:44.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.308+0000 7fe120e00640 1 --2- 172.21.15.16:0/1757805228 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fe1040a1ef0 0x7fe1040a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.308+0000 7fe113fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5f0f26d90 0x55d5f0f2f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.308+0000 7fe121601640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d5f0f26470 0x55d5f0f26850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.308+0000 7fe113fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5f0f26d90 0x55d5f0f2f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.16:54108/0 (socket says 172.21.15.16:54108) 2024-03-28T22:14:44.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.308+0000 7fe113fff640 1 -- 172.21.15.16:0/1765722285 learned_addr learned my addr 172.21.15.16:0/1765722285 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:44.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.308+0000 7fe121601640 1 --2- 172.21.15.16:0/1765722285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5f0f2f970 0x55d5f0f31d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.308+0000 7fe113fff640 1 -- 172.21.15.16:0/1765722285 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d5f0f26470 msgr2=0x55d5f0f26850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:44.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.308+0000 7fe113fff640 1 --2- 172.21.15.16:0/1765722285 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d5f0f26470 0x55d5f0f26850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.308+0000 7fe113fff640 1 -- 172.21.15.16:0/1765722285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5f0f2f970 msgr2=0x55d5f0f31d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:44.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.308+0000 7fe113fff640 1 --2- 172.21.15.16:0/1765722285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5f0f2f970 0x55d5f0f31d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.308+0000 7fe113fff640 1 -- 172.21.15.16:0/1765722285 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5f0e523d0 con 0x55d5f0f26d90 2024-03-28T22:14:44.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.308+0000 7fe120e00640 1 --2- 172.21.15.16:0/1757805228 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fe1040a1ef0 0x7fe1040a43b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe11802c170 tx=0x7fe11802df90 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:44.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.308+0000 7fe113fff640 1 --2- 172.21.15.16:0/1765722285 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5f0f26d90 0x55d5f0f2f430 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe10c0973d0 tx=0x7fe10c09b670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=565b7115c14e518a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:44.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.309+0000 7fe102ffd640 1 -- 172.21.15.16:0/1765722285 <== mon.1 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe10c09af70 con 0x55d5f0f26d90 2024-03-28T22:14:44.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.309+0000 7fe102ffd640 1 -- 172.21.15.16:0/1765722285 <== mon.1 v2:172.21.15.63:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe10c09b900 con 0x55d5f0f26d90 2024-03-28T22:14:44.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.309+0000 7fe102ffd640 1 -- 172.21.15.16:0/1765722285 <== mon.1 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe10c0975e0 con 0x55d5f0f26d90 2024-03-28T22:14:44.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.309+0000 7fe125c44740 1 -- 172.21.15.16:0/1765722285 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5f0f26d90 msgr2=0x55d5f0f2f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:44.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.309+0000 7fe125c44740 1 --2- 172.21.15.16:0/1765722285 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5f0f26d90 0x55d5f0f2f430 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe10c0973d0 tx=0x7fe10c09b670 comp rx=0 tx=0).stop 2024-03-28T22:14:44.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.309+0000 7fe125c44740 1 -- 172.21.15.16:0/1765722285 shutdown_connections 2024-03-28T22:14:44.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.309+0000 7fe125c44740 1 --2- 172.21.15.16:0/1765722285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5f0f2f970 0x55d5f0f31d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.309+0000 7fe125c44740 1 --2- 172.21.15.16:0/1765722285 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5f0f26d90 0x55d5f0f2f430 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.309+0000 7fe125c44740 1 --2- 172.21.15.16:0/1765722285 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d5f0f26470 0x55d5f0f26850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.309+0000 7fe125c44740 1 -- 172.21.15.16:0/1765722285 >> 172.21.15.16:0/1765722285 conn(0x55d5f0f20ba0 msgr2=0x55d5f0f20f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:44.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.309+0000 7fe125c44740 1 -- 172.21.15.16:0/1765722285 shutdown_connections 2024-03-28T22:14:44.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.309+0000 7fe125c44740 1 -- 172.21.15.16:0/1765722285 wait complete. 2024-03-28T22:14:44.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.310+0000 7fe125c44740 1 -- start start 2024-03-28T22:14:44.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.310+0000 7fe125c44740 1 Processor -- start 2024-03-28T22:14:44.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.310+0000 7fe125c44740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5f0f26470 0x55d5f0f2ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.310+0000 7fe125c44740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d5f0f26d90 0x55d5f0f2cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.310+0000 7fe125c44740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5f0f2f970 0x55d5f0f2d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.310+0000 7fe125c44740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d5f0f32760 con 0x55d5f0f2f970 2024-03-28T22:14:44.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.310+0000 7fe125c44740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55d5f0e28470 con 0x55d5f0f26d90 2024-03-28T22:14:44.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.310+0000 7fe125c44740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d5f0f328e0 con 0x55d5f0f26470 2024-03-28T22:14:44.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.310+0000 7fe125c44740 10 client.0 _ll_register_callbacks cb 0x55d5f0e45cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:14:44.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.310+0000 7fe121601640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5f0f2f970 0x55d5f0f2d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.310+0000 7fe121601640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5f0f2f970 0x55d5f0f2d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47636/0 (socket says 172.21.15.16:47636) 2024-03-28T22:14:44.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.310+0000 7fe120e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5f0f26470 0x55d5f0f2ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.310+0000 7fe121601640 1 -- 172.21.15.16:0/3812079898 learned_addr learned my addr 172.21.15.16:0/3812079898 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:44.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.310+0000 7fe113fff640 1 --2- 172.21.15.16:0/3812079898 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d5f0f26d90 0x55d5f0f2cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.311+0000 7fe121601640 1 -- 172.21.15.16:0/3812079898 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5f0f26470 msgr2=0x55d5f0f2ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:44.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.311+0000 7fe121601640 1 --2- 172.21.15.16:0/3812079898 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5f0f26470 0x55d5f0f2ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.311+0000 7fe121601640 1 -- 172.21.15.16:0/3812079898 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d5f0f26d90 msgr2=0x55d5f0f2cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:44.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.311+0000 7fe121601640 1 --2- 172.21.15.16:0/3812079898 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d5f0f26d90 0x55d5f0f2cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.311+0000 7fe121601640 1 -- 172.21.15.16:0/3812079898 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5f0b9e3f0 con 0x55d5f0f2f970 2024-03-28T22:14:44.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.311+0000 7fe121601640 1 --2- 172.21.15.16:0/3812079898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5f0f2f970 0x55d5f0f2d490 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fe114003bb0 tx=0x7fe114009d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9af4b92605b608 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:44.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.311+0000 7fe0fbfff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:14:44.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.311+0000 7fe0fbfff640 1 -- 172.21.15.16:0/3812079898 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe114014b40 con 0x55d5f0f2f970 2024-03-28T22:14:44.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.311+0000 7fe0fbfff640 1 -- 172.21.15.16:0/3812079898 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fe114014ce0 con 0x55d5f0f2f970 2024-03-28T22:14:44.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.311+0000 7fe125c44740 1 -- 172.21.15.16:0/3812079898 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe10c09b900 con 0x55d5f0f2f970 2024-03-28T22:14:44.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.311+0000 7fe0fbfff640 1 -- 172.21.15.16:0/3812079898 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe114026050 con 0x55d5f0f2f970 2024-03-28T22:14:44.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.311+0000 7fe125c44740 10 client.16392 fetch_fsmap learned FSMap version 26 2024-03-28T22:14:44.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.311+0000 7fe125c44740 1 -- 172.21.15.16:0/3812079898 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d5f0c687d0 con 0x55d5f0f2f970 2024-03-28T22:14:44.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.311+0000 7fe0fbfff640 1 -- 172.21.15.16:0/3812079898 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe10c09b900 con 0x55d5f0f2f970 2024-03-28T22:14:44.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.311+0000 7fe0fbfff640 1 -- 172.21.15.16:0/3812079898 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 26) ==== 8342+0+0 (secure 0 0 0) 0x7fe114012450 con 0x55d5f0f2f970 2024-03-28T22:14:44.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.312+0000 7fe125c44740 10 client.16392 fetch_fsmap finished waiting for FSMap version 26 2024-03-28T22:14:44.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.312+0000 7fe125c44740 10 client.16392 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:14:44.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.312+0000 7fe125c44740 20 client.16392 populate_metadata read hostname 'smithi016' 2024-03-28T22:14:44.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.312+0000 7fe125c44740 1 --2- 172.21.15.16:0/3812079898 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d5f0f3f9c0 0x55d5f0f41e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.312+0000 7fe125c44740 4 client.16392 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:14:44.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.312+0000 7fe125c44740 1 -- 172.21.15.16:0/3812079898 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d5f0e976f0 con 0x55d5f0f3f9c0 2024-03-28T22:14:44.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.312+0000 7fe120e00640 1 --2- 172.21.15.16:0/3812079898 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d5f0f3f9c0 0x55d5f0f41e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.312+0000 7fe0fbfff640 10 client.16392 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:14:44.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.312+0000 7fe120e00640 1 --2- 172.21.15.16:0/3812079898 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d5f0f3f9c0 0x55d5f0f41e00 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62416da7e20815f2 server_cookie=c67e27796eadf1b7 in_seq=0 out_seq=0 2024-03-28T22:14:44.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.313+0000 7fe0fbfff640 1 -- 172.21.15.16:0/3812079898 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d5f0e976f0 con 0x55d5f0f3f9c0 2024-03-28T22:14:44.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.313+0000 7fe0fbfff640 10 client.16392 handle_command_reply: tid=0 2024-03-28T22:14:44.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.313+0000 7fe125c44740 1 client.16392 shutdown 2024-03-28T22:14:44.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.313+0000 7fe125c44740 1 -- 172.21.15.16:0/3812079898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5f0f2f970 msgr2=0x55d5f0f2d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:44.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.313+0000 7fe125c44740 1 --2- 172.21.15.16:0/3812079898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5f0f2f970 0x55d5f0f2d490 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fe114003bb0 tx=0x7fe114009d50 comp rx=0 tx=0).stop 2024-03-28T22:14:44.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.313+0000 7fe125c44740 1 -- 172.21.15.16:0/3812079898 shutdown_connections 2024-03-28T22:14:44.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.313+0000 7fe125c44740 1 --2- 172.21.15.16:0/3812079898 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d5f0f3f9c0 0x55d5f0f41e00 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.313+0000 7fe125c44740 1 --2- 172.21.15.16:0/3812079898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5f0f2f970 0x55d5f0f2d490 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.313+0000 7fe0fbfff640 0 client.16392 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:14:44.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.313+0000 7fe125c44740 1 --2- 172.21.15.16:0/3812079898 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d5f0f26d90 0x55d5f0f2cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.313+0000 7fe125c44740 1 --2- 172.21.15.16:0/3812079898 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5f0f26470 0x55d5f0f2ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.313+0000 7fe125c44740 1 -- 172.21.15.16:0/3812079898 >> 172.21.15.16:0/3812079898 conn(0x55d5f0f20ba0 msgr2=0x55d5f0f228f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:44.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.314+0000 7fe125c44740 1 -- 172.21.15.16:0/3812079898 shutdown_connections 2024-03-28T22:14:44.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.314+0000 7fe125c44740 1 -- 172.21.15.16:0/3812079898 wait complete. 2024-03-28T22:14:44.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.314+0000 7fe125c44740 20 client.16392 trim_cache size 0 max 16384 2024-03-28T22:14:44.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.328+0000 7fe125c44740 1 Processor -- start 2024-03-28T22:14:44.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.328+0000 7fe125c44740 1 -- start start 2024-03-28T22:14:44.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.328+0000 7fe125c44740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d5f0f26470 0x55d5f0f2f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.328+0000 7fe125c44740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5f0f26d90 0x55d5f0f3b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.328+0000 7fe125c44740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5f0f2f970 0x55d5f0f476d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.328+0000 7fe125c44740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe11c1b1f30 con 0x55d5f0f2f970 2024-03-28T22:14:44.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.328+0000 7fe125c44740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe11c1af800 con 0x55d5f0f26470 2024-03-28T22:14:44.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.328+0000 7fe125c44740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe11c1ad650 con 0x55d5f0f26d90 2024-03-28T22:14:44.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.328+0000 7fe121601640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5f0f2f970 0x55d5f0f476d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.328+0000 7fe121601640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5f0f2f970 0x55d5f0f476d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47644/0 (socket says 172.21.15.16:47644) 2024-03-28T22:14:44.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.328+0000 7fe121601640 1 -- 172.21.15.16:0/2732162154 learned_addr learned my addr 172.21.15.16:0/2732162154 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:44.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.328+0000 7fe120e00640 1 --2- 172.21.15.16:0/2732162154 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d5f0f26470 0x55d5f0f2f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.328+0000 7fe121601640 1 -- 172.21.15.16:0/2732162154 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5f0f26d90 msgr2=0x55d5f0f3b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:44.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.328+0000 7fe113fff640 1 --2- 172.21.15.16:0/2732162154 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5f0f26d90 0x55d5f0f3b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.328+0000 7fe121601640 1 --2- 172.21.15.16:0/2732162154 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5f0f26d90 0x55d5f0f3b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.328+0000 7fe121601640 1 -- 172.21.15.16:0/2732162154 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d5f0f26470 msgr2=0x55d5f0f2f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:44.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.328+0000 7fe121601640 1 --2- 172.21.15.16:0/2732162154 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d5f0f26470 0x55d5f0f2f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.328+0000 7fe121601640 1 -- 172.21.15.16:0/2732162154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5f0f26bc0 con 0x55d5f0f2f970 2024-03-28T22:14:44.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.328+0000 7fe113fff640 1 --2- 172.21.15.16:0/2732162154 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5f0f26d90 0x55d5f0f3b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:14:44.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.328+0000 7fe121601640 1 --2- 172.21.15.16:0/2732162154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5f0f2f970 0x55d5f0f476d0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fe114003b80 tx=0x7fe1140095a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b59d70380ff1dfab server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:44.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.329+0000 7fe1027fc640 1 -- 172.21.15.16:0/2732162154 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe11402e020 con 0x55d5f0f2f970 2024-03-28T22:14:44.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.329+0000 7fe1027fc640 1 -- 172.21.15.16:0/2732162154 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1140099b0 con 0x55d5f0f2f970 2024-03-28T22:14:44.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.329+0000 7fe1027fc640 1 -- 172.21.15.16:0/2732162154 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1140048a0 con 0x55d5f0f2f970 2024-03-28T22:14:44.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.329+0000 7fe125c44740 1 -- 172.21.15.16:0/2732162154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5f0f2f970 msgr2=0x55d5f0f476d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:44.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.329+0000 7fe125c44740 1 --2- 172.21.15.16:0/2732162154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5f0f2f970 0x55d5f0f476d0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fe114003b80 tx=0x7fe1140095a0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.329+0000 7fe125c44740 1 -- 172.21.15.16:0/2732162154 shutdown_connections 2024-03-28T22:14:44.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.329+0000 7fe125c44740 1 --2- 172.21.15.16:0/2732162154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5f0f2f970 0x55d5f0f476d0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.329+0000 7fe125c44740 1 --2- 172.21.15.16:0/2732162154 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5f0f26d90 0x55d5f0f3b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.329+0000 7fe125c44740 1 --2- 172.21.15.16:0/2732162154 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d5f0f26470 0x55d5f0f2f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.329+0000 7fe125c44740 1 -- 172.21.15.16:0/2732162154 >> 172.21.15.16:0/2732162154 conn(0x55d5f0f20ba0 msgr2=0x55d5f0f1fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:44.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.329+0000 7fe125c44740 1 -- 172.21.15.16:0/2732162154 shutdown_connections 2024-03-28T22:14:44.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.329+0000 7fe125c44740 1 -- 172.21.15.16:0/2732162154 wait complete. 2024-03-28T22:14:44.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.330+0000 7fe125c44740 1 -- start start 2024-03-28T22:14:44.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.330+0000 7fe125c44740 1 Processor -- start 2024-03-28T22:14:44.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.330+0000 7fe125c44740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5f0f26470 0x55d5f0f29dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.330+0000 7fe125c44740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d5f0f26d90 0x55d5f0f2a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.330+0000 7fe125c44740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5f0f2f970 0x55d5f0fa4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.330+0000 7fe125c44740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe11c1adcc0 con 0x55d5f0f2f970 2024-03-28T22:14:44.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.330+0000 7fe125c44740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe11c1af130 con 0x55d5f0f26d90 2024-03-28T22:14:44.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.330+0000 7fe125c44740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe11c1ae390 con 0x55d5f0f26470 2024-03-28T22:14:44.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.330+0000 7fe125c44740 10 client.0 _ll_register_callbacks cb 0x7fe114014350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:14:44.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.330+0000 7fe113fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d5f0f26d90 0x55d5f0f2a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.330+0000 7fe120e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5f0f26470 0x55d5f0f29dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.330+0000 7fe121601640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5f0f2f970 0x55d5f0fa4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.330+0000 7fe113fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d5f0f26d90 0x55d5f0f2a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:51152/0 (socket says 172.21.15.16:51152) 2024-03-28T22:14:44.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.330+0000 7fe113fff640 1 -- 172.21.15.16:0/324345937 learned_addr learned my addr 172.21.15.16:0/324345937 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:44.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.331+0000 7fe113fff640 1 -- 172.21.15.16:0/324345937 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5f0f26470 msgr2=0x55d5f0f29dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:44.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.331+0000 7fe113fff640 1 --2- 172.21.15.16:0/324345937 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5f0f26470 0x55d5f0f29dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.331+0000 7fe113fff640 1 -- 172.21.15.16:0/324345937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5f0f2f970 msgr2=0x55d5f0fa4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:44.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.331+0000 7fe113fff640 1 --2- 172.21.15.16:0/324345937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5f0f2f970 0x55d5f0fa4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.331+0000 7fe113fff640 1 -- 172.21.15.16:0/324345937 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5f0fa5310 con 0x55d5f0f26d90 2024-03-28T22:14:44.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.331+0000 7fe121601640 1 --2- 172.21.15.16:0/324345937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5f0f2f970 0x55d5f0fa4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:14:44.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.331+0000 7fe120e00640 1 --2- 172.21.15.16:0/324345937 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5f0f26470 0x55d5f0f29dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:14:44.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.331+0000 7fe113fff640 1 --2- 172.21.15.16:0/324345937 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d5f0f26d90 0x55d5f0f2a310 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe10c000c00 tx=0x7fe10c0a4750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=882382bb862c13e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:44.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.331+0000 7fe0fbfff640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-03-28T22:14:44.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.331+0000 7fe0fbfff640 1 -- 172.21.15.16:0/324345937 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe10c0ae020 con 0x55d5f0f26d90 2024-03-28T22:14:44.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.331+0000 7fe125c44740 1 -- 172.21.15.16:0/324345937 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1140099b0 con 0x55d5f0f26d90 2024-03-28T22:14:44.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.331+0000 7fe0fbfff640 1 -- 172.21.15.16:0/324345937 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fe10c09c4b0 con 0x55d5f0f26d90 2024-03-28T22:14:44.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.331+0000 7fe0fbfff640 1 -- 172.21.15.16:0/324345937 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe10c09af70 con 0x55d5f0f26d90 2024-03-28T22:14:44.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.331+0000 7fe125c44740 10 client.26300 fetch_fsmap learned FSMap version 26 2024-03-28T22:14:44.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.331+0000 7fe125c44740 1 -- 172.21.15.16:0/324345937 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d5f0fa6580 con 0x55d5f0f26d90 2024-03-28T22:14:44.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.331+0000 7fe0fbfff640 1 -- 172.21.15.16:0/324345937 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1140099b0 con 0x55d5f0f26d90 2024-03-28T22:14:44.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.332+0000 7fe0fbfff640 1 -- 172.21.15.16:0/324345937 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 26) ==== 8342+0+0 (secure 0 0 0) 0x7fe10c018b30 con 0x55d5f0f26d90 2024-03-28T22:14:44.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.332+0000 7fe125c44740 10 client.26300 fetch_fsmap finished waiting for FSMap version 26 2024-03-28T22:14:44.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.332+0000 7fe125c44740 10 client.26300 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:14:44.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.332+0000 7fe125c44740 20 client.26300 populate_metadata read hostname 'smithi016' 2024-03-28T22:14:44.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.332+0000 7fe125c44740 1 --2- 172.21.15.16:0/324345937 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d5f0fa6970 0x55d5f0fa8d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:44.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.332+0000 7fe125c44740 4 client.26300 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:14:44.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.332+0000 7fe125c44740 1 -- 172.21.15.16:0/324345937 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d5f0e91a40 con 0x55d5f0fa6970 2024-03-28T22:14:44.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.332+0000 7fe120e00640 1 --2- 172.21.15.16:0/324345937 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d5f0fa6970 0x55d5f0fa8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:44.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.332+0000 7fe0fbfff640 10 client.26300 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:14:44.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.332+0000 7fe120e00640 1 --2- 172.21.15.16:0/324345937 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d5f0fa6970 0x55d5f0fa8d60 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=315e8ba926cc8fca server_cookie=cd42198fa8333503 in_seq=0 out_seq=0 2024-03-28T22:14:44.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.332+0000 7fe0fbfff640 1 -- 172.21.15.16:0/324345937 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d5f0e91a40 con 0x55d5f0fa6970 2024-03-28T22:14:44.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.332+0000 7fe0fbfff640 10 client.26300 handle_command_reply: tid=0 2024-03-28T22:14:44.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.332+0000 7fe125c44740 1 client.26300 shutdown 2024-03-28T22:14:44.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.333+0000 7fe125c44740 1 -- 172.21.15.16:0/324345937 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d5f0f26d90 msgr2=0x55d5f0f2a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:44.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.333+0000 7fe125c44740 1 --2- 172.21.15.16:0/324345937 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d5f0f26d90 0x55d5f0f2a310 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe10c000c00 tx=0x7fe10c0a4750 comp rx=0 tx=0).stop 2024-03-28T22:14:44.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.333+0000 7fe125c44740 1 -- 172.21.15.16:0/324345937 shutdown_connections 2024-03-28T22:14:44.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.333+0000 7fe125c44740 1 --2- 172.21.15.16:0/324345937 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d5f0fa6970 0x55d5f0fa8d60 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.333+0000 7fe125c44740 1 --2- 172.21.15.16:0/324345937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5f0f2f970 0x55d5f0fa4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.333+0000 7fe125c44740 1 --2- 172.21.15.16:0/324345937 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d5f0f26d90 0x55d5f0f2a310 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.333+0000 7fe0fbfff640 0 client.26300 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:14:44.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.333+0000 7fe125c44740 1 --2- 172.21.15.16:0/324345937 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5f0f26470 0x55d5f0f29dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.333+0000 7fe125c44740 1 -- 172.21.15.16:0/324345937 >> 172.21.15.16:0/324345937 conn(0x55d5f0f20ba0 msgr2=0x55d5f0f2c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:44.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.333+0000 7fe125c44740 1 -- 172.21.15.16:0/324345937 shutdown_connections 2024-03-28T22:14:44.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.333+0000 7fe125c44740 1 -- 172.21.15.16:0/324345937 wait complete. 2024-03-28T22:14:44.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.333+0000 7fe125c44740 20 client.26300 trim_cache size 0 max 16384 2024-03-28T22:14:44.347 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:14:44.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.333+0000 7fe123064640 1 -- 172.21.15.16:0/1757805228 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fe1040a1ef0 msgr2=0x7fe1040a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:44.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.333+0000 7fe123064640 1 --2- 172.21.15.16:0/1757805228 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fe1040a1ef0 0x7fe1040a43b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe11802c170 tx=0x7fe11802df90 comp rx=0 tx=0).stop 2024-03-28T22:14:44.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.333+0000 7fe123064640 1 -- 172.21.15.16:0/1757805228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe11c100fc0 msgr2=0x7fe11c10e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:44.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.333+0000 7fe123064640 1 --2- 172.21.15.16:0/1757805228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe11c100fc0 0x7fe11c10e4a0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fe10c00e9f0 tx=0x7fe10c00eeb0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.333+0000 7fe123064640 1 -- 172.21.15.16:0/1757805228 shutdown_connections 2024-03-28T22:14:44.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.333+0000 7fe123064640 1 --2- 172.21.15.16:0/1757805228 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fe1040a1ef0 0x7fe1040a43b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.333+0000 7fe123064640 1 --2- 172.21.15.16:0/1757805228 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe11c1066a0 0x7fe11c10e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.333+0000 7fe123064640 1 --2- 172.21.15.16:0/1757805228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe11c100fc0 0x7fe11c10e4a0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.333+0000 7fe123064640 1 --2- 172.21.15.16:0/1757805228 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe11c100600 0x7fe11c10df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:44.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.334+0000 7fe123064640 1 -- 172.21.15.16:0/1757805228 >> 172.21.15.16:0/1757805228 conn(0x7fe11c077770 msgr2=0x7fe11c0ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:44.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.334+0000 7fe123064640 1 -- 172.21.15.16:0/1757805228 shutdown_connections 2024-03-28T22:14:44.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:44.334+0000 7fe123064640 1 -- 172.21.15.16:0/1757805228 wait complete. 2024-03-28T22:14:44.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:44 smithi063 ceph-mon[36249]: pgmap v127: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:14:44.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:44 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:14:44.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:44 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/721300580' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:14:44.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:44 smithi053 ceph-mon[35979]: pgmap v127: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:14:44.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:44 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:14:44.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:44 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/721300580' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:14:45.351 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:14:45.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:14:45.351 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:14:45.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:45 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:14:45.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:45 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:14:45.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:45 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:14:45.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:45 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:14:45.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:45 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:14:45.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:45 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:14:45.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.469+0000 7f89bad72640 1 -- 172.21.15.16:0/3711157910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89b4100fb0 msgr2=0x7f89b410d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:45.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.469+0000 7f89bad72640 1 --2- 172.21.15.16:0/3711157910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89b4100fb0 0x7f89b410d4c0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f89ac009cd0 tx=0x7f89ac02c0f0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.469+0000 7f89bad72640 1 -- 172.21.15.16:0/3711157910 shutdown_connections 2024-03-28T22:14:45.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.469+0000 7f89bad72640 1 --2- 172.21.15.16:0/3711157910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89b4100fb0 0x7f89b410d4c0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.469+0000 7f89bad72640 1 --2- 172.21.15.16:0/3711157910 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89b41005f0 0x7f89b4100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.469+0000 7f89bad72640 1 --2- 172.21.15.16:0/3711157910 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89b4106690 0x7f89b4106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.469+0000 7f89bad72640 1 -- 172.21.15.16:0/3711157910 >> 172.21.15.16:0/3711157910 conn(0x7f89b4077800 msgr2=0x7f89b4077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:45.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.469+0000 7f89bad72640 1 -- 172.21.15.16:0/3711157910 shutdown_connections 2024-03-28T22:14:45.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.469+0000 7f89bad72640 1 -- 172.21.15.16:0/3711157910 wait complete. 2024-03-28T22:14:45.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.470+0000 7f89bad72640 1 Processor -- start 2024-03-28T22:14:45.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.470+0000 7f89bad72640 1 -- start start 2024-03-28T22:14:45.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.470+0000 7f89bad72640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89b41005f0 0x7f89b4197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:45.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.470+0000 7f89bad72640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89b4100fb0 0x7f89b4198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:45.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.470+0000 7f89bad72640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89b4106690 0x7f89b419c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.470+0000 7f89bad72640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f89b41109d0 con 0x7f89b41005f0 2024-03-28T22:14:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.470+0000 7f89bad72640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f89b4110850 con 0x7f89b4100fb0 2024-03-28T22:14:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.470+0000 7f89bad72640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f89b4110b50 con 0x7f89b4106690 2024-03-28T22:14:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.470+0000 7f89b8b0e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89b41005f0 0x7f89b4197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.470+0000 7f89b8b0e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89b41005f0 0x7f89b4197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47676/0 (socket says 172.21.15.16:47676) 2024-03-28T22:14:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.470+0000 7f89b8b0e640 1 -- 172.21.15.16:0/3797962689 learned_addr learned my addr 172.21.15.16:0/3797962689 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.470+0000 7f89b3fff640 1 --2- 172.21.15.16:0/3797962689 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89b4100fb0 0x7f89b4198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.470+0000 7f89b930f640 1 --2- 172.21.15.16:0/3797962689 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89b4106690 0x7f89b419c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:45.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.470+0000 7f89b8b0e640 1 -- 172.21.15.16:0/3797962689 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89b4106690 msgr2=0x7f89b419c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:45.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.470+0000 7f89b8b0e640 1 --2- 172.21.15.16:0/3797962689 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89b4106690 0x7f89b419c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.470+0000 7f89b8b0e640 1 -- 172.21.15.16:0/3797962689 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89b4100fb0 msgr2=0x7f89b4198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:45.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.470+0000 7f89b8b0e640 1 --2- 172.21.15.16:0/3797962689 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89b4100fb0 0x7f89b4198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.470+0000 7f89b8b0e640 1 -- 172.21.15.16:0/3797962689 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89b419ced0 con 0x7f89b41005f0 2024-03-28T22:14:45.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.470+0000 7f89b930f640 1 --2- 172.21.15.16:0/3797962689 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89b4106690 0x7f89b419c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:14:45.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.470+0000 7f89b8b0e640 1 --2- 172.21.15.16:0/3797962689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89b41005f0 0x7f89b4197ee0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f89a800d770 tx=0x7f89a800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:45.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.471+0000 7f89b1ffb640 1 -- 172.21.15.16:0/3797962689 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a8017070 con 0x7f89b41005f0 2024-03-28T22:14:45.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.471+0000 7f89b1ffb640 1 -- 172.21.15.16:0/3797962689 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89a8009c30 con 0x7f89b41005f0 2024-03-28T22:14:45.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.471+0000 7f89bad72640 1 -- 172.21.15.16:0/3797962689 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89b419d1c0 con 0x7f89b41005f0 2024-03-28T22:14:45.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.471+0000 7f89b1ffb640 1 -- 172.21.15.16:0/3797962689 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a8003dd0 con 0x7f89b41005f0 2024-03-28T22:14:45.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.471+0000 7f89bad72640 1 -- 172.21.15.16:0/3797962689 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89b41a4a50 con 0x7f89b41005f0 2024-03-28T22:14:45.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.473+0000 7f89b1ffb640 1 -- 172.21.15.16:0/3797962689 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f89a8003fb0 con 0x7f89b41005f0 2024-03-28T22:14:45.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.473+0000 7f89b1ffb640 1 --2- 172.21.15.16:0/3797962689 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f89a40a1f70 0x7f89a40a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:45.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.473+0000 7f89bda2a740 1 Processor -- start 2024-03-28T22:14:45.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.473+0000 7f89bda2a740 1 -- start start 2024-03-28T22:14:45.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.474+0000 7f89b3fff640 1 --2- 172.21.15.16:0/3797962689 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f89a40a1f70 0x7f89a40a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:45.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.474+0000 7f89b1ffb640 1 -- 172.21.15.16:0/3797962689 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f89a80cdac0 con 0x7f89b41005f0 2024-03-28T22:14:45.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.474+0000 7f89bda2a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8bb10e6a0 0x55f8bb10ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:45.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.474+0000 7f89bda2a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f8bb10efc0 0x55f8bb11bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:45.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.474+0000 7f89bda2a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f8bb11c300 0x55f8bb11e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:45.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.474+0000 7f89bda2a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f8bb00f0b0 con 0x55f8bb10efc0 2024-03-28T22:14:45.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.474+0000 7f89bda2a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f8bb071ea0 con 0x55f8bb11c300 2024-03-28T22:14:45.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.474+0000 7f89bda2a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f8bb01ac00 con 0x55f8bb10e6a0 2024-03-28T22:14:45.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.474+0000 7f89b3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f8bb10efc0 0x55f8bb11bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:45.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.474+0000 7f89b930f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f8bb11c300 0x55f8bb11e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:45.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.474+0000 7f89b930f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f8bb11c300 0x55f8bb11e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:51182/0 (socket says 172.21.15.16:51182) 2024-03-28T22:14:45.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.474+0000 7f89b3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f8bb10efc0 0x55f8bb11bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47692/0 (socket says 172.21.15.16:47692) 2024-03-28T22:14:45.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.474+0000 7f89b930f640 1 -- 172.21.15.16:0/3047294313 learned_addr learned my addr 172.21.15.16:0/3047294313 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:45.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.474+0000 7f89b8b0e640 1 --2- 172.21.15.16:0/3047294313 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8bb10e6a0 0x55f8bb10ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:45.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.474+0000 7f89b3fff640 1 --2- 172.21.15.16:0/3797962689 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f89a40a1f70 0x7f89a40a4430 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f89b4069b00 tx=0x7f89a0011440 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:45.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.474+0000 7f89b930f640 1 -- 172.21.15.16:0/3047294313 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8bb10e6a0 msgr2=0x55f8bb10ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:45.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.474+0000 7f89b930f640 1 --2- 172.21.15.16:0/3047294313 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8bb10e6a0 0x55f8bb10ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.474+0000 7f89b930f640 1 -- 172.21.15.16:0/3047294313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f8bb10efc0 msgr2=0x55f8bb11bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:45.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.474+0000 7f89b930f640 1 --2- 172.21.15.16:0/3047294313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f8bb10efc0 0x55f8bb11bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.474+0000 7f89b930f640 1 -- 172.21.15.16:0/3047294313 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8bae4e7e0 con 0x55f8bb11c300 2024-03-28T22:14:45.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.474+0000 7f89b3fff640 1 --2- 172.21.15.16:0/3047294313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f8bb10efc0 0x55f8bb11bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:14:45.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.474+0000 7f89b930f640 1 --2- 172.21.15.16:0/3047294313 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f8bb11c300 0x55f8bb11e6f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f89ac037040 tx=0x7f89ac032490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2190802125569244 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:45.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.475+0000 7f899effd640 1 -- 172.21.15.16:0/3047294313 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ac039050 con 0x55f8bb11c300 2024-03-28T22:14:45.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.475+0000 7f899effd640 1 -- 172.21.15.16:0/3047294313 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89ac045070 con 0x55f8bb11c300 2024-03-28T22:14:45.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.475+0000 7f899effd640 1 -- 172.21.15.16:0/3047294313 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ac0042a0 con 0x55f8bb11c300 2024-03-28T22:14:45.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.475+0000 7f89bda2a740 1 -- 172.21.15.16:0/3047294313 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f8bb11c300 msgr2=0x55f8bb11e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:45.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.475+0000 7f89bda2a740 1 --2- 172.21.15.16:0/3047294313 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f8bb11c300 0x55f8bb11e6f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f89ac037040 tx=0x7f89ac032490 comp rx=0 tx=0).stop 2024-03-28T22:14:45.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.475+0000 7f89bda2a740 1 -- 172.21.15.16:0/3047294313 shutdown_connections 2024-03-28T22:14:45.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.475+0000 7f89bda2a740 1 --2- 172.21.15.16:0/3047294313 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f8bb11c300 0x55f8bb11e6f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.475+0000 7f89bda2a740 1 --2- 172.21.15.16:0/3047294313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f8bb10efc0 0x55f8bb11bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.475+0000 7f89bda2a740 1 --2- 172.21.15.16:0/3047294313 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8bb10e6a0 0x55f8bb10ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.475+0000 7f89bda2a740 1 -- 172.21.15.16:0/3047294313 >> 172.21.15.16:0/3047294313 conn(0x55f8bb109120 msgr2=0x55f8bb109500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:45.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.475+0000 7f89bda2a740 1 -- 172.21.15.16:0/3047294313 shutdown_connections 2024-03-28T22:14:45.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.475+0000 7f89bda2a740 1 -- 172.21.15.16:0/3047294313 wait complete. 2024-03-28T22:14:45.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.476+0000 7f89bda2a740 1 -- start start 2024-03-28T22:14:45.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.476+0000 7f89bda2a740 1 Processor -- start 2024-03-28T22:14:45.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.476+0000 7f89bda2a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8bb10e6a0 0x55f8bb119220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:45.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.476+0000 7f89bda2a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f8bb10efc0 0x55f8bb119760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:45.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.476+0000 7f89bda2a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f8bb11c300 0x55f8bb119ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:45.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.476+0000 7f89bda2a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f8bb0103d0 con 0x55f8bb10efc0 2024-03-28T22:14:45.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.476+0000 7f89bda2a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f8bb06df40 con 0x55f8bb11c300 2024-03-28T22:14:45.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.476+0000 7f89bda2a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f8bb11f0e0 con 0x55f8bb10e6a0 2024-03-28T22:14:45.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.476+0000 7f89bda2a740 10 client.0 _ll_register_callbacks cb 0x55f8bb02db70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:14:45.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.476+0000 7f89b3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f8bb10efc0 0x55f8bb119760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:45.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.476+0000 7f89b3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f8bb10efc0 0x55f8bb119760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47708/0 (socket says 172.21.15.16:47708) 2024-03-28T22:14:45.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.476+0000 7f89b3fff640 1 -- 172.21.15.16:0/1425554401 learned_addr learned my addr 172.21.15.16:0/1425554401 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:45.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.476+0000 7f89b930f640 1 --2- 172.21.15.16:0/1425554401 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f8bb11c300 0x55f8bb119ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:45.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.477+0000 7f89b8b0e640 1 --2- 172.21.15.16:0/1425554401 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8bb10e6a0 0x55f8bb119220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:45.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.477+0000 7f89b3fff640 1 -- 172.21.15.16:0/1425554401 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8bb10e6a0 msgr2=0x55f8bb119220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:45.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.477+0000 7f89b3fff640 1 --2- 172.21.15.16:0/1425554401 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8bb10e6a0 0x55f8bb119220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.477+0000 7f89b3fff640 1 -- 172.21.15.16:0/1425554401 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f8bb11c300 msgr2=0x55f8bb119ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:45.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.477+0000 7f89b3fff640 1 --2- 172.21.15.16:0/1425554401 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f8bb11c300 0x55f8bb119ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.477+0000 7f89b3fff640 1 -- 172.21.15.16:0/1425554401 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8baeaef70 con 0x55f8bb10efc0 2024-03-28T22:14:45.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.477+0000 7f89b930f640 1 --2- 172.21.15.16:0/1425554401 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f8bb11c300 0x55f8bb119ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:14:45.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.477+0000 7f89b3fff640 1 --2- 172.21.15.16:0/1425554401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f8bb10efc0 0x55f8bb119760 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f89a001e9d0 tx=0x7f89a0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16ceb4205ded0bf1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:45.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.477+0000 7f8993fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:14:45.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.477+0000 7f8993fff640 1 -- 172.21.15.16:0/1425554401 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a0017040 con 0x55f8bb10efc0 2024-03-28T22:14:45.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.477+0000 7f8993fff640 1 -- 172.21.15.16:0/1425554401 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f89a000f480 con 0x55f8bb10efc0 2024-03-28T22:14:45.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.477+0000 7f89bda2a740 1 -- 172.21.15.16:0/1425554401 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89ac045070 con 0x55f8bb10efc0 2024-03-28T22:14:45.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.477+0000 7f8993fff640 1 -- 172.21.15.16:0/1425554401 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f89a001e320 con 0x55f8bb10efc0 2024-03-28T22:14:45.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.477+0000 7f89bda2a740 10 client.16410 fetch_fsmap learned FSMap version 26 2024-03-28T22:14:45.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.477+0000 7f89bda2a740 1 -- 172.21.15.16:0/1425554401 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f8bad46170 con 0x55f8bb10efc0 2024-03-28T22:14:45.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.477+0000 7f8993fff640 1 -- 172.21.15.16:0/1425554401 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89ac045070 con 0x55f8bb10efc0 2024-03-28T22:14:45.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.478+0000 7f8993fff640 1 -- 172.21.15.16:0/1425554401 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 26) ==== 8342+0+0 (secure 0 0 0) 0x7f89a002e070 con 0x55f8bb10efc0 2024-03-28T22:14:45.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.478+0000 7f89bda2a740 10 client.16410 fetch_fsmap finished waiting for FSMap version 26 2024-03-28T22:14:45.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.478+0000 7f89bda2a740 10 client.16410 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:14:45.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.478+0000 7f89bda2a740 20 client.16410 populate_metadata read hostname 'smithi016' 2024-03-28T22:14:45.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.478+0000 7f89bda2a740 1 --2- 172.21.15.16:0/1425554401 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f8bb12c290 0x55f8bb12e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:45.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.478+0000 7f89bda2a740 4 client.16410 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:14:45.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.478+0000 7f89bda2a740 1 -- 172.21.15.16:0/1425554401 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f8bb07f3e0 con 0x55f8bb12c290 2024-03-28T22:14:45.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.478+0000 7f89b8b0e640 1 --2- 172.21.15.16:0/1425554401 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f8bb12c290 0x55f8bb12e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:45.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.478+0000 7f89b8b0e640 1 --2- 172.21.15.16:0/1425554401 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f8bb12c290 0x55f8bb12e6b0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac06d7ad3a5a9ca7 server_cookie=80a17c0f512bd8d2 in_seq=0 out_seq=0 2024-03-28T22:14:45.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.478+0000 7f8993fff640 10 client.16410 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:14:45.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.479+0000 7f8993fff640 1 -- 172.21.15.16:0/1425554401 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f8bb07f3e0 con 0x55f8bb12c290 2024-03-28T22:14:45.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.479+0000 7f8993fff640 10 client.16410 handle_command_reply: tid=0 2024-03-28T22:14:45.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.479+0000 7f89bda2a740 1 client.16410 shutdown 2024-03-28T22:14:45.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.479+0000 7f89bda2a740 1 -- 172.21.15.16:0/1425554401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f8bb10efc0 msgr2=0x55f8bb119760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:45.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.479+0000 7f89bda2a740 1 --2- 172.21.15.16:0/1425554401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f8bb10efc0 0x55f8bb119760 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f89a001e9d0 tx=0x7f89a0004730 comp rx=0 tx=0).stop 2024-03-28T22:14:45.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.479+0000 7f89bda2a740 1 -- 172.21.15.16:0/1425554401 shutdown_connections 2024-03-28T22:14:45.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.479+0000 7f89bda2a740 1 --2- 172.21.15.16:0/1425554401 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f8bb12c290 0x55f8bb12e6b0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.479+0000 7f89bda2a740 1 --2- 172.21.15.16:0/1425554401 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f8bb11c300 0x55f8bb119ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.479+0000 7f89bda2a740 1 --2- 172.21.15.16:0/1425554401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f8bb10efc0 0x55f8bb119760 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.479+0000 7f89bda2a740 1 --2- 172.21.15.16:0/1425554401 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8bb10e6a0 0x55f8bb119220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.479+0000 7f8993fff640 0 client.16410 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:14:45.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.479+0000 7f89bda2a740 1 -- 172.21.15.16:0/1425554401 >> 172.21.15.16:0/1425554401 conn(0x55f8bb109120 msgr2=0x55f8bb10ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:45.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.479+0000 7f89bda2a740 1 -- 172.21.15.16:0/1425554401 shutdown_connections 2024-03-28T22:14:45.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.479+0000 7f89bda2a740 1 -- 172.21.15.16:0/1425554401 wait complete. 2024-03-28T22:14:45.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.479+0000 7f89bda2a740 20 client.16410 trim_cache size 0 max 16384 2024-03-28T22:14:45.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.497+0000 7f89bda2a740 1 Processor -- start 2024-03-28T22:14:45.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.497+0000 7f89bda2a740 1 -- start start 2024-03-28T22:14:45.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.497+0000 7f89bda2a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f8bb10e6a0 0x55f8bb11b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:45.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.497+0000 7f89bda2a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8bb10efc0 0x55f8bb127dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:45.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.497+0000 7f89bda2a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f8bb11c300 0x55f8bb1180d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:45.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.497+0000 7f89bda2a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f89b41ad8c0 con 0x55f8bb11c300 2024-03-28T22:14:45.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.497+0000 7f89bda2a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f89b41ab190 con 0x55f8bb10e6a0 2024-03-28T22:14:45.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.497+0000 7f89bda2a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f89b41a8fe0 con 0x55f8bb10efc0 2024-03-28T22:14:45.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.497+0000 7f89b930f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f8bb11c300 0x55f8bb1180d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:45.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.497+0000 7f89b930f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f8bb11c300 0x55f8bb1180d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47714/0 (socket says 172.21.15.16:47714) 2024-03-28T22:14:45.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.497+0000 7f89b930f640 1 -- 172.21.15.16:0/3417307987 learned_addr learned my addr 172.21.15.16:0/3417307987 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:45.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.498+0000 7f89b8b0e640 1 --2- 172.21.15.16:0/3417307987 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f8bb10e6a0 0x55f8bb11b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:45.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.498+0000 7f89b3fff640 1 --2- 172.21.15.16:0/3417307987 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8bb10efc0 0x55f8bb127dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:45.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.498+0000 7f89b930f640 1 -- 172.21.15.16:0/3417307987 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8bb10efc0 msgr2=0x55f8bb127dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:45.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.498+0000 7f89b930f640 1 --2- 172.21.15.16:0/3417307987 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8bb10efc0 0x55f8bb127dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.498+0000 7f89b930f640 1 -- 172.21.15.16:0/3417307987 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f8bb10e6a0 msgr2=0x55f8bb11b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:45.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.498+0000 7f89b930f640 1 --2- 172.21.15.16:0/3417307987 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f8bb10e6a0 0x55f8bb11b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.498+0000 7f89b930f640 1 -- 172.21.15.16:0/3417307987 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8bb113f90 con 0x55f8bb11c300 2024-03-28T22:14:45.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.498+0000 7f89b930f640 1 --2- 172.21.15.16:0/3417307987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f8bb11c300 0x55f8bb1180d0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f89ac037040 tx=0x7f89ac02c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bab821223252a03 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:45.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.498+0000 7f899e7fc640 1 -- 172.21.15.16:0/3417307987 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ac003970 con 0x55f8bb11c300 2024-03-28T22:14:45.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.498+0000 7f899e7fc640 1 -- 172.21.15.16:0/3417307987 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89ac045040 con 0x55f8bb11c300 2024-03-28T22:14:45.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.498+0000 7f899e7fc640 1 -- 172.21.15.16:0/3417307987 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ac040950 con 0x55f8bb11c300 2024-03-28T22:14:45.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.499+0000 7f89bda2a740 1 -- 172.21.15.16:0/3417307987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f8bb11c300 msgr2=0x55f8bb1180d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:45.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.499+0000 7f89bda2a740 1 --2- 172.21.15.16:0/3417307987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f8bb11c300 0x55f8bb1180d0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f89ac037040 tx=0x7f89ac02c0f0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.499+0000 7f89bda2a740 1 -- 172.21.15.16:0/3417307987 shutdown_connections 2024-03-28T22:14:45.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.499+0000 7f89bda2a740 1 --2- 172.21.15.16:0/3417307987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f8bb11c300 0x55f8bb1180d0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.499+0000 7f89bda2a740 1 --2- 172.21.15.16:0/3417307987 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8bb10efc0 0x55f8bb127dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.499+0000 7f89bda2a740 1 --2- 172.21.15.16:0/3417307987 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f8bb10e6a0 0x55f8bb11b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.499+0000 7f89bda2a740 1 -- 172.21.15.16:0/3417307987 >> 172.21.15.16:0/3417307987 conn(0x55f8bb109120 msgr2=0x55f8bb1179c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:45.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.499+0000 7f89bda2a740 1 -- 172.21.15.16:0/3417307987 shutdown_connections 2024-03-28T22:14:45.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.499+0000 7f89bda2a740 1 -- 172.21.15.16:0/3417307987 wait complete. 2024-03-28T22:14:45.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.499+0000 7f89bda2a740 1 -- start start 2024-03-28T22:14:45.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.499+0000 7f89bda2a740 1 Processor -- start 2024-03-28T22:14:45.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.500+0000 7f89bda2a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f8bb10e6a0 0x55f8bb120500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:45.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.500+0000 7f89bda2a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8bb10efc0 0x55f8bb120a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:45.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.500+0000 7f89bda2a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f8bb11c300 0x55f8bb1915d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:45.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.500+0000 7f89bda2a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f89b41a9650 con 0x55f8bb10e6a0 2024-03-28T22:14:45.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.500+0000 7f89bda2a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f89b41aaac0 con 0x55f8bb11c300 2024-03-28T22:14:45.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.500+0000 7f89bda2a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f89b41a9d20 con 0x55f8bb10efc0 2024-03-28T22:14:45.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.500+0000 7f89bda2a740 10 client.0 _ll_register_callbacks cb 0x7f89a001e8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:14:45.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.500+0000 7f89b8b0e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f8bb10e6a0 0x55f8bb120500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:45.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.500+0000 7f89b8b0e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f8bb10e6a0 0x55f8bb120500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47724/0 (socket says 172.21.15.16:47724) 2024-03-28T22:14:45.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.500+0000 7f89b8b0e640 1 -- 172.21.15.16:0/3779767679 learned_addr learned my addr 172.21.15.16:0/3779767679 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:45.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.500+0000 7f89b930f640 1 --2- 172.21.15.16:0/3779767679 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f8bb11c300 0x55f8bb1915d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:45.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.500+0000 7f89b8b0e640 1 -- 172.21.15.16:0/3779767679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8bb10efc0 msgr2=0x55f8bb120a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:45.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.500+0000 7f89b8b0e640 1 --2- 172.21.15.16:0/3779767679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8bb10efc0 0x55f8bb120a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.500+0000 7f89b8b0e640 1 -- 172.21.15.16:0/3779767679 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f8bb11c300 msgr2=0x55f8bb1915d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:45.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.500+0000 7f89b8b0e640 1 --2- 172.21.15.16:0/3779767679 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f8bb11c300 0x55f8bb1915d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.500+0000 7f89b8b0e640 1 -- 172.21.15.16:0/3779767679 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8bb191be0 con 0x55f8bb10e6a0 2024-03-28T22:14:45.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.500+0000 7f89b8b0e640 1 --2- 172.21.15.16:0/3779767679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f8bb10e6a0 0x55f8bb120500 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f89a809a460 tx=0x7f89a80ba040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c328274af5ba155a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:45.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.500+0000 7f8993fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:14:45.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.500+0000 7f8993fff640 1 -- 172.21.15.16:0/3779767679 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a8020040 con 0x55f8bb10e6a0 2024-03-28T22:14:45.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.500+0000 7f8993fff640 1 -- 172.21.15.16:0/3779767679 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f89a80cb4b0 con 0x55f8bb10e6a0 2024-03-28T22:14:45.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.500+0000 7f89bda2a740 1 -- 172.21.15.16:0/3779767679 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89ac045040 con 0x55f8bb10e6a0 2024-03-28T22:14:45.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.501+0000 7f8993fff640 1 -- 172.21.15.16:0/3779767679 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f89a809e620 con 0x55f8bb10e6a0 2024-03-28T22:14:45.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.501+0000 7f89bda2a740 10 client.16416 fetch_fsmap learned FSMap version 27 2024-03-28T22:14:45.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.501+0000 7f89bda2a740 1 -- 172.21.15.16:0/3779767679 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f8bb192f20 con 0x55f8bb10e6a0 2024-03-28T22:14:45.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.501+0000 7f8993fff640 1 -- 172.21.15.16:0/3779767679 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89ac045040 con 0x55f8bb10e6a0 2024-03-28T22:14:45.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.501+0000 7f8993fff640 1 -- 172.21.15.16:0/3779767679 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f89a807a020 con 0x55f8bb10e6a0 2024-03-28T22:14:45.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.501+0000 7f89bda2a740 10 client.16416 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:14:45.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.501+0000 7f89bda2a740 10 client.16416 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:14:45.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.501+0000 7f89bda2a740 20 client.16416 populate_metadata read hostname 'smithi016' 2024-03-28T22:14:45.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.501+0000 7f89bda2a740 1 --2- 172.21.15.16:0/3779767679 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f8bb1935f0 0x55f8bb195a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:45.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.501+0000 7f89bda2a740 4 client.16416 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:14:45.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.501+0000 7f89bda2a740 1 -- 172.21.15.16:0/3779767679 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f8bb079730 con 0x55f8bb1935f0 2024-03-28T22:14:45.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.501+0000 7f89b3fff640 1 --2- 172.21.15.16:0/3779767679 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f8bb1935f0 0x55f8bb195a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:45.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.501+0000 7f8993fff640 10 client.16416 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:14:45.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.501+0000 7f89b3fff640 1 --2- 172.21.15.16:0/3779767679 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f8bb1935f0 0x55f8bb195a30 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec6c9252e64689a4 server_cookie=8aa9cb4af81e7e75 in_seq=0 out_seq=0 2024-03-28T22:14:45.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.503+0000 7f8993fff640 1 -- 172.21.15.16:0/3779767679 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f8bb079730 con 0x55f8bb1935f0 2024-03-28T22:14:45.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.503+0000 7f8993fff640 10 client.16416 handle_command_reply: tid=0 2024-03-28T22:14:45.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.503+0000 7f89bda2a740 1 client.16416 shutdown 2024-03-28T22:14:45.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.503+0000 7f89bda2a740 1 -- 172.21.15.16:0/3779767679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f8bb10e6a0 msgr2=0x55f8bb120500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:45.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.503+0000 7f89bda2a740 1 --2- 172.21.15.16:0/3779767679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f8bb10e6a0 0x55f8bb120500 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f89a809a460 tx=0x7f89a80ba040 comp rx=0 tx=0).stop 2024-03-28T22:14:45.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.503+0000 7f89bda2a740 1 -- 172.21.15.16:0/3779767679 shutdown_connections 2024-03-28T22:14:45.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.503+0000 7f89bda2a740 1 --2- 172.21.15.16:0/3779767679 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f8bb1935f0 0x55f8bb195a30 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.503+0000 7f89bda2a740 1 --2- 172.21.15.16:0/3779767679 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f8bb11c300 0x55f8bb1915d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.503+0000 7f89bda2a740 1 --2- 172.21.15.16:0/3779767679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8bb10efc0 0x55f8bb120a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.503+0000 7f89bda2a740 1 --2- 172.21.15.16:0/3779767679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f8bb10e6a0 0x55f8bb120500 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.503+0000 7f89bda2a740 1 -- 172.21.15.16:0/3779767679 >> 172.21.15.16:0/3779767679 conn(0x55f8bb109120 msgr2=0x55f8bb119a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:45.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.503+0000 7f8993fff640 0 client.16416 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:14:45.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.503+0000 7f89bda2a740 1 -- 172.21.15.16:0/3779767679 shutdown_connections 2024-03-28T22:14:45.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.503+0000 7f89bda2a740 1 -- 172.21.15.16:0/3779767679 wait complete. 2024-03-28T22:14:45.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.503+0000 7f89bda2a740 20 client.16416 trim_cache size 0 max 16384 2024-03-28T22:14:45.517 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:14:45.517 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-03-28T22:14:45.517 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "9647b74d-eb84-4a1c-b29b-16c2d15860a7", 2024-03-28T22:14:45.517 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-03-28T22:14:45.518 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:14:45.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.504+0000 7f89bad72640 1 -- 172.21.15.16:0/3797962689 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f89a40a1f70 msgr2=0x7f89a40a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:45.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.504+0000 7f89bad72640 1 --2- 172.21.15.16:0/3797962689 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f89a40a1f70 0x7f89a40a4430 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f89b4069b00 tx=0x7f89a0011440 comp rx=0 tx=0).stop 2024-03-28T22:14:45.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.504+0000 7f89bad72640 1 -- 172.21.15.16:0/3797962689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89b41005f0 msgr2=0x7f89b4197ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:45.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.504+0000 7f89bad72640 1 --2- 172.21.15.16:0/3797962689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89b41005f0 0x7f89b4197ee0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f89a800d770 tx=0x7f89a800dc30 comp rx=0 tx=0).stop 2024-03-28T22:14:45.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.504+0000 7f89bad72640 1 -- 172.21.15.16:0/3797962689 shutdown_connections 2024-03-28T22:14:45.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.504+0000 7f89bad72640 1 --2- 172.21.15.16:0/3797962689 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f89a40a1f70 0x7f89a40a4430 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.504+0000 7f89bad72640 1 --2- 172.21.15.16:0/3797962689 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89b4106690 0x7f89b419c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.504+0000 7f89bad72640 1 --2- 172.21.15.16:0/3797962689 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89b4100fb0 0x7f89b4198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.504+0000 7f89bad72640 1 --2- 172.21.15.16:0/3797962689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89b41005f0 0x7f89b4197ee0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.504+0000 7f89bad72640 1 -- 172.21.15.16:0/3797962689 >> 172.21.15.16:0/3797962689 conn(0x7f89b4077800 msgr2=0x7f89b4104660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:45.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.504+0000 7f89bad72640 1 -- 172.21.15.16:0/3797962689 shutdown_connections 2024-03-28T22:14:45.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.504+0000 7f89bad72640 1 -- 172.21.15.16:0/3797962689 wait complete. 2024-03-28T22:14:45.521 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:14:45.521 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:14:45.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:45 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:14:45.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:45 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:14:45.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:45 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:14:45.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.626+0000 7f68cd510640 1 -- 172.21.15.16:0/3928014511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68c810e0a0 msgr2=0x7f68c8110490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:45.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.626+0000 7f68cd510640 1 --2- 172.21.15.16:0/3928014511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68c810e0a0 0x7f68c8110490 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f68b8009900 tx=0x7f68b802bcd0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.626+0000 7f68cd510640 1 -- 172.21.15.16:0/3928014511 shutdown_connections 2024-03-28T22:14:45.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.626+0000 7f68cd510640 1 --2- 172.21.15.16:0/3928014511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68c810e0a0 0x7f68c8110490 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.626+0000 7f68cd510640 1 --2- 172.21.15.16:0/3928014511 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68c8100e10 0x7f68c810db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.626+0000 7f68cd510640 1 --2- 172.21.15.16:0/3928014511 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68c81004f0 0x7f68c81008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.626+0000 7f68cd510640 1 -- 172.21.15.16:0/3928014511 >> 172.21.15.16:0/3928014511 conn(0x7f68c8077800 msgr2=0x7f68c8077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:45.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.626+0000 7f68cd510640 1 -- 172.21.15.16:0/3928014511 shutdown_connections 2024-03-28T22:14:45.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.626+0000 7f68cd510640 1 -- 172.21.15.16:0/3928014511 wait complete. 2024-03-28T22:14:45.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.626+0000 7f68cd510640 1 Processor -- start 2024-03-28T22:14:45.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.627+0000 7f68cd510640 1 -- start start 2024-03-28T22:14:45.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.627+0000 7f68cd510640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68c81004f0 0x7f68c819a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:45.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.627+0000 7f68cd510640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68c8100e10 0x7f68c819a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:45.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.627+0000 7f68cd510640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68c810e0a0 0x7f68c819eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:45.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.627+0000 7f68cd510640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f68c8113d00 con 0x7f68c81004f0 2024-03-28T22:14:45.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.627+0000 7f68cd510640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f68c8113b80 con 0x7f68c810e0a0 2024-03-28T22:14:45.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.627+0000 7f68cd510640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f68c8113e80 con 0x7f68c8100e10 2024-03-28T22:14:45.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.627+0000 7f68c6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68c81004f0 0x7f68c819a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:45.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.627+0000 7f68c6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68c81004f0 0x7f68c819a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47738/0 (socket says 172.21.15.16:47738) 2024-03-28T22:14:45.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.627+0000 7f68c6ffd640 1 -- 172.21.15.16:0/3920478537 learned_addr learned my addr 172.21.15.16:0/3920478537 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:45.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.627+0000 7f68c77fe640 1 --2- 172.21.15.16:0/3920478537 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68c810e0a0 0x7f68c819eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:45.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.627+0000 7f68c67fc640 1 --2- 172.21.15.16:0/3920478537 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68c8100e10 0x7f68c819a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.627+0000 7f68c6ffd640 1 -- 172.21.15.16:0/3920478537 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68c8100e10 msgr2=0x7f68c819a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.627+0000 7f68c6ffd640 1 --2- 172.21.15.16:0/3920478537 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68c8100e10 0x7f68c819a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.627+0000 7f68c6ffd640 1 -- 172.21.15.16:0/3920478537 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68c810e0a0 msgr2=0x7f68c819eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.627+0000 7f68c6ffd640 1 --2- 172.21.15.16:0/3920478537 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68c810e0a0 0x7f68c819eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.627+0000 7f68c6ffd640 1 -- 172.21.15.16:0/3920478537 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68c819f1d0 con 0x7f68c81004f0 2024-03-28T22:14:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.628+0000 7f68c6ffd640 1 --2- 172.21.15.16:0/3920478537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68c81004f0 0x7f68c819a1e0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f68c000ea80 tx=0x7f68c000ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.628+0000 7f68bffff640 1 -- 172.21.15.16:0/3920478537 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c000ce40 con 0x7f68c81004f0 2024-03-28T22:14:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.628+0000 7f68cd510640 1 -- 172.21.15.16:0/3920478537 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68c819f4c0 con 0x7f68c81004f0 2024-03-28T22:14:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.628+0000 7f68bffff640 1 -- 172.21.15.16:0/3920478537 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68c0005020 con 0x7f68c81004f0 2024-03-28T22:14:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.628+0000 7f68bffff640 1 -- 172.21.15.16:0/3920478537 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c0011760 con 0x7f68c81004f0 2024-03-28T22:14:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.628+0000 7f68cd510640 1 -- 172.21.15.16:0/3920478537 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68c81a6d50 con 0x7f68c81004f0 2024-03-28T22:14:45.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.630+0000 7f68bffff640 1 -- 172.21.15.16:0/3920478537 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f68c00051d0 con 0x7f68c81004f0 2024-03-28T22:14:45.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.631+0000 7f68d01cd740 1 Processor -- start 2024-03-28T22:14:45.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.631+0000 7f68d01cd740 1 -- start start 2024-03-28T22:14:45.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.631+0000 7f68bffff640 1 --2- 172.21.15.16:0/3920478537 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f68b40a1ef0 0x7f68b40a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:45.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.631+0000 7f68d01cd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558906a1e290 0x558906a1e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:45.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.631+0000 7f68d01cd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558906a1ebb0 0x558906a27250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:45.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.631+0000 7f68d01cd740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558906a27790 0x558906a29b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:45.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.631+0000 7f68d01cd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558906982180 con 0x558906a1e290 2024-03-28T22:14:45.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.631+0000 7f68d01cd740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55890692ab70 con 0x558906a27790 2024-03-28T22:14:45.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.631+0000 7f68d01cd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55890697e220 con 0x558906a1ebb0 2024-03-28T22:14:45.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.631+0000 7f68c77fe640 1 --2- 172.21.15.16:0/3920478537 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f68b40a1ef0 0x7f68b40a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:45.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.631+0000 7f68c6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558906a1e290 0x558906a1e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:45.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.631+0000 7f68c67fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558906a1ebb0 0x558906a27250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:45.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.631+0000 7f68c6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558906a1e290 0x558906a1e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47750/0 (socket says 172.21.15.16:47750) 2024-03-28T22:14:45.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.631+0000 7f68c67fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558906a1ebb0 0x558906a27250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.16:54220/0 (socket says 172.21.15.16:54220) 2024-03-28T22:14:45.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.631+0000 7f68c77fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558906a27790 0x558906a29b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:45.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.631+0000 7f68c6ffd640 1 -- 172.21.15.16:0/1943538819 learned_addr learned my addr 172.21.15.16:0/1943538819 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:45.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.631+0000 7f68bffff640 1 -- 172.21.15.16:0/3920478537 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f68c00ce780 con 0x7f68c81004f0 2024-03-28T22:14:45.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.632+0000 7f68c6ffd640 1 -- 172.21.15.16:0/1943538819 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558906a1ebb0 msgr2=0x558906a27250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:45.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.632+0000 7f68c6ffd640 1 --2- 172.21.15.16:0/1943538819 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558906a1ebb0 0x558906a27250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.632+0000 7f68c6ffd640 1 -- 172.21.15.16:0/1943538819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558906a27790 msgr2=0x558906a29b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:45.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.632+0000 7f68c6ffd640 1 --2- 172.21.15.16:0/1943538819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558906a27790 0x558906a29b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.632+0000 7f68c6ffd640 1 -- 172.21.15.16:0/1943538819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55890693e7c0 con 0x558906a1e290 2024-03-28T22:14:45.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.632+0000 7f68c67fc640 1 --2- 172.21.15.16:0/1943538819 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558906a1ebb0 0x558906a27250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:14:45.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.632+0000 7f68c77fe640 1 --2- 172.21.15.16:0/3920478537 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f68b40a1ef0 0x7f68b40a43b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f68b8009230 tx=0x7f68b8005de0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:45.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.632+0000 7f68c6ffd640 1 --2- 172.21.15.16:0/1943538819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558906a1e290 0x558906a1e670 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f68c00cbef0 tx=0x7f68c001f590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fac9792f1744299d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:45.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.632+0000 7f68bd7fa640 1 -- 172.21.15.16:0/1943538819 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c001f040 con 0x558906a1e290 2024-03-28T22:14:45.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.632+0000 7f68bd7fa640 1 -- 172.21.15.16:0/1943538819 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68c009f280 con 0x558906a1e290 2024-03-28T22:14:45.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.632+0000 7f68bd7fa640 1 -- 172.21.15.16:0/1943538819 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c00b4220 con 0x558906a1e290 2024-03-28T22:14:45.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.633+0000 7f68d01cd740 1 -- 172.21.15.16:0/1943538819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558906a1e290 msgr2=0x558906a1e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:45.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.633+0000 7f68d01cd740 1 --2- 172.21.15.16:0/1943538819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558906a1e290 0x558906a1e670 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f68c00cbef0 tx=0x7f68c001f590 comp rx=0 tx=0).stop 2024-03-28T22:14:45.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.633+0000 7f68d01cd740 1 -- 172.21.15.16:0/1943538819 shutdown_connections 2024-03-28T22:14:45.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.633+0000 7f68d01cd740 1 --2- 172.21.15.16:0/1943538819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558906a27790 0x558906a29b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.633+0000 7f68d01cd740 1 --2- 172.21.15.16:0/1943538819 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558906a1ebb0 0x558906a27250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.633+0000 7f68d01cd740 1 --2- 172.21.15.16:0/1943538819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558906a1e290 0x558906a1e670 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.633+0000 7f68d01cd740 1 -- 172.21.15.16:0/1943538819 >> 172.21.15.16:0/1943538819 conn(0x558906a189c0 msgr2=0x558906a18da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:45.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.633+0000 7f68d01cd740 1 -- 172.21.15.16:0/1943538819 shutdown_connections 2024-03-28T22:14:45.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.633+0000 7f68d01cd740 1 -- 172.21.15.16:0/1943538819 wait complete. 2024-03-28T22:14:45.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.633+0000 7f68d01cd740 1 -- start start 2024-03-28T22:14:45.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.633+0000 7f68d01cd740 1 Processor -- start 2024-03-28T22:14:45.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.633+0000 7f68d01cd740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558906a1e290 0x558906a24860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:45.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.633+0000 7f68d01cd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558906a1ebb0 0x558906a24da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:45.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.633+0000 7f68d01cd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558906a27790 0x558906a252e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:45.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.633+0000 7f68d01cd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558906a2a4c0 con 0x558906a1ebb0 2024-03-28T22:14:45.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.633+0000 7f68d01cd740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5589069204a0 con 0x558906a1e290 2024-03-28T22:14:45.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.633+0000 7f68d01cd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x558906a2a640 con 0x558906a27790 2024-03-28T22:14:45.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.633+0000 7f68d01cd740 10 client.0 _ll_register_callbacks cb 0x558906923d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:14:45.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.633+0000 7f68c67fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558906a1ebb0 0x558906a24da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:45.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.634+0000 7f68c6ffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558906a1e290 0x558906a24860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:45.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.634+0000 7f68c67fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558906a1ebb0 0x558906a24da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47754/0 (socket says 172.21.15.16:47754) 2024-03-28T22:14:45.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.634+0000 7f68c67fc640 1 -- 172.21.15.16:0/1444331621 learned_addr learned my addr 172.21.15.16:0/1444331621 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:45.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.634+0000 7f68c77fe640 1 --2- 172.21.15.16:0/1444331621 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558906a27790 0x558906a252e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:45.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.634+0000 7f68c67fc640 1 -- 172.21.15.16:0/1444331621 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558906a27790 msgr2=0x558906a252e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:45.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.634+0000 7f68c67fc640 1 --2- 172.21.15.16:0/1444331621 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558906a27790 0x558906a252e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.634+0000 7f68c67fc640 1 -- 172.21.15.16:0/1444331621 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558906a1e290 msgr2=0x558906a24860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:45.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.634+0000 7f68c67fc640 1 --2- 172.21.15.16:0/1444331621 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558906a1e290 0x558906a24860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.634+0000 7f68c67fc640 1 -- 172.21.15.16:0/1444331621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589067607d0 con 0x558906a1ebb0 2024-03-28T22:14:45.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.634+0000 7f68c67fc640 1 --2- 172.21.15.16:0/1444331621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558906a1ebb0 0x558906a24da0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f68b000bf00 tx=0x7f68b0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9cd23c2ae9beb25 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:45.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.634+0000 7f68ae7fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:14:45.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.634+0000 7f68ae7fc640 1 -- 172.21.15.16:0/1444331621 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b0004020 con 0x558906a1ebb0 2024-03-28T22:14:45.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.634+0000 7f68ae7fc640 1 -- 172.21.15.16:0/1444331621 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f68b00041c0 con 0x558906a1ebb0 2024-03-28T22:14:45.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.634+0000 7f68d01cd740 1 -- 172.21.15.16:0/1444331621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f68c009f280 con 0x558906a1ebb0 2024-03-28T22:14:45.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.634+0000 7f68ae7fc640 1 -- 172.21.15.16:0/1444331621 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f68b00043c0 con 0x558906a1ebb0 2024-03-28T22:14:45.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.634+0000 7f68d01cd740 10 client.16428 fetch_fsmap learned FSMap version 27 2024-03-28T22:14:45.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.634+0000 7f68d01cd740 1 -- 172.21.15.16:0/1444331621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5589066963f0 con 0x558906a1ebb0 2024-03-28T22:14:45.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.634+0000 7f68ae7fc640 1 -- 172.21.15.16:0/1444331621 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68c009f280 con 0x558906a1ebb0 2024-03-28T22:14:45.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.635+0000 7f68ae7fc640 1 -- 172.21.15.16:0/1444331621 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f68b0021030 con 0x558906a1ebb0 2024-03-28T22:14:45.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.635+0000 7f68d01cd740 10 client.16428 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:14:45.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.635+0000 7f68d01cd740 10 client.16428 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:14:45.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.635+0000 7f68d01cd740 20 client.16428 populate_metadata read hostname 'smithi016' 2024-03-28T22:14:45.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.635+0000 7f68d01cd740 1 --2- 172.21.15.16:0/1444331621 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558906a37ae0 0x558906a39f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:45.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.635+0000 7f68d01cd740 4 client.16428 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:14:45.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.635+0000 7f68d01cd740 1 -- 172.21.15.16:0/1444331621 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55890698f6c0 con 0x558906a37ae0 2024-03-28T22:14:45.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.635+0000 7f68c6ffd640 1 --2- 172.21.15.16:0/1444331621 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558906a37ae0 0x558906a39f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:45.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.635+0000 7f68ae7fc640 10 client.16428 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:14:45.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.635+0000 7f68c6ffd640 1 --2- 172.21.15.16:0/1444331621 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558906a37ae0 0x558906a39f80 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8c553fda308b044 server_cookie=cc69e5ffc7519c85 in_seq=0 out_seq=0 2024-03-28T22:14:45.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.636+0000 7f68ae7fc640 1 -- 172.21.15.16:0/1444331621 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55890698f6c0 con 0x558906a37ae0 2024-03-28T22:14:45.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.636+0000 7f68ae7fc640 10 client.16428 handle_command_reply: tid=0 2024-03-28T22:14:45.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.636+0000 7f68d01cd740 1 client.16428 shutdown 2024-03-28T22:14:45.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.636+0000 7f68d01cd740 1 -- 172.21.15.16:0/1444331621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558906a1ebb0 msgr2=0x558906a24da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:45.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.636+0000 7f68d01cd740 1 --2- 172.21.15.16:0/1444331621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558906a1ebb0 0x558906a24da0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f68b000bf00 tx=0x7f68b0011810 comp rx=0 tx=0).stop 2024-03-28T22:14:45.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.636+0000 7f68d01cd740 1 -- 172.21.15.16:0/1444331621 shutdown_connections 2024-03-28T22:14:45.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.636+0000 7f68d01cd740 1 --2- 172.21.15.16:0/1444331621 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558906a37ae0 0x558906a39f80 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.637+0000 7f68d01cd740 1 --2- 172.21.15.16:0/1444331621 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558906a27790 0x558906a252e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.637+0000 7f68d01cd740 1 --2- 172.21.15.16:0/1444331621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558906a1ebb0 0x558906a24da0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.637+0000 7f68ae7fc640 0 client.16428 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:14:45.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.637+0000 7f68d01cd740 1 --2- 172.21.15.16:0/1444331621 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558906a1e290 0x558906a24860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.637+0000 7f68d01cd740 1 -- 172.21.15.16:0/1444331621 >> 172.21.15.16:0/1444331621 conn(0x558906a189c0 msgr2=0x558906a1a710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:45.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.637+0000 7f68d01cd740 1 -- 172.21.15.16:0/1444331621 shutdown_connections 2024-03-28T22:14:45.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.637+0000 7f68d01cd740 1 -- 172.21.15.16:0/1444331621 wait complete. 2024-03-28T22:14:45.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.637+0000 7f68d01cd740 20 client.16428 trim_cache size 0 max 16384 2024-03-28T22:14:45.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.656+0000 7f68d01cd740 1 Processor -- start 2024-03-28T22:14:45.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.656+0000 7f68d01cd740 1 -- start start 2024-03-28T22:14:45.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.656+0000 7f68d01cd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558906a189c0 0x558906a26f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:45.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.656+0000 7f68d01cd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558906a1e290 0x558906a33260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:45.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.656+0000 7f68d01cd740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558906a1ebb0 0x558906a3f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:45.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.656+0000 7f68d01cd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f68c81b1f40 con 0x558906a1e290 2024-03-28T22:14:45.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.656+0000 7f68d01cd740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f68c81af810 con 0x558906a1ebb0 2024-03-28T22:14:45.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.656+0000 7f68d01cd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f68c81ad660 con 0x558906a189c0 2024-03-28T22:14:45.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.656+0000 7f68c67fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558906a1e290 0x558906a33260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:45.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.656+0000 7f68c67fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558906a1e290 0x558906a33260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47766/0 (socket says 172.21.15.16:47766) 2024-03-28T22:14:45.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.656+0000 7f68c67fc640 1 -- 172.21.15.16:0/710750475 learned_addr learned my addr 172.21.15.16:0/710750475 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:45.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.656+0000 7f68c77fe640 1 --2- 172.21.15.16:0/710750475 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558906a1ebb0 0x558906a3f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:45.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.657+0000 7f68c6ffd640 1 --2- 172.21.15.16:0/710750475 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558906a189c0 0x558906a26f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:45.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.657+0000 7f68c67fc640 1 -- 172.21.15.16:0/710750475 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558906a189c0 msgr2=0x558906a26f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:45.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.657+0000 7f68c67fc640 1 --2- 172.21.15.16:0/710750475 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558906a189c0 0x558906a26f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.657+0000 7f68c67fc640 1 -- 172.21.15.16:0/710750475 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558906a1ebb0 msgr2=0x558906a3f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:45.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.657+0000 7f68c67fc640 1 --2- 172.21.15.16:0/710750475 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558906a1ebb0 0x558906a3f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.657+0000 7f68c67fc640 1 -- 172.21.15.16:0/710750475 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558906a1e9e0 con 0x558906a1e290 2024-03-28T22:14:45.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.657+0000 7f68c67fc640 1 --2- 172.21.15.16:0/710750475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558906a1e290 0x558906a33260 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f68b000bed0 tx=0x7f68b0011d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c143c7a78cd8ff5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:45.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.657+0000 7f68bcff9640 1 -- 172.21.15.16:0/710750475 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b000b480 con 0x558906a1e290 2024-03-28T22:14:45.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.657+0000 7f68bcff9640 1 -- 172.21.15.16:0/710750475 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68b0004b40 con 0x558906a1e290 2024-03-28T22:14:45.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.657+0000 7f68bcff9640 1 -- 172.21.15.16:0/710750475 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b0005310 con 0x558906a1e290 2024-03-28T22:14:45.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.658+0000 7f68d01cd740 1 -- 172.21.15.16:0/710750475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558906a1e290 msgr2=0x558906a33260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:45.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.658+0000 7f68d01cd740 1 --2- 172.21.15.16:0/710750475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558906a1e290 0x558906a33260 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f68b000bed0 tx=0x7f68b0011d10 comp rx=0 tx=0).stop 2024-03-28T22:14:45.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.658+0000 7f68d01cd740 1 -- 172.21.15.16:0/710750475 shutdown_connections 2024-03-28T22:14:45.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.658+0000 7f68d01cd740 1 --2- 172.21.15.16:0/710750475 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558906a1ebb0 0x558906a3f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.658+0000 7f68d01cd740 1 --2- 172.21.15.16:0/710750475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558906a1e290 0x558906a33260 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.658+0000 7f68d01cd740 1 --2- 172.21.15.16:0/710750475 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558906a189c0 0x558906a26f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.658+0000 7f68d01cd740 1 -- 172.21.15.16:0/710750475 >> 172.21.15.16:0/710750475 conn(0x558906a234a0 msgr2=0x558906a17c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:45.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.658+0000 7f68d01cd740 1 -- 172.21.15.16:0/710750475 shutdown_connections 2024-03-28T22:14:45.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.658+0000 7f68d01cd740 1 -- 172.21.15.16:0/710750475 wait complete. 2024-03-28T22:14:45.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.658+0000 7f68d01cd740 1 -- start start 2024-03-28T22:14:45.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.658+0000 7f68d01cd740 1 Processor -- start 2024-03-28T22:14:45.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.658+0000 7f68d01cd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558906a189c0 0x558906a2f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:45.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.658+0000 7f68d01cd740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558906a1e290 0x558906a2f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:45.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.658+0000 7f68d01cd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558906a1ebb0 0x558906a2b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:45.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.658+0000 7f68d01cd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f68c81adcd0 con 0x558906a189c0 2024-03-28T22:14:45.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.658+0000 7f68d01cd740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f68c81af140 con 0x558906a1e290 2024-03-28T22:14:45.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.658+0000 7f68d01cd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f68c81ae3a0 con 0x558906a1ebb0 2024-03-28T22:14:45.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.659+0000 7f68d01cd740 10 client.0 _ll_register_callbacks cb 0x7f68b000bb10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:14:45.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.659+0000 7f68c6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558906a189c0 0x558906a2f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:45.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.659+0000 7f68c67fc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558906a1e290 0x558906a2f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:45.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.659+0000 7f68c6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558906a189c0 0x558906a2f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47778/0 (socket says 172.21.15.16:47778) 2024-03-28T22:14:45.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.659+0000 7f68c6ffd640 1 -- 172.21.15.16:0/180761001 learned_addr learned my addr 172.21.15.16:0/180761001 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:14:45.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.659+0000 7f68c77fe640 1 --2- 172.21.15.16:0/180761001 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558906a1ebb0 0x558906a2b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:45.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.659+0000 7f68c67fc640 1 -- 172.21.15.16:0/180761001 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558906a1ebb0 msgr2=0x558906a2b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:45.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.659+0000 7f68c67fc640 1 --2- 172.21.15.16:0/180761001 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558906a1ebb0 0x558906a2b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.659+0000 7f68c67fc640 1 -- 172.21.15.16:0/180761001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558906a189c0 msgr2=0x558906a2f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:14:45.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.659+0000 7f68c67fc640 1 --2- 172.21.15.16:0/180761001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558906a189c0 0x558906a2f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.659+0000 7f68c67fc640 1 -- 172.21.15.16:0/180761001 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589066560f0 con 0x558906a1e290 2024-03-28T22:14:45.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.659+0000 7f68c67fc640 1 --2- 172.21.15.16:0/180761001 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558906a1e290 0x558906a2f9c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x558906a0d400 tx=0x7f68b00051b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4865a1d1ef308d90 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:14:45.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.659+0000 7f68ae7fc640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-03-28T22:14:45.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.659+0000 7f68ae7fc640 1 -- 172.21.15.16:0/180761001 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b0010910 con 0x558906a1e290 2024-03-28T22:14:45.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.659+0000 7f68d01cd740 1 -- 172.21.15.16:0/180761001 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f68b0004b40 con 0x558906a1e290 2024-03-28T22:14:45.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.659+0000 7f68ae7fc640 1 -- 172.21.15.16:0/180761001 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f68b0004660 con 0x558906a1e290 2024-03-28T22:14:45.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.660+0000 7f68ae7fc640 1 -- 172.21.15.16:0/180761001 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f68b003a050 con 0x558906a1e290 2024-03-28T22:14:45.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.660+0000 7f68d01cd740 10 client.26336 fetch_fsmap learned FSMap version 27 2024-03-28T22:14:45.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.660+0000 7f68d01cd740 1 -- 172.21.15.16:0/180761001 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558906aa6f30 con 0x558906a1e290 2024-03-28T22:14:45.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.660+0000 7f68ae7fc640 1 -- 172.21.15.16:0/180761001 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68b0004b40 con 0x558906a1e290 2024-03-28T22:14:45.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.660+0000 7f68ae7fc640 1 -- 172.21.15.16:0/180761001 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f68b0021040 con 0x558906a1e290 2024-03-28T22:14:45.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.660+0000 7f68d01cd740 10 client.26336 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:14:45.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.660+0000 7f68d01cd740 10 client.26336 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:14:45.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.660+0000 7f68d01cd740 20 client.26336 populate_metadata read hostname 'smithi016' 2024-03-28T22:14:45.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.660+0000 7f68d01cd740 1 --2- 172.21.15.16:0/180761001 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558906a27790 0x558906aa9620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:14:45.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.660+0000 7f68d01cd740 4 client.26336 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:14:45.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.660+0000 7f68d01cd740 1 -- 172.21.15.16:0/180761001 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558906989a10 con 0x558906a27790 2024-03-28T22:14:45.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.660+0000 7f68c6ffd640 1 --2- 172.21.15.16:0/180761001 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558906a27790 0x558906aa9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:14:45.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.661+0000 7f68ae7fc640 10 client.26336 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:14:45.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.661+0000 7f68c6ffd640 1 --2- 172.21.15.16:0/180761001 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558906a27790 0x558906aa9620 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf94ec5b9602384d server_cookie=63fb3fc17842213c in_seq=0 out_seq=0 2024-03-28T22:14:45.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.661+0000 7f68ae7fc640 1 -- 172.21.15.16:0/180761001 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x558906989a10 con 0x558906a27790 2024-03-28T22:14:45.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.661+0000 7f68ae7fc640 10 client.26336 handle_command_reply: tid=0 2024-03-28T22:14:45.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.661+0000 7f68d01cd740 1 client.26336 shutdown 2024-03-28T22:14:45.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.661+0000 7f68d01cd740 1 -- 172.21.15.16:0/180761001 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558906a1e290 msgr2=0x558906a2f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:45.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.661+0000 7f68d01cd740 1 --2- 172.21.15.16:0/180761001 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558906a1e290 0x558906a2f9c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x558906a0d400 tx=0x7f68b00051b0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.661+0000 7f68d01cd740 1 -- 172.21.15.16:0/180761001 shutdown_connections 2024-03-28T22:14:45.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.661+0000 7f68d01cd740 1 --2- 172.21.15.16:0/180761001 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558906a27790 0x558906aa9620 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.661+0000 7f68d01cd740 1 --2- 172.21.15.16:0/180761001 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558906a1ebb0 0x558906a2b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.661+0000 7f68ae7fc640 0 client.26336 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:14:45.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.661+0000 7f68d01cd740 1 --2- 172.21.15.16:0/180761001 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558906a1e290 0x558906a2f9c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.661+0000 7f68d01cd740 1 --2- 172.21.15.16:0/180761001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558906a189c0 0x558906a2f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.661+0000 7f68d01cd740 1 -- 172.21.15.16:0/180761001 >> 172.21.15.16:0/180761001 conn(0x558906a234a0 msgr2=0x558906a17c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:45.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.661+0000 7f68d01cd740 1 -- 172.21.15.16:0/180761001 shutdown_connections 2024-03-28T22:14:45.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.661+0000 7f68d01cd740 1 -- 172.21.15.16:0/180761001 wait complete. 2024-03-28T22:14:45.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.661+0000 7f68d01cd740 20 client.26336 trim_cache size 0 max 16384 2024-03-28T22:14:45.675 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:14:45.675 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T22:14:45.676 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-03-28T22:14:45.676 INFO:teuthology.orchestra.run.smithi016.stdout: "9647b74d-eb84-4a1c-b29b-16c2d15860a7": { 2024-03-28T22:14:45.676 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-03-28T22:14:45.676 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "9647b74d-eb84-4a1c-b29b-16c2d15860a7", 2024-03-28T22:14:45.676 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-03-28T22:14:45.676 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:14:45.676 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:14:45.676 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:14:45.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.662+0000 7f68cd510640 1 -- 172.21.15.16:0/3920478537 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f68b40a1ef0 msgr2=0x7f68b40a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:45.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.662+0000 7f68cd510640 1 --2- 172.21.15.16:0/3920478537 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f68b40a1ef0 0x7f68b40a43b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f68b8009230 tx=0x7f68b8005de0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.662+0000 7f68cd510640 1 -- 172.21.15.16:0/3920478537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68c81004f0 msgr2=0x7f68c819a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:14:45.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.662+0000 7f68cd510640 1 --2- 172.21.15.16:0/3920478537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68c81004f0 0x7f68c819a1e0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f68c000ea80 tx=0x7f68c000ef40 comp rx=0 tx=0).stop 2024-03-28T22:14:45.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.662+0000 7f68cd510640 1 -- 172.21.15.16:0/3920478537 shutdown_connections 2024-03-28T22:14:45.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.662+0000 7f68cd510640 1 --2- 172.21.15.16:0/3920478537 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f68b40a1ef0 0x7f68b40a43b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.662+0000 7f68cd510640 1 --2- 172.21.15.16:0/3920478537 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68c810e0a0 0x7f68c819eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.662+0000 7f68cd510640 1 --2- 172.21.15.16:0/3920478537 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68c8100e10 0x7f68c819a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.662+0000 7f68cd510640 1 --2- 172.21.15.16:0/3920478537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68c81004f0 0x7f68c819a1e0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:14:45.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.662+0000 7f68cd510640 1 -- 172.21.15.16:0/3920478537 >> 172.21.15.16:0/3920478537 conn(0x7f68c8077800 msgr2=0x7f68c810f550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:14:45.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.662+0000 7f68cd510640 1 -- 172.21.15.16:0/3920478537 shutdown_connections 2024-03-28T22:14:45.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:14:45.662+0000 7f68cd510640 1 -- 172.21.15.16:0/3920478537 wait complete. 2024-03-28T22:14:45.680 INFO:tasks.cephfs.filesystem:scrub status for tag:9647b74d-eb84-4a1c-b29b-16c2d15860a7 - {'path': '/', 'tag': '9647b74d-eb84-4a1c-b29b-16c2d15860a7', 'options': 'recursive,force'} 2024-03-28T22:14:46.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:46 smithi063 ceph-mon[36249]: pgmap v128: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:14:46.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:46 smithi063 ceph-mon[36249]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T22:14:46.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:46 smithi063 ceph-mon[36249]: scrub queued for path: / 2024-03-28T22:14:46.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:46 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:14:46.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:46 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:14:46.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:46 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:14:46.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:46 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:14:46.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:46 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:14:46.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:46 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:14:46.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:46 smithi016 ceph-mon[32374]: pgmap v128: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:14:46.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:46 smithi016 ceph-mon[32374]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T22:14:46.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:46 smithi016 ceph-mon[32374]: scrub queued for path: / 2024-03-28T22:14:46.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:46 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:14:46.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:46 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:14:46.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:46 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:14:46.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:46 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:14:46.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:46 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:14:46.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:46 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:14:46.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:46 smithi053 ceph-mon[35979]: pgmap v128: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:14:46.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:46 smithi053 ceph-mon[35979]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-03-28T22:14:46.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:46 smithi053 ceph-mon[35979]: scrub queued for path: / 2024-03-28T22:14:46.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:46 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:14:46.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:46 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:14:46.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:46 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:14:46.549 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:46 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:14:46.549 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:46 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:14:46.549 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:46 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:14:47.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:47 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:14:47.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:47 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:14:47.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:47 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:14:47.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:47 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:14:47.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:47 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:14:47.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:47 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:14:47.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:47 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:14:47.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:47 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:14:47.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:47 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:14:48.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:48 smithi063 ceph-mon[36249]: pgmap v129: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:14:48.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:48 smithi016 ceph-mon[32374]: pgmap v129: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:14:48.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:48 smithi053 ceph-mon[35979]: pgmap v129: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:14:50.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:50 smithi063 ceph-mon[36249]: pgmap v130: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:14:50.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:50 smithi016 ceph-mon[32374]: pgmap v130: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:14:50.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:50 smithi053 ceph-mon[35979]: pgmap v130: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:14:52.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:52 smithi063 ceph-mon[36249]: pgmap v131: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:14:52.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:52 smithi016 ceph-mon[32374]: pgmap v131: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:14:52.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:52 smithi053 ceph-mon[35979]: pgmap v131: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:14:54.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:54 smithi063 ceph-mon[36249]: pgmap v132: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:14:54.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:54 smithi016 ceph-mon[32374]: pgmap v132: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:14:54.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:54 smithi053 ceph-mon[35979]: pgmap v132: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:14:56.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:56 smithi063 ceph-mon[36249]: pgmap v133: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:14:56.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:56 smithi016 ceph-mon[32374]: pgmap v133: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:14:56.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:56 smithi053 ceph-mon[35979]: pgmap v133: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:14:58.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:14:58 smithi063 ceph-mon[36249]: pgmap v134: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:14:58.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:14:58 smithi016 ceph-mon[32374]: pgmap v134: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:14:58.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:14:58 smithi053 ceph-mon[35979]: pgmap v134: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:15:00.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:00 smithi063 ceph-mon[36249]: pgmap v135: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:15:00.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:00 smithi063 ceph-mon[36249]: scrub summary: idle 2024-03-28T22:15:00.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:00 smithi016 ceph-mon[32374]: pgmap v135: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:15:00.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:00 smithi016 ceph-mon[32374]: scrub summary: idle 2024-03-28T22:15:00.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:00 smithi053 ceph-mon[35979]: pgmap v135: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:15:00.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:00 smithi053 ceph-mon[35979]: scrub summary: idle 2024-03-28T22:15:01.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:01 smithi063 ceph-mon[36249]: scrub summary: idle 2024-03-28T22:15:01.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:01 smithi016 ceph-mon[32374]: scrub summary: idle 2024-03-28T22:15:01.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:01 smithi053 ceph-mon[35979]: scrub summary: idle 2024-03-28T22:15:02.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:02 smithi063 ceph-mon[36249]: pgmap v136: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T22:15:02.501 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:02 smithi016 ceph-mon[32374]: pgmap v136: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T22:15:02.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:02 smithi053 ceph-mon[35979]: pgmap v136: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T22:15:04.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:04 smithi016 ceph-mon[32374]: pgmap v137: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-28T22:15:04.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:04 smithi053 ceph-mon[35979]: pgmap v137: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-28T22:15:04.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:04 smithi063 ceph-mon[36249]: pgmap v137: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-28T22:15:06.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:06 smithi016 ceph-mon[32374]: pgmap v138: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-28T22:15:06.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:06 smithi053 ceph-mon[35979]: pgmap v138: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-28T22:15:06.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:06 smithi063 ceph-mon[36249]: pgmap v138: 97 pgs: 97 active+clean; 96 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-28T22:15:08.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:08 smithi053 ceph-mon[35979]: pgmap v139: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T22:15:08.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:08 smithi063 ceph-mon[36249]: pgmap v139: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T22:15:08.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:08 smithi016 ceph-mon[32374]: pgmap v139: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T22:15:10.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:10 smithi053 ceph-mon[35979]: pgmap v140: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-28T22:15:10.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:10 smithi063 ceph-mon[36249]: pgmap v140: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-28T22:15:10.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:10 smithi016 ceph-mon[32374]: pgmap v140: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-28T22:15:12.483 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:12 smithi063 ceph-mon[36249]: pgmap v141: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T22:15:12.487 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:12 smithi016 ceph-mon[32374]: pgmap v141: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T22:15:12.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:12 smithi053 ceph-mon[35979]: pgmap v141: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T22:15:14.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:14 smithi053 ceph-mon[35979]: pgmap v142: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T22:15:14.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:14 smithi063 ceph-mon[36249]: pgmap v142: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T22:15:14.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:14 smithi016 ceph-mon[32374]: pgmap v142: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T22:15:15.680 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:15:15.681 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:15:15.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.796+0000 7f29ae301640 1 -- 172.21.15.16:0/2315130013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a81021e0 msgr2=0x7f29a8102640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:15.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.796+0000 7f29ae301640 1 --2- 172.21.15.16:0/2315130013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a81021e0 0x7f29a8102640 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f299c009900 tx=0x7f299c02bd10 comp rx=0 tx=0).stop 2024-03-28T22:15:15.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.796+0000 7f29ae301640 1 -- 172.21.15.16:0/2315130013 shutdown_connections 2024-03-28T22:15:15.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.796+0000 7f29ae301640 1 --2- 172.21.15.16:0/2315130013 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29a8102b80 0x7f29a810d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:15.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.796+0000 7f29ae301640 1 --2- 172.21.15.16:0/2315130013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a81021e0 0x7f29a8102640 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:15.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.796+0000 7f29ae301640 1 --2- 172.21.15.16:0/2315130013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29a81009d0 0x7f29a8100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:15.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.796+0000 7f29ae301640 1 -- 172.21.15.16:0/2315130013 >> 172.21.15.16:0/2315130013 conn(0x7f29a8077830 msgr2=0x7f29a8077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:15.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.797+0000 7f29ae301640 1 -- 172.21.15.16:0/2315130013 shutdown_connections 2024-03-28T22:15:15.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.797+0000 7f29ae301640 1 -- 172.21.15.16:0/2315130013 wait complete. 2024-03-28T22:15:15.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.797+0000 7f29ae301640 1 Processor -- start 2024-03-28T22:15:15.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.797+0000 7f29ae301640 1 -- start start 2024-03-28T22:15:15.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.798+0000 7f29ae301640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29a81009d0 0x7f29a80fd590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:15.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.798+0000 7f29ae301640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a81021e0 0x7f29a80fdad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:15.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.798+0000 7f29ae301640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29a8102b80 0x7f29a80fe010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:15.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.798+0000 7f29ae301640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f29a8110b20 con 0x7f29a81021e0 2024-03-28T22:15:15.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.798+0000 7f29ae301640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f29a81109a0 con 0x7f29a81009d0 2024-03-28T22:15:15.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.798+0000 7f29ae301640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f29a8110ca0 con 0x7f29a8102b80 2024-03-28T22:15:15.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.798+0000 7f29a77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a81021e0 0x7f29a80fdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:15.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.798+0000 7f29a7fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29a81009d0 0x7f29a80fd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:15.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.798+0000 7f29a77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a81021e0 0x7f29a80fdad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59336/0 (socket says 172.21.15.16:59336) 2024-03-28T22:15:15.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.798+0000 7f29a77fe640 1 -- 172.21.15.16:0/3903657026 learned_addr learned my addr 172.21.15.16:0/3903657026 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:15.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.798+0000 7f29ac89e640 1 --2- 172.21.15.16:0/3903657026 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29a8102b80 0x7f29a80fe010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:15.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.798+0000 7f29a77fe640 1 -- 172.21.15.16:0/3903657026 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29a8102b80 msgr2=0x7f29a80fe010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:15.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.798+0000 7f29a77fe640 1 --2- 172.21.15.16:0/3903657026 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29a8102b80 0x7f29a80fe010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:15.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.798+0000 7f29a77fe640 1 -- 172.21.15.16:0/3903657026 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29a81009d0 msgr2=0x7f29a80fd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:15.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.798+0000 7f29a77fe640 1 --2- 172.21.15.16:0/3903657026 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29a81009d0 0x7f29a80fd590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:15.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.798+0000 7f29a77fe640 1 -- 172.21.15.16:0/3903657026 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29a81a6560 con 0x7f29a81021e0 2024-03-28T22:15:15.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.798+0000 7f29a7fff640 1 --2- 172.21.15.16:0/3903657026 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29a81009d0 0x7f29a80fd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:15:15.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.798+0000 7f29a77fe640 1 --2- 172.21.15.16:0/3903657026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a81021e0 0x7f29a80fdad0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f299c006010 tx=0x7f299c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:15.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.798+0000 7f29a57fa640 1 -- 172.21.15.16:0/3903657026 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f299c042020 con 0x7f29a81021e0 2024-03-28T22:15:15.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.798+0000 7f29ae301640 1 -- 172.21.15.16:0/3903657026 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29a81a67f0 con 0x7f29a81021e0 2024-03-28T22:15:15.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.798+0000 7f29a57fa640 1 -- 172.21.15.16:0/3903657026 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f299c02ce10 con 0x7f29a81021e0 2024-03-28T22:15:15.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.798+0000 7f29a57fa640 1 -- 172.21.15.16:0/3903657026 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f299c003ab0 con 0x7f29a81021e0 2024-03-28T22:15:15.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.799+0000 7f29ae301640 1 -- 172.21.15.16:0/3903657026 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29a81a6d40 con 0x7f29a81021e0 2024-03-28T22:15:15.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.801+0000 7f29a57fa640 1 -- 172.21.15.16:0/3903657026 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f299c045020 con 0x7f29a81021e0 2024-03-28T22:15:15.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.801+0000 7f29b0fb7740 1 Processor -- start 2024-03-28T22:15:15.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.801+0000 7f29b0fb7740 1 -- start start 2024-03-28T22:15:15.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.801+0000 7f29a57fa640 1 --2- 172.21.15.16:0/3903657026 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f29900a1ef0 0x7f29900a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:15.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.801+0000 7f29b0fb7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564754dc0290 0x564754dc0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:15.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.801+0000 7f29b0fb7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564754dc0bb0 0x564754dc9250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:15.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.801+0000 7f29b0fb7740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564754dc9790 0x564754dcbb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:15.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.801+0000 7f29b0fb7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564754d24180 con 0x564754dc0bb0 2024-03-28T22:15:15.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.801+0000 7f29b0fb7740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x564754cccb70 con 0x564754dc9790 2024-03-28T22:15:15.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.801+0000 7f29b0fb7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x564754d20220 con 0x564754dc0290 2024-03-28T22:15:15.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.801+0000 7f29a57fa640 1 -- 172.21.15.16:0/3903657026 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f299c0efe80 con 0x7f29a81021e0 2024-03-28T22:15:15.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.801+0000 7f29ac89e640 1 --2- 172.21.15.16:0/3903657026 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f29900a1ef0 0x7f29900a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:15.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.801+0000 7f29a7fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564754dc0290 0x564754dc0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:15.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.801+0000 7f29a77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564754dc0bb0 0x564754dc9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:15.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.801+0000 7f29a7fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564754dc0290 0x564754dc0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.16:44620/0 (socket says 172.21.15.16:44620) 2024-03-28T22:15:15.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.801+0000 7f29ac89e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564754dc9790 0x564754dcbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:15.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.801+0000 7f29a7fff640 1 -- 172.21.15.16:0/3211912671 learned_addr learned my addr 172.21.15.16:0/3211912671 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:15.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.802+0000 7f29a7fff640 1 -- 172.21.15.16:0/3211912671 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564754dc9790 msgr2=0x564754dcbb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:15.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.802+0000 7f29a7fff640 1 --2- 172.21.15.16:0/3211912671 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564754dc9790 0x564754dcbb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:15.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.802+0000 7f29a7fff640 1 -- 172.21.15.16:0/3211912671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564754dc0bb0 msgr2=0x564754dc9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:15.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.802+0000 7f29a7fff640 1 --2- 172.21.15.16:0/3211912671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564754dc0bb0 0x564754dc9250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:15.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.802+0000 7f29a7fff640 1 -- 172.21.15.16:0/3211912671 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564754ce07c0 con 0x564754dc0290 2024-03-28T22:15:15.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.802+0000 7f29a77fe640 1 --2- 172.21.15.16:0/3211912671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564754dc0bb0 0x564754dc9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:15:15.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.802+0000 7f29ac89e640 1 --2- 172.21.15.16:0/3903657026 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f29900a1ef0 0x7f29900a43b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f29a0005fd0 tx=0x7f29a0009e80 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:15.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.802+0000 7f29a7fff640 1 --2- 172.21.15.16:0/3211912671 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564754dc0290 0x564754dc0670 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2998011540 tx=0x7f2998011a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44a40f56cb46ff82 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:15.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.802+0000 7f298e7fc640 1 -- 172.21.15.16:0/3211912671 <== mon.1 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f299800f5c0 con 0x564754dc0290 2024-03-28T22:15:15.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.802+0000 7f298e7fc640 1 -- 172.21.15.16:0/3211912671 <== mon.1 v2:172.21.15.63:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f299800fc40 con 0x564754dc0290 2024-03-28T22:15:15.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.802+0000 7f298e7fc640 1 -- 172.21.15.16:0/3211912671 <== mon.1 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2998019cd0 con 0x564754dc0290 2024-03-28T22:15:15.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.803+0000 7f29b0fb7740 1 -- 172.21.15.16:0/3211912671 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564754dc0290 msgr2=0x564754dc0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:15.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.803+0000 7f29b0fb7740 1 --2- 172.21.15.16:0/3211912671 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564754dc0290 0x564754dc0670 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2998011540 tx=0x7f2998011a00 comp rx=0 tx=0).stop 2024-03-28T22:15:15.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.803+0000 7f29b0fb7740 1 -- 172.21.15.16:0/3211912671 shutdown_connections 2024-03-28T22:15:15.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.803+0000 7f29b0fb7740 1 --2- 172.21.15.16:0/3211912671 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564754dc9790 0x564754dcbb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:15.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.803+0000 7f29b0fb7740 1 --2- 172.21.15.16:0/3211912671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564754dc0bb0 0x564754dc9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:15.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.803+0000 7f29b0fb7740 1 --2- 172.21.15.16:0/3211912671 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564754dc0290 0x564754dc0670 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:15.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.803+0000 7f29b0fb7740 1 -- 172.21.15.16:0/3211912671 >> 172.21.15.16:0/3211912671 conn(0x564754dba9c0 msgr2=0x564754dbada0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:15.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.803+0000 7f29b0fb7740 1 -- 172.21.15.16:0/3211912671 shutdown_connections 2024-03-28T22:15:15.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.803+0000 7f29b0fb7740 1 -- 172.21.15.16:0/3211912671 wait complete. 2024-03-28T22:15:15.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.803+0000 7f29b0fb7740 1 -- start start 2024-03-28T22:15:15.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.804+0000 7f29b0fb7740 1 Processor -- start 2024-03-28T22:15:15.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.804+0000 7f29b0fb7740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564754dc0290 0x564754dc67b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:15.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.804+0000 7f29b0fb7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564754dc0bb0 0x564754dc6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:15.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.804+0000 7f29b0fb7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564754dc9790 0x564754dc7230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:15.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.804+0000 7f29b0fb7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564754dcc580 con 0x564754dc9790 2024-03-28T22:15:15.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.804+0000 7f29b0fb7740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x564754cc24a0 con 0x564754dc0290 2024-03-28T22:15:15.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.804+0000 7f29b0fb7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x564754dcc700 con 0x564754dc0bb0 2024-03-28T22:15:15.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.804+0000 7f29b0fb7740 10 client.0 _ll_register_callbacks cb 0x564754cc5d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:15:15.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.804+0000 7f29a7fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564754dc0290 0x564754dc67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:15.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.804+0000 7f29ac89e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564754dc9790 0x564754dc7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:15.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.804+0000 7f29a77fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564754dc0bb0 0x564754dc6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:15.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.804+0000 7f29a7fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564754dc0290 0x564754dc67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:51478/0 (socket says 172.21.15.16:51478) 2024-03-28T22:15:15.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.804+0000 7f29a7fff640 1 -- 172.21.15.16:0/165810508 learned_addr learned my addr 172.21.15.16:0/165810508 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:15.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.804+0000 7f29ac89e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564754dc9790 0x564754dc7230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59358/0 (socket says 172.21.15.16:59358) 2024-03-28T22:15:15.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.804+0000 7f29ac89e640 1 -- 172.21.15.16:0/165810508 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564754dc0bb0 msgr2=0x564754dc6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:15.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.804+0000 7f29ac89e640 1 --2- 172.21.15.16:0/165810508 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564754dc0bb0 0x564754dc6cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:15.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.804+0000 7f29ac89e640 1 -- 172.21.15.16:0/165810508 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564754dc0290 msgr2=0x564754dc67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:15.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.804+0000 7f29ac89e640 1 --2- 172.21.15.16:0/165810508 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564754dc0290 0x564754dc67b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:15.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.804+0000 7f29ac89e640 1 -- 172.21.15.16:0/165810508 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564754b027d0 con 0x564754dc9790 2024-03-28T22:15:15.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.804+0000 7f29ac89e640 1 --2- 172.21.15.16:0/165810508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564754dc9790 0x564754dc7230 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f29a0009b30 tx=0x7f29a001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ac68cd097746701 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:15.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.804+0000 7f29877fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:15:15.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.805+0000 7f29877fe640 1 -- 172.21.15.16:0/165810508 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a0024070 con 0x564754dc9790 2024-03-28T22:15:15.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.805+0000 7f29877fe640 1 -- 172.21.15.16:0/165810508 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f29a0004710 con 0x564754dc9790 2024-03-28T22:15:15.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.805+0000 7f29b0fb7740 1 -- 172.21.15.16:0/165810508 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f299800fc40 con 0x564754dc9790 2024-03-28T22:15:15.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.805+0000 7f29877fe640 1 -- 172.21.15.16:0/165810508 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f29a0004b20 con 0x564754dc9790 2024-03-28T22:15:15.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.805+0000 7f29b0fb7740 10 client.16446 fetch_fsmap learned FSMap version 27 2024-03-28T22:15:15.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.805+0000 7f29b0fb7740 1 -- 172.21.15.16:0/165810508 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564754a383f0 con 0x564754dc9790 2024-03-28T22:15:15.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.805+0000 7f29877fe640 1 -- 172.21.15.16:0/165810508 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f299800fc40 con 0x564754dc9790 2024-03-28T22:15:15.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.805+0000 7f29877fe640 1 -- 172.21.15.16:0/165810508 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f29a002f070 con 0x564754dc9790 2024-03-28T22:15:15.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.805+0000 7f29b0fb7740 10 client.16446 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:15:15.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.805+0000 7f29b0fb7740 10 client.16446 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:15:15.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.805+0000 7f29b0fb7740 20 client.16446 populate_metadata read hostname 'smithi016' 2024-03-28T22:15:15.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.805+0000 7f29b0fb7740 1 --2- 172.21.15.16:0/165810508 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x564754dd9b40 0x564754ddbf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:15.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.805+0000 7f29b0fb7740 4 client.16446 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:15:15.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.805+0000 7f29b0fb7740 1 -- 172.21.15.16:0/165810508 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564754d316c0 con 0x564754dd9b40 2024-03-28T22:15:15.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.805+0000 7f29a7fff640 1 --2- 172.21.15.16:0/165810508 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x564754dd9b40 0x564754ddbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:15.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.806+0000 7f29877fe640 10 client.16446 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:15:15.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.806+0000 7f29a7fff640 1 --2- 172.21.15.16:0/165810508 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x564754dd9b40 0x564754ddbf70 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4314e37b94733f9b server_cookie=ef78a52875673efb in_seq=0 out_seq=0 2024-03-28T22:15:15.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.806+0000 7f29877fe640 1 -- 172.21.15.16:0/165810508 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564754d316c0 con 0x564754dd9b40 2024-03-28T22:15:15.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.806+0000 7f29877fe640 10 client.16446 handle_command_reply: tid=0 2024-03-28T22:15:15.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.807+0000 7f29b0fb7740 1 client.16446 shutdown 2024-03-28T22:15:15.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.807+0000 7f29b0fb7740 1 -- 172.21.15.16:0/165810508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564754dc9790 msgr2=0x564754dc7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:15.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.807+0000 7f29b0fb7740 1 --2- 172.21.15.16:0/165810508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564754dc9790 0x564754dc7230 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f29a0009b30 tx=0x7f29a001d000 comp rx=0 tx=0).stop 2024-03-28T22:15:15.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.807+0000 7f29b0fb7740 1 -- 172.21.15.16:0/165810508 shutdown_connections 2024-03-28T22:15:15.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.807+0000 7f29b0fb7740 1 --2- 172.21.15.16:0/165810508 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x564754dd9b40 0x564754ddbf70 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:15.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.807+0000 7f29b0fb7740 1 --2- 172.21.15.16:0/165810508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564754dc9790 0x564754dc7230 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:15.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.807+0000 7f29877fe640 0 client.16446 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:15:15.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.807+0000 7f29b0fb7740 1 --2- 172.21.15.16:0/165810508 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564754dc0bb0 0x564754dc6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:15.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.807+0000 7f29b0fb7740 1 --2- 172.21.15.16:0/165810508 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564754dc0290 0x564754dc67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:15.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.807+0000 7f29b0fb7740 1 -- 172.21.15.16:0/165810508 >> 172.21.15.16:0/165810508 conn(0x564754dba9c0 msgr2=0x564754dbc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:15.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.807+0000 7f29b0fb7740 1 -- 172.21.15.16:0/165810508 shutdown_connections 2024-03-28T22:15:15.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.807+0000 7f29b0fb7740 1 -- 172.21.15.16:0/165810508 wait complete. 2024-03-28T22:15:15.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.807+0000 7f29b0fb7740 20 client.16446 trim_cache size 0 max 16384 2024-03-28T22:15:15.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.824+0000 7f29b0fb7740 1 Processor -- start 2024-03-28T22:15:15.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.824+0000 7f29b0fb7740 1 -- start start 2024-03-28T22:15:15.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.824+0000 7f29b0fb7740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564754dc0290 0x564754dc8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:15.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.824+0000 7f29b0fb7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564754dc0bb0 0x564754dd5320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:15.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.824+0000 7f29b0fb7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564754dc9790 0x564754db9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:15.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.824+0000 7f29b0fb7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f29a81b1de0 con 0x564754dc9790 2024-03-28T22:15:15.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.824+0000 7f29b0fb7740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f29a81af6b0 con 0x564754dc0290 2024-03-28T22:15:15.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.824+0000 7f29b0fb7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f29a81ad520 con 0x564754dc0bb0 2024-03-28T22:15:15.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.825+0000 7f29ac89e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564754dc9790 0x564754db9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:15.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.825+0000 7f29ac89e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564754dc9790 0x564754db9aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59360/0 (socket says 172.21.15.16:59360) 2024-03-28T22:15:15.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.825+0000 7f29ac89e640 1 -- 172.21.15.16:0/2610339203 learned_addr learned my addr 172.21.15.16:0/2610339203 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:15.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.825+0000 7f29a7fff640 1 --2- 172.21.15.16:0/2610339203 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564754dc0290 0x564754dc8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:15.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.825+0000 7f29a77fe640 1 --2- 172.21.15.16:0/2610339203 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564754dc0bb0 0x564754dd5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:15.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.825+0000 7f29ac89e640 1 -- 172.21.15.16:0/2610339203 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564754dc0bb0 msgr2=0x564754dd5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:15.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.825+0000 7f29ac89e640 1 --2- 172.21.15.16:0/2610339203 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564754dc0bb0 0x564754dd5320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:15.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.825+0000 7f29ac89e640 1 -- 172.21.15.16:0/2610339203 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564754dc0290 msgr2=0x564754dc8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:15.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.825+0000 7f29ac89e640 1 --2- 172.21.15.16:0/2610339203 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564754dc0290 0x564754dc8ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:15.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.825+0000 7f29ac89e640 1 -- 172.21.15.16:0/2610339203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564754dc09e0 con 0x564754dc9790 2024-03-28T22:15:15.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.825+0000 7f29ac89e640 1 --2- 172.21.15.16:0/2610339203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564754dc9790 0x564754db9aa0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f29a000f850 tx=0x7f29a00094e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90d36c9e0db49fc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:15.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.825+0000 7f298dffb640 1 -- 172.21.15.16:0/2610339203 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a0035070 con 0x564754dc9790 2024-03-28T22:15:15.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.825+0000 7f298dffb640 1 -- 172.21.15.16:0/2610339203 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29a0011560 con 0x564754dc9790 2024-03-28T22:15:15.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.825+0000 7f298dffb640 1 -- 172.21.15.16:0/2610339203 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a0024040 con 0x564754dc9790 2024-03-28T22:15:15.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.826+0000 7f29b0fb7740 1 -- 172.21.15.16:0/2610339203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564754dc9790 msgr2=0x564754db9aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:15.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.826+0000 7f29b0fb7740 1 --2- 172.21.15.16:0/2610339203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564754dc9790 0x564754db9aa0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f29a000f850 tx=0x7f29a00094e0 comp rx=0 tx=0).stop 2024-03-28T22:15:15.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.826+0000 7f29b0fb7740 1 -- 172.21.15.16:0/2610339203 shutdown_connections 2024-03-28T22:15:15.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.826+0000 7f29b0fb7740 1 --2- 172.21.15.16:0/2610339203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564754dc9790 0x564754db9aa0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:15.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.826+0000 7f29b0fb7740 1 --2- 172.21.15.16:0/2610339203 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564754dc0bb0 0x564754dd5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:15.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.826+0000 7f29b0fb7740 1 --2- 172.21.15.16:0/2610339203 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564754dc0290 0x564754dc8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:15.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.826+0000 7f29b0fb7740 1 -- 172.21.15.16:0/2610339203 >> 172.21.15.16:0/2610339203 conn(0x564754dba9c0 msgr2=0x564754db9390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:15.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.826+0000 7f29b0fb7740 1 -- 172.21.15.16:0/2610339203 shutdown_connections 2024-03-28T22:15:15.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.826+0000 7f29b0fb7740 1 -- 172.21.15.16:0/2610339203 wait complete. 2024-03-28T22:15:15.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.826+0000 7f29b0fb7740 1 -- start start 2024-03-28T22:15:15.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.826+0000 7f29b0fb7740 1 Processor -- start 2024-03-28T22:15:15.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.827+0000 7f29b0fb7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564754dc0290 0x564754dd1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:15.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.827+0000 7f29b0fb7740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564754dc0bb0 0x564754dc3210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:15.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.827+0000 7f29b0fb7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564754dc9790 0x564754dc3950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:15.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.827+0000 7f29b0fb7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f29a81adb70 con 0x564754dc0290 2024-03-28T22:15:15.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.827+0000 7f29b0fb7740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f29a81aefe0 con 0x564754dc0bb0 2024-03-28T22:15:15.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.827+0000 7f29b0fb7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f29a81ae240 con 0x564754dc9790 2024-03-28T22:15:15.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.827+0000 7f29b0fb7740 10 client.0 _ll_register_callbacks cb 0x7f29a0009a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:15:15.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.827+0000 7f29a7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564754dc0290 0x564754dd1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:15.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.827+0000 7f29a7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564754dc0290 0x564754dd1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59368/0 (socket says 172.21.15.16:59368) 2024-03-28T22:15:15.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.827+0000 7f29a7fff640 1 -- 172.21.15.16:0/449860684 learned_addr learned my addr 172.21.15.16:0/449860684 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:15.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.827+0000 7f29a77fe640 1 --2- 172.21.15.16:0/449860684 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564754dc0bb0 0x564754dc3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:15.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.827+0000 7f29ac89e640 1 --2- 172.21.15.16:0/449860684 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564754dc9790 0x564754dc3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:15.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.827+0000 7f29a7fff640 1 -- 172.21.15.16:0/449860684 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564754dc9790 msgr2=0x564754dc3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:15.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.827+0000 7f29a7fff640 1 --2- 172.21.15.16:0/449860684 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564754dc9790 0x564754dc3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:15.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.827+0000 7f29a7fff640 1 -- 172.21.15.16:0/449860684 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564754dc0bb0 msgr2=0x564754dc3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:15.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.827+0000 7f29a7fff640 1 --2- 172.21.15.16:0/449860684 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564754dc0bb0 0x564754dc3210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:15.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.827+0000 7f29a7fff640 1 -- 172.21.15.16:0/449860684 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647549f80f0 con 0x564754dc0290 2024-03-28T22:15:15.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.827+0000 7f29ac89e640 1 --2- 172.21.15.16:0/449860684 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564754dc9790 0x564754dc3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:15:15.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.827+0000 7f29a7fff640 1 --2- 172.21.15.16:0/449860684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564754dc0290 0x564754dd1da0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f2998011f00 tx=0x7f2998019620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaed8f987929481a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:15.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.827+0000 7f29877fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:15:15.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.827+0000 7f29877fe640 1 -- 172.21.15.16:0/449860684 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2998009d00 con 0x564754dc0290 2024-03-28T22:15:15.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.827+0000 7f29877fe640 1 -- 172.21.15.16:0/449860684 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f29980052c0 con 0x564754dc0290 2024-03-28T22:15:15.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.827+0000 7f29b0fb7740 1 -- 172.21.15.16:0/449860684 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564754d2ba10 con 0x564754dc0290 2024-03-28T22:15:15.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.827+0000 7f29877fe640 1 -- 172.21.15.16:0/449860684 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f29980054c0 con 0x564754dc0290 2024-03-28T22:15:15.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.827+0000 7f29b0fb7740 10 client.16452 fetch_fsmap learned FSMap version 27 2024-03-28T22:15:15.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.828+0000 7f29b0fb7740 1 -- 172.21.15.16:0/449860684 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564754e3fe50 con 0x564754dc0290 2024-03-28T22:15:15.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.828+0000 7f29877fe640 1 -- 172.21.15.16:0/449860684 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x564754d2ba10 con 0x564754dc0290 2024-03-28T22:15:15.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.828+0000 7f29877fe640 1 -- 172.21.15.16:0/449860684 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f2998034020 con 0x564754dc0290 2024-03-28T22:15:15.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.828+0000 7f29b0fb7740 10 client.16452 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:15:15.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.828+0000 7f29b0fb7740 10 client.16452 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:15:15.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.828+0000 7f29b0fb7740 20 client.16452 populate_metadata read hostname 'smithi016' 2024-03-28T22:15:15.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.828+0000 7f29b0fb7740 1 --2- 172.21.15.16:0/449860684 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x564754e40440 0x564754e42860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:15.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.828+0000 7f29b0fb7740 4 client.16452 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:15:15.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.828+0000 7f29b0fb7740 1 -- 172.21.15.16:0/449860684 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564754ca9540 con 0x564754e40440 2024-03-28T22:15:15.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.828+0000 7f29a77fe640 1 --2- 172.21.15.16:0/449860684 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x564754e40440 0x564754e42860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:15.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.828+0000 7f29877fe640 10 client.16452 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:15:15.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.828+0000 7f29a77fe640 1 --2- 172.21.15.16:0/449860684 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x564754e40440 0x564754e42860 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a18df9d9d9e73846 server_cookie=951309b9557d2094 in_seq=0 out_seq=0 2024-03-28T22:15:15.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.829+0000 7f29877fe640 1 -- 172.21.15.16:0/449860684 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x564754ca9540 con 0x564754e40440 2024-03-28T22:15:15.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.829+0000 7f29877fe640 10 client.16452 handle_command_reply: tid=0 2024-03-28T22:15:15.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.829+0000 7f29b0fb7740 1 client.16452 shutdown 2024-03-28T22:15:15.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.829+0000 7f29b0fb7740 1 -- 172.21.15.16:0/449860684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564754dc0290 msgr2=0x564754dd1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:15.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.829+0000 7f29b0fb7740 1 --2- 172.21.15.16:0/449860684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564754dc0290 0x564754dd1da0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f2998011f00 tx=0x7f2998019620 comp rx=0 tx=0).stop 2024-03-28T22:15:15.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.829+0000 7f29b0fb7740 1 -- 172.21.15.16:0/449860684 shutdown_connections 2024-03-28T22:15:15.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.829+0000 7f29b0fb7740 1 --2- 172.21.15.16:0/449860684 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x564754e40440 0x564754e42860 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:15.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.829+0000 7f29b0fb7740 1 --2- 172.21.15.16:0/449860684 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564754dc9790 0x564754dc3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:15.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.829+0000 7f29b0fb7740 1 --2- 172.21.15.16:0/449860684 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564754dc0bb0 0x564754dc3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:15.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.829+0000 7f29b0fb7740 1 --2- 172.21.15.16:0/449860684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564754dc0290 0x564754dd1da0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:15.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.829+0000 7f29877fe640 0 client.16452 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:15:15.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.829+0000 7f29b0fb7740 1 -- 172.21.15.16:0/449860684 >> 172.21.15.16:0/449860684 conn(0x564754dba9c0 msgr2=0x564754dbc630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:15.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.829+0000 7f29b0fb7740 1 -- 172.21.15.16:0/449860684 shutdown_connections 2024-03-28T22:15:15.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.829+0000 7f29b0fb7740 1 -- 172.21.15.16:0/449860684 wait complete. 2024-03-28T22:15:15.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.829+0000 7f29b0fb7740 20 client.16452 trim_cache size 0 max 16384 2024-03-28T22:15:15.842 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:15:15.842 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-03-28T22:15:15.843 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-03-28T22:15:15.843 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:15:15.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.829+0000 7f29ae301640 1 -- 172.21.15.16:0/3903657026 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f29900a1ef0 msgr2=0x7f29900a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:15.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.829+0000 7f29ae301640 1 --2- 172.21.15.16:0/3903657026 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f29900a1ef0 0x7f29900a43b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f29a0005fd0 tx=0x7f29a0009e80 comp rx=0 tx=0).stop 2024-03-28T22:15:15.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.829+0000 7f29ae301640 1 -- 172.21.15.16:0/3903657026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a81021e0 msgr2=0x7f29a80fdad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:15.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.829+0000 7f29ae301640 1 --2- 172.21.15.16:0/3903657026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a81021e0 0x7f29a80fdad0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f299c006010 tx=0x7f299c004270 comp rx=0 tx=0).stop 2024-03-28T22:15:15.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.830+0000 7f29ae301640 1 -- 172.21.15.16:0/3903657026 shutdown_connections 2024-03-28T22:15:15.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.830+0000 7f29ae301640 1 --2- 172.21.15.16:0/3903657026 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f29900a1ef0 0x7f29900a43b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:15.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.830+0000 7f29ae301640 1 --2- 172.21.15.16:0/3903657026 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29a8102b80 0x7f29a80fe010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:15.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.830+0000 7f29ae301640 1 --2- 172.21.15.16:0/3903657026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a81021e0 0x7f29a80fdad0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:15.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.830+0000 7f29ae301640 1 --2- 172.21.15.16:0/3903657026 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29a81009d0 0x7f29a80fd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:15.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.830+0000 7f29ae301640 1 -- 172.21.15.16:0/3903657026 >> 172.21.15.16:0/3903657026 conn(0x7f29a8077830 msgr2=0x7f29a806a0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:15.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.830+0000 7f29ae301640 1 -- 172.21.15.16:0/3903657026 shutdown_connections 2024-03-28T22:15:15.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.830+0000 7f29ae301640 1 -- 172.21.15.16:0/3903657026 wait complete. 2024-03-28T22:15:15.846 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:15:15.846 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:15:15.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.948+0000 7fd4ebec2640 1 -- 172.21.15.16:0/4017366658 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4e410bd40 msgr2=0x7fd4e410e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:15.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.948+0000 7fd4ebec2640 1 --2- 172.21.15.16:0/4017366658 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4e410bd40 0x7fd4e410e190 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd4e0009ca0 tx=0x7fd4e002c180 comp rx=0 tx=0).stop 2024-03-28T22:15:15.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.948+0000 7fd4ebec2640 1 -- 172.21.15.16:0/4017366658 shutdown_connections 2024-03-28T22:15:15.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.948+0000 7fd4ebec2640 1 --2- 172.21.15.16:0/4017366658 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4e410bd40 0x7fd4e410e190 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:15.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.948+0000 7fd4ebec2640 1 --2- 172.21.15.16:0/4017366658 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd4e4103290 0x7fd4e410b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:15.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.948+0000 7fd4ebec2640 1 --2- 172.21.15.16:0/4017366658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4e4102970 0x7fd4e4102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:15.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.948+0000 7fd4ebec2640 1 -- 172.21.15.16:0/4017366658 >> 172.21.15.16:0/4017366658 conn(0x7fd4e40777d0 msgr2=0x7fd4e4077be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:15.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.948+0000 7fd4ebec2640 1 -- 172.21.15.16:0/4017366658 shutdown_connections 2024-03-28T22:15:15.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.948+0000 7fd4ebec2640 1 -- 172.21.15.16:0/4017366658 wait complete. 2024-03-28T22:15:15.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.948+0000 7fd4ebec2640 1 Processor -- start 2024-03-28T22:15:15.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.948+0000 7fd4ebec2640 1 -- start start 2024-03-28T22:15:15.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.948+0000 7fd4ebec2640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd4e4102970 0x7fd4e419a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:15.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.948+0000 7fd4ebec2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4e4103290 0x7fd4e419a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:15.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.948+0000 7fd4ebec2640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4e410bd40 0x7fd4e419ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:15.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.948+0000 7fd4ebec2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd4e4111ae0 con 0x7fd4e4103290 2024-03-28T22:15:15.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.948+0000 7fd4ebec2640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd4e4111960 con 0x7fd4e410bd40 2024-03-28T22:15:15.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.948+0000 7fd4ebec2640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd4e4111c60 con 0x7fd4e4102970 2024-03-28T22:15:15.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.949+0000 7fd4e945d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4e4103290 0x7fd4e419a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:15.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.949+0000 7fd4e945d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4e4103290 0x7fd4e419a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59378/0 (socket says 172.21.15.16:59378) 2024-03-28T22:15:15.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.949+0000 7fd4e945d640 1 -- 172.21.15.16:0/2667444867 learned_addr learned my addr 172.21.15.16:0/2667444867 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:15.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.949+0000 7fd4e9c5e640 1 --2- 172.21.15.16:0/2667444867 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd4e4102970 0x7fd4e419a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:15.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.949+0000 7fd4ea45f640 1 --2- 172.21.15.16:0/2667444867 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4e410bd40 0x7fd4e419ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:15.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.949+0000 7fd4e945d640 1 -- 172.21.15.16:0/2667444867 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd4e4102970 msgr2=0x7fd4e419a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:15.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.949+0000 7fd4e945d640 1 --2- 172.21.15.16:0/2667444867 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd4e4102970 0x7fd4e419a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:15.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.949+0000 7fd4e945d640 1 -- 172.21.15.16:0/2667444867 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4e410bd40 msgr2=0x7fd4e419ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:15.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.949+0000 7fd4e945d640 1 --2- 172.21.15.16:0/2667444867 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4e410bd40 0x7fd4e419ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:15.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.949+0000 7fd4e945d640 1 -- 172.21.15.16:0/2667444867 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4e419f1a0 con 0x7fd4e4103290 2024-03-28T22:15:15.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.949+0000 7fd4ea45f640 1 --2- 172.21.15.16:0/2667444867 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4e410bd40 0x7fd4e419ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:15:15.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.949+0000 7fd4e945d640 1 --2- 172.21.15.16:0/2667444867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4e4103290 0x7fd4e419a740 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fd4d400e9f0 tx=0x7fd4d400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:15.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.949+0000 7fd4daffd640 1 -- 172.21.15.16:0/2667444867 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d400cd80 con 0x7fd4e4103290 2024-03-28T22:15:15.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.949+0000 7fd4daffd640 1 -- 172.21.15.16:0/2667444867 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4d4005500 con 0x7fd4e4103290 2024-03-28T22:15:15.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.949+0000 7fd4ebec2640 1 -- 172.21.15.16:0/2667444867 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4e419f490 con 0x7fd4e4103290 2024-03-28T22:15:15.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.949+0000 7fd4daffd640 1 -- 172.21.15.16:0/2667444867 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d4010690 con 0x7fd4e4103290 2024-03-28T22:15:15.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.949+0000 7fd4ebec2640 1 -- 172.21.15.16:0/2667444867 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4e41a6d20 con 0x7fd4e4103290 2024-03-28T22:15:15.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.950+0000 7fd4ebec2640 1 -- 172.21.15.16:0/2667444867 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4e40685d0 con 0x7fd4e4103290 2024-03-28T22:15:15.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.951+0000 7fd4daffd640 1 -- 172.21.15.16:0/2667444867 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7fd4d40046c0 con 0x7fd4e4103290 2024-03-28T22:15:15.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.952+0000 7fd4daffd640 1 --2- 172.21.15.16:0/2667444867 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fd4d00a1ef0 0x7fd4d00a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:15.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.952+0000 7fd4daffd640 1 -- 172.21.15.16:0/2667444867 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fd4d40ce6f0 con 0x7fd4e4103290 2024-03-28T22:15:15.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.952+0000 7fd4e9c5e640 1 --2- 172.21.15.16:0/2667444867 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fd4d00a1ef0 0x7fd4d00a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:15.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.952+0000 7fd4e9c5e640 1 --2- 172.21.15.16:0/2667444867 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fd4d00a1ef0 0x7fd4d00a43b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc009bd0 tx=0x7fd4dc005f90 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:15.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:15.956+0000 7fd4daffd640 1 -- 172.21.15.16:0/2667444867 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd4d4053280 con 0x7fd4e4103290 2024-03-28T22:15:16.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.181+0000 7fd4ebec2640 1 -- 172.21.15.16:0/2667444867 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd4e419b500 con 0x7fd4e4103290 2024-03-28T22:15:16.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.181+0000 7fd4daffd640 1 -- 172.21.15.16:0/2667444867 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11851 (secure 0 0 0) 0x7fd4e419b500 con 0x7fd4e4103290 2024-03-28T22:15:16.183 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:15:16.184 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/1644581227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1644581227},{"type":"v1","addr":"172.21.15.16:6839","nonce":1644581227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/2593062669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2593062669},{"type":"v1","addr":"172.21.15.16:6841","nonce":2593062669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:08:29.672976+0000","modified":"2024-03-28T22:14:44.922044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24566,"mds_1":24524,"mds_2":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24466":{"gid":24466,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/247249857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":247249857},{"type":"v1","addr":"172.21.15.63:6835","nonce":247249857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6837/2324231032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":2324231032},{"type":"v1","addr":"172.21.15.63:6837","nonce":2324231032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6839/1177197077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":1177197077},{"type":"v1","addr":"172.21.15.63:6839","nonce":1177197077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"e","rank":2,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.53:6835/350727634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":350727634},{"type":"v1","addr":"172.21.15.53:6835","nonce":350727634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/349877573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":349877573},{"type":"v1","addr":"172.21.15.53:6837","nonce":349877573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6839/3714153956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":3714153956},{"type":"v1","addr":"172.21.15.53:6839","nonce":3714153956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24566,"qdb_cluster":[24500,24524,24566]},"id":1}]} 2024-03-28T22:15:16.184 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2024-03-28T22:15:16.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.184+0000 7fd4ebec2640 1 -- 172.21.15.16:0/2667444867 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fd4d00a1ef0 msgr2=0x7fd4d00a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:16.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.184+0000 7fd4ebec2640 1 --2- 172.21.15.16:0/2667444867 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fd4d00a1ef0 0x7fd4d00a43b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc009bd0 tx=0x7fd4dc005f90 comp rx=0 tx=0).stop 2024-03-28T22:15:16.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.184+0000 7fd4ebec2640 1 -- 172.21.15.16:0/2667444867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4e4103290 msgr2=0x7fd4e419a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:16.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.184+0000 7fd4ebec2640 1 --2- 172.21.15.16:0/2667444867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4e4103290 0x7fd4e419a740 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fd4d400e9f0 tx=0x7fd4d400edf0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.184+0000 7fd4ebec2640 1 -- 172.21.15.16:0/2667444867 shutdown_connections 2024-03-28T22:15:16.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.184+0000 7fd4ebec2640 1 --2- 172.21.15.16:0/2667444867 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fd4d00a1ef0 0x7fd4d00a43b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.184+0000 7fd4ebec2640 1 --2- 172.21.15.16:0/2667444867 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4e410bd40 0x7fd4e419ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.184+0000 7fd4ebec2640 1 --2- 172.21.15.16:0/2667444867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4e4103290 0x7fd4e419a740 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.184+0000 7fd4ebec2640 1 --2- 172.21.15.16:0/2667444867 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd4e4102970 0x7fd4e419a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.184+0000 7fd4ebec2640 1 -- 172.21.15.16:0/2667444867 >> 172.21.15.16:0/2667444867 conn(0x7fd4e40777d0 msgr2=0x7fd4e4107bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:16.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.184+0000 7fd4ebec2640 1 -- 172.21.15.16:0/2667444867 shutdown_connections 2024-03-28T22:15:16.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.184+0000 7fd4ebec2640 1 -- 172.21.15.16:0/2667444867 wait complete. 2024-03-28T22:15:16.196 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:15:16.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.299+0000 7fbed42f4640 1 -- 172.21.15.16:0/2622872093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbecc1005e0 msgr2=0x7fbecc100a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:16.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.299+0000 7fbed42f4640 1 --2- 172.21.15.16:0/2622872093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbecc1005e0 0x7fbecc100a60 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fbebc009930 tx=0x7fbebc02bc30 comp rx=0 tx=0).stop 2024-03-28T22:15:16.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.299+0000 7fbed42f4640 1 -- 172.21.15.16:0/2622872093 shutdown_connections 2024-03-28T22:15:16.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.299+0000 7fbed42f4640 1 --2- 172.21.15.16:0/2622872093 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbecc100fa0 0x7fbecc10d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.299+0000 7fbed42f4640 1 --2- 172.21.15.16:0/2622872093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbecc1005e0 0x7fbecc100a60 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.299+0000 7fbed42f4640 1 --2- 172.21.15.16:0/2622872093 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbecc106680 0x7fbecc106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.299+0000 7fbed42f4640 1 -- 172.21.15.16:0/2622872093 >> 172.21.15.16:0/2622872093 conn(0x7fbecc077770 msgr2=0x7fbecc077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:16.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.299+0000 7fbed42f4640 1 -- 172.21.15.16:0/2622872093 shutdown_connections 2024-03-28T22:15:16.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.299+0000 7fbed42f4640 1 -- 172.21.15.16:0/2622872093 wait complete. 2024-03-28T22:15:16.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.300+0000 7fbed42f4640 1 Processor -- start 2024-03-28T22:15:16.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.300+0000 7fbed42f4640 1 -- start start 2024-03-28T22:15:16.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.300+0000 7fbed42f4640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbecc1005e0 0x7fbecc196f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.300+0000 7fbed42f4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbecc100fa0 0x7fbecc197480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.300+0000 7fbed42f4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbecc106680 0x7fbecc19b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.300+0000 7fbed42f4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc1109c0 con 0x7fbecc100fa0 2024-03-28T22:15:16.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.300+0000 7fbed42f4640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc110840 con 0x7fbecc1005e0 2024-03-28T22:15:16.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.300+0000 7fbed42f4640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc110b40 con 0x7fbecc106680 2024-03-28T22:15:16.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.300+0000 7fbed188f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbecc100fa0 0x7fbecc197480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.300+0000 7fbed2090640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbecc1005e0 0x7fbecc196f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.300+0000 7fbed2090640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbecc1005e0 0x7fbecc196f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:51506/0 (socket says 172.21.15.16:51506) 2024-03-28T22:15:16.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.300+0000 7fbed188f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbecc100fa0 0x7fbecc197480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59400/0 (socket says 172.21.15.16:59400) 2024-03-28T22:15:16.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.300+0000 7fbed2090640 1 -- 172.21.15.16:0/2902480774 learned_addr learned my addr 172.21.15.16:0/2902480774 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:16.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.300+0000 7fbed2891640 1 --2- 172.21.15.16:0/2902480774 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbecc106680 0x7fbecc19b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.301+0000 7fbed188f640 1 -- 172.21.15.16:0/2902480774 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbecc106680 msgr2=0x7fbecc19b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:16.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.301+0000 7fbed188f640 1 --2- 172.21.15.16:0/2902480774 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbecc106680 0x7fbecc19b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.301+0000 7fbed188f640 1 -- 172.21.15.16:0/2902480774 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbecc1005e0 msgr2=0x7fbecc196f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:16.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.301+0000 7fbed188f640 1 --2- 172.21.15.16:0/2902480774 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbecc1005e0 0x7fbecc196f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.301+0000 7fbed188f640 1 -- 172.21.15.16:0/2902480774 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbecc19bf30 con 0x7fbecc100fa0 2024-03-28T22:15:16.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.301+0000 7fbed188f640 1 --2- 172.21.15.16:0/2902480774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbecc100fa0 0x7fbecc197480 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fbebc009230 tx=0x7fbebc02cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:16.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.301+0000 7fbec37fe640 1 -- 172.21.15.16:0/2902480774 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbebc041070 con 0x7fbecc100fa0 2024-03-28T22:15:16.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.301+0000 7fbed42f4640 1 -- 172.21.15.16:0/2902480774 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbecc19c1c0 con 0x7fbecc100fa0 2024-03-28T22:15:16.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.301+0000 7fbec37fe640 1 -- 172.21.15.16:0/2902480774 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbebc004450 con 0x7fbecc100fa0 2024-03-28T22:15:16.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.301+0000 7fbec37fe640 1 -- 172.21.15.16:0/2902480774 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbebc003df0 con 0x7fbecc100fa0 2024-03-28T22:15:16.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.301+0000 7fbed42f4640 1 -- 172.21.15.16:0/2902480774 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbecc069df0 con 0x7fbecc100fa0 2024-03-28T22:15:16.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.303+0000 7fbec37fe640 1 -- 172.21.15.16:0/2902480774 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7fbebc044070 con 0x7fbecc100fa0 2024-03-28T22:15:16.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.303+0000 7fbed6fb1740 1 Processor -- start 2024-03-28T22:15:16.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.304+0000 7fbed6fb1740 1 -- start start 2024-03-28T22:15:16.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.304+0000 7fbed6fb1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55742f190470 0x55742f190850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.304+0000 7fbed6fb1740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55742f190d90 0x55742f199430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.304+0000 7fbed6fb1740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55742f199970 0x55742f19bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.304+0000 7fbed6fb1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55742f0f41b0 con 0x55742f190470 2024-03-28T22:15:16.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.304+0000 7fbed6fb1740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55742f09cc00 con 0x55742f190d90 2024-03-28T22:15:16.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.304+0000 7fbed6fb1740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55742f0f0250 con 0x55742f199970 2024-03-28T22:15:16.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.304+0000 7fbec37fe640 1 --2- 172.21.15.16:0/2902480774 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fbeb80a1ef0 0x7fbeb80a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.304+0000 7fbed2891640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55742f190470 0x55742f190850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.304+0000 7fbed2090640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55742f190d90 0x55742f199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.304+0000 7fbed2891640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55742f190470 0x55742f190850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59402/0 (socket says 172.21.15.16:59402) 2024-03-28T22:15:16.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.304+0000 7fbed2090640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55742f190d90 0x55742f199430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:51514/0 (socket says 172.21.15.16:51514) 2024-03-28T22:15:16.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.304+0000 7fbed2891640 1 -- 172.21.15.16:0/3313820900 learned_addr learned my addr 172.21.15.16:0/3313820900 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:16.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.304+0000 7fbed2891640 1 --2- 172.21.15.16:0/2902480774 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fbeb80a1ef0 0x7fbeb80a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.304+0000 7fbec37fe640 1 -- 172.21.15.16:0/2902480774 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fbebc0efd40 con 0x7fbecc100fa0 2024-03-28T22:15:16.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.304+0000 7fbed188f640 1 --2- 172.21.15.16:0/3313820900 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55742f199970 0x55742f19bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.304+0000 7fbed2891640 1 -- 172.21.15.16:0/3313820900 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55742f199970 msgr2=0x55742f19bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:16.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.304+0000 7fbed2891640 1 --2- 172.21.15.16:0/3313820900 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55742f199970 0x55742f19bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.305+0000 7fbed2891640 1 -- 172.21.15.16:0/3313820900 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55742f190d90 msgr2=0x55742f199430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:16.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.305+0000 7fbed2891640 1 --2- 172.21.15.16:0/3313820900 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55742f190d90 0x55742f199430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.305+0000 7fbed2891640 1 -- 172.21.15.16:0/3313820900 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55742f0bc3d0 con 0x55742f190470 2024-03-28T22:15:16.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.305+0000 7fbed2090640 1 --2- 172.21.15.16:0/3313820900 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55742f190d90 0x55742f199430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:15:16.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.305+0000 7fbed2891640 1 --2- 172.21.15.16:0/3313820900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55742f190470 0x55742f190850 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fbec4011e20 tx=0x7fbec400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22107e7a497da381 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:16.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.305+0000 7fbed2891640 1 --2- 172.21.15.16:0/2902480774 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fbeb80a1ef0 0x7fbeb80a43b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fbec400fc90 tx=0x7fbec401e000 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:16.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.305+0000 7fbec0bfe640 1 -- 172.21.15.16:0/3313820900 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec401dbf0 con 0x55742f190470 2024-03-28T22:15:16.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.305+0000 7fbec0bfe640 1 -- 172.21.15.16:0/3313820900 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbec401dd90 con 0x55742f190470 2024-03-28T22:15:16.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.305+0000 7fbec0bfe640 1 -- 172.21.15.16:0/3313820900 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec4004290 con 0x55742f190470 2024-03-28T22:15:16.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.306+0000 7fbed6fb1740 1 -- 172.21.15.16:0/3313820900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55742f190470 msgr2=0x55742f190850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:16.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.306+0000 7fbed6fb1740 1 --2- 172.21.15.16:0/3313820900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55742f190470 0x55742f190850 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fbec4011e20 tx=0x7fbec400f790 comp rx=0 tx=0).stop 2024-03-28T22:15:16.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.306+0000 7fbed6fb1740 1 -- 172.21.15.16:0/3313820900 shutdown_connections 2024-03-28T22:15:16.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.306+0000 7fbed6fb1740 1 --2- 172.21.15.16:0/3313820900 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55742f199970 0x55742f19bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.306+0000 7fbed6fb1740 1 --2- 172.21.15.16:0/3313820900 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55742f190d90 0x55742f199430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.306+0000 7fbed6fb1740 1 --2- 172.21.15.16:0/3313820900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55742f190470 0x55742f190850 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.306+0000 7fbed6fb1740 1 -- 172.21.15.16:0/3313820900 >> 172.21.15.16:0/3313820900 conn(0x55742f18aba0 msgr2=0x55742f18af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:16.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.306+0000 7fbed6fb1740 1 -- 172.21.15.16:0/3313820900 shutdown_connections 2024-03-28T22:15:16.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.306+0000 7fbed6fb1740 1 -- 172.21.15.16:0/3313820900 wait complete. 2024-03-28T22:15:16.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.306+0000 7fbed6fb1740 1 -- start start 2024-03-28T22:15:16.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.306+0000 7fbed6fb1740 1 Processor -- start 2024-03-28T22:15:16.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.306+0000 7fbed6fb1740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55742f190470 0x55742f196a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.306+0000 7fbed6fb1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55742f190d90 0x55742f196f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.306+0000 7fbed6fb1740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55742f199970 0x55742f1974d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.306+0000 7fbed6fb1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55742f19c6a0 con 0x55742f190d90 2024-03-28T22:15:16.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.306+0000 7fbed6fb1740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55742f092470 con 0x55742f190470 2024-03-28T22:15:16.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.306+0000 7fbed6fb1740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55742f19c820 con 0x55742f199970 2024-03-28T22:15:16.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.306+0000 7fbed6fb1740 10 client.0 _ll_register_callbacks cb 0x55742f0afcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:15:16.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.307+0000 7fbed2090640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55742f190470 0x55742f196a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.307+0000 7fbed188f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55742f190d90 0x55742f196f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.307+0000 7fbed2090640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55742f190470 0x55742f196a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:51516/0 (socket says 172.21.15.16:51516) 2024-03-28T22:15:16.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.307+0000 7fbed2090640 1 -- 172.21.15.16:0/1185191351 learned_addr learned my addr 172.21.15.16:0/1185191351 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:16.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.307+0000 7fbed2891640 1 --2- 172.21.15.16:0/1185191351 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55742f199970 0x55742f1974d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.307+0000 7fbed188f640 1 -- 172.21.15.16:0/1185191351 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55742f199970 msgr2=0x55742f1974d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:16.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.307+0000 7fbed188f640 1 --2- 172.21.15.16:0/1185191351 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55742f199970 0x55742f1974d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.307+0000 7fbed188f640 1 -- 172.21.15.16:0/1185191351 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55742f190470 msgr2=0x55742f196a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:16.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.307+0000 7fbed188f640 1 --2- 172.21.15.16:0/1185191351 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55742f190470 0x55742f196a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.307+0000 7fbed188f640 1 -- 172.21.15.16:0/1185191351 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55742ee083f0 con 0x55742f190d90 2024-03-28T22:15:16.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.307+0000 7fbed2090640 1 --2- 172.21.15.16:0/1185191351 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55742f190470 0x55742f196a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:15:16.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.307+0000 7fbed188f640 1 --2- 172.21.15.16:0/1185191351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55742f190d90 0x55742f196f90 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fbebc0ed5c0 tx=0x7fbebc009620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8036ac710a0dfab9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:16.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.307+0000 7fbeb5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:15:16.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.307+0000 7fbeb5ffb640 1 -- 172.21.15.16:0/1185191351 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbebc04b040 con 0x55742f190d90 2024-03-28T22:15:16.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.307+0000 7fbeb5ffb640 1 -- 172.21.15.16:0/1185191351 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fbebc0d5290 con 0x55742f190d90 2024-03-28T22:15:16.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.307+0000 7fbed6fb1740 1 -- 172.21.15.16:0/1185191351 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbec401dd90 con 0x55742f190d90 2024-03-28T22:15:16.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.307+0000 7fbeb5ffb640 1 -- 172.21.15.16:0/1185191351 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbebc0bc270 con 0x55742f190d90 2024-03-28T22:15:16.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.307+0000 7fbed6fb1740 10 client.16470 fetch_fsmap learned FSMap version 27 2024-03-28T22:15:16.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.307+0000 7fbed6fb1740 1 -- 172.21.15.16:0/1185191351 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55742eed27d0 con 0x55742f190d90 2024-03-28T22:15:16.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.308+0000 7fbeb5ffb640 1 -- 172.21.15.16:0/1185191351 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbec401dd90 con 0x55742f190d90 2024-03-28T22:15:16.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.308+0000 7fbeb5ffb640 1 -- 172.21.15.16:0/1185191351 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7fbebc0a70b0 con 0x55742f190d90 2024-03-28T22:15:16.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.308+0000 7fbed6fb1740 10 client.16470 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:15:16.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.308+0000 7fbed6fb1740 10 client.16470 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-28T22:15:16.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.308+0000 7fbed6fb1740 20 client.16470 populate_metadata read hostname 'smithi016' 2024-03-28T22:15:16.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.308+0000 7fbed6fb1740 1 --2- 172.21.15.16:0/1185191351 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55742f1a9840 0x55742f1abc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.308+0000 7fbed6fb1740 4 client.16470 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:15:16.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.308+0000 7fbed6fb1740 1 -- 172.21.15.16:0/1185191351 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55742f1016f0 con 0x55742f1a9840 2024-03-28T22:15:16.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.308+0000 7fbed2090640 1 --2- 172.21.15.16:0/1185191351 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55742f1a9840 0x55742f1abc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.308+0000 7fbeb5ffb640 10 client.16470 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:15:16.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.308+0000 7fbed2090640 1 --2- 172.21.15.16:0/1185191351 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55742f1a9840 0x55742f1abc50 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=acd5b455d110e156 server_cookie=c02790834a662475 in_seq=0 out_seq=0 2024-03-28T22:15:16.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.309+0000 7fbeb5ffb640 1 -- 172.21.15.16:0/1185191351 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55742f1016f0 con 0x55742f1a9840 2024-03-28T22:15:16.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.309+0000 7fbeb5ffb640 10 client.16470 handle_command_reply: tid=0 2024-03-28T22:15:16.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.309+0000 7fbed6fb1740 1 client.16470 shutdown 2024-03-28T22:15:16.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.310+0000 7fbed6fb1740 1 -- 172.21.15.16:0/1185191351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55742f190d90 msgr2=0x55742f196f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:16.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.310+0000 7fbed6fb1740 1 --2- 172.21.15.16:0/1185191351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55742f190d90 0x55742f196f90 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fbebc0ed5c0 tx=0x7fbebc009620 comp rx=0 tx=0).stop 2024-03-28T22:15:16.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.310+0000 7fbed6fb1740 1 -- 172.21.15.16:0/1185191351 shutdown_connections 2024-03-28T22:15:16.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.310+0000 7fbed6fb1740 1 --2- 172.21.15.16:0/1185191351 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55742f1a9840 0x55742f1abc50 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.310+0000 7fbed6fb1740 1 --2- 172.21.15.16:0/1185191351 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55742f199970 0x55742f1974d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.310+0000 7fbed6fb1740 1 --2- 172.21.15.16:0/1185191351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55742f190d90 0x55742f196f90 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.310+0000 7fbeb5ffb640 0 client.16470 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:15:16.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.310+0000 7fbed6fb1740 1 --2- 172.21.15.16:0/1185191351 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55742f190470 0x55742f196a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.310+0000 7fbed6fb1740 1 -- 172.21.15.16:0/1185191351 >> 172.21.15.16:0/1185191351 conn(0x55742f18aba0 msgr2=0x55742f18c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:16.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.310+0000 7fbed6fb1740 1 -- 172.21.15.16:0/1185191351 shutdown_connections 2024-03-28T22:15:16.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.310+0000 7fbed6fb1740 1 -- 172.21.15.16:0/1185191351 wait complete. 2024-03-28T22:15:16.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.310+0000 7fbed6fb1740 20 client.16470 trim_cache size 0 max 16384 2024-03-28T22:15:16.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.323+0000 7fbed6fb1740 1 Processor -- start 2024-03-28T22:15:16.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.323+0000 7fbed6fb1740 1 -- start start 2024-03-28T22:15:16.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.323+0000 7fbed6fb1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55742f190470 0x55742f1a5440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.323+0000 7fbed6fb1740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55742f190d90 0x55742f1b1530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.323+0000 7fbed6fb1740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55742f199970 0x55742f189bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.323+0000 7fbed6fb1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc1ada10 con 0x55742f190470 2024-03-28T22:15:16.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.323+0000 7fbed6fb1740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc1ab2e0 con 0x55742f199970 2024-03-28T22:15:16.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.323+0000 7fbed6fb1740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc1a9130 con 0x55742f190d90 2024-03-28T22:15:16.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.323+0000 7fbed2090640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55742f190470 0x55742f1a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.323+0000 7fbed2090640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55742f190470 0x55742f1a5440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59434/0 (socket says 172.21.15.16:59434) 2024-03-28T22:15:16.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.323+0000 7fbed2090640 1 -- 172.21.15.16:0/2938414232 learned_addr learned my addr 172.21.15.16:0/2938414232 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:16.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.323+0000 7fbed2891640 1 --2- 172.21.15.16:0/2938414232 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55742f199970 0x55742f189bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.324+0000 7fbed2090640 1 -- 172.21.15.16:0/2938414232 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55742f190d90 msgr2=0x55742f1b1530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:16.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.324+0000 7fbed2090640 1 --2- 172.21.15.16:0/2938414232 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55742f190d90 0x55742f1b1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.324+0000 7fbed2090640 1 -- 172.21.15.16:0/2938414232 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55742f199970 msgr2=0x55742f189bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:16.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.324+0000 7fbed2090640 1 --2- 172.21.15.16:0/2938414232 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55742f199970 0x55742f189bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.324+0000 7fbed2090640 1 -- 172.21.15.16:0/2938414232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55742f1ac2a0 con 0x55742f190470 2024-03-28T22:15:16.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.324+0000 7fbed2090640 1 --2- 172.21.15.16:0/2938414232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55742f190470 0x55742f1a5440 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x55742f1ac470 tx=0x7fbec80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a7d8b2ea6850e7a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:16.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.324+0000 7fbeb7fff640 1 -- 172.21.15.16:0/2938414232 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec8016d00 con 0x55742f190470 2024-03-28T22:15:16.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.324+0000 7fbeb7fff640 1 -- 172.21.15.16:0/2938414232 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbec8027070 con 0x55742f190470 2024-03-28T22:15:16.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.324+0000 7fbeb7fff640 1 -- 172.21.15.16:0/2938414232 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec800ca20 con 0x55742f190470 2024-03-28T22:15:16.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.324+0000 7fbed6fb1740 1 -- 172.21.15.16:0/2938414232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55742f190470 msgr2=0x55742f1a5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:16.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.324+0000 7fbed6fb1740 1 --2- 172.21.15.16:0/2938414232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55742f190470 0x55742f1a5440 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x55742f1ac470 tx=0x7fbec80046c0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.324+0000 7fbed6fb1740 1 -- 172.21.15.16:0/2938414232 shutdown_connections 2024-03-28T22:15:16.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.324+0000 7fbed6fb1740 1 --2- 172.21.15.16:0/2938414232 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55742f199970 0x55742f189bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.324+0000 7fbed6fb1740 1 --2- 172.21.15.16:0/2938414232 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55742f190d90 0x55742f1b1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.324+0000 7fbed6fb1740 1 --2- 172.21.15.16:0/2938414232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55742f190470 0x55742f1a5440 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.325+0000 7fbed6fb1740 1 -- 172.21.15.16:0/2938414232 >> 172.21.15.16:0/2938414232 conn(0x55742f18aba0 msgr2=0x55742f1894b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:16.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.325+0000 7fbed6fb1740 1 -- 172.21.15.16:0/2938414232 shutdown_connections 2024-03-28T22:15:16.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.325+0000 7fbed6fb1740 1 -- 172.21.15.16:0/2938414232 wait complete. 2024-03-28T22:15:16.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.325+0000 7fbed6fb1740 1 -- start start 2024-03-28T22:15:16.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.325+0000 7fbed6fb1740 1 Processor -- start 2024-03-28T22:15:16.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.325+0000 7fbed6fb1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55742f190470 0x55742f1a1a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.325+0000 7fbed6fb1740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55742f190d90 0x55742f19ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.325+0000 7fbed6fb1740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55742f199970 0x55742f19ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.325+0000 7fbed6fb1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc1a97a0 con 0x55742f190470 2024-03-28T22:15:16.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.325+0000 7fbed6fb1740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc1aac10 con 0x55742f199970 2024-03-28T22:15:16.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.325+0000 7fbed6fb1740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc1a9e70 con 0x55742f190d90 2024-03-28T22:15:16.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.325+0000 7fbed6fb1740 10 client.0 _ll_register_callbacks cb 0x7fbebc005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:15:16.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.325+0000 7fbed2090640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55742f190470 0x55742f1a1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.325+0000 7fbed2090640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55742f190470 0x55742f1a1a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59450/0 (socket says 172.21.15.16:59450) 2024-03-28T22:15:16.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.325+0000 7fbed2090640 1 -- 172.21.15.16:0/432014807 learned_addr learned my addr 172.21.15.16:0/432014807 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:16.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.325+0000 7fbed2891640 1 --2- 172.21.15.16:0/432014807 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55742f199970 0x55742f19ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.325+0000 7fbed188f640 1 --2- 172.21.15.16:0/432014807 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55742f190d90 0x55742f19ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.325+0000 7fbed2090640 1 -- 172.21.15.16:0/432014807 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55742f190d90 msgr2=0x55742f19ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:16.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.325+0000 7fbed2090640 1 --2- 172.21.15.16:0/432014807 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55742f190d90 0x55742f19ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.325+0000 7fbed2090640 1 -- 172.21.15.16:0/432014807 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55742f199970 msgr2=0x55742f19ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:16.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.325+0000 7fbed2090640 1 --2- 172.21.15.16:0/432014807 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55742f199970 0x55742f19ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.325+0000 7fbed2090640 1 -- 172.21.15.16:0/432014807 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55742f19f4c0 con 0x55742f190470 2024-03-28T22:15:16.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.325+0000 7fbed188f640 1 --2- 172.21.15.16:0/432014807 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55742f190d90 0x55742f19ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:15:16.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.326+0000 7fbed2090640 1 --2- 172.21.15.16:0/432014807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55742f190470 0x55742f1a1a70 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fbec8005c50 tx=0x7fbec800d3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9e3e048b7a6a9e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:16.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.326+0000 7fbeb5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:15:16.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.326+0000 7fbeb5ffb640 1 -- 172.21.15.16:0/432014807 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec8021310 con 0x55742f190470 2024-03-28T22:15:16.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.326+0000 7fbeb5ffb640 1 -- 172.21.15.16:0/432014807 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fbec80214b0 con 0x55742f190470 2024-03-28T22:15:16.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.326+0000 7fbed6fb1740 1 -- 172.21.15.16:0/432014807 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbec8027070 con 0x55742f190470 2024-03-28T22:15:16.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.326+0000 7fbeb5ffb640 1 -- 172.21.15.16:0/432014807 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbec803b050 con 0x55742f190470 2024-03-28T22:15:16.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.326+0000 7fbed6fb1740 10 client.16476 fetch_fsmap learned FSMap version 27 2024-03-28T22:15:16.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.326+0000 7fbed6fb1740 1 -- 172.21.15.16:0/432014807 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55742f2108b0 con 0x55742f190470 2024-03-28T22:15:16.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.326+0000 7fbeb5ffb640 1 -- 172.21.15.16:0/432014807 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbec8027070 con 0x55742f190470 2024-03-28T22:15:16.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.326+0000 7fbeb5ffb640 1 -- 172.21.15.16:0/432014807 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7fbec802d460 con 0x55742f190470 2024-03-28T22:15:16.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.326+0000 7fbed6fb1740 10 client.16476 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:15:16.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.326+0000 7fbed6fb1740 10 client.16476 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-28T22:15:16.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.326+0000 7fbed6fb1740 20 client.16476 populate_metadata read hostname 'smithi016' 2024-03-28T22:15:16.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.326+0000 7fbed6fb1740 1 --2- 172.21.15.16:0/432014807 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55742f210fd0 0x55742f2133f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.326+0000 7fbed6fb1740 4 client.16476 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:15:16.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.326+0000 7fbed6fb1740 1 -- 172.21.15.16:0/432014807 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55742f0fba40 con 0x55742f210fd0 2024-03-28T22:15:16.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.327+0000 7fbed188f640 1 --2- 172.21.15.16:0/432014807 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55742f210fd0 0x55742f2133f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.327+0000 7fbeb5ffb640 10 client.16476 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:15:16.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.327+0000 7fbed188f640 1 --2- 172.21.15.16:0/432014807 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55742f210fd0 0x55742f2133f0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=425ef163c4f8fbf0 server_cookie=190e4bba0ee26d1e in_seq=0 out_seq=0 2024-03-28T22:15:16.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.327+0000 7fbeb5ffb640 1 -- 172.21.15.16:0/432014807 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55742f0fba40 con 0x55742f210fd0 2024-03-28T22:15:16.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.327+0000 7fbeb5ffb640 10 client.16476 handle_command_reply: tid=0 2024-03-28T22:15:16.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.327+0000 7fbed6fb1740 1 client.16476 shutdown 2024-03-28T22:15:16.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.327+0000 7fbed6fb1740 1 -- 172.21.15.16:0/432014807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55742f190470 msgr2=0x55742f1a1a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:16.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.327+0000 7fbed6fb1740 1 --2- 172.21.15.16:0/432014807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55742f190470 0x55742f1a1a70 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fbec8005c50 tx=0x7fbec800d3d0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.327+0000 7fbed6fb1740 1 -- 172.21.15.16:0/432014807 shutdown_connections 2024-03-28T22:15:16.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.327+0000 7fbed6fb1740 1 --2- 172.21.15.16:0/432014807 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55742f210fd0 0x55742f2133f0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.327+0000 7fbed6fb1740 1 --2- 172.21.15.16:0/432014807 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55742f199970 0x55742f19ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.327+0000 7fbed6fb1740 1 --2- 172.21.15.16:0/432014807 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55742f190d90 0x55742f19ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.327+0000 7fbed6fb1740 1 --2- 172.21.15.16:0/432014807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55742f190470 0x55742f1a1a70 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.327+0000 7fbeb5ffb640 0 client.16476 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:15:16.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.327+0000 7fbed6fb1740 1 -- 172.21.15.16:0/432014807 >> 172.21.15.16:0/432014807 conn(0x55742f18aba0 msgr2=0x55742f1969f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:16.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.327+0000 7fbed6fb1740 1 -- 172.21.15.16:0/432014807 shutdown_connections 2024-03-28T22:15:16.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.327+0000 7fbed6fb1740 1 -- 172.21.15.16:0/432014807 wait complete. 2024-03-28T22:15:16.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.327+0000 7fbed6fb1740 20 client.16476 trim_cache size 0 max 16384 2024-03-28T22:15:16.343 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:15:16.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.328+0000 7fbed42f4640 1 -- 172.21.15.16:0/2902480774 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fbeb80a1ef0 msgr2=0x7fbeb80a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:16.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.328+0000 7fbed42f4640 1 --2- 172.21.15.16:0/2902480774 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fbeb80a1ef0 0x7fbeb80a43b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fbec400fc90 tx=0x7fbec401e000 comp rx=0 tx=0).stop 2024-03-28T22:15:16.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.328+0000 7fbed42f4640 1 -- 172.21.15.16:0/2902480774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbecc100fa0 msgr2=0x7fbecc197480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:16.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.328+0000 7fbed42f4640 1 --2- 172.21.15.16:0/2902480774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbecc100fa0 0x7fbecc197480 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fbebc009230 tx=0x7fbebc02cbf0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.328+0000 7fbed42f4640 1 -- 172.21.15.16:0/2902480774 shutdown_connections 2024-03-28T22:15:16.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.328+0000 7fbed42f4640 1 --2- 172.21.15.16:0/2902480774 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fbeb80a1ef0 0x7fbeb80a43b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.328+0000 7fbed42f4640 1 --2- 172.21.15.16:0/2902480774 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbecc106680 0x7fbecc19b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.328+0000 7fbed42f4640 1 --2- 172.21.15.16:0/2902480774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbecc100fa0 0x7fbecc197480 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.328+0000 7fbed42f4640 1 --2- 172.21.15.16:0/2902480774 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbecc1005e0 0x7fbecc196f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.328+0000 7fbed42f4640 1 -- 172.21.15.16:0/2902480774 >> 172.21.15.16:0/2902480774 conn(0x7fbecc077770 msgr2=0x7fbecc0ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:16.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.328+0000 7fbed42f4640 1 -- 172.21.15.16:0/2902480774 shutdown_connections 2024-03-28T22:15:16.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.328+0000 7fbed42f4640 1 -- 172.21.15.16:0/2902480774 wait complete. 2024-03-28T22:15:16.346 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:15:16.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.449+0000 7f0316dbc640 1 -- 172.21.15.16:0/3481330456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0310103290 msgr2=0x7f031010b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:16.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.449+0000 7f0316dbc640 1 --2- 172.21.15.16:0/3481330456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0310103290 0x7f031010b770 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f02fc009900 tx=0x7f02fc02bcd0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.449+0000 7f0316dbc640 1 -- 172.21.15.16:0/3481330456 shutdown_connections 2024-03-28T22:15:16.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.449+0000 7f0316dbc640 1 --2- 172.21.15.16:0/3481330456 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f031010bd40 0x7f031010e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.449+0000 7f0316dbc640 1 --2- 172.21.15.16:0/3481330456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0310103290 0x7f031010b770 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.449+0000 7f0316dbc640 1 --2- 172.21.15.16:0/3481330456 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0310102970 0x7f0310102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.449+0000 7f0316dbc640 1 -- 172.21.15.16:0/3481330456 >> 172.21.15.16:0/3481330456 conn(0x7f0310077770 msgr2=0x7f0310077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:16.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.449+0000 7f0316dbc640 1 -- 172.21.15.16:0/3481330456 shutdown_connections 2024-03-28T22:15:16.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.449+0000 7f0316dbc640 1 -- 172.21.15.16:0/3481330456 wait complete. 2024-03-28T22:15:16.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.450+0000 7f0316dbc640 1 Processor -- start 2024-03-28T22:15:16.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.450+0000 7f0316dbc640 1 -- start start 2024-03-28T22:15:16.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.450+0000 7f0316dbc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0310102970 0x7f0310198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.450+0000 7f0316dbc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0310103290 0x7f0310198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.450+0000 7f0316dbc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f031010bd40 0x7f031019c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.450+0000 7f0316dbc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0310111ad0 con 0x7f0310102970 2024-03-28T22:15:16.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.450+0000 7f0316dbc640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0310111950 con 0x7f0310103290 2024-03-28T22:15:16.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.450+0000 7f0316dbc640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0310111c50 con 0x7f031010bd40 2024-03-28T22:15:16.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.450+0000 7f0314b58640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0310102970 0x7f0310198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.450+0000 7f0314b58640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0310102970 0x7f0310198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59462/0 (socket says 172.21.15.16:59462) 2024-03-28T22:15:16.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.450+0000 7f0314b58640 1 -- 172.21.15.16:0/3830202006 learned_addr learned my addr 172.21.15.16:0/3830202006 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:16.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.450+0000 7f030ffff640 1 --2- 172.21.15.16:0/3830202006 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0310103290 0x7f0310198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.450+0000 7f0315359640 1 --2- 172.21.15.16:0/3830202006 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f031010bd40 0x7f031019c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.450+0000 7f0314b58640 1 -- 172.21.15.16:0/3830202006 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f031010bd40 msgr2=0x7f031019c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:16.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.450+0000 7f0314b58640 1 --2- 172.21.15.16:0/3830202006 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f031010bd40 0x7f031019c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.450+0000 7f0314b58640 1 -- 172.21.15.16:0/3830202006 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0310103290 msgr2=0x7f0310198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:16.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.450+0000 7f0314b58640 1 --2- 172.21.15.16:0/3830202006 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0310103290 0x7f0310198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.450+0000 7f0314b58640 1 -- 172.21.15.16:0/3830202006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f031019d020 con 0x7f0310102970 2024-03-28T22:15:16.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.450+0000 7f0315359640 1 --2- 172.21.15.16:0/3830202006 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f031010bd40 0x7f031019c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:15:16.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.451+0000 7f0314b58640 1 --2- 172.21.15.16:0/3830202006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0310102970 0x7f0310198030 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f030400e9f0 tx=0x7f030400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:16.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.451+0000 7f030dffb640 1 -- 172.21.15.16:0/3830202006 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f030400cd80 con 0x7f0310102970 2024-03-28T22:15:16.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.451+0000 7f030dffb640 1 -- 172.21.15.16:0/3830202006 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0304005500 con 0x7f0310102970 2024-03-28T22:15:16.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.451+0000 7f0316dbc640 1 -- 172.21.15.16:0/3830202006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f031019d310 con 0x7f0310102970 2024-03-28T22:15:16.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.451+0000 7f030dffb640 1 -- 172.21.15.16:0/3830202006 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0304010690 con 0x7f0310102970 2024-03-28T22:15:16.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.451+0000 7f0316dbc640 1 -- 172.21.15.16:0/3830202006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03101a4ba0 con 0x7f0310102970 2024-03-28T22:15:16.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.453+0000 7f030dffb640 1 -- 172.21.15.16:0/3830202006 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f03040046c0 con 0x7f0310102970 2024-03-28T22:15:16.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.453+0000 7f031999f740 1 Processor -- start 2024-03-28T22:15:16.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.453+0000 7f031999f740 1 -- start start 2024-03-28T22:15:16.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.454+0000 7f030dffb640 1 --2- 172.21.15.16:0/3830202006 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f03000a1ef0 0x7f03000a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.454+0000 7f031999f740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec315c6470 0x55ec315c6850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.454+0000 7f031999f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec315c6d90 0x55ec315cf430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.454+0000 7f031999f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ec315cf970 0x55ec315d1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.454+0000 7f031999f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ec3152a1b0 con 0x55ec315c6d90 2024-03-28T22:15:16.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.454+0000 7f031999f740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55ec314d2c00 con 0x55ec315c6470 2024-03-28T22:15:16.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.454+0000 7f031999f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55ec31526250 con 0x55ec315cf970 2024-03-28T22:15:16.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.454+0000 7f0315359640 1 --2- 172.21.15.16:0/3830202006 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f03000a1ef0 0x7f03000a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.454+0000 7f030ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec315c6d90 0x55ec315cf430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.454+0000 7f0314b58640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec315c6470 0x55ec315c6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.454+0000 7f030ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec315c6d90 0x55ec315cf430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59476/0 (socket says 172.21.15.16:59476) 2024-03-28T22:15:16.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.454+0000 7f030ffff640 1 -- 172.21.15.16:0/607823106 learned_addr learned my addr 172.21.15.16:0/607823106 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:16.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.454+0000 7f030dffb640 1 -- 172.21.15.16:0/3830202006 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f03040ce640 con 0x7f0310102970 2024-03-28T22:15:16.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.454+0000 7f0315359640 1 --2- 172.21.15.16:0/607823106 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ec315cf970 0x55ec315d1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.454+0000 7f030ffff640 1 -- 172.21.15.16:0/607823106 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ec315cf970 msgr2=0x55ec315d1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:16.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.454+0000 7f030ffff640 1 --2- 172.21.15.16:0/607823106 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ec315cf970 0x55ec315d1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.454+0000 7f030ffff640 1 -- 172.21.15.16:0/607823106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec315c6470 msgr2=0x55ec315c6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:16.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.454+0000 7f030ffff640 1 --2- 172.21.15.16:0/607823106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec315c6470 0x55ec315c6850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.454+0000 7f030ffff640 1 -- 172.21.15.16:0/607823106 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec314f23d0 con 0x55ec315c6d90 2024-03-28T22:15:16.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.454+0000 7f0315359640 1 --2- 172.21.15.16:0/3830202006 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f03000a1ef0 0x7f03000a43b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f031006aed0 tx=0x7f0308012b30 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:16.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.454+0000 7f030ffff640 1 --2- 172.21.15.16:0/607823106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec315c6d90 0x55ec315cf430 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f02fc009230 tx=0x7f02fc0095f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d8886582636e180 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:16.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.454+0000 7f02faffd640 1 -- 172.21.15.16:0/607823106 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02fc042070 con 0x55ec315c6d90 2024-03-28T22:15:16.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.455+0000 7f02faffd640 1 -- 172.21.15.16:0/607823106 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02fc004040 con 0x55ec315c6d90 2024-03-28T22:15:16.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.455+0000 7f02faffd640 1 -- 172.21.15.16:0/607823106 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02fc004c00 con 0x55ec315c6d90 2024-03-28T22:15:16.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.455+0000 7f031999f740 1 -- 172.21.15.16:0/607823106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec315c6d90 msgr2=0x55ec315cf430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:16.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.455+0000 7f031999f740 1 --2- 172.21.15.16:0/607823106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec315c6d90 0x55ec315cf430 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f02fc009230 tx=0x7f02fc0095f0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.455+0000 7f031999f740 1 -- 172.21.15.16:0/607823106 shutdown_connections 2024-03-28T22:15:16.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.455+0000 7f031999f740 1 --2- 172.21.15.16:0/607823106 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ec315cf970 0x55ec315d1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.455+0000 7f031999f740 1 --2- 172.21.15.16:0/607823106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec315c6d90 0x55ec315cf430 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.455+0000 7f031999f740 1 --2- 172.21.15.16:0/607823106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec315c6470 0x55ec315c6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.455+0000 7f031999f740 1 -- 172.21.15.16:0/607823106 >> 172.21.15.16:0/607823106 conn(0x55ec315c0ba0 msgr2=0x55ec315c0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:16.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.455+0000 7f031999f740 1 -- 172.21.15.16:0/607823106 shutdown_connections 2024-03-28T22:15:16.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.455+0000 7f031999f740 1 -- 172.21.15.16:0/607823106 wait complete. 2024-03-28T22:15:16.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.456+0000 7f031999f740 1 -- start start 2024-03-28T22:15:16.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.456+0000 7f031999f740 1 Processor -- start 2024-03-28T22:15:16.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.456+0000 7f031999f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec315c6470 0x55ec315cc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.456+0000 7f031999f740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec315c6d90 0x55ec315ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.456+0000 7f031999f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ec315cf970 0x55ec315cd420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.456+0000 7f031999f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ec315d2700 con 0x55ec315c6470 2024-03-28T22:15:16.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.456+0000 7f031999f740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55ec314c8470 con 0x55ec315c6d90 2024-03-28T22:15:16.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.456+0000 7f031999f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55ec315d2880 con 0x55ec315cf970 2024-03-28T22:15:16.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.456+0000 7f031999f740 10 client.0 _ll_register_callbacks cb 0x55ec314e5cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:15:16.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.456+0000 7f0314b58640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec315c6470 0x55ec315cc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.456+0000 7f0314b58640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec315c6470 0x55ec315cc9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59484/0 (socket says 172.21.15.16:59484) 2024-03-28T22:15:16.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.456+0000 7f0314b58640 1 -- 172.21.15.16:0/3248807409 learned_addr learned my addr 172.21.15.16:0/3248807409 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:16.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.456+0000 7f030ffff640 1 --2- 172.21.15.16:0/3248807409 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec315c6d90 0x55ec315ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.456+0000 7f0315359640 1 --2- 172.21.15.16:0/3248807409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ec315cf970 0x55ec315cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.456+0000 7f0314b58640 1 -- 172.21.15.16:0/3248807409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ec315cf970 msgr2=0x55ec315cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:16.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.456+0000 7f0314b58640 1 --2- 172.21.15.16:0/3248807409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ec315cf970 0x55ec315cd420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.456+0000 7f0314b58640 1 -- 172.21.15.16:0/3248807409 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec315c6d90 msgr2=0x55ec315ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:16.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.456+0000 7f0314b58640 1 --2- 172.21.15.16:0/3248807409 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec315c6d90 0x55ec315ccee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.456+0000 7f0314b58640 1 -- 172.21.15.16:0/3248807409 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec3123e3f0 con 0x55ec315c6470 2024-03-28T22:15:16.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.456+0000 7f0315359640 1 --2- 172.21.15.16:0/3248807409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ec315cf970 0x55ec315cd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:15:16.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.457+0000 7f0314b58640 1 --2- 172.21.15.16:0/3248807409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec315c6470 0x55ec315cc9a0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f03040974b0 tx=0x7f030409aae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f2992cac824be7c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:16.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.457+0000 7f02effff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:15:16.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.457+0000 7f02effff640 1 -- 172.21.15.16:0/3248807409 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f030406f540 con 0x55ec315c6470 2024-03-28T22:15:16.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.457+0000 7f02effff640 1 -- 172.21.15.16:0/3248807409 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f0304075880 con 0x55ec315c6470 2024-03-28T22:15:16.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.457+0000 7f031999f740 1 -- 172.21.15.16:0/3248807409 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02fc004040 con 0x55ec315c6470 2024-03-28T22:15:16.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.457+0000 7f02effff640 1 -- 172.21.15.16:0/3248807409 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0304075a80 con 0x55ec315c6470 2024-03-28T22:15:16.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.457+0000 7f031999f740 10 client.16488 fetch_fsmap learned FSMap version 27 2024-03-28T22:15:16.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.457+0000 7f031999f740 1 -- 172.21.15.16:0/3248807409 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ec313087d0 con 0x55ec315c6470 2024-03-28T22:15:16.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.457+0000 7f02effff640 1 -- 172.21.15.16:0/3248807409 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02fc004040 con 0x55ec315c6470 2024-03-28T22:15:16.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.457+0000 7f02effff640 1 -- 172.21.15.16:0/3248807409 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f0304095bc0 con 0x55ec315c6470 2024-03-28T22:15:16.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.457+0000 7f031999f740 10 client.16488 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:15:16.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.457+0000 7f031999f740 10 client.16488 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T22:15:16.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.457+0000 7f031999f740 20 client.16488 populate_metadata read hostname 'smithi016' 2024-03-28T22:15:16.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.457+0000 7f031999f740 1 --2- 172.21.15.16:0/3248807409 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55ec315df8b0 0x55ec315e1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.457+0000 7f031999f740 4 client.16488 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:15:16.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.457+0000 7f031999f740 1 -- 172.21.15.16:0/3248807409 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ec315376f0 con 0x55ec315df8b0 2024-03-28T22:15:16.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.457+0000 7f030ffff640 1 --2- 172.21.15.16:0/3248807409 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55ec315df8b0 0x55ec315e1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.458+0000 7f02effff640 10 client.16488 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:15:16.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.458+0000 7f030ffff640 1 --2- 172.21.15.16:0/3248807409 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55ec315df8b0 0x55ec315e1d20 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8303c79b4e3e55bc server_cookie=86fb8901bbd53ee in_seq=0 out_seq=0 2024-03-28T22:15:16.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.459+0000 7f02effff640 1 -- 172.21.15.16:0/3248807409 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ec315376f0 con 0x55ec315df8b0 2024-03-28T22:15:16.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.459+0000 7f02effff640 10 client.16488 handle_command_reply: tid=0 2024-03-28T22:15:16.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.459+0000 7f031999f740 1 client.16488 shutdown 2024-03-28T22:15:16.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.459+0000 7f031999f740 1 -- 172.21.15.16:0/3248807409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec315c6470 msgr2=0x55ec315cc9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:16.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.459+0000 7f031999f740 1 --2- 172.21.15.16:0/3248807409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec315c6470 0x55ec315cc9a0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f03040974b0 tx=0x7f030409aae0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.459+0000 7f031999f740 1 -- 172.21.15.16:0/3248807409 shutdown_connections 2024-03-28T22:15:16.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.459+0000 7f031999f740 1 --2- 172.21.15.16:0/3248807409 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55ec315df8b0 0x55ec315e1d20 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.459+0000 7f031999f740 1 --2- 172.21.15.16:0/3248807409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ec315cf970 0x55ec315cd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.459+0000 7f02effff640 0 client.16488 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:15:16.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.459+0000 7f031999f740 1 --2- 172.21.15.16:0/3248807409 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec315c6d90 0x55ec315ccee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.459+0000 7f031999f740 1 --2- 172.21.15.16:0/3248807409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec315c6470 0x55ec315cc9a0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.459+0000 7f031999f740 1 -- 172.21.15.16:0/3248807409 >> 172.21.15.16:0/3248807409 conn(0x55ec315c0ba0 msgr2=0x55ec315c28f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:16.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.459+0000 7f031999f740 1 -- 172.21.15.16:0/3248807409 shutdown_connections 2024-03-28T22:15:16.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.459+0000 7f031999f740 1 -- 172.21.15.16:0/3248807409 wait complete. 2024-03-28T22:15:16.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.459+0000 7f031999f740 20 client.16488 trim_cache size 0 max 16384 2024-03-28T22:15:16.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.472+0000 7f031999f740 1 Processor -- start 2024-03-28T22:15:16.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.473+0000 7f031999f740 1 -- start start 2024-03-28T22:15:16.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.473+0000 7f031999f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec315c6470 0x55ec315cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.473+0000 7f031999f740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec315c6d90 0x55ec315db500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.473+0000 7f031999f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ec315cf970 0x55ec315e7610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.473+0000 7f031999f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f03101afb50 con 0x55ec315c6470 2024-03-28T22:15:16.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.473+0000 7f031999f740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f03101ad420 con 0x55ec315c6d90 2024-03-28T22:15:16.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.473+0000 7f031999f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f03101ab270 con 0x55ec315cf970 2024-03-28T22:15:16.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.473+0000 7f0314b58640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec315c6470 0x55ec315cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.473+0000 7f0314b58640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec315c6470 0x55ec315cf0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59500/0 (socket says 172.21.15.16:59500) 2024-03-28T22:15:16.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.473+0000 7f0314b58640 1 -- 172.21.15.16:0/2105637628 learned_addr learned my addr 172.21.15.16:0/2105637628 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:16.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.473+0000 7f030ffff640 1 --2- 172.21.15.16:0/2105637628 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec315c6d90 0x55ec315db500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.473+0000 7f0315359640 1 --2- 172.21.15.16:0/2105637628 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ec315cf970 0x55ec315e7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.473+0000 7f0314b58640 1 -- 172.21.15.16:0/2105637628 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ec315cf970 msgr2=0x55ec315e7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:16.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.473+0000 7f0314b58640 1 --2- 172.21.15.16:0/2105637628 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ec315cf970 0x55ec315e7610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.473+0000 7f0314b58640 1 -- 172.21.15.16:0/2105637628 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec315c6d90 msgr2=0x55ec315db500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:16.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.473+0000 7f0314b58640 1 --2- 172.21.15.16:0/2105637628 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec315c6d90 0x55ec315db500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.473+0000 7f0314b58640 1 -- 172.21.15.16:0/2105637628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec315c6bc0 con 0x55ec315c6470 2024-03-28T22:15:16.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.473+0000 7f030ffff640 1 --2- 172.21.15.16:0/2105637628 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec315c6d90 0x55ec315db500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:15:16.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.473+0000 7f0314b58640 1 --2- 172.21.15.16:0/2105637628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec315c6470 0x55ec315cf0d0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f03040cc3b0 tx=0x7f0304097420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f2e26d06135b478 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:16.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.473+0000 7f02fa7fc640 1 -- 172.21.15.16:0/2105637628 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03040b7070 con 0x55ec315c6470 2024-03-28T22:15:16.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.473+0000 7f02fa7fc640 1 -- 172.21.15.16:0/2105637628 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0304065070 con 0x55ec315c6470 2024-03-28T22:15:16.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.473+0000 7f02fa7fc640 1 -- 172.21.15.16:0/2105637628 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f030409b050 con 0x55ec315c6470 2024-03-28T22:15:16.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.474+0000 7f031999f740 1 -- 172.21.15.16:0/2105637628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec315c6470 msgr2=0x55ec315cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:16.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.474+0000 7f031999f740 1 --2- 172.21.15.16:0/2105637628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec315c6470 0x55ec315cf0d0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f03040cc3b0 tx=0x7f0304097420 comp rx=0 tx=0).stop 2024-03-28T22:15:16.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.474+0000 7f031999f740 1 -- 172.21.15.16:0/2105637628 shutdown_connections 2024-03-28T22:15:16.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.474+0000 7f031999f740 1 --2- 172.21.15.16:0/2105637628 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ec315cf970 0x55ec315e7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.474+0000 7f031999f740 1 --2- 172.21.15.16:0/2105637628 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec315c6d90 0x55ec315db500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.474+0000 7f031999f740 1 --2- 172.21.15.16:0/2105637628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec315c6470 0x55ec315cf0d0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.474+0000 7f031999f740 1 -- 172.21.15.16:0/2105637628 >> 172.21.15.16:0/2105637628 conn(0x55ec315c0ba0 msgr2=0x55ec315c2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:16.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.474+0000 7f031999f740 1 -- 172.21.15.16:0/2105637628 shutdown_connections 2024-03-28T22:15:16.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.474+0000 7f031999f740 1 -- 172.21.15.16:0/2105637628 wait complete. 2024-03-28T22:15:16.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.474+0000 7f031999f740 1 -- start start 2024-03-28T22:15:16.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.474+0000 7f031999f740 1 Processor -- start 2024-03-28T22:15:16.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.475+0000 7f031999f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec315c6470 0x55ec315d7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.475+0000 7f031999f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ec315c6d90 0x55ec315d4b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.475+0000 7f031999f740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec315cf970 0x55ec315d5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.475+0000 7f031999f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f03101ab8e0 con 0x55ec315c6470 2024-03-28T22:15:16.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.475+0000 7f031999f740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f03101acd50 con 0x55ec315cf970 2024-03-28T22:15:16.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.475+0000 7f031999f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f03101abfb0 con 0x55ec315c6d90 2024-03-28T22:15:16.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.475+0000 7f031999f740 10 client.0 _ll_register_callbacks cb 0x7f030409ac00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:15:16.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.475+0000 7f0314b58640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec315c6470 0x55ec315d7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.475+0000 7f0314b58640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec315c6470 0x55ec315d7b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59510/0 (socket says 172.21.15.16:59510) 2024-03-28T22:15:16.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.475+0000 7f0314b58640 1 -- 172.21.15.16:0/1712174941 learned_addr learned my addr 172.21.15.16:0/1712174941 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:16.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.475+0000 7f0315359640 1 --2- 172.21.15.16:0/1712174941 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec315cf970 0x55ec315d5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.475+0000 7f0314b58640 1 -- 172.21.15.16:0/1712174941 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ec315c6d90 msgr2=0x55ec315d4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:16.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.475+0000 7f0314b58640 1 --2- 172.21.15.16:0/1712174941 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ec315c6d90 0x55ec315d4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.475+0000 7f0314b58640 1 -- 172.21.15.16:0/1712174941 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec315cf970 msgr2=0x55ec315d5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:16.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.475+0000 7f0314b58640 1 --2- 172.21.15.16:0/1712174941 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec315cf970 0x55ec315d5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.475+0000 7f0314b58640 1 -- 172.21.15.16:0/1712174941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec315d8240 con 0x55ec315c6470 2024-03-28T22:15:16.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.475+0000 7f0314b58640 1 --2- 172.21.15.16:0/1712174941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec315c6470 0x55ec315d7b30 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f0304099300 tx=0x7f0304004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41e8eadf3d854baa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:16.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.475+0000 7f02effff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:15:16.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.475+0000 7f02effff640 1 -- 172.21.15.16:0/1712174941 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03040a7020 con 0x55ec315c6470 2024-03-28T22:15:16.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.475+0000 7f02effff640 1 -- 172.21.15.16:0/1712174941 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f030409a4a0 con 0x55ec315c6470 2024-03-28T22:15:16.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.475+0000 7f031999f740 1 -- 172.21.15.16:0/1712174941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0304065070 con 0x55ec315c6470 2024-03-28T22:15:16.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.476+0000 7f02effff640 1 -- 172.21.15.16:0/1712174941 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f03040b7030 con 0x55ec315c6470 2024-03-28T22:15:16.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.476+0000 7f031999f740 10 client.16494 fetch_fsmap learned FSMap version 27 2024-03-28T22:15:16.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.476+0000 7f031999f740 1 -- 172.21.15.16:0/1712174941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ec315d8240 con 0x55ec315c6470 2024-03-28T22:15:16.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.476+0000 7f02effff640 1 -- 172.21.15.16:0/1712174941 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0304065070 con 0x55ec315c6470 2024-03-28T22:15:16.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.476+0000 7f02effff640 1 -- 172.21.15.16:0/1712174941 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f0304096da0 con 0x55ec315c6470 2024-03-28T22:15:16.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.476+0000 7f031999f740 10 client.16494 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:15:16.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.476+0000 7f031999f740 10 client.16494 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T22:15:16.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.476+0000 7f031999f740 20 client.16494 populate_metadata read hostname 'smithi016' 2024-03-28T22:15:16.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.476+0000 7f031999f740 1 --2- 172.21.15.16:0/1712174941 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55ec31646980 0x55ec31648d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.476+0000 7f031999f740 4 client.16494 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:15:16.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.476+0000 7f031999f740 1 -- 172.21.15.16:0/1712174941 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ec31531a40 con 0x55ec31646980 2024-03-28T22:15:16.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.476+0000 7f030ffff640 1 --2- 172.21.15.16:0/1712174941 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55ec31646980 0x55ec31648d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.477+0000 7f02effff640 10 client.16494 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:15:16.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.477+0000 7f030ffff640 1 --2- 172.21.15.16:0/1712174941 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55ec31646980 0x55ec31648d70 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7357c497c0fa5a7f server_cookie=5c5fee044a888027 in_seq=0 out_seq=0 2024-03-28T22:15:16.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.477+0000 7f02effff640 1 -- 172.21.15.16:0/1712174941 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ec31531a40 con 0x55ec31646980 2024-03-28T22:15:16.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.477+0000 7f02effff640 10 client.16494 handle_command_reply: tid=0 2024-03-28T22:15:16.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.477+0000 7f031999f740 1 client.16494 shutdown 2024-03-28T22:15:16.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.477+0000 7f031999f740 1 -- 172.21.15.16:0/1712174941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec315c6470 msgr2=0x55ec315d7b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:16.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.477+0000 7f031999f740 1 --2- 172.21.15.16:0/1712174941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec315c6470 0x55ec315d7b30 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f0304099300 tx=0x7f0304004a40 comp rx=0 tx=0).stop 2024-03-28T22:15:16.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.477+0000 7f031999f740 1 -- 172.21.15.16:0/1712174941 shutdown_connections 2024-03-28T22:15:16.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.477+0000 7f031999f740 1 --2- 172.21.15.16:0/1712174941 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55ec31646980 0x55ec31648d70 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.477+0000 7f031999f740 1 --2- 172.21.15.16:0/1712174941 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec315cf970 0x55ec315d5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.477+0000 7f02effff640 0 client.16494 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:15:16.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.477+0000 7f031999f740 1 --2- 172.21.15.16:0/1712174941 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ec315c6d90 0x55ec315d4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.477+0000 7f031999f740 1 --2- 172.21.15.16:0/1712174941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec315c6470 0x55ec315d7b30 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.477+0000 7f031999f740 1 -- 172.21.15.16:0/1712174941 >> 172.21.15.16:0/1712174941 conn(0x55ec315c0ba0 msgr2=0x55ec315bf4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:16.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.477+0000 7f031999f740 1 -- 172.21.15.16:0/1712174941 shutdown_connections 2024-03-28T22:15:16.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.477+0000 7f031999f740 1 -- 172.21.15.16:0/1712174941 wait complete. 2024-03-28T22:15:16.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.477+0000 7f031999f740 20 client.16494 trim_cache size 0 max 16384 2024-03-28T22:15:16.490 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:15:16.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.478+0000 7f0316dbc640 1 -- 172.21.15.16:0/3830202006 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f03000a1ef0 msgr2=0x7f03000a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:16.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.478+0000 7f0316dbc640 1 --2- 172.21.15.16:0/3830202006 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f03000a1ef0 0x7f03000a43b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f031006aed0 tx=0x7f0308012b30 comp rx=0 tx=0).stop 2024-03-28T22:15:16.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.478+0000 7f0316dbc640 1 -- 172.21.15.16:0/3830202006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0310102970 msgr2=0x7f0310198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:16.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.478+0000 7f0316dbc640 1 --2- 172.21.15.16:0/3830202006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0310102970 0x7f0310198030 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f030400e9f0 tx=0x7f030400edf0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.478+0000 7f0316dbc640 1 -- 172.21.15.16:0/3830202006 shutdown_connections 2024-03-28T22:15:16.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.478+0000 7f0316dbc640 1 --2- 172.21.15.16:0/3830202006 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f03000a1ef0 0x7f03000a43b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.478+0000 7f0316dbc640 1 --2- 172.21.15.16:0/3830202006 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f031010bd40 0x7f031019c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.478+0000 7f0316dbc640 1 --2- 172.21.15.16:0/3830202006 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0310103290 0x7f0310198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.478+0000 7f0316dbc640 1 --2- 172.21.15.16:0/3830202006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0310102970 0x7f0310198030 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.478+0000 7f0316dbc640 1 -- 172.21.15.16:0/3830202006 >> 172.21.15.16:0/3830202006 conn(0x7f0310077770 msgr2=0x7f03100fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:16.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.478+0000 7f0316dbc640 1 -- 172.21.15.16:0/3830202006 shutdown_connections 2024-03-28T22:15:16.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.478+0000 7f0316dbc640 1 -- 172.21.15.16:0/3830202006 wait complete. 2024-03-28T22:15:16.493 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:16 smithi016 ceph-mon[32374]: pgmap v143: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T22:15:16.493 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:16 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2667444867' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:15:16.494 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:15:16.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:16 smithi053 ceph-mon[35979]: pgmap v143: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T22:15:16.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:16 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2667444867' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:15:16.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.597+0000 7f62ea772640 1 -- 172.21.15.16:0/2482623218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62e4100d50 msgr2=0x7f62e41011d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:16.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.597+0000 7f62ea772640 1 --2- 172.21.15.16:0/2482623218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62e4100d50 0x7f62e41011d0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f62d8009930 tx=0x7f62d802bc30 comp rx=0 tx=0).stop 2024-03-28T22:15:16.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.597+0000 7f62ea772640 1 -- 172.21.15.16:0/2482623218 shutdown_connections 2024-03-28T22:15:16.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.597+0000 7f62ea772640 1 --2- 172.21.15.16:0/2482623218 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62e4101710 0x7f62e410d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.597+0000 7f62ea772640 1 --2- 172.21.15.16:0/2482623218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62e4100d50 0x7f62e41011d0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.597+0000 7f62ea772640 1 --2- 172.21.15.16:0/2482623218 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62e41065f0 0x7f62e41069d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.597+0000 7f62ea772640 1 -- 172.21.15.16:0/2482623218 >> 172.21.15.16:0/2482623218 conn(0x7f62e4076b50 msgr2=0x7f62e40fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:16.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.597+0000 7f62ea772640 1 -- 172.21.15.16:0/2482623218 shutdown_connections 2024-03-28T22:15:16.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.597+0000 7f62ea772640 1 -- 172.21.15.16:0/2482623218 wait complete. 2024-03-28T22:15:16.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.598+0000 7f62ea772640 1 Processor -- start 2024-03-28T22:15:16.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.598+0000 7f62ea772640 1 -- start start 2024-03-28T22:15:16.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.598+0000 7f62ea772640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62e4100d50 0x7f62e4196f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.598+0000 7f62ea772640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62e4101710 0x7f62e4197460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.598+0000 7f62ea772640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62e41065f0 0x7f62e419b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.598+0000 7f62ea772640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f62e4110a00 con 0x7f62e4101710 2024-03-28T22:15:16.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.598+0000 7f62ea772640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f62e4110880 con 0x7f62e4100d50 2024-03-28T22:15:16.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.598+0000 7f62ea772640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f62e4110b80 con 0x7f62e41065f0 2024-03-28T22:15:16.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.598+0000 7f62e37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62e4101710 0x7f62e4197460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.598+0000 7f62e37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62e4101710 0x7f62e4197460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59530/0 (socket says 172.21.15.16:59530) 2024-03-28T22:15:16.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.598+0000 7f62e37fe640 1 -- 172.21.15.16:0/2559114137 learned_addr learned my addr 172.21.15.16:0/2559114137 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:16.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.598+0000 7f62e3fff640 1 --2- 172.21.15.16:0/2559114137 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62e4100d50 0x7f62e4196f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.598+0000 7f62e8d0f640 1 --2- 172.21.15.16:0/2559114137 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62e41065f0 0x7f62e419b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.598+0000 7f62e37fe640 1 -- 172.21.15.16:0/2559114137 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62e41065f0 msgr2=0x7f62e419b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:16.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.598+0000 7f62e37fe640 1 --2- 172.21.15.16:0/2559114137 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62e41065f0 0x7f62e419b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.598+0000 7f62e37fe640 1 -- 172.21.15.16:0/2559114137 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62e4100d50 msgr2=0x7f62e4196f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:16.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.598+0000 7f62e37fe640 1 --2- 172.21.15.16:0/2559114137 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62e4100d50 0x7f62e4196f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.598+0000 7f62e37fe640 1 -- 172.21.15.16:0/2559114137 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62e419bf10 con 0x7f62e4101710 2024-03-28T22:15:16.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.599+0000 7f62e37fe640 1 --2- 172.21.15.16:0/2559114137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62e4101710 0x7f62e4197460 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f62d8009230 tx=0x7f62d802cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:16.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.599+0000 7f62e17fa640 1 -- 172.21.15.16:0/2559114137 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d8004450 con 0x7f62e4101710 2024-03-28T22:15:16.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.599+0000 7f62e17fa640 1 -- 172.21.15.16:0/2559114137 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62d80054e0 con 0x7f62e4101710 2024-03-28T22:15:16.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.599+0000 7f62ea772640 1 -- 172.21.15.16:0/2559114137 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62e419c1a0 con 0x7f62e4101710 2024-03-28T22:15:16.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.599+0000 7f62e17fa640 1 -- 172.21.15.16:0/2559114137 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d803b440 con 0x7f62e4101710 2024-03-28T22:15:16.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.599+0000 7f62ea772640 1 -- 172.21.15.16:0/2559114137 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62e40ffc40 con 0x7f62e4101710 2024-03-28T22:15:16.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.601+0000 7f62e17fa640 1 -- 172.21.15.16:0/2559114137 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f62d8003710 con 0x7f62e4101710 2024-03-28T22:15:16.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.601+0000 7f62ed42a740 1 Processor -- start 2024-03-28T22:15:16.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.601+0000 7f62ed42a740 1 -- start start 2024-03-28T22:15:16.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.601+0000 7f62e17fa640 1 --2- 172.21.15.16:0/2559114137 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f62cc0a1ef0 0x7f62cc0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.602+0000 7f62ed42a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3eb50b880 0x55f3eb50bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.602+0000 7f62ed42a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3eb50c1a0 0x55f3eb514840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.602+0000 7f62ed42a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3eb514d80 0x55f3eb517170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.602+0000 7f62ed42a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f3eb46b1b0 con 0x55f3eb514d80 2024-03-28T22:15:16.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.602+0000 7f62ed42a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f3eb413c00 con 0x55f3eb50c1a0 2024-03-28T22:15:16.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.602+0000 7f62ed42a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f3eb467250 con 0x55f3eb50b880 2024-03-28T22:15:16.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.602+0000 7f62e8d0f640 1 --2- 172.21.15.16:0/2559114137 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f62cc0a1ef0 0x7f62cc0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.602+0000 7f62e17fa640 1 -- 172.21.15.16:0/2559114137 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f62d80f1b20 con 0x7f62e4101710 2024-03-28T22:15:16.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.602+0000 7f62e37fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3eb50c1a0 0x55f3eb514840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.602+0000 7f62e8d0f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3eb514d80 0x55f3eb517170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.602+0000 7f62e37fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3eb50c1a0 0x55f3eb514840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:51614/0 (socket says 172.21.15.16:51614) 2024-03-28T22:15:16.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.602+0000 7f62e37fe640 1 -- 172.21.15.16:0/2337416770 learned_addr learned my addr 172.21.15.16:0/2337416770 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:16.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.602+0000 7f62e3fff640 1 --2- 172.21.15.16:0/2337416770 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3eb50b880 0x55f3eb50bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.602+0000 7f62e37fe640 1 -- 172.21.15.16:0/2337416770 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3eb50b880 msgr2=0x55f3eb50bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:16.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.602+0000 7f62e37fe640 1 --2- 172.21.15.16:0/2337416770 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3eb50b880 0x55f3eb50bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.602+0000 7f62e37fe640 1 -- 172.21.15.16:0/2337416770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3eb514d80 msgr2=0x55f3eb517170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:16.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.602+0000 7f62e37fe640 1 --2- 172.21.15.16:0/2337416770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3eb514d80 0x55f3eb517170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.602+0000 7f62e37fe640 1 -- 172.21.15.16:0/2337416770 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3eb4333d0 con 0x55f3eb50c1a0 2024-03-28T22:15:16.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.602+0000 7f62e8d0f640 1 --2- 172.21.15.16:0/2559114137 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f62cc0a1ef0 0x7f62cc0a43b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f62dc005fd0 tx=0x7f62dc009f30 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:16.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.602+0000 7f62e37fe640 1 --2- 172.21.15.16:0/2337416770 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3eb50c1a0 0x55f3eb514840 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f62d80bb010 tx=0x7f62d80ba9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7423a3734ef31943 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:16.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.602+0000 7f62ca7fc640 1 -- 172.21.15.16:0/2337416770 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d80bdfa0 con 0x55f3eb50c1a0 2024-03-28T22:15:16.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.602+0000 7f62ca7fc640 1 -- 172.21.15.16:0/2337416770 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62d8004c80 con 0x55f3eb50c1a0 2024-03-28T22:15:16.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.602+0000 7f62ca7fc640 1 -- 172.21.15.16:0/2337416770 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d80c6de0 con 0x55f3eb50c1a0 2024-03-28T22:15:16.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.603+0000 7f62ed42a740 1 -- 172.21.15.16:0/2337416770 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3eb50c1a0 msgr2=0x55f3eb514840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:16.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.603+0000 7f62ed42a740 1 --2- 172.21.15.16:0/2337416770 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3eb50c1a0 0x55f3eb514840 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f62d80bb010 tx=0x7f62d80ba9a0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.603+0000 7f62ed42a740 1 -- 172.21.15.16:0/2337416770 shutdown_connections 2024-03-28T22:15:16.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.603+0000 7f62ed42a740 1 --2- 172.21.15.16:0/2337416770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3eb514d80 0x55f3eb517170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.603+0000 7f62ed42a740 1 --2- 172.21.15.16:0/2337416770 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3eb50c1a0 0x55f3eb514840 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.603+0000 7f62ed42a740 1 --2- 172.21.15.16:0/2337416770 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3eb50b880 0x55f3eb50bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.603+0000 7f62ed42a740 1 -- 172.21.15.16:0/2337416770 >> 172.21.15.16:0/2337416770 conn(0x55f3eb505fb0 msgr2=0x55f3eb506390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:16.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.603+0000 7f62ed42a740 1 -- 172.21.15.16:0/2337416770 shutdown_connections 2024-03-28T22:15:16.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.603+0000 7f62ed42a740 1 -- 172.21.15.16:0/2337416770 wait complete. 2024-03-28T22:15:16.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.603+0000 7f62ed42a740 1 -- start start 2024-03-28T22:15:16.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.604+0000 7f62ed42a740 1 Processor -- start 2024-03-28T22:15:16.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.604+0000 7f62ed42a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3eb50b880 0x55f3eb511d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.604+0000 7f62ed42a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3eb50c1a0 0x55f3eb512280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.604+0000 7f62ed42a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3eb514d80 0x55f3eb5127c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.604+0000 7f62ed42a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f3eb413c00 con 0x55f3eb50c1a0 2024-03-28T22:15:16.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.604+0000 7f62ed42a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f3eb409470 con 0x55f3eb514d80 2024-03-28T22:15:16.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.604+0000 7f62ed42a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f3eb517930 con 0x55f3eb50b880 2024-03-28T22:15:16.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.604+0000 7f62e37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3eb50c1a0 0x55f3eb512280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.604+0000 7f62ed42a740 10 client.0 _ll_register_callbacks cb 0x55f3eb426cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:15:16.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.604+0000 7f62e37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3eb50c1a0 0x55f3eb512280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59552/0 (socket says 172.21.15.16:59552) 2024-03-28T22:15:16.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.604+0000 7f62e37fe640 1 -- 172.21.15.16:0/570895749 learned_addr learned my addr 172.21.15.16:0/570895749 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:16.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.604+0000 7f62e8d0f640 1 --2- 172.21.15.16:0/570895749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3eb514d80 0x55f3eb5127c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.604+0000 7f62e3fff640 1 --2- 172.21.15.16:0/570895749 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3eb50b880 0x55f3eb511d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.604+0000 7f62e37fe640 1 -- 172.21.15.16:0/570895749 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3eb50b880 msgr2=0x55f3eb511d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:16.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.604+0000 7f62e37fe640 1 --2- 172.21.15.16:0/570895749 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3eb50b880 0x55f3eb511d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.604+0000 7f62e37fe640 1 -- 172.21.15.16:0/570895749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3eb514d80 msgr2=0x55f3eb5127c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:16.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.604+0000 7f62e37fe640 1 --2- 172.21.15.16:0/570895749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3eb514d80 0x55f3eb5127c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.604+0000 7f62e37fe640 1 -- 172.21.15.16:0/570895749 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3eb17f3f0 con 0x55f3eb50c1a0 2024-03-28T22:15:16.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.604+0000 7f62e3fff640 1 --2- 172.21.15.16:0/570895749 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3eb50b880 0x55f3eb511d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:15:16.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.604+0000 7f62e8d0f640 1 --2- 172.21.15.16:0/570895749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3eb514d80 0x55f3eb5127c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:15:16.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.604+0000 7f62e37fe640 1 --2- 172.21.15.16:0/570895749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3eb50c1a0 0x55f3eb512280 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f62d80ef890 tx=0x7f62d80be090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b6bf6056c1aae57 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:16.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.604+0000 7f62c37fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:15:16.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.605+0000 7f62c37fe640 1 -- 172.21.15.16:0/570895749 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d80be640 con 0x55f3eb50c1a0 2024-03-28T22:15:16.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.605+0000 7f62c37fe640 1 -- 172.21.15.16:0/570895749 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f62d80be8a0 con 0x55f3eb50c1a0 2024-03-28T22:15:16.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.605+0000 7f62ed42a740 1 -- 172.21.15.16:0/570895749 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62d8004c80 con 0x55f3eb50c1a0 2024-03-28T22:15:16.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.605+0000 7f62c37fe640 1 -- 172.21.15.16:0/570895749 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f62d80ef680 con 0x55f3eb50c1a0 2024-03-28T22:15:16.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.605+0000 7f62ed42a740 10 client.16506 fetch_fsmap learned FSMap version 27 2024-03-28T22:15:16.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.605+0000 7f62ed42a740 1 -- 172.21.15.16:0/570895749 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f3eb2497d0 con 0x55f3eb50c1a0 2024-03-28T22:15:16.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.605+0000 7f62c37fe640 1 -- 172.21.15.16:0/570895749 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62d8004c80 con 0x55f3eb50c1a0 2024-03-28T22:15:16.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.605+0000 7f62c37fe640 1 -- 172.21.15.16:0/570895749 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f62d8042070 con 0x55f3eb50c1a0 2024-03-28T22:15:16.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.605+0000 7f62ed42a740 10 client.16506 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:15:16.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.605+0000 7f62ed42a740 10 client.16506 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:15:16.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.605+0000 7f62ed42a740 20 client.16506 populate_metadata read hostname 'smithi016' 2024-03-28T22:15:16.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.605+0000 7f62ed42a740 1 --2- 172.21.15.16:0/570895749 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f3eb524730 0x55f3eb526b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.605+0000 7f62ed42a740 4 client.16506 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:15:16.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.605+0000 7f62ed42a740 1 -- 172.21.15.16:0/570895749 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f3eb4786f0 con 0x55f3eb524730 2024-03-28T22:15:16.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.605+0000 7f62e3fff640 1 --2- 172.21.15.16:0/570895749 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f3eb524730 0x55f3eb526b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.606+0000 7f62c37fe640 10 client.16506 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:15:16.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.606+0000 7f62e3fff640 1 --2- 172.21.15.16:0/570895749 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f3eb524730 0x55f3eb526b20 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f0587e2ac2386a0 server_cookie=ee6d6f026b7424ed in_seq=0 out_seq=0 2024-03-28T22:15:16.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.606+0000 7f62c37fe640 1 -- 172.21.15.16:0/570895749 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f3eb4786f0 con 0x55f3eb524730 2024-03-28T22:15:16.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.606+0000 7f62c37fe640 10 client.16506 handle_command_reply: tid=0 2024-03-28T22:15:16.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.606+0000 7f62ed42a740 1 client.16506 shutdown 2024-03-28T22:15:16.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.606+0000 7f62ed42a740 1 -- 172.21.15.16:0/570895749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3eb50c1a0 msgr2=0x55f3eb512280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:16.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.606+0000 7f62ed42a740 1 --2- 172.21.15.16:0/570895749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3eb50c1a0 0x55f3eb512280 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f62d80ef890 tx=0x7f62d80be090 comp rx=0 tx=0).stop 2024-03-28T22:15:16.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.607+0000 7f62ed42a740 1 -- 172.21.15.16:0/570895749 shutdown_connections 2024-03-28T22:15:16.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.607+0000 7f62ed42a740 1 --2- 172.21.15.16:0/570895749 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f3eb524730 0x55f3eb526b20 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.607+0000 7f62ed42a740 1 --2- 172.21.15.16:0/570895749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3eb514d80 0x55f3eb5127c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.607+0000 7f62ed42a740 1 --2- 172.21.15.16:0/570895749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3eb50c1a0 0x55f3eb512280 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.607+0000 7f62c37fe640 0 client.16506 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:15:16.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.607+0000 7f62ed42a740 1 --2- 172.21.15.16:0/570895749 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3eb50b880 0x55f3eb511d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.607+0000 7f62ed42a740 1 -- 172.21.15.16:0/570895749 >> 172.21.15.16:0/570895749 conn(0x55f3eb505fb0 msgr2=0x55f3eb500d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:16.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.607+0000 7f62ed42a740 1 -- 172.21.15.16:0/570895749 shutdown_connections 2024-03-28T22:15:16.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.607+0000 7f62ed42a740 1 -- 172.21.15.16:0/570895749 wait complete. 2024-03-28T22:15:16.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.607+0000 7f62ed42a740 20 client.16506 trim_cache size 0 max 16384 2024-03-28T22:15:16.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.620+0000 7f62ed42a740 1 Processor -- start 2024-03-28T22:15:16.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.620+0000 7f62ed42a740 1 -- start start 2024-03-28T22:15:16.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.620+0000 7f62ed42a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3eb50b880 0x55f3eb5204c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.620+0000 7f62ed42a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3eb50c1a0 0x55f3eb4fcbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.620+0000 7f62ed42a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3eb514d80 0x55f3eb52c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.620+0000 7f62ed42a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f62e41ad9b0 con 0x55f3eb50c1a0 2024-03-28T22:15:16.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.620+0000 7f62ed42a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f62e41ab280 con 0x55f3eb514d80 2024-03-28T22:15:16.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.620+0000 7f62ed42a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f62e41a90f0 con 0x55f3eb50b880 2024-03-28T22:15:16.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.620+0000 7f62e37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3eb50c1a0 0x55f3eb4fcbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.620+0000 7f62e37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3eb50c1a0 0x55f3eb4fcbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59560/0 (socket says 172.21.15.16:59560) 2024-03-28T22:15:16.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.620+0000 7f62e37fe640 1 -- 172.21.15.16:0/1684583477 learned_addr learned my addr 172.21.15.16:0/1684583477 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:16.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.620+0000 7f62e8d0f640 1 --2- 172.21.15.16:0/1684583477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3eb514d80 0x55f3eb52c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.620+0000 7f62e3fff640 1 --2- 172.21.15.16:0/1684583477 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3eb50b880 0x55f3eb5204c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.620+0000 7f62e37fe640 1 -- 172.21.15.16:0/1684583477 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3eb50b880 msgr2=0x55f3eb5204c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:16.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.620+0000 7f62e37fe640 1 --2- 172.21.15.16:0/1684583477 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3eb50b880 0x55f3eb5204c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.620+0000 7f62e37fe640 1 -- 172.21.15.16:0/1684583477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3eb514d80 msgr2=0x55f3eb52c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:16.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.620+0000 7f62e37fe640 1 --2- 172.21.15.16:0/1684583477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3eb514d80 0x55f3eb52c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.620+0000 7f62e37fe640 1 -- 172.21.15.16:0/1684583477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3eb505de0 con 0x55f3eb50c1a0 2024-03-28T22:15:16.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.620+0000 7f62e3fff640 1 --2- 172.21.15.16:0/1684583477 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3eb50b880 0x55f3eb5204c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:15:16.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.621+0000 7f62e37fe640 1 --2- 172.21.15.16:0/1684583477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3eb50c1a0 0x55f3eb4fcbd0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f62d8003af0 tx=0x7f62d8005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf858d96b412044b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:16.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.621+0000 7f62c9ffb640 1 -- 172.21.15.16:0/1684583477 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d8004c80 con 0x55f3eb50c1a0 2024-03-28T22:15:16.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.621+0000 7f62c9ffb640 1 -- 172.21.15.16:0/1684583477 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62d8005690 con 0x55f3eb50c1a0 2024-03-28T22:15:16.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.621+0000 7f62c9ffb640 1 -- 172.21.15.16:0/1684583477 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d80c6bb0 con 0x55f3eb50c1a0 2024-03-28T22:15:16.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.621+0000 7f62ed42a740 1 -- 172.21.15.16:0/1684583477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3eb50c1a0 msgr2=0x55f3eb4fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:16.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.621+0000 7f62ed42a740 1 --2- 172.21.15.16:0/1684583477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3eb50c1a0 0x55f3eb4fcbd0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f62d8003af0 tx=0x7f62d8005fd0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.621+0000 7f62ed42a740 1 -- 172.21.15.16:0/1684583477 shutdown_connections 2024-03-28T22:15:16.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.621+0000 7f62ed42a740 1 --2- 172.21.15.16:0/1684583477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3eb514d80 0x55f3eb52c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.621+0000 7f62ed42a740 1 --2- 172.21.15.16:0/1684583477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3eb50c1a0 0x55f3eb4fcbd0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.621+0000 7f62ed42a740 1 --2- 172.21.15.16:0/1684583477 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3eb50b880 0x55f3eb5204c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.621+0000 7f62ed42a740 1 -- 172.21.15.16:0/1684583477 >> 172.21.15.16:0/1684583477 conn(0x55f3eb505fb0 msgr2=0x55f3eb500d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:16.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.621+0000 7f62ed42a740 1 -- 172.21.15.16:0/1684583477 shutdown_connections 2024-03-28T22:15:16.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.621+0000 7f62ed42a740 1 -- 172.21.15.16:0/1684583477 wait complete. 2024-03-28T22:15:16.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.622+0000 7f62ed42a740 1 -- start start 2024-03-28T22:15:16.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.622+0000 7f62ed42a740 1 Processor -- start 2024-03-28T22:15:16.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.622+0000 7f62ed42a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3eb50b880 0x55f3eb51cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.622+0000 7f62ed42a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3eb50c1a0 0x55f3eb519fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.622+0000 7f62ed42a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3eb514d80 0x55f3eb51a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.622+0000 7f62ed42a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f62e41a9740 con 0x55f3eb50b880 2024-03-28T22:15:16.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.622+0000 7f62ed42a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f62e41aabb0 con 0x55f3eb50c1a0 2024-03-28T22:15:16.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.622+0000 7f62ed42a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f62e41a9e10 con 0x55f3eb514d80 2024-03-28T22:15:16.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.622+0000 7f62ed42a740 10 client.0 _ll_register_callbacks cb 0x7f62d80ba260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:15:16.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.622+0000 7f62e3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3eb50b880 0x55f3eb51cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.622+0000 7f62e3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3eb50b880 0x55f3eb51cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59564/0 (socket says 172.21.15.16:59564) 2024-03-28T22:15:16.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.622+0000 7f62e3fff640 1 -- 172.21.15.16:0/1318076271 learned_addr learned my addr 172.21.15.16:0/1318076271 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:16.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.622+0000 7f62e37fe640 1 --2- 172.21.15.16:0/1318076271 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3eb50c1a0 0x55f3eb519fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.622+0000 7f62e8d0f640 1 --2- 172.21.15.16:0/1318076271 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3eb514d80 0x55f3eb51a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.622+0000 7f62e3fff640 1 -- 172.21.15.16:0/1318076271 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3eb514d80 msgr2=0x55f3eb51a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:16.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.622+0000 7f62e3fff640 1 --2- 172.21.15.16:0/1318076271 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3eb514d80 0x55f3eb51a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.622+0000 7f62e3fff640 1 -- 172.21.15.16:0/1318076271 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3eb50c1a0 msgr2=0x55f3eb519fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:16.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.622+0000 7f62e3fff640 1 --2- 172.21.15.16:0/1318076271 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3eb50c1a0 0x55f3eb519fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.622+0000 7f62e3fff640 1 -- 172.21.15.16:0/1318076271 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3eb51aba0 con 0x55f3eb50b880 2024-03-28T22:15:16.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.623+0000 7f62e3fff640 1 --2- 172.21.15.16:0/1318076271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3eb50b880 0x55f3eb51cc30 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f62d40045d0 tx=0x7f62d400bb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=214f7a944cb3d03f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:16.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.623+0000 7f62c37fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:15:16.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.623+0000 7f62c37fe640 1 -- 172.21.15.16:0/1318076271 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d4003cc0 con 0x55f3eb50b880 2024-03-28T22:15:16.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.623+0000 7f62c37fe640 1 -- 172.21.15.16:0/1318076271 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f62d401b2c0 con 0x55f3eb50b880 2024-03-28T22:15:16.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.623+0000 7f62ed42a740 1 -- 172.21.15.16:0/1318076271 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f3eb472a40 con 0x55f3eb50b880 2024-03-28T22:15:16.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.623+0000 7f62c37fe640 1 -- 172.21.15.16:0/1318076271 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f62d400be80 con 0x55f3eb50b880 2024-03-28T22:15:16.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.623+0000 7f62ed42a740 10 client.16512 fetch_fsmap learned FSMap version 27 2024-03-28T22:15:16.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.623+0000 7f62ed42a740 1 -- 172.21.15.16:0/1318076271 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f3eb58a850 con 0x55f3eb50b880 2024-03-28T22:15:16.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.623+0000 7f62c37fe640 1 -- 172.21.15.16:0/1318076271 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f3eb472a40 con 0x55f3eb50b880 2024-03-28T22:15:16.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.623+0000 7f62c37fe640 1 -- 172.21.15.16:0/1318076271 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f62d402b020 con 0x55f3eb50b880 2024-03-28T22:15:16.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.623+0000 7f62ed42a740 10 client.16512 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:15:16.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.623+0000 7f62ed42a740 10 client.16512 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:15:16.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.623+0000 7f62ed42a740 20 client.16512 populate_metadata read hostname 'smithi016' 2024-03-28T22:15:16.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.623+0000 7f62ed42a740 1 --2- 172.21.15.16:0/1318076271 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f3eb58af80 0x55f3eb58d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:16.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.623+0000 7f62ed42a740 4 client.16512 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:15:16.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.623+0000 7f62ed42a740 1 -- 172.21.15.16:0/1318076271 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f3eb3f06a0 con 0x55f3eb58af80 2024-03-28T22:15:16.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.623+0000 7f62e37fe640 1 --2- 172.21.15.16:0/1318076271 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f3eb58af80 0x55f3eb58d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:16.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.624+0000 7f62e37fe640 1 --2- 172.21.15.16:0/1318076271 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f3eb58af80 0x55f3eb58d3a0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b82861cc65bab4b4 server_cookie=f43cb37429640831 in_seq=0 out_seq=0 2024-03-28T22:15:16.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.624+0000 7f62c37fe640 10 client.16512 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:15:16.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.624+0000 7f62c37fe640 1 -- 172.21.15.16:0/1318076271 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f3eb3f06a0 con 0x55f3eb58af80 2024-03-28T22:15:16.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.624+0000 7f62c37fe640 10 client.16512 handle_command_reply: tid=0 2024-03-28T22:15:16.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.624+0000 7f62ed42a740 1 client.16512 shutdown 2024-03-28T22:15:16.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.624+0000 7f62ed42a740 1 -- 172.21.15.16:0/1318076271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3eb50b880 msgr2=0x55f3eb51cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:16.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.624+0000 7f62ed42a740 1 --2- 172.21.15.16:0/1318076271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3eb50b880 0x55f3eb51cc30 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f62d40045d0 tx=0x7f62d400bb40 comp rx=0 tx=0).stop 2024-03-28T22:15:16.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.624+0000 7f62ed42a740 1 -- 172.21.15.16:0/1318076271 shutdown_connections 2024-03-28T22:15:16.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.624+0000 7f62ed42a740 1 --2- 172.21.15.16:0/1318076271 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f3eb58af80 0x55f3eb58d3a0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.624+0000 7f62ed42a740 1 --2- 172.21.15.16:0/1318076271 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3eb514d80 0x55f3eb51a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.624+0000 7f62ed42a740 1 --2- 172.21.15.16:0/1318076271 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f3eb50c1a0 0x55f3eb519fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.624+0000 7f62ed42a740 1 --2- 172.21.15.16:0/1318076271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3eb50b880 0x55f3eb51cc30 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.624+0000 7f62c37fe640 0 client.16512 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:15:16.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.624+0000 7f62ed42a740 1 -- 172.21.15.16:0/1318076271 >> 172.21.15.16:0/1318076271 conn(0x55f3eb505fb0 msgr2=0x55f3eb507260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:16.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.624+0000 7f62ed42a740 1 -- 172.21.15.16:0/1318076271 shutdown_connections 2024-03-28T22:15:16.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.624+0000 7f62ed42a740 1 -- 172.21.15.16:0/1318076271 wait complete. 2024-03-28T22:15:16.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.624+0000 7f62ed42a740 20 client.16512 trim_cache size 0 max 16384 2024-03-28T22:15:16.638 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:15:16.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.625+0000 7f62ea772640 1 -- 172.21.15.16:0/2559114137 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f62cc0a1ef0 msgr2=0x7f62cc0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:16.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.625+0000 7f62ea772640 1 --2- 172.21.15.16:0/2559114137 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f62cc0a1ef0 0x7f62cc0a43b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f62dc005fd0 tx=0x7f62dc009f30 comp rx=0 tx=0).stop 2024-03-28T22:15:16.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.625+0000 7f62ea772640 1 -- 172.21.15.16:0/2559114137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62e4101710 msgr2=0x7f62e4197460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:16.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.625+0000 7f62ea772640 1 --2- 172.21.15.16:0/2559114137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62e4101710 0x7f62e4197460 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f62d8009230 tx=0x7f62d802cba0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.625+0000 7f62ea772640 1 -- 172.21.15.16:0/2559114137 shutdown_connections 2024-03-28T22:15:16.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.625+0000 7f62ea772640 1 --2- 172.21.15.16:0/2559114137 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f62cc0a1ef0 0x7f62cc0a43b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.625+0000 7f62ea772640 1 --2- 172.21.15.16:0/2559114137 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62e41065f0 0x7f62e419b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.625+0000 7f62ea772640 1 --2- 172.21.15.16:0/2559114137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62e4101710 0x7f62e4197460 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.625+0000 7f62ea772640 1 --2- 172.21.15.16:0/2559114137 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62e4100d50 0x7f62e4196f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:16.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.625+0000 7f62ea772640 1 -- 172.21.15.16:0/2559114137 >> 172.21.15.16:0/2559114137 conn(0x7f62e4076b50 msgr2=0x7f62e40fe3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:16.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.625+0000 7f62ea772640 1 -- 172.21.15.16:0/2559114137 shutdown_connections 2024-03-28T22:15:16.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:16.625+0000 7f62ea772640 1 -- 172.21.15.16:0/2559114137 wait complete. 2024-03-28T22:15:16.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:16 smithi063 ceph-mon[36249]: pgmap v143: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T22:15:16.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:16 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2667444867' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:15:17.641 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:15:17.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:15:17.642 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:15:17.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.760+0000 7f759f72c640 1 -- 172.21.15.16:0/1662291918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7598103290 msgr2=0x7f759810b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:17.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.760+0000 7f759f72c640 1 --2- 172.21.15.16:0/1662291918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7598103290 0x7f759810b770 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f7588009900 tx=0x7f758802bcd0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.760+0000 7f759f72c640 1 -- 172.21.15.16:0/1662291918 shutdown_connections 2024-03-28T22:15:17.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.760+0000 7f759f72c640 1 --2- 172.21.15.16:0/1662291918 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f759810bd40 0x7f759810e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.760+0000 7f759f72c640 1 --2- 172.21.15.16:0/1662291918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7598103290 0x7f759810b770 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.760+0000 7f759f72c640 1 --2- 172.21.15.16:0/1662291918 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7598102970 0x7f7598102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.760+0000 7f759f72c640 1 -- 172.21.15.16:0/1662291918 >> 172.21.15.16:0/1662291918 conn(0x7f7598077770 msgr2=0x7f7598077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:17.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.760+0000 7f759f72c640 1 -- 172.21.15.16:0/1662291918 shutdown_connections 2024-03-28T22:15:17.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.760+0000 7f759f72c640 1 -- 172.21.15.16:0/1662291918 wait complete. 2024-03-28T22:15:17.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.761+0000 7f759f72c640 1 Processor -- start 2024-03-28T22:15:17.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.761+0000 7f759f72c640 1 -- start start 2024-03-28T22:15:17.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.761+0000 7f759f72c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7598102970 0x7f75981005c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:17.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.761+0000 7f759f72c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7598103290 0x7f7598100b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:17.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.761+0000 7f759f72c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f759810bd40 0x7f7598101040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:17.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.761+0000 7f759f72c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7598111ad0 con 0x7f7598103290 2024-03-28T22:15:17.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.761+0000 7f759f72c640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7598111950 con 0x7f7598102970 2024-03-28T22:15:17.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.761+0000 7f759f72c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7598111c50 con 0x7f759810bd40 2024-03-28T22:15:17.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.761+0000 7f759ccc7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7598103290 0x7f7598100b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:17.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.761+0000 7f759d4c8640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7598102970 0x7f75981005c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:17.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.761+0000 7f759ccc7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7598103290 0x7f7598100b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59580/0 (socket says 172.21.15.16:59580) 2024-03-28T22:15:17.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.761+0000 7f759ccc7640 1 -- 172.21.15.16:0/78349542 learned_addr learned my addr 172.21.15.16:0/78349542 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:17.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.762+0000 7f759dcc9640 1 --2- 172.21.15.16:0/78349542 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f759810bd40 0x7f7598101040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:17.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.762+0000 7f759d4c8640 1 -- 172.21.15.16:0/78349542 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f759810bd40 msgr2=0x7f7598101040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:17.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.762+0000 7f759d4c8640 1 --2- 172.21.15.16:0/78349542 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f759810bd40 0x7f7598101040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.762+0000 7f759d4c8640 1 -- 172.21.15.16:0/78349542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7598103290 msgr2=0x7f7598100b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:17.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.762+0000 7f759d4c8640 1 --2- 172.21.15.16:0/78349542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7598103290 0x7f7598100b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.762+0000 7f759d4c8640 1 -- 172.21.15.16:0/78349542 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75981a64d0 con 0x7f7598102970 2024-03-28T22:15:17.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.762+0000 7f759ccc7640 1 --2- 172.21.15.16:0/78349542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7598103290 0x7f7598100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:15:17.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.762+0000 7f759d4c8640 1 --2- 172.21.15.16:0/78349542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7598102970 0x7f75981005c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f759400e9f0 tx=0x7f759400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:17.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.762+0000 7f758e7fc640 1 -- 172.21.15.16:0/78349542 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f759400cd80 con 0x7f7598102970 2024-03-28T22:15:17.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.762+0000 7f759f72c640 1 -- 172.21.15.16:0/78349542 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75981a67c0 con 0x7f7598102970 2024-03-28T22:15:17.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.762+0000 7f758e7fc640 1 -- 172.21.15.16:0/78349542 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75940035f0 con 0x7f7598102970 2024-03-28T22:15:17.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.762+0000 7f758e7fc640 1 -- 172.21.15.16:0/78349542 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7594010670 con 0x7f7598102970 2024-03-28T22:15:17.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.762+0000 7f759f72c640 1 -- 172.21.15.16:0/78349542 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75981a6d30 con 0x7f7598102970 2024-03-28T22:15:17.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.764+0000 7f758e7fc640 1 -- 172.21.15.16:0/78349542 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f75940037a0 con 0x7f7598102970 2024-03-28T22:15:17.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.765+0000 7f75a23ec740 1 Processor -- start 2024-03-28T22:15:17.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.765+0000 7f75a23ec740 1 -- start start 2024-03-28T22:15:17.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.765+0000 7f75a23ec740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643fd6499f0 0x5643fd649dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:17.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.765+0000 7f75a23ec740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643fd64a310 0x5643fd6529b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:17.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.765+0000 7f758e7fc640 1 --2- 172.21.15.16:0/78349542 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f75840a1f20 0x7f75840a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:17.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.765+0000 7f75a23ec740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643fd652ef0 0x5643fd6552e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:17.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.765+0000 7f75a23ec740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5643fd54a0b0 con 0x5643fd64a310 2024-03-28T22:15:17.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.765+0000 7f75a23ec740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5643fd5acea0 con 0x5643fd6499f0 2024-03-28T22:15:17.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.765+0000 7f75a23ec740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5643fd555c00 con 0x5643fd652ef0 2024-03-28T22:15:17.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.765+0000 7f759ccc7640 1 --2- 172.21.15.16:0/78349542 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f75840a1f20 0x7f75840a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:17.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.765+0000 7f759d4c8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643fd64a310 0x5643fd6529b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:17.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.765+0000 7f759d4c8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643fd64a310 0x5643fd6529b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59584/0 (socket says 172.21.15.16:59584) 2024-03-28T22:15:17.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.765+0000 7f759d4c8640 1 -- 172.21.15.16:0/1570140578 learned_addr learned my addr 172.21.15.16:0/1570140578 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:17.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.765+0000 7f759dcc9640 1 --2- 172.21.15.16:0/1570140578 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643fd6499f0 0x5643fd649dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:17.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.765+0000 7f758e7fc640 1 -- 172.21.15.16:0/78349542 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f75940ce4f0 con 0x7f7598102970 2024-03-28T22:15:17.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.765+0000 7f759dcc9640 1 --2- 172.21.15.16:0/1570140578 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643fd652ef0 0x5643fd6552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:17.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.766+0000 7f759d4c8640 1 -- 172.21.15.16:0/1570140578 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643fd652ef0 msgr2=0x5643fd6552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:17.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.766+0000 7f759d4c8640 1 --2- 172.21.15.16:0/1570140578 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643fd652ef0 0x5643fd6552e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.766+0000 7f759d4c8640 1 -- 172.21.15.16:0/1570140578 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643fd6499f0 msgr2=0x5643fd649dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:17.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.766+0000 7f759d4c8640 1 --2- 172.21.15.16:0/1570140578 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643fd6499f0 0x5643fd649dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.766+0000 7f759d4c8640 1 -- 172.21.15.16:0/1570140578 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643fd3897e0 con 0x5643fd64a310 2024-03-28T22:15:17.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.766+0000 7f759ccc7640 1 --2- 172.21.15.16:0/78349542 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f75840a1f20 0x7f75840a43e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7588000c00 tx=0x7f758802cf90 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:17.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.766+0000 7f759d4c8640 1 --2- 172.21.15.16:0/1570140578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643fd64a310 0x5643fd6529b0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f759401f000 tx=0x7f759409aaa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bf6f7e99641dcfc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:17.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.766+0000 7f75837fe640 1 -- 172.21.15.16:0/1570140578 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f759409eff0 con 0x5643fd64a310 2024-03-28T22:15:17.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.766+0000 7f75837fe640 1 -- 172.21.15.16:0/1570140578 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75940a3640 con 0x5643fd64a310 2024-03-28T22:15:17.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.766+0000 7f75837fe640 1 -- 172.21.15.16:0/1570140578 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75940750b0 con 0x5643fd64a310 2024-03-28T22:15:17.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.767+0000 7f75a23ec740 1 -- 172.21.15.16:0/1570140578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643fd64a310 msgr2=0x5643fd6529b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:17.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.767+0000 7f75a23ec740 1 --2- 172.21.15.16:0/1570140578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643fd64a310 0x5643fd6529b0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f759401f000 tx=0x7f759409aaa0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.767+0000 7f75a23ec740 1 -- 172.21.15.16:0/1570140578 shutdown_connections 2024-03-28T22:15:17.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.767+0000 7f75a23ec740 1 --2- 172.21.15.16:0/1570140578 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643fd652ef0 0x5643fd6552e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.767+0000 7f75a23ec740 1 --2- 172.21.15.16:0/1570140578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643fd64a310 0x5643fd6529b0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.767+0000 7f75a23ec740 1 --2- 172.21.15.16:0/1570140578 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643fd6499f0 0x5643fd649dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.767+0000 7f75a23ec740 1 -- 172.21.15.16:0/1570140578 >> 172.21.15.16:0/1570140578 conn(0x5643fd644120 msgr2=0x5643fd644500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:17.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.767+0000 7f75a23ec740 1 -- 172.21.15.16:0/1570140578 shutdown_connections 2024-03-28T22:15:17.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.767+0000 7f75a23ec740 1 -- 172.21.15.16:0/1570140578 wait complete. 2024-03-28T22:15:17.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.767+0000 7f75a23ec740 1 -- start start 2024-03-28T22:15:17.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.767+0000 7f75a23ec740 1 Processor -- start 2024-03-28T22:15:17.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.768+0000 7f75a23ec740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643fd6499f0 0x5643fd64fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:17.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.768+0000 7f75a23ec740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643fd64a310 0x5643fd650430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:17.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.768+0000 7f75a23ec740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643fd652ef0 0x5643fd650970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:17.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.768+0000 7f75a23ec740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5643fd54b3d0 con 0x5643fd64a310 2024-03-28T22:15:17.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.768+0000 7f75a23ec740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5643fd5a8f40 con 0x5643fd6499f0 2024-03-28T22:15:17.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.768+0000 7f75a23ec740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5643fd655ce0 con 0x5643fd652ef0 2024-03-28T22:15:17.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.768+0000 7f75a23ec740 10 client.0 _ll_register_callbacks cb 0x5643fd568b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:15:17.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.768+0000 7f759ccc7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643fd64a310 0x5643fd650430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:17.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.768+0000 7f759ccc7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643fd64a310 0x5643fd650430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59594/0 (socket says 172.21.15.16:59594) 2024-03-28T22:15:17.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.768+0000 7f759ccc7640 1 -- 172.21.15.16:0/2285716607 learned_addr learned my addr 172.21.15.16:0/2285716607 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:17.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.768+0000 7f759d4c8640 1 --2- 172.21.15.16:0/2285716607 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643fd6499f0 0x5643fd64fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:17.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.768+0000 7f759dcc9640 1 --2- 172.21.15.16:0/2285716607 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643fd652ef0 0x5643fd650970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:17.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.768+0000 7f759ccc7640 1 -- 172.21.15.16:0/2285716607 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643fd652ef0 msgr2=0x5643fd650970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:17.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.768+0000 7f759ccc7640 1 --2- 172.21.15.16:0/2285716607 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643fd652ef0 0x5643fd650970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.768+0000 7f759ccc7640 1 -- 172.21.15.16:0/2285716607 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643fd6499f0 msgr2=0x5643fd64fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:17.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.768+0000 7f759ccc7640 1 --2- 172.21.15.16:0/2285716607 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643fd6499f0 0x5643fd64fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.768+0000 7f759ccc7640 1 -- 172.21.15.16:0/2285716607 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643fd3e9f70 con 0x5643fd64a310 2024-03-28T22:15:17.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.768+0000 7f759d4c8640 1 --2- 172.21.15.16:0/2285716607 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643fd6499f0 0x5643fd64fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:15:17.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.768+0000 7f759ccc7640 1 --2- 172.21.15.16:0/2285716607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643fd64a310 0x5643fd650430 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f758802c8c0 tx=0x7f7588009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4066ec3548a0aefa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:17.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.768+0000 7f7580ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:15:17.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.768+0000 7f7580ff9640 1 -- 172.21.15.16:0/2285716607 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f758804f070 con 0x5643fd64a310 2024-03-28T22:15:17.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.768+0000 7f7580ff9640 1 -- 172.21.15.16:0/2285716607 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f75880050c0 con 0x5643fd64a310 2024-03-28T22:15:17.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.768+0000 7f75a23ec740 1 -- 172.21.15.16:0/2285716607 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75940a3640 con 0x5643fd64a310 2024-03-28T22:15:17.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.769+0000 7f7580ff9640 1 -- 172.21.15.16:0/2285716607 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f758805b050 con 0x5643fd64a310 2024-03-28T22:15:17.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.769+0000 7f75a23ec740 10 client.16524 fetch_fsmap learned FSMap version 27 2024-03-28T22:15:17.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.769+0000 7f75a23ec740 1 -- 172.21.15.16:0/2285716607 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5643fd281170 con 0x5643fd64a310 2024-03-28T22:15:17.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.769+0000 7f7580ff9640 1 -- 172.21.15.16:0/2285716607 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75940a3640 con 0x5643fd64a310 2024-03-28T22:15:17.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.769+0000 7f7580ff9640 1 -- 172.21.15.16:0/2285716607 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f758804b450 con 0x5643fd64a310 2024-03-28T22:15:17.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.769+0000 7f75a23ec740 10 client.16524 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:15:17.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.769+0000 7f75a23ec740 10 client.16524 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:15:17.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.769+0000 7f75a23ec740 20 client.16524 populate_metadata read hostname 'smithi016' 2024-03-28T22:15:17.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.769+0000 7f75a23ec740 1 --2- 172.21.15.16:0/2285716607 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5643fd662cd0 0x5643fd665100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:17.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.769+0000 7f75a23ec740 4 client.16524 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:15:17.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.769+0000 7f75a23ec740 1 -- 172.21.15.16:0/2285716607 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5643fd5ba3e0 con 0x5643fd662cd0 2024-03-28T22:15:17.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.769+0000 7f759d4c8640 1 --2- 172.21.15.16:0/2285716607 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5643fd662cd0 0x5643fd665100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:17.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.770+0000 7f759d4c8640 1 --2- 172.21.15.16:0/2285716607 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5643fd662cd0 0x5643fd665100 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a17fc3508fe3599e server_cookie=e470432afad627e3 in_seq=0 out_seq=0 2024-03-28T22:15:17.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.770+0000 7f7580ff9640 10 client.16524 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:15:17.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.770+0000 7f7580ff9640 1 -- 172.21.15.16:0/2285716607 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5643fd5ba3e0 con 0x5643fd662cd0 2024-03-28T22:15:17.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.770+0000 7f7580ff9640 10 client.16524 handle_command_reply: tid=0 2024-03-28T22:15:17.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.770+0000 7f75a23ec740 1 client.16524 shutdown 2024-03-28T22:15:17.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.770+0000 7f75a23ec740 1 -- 172.21.15.16:0/2285716607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643fd64a310 msgr2=0x5643fd650430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:17.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.770+0000 7f75a23ec740 1 --2- 172.21.15.16:0/2285716607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643fd64a310 0x5643fd650430 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f758802c8c0 tx=0x7f7588009260 comp rx=0 tx=0).stop 2024-03-28T22:15:17.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.770+0000 7f75a23ec740 1 -- 172.21.15.16:0/2285716607 shutdown_connections 2024-03-28T22:15:17.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.770+0000 7f75a23ec740 1 --2- 172.21.15.16:0/2285716607 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5643fd662cd0 0x5643fd665100 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.770+0000 7f75a23ec740 1 --2- 172.21.15.16:0/2285716607 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643fd652ef0 0x5643fd650970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.770+0000 7f75a23ec740 1 --2- 172.21.15.16:0/2285716607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643fd64a310 0x5643fd650430 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.770+0000 7f7580ff9640 0 client.16524 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:15:17.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.770+0000 7f75a23ec740 1 --2- 172.21.15.16:0/2285716607 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643fd6499f0 0x5643fd64fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.770+0000 7f75a23ec740 1 -- 172.21.15.16:0/2285716607 >> 172.21.15.16:0/2285716607 conn(0x5643fd644120 msgr2=0x5643fd645e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:17.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.771+0000 7f75a23ec740 1 -- 172.21.15.16:0/2285716607 shutdown_connections 2024-03-28T22:15:17.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.771+0000 7f75a23ec740 1 -- 172.21.15.16:0/2285716607 wait complete. 2024-03-28T22:15:17.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.771+0000 7f75a23ec740 20 client.16524 trim_cache size 0 max 16384 2024-03-28T22:15:17.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.788+0000 7f75a23ec740 1 Processor -- start 2024-03-28T22:15:17.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.789+0000 7f75a23ec740 1 -- start start 2024-03-28T22:15:17.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.789+0000 7f75a23ec740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643fd6499f0 0x5643fd652620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:17.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.789+0000 7f75a23ec740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643fd64a310 0x5643fd65e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:17.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.789+0000 7f75a23ec740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643fd652ef0 0x5643fd66aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:17.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.789+0000 7f75a23ec740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f75981b1ea0 con 0x5643fd652ef0 2024-03-28T22:15:17.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.789+0000 7f75a23ec740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f75981af770 con 0x5643fd64a310 2024-03-28T22:15:17.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.789+0000 7f75a23ec740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f75981ad5c0 con 0x5643fd6499f0 2024-03-28T22:15:17.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.789+0000 7f759dcc9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643fd652ef0 0x5643fd66aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:17.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.789+0000 7f759dcc9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643fd652ef0 0x5643fd66aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59608/0 (socket says 172.21.15.16:59608) 2024-03-28T22:15:17.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.789+0000 7f759dcc9640 1 -- 172.21.15.16:0/1552029217 learned_addr learned my addr 172.21.15.16:0/1552029217 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:17.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.789+0000 7f759ccc7640 1 --2- 172.21.15.16:0/1552029217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643fd64a310 0x5643fd65e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:17.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.789+0000 7f759dcc9640 1 -- 172.21.15.16:0/1552029217 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643fd6499f0 msgr2=0x5643fd652620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:17.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.789+0000 7f759dcc9640 1 --2- 172.21.15.16:0/1552029217 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643fd6499f0 0x5643fd652620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.789+0000 7f759dcc9640 1 -- 172.21.15.16:0/1552029217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643fd64a310 msgr2=0x5643fd65e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:17.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.789+0000 7f759dcc9640 1 --2- 172.21.15.16:0/1552029217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643fd64a310 0x5643fd65e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.789+0000 7f759dcc9640 1 -- 172.21.15.16:0/1552029217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643fd65eea0 con 0x5643fd652ef0 2024-03-28T22:15:17.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.789+0000 7f759dcc9640 1 --2- 172.21.15.16:0/1552029217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643fd652ef0 0x5643fd66aa60 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f7590009f50 tx=0x7f7590004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90ca5220e285009a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:17.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.789+0000 7f7582ffd640 1 -- 172.21.15.16:0/1552029217 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7590003e00 con 0x5643fd652ef0 2024-03-28T22:15:17.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.789+0000 7f7582ffd640 1 -- 172.21.15.16:0/1552029217 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7590012bf0 con 0x5643fd652ef0 2024-03-28T22:15:17.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.789+0000 7f7582ffd640 1 -- 172.21.15.16:0/1552029217 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7590011440 con 0x5643fd652ef0 2024-03-28T22:15:17.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.790+0000 7f75a23ec740 1 -- 172.21.15.16:0/1552029217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643fd652ef0 msgr2=0x5643fd66aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:17.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.790+0000 7f75a23ec740 1 --2- 172.21.15.16:0/1552029217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643fd652ef0 0x5643fd66aa60 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f7590009f50 tx=0x7f7590004ab0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.790+0000 7f75a23ec740 1 -- 172.21.15.16:0/1552029217 shutdown_connections 2024-03-28T22:15:17.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.790+0000 7f75a23ec740 1 --2- 172.21.15.16:0/1552029217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643fd652ef0 0x5643fd66aa60 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.790+0000 7f75a23ec740 1 --2- 172.21.15.16:0/1552029217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643fd64a310 0x5643fd65e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.790+0000 7f75a23ec740 1 --2- 172.21.15.16:0/1552029217 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643fd6499f0 0x5643fd652620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.790+0000 7f75a23ec740 1 -- 172.21.15.16:0/1552029217 >> 172.21.15.16:0/1552029217 conn(0x5643fd644120 msgr2=0x5643fd645500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:17.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.790+0000 7f75a23ec740 1 -- 172.21.15.16:0/1552029217 shutdown_connections 2024-03-28T22:15:17.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.790+0000 7f75a23ec740 1 -- 172.21.15.16:0/1552029217 wait complete. 2024-03-28T22:15:17.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.790+0000 7f75a23ec740 1 -- start start 2024-03-28T22:15:17.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.790+0000 7f75a23ec740 1 Processor -- start 2024-03-28T22:15:17.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.791+0000 7f75a23ec740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643fd6499f0 0x5643fd65b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:17.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.791+0000 7f75a23ec740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643fd64a310 0x5643fd657500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:17.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.791+0000 7f75a23ec740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643fd652ef0 0x5643fd657a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:17.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.791+0000 7f75a23ec740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f75981adc30 con 0x5643fd64a310 2024-03-28T22:15:17.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.791+0000 7f75a23ec740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f75981af0a0 con 0x5643fd6499f0 2024-03-28T22:15:17.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.791+0000 7f75a23ec740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f75981ae300 con 0x5643fd652ef0 2024-03-28T22:15:17.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.791+0000 7f75a23ec740 10 client.0 _ll_register_callbacks cb 0x7f7588009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:15:17.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.791+0000 7f759ccc7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643fd64a310 0x5643fd657500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:17.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.791+0000 7f759ccc7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643fd64a310 0x5643fd657500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59614/0 (socket says 172.21.15.16:59614) 2024-03-28T22:15:17.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.791+0000 7f759ccc7640 1 -- 172.21.15.16:0/995621725 learned_addr learned my addr 172.21.15.16:0/995621725 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:17.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.791+0000 7f759d4c8640 1 --2- 172.21.15.16:0/995621725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643fd6499f0 0x5643fd65b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:17.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.791+0000 7f759ccc7640 1 -- 172.21.15.16:0/995621725 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643fd652ef0 msgr2=0x5643fd657a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:17.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.791+0000 7f759dcc9640 1 --2- 172.21.15.16:0/995621725 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643fd652ef0 0x5643fd657a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:17.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.791+0000 7f759ccc7640 1 --2- 172.21.15.16:0/995621725 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643fd652ef0 0x5643fd657a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.791+0000 7f759ccc7640 1 -- 172.21.15.16:0/995621725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643fd6499f0 msgr2=0x5643fd65b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:17.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.791+0000 7f759ccc7640 1 --2- 172.21.15.16:0/995621725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643fd6499f0 0x5643fd65b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.791+0000 7f759ccc7640 1 -- 172.21.15.16:0/995621725 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643fd658240 con 0x5643fd64a310 2024-03-28T22:15:17.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.791+0000 7f759ccc7640 1 --2- 172.21.15.16:0/995621725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643fd64a310 0x5643fd657500 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f7588005ec0 tx=0x7f758804ba60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85c9ad88601b1616 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:17.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.791+0000 7f7580ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:15:17.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.792+0000 7f7580ff9640 1 -- 172.21.15.16:0/995621725 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7588063020 con 0x5643fd64a310 2024-03-28T22:15:17.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.792+0000 7f7580ff9640 1 -- 172.21.15.16:0/995621725 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f7588003a00 con 0x5643fd64a310 2024-03-28T22:15:17.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.792+0000 7f75a23ec740 1 -- 172.21.15.16:0/995621725 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7590012bf0 con 0x5643fd64a310 2024-03-28T22:15:17.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.792+0000 7f7580ff9640 1 -- 172.21.15.16:0/995621725 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f758806e050 con 0x5643fd64a310 2024-03-28T22:15:17.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.792+0000 7f75a23ec740 10 client.16530 fetch_fsmap learned FSMap version 27 2024-03-28T22:15:17.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.792+0000 7f75a23ec740 1 -- 172.21.15.16:0/995621725 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5643fd6ca030 con 0x5643fd64a310 2024-03-28T22:15:17.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.792+0000 7f7580ff9640 1 -- 172.21.15.16:0/995621725 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7590012bf0 con 0x5643fd64a310 2024-03-28T22:15:17.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.792+0000 7f7580ff9640 1 -- 172.21.15.16:0/995621725 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f7588054c90 con 0x5643fd64a310 2024-03-28T22:15:17.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.792+0000 7f75a23ec740 10 client.16530 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:15:17.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.792+0000 7f75a23ec740 10 client.16530 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:15:17.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.792+0000 7f75a23ec740 20 client.16530 populate_metadata read hostname 'smithi016' 2024-03-28T22:15:17.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.792+0000 7f75a23ec740 1 --2- 172.21.15.16:0/995621725 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5643fd6ca800 0x5643fd6ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:17.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.792+0000 7f75a23ec740 4 client.16530 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:15:17.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.792+0000 7f75a23ec740 1 -- 172.21.15.16:0/995621725 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5643fd5b4730 con 0x5643fd6ca800 2024-03-28T22:15:17.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.792+0000 7f759d4c8640 1 --2- 172.21.15.16:0/995621725 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5643fd6ca800 0x5643fd6ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:17.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.793+0000 7f759d4c8640 1 --2- 172.21.15.16:0/995621725 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5643fd6ca800 0x5643fd6ccc20 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d7e27c4b4d1be43 server_cookie=74950978f6ef67ce in_seq=0 out_seq=0 2024-03-28T22:15:17.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.793+0000 7f7580ff9640 10 client.16530 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:15:17.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.794+0000 7f7580ff9640 1 -- 172.21.15.16:0/995621725 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5643fd5b4730 con 0x5643fd6ca800 2024-03-28T22:15:17.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.794+0000 7f7580ff9640 10 client.16530 handle_command_reply: tid=0 2024-03-28T22:15:17.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.794+0000 7f75a23ec740 1 client.16530 shutdown 2024-03-28T22:15:17.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.794+0000 7f75a23ec740 1 -- 172.21.15.16:0/995621725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643fd64a310 msgr2=0x5643fd657500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:17.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.794+0000 7f75a23ec740 1 --2- 172.21.15.16:0/995621725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643fd64a310 0x5643fd657500 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f7588005ec0 tx=0x7f758804ba60 comp rx=0 tx=0).stop 2024-03-28T22:15:17.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.794+0000 7f75a23ec740 1 -- 172.21.15.16:0/995621725 shutdown_connections 2024-03-28T22:15:17.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.794+0000 7f75a23ec740 1 --2- 172.21.15.16:0/995621725 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5643fd6ca800 0x5643fd6ccc20 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.794+0000 7f7580ff9640 0 client.16530 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:15:17.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.794+0000 7f75a23ec740 1 --2- 172.21.15.16:0/995621725 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643fd652ef0 0x5643fd657a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.794+0000 7f75a23ec740 1 --2- 172.21.15.16:0/995621725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643fd64a310 0x5643fd657500 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.794+0000 7f75a23ec740 1 --2- 172.21.15.16:0/995621725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643fd6499f0 0x5643fd65b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.794+0000 7f75a23ec740 1 -- 172.21.15.16:0/995621725 >> 172.21.15.16:0/995621725 conn(0x5643fd644120 msgr2=0x5643fd650720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:17.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.794+0000 7f75a23ec740 1 -- 172.21.15.16:0/995621725 shutdown_connections 2024-03-28T22:15:17.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.794+0000 7f75a23ec740 1 -- 172.21.15.16:0/995621725 wait complete. 2024-03-28T22:15:17.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.794+0000 7f75a23ec740 20 client.16530 trim_cache size 0 max 16384 2024-03-28T22:15:17.807 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:15:17.807 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-03-28T22:15:17.807 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "32763064-9768-4788-b8b1-855f11c4a142", 2024-03-28T22:15:17.807 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-03-28T22:15:17.807 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:15:17.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.795+0000 7f759f72c640 1 -- 172.21.15.16:0/78349542 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f75840a1f20 msgr2=0x7f75840a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:17.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.795+0000 7f759f72c640 1 --2- 172.21.15.16:0/78349542 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f75840a1f20 0x7f75840a43e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7588000c00 tx=0x7f758802cf90 comp rx=0 tx=0).stop 2024-03-28T22:15:17.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.795+0000 7f759f72c640 1 -- 172.21.15.16:0/78349542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7598102970 msgr2=0x7f75981005c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:17.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.795+0000 7f759f72c640 1 --2- 172.21.15.16:0/78349542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7598102970 0x7f75981005c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f759400e9f0 tx=0x7f759400eeb0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.795+0000 7f759f72c640 1 -- 172.21.15.16:0/78349542 shutdown_connections 2024-03-28T22:15:17.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.795+0000 7f759f72c640 1 --2- 172.21.15.16:0/78349542 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f75840a1f20 0x7f75840a43e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.795+0000 7f759f72c640 1 --2- 172.21.15.16:0/78349542 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f759810bd40 0x7f7598101040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.795+0000 7f759f72c640 1 --2- 172.21.15.16:0/78349542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7598103290 0x7f7598100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.795+0000 7f759f72c640 1 --2- 172.21.15.16:0/78349542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7598102970 0x7f75981005c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.795+0000 7f759f72c640 1 -- 172.21.15.16:0/78349542 >> 172.21.15.16:0/78349542 conn(0x7f7598077770 msgr2=0x7f75980fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:17.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.795+0000 7f759f72c640 1 -- 172.21.15.16:0/78349542 shutdown_connections 2024-03-28T22:15:17.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.795+0000 7f759f72c640 1 -- 172.21.15.16:0/78349542 wait complete. 2024-03-28T22:15:17.810 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:15:17.810 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:15:17.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.914+0000 7f4fb1f13640 1 -- 172.21.15.16:0/2063134761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4fac1009d0 msgr2=0x7f4fac100db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:17.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.914+0000 7f4fb1f13640 1 --2- 172.21.15.16:0/2063134761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4fac1009d0 0x7f4fac100db0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c009900 tx=0x7f4f9c02bd10 comp rx=0 tx=0).stop 2024-03-28T22:15:17.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.914+0000 7f4fb1f13640 1 -- 172.21.15.16:0/2063134761 shutdown_connections 2024-03-28T22:15:17.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.914+0000 7f4fb1f13640 1 --2- 172.21.15.16:0/2063134761 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fac102b80 0x7f4fac10d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.914+0000 7f4fb1f13640 1 --2- 172.21.15.16:0/2063134761 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4fac1021e0 0x7f4fac102640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.914+0000 7f4fb1f13640 1 --2- 172.21.15.16:0/2063134761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4fac1009d0 0x7f4fac100db0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.914+0000 7f4fb1f13640 1 -- 172.21.15.16:0/2063134761 >> 172.21.15.16:0/2063134761 conn(0x7f4fac077830 msgr2=0x7f4fac077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:17.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.914+0000 7f4fb1f13640 1 -- 172.21.15.16:0/2063134761 shutdown_connections 2024-03-28T22:15:17.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.914+0000 7f4fb1f13640 1 -- 172.21.15.16:0/2063134761 wait complete. 2024-03-28T22:15:17.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.915+0000 7f4fb1f13640 1 Processor -- start 2024-03-28T22:15:17.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.915+0000 7f4fb1f13640 1 -- start start 2024-03-28T22:15:17.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.915+0000 7f4fb1f13640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fac1009d0 0x7f4fac198e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:17.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.915+0000 7f4fb1f13640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4fac1021e0 0x7f4fac199360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:17.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.915+0000 7f4fb1f13640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4fac102b80 0x7f4fac1a3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:17.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.915+0000 7f4fb1f13640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4fac110b20 con 0x7f4fac102b80 2024-03-28T22:15:17.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.915+0000 7f4fb1f13640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4fac1109a0 con 0x7f4fac1021e0 2024-03-28T22:15:17.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.915+0000 7f4fb1f13640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4fac110ca0 con 0x7f4fac1009d0 2024-03-28T22:15:17.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.915+0000 7f4fabfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4fac102b80 0x7f4fac1a3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:17.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.915+0000 7f4fabfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4fac102b80 0x7f4fac1a3ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59618/0 (socket says 172.21.15.16:59618) 2024-03-28T22:15:17.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.915+0000 7f4fabfff640 1 -- 172.21.15.16:0/608910282 learned_addr learned my addr 172.21.15.16:0/608910282 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:17.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.915+0000 7f4faaffd640 1 --2- 172.21.15.16:0/608910282 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4fac1021e0 0x7f4fac199360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:17.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.915+0000 7f4fab7fe640 1 --2- 172.21.15.16:0/608910282 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fac1009d0 0x7f4fac198e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:17.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.915+0000 7f4fabfff640 1 -- 172.21.15.16:0/608910282 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fac1009d0 msgr2=0x7f4fac198e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:17.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.915+0000 7f4fabfff640 1 --2- 172.21.15.16:0/608910282 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fac1009d0 0x7f4fac198e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.915+0000 7f4fabfff640 1 -- 172.21.15.16:0/608910282 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4fac1021e0 msgr2=0x7f4fac199360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:17.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.915+0000 7f4fabfff640 1 --2- 172.21.15.16:0/608910282 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4fac1021e0 0x7f4fac199360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.915+0000 7f4fabfff640 1 -- 172.21.15.16:0/608910282 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fac1a43e0 con 0x7f4fac102b80 2024-03-28T22:15:17.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.916+0000 7f4fabfff640 1 --2- 172.21.15.16:0/608910282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4fac102b80 0x7f4fac1a3ea0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f4fa400cab0 tx=0x7f4fa400cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:17.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.916+0000 7f4fa8ff9640 1 -- 172.21.15.16:0/608910282 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa400ae20 con 0x7f4fac102b80 2024-03-28T22:15:17.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.916+0000 7f4fa8ff9640 1 -- 172.21.15.16:0/608910282 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4fa400f930 con 0x7f4fac102b80 2024-03-28T22:15:17.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.916+0000 7f4fb1f13640 1 -- 172.21.15.16:0/608910282 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fac1a45b0 con 0x7f4fac102b80 2024-03-28T22:15:17.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.916+0000 7f4fa8ff9640 1 -- 172.21.15.16:0/608910282 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa4018b00 con 0x7f4fac102b80 2024-03-28T22:15:17.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.916+0000 7f4fb1f13640 1 -- 172.21.15.16:0/608910282 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fac1a4a50 con 0x7f4fac102b80 2024-03-28T22:15:17.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.918+0000 7f4fa8ff9640 1 -- 172.21.15.16:0/608910282 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f4fa400fae0 con 0x7f4fac102b80 2024-03-28T22:15:17.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.918+0000 7f4fb4bd2740 1 Processor -- start 2024-03-28T22:15:17.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.918+0000 7f4fb4bd2740 1 -- start start 2024-03-28T22:15:17.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.918+0000 7f4fa8ff9640 1 --2- 172.21.15.16:0/608910282 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4f940a1ef0 0x7f4f940a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:17.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.919+0000 7f4fb4bd2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fa5d5e290 0x558fa5d5e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:17.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.919+0000 7f4faaffd640 1 --2- 172.21.15.16:0/608910282 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4f940a1ef0 0x7f4f940a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:17.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.919+0000 7f4fb4bd2740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa5d5ebb0 0x558fa5d67250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:17.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.919+0000 7f4fa8ff9640 1 -- 172.21.15.16:0/608910282 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f4fa40ce6e0 con 0x7f4fac102b80 2024-03-28T22:15:17.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.919+0000 7f4fb4bd2740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558fa5d67790 0x558fa5d69b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:17.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.919+0000 7f4fb4bd2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558fa5cc2180 con 0x558fa5d5e290 2024-03-28T22:15:17.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.919+0000 7f4fb4bd2740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x558fa5c6ab70 con 0x558fa5d67790 2024-03-28T22:15:17.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.919+0000 7f4fb4bd2740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x558fa5cbe220 con 0x558fa5d5ebb0 2024-03-28T22:15:17.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.919+0000 7f4fab7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fa5d5e290 0x558fa5d5e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:17.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.919+0000 7f4fab7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fa5d5e290 0x558fa5d5e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59630/0 (socket says 172.21.15.16:59630) 2024-03-28T22:15:17.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.919+0000 7f4fab7fe640 1 -- 172.21.15.16:0/3780676913 learned_addr learned my addr 172.21.15.16:0/3780676913 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:17.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.919+0000 7f4fabfff640 1 --2- 172.21.15.16:0/3780676913 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558fa5d67790 0x558fa5d69b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:17.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.919+0000 7f4faaffd640 1 --2- 172.21.15.16:0/3780676913 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa5d5ebb0 0x558fa5d67250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:17.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.919+0000 7f4fab7fe640 1 -- 172.21.15.16:0/3780676913 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa5d5ebb0 msgr2=0x558fa5d67250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:17.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.919+0000 7f4fab7fe640 1 --2- 172.21.15.16:0/3780676913 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa5d5ebb0 0x558fa5d67250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.919+0000 7f4fab7fe640 1 -- 172.21.15.16:0/3780676913 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558fa5d67790 msgr2=0x558fa5d69b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:17.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.919+0000 7f4fab7fe640 1 --2- 172.21.15.16:0/3780676913 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558fa5d67790 0x558fa5d69b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.919+0000 7f4fab7fe640 1 -- 172.21.15.16:0/3780676913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558fa5c7e7c0 con 0x558fa5d5e290 2024-03-28T22:15:17.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.919+0000 7f4faaffd640 1 --2- 172.21.15.16:0/608910282 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4f940a1ef0 0x7f4f940a43b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4fac06a910 tx=0x7f4fa000b7d0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:17.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.919+0000 7f4fab7fe640 1 --2- 172.21.15.16:0/3780676913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fa5d5e290 0x558fa5d5e670 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c009900 tx=0x7f4f9c02c700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2a0a77a5826dd54 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:17.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.919+0000 7f4f91ffb640 1 -- 172.21.15.16:0/3780676913 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f9c043070 con 0x558fa5d5e290 2024-03-28T22:15:17.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.919+0000 7f4f91ffb640 1 -- 172.21.15.16:0/3780676913 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f9c02c930 con 0x558fa5d5e290 2024-03-28T22:15:17.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.919+0000 7f4f91ffb640 1 -- 172.21.15.16:0/3780676913 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f9c004c40 con 0x558fa5d5e290 2024-03-28T22:15:17.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.920+0000 7f4fb4bd2740 1 -- 172.21.15.16:0/3780676913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fa5d5e290 msgr2=0x558fa5d5e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:17.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.920+0000 7f4fb4bd2740 1 --2- 172.21.15.16:0/3780676913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fa5d5e290 0x558fa5d5e670 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c009900 tx=0x7f4f9c02c700 comp rx=0 tx=0).stop 2024-03-28T22:15:17.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.920+0000 7f4fb4bd2740 1 -- 172.21.15.16:0/3780676913 shutdown_connections 2024-03-28T22:15:17.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.920+0000 7f4fb4bd2740 1 --2- 172.21.15.16:0/3780676913 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558fa5d67790 0x558fa5d69b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.920+0000 7f4fb4bd2740 1 --2- 172.21.15.16:0/3780676913 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa5d5ebb0 0x558fa5d67250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.920+0000 7f4fb4bd2740 1 --2- 172.21.15.16:0/3780676913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fa5d5e290 0x558fa5d5e670 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.920+0000 7f4fb4bd2740 1 -- 172.21.15.16:0/3780676913 >> 172.21.15.16:0/3780676913 conn(0x558fa5d589c0 msgr2=0x558fa5d58da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:17.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.920+0000 7f4fb4bd2740 1 -- 172.21.15.16:0/3780676913 shutdown_connections 2024-03-28T22:15:17.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.920+0000 7f4fb4bd2740 1 -- 172.21.15.16:0/3780676913 wait complete. 2024-03-28T22:15:17.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.920+0000 7f4fb4bd2740 1 -- start start 2024-03-28T22:15:17.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.921+0000 7f4fb4bd2740 1 Processor -- start 2024-03-28T22:15:17.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.921+0000 7f4fb4bd2740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa5d5e290 0x558fa5d647c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:17.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.921+0000 7f4fb4bd2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fa5d5ebb0 0x558fa5d64d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:17.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.921+0000 7f4fb4bd2740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558fa5d67790 0x558fa5d65240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:17.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.921+0000 7f4fb4bd2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558fa5d6a520 con 0x558fa5d5ebb0 2024-03-28T22:15:17.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.921+0000 7f4fb4bd2740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x558fa5c604a0 con 0x558fa5d67790 2024-03-28T22:15:17.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.921+0000 7f4fb4bd2740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x558fa5d6a6a0 con 0x558fa5d5e290 2024-03-28T22:15:17.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.921+0000 7f4fb4bd2740 10 client.0 _ll_register_callbacks cb 0x558fa5c63d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:15:17.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.921+0000 7f4faaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fa5d5ebb0 0x558fa5d64d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:17.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.921+0000 7f4faaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fa5d5ebb0 0x558fa5d64d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59632/0 (socket says 172.21.15.16:59632) 2024-03-28T22:15:17.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.921+0000 7f4faaffd640 1 -- 172.21.15.16:0/3545560060 learned_addr learned my addr 172.21.15.16:0/3545560060 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:17.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.921+0000 7f4fabfff640 1 --2- 172.21.15.16:0/3545560060 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558fa5d67790 0x558fa5d65240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:17.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.921+0000 7f4fab7fe640 1 --2- 172.21.15.16:0/3545560060 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa5d5e290 0x558fa5d647c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:17.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.921+0000 7f4faaffd640 1 -- 172.21.15.16:0/3545560060 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa5d5e290 msgr2=0x558fa5d647c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:17.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.921+0000 7f4faaffd640 1 --2- 172.21.15.16:0/3545560060 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa5d5e290 0x558fa5d647c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.921+0000 7f4faaffd640 1 -- 172.21.15.16:0/3545560060 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558fa5d67790 msgr2=0x558fa5d65240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:17.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.921+0000 7f4faaffd640 1 --2- 172.21.15.16:0/3545560060 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558fa5d67790 0x558fa5d65240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.921+0000 7f4faaffd640 1 -- 172.21.15.16:0/3545560060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558fa5aa07d0 con 0x558fa5d5ebb0 2024-03-28T22:15:17.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.922+0000 7f4faaffd640 1 --2- 172.21.15.16:0/3545560060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fa5d5ebb0 0x558fa5d64d00 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f4fa001e7c0 tx=0x7f4fa0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a864fd1edaba803c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:17.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.922+0000 7f4f8affd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:15:17.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.922+0000 7f4f8affd640 1 -- 172.21.15.16:0/3545560060 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa002b060 con 0x558fa5d5ebb0 2024-03-28T22:15:17.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.922+0000 7f4fb4bd2740 1 -- 172.21.15.16:0/3545560060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f9c02c930 con 0x558fa5d5ebb0 2024-03-28T22:15:17.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.922+0000 7f4f8affd640 1 -- 172.21.15.16:0/3545560060 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f4fa001c800 con 0x558fa5d5ebb0 2024-03-28T22:15:17.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.922+0000 7f4f8affd640 1 -- 172.21.15.16:0/3545560060 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4fa001ca00 con 0x558fa5d5ebb0 2024-03-28T22:15:17.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.922+0000 7f4fb4bd2740 10 client.16542 fetch_fsmap learned FSMap version 27 2024-03-28T22:15:17.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.922+0000 7f4fb4bd2740 1 -- 172.21.15.16:0/3545560060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558fa59d63f0 con 0x558fa5d5ebb0 2024-03-28T22:15:17.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.922+0000 7f4f8affd640 1 -- 172.21.15.16:0/3545560060 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f9c02c930 con 0x558fa5d5ebb0 2024-03-28T22:15:17.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.922+0000 7f4f8affd640 1 -- 172.21.15.16:0/3545560060 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f4fa000f3c0 con 0x558fa5d5ebb0 2024-03-28T22:15:17.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.922+0000 7f4fb4bd2740 10 client.16542 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:15:17.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.922+0000 7f4fb4bd2740 10 client.16542 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:15:17.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.922+0000 7f4fb4bd2740 20 client.16542 populate_metadata read hostname 'smithi016' 2024-03-28T22:15:17.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.922+0000 7f4fb4bd2740 1 --2- 172.21.15.16:0/3545560060 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558fa5d77b60 0x558fa5d79f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:17.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.922+0000 7f4fb4bd2740 4 client.16542 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:15:17.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.922+0000 7f4fb4bd2740 1 -- 172.21.15.16:0/3545560060 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558fa5ccf6c0 con 0x558fa5d77b60 2024-03-28T22:15:17.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.922+0000 7f4fab7fe640 1 --2- 172.21.15.16:0/3545560060 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558fa5d77b60 0x558fa5d79f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:17.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.923+0000 7f4f8affd640 10 client.16542 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:15:17.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.923+0000 7f4fab7fe640 1 --2- 172.21.15.16:0/3545560060 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558fa5d77b60 0x558fa5d79f80 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d1d48be46b1969e server_cookie=1a360e713d049dd9 in_seq=0 out_seq=0 2024-03-28T22:15:17.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.924+0000 7f4f8affd640 1 -- 172.21.15.16:0/3545560060 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558fa5ccf6c0 con 0x558fa5d77b60 2024-03-28T22:15:17.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.924+0000 7f4f8affd640 10 client.16542 handle_command_reply: tid=0 2024-03-28T22:15:17.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.924+0000 7f4fb4bd2740 1 client.16542 shutdown 2024-03-28T22:15:17.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.924+0000 7f4fb4bd2740 1 -- 172.21.15.16:0/3545560060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fa5d5ebb0 msgr2=0x558fa5d64d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:17.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.924+0000 7f4fb4bd2740 1 --2- 172.21.15.16:0/3545560060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fa5d5ebb0 0x558fa5d64d00 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f4fa001e7c0 tx=0x7f4fa0004730 comp rx=0 tx=0).stop 2024-03-28T22:15:17.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.924+0000 7f4fb4bd2740 1 -- 172.21.15.16:0/3545560060 shutdown_connections 2024-03-28T22:15:17.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.924+0000 7f4fb4bd2740 1 --2- 172.21.15.16:0/3545560060 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558fa5d77b60 0x558fa5d79f80 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.924+0000 7f4fb4bd2740 1 --2- 172.21.15.16:0/3545560060 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558fa5d67790 0x558fa5d65240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.924+0000 7f4f8affd640 0 client.16542 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:15:17.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.924+0000 7f4fb4bd2740 1 --2- 172.21.15.16:0/3545560060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fa5d5ebb0 0x558fa5d64d00 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.924+0000 7f4fb4bd2740 1 --2- 172.21.15.16:0/3545560060 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa5d5e290 0x558fa5d647c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.924+0000 7f4fb4bd2740 1 -- 172.21.15.16:0/3545560060 >> 172.21.15.16:0/3545560060 conn(0x558fa5d589c0 msgr2=0x558fa5d5a710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:17.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.924+0000 7f4fb4bd2740 1 -- 172.21.15.16:0/3545560060 shutdown_connections 2024-03-28T22:15:17.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.924+0000 7f4fb4bd2740 1 -- 172.21.15.16:0/3545560060 wait complete. 2024-03-28T22:15:17.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.924+0000 7f4fb4bd2740 20 client.16542 trim_cache size 0 max 16384 2024-03-28T22:15:17.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.942+0000 7f4fb4bd2740 1 Processor -- start 2024-03-28T22:15:17.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.942+0000 7f4fb4bd2740 1 -- start start 2024-03-28T22:15:17.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.942+0000 7f4fb4bd2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fa5d5e290 0x558fa5d66ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:17.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.942+0000 7f4fb4bd2740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa5d5ebb0 0x558fa5d732c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:17.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.942+0000 7f4fb4bd2740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558fa5d67790 0x558fa5d7f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:17.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.942+0000 7f4fb4bd2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4fac1afa90 con 0x558fa5d5e290 2024-03-28T22:15:17.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.942+0000 7f4fb4bd2740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4fac1ad360 con 0x558fa5d67790 2024-03-28T22:15:17.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.942+0000 7f4fb4bd2740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4fac1ab1b0 con 0x558fa5d5ebb0 2024-03-28T22:15:17.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.942+0000 7f4fab7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fa5d5e290 0x558fa5d66ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:17.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.942+0000 7f4fab7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fa5d5e290 0x558fa5d66ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59642/0 (socket says 172.21.15.16:59642) 2024-03-28T22:15:17.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.942+0000 7f4fab7fe640 1 -- 172.21.15.16:0/2644251116 learned_addr learned my addr 172.21.15.16:0/2644251116 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:17.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.942+0000 7f4fabfff640 1 --2- 172.21.15.16:0/2644251116 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558fa5d67790 0x558fa5d7f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:17.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.942+0000 7f4faaffd640 1 --2- 172.21.15.16:0/2644251116 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa5d5ebb0 0x558fa5d732c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:17.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.942+0000 7f4fab7fe640 1 -- 172.21.15.16:0/2644251116 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa5d5ebb0 msgr2=0x558fa5d732c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:17.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.942+0000 7f4fab7fe640 1 --2- 172.21.15.16:0/2644251116 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa5d5ebb0 0x558fa5d732c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.942+0000 7f4fab7fe640 1 -- 172.21.15.16:0/2644251116 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558fa5d67790 msgr2=0x558fa5d7f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:17.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.942+0000 7f4fab7fe640 1 --2- 172.21.15.16:0/2644251116 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558fa5d67790 0x558fa5d7f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.942+0000 7f4fab7fe640 1 -- 172.21.15.16:0/2644251116 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558fa59960f0 con 0x558fa5d5e290 2024-03-28T22:15:17.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.942+0000 7f4fab7fe640 1 --2- 172.21.15.16:0/2644251116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fa5d5e290 0x558fa5d66ef0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c036040 tx=0x7f4f9c02bd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f436df10ac05b8e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:17.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.943+0000 7f4f917fa640 1 -- 172.21.15.16:0/2644251116 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f9c043070 con 0x558fa5d5e290 2024-03-28T22:15:17.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.943+0000 7f4f917fa640 1 -- 172.21.15.16:0/2644251116 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f9c02c930 con 0x558fa5d5e290 2024-03-28T22:15:17.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.943+0000 7f4f917fa640 1 -- 172.21.15.16:0/2644251116 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f9c005360 con 0x558fa5d5e290 2024-03-28T22:15:17.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.943+0000 7f4fb4bd2740 1 -- 172.21.15.16:0/2644251116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fa5d5e290 msgr2=0x558fa5d66ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:17.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.943+0000 7f4fb4bd2740 1 --2- 172.21.15.16:0/2644251116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fa5d5e290 0x558fa5d66ef0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c036040 tx=0x7f4f9c02bd10 comp rx=0 tx=0).stop 2024-03-28T22:15:17.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.943+0000 7f4fb4bd2740 1 -- 172.21.15.16:0/2644251116 shutdown_connections 2024-03-28T22:15:17.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.943+0000 7f4fb4bd2740 1 --2- 172.21.15.16:0/2644251116 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558fa5d67790 0x558fa5d7f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.943+0000 7f4fb4bd2740 1 --2- 172.21.15.16:0/2644251116 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa5d5ebb0 0x558fa5d732c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.943+0000 7f4fb4bd2740 1 --2- 172.21.15.16:0/2644251116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fa5d5e290 0x558fa5d66ef0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.943+0000 7f4fb4bd2740 1 -- 172.21.15.16:0/2644251116 >> 172.21.15.16:0/2644251116 conn(0x558fa5d589c0 msgr2=0x558fa5d57c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:17.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.943+0000 7f4fb4bd2740 1 -- 172.21.15.16:0/2644251116 shutdown_connections 2024-03-28T22:15:17.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.943+0000 7f4fb4bd2740 1 -- 172.21.15.16:0/2644251116 wait complete. 2024-03-28T22:15:17.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.944+0000 7f4fb4bd2740 1 -- start start 2024-03-28T22:15:17.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.944+0000 7f4fb4bd2740 1 Processor -- start 2024-03-28T22:15:17.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.944+0000 7f4fb4bd2740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa5d5e290 0x558fa5d6b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:17.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.944+0000 7f4fb4bd2740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558fa5d5ebb0 0x558fa5d6ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:17.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.944+0000 7f4fb4bd2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fa5d67790 0x558fa5d6bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:17.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.944+0000 7f4fb4bd2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4fac1ab820 con 0x558fa5d67790 2024-03-28T22:15:17.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.944+0000 7f4fb4bd2740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4fac1acc90 con 0x558fa5d5ebb0 2024-03-28T22:15:17.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.944+0000 7f4fb4bd2740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4fac1abef0 con 0x558fa5d5e290 2024-03-28T22:15:17.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.944+0000 7f4fb4bd2740 10 client.0 _ll_register_callbacks cb 0x7f4fa000bd70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:15:17.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.944+0000 7f4fabfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fa5d67790 0x558fa5d6bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:17.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.944+0000 7f4fab7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa5d5e290 0x558fa5d6b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:17.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.944+0000 7f4fabfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fa5d67790 0x558fa5d6bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59644/0 (socket says 172.21.15.16:59644) 2024-03-28T22:15:17.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.944+0000 7f4fabfff640 1 -- 172.21.15.16:0/339512094 learned_addr learned my addr 172.21.15.16:0/339512094 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:17.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.944+0000 7f4fab7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa5d5e290 0x558fa5d6b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.16:44936/0 (socket says 172.21.15.16:44936) 2024-03-28T22:15:17.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.944+0000 7f4faaffd640 1 --2- 172.21.15.16:0/339512094 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558fa5d5ebb0 0x558fa5d6ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:17.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.944+0000 7f4fabfff640 1 -- 172.21.15.16:0/339512094 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa5d5e290 msgr2=0x558fa5d6b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:17.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.944+0000 7f4fabfff640 1 --2- 172.21.15.16:0/339512094 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa5d5e290 0x558fa5d6b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.945+0000 7f4fabfff640 1 -- 172.21.15.16:0/339512094 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558fa5d5ebb0 msgr2=0x558fa5d6ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:17.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.945+0000 7f4fabfff640 1 --2- 172.21.15.16:0/339512094 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558fa5d5ebb0 0x558fa5d6ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.945+0000 7f4fabfff640 1 -- 172.21.15.16:0/339512094 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558fa5de1510 con 0x558fa5d67790 2024-03-28T22:15:17.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.945+0000 7f4fabfff640 1 --2- 172.21.15.16:0/339512094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fa5d67790 0x558fa5d6bf50 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f4fa409b080 tx=0x7f4fa409aa50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b6beeb2e85a5f96 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:17.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.945+0000 7f4f8affd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:15:17.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.945+0000 7f4f8affd640 1 -- 172.21.15.16:0/339512094 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa40b7020 con 0x558fa5d67790 2024-03-28T22:15:17.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.945+0000 7f4f8affd640 1 -- 172.21.15.16:0/339512094 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f4fa40a3400 con 0x558fa5d67790 2024-03-28T22:15:17.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.945+0000 7f4fb4bd2740 1 -- 172.21.15.16:0/339512094 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f9c02c930 con 0x558fa5d67790 2024-03-28T22:15:17.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.945+0000 7f4f8affd640 1 -- 172.21.15.16:0/339512094 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4fa409f240 con 0x558fa5d67790 2024-03-28T22:15:17.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.945+0000 7f4fb4bd2740 10 client.16548 fetch_fsmap learned FSMap version 27 2024-03-28T22:15:17.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.945+0000 7f4fb4bd2740 1 -- 172.21.15.16:0/339512094 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558fa5de2880 con 0x558fa5d67790 2024-03-28T22:15:17.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.945+0000 7f4f8affd640 1 -- 172.21.15.16:0/339512094 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f9c02c930 con 0x558fa5d67790 2024-03-28T22:15:17.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.945+0000 7f4f8affd640 1 -- 172.21.15.16:0/339512094 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f4fa40b92c0 con 0x558fa5d67790 2024-03-28T22:15:17.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.945+0000 7f4fb4bd2740 10 client.16548 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:15:17.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.945+0000 7f4fb4bd2740 10 client.16548 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:15:17.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.945+0000 7f4fb4bd2740 20 client.16548 populate_metadata read hostname 'smithi016' 2024-03-28T22:15:17.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.945+0000 7f4fb4bd2740 1 --2- 172.21.15.16:0/339512094 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558fa5de2f80 0x558fa5de53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:17.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.946+0000 7f4fb4bd2740 4 client.16548 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:15:17.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.946+0000 7f4fb4bd2740 1 -- 172.21.15.16:0/339512094 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558fa5cc9a10 con 0x558fa5de2f80 2024-03-28T22:15:17.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.946+0000 7f4fab7fe640 1 --2- 172.21.15.16:0/339512094 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558fa5de2f80 0x558fa5de53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:17.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.946+0000 7f4fab7fe640 1 --2- 172.21.15.16:0/339512094 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558fa5de2f80 0x558fa5de53a0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1cd419f9de9acf7d server_cookie=b0d2fe20150a97d1 in_seq=0 out_seq=0 2024-03-28T22:15:17.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.946+0000 7f4f8affd640 10 client.16548 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:15:17.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.946+0000 7f4f8affd640 1 -- 172.21.15.16:0/339512094 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x558fa5cc9a10 con 0x558fa5de2f80 2024-03-28T22:15:17.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.946+0000 7f4f8affd640 10 client.16548 handle_command_reply: tid=0 2024-03-28T22:15:17.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.946+0000 7f4fb4bd2740 1 client.16548 shutdown 2024-03-28T22:15:17.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.947+0000 7f4fb4bd2740 1 -- 172.21.15.16:0/339512094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fa5d67790 msgr2=0x558fa5d6bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:17.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.947+0000 7f4fb4bd2740 1 --2- 172.21.15.16:0/339512094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fa5d67790 0x558fa5d6bf50 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f4fa409b080 tx=0x7f4fa409aa50 comp rx=0 tx=0).stop 2024-03-28T22:15:17.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.947+0000 7f4fb4bd2740 1 -- 172.21.15.16:0/339512094 shutdown_connections 2024-03-28T22:15:17.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.947+0000 7f4fb4bd2740 1 --2- 172.21.15.16:0/339512094 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558fa5de2f80 0x558fa5de53a0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.947+0000 7f4fb4bd2740 1 --2- 172.21.15.16:0/339512094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fa5d67790 0x558fa5d6bf50 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.947+0000 7f4fb4bd2740 1 --2- 172.21.15.16:0/339512094 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558fa5d5ebb0 0x558fa5d6ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.947+0000 7f4f8affd640 0 client.16548 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:15:17.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.947+0000 7f4fb4bd2740 1 --2- 172.21.15.16:0/339512094 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa5d5e290 0x558fa5d6b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.947+0000 7f4fb4bd2740 1 -- 172.21.15.16:0/339512094 >> 172.21.15.16:0/339512094 conn(0x558fa5d589c0 msgr2=0x558fa5d64780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:17.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.947+0000 7f4fb4bd2740 1 -- 172.21.15.16:0/339512094 shutdown_connections 2024-03-28T22:15:17.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.947+0000 7f4fb4bd2740 1 -- 172.21.15.16:0/339512094 wait complete. 2024-03-28T22:15:17.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.947+0000 7f4fb4bd2740 20 client.16548 trim_cache size 0 max 16384 2024-03-28T22:15:17.961 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:15:17.962 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T22:15:17.962 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-03-28T22:15:17.962 INFO:teuthology.orchestra.run.smithi016.stdout: "32763064-9768-4788-b8b1-855f11c4a142": { 2024-03-28T22:15:17.962 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-03-28T22:15:17.962 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "32763064-9768-4788-b8b1-855f11c4a142", 2024-03-28T22:15:17.962 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-03-28T22:15:17.962 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:15:17.963 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:15:17.963 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:15:17.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.947+0000 7f4fb1f13640 1 -- 172.21.15.16:0/608910282 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4f940a1ef0 msgr2=0x7f4f940a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:17.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.947+0000 7f4fb1f13640 1 --2- 172.21.15.16:0/608910282 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4f940a1ef0 0x7f4f940a43b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4fac06a910 tx=0x7f4fa000b7d0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.947+0000 7f4fb1f13640 1 -- 172.21.15.16:0/608910282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4fac102b80 msgr2=0x7f4fac1a3ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:17.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.947+0000 7f4fb1f13640 1 --2- 172.21.15.16:0/608910282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4fac102b80 0x7f4fac1a3ea0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f4fa400cab0 tx=0x7f4fa400cf70 comp rx=0 tx=0).stop 2024-03-28T22:15:17.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.947+0000 7f4fb1f13640 1 -- 172.21.15.16:0/608910282 shutdown_connections 2024-03-28T22:15:17.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.947+0000 7f4fb1f13640 1 --2- 172.21.15.16:0/608910282 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4f940a1ef0 0x7f4f940a43b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.947+0000 7f4fb1f13640 1 --2- 172.21.15.16:0/608910282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4fac102b80 0x7f4fac1a3ea0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.947+0000 7f4fb1f13640 1 --2- 172.21.15.16:0/608910282 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4fac1021e0 0x7f4fac199360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.947+0000 7f4fb1f13640 1 --2- 172.21.15.16:0/608910282 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fac1009d0 0x7f4fac198e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:17.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.947+0000 7f4fb1f13640 1 -- 172.21.15.16:0/608910282 >> 172.21.15.16:0/608910282 conn(0x7f4fac077830 msgr2=0x7f4fac1042c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:17.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.947+0000 7f4fb1f13640 1 -- 172.21.15.16:0/608910282 shutdown_connections 2024-03-28T22:15:17.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:17.947+0000 7f4fb1f13640 1 -- 172.21.15.16:0/608910282 wait complete. 2024-03-28T22:15:17.966 INFO:tasks.cephfs.filesystem:scrub status for tag:32763064-9768-4788-b8b1-855f11c4a142 - {'path': '/', 'tag': '32763064-9768-4788-b8b1-855f11c4a142', 'options': 'recursive,force'} 2024-03-28T22:15:18.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:18 smithi016 ceph-mon[32374]: pgmap v144: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 7 op/s 2024-03-28T22:15:18.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:18 smithi016 ceph-mon[32374]: scrub queued for path: / 2024-03-28T22:15:18.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:18 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:15:18.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:18 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:15:18.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:18 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:15:18.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:18 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:15:18.437 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:18 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:15:18.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:18 smithi053 ceph-mon[35979]: pgmap v144: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 7 op/s 2024-03-28T22:15:18.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:18 smithi053 ceph-mon[35979]: scrub queued for path: / 2024-03-28T22:15:18.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:18 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:15:18.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:18 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:15:18.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:18 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:15:18.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:18 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:15:18.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:18 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:15:18.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:18 smithi063 ceph-mon[36249]: pgmap v144: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 7 op/s 2024-03-28T22:15:18.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:18 smithi063 ceph-mon[36249]: scrub queued for path: / 2024-03-28T22:15:18.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:18 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:15:18.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:18 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:15:18.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:18 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:15:18.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:18 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:15:18.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:18 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:15:19.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:19 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:15:19.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:19 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:15:19.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:19 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:15:19.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:19 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:15:19.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:19 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:15:19.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:19 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:15:19.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:19 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:15:19.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:19 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:15:19.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:19 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:15:19.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:19 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:15:19.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:19 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:15:19.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:19 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:15:20.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:20 smithi053 ceph-mon[35979]: pgmap v145: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-28T22:15:20.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:20 smithi063 ceph-mon[36249]: pgmap v145: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-28T22:15:20.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:20 smithi016 ceph-mon[32374]: pgmap v145: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-28T22:15:22.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:22 smithi053 ceph-mon[35979]: pgmap v146: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-28T22:15:22.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:22 smithi063 ceph-mon[36249]: pgmap v146: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-28T22:15:22.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:22 smithi016 ceph-mon[32374]: pgmap v146: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-28T22:15:24.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:24 smithi053 ceph-mon[35979]: pgmap v147: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:15:24.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:24 smithi063 ceph-mon[36249]: pgmap v147: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:15:24.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:24 smithi016 ceph-mon[32374]: pgmap v147: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:15:26.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:26 smithi053 ceph-mon[35979]: pgmap v148: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:15:26.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:26 smithi063 ceph-mon[36249]: pgmap v148: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:15:26.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:26 smithi016 ceph-mon[32374]: pgmap v148: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:15:28.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:28 smithi053 ceph-mon[35979]: pgmap v149: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:15:28.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:28 smithi063 ceph-mon[36249]: pgmap v149: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:15:28.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:28 smithi016 ceph-mon[32374]: pgmap v149: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:15:30.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:30 smithi053 ceph-mon[35979]: pgmap v150: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:15:30.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:30 smithi053 ceph-mon[35979]: scrub summary: idle 2024-03-28T22:15:30.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:30 smithi063 ceph-mon[36249]: pgmap v150: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:15:30.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:30 smithi063 ceph-mon[36249]: scrub summary: idle 2024-03-28T22:15:30.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:30 smithi016 ceph-mon[32374]: pgmap v150: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:15:30.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:30 smithi016 ceph-mon[32374]: scrub summary: idle 2024-03-28T22:15:31.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:31 smithi053 ceph-mon[35979]: scrub summary: idle 2024-03-28T22:15:31.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:31 smithi063 ceph-mon[36249]: scrub summary: idle 2024-03-28T22:15:31.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:31 smithi016 ceph-mon[32374]: scrub summary: idle 2024-03-28T22:15:32.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:32 smithi053 ceph-mon[35979]: pgmap v151: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:15:32.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:32 smithi063 ceph-mon[36249]: pgmap v151: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:15:32.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:32 smithi016 ceph-mon[32374]: pgmap v151: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:15:34.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:34 smithi053 ceph-mon[35979]: pgmap v152: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:15:34.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:34 smithi063 ceph-mon[36249]: pgmap v152: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:15:34.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:34 smithi016 ceph-mon[32374]: pgmap v152: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:15:36.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:36 smithi053 ceph-mon[35979]: pgmap v153: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:15:36.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:36 smithi063 ceph-mon[36249]: pgmap v153: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:15:36.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:36 smithi016 ceph-mon[32374]: pgmap v153: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:15:38.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:38 smithi053 ceph-mon[35979]: pgmap v154: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:15:38.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:38 smithi063 ceph-mon[36249]: pgmap v154: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:15:38.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:38 smithi016 ceph-mon[32374]: pgmap v154: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:15:40.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:40 smithi053 ceph-mon[35979]: pgmap v155: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:15:40.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:40 smithi063 ceph-mon[36249]: pgmap v155: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:15:40.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:40 smithi016 ceph-mon[32374]: pgmap v155: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:15:42.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:42 smithi053 ceph-mon[35979]: pgmap v156: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:15:42.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:42 smithi063 ceph-mon[36249]: pgmap v156: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:15:42.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:42 smithi016 ceph-mon[32374]: pgmap v156: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:15:44.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:44 smithi053 ceph-mon[35979]: pgmap v157: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:15:44.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:44 smithi063 ceph-mon[36249]: pgmap v157: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:15:44.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:44 smithi016 ceph-mon[32374]: pgmap v157: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:15:45.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:45 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:15:45.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:45 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:15:45.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:45 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:15:46.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:46 smithi053 ceph-mon[35979]: pgmap v158: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:15:46.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:46 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:15:46.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:46 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:15:46.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:46 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:15:46.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:46 smithi063 ceph-mon[36249]: pgmap v158: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:15:46.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:46 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:15:46.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:46 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:15:46.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:46 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:15:46.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:46 smithi016 ceph-mon[32374]: pgmap v158: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:15:46.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:46 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:15:46.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:46 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:15:46.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:46 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:15:47.967 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:15:47.968 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:15:48.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.083+0000 7f0345993640 1 -- 172.21.15.16:0/475021670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f034010bd40 msgr2=0x7f034010e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.083+0000 7f0345993640 1 --2- 172.21.15.16:0/475021670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f034010bd40 0x7f034010e170 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f0338009ca0 tx=0x7f033802c180 comp rx=0 tx=0).stop 2024-03-28T22:15:48.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.083+0000 7f0345993640 1 -- 172.21.15.16:0/475021670 shutdown_connections 2024-03-28T22:15:48.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.083+0000 7f0345993640 1 --2- 172.21.15.16:0/475021670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f034010bd40 0x7f034010e170 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.083+0000 7f0345993640 1 --2- 172.21.15.16:0/475021670 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0340103290 0x7f034010b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.083+0000 7f0345993640 1 --2- 172.21.15.16:0/475021670 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0340102970 0x7f0340102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.083+0000 7f0345993640 1 -- 172.21.15.16:0/475021670 >> 172.21.15.16:0/475021670 conn(0x7f0340077800 msgr2=0x7f0340077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:48.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.083+0000 7f0345993640 1 -- 172.21.15.16:0/475021670 shutdown_connections 2024-03-28T22:15:48.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.083+0000 7f0345993640 1 -- 172.21.15.16:0/475021670 wait complete. 2024-03-28T22:15:48.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.083+0000 7f0345993640 1 Processor -- start 2024-03-28T22:15:48.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.083+0000 7f0345993640 1 -- start start 2024-03-28T22:15:48.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.084+0000 7f0345993640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0340102970 0x7f0340197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.084+0000 7f0345993640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0340103290 0x7f03401984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.084+0000 7f0345993640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f034010bd40 0x7f034019c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.084+0000 7f0345993640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0340111ad0 con 0x7f0340103290 2024-03-28T22:15:48.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.084+0000 7f0345993640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0340111950 con 0x7f0340102970 2024-03-28T22:15:48.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.084+0000 7f0345993640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0340111c50 con 0x7f034010bd40 2024-03-28T22:15:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.084+0000 7f033e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0340103290 0x7f03401984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.084+0000 7f033effd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0340102970 0x7f0340197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.084+0000 7f033e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0340103290 0x7f03401984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56862/0 (socket says 172.21.15.16:56862) 2024-03-28T22:15:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.084+0000 7f033e7fc640 1 -- 172.21.15.16:0/1071083477 learned_addr learned my addr 172.21.15.16:0/1071083477 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.084+0000 7f033f7fe640 1 --2- 172.21.15.16:0/1071083477 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f034010bd40 0x7f034019c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.084+0000 7f033effd640 1 -- 172.21.15.16:0/1071083477 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f034010bd40 msgr2=0x7f034019c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.084+0000 7f033effd640 1 --2- 172.21.15.16:0/1071083477 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f034010bd40 0x7f034019c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.084+0000 7f033effd640 1 -- 172.21.15.16:0/1071083477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0340103290 msgr2=0x7f03401984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.084+0000 7f033effd640 1 --2- 172.21.15.16:0/1071083477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0340103290 0x7f03401984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.084+0000 7f033effd640 1 -- 172.21.15.16:0/1071083477 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f034019cf60 con 0x7f0340102970 2024-03-28T22:15:48.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.084+0000 7f033effd640 1 --2- 172.21.15.16:0/1071083477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0340102970 0x7f0340197f70 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f033000c970 tx=0x7f033000ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:48.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.084+0000 7f0344991640 1 -- 172.21.15.16:0/1071083477 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0330007bb0 con 0x7f0340102970 2024-03-28T22:15:48.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.085+0000 7f0345993640 1 -- 172.21.15.16:0/1071083477 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f034019d250 con 0x7f0340102970 2024-03-28T22:15:48.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.085+0000 7f0344991640 1 -- 172.21.15.16:0/1071083477 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0330007d50 con 0x7f0340102970 2024-03-28T22:15:48.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.085+0000 7f0344991640 1 -- 172.21.15.16:0/1071083477 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0330004e60 con 0x7f0340102970 2024-03-28T22:15:48.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.085+0000 7f0345993640 1 -- 172.21.15.16:0/1071083477 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03401a4ae0 con 0x7f0340102970 2024-03-28T22:15:48.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.087+0000 7f0344991640 1 -- 172.21.15.16:0/1071083477 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f0330005000 con 0x7f0340102970 2024-03-28T22:15:48.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.087+0000 7f0348570740 1 Processor -- start 2024-03-28T22:15:48.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.087+0000 7f0348570740 1 -- start start 2024-03-28T22:15:48.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.087+0000 7f0348570740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c976361fc0 0x55c9763623a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.087+0000 7f0344991640 1 --2- 172.21.15.16:0/1071083477 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f03280a1ef0 0x7f03280a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.087+0000 7f0348570740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c9763628e0 0x55c97636f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.087+0000 7f0348570740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c97636fba0 0x55c976371f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.087+0000 7f0348570740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c9762c6180 con 0x55c97636fba0 2024-03-28T22:15:48.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.087+0000 7f0348570740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55c97626eb70 con 0x55c9763628e0 2024-03-28T22:15:48.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.087+0000 7f0348570740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55c9762c2220 con 0x55c976361fc0 2024-03-28T22:15:48.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.088+0000 7f033effd640 1 --2- 172.21.15.16:0/1071083477 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f03280a1ef0 0x7f03280a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.088+0000 7f033e7fc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c9763628e0 0x55c97636f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.088+0000 7f033e7fc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c9763628e0 0x55c97636f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:48756/0 (socket says 172.21.15.16:48756) 2024-03-28T22:15:48.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.088+0000 7f033e7fc640 1 -- 172.21.15.16:0/1736220277 learned_addr learned my addr 172.21.15.16:0/1736220277 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:48.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.088+0000 7f033f7fe640 1 --2- 172.21.15.16:0/1736220277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c97636fba0 0x55c976371f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.088+0000 7f0344991640 1 -- 172.21.15.16:0/1071083477 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f03300cddd0 con 0x7f0340102970 2024-03-28T22:15:48.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.088+0000 7f033f7fe640 1 --2- 172.21.15.16:0/1736220277 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c976361fc0 0x55c9763623a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.088+0000 7f033e7fc640 1 -- 172.21.15.16:0/1736220277 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c976361fc0 msgr2=0x55c9763623a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:48.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.088+0000 7f033e7fc640 1 --2- 172.21.15.16:0/1736220277 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c976361fc0 0x55c9763623a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.088+0000 7f033e7fc640 1 -- 172.21.15.16:0/1736220277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c97636fba0 msgr2=0x55c976371f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:48.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.088+0000 7f033e7fc640 1 --2- 172.21.15.16:0/1736220277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c97636fba0 0x55c976371f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.088+0000 7f033e7fc640 1 -- 172.21.15.16:0/1736220277 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9762827c0 con 0x55c9763628e0 2024-03-28T22:15:48.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.088+0000 7f033f7fe640 1 --2- 172.21.15.16:0/1736220277 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c976361fc0 0x55c9763623a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:15:48.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.088+0000 7f033effd640 1 --2- 172.21.15.16:0/1071083477 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f03280a1ef0 0x7f03280a43b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f03300cb5f0 tx=0x7f0330085040 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:48.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.088+0000 7f033e7fc640 1 --2- 172.21.15.16:0/1736220277 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c9763628e0 0x55c97636f660 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0334009cf0 tx=0x7f0334011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=176007b2e5b32c5f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:48.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.088+0000 7f0325ffb640 1 -- 172.21.15.16:0/1736220277 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0334005220 con 0x55c9763628e0 2024-03-28T22:15:48.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.088+0000 7f0325ffb640 1 -- 172.21.15.16:0/1736220277 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0334004c80 con 0x55c9763628e0 2024-03-28T22:15:48.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.088+0000 7f0325ffb640 1 -- 172.21.15.16:0/1736220277 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0334019900 con 0x55c9763628e0 2024-03-28T22:15:48.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.089+0000 7f0348570740 1 -- 172.21.15.16:0/1736220277 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c9763628e0 msgr2=0x55c97636f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.089+0000 7f0348570740 1 --2- 172.21.15.16:0/1736220277 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c9763628e0 0x55c97636f660 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0334009cf0 tx=0x7f0334011810 comp rx=0 tx=0).stop 2024-03-28T22:15:48.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.089+0000 7f0348570740 1 -- 172.21.15.16:0/1736220277 shutdown_connections 2024-03-28T22:15:48.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.089+0000 7f0348570740 1 --2- 172.21.15.16:0/1736220277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c97636fba0 0x55c976371f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.089+0000 7f0348570740 1 --2- 172.21.15.16:0/1736220277 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c9763628e0 0x55c97636f660 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.089+0000 7f0348570740 1 --2- 172.21.15.16:0/1736220277 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c976361fc0 0x55c9763623a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.089+0000 7f0348570740 1 -- 172.21.15.16:0/1736220277 >> 172.21.15.16:0/1736220277 conn(0x55c97635c9c0 msgr2=0x55c97635cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:48.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.089+0000 7f0348570740 1 -- 172.21.15.16:0/1736220277 shutdown_connections 2024-03-28T22:15:48.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.089+0000 7f0348570740 1 -- 172.21.15.16:0/1736220277 wait complete. 2024-03-28T22:15:48.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.089+0000 7f0348570740 1 -- start start 2024-03-28T22:15:48.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.090+0000 7f0348570740 1 Processor -- start 2024-03-28T22:15:48.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.090+0000 7f0348570740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c976361fc0 0x55c97636cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.090+0000 7f0348570740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9763628e0 0x55c97636d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.090+0000 7f0348570740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c97636fba0 0x55c97636d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.090+0000 7f0348570740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c976372960 con 0x55c9763628e0 2024-03-28T22:15:48.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.090+0000 7f0348570740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55c9762644a0 con 0x55c97636fba0 2024-03-28T22:15:48.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.090+0000 7f0348570740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55c976372ae0 con 0x55c976361fc0 2024-03-28T22:15:48.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.090+0000 7f0348570740 10 client.0 _ll_register_callbacks cb 0x55c976267d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:15:48.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.090+0000 7f033e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9763628e0 0x55c97636d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.090+0000 7f033e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9763628e0 0x55c97636d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56872/0 (socket says 172.21.15.16:56872) 2024-03-28T22:15:48.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.090+0000 7f033e7fc640 1 -- 172.21.15.16:0/2776533565 learned_addr learned my addr 172.21.15.16:0/2776533565 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:48.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.090+0000 7f033f7fe640 1 --2- 172.21.15.16:0/2776533565 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c97636fba0 0x55c97636d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.090+0000 7f033f7fe640 1 --2- 172.21.15.16:0/2776533565 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c976361fc0 0x55c97636cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.090+0000 7f033e7fc640 1 -- 172.21.15.16:0/2776533565 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c976361fc0 msgr2=0x55c97636cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:48.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.090+0000 7f033e7fc640 1 --2- 172.21.15.16:0/2776533565 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c976361fc0 0x55c97636cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.090+0000 7f033e7fc640 1 -- 172.21.15.16:0/2776533565 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c97636fba0 msgr2=0x55c97636d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:48.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.090+0000 7f033e7fc640 1 --2- 172.21.15.16:0/2776533565 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c97636fba0 0x55c97636d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.090+0000 7f033e7fc640 1 -- 172.21.15.16:0/2776533565 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9760a47d0 con 0x55c9763628e0 2024-03-28T22:15:48.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.090+0000 7f033f7fe640 1 --2- 172.21.15.16:0/2776533565 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c97636fba0 0x55c97636d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:15:48.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.091+0000 7f033e7fc640 1 --2- 172.21.15.16:0/2776533565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9763628e0 0x55c97636d140 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f03340195a0 tx=0x7f03340052b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91fccc620fab9b9e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:48.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.091+0000 7f031effd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:15:48.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.091+0000 7f031effd640 1 -- 172.21.15.16:0/2776533565 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0334009ea0 con 0x55c9763628e0 2024-03-28T22:15:48.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.091+0000 7f031effd640 1 -- 172.21.15.16:0/2776533565 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f0334004230 con 0x55c9763628e0 2024-03-28T22:15:48.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.091+0000 7f0348570740 1 -- 172.21.15.16:0/2776533565 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0334004c80 con 0x55c9763628e0 2024-03-28T22:15:48.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.091+0000 7f031effd640 1 -- 172.21.15.16:0/2776533565 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0334034050 con 0x55c9763628e0 2024-03-28T22:15:48.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.091+0000 7f0348570740 10 client.16560 fetch_fsmap learned FSMap version 27 2024-03-28T22:15:48.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.091+0000 7f0348570740 1 -- 172.21.15.16:0/2776533565 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c975fda3f0 con 0x55c9763628e0 2024-03-28T22:15:48.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.091+0000 7f031effd640 1 -- 172.21.15.16:0/2776533565 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0334004c80 con 0x55c9763628e0 2024-03-28T22:15:48.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.091+0000 7f031effd640 1 -- 172.21.15.16:0/2776533565 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f03340035f0 con 0x55c9763628e0 2024-03-28T22:15:48.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.091+0000 7f0348570740 10 client.16560 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:15:48.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.091+0000 7f0348570740 10 client.16560 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:15:48.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.091+0000 7f0348570740 20 client.16560 populate_metadata read hostname 'smithi016' 2024-03-28T22:15:48.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.091+0000 7f0348570740 1 --2- 172.21.15.16:0/2776533565 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55c97637fdc0 0x55c9763821b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.091+0000 7f0348570740 4 client.16560 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:15:48.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.091+0000 7f0348570740 1 -- 172.21.15.16:0/2776533565 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c9762d36c0 con 0x55c97637fdc0 2024-03-28T22:15:48.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.092+0000 7f033f7fe640 1 --2- 172.21.15.16:0/2776533565 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55c97637fdc0 0x55c9763821b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.092+0000 7f033f7fe640 1 --2- 172.21.15.16:0/2776533565 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55c97637fdc0 0x55c9763821b0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb8e5a40af58796e server_cookie=56f1978bc263d240 in_seq=0 out_seq=0 2024-03-28T22:15:48.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.092+0000 7f031effd640 10 client.16560 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:15:48.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.093+0000 7f031effd640 1 -- 172.21.15.16:0/2776533565 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c9762d36c0 con 0x55c97637fdc0 2024-03-28T22:15:48.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.093+0000 7f031effd640 10 client.16560 handle_command_reply: tid=0 2024-03-28T22:15:48.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.093+0000 7f0348570740 1 client.16560 shutdown 2024-03-28T22:15:48.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.093+0000 7f0348570740 1 -- 172.21.15.16:0/2776533565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9763628e0 msgr2=0x55c97636d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.093+0000 7f0348570740 1 --2- 172.21.15.16:0/2776533565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9763628e0 0x55c97636d140 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f03340195a0 tx=0x7f03340052b0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.093+0000 7f0348570740 1 -- 172.21.15.16:0/2776533565 shutdown_connections 2024-03-28T22:15:48.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.093+0000 7f0348570740 1 --2- 172.21.15.16:0/2776533565 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55c97637fdc0 0x55c9763821b0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.093+0000 7f0348570740 1 --2- 172.21.15.16:0/2776533565 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c97636fba0 0x55c97636d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.093+0000 7f0348570740 1 --2- 172.21.15.16:0/2776533565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9763628e0 0x55c97636d140 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.093+0000 7f0348570740 1 --2- 172.21.15.16:0/2776533565 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c976361fc0 0x55c97636cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.093+0000 7f031effd640 0 client.16560 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:15:48.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.093+0000 7f0348570740 1 -- 172.21.15.16:0/2776533565 >> 172.21.15.16:0/2776533565 conn(0x55c97635c9c0 msgr2=0x55c97635e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:48.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.093+0000 7f0348570740 1 -- 172.21.15.16:0/2776533565 shutdown_connections 2024-03-28T22:15:48.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.093+0000 7f0348570740 1 -- 172.21.15.16:0/2776533565 wait complete. 2024-03-28T22:15:48.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.093+0000 7f0348570740 20 client.16560 trim_cache size 0 max 16384 2024-03-28T22:15:48.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.110+0000 7f0348570740 1 Processor -- start 2024-03-28T22:15:48.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.110+0000 7f0348570740 1 -- start start 2024-03-28T22:15:48.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.111+0000 7f0348570740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c976361fc0 0x55c97636f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.111+0000 7f0348570740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c9763628e0 0x55c97637b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.111+0000 7f0348570740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c97636fba0 0x55c97635baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.111+0000 7f0348570740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f03401adb20 con 0x55c97636fba0 2024-03-28T22:15:48.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.111+0000 7f0348570740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f03401ab3f0 con 0x55c976361fc0 2024-03-28T22:15:48.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.111+0000 7f0348570740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f03401a9240 con 0x55c9763628e0 2024-03-28T22:15:48.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.111+0000 7f033f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c97636fba0 0x55c97635baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.111+0000 7f033f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c97636fba0 0x55c97635baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56888/0 (socket says 172.21.15.16:56888) 2024-03-28T22:15:48.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.111+0000 7f033f7fe640 1 -- 172.21.15.16:0/3931892496 learned_addr learned my addr 172.21.15.16:0/3931892496 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:48.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.111+0000 7f033f7fe640 1 --2- 172.21.15.16:0/3931892496 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c976361fc0 0x55c97636f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.111+0000 7f033f7fe640 1 -- 172.21.15.16:0/3931892496 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c9763628e0 msgr2=0x55c97637b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:48.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.111+0000 7f033f7fe640 1 --2- 172.21.15.16:0/3931892496 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c9763628e0 0x55c97637b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.111+0000 7f033f7fe640 1 -- 172.21.15.16:0/3931892496 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c976361fc0 msgr2=0x55c97636f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:48.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.111+0000 7f033f7fe640 1 --2- 172.21.15.16:0/3931892496 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c976361fc0 0x55c97636f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.111+0000 7f033f7fe640 1 -- 172.21.15.16:0/3931892496 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c975f9a0f0 con 0x55c97636fba0 2024-03-28T22:15:48.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.111+0000 7f033f7fe640 1 --2- 172.21.15.16:0/3931892496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c97636fba0 0x55c97635baa0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f033802c1c0 tx=0x7f033802e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc1342bab8b9c92a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:48.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.111+0000 7f03257fa640 1 -- 172.21.15.16:0/3931892496 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0338054070 con 0x55c97636fba0 2024-03-28T22:15:48.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.111+0000 7f03257fa640 1 -- 172.21.15.16:0/3931892496 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0338003e50 con 0x55c97636fba0 2024-03-28T22:15:48.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.111+0000 7f03257fa640 1 -- 172.21.15.16:0/3931892496 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0338039040 con 0x55c97636fba0 2024-03-28T22:15:48.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.112+0000 7f0348570740 1 -- 172.21.15.16:0/3931892496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c97636fba0 msgr2=0x55c97635baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.112+0000 7f0348570740 1 --2- 172.21.15.16:0/3931892496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c97636fba0 0x55c97635baa0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f033802c1c0 tx=0x7f033802e040 comp rx=0 tx=0).stop 2024-03-28T22:15:48.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.112+0000 7f0348570740 1 -- 172.21.15.16:0/3931892496 shutdown_connections 2024-03-28T22:15:48.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.112+0000 7f0348570740 1 --2- 172.21.15.16:0/3931892496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c97636fba0 0x55c97635baa0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.112+0000 7f0348570740 1 --2- 172.21.15.16:0/3931892496 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c9763628e0 0x55c97637b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.112+0000 7f0348570740 1 --2- 172.21.15.16:0/3931892496 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c976361fc0 0x55c97636f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.112+0000 7f0348570740 1 -- 172.21.15.16:0/3931892496 >> 172.21.15.16:0/3931892496 conn(0x55c97635c9c0 msgr2=0x55c97635b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:48.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.112+0000 7f0348570740 1 -- 172.21.15.16:0/3931892496 shutdown_connections 2024-03-28T22:15:48.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.112+0000 7f0348570740 1 -- 172.21.15.16:0/3931892496 wait complete. 2024-03-28T22:15:48.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.112+0000 7f0348570740 1 -- start start 2024-03-28T22:15:48.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.112+0000 7f0348570740 1 Processor -- start 2024-03-28T22:15:48.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.113+0000 7f0348570740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c976361fc0 0x55c976377d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.113+0000 7f0348570740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c9763628e0 0x55c976373100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.113+0000 7f0348570740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c97636fba0 0x55c976373640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.113+0000 7f0348570740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f03401a98b0 con 0x55c97636fba0 2024-03-28T22:15:48.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.113+0000 7f0348570740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f03401aad20 con 0x55c9763628e0 2024-03-28T22:15:48.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.113+0000 7f0348570740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f03401a9f80 con 0x55c976361fc0 2024-03-28T22:15:48.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.113+0000 7f0348570740 10 client.0 _ll_register_callbacks cb 0x7f034006afe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:15:48.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.113+0000 7f033f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c97636fba0 0x55c976373640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.113+0000 7f033f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c97636fba0 0x55c976373640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56890/0 (socket says 172.21.15.16:56890) 2024-03-28T22:15:48.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.113+0000 7f033f7fe640 1 -- 172.21.15.16:0/1594896303 learned_addr learned my addr 172.21.15.16:0/1594896303 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:48.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.113+0000 7f033e7fc640 1 --2- 172.21.15.16:0/1594896303 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c9763628e0 0x55c976373100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.113+0000 7f033f7fe640 1 --2- 172.21.15.16:0/1594896303 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c976361fc0 0x55c976377d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.113+0000 7f033f7fe640 1 -- 172.21.15.16:0/1594896303 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c976361fc0 msgr2=0x55c976377d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:48.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.113+0000 7f033f7fe640 1 --2- 172.21.15.16:0/1594896303 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c976361fc0 0x55c976377d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.113+0000 7f033f7fe640 1 -- 172.21.15.16:0/1594896303 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c9763628e0 msgr2=0x55c976373100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:48.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.113+0000 7f033f7fe640 1 --2- 172.21.15.16:0/1594896303 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c9763628e0 0x55c976373100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.113+0000 7f033f7fe640 1 -- 172.21.15.16:0/1594896303 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c976373bb0 con 0x55c97636fba0 2024-03-28T22:15:48.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.113+0000 7f033f7fe640 1 --2- 172.21.15.16:0/1594896303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c97636fba0 0x55c976373640 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f033802e010 tx=0x7f033804d350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6fd6a77f2bfedc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:48.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.113+0000 7f031effd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:15:48.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.113+0000 7f031effd640 1 -- 172.21.15.16:0/1594896303 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0338054070 con 0x55c97636fba0 2024-03-28T22:15:48.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.113+0000 7f031effd640 1 -- 172.21.15.16:0/1594896303 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f033804ed00 con 0x55c97636fba0 2024-03-28T22:15:48.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.113+0000 7f0348570740 1 -- 172.21.15.16:0/1594896303 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0338003e50 con 0x55c97636fba0 2024-03-28T22:15:48.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.114+0000 7f031effd640 1 -- 172.21.15.16:0/1594896303 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0338039040 con 0x55c97636fba0 2024-03-28T22:15:48.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.114+0000 7f0348570740 10 client.16566 fetch_fsmap learned FSMap version 27 2024-03-28T22:15:48.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.114+0000 7f0348570740 1 -- 172.21.15.16:0/1594896303 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c9763e5dd0 con 0x55c97636fba0 2024-03-28T22:15:48.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.114+0000 7f031effd640 1 -- 172.21.15.16:0/1594896303 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0338003e50 con 0x55c97636fba0 2024-03-28T22:15:48.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.114+0000 7f031effd640 1 -- 172.21.15.16:0/1594896303 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f0338061020 con 0x55c97636fba0 2024-03-28T22:15:48.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.114+0000 7f0348570740 10 client.16566 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:15:48.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.114+0000 7f0348570740 10 client.16566 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:15:48.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.114+0000 7f0348570740 20 client.16566 populate_metadata read hostname 'smithi016' 2024-03-28T22:15:48.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.114+0000 7f0348570740 1 --2- 172.21.15.16:0/1594896303 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55c9763e62f0 0x55c9763e8710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.114+0000 7f0348570740 4 client.16566 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:15:48.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.114+0000 7f0348570740 1 -- 172.21.15.16:0/1594896303 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c9762cda10 con 0x55c9763e62f0 2024-03-28T22:15:48.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.114+0000 7f033e7fc640 1 --2- 172.21.15.16:0/1594896303 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55c9763e62f0 0x55c9763e8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.115+0000 7f033e7fc640 1 --2- 172.21.15.16:0/1594896303 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55c9763e62f0 0x55c9763e8710 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee8f13482c3e79f server_cookie=dcd0d3d0b0c4a068 in_seq=0 out_seq=0 2024-03-28T22:15:48.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.115+0000 7f031effd640 10 client.16566 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:15:48.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.115+0000 7f031effd640 1 -- 172.21.15.16:0/1594896303 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c9762cda10 con 0x55c9763e62f0 2024-03-28T22:15:48.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.115+0000 7f031effd640 10 client.16566 handle_command_reply: tid=0 2024-03-28T22:15:48.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.115+0000 7f0348570740 1 client.16566 shutdown 2024-03-28T22:15:48.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.115+0000 7f0348570740 1 -- 172.21.15.16:0/1594896303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c97636fba0 msgr2=0x55c976373640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.115+0000 7f0348570740 1 --2- 172.21.15.16:0/1594896303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c97636fba0 0x55c976373640 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f033802e010 tx=0x7f033804d350 comp rx=0 tx=0).stop 2024-03-28T22:15:48.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.115+0000 7f0348570740 1 -- 172.21.15.16:0/1594896303 shutdown_connections 2024-03-28T22:15:48.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.115+0000 7f0348570740 1 --2- 172.21.15.16:0/1594896303 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55c9763e62f0 0x55c9763e8710 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.115+0000 7f0348570740 1 --2- 172.21.15.16:0/1594896303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c97636fba0 0x55c976373640 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.115+0000 7f0348570740 1 --2- 172.21.15.16:0/1594896303 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c9763628e0 0x55c976373100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.115+0000 7f031effd640 0 client.16566 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:15:48.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.115+0000 7f0348570740 1 --2- 172.21.15.16:0/1594896303 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c976361fc0 0x55c976377d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.115+0000 7f0348570740 1 -- 172.21.15.16:0/1594896303 >> 172.21.15.16:0/1594896303 conn(0x55c97635c9c0 msgr2=0x55c97635bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:48.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.115+0000 7f0348570740 1 -- 172.21.15.16:0/1594896303 shutdown_connections 2024-03-28T22:15:48.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.115+0000 7f0348570740 1 -- 172.21.15.16:0/1594896303 wait complete. 2024-03-28T22:15:48.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.115+0000 7f0348570740 20 client.16566 trim_cache size 0 max 16384 2024-03-28T22:15:48.129 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:15:48.129 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-03-28T22:15:48.130 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-03-28T22:15:48.130 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:15:48.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.116+0000 7f0345993640 1 -- 172.21.15.16:0/1071083477 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f03280a1ef0 msgr2=0x7f03280a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.116+0000 7f0345993640 1 --2- 172.21.15.16:0/1071083477 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f03280a1ef0 0x7f03280a43b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f03300cb5f0 tx=0x7f0330085040 comp rx=0 tx=0).stop 2024-03-28T22:15:48.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.116+0000 7f0345993640 1 -- 172.21.15.16:0/1071083477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0340102970 msgr2=0x7f0340197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.116+0000 7f0345993640 1 --2- 172.21.15.16:0/1071083477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0340102970 0x7f0340197f70 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f033000c970 tx=0x7f033000ce30 comp rx=0 tx=0).stop 2024-03-28T22:15:48.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.116+0000 7f0345993640 1 -- 172.21.15.16:0/1071083477 shutdown_connections 2024-03-28T22:15:48.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.116+0000 7f0345993640 1 --2- 172.21.15.16:0/1071083477 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f03280a1ef0 0x7f03280a43b0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.116+0000 7f0345993640 1 --2- 172.21.15.16:0/1071083477 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f034010bd40 0x7f034019c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.116+0000 7f0345993640 1 --2- 172.21.15.16:0/1071083477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0340103290 0x7f03401984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.116+0000 7f0345993640 1 --2- 172.21.15.16:0/1071083477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0340102970 0x7f0340197f70 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.116+0000 7f0345993640 1 -- 172.21.15.16:0/1071083477 >> 172.21.15.16:0/1071083477 conn(0x7f0340077800 msgr2=0x7f03400fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:48.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.116+0000 7f0345993640 1 -- 172.21.15.16:0/1071083477 shutdown_connections 2024-03-28T22:15:48.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.116+0000 7f0345993640 1 -- 172.21.15.16:0/1071083477 wait complete. 2024-03-28T22:15:48.133 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:15:48.133 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:15:48.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.233+0000 7fdcd3a07640 1 -- 172.21.15.16:0/1808357559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdccc106680 msgr2=0x7fdccc106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.233+0000 7fdcd3a07640 1 --2- 172.21.15.16:0/1808357559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdccc106680 0x7fdccc106a60 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fdcc4009930 tx=0x7fdcc402bc30 comp rx=0 tx=0).stop 2024-03-28T22:15:48.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.233+0000 7fdcd3a07640 1 -- 172.21.15.16:0/1808357559 shutdown_connections 2024-03-28T22:15:48.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.233+0000 7fdcd3a07640 1 --2- 172.21.15.16:0/1808357559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdccc100fa0 0x7fdccc10d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.233+0000 7fdcd3a07640 1 --2- 172.21.15.16:0/1808357559 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdccc1005e0 0x7fdccc100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.233+0000 7fdcd3a07640 1 --2- 172.21.15.16:0/1808357559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdccc106680 0x7fdccc106a60 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.233+0000 7fdcd3a07640 1 -- 172.21.15.16:0/1808357559 >> 172.21.15.16:0/1808357559 conn(0x7fdccc0fc0d0 msgr2=0x7fdccc0fe4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:48.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.234+0000 7fdcd3a07640 1 -- 172.21.15.16:0/1808357559 shutdown_connections 2024-03-28T22:15:48.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.234+0000 7fdcd3a07640 1 -- 172.21.15.16:0/1808357559 wait complete. 2024-03-28T22:15:48.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.234+0000 7fdcd3a07640 1 Processor -- start 2024-03-28T22:15:48.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.234+0000 7fdcd3a07640 1 -- start start 2024-03-28T22:15:48.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.234+0000 7fdcd3a07640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdccc1005e0 0x7fdccc197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.234+0000 7fdcd3a07640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdccc100fa0 0x7fdccc198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.234+0000 7fdcd3a07640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdccc106680 0x7fdccc19c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.234+0000 7fdcd3a07640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdccc1109c0 con 0x7fdccc100fa0 2024-03-28T22:15:48.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.234+0000 7fdcd3a07640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fdccc110840 con 0x7fdccc106680 2024-03-28T22:15:48.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.234+0000 7fdcd3a07640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fdccc110b40 con 0x7fdccc1005e0 2024-03-28T22:15:48.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.234+0000 7fdcd0fa2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdccc100fa0 0x7fdccc198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.234+0000 7fdcd0fa2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdccc100fa0 0x7fdccc198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56918/0 (socket says 172.21.15.16:56918) 2024-03-28T22:15:48.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.235+0000 7fdcd0fa2640 1 -- 172.21.15.16:0/550643783 learned_addr learned my addr 172.21.15.16:0/550643783 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:48.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.235+0000 7fdcd1fa4640 1 --2- 172.21.15.16:0/550643783 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdccc106680 0x7fdccc19c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.235+0000 7fdcd17a3640 1 --2- 172.21.15.16:0/550643783 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdccc1005e0 0x7fdccc197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.235+0000 7fdcd0fa2640 1 -- 172.21.15.16:0/550643783 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdccc1005e0 msgr2=0x7fdccc197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.235+0000 7fdcd0fa2640 1 --2- 172.21.15.16:0/550643783 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdccc1005e0 0x7fdccc197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.235+0000 7fdcd0fa2640 1 -- 172.21.15.16:0/550643783 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdccc106680 msgr2=0x7fdccc19c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.235+0000 7fdcd0fa2640 1 --2- 172.21.15.16:0/550643783 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdccc106680 0x7fdccc19c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.235+0000 7fdcd0fa2640 1 -- 172.21.15.16:0/550643783 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdccc19cfb0 con 0x7fdccc100fa0 2024-03-28T22:15:48.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.235+0000 7fdcd1fa4640 1 --2- 172.21.15.16:0/550643783 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdccc106680 0x7fdccc19c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:15:48.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.235+0000 7fdcd0fa2640 1 --2- 172.21.15.16:0/550643783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdccc100fa0 0x7fdccc198500 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fdcc000e9f0 tx=0x7fdcc000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:48.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.235+0000 7fdcca7fc640 1 -- 172.21.15.16:0/550643783 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc000cd80 con 0x7fdccc100fa0 2024-03-28T22:15:48.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.235+0000 7fdcd3a07640 1 -- 172.21.15.16:0/550643783 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdccc19d2a0 con 0x7fdccc100fa0 2024-03-28T22:15:48.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.235+0000 7fdcca7fc640 1 -- 172.21.15.16:0/550643783 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdcc0010910 con 0x7fdccc100fa0 2024-03-28T22:15:48.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.235+0000 7fdcca7fc640 1 -- 172.21.15.16:0/550643783 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc0018990 con 0x7fdccc100fa0 2024-03-28T22:15:48.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.235+0000 7fdcd3a07640 1 -- 172.21.15.16:0/550643783 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdccc1a4b30 con 0x7fdccc100fa0 2024-03-28T22:15:48.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.236+0000 7fdcd3a07640 1 -- 172.21.15.16:0/550643783 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdccc101d20 con 0x7fdccc100fa0 2024-03-28T22:15:48.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.237+0000 7fdcca7fc640 1 -- 172.21.15.16:0/550643783 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7fdcc0010ac0 con 0x7fdccc100fa0 2024-03-28T22:15:48.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.238+0000 7fdcca7fc640 1 --2- 172.21.15.16:0/550643783 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fdcb40a1ef0 0x7fdcb40a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.238+0000 7fdcd17a3640 1 --2- 172.21.15.16:0/550643783 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fdcb40a1ef0 0x7fdcb40a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.238+0000 7fdcca7fc640 1 -- 172.21.15.16:0/550643783 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fdcc00ce920 con 0x7fdccc100fa0 2024-03-28T22:15:48.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.238+0000 7fdcd17a3640 1 --2- 172.21.15.16:0/550643783 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fdcb40a1ef0 0x7fdcb40a43b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fdcc4000c00 tx=0x7fdcc4005fd0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:48.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.242+0000 7fdcca7fc640 1 -- 172.21.15.16:0/550643783 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fdcc009b380 con 0x7fdccc100fa0 2024-03-28T22:15:48.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.465+0000 7fdcd3a07640 1 -- 172.21.15.16:0/550643783 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdccc199310 con 0x7fdccc100fa0 2024-03-28T22:15:48.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.465+0000 7fdcca7fc640 1 -- 172.21.15.16:0/550643783 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11851 (secure 0 0 0) 0x7fdcc009f420 con 0x7fdccc100fa0 2024-03-28T22:15:48.468 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:15:48.469 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/1644581227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1644581227},{"type":"v1","addr":"172.21.15.16:6839","nonce":1644581227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/2593062669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2593062669},{"type":"v1","addr":"172.21.15.16:6841","nonce":2593062669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:08:29.672976+0000","modified":"2024-03-28T22:14:44.922044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24566,"mds_1":24524,"mds_2":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24466":{"gid":24466,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/247249857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":247249857},{"type":"v1","addr":"172.21.15.63:6835","nonce":247249857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6837/2324231032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":2324231032},{"type":"v1","addr":"172.21.15.63:6837","nonce":2324231032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6839/1177197077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":1177197077},{"type":"v1","addr":"172.21.15.63:6839","nonce":1177197077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"e","rank":2,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.53:6835/350727634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":350727634},{"type":"v1","addr":"172.21.15.53:6835","nonce":350727634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/349877573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":349877573},{"type":"v1","addr":"172.21.15.53:6837","nonce":349877573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6839/3714153956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":3714153956},{"type":"v1","addr":"172.21.15.53:6839","nonce":3714153956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24566,"qdb_cluster":[24500,24524,24566]},"id":1}]} 2024-03-28T22:15:48.469 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2024-03-28T22:15:48.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.468+0000 7fdcd3a07640 1 -- 172.21.15.16:0/550643783 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fdcb40a1ef0 msgr2=0x7fdcb40a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.468+0000 7fdcd3a07640 1 --2- 172.21.15.16:0/550643783 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fdcb40a1ef0 0x7fdcb40a43b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fdcc4000c00 tx=0x7fdcc4005fd0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.468+0000 7fdcd3a07640 1 -- 172.21.15.16:0/550643783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdccc100fa0 msgr2=0x7fdccc198500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.468+0000 7fdcd3a07640 1 --2- 172.21.15.16:0/550643783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdccc100fa0 0x7fdccc198500 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fdcc000e9f0 tx=0x7fdcc000eeb0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.468+0000 7fdcd3a07640 1 -- 172.21.15.16:0/550643783 shutdown_connections 2024-03-28T22:15:48.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.468+0000 7fdcd3a07640 1 --2- 172.21.15.16:0/550643783 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fdcb40a1ef0 0x7fdcb40a43b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.468+0000 7fdcd3a07640 1 --2- 172.21.15.16:0/550643783 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdccc106680 0x7fdccc19c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.468+0000 7fdcd3a07640 1 --2- 172.21.15.16:0/550643783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdccc100fa0 0x7fdccc198500 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.468+0000 7fdcd3a07640 1 --2- 172.21.15.16:0/550643783 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdccc1005e0 0x7fdccc197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.468+0000 7fdcd3a07640 1 -- 172.21.15.16:0/550643783 >> 172.21.15.16:0/550643783 conn(0x7fdccc0fc0d0 msgr2=0x7fdccc06a130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:48.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.468+0000 7fdcd3a07640 1 -- 172.21.15.16:0/550643783 shutdown_connections 2024-03-28T22:15:48.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.468+0000 7fdcd3a07640 1 -- 172.21.15.16:0/550643783 wait complete. 2024-03-28T22:15:48.482 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:15:48.507 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:48 smithi016 ceph-mon[32374]: pgmap v159: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:15:48.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:48 smithi053 ceph-mon[35979]: pgmap v159: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:15:48.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.584+0000 7f955090d640 1 -- 172.21.15.16:0/1617772754 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f954c106680 msgr2=0x7f954c106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.584+0000 7f955090d640 1 --2- 172.21.15.16:0/1617772754 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f954c106680 0x7f954c106a60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9544009900 tx=0x7f954402bc30 comp rx=0 tx=0).stop 2024-03-28T22:15:48.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.584+0000 7f955090d640 1 -- 172.21.15.16:0/1617772754 shutdown_connections 2024-03-28T22:15:48.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.584+0000 7f955090d640 1 --2- 172.21.15.16:0/1617772754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f954c100fa0 0x7f954c10d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.584+0000 7f955090d640 1 --2- 172.21.15.16:0/1617772754 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f954c1005e0 0x7f954c100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.584+0000 7f955090d640 1 --2- 172.21.15.16:0/1617772754 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f954c106680 0x7f954c106a60 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.584+0000 7f955090d640 1 -- 172.21.15.16:0/1617772754 >> 172.21.15.16:0/1617772754 conn(0x7f954c0fc0b0 msgr2=0x7f954c0fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:48.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.585+0000 7f955090d640 1 -- 172.21.15.16:0/1617772754 shutdown_connections 2024-03-28T22:15:48.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.585+0000 7f955090d640 1 -- 172.21.15.16:0/1617772754 wait complete. 2024-03-28T22:15:48.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.585+0000 7f955090d640 1 Processor -- start 2024-03-28T22:15:48.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.585+0000 7f955090d640 1 -- start start 2024-03-28T22:15:48.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.585+0000 7f955090d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f954c1005e0 0x7f954c197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.585+0000 7f955090d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f954c100fa0 0x7f954c198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.585+0000 7f955090d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f954c106680 0x7f954c19c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.585+0000 7f955090d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f954c111b00 con 0x7f954c100fa0 2024-03-28T22:15:48.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.585+0000 7f955090d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f954c111980 con 0x7f954c106680 2024-03-28T22:15:48.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.585+0000 7f955090d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f954c111c80 con 0x7f954c1005e0 2024-03-28T22:15:48.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.586+0000 7f9549d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f954c100fa0 0x7f954c198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.586+0000 7f9549d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f954c100fa0 0x7f954c198480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56936/0 (socket says 172.21.15.16:56936) 2024-03-28T22:15:48.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.586+0000 7f9549d9b640 1 -- 172.21.15.16:0/3249799002 learned_addr learned my addr 172.21.15.16:0/3249799002 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:48.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.586+0000 7f954ad9d640 1 --2- 172.21.15.16:0/3249799002 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f954c106680 0x7f954c19c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.586+0000 7f954a59c640 1 --2- 172.21.15.16:0/3249799002 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f954c1005e0 0x7f954c197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.586+0000 7f9549d9b640 1 -- 172.21.15.16:0/3249799002 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f954c1005e0 msgr2=0x7f954c197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.586+0000 7f9549d9b640 1 --2- 172.21.15.16:0/3249799002 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f954c1005e0 0x7f954c197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.586+0000 7f9549d9b640 1 -- 172.21.15.16:0/3249799002 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f954c106680 msgr2=0x7f954c19c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.586+0000 7f9549d9b640 1 --2- 172.21.15.16:0/3249799002 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f954c106680 0x7f954c19c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.586+0000 7f9549d9b640 1 -- 172.21.15.16:0/3249799002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f954c19cf30 con 0x7f954c100fa0 2024-03-28T22:15:48.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.586+0000 7f954ad9d640 1 --2- 172.21.15.16:0/3249799002 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f954c106680 0x7f954c19c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:15:48.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.586+0000 7f9549d9b640 1 --2- 172.21.15.16:0/3249799002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f954c100fa0 0x7f954c198480 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f953400d950 tx=0x7f953400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:48.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.586+0000 7f95437fe640 1 -- 172.21.15.16:0/3249799002 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9534017070 con 0x7f954c100fa0 2024-03-28T22:15:48.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.586+0000 7f955090d640 1 -- 172.21.15.16:0/3249799002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f954c19d220 con 0x7f954c100fa0 2024-03-28T22:15:48.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.586+0000 7f95437fe640 1 -- 172.21.15.16:0/3249799002 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9534009c30 con 0x7f954c100fa0 2024-03-28T22:15:48.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.586+0000 7f95437fe640 1 -- 172.21.15.16:0/3249799002 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f953400f6b0 con 0x7f954c100fa0 2024-03-28T22:15:48.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.586+0000 7f955090d640 1 -- 172.21.15.16:0/3249799002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f954c1a4ab0 con 0x7f954c100fa0 2024-03-28T22:15:48.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.588+0000 7f95437fe640 1 -- 172.21.15.16:0/3249799002 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f95340109d0 con 0x7f954c100fa0 2024-03-28T22:15:48.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.589+0000 7f95535c9740 1 Processor -- start 2024-03-28T22:15:48.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.589+0000 7f95437fe640 1 --2- 172.21.15.16:0/3249799002 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f95380a1ef0 0x7f95380a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.589+0000 7f95437fe640 1 -- 172.21.15.16:0/3249799002 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f95340cdd10 con 0x7f954c100fa0 2024-03-28T22:15:48.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.589+0000 7f954ad9d640 1 --2- 172.21.15.16:0/3249799002 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f95380a1ef0 0x7f95380a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.589+0000 7f95535c9740 1 -- start start 2024-03-28T22:15:48.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.589+0000 7f95535c9740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564889ee2470 0x564889ee2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.589+0000 7f95535c9740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564889ee2d90 0x564889eeb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.589+0000 7f95535c9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564889eeb970 0x564889eedd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.589+0000 7f95535c9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564889e461b0 con 0x564889eeb970 2024-03-28T22:15:48.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.589+0000 7f95535c9740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x564889deec00 con 0x564889ee2470 2024-03-28T22:15:48.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.589+0000 7f95535c9740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x564889e42250 con 0x564889ee2d90 2024-03-28T22:15:48.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.589+0000 7f954ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564889eeb970 0x564889eedd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.589+0000 7f954a59c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564889ee2470 0x564889ee2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.590+0000 7f954a59c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564889ee2470 0x564889ee2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:48836/0 (socket says 172.21.15.16:48836) 2024-03-28T22:15:48.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.590+0000 7f954a59c640 1 -- 172.21.15.16:0/698056504 learned_addr learned my addr 172.21.15.16:0/698056504 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:48.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.590+0000 7f954ad9d640 1 --2- 172.21.15.16:0/3249799002 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f95380a1ef0 0x7f95380a43b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f953c005fd0 tx=0x7f953c009500 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:48.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.590+0000 7f9549d9b640 1 --2- 172.21.15.16:0/698056504 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564889ee2d90 0x564889eeb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.590+0000 7f954a59c640 1 -- 172.21.15.16:0/698056504 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564889ee2d90 msgr2=0x564889eeb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:48.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.590+0000 7f954a59c640 1 --2- 172.21.15.16:0/698056504 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564889ee2d90 0x564889eeb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.590+0000 7f954a59c640 1 -- 172.21.15.16:0/698056504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564889eeb970 msgr2=0x564889eedd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:48.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.590+0000 7f954a59c640 1 --2- 172.21.15.16:0/698056504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564889eeb970 0x564889eedd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.590+0000 7f954a59c640 1 -- 172.21.15.16:0/698056504 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564889e0e3d0 con 0x564889ee2470 2024-03-28T22:15:48.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.590+0000 7f954ad9d640 1 --2- 172.21.15.16:0/698056504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564889eeb970 0x564889eedd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:15:48.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.590+0000 7f954a59c640 1 --2- 172.21.15.16:0/698056504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564889ee2470 0x564889ee2850 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9544009900 tx=0x7f9544009610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9afa7472f651c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:48.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.590+0000 7f9540ff9640 1 -- 172.21.15.16:0/698056504 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9544042070 con 0x564889ee2470 2024-03-28T22:15:48.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.590+0000 7f9540ff9640 1 -- 172.21.15.16:0/698056504 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f954402c6d0 con 0x564889ee2470 2024-03-28T22:15:48.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.590+0000 7f9540ff9640 1 -- 172.21.15.16:0/698056504 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9544004f20 con 0x564889ee2470 2024-03-28T22:15:48.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.591+0000 7f95535c9740 1 -- 172.21.15.16:0/698056504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564889ee2470 msgr2=0x564889ee2850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.591+0000 7f95535c9740 1 --2- 172.21.15.16:0/698056504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564889ee2470 0x564889ee2850 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9544009900 tx=0x7f9544009610 comp rx=0 tx=0).stop 2024-03-28T22:15:48.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.591+0000 7f95535c9740 1 -- 172.21.15.16:0/698056504 shutdown_connections 2024-03-28T22:15:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.591+0000 7f95535c9740 1 --2- 172.21.15.16:0/698056504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564889eeb970 0x564889eedd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.591+0000 7f95535c9740 1 --2- 172.21.15.16:0/698056504 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564889ee2d90 0x564889eeb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.591+0000 7f95535c9740 1 --2- 172.21.15.16:0/698056504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564889ee2470 0x564889ee2850 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.591+0000 7f95535c9740 1 -- 172.21.15.16:0/698056504 >> 172.21.15.16:0/698056504 conn(0x564889edcba0 msgr2=0x564889edcf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.591+0000 7f95535c9740 1 -- 172.21.15.16:0/698056504 shutdown_connections 2024-03-28T22:15:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.591+0000 7f95535c9740 1 -- 172.21.15.16:0/698056504 wait complete. 2024-03-28T22:15:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.591+0000 7f95535c9740 1 -- start start 2024-03-28T22:15:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.591+0000 7f95535c9740 1 Processor -- start 2024-03-28T22:15:48.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.592+0000 7f95535c9740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564889ee2470 0x564889ee89d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.592+0000 7f95535c9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564889ee2d90 0x564889ee8f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.592+0000 7f95535c9740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564889eeb970 0x564889ee9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.592+0000 7f95535c9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564889eee700 con 0x564889ee2d90 2024-03-28T22:15:48.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.592+0000 7f95535c9740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x564889de4470 con 0x564889eeb970 2024-03-28T22:15:48.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.592+0000 7f95535c9740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x564889eee880 con 0x564889ee2470 2024-03-28T22:15:48.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.592+0000 7f95535c9740 10 client.0 _ll_register_callbacks cb 0x564889e01cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:15:48.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.592+0000 7f9549d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564889ee2d90 0x564889ee8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.592+0000 7f9549d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564889ee2d90 0x564889ee8f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56960/0 (socket says 172.21.15.16:56960) 2024-03-28T22:15:48.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.592+0000 7f9549d9b640 1 -- 172.21.15.16:0/2422667316 learned_addr learned my addr 172.21.15.16:0/2422667316 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:48.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.592+0000 7f954ad9d640 1 --2- 172.21.15.16:0/2422667316 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564889eeb970 0x564889ee9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.592+0000 7f954a59c640 1 --2- 172.21.15.16:0/2422667316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564889ee2470 0x564889ee89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.592+0000 7f9549d9b640 1 -- 172.21.15.16:0/2422667316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564889ee2470 msgr2=0x564889ee89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:48.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.592+0000 7f9549d9b640 1 --2- 172.21.15.16:0/2422667316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564889ee2470 0x564889ee89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.592+0000 7f9549d9b640 1 -- 172.21.15.16:0/2422667316 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564889eeb970 msgr2=0x564889ee9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:48.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.592+0000 7f9549d9b640 1 --2- 172.21.15.16:0/2422667316 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564889eeb970 0x564889ee9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.592+0000 7f9549d9b640 1 -- 172.21.15.16:0/2422667316 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564889b5a3f0 con 0x564889ee2d90 2024-03-28T22:15:48.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.592+0000 7f9549d9b640 1 --2- 172.21.15.16:0/2422667316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564889ee2d90 0x564889ee8f10 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f95340cba80 tx=0x7f9534009ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0ae4b0122c6e9e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:48.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.592+0000 7f9531ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:15:48.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.592+0000 7f9531ffb640 1 -- 172.21.15.16:0/2422667316 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9534020040 con 0x564889ee2d90 2024-03-28T22:15:48.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.592+0000 7f9531ffb640 1 -- 172.21.15.16:0/2422667316 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f9534096e40 con 0x564889ee2d90 2024-03-28T22:15:48.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.592+0000 7f95535c9740 1 -- 172.21.15.16:0/2422667316 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f954402c6d0 con 0x564889ee2d90 2024-03-28T22:15:48.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.592+0000 7f9531ffb640 1 -- 172.21.15.16:0/2422667316 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f95340a2ec0 con 0x564889ee2d90 2024-03-28T22:15:48.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.592+0000 7f95535c9740 10 client.16584 fetch_fsmap learned FSMap version 27 2024-03-28T22:15:48.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.592+0000 7f95535c9740 1 -- 172.21.15.16:0/2422667316 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564889c247d0 con 0x564889ee2d90 2024-03-28T22:15:48.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.593+0000 7f9531ffb640 1 -- 172.21.15.16:0/2422667316 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f954402c6d0 con 0x564889ee2d90 2024-03-28T22:15:48.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.593+0000 7f9531ffb640 1 -- 172.21.15.16:0/2422667316 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f95340ba0c0 con 0x564889ee2d90 2024-03-28T22:15:48.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.593+0000 7f95535c9740 10 client.16584 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:15:48.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.593+0000 7f95535c9740 10 client.16584 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-28T22:15:48.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.593+0000 7f95535c9740 20 client.16584 populate_metadata read hostname 'smithi016' 2024-03-28T22:15:48.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.593+0000 7f95535c9740 1 --2- 172.21.15.16:0/2422667316 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x564889efb970 0x564889efdd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.593+0000 7f95535c9740 4 client.16584 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:15:48.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.593+0000 7f95535c9740 1 -- 172.21.15.16:0/2422667316 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564889e536f0 con 0x564889efb970 2024-03-28T22:15:48.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.593+0000 7f954a59c640 1 --2- 172.21.15.16:0/2422667316 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x564889efb970 0x564889efdd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.593+0000 7f9531ffb640 10 client.16584 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:15:48.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.593+0000 7f954a59c640 1 --2- 172.21.15.16:0/2422667316 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x564889efb970 0x564889efdd60 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=58436504802de1ef server_cookie=7d3fac24c9807a6a in_seq=0 out_seq=0 2024-03-28T22:15:48.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.594+0000 7f9531ffb640 1 -- 172.21.15.16:0/2422667316 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564889e536f0 con 0x564889efb970 2024-03-28T22:15:48.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.594+0000 7f9531ffb640 10 client.16584 handle_command_reply: tid=0 2024-03-28T22:15:48.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.594+0000 7f95535c9740 1 client.16584 shutdown 2024-03-28T22:15:48.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.594+0000 7f95535c9740 1 -- 172.21.15.16:0/2422667316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564889ee2d90 msgr2=0x564889ee8f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.594+0000 7f95535c9740 1 --2- 172.21.15.16:0/2422667316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564889ee2d90 0x564889ee8f10 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f95340cba80 tx=0x7f9534009ea0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.594+0000 7f95535c9740 1 -- 172.21.15.16:0/2422667316 shutdown_connections 2024-03-28T22:15:48.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.594+0000 7f95535c9740 1 --2- 172.21.15.16:0/2422667316 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x564889efb970 0x564889efdd60 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.594+0000 7f95535c9740 1 --2- 172.21.15.16:0/2422667316 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564889eeb970 0x564889ee9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.594+0000 7f95535c9740 1 --2- 172.21.15.16:0/2422667316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564889ee2d90 0x564889ee8f10 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.594+0000 7f9531ffb640 0 client.16584 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:15:48.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.594+0000 7f95535c9740 1 --2- 172.21.15.16:0/2422667316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564889ee2470 0x564889ee89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.594+0000 7f95535c9740 1 -- 172.21.15.16:0/2422667316 >> 172.21.15.16:0/2422667316 conn(0x564889edcba0 msgr2=0x564889ede8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:48.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.594+0000 7f95535c9740 1 -- 172.21.15.16:0/2422667316 shutdown_connections 2024-03-28T22:15:48.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.594+0000 7f95535c9740 1 -- 172.21.15.16:0/2422667316 wait complete. 2024-03-28T22:15:48.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.594+0000 7f95535c9740 20 client.16584 trim_cache size 0 max 16384 2024-03-28T22:15:48.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.607+0000 7f95535c9740 1 Processor -- start 2024-03-28T22:15:48.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.607+0000 7f95535c9740 1 -- start start 2024-03-28T22:15:48.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.608+0000 7f95535c9740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564889ee2470 0x564889ef74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.608+0000 7f95535c9740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564889ee2d90 0x564889eea800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.608+0000 7f95535c9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564889eeb970 0x564889f03610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.608+0000 7f95535c9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f954c1afac0 con 0x564889eeb970 2024-03-28T22:15:48.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.608+0000 7f95535c9740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f954c1ad390 con 0x564889ee2470 2024-03-28T22:15:48.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.608+0000 7f95535c9740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f954c1ab1e0 con 0x564889ee2d90 2024-03-28T22:15:48.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.608+0000 7f954ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564889eeb970 0x564889f03610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.608+0000 7f954ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564889eeb970 0x564889f03610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56966/0 (socket says 172.21.15.16:56966) 2024-03-28T22:15:48.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.608+0000 7f954ad9d640 1 -- 172.21.15.16:0/3410889355 learned_addr learned my addr 172.21.15.16:0/3410889355 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:48.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.608+0000 7f954a59c640 1 --2- 172.21.15.16:0/3410889355 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564889ee2470 0x564889ef74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.608+0000 7f954ad9d640 1 -- 172.21.15.16:0/3410889355 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564889ee2d90 msgr2=0x564889eea800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:48.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.608+0000 7f954ad9d640 1 --2- 172.21.15.16:0/3410889355 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564889ee2d90 0x564889eea800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.608+0000 7f954ad9d640 1 -- 172.21.15.16:0/3410889355 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564889ee2470 msgr2=0x564889ef74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:48.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.608+0000 7f954ad9d640 1 --2- 172.21.15.16:0/3410889355 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564889ee2470 0x564889ef74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.608+0000 7f954ad9d640 1 -- 172.21.15.16:0/3410889355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564889ef79e0 con 0x564889eeb970 2024-03-28T22:15:48.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.608+0000 7f954a59c640 1 --2- 172.21.15.16:0/3410889355 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564889ee2470 0x564889ef74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:15:48.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.608+0000 7f954ad9d640 1 --2- 172.21.15.16:0/3410889355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564889eeb970 0x564889f03610 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f953c005c30 tx=0x7f953c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6875a9f267fbeafe server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:48.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.608+0000 7f9533fff640 1 -- 172.21.15.16:0/3410889355 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f953c004530 con 0x564889eeb970 2024-03-28T22:15:48.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.608+0000 7f9533fff640 1 -- 172.21.15.16:0/3410889355 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f953c0046d0 con 0x564889eeb970 2024-03-28T22:15:48.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.608+0000 7f9533fff640 1 -- 172.21.15.16:0/3410889355 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f953c0286f0 con 0x564889eeb970 2024-03-28T22:15:48.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.609+0000 7f95535c9740 1 -- 172.21.15.16:0/3410889355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564889eeb970 msgr2=0x564889f03610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.609+0000 7f95535c9740 1 --2- 172.21.15.16:0/3410889355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564889eeb970 0x564889f03610 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f953c005c30 tx=0x7f953c014000 comp rx=0 tx=0).stop 2024-03-28T22:15:48.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.609+0000 7f95535c9740 1 -- 172.21.15.16:0/3410889355 shutdown_connections 2024-03-28T22:15:48.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.609+0000 7f95535c9740 1 --2- 172.21.15.16:0/3410889355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564889eeb970 0x564889f03610 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.609+0000 7f95535c9740 1 --2- 172.21.15.16:0/3410889355 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564889ee2d90 0x564889eea800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.609+0000 7f95535c9740 1 --2- 172.21.15.16:0/3410889355 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564889ee2470 0x564889ef74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.609+0000 7f95535c9740 1 -- 172.21.15.16:0/3410889355 >> 172.21.15.16:0/3410889355 conn(0x564889edcba0 msgr2=0x564889eea0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:48.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.609+0000 7f95535c9740 1 -- 172.21.15.16:0/3410889355 shutdown_connections 2024-03-28T22:15:48.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.609+0000 7f95535c9740 1 -- 172.21.15.16:0/3410889355 wait complete. 2024-03-28T22:15:48.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.609+0000 7f95535c9740 1 -- start start 2024-03-28T22:15:48.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.609+0000 7f95535c9740 1 Processor -- start 2024-03-28T22:15:48.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.610+0000 7f95535c9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564889ee2470 0x564889ef3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.610+0000 7f95535c9740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564889ee2d90 0x564889ef0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.610+0000 7f95535c9740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564889eeb970 0x564889ef0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.610+0000 7f95535c9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f954c1ab850 con 0x564889ee2470 2024-03-28T22:15:48.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.610+0000 7f95535c9740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f954c1accc0 con 0x564889eeb970 2024-03-28T22:15:48.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.610+0000 7f95535c9740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f954c1abf20 con 0x564889ee2d90 2024-03-28T22:15:48.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.610+0000 7f95535c9740 10 client.0 _ll_register_callbacks cb 0x7f953409a130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:15:48.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.610+0000 7f954a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564889ee2470 0x564889ef3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.610+0000 7f954a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564889ee2470 0x564889ef3cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56974/0 (socket says 172.21.15.16:56974) 2024-03-28T22:15:48.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.610+0000 7f954a59c640 1 -- 172.21.15.16:0/3698620197 learned_addr learned my addr 172.21.15.16:0/3698620197 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:48.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.610+0000 7f954ad9d640 1 --2- 172.21.15.16:0/3698620197 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564889eeb970 0x564889ef0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.610+0000 7f9549d9b640 1 --2- 172.21.15.16:0/3698620197 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564889ee2d90 0x564889ef0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.610+0000 7f954a59c640 1 -- 172.21.15.16:0/3698620197 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564889ee2d90 msgr2=0x564889ef0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:48.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.610+0000 7f954a59c640 1 --2- 172.21.15.16:0/3698620197 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564889ee2d90 0x564889ef0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.610+0000 7f954a59c640 1 -- 172.21.15.16:0/3698620197 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564889eeb970 msgr2=0x564889ef0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:48.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.610+0000 7f954a59c640 1 --2- 172.21.15.16:0/3698620197 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564889eeb970 0x564889ef0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.610+0000 7f954a59c640 1 -- 172.21.15.16:0/3698620197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564889ef1620 con 0x564889ee2470 2024-03-28T22:15:48.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.610+0000 7f9549d9b640 1 --2- 172.21.15.16:0/3698620197 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564889ee2d90 0x564889ef0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:15:48.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.610+0000 7f954a59c640 1 --2- 172.21.15.16:0/3698620197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564889ee2470 0x564889ef3cc0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f954402f020 tx=0x7f9544009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afaa31f3dec50de9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:48.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.610+0000 7f9531ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:15:48.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.610+0000 7f9531ffb640 1 -- 172.21.15.16:0/3698620197 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9544042040 con 0x564889ee2470 2024-03-28T22:15:48.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.610+0000 7f9531ffb640 1 -- 172.21.15.16:0/3698620197 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f9544050a30 con 0x564889ee2470 2024-03-28T22:15:48.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.610+0000 7f95535c9740 1 -- 172.21.15.16:0/3698620197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f953c0046d0 con 0x564889ee2470 2024-03-28T22:15:48.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.611+0000 7f9531ffb640 1 -- 172.21.15.16:0/3698620197 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9544050c50 con 0x564889ee2470 2024-03-28T22:15:48.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.611+0000 7f95535c9740 10 client.16590 fetch_fsmap learned FSMap version 27 2024-03-28T22:15:48.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.611+0000 7f95535c9740 1 -- 172.21.15.16:0/3698620197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564889f62b20 con 0x564889ee2470 2024-03-28T22:15:48.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.611+0000 7f9531ffb640 1 -- 172.21.15.16:0/3698620197 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f953c0046d0 con 0x564889ee2470 2024-03-28T22:15:48.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.611+0000 7f9531ffb640 1 -- 172.21.15.16:0/3698620197 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f9544004040 con 0x564889ee2470 2024-03-28T22:15:48.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.611+0000 7f95535c9740 10 client.16590 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:15:48.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.611+0000 7f95535c9740 10 client.16590 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-28T22:15:48.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.611+0000 7f95535c9740 20 client.16590 populate_metadata read hostname 'smithi016' 2024-03-28T22:15:48.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.611+0000 7f95535c9740 1 --2- 172.21.15.16:0/3698620197 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x564889f62e80 0x564889f652a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.611+0000 7f95535c9740 4 client.16590 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:15:48.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.611+0000 7f95535c9740 1 -- 172.21.15.16:0/3698620197 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564889e4da40 con 0x564889f62e80 2024-03-28T22:15:48.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.611+0000 7f9549d9b640 1 --2- 172.21.15.16:0/3698620197 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x564889f62e80 0x564889f652a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.611+0000 7f9549d9b640 1 --2- 172.21.15.16:0/3698620197 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x564889f62e80 0x564889f652a0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a95f5586be55474d server_cookie=318372c293b969cf in_seq=0 out_seq=0 2024-03-28T22:15:48.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.611+0000 7f9531ffb640 10 client.16590 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:15:48.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.612+0000 7f9531ffb640 1 -- 172.21.15.16:0/3698620197 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564889e4da40 con 0x564889f62e80 2024-03-28T22:15:48.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.612+0000 7f9531ffb640 10 client.16590 handle_command_reply: tid=0 2024-03-28T22:15:48.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.612+0000 7f95535c9740 1 client.16590 shutdown 2024-03-28T22:15:48.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.612+0000 7f95535c9740 1 -- 172.21.15.16:0/3698620197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564889ee2470 msgr2=0x564889ef3cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.612+0000 7f95535c9740 1 --2- 172.21.15.16:0/3698620197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564889ee2470 0x564889ef3cc0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f954402f020 tx=0x7f9544009680 comp rx=0 tx=0).stop 2024-03-28T22:15:48.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.612+0000 7f95535c9740 1 -- 172.21.15.16:0/3698620197 shutdown_connections 2024-03-28T22:15:48.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.612+0000 7f95535c9740 1 --2- 172.21.15.16:0/3698620197 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x564889f62e80 0x564889f652a0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.612+0000 7f95535c9740 1 --2- 172.21.15.16:0/3698620197 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564889eeb970 0x564889ef0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.612+0000 7f95535c9740 1 --2- 172.21.15.16:0/3698620197 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564889ee2d90 0x564889ef0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.612+0000 7f9531ffb640 0 client.16590 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:15:48.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.612+0000 7f95535c9740 1 --2- 172.21.15.16:0/3698620197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564889ee2470 0x564889ef3cc0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.612+0000 7f95535c9740 1 -- 172.21.15.16:0/3698620197 >> 172.21.15.16:0/3698620197 conn(0x564889edcba0 msgr2=0x564889edbd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:48.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.612+0000 7f95535c9740 1 -- 172.21.15.16:0/3698620197 shutdown_connections 2024-03-28T22:15:48.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.612+0000 7f95535c9740 1 -- 172.21.15.16:0/3698620197 wait complete. 2024-03-28T22:15:48.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.612+0000 7f95535c9740 20 client.16590 trim_cache size 0 max 16384 2024-03-28T22:15:48.635 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:15:48.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.612+0000 7f955090d640 1 -- 172.21.15.16:0/3249799002 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f95380a1ef0 msgr2=0x7f95380a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.612+0000 7f955090d640 1 --2- 172.21.15.16:0/3249799002 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f95380a1ef0 0x7f95380a43b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f953c005fd0 tx=0x7f953c009500 comp rx=0 tx=0).stop 2024-03-28T22:15:48.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.612+0000 7f955090d640 1 -- 172.21.15.16:0/3249799002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f954c100fa0 msgr2=0x7f954c198480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.612+0000 7f955090d640 1 --2- 172.21.15.16:0/3249799002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f954c100fa0 0x7f954c198480 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f953400d950 tx=0x7f953400de10 comp rx=0 tx=0).stop 2024-03-28T22:15:48.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.612+0000 7f955090d640 1 -- 172.21.15.16:0/3249799002 shutdown_connections 2024-03-28T22:15:48.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.612+0000 7f955090d640 1 --2- 172.21.15.16:0/3249799002 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f95380a1ef0 0x7f95380a43b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.612+0000 7f955090d640 1 --2- 172.21.15.16:0/3249799002 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f954c106680 0x7f954c19c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.612+0000 7f955090d640 1 --2- 172.21.15.16:0/3249799002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f954c100fa0 0x7f954c198480 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.612+0000 7f955090d640 1 --2- 172.21.15.16:0/3249799002 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f954c1005e0 0x7f954c197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.612+0000 7f955090d640 1 -- 172.21.15.16:0/3249799002 >> 172.21.15.16:0/3249799002 conn(0x7f954c0fc0b0 msgr2=0x7f954c0fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:48.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.612+0000 7f955090d640 1 -- 172.21.15.16:0/3249799002 shutdown_connections 2024-03-28T22:15:48.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.612+0000 7f955090d640 1 -- 172.21.15.16:0/3249799002 wait complete. 2024-03-28T22:15:48.639 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:15:48.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:48 smithi063 ceph-mon[36249]: pgmap v159: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:15:48.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.779+0000 7f7460a64640 1 -- 172.21.15.16:0/892173907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f745c100fa0 msgr2=0x7f745c10d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.779+0000 7f7460a64640 1 --2- 172.21.15.16:0/892173907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f745c100fa0 0x7f745c10d4b0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f744c009cd0 tx=0x7f744c02c0f0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.779+0000 7f7460a64640 1 -- 172.21.15.16:0/892173907 shutdown_connections 2024-03-28T22:15:48.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.779+0000 7f7460a64640 1 --2- 172.21.15.16:0/892173907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f745c100fa0 0x7f745c10d4b0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.779+0000 7f7460a64640 1 --2- 172.21.15.16:0/892173907 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f745c1005e0 0x7f745c100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.779+0000 7f7460a64640 1 --2- 172.21.15.16:0/892173907 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f745c106680 0x7f745c106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.779+0000 7f7460a64640 1 -- 172.21.15.16:0/892173907 >> 172.21.15.16:0/892173907 conn(0x7f745c077770 msgr2=0x7f745c077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:48.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.779+0000 7f7460a64640 1 -- 172.21.15.16:0/892173907 shutdown_connections 2024-03-28T22:15:48.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.779+0000 7f7460a64640 1 -- 172.21.15.16:0/892173907 wait complete. 2024-03-28T22:15:48.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.780+0000 7f7460a64640 1 Processor -- start 2024-03-28T22:15:48.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.780+0000 7f7460a64640 1 -- start start 2024-03-28T22:15:48.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.780+0000 7f7460a64640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f745c1005e0 0x7f745c197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.780+0000 7f7460a64640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f745c100fa0 0x7f745c198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.780+0000 7f7460a64640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f745c106680 0x7f745c19c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.780+0000 7f7460a64640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f745c110a00 con 0x7f745c1005e0 2024-03-28T22:15:48.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.780+0000 7f7460a64640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f745c110880 con 0x7f745c100fa0 2024-03-28T22:15:48.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.780+0000 7f7460a64640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f745c110b80 con 0x7f745c106680 2024-03-28T22:15:48.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.780+0000 7f745a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f745c1005e0 0x7f745c197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.781+0000 7f745a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f745c1005e0 0x7f745c197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56994/0 (socket says 172.21.15.16:56994) 2024-03-28T22:15:48.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.781+0000 7f745a59c640 1 -- 172.21.15.16:0/3500113639 learned_addr learned my addr 172.21.15.16:0/3500113639 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:48.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.781+0000 7f7459d9b640 1 --2- 172.21.15.16:0/3500113639 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f745c100fa0 0x7f745c198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.781+0000 7f745ad9d640 1 --2- 172.21.15.16:0/3500113639 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f745c106680 0x7f745c19c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.781+0000 7f745a59c640 1 -- 172.21.15.16:0/3500113639 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f745c106680 msgr2=0x7f745c19c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.781+0000 7f745a59c640 1 --2- 172.21.15.16:0/3500113639 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f745c106680 0x7f745c19c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.781+0000 7f745a59c640 1 -- 172.21.15.16:0/3500113639 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f745c100fa0 msgr2=0x7f745c198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.781+0000 7f745a59c640 1 --2- 172.21.15.16:0/3500113639 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f745c100fa0 0x7f745c198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.781+0000 7f745a59c640 1 -- 172.21.15.16:0/3500113639 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f745c19cef0 con 0x7f745c1005e0 2024-03-28T22:15:48.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.781+0000 7f7459d9b640 1 --2- 172.21.15.16:0/3500113639 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f745c100fa0 0x7f745c198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:15:48.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.781+0000 7f745a59c640 1 --2- 172.21.15.16:0/3500113639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f745c1005e0 0x7f745c197f00 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f745400d950 tx=0x7f745400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:48.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.781+0000 7f744b7fe640 1 -- 172.21.15.16:0/3500113639 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7454017070 con 0x7f745c1005e0 2024-03-28T22:15:48.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.781+0000 7f7460a64640 1 -- 172.21.15.16:0/3500113639 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f745c19d1e0 con 0x7f745c1005e0 2024-03-28T22:15:48.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.781+0000 7f744b7fe640 1 -- 172.21.15.16:0/3500113639 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7454009c30 con 0x7f745c1005e0 2024-03-28T22:15:48.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.781+0000 7f744b7fe640 1 -- 172.21.15.16:0/3500113639 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f745400f600 con 0x7f745c1005e0 2024-03-28T22:15:48.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.781+0000 7f7460a64640 1 -- 172.21.15.16:0/3500113639 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f745c1a4a70 con 0x7f745c1005e0 2024-03-28T22:15:48.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.783+0000 7f744b7fe640 1 -- 172.21.15.16:0/3500113639 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f745401a070 con 0x7f745c1005e0 2024-03-28T22:15:48.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.784+0000 7f7463643740 1 Processor -- start 2024-03-28T22:15:48.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.784+0000 7f7463643740 1 -- start start 2024-03-28T22:15:48.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.784+0000 7f7463643740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e9df1f1470 0x55e9df1f1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.784+0000 7f7463643740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9df1f1d90 0x55e9df1fa430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.784+0000 7f7463643740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9df1fa970 0x55e9df1fcd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.784+0000 7f7463643740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e9df1551b0 con 0x55e9df1f1d90 2024-03-28T22:15:48.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.784+0000 7f7463643740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55e9df0fdc00 con 0x55e9df1f1470 2024-03-28T22:15:48.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.784+0000 7f7463643740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e9df151250 con 0x55e9df1fa970 2024-03-28T22:15:48.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.784+0000 7f744b7fe640 1 --2- 172.21.15.16:0/3500113639 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f74440a1ef0 0x7f74440a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.784+0000 7f745a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9df1f1d90 0x55e9df1fa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.784+0000 7f745a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9df1f1d90 0x55e9df1fa430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57010/0 (socket says 172.21.15.16:57010) 2024-03-28T22:15:48.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.784+0000 7f745a59c640 1 -- 172.21.15.16:0/2658111767 learned_addr learned my addr 172.21.15.16:0/2658111767 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:48.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.784+0000 7f7459d9b640 1 --2- 172.21.15.16:0/2658111767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9df1fa970 0x55e9df1fcd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.784+0000 7f745ad9d640 1 --2- 172.21.15.16:0/2658111767 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e9df1f1470 0x55e9df1f1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.784+0000 7f745ad9d640 1 --2- 172.21.15.16:0/3500113639 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f74440a1ef0 0x7f74440a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.784+0000 7f744b7fe640 1 -- 172.21.15.16:0/3500113639 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f74540cdc10 con 0x7f745c1005e0 2024-03-28T22:15:48.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.784+0000 7f7459d9b640 1 -- 172.21.15.16:0/2658111767 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e9df1f1470 msgr2=0x55e9df1f1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:48.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.784+0000 7f7459d9b640 1 --2- 172.21.15.16:0/2658111767 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e9df1f1470 0x55e9df1f1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.784+0000 7f7459d9b640 1 -- 172.21.15.16:0/2658111767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9df1f1d90 msgr2=0x55e9df1fa430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:48.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.784+0000 7f7459d9b640 1 --2- 172.21.15.16:0/2658111767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9df1f1d90 0x55e9df1fa430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.784+0000 7f7459d9b640 1 -- 172.21.15.16:0/2658111767 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9df11d3d0 con 0x55e9df1fa970 2024-03-28T22:15:48.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.785+0000 7f745ad9d640 1 --2- 172.21.15.16:0/3500113639 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f74440a1ef0 0x7f74440a43b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f744c006010 tx=0x7f744c0095c0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:48.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.785+0000 7f7459d9b640 1 --2- 172.21.15.16:0/2658111767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9df1fa970 0x55e9df1fcd60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7450009fa0 tx=0x7f74500118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da0f791df376ccd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:48.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.785+0000 7f7448ff9640 1 -- 172.21.15.16:0/2658111767 <== mon.1 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7450004020 con 0x55e9df1fa970 2024-03-28T22:15:48.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.785+0000 7f7448ff9640 1 -- 172.21.15.16:0/2658111767 <== mon.1 v2:172.21.15.63:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74500041c0 con 0x55e9df1fa970 2024-03-28T22:15:48.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.785+0000 7f7448ff9640 1 -- 172.21.15.16:0/2658111767 <== mon.1 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f745000f8f0 con 0x55e9df1fa970 2024-03-28T22:15:48.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.786+0000 7f7463643740 1 -- 172.21.15.16:0/2658111767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9df1fa970 msgr2=0x55e9df1fcd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.786+0000 7f7463643740 1 --2- 172.21.15.16:0/2658111767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9df1fa970 0x55e9df1fcd60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7450009fa0 tx=0x7f74500118d0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.786+0000 7f7463643740 1 -- 172.21.15.16:0/2658111767 shutdown_connections 2024-03-28T22:15:48.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.786+0000 7f7463643740 1 --2- 172.21.15.16:0/2658111767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9df1fa970 0x55e9df1fcd60 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.786+0000 7f7463643740 1 --2- 172.21.15.16:0/2658111767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9df1f1d90 0x55e9df1fa430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.786+0000 7f7463643740 1 --2- 172.21.15.16:0/2658111767 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e9df1f1470 0x55e9df1f1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.786+0000 7f7463643740 1 -- 172.21.15.16:0/2658111767 >> 172.21.15.16:0/2658111767 conn(0x55e9df1ebba0 msgr2=0x55e9df1ebf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:48.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.786+0000 7f7463643740 1 -- 172.21.15.16:0/2658111767 shutdown_connections 2024-03-28T22:15:48.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.786+0000 7f7463643740 1 -- 172.21.15.16:0/2658111767 wait complete. 2024-03-28T22:15:48.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.786+0000 7f7463643740 1 -- start start 2024-03-28T22:15:48.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.786+0000 7f7463643740 1 Processor -- start 2024-03-28T22:15:48.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.786+0000 7f7463643740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9df1f1470 0x55e9df1f79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.786+0000 7f7463643740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e9df1f1d90 0x55e9df1f7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.786+0000 7f7463643740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9df1fa970 0x55e9df1f8430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.786+0000 7f7463643740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e9df1fd760 con 0x55e9df1fa970 2024-03-28T22:15:48.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.786+0000 7f7463643740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55e9df0f3470 con 0x55e9df1f1d90 2024-03-28T22:15:48.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.786+0000 7f7463643740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e9df1fd8e0 con 0x55e9df1f1470 2024-03-28T22:15:48.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.786+0000 7f7463643740 10 client.0 _ll_register_callbacks cb 0x55e9df110cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:15:48.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.787+0000 7f745ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9df1fa970 0x55e9df1f8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.787+0000 7f745ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9df1fa970 0x55e9df1f8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57020/0 (socket says 172.21.15.16:57020) 2024-03-28T22:15:48.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.787+0000 7f7459d9b640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e9df1f1d90 0x55e9df1f7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.787+0000 7f745ad9d640 1 -- 172.21.15.16:0/813098107 learned_addr learned my addr 172.21.15.16:0/813098107 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:48.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.787+0000 7f745a59c640 1 --2- 172.21.15.16:0/813098107 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9df1f1470 0x55e9df1f79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.787+0000 7f7459d9b640 1 -- 172.21.15.16:0/813098107 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9df1f1470 msgr2=0x55e9df1f79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:48.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.787+0000 7f7459d9b640 1 --2- 172.21.15.16:0/813098107 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9df1f1470 0x55e9df1f79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.787+0000 7f7459d9b640 1 -- 172.21.15.16:0/813098107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9df1fa970 msgr2=0x55e9df1f8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:48.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.787+0000 7f7459d9b640 1 --2- 172.21.15.16:0/813098107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9df1fa970 0x55e9df1f8430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.787+0000 7f7459d9b640 1 -- 172.21.15.16:0/813098107 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9dee693f0 con 0x55e9df1f1d90 2024-03-28T22:15:48.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.787+0000 7f745ad9d640 1 --2- 172.21.15.16:0/813098107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9df1fa970 0x55e9df1f8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:15:48.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.787+0000 7f745a59c640 1 --2- 172.21.15.16:0/813098107 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9df1f1470 0x55e9df1f79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:15:48.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.787+0000 7f7459d9b640 1 --2- 172.21.15.16:0/813098107 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e9df1f1d90 0x55e9df1f7ef0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x55e9df1ecd30 tx=0x7f7450003bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18f401d293779998 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:48.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.787+0000 7f7441ffb640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-03-28T22:15:48.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.787+0000 7f7441ffb640 1 -- 172.21.15.16:0/813098107 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f745001d050 con 0x55e9df1f1d90 2024-03-28T22:15:48.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.787+0000 7f7441ffb640 1 -- 172.21.15.16:0/813098107 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f7450003f50 con 0x55e9df1f1d90 2024-03-28T22:15:48.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.787+0000 7f7463643740 1 -- 172.21.15.16:0/813098107 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74500041c0 con 0x55e9df1f1d90 2024-03-28T22:15:48.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.787+0000 7f7441ffb640 1 -- 172.21.15.16:0/813098107 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7450032050 con 0x55e9df1f1d90 2024-03-28T22:15:48.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.787+0000 7f7463643740 10 client.26504 fetch_fsmap learned FSMap version 27 2024-03-28T22:15:48.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.787+0000 7f7463643740 1 -- 172.21.15.16:0/813098107 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e9def337d0 con 0x55e9df1f1d90 2024-03-28T22:15:48.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.788+0000 7f7441ffb640 1 -- 172.21.15.16:0/813098107 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74500041c0 con 0x55e9df1f1d90 2024-03-28T22:15:48.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.788+0000 7f7441ffb640 1 -- 172.21.15.16:0/813098107 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f7450004c90 con 0x55e9df1f1d90 2024-03-28T22:15:48.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.788+0000 7f7463643740 10 client.26504 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:15:48.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.788+0000 7f7463643740 10 client.26504 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T22:15:48.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.788+0000 7f7463643740 20 client.26504 populate_metadata read hostname 'smithi016' 2024-03-28T22:15:48.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.788+0000 7f7463643740 1 --2- 172.21.15.16:0/813098107 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55e9df20a950 0x55e9df20cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.788+0000 7f7463643740 4 client.26504 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:15:48.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.788+0000 7f7463643740 1 -- 172.21.15.16:0/813098107 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e9df1626f0 con 0x55e9df20a950 2024-03-28T22:15:48.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.788+0000 7f745a59c640 1 --2- 172.21.15.16:0/813098107 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55e9df20a950 0x55e9df20cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.788+0000 7f745a59c640 1 --2- 172.21.15.16:0/813098107 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55e9df20a950 0x55e9df20cd40 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=856667ebf6ca2299 server_cookie=9daf21e6b2fb6f10 in_seq=0 out_seq=0 2024-03-28T22:15:48.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.788+0000 7f7441ffb640 10 client.26504 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:15:48.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.789+0000 7f7441ffb640 1 -- 172.21.15.16:0/813098107 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e9df1626f0 con 0x55e9df20a950 2024-03-28T22:15:48.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.789+0000 7f7441ffb640 10 client.26504 handle_command_reply: tid=0 2024-03-28T22:15:48.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.789+0000 7f7463643740 1 client.26504 shutdown 2024-03-28T22:15:48.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.789+0000 7f7463643740 1 -- 172.21.15.16:0/813098107 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e9df1f1d90 msgr2=0x55e9df1f7ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.789+0000 7f7463643740 1 --2- 172.21.15.16:0/813098107 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e9df1f1d90 0x55e9df1f7ef0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x55e9df1ecd30 tx=0x7f7450003bf0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.790+0000 7f7463643740 1 -- 172.21.15.16:0/813098107 shutdown_connections 2024-03-28T22:15:48.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.790+0000 7f7463643740 1 --2- 172.21.15.16:0/813098107 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55e9df20a950 0x55e9df20cd40 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.790+0000 7f7463643740 1 --2- 172.21.15.16:0/813098107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9df1fa970 0x55e9df1f8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.790+0000 7f7463643740 1 --2- 172.21.15.16:0/813098107 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e9df1f1d90 0x55e9df1f7ef0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.790+0000 7f7463643740 1 --2- 172.21.15.16:0/813098107 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9df1f1470 0x55e9df1f79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.790+0000 7f7441ffb640 0 client.26504 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:15:48.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.790+0000 7f7463643740 1 -- 172.21.15.16:0/813098107 >> 172.21.15.16:0/813098107 conn(0x55e9df1ebba0 msgr2=0x55e9df1ed8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:48.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.790+0000 7f7463643740 1 -- 172.21.15.16:0/813098107 shutdown_connections 2024-03-28T22:15:48.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.790+0000 7f7463643740 1 -- 172.21.15.16:0/813098107 wait complete. 2024-03-28T22:15:48.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.790+0000 7f7463643740 20 client.26504 trim_cache size 0 max 16384 2024-03-28T22:15:48.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.803+0000 7f7463643740 1 Processor -- start 2024-03-28T22:15:48.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.803+0000 7f7463643740 1 -- start start 2024-03-28T22:15:48.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.803+0000 7f7463643740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9df1f1470 0x55e9df1fa0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.803+0000 7f7463643740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e9df1f1d90 0x55e9df206530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.803+0000 7f7463643740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9df1fa970 0x55e9df1ed860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.803+0000 7f7463643740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f745c1afc00 con 0x55e9df1fa970 2024-03-28T22:15:48.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.803+0000 7f7463643740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f745c1ad4d0 con 0x55e9df1f1d90 2024-03-28T22:15:48.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.803+0000 7f7463643740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f745c1ab320 con 0x55e9df1f1470 2024-03-28T22:15:48.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.803+0000 7f745ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9df1fa970 0x55e9df1ed860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.803+0000 7f745ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9df1fa970 0x55e9df1ed860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57026/0 (socket says 172.21.15.16:57026) 2024-03-28T22:15:48.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.803+0000 7f745ad9d640 1 -- 172.21.15.16:0/313360594 learned_addr learned my addr 172.21.15.16:0/313360594 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:48.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.803+0000 7f7459d9b640 1 --2- 172.21.15.16:0/313360594 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e9df1f1d90 0x55e9df206530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.803+0000 7f745a59c640 1 --2- 172.21.15.16:0/313360594 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9df1f1470 0x55e9df1fa0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.803+0000 7f745ad9d640 1 -- 172.21.15.16:0/313360594 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9df1f1470 msgr2=0x55e9df1fa0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:48.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.803+0000 7f745ad9d640 1 --2- 172.21.15.16:0/313360594 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9df1f1470 0x55e9df1fa0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.803+0000 7f745ad9d640 1 -- 172.21.15.16:0/313360594 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e9df1f1d90 msgr2=0x55e9df206530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:48.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.803+0000 7f745ad9d640 1 --2- 172.21.15.16:0/313360594 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e9df1f1d90 0x55e9df206530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.803+0000 7f745ad9d640 1 -- 172.21.15.16:0/313360594 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9df1f6960 con 0x55e9df1fa970 2024-03-28T22:15:48.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.803+0000 7f745a59c640 1 --2- 172.21.15.16:0/313360594 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9df1f1470 0x55e9df1fa0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:15:48.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.803+0000 7f7459d9b640 1 --2- 172.21.15.16:0/313360594 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e9df1f1d90 0x55e9df206530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:15:48.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.804+0000 7f745ad9d640 1 --2- 172.21.15.16:0/313360594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9df1fa970 0x55e9df1ed860 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f744c02c700 tx=0x7f744c041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc0df8eb826d44a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:48.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.804+0000 7f7443fff640 1 -- 172.21.15.16:0/313360594 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f744c039070 con 0x55e9df1fa970 2024-03-28T22:15:48.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.804+0000 7f7443fff640 1 -- 172.21.15.16:0/313360594 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f744c003e50 con 0x55e9df1fa970 2024-03-28T22:15:48.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.804+0000 7f7443fff640 1 -- 172.21.15.16:0/313360594 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f744c0042a0 con 0x55e9df1fa970 2024-03-28T22:15:48.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.804+0000 7f7463643740 1 -- 172.21.15.16:0/313360594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9df1fa970 msgr2=0x55e9df1ed860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.804+0000 7f7463643740 1 --2- 172.21.15.16:0/313360594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9df1fa970 0x55e9df1ed860 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f744c02c700 tx=0x7f744c041000 comp rx=0 tx=0).stop 2024-03-28T22:15:48.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.804+0000 7f7463643740 1 -- 172.21.15.16:0/313360594 shutdown_connections 2024-03-28T22:15:48.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.804+0000 7f7463643740 1 --2- 172.21.15.16:0/313360594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9df1fa970 0x55e9df1ed860 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.804+0000 7f7463643740 1 --2- 172.21.15.16:0/313360594 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e9df1f1d90 0x55e9df206530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.804+0000 7f7463643740 1 --2- 172.21.15.16:0/313360594 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9df1f1470 0x55e9df1fa0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.804+0000 7f7463643740 1 -- 172.21.15.16:0/313360594 >> 172.21.15.16:0/313360594 conn(0x55e9df1ebba0 msgr2=0x55e9df1ebf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:48.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.804+0000 7f7463643740 1 -- 172.21.15.16:0/313360594 shutdown_connections 2024-03-28T22:15:48.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.804+0000 7f7463643740 1 -- 172.21.15.16:0/313360594 wait complete. 2024-03-28T22:15:48.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.805+0000 7f7463643740 1 -- start start 2024-03-28T22:15:48.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.805+0000 7f7463643740 1 Processor -- start 2024-03-28T22:15:48.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.805+0000 7f7463643740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9df1f1470 0x55e9df1feb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.805+0000 7f7463643740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9df1f1d90 0x55e9df1ff090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.805+0000 7f7463643740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e9df1fa970 0x55e9df273f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.805+0000 7f7463643740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f745c1ab990 con 0x55e9df1f1d90 2024-03-28T22:15:48.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.805+0000 7f7463643740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f745c1ace00 con 0x55e9df1fa970 2024-03-28T22:15:48.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.805+0000 7f7463643740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f745c1ac060 con 0x55e9df1f1470 2024-03-28T22:15:48.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.805+0000 7f7463643740 10 client.0 _ll_register_callbacks cb 0x7f7450009e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:15:48.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.805+0000 7f7459d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9df1f1d90 0x55e9df1ff090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.805+0000 7f7459d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9df1f1d90 0x55e9df1ff090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57042/0 (socket says 172.21.15.16:57042) 2024-03-28T22:15:48.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.805+0000 7f7459d9b640 1 -- 172.21.15.16:0/1797425934 learned_addr learned my addr 172.21.15.16:0/1797425934 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:48.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.805+0000 7f745ad9d640 1 --2- 172.21.15.16:0/1797425934 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e9df1fa970 0x55e9df273f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.805+0000 7f745a59c640 1 --2- 172.21.15.16:0/1797425934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9df1f1470 0x55e9df1feb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.805+0000 7f7459d9b640 1 -- 172.21.15.16:0/1797425934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9df1f1470 msgr2=0x55e9df1feb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:48.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.805+0000 7f7459d9b640 1 --2- 172.21.15.16:0/1797425934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9df1f1470 0x55e9df1feb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.805+0000 7f7459d9b640 1 -- 172.21.15.16:0/1797425934 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e9df1fa970 msgr2=0x55e9df273f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:48.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.805+0000 7f7459d9b640 1 --2- 172.21.15.16:0/1797425934 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e9df1fa970 0x55e9df273f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.805+0000 7f7459d9b640 1 -- 172.21.15.16:0/1797425934 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9df2744d0 con 0x55e9df1f1d90 2024-03-28T22:15:48.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.806+0000 7f7459d9b640 1 --2- 172.21.15.16:0/1797425934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9df1f1d90 0x55e9df1ff090 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x55e9df1f7210 tx=0x7f7450005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6aa2da4028f79ad server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:48.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.806+0000 7f7441ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:15:48.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.806+0000 7f7441ffb640 1 -- 172.21.15.16:0/1797425934 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f745001d040 con 0x55e9df1f1d90 2024-03-28T22:15:48.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.806+0000 7f7441ffb640 1 -- 172.21.15.16:0/1797425934 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f7450004120 con 0x55e9df1f1d90 2024-03-28T22:15:48.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.806+0000 7f7463643740 1 -- 172.21.15.16:0/1797425934 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f744c003e50 con 0x55e9df1f1d90 2024-03-28T22:15:48.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.806+0000 7f7441ffb640 1 -- 172.21.15.16:0/1797425934 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f745000fe60 con 0x55e9df1f1d90 2024-03-28T22:15:48.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.806+0000 7f7463643740 10 client.16608 fetch_fsmap learned FSMap version 27 2024-03-28T22:15:48.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.806+0000 7f7463643740 1 -- 172.21.15.16:0/1797425934 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e9df275780 con 0x55e9df1f1d90 2024-03-28T22:15:48.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.806+0000 7f7441ffb640 1 -- 172.21.15.16:0/1797425934 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f744c003e50 con 0x55e9df1f1d90 2024-03-28T22:15:48.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.806+0000 7f7441ffb640 1 -- 172.21.15.16:0/1797425934 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f745003d030 con 0x55e9df1f1d90 2024-03-28T22:15:48.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.806+0000 7f7463643740 10 client.16608 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:15:48.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.806+0000 7f7463643740 10 client.16608 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T22:15:48.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.806+0000 7f7463643740 20 client.16608 populate_metadata read hostname 'smithi016' 2024-03-28T22:15:48.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.806+0000 7f7463643740 1 --2- 172.21.15.16:0/1797425934 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55e9df275ea0 0x55e9df2782c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.806+0000 7f7463643740 4 client.16608 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:15:48.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.806+0000 7f7463643740 1 -- 172.21.15.16:0/1797425934 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e9df15ca40 con 0x55e9df275ea0 2024-03-28T22:15:48.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.806+0000 7f745a59c640 1 --2- 172.21.15.16:0/1797425934 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55e9df275ea0 0x55e9df2782c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.807+0000 7f7441ffb640 10 client.16608 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:15:48.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.807+0000 7f745a59c640 1 --2- 172.21.15.16:0/1797425934 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55e9df275ea0 0x55e9df2782c0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3e0f8c94e56fa507 server_cookie=e0943f3bc7397232 in_seq=0 out_seq=0 2024-03-28T22:15:48.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.807+0000 7f7441ffb640 1 -- 172.21.15.16:0/1797425934 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e9df15ca40 con 0x55e9df275ea0 2024-03-28T22:15:48.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.807+0000 7f7441ffb640 10 client.16608 handle_command_reply: tid=0 2024-03-28T22:15:48.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.807+0000 7f7463643740 1 client.16608 shutdown 2024-03-28T22:15:48.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.807+0000 7f7463643740 1 -- 172.21.15.16:0/1797425934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9df1f1d90 msgr2=0x55e9df1ff090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.807+0000 7f7463643740 1 --2- 172.21.15.16:0/1797425934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9df1f1d90 0x55e9df1ff090 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x55e9df1f7210 tx=0x7f7450005850 comp rx=0 tx=0).stop 2024-03-28T22:15:48.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.807+0000 7f7463643740 1 -- 172.21.15.16:0/1797425934 shutdown_connections 2024-03-28T22:15:48.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.807+0000 7f7463643740 1 --2- 172.21.15.16:0/1797425934 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55e9df275ea0 0x55e9df2782c0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.807+0000 7f7463643740 1 --2- 172.21.15.16:0/1797425934 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e9df1fa970 0x55e9df273f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.807+0000 7f7463643740 1 --2- 172.21.15.16:0/1797425934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e9df1f1d90 0x55e9df1ff090 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.807+0000 7f7463643740 1 --2- 172.21.15.16:0/1797425934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9df1f1470 0x55e9df1feb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.807+0000 7f7441ffb640 0 client.16608 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:15:48.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.807+0000 7f7463643740 1 -- 172.21.15.16:0/1797425934 >> 172.21.15.16:0/1797425934 conn(0x55e9df1ebba0 msgr2=0x55e9df1f81e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:48.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.807+0000 7f7463643740 1 -- 172.21.15.16:0/1797425934 shutdown_connections 2024-03-28T22:15:48.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.807+0000 7f7463643740 1 -- 172.21.15.16:0/1797425934 wait complete. 2024-03-28T22:15:48.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.807+0000 7f7463643740 20 client.16608 trim_cache size 0 max 16384 2024-03-28T22:15:48.827 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:15:48.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.808+0000 7f7460a64640 1 -- 172.21.15.16:0/3500113639 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f74440a1ef0 msgr2=0x7f74440a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.808+0000 7f7460a64640 1 --2- 172.21.15.16:0/3500113639 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f74440a1ef0 0x7f74440a43b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f744c006010 tx=0x7f744c0095c0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.808+0000 7f7460a64640 1 -- 172.21.15.16:0/3500113639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f745c1005e0 msgr2=0x7f745c197f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.808+0000 7f7460a64640 1 --2- 172.21.15.16:0/3500113639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f745c1005e0 0x7f745c197f00 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f745400d950 tx=0x7f745400de10 comp rx=0 tx=0).stop 2024-03-28T22:15:48.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.808+0000 7f7460a64640 1 -- 172.21.15.16:0/3500113639 shutdown_connections 2024-03-28T22:15:48.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.808+0000 7f7460a64640 1 --2- 172.21.15.16:0/3500113639 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f74440a1ef0 0x7f74440a43b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.808+0000 7f7460a64640 1 --2- 172.21.15.16:0/3500113639 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f745c106680 0x7f745c19c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.808+0000 7f7460a64640 1 --2- 172.21.15.16:0/3500113639 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f745c100fa0 0x7f745c198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.808+0000 7f7460a64640 1 --2- 172.21.15.16:0/3500113639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f745c1005e0 0x7f745c197f00 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.808+0000 7f7460a64640 1 -- 172.21.15.16:0/3500113639 >> 172.21.15.16:0/3500113639 conn(0x7f745c077770 msgr2=0x7f745c104650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:48.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.808+0000 7f7460a64640 1 -- 172.21.15.16:0/3500113639 shutdown_connections 2024-03-28T22:15:48.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.808+0000 7f7460a64640 1 -- 172.21.15.16:0/3500113639 wait complete. 2024-03-28T22:15:48.831 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:15:48.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.933+0000 7f5cfa8c0640 1 -- 172.21.15.16:0/2516388881 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cf4107e90 msgr2=0x7f5cf410e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.933+0000 7f5cfa8c0640 1 --2- 172.21.15.16:0/2516388881 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cf4107e90 0x7f5cf410e810 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5cec009c10 tx=0x7f5cec02bef0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.933+0000 7f5cfa8c0640 1 -- 172.21.15.16:0/2516388881 shutdown_connections 2024-03-28T22:15:48.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.933+0000 7f5cfa8c0640 1 --2- 172.21.15.16:0/2516388881 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cf4107e90 0x7f5cf410e810 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.933+0000 7f5cfa8c0640 1 --2- 172.21.15.16:0/2516388881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cf4103330 0x7f5cf4107760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.933+0000 7f5cfa8c0640 1 --2- 172.21.15.16:0/2516388881 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5cf4102980 0x7f5cf4102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.933+0000 7f5cfa8c0640 1 -- 172.21.15.16:0/2516388881 >> 172.21.15.16:0/2516388881 conn(0x7f5cf4077830 msgr2=0x7f5cf4077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:48.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.934+0000 7f5cfa8c0640 1 -- 172.21.15.16:0/2516388881 shutdown_connections 2024-03-28T22:15:48.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.934+0000 7f5cfa8c0640 1 -- 172.21.15.16:0/2516388881 wait complete. 2024-03-28T22:15:48.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.934+0000 7f5cfa8c0640 1 Processor -- start 2024-03-28T22:15:48.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.934+0000 7f5cfa8c0640 1 -- start start 2024-03-28T22:15:48.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.934+0000 7f5cfa8c0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cf4102980 0x7f5cf4198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.934+0000 7f5cfa8c0640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cf4107e90 0x7f5cf4198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.934+0000 7f5cfa8c0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5cf419c9c0 0x7f5cf419ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.934+0000 7f5cfa8c0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf4111cc0 con 0x7f5cf4102980 2024-03-28T22:15:48.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.934+0000 7f5cfa8c0640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf4111b40 con 0x7f5cf4107e90 2024-03-28T22:15:48.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.934+0000 7f5cfa8c0640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf4111e40 con 0x7f5cf419c9c0 2024-03-28T22:15:48.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.935+0000 7f5cf3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cf4102980 0x7f5cf4198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.935+0000 7f5cf3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cf4102980 0x7f5cf4198100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57060/0 (socket says 172.21.15.16:57060) 2024-03-28T22:15:48.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.935+0000 7f5cf3fff640 1 -- 172.21.15.16:0/1470582806 learned_addr learned my addr 172.21.15.16:0/1470582806 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:48.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.935+0000 7f5cf37fe640 1 --2- 172.21.15.16:0/1470582806 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cf4107e90 0x7f5cf4198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.935+0000 7f5cf8e5d640 1 --2- 172.21.15.16:0/1470582806 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5cf419c9c0 0x7f5cf419ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.935+0000 7f5cf3fff640 1 -- 172.21.15.16:0/1470582806 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5cf419c9c0 msgr2=0x7f5cf419ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.935+0000 7f5cf3fff640 1 --2- 172.21.15.16:0/1470582806 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5cf419c9c0 0x7f5cf419ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.935+0000 7f5cf3fff640 1 -- 172.21.15.16:0/1470582806 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cf4107e90 msgr2=0x7f5cf4198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.935+0000 7f5cf3fff640 1 --2- 172.21.15.16:0/1470582806 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cf4107e90 0x7f5cf4198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.935+0000 7f5cf3fff640 1 -- 172.21.15.16:0/1470582806 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cf419d4b0 con 0x7f5cf4102980 2024-03-28T22:15:48.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.935+0000 7f5cf8e5d640 1 --2- 172.21.15.16:0/1470582806 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5cf419c9c0 0x7f5cf419ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:15:48.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.935+0000 7f5cf3fff640 1 --2- 172.21.15.16:0/1470582806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cf4102980 0x7f5cf4198100 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f5ce400c890 tx=0x7f5ce400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:48.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.935+0000 7f5cf17fa640 1 -- 172.21.15.16:0/1470582806 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce4004220 con 0x7f5cf4102980 2024-03-28T22:15:48.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.935+0000 7f5cfa8c0640 1 -- 172.21.15.16:0/1470582806 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cf419d7a0 con 0x7f5cf4102980 2024-03-28T22:15:48.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.935+0000 7f5cf17fa640 1 -- 172.21.15.16:0/1470582806 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ce40043c0 con 0x7f5cf4102980 2024-03-28T22:15:48.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.935+0000 7f5cf17fa640 1 -- 172.21.15.16:0/1470582806 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce4004a00 con 0x7f5cf4102980 2024-03-28T22:15:48.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.935+0000 7f5cfa8c0640 1 -- 172.21.15.16:0/1470582806 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cf41a4f90 con 0x7f5cf4102980 2024-03-28T22:15:48.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.937+0000 7f5cf17fa640 1 -- 172.21.15.16:0/1470582806 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f5ce40079b0 con 0x7f5cf4102980 2024-03-28T22:15:48.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.938+0000 7f5cfd57d740 1 Processor -- start 2024-03-28T22:15:48.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.938+0000 7f5cfd57d740 1 -- start start 2024-03-28T22:15:48.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.938+0000 7f5cfd57d740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562513163470 0x562513163850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.938+0000 7f5cfd57d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562513163d90 0x56251316c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.938+0000 7f5cfd57d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56251316c970 0x56251316ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.938+0000 7f5cfd57d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5625130c71b0 con 0x56251316c970 2024-03-28T22:15:48.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.938+0000 7f5cfd57d740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x56251306fc00 con 0x562513163470 2024-03-28T22:15:48.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.938+0000 7f5cfd57d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5625130c3250 con 0x562513163d90 2024-03-28T22:15:48.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.938+0000 7f5cf37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56251316c970 0x56251316ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.938+0000 7f5cf8e5d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562513163470 0x562513163850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.938+0000 7f5cf37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56251316c970 0x56251316ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57076/0 (socket says 172.21.15.16:57076) 2024-03-28T22:15:48.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.938+0000 7f5cf37fe640 1 -- 172.21.15.16:0/3018072751 learned_addr learned my addr 172.21.15.16:0/3018072751 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:48.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.938+0000 7f5cf17fa640 1 --2- 172.21.15.16:0/1470582806 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f5ce00a1ef0 0x7f5ce00a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.938+0000 7f5cf8e5d640 1 --2- 172.21.15.16:0/1470582806 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f5ce00a1ef0 0x7f5ce00a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.938+0000 7f5cf17fa640 1 -- 172.21.15.16:0/1470582806 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f5ce40cded0 con 0x7f5cf4102980 2024-03-28T22:15:48.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.938+0000 7f5cf3fff640 1 --2- 172.21.15.16:0/3018072751 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562513163d90 0x56251316c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.938+0000 7f5cf37fe640 1 -- 172.21.15.16:0/3018072751 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562513163d90 msgr2=0x56251316c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:48.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.938+0000 7f5cf37fe640 1 --2- 172.21.15.16:0/3018072751 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562513163d90 0x56251316c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.938+0000 7f5cf37fe640 1 -- 172.21.15.16:0/3018072751 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562513163470 msgr2=0x562513163850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:48.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.938+0000 7f5cf37fe640 1 --2- 172.21.15.16:0/3018072751 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562513163470 0x562513163850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.938+0000 7f5cf37fe640 1 -- 172.21.15.16:0/3018072751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56251308f3d0 con 0x56251316c970 2024-03-28T22:15:48.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.938+0000 7f5cf8e5d640 1 --2- 172.21.15.16:0/3018072751 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562513163470 0x562513163850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:15:48.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.939+0000 7f5cf37fe640 1 --2- 172.21.15.16:0/3018072751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56251316c970 0x56251316ed60 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc009c30 tx=0x7f5cdc011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5754cdcda3fc425 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:48.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.939+0000 7f5cf8e5d640 1 --2- 172.21.15.16:0/1470582806 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f5ce00a1ef0 0x7f5ce00a43b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5cec006010 tx=0x7f5cec02bef0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:48.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.939+0000 7f5cea7fc640 1 -- 172.21.15.16:0/3018072751 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cdc009de0 con 0x56251316c970 2024-03-28T22:15:48.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.939+0000 7f5cea7fc640 1 -- 172.21.15.16:0/3018072751 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5cdc0035f0 con 0x56251316c970 2024-03-28T22:15:48.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.939+0000 7f5cea7fc640 1 -- 172.21.15.16:0/3018072751 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cdc019650 con 0x56251316c970 2024-03-28T22:15:48.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.939+0000 7f5cfd57d740 1 -- 172.21.15.16:0/3018072751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56251316c970 msgr2=0x56251316ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.939+0000 7f5cfd57d740 1 --2- 172.21.15.16:0/3018072751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56251316c970 0x56251316ed60 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc009c30 tx=0x7f5cdc011810 comp rx=0 tx=0).stop 2024-03-28T22:15:48.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.939+0000 7f5cfd57d740 1 -- 172.21.15.16:0/3018072751 shutdown_connections 2024-03-28T22:15:48.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.939+0000 7f5cfd57d740 1 --2- 172.21.15.16:0/3018072751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56251316c970 0x56251316ed60 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.939+0000 7f5cfd57d740 1 --2- 172.21.15.16:0/3018072751 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562513163d90 0x56251316c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.939+0000 7f5cfd57d740 1 --2- 172.21.15.16:0/3018072751 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562513163470 0x562513163850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.939+0000 7f5cfd57d740 1 -- 172.21.15.16:0/3018072751 >> 172.21.15.16:0/3018072751 conn(0x56251315dba0 msgr2=0x56251315df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:48.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.939+0000 7f5cfd57d740 1 -- 172.21.15.16:0/3018072751 shutdown_connections 2024-03-28T22:15:48.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.939+0000 7f5cfd57d740 1 -- 172.21.15.16:0/3018072751 wait complete. 2024-03-28T22:15:48.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.940+0000 7f5cfd57d740 1 -- start start 2024-03-28T22:15:48.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.940+0000 7f5cfd57d740 1 Processor -- start 2024-03-28T22:15:48.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.940+0000 7f5cfd57d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562513163470 0x562513169950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.940+0000 7f5cfd57d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562513163d90 0x562513169e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.940+0000 7f5cfd57d740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56251316c970 0x56251316a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.940+0000 7f5cfd57d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56251316f760 con 0x562513163470 2024-03-28T22:15:48.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.940+0000 7f5cfd57d740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x562513065470 con 0x56251316c970 2024-03-28T22:15:48.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.940+0000 7f5cfd57d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56251316f8e0 con 0x562513163d90 2024-03-28T22:15:48.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.940+0000 7f5cfd57d740 10 client.0 _ll_register_callbacks cb 0x562513082cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:15:48.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.940+0000 7f5cf3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562513163470 0x562513169950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.940+0000 7f5cf3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562513163470 0x562513169950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57090/0 (socket says 172.21.15.16:57090) 2024-03-28T22:15:48.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.940+0000 7f5cf3fff640 1 -- 172.21.15.16:0/2492346597 learned_addr learned my addr 172.21.15.16:0/2492346597 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:48.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.940+0000 7f5cf8e5d640 1 --2- 172.21.15.16:0/2492346597 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56251316c970 0x56251316a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.940+0000 7f5cf37fe640 1 --2- 172.21.15.16:0/2492346597 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562513163d90 0x562513169e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.941+0000 7f5cf3fff640 1 -- 172.21.15.16:0/2492346597 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562513163d90 msgr2=0x562513169e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:48.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.941+0000 7f5cf3fff640 1 --2- 172.21.15.16:0/2492346597 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562513163d90 0x562513169e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.941+0000 7f5cf3fff640 1 -- 172.21.15.16:0/2492346597 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56251316c970 msgr2=0x56251316a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:48.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.941+0000 7f5cf3fff640 1 --2- 172.21.15.16:0/2492346597 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56251316c970 0x56251316a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.941+0000 7f5cf3fff640 1 -- 172.21.15.16:0/2492346597 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562512ddb3f0 con 0x562513163470 2024-03-28T22:15:48.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.941+0000 7f5cf8e5d640 1 --2- 172.21.15.16:0/2492346597 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56251316c970 0x56251316a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:15:48.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.941+0000 7f5cf3fff640 1 --2- 172.21.15.16:0/2492346597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562513163470 0x562513169950 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f5ce40058c0 tx=0x7f5ce4096c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1dc386a1a895045 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:48.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.941+0000 7f5cd37fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:15:48.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.941+0000 7f5cd37fe640 1 -- 172.21.15.16:0/2492346597 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce40b3060 con 0x562513163470 2024-03-28T22:15:48.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.941+0000 7f5cd37fe640 1 -- 172.21.15.16:0/2492346597 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f5ce40b3200 con 0x562513163470 2024-03-28T22:15:48.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.941+0000 7f5cfd57d740 1 -- 172.21.15.16:0/2492346597 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5cdc0035f0 con 0x562513163470 2024-03-28T22:15:48.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.941+0000 7f5cd37fe640 1 -- 172.21.15.16:0/2492346597 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5ce40b3400 con 0x562513163470 2024-03-28T22:15:48.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.941+0000 7f5cfd57d740 10 client.16620 fetch_fsmap learned FSMap version 27 2024-03-28T22:15:48.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.941+0000 7f5cfd57d740 1 -- 172.21.15.16:0/2492346597 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562512ea57d0 con 0x562513163470 2024-03-28T22:15:48.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.941+0000 7f5cd37fe640 1 -- 172.21.15.16:0/2492346597 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5cdc0035f0 con 0x562513163470 2024-03-28T22:15:48.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.941+0000 7f5cd37fe640 1 -- 172.21.15.16:0/2492346597 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f5ce4085070 con 0x562513163470 2024-03-28T22:15:48.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.941+0000 7f5cfd57d740 10 client.16620 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:15:48.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.941+0000 7f5cfd57d740 10 client.16620 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:15:48.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.942+0000 7f5cfd57d740 20 client.16620 populate_metadata read hostname 'smithi016' 2024-03-28T22:15:48.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.942+0000 7f5cfd57d740 1 --2- 172.21.15.16:0/2492346597 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x56251317c9a0 0x56251317ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.942+0000 7f5cfd57d740 4 client.16620 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:15:48.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.942+0000 7f5cfd57d740 1 -- 172.21.15.16:0/2492346597 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5625130d46f0 con 0x56251317c9a0 2024-03-28T22:15:48.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.942+0000 7f5cf37fe640 1 --2- 172.21.15.16:0/2492346597 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x56251317c9a0 0x56251317ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.942+0000 7f5cf37fe640 1 --2- 172.21.15.16:0/2492346597 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x56251317c9a0 0x56251317ed90 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0a49a6f8a9f193c server_cookie=102ad40516e02aae in_seq=0 out_seq=0 2024-03-28T22:15:48.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.942+0000 7f5cd37fe640 10 client.16620 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:15:48.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.943+0000 7f5cd37fe640 1 -- 172.21.15.16:0/2492346597 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5625130d46f0 con 0x56251317c9a0 2024-03-28T22:15:48.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.943+0000 7f5cd37fe640 10 client.16620 handle_command_reply: tid=0 2024-03-28T22:15:48.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.943+0000 7f5cfd57d740 1 client.16620 shutdown 2024-03-28T22:15:48.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.943+0000 7f5cfd57d740 1 -- 172.21.15.16:0/2492346597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562513163470 msgr2=0x562513169950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.943+0000 7f5cfd57d740 1 --2- 172.21.15.16:0/2492346597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562513163470 0x562513169950 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f5ce40058c0 tx=0x7f5ce4096c80 comp rx=0 tx=0).stop 2024-03-28T22:15:48.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.943+0000 7f5cfd57d740 1 -- 172.21.15.16:0/2492346597 shutdown_connections 2024-03-28T22:15:48.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.943+0000 7f5cfd57d740 1 --2- 172.21.15.16:0/2492346597 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x56251317c9a0 0x56251317ed90 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.943+0000 7f5cfd57d740 1 --2- 172.21.15.16:0/2492346597 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56251316c970 0x56251316a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.943+0000 7f5cfd57d740 1 --2- 172.21.15.16:0/2492346597 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562513163d90 0x562513169e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.943+0000 7f5cd37fe640 0 client.16620 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:15:48.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.943+0000 7f5cfd57d740 1 --2- 172.21.15.16:0/2492346597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562513163470 0x562513169950 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.943+0000 7f5cfd57d740 1 -- 172.21.15.16:0/2492346597 >> 172.21.15.16:0/2492346597 conn(0x56251315dba0 msgr2=0x56251315f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:48.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.943+0000 7f5cfd57d740 1 -- 172.21.15.16:0/2492346597 shutdown_connections 2024-03-28T22:15:48.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.943+0000 7f5cfd57d740 1 -- 172.21.15.16:0/2492346597 wait complete. 2024-03-28T22:15:48.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.944+0000 7f5cfd57d740 20 client.16620 trim_cache size 0 max 16384 2024-03-28T22:15:48.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.957+0000 7f5cfd57d740 1 Processor -- start 2024-03-28T22:15:48.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.957+0000 7f5cfd57d740 1 -- start start 2024-03-28T22:15:48.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.957+0000 7f5cfd57d740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562513163470 0x56251316c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.957+0000 7f5cfd57d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562513163d90 0x562513178530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.957+0000 7f5cfd57d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56251316c970 0x562513184630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.957+0000 7f5cfd57d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf41afc50 con 0x562513163d90 2024-03-28T22:15:48.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.957+0000 7f5cfd57d740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf41ad520 con 0x562513163470 2024-03-28T22:15:48.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.957+0000 7f5cfd57d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf41ab390 con 0x56251316c970 2024-03-28T22:15:48.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.957+0000 7f5cf37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562513163d90 0x562513178530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.957+0000 7f5cf37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562513163d90 0x562513178530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57098/0 (socket says 172.21.15.16:57098) 2024-03-28T22:15:48.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.957+0000 7f5cf37fe640 1 -- 172.21.15.16:0/388831274 learned_addr learned my addr 172.21.15.16:0/388831274 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:48.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.957+0000 7f5cf3fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562513163470 0x56251316c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.957+0000 7f5cf37fe640 1 -- 172.21.15.16:0/388831274 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56251316c970 msgr2=0x562513184630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:48.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.957+0000 7f5cf37fe640 1 --2- 172.21.15.16:0/388831274 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56251316c970 0x562513184630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.957+0000 7f5cf37fe640 1 -- 172.21.15.16:0/388831274 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562513163470 msgr2=0x56251316c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:48.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.957+0000 7f5cf37fe640 1 --2- 172.21.15.16:0/388831274 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562513163470 0x56251316c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.957+0000 7f5cf37fe640 1 -- 172.21.15.16:0/388831274 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562513159460 con 0x562513163d90 2024-03-28T22:15:48.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.957+0000 7f5cf37fe640 1 --2- 172.21.15.16:0/388831274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562513163d90 0x562513178530 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc000c00 tx=0x7f5cdc0097b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a799c69f23198011 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:48.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.957+0000 7f5ce9ffb640 1 -- 172.21.15.16:0/388831274 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cdc031070 con 0x562513163d90 2024-03-28T22:15:48.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.958+0000 7f5ce9ffb640 1 -- 172.21.15.16:0/388831274 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5cdc009d20 con 0x562513163d90 2024-03-28T22:15:48.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.958+0000 7f5ce9ffb640 1 -- 172.21.15.16:0/388831274 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cdc004a80 con 0x562513163d90 2024-03-28T22:15:48.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.958+0000 7f5cfd57d740 1 -- 172.21.15.16:0/388831274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562513163d90 msgr2=0x562513178530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.958+0000 7f5cfd57d740 1 --2- 172.21.15.16:0/388831274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562513163d90 0x562513178530 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc000c00 tx=0x7f5cdc0097b0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.958+0000 7f5cfd57d740 1 -- 172.21.15.16:0/388831274 shutdown_connections 2024-03-28T22:15:48.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.958+0000 7f5cfd57d740 1 --2- 172.21.15.16:0/388831274 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56251316c970 0x562513184630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.958+0000 7f5cfd57d740 1 --2- 172.21.15.16:0/388831274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562513163d90 0x562513178530 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.958+0000 7f5cfd57d740 1 --2- 172.21.15.16:0/388831274 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562513163470 0x56251316c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.958+0000 7f5cfd57d740 1 -- 172.21.15.16:0/388831274 >> 172.21.15.16:0/388831274 conn(0x56251315dba0 msgr2=0x56251315cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:48.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.958+0000 7f5cfd57d740 1 -- 172.21.15.16:0/388831274 shutdown_connections 2024-03-28T22:15:48.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.958+0000 7f5cfd57d740 1 -- 172.21.15.16:0/388831274 wait complete. 2024-03-28T22:15:48.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.958+0000 7f5cfd57d740 1 -- start start 2024-03-28T22:15:48.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.959+0000 7f5cfd57d740 1 Processor -- start 2024-03-28T22:15:48.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.959+0000 7f5cfd57d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562513163470 0x562513170c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.959+0000 7f5cfd57d740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562513163d90 0x5625131711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.959+0000 7f5cfd57d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56251316c970 0x5625131dbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.959+0000 7f5cfd57d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf41ab9e0 con 0x56251316c970 2024-03-28T22:15:48.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.959+0000 7f5cfd57d740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf41ace50 con 0x562513163d90 2024-03-28T22:15:48.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.959+0000 7f5cfd57d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf41ac0b0 con 0x562513163470 2024-03-28T22:15:48.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.959+0000 7f5cfd57d740 10 client.0 _ll_register_callbacks cb 0x7f5ce400a520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:15:48.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.959+0000 7f5cf8e5d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56251316c970 0x5625131dbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.959+0000 7f5cf37fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562513163d90 0x5625131711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.959+0000 7f5cf37fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562513163d90 0x5625131711b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:48964/0 (socket says 172.21.15.16:48964) 2024-03-28T22:15:48.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.959+0000 7f5cf37fe640 1 -- 172.21.15.16:0/579449828 learned_addr learned my addr 172.21.15.16:0/579449828 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:48.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.959+0000 7f5cf8e5d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56251316c970 0x5625131dbc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57112/0 (socket says 172.21.15.16:57112) 2024-03-28T22:15:48.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.959+0000 7f5cf3fff640 1 --2- 172.21.15.16:0/579449828 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562513163470 0x562513170c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.959+0000 7f5cf8e5d640 1 -- 172.21.15.16:0/579449828 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562513163470 msgr2=0x562513170c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:48.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.959+0000 7f5cf8e5d640 1 --2- 172.21.15.16:0/579449828 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562513163470 0x562513170c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.959+0000 7f5cf8e5d640 1 -- 172.21.15.16:0/579449828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562513163d90 msgr2=0x5625131711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:48.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.959+0000 7f5cf8e5d640 1 --2- 172.21.15.16:0/579449828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562513163d90 0x5625131711b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.959+0000 7f5cf8e5d640 1 -- 172.21.15.16:0/579449828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625131dc1f0 con 0x56251316c970 2024-03-28T22:15:48.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.959+0000 7f5cf37fe640 1 --2- 172.21.15.16:0/579449828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562513163d90 0x5625131711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:15:48.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.959+0000 7f5cf8e5d640 1 --2- 172.21.15.16:0/579449828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56251316c970 0x5625131dbc30 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f5cec02c500 tx=0x7f5cec033cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e16fd79d65c09f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:48.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.959+0000 7f5cd37fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:15:48.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.960+0000 7f5cd37fe640 1 -- 172.21.15.16:0/579449828 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cec0338b0 con 0x56251316c970 2024-03-28T22:15:48.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.960+0000 7f5cd37fe640 1 -- 172.21.15.16:0/579449828 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f5cec004490 con 0x56251316c970 2024-03-28T22:15:48.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.960+0000 7f5cfd57d740 1 -- 172.21.15.16:0/579449828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5cdc009d20 con 0x56251316c970 2024-03-28T22:15:48.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.960+0000 7f5cd37fe640 1 -- 172.21.15.16:0/579449828 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5cec004690 con 0x56251316c970 2024-03-28T22:15:48.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.960+0000 7f5cfd57d740 10 client.16626 fetch_fsmap learned FSMap version 27 2024-03-28T22:15:48.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.960+0000 7f5cfd57d740 1 -- 172.21.15.16:0/579449828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5625131e3580 con 0x56251316c970 2024-03-28T22:15:48.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.960+0000 7f5cd37fe640 1 -- 172.21.15.16:0/579449828 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5cdc009d20 con 0x56251316c970 2024-03-28T22:15:48.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.960+0000 7f5cd37fe640 1 -- 172.21.15.16:0/579449828 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f5cec040040 con 0x56251316c970 2024-03-28T22:15:48.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.960+0000 7f5cfd57d740 10 client.16626 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:15:48.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.960+0000 7f5cfd57d740 10 client.16626 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:15:48.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.960+0000 7f5cfd57d740 20 client.16626 populate_metadata read hostname 'smithi016' 2024-03-28T22:15:48.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.960+0000 7f5cfd57d740 1 --2- 172.21.15.16:0/579449828 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5625131e3d20 0x5625131e6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:48.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.960+0000 7f5cfd57d740 4 client.16626 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:15:48.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.960+0000 7f5cfd57d740 1 -- 172.21.15.16:0/579449828 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5625130cea40 con 0x5625131e3d20 2024-03-28T22:15:48.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.960+0000 7f5cf3fff640 1 --2- 172.21.15.16:0/579449828 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5625131e3d20 0x5625131e6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:48.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.961+0000 7f5cd37fe640 10 client.16626 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:15:48.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.961+0000 7f5cf3fff640 1 --2- 172.21.15.16:0/579449828 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5625131e3d20 0x5625131e6140 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dda012e779f5613d server_cookie=26e73cbc92ecaf31 in_seq=0 out_seq=0 2024-03-28T22:15:48.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.961+0000 7f5cd37fe640 1 -- 172.21.15.16:0/579449828 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5625130cea40 con 0x5625131e3d20 2024-03-28T22:15:48.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.961+0000 7f5cd37fe640 10 client.16626 handle_command_reply: tid=0 2024-03-28T22:15:48.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.961+0000 7f5cfd57d740 1 client.16626 shutdown 2024-03-28T22:15:48.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.961+0000 7f5cfd57d740 1 -- 172.21.15.16:0/579449828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56251316c970 msgr2=0x5625131dbc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.961+0000 7f5cfd57d740 1 --2- 172.21.15.16:0/579449828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56251316c970 0x5625131dbc30 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f5cec02c500 tx=0x7f5cec033cb0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.961+0000 7f5cfd57d740 1 -- 172.21.15.16:0/579449828 shutdown_connections 2024-03-28T22:15:48.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.961+0000 7f5cfd57d740 1 --2- 172.21.15.16:0/579449828 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5625131e3d20 0x5625131e6140 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.961+0000 7f5cfd57d740 1 --2- 172.21.15.16:0/579449828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56251316c970 0x5625131dbc30 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.961+0000 7f5cfd57d740 1 --2- 172.21.15.16:0/579449828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562513163d90 0x5625131711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.961+0000 7f5cd37fe640 0 client.16626 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:15:48.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.961+0000 7f5cfd57d740 1 --2- 172.21.15.16:0/579449828 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562513163470 0x562513170c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.961+0000 7f5cfd57d740 1 -- 172.21.15.16:0/579449828 >> 172.21.15.16:0/579449828 conn(0x56251315dba0 msgr2=0x562513169910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:48.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.961+0000 7f5cfd57d740 1 -- 172.21.15.16:0/579449828 shutdown_connections 2024-03-28T22:15:48.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.961+0000 7f5cfd57d740 1 -- 172.21.15.16:0/579449828 wait complete. 2024-03-28T22:15:48.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.961+0000 7f5cfd57d740 20 client.16626 trim_cache size 0 max 16384 2024-03-28T22:15:48.977 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:15:48.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.962+0000 7f5cfa8c0640 1 -- 172.21.15.16:0/1470582806 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f5ce00a1ef0 msgr2=0x7f5ce00a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.962+0000 7f5cfa8c0640 1 --2- 172.21.15.16:0/1470582806 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f5ce00a1ef0 0x7f5ce00a43b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5cec006010 tx=0x7f5cec02bef0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.962+0000 7f5cfa8c0640 1 -- 172.21.15.16:0/1470582806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cf4102980 msgr2=0x7f5cf4198100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:48.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.962+0000 7f5cfa8c0640 1 --2- 172.21.15.16:0/1470582806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cf4102980 0x7f5cf4198100 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f5ce400c890 tx=0x7f5ce400cd50 comp rx=0 tx=0).stop 2024-03-28T22:15:48.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.962+0000 7f5cfa8c0640 1 -- 172.21.15.16:0/1470582806 shutdown_connections 2024-03-28T22:15:48.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.962+0000 7f5cfa8c0640 1 --2- 172.21.15.16:0/1470582806 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f5ce00a1ef0 0x7f5ce00a43b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.962+0000 7f5cfa8c0640 1 --2- 172.21.15.16:0/1470582806 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5cf419c9c0 0x7f5cf419ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.962+0000 7f5cfa8c0640 1 --2- 172.21.15.16:0/1470582806 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cf4107e90 0x7f5cf4198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.962+0000 7f5cfa8c0640 1 --2- 172.21.15.16:0/1470582806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5cf4102980 0x7f5cf4198100 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:48.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.962+0000 7f5cfa8c0640 1 -- 172.21.15.16:0/1470582806 >> 172.21.15.16:0/1470582806 conn(0x7f5cf4077830 msgr2=0x7f5cf40ff660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:48.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.962+0000 7f5cfa8c0640 1 -- 172.21.15.16:0/1470582806 shutdown_connections 2024-03-28T22:15:48.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:48.962+0000 7f5cfa8c0640 1 -- 172.21.15.16:0/1470582806 wait complete. 2024-03-28T22:15:49.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:49 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/550643783' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:15:49.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:49 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/550643783' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:15:49.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:49 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/550643783' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:15:49.981 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:15:49.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:15:49.982 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:15:50.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.097+0000 7f236f3e2640 1 -- 172.21.15.16:0/4275015443 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23681020a0 msgr2=0x7f236810e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:50.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.097+0000 7f236f3e2640 1 --2- 172.21.15.16:0/4275015443 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23681020a0 0x7f236810e5b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2360009be0 tx=0x7f236002bef0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.097+0000 7f236f3e2640 1 -- 172.21.15.16:0/4275015443 shutdown_connections 2024-03-28T22:15:50.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.097+0000 7f236f3e2640 1 --2- 172.21.15.16:0/4275015443 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23681020a0 0x7f236810e5b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.097+0000 7f236f3e2640 1 --2- 172.21.15.16:0/4275015443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23681016e0 0x7f2368101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.097+0000 7f236f3e2640 1 --2- 172.21.15.16:0/4275015443 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f23681076f0 0x7f2368107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.097+0000 7f236f3e2640 1 -- 172.21.15.16:0/4275015443 >> 172.21.15.16:0/4275015443 conn(0x7f23680fd1f0 msgr2=0x7f23680ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:50.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.097+0000 7f236f3e2640 1 -- 172.21.15.16:0/4275015443 shutdown_connections 2024-03-28T22:15:50.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.097+0000 7f236f3e2640 1 -- 172.21.15.16:0/4275015443 wait complete. 2024-03-28T22:15:50.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.098+0000 7f236f3e2640 1 Processor -- start 2024-03-28T22:15:50.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.098+0000 7f236f3e2640 1 -- start start 2024-03-28T22:15:50.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.098+0000 7f236f3e2640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f23681016e0 0x7f2368073a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:50.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.098+0000 7f236f3e2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23681020a0 0x7f2368073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:50.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.098+0000 7f236f3e2640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23681076f0 0x7f23680744d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:50.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.098+0000 7f236f3e2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2368111b00 con 0x7f23681020a0 2024-03-28T22:15:50.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.098+0000 7f236f3e2640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f2368111980 con 0x7f23681076f0 2024-03-28T22:15:50.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.098+0000 7f236f3e2640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2368111c80 con 0x7f23681016e0 2024-03-28T22:15:50.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.098+0000 7f236c97d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23681020a0 0x7f2368073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:50.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.098+0000 7f236c97d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23681020a0 0x7f2368073f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57124/0 (socket says 172.21.15.16:57124) 2024-03-28T22:15:50.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.098+0000 7f236c97d640 1 -- 172.21.15.16:0/1852586634 learned_addr learned my addr 172.21.15.16:0/1852586634 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:50.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.098+0000 7f236d97f640 1 --2- 172.21.15.16:0/1852586634 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23681076f0 0x7f23680744d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:50.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.098+0000 7f236d17e640 1 --2- 172.21.15.16:0/1852586634 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f23681016e0 0x7f2368073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:50.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.099+0000 7f236c97d640 1 -- 172.21.15.16:0/1852586634 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f23681016e0 msgr2=0x7f2368073a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:50.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.099+0000 7f236c97d640 1 --2- 172.21.15.16:0/1852586634 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f23681016e0 0x7f2368073a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.099+0000 7f236c97d640 1 -- 172.21.15.16:0/1852586634 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23681076f0 msgr2=0x7f23680744d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:50.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.099+0000 7f236c97d640 1 --2- 172.21.15.16:0/1852586634 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23681076f0 0x7f23680744d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.099+0000 7f236c97d640 1 -- 172.21.15.16:0/1852586634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23681a64a0 con 0x7f23681020a0 2024-03-28T22:15:50.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.099+0000 7f236d97f640 1 --2- 172.21.15.16:0/1852586634 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23681076f0 0x7f23680744d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:15:50.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.099+0000 7f236c97d640 1 --2- 172.21.15.16:0/1852586634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23681020a0 0x7f2368073f90 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f235800e9c0 tx=0x7f235800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:50.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.099+0000 7f235e7fc640 1 -- 172.21.15.16:0/1852586634 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f235800cde0 con 0x7f23681020a0 2024-03-28T22:15:50.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.099+0000 7f235e7fc640 1 -- 172.21.15.16:0/1852586634 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23580035f0 con 0x7f23681020a0 2024-03-28T22:15:50.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.099+0000 7f236f3e2640 1 -- 172.21.15.16:0/1852586634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23681a6790 con 0x7f23681020a0 2024-03-28T22:15:50.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.099+0000 7f235e7fc640 1 -- 172.21.15.16:0/1852586634 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2358010690 con 0x7f23681020a0 2024-03-28T22:15:50.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.099+0000 7f236f3e2640 1 -- 172.21.15.16:0/1852586634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23681a6ce0 con 0x7f23681020a0 2024-03-28T22:15:50.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.101+0000 7f235e7fc640 1 -- 172.21.15.16:0/1852586634 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f23580037a0 con 0x7f23681020a0 2024-03-28T22:15:50.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.102+0000 7f2371fc0740 1 Processor -- start 2024-03-28T22:15:50.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.102+0000 7f2371fc0740 1 -- start start 2024-03-28T22:15:50.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.102+0000 7f2371fc0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e296de99f0 0x55e296de9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:50.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.102+0000 7f2371fc0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e296dea310 0x55e296df29b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:50.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.102+0000 7f2371fc0740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e296df2ef0 0x55e296df52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:50.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.102+0000 7f2371fc0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e296cea0b0 con 0x55e296dea310 2024-03-28T22:15:50.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.102+0000 7f235e7fc640 1 --2- 172.21.15.16:0/1852586634 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f23540a1ef0 0x7f23540a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:50.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.102+0000 7f2371fc0740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55e296d4cea0 con 0x55e296df2ef0 2024-03-28T22:15:50.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.102+0000 7f2371fc0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e296cf5c00 con 0x55e296de99f0 2024-03-28T22:15:50.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.102+0000 7f236d17e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e296dea310 0x55e296df29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:50.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.102+0000 7f236d17e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e296dea310 0x55e296df29b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57126/0 (socket says 172.21.15.16:57126) 2024-03-28T22:15:50.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.102+0000 7f236d17e640 1 -- 172.21.15.16:0/3993103190 learned_addr learned my addr 172.21.15.16:0/3993103190 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:50.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.102+0000 7f236d97f640 1 --2- 172.21.15.16:0/1852586634 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f23540a1ef0 0x7f23540a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:50.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.102+0000 7f236c97d640 1 --2- 172.21.15.16:0/3993103190 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e296df2ef0 0x55e296df52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:50.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.102+0000 7f236d97f640 1 --2- 172.21.15.16:0/3993103190 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e296de99f0 0x55e296de9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:50.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.102+0000 7f235e7fc640 1 -- 172.21.15.16:0/1852586634 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f23580ce500 con 0x7f23681020a0 2024-03-28T22:15:50.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.102+0000 7f236d17e640 1 -- 172.21.15.16:0/3993103190 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e296de99f0 msgr2=0x55e296de9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:50.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.102+0000 7f236d17e640 1 --2- 172.21.15.16:0/3993103190 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e296de99f0 0x55e296de9dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.102+0000 7f236d17e640 1 -- 172.21.15.16:0/3993103190 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e296df2ef0 msgr2=0x55e296df52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:50.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.102+0000 7f236d17e640 1 --2- 172.21.15.16:0/3993103190 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e296df2ef0 0x55e296df52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.103+0000 7f236d17e640 1 -- 172.21.15.16:0/3993103190 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e296b297e0 con 0x55e296dea310 2024-03-28T22:15:50.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.103+0000 7f236d97f640 1 --2- 172.21.15.16:0/1852586634 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f23540a1ef0 0x7f23540a43b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2360006010 tx=0x7f2360033f30 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:50.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.103+0000 7f236d17e640 1 --2- 172.21.15.16:0/3993103190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e296dea310 0x55e296df29b0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f236400b710 tx=0x7f2364004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d823089067978b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:50.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.103+0000 7f23537fe640 1 -- 172.21.15.16:0/3993103190 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2364003bd0 con 0x55e296dea310 2024-03-28T22:15:50.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.103+0000 7f23537fe640 1 -- 172.21.15.16:0/3993103190 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2364015070 con 0x55e296dea310 2024-03-28T22:15:50.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.103+0000 7f23537fe640 1 -- 172.21.15.16:0/3993103190 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2364004e40 con 0x55e296dea310 2024-03-28T22:15:50.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.104+0000 7f2371fc0740 1 -- 172.21.15.16:0/3993103190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e296dea310 msgr2=0x55e296df29b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:50.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.104+0000 7f2371fc0740 1 --2- 172.21.15.16:0/3993103190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e296dea310 0x55e296df29b0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f236400b710 tx=0x7f2364004610 comp rx=0 tx=0).stop 2024-03-28T22:15:50.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.104+0000 7f2371fc0740 1 -- 172.21.15.16:0/3993103190 shutdown_connections 2024-03-28T22:15:50.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.104+0000 7f2371fc0740 1 --2- 172.21.15.16:0/3993103190 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e296df2ef0 0x55e296df52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.104+0000 7f2371fc0740 1 --2- 172.21.15.16:0/3993103190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e296dea310 0x55e296df29b0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.104+0000 7f2371fc0740 1 --2- 172.21.15.16:0/3993103190 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e296de99f0 0x55e296de9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.104+0000 7f2371fc0740 1 -- 172.21.15.16:0/3993103190 >> 172.21.15.16:0/3993103190 conn(0x55e296de4120 msgr2=0x55e296de4500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:50.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.104+0000 7f2371fc0740 1 -- 172.21.15.16:0/3993103190 shutdown_connections 2024-03-28T22:15:50.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.104+0000 7f2371fc0740 1 -- 172.21.15.16:0/3993103190 wait complete. 2024-03-28T22:15:50.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.104+0000 7f2371fc0740 1 -- start start 2024-03-28T22:15:50.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.104+0000 7f2371fc0740 1 Processor -- start 2024-03-28T22:15:50.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.104+0000 7f2371fc0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e296de99f0 0x55e296defee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:50.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.105+0000 7f2371fc0740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e296dea310 0x55e296df0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:50.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.105+0000 7f2371fc0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e296df2ef0 0x55e296df0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:50.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.105+0000 7f2371fc0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e296ceb3d0 con 0x55e296de99f0 2024-03-28T22:15:50.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.105+0000 7f2371fc0740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55e296d48f40 con 0x55e296dea310 2024-03-28T22:15:50.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.105+0000 7f2371fc0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e296df5ce0 con 0x55e296df2ef0 2024-03-28T22:15:50.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.105+0000 7f2371fc0740 10 client.0 _ll_register_callbacks cb 0x55e296d08b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:15:50.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.105+0000 7f236d17e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e296de99f0 0x55e296defee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:50.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.105+0000 7f236d17e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e296de99f0 0x55e296defee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57136/0 (socket says 172.21.15.16:57136) 2024-03-28T22:15:50.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.105+0000 7f236d17e640 1 -- 172.21.15.16:0/1373785122 learned_addr learned my addr 172.21.15.16:0/1373785122 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:50.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.105+0000 7f236c97d640 1 --2- 172.21.15.16:0/1373785122 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e296dea310 0x55e296df0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:50.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.105+0000 7f236d97f640 1 --2- 172.21.15.16:0/1373785122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e296df2ef0 0x55e296df0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:50.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.105+0000 7f236d17e640 1 -- 172.21.15.16:0/1373785122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e296df2ef0 msgr2=0x55e296df0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:50.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.105+0000 7f236d17e640 1 --2- 172.21.15.16:0/1373785122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e296df2ef0 0x55e296df0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.105+0000 7f236d17e640 1 -- 172.21.15.16:0/1373785122 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e296dea310 msgr2=0x55e296df0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:50.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.105+0000 7f236d17e640 1 --2- 172.21.15.16:0/1373785122 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e296dea310 0x55e296df0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.105+0000 7f236d17e640 1 -- 172.21.15.16:0/1373785122 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e296b89f70 con 0x55e296de99f0 2024-03-28T22:15:50.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.105+0000 7f236c97d640 1 --2- 172.21.15.16:0/1373785122 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e296dea310 0x55e296df0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:15:50.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.105+0000 7f236d17e640 1 --2- 172.21.15.16:0/1373785122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e296de99f0 0x55e296defee0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f23640039c0 tx=0x7f236400bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1346ebcadd295f1b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:50.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.105+0000 7f2350ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:15:50.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.105+0000 7f2350ff9640 1 -- 172.21.15.16:0/1373785122 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2364028060 con 0x55e296de99f0 2024-03-28T22:15:50.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.105+0000 7f2350ff9640 1 -- 172.21.15.16:0/1373785122 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f2364011440 con 0x55e296de99f0 2024-03-28T22:15:50.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.105+0000 7f2371fc0740 1 -- 172.21.15.16:0/1373785122 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2364015070 con 0x55e296de99f0 2024-03-28T22:15:50.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.106+0000 7f2350ff9640 1 -- 172.21.15.16:0/1373785122 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2364011640 con 0x55e296de99f0 2024-03-28T22:15:50.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.106+0000 7f2371fc0740 10 client.16638 fetch_fsmap learned FSMap version 27 2024-03-28T22:15:50.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.106+0000 7f2371fc0740 1 -- 172.21.15.16:0/1373785122 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e296a21170 con 0x55e296de99f0 2024-03-28T22:15:50.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.106+0000 7f2350ff9640 1 -- 172.21.15.16:0/1373785122 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2364015070 con 0x55e296de99f0 2024-03-28T22:15:50.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.106+0000 7f2350ff9640 1 -- 172.21.15.16:0/1373785122 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f2364022070 con 0x55e296de99f0 2024-03-28T22:15:50.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.106+0000 7f2371fc0740 10 client.16638 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:15:50.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.106+0000 7f2371fc0740 10 client.16638 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:15:50.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.106+0000 7f2371fc0740 20 client.16638 populate_metadata read hostname 'smithi016' 2024-03-28T22:15:50.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.106+0000 7f2371fc0740 1 --2- 172.21.15.16:0/1373785122 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e296e02ce0 0x55e296e050d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:50.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.106+0000 7f2371fc0740 4 client.16638 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:15:50.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.106+0000 7f2371fc0740 1 -- 172.21.15.16:0/1373785122 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e296d5a3e0 con 0x55e296e02ce0 2024-03-28T22:15:50.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.106+0000 7f236c97d640 1 --2- 172.21.15.16:0/1373785122 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e296e02ce0 0x55e296e050d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:50.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.107+0000 7f2350ff9640 10 client.16638 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:15:50.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.107+0000 7f236c97d640 1 --2- 172.21.15.16:0/1373785122 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e296e02ce0 0x55e296e050d0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cce31601edeac53b server_cookie=efad8f4a18821160 in_seq=0 out_seq=0 2024-03-28T22:15:50.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.107+0000 7f2350ff9640 1 -- 172.21.15.16:0/1373785122 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e296d5a3e0 con 0x55e296e02ce0 2024-03-28T22:15:50.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.107+0000 7f2350ff9640 10 client.16638 handle_command_reply: tid=0 2024-03-28T22:15:50.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.107+0000 7f2371fc0740 1 client.16638 shutdown 2024-03-28T22:15:50.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.107+0000 7f2371fc0740 1 -- 172.21.15.16:0/1373785122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e296de99f0 msgr2=0x55e296defee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:50.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.107+0000 7f2371fc0740 1 --2- 172.21.15.16:0/1373785122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e296de99f0 0x55e296defee0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f23640039c0 tx=0x7f236400bdb0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.107+0000 7f2371fc0740 1 -- 172.21.15.16:0/1373785122 shutdown_connections 2024-03-28T22:15:50.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.107+0000 7f2371fc0740 1 --2- 172.21.15.16:0/1373785122 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e296e02ce0 0x55e296e050d0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.107+0000 7f2371fc0740 1 --2- 172.21.15.16:0/1373785122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e296df2ef0 0x55e296df0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.107+0000 7f2371fc0740 1 --2- 172.21.15.16:0/1373785122 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e296dea310 0x55e296df0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.107+0000 7f2350ff9640 0 client.16638 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:15:50.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.107+0000 7f2371fc0740 1 --2- 172.21.15.16:0/1373785122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e296de99f0 0x55e296defee0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.107+0000 7f2371fc0740 1 -- 172.21.15.16:0/1373785122 >> 172.21.15.16:0/1373785122 conn(0x55e296de4120 msgr2=0x55e296de5e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:50.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.108+0000 7f2371fc0740 1 -- 172.21.15.16:0/1373785122 shutdown_connections 2024-03-28T22:15:50.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.108+0000 7f2371fc0740 1 -- 172.21.15.16:0/1373785122 wait complete. 2024-03-28T22:15:50.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.108+0000 7f2371fc0740 20 client.16638 trim_cache size 0 max 16384 2024-03-28T22:15:50.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.125+0000 7f2371fc0740 1 Processor -- start 2024-03-28T22:15:50.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.125+0000 7f2371fc0740 1 -- start start 2024-03-28T22:15:50.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.125+0000 7f2371fc0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e296de99f0 0x55e296df2610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:50.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.125+0000 7f2371fc0740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e296dea310 0x55e296dfe930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:50.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.125+0000 7f2371fc0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e296df2ef0 0x55e296de3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:50.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.125+0000 7f2371fc0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f23681b1e40 con 0x55e296df2ef0 2024-03-28T22:15:50.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.125+0000 7f2371fc0740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f23681af710 con 0x55e296dea310 2024-03-28T22:15:50.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.125+0000 7f2371fc0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f23681ad580 con 0x55e296de99f0 2024-03-28T22:15:50.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.126+0000 7f236d97f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e296df2ef0 0x55e296de3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:50.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.126+0000 7f236d97f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e296df2ef0 0x55e296de3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57146/0 (socket says 172.21.15.16:57146) 2024-03-28T22:15:50.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.126+0000 7f236d97f640 1 -- 172.21.15.16:0/1006605189 learned_addr learned my addr 172.21.15.16:0/1006605189 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:50.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.126+0000 7f236c97d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e296dea310 0x55e296dfe930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:50.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.126+0000 7f236d17e640 1 --2- 172.21.15.16:0/1006605189 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e296de99f0 0x55e296df2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:50.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.126+0000 7f236d97f640 1 -- 172.21.15.16:0/1006605189 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e296de99f0 msgr2=0x55e296df2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:50.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.126+0000 7f236d97f640 1 --2- 172.21.15.16:0/1006605189 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e296de99f0 0x55e296df2610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.126+0000 7f236d97f640 1 -- 172.21.15.16:0/1006605189 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e296dea310 msgr2=0x55e296dfe930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:50.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.126+0000 7f236d97f640 1 --2- 172.21.15.16:0/1006605189 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e296dea310 0x55e296dfe930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.126+0000 7f236d97f640 1 -- 172.21.15.16:0/1006605189 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e296deacc0 con 0x55e296df2ef0 2024-03-28T22:15:50.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.126+0000 7f236c97d640 1 --2- 172.21.15.16:0/1006605189 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e296dea310 0x55e296dfe930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:15:50.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.126+0000 7f236d17e640 1 --2- 172.21.15.16:0/1006605189 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e296de99f0 0x55e296df2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:15:50.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.126+0000 7f236d97f640 1 --2- 172.21.15.16:0/1006605189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e296df2ef0 0x55e296de3000 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f2360036040 tx=0x7f2360009280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af64fe906178485f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:50.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.126+0000 7f2352ffd640 1 -- 172.21.15.16:0/1006605189 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f236002da80 con 0x55e296df2ef0 2024-03-28T22:15:50.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.126+0000 7f2352ffd640 1 -- 172.21.15.16:0/1006605189 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2360004520 con 0x55e296df2ef0 2024-03-28T22:15:50.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.126+0000 7f2352ffd640 1 -- 172.21.15.16:0/1006605189 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2360034950 con 0x55e296df2ef0 2024-03-28T22:15:50.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.127+0000 7f2371fc0740 1 -- 172.21.15.16:0/1006605189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e296df2ef0 msgr2=0x55e296de3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:50.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.127+0000 7f2371fc0740 1 --2- 172.21.15.16:0/1006605189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e296df2ef0 0x55e296de3000 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f2360036040 tx=0x7f2360009280 comp rx=0 tx=0).stop 2024-03-28T22:15:50.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.127+0000 7f2371fc0740 1 -- 172.21.15.16:0/1006605189 shutdown_connections 2024-03-28T22:15:50.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.127+0000 7f2371fc0740 1 --2- 172.21.15.16:0/1006605189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e296df2ef0 0x55e296de3000 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.127+0000 7f2371fc0740 1 --2- 172.21.15.16:0/1006605189 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e296dea310 0x55e296dfe930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.127+0000 7f2371fc0740 1 --2- 172.21.15.16:0/1006605189 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e296de99f0 0x55e296df2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.127+0000 7f2371fc0740 1 -- 172.21.15.16:0/1006605189 >> 172.21.15.16:0/1006605189 conn(0x55e296de4120 msgr2=0x55e296de28f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:50.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.127+0000 7f2371fc0740 1 -- 172.21.15.16:0/1006605189 shutdown_connections 2024-03-28T22:15:50.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.127+0000 7f2371fc0740 1 -- 172.21.15.16:0/1006605189 wait complete. 2024-03-28T22:15:50.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.127+0000 7f2371fc0740 1 -- start start 2024-03-28T22:15:50.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.127+0000 7f2371fc0740 1 Processor -- start 2024-03-28T22:15:50.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.128+0000 7f2371fc0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e296de99f0 0x55e296dfb230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:50.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.128+0000 7f2371fc0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e296dea310 0x55e296df7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:50.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.128+0000 7f2371fc0740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e296df2ef0 0x55e296df8470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:50.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.128+0000 7f2371fc0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f23681adbd0 con 0x55e296de99f0 2024-03-28T22:15:50.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.128+0000 7f2371fc0740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f23681af040 con 0x55e296df2ef0 2024-03-28T22:15:50.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.128+0000 7f2371fc0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f23681ae2a0 con 0x55e296dea310 2024-03-28T22:15:50.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.128+0000 7f2371fc0740 10 client.0 _ll_register_callbacks cb 0x7f2364004510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:15:50.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.128+0000 7f236d17e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e296de99f0 0x55e296dfb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:50.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.128+0000 7f236d17e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e296de99f0 0x55e296dfb230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57148/0 (socket says 172.21.15.16:57148) 2024-03-28T22:15:50.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.128+0000 7f236d17e640 1 -- 172.21.15.16:0/1816510842 learned_addr learned my addr 172.21.15.16:0/1816510842 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:50.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.128+0000 7f236d97f640 1 --2- 172.21.15.16:0/1816510842 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e296df2ef0 0x55e296df8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:50.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.128+0000 7f236d17e640 1 -- 172.21.15.16:0/1816510842 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e296dea310 msgr2=0x55e296df7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:50.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.128+0000 7f236d17e640 1 --2- 172.21.15.16:0/1816510842 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e296dea310 0x55e296df7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.128+0000 7f236d17e640 1 -- 172.21.15.16:0/1816510842 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e296df2ef0 msgr2=0x55e296df8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:50.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.128+0000 7f236d17e640 1 --2- 172.21.15.16:0/1816510842 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e296df2ef0 0x55e296df8470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.128+0000 7f236d17e640 1 -- 172.21.15.16:0/1816510842 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e296df8ca0 con 0x55e296de99f0 2024-03-28T22:15:50.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.128+0000 7f236d97f640 1 --2- 172.21.15.16:0/1816510842 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e296df2ef0 0x55e296df8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:15:50.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.128+0000 7f236d17e640 1 --2- 172.21.15.16:0/1816510842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e296de99f0 0x55e296dfb230 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f236400b620 tx=0x7f23640048d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60e6bced9561d0cb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:50.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.128+0000 7f2350ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:15:50.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.128+0000 7f2350ff9640 1 -- 172.21.15.16:0/1816510842 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2364028070 con 0x55e296de99f0 2024-03-28T22:15:50.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.128+0000 7f2371fc0740 1 -- 172.21.15.16:0/1816510842 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2360004520 con 0x55e296de99f0 2024-03-28T22:15:50.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.128+0000 7f2350ff9640 1 -- 172.21.15.16:0/1816510842 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f2364003ff0 con 0x55e296de99f0 2024-03-28T22:15:50.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.129+0000 7f2350ff9640 1 -- 172.21.15.16:0/1816510842 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f236403a050 con 0x55e296de99f0 2024-03-28T22:15:50.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.129+0000 7f2371fc0740 10 client.16644 fetch_fsmap learned FSMap version 27 2024-03-28T22:15:50.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.129+0000 7f2371fc0740 1 -- 172.21.15.16:0/1816510842 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e296e69d20 con 0x55e296de99f0 2024-03-28T22:15:50.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.129+0000 7f2350ff9640 1 -- 172.21.15.16:0/1816510842 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2360004520 con 0x55e296de99f0 2024-03-28T22:15:50.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.129+0000 7f2350ff9640 1 -- 172.21.15.16:0/1816510842 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f2364022040 con 0x55e296de99f0 2024-03-28T22:15:50.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.129+0000 7f2371fc0740 10 client.16644 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:15:50.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.129+0000 7f2371fc0740 10 client.16644 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:15:50.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.129+0000 7f2371fc0740 20 client.16644 populate_metadata read hostname 'smithi016' 2024-03-28T22:15:50.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.129+0000 7f2371fc0740 1 --2- 172.21.15.16:0/1816510842 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e296e6a440 0x55e296e6c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:50.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.129+0000 7f2371fc0740 4 client.16644 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:15:50.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.129+0000 7f2371fc0740 1 -- 172.21.15.16:0/1816510842 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55e296d54730 con 0x55e296e6a440 2024-03-28T22:15:50.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.129+0000 7f236c97d640 1 --2- 172.21.15.16:0/1816510842 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e296e6a440 0x55e296e6c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:50.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.130+0000 7f2350ff9640 10 client.16644 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:15:50.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.130+0000 7f236c97d640 1 --2- 172.21.15.16:0/1816510842 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e296e6a440 0x55e296e6c860 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a731d11cd26ac06d server_cookie=7d567da77d4ed1ba in_seq=0 out_seq=0 2024-03-28T22:15:50.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.131+0000 7f2350ff9640 1 -- 172.21.15.16:0/1816510842 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55e296d54730 con 0x55e296e6a440 2024-03-28T22:15:50.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.131+0000 7f2350ff9640 10 client.16644 handle_command_reply: tid=0 2024-03-28T22:15:50.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.131+0000 7f2371fc0740 1 client.16644 shutdown 2024-03-28T22:15:50.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.131+0000 7f2371fc0740 1 -- 172.21.15.16:0/1816510842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e296de99f0 msgr2=0x55e296dfb230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:50.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.131+0000 7f2371fc0740 1 --2- 172.21.15.16:0/1816510842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e296de99f0 0x55e296dfb230 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f236400b620 tx=0x7f23640048d0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.131+0000 7f2371fc0740 1 -- 172.21.15.16:0/1816510842 shutdown_connections 2024-03-28T22:15:50.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.131+0000 7f2371fc0740 1 --2- 172.21.15.16:0/1816510842 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e296e6a440 0x55e296e6c860 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.131+0000 7f2371fc0740 1 --2- 172.21.15.16:0/1816510842 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e296df2ef0 0x55e296df8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.131+0000 7f2371fc0740 1 --2- 172.21.15.16:0/1816510842 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e296dea310 0x55e296df7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.131+0000 7f2371fc0740 1 --2- 172.21.15.16:0/1816510842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e296de99f0 0x55e296dfb230 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.131+0000 7f2350ff9640 0 client.16644 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:15:50.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.131+0000 7f2371fc0740 1 -- 172.21.15.16:0/1816510842 >> 172.21.15.16:0/1816510842 conn(0x55e296de4120 msgr2=0x55e296df0710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:50.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.131+0000 7f2371fc0740 1 -- 172.21.15.16:0/1816510842 shutdown_connections 2024-03-28T22:15:50.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.131+0000 7f2371fc0740 1 -- 172.21.15.16:0/1816510842 wait complete. 2024-03-28T22:15:50.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.131+0000 7f2371fc0740 20 client.16644 trim_cache size 0 max 16384 2024-03-28T22:15:50.145 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:15:50.145 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-03-28T22:15:50.145 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "a71f7fc3-f767-448e-bc88-298b94956a81", 2024-03-28T22:15:50.145 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-03-28T22:15:50.145 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:15:50.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.132+0000 7f236f3e2640 1 -- 172.21.15.16:0/1852586634 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f23540a1ef0 msgr2=0x7f23540a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:50.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.132+0000 7f236f3e2640 1 --2- 172.21.15.16:0/1852586634 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f23540a1ef0 0x7f23540a43b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2360006010 tx=0x7f2360033f30 comp rx=0 tx=0).stop 2024-03-28T22:15:50.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.132+0000 7f236f3e2640 1 -- 172.21.15.16:0/1852586634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23681020a0 msgr2=0x7f2368073f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:50.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.132+0000 7f236f3e2640 1 --2- 172.21.15.16:0/1852586634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23681020a0 0x7f2368073f90 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f235800e9c0 tx=0x7f235800ee80 comp rx=0 tx=0).stop 2024-03-28T22:15:50.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.132+0000 7f236f3e2640 1 -- 172.21.15.16:0/1852586634 shutdown_connections 2024-03-28T22:15:50.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.132+0000 7f236f3e2640 1 --2- 172.21.15.16:0/1852586634 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f23540a1ef0 0x7f23540a43b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.132+0000 7f236f3e2640 1 --2- 172.21.15.16:0/1852586634 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23681076f0 0x7f23680744d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.132+0000 7f236f3e2640 1 --2- 172.21.15.16:0/1852586634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23681020a0 0x7f2368073f90 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.132+0000 7f236f3e2640 1 --2- 172.21.15.16:0/1852586634 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f23681016e0 0x7f2368073a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.132+0000 7f236f3e2640 1 -- 172.21.15.16:0/1852586634 >> 172.21.15.16:0/1852586634 conn(0x7f23680fd1f0 msgr2=0x7f23680fd910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:50.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.132+0000 7f236f3e2640 1 -- 172.21.15.16:0/1852586634 shutdown_connections 2024-03-28T22:15:50.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.132+0000 7f236f3e2640 1 -- 172.21.15.16:0/1852586634 wait complete. 2024-03-28T22:15:50.148 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:15:50.149 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:15:50.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.252+0000 7f19175dd640 1 -- 172.21.15.16:0/942528962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f191010bd40 msgr2=0x7f191010e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:50.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.252+0000 7f19175dd640 1 --2- 172.21.15.16:0/942528962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f191010bd40 0x7f191010e170 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f1900009ca0 tx=0x7f190002c180 comp rx=0 tx=0).stop 2024-03-28T22:15:50.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.253+0000 7f19175dd640 1 -- 172.21.15.16:0/942528962 shutdown_connections 2024-03-28T22:15:50.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.253+0000 7f19175dd640 1 --2- 172.21.15.16:0/942528962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f191010bd40 0x7f191010e170 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.253+0000 7f19175dd640 1 --2- 172.21.15.16:0/942528962 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1910103290 0x7f191010b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.253+0000 7f19175dd640 1 --2- 172.21.15.16:0/942528962 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1910102970 0x7f1910102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.253+0000 7f19175dd640 1 -- 172.21.15.16:0/942528962 >> 172.21.15.16:0/942528962 conn(0x7f1910077800 msgr2=0x7f1910077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:50.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.253+0000 7f19175dd640 1 -- 172.21.15.16:0/942528962 shutdown_connections 2024-03-28T22:15:50.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.253+0000 7f19175dd640 1 -- 172.21.15.16:0/942528962 wait complete. 2024-03-28T22:15:50.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.253+0000 7f19175dd640 1 Processor -- start 2024-03-28T22:15:50.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.253+0000 7f19175dd640 1 -- start start 2024-03-28T22:15:50.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.253+0000 7f19175dd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1910102970 0x7f1910197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:50.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.253+0000 7f19175dd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1910103290 0x7f19101984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:50.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.253+0000 7f19175dd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f191010bd40 0x7f191019c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:50.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.253+0000 7f19175dd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1910111ad0 con 0x7f191010bd40 2024-03-28T22:15:50.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.253+0000 7f19175dd640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f1910111950 con 0x7f1910103290 2024-03-28T22:15:50.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.253+0000 7f19175dd640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1910111c50 con 0x7f1910102970 2024-03-28T22:15:50.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.254+0000 7f1915b7a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f191010bd40 0x7f191019c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:50.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.254+0000 7f1915b7a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f191010bd40 0x7f191019c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57172/0 (socket says 172.21.15.16:57172) 2024-03-28T22:15:50.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.254+0000 7f1915b7a640 1 -- 172.21.15.16:0/1596319661 learned_addr learned my addr 172.21.15.16:0/1596319661 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:50.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.254+0000 7f1914b78640 1 --2- 172.21.15.16:0/1596319661 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1910103290 0x7f19101984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:50.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.254+0000 7f1915379640 1 --2- 172.21.15.16:0/1596319661 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1910102970 0x7f1910197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:50.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.254+0000 7f1915b7a640 1 -- 172.21.15.16:0/1596319661 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1910102970 msgr2=0x7f1910197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:50.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.254+0000 7f1915b7a640 1 --2- 172.21.15.16:0/1596319661 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1910102970 0x7f1910197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.254+0000 7f1915b7a640 1 -- 172.21.15.16:0/1596319661 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1910103290 msgr2=0x7f19101984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:50.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.254+0000 7f1915b7a640 1 --2- 172.21.15.16:0/1596319661 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1910103290 0x7f19101984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.254+0000 7f1915b7a640 1 -- 172.21.15.16:0/1596319661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f191019cf60 con 0x7f191010bd40 2024-03-28T22:15:50.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.254+0000 7f1915b7a640 1 --2- 172.21.15.16:0/1596319661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f191010bd40 0x7f191019c830 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f190002c790 tx=0x7f1900035ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:50.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.254+0000 7f190e7fc640 1 -- 172.21.15.16:0/1596319661 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1900039040 con 0x7f191010bd40 2024-03-28T22:15:50.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.254+0000 7f190e7fc640 1 -- 172.21.15.16:0/1596319661 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1900044070 con 0x7f191010bd40 2024-03-28T22:15:50.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.254+0000 7f19175dd640 1 -- 172.21.15.16:0/1596319661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f191019d1f0 con 0x7f191010bd40 2024-03-28T22:15:50.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.254+0000 7f190e7fc640 1 -- 172.21.15.16:0/1596319661 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19000329f0 con 0x7f191010bd40 2024-03-28T22:15:50.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.254+0000 7f19175dd640 1 -- 172.21.15.16:0/1596319661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19101a4ae0 con 0x7f191010bd40 2024-03-28T22:15:50.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.256+0000 7f190e7fc640 1 -- 172.21.15.16:0/1596319661 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f1900032b90 con 0x7f191010bd40 2024-03-28T22:15:50.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.257+0000 7f191a1b6740 1 Processor -- start 2024-03-28T22:15:50.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.257+0000 7f191a1b6740 1 -- start start 2024-03-28T22:15:50.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.257+0000 7f191a1b6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3dfea2290 0x55d3dfea2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:50.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.257+0000 7f191a1b6740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3dfea2bb0 0x55d3dfeab250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:50.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.257+0000 7f191a1b6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3dfeab790 0x55d3dfeadb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:50.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.257+0000 7f191a1b6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d3dfe06180 con 0x55d3dfeab790 2024-03-28T22:15:50.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.257+0000 7f191a1b6740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55d3dfdaeb70 con 0x55d3dfea2bb0 2024-03-28T22:15:50.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.257+0000 7f191a1b6740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d3dfe02220 con 0x55d3dfea2290 2024-03-28T22:15:50.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.257+0000 7f190e7fc640 1 --2- 172.21.15.16:0/1596319661 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f18f80a1ef0 0x7f18f80a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:50.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.257+0000 7f1915379640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3dfea2bb0 0x55d3dfeab250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:50.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.257+0000 7f1915b7a640 1 --2- 172.21.15.16:0/1596319661 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f18f80a1ef0 0x7f18f80a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:50.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.257+0000 7f1915379640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3dfea2bb0 0x55d3dfeab250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:49050/0 (socket says 172.21.15.16:49050) 2024-03-28T22:15:50.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.257+0000 7f1914b78640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3dfeab790 0x55d3dfeadb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:50.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.257+0000 7f1915379640 1 -- 172.21.15.16:0/1788502878 learned_addr learned my addr 172.21.15.16:0/1788502878 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:50.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.257+0000 7f1914b78640 1 --2- 172.21.15.16:0/1788502878 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3dfea2290 0x55d3dfea2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:50.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.257+0000 7f190e7fc640 1 -- 172.21.15.16:0/1596319661 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f19000f0700 con 0x7f191010bd40 2024-03-28T22:15:50.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.258+0000 7f1915379640 1 -- 172.21.15.16:0/1788502878 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3dfea2290 msgr2=0x55d3dfea2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:50.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.258+0000 7f1915379640 1 --2- 172.21.15.16:0/1788502878 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3dfea2290 0x55d3dfea2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.258+0000 7f1915379640 1 -- 172.21.15.16:0/1788502878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3dfeab790 msgr2=0x55d3dfeadb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:50.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.258+0000 7f1915379640 1 --2- 172.21.15.16:0/1788502878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3dfeab790 0x55d3dfeadb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.258+0000 7f1915379640 1 -- 172.21.15.16:0/1788502878 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3dfdc27c0 con 0x55d3dfea2bb0 2024-03-28T22:15:50.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.258+0000 7f1914b78640 1 --2- 172.21.15.16:0/1788502878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3dfeab790 0x55d3dfeadb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:15:50.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.258+0000 7f1915b7a640 1 --2- 172.21.15.16:0/1596319661 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f18f80a1ef0 0x7f18f80a43b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f19000bd0b0 tx=0x7f190008f0e0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:50.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.258+0000 7f1915379640 1 --2- 172.21.15.16:0/1788502878 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3dfea2bb0 0x55d3dfeab250 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1908009b60 tx=0x7f190800e6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e2a7f5197713b4e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:50.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.258+0000 7f18f77fe640 1 -- 172.21.15.16:0/1788502878 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1908016070 con 0x55d3dfea2bb0 2024-03-28T22:15:50.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.258+0000 7f18f77fe640 1 -- 172.21.15.16:0/1788502878 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1908009e50 con 0x55d3dfea2bb0 2024-03-28T22:15:50.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.258+0000 7f18f77fe640 1 -- 172.21.15.16:0/1788502878 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1908004df0 con 0x55d3dfea2bb0 2024-03-28T22:15:50.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.259+0000 7f191a1b6740 1 -- 172.21.15.16:0/1788502878 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3dfea2bb0 msgr2=0x55d3dfeab250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:50.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.259+0000 7f191a1b6740 1 --2- 172.21.15.16:0/1788502878 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3dfea2bb0 0x55d3dfeab250 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1908009b60 tx=0x7f190800e6c0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.259+0000 7f191a1b6740 1 -- 172.21.15.16:0/1788502878 shutdown_connections 2024-03-28T22:15:50.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.259+0000 7f191a1b6740 1 --2- 172.21.15.16:0/1788502878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3dfeab790 0x55d3dfeadb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.259+0000 7f191a1b6740 1 --2- 172.21.15.16:0/1788502878 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3dfea2bb0 0x55d3dfeab250 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.259+0000 7f191a1b6740 1 --2- 172.21.15.16:0/1788502878 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3dfea2290 0x55d3dfea2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.259+0000 7f191a1b6740 1 -- 172.21.15.16:0/1788502878 >> 172.21.15.16:0/1788502878 conn(0x55d3dfe9c9c0 msgr2=0x55d3dfe9cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:50.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.259+0000 7f191a1b6740 1 -- 172.21.15.16:0/1788502878 shutdown_connections 2024-03-28T22:15:50.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.259+0000 7f191a1b6740 1 -- 172.21.15.16:0/1788502878 wait complete. 2024-03-28T22:15:50.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.259+0000 7f191a1b6740 1 -- start start 2024-03-28T22:15:50.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.259+0000 7f191a1b6740 1 Processor -- start 2024-03-28T22:15:50.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.260+0000 7f191a1b6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3dfea2290 0x55d3dfea87b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:50.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.260+0000 7f191a1b6740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3dfea2bb0 0x55d3dfea8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:50.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.260+0000 7f191a1b6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3dfeab790 0x55d3dfea9230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:50.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.260+0000 7f191a1b6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d3dfeae580 con 0x55d3dfeab790 2024-03-28T22:15:50.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.260+0000 7f191a1b6740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55d3dfda44a0 con 0x55d3dfea2bb0 2024-03-28T22:15:50.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.260+0000 7f191a1b6740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d3dfeae700 con 0x55d3dfea2290 2024-03-28T22:15:50.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.260+0000 7f191a1b6740 10 client.0 _ll_register_callbacks cb 0x55d3dfda7d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:15:50.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.260+0000 7f1914b78640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3dfeab790 0x55d3dfea9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:50.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.260+0000 7f1915379640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3dfea2bb0 0x55d3dfea8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:50.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.260+0000 7f1914b78640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3dfeab790 0x55d3dfea9230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57196/0 (socket says 172.21.15.16:57196) 2024-03-28T22:15:50.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.260+0000 7f1915379640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3dfea2bb0 0x55d3dfea8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:49056/0 (socket says 172.21.15.16:49056) 2024-03-28T22:15:50.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.260+0000 7f1914b78640 1 -- 172.21.15.16:0/94176558 learned_addr learned my addr 172.21.15.16:0/94176558 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:50.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.260+0000 7f1914b78640 1 --2- 172.21.15.16:0/94176558 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3dfea2290 0x55d3dfea87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:50.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.260+0000 7f1914b78640 1 -- 172.21.15.16:0/94176558 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3dfea2290 msgr2=0x55d3dfea87b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:50.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.260+0000 7f1914b78640 1 --2- 172.21.15.16:0/94176558 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3dfea2290 0x55d3dfea87b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.260+0000 7f1914b78640 1 -- 172.21.15.16:0/94176558 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3dfea2bb0 msgr2=0x55d3dfea8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:50.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.260+0000 7f1914b78640 1 --2- 172.21.15.16:0/94176558 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3dfea2bb0 0x55d3dfea8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.260+0000 7f1914b78640 1 -- 172.21.15.16:0/94176558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3dfbe47d0 con 0x55d3dfeab790 2024-03-28T22:15:50.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.261+0000 7f1914b78640 1 --2- 172.21.15.16:0/94176558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3dfeab790 0x55d3dfea9230 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f1904012d50 tx=0x7f1904004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ae72e26d3720bfe server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:50.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.261+0000 7f18f4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:15:50.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.263+0000 7f18f4ff9640 1 -- 172.21.15.16:0/94176558 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1904004050 con 0x55d3dfeab790 2024-03-28T22:15:50.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.263+0000 7f18f4ff9640 1 -- 172.21.15.16:0/94176558 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f19040041f0 con 0x55d3dfeab790 2024-03-28T22:15:50.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.263+0000 7f191a1b6740 1 -- 172.21.15.16:0/94176558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1908009e50 con 0x55d3dfeab790 2024-03-28T22:15:50.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.263+0000 7f18f4ff9640 1 -- 172.21.15.16:0/94176558 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f190402a050 con 0x55d3dfeab790 2024-03-28T22:15:50.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.263+0000 7f191a1b6740 10 client.16656 fetch_fsmap learned FSMap version 27 2024-03-28T22:15:50.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.263+0000 7f191a1b6740 1 -- 172.21.15.16:0/94176558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d3dfb1a3f0 con 0x55d3dfeab790 2024-03-28T22:15:50.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.263+0000 7f18f4ff9640 1 -- 172.21.15.16:0/94176558 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1908009e50 con 0x55d3dfeab790 2024-03-28T22:15:50.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.263+0000 7f18f4ff9640 1 -- 172.21.15.16:0/94176558 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f1904025440 con 0x55d3dfeab790 2024-03-28T22:15:50.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.263+0000 7f191a1b6740 10 client.16656 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:15:50.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.263+0000 7f191a1b6740 10 client.16656 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:15:50.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.263+0000 7f191a1b6740 20 client.16656 populate_metadata read hostname 'smithi016' 2024-03-28T22:15:50.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.263+0000 7f191a1b6740 1 --2- 172.21.15.16:0/94176558 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d3dfebbb80 0x55d3dfebdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:50.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.263+0000 7f191a1b6740 4 client.16656 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:15:50.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.263+0000 7f191a1b6740 1 -- 172.21.15.16:0/94176558 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d3dfe136c0 con 0x55d3dfebbb80 2024-03-28T22:15:50.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.263+0000 7f1915379640 1 --2- 172.21.15.16:0/94176558 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d3dfebbb80 0x55d3dfebdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:50.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.264+0000 7f1915379640 1 --2- 172.21.15.16:0/94176558 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d3dfebbb80 0x55d3dfebdfb0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3591f784addb55b2 server_cookie=4df9764031a04514 in_seq=0 out_seq=0 2024-03-28T22:15:50.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.264+0000 7f18f4ff9640 10 client.16656 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:15:50.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.264+0000 7f18f4ff9640 1 -- 172.21.15.16:0/94176558 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d3dfe136c0 con 0x55d3dfebbb80 2024-03-28T22:15:50.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.265+0000 7f18f4ff9640 10 client.16656 handle_command_reply: tid=0 2024-03-28T22:15:50.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.265+0000 7f191a1b6740 1 client.16656 shutdown 2024-03-28T22:15:50.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.265+0000 7f191a1b6740 1 -- 172.21.15.16:0/94176558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3dfeab790 msgr2=0x55d3dfea9230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:50.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.265+0000 7f191a1b6740 1 --2- 172.21.15.16:0/94176558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3dfeab790 0x55d3dfea9230 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f1904012d50 tx=0x7f1904004730 comp rx=0 tx=0).stop 2024-03-28T22:15:50.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.265+0000 7f191a1b6740 1 -- 172.21.15.16:0/94176558 shutdown_connections 2024-03-28T22:15:50.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.265+0000 7f191a1b6740 1 --2- 172.21.15.16:0/94176558 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d3dfebbb80 0x55d3dfebdfb0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.265+0000 7f191a1b6740 1 --2- 172.21.15.16:0/94176558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3dfeab790 0x55d3dfea9230 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.265+0000 7f191a1b6740 1 --2- 172.21.15.16:0/94176558 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3dfea2bb0 0x55d3dfea8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.265+0000 7f18f4ff9640 0 client.16656 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:15:50.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.265+0000 7f191a1b6740 1 --2- 172.21.15.16:0/94176558 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3dfea2290 0x55d3dfea87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.265+0000 7f191a1b6740 1 -- 172.21.15.16:0/94176558 >> 172.21.15.16:0/94176558 conn(0x55d3dfe9c9c0 msgr2=0x55d3dfe9bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:50.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.265+0000 7f191a1b6740 1 -- 172.21.15.16:0/94176558 shutdown_connections 2024-03-28T22:15:50.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.265+0000 7f191a1b6740 1 -- 172.21.15.16:0/94176558 wait complete. 2024-03-28T22:15:50.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.265+0000 7f191a1b6740 20 client.16656 trim_cache size 0 max 16384 2024-03-28T22:15:50.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.282+0000 7f191a1b6740 1 Processor -- start 2024-03-28T22:15:50.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.282+0000 7f191a1b6740 1 -- start start 2024-03-28T22:15:50.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.282+0000 7f191a1b6740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3dfea2290 0x55d3dfeaaee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:50.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.282+0000 7f191a1b6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3dfea2bb0 0x55d3dfeb7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:50.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.283+0000 7f191a1b6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3dfeab790 0x55d3dfe9bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:50.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.283+0000 7f191a1b6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f19101afbb0 con 0x55d3dfeab790 2024-03-28T22:15:50.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.283+0000 7f191a1b6740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f19101ad480 con 0x55d3dfea2290 2024-03-28T22:15:50.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.283+0000 7f191a1b6740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f19101ab2f0 con 0x55d3dfea2bb0 2024-03-28T22:15:50.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.283+0000 7f1914b78640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3dfeab790 0x55d3dfe9bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:50.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.283+0000 7f1914b78640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3dfeab790 0x55d3dfe9bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57208/0 (socket says 172.21.15.16:57208) 2024-03-28T22:15:50.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.283+0000 7f1914b78640 1 -- 172.21.15.16:0/1348490986 learned_addr learned my addr 172.21.15.16:0/1348490986 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:50.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.283+0000 7f1914b78640 1 --2- 172.21.15.16:0/1348490986 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3dfea2290 0x55d3dfeaaee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:50.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.283+0000 7f1915379640 1 --2- 172.21.15.16:0/1348490986 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3dfea2bb0 0x55d3dfeb7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:50.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.283+0000 7f1914b78640 1 -- 172.21.15.16:0/1348490986 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3dfea2bb0 msgr2=0x55d3dfeb7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:50.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.283+0000 7f1914b78640 1 --2- 172.21.15.16:0/1348490986 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3dfea2bb0 0x55d3dfeb7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.283+0000 7f1914b78640 1 -- 172.21.15.16:0/1348490986 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3dfea2290 msgr2=0x55d3dfeaaee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:50.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.283+0000 7f1914b78640 1 --2- 172.21.15.16:0/1348490986 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3dfea2290 0x55d3dfeaaee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.283+0000 7f1914b78640 1 -- 172.21.15.16:0/1348490986 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3dfada0f0 con 0x55d3dfeab790 2024-03-28T22:15:50.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.283+0000 7f1915379640 1 --2- 172.21.15.16:0/1348490986 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3dfea2bb0 0x55d3dfeb7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:15:50.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.283+0000 7f1914b78640 1 --2- 172.21.15.16:0/1348490986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3dfeab790 0x55d3dfe9bc20 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f190400bd60 tx=0x7f19040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d2654984a5f3f9b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:50.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.283+0000 7f18f6ffd640 1 -- 172.21.15.16:0/1348490986 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1904026e60 con 0x55d3dfeab790 2024-03-28T22:15:50.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.283+0000 7f18f6ffd640 1 -- 172.21.15.16:0/1348490986 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1904004030 con 0x55d3dfeab790 2024-03-28T22:15:50.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.283+0000 7f18f6ffd640 1 -- 172.21.15.16:0/1348490986 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1904003850 con 0x55d3dfeab790 2024-03-28T22:15:50.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.284+0000 7f191a1b6740 1 -- 172.21.15.16:0/1348490986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3dfeab790 msgr2=0x55d3dfe9bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:50.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.284+0000 7f191a1b6740 1 --2- 172.21.15.16:0/1348490986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3dfeab790 0x55d3dfe9bc20 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f190400bd60 tx=0x7f19040046c0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.284+0000 7f191a1b6740 1 -- 172.21.15.16:0/1348490986 shutdown_connections 2024-03-28T22:15:50.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.284+0000 7f191a1b6740 1 --2- 172.21.15.16:0/1348490986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3dfeab790 0x55d3dfe9bc20 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.284+0000 7f191a1b6740 1 --2- 172.21.15.16:0/1348490986 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3dfea2bb0 0x55d3dfeb7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.284+0000 7f191a1b6740 1 --2- 172.21.15.16:0/1348490986 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3dfea2290 0x55d3dfeaaee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.284+0000 7f191a1b6740 1 -- 172.21.15.16:0/1348490986 >> 172.21.15.16:0/1348490986 conn(0x55d3dfe9c9c0 msgr2=0x55d3dfea76f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:50.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.284+0000 7f191a1b6740 1 -- 172.21.15.16:0/1348490986 shutdown_connections 2024-03-28T22:15:50.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.284+0000 7f191a1b6740 1 -- 172.21.15.16:0/1348490986 wait complete. 2024-03-28T22:15:50.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.284+0000 7f191a1b6740 1 -- start start 2024-03-28T22:15:50.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.284+0000 7f191a1b6740 1 Processor -- start 2024-03-28T22:15:50.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.285+0000 7f191a1b6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3dfea2290 0x55d3dfeb3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:50.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.285+0000 7f191a1b6740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3dfea2bb0 0x55d3dfeaf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:50.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.285+0000 7f191a1b6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3dfeab790 0x55d3dfeaf890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:50.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.285+0000 7f191a1b6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f19101ab940 con 0x55d3dfeab790 2024-03-28T22:15:50.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.285+0000 7f191a1b6740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f19101acdb0 con 0x55d3dfea2bb0 2024-03-28T22:15:50.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.285+0000 7f191a1b6740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f19101ac010 con 0x55d3dfea2290 2024-03-28T22:15:50.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.285+0000 7f191a1b6740 10 client.0 _ll_register_callbacks cb 0x7f190400bc90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:15:50.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.285+0000 7f1914b78640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3dfeab790 0x55d3dfeaf890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:50.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.285+0000 7f1915379640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3dfea2bb0 0x55d3dfeaf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:50.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.285+0000 7f1914b78640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3dfeab790 0x55d3dfeaf890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57212/0 (socket says 172.21.15.16:57212) 2024-03-28T22:15:50.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.285+0000 7f1914b78640 1 -- 172.21.15.16:0/3574252009 learned_addr learned my addr 172.21.15.16:0/3574252009 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:15:50.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.285+0000 7f1914b78640 1 --2- 172.21.15.16:0/3574252009 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3dfea2290 0x55d3dfeb3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:50.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.285+0000 7f1915379640 1 -- 172.21.15.16:0/3574252009 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3dfea2290 msgr2=0x55d3dfeb3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:50.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.285+0000 7f1915379640 1 --2- 172.21.15.16:0/3574252009 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3dfea2290 0x55d3dfeb3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.285+0000 7f1915379640 1 -- 172.21.15.16:0/3574252009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3dfeab790 msgr2=0x55d3dfeaf890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:15:50.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.285+0000 7f1915379640 1 --2- 172.21.15.16:0/3574252009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3dfeab790 0x55d3dfeaf890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.285+0000 7f1915379640 1 -- 172.21.15.16:0/3574252009 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3dff20eb0 con 0x55d3dfea2bb0 2024-03-28T22:15:50.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.285+0000 7f1914b78640 1 --2- 172.21.15.16:0/3574252009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3dfeab790 0x55d3dfeaf890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:15:50.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.285+0000 7f1915379640 1 --2- 172.21.15.16:0/3574252009 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3dfea2bb0 0x55d3dfeaf150 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f190800ebc0 tx=0x7f1908009330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7b049382e6f8b62 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:15:50.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.285+0000 7f18f4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-03-28T22:15:50.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.286+0000 7f18f4ff9640 1 -- 172.21.15.16:0/3574252009 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1908016040 con 0x55d3dfea2bb0 2024-03-28T22:15:50.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.286+0000 7f191a1b6740 1 -- 172.21.15.16:0/3574252009 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1904004030 con 0x55d3dfea2bb0 2024-03-28T22:15:50.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.286+0000 7f18f4ff9640 1 -- 172.21.15.16:0/3574252009 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f1908009600 con 0x55d3dfea2bb0 2024-03-28T22:15:50.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.286+0000 7f18f4ff9640 1 -- 172.21.15.16:0/3574252009 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f190803a050 con 0x55d3dfea2bb0 2024-03-28T22:15:50.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.286+0000 7f191a1b6740 10 client.26564 fetch_fsmap learned FSMap version 27 2024-03-28T22:15:50.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.286+0000 7f191a1b6740 1 -- 172.21.15.16:0/3574252009 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d3dff220e0 con 0x55d3dfea2bb0 2024-03-28T22:15:50.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.286+0000 7f18f4ff9640 1 -- 172.21.15.16:0/3574252009 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1904004030 con 0x55d3dfea2bb0 2024-03-28T22:15:50.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.286+0000 7f18f4ff9640 1 -- 172.21.15.16:0/3574252009 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f190800f040 con 0x55d3dfea2bb0 2024-03-28T22:15:50.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.286+0000 7f191a1b6740 10 client.26564 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:15:50.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.286+0000 7f191a1b6740 10 client.26564 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:15:50.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.286+0000 7f191a1b6740 20 client.26564 populate_metadata read hostname 'smithi016' 2024-03-28T22:15:50.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.286+0000 7f191a1b6740 1 --2- 172.21.15.16:0/3574252009 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d3dff22600 0x55d3dff24a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:15:50.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.286+0000 7f191a1b6740 4 client.26564 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:15:50.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.286+0000 7f191a1b6740 1 -- 172.21.15.16:0/3574252009 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d3dfe0da10 con 0x55d3dff22600 2024-03-28T22:15:50.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.286+0000 7f1914b78640 1 --2- 172.21.15.16:0/3574252009 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d3dff22600 0x55d3dff24a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:15:50.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.287+0000 7f18f4ff9640 10 client.26564 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:15:50.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.287+0000 7f1914b78640 1 --2- 172.21.15.16:0/3574252009 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d3dff22600 0x55d3dff24a20 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2cec0d5bc74a896 server_cookie=f802928f01c15ade in_seq=0 out_seq=0 2024-03-28T22:15:50.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.287+0000 7f18f4ff9640 1 -- 172.21.15.16:0/3574252009 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55d3dfe0da10 con 0x55d3dff22600 2024-03-28T22:15:50.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.287+0000 7f18f4ff9640 10 client.26564 handle_command_reply: tid=0 2024-03-28T22:15:50.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.287+0000 7f191a1b6740 1 client.26564 shutdown 2024-03-28T22:15:50.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.287+0000 7f191a1b6740 1 -- 172.21.15.16:0/3574252009 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3dfea2bb0 msgr2=0x55d3dfeaf150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:50.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.287+0000 7f191a1b6740 1 --2- 172.21.15.16:0/3574252009 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3dfea2bb0 0x55d3dfeaf150 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f190800ebc0 tx=0x7f1908009330 comp rx=0 tx=0).stop 2024-03-28T22:15:50.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.287+0000 7f191a1b6740 1 -- 172.21.15.16:0/3574252009 shutdown_connections 2024-03-28T22:15:50.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.287+0000 7f191a1b6740 1 --2- 172.21.15.16:0/3574252009 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d3dff22600 0x55d3dff24a20 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.287+0000 7f191a1b6740 1 --2- 172.21.15.16:0/3574252009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d3dfeab790 0x55d3dfeaf890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.287+0000 7f191a1b6740 1 --2- 172.21.15.16:0/3574252009 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d3dfea2bb0 0x55d3dfeaf150 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.287+0000 7f191a1b6740 1 --2- 172.21.15.16:0/3574252009 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3dfea2290 0x55d3dfeb3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.287+0000 7f18f4ff9640 0 client.26564 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:15:50.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.287+0000 7f191a1b6740 1 -- 172.21.15.16:0/3574252009 >> 172.21.15.16:0/3574252009 conn(0x55d3dfe9c9c0 msgr2=0x55d3dfe9bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:50.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.287+0000 7f191a1b6740 1 -- 172.21.15.16:0/3574252009 shutdown_connections 2024-03-28T22:15:50.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.287+0000 7f191a1b6740 1 -- 172.21.15.16:0/3574252009 wait complete. 2024-03-28T22:15:50.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.287+0000 7f191a1b6740 20 client.26564 trim_cache size 0 max 16384 2024-03-28T22:15:50.302 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:15:50.302 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T22:15:50.303 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-03-28T22:15:50.303 INFO:teuthology.orchestra.run.smithi016.stdout: "a71f7fc3-f767-448e-bc88-298b94956a81": { 2024-03-28T22:15:50.303 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-03-28T22:15:50.303 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "a71f7fc3-f767-448e-bc88-298b94956a81", 2024-03-28T22:15:50.303 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-03-28T22:15:50.303 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:15:50.303 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:15:50.303 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:15:50.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.288+0000 7f19175dd640 1 -- 172.21.15.16:0/1596319661 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f18f80a1ef0 msgr2=0x7f18f80a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:50.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.288+0000 7f19175dd640 1 --2- 172.21.15.16:0/1596319661 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f18f80a1ef0 0x7f18f80a43b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f19000bd0b0 tx=0x7f190008f0e0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.288+0000 7f19175dd640 1 -- 172.21.15.16:0/1596319661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f191010bd40 msgr2=0x7f191019c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:15:50.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.288+0000 7f19175dd640 1 --2- 172.21.15.16:0/1596319661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f191010bd40 0x7f191019c830 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f190002c790 tx=0x7f1900035ed0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.288+0000 7f19175dd640 1 -- 172.21.15.16:0/1596319661 shutdown_connections 2024-03-28T22:15:50.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.288+0000 7f19175dd640 1 --2- 172.21.15.16:0/1596319661 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f18f80a1ef0 0x7f18f80a43b0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.288+0000 7f19175dd640 1 --2- 172.21.15.16:0/1596319661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f191010bd40 0x7f191019c830 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.288+0000 7f19175dd640 1 --2- 172.21.15.16:0/1596319661 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1910103290 0x7f19101984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.288+0000 7f19175dd640 1 --2- 172.21.15.16:0/1596319661 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1910102970 0x7f1910197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:15:50.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.288+0000 7f19175dd640 1 -- 172.21.15.16:0/1596319661 >> 172.21.15.16:0/1596319661 conn(0x7f1910077800 msgr2=0x7f19100fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:15:50.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.288+0000 7f19175dd640 1 -- 172.21.15.16:0/1596319661 shutdown_connections 2024-03-28T22:15:50.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:15:50.288+0000 7f19175dd640 1 -- 172.21.15.16:0/1596319661 wait complete. 2024-03-28T22:15:50.307 INFO:tasks.cephfs.filesystem:scrub status for tag:a71f7fc3-f767-448e-bc88-298b94956a81 - {'path': '/', 'tag': 'a71f7fc3-f767-448e-bc88-298b94956a81', 'options': 'recursive,force'} 2024-03-28T22:15:50.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:50 smithi053 ceph-mon[35979]: pgmap v160: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:15:50.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:50 smithi063 ceph-mon[36249]: pgmap v160: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:15:50.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:50 smithi016 ceph-mon[32374]: pgmap v160: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:15:51.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:51 smithi053 ceph-mon[35979]: scrub queued for path: / 2024-03-28T22:15:51.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:51 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:15:51.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:51 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:15:51.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:51 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:15:51.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:51 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:15:51.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:51 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:15:51.549 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:51 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:15:51.549 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:51 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:15:51.549 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:51 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:15:51.549 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:51 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:15:51.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:51 smithi063 ceph-mon[36249]: scrub queued for path: / 2024-03-28T22:15:51.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:51 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:15:51.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:51 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:15:51.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:51 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:15:51.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:51 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:15:51.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:51 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:15:51.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:51 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:15:51.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:51 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:15:51.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:51 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:15:51.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:51 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:15:51.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:51 smithi016 ceph-mon[32374]: scrub queued for path: / 2024-03-28T22:15:51.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:51 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:15:51.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:51 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:15:51.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:51 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:15:51.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:51 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:15:51.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:51 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:15:51.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:51 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:15:51.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:51 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:15:51.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:51 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:15:51.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:51 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:15:52.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:52 smithi053 ceph-mon[35979]: pgmap v161: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:15:52.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:52 smithi063 ceph-mon[36249]: pgmap v161: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:15:52.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:52 smithi016 ceph-mon[32374]: pgmap v161: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:15:54.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:54 smithi053 ceph-mon[35979]: pgmap v162: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:15:54.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:54 smithi063 ceph-mon[36249]: pgmap v162: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:15:54.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:54 smithi016 ceph-mon[32374]: pgmap v162: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:15:56.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:56 smithi053 ceph-mon[35979]: pgmap v163: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:15:56.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:56 smithi063 ceph-mon[36249]: pgmap v163: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:15:56.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:56 smithi016 ceph-mon[32374]: pgmap v163: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:15:58.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:15:58 smithi053 ceph-mon[35979]: pgmap v164: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:15:58.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:15:58 smithi063 ceph-mon[36249]: pgmap v164: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:15:58.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:15:58 smithi016 ceph-mon[32374]: pgmap v164: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:16:00.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:00 smithi053 ceph-mon[35979]: pgmap v165: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:16:00.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:00 smithi063 ceph-mon[36249]: pgmap v165: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:16:00.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:00 smithi016 ceph-mon[32374]: pgmap v165: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:16:02.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:02 smithi053 ceph-mon[35979]: pgmap v166: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T22:16:02.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:02 smithi063 ceph-mon[36249]: pgmap v166: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T22:16:02.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:02 smithi016 ceph-mon[32374]: pgmap v166: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T22:16:04.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:04 smithi053 ceph-mon[35979]: pgmap v167: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-28T22:16:04.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:04 smithi063 ceph-mon[36249]: pgmap v167: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-28T22:16:04.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:04 smithi016 ceph-mon[32374]: pgmap v167: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-28T22:16:05.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:05 smithi053 ceph-mon[35979]: scrub summary: idle 2024-03-28T22:16:05.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:05 smithi063 ceph-mon[36249]: scrub summary: idle 2024-03-28T22:16:05.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:05 smithi016 ceph-mon[32374]: scrub summary: idle 2024-03-28T22:16:06.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:06 smithi053 ceph-mon[35979]: pgmap v168: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-28T22:16:06.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:06 smithi053 ceph-mon[35979]: scrub summary: idle 2024-03-28T22:16:06.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:06 smithi063 ceph-mon[36249]: pgmap v168: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-28T22:16:06.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:06 smithi063 ceph-mon[36249]: scrub summary: idle 2024-03-28T22:16:06.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:06 smithi016 ceph-mon[32374]: pgmap v168: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-28T22:16:06.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:06 smithi016 ceph-mon[32374]: scrub summary: idle 2024-03-28T22:16:08.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:08 smithi053 ceph-mon[35979]: pgmap v169: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T22:16:08.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:08 smithi063 ceph-mon[36249]: pgmap v169: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T22:16:08.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:08 smithi016 ceph-mon[32374]: pgmap v169: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T22:16:10.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:10 smithi053 ceph-mon[35979]: pgmap v170: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-28T22:16:10.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:10 smithi063 ceph-mon[36249]: pgmap v170: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-28T22:16:10.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:10 smithi016 ceph-mon[32374]: pgmap v170: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-28T22:16:12.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:12 smithi063 ceph-mon[36249]: pgmap v171: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T22:16:12.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:12 smithi016 ceph-mon[32374]: pgmap v171: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T22:16:12.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:12 smithi053 ceph-mon[35979]: pgmap v171: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-28T22:16:14.599 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:14 smithi053 ceph-mon[35979]: pgmap v172: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T22:16:14.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:14 smithi063 ceph-mon[36249]: pgmap v172: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T22:16:14.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:14 smithi016 ceph-mon[32374]: pgmap v172: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T22:16:16.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:16 smithi063 ceph-mon[36249]: pgmap v173: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T22:16:16.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:16 smithi016 ceph-mon[32374]: pgmap v173: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T22:16:16.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:16 smithi053 ceph-mon[35979]: pgmap v173: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-28T22:16:18.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:18 smithi063 ceph-mon[36249]: pgmap v174: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 7 op/s 2024-03-28T22:16:18.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:18 smithi016 ceph-mon[32374]: pgmap v174: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 7 op/s 2024-03-28T22:16:18.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:18 smithi053 ceph-mon[35979]: pgmap v174: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 7 op/s 2024-03-28T22:16:20.308 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:16:20.308 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:16:20.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.425+0000 7fd9aba6b640 1 -- 172.21.15.16:0/1168214978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9a41021e0 msgr2=0x7fd9a4102640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:20.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.425+0000 7fd9aba6b640 1 --2- 172.21.15.16:0/1168214978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9a41021e0 0x7fd9a4102640 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fd994009900 tx=0x7fd99402bd10 comp rx=0 tx=0).stop 2024-03-28T22:16:20.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.425+0000 7fd9aba6b640 1 -- 172.21.15.16:0/1168214978 shutdown_connections 2024-03-28T22:16:20.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.425+0000 7fd9aba6b640 1 --2- 172.21.15.16:0/1168214978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd9a4102b80 0x7fd9a410d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.425+0000 7fd9aba6b640 1 --2- 172.21.15.16:0/1168214978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9a41021e0 0x7fd9a4102640 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.425+0000 7fd9aba6b640 1 --2- 172.21.15.16:0/1168214978 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9a41009d0 0x7fd9a4100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.425+0000 7fd9aba6b640 1 -- 172.21.15.16:0/1168214978 >> 172.21.15.16:0/1168214978 conn(0x7fd9a4077830 msgr2=0x7fd9a4077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:20.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.425+0000 7fd9aba6b640 1 -- 172.21.15.16:0/1168214978 shutdown_connections 2024-03-28T22:16:20.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.425+0000 7fd9aba6b640 1 -- 172.21.15.16:0/1168214978 wait complete. 2024-03-28T22:16:20.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.425+0000 7fd9aba6b640 1 Processor -- start 2024-03-28T22:16:20.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.425+0000 7fd9aba6b640 1 -- start start 2024-03-28T22:16:20.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.426+0000 7fd9aba6b640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9a41009d0 0x7fd9a40fd590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.426+0000 7fd9aba6b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd9a41021e0 0x7fd9a40fdad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.426+0000 7fd9aba6b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9a4102b80 0x7fd9a40fe010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.426+0000 7fd9aba6b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a4110ae0 con 0x7fd9a4102b80 2024-03-28T22:16:20.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.426+0000 7fd9aba6b640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a4110960 con 0x7fd9a41009d0 2024-03-28T22:16:20.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.426+0000 7fd9aba6b640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a4110c60 con 0x7fd9a41021e0 2024-03-28T22:16:20.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.426+0000 7fd9aa008640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9a4102b80 0x7fd9a40fe010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.426+0000 7fd9aa008640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9a4102b80 0x7fd9a40fe010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55168/0 (socket says 172.21.15.16:55168) 2024-03-28T22:16:20.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.426+0000 7fd9a9807640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9a41009d0 0x7fd9a40fd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.426+0000 7fd9aa008640 1 -- 172.21.15.16:0/716278166 learned_addr learned my addr 172.21.15.16:0/716278166 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:20.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.426+0000 7fd9a9006640 1 --2- 172.21.15.16:0/716278166 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd9a41021e0 0x7fd9a40fdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.426+0000 7fd9aa008640 1 -- 172.21.15.16:0/716278166 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd9a41021e0 msgr2=0x7fd9a40fdad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:20.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.426+0000 7fd9aa008640 1 --2- 172.21.15.16:0/716278166 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd9a41021e0 0x7fd9a40fdad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.426+0000 7fd9aa008640 1 -- 172.21.15.16:0/716278166 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9a41009d0 msgr2=0x7fd9a40fd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:20.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.426+0000 7fd9aa008640 1 --2- 172.21.15.16:0/716278166 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9a41009d0 0x7fd9a40fd590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.426+0000 7fd9aa008640 1 -- 172.21.15.16:0/716278166 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9a41a64c0 con 0x7fd9a4102b80 2024-03-28T22:16:20.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.426+0000 7fd9a9807640 1 --2- 172.21.15.16:0/716278166 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9a41009d0 0x7fd9a40fd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:16:20.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.426+0000 7fd9aa008640 1 --2- 172.21.15.16:0/716278166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9a4102b80 0x7fd9a40fe010 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fd99c00cab0 tx=0x7fd99c00cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:20.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.426+0000 7fd99affd640 1 -- 172.21.15.16:0/716278166 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd99c00ae20 con 0x7fd9a4102b80 2024-03-28T22:16:20.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.427+0000 7fd99affd640 1 -- 172.21.15.16:0/716278166 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd99c00f930 con 0x7fd9a4102b80 2024-03-28T22:16:20.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.427+0000 7fd9aba6b640 1 -- 172.21.15.16:0/716278166 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9a41a67b0 con 0x7fd9a4102b80 2024-03-28T22:16:20.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.427+0000 7fd99affd640 1 -- 172.21.15.16:0/716278166 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd99c018b00 con 0x7fd9a4102b80 2024-03-28T22:16:20.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.427+0000 7fd9aba6b640 1 -- 172.21.15.16:0/716278166 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9a41a6c80 con 0x7fd9a4102b80 2024-03-28T22:16:20.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.429+0000 7fd99affd640 1 -- 172.21.15.16:0/716278166 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7fd99c00fae0 con 0x7fd9a4102b80 2024-03-28T22:16:20.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.429+0000 7fd9ae647740 1 Processor -- start 2024-03-28T22:16:20.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.429+0000 7fd9ae647740 1 -- start start 2024-03-28T22:16:20.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.430+0000 7fd9ae647740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b8e2607290 0x55b8e2607670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.430+0000 7fd9ae647740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e2607bb0 0x55b8e2610250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.430+0000 7fd9ae647740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8e2610790 0x55b8e2612b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.430+0000 7fd9ae647740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b8e256b180 con 0x55b8e2610790 2024-03-28T22:16:20.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.430+0000 7fd9ae647740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55b8e2513b70 con 0x55b8e2607290 2024-03-28T22:16:20.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.430+0000 7fd9ae647740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55b8e2567220 con 0x55b8e2607bb0 2024-03-28T22:16:20.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.430+0000 7fd9a9006640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b8e2607290 0x55b8e2607670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.430+0000 7fd9a9006640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b8e2607290 0x55b8e2607670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:51888/0 (socket says 172.21.15.16:51888) 2024-03-28T22:16:20.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.430+0000 7fd9a9006640 1 -- 172.21.15.16:0/3229895789 learned_addr learned my addr 172.21.15.16:0/3229895789 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:20.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.430+0000 7fd9a9006640 1 --2- 172.21.15.16:0/3229895789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8e2610790 0x55b8e2612b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.430+0000 7fd99affd640 1 --2- 172.21.15.16:0/716278166 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fd9900a1f20 0x7fd9900a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.430+0000 7fd9a9807640 1 --2- 172.21.15.16:0/3229895789 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e2607bb0 0x55b8e2610250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.430+0000 7fd9aa008640 1 --2- 172.21.15.16:0/716278166 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fd9900a1f20 0x7fd9900a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.430+0000 7fd9a9006640 1 -- 172.21.15.16:0/3229895789 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e2607bb0 msgr2=0x55b8e2610250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:20.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.430+0000 7fd9a9006640 1 --2- 172.21.15.16:0/3229895789 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e2607bb0 0x55b8e2610250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.430+0000 7fd9a9006640 1 -- 172.21.15.16:0/3229895789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8e2610790 msgr2=0x55b8e2612b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:20.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.430+0000 7fd9a9006640 1 --2- 172.21.15.16:0/3229895789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8e2610790 0x55b8e2612b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.430+0000 7fd9a9006640 1 -- 172.21.15.16:0/3229895789 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8e25277c0 con 0x55b8e2607290 2024-03-28T22:16:20.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.430+0000 7fd99affd640 1 -- 172.21.15.16:0/716278166 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fd99c0ce7a0 con 0x7fd9a4102b80 2024-03-28T22:16:20.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.430+0000 7fd9a9006640 1 --2- 172.21.15.16:0/3229895789 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b8e2607290 0x55b8e2607670 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd99402c250 tx=0x7fd994005e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8a2cc392d933fb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:20.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.431+0000 7fd9aa008640 1 --2- 172.21.15.16:0/716278166 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fd9900a1f20 0x7fd9900a43e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd99c0cbfc0 tx=0x7fd99c08d040 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:20.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.431+0000 7fd98ffff640 1 -- 172.21.15.16:0/3229895789 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd99402c930 con 0x55b8e2607290 2024-03-28T22:16:20.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.431+0000 7fd98ffff640 1 -- 172.21.15.16:0/3229895789 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9940046c0 con 0x55b8e2607290 2024-03-28T22:16:20.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.431+0000 7fd98ffff640 1 -- 172.21.15.16:0/3229895789 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd994005220 con 0x55b8e2607290 2024-03-28T22:16:20.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.431+0000 7fd9ae647740 1 -- 172.21.15.16:0/3229895789 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b8e2607290 msgr2=0x55b8e2607670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:20.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.431+0000 7fd9ae647740 1 --2- 172.21.15.16:0/3229895789 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b8e2607290 0x55b8e2607670 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd99402c250 tx=0x7fd994005e40 comp rx=0 tx=0).stop 2024-03-28T22:16:20.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.431+0000 7fd9ae647740 1 -- 172.21.15.16:0/3229895789 shutdown_connections 2024-03-28T22:16:20.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.431+0000 7fd9ae647740 1 --2- 172.21.15.16:0/3229895789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8e2610790 0x55b8e2612b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.431+0000 7fd9ae647740 1 --2- 172.21.15.16:0/3229895789 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e2607bb0 0x55b8e2610250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.431+0000 7fd9ae647740 1 --2- 172.21.15.16:0/3229895789 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b8e2607290 0x55b8e2607670 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.431+0000 7fd9ae647740 1 -- 172.21.15.16:0/3229895789 >> 172.21.15.16:0/3229895789 conn(0x55b8e26019c0 msgr2=0x55b8e2601da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:20.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.431+0000 7fd9ae647740 1 -- 172.21.15.16:0/3229895789 shutdown_connections 2024-03-28T22:16:20.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.431+0000 7fd9ae647740 1 -- 172.21.15.16:0/3229895789 wait complete. 2024-03-28T22:16:20.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.432+0000 7fd9ae647740 1 -- start start 2024-03-28T22:16:20.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.432+0000 7fd9ae647740 1 Processor -- start 2024-03-28T22:16:20.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.432+0000 7fd9ae647740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b8e2607290 0x55b8e260d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.432+0000 7fd9ae647740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e2607bb0 0x55b8e260dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.432+0000 7fd9ae647740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8e2610790 0x55b8e260e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.432+0000 7fd9ae647740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b8e2613520 con 0x55b8e2610790 2024-03-28T22:16:20.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.432+0000 7fd9ae647740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55b8e25094a0 con 0x55b8e2607290 2024-03-28T22:16:20.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.432+0000 7fd9ae647740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55b8e26136a0 con 0x55b8e2607bb0 2024-03-28T22:16:20.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.432+0000 7fd9ae647740 10 client.0 _ll_register_callbacks cb 0x55b8e250cd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:16:20.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.432+0000 7fd9a9006640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b8e2607290 0x55b8e260d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.432+0000 7fd9a9006640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8e2610790 0x55b8e260e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.433+0000 7fd9a9006640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8e2610790 0x55b8e260e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55184/0 (socket says 172.21.15.16:55184) 2024-03-28T22:16:20.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.433+0000 7fd9a9006640 1 -- 172.21.15.16:0/3810256648 learned_addr learned my addr 172.21.15.16:0/3810256648 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:20.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.433+0000 7fd9a9807640 1 --2- 172.21.15.16:0/3810256648 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e2607bb0 0x55b8e260dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.433+0000 7fd9a9006640 1 -- 172.21.15.16:0/3810256648 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e2607bb0 msgr2=0x55b8e260dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:20.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.433+0000 7fd9a9006640 1 --2- 172.21.15.16:0/3810256648 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e2607bb0 0x55b8e260dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.433+0000 7fd9a9006640 1 -- 172.21.15.16:0/3810256648 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b8e2607290 msgr2=0x55b8e260d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:20.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.433+0000 7fd9a9006640 1 --2- 172.21.15.16:0/3810256648 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b8e2607290 0x55b8e260d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.433+0000 7fd9a9006640 1 -- 172.21.15.16:0/3810256648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8e23497d0 con 0x55b8e2610790 2024-03-28T22:16:20.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.433+0000 7fd9a9807640 1 --2- 172.21.15.16:0/3810256648 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e2607bb0 0x55b8e260dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:16:20.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.433+0000 7fd9a9006640 1 --2- 172.21.15.16:0/3810256648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8e2610790 0x55b8e260e2d0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fd99402c930 tx=0x7fd994005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ae7690d71a7061f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:20.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.433+0000 7fd98d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:16:20.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.433+0000 7fd98d7fa640 1 -- 172.21.15.16:0/3810256648 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd99402cda0 con 0x55b8e2610790 2024-03-28T22:16:20.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.433+0000 7fd98d7fa640 1 -- 172.21.15.16:0/3810256648 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fd994004fa0 con 0x55b8e2610790 2024-03-28T22:16:20.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.433+0000 7fd9ae647740 1 -- 172.21.15.16:0/3810256648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9940046c0 con 0x55b8e2610790 2024-03-28T22:16:20.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.433+0000 7fd98d7fa640 1 -- 172.21.15.16:0/3810256648 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd9940051a0 con 0x55b8e2610790 2024-03-28T22:16:20.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.433+0000 7fd9ae647740 10 client.16674 fetch_fsmap learned FSMap version 27 2024-03-28T22:16:20.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.433+0000 7fd9ae647740 1 -- 172.21.15.16:0/3810256648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b8e227f3f0 con 0x55b8e2610790 2024-03-28T22:16:20.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.433+0000 7fd98d7fa640 1 -- 172.21.15.16:0/3810256648 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9940046c0 con 0x55b8e2610790 2024-03-28T22:16:20.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.434+0000 7fd98d7fa640 1 -- 172.21.15.16:0/3810256648 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7fd994058030 con 0x55b8e2610790 2024-03-28T22:16:20.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.434+0000 7fd9ae647740 10 client.16674 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:16:20.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.434+0000 7fd9ae647740 10 client.16674 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:16:20.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.434+0000 7fd9ae647740 20 client.16674 populate_metadata read hostname 'smithi016' 2024-03-28T22:16:20.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.434+0000 7fd9ae647740 1 --2- 172.21.15.16:0/3810256648 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b8e26209d0 0x55b8e2622dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.434+0000 7fd9ae647740 4 client.16674 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:16:20.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.434+0000 7fd9ae647740 1 -- 172.21.15.16:0/3810256648 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b8e25786c0 con 0x55b8e26209d0 2024-03-28T22:16:20.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.434+0000 7fd9a9807640 1 --2- 172.21.15.16:0/3810256648 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b8e26209d0 0x55b8e2622dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.434+0000 7fd9a9807640 1 --2- 172.21.15.16:0/3810256648 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b8e26209d0 0x55b8e2622dc0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c8c914ceefe2f42 server_cookie=2837566571359664 in_seq=0 out_seq=0 2024-03-28T22:16:20.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.434+0000 7fd98d7fa640 10 client.16674 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:16:20.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.435+0000 7fd98d7fa640 1 -- 172.21.15.16:0/3810256648 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b8e25786c0 con 0x55b8e26209d0 2024-03-28T22:16:20.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.435+0000 7fd98d7fa640 10 client.16674 handle_command_reply: tid=0 2024-03-28T22:16:20.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.435+0000 7fd9ae647740 1 client.16674 shutdown 2024-03-28T22:16:20.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.435+0000 7fd9ae647740 1 -- 172.21.15.16:0/3810256648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8e2610790 msgr2=0x55b8e260e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:20.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.435+0000 7fd9ae647740 1 --2- 172.21.15.16:0/3810256648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8e2610790 0x55b8e260e2d0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fd99402c930 tx=0x7fd994005fd0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.435+0000 7fd9ae647740 1 -- 172.21.15.16:0/3810256648 shutdown_connections 2024-03-28T22:16:20.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.435+0000 7fd9ae647740 1 --2- 172.21.15.16:0/3810256648 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b8e26209d0 0x55b8e2622dc0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.435+0000 7fd9ae647740 1 --2- 172.21.15.16:0/3810256648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8e2610790 0x55b8e260e2d0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.435+0000 7fd9ae647740 1 --2- 172.21.15.16:0/3810256648 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e2607bb0 0x55b8e260dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.435+0000 7fd9ae647740 1 --2- 172.21.15.16:0/3810256648 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b8e2607290 0x55b8e260d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.435+0000 7fd98d7fa640 0 client.16674 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:16:20.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.435+0000 7fd9ae647740 1 -- 172.21.15.16:0/3810256648 >> 172.21.15.16:0/3810256648 conn(0x55b8e26019c0 msgr2=0x55b8e2600c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:20.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.435+0000 7fd9ae647740 1 -- 172.21.15.16:0/3810256648 shutdown_connections 2024-03-28T22:16:20.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.435+0000 7fd9ae647740 1 -- 172.21.15.16:0/3810256648 wait complete. 2024-03-28T22:16:20.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.435+0000 7fd9ae647740 20 client.16674 trim_cache size 0 max 16384 2024-03-28T22:16:20.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.453+0000 7fd9ae647740 1 Processor -- start 2024-03-28T22:16:20.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.453+0000 7fd9ae647740 1 -- start start 2024-03-28T22:16:20.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.453+0000 7fd9ae647740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b8e2607290 0x55b8e260ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.453+0000 7fd9ae647740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8e2607bb0 0x55b8e261c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.453+0000 7fd9ae647740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e2610790 0x55b8e2628360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.453+0000 7fd9ae647740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a41b1d30 con 0x55b8e2607bb0 2024-03-28T22:16:20.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.453+0000 7fd9ae647740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a41af600 con 0x55b8e2607290 2024-03-28T22:16:20.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.453+0000 7fd9ae647740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a41ad450 con 0x55b8e2610790 2024-03-28T22:16:20.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.453+0000 7fd9a9807640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8e2607bb0 0x55b8e261c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.453+0000 7fd9a9807640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8e2607bb0 0x55b8e261c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55186/0 (socket says 172.21.15.16:55186) 2024-03-28T22:16:20.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.453+0000 7fd9a9807640 1 -- 172.21.15.16:0/3964574283 learned_addr learned my addr 172.21.15.16:0/3964574283 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:20.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.453+0000 7fd9a9006640 1 --2- 172.21.15.16:0/3964574283 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b8e2607290 0x55b8e260ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.453+0000 7fd9a9006640 1 --2- 172.21.15.16:0/3964574283 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e2610790 0x55b8e2628360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.453+0000 7fd9a9807640 1 -- 172.21.15.16:0/3964574283 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e2610790 msgr2=0x55b8e2628360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:20.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.453+0000 7fd9a9807640 1 --2- 172.21.15.16:0/3964574283 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e2610790 0x55b8e2628360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.453+0000 7fd9a9807640 1 -- 172.21.15.16:0/3964574283 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b8e2607290 msgr2=0x55b8e260ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:20.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.453+0000 7fd9a9807640 1 --2- 172.21.15.16:0/3964574283 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b8e2607290 0x55b8e260ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.454+0000 7fd9a9807640 1 -- 172.21.15.16:0/3964574283 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8e2623130 con 0x55b8e2607bb0 2024-03-28T22:16:20.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.454+0000 7fd9a9807640 1 --2- 172.21.15.16:0/3964574283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8e2607bb0 0x55b8e261c2c0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fd9a0010540 tx=0x7fd9a0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6b1a456b675d555 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:20.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.454+0000 7fd98f7fe640 1 -- 172.21.15.16:0/3964574283 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a000fad0 con 0x55b8e2607bb0 2024-03-28T22:16:20.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.454+0000 7fd98f7fe640 1 -- 172.21.15.16:0/3964574283 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9a0005500 con 0x55b8e2607bb0 2024-03-28T22:16:20.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.454+0000 7fd98f7fe640 1 -- 172.21.15.16:0/3964574283 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a0004d00 con 0x55b8e2607bb0 2024-03-28T22:16:20.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.454+0000 7fd9ae647740 1 -- 172.21.15.16:0/3964574283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8e2607bb0 msgr2=0x55b8e261c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:20.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.454+0000 7fd9ae647740 1 --2- 172.21.15.16:0/3964574283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8e2607bb0 0x55b8e261c2c0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fd9a0010540 tx=0x7fd9a0005f90 comp rx=0 tx=0).stop 2024-03-28T22:16:20.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.454+0000 7fd9ae647740 1 -- 172.21.15.16:0/3964574283 shutdown_connections 2024-03-28T22:16:20.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.454+0000 7fd9ae647740 1 --2- 172.21.15.16:0/3964574283 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e2610790 0x55b8e2628360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.454+0000 7fd9ae647740 1 --2- 172.21.15.16:0/3964574283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8e2607bb0 0x55b8e261c2c0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.454+0000 7fd9ae647740 1 --2- 172.21.15.16:0/3964574283 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b8e2607290 0x55b8e260ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.454+0000 7fd9ae647740 1 -- 172.21.15.16:0/3964574283 >> 172.21.15.16:0/3964574283 conn(0x55b8e26019c0 msgr2=0x55b8e2600c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:20.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.455+0000 7fd9ae647740 1 -- 172.21.15.16:0/3964574283 shutdown_connections 2024-03-28T22:16:20.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.455+0000 7fd9ae647740 1 -- 172.21.15.16:0/3964574283 wait complete. 2024-03-28T22:16:20.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.455+0000 7fd9ae647740 1 -- start start 2024-03-28T22:16:20.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.455+0000 7fd9ae647740 1 Processor -- start 2024-03-28T22:16:20.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.455+0000 7fd9ae647740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e2607290 0x55b8e260aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.455+0000 7fd9ae647740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b8e2607bb0 0x55b8e260af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.455+0000 7fd9ae647740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8e2610790 0x55b8e2685990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.455+0000 7fd9ae647740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a41adac0 con 0x55b8e2610790 2024-03-28T22:16:20.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.455+0000 7fd9ae647740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a41aef30 con 0x55b8e2607bb0 2024-03-28T22:16:20.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.455+0000 7fd9ae647740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a41ae190 con 0x55b8e2607290 2024-03-28T22:16:20.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.455+0000 7fd9ae647740 10 client.0 _ll_register_callbacks cb 0x7fd9940040c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:16:20.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.455+0000 7fd9a9006640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8e2610790 0x55b8e2685990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.456+0000 7fd9a9807640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b8e2607bb0 0x55b8e260af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.456+0000 7fd9a9006640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8e2610790 0x55b8e2685990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55198/0 (socket says 172.21.15.16:55198) 2024-03-28T22:16:20.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.456+0000 7fd9a9006640 1 -- 172.21.15.16:0/2303472869 learned_addr learned my addr 172.21.15.16:0/2303472869 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:20.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.456+0000 7fd9a9006640 1 --2- 172.21.15.16:0/2303472869 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e2607290 0x55b8e260aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.456+0000 7fd9a9807640 1 -- 172.21.15.16:0/2303472869 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e2607290 msgr2=0x55b8e260aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:20.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.456+0000 7fd9a9807640 1 --2- 172.21.15.16:0/2303472869 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e2607290 0x55b8e260aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.456+0000 7fd9a9807640 1 -- 172.21.15.16:0/2303472869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8e2610790 msgr2=0x55b8e2685990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:20.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.456+0000 7fd9a9807640 1 --2- 172.21.15.16:0/2303472869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8e2610790 0x55b8e2685990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.456+0000 7fd9a9807640 1 -- 172.21.15.16:0/2303472869 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8e26079e0 con 0x55b8e2607bb0 2024-03-28T22:16:20.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.456+0000 7fd9a9006640 1 --2- 172.21.15.16:0/2303472869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8e2610790 0x55b8e2685990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:16:20.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.456+0000 7fd9a9807640 1 --2- 172.21.15.16:0/2303472869 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b8e2607bb0 0x55b8e260af60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x55b8e2513bb0 tx=0x7fd9a0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6eee3f30fb361132 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:20.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.456+0000 7fd98d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-03-28T22:16:20.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.456+0000 7fd98d7fa640 1 -- 172.21.15.16:0/2303472869 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a0005220 con 0x55b8e2607bb0 2024-03-28T22:16:20.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.456+0000 7fd9ae647740 1 -- 172.21.15.16:0/2303472869 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9a0005500 con 0x55b8e2607bb0 2024-03-28T22:16:20.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.456+0000 7fd98d7fa640 1 -- 172.21.15.16:0/2303472869 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fd9a0004a80 con 0x55b8e2607bb0 2024-03-28T22:16:20.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.456+0000 7fd98d7fa640 1 -- 172.21.15.16:0/2303472869 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd9a0037050 con 0x55b8e2607bb0 2024-03-28T22:16:20.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.456+0000 7fd9ae647740 10 client.26582 fetch_fsmap learned FSMap version 27 2024-03-28T22:16:20.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.456+0000 7fd9ae647740 1 -- 172.21.15.16:0/2303472869 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b8e223f0f0 con 0x55b8e2607bb0 2024-03-28T22:16:20.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.457+0000 7fd98d7fa640 1 -- 172.21.15.16:0/2303472869 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9a0005500 con 0x55b8e2607bb0 2024-03-28T22:16:20.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.457+0000 7fd98d7fa640 1 -- 172.21.15.16:0/2303472869 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7fd9a00112e0 con 0x55b8e2607bb0 2024-03-28T22:16:20.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.457+0000 7fd9ae647740 10 client.26582 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:16:20.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.457+0000 7fd9ae647740 10 client.26582 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:16:20.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.457+0000 7fd9ae647740 20 client.26582 populate_metadata read hostname 'smithi016' 2024-03-28T22:16:20.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.457+0000 7fd9ae647740 1 --2- 172.21.15.16:0/2303472869 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b8e2687440 0x55b8e2689860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.457+0000 7fd9ae647740 4 client.26582 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:16:20.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.457+0000 7fd9ae647740 1 -- 172.21.15.16:0/2303472869 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b8e2572a10 con 0x55b8e2687440 2024-03-28T22:16:20.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.457+0000 7fd9a9006640 1 --2- 172.21.15.16:0/2303472869 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b8e2687440 0x55b8e2689860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.457+0000 7fd98d7fa640 10 client.26582 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:16:20.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.457+0000 7fd9a9006640 1 --2- 172.21.15.16:0/2303472869 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b8e2687440 0x55b8e2689860 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a13c5ed07ffc97f server_cookie=169dd11acb9f9ba8 in_seq=0 out_seq=0 2024-03-28T22:16:20.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.458+0000 7fd98d7fa640 1 -- 172.21.15.16:0/2303472869 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55b8e2572a10 con 0x55b8e2687440 2024-03-28T22:16:20.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.458+0000 7fd98d7fa640 10 client.26582 handle_command_reply: tid=0 2024-03-28T22:16:20.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.458+0000 7fd9ae647740 1 client.26582 shutdown 2024-03-28T22:16:20.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.458+0000 7fd9ae647740 1 -- 172.21.15.16:0/2303472869 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b8e2607bb0 msgr2=0x55b8e260af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:20.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.458+0000 7fd9ae647740 1 --2- 172.21.15.16:0/2303472869 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b8e2607bb0 0x55b8e260af60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x55b8e2513bb0 tx=0x7fd9a0004730 comp rx=0 tx=0).stop 2024-03-28T22:16:20.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.458+0000 7fd9ae647740 1 -- 172.21.15.16:0/2303472869 shutdown_connections 2024-03-28T22:16:20.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.458+0000 7fd9ae647740 1 --2- 172.21.15.16:0/2303472869 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b8e2687440 0x55b8e2689860 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.458+0000 7fd9ae647740 1 --2- 172.21.15.16:0/2303472869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8e2610790 0x55b8e2685990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.458+0000 7fd9ae647740 1 --2- 172.21.15.16:0/2303472869 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b8e2607bb0 0x55b8e260af60 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.458+0000 7fd9ae647740 1 --2- 172.21.15.16:0/2303472869 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e2607290 0x55b8e260aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.458+0000 7fd98d7fa640 0 client.26582 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:16:20.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.458+0000 7fd9ae647740 1 -- 172.21.15.16:0/2303472869 >> 172.21.15.16:0/2303472869 conn(0x55b8e26019c0 msgr2=0x55b8e260d810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:20.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.458+0000 7fd9ae647740 1 -- 172.21.15.16:0/2303472869 shutdown_connections 2024-03-28T22:16:20.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.458+0000 7fd9ae647740 1 -- 172.21.15.16:0/2303472869 wait complete. 2024-03-28T22:16:20.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.458+0000 7fd9ae647740 20 client.26582 trim_cache size 0 max 16384 2024-03-28T22:16:20.472 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:16:20.472 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-03-28T22:16:20.472 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-03-28T22:16:20.472 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:16:20.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.458+0000 7fd9aba6b640 1 -- 172.21.15.16:0/716278166 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fd9900a1f20 msgr2=0x7fd9900a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:20.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.458+0000 7fd9aba6b640 1 --2- 172.21.15.16:0/716278166 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fd9900a1f20 0x7fd9900a43e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd99c0cbfc0 tx=0x7fd99c08d040 comp rx=0 tx=0).stop 2024-03-28T22:16:20.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.458+0000 7fd9aba6b640 1 -- 172.21.15.16:0/716278166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9a4102b80 msgr2=0x7fd9a40fe010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:20.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.458+0000 7fd9aba6b640 1 --2- 172.21.15.16:0/716278166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9a4102b80 0x7fd9a40fe010 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fd99c00cab0 tx=0x7fd99c00cf70 comp rx=0 tx=0).stop 2024-03-28T22:16:20.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.458+0000 7fd9aba6b640 1 -- 172.21.15.16:0/716278166 shutdown_connections 2024-03-28T22:16:20.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.458+0000 7fd9aba6b640 1 --2- 172.21.15.16:0/716278166 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fd9900a1f20 0x7fd9900a43e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.458+0000 7fd9aba6b640 1 --2- 172.21.15.16:0/716278166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9a4102b80 0x7fd9a40fe010 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.458+0000 7fd9aba6b640 1 --2- 172.21.15.16:0/716278166 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd9a41021e0 0x7fd9a40fdad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.458+0000 7fd9aba6b640 1 --2- 172.21.15.16:0/716278166 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9a41009d0 0x7fd9a40fd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.458+0000 7fd9aba6b640 1 -- 172.21.15.16:0/716278166 >> 172.21.15.16:0/716278166 conn(0x7fd9a4077830 msgr2=0x7fd9a406a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:20.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.459+0000 7fd9aba6b640 1 -- 172.21.15.16:0/716278166 shutdown_connections 2024-03-28T22:16:20.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.459+0000 7fd9aba6b640 1 -- 172.21.15.16:0/716278166 wait complete. 2024-03-28T22:16:20.475 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:16:20.475 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:16:20.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.577+0000 7ff3c94c9640 1 -- 172.21.15.16:0/2327133546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3c4100fc0 msgr2=0x7ff3c410d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:20.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.577+0000 7ff3c94c9640 1 --2- 172.21.15.16:0/2327133546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3c4100fc0 0x7ff3c410d4d0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4009cd0 tx=0x7ff3b402c0f0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.577+0000 7ff3c94c9640 1 -- 172.21.15.16:0/2327133546 shutdown_connections 2024-03-28T22:16:20.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.577+0000 7ff3c94c9640 1 --2- 172.21.15.16:0/2327133546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3c4100fc0 0x7ff3c410d4d0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.577+0000 7ff3c94c9640 1 --2- 172.21.15.16:0/2327133546 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3c4100600 0x7ff3c4100a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.577+0000 7ff3c94c9640 1 --2- 172.21.15.16:0/2327133546 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3c41066a0 0x7ff3c4106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.577+0000 7ff3c94c9640 1 -- 172.21.15.16:0/2327133546 >> 172.21.15.16:0/2327133546 conn(0x7ff3c40777d0 msgr2=0x7ff3c4077be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:20.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.577+0000 7ff3c94c9640 1 -- 172.21.15.16:0/2327133546 shutdown_connections 2024-03-28T22:16:20.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.577+0000 7ff3c94c9640 1 -- 172.21.15.16:0/2327133546 wait complete. 2024-03-28T22:16:20.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.577+0000 7ff3c94c9640 1 Processor -- start 2024-03-28T22:16:20.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.578+0000 7ff3c94c9640 1 -- start start 2024-03-28T22:16:20.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.578+0000 7ff3c94c9640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3c4100600 0x7ff3c4196f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.578+0000 7ff3c94c9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3c4100fc0 0x7ff3c4197440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.578+0000 7ff3c94c9640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3c41066a0 0x7ff3c419b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.578+0000 7ff3c94c9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c4110a20 con 0x7ff3c4100fc0 2024-03-28T22:16:20.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.578+0000 7ff3c94c9640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c41108a0 con 0x7ff3c4100600 2024-03-28T22:16:20.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.578+0000 7ff3c94c9640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c4110ba0 con 0x7ff3c41066a0 2024-03-28T22:16:20.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.578+0000 7ff3c27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3c4100fc0 0x7ff3c4197440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.578+0000 7ff3c27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3c4100fc0 0x7ff3c4197440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40230/0 (socket says 172.21.15.16:40230) 2024-03-28T22:16:20.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.578+0000 7ff3c27fc640 1 -- 172.21.15.16:0/3755812130 learned_addr learned my addr 172.21.15.16:0/3755812130 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:20.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.578+0000 7ff3c2ffd640 1 --2- 172.21.15.16:0/3755812130 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3c4100600 0x7ff3c4196f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.578+0000 7ff3c37fe640 1 --2- 172.21.15.16:0/3755812130 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3c41066a0 0x7ff3c419b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.578+0000 7ff3c27fc640 1 -- 172.21.15.16:0/3755812130 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3c41066a0 msgr2=0x7ff3c419b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:20.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.578+0000 7ff3c27fc640 1 --2- 172.21.15.16:0/3755812130 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3c41066a0 0x7ff3c419b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.578+0000 7ff3c27fc640 1 -- 172.21.15.16:0/3755812130 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3c4100600 msgr2=0x7ff3c4196f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:20.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.578+0000 7ff3c27fc640 1 --2- 172.21.15.16:0/3755812130 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3c4100600 0x7ff3c4196f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.578+0000 7ff3c27fc640 1 -- 172.21.15.16:0/3755812130 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3c419bef0 con 0x7ff3c4100fc0 2024-03-28T22:16:20.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.578+0000 7ff3c2ffd640 1 --2- 172.21.15.16:0/3755812130 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3c4100600 0x7ff3c4196f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:16:20.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.578+0000 7ff3c27fc640 1 --2- 172.21.15.16:0/3755812130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3c4100fc0 0x7ff3c4197440 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac00b4a0 tx=0x7ff3ac00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:20.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.579+0000 7ff3bbfff640 1 -- 172.21.15.16:0/3755812130 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac014070 con 0x7ff3c4100fc0 2024-03-28T22:16:20.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.579+0000 7ff3c94c9640 1 -- 172.21.15.16:0/3755812130 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3c419c1e0 con 0x7ff3c4100fc0 2024-03-28T22:16:20.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.579+0000 7ff3bbfff640 1 -- 172.21.15.16:0/3755812130 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3ac010af0 con 0x7ff3c4100fc0 2024-03-28T22:16:20.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.579+0000 7ff3bbfff640 1 -- 172.21.15.16:0/3755812130 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac017d10 con 0x7ff3c4100fc0 2024-03-28T22:16:20.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.579+0000 7ff3c94c9640 1 -- 172.21.15.16:0/3755812130 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3c41a3a70 con 0x7ff3c4100fc0 2024-03-28T22:16:20.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.580+0000 7ff3c94c9640 1 -- 172.21.15.16:0/3755812130 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3c40685d0 con 0x7ff3c4100fc0 2024-03-28T22:16:20.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.581+0000 7ff3bbfff640 1 -- 172.21.15.16:0/3755812130 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7ff3ac017070 con 0x7ff3c4100fc0 2024-03-28T22:16:20.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.581+0000 7ff3bbfff640 1 --2- 172.21.15.16:0/3755812130 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff3b00a1ef0 0x7ff3b00a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.581+0000 7ff3c2ffd640 1 --2- 172.21.15.16:0/3755812130 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff3b00a1ef0 0x7ff3b00a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.582+0000 7ff3bbfff640 1 -- 172.21.15.16:0/3755812130 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7ff3ac0cd860 con 0x7ff3c4100fc0 2024-03-28T22:16:20.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.582+0000 7ff3c2ffd640 1 --2- 172.21.15.16:0/3755812130 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff3b00a1ef0 0x7ff3b00a43b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff3c4101740 tx=0x7ff3bc005f90 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:20.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.585+0000 7ff3bbfff640 1 -- 172.21.15.16:0/3755812130 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff3ac09aa20 con 0x7ff3c4100fc0 2024-03-28T22:16:20.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:20 smithi063 ceph-mon[36249]: pgmap v175: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-28T22:16:20.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:20 smithi016 ceph-mon[32374]: pgmap v175: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-28T22:16:20.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:20 smithi053 ceph-mon[35979]: pgmap v175: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-28T22:16:20.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.810+0000 7ff3c94c9640 1 -- 172.21.15.16:0/3755812130 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff3c4198250 con 0x7ff3c4100fc0 2024-03-28T22:16:20.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.811+0000 7ff3bbfff640 1 -- 172.21.15.16:0/3755812130 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11851 (secure 0 0 0) 0x7ff3ac09a320 con 0x7ff3c4100fc0 2024-03-28T22:16:20.813 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:16:20.813 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/1644581227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1644581227},{"type":"v1","addr":"172.21.15.16:6839","nonce":1644581227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/2593062669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2593062669},{"type":"v1","addr":"172.21.15.16:6841","nonce":2593062669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:08:29.672976+0000","modified":"2024-03-28T22:14:44.922044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24566,"mds_1":24524,"mds_2":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24466":{"gid":24466,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/247249857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":247249857},{"type":"v1","addr":"172.21.15.63:6835","nonce":247249857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6837/2324231032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":2324231032},{"type":"v1","addr":"172.21.15.63:6837","nonce":2324231032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6839/1177197077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":1177197077},{"type":"v1","addr":"172.21.15.63:6839","nonce":1177197077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"e","rank":2,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.53:6835/350727634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":350727634},{"type":"v1","addr":"172.21.15.53:6835","nonce":350727634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/349877573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":349877573},{"type":"v1","addr":"172.21.15.53:6837","nonce":349877573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6839/3714153956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":3714153956},{"type":"v1","addr":"172.21.15.53:6839","nonce":3714153956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24566,"qdb_cluster":[24500,24524,24566]},"id":1}]} 2024-03-28T22:16:20.813 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2024-03-28T22:16:20.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.813+0000 7ff3c94c9640 1 -- 172.21.15.16:0/3755812130 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff3b00a1ef0 msgr2=0x7ff3b00a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:20.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.813+0000 7ff3c94c9640 1 --2- 172.21.15.16:0/3755812130 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff3b00a1ef0 0x7ff3b00a43b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff3c4101740 tx=0x7ff3bc005f90 comp rx=0 tx=0).stop 2024-03-28T22:16:20.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.813+0000 7ff3c94c9640 1 -- 172.21.15.16:0/3755812130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3c4100fc0 msgr2=0x7ff3c4197440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:20.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.813+0000 7ff3c94c9640 1 --2- 172.21.15.16:0/3755812130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3c4100fc0 0x7ff3c4197440 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac00b4a0 tx=0x7ff3ac00b960 comp rx=0 tx=0).stop 2024-03-28T22:16:20.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.813+0000 7ff3c94c9640 1 -- 172.21.15.16:0/3755812130 shutdown_connections 2024-03-28T22:16:20.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.813+0000 7ff3c94c9640 1 --2- 172.21.15.16:0/3755812130 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff3b00a1ef0 0x7ff3b00a43b0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.813+0000 7ff3c94c9640 1 --2- 172.21.15.16:0/3755812130 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3c41066a0 0x7ff3c419b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.813+0000 7ff3c94c9640 1 --2- 172.21.15.16:0/3755812130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3c4100fc0 0x7ff3c4197440 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.813+0000 7ff3c94c9640 1 --2- 172.21.15.16:0/3755812130 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3c4100600 0x7ff3c4196f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.813+0000 7ff3c94c9640 1 -- 172.21.15.16:0/3755812130 >> 172.21.15.16:0/3755812130 conn(0x7ff3c40777d0 msgr2=0x7ff3c40ff180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:20.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.813+0000 7ff3c94c9640 1 -- 172.21.15.16:0/3755812130 shutdown_connections 2024-03-28T22:16:20.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.813+0000 7ff3c94c9640 1 -- 172.21.15.16:0/3755812130 wait complete. 2024-03-28T22:16:20.826 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:16:20.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.929+0000 7f79621f6640 1 -- 172.21.15.16:0/3348107146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f795c102970 msgr2=0x7f795c102d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:20.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.929+0000 7f79621f6640 1 --2- 172.21.15.16:0/3348107146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f795c102970 0x7f795c102d50 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f794c009900 tx=0x7f794c02bcd0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.929+0000 7f79621f6640 1 -- 172.21.15.16:0/3348107146 shutdown_connections 2024-03-28T22:16:20.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.929+0000 7f79621f6640 1 --2- 172.21.15.16:0/3348107146 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f795c10bd40 0x7f795c10e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.929+0000 7f79621f6640 1 --2- 172.21.15.16:0/3348107146 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f795c103290 0x7f795c10b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.929+0000 7f79621f6640 1 --2- 172.21.15.16:0/3348107146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f795c102970 0x7f795c102d50 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.929+0000 7f79621f6640 1 -- 172.21.15.16:0/3348107146 >> 172.21.15.16:0/3348107146 conn(0x7f795c077800 msgr2=0x7f795c077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:20.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.929+0000 7f79621f6640 1 -- 172.21.15.16:0/3348107146 shutdown_connections 2024-03-28T22:16:20.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.929+0000 7f79621f6640 1 -- 172.21.15.16:0/3348107146 wait complete. 2024-03-28T22:16:20.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.929+0000 7f79621f6640 1 Processor -- start 2024-03-28T22:16:20.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.929+0000 7f79621f6640 1 -- start start 2024-03-28T22:16:20.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.930+0000 7f79621f6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f795c102970 0x7f795c197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.930+0000 7f79621f6640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f795c103290 0x7f795c198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.930+0000 7f79621f6640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f795c10bd40 0x7f795c19c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.930+0000 7f79621f6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f795c111ad0 con 0x7f795c102970 2024-03-28T22:16:20.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.930+0000 7f79621f6640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f795c111950 con 0x7f795c10bd40 2024-03-28T22:16:20.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.930+0000 7f79621f6640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f795c111c50 con 0x7f795c103290 2024-03-28T22:16:20.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.930+0000 7f795b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f795c102970 0x7f795c197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.930+0000 7f795b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f795c102970 0x7f795c197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40246/0 (socket says 172.21.15.16:40246) 2024-03-28T22:16:20.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.930+0000 7f795b7fe640 1 -- 172.21.15.16:0/3285665417 learned_addr learned my addr 172.21.15.16:0/3285665417 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:20.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.930+0000 7f795bfff640 1 --2- 172.21.15.16:0/3285665417 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f795c10bd40 0x7f795c19c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.930+0000 7f795affd640 1 --2- 172.21.15.16:0/3285665417 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f795c103290 0x7f795c198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.930+0000 7f795b7fe640 1 -- 172.21.15.16:0/3285665417 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f795c103290 msgr2=0x7f795c198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:20.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.930+0000 7f795b7fe640 1 --2- 172.21.15.16:0/3285665417 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f795c103290 0x7f795c198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.930+0000 7f795b7fe640 1 -- 172.21.15.16:0/3285665417 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f795c10bd40 msgr2=0x7f795c19c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:20.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.930+0000 7f795b7fe640 1 --2- 172.21.15.16:0/3285665417 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f795c10bd40 0x7f795c19c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.930+0000 7f795b7fe640 1 -- 172.21.15.16:0/3285665417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f795c19cf90 con 0x7f795c102970 2024-03-28T22:16:20.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.930+0000 7f795bfff640 1 --2- 172.21.15.16:0/3285665417 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f795c10bd40 0x7f795c19c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:16:20.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.930+0000 7f795b7fe640 1 --2- 172.21.15.16:0/3285665417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f795c102970 0x7f795c197ff0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f794c005940 tx=0x7f794c02cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:20.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.930+0000 7f7958ff9640 1 -- 172.21.15.16:0/3285665417 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f794c041070 con 0x7f795c102970 2024-03-28T22:16:20.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.930+0000 7f7958ff9640 1 -- 172.21.15.16:0/3285665417 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f794c004450 con 0x7f795c102970 2024-03-28T22:16:20.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.930+0000 7f79621f6640 1 -- 172.21.15.16:0/3285665417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f795c19d220 con 0x7f795c102970 2024-03-28T22:16:20.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.930+0000 7f7958ff9640 1 -- 172.21.15.16:0/3285665417 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f794c003df0 con 0x7f795c102970 2024-03-28T22:16:20.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.930+0000 7f79621f6640 1 -- 172.21.15.16:0/3285665417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f795c1a4b10 con 0x7f795c102970 2024-03-28T22:16:20.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.933+0000 7f7958ff9640 1 -- 172.21.15.16:0/3285665417 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f794c044070 con 0x7f795c102970 2024-03-28T22:16:20.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.933+0000 7f7964dd6740 1 Processor -- start 2024-03-28T22:16:20.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.933+0000 7f7964dd6740 1 -- start start 2024-03-28T22:16:20.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.934+0000 7f7964dd6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f85af8d470 0x55f85af8d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.934+0000 7f7964dd6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f85af8dd90 0x55f85af96430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.934+0000 7f7964dd6740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f85af96970 0x55f85af98d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.934+0000 7f7964dd6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f85aef11b0 con 0x55f85af8dd90 2024-03-28T22:16:20.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.934+0000 7f7964dd6740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f85ae99c00 con 0x55f85af96970 2024-03-28T22:16:20.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.934+0000 7f7964dd6740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f85aeed250 con 0x55f85af8d470 2024-03-28T22:16:20.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.934+0000 7f795b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f85af8dd90 0x55f85af96430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.934+0000 7f7958ff9640 1 --2- 172.21.15.16:0/3285665417 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f79440a1ef0 0x7f79440a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.934+0000 7f795b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f85af8dd90 0x55f85af96430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40248/0 (socket says 172.21.15.16:40248) 2024-03-28T22:16:20.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.934+0000 7f795b7fe640 1 -- 172.21.15.16:0/1895612581 learned_addr learned my addr 172.21.15.16:0/1895612581 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:20.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.934+0000 7f795affd640 1 --2- 172.21.15.16:0/1895612581 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f85af96970 0x55f85af98d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.934+0000 7f795bfff640 1 --2- 172.21.15.16:0/1895612581 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f85af8d470 0x55f85af8d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.934+0000 7f795bfff640 1 --2- 172.21.15.16:0/3285665417 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f79440a1ef0 0x7f79440a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.934+0000 7f7958ff9640 1 -- 172.21.15.16:0/3285665417 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f794c0efc20 con 0x7f795c102970 2024-03-28T22:16:20.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.934+0000 7f795affd640 1 -- 172.21.15.16:0/1895612581 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f85af8d470 msgr2=0x55f85af8d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:20.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.934+0000 7f795affd640 1 --2- 172.21.15.16:0/1895612581 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f85af8d470 0x55f85af8d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.934+0000 7f795affd640 1 -- 172.21.15.16:0/1895612581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f85af8dd90 msgr2=0x55f85af96430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:20.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.934+0000 7f795affd640 1 --2- 172.21.15.16:0/1895612581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f85af8dd90 0x55f85af96430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.934+0000 7f795affd640 1 -- 172.21.15.16:0/1895612581 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f85aeb93d0 con 0x55f85af96970 2024-03-28T22:16:20.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.934+0000 7f795affd640 1 --2- 172.21.15.16:0/1895612581 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f85af96970 0x55f85af98d60 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7950009a60 tx=0x7f7950009f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35d56c2ec60e0d1b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:20.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.934+0000 7f795bfff640 1 --2- 172.21.15.16:0/3285665417 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f79440a1ef0 0x7f79440a43b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7954009800 tx=0x7f7954009550 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:20.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.934+0000 7f7941ffb640 1 -- 172.21.15.16:0/1895612581 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7950010cd0 con 0x55f85af96970 2024-03-28T22:16:20.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.935+0000 7f7941ffb640 1 -- 172.21.15.16:0/1895612581 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7950010e70 con 0x55f85af96970 2024-03-28T22:16:20.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.935+0000 7f7941ffb640 1 -- 172.21.15.16:0/1895612581 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79500128f0 con 0x55f85af96970 2024-03-28T22:16:20.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.935+0000 7f7964dd6740 1 -- 172.21.15.16:0/1895612581 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f85af96970 msgr2=0x55f85af98d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:20.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.935+0000 7f7964dd6740 1 --2- 172.21.15.16:0/1895612581 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f85af96970 0x55f85af98d60 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7950009a60 tx=0x7f7950009f20 comp rx=0 tx=0).stop 2024-03-28T22:16:20.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.935+0000 7f7964dd6740 1 -- 172.21.15.16:0/1895612581 shutdown_connections 2024-03-28T22:16:20.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.935+0000 7f7964dd6740 1 --2- 172.21.15.16:0/1895612581 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f85af96970 0x55f85af98d60 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.935+0000 7f7964dd6740 1 --2- 172.21.15.16:0/1895612581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f85af8dd90 0x55f85af96430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.935+0000 7f7964dd6740 1 --2- 172.21.15.16:0/1895612581 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f85af8d470 0x55f85af8d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.935+0000 7f7964dd6740 1 -- 172.21.15.16:0/1895612581 >> 172.21.15.16:0/1895612581 conn(0x55f85af87ba0 msgr2=0x55f85af87f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:20.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.935+0000 7f7964dd6740 1 -- 172.21.15.16:0/1895612581 shutdown_connections 2024-03-28T22:16:20.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.935+0000 7f7964dd6740 1 -- 172.21.15.16:0/1895612581 wait complete. 2024-03-28T22:16:20.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.936+0000 7f7964dd6740 1 -- start start 2024-03-28T22:16:20.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.936+0000 7f7964dd6740 1 Processor -- start 2024-03-28T22:16:20.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.936+0000 7f7964dd6740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f85af8d470 0x55f85af93950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.936+0000 7f7964dd6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f85af8dd90 0x55f85af93e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.936+0000 7f7964dd6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f85af96970 0x55f85af943d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.936+0000 7f7964dd6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f85af99760 con 0x55f85af8dd90 2024-03-28T22:16:20.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.936+0000 7f7964dd6740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f85ae8f470 con 0x55f85af8d470 2024-03-28T22:16:20.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.936+0000 7f7964dd6740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f85af998e0 con 0x55f85af96970 2024-03-28T22:16:20.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.936+0000 7f7964dd6740 10 client.0 _ll_register_callbacks cb 0x55f85aeaccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:16:20.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.936+0000 7f795affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f85af8dd90 0x55f85af93e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.936+0000 7f795b7fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f85af8d470 0x55f85af93950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.936+0000 7f795affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f85af8dd90 0x55f85af93e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40250/0 (socket says 172.21.15.16:40250) 2024-03-28T22:16:20.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.936+0000 7f795affd640 1 -- 172.21.15.16:0/1624888256 learned_addr learned my addr 172.21.15.16:0/1624888256 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:20.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.936+0000 7f795bfff640 1 --2- 172.21.15.16:0/1624888256 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f85af96970 0x55f85af943d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.936+0000 7f795b7fe640 1 -- 172.21.15.16:0/1624888256 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f85af96970 msgr2=0x55f85af943d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:20.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.936+0000 7f795b7fe640 1 --2- 172.21.15.16:0/1624888256 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f85af96970 0x55f85af943d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.936+0000 7f795b7fe640 1 -- 172.21.15.16:0/1624888256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f85af8dd90 msgr2=0x55f85af93e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:20.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.936+0000 7f795b7fe640 1 --2- 172.21.15.16:0/1624888256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f85af8dd90 0x55f85af93e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.936+0000 7f795b7fe640 1 -- 172.21.15.16:0/1624888256 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f85ac053f0 con 0x55f85af8d470 2024-03-28T22:16:20.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.936+0000 7f795affd640 1 --2- 172.21.15.16:0/1624888256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f85af8dd90 0x55f85af93e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:16:20.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.937+0000 7f795b7fe640 1 --2- 172.21.15.16:0/1624888256 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f85af8d470 0x55f85af93950 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f794c0bc900 tx=0x7f794c009620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a95c8f4ed219531a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:20.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.937+0000 7f793affd640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-03-28T22:16:20.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.937+0000 7f793affd640 1 -- 172.21.15.16:0/1624888256 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f794c04b020 con 0x55f85af8d470 2024-03-28T22:16:20.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.937+0000 7f793affd640 1 -- 172.21.15.16:0/1624888256 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f794c0b8db0 con 0x55f85af8d470 2024-03-28T22:16:20.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.937+0000 7f7964dd6740 1 -- 172.21.15.16:0/1624888256 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7950010e70 con 0x55f85af8d470 2024-03-28T22:16:20.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.937+0000 7f793affd640 1 -- 172.21.15.16:0/1624888256 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f794c0d5050 con 0x55f85af8d470 2024-03-28T22:16:20.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.937+0000 7f7964dd6740 10 client.26600 fetch_fsmap learned FSMap version 27 2024-03-28T22:16:20.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.937+0000 7f7964dd6740 1 -- 172.21.15.16:0/1624888256 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f85accf7d0 con 0x55f85af8d470 2024-03-28T22:16:20.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.937+0000 7f793affd640 1 -- 172.21.15.16:0/1624888256 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7950010e70 con 0x55f85af8d470 2024-03-28T22:16:20.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.937+0000 7f793affd640 1 -- 172.21.15.16:0/1624888256 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f794c0dc0e0 con 0x55f85af8d470 2024-03-28T22:16:20.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.937+0000 7f7964dd6740 10 client.26600 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:16:20.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.937+0000 7f7964dd6740 10 client.26600 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-28T22:16:20.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.937+0000 7f7964dd6740 20 client.26600 populate_metadata read hostname 'smithi016' 2024-03-28T22:16:20.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.937+0000 7f7964dd6740 1 --2- 172.21.15.16:0/1624888256 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55f85afa6840 0x55f85afa8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.937+0000 7f7964dd6740 4 client.26600 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:16:20.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.937+0000 7f7964dd6740 1 -- 172.21.15.16:0/1624888256 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f85aefe6f0 con 0x55f85afa6840 2024-03-28T22:16:20.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.938+0000 7f795affd640 1 --2- 172.21.15.16:0/1624888256 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55f85afa6840 0x55f85afa8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.938+0000 7f793affd640 10 client.26600 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:16:20.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.938+0000 7f795affd640 1 --2- 172.21.15.16:0/1624888256 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55f85afa6840 0x55f85afa8d00 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=760ac0549f889740 server_cookie=48b1c4927afb1b03 in_seq=0 out_seq=0 2024-03-28T22:16:20.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.939+0000 7f793affd640 1 -- 172.21.15.16:0/1624888256 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f85aefe6f0 con 0x55f85afa6840 2024-03-28T22:16:20.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.939+0000 7f793affd640 10 client.26600 handle_command_reply: tid=0 2024-03-28T22:16:20.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.939+0000 7f7964dd6740 1 client.26600 shutdown 2024-03-28T22:16:20.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.939+0000 7f7964dd6740 1 -- 172.21.15.16:0/1624888256 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f85af8d470 msgr2=0x55f85af93950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:20.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.939+0000 7f7964dd6740 1 --2- 172.21.15.16:0/1624888256 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f85af8d470 0x55f85af93950 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f794c0bc900 tx=0x7f794c009620 comp rx=0 tx=0).stop 2024-03-28T22:16:20.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.939+0000 7f7964dd6740 1 -- 172.21.15.16:0/1624888256 shutdown_connections 2024-03-28T22:16:20.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.939+0000 7f7964dd6740 1 --2- 172.21.15.16:0/1624888256 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55f85afa6840 0x55f85afa8d00 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.939+0000 7f7964dd6740 1 --2- 172.21.15.16:0/1624888256 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f85af96970 0x55f85af943d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.939+0000 7f793affd640 0 client.26600 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:16:20.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.939+0000 7f7964dd6740 1 --2- 172.21.15.16:0/1624888256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f85af8dd90 0x55f85af93e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.939+0000 7f7964dd6740 1 --2- 172.21.15.16:0/1624888256 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f85af8d470 0x55f85af93950 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.939+0000 7f7964dd6740 1 -- 172.21.15.16:0/1624888256 >> 172.21.15.16:0/1624888256 conn(0x55f85af87ba0 msgr2=0x55f85af898f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:20.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.939+0000 7f7964dd6740 1 -- 172.21.15.16:0/1624888256 shutdown_connections 2024-03-28T22:16:20.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.939+0000 7f7964dd6740 1 -- 172.21.15.16:0/1624888256 wait complete. 2024-03-28T22:16:20.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.940+0000 7f7964dd6740 20 client.26600 trim_cache size 0 max 16384 2024-03-28T22:16:20.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.953+0000 7f7964dd6740 1 Processor -- start 2024-03-28T22:16:20.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.953+0000 7f7964dd6740 1 -- start start 2024-03-28T22:16:20.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.953+0000 7f7964dd6740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f85af8d470 0x55f85af96080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.953+0000 7f7964dd6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f85af8dd90 0x55f85afa2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.953+0000 7f7964dd6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f85af96970 0x55f85afae5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.953+0000 7f7964dd6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f795c1afc30 con 0x55f85af96970 2024-03-28T22:16:20.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.953+0000 7f7964dd6740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f795c1ad500 con 0x55f85af8d470 2024-03-28T22:16:20.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.953+0000 7f7964dd6740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f795c1ab350 con 0x55f85af8dd90 2024-03-28T22:16:20.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.953+0000 7f795bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f85af96970 0x55f85afae5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.953+0000 7f795bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f85af96970 0x55f85afae5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40264/0 (socket says 172.21.15.16:40264) 2024-03-28T22:16:20.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.953+0000 7f795b7fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f85af8d470 0x55f85af96080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.953+0000 7f795bfff640 1 -- 172.21.15.16:0/238676804 learned_addr learned my addr 172.21.15.16:0/238676804 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:20.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.953+0000 7f795bfff640 1 -- 172.21.15.16:0/238676804 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f85af8dd90 msgr2=0x55f85afa2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:20.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.953+0000 7f795affd640 1 --2- 172.21.15.16:0/238676804 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f85af8dd90 0x55f85afa2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.953+0000 7f795bfff640 1 --2- 172.21.15.16:0/238676804 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f85af8dd90 0x55f85afa2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.953+0000 7f795bfff640 1 -- 172.21.15.16:0/238676804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f85af8d470 msgr2=0x55f85af96080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:20.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.953+0000 7f795bfff640 1 --2- 172.21.15.16:0/238676804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f85af8d470 0x55f85af96080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.953+0000 7f795bfff640 1 -- 172.21.15.16:0/238676804 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f85af8dbc0 con 0x55f85af96970 2024-03-28T22:16:20.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.953+0000 7f795b7fe640 1 --2- 172.21.15.16:0/238676804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f85af8d470 0x55f85af96080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:16:20.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.953+0000 7f795affd640 1 --2- 172.21.15.16:0/238676804 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f85af8dd90 0x55f85afa2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:16:20.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.954+0000 7f795bfff640 1 --2- 172.21.15.16:0/238676804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f85af96970 0x55f85afae5d0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f7954012df0 tx=0x7f795400c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c30d62143b11ee58 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:20.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.954+0000 7f79417fa640 1 -- 172.21.15.16:0/238676804 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795401e970 con 0x55f85af96970 2024-03-28T22:16:20.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.954+0000 7f79417fa640 1 -- 172.21.15.16:0/238676804 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f795402d070 con 0x55f85af96970 2024-03-28T22:16:20.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.954+0000 7f79417fa640 1 -- 172.21.15.16:0/238676804 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79540103d0 con 0x55f85af96970 2024-03-28T22:16:20.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.954+0000 7f7964dd6740 1 -- 172.21.15.16:0/238676804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f85af96970 msgr2=0x55f85afae5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:20.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.954+0000 7f7964dd6740 1 --2- 172.21.15.16:0/238676804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f85af96970 0x55f85afae5d0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f7954012df0 tx=0x7f795400c000 comp rx=0 tx=0).stop 2024-03-28T22:16:20.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.954+0000 7f7964dd6740 1 -- 172.21.15.16:0/238676804 shutdown_connections 2024-03-28T22:16:20.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.954+0000 7f7964dd6740 1 --2- 172.21.15.16:0/238676804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f85af96970 0x55f85afae5d0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.954+0000 7f7964dd6740 1 --2- 172.21.15.16:0/238676804 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f85af8dd90 0x55f85afa2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.954+0000 7f7964dd6740 1 --2- 172.21.15.16:0/238676804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f85af8d470 0x55f85af96080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.954+0000 7f7964dd6740 1 -- 172.21.15.16:0/238676804 >> 172.21.15.16:0/238676804 conn(0x55f85af87ba0 msgr2=0x55f85af86d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:20.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.954+0000 7f7964dd6740 1 -- 172.21.15.16:0/238676804 shutdown_connections 2024-03-28T22:16:20.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.954+0000 7f7964dd6740 1 -- 172.21.15.16:0/238676804 wait complete. 2024-03-28T22:16:20.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.955+0000 7f7964dd6740 1 -- start start 2024-03-28T22:16:20.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.955+0000 7f7964dd6740 1 Processor -- start 2024-03-28T22:16:20.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.955+0000 7f7964dd6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f85af8d470 0x55f85af9ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.955+0000 7f7964dd6740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f85af8dd90 0x55f85af902a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.955+0000 7f7964dd6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f85af96970 0x55f85af90800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.955+0000 7f7964dd6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f795c1ab9c0 con 0x55f85af8d470 2024-03-28T22:16:20.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.955+0000 7f7964dd6740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f795c1ace30 con 0x55f85af8dd90 2024-03-28T22:16:20.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.955+0000 7f7964dd6740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f795c1ac090 con 0x55f85af96970 2024-03-28T22:16:20.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.955+0000 7f7964dd6740 10 client.0 _ll_register_callbacks cb 0x7f794c005770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:16:20.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.955+0000 7f795b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f85af8d470 0x55f85af9ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.955+0000 7f795b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f85af8d470 0x55f85af9ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40274/0 (socket says 172.21.15.16:40274) 2024-03-28T22:16:20.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.955+0000 7f795b7fe640 1 -- 172.21.15.16:0/1784231583 learned_addr learned my addr 172.21.15.16:0/1784231583 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:20.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.955+0000 7f795affd640 1 --2- 172.21.15.16:0/1784231583 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f85af8dd90 0x55f85af902a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.955+0000 7f795b7fe640 1 -- 172.21.15.16:0/1784231583 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f85af96970 msgr2=0x55f85af90800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:20.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.955+0000 7f795b7fe640 1 --2- 172.21.15.16:0/1784231583 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f85af96970 0x55f85af90800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.955+0000 7f795b7fe640 1 -- 172.21.15.16:0/1784231583 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f85af8dd90 msgr2=0x55f85af902a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:20.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.955+0000 7f795b7fe640 1 --2- 172.21.15.16:0/1784231583 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f85af8dd90 0x55f85af902a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.955+0000 7f795b7fe640 1 -- 172.21.15.16:0/1784231583 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f85af91030 con 0x55f85af8d470 2024-03-28T22:16:20.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.955+0000 7f795b7fe640 1 --2- 172.21.15.16:0/1784231583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f85af8d470 0x55f85af9ee20 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f794c006010 tx=0x7f794c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c28f961fb76910c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:20.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.955+0000 7f793affd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:16:20.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.956+0000 7f793affd640 1 -- 172.21.15.16:0/1784231583 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f794c0d9030 con 0x55f85af8d470 2024-03-28T22:16:20.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.956+0000 7f793affd640 1 -- 172.21.15.16:0/1784231583 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f794c0ed550 con 0x55f85af8d470 2024-03-28T22:16:20.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.956+0000 7f7964dd6740 1 -- 172.21.15.16:0/1784231583 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f795402d070 con 0x55f85af8d470 2024-03-28T22:16:20.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.956+0000 7f793affd640 1 -- 172.21.15.16:0/1784231583 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f794c0a0050 con 0x55f85af8d470 2024-03-28T22:16:20.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.956+0000 7f7964dd6740 10 client.16704 fetch_fsmap learned FSMap version 27 2024-03-28T22:16:20.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.956+0000 7f7964dd6740 1 -- 172.21.15.16:0/1784231583 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f85b00cc00 con 0x55f85af8d470 2024-03-28T22:16:20.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.956+0000 7f793affd640 1 -- 172.21.15.16:0/1784231583 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f795402d070 con 0x55f85af8d470 2024-03-28T22:16:20.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.956+0000 7f793affd640 1 -- 172.21.15.16:0/1784231583 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f794c0bbb90 con 0x55f85af8d470 2024-03-28T22:16:20.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.956+0000 7f7964dd6740 10 client.16704 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:16:20.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.956+0000 7f7964dd6740 10 client.16704 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-28T22:16:20.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.956+0000 7f7964dd6740 20 client.16704 populate_metadata read hostname 'smithi016' 2024-03-28T22:16:20.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.956+0000 7f7964dd6740 1 --2- 172.21.15.16:0/1784231583 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55f85b00d130 0x55f85b00f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:20.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.956+0000 7f7964dd6740 4 client.16704 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:16:20.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.956+0000 7f7964dd6740 1 -- 172.21.15.16:0/1784231583 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f85aef8a40 con 0x55f85b00d130 2024-03-28T22:16:20.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.956+0000 7f795affd640 1 --2- 172.21.15.16:0/1784231583 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55f85b00d130 0x55f85b00f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:20.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.957+0000 7f793affd640 10 client.16704 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:16:20.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.957+0000 7f795affd640 1 --2- 172.21.15.16:0/1784231583 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55f85b00d130 0x55f85b00f520 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=13064ec1ba3feb server_cookie=bbd88b9b9c0bfcf8 in_seq=0 out_seq=0 2024-03-28T22:16:20.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.957+0000 7f793affd640 1 -- 172.21.15.16:0/1784231583 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f85aef8a40 con 0x55f85b00d130 2024-03-28T22:16:20.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.957+0000 7f793affd640 10 client.16704 handle_command_reply: tid=0 2024-03-28T22:16:20.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.957+0000 7f7964dd6740 1 client.16704 shutdown 2024-03-28T22:16:20.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.957+0000 7f7964dd6740 1 -- 172.21.15.16:0/1784231583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f85af8d470 msgr2=0x55f85af9ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:20.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.957+0000 7f7964dd6740 1 --2- 172.21.15.16:0/1784231583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f85af8d470 0x55f85af9ee20 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f794c006010 tx=0x7f794c000bc0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.957+0000 7f7964dd6740 1 -- 172.21.15.16:0/1784231583 shutdown_connections 2024-03-28T22:16:20.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.957+0000 7f7964dd6740 1 --2- 172.21.15.16:0/1784231583 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55f85b00d130 0x55f85b00f520 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.957+0000 7f7964dd6740 1 --2- 172.21.15.16:0/1784231583 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f85af96970 0x55f85af90800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.957+0000 7f7964dd6740 1 --2- 172.21.15.16:0/1784231583 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f85af8dd90 0x55f85af902a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.957+0000 7f793affd640 0 client.16704 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:16:20.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.957+0000 7f7964dd6740 1 --2- 172.21.15.16:0/1784231583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f85af8d470 0x55f85af9ee20 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.957+0000 7f7964dd6740 1 -- 172.21.15.16:0/1784231583 >> 172.21.15.16:0/1784231583 conn(0x55f85af87ba0 msgr2=0x55f85af864b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:20.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.957+0000 7f7964dd6740 1 -- 172.21.15.16:0/1784231583 shutdown_connections 2024-03-28T22:16:20.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.957+0000 7f7964dd6740 1 -- 172.21.15.16:0/1784231583 wait complete. 2024-03-28T22:16:20.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.957+0000 7f7964dd6740 20 client.16704 trim_cache size 0 max 16384 2024-03-28T22:16:20.980 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:16:20.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.958+0000 7f79621f6640 1 -- 172.21.15.16:0/3285665417 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f79440a1ef0 msgr2=0x7f79440a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:20.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.958+0000 7f79621f6640 1 --2- 172.21.15.16:0/3285665417 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f79440a1ef0 0x7f79440a43b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7954009800 tx=0x7f7954009550 comp rx=0 tx=0).stop 2024-03-28T22:16:20.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.958+0000 7f79621f6640 1 -- 172.21.15.16:0/3285665417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f795c102970 msgr2=0x7f795c197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:20.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.958+0000 7f79621f6640 1 --2- 172.21.15.16:0/3285665417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f795c102970 0x7f795c197ff0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f794c005940 tx=0x7f794c02cc40 comp rx=0 tx=0).stop 2024-03-28T22:16:20.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.958+0000 7f79621f6640 1 -- 172.21.15.16:0/3285665417 shutdown_connections 2024-03-28T22:16:20.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.958+0000 7f79621f6640 1 --2- 172.21.15.16:0/3285665417 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f79440a1ef0 0x7f79440a43b0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.958+0000 7f79621f6640 1 --2- 172.21.15.16:0/3285665417 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f795c10bd40 0x7f795c19c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.958+0000 7f79621f6640 1 --2- 172.21.15.16:0/3285665417 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f795c103290 0x7f795c198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.958+0000 7f79621f6640 1 --2- 172.21.15.16:0/3285665417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f795c102970 0x7f795c197ff0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:20.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.958+0000 7f79621f6640 1 -- 172.21.15.16:0/3285665417 >> 172.21.15.16:0/3285665417 conn(0x7f795c077800 msgr2=0x7f795c0fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:20.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.958+0000 7f79621f6640 1 -- 172.21.15.16:0/3285665417 shutdown_connections 2024-03-28T22:16:20.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:20.958+0000 7f79621f6640 1 -- 172.21.15.16:0/3285665417 wait complete. 2024-03-28T22:16:20.983 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:16:21.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.086+0000 7ff222502640 1 -- 172.21.15.16:0/3918830140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff21c1076f0 msgr2=0x7ff21c107ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:21.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.086+0000 7ff222502640 1 --2- 172.21.15.16:0/3918830140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff21c1076f0 0x7ff21c107ad0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff20c009900 tx=0x7ff20c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:16:21.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.086+0000 7ff222502640 1 -- 172.21.15.16:0/3918830140 shutdown_connections 2024-03-28T22:16:21.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.086+0000 7ff222502640 1 --2- 172.21.15.16:0/3918830140 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff21c100560 0x7ff21c10e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.086+0000 7ff222502640 1 --2- 172.21.15.16:0/3918830140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff21c0ffbc0 0x7ff21c100020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.086+0000 7ff222502640 1 --2- 172.21.15.16:0/3918830140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff21c1076f0 0x7ff21c107ad0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.086+0000 7ff222502640 1 -- 172.21.15.16:0/3918830140 >> 172.21.15.16:0/3918830140 conn(0x7ff21c076b50 msgr2=0x7ff21c0fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:21.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.086+0000 7ff222502640 1 -- 172.21.15.16:0/3918830140 shutdown_connections 2024-03-28T22:16:21.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.086+0000 7ff222502640 1 -- 172.21.15.16:0/3918830140 wait complete. 2024-03-28T22:16:21.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.087+0000 7ff222502640 1 Processor -- start 2024-03-28T22:16:21.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.087+0000 7ff222502640 1 -- start start 2024-03-28T22:16:21.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.087+0000 7ff222502640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff21c0ffbc0 0x7ff21c197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:21.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.087+0000 7ff222502640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff21c100560 0x7ff21c198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:21.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.087+0000 7ff222502640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff21c1076f0 0x7ff21c19c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:21.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.087+0000 7ff222502640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff21c111b80 con 0x7ff21c1076f0 2024-03-28T22:16:21.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.087+0000 7ff222502640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff21c111a00 con 0x7ff21c100560 2024-03-28T22:16:21.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.087+0000 7ff222502640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff21c111d00 con 0x7ff21c0ffbc0 2024-03-28T22:16:21.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.087+0000 7ff220a9f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff21c1076f0 0x7ff21c19c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:21.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.087+0000 7ff21b7fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff21c100560 0x7ff21c198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:21.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.087+0000 7ff21bfff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff21c0ffbc0 0x7ff21c197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:21.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.087+0000 7ff220a9f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff21c1076f0 0x7ff21c19c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40292/0 (socket says 172.21.15.16:40292) 2024-03-28T22:16:21.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.087+0000 7ff220a9f640 1 -- 172.21.15.16:0/2637598121 learned_addr learned my addr 172.21.15.16:0/2637598121 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:21.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.087+0000 7ff220a9f640 1 -- 172.21.15.16:0/2637598121 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff21c0ffbc0 msgr2=0x7ff21c197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:21.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.087+0000 7ff220a9f640 1 --2- 172.21.15.16:0/2637598121 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff21c0ffbc0 0x7ff21c197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.087+0000 7ff220a9f640 1 -- 172.21.15.16:0/2637598121 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff21c100560 msgr2=0x7ff21c198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:21.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.087+0000 7ff220a9f640 1 --2- 172.21.15.16:0/2637598121 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff21c100560 0x7ff21c198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.087+0000 7ff220a9f640 1 -- 172.21.15.16:0/2637598121 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff21c19cf10 con 0x7ff21c1076f0 2024-03-28T22:16:21.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.088+0000 7ff220a9f640 1 --2- 172.21.15.16:0/2637598121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff21c1076f0 0x7ff21c19c7e0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7ff21400cce0 tx=0x7ff21400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:21.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.088+0000 7ff2197fa640 1 -- 172.21.15.16:0/2637598121 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff21400f450 con 0x7ff21c1076f0 2024-03-28T22:16:21.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.088+0000 7ff222502640 1 -- 172.21.15.16:0/2637598121 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff21c19d200 con 0x7ff21c1076f0 2024-03-28T22:16:21.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.088+0000 7ff2197fa640 1 -- 172.21.15.16:0/2637598121 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff21400fad0 con 0x7ff21c1076f0 2024-03-28T22:16:21.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.088+0000 7ff2197fa640 1 -- 172.21.15.16:0/2637598121 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff214018c10 con 0x7ff21c1076f0 2024-03-28T22:16:21.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.088+0000 7ff222502640 1 -- 172.21.15.16:0/2637598121 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff21c1a4a90 con 0x7ff21c1076f0 2024-03-28T22:16:21.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.090+0000 7ff2197fa640 1 -- 172.21.15.16:0/2637598121 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7ff21400f5f0 con 0x7ff21c1076f0 2024-03-28T22:16:21.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.090+0000 7ff2251c0740 1 Processor -- start 2024-03-28T22:16:21.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.090+0000 7ff2251c0740 1 -- start start 2024-03-28T22:16:21.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.091+0000 7ff2197fa640 1 --2- 172.21.15.16:0/2637598121 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff2040a1ef0 0x7ff2040a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:21.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.091+0000 7ff2251c0740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643230b3470 0x5643230b3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:21.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.091+0000 7ff2251c0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643230b3d90 0x5643230bc430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:21.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.091+0000 7ff2251c0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643230bc970 0x5643230bed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:21.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.091+0000 7ff2251c0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5643230171b0 con 0x5643230bc970 2024-03-28T22:16:21.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.091+0000 7ff2251c0740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x564322fbfc00 con 0x5643230b3470 2024-03-28T22:16:21.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.091+0000 7ff2251c0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x564323013250 con 0x5643230b3d90 2024-03-28T22:16:21.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.091+0000 7ff21b7fe640 1 --2- 172.21.15.16:0/2637598121 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff2040a1ef0 0x7ff2040a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:21.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.091+0000 7ff2197fa640 1 -- 172.21.15.16:0/2637598121 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7ff2140ce770 con 0x7ff21c1076f0 2024-03-28T22:16:21.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.091+0000 7ff220a9f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643230bc970 0x5643230bed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:21.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.091+0000 7ff220a9f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643230bc970 0x5643230bed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40308/0 (socket says 172.21.15.16:40308) 2024-03-28T22:16:21.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.091+0000 7ff21bfff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643230b3470 0x5643230b3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:21.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.091+0000 7ff220a9f640 1 -- 172.21.15.16:0/1237102922 learned_addr learned my addr 172.21.15.16:0/1237102922 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:21.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.091+0000 7ff21b7fe640 1 --2- 172.21.15.16:0/1237102922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643230b3d90 0x5643230bc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:21.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.091+0000 7ff220a9f640 1 -- 172.21.15.16:0/1237102922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643230b3d90 msgr2=0x5643230bc430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:21.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.091+0000 7ff220a9f640 1 --2- 172.21.15.16:0/1237102922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643230b3d90 0x5643230bc430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.091+0000 7ff220a9f640 1 -- 172.21.15.16:0/1237102922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643230b3470 msgr2=0x5643230b3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:21.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.091+0000 7ff220a9f640 1 --2- 172.21.15.16:0/1237102922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643230b3470 0x5643230b3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.091+0000 7ff220a9f640 1 -- 172.21.15.16:0/1237102922 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564322fdf3d0 con 0x5643230bc970 2024-03-28T22:16:21.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.091+0000 7ff21b7fe640 1 --2- 172.21.15.16:0/2637598121 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff2040a1ef0 0x7ff2040a43b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff21c106c80 tx=0x7ff210013830 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:21.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.091+0000 7ff220a9f640 1 --2- 172.21.15.16:0/1237102922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643230bc970 0x5643230bed60 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7ff2140975e0 tx=0x7ff21409ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4004cefff751dc9e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:21.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.091+0000 7ff2027fc640 1 -- 172.21.15.16:0/1237102922 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff21409b180 con 0x5643230bc970 2024-03-28T22:16:21.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.091+0000 7ff2027fc640 1 -- 172.21.15.16:0/1237102922 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff21409f2d0 con 0x5643230bc970 2024-03-28T22:16:21.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.092+0000 7ff2027fc640 1 -- 172.21.15.16:0/1237102922 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2140cc160 con 0x5643230bc970 2024-03-28T22:16:21.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.092+0000 7ff2251c0740 1 -- 172.21.15.16:0/1237102922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643230bc970 msgr2=0x5643230bed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:21.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.092+0000 7ff2251c0740 1 --2- 172.21.15.16:0/1237102922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643230bc970 0x5643230bed60 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7ff2140975e0 tx=0x7ff21409ad80 comp rx=0 tx=0).stop 2024-03-28T22:16:21.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.092+0000 7ff2251c0740 1 -- 172.21.15.16:0/1237102922 shutdown_connections 2024-03-28T22:16:21.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.092+0000 7ff2251c0740 1 --2- 172.21.15.16:0/1237102922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643230bc970 0x5643230bed60 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.092+0000 7ff2251c0740 1 --2- 172.21.15.16:0/1237102922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643230b3d90 0x5643230bc430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.092+0000 7ff2251c0740 1 --2- 172.21.15.16:0/1237102922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643230b3470 0x5643230b3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.092+0000 7ff2251c0740 1 -- 172.21.15.16:0/1237102922 >> 172.21.15.16:0/1237102922 conn(0x5643230adba0 msgr2=0x5643230adf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:21.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.092+0000 7ff2251c0740 1 -- 172.21.15.16:0/1237102922 shutdown_connections 2024-03-28T22:16:21.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.092+0000 7ff2251c0740 1 -- 172.21.15.16:0/1237102922 wait complete. 2024-03-28T22:16:21.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.092+0000 7ff2251c0740 1 -- start start 2024-03-28T22:16:21.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.093+0000 7ff2251c0740 1 Processor -- start 2024-03-28T22:16:21.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.093+0000 7ff2251c0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643230b3470 0x5643230b9990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:21.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.093+0000 7ff2251c0740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643230b3d90 0x5643230b9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:21.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.093+0000 7ff2251c0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643230bc970 0x5643230ba410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:21.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.093+0000 7ff2251c0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5643230bf760 con 0x5643230b3470 2024-03-28T22:16:21.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.093+0000 7ff2251c0740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x564322fb5470 con 0x5643230b3d90 2024-03-28T22:16:21.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.093+0000 7ff2251c0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5643230bf8e0 con 0x5643230bc970 2024-03-28T22:16:21.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.093+0000 7ff2251c0740 10 client.0 _ll_register_callbacks cb 0x564322fd2cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:16:21.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.093+0000 7ff21bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643230b3470 0x5643230b9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:21.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.093+0000 7ff21bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643230b3470 0x5643230b9990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40314/0 (socket says 172.21.15.16:40314) 2024-03-28T22:16:21.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.093+0000 7ff21bfff640 1 -- 172.21.15.16:0/4082455909 learned_addr learned my addr 172.21.15.16:0/4082455909 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:21.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.093+0000 7ff21b7fe640 1 --2- 172.21.15.16:0/4082455909 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643230b3d90 0x5643230b9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:21.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.093+0000 7ff220a9f640 1 --2- 172.21.15.16:0/4082455909 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643230bc970 0x5643230ba410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:21.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.093+0000 7ff21bfff640 1 -- 172.21.15.16:0/4082455909 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643230bc970 msgr2=0x5643230ba410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:21.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.093+0000 7ff21bfff640 1 --2- 172.21.15.16:0/4082455909 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643230bc970 0x5643230ba410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.093+0000 7ff21bfff640 1 -- 172.21.15.16:0/4082455909 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643230b3d90 msgr2=0x5643230b9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:21.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.093+0000 7ff21bfff640 1 --2- 172.21.15.16:0/4082455909 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643230b3d90 0x5643230b9ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.093+0000 7ff21bfff640 1 -- 172.21.15.16:0/4082455909 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564322d2b3f0 con 0x5643230b3470 2024-03-28T22:16:21.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.094+0000 7ff21bfff640 1 --2- 172.21.15.16:0/4082455909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643230b3470 0x5643230b9990 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7ff20c009230 tx=0x7ff20c02c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41e875cc017abb2c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:21.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.094+0000 7ff1fb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:16:21.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.094+0000 7ff1fb7fe640 1 -- 172.21.15.16:0/4082455909 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff20c0046c0 con 0x5643230b3470 2024-03-28T22:16:21.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.094+0000 7ff1fb7fe640 1 -- 172.21.15.16:0/4082455909 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7ff20c004860 con 0x5643230b3470 2024-03-28T22:16:21.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.094+0000 7ff2251c0740 1 -- 172.21.15.16:0/4082455909 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff21409f2d0 con 0x5643230b3470 2024-03-28T22:16:21.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.094+0000 7ff1fb7fe640 1 -- 172.21.15.16:0/4082455909 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff20c004a60 con 0x5643230b3470 2024-03-28T22:16:21.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.094+0000 7ff2251c0740 10 client.16716 fetch_fsmap learned FSMap version 27 2024-03-28T22:16:21.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.094+0000 7ff2251c0740 1 -- 172.21.15.16:0/4082455909 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564322df57d0 con 0x5643230b3470 2024-03-28T22:16:21.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.094+0000 7ff1fb7fe640 1 -- 172.21.15.16:0/4082455909 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff21409f2d0 con 0x5643230b3470 2024-03-28T22:16:21.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.094+0000 7ff1fb7fe640 1 -- 172.21.15.16:0/4082455909 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7ff20c04e070 con 0x5643230b3470 2024-03-28T22:16:21.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.094+0000 7ff2251c0740 10 client.16716 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:16:21.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.094+0000 7ff2251c0740 10 client.16716 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T22:16:21.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.094+0000 7ff2251c0740 20 client.16716 populate_metadata read hostname 'smithi016' 2024-03-28T22:16:21.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.094+0000 7ff2251c0740 1 --2- 172.21.15.16:0/4082455909 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5643230cc910 0x5643230ceda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:21.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.094+0000 7ff2251c0740 4 client.16716 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:16:21.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.094+0000 7ff2251c0740 1 -- 172.21.15.16:0/4082455909 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5643230246f0 con 0x5643230cc910 2024-03-28T22:16:21.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.094+0000 7ff21b7fe640 1 --2- 172.21.15.16:0/4082455909 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5643230cc910 0x5643230ceda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:21.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.095+0000 7ff1fb7fe640 10 client.16716 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:16:21.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.095+0000 7ff21b7fe640 1 --2- 172.21.15.16:0/4082455909 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5643230cc910 0x5643230ceda0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c75846e14bccfd01 server_cookie=7951bcbde5378d9c in_seq=0 out_seq=0 2024-03-28T22:16:21.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.096+0000 7ff1fb7fe640 1 -- 172.21.15.16:0/4082455909 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5643230246f0 con 0x5643230cc910 2024-03-28T22:16:21.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.096+0000 7ff1fb7fe640 10 client.16716 handle_command_reply: tid=0 2024-03-28T22:16:21.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.096+0000 7ff2251c0740 1 client.16716 shutdown 2024-03-28T22:16:21.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.096+0000 7ff2251c0740 1 -- 172.21.15.16:0/4082455909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643230b3470 msgr2=0x5643230b9990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:21.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.096+0000 7ff2251c0740 1 --2- 172.21.15.16:0/4082455909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643230b3470 0x5643230b9990 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7ff20c009230 tx=0x7ff20c02c7e0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.096+0000 7ff2251c0740 1 -- 172.21.15.16:0/4082455909 shutdown_connections 2024-03-28T22:16:21.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.096+0000 7ff2251c0740 1 --2- 172.21.15.16:0/4082455909 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5643230cc910 0x5643230ceda0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.096+0000 7ff2251c0740 1 --2- 172.21.15.16:0/4082455909 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643230bc970 0x5643230ba410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.096+0000 7ff2251c0740 1 --2- 172.21.15.16:0/4082455909 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643230b3d90 0x5643230b9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.096+0000 7ff1fb7fe640 0 client.16716 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:16:21.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.096+0000 7ff2251c0740 1 --2- 172.21.15.16:0/4082455909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643230b3470 0x5643230b9990 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.096+0000 7ff2251c0740 1 -- 172.21.15.16:0/4082455909 >> 172.21.15.16:0/4082455909 conn(0x5643230adba0 msgr2=0x5643230af8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:21.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.096+0000 7ff2251c0740 1 -- 172.21.15.16:0/4082455909 shutdown_connections 2024-03-28T22:16:21.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.096+0000 7ff2251c0740 1 -- 172.21.15.16:0/4082455909 wait complete. 2024-03-28T22:16:21.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.097+0000 7ff2251c0740 20 client.16716 trim_cache size 0 max 16384 2024-03-28T22:16:21.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.110+0000 7ff2251c0740 1 Processor -- start 2024-03-28T22:16:21.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.110+0000 7ff2251c0740 1 -- start start 2024-03-28T22:16:21.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.110+0000 7ff2251c0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643230b3470 0x5643230c8530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:21.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.110+0000 7ff2251c0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643230b3d90 0x5643230acbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:21.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.110+0000 7ff2251c0740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643230bc970 0x5643230d4670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:21.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.110+0000 7ff2251c0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff21c1afa80 con 0x5643230b3d90 2024-03-28T22:16:21.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.110+0000 7ff2251c0740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff21c1ad350 con 0x5643230bc970 2024-03-28T22:16:21.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.110+0000 7ff2251c0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff21c1ab1a0 con 0x5643230b3470 2024-03-28T22:16:21.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.110+0000 7ff21b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643230b3d90 0x5643230acbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:21.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.110+0000 7ff21b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643230b3d90 0x5643230acbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40328/0 (socket says 172.21.15.16:40328) 2024-03-28T22:16:21.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.110+0000 7ff21b7fe640 1 -- 172.21.15.16:0/2432846477 learned_addr learned my addr 172.21.15.16:0/2432846477 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:21.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.110+0000 7ff220a9f640 1 --2- 172.21.15.16:0/2432846477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643230bc970 0x5643230d4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:21.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.110+0000 7ff21b7fe640 1 -- 172.21.15.16:0/2432846477 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643230b3470 msgr2=0x5643230c8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:21.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.110+0000 7ff21b7fe640 1 --2- 172.21.15.16:0/2432846477 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643230b3470 0x5643230c8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.110+0000 7ff21b7fe640 1 -- 172.21.15.16:0/2432846477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643230bc970 msgr2=0x5643230d4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:21.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.110+0000 7ff21b7fe640 1 --2- 172.21.15.16:0/2432846477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643230bc970 0x5643230d4670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.110+0000 7ff21b7fe640 1 -- 172.21.15.16:0/2432846477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643230bed60 con 0x5643230b3d90 2024-03-28T22:16:21.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.110+0000 7ff21b7fe640 1 --2- 172.21.15.16:0/2432846477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643230b3d90 0x5643230acbc0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x56432309efb0 tx=0x7ff210012040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6613c08d9d77267 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:21.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.110+0000 7ff201ffb640 1 -- 172.21.15.16:0/2432846477 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff210004380 con 0x5643230b3d90 2024-03-28T22:16:21.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.110+0000 7ff201ffb640 1 -- 172.21.15.16:0/2432846477 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2100262e0 con 0x5643230b3d90 2024-03-28T22:16:21.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.110+0000 7ff201ffb640 1 -- 172.21.15.16:0/2432846477 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff210005020 con 0x5643230b3d90 2024-03-28T22:16:21.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.111+0000 7ff2251c0740 1 -- 172.21.15.16:0/2432846477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643230b3d90 msgr2=0x5643230acbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:21.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.111+0000 7ff2251c0740 1 --2- 172.21.15.16:0/2432846477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643230b3d90 0x5643230acbc0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x56432309efb0 tx=0x7ff210012040 comp rx=0 tx=0).stop 2024-03-28T22:16:21.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.111+0000 7ff2251c0740 1 -- 172.21.15.16:0/2432846477 shutdown_connections 2024-03-28T22:16:21.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.111+0000 7ff2251c0740 1 --2- 172.21.15.16:0/2432846477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643230bc970 0x5643230d4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.111+0000 7ff2251c0740 1 --2- 172.21.15.16:0/2432846477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643230b3d90 0x5643230acbc0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.111+0000 7ff2251c0740 1 --2- 172.21.15.16:0/2432846477 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643230b3470 0x5643230c8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.111+0000 7ff2251c0740 1 -- 172.21.15.16:0/2432846477 >> 172.21.15.16:0/2432846477 conn(0x5643230adba0 msgr2=0x5643230ac4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:21.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.111+0000 7ff2251c0740 1 -- 172.21.15.16:0/2432846477 shutdown_connections 2024-03-28T22:16:21.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.111+0000 7ff2251c0740 1 -- 172.21.15.16:0/2432846477 wait complete. 2024-03-28T22:16:21.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.111+0000 7ff2251c0740 1 -- start start 2024-03-28T22:16:21.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.111+0000 7ff2251c0740 1 Processor -- start 2024-03-28T22:16:21.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.112+0000 7ff2251c0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643230b3470 0x5643230b6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:21.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.112+0000 7ff2251c0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643230b3d90 0x5643230b6d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:21.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.112+0000 7ff2251c0740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643230bc970 0x5643230b72a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:21.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.112+0000 7ff2251c0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff21c1ab810 con 0x5643230b3470 2024-03-28T22:16:21.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.112+0000 7ff2251c0740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff21c1acc80 con 0x5643230bc970 2024-03-28T22:16:21.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.112+0000 7ff2251c0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff21c1abee0 con 0x5643230b3d90 2024-03-28T22:16:21.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.112+0000 7ff2251c0740 10 client.0 _ll_register_callbacks cb 0x7ff20c0043e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:16:21.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.112+0000 7ff21bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643230b3470 0x5643230b6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:21.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.112+0000 7ff21bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643230b3470 0x5643230b6800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40340/0 (socket says 172.21.15.16:40340) 2024-03-28T22:16:21.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.112+0000 7ff21bfff640 1 -- 172.21.15.16:0/3452530778 learned_addr learned my addr 172.21.15.16:0/3452530778 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:21.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.112+0000 7ff220a9f640 1 --2- 172.21.15.16:0/3452530778 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643230bc970 0x5643230b72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:21.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.112+0000 7ff21bfff640 1 -- 172.21.15.16:0/3452530778 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643230b3d90 msgr2=0x5643230b6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:21.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.112+0000 7ff21bfff640 1 --2- 172.21.15.16:0/3452530778 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643230b3d90 0x5643230b6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.112+0000 7ff21bfff640 1 -- 172.21.15.16:0/3452530778 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643230bc970 msgr2=0x5643230b72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:21.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.112+0000 7ff21bfff640 1 --2- 172.21.15.16:0/3452530778 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643230bc970 0x5643230b72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.112+0000 7ff21bfff640 1 -- 172.21.15.16:0/3452530778 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564323131fc0 con 0x5643230b3470 2024-03-28T22:16:21.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.112+0000 7ff21bfff640 1 --2- 172.21.15.16:0/3452530778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643230b3470 0x5643230b6800 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7ff20c005eb0 tx=0x7ff20c003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2fd64136777eea server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:21.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.112+0000 7ff1fb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:16:21.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.112+0000 7ff1fb7fe640 1 -- 172.21.15.16:0/3452530778 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff20c054070 con 0x5643230b3470 2024-03-28T22:16:21.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.112+0000 7ff1fb7fe640 1 -- 172.21.15.16:0/3452530778 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7ff20c0046c0 con 0x5643230b3470 2024-03-28T22:16:21.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.112+0000 7ff2251c0740 1 -- 172.21.15.16:0/3452530778 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2100262e0 con 0x5643230b3470 2024-03-28T22:16:21.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.113+0000 7ff1fb7fe640 1 -- 172.21.15.16:0/3452530778 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff20c05d050 con 0x5643230b3470 2024-03-28T22:16:21.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.113+0000 7ff2251c0740 10 client.16722 fetch_fsmap learned FSMap version 27 2024-03-28T22:16:21.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.113+0000 7ff2251c0740 1 -- 172.21.15.16:0/3452530778 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564323133260 con 0x5643230b3470 2024-03-28T22:16:21.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.113+0000 7ff1fb7fe640 1 -- 172.21.15.16:0/3452530778 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2100262e0 con 0x5643230b3470 2024-03-28T22:16:21.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.113+0000 7ff1fb7fe640 1 -- 172.21.15.16:0/3452530778 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7ff20c057020 con 0x5643230b3470 2024-03-28T22:16:21.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.113+0000 7ff2251c0740 10 client.16722 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:16:21.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.113+0000 7ff2251c0740 10 client.16722 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T22:16:21.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.113+0000 7ff2251c0740 20 client.16722 populate_metadata read hostname 'smithi016' 2024-03-28T22:16:21.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.113+0000 7ff2251c0740 1 --2- 172.21.15.16:0/3452530778 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x564323133a00 0x564323135e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:21.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.113+0000 7ff2251c0740 4 client.16722 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:16:21.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.113+0000 7ff2251c0740 1 -- 172.21.15.16:0/3452530778 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56432301ea40 con 0x564323133a00 2024-03-28T22:16:21.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.113+0000 7ff21b7fe640 1 --2- 172.21.15.16:0/3452530778 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x564323133a00 0x564323135e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:21.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.113+0000 7ff21b7fe640 1 --2- 172.21.15.16:0/3452530778 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x564323133a00 0x564323135e40 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b8dad301693d8dbd server_cookie=ba613967207d5aca in_seq=0 out_seq=0 2024-03-28T22:16:21.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.113+0000 7ff1fb7fe640 10 client.16722 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:16:21.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.114+0000 7ff1fb7fe640 1 -- 172.21.15.16:0/3452530778 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56432301ea40 con 0x564323133a00 2024-03-28T22:16:21.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.114+0000 7ff1fb7fe640 10 client.16722 handle_command_reply: tid=0 2024-03-28T22:16:21.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.114+0000 7ff2251c0740 1 client.16722 shutdown 2024-03-28T22:16:21.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.114+0000 7ff2251c0740 1 -- 172.21.15.16:0/3452530778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643230b3470 msgr2=0x5643230b6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:21.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.114+0000 7ff2251c0740 1 --2- 172.21.15.16:0/3452530778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643230b3470 0x5643230b6800 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7ff20c005eb0 tx=0x7ff20c003820 comp rx=0 tx=0).stop 2024-03-28T22:16:21.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.114+0000 7ff2251c0740 1 -- 172.21.15.16:0/3452530778 shutdown_connections 2024-03-28T22:16:21.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.114+0000 7ff2251c0740 1 --2- 172.21.15.16:0/3452530778 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x564323133a00 0x564323135e40 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.114+0000 7ff2251c0740 1 --2- 172.21.15.16:0/3452530778 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5643230bc970 0x5643230b72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.114+0000 7ff2251c0740 1 --2- 172.21.15.16:0/3452530778 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643230b3d90 0x5643230b6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.114+0000 7ff2251c0740 1 --2- 172.21.15.16:0/3452530778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5643230b3470 0x5643230b6800 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.114+0000 7ff1fb7fe640 0 client.16722 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:16:21.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.114+0000 7ff2251c0740 1 -- 172.21.15.16:0/3452530778 >> 172.21.15.16:0/3452530778 conn(0x5643230adba0 msgr2=0x5643230acd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:21.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.114+0000 7ff2251c0740 1 -- 172.21.15.16:0/3452530778 shutdown_connections 2024-03-28T22:16:21.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.114+0000 7ff2251c0740 1 -- 172.21.15.16:0/3452530778 wait complete. 2024-03-28T22:16:21.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.114+0000 7ff2251c0740 20 client.16722 trim_cache size 0 max 16384 2024-03-28T22:16:21.144 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:16:21.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.114+0000 7ff222502640 1 -- 172.21.15.16:0/2637598121 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff2040a1ef0 msgr2=0x7ff2040a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:21.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.114+0000 7ff222502640 1 --2- 172.21.15.16:0/2637598121 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff2040a1ef0 0x7ff2040a43b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff21c106c80 tx=0x7ff210013830 comp rx=0 tx=0).stop 2024-03-28T22:16:21.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.114+0000 7ff222502640 1 -- 172.21.15.16:0/2637598121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff21c1076f0 msgr2=0x7ff21c19c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:21.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.114+0000 7ff222502640 1 --2- 172.21.15.16:0/2637598121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff21c1076f0 0x7ff21c19c7e0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7ff21400cce0 tx=0x7ff21400a620 comp rx=0 tx=0).stop 2024-03-28T22:16:21.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.115+0000 7ff222502640 1 -- 172.21.15.16:0/2637598121 shutdown_connections 2024-03-28T22:16:21.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.115+0000 7ff222502640 1 --2- 172.21.15.16:0/2637598121 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff2040a1ef0 0x7ff2040a43b0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.115+0000 7ff222502640 1 --2- 172.21.15.16:0/2637598121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff21c1076f0 0x7ff21c19c7e0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.115+0000 7ff222502640 1 --2- 172.21.15.16:0/2637598121 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff21c100560 0x7ff21c198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.115+0000 7ff222502640 1 --2- 172.21.15.16:0/2637598121 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff21c0ffbc0 0x7ff21c197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.115+0000 7ff222502640 1 -- 172.21.15.16:0/2637598121 >> 172.21.15.16:0/2637598121 conn(0x7ff21c076b50 msgr2=0x7ff21c0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:21.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.115+0000 7ff222502640 1 -- 172.21.15.16:0/2637598121 shutdown_connections 2024-03-28T22:16:21.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.115+0000 7ff222502640 1 -- 172.21.15.16:0/2637598121 wait complete. 2024-03-28T22:16:21.147 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:16:21.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.250+0000 7f4b890ed640 1 -- 172.21.15.16:0/2522673150 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b84100560 msgr2=0x7f4b8410e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:21.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.250+0000 7f4b890ed640 1 --2- 172.21.15.16:0/2522673150 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b84100560 0x7f4b8410e5b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c009900 tx=0x7f4b7c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:16:21.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.250+0000 7f4b890ed640 1 -- 172.21.15.16:0/2522673150 shutdown_connections 2024-03-28T22:16:21.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.250+0000 7f4b890ed640 1 --2- 172.21.15.16:0/2522673150 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b84100560 0x7f4b8410e5b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.250+0000 7f4b890ed640 1 --2- 172.21.15.16:0/2522673150 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b840ffbc0 0x7f4b84100020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.250+0000 7f4b890ed640 1 --2- 172.21.15.16:0/2522673150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b841076f0 0x7f4b84107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.250+0000 7f4b890ed640 1 -- 172.21.15.16:0/2522673150 >> 172.21.15.16:0/2522673150 conn(0x7f4b84076b50 msgr2=0x7f4b840fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:21.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.250+0000 7f4b890ed640 1 -- 172.21.15.16:0/2522673150 shutdown_connections 2024-03-28T22:16:21.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.250+0000 7f4b890ed640 1 -- 172.21.15.16:0/2522673150 wait complete. 2024-03-28T22:16:21.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.250+0000 7f4b890ed640 1 Processor -- start 2024-03-28T22:16:21.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.251+0000 7f4b890ed640 1 -- start start 2024-03-28T22:16:21.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.251+0000 7f4b890ed640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b840ffbc0 0x7f4b84197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:21.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.251+0000 7f4b890ed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b84100560 0x7f4b841984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:21.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.251+0000 7f4b890ed640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b841076f0 0x7f4b8419c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:21.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.251+0000 7f4b890ed640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4b84111b30 con 0x7f4b84100560 2024-03-28T22:16:21.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.251+0000 7f4b890ed640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4b841119b0 con 0x7f4b841076f0 2024-03-28T22:16:21.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.251+0000 7f4b890ed640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4b84111cb0 con 0x7f4b840ffbc0 2024-03-28T22:16:21.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.251+0000 7f4b8259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b84100560 0x7f4b841984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:21.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.251+0000 7f4b8359e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b841076f0 0x7f4b8419c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:21.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.251+0000 7f4b8259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b84100560 0x7f4b841984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40350/0 (socket says 172.21.15.16:40350) 2024-03-28T22:16:21.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.251+0000 7f4b8259c640 1 -- 172.21.15.16:0/1203908236 learned_addr learned my addr 172.21.15.16:0/1203908236 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:21.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.251+0000 7f4b82d9d640 1 --2- 172.21.15.16:0/1203908236 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b840ffbc0 0x7f4b84197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:21.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.251+0000 7f4b8359e640 1 -- 172.21.15.16:0/1203908236 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b840ffbc0 msgr2=0x7f4b84197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:21.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.251+0000 7f4b8359e640 1 --2- 172.21.15.16:0/1203908236 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b840ffbc0 0x7f4b84197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.251+0000 7f4b8359e640 1 -- 172.21.15.16:0/1203908236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b84100560 msgr2=0x7f4b841984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:21.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.251+0000 7f4b8359e640 1 --2- 172.21.15.16:0/1203908236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b84100560 0x7f4b841984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.251+0000 7f4b8359e640 1 -- 172.21.15.16:0/1203908236 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b8419cf80 con 0x7f4b841076f0 2024-03-28T22:16:21.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.251+0000 7f4b8259c640 1 --2- 172.21.15.16:0/1203908236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b84100560 0x7f4b841984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:16:21.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.252+0000 7f4b8359e640 1 --2- 172.21.15.16:0/1203908236 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b841076f0 0x7f4b8419c850 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c009230 tx=0x7f4b7c004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:21.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.252+0000 7f4b7bfff640 1 -- 172.21.15.16:0/1203908236 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c041070 con 0x7f4b841076f0 2024-03-28T22:16:21.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.252+0000 7f4b7bfff640 1 -- 172.21.15.16:0/1203908236 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b7c0054e0 con 0x7f4b841076f0 2024-03-28T22:16:21.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.252+0000 7f4b890ed640 1 -- 172.21.15.16:0/1203908236 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b8419d210 con 0x7f4b841076f0 2024-03-28T22:16:21.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.252+0000 7f4b7bfff640 1 -- 172.21.15.16:0/1203908236 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c003c90 con 0x7f4b841076f0 2024-03-28T22:16:21.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.252+0000 7f4b890ed640 1 -- 172.21.15.16:0/1203908236 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b841a4b00 con 0x7f4b841076f0 2024-03-28T22:16:21.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.254+0000 7f4b8bdae740 1 Processor -- start 2024-03-28T22:16:21.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.254+0000 7f4b7bfff640 1 -- 172.21.15.16:0/1203908236 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f4b7c044070 con 0x7f4b841076f0 2024-03-28T22:16:21.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.254+0000 7f4b8bdae740 1 -- start start 2024-03-28T22:16:21.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.255+0000 7f4b8bdae740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e32fc25470 0x55e32fc25850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:21.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.255+0000 7f4b8bdae740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e32fc25d90 0x55e32fc2e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:21.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.255+0000 7f4b8bdae740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e32fc2e970 0x55e32fc30d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:21.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.255+0000 7f4b8bdae740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e32fb891b0 con 0x55e32fc25470 2024-03-28T22:16:21.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.255+0000 7f4b8bdae740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55e32fb31c00 con 0x55e32fc2e970 2024-03-28T22:16:21.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.255+0000 7f4b8bdae740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e32fb85250 con 0x55e32fc25d90 2024-03-28T22:16:21.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.255+0000 7f4b8259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e32fc25470 0x55e32fc25850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:21.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.255+0000 7f4b8259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e32fc25470 0x55e32fc25850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40356/0 (socket says 172.21.15.16:40356) 2024-03-28T22:16:21.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.255+0000 7f4b8259c640 1 -- 172.21.15.16:0/1137689473 learned_addr learned my addr 172.21.15.16:0/1137689473 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:21.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.255+0000 7f4b7bfff640 1 --2- 172.21.15.16:0/1203908236 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4b700a1f70 0x7f4b700a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:21.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.255+0000 7f4b82d9d640 1 --2- 172.21.15.16:0/1137689473 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e32fc25d90 0x55e32fc2e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:21.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.255+0000 7f4b8259c640 1 --2- 172.21.15.16:0/1137689473 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e32fc2e970 0x55e32fc30d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:21.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.255+0000 7f4b8359e640 1 --2- 172.21.15.16:0/1203908236 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4b700a1f70 0x7f4b700a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:21.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.255+0000 7f4b7bfff640 1 -- 172.21.15.16:0/1203908236 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f4b7c0eff80 con 0x7f4b841076f0 2024-03-28T22:16:21.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.255+0000 7f4b8259c640 1 -- 172.21.15.16:0/1137689473 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e32fc25d90 msgr2=0x55e32fc2e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:21.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.255+0000 7f4b8259c640 1 --2- 172.21.15.16:0/1137689473 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e32fc25d90 0x55e32fc2e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.255+0000 7f4b8259c640 1 -- 172.21.15.16:0/1137689473 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e32fc2e970 msgr2=0x55e32fc30d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:21.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.255+0000 7f4b8259c640 1 --2- 172.21.15.16:0/1137689473 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e32fc2e970 0x55e32fc30d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.255+0000 7f4b8259c640 1 -- 172.21.15.16:0/1137689473 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e32fb513d0 con 0x55e32fc25470 2024-03-28T22:16:21.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.256+0000 7f4b8259c640 1 --2- 172.21.15.16:0/1137689473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e32fc25470 0x55e32fc25850 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c012c70 tx=0x7f4b6c00bbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe9bef5864915062 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:21.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.256+0000 7f4b8359e640 1 --2- 172.21.15.16:0/1203908236 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4b700a1f70 0x7f4b700a4430 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c0b94c0 tx=0x7f4b7c02bc30 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:21.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.256+0000 7f4b797fa640 1 -- 172.21.15.16:0/1137689473 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c012d60 con 0x55e32fc25470 2024-03-28T22:16:21.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.256+0000 7f4b797fa640 1 -- 172.21.15.16:0/1137689473 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b6c005500 con 0x55e32fc25470 2024-03-28T22:16:21.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.256+0000 7f4b797fa640 1 -- 172.21.15.16:0/1137689473 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c0049a0 con 0x55e32fc25470 2024-03-28T22:16:21.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.256+0000 7f4b8bdae740 1 -- 172.21.15.16:0/1137689473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e32fc25470 msgr2=0x55e32fc25850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:21.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.256+0000 7f4b8bdae740 1 --2- 172.21.15.16:0/1137689473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e32fc25470 0x55e32fc25850 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c012c70 tx=0x7f4b6c00bbe0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.257+0000 7f4b8bdae740 1 -- 172.21.15.16:0/1137689473 shutdown_connections 2024-03-28T22:16:21.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.257+0000 7f4b8bdae740 1 --2- 172.21.15.16:0/1137689473 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e32fc2e970 0x55e32fc30d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.257+0000 7f4b8bdae740 1 --2- 172.21.15.16:0/1137689473 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e32fc25d90 0x55e32fc2e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.257+0000 7f4b8bdae740 1 --2- 172.21.15.16:0/1137689473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e32fc25470 0x55e32fc25850 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.257+0000 7f4b8bdae740 1 -- 172.21.15.16:0/1137689473 >> 172.21.15.16:0/1137689473 conn(0x55e32fc1fba0 msgr2=0x55e32fc1ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:21.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.257+0000 7f4b8bdae740 1 -- 172.21.15.16:0/1137689473 shutdown_connections 2024-03-28T22:16:21.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.257+0000 7f4b8bdae740 1 -- 172.21.15.16:0/1137689473 wait complete. 2024-03-28T22:16:21.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.257+0000 7f4b8bdae740 1 -- start start 2024-03-28T22:16:21.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.257+0000 7f4b8bdae740 1 Processor -- start 2024-03-28T22:16:21.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.257+0000 7f4b8bdae740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e32fc25470 0x55e32fc2b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:21.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.257+0000 7f4b8bdae740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e32fc25d90 0x55e32fc2bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:21.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.257+0000 7f4b8bdae740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e32fc2e970 0x55e32fc2c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:21.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.257+0000 7f4b8bdae740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e32fc31700 con 0x55e32fc2e970 2024-03-28T22:16:21.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.257+0000 7f4b8bdae740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55e32fb27470 con 0x55e32fc25470 2024-03-28T22:16:21.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.257+0000 7f4b8bdae740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e32fc31880 con 0x55e32fc25d90 2024-03-28T22:16:21.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.257+0000 7f4b8bdae740 10 client.0 _ll_register_callbacks cb 0x55e32fb44cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:16:21.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.257+0000 7f4b8259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e32fc2e970 0x55e32fc2c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:21.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.257+0000 7f4b8259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e32fc2e970 0x55e32fc2c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40368/0 (socket says 172.21.15.16:40368) 2024-03-28T22:16:21.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.257+0000 7f4b8259c640 1 -- 172.21.15.16:0/532595993 learned_addr learned my addr 172.21.15.16:0/532595993 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:21.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.257+0000 7f4b8259c640 1 --2- 172.21.15.16:0/532595993 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e32fc25470 0x55e32fc2b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:21.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.257+0000 7f4b82d9d640 1 --2- 172.21.15.16:0/532595993 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e32fc25d90 0x55e32fc2bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:21.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.258+0000 7f4b8259c640 1 -- 172.21.15.16:0/532595993 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e32fc25d90 msgr2=0x55e32fc2bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:21.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.258+0000 7f4b8259c640 1 --2- 172.21.15.16:0/532595993 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e32fc25d90 0x55e32fc2bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.258+0000 7f4b8259c640 1 -- 172.21.15.16:0/532595993 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e32fc25470 msgr2=0x55e32fc2b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:21.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.258+0000 7f4b8259c640 1 --2- 172.21.15.16:0/532595993 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e32fc25470 0x55e32fc2b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.258+0000 7f4b8259c640 1 -- 172.21.15.16:0/532595993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e32f89d3f0 con 0x55e32fc2e970 2024-03-28T22:16:21.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.258+0000 7f4b8259c640 1 --2- 172.21.15.16:0/532595993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e32fc2e970 0x55e32fc2c400 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c012f90 tx=0x7f4b6c014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28d3c7d2ac792786 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:21.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.258+0000 7f4b6a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:16:21.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.258+0000 7f4b6a7fc640 1 -- 172.21.15.16:0/532595993 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c01f460 con 0x55e32fc2e970 2024-03-28T22:16:21.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.258+0000 7f4b6a7fc640 1 -- 172.21.15.16:0/532595993 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f4b6c01f600 con 0x55e32fc2e970 2024-03-28T22:16:21.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.258+0000 7f4b8bdae740 1 -- 172.21.15.16:0/532595993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b6c005500 con 0x55e32fc2e970 2024-03-28T22:16:21.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.258+0000 7f4b6a7fc640 1 -- 172.21.15.16:0/532595993 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4b6c034050 con 0x55e32fc2e970 2024-03-28T22:16:21.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.258+0000 7f4b8bdae740 10 client.16734 fetch_fsmap learned FSMap version 27 2024-03-28T22:16:21.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.258+0000 7f4b8bdae740 1 -- 172.21.15.16:0/532595993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e32f9677d0 con 0x55e32fc2e970 2024-03-28T22:16:21.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.258+0000 7f4b6a7fc640 1 -- 172.21.15.16:0/532595993 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b6c005500 con 0x55e32fc2e970 2024-03-28T22:16:21.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.259+0000 7f4b6a7fc640 1 -- 172.21.15.16:0/532595993 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f4b6c010800 con 0x55e32fc2e970 2024-03-28T22:16:21.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.259+0000 7f4b8bdae740 10 client.16734 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:16:21.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.259+0000 7f4b8bdae740 10 client.16734 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:16:21.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.259+0000 7f4b8bdae740 20 client.16734 populate_metadata read hostname 'smithi016' 2024-03-28T22:16:21.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.259+0000 7f4b8bdae740 1 --2- 172.21.15.16:0/532595993 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e32fc3e780 0x55e32fc40b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:21.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.259+0000 7f4b8bdae740 4 client.16734 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:16:21.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.259+0000 7f4b8bdae740 1 -- 172.21.15.16:0/532595993 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e32fb966f0 con 0x55e32fc3e780 2024-03-28T22:16:21.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.259+0000 7f4b82d9d640 1 --2- 172.21.15.16:0/532595993 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e32fc3e780 0x55e32fc40b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:21.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.259+0000 7f4b82d9d640 1 --2- 172.21.15.16:0/532595993 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e32fc3e780 0x55e32fc40b70 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29dbaafc01ff0f14 server_cookie=50040b4d32f503ee in_seq=0 out_seq=0 2024-03-28T22:16:21.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.259+0000 7f4b6a7fc640 10 client.16734 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:16:21.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.260+0000 7f4b6a7fc640 1 -- 172.21.15.16:0/532595993 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e32fb966f0 con 0x55e32fc3e780 2024-03-28T22:16:21.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.260+0000 7f4b6a7fc640 10 client.16734 handle_command_reply: tid=0 2024-03-28T22:16:21.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.260+0000 7f4b8bdae740 1 client.16734 shutdown 2024-03-28T22:16:21.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.260+0000 7f4b8bdae740 1 -- 172.21.15.16:0/532595993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e32fc2e970 msgr2=0x55e32fc2c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:21.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.260+0000 7f4b8bdae740 1 --2- 172.21.15.16:0/532595993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e32fc2e970 0x55e32fc2c400 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c012f90 tx=0x7f4b6c014040 comp rx=0 tx=0).stop 2024-03-28T22:16:21.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.260+0000 7f4b8bdae740 1 -- 172.21.15.16:0/532595993 shutdown_connections 2024-03-28T22:16:21.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.260+0000 7f4b8bdae740 1 --2- 172.21.15.16:0/532595993 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e32fc3e780 0x55e32fc40b70 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.260+0000 7f4b8bdae740 1 --2- 172.21.15.16:0/532595993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e32fc2e970 0x55e32fc2c400 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.260+0000 7f4b8bdae740 1 --2- 172.21.15.16:0/532595993 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e32fc25d90 0x55e32fc2bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.260+0000 7f4b6a7fc640 0 client.16734 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:16:21.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.260+0000 7f4b8bdae740 1 --2- 172.21.15.16:0/532595993 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e32fc25470 0x55e32fc2b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.260+0000 7f4b8bdae740 1 -- 172.21.15.16:0/532595993 >> 172.21.15.16:0/532595993 conn(0x55e32fc1fba0 msgr2=0x55e32fc218f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:21.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.260+0000 7f4b8bdae740 1 -- 172.21.15.16:0/532595993 shutdown_connections 2024-03-28T22:16:21.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.260+0000 7f4b8bdae740 1 -- 172.21.15.16:0/532595993 wait complete. 2024-03-28T22:16:21.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.260+0000 7f4b8bdae740 20 client.16734 trim_cache size 0 max 16384 2024-03-28T22:16:21.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.273+0000 7f4b8bdae740 1 Processor -- start 2024-03-28T22:16:21.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.273+0000 7f4b8bdae740 1 -- start start 2024-03-28T22:16:21.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.274+0000 7f4b8bdae740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e32fc25470 0x55e32fc2e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:21.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.274+0000 7f4b8bdae740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e32fc25d90 0x55e32fc3a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:21.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.274+0000 7f4b8bdae740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e32fc2e970 0x55e32fc1ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:21.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.274+0000 7f4b8bdae740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4b841afbc0 con 0x55e32fc25d90 2024-03-28T22:16:21.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.274+0000 7f4b8bdae740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4b841ad490 con 0x55e32fc25470 2024-03-28T22:16:21.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.274+0000 7f4b8bdae740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4b841ab2e0 con 0x55e32fc2e970 2024-03-28T22:16:21.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.274+0000 7f4b82d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e32fc25d90 0x55e32fc3a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:21.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.274+0000 7f4b82d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e32fc25d90 0x55e32fc3a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40372/0 (socket says 172.21.15.16:40372) 2024-03-28T22:16:21.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.274+0000 7f4b82d9d640 1 -- 172.21.15.16:0/241639272 learned_addr learned my addr 172.21.15.16:0/241639272 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:21.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.274+0000 7f4b8259c640 1 --2- 172.21.15.16:0/241639272 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e32fc25470 0x55e32fc2e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:21.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.274+0000 7f4b82d9d640 1 -- 172.21.15.16:0/241639272 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e32fc2e970 msgr2=0x55e32fc1ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:21.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.274+0000 7f4b82d9d640 1 --2- 172.21.15.16:0/241639272 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e32fc2e970 0x55e32fc1ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.274+0000 7f4b82d9d640 1 -- 172.21.15.16:0/241639272 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e32fc25470 msgr2=0x55e32fc2e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:21.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.274+0000 7f4b82d9d640 1 --2- 172.21.15.16:0/241639272 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e32fc25470 0x55e32fc2e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.274+0000 7f4b82d9d640 1 -- 172.21.15.16:0/241639272 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e32fc25bc0 con 0x55e32fc25d90 2024-03-28T22:16:21.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.274+0000 7f4b82d9d640 1 --2- 172.21.15.16:0/241639272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e32fc25d90 0x55e32fc3a4d0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x55e32fc2b1e0 tx=0x7f4b74009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a5152e378ee25f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:21.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.274+0000 7f4b78ff9640 1 -- 172.21.15.16:0/241639272 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b74024030 con 0x55e32fc25d90 2024-03-28T22:16:21.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.275+0000 7f4b78ff9640 1 -- 172.21.15.16:0/241639272 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b74011de0 con 0x55e32fc25d90 2024-03-28T22:16:21.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.275+0000 7f4b78ff9640 1 -- 172.21.15.16:0/241639272 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b7401e400 con 0x55e32fc25d90 2024-03-28T22:16:21.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.275+0000 7f4b8bdae740 1 -- 172.21.15.16:0/241639272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e32fc25d90 msgr2=0x55e32fc3a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:21.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.275+0000 7f4b8bdae740 1 --2- 172.21.15.16:0/241639272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e32fc25d90 0x55e32fc3a4d0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x55e32fc2b1e0 tx=0x7f4b74009500 comp rx=0 tx=0).stop 2024-03-28T22:16:21.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.275+0000 7f4b8bdae740 1 -- 172.21.15.16:0/241639272 shutdown_connections 2024-03-28T22:16:21.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.275+0000 7f4b8bdae740 1 --2- 172.21.15.16:0/241639272 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e32fc2e970 0x55e32fc1ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.275+0000 7f4b8bdae740 1 --2- 172.21.15.16:0/241639272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e32fc25d90 0x55e32fc3a4d0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.275+0000 7f4b8bdae740 1 --2- 172.21.15.16:0/241639272 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e32fc25470 0x55e32fc2e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.275+0000 7f4b8bdae740 1 -- 172.21.15.16:0/241639272 >> 172.21.15.16:0/241639272 conn(0x55e32fc1fba0 msgr2=0x55e32fc1e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:21.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.275+0000 7f4b8bdae740 1 -- 172.21.15.16:0/241639272 shutdown_connections 2024-03-28T22:16:21.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.275+0000 7f4b8bdae740 1 -- 172.21.15.16:0/241639272 wait complete. 2024-03-28T22:16:21.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.275+0000 7f4b8bdae740 1 -- start start 2024-03-28T22:16:21.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.276+0000 7f4b8bdae740 1 Processor -- start 2024-03-28T22:16:21.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.276+0000 7f4b8bdae740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e32fc25470 0x55e32fc28520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:21.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.276+0000 7f4b8bdae740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e32fc25d90 0x55e32fc28a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:21.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.276+0000 7f4b8bdae740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e32fc2e970 0x55e32fc28fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:21.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.276+0000 7f4b8bdae740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4b841ab950 con 0x55e32fc2e970 2024-03-28T22:16:21.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.276+0000 7f4b8bdae740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4b841acdc0 con 0x55e32fc25d90 2024-03-28T22:16:21.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.276+0000 7f4b8bdae740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4b841ac020 con 0x55e32fc25470 2024-03-28T22:16:21.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.276+0000 7f4b8bdae740 10 client.0 _ll_register_callbacks cb 0x7f4b6c00ecd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:16:21.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.276+0000 7f4b8259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e32fc2e970 0x55e32fc28fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:21.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.276+0000 7f4b82d9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e32fc25d90 0x55e32fc28a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:21.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.276+0000 7f4b82d9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e32fc25d90 0x55e32fc28a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:60592/0 (socket says 172.21.15.16:60592) 2024-03-28T22:16:21.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.276+0000 7f4b8259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e32fc2e970 0x55e32fc28fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40386/0 (socket says 172.21.15.16:40386) 2024-03-28T22:16:21.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.276+0000 7f4b82d9d640 1 -- 172.21.15.16:0/2521329142 learned_addr learned my addr 172.21.15.16:0/2521329142 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:21.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.276+0000 7f4b8259c640 1 --2- 172.21.15.16:0/2521329142 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e32fc25470 0x55e32fc28520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:21.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.276+0000 7f4b8259c640 1 -- 172.21.15.16:0/2521329142 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e32fc25470 msgr2=0x55e32fc28520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:21.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.276+0000 7f4b8259c640 1 --2- 172.21.15.16:0/2521329142 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e32fc25470 0x55e32fc28520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.276+0000 7f4b8259c640 1 -- 172.21.15.16:0/2521329142 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e32fc25d90 msgr2=0x55e32fc28a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:21.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.276+0000 7f4b8259c640 1 --2- 172.21.15.16:0/2521329142 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e32fc25d90 0x55e32fc28a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.276+0000 7f4b8259c640 1 -- 172.21.15.16:0/2521329142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e32fca39f0 con 0x55e32fc2e970 2024-03-28T22:16:21.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.276+0000 7f4b82d9d640 1 --2- 172.21.15.16:0/2521329142 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e32fc25d90 0x55e32fc28a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:16:21.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.276+0000 7f4b8259c640 1 --2- 172.21.15.16:0/2521329142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e32fc2e970 0x55e32fc28fa0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c00c010 tx=0x7f4b6c00b410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a79b7c380385494d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:21.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.276+0000 7f4b6a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:16:21.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.276+0000 7f4b6a7fc640 1 -- 172.21.15.16:0/2521329142 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c012430 con 0x55e32fc2e970 2024-03-28T22:16:21.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.276+0000 7f4b6a7fc640 1 -- 172.21.15.16:0/2521329142 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f4b6c010460 con 0x55e32fc2e970 2024-03-28T22:16:21.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.276+0000 7f4b8bdae740 1 -- 172.21.15.16:0/2521329142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b74011de0 con 0x55e32fc2e970 2024-03-28T22:16:21.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.277+0000 7f4b6a7fc640 1 -- 172.21.15.16:0/2521329142 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4b6c010660 con 0x55e32fc2e970 2024-03-28T22:16:21.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.277+0000 7f4b8bdae740 10 client.16740 fetch_fsmap learned FSMap version 27 2024-03-28T22:16:21.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.277+0000 7f4b8bdae740 1 -- 172.21.15.16:0/2521329142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e32fca4ca0 con 0x55e32fc2e970 2024-03-28T22:16:21.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.277+0000 7f4b6a7fc640 1 -- 172.21.15.16:0/2521329142 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b74011de0 con 0x55e32fc2e970 2024-03-28T22:16:21.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.277+0000 7f4b6a7fc640 1 -- 172.21.15.16:0/2521329142 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f4b6c041050 con 0x55e32fc2e970 2024-03-28T22:16:21.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.277+0000 7f4b8bdae740 10 client.16740 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:16:21.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.277+0000 7f4b8bdae740 10 client.16740 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:16:21.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.277+0000 7f4b8bdae740 20 client.16740 populate_metadata read hostname 'smithi016' 2024-03-28T22:16:21.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.277+0000 7f4b8bdae740 1 --2- 172.21.15.16:0/2521329142 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e32fca5260 0x55e32fca7680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:21.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.277+0000 7f4b8bdae740 4 client.16740 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:16:21.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.277+0000 7f4b8bdae740 1 -- 172.21.15.16:0/2521329142 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e32fb90a40 con 0x55e32fca5260 2024-03-28T22:16:21.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.277+0000 7f4b82d9d640 1 --2- 172.21.15.16:0/2521329142 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e32fca5260 0x55e32fca7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:21.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.278+0000 7f4b82d9d640 1 --2- 172.21.15.16:0/2521329142 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e32fca5260 0x55e32fca7680 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7761d8ee944c8324 server_cookie=fcf5844041a911c1 in_seq=0 out_seq=0 2024-03-28T22:16:21.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.278+0000 7f4b6a7fc640 10 client.16740 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:16:21.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.278+0000 7f4b6a7fc640 1 -- 172.21.15.16:0/2521329142 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e32fb90a40 con 0x55e32fca5260 2024-03-28T22:16:21.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.278+0000 7f4b6a7fc640 10 client.16740 handle_command_reply: tid=0 2024-03-28T22:16:21.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.278+0000 7f4b8bdae740 1 client.16740 shutdown 2024-03-28T22:16:21.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.278+0000 7f4b8bdae740 1 -- 172.21.15.16:0/2521329142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e32fc2e970 msgr2=0x55e32fc28fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:21.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.278+0000 7f4b8bdae740 1 --2- 172.21.15.16:0/2521329142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e32fc2e970 0x55e32fc28fa0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c00c010 tx=0x7f4b6c00b410 comp rx=0 tx=0).stop 2024-03-28T22:16:21.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.278+0000 7f4b8bdae740 1 -- 172.21.15.16:0/2521329142 shutdown_connections 2024-03-28T22:16:21.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.278+0000 7f4b8bdae740 1 --2- 172.21.15.16:0/2521329142 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e32fca5260 0x55e32fca7680 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.278+0000 7f4b8bdae740 1 --2- 172.21.15.16:0/2521329142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e32fc2e970 0x55e32fc28fa0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.278+0000 7f4b6a7fc640 0 client.16740 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:16:21.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.278+0000 7f4b8bdae740 1 --2- 172.21.15.16:0/2521329142 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e32fc25d90 0x55e32fc28a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.278+0000 7f4b8bdae740 1 --2- 172.21.15.16:0/2521329142 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e32fc25470 0x55e32fc28520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.278+0000 7f4b8bdae740 1 -- 172.21.15.16:0/2521329142 >> 172.21.15.16:0/2521329142 conn(0x55e32fc1fba0 msgr2=0x55e32fc1ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:21.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.278+0000 7f4b8bdae740 1 -- 172.21.15.16:0/2521329142 shutdown_connections 2024-03-28T22:16:21.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.278+0000 7f4b8bdae740 1 -- 172.21.15.16:0/2521329142 wait complete. 2024-03-28T22:16:21.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.278+0000 7f4b8bdae740 20 client.16740 trim_cache size 0 max 16384 2024-03-28T22:16:21.296 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:16:21.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.278+0000 7f4b890ed640 1 -- 172.21.15.16:0/1203908236 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4b700a1f70 msgr2=0x7f4b700a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:21.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.278+0000 7f4b890ed640 1 --2- 172.21.15.16:0/1203908236 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4b700a1f70 0x7f4b700a4430 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c0b94c0 tx=0x7f4b7c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:16:21.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.278+0000 7f4b890ed640 1 -- 172.21.15.16:0/1203908236 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b841076f0 msgr2=0x7f4b8419c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:21.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.278+0000 7f4b890ed640 1 --2- 172.21.15.16:0/1203908236 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b841076f0 0x7f4b8419c850 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c009230 tx=0x7f4b7c004270 comp rx=0 tx=0).stop 2024-03-28T22:16:21.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.279+0000 7f4b890ed640 1 -- 172.21.15.16:0/1203908236 shutdown_connections 2024-03-28T22:16:21.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.279+0000 7f4b890ed640 1 --2- 172.21.15.16:0/1203908236 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4b700a1f70 0x7f4b700a4430 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.279+0000 7f4b890ed640 1 --2- 172.21.15.16:0/1203908236 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b841076f0 0x7f4b8419c850 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.279+0000 7f4b890ed640 1 --2- 172.21.15.16:0/1203908236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b84100560 0x7f4b841984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.279+0000 7f4b890ed640 1 --2- 172.21.15.16:0/1203908236 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b840ffbc0 0x7f4b84197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:21.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.279+0000 7f4b890ed640 1 -- 172.21.15.16:0/1203908236 >> 172.21.15.16:0/1203908236 conn(0x7f4b84076b50 msgr2=0x7f4b840fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:21.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.279+0000 7f4b890ed640 1 -- 172.21.15.16:0/1203908236 shutdown_connections 2024-03-28T22:16:21.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:21.279+0000 7f4b890ed640 1 -- 172.21.15.16:0/1203908236 wait complete. 2024-03-28T22:16:21.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:21 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3755812130' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:16:21.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:21 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/3755812130' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:16:21.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:21 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3755812130' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:16:22.301 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:16:22.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:16:22.301 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:16:22.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.420+0000 7fb8c3b3e640 1 -- 172.21.15.16:0/2254741203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8bc107e50 msgr2=0x7fb8bc10e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:22.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.420+0000 7fb8c3b3e640 1 --2- 172.21.15.16:0/2254741203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8bc107e50 0x7fb8bc10e7f0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4009cd0 tx=0x7fb8b402c0f0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.420+0000 7fb8c3b3e640 1 -- 172.21.15.16:0/2254741203 shutdown_connections 2024-03-28T22:16:22.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.420+0000 7fb8c3b3e640 1 --2- 172.21.15.16:0/2254741203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8bc107e50 0x7fb8bc10e7f0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.420+0000 7fb8c3b3e640 1 --2- 172.21.15.16:0/2254741203 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8bc103310 0x7fb8bc107740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.420+0000 7fb8c3b3e640 1 --2- 172.21.15.16:0/2254741203 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8bc102960 0x7fb8bc102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.420+0000 7fb8c3b3e640 1 -- 172.21.15.16:0/2254741203 >> 172.21.15.16:0/2254741203 conn(0x7fb8bc077830 msgr2=0x7fb8bc077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:22.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.420+0000 7fb8c3b3e640 1 -- 172.21.15.16:0/2254741203 shutdown_connections 2024-03-28T22:16:22.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.420+0000 7fb8c3b3e640 1 -- 172.21.15.16:0/2254741203 wait complete. 2024-03-28T22:16:22.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.421+0000 7fb8c3b3e640 1 Processor -- start 2024-03-28T22:16:22.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.421+0000 7fb8c3b3e640 1 -- start start 2024-03-28T22:16:22.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.421+0000 7fb8c3b3e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8bc102960 0x7fb8bc198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:22.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.421+0000 7fb8c3b3e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8bc107e50 0x7fb8bc1986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:22.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.421+0000 7fb8c3b3e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8bc19ca50 0x7fb8bc19cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:22.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.421+0000 7fb8c3b3e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc111c50 con 0x7fb8bc19ca50 2024-03-28T22:16:22.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.421+0000 7fb8c3b3e640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc111ad0 con 0x7fb8bc107e50 2024-03-28T22:16:22.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.421+0000 7fb8c3b3e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc111dd0 con 0x7fb8bc102960 2024-03-28T22:16:22.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.421+0000 7fb8c20db640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8bc19ca50 0x7fb8bc19cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:22.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.421+0000 7fb8c10d9640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8bc107e50 0x7fb8bc1986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:22.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.421+0000 7fb8c20db640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8bc19ca50 0x7fb8bc19cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40390/0 (socket says 172.21.15.16:40390) 2024-03-28T22:16:22.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.421+0000 7fb8c20db640 1 -- 172.21.15.16:0/1343465526 learned_addr learned my addr 172.21.15.16:0/1343465526 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:22.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.421+0000 7fb8c18da640 1 --2- 172.21.15.16:0/1343465526 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8bc102960 0x7fb8bc198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:22.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.421+0000 7fb8c10d9640 1 -- 172.21.15.16:0/1343465526 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8bc102960 msgr2=0x7fb8bc198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:22.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.421+0000 7fb8c10d9640 1 --2- 172.21.15.16:0/1343465526 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8bc102960 0x7fb8bc198190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.421+0000 7fb8c10d9640 1 -- 172.21.15.16:0/1343465526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8bc19ca50 msgr2=0x7fb8bc19cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:22.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.421+0000 7fb8c10d9640 1 --2- 172.21.15.16:0/1343465526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8bc19ca50 0x7fb8bc19cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.421+0000 7fb8c10d9640 1 -- 172.21.15.16:0/1343465526 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8bc19d540 con 0x7fb8bc107e50 2024-03-28T22:16:22.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.421+0000 7fb8c20db640 1 --2- 172.21.15.16:0/1343465526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8bc19ca50 0x7fb8bc19cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:16:22.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.422+0000 7fb8c10d9640 1 --2- 172.21.15.16:0/1343465526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8bc107e50 0x7fb8bc1986d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc069890 tx=0x7fb8ac00ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:22.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.422+0000 7fb8b2ffd640 1 -- 172.21.15.16:0/1343465526 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac00cc40 con 0x7fb8bc107e50 2024-03-28T22:16:22.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.422+0000 7fb8b2ffd640 1 -- 172.21.15.16:0/1343465526 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8ac00cde0 con 0x7fb8bc107e50 2024-03-28T22:16:22.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.422+0000 7fb8c3b3e640 1 -- 172.21.15.16:0/1343465526 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8bc19d830 con 0x7fb8bc107e50 2024-03-28T22:16:22.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.422+0000 7fb8b2ffd640 1 -- 172.21.15.16:0/1343465526 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac0188e0 con 0x7fb8bc107e50 2024-03-28T22:16:22.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.422+0000 7fb8c3b3e640 1 -- 172.21.15.16:0/1343465526 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8bc1a5020 con 0x7fb8bc107e50 2024-03-28T22:16:22.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.424+0000 7fb8b2ffd640 1 -- 172.21.15.16:0/1343465526 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7fb8ac018a80 con 0x7fb8bc107e50 2024-03-28T22:16:22.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.425+0000 7fb8c67f8740 1 Processor -- start 2024-03-28T22:16:22.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.425+0000 7fb8c67f8740 1 -- start start 2024-03-28T22:16:22.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.425+0000 7fb8b2ffd640 1 --2- 172.21.15.16:0/1343465526 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fb8a80a1ef0 0x7fb8a80a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:22.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.425+0000 7fb8c67f8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d96a1f9720 0x55d96a1f9b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:22.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.425+0000 7fb8c67f8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d96a1fa040 0x55d96a206dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:22.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.425+0000 7fb8c67f8740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d96a207300 0x55d96a2096f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:22.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.425+0000 7fb8c67f8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d96a0fa0b0 con 0x55d96a1f9720 2024-03-28T22:16:22.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.425+0000 7fb8c67f8740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55d96a15cea0 con 0x55d96a207300 2024-03-28T22:16:22.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.426+0000 7fb8c67f8740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d96a105c00 con 0x55d96a1fa040 2024-03-28T22:16:22.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.426+0000 7fb8c18da640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d96a1f9720 0x55d96a1f9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:22.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.426+0000 7fb8c20db640 1 --2- 172.21.15.16:0/1343465526 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fb8a80a1ef0 0x7fb8a80a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:22.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.426+0000 7fb8c18da640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d96a1f9720 0x55d96a1f9b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40400/0 (socket says 172.21.15.16:40400) 2024-03-28T22:16:22.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.426+0000 7fb8b2ffd640 1 -- 172.21.15.16:0/1343465526 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fb8ac0ce470 con 0x7fb8bc107e50 2024-03-28T22:16:22.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.426+0000 7fb8c18da640 1 -- 172.21.15.16:0/3465573080 learned_addr learned my addr 172.21.15.16:0/3465573080 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:22.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.426+0000 7fb8c20db640 1 --2- 172.21.15.16:0/3465573080 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d96a207300 0x55d96a2096f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:22.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.426+0000 7fb8c10d9640 1 --2- 172.21.15.16:0/3465573080 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d96a1fa040 0x55d96a206dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:22.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.426+0000 7fb8c18da640 1 -- 172.21.15.16:0/3465573080 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d96a1fa040 msgr2=0x55d96a206dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:22.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.426+0000 7fb8c18da640 1 --2- 172.21.15.16:0/3465573080 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d96a1fa040 0x55d96a206dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.426+0000 7fb8c18da640 1 -- 172.21.15.16:0/3465573080 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d96a207300 msgr2=0x55d96a2096f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:22.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.426+0000 7fb8c18da640 1 --2- 172.21.15.16:0/3465573080 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d96a207300 0x55d96a2096f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.426+0000 7fb8c18da640 1 -- 172.21.15.16:0/3465573080 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d969f397e0 con 0x55d96a1f9720 2024-03-28T22:16:22.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.426+0000 7fb8c20db640 1 --2- 172.21.15.16:0/1343465526 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fb8a80a1ef0 0x7fb8a80a43b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4006010 tx=0x7fb8b4032430 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:22.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.426+0000 7fb8c18da640 1 --2- 172.21.15.16:0/3465573080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d96a1f9720 0x55d96a1f9b00 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fb8b800bcf0 tx=0x7fb8b800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60701ef7c0434535 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:22.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.426+0000 7fb8a7fff640 1 -- 172.21.15.16:0/3465573080 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b8015070 con 0x55d96a1f9720 2024-03-28T22:16:22.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.426+0000 7fb8a7fff640 1 -- 172.21.15.16:0/3465573080 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8b8011920 con 0x55d96a1f9720 2024-03-28T22:16:22.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.426+0000 7fb8a7fff640 1 -- 172.21.15.16:0/3465573080 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b8010910 con 0x55d96a1f9720 2024-03-28T22:16:22.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.427+0000 7fb8c67f8740 1 -- 172.21.15.16:0/3465573080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d96a1f9720 msgr2=0x55d96a1f9b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:22.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.427+0000 7fb8c67f8740 1 --2- 172.21.15.16:0/3465573080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d96a1f9720 0x55d96a1f9b00 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fb8b800bcf0 tx=0x7fb8b800f730 comp rx=0 tx=0).stop 2024-03-28T22:16:22.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.427+0000 7fb8c67f8740 1 -- 172.21.15.16:0/3465573080 shutdown_connections 2024-03-28T22:16:22.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.427+0000 7fb8c67f8740 1 --2- 172.21.15.16:0/3465573080 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d96a207300 0x55d96a2096f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.427+0000 7fb8c67f8740 1 --2- 172.21.15.16:0/3465573080 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d96a1fa040 0x55d96a206dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.427+0000 7fb8c67f8740 1 --2- 172.21.15.16:0/3465573080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d96a1f9720 0x55d96a1f9b00 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.427+0000 7fb8c67f8740 1 -- 172.21.15.16:0/3465573080 >> 172.21.15.16:0/3465573080 conn(0x55d96a1f4120 msgr2=0x55d96a1f4500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:22.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.427+0000 7fb8c67f8740 1 -- 172.21.15.16:0/3465573080 shutdown_connections 2024-03-28T22:16:22.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.427+0000 7fb8c67f8740 1 -- 172.21.15.16:0/3465573080 wait complete. 2024-03-28T22:16:22.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.428+0000 7fb8c67f8740 1 -- start start 2024-03-28T22:16:22.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.428+0000 7fb8c67f8740 1 Processor -- start 2024-03-28T22:16:22.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.428+0000 7fb8c67f8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d96a1f9720 0x55d96a204310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:22.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.428+0000 7fb8c67f8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d96a1fa040 0x55d96a204850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:22.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.428+0000 7fb8c67f8740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d96a207300 0x55d96a204d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:22.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.428+0000 7fb8c67f8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d96a0fb3d0 con 0x55d96a1f9720 2024-03-28T22:16:22.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.428+0000 7fb8c67f8740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55d96a158f40 con 0x55d96a207300 2024-03-28T22:16:22.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.428+0000 7fb8c67f8740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d96a20a0c0 con 0x55d96a1fa040 2024-03-28T22:16:22.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.428+0000 7fb8c67f8740 10 client.0 _ll_register_callbacks cb 0x55d96a118b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:16:22.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.428+0000 7fb8c18da640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d96a1f9720 0x55d96a204310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:22.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.428+0000 7fb8c18da640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d96a1f9720 0x55d96a204310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40402/0 (socket says 172.21.15.16:40402) 2024-03-28T22:16:22.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.428+0000 7fb8c18da640 1 -- 172.21.15.16:0/2474356120 learned_addr learned my addr 172.21.15.16:0/2474356120 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:22.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.428+0000 7fb8c20db640 1 --2- 172.21.15.16:0/2474356120 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d96a207300 0x55d96a204d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:22.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.428+0000 7fb8c10d9640 1 --2- 172.21.15.16:0/2474356120 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d96a1fa040 0x55d96a204850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:22.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.429+0000 7fb8c18da640 1 -- 172.21.15.16:0/2474356120 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d96a1fa040 msgr2=0x55d96a204850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:22.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.429+0000 7fb8c18da640 1 --2- 172.21.15.16:0/2474356120 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d96a1fa040 0x55d96a204850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.429+0000 7fb8c18da640 1 -- 172.21.15.16:0/2474356120 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d96a207300 msgr2=0x55d96a204d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:22.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.429+0000 7fb8c18da640 1 --2- 172.21.15.16:0/2474356120 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d96a207300 0x55d96a204d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.429+0000 7fb8c18da640 1 -- 172.21.15.16:0/2474356120 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d969f99f70 con 0x55d96a1f9720 2024-03-28T22:16:22.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.429+0000 7fb8c10d9640 1 --2- 172.21.15.16:0/2474356120 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d96a1fa040 0x55d96a204850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:16:22.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.429+0000 7fb8c18da640 1 --2- 172.21.15.16:0/2474356120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d96a1f9720 0x55d96a204310 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fb8b8010410 tx=0x7fb8b800ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8f2c826329d6020 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:22.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.429+0000 7fb8a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:16:22.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.429+0000 7fb8a57fa640 1 -- 172.21.15.16:0/2474356120 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b8028060 con 0x55d96a1f9720 2024-03-28T22:16:22.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.429+0000 7fb8a57fa640 1 -- 172.21.15.16:0/2474356120 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fb8b8011ad0 con 0x55d96a1f9720 2024-03-28T22:16:22.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.429+0000 7fb8c67f8740 1 -- 172.21.15.16:0/2474356120 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8b8011920 con 0x55d96a1f9720 2024-03-28T22:16:22.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.429+0000 7fb8a57fa640 1 -- 172.21.15.16:0/2474356120 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb8b8015070 con 0x55d96a1f9720 2024-03-28T22:16:22.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.429+0000 7fb8c67f8740 10 client.16752 fetch_fsmap learned FSMap version 27 2024-03-28T22:16:22.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.429+0000 7fb8c67f8740 1 -- 172.21.15.16:0/2474356120 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d969e31170 con 0x55d96a1f9720 2024-03-28T22:16:22.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.429+0000 7fb8a57fa640 1 -- 172.21.15.16:0/2474356120 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8b8011920 con 0x55d96a1f9720 2024-03-28T22:16:22.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.430+0000 7fb8a57fa640 1 -- 172.21.15.16:0/2474356120 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7fb8b8004850 con 0x55d96a1f9720 2024-03-28T22:16:22.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.430+0000 7fb8c67f8740 10 client.16752 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:16:22.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.430+0000 7fb8c67f8740 10 client.16752 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:16:22.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.430+0000 7fb8c67f8740 20 client.16752 populate_metadata read hostname 'smithi016' 2024-03-28T22:16:22.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.430+0000 7fb8c67f8740 1 --2- 172.21.15.16:0/2474356120 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d96a2170d0 0x55d96a2194c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:22.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.430+0000 7fb8c67f8740 4 client.16752 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:16:22.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.430+0000 7fb8c67f8740 1 -- 172.21.15.16:0/2474356120 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d96a16a3e0 con 0x55d96a2170d0 2024-03-28T22:16:22.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.430+0000 7fb8c10d9640 1 --2- 172.21.15.16:0/2474356120 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d96a2170d0 0x55d96a2194c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:22.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.430+0000 7fb8c10d9640 1 --2- 172.21.15.16:0/2474356120 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d96a2170d0 0x55d96a2194c0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ace1bf5b7e56172 server_cookie=6e2947227756f0b7 in_seq=0 out_seq=0 2024-03-28T22:16:22.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.430+0000 7fb8a57fa640 10 client.16752 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:16:22.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.431+0000 7fb8a57fa640 1 -- 172.21.15.16:0/2474356120 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d96a16a3e0 con 0x55d96a2170d0 2024-03-28T22:16:22.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.431+0000 7fb8a57fa640 10 client.16752 handle_command_reply: tid=0 2024-03-28T22:16:22.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.431+0000 7fb8c67f8740 1 client.16752 shutdown 2024-03-28T22:16:22.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.431+0000 7fb8c67f8740 1 -- 172.21.15.16:0/2474356120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d96a1f9720 msgr2=0x55d96a204310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:22.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.431+0000 7fb8c67f8740 1 --2- 172.21.15.16:0/2474356120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d96a1f9720 0x55d96a204310 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fb8b8010410 tx=0x7fb8b800ba30 comp rx=0 tx=0).stop 2024-03-28T22:16:22.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.431+0000 7fb8c67f8740 1 -- 172.21.15.16:0/2474356120 shutdown_connections 2024-03-28T22:16:22.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.431+0000 7fb8c67f8740 1 --2- 172.21.15.16:0/2474356120 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d96a2170d0 0x55d96a2194c0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.431+0000 7fb8c67f8740 1 --2- 172.21.15.16:0/2474356120 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d96a207300 0x55d96a204d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.431+0000 7fb8c67f8740 1 --2- 172.21.15.16:0/2474356120 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d96a1fa040 0x55d96a204850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.431+0000 7fb8a57fa640 0 client.16752 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:16:22.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.431+0000 7fb8c67f8740 1 --2- 172.21.15.16:0/2474356120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d96a1f9720 0x55d96a204310 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.431+0000 7fb8c67f8740 1 -- 172.21.15.16:0/2474356120 >> 172.21.15.16:0/2474356120 conn(0x55d96a1f4120 msgr2=0x55d96a1f5e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:22.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.432+0000 7fb8c67f8740 1 -- 172.21.15.16:0/2474356120 shutdown_connections 2024-03-28T22:16:22.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.432+0000 7fb8c67f8740 1 -- 172.21.15.16:0/2474356120 wait complete. 2024-03-28T22:16:22.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.432+0000 7fb8c67f8740 20 client.16752 trim_cache size 0 max 16384 2024-03-28T22:16:22.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.449+0000 7fb8c67f8740 1 Processor -- start 2024-03-28T22:16:22.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.449+0000 7fb8c67f8740 1 -- start start 2024-03-28T22:16:22.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.449+0000 7fb8c67f8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d96a1f9720 0x55d96a206a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:22.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.449+0000 7fb8c67f8740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d96a1fa040 0x55d96a212d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:22.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.449+0000 7fb8c67f8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d96a207300 0x55d96a21eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:22.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.449+0000 7fb8c67f8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc1adad0 con 0x55d96a1f9720 2024-03-28T22:16:22.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.449+0000 7fb8c67f8740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc1ab3a0 con 0x55d96a1fa040 2024-03-28T22:16:22.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.449+0000 7fb8c67f8740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc1a9230 con 0x55d96a207300 2024-03-28T22:16:22.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.450+0000 7fb8c18da640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d96a1f9720 0x55d96a206a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:22.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.450+0000 7fb8c18da640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d96a1f9720 0x55d96a206a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40416/0 (socket says 172.21.15.16:40416) 2024-03-28T22:16:22.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.450+0000 7fb8c18da640 1 -- 172.21.15.16:0/4270518290 learned_addr learned my addr 172.21.15.16:0/4270518290 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:22.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.450+0000 7fb8c10d9640 1 --2- 172.21.15.16:0/4270518290 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d96a1fa040 0x55d96a212d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:22.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.450+0000 7fb8c20db640 1 --2- 172.21.15.16:0/4270518290 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d96a207300 0x55d96a21eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:22.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.450+0000 7fb8c18da640 1 -- 172.21.15.16:0/4270518290 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d96a207300 msgr2=0x55d96a21eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:22.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.450+0000 7fb8c18da640 1 --2- 172.21.15.16:0/4270518290 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d96a207300 0x55d96a21eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.450+0000 7fb8c18da640 1 -- 172.21.15.16:0/4270518290 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d96a1fa040 msgr2=0x55d96a212d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:22.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.450+0000 7fb8c18da640 1 --2- 172.21.15.16:0/4270518290 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d96a1fa040 0x55d96a212d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.450+0000 7fb8c18da640 1 -- 172.21.15.16:0/4270518290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d96a202860 con 0x55d96a1f9720 2024-03-28T22:16:22.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.450+0000 7fb8c20db640 1 --2- 172.21.15.16:0/4270518290 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d96a207300 0x55d96a21eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:16:22.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.450+0000 7fb8c18da640 1 --2- 172.21.15.16:0/4270518290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d96a1f9720 0x55d96a206a40 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fb8b8004c20 tx=0x7fb8b80104b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5079abe58654f83 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:22.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.450+0000 7fb8a77fe640 1 -- 172.21.15.16:0/4270518290 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b8028070 con 0x55d96a1f9720 2024-03-28T22:16:22.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.450+0000 7fb8a77fe640 1 -- 172.21.15.16:0/4270518290 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8b800be90 con 0x55d96a1f9720 2024-03-28T22:16:22.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.450+0000 7fb8a77fe640 1 -- 172.21.15.16:0/4270518290 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b8005520 con 0x55d96a1f9720 2024-03-28T22:16:22.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.451+0000 7fb8c67f8740 1 -- 172.21.15.16:0/4270518290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d96a1f9720 msgr2=0x55d96a206a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:22.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.451+0000 7fb8c67f8740 1 --2- 172.21.15.16:0/4270518290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d96a1f9720 0x55d96a206a40 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fb8b8004c20 tx=0x7fb8b80104b0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.451+0000 7fb8c67f8740 1 -- 172.21.15.16:0/4270518290 shutdown_connections 2024-03-28T22:16:22.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.451+0000 7fb8c67f8740 1 --2- 172.21.15.16:0/4270518290 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d96a207300 0x55d96a21eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.451+0000 7fb8c67f8740 1 --2- 172.21.15.16:0/4270518290 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d96a1fa040 0x55d96a212d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.451+0000 7fb8c67f8740 1 --2- 172.21.15.16:0/4270518290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d96a1f9720 0x55d96a206a40 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.451+0000 7fb8c67f8740 1 -- 172.21.15.16:0/4270518290 >> 172.21.15.16:0/4270518290 conn(0x55d96a1f4120 msgr2=0x55d96a1f3180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:22.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.451+0000 7fb8c67f8740 1 -- 172.21.15.16:0/4270518290 shutdown_connections 2024-03-28T22:16:22.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.451+0000 7fb8c67f8740 1 -- 172.21.15.16:0/4270518290 wait complete. 2024-03-28T22:16:22.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.451+0000 7fb8c67f8740 1 -- start start 2024-03-28T22:16:22.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.451+0000 7fb8c67f8740 1 Processor -- start 2024-03-28T22:16:22.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.452+0000 7fb8c67f8740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d96a1f9720 0x55d96a20f380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:22.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.452+0000 7fb8c67f8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d96a1fa040 0x55d96a20c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:22.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.452+0000 7fb8c67f8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d96a207300 0x55d96a20c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:22.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.452+0000 7fb8c67f8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc1a9880 con 0x55d96a1fa040 2024-03-28T22:16:22.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.452+0000 7fb8c67f8740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc1aacd0 con 0x55d96a1f9720 2024-03-28T22:16:22.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.452+0000 7fb8c67f8740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc1a9f30 con 0x55d96a207300 2024-03-28T22:16:22.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.452+0000 7fb8c67f8740 10 client.0 _ll_register_callbacks cb 0x7fb8b800bde0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:16:22.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.452+0000 7fb8c10d9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d96a1fa040 0x55d96a20c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:22.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.452+0000 7fb8c18da640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d96a1f9720 0x55d96a20f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:22.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.452+0000 7fb8c10d9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d96a1fa040 0x55d96a20c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40430/0 (socket says 172.21.15.16:40430) 2024-03-28T22:16:22.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.452+0000 7fb8c10d9640 1 -- 172.21.15.16:0/2552104750 learned_addr learned my addr 172.21.15.16:0/2552104750 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:22.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.452+0000 7fb8c20db640 1 --2- 172.21.15.16:0/2552104750 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d96a207300 0x55d96a20c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:22.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.452+0000 7fb8c10d9640 1 -- 172.21.15.16:0/2552104750 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d96a207300 msgr2=0x55d96a20c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:22.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.452+0000 7fb8c10d9640 1 --2- 172.21.15.16:0/2552104750 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d96a207300 0x55d96a20c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.452+0000 7fb8c10d9640 1 -- 172.21.15.16:0/2552104750 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d96a1f9720 msgr2=0x55d96a20f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:22.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.452+0000 7fb8c10d9640 1 --2- 172.21.15.16:0/2552104750 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d96a1f9720 0x55d96a20f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.452+0000 7fb8c10d9640 1 -- 172.21.15.16:0/2552104750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d96a20cdd0 con 0x55d96a1fa040 2024-03-28T22:16:22.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.452+0000 7fb8c18da640 1 --2- 172.21.15.16:0/2552104750 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d96a1f9720 0x55d96a20f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:16:22.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.452+0000 7fb8c10d9640 1 --2- 172.21.15.16:0/2552104750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d96a1fa040 0x55d96a20c350 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac0972e0 tx=0x7fb8ac09aae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ff596c769281bca server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:22.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.452+0000 7fb8a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:16:22.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.452+0000 7fb8a57fa640 1 -- 172.21.15.16:0/2552104750 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac099240 con 0x55d96a1fa040 2024-03-28T22:16:22.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.452+0000 7fb8a57fa640 1 -- 172.21.15.16:0/2552104750 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fb8ac0974f0 con 0x55d96a1fa040 2024-03-28T22:16:22.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.453+0000 7fb8c67f8740 1 -- 172.21.15.16:0/2552104750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8b800be90 con 0x55d96a1fa040 2024-03-28T22:16:22.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.453+0000 7fb8a57fa640 1 -- 172.21.15.16:0/2552104750 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb8ac0a3190 con 0x55d96a1fa040 2024-03-28T22:16:22.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.453+0000 7fb8c67f8740 10 client.16758 fetch_fsmap learned FSMap version 27 2024-03-28T22:16:22.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.453+0000 7fb8c67f8740 1 -- 172.21.15.16:0/2552104750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d96a27ddf0 con 0x55d96a1fa040 2024-03-28T22:16:22.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.453+0000 7fb8a57fa640 1 -- 172.21.15.16:0/2552104750 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8b800be90 con 0x55d96a1fa040 2024-03-28T22:16:22.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.453+0000 7fb8a57fa640 1 -- 172.21.15.16:0/2552104750 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7fb8ac099d40 con 0x55d96a1fa040 2024-03-28T22:16:22.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.453+0000 7fb8c67f8740 10 client.16758 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:16:22.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.453+0000 7fb8c67f8740 10 client.16758 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:16:22.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.453+0000 7fb8c67f8740 20 client.16758 populate_metadata read hostname 'smithi016' 2024-03-28T22:16:22.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.453+0000 7fb8c67f8740 1 --2- 172.21.15.16:0/2552104750 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d96a27e4f0 0x55d96a2808e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:22.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.453+0000 7fb8c67f8740 4 client.16758 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:16:22.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.453+0000 7fb8c67f8740 1 -- 172.21.15.16:0/2552104750 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d96a164730 con 0x55d96a27e4f0 2024-03-28T22:16:22.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.453+0000 7fb8c18da640 1 --2- 172.21.15.16:0/2552104750 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d96a27e4f0 0x55d96a2808e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:22.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.454+0000 7fb8a57fa640 10 client.16758 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:16:22.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.454+0000 7fb8c18da640 1 --2- 172.21.15.16:0/2552104750 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d96a27e4f0 0x55d96a2808e0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47712d5090e89aa6 server_cookie=ab7a337a19ad0c59 in_seq=0 out_seq=0 2024-03-28T22:16:22.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.455+0000 7fb8a57fa640 1 -- 172.21.15.16:0/2552104750 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d96a164730 con 0x55d96a27e4f0 2024-03-28T22:16:22.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.455+0000 7fb8a57fa640 10 client.16758 handle_command_reply: tid=0 2024-03-28T22:16:22.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.455+0000 7fb8c67f8740 1 client.16758 shutdown 2024-03-28T22:16:22.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.455+0000 7fb8c67f8740 1 -- 172.21.15.16:0/2552104750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d96a1fa040 msgr2=0x55d96a20c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:22.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.455+0000 7fb8c67f8740 1 --2- 172.21.15.16:0/2552104750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d96a1fa040 0x55d96a20c350 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac0972e0 tx=0x7fb8ac09aae0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.455+0000 7fb8c67f8740 1 -- 172.21.15.16:0/2552104750 shutdown_connections 2024-03-28T22:16:22.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.455+0000 7fb8c67f8740 1 --2- 172.21.15.16:0/2552104750 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d96a27e4f0 0x55d96a2808e0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.455+0000 7fb8c67f8740 1 --2- 172.21.15.16:0/2552104750 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d96a207300 0x55d96a20c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.455+0000 7fb8c67f8740 1 --2- 172.21.15.16:0/2552104750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d96a1fa040 0x55d96a20c350 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.455+0000 7fb8a57fa640 0 client.16758 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:16:22.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.455+0000 7fb8c67f8740 1 --2- 172.21.15.16:0/2552104750 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d96a1f9720 0x55d96a20f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.455+0000 7fb8c67f8740 1 -- 172.21.15.16:0/2552104750 >> 172.21.15.16:0/2552104750 conn(0x55d96a1f4120 msgr2=0x55d96a1f54c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:22.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.455+0000 7fb8c67f8740 1 -- 172.21.15.16:0/2552104750 shutdown_connections 2024-03-28T22:16:22.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.455+0000 7fb8c67f8740 1 -- 172.21.15.16:0/2552104750 wait complete. 2024-03-28T22:16:22.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.455+0000 7fb8c67f8740 20 client.16758 trim_cache size 0 max 16384 2024-03-28T22:16:22.469 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:16:22.469 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-03-28T22:16:22.469 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "669fc00c-77e6-4491-81f6-ffd0f4d8d686", 2024-03-28T22:16:22.469 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-03-28T22:16:22.469 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:16:22.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.456+0000 7fb8c3b3e640 1 -- 172.21.15.16:0/1343465526 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fb8a80a1ef0 msgr2=0x7fb8a80a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:22.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.456+0000 7fb8c3b3e640 1 --2- 172.21.15.16:0/1343465526 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fb8a80a1ef0 0x7fb8a80a43b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4006010 tx=0x7fb8b4032430 comp rx=0 tx=0).stop 2024-03-28T22:16:22.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.456+0000 7fb8c3b3e640 1 -- 172.21.15.16:0/1343465526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8bc107e50 msgr2=0x7fb8bc1986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:22.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.456+0000 7fb8c3b3e640 1 --2- 172.21.15.16:0/1343465526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8bc107e50 0x7fb8bc1986d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc069890 tx=0x7fb8ac00ee50 comp rx=0 tx=0).stop 2024-03-28T22:16:22.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.456+0000 7fb8c3b3e640 1 -- 172.21.15.16:0/1343465526 shutdown_connections 2024-03-28T22:16:22.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.456+0000 7fb8c3b3e640 1 --2- 172.21.15.16:0/1343465526 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fb8a80a1ef0 0x7fb8a80a43b0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.456+0000 7fb8c3b3e640 1 --2- 172.21.15.16:0/1343465526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8bc19ca50 0x7fb8bc19cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.456+0000 7fb8c3b3e640 1 --2- 172.21.15.16:0/1343465526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8bc107e50 0x7fb8bc1986d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.456+0000 7fb8c3b3e640 1 --2- 172.21.15.16:0/1343465526 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8bc102960 0x7fb8bc198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.456+0000 7fb8c3b3e640 1 -- 172.21.15.16:0/1343465526 >> 172.21.15.16:0/1343465526 conn(0x7fb8bc077830 msgr2=0x7fb8bc0ff780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:22.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.456+0000 7fb8c3b3e640 1 -- 172.21.15.16:0/1343465526 shutdown_connections 2024-03-28T22:16:22.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.456+0000 7fb8c3b3e640 1 -- 172.21.15.16:0/1343465526 wait complete. 2024-03-28T22:16:22.473 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:16:22.473 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:16:22.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.578+0000 7f7a0bfb6640 1 -- 172.21.15.16:0/2514524059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a04103b90 msgr2=0x7f7a0410e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:22.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.578+0000 7f7a0bfb6640 1 --2- 172.21.15.16:0/2514524059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a04103b90 0x7f7a0410e6c0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f79fc009ca0 tx=0x7f79fc02c1d0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.578+0000 7f7a0bfb6640 1 -- 172.21.15.16:0/2514524059 shutdown_connections 2024-03-28T22:16:22.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.578+0000 7f7a0bfb6640 1 --2- 172.21.15.16:0/2514524059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a04103b90 0x7f7a0410e6c0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.578+0000 7f7a0bfb6640 1 --2- 172.21.15.16:0/2514524059 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a041031f0 0x7f7a04103650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.578+0000 7f7a0bfb6640 1 --2- 172.21.15.16:0/2514524059 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a04101ab0 0x7f7a04101e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.578+0000 7f7a0bfb6640 1 -- 172.21.15.16:0/2514524059 >> 172.21.15.16:0/2514524059 conn(0x7f7a040fd1e0 msgr2=0x7f7a040ff600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:22.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.578+0000 7f7a0bfb6640 1 -- 172.21.15.16:0/2514524059 shutdown_connections 2024-03-28T22:16:22.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.578+0000 7f7a0bfb6640 1 -- 172.21.15.16:0/2514524059 wait complete. 2024-03-28T22:16:22.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.579+0000 7f7a0bfb6640 1 Processor -- start 2024-03-28T22:16:22.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.579+0000 7f7a0bfb6640 1 -- start start 2024-03-28T22:16:22.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.579+0000 7f7a0bfb6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a04101ab0 0x7f7a041980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:22.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.579+0000 7f7a0bfb6640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a041031f0 0x7f7a04198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:22.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.579+0000 7f7a0bfb6640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a04103b90 0x7f7a0419c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:22.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.579+0000 7f7a0bfb6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7a04111c20 con 0x7f7a04101ab0 2024-03-28T22:16:22.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.579+0000 7f7a0bfb6640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7a04111aa0 con 0x7f7a041031f0 2024-03-28T22:16:22.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.579+0000 7f7a0bfb6640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7a04111da0 con 0x7f7a04103b90 2024-03-28T22:16:22.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.579+0000 7f7a09d52640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a04101ab0 0x7f7a041980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:22.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.579+0000 7f7a09d52640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a04101ab0 0x7f7a041980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40446/0 (socket says 172.21.15.16:40446) 2024-03-28T22:16:22.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.579+0000 7f7a09d52640 1 -- 172.21.15.16:0/551145451 learned_addr learned my addr 172.21.15.16:0/551145451 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:22.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.579+0000 7f7a09551640 1 --2- 172.21.15.16:0/551145451 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a041031f0 0x7f7a04198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:22.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.579+0000 7f7a0a553640 1 --2- 172.21.15.16:0/551145451 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a04103b90 0x7f7a0419c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:22.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.579+0000 7f7a09d52640 1 -- 172.21.15.16:0/551145451 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a04103b90 msgr2=0x7f7a0419c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:22.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.579+0000 7f7a09d52640 1 --2- 172.21.15.16:0/551145451 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a04103b90 0x7f7a0419c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.579+0000 7f7a09d52640 1 -- 172.21.15.16:0/551145451 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a041031f0 msgr2=0x7f7a04198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:22.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.579+0000 7f7a09d52640 1 --2- 172.21.15.16:0/551145451 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a041031f0 0x7f7a04198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.579+0000 7f7a09d52640 1 -- 172.21.15.16:0/551145451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a0419d060 con 0x7f7a04101ab0 2024-03-28T22:16:22.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.579+0000 7f7a0a553640 1 --2- 172.21.15.16:0/551145451 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a04103b90 0x7f7a0419c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:16:22.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.579+0000 7f7a09d52640 1 --2- 172.21.15.16:0/551145451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a04101ab0 0x7f7a041980c0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f7a0000b4a0 tx=0x7f7a0000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:22.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.580+0000 7f79faffd640 1 -- 172.21.15.16:0/551145451 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a00014070 con 0x7f7a04101ab0 2024-03-28T22:16:22.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.580+0000 7f79faffd640 1 -- 172.21.15.16:0/551145451 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a00010af0 con 0x7f7a04101ab0 2024-03-28T22:16:22.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.580+0000 7f7a0bfb6640 1 -- 172.21.15.16:0/551145451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a0419d350 con 0x7f7a04101ab0 2024-03-28T22:16:22.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.580+0000 7f79faffd640 1 -- 172.21.15.16:0/551145451 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a00017d10 con 0x7f7a04101ab0 2024-03-28T22:16:22.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.580+0000 7f7a0bfb6640 1 -- 172.21.15.16:0/551145451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a041a4be0 con 0x7f7a04101ab0 2024-03-28T22:16:22.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.582+0000 7f79faffd640 1 -- 172.21.15.16:0/551145451 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f7a00017070 con 0x7f7a04101ab0 2024-03-28T22:16:22.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.582+0000 7f7a0eb91740 1 Processor -- start 2024-03-28T22:16:22.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.582+0000 7f7a0eb91740 1 -- start start 2024-03-28T22:16:22.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.583+0000 7f7a0eb91740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5638f9cc4f40 0x5638f9cc5320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:22.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.583+0000 7f7a0eb91740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638f9cc5860 0x5638f9cd2660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:22.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.583+0000 7f7a0eb91740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5638f9cd2ba0 0x5638f9cd4f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:22.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.583+0000 7f7a0eb91740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5638f9c29180 con 0x5638f9cc5860 2024-03-28T22:16:22.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.583+0000 7f7a0eb91740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5638f9bd1b70 con 0x5638f9cc4f40 2024-03-28T22:16:22.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.583+0000 7f7a0eb91740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5638f9c25220 con 0x5638f9cd2ba0 2024-03-28T22:16:22.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.583+0000 7f7a09d52640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638f9cc5860 0x5638f9cd2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:22.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.583+0000 7f7a09d52640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638f9cc5860 0x5638f9cd2660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40454/0 (socket says 172.21.15.16:40454) 2024-03-28T22:16:22.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.583+0000 7f7a09d52640 1 -- 172.21.15.16:0/858679036 learned_addr learned my addr 172.21.15.16:0/858679036 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:22.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.583+0000 7f7a0a553640 1 --2- 172.21.15.16:0/858679036 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5638f9cc4f40 0x5638f9cc5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:22.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.583+0000 7f79faffd640 1 --2- 172.21.15.16:0/551145451 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f79f00a1f70 0x7f79f00a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:22.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.583+0000 7f7a0a553640 1 --2- 172.21.15.16:0/551145451 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f79f00a1f70 0x7f79f00a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:22.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.583+0000 7f7a09551640 1 --2- 172.21.15.16:0/858679036 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5638f9cd2ba0 0x5638f9cd4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:22.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.583+0000 7f79faffd640 1 -- 172.21.15.16:0/551145451 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f7a000cd3b0 con 0x7f7a04101ab0 2024-03-28T22:16:22.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.583+0000 7f7a09d52640 1 -- 172.21.15.16:0/858679036 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5638f9cd2ba0 msgr2=0x5638f9cd4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:22.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.583+0000 7f7a09d52640 1 --2- 172.21.15.16:0/858679036 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5638f9cd2ba0 0x5638f9cd4f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.583+0000 7f7a09d52640 1 -- 172.21.15.16:0/858679036 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5638f9cc4f40 msgr2=0x5638f9cc5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:22.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.583+0000 7f7a09d52640 1 --2- 172.21.15.16:0/858679036 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5638f9cc4f40 0x5638f9cc5320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.583+0000 7f7a09d52640 1 -- 172.21.15.16:0/858679036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638f9be57c0 con 0x5638f9cc5860 2024-03-28T22:16:22.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.583+0000 7f7a0a553640 1 --2- 172.21.15.16:0/858679036 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5638f9cc4f40 0x5638f9cc5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:16:22.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.583+0000 7f7a09d52640 1 --2- 172.21.15.16:0/858679036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638f9cc5860 0x5638f9cd2660 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f7a00099b70 tx=0x7f7a0000bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c317bad109784c05 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:22.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.584+0000 7f79effff640 1 -- 172.21.15.16:0/858679036 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0001e070 con 0x5638f9cc5860 2024-03-28T22:16:22.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.584+0000 7f7a0a553640 1 --2- 172.21.15.16:0/551145451 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f79f00a1f70 0x7f79f00a4430 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f79fc004af0 tx=0x7f79fc02c1d0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:22.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.584+0000 7f79effff640 1 -- 172.21.15.16:0/858679036 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a00099e70 con 0x5638f9cc5860 2024-03-28T22:16:22.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.584+0000 7f79effff640 1 -- 172.21.15.16:0/858679036 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a000964e0 con 0x5638f9cc5860 2024-03-28T22:16:22.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.584+0000 7f7a0eb91740 1 -- 172.21.15.16:0/858679036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638f9cc5860 msgr2=0x5638f9cd2660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:22.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.584+0000 7f7a0eb91740 1 --2- 172.21.15.16:0/858679036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638f9cc5860 0x5638f9cd2660 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f7a00099b70 tx=0x7f7a0000bf90 comp rx=0 tx=0).stop 2024-03-28T22:16:22.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.584+0000 7f7a0eb91740 1 -- 172.21.15.16:0/858679036 shutdown_connections 2024-03-28T22:16:22.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.584+0000 7f7a0eb91740 1 --2- 172.21.15.16:0/858679036 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5638f9cd2ba0 0x5638f9cd4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.584+0000 7f7a0eb91740 1 --2- 172.21.15.16:0/858679036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638f9cc5860 0x5638f9cd2660 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.584+0000 7f7a0eb91740 1 --2- 172.21.15.16:0/858679036 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5638f9cc4f40 0x5638f9cc5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.584+0000 7f7a0eb91740 1 -- 172.21.15.16:0/858679036 >> 172.21.15.16:0/858679036 conn(0x5638f9cbf9c0 msgr2=0x5638f9cbfda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:22.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.584+0000 7f7a0eb91740 1 -- 172.21.15.16:0/858679036 shutdown_connections 2024-03-28T22:16:22.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.584+0000 7f7a0eb91740 1 -- 172.21.15.16:0/858679036 wait complete. 2024-03-28T22:16:22.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.585+0000 7f7a0eb91740 1 -- start start 2024-03-28T22:16:22.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.585+0000 7f7a0eb91740 1 Processor -- start 2024-03-28T22:16:22.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.585+0000 7f7a0eb91740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638f9cc4f40 0x5638f9ccfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:22.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.585+0000 7f7a0eb91740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5638f9cc5860 0x5638f9cd00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:22.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.585+0000 7f7a0eb91740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5638f9cd2ba0 0x5638f9cd05f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:22.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.585+0000 7f7a0eb91740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5638f9cd5980 con 0x5638f9cc4f40 2024-03-28T22:16:22.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.585+0000 7f7a0eb91740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5638f9bc74a0 con 0x5638f9cd2ba0 2024-03-28T22:16:22.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.585+0000 7f7a0eb91740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5638f9cd5b00 con 0x5638f9cc5860 2024-03-28T22:16:22.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.585+0000 7f7a0eb91740 10 client.0 _ll_register_callbacks cb 0x5638f9bcad90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:16:22.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.585+0000 7f7a09d52640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638f9cc4f40 0x5638f9ccfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:22.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.585+0000 7f7a09d52640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638f9cc4f40 0x5638f9ccfb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40468/0 (socket says 172.21.15.16:40468) 2024-03-28T22:16:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.585+0000 7f7a09d52640 1 -- 172.21.15.16:0/317161637 learned_addr learned my addr 172.21.15.16:0/317161637 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.585+0000 7f7a0a553640 1 --2- 172.21.15.16:0/317161637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5638f9cd2ba0 0x5638f9cd05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.586+0000 7f7a09551640 1 --2- 172.21.15.16:0/317161637 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5638f9cc5860 0x5638f9cd00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.586+0000 7f7a09d52640 1 -- 172.21.15.16:0/317161637 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5638f9cc5860 msgr2=0x5638f9cd00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.586+0000 7f7a09d52640 1 --2- 172.21.15.16:0/317161637 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5638f9cc5860 0x5638f9cd00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.586+0000 7f7a09d52640 1 -- 172.21.15.16:0/317161637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5638f9cd2ba0 msgr2=0x5638f9cd05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.586+0000 7f7a09d52640 1 --2- 172.21.15.16:0/317161637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5638f9cd2ba0 0x5638f9cd05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.586+0000 7f7a09d52640 1 -- 172.21.15.16:0/317161637 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638f9a077d0 con 0x5638f9cc4f40 2024-03-28T22:16:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.586+0000 7f7a09d52640 1 --2- 172.21.15.16:0/317161637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638f9cc4f40 0x5638f9ccfb70 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f7a000962d0 tx=0x7f7a00017730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=135fd6a0a488816b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:22.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.586+0000 7f79ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:16:22.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.586+0000 7f79ed7fa640 1 -- 172.21.15.16:0/317161637 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0001e070 con 0x5638f9cc4f40 2024-03-28T22:16:22.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.586+0000 7f79ed7fa640 1 -- 172.21.15.16:0/317161637 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f7a00095bc0 con 0x5638f9cc4f40 2024-03-28T22:16:22.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.586+0000 7f7a0eb91740 1 -- 172.21.15.16:0/317161637 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a00099e70 con 0x5638f9cc4f40 2024-03-28T22:16:22.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.586+0000 7f79ed7fa640 1 -- 172.21.15.16:0/317161637 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7a0009dfc0 con 0x5638f9cc4f40 2024-03-28T22:16:22.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.586+0000 7f7a0eb91740 10 client.16770 fetch_fsmap learned FSMap version 27 2024-03-28T22:16:22.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.586+0000 7f7a0eb91740 1 -- 172.21.15.16:0/317161637 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5638f993d3f0 con 0x5638f9cc4f40 2024-03-28T22:16:22.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.587+0000 7f79ed7fa640 1 -- 172.21.15.16:0/317161637 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a00099e70 con 0x5638f9cc4f40 2024-03-28T22:16:22.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.587+0000 7f79ed7fa640 1 -- 172.21.15.16:0/317161637 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f7a000cad40 con 0x5638f9cc4f40 2024-03-28T22:16:22.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.587+0000 7f7a0eb91740 10 client.16770 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:16:22.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.587+0000 7f7a0eb91740 10 client.16770 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:16:22.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.587+0000 7f7a0eb91740 20 client.16770 populate_metadata read hostname 'smithi016' 2024-03-28T22:16:22.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.587+0000 7f7a0eb91740 1 --2- 172.21.15.16:0/317161637 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5638f9ce2de0 0x5638f9ce51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:22.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.587+0000 7f7a0eb91740 4 client.16770 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:16:22.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.587+0000 7f7a0eb91740 1 -- 172.21.15.16:0/317161637 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5638f9c366c0 con 0x5638f9ce2de0 2024-03-28T22:16:22.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.587+0000 7f7a09551640 1 --2- 172.21.15.16:0/317161637 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5638f9ce2de0 0x5638f9ce51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:22.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.588+0000 7f7a09551640 1 --2- 172.21.15.16:0/317161637 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5638f9ce2de0 0x5638f9ce51d0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b2d99f030f6fbbd server_cookie=85d204ada71c7776 in_seq=0 out_seq=0 2024-03-28T22:16:22.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.588+0000 7f79ed7fa640 10 client.16770 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:16:22.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.589+0000 7f79ed7fa640 1 -- 172.21.15.16:0/317161637 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5638f9c366c0 con 0x5638f9ce2de0 2024-03-28T22:16:22.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.589+0000 7f79ed7fa640 10 client.16770 handle_command_reply: tid=0 2024-03-28T22:16:22.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.589+0000 7f7a0eb91740 1 client.16770 shutdown 2024-03-28T22:16:22.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.589+0000 7f7a0eb91740 1 -- 172.21.15.16:0/317161637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638f9cc4f40 msgr2=0x5638f9ccfb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:22.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.589+0000 7f7a0eb91740 1 --2- 172.21.15.16:0/317161637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638f9cc4f40 0x5638f9ccfb70 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f7a000962d0 tx=0x7f7a00017730 comp rx=0 tx=0).stop 2024-03-28T22:16:22.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.589+0000 7f7a0eb91740 1 -- 172.21.15.16:0/317161637 shutdown_connections 2024-03-28T22:16:22.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.589+0000 7f7a0eb91740 1 --2- 172.21.15.16:0/317161637 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5638f9ce2de0 0x5638f9ce51d0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.589+0000 7f7a0eb91740 1 --2- 172.21.15.16:0/317161637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5638f9cd2ba0 0x5638f9cd05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.589+0000 7f7a0eb91740 1 --2- 172.21.15.16:0/317161637 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5638f9cc5860 0x5638f9cd00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.589+0000 7f79ed7fa640 0 client.16770 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:16:22.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.589+0000 7f7a0eb91740 1 --2- 172.21.15.16:0/317161637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638f9cc4f40 0x5638f9ccfb70 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.589+0000 7f7a0eb91740 1 -- 172.21.15.16:0/317161637 >> 172.21.15.16:0/317161637 conn(0x5638f9cbf9c0 msgr2=0x5638f9cc1710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:22.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.589+0000 7f7a0eb91740 1 -- 172.21.15.16:0/317161637 shutdown_connections 2024-03-28T22:16:22.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.589+0000 7f7a0eb91740 1 -- 172.21.15.16:0/317161637 wait complete. 2024-03-28T22:16:22.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.589+0000 7f7a0eb91740 20 client.16770 trim_cache size 0 max 16384 2024-03-28T22:16:22.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.608+0000 7f7a0eb91740 1 Processor -- start 2024-03-28T22:16:22.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.608+0000 7f7a0eb91740 1 -- start start 2024-03-28T22:16:22.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.608+0000 7f7a0eb91740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5638f9cc4f40 0x5638f9cd22a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:22.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.608+0000 7f7a0eb91740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638f9cc5860 0x5638f9cbeaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:22.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.608+0000 7f7a0eb91740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5638f9cd2ba0 0x5638f9ce81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:22.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.608+0000 7f7a0eb91740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7a041ada60 con 0x5638f9cc5860 2024-03-28T22:16:22.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.608+0000 7f7a0eb91740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7a041ab330 con 0x5638f9cc4f40 2024-03-28T22:16:22.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.608+0000 7f7a0eb91740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7a041a9180 con 0x5638f9cd2ba0 2024-03-28T22:16:22.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.609+0000 7f7a09551640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638f9cc5860 0x5638f9cbeaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:22.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.609+0000 7f7a0a553640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5638f9cd2ba0 0x5638f9ce81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:22.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.609+0000 7f7a09d52640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5638f9cc4f40 0x5638f9cd22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:22.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.609+0000 7f7a09551640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638f9cc5860 0x5638f9cbeaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40480/0 (socket says 172.21.15.16:40480) 2024-03-28T22:16:22.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.609+0000 7f7a09551640 1 -- 172.21.15.16:0/1291991945 learned_addr learned my addr 172.21.15.16:0/1291991945 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.609+0000 7f7a09551640 1 -- 172.21.15.16:0/1291991945 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5638f9cd2ba0 msgr2=0x5638f9ce81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.609+0000 7f7a09551640 1 --2- 172.21.15.16:0/1291991945 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5638f9cd2ba0 0x5638f9ce81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.609+0000 7f7a09551640 1 -- 172.21.15.16:0/1291991945 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5638f9cc4f40 msgr2=0x5638f9cd22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.609+0000 7f7a09551640 1 --2- 172.21.15.16:0/1291991945 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5638f9cc4f40 0x5638f9cd22a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.609+0000 7f7a09551640 1 -- 172.21.15.16:0/1291991945 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638f98fd0f0 con 0x5638f9cc5860 2024-03-28T22:16:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.609+0000 7f7a09551640 1 --2- 172.21.15.16:0/1291991945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638f9cc5860 0x5638f9cbeaa0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f7a040ff6b0 tx=0x7f79f401df60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e6c57ab7c2edd67 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.609+0000 7f79ef7fe640 1 -- 172.21.15.16:0/1291991945 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f4013b10 con 0x5638f9cc5860 2024-03-28T22:16:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.609+0000 7f79ef7fe640 1 -- 172.21.15.16:0/1291991945 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79f4013cb0 con 0x5638f9cc5860 2024-03-28T22:16:22.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.609+0000 7f79ef7fe640 1 -- 172.21.15.16:0/1291991945 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f4004a00 con 0x5638f9cc5860 2024-03-28T22:16:22.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.610+0000 7f7a0eb91740 1 -- 172.21.15.16:0/1291991945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638f9cc5860 msgr2=0x5638f9cbeaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:22.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.610+0000 7f7a0eb91740 1 --2- 172.21.15.16:0/1291991945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638f9cc5860 0x5638f9cbeaa0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f7a040ff6b0 tx=0x7f79f401df60 comp rx=0 tx=0).stop 2024-03-28T22:16:22.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.610+0000 7f7a0eb91740 1 -- 172.21.15.16:0/1291991945 shutdown_connections 2024-03-28T22:16:22.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.610+0000 7f7a0eb91740 1 --2- 172.21.15.16:0/1291991945 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5638f9cd2ba0 0x5638f9ce81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.610+0000 7f7a0eb91740 1 --2- 172.21.15.16:0/1291991945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638f9cc5860 0x5638f9cbeaa0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.610+0000 7f7a0eb91740 1 --2- 172.21.15.16:0/1291991945 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5638f9cc4f40 0x5638f9cd22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.610+0000 7f7a0eb91740 1 -- 172.21.15.16:0/1291991945 >> 172.21.15.16:0/1291991945 conn(0x5638f9cbf9c0 msgr2=0x5638f9cbe390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:22.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.610+0000 7f7a0eb91740 1 -- 172.21.15.16:0/1291991945 shutdown_connections 2024-03-28T22:16:22.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.610+0000 7f7a0eb91740 1 -- 172.21.15.16:0/1291991945 wait complete. 2024-03-28T22:16:22.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.610+0000 7f7a0eb91740 1 -- start start 2024-03-28T22:16:22.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.610+0000 7f7a0eb91740 1 Processor -- start 2024-03-28T22:16:22.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.611+0000 7f7a0eb91740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638f9cc4f40 0x5638f9ce5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:22.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.611+0000 7f7a0eb91740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5638f9cc5860 0x5638f9ce5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:22.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.611+0000 7f7a0eb91740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5638f9cd2ba0 0x5638f9ce64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:22.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.611+0000 7f7a0eb91740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7a041a97f0 con 0x5638f9cc4f40 2024-03-28T22:16:22.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.611+0000 7f7a0eb91740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7a041aac60 con 0x5638f9cd2ba0 2024-03-28T22:16:22.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.611+0000 7f7a0eb91740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7a041a9ec0 con 0x5638f9cc5860 2024-03-28T22:16:22.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.611+0000 7f7a0eb91740 10 client.0 _ll_register_callbacks cb 0x7f79fc037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:16:22.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.611+0000 7f7a09d52640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638f9cc4f40 0x5638f9ce5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:22.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.611+0000 7f7a09d52640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638f9cc4f40 0x5638f9ce5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40490/0 (socket says 172.21.15.16:40490) 2024-03-28T22:16:22.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.611+0000 7f7a09d52640 1 -- 172.21.15.16:0/3357492138 learned_addr learned my addr 172.21.15.16:0/3357492138 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:22.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.611+0000 7f7a09551640 1 --2- 172.21.15.16:0/3357492138 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5638f9cc5860 0x5638f9ce5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:22.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.611+0000 7f7a0a553640 1 --2- 172.21.15.16:0/3357492138 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5638f9cd2ba0 0x5638f9ce64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:22.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.611+0000 7f7a09d52640 1 -- 172.21.15.16:0/3357492138 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5638f9cc5860 msgr2=0x5638f9ce5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:22.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.611+0000 7f7a09d52640 1 --2- 172.21.15.16:0/3357492138 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5638f9cc5860 0x5638f9ce5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.611+0000 7f7a09d52640 1 -- 172.21.15.16:0/3357492138 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5638f9cd2ba0 msgr2=0x5638f9ce64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:22.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.611+0000 7f7a09d52640 1 --2- 172.21.15.16:0/3357492138 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5638f9cd2ba0 0x5638f9ce64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.611+0000 7f7a09d52640 1 -- 172.21.15.16:0/3357492138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638f9d47d80 con 0x5638f9cc4f40 2024-03-28T22:16:22.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.611+0000 7f7a09551640 1 --2- 172.21.15.16:0/3357492138 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5638f9cc5860 0x5638f9ce5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:16:22.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.612+0000 7f7a09d52640 1 --2- 172.21.15.16:0/3357492138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638f9cc4f40 0x5638f9ce5a70 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f7a00098120 tx=0x7f7a00096110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8172633af40c31e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:22.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.612+0000 7f79ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:16:22.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.612+0000 7f79ed7fa640 1 -- 172.21.15.16:0/3357492138 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0001e040 con 0x5638f9cc4f40 2024-03-28T22:16:22.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.612+0000 7f79ed7fa640 1 -- 172.21.15.16:0/3357492138 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f7a000994d0 con 0x5638f9cc4f40 2024-03-28T22:16:22.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.612+0000 7f7a0eb91740 1 -- 172.21.15.16:0/3357492138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79f4013cb0 con 0x5638f9cc4f40 2024-03-28T22:16:22.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.612+0000 7f79ed7fa640 1 -- 172.21.15.16:0/3357492138 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7a00099e70 con 0x5638f9cc4f40 2024-03-28T22:16:22.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.612+0000 7f7a0eb91740 10 client.16776 fetch_fsmap learned FSMap version 27 2024-03-28T22:16:22.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.612+0000 7f7a0eb91740 1 -- 172.21.15.16:0/3357492138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5638f9d47d80 con 0x5638f9cc4f40 2024-03-28T22:16:22.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.612+0000 7f79ed7fa640 1 -- 172.21.15.16:0/3357492138 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79f4013cb0 con 0x5638f9cc4f40 2024-03-28T22:16:22.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.612+0000 7f79ed7fa640 1 -- 172.21.15.16:0/3357492138 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f7a00095bc0 con 0x5638f9cc4f40 2024-03-28T22:16:22.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.612+0000 7f7a0eb91740 10 client.16776 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:16:22.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.612+0000 7f7a0eb91740 10 client.16776 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:16:22.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.612+0000 7f7a0eb91740 20 client.16776 populate_metadata read hostname 'smithi016' 2024-03-28T22:16:22.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.612+0000 7f7a0eb91740 1 --2- 172.21.15.16:0/3357492138 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5638f9d49350 0x5638f9d4b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:22.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.612+0000 7f7a0eb91740 4 client.16776 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:16:22.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.612+0000 7f7a0eb91740 1 -- 172.21.15.16:0/3357492138 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5638f9c30a10 con 0x5638f9d49350 2024-03-28T22:16:22.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.613+0000 7f7a09551640 1 --2- 172.21.15.16:0/3357492138 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5638f9d49350 0x5638f9d4b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:22.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.613+0000 7f7a09551640 1 --2- 172.21.15.16:0/3357492138 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5638f9d49350 0x5638f9d4b770 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0388c24801285b6 server_cookie=a9d17081431f43ab in_seq=0 out_seq=0 2024-03-28T22:16:22.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.613+0000 7f79ed7fa640 10 client.16776 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:16:22.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.613+0000 7f79ed7fa640 1 -- 172.21.15.16:0/3357492138 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5638f9c30a10 con 0x5638f9d49350 2024-03-28T22:16:22.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.613+0000 7f79ed7fa640 10 client.16776 handle_command_reply: tid=0 2024-03-28T22:16:22.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.613+0000 7f7a0eb91740 1 client.16776 shutdown 2024-03-28T22:16:22.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.613+0000 7f7a0eb91740 1 -- 172.21.15.16:0/3357492138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638f9cc4f40 msgr2=0x5638f9ce5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:22.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.613+0000 7f7a0eb91740 1 --2- 172.21.15.16:0/3357492138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638f9cc4f40 0x5638f9ce5a70 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f7a00098120 tx=0x7f7a00096110 comp rx=0 tx=0).stop 2024-03-28T22:16:22.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.614+0000 7f7a0eb91740 1 -- 172.21.15.16:0/3357492138 shutdown_connections 2024-03-28T22:16:22.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.614+0000 7f7a0eb91740 1 --2- 172.21.15.16:0/3357492138 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5638f9d49350 0x5638f9d4b770 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.614+0000 7f7a0eb91740 1 --2- 172.21.15.16:0/3357492138 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5638f9cd2ba0 0x5638f9ce64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.614+0000 7f79ed7fa640 0 client.16776 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:16:22.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.614+0000 7f7a0eb91740 1 --2- 172.21.15.16:0/3357492138 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5638f9cc5860 0x5638f9ce5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.614+0000 7f7a0eb91740 1 --2- 172.21.15.16:0/3357492138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638f9cc4f40 0x5638f9ce5a70 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.614+0000 7f7a0eb91740 1 -- 172.21.15.16:0/3357492138 >> 172.21.15.16:0/3357492138 conn(0x5638f9cbf9c0 msgr2=0x5638f9cbec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:22.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.614+0000 7f7a0eb91740 1 -- 172.21.15.16:0/3357492138 shutdown_connections 2024-03-28T22:16:22.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.614+0000 7f7a0eb91740 1 -- 172.21.15.16:0/3357492138 wait complete. 2024-03-28T22:16:22.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.614+0000 7f7a0eb91740 20 client.16776 trim_cache size 0 max 16384 2024-03-28T22:16:22.626 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:16:22.626 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T22:16:22.627 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-03-28T22:16:22.627 INFO:teuthology.orchestra.run.smithi016.stdout: "669fc00c-77e6-4491-81f6-ffd0f4d8d686": { 2024-03-28T22:16:22.627 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-03-28T22:16:22.627 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "669fc00c-77e6-4491-81f6-ffd0f4d8d686", 2024-03-28T22:16:22.627 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-03-28T22:16:22.627 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:16:22.627 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:16:22.627 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:16:22.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.614+0000 7f7a0bfb6640 1 -- 172.21.15.16:0/551145451 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f79f00a1f70 msgr2=0x7f79f00a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:22.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.614+0000 7f7a0bfb6640 1 --2- 172.21.15.16:0/551145451 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f79f00a1f70 0x7f79f00a4430 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f79fc004af0 tx=0x7f79fc02c1d0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.614+0000 7f7a0bfb6640 1 -- 172.21.15.16:0/551145451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a04101ab0 msgr2=0x7f7a041980c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:22.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.614+0000 7f7a0bfb6640 1 --2- 172.21.15.16:0/551145451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a04101ab0 0x7f7a041980c0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f7a0000b4a0 tx=0x7f7a0000b960 comp rx=0 tx=0).stop 2024-03-28T22:16:22.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.614+0000 7f7a0bfb6640 1 -- 172.21.15.16:0/551145451 shutdown_connections 2024-03-28T22:16:22.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.614+0000 7f7a0bfb6640 1 --2- 172.21.15.16:0/551145451 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f79f00a1f70 0x7f79f00a4430 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.614+0000 7f7a0bfb6640 1 --2- 172.21.15.16:0/551145451 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a04103b90 0x7f7a0419c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.614+0000 7f7a0bfb6640 1 --2- 172.21.15.16:0/551145451 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a041031f0 0x7f7a04198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.614+0000 7f7a0bfb6640 1 --2- 172.21.15.16:0/551145451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a04101ab0 0x7f7a041980c0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:22.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.614+0000 7f7a0bfb6640 1 -- 172.21.15.16:0/551145451 >> 172.21.15.16:0/551145451 conn(0x7f7a040fd1e0 msgr2=0x7f7a04107560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:22.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.615+0000 7f7a0bfb6640 1 -- 172.21.15.16:0/551145451 shutdown_connections 2024-03-28T22:16:22.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:22.615+0000 7f7a0bfb6640 1 -- 172.21.15.16:0/551145451 wait complete. 2024-03-28T22:16:22.630 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:22 smithi016 ceph-mon[32374]: pgmap v176: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-28T22:16:22.631 INFO:tasks.cephfs.filesystem:scrub status for tag:669fc00c-77e6-4491-81f6-ffd0f4d8d686 - {'path': '/', 'tag': '669fc00c-77e6-4491-81f6-ffd0f4d8d686', 'options': 'recursive,force'} 2024-03-28T22:16:22.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:22 smithi063 ceph-mon[36249]: pgmap v176: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-28T22:16:22.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:22 smithi053 ceph-mon[35979]: pgmap v176: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-28T22:16:23.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:23 smithi063 ceph-mon[36249]: pgmap v177: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:16:23.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:23 smithi063 ceph-mon[36249]: scrub queued for path: / 2024-03-28T22:16:23.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:23 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:16:23.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:23 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:16:23.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:23 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:16:23.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:23 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:16:23.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:23 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:16:23.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:23 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:16:23.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:23 smithi016 ceph-mon[32374]: pgmap v177: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:16:23.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:23 smithi016 ceph-mon[32374]: scrub queued for path: / 2024-03-28T22:16:23.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:23 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:16:23.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:23 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:16:23.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:23 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:16:23.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:23 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:16:23.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:23 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:16:23.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:23 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:16:23.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:23 smithi053 ceph-mon[35979]: pgmap v177: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:16:23.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:23 smithi053 ceph-mon[35979]: scrub queued for path: / 2024-03-28T22:16:23.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:23 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:16:23.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:23 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:16:23.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:23 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:16:23.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:23 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:16:23.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:23 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:16:23.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:23 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:16:24.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:24 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:16:24.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:24 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:16:24.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:24 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:16:24.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:24 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:16:24.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:24 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:16:24.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:24 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:16:24.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:24 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:16:24.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:24 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:16:24.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:24 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:16:25.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:25 smithi063 ceph-mon[36249]: pgmap v178: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:16:25.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:25 smithi016 ceph-mon[32374]: pgmap v178: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:16:25.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:25 smithi053 ceph-mon[35979]: pgmap v178: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:16:27.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:27 smithi053 ceph-mon[35979]: pgmap v179: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:16:27.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:27 smithi063 ceph-mon[36249]: pgmap v179: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:16:27.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:27 smithi016 ceph-mon[32374]: pgmap v179: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:16:29.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:29 smithi053 ceph-mon[35979]: pgmap v180: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:16:29.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:29 smithi063 ceph-mon[36249]: pgmap v180: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:16:29.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:29 smithi016 ceph-mon[32374]: pgmap v180: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:16:31.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:31 smithi053 ceph-mon[35979]: pgmap v181: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:16:31.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:31 smithi063 ceph-mon[36249]: pgmap v181: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:16:31.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:31 smithi016 ceph-mon[32374]: pgmap v181: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:16:33.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:33 smithi053 ceph-mon[35979]: pgmap v182: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:16:33.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:33 smithi063 ceph-mon[36249]: pgmap v182: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:16:33.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:33 smithi016 ceph-mon[32374]: pgmap v182: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:16:35.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:35 smithi053 ceph-mon[35979]: pgmap v183: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:16:35.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:35 smithi053 ceph-mon[35979]: scrub summary: idle 2024-03-28T22:16:35.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:35 smithi063 ceph-mon[36249]: pgmap v183: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:16:35.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:35 smithi063 ceph-mon[36249]: scrub summary: idle 2024-03-28T22:16:35.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:35 smithi016 ceph-mon[32374]: pgmap v183: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:16:35.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:35 smithi016 ceph-mon[32374]: scrub summary: idle 2024-03-28T22:16:36.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:36 smithi053 ceph-mon[35979]: scrub summary: idle 2024-03-28T22:16:36.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:36 smithi063 ceph-mon[36249]: scrub summary: idle 2024-03-28T22:16:36.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:36 smithi016 ceph-mon[32374]: scrub summary: idle 2024-03-28T22:16:37.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:37 smithi053 ceph-mon[35979]: pgmap v184: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:16:37.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:37 smithi063 ceph-mon[36249]: pgmap v184: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:16:37.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:37 smithi016 ceph-mon[32374]: pgmap v184: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:16:39.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:39 smithi053 ceph-mon[35979]: pgmap v185: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:16:39.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:39 smithi063 ceph-mon[36249]: pgmap v185: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:16:39.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:39 smithi016 ceph-mon[32374]: pgmap v185: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:16:41.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:41 smithi053 ceph-mon[35979]: pgmap v186: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:16:41.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:41 smithi063 ceph-mon[36249]: pgmap v186: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:16:41.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:41 smithi016 ceph-mon[32374]: pgmap v186: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:16:43.739 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:43 smithi016 ceph-mon[32374]: pgmap v187: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:16:43.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:43 smithi053 ceph-mon[35979]: pgmap v187: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:16:43.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:43 smithi063 ceph-mon[36249]: pgmap v187: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:16:45.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:45 smithi053 ceph-mon[35979]: pgmap v188: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:16:45.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:45 smithi063 ceph-mon[36249]: pgmap v188: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:16:45.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:45 smithi016 ceph-mon[32374]: pgmap v188: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:16:46.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:46 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:16:46.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:46 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:16:46.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:46 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:16:46.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:46 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:16:46.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:46 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:16:46.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:46 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:16:46.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:46 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:16:46.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:46 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:16:46.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:46 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:16:46.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:46 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:16:46.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:46 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:16:46.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:46 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:16:48.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:48 smithi053 ceph-mon[35979]: pgmap v189: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:16:48.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:48 smithi063 ceph-mon[36249]: pgmap v189: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:16:48.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:48 smithi016 ceph-mon[32374]: pgmap v189: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:16:50.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:50 smithi053 ceph-mon[35979]: pgmap v190: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:16:50.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:50 smithi063 ceph-mon[36249]: pgmap v190: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:16:50.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:50 smithi016 ceph-mon[32374]: pgmap v190: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:16:52.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:52 smithi053 ceph-mon[35979]: pgmap v191: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:16:52.632 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:16:52.632 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:16:52.661 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:52 smithi016 ceph-mon[32374]: pgmap v191: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:16:52.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:52 smithi063 ceph-mon[36249]: pgmap v191: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:16:52.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.748+0000 7f3a79d25640 1 -- 172.21.15.16:0/4020698805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a74106a80 msgr2=0x7f3a7410e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:52.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.748+0000 7f3a79d25640 1 --2- 172.21.15.16:0/4020698805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a74106a80 0x7f3a7410e670 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c009900 tx=0x7f3a6c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:16:52.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.748+0000 7f3a79d25640 1 -- 172.21.15.16:0/4020698805 shutdown_connections 2024-03-28T22:16:52.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.748+0000 7f3a79d25640 1 --2- 172.21.15.16:0/4020698805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a74106a80 0x7f3a7410e670 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:52.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.748+0000 7f3a79d25640 1 --2- 172.21.15.16:0/4020698805 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a74100c70 0x7f3a741063a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:52.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.748+0000 7f3a79d25640 1 --2- 172.21.15.16:0/4020698805 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a741002c0 0x7f3a741006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:52.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.748+0000 7f3a79d25640 1 -- 172.21.15.16:0/4020698805 >> 172.21.15.16:0/4020698805 conn(0x7f3a740758a0 msgr2=0x7f3a74075cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:52.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.748+0000 7f3a79d25640 1 -- 172.21.15.16:0/4020698805 shutdown_connections 2024-03-28T22:16:52.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.748+0000 7f3a79d25640 1 -- 172.21.15.16:0/4020698805 wait complete. 2024-03-28T22:16:52.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.748+0000 7f3a79d25640 1 Processor -- start 2024-03-28T22:16:52.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.748+0000 7f3a79d25640 1 -- start start 2024-03-28T22:16:52.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.749+0000 7f3a79d25640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a741002c0 0x7f3a74198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:52.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.749+0000 7f3a79d25640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a74106a80 0x7f3a74198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:52.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.749+0000 7f3a79d25640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a7419c8d0 0x7f3a7419cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:52.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.749+0000 7f3a79d25640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3a74111b50 con 0x7f3a7419c8d0 2024-03-28T22:16:52.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.749+0000 7f3a79d25640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3a741119d0 con 0x7f3a741002c0 2024-03-28T22:16:52.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.749+0000 7f3a79d25640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3a74111cd0 con 0x7f3a74106a80 2024-03-28T22:16:52.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.749+0000 7f3a73fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a7419c8d0 0x7f3a7419cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:52.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.749+0000 7f3a737fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a741002c0 0x7f3a74198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:52.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.749+0000 7f3a73fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a7419c8d0 0x7f3a7419cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60142/0 (socket says 172.21.15.16:60142) 2024-03-28T22:16:52.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.749+0000 7f3a73fff640 1 -- 172.21.15.16:0/3034508716 learned_addr learned my addr 172.21.15.16:0/3034508716 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:52.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.749+0000 7f3a72ffd640 1 --2- 172.21.15.16:0/3034508716 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a74106a80 0x7f3a74198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:52.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.749+0000 7f3a73fff640 1 -- 172.21.15.16:0/3034508716 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a74106a80 msgr2=0x7f3a74198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:52.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.749+0000 7f3a73fff640 1 --2- 172.21.15.16:0/3034508716 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a74106a80 0x7f3a74198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:52.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.749+0000 7f3a73fff640 1 -- 172.21.15.16:0/3034508716 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a741002c0 msgr2=0x7f3a74198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:52.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.749+0000 7f3a73fff640 1 --2- 172.21.15.16:0/3034508716 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a741002c0 0x7f3a74198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:52.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.749+0000 7f3a73fff640 1 -- 172.21.15.16:0/3034508716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a7419d3c0 con 0x7f3a7419c8d0 2024-03-28T22:16:52.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.749+0000 7f3a73fff640 1 --2- 172.21.15.16:0/3034508716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a7419c8d0 0x7f3a7419cd80 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c009230 tx=0x7f3a6c02cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:52.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.750+0000 7f3a70ff9640 1 -- 172.21.15.16:0/3034508716 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c041070 con 0x7f3a7419c8d0 2024-03-28T22:16:52.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.750+0000 7f3a79d25640 1 -- 172.21.15.16:0/3034508716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a7419d650 con 0x7f3a7419c8d0 2024-03-28T22:16:52.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.750+0000 7f3a70ff9640 1 -- 172.21.15.16:0/3034508716 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a6c004450 con 0x7f3a7419c8d0 2024-03-28T22:16:52.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.750+0000 7f3a70ff9640 1 -- 172.21.15.16:0/3034508716 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c003d90 con 0x7f3a7419c8d0 2024-03-28T22:16:52.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.750+0000 7f3a79d25640 1 -- 172.21.15.16:0/3034508716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a741a4e10 con 0x7f3a7419c8d0 2024-03-28T22:16:52.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.752+0000 7f3a70ff9640 1 -- 172.21.15.16:0/3034508716 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f3a6c044070 con 0x7f3a7419c8d0 2024-03-28T22:16:52.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.752+0000 7f3a7c9e4740 1 Processor -- start 2024-03-28T22:16:52.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.752+0000 7f3a7c9e4740 1 -- start start 2024-03-28T22:16:52.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.752+0000 7f3a7c9e4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cb85b4290 0x561cb85b4670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:52.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.752+0000 7f3a7c9e4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561cb85b4bb0 0x561cb85bd250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:52.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.752+0000 7f3a7c9e4740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cb85bd790 0x561cb85bfb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:52.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.752+0000 7f3a7c9e4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561cb8518180 con 0x561cb85b4290 2024-03-28T22:16:52.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.752+0000 7f3a7c9e4740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x561cb84c0b70 con 0x561cb85bd790 2024-03-28T22:16:52.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.752+0000 7f3a7c9e4740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x561cb8514220 con 0x561cb85b4bb0 2024-03-28T22:16:52.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.752+0000 7f3a70ff9640 1 --2- 172.21.15.16:0/3034508716 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f3a5c0a1ef0 0x7f3a5c0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:52.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.752+0000 7f3a72ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cb85b4290 0x561cb85b4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:52.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.752+0000 7f3a72ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cb85b4290 0x561cb85b4670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60150/0 (socket says 172.21.15.16:60150) 2024-03-28T22:16:52.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.752+0000 7f3a72ffd640 1 -- 172.21.15.16:0/4111360961 learned_addr learned my addr 172.21.15.16:0/4111360961 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:52.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.753+0000 7f3a73fff640 1 --2- 172.21.15.16:0/3034508716 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f3a5c0a1ef0 0x7f3a5c0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:52.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.753+0000 7f3a737fe640 1 --2- 172.21.15.16:0/4111360961 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561cb85b4bb0 0x561cb85bd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:52.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.753+0000 7f3a72ffd640 1 --2- 172.21.15.16:0/4111360961 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cb85bd790 0x561cb85bfb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:52.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.753+0000 7f3a70ff9640 1 -- 172.21.15.16:0/3034508716 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f3a6c0efc70 con 0x7f3a7419c8d0 2024-03-28T22:16:52.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.753+0000 7f3a72ffd640 1 -- 172.21.15.16:0/4111360961 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561cb85b4bb0 msgr2=0x561cb85bd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:52.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.753+0000 7f3a72ffd640 1 --2- 172.21.15.16:0/4111360961 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561cb85b4bb0 0x561cb85bd250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:52.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.753+0000 7f3a72ffd640 1 -- 172.21.15.16:0/4111360961 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cb85bd790 msgr2=0x561cb85bfb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:52.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.753+0000 7f3a72ffd640 1 --2- 172.21.15.16:0/4111360961 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cb85bd790 0x561cb85bfb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:52.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.753+0000 7f3a72ffd640 1 -- 172.21.15.16:0/4111360961 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cb84d47c0 con 0x561cb85b4290 2024-03-28T22:16:52.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.753+0000 7f3a73fff640 1 --2- 172.21.15.16:0/3034508716 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f3a5c0a1ef0 0x7f3a5c0a43b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c0ed9e0 tx=0x7f3a6c0a7040 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:52.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.753+0000 7f3a72ffd640 1 --2- 172.21.15.16:0/4111360961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cb85b4290 0x561cb85b4670 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f3a680058c0 tx=0x7f3a6800b410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=929e57e968c5d1e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:52.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.753+0000 7f3a59ffb640 1 -- 172.21.15.16:0/4111360961 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6800d6e0 con 0x561cb85b4290 2024-03-28T22:16:52.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.753+0000 7f3a59ffb640 1 -- 172.21.15.16:0/4111360961 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a6800dd60 con 0x561cb85b4290 2024-03-28T22:16:52.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.753+0000 7f3a59ffb640 1 -- 172.21.15.16:0/4111360961 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a68011a40 con 0x561cb85b4290 2024-03-28T22:16:52.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.754+0000 7f3a7c9e4740 1 -- 172.21.15.16:0/4111360961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cb85b4290 msgr2=0x561cb85b4670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:52.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.754+0000 7f3a7c9e4740 1 --2- 172.21.15.16:0/4111360961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cb85b4290 0x561cb85b4670 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f3a680058c0 tx=0x7f3a6800b410 comp rx=0 tx=0).stop 2024-03-28T22:16:52.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.754+0000 7f3a7c9e4740 1 -- 172.21.15.16:0/4111360961 shutdown_connections 2024-03-28T22:16:52.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.754+0000 7f3a7c9e4740 1 --2- 172.21.15.16:0/4111360961 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cb85bd790 0x561cb85bfb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:52.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.754+0000 7f3a7c9e4740 1 --2- 172.21.15.16:0/4111360961 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561cb85b4bb0 0x561cb85bd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:52.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.754+0000 7f3a7c9e4740 1 --2- 172.21.15.16:0/4111360961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cb85b4290 0x561cb85b4670 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:52.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.754+0000 7f3a7c9e4740 1 -- 172.21.15.16:0/4111360961 >> 172.21.15.16:0/4111360961 conn(0x561cb85ae9c0 msgr2=0x561cb85aeda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:52.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.754+0000 7f3a7c9e4740 1 -- 172.21.15.16:0/4111360961 shutdown_connections 2024-03-28T22:16:52.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.754+0000 7f3a7c9e4740 1 -- 172.21.15.16:0/4111360961 wait complete. 2024-03-28T22:16:52.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.754+0000 7f3a7c9e4740 1 -- start start 2024-03-28T22:16:52.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.755+0000 7f3a7c9e4740 1 Processor -- start 2024-03-28T22:16:52.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.755+0000 7f3a7c9e4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561cb85b4290 0x561cb85ba790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:52.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.755+0000 7f3a7c9e4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cb85b4bb0 0x561cb85bacd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:52.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.755+0000 7f3a7c9e4740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cb85bd790 0x561cb85bb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:52.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.755+0000 7f3a7c9e4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561cb85c0520 con 0x561cb85b4bb0 2024-03-28T22:16:52.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.755+0000 7f3a7c9e4740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x561cb84b64a0 con 0x561cb85bd790 2024-03-28T22:16:52.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.755+0000 7f3a7c9e4740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x561cb85c06a0 con 0x561cb85b4290 2024-03-28T22:16:52.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.755+0000 7f3a7c9e4740 10 client.0 _ll_register_callbacks cb 0x561cb84b9d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:16:52.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.755+0000 7f3a737fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cb85b4bb0 0x561cb85bacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:52.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.755+0000 7f3a737fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cb85b4bb0 0x561cb85bacd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60166/0 (socket says 172.21.15.16:60166) 2024-03-28T22:16:52.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.755+0000 7f3a737fe640 1 -- 172.21.15.16:0/2759146895 learned_addr learned my addr 172.21.15.16:0/2759146895 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:52.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.755+0000 7f3a72ffd640 1 --2- 172.21.15.16:0/2759146895 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cb85bd790 0x561cb85bb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:52.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.755+0000 7f3a72ffd640 1 --2- 172.21.15.16:0/2759146895 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561cb85b4290 0x561cb85ba790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:52.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.755+0000 7f3a737fe640 1 -- 172.21.15.16:0/2759146895 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561cb85b4290 msgr2=0x561cb85ba790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:52.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.755+0000 7f3a737fe640 1 --2- 172.21.15.16:0/2759146895 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561cb85b4290 0x561cb85ba790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:52.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.755+0000 7f3a737fe640 1 -- 172.21.15.16:0/2759146895 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cb85bd790 msgr2=0x561cb85bb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:52.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.755+0000 7f3a737fe640 1 --2- 172.21.15.16:0/2759146895 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cb85bd790 0x561cb85bb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:52.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.755+0000 7f3a737fe640 1 -- 172.21.15.16:0/2759146895 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cb82f67d0 con 0x561cb85b4bb0 2024-03-28T22:16:52.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.755+0000 7f3a737fe640 1 --2- 172.21.15.16:0/2759146895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cb85b4bb0 0x561cb85bacd0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f3a64011d10 tx=0x7f3a6400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7d629335b005f36 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:52.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.755+0000 7f3a52ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:16:52.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.756+0000 7f3a52ffd640 1 -- 172.21.15.16:0/2759146895 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6401f060 con 0x561cb85b4bb0 2024-03-28T22:16:52.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.756+0000 7f3a52ffd640 1 -- 172.21.15.16:0/2759146895 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f3a64017050 con 0x561cb85b4bb0 2024-03-28T22:16:52.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.756+0000 7f3a7c9e4740 1 -- 172.21.15.16:0/2759146895 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a6800dd60 con 0x561cb85b4bb0 2024-03-28T22:16:52.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.756+0000 7f3a52ffd640 1 -- 172.21.15.16:0/2759146895 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3a640233f0 con 0x561cb85b4bb0 2024-03-28T22:16:52.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.756+0000 7f3a7c9e4740 10 client.16788 fetch_fsmap learned FSMap version 27 2024-03-28T22:16:52.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.756+0000 7f3a7c9e4740 1 -- 172.21.15.16:0/2759146895 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561cb822c3f0 con 0x561cb85b4bb0 2024-03-28T22:16:52.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.756+0000 7f3a52ffd640 1 -- 172.21.15.16:0/2759146895 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a6800dd60 con 0x561cb85b4bb0 2024-03-28T22:16:52.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.756+0000 7f3a52ffd640 1 -- 172.21.15.16:0/2759146895 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f3a64019030 con 0x561cb85b4bb0 2024-03-28T22:16:52.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.756+0000 7f3a7c9e4740 10 client.16788 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:16:52.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.756+0000 7f3a7c9e4740 10 client.16788 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:16:52.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.756+0000 7f3a7c9e4740 20 client.16788 populate_metadata read hostname 'smithi016' 2024-03-28T22:16:52.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.756+0000 7f3a7c9e4740 1 --2- 172.21.15.16:0/2759146895 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x561cb85cdb50 0x561cb85cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:52.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.756+0000 7f3a7c9e4740 4 client.16788 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:16:52.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.756+0000 7f3a7c9e4740 1 -- 172.21.15.16:0/2759146895 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561cb85256c0 con 0x561cb85cdb50 2024-03-28T22:16:52.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.756+0000 7f3a72ffd640 1 --2- 172.21.15.16:0/2759146895 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x561cb85cdb50 0x561cb85cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:52.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.757+0000 7f3a72ffd640 1 --2- 172.21.15.16:0/2759146895 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x561cb85cdb50 0x561cb85cff40 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1edff05904b9a7b2 server_cookie=39857fe0710796b2 in_seq=0 out_seq=0 2024-03-28T22:16:52.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.757+0000 7f3a52ffd640 10 client.16788 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:16:52.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.757+0000 7f3a52ffd640 1 -- 172.21.15.16:0/2759146895 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561cb85256c0 con 0x561cb85cdb50 2024-03-28T22:16:52.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.757+0000 7f3a52ffd640 10 client.16788 handle_command_reply: tid=0 2024-03-28T22:16:52.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.758+0000 7f3a7c9e4740 1 client.16788 shutdown 2024-03-28T22:16:52.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.758+0000 7f3a7c9e4740 1 -- 172.21.15.16:0/2759146895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cb85b4bb0 msgr2=0x561cb85bacd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:52.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.758+0000 7f3a7c9e4740 1 --2- 172.21.15.16:0/2759146895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cb85b4bb0 0x561cb85bacd0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f3a64011d10 tx=0x7f3a6400f790 comp rx=0 tx=0).stop 2024-03-28T22:16:52.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.758+0000 7f3a7c9e4740 1 -- 172.21.15.16:0/2759146895 shutdown_connections 2024-03-28T22:16:52.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.758+0000 7f3a7c9e4740 1 --2- 172.21.15.16:0/2759146895 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x561cb85cdb50 0x561cb85cff40 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:52.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.758+0000 7f3a7c9e4740 1 --2- 172.21.15.16:0/2759146895 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cb85bd790 0x561cb85bb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:52.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.758+0000 7f3a52ffd640 0 client.16788 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:16:52.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.758+0000 7f3a7c9e4740 1 --2- 172.21.15.16:0/2759146895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cb85b4bb0 0x561cb85bacd0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:52.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.758+0000 7f3a7c9e4740 1 --2- 172.21.15.16:0/2759146895 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561cb85b4290 0x561cb85ba790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:52.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.758+0000 7f3a7c9e4740 1 -- 172.21.15.16:0/2759146895 >> 172.21.15.16:0/2759146895 conn(0x561cb85ae9c0 msgr2=0x561cb85b0710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:52.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.758+0000 7f3a7c9e4740 1 -- 172.21.15.16:0/2759146895 shutdown_connections 2024-03-28T22:16:52.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.758+0000 7f3a7c9e4740 1 -- 172.21.15.16:0/2759146895 wait complete. 2024-03-28T22:16:52.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.758+0000 7f3a7c9e4740 20 client.16788 trim_cache size 0 max 16384 2024-03-28T22:16:52.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.776+0000 7f3a7c9e4740 1 Processor -- start 2024-03-28T22:16:52.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.776+0000 7f3a7c9e4740 1 -- start start 2024-03-28T22:16:52.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.776+0000 7f3a7c9e4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561cb85b4290 0x561cb85bcec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:52.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.776+0000 7f3a7c9e4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cb85b4bb0 0x561cb85bb030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:52.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.776+0000 7f3a7c9e4740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cb85bd790 0x561cb85d5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:52.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.776+0000 7f3a7c9e4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3a741afbe0 con 0x561cb85b4bb0 2024-03-28T22:16:52.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.776+0000 7f3a7c9e4740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3a741ad4b0 con 0x561cb85bd790 2024-03-28T22:16:52.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.776+0000 7f3a7c9e4740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3a741ab320 con 0x561cb85b4290 2024-03-28T22:16:52.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.776+0000 7f3a737fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cb85b4bb0 0x561cb85bb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:52.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.776+0000 7f3a737fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cb85b4bb0 0x561cb85bb030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60168/0 (socket says 172.21.15.16:60168) 2024-03-28T22:16:52.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.776+0000 7f3a737fe640 1 -- 172.21.15.16:0/2819827594 learned_addr learned my addr 172.21.15.16:0/2819827594 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:52.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.776+0000 7f3a72ffd640 1 --2- 172.21.15.16:0/2819827594 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cb85bd790 0x561cb85d5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:52.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.776+0000 7f3a737fe640 1 -- 172.21.15.16:0/2819827594 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561cb85b4290 msgr2=0x561cb85bcec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:52.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.776+0000 7f3a737fe640 1 --2- 172.21.15.16:0/2819827594 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561cb85b4290 0x561cb85bcec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:52.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.776+0000 7f3a737fe640 1 -- 172.21.15.16:0/2819827594 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cb85bd790 msgr2=0x561cb85d5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:52.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.776+0000 7f3a737fe640 1 --2- 172.21.15.16:0/2819827594 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cb85bd790 0x561cb85d5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:52.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.776+0000 7f3a737fe640 1 -- 172.21.15.16:0/2819827594 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cb81ec0f0 con 0x561cb85b4bb0 2024-03-28T22:16:52.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.776+0000 7f3a737fe640 1 --2- 172.21.15.16:0/2819827594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cb85b4bb0 0x561cb85bb030 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f3a64009a00 tx=0x7f3a6400fc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44b155ff9a45e76a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:52.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.777+0000 7f3a597fa640 1 -- 172.21.15.16:0/2819827594 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6400ed70 con 0x561cb85b4bb0 2024-03-28T22:16:52.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.777+0000 7f3a597fa640 1 -- 172.21.15.16:0/2819827594 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a64003e50 con 0x561cb85b4bb0 2024-03-28T22:16:52.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.777+0000 7f3a597fa640 1 -- 172.21.15.16:0/2819827594 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a64023910 con 0x561cb85b4bb0 2024-03-28T22:16:52.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.777+0000 7f3a7c9e4740 1 -- 172.21.15.16:0/2819827594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cb85b4bb0 msgr2=0x561cb85bb030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:52.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.777+0000 7f3a7c9e4740 1 --2- 172.21.15.16:0/2819827594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cb85b4bb0 0x561cb85bb030 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f3a64009a00 tx=0x7f3a6400fc90 comp rx=0 tx=0).stop 2024-03-28T22:16:52.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.777+0000 7f3a7c9e4740 1 -- 172.21.15.16:0/2819827594 shutdown_connections 2024-03-28T22:16:52.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.777+0000 7f3a7c9e4740 1 --2- 172.21.15.16:0/2819827594 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cb85bd790 0x561cb85d5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:52.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.777+0000 7f3a7c9e4740 1 --2- 172.21.15.16:0/2819827594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cb85b4bb0 0x561cb85bb030 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:52.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.777+0000 7f3a7c9e4740 1 --2- 172.21.15.16:0/2819827594 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561cb85b4290 0x561cb85bcec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:52.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.777+0000 7f3a7c9e4740 1 -- 172.21.15.16:0/2819827594 >> 172.21.15.16:0/2819827594 conn(0x561cb85ae9c0 msgr2=0x561cb85afcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:52.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.777+0000 7f3a7c9e4740 1 -- 172.21.15.16:0/2819827594 shutdown_connections 2024-03-28T22:16:52.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.777+0000 7f3a7c9e4740 1 -- 172.21.15.16:0/2819827594 wait complete. 2024-03-28T22:16:52.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.778+0000 7f3a7c9e4740 1 -- start start 2024-03-28T22:16:52.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.778+0000 7f3a7c9e4740 1 Processor -- start 2024-03-28T22:16:52.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.778+0000 7f3a7c9e4740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cb85b4290 0x561cb85c13c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:52.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.778+0000 7f3a7c9e4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cb85b4bb0 0x561cb85c1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:52.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.778+0000 7f3a7c9e4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561cb85bd790 0x561cb85c1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:52.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.778+0000 7f3a7c9e4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3a741ab970 con 0x561cb85b4bb0 2024-03-28T22:16:52.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.778+0000 7f3a7c9e4740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3a741acde0 con 0x561cb85b4290 2024-03-28T22:16:52.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.778+0000 7f3a7c9e4740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3a741ac040 con 0x561cb85bd790 2024-03-28T22:16:52.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.778+0000 7f3a7c9e4740 10 client.0 _ll_register_callbacks cb 0x7f3a64009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:16:52.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.778+0000 7f3a737fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cb85b4bb0 0x561cb85c1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:52.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.778+0000 7f3a737fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cb85b4bb0 0x561cb85c1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60170/0 (socket says 172.21.15.16:60170) 2024-03-28T22:16:52.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.778+0000 7f3a737fe640 1 -- 172.21.15.16:0/3075195252 learned_addr learned my addr 172.21.15.16:0/3075195252 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:52.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.778+0000 7f3a72ffd640 1 --2- 172.21.15.16:0/3075195252 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cb85b4290 0x561cb85c13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:52.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.778+0000 7f3a72ffd640 1 --2- 172.21.15.16:0/3075195252 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561cb85bd790 0x561cb85c1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:52.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.778+0000 7f3a737fe640 1 -- 172.21.15.16:0/3075195252 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561cb85bd790 msgr2=0x561cb85c1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:52.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.778+0000 7f3a737fe640 1 --2- 172.21.15.16:0/3075195252 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561cb85bd790 0x561cb85c1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:52.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.778+0000 7f3a737fe640 1 -- 172.21.15.16:0/3075195252 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cb85b4290 msgr2=0x561cb85c13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:52.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.778+0000 7f3a737fe640 1 --2- 172.21.15.16:0/3075195252 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cb85b4290 0x561cb85c13c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:52.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.778+0000 7f3a737fe640 1 -- 172.21.15.16:0/3075195252 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cb8632fe0 con 0x561cb85b4bb0 2024-03-28T22:16:52.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.779+0000 7f3a737fe640 1 --2- 172.21.15.16:0/3075195252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cb85b4bb0 0x561cb85c1900 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f3a64013040 tx=0x7f3a6400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eadbf0ae6fc0a0fc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:52.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.779+0000 7f3a52ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:16:52.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.779+0000 7f3a52ffd640 1 -- 172.21.15.16:0/3075195252 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6401f070 con 0x561cb85b4bb0 2024-03-28T22:16:52.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.779+0000 7f3a52ffd640 1 -- 172.21.15.16:0/3075195252 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f3a64011e00 con 0x561cb85b4bb0 2024-03-28T22:16:52.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.779+0000 7f3a7c9e4740 1 -- 172.21.15.16:0/3075195252 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a64003e50 con 0x561cb85b4bb0 2024-03-28T22:16:52.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.779+0000 7f3a52ffd640 1 -- 172.21.15.16:0/3075195252 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3a6400e920 con 0x561cb85b4bb0 2024-03-28T22:16:52.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.779+0000 7f3a7c9e4740 10 client.16794 fetch_fsmap learned FSMap version 27 2024-03-28T22:16:52.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.779+0000 7f3a7c9e4740 1 -- 172.21.15.16:0/3075195252 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561cb86342d0 con 0x561cb85b4bb0 2024-03-28T22:16:52.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.779+0000 7f3a52ffd640 1 -- 172.21.15.16:0/3075195252 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a64003e50 con 0x561cb85b4bb0 2024-03-28T22:16:52.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.779+0000 7f3a52ffd640 1 -- 172.21.15.16:0/3075195252 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f3a6403c030 con 0x561cb85b4bb0 2024-03-28T22:16:52.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.779+0000 7f3a7c9e4740 10 client.16794 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:16:52.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.779+0000 7f3a7c9e4740 10 client.16794 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:16:52.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.779+0000 7f3a7c9e4740 20 client.16794 populate_metadata read hostname 'smithi016' 2024-03-28T22:16:52.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.779+0000 7f3a7c9e4740 1 --2- 172.21.15.16:0/3075195252 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x561cb8634a00 0x561cb8636e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:52.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.779+0000 7f3a7c9e4740 4 client.16794 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:16:52.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.779+0000 7f3a7c9e4740 1 -- 172.21.15.16:0/3075195252 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561cb851fa10 con 0x561cb8634a00 2024-03-28T22:16:52.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.779+0000 7f3a72ffd640 1 --2- 172.21.15.16:0/3075195252 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x561cb8634a00 0x561cb8636e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:52.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.780+0000 7f3a72ffd640 1 --2- 172.21.15.16:0/3075195252 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x561cb8634a00 0x561cb8636e20 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfb7ab6a07d23032 server_cookie=d9be1291b475840b in_seq=0 out_seq=0 2024-03-28T22:16:52.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.780+0000 7f3a52ffd640 10 client.16794 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:16:52.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.780+0000 7f3a52ffd640 1 -- 172.21.15.16:0/3075195252 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x561cb851fa10 con 0x561cb8634a00 2024-03-28T22:16:52.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.780+0000 7f3a52ffd640 10 client.16794 handle_command_reply: tid=0 2024-03-28T22:16:52.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.780+0000 7f3a7c9e4740 1 client.16794 shutdown 2024-03-28T22:16:52.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.780+0000 7f3a7c9e4740 1 -- 172.21.15.16:0/3075195252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cb85b4bb0 msgr2=0x561cb85c1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:52.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.780+0000 7f3a7c9e4740 1 --2- 172.21.15.16:0/3075195252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cb85b4bb0 0x561cb85c1900 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f3a64013040 tx=0x7f3a6400ef60 comp rx=0 tx=0).stop 2024-03-28T22:16:52.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.780+0000 7f3a7c9e4740 1 -- 172.21.15.16:0/3075195252 shutdown_connections 2024-03-28T22:16:52.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.780+0000 7f3a7c9e4740 1 --2- 172.21.15.16:0/3075195252 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x561cb8634a00 0x561cb8636e20 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:52.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.780+0000 7f3a7c9e4740 1 --2- 172.21.15.16:0/3075195252 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561cb85bd790 0x561cb85c1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:52.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.780+0000 7f3a52ffd640 0 client.16794 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:16:52.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.780+0000 7f3a7c9e4740 1 --2- 172.21.15.16:0/3075195252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561cb85b4bb0 0x561cb85c1900 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:52.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.780+0000 7f3a7c9e4740 1 --2- 172.21.15.16:0/3075195252 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cb85b4290 0x561cb85c13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:52.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.780+0000 7f3a7c9e4740 1 -- 172.21.15.16:0/3075195252 >> 172.21.15.16:0/3075195252 conn(0x561cb85ae9c0 msgr2=0x561cb85b0550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:52.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.780+0000 7f3a7c9e4740 1 -- 172.21.15.16:0/3075195252 shutdown_connections 2024-03-28T22:16:52.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.780+0000 7f3a7c9e4740 1 -- 172.21.15.16:0/3075195252 wait complete. 2024-03-28T22:16:52.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.780+0000 7f3a7c9e4740 20 client.16794 trim_cache size 0 max 16384 2024-03-28T22:16:52.796 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:16:52.797 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-03-28T22:16:52.797 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-03-28T22:16:52.797 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:16:52.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.781+0000 7f3a79d25640 1 -- 172.21.15.16:0/3034508716 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f3a5c0a1ef0 msgr2=0x7f3a5c0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:52.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.781+0000 7f3a79d25640 1 --2- 172.21.15.16:0/3034508716 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f3a5c0a1ef0 0x7f3a5c0a43b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c0ed9e0 tx=0x7f3a6c0a7040 comp rx=0 tx=0).stop 2024-03-28T22:16:52.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.781+0000 7f3a79d25640 1 -- 172.21.15.16:0/3034508716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a7419c8d0 msgr2=0x7f3a7419cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:52.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.781+0000 7f3a79d25640 1 --2- 172.21.15.16:0/3034508716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a7419c8d0 0x7f3a7419cd80 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c009230 tx=0x7f3a6c02cba0 comp rx=0 tx=0).stop 2024-03-28T22:16:52.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.781+0000 7f3a79d25640 1 -- 172.21.15.16:0/3034508716 shutdown_connections 2024-03-28T22:16:52.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.781+0000 7f3a79d25640 1 --2- 172.21.15.16:0/3034508716 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f3a5c0a1ef0 0x7f3a5c0a43b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:52.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.781+0000 7f3a79d25640 1 --2- 172.21.15.16:0/3034508716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a7419c8d0 0x7f3a7419cd80 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:52.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.781+0000 7f3a79d25640 1 --2- 172.21.15.16:0/3034508716 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a74106a80 0x7f3a74198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:52.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.781+0000 7f3a79d25640 1 --2- 172.21.15.16:0/3034508716 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a741002c0 0x7f3a74198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:52.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.781+0000 7f3a79d25640 1 -- 172.21.15.16:0/3034508716 >> 172.21.15.16:0/3034508716 conn(0x7f3a740758a0 msgr2=0x7f3a740fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:52.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.781+0000 7f3a79d25640 1 -- 172.21.15.16:0/3034508716 shutdown_connections 2024-03-28T22:16:52.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.781+0000 7f3a79d25640 1 -- 172.21.15.16:0/3034508716 wait complete. 2024-03-28T22:16:52.800 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:16:52.800 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:16:52.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.939+0000 7f9f517d0640 1 -- 172.21.15.16:0/830741671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f4c100600 msgr2=0x7f9f4c100a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:52.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.939+0000 7f9f517d0640 1 --2- 172.21.15.16:0/830741671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f4c100600 0x7f9f4c100a80 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f9f34009930 tx=0x7f9f3402bc30 comp rx=0 tx=0).stop 2024-03-28T22:16:52.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.939+0000 7f9f517d0640 1 -- 172.21.15.16:0/830741671 shutdown_connections 2024-03-28T22:16:52.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.939+0000 7f9f517d0640 1 --2- 172.21.15.16:0/830741671 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f4c100fc0 0x7f9f4c10d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:52.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.939+0000 7f9f517d0640 1 --2- 172.21.15.16:0/830741671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f4c100600 0x7f9f4c100a80 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:52.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.939+0000 7f9f517d0640 1 --2- 172.21.15.16:0/830741671 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f4c1066a0 0x7f9f4c106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:52.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.939+0000 7f9f517d0640 1 -- 172.21.15.16:0/830741671 >> 172.21.15.16:0/830741671 conn(0x7f9f4c0777d0 msgr2=0x7f9f4c077be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:52.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.939+0000 7f9f517d0640 1 -- 172.21.15.16:0/830741671 shutdown_connections 2024-03-28T22:16:52.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.939+0000 7f9f517d0640 1 -- 172.21.15.16:0/830741671 wait complete. 2024-03-28T22:16:52.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.940+0000 7f9f517d0640 1 Processor -- start 2024-03-28T22:16:52.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.940+0000 7f9f517d0640 1 -- start start 2024-03-28T22:16:52.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.940+0000 7f9f517d0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f4c100600 0x7f9f4c196ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:52.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.940+0000 7f9f517d0640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f4c100fc0 0x7f9f4c197420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:52.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.940+0000 7f9f517d0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f4c1066a0 0x7f9f4c19b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:52.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.940+0000 7f9f517d0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9f4c1109e0 con 0x7f9f4c100600 2024-03-28T22:16:52.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.940+0000 7f9f517d0640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9f4c110860 con 0x7f9f4c100fc0 2024-03-28T22:16:52.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.940+0000 7f9f517d0640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9f4c110b60 con 0x7f9f4c1066a0 2024-03-28T22:16:52.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.940+0000 7f9f4affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f4c100600 0x7f9f4c196ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:52.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.940+0000 7f9f43fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f4c100fc0 0x7f9f4c197420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:52.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.940+0000 7f9f4affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f4c100600 0x7f9f4c196ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60198/0 (socket says 172.21.15.16:60198) 2024-03-28T22:16:52.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.940+0000 7f9f4affd640 1 -- 172.21.15.16:0/4005563281 learned_addr learned my addr 172.21.15.16:0/4005563281 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:52.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.940+0000 7f9f4b7fe640 1 --2- 172.21.15.16:0/4005563281 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f4c1066a0 0x7f9f4c19b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:52.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.941+0000 7f9f4affd640 1 -- 172.21.15.16:0/4005563281 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f4c1066a0 msgr2=0x7f9f4c19b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:52.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.941+0000 7f9f4affd640 1 --2- 172.21.15.16:0/4005563281 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f4c1066a0 0x7f9f4c19b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:52.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.941+0000 7f9f4affd640 1 -- 172.21.15.16:0/4005563281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f4c100fc0 msgr2=0x7f9f4c197420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:52.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.941+0000 7f9f4affd640 1 --2- 172.21.15.16:0/4005563281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f4c100fc0 0x7f9f4c197420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:52.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.941+0000 7f9f4affd640 1 -- 172.21.15.16:0/4005563281 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f4c19bed0 con 0x7f9f4c100600 2024-03-28T22:16:52.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.941+0000 7f9f43fff640 1 --2- 172.21.15.16:0/4005563281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f4c100fc0 0x7f9f4c197420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:16:52.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.941+0000 7f9f4affd640 1 --2- 172.21.15.16:0/4005563281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f4c100600 0x7f9f4c196ee0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f9f3c00d950 tx=0x7f9f3c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:52.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.941+0000 7f9f48ff9640 1 -- 172.21.15.16:0/4005563281 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f3c017020 con 0x7f9f4c100600 2024-03-28T22:16:52.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.941+0000 7f9f48ff9640 1 -- 172.21.15.16:0/4005563281 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f3c009c30 con 0x7f9f4c100600 2024-03-28T22:16:52.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.941+0000 7f9f517d0640 1 -- 172.21.15.16:0/4005563281 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f4c19c1c0 con 0x7f9f4c100600 2024-03-28T22:16:52.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.941+0000 7f9f48ff9640 1 -- 172.21.15.16:0/4005563281 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f3c00f720 con 0x7f9f4c100600 2024-03-28T22:16:52.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.941+0000 7f9f517d0640 1 -- 172.21.15.16:0/4005563281 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f4c1a3a50 con 0x7f9f4c100600 2024-03-28T22:16:52.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.942+0000 7f9f517d0640 1 -- 172.21.15.16:0/4005563281 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f4c068a20 con 0x7f9f4c100600 2024-03-28T22:16:52.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.944+0000 7f9f48ff9640 1 -- 172.21.15.16:0/4005563281 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f9f3c01a020 con 0x7f9f4c100600 2024-03-28T22:16:52.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.944+0000 7f9f48ff9640 1 --2- 172.21.15.16:0/4005563281 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f9f380a1ef0 0x7f9f380a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:52.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.944+0000 7f9f48ff9640 1 -- 172.21.15.16:0/4005563281 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f9f3c0ce4e0 con 0x7f9f4c100600 2024-03-28T22:16:52.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.944+0000 7f9f43fff640 1 --2- 172.21.15.16:0/4005563281 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f9f380a1ef0 0x7f9f380a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:52.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.945+0000 7f9f43fff640 1 --2- 172.21.15.16:0/4005563281 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f9f380a1ef0 0x7f9f380a43b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9f34005ab0 tx=0x7f9f34005fd0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:52.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:52.948+0000 7f9f48ff9640 1 -- 172.21.15.16:0/4005563281 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9f3c021020 con 0x7f9f4c100600 2024-03-28T22:16:53.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.169+0000 7f9f517d0640 1 -- 172.21.15.16:0/4005563281 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9f4c198230 con 0x7f9f4c100600 2024-03-28T22:16:53.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.169+0000 7f9f48ff9640 1 -- 172.21.15.16:0/4005563281 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11851 (secure 0 0 0) 0x7f9f3c009de0 con 0x7f9f4c100600 2024-03-28T22:16:53.172 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:16:53.173 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/1644581227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1644581227},{"type":"v1","addr":"172.21.15.16:6839","nonce":1644581227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/2593062669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2593062669},{"type":"v1","addr":"172.21.15.16:6841","nonce":2593062669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:08:29.672976+0000","modified":"2024-03-28T22:14:44.922044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24566,"mds_1":24524,"mds_2":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24466":{"gid":24466,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/247249857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":247249857},{"type":"v1","addr":"172.21.15.63:6835","nonce":247249857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6837/2324231032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":2324231032},{"type":"v1","addr":"172.21.15.63:6837","nonce":2324231032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6839/1177197077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":1177197077},{"type":"v1","addr":"172.21.15.63:6839","nonce":1177197077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"e","rank":2,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.53:6835/350727634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":350727634},{"type":"v1","addr":"172.21.15.53:6835","nonce":350727634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/349877573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":349877573},{"type":"v1","addr":"172.21.15.53:6837","nonce":349877573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6839/3714153956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":3714153956},{"type":"v1","addr":"172.21.15.53:6839","nonce":3714153956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24566,"qdb_cluster":[24500,24524,24566]},"id":1}]} 2024-03-28T22:16:53.173 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2024-03-28T22:16:53.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.171+0000 7f9f517d0640 1 -- 172.21.15.16:0/4005563281 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f9f380a1ef0 msgr2=0x7f9f380a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:53.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.171+0000 7f9f517d0640 1 --2- 172.21.15.16:0/4005563281 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f9f380a1ef0 0x7f9f380a43b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9f34005ab0 tx=0x7f9f34005fd0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.171+0000 7f9f517d0640 1 -- 172.21.15.16:0/4005563281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f4c100600 msgr2=0x7f9f4c196ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:53.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.171+0000 7f9f517d0640 1 --2- 172.21.15.16:0/4005563281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f4c100600 0x7f9f4c196ee0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f9f3c00d950 tx=0x7f9f3c00de10 comp rx=0 tx=0).stop 2024-03-28T22:16:53.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.171+0000 7f9f517d0640 1 -- 172.21.15.16:0/4005563281 shutdown_connections 2024-03-28T22:16:53.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.171+0000 7f9f517d0640 1 --2- 172.21.15.16:0/4005563281 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f9f380a1ef0 0x7f9f380a43b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.171+0000 7f9f517d0640 1 --2- 172.21.15.16:0/4005563281 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f4c1066a0 0x7f9f4c19b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.171+0000 7f9f517d0640 1 --2- 172.21.15.16:0/4005563281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f4c100fc0 0x7f9f4c197420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.171+0000 7f9f517d0640 1 --2- 172.21.15.16:0/4005563281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f4c100600 0x7f9f4c196ee0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.171+0000 7f9f517d0640 1 -- 172.21.15.16:0/4005563281 >> 172.21.15.16:0/4005563281 conn(0x7f9f4c0777d0 msgr2=0x7f9f4c0ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:53.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.171+0000 7f9f517d0640 1 -- 172.21.15.16:0/4005563281 shutdown_connections 2024-03-28T22:16:53.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.171+0000 7f9f517d0640 1 -- 172.21.15.16:0/4005563281 wait complete. 2024-03-28T22:16:53.185 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:16:53.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.287+0000 7fdc31a47640 1 -- 172.21.15.16:0/4152206569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc2c1020b0 msgr2=0x7fdc2c10e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:53.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.287+0000 7fdc31a47640 1 --2- 172.21.15.16:0/4152206569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc2c1020b0 0x7fdc2c10e5c0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c009cd0 tx=0x7fdc1c02c0f0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.287+0000 7fdc31a47640 1 -- 172.21.15.16:0/4152206569 shutdown_connections 2024-03-28T22:16:53.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.287+0000 7fdc31a47640 1 --2- 172.21.15.16:0/4152206569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc2c1020b0 0x7fdc2c10e5c0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.287+0000 7fdc31a47640 1 --2- 172.21.15.16:0/4152206569 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc2c1016f0 0x7fdc2c101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.287+0000 7fdc31a47640 1 --2- 172.21.15.16:0/4152206569 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc2c107700 0x7fdc2c107ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.287+0000 7fdc31a47640 1 -- 172.21.15.16:0/4152206569 >> 172.21.15.16:0/4152206569 conn(0x7fdc2c0fd1e0 msgr2=0x7fdc2c0ff600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:53.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.287+0000 7fdc31a47640 1 -- 172.21.15.16:0/4152206569 shutdown_connections 2024-03-28T22:16:53.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.287+0000 7fdc31a47640 1 -- 172.21.15.16:0/4152206569 wait complete. 2024-03-28T22:16:53.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.288+0000 7fdc31a47640 1 Processor -- start 2024-03-28T22:16:53.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.288+0000 7fdc31a47640 1 -- start start 2024-03-28T22:16:53.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.288+0000 7fdc31a47640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc2c1016f0 0x7fdc2c197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.288+0000 7fdc31a47640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc2c1020b0 0x7fdc2c198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.288+0000 7fdc31a47640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc2c107700 0x7fdc2c19c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.288+0000 7fdc31a47640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdc2c111b10 con 0x7fdc2c107700 2024-03-28T22:16:53.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.288+0000 7fdc31a47640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fdc2c111990 con 0x7fdc2c1016f0 2024-03-28T22:16:53.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.288+0000 7fdc31a47640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fdc2c111c90 con 0x7fdc2c1020b0 2024-03-28T22:16:53.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.288+0000 7fdc2b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc2c107700 0x7fdc2c19c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.288+0000 7fdc2affd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc2c1016f0 0x7fdc2c197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.288+0000 7fdc2b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc2c107700 0x7fdc2c19c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60216/0 (socket says 172.21.15.16:60216) 2024-03-28T22:16:53.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.288+0000 7fdc2b7fe640 1 -- 172.21.15.16:0/2411114108 learned_addr learned my addr 172.21.15.16:0/2411114108 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:53.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.289+0000 7fdc2a7fc640 1 --2- 172.21.15.16:0/2411114108 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc2c1020b0 0x7fdc2c198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.289+0000 7fdc2b7fe640 1 -- 172.21.15.16:0/2411114108 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc2c1020b0 msgr2=0x7fdc2c198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:53.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.289+0000 7fdc2b7fe640 1 --2- 172.21.15.16:0/2411114108 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc2c1020b0 0x7fdc2c198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.289+0000 7fdc2b7fe640 1 -- 172.21.15.16:0/2411114108 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc2c1016f0 msgr2=0x7fdc2c197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:53.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.289+0000 7fdc2b7fe640 1 --2- 172.21.15.16:0/2411114108 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc2c1016f0 0x7fdc2c197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.289+0000 7fdc2b7fe640 1 -- 172.21.15.16:0/2411114108 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc2c19cfc0 con 0x7fdc2c107700 2024-03-28T22:16:53.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.289+0000 7fdc2b7fe640 1 --2- 172.21.15.16:0/2411114108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc2c107700 0x7fdc2c19c890 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c004af0 tx=0x7fdc1c032ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:53.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.289+0000 7fdc30a45640 1 -- 172.21.15.16:0/2411114108 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c041020 con 0x7fdc2c107700 2024-03-28T22:16:53.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.289+0000 7fdc30a45640 1 -- 172.21.15.16:0/2411114108 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc1c034cd0 con 0x7fdc2c107700 2024-03-28T22:16:53.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.289+0000 7fdc31a47640 1 -- 172.21.15.16:0/2411114108 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc2c19d250 con 0x7fdc2c107700 2024-03-28T22:16:53.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.289+0000 7fdc30a45640 1 -- 172.21.15.16:0/2411114108 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c039040 con 0x7fdc2c107700 2024-03-28T22:16:53.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.289+0000 7fdc31a47640 1 -- 172.21.15.16:0/2411114108 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc2c1a4b40 con 0x7fdc2c107700 2024-03-28T22:16:53.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.292+0000 7fdc30a45640 1 -- 172.21.15.16:0/2411114108 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7fdc1c044020 con 0x7fdc2c107700 2024-03-28T22:16:53.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.292+0000 7fdc3462e740 1 Processor -- start 2024-03-28T22:16:53.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.292+0000 7fdc3462e740 1 -- start start 2024-03-28T22:16:53.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.292+0000 7fdc3462e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558221c24470 0x558221c24850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.292+0000 7fdc3462e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558221c24d90 0x558221c2d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.292+0000 7fdc3462e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558221c2d970 0x558221c2fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.292+0000 7fdc3462e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558221b881b0 con 0x558221c24d90 2024-03-28T22:16:53.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.292+0000 7fdc3462e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x558221b30c00 con 0x558221c2d970 2024-03-28T22:16:53.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.292+0000 7fdc3462e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x558221b84250 con 0x558221c24470 2024-03-28T22:16:53.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.292+0000 7fdc2affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558221c24d90 0x558221c2d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.292+0000 7fdc2affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558221c24d90 0x558221c2d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60232/0 (socket says 172.21.15.16:60232) 2024-03-28T22:16:53.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.292+0000 7fdc2affd640 1 -- 172.21.15.16:0/3037998643 learned_addr learned my addr 172.21.15.16:0/3037998643 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:53.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.292+0000 7fdc2a7fc640 1 --2- 172.21.15.16:0/3037998643 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558221c2d970 0x558221c2fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.292+0000 7fdc30a45640 1 --2- 172.21.15.16:0/2411114108 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fdc180a1ef0 0x7fdc180a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.292+0000 7fdc2a7fc640 1 --2- 172.21.15.16:0/3037998643 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558221c24470 0x558221c24850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.293+0000 7fdc2b7fe640 1 --2- 172.21.15.16:0/2411114108 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fdc180a1ef0 0x7fdc180a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.293+0000 7fdc2affd640 1 -- 172.21.15.16:0/3037998643 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558221c24470 msgr2=0x558221c24850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:53.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.293+0000 7fdc2affd640 1 --2- 172.21.15.16:0/3037998643 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558221c24470 0x558221c24850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.293+0000 7fdc2affd640 1 -- 172.21.15.16:0/3037998643 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558221c2d970 msgr2=0x558221c2fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:53.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.293+0000 7fdc2affd640 1 --2- 172.21.15.16:0/3037998643 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558221c2d970 0x558221c2fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.293+0000 7fdc2affd640 1 -- 172.21.15.16:0/3037998643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558221b503d0 con 0x558221c24d90 2024-03-28T22:16:53.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.293+0000 7fdc2a7fc640 1 --2- 172.21.15.16:0/3037998643 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558221c2d970 0x558221c2fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:16:53.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.293+0000 7fdc30a45640 1 -- 172.21.15.16:0/2411114108 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fdc1c0efda0 con 0x7fdc2c107700 2024-03-28T22:16:53.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.293+0000 7fdc2affd640 1 --2- 172.21.15.16:0/3037998643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558221c24d90 0x558221c2d430 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fdc2400b710 tx=0x7fdc24004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35a9cb85e63b2114 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:53.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.293+0000 7fdc21ffb640 1 -- 172.21.15.16:0/3037998643 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc24003bd0 con 0x558221c24d90 2024-03-28T22:16:53.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.293+0000 7fdc21ffb640 1 -- 172.21.15.16:0/3037998643 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc24015070 con 0x558221c24d90 2024-03-28T22:16:53.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.293+0000 7fdc21ffb640 1 -- 172.21.15.16:0/3037998643 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc24004d90 con 0x558221c24d90 2024-03-28T22:16:53.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.294+0000 7fdc3462e740 1 -- 172.21.15.16:0/3037998643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558221c24d90 msgr2=0x558221c2d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:53.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.294+0000 7fdc3462e740 1 --2- 172.21.15.16:0/3037998643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558221c24d90 0x558221c2d430 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fdc2400b710 tx=0x7fdc24004610 comp rx=0 tx=0).stop 2024-03-28T22:16:53.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.294+0000 7fdc2b7fe640 1 --2- 172.21.15.16:0/2411114108 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fdc180a1ef0 0x7fdc180a43b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c0bc560 tx=0x7fdc1c0bc630 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:53.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.294+0000 7fdc3462e740 1 -- 172.21.15.16:0/3037998643 shutdown_connections 2024-03-28T22:16:53.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.294+0000 7fdc3462e740 1 --2- 172.21.15.16:0/3037998643 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558221c2d970 0x558221c2fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.294+0000 7fdc3462e740 1 --2- 172.21.15.16:0/3037998643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558221c24d90 0x558221c2d430 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.294+0000 7fdc3462e740 1 --2- 172.21.15.16:0/3037998643 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558221c24470 0x558221c24850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.294+0000 7fdc3462e740 1 -- 172.21.15.16:0/3037998643 >> 172.21.15.16:0/3037998643 conn(0x558221c1eba0 msgr2=0x558221c1ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:53.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.294+0000 7fdc3462e740 1 -- 172.21.15.16:0/3037998643 shutdown_connections 2024-03-28T22:16:53.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.294+0000 7fdc3462e740 1 -- 172.21.15.16:0/3037998643 wait complete. 2024-03-28T22:16:53.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.294+0000 7fdc3462e740 1 -- start start 2024-03-28T22:16:53.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.294+0000 7fdc3462e740 1 Processor -- start 2024-03-28T22:16:53.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.295+0000 7fdc3462e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558221c24470 0x558221c2aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.295+0000 7fdc3462e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558221c24d90 0x558221c2af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.295+0000 7fdc3462e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558221c2d970 0x558221c2b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.295+0000 7fdc3462e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558221c34b70 con 0x558221c24d90 2024-03-28T22:16:53.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.295+0000 7fdc3462e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x558221b26470 con 0x558221c24470 2024-03-28T22:16:53.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.295+0000 7fdc3462e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x558221c34cf0 con 0x558221c2d970 2024-03-28T22:16:53.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.295+0000 7fdc3462e740 10 client.0 _ll_register_callbacks cb 0x558221b43cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:16:53.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.295+0000 7fdc2affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558221c24d90 0x558221c2af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.295+0000 7fdc2affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558221c24d90 0x558221c2af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60240/0 (socket says 172.21.15.16:60240) 2024-03-28T22:16:53.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.295+0000 7fdc2affd640 1 -- 172.21.15.16:0/347806398 learned_addr learned my addr 172.21.15.16:0/347806398 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:53.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.295+0000 7fdc2a7fc640 1 --2- 172.21.15.16:0/347806398 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558221c24470 0x558221c2aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.295+0000 7fdc2a7fc640 1 --2- 172.21.15.16:0/347806398 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558221c2d970 0x558221c2b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.295+0000 7fdc2affd640 1 -- 172.21.15.16:0/347806398 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558221c2d970 msgr2=0x558221c2b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:53.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.295+0000 7fdc2affd640 1 --2- 172.21.15.16:0/347806398 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558221c2d970 0x558221c2b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.295+0000 7fdc2affd640 1 -- 172.21.15.16:0/347806398 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558221c24470 msgr2=0x558221c2aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:53.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.295+0000 7fdc2affd640 1 --2- 172.21.15.16:0/347806398 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558221c24470 0x558221c2aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.295+0000 7fdc2affd640 1 -- 172.21.15.16:0/347806398 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55822189c3f0 con 0x558221c24d90 2024-03-28T22:16:53.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.295+0000 7fdc2affd640 1 --2- 172.21.15.16:0/347806398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558221c24d90 0x558221c2af50 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fdc24004290 tx=0x7fdc2400bfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58e1fba9551bdd44 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:53.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.295+0000 7fdc0affd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:16:53.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.297+0000 7fdc0affd640 1 -- 172.21.15.16:0/347806398 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc24028060 con 0x558221c24d90 2024-03-28T22:16:53.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.297+0000 7fdc3462e740 1 -- 172.21.15.16:0/347806398 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc24015070 con 0x558221c24d90 2024-03-28T22:16:53.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.297+0000 7fdc0affd640 1 -- 172.21.15.16:0/347806398 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fdc24011440 con 0x558221c24d90 2024-03-28T22:16:53.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.297+0000 7fdc0affd640 1 -- 172.21.15.16:0/347806398 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdc240046c0 con 0x558221c24d90 2024-03-28T22:16:53.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.297+0000 7fdc3462e740 10 client.16812 fetch_fsmap learned FSMap version 27 2024-03-28T22:16:53.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.297+0000 7fdc3462e740 1 -- 172.21.15.16:0/347806398 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5582219667d0 con 0x558221c24d90 2024-03-28T22:16:53.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.297+0000 7fdc0affd640 1 -- 172.21.15.16:0/347806398 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc24015070 con 0x558221c24d90 2024-03-28T22:16:53.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.297+0000 7fdc0affd640 1 -- 172.21.15.16:0/347806398 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7fdc24022070 con 0x558221c24d90 2024-03-28T22:16:53.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.297+0000 7fdc3462e740 10 client.16812 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:16:53.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.297+0000 7fdc3462e740 10 client.16812 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-28T22:16:53.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.297+0000 7fdc3462e740 20 client.16812 populate_metadata read hostname 'smithi016' 2024-03-28T22:16:53.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.297+0000 7fdc3462e740 1 --2- 172.21.15.16:0/347806398 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x558221c41d40 0x558221c44130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.297+0000 7fdc3462e740 4 client.16812 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:16:53.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.297+0000 7fdc3462e740 1 -- 172.21.15.16:0/347806398 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558221b956f0 con 0x558221c41d40 2024-03-28T22:16:53.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.298+0000 7fdc2a7fc640 1 --2- 172.21.15.16:0/347806398 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x558221c41d40 0x558221c44130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.298+0000 7fdc0affd640 10 client.16812 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:16:53.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.298+0000 7fdc2a7fc640 1 --2- 172.21.15.16:0/347806398 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x558221c41d40 0x558221c44130 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7da87d180ab97bb9 server_cookie=d7a47a0f8006704 in_seq=0 out_seq=0 2024-03-28T22:16:53.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.299+0000 7fdc0affd640 1 -- 172.21.15.16:0/347806398 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558221b956f0 con 0x558221c41d40 2024-03-28T22:16:53.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.299+0000 7fdc0affd640 10 client.16812 handle_command_reply: tid=0 2024-03-28T22:16:53.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.299+0000 7fdc3462e740 1 client.16812 shutdown 2024-03-28T22:16:53.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.299+0000 7fdc3462e740 1 -- 172.21.15.16:0/347806398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558221c24d90 msgr2=0x558221c2af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:53.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.299+0000 7fdc3462e740 1 --2- 172.21.15.16:0/347806398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558221c24d90 0x558221c2af50 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fdc24004290 tx=0x7fdc2400bfc0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.299+0000 7fdc3462e740 1 -- 172.21.15.16:0/347806398 shutdown_connections 2024-03-28T22:16:53.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.299+0000 7fdc3462e740 1 --2- 172.21.15.16:0/347806398 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x558221c41d40 0x558221c44130 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.299+0000 7fdc3462e740 1 --2- 172.21.15.16:0/347806398 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558221c2d970 0x558221c2b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.299+0000 7fdc0affd640 0 client.16812 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:16:53.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.299+0000 7fdc3462e740 1 --2- 172.21.15.16:0/347806398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558221c24d90 0x558221c2af50 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.299+0000 7fdc3462e740 1 --2- 172.21.15.16:0/347806398 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558221c24470 0x558221c2aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.299+0000 7fdc3462e740 1 -- 172.21.15.16:0/347806398 >> 172.21.15.16:0/347806398 conn(0x558221c1eba0 msgr2=0x558221c208f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:53.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.299+0000 7fdc3462e740 1 -- 172.21.15.16:0/347806398 shutdown_connections 2024-03-28T22:16:53.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.299+0000 7fdc3462e740 1 -- 172.21.15.16:0/347806398 wait complete. 2024-03-28T22:16:53.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.299+0000 7fdc3462e740 20 client.16812 trim_cache size 0 max 16384 2024-03-28T22:16:53.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.312+0000 7fdc3462e740 1 Processor -- start 2024-03-28T22:16:53.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.312+0000 7fdc3462e740 1 -- start start 2024-03-28T22:16:53.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.313+0000 7fdc3462e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558221c24470 0x558221c3d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.313+0000 7fdc3462e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558221c24d90 0x558221c496d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.313+0000 7fdc3462e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558221c2d970 0x558221c27c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.313+0000 7fdc3462e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdc2c1adae0 con 0x558221c2d970 2024-03-28T22:16:53.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.313+0000 7fdc3462e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fdc2c1ab3b0 con 0x558221c24d90 2024-03-28T22:16:53.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.313+0000 7fdc3462e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fdc2c1a9200 con 0x558221c24470 2024-03-28T22:16:53.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.313+0000 7fdc2a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558221c2d970 0x558221c27c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.313+0000 7fdc2a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558221c2d970 0x558221c27c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60256/0 (socket says 172.21.15.16:60256) 2024-03-28T22:16:53.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.313+0000 7fdc2a7fc640 1 -- 172.21.15.16:0/1956515053 learned_addr learned my addr 172.21.15.16:0/1956515053 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:53.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.313+0000 7fdc2affd640 1 --2- 172.21.15.16:0/1956515053 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558221c24d90 0x558221c496d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.313+0000 7fdc2a7fc640 1 -- 172.21.15.16:0/1956515053 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558221c24470 msgr2=0x558221c3d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:53.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.313+0000 7fdc2a7fc640 1 --2- 172.21.15.16:0/1956515053 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558221c24470 0x558221c3d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.313+0000 7fdc2a7fc640 1 -- 172.21.15.16:0/1956515053 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558221c24d90 msgr2=0x558221c496d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:53.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.313+0000 7fdc2a7fc640 1 --2- 172.21.15.16:0/1956515053 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558221c24d90 0x558221c496d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.313+0000 7fdc2a7fc640 1 -- 172.21.15.16:0/1956515053 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558221c2fd60 con 0x558221c2d970 2024-03-28T22:16:53.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.313+0000 7fdc2a7fc640 1 --2- 172.21.15.16:0/1956515053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558221c2d970 0x558221c27c70 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fdc14012770 tx=0x7fdc14014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a450511e18bc8268 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:53.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.313+0000 7fdc217fa640 1 -- 172.21.15.16:0/1956515053 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc14028020 con 0x558221c2d970 2024-03-28T22:16:53.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.313+0000 7fdc217fa640 1 -- 172.21.15.16:0/1956515053 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc14024bf0 con 0x558221c2d970 2024-03-28T22:16:53.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.313+0000 7fdc217fa640 1 -- 172.21.15.16:0/1956515053 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc14012e60 con 0x558221c2d970 2024-03-28T22:16:53.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.314+0000 7fdc3462e740 1 -- 172.21.15.16:0/1956515053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558221c2d970 msgr2=0x558221c27c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:53.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.314+0000 7fdc3462e740 1 --2- 172.21.15.16:0/1956515053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558221c2d970 0x558221c27c70 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fdc14012770 tx=0x7fdc14014040 comp rx=0 tx=0).stop 2024-03-28T22:16:53.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.314+0000 7fdc3462e740 1 -- 172.21.15.16:0/1956515053 shutdown_connections 2024-03-28T22:16:53.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.314+0000 7fdc3462e740 1 --2- 172.21.15.16:0/1956515053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558221c2d970 0x558221c27c70 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.314+0000 7fdc3462e740 1 --2- 172.21.15.16:0/1956515053 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558221c24d90 0x558221c496d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.314+0000 7fdc3462e740 1 --2- 172.21.15.16:0/1956515053 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558221c24470 0x558221c3d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.314+0000 7fdc3462e740 1 -- 172.21.15.16:0/1956515053 >> 172.21.15.16:0/1956515053 conn(0x558221c1eba0 msgr2=0x558221c2c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:53.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.314+0000 7fdc3462e740 1 -- 172.21.15.16:0/1956515053 shutdown_connections 2024-03-28T22:16:53.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.314+0000 7fdc3462e740 1 -- 172.21.15.16:0/1956515053 wait complete. 2024-03-28T22:16:53.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.315+0000 7fdc3462e740 1 -- start start 2024-03-28T22:16:53.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.315+0000 7fdc3462e740 1 Processor -- start 2024-03-28T22:16:53.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.315+0000 7fdc3462e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558221c24470 0x558221c35f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.315+0000 7fdc3462e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558221c24d90 0x558221c36440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.315+0000 7fdc3462e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558221c2d970 0x558221ca6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.315+0000 7fdc3462e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdc2c1a9870 con 0x558221c2d970 2024-03-28T22:16:53.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.315+0000 7fdc3462e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fdc2c1aace0 con 0x558221c24470 2024-03-28T22:16:53.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.315+0000 7fdc3462e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fdc2c1a9f40 con 0x558221c24d90 2024-03-28T22:16:53.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.315+0000 7fdc3462e740 10 client.0 _ll_register_callbacks cb 0x7fdc24004510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:16:53.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.315+0000 7fdc2a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558221c2d970 0x558221ca6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.315+0000 7fdc2a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558221c2d970 0x558221ca6d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60266/0 (socket says 172.21.15.16:60266) 2024-03-28T22:16:53.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.315+0000 7fdc2a7fc640 1 -- 172.21.15.16:0/1004733682 learned_addr learned my addr 172.21.15.16:0/1004733682 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:53.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.315+0000 7fdc2a7fc640 1 --2- 172.21.15.16:0/1004733682 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558221c24470 0x558221c35f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.315+0000 7fdc2affd640 1 --2- 172.21.15.16:0/1004733682 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558221c24d90 0x558221c36440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.315+0000 7fdc2a7fc640 1 -- 172.21.15.16:0/1004733682 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558221c24d90 msgr2=0x558221c36440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:53.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.315+0000 7fdc2a7fc640 1 --2- 172.21.15.16:0/1004733682 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558221c24d90 0x558221c36440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.315+0000 7fdc2a7fc640 1 -- 172.21.15.16:0/1004733682 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558221c24470 msgr2=0x558221c35f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:53.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.315+0000 7fdc2a7fc640 1 --2- 172.21.15.16:0/1004733682 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558221c24470 0x558221c35f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.315+0000 7fdc2a7fc640 1 -- 172.21.15.16:0/1004733682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558221ca7240 con 0x558221c2d970 2024-03-28T22:16:53.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.316+0000 7fdc2a7fc640 1 --2- 172.21.15.16:0/1004733682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558221c2d970 0x558221ca6d00 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fdc14010a90 tx=0x7fdc14009370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ece9d8e2259d7f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:53.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.316+0000 7fdc0affd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:16:53.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.316+0000 7fdc0affd640 1 -- 172.21.15.16:0/1004733682 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc14032030 con 0x558221c2d970 2024-03-28T22:16:53.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.316+0000 7fdc0affd640 1 -- 172.21.15.16:0/1004733682 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7fdc14004020 con 0x558221c2d970 2024-03-28T22:16:53.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.316+0000 7fdc3462e740 1 -- 172.21.15.16:0/1004733682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc14024bf0 con 0x558221c2d970 2024-03-28T22:16:53.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.316+0000 7fdc0affd640 1 -- 172.21.15.16:0/1004733682 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdc14041050 con 0x558221c2d970 2024-03-28T22:16:53.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.316+0000 7fdc3462e740 10 client.16818 fetch_fsmap learned FSMap version 27 2024-03-28T22:16:53.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.316+0000 7fdc3462e740 1 -- 172.21.15.16:0/1004733682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558221ca84b0 con 0x558221c2d970 2024-03-28T22:16:53.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.316+0000 7fdc0affd640 1 -- 172.21.15.16:0/1004733682 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc14024bf0 con 0x558221c2d970 2024-03-28T22:16:53.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.316+0000 7fdc0affd640 1 -- 172.21.15.16:0/1004733682 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7fdc140039a0 con 0x558221c2d970 2024-03-28T22:16:53.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.316+0000 7fdc3462e740 10 client.16818 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:16:53.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.316+0000 7fdc3462e740 10 client.16818 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-28T22:16:53.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.316+0000 7fdc3462e740 20 client.16818 populate_metadata read hostname 'smithi016' 2024-03-28T22:16:53.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.316+0000 7fdc3462e740 1 --2- 172.21.15.16:0/1004733682 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x558221ca8b40 0x558221caaf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.316+0000 7fdc3462e740 4 client.16818 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:16:53.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.316+0000 7fdc3462e740 1 -- 172.21.15.16:0/1004733682 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558221b8fa40 con 0x558221ca8b40 2024-03-28T22:16:53.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.317+0000 7fdc2affd640 1 --2- 172.21.15.16:0/1004733682 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x558221ca8b40 0x558221caaf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.317+0000 7fdc0affd640 10 client.16818 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:16:53.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.317+0000 7fdc2affd640 1 --2- 172.21.15.16:0/1004733682 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x558221ca8b40 0x558221caaf60 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5c57da8a263b70db server_cookie=a368c69f59ef7bc4 in_seq=0 out_seq=0 2024-03-28T22:16:53.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.317+0000 7fdc0affd640 1 -- 172.21.15.16:0/1004733682 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558221b8fa40 con 0x558221ca8b40 2024-03-28T22:16:53.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.317+0000 7fdc0affd640 10 client.16818 handle_command_reply: tid=0 2024-03-28T22:16:53.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.317+0000 7fdc3462e740 1 client.16818 shutdown 2024-03-28T22:16:53.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.317+0000 7fdc3462e740 1 -- 172.21.15.16:0/1004733682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558221c2d970 msgr2=0x558221ca6d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:53.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.317+0000 7fdc3462e740 1 --2- 172.21.15.16:0/1004733682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558221c2d970 0x558221ca6d00 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fdc14010a90 tx=0x7fdc14009370 comp rx=0 tx=0).stop 2024-03-28T22:16:53.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.317+0000 7fdc3462e740 1 -- 172.21.15.16:0/1004733682 shutdown_connections 2024-03-28T22:16:53.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.317+0000 7fdc3462e740 1 --2- 172.21.15.16:0/1004733682 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x558221ca8b40 0x558221caaf60 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.317+0000 7fdc3462e740 1 --2- 172.21.15.16:0/1004733682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558221c2d970 0x558221ca6d00 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.317+0000 7fdc0affd640 0 client.16818 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:16:53.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.317+0000 7fdc3462e740 1 --2- 172.21.15.16:0/1004733682 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558221c24d90 0x558221c36440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.317+0000 7fdc3462e740 1 --2- 172.21.15.16:0/1004733682 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558221c24470 0x558221c35f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.317+0000 7fdc3462e740 1 -- 172.21.15.16:0/1004733682 >> 172.21.15.16:0/1004733682 conn(0x558221c1eba0 msgr2=0x558221c2c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:53.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.317+0000 7fdc3462e740 1 -- 172.21.15.16:0/1004733682 shutdown_connections 2024-03-28T22:16:53.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.317+0000 7fdc3462e740 1 -- 172.21.15.16:0/1004733682 wait complete. 2024-03-28T22:16:53.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.317+0000 7fdc3462e740 20 client.16818 trim_cache size 0 max 16384 2024-03-28T22:16:53.330 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:16:53.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.318+0000 7fdc31a47640 1 -- 172.21.15.16:0/2411114108 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fdc180a1ef0 msgr2=0x7fdc180a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:53.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.318+0000 7fdc31a47640 1 --2- 172.21.15.16:0/2411114108 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fdc180a1ef0 0x7fdc180a43b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c0bc560 tx=0x7fdc1c0bc630 comp rx=0 tx=0).stop 2024-03-28T22:16:53.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.318+0000 7fdc31a47640 1 -- 172.21.15.16:0/2411114108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc2c107700 msgr2=0x7fdc2c19c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:53.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.318+0000 7fdc31a47640 1 --2- 172.21.15.16:0/2411114108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc2c107700 0x7fdc2c19c890 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c004af0 tx=0x7fdc1c032ec0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.318+0000 7fdc31a47640 1 -- 172.21.15.16:0/2411114108 shutdown_connections 2024-03-28T22:16:53.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.318+0000 7fdc31a47640 1 --2- 172.21.15.16:0/2411114108 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fdc180a1ef0 0x7fdc180a43b0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.318+0000 7fdc31a47640 1 --2- 172.21.15.16:0/2411114108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc2c107700 0x7fdc2c19c890 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.318+0000 7fdc31a47640 1 --2- 172.21.15.16:0/2411114108 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc2c1020b0 0x7fdc2c198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.318+0000 7fdc31a47640 1 --2- 172.21.15.16:0/2411114108 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc2c1016f0 0x7fdc2c197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.318+0000 7fdc31a47640 1 -- 172.21.15.16:0/2411114108 >> 172.21.15.16:0/2411114108 conn(0x7fdc2c0fd1e0 msgr2=0x7fdc2c0fd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:53.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.318+0000 7fdc31a47640 1 -- 172.21.15.16:0/2411114108 shutdown_connections 2024-03-28T22:16:53.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.318+0000 7fdc31a47640 1 -- 172.21.15.16:0/2411114108 wait complete. 2024-03-28T22:16:53.334 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:16:53.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:53 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/4005563281' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:16:53.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.436+0000 7f427202e640 1 -- 172.21.15.16:0/4267909967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f426c1029b0 msgr2=0x7f426c102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:53.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.436+0000 7f427202e640 1 --2- 172.21.15.16:0/4267909967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f426c1029b0 0x7f426c102d90 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f425c009ca0 tx=0x7f425c02c180 comp rx=0 tx=0).stop 2024-03-28T22:16:53.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.436+0000 7f427202e640 1 -- 172.21.15.16:0/4267909967 shutdown_connections 2024-03-28T22:16:53.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.436+0000 7f427202e640 1 --2- 172.21.15.16:0/4267909967 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f426c10bcf0 0x7f426c10e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.436+0000 7f427202e640 1 --2- 172.21.15.16:0/4267909967 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f426c1032d0 0x7f426c10b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.436+0000 7f427202e640 1 --2- 172.21.15.16:0/4267909967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f426c1029b0 0x7f426c102d90 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.436+0000 7f427202e640 1 -- 172.21.15.16:0/4267909967 >> 172.21.15.16:0/4267909967 conn(0x7f426c077800 msgr2=0x7f426c077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:53.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.437+0000 7f427202e640 1 -- 172.21.15.16:0/4267909967 shutdown_connections 2024-03-28T22:16:53.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.437+0000 7f427202e640 1 -- 172.21.15.16:0/4267909967 wait complete. 2024-03-28T22:16:53.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.437+0000 7f427202e640 1 Processor -- start 2024-03-28T22:16:53.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.437+0000 7f427202e640 1 -- start start 2024-03-28T22:16:53.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.437+0000 7f427202e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f426c1029b0 0x7f426c19a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.437+0000 7f427202e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f426c1032d0 0x7f426c19a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.437+0000 7f427202e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f426c10bcf0 0x7f426c19ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.437+0000 7f427202e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f426c111ad0 con 0x7f426c1029b0 2024-03-28T22:16:53.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.437+0000 7f427202e640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f426c111950 con 0x7f426c10bcf0 2024-03-28T22:16:53.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.437+0000 7f427202e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f426c111c50 con 0x7f426c1032d0 2024-03-28T22:16:53.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.438+0000 7f426b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f426c1029b0 0x7f426c19a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.438+0000 7f426b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f426c1029b0 0x7f426c19a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60272/0 (socket says 172.21.15.16:60272) 2024-03-28T22:16:53.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.438+0000 7f426b7fe640 1 -- 172.21.15.16:0/2964276468 learned_addr learned my addr 172.21.15.16:0/2964276468 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:53.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.438+0000 7f426bfff640 1 --2- 172.21.15.16:0/2964276468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f426c10bcf0 0x7f426c19ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.438+0000 7f426affd640 1 --2- 172.21.15.16:0/2964276468 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f426c1032d0 0x7f426c19a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.438+0000 7f426b7fe640 1 -- 172.21.15.16:0/2964276468 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f426c1032d0 msgr2=0x7f426c19a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:53.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.438+0000 7f426b7fe640 1 --2- 172.21.15.16:0/2964276468 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f426c1032d0 0x7f426c19a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.438+0000 7f426b7fe640 1 -- 172.21.15.16:0/2964276468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f426c10bcf0 msgr2=0x7f426c19ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:53.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.438+0000 7f426b7fe640 1 --2- 172.21.15.16:0/2964276468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f426c10bcf0 0x7f426c19ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.438+0000 7f426b7fe640 1 -- 172.21.15.16:0/2964276468 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f426c19f180 con 0x7f426c1029b0 2024-03-28T22:16:53.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.438+0000 7f426affd640 1 --2- 172.21.15.16:0/2964276468 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f426c1032d0 0x7f426c19a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:16:53.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.438+0000 7f426b7fe640 1 --2- 172.21.15.16:0/2964276468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f426c1029b0 0x7f426c19a1e0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f425c005e30 tx=0x7f425c035d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:53.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.438+0000 7f4268ff9640 1 -- 172.21.15.16:0/2964276468 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f425c039040 con 0x7f426c1029b0 2024-03-28T22:16:53.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.438+0000 7f4268ff9640 1 -- 172.21.15.16:0/2964276468 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f425c044070 con 0x7f426c1029b0 2024-03-28T22:16:53.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.438+0000 7f4268ff9640 1 -- 172.21.15.16:0/2964276468 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f425c0328d0 con 0x7f426c1029b0 2024-03-28T22:16:53.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.438+0000 7f427202e640 1 -- 172.21.15.16:0/2964276468 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f426c19f410 con 0x7f426c1029b0 2024-03-28T22:16:53.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.438+0000 7f427202e640 1 -- 172.21.15.16:0/2964276468 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f426c1a6d00 con 0x7f426c1029b0 2024-03-28T22:16:53.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.440+0000 7f4268ff9640 1 -- 172.21.15.16:0/2964276468 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f425c032a70 con 0x7f426c1029b0 2024-03-28T22:16:53.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.441+0000 7f4274c11740 1 Processor -- start 2024-03-28T22:16:53.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.441+0000 7f4274c11740 1 -- start start 2024-03-28T22:16:53.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.441+0000 7f4268ff9640 1 --2- 172.21.15.16:0/2964276468 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f42540a1ef0 0x7f42540a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.441+0000 7f4274c11740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56199fc26470 0x56199fc26850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.441+0000 7f4274c11740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56199fc26d90 0x56199fc2f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.441+0000 7f426bfff640 1 --2- 172.21.15.16:0/2964276468 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f42540a1ef0 0x7f42540a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.441+0000 7f4274c11740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199fc2f970 0x56199fc31d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.441+0000 7f4274c11740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56199fb8a1b0 con 0x56199fc2f970 2024-03-28T22:16:53.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.441+0000 7f4274c11740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x56199fb32c00 con 0x56199fc26470 2024-03-28T22:16:53.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.441+0000 7f4274c11740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56199fb86250 con 0x56199fc26d90 2024-03-28T22:16:53.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.441+0000 7f4268ff9640 1 -- 172.21.15.16:0/2964276468 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f425c0f05a0 con 0x7f426c1029b0 2024-03-28T22:16:53.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.441+0000 7f426bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199fc2f970 0x56199fc31d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.441+0000 7f426bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199fc2f970 0x56199fc31d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60288/0 (socket says 172.21.15.16:60288) 2024-03-28T22:16:53.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.441+0000 7f426b7fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56199fc26470 0x56199fc26850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.441+0000 7f426bfff640 1 -- 172.21.15.16:0/789920353 learned_addr learned my addr 172.21.15.16:0/789920353 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:53.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.441+0000 7f426affd640 1 --2- 172.21.15.16:0/789920353 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56199fc26d90 0x56199fc2f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.441+0000 7f426b7fe640 1 -- 172.21.15.16:0/789920353 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56199fc26d90 msgr2=0x56199fc2f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:53.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.442+0000 7f426b7fe640 1 --2- 172.21.15.16:0/789920353 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56199fc26d90 0x56199fc2f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.442+0000 7f426b7fe640 1 -- 172.21.15.16:0/789920353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199fc2f970 msgr2=0x56199fc31d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:53.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.442+0000 7f426b7fe640 1 --2- 172.21.15.16:0/789920353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199fc2f970 0x56199fc31d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.442+0000 7f426b7fe640 1 -- 172.21.15.16:0/789920353 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56199fb523d0 con 0x56199fc26470 2024-03-28T22:16:53.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.442+0000 7f426bfff640 1 --2- 172.21.15.16:0/789920353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199fc2f970 0x56199fc31d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:16:53.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.442+0000 7f426bfff640 1 --2- 172.21.15.16:0/2964276468 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f42540a1ef0 0x7f42540a43b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f426400bed0 tx=0x7f4264004550 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:53.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.442+0000 7f426b7fe640 1 --2- 172.21.15.16:0/789920353 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56199fc26470 0x56199fc26850 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f425c0eddc0 tx=0x7f425c004700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac885232217a169e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:53.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.442+0000 7f4251ffb640 1 -- 172.21.15.16:0/789920353 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f425c003e10 con 0x56199fc26470 2024-03-28T22:16:53.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.442+0000 7f4251ffb640 1 -- 172.21.15.16:0/789920353 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f425c0bd760 con 0x56199fc26470 2024-03-28T22:16:53.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.442+0000 7f4251ffb640 1 -- 172.21.15.16:0/789920353 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f425c036ca0 con 0x56199fc26470 2024-03-28T22:16:53.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.442+0000 7f4274c11740 1 -- 172.21.15.16:0/789920353 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56199fc26470 msgr2=0x56199fc26850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:53.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.442+0000 7f4274c11740 1 --2- 172.21.15.16:0/789920353 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56199fc26470 0x56199fc26850 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f425c0eddc0 tx=0x7f425c004700 comp rx=0 tx=0).stop 2024-03-28T22:16:53.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.443+0000 7f4274c11740 1 -- 172.21.15.16:0/789920353 shutdown_connections 2024-03-28T22:16:53.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.443+0000 7f4274c11740 1 --2- 172.21.15.16:0/789920353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199fc2f970 0x56199fc31d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.443+0000 7f4274c11740 1 --2- 172.21.15.16:0/789920353 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56199fc26d90 0x56199fc2f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.443+0000 7f4274c11740 1 --2- 172.21.15.16:0/789920353 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56199fc26470 0x56199fc26850 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.443+0000 7f4274c11740 1 -- 172.21.15.16:0/789920353 >> 172.21.15.16:0/789920353 conn(0x56199fc20ba0 msgr2=0x56199fc20f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:53.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.443+0000 7f4274c11740 1 -- 172.21.15.16:0/789920353 shutdown_connections 2024-03-28T22:16:53.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.443+0000 7f4274c11740 1 -- 172.21.15.16:0/789920353 wait complete. 2024-03-28T22:16:53.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.443+0000 7f4274c11740 1 -- start start 2024-03-28T22:16:53.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.443+0000 7f4274c11740 1 Processor -- start 2024-03-28T22:16:53.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.443+0000 7f4274c11740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56199fc26470 0x56199fc2c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.443+0000 7f4274c11740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56199fc26d90 0x56199fc2ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.443+0000 7f4274c11740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199fc2f970 0x56199fc2d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.443+0000 7f4274c11740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56199fc32520 con 0x56199fc2f970 2024-03-28T22:16:53.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.443+0000 7f4274c11740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x56199fb28470 con 0x56199fc26470 2024-03-28T22:16:53.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.443+0000 7f4274c11740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56199fc326a0 con 0x56199fc26d90 2024-03-28T22:16:53.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.443+0000 7f4274c11740 10 client.0 _ll_register_callbacks cb 0x56199fb45cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:16:53.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.444+0000 7f426bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199fc2f970 0x56199fc2d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.444+0000 7f426b7fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56199fc26470 0x56199fc2c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.444+0000 7f426bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199fc2f970 0x56199fc2d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60302/0 (socket says 172.21.15.16:60302) 2024-03-28T22:16:53.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.444+0000 7f426bfff640 1 -- 172.21.15.16:0/126600121 learned_addr learned my addr 172.21.15.16:0/126600121 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:53.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.444+0000 7f426affd640 1 --2- 172.21.15.16:0/126600121 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56199fc26d90 0x56199fc2ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.444+0000 7f426b7fe640 1 -- 172.21.15.16:0/126600121 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56199fc26d90 msgr2=0x56199fc2ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:53.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.444+0000 7f426b7fe640 1 --2- 172.21.15.16:0/126600121 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56199fc26d90 0x56199fc2ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.444+0000 7f426b7fe640 1 -- 172.21.15.16:0/126600121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199fc2f970 msgr2=0x56199fc2d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:53.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.444+0000 7f426b7fe640 1 --2- 172.21.15.16:0/126600121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199fc2f970 0x56199fc2d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.444+0000 7f426b7fe640 1 -- 172.21.15.16:0/126600121 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56199f89e3f0 con 0x56199fc26470 2024-03-28T22:16:53.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.444+0000 7f426bfff640 1 --2- 172.21.15.16:0/126600121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199fc2f970 0x56199fc2d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:16:53.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.444+0000 7f426b7fe640 1 --2- 172.21.15.16:0/126600121 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56199fc26470 0x56199fc2c970 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f425c0bd220 tx=0x7f425c036f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d93c8292c267f15 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:53.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.444+0000 7f424affd640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-03-28T22:16:53.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.444+0000 7f424affd640 1 -- 172.21.15.16:0/126600121 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f425c003e10 con 0x56199fc26470 2024-03-28T22:16:53.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.444+0000 7f424affd640 1 -- 172.21.15.16:0/126600121 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f425c0bd4d0 con 0x56199fc26470 2024-03-28T22:16:53.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.444+0000 7f4274c11740 1 -- 172.21.15.16:0/126600121 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f425c0bd760 con 0x56199fc26470 2024-03-28T22:16:53.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.444+0000 7f424affd640 1 -- 172.21.15.16:0/126600121 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f425c0bd060 con 0x56199fc26470 2024-03-28T22:16:53.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.444+0000 7f4274c11740 10 client.26732 fetch_fsmap learned FSMap version 27 2024-03-28T22:16:53.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.444+0000 7f4274c11740 1 -- 172.21.15.16:0/126600121 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56199f9687d0 con 0x56199fc26470 2024-03-28T22:16:53.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.445+0000 7f424affd640 1 -- 172.21.15.16:0/126600121 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f425c0bd760 con 0x56199fc26470 2024-03-28T22:16:53.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.445+0000 7f424affd640 1 -- 172.21.15.16:0/126600121 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f425c03c030 con 0x56199fc26470 2024-03-28T22:16:53.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.445+0000 7f4274c11740 10 client.26732 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:16:53.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.445+0000 7f4274c11740 10 client.26732 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T22:16:53.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.445+0000 7f4274c11740 20 client.26732 populate_metadata read hostname 'smithi016' 2024-03-28T22:16:53.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.445+0000 7f4274c11740 1 --2- 172.21.15.16:0/126600121 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x56199fc3f4e0 0x56199fc41910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.445+0000 7f4274c11740 4 client.26732 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:16:53.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.445+0000 7f4274c11740 1 -- 172.21.15.16:0/126600121 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56199fb976f0 con 0x56199fc3f4e0 2024-03-28T22:16:53.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.445+0000 7f426affd640 1 --2- 172.21.15.16:0/126600121 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x56199fc3f4e0 0x56199fc41910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.445+0000 7f426affd640 1 --2- 172.21.15.16:0/126600121 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x56199fc3f4e0 0x56199fc41910 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1846e15bc9cf9326 server_cookie=77aac3dc7998363b in_seq=0 out_seq=0 2024-03-28T22:16:53.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.445+0000 7f424affd640 10 client.26732 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:16:53.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.446+0000 7f424affd640 1 -- 172.21.15.16:0/126600121 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56199fb976f0 con 0x56199fc3f4e0 2024-03-28T22:16:53.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.446+0000 7f424affd640 10 client.26732 handle_command_reply: tid=0 2024-03-28T22:16:53.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.446+0000 7f4274c11740 1 client.26732 shutdown 2024-03-28T22:16:53.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.446+0000 7f4274c11740 1 -- 172.21.15.16:0/126600121 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56199fc26470 msgr2=0x56199fc2c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:53.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.446+0000 7f4274c11740 1 --2- 172.21.15.16:0/126600121 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56199fc26470 0x56199fc2c970 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f425c0bd220 tx=0x7f425c036f50 comp rx=0 tx=0).stop 2024-03-28T22:16:53.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.446+0000 7f4274c11740 1 -- 172.21.15.16:0/126600121 shutdown_connections 2024-03-28T22:16:53.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.446+0000 7f4274c11740 1 --2- 172.21.15.16:0/126600121 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x56199fc3f4e0 0x56199fc41910 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.446+0000 7f4274c11740 1 --2- 172.21.15.16:0/126600121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199fc2f970 0x56199fc2d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.446+0000 7f4274c11740 1 --2- 172.21.15.16:0/126600121 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56199fc26d90 0x56199fc2ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.446+0000 7f424affd640 0 client.26732 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:16:53.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.446+0000 7f4274c11740 1 --2- 172.21.15.16:0/126600121 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56199fc26470 0x56199fc2c970 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.446+0000 7f4274c11740 1 -- 172.21.15.16:0/126600121 >> 172.21.15.16:0/126600121 conn(0x56199fc20ba0 msgr2=0x56199fc228f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:53.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.446+0000 7f4274c11740 1 -- 172.21.15.16:0/126600121 shutdown_connections 2024-03-28T22:16:53.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.446+0000 7f4274c11740 1 -- 172.21.15.16:0/126600121 wait complete. 2024-03-28T22:16:53.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.446+0000 7f4274c11740 20 client.26732 trim_cache size 0 max 16384 2024-03-28T22:16:53.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.459+0000 7f4274c11740 1 Processor -- start 2024-03-28T22:16:53.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.459+0000 7f4274c11740 1 -- start start 2024-03-28T22:16:53.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.460+0000 7f4274c11740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56199fc26470 0x56199fc2f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.460+0000 7f4274c11740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56199fc26d90 0x56199fc2bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.460+0000 7f4274c11740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199fc2f970 0x56199fc3b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.460+0000 7f4274c11740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f426c1b1e80 con 0x56199fc2f970 2024-03-28T22:16:53.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.460+0000 7f4274c11740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f426c1af750 con 0x56199fc26d90 2024-03-28T22:16:53.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.460+0000 7f4274c11740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f426c1ad5a0 con 0x56199fc26470 2024-03-28T22:16:53.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.460+0000 7f426bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199fc2f970 0x56199fc3b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.460+0000 7f426bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199fc2f970 0x56199fc3b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60308/0 (socket says 172.21.15.16:60308) 2024-03-28T22:16:53.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.460+0000 7f426bfff640 1 -- 172.21.15.16:0/3569293854 learned_addr learned my addr 172.21.15.16:0/3569293854 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:53.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.460+0000 7f426affd640 1 --2- 172.21.15.16:0/3569293854 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56199fc26d90 0x56199fc2bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.460+0000 7f426bfff640 1 -- 172.21.15.16:0/3569293854 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56199fc26470 msgr2=0x56199fc2f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:53.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.460+0000 7f426b7fe640 1 --2- 172.21.15.16:0/3569293854 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56199fc26470 0x56199fc2f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.460+0000 7f426bfff640 1 --2- 172.21.15.16:0/3569293854 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56199fc26470 0x56199fc2f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.460+0000 7f426bfff640 1 -- 172.21.15.16:0/3569293854 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56199fc26d90 msgr2=0x56199fc2bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:53.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.460+0000 7f426bfff640 1 --2- 172.21.15.16:0/3569293854 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56199fc26d90 0x56199fc2bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.460+0000 7f426bfff640 1 -- 172.21.15.16:0/3569293854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56199fc3b720 con 0x56199fc2f970 2024-03-28T22:16:53.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.460+0000 7f426affd640 1 --2- 172.21.15.16:0/3569293854 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56199fc26d90 0x56199fc2bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:16:53.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.460+0000 7f426b7fe640 1 --2- 172.21.15.16:0/3569293854 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56199fc26470 0x56199fc2f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:16:53.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.460+0000 7f426bfff640 1 --2- 172.21.15.16:0/3569293854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199fc2f970 0x56199fc3b1e0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f4264003e50 tx=0x7f426400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac8c5f6235657682 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:53.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.460+0000 7f42517fa640 1 -- 172.21.15.16:0/3569293854 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4264012b30 con 0x56199fc2f970 2024-03-28T22:16:53.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.460+0000 7f42517fa640 1 -- 172.21.15.16:0/3569293854 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4264012cd0 con 0x56199fc2f970 2024-03-28T22:16:53.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.460+0000 7f42517fa640 1 -- 172.21.15.16:0/3569293854 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4264021810 con 0x56199fc2f970 2024-03-28T22:16:53.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.461+0000 7f4274c11740 1 -- 172.21.15.16:0/3569293854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199fc2f970 msgr2=0x56199fc3b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:53.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.461+0000 7f4274c11740 1 --2- 172.21.15.16:0/3569293854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199fc2f970 0x56199fc3b1e0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f4264003e50 tx=0x7f426400b260 comp rx=0 tx=0).stop 2024-03-28T22:16:53.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.461+0000 7f4274c11740 1 -- 172.21.15.16:0/3569293854 shutdown_connections 2024-03-28T22:16:53.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.461+0000 7f4274c11740 1 --2- 172.21.15.16:0/3569293854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199fc2f970 0x56199fc3b1e0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.461+0000 7f4274c11740 1 --2- 172.21.15.16:0/3569293854 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56199fc26d90 0x56199fc2bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.461+0000 7f4274c11740 1 --2- 172.21.15.16:0/3569293854 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56199fc26470 0x56199fc2f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.461+0000 7f4274c11740 1 -- 172.21.15.16:0/3569293854 >> 172.21.15.16:0/3569293854 conn(0x56199fc20ba0 msgr2=0x56199fc1f4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:53.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.461+0000 7f4274c11740 1 -- 172.21.15.16:0/3569293854 shutdown_connections 2024-03-28T22:16:53.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.461+0000 7f4274c11740 1 -- 172.21.15.16:0/3569293854 wait complete. 2024-03-28T22:16:53.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.461+0000 7f4274c11740 1 -- start start 2024-03-28T22:16:53.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.462+0000 7f4274c11740 1 Processor -- start 2024-03-28T22:16:53.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.462+0000 7f4274c11740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56199fc26470 0x56199fc37ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.462+0000 7f4274c11740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199fc26d90 0x56199fc32ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.462+0000 7f4274c11740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56199fc2f970 0x56199fc33460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.462+0000 7f4274c11740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f426c1adc10 con 0x56199fc26d90 2024-03-28T22:16:53.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.462+0000 7f4274c11740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f426c1af080 con 0x56199fc26470 2024-03-28T22:16:53.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.462+0000 7f4274c11740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f426c1ae2e0 con 0x56199fc2f970 2024-03-28T22:16:53.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.462+0000 7f4274c11740 10 client.0 _ll_register_callbacks cb 0x7f425c0bce70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:16:53.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.462+0000 7f426affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199fc26d90 0x56199fc32ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.462+0000 7f426affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199fc26d90 0x56199fc32ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60314/0 (socket says 172.21.15.16:60314) 2024-03-28T22:16:53.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.462+0000 7f426affd640 1 -- 172.21.15.16:0/630513920 learned_addr learned my addr 172.21.15.16:0/630513920 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:53.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.462+0000 7f426b7fe640 1 --2- 172.21.15.16:0/630513920 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56199fc26470 0x56199fc37ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.462+0000 7f426affd640 1 -- 172.21.15.16:0/630513920 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56199fc2f970 msgr2=0x56199fc33460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:53.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.462+0000 7f426affd640 1 --2- 172.21.15.16:0/630513920 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56199fc2f970 0x56199fc33460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.462+0000 7f426affd640 1 -- 172.21.15.16:0/630513920 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56199fc26470 msgr2=0x56199fc37ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:53.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.462+0000 7f426affd640 1 --2- 172.21.15.16:0/630513920 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56199fc26470 0x56199fc37ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.462+0000 7f426affd640 1 -- 172.21.15.16:0/630513920 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56199fc33c90 con 0x56199fc26d90 2024-03-28T22:16:53.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.462+0000 7f426affd640 1 --2- 172.21.15.16:0/630513920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199fc26d90 0x56199fc32ec0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f4260005940 tx=0x7f426001ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c3835a30bb3a15e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:53.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.462+0000 7f424affd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:16:53.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.462+0000 7f424affd640 1 -- 172.21.15.16:0/630513920 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4260014d80 con 0x56199fc26d90 2024-03-28T22:16:53.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.463+0000 7f4274c11740 1 -- 172.21.15.16:0/630513920 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4264012cd0 con 0x56199fc26d90 2024-03-28T22:16:53.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.463+0000 7f424affd640 1 -- 172.21.15.16:0/630513920 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f426001e380 con 0x56199fc26d90 2024-03-28T22:16:53.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.463+0000 7f424affd640 1 -- 172.21.15.16:0/630513920 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f426001e580 con 0x56199fc26d90 2024-03-28T22:16:53.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.463+0000 7f4274c11740 10 client.16836 fetch_fsmap learned FSMap version 27 2024-03-28T22:16:53.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.463+0000 7f4274c11740 1 -- 172.21.15.16:0/630513920 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56199fca58b0 con 0x56199fc26d90 2024-03-28T22:16:53.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.463+0000 7f424affd640 1 -- 172.21.15.16:0/630513920 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4264012cd0 con 0x56199fc26d90 2024-03-28T22:16:53.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.463+0000 7f424affd640 1 -- 172.21.15.16:0/630513920 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f426000f040 con 0x56199fc26d90 2024-03-28T22:16:53.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.463+0000 7f4274c11740 10 client.16836 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:16:53.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.463+0000 7f4274c11740 10 client.16836 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T22:16:53.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.463+0000 7f4274c11740 20 client.16836 populate_metadata read hostname 'smithi016' 2024-03-28T22:16:53.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.463+0000 7f4274c11740 1 --2- 172.21.15.16:0/630513920 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x56199fca5fb0 0x56199fca83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.463+0000 7f4274c11740 4 client.16836 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:16:53.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.463+0000 7f4274c11740 1 -- 172.21.15.16:0/630513920 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56199fb91a40 con 0x56199fca5fb0 2024-03-28T22:16:53.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.463+0000 7f426b7fe640 1 --2- 172.21.15.16:0/630513920 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x56199fca5fb0 0x56199fca83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.464+0000 7f424affd640 10 client.16836 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:16:53.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.464+0000 7f426b7fe640 1 --2- 172.21.15.16:0/630513920 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x56199fca5fb0 0x56199fca83d0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b61a5ed3a2286b9e server_cookie=6a2ea6a810c5d9b9 in_seq=0 out_seq=0 2024-03-28T22:16:53.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.464+0000 7f424affd640 1 -- 172.21.15.16:0/630513920 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56199fb91a40 con 0x56199fca5fb0 2024-03-28T22:16:53.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.464+0000 7f424affd640 10 client.16836 handle_command_reply: tid=0 2024-03-28T22:16:53.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.464+0000 7f4274c11740 1 client.16836 shutdown 2024-03-28T22:16:53.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.464+0000 7f4274c11740 1 -- 172.21.15.16:0/630513920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199fc26d90 msgr2=0x56199fc32ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:53.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.464+0000 7f4274c11740 1 --2- 172.21.15.16:0/630513920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199fc26d90 0x56199fc32ec0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f4260005940 tx=0x7f426001ef60 comp rx=0 tx=0).stop 2024-03-28T22:16:53.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.464+0000 7f4274c11740 1 -- 172.21.15.16:0/630513920 shutdown_connections 2024-03-28T22:16:53.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.464+0000 7f4274c11740 1 --2- 172.21.15.16:0/630513920 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x56199fca5fb0 0x56199fca83d0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.464+0000 7f4274c11740 1 --2- 172.21.15.16:0/630513920 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56199fc2f970 0x56199fc33460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.464+0000 7f4274c11740 1 --2- 172.21.15.16:0/630513920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199fc26d90 0x56199fc32ec0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.464+0000 7f4274c11740 1 --2- 172.21.15.16:0/630513920 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56199fc26470 0x56199fc37ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.464+0000 7f4274c11740 1 -- 172.21.15.16:0/630513920 >> 172.21.15.16:0/630513920 conn(0x56199fc20ba0 msgr2=0x56199fc2d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:53.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.464+0000 7f424affd640 0 client.16836 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:16:53.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.464+0000 7f4274c11740 1 -- 172.21.15.16:0/630513920 shutdown_connections 2024-03-28T22:16:53.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.464+0000 7f4274c11740 1 -- 172.21.15.16:0/630513920 wait complete. 2024-03-28T22:16:53.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.464+0000 7f4274c11740 20 client.16836 trim_cache size 0 max 16384 2024-03-28T22:16:53.482 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:16:53.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.464+0000 7f427202e640 1 -- 172.21.15.16:0/2964276468 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f42540a1ef0 msgr2=0x7f42540a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:53.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.464+0000 7f427202e640 1 --2- 172.21.15.16:0/2964276468 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f42540a1ef0 0x7f42540a43b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f426400bed0 tx=0x7f4264004550 comp rx=0 tx=0).stop 2024-03-28T22:16:53.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.465+0000 7f427202e640 1 -- 172.21.15.16:0/2964276468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f426c1029b0 msgr2=0x7f426c19a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:53.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.465+0000 7f427202e640 1 --2- 172.21.15.16:0/2964276468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f426c1029b0 0x7f426c19a1e0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f425c005e30 tx=0x7f425c035d70 comp rx=0 tx=0).stop 2024-03-28T22:16:53.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.465+0000 7f427202e640 1 -- 172.21.15.16:0/2964276468 shutdown_connections 2024-03-28T22:16:53.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.465+0000 7f427202e640 1 --2- 172.21.15.16:0/2964276468 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f42540a1ef0 0x7f42540a43b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.465+0000 7f427202e640 1 --2- 172.21.15.16:0/2964276468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f426c10bcf0 0x7f426c19ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.465+0000 7f427202e640 1 --2- 172.21.15.16:0/2964276468 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f426c1032d0 0x7f426c19a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.465+0000 7f427202e640 1 --2- 172.21.15.16:0/2964276468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f426c1029b0 0x7f426c19a1e0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.465+0000 7f427202e640 1 -- 172.21.15.16:0/2964276468 >> 172.21.15.16:0/2964276468 conn(0x7f426c077800 msgr2=0x7f426c107ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:53.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.465+0000 7f427202e640 1 -- 172.21.15.16:0/2964276468 shutdown_connections 2024-03-28T22:16:53.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.465+0000 7f427202e640 1 -- 172.21.15.16:0/2964276468 wait complete. 2024-03-28T22:16:53.486 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:16:53.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.587+0000 7f82ea4eb640 1 -- 172.21.15.16:0/1229813584 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82e4102970 msgr2=0x7f82e4102d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:53.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.587+0000 7f82ea4eb640 1 --2- 172.21.15.16:0/1229813584 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82e4102970 0x7f82e4102d50 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f82d4009860 tx=0x7f82d402bc00 comp rx=0 tx=0).stop 2024-03-28T22:16:53.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.587+0000 7f82ea4eb640 1 -- 172.21.15.16:0/1229813584 shutdown_connections 2024-03-28T22:16:53.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.587+0000 7f82ea4eb640 1 --2- 172.21.15.16:0/1229813584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82e410bd40 0x7f82e410e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.587+0000 7f82ea4eb640 1 --2- 172.21.15.16:0/1229813584 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f82e4103290 0x7f82e410b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.587+0000 7f82ea4eb640 1 --2- 172.21.15.16:0/1229813584 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82e4102970 0x7f82e4102d50 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.588+0000 7f82ea4eb640 1 -- 172.21.15.16:0/1229813584 >> 172.21.15.16:0/1229813584 conn(0x7f82e4077800 msgr2=0x7f82e4077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:53.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.588+0000 7f82ea4eb640 1 -- 172.21.15.16:0/1229813584 shutdown_connections 2024-03-28T22:16:53.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.588+0000 7f82ea4eb640 1 -- 172.21.15.16:0/1229813584 wait complete. 2024-03-28T22:16:53.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.588+0000 7f82ea4eb640 1 Processor -- start 2024-03-28T22:16:53.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.588+0000 7f82ea4eb640 1 -- start start 2024-03-28T22:16:53.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.588+0000 7f82ea4eb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82e4102970 0x7f82e4198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.588+0000 7f82ea4eb640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82e4103290 0x7f82e41985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.588+0000 7f82ea4eb640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f82e410bd40 0x7f82e419c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.588+0000 7f82ea4eb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f82e4111ad0 con 0x7f82e4102970 2024-03-28T22:16:53.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.588+0000 7f82ea4eb640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f82e4111950 con 0x7f82e4103290 2024-03-28T22:16:53.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.588+0000 7f82ea4eb640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f82e4111c50 con 0x7f82e410bd40 2024-03-28T22:16:53.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.589+0000 7f82e3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82e4102970 0x7f82e4198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.589+0000 7f82e3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82e4102970 0x7f82e4198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60332/0 (socket says 172.21.15.16:60332) 2024-03-28T22:16:53.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.589+0000 7f82e3fff640 1 -- 172.21.15.16:0/2040685106 learned_addr learned my addr 172.21.15.16:0/2040685106 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:53.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.589+0000 7f82e37fe640 1 --2- 172.21.15.16:0/2040685106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82e4103290 0x7f82e41985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.589+0000 7f82e8a88640 1 --2- 172.21.15.16:0/2040685106 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f82e410bd40 0x7f82e419c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.589+0000 7f82e3fff640 1 -- 172.21.15.16:0/2040685106 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f82e410bd40 msgr2=0x7f82e419c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:53.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.589+0000 7f82e3fff640 1 --2- 172.21.15.16:0/2040685106 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f82e410bd40 0x7f82e419c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.589+0000 7f82e3fff640 1 -- 172.21.15.16:0/2040685106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82e4103290 msgr2=0x7f82e41985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:53.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.589+0000 7f82e3fff640 1 --2- 172.21.15.16:0/2040685106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82e4103290 0x7f82e41985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.589+0000 7f82e3fff640 1 -- 172.21.15.16:0/2040685106 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82e419d010 con 0x7f82e4102970 2024-03-28T22:16:53.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.589+0000 7f82e8a88640 1 --2- 172.21.15.16:0/2040685106 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f82e410bd40 0x7f82e419c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:16:53.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.589+0000 7f82e37fe640 1 --2- 172.21.15.16:0/2040685106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82e4103290 0x7f82e41985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:16:53.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.589+0000 7f82e3fff640 1 --2- 172.21.15.16:0/2040685106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82e4102970 0x7f82e4198070 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f82d4037040 tx=0x7f82d40346f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:53.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.589+0000 7f82e17fa640 1 -- 172.21.15.16:0/2040685106 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d4041020 con 0x7f82e4102970 2024-03-28T22:16:53.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.589+0000 7f82ea4eb640 1 -- 172.21.15.16:0/2040685106 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82e419d2a0 con 0x7f82e4102970 2024-03-28T22:16:53.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.589+0000 7f82e17fa640 1 -- 172.21.15.16:0/2040685106 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82d40044d0 con 0x7f82e4102970 2024-03-28T22:16:53.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.589+0000 7f82e17fa640 1 -- 172.21.15.16:0/2040685106 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d4039040 con 0x7f82e4102970 2024-03-28T22:16:53.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.589+0000 7f82ea4eb640 1 -- 172.21.15.16:0/2040685106 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82e41a4b90 con 0x7f82e4102970 2024-03-28T22:16:53.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.591+0000 7f82e17fa640 1 -- 172.21.15.16:0/2040685106 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f82d4044020 con 0x7f82e4102970 2024-03-28T22:16:53.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.592+0000 7f82ed1a4740 1 Processor -- start 2024-03-28T22:16:53.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.592+0000 7f82ed1a4740 1 -- start start 2024-03-28T22:16:53.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.592+0000 7f82e17fa640 1 --2- 172.21.15.16:0/2040685106 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f82cc0a1ef0 0x7f82cc0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.592+0000 7f82ed1a4740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b908f6f470 0x55b908f6f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.592+0000 7f82ed1a4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b908f6fd90 0x55b908f78430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.592+0000 7f82ed1a4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b908f78970 0x55b908f7ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.592+0000 7f82ed1a4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b908ed31b0 con 0x55b908f78970 2024-03-28T22:16:53.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.592+0000 7f82ed1a4740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55b908e7bc00 con 0x55b908f6f470 2024-03-28T22:16:53.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.592+0000 7f82ed1a4740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55b908ecf250 con 0x55b908f6fd90 2024-03-28T22:16:53.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.592+0000 7f82e8a88640 1 --2- 172.21.15.16:0/2040685106 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f82cc0a1ef0 0x7f82cc0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.592+0000 7f82e3fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b908f6f470 0x55b908f6f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.592+0000 7f82e17fa640 1 -- 172.21.15.16:0/2040685106 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f82d40ef900 con 0x7f82e4102970 2024-03-28T22:16:53.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.592+0000 7f82e3fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b908f6f470 0x55b908f6f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:44370/0 (socket says 172.21.15.16:44370) 2024-03-28T22:16:53.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.592+0000 7f82e3fff640 1 -- 172.21.15.16:0/141943141 learned_addr learned my addr 172.21.15.16:0/141943141 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:53.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.592+0000 7f82e8a88640 1 --2- 172.21.15.16:0/141943141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b908f78970 0x55b908f7ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.592+0000 7f82e37fe640 1 --2- 172.21.15.16:0/141943141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b908f6fd90 0x55b908f78430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.592+0000 7f82e3fff640 1 -- 172.21.15.16:0/141943141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b908f6fd90 msgr2=0x55b908f78430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:53.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.592+0000 7f82e3fff640 1 --2- 172.21.15.16:0/141943141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b908f6fd90 0x55b908f78430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.592+0000 7f82e3fff640 1 -- 172.21.15.16:0/141943141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b908f78970 msgr2=0x55b908f7ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:53.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.592+0000 7f82e3fff640 1 --2- 172.21.15.16:0/141943141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b908f78970 0x55b908f7ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.592+0000 7f82e3fff640 1 -- 172.21.15.16:0/141943141 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b908e9b3d0 con 0x55b908f6f470 2024-03-28T22:16:53.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.592+0000 7f82e37fe640 1 --2- 172.21.15.16:0/141943141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b908f6fd90 0x55b908f78430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:16:53.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.593+0000 7f82e8a88640 1 --2- 172.21.15.16:0/2040685106 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f82cc0a1ef0 0x7f82cc0a43b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f82dc005e30 tx=0x7f82dc009e50 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:53.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.593+0000 7f82e3fff640 1 --2- 172.21.15.16:0/141943141 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b908f6f470 0x55b908f6f850 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f82d40b8df0 tx=0x7f82d40b6f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ef07c1801048853 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:53.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.593+0000 7f82ca7fc640 1 -- 172.21.15.16:0/141943141 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d40cee50 con 0x55b908f6f470 2024-03-28T22:16:53.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.593+0000 7f82ca7fc640 1 -- 172.21.15.16:0/141943141 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82d40bcb20 con 0x55b908f6f470 2024-03-28T22:16:53.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.593+0000 7f82ca7fc640 1 -- 172.21.15.16:0/141943141 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d40d1840 con 0x55b908f6f470 2024-03-28T22:16:53.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.593+0000 7f82ed1a4740 1 -- 172.21.15.16:0/141943141 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b908f6f470 msgr2=0x55b908f6f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:53.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.593+0000 7f82ed1a4740 1 --2- 172.21.15.16:0/141943141 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b908f6f470 0x55b908f6f850 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f82d40b8df0 tx=0x7f82d40b6f90 comp rx=0 tx=0).stop 2024-03-28T22:16:53.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.593+0000 7f82ed1a4740 1 -- 172.21.15.16:0/141943141 shutdown_connections 2024-03-28T22:16:53.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.593+0000 7f82ed1a4740 1 --2- 172.21.15.16:0/141943141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b908f78970 0x55b908f7ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.593+0000 7f82ed1a4740 1 --2- 172.21.15.16:0/141943141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b908f6fd90 0x55b908f78430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.593+0000 7f82ed1a4740 1 --2- 172.21.15.16:0/141943141 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b908f6f470 0x55b908f6f850 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.594+0000 7f82ed1a4740 1 -- 172.21.15.16:0/141943141 >> 172.21.15.16:0/141943141 conn(0x55b908f69ba0 msgr2=0x55b908f69f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:53.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.594+0000 7f82ed1a4740 1 -- 172.21.15.16:0/141943141 shutdown_connections 2024-03-28T22:16:53.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.594+0000 7f82ed1a4740 1 -- 172.21.15.16:0/141943141 wait complete. 2024-03-28T22:16:53.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.594+0000 7f82ed1a4740 1 -- start start 2024-03-28T22:16:53.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.594+0000 7f82ed1a4740 1 Processor -- start 2024-03-28T22:16:53.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.594+0000 7f82ed1a4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b908f6f470 0x55b908f75960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.594+0000 7f82ed1a4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b908f6fd90 0x55b908f75ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.594+0000 7f82ed1a4740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b908f78970 0x55b908f763e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.594+0000 7f82ed1a4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b908f7b520 con 0x55b908f6fd90 2024-03-28T22:16:53.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.594+0000 7f82ed1a4740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55b908e71470 con 0x55b908f78970 2024-03-28T22:16:53.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.594+0000 7f82ed1a4740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55b908f7b6a0 con 0x55b908f6f470 2024-03-28T22:16:53.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.594+0000 7f82ed1a4740 10 client.0 _ll_register_callbacks cb 0x55b908e8ecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:16:53.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.594+0000 7f82e37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b908f6fd90 0x55b908f75ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.595+0000 7f82e37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b908f6fd90 0x55b908f75ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60352/0 (socket says 172.21.15.16:60352) 2024-03-28T22:16:53.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.595+0000 7f82e37fe640 1 -- 172.21.15.16:0/2218511057 learned_addr learned my addr 172.21.15.16:0/2218511057 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:53.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.595+0000 7f82e8a88640 1 --2- 172.21.15.16:0/2218511057 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b908f78970 0x55b908f763e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.595+0000 7f82e3fff640 1 --2- 172.21.15.16:0/2218511057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b908f6f470 0x55b908f75960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.595+0000 7f82e37fe640 1 -- 172.21.15.16:0/2218511057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b908f6f470 msgr2=0x55b908f75960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:53.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.595+0000 7f82e37fe640 1 --2- 172.21.15.16:0/2218511057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b908f6f470 0x55b908f75960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.595+0000 7f82e37fe640 1 -- 172.21.15.16:0/2218511057 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b908f78970 msgr2=0x55b908f763e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:53.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.595+0000 7f82e37fe640 1 --2- 172.21.15.16:0/2218511057 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b908f78970 0x55b908f763e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.595+0000 7f82e37fe640 1 -- 172.21.15.16:0/2218511057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b908be73f0 con 0x55b908f6fd90 2024-03-28T22:16:53.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.595+0000 7f82e37fe640 1 --2- 172.21.15.16:0/2218511057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b908f6fd90 0x55b908f75ea0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f82d8012780 tx=0x7f82d8012c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39778815c731569b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:53.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.595+0000 7f82c37fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:16:53.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.595+0000 7f82c37fe640 1 -- 172.21.15.16:0/2218511057 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d8010dc0 con 0x55b908f6fd90 2024-03-28T22:16:53.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.595+0000 7f82c37fe640 1 -- 172.21.15.16:0/2218511057 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f82d8004020 con 0x55b908f6fd90 2024-03-28T22:16:53.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.595+0000 7f82ed1a4740 1 -- 172.21.15.16:0/2218511057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82d40bcb20 con 0x55b908f6fd90 2024-03-28T22:16:53.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.595+0000 7f82c37fe640 1 -- 172.21.15.16:0/2218511057 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f82d8004220 con 0x55b908f6fd90 2024-03-28T22:16:53.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.595+0000 7f82ed1a4740 10 client.16848 fetch_fsmap learned FSMap version 27 2024-03-28T22:16:53.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.595+0000 7f82ed1a4740 1 -- 172.21.15.16:0/2218511057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b908cb17d0 con 0x55b908f6fd90 2024-03-28T22:16:53.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.595+0000 7f82c37fe640 1 -- 172.21.15.16:0/2218511057 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82d40bcb20 con 0x55b908f6fd90 2024-03-28T22:16:53.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.596+0000 7f82c37fe640 1 -- 172.21.15.16:0/2218511057 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f82d8004b20 con 0x55b908f6fd90 2024-03-28T22:16:53.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.596+0000 7f82ed1a4740 10 client.16848 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:16:53.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.596+0000 7f82ed1a4740 10 client.16848 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:16:53.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.596+0000 7f82ed1a4740 20 client.16848 populate_metadata read hostname 'smithi016' 2024-03-28T22:16:53.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.596+0000 7f82ed1a4740 1 --2- 172.21.15.16:0/2218511057 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b908f886a0 0x55b908f8ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.596+0000 7f82ed1a4740 4 client.16848 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:16:53.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.596+0000 7f82ed1a4740 1 -- 172.21.15.16:0/2218511057 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b908ee06f0 con 0x55b908f886a0 2024-03-28T22:16:53.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.596+0000 7f82e3fff640 1 --2- 172.21.15.16:0/2218511057 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b908f886a0 0x55b908f8ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.596+0000 7f82e3fff640 1 --2- 172.21.15.16:0/2218511057 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b908f886a0 0x55b908f8ab10 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a1de96bbb19d4ca server_cookie=951337b3ed119a7d in_seq=0 out_seq=0 2024-03-28T22:16:53.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.596+0000 7f82c37fe640 10 client.16848 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:16:53.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.597+0000 7f82c37fe640 1 -- 172.21.15.16:0/2218511057 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b908ee06f0 con 0x55b908f886a0 2024-03-28T22:16:53.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.597+0000 7f82c37fe640 10 client.16848 handle_command_reply: tid=0 2024-03-28T22:16:53.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.597+0000 7f82ed1a4740 1 client.16848 shutdown 2024-03-28T22:16:53.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.597+0000 7f82ed1a4740 1 -- 172.21.15.16:0/2218511057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b908f6fd90 msgr2=0x55b908f75ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:53.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.597+0000 7f82ed1a4740 1 --2- 172.21.15.16:0/2218511057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b908f6fd90 0x55b908f75ea0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f82d8012780 tx=0x7f82d8012c40 comp rx=0 tx=0).stop 2024-03-28T22:16:53.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.597+0000 7f82ed1a4740 1 -- 172.21.15.16:0/2218511057 shutdown_connections 2024-03-28T22:16:53.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.597+0000 7f82ed1a4740 1 --2- 172.21.15.16:0/2218511057 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b908f886a0 0x55b908f8ab10 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.597+0000 7f82ed1a4740 1 --2- 172.21.15.16:0/2218511057 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b908f78970 0x55b908f763e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.597+0000 7f82ed1a4740 1 --2- 172.21.15.16:0/2218511057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b908f6fd90 0x55b908f75ea0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.597+0000 7f82c37fe640 0 client.16848 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:16:53.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.597+0000 7f82ed1a4740 1 --2- 172.21.15.16:0/2218511057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b908f6f470 0x55b908f75960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.597+0000 7f82ed1a4740 1 -- 172.21.15.16:0/2218511057 >> 172.21.15.16:0/2218511057 conn(0x55b908f69ba0 msgr2=0x55b908f6b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:53.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.597+0000 7f82ed1a4740 1 -- 172.21.15.16:0/2218511057 shutdown_connections 2024-03-28T22:16:53.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.597+0000 7f82ed1a4740 1 -- 172.21.15.16:0/2218511057 wait complete. 2024-03-28T22:16:53.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.597+0000 7f82ed1a4740 20 client.16848 trim_cache size 0 max 16384 2024-03-28T22:16:53.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.610+0000 7f82ed1a4740 1 Processor -- start 2024-03-28T22:16:53.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.610+0000 7f82ed1a4740 1 -- start start 2024-03-28T22:16:53.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.610+0000 7f82ed1a4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b908f6f470 0x55b908f84210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.610+0000 7f82ed1a4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b908f6fd90 0x55b908f68bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.610+0000 7f82ed1a4740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b908f78970 0x55b908f903d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.610+0000 7f82ed1a4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f82e41afca0 con 0x55b908f6fd90 2024-03-28T22:16:53.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.610+0000 7f82ed1a4740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f82e41ad570 con 0x55b908f78970 2024-03-28T22:16:53.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.610+0000 7f82ed1a4740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f82e41ab3c0 con 0x55b908f6f470 2024-03-28T22:16:53.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.611+0000 7f82e37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b908f6fd90 0x55b908f68bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.611+0000 7f82e37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b908f6fd90 0x55b908f68bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60366/0 (socket says 172.21.15.16:60366) 2024-03-28T22:16:53.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.611+0000 7f82e37fe640 1 -- 172.21.15.16:0/943291247 learned_addr learned my addr 172.21.15.16:0/943291247 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:53.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.611+0000 7f82e8a88640 1 --2- 172.21.15.16:0/943291247 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b908f78970 0x55b908f903d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.611+0000 7f82e3fff640 1 --2- 172.21.15.16:0/943291247 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b908f6f470 0x55b908f84210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.611+0000 7f82e37fe640 1 -- 172.21.15.16:0/943291247 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b908f6f470 msgr2=0x55b908f84210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:53.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.611+0000 7f82e37fe640 1 --2- 172.21.15.16:0/943291247 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b908f6f470 0x55b908f84210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.611+0000 7f82e37fe640 1 -- 172.21.15.16:0/943291247 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b908f78970 msgr2=0x55b908f903d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:53.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.611+0000 7f82e37fe640 1 --2- 172.21.15.16:0/943291247 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b908f78970 0x55b908f903d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.611+0000 7f82e37fe640 1 -- 172.21.15.16:0/943291247 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b908f73eb0 con 0x55b908f6fd90 2024-03-28T22:16:53.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.611+0000 7f82e37fe640 1 --2- 172.21.15.16:0/943291247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b908f6fd90 0x55b908f68bc0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f82d8000c00 tx=0x7f82d8010820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0652189bd68d6f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:53.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.611+0000 7f82c9ffb640 1 -- 172.21.15.16:0/943291247 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d8028040 con 0x55b908f6fd90 2024-03-28T22:16:53.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.611+0000 7f82c9ffb640 1 -- 172.21.15.16:0/943291247 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82d8010dc0 con 0x55b908f6fd90 2024-03-28T22:16:53.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.611+0000 7f82c9ffb640 1 -- 172.21.15.16:0/943291247 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d8004d20 con 0x55b908f6fd90 2024-03-28T22:16:53.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.612+0000 7f82ed1a4740 1 -- 172.21.15.16:0/943291247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b908f6fd90 msgr2=0x55b908f68bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:53.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.612+0000 7f82ed1a4740 1 --2- 172.21.15.16:0/943291247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b908f6fd90 0x55b908f68bc0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f82d8000c00 tx=0x7f82d8010820 comp rx=0 tx=0).stop 2024-03-28T22:16:53.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.612+0000 7f82ed1a4740 1 -- 172.21.15.16:0/943291247 shutdown_connections 2024-03-28T22:16:53.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.612+0000 7f82ed1a4740 1 --2- 172.21.15.16:0/943291247 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b908f78970 0x55b908f903d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.612+0000 7f82ed1a4740 1 --2- 172.21.15.16:0/943291247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b908f6fd90 0x55b908f68bc0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.612+0000 7f82ed1a4740 1 --2- 172.21.15.16:0/943291247 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b908f6f470 0x55b908f84210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.612+0000 7f82ed1a4740 1 -- 172.21.15.16:0/943291247 >> 172.21.15.16:0/943291247 conn(0x55b908f69ba0 msgr2=0x55b908f684b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:53.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.612+0000 7f82ed1a4740 1 -- 172.21.15.16:0/943291247 shutdown_connections 2024-03-28T22:16:53.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.612+0000 7f82ed1a4740 1 -- 172.21.15.16:0/943291247 wait complete. 2024-03-28T22:16:53.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.612+0000 7f82ed1a4740 1 -- start start 2024-03-28T22:16:53.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.612+0000 7f82ed1a4740 1 Processor -- start 2024-03-28T22:16:53.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.612+0000 7f82ed1a4740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b908f6f470 0x55b908f80c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.612+0000 7f82ed1a4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b908f6fd90 0x55b908f7c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.613+0000 7f82ed1a4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b908f78970 0x55b908f7c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.613+0000 7f82ed1a4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f82e41aba30 con 0x55b908f6fd90 2024-03-28T22:16:53.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.613+0000 7f82ed1a4740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f82e41acea0 con 0x55b908f6f470 2024-03-28T22:16:53.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.613+0000 7f82ed1a4740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f82e41ac100 con 0x55b908f78970 2024-03-28T22:16:53.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.613+0000 7f82ed1a4740 10 client.0 _ll_register_callbacks cb 0x55b908e8ecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:16:53.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.613+0000 7f82e37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b908f6fd90 0x55b908f7c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.613+0000 7f82e3fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b908f6f470 0x55b908f80c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.613+0000 7f82e3fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b908f6f470 0x55b908f80c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:44396/0 (socket says 172.21.15.16:44396) 2024-03-28T22:16:53.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.613+0000 7f82e37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b908f6fd90 0x55b908f7c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60382/0 (socket says 172.21.15.16:60382) 2024-03-28T22:16:53.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.613+0000 7f82e3fff640 1 -- 172.21.15.16:0/1793499015 learned_addr learned my addr 172.21.15.16:0/1793499015 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:53.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.613+0000 7f82e8a88640 1 --2- 172.21.15.16:0/1793499015 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b908f78970 0x55b908f7c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.613+0000 7f82e3fff640 1 -- 172.21.15.16:0/1793499015 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b908f78970 msgr2=0x55b908f7c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:53.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.613+0000 7f82e3fff640 1 --2- 172.21.15.16:0/1793499015 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b908f78970 0x55b908f7c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.613+0000 7f82e3fff640 1 -- 172.21.15.16:0/1793499015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b908f6fd90 msgr2=0x55b908f7c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:53.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.613+0000 7f82e3fff640 1 --2- 172.21.15.16:0/1793499015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b908f6fd90 0x55b908f7c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.613+0000 7f82e3fff640 1 -- 172.21.15.16:0/1793499015 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b908feda00 con 0x55b908f6f470 2024-03-28T22:16:53.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.613+0000 7f82e37fe640 1 --2- 172.21.15.16:0/1793499015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b908f6fd90 0x55b908f7c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:16:53.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.613+0000 7f82e8a88640 1 --2- 172.21.15.16:0/1793499015 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b908f78970 0x55b908f7c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:16:53.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.613+0000 7f82e3fff640 1 --2- 172.21.15.16:0/1793499015 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b908f6f470 0x55b908f80c90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f82d4009830 tx=0x7f82d40035f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7de570a0cd2a465 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:53.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.613+0000 7f82c37fe640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-03-28T22:16:53.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.613+0000 7f82c37fe640 1 -- 172.21.15.16:0/1793499015 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d40cf030 con 0x55b908f6f470 2024-03-28T22:16:53.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.613+0000 7f82c37fe640 1 -- 172.21.15.16:0/1793499015 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f82d40ba720 con 0x55b908f6f470 2024-03-28T22:16:53.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.613+0000 7f82ed1a4740 1 -- 172.21.15.16:0/1793499015 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82d8010dc0 con 0x55b908f6f470 2024-03-28T22:16:53.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.613+0000 7f82c37fe640 1 -- 172.21.15.16:0/1793499015 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f82d40cea90 con 0x55b908f6f470 2024-03-28T22:16:53.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.613+0000 7f82ed1a4740 10 client.26756 fetch_fsmap learned FSMap version 27 2024-03-28T22:16:53.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.613+0000 7f82ed1a4740 1 -- 172.21.15.16:0/1793499015 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b908feeba0 con 0x55b908f6f470 2024-03-28T22:16:53.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.614+0000 7f82c37fe640 1 -- 172.21.15.16:0/1793499015 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82d8010dc0 con 0x55b908f6f470 2024-03-28T22:16:53.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.614+0000 7f82c37fe640 1 -- 172.21.15.16:0/1793499015 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f82d403c030 con 0x55b908f6f470 2024-03-28T22:16:53.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.614+0000 7f82ed1a4740 10 client.26756 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:16:53.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.614+0000 7f82ed1a4740 10 client.26756 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:16:53.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.614+0000 7f82ed1a4740 20 client.26756 populate_metadata read hostname 'smithi016' 2024-03-28T22:16:53.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.614+0000 7f82ed1a4740 1 --2- 172.21.15.16:0/1793499015 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b908feefa0 0x55b908ff1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.614+0000 7f82ed1a4740 4 client.26756 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:16:53.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.614+0000 7f82ed1a4740 1 -- 172.21.15.16:0/1793499015 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b908edaa40 con 0x55b908feefa0 2024-03-28T22:16:53.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.614+0000 7f82e37fe640 1 --2- 172.21.15.16:0/1793499015 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b908feefa0 0x55b908ff1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:53.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.614+0000 7f82c37fe640 10 client.26756 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:16:53.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.614+0000 7f82e37fe640 1 --2- 172.21.15.16:0/1793499015 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b908feefa0 0x55b908ff1390 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f92fcb0b7cbae25 server_cookie=5b8a591b419838f2 in_seq=0 out_seq=0 2024-03-28T22:16:53.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.614+0000 7f82c37fe640 1 -- 172.21.15.16:0/1793499015 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b908edaa40 con 0x55b908feefa0 2024-03-28T22:16:53.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.614+0000 7f82c37fe640 10 client.26756 handle_command_reply: tid=0 2024-03-28T22:16:53.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.614+0000 7f82ed1a4740 1 client.26756 shutdown 2024-03-28T22:16:53.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.615+0000 7f82ed1a4740 1 -- 172.21.15.16:0/1793499015 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b908f6f470 msgr2=0x55b908f80c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:53.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.615+0000 7f82ed1a4740 1 --2- 172.21.15.16:0/1793499015 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b908f6f470 0x55b908f80c90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f82d4009830 tx=0x7f82d40035f0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.615+0000 7f82ed1a4740 1 -- 172.21.15.16:0/1793499015 shutdown_connections 2024-03-28T22:16:53.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.615+0000 7f82ed1a4740 1 --2- 172.21.15.16:0/1793499015 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b908feefa0 0x55b908ff1390 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.615+0000 7f82ed1a4740 1 --2- 172.21.15.16:0/1793499015 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b908f78970 0x55b908f7c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.615+0000 7f82ed1a4740 1 --2- 172.21.15.16:0/1793499015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b908f6fd90 0x55b908f7c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.615+0000 7f82ed1a4740 1 --2- 172.21.15.16:0/1793499015 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b908f6f470 0x55b908f80c90 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.615+0000 7f82c37fe640 0 client.26756 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:16:53.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.615+0000 7f82ed1a4740 1 -- 172.21.15.16:0/1793499015 >> 172.21.15.16:0/1793499015 conn(0x55b908f69ba0 msgr2=0x55b908f68d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:53.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.615+0000 7f82ed1a4740 1 -- 172.21.15.16:0/1793499015 shutdown_connections 2024-03-28T22:16:53.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.615+0000 7f82ed1a4740 1 -- 172.21.15.16:0/1793499015 wait complete. 2024-03-28T22:16:53.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.615+0000 7f82ed1a4740 20 client.26756 trim_cache size 0 max 16384 2024-03-28T22:16:53.631 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:16:53.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.615+0000 7f82ea4eb640 1 -- 172.21.15.16:0/2040685106 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f82cc0a1ef0 msgr2=0x7f82cc0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:53.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.615+0000 7f82ea4eb640 1 --2- 172.21.15.16:0/2040685106 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f82cc0a1ef0 0x7f82cc0a43b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f82dc005e30 tx=0x7f82dc009e50 comp rx=0 tx=0).stop 2024-03-28T22:16:53.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.615+0000 7f82ea4eb640 1 -- 172.21.15.16:0/2040685106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82e4102970 msgr2=0x7f82e4198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:53.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.615+0000 7f82ea4eb640 1 --2- 172.21.15.16:0/2040685106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82e4102970 0x7f82e4198070 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f82d4037040 tx=0x7f82d40346f0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.615+0000 7f82ea4eb640 1 -- 172.21.15.16:0/2040685106 shutdown_connections 2024-03-28T22:16:53.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.615+0000 7f82ea4eb640 1 --2- 172.21.15.16:0/2040685106 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f82cc0a1ef0 0x7f82cc0a43b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.615+0000 7f82ea4eb640 1 --2- 172.21.15.16:0/2040685106 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f82e410bd40 0x7f82e419c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.615+0000 7f82ea4eb640 1 --2- 172.21.15.16:0/2040685106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82e4103290 0x7f82e41985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.615+0000 7f82ea4eb640 1 --2- 172.21.15.16:0/2040685106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82e4102970 0x7f82e4198070 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.615+0000 7f82ea4eb640 1 -- 172.21.15.16:0/2040685106 >> 172.21.15.16:0/2040685106 conn(0x7f82e4077800 msgr2=0x7f82e40fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:53.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.615+0000 7f82ea4eb640 1 -- 172.21.15.16:0/2040685106 shutdown_connections 2024-03-28T22:16:53.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:53.615+0000 7f82ea4eb640 1 -- 172.21.15.16:0/2040685106 wait complete. 2024-03-28T22:16:53.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:53 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/4005563281' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:16:53.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:53 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/4005563281' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:16:54.638 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:16:54.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:16:54.639 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:16:54.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:54 smithi016 ceph-mon[32374]: pgmap v192: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:16:54.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:54 smithi063 ceph-mon[36249]: pgmap v192: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:16:54.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.756+0000 7faeed2ed640 1 -- 172.21.15.16:0/14139778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faee8103b90 msgr2=0x7faee810e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:54.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.756+0000 7faeed2ed640 1 --2- 172.21.15.16:0/14139778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faee8103b90 0x7faee810e6c0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7faee0009900 tx=0x7faee002bd10 comp rx=0 tx=0).stop 2024-03-28T22:16:54.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.756+0000 7faeed2ed640 1 -- 172.21.15.16:0/14139778 shutdown_connections 2024-03-28T22:16:54.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.756+0000 7faeed2ed640 1 --2- 172.21.15.16:0/14139778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faee8103b90 0x7faee810e6c0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.756+0000 7faeed2ed640 1 --2- 172.21.15.16:0/14139778 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faee81031f0 0x7faee8103650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.756+0000 7faeed2ed640 1 --2- 172.21.15.16:0/14139778 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faee8101ab0 0x7faee8101e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.756+0000 7faeed2ed640 1 -- 172.21.15.16:0/14139778 >> 172.21.15.16:0/14139778 conn(0x7faee80fd1e0 msgr2=0x7faee80ff600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:54.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.756+0000 7faeed2ed640 1 -- 172.21.15.16:0/14139778 shutdown_connections 2024-03-28T22:16:54.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.756+0000 7faeed2ed640 1 -- 172.21.15.16:0/14139778 wait complete. 2024-03-28T22:16:54.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.757+0000 7faeed2ed640 1 Processor -- start 2024-03-28T22:16:54.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.757+0000 7faeed2ed640 1 -- start start 2024-03-28T22:16:54.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.757+0000 7faeed2ed640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faee8101ab0 0x7faee8197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:54.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.757+0000 7faeed2ed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faee81031f0 0x7faee8198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:54.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.757+0000 7faeed2ed640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faee8103b90 0x7faee819c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:54.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.757+0000 7faeed2ed640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faee8111c00 con 0x7faee81031f0 2024-03-28T22:16:54.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.757+0000 7faeed2ed640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7faee8111a80 con 0x7faee8103b90 2024-03-28T22:16:54.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.757+0000 7faeed2ed640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7faee8111d80 con 0x7faee8101ab0 2024-03-28T22:16:54.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.757+0000 7faee67fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faee81031f0 0x7faee8198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:54.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.757+0000 7faee77fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faee8103b90 0x7faee819c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:54.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.757+0000 7faee67fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faee81031f0 0x7faee8198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60396/0 (socket says 172.21.15.16:60396) 2024-03-28T22:16:54.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.757+0000 7faee77fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faee8103b90 0x7faee819c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:44412/0 (socket says 172.21.15.16:44412) 2024-03-28T22:16:54.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.757+0000 7faee67fc640 1 -- 172.21.15.16:0/3626213089 learned_addr learned my addr 172.21.15.16:0/3626213089 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:54.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.757+0000 7faee6ffd640 1 --2- 172.21.15.16:0/3626213089 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faee8101ab0 0x7faee8197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:54.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.758+0000 7faee67fc640 1 -- 172.21.15.16:0/3626213089 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faee8101ab0 msgr2=0x7faee8197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:54.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.758+0000 7faee67fc640 1 --2- 172.21.15.16:0/3626213089 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faee8101ab0 0x7faee8197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.758+0000 7faee67fc640 1 -- 172.21.15.16:0/3626213089 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faee8103b90 msgr2=0x7faee819c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:54.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.758+0000 7faee67fc640 1 --2- 172.21.15.16:0/3626213089 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faee8103b90 0x7faee819c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.758+0000 7faee67fc640 1 -- 172.21.15.16:0/3626213089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faee819cfd0 con 0x7faee81031f0 2024-03-28T22:16:54.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.758+0000 7faee77fe640 1 --2- 172.21.15.16:0/3626213089 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faee8103b90 0x7faee819c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:16:54.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.758+0000 7faee67fc640 1 --2- 172.21.15.16:0/3626213089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faee81031f0 0x7faee8198520 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7faed000c920 tx=0x7faed000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:54.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.758+0000 7faedffff640 1 -- 172.21.15.16:0/3626213089 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed0007c70 con 0x7faee81031f0 2024-03-28T22:16:54.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.758+0000 7faeed2ed640 1 -- 172.21.15.16:0/3626213089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faee819d2c0 con 0x7faee81031f0 2024-03-28T22:16:54.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.758+0000 7faedffff640 1 -- 172.21.15.16:0/3626213089 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faed0007e10 con 0x7faee81031f0 2024-03-28T22:16:54.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.758+0000 7faedffff640 1 -- 172.21.15.16:0/3626213089 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed0018890 con 0x7faee81031f0 2024-03-28T22:16:54.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.758+0000 7faeed2ed640 1 -- 172.21.15.16:0/3626213089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faee81a4b50 con 0x7faee81031f0 2024-03-28T22:16:54.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.760+0000 7faedffff640 1 -- 172.21.15.16:0/3626213089 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7faed0018a70 con 0x7faee81031f0 2024-03-28T22:16:54.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.761+0000 7faeeffae740 1 Processor -- start 2024-03-28T22:16:54.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.761+0000 7faeeffae740 1 -- start start 2024-03-28T22:16:54.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.761+0000 7faeeffae740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562c3f7ba720 0x562c3f7bab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:54.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.761+0000 7faedffff640 1 --2- 172.21.15.16:0/3626213089 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7faed40a1ef0 0x7faed40a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:54.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.761+0000 7faeeffae740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562c3f7bb040 0x562c3f7c7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:54.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.761+0000 7faeeffae740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562c3f7c8300 0x562c3f7ca6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:54.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.761+0000 7faeeffae740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562c3f6bb0b0 con 0x562c3f7c8300 2024-03-28T22:16:54.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.761+0000 7faeeffae740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x562c3f71dea0 con 0x562c3f7ba720 2024-03-28T22:16:54.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.761+0000 7faeeffae740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x562c3f6c6c00 con 0x562c3f7bb040 2024-03-28T22:16:54.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.761+0000 7faee77fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562c3f7ba720 0x562c3f7bab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:54.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.761+0000 7faee6ffd640 1 --2- 172.21.15.16:0/3626213089 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7faed40a1ef0 0x7faed40a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:54.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.761+0000 7faee77fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562c3f7ba720 0x562c3f7bab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:44426/0 (socket says 172.21.15.16:44426) 2024-03-28T22:16:54.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.761+0000 7faee77fe640 1 -- 172.21.15.16:0/3430934295 learned_addr learned my addr 172.21.15.16:0/3430934295 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:54.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.761+0000 7faedffff640 1 -- 172.21.15.16:0/3626213089 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7faed00ce240 con 0x7faee81031f0 2024-03-28T22:16:54.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.762+0000 7faee77fe640 1 --2- 172.21.15.16:0/3430934295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562c3f7c8300 0x562c3f7ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:54.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.762+0000 7faee67fc640 1 --2- 172.21.15.16:0/3430934295 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562c3f7bb040 0x562c3f7c7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:54.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.762+0000 7faee77fe640 1 -- 172.21.15.16:0/3430934295 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562c3f7bb040 msgr2=0x562c3f7c7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:54.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.762+0000 7faee77fe640 1 --2- 172.21.15.16:0/3430934295 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562c3f7bb040 0x562c3f7c7dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.762+0000 7faee77fe640 1 -- 172.21.15.16:0/3430934295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562c3f7c8300 msgr2=0x562c3f7ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:54.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.762+0000 7faee77fe640 1 --2- 172.21.15.16:0/3430934295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562c3f7c8300 0x562c3f7ca6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.762+0000 7faee77fe640 1 -- 172.21.15.16:0/3430934295 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c3f4fa7e0 con 0x562c3f7ba720 2024-03-28T22:16:54.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.762+0000 7faee67fc640 1 --2- 172.21.15.16:0/3430934295 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562c3f7bb040 0x562c3f7c7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:16:54.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.762+0000 7faee6ffd640 1 --2- 172.21.15.16:0/3626213089 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7faed40a1ef0 0x7faed40a43b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7faed8005fd0 tx=0x7faed8009500 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:54.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.762+0000 7faee77fe640 1 --2- 172.21.15.16:0/3430934295 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562c3f7ba720 0x562c3f7bab00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7faee0036040 tx=0x7faee00095a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ffeb20e54bda8a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:54.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.762+0000 7faedd7fa640 1 -- 172.21.15.16:0/3430934295 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee00040f0 con 0x562c3f7ba720 2024-03-28T22:16:54.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.762+0000 7faedd7fa640 1 -- 172.21.15.16:0/3430934295 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faee002ce10 con 0x562c3f7ba720 2024-03-28T22:16:54.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.762+0000 7faedd7fa640 1 -- 172.21.15.16:0/3430934295 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee0005530 con 0x562c3f7ba720 2024-03-28T22:16:54.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.763+0000 7faeeffae740 1 -- 172.21.15.16:0/3430934295 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562c3f7ba720 msgr2=0x562c3f7bab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:54.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.763+0000 7faeeffae740 1 --2- 172.21.15.16:0/3430934295 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562c3f7ba720 0x562c3f7bab00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7faee0036040 tx=0x7faee00095a0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.763+0000 7faeeffae740 1 -- 172.21.15.16:0/3430934295 shutdown_connections 2024-03-28T22:16:54.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.763+0000 7faeeffae740 1 --2- 172.21.15.16:0/3430934295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562c3f7c8300 0x562c3f7ca6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.763+0000 7faeeffae740 1 --2- 172.21.15.16:0/3430934295 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562c3f7bb040 0x562c3f7c7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.763+0000 7faeeffae740 1 --2- 172.21.15.16:0/3430934295 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562c3f7ba720 0x562c3f7bab00 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.763+0000 7faeeffae740 1 -- 172.21.15.16:0/3430934295 >> 172.21.15.16:0/3430934295 conn(0x562c3f7b5120 msgr2=0x562c3f7b5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:54.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.763+0000 7faeeffae740 1 -- 172.21.15.16:0/3430934295 shutdown_connections 2024-03-28T22:16:54.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.763+0000 7faeeffae740 1 -- 172.21.15.16:0/3430934295 wait complete. 2024-03-28T22:16:54.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.763+0000 7faeeffae740 1 -- start start 2024-03-28T22:16:54.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.763+0000 7faeeffae740 1 Processor -- start 2024-03-28T22:16:54.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.764+0000 7faeeffae740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562c3f7ba720 0x562c3f7c5320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:54.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.764+0000 7faeeffae740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562c3f7bb040 0x562c3f7c5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:54.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.764+0000 7faeeffae740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562c3f7c8300 0x562c3f7c5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:54.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.764+0000 7faeeffae740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562c3f6bc3d0 con 0x562c3f7bb040 2024-03-28T22:16:54.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.764+0000 7faeeffae740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x562c3f719f40 con 0x562c3f7c8300 2024-03-28T22:16:54.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.764+0000 7faeeffae740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x562c3f7cb060 con 0x562c3f7ba720 2024-03-28T22:16:54.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.764+0000 7faeeffae740 10 client.0 _ll_register_callbacks cb 0x562c3f6d9b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:16:54.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.764+0000 7faee67fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562c3f7bb040 0x562c3f7c5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:54.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.764+0000 7faee67fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562c3f7bb040 0x562c3f7c5860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60422/0 (socket says 172.21.15.16:60422) 2024-03-28T22:16:54.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.764+0000 7faee67fc640 1 -- 172.21.15.16:0/2821318970 learned_addr learned my addr 172.21.15.16:0/2821318970 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:54.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.764+0000 7faee77fe640 1 --2- 172.21.15.16:0/2821318970 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562c3f7c8300 0x562c3f7c5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:54.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.764+0000 7faee67fc640 1 -- 172.21.15.16:0/2821318970 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562c3f7ba720 msgr2=0x562c3f7c5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:54.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.764+0000 7faee6ffd640 1 --2- 172.21.15.16:0/2821318970 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562c3f7ba720 0x562c3f7c5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:54.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.764+0000 7faee67fc640 1 --2- 172.21.15.16:0/2821318970 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562c3f7ba720 0x562c3f7c5320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.764+0000 7faee67fc640 1 -- 172.21.15.16:0/2821318970 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562c3f7c8300 msgr2=0x562c3f7c5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:54.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.764+0000 7faee67fc640 1 --2- 172.21.15.16:0/2821318970 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562c3f7c8300 0x562c3f7c5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.764+0000 7faee67fc640 1 -- 172.21.15.16:0/2821318970 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c3f55af70 con 0x562c3f7bb040 2024-03-28T22:16:54.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.764+0000 7faee67fc640 1 --2- 172.21.15.16:0/2821318970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562c3f7bb040 0x562c3f7c5860 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7faed00cba10 tx=0x7faed009a670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af2f71926ad78cbc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:54.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.764+0000 7faece7fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:16:54.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.764+0000 7faece7fc640 1 -- 172.21.15.16:0/2821318970 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed00131b0 con 0x562c3f7bb040 2024-03-28T22:16:54.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.764+0000 7faece7fc640 1 -- 172.21.15.16:0/2821318970 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7faed00972c0 con 0x562c3f7bb040 2024-03-28T22:16:54.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.764+0000 7faeeffae740 1 -- 172.21.15.16:0/2821318970 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faee002ce10 con 0x562c3f7bb040 2024-03-28T22:16:54.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.765+0000 7faece7fc640 1 -- 172.21.15.16:0/2821318970 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7faed009ac50 con 0x562c3f7bb040 2024-03-28T22:16:54.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.765+0000 7faeeffae740 10 client.16866 fetch_fsmap learned FSMap version 27 2024-03-28T22:16:54.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.765+0000 7faeeffae740 1 -- 172.21.15.16:0/2821318970 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562c3f3f2170 con 0x562c3f7bb040 2024-03-28T22:16:54.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.765+0000 7faece7fc640 1 -- 172.21.15.16:0/2821318970 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faee002ce10 con 0x562c3f7bb040 2024-03-28T22:16:54.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.765+0000 7faece7fc640 1 -- 172.21.15.16:0/2821318970 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7faed00a39c0 con 0x562c3f7bb040 2024-03-28T22:16:54.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.765+0000 7faeeffae740 10 client.16866 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:16:54.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.765+0000 7faeeffae740 10 client.16866 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:16:54.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.765+0000 7faeeffae740 20 client.16866 populate_metadata read hostname 'smithi016' 2024-03-28T22:16:54.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.765+0000 7faeeffae740 1 --2- 172.21.15.16:0/2821318970 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x562c3f7d81b0 0x562c3f7da600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:54.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.765+0000 7faeeffae740 4 client.16866 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:16:54.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.765+0000 7faeeffae740 1 -- 172.21.15.16:0/2821318970 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562c3f72b3e0 con 0x562c3f7d81b0 2024-03-28T22:16:54.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.765+0000 7faee6ffd640 1 --2- 172.21.15.16:0/2821318970 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x562c3f7d81b0 0x562c3f7da600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:54.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.766+0000 7faece7fc640 10 client.16866 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:16:54.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.766+0000 7faee6ffd640 1 --2- 172.21.15.16:0/2821318970 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x562c3f7d81b0 0x562c3f7da600 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef8daa8c3d3cb853 server_cookie=d566fa39e1d36e27 in_seq=0 out_seq=0 2024-03-28T22:16:54.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.766+0000 7faece7fc640 1 -- 172.21.15.16:0/2821318970 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562c3f72b3e0 con 0x562c3f7d81b0 2024-03-28T22:16:54.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.766+0000 7faece7fc640 10 client.16866 handle_command_reply: tid=0 2024-03-28T22:16:54.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.766+0000 7faeeffae740 1 client.16866 shutdown 2024-03-28T22:16:54.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.766+0000 7faeeffae740 1 -- 172.21.15.16:0/2821318970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562c3f7bb040 msgr2=0x562c3f7c5860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:54.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.766+0000 7faeeffae740 1 --2- 172.21.15.16:0/2821318970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562c3f7bb040 0x562c3f7c5860 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7faed00cba10 tx=0x7faed009a670 comp rx=0 tx=0).stop 2024-03-28T22:16:54.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.766+0000 7faeeffae740 1 -- 172.21.15.16:0/2821318970 shutdown_connections 2024-03-28T22:16:54.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.766+0000 7faeeffae740 1 --2- 172.21.15.16:0/2821318970 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x562c3f7d81b0 0x562c3f7da600 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.766+0000 7faeeffae740 1 --2- 172.21.15.16:0/2821318970 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562c3f7c8300 0x562c3f7c5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.766+0000 7faeeffae740 1 --2- 172.21.15.16:0/2821318970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562c3f7bb040 0x562c3f7c5860 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.766+0000 7faeeffae740 1 --2- 172.21.15.16:0/2821318970 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562c3f7ba720 0x562c3f7c5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.766+0000 7faece7fc640 0 client.16866 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:16:54.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.766+0000 7faeeffae740 1 -- 172.21.15.16:0/2821318970 >> 172.21.15.16:0/2821318970 conn(0x562c3f7b5120 msgr2=0x562c3f7b6e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:54.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.767+0000 7faeeffae740 1 -- 172.21.15.16:0/2821318970 shutdown_connections 2024-03-28T22:16:54.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.767+0000 7faeeffae740 1 -- 172.21.15.16:0/2821318970 wait complete. 2024-03-28T22:16:54.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.767+0000 7faeeffae740 20 client.16866 trim_cache size 0 max 16384 2024-03-28T22:16:54.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.784+0000 7faeeffae740 1 Processor -- start 2024-03-28T22:16:54.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.784+0000 7faeeffae740 1 -- start start 2024-03-28T22:16:54.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.785+0000 7faeeffae740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562c3f7ba720 0x562c3f7c7a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:54.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.785+0000 7faeeffae740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562c3f7bb040 0x562c3f7d3d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:54.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.785+0000 7faeeffae740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562c3f7c8300 0x562c3f7b6c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:54.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.785+0000 7faeeffae740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faee81ad8d0 con 0x562c3f7c8300 2024-03-28T22:16:54.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.785+0000 7faeeffae740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7faee81ab1a0 con 0x562c3f7bb040 2024-03-28T22:16:54.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.785+0000 7faeeffae740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7faee81a9010 con 0x562c3f7ba720 2024-03-28T22:16:54.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.785+0000 7faee77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562c3f7c8300 0x562c3f7b6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:54.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.785+0000 7faee67fc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562c3f7bb040 0x562c3f7d3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:54.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.785+0000 7faee67fc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562c3f7bb040 0x562c3f7d3d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:44444/0 (socket says 172.21.15.16:44444) 2024-03-28T22:16:54.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.785+0000 7faee67fc640 1 -- 172.21.15.16:0/4095562754 learned_addr learned my addr 172.21.15.16:0/4095562754 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:54.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.785+0000 7faee6ffd640 1 --2- 172.21.15.16:0/4095562754 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562c3f7ba720 0x562c3f7c7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:54.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.785+0000 7faee77fe640 1 -- 172.21.15.16:0/4095562754 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562c3f7ba720 msgr2=0x562c3f7c7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:54.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.785+0000 7faee77fe640 1 --2- 172.21.15.16:0/4095562754 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562c3f7ba720 0x562c3f7c7a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.785+0000 7faee77fe640 1 -- 172.21.15.16:0/4095562754 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562c3f7bb040 msgr2=0x562c3f7d3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:54.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.785+0000 7faee77fe640 1 --2- 172.21.15.16:0/4095562754 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562c3f7bb040 0x562c3f7d3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.785+0000 7faee77fe640 1 -- 172.21.15.16:0/4095562754 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c3f7bae70 con 0x562c3f7c8300 2024-03-28T22:16:54.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.785+0000 7faee6ffd640 1 --2- 172.21.15.16:0/4095562754 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562c3f7ba720 0x562c3f7c7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:16:54.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.785+0000 7faee67fc640 1 --2- 172.21.15.16:0/4095562754 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562c3f7bb040 0x562c3f7d3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:16:54.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.785+0000 7faee77fe640 1 --2- 172.21.15.16:0/4095562754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562c3f7c8300 0x562c3f7b6c10 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7faee002c250 tx=0x7faee00050d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a788a7535b229007 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:54.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.785+0000 7faedcff9640 1 -- 172.21.15.16:0/4095562754 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee0004f30 con 0x562c3f7c8300 2024-03-28T22:16:54.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.785+0000 7faedcff9640 1 -- 172.21.15.16:0/4095562754 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faee002ce10 con 0x562c3f7c8300 2024-03-28T22:16:54.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.785+0000 7faedcff9640 1 -- 172.21.15.16:0/4095562754 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee00035f0 con 0x562c3f7c8300 2024-03-28T22:16:54.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.786+0000 7faeeffae740 1 -- 172.21.15.16:0/4095562754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562c3f7c8300 msgr2=0x562c3f7b6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:54.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.786+0000 7faeeffae740 1 --2- 172.21.15.16:0/4095562754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562c3f7c8300 0x562c3f7b6c10 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7faee002c250 tx=0x7faee00050d0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.786+0000 7faeeffae740 1 -- 172.21.15.16:0/4095562754 shutdown_connections 2024-03-28T22:16:54.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.786+0000 7faeeffae740 1 --2- 172.21.15.16:0/4095562754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562c3f7c8300 0x562c3f7b6c10 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.786+0000 7faeeffae740 1 --2- 172.21.15.16:0/4095562754 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562c3f7bb040 0x562c3f7d3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.786+0000 7faeeffae740 1 --2- 172.21.15.16:0/4095562754 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562c3f7ba720 0x562c3f7c7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.786+0000 7faeeffae740 1 -- 172.21.15.16:0/4095562754 >> 172.21.15.16:0/4095562754 conn(0x562c3f7b5120 msgr2=0x562c3f7b6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:54.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.786+0000 7faeeffae740 1 -- 172.21.15.16:0/4095562754 shutdown_connections 2024-03-28T22:16:54.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.786+0000 7faeeffae740 1 -- 172.21.15.16:0/4095562754 wait complete. 2024-03-28T22:16:54.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.786+0000 7faeeffae740 1 -- start start 2024-03-28T22:16:54.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.787+0000 7faeeffae740 1 Processor -- start 2024-03-28T22:16:54.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.787+0000 7faeeffae740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562c3f7ba720 0x562c3f7c2730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:54.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.787+0000 7faeeffae740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562c3f7bb040 0x562c3f7c2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:54.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.787+0000 7faeeffae740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562c3f7c8300 0x562c3f837480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:54.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.787+0000 7faeeffae740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faee81a9660 con 0x562c3f7c8300 2024-03-28T22:16:54.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.787+0000 7faeeffae740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7faee81aaad0 con 0x562c3f7bb040 2024-03-28T22:16:54.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.787+0000 7faeeffae740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7faee81a9d30 con 0x562c3f7ba720 2024-03-28T22:16:54.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.787+0000 7faeeffae740 10 client.0 _ll_register_callbacks cb 0x7faed009aa90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:16:54.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.787+0000 7faee77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562c3f7c8300 0x562c3f837480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:54.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.787+0000 7faee77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562c3f7c8300 0x562c3f837480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60450/0 (socket says 172.21.15.16:60450) 2024-03-28T22:16:54.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.787+0000 7faee77fe640 1 -- 172.21.15.16:0/1562666085 learned_addr learned my addr 172.21.15.16:0/1562666085 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:54.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.787+0000 7faee67fc640 1 --2- 172.21.15.16:0/1562666085 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562c3f7bb040 0x562c3f7c2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:54.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.787+0000 7faee6ffd640 1 --2- 172.21.15.16:0/1562666085 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562c3f7ba720 0x562c3f7c2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:54.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.787+0000 7faee77fe640 1 -- 172.21.15.16:0/1562666085 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562c3f7ba720 msgr2=0x562c3f7c2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:54.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.787+0000 7faee77fe640 1 --2- 172.21.15.16:0/1562666085 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562c3f7ba720 0x562c3f7c2730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.787+0000 7faee77fe640 1 -- 172.21.15.16:0/1562666085 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562c3f7bb040 msgr2=0x562c3f7c2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:54.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.787+0000 7faee77fe640 1 --2- 172.21.15.16:0/1562666085 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562c3f7bb040 0x562c3f7c2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.787+0000 7faee77fe640 1 -- 172.21.15.16:0/1562666085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c3f837a70 con 0x562c3f7c8300 2024-03-28T22:16:54.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.787+0000 7faee67fc640 1 --2- 172.21.15.16:0/1562666085 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562c3f7bb040 0x562c3f7c2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:16:54.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.787+0000 7faee77fe640 1 --2- 172.21.15.16:0/1562666085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562c3f7c8300 0x562c3f837480 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7faee002bd50 tx=0x7faee0004c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c80b69702a44280f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:54.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.787+0000 7faece7fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:16:54.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.788+0000 7faece7fc640 1 -- 172.21.15.16:0/1562666085 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee002cb30 con 0x562c3f7c8300 2024-03-28T22:16:54.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.788+0000 7faece7fc640 1 -- 172.21.15.16:0/1562666085 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7faee00040f0 con 0x562c3f7c8300 2024-03-28T22:16:54.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.788+0000 7faeeffae740 1 -- 172.21.15.16:0/1562666085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faee002ce10 con 0x562c3f7c8300 2024-03-28T22:16:54.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.788+0000 7faece7fc640 1 -- 172.21.15.16:0/1562666085 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7faee00042f0 con 0x562c3f7c8300 2024-03-28T22:16:54.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.788+0000 7faeeffae740 10 client.16872 fetch_fsmap learned FSMap version 27 2024-03-28T22:16:54.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.788+0000 7faeeffae740 1 -- 172.21.15.16:0/1562666085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562c3f83ee30 con 0x562c3f7c8300 2024-03-28T22:16:54.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.788+0000 7faece7fc640 1 -- 172.21.15.16:0/1562666085 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faee002ce10 con 0x562c3f7c8300 2024-03-28T22:16:54.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.788+0000 7faece7fc640 1 -- 172.21.15.16:0/1562666085 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7faee0043480 con 0x562c3f7c8300 2024-03-28T22:16:54.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.788+0000 7faeeffae740 10 client.16872 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:16:54.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.788+0000 7faeeffae740 10 client.16872 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:16:54.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.788+0000 7faeeffae740 20 client.16872 populate_metadata read hostname 'smithi016' 2024-03-28T22:16:54.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.788+0000 7faeeffae740 1 --2- 172.21.15.16:0/1562666085 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x562c3f83f0b0 0x562c3f8414a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:54.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.788+0000 7faeeffae740 4 client.16872 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:16:54.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.788+0000 7faeeffae740 1 -- 172.21.15.16:0/1562666085 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x562c3f725730 con 0x562c3f83f0b0 2024-03-28T22:16:54.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.788+0000 7faee6ffd640 1 --2- 172.21.15.16:0/1562666085 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x562c3f83f0b0 0x562c3f8414a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:54.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.789+0000 7faece7fc640 10 client.16872 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:16:54.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.789+0000 7faee6ffd640 1 --2- 172.21.15.16:0/1562666085 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x562c3f83f0b0 0x562c3f8414a0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8be53de33ef167a8 server_cookie=ceeeb54777c761eb in_seq=0 out_seq=0 2024-03-28T22:16:54.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.790+0000 7faece7fc640 1 -- 172.21.15.16:0/1562666085 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x562c3f725730 con 0x562c3f83f0b0 2024-03-28T22:16:54.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.790+0000 7faece7fc640 10 client.16872 handle_command_reply: tid=0 2024-03-28T22:16:54.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.790+0000 7faeeffae740 1 client.16872 shutdown 2024-03-28T22:16:54.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.790+0000 7faeeffae740 1 -- 172.21.15.16:0/1562666085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562c3f7c8300 msgr2=0x562c3f837480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:54.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.790+0000 7faeeffae740 1 --2- 172.21.15.16:0/1562666085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562c3f7c8300 0x562c3f837480 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7faee002bd50 tx=0x7faee0004c20 comp rx=0 tx=0).stop 2024-03-28T22:16:54.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.790+0000 7faeeffae740 1 -- 172.21.15.16:0/1562666085 shutdown_connections 2024-03-28T22:16:54.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.790+0000 7faeeffae740 1 --2- 172.21.15.16:0/1562666085 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x562c3f83f0b0 0x562c3f8414a0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.790+0000 7faeeffae740 1 --2- 172.21.15.16:0/1562666085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562c3f7c8300 0x562c3f837480 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.790+0000 7faeeffae740 1 --2- 172.21.15.16:0/1562666085 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562c3f7bb040 0x562c3f7c2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.790+0000 7faeeffae740 1 --2- 172.21.15.16:0/1562666085 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562c3f7ba720 0x562c3f7c2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.790+0000 7faece7fc640 0 client.16872 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:16:54.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.790+0000 7faeeffae740 1 -- 172.21.15.16:0/1562666085 >> 172.21.15.16:0/1562666085 conn(0x562c3f7b5120 msgr2=0x562c3f7b4180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:54.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.790+0000 7faeeffae740 1 -- 172.21.15.16:0/1562666085 shutdown_connections 2024-03-28T22:16:54.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.790+0000 7faeeffae740 1 -- 172.21.15.16:0/1562666085 wait complete. 2024-03-28T22:16:54.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.791+0000 7faeeffae740 20 client.16872 trim_cache size 0 max 16384 2024-03-28T22:16:54.803 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:16:54.803 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-03-28T22:16:54.804 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "d5f1e4fb-ec2c-4b67-9be9-342dc88f51e7", 2024-03-28T22:16:54.804 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-03-28T22:16:54.804 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:16:54.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.791+0000 7faeed2ed640 1 -- 172.21.15.16:0/3626213089 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7faed40a1ef0 msgr2=0x7faed40a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:54.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.791+0000 7faeed2ed640 1 --2- 172.21.15.16:0/3626213089 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7faed40a1ef0 0x7faed40a43b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7faed8005fd0 tx=0x7faed8009500 comp rx=0 tx=0).stop 2024-03-28T22:16:54.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.791+0000 7faeed2ed640 1 -- 172.21.15.16:0/3626213089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faee81031f0 msgr2=0x7faee8198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:54.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.791+0000 7faeed2ed640 1 --2- 172.21.15.16:0/3626213089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faee81031f0 0x7faee8198520 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7faed000c920 tx=0x7faed000cde0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.791+0000 7faeed2ed640 1 -- 172.21.15.16:0/3626213089 shutdown_connections 2024-03-28T22:16:54.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.791+0000 7faeed2ed640 1 --2- 172.21.15.16:0/3626213089 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7faed40a1ef0 0x7faed40a43b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.791+0000 7faeed2ed640 1 --2- 172.21.15.16:0/3626213089 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faee8103b90 0x7faee819c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.791+0000 7faeed2ed640 1 --2- 172.21.15.16:0/3626213089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faee81031f0 0x7faee8198520 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.791+0000 7faeed2ed640 1 --2- 172.21.15.16:0/3626213089 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faee8101ab0 0x7faee8197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.791+0000 7faeed2ed640 1 -- 172.21.15.16:0/3626213089 >> 172.21.15.16:0/3626213089 conn(0x7faee80fd1e0 msgr2=0x7faee8107590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:54.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.791+0000 7faeed2ed640 1 -- 172.21.15.16:0/3626213089 shutdown_connections 2024-03-28T22:16:54.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.791+0000 7faeed2ed640 1 -- 172.21.15.16:0/3626213089 wait complete. 2024-03-28T22:16:54.807 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:16:54.807 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:16:54.808 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:54 smithi053 ceph-mon[35979]: pgmap v192: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:16:54.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.952+0000 7f5a8d21c640 1 -- 172.21.15.16:0/3874484541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a88106690 msgr2=0x7f5a88106a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:54.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.952+0000 7f5a8d21c640 1 --2- 172.21.15.16:0/3874484541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a88106690 0x7f5a88106a70 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f5a78009890 tx=0x7f5a7802bb70 comp rx=0 tx=0).stop 2024-03-28T22:16:54.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.952+0000 7f5a8d21c640 1 -- 172.21.15.16:0/3874484541 shutdown_connections 2024-03-28T22:16:54.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.952+0000 7f5a8d21c640 1 --2- 172.21.15.16:0/3874484541 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a88100fb0 0x7f5a8810d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.952+0000 7f5a8d21c640 1 --2- 172.21.15.16:0/3874484541 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a881005f0 0x7f5a88100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.952+0000 7f5a8d21c640 1 --2- 172.21.15.16:0/3874484541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a88106690 0x7f5a88106a70 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.952+0000 7f5a8d21c640 1 -- 172.21.15.16:0/3874484541 >> 172.21.15.16:0/3874484541 conn(0x7f5a88077800 msgr2=0x7f5a88077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:54.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.952+0000 7f5a8d21c640 1 -- 172.21.15.16:0/3874484541 shutdown_connections 2024-03-28T22:16:54.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.952+0000 7f5a8d21c640 1 -- 172.21.15.16:0/3874484541 wait complete. 2024-03-28T22:16:54.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.953+0000 7f5a8d21c640 1 Processor -- start 2024-03-28T22:16:54.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.953+0000 7f5a8d21c640 1 -- start start 2024-03-28T22:16:54.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.953+0000 7f5a8d21c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a881005f0 0x7f5a8819a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:54.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.953+0000 7f5a8d21c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a88100fb0 0x7f5a8819a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:54.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.953+0000 7f5a8d21c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a88106690 0x7f5a8819e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.953+0000 7f5a8d21c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5a88110a10 con 0x7f5a88106690 2024-03-28T22:16:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.953+0000 7f5a8d21c640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5a88110890 con 0x7f5a88100fb0 2024-03-28T22:16:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.953+0000 7f5a8d21c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5a88110b90 con 0x7f5a881005f0 2024-03-28T22:16:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.954+0000 7f5a8759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a88106690 0x7f5a8819e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.954+0000 7f5a86d9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a881005f0 0x7f5a8819a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.954+0000 7f5a7ffff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a88100fb0 0x7f5a8819a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.954+0000 7f5a8759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a88106690 0x7f5a8819e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60472/0 (socket says 172.21.15.16:60472) 2024-03-28T22:16:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.954+0000 7f5a8759e640 1 -- 172.21.15.16:0/1235567244 learned_addr learned my addr 172.21.15.16:0/1235567244 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.954+0000 7f5a8759e640 1 -- 172.21.15.16:0/1235567244 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a881005f0 msgr2=0x7f5a8819a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:54.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.954+0000 7f5a8759e640 1 --2- 172.21.15.16:0/1235567244 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a881005f0 0x7f5a8819a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.954+0000 7f5a8759e640 1 -- 172.21.15.16:0/1235567244 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a88100fb0 msgr2=0x7f5a8819a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:54.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.954+0000 7f5a8759e640 1 --2- 172.21.15.16:0/1235567244 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a88100fb0 0x7f5a8819a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.954+0000 7f5a8759e640 1 -- 172.21.15.16:0/1235567244 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a8819f120 con 0x7f5a88106690 2024-03-28T22:16:54.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.954+0000 7f5a8759e640 1 --2- 172.21.15.16:0/1235567244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a88106690 0x7f5a8819e9f0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f5a8000ea60 tx=0x7f5a8000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:54.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.954+0000 7f5a84d99640 1 -- 172.21.15.16:0/1235567244 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a8000cd80 con 0x7f5a88106690 2024-03-28T22:16:54.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.954+0000 7f5a84d99640 1 -- 172.21.15.16:0/1235567244 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a80005020 con 0x7f5a88106690 2024-03-28T22:16:54.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.954+0000 7f5a8d21c640 1 -- 172.21.15.16:0/1235567244 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a8819f410 con 0x7f5a88106690 2024-03-28T22:16:54.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.954+0000 7f5a84d99640 1 -- 172.21.15.16:0/1235567244 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a80010690 con 0x7f5a88106690 2024-03-28T22:16:54.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.954+0000 7f5a8d21c640 1 -- 172.21.15.16:0/1235567244 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a881a6ca0 con 0x7f5a88106690 2024-03-28T22:16:54.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.957+0000 7f5a84d99640 1 -- 172.21.15.16:0/1235567244 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f5a800051d0 con 0x7f5a88106690 2024-03-28T22:16:54.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.957+0000 7f5a8fe01740 1 Processor -- start 2024-03-28T22:16:54.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.957+0000 7f5a8fe01740 1 -- start start 2024-03-28T22:16:54.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.958+0000 7f5a8fe01740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55759f17f290 0x55759f17f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:54.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.958+0000 7f5a8fe01740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55759f17fbb0 0x55759f188250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:54.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.958+0000 7f5a8fe01740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55759f188790 0x55759f18ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:54.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.958+0000 7f5a8fe01740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55759f0e3180 con 0x55759f17fbb0 2024-03-28T22:16:54.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.958+0000 7f5a8fe01740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55759f08bb70 con 0x55759f17f290 2024-03-28T22:16:54.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.958+0000 7f5a8fe01740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55759f0df220 con 0x55759f188790 2024-03-28T22:16:54.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.958+0000 7f5a86d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55759f17fbb0 0x55759f188250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:54.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.958+0000 7f5a86d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55759f17fbb0 0x55759f188250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60476/0 (socket says 172.21.15.16:60476) 2024-03-28T22:16:54.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.958+0000 7f5a86d9d640 1 -- 172.21.15.16:0/4231847481 learned_addr learned my addr 172.21.15.16:0/4231847481 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:54.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.958+0000 7f5a7ffff640 1 --2- 172.21.15.16:0/4231847481 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55759f17f290 0x55759f17f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:54.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.958+0000 7f5a7ffff640 1 --2- 172.21.15.16:0/4231847481 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55759f188790 0x55759f18ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:54.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.958+0000 7f5a84d99640 1 --2- 172.21.15.16:0/1235567244 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f5a740a1ef0 0x7f5a740a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:54.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.958+0000 7f5a86d9d640 1 -- 172.21.15.16:0/4231847481 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55759f188790 msgr2=0x55759f18ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:54.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.958+0000 7f5a86d9d640 1 --2- 172.21.15.16:0/4231847481 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55759f188790 0x55759f18ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.958+0000 7f5a86d9d640 1 -- 172.21.15.16:0/4231847481 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55759f17f290 msgr2=0x55759f17f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:54.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.958+0000 7f5a86d9d640 1 --2- 172.21.15.16:0/4231847481 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55759f17f290 0x55759f17f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.958+0000 7f5a86d9d640 1 -- 172.21.15.16:0/4231847481 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55759f09f7c0 con 0x55759f17fbb0 2024-03-28T22:16:54.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.958+0000 7f5a8759e640 1 --2- 172.21.15.16:0/1235567244 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f5a740a1ef0 0x7f5a740a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:54.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.958+0000 7f5a84d99640 1 -- 172.21.15.16:0/1235567244 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f5a800ce7d0 con 0x7f5a88106690 2024-03-28T22:16:54.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.958+0000 7f5a86d9d640 1 --2- 172.21.15.16:0/4231847481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55759f17fbb0 0x55759f188250 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f5a78009890 tx=0x7f5a78005b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfe9842638235ccc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:54.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.959+0000 7f5a7d7fa640 1 -- 172.21.15.16:0/4231847481 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a78039040 con 0x55759f17fbb0 2024-03-28T22:16:54.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.959+0000 7f5a7d7fa640 1 -- 172.21.15.16:0/4231847481 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a780323c0 con 0x55759f17fbb0 2024-03-28T22:16:54.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.959+0000 7f5a7d7fa640 1 -- 172.21.15.16:0/4231847481 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a7802c510 con 0x55759f17fbb0 2024-03-28T22:16:54.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.959+0000 7f5a8fe01740 1 -- 172.21.15.16:0/4231847481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55759f17fbb0 msgr2=0x55759f188250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:54.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.959+0000 7f5a8759e640 1 --2- 172.21.15.16:0/1235567244 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f5a740a1ef0 0x7f5a740a43b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5a800cbff0 tx=0x7f5a8006e250 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:54.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.959+0000 7f5a8fe01740 1 --2- 172.21.15.16:0/4231847481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55759f17fbb0 0x55759f188250 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f5a78009890 tx=0x7f5a78005b20 comp rx=0 tx=0).stop 2024-03-28T22:16:54.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.960+0000 7f5a8fe01740 1 -- 172.21.15.16:0/4231847481 shutdown_connections 2024-03-28T22:16:54.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.960+0000 7f5a8fe01740 1 --2- 172.21.15.16:0/4231847481 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55759f188790 0x55759f18ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.960+0000 7f5a8fe01740 1 --2- 172.21.15.16:0/4231847481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55759f17fbb0 0x55759f188250 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.960+0000 7f5a8fe01740 1 --2- 172.21.15.16:0/4231847481 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55759f17f290 0x55759f17f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.960+0000 7f5a8fe01740 1 -- 172.21.15.16:0/4231847481 >> 172.21.15.16:0/4231847481 conn(0x55759f1799c0 msgr2=0x55759f179da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:54.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.960+0000 7f5a8fe01740 1 -- 172.21.15.16:0/4231847481 shutdown_connections 2024-03-28T22:16:54.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.960+0000 7f5a8fe01740 1 -- 172.21.15.16:0/4231847481 wait complete. 2024-03-28T22:16:54.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.960+0000 7f5a8fe01740 1 -- start start 2024-03-28T22:16:54.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.960+0000 7f5a8fe01740 1 Processor -- start 2024-03-28T22:16:54.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.960+0000 7f5a8fe01740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55759f17f290 0x55759f185790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:54.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.960+0000 7f5a8fe01740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55759f17fbb0 0x55759f185cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:54.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.960+0000 7f5a8fe01740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55759f188790 0x55759f186210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:54.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.961+0000 7f5a8fe01740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55759f18b520 con 0x55759f17f290 2024-03-28T22:16:54.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.961+0000 7f5a8fe01740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55759f0814a0 con 0x55759f17fbb0 2024-03-28T22:16:54.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.961+0000 7f5a8fe01740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55759f18b6a0 con 0x55759f188790 2024-03-28T22:16:54.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.961+0000 7f5a8fe01740 10 client.0 _ll_register_callbacks cb 0x55759f084d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:16:54.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.961+0000 7f5a86d9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55759f17fbb0 0x55759f185cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:54.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.961+0000 7f5a86d9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55759f17fbb0 0x55759f185cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:44498/0 (socket says 172.21.15.16:44498) 2024-03-28T22:16:54.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.961+0000 7f5a86d9d640 1 -- 172.21.15.16:0/106485583 learned_addr learned my addr 172.21.15.16:0/106485583 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:54.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.961+0000 7f5a7ffff640 1 --2- 172.21.15.16:0/106485583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55759f17f290 0x55759f185790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:54.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.961+0000 7f5a7ffff640 1 --2- 172.21.15.16:0/106485583 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55759f188790 0x55759f186210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:54.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.961+0000 7f5a86d9d640 1 -- 172.21.15.16:0/106485583 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55759f188790 msgr2=0x55759f186210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:54.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.961+0000 7f5a86d9d640 1 --2- 172.21.15.16:0/106485583 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55759f188790 0x55759f186210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.961+0000 7f5a86d9d640 1 -- 172.21.15.16:0/106485583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55759f17f290 msgr2=0x55759f185790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:54.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.961+0000 7f5a86d9d640 1 --2- 172.21.15.16:0/106485583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55759f17f290 0x55759f185790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.961+0000 7f5a86d9d640 1 -- 172.21.15.16:0/106485583 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55759eec17d0 con 0x55759f17fbb0 2024-03-28T22:16:54.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.961+0000 7f5a7ffff640 1 --2- 172.21.15.16:0/106485583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55759f17f290 0x55759f185790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:16:54.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.961+0000 7f5a86d9d640 1 --2- 172.21.15.16:0/106485583 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55759f17fbb0 0x55759f185cd0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5a7802bbb0 tx=0x7f5a78034ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f41b8b60a85de81 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:54.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.961+0000 7f5a6e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-03-28T22:16:54.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.962+0000 7f5a6e7fc640 1 -- 172.21.15.16:0/106485583 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a78039040 con 0x55759f17fbb0 2024-03-28T22:16:54.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.962+0000 7f5a8fe01740 1 -- 172.21.15.16:0/106485583 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a780323c0 con 0x55759f17fbb0 2024-03-28T22:16:54.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.962+0000 7f5a6e7fc640 1 -- 172.21.15.16:0/106485583 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f5a78034650 con 0x55759f17fbb0 2024-03-28T22:16:54.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.962+0000 7f5a6e7fc640 1 -- 172.21.15.16:0/106485583 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5a78034850 con 0x55759f17fbb0 2024-03-28T22:16:54.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.962+0000 7f5a8fe01740 10 client.26786 fetch_fsmap learned FSMap version 27 2024-03-28T22:16:54.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.962+0000 7f5a8fe01740 1 -- 172.21.15.16:0/106485583 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55759edf73f0 con 0x55759f17fbb0 2024-03-28T22:16:54.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.962+0000 7f5a6e7fc640 1 -- 172.21.15.16:0/106485583 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a780323c0 con 0x55759f17fbb0 2024-03-28T22:16:54.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.962+0000 7f5a6e7fc640 1 -- 172.21.15.16:0/106485583 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f5a78057020 con 0x55759f17fbb0 2024-03-28T22:16:54.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.962+0000 7f5a8fe01740 10 client.26786 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:16:54.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.962+0000 7f5a8fe01740 10 client.26786 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:16:54.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.962+0000 7f5a8fe01740 20 client.26786 populate_metadata read hostname 'smithi016' 2024-03-28T22:16:54.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.962+0000 7f5a8fe01740 1 --2- 172.21.15.16:0/106485583 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55759f198930 0x55759f19ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:54.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.962+0000 7f5a8fe01740 4 client.26786 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:16:54.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.963+0000 7f5a8fe01740 1 -- 172.21.15.16:0/106485583 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55759f0f06c0 con 0x55759f198930 2024-03-28T22:16:54.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.963+0000 7f5a7ffff640 1 --2- 172.21.15.16:0/106485583 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55759f198930 0x55759f19ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:54.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.963+0000 7f5a6e7fc640 10 client.26786 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:16:54.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.963+0000 7f5a7ffff640 1 --2- 172.21.15.16:0/106485583 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55759f198930 0x55759f19ad20 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f775de38d8de65f server_cookie=70328c864c872db6 in_seq=0 out_seq=0 2024-03-28T22:16:54.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.964+0000 7f5a6e7fc640 1 -- 172.21.15.16:0/106485583 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55759f0f06c0 con 0x55759f198930 2024-03-28T22:16:54.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.964+0000 7f5a6e7fc640 10 client.26786 handle_command_reply: tid=0 2024-03-28T22:16:54.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.964+0000 7f5a8fe01740 1 client.26786 shutdown 2024-03-28T22:16:54.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.964+0000 7f5a8fe01740 1 -- 172.21.15.16:0/106485583 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55759f17fbb0 msgr2=0x55759f185cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:54.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.964+0000 7f5a8fe01740 1 --2- 172.21.15.16:0/106485583 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55759f17fbb0 0x55759f185cd0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5a7802bbb0 tx=0x7f5a78034ec0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.964+0000 7f5a8fe01740 1 -- 172.21.15.16:0/106485583 shutdown_connections 2024-03-28T22:16:54.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.964+0000 7f5a8fe01740 1 --2- 172.21.15.16:0/106485583 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55759f198930 0x55759f19ad20 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.964+0000 7f5a8fe01740 1 --2- 172.21.15.16:0/106485583 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55759f188790 0x55759f186210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.964+0000 7f5a6e7fc640 0 client.26786 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:16:54.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.964+0000 7f5a8fe01740 1 --2- 172.21.15.16:0/106485583 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55759f17fbb0 0x55759f185cd0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.964+0000 7f5a8fe01740 1 --2- 172.21.15.16:0/106485583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55759f17f290 0x55759f185790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.964+0000 7f5a8fe01740 1 -- 172.21.15.16:0/106485583 >> 172.21.15.16:0/106485583 conn(0x55759f1799c0 msgr2=0x55759f17b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:54.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.964+0000 7f5a8fe01740 1 -- 172.21.15.16:0/106485583 shutdown_connections 2024-03-28T22:16:54.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.964+0000 7f5a8fe01740 1 -- 172.21.15.16:0/106485583 wait complete. 2024-03-28T22:16:54.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.964+0000 7f5a8fe01740 20 client.26786 trim_cache size 0 max 16384 2024-03-28T22:16:54.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.984+0000 7f5a8fe01740 1 Processor -- start 2024-03-28T22:16:54.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.984+0000 7f5a8fe01740 1 -- start start 2024-03-28T22:16:54.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.984+0000 7f5a8fe01740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55759f1799c0 0x55759f187ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:54.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.984+0000 7f5a8fe01740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55759f17f290 0x55759f19dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:54.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.984+0000 7f5a8fe01740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55759f17fbb0 0x55759f1829f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:54.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.984+0000 7f5a8fe01740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5a881b1e10 con 0x55759f17fbb0 2024-03-28T22:16:54.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.985+0000 7f5a8fe01740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5a881af6e0 con 0x55759f1799c0 2024-03-28T22:16:54.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.985+0000 7f5a8fe01740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5a881ad530 con 0x55759f17f290 2024-03-28T22:16:54.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.985+0000 7f5a86d9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55759f17f290 0x55759f19dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:54.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.985+0000 7f5a86d9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55759f17f290 0x55759f19dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.16:51204/0 (socket says 172.21.15.16:51204) 2024-03-28T22:16:54.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.985+0000 7f5a86d9d640 1 -- 172.21.15.16:0/3331739024 learned_addr learned my addr 172.21.15.16:0/3331739024 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:54.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.985+0000 7f5a7ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55759f17fbb0 0x55759f1829f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:54.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.985+0000 7f5a7ffff640 1 --2- 172.21.15.16:0/3331739024 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55759f1799c0 0x55759f187ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:54.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.985+0000 7f5a86d9d640 1 -- 172.21.15.16:0/3331739024 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55759f1799c0 msgr2=0x55759f187ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:54.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.985+0000 7f5a86d9d640 1 --2- 172.21.15.16:0/3331739024 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55759f1799c0 0x55759f187ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.985+0000 7f5a86d9d640 1 -- 172.21.15.16:0/3331739024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55759f17fbb0 msgr2=0x55759f1829f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:54.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.985+0000 7f5a86d9d640 1 --2- 172.21.15.16:0/3331739024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55759f17fbb0 0x55759f1829f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.985+0000 7f5a86d9d640 1 -- 172.21.15.16:0/3331739024 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55759f17f9e0 con 0x55759f17f290 2024-03-28T22:16:54.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.985+0000 7f5a7ffff640 1 --2- 172.21.15.16:0/3331739024 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55759f1799c0 0x55759f187ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:16:54.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.985+0000 7f5a86d9d640 1 --2- 172.21.15.16:0/3331739024 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55759f17f290 0x55759f19dd40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5a78000c00 tx=0x7f5a78009590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3519b86f02d0d6da server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:54.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.985+0000 7f5a7cff9640 1 -- 172.21.15.16:0/3331739024 <== mon.1 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a780323c0 con 0x55759f17f290 2024-03-28T22:16:54.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.986+0000 7f5a7cff9640 1 -- 172.21.15.16:0/3331739024 <== mon.1 v2:172.21.15.63:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a7805e020 con 0x55759f17f290 2024-03-28T22:16:54.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.986+0000 7f5a7cff9640 1 -- 172.21.15.16:0/3331739024 <== mon.1 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a7802c9f0 con 0x55759f17f290 2024-03-28T22:16:54.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.986+0000 7f5a8fe01740 1 -- 172.21.15.16:0/3331739024 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55759f17f290 msgr2=0x55759f19dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:54.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.986+0000 7f5a8fe01740 1 --2- 172.21.15.16:0/3331739024 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55759f17f290 0x55759f19dd40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5a78000c00 tx=0x7f5a78009590 comp rx=0 tx=0).stop 2024-03-28T22:16:54.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.986+0000 7f5a8fe01740 1 -- 172.21.15.16:0/3331739024 shutdown_connections 2024-03-28T22:16:54.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.986+0000 7f5a8fe01740 1 --2- 172.21.15.16:0/3331739024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55759f17fbb0 0x55759f1829f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.986+0000 7f5a8fe01740 1 --2- 172.21.15.16:0/3331739024 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55759f17f290 0x55759f19dd40 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.986+0000 7f5a8fe01740 1 --2- 172.21.15.16:0/3331739024 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55759f1799c0 0x55759f187ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.986+0000 7f5a8fe01740 1 -- 172.21.15.16:0/3331739024 >> 172.21.15.16:0/3331739024 conn(0x55759f1843d0 msgr2=0x55759f178c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:54.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.986+0000 7f5a8fe01740 1 -- 172.21.15.16:0/3331739024 shutdown_connections 2024-03-28T22:16:54.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.986+0000 7f5a8fe01740 1 -- 172.21.15.16:0/3331739024 wait complete. 2024-03-28T22:16:54.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.987+0000 7f5a8fe01740 1 -- start start 2024-03-28T22:16:54.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.987+0000 7f5a8fe01740 1 Processor -- start 2024-03-28T22:16:54.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.987+0000 7f5a8fe01740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55759f1799c0 0x55759f19dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:54.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.987+0000 7f5a8fe01740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55759f17f290 0x55759f19e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:54.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.987+0000 7f5a8fe01740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55759f17fbb0 0x55759f19c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:54.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.987+0000 7f5a8fe01740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5a881adba0 con 0x55759f17fbb0 2024-03-28T22:16:54.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.987+0000 7f5a8fe01740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5a881af010 con 0x55759f17f290 2024-03-28T22:16:54.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.987+0000 7f5a8fe01740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5a881ae270 con 0x55759f1799c0 2024-03-28T22:16:54.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.987+0000 7f5a7ffff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55759f1799c0 0x55759f19dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:54.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.987+0000 7f5a8fe01740 10 client.0 _ll_register_callbacks cb 0x7f5a70009a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:16:54.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.987+0000 7f5a7ffff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55759f1799c0 0x55759f19dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.16:51206/0 (socket says 172.21.15.16:51206) 2024-03-28T22:16:54.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.987+0000 7f5a7ffff640 1 -- 172.21.15.16:0/2931756946 learned_addr learned my addr 172.21.15.16:0/2931756946 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:16:54.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.987+0000 7f5a86d9d640 1 --2- 172.21.15.16:0/2931756946 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55759f17f290 0x55759f19e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:54.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.987+0000 7f5a7ffff640 1 --2- 172.21.15.16:0/2931756946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55759f17fbb0 0x55759f19c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:54.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.988+0000 7f5a7ffff640 1 -- 172.21.15.16:0/2931756946 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55759f17f290 msgr2=0x55759f19e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:54.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.988+0000 7f5a7ffff640 1 --2- 172.21.15.16:0/2931756946 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55759f17f290 0x55759f19e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.988+0000 7f5a7ffff640 1 -- 172.21.15.16:0/2931756946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55759f17fbb0 msgr2=0x55759f19c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:16:54.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.988+0000 7f5a7ffff640 1 --2- 172.21.15.16:0/2931756946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55759f17fbb0 0x55759f19c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:54.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.988+0000 7f5a7ffff640 1 -- 172.21.15.16:0/2931756946 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55759edb70f0 con 0x55759f1799c0 2024-03-28T22:16:54.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.988+0000 7f5a7ffff640 1 --2- 172.21.15.16:0/2931756946 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55759f1799c0 0x55759f19dc70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5a70004650 tx=0x7f5a70009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=daa6bbc4064b036d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:16:54.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.988+0000 7f5a6e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-03-28T22:16:54.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.988+0000 7f5a6e7fc640 1 -- 172.21.15.16:0/2931756946 <== mon.1 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a70013ea0 con 0x55759f1799c0 2024-03-28T22:16:54.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.988+0000 7f5a8fe01740 1 -- 172.21.15.16:0/2931756946 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a7805e020 con 0x55759f1799c0 2024-03-28T22:16:54.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.988+0000 7f5a6e7fc640 1 -- 172.21.15.16:0/2931756946 <== mon.1 v2:172.21.15.63:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8483+0+0 (secure 0 0 0) 0x7f5a7001a040 con 0x55759f1799c0 2024-03-28T22:16:55.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.988+0000 7f5a6e7fc640 1 -- 172.21.15.16:0/2931756946 <== mon.1 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5a70005020 con 0x55759f1799c0 2024-03-28T22:16:55.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.988+0000 7f5a8fe01740 10 client.26140 fetch_fsmap learned FSMap version 27 2024-03-28T22:16:55.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.988+0000 7f5a8fe01740 1 -- 172.21.15.16:0/2931756946 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55759f1ff460 con 0x55759f1799c0 2024-03-28T22:16:55.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.989+0000 7f5a6e7fc640 1 -- 172.21.15.16:0/2931756946 <== mon.1 v2:172.21.15.63:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a7805e020 con 0x55759f1799c0 2024-03-28T22:16:55.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.989+0000 7f5a6e7fc640 1 -- 172.21.15.16:0/2931756946 <== mon.1 v2:172.21.15.63:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f5a70039020 con 0x55759f1799c0 2024-03-28T22:16:55.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.989+0000 7f5a8fe01740 10 client.26140 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:16:55.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.989+0000 7f5a8fe01740 10 client.26140 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:16:55.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.989+0000 7f5a8fe01740 20 client.26140 populate_metadata read hostname 'smithi016' 2024-03-28T22:16:55.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.989+0000 7f5a8fe01740 1 --2- 172.21.15.16:0/2931756946 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55759f188790 0x55759f201b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:55.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.989+0000 7f5a8fe01740 4 client.26140 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:16:55.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.989+0000 7f5a8fe01740 1 -- 172.21.15.16:0/2931756946 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55759f0eaa10 con 0x55759f188790 2024-03-28T22:16:55.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.989+0000 7f5a86d9d640 1 --2- 172.21.15.16:0/2931756946 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55759f188790 0x55759f201b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:16:55.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.990+0000 7f5a86d9d640 1 --2- 172.21.15.16:0/2931756946 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55759f188790 0x55759f201b80 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=167efcf97c2c393 server_cookie=fbfb1547d27bd36b in_seq=0 out_seq=0 2024-03-28T22:16:55.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.990+0000 7f5a6e7fc640 10 client.26140 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:16:55.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.990+0000 7f5a6e7fc640 1 -- 172.21.15.16:0/2931756946 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55759f0eaa10 con 0x55759f188790 2024-03-28T22:16:55.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.990+0000 7f5a6e7fc640 10 client.26140 handle_command_reply: tid=0 2024-03-28T22:16:55.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.990+0000 7f5a8fe01740 1 client.26140 shutdown 2024-03-28T22:16:55.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.990+0000 7f5a8fe01740 1 -- 172.21.15.16:0/2931756946 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55759f1799c0 msgr2=0x55759f19dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:55.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.990+0000 7f5a8fe01740 1 --2- 172.21.15.16:0/2931756946 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55759f1799c0 0x55759f19dc70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5a70004650 tx=0x7f5a70009260 comp rx=0 tx=0).stop 2024-03-28T22:16:55.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.990+0000 7f5a8fe01740 1 -- 172.21.15.16:0/2931756946 shutdown_connections 2024-03-28T22:16:55.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.990+0000 7f5a8fe01740 1 --2- 172.21.15.16:0/2931756946 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55759f188790 0x55759f201b80 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:55.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.990+0000 7f5a8fe01740 1 --2- 172.21.15.16:0/2931756946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55759f17fbb0 0x55759f19c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:55.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.990+0000 7f5a8fe01740 1 --2- 172.21.15.16:0/2931756946 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55759f17f290 0x55759f19e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:55.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.990+0000 7f5a8fe01740 1 --2- 172.21.15.16:0/2931756946 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55759f1799c0 0x55759f19dc70 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:55.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.990+0000 7f5a6e7fc640 0 client.26140 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:16:55.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.990+0000 7f5a8fe01740 1 -- 172.21.15.16:0/2931756946 >> 172.21.15.16:0/2931756946 conn(0x55759f1843d0 msgr2=0x55759f17b630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:55.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.990+0000 7f5a8fe01740 1 -- 172.21.15.16:0/2931756946 shutdown_connections 2024-03-28T22:16:55.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.990+0000 7f5a8fe01740 1 -- 172.21.15.16:0/2931756946 wait complete. 2024-03-28T22:16:55.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.990+0000 7f5a8fe01740 20 client.26140 trim_cache size 0 max 16384 2024-03-28T22:16:55.004 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:16:55.004 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T22:16:55.004 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-03-28T22:16:55.004 INFO:teuthology.orchestra.run.smithi016.stdout: "d5f1e4fb-ec2c-4b67-9be9-342dc88f51e7": { 2024-03-28T22:16:55.004 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-03-28T22:16:55.005 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "d5f1e4fb-ec2c-4b67-9be9-342dc88f51e7", 2024-03-28T22:16:55.005 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-03-28T22:16:55.005 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:16:55.005 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:16:55.005 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:16:55.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.991+0000 7f5a8d21c640 1 -- 172.21.15.16:0/1235567244 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f5a740a1ef0 msgr2=0x7f5a740a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:55.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.991+0000 7f5a8d21c640 1 --2- 172.21.15.16:0/1235567244 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f5a740a1ef0 0x7f5a740a43b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5a800cbff0 tx=0x7f5a8006e250 comp rx=0 tx=0).stop 2024-03-28T22:16:55.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.991+0000 7f5a8d21c640 1 -- 172.21.15.16:0/1235567244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a88106690 msgr2=0x7f5a8819e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:16:55.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.991+0000 7f5a8d21c640 1 --2- 172.21.15.16:0/1235567244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a88106690 0x7f5a8819e9f0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f5a8000ea60 tx=0x7f5a8000ee60 comp rx=0 tx=0).stop 2024-03-28T22:16:55.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.991+0000 7f5a8d21c640 1 -- 172.21.15.16:0/1235567244 shutdown_connections 2024-03-28T22:16:55.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.991+0000 7f5a8d21c640 1 --2- 172.21.15.16:0/1235567244 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f5a740a1ef0 0x7f5a740a43b0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:55.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.991+0000 7f5a8d21c640 1 --2- 172.21.15.16:0/1235567244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a88106690 0x7f5a8819e9f0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:55.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.991+0000 7f5a8d21c640 1 --2- 172.21.15.16:0/1235567244 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a88100fb0 0x7f5a8819a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:55.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.991+0000 7f5a8d21c640 1 --2- 172.21.15.16:0/1235567244 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a881005f0 0x7f5a8819a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:55.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.991+0000 7f5a8d21c640 1 -- 172.21.15.16:0/1235567244 >> 172.21.15.16:0/1235567244 conn(0x7f5a88077800 msgr2=0x7f5a88069d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:16:55.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.991+0000 7f5a8d21c640 1 -- 172.21.15.16:0/1235567244 shutdown_connections 2024-03-28T22:16:55.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:16:54.991+0000 7f5a8d21c640 1 -- 172.21.15.16:0/1235567244 wait complete. 2024-03-28T22:16:55.008 INFO:tasks.cephfs.filesystem:scrub status for tag:d5f1e4fb-ec2c-4b67-9be9-342dc88f51e7 - {'path': '/', 'tag': 'd5f1e4fb-ec2c-4b67-9be9-342dc88f51e7', 'options': 'recursive,force'} 2024-03-28T22:16:55.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:55 smithi063 ceph-mon[36249]: scrub queued for path: / 2024-03-28T22:16:55.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:55 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:16:55.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:55 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:16:55.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:55 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:16:55.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:55 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:16:55.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:55 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:16:55.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:55 smithi016 ceph-mon[32374]: scrub queued for path: / 2024-03-28T22:16:55.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:55 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:16:55.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:55 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:16:55.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:55 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:16:55.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:55 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:16:55.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:55 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:16:55.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:55 smithi053 ceph-mon[35979]: scrub queued for path: / 2024-03-28T22:16:55.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:55 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:16:55.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:55 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:16:55.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:55 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:16:55.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:55 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:16:55.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:55 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:16:56.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:56 smithi063 ceph-mon[36249]: pgmap v193: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:16:56.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:56 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:16:56.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:56 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:16:56.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:56 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:16:56.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:56 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:16:56.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:56 smithi016 ceph-mon[32374]: pgmap v193: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:16:56.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:56 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:16:56.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:56 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:16:56.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:56 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:16:56.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:56 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:16:56.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:56 smithi053 ceph-mon[35979]: pgmap v193: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:16:56.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:56 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:16:56.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:56 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:16:56.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:56 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:16:56.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:56 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:16:58.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:16:58 smithi063 ceph-mon[36249]: pgmap v194: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:16:58.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:16:58 smithi016 ceph-mon[32374]: pgmap v194: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:16:58.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:16:58 smithi053 ceph-mon[35979]: pgmap v194: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:17:00.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:00 smithi063 ceph-mon[36249]: pgmap v195: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:17:00.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:00 smithi016 ceph-mon[32374]: pgmap v195: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:17:00.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:00 smithi053 ceph-mon[35979]: pgmap v195: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:17:02.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:02 smithi063 ceph-mon[36249]: pgmap v196: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-28T22:17:02.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:02 smithi016 ceph-mon[32374]: pgmap v196: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-28T22:17:02.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:02 smithi053 ceph-mon[35979]: pgmap v196: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-28T22:17:04.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:04 smithi063 ceph-mon[36249]: pgmap v197: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 18 op/s 2024-03-28T22:17:04.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:04 smithi016 ceph-mon[32374]: pgmap v197: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 18 op/s 2024-03-28T22:17:04.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:04 smithi053 ceph-mon[35979]: pgmap v197: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 18 op/s 2024-03-28T22:17:05.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:05 smithi063 ceph-mon[36249]: pgmap v198: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 18 op/s 2024-03-28T22:17:05.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:05 smithi016 ceph-mon[32374]: pgmap v198: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 18 op/s 2024-03-28T22:17:05.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:05 smithi053 ceph-mon[35979]: pgmap v198: 97 pgs: 97 active+clean; 96 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 18 op/s 2024-03-28T22:17:07.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:07 smithi053 ceph-mon[35979]: pgmap v199: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-28T22:17:07.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:07 smithi063 ceph-mon[36249]: pgmap v199: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-28T22:17:07.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:07 smithi016 ceph-mon[32374]: pgmap v199: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-28T22:17:09.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:09 smithi053 ceph-mon[35979]: pgmap v200: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-28T22:17:09.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:09 smithi063 ceph-mon[36249]: pgmap v200: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-28T22:17:09.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:09 smithi016 ceph-mon[32374]: pgmap v200: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-28T22:17:10.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:10 smithi053 ceph-mon[35979]: scrub summary: idle 2024-03-28T22:17:10.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:10 smithi063 ceph-mon[36249]: scrub summary: idle 2024-03-28T22:17:10.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:10 smithi016 ceph-mon[32374]: scrub summary: idle 2024-03-28T22:17:11.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:11 smithi053 ceph-mon[35979]: scrub summary: idle 2024-03-28T22:17:11.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:11 smithi053 ceph-mon[35979]: pgmap v201: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-28T22:17:11.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:11 smithi053 ceph-mon[35979]: osdmap e90: 12 total, 12 up, 12 in 2024-03-28T22:17:11.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:11 smithi063 ceph-mon[36249]: scrub summary: idle 2024-03-28T22:17:11.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:11 smithi063 ceph-mon[36249]: pgmap v201: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-28T22:17:11.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:11 smithi063 ceph-mon[36249]: osdmap e90: 12 total, 12 up, 12 in 2024-03-28T22:17:11.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:11 smithi016 ceph-mon[32374]: scrub summary: idle 2024-03-28T22:17:11.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:11 smithi016 ceph-mon[32374]: pgmap v201: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-28T22:17:11.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:11 smithi016 ceph-mon[32374]: osdmap e90: 12 total, 12 up, 12 in 2024-03-28T22:17:13.738 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:13 smithi016 ceph-mon[32374]: pgmap v203: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 27 KiB/s wr, 11 op/s 2024-03-28T22:17:13.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:13 smithi053 ceph-mon[35979]: pgmap v203: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 27 KiB/s wr, 11 op/s 2024-03-28T22:17:13.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:13 smithi063 ceph-mon[36249]: pgmap v203: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 27 KiB/s wr, 11 op/s 2024-03-28T22:17:15.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:15 smithi053 ceph-mon[35979]: pgmap v204: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 27 KiB/s wr, 11 op/s 2024-03-28T22:17:15.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:15 smithi063 ceph-mon[36249]: pgmap v204: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 27 KiB/s wr, 11 op/s 2024-03-28T22:17:15.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:15 smithi016 ceph-mon[32374]: pgmap v204: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 27 KiB/s wr, 11 op/s 2024-03-28T22:17:17.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:17 smithi053 ceph-mon[35979]: pgmap v205: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T22:17:17.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:17 smithi053 ceph-mon[35979]: osdmap e91: 12 total, 12 up, 12 in 2024-03-28T22:17:17.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:17 smithi063 ceph-mon[36249]: pgmap v205: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T22:17:17.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:17 smithi063 ceph-mon[36249]: osdmap e91: 12 total, 12 up, 12 in 2024-03-28T22:17:17.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:17 smithi016 ceph-mon[32374]: pgmap v205: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T22:17:17.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:17 smithi016 ceph-mon[32374]: osdmap e91: 12 total, 12 up, 12 in 2024-03-28T22:17:19.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:19 smithi053 ceph-mon[35979]: pgmap v207: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-28T22:17:19.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:19 smithi063 ceph-mon[36249]: pgmap v207: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-28T22:17:19.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:19 smithi016 ceph-mon[32374]: pgmap v207: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-28T22:17:21.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:21 smithi053 ceph-mon[35979]: pgmap v208: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 103 B/s wr, 6 op/s 2024-03-28T22:17:21.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:21 smithi063 ceph-mon[36249]: pgmap v208: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 103 B/s wr, 6 op/s 2024-03-28T22:17:21.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:21 smithi016 ceph-mon[32374]: pgmap v208: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 103 B/s wr, 6 op/s 2024-03-28T22:17:23.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:23 smithi053 ceph-mon[35979]: pgmap v209: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T22:17:23.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:23 smithi063 ceph-mon[36249]: pgmap v209: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T22:17:23.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:23 smithi016 ceph-mon[32374]: pgmap v209: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T22:17:25.010 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:17:25.010 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:17:25.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.125+0000 7f52a4623640 1 -- 172.21.15.16:0/3676407722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f529c073d00 msgr2=0x7f529c0740e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.125+0000 7f52a4623640 1 --2- 172.21.15.16:0/3676407722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f529c073d00 0x7f529c0740e0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f5298009900 tx=0x7f529802bc30 comp rx=0 tx=0).stop 2024-03-28T22:17:25.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.125+0000 7f52a4623640 1 -- 172.21.15.16:0/3676407722 shutdown_connections 2024-03-28T22:17:25.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.125+0000 7f52a4623640 1 --2- 172.21.15.16:0/3676407722 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f529c10e0e0 0x7f529c1104d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.125+0000 7f52a4623640 1 --2- 172.21.15.16:0/3676407722 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f529c074620 0x7f529c10da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.125+0000 7f52a4623640 1 --2- 172.21.15.16:0/3676407722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f529c073d00 0x7f529c0740e0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.125+0000 7f52a4623640 1 -- 172.21.15.16:0/3676407722 >> 172.21.15.16:0/3676407722 conn(0x7f529c0fd1e0 msgr2=0x7f529c0ff600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:25.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.126+0000 7f52a4623640 1 -- 172.21.15.16:0/3676407722 shutdown_connections 2024-03-28T22:17:25.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.126+0000 7f52a4623640 1 -- 172.21.15.16:0/3676407722 wait complete. 2024-03-28T22:17:25.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.126+0000 7f52a4623640 1 Processor -- start 2024-03-28T22:17:25.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.126+0000 7f52a4623640 1 -- start start 2024-03-28T22:17:25.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.126+0000 7f52a4623640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f529c073d00 0x7f529c19a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.126+0000 7f52a4623640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f529c074620 0x7f529c19a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.126+0000 7f52a4623640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f529c10e0e0 0x7f529c19ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.126+0000 7f52a4623640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f529c113de0 con 0x7f529c074620 2024-03-28T22:17:25.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.126+0000 7f52a4623640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f529c113c60 con 0x7f529c073d00 2024-03-28T22:17:25.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.126+0000 7f52a4623640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f529c113f60 con 0x7f529c10e0e0 2024-03-28T22:17:25.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.126+0000 7f52a1bbe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f529c074620 0x7f529c19a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.127+0000 7f52a1bbe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f529c074620 0x7f529c19a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45440/0 (socket says 172.21.15.16:45440) 2024-03-28T22:17:25.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.127+0000 7f52a23bf640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f529c073d00 0x7f529c19a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.127+0000 7f52a1bbe640 1 -- 172.21.15.16:0/2748128081 learned_addr learned my addr 172.21.15.16:0/2748128081 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:25.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.127+0000 7f52a2bc0640 1 --2- 172.21.15.16:0/2748128081 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f529c10e0e0 0x7f529c19ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.127+0000 7f52a23bf640 1 -- 172.21.15.16:0/2748128081 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f529c10e0e0 msgr2=0x7f529c19ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.127+0000 7f52a23bf640 1 --2- 172.21.15.16:0/2748128081 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f529c10e0e0 0x7f529c19ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.127+0000 7f52a23bf640 1 -- 172.21.15.16:0/2748128081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f529c074620 msgr2=0x7f529c19a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.127+0000 7f52a23bf640 1 --2- 172.21.15.16:0/2748128081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f529c074620 0x7f529c19a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.127+0000 7f52a23bf640 1 -- 172.21.15.16:0/2748128081 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f529c19f300 con 0x7f529c073d00 2024-03-28T22:17:25.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.127+0000 7f52a1bbe640 1 --2- 172.21.15.16:0/2748128081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f529c074620 0x7f529c19a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:17:25.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.127+0000 7f52a23bf640 1 --2- 172.21.15.16:0/2748128081 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f529c073d00 0x7f529c19a360 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5298036040 tx=0x7f529802cb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:25.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.127+0000 7f52937fe640 1 -- 172.21.15.16:0/2748128081 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5298041070 con 0x7f529c073d00 2024-03-28T22:17:25.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.127+0000 7f52a4623640 1 -- 172.21.15.16:0/2748128081 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f529c19f590 con 0x7f529c073d00 2024-03-28T22:17:25.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.127+0000 7f52937fe640 1 -- 172.21.15.16:0/2748128081 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5298004450 con 0x7f529c073d00 2024-03-28T22:17:25.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.127+0000 7f52937fe640 1 -- 172.21.15.16:0/2748128081 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5298003d90 con 0x7f529c073d00 2024-03-28T22:17:25.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.127+0000 7f52a4623640 1 -- 172.21.15.16:0/2748128081 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f529c1a6e80 con 0x7f529c073d00 2024-03-28T22:17:25.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.130+0000 7f52a7208740 1 Processor -- start 2024-03-28T22:17:25.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.130+0000 7f52937fe640 1 -- 172.21.15.16:0/2748128081 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f5298044070 con 0x7f529c073d00 2024-03-28T22:17:25.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.131+0000 7f52a7208740 1 -- start start 2024-03-28T22:17:25.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.131+0000 7f52a7208740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ed5b6e290 0x564ed5b6e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.131+0000 7f52a7208740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ed5b6ebb0 0x564ed5b77250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.131+0000 7f52a7208740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed5b77790 0x564ed5b79b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.131+0000 7f52a7208740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564ed5ad2180 con 0x564ed5b77790 2024-03-28T22:17:25.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.131+0000 7f52a7208740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x564ed5a7ab70 con 0x564ed5b6ebb0 2024-03-28T22:17:25.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.131+0000 7f52a7208740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x564ed5ace220 con 0x564ed5b6e290 2024-03-28T22:17:25.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.131+0000 7f52a1bbe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed5b77790 0x564ed5b79b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.131+0000 7f52a2bc0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ed5b6e290 0x564ed5b6e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.131+0000 7f52a23bf640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ed5b6ebb0 0x564ed5b77250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.131+0000 7f52a1bbe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed5b77790 0x564ed5b79b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45454/0 (socket says 172.21.15.16:45454) 2024-03-28T22:17:25.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.131+0000 7f52a1bbe640 1 -- 172.21.15.16:0/1476438669 learned_addr learned my addr 172.21.15.16:0/1476438669 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:25.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.131+0000 7f52937fe640 1 --2- 172.21.15.16:0/2748128081 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f52880a1ef0 0x7f52880a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.131+0000 7f52a1bbe640 1 -- 172.21.15.16:0/1476438669 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ed5b6e290 msgr2=0x564ed5b6e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:25.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.131+0000 7f52a1bbe640 1 --2- 172.21.15.16:0/1476438669 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ed5b6e290 0x564ed5b6e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.131+0000 7f52a1bbe640 1 -- 172.21.15.16:0/1476438669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ed5b6ebb0 msgr2=0x564ed5b77250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:25.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.131+0000 7f52a1bbe640 1 --2- 172.21.15.16:0/1476438669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ed5b6ebb0 0x564ed5b77250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.131+0000 7f52a1bbe640 1 -- 172.21.15.16:0/1476438669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ed5a8e7c0 con 0x564ed5b77790 2024-03-28T22:17:25.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.132+0000 7f52a2bc0640 1 --2- 172.21.15.16:0/1476438669 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ed5b6e290 0x564ed5b6e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:17:25.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.132+0000 7f52937fe640 1 -- 172.21.15.16:0/2748128081 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8527+0+0 (secure 0 0 0) 0x7f52980efa70 con 0x7f529c073d00 2024-03-28T22:17:25.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.132+0000 7f52a2bc0640 1 --2- 172.21.15.16:0/2748128081 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f52880a1ef0 0x7f52880a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.132+0000 7f52a1bbe640 1 --2- 172.21.15.16:0/1476438669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed5b77790 0x564ed5b79b80 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f528c011570 tx=0x7f528c011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f63cdd6cd2ea9739 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:25.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.132+0000 7f5290bfe640 1 -- 172.21.15.16:0/1476438669 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f528c005020 con 0x564ed5b77790 2024-03-28T22:17:25.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.132+0000 7f5290bfe640 1 -- 172.21.15.16:0/1476438669 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f528c0047a0 con 0x564ed5b77790 2024-03-28T22:17:25.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.132+0000 7f5290bfe640 1 -- 172.21.15.16:0/1476438669 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f528c00f9a0 con 0x564ed5b77790 2024-03-28T22:17:25.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.133+0000 7f52a7208740 1 -- 172.21.15.16:0/1476438669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed5b77790 msgr2=0x564ed5b79b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.133+0000 7f52a2bc0640 1 --2- 172.21.15.16:0/2748128081 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f52880a1ef0 0x7f52880a43b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5294009fd0 tx=0x7f5294004a40 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:25.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.133+0000 7f52a7208740 1 --2- 172.21.15.16:0/1476438669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed5b77790 0x564ed5b79b80 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f528c011570 tx=0x7f528c011a30 comp rx=0 tx=0).stop 2024-03-28T22:17:25.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.133+0000 7f52a7208740 1 -- 172.21.15.16:0/1476438669 shutdown_connections 2024-03-28T22:17:25.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.133+0000 7f52a7208740 1 --2- 172.21.15.16:0/1476438669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed5b77790 0x564ed5b79b80 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.133+0000 7f52a7208740 1 --2- 172.21.15.16:0/1476438669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ed5b6ebb0 0x564ed5b77250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.133+0000 7f52a7208740 1 --2- 172.21.15.16:0/1476438669 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ed5b6e290 0x564ed5b6e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.133+0000 7f52a7208740 1 -- 172.21.15.16:0/1476438669 >> 172.21.15.16:0/1476438669 conn(0x564ed5b689c0 msgr2=0x564ed5b68da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:25.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.133+0000 7f52a7208740 1 -- 172.21.15.16:0/1476438669 shutdown_connections 2024-03-28T22:17:25.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.133+0000 7f52a7208740 1 -- 172.21.15.16:0/1476438669 wait complete. 2024-03-28T22:17:25.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.133+0000 7f52a7208740 1 -- start start 2024-03-28T22:17:25.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.133+0000 7f52a7208740 1 Processor -- start 2024-03-28T22:17:25.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.133+0000 7f52a7208740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ed5b6e290 0x564ed5b74790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.133+0000 7f52a7208740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed5b6ebb0 0x564ed5b74cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.133+0000 7f52a7208740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ed5b77790 0x564ed5b75210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.133+0000 7f52a7208740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564ed5b7a580 con 0x564ed5b6ebb0 2024-03-28T22:17:25.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.133+0000 7f52a7208740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x564ed5a704a0 con 0x564ed5b6e290 2024-03-28T22:17:25.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.133+0000 7f52a7208740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x564ed5b7a700 con 0x564ed5b77790 2024-03-28T22:17:25.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.134+0000 7f52a7208740 10 client.0 _ll_register_callbacks cb 0x564ed5a73d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:17:25.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.134+0000 7f52a1bbe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed5b6ebb0 0x564ed5b74cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.134+0000 7f52a23bf640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ed5b6e290 0x564ed5b74790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.134+0000 7f52a1bbe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed5b6ebb0 0x564ed5b74cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45464/0 (socket says 172.21.15.16:45464) 2024-03-28T22:17:25.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.134+0000 7f52a1bbe640 1 -- 172.21.15.16:0/2495933205 learned_addr learned my addr 172.21.15.16:0/2495933205 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:25.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.134+0000 7f52a2bc0640 1 --2- 172.21.15.16:0/2495933205 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ed5b77790 0x564ed5b75210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.134+0000 7f52a1bbe640 1 -- 172.21.15.16:0/2495933205 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ed5b77790 msgr2=0x564ed5b75210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:25.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.134+0000 7f52a1bbe640 1 --2- 172.21.15.16:0/2495933205 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ed5b77790 0x564ed5b75210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.134+0000 7f52a1bbe640 1 -- 172.21.15.16:0/2495933205 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ed5b6e290 msgr2=0x564ed5b74790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:25.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.134+0000 7f52a1bbe640 1 --2- 172.21.15.16:0/2495933205 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ed5b6e290 0x564ed5b74790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.134+0000 7f52a1bbe640 1 -- 172.21.15.16:0/2495933205 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ed58b07d0 con 0x564ed5b6ebb0 2024-03-28T22:17:25.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.134+0000 7f52a23bf640 1 --2- 172.21.15.16:0/2495933205 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ed5b6e290 0x564ed5b74790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:17:25.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.134+0000 7f52a1bbe640 1 --2- 172.21.15.16:0/2495933205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed5b6ebb0 0x564ed5b74cd0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f528c0057c0 tx=0x7f528c0052b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0c6818e4db07346 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:25.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.134+0000 7f5285ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:17:25.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.134+0000 7f5285ffb640 1 -- 172.21.15.16:0/2495933205 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f528c028070 con 0x564ed5b6ebb0 2024-03-28T22:17:25.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.134+0000 7f5285ffb640 1 -- 172.21.15.16:0/2495933205 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8527+0+0 (secure 0 0 0) 0x7f528c004020 con 0x564ed5b6ebb0 2024-03-28T22:17:25.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.134+0000 7f52a7208740 1 -- 172.21.15.16:0/2495933205 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f528c0047a0 con 0x564ed5b6ebb0 2024-03-28T22:17:25.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.134+0000 7f5285ffb640 1 -- 172.21.15.16:0/2495933205 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f528c033050 con 0x564ed5b6ebb0 2024-03-28T22:17:25.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.134+0000 7f52a7208740 10 client.16902 fetch_fsmap learned FSMap version 27 2024-03-28T22:17:25.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.134+0000 7f52a7208740 1 -- 172.21.15.16:0/2495933205 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564ed57e63f0 con 0x564ed5b6ebb0 2024-03-28T22:17:25.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.135+0000 7f5285ffb640 1 -- 172.21.15.16:0/2495933205 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f528c0047a0 con 0x564ed5b6ebb0 2024-03-28T22:17:25.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.135+0000 7f5285ffb640 1 -- 172.21.15.16:0/2495933205 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f528c0035f0 con 0x564ed5b6ebb0 2024-03-28T22:17:25.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.135+0000 7f52a7208740 10 client.16902 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:17:25.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.135+0000 7f52a7208740 10 client.16902 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:17:25.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.135+0000 7f52a7208740 20 client.16902 populate_metadata read hostname 'smithi016' 2024-03-28T22:17:25.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.135+0000 7f52a7208740 1 --2- 172.21.15.16:0/2495933205 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x564ed5b87a20 0x564ed5b89e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.135+0000 7f52a7208740 4 client.16902 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:17:25.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.135+0000 7f52a7208740 1 -- 172.21.15.16:0/2495933205 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ed5adf6c0 con 0x564ed5b87a20 2024-03-28T22:17:25.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.135+0000 7f52a23bf640 1 --2- 172.21.15.16:0/2495933205 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x564ed5b87a20 0x564ed5b89e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.135+0000 7f52a23bf640 1 --2- 172.21.15.16:0/2495933205 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x564ed5b87a20 0x564ed5b89e10 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f35609205d8e10a2 server_cookie=92f185e59157871c in_seq=0 out_seq=0 2024-03-28T22:17:25.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.135+0000 7f5285ffb640 10 client.16902 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:17:25.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.136+0000 7f5285ffb640 1 -- 172.21.15.16:0/2495933205 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564ed5adf6c0 con 0x564ed5b87a20 2024-03-28T22:17:25.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.136+0000 7f5285ffb640 10 client.16902 handle_command_reply: tid=0 2024-03-28T22:17:25.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.136+0000 7f52a7208740 1 client.16902 shutdown 2024-03-28T22:17:25.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.136+0000 7f52a7208740 1 -- 172.21.15.16:0/2495933205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed5b6ebb0 msgr2=0x564ed5b74cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.136+0000 7f52a7208740 1 --2- 172.21.15.16:0/2495933205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed5b6ebb0 0x564ed5b74cd0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f528c0057c0 tx=0x7f528c0052b0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.136+0000 7f52a7208740 1 -- 172.21.15.16:0/2495933205 shutdown_connections 2024-03-28T22:17:25.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.136+0000 7f52a7208740 1 --2- 172.21.15.16:0/2495933205 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x564ed5b87a20 0x564ed5b89e10 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.136+0000 7f52a7208740 1 --2- 172.21.15.16:0/2495933205 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ed5b77790 0x564ed5b75210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.136+0000 7f5285ffb640 0 client.16902 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:17:25.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.136+0000 7f52a7208740 1 --2- 172.21.15.16:0/2495933205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed5b6ebb0 0x564ed5b74cd0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.136+0000 7f52a7208740 1 --2- 172.21.15.16:0/2495933205 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ed5b6e290 0x564ed5b74790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.136+0000 7f52a7208740 1 -- 172.21.15.16:0/2495933205 >> 172.21.15.16:0/2495933205 conn(0x564ed5b689c0 msgr2=0x564ed5b67c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:25.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.136+0000 7f52a7208740 1 -- 172.21.15.16:0/2495933205 shutdown_connections 2024-03-28T22:17:25.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.136+0000 7f52a7208740 1 -- 172.21.15.16:0/2495933205 wait complete. 2024-03-28T22:17:25.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.136+0000 7f52a7208740 20 client.16902 trim_cache size 0 max 16384 2024-03-28T22:17:25.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.154+0000 7f52a7208740 1 Processor -- start 2024-03-28T22:17:25.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.154+0000 7f52a7208740 1 -- start start 2024-03-28T22:17:25.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.154+0000 7f52a7208740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed5b6e290 0x564ed5b76ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.154+0000 7f52a7208740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ed5b6ebb0 0x564ed5b83320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.154+0000 7f52a7208740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ed5b77790 0x564ed5b6a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.154+0000 7f52a7208740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f529c1b1f60 con 0x564ed5b6e290 2024-03-28T22:17:25.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.154+0000 7f52a7208740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f529c1af830 con 0x564ed5b77790 2024-03-28T22:17:25.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.154+0000 7f52a7208740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f529c1ad6a0 con 0x564ed5b6ebb0 2024-03-28T22:17:25.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.154+0000 7f52a23bf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed5b6e290 0x564ed5b76ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.154+0000 7f52a23bf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed5b6e290 0x564ed5b76ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45472/0 (socket says 172.21.15.16:45472) 2024-03-28T22:17:25.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.154+0000 7f52a23bf640 1 -- 172.21.15.16:0/541315487 learned_addr learned my addr 172.21.15.16:0/541315487 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:25.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.154+0000 7f52a2bc0640 1 --2- 172.21.15.16:0/541315487 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ed5b77790 0x564ed5b6a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.155+0000 7f52a23bf640 1 -- 172.21.15.16:0/541315487 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ed5b6ebb0 msgr2=0x564ed5b83320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:25.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.155+0000 7f52a23bf640 1 --2- 172.21.15.16:0/541315487 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ed5b6ebb0 0x564ed5b83320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.155+0000 7f52a23bf640 1 -- 172.21.15.16:0/541315487 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ed5b77790 msgr2=0x564ed5b6a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:25.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.155+0000 7f52a23bf640 1 --2- 172.21.15.16:0/541315487 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ed5b77790 0x564ed5b6a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.155+0000 7f52a23bf640 1 -- 172.21.15.16:0/541315487 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ed5b8a180 con 0x564ed5b6e290 2024-03-28T22:17:25.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.155+0000 7f52a23bf640 1 --2- 172.21.15.16:0/541315487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed5b6e290 0x564ed5b76ec0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f52980ed330 tx=0x7f52980a7040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72cf2356aef73883 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:25.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.155+0000 7f5287fff640 1 -- 172.21.15.16:0/541315487 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f529804b040 con 0x564ed5b6e290 2024-03-28T22:17:25.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.155+0000 7f5287fff640 1 -- 172.21.15.16:0/541315487 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5298004c80 con 0x564ed5b6e290 2024-03-28T22:17:25.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.155+0000 7f5287fff640 1 -- 172.21.15.16:0/541315487 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52980c06c0 con 0x564ed5b6e290 2024-03-28T22:17:25.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.155+0000 7f52a7208740 1 -- 172.21.15.16:0/541315487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed5b6e290 msgr2=0x564ed5b76ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.155+0000 7f52a7208740 1 --2- 172.21.15.16:0/541315487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed5b6e290 0x564ed5b76ec0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f52980ed330 tx=0x7f52980a7040 comp rx=0 tx=0).stop 2024-03-28T22:17:25.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.155+0000 7f52a7208740 1 -- 172.21.15.16:0/541315487 shutdown_connections 2024-03-28T22:17:25.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.155+0000 7f52a7208740 1 --2- 172.21.15.16:0/541315487 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ed5b77790 0x564ed5b6a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.155+0000 7f52a7208740 1 --2- 172.21.15.16:0/541315487 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ed5b6ebb0 0x564ed5b83320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.155+0000 7f52a7208740 1 --2- 172.21.15.16:0/541315487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed5b6e290 0x564ed5b76ec0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.155+0000 7f52a7208740 1 -- 172.21.15.16:0/541315487 >> 172.21.15.16:0/541315487 conn(0x564ed5b689c0 msgr2=0x564ed5b69da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:25.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.156+0000 7f52a7208740 1 -- 172.21.15.16:0/541315487 shutdown_connections 2024-03-28T22:17:25.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.156+0000 7f52a7208740 1 -- 172.21.15.16:0/541315487 wait complete. 2024-03-28T22:17:25.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.156+0000 7f52a7208740 1 -- start start 2024-03-28T22:17:25.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.156+0000 7f52a7208740 1 Processor -- start 2024-03-28T22:17:25.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.156+0000 7f52a7208740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ed5b6e290 0x564ed5b7b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.156+0000 7f52a7208740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ed5b6ebb0 0x564ed5b7bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.156+0000 7f52a7208740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed5b77790 0x564ed5becc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.156+0000 7f52a7208740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f529c1adcf0 con 0x564ed5b77790 2024-03-28T22:17:25.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.156+0000 7f52a7208740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f529c1af160 con 0x564ed5b6ebb0 2024-03-28T22:17:25.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.156+0000 7f52a7208740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f529c1ae3c0 con 0x564ed5b6e290 2024-03-28T22:17:25.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.156+0000 7f52a7208740 10 client.0 _ll_register_callbacks cb 0x7f52980b8940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:17:25.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.156+0000 7f52a2bc0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed5b77790 0x564ed5becc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.156+0000 7f52a1bbe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ed5b6ebb0 0x564ed5b7bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.156+0000 7f52a2bc0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed5b77790 0x564ed5becc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45482/0 (socket says 172.21.15.16:45482) 2024-03-28T22:17:25.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.156+0000 7f52a1bbe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ed5b6ebb0 0x564ed5b7bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:52474/0 (socket says 172.21.15.16:52474) 2024-03-28T22:17:25.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.156+0000 7f52a23bf640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ed5b6e290 0x564ed5b7b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.156+0000 7f52a1bbe640 1 -- 172.21.15.16:0/350788077 learned_addr learned my addr 172.21.15.16:0/350788077 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:25.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.157+0000 7f52a1bbe640 1 -- 172.21.15.16:0/350788077 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ed5b6e290 msgr2=0x564ed5b7b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:25.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.157+0000 7f52a1bbe640 1 --2- 172.21.15.16:0/350788077 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ed5b6e290 0x564ed5b7b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.157+0000 7f52a1bbe640 1 -- 172.21.15.16:0/350788077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed5b77790 msgr2=0x564ed5becc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:25.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.157+0000 7f52a1bbe640 1 --2- 172.21.15.16:0/350788077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed5b77790 0x564ed5becc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.157+0000 7f52a1bbe640 1 -- 172.21.15.16:0/350788077 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ed57a60f0 con 0x564ed5b6ebb0 2024-03-28T22:17:25.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.157+0000 7f52a2bc0640 1 --2- 172.21.15.16:0/350788077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed5b77790 0x564ed5becc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:17:25.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.157+0000 7f52a1bbe640 1 --2- 172.21.15.16:0/350788077 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ed5b6ebb0 0x564ed5b7bb10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f528c0039c0 tx=0x7f528c00f440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d1427fc584a7607 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:25.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.157+0000 7f5285ffb640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-03-28T22:17:25.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.157+0000 7f5285ffb640 1 -- 172.21.15.16:0/350788077 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f528c028040 con 0x564ed5b6ebb0 2024-03-28T22:17:25.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.157+0000 7f52a7208740 1 -- 172.21.15.16:0/350788077 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5298004c80 con 0x564ed5b6ebb0 2024-03-28T22:17:25.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.157+0000 7f5285ffb640 1 -- 172.21.15.16:0/350788077 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8527+0+0 (secure 0 0 0) 0x7f528c0035f0 con 0x564ed5b6ebb0 2024-03-28T22:17:25.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.157+0000 7f5285ffb640 1 -- 172.21.15.16:0/350788077 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f528c03e050 con 0x564ed5b6ebb0 2024-03-28T22:17:25.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.157+0000 7f52a7208740 10 client.26810 fetch_fsmap learned FSMap version 27 2024-03-28T22:17:25.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.157+0000 7f52a7208740 1 -- 172.21.15.16:0/350788077 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564ed5bee0e0 con 0x564ed5b6ebb0 2024-03-28T22:17:25.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.157+0000 7f5285ffb640 1 -- 172.21.15.16:0/350788077 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5298004c80 con 0x564ed5b6ebb0 2024-03-28T22:17:25.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.158+0000 7f5285ffb640 1 -- 172.21.15.16:0/350788077 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f528c00f9a0 con 0x564ed5b6ebb0 2024-03-28T22:17:25.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.158+0000 7f52a7208740 10 client.26810 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:17:25.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.158+0000 7f52a7208740 10 client.26810 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:17:25.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.158+0000 7f52a7208740 20 client.26810 populate_metadata read hostname 'smithi016' 2024-03-28T22:17:25.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.158+0000 7f52a7208740 1 --2- 172.21.15.16:0/350788077 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x564ed5bee7d0 0x564ed5bf0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.158+0000 7f52a7208740 4 client.26810 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:17:25.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.158+0000 7f52a7208740 1 -- 172.21.15.16:0/350788077 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564ed5ad9a10 con 0x564ed5bee7d0 2024-03-28T22:17:25.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.158+0000 7f52a23bf640 1 --2- 172.21.15.16:0/350788077 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x564ed5bee7d0 0x564ed5bf0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.158+0000 7f52a23bf640 1 --2- 172.21.15.16:0/350788077 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x564ed5bee7d0 0x564ed5bf0bc0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f26dd33a46e511c server_cookie=9d4a0ea6dcea1e28 in_seq=0 out_seq=0 2024-03-28T22:17:25.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.158+0000 7f5285ffb640 10 client.26810 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:17:25.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.158+0000 7f5285ffb640 1 -- 172.21.15.16:0/350788077 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x564ed5ad9a10 con 0x564ed5bee7d0 2024-03-28T22:17:25.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.158+0000 7f5285ffb640 10 client.26810 handle_command_reply: tid=0 2024-03-28T22:17:25.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.158+0000 7f52a7208740 1 client.26810 shutdown 2024-03-28T22:17:25.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.159+0000 7f52a7208740 1 -- 172.21.15.16:0/350788077 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ed5b6ebb0 msgr2=0x564ed5b7bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.159+0000 7f52a7208740 1 --2- 172.21.15.16:0/350788077 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ed5b6ebb0 0x564ed5b7bb10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f528c0039c0 tx=0x7f528c00f440 comp rx=0 tx=0).stop 2024-03-28T22:17:25.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.159+0000 7f52a7208740 1 -- 172.21.15.16:0/350788077 shutdown_connections 2024-03-28T22:17:25.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.159+0000 7f52a7208740 1 --2- 172.21.15.16:0/350788077 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x564ed5bee7d0 0x564ed5bf0bc0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.159+0000 7f52a7208740 1 --2- 172.21.15.16:0/350788077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564ed5b77790 0x564ed5becc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.159+0000 7f52a7208740 1 --2- 172.21.15.16:0/350788077 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564ed5b6ebb0 0x564ed5b7bb10 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.159+0000 7f52a7208740 1 --2- 172.21.15.16:0/350788077 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ed5b6e290 0x564ed5b7b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.159+0000 7f5285ffb640 0 client.26810 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:17:25.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.159+0000 7f52a7208740 1 -- 172.21.15.16:0/350788077 >> 172.21.15.16:0/350788077 conn(0x564ed5b689c0 msgr2=0x564ed5b74fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:25.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.159+0000 7f52a7208740 1 -- 172.21.15.16:0/350788077 shutdown_connections 2024-03-28T22:17:25.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.159+0000 7f52a7208740 1 -- 172.21.15.16:0/350788077 wait complete. 2024-03-28T22:17:25.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.159+0000 7f52a7208740 20 client.26810 trim_cache size 0 max 16384 2024-03-28T22:17:25.172 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:17:25.172 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-03-28T22:17:25.172 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-03-28T22:17:25.172 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:17:25.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.159+0000 7f52a4623640 1 -- 172.21.15.16:0/2748128081 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f52880a1ef0 msgr2=0x7f52880a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.159+0000 7f52a4623640 1 --2- 172.21.15.16:0/2748128081 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f52880a1ef0 0x7f52880a43b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5294009fd0 tx=0x7f5294004a40 comp rx=0 tx=0).stop 2024-03-28T22:17:25.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.159+0000 7f52a4623640 1 -- 172.21.15.16:0/2748128081 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f529c073d00 msgr2=0x7f529c19a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.159+0000 7f52a4623640 1 --2- 172.21.15.16:0/2748128081 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f529c073d00 0x7f529c19a360 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5298036040 tx=0x7f529802cb20 comp rx=0 tx=0).stop 2024-03-28T22:17:25.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.159+0000 7f52a4623640 1 -- 172.21.15.16:0/2748128081 shutdown_connections 2024-03-28T22:17:25.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.159+0000 7f52a4623640 1 --2- 172.21.15.16:0/2748128081 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f52880a1ef0 0x7f52880a43b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.159+0000 7f52a4623640 1 --2- 172.21.15.16:0/2748128081 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f529c10e0e0 0x7f529c19ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.159+0000 7f52a4623640 1 --2- 172.21.15.16:0/2748128081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f529c074620 0x7f529c19a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.159+0000 7f52a4623640 1 --2- 172.21.15.16:0/2748128081 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f529c073d00 0x7f529c19a360 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.159+0000 7f52a4623640 1 -- 172.21.15.16:0/2748128081 >> 172.21.15.16:0/2748128081 conn(0x7f529c0fd1e0 msgr2=0x7f529c0fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:25.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.159+0000 7f52a4623640 1 -- 172.21.15.16:0/2748128081 shutdown_connections 2024-03-28T22:17:25.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.159+0000 7f52a4623640 1 -- 172.21.15.16:0/2748128081 wait complete. 2024-03-28T22:17:25.175 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:17:25.175 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:17:25.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.277+0000 7fde2288f640 1 -- 172.21.15.16:0/3487461618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde1c100fc0 msgr2=0x7fde1c10d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.277+0000 7fde2288f640 1 --2- 172.21.15.16:0/3487461618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde1c100fc0 0x7fde1c10d4d0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fde14009cd0 tx=0x7fde1402c0f0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.277+0000 7fde2288f640 1 -- 172.21.15.16:0/3487461618 shutdown_connections 2024-03-28T22:17:25.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.277+0000 7fde2288f640 1 --2- 172.21.15.16:0/3487461618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde1c100fc0 0x7fde1c10d4d0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.277+0000 7fde2288f640 1 --2- 172.21.15.16:0/3487461618 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde1c100600 0x7fde1c100a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.277+0000 7fde2288f640 1 --2- 172.21.15.16:0/3487461618 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde1c1066a0 0x7fde1c106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.277+0000 7fde2288f640 1 -- 172.21.15.16:0/3487461618 >> 172.21.15.16:0/3487461618 conn(0x7fde1c0777d0 msgr2=0x7fde1c077be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:25.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.277+0000 7fde2288f640 1 -- 172.21.15.16:0/3487461618 shutdown_connections 2024-03-28T22:17:25.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.277+0000 7fde2288f640 1 -- 172.21.15.16:0/3487461618 wait complete. 2024-03-28T22:17:25.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.277+0000 7fde2288f640 1 Processor -- start 2024-03-28T22:17:25.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.278+0000 7fde2288f640 1 -- start start 2024-03-28T22:17:25.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.278+0000 7fde2288f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde1c100600 0x7fde1c197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.278+0000 7fde2288f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde1c100fc0 0x7fde1c1984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.278+0000 7fde2288f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde1c1066a0 0x7fde1c19c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.278+0000 7fde2288f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fde1c111b20 con 0x7fde1c100fc0 2024-03-28T22:17:25.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.278+0000 7fde2288f640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fde1c1119a0 con 0x7fde1c100600 2024-03-28T22:17:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.278+0000 7fde2288f640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fde1c111ca0 con 0x7fde1c1066a0 2024-03-28T22:17:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.278+0000 7fde1b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde1c100fc0 0x7fde1c1984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.278+0000 7fde1b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde1c100fc0 0x7fde1c1984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45500/0 (socket says 172.21.15.16:45500) 2024-03-28T22:17:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.278+0000 7fde1b7fe640 1 -- 172.21.15.16:0/525888522 learned_addr learned my addr 172.21.15.16:0/525888522 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.278+0000 7fde1bfff640 1 --2- 172.21.15.16:0/525888522 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde1c100600 0x7fde1c197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.278+0000 7fde20e2c640 1 --2- 172.21.15.16:0/525888522 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde1c1066a0 0x7fde1c19c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.278+0000 7fde1b7fe640 1 -- 172.21.15.16:0/525888522 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde1c1066a0 msgr2=0x7fde1c19c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.278+0000 7fde1b7fe640 1 --2- 172.21.15.16:0/525888522 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde1c1066a0 0x7fde1c19c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.278+0000 7fde1b7fe640 1 -- 172.21.15.16:0/525888522 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde1c100600 msgr2=0x7fde1c197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.278+0000 7fde1b7fe640 1 --2- 172.21.15.16:0/525888522 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde1c100600 0x7fde1c197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.278+0000 7fde1b7fe640 1 -- 172.21.15.16:0/525888522 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde1c19cf50 con 0x7fde1c100fc0 2024-03-28T22:17:25.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.278+0000 7fde20e2c640 1 --2- 172.21.15.16:0/525888522 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde1c1066a0 0x7fde1c19c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:17:25.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.278+0000 7fde1b7fe640 1 --2- 172.21.15.16:0/525888522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde1c100fc0 0x7fde1c1984a0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fde1000e9f0 tx=0x7fde1000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:25.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.279+0000 7fde197fa640 1 -- 172.21.15.16:0/525888522 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde1000cd80 con 0x7fde1c100fc0 2024-03-28T22:17:25.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.279+0000 7fde2288f640 1 -- 172.21.15.16:0/525888522 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde1c19d240 con 0x7fde1c100fc0 2024-03-28T22:17:25.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.279+0000 7fde197fa640 1 -- 172.21.15.16:0/525888522 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde10005500 con 0x7fde1c100fc0 2024-03-28T22:17:25.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.279+0000 7fde197fa640 1 -- 172.21.15.16:0/525888522 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde10010690 con 0x7fde1c100fc0 2024-03-28T22:17:25.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.279+0000 7fde2288f640 1 -- 172.21.15.16:0/525888522 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde1c1a4ad0 con 0x7fde1c100fc0 2024-03-28T22:17:25.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.280+0000 7fde2288f640 1 -- 172.21.15.16:0/525888522 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde1c068a20 con 0x7fde1c100fc0 2024-03-28T22:17:25.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.281+0000 7fde197fa640 1 -- 172.21.15.16:0/525888522 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7fde100046c0 con 0x7fde1c100fc0 2024-03-28T22:17:25.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.281+0000 7fde197fa640 1 --2- 172.21.15.16:0/525888522 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fde040a1ef0 0x7fde040a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.281+0000 7fde197fa640 1 -- 172.21.15.16:0/525888522 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8527+0+0 (secure 0 0 0) 0x7fde100ce6c0 con 0x7fde1c100fc0 2024-03-28T22:17:25.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.281+0000 7fde1bfff640 1 --2- 172.21.15.16:0/525888522 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fde040a1ef0 0x7fde040a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.282+0000 7fde1bfff640 1 --2- 172.21.15.16:0/525888522 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fde040a1ef0 0x7fde040a43b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fde1c101740 tx=0x7fde0c005f90 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:25.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.286+0000 7fde197fa640 1 -- 172.21.15.16:0/525888522 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fde1009b750 con 0x7fde1c100fc0 2024-03-28T22:17:25.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.510+0000 7fde2288f640 1 -- 172.21.15.16:0/525888522 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fde1c108600 con 0x7fde1c100fc0 2024-03-28T22:17:25.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.510+0000 7fde197fa640 1 -- 172.21.15.16:0/525888522 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11851 (secure 0 0 0) 0x7fde1009b050 con 0x7fde1c100fc0 2024-03-28T22:17:25.512 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:17:25.512 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/1644581227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1644581227},{"type":"v1","addr":"172.21.15.16:6839","nonce":1644581227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/2593062669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2593062669},{"type":"v1","addr":"172.21.15.16:6841","nonce":2593062669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:08:29.672976+0000","modified":"2024-03-28T22:14:44.922044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24566,"mds_1":24524,"mds_2":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24466":{"gid":24466,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/247249857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":247249857},{"type":"v1","addr":"172.21.15.63:6835","nonce":247249857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6837/2324231032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":2324231032},{"type":"v1","addr":"172.21.15.63:6837","nonce":2324231032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6839/1177197077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":1177197077},{"type":"v1","addr":"172.21.15.63:6839","nonce":1177197077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"e","rank":2,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.53:6835/350727634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":350727634},{"type":"v1","addr":"172.21.15.53:6835","nonce":350727634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/349877573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":349877573},{"type":"v1","addr":"172.21.15.53:6837","nonce":349877573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6839/3714153956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":3714153956},{"type":"v1","addr":"172.21.15.53:6839","nonce":3714153956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24566,"qdb_cluster":[24500,24524,24566]},"id":1}]} 2024-03-28T22:17:25.513 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2024-03-28T22:17:25.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.513+0000 7fde2288f640 1 -- 172.21.15.16:0/525888522 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fde040a1ef0 msgr2=0x7fde040a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.513+0000 7fde2288f640 1 --2- 172.21.15.16:0/525888522 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fde040a1ef0 0x7fde040a43b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fde1c101740 tx=0x7fde0c005f90 comp rx=0 tx=0).stop 2024-03-28T22:17:25.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.513+0000 7fde2288f640 1 -- 172.21.15.16:0/525888522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde1c100fc0 msgr2=0x7fde1c1984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.513+0000 7fde2288f640 1 --2- 172.21.15.16:0/525888522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde1c100fc0 0x7fde1c1984a0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fde1000e9f0 tx=0x7fde1000edf0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.513+0000 7fde2288f640 1 -- 172.21.15.16:0/525888522 shutdown_connections 2024-03-28T22:17:25.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.513+0000 7fde2288f640 1 --2- 172.21.15.16:0/525888522 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fde040a1ef0 0x7fde040a43b0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.513+0000 7fde2288f640 1 --2- 172.21.15.16:0/525888522 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde1c1066a0 0x7fde1c19c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.513+0000 7fde2288f640 1 --2- 172.21.15.16:0/525888522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde1c100fc0 0x7fde1c1984a0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.513+0000 7fde2288f640 1 --2- 172.21.15.16:0/525888522 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde1c100600 0x7fde1c197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.513+0000 7fde2288f640 1 -- 172.21.15.16:0/525888522 >> 172.21.15.16:0/525888522 conn(0x7fde1c0777d0 msgr2=0x7fde1c0ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:25.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.513+0000 7fde2288f640 1 -- 172.21.15.16:0/525888522 shutdown_connections 2024-03-28T22:17:25.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.513+0000 7fde2288f640 1 -- 172.21.15.16:0/525888522 wait complete. 2024-03-28T22:17:25.525 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:17:25.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.628+0000 7f969fcfb640 1 -- 172.21.15.16:0/2701312075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9698100fb0 msgr2=0x7f969810d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.628+0000 7f969fcfb640 1 --2- 172.21.15.16:0/2701312075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9698100fb0 0x7f969810d4c0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f9690009cd0 tx=0x7f969002c0f0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.628+0000 7f969fcfb640 1 -- 172.21.15.16:0/2701312075 shutdown_connections 2024-03-28T22:17:25.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.628+0000 7f969fcfb640 1 --2- 172.21.15.16:0/2701312075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9698100fb0 0x7f969810d4c0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.628+0000 7f969fcfb640 1 --2- 172.21.15.16:0/2701312075 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96981005f0 0x7f9698100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.628+0000 7f969fcfb640 1 --2- 172.21.15.16:0/2701312075 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9698106690 0x7f9698106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.628+0000 7f969fcfb640 1 -- 172.21.15.16:0/2701312075 >> 172.21.15.16:0/2701312075 conn(0x7f9698077800 msgr2=0x7f9698077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:25.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.628+0000 7f969fcfb640 1 -- 172.21.15.16:0/2701312075 shutdown_connections 2024-03-28T22:17:25.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.628+0000 7f969fcfb640 1 -- 172.21.15.16:0/2701312075 wait complete. 2024-03-28T22:17:25.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.628+0000 7f969fcfb640 1 Processor -- start 2024-03-28T22:17:25.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.628+0000 7f969fcfb640 1 -- start start 2024-03-28T22:17:25.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.629+0000 7f969fcfb640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96981005f0 0x7f9698197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.629+0000 7f969fcfb640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9698100fb0 0x7f9698198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.629+0000 7f969fcfb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9698106690 0x7f969819c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.629+0000 7f969fcfb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9698111b10 con 0x7f9698106690 2024-03-28T22:17:25.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.629+0000 7f969fcfb640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9698111990 con 0x7f9698100fb0 2024-03-28T22:17:25.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.629+0000 7f969fcfb640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9698111c90 con 0x7f96981005f0 2024-03-28T22:17:25.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.629+0000 7f969e298640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9698106690 0x7f969819c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.629+0000 7f969e298640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9698106690 0x7f969819c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45516/0 (socket says 172.21.15.16:45516) 2024-03-28T22:17:25.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.629+0000 7f969e298640 1 -- 172.21.15.16:0/2675929615 learned_addr learned my addr 172.21.15.16:0/2675929615 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:25.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.629+0000 7f969d296640 1 --2- 172.21.15.16:0/2675929615 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9698100fb0 0x7f9698198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.629+0000 7f969da97640 1 --2- 172.21.15.16:0/2675929615 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96981005f0 0x7f9698197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.629+0000 7f969e298640 1 -- 172.21.15.16:0/2675929615 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96981005f0 msgr2=0x7f9698197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.629+0000 7f969e298640 1 --2- 172.21.15.16:0/2675929615 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96981005f0 0x7f9698197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.629+0000 7f969e298640 1 -- 172.21.15.16:0/2675929615 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9698100fb0 msgr2=0x7f9698198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.629+0000 7f969e298640 1 --2- 172.21.15.16:0/2675929615 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9698100fb0 0x7f9698198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.629+0000 7f969e298640 1 -- 172.21.15.16:0/2675929615 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f969819cec0 con 0x7f9698106690 2024-03-28T22:17:25.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.629+0000 7f969d296640 1 --2- 172.21.15.16:0/2675929615 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9698100fb0 0x7f9698198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:17:25.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.629+0000 7f969e298640 1 --2- 172.21.15.16:0/2675929615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9698106690 0x7f969819c790 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f96900348f0 tx=0x7f9690032c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:25.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.629+0000 7f968effd640 1 -- 172.21.15.16:0/2675929615 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9690041020 con 0x7f9698106690 2024-03-28T22:17:25.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.629+0000 7f968effd640 1 -- 172.21.15.16:0/2675929615 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9690032e80 con 0x7f9698106690 2024-03-28T22:17:25.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.629+0000 7f969fcfb640 1 -- 172.21.15.16:0/2675929615 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f969819d150 con 0x7f9698106690 2024-03-28T22:17:25.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.629+0000 7f968effd640 1 -- 172.21.15.16:0/2675929615 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9690039040 con 0x7f9698106690 2024-03-28T22:17:25.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.629+0000 7f969fcfb640 1 -- 172.21.15.16:0/2675929615 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96981a4a40 con 0x7f9698106690 2024-03-28T22:17:25.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.632+0000 7f968effd640 1 -- 172.21.15.16:0/2675929615 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f9690044020 con 0x7f9698106690 2024-03-28T22:17:25.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.632+0000 7f96a29ba740 1 Processor -- start 2024-03-28T22:17:25.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.632+0000 7f96a29ba740 1 -- start start 2024-03-28T22:17:25.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.632+0000 7f96a29ba740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7237d3470 0x55a7237d3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.632+0000 7f96a29ba740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7237d3d90 0x55a7237dc430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.632+0000 7f968effd640 1 --2- 172.21.15.16:0/2675929615 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f96840a1f70 0x7f96840a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.632+0000 7f96a29ba740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a7237dc970 0x55a7237ded60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.633+0000 7f96a29ba740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a7237371b0 con 0x55a7237d3470 2024-03-28T22:17:25.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.633+0000 7f96a29ba740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55a7236dfc00 con 0x55a7237dc970 2024-03-28T22:17:25.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.633+0000 7f96a29ba740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55a723733250 con 0x55a7237d3d90 2024-03-28T22:17:25.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.633+0000 7f969d296640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7237d3470 0x55a7237d3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.633+0000 7f969da97640 1 --2- 172.21.15.16:0/2675929615 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f96840a1f70 0x7f96840a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.633+0000 7f969d296640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7237d3470 0x55a7237d3850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45520/0 (socket says 172.21.15.16:45520) 2024-03-28T22:17:25.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.633+0000 7f969d296640 1 -- 172.21.15.16:0/1694992744 learned_addr learned my addr 172.21.15.16:0/1694992744 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:25.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.633+0000 7f969e298640 1 --2- 172.21.15.16:0/1694992744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a7237dc970 0x55a7237ded60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.633+0000 7f968effd640 1 -- 172.21.15.16:0/2675929615 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8527+0+0 (secure 0 0 0) 0x7f96900efca0 con 0x7f9698106690 2024-03-28T22:17:25.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.633+0000 7f969d296640 1 --2- 172.21.15.16:0/1694992744 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7237d3d90 0x55a7237dc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.633+0000 7f969e298640 1 -- 172.21.15.16:0/1694992744 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7237d3d90 msgr2=0x55a7237dc430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:25.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.633+0000 7f969e298640 1 --2- 172.21.15.16:0/1694992744 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7237d3d90 0x55a7237dc430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.633+0000 7f969e298640 1 -- 172.21.15.16:0/1694992744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7237d3470 msgr2=0x55a7237d3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:25.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.633+0000 7f969e298640 1 --2- 172.21.15.16:0/1694992744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7237d3470 0x55a7237d3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.633+0000 7f969e298640 1 -- 172.21.15.16:0/1694992744 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7236ff3d0 con 0x55a7237dc970 2024-03-28T22:17:25.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.633+0000 7f969d296640 1 --2- 172.21.15.16:0/1694992744 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7237d3d90 0x55a7237dc430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:17:25.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.633+0000 7f969da97640 1 --2- 172.21.15.16:0/2675929615 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f96840a1f70 0x7f96840a4430 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9694005fd0 tx=0x7f969400b5b0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:25.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.633+0000 7f969e298640 1 --2- 172.21.15.16:0/1694992744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a7237dc970 0x55a7237ded60 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f96900ed500 tx=0x7f9690003f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b9a1cb7d742abf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:25.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.633+0000 7f9683fff640 1 -- 172.21.15.16:0/1694992744 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96900bc2e0 con 0x55a7237dc970 2024-03-28T22:17:25.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.633+0000 7f9683fff640 1 -- 172.21.15.16:0/1694992744 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9690034920 con 0x55a7237dc970 2024-03-28T22:17:25.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.633+0000 7f9683fff640 1 -- 172.21.15.16:0/1694992744 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9690034e60 con 0x55a7237dc970 2024-03-28T22:17:25.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.634+0000 7f96a29ba740 1 -- 172.21.15.16:0/1694992744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a7237dc970 msgr2=0x55a7237ded60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.634+0000 7f96a29ba740 1 --2- 172.21.15.16:0/1694992744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a7237dc970 0x55a7237ded60 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f96900ed500 tx=0x7f9690003f90 comp rx=0 tx=0).stop 2024-03-28T22:17:25.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.634+0000 7f96a29ba740 1 -- 172.21.15.16:0/1694992744 shutdown_connections 2024-03-28T22:17:25.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.634+0000 7f96a29ba740 1 --2- 172.21.15.16:0/1694992744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a7237dc970 0x55a7237ded60 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.634+0000 7f96a29ba740 1 --2- 172.21.15.16:0/1694992744 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7237d3d90 0x55a7237dc430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.634+0000 7f96a29ba740 1 --2- 172.21.15.16:0/1694992744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7237d3470 0x55a7237d3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.634+0000 7f96a29ba740 1 -- 172.21.15.16:0/1694992744 >> 172.21.15.16:0/1694992744 conn(0x55a7237cdba0 msgr2=0x55a7237cdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:25.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.634+0000 7f96a29ba740 1 -- 172.21.15.16:0/1694992744 shutdown_connections 2024-03-28T22:17:25.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.634+0000 7f96a29ba740 1 -- 172.21.15.16:0/1694992744 wait complete. 2024-03-28T22:17:25.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.635+0000 7f96a29ba740 1 -- start start 2024-03-28T22:17:25.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.635+0000 7f96a29ba740 1 Processor -- start 2024-03-28T22:17:25.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.635+0000 7f96a29ba740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a7237d3470 0x55a7237d98f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.635+0000 7f96a29ba740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7237d3d90 0x55a7237d9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.635+0000 7f96a29ba740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7237dc970 0x55a7237da370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.635+0000 7f96a29ba740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a7237df520 con 0x55a7237d3d90 2024-03-28T22:17:25.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.635+0000 7f96a29ba740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55a7236d5470 con 0x55a7237d3470 2024-03-28T22:17:25.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.635+0000 7f96a29ba740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55a7237df6a0 con 0x55a7237dc970 2024-03-28T22:17:25.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.635+0000 7f96a29ba740 10 client.0 _ll_register_callbacks cb 0x55a7236f2cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:17:25.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.635+0000 7f969d296640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7237d3d90 0x55a7237d9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.635+0000 7f969da97640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a7237d3470 0x55a7237d98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.635+0000 7f969d296640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7237d3d90 0x55a7237d9e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45522/0 (socket says 172.21.15.16:45522) 2024-03-28T22:17:25.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.635+0000 7f969d296640 1 -- 172.21.15.16:0/4160004029 learned_addr learned my addr 172.21.15.16:0/4160004029 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:25.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.635+0000 7f969e298640 1 --2- 172.21.15.16:0/4160004029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7237dc970 0x55a7237da370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.635+0000 7f969d296640 1 -- 172.21.15.16:0/4160004029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7237dc970 msgr2=0x55a7237da370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:25.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.635+0000 7f969d296640 1 --2- 172.21.15.16:0/4160004029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7237dc970 0x55a7237da370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.635+0000 7f969d296640 1 -- 172.21.15.16:0/4160004029 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a7237d3470 msgr2=0x55a7237d98f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:25.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.635+0000 7f969d296640 1 --2- 172.21.15.16:0/4160004029 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a7237d3470 0x55a7237d98f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.635+0000 7f969d296640 1 -- 172.21.15.16:0/4160004029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a72344b3f0 con 0x55a7237d3d90 2024-03-28T22:17:25.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.635+0000 7f969da97640 1 --2- 172.21.15.16:0/4160004029 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a7237d3470 0x55a7237d98f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:17:25.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.636+0000 7f969d296640 1 --2- 172.21.15.16:0/4160004029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7237d3d90 0x55a7237d9e30 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f9688012ca0 tx=0x7f968800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d5ff97f7a6bb352 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:25.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.636+0000 7f96817fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:17:25.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.636+0000 7f96817fa640 1 -- 172.21.15.16:0/4160004029 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f968800fe80 con 0x55a7237d3d90 2024-03-28T22:17:25.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.636+0000 7f96817fa640 1 -- 172.21.15.16:0/4160004029 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8527+0+0 (secure 0 0 0) 0x7f9688004050 con 0x55a7237d3d90 2024-03-28T22:17:25.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.636+0000 7f96a29ba740 1 -- 172.21.15.16:0/4160004029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9690034920 con 0x55a7237d3d90 2024-03-28T22:17:25.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.636+0000 7f96817fa640 1 -- 172.21.15.16:0/4160004029 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9688004290 con 0x55a7237d3d90 2024-03-28T22:17:25.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.636+0000 7f96a29ba740 10 client.16926 fetch_fsmap learned FSMap version 27 2024-03-28T22:17:25.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.636+0000 7f96a29ba740 1 -- 172.21.15.16:0/4160004029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a7235157d0 con 0x55a7237d3d90 2024-03-28T22:17:25.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.636+0000 7f96817fa640 1 -- 172.21.15.16:0/4160004029 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9690034920 con 0x55a7237d3d90 2024-03-28T22:17:25.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.636+0000 7f96817fa640 1 -- 172.21.15.16:0/4160004029 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f96880104c0 con 0x55a7237d3d90 2024-03-28T22:17:25.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.636+0000 7f96a29ba740 10 client.16926 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:17:25.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.636+0000 7f96a29ba740 10 client.16926 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-28T22:17:25.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.636+0000 7f96a29ba740 20 client.16926 populate_metadata read hostname 'smithi016' 2024-03-28T22:17:25.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.636+0000 7f96a29ba740 1 --2- 172.21.15.16:0/4160004029 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55a7237ec5a0 0x55a7237ee9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.636+0000 7f96a29ba740 4 client.16926 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:17:25.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.636+0000 7f96a29ba740 1 -- 172.21.15.16:0/4160004029 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a7237446f0 con 0x55a7237ec5a0 2024-03-28T22:17:25.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.637+0000 7f969da97640 1 --2- 172.21.15.16:0/4160004029 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55a7237ec5a0 0x55a7237ee9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.637+0000 7f96817fa640 10 client.16926 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:17:25.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.637+0000 7f969da97640 1 --2- 172.21.15.16:0/4160004029 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55a7237ec5a0 0x55a7237ee9f0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d214959f42874696 server_cookie=8343f597b853d67b in_seq=0 out_seq=0 2024-03-28T22:17:25.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.638+0000 7f96817fa640 1 -- 172.21.15.16:0/4160004029 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a7237446f0 con 0x55a7237ec5a0 2024-03-28T22:17:25.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.638+0000 7f96817fa640 10 client.16926 handle_command_reply: tid=0 2024-03-28T22:17:25.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.638+0000 7f96a29ba740 1 client.16926 shutdown 2024-03-28T22:17:25.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.638+0000 7f96a29ba740 1 -- 172.21.15.16:0/4160004029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7237d3d90 msgr2=0x55a7237d9e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.638+0000 7f96a29ba740 1 --2- 172.21.15.16:0/4160004029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7237d3d90 0x55a7237d9e30 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f9688012ca0 tx=0x7f968800f790 comp rx=0 tx=0).stop 2024-03-28T22:17:25.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.638+0000 7f96a29ba740 1 -- 172.21.15.16:0/4160004029 shutdown_connections 2024-03-28T22:17:25.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.638+0000 7f96a29ba740 1 --2- 172.21.15.16:0/4160004029 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55a7237ec5a0 0x55a7237ee9f0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.638+0000 7f96a29ba740 1 --2- 172.21.15.16:0/4160004029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7237dc970 0x55a7237da370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.638+0000 7f96a29ba740 1 --2- 172.21.15.16:0/4160004029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7237d3d90 0x55a7237d9e30 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.638+0000 7f96a29ba740 1 --2- 172.21.15.16:0/4160004029 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a7237d3470 0x55a7237d98f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.638+0000 7f96817fa640 0 client.16926 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:17:25.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.638+0000 7f96a29ba740 1 -- 172.21.15.16:0/4160004029 >> 172.21.15.16:0/4160004029 conn(0x55a7237cdba0 msgr2=0x55a7237cf8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:25.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.638+0000 7f96a29ba740 1 -- 172.21.15.16:0/4160004029 shutdown_connections 2024-03-28T22:17:25.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.638+0000 7f96a29ba740 1 -- 172.21.15.16:0/4160004029 wait complete. 2024-03-28T22:17:25.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.638+0000 7f96a29ba740 20 client.16926 trim_cache size 0 max 16384 2024-03-28T22:17:25.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.651+0000 7f96a29ba740 1 Processor -- start 2024-03-28T22:17:25.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.651+0000 7f96a29ba740 1 -- start start 2024-03-28T22:17:25.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.652+0000 7f96a29ba740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a7237d3470 0x55a7237dc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.652+0000 7f96a29ba740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7237d3d90 0x55a7237c8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.652+0000 7f96a29ba740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7237dc970 0x55a7237e8210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.652+0000 7f96a29ba740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f96981afbe0 con 0x55a7237d3d90 2024-03-28T22:17:25.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.652+0000 7f96a29ba740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f96981ad4b0 con 0x55a7237d3470 2024-03-28T22:17:25.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.652+0000 7f96a29ba740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f96981ab320 con 0x55a7237dc970 2024-03-28T22:17:25.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.652+0000 7f969d296640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7237d3d90 0x55a7237c8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.652+0000 7f969d296640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7237d3d90 0x55a7237c8bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45528/0 (socket says 172.21.15.16:45528) 2024-03-28T22:17:25.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.652+0000 7f969d296640 1 -- 172.21.15.16:0/3479651021 learned_addr learned my addr 172.21.15.16:0/3479651021 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:25.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.652+0000 7f969da97640 1 --2- 172.21.15.16:0/3479651021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a7237d3470 0x55a7237dc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.652+0000 7f969e298640 1 --2- 172.21.15.16:0/3479651021 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7237dc970 0x55a7237e8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.652+0000 7f969d296640 1 -- 172.21.15.16:0/3479651021 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7237dc970 msgr2=0x55a7237e8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:25.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.652+0000 7f969d296640 1 --2- 172.21.15.16:0/3479651021 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7237dc970 0x55a7237e8210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.652+0000 7f969d296640 1 -- 172.21.15.16:0/3479651021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a7237d3470 msgr2=0x55a7237dc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:25.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.652+0000 7f969d296640 1 --2- 172.21.15.16:0/3479651021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a7237d3470 0x55a7237dc020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.652+0000 7f969d296640 1 -- 172.21.15.16:0/3479651021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7237d88a0 con 0x55a7237d3d90 2024-03-28T22:17:25.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.652+0000 7f969e298640 1 --2- 172.21.15.16:0/3479651021 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7237dc970 0x55a7237e8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:17:25.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.652+0000 7f969d296640 1 --2- 172.21.15.16:0/3479651021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7237d3d90 0x55a7237c8bd0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f9688001eb0 tx=0x7f9688005db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb5d8b58a75bf22c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:25.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.652+0000 7f96837fe640 1 -- 172.21.15.16:0/3479651021 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9688033070 con 0x55a7237d3d90 2024-03-28T22:17:25.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.652+0000 7f96837fe640 1 -- 172.21.15.16:0/3479651021 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9688012e50 con 0x55a7237d3d90 2024-03-28T22:17:25.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.652+0000 7f96837fe640 1 -- 172.21.15.16:0/3479651021 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f968802a070 con 0x55a7237d3d90 2024-03-28T22:17:25.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.653+0000 7f96a29ba740 1 -- 172.21.15.16:0/3479651021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7237d3d90 msgr2=0x55a7237c8bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.653+0000 7f96a29ba740 1 --2- 172.21.15.16:0/3479651021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7237d3d90 0x55a7237c8bd0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f9688001eb0 tx=0x7f9688005db0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.653+0000 7f96a29ba740 1 -- 172.21.15.16:0/3479651021 shutdown_connections 2024-03-28T22:17:25.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.653+0000 7f96a29ba740 1 --2- 172.21.15.16:0/3479651021 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7237dc970 0x55a7237e8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.653+0000 7f96a29ba740 1 --2- 172.21.15.16:0/3479651021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7237d3d90 0x55a7237c8bd0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.653+0000 7f96a29ba740 1 --2- 172.21.15.16:0/3479651021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a7237d3470 0x55a7237dc020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.653+0000 7f96a29ba740 1 -- 172.21.15.16:0/3479651021 >> 172.21.15.16:0/3479651021 conn(0x55a7237cdba0 msgr2=0x55a7237ccd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:25.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.653+0000 7f96a29ba740 1 -- 172.21.15.16:0/3479651021 shutdown_connections 2024-03-28T22:17:25.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.653+0000 7f96a29ba740 1 -- 172.21.15.16:0/3479651021 wait complete. 2024-03-28T22:17:25.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.653+0000 7f96a29ba740 1 -- start start 2024-03-28T22:17:25.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.653+0000 7f96a29ba740 1 Processor -- start 2024-03-28T22:17:25.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.654+0000 7f96a29ba740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7237d3470 0x55a7237d6c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.654+0000 7f96a29ba740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7237d3d90 0x55a7237d7150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.654+0000 7f96a29ba740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a7237dc970 0x55a723851930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.654+0000 7f96a29ba740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f96981ab970 con 0x55a7237d3d90 2024-03-28T22:17:25.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.654+0000 7f96a29ba740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f96981acde0 con 0x55a7237dc970 2024-03-28T22:17:25.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.654+0000 7f96a29ba740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f96981ac040 con 0x55a7237d3470 2024-03-28T22:17:25.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.654+0000 7f96a29ba740 10 client.0 _ll_register_callbacks cb 0x7f96880127f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:17:25.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.654+0000 7f969d296640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7237d3d90 0x55a7237d7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.654+0000 7f969d296640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7237d3d90 0x55a7237d7150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45530/0 (socket says 172.21.15.16:45530) 2024-03-28T22:17:25.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.654+0000 7f969d296640 1 -- 172.21.15.16:0/1202041253 learned_addr learned my addr 172.21.15.16:0/1202041253 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:25.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.654+0000 7f969e298640 1 --2- 172.21.15.16:0/1202041253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a7237dc970 0x55a723851930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.654+0000 7f969da97640 1 --2- 172.21.15.16:0/1202041253 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7237d3470 0x55a7237d6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.654+0000 7f969d296640 1 -- 172.21.15.16:0/1202041253 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7237d3470 msgr2=0x55a7237d6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:25.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.654+0000 7f969d296640 1 --2- 172.21.15.16:0/1202041253 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7237d3470 0x55a7237d6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.654+0000 7f969d296640 1 -- 172.21.15.16:0/1202041253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a7237dc970 msgr2=0x55a723851930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:25.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.654+0000 7f969d296640 1 --2- 172.21.15.16:0/1202041253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a7237dc970 0x55a723851930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.654+0000 7f969d296640 1 -- 172.21.15.16:0/1202041253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a723851ec0 con 0x55a7237d3d90 2024-03-28T22:17:25.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.654+0000 7f969e298640 1 --2- 172.21.15.16:0/1202041253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a7237dc970 0x55a723851930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:17:25.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.654+0000 7f969d296640 1 --2- 172.21.15.16:0/1202041253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7237d3d90 0x55a7237d7150 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f9688012420 tx=0x7f9688010660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=747ada810954ec03 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:25.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.654+0000 7f96817fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:17:25.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.655+0000 7f96817fa640 1 -- 172.21.15.16:0/1202041253 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9688033040 con 0x55a7237d3d90 2024-03-28T22:17:25.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.655+0000 7f96817fa640 1 -- 172.21.15.16:0/1202041253 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8527+0+0 (secure 0 0 0) 0x7f9688005040 con 0x55a7237d3d90 2024-03-28T22:17:25.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.655+0000 7f96a29ba740 1 -- 172.21.15.16:0/1202041253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9688012e50 con 0x55a7237d3d90 2024-03-28T22:17:25.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.655+0000 7f96817fa640 1 -- 172.21.15.16:0/1202041253 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f968802a040 con 0x55a7237d3d90 2024-03-28T22:17:25.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.655+0000 7f96a29ba740 10 client.16932 fetch_fsmap learned FSMap version 27 2024-03-28T22:17:25.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.655+0000 7f96a29ba740 1 -- 172.21.15.16:0/1202041253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a723853130 con 0x55a7237d3d90 2024-03-28T22:17:25.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.655+0000 7f96817fa640 1 -- 172.21.15.16:0/1202041253 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9688012e50 con 0x55a7237d3d90 2024-03-28T22:17:25.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.655+0000 7f96817fa640 1 -- 172.21.15.16:0/1202041253 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f968802ea20 con 0x55a7237d3d90 2024-03-28T22:17:25.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.655+0000 7f96a29ba740 10 client.16932 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:17:25.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.655+0000 7f96a29ba740 10 client.16932 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-28T22:17:25.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.655+0000 7f96a29ba740 20 client.16932 populate_metadata read hostname 'smithi016' 2024-03-28T22:17:25.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.655+0000 7f96a29ba740 1 --2- 172.21.15.16:0/1202041253 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55a723853790 0x55a723855bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.655+0000 7f96a29ba740 4 client.16932 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:17:25.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.655+0000 7f96a29ba740 1 -- 172.21.15.16:0/1202041253 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a72373ea40 con 0x55a723853790 2024-03-28T22:17:25.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.655+0000 7f969da97640 1 --2- 172.21.15.16:0/1202041253 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55a723853790 0x55a723855bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.656+0000 7f96817fa640 10 client.16932 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:17:25.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.656+0000 7f969da97640 1 --2- 172.21.15.16:0/1202041253 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55a723853790 0x55a723855bb0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5fb6d2bbb8b0871f server_cookie=29ba902e87b668c in_seq=0 out_seq=0 2024-03-28T22:17:25.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.656+0000 7f96817fa640 1 -- 172.21.15.16:0/1202041253 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a72373ea40 con 0x55a723853790 2024-03-28T22:17:25.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.656+0000 7f96817fa640 10 client.16932 handle_command_reply: tid=0 2024-03-28T22:17:25.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.656+0000 7f96a29ba740 1 client.16932 shutdown 2024-03-28T22:17:25.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.656+0000 7f96a29ba740 1 -- 172.21.15.16:0/1202041253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7237d3d90 msgr2=0x55a7237d7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.656+0000 7f96a29ba740 1 --2- 172.21.15.16:0/1202041253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7237d3d90 0x55a7237d7150 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f9688012420 tx=0x7f9688010660 comp rx=0 tx=0).stop 2024-03-28T22:17:25.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.656+0000 7f96a29ba740 1 -- 172.21.15.16:0/1202041253 shutdown_connections 2024-03-28T22:17:25.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.656+0000 7f96a29ba740 1 --2- 172.21.15.16:0/1202041253 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55a723853790 0x55a723855bb0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.656+0000 7f96a29ba740 1 --2- 172.21.15.16:0/1202041253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a7237dc970 0x55a723851930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.656+0000 7f96a29ba740 1 --2- 172.21.15.16:0/1202041253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a7237d3d90 0x55a7237d7150 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.656+0000 7f96a29ba740 1 --2- 172.21.15.16:0/1202041253 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7237d3470 0x55a7237d6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.656+0000 7f96a29ba740 1 -- 172.21.15.16:0/1202041253 >> 172.21.15.16:0/1202041253 conn(0x55a7237cdba0 msgr2=0x55a7237d9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:25.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.656+0000 7f96817fa640 0 client.16932 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:17:25.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.656+0000 7f96a29ba740 1 -- 172.21.15.16:0/1202041253 shutdown_connections 2024-03-28T22:17:25.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.656+0000 7f96a29ba740 1 -- 172.21.15.16:0/1202041253 wait complete. 2024-03-28T22:17:25.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.656+0000 7f96a29ba740 20 client.16932 trim_cache size 0 max 16384 2024-03-28T22:17:25.669 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:17:25.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.657+0000 7f969fcfb640 1 -- 172.21.15.16:0/2675929615 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f96840a1f70 msgr2=0x7f96840a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.657+0000 7f969fcfb640 1 --2- 172.21.15.16:0/2675929615 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f96840a1f70 0x7f96840a4430 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9694005fd0 tx=0x7f969400b5b0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.657+0000 7f969fcfb640 1 -- 172.21.15.16:0/2675929615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9698106690 msgr2=0x7f969819c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.657+0000 7f969fcfb640 1 --2- 172.21.15.16:0/2675929615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9698106690 0x7f969819c790 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f96900348f0 tx=0x7f9690032c90 comp rx=0 tx=0).stop 2024-03-28T22:17:25.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.657+0000 7f969fcfb640 1 -- 172.21.15.16:0/2675929615 shutdown_connections 2024-03-28T22:17:25.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.657+0000 7f969fcfb640 1 --2- 172.21.15.16:0/2675929615 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f96840a1f70 0x7f96840a4430 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.657+0000 7f969fcfb640 1 --2- 172.21.15.16:0/2675929615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9698106690 0x7f969819c790 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.657+0000 7f969fcfb640 1 --2- 172.21.15.16:0/2675929615 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9698100fb0 0x7f9698198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.657+0000 7f969fcfb640 1 --2- 172.21.15.16:0/2675929615 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96981005f0 0x7f9698197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.657+0000 7f969fcfb640 1 -- 172.21.15.16:0/2675929615 >> 172.21.15.16:0/2675929615 conn(0x7f9698077800 msgr2=0x7f96980ff100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:25.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.657+0000 7f969fcfb640 1 -- 172.21.15.16:0/2675929615 shutdown_connections 2024-03-28T22:17:25.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.657+0000 7f969fcfb640 1 -- 172.21.15.16:0/2675929615 wait complete. 2024-03-28T22:17:25.672 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:17:25.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.775+0000 7f9d588df640 1 -- 172.21.15.16:0/2240286818 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d54102960 msgr2=0x7f9d54102d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.775+0000 7f9d588df640 1 --2- 172.21.15.16:0/2240286818 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d54102960 0x7f9d54102d40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c009890 tx=0x7f9d4c02bb70 comp rx=0 tx=0).stop 2024-03-28T22:17:25.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.776+0000 7f9d588df640 1 -- 172.21.15.16:0/2240286818 shutdown_connections 2024-03-28T22:17:25.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.776+0000 7f9d588df640 1 --2- 172.21.15.16:0/2240286818 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d54107e50 0x7f9d5410e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.776+0000 7f9d588df640 1 --2- 172.21.15.16:0/2240286818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d54103310 0x7f9d54107740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.776+0000 7f9d588df640 1 --2- 172.21.15.16:0/2240286818 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d54102960 0x7f9d54102d40 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.776+0000 7f9d588df640 1 -- 172.21.15.16:0/2240286818 >> 172.21.15.16:0/2240286818 conn(0x7f9d54077830 msgr2=0x7f9d54077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:25.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.776+0000 7f9d588df640 1 -- 172.21.15.16:0/2240286818 shutdown_connections 2024-03-28T22:17:25.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.776+0000 7f9d588df640 1 -- 172.21.15.16:0/2240286818 wait complete. 2024-03-28T22:17:25.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.776+0000 7f9d588df640 1 Processor -- start 2024-03-28T22:17:25.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.776+0000 7f9d588df640 1 -- start start 2024-03-28T22:17:25.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.776+0000 7f9d588df640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d54102960 0x7f9d54198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.776+0000 7f9d588df640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d54107e50 0x7f9d541986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.777+0000 7f9d588df640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d5419ca30 0x7f9d5419cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.777+0000 7f9d588df640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9d54111c50 con 0x7f9d54102960 2024-03-28T22:17:25.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.777+0000 7f9d588df640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9d54111ad0 con 0x7f9d54107e50 2024-03-28T22:17:25.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.777+0000 7f9d588df640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9d54111dd0 con 0x7f9d5419ca30 2024-03-28T22:17:25.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.777+0000 7f9d5259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d54102960 0x7f9d54198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.777+0000 7f9d5259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d54102960 0x7f9d54198170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45554/0 (socket says 172.21.15.16:45554) 2024-03-28T22:17:25.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.777+0000 7f9d5259c640 1 -- 172.21.15.16:0/1267221368 learned_addr learned my addr 172.21.15.16:0/1267221368 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:25.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.777+0000 7f9d51d9b640 1 --2- 172.21.15.16:0/1267221368 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d54107e50 0x7f9d541986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.777+0000 7f9d52d9d640 1 --2- 172.21.15.16:0/1267221368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d5419ca30 0x7f9d5419cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.777+0000 7f9d5259c640 1 -- 172.21.15.16:0/1267221368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d5419ca30 msgr2=0x7f9d5419cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.777+0000 7f9d5259c640 1 --2- 172.21.15.16:0/1267221368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d5419ca30 0x7f9d5419cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.777+0000 7f9d5259c640 1 -- 172.21.15.16:0/1267221368 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d54107e50 msgr2=0x7f9d541986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.777+0000 7f9d5259c640 1 --2- 172.21.15.16:0/1267221368 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d54107e50 0x7f9d541986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.777+0000 7f9d5259c640 1 -- 172.21.15.16:0/1267221368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d5419d520 con 0x7f9d54102960 2024-03-28T22:17:25.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.777+0000 7f9d51d9b640 1 --2- 172.21.15.16:0/1267221368 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d54107e50 0x7f9d541986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:17:25.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.777+0000 7f9d52d9d640 1 --2- 172.21.15.16:0/1267221368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d5419ca30 0x7f9d5419cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:17:25.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.777+0000 7f9d5259c640 1 --2- 172.21.15.16:0/1267221368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d54102960 0x7f9d54198170 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c02c180 tx=0x7f9d4c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:25.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.777+0000 7f9d4b7fe640 1 -- 172.21.15.16:0/1267221368 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c041020 con 0x7f9d54102960 2024-03-28T22:17:25.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.777+0000 7f9d588df640 1 -- 172.21.15.16:0/1267221368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d5419d7b0 con 0x7f9d54102960 2024-03-28T22:17:25.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.777+0000 7f9d4b7fe640 1 -- 172.21.15.16:0/1267221368 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d4c02c9f0 con 0x7f9d54102960 2024-03-28T22:17:25.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.777+0000 7f9d588df640 1 -- 172.21.15.16:0/1267221368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d541a4fd0 con 0x7f9d54102960 2024-03-28T22:17:25.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.777+0000 7f9d4b7fe640 1 -- 172.21.15.16:0/1267221368 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c039040 con 0x7f9d54102960 2024-03-28T22:17:25.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.779+0000 7f9d4b7fe640 1 -- 172.21.15.16:0/1267221368 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f9d4c044020 con 0x7f9d54102960 2024-03-28T22:17:25.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.780+0000 7f9d5b59e740 1 Processor -- start 2024-03-28T22:17:25.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.780+0000 7f9d5b59e740 1 -- start start 2024-03-28T22:17:25.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.780+0000 7f9d4b7fe640 1 --2- 172.21.15.16:0/1267221368 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f9d400a1ef0 0x7f9d400a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.780+0000 7f9d5b59e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a6739b2470 0x55a6739b2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.780+0000 7f9d5b59e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a6739b2d90 0x55a6739bb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.780+0000 7f9d5b59e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6739bb970 0x55a6739bdd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.780+0000 7f9d5b59e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a6739161b0 con 0x55a6739bb970 2024-03-28T22:17:25.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.780+0000 7f9d5b59e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55a6738bec00 con 0x55a6739b2470 2024-03-28T22:17:25.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.780+0000 7f9d5b59e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55a673912250 con 0x55a6739b2d90 2024-03-28T22:17:25.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.780+0000 7f9d5259c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a6739b2470 0x55a6739b2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.780+0000 7f9d52d9d640 1 --2- 172.21.15.16:0/1267221368 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f9d400a1ef0 0x7f9d400a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.780+0000 7f9d5259c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a6739b2470 0x55a6739b2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:52550/0 (socket says 172.21.15.16:52550) 2024-03-28T22:17:25.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.780+0000 7f9d5259c640 1 -- 172.21.15.16:0/2273947230 learned_addr learned my addr 172.21.15.16:0/2273947230 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:25.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.780+0000 7f9d4b7fe640 1 -- 172.21.15.16:0/1267221368 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8527+0+0 (secure 0 0 0) 0x7f9d4c0ef950 con 0x7f9d54102960 2024-03-28T22:17:25.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.780+0000 7f9d52d9d640 1 --2- 172.21.15.16:0/2273947230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6739bb970 0x55a6739bdd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.781+0000 7f9d51d9b640 1 --2- 172.21.15.16:0/2273947230 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a6739b2d90 0x55a6739bb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.781+0000 7f9d5259c640 1 -- 172.21.15.16:0/2273947230 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a6739b2d90 msgr2=0x55a6739bb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:25.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.781+0000 7f9d5259c640 1 --2- 172.21.15.16:0/2273947230 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a6739b2d90 0x55a6739bb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.781+0000 7f9d5259c640 1 -- 172.21.15.16:0/2273947230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6739bb970 msgr2=0x55a6739bdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:25.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.781+0000 7f9d5259c640 1 --2- 172.21.15.16:0/2273947230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6739bb970 0x55a6739bdd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.781+0000 7f9d5259c640 1 -- 172.21.15.16:0/2273947230 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6738de3d0 con 0x55a6739b2470 2024-03-28T22:17:25.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.781+0000 7f9d52d9d640 1 --2- 172.21.15.16:0/1267221368 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f9d400a1ef0 0x7f9d400a43b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9d44012990 tx=0x7f9d4400ac90 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:25.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.781+0000 7f9d5259c640 1 --2- 172.21.15.16:0/2273947230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a6739b2470 0x55a6739b2850 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c0b8850 tx=0x7f9d4c005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59544ec5f6af7976 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:25.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.781+0000 7f9d48ff9640 1 -- 172.21.15.16:0/2273947230 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c0bc1c0 con 0x55a6739b2470 2024-03-28T22:17:25.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.781+0000 7f9d48ff9640 1 -- 172.21.15.16:0/2273947230 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d4c0bcaf0 con 0x55a6739b2470 2024-03-28T22:17:25.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.781+0000 7f9d48ff9640 1 -- 172.21.15.16:0/2273947230 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c0d5120 con 0x55a6739b2470 2024-03-28T22:17:25.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.782+0000 7f9d5b59e740 1 -- 172.21.15.16:0/2273947230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a6739b2470 msgr2=0x55a6739b2850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.782+0000 7f9d5b59e740 1 --2- 172.21.15.16:0/2273947230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a6739b2470 0x55a6739b2850 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c0b8850 tx=0x7f9d4c005f90 comp rx=0 tx=0).stop 2024-03-28T22:17:25.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.782+0000 7f9d5b59e740 1 -- 172.21.15.16:0/2273947230 shutdown_connections 2024-03-28T22:17:25.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.782+0000 7f9d5b59e740 1 --2- 172.21.15.16:0/2273947230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6739bb970 0x55a6739bdd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.782+0000 7f9d5b59e740 1 --2- 172.21.15.16:0/2273947230 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a6739b2d90 0x55a6739bb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.782+0000 7f9d5b59e740 1 --2- 172.21.15.16:0/2273947230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a6739b2470 0x55a6739b2850 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.782+0000 7f9d5b59e740 1 -- 172.21.15.16:0/2273947230 >> 172.21.15.16:0/2273947230 conn(0x55a6739acba0 msgr2=0x55a6739acf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:25.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.782+0000 7f9d5b59e740 1 -- 172.21.15.16:0/2273947230 shutdown_connections 2024-03-28T22:17:25.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.782+0000 7f9d5b59e740 1 -- 172.21.15.16:0/2273947230 wait complete. 2024-03-28T22:17:25.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.782+0000 7f9d5b59e740 1 -- start start 2024-03-28T22:17:25.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.782+0000 7f9d5b59e740 1 Processor -- start 2024-03-28T22:17:25.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.782+0000 7f9d5b59e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a6739b2470 0x55a6739b89f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.782+0000 7f9d5b59e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a6739b2d90 0x55a6739b8f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.782+0000 7f9d5b59e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6739bb970 0x55a6739b9470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.782+0000 7f9d5b59e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a6739be520 con 0x55a6739bb970 2024-03-28T22:17:25.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.782+0000 7f9d5b59e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55a6738b4470 con 0x55a6739b2d90 2024-03-28T22:17:25.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.782+0000 7f9d5b59e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55a6739be6a0 con 0x55a6739b2470 2024-03-28T22:17:25.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.782+0000 7f9d5b59e740 10 client.0 _ll_register_callbacks cb 0x55a6738d1cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:17:25.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.783+0000 7f9d52d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6739bb970 0x55a6739b9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.783+0000 7f9d51d9b640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a6739b2d90 0x55a6739b8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.783+0000 7f9d52d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6739bb970 0x55a6739b9470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45576/0 (socket says 172.21.15.16:45576) 2024-03-28T22:17:25.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.783+0000 7f9d52d9d640 1 -- 172.21.15.16:0/2387652552 learned_addr learned my addr 172.21.15.16:0/2387652552 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:25.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.783+0000 7f9d5259c640 1 --2- 172.21.15.16:0/2387652552 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a6739b2470 0x55a6739b89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.783+0000 7f9d52d9d640 1 -- 172.21.15.16:0/2387652552 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a6739b2470 msgr2=0x55a6739b89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:25.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.783+0000 7f9d52d9d640 1 --2- 172.21.15.16:0/2387652552 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a6739b2470 0x55a6739b89f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.783+0000 7f9d52d9d640 1 -- 172.21.15.16:0/2387652552 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a6739b2d90 msgr2=0x55a6739b8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:25.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.783+0000 7f9d52d9d640 1 --2- 172.21.15.16:0/2387652552 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a6739b2d90 0x55a6739b8f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.783+0000 7f9d52d9d640 1 -- 172.21.15.16:0/2387652552 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a67362a3f0 con 0x55a6739bb970 2024-03-28T22:17:25.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.783+0000 7f9d51d9b640 1 --2- 172.21.15.16:0/2387652552 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a6739b2d90 0x55a6739b8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:17:25.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.783+0000 7f9d52d9d640 1 --2- 172.21.15.16:0/2387652552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6739bb970 0x55a6739b9470 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f9d4400a580 tx=0x7f9d44005c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fad671543e22ba3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:25.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.783+0000 7f9d39ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:17:25.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.783+0000 7f9d39ffb640 1 -- 172.21.15.16:0/2387652552 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d44023070 con 0x55a6739bb970 2024-03-28T22:17:25.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.783+0000 7f9d39ffb640 1 -- 172.21.15.16:0/2387652552 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8527+0+0 (secure 0 0 0) 0x7f9d44011b70 con 0x55a6739bb970 2024-03-28T22:17:25.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.783+0000 7f9d5b59e740 1 -- 172.21.15.16:0/2387652552 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d4c0bcaf0 con 0x55a6739bb970 2024-03-28T22:17:25.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.783+0000 7f9d39ffb640 1 -- 172.21.15.16:0/2387652552 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9d44011d70 con 0x55a6739bb970 2024-03-28T22:17:25.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.783+0000 7f9d5b59e740 10 client.16944 fetch_fsmap learned FSMap version 27 2024-03-28T22:17:25.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.783+0000 7f9d5b59e740 1 -- 172.21.15.16:0/2387652552 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a6736f47d0 con 0x55a6739bb970 2024-03-28T22:17:25.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.784+0000 7f9d39ffb640 1 -- 172.21.15.16:0/2387652552 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d4c0bcaf0 con 0x55a6739bb970 2024-03-28T22:17:25.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.784+0000 7f9d39ffb640 1 -- 172.21.15.16:0/2387652552 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f9d440286c0 con 0x55a6739bb970 2024-03-28T22:17:25.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.784+0000 7f9d5b59e740 10 client.16944 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:17:25.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.784+0000 7f9d5b59e740 10 client.16944 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T22:17:25.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.784+0000 7f9d5b59e740 20 client.16944 populate_metadata read hostname 'smithi016' 2024-03-28T22:17:25.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.784+0000 7f9d5b59e740 1 --2- 172.21.15.16:0/2387652552 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55a6739cb680 0x55a6739cdae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.784+0000 7f9d5b59e740 4 client.16944 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:17:25.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.784+0000 7f9d5b59e740 1 -- 172.21.15.16:0/2387652552 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a6739236f0 con 0x55a6739cb680 2024-03-28T22:17:25.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.784+0000 7f9d5259c640 1 --2- 172.21.15.16:0/2387652552 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55a6739cb680 0x55a6739cdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.784+0000 7f9d5259c640 1 --2- 172.21.15.16:0/2387652552 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55a6739cb680 0x55a6739cdae0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=71dc91c4485b6d57 server_cookie=7da2af9810e394b3 in_seq=0 out_seq=0 2024-03-28T22:17:25.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.784+0000 7f9d39ffb640 10 client.16944 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:17:25.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.785+0000 7f9d39ffb640 1 -- 172.21.15.16:0/2387652552 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a6739236f0 con 0x55a6739cb680 2024-03-28T22:17:25.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.785+0000 7f9d39ffb640 10 client.16944 handle_command_reply: tid=0 2024-03-28T22:17:25.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.785+0000 7f9d5b59e740 1 client.16944 shutdown 2024-03-28T22:17:25.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.785+0000 7f9d5b59e740 1 -- 172.21.15.16:0/2387652552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6739bb970 msgr2=0x55a6739b9470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.785+0000 7f9d5b59e740 1 --2- 172.21.15.16:0/2387652552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6739bb970 0x55a6739b9470 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f9d4400a580 tx=0x7f9d44005c80 comp rx=0 tx=0).stop 2024-03-28T22:17:25.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.785+0000 7f9d5b59e740 1 -- 172.21.15.16:0/2387652552 shutdown_connections 2024-03-28T22:17:25.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.785+0000 7f9d5b59e740 1 --2- 172.21.15.16:0/2387652552 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55a6739cb680 0x55a6739cdae0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.785+0000 7f9d5b59e740 1 --2- 172.21.15.16:0/2387652552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6739bb970 0x55a6739b9470 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.785+0000 7f9d5b59e740 1 --2- 172.21.15.16:0/2387652552 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a6739b2d90 0x55a6739b8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.785+0000 7f9d5b59e740 1 --2- 172.21.15.16:0/2387652552 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a6739b2470 0x55a6739b89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.785+0000 7f9d39ffb640 0 client.16944 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:17:25.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.785+0000 7f9d5b59e740 1 -- 172.21.15.16:0/2387652552 >> 172.21.15.16:0/2387652552 conn(0x55a6739acba0 msgr2=0x55a6739ae8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:25.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.785+0000 7f9d5b59e740 1 -- 172.21.15.16:0/2387652552 shutdown_connections 2024-03-28T22:17:25.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.785+0000 7f9d5b59e740 1 -- 172.21.15.16:0/2387652552 wait complete. 2024-03-28T22:17:25.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.785+0000 7f9d5b59e740 20 client.16944 trim_cache size 0 max 16384 2024-03-28T22:17:25.804 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:25 smithi053 ceph-mon[35979]: pgmap v210: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T22:17:25.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.798+0000 7f9d5b59e740 1 Processor -- start 2024-03-28T22:17:25.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.798+0000 7f9d5b59e740 1 -- start start 2024-03-28T22:17:25.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.799+0000 7f9d5b59e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6739b2470 0x55a6739c71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.799+0000 7f9d5b59e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a6739b2d90 0x55a6739d33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.799+0000 7f9d5b59e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a6739bb970 0x55a6739c37c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.799+0000 7f9d5b59e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9d541afe20 con 0x55a6739b2470 2024-03-28T22:17:25.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.799+0000 7f9d5b59e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9d541ad6f0 con 0x55a6739b2d90 2024-03-28T22:17:25.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.799+0000 7f9d5b59e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9d541ab560 con 0x55a6739bb970 2024-03-28T22:17:25.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.799+0000 7f9d5259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6739b2470 0x55a6739c71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.799+0000 7f9d5259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6739b2470 0x55a6739c71a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45586/0 (socket says 172.21.15.16:45586) 2024-03-28T22:17:25.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.799+0000 7f9d5259c640 1 -- 172.21.15.16:0/1238963555 learned_addr learned my addr 172.21.15.16:0/1238963555 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:25.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.799+0000 7f9d51d9b640 1 --2- 172.21.15.16:0/1238963555 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a6739b2d90 0x55a6739d33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.799+0000 7f9d52d9d640 1 --2- 172.21.15.16:0/1238963555 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a6739bb970 0x55a6739c37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.799+0000 7f9d5259c640 1 -- 172.21.15.16:0/1238963555 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a6739bb970 msgr2=0x55a6739c37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:25.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.799+0000 7f9d5259c640 1 --2- 172.21.15.16:0/1238963555 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a6739bb970 0x55a6739c37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.799+0000 7f9d5259c640 1 -- 172.21.15.16:0/1238963555 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a6739b2d90 msgr2=0x55a6739d33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:25.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.799+0000 7f9d5259c640 1 --2- 172.21.15.16:0/1238963555 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a6739b2d90 0x55a6739d33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.799+0000 7f9d5259c640 1 -- 172.21.15.16:0/1238963555 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6739b2bc0 con 0x55a6739b2470 2024-03-28T22:17:25.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.799+0000 7f9d52d9d640 1 --2- 172.21.15.16:0/1238963555 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a6739bb970 0x55a6739c37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:17:25.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.799+0000 7f9d5259c640 1 --2- 172.21.15.16:0/1238963555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6739b2470 0x55a6739c71a0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c0bc0f0 tx=0x7f9d4c02bb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4087eaceb472d3fb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:25.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.799+0000 7f9d3bfff640 1 -- 172.21.15.16:0/1238963555 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c0d5300 con 0x55a6739b2470 2024-03-28T22:17:25.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.799+0000 7f9d3bfff640 1 -- 172.21.15.16:0/1238963555 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d4c0d5120 con 0x55a6739b2470 2024-03-28T22:17:25.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.799+0000 7f9d3bfff640 1 -- 172.21.15.16:0/1238963555 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c0bbd50 con 0x55a6739b2470 2024-03-28T22:17:25.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.800+0000 7f9d5b59e740 1 -- 172.21.15.16:0/1238963555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6739b2470 msgr2=0x55a6739c71a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.800+0000 7f9d5b59e740 1 --2- 172.21.15.16:0/1238963555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6739b2470 0x55a6739c71a0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c0bc0f0 tx=0x7f9d4c02bb70 comp rx=0 tx=0).stop 2024-03-28T22:17:25.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.800+0000 7f9d5b59e740 1 -- 172.21.15.16:0/1238963555 shutdown_connections 2024-03-28T22:17:25.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.800+0000 7f9d5b59e740 1 --2- 172.21.15.16:0/1238963555 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a6739bb970 0x55a6739c37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.800+0000 7f9d5b59e740 1 --2- 172.21.15.16:0/1238963555 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a6739b2d90 0x55a6739d33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.800+0000 7f9d5b59e740 1 --2- 172.21.15.16:0/1238963555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6739b2470 0x55a6739c71a0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.800+0000 7f9d5b59e740 1 -- 172.21.15.16:0/1238963555 >> 172.21.15.16:0/1238963555 conn(0x55a6739acba0 msgr2=0x55a6739b89b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:25.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.800+0000 7f9d5b59e740 1 -- 172.21.15.16:0/1238963555 shutdown_connections 2024-03-28T22:17:25.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.800+0000 7f9d5b59e740 1 -- 172.21.15.16:0/1238963555 wait complete. 2024-03-28T22:17:25.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.800+0000 7f9d5b59e740 1 -- start start 2024-03-28T22:17:25.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.800+0000 7f9d5b59e740 1 Processor -- start 2024-03-28T22:17:25.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.800+0000 7f9d5b59e740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a6739b2470 0x55a6739c39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.800+0000 7f9d5b59e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6739b2d90 0x55a6739c07a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.800+0000 7f9d5b59e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a6739bb970 0x55a6739c0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.800+0000 7f9d5b59e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9d541abbb0 con 0x55a6739b2d90 2024-03-28T22:17:25.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.800+0000 7f9d5b59e740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9d541ad020 con 0x55a6739b2470 2024-03-28T22:17:25.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.800+0000 7f9d5b59e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9d541ac280 con 0x55a6739bb970 2024-03-28T22:17:25.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.800+0000 7f9d5b59e740 10 client.0 _ll_register_callbacks cb 0x7f9d44012270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:17:25.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.800+0000 7f9d51d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6739b2d90 0x55a6739c07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.801+0000 7f9d51d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6739b2d90 0x55a6739c07a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45598/0 (socket says 172.21.15.16:45598) 2024-03-28T22:17:25.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.801+0000 7f9d51d9b640 1 -- 172.21.15.16:0/780032360 learned_addr learned my addr 172.21.15.16:0/780032360 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:25.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.801+0000 7f9d5259c640 1 --2- 172.21.15.16:0/780032360 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a6739b2470 0x55a6739c39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.801+0000 7f9d52d9d640 1 --2- 172.21.15.16:0/780032360 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a6739bb970 0x55a6739c0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.801+0000 7f9d51d9b640 1 -- 172.21.15.16:0/780032360 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a6739bb970 msgr2=0x55a6739c0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:25.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.801+0000 7f9d51d9b640 1 --2- 172.21.15.16:0/780032360 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a6739bb970 0x55a6739c0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.801+0000 7f9d51d9b640 1 -- 172.21.15.16:0/780032360 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a6739b2470 msgr2=0x55a6739c39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:25.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.801+0000 7f9d51d9b640 1 --2- 172.21.15.16:0/780032360 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a6739b2470 0x55a6739c39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.801+0000 7f9d51d9b640 1 -- 172.21.15.16:0/780032360 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6739c1320 con 0x55a6739b2d90 2024-03-28T22:17:25.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.801+0000 7f9d51d9b640 1 --2- 172.21.15.16:0/780032360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6739b2d90 0x55a6739c07a0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f9d54069890 tx=0x7f9d3c00f850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0d67e8a40c3e40f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:25.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.801+0000 7f9d39ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:17:25.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.801+0000 7f9d39ffb640 1 -- 172.21.15.16:0/780032360 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c004250 con 0x55a6739b2d90 2024-03-28T22:17:25.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.801+0000 7f9d39ffb640 1 -- 172.21.15.16:0/780032360 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8527+0+0 (secure 0 0 0) 0x7f9d3c0043f0 con 0x55a6739b2d90 2024-03-28T22:17:25.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.801+0000 7f9d5b59e740 1 -- 172.21.15.16:0/780032360 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d4c0d5120 con 0x55a6739b2d90 2024-03-28T22:17:25.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.801+0000 7f9d39ffb640 1 -- 172.21.15.16:0/780032360 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9d3c02b050 con 0x55a6739b2d90 2024-03-28T22:17:25.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.801+0000 7f9d5b59e740 10 client.16950 fetch_fsmap learned FSMap version 27 2024-03-28T22:17:25.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.801+0000 7f9d5b59e740 1 -- 172.21.15.16:0/780032360 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a673a36b40 con 0x55a6739b2d90 2024-03-28T22:17:25.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.801+0000 7f9d39ffb640 1 -- 172.21.15.16:0/780032360 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d4c0d5120 con 0x55a6739b2d90 2024-03-28T22:17:25.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.802+0000 7f9d39ffb640 1 -- 172.21.15.16:0/780032360 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f9d3c01d680 con 0x55a6739b2d90 2024-03-28T22:17:25.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.802+0000 7f9d5b59e740 10 client.16950 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:17:25.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.802+0000 7f9d5b59e740 10 client.16950 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T22:17:25.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.802+0000 7f9d5b59e740 20 client.16950 populate_metadata read hostname 'smithi016' 2024-03-28T22:17:25.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.802+0000 7f9d5b59e740 1 --2- 172.21.15.16:0/780032360 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55a673a372e0 0x55a673a39700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.802+0000 7f9d5b59e740 4 client.16950 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:17:25.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.802+0000 7f9d5b59e740 1 -- 172.21.15.16:0/780032360 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a67391da40 con 0x55a673a372e0 2024-03-28T22:17:25.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.802+0000 7f9d5259c640 1 --2- 172.21.15.16:0/780032360 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55a673a372e0 0x55a673a39700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.802+0000 7f9d5259c640 1 --2- 172.21.15.16:0/780032360 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55a673a372e0 0x55a673a39700 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=85292fe8916e26d7 server_cookie=d39e4a502bcbd4ae in_seq=0 out_seq=0 2024-03-28T22:17:25.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.802+0000 7f9d39ffb640 10 client.16950 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:17:25.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.802+0000 7f9d39ffb640 1 -- 172.21.15.16:0/780032360 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a67391da40 con 0x55a673a372e0 2024-03-28T22:17:25.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.802+0000 7f9d39ffb640 10 client.16950 handle_command_reply: tid=0 2024-03-28T22:17:25.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.802+0000 7f9d5b59e740 1 client.16950 shutdown 2024-03-28T22:17:25.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.802+0000 7f9d5b59e740 1 -- 172.21.15.16:0/780032360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6739b2d90 msgr2=0x55a6739c07a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.802+0000 7f9d5b59e740 1 --2- 172.21.15.16:0/780032360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6739b2d90 0x55a6739c07a0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f9d54069890 tx=0x7f9d3c00f850 comp rx=0 tx=0).stop 2024-03-28T22:17:25.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.802+0000 7f9d5b59e740 1 -- 172.21.15.16:0/780032360 shutdown_connections 2024-03-28T22:17:25.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.802+0000 7f9d5b59e740 1 --2- 172.21.15.16:0/780032360 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55a673a372e0 0x55a673a39700 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.802+0000 7f9d5b59e740 1 --2- 172.21.15.16:0/780032360 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a6739bb970 0x55a6739c0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.802+0000 7f9d5b59e740 1 --2- 172.21.15.16:0/780032360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a6739b2d90 0x55a6739c07a0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.802+0000 7f9d5b59e740 1 --2- 172.21.15.16:0/780032360 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a6739b2470 0x55a6739c39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.802+0000 7f9d39ffb640 0 client.16950 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:17:25.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.802+0000 7f9d5b59e740 1 -- 172.21.15.16:0/780032360 >> 172.21.15.16:0/780032360 conn(0x55a6739acba0 msgr2=0x55a6739b89b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:25.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.802+0000 7f9d5b59e740 1 -- 172.21.15.16:0/780032360 shutdown_connections 2024-03-28T22:17:25.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.803+0000 7f9d5b59e740 1 -- 172.21.15.16:0/780032360 wait complete. 2024-03-28T22:17:25.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.803+0000 7f9d5b59e740 20 client.16950 trim_cache size 0 max 16384 2024-03-28T22:17:25.820 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:17:25.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.803+0000 7f9d588df640 1 -- 172.21.15.16:0/1267221368 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f9d400a1ef0 msgr2=0x7f9d400a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.803+0000 7f9d588df640 1 --2- 172.21.15.16:0/1267221368 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f9d400a1ef0 0x7f9d400a43b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9d44012990 tx=0x7f9d4400ac90 comp rx=0 tx=0).stop 2024-03-28T22:17:25.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.803+0000 7f9d588df640 1 -- 172.21.15.16:0/1267221368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d54102960 msgr2=0x7f9d54198170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.803+0000 7f9d588df640 1 --2- 172.21.15.16:0/1267221368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d54102960 0x7f9d54198170 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c02c180 tx=0x7f9d4c004600 comp rx=0 tx=0).stop 2024-03-28T22:17:25.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.803+0000 7f9d588df640 1 -- 172.21.15.16:0/1267221368 shutdown_connections 2024-03-28T22:17:25.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.803+0000 7f9d588df640 1 --2- 172.21.15.16:0/1267221368 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f9d400a1ef0 0x7f9d400a43b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.803+0000 7f9d588df640 1 --2- 172.21.15.16:0/1267221368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d5419ca30 0x7f9d5419cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.803+0000 7f9d588df640 1 --2- 172.21.15.16:0/1267221368 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d54107e50 0x7f9d541986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.803+0000 7f9d588df640 1 --2- 172.21.15.16:0/1267221368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d54102960 0x7f9d54198170 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.803+0000 7f9d588df640 1 -- 172.21.15.16:0/1267221368 >> 172.21.15.16:0/1267221368 conn(0x7f9d54077830 msgr2=0x7f9d54105fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:25.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.803+0000 7f9d588df640 1 -- 172.21.15.16:0/1267221368 shutdown_connections 2024-03-28T22:17:25.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.803+0000 7f9d588df640 1 -- 172.21.15.16:0/1267221368 wait complete. 2024-03-28T22:17:25.823 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:25 smithi016 ceph-mon[32374]: pgmap v210: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T22:17:25.823 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:17:25.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.927+0000 7f4aaac54640 1 -- 172.21.15.16:0/4073464858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aa410bd30 msgr2=0x7f4aa410e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.927+0000 7f4aaac54640 1 --2- 172.21.15.16:0/4073464858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aa410bd30 0x7f4aa410e180 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c009ca0 tx=0x7f4a9c02c180 comp rx=0 tx=0).stop 2024-03-28T22:17:25.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.927+0000 7f4aaac54640 1 -- 172.21.15.16:0/4073464858 shutdown_connections 2024-03-28T22:17:25.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.927+0000 7f4aaac54640 1 --2- 172.21.15.16:0/4073464858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aa410bd30 0x7f4aa410e180 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.927+0000 7f4aaac54640 1 --2- 172.21.15.16:0/4073464858 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4aa4103280 0x7f4aa410b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.927+0000 7f4aaac54640 1 --2- 172.21.15.16:0/4073464858 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4aa4102960 0x7f4aa4102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.927+0000 7f4aaac54640 1 -- 172.21.15.16:0/4073464858 >> 172.21.15.16:0/4073464858 conn(0x7f4aa4077820 msgr2=0x7f4aa4077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:25.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.927+0000 7f4aaac54640 1 -- 172.21.15.16:0/4073464858 shutdown_connections 2024-03-28T22:17:25.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.927+0000 7f4aaac54640 1 -- 172.21.15.16:0/4073464858 wait complete. 2024-03-28T22:17:25.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.928+0000 7f4aaac54640 1 Processor -- start 2024-03-28T22:17:25.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.928+0000 7f4aaac54640 1 -- start start 2024-03-28T22:17:25.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.928+0000 7f4aaac54640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4aa4102960 0x7f4aa4197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.928+0000 7f4aaac54640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4aa4103280 0x7f4aa41984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.928+0000 7f4aaac54640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aa410bd30 0x7f4aa419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.928+0000 7f4aaac54640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa4111ae0 con 0x7f4aa410bd30 2024-03-28T22:17:25.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.928+0000 7f4aaac54640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa4111960 con 0x7f4aa4102960 2024-03-28T22:17:25.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.928+0000 7f4aaac54640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa4111c60 con 0x7f4aa4103280 2024-03-28T22:17:25.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.928+0000 7f4aa91f1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aa410bd30 0x7f4aa419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.928+0000 7f4aa91f1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aa410bd30 0x7f4aa419c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45614/0 (socket says 172.21.15.16:45614) 2024-03-28T22:17:25.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.928+0000 7f4aa91f1640 1 -- 172.21.15.16:0/2459202592 learned_addr learned my addr 172.21.15.16:0/2459202592 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:25.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.928+0000 7f4aa89f0640 1 --2- 172.21.15.16:0/2459202592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4aa4102960 0x7f4aa4197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.928+0000 7f4aa3fff640 1 --2- 172.21.15.16:0/2459202592 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4aa4103280 0x7f4aa41984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.928+0000 7f4aa91f1640 1 -- 172.21.15.16:0/2459202592 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4aa4103280 msgr2=0x7f4aa41984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.928+0000 7f4aa91f1640 1 --2- 172.21.15.16:0/2459202592 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4aa4103280 0x7f4aa41984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.928+0000 7f4aa91f1640 1 -- 172.21.15.16:0/2459202592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4aa4102960 msgr2=0x7f4aa4197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.928+0000 7f4aa91f1640 1 --2- 172.21.15.16:0/2459202592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4aa4102960 0x7f4aa4197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.928+0000 7f4aa91f1640 1 -- 172.21.15.16:0/2459202592 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aa419cf50 con 0x7f4aa410bd30 2024-03-28T22:17:25.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.928+0000 7f4aa3fff640 1 --2- 172.21.15.16:0/2459202592 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4aa4103280 0x7f4aa41984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:17:25.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.928+0000 7f4aa91f1640 1 --2- 172.21.15.16:0/2459202592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aa410bd30 0x7f4aa419c820 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c02c790 tx=0x7f4a9c035e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:25.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.929+0000 7f4aa1ffb640 1 -- 172.21.15.16:0/2459202592 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c039040 con 0x7f4aa410bd30 2024-03-28T22:17:25.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.929+0000 7f4aa1ffb640 1 -- 172.21.15.16:0/2459202592 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a9c044070 con 0x7f4aa410bd30 2024-03-28T22:17:25.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.929+0000 7f4aa1ffb640 1 -- 172.21.15.16:0/2459202592 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c032940 con 0x7f4aa410bd30 2024-03-28T22:17:25.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.929+0000 7f4aaac54640 1 -- 172.21.15.16:0/2459202592 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4aa419d1e0 con 0x7f4aa410bd30 2024-03-28T22:17:25.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.929+0000 7f4aaac54640 1 -- 172.21.15.16:0/2459202592 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4aa41a4ad0 con 0x7f4aa410bd30 2024-03-28T22:17:25.935 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:25 smithi063 ceph-mon[36249]: pgmap v210: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T22:17:25.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.932+0000 7f4aa1ffb640 1 -- 172.21.15.16:0/2459202592 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f4a9c032ae0 con 0x7f4aa410bd30 2024-03-28T22:17:25.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.932+0000 7f4aad830740 1 Processor -- start 2024-03-28T22:17:25.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.932+0000 7f4aad830740 1 -- start start 2024-03-28T22:17:25.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.932+0000 7f4aad830740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561f10f9f470 0x561f10f9f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.932+0000 7f4aad830740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f10f9fd90 0x561f10fa8430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.932+0000 7f4aad830740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f10fa8970 0x561f10faad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.932+0000 7f4aad830740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561f10f031b0 con 0x561f10f9fd90 2024-03-28T22:17:25.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.932+0000 7f4aad830740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x561f10eabc00 con 0x561f10f9f470 2024-03-28T22:17:25.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.932+0000 7f4aad830740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x561f10eff250 con 0x561f10fa8970 2024-03-28T22:17:25.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.932+0000 7f4aa89f0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f10f9fd90 0x561f10fa8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.932+0000 7f4aa3fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561f10f9f470 0x561f10f9f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.932+0000 7f4aa89f0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f10f9fd90 0x561f10fa8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45622/0 (socket says 172.21.15.16:45622) 2024-03-28T22:17:25.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.932+0000 7f4aa89f0640 1 -- 172.21.15.16:0/2520428478 learned_addr learned my addr 172.21.15.16:0/2520428478 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:25.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.932+0000 7f4aa1ffb640 1 --2- 172.21.15.16:0/2459202592 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4a900a1ef0 0x7f4a900a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.932+0000 7f4aa91f1640 1 --2- 172.21.15.16:0/2459202592 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4a900a1ef0 0x7f4a900a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.932+0000 7f4aa3fff640 1 --2- 172.21.15.16:0/2520428478 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f10fa8970 0x561f10faad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.932+0000 7f4aa89f0640 1 -- 172.21.15.16:0/2520428478 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f10fa8970 msgr2=0x561f10faad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:25.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.932+0000 7f4aa89f0640 1 --2- 172.21.15.16:0/2520428478 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f10fa8970 0x561f10faad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.932+0000 7f4aa89f0640 1 -- 172.21.15.16:0/2520428478 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561f10f9f470 msgr2=0x561f10f9f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:25.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.932+0000 7f4aa89f0640 1 --2- 172.21.15.16:0/2520428478 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561f10f9f470 0x561f10f9f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.932+0000 7f4aa89f0640 1 -- 172.21.15.16:0/2520428478 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f10ecb3d0 con 0x561f10f9fd90 2024-03-28T22:17:25.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.932+0000 7f4aa3fff640 1 --2- 172.21.15.16:0/2520428478 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561f10f9f470 0x561f10f9f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:17:25.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.932+0000 7f4aa1ffb640 1 -- 172.21.15.16:0/2459202592 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8527+0+0 (secure 0 0 0) 0x7f4a9c0f0620 con 0x7f4aa410bd30 2024-03-28T22:17:25.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.933+0000 7f4aa89f0640 1 --2- 172.21.15.16:0/2520428478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f10f9fd90 0x561f10fa8430 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c00bd10 tx=0x7f4a8c0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed075c71487250ee server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:25.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.933+0000 7f4a9affd640 1 -- 172.21.15.16:0/2520428478 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c004030 con 0x561f10f9fd90 2024-03-28T22:17:25.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.933+0000 7f4aa91f1640 1 --2- 172.21.15.16:0/2459202592 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4a900a1ef0 0x7f4a900a43b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c0ee370 tx=0x7f4a9c02c180 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:25.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.933+0000 7f4a9affd640 1 -- 172.21.15.16:0/2520428478 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a8c0041d0 con 0x561f10f9fd90 2024-03-28T22:17:25.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.933+0000 7f4a9affd640 1 -- 172.21.15.16:0/2520428478 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c0117e0 con 0x561f10f9fd90 2024-03-28T22:17:25.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.933+0000 7f4aad830740 1 -- 172.21.15.16:0/2520428478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f10f9fd90 msgr2=0x561f10fa8430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.933+0000 7f4aad830740 1 --2- 172.21.15.16:0/2520428478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f10f9fd90 0x561f10fa8430 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c00bd10 tx=0x7f4a8c0107a0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.933+0000 7f4aad830740 1 -- 172.21.15.16:0/2520428478 shutdown_connections 2024-03-28T22:17:25.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.934+0000 7f4aad830740 1 --2- 172.21.15.16:0/2520428478 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f10fa8970 0x561f10faad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.934+0000 7f4aad830740 1 --2- 172.21.15.16:0/2520428478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f10f9fd90 0x561f10fa8430 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.934+0000 7f4aad830740 1 --2- 172.21.15.16:0/2520428478 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561f10f9f470 0x561f10f9f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.934+0000 7f4aad830740 1 -- 172.21.15.16:0/2520428478 >> 172.21.15.16:0/2520428478 conn(0x561f10f99ba0 msgr2=0x561f10f99f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:25.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.934+0000 7f4aad830740 1 -- 172.21.15.16:0/2520428478 shutdown_connections 2024-03-28T22:17:25.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.934+0000 7f4aad830740 1 -- 172.21.15.16:0/2520428478 wait complete. 2024-03-28T22:17:25.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.934+0000 7f4aad830740 1 -- start start 2024-03-28T22:17:25.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.934+0000 7f4aad830740 1 Processor -- start 2024-03-28T22:17:25.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.934+0000 7f4aad830740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f10f9f470 0x561f10fa59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.934+0000 7f4aad830740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561f10f9fd90 0x561f10fa5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.934+0000 7f4aad830740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f10fa8970 0x561f10fa6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.934+0000 7f4aad830740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561f10fab760 con 0x561f10f9f470 2024-03-28T22:17:25.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.934+0000 7f4aad830740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x561f10ea1470 con 0x561f10f9fd90 2024-03-28T22:17:25.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.934+0000 7f4aad830740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x561f10fab8e0 con 0x561f10fa8970 2024-03-28T22:17:25.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.934+0000 7f4aad830740 10 client.0 _ll_register_callbacks cb 0x561f10ebecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:17:25.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.934+0000 7f4aa3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f10f9f470 0x561f10fa59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.934+0000 7f4aa3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f10f9f470 0x561f10fa59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45624/0 (socket says 172.21.15.16:45624) 2024-03-28T22:17:25.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.934+0000 7f4aa3fff640 1 -- 172.21.15.16:0/4030974288 learned_addr learned my addr 172.21.15.16:0/4030974288 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:25.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.935+0000 7f4aa89f0640 1 --2- 172.21.15.16:0/4030974288 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561f10f9fd90 0x561f10fa5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.935+0000 7f4aa3fff640 1 -- 172.21.15.16:0/4030974288 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f10fa8970 msgr2=0x561f10fa6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:25.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.935+0000 7f4aa3fff640 1 --2- 172.21.15.16:0/4030974288 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f10fa8970 0x561f10fa6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.935+0000 7f4aa3fff640 1 -- 172.21.15.16:0/4030974288 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561f10f9fd90 msgr2=0x561f10fa5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:25.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.935+0000 7f4aa3fff640 1 --2- 172.21.15.16:0/4030974288 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561f10f9fd90 0x561f10fa5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.935+0000 7f4aa3fff640 1 -- 172.21.15.16:0/4030974288 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f10c173f0 con 0x561f10f9f470 2024-03-28T22:17:25.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.935+0000 7f4aa89f0640 1 --2- 172.21.15.16:0/4030974288 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561f10f9fd90 0x561f10fa5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:17:25.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.935+0000 7f4aa3fff640 1 --2- 172.21.15.16:0/4030974288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f10f9f470 0x561f10fa59a0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f4a94009fb0 tx=0x7f4a94013c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=319a6ca8dc3e9a08 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:25.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.935+0000 7f4a83fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:17:25.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.935+0000 7f4a83fff640 1 -- 172.21.15.16:0/4030974288 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a94004460 con 0x561f10f9f470 2024-03-28T22:17:25.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.935+0000 7f4a83fff640 1 -- 172.21.15.16:0/4030974288 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8527+0+0 (secure 0 0 0) 0x7f4a9400f860 con 0x561f10f9f470 2024-03-28T22:17:25.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.935+0000 7f4aad830740 1 -- 172.21.15.16:0/4030974288 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a8c0041d0 con 0x561f10f9f470 2024-03-28T22:17:25.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.935+0000 7f4a83fff640 1 -- 172.21.15.16:0/4030974288 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4a94025050 con 0x561f10f9f470 2024-03-28T22:17:25.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.935+0000 7f4aad830740 10 client.16962 fetch_fsmap learned FSMap version 27 2024-03-28T22:17:25.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.935+0000 7f4aad830740 1 -- 172.21.15.16:0/4030974288 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561f10ce17d0 con 0x561f10f9f470 2024-03-28T22:17:25.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.935+0000 7f4a83fff640 1 -- 172.21.15.16:0/4030974288 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a8c0041d0 con 0x561f10f9f470 2024-03-28T22:17:25.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.936+0000 7f4a83fff640 1 -- 172.21.15.16:0/4030974288 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f4a9400f040 con 0x561f10f9f470 2024-03-28T22:17:25.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.936+0000 7f4aad830740 10 client.16962 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:17:25.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.936+0000 7f4aad830740 10 client.16962 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:17:25.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.936+0000 7f4aad830740 20 client.16962 populate_metadata read hostname 'smithi016' 2024-03-28T22:17:25.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.936+0000 7f4aad830740 1 --2- 172.21.15.16:0/4030974288 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x561f10fb89c0 0x561f10fbae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.936+0000 7f4aad830740 4 client.16962 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:17:25.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.936+0000 7f4aad830740 1 -- 172.21.15.16:0/4030974288 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f10f106f0 con 0x561f10fb89c0 2024-03-28T22:17:25.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.936+0000 7f4aa89f0640 1 --2- 172.21.15.16:0/4030974288 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x561f10fb89c0 0x561f10fbae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.936+0000 7f4a83fff640 10 client.16962 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:17:25.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.936+0000 7f4aa89f0640 1 --2- 172.21.15.16:0/4030974288 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x561f10fb89c0 0x561f10fbae20 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72a73494b43f71a server_cookie=785bbbcabb9eed62 in_seq=0 out_seq=0 2024-03-28T22:17:25.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.937+0000 7f4a83fff640 1 -- 172.21.15.16:0/4030974288 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561f10f106f0 con 0x561f10fb89c0 2024-03-28T22:17:25.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.937+0000 7f4a83fff640 10 client.16962 handle_command_reply: tid=0 2024-03-28T22:17:25.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.937+0000 7f4aad830740 1 client.16962 shutdown 2024-03-28T22:17:25.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.937+0000 7f4aad830740 1 -- 172.21.15.16:0/4030974288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f10f9f470 msgr2=0x561f10fa59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.937+0000 7f4aad830740 1 --2- 172.21.15.16:0/4030974288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f10f9f470 0x561f10fa59a0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f4a94009fb0 tx=0x7f4a94013c50 comp rx=0 tx=0).stop 2024-03-28T22:17:25.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.937+0000 7f4aad830740 1 -- 172.21.15.16:0/4030974288 shutdown_connections 2024-03-28T22:17:25.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.937+0000 7f4aad830740 1 --2- 172.21.15.16:0/4030974288 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x561f10fb89c0 0x561f10fbae20 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.937+0000 7f4aad830740 1 --2- 172.21.15.16:0/4030974288 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f10fa8970 0x561f10fa6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.937+0000 7f4aad830740 1 --2- 172.21.15.16:0/4030974288 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561f10f9fd90 0x561f10fa5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.937+0000 7f4a83fff640 0 client.16962 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:17:25.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.937+0000 7f4aad830740 1 --2- 172.21.15.16:0/4030974288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f10f9f470 0x561f10fa59a0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.937+0000 7f4aad830740 1 -- 172.21.15.16:0/4030974288 >> 172.21.15.16:0/4030974288 conn(0x561f10f99ba0 msgr2=0x561f10f9b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:25.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.937+0000 7f4aad830740 1 -- 172.21.15.16:0/4030974288 shutdown_connections 2024-03-28T22:17:25.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.937+0000 7f4aad830740 1 -- 172.21.15.16:0/4030974288 wait complete. 2024-03-28T22:17:25.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.937+0000 7f4aad830740 20 client.16962 trim_cache size 0 max 16384 2024-03-28T22:17:25.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.950+0000 7f4aad830740 1 Processor -- start 2024-03-28T22:17:25.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.950+0000 7f4aad830740 1 -- start start 2024-03-28T22:17:25.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.951+0000 7f4aad830740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f10f9f470 0x561f10fa80d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.951+0000 7f4aad830740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561f10f9fd90 0x561f10fa4b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.951+0000 7f4aad830740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f10fa8970 0x561f10fb4560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.951+0000 7f4aad830740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa41afd50 con 0x561f10fa8970 2024-03-28T22:17:25.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.951+0000 7f4aad830740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa41ad620 con 0x561f10f9fd90 2024-03-28T22:17:25.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.951+0000 7f4aad830740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa41ab490 con 0x561f10f9f470 2024-03-28T22:17:25.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.951+0000 7f4aa3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f10fa8970 0x561f10fb4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.951+0000 7f4aa3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f10fa8970 0x561f10fb4560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45626/0 (socket says 172.21.15.16:45626) 2024-03-28T22:17:25.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.951+0000 7f4aa3fff640 1 -- 172.21.15.16:0/2861470685 learned_addr learned my addr 172.21.15.16:0/2861470685 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:25.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.951+0000 7f4aa89f0640 1 --2- 172.21.15.16:0/2861470685 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561f10f9fd90 0x561f10fa4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.951+0000 7f4aa3fff640 1 -- 172.21.15.16:0/2861470685 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f10f9f470 msgr2=0x561f10fa80d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:25.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.951+0000 7f4aa3fff640 1 --2- 172.21.15.16:0/2861470685 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f10f9f470 0x561f10fa80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.951+0000 7f4aa3fff640 1 -- 172.21.15.16:0/2861470685 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561f10f9fd90 msgr2=0x561f10fa4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:25.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.951+0000 7f4aa3fff640 1 --2- 172.21.15.16:0/2861470685 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561f10f9fd90 0x561f10fa4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.951+0000 7f4aa3fff640 1 -- 172.21.15.16:0/2861470685 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f10f996d0 con 0x561f10fa8970 2024-03-28T22:17:25.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.951+0000 7f4aa3fff640 1 --2- 172.21.15.16:0/2861470685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f10fa8970 0x561f10fb4560 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f4a940059d0 tx=0x7f4a94005730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0061895805a5d76 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:25.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.951+0000 7f4a9a7fc640 1 -- 172.21.15.16:0/2861470685 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a94004260 con 0x561f10fa8970 2024-03-28T22:17:25.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.951+0000 7f4a9a7fc640 1 -- 172.21.15.16:0/2861470685 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a94005510 con 0x561f10fa8970 2024-03-28T22:17:25.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.951+0000 7f4a9a7fc640 1 -- 172.21.15.16:0/2861470685 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a94011820 con 0x561f10fa8970 2024-03-28T22:17:25.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.952+0000 7f4aad830740 1 -- 172.21.15.16:0/2861470685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f10fa8970 msgr2=0x561f10fb4560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.952+0000 7f4aad830740 1 --2- 172.21.15.16:0/2861470685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f10fa8970 0x561f10fb4560 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f4a940059d0 tx=0x7f4a94005730 comp rx=0 tx=0).stop 2024-03-28T22:17:25.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.952+0000 7f4aad830740 1 -- 172.21.15.16:0/2861470685 shutdown_connections 2024-03-28T22:17:25.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.952+0000 7f4aad830740 1 --2- 172.21.15.16:0/2861470685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f10fa8970 0x561f10fb4560 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.952+0000 7f4aad830740 1 --2- 172.21.15.16:0/2861470685 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561f10f9fd90 0x561f10fa4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.952+0000 7f4aad830740 1 --2- 172.21.15.16:0/2861470685 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f10f9f470 0x561f10fa80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.952+0000 7f4aad830740 1 -- 172.21.15.16:0/2861470685 >> 172.21.15.16:0/2861470685 conn(0x561f10f99ba0 msgr2=0x561f10f984b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.952+0000 7f4aad830740 1 -- 172.21.15.16:0/2861470685 shutdown_connections 2024-03-28T22:17:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.952+0000 7f4aad830740 1 -- 172.21.15.16:0/2861470685 wait complete. 2024-03-28T22:17:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.952+0000 7f4aad830740 1 -- start start 2024-03-28T22:17:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.952+0000 7f4aad830740 1 Processor -- start 2024-03-28T22:17:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.953+0000 7f4aad830740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f10f9f470 0x561f10fb0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.953+0000 7f4aad830740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561f10f9fd90 0x561f10fac150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.953+0000 7f4aad830740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f10fa8970 0x561f10fac690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.953+0000 7f4aad830740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa41abae0 con 0x561f10f9f470 2024-03-28T22:17:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.953+0000 7f4aad830740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa41acf50 con 0x561f10f9fd90 2024-03-28T22:17:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.953+0000 7f4aad830740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa41ac1b0 con 0x561f10fa8970 2024-03-28T22:17:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.953+0000 7f4aad830740 10 client.0 _ll_register_callbacks cb 0x7f4a94013400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:17:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.953+0000 7f4aa3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f10f9f470 0x561f10fb0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.953+0000 7f4aa3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f10f9f470 0x561f10fb0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45630/0 (socket says 172.21.15.16:45630) 2024-03-28T22:17:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.953+0000 7f4aa3fff640 1 -- 172.21.15.16:0/1720955345 learned_addr learned my addr 172.21.15.16:0/1720955345 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.953+0000 7f4aa89f0640 1 --2- 172.21.15.16:0/1720955345 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561f10f9fd90 0x561f10fac150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.953+0000 7f4aa3fff640 1 -- 172.21.15.16:0/1720955345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f10fa8970 msgr2=0x561f10fac690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.953+0000 7f4aa3fff640 1 --2- 172.21.15.16:0/1720955345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f10fa8970 0x561f10fac690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.953+0000 7f4aa3fff640 1 -- 172.21.15.16:0/1720955345 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561f10f9fd90 msgr2=0x561f10fac150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.953+0000 7f4aa3fff640 1 --2- 172.21.15.16:0/1720955345 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561f10f9fd90 0x561f10fac150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.953+0000 7f4aa3fff640 1 -- 172.21.15.16:0/1720955345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f10faccd0 con 0x561f10f9f470 2024-03-28T22:17:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.953+0000 7f4aa89f0640 1 --2- 172.21.15.16:0/1720955345 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561f10f9fd90 0x561f10fac150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:17:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.953+0000 7f4aa3fff640 1 --2- 172.21.15.16:0/1720955345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f10f9f470 0x561f10fb0cb0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f4a940096e0 tx=0x7f4a940039a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79d9b1709ebbe2e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.953+0000 7f4a83fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:17:25.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.953+0000 7f4a83fff640 1 -- 172.21.15.16:0/1720955345 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a94011820 con 0x561f10f9f470 2024-03-28T22:17:25.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.953+0000 7f4a83fff640 1 -- 172.21.15.16:0/1720955345 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8527+0+0 (secure 0 0 0) 0x7f4a940119c0 con 0x561f10f9f470 2024-03-28T22:17:25.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.953+0000 7f4aad830740 1 -- 172.21.15.16:0/1720955345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a94005510 con 0x561f10f9f470 2024-03-28T22:17:25.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.954+0000 7f4a83fff640 1 -- 172.21.15.16:0/1720955345 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4a94011be0 con 0x561f10f9f470 2024-03-28T22:17:25.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.954+0000 7f4aad830740 10 client.16968 fetch_fsmap learned FSMap version 27 2024-03-28T22:17:25.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.954+0000 7f4aad830740 1 -- 172.21.15.16:0/1720955345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561f1101eb30 con 0x561f10f9f470 2024-03-28T22:17:25.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.954+0000 7f4a83fff640 1 -- 172.21.15.16:0/1720955345 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a94005510 con 0x561f10f9f470 2024-03-28T22:17:25.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.954+0000 7f4a83fff640 1 -- 172.21.15.16:0/1720955345 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f4a9403d460 con 0x561f10f9f470 2024-03-28T22:17:25.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.954+0000 7f4aad830740 10 client.16968 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:17:25.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.954+0000 7f4aad830740 10 client.16968 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:17:25.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.954+0000 7f4aad830740 20 client.16968 populate_metadata read hostname 'smithi016' 2024-03-28T22:17:25.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.954+0000 7f4aad830740 1 --2- 172.21.15.16:0/1720955345 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x561f1101f1e0 0x561f11021600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:25.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.954+0000 7f4aad830740 4 client.16968 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:17:25.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.954+0000 7f4aad830740 1 -- 172.21.15.16:0/1720955345 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561f10f0aa40 con 0x561f1101f1e0 2024-03-28T22:17:25.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.954+0000 7f4aa89f0640 1 --2- 172.21.15.16:0/1720955345 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x561f1101f1e0 0x561f11021600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:25.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.955+0000 7f4a83fff640 10 client.16968 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:17:25.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.955+0000 7f4aa89f0640 1 --2- 172.21.15.16:0/1720955345 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x561f1101f1e0 0x561f11021600 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfe94d677c0fbc2c server_cookie=ba4a4046585d7428 in_seq=0 out_seq=0 2024-03-28T22:17:25.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.955+0000 7f4a83fff640 1 -- 172.21.15.16:0/1720955345 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561f10f0aa40 con 0x561f1101f1e0 2024-03-28T22:17:25.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.955+0000 7f4a83fff640 10 client.16968 handle_command_reply: tid=0 2024-03-28T22:17:25.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.955+0000 7f4aad830740 1 client.16968 shutdown 2024-03-28T22:17:25.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.955+0000 7f4aad830740 1 -- 172.21.15.16:0/1720955345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f10f9f470 msgr2=0x561f10fb0cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.955+0000 7f4aad830740 1 --2- 172.21.15.16:0/1720955345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f10f9f470 0x561f10fb0cb0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f4a940096e0 tx=0x7f4a940039a0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.955+0000 7f4aad830740 1 -- 172.21.15.16:0/1720955345 shutdown_connections 2024-03-28T22:17:25.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.955+0000 7f4aad830740 1 --2- 172.21.15.16:0/1720955345 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x561f1101f1e0 0x561f11021600 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.955+0000 7f4aad830740 1 --2- 172.21.15.16:0/1720955345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f10fa8970 0x561f10fac690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.955+0000 7f4a83fff640 0 client.16968 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:17:25.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.955+0000 7f4aad830740 1 --2- 172.21.15.16:0/1720955345 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561f10f9fd90 0x561f10fac150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.955+0000 7f4aad830740 1 --2- 172.21.15.16:0/1720955345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f10f9f470 0x561f10fb0cb0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.955+0000 7f4aad830740 1 -- 172.21.15.16:0/1720955345 >> 172.21.15.16:0/1720955345 conn(0x561f10f99ba0 msgr2=0x561f10fa5960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:25.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.955+0000 7f4aad830740 1 -- 172.21.15.16:0/1720955345 shutdown_connections 2024-03-28T22:17:25.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.955+0000 7f4aad830740 1 -- 172.21.15.16:0/1720955345 wait complete. 2024-03-28T22:17:25.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.955+0000 7f4aad830740 20 client.16968 trim_cache size 0 max 16384 2024-03-28T22:17:25.969 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:17:25.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.956+0000 7f4aaac54640 1 -- 172.21.15.16:0/2459202592 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4a900a1ef0 msgr2=0x7f4a900a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.956+0000 7f4aaac54640 1 --2- 172.21.15.16:0/2459202592 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4a900a1ef0 0x7f4a900a43b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c0ee370 tx=0x7f4a9c02c180 comp rx=0 tx=0).stop 2024-03-28T22:17:25.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.956+0000 7f4aaac54640 1 -- 172.21.15.16:0/2459202592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aa410bd30 msgr2=0x7f4aa419c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:25.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.956+0000 7f4aaac54640 1 --2- 172.21.15.16:0/2459202592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aa410bd30 0x7f4aa419c820 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c02c790 tx=0x7f4a9c035e00 comp rx=0 tx=0).stop 2024-03-28T22:17:25.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.956+0000 7f4aaac54640 1 -- 172.21.15.16:0/2459202592 shutdown_connections 2024-03-28T22:17:25.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.956+0000 7f4aaac54640 1 --2- 172.21.15.16:0/2459202592 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4a900a1ef0 0x7f4a900a43b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.956+0000 7f4aaac54640 1 --2- 172.21.15.16:0/2459202592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aa410bd30 0x7f4aa419c820 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.956+0000 7f4aaac54640 1 --2- 172.21.15.16:0/2459202592 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4aa4103280 0x7f4aa41984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.956+0000 7f4aaac54640 1 --2- 172.21.15.16:0/2459202592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4aa4102960 0x7f4aa4197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:25.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.956+0000 7f4aaac54640 1 -- 172.21.15.16:0/2459202592 >> 172.21.15.16:0/2459202592 conn(0x7f4aa4077820 msgr2=0x7f4aa40fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:25.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.956+0000 7f4aaac54640 1 -- 172.21.15.16:0/2459202592 shutdown_connections 2024-03-28T22:17:25.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:25.956+0000 7f4aaac54640 1 -- 172.21.15.16:0/2459202592 wait complete. 2024-03-28T22:17:26.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:26 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/525888522' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:17:26.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:26 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/525888522' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:17:26.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:26 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/525888522' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:17:26.972 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:17:26.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:17:26.973 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:17:27.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.091+0000 7f47fb72c640 1 -- 172.21.15.16:0/1161884801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47f41004f0 msgr2=0x7f47f41008d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:27.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.091+0000 7f47fb72c640 1 --2- 172.21.15.16:0/1161884801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47f41004f0 0x7f47f41008d0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f47ec009900 tx=0x7f47ec02bcd0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.091+0000 7f47fb72c640 1 -- 172.21.15.16:0/1161884801 shutdown_connections 2024-03-28T22:17:27.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.091+0000 7f47fb72c640 1 --2- 172.21.15.16:0/1161884801 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47f410e0a0 0x7f47f4110490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.091+0000 7f47fb72c640 1 --2- 172.21.15.16:0/1161884801 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47f4100e10 0x7f47f410db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.091+0000 7f47fb72c640 1 --2- 172.21.15.16:0/1161884801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47f41004f0 0x7f47f41008d0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.091+0000 7f47fb72c640 1 -- 172.21.15.16:0/1161884801 >> 172.21.15.16:0/1161884801 conn(0x7f47f4077800 msgr2=0x7f47f4077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:27.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.091+0000 7f47fb72c640 1 -- 172.21.15.16:0/1161884801 shutdown_connections 2024-03-28T22:17:27.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.091+0000 7f47fb72c640 1 -- 172.21.15.16:0/1161884801 wait complete. 2024-03-28T22:17:27.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.091+0000 7f47fb72c640 1 Processor -- start 2024-03-28T22:17:27.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.091+0000 7f47fb72c640 1 -- start start 2024-03-28T22:17:27.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.092+0000 7f47fb72c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47f41004f0 0x7f47f419a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:27.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.092+0000 7f47fb72c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47f4100e10 0x7f47f419a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:27.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.092+0000 7f47fb72c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47f410e0a0 0x7f47f419eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:27.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.092+0000 7f47fb72c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f47f4113d00 con 0x7f47f410e0a0 2024-03-28T22:17:27.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.092+0000 7f47fb72c640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f47f4113b80 con 0x7f47f41004f0 2024-03-28T22:17:27.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.092+0000 7f47fb72c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f47f4113e80 con 0x7f47f4100e10 2024-03-28T22:17:27.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.092+0000 7f47f9cc9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47f410e0a0 0x7f47f419eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:27.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.092+0000 7f47f9cc9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47f410e0a0 0x7f47f419eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45652/0 (socket says 172.21.15.16:45652) 2024-03-28T22:17:27.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.092+0000 7f47f9cc9640 1 -- 172.21.15.16:0/2365364167 learned_addr learned my addr 172.21.15.16:0/2365364167 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:27.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.092+0000 7f47f94c8640 1 --2- 172.21.15.16:0/2365364167 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47f41004f0 0x7f47f419a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:27.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.092+0000 7f47f8cc7640 1 --2- 172.21.15.16:0/2365364167 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47f4100e10 0x7f47f419a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:27.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.092+0000 7f47f9cc9640 1 -- 172.21.15.16:0/2365364167 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47f4100e10 msgr2=0x7f47f419a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:27.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.092+0000 7f47f9cc9640 1 --2- 172.21.15.16:0/2365364167 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47f4100e10 0x7f47f419a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.092+0000 7f47f9cc9640 1 -- 172.21.15.16:0/2365364167 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47f41004f0 msgr2=0x7f47f419a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:27.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.092+0000 7f47f9cc9640 1 --2- 172.21.15.16:0/2365364167 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47f41004f0 0x7f47f419a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.092+0000 7f47f9cc9640 1 -- 172.21.15.16:0/2365364167 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47f419f210 con 0x7f47f410e0a0 2024-03-28T22:17:27.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.092+0000 7f47f9cc9640 1 --2- 172.21.15.16:0/2365364167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47f410e0a0 0x7f47f419eae0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f47f000e8f0 tx=0x7f47f000edb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:27.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.092+0000 7f47ea7fc640 1 -- 172.21.15.16:0/2365364167 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f0004d20 con 0x7f47f410e0a0 2024-03-28T22:17:27.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.092+0000 7f47ea7fc640 1 -- 172.21.15.16:0/2365364167 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47f0005020 con 0x7f47f410e0a0 2024-03-28T22:17:27.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.092+0000 7f47fb72c640 1 -- 172.21.15.16:0/2365364167 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47f419f500 con 0x7f47f410e0a0 2024-03-28T22:17:27.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.092+0000 7f47ea7fc640 1 -- 172.21.15.16:0/2365364167 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f0011690 con 0x7f47f410e0a0 2024-03-28T22:17:27.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.092+0000 7f47fb72c640 1 -- 172.21.15.16:0/2365364167 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47f41a6d90 con 0x7f47f410e0a0 2024-03-28T22:17:27.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.095+0000 7f47ea7fc640 1 -- 172.21.15.16:0/2365364167 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f47f00051d0 con 0x7f47f410e0a0 2024-03-28T22:17:27.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.095+0000 7f47fe3eb740 1 Processor -- start 2024-03-28T22:17:27.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.095+0000 7f47fe3eb740 1 -- start start 2024-03-28T22:17:27.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.095+0000 7f47fe3eb740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55876961f9f0 0x55876961fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:27.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.095+0000 7f47fe3eb740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558769620310 0x5587696289b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:27.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.095+0000 7f47ea7fc640 1 --2- 172.21.15.16:0/2365364167 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f47e00a1ef0 0x7f47e00a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:27.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.095+0000 7f47fe3eb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558769628ef0 0x55876962b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:27.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.095+0000 7f47fe3eb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5587695200b0 con 0x558769628ef0 2024-03-28T22:17:27.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.096+0000 7f47fe3eb740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x558769582ea0 con 0x55876961f9f0 2024-03-28T22:17:27.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.096+0000 7f47fe3eb740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55876952bc00 con 0x558769620310 2024-03-28T22:17:27.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.096+0000 7f47f9cc9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558769628ef0 0x55876962b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:27.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.096+0000 7f47f8cc7640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55876961f9f0 0x55876961fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:27.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.096+0000 7f47f8cc7640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55876961f9f0 0x55876961fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:52674/0 (socket says 172.21.15.16:52674) 2024-03-28T22:17:27.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.096+0000 7f47f8cc7640 1 -- 172.21.15.16:0/2678027254 learned_addr learned my addr 172.21.15.16:0/2678027254 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:27.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.096+0000 7f47f94c8640 1 --2- 172.21.15.16:0/2678027254 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558769620310 0x5587696289b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:27.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.096+0000 7f47f8cc7640 1 --2- 172.21.15.16:0/2365364167 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f47e00a1ef0 0x7f47e00a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:27.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.096+0000 7f47ea7fc640 1 -- 172.21.15.16:0/2365364167 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8527+0+0 (secure 0 0 0) 0x7f47f00ce550 con 0x7f47f410e0a0 2024-03-28T22:17:27.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.096+0000 7f47f8cc7640 1 -- 172.21.15.16:0/2678027254 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558769620310 msgr2=0x5587696289b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:27.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.096+0000 7f47f8cc7640 1 --2- 172.21.15.16:0/2678027254 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558769620310 0x5587696289b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.096+0000 7f47f8cc7640 1 -- 172.21.15.16:0/2678027254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558769628ef0 msgr2=0x55876962b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:27.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.096+0000 7f47f8cc7640 1 --2- 172.21.15.16:0/2678027254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558769628ef0 0x55876962b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.096+0000 7f47f8cc7640 1 -- 172.21.15.16:0/2678027254 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55876935f7e0 con 0x55876961f9f0 2024-03-28T22:17:27.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.096+0000 7f47f9cc9640 1 --2- 172.21.15.16:0/2678027254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558769628ef0 0x55876962b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:17:27.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.096+0000 7f47f8cc7640 1 --2- 172.21.15.16:0/2365364167 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f47e00a1ef0 0x7f47e00a43b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f47e4011fd0 tx=0x7f47e4018000 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:27.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.096+0000 7f47f8cc7640 1 --2- 172.21.15.16:0/2678027254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55876961f9f0 0x55876961fdd0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f47e40058c0 tx=0x7f47e4011f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c122722d74655e9a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:27.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.097+0000 7f47df7fe640 1 -- 172.21.15.16:0/2678027254 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e4011050 con 0x55876961f9f0 2024-03-28T22:17:27.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.097+0000 7f47df7fe640 1 -- 172.21.15.16:0/2678027254 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47e4009a20 con 0x55876961f9f0 2024-03-28T22:17:27.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.097+0000 7f47df7fe640 1 -- 172.21.15.16:0/2678027254 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e40035f0 con 0x55876961f9f0 2024-03-28T22:17:27.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.097+0000 7f47fe3eb740 1 -- 172.21.15.16:0/2678027254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55876961f9f0 msgr2=0x55876961fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:27.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.097+0000 7f47fe3eb740 1 --2- 172.21.15.16:0/2678027254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55876961f9f0 0x55876961fdd0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f47e40058c0 tx=0x7f47e4011f60 comp rx=0 tx=0).stop 2024-03-28T22:17:27.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.097+0000 7f47fe3eb740 1 -- 172.21.15.16:0/2678027254 shutdown_connections 2024-03-28T22:17:27.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.097+0000 7f47fe3eb740 1 --2- 172.21.15.16:0/2678027254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558769628ef0 0x55876962b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.097+0000 7f47fe3eb740 1 --2- 172.21.15.16:0/2678027254 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558769620310 0x5587696289b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.097+0000 7f47fe3eb740 1 --2- 172.21.15.16:0/2678027254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55876961f9f0 0x55876961fdd0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.097+0000 7f47fe3eb740 1 -- 172.21.15.16:0/2678027254 >> 172.21.15.16:0/2678027254 conn(0x55876961a120 msgr2=0x55876961a500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:27.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.097+0000 7f47fe3eb740 1 -- 172.21.15.16:0/2678027254 shutdown_connections 2024-03-28T22:17:27.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.097+0000 7f47fe3eb740 1 -- 172.21.15.16:0/2678027254 wait complete. 2024-03-28T22:17:27.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.098+0000 7f47fe3eb740 1 -- start start 2024-03-28T22:17:27.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.098+0000 7f47fe3eb740 1 Processor -- start 2024-03-28T22:17:27.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.098+0000 7f47fe3eb740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55876961f9f0 0x558769625fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:27.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.098+0000 7f47fe3eb740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558769620310 0x558769626500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:27.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.098+0000 7f47fe3eb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558769628ef0 0x558769626a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:27.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.098+0000 7f47fe3eb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5587695213d0 con 0x558769628ef0 2024-03-28T22:17:27.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.098+0000 7f47fe3eb740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55876957ef40 con 0x55876961f9f0 2024-03-28T22:17:27.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.098+0000 7f47fe3eb740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55876962bc80 con 0x558769620310 2024-03-28T22:17:27.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.098+0000 7f47fe3eb740 10 client.0 _ll_register_callbacks cb 0x55876953eb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:17:27.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.098+0000 7f47f9cc9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558769628ef0 0x558769626a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:27.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.098+0000 7f47f9cc9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558769628ef0 0x558769626a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45670/0 (socket says 172.21.15.16:45670) 2024-03-28T22:17:27.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.098+0000 7f47f9cc9640 1 -- 172.21.15.16:0/3949080957 learned_addr learned my addr 172.21.15.16:0/3949080957 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:27.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.098+0000 7f47f94c8640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55876961f9f0 0x558769625fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:27.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.098+0000 7f47f8cc7640 1 --2- 172.21.15.16:0/3949080957 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558769620310 0x558769626500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:27.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.099+0000 7f47f9cc9640 1 -- 172.21.15.16:0/3949080957 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558769620310 msgr2=0x558769626500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:27.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.099+0000 7f47f9cc9640 1 --2- 172.21.15.16:0/3949080957 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558769620310 0x558769626500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.099+0000 7f47f9cc9640 1 -- 172.21.15.16:0/3949080957 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55876961f9f0 msgr2=0x558769625fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:27.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.099+0000 7f47f9cc9640 1 --2- 172.21.15.16:0/3949080957 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55876961f9f0 0x558769625fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.099+0000 7f47f9cc9640 1 -- 172.21.15.16:0/3949080957 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587693bff70 con 0x558769628ef0 2024-03-28T22:17:27.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.099+0000 7f47f8cc7640 1 --2- 172.21.15.16:0/3949080957 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558769620310 0x558769626500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:17:27.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.099+0000 7f47f9cc9640 1 --2- 172.21.15.16:0/3949080957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558769628ef0 0x558769626a40 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f47f00979c0 tx=0x7f47f009abf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec219d14751ca9c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:27.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.099+0000 7f47dcff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:17:27.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.099+0000 7f47dcff9640 1 -- 172.21.15.16:0/3949080957 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f001f050 con 0x558769628ef0 2024-03-28T22:17:27.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.099+0000 7f47dcff9640 1 -- 172.21.15.16:0/3949080957 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8527+0+0 (secure 0 0 0) 0x7f47f00992f0 con 0x558769628ef0 2024-03-28T22:17:27.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.099+0000 7f47fe3eb740 1 -- 172.21.15.16:0/3949080957 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47e4009a20 con 0x558769628ef0 2024-03-28T22:17:27.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.099+0000 7f47dcff9640 1 -- 172.21.15.16:0/3949080957 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f47f009f090 con 0x558769628ef0 2024-03-28T22:17:27.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.099+0000 7f47fe3eb740 10 client.16980 fetch_fsmap learned FSMap version 27 2024-03-28T22:17:27.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.099+0000 7f47fe3eb740 1 -- 172.21.15.16:0/3949080957 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558769257170 con 0x558769628ef0 2024-03-28T22:17:27.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.099+0000 7f47dcff9640 1 -- 172.21.15.16:0/3949080957 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47e4009a20 con 0x558769628ef0 2024-03-28T22:17:27.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.099+0000 7f47dcff9640 1 -- 172.21.15.16:0/3949080957 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f47f000f040 con 0x558769628ef0 2024-03-28T22:17:27.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.099+0000 7f47fe3eb740 10 client.16980 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:17:27.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.099+0000 7f47fe3eb740 10 client.16980 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:17:27.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.100+0000 7f47fe3eb740 20 client.16980 populate_metadata read hostname 'smithi016' 2024-03-28T22:17:27.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.100+0000 7f47fe3eb740 1 --2- 172.21.15.16:0/3949080957 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558769638e50 0x55876963b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:27.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.100+0000 7f47fe3eb740 4 client.16980 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:17:27.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.100+0000 7f47fe3eb740 1 -- 172.21.15.16:0/3949080957 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5587695903e0 con 0x558769638e50 2024-03-28T22:17:27.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.100+0000 7f47f94c8640 1 --2- 172.21.15.16:0/3949080957 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558769638e50 0x55876963b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:27.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.100+0000 7f47dcff9640 10 client.16980 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:17:27.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.100+0000 7f47f94c8640 1 --2- 172.21.15.16:0/3949080957 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558769638e50 0x55876963b240 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21cb506f4eeeb30c server_cookie=fe270f0f74031fd0 in_seq=0 out_seq=0 2024-03-28T22:17:27.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.101+0000 7f47dcff9640 1 -- 172.21.15.16:0/3949080957 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5587695903e0 con 0x558769638e50 2024-03-28T22:17:27.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.101+0000 7f47dcff9640 10 client.16980 handle_command_reply: tid=0 2024-03-28T22:17:27.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.101+0000 7f47fe3eb740 1 client.16980 shutdown 2024-03-28T22:17:27.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.101+0000 7f47fe3eb740 1 -- 172.21.15.16:0/3949080957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558769628ef0 msgr2=0x558769626a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:27.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.101+0000 7f47fe3eb740 1 --2- 172.21.15.16:0/3949080957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558769628ef0 0x558769626a40 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f47f00979c0 tx=0x7f47f009abf0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.102+0000 7f47fe3eb740 1 -- 172.21.15.16:0/3949080957 shutdown_connections 2024-03-28T22:17:27.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.102+0000 7f47fe3eb740 1 --2- 172.21.15.16:0/3949080957 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558769638e50 0x55876963b240 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.102+0000 7f47fe3eb740 1 --2- 172.21.15.16:0/3949080957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558769628ef0 0x558769626a40 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.102+0000 7f47fe3eb740 1 --2- 172.21.15.16:0/3949080957 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558769620310 0x558769626500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.102+0000 7f47dcff9640 0 client.16980 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:17:27.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.102+0000 7f47fe3eb740 1 --2- 172.21.15.16:0/3949080957 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55876961f9f0 0x558769625fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.102+0000 7f47fe3eb740 1 -- 172.21.15.16:0/3949080957 >> 172.21.15.16:0/3949080957 conn(0x55876961a120 msgr2=0x55876961be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:27.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.102+0000 7f47fe3eb740 1 -- 172.21.15.16:0/3949080957 shutdown_connections 2024-03-28T22:17:27.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.102+0000 7f47fe3eb740 1 -- 172.21.15.16:0/3949080957 wait complete. 2024-03-28T22:17:27.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.102+0000 7f47fe3eb740 20 client.16980 trim_cache size 0 max 16384 2024-03-28T22:17:27.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.119+0000 7f47fe3eb740 1 Processor -- start 2024-03-28T22:17:27.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.119+0000 7f47fe3eb740 1 -- start start 2024-03-28T22:17:27.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.120+0000 7f47fe3eb740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55876961f9f0 0x5587696286f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:27.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.120+0000 7f47fe3eb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558769620310 0x5587696348a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:27.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.120+0000 7f47fe3eb740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558769628ef0 0x55876961bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:27.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.120+0000 7f47fe3eb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f47f41b1e20 con 0x558769620310 2024-03-28T22:17:27.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.120+0000 7f47fe3eb740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f47f41af6f0 con 0x558769628ef0 2024-03-28T22:17:27.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.120+0000 7f47fe3eb740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f47f41ad560 con 0x55876961f9f0 2024-03-28T22:17:27.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.120+0000 7f47f8cc7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558769620310 0x5587696348a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:27.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.120+0000 7f47f8cc7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558769620310 0x5587696348a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45680/0 (socket says 172.21.15.16:45680) 2024-03-28T22:17:27.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.120+0000 7f47f8cc7640 1 -- 172.21.15.16:0/1811214738 learned_addr learned my addr 172.21.15.16:0/1811214738 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:27.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.120+0000 7f47f9cc9640 1 --2- 172.21.15.16:0/1811214738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558769628ef0 0x55876961bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:27.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.120+0000 7f47f94c8640 1 --2- 172.21.15.16:0/1811214738 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55876961f9f0 0x5587696286f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:27.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.120+0000 7f47f8cc7640 1 -- 172.21.15.16:0/1811214738 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55876961f9f0 msgr2=0x5587696286f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:27.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.120+0000 7f47f8cc7640 1 --2- 172.21.15.16:0/1811214738 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55876961f9f0 0x5587696286f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.120+0000 7f47f8cc7640 1 -- 172.21.15.16:0/1811214738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558769628ef0 msgr2=0x55876961bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:27.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.120+0000 7f47f8cc7640 1 --2- 172.21.15.16:0/1811214738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558769628ef0 0x55876961bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.120+0000 7f47f8cc7640 1 -- 172.21.15.16:0/1811214738 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558769624510 con 0x558769620310 2024-03-28T22:17:27.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.120+0000 7f47f8cc7640 1 --2- 172.21.15.16:0/1811214738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558769620310 0x5587696348a0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f47e4009c80 tx=0x7f47e4013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c07cb09f7271ce2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:27.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.120+0000 7f47deffd640 1 -- 172.21.15.16:0/1811214738 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e40116a0 con 0x558769620310 2024-03-28T22:17:27.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.120+0000 7f47deffd640 1 -- 172.21.15.16:0/1811214738 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47e4011250 con 0x558769620310 2024-03-28T22:17:27.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.120+0000 7f47deffd640 1 -- 172.21.15.16:0/1811214738 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e400e7c0 con 0x558769620310 2024-03-28T22:17:27.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.121+0000 7f47fe3eb740 1 -- 172.21.15.16:0/1811214738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558769620310 msgr2=0x5587696348a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:27.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.121+0000 7f47fe3eb740 1 --2- 172.21.15.16:0/1811214738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558769620310 0x5587696348a0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f47e4009c80 tx=0x7f47e4013040 comp rx=0 tx=0).stop 2024-03-28T22:17:27.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.121+0000 7f47fe3eb740 1 -- 172.21.15.16:0/1811214738 shutdown_connections 2024-03-28T22:17:27.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.121+0000 7f47fe3eb740 1 --2- 172.21.15.16:0/1811214738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558769628ef0 0x55876961bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.121+0000 7f47fe3eb740 1 --2- 172.21.15.16:0/1811214738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558769620310 0x5587696348a0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.121+0000 7f47fe3eb740 1 --2- 172.21.15.16:0/1811214738 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55876961f9f0 0x5587696286f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.121+0000 7f47fe3eb740 1 -- 172.21.15.16:0/1811214738 >> 172.21.15.16:0/1811214738 conn(0x55876961a120 msgr2=0x55876961b550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:27.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.121+0000 7f47fe3eb740 1 -- 172.21.15.16:0/1811214738 shutdown_connections 2024-03-28T22:17:27.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.121+0000 7f47fe3eb740 1 -- 172.21.15.16:0/1811214738 wait complete. 2024-03-28T22:17:27.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.121+0000 7f47fe3eb740 1 -- start start 2024-03-28T22:17:27.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.121+0000 7f47fe3eb740 1 Processor -- start 2024-03-28T22:17:27.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.122+0000 7f47fe3eb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55876961f9f0 0x55876962cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:27.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.122+0000 7f47fe3eb740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558769620310 0x55876962d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:27.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.122+0000 7f47fe3eb740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558769628ef0 0x55876969e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:27.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.122+0000 7f47fe3eb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f47f41adbb0 con 0x55876961f9f0 2024-03-28T22:17:27.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.122+0000 7f47fe3eb740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f47f41af020 con 0x558769628ef0 2024-03-28T22:17:27.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.122+0000 7f47fe3eb740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f47f41ae280 con 0x558769620310 2024-03-28T22:17:27.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.122+0000 7f47fe3eb740 10 client.0 _ll_register_callbacks cb 0x7f47f009adc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:17:27.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.122+0000 7f47f94c8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55876961f9f0 0x55876962cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:27.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.122+0000 7f47f94c8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55876961f9f0 0x55876962cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45694/0 (socket says 172.21.15.16:45694) 2024-03-28T22:17:27.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.122+0000 7f47f94c8640 1 -- 172.21.15.16:0/723278644 learned_addr learned my addr 172.21.15.16:0/723278644 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:27.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.122+0000 7f47f9cc9640 1 --2- 172.21.15.16:0/723278644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558769628ef0 0x55876969e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:27.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.122+0000 7f47f8cc7640 1 --2- 172.21.15.16:0/723278644 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558769620310 0x55876962d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:27.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.122+0000 7f47f94c8640 1 -- 172.21.15.16:0/723278644 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558769620310 msgr2=0x55876962d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:27.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.122+0000 7f47f94c8640 1 --2- 172.21.15.16:0/723278644 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558769620310 0x55876962d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.122+0000 7f47f94c8640 1 -- 172.21.15.16:0/723278644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558769628ef0 msgr2=0x55876969e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:27.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.122+0000 7f47f94c8640 1 --2- 172.21.15.16:0/723278644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558769628ef0 0x55876969e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.122+0000 7f47f94c8640 1 -- 172.21.15.16:0/723278644 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55876962d9e0 con 0x55876961f9f0 2024-03-28T22:17:27.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.122+0000 7f47f8cc7640 1 --2- 172.21.15.16:0/723278644 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558769620310 0x55876962d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:17:27.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.122+0000 7f47f9cc9640 1 --2- 172.21.15.16:0/723278644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558769628ef0 0x55876969e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:17:27.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.122+0000 7f47f94c8640 1 --2- 172.21.15.16:0/723278644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55876961f9f0 0x55876962cbd0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f47ec009230 tx=0x7f47ec004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c72997875a933ecd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:27.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.122+0000 7f47dcff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:17:27.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.122+0000 7f47dcff9640 1 -- 172.21.15.16:0/723278644 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47ec02cd30 con 0x55876961f9f0 2024-03-28T22:17:27.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.123+0000 7f47dcff9640 1 -- 172.21.15.16:0/723278644 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8527+0+0 (secure 0 0 0) 0x7f47ec0046c0 con 0x55876961f9f0 2024-03-28T22:17:27.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.123+0000 7f47fe3eb740 1 -- 172.21.15.16:0/723278644 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47e4011250 con 0x55876961f9f0 2024-03-28T22:17:27.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.123+0000 7f47dcff9640 1 -- 172.21.15.16:0/723278644 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f47ec05a050 con 0x55876961f9f0 2024-03-28T22:17:27.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.123+0000 7f47fe3eb740 10 client.16986 fetch_fsmap learned FSMap version 27 2024-03-28T22:17:27.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.123+0000 7f47fe3eb740 1 -- 172.21.15.16:0/723278644 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55876969f820 con 0x55876961f9f0 2024-03-28T22:17:27.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.123+0000 7f47dcff9640 1 -- 172.21.15.16:0/723278644 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47e4011250 con 0x55876961f9f0 2024-03-28T22:17:27.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.123+0000 7f47dcff9640 1 -- 172.21.15.16:0/723278644 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f47ec04a050 con 0x55876961f9f0 2024-03-28T22:17:27.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.123+0000 7f47fe3eb740 10 client.16986 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:17:27.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.123+0000 7f47fe3eb740 10 client.16986 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:17:27.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.123+0000 7f47fe3eb740 20 client.16986 populate_metadata read hostname 'smithi016' 2024-03-28T22:17:27.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.123+0000 7f47fe3eb740 1 --2- 172.21.15.16:0/723278644 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55876969ff50 0x5587696a2370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:27.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.123+0000 7f47fe3eb740 4 client.16986 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:17:27.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.123+0000 7f47fe3eb740 1 -- 172.21.15.16:0/723278644 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55876958a730 con 0x55876969ff50 2024-03-28T22:17:27.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.123+0000 7f47f8cc7640 1 --2- 172.21.15.16:0/723278644 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55876969ff50 0x5587696a2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:27.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.124+0000 7f47f8cc7640 1 --2- 172.21.15.16:0/723278644 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55876969ff50 0x5587696a2370 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b84ececf9aa2163 server_cookie=d54ae32279e3babf in_seq=0 out_seq=0 2024-03-28T22:17:27.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.124+0000 7f47dcff9640 10 client.16986 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:17:27.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.125+0000 7f47dcff9640 1 -- 172.21.15.16:0/723278644 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55876958a730 con 0x55876969ff50 2024-03-28T22:17:27.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.125+0000 7f47dcff9640 10 client.16986 handle_command_reply: tid=0 2024-03-28T22:17:27.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.125+0000 7f47fe3eb740 1 client.16986 shutdown 2024-03-28T22:17:27.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.125+0000 7f47fe3eb740 1 -- 172.21.15.16:0/723278644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55876961f9f0 msgr2=0x55876962cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:27.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.125+0000 7f47fe3eb740 1 --2- 172.21.15.16:0/723278644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55876961f9f0 0x55876962cbd0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f47ec009230 tx=0x7f47ec004410 comp rx=0 tx=0).stop 2024-03-28T22:17:27.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.125+0000 7f47fe3eb740 1 -- 172.21.15.16:0/723278644 shutdown_connections 2024-03-28T22:17:27.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.125+0000 7f47fe3eb740 1 --2- 172.21.15.16:0/723278644 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55876969ff50 0x5587696a2370 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.125+0000 7f47fe3eb740 1 --2- 172.21.15.16:0/723278644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558769628ef0 0x55876969e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.125+0000 7f47fe3eb740 1 --2- 172.21.15.16:0/723278644 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558769620310 0x55876962d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.125+0000 7f47dcff9640 0 client.16986 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:17:27.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.125+0000 7f47fe3eb740 1 --2- 172.21.15.16:0/723278644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55876961f9f0 0x55876962cbd0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.125+0000 7f47fe3eb740 1 -- 172.21.15.16:0/723278644 >> 172.21.15.16:0/723278644 conn(0x55876961a120 msgr2=0x55876961bde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:27.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.125+0000 7f47fe3eb740 1 -- 172.21.15.16:0/723278644 shutdown_connections 2024-03-28T22:17:27.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.125+0000 7f47fe3eb740 1 -- 172.21.15.16:0/723278644 wait complete. 2024-03-28T22:17:27.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.125+0000 7f47fe3eb740 20 client.16986 trim_cache size 0 max 16384 2024-03-28T22:17:27.137 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:17:27.138 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-03-28T22:17:27.138 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "3719abed-052e-49da-980e-def6fdeed7ad", 2024-03-28T22:17:27.138 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-03-28T22:17:27.138 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:17:27.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.125+0000 7f47fb72c640 1 -- 172.21.15.16:0/2365364167 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f47e00a1ef0 msgr2=0x7f47e00a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:27.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.125+0000 7f47fb72c640 1 --2- 172.21.15.16:0/2365364167 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f47e00a1ef0 0x7f47e00a43b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f47e4011fd0 tx=0x7f47e4018000 comp rx=0 tx=0).stop 2024-03-28T22:17:27.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.125+0000 7f47fb72c640 1 -- 172.21.15.16:0/2365364167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47f410e0a0 msgr2=0x7f47f419eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:27.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.125+0000 7f47fb72c640 1 --2- 172.21.15.16:0/2365364167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47f410e0a0 0x7f47f419eae0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f47f000e8f0 tx=0x7f47f000edb0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.125+0000 7f47fb72c640 1 -- 172.21.15.16:0/2365364167 shutdown_connections 2024-03-28T22:17:27.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.126+0000 7f47fb72c640 1 --2- 172.21.15.16:0/2365364167 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f47e00a1ef0 0x7f47e00a43b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.126+0000 7f47fb72c640 1 --2- 172.21.15.16:0/2365364167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47f410e0a0 0x7f47f419eae0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.126+0000 7f47fb72c640 1 --2- 172.21.15.16:0/2365364167 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47f4100e10 0x7f47f419a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.126+0000 7f47fb72c640 1 --2- 172.21.15.16:0/2365364167 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47f41004f0 0x7f47f419a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.126+0000 7f47fb72c640 1 -- 172.21.15.16:0/2365364167 >> 172.21.15.16:0/2365364167 conn(0x7f47f4077800 msgr2=0x7f47f410f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:27.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.126+0000 7f47fb72c640 1 -- 172.21.15.16:0/2365364167 shutdown_connections 2024-03-28T22:17:27.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.126+0000 7f47fb72c640 1 -- 172.21.15.16:0/2365364167 wait complete. 2024-03-28T22:17:27.141 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:17:27.142 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:17:27.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.247+0000 7f09ad002640 1 -- 172.21.15.16:0/3519827447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09a810bb80 msgr2=0x7f09a810dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:27.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.247+0000 7f09ad002640 1 --2- 172.21.15.16:0/3519827447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09a810bb80 0x7f09a810dfd0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f09a0009ca0 tx=0x7f09a002c180 comp rx=0 tx=0).stop 2024-03-28T22:17:27.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.247+0000 7f09ad002640 1 -- 172.21.15.16:0/3519827447 shutdown_connections 2024-03-28T22:17:27.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.247+0000 7f09ad002640 1 --2- 172.21.15.16:0/3519827447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09a810bb80 0x7f09a810dfd0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.247+0000 7f09ad002640 1 --2- 172.21.15.16:0/3519827447 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09a81030d0 0x7f09a810b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.247+0000 7f09ad002640 1 --2- 172.21.15.16:0/3519827447 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09a81027b0 0x7f09a8102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.247+0000 7f09ad002640 1 -- 172.21.15.16:0/3519827447 >> 172.21.15.16:0/3519827447 conn(0x7f09a80fbf50 msgr2=0x7f09a80fe370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:27.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.247+0000 7f09ad002640 1 -- 172.21.15.16:0/3519827447 shutdown_connections 2024-03-28T22:17:27.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.247+0000 7f09ad002640 1 -- 172.21.15.16:0/3519827447 wait complete. 2024-03-28T22:17:27.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.248+0000 7f09ad002640 1 Processor -- start 2024-03-28T22:17:27.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.248+0000 7f09ad002640 1 -- start start 2024-03-28T22:17:27.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.248+0000 7f09ad002640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09a81027b0 0x7f09a8197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:27.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.248+0000 7f09ad002640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09a81030d0 0x7f09a8198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:27.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.248+0000 7f09ad002640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09a810bb80 0x7f09a819c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.248+0000 7f09ad002640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f09a8111900 con 0x7f09a81027b0 2024-03-28T22:17:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.248+0000 7f09ad002640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f09a8111780 con 0x7f09a810bb80 2024-03-28T22:17:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.248+0000 7f09ad002640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f09a8111a80 con 0x7f09a81030d0 2024-03-28T22:17:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.248+0000 7f09a6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09a81027b0 0x7f09a8197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.248+0000 7f09a6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09a81027b0 0x7f09a8197e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45708/0 (socket says 172.21.15.16:45708) 2024-03-28T22:17:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.248+0000 7f09a6d9d640 1 -- 172.21.15.16:0/3924046210 learned_addr learned my addr 172.21.15.16:0/3924046210 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.248+0000 7f09a759e640 1 --2- 172.21.15.16:0/3924046210 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09a810bb80 0x7f09a819c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.248+0000 7f099ffff640 1 --2- 172.21.15.16:0/3924046210 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09a81030d0 0x7f09a8198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.248+0000 7f09a6d9d640 1 -- 172.21.15.16:0/3924046210 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09a81030d0 msgr2=0x7f09a8198360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:27.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.248+0000 7f09a6d9d640 1 --2- 172.21.15.16:0/3924046210 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09a81030d0 0x7f09a8198360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.248+0000 7f09a6d9d640 1 -- 172.21.15.16:0/3924046210 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09a810bb80 msgr2=0x7f09a819c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:27.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.248+0000 7f09a6d9d640 1 --2- 172.21.15.16:0/3924046210 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09a810bb80 0x7f09a819c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.248+0000 7f09a6d9d640 1 -- 172.21.15.16:0/3924046210 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09a819ce10 con 0x7f09a81027b0 2024-03-28T22:17:27.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.249+0000 7f09a6d9d640 1 --2- 172.21.15.16:0/3924046210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09a81027b0 0x7f09a8197e20 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f099800d770 tx=0x7f099800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:27.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.249+0000 7f09a4d99640 1 -- 172.21.15.16:0/3924046210 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0998017070 con 0x7f09a81027b0 2024-03-28T22:17:27.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.249+0000 7f09ad002640 1 -- 172.21.15.16:0/3924046210 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09a819d100 con 0x7f09a81027b0 2024-03-28T22:17:27.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.249+0000 7f09a4d99640 1 -- 172.21.15.16:0/3924046210 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0998005500 con 0x7f09a81027b0 2024-03-28T22:17:27.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.249+0000 7f09a4d99640 1 -- 172.21.15.16:0/3924046210 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0998004e00 con 0x7f09a81027b0 2024-03-28T22:17:27.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.249+0000 7f09ad002640 1 -- 172.21.15.16:0/3924046210 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09a81a4990 con 0x7f09a81027b0 2024-03-28T22:17:27.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.251+0000 7f09a4d99640 1 -- 172.21.15.16:0/3924046210 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f0998004fa0 con 0x7f09a81027b0 2024-03-28T22:17:27.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.251+0000 7f09afbdd740 1 Processor -- start 2024-03-28T22:17:27.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.251+0000 7f09afbdd740 1 -- start start 2024-03-28T22:17:27.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.252+0000 7f09afbdd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b4faa470 0x55f6b4faa850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:27.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.252+0000 7f09afbdd740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b4faad90 0x55f6b4fb3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:27.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.252+0000 7f09a4d99640 1 --2- 172.21.15.16:0/3924046210 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f09940a1ef0 0x7f09940a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:27.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.252+0000 7f09afbdd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b4fb3970 0x55f6b4fb5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:27.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.252+0000 7f09afbdd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f6b4f0e180 con 0x55f6b4faa470 2024-03-28T22:17:27.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.252+0000 7f09afbdd740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f6b4eb6b70 con 0x55f6b4faad90 2024-03-28T22:17:27.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.252+0000 7f09afbdd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f6b4f0a220 con 0x55f6b4fb3970 2024-03-28T22:17:27.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.252+0000 7f09a759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b4faa470 0x55f6b4faa850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:27.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.252+0000 7f09a6d9d640 1 --2- 172.21.15.16:0/3924046210 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f09940a1ef0 0x7f09940a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:27.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.252+0000 7f099ffff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b4faad90 0x55f6b4fb3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:27.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.252+0000 7f09a759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b4faa470 0x55f6b4faa850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45716/0 (socket says 172.21.15.16:45716) 2024-03-28T22:17:27.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.252+0000 7f09a759e640 1 -- 172.21.15.16:0/4135313104 learned_addr learned my addr 172.21.15.16:0/4135313104 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:27.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.252+0000 7f09a4d99640 1 -- 172.21.15.16:0/3924046210 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8527+0+0 (secure 0 0 0) 0x7f09980cdba0 con 0x7f09a81027b0 2024-03-28T22:17:27.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.252+0000 7f09a759e640 1 --2- 172.21.15.16:0/4135313104 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b4fb3970 0x55f6b4fb5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:27.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.252+0000 7f09a759e640 1 -- 172.21.15.16:0/4135313104 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b4fb3970 msgr2=0x55f6b4fb5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:27.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.252+0000 7f09a759e640 1 --2- 172.21.15.16:0/4135313104 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b4fb3970 0x55f6b4fb5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.252+0000 7f09a759e640 1 -- 172.21.15.16:0/4135313104 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b4faad90 msgr2=0x55f6b4fb3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:27.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.252+0000 7f09a759e640 1 --2- 172.21.15.16:0/4135313104 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b4faad90 0x55f6b4fb3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.252+0000 7f09a759e640 1 -- 172.21.15.16:0/4135313104 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6b4eca7c0 con 0x55f6b4faa470 2024-03-28T22:17:27.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.252+0000 7f099ffff640 1 --2- 172.21.15.16:0/4135313104 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b4faad90 0x55f6b4fb3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:17:27.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.252+0000 7f09a6d9d640 1 --2- 172.21.15.16:0/3924046210 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f09940a1ef0 0x7f09940a43b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f099806d620 tx=0x7f0998095040 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:27.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.253+0000 7f09a759e640 1 --2- 172.21.15.16:0/4135313104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b4faa470 0x55f6b4faa850 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f09a0037040 tx=0x7f09a0009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9ddd3f2bd976916 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:27.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.253+0000 7f099d7fa640 1 -- 172.21.15.16:0/4135313104 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a0048070 con 0x55f6b4faa470 2024-03-28T22:17:27.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.253+0000 7f099d7fa640 1 -- 172.21.15.16:0/4135313104 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09a0032d50 con 0x55f6b4faa470 2024-03-28T22:17:27.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.253+0000 7f099d7fa640 1 -- 172.21.15.16:0/4135313104 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a0039040 con 0x55f6b4faa470 2024-03-28T22:17:27.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.253+0000 7f09afbdd740 1 -- 172.21.15.16:0/4135313104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b4faa470 msgr2=0x55f6b4faa850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:27.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.253+0000 7f09afbdd740 1 --2- 172.21.15.16:0/4135313104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b4faa470 0x55f6b4faa850 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f09a0037040 tx=0x7f09a0009360 comp rx=0 tx=0).stop 2024-03-28T22:17:27.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.254+0000 7f09afbdd740 1 -- 172.21.15.16:0/4135313104 shutdown_connections 2024-03-28T22:17:27.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.254+0000 7f09afbdd740 1 --2- 172.21.15.16:0/4135313104 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b4fb3970 0x55f6b4fb5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.254+0000 7f09afbdd740 1 --2- 172.21.15.16:0/4135313104 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b4faad90 0x55f6b4fb3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.254+0000 7f09afbdd740 1 --2- 172.21.15.16:0/4135313104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b4faa470 0x55f6b4faa850 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.254+0000 7f09afbdd740 1 -- 172.21.15.16:0/4135313104 >> 172.21.15.16:0/4135313104 conn(0x55f6b4fa4ba0 msgr2=0x55f6b4fa4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:27.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.254+0000 7f09afbdd740 1 -- 172.21.15.16:0/4135313104 shutdown_connections 2024-03-28T22:17:27.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.254+0000 7f09afbdd740 1 -- 172.21.15.16:0/4135313104 wait complete. 2024-03-28T22:17:27.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.254+0000 7f09afbdd740 1 -- start start 2024-03-28T22:17:27.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.254+0000 7f09afbdd740 1 Processor -- start 2024-03-28T22:17:27.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.254+0000 7f09afbdd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b4faa470 0x55f6b4fb09c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:27.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.254+0000 7f09afbdd740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b4faad90 0x55f6b4fb0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:27.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.254+0000 7f09afbdd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b4fb3970 0x55f6b4fb1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:27.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.254+0000 7f09afbdd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f6b4fb6700 con 0x55f6b4fb3970 2024-03-28T22:17:27.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.254+0000 7f09afbdd740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f6b4eac4a0 con 0x55f6b4faad90 2024-03-28T22:17:27.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.254+0000 7f09afbdd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f6b4fb6880 con 0x55f6b4faa470 2024-03-28T22:17:27.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.254+0000 7f09afbdd740 10 client.0 _ll_register_callbacks cb 0x55f6b4eafd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:17:27.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.254+0000 7f09a759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b4fb3970 0x55f6b4fb1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:27.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.254+0000 7f09a759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b4fb3970 0x55f6b4fb1440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45728/0 (socket says 172.21.15.16:45728) 2024-03-28T22:17:27.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.254+0000 7f09a759e640 1 -- 172.21.15.16:0/260604609 learned_addr learned my addr 172.21.15.16:0/260604609 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:27.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.255+0000 7f099ffff640 1 --2- 172.21.15.16:0/260604609 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b4faad90 0x55f6b4fb0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:27.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.255+0000 7f09a759e640 1 --2- 172.21.15.16:0/260604609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b4faa470 0x55f6b4fb09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:27.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.255+0000 7f09a759e640 1 -- 172.21.15.16:0/260604609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b4faa470 msgr2=0x55f6b4fb09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:27.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.255+0000 7f09a759e640 1 --2- 172.21.15.16:0/260604609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b4faa470 0x55f6b4fb09c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.255+0000 7f09a759e640 1 -- 172.21.15.16:0/260604609 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b4faad90 msgr2=0x55f6b4fb0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:27.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.255+0000 7f09a759e640 1 --2- 172.21.15.16:0/260604609 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b4faad90 0x55f6b4fb0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.255+0000 7f09a759e640 1 -- 172.21.15.16:0/260604609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6b4cec7d0 con 0x55f6b4fb3970 2024-03-28T22:17:27.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.255+0000 7f099ffff640 1 --2- 172.21.15.16:0/260604609 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b4faad90 0x55f6b4fb0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:17:27.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.255+0000 7f09a759e640 1 --2- 172.21.15.16:0/260604609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b4fb3970 0x55f6b4fb1440 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f09a0033000 tx=0x7f09a0034a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7123d7b00bf23a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:27.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.255+0000 7f098e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:17:27.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.255+0000 7f098e7fc640 1 -- 172.21.15.16:0/260604609 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a0048070 con 0x55f6b4fb3970 2024-03-28T22:17:27.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.255+0000 7f09afbdd740 1 -- 172.21.15.16:0/260604609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09a0032d50 con 0x55f6b4fb3970 2024-03-28T22:17:27.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.255+0000 7f098e7fc640 1 -- 172.21.15.16:0/260604609 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8527+0+0 (secure 0 0 0) 0x7f09a0032870 con 0x55f6b4fb3970 2024-03-28T22:17:27.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.255+0000 7f098e7fc640 1 -- 172.21.15.16:0/260604609 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f09a0058050 con 0x55f6b4fb3970 2024-03-28T22:17:27.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.255+0000 7f09afbdd740 10 client.16998 fetch_fsmap learned FSMap version 27 2024-03-28T22:17:27.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.255+0000 7f09afbdd740 1 -- 172.21.15.16:0/260604609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f6b4c223f0 con 0x55f6b4fb3970 2024-03-28T22:17:27.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.256+0000 7f098e7fc640 1 -- 172.21.15.16:0/260604609 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09a0032d50 con 0x55f6b4fb3970 2024-03-28T22:17:27.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.256+0000 7f098e7fc640 1 -- 172.21.15.16:0/260604609 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f09a003f040 con 0x55f6b4fb3970 2024-03-28T22:17:27.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.256+0000 7f09afbdd740 10 client.16998 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:17:27.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.256+0000 7f09afbdd740 10 client.16998 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:17:27.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.256+0000 7f09afbdd740 20 client.16998 populate_metadata read hostname 'smithi016' 2024-03-28T22:17:27.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.256+0000 7f09afbdd740 1 --2- 172.21.15.16:0/260604609 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f6b4fc3b70 0x55f6b4fc5f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:27.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.256+0000 7f09afbdd740 4 client.16998 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:17:27.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.256+0000 7f09afbdd740 1 -- 172.21.15.16:0/260604609 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f6b4f1b6c0 con 0x55f6b4fc3b70 2024-03-28T22:17:27.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.256+0000 7f099ffff640 1 --2- 172.21.15.16:0/260604609 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f6b4fc3b70 0x55f6b4fc5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:27.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.257+0000 7f099ffff640 1 --2- 172.21.15.16:0/260604609 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f6b4fc3b70 0x55f6b4fc5f60 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=636c52e82bbe5d4a server_cookie=c29ab7ab6cb8f088 in_seq=0 out_seq=0 2024-03-28T22:17:27.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.257+0000 7f098e7fc640 10 client.16998 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:17:27.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.258+0000 7f098e7fc640 1 -- 172.21.15.16:0/260604609 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f6b4f1b6c0 con 0x55f6b4fc3b70 2024-03-28T22:17:27.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.258+0000 7f098e7fc640 10 client.16998 handle_command_reply: tid=0 2024-03-28T22:17:27.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.258+0000 7f09afbdd740 1 client.16998 shutdown 2024-03-28T22:17:27.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.258+0000 7f09afbdd740 1 -- 172.21.15.16:0/260604609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b4fb3970 msgr2=0x55f6b4fb1440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:27.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.258+0000 7f09afbdd740 1 --2- 172.21.15.16:0/260604609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b4fb3970 0x55f6b4fb1440 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f09a0033000 tx=0x7f09a0034a20 comp rx=0 tx=0).stop 2024-03-28T22:17:27.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.258+0000 7f09afbdd740 1 -- 172.21.15.16:0/260604609 shutdown_connections 2024-03-28T22:17:27.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.258+0000 7f09afbdd740 1 --2- 172.21.15.16:0/260604609 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f6b4fc3b70 0x55f6b4fc5f60 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.258+0000 7f09afbdd740 1 --2- 172.21.15.16:0/260604609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b4fb3970 0x55f6b4fb1440 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.258+0000 7f09afbdd740 1 --2- 172.21.15.16:0/260604609 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b4faad90 0x55f6b4fb0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.258+0000 7f09afbdd740 1 --2- 172.21.15.16:0/260604609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b4faa470 0x55f6b4fb09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.258+0000 7f098e7fc640 0 client.16998 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:17:27.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.258+0000 7f09afbdd740 1 -- 172.21.15.16:0/260604609 >> 172.21.15.16:0/260604609 conn(0x55f6b4fa4ba0 msgr2=0x55f6b4fa68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:27.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.258+0000 7f09afbdd740 1 -- 172.21.15.16:0/260604609 shutdown_connections 2024-03-28T22:17:27.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.258+0000 7f09afbdd740 1 -- 172.21.15.16:0/260604609 wait complete. 2024-03-28T22:17:27.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.258+0000 7f09afbdd740 20 client.16998 trim_cache size 0 max 16384 2024-03-28T22:17:27.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.277+0000 7f09afbdd740 1 Processor -- start 2024-03-28T22:17:27.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.277+0000 7f09afbdd740 1 -- start start 2024-03-28T22:17:27.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.278+0000 7f09afbdd740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b4faa470 0x55f6b4fb30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:27.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.278+0000 7f09afbdd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b4faad90 0x55f6b4faf9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:27.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.278+0000 7f09afbdd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b4fb3970 0x55f6b4fb1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:27.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.278+0000 7f09afbdd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f09a81afa90 con 0x55f6b4fb3970 2024-03-28T22:17:27.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.278+0000 7f09afbdd740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f09a81ad360 con 0x55f6b4faa470 2024-03-28T22:17:27.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.278+0000 7f09afbdd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f09a81ab1d0 con 0x55f6b4faad90 2024-03-28T22:17:27.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.278+0000 7f09a759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b4fb3970 0x55f6b4fb1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:27.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.278+0000 7f099ffff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b4faad90 0x55f6b4faf9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:27.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.278+0000 7f09a759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b4fb3970 0x55f6b4fb1090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45732/0 (socket says 172.21.15.16:45732) 2024-03-28T22:17:27.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.278+0000 7f09a759e640 1 -- 172.21.15.16:0/131085194 learned_addr learned my addr 172.21.15.16:0/131085194 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:27.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.278+0000 7f09a759e640 1 --2- 172.21.15.16:0/131085194 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b4faa470 0x55f6b4fb30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:27.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.278+0000 7f09a759e640 1 -- 172.21.15.16:0/131085194 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b4faad90 msgr2=0x55f6b4faf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:27.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.278+0000 7f09a759e640 1 --2- 172.21.15.16:0/131085194 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b4faad90 0x55f6b4faf9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.278+0000 7f09a759e640 1 -- 172.21.15.16:0/131085194 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b4faa470 msgr2=0x55f6b4fb30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:27.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.278+0000 7f09a759e640 1 --2- 172.21.15.16:0/131085194 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b4faa470 0x55f6b4fb30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.278+0000 7f09a759e640 1 -- 172.21.15.16:0/131085194 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6b4be20f0 con 0x55f6b4fb3970 2024-03-28T22:17:27.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.278+0000 7f099ffff640 1 --2- 172.21.15.16:0/131085194 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b4faad90 0x55f6b4faf9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:17:27.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.278+0000 7f09a759e640 1 --2- 172.21.15.16:0/131085194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b4fb3970 0x55f6b4fb1090 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f09a00436b0 tx=0x7f09a0034dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2d3770c470f4845 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:27.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.278+0000 7f099cff9640 1 -- 172.21.15.16:0/131085194 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a0048070 con 0x55f6b4fb3970 2024-03-28T22:17:27.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.279+0000 7f099cff9640 1 -- 172.21.15.16:0/131085194 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09a0049040 con 0x55f6b4fb3970 2024-03-28T22:17:27.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.279+0000 7f099cff9640 1 -- 172.21.15.16:0/131085194 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a0039040 con 0x55f6b4fb3970 2024-03-28T22:17:27.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.279+0000 7f09afbdd740 1 -- 172.21.15.16:0/131085194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b4fb3970 msgr2=0x55f6b4fb1090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:27.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.279+0000 7f09afbdd740 1 --2- 172.21.15.16:0/131085194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b4fb3970 0x55f6b4fb1090 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f09a00436b0 tx=0x7f09a0034dd0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.279+0000 7f09afbdd740 1 -- 172.21.15.16:0/131085194 shutdown_connections 2024-03-28T22:17:27.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.279+0000 7f09afbdd740 1 --2- 172.21.15.16:0/131085194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b4fb3970 0x55f6b4fb1090 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.279+0000 7f09afbdd740 1 --2- 172.21.15.16:0/131085194 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b4faad90 0x55f6b4faf9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.279+0000 7f09afbdd740 1 --2- 172.21.15.16:0/131085194 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b4faa470 0x55f6b4fb30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.279+0000 7f09afbdd740 1 -- 172.21.15.16:0/131085194 >> 172.21.15.16:0/131085194 conn(0x55f6b4fa4ba0 msgr2=0x55f6b4fb0980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:27.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.279+0000 7f09afbdd740 1 -- 172.21.15.16:0/131085194 shutdown_connections 2024-03-28T22:17:27.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.280+0000 7f09afbdd740 1 -- 172.21.15.16:0/131085194 wait complete. 2024-03-28T22:17:27.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.280+0000 7f09afbdd740 1 -- start start 2024-03-28T22:17:27.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.280+0000 7f09afbdd740 1 Processor -- start 2024-03-28T22:17:27.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.280+0000 7f09afbdd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b4faa470 0x55f6b4fc7100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:27.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.280+0000 7f09afbdd740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b4faad90 0x55f6b4fc7640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:27.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.280+0000 7f09afbdd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b4fb3970 0x55f6b5022ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:27.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.280+0000 7f09afbdd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f09a81ab820 con 0x55f6b4fb3970 2024-03-28T22:17:27.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.280+0000 7f09afbdd740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f09a81acc90 con 0x55f6b4faad90 2024-03-28T22:17:27.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.280+0000 7f09afbdd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f09a81abef0 con 0x55f6b4faa470 2024-03-28T22:17:27.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.280+0000 7f09afbdd740 10 client.0 _ll_register_callbacks cb 0x7f09a0046070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:17:27.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.280+0000 7f09a759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b4fb3970 0x55f6b5022ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:27.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.280+0000 7f099ffff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b4faad90 0x55f6b4fc7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:27.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.280+0000 7f09a759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b4fb3970 0x55f6b5022ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45734/0 (socket says 172.21.15.16:45734) 2024-03-28T22:17:27.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.280+0000 7f09a759e640 1 -- 172.21.15.16:0/911605199 learned_addr learned my addr 172.21.15.16:0/911605199 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:27.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.280+0000 7f09a759e640 1 --2- 172.21.15.16:0/911605199 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b4faa470 0x55f6b4fc7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:27.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.280+0000 7f09a759e640 1 -- 172.21.15.16:0/911605199 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b4faa470 msgr2=0x55f6b4fc7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:27.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.280+0000 7f09a759e640 1 --2- 172.21.15.16:0/911605199 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b4faa470 0x55f6b4fc7100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.281+0000 7f09a759e640 1 -- 172.21.15.16:0/911605199 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b4faad90 msgr2=0x55f6b4fc7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:27.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.281+0000 7f09a759e640 1 --2- 172.21.15.16:0/911605199 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b4faad90 0x55f6b4fc7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.281+0000 7f09a759e640 1 -- 172.21.15.16:0/911605199 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6b50230d0 con 0x55f6b4fb3970 2024-03-28T22:17:27.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.281+0000 7f099ffff640 1 --2- 172.21.15.16:0/911605199 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b4faad90 0x55f6b4fc7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:17:27.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.281+0000 7f09a759e640 1 --2- 172.21.15.16:0/911605199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b4fb3970 0x55f6b5022ae0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f09a0043590 tx=0x7f09a0032c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=283f14b8e9d7550f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:27.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.281+0000 7f098e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:17:27.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.281+0000 7f098e7fc640 1 -- 172.21.15.16:0/911605199 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a0048070 con 0x55f6b4fb3970 2024-03-28T22:17:27.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.281+0000 7f098e7fc640 1 -- 172.21.15.16:0/911605199 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8527+0+0 (secure 0 0 0) 0x7f09a0032880 con 0x55f6b4fb3970 2024-03-28T22:17:27.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.281+0000 7f09afbdd740 1 -- 172.21.15.16:0/911605199 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09a0049040 con 0x55f6b4fb3970 2024-03-28T22:17:27.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.281+0000 7f098e7fc640 1 -- 172.21.15.16:0/911605199 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f09a0039040 con 0x55f6b4fb3970 2024-03-28T22:17:27.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.281+0000 7f09afbdd740 10 client.17004 fetch_fsmap learned FSMap version 27 2024-03-28T22:17:27.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.281+0000 7f09afbdd740 1 -- 172.21.15.16:0/911605199 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f6b502a460 con 0x55f6b4fb3970 2024-03-28T22:17:27.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.282+0000 7f098e7fc640 1 -- 172.21.15.16:0/911605199 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09a0049040 con 0x55f6b4fb3970 2024-03-28T22:17:27.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.282+0000 7f098e7fc640 1 -- 172.21.15.16:0/911605199 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f09a003f070 con 0x55f6b4fb3970 2024-03-28T22:17:27.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.282+0000 7f09afbdd740 10 client.17004 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:17:27.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.282+0000 7f09afbdd740 10 client.17004 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:17:27.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.282+0000 7f09afbdd740 20 client.17004 populate_metadata read hostname 'smithi016' 2024-03-28T22:17:27.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.282+0000 7f09afbdd740 1 --2- 172.21.15.16:0/911605199 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f6b502aab0 0x55f6b502ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:27.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.282+0000 7f09afbdd740 4 client.17004 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:17:27.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.282+0000 7f09afbdd740 1 -- 172.21.15.16:0/911605199 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f6b4fb0070 con 0x55f6b502aab0 2024-03-28T22:17:27.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.282+0000 7f099ffff640 1 --2- 172.21.15.16:0/911605199 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f6b502aab0 0x55f6b502ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:27.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.283+0000 7f098e7fc640 10 client.17004 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:17:27.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.283+0000 7f099ffff640 1 --2- 172.21.15.16:0/911605199 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f6b502aab0 0x55f6b502ced0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7b3ece98c41b7cd server_cookie=ca6b3deae3b2870d in_seq=0 out_seq=0 2024-03-28T22:17:27.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.283+0000 7f098e7fc640 1 -- 172.21.15.16:0/911605199 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55f6b4fb0070 con 0x55f6b502aab0 2024-03-28T22:17:27.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.283+0000 7f098e7fc640 10 client.17004 handle_command_reply: tid=0 2024-03-28T22:17:27.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.283+0000 7f09afbdd740 1 client.17004 shutdown 2024-03-28T22:17:27.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.283+0000 7f09afbdd740 1 -- 172.21.15.16:0/911605199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b4fb3970 msgr2=0x55f6b5022ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:27.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.283+0000 7f09afbdd740 1 --2- 172.21.15.16:0/911605199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b4fb3970 0x55f6b5022ae0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f09a0043590 tx=0x7f09a0032c90 comp rx=0 tx=0).stop 2024-03-28T22:17:27.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.283+0000 7f09afbdd740 1 -- 172.21.15.16:0/911605199 shutdown_connections 2024-03-28T22:17:27.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.283+0000 7f09afbdd740 1 --2- 172.21.15.16:0/911605199 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55f6b502aab0 0x55f6b502ced0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.283+0000 7f09afbdd740 1 --2- 172.21.15.16:0/911605199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6b4fb3970 0x55f6b5022ae0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.283+0000 7f09afbdd740 1 --2- 172.21.15.16:0/911605199 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6b4faad90 0x55f6b4fc7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.283+0000 7f098e7fc640 0 client.17004 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:17:27.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.283+0000 7f09afbdd740 1 --2- 172.21.15.16:0/911605199 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6b4faa470 0x55f6b4fc7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.283+0000 7f09afbdd740 1 -- 172.21.15.16:0/911605199 >> 172.21.15.16:0/911605199 conn(0x55f6b4fa4ba0 msgr2=0x55f6b4fbf390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:27.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.283+0000 7f09afbdd740 1 -- 172.21.15.16:0/911605199 shutdown_connections 2024-03-28T22:17:27.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.283+0000 7f09afbdd740 1 -- 172.21.15.16:0/911605199 wait complete. 2024-03-28T22:17:27.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.283+0000 7f09afbdd740 20 client.17004 trim_cache size 0 max 16384 2024-03-28T22:17:27.294 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:17:27.294 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T22:17:27.295 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-03-28T22:17:27.295 INFO:teuthology.orchestra.run.smithi016.stdout: "3719abed-052e-49da-980e-def6fdeed7ad": { 2024-03-28T22:17:27.295 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-03-28T22:17:27.295 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "3719abed-052e-49da-980e-def6fdeed7ad", 2024-03-28T22:17:27.295 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-03-28T22:17:27.295 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:17:27.295 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:17:27.295 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:17:27.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.284+0000 7f09ad002640 1 -- 172.21.15.16:0/3924046210 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f09940a1ef0 msgr2=0x7f09940a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:27.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.284+0000 7f09ad002640 1 --2- 172.21.15.16:0/3924046210 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f09940a1ef0 0x7f09940a43b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f099806d620 tx=0x7f0998095040 comp rx=0 tx=0).stop 2024-03-28T22:17:27.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.284+0000 7f09ad002640 1 -- 172.21.15.16:0/3924046210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09a81027b0 msgr2=0x7f09a8197e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:27.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.284+0000 7f09ad002640 1 --2- 172.21.15.16:0/3924046210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09a81027b0 0x7f09a8197e20 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f099800d770 tx=0x7f099800dc30 comp rx=0 tx=0).stop 2024-03-28T22:17:27.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.284+0000 7f09ad002640 1 -- 172.21.15.16:0/3924046210 shutdown_connections 2024-03-28T22:17:27.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.284+0000 7f09ad002640 1 --2- 172.21.15.16:0/3924046210 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f09940a1ef0 0x7f09940a43b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.284+0000 7f09ad002640 1 --2- 172.21.15.16:0/3924046210 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09a810bb80 0x7f09a819c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.284+0000 7f09ad002640 1 --2- 172.21.15.16:0/3924046210 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09a81030d0 0x7f09a8198360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.284+0000 7f09ad002640 1 --2- 172.21.15.16:0/3924046210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09a81027b0 0x7f09a8197e20 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:27.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.284+0000 7f09ad002640 1 -- 172.21.15.16:0/3924046210 >> 172.21.15.16:0/3924046210 conn(0x7f09a80fbf50 msgr2=0x7f09a80fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:27.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.284+0000 7f09ad002640 1 -- 172.21.15.16:0/3924046210 shutdown_connections 2024-03-28T22:17:27.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:27.284+0000 7f09ad002640 1 -- 172.21.15.16:0/3924046210 wait complete. 2024-03-28T22:17:27.298 INFO:tasks.cephfs.filesystem:scrub status for tag:3719abed-052e-49da-980e-def6fdeed7ad - {'path': '/', 'tag': '3719abed-052e-49da-980e-def6fdeed7ad', 'options': 'recursive,force'} 2024-03-28T22:17:27.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:27 smithi016 ceph-mon[32374]: pgmap v211: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T22:17:27.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:27 smithi053 ceph-mon[35979]: pgmap v211: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T22:17:27.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:27 smithi063 ceph-mon[36249]: pgmap v211: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-28T22:17:28.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:28 smithi053 ceph-mon[35979]: scrub queued for path: / 2024-03-28T22:17:28.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:28 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:17:28.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:28 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:17:28.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:28 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:17:28.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:28 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:17:28.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:28 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:17:28.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:28 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:17:28.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:28 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:17:28.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:28 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:17:28.799 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:28 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:17:28.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:28 smithi063 ceph-mon[36249]: scrub queued for path: / 2024-03-28T22:17:28.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:28 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:17:28.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:28 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:17:28.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:28 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:17:28.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:28 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:17:28.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:28 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:17:28.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:28 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:17:28.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:28 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:17:28.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:28 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:17:28.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:28 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:17:28.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:28 smithi016 ceph-mon[32374]: scrub queued for path: / 2024-03-28T22:17:28.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:28 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:17:28.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:28 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:17:28.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:28 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:17:28.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:28 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:17:28.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:28 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:17:28.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:28 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:17:28.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:28 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:17:28.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:28 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:17:28.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:28 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:17:29.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:29 smithi053 ceph-mon[35979]: pgmap v212: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:17:29.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:29 smithi063 ceph-mon[36249]: pgmap v212: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:17:29.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:29 smithi016 ceph-mon[32374]: pgmap v212: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:17:31.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:31 smithi053 ceph-mon[35979]: pgmap v213: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:17:31.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:31 smithi063 ceph-mon[36249]: pgmap v213: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:17:31.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:31 smithi016 ceph-mon[32374]: pgmap v213: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:17:33.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:33 smithi053 ceph-mon[35979]: pgmap v214: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:17:33.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:33 smithi063 ceph-mon[36249]: pgmap v214: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:17:33.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:33 smithi016 ceph-mon[32374]: pgmap v214: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:17:35.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:35 smithi053 ceph-mon[35979]: pgmap v215: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:17:35.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:35 smithi063 ceph-mon[36249]: pgmap v215: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:17:35.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:35 smithi016 ceph-mon[32374]: pgmap v215: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:17:37.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:37 smithi053 ceph-mon[35979]: pgmap v216: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:17:37.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:37 smithi063 ceph-mon[36249]: pgmap v216: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:17:37.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:37 smithi016 ceph-mon[32374]: pgmap v216: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:17:39.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:39 smithi053 ceph-mon[35979]: pgmap v217: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:17:39.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:39 smithi063 ceph-mon[36249]: pgmap v217: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:17:39.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:39 smithi016 ceph-mon[32374]: pgmap v217: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:17:40.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:40 smithi053 ceph-mon[35979]: scrub summary: idle 2024-03-28T22:17:40.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:40 smithi063 ceph-mon[36249]: scrub summary: idle 2024-03-28T22:17:40.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:40 smithi016 ceph-mon[32374]: scrub summary: idle 2024-03-28T22:17:41.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:41 smithi053 ceph-mon[35979]: scrub summary: idle 2024-03-28T22:17:41.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:41 smithi053 ceph-mon[35979]: pgmap v218: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:17:41.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:41 smithi063 ceph-mon[36249]: scrub summary: idle 2024-03-28T22:17:41.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:41 smithi063 ceph-mon[36249]: pgmap v218: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:17:41.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:41 smithi016 ceph-mon[32374]: scrub summary: idle 2024-03-28T22:17:41.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:41 smithi016 ceph-mon[32374]: pgmap v218: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:17:43.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:43 smithi053 ceph-mon[35979]: pgmap v219: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:17:43.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:43 smithi063 ceph-mon[36249]: pgmap v219: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:17:43.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:43 smithi016 ceph-mon[32374]: pgmap v219: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:17:45.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:45 smithi053 ceph-mon[35979]: pgmap v220: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:17:45.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:45 smithi063 ceph-mon[36249]: pgmap v220: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:17:45.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:45 smithi016 ceph-mon[32374]: pgmap v220: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:17:46.797 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:46 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:17:46.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:46 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:17:46.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:46 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:17:47.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:47 smithi063 ceph-mon[36249]: pgmap v221: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:17:47.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:47 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:17:47.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:47 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:17:47.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:47 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:17:47.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:47 smithi016 ceph-mon[32374]: pgmap v221: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:17:47.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:47 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:17:47.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:47 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:17:47.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:47 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:17:48.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:47 smithi053 ceph-mon[35979]: pgmap v221: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:17:48.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:47 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:17:48.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:47 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:17:48.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:47 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:17:49.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:49 smithi063 ceph-mon[36249]: pgmap v222: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:17:49.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:49 smithi016 ceph-mon[32374]: pgmap v222: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:17:50.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:49 smithi053 ceph-mon[35979]: pgmap v222: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:17:51.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:51 smithi063 ceph-mon[36249]: pgmap v223: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:17:51.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:51 smithi016 ceph-mon[32374]: pgmap v223: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:17:52.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:51 smithi053 ceph-mon[35979]: pgmap v223: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:17:53.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:53 smithi063 ceph-mon[36249]: pgmap v224: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:17:53.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:53 smithi016 ceph-mon[32374]: pgmap v224: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:17:54.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:53 smithi053 ceph-mon[35979]: pgmap v224: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:17:55.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:55 smithi063 ceph-mon[36249]: pgmap v225: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:17:55.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:55 smithi016 ceph-mon[32374]: pgmap v225: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:17:56.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:55 smithi053 ceph-mon[35979]: pgmap v225: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:17:57.299 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:17:57.300 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:17:57.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.417+0000 7f75a7207640 1 -- 172.21.15.16:0/2280767462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75a0106690 msgr2=0x7f75a0106a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:57.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.417+0000 7f75a7207640 1 --2- 172.21.15.16:0/2280767462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75a0106690 0x7f75a0106a70 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f7594009930 tx=0x7f759402bc30 comp rx=0 tx=0).stop 2024-03-28T22:17:57.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.417+0000 7f75a7207640 1 -- 172.21.15.16:0/2280767462 shutdown_connections 2024-03-28T22:17:57.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.417+0000 7f75a7207640 1 --2- 172.21.15.16:0/2280767462 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75a0100fb0 0x7f75a010d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.417+0000 7f75a7207640 1 --2- 172.21.15.16:0/2280767462 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75a01005f0 0x7f75a0100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.417+0000 7f75a7207640 1 --2- 172.21.15.16:0/2280767462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75a0106690 0x7f75a0106a70 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.417+0000 7f75a7207640 1 -- 172.21.15.16:0/2280767462 >> 172.21.15.16:0/2280767462 conn(0x7f75a0077800 msgr2=0x7f75a0077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:57.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.417+0000 7f75a7207640 1 -- 172.21.15.16:0/2280767462 shutdown_connections 2024-03-28T22:17:57.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.417+0000 7f75a7207640 1 -- 172.21.15.16:0/2280767462 wait complete. 2024-03-28T22:17:57.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.417+0000 7f75a7207640 1 Processor -- start 2024-03-28T22:17:57.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.417+0000 7f75a7207640 1 -- start start 2024-03-28T22:17:57.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.418+0000 7f75a7207640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75a01005f0 0x7f75a0102290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.418+0000 7f75a7207640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75a0100fb0 0x7f75a01027d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.418+0000 7f75a7207640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75a0106690 0x7f75a0102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.418+0000 7f75a7207640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f75a0110a10 con 0x7f75a01005f0 2024-03-28T22:17:57.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.418+0000 7f75a7207640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f75a0110890 con 0x7f75a0100fb0 2024-03-28T22:17:57.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.418+0000 7f75a7207640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f75a0110b90 con 0x7f75a0106690 2024-03-28T22:17:57.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.418+0000 7f75a4fa3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75a01005f0 0x7f75a0102290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:57.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.418+0000 7f75a4fa3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75a01005f0 0x7f75a0102290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55014/0 (socket says 172.21.15.16:55014) 2024-03-28T22:17:57.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.418+0000 7f75a4fa3640 1 -- 172.21.15.16:0/566419330 learned_addr learned my addr 172.21.15.16:0/566419330 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:57.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.418+0000 7f759ffff640 1 --2- 172.21.15.16:0/566419330 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75a0100fb0 0x7f75a01027d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:57.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.418+0000 7f75a57a4640 1 --2- 172.21.15.16:0/566419330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75a0106690 0x7f75a0102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:57.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.418+0000 7f75a4fa3640 1 -- 172.21.15.16:0/566419330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75a0106690 msgr2=0x7f75a0102d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:57.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.418+0000 7f75a4fa3640 1 --2- 172.21.15.16:0/566419330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75a0106690 0x7f75a0102d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.418+0000 7f75a4fa3640 1 -- 172.21.15.16:0/566419330 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75a0100fb0 msgr2=0x7f75a01027d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:57.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.418+0000 7f75a4fa3640 1 --2- 172.21.15.16:0/566419330 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75a0100fb0 0x7f75a01027d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.418+0000 7f75a4fa3640 1 -- 172.21.15.16:0/566419330 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75a01a6490 con 0x7f75a01005f0 2024-03-28T22:17:57.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.418+0000 7f75a57a4640 1 --2- 172.21.15.16:0/566419330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75a0106690 0x7f75a0102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:17:57.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.418+0000 7f759ffff640 1 --2- 172.21.15.16:0/566419330 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75a0100fb0 0x7f75a01027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:17:57.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.418+0000 7f75a4fa3640 1 --2- 172.21.15.16:0/566419330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75a01005f0 0x7f75a0102290 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f7594009230 tx=0x7f759402cb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:57.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.418+0000 7f759dffb640 1 -- 172.21.15.16:0/566419330 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7594041070 con 0x7f75a01005f0 2024-03-28T22:17:57.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.419+0000 7f75a7207640 1 -- 172.21.15.16:0/566419330 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75a01a6720 con 0x7f75a01005f0 2024-03-28T22:17:57.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.419+0000 7f759dffb640 1 -- 172.21.15.16:0/566419330 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7594004450 con 0x7f75a01005f0 2024-03-28T22:17:57.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.419+0000 7f759dffb640 1 -- 172.21.15.16:0/566419330 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7594003d90 con 0x7f75a01005f0 2024-03-28T22:17:57.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.419+0000 7f75a7207640 1 -- 172.21.15.16:0/566419330 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75a01a6c60 con 0x7f75a01005f0 2024-03-28T22:17:57.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.421+0000 7f759dffb640 1 -- 172.21.15.16:0/566419330 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f7594044070 con 0x7f75a01005f0 2024-03-28T22:17:57.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.421+0000 7f75a9ded740 1 Processor -- start 2024-03-28T22:17:57.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.422+0000 7f75a9ded740 1 -- start start 2024-03-28T22:17:57.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.422+0000 7f75a9ded740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558808c69290 0x558808c69670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.422+0000 7f75a9ded740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558808c69bb0 0x558808c72250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.422+0000 7f759dffb640 1 --2- 172.21.15.16:0/566419330 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f75900a1ef0 0x7f75900a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.422+0000 7f75a9ded740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558808c72790 0x558808c74b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.422+0000 7f75a9ded740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558808bcd180 con 0x558808c69290 2024-03-28T22:17:57.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.422+0000 7f75a9ded740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x558808b75b70 con 0x558808c69bb0 2024-03-28T22:17:57.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.422+0000 7f75a9ded740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x558808bc9220 con 0x558808c72790 2024-03-28T22:17:57.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.422+0000 7f75a57a4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558808c69290 0x558808c69670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:57.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.422+0000 7f759ffff640 1 --2- 172.21.15.16:0/566419330 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f75900a1ef0 0x7f75900a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:57.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.422+0000 7f75a57a4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558808c69290 0x558808c69670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55028/0 (socket says 172.21.15.16:55028) 2024-03-28T22:17:57.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.422+0000 7f75a57a4640 1 -- 172.21.15.16:0/2670345154 learned_addr learned my addr 172.21.15.16:0/2670345154 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:57.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.422+0000 7f759dffb640 1 -- 172.21.15.16:0/566419330 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8527+0+0 (secure 0 0 0) 0x7f75940efbb0 con 0x7f75a01005f0 2024-03-28T22:17:57.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.422+0000 7f75a4fa3640 1 --2- 172.21.15.16:0/2670345154 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558808c69bb0 0x558808c72250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:57.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.422+0000 7f75a57a4640 1 --2- 172.21.15.16:0/2670345154 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558808c72790 0x558808c74b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:57.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.422+0000 7f75a4fa3640 1 -- 172.21.15.16:0/2670345154 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558808c72790 msgr2=0x558808c74b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:57.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.422+0000 7f75a4fa3640 1 --2- 172.21.15.16:0/2670345154 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558808c72790 0x558808c74b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.422+0000 7f75a4fa3640 1 -- 172.21.15.16:0/2670345154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558808c69290 msgr2=0x558808c69670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:57.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.422+0000 7f75a4fa3640 1 --2- 172.21.15.16:0/2670345154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558808c69290 0x558808c69670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.422+0000 7f75a4fa3640 1 -- 172.21.15.16:0/2670345154 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558808b897c0 con 0x558808c69bb0 2024-03-28T22:17:57.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.422+0000 7f75a57a4640 1 --2- 172.21.15.16:0/2670345154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558808c69290 0x558808c69670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:17:57.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.422+0000 7f759ffff640 1 --2- 172.21.15.16:0/566419330 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f75900a1ef0 0x7f75900a43b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f75a0103d20 tx=0x7f758c005f90 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:57.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.423+0000 7f75a4fa3640 1 --2- 172.21.15.16:0/2670345154 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558808c69bb0 0x558808c72250 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f75940ed410 tx=0x7f7594004ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be9adeee58db4d25 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:57.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.423+0000 7f758affd640 1 -- 172.21.15.16:0/2670345154 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f759404b040 con 0x558808c69bb0 2024-03-28T22:17:57.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.423+0000 7f758affd640 1 -- 172.21.15.16:0/2670345154 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75940bcdb0 con 0x558808c69bb0 2024-03-28T22:17:57.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.423+0000 7f758affd640 1 -- 172.21.15.16:0/2670345154 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75940035f0 con 0x558808c69bb0 2024-03-28T22:17:57.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.423+0000 7f75a9ded740 1 -- 172.21.15.16:0/2670345154 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558808c69bb0 msgr2=0x558808c72250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:57.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.423+0000 7f75a9ded740 1 --2- 172.21.15.16:0/2670345154 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558808c69bb0 0x558808c72250 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f75940ed410 tx=0x7f7594004ee0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.423+0000 7f75a9ded740 1 -- 172.21.15.16:0/2670345154 shutdown_connections 2024-03-28T22:17:57.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.423+0000 7f75a9ded740 1 --2- 172.21.15.16:0/2670345154 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558808c72790 0x558808c74b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.423+0000 7f75a9ded740 1 --2- 172.21.15.16:0/2670345154 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558808c69bb0 0x558808c72250 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.423+0000 7f75a9ded740 1 --2- 172.21.15.16:0/2670345154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558808c69290 0x558808c69670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.423+0000 7f75a9ded740 1 -- 172.21.15.16:0/2670345154 >> 172.21.15.16:0/2670345154 conn(0x558808c639c0 msgr2=0x558808c63da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:57.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.423+0000 7f75a9ded740 1 -- 172.21.15.16:0/2670345154 shutdown_connections 2024-03-28T22:17:57.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.423+0000 7f75a9ded740 1 -- 172.21.15.16:0/2670345154 wait complete. 2024-03-28T22:17:57.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.424+0000 7f75a9ded740 1 -- start start 2024-03-28T22:17:57.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.424+0000 7f75a9ded740 1 Processor -- start 2024-03-28T22:17:57.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.424+0000 7f75a9ded740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558808c69290 0x558808c6f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.424+0000 7f75a9ded740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558808c69bb0 0x558808c6fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.424+0000 7f75a9ded740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558808c72790 0x558808c70200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.424+0000 7f75a9ded740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558808c75340 con 0x558808c69bb0 2024-03-28T22:17:57.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.424+0000 7f75a9ded740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x558808b6b4a0 con 0x558808c69290 2024-03-28T22:17:57.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.424+0000 7f75a9ded740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x558808c754c0 con 0x558808c72790 2024-03-28T22:17:57.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.424+0000 7f75a9ded740 10 client.0 _ll_register_callbacks cb 0x558808b6ed90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:17:57.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.424+0000 7f759ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558808c69bb0 0x558808c6fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:57.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.424+0000 7f759ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558808c69bb0 0x558808c6fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55034/0 (socket says 172.21.15.16:55034) 2024-03-28T22:17:57.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.424+0000 7f759ffff640 1 -- 172.21.15.16:0/1941459224 learned_addr learned my addr 172.21.15.16:0/1941459224 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:57.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.424+0000 7f75a4fa3640 1 --2- 172.21.15.16:0/1941459224 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558808c69290 0x558808c6f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:57.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.425+0000 7f75a57a4640 1 --2- 172.21.15.16:0/1941459224 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558808c72790 0x558808c70200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:57.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.425+0000 7f759ffff640 1 -- 172.21.15.16:0/1941459224 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558808c72790 msgr2=0x558808c70200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:57.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.425+0000 7f759ffff640 1 --2- 172.21.15.16:0/1941459224 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558808c72790 0x558808c70200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.425+0000 7f759ffff640 1 -- 172.21.15.16:0/1941459224 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558808c69290 msgr2=0x558808c6f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:57.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.425+0000 7f759ffff640 1 --2- 172.21.15.16:0/1941459224 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558808c69290 0x558808c6f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.425+0000 7f759ffff640 1 -- 172.21.15.16:0/1941459224 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588089ab7d0 con 0x558808c69bb0 2024-03-28T22:17:57.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.425+0000 7f75a4fa3640 1 --2- 172.21.15.16:0/1941459224 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558808c69290 0x558808c6f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:17:57.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.425+0000 7f759ffff640 1 --2- 172.21.15.16:0/1941459224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558808c69bb0 0x558808c6fcc0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f758c003800 tx=0x7f758c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca9cddc0d6d53d1c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:57.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.425+0000 7f757ffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:17:57.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.425+0000 7f757ffff640 1 -- 172.21.15.16:0/1941459224 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f758c00f980 con 0x558808c69bb0 2024-03-28T22:17:57.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.425+0000 7f75a9ded740 1 -- 172.21.15.16:0/1941459224 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75940bcdb0 con 0x558808c69bb0 2024-03-28T22:17:57.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.425+0000 7f757ffff640 1 -- 172.21.15.16:0/1941459224 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8527+0+0 (secure 0 0 0) 0x7f758c00fb20 con 0x558808c69bb0 2024-03-28T22:17:57.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.425+0000 7f757ffff640 1 -- 172.21.15.16:0/1941459224 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f758c00fd20 con 0x558808c69bb0 2024-03-28T22:17:57.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.425+0000 7f75a9ded740 10 client.17016 fetch_fsmap learned FSMap version 27 2024-03-28T22:17:57.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.425+0000 7f75a9ded740 1 -- 172.21.15.16:0/1941459224 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5588088e13f0 con 0x558808c69bb0 2024-03-28T22:17:57.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.426+0000 7f757ffff640 1 -- 172.21.15.16:0/1941459224 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75940bcdb0 con 0x558808c69bb0 2024-03-28T22:17:57.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.426+0000 7f757ffff640 1 -- 172.21.15.16:0/1941459224 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f758c01a070 con 0x558808c69bb0 2024-03-28T22:17:57.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.426+0000 7f75a9ded740 10 client.17016 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:17:57.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.426+0000 7f75a9ded740 10 client.17016 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:17:57.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.426+0000 7f75a9ded740 20 client.17016 populate_metadata read hostname 'smithi016' 2024-03-28T22:17:57.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.426+0000 7f75a9ded740 1 --2- 172.21.15.16:0/1941459224 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558808c827f0 0x558808c84c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.426+0000 7f75a9ded740 4 client.17016 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:17:57.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.426+0000 7f75a9ded740 1 -- 172.21.15.16:0/1941459224 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558808bda6c0 con 0x558808c827f0 2024-03-28T22:17:57.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.426+0000 7f75a4fa3640 1 --2- 172.21.15.16:0/1941459224 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558808c827f0 0x558808c84c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:57.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.426+0000 7f757ffff640 10 client.17016 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:17:57.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.426+0000 7f75a4fa3640 1 --2- 172.21.15.16:0/1941459224 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558808c827f0 0x558808c84c50 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76d158d778fbb6ea server_cookie=3a839c0742e3affd in_seq=0 out_seq=0 2024-03-28T22:17:57.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.427+0000 7f757ffff640 1 -- 172.21.15.16:0/1941459224 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558808bda6c0 con 0x558808c827f0 2024-03-28T22:17:57.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.427+0000 7f757ffff640 10 client.17016 handle_command_reply: tid=0 2024-03-28T22:17:57.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.427+0000 7f75a9ded740 1 client.17016 shutdown 2024-03-28T22:17:57.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.427+0000 7f75a9ded740 1 -- 172.21.15.16:0/1941459224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558808c69bb0 msgr2=0x558808c6fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:57.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.427+0000 7f75a9ded740 1 --2- 172.21.15.16:0/1941459224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558808c69bb0 0x558808c6fcc0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f758c003800 tx=0x7f758c004730 comp rx=0 tx=0).stop 2024-03-28T22:17:57.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.427+0000 7f75a9ded740 1 -- 172.21.15.16:0/1941459224 shutdown_connections 2024-03-28T22:17:57.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.427+0000 7f75a9ded740 1 --2- 172.21.15.16:0/1941459224 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558808c827f0 0x558808c84c50 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.427+0000 7f75a9ded740 1 --2- 172.21.15.16:0/1941459224 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558808c72790 0x558808c70200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.427+0000 7f75a9ded740 1 --2- 172.21.15.16:0/1941459224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558808c69bb0 0x558808c6fcc0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.427+0000 7f757ffff640 0 client.17016 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:17:57.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.427+0000 7f75a9ded740 1 --2- 172.21.15.16:0/1941459224 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558808c69290 0x558808c6f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.427+0000 7f75a9ded740 1 -- 172.21.15.16:0/1941459224 >> 172.21.15.16:0/1941459224 conn(0x558808c639c0 msgr2=0x558808c62c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:57.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.428+0000 7f75a9ded740 1 -- 172.21.15.16:0/1941459224 shutdown_connections 2024-03-28T22:17:57.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.428+0000 7f75a9ded740 1 -- 172.21.15.16:0/1941459224 wait complete. 2024-03-28T22:17:57.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.428+0000 7f75a9ded740 20 client.17016 trim_cache size 0 max 16384 2024-03-28T22:17:57.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.445+0000 7f75a9ded740 1 Processor -- start 2024-03-28T22:17:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.445+0000 7f75a9ded740 1 -- start start 2024-03-28T22:17:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.445+0000 7f75a9ded740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558808c69290 0x558808c71eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.445+0000 7f75a9ded740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558808c69bb0 0x558808c7e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.445+0000 7f75a9ded740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558808c72790 0x558808c8a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.445+0000 7f75a9ded740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f75a01b1ee0 con 0x558808c69290 2024-03-28T22:17:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.445+0000 7f75a9ded740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f75a01af7b0 con 0x558808c72790 2024-03-28T22:17:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.445+0000 7f75a9ded740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f75a01ad620 con 0x558808c69bb0 2024-03-28T22:17:57.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.445+0000 7f75a4fa3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558808c69290 0x558808c71eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:57.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.445+0000 7f75a4fa3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558808c69290 0x558808c71eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55040/0 (socket says 172.21.15.16:55040) 2024-03-28T22:17:57.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.445+0000 7f75a4fa3640 1 -- 172.21.15.16:0/2030246622 learned_addr learned my addr 172.21.15.16:0/2030246622 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:57.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.446+0000 7f75a57a4640 1 --2- 172.21.15.16:0/2030246622 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558808c72790 0x558808c8a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:57.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.446+0000 7f75a4fa3640 1 -- 172.21.15.16:0/2030246622 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558808c69bb0 msgr2=0x558808c7e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:57.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.446+0000 7f75a4fa3640 1 --2- 172.21.15.16:0/2030246622 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558808c69bb0 0x558808c7e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.446+0000 7f75a4fa3640 1 -- 172.21.15.16:0/2030246622 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558808c72790 msgr2=0x558808c8a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:57.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.446+0000 7f75a4fa3640 1 --2- 172.21.15.16:0/2030246622 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558808c72790 0x558808c8a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.446+0000 7f75a4fa3640 1 -- 172.21.15.16:0/2030246622 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558808c699e0 con 0x558808c69290 2024-03-28T22:17:57.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.446+0000 7f75a4fa3640 1 --2- 172.21.15.16:0/2030246622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558808c69290 0x558808c71eb0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f75940bc3b0 tx=0x7f75940a7040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f312025dfa6e94a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:57.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.446+0000 7f758a7fc640 1 -- 172.21.15.16:0/2030246622 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f759404b040 con 0x558808c69290 2024-03-28T22:17:57.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.446+0000 7f758a7fc640 1 -- 172.21.15.16:0/2030246622 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75940bc6b0 con 0x558808c69290 2024-03-28T22:17:57.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.446+0000 7f758a7fc640 1 -- 172.21.15.16:0/2030246622 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75940ed4c0 con 0x558808c69290 2024-03-28T22:17:57.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.447+0000 7f75a9ded740 1 -- 172.21.15.16:0/2030246622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558808c69290 msgr2=0x558808c71eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:57.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.447+0000 7f75a9ded740 1 --2- 172.21.15.16:0/2030246622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558808c69290 0x558808c71eb0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f75940bc3b0 tx=0x7f75940a7040 comp rx=0 tx=0).stop 2024-03-28T22:17:57.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.447+0000 7f75a9ded740 1 -- 172.21.15.16:0/2030246622 shutdown_connections 2024-03-28T22:17:57.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.447+0000 7f75a9ded740 1 --2- 172.21.15.16:0/2030246622 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558808c72790 0x558808c8a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.447+0000 7f75a9ded740 1 --2- 172.21.15.16:0/2030246622 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558808c69bb0 0x558808c7e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.447+0000 7f75a9ded740 1 --2- 172.21.15.16:0/2030246622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558808c69290 0x558808c71eb0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.447+0000 7f75a9ded740 1 -- 172.21.15.16:0/2030246622 >> 172.21.15.16:0/2030246622 conn(0x558808c639c0 msgr2=0x558808c62c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:57.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.447+0000 7f75a9ded740 1 -- 172.21.15.16:0/2030246622 shutdown_connections 2024-03-28T22:17:57.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.447+0000 7f75a9ded740 1 -- 172.21.15.16:0/2030246622 wait complete. 2024-03-28T22:17:57.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.447+0000 7f75a9ded740 1 -- start start 2024-03-28T22:17:57.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.447+0000 7f75a9ded740 1 Processor -- start 2024-03-28T22:17:57.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.447+0000 7f75a9ded740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558808c69290 0x558808c7a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.447+0000 7f75a9ded740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558808c69bb0 0x558808c76940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.447+0000 7f75a9ded740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558808c72790 0x558808c76e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.447+0000 7f75a9ded740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f75a01adc70 con 0x558808c69bb0 2024-03-28T22:17:57.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.447+0000 7f75a9ded740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f75a01af0e0 con 0x558808c72790 2024-03-28T22:17:57.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.447+0000 7f75a9ded740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f75a01ae340 con 0x558808c69290 2024-03-28T22:17:57.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.447+0000 7f75a9ded740 10 client.0 _ll_register_callbacks cb 0x7f758c0036c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:17:57.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.448+0000 7f759ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558808c69bb0 0x558808c76940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:57.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.448+0000 7f759ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558808c69bb0 0x558808c76940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55050/0 (socket says 172.21.15.16:55050) 2024-03-28T22:17:57.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.448+0000 7f759ffff640 1 -- 172.21.15.16:0/1422601792 learned_addr learned my addr 172.21.15.16:0/1422601792 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:57.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.448+0000 7f75a57a4640 1 --2- 172.21.15.16:0/1422601792 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558808c72790 0x558808c76e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:57.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.448+0000 7f75a4fa3640 1 --2- 172.21.15.16:0/1422601792 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558808c69290 0x558808c7a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:57.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.448+0000 7f759ffff640 1 -- 172.21.15.16:0/1422601792 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558808c69290 msgr2=0x558808c7a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:57.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.448+0000 7f759ffff640 1 --2- 172.21.15.16:0/1422601792 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558808c69290 0x558808c7a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.448+0000 7f759ffff640 1 -- 172.21.15.16:0/1422601792 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558808c72790 msgr2=0x558808c76e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:57.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.448+0000 7f759ffff640 1 --2- 172.21.15.16:0/1422601792 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558808c72790 0x558808c76e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.448+0000 7f759ffff640 1 -- 172.21.15.16:0/1422601792 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588088a10f0 con 0x558808c69bb0 2024-03-28T22:17:57.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.448+0000 7f75a4fa3640 1 --2- 172.21.15.16:0/1422601792 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558808c69290 0x558808c7a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:17:57.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.448+0000 7f759ffff640 1 --2- 172.21.15.16:0/1422601792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558808c69bb0 0x558808c76940 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f758c02e7f0 tx=0x7f758c02eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bedf4282eb9513d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:57.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.448+0000 7f757ffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:17:57.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.448+0000 7f757ffff640 1 -- 172.21.15.16:0/1422601792 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f758c02e400 con 0x558808c69bb0 2024-03-28T22:17:57.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.448+0000 7f757ffff640 1 -- 172.21.15.16:0/1422601792 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8527+0+0 (secure 0 0 0) 0x7f758c003d80 con 0x558808c69bb0 2024-03-28T22:17:57.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.448+0000 7f75a9ded740 1 -- 172.21.15.16:0/1422601792 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558808bd4a10 con 0x558808c69bb0 2024-03-28T22:17:57.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.448+0000 7f757ffff640 1 -- 172.21.15.16:0/1422601792 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f758c03d050 con 0x558808c69bb0 2024-03-28T22:17:57.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.448+0000 7f75a9ded740 10 client.17022 fetch_fsmap learned FSMap version 27 2024-03-28T22:17:57.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.448+0000 7f75a9ded740 1 -- 172.21.15.16:0/1422601792 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558808ce9670 con 0x558808c69bb0 2024-03-28T22:17:57.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.449+0000 7f757ffff640 1 -- 172.21.15.16:0/1422601792 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558808bd4a10 con 0x558808c69bb0 2024-03-28T22:17:57.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.449+0000 7f757ffff640 1 -- 172.21.15.16:0/1422601792 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f758c035020 con 0x558808c69bb0 2024-03-28T22:17:57.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.449+0000 7f75a9ded740 10 client.17022 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:17:57.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.449+0000 7f75a9ded740 10 client.17022 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:17:57.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.449+0000 7f75a9ded740 20 client.17022 populate_metadata read hostname 'smithi016' 2024-03-28T22:17:57.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.449+0000 7f75a9ded740 1 --2- 172.21.15.16:0/1422601792 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558808ce9de0 0x558808cec1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.449+0000 7f75a9ded740 4 client.17022 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:17:57.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.449+0000 7f75a9ded740 1 -- 172.21.15.16:0/1422601792 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558808b52540 con 0x558808ce9de0 2024-03-28T22:17:57.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.449+0000 7f75a4fa3640 1 --2- 172.21.15.16:0/1422601792 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558808ce9de0 0x558808cec1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:57.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.449+0000 7f757ffff640 10 client.17022 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:17:57.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.449+0000 7f75a4fa3640 1 --2- 172.21.15.16:0/1422601792 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558808ce9de0 0x558808cec1d0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b365d1a6f5b9d47b server_cookie=acaf25630ac535f0 in_seq=0 out_seq=0 2024-03-28T22:17:57.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.449+0000 7f757ffff640 1 -- 172.21.15.16:0/1422601792 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x558808b52540 con 0x558808ce9de0 2024-03-28T22:17:57.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.450+0000 7f757ffff640 10 client.17022 handle_command_reply: tid=0 2024-03-28T22:17:57.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.450+0000 7f75a9ded740 1 client.17022 shutdown 2024-03-28T22:17:57.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.450+0000 7f75a9ded740 1 -- 172.21.15.16:0/1422601792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558808c69bb0 msgr2=0x558808c76940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:57.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.450+0000 7f75a9ded740 1 --2- 172.21.15.16:0/1422601792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558808c69bb0 0x558808c76940 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f758c02e7f0 tx=0x7f758c02eb20 comp rx=0 tx=0).stop 2024-03-28T22:17:57.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.450+0000 7f75a9ded740 1 -- 172.21.15.16:0/1422601792 shutdown_connections 2024-03-28T22:17:57.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.450+0000 7f75a9ded740 1 --2- 172.21.15.16:0/1422601792 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558808ce9de0 0x558808cec1d0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.450+0000 7f75a9ded740 1 --2- 172.21.15.16:0/1422601792 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558808c72790 0x558808c76e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.450+0000 7f75a9ded740 1 --2- 172.21.15.16:0/1422601792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558808c69bb0 0x558808c76940 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.450+0000 7f757ffff640 0 client.17022 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:17:57.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.450+0000 7f75a9ded740 1 --2- 172.21.15.16:0/1422601792 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558808c69290 0x558808c7a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.450+0000 7f75a9ded740 1 -- 172.21.15.16:0/1422601792 >> 172.21.15.16:0/1422601792 conn(0x558808c639c0 msgr2=0x558808c6f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:57.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.450+0000 7f75a9ded740 1 -- 172.21.15.16:0/1422601792 shutdown_connections 2024-03-28T22:17:57.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.450+0000 7f75a9ded740 1 -- 172.21.15.16:0/1422601792 wait complete. 2024-03-28T22:17:57.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.450+0000 7f75a9ded740 20 client.17022 trim_cache size 0 max 16384 2024-03-28T22:17:57.464 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:17:57.464 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-03-28T22:17:57.465 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-03-28T22:17:57.465 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:17:57.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.450+0000 7f75a7207640 1 -- 172.21.15.16:0/566419330 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f75900a1ef0 msgr2=0x7f75900a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:57.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.450+0000 7f75a7207640 1 --2- 172.21.15.16:0/566419330 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f75900a1ef0 0x7f75900a43b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f75a0103d20 tx=0x7f758c005f90 comp rx=0 tx=0).stop 2024-03-28T22:17:57.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.450+0000 7f75a7207640 1 -- 172.21.15.16:0/566419330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75a01005f0 msgr2=0x7f75a0102290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:57.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.450+0000 7f75a7207640 1 --2- 172.21.15.16:0/566419330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75a01005f0 0x7f75a0102290 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f7594009230 tx=0x7f759402cb20 comp rx=0 tx=0).stop 2024-03-28T22:17:57.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.450+0000 7f75a7207640 1 -- 172.21.15.16:0/566419330 shutdown_connections 2024-03-28T22:17:57.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.450+0000 7f75a7207640 1 --2- 172.21.15.16:0/566419330 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f75900a1ef0 0x7f75900a43b0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.451+0000 7f75a7207640 1 --2- 172.21.15.16:0/566419330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75a0106690 0x7f75a0102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.451+0000 7f75a7207640 1 --2- 172.21.15.16:0/566419330 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75a0100fb0 0x7f75a01027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.451+0000 7f75a7207640 1 --2- 172.21.15.16:0/566419330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75a01005f0 0x7f75a0102290 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.451+0000 7f75a7207640 1 -- 172.21.15.16:0/566419330 >> 172.21.15.16:0/566419330 conn(0x7f75a0077800 msgr2=0x7f75a00ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:57.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.451+0000 7f75a7207640 1 -- 172.21.15.16:0/566419330 shutdown_connections 2024-03-28T22:17:57.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.451+0000 7f75a7207640 1 -- 172.21.15.16:0/566419330 wait complete. 2024-03-28T22:17:57.468 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:17:57.468 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:17:57.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.569+0000 7f22eeaae640 1 -- 172.21.15.16:0/3425588073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22e8101700 msgr2=0x7f22e8101b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:57.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.569+0000 7f22eeaae640 1 --2- 172.21.15.16:0/3425588073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22e8101700 0x7f22e8101b80 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f22d8009930 tx=0x7f22d802bc30 comp rx=0 tx=0).stop 2024-03-28T22:17:57.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.570+0000 7f22eeaae640 1 -- 172.21.15.16:0/3425588073 shutdown_connections 2024-03-28T22:17:57.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.570+0000 7f22eeaae640 1 --2- 172.21.15.16:0/3425588073 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f22e81020c0 0x7f22e810e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.570+0000 7f22eeaae640 1 --2- 172.21.15.16:0/3425588073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22e8101700 0x7f22e8101b80 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.570+0000 7f22eeaae640 1 --2- 172.21.15.16:0/3425588073 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22e8107710 0x7f22e8107af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.570+0000 7f22eeaae640 1 -- 172.21.15.16:0/3425588073 >> 172.21.15.16:0/3425588073 conn(0x7f22e80fd210 msgr2=0x7f22e80ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:57.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.570+0000 7f22eeaae640 1 -- 172.21.15.16:0/3425588073 shutdown_connections 2024-03-28T22:17:57.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.570+0000 7f22eeaae640 1 -- 172.21.15.16:0/3425588073 wait complete. 2024-03-28T22:17:57.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.570+0000 7f22eeaae640 1 Processor -- start 2024-03-28T22:17:57.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.570+0000 7f22eeaae640 1 -- start start 2024-03-28T22:17:57.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.570+0000 7f22eeaae640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f22e8101700 0x7f22e81980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.570+0000 7f22eeaae640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22e81020c0 0x7f22e8198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.570+0000 7f22eeaae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22e8107710 0x7f22e819c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.570+0000 7f22eeaae640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f22e8111b20 con 0x7f22e8107710 2024-03-28T22:17:57.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.570+0000 7f22eeaae640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f22e81119a0 con 0x7f22e81020c0 2024-03-28T22:17:57.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.570+0000 7f22eeaae640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f22e8111ca0 con 0x7f22e8101700 2024-03-28T22:17:57.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.571+0000 7f22ed04b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22e8107710 0x7f22e819c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:57.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.571+0000 7f22dffff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22e81020c0 0x7f22e8198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:57.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.571+0000 7f22ed04b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22e8107710 0x7f22e819c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55070/0 (socket says 172.21.15.16:55070) 2024-03-28T22:17:57.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.571+0000 7f22ed04b640 1 -- 172.21.15.16:0/108608525 learned_addr learned my addr 172.21.15.16:0/108608525 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:57.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.571+0000 7f22ec84a640 1 --2- 172.21.15.16:0/108608525 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f22e8101700 0x7f22e81980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:57.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.571+0000 7f22ed04b640 1 -- 172.21.15.16:0/108608525 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f22e8101700 msgr2=0x7f22e81980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:57.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.571+0000 7f22ed04b640 1 --2- 172.21.15.16:0/108608525 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f22e8101700 0x7f22e81980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.571+0000 7f22ed04b640 1 -- 172.21.15.16:0/108608525 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22e81020c0 msgr2=0x7f22e8198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:57.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.571+0000 7f22ed04b640 1 --2- 172.21.15.16:0/108608525 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22e81020c0 0x7f22e8198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.571+0000 7f22ed04b640 1 -- 172.21.15.16:0/108608525 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22e819d0d0 con 0x7f22e8107710 2024-03-28T22:17:57.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.571+0000 7f22dffff640 1 --2- 172.21.15.16:0/108608525 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22e81020c0 0x7f22e8198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:17:57.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.571+0000 7f22ed04b640 1 --2- 172.21.15.16:0/108608525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22e8107710 0x7f22e819c9a0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f22e000b970 tx=0x7f22e000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:57.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.571+0000 7f22ddffb640 1 -- 172.21.15.16:0/108608525 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e0019070 con 0x7f22e8107710 2024-03-28T22:17:57.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.571+0000 7f22ddffb640 1 -- 172.21.15.16:0/108608525 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22e000cb40 con 0x7f22e8107710 2024-03-28T22:17:57.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.571+0000 7f22eeaae640 1 -- 172.21.15.16:0/108608525 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22e819d3c0 con 0x7f22e8107710 2024-03-28T22:17:57.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.571+0000 7f22ddffb640 1 -- 172.21.15.16:0/108608525 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e0007650 con 0x7f22e8107710 2024-03-28T22:17:57.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.571+0000 7f22eeaae640 1 -- 172.21.15.16:0/108608525 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22e81a4c50 con 0x7f22e8107710 2024-03-28T22:17:57.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.572+0000 7f22eeaae640 1 -- 172.21.15.16:0/108608525 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22e80685d0 con 0x7f22e8107710 2024-03-28T22:17:57.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.573+0000 7f22ddffb640 1 -- 172.21.15.16:0/108608525 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f22e001c070 con 0x7f22e8107710 2024-03-28T22:17:57.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.574+0000 7f22ddffb640 1 --2- 172.21.15.16:0/108608525 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f22d00a1ef0 0x7f22d00a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.574+0000 7f22ec84a640 1 --2- 172.21.15.16:0/108608525 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f22d00a1ef0 0x7f22d00a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:57.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.574+0000 7f22ddffb640 1 -- 172.21.15.16:0/108608525 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8527+0+0 (secure 0 0 0) 0x7f22e00cd6b0 con 0x7f22e8107710 2024-03-28T22:17:57.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.574+0000 7f22ec84a640 1 --2- 172.21.15.16:0/108608525 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f22d00a1ef0 0x7f22d00a43b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f22e4009b60 tx=0x7f22e4005f90 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:57.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.579+0000 7f22ddffb640 1 -- 172.21.15.16:0/108608525 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f22e009a200 con 0x7f22e8107710 2024-03-28T22:17:57.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:57 smithi016 ceph-mon[32374]: pgmap v226: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:17:57.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.799+0000 7f22eeaae640 1 -- 172.21.15.16:0/108608525 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f22e8109c90 con 0x7f22e8107710 2024-03-28T22:17:57.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.799+0000 7f22ddffb640 1 -- 172.21.15.16:0/108608525 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11851 (secure 0 0 0) 0x7f22e009e2a0 con 0x7f22e8107710 2024-03-28T22:17:57.802 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:17:57.802 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/1644581227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1644581227},{"type":"v1","addr":"172.21.15.16:6839","nonce":1644581227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/2593062669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2593062669},{"type":"v1","addr":"172.21.15.16:6841","nonce":2593062669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:08:29.672976+0000","modified":"2024-03-28T22:14:44.922044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24566,"mds_1":24524,"mds_2":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24466":{"gid":24466,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/247249857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":247249857},{"type":"v1","addr":"172.21.15.63:6835","nonce":247249857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6837/2324231032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":2324231032},{"type":"v1","addr":"172.21.15.63:6837","nonce":2324231032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6839/1177197077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":1177197077},{"type":"v1","addr":"172.21.15.63:6839","nonce":1177197077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"e","rank":2,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.53:6835/350727634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":350727634},{"type":"v1","addr":"172.21.15.53:6835","nonce":350727634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/349877573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":349877573},{"type":"v1","addr":"172.21.15.53:6837","nonce":349877573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6839/3714153956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":3714153956},{"type":"v1","addr":"172.21.15.53:6839","nonce":3714153956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24566,"qdb_cluster":[24500,24524,24566]},"id":1}]} 2024-03-28T22:17:57.802 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2024-03-28T22:17:57.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.802+0000 7f22eeaae640 1 -- 172.21.15.16:0/108608525 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f22d00a1ef0 msgr2=0x7f22d00a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:57.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.802+0000 7f22eeaae640 1 --2- 172.21.15.16:0/108608525 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f22d00a1ef0 0x7f22d00a43b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f22e4009b60 tx=0x7f22e4005f90 comp rx=0 tx=0).stop 2024-03-28T22:17:57.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.802+0000 7f22eeaae640 1 -- 172.21.15.16:0/108608525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22e8107710 msgr2=0x7f22e819c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:57.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.802+0000 7f22eeaae640 1 --2- 172.21.15.16:0/108608525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22e8107710 0x7f22e819c9a0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f22e000b970 tx=0x7f22e000be30 comp rx=0 tx=0).stop 2024-03-28T22:17:57.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.802+0000 7f22eeaae640 1 -- 172.21.15.16:0/108608525 shutdown_connections 2024-03-28T22:17:57.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.802+0000 7f22eeaae640 1 --2- 172.21.15.16:0/108608525 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f22d00a1ef0 0x7f22d00a43b0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.802+0000 7f22eeaae640 1 --2- 172.21.15.16:0/108608525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22e8107710 0x7f22e819c9a0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.802+0000 7f22eeaae640 1 --2- 172.21.15.16:0/108608525 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22e81020c0 0x7f22e8198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.802+0000 7f22eeaae640 1 --2- 172.21.15.16:0/108608525 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f22e8101700 0x7f22e81980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.802+0000 7f22eeaae640 1 -- 172.21.15.16:0/108608525 >> 172.21.15.16:0/108608525 conn(0x7f22e80fd210 msgr2=0x7f22e80fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:57.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.802+0000 7f22eeaae640 1 -- 172.21.15.16:0/108608525 shutdown_connections 2024-03-28T22:17:57.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.802+0000 7f22eeaae640 1 -- 172.21.15.16:0/108608525 wait complete. 2024-03-28T22:17:57.815 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:17:57.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.917+0000 7f167598c640 1 -- 172.21.15.16:0/2921157799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16701020a0 msgr2=0x7f167010e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:57.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.917+0000 7f167598c640 1 --2- 172.21.15.16:0/2921157799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16701020a0 0x7f167010e5b0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f1668009cd0 tx=0x7f166802c0f0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.917+0000 7f167598c640 1 -- 172.21.15.16:0/2921157799 shutdown_connections 2024-03-28T22:17:57.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.917+0000 7f167598c640 1 --2- 172.21.15.16:0/2921157799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16701020a0 0x7f167010e5b0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.917+0000 7f167598c640 1 --2- 172.21.15.16:0/2921157799 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16701016e0 0x7f1670101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.917+0000 7f167598c640 1 --2- 172.21.15.16:0/2921157799 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16701076f0 0x7f1670107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.917+0000 7f167598c640 1 -- 172.21.15.16:0/2921157799 >> 172.21.15.16:0/2921157799 conn(0x7f16700fd1f0 msgr2=0x7f16700ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:57.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.917+0000 7f167598c640 1 -- 172.21.15.16:0/2921157799 shutdown_connections 2024-03-28T22:17:57.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.917+0000 7f167598c640 1 -- 172.21.15.16:0/2921157799 wait complete. 2024-03-28T22:17:57.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.918+0000 7f167598c640 1 Processor -- start 2024-03-28T22:17:57.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.918+0000 7f167598c640 1 -- start start 2024-03-28T22:17:57.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.918+0000 7f167598c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16701016e0 0x7f1670197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.918+0000 7f167598c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16701020a0 0x7f16701984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.918+0000 7f167598c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16701076f0 0x7f167019c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.918+0000 7f167598c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1670111ac0 con 0x7f16701076f0 2024-03-28T22:17:57.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.918+0000 7f167598c640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f1670111940 con 0x7f16701016e0 2024-03-28T22:17:57.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.918+0000 7f167598c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1670111c40 con 0x7f16701020a0 2024-03-28T22:17:57.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.918+0000 7f166f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16701076f0 0x7f167019c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:57.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.918+0000 7f166f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16701076f0 0x7f167019c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55092/0 (socket says 172.21.15.16:55092) 2024-03-28T22:17:57.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.918+0000 7f166f7fe640 1 -- 172.21.15.16:0/1206562146 learned_addr learned my addr 172.21.15.16:0/1206562146 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:57.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.918+0000 7f166effd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16701016e0 0x7f1670197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:57.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.918+0000 7f166e7fc640 1 --2- 172.21.15.16:0/1206562146 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16701020a0 0x7f16701984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:57.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.918+0000 7f166f7fe640 1 -- 172.21.15.16:0/1206562146 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16701020a0 msgr2=0x7f16701984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:57.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.918+0000 7f166f7fe640 1 --2- 172.21.15.16:0/1206562146 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16701020a0 0x7f16701984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.918+0000 7f166f7fe640 1 -- 172.21.15.16:0/1206562146 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16701016e0 msgr2=0x7f1670197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:57.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.918+0000 7f166f7fe640 1 --2- 172.21.15.16:0/1206562146 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16701016e0 0x7f1670197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.918+0000 7f166f7fe640 1 -- 172.21.15.16:0/1206562146 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f167019cf70 con 0x7f16701076f0 2024-03-28T22:17:57.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.919+0000 7f166f7fe640 1 --2- 172.21.15.16:0/1206562146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16701076f0 0x7f167019c840 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f16680348f0 tx=0x7f1668032c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:57.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.919+0000 7f167498a640 1 -- 172.21.15.16:0/1206562146 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1668041020 con 0x7f16701076f0 2024-03-28T22:17:57.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.919+0000 7f167498a640 1 -- 172.21.15.16:0/1206562146 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1668032e80 con 0x7f16701076f0 2024-03-28T22:17:57.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.919+0000 7f167598c640 1 -- 172.21.15.16:0/1206562146 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f167019d200 con 0x7f16701076f0 2024-03-28T22:17:57.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.919+0000 7f167498a640 1 -- 172.21.15.16:0/1206562146 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1668039040 con 0x7f16701076f0 2024-03-28T22:17:57.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.919+0000 7f167598c640 1 -- 172.21.15.16:0/1206562146 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16701a4af0 con 0x7f16701076f0 2024-03-28T22:17:57.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.921+0000 7f167498a640 1 -- 172.21.15.16:0/1206562146 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f1668044020 con 0x7f16701076f0 2024-03-28T22:17:57.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.921+0000 7f1678571740 1 Processor -- start 2024-03-28T22:17:57.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.921+0000 7f167498a640 1 --2- 172.21.15.16:0/1206562146 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f16580a1f70 0x7f16580a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.921+0000 7f1678571740 1 -- start start 2024-03-28T22:17:57.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.922+0000 7f166e7fc640 1 --2- 172.21.15.16:0/1206562146 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f16580a1f70 0x7f16580a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:57.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.922+0000 7f167498a640 1 -- 172.21.15.16:0/1206562146 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8527+0+0 (secure 0 0 0) 0x7f16680efcb0 con 0x7f16701076f0 2024-03-28T22:17:57.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.922+0000 7f1678571740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d16ee7a470 0x55d16ee7a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.922+0000 7f1678571740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d16ee7ad90 0x55d16ee83430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.922+0000 7f1678571740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d16ee83970 0x55d16ee85d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.922+0000 7f1678571740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d16edde1b0 con 0x55d16ee7ad90 2024-03-28T22:17:57.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.922+0000 7f1678571740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55d16ed86c00 con 0x55d16ee83970 2024-03-28T22:17:57.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.922+0000 7f1678571740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d16edda250 con 0x55d16ee7a470 2024-03-28T22:17:57.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.922+0000 7f166e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d16ee7ad90 0x55d16ee83430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:57.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.922+0000 7f166e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d16ee7ad90 0x55d16ee83430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55094/0 (socket says 172.21.15.16:55094) 2024-03-28T22:17:57.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.922+0000 7f166e7fc640 1 -- 172.21.15.16:0/2152773826 learned_addr learned my addr 172.21.15.16:0/2152773826 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:57.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.922+0000 7f166f7fe640 1 --2- 172.21.15.16:0/2152773826 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d16ee83970 0x55d16ee85d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:57.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.922+0000 7f166effd640 1 --2- 172.21.15.16:0/2152773826 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d16ee7a470 0x55d16ee7a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:57.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.922+0000 7f166f7fe640 1 -- 172.21.15.16:0/2152773826 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d16ee7a470 msgr2=0x55d16ee7a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:57.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.922+0000 7f166f7fe640 1 --2- 172.21.15.16:0/2152773826 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d16ee7a470 0x55d16ee7a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.922+0000 7f166f7fe640 1 -- 172.21.15.16:0/2152773826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d16ee7ad90 msgr2=0x55d16ee83430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:57.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.922+0000 7f166f7fe640 1 --2- 172.21.15.16:0/2152773826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d16ee7ad90 0x55d16ee83430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.922+0000 7f166f7fe640 1 -- 172.21.15.16:0/2152773826 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d16eda63d0 con 0x55d16ee83970 2024-03-28T22:17:57.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.922+0000 7f166e7fc640 1 --2- 172.21.15.16:0/1206562146 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f16580a1f70 0x7f16580a4430 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1664009fd0 tx=0x7f1664013820 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:57.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.922+0000 7f166effd640 1 --2- 172.21.15.16:0/2152773826 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d16ee7a470 0x55d16ee7a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:17:57.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.922+0000 7f166f7fe640 1 --2- 172.21.15.16:0/2152773826 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d16ee83970 0x55d16ee85d60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f16680ed510 tx=0x7f16680b8420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9944bc3cfa340964 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:57.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.922+0000 7f1655ffb640 1 -- 172.21.15.16:0/2152773826 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16680bc110 con 0x55d16ee83970 2024-03-28T22:17:57.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.923+0000 7f1655ffb640 1 -- 172.21.15.16:0/2152773826 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1668034920 con 0x55d16ee83970 2024-03-28T22:17:57.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.923+0000 7f1655ffb640 1 -- 172.21.15.16:0/2152773826 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1668034e60 con 0x55d16ee83970 2024-03-28T22:17:57.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.923+0000 7f1678571740 1 -- 172.21.15.16:0/2152773826 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d16ee83970 msgr2=0x55d16ee85d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:57.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.923+0000 7f1678571740 1 --2- 172.21.15.16:0/2152773826 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d16ee83970 0x55d16ee85d60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f16680ed510 tx=0x7f16680b8420 comp rx=0 tx=0).stop 2024-03-28T22:17:57.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.923+0000 7f1678571740 1 -- 172.21.15.16:0/2152773826 shutdown_connections 2024-03-28T22:17:57.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.923+0000 7f1678571740 1 --2- 172.21.15.16:0/2152773826 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d16ee83970 0x55d16ee85d60 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.923+0000 7f1678571740 1 --2- 172.21.15.16:0/2152773826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d16ee7ad90 0x55d16ee83430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.923+0000 7f1678571740 1 --2- 172.21.15.16:0/2152773826 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d16ee7a470 0x55d16ee7a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.923+0000 7f1678571740 1 -- 172.21.15.16:0/2152773826 >> 172.21.15.16:0/2152773826 conn(0x55d16ee74ba0 msgr2=0x55d16ee74f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:57.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.923+0000 7f1678571740 1 -- 172.21.15.16:0/2152773826 shutdown_connections 2024-03-28T22:17:57.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.923+0000 7f1678571740 1 -- 172.21.15.16:0/2152773826 wait complete. 2024-03-28T22:17:57.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.923+0000 7f1678571740 1 -- start start 2024-03-28T22:17:57.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.924+0000 7f1678571740 1 Processor -- start 2024-03-28T22:17:57.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.924+0000 7f1678571740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d16ee7a470 0x55d16ee809f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.924+0000 7f1678571740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d16ee7ad90 0x55d16ee80f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.924+0000 7f1678571740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d16ee83970 0x55d16ee81470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.924+0000 7f1678571740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d16ee86520 con 0x55d16ee7a470 2024-03-28T22:17:57.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.924+0000 7f1678571740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55d16ed7c470 con 0x55d16ee83970 2024-03-28T22:17:57.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.924+0000 7f1678571740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d16ee866a0 con 0x55d16ee7ad90 2024-03-28T22:17:57.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.924+0000 7f1678571740 10 client.0 _ll_register_callbacks cb 0x55d16ed99cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:17:57.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.924+0000 7f166effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d16ee7a470 0x55d16ee809f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:57.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.924+0000 7f166effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d16ee7a470 0x55d16ee809f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55104/0 (socket says 172.21.15.16:55104) 2024-03-28T22:17:57.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.924+0000 7f166effd640 1 -- 172.21.15.16:0/2305720062 learned_addr learned my addr 172.21.15.16:0/2305720062 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:57.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.924+0000 7f166f7fe640 1 --2- 172.21.15.16:0/2305720062 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d16ee83970 0x55d16ee81470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:57.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.924+0000 7f166effd640 1 -- 172.21.15.16:0/2305720062 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d16ee7ad90 msgr2=0x55d16ee80f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:57.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.924+0000 7f166effd640 1 --2- 172.21.15.16:0/2305720062 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d16ee7ad90 0x55d16ee80f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.924+0000 7f166effd640 1 -- 172.21.15.16:0/2305720062 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d16ee83970 msgr2=0x55d16ee81470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:57.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.924+0000 7f166effd640 1 --2- 172.21.15.16:0/2305720062 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d16ee83970 0x55d16ee81470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.924+0000 7f166effd640 1 -- 172.21.15.16:0/2305720062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d16eaf23f0 con 0x55d16ee7a470 2024-03-28T22:17:57.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.924+0000 7f166f7fe640 1 --2- 172.21.15.16:0/2305720062 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d16ee83970 0x55d16ee81470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:17:57.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.925+0000 7f166effd640 1 --2- 172.21.15.16:0/2305720062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d16ee7a470 0x55d16ee809f0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f16600136e0 tx=0x7f1660013ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e40dc39c77175ccd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:57.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.925+0000 7f164effd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:17:57.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.925+0000 7f164effd640 1 -- 172.21.15.16:0/2305720062 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1660004460 con 0x55d16ee7a470 2024-03-28T22:17:57.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.925+0000 7f164effd640 1 -- 172.21.15.16:0/2305720062 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8527+0+0 (secure 0 0 0) 0x7f16600119c0 con 0x55d16ee7a470 2024-03-28T22:17:57.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.925+0000 7f1678571740 1 -- 172.21.15.16:0/2305720062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1668034920 con 0x55d16ee7a470 2024-03-28T22:17:57.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.925+0000 7f164effd640 1 -- 172.21.15.16:0/2305720062 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1660011bc0 con 0x55d16ee7a470 2024-03-28T22:17:57.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.925+0000 7f1678571740 10 client.17040 fetch_fsmap learned FSMap version 27 2024-03-28T22:17:57.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.925+0000 7f1678571740 1 -- 172.21.15.16:0/2305720062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d16ebbc7d0 con 0x55d16ee7a470 2024-03-28T22:17:57.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.925+0000 7f164effd640 1 -- 172.21.15.16:0/2305720062 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1668034920 con 0x55d16ee7a470 2024-03-28T22:17:57.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.925+0000 7f164effd640 1 -- 172.21.15.16:0/2305720062 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f166001a070 con 0x55d16ee7a470 2024-03-28T22:17:57.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.925+0000 7f1678571740 10 client.17040 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:17:57.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.925+0000 7f1678571740 10 client.17040 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-28T22:17:57.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.925+0000 7f1678571740 20 client.17040 populate_metadata read hostname 'smithi016' 2024-03-28T22:17:57.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.925+0000 7f1678571740 1 --2- 172.21.15.16:0/2305720062 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55d16ee936d0 0x55d16ee95af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.925+0000 7f1678571740 4 client.17040 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:17:57.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.925+0000 7f1678571740 1 -- 172.21.15.16:0/2305720062 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d16edeb6f0 con 0x55d16ee936d0 2024-03-28T22:17:57.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.926+0000 7f166e7fc640 1 --2- 172.21.15.16:0/2305720062 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55d16ee936d0 0x55d16ee95af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:57.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.926+0000 7f166e7fc640 1 --2- 172.21.15.16:0/2305720062 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55d16ee936d0 0x55d16ee95af0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=877153a32490ec97 server_cookie=fe9dc456d5171d75 in_seq=0 out_seq=0 2024-03-28T22:17:57.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.926+0000 7f164effd640 10 client.17040 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:17:57.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.927+0000 7f164effd640 1 -- 172.21.15.16:0/2305720062 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d16edeb6f0 con 0x55d16ee936d0 2024-03-28T22:17:57.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.927+0000 7f164effd640 10 client.17040 handle_command_reply: tid=0 2024-03-28T22:17:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.927+0000 7f1678571740 1 client.17040 shutdown 2024-03-28T22:17:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.927+0000 7f1678571740 1 -- 172.21.15.16:0/2305720062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d16ee7a470 msgr2=0x55d16ee809f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.927+0000 7f1678571740 1 --2- 172.21.15.16:0/2305720062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d16ee7a470 0x55d16ee809f0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f16600136e0 tx=0x7f1660013ba0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.927+0000 7f1678571740 1 -- 172.21.15.16:0/2305720062 shutdown_connections 2024-03-28T22:17:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.927+0000 7f1678571740 1 --2- 172.21.15.16:0/2305720062 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55d16ee936d0 0x55d16ee95af0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.927+0000 7f1678571740 1 --2- 172.21.15.16:0/2305720062 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d16ee83970 0x55d16ee81470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.927+0000 7f164effd640 0 client.17040 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:17:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.927+0000 7f1678571740 1 --2- 172.21.15.16:0/2305720062 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d16ee7ad90 0x55d16ee80f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.927+0000 7f1678571740 1 --2- 172.21.15.16:0/2305720062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d16ee7a470 0x55d16ee809f0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.927+0000 7f1678571740 1 -- 172.21.15.16:0/2305720062 >> 172.21.15.16:0/2305720062 conn(0x55d16ee74ba0 msgr2=0x55d16ee768f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:57.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.927+0000 7f1678571740 1 -- 172.21.15.16:0/2305720062 shutdown_connections 2024-03-28T22:17:57.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.927+0000 7f1678571740 1 -- 172.21.15.16:0/2305720062 wait complete. 2024-03-28T22:17:57.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.927+0000 7f1678571740 20 client.17040 trim_cache size 0 max 16384 2024-03-28T22:17:57.943 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:57 smithi063 ceph-mon[36249]: pgmap v226: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:17:57.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.940+0000 7f1678571740 1 Processor -- start 2024-03-28T22:17:57.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.940+0000 7f1678571740 1 -- start start 2024-03-28T22:17:57.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.940+0000 7f1678571740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d16ee7a470 0x55d16ee8f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.940+0000 7f1678571740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d16ee7ad90 0x55d16ee81290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.940+0000 7f1678571740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d16ee83970 0x55d16ee76700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.940+0000 7f1678571740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f16701afca0 con 0x55d16ee7ad90 2024-03-28T22:17:57.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.940+0000 7f1678571740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f16701ad570 con 0x55d16ee7a470 2024-03-28T22:17:57.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.940+0000 7f1678571740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f16701ab3e0 con 0x55d16ee83970 2024-03-28T22:17:57.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.940+0000 7f166e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d16ee7ad90 0x55d16ee81290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:57.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.940+0000 7f166effd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d16ee7a470 0x55d16ee8f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:57.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.940+0000 7f166e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d16ee7ad90 0x55d16ee81290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55116/0 (socket says 172.21.15.16:55116) 2024-03-28T22:17:57.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.940+0000 7f166e7fc640 1 -- 172.21.15.16:0/10699104 learned_addr learned my addr 172.21.15.16:0/10699104 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:57.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.941+0000 7f166f7fe640 1 --2- 172.21.15.16:0/10699104 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d16ee83970 0x55d16ee76700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:57.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.941+0000 7f166e7fc640 1 -- 172.21.15.16:0/10699104 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d16ee83970 msgr2=0x55d16ee76700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:57.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.941+0000 7f166e7fc640 1 --2- 172.21.15.16:0/10699104 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d16ee83970 0x55d16ee76700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.941+0000 7f166e7fc640 1 -- 172.21.15.16:0/10699104 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d16ee7a470 msgr2=0x55d16ee8f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:57.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.941+0000 7f166e7fc640 1 --2- 172.21.15.16:0/10699104 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d16ee7a470 0x55d16ee8f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.941+0000 7f166e7fc640 1 -- 172.21.15.16:0/10699104 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d16ee75d30 con 0x55d16ee7ad90 2024-03-28T22:17:57.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.941+0000 7f166effd640 1 --2- 172.21.15.16:0/10699104 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d16ee7a470 0x55d16ee8f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:17:57.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.941+0000 7f166e7fc640 1 --2- 172.21.15.16:0/10699104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d16ee7ad90 0x55d16ee81290 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f1664013fd0 tx=0x7f166401f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b542e2e79f88938 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:57.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.941+0000 7f16557fa640 1 -- 172.21.15.16:0/10699104 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166402adf0 con 0x55d16ee7ad90 2024-03-28T22:17:57.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.941+0000 7f16557fa640 1 -- 172.21.15.16:0/10699104 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1664020430 con 0x55d16ee7ad90 2024-03-28T22:17:57.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.941+0000 7f16557fa640 1 -- 172.21.15.16:0/10699104 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1664029620 con 0x55d16ee7ad90 2024-03-28T22:17:57.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.941+0000 7f1678571740 1 -- 172.21.15.16:0/10699104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d16ee7ad90 msgr2=0x55d16ee81290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:57.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.941+0000 7f1678571740 1 --2- 172.21.15.16:0/10699104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d16ee7ad90 0x55d16ee81290 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f1664013fd0 tx=0x7f166401f000 comp rx=0 tx=0).stop 2024-03-28T22:17:57.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.941+0000 7f1678571740 1 -- 172.21.15.16:0/10699104 shutdown_connections 2024-03-28T22:17:57.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.941+0000 7f1678571740 1 --2- 172.21.15.16:0/10699104 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d16ee83970 0x55d16ee76700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.941+0000 7f1678571740 1 --2- 172.21.15.16:0/10699104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d16ee7ad90 0x55d16ee81290 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.941+0000 7f1678571740 1 --2- 172.21.15.16:0/10699104 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d16ee7a470 0x55d16ee8f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.941+0000 7f1678571740 1 -- 172.21.15.16:0/10699104 >> 172.21.15.16:0/10699104 conn(0x55d16ee74ba0 msgr2=0x55d16ee75ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:57.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.941+0000 7f1678571740 1 -- 172.21.15.16:0/10699104 shutdown_connections 2024-03-28T22:17:57.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.942+0000 7f1678571740 1 -- 172.21.15.16:0/10699104 wait complete. 2024-03-28T22:17:57.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.942+0000 7f1678571740 1 -- start start 2024-03-28T22:17:57.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.942+0000 7f1678571740 1 Processor -- start 2024-03-28T22:17:57.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.942+0000 7f1678571740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d16ee7a470 0x55d16ee7d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.942+0000 7f1678571740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d16ee7ad90 0x55d16ee7db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.942+0000 7f1678571740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d16ee83970 0x55d16ee7e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.942+0000 7f1678571740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f16701aba30 con 0x55d16ee83970 2024-03-28T22:17:57.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.942+0000 7f1678571740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f16701acea0 con 0x55d16ee7a470 2024-03-28T22:17:57.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.942+0000 7f1678571740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f16701ac100 con 0x55d16ee7ad90 2024-03-28T22:17:57.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.942+0000 7f1678571740 10 client.0 _ll_register_callbacks cb 0x7f1660009e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:17:57.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.942+0000 7f166f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d16ee83970 0x55d16ee7e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:57.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.942+0000 7f166f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d16ee83970 0x55d16ee7e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55128/0 (socket says 172.21.15.16:55128) 2024-03-28T22:17:57.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.942+0000 7f166effd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d16ee7a470 0x55d16ee7d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:57.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.942+0000 7f166f7fe640 1 -- 172.21.15.16:0/1898938335 learned_addr learned my addr 172.21.15.16:0/1898938335 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:57.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.942+0000 7f166e7fc640 1 --2- 172.21.15.16:0/1898938335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d16ee7ad90 0x55d16ee7db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:57.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.943+0000 7f166f7fe640 1 -- 172.21.15.16:0/1898938335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d16ee7ad90 msgr2=0x55d16ee7db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:57.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.943+0000 7f166f7fe640 1 --2- 172.21.15.16:0/1898938335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d16ee7ad90 0x55d16ee7db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.943+0000 7f166f7fe640 1 -- 172.21.15.16:0/1898938335 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d16ee7a470 msgr2=0x55d16ee7d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:57.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.943+0000 7f166f7fe640 1 --2- 172.21.15.16:0/1898938335 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d16ee7a470 0x55d16ee7d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.943+0000 7f166f7fe640 1 -- 172.21.15.16:0/1898938335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d16eef8aa0 con 0x55d16ee83970 2024-03-28T22:17:57.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.943+0000 7f166effd640 1 --2- 172.21.15.16:0/1898938335 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d16ee7a470 0x55d16ee7d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:17:57.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.943+0000 7f166f7fe640 1 --2- 172.21.15.16:0/1898938335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d16ee83970 0x55d16ee7e0b0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f1668009430 tx=0x7f1668009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=516870e279f5e313 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:57.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.943+0000 7f164effd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:17:57.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.943+0000 7f164effd640 1 -- 172.21.15.16:0/1898938335 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1668034e60 con 0x55d16ee83970 2024-03-28T22:17:57.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.943+0000 7f164effd640 1 -- 172.21.15.16:0/1898938335 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8527+0+0 (secure 0 0 0) 0x7f16680b8e20 con 0x55d16ee83970 2024-03-28T22:17:57.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.943+0000 7f1678571740 1 -- 172.21.15.16:0/1898938335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1664020430 con 0x55d16ee83970 2024-03-28T22:17:57.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.943+0000 7f164effd640 1 -- 172.21.15.16:0/1898938335 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1668034920 con 0x55d16ee83970 2024-03-28T22:17:57.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.943+0000 7f1678571740 10 client.17046 fetch_fsmap learned FSMap version 27 2024-03-28T22:17:57.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.943+0000 7f1678571740 1 -- 172.21.15.16:0/1898938335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d16eef9d60 con 0x55d16ee83970 2024-03-28T22:17:57.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.943+0000 7f164effd640 1 -- 172.21.15.16:0/1898938335 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1664020430 con 0x55d16ee83970 2024-03-28T22:17:57.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.943+0000 7f164effd640 1 -- 172.21.15.16:0/1898938335 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f166803c030 con 0x55d16ee83970 2024-03-28T22:17:57.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.943+0000 7f1678571740 10 client.17046 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:17:57.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.943+0000 7f1678571740 10 client.17046 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-28T22:17:57.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.944+0000 7f1678571740 20 client.17046 populate_metadata read hostname 'smithi016' 2024-03-28T22:17:57.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.944+0000 7f1678571740 1 --2- 172.21.15.16:0/1898938335 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55d16eefa2f0 0x55d16eefc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:57.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.944+0000 7f1678571740 4 client.17046 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:17:57.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.944+0000 7f1678571740 1 -- 172.21.15.16:0/1898938335 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d16ede5a40 con 0x55d16eefa2f0 2024-03-28T22:17:57.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.944+0000 7f166effd640 1 --2- 172.21.15.16:0/1898938335 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55d16eefa2f0 0x55d16eefc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:57.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.944+0000 7f166effd640 1 --2- 172.21.15.16:0/1898938335 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55d16eefa2f0 0x55d16eefc710 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3035772c4b70ed1d server_cookie=c2d70c325d6a74cc in_seq=0 out_seq=0 2024-03-28T22:17:57.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.944+0000 7f164effd640 10 client.17046 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:17:57.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.944+0000 7f164effd640 1 -- 172.21.15.16:0/1898938335 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d16ede5a40 con 0x55d16eefa2f0 2024-03-28T22:17:57.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.944+0000 7f164effd640 10 client.17046 handle_command_reply: tid=0 2024-03-28T22:17:57.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.944+0000 7f1678571740 1 client.17046 shutdown 2024-03-28T22:17:57.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.944+0000 7f1678571740 1 -- 172.21.15.16:0/1898938335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d16ee83970 msgr2=0x55d16ee7e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:57.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.944+0000 7f1678571740 1 --2- 172.21.15.16:0/1898938335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d16ee83970 0x55d16ee7e0b0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f1668009430 tx=0x7f1668009650 comp rx=0 tx=0).stop 2024-03-28T22:17:57.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.944+0000 7f1678571740 1 -- 172.21.15.16:0/1898938335 shutdown_connections 2024-03-28T22:17:57.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.944+0000 7f1678571740 1 --2- 172.21.15.16:0/1898938335 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55d16eefa2f0 0x55d16eefc710 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.944+0000 7f1678571740 1 --2- 172.21.15.16:0/1898938335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d16ee83970 0x55d16ee7e0b0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.944+0000 7f1678571740 1 --2- 172.21.15.16:0/1898938335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d16ee7ad90 0x55d16ee7db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.944+0000 7f164effd640 0 client.17046 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:17:57.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.945+0000 7f1678571740 1 --2- 172.21.15.16:0/1898938335 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d16ee7a470 0x55d16ee7d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.945+0000 7f1678571740 1 -- 172.21.15.16:0/1898938335 >> 172.21.15.16:0/1898938335 conn(0x55d16ee74ba0 msgr2=0x55d16ee76880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:57.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.945+0000 7f1678571740 1 -- 172.21.15.16:0/1898938335 shutdown_connections 2024-03-28T22:17:57.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.945+0000 7f1678571740 1 -- 172.21.15.16:0/1898938335 wait complete. 2024-03-28T22:17:57.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.945+0000 7f1678571740 20 client.17046 trim_cache size 0 max 16384 2024-03-28T22:17:57.959 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:17:57.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.945+0000 7f167598c640 1 -- 172.21.15.16:0/1206562146 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f16580a1f70 msgr2=0x7f16580a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:57.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.945+0000 7f167598c640 1 --2- 172.21.15.16:0/1206562146 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f16580a1f70 0x7f16580a4430 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1664009fd0 tx=0x7f1664013820 comp rx=0 tx=0).stop 2024-03-28T22:17:57.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.945+0000 7f167598c640 1 -- 172.21.15.16:0/1206562146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16701076f0 msgr2=0x7f167019c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:57.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.945+0000 7f167598c640 1 --2- 172.21.15.16:0/1206562146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16701076f0 0x7f167019c840 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f16680348f0 tx=0x7f1668032c90 comp rx=0 tx=0).stop 2024-03-28T22:17:57.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.945+0000 7f167598c640 1 -- 172.21.15.16:0/1206562146 shutdown_connections 2024-03-28T22:17:57.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.945+0000 7f167598c640 1 --2- 172.21.15.16:0/1206562146 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f16580a1f70 0x7f16580a4430 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.945+0000 7f167598c640 1 --2- 172.21.15.16:0/1206562146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16701076f0 0x7f167019c840 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.945+0000 7f167598c640 1 --2- 172.21.15.16:0/1206562146 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16701020a0 0x7f16701984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.945+0000 7f167598c640 1 --2- 172.21.15.16:0/1206562146 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16701016e0 0x7f1670197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:57.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.945+0000 7f167598c640 1 -- 172.21.15.16:0/1206562146 >> 172.21.15.16:0/1206562146 conn(0x7f16700fd1f0 msgr2=0x7f16700fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:57.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.945+0000 7f167598c640 1 -- 172.21.15.16:0/1206562146 shutdown_connections 2024-03-28T22:17:57.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:57.945+0000 7f167598c640 1 -- 172.21.15.16:0/1206562146 wait complete. 2024-03-28T22:17:57.962 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:17:58.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:57 smithi053 ceph-mon[35979]: pgmap v226: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:17:58.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.066+0000 7ff71f047640 1 -- 172.21.15.16:0/2379053351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff718103b90 msgr2=0x7ff71810e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:58.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.066+0000 7ff71f047640 1 --2- 172.21.15.16:0/2379053351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff718103b90 0x7ff71810e6c0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7ff710009ca0 tx=0x7ff71002c1d0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.066+0000 7ff71f047640 1 -- 172.21.15.16:0/2379053351 shutdown_connections 2024-03-28T22:17:58.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.066+0000 7ff71f047640 1 --2- 172.21.15.16:0/2379053351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff718103b90 0x7ff71810e6c0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.066+0000 7ff71f047640 1 --2- 172.21.15.16:0/2379053351 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7181031f0 0x7ff718103650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.066+0000 7ff71f047640 1 --2- 172.21.15.16:0/2379053351 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff718101ab0 0x7ff718101e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.066+0000 7ff71f047640 1 -- 172.21.15.16:0/2379053351 >> 172.21.15.16:0/2379053351 conn(0x7ff7180fd1e0 msgr2=0x7ff7180ff600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:58.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.066+0000 7ff71f047640 1 -- 172.21.15.16:0/2379053351 shutdown_connections 2024-03-28T22:17:58.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.066+0000 7ff71f047640 1 -- 172.21.15.16:0/2379053351 wait complete. 2024-03-28T22:17:58.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.067+0000 7ff71f047640 1 Processor -- start 2024-03-28T22:17:58.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.067+0000 7ff71f047640 1 -- start start 2024-03-28T22:17:58.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.067+0000 7ff71f047640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff718101ab0 0x7ff718198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:58.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.067+0000 7ff71f047640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7181031f0 0x7ff7181985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:58.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.067+0000 7ff71f047640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff718103b90 0x7ff71819c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:58.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.067+0000 7ff71f047640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff718111c00 con 0x7ff718101ab0 2024-03-28T22:17:58.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.067+0000 7ff71f047640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff718111a80 con 0x7ff7181031f0 2024-03-28T22:17:58.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.067+0000 7ff71f047640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff718111d80 con 0x7ff718103b90 2024-03-28T22:17:58.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.067+0000 7ff71cde3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff718101ab0 0x7ff718198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:58.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.067+0000 7ff71cde3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff718101ab0 0x7ff718198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55142/0 (socket says 172.21.15.16:55142) 2024-03-28T22:17:58.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.067+0000 7ff71cde3640 1 -- 172.21.15.16:0/3295770024 learned_addr learned my addr 172.21.15.16:0/3295770024 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:58.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.067+0000 7ff717fff640 1 --2- 172.21.15.16:0/3295770024 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7181031f0 0x7ff7181985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:58.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.067+0000 7ff71d5e4640 1 --2- 172.21.15.16:0/3295770024 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff718103b90 0x7ff71819c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:58.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.068+0000 7ff71cde3640 1 -- 172.21.15.16:0/3295770024 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff718103b90 msgr2=0x7ff71819c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:58.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.068+0000 7ff71cde3640 1 --2- 172.21.15.16:0/3295770024 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff718103b90 0x7ff71819c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.068+0000 7ff71cde3640 1 -- 172.21.15.16:0/3295770024 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7181031f0 msgr2=0x7ff7181985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:58.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.068+0000 7ff71cde3640 1 --2- 172.21.15.16:0/3295770024 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7181031f0 0x7ff7181985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.068+0000 7ff71cde3640 1 -- 172.21.15.16:0/3295770024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff71819d000 con 0x7ff718101ab0 2024-03-28T22:17:58.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.068+0000 7ff717fff640 1 --2- 172.21.15.16:0/3295770024 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7181031f0 0x7ff7181985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:17:58.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.068+0000 7ff71cde3640 1 --2- 172.21.15.16:0/3295770024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff718101ab0 0x7ff718198060 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7ff70c00ea60 tx=0x7ff70c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:58.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.068+0000 7ff715ffb640 1 -- 172.21.15.16:0/3295770024 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff70c00cde0 con 0x7ff718101ab0 2024-03-28T22:17:58.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.068+0000 7ff71f047640 1 -- 172.21.15.16:0/3295770024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff71819d2f0 con 0x7ff718101ab0 2024-03-28T22:17:58.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.068+0000 7ff715ffb640 1 -- 172.21.15.16:0/3295770024 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff70c0035f0 con 0x7ff718101ab0 2024-03-28T22:17:58.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.068+0000 7ff715ffb640 1 -- 172.21.15.16:0/3295770024 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff70c010670 con 0x7ff718101ab0 2024-03-28T22:17:58.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.068+0000 7ff71f047640 1 -- 172.21.15.16:0/3295770024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7181a4b80 con 0x7ff718101ab0 2024-03-28T22:17:58.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.071+0000 7ff715ffb640 1 -- 172.21.15.16:0/3295770024 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7ff70c0037a0 con 0x7ff718101ab0 2024-03-28T22:17:58.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.071+0000 7ff721c2c740 1 Processor -- start 2024-03-28T22:17:58.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.071+0000 7ff721c2c740 1 -- start start 2024-03-28T22:17:58.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.071+0000 7ff715ffb640 1 --2- 172.21.15.16:0/3295770024 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff7080a1ef0 0x7ff7080a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:58.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.071+0000 7ff721c2c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1cf147470 0x55f1cf147850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:58.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.071+0000 7ff721c2c740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f1cf147d90 0x55f1cf150430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:58.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.071+0000 7ff721c2c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1cf150970 0x55f1cf152d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:58.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.071+0000 7ff721c2c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f1cf0ab1b0 con 0x55f1cf147470 2024-03-28T22:17:58.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.071+0000 7ff721c2c740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f1cf053c00 con 0x55f1cf147d90 2024-03-28T22:17:58.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.071+0000 7ff721c2c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f1cf0a7250 con 0x55f1cf150970 2024-03-28T22:17:58.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.071+0000 7ff71cde3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1cf147470 0x55f1cf147850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:58.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.071+0000 7ff717fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f1cf147d90 0x55f1cf150430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:58.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.071+0000 7ff71d5e4640 1 --2- 172.21.15.16:0/3295770024 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff7080a1ef0 0x7ff7080a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:58.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.071+0000 7ff717fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f1cf147d90 0x55f1cf150430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:39598/0 (socket says 172.21.15.16:39598) 2024-03-28T22:17:58.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.071+0000 7ff717fff640 1 -- 172.21.15.16:0/2393165373 learned_addr learned my addr 172.21.15.16:0/2393165373 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:58.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.072+0000 7ff715ffb640 1 -- 172.21.15.16:0/3295770024 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8527+0+0 (secure 0 0 0) 0x7ff70c0ce450 con 0x7ff718101ab0 2024-03-28T22:17:58.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.072+0000 7ff71d5e4640 1 --2- 172.21.15.16:0/2393165373 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1cf150970 0x55f1cf152d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:58.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.072+0000 7ff717fff640 1 -- 172.21.15.16:0/2393165373 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1cf150970 msgr2=0x55f1cf152d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:58.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.072+0000 7ff717fff640 1 --2- 172.21.15.16:0/2393165373 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1cf150970 0x55f1cf152d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.072+0000 7ff717fff640 1 -- 172.21.15.16:0/2393165373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1cf147470 msgr2=0x55f1cf147850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:58.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.072+0000 7ff717fff640 1 --2- 172.21.15.16:0/2393165373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1cf147470 0x55f1cf147850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.072+0000 7ff717fff640 1 -- 172.21.15.16:0/2393165373 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1cf0733d0 con 0x55f1cf147d90 2024-03-28T22:17:58.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.072+0000 7ff71d5e4640 1 --2- 172.21.15.16:0/2393165373 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1cf150970 0x55f1cf152d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:17:58.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.072+0000 7ff71d5e4640 1 --2- 172.21.15.16:0/3295770024 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff7080a1ef0 0x7ff7080a43b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff710004af0 tx=0x7ff710009820 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:58.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.072+0000 7ff717fff640 1 --2- 172.21.15.16:0/2393165373 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f1cf147d90 0x55f1cf150430 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff704010450 tx=0x7ff704010910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbef96f36d841172 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:58.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.072+0000 7ff702ffd640 1 -- 172.21.15.16:0/2393165373 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff704011760 con 0x55f1cf147d90 2024-03-28T22:17:58.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.072+0000 7ff702ffd640 1 -- 172.21.15.16:0/2393165373 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff704011de0 con 0x55f1cf147d90 2024-03-28T22:17:58.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.072+0000 7ff702ffd640 1 -- 172.21.15.16:0/2393165373 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff70401a380 con 0x55f1cf147d90 2024-03-28T22:17:58.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.073+0000 7ff721c2c740 1 -- 172.21.15.16:0/2393165373 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f1cf147d90 msgr2=0x55f1cf150430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:58.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.073+0000 7ff721c2c740 1 --2- 172.21.15.16:0/2393165373 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f1cf147d90 0x55f1cf150430 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff704010450 tx=0x7ff704010910 comp rx=0 tx=0).stop 2024-03-28T22:17:58.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.073+0000 7ff721c2c740 1 -- 172.21.15.16:0/2393165373 shutdown_connections 2024-03-28T22:17:58.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.073+0000 7ff721c2c740 1 --2- 172.21.15.16:0/2393165373 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1cf150970 0x55f1cf152d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.073+0000 7ff721c2c740 1 --2- 172.21.15.16:0/2393165373 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f1cf147d90 0x55f1cf150430 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.073+0000 7ff721c2c740 1 --2- 172.21.15.16:0/2393165373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1cf147470 0x55f1cf147850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.073+0000 7ff721c2c740 1 -- 172.21.15.16:0/2393165373 >> 172.21.15.16:0/2393165373 conn(0x55f1cf141ba0 msgr2=0x55f1cf141f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:58.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.073+0000 7ff721c2c740 1 -- 172.21.15.16:0/2393165373 shutdown_connections 2024-03-28T22:17:58.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.073+0000 7ff721c2c740 1 -- 172.21.15.16:0/2393165373 wait complete. 2024-03-28T22:17:58.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.073+0000 7ff721c2c740 1 -- start start 2024-03-28T22:17:58.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.073+0000 7ff721c2c740 1 Processor -- start 2024-03-28T22:17:58.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.074+0000 7ff721c2c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1cf147470 0x55f1cf14d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:58.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.074+0000 7ff721c2c740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f1cf147d90 0x55f1cf14deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:58.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.074+0000 7ff721c2c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1cf150970 0x55f1cf14e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:58.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.074+0000 7ff721c2c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f1cf153760 con 0x55f1cf150970 2024-03-28T22:17:58.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.074+0000 7ff721c2c740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f1cf049470 con 0x55f1cf147d90 2024-03-28T22:17:58.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.074+0000 7ff721c2c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f1cf1538e0 con 0x55f1cf147470 2024-03-28T22:17:58.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.074+0000 7ff721c2c740 10 client.0 _ll_register_callbacks cb 0x55f1cf066cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:17:58.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.074+0000 7ff71d5e4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1cf150970 0x55f1cf14e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:58.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.074+0000 7ff717fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f1cf147d90 0x55f1cf14deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:58.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.074+0000 7ff71d5e4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1cf150970 0x55f1cf14e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55150/0 (socket says 172.21.15.16:55150) 2024-03-28T22:17:58.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.074+0000 7ff71d5e4640 1 -- 172.21.15.16:0/3145732630 learned_addr learned my addr 172.21.15.16:0/3145732630 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:58.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.074+0000 7ff71cde3640 1 --2- 172.21.15.16:0/3145732630 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1cf147470 0x55f1cf14d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:58.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.074+0000 7ff71d5e4640 1 -- 172.21.15.16:0/3145732630 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1cf147470 msgr2=0x55f1cf14d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:58.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.074+0000 7ff71d5e4640 1 --2- 172.21.15.16:0/3145732630 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1cf147470 0x55f1cf14d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.074+0000 7ff71d5e4640 1 -- 172.21.15.16:0/3145732630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f1cf147d90 msgr2=0x55f1cf14deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:58.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.074+0000 7ff71d5e4640 1 --2- 172.21.15.16:0/3145732630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f1cf147d90 0x55f1cf14deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.074+0000 7ff71d5e4640 1 -- 172.21.15.16:0/3145732630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1cedbf3f0 con 0x55f1cf150970 2024-03-28T22:17:58.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.074+0000 7ff717fff640 1 --2- 172.21.15.16:0/3145732630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f1cf147d90 0x55f1cf14deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:17:58.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.074+0000 7ff71d5e4640 1 --2- 172.21.15.16:0/3145732630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1cf150970 0x55f1cf14e3f0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7ff710036000 tx=0x7ff71002c1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d51f8deac0c1b208 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:58.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.074+0000 7ff6f7fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:17:58.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.074+0000 7ff6f7fff640 1 -- 172.21.15.16:0/3145732630 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff710052020 con 0x55f1cf150970 2024-03-28T22:17:58.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.075+0000 7ff6f7fff640 1 -- 172.21.15.16:0/3145732630 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8527+0+0 (secure 0 0 0) 0x7ff710034bc0 con 0x55f1cf150970 2024-03-28T22:17:58.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.075+0000 7ff721c2c740 1 -- 172.21.15.16:0/3145732630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff704011de0 con 0x55f1cf150970 2024-03-28T22:17:58.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.075+0000 7ff6f7fff640 1 -- 172.21.15.16:0/3145732630 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff7100039d0 con 0x55f1cf150970 2024-03-28T22:17:58.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.075+0000 7ff721c2c740 10 client.17058 fetch_fsmap learned FSMap version 27 2024-03-28T22:17:58.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.075+0000 7ff721c2c740 1 -- 172.21.15.16:0/3145732630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f1cee897d0 con 0x55f1cf150970 2024-03-28T22:17:58.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.075+0000 7ff6f7fff640 1 -- 172.21.15.16:0/3145732630 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff704011de0 con 0x55f1cf150970 2024-03-28T22:17:58.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.075+0000 7ff6f7fff640 1 -- 172.21.15.16:0/3145732630 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7ff71003f070 con 0x55f1cf150970 2024-03-28T22:17:58.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.075+0000 7ff721c2c740 10 client.17058 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:17:58.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.075+0000 7ff721c2c740 10 client.17058 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T22:17:58.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.075+0000 7ff721c2c740 20 client.17058 populate_metadata read hostname 'smithi016' 2024-03-28T22:17:58.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.075+0000 7ff721c2c740 1 --2- 172.21.15.16:0/3145732630 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55f1cf160980 0x55f1cf162dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:58.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.075+0000 7ff721c2c740 4 client.17058 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:17:58.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.075+0000 7ff721c2c740 1 -- 172.21.15.16:0/3145732630 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f1cf0b86f0 con 0x55f1cf160980 2024-03-28T22:17:58.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.075+0000 7ff71cde3640 1 --2- 172.21.15.16:0/3145732630 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55f1cf160980 0x55f1cf162dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:58.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.076+0000 7ff6f7fff640 10 client.17058 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:17:58.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.076+0000 7ff71cde3640 1 --2- 172.21.15.16:0/3145732630 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55f1cf160980 0x55f1cf162dc0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2daca5ca0cbefc8a server_cookie=9dc77fdab21a73b1 in_seq=0 out_seq=0 2024-03-28T22:17:58.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.076+0000 7ff6f7fff640 1 -- 172.21.15.16:0/3145732630 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f1cf0b86f0 con 0x55f1cf160980 2024-03-28T22:17:58.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.076+0000 7ff6f7fff640 10 client.17058 handle_command_reply: tid=0 2024-03-28T22:17:58.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.076+0000 7ff721c2c740 1 client.17058 shutdown 2024-03-28T22:17:58.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.077+0000 7ff721c2c740 1 -- 172.21.15.16:0/3145732630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1cf150970 msgr2=0x55f1cf14e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:58.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.077+0000 7ff721c2c740 1 --2- 172.21.15.16:0/3145732630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1cf150970 0x55f1cf14e3f0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7ff710036000 tx=0x7ff71002c1d0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.077+0000 7ff721c2c740 1 -- 172.21.15.16:0/3145732630 shutdown_connections 2024-03-28T22:17:58.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.077+0000 7ff721c2c740 1 --2- 172.21.15.16:0/3145732630 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55f1cf160980 0x55f1cf162dc0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.077+0000 7ff721c2c740 1 --2- 172.21.15.16:0/3145732630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1cf150970 0x55f1cf14e3f0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.077+0000 7ff721c2c740 1 --2- 172.21.15.16:0/3145732630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f1cf147d90 0x55f1cf14deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.077+0000 7ff721c2c740 1 --2- 172.21.15.16:0/3145732630 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1cf147470 0x55f1cf14d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.077+0000 7ff6f7fff640 0 client.17058 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:17:58.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.077+0000 7ff721c2c740 1 -- 172.21.15.16:0/3145732630 >> 172.21.15.16:0/3145732630 conn(0x55f1cf141ba0 msgr2=0x55f1cf1438f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:58.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.077+0000 7ff721c2c740 1 -- 172.21.15.16:0/3145732630 shutdown_connections 2024-03-28T22:17:58.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.077+0000 7ff721c2c740 1 -- 172.21.15.16:0/3145732630 wait complete. 2024-03-28T22:17:58.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.077+0000 7ff721c2c740 20 client.17058 trim_cache size 0 max 16384 2024-03-28T22:17:58.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.090+0000 7ff721c2c740 1 Processor -- start 2024-03-28T22:17:58.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.090+0000 7ff721c2c740 1 -- start start 2024-03-28T22:17:58.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.090+0000 7ff721c2c740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f1cf147470 0x55f1cf15c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:58.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.090+0000 7ff721c2c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1cf147d90 0x55f1cf1436e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:58.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.090+0000 7ff721c2c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1cf150970 0x55f1cf168690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:58.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.090+0000 7ff721c2c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff7181afc20 con 0x55f1cf150970 2024-03-28T22:17:58.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.090+0000 7ff721c2c740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff7181ad4f0 con 0x55f1cf147470 2024-03-28T22:17:58.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.090+0000 7ff721c2c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff7181ab360 con 0x55f1cf147d90 2024-03-28T22:17:58.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.090+0000 7ff71d5e4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1cf150970 0x55f1cf168690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:58.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.090+0000 7ff71d5e4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1cf150970 0x55f1cf168690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55160/0 (socket says 172.21.15.16:55160) 2024-03-28T22:17:58.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.090+0000 7ff71d5e4640 1 -- 172.21.15.16:0/351678967 learned_addr learned my addr 172.21.15.16:0/351678967 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:58.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.090+0000 7ff71cde3640 1 --2- 172.21.15.16:0/351678967 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f1cf147470 0x55f1cf15c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:58.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.090+0000 7ff71d5e4640 1 -- 172.21.15.16:0/351678967 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1cf147d90 msgr2=0x55f1cf1436e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:58.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.090+0000 7ff71d5e4640 1 --2- 172.21.15.16:0/351678967 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1cf147d90 0x55f1cf1436e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.090+0000 7ff71d5e4640 1 -- 172.21.15.16:0/351678967 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f1cf147470 msgr2=0x55f1cf15c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:58.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.090+0000 7ff71d5e4640 1 --2- 172.21.15.16:0/351678967 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f1cf147470 0x55f1cf15c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.090+0000 7ff71d5e4640 1 -- 172.21.15.16:0/351678967 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1cf152d60 con 0x55f1cf150970 2024-03-28T22:17:58.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.091+0000 7ff71d5e4640 1 --2- 172.21.15.16:0/351678967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1cf150970 0x55f1cf168690 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7ff710037040 tx=0x7ff710009620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1cf9448fee4eb45 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:58.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.091+0000 7ff7027fc640 1 -- 172.21.15.16:0/351678967 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff710052040 con 0x55f1cf150970 2024-03-28T22:17:58.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.091+0000 7ff7027fc640 1 -- 172.21.15.16:0/351678967 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff710067070 con 0x55f1cf150970 2024-03-28T22:17:58.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.091+0000 7ff7027fc640 1 -- 172.21.15.16:0/351678967 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff710061d00 con 0x55f1cf150970 2024-03-28T22:17:58.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.091+0000 7ff721c2c740 1 -- 172.21.15.16:0/351678967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1cf150970 msgr2=0x55f1cf168690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:58.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.091+0000 7ff721c2c740 1 --2- 172.21.15.16:0/351678967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1cf150970 0x55f1cf168690 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7ff710037040 tx=0x7ff710009620 comp rx=0 tx=0).stop 2024-03-28T22:17:58.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.091+0000 7ff721c2c740 1 -- 172.21.15.16:0/351678967 shutdown_connections 2024-03-28T22:17:58.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.091+0000 7ff721c2c740 1 --2- 172.21.15.16:0/351678967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1cf150970 0x55f1cf168690 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.091+0000 7ff721c2c740 1 --2- 172.21.15.16:0/351678967 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1cf147d90 0x55f1cf1436e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.091+0000 7ff721c2c740 1 --2- 172.21.15.16:0/351678967 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f1cf147470 0x55f1cf15c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.091+0000 7ff721c2c740 1 -- 172.21.15.16:0/351678967 >> 172.21.15.16:0/351678967 conn(0x55f1cf141ba0 msgr2=0x55f1cf142fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:58.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.091+0000 7ff721c2c740 1 -- 172.21.15.16:0/351678967 shutdown_connections 2024-03-28T22:17:58.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.091+0000 7ff721c2c740 1 -- 172.21.15.16:0/351678967 wait complete. 2024-03-28T22:17:58.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.092+0000 7ff721c2c740 1 -- start start 2024-03-28T22:17:58.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.092+0000 7ff721c2c740 1 Processor -- start 2024-03-28T22:17:58.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.092+0000 7ff721c2c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1cf147470 0x55f1cf1547a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:58.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.092+0000 7ff721c2c740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f1cf147d90 0x55f1cf154ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:58.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.092+0000 7ff721c2c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1cf150970 0x55f1cf1c5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:58.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.092+0000 7ff721c2c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff7181ab9b0 con 0x55f1cf147470 2024-03-28T22:17:58.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.092+0000 7ff721c2c740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff7181ace20 con 0x55f1cf147d90 2024-03-28T22:17:58.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.092+0000 7ff721c2c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff7181ac080 con 0x55f1cf150970 2024-03-28T22:17:58.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.092+0000 7ff721c2c740 10 client.0 _ll_register_callbacks cb 0x7ff710032930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:17:58.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.092+0000 7ff71cde3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1cf147470 0x55f1cf1547a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:58.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.092+0000 7ff71cde3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1cf147470 0x55f1cf1547a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55170/0 (socket says 172.21.15.16:55170) 2024-03-28T22:17:58.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.092+0000 7ff71cde3640 1 -- 172.21.15.16:0/2117799659 learned_addr learned my addr 172.21.15.16:0/2117799659 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:58.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.092+0000 7ff717fff640 1 --2- 172.21.15.16:0/2117799659 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f1cf147d90 0x55f1cf154ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:58.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.092+0000 7ff71d5e4640 1 --2- 172.21.15.16:0/2117799659 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1cf150970 0x55f1cf1c5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:58.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.092+0000 7ff71cde3640 1 -- 172.21.15.16:0/2117799659 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1cf150970 msgr2=0x55f1cf1c5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:58.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.092+0000 7ff71cde3640 1 --2- 172.21.15.16:0/2117799659 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1cf150970 0x55f1cf1c5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.092+0000 7ff71cde3640 1 -- 172.21.15.16:0/2117799659 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f1cf147d90 msgr2=0x55f1cf154ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:58.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.092+0000 7ff71cde3640 1 --2- 172.21.15.16:0/2117799659 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f1cf147d90 0x55f1cf154ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.093+0000 7ff71cde3640 1 -- 172.21.15.16:0/2117799659 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1cf1555e0 con 0x55f1cf147470 2024-03-28T22:17:58.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.093+0000 7ff71cde3640 1 --2- 172.21.15.16:0/2117799659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1cf147470 0x55f1cf1547a0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7ff70c09add0 tx=0x7ff70c014fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17576f24fd26694a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:58.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.093+0000 7ff6f7fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:17:58.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.093+0000 7ff6f7fff640 1 -- 172.21.15.16:0/2117799659 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff70c0b7030 con 0x55f1cf147470 2024-03-28T22:17:58.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.093+0000 7ff6f7fff640 1 -- 172.21.15.16:0/2117799659 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8527+0+0 (secure 0 0 0) 0x7ff70c0b92c0 con 0x55f1cf147470 2024-03-28T22:17:58.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.093+0000 7ff721c2c740 1 -- 172.21.15.16:0/2117799659 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff710067070 con 0x55f1cf147470 2024-03-28T22:17:58.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.093+0000 7ff6f7fff640 1 -- 172.21.15.16:0/2117799659 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff70c09d040 con 0x55f1cf147470 2024-03-28T22:17:58.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.093+0000 7ff721c2c740 10 client.17064 fetch_fsmap learned FSMap version 27 2024-03-28T22:17:58.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.093+0000 7ff721c2c740 1 -- 172.21.15.16:0/2117799659 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f1cf1c7330 con 0x55f1cf147470 2024-03-28T22:17:58.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.093+0000 7ff6f7fff640 1 -- 172.21.15.16:0/2117799659 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff710067070 con 0x55f1cf147470 2024-03-28T22:17:58.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.093+0000 7ff6f7fff640 1 -- 172.21.15.16:0/2117799659 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7ff70c0cd040 con 0x55f1cf147470 2024-03-28T22:17:58.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.093+0000 7ff721c2c740 10 client.17064 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:17:58.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.093+0000 7ff721c2c740 10 client.17064 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T22:17:58.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.093+0000 7ff721c2c740 20 client.17064 populate_metadata read hostname 'smithi016' 2024-03-28T22:17:58.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.093+0000 7ff721c2c740 1 --2- 172.21.15.16:0/2117799659 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55f1cf1c7a50 0x55f1cf1c9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:58.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.093+0000 7ff721c2c740 4 client.17064 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:17:58.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.093+0000 7ff721c2c740 1 -- 172.21.15.16:0/2117799659 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f1cf0b2a40 con 0x55f1cf1c7a50 2024-03-28T22:17:58.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.094+0000 7ff717fff640 1 --2- 172.21.15.16:0/2117799659 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55f1cf1c7a50 0x55f1cf1c9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:58.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.094+0000 7ff6f7fff640 10 client.17064 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:17:58.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.094+0000 7ff717fff640 1 --2- 172.21.15.16:0/2117799659 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55f1cf1c7a50 0x55f1cf1c9e40 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bead9799f2c577f server_cookie=974a69c5cfa54bf7 in_seq=0 out_seq=0 2024-03-28T22:17:58.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.094+0000 7ff6f7fff640 1 -- 172.21.15.16:0/2117799659 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f1cf0b2a40 con 0x55f1cf1c7a50 2024-03-28T22:17:58.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.094+0000 7ff6f7fff640 10 client.17064 handle_command_reply: tid=0 2024-03-28T22:17:58.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.094+0000 7ff721c2c740 1 client.17064 shutdown 2024-03-28T22:17:58.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.094+0000 7ff721c2c740 1 -- 172.21.15.16:0/2117799659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1cf147470 msgr2=0x55f1cf1547a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:58.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.094+0000 7ff721c2c740 1 --2- 172.21.15.16:0/2117799659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1cf147470 0x55f1cf1547a0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7ff70c09add0 tx=0x7ff70c014fb0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.094+0000 7ff721c2c740 1 -- 172.21.15.16:0/2117799659 shutdown_connections 2024-03-28T22:17:58.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.094+0000 7ff721c2c740 1 --2- 172.21.15.16:0/2117799659 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55f1cf1c7a50 0x55f1cf1c9e40 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.094+0000 7ff721c2c740 1 --2- 172.21.15.16:0/2117799659 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1cf150970 0x55f1cf1c5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.094+0000 7ff721c2c740 1 --2- 172.21.15.16:0/2117799659 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f1cf147d90 0x55f1cf154ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.094+0000 7ff6f7fff640 0 client.17064 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:17:58.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.094+0000 7ff721c2c740 1 --2- 172.21.15.16:0/2117799659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1cf147470 0x55f1cf1547a0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.094+0000 7ff721c2c740 1 -- 172.21.15.16:0/2117799659 >> 172.21.15.16:0/2117799659 conn(0x55f1cf141ba0 msgr2=0x55f1cf143860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:58.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.094+0000 7ff721c2c740 1 -- 172.21.15.16:0/2117799659 shutdown_connections 2024-03-28T22:17:58.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.094+0000 7ff721c2c740 1 -- 172.21.15.16:0/2117799659 wait complete. 2024-03-28T22:17:58.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.094+0000 7ff721c2c740 20 client.17064 trim_cache size 0 max 16384 2024-03-28T22:17:58.108 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:17:58.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.095+0000 7ff71f047640 1 -- 172.21.15.16:0/3295770024 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff7080a1ef0 msgr2=0x7ff7080a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:58.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.095+0000 7ff71f047640 1 --2- 172.21.15.16:0/3295770024 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff7080a1ef0 0x7ff7080a43b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff710004af0 tx=0x7ff710009820 comp rx=0 tx=0).stop 2024-03-28T22:17:58.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.095+0000 7ff71f047640 1 -- 172.21.15.16:0/3295770024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff718101ab0 msgr2=0x7ff718198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:58.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.095+0000 7ff71f047640 1 --2- 172.21.15.16:0/3295770024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff718101ab0 0x7ff718198060 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7ff70c00ea60 tx=0x7ff70c00ef20 comp rx=0 tx=0).stop 2024-03-28T22:17:58.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.095+0000 7ff71f047640 1 -- 172.21.15.16:0/3295770024 shutdown_connections 2024-03-28T22:17:58.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.095+0000 7ff71f047640 1 --2- 172.21.15.16:0/3295770024 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff7080a1ef0 0x7ff7080a43b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.095+0000 7ff71f047640 1 --2- 172.21.15.16:0/3295770024 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff718103b90 0x7ff71819c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.095+0000 7ff71f047640 1 --2- 172.21.15.16:0/3295770024 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7181031f0 0x7ff7181985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.095+0000 7ff71f047640 1 --2- 172.21.15.16:0/3295770024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff718101ab0 0x7ff718198060 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.095+0000 7ff71f047640 1 -- 172.21.15.16:0/3295770024 >> 172.21.15.16:0/3295770024 conn(0x7ff7180fd1e0 msgr2=0x7ff718107560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:58.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.095+0000 7ff71f047640 1 -- 172.21.15.16:0/3295770024 shutdown_connections 2024-03-28T22:17:58.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.095+0000 7ff71f047640 1 -- 172.21.15.16:0/3295770024 wait complete. 2024-03-28T22:17:58.111 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:17:58.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.215+0000 7fe9b6a47640 1 -- 172.21.15.16:0/371048232 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9b0100fa0 msgr2=0x7fe9b010d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:58.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.215+0000 7fe9b6a47640 1 --2- 172.21.15.16:0/371048232 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9b0100fa0 0x7fe9b010d4b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe9a8009ca0 tx=0x7fe9a802c0f0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.215+0000 7fe9b6a47640 1 -- 172.21.15.16:0/371048232 shutdown_connections 2024-03-28T22:17:58.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.215+0000 7fe9b6a47640 1 --2- 172.21.15.16:0/371048232 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9b0100fa0 0x7fe9b010d4b0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.215+0000 7fe9b6a47640 1 --2- 172.21.15.16:0/371048232 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe9b01005e0 0x7fe9b0100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.215+0000 7fe9b6a47640 1 --2- 172.21.15.16:0/371048232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9b0106680 0x7fe9b0106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.215+0000 7fe9b6a47640 1 -- 172.21.15.16:0/371048232 >> 172.21.15.16:0/371048232 conn(0x7fe9b0077770 msgr2=0x7fe9b0077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:58.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.215+0000 7fe9b6a47640 1 -- 172.21.15.16:0/371048232 shutdown_connections 2024-03-28T22:17:58.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.215+0000 7fe9b6a47640 1 -- 172.21.15.16:0/371048232 wait complete. 2024-03-28T22:17:58.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.216+0000 7fe9b6a47640 1 Processor -- start 2024-03-28T22:17:58.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.216+0000 7fe9b6a47640 1 -- start start 2024-03-28T22:17:58.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.216+0000 7fe9b6a47640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9b01005e0 0x7fe9b0197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:58.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.216+0000 7fe9b6a47640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9b0100fa0 0x7fe9b0198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:58.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.216+0000 7fe9b6a47640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe9b0106680 0x7fe9b019c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:58.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.216+0000 7fe9b6a47640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe9b01109c0 con 0x7fe9b01005e0 2024-03-28T22:17:58.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.216+0000 7fe9b6a47640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe9b0110840 con 0x7fe9b0100fa0 2024-03-28T22:17:58.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.216+0000 7fe9b6a47640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe9b0110b40 con 0x7fe9b0106680 2024-03-28T22:17:58.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.216+0000 7fe9affff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9b01005e0 0x7fe9b0197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:58.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.216+0000 7fe9affff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9b01005e0 0x7fe9b0197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55186/0 (socket says 172.21.15.16:55186) 2024-03-28T22:17:58.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.216+0000 7fe9affff640 1 -- 172.21.15.16:0/2097108852 learned_addr learned my addr 172.21.15.16:0/2097108852 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:58.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.216+0000 7fe9af7fe640 1 --2- 172.21.15.16:0/2097108852 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9b0100fa0 0x7fe9b0198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:58.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.216+0000 7fe9b4fe4640 1 --2- 172.21.15.16:0/2097108852 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe9b0106680 0x7fe9b019c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:58.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.216+0000 7fe9affff640 1 -- 172.21.15.16:0/2097108852 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe9b0106680 msgr2=0x7fe9b019c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:58.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.216+0000 7fe9affff640 1 --2- 172.21.15.16:0/2097108852 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe9b0106680 0x7fe9b019c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.216+0000 7fe9affff640 1 -- 172.21.15.16:0/2097108852 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9b0100fa0 msgr2=0x7fe9b0198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:58.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.216+0000 7fe9affff640 1 --2- 172.21.15.16:0/2097108852 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9b0100fa0 0x7fe9b0198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.216+0000 7fe9affff640 1 -- 172.21.15.16:0/2097108852 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9b019ceb0 con 0x7fe9b01005e0 2024-03-28T22:17:58.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.216+0000 7fe9b4fe4640 1 --2- 172.21.15.16:0/2097108852 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe9b0106680 0x7fe9b019c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:17:58.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.217+0000 7fe9affff640 1 --2- 172.21.15.16:0/2097108852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9b01005e0 0x7fe9b0197f10 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fe9a000e9c0 tx=0x7fe9a000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:58.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.217+0000 7fe9ad7fa640 1 -- 172.21.15.16:0/2097108852 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a000cd80 con 0x7fe9b01005e0 2024-03-28T22:17:58.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.217+0000 7fe9ad7fa640 1 -- 172.21.15.16:0/2097108852 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9a00035f0 con 0x7fe9b01005e0 2024-03-28T22:17:58.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.217+0000 7fe9b6a47640 1 -- 172.21.15.16:0/2097108852 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9b019d1a0 con 0x7fe9b01005e0 2024-03-28T22:17:58.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.217+0000 7fe9ad7fa640 1 -- 172.21.15.16:0/2097108852 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a0010690 con 0x7fe9b01005e0 2024-03-28T22:17:58.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.217+0000 7fe9b6a47640 1 -- 172.21.15.16:0/2097108852 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9b01a4a30 con 0x7fe9b01005e0 2024-03-28T22:17:58.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.219+0000 7fe9ad7fa640 1 -- 172.21.15.16:0/2097108852 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7fe9a00037a0 con 0x7fe9b01005e0 2024-03-28T22:17:58.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.220+0000 7fe9b962f740 1 Processor -- start 2024-03-28T22:17:58.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.220+0000 7fe9b962f740 1 -- start start 2024-03-28T22:17:58.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.220+0000 7fe9b962f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b727110880 0x55b727110c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:58.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.220+0000 7fe9b962f740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b7271111a0 0x55b727119840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:58.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.220+0000 7fe9b962f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b727119d80 0x55b72711c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:58.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.220+0000 7fe9b962f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b7270701b0 con 0x55b727119d80 2024-03-28T22:17:58.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.220+0000 7fe9b962f740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55b727018c00 con 0x55b7271111a0 2024-03-28T22:17:58.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.220+0000 7fe9b962f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55b72706c250 con 0x55b727110880 2024-03-28T22:17:58.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.220+0000 7fe9af7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b727119d80 0x55b72711c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:58.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.220+0000 7fe9ad7fa640 1 --2- 172.21.15.16:0/2097108852 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fe9980a1ef0 0x7fe9980a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:58.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.220+0000 7fe9af7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b727119d80 0x55b72711c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55200/0 (socket says 172.21.15.16:55200) 2024-03-28T22:17:58.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.220+0000 7fe9affff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b7271111a0 0x55b727119840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:58.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.220+0000 7fe9af7fe640 1 -- 172.21.15.16:0/734061891 learned_addr learned my addr 172.21.15.16:0/734061891 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:58.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.220+0000 7fe9b4fe4640 1 --2- 172.21.15.16:0/734061891 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b727110880 0x55b727110c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:58.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.220+0000 7fe9b4fe4640 1 --2- 172.21.15.16:0/2097108852 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fe9980a1ef0 0x7fe9980a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:58.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.220+0000 7fe9af7fe640 1 -- 172.21.15.16:0/734061891 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b727110880 msgr2=0x55b727110c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:58.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.221+0000 7fe9af7fe640 1 --2- 172.21.15.16:0/734061891 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b727110880 0x55b727110c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.221+0000 7fe9af7fe640 1 -- 172.21.15.16:0/734061891 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b7271111a0 msgr2=0x55b727119840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:58.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.221+0000 7fe9af7fe640 1 --2- 172.21.15.16:0/734061891 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b7271111a0 0x55b727119840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.221+0000 7fe9ad7fa640 1 -- 172.21.15.16:0/2097108852 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8527+0+0 (secure 0 0 0) 0x7fe9a00ce5a0 con 0x7fe9b01005e0 2024-03-28T22:17:58.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.221+0000 7fe9af7fe640 1 -- 172.21.15.16:0/734061891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7270383d0 con 0x55b727119d80 2024-03-28T22:17:58.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.221+0000 7fe9af7fe640 1 --2- 172.21.15.16:0/734061891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b727119d80 0x55b72711c170 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fe9a400bd10 tx=0x7fe9a40107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9242dacf7d4dd00d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:58.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.221+0000 7fe9b4fe4640 1 --2- 172.21.15.16:0/2097108852 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fe9980a1ef0 0x7fe9980a43b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe9a8037040 tx=0x7fe9a80095c0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:58.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.221+0000 7fe9967fc640 1 -- 172.21.15.16:0/734061891 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a4005220 con 0x55b727119d80 2024-03-28T22:17:58.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.221+0000 7fe9967fc640 1 -- 172.21.15.16:0/734061891 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9a4004c80 con 0x55b727119d80 2024-03-28T22:17:58.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.221+0000 7fe9967fc640 1 -- 172.21.15.16:0/734061891 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a4011950 con 0x55b727119d80 2024-03-28T22:17:58.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.222+0000 7fe9b962f740 1 -- 172.21.15.16:0/734061891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b727119d80 msgr2=0x55b72711c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:58.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.222+0000 7fe9b962f740 1 --2- 172.21.15.16:0/734061891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b727119d80 0x55b72711c170 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fe9a400bd10 tx=0x7fe9a40107a0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.222+0000 7fe9b962f740 1 -- 172.21.15.16:0/734061891 shutdown_connections 2024-03-28T22:17:58.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.222+0000 7fe9b962f740 1 --2- 172.21.15.16:0/734061891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b727119d80 0x55b72711c170 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.222+0000 7fe9b962f740 1 --2- 172.21.15.16:0/734061891 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b7271111a0 0x55b727119840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.222+0000 7fe9b962f740 1 --2- 172.21.15.16:0/734061891 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b727110880 0x55b727110c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.222+0000 7fe9b962f740 1 -- 172.21.15.16:0/734061891 >> 172.21.15.16:0/734061891 conn(0x55b72710afb0 msgr2=0x55b72710b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:58.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.222+0000 7fe9b962f740 1 -- 172.21.15.16:0/734061891 shutdown_connections 2024-03-28T22:17:58.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.222+0000 7fe9b962f740 1 -- 172.21.15.16:0/734061891 wait complete. 2024-03-28T22:17:58.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.222+0000 7fe9b962f740 1 -- start start 2024-03-28T22:17:58.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.222+0000 7fe9b962f740 1 Processor -- start 2024-03-28T22:17:58.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.222+0000 7fe9b962f740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b727110880 0x55b727116dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:58.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.222+0000 7fe9b962f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b7271111a0 0x55b727117300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:58.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.222+0000 7fe9b962f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b727119d80 0x55b727117840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:58.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.222+0000 7fe9b962f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b72711cb70 con 0x55b7271111a0 2024-03-28T22:17:58.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.222+0000 7fe9b962f740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55b72700e470 con 0x55b727110880 2024-03-28T22:17:58.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.222+0000 7fe9b962f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55b72711ccf0 con 0x55b727119d80 2024-03-28T22:17:58.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.222+0000 7fe9b962f740 10 client.0 _ll_register_callbacks cb 0x55b72702bcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:17:58.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.223+0000 7fe9af7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b7271111a0 0x55b727117300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:58.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.223+0000 7fe9affff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b727110880 0x55b727116dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:58.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.223+0000 7fe9af7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b7271111a0 0x55b727117300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55216/0 (socket says 172.21.15.16:55216) 2024-03-28T22:17:58.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.223+0000 7fe9af7fe640 1 -- 172.21.15.16:0/1942495690 learned_addr learned my addr 172.21.15.16:0/1942495690 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:58.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.223+0000 7fe9b4fe4640 1 --2- 172.21.15.16:0/1942495690 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b727119d80 0x55b727117840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:58.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.223+0000 7fe9af7fe640 1 -- 172.21.15.16:0/1942495690 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b727119d80 msgr2=0x55b727117840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:58.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.223+0000 7fe9af7fe640 1 --2- 172.21.15.16:0/1942495690 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b727119d80 0x55b727117840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.223+0000 7fe9af7fe640 1 -- 172.21.15.16:0/1942495690 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b727110880 msgr2=0x55b727116dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:58.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.223+0000 7fe9af7fe640 1 --2- 172.21.15.16:0/1942495690 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b727110880 0x55b727116dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.223+0000 7fe9af7fe640 1 -- 172.21.15.16:0/1942495690 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b726d843f0 con 0x55b7271111a0 2024-03-28T22:17:58.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.223+0000 7fe9b4fe4640 1 --2- 172.21.15.16:0/1942495690 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b727119d80 0x55b727117840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:17:58.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.223+0000 7fe9af7fe640 1 --2- 172.21.15.16:0/1942495690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b7271111a0 0x55b727117300 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4010ca0 tx=0x7fe9a40048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78b17d6db67f0e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:58.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.223+0000 7fe98f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:17:58.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.223+0000 7fe98f7fe640 1 -- 172.21.15.16:0/1942495690 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a4011440 con 0x55b7271111a0 2024-03-28T22:17:58.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.223+0000 7fe98f7fe640 1 -- 172.21.15.16:0/1942495690 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8527+0+0 (secure 0 0 0) 0x7fe9a4005040 con 0x55b7271111a0 2024-03-28T22:17:58.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.223+0000 7fe9b962f740 1 -- 172.21.15.16:0/1942495690 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9a4004c80 con 0x55b7271111a0 2024-03-28T22:17:58.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.223+0000 7fe98f7fe640 1 -- 172.21.15.16:0/1942495690 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe9a4004ae0 con 0x55b7271111a0 2024-03-28T22:17:58.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.223+0000 7fe9b962f740 10 client.17076 fetch_fsmap learned FSMap version 27 2024-03-28T22:17:58.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.223+0000 7fe9b962f740 1 -- 172.21.15.16:0/1942495690 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b726e4e7d0 con 0x55b7271111a0 2024-03-28T22:17:58.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.224+0000 7fe98f7fe640 1 -- 172.21.15.16:0/1942495690 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9a4004c80 con 0x55b7271111a0 2024-03-28T22:17:58.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.224+0000 7fe98f7fe640 1 -- 172.21.15.16:0/1942495690 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7fe9a401b430 con 0x55b7271111a0 2024-03-28T22:17:58.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.224+0000 7fe9b962f740 10 client.17076 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:17:58.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.224+0000 7fe9b962f740 10 client.17076 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:17:58.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.224+0000 7fe9b962f740 20 client.17076 populate_metadata read hostname 'smithi016' 2024-03-28T22:17:58.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.224+0000 7fe9b962f740 1 --2- 172.21.15.16:0/1942495690 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b727129d60 0x55b72712c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:58.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.224+0000 7fe9b962f740 4 client.17076 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:17:58.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.224+0000 7fe9b962f740 1 -- 172.21.15.16:0/1942495690 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b72707d6f0 con 0x55b727129d60 2024-03-28T22:17:58.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.224+0000 7fe9affff640 1 --2- 172.21.15.16:0/1942495690 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b727129d60 0x55b72712c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:58.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.224+0000 7fe9affff640 1 --2- 172.21.15.16:0/1942495690 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b727129d60 0x55b72712c150 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74d10b97bd5de202 server_cookie=2cb40ff39036e80a in_seq=0 out_seq=0 2024-03-28T22:17:58.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.224+0000 7fe98f7fe640 10 client.17076 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:17:58.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.225+0000 7fe98f7fe640 1 -- 172.21.15.16:0/1942495690 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b72707d6f0 con 0x55b727129d60 2024-03-28T22:17:58.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.225+0000 7fe98f7fe640 10 client.17076 handle_command_reply: tid=0 2024-03-28T22:17:58.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.225+0000 7fe9b962f740 1 client.17076 shutdown 2024-03-28T22:17:58.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.225+0000 7fe9b962f740 1 -- 172.21.15.16:0/1942495690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b7271111a0 msgr2=0x55b727117300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:58.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.225+0000 7fe9b962f740 1 --2- 172.21.15.16:0/1942495690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b7271111a0 0x55b727117300 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4010ca0 tx=0x7fe9a40048f0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.225+0000 7fe9b962f740 1 -- 172.21.15.16:0/1942495690 shutdown_connections 2024-03-28T22:17:58.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.225+0000 7fe9b962f740 1 --2- 172.21.15.16:0/1942495690 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b727129d60 0x55b72712c150 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.225+0000 7fe9b962f740 1 --2- 172.21.15.16:0/1942495690 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b727119d80 0x55b727117840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.225+0000 7fe9b962f740 1 --2- 172.21.15.16:0/1942495690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b7271111a0 0x55b727117300 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.225+0000 7fe9b962f740 1 --2- 172.21.15.16:0/1942495690 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b727110880 0x55b727116dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.225+0000 7fe98f7fe640 0 client.17076 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:17:58.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.225+0000 7fe9b962f740 1 -- 172.21.15.16:0/1942495690 >> 172.21.15.16:0/1942495690 conn(0x55b72710afb0 msgr2=0x55b72710c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:58.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.226+0000 7fe9b962f740 1 -- 172.21.15.16:0/1942495690 shutdown_connections 2024-03-28T22:17:58.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.226+0000 7fe9b962f740 1 -- 172.21.15.16:0/1942495690 wait complete. 2024-03-28T22:17:58.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.226+0000 7fe9b962f740 20 client.17076 trim_cache size 0 max 16384 2024-03-28T22:17:58.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.239+0000 7fe9b962f740 1 Processor -- start 2024-03-28T22:17:58.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.239+0000 7fe9b962f740 1 -- start start 2024-03-28T22:17:58.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.239+0000 7fe9b962f740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b727110880 0x55b7271194f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:58.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.239+0000 7fe9b962f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b7271111a0 0x55b727125940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:58.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.239+0000 7fe9b962f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b727119d80 0x55b727105bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:58.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.239+0000 7fe9b962f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe9b01adaa0 con 0x55b7271111a0 2024-03-28T22:17:58.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.239+0000 7fe9b962f740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe9b01ab370 con 0x55b727110880 2024-03-28T22:17:58.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.239+0000 7fe9b962f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe9b01a91e0 con 0x55b727119d80 2024-03-28T22:17:58.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.239+0000 7fe9af7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b7271111a0 0x55b727125940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:58.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.239+0000 7fe9af7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b7271111a0 0x55b727125940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55220/0 (socket says 172.21.15.16:55220) 2024-03-28T22:17:58.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.239+0000 7fe9af7fe640 1 -- 172.21.15.16:0/319674776 learned_addr learned my addr 172.21.15.16:0/319674776 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:58.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.239+0000 7fe9affff640 1 --2- 172.21.15.16:0/319674776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b727110880 0x55b7271194f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:58.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.239+0000 7fe9af7fe640 1 -- 172.21.15.16:0/319674776 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b727119d80 msgr2=0x55b727105bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:58.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.239+0000 7fe9af7fe640 1 --2- 172.21.15.16:0/319674776 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b727119d80 0x55b727105bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.239+0000 7fe9af7fe640 1 -- 172.21.15.16:0/319674776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b727110880 msgr2=0x55b7271194f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:58.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.239+0000 7fe9af7fe640 1 --2- 172.21.15.16:0/319674776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b727110880 0x55b7271194f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.239+0000 7fe9af7fe640 1 -- 172.21.15.16:0/319674776 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b727110fd0 con 0x55b7271111a0 2024-03-28T22:17:58.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.239+0000 7fe9af7fe640 1 --2- 172.21.15.16:0/319674776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b7271111a0 0x55b727125940 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fe9a400bc30 tx=0x7fe9a40107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=324a29c47cd0572f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:58.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.240+0000 7fe995ffb640 1 -- 172.21.15.16:0/319674776 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a400be00 con 0x55b7271111a0 2024-03-28T22:17:58.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.240+0000 7fe995ffb640 1 -- 172.21.15.16:0/319674776 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9a400b810 con 0x55b7271111a0 2024-03-28T22:17:58.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.240+0000 7fe995ffb640 1 -- 172.21.15.16:0/319674776 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a40052a0 con 0x55b7271111a0 2024-03-28T22:17:58.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.240+0000 7fe9b962f740 1 -- 172.21.15.16:0/319674776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b7271111a0 msgr2=0x55b727125940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:58.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.240+0000 7fe9b962f740 1 --2- 172.21.15.16:0/319674776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b7271111a0 0x55b727125940 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fe9a400bc30 tx=0x7fe9a40107a0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.240+0000 7fe9b962f740 1 -- 172.21.15.16:0/319674776 shutdown_connections 2024-03-28T22:17:58.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.240+0000 7fe9b962f740 1 --2- 172.21.15.16:0/319674776 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b727119d80 0x55b727105bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.240+0000 7fe9b962f740 1 --2- 172.21.15.16:0/319674776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b7271111a0 0x55b727125940 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.240+0000 7fe9b962f740 1 --2- 172.21.15.16:0/319674776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b727110880 0x55b7271194f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.240+0000 7fe9b962f740 1 -- 172.21.15.16:0/319674776 >> 172.21.15.16:0/319674776 conn(0x55b72710afb0 msgr2=0x55b7271054b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:58.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.240+0000 7fe9b962f740 1 -- 172.21.15.16:0/319674776 shutdown_connections 2024-03-28T22:17:58.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.240+0000 7fe9b962f740 1 -- 172.21.15.16:0/319674776 wait complete. 2024-03-28T22:17:58.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.240+0000 7fe9b962f740 1 -- start start 2024-03-28T22:17:58.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.241+0000 7fe9b962f740 1 Processor -- start 2024-03-28T22:17:58.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.241+0000 7fe9b962f740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b727110880 0x55b72711db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:58.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.241+0000 7fe9b962f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b7271111a0 0x55b72711e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:58.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.241+0000 7fe9b962f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b727119d80 0x55b72711e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:58.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.241+0000 7fe9b962f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe9b01a9830 con 0x55b727119d80 2024-03-28T22:17:58.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.241+0000 7fe9b962f740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe9b01aaca0 con 0x55b727110880 2024-03-28T22:17:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.241+0000 7fe9b962f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe9b01a9f00 con 0x55b7271111a0 2024-03-28T22:17:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.241+0000 7fe9b962f740 10 client.0 _ll_register_callbacks cb 0x7fe9a400bb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:17:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.241+0000 7fe9b4fe4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b727119d80 0x55b72711e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.241+0000 7fe9b4fe4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b727119d80 0x55b72711e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55222/0 (socket says 172.21.15.16:55222) 2024-03-28T22:17:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.241+0000 7fe9b4fe4640 1 -- 172.21.15.16:0/1156700340 learned_addr learned my addr 172.21.15.16:0/1156700340 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.241+0000 7fe9affff640 1 --2- 172.21.15.16:0/1156700340 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b727110880 0x55b72711db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.241+0000 7fe9af7fe640 1 --2- 172.21.15.16:0/1156700340 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b7271111a0 0x55b72711e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.241+0000 7fe9b4fe4640 1 -- 172.21.15.16:0/1156700340 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b7271111a0 msgr2=0x55b72711e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.241+0000 7fe9b4fe4640 1 --2- 172.21.15.16:0/1156700340 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b7271111a0 0x55b72711e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.241+0000 7fe9b4fe4640 1 -- 172.21.15.16:0/1156700340 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b727110880 msgr2=0x55b72711db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.241+0000 7fe9b4fe4640 1 --2- 172.21.15.16:0/1156700340 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b727110880 0x55b72711db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.241+0000 7fe9b4fe4640 1 -- 172.21.15.16:0/1156700340 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b72718f190 con 0x55b727119d80 2024-03-28T22:17:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.241+0000 7fe9b4fe4640 1 --2- 172.21.15.16:0/1156700340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b727119d80 0x55b72711e610 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fe9a8036000 tx=0x7fe9a802c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69dc02cbaf08a627 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.241+0000 7fe98f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:17:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.242+0000 7fe98f7fe640 1 -- 172.21.15.16:0/1156700340 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a8039070 con 0x55b727119d80 2024-03-28T22:17:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.242+0000 7fe98f7fe640 1 -- 172.21.15.16:0/1156700340 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8527+0+0 (secure 0 0 0) 0x7fe9a80046c0 con 0x55b727119d80 2024-03-28T22:17:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.242+0000 7fe9b962f740 1 -- 172.21.15.16:0/1156700340 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9a400b810 con 0x55b727119d80 2024-03-28T22:17:58.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.242+0000 7fe98f7fe640 1 -- 172.21.15.16:0/1156700340 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe9a8032590 con 0x55b727119d80 2024-03-28T22:17:58.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.242+0000 7fe9b962f740 10 client.17082 fetch_fsmap learned FSMap version 27 2024-03-28T22:17:58.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.242+0000 7fe9b962f740 1 -- 172.21.15.16:0/1156700340 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b7271904c0 con 0x55b727119d80 2024-03-28T22:17:58.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.242+0000 7fe98f7fe640 1 -- 172.21.15.16:0/1156700340 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9a400b810 con 0x55b727119d80 2024-03-28T22:17:58.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.242+0000 7fe98f7fe640 1 -- 172.21.15.16:0/1156700340 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7fe9a8043400 con 0x55b727119d80 2024-03-28T22:17:58.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.242+0000 7fe9b962f740 10 client.17082 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:17:58.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.242+0000 7fe9b962f740 10 client.17082 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:17:58.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.242+0000 7fe9b962f740 20 client.17082 populate_metadata read hostname 'smithi016' 2024-03-28T22:17:58.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.242+0000 7fe9b962f740 1 --2- 172.21.15.16:0/1156700340 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b727190cf0 0x55b727193110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:58.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.242+0000 7fe9b962f740 4 client.17082 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:17:58.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.242+0000 7fe9b962f740 1 -- 172.21.15.16:0/1156700340 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b727077a40 con 0x55b727190cf0 2024-03-28T22:17:58.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.242+0000 7fe9affff640 1 --2- 172.21.15.16:0/1156700340 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b727190cf0 0x55b727193110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:58.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.243+0000 7fe9affff640 1 --2- 172.21.15.16:0/1156700340 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b727190cf0 0x55b727193110 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7996455b0aef6244 server_cookie=18e87e03abb0d45c in_seq=0 out_seq=0 2024-03-28T22:17:58.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.243+0000 7fe98f7fe640 10 client.17082 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:17:58.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.243+0000 7fe98f7fe640 1 -- 172.21.15.16:0/1156700340 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b727077a40 con 0x55b727190cf0 2024-03-28T22:17:58.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.243+0000 7fe98f7fe640 10 client.17082 handle_command_reply: tid=0 2024-03-28T22:17:58.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.243+0000 7fe9b962f740 1 client.17082 shutdown 2024-03-28T22:17:58.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.243+0000 7fe9b962f740 1 -- 172.21.15.16:0/1156700340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b727119d80 msgr2=0x55b72711e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:58.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.243+0000 7fe9b962f740 1 --2- 172.21.15.16:0/1156700340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b727119d80 0x55b72711e610 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fe9a8036000 tx=0x7fe9a802c0f0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.243+0000 7fe9b962f740 1 -- 172.21.15.16:0/1156700340 shutdown_connections 2024-03-28T22:17:58.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.243+0000 7fe9b962f740 1 --2- 172.21.15.16:0/1156700340 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55b727190cf0 0x55b727193110 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.243+0000 7fe9b962f740 1 --2- 172.21.15.16:0/1156700340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b727119d80 0x55b72711e610 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.243+0000 7fe9b962f740 1 --2- 172.21.15.16:0/1156700340 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b7271111a0 0x55b72711e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.243+0000 7fe98f7fe640 0 client.17082 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:17:58.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.243+0000 7fe9b962f740 1 --2- 172.21.15.16:0/1156700340 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b727110880 0x55b72711db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.243+0000 7fe9b962f740 1 -- 172.21.15.16:0/1156700340 >> 172.21.15.16:0/1156700340 conn(0x55b72710afb0 msgr2=0x55b727116d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:58.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.243+0000 7fe9b962f740 1 -- 172.21.15.16:0/1156700340 shutdown_connections 2024-03-28T22:17:58.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.243+0000 7fe9b962f740 1 -- 172.21.15.16:0/1156700340 wait complete. 2024-03-28T22:17:58.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.243+0000 7fe9b962f740 20 client.17082 trim_cache size 0 max 16384 2024-03-28T22:17:58.256 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:17:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.244+0000 7fe9b6a47640 1 -- 172.21.15.16:0/2097108852 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fe9980a1ef0 msgr2=0x7fe9980a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.244+0000 7fe9b6a47640 1 --2- 172.21.15.16:0/2097108852 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fe9980a1ef0 0x7fe9980a43b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe9a8037040 tx=0x7fe9a80095c0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.244+0000 7fe9b6a47640 1 -- 172.21.15.16:0/2097108852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9b01005e0 msgr2=0x7fe9b0197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.244+0000 7fe9b6a47640 1 --2- 172.21.15.16:0/2097108852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9b01005e0 0x7fe9b0197f10 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fe9a000e9c0 tx=0x7fe9a000ee80 comp rx=0 tx=0).stop 2024-03-28T22:17:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.244+0000 7fe9b6a47640 1 -- 172.21.15.16:0/2097108852 shutdown_connections 2024-03-28T22:17:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.244+0000 7fe9b6a47640 1 --2- 172.21.15.16:0/2097108852 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fe9980a1ef0 0x7fe9980a43b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.244+0000 7fe9b6a47640 1 --2- 172.21.15.16:0/2097108852 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe9b0106680 0x7fe9b019c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.244+0000 7fe9b6a47640 1 --2- 172.21.15.16:0/2097108852 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9b0100fa0 0x7fe9b0198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.244+0000 7fe9b6a47640 1 --2- 172.21.15.16:0/2097108852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9b01005e0 0x7fe9b0197f10 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.244+0000 7fe9b6a47640 1 -- 172.21.15.16:0/2097108852 >> 172.21.15.16:0/2097108852 conn(0x7fe9b0077770 msgr2=0x7fe9b00ff780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.244+0000 7fe9b6a47640 1 -- 172.21.15.16:0/2097108852 shutdown_connections 2024-03-28T22:17:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:58.244+0000 7fe9b6a47640 1 -- 172.21.15.16:0/2097108852 wait complete. 2024-03-28T22:17:58.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:58 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/108608525' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:17:58.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:58 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/108608525' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:17:59.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:58 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/108608525' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:17:59.261 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:17:59.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:17:59.261 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:17:59.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.379+0000 7f115cd7a640 1 -- 172.21.15.16:0/898504063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11581021e0 msgr2=0x7f1158102640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:59.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.379+0000 7f115cd7a640 1 --2- 172.21.15.16:0/898504063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11581021e0 0x7f1158102640 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f1140009900 tx=0x7f114002bd10 comp rx=0 tx=0).stop 2024-03-28T22:17:59.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.379+0000 7f115cd7a640 1 -- 172.21.15.16:0/898504063 shutdown_connections 2024-03-28T22:17:59.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.379+0000 7f115cd7a640 1 --2- 172.21.15.16:0/898504063 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1158102b80 0x7f115810d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.379+0000 7f115cd7a640 1 --2- 172.21.15.16:0/898504063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11581021e0 0x7f1158102640 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.379+0000 7f115cd7a640 1 --2- 172.21.15.16:0/898504063 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11581009d0 0x7f1158100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.379+0000 7f115cd7a640 1 -- 172.21.15.16:0/898504063 >> 172.21.15.16:0/898504063 conn(0x7f1158077830 msgr2=0x7f1158077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:59.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.379+0000 7f115cd7a640 1 -- 172.21.15.16:0/898504063 shutdown_connections 2024-03-28T22:17:59.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.379+0000 7f115cd7a640 1 -- 172.21.15.16:0/898504063 wait complete. 2024-03-28T22:17:59.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.380+0000 7f115cd7a640 1 Processor -- start 2024-03-28T22:17:59.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.380+0000 7f115cd7a640 1 -- start start 2024-03-28T22:17:59.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.380+0000 7f115cd7a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11581009d0 0x7f11581980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:59.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.380+0000 7f115cd7a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11581021e0 0x7f11581985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:59.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.380+0000 7f115cd7a640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1158102b80 0x7f115819c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:59.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.380+0000 7f115cd7a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1158111c50 con 0x7f11581021e0 2024-03-28T22:17:59.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.380+0000 7f115cd7a640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f1158111ad0 con 0x7f1158102b80 2024-03-28T22:17:59.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.380+0000 7f115cd7a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1158111dd0 con 0x7f11581009d0 2024-03-28T22:17:59.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.380+0000 7f114ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11581021e0 0x7f11581985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:59.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.380+0000 7f114ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11581021e0 0x7f11581985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55234/0 (socket says 172.21.15.16:55234) 2024-03-28T22:17:59.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.380+0000 7f114ffff640 1 -- 172.21.15.16:0/2687610644 learned_addr learned my addr 172.21.15.16:0/2687610644 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:59.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.380+0000 7f1156d9d640 1 --2- 172.21.15.16:0/2687610644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1158102b80 0x7f115819c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:59.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.380+0000 7f115659c640 1 --2- 172.21.15.16:0/2687610644 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11581009d0 0x7f11581980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:59.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.380+0000 7f114ffff640 1 -- 172.21.15.16:0/2687610644 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11581009d0 msgr2=0x7f11581980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:59.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.380+0000 7f114ffff640 1 --2- 172.21.15.16:0/2687610644 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11581009d0 0x7f11581980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.380+0000 7f114ffff640 1 -- 172.21.15.16:0/2687610644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1158102b80 msgr2=0x7f115819c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:59.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.380+0000 7f114ffff640 1 --2- 172.21.15.16:0/2687610644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1158102b80 0x7f115819c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.380+0000 7f114ffff640 1 -- 172.21.15.16:0/2687610644 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f115819d0a0 con 0x7f11581021e0 2024-03-28T22:17:59.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.380+0000 7f115659c640 1 --2- 172.21.15.16:0/2687610644 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11581009d0 0x7f11581980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:17:59.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.381+0000 7f114ffff640 1 --2- 172.21.15.16:0/2687610644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11581021e0 0x7f11581985f0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f1140006010 tx=0x7f11400042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:59.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.381+0000 7f114f7fe640 1 -- 172.21.15.16:0/2687610644 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1140042020 con 0x7f11581021e0 2024-03-28T22:17:59.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.381+0000 7f114f7fe640 1 -- 172.21.15.16:0/2687610644 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11400054e0 con 0x7f11581021e0 2024-03-28T22:17:59.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.381+0000 7f115cd7a640 1 -- 172.21.15.16:0/2687610644 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f115819d330 con 0x7f11581021e0 2024-03-28T22:17:59.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.381+0000 7f114f7fe640 1 -- 172.21.15.16:0/2687610644 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1140003ad0 con 0x7f11581021e0 2024-03-28T22:17:59.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.381+0000 7f115cd7a640 1 -- 172.21.15.16:0/2687610644 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11581a4c20 con 0x7f11581021e0 2024-03-28T22:17:59.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.383+0000 7f114f7fe640 1 -- 172.21.15.16:0/2687610644 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f1140045020 con 0x7f11581021e0 2024-03-28T22:17:59.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.384+0000 7f115fa3a740 1 Processor -- start 2024-03-28T22:17:59.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.384+0000 7f115fa3a740 1 -- start start 2024-03-28T22:17:59.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.384+0000 7f115fa3a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef65a0d9f0 0x55ef65a0ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:59.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.384+0000 7f115fa3a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef65a0e310 0x55ef65a169b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:59.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.384+0000 7f115fa3a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ef65a16ef0 0x55ef65a192e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:59.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.384+0000 7f115fa3a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ef6590e0b0 con 0x55ef65a0e310 2024-03-28T22:17:59.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.384+0000 7f115fa3a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55ef65970ea0 con 0x55ef65a16ef0 2024-03-28T22:17:59.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.384+0000 7f115fa3a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55ef65919c00 con 0x55ef65a0d9f0 2024-03-28T22:17:59.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.384+0000 7f114f7fe640 1 --2- 172.21.15.16:0/2687610644 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f11440a1ef0 0x7f11440a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:59.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.384+0000 7f115659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef65a0e310 0x55ef65a169b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:59.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.384+0000 7f114ffff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ef65a16ef0 0x55ef65a192e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:59.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.384+0000 7f115659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef65a0e310 0x55ef65a169b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55236/0 (socket says 172.21.15.16:55236) 2024-03-28T22:17:59.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.384+0000 7f115659c640 1 -- 172.21.15.16:0/3235418364 learned_addr learned my addr 172.21.15.16:0/3235418364 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:59.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.384+0000 7f1156d9d640 1 --2- 172.21.15.16:0/3235418364 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef65a0d9f0 0x55ef65a0ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:59.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.384+0000 7f1156d9d640 1 --2- 172.21.15.16:0/2687610644 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f11440a1ef0 0x7f11440a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:59.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.384+0000 7f114f7fe640 1 -- 172.21.15.16:0/2687610644 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8527+0+0 (secure 0 0 0) 0x7f11400efe20 con 0x7f11581021e0 2024-03-28T22:17:59.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.385+0000 7f114ffff640 1 -- 172.21.15.16:0/3235418364 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef65a0d9f0 msgr2=0x55ef65a0ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:59.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.385+0000 7f114ffff640 1 --2- 172.21.15.16:0/3235418364 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef65a0d9f0 0x55ef65a0ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.385+0000 7f114ffff640 1 -- 172.21.15.16:0/3235418364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef65a0e310 msgr2=0x55ef65a169b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:59.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.385+0000 7f114ffff640 1 --2- 172.21.15.16:0/3235418364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef65a0e310 0x55ef65a169b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.385+0000 7f114ffff640 1 -- 172.21.15.16:0/3235418364 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef6574d7e0 con 0x55ef65a16ef0 2024-03-28T22:17:59.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.385+0000 7f114ffff640 1 --2- 172.21.15.16:0/3235418364 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ef65a16ef0 0x55ef65a192e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f11400ed680 tx=0x7f114008f990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc91e554aa8a9ec8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:59.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.385+0000 7f1156d9d640 1 --2- 172.21.15.16:0/2687610644 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f11440a1ef0 0x7f11440a43b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1150009fd0 tx=0x7f11500095a0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:59.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.385+0000 7f114cff9640 1 -- 172.21.15.16:0/3235418364 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11400a5e60 con 0x55ef65a16ef0 2024-03-28T22:17:59.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.385+0000 7f114cff9640 1 -- 172.21.15.16:0/3235418364 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11400bd020 con 0x55ef65a16ef0 2024-03-28T22:17:59.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.385+0000 7f114cff9640 1 -- 172.21.15.16:0/3235418364 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11400c0a70 con 0x55ef65a16ef0 2024-03-28T22:17:59.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.386+0000 7f115fa3a740 1 -- 172.21.15.16:0/3235418364 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ef65a16ef0 msgr2=0x55ef65a192e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:59.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.386+0000 7f115fa3a740 1 --2- 172.21.15.16:0/3235418364 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ef65a16ef0 0x55ef65a192e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f11400ed680 tx=0x7f114008f990 comp rx=0 tx=0).stop 2024-03-28T22:17:59.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.386+0000 7f115fa3a740 1 -- 172.21.15.16:0/3235418364 shutdown_connections 2024-03-28T22:17:59.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.386+0000 7f115fa3a740 1 --2- 172.21.15.16:0/3235418364 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ef65a16ef0 0x55ef65a192e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.386+0000 7f115fa3a740 1 --2- 172.21.15.16:0/3235418364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef65a0e310 0x55ef65a169b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.386+0000 7f115fa3a740 1 --2- 172.21.15.16:0/3235418364 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef65a0d9f0 0x55ef65a0ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.386+0000 7f115fa3a740 1 -- 172.21.15.16:0/3235418364 >> 172.21.15.16:0/3235418364 conn(0x55ef65a08120 msgr2=0x55ef65a08500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:59.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.386+0000 7f115fa3a740 1 -- 172.21.15.16:0/3235418364 shutdown_connections 2024-03-28T22:17:59.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.386+0000 7f115fa3a740 1 -- 172.21.15.16:0/3235418364 wait complete. 2024-03-28T22:17:59.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.386+0000 7f115fa3a740 1 -- start start 2024-03-28T22:17:59.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.386+0000 7f115fa3a740 1 Processor -- start 2024-03-28T22:17:59.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.387+0000 7f115fa3a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef65a0d9f0 0x55ef65a13ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:59.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.387+0000 7f115fa3a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef65a0e310 0x55ef65a14430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:59.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.387+0000 7f115fa3a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ef65a16ef0 0x55ef65a14970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:59.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.387+0000 7f115fa3a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ef6590f3d0 con 0x55ef65a0d9f0 2024-03-28T22:17:59.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.387+0000 7f115fa3a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55ef6596cf40 con 0x55ef65a16ef0 2024-03-28T22:17:59.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.387+0000 7f115fa3a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55ef65a19aa0 con 0x55ef65a0e310 2024-03-28T22:17:59.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.387+0000 7f115fa3a740 10 client.0 _ll_register_callbacks cb 0x55ef6592cb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:17:59.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.387+0000 7f115659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef65a0d9f0 0x55ef65a13ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:59.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.387+0000 7f115659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef65a0d9f0 0x55ef65a13ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55248/0 (socket says 172.21.15.16:55248) 2024-03-28T22:17:59.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.387+0000 7f115659c640 1 -- 172.21.15.16:0/3003776076 learned_addr learned my addr 172.21.15.16:0/3003776076 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:59.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.387+0000 7f1156d9d640 1 --2- 172.21.15.16:0/3003776076 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ef65a16ef0 0x55ef65a14970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:59.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.387+0000 7f114ffff640 1 --2- 172.21.15.16:0/3003776076 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef65a0e310 0x55ef65a14430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:59.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.387+0000 7f115659c640 1 -- 172.21.15.16:0/3003776076 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef65a0e310 msgr2=0x55ef65a14430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:59.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.387+0000 7f115659c640 1 --2- 172.21.15.16:0/3003776076 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef65a0e310 0x55ef65a14430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.387+0000 7f115659c640 1 -- 172.21.15.16:0/3003776076 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ef65a16ef0 msgr2=0x55ef65a14970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:59.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.387+0000 7f115659c640 1 --2- 172.21.15.16:0/3003776076 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ef65a16ef0 0x55ef65a14970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.387+0000 7f115659c640 1 -- 172.21.15.16:0/3003776076 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef657adf70 con 0x55ef65a0d9f0 2024-03-28T22:17:59.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.387+0000 7f1156d9d640 1 --2- 172.21.15.16:0/3003776076 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ef65a16ef0 0x55ef65a14970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:17:59.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.387+0000 7f115659c640 1 --2- 172.21.15.16:0/3003776076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef65a0d9f0 0x55ef65a13ef0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f11480127a0 tx=0x7f1148012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae0d454b9a4e6a0c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:59.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.387+0000 7f113dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:17:59.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.387+0000 7f113dffb640 1 -- 172.21.15.16:0/3003776076 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1148004050 con 0x55ef65a0d9f0 2024-03-28T22:17:59.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.387+0000 7f113dffb640 1 -- 172.21.15.16:0/3003776076 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8527+0+0 (secure 0 0 0) 0x7f11480041f0 con 0x55ef65a0d9f0 2024-03-28T22:17:59.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.387+0000 7f115fa3a740 1 -- 172.21.15.16:0/3003776076 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11400bd020 con 0x55ef65a0d9f0 2024-03-28T22:17:59.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.388+0000 7f113dffb640 1 -- 172.21.15.16:0/3003776076 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f11480043f0 con 0x55ef65a0d9f0 2024-03-28T22:17:59.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.388+0000 7f115fa3a740 10 client.17094 fetch_fsmap learned FSMap version 27 2024-03-28T22:17:59.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.388+0000 7f115fa3a740 1 -- 172.21.15.16:0/3003776076 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ef65645170 con 0x55ef65a0d9f0 2024-03-28T22:17:59.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.388+0000 7f113dffb640 1 -- 172.21.15.16:0/3003776076 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11400bd020 con 0x55ef65a0d9f0 2024-03-28T22:17:59.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.388+0000 7f113dffb640 1 -- 172.21.15.16:0/3003776076 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f1148025480 con 0x55ef65a0d9f0 2024-03-28T22:17:59.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.388+0000 7f115fa3a740 10 client.17094 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:17:59.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.388+0000 7f115fa3a740 10 client.17094 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:17:59.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.388+0000 7f115fa3a740 20 client.17094 populate_metadata read hostname 'smithi016' 2024-03-28T22:17:59.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.388+0000 7f115fa3a740 1 --2- 172.21.15.16:0/3003776076 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55ef65a26ba0 0x55ef65a29030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:59.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.388+0000 7f115fa3a740 4 client.17094 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:17:59.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.388+0000 7f115fa3a740 1 -- 172.21.15.16:0/3003776076 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef6597e3e0 con 0x55ef65a26ba0 2024-03-28T22:17:59.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.388+0000 7f114ffff640 1 --2- 172.21.15.16:0/3003776076 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55ef65a26ba0 0x55ef65a29030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:59.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.389+0000 7f113dffb640 10 client.17094 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:17:59.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.389+0000 7f114ffff640 1 --2- 172.21.15.16:0/3003776076 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55ef65a26ba0 0x55ef65a29030 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb6b069a91549b0 server_cookie=29c77459a4321c86 in_seq=0 out_seq=0 2024-03-28T22:17:59.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.389+0000 7f113dffb640 1 -- 172.21.15.16:0/3003776076 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ef6597e3e0 con 0x55ef65a26ba0 2024-03-28T22:17:59.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.389+0000 7f113dffb640 10 client.17094 handle_command_reply: tid=0 2024-03-28T22:17:59.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.389+0000 7f115fa3a740 1 client.17094 shutdown 2024-03-28T22:17:59.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.389+0000 7f115fa3a740 1 -- 172.21.15.16:0/3003776076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef65a0d9f0 msgr2=0x55ef65a13ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:59.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.389+0000 7f115fa3a740 1 --2- 172.21.15.16:0/3003776076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef65a0d9f0 0x55ef65a13ef0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f11480127a0 tx=0x7f1148012ba0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.389+0000 7f115fa3a740 1 -- 172.21.15.16:0/3003776076 shutdown_connections 2024-03-28T22:17:59.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.389+0000 7f115fa3a740 1 --2- 172.21.15.16:0/3003776076 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55ef65a26ba0 0x55ef65a29030 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.389+0000 7f115fa3a740 1 --2- 172.21.15.16:0/3003776076 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ef65a16ef0 0x55ef65a14970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.389+0000 7f115fa3a740 1 --2- 172.21.15.16:0/3003776076 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef65a0e310 0x55ef65a14430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.389+0000 7f113dffb640 0 client.17094 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:17:59.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.389+0000 7f115fa3a740 1 --2- 172.21.15.16:0/3003776076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef65a0d9f0 0x55ef65a13ef0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.389+0000 7f115fa3a740 1 -- 172.21.15.16:0/3003776076 >> 172.21.15.16:0/3003776076 conn(0x55ef65a08120 msgr2=0x55ef65a09e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:59.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.390+0000 7f115fa3a740 1 -- 172.21.15.16:0/3003776076 shutdown_connections 2024-03-28T22:17:59.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.390+0000 7f115fa3a740 1 -- 172.21.15.16:0/3003776076 wait complete. 2024-03-28T22:17:59.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.390+0000 7f115fa3a740 20 client.17094 trim_cache size 0 max 16384 2024-03-28T22:17:59.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.407+0000 7f115fa3a740 1 Processor -- start 2024-03-28T22:17:59.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.407+0000 7f115fa3a740 1 -- start start 2024-03-28T22:17:59.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.407+0000 7f115fa3a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ef65a0d9f0 0x55ef65a226c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:59.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.407+0000 7f115fa3a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef65a0e310 0x55ef65a2e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:59.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.407+0000 7f115fa3a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef65a16ef0 0x55ef65a1ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:59.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.407+0000 7f115fa3a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f11581afb60 con 0x55ef65a0e310 2024-03-28T22:17:59.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.407+0000 7f115fa3a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f11581ad430 con 0x55ef65a0d9f0 2024-03-28T22:17:59.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.407+0000 7f115fa3a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f11581ab2a0 con 0x55ef65a16ef0 2024-03-28T22:17:59.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.408+0000 7f114ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef65a0e310 0x55ef65a2e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:59.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.408+0000 7f114ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef65a0e310 0x55ef65a2e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55254/0 (socket says 172.21.15.16:55254) 2024-03-28T22:17:59.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.408+0000 7f114ffff640 1 -- 172.21.15.16:0/2569468424 learned_addr learned my addr 172.21.15.16:0/2569468424 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:59.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.408+0000 7f115659c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ef65a0d9f0 0x55ef65a226c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:59.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.408+0000 7f1156d9d640 1 --2- 172.21.15.16:0/2569468424 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef65a16ef0 0x55ef65a1ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:59.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.408+0000 7f114ffff640 1 -- 172.21.15.16:0/2569468424 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef65a16ef0 msgr2=0x55ef65a1ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:59.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.408+0000 7f114ffff640 1 --2- 172.21.15.16:0/2569468424 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef65a16ef0 0x55ef65a1ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.408+0000 7f114ffff640 1 -- 172.21.15.16:0/2569468424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ef65a0d9f0 msgr2=0x55ef65a226c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:59.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.408+0000 7f114ffff640 1 --2- 172.21.15.16:0/2569468424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ef65a0d9f0 0x55ef65a226c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.408+0000 7f114ffff640 1 -- 172.21.15.16:0/2569468424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef65a0e140 con 0x55ef65a0e310 2024-03-28T22:17:59.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.408+0000 7f115659c640 1 --2- 172.21.15.16:0/2569468424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ef65a0d9f0 0x55ef65a226c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:17:59.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.408+0000 7f114ffff640 1 --2- 172.21.15.16:0/2569468424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef65a0e310 0x55ef65a2e900 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f114002c8d0 tx=0x7f11400e6fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=575af11049a68079 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:59.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.408+0000 7f113ffff640 1 -- 172.21.15.16:0/2569468424 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11400a5e60 con 0x55ef65a0e310 2024-03-28T22:17:59.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.408+0000 7f113ffff640 1 -- 172.21.15.16:0/2569468424 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11400bd020 con 0x55ef65a0e310 2024-03-28T22:17:59.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.408+0000 7f113ffff640 1 -- 172.21.15.16:0/2569468424 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11400bacb0 con 0x55ef65a0e310 2024-03-28T22:17:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.409+0000 7f115fa3a740 1 -- 172.21.15.16:0/2569468424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef65a0e310 msgr2=0x55ef65a2e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.409+0000 7f115fa3a740 1 --2- 172.21.15.16:0/2569468424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef65a0e310 0x55ef65a2e900 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f114002c8d0 tx=0x7f11400e6fe0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.409+0000 7f115fa3a740 1 -- 172.21.15.16:0/2569468424 shutdown_connections 2024-03-28T22:17:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.409+0000 7f115fa3a740 1 --2- 172.21.15.16:0/2569468424 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef65a16ef0 0x55ef65a1ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.409+0000 7f115fa3a740 1 --2- 172.21.15.16:0/2569468424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef65a0e310 0x55ef65a2e900 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.409+0000 7f115fa3a740 1 --2- 172.21.15.16:0/2569468424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ef65a0d9f0 0x55ef65a226c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.409+0000 7f115fa3a740 1 -- 172.21.15.16:0/2569468424 >> 172.21.15.16:0/2569468424 conn(0x55ef65a08120 msgr2=0x55ef65a13eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.409+0000 7f115fa3a740 1 -- 172.21.15.16:0/2569468424 shutdown_connections 2024-03-28T22:17:59.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.409+0000 7f115fa3a740 1 -- 172.21.15.16:0/2569468424 wait complete. 2024-03-28T22:17:59.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.409+0000 7f115fa3a740 1 -- start start 2024-03-28T22:17:59.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.409+0000 7f115fa3a740 1 Processor -- start 2024-03-28T22:17:59.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.409+0000 7f115fa3a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ef65a0d9f0 0x55ef65a1ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:59.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.409+0000 7f115fa3a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef65a0e310 0x55ef65a1afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:59.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.409+0000 7f115fa3a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef65a16ef0 0x55ef65a1b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:59.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.409+0000 7f115fa3a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f11581ab8f0 con 0x55ef65a16ef0 2024-03-28T22:17:59.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.409+0000 7f115fa3a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f11581acd60 con 0x55ef65a0d9f0 2024-03-28T22:17:59.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.409+0000 7f115fa3a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f11581abfc0 con 0x55ef65a0e310 2024-03-28T22:17:59.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.410+0000 7f115fa3a740 10 client.0 _ll_register_callbacks cb 0x7f1148009ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:17:59.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.410+0000 7f1156d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef65a16ef0 0x55ef65a1b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:59.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.410+0000 7f115659c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ef65a0d9f0 0x55ef65a1ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:59.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.410+0000 7f1156d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef65a16ef0 0x55ef65a1b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55266/0 (socket says 172.21.15.16:55266) 2024-03-28T22:17:59.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.410+0000 7f1156d9d640 1 -- 172.21.15.16:0/160686975 learned_addr learned my addr 172.21.15.16:0/160686975 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:59.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.410+0000 7f114ffff640 1 --2- 172.21.15.16:0/160686975 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef65a0e310 0x55ef65a1afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:59.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.410+0000 7f1156d9d640 1 -- 172.21.15.16:0/160686975 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef65a0e310 msgr2=0x55ef65a1afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:59.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.410+0000 7f1156d9d640 1 --2- 172.21.15.16:0/160686975 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef65a0e310 0x55ef65a1afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.410+0000 7f1156d9d640 1 -- 172.21.15.16:0/160686975 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ef65a0d9f0 msgr2=0x55ef65a1ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:59.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.410+0000 7f1156d9d640 1 --2- 172.21.15.16:0/160686975 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ef65a0d9f0 0x55ef65a1ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.410+0000 7f1156d9d640 1 -- 172.21.15.16:0/160686975 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef65a1ba50 con 0x55ef65a16ef0 2024-03-28T22:17:59.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.410+0000 7f1156d9d640 1 --2- 172.21.15.16:0/160686975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef65a16ef0 0x55ef65a1b510 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f115001a6a0 tx=0x7f1150004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b36dccfa0da3f381 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:59.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.410+0000 7f113dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:17:59.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.410+0000 7f113dffb640 1 -- 172.21.15.16:0/160686975 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1150014040 con 0x55ef65a16ef0 2024-03-28T22:17:59.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.410+0000 7f113dffb640 1 -- 172.21.15.16:0/160686975 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8527+0+0 (secure 0 0 0) 0x7f11500267e0 con 0x55ef65a16ef0 2024-03-28T22:17:59.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.410+0000 7f115fa3a740 1 -- 172.21.15.16:0/160686975 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11400bd020 con 0x55ef65a16ef0 2024-03-28T22:17:59.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.410+0000 7f113dffb640 1 -- 172.21.15.16:0/160686975 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f115000be10 con 0x55ef65a16ef0 2024-03-28T22:17:59.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.410+0000 7f115fa3a740 10 client.17100 fetch_fsmap learned FSMap version 27 2024-03-28T22:17:59.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.410+0000 7f115fa3a740 1 -- 172.21.15.16:0/160686975 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ef65a8dc40 con 0x55ef65a16ef0 2024-03-28T22:17:59.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.411+0000 7f113dffb640 1 -- 172.21.15.16:0/160686975 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11400bd020 con 0x55ef65a16ef0 2024-03-28T22:17:59.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.411+0000 7f113dffb640 1 -- 172.21.15.16:0/160686975 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f1150027b10 con 0x55ef65a16ef0 2024-03-28T22:17:59.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.411+0000 7f115fa3a740 10 client.17100 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:17:59.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.411+0000 7f115fa3a740 10 client.17100 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:17:59.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.411+0000 7f115fa3a740 20 client.17100 populate_metadata read hostname 'smithi016' 2024-03-28T22:17:59.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.411+0000 7f115fa3a740 1 --2- 172.21.15.16:0/160686975 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55ef65a8e470 0x55ef65a90890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:59.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.411+0000 7f115fa3a740 4 client.17100 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:17:59.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.411+0000 7f115fa3a740 1 -- 172.21.15.16:0/160686975 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ef65978730 con 0x55ef65a8e470 2024-03-28T22:17:59.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.411+0000 7f115659c640 1 --2- 172.21.15.16:0/160686975 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55ef65a8e470 0x55ef65a90890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:59.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.411+0000 7f113dffb640 10 client.17100 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:17:59.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.411+0000 7f115659c640 1 --2- 172.21.15.16:0/160686975 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55ef65a8e470 0x55ef65a90890 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72e4d452cecd9717 server_cookie=310a7784082fcdc2 in_seq=0 out_seq=0 2024-03-28T22:17:59.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.412+0000 7f113dffb640 1 -- 172.21.15.16:0/160686975 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ef65978730 con 0x55ef65a8e470 2024-03-28T22:17:59.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.412+0000 7f113dffb640 10 client.17100 handle_command_reply: tid=0 2024-03-28T22:17:59.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.412+0000 7f115fa3a740 1 client.17100 shutdown 2024-03-28T22:17:59.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.412+0000 7f115fa3a740 1 -- 172.21.15.16:0/160686975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef65a16ef0 msgr2=0x55ef65a1b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:59.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.412+0000 7f115fa3a740 1 --2- 172.21.15.16:0/160686975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef65a16ef0 0x55ef65a1b510 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f115001a6a0 tx=0x7f1150004b50 comp rx=0 tx=0).stop 2024-03-28T22:17:59.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.413+0000 7f115fa3a740 1 -- 172.21.15.16:0/160686975 shutdown_connections 2024-03-28T22:17:59.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.413+0000 7f115fa3a740 1 --2- 172.21.15.16:0/160686975 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55ef65a8e470 0x55ef65a90890 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.413+0000 7f115fa3a740 1 --2- 172.21.15.16:0/160686975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef65a16ef0 0x55ef65a1b510 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.413+0000 7f113dffb640 0 client.17100 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:17:59.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.413+0000 7f115fa3a740 1 --2- 172.21.15.16:0/160686975 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef65a0e310 0x55ef65a1afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.413+0000 7f115fa3a740 1 --2- 172.21.15.16:0/160686975 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ef65a0d9f0 0x55ef65a1ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.413+0000 7f115fa3a740 1 -- 172.21.15.16:0/160686975 >> 172.21.15.16:0/160686975 conn(0x55ef65a08120 msgr2=0x55ef65a14810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:59.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.413+0000 7f115fa3a740 1 -- 172.21.15.16:0/160686975 shutdown_connections 2024-03-28T22:17:59.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.413+0000 7f115fa3a740 1 -- 172.21.15.16:0/160686975 wait complete. 2024-03-28T22:17:59.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.413+0000 7f115fa3a740 20 client.17100 trim_cache size 0 max 16384 2024-03-28T22:17:59.428 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:17:59.428 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-03-28T22:17:59.428 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "741e9a7c-55be-48ee-bc62-2e700a17d40e", 2024-03-28T22:17:59.428 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-03-28T22:17:59.428 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:17:59.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.413+0000 7f115cd7a640 1 -- 172.21.15.16:0/2687610644 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f11440a1ef0 msgr2=0x7f11440a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:59.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.413+0000 7f115cd7a640 1 --2- 172.21.15.16:0/2687610644 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f11440a1ef0 0x7f11440a43b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1150009fd0 tx=0x7f11500095a0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.413+0000 7f115cd7a640 1 -- 172.21.15.16:0/2687610644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11581021e0 msgr2=0x7f11581985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:59.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.413+0000 7f115cd7a640 1 --2- 172.21.15.16:0/2687610644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11581021e0 0x7f11581985f0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f1140006010 tx=0x7f11400042f0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.413+0000 7f115cd7a640 1 -- 172.21.15.16:0/2687610644 shutdown_connections 2024-03-28T22:17:59.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.413+0000 7f115cd7a640 1 --2- 172.21.15.16:0/2687610644 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f11440a1ef0 0x7f11440a43b0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.413+0000 7f115cd7a640 1 --2- 172.21.15.16:0/2687610644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1158102b80 0x7f115819c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.413+0000 7f115cd7a640 1 --2- 172.21.15.16:0/2687610644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11581021e0 0x7f11581985f0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.413+0000 7f115cd7a640 1 --2- 172.21.15.16:0/2687610644 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11581009d0 0x7f11581980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.413+0000 7f115cd7a640 1 -- 172.21.15.16:0/2687610644 >> 172.21.15.16:0/2687610644 conn(0x7f1158077830 msgr2=0x7f1158104360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:59.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.413+0000 7f115cd7a640 1 -- 172.21.15.16:0/2687610644 shutdown_connections 2024-03-28T22:17:59.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.413+0000 7f115cd7a640 1 -- 172.21.15.16:0/2687610644 wait complete. 2024-03-28T22:17:59.431 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:17:59.432 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:17:59.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.537+0000 7f011a756640 1 -- 172.21.15.16:0/1388446117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0114106690 msgr2=0x7f0114106a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:59.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.537+0000 7f011a756640 1 --2- 172.21.15.16:0/1388446117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0114106690 0x7f0114106a70 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f0104009930 tx=0x7f010402bc30 comp rx=0 tx=0).stop 2024-03-28T22:17:59.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.537+0000 7f011a756640 1 -- 172.21.15.16:0/1388446117 shutdown_connections 2024-03-28T22:17:59.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.537+0000 7f011a756640 1 --2- 172.21.15.16:0/1388446117 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0114100fb0 0x7f011410d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.537+0000 7f011a756640 1 --2- 172.21.15.16:0/1388446117 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01141005f0 0x7f0114100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.537+0000 7f011a756640 1 --2- 172.21.15.16:0/1388446117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0114106690 0x7f0114106a70 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.537+0000 7f011a756640 1 -- 172.21.15.16:0/1388446117 >> 172.21.15.16:0/1388446117 conn(0x7f0114077800 msgr2=0x7f0114077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:59.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.538+0000 7f011a756640 1 -- 172.21.15.16:0/1388446117 shutdown_connections 2024-03-28T22:17:59.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.538+0000 7f011a756640 1 -- 172.21.15.16:0/1388446117 wait complete. 2024-03-28T22:17:59.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.538+0000 7f011a756640 1 Processor -- start 2024-03-28T22:17:59.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.538+0000 7f011a756640 1 -- start start 2024-03-28T22:17:59.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.538+0000 7f011a756640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01141005f0 0x7f0114198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:59.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.538+0000 7f011a756640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0114100fb0 0x7f01141985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:59.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.538+0000 7f011a756640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0114106690 0x7f011419c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:59.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.538+0000 7f011a756640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0114110a60 con 0x7f0114100fb0 2024-03-28T22:17:59.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.538+0000 7f011a756640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f01141108e0 con 0x7f01141005f0 2024-03-28T22:17:59.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.538+0000 7f011a756640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0114110be0 con 0x7f0114106690 2024-03-28T22:17:59.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.539+0000 7f010bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0114100fb0 0x7f01141985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:59.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.539+0000 7f010bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0114100fb0 0x7f01141985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55294/0 (socket says 172.21.15.16:55294) 2024-03-28T22:17:59.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.539+0000 7f010bfff640 1 -- 172.21.15.16:0/112948767 learned_addr learned my addr 172.21.15.16:0/112948767 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:59.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.539+0000 7f0113fff640 1 --2- 172.21.15.16:0/112948767 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01141005f0 0x7f0114198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:59.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.539+0000 7f0118cf3640 1 --2- 172.21.15.16:0/112948767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0114106690 0x7f011419c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:59.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.539+0000 7f010bfff640 1 -- 172.21.15.16:0/112948767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0114106690 msgr2=0x7f011419c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:59.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.539+0000 7f010bfff640 1 --2- 172.21.15.16:0/112948767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0114106690 0x7f011419c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.539+0000 7f010bfff640 1 -- 172.21.15.16:0/112948767 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01141005f0 msgr2=0x7f0114198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:59.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.539+0000 7f010bfff640 1 --2- 172.21.15.16:0/112948767 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01141005f0 0x7f0114198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.539+0000 7f010bfff640 1 -- 172.21.15.16:0/112948767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f011419d060 con 0x7f0114100fb0 2024-03-28T22:17:59.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.539+0000 7f0118cf3640 1 --2- 172.21.15.16:0/112948767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0114106690 0x7f011419c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:17:59.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.539+0000 7f010bfff640 1 --2- 172.21.15.16:0/112948767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0114100fb0 0x7f01141985b0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f00fc00e9f0 tx=0x7f00fc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:59.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.539+0000 7f0111ffb640 1 -- 172.21.15.16:0/112948767 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00fc00cd80 con 0x7f0114100fb0 2024-03-28T22:17:59.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.539+0000 7f0111ffb640 1 -- 172.21.15.16:0/112948767 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00fc010910 con 0x7f0114100fb0 2024-03-28T22:17:59.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.539+0000 7f0111ffb640 1 -- 172.21.15.16:0/112948767 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00fc018990 con 0x7f0114100fb0 2024-03-28T22:17:59.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.539+0000 7f011a756640 1 -- 172.21.15.16:0/112948767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f011419d350 con 0x7f0114100fb0 2024-03-28T22:17:59.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.539+0000 7f011a756640 1 -- 172.21.15.16:0/112948767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01141a4be0 con 0x7f0114100fb0 2024-03-28T22:17:59.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.541+0000 7f0111ffb640 1 -- 172.21.15.16:0/112948767 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f00fc010ac0 con 0x7f0114100fb0 2024-03-28T22:17:59.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.542+0000 7f011d415740 1 Processor -- start 2024-03-28T22:17:59.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.542+0000 7f011d415740 1 -- start start 2024-03-28T22:17:59.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.542+0000 7f011d415740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590eb9b3290 0x5590eb9b3670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:59.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.542+0000 7f0111ffb640 1 --2- 172.21.15.16:0/112948767 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f01000a1ef0 0x7f01000a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:59.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.542+0000 7f011d415740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590eb9b3bb0 0x5590eb9bc250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:59.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.542+0000 7f011d415740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590eb9bc790 0x5590eb9beb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:59.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.542+0000 7f011d415740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5590eb917180 con 0x5590eb9bc790 2024-03-28T22:17:59.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.542+0000 7f011d415740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5590eb8bfb70 con 0x5590eb9b3bb0 2024-03-28T22:17:59.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.542+0000 7f011d415740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5590eb913220 con 0x5590eb9b3290 2024-03-28T22:17:59.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.542+0000 7f0118cf3640 1 --2- 172.21.15.16:0/112948767 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f01000a1ef0 0x7f01000a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:59.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.542+0000 7f0111ffb640 1 -- 172.21.15.16:0/112948767 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8527+0+0 (secure 0 0 0) 0x7f00fc0ce540 con 0x7f0114100fb0 2024-03-28T22:17:59.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.542+0000 7f010bfff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590eb9b3bb0 0x5590eb9bc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:59.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.542+0000 7f0113fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590eb9b3290 0x5590eb9b3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:59.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.542+0000 7f0118cf3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590eb9bc790 0x5590eb9beb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:59.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.542+0000 7f010bfff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590eb9b3bb0 0x5590eb9bc250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:39774/0 (socket says 172.21.15.16:39774) 2024-03-28T22:17:59.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.542+0000 7f010bfff640 1 -- 172.21.15.16:0/1302103531 learned_addr learned my addr 172.21.15.16:0/1302103531 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:59.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.542+0000 7f0113fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590eb9b3290 0x5590eb9b3670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.16:41842/0 (socket says 172.21.15.16:41842) 2024-03-28T22:17:59.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.543+0000 7f010bfff640 1 -- 172.21.15.16:0/1302103531 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590eb9b3290 msgr2=0x5590eb9b3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:59.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.543+0000 7f010bfff640 1 --2- 172.21.15.16:0/1302103531 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590eb9b3290 0x5590eb9b3670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.543+0000 7f010bfff640 1 -- 172.21.15.16:0/1302103531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590eb9bc790 msgr2=0x5590eb9beb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:59.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.543+0000 7f010bfff640 1 --2- 172.21.15.16:0/1302103531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590eb9bc790 0x5590eb9beb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.543+0000 7f010bfff640 1 -- 172.21.15.16:0/1302103531 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590eb8d37c0 con 0x5590eb9b3bb0 2024-03-28T22:17:59.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.543+0000 7f0118cf3640 1 --2- 172.21.15.16:0/1302103531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590eb9bc790 0x5590eb9beb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:17:59.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.543+0000 7f0113fff640 1 --2- 172.21.15.16:0/1302103531 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590eb9b3290 0x5590eb9b3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:17:59.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.543+0000 7f0118cf3640 1 --2- 172.21.15.16:0/112948767 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f01000a1ef0 0x7f01000a43b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f010c009810 tx=0x7f010c009e80 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:59.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.543+0000 7f010bfff640 1 --2- 172.21.15.16:0/1302103531 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590eb9b3bb0 0x5590eb9bc250 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f00fc0cbcf0 tx=0x7f00fc01f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f126afbb011f468d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:59.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.543+0000 7f010a7fc640 1 -- 172.21.15.16:0/1302103531 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00fc0a4590 con 0x5590eb9b3bb0 2024-03-28T22:17:59.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.543+0000 7f010a7fc640 1 -- 172.21.15.16:0/1302103531 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00fc09af30 con 0x5590eb9b3bb0 2024-03-28T22:17:59.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.543+0000 7f010a7fc640 1 -- 172.21.15.16:0/1302103531 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00fc0975a0 con 0x5590eb9b3bb0 2024-03-28T22:17:59.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.544+0000 7f011d415740 1 -- 172.21.15.16:0/1302103531 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590eb9b3bb0 msgr2=0x5590eb9bc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:59.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.544+0000 7f011d415740 1 --2- 172.21.15.16:0/1302103531 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590eb9b3bb0 0x5590eb9bc250 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f00fc0cbcf0 tx=0x7f00fc01f000 comp rx=0 tx=0).stop 2024-03-28T22:17:59.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.544+0000 7f011d415740 1 -- 172.21.15.16:0/1302103531 shutdown_connections 2024-03-28T22:17:59.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.544+0000 7f011d415740 1 --2- 172.21.15.16:0/1302103531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590eb9bc790 0x5590eb9beb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.544+0000 7f011d415740 1 --2- 172.21.15.16:0/1302103531 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590eb9b3bb0 0x5590eb9bc250 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.544+0000 7f011d415740 1 --2- 172.21.15.16:0/1302103531 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590eb9b3290 0x5590eb9b3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.544+0000 7f011d415740 1 -- 172.21.15.16:0/1302103531 >> 172.21.15.16:0/1302103531 conn(0x5590eb9ad9c0 msgr2=0x5590eb9adda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:59.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.544+0000 7f011d415740 1 -- 172.21.15.16:0/1302103531 shutdown_connections 2024-03-28T22:17:59.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.544+0000 7f011d415740 1 -- 172.21.15.16:0/1302103531 wait complete. 2024-03-28T22:17:59.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.544+0000 7f011d415740 1 -- start start 2024-03-28T22:17:59.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.545+0000 7f011d415740 1 Processor -- start 2024-03-28T22:17:59.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.545+0000 7f011d415740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590eb9b3290 0x5590eb9b9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:59.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.545+0000 7f011d415740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590eb9b3bb0 0x5590eb9b9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:59.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.545+0000 7f011d415740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590eb9bc790 0x5590eb9ba1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:59.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.545+0000 7f011d415740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5590eb9bf580 con 0x5590eb9b3bb0 2024-03-28T22:17:59.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.545+0000 7f011d415740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5590eb8b54a0 con 0x5590eb9bc790 2024-03-28T22:17:59.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.545+0000 7f011d415740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5590eb9bf700 con 0x5590eb9b3290 2024-03-28T22:17:59.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.545+0000 7f011d415740 10 client.0 _ll_register_callbacks cb 0x5590eb8b8d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:17:59.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.545+0000 7f010bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590eb9b3bb0 0x5590eb9b9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:59.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.545+0000 7f010bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590eb9b3bb0 0x5590eb9b9ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55312/0 (socket says 172.21.15.16:55312) 2024-03-28T22:17:59.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.545+0000 7f010bfff640 1 -- 172.21.15.16:0/3877774521 learned_addr learned my addr 172.21.15.16:0/3877774521 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:59.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.545+0000 7f0113fff640 1 --2- 172.21.15.16:0/3877774521 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590eb9b3290 0x5590eb9b9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:59.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.545+0000 7f0118cf3640 1 --2- 172.21.15.16:0/3877774521 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590eb9bc790 0x5590eb9ba1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:59.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.545+0000 7f010bfff640 1 -- 172.21.15.16:0/3877774521 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590eb9b3290 msgr2=0x5590eb9b9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:59.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.545+0000 7f010bfff640 1 --2- 172.21.15.16:0/3877774521 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590eb9b3290 0x5590eb9b9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.545+0000 7f010bfff640 1 -- 172.21.15.16:0/3877774521 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590eb9bc790 msgr2=0x5590eb9ba1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:59.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.545+0000 7f010bfff640 1 --2- 172.21.15.16:0/3877774521 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590eb9bc790 0x5590eb9ba1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.545+0000 7f010bfff640 1 -- 172.21.15.16:0/3877774521 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590eb6f57d0 con 0x5590eb9b3bb0 2024-03-28T22:17:59.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.545+0000 7f0118cf3640 1 --2- 172.21.15.16:0/3877774521 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590eb9bc790 0x5590eb9ba1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:17:59.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.546+0000 7f010bfff640 1 --2- 172.21.15.16:0/3877774521 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590eb9b3bb0 0x5590eb9b9ca0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f00fc097900 tx=0x7f00fc0cc1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9339bcff1a94549 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:59.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.546+0000 7f00f37fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:17:59.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.546+0000 7f00f37fe640 1 -- 172.21.15.16:0/3877774521 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00fc0b3060 con 0x5590eb9b3bb0 2024-03-28T22:17:59.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.546+0000 7f00f37fe640 1 -- 172.21.15.16:0/3877774521 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8527+0+0 (secure 0 0 0) 0x7f00fc09c450 con 0x5590eb9b3bb0 2024-03-28T22:17:59.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.546+0000 7f011d415740 1 -- 172.21.15.16:0/3877774521 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00fc09af30 con 0x5590eb9b3bb0 2024-03-28T22:17:59.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.546+0000 7f00f37fe640 1 -- 172.21.15.16:0/3877774521 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f00fc09b8c0 con 0x5590eb9b3bb0 2024-03-28T22:17:59.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.546+0000 7f011d415740 10 client.17112 fetch_fsmap learned FSMap version 27 2024-03-28T22:17:59.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.546+0000 7f011d415740 1 -- 172.21.15.16:0/3877774521 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5590eb62b3f0 con 0x5590eb9b3bb0 2024-03-28T22:17:59.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.546+0000 7f00f37fe640 1 -- 172.21.15.16:0/3877774521 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00fc09af30 con 0x5590eb9b3bb0 2024-03-28T22:17:59.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.546+0000 7f00f37fe640 1 -- 172.21.15.16:0/3877774521 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f00fc018b30 con 0x5590eb9b3bb0 2024-03-28T22:17:59.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.546+0000 7f011d415740 10 client.17112 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:17:59.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.546+0000 7f011d415740 10 client.17112 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:17:59.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.546+0000 7f011d415740 20 client.17112 populate_metadata read hostname 'smithi016' 2024-03-28T22:17:59.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.546+0000 7f011d415740 1 --2- 172.21.15.16:0/3877774521 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5590eb9cc940 0x5590eb9ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:59.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.546+0000 7f011d415740 4 client.17112 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:17:59.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.546+0000 7f011d415740 1 -- 172.21.15.16:0/3877774521 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5590eb9246c0 con 0x5590eb9cc940 2024-03-28T22:17:59.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.547+0000 7f0113fff640 1 --2- 172.21.15.16:0/3877774521 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5590eb9cc940 0x5590eb9ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:59.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.547+0000 7f0113fff640 1 --2- 172.21.15.16:0/3877774521 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5590eb9cc940 0x5590eb9ced30 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6cdab1eb6076eba server_cookie=f4de490487a549f in_seq=0 out_seq=0 2024-03-28T22:17:59.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.547+0000 7f00f37fe640 10 client.17112 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:17:59.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.548+0000 7f00f37fe640 1 -- 172.21.15.16:0/3877774521 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5590eb9246c0 con 0x5590eb9cc940 2024-03-28T22:17:59.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.548+0000 7f00f37fe640 10 client.17112 handle_command_reply: tid=0 2024-03-28T22:17:59.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.548+0000 7f011d415740 1 client.17112 shutdown 2024-03-28T22:17:59.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.548+0000 7f011d415740 1 -- 172.21.15.16:0/3877774521 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590eb9b3bb0 msgr2=0x5590eb9b9ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:59.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.548+0000 7f011d415740 1 --2- 172.21.15.16:0/3877774521 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590eb9b3bb0 0x5590eb9b9ca0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f00fc097900 tx=0x7f00fc0cc1c0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.548+0000 7f011d415740 1 -- 172.21.15.16:0/3877774521 shutdown_connections 2024-03-28T22:17:59.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.548+0000 7f011d415740 1 --2- 172.21.15.16:0/3877774521 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5590eb9cc940 0x5590eb9ced30 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.548+0000 7f011d415740 1 --2- 172.21.15.16:0/3877774521 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590eb9bc790 0x5590eb9ba1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.548+0000 7f011d415740 1 --2- 172.21.15.16:0/3877774521 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590eb9b3bb0 0x5590eb9b9ca0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.548+0000 7f011d415740 1 --2- 172.21.15.16:0/3877774521 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590eb9b3290 0x5590eb9b9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.548+0000 7f00f37fe640 0 client.17112 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:17:59.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.548+0000 7f011d415740 1 -- 172.21.15.16:0/3877774521 >> 172.21.15.16:0/3877774521 conn(0x5590eb9ad9c0 msgr2=0x5590eb9acc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:59.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.549+0000 7f011d415740 1 -- 172.21.15.16:0/3877774521 shutdown_connections 2024-03-28T22:17:59.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.549+0000 7f011d415740 1 -- 172.21.15.16:0/3877774521 wait complete. 2024-03-28T22:17:59.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.549+0000 7f011d415740 20 client.17112 trim_cache size 0 max 16384 2024-03-28T22:17:59.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.569+0000 7f011d415740 1 Processor -- start 2024-03-28T22:17:59.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.569+0000 7f011d415740 1 -- start start 2024-03-28T22:17:59.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.569+0000 7f011d415740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590eb9ad9c0 0x5590eb9bbe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:59.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.569+0000 7f011d415740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590eb9b3290 0x5590eb9ba000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:59.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.569+0000 7f011d415740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590eb9b3bb0 0x5590eb9d1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:59.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.569+0000 7f011d415740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f01141afd90 con 0x5590eb9b3bb0 2024-03-28T22:17:59.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.569+0000 7f011d415740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f01141ad660 con 0x5590eb9b3290 2024-03-28T22:17:59.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.569+0000 7f011d415740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f01141ab4d0 con 0x5590eb9ad9c0 2024-03-28T22:17:59.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.569+0000 7f010bfff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590eb9b3290 0x5590eb9ba000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:59.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.569+0000 7f0113fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590eb9ad9c0 0x5590eb9bbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:59.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.569+0000 7f010bfff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590eb9b3290 0x5590eb9ba000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:39784/0 (socket says 172.21.15.16:39784) 2024-03-28T22:17:59.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.569+0000 7f0113fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590eb9ad9c0 0x5590eb9bbe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.16:41870/0 (socket says 172.21.15.16:41870) 2024-03-28T22:17:59.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.569+0000 7f010bfff640 1 -- 172.21.15.16:0/3008814711 learned_addr learned my addr 172.21.15.16:0/3008814711 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:59.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.569+0000 7f0118cf3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590eb9b3bb0 0x5590eb9d1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:59.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.569+0000 7f0113fff640 1 -- 172.21.15.16:0/3008814711 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590eb9b3290 msgr2=0x5590eb9ba000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:59.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.569+0000 7f0113fff640 1 --2- 172.21.15.16:0/3008814711 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590eb9b3290 0x5590eb9ba000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.569+0000 7f0113fff640 1 -- 172.21.15.16:0/3008814711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590eb9b3bb0 msgr2=0x5590eb9d1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:59.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.569+0000 7f0113fff640 1 --2- 172.21.15.16:0/3008814711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590eb9b3bb0 0x5590eb9d1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.569+0000 7f0113fff640 1 -- 172.21.15.16:0/3008814711 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590eb5eb0f0 con 0x5590eb9ad9c0 2024-03-28T22:17:59.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.569+0000 7f0118cf3640 1 --2- 172.21.15.16:0/3008814711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590eb9b3bb0 0x5590eb9d1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:17:59.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.570+0000 7f0113fff640 1 --2- 172.21.15.16:0/3008814711 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590eb9ad9c0 0x5590eb9bbe90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0104009230 tx=0x7f0104009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44c6919a7de9734e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:59.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.570+0000 7f0109ffb640 1 -- 172.21.15.16:0/3008814711 <== mon.1 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f010404f070 con 0x5590eb9ad9c0 2024-03-28T22:17:59.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.570+0000 7f0109ffb640 1 -- 172.21.15.16:0/3008814711 <== mon.1 v2:172.21.15.63:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0104004300 con 0x5590eb9ad9c0 2024-03-28T22:17:59.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.570+0000 7f0109ffb640 1 -- 172.21.15.16:0/3008814711 <== mon.1 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f010404b310 con 0x5590eb9ad9c0 2024-03-28T22:17:59.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.571+0000 7f011d415740 1 -- 172.21.15.16:0/3008814711 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590eb9ad9c0 msgr2=0x5590eb9bbe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:59.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.571+0000 7f011d415740 1 --2- 172.21.15.16:0/3008814711 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590eb9ad9c0 0x5590eb9bbe90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0104009230 tx=0x7f0104009260 comp rx=0 tx=0).stop 2024-03-28T22:17:59.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.571+0000 7f011d415740 1 -- 172.21.15.16:0/3008814711 shutdown_connections 2024-03-28T22:17:59.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.571+0000 7f011d415740 1 --2- 172.21.15.16:0/3008814711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590eb9b3bb0 0x5590eb9d1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.571+0000 7f011d415740 1 --2- 172.21.15.16:0/3008814711 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590eb9b3290 0x5590eb9ba000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.571+0000 7f011d415740 1 --2- 172.21.15.16:0/3008814711 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590eb9ad9c0 0x5590eb9bbe90 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.571+0000 7f011d415740 1 -- 172.21.15.16:0/3008814711 >> 172.21.15.16:0/3008814711 conn(0x5590eb9b83a0 msgr2=0x5590eb9acc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:59.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.571+0000 7f011d415740 1 -- 172.21.15.16:0/3008814711 shutdown_connections 2024-03-28T22:17:59.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.571+0000 7f011d415740 1 -- 172.21.15.16:0/3008814711 wait complete. 2024-03-28T22:17:59.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.571+0000 7f011d415740 1 -- start start 2024-03-28T22:17:59.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.571+0000 7f011d415740 1 Processor -- start 2024-03-28T22:17:59.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.572+0000 7f011d415740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590eb9ad9c0 0x5590eb9cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:59.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.572+0000 7f011d415740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590eb9b3290 0x5590eb9cfc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:59.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.572+0000 7f011d415740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590eb9b3bb0 0x5590eb9d01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:59.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.572+0000 7f011d415740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f01141abb20 con 0x5590eb9ad9c0 2024-03-28T22:17:59.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.572+0000 7f011d415740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f01141acf90 con 0x5590eb9b3bb0 2024-03-28T22:17:59.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.572+0000 7f011d415740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f01141ac1f0 con 0x5590eb9b3290 2024-03-28T22:17:59.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.572+0000 7f011d415740 10 client.0 _ll_register_callbacks cb 0x7f00fc09aa70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:17:59.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.572+0000 7f0113fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590eb9ad9c0 0x5590eb9cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:59.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.572+0000 7f010bfff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590eb9b3290 0x5590eb9cfc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:59.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.572+0000 7f0113fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590eb9ad9c0 0x5590eb9cf750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55320/0 (socket says 172.21.15.16:55320) 2024-03-28T22:17:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.572+0000 7f0113fff640 1 -- 172.21.15.16:0/3905525951 learned_addr learned my addr 172.21.15.16:0/3905525951 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:17:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.572+0000 7f0118cf3640 1 --2- 172.21.15.16:0/3905525951 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590eb9b3bb0 0x5590eb9d01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.572+0000 7f0113fff640 1 -- 172.21.15.16:0/3905525951 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590eb9b3290 msgr2=0x5590eb9cfc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.572+0000 7f0113fff640 1 --2- 172.21.15.16:0/3905525951 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590eb9b3290 0x5590eb9cfc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.572+0000 7f0113fff640 1 -- 172.21.15.16:0/3905525951 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590eb9b3bb0 msgr2=0x5590eb9d01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:17:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.572+0000 7f0113fff640 1 --2- 172.21.15.16:0/3905525951 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590eb9b3bb0 0x5590eb9d01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.572+0000 7f0113fff640 1 -- 172.21.15.16:0/3905525951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590eba319b0 con 0x5590eb9ad9c0 2024-03-28T22:17:59.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.572+0000 7f010bfff640 1 --2- 172.21.15.16:0/3905525951 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590eb9b3290 0x5590eb9cfc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:17:59.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.572+0000 7f0113fff640 1 --2- 172.21.15.16:0/3905525951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590eb9ad9c0 0x5590eb9cf750 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f0104000c00 tx=0x7f01040056f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fad1f433f0ab0672 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:17:59.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.572+0000 7f00f37fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:17:59.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.572+0000 7f00f37fe640 1 -- 172.21.15.16:0/3905525951 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f010404f040 con 0x5590eb9ad9c0 2024-03-28T22:17:59.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.573+0000 7f00f37fe640 1 -- 172.21.15.16:0/3905525951 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8527+0+0 (secure 0 0 0) 0x7f010402c880 con 0x5590eb9ad9c0 2024-03-28T22:17:59.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.573+0000 7f011d415740 1 -- 172.21.15.16:0/3905525951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0104004300 con 0x5590eb9ad9c0 2024-03-28T22:17:59.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.573+0000 7f00f37fe640 1 -- 172.21.15.16:0/3905525951 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0104060050 con 0x5590eb9ad9c0 2024-03-28T22:17:59.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.573+0000 7f011d415740 10 client.17118 fetch_fsmap learned FSMap version 27 2024-03-28T22:17:59.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.573+0000 7f011d415740 1 -- 172.21.15.16:0/3905525951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5590eba32cc0 con 0x5590eb9ad9c0 2024-03-28T22:17:59.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.573+0000 7f00f37fe640 1 -- 172.21.15.16:0/3905525951 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0104004300 con 0x5590eb9ad9c0 2024-03-28T22:17:59.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.573+0000 7f00f37fe640 1 -- 172.21.15.16:0/3905525951 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f010405b030 con 0x5590eb9ad9c0 2024-03-28T22:17:59.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.573+0000 7f011d415740 10 client.17118 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:17:59.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.573+0000 7f011d415740 10 client.17118 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:17:59.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.573+0000 7f011d415740 20 client.17118 populate_metadata read hostname 'smithi016' 2024-03-28T22:17:59.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.573+0000 7f011d415740 1 --2- 172.21.15.16:0/3905525951 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5590eb9bc790 0x5590eba353f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:59.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.573+0000 7f011d415740 4 client.17118 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:17:59.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.573+0000 7f011d415740 1 -- 172.21.15.16:0/3905525951 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5590eb91ea10 con 0x5590eb9bc790 2024-03-28T22:17:59.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.573+0000 7f010bfff640 1 --2- 172.21.15.16:0/3905525951 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5590eb9bc790 0x5590eba353f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:17:59.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.574+0000 7f010bfff640 1 --2- 172.21.15.16:0/3905525951 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5590eb9bc790 0x5590eba353f0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a68d92cc9723158 server_cookie=ddcf8c5652d24884 in_seq=0 out_seq=0 2024-03-28T22:17:59.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.574+0000 7f00f37fe640 10 client.17118 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:17:59.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.574+0000 7f00f37fe640 1 -- 172.21.15.16:0/3905525951 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5590eb91ea10 con 0x5590eb9bc790 2024-03-28T22:17:59.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.574+0000 7f00f37fe640 10 client.17118 handle_command_reply: tid=0 2024-03-28T22:17:59.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.574+0000 7f011d415740 1 client.17118 shutdown 2024-03-28T22:17:59.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.574+0000 7f011d415740 1 -- 172.21.15.16:0/3905525951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590eb9ad9c0 msgr2=0x5590eb9cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:59.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.574+0000 7f011d415740 1 --2- 172.21.15.16:0/3905525951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590eb9ad9c0 0x5590eb9cf750 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f0104000c00 tx=0x7f01040056f0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.574+0000 7f011d415740 1 -- 172.21.15.16:0/3905525951 shutdown_connections 2024-03-28T22:17:59.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.574+0000 7f011d415740 1 --2- 172.21.15.16:0/3905525951 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5590eb9bc790 0x5590eba353f0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.574+0000 7f011d415740 1 --2- 172.21.15.16:0/3905525951 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5590eb9b3bb0 0x5590eb9d01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.574+0000 7f011d415740 1 --2- 172.21.15.16:0/3905525951 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590eb9b3290 0x5590eb9cfc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.574+0000 7f011d415740 1 --2- 172.21.15.16:0/3905525951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590eb9ad9c0 0x5590eb9cf750 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.574+0000 7f011d415740 1 -- 172.21.15.16:0/3905525951 >> 172.21.15.16:0/3905525951 conn(0x5590eb9b83a0 msgr2=0x5590eb9acc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:59.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.574+0000 7f00f37fe640 0 client.17118 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:17:59.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.574+0000 7f011d415740 1 -- 172.21.15.16:0/3905525951 shutdown_connections 2024-03-28T22:17:59.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.575+0000 7f011d415740 1 -- 172.21.15.16:0/3905525951 wait complete. 2024-03-28T22:17:59.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.575+0000 7f011d415740 20 client.17118 trim_cache size 0 max 16384 2024-03-28T22:17:59.587 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:17:59.587 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T22:17:59.587 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-03-28T22:17:59.587 INFO:teuthology.orchestra.run.smithi016.stdout: "741e9a7c-55be-48ee-bc62-2e700a17d40e": { 2024-03-28T22:17:59.587 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-03-28T22:17:59.587 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "741e9a7c-55be-48ee-bc62-2e700a17d40e", 2024-03-28T22:17:59.588 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-03-28T22:17:59.588 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:17:59.588 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:17:59.588 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:17:59.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.575+0000 7f011a756640 1 -- 172.21.15.16:0/112948767 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f01000a1ef0 msgr2=0x7f01000a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:59.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.575+0000 7f011a756640 1 --2- 172.21.15.16:0/112948767 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f01000a1ef0 0x7f01000a43b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f010c009810 tx=0x7f010c009e80 comp rx=0 tx=0).stop 2024-03-28T22:17:59.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.575+0000 7f011a756640 1 -- 172.21.15.16:0/112948767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0114100fb0 msgr2=0x7f01141985b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:17:59.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.575+0000 7f011a756640 1 --2- 172.21.15.16:0/112948767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0114100fb0 0x7f01141985b0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f00fc00e9f0 tx=0x7f00fc00eeb0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.575+0000 7f011a756640 1 -- 172.21.15.16:0/112948767 shutdown_connections 2024-03-28T22:17:59.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.575+0000 7f011a756640 1 --2- 172.21.15.16:0/112948767 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f01000a1ef0 0x7f01000a43b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.575+0000 7f011a756640 1 --2- 172.21.15.16:0/112948767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0114106690 0x7f011419c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.575+0000 7f011a756640 1 --2- 172.21.15.16:0/112948767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0114100fb0 0x7f01141985b0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.575+0000 7f011a756640 1 --2- 172.21.15.16:0/112948767 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01141005f0 0x7f0114198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:59.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.575+0000 7f011a756640 1 -- 172.21.15.16:0/112948767 >> 172.21.15.16:0/112948767 conn(0x7f0114077800 msgr2=0x7f01140ff780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:17:59.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.576+0000 7f011a756640 1 -- 172.21.15.16:0/112948767 shutdown_connections 2024-03-28T22:17:59.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:17:59.576+0000 7f011a756640 1 -- 172.21.15.16:0/112948767 wait complete. 2024-03-28T22:17:59.591 INFO:tasks.cephfs.filesystem:scrub status for tag:741e9a7c-55be-48ee-bc62-2e700a17d40e - {'path': '/', 'tag': '741e9a7c-55be-48ee-bc62-2e700a17d40e', 'options': 'recursive,force'} 2024-03-28T22:17:59.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:17:59 smithi063 ceph-mon[36249]: pgmap v227: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T22:17:59.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:17:59 smithi016 ceph-mon[32374]: pgmap v227: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T22:18:00.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:17:59 smithi053 ceph-mon[35979]: pgmap v227: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T22:18:00.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:00 smithi063 ceph-mon[36249]: scrub queued for path: / 2024-03-28T22:18:00.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:00 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:18:00.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:00 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:18:00.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:00 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:18:00.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:00 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:18:00.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:00 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:18:00.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:00 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:18:00.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:00 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:18:00.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:00 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:18:00.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:00 smithi016 ceph-mon[32374]: scrub queued for path: / 2024-03-28T22:18:00.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:00 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:18:00.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:00 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:18:00.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:00 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:18:00.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:00 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:18:00.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:00 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:18:00.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:00 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:18:00.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:00 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:18:00.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:00 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:18:01.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:00 smithi053 ceph-mon[35979]: scrub queued for path: / 2024-03-28T22:18:01.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:00 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:18:01.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:00 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:18:01.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:00 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:18:01.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:00 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:18:01.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:00 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:18:01.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:00 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:18:01.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:00 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:18:01.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:00 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:18:01.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:01 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:18:01.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:01 smithi063 ceph-mon[36249]: pgmap v228: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-03-28T22:18:01.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:01 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:18:01.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:01 smithi016 ceph-mon[32374]: pgmap v228: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-03-28T22:18:02.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:01 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:18:02.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:01 smithi053 ceph-mon[35979]: pgmap v228: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-03-28T22:18:03.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:03 smithi063 ceph-mon[36249]: pgmap v229: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-28T22:18:03.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:03 smithi016 ceph-mon[32374]: pgmap v229: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-28T22:18:04.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:03 smithi053 ceph-mon[35979]: pgmap v229: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-28T22:18:05.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:05 smithi063 ceph-mon[36249]: pgmap v230: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-28T22:18:05.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:05 smithi016 ceph-mon[32374]: pgmap v230: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-28T22:18:06.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:05 smithi053 ceph-mon[35979]: pgmap v230: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-28T22:18:07.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:07 smithi063 ceph-mon[36249]: pgmap v231: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-28T22:18:07.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:07 smithi016 ceph-mon[32374]: pgmap v231: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-28T22:18:08.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:07 smithi053 ceph-mon[35979]: pgmap v231: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-28T22:18:09.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:09 smithi063 ceph-mon[36249]: pgmap v232: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-28T22:18:09.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:09 smithi016 ceph-mon[32374]: pgmap v232: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-28T22:18:10.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:09 smithi053 ceph-mon[35979]: pgmap v232: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-28T22:18:10.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:10 smithi063 ceph-mon[36249]: scrub summary: idle 2024-03-28T22:18:10.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:10 smithi016 ceph-mon[32374]: scrub summary: idle 2024-03-28T22:18:11.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:10 smithi053 ceph-mon[35979]: scrub summary: idle 2024-03-28T22:18:11.934 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:11 smithi063 ceph-mon[36249]: scrub summary: idle 2024-03-28T22:18:11.934 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:11 smithi063 ceph-mon[36249]: pgmap v233: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-03-28T22:18:11.934 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:11 smithi063 ceph-mon[36249]: osdmap e92: 12 total, 12 up, 12 in 2024-03-28T22:18:11.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:11 smithi016 ceph-mon[32374]: scrub summary: idle 2024-03-28T22:18:11.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:11 smithi016 ceph-mon[32374]: pgmap v233: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-03-28T22:18:11.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:11 smithi016 ceph-mon[32374]: osdmap e92: 12 total, 12 up, 12 in 2024-03-28T22:18:12.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:11 smithi053 ceph-mon[35979]: scrub summary: idle 2024-03-28T22:18:12.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:11 smithi053 ceph-mon[35979]: pgmap v233: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-03-28T22:18:12.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:11 smithi053 ceph-mon[35979]: osdmap e92: 12 total, 12 up, 12 in 2024-03-28T22:18:13.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:13 smithi063 ceph-mon[36249]: pgmap v235: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-28T22:18:13.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:13 smithi016 ceph-mon[32374]: pgmap v235: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-28T22:18:14.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:13 smithi053 ceph-mon[35979]: pgmap v235: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-28T22:18:15.849 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:15 smithi053 ceph-mon[35979]: pgmap v236: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-28T22:18:15.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:15 smithi063 ceph-mon[36249]: pgmap v236: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-28T22:18:15.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:15 smithi016 ceph-mon[32374]: pgmap v236: 97 pgs: 97 active+clean; 96 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-28T22:18:17.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:17 smithi063 ceph-mon[36249]: pgmap v237: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T22:18:17.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:17 smithi063 ceph-mon[36249]: osdmap e93: 12 total, 12 up, 12 in 2024-03-28T22:18:17.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:17 smithi016 ceph-mon[32374]: pgmap v237: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T22:18:17.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:17 smithi016 ceph-mon[32374]: osdmap e93: 12 total, 12 up, 12 in 2024-03-28T22:18:18.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:17 smithi053 ceph-mon[35979]: pgmap v237: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T22:18:18.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:17 smithi053 ceph-mon[35979]: osdmap e93: 12 total, 12 up, 12 in 2024-03-28T22:18:19.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:19 smithi063 ceph-mon[36249]: pgmap v239: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-28T22:18:19.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:19 smithi016 ceph-mon[32374]: pgmap v239: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-28T22:18:20.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:19 smithi053 ceph-mon[35979]: pgmap v239: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-28T22:18:21.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:21 smithi063 ceph-mon[36249]: pgmap v240: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 104 B/s wr, 6 op/s 2024-03-28T22:18:21.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:21 smithi016 ceph-mon[32374]: pgmap v240: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 104 B/s wr, 6 op/s 2024-03-28T22:18:22.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:21 smithi053 ceph-mon[35979]: pgmap v240: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 104 B/s wr, 6 op/s 2024-03-28T22:18:23.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:23 smithi063 ceph-mon[36249]: pgmap v241: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T22:18:23.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:23 smithi016 ceph-mon[32374]: pgmap v241: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T22:18:24.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:23 smithi053 ceph-mon[35979]: pgmap v241: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T22:18:25.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:25 smithi063 ceph-mon[36249]: pgmap v242: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T22:18:25.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:25 smithi016 ceph-mon[32374]: pgmap v242: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T22:18:26.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:25 smithi053 ceph-mon[35979]: pgmap v242: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T22:18:27.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:27 smithi063 ceph-mon[36249]: pgmap v243: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-28T22:18:27.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:27 smithi016 ceph-mon[32374]: pgmap v243: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-28T22:18:28.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:27 smithi053 ceph-mon[35979]: pgmap v243: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-28T22:18:29.592 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:18:29.592 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:18:29.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.709+0000 7fcfc19a1640 1 -- 172.21.15.16:0/2100446225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfbc102970 msgr2=0x7fcfbc102d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:29.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.709+0000 7fcfc19a1640 1 --2- 172.21.15.16:0/2100446225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfbc102970 0x7fcfbc102d50 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fcfac009900 tx=0x7fcfac02bcd0 comp rx=0 tx=0).stop 2024-03-28T22:18:29.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.710+0000 7fcfc19a1640 1 -- 172.21.15.16:0/2100446225 shutdown_connections 2024-03-28T22:18:29.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.710+0000 7fcfc19a1640 1 --2- 172.21.15.16:0/2100446225 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcfbc10bd40 0x7fcfbc10e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:29.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.710+0000 7fcfc19a1640 1 --2- 172.21.15.16:0/2100446225 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcfbc103290 0x7fcfbc10b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:29.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.710+0000 7fcfc19a1640 1 --2- 172.21.15.16:0/2100446225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfbc102970 0x7fcfbc102d50 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:29.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.710+0000 7fcfc19a1640 1 -- 172.21.15.16:0/2100446225 >> 172.21.15.16:0/2100446225 conn(0x7fcfbc077800 msgr2=0x7fcfbc077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:18:29.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.710+0000 7fcfc19a1640 1 -- 172.21.15.16:0/2100446225 shutdown_connections 2024-03-28T22:18:29.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.710+0000 7fcfc19a1640 1 -- 172.21.15.16:0/2100446225 wait complete. 2024-03-28T22:18:29.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.710+0000 7fcfc19a1640 1 Processor -- start 2024-03-28T22:18:29.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.710+0000 7fcfc19a1640 1 -- start start 2024-03-28T22:18:29.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.711+0000 7fcfc19a1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcfbc102970 0x7fcfbc197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:29.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.711+0000 7fcfc19a1640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcfbc103290 0x7fcfbc198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:29.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.711+0000 7fcfc19a1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfbc10bd40 0x7fcfbc19c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:29.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.711+0000 7fcfc19a1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcfbc111ad0 con 0x7fcfbc10bd40 2024-03-28T22:18:29.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.711+0000 7fcfc19a1640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcfbc111950 con 0x7fcfbc103290 2024-03-28T22:18:29.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.711+0000 7fcfc19a1640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcfbc111c50 con 0x7fcfbc102970 2024-03-28T22:18:29.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.711+0000 7fcfbb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfbc10bd40 0x7fcfbc19c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:29.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.711+0000 7fcfbb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfbc10bd40 0x7fcfbc19c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53538/0 (socket says 172.21.15.16:53538) 2024-03-28T22:18:29.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.711+0000 7fcfbb7fe640 1 -- 172.21.15.16:0/2851107300 learned_addr learned my addr 172.21.15.16:0/2851107300 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:18:29.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.711+0000 7fcfba7fc640 1 --2- 172.21.15.16:0/2851107300 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcfbc103290 0x7fcfbc198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:29.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.711+0000 7fcfbaffd640 1 --2- 172.21.15.16:0/2851107300 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcfbc102970 0x7fcfbc197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:29.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.711+0000 7fcfbb7fe640 1 -- 172.21.15.16:0/2851107300 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcfbc102970 msgr2=0x7fcfbc197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:29.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.711+0000 7fcfbb7fe640 1 --2- 172.21.15.16:0/2851107300 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcfbc102970 0x7fcfbc197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:29.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.711+0000 7fcfbb7fe640 1 -- 172.21.15.16:0/2851107300 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcfbc103290 msgr2=0x7fcfbc198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:29.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.711+0000 7fcfbb7fe640 1 --2- 172.21.15.16:0/2851107300 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcfbc103290 0x7fcfbc198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:29.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.711+0000 7fcfbb7fe640 1 -- 172.21.15.16:0/2851107300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfbc19cf40 con 0x7fcfbc10bd40 2024-03-28T22:18:29.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.711+0000 7fcfbaffd640 1 --2- 172.21.15.16:0/2851107300 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcfbc102970 0x7fcfbc197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:18:29.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.711+0000 7fcfba7fc640 1 --2- 172.21.15.16:0/2851107300 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcfbc103290 0x7fcfbc198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:18:29.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.711+0000 7fcfbb7fe640 1 --2- 172.21.15.16:0/2851107300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfbc10bd40 0x7fcfbc19c810 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fcfb400cab0 tx=0x7fcfb400cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:18:29.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.711+0000 7fcfc099f640 1 -- 172.21.15.16:0/2851107300 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb400ae20 con 0x7fcfbc10bd40 2024-03-28T22:18:29.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.711+0000 7fcfc099f640 1 -- 172.21.15.16:0/2851107300 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfb400f930 con 0x7fcfbc10bd40 2024-03-28T22:18:29.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.712+0000 7fcfc19a1640 1 -- 172.21.15.16:0/2851107300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfbc19d230 con 0x7fcfbc10bd40 2024-03-28T22:18:29.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.712+0000 7fcfc099f640 1 -- 172.21.15.16:0/2851107300 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb4018b00 con 0x7fcfbc10bd40 2024-03-28T22:18:29.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.712+0000 7fcfc19a1640 1 -- 172.21.15.16:0/2851107300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfbc1a4ac0 con 0x7fcfbc10bd40 2024-03-28T22:18:29.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.714+0000 7fcfc099f640 1 -- 172.21.15.16:0/2851107300 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7fcfb400fae0 con 0x7fcfbc10bd40 2024-03-28T22:18:29.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.714+0000 7fcfc4589740 1 Processor -- start 2024-03-28T22:18:29.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.715+0000 7fcfc4589740 1 -- start start 2024-03-28T22:18:29.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.715+0000 7fcfc4589740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e63855f40 0x559e63856320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:29.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.715+0000 7fcfc4589740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e63856860 0x559e63863660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:29.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.715+0000 7fcfc4589740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e63863ba0 0x559e63865f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:29.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.715+0000 7fcfc4589740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559e637ba180 con 0x559e63863ba0 2024-03-28T22:18:29.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.715+0000 7fcfc4589740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x559e63762b70 con 0x559e63855f40 2024-03-28T22:18:29.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.715+0000 7fcfc4589740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x559e637b6220 con 0x559e63856860 2024-03-28T22:18:29.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.715+0000 7fcfba7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e63863ba0 0x559e63865f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:29.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.715+0000 7fcfba7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e63863ba0 0x559e63865f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53550/0 (socket says 172.21.15.16:53550) 2024-03-28T22:18:29.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.715+0000 7fcfba7fc640 1 -- 172.21.15.16:0/4010437518 learned_addr learned my addr 172.21.15.16:0/4010437518 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:18:29.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.715+0000 7fcfc099f640 1 --2- 172.21.15.16:0/2851107300 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fcfa40a1ef0 0x7fcfa40a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:29.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.715+0000 7fcfbaffd640 1 --2- 172.21.15.16:0/4010437518 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e63856860 0x559e63863660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:29.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.715+0000 7fcfba7fc640 1 --2- 172.21.15.16:0/4010437518 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e63855f40 0x559e63856320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:29.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.715+0000 7fcfbb7fe640 1 --2- 172.21.15.16:0/2851107300 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fcfa40a1ef0 0x7fcfa40a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:29.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.715+0000 7fcfba7fc640 1 -- 172.21.15.16:0/4010437518 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e63856860 msgr2=0x559e63863660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:29.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.715+0000 7fcfba7fc640 1 --2- 172.21.15.16:0/4010437518 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e63856860 0x559e63863660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:29.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.715+0000 7fcfba7fc640 1 -- 172.21.15.16:0/4010437518 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e63855f40 msgr2=0x559e63856320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:29.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.715+0000 7fcfba7fc640 1 --2- 172.21.15.16:0/4010437518 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e63855f40 0x559e63856320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:29.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.715+0000 7fcfba7fc640 1 -- 172.21.15.16:0/4010437518 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e637767c0 con 0x559e63863ba0 2024-03-28T22:18:29.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.715+0000 7fcfc099f640 1 -- 172.21.15.16:0/2851107300 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7fcfb40ce890 con 0x7fcfbc10bd40 2024-03-28T22:18:29.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.716+0000 7fcfba7fc640 1 --2- 172.21.15.16:0/4010437518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e63863ba0 0x559e63865f90 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fcfb00058c0 tx=0x7fcfb0009770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed21e6a464d69e6d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:18:29.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.716+0000 7fcfbb7fe640 1 --2- 172.21.15.16:0/2851107300 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fcfa40a1ef0 0x7fcfa40a43b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fcfb40cc090 tx=0x7fcfb4018400 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:18:29.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.716+0000 7fcfa1ffb640 1 -- 172.21.15.16:0/4010437518 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb000fc50 con 0x559e63863ba0 2024-03-28T22:18:29.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.716+0000 7fcfa1ffb640 1 -- 172.21.15.16:0/4010437518 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfb000fdf0 con 0x559e63863ba0 2024-03-28T22:18:29.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.716+0000 7fcfa1ffb640 1 -- 172.21.15.16:0/4010437518 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb001e3a0 con 0x559e63863ba0 2024-03-28T22:18:29.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.716+0000 7fcfc4589740 1 -- 172.21.15.16:0/4010437518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e63863ba0 msgr2=0x559e63865f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:29.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.716+0000 7fcfc4589740 1 --2- 172.21.15.16:0/4010437518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e63863ba0 0x559e63865f90 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fcfb00058c0 tx=0x7fcfb0009770 comp rx=0 tx=0).stop 2024-03-28T22:18:29.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.716+0000 7fcfc4589740 1 -- 172.21.15.16:0/4010437518 shutdown_connections 2024-03-28T22:18:29.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.716+0000 7fcfc4589740 1 --2- 172.21.15.16:0/4010437518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e63863ba0 0x559e63865f90 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:29.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.716+0000 7fcfc4589740 1 --2- 172.21.15.16:0/4010437518 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e63856860 0x559e63863660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:29.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.716+0000 7fcfc4589740 1 --2- 172.21.15.16:0/4010437518 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e63855f40 0x559e63856320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:29.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.716+0000 7fcfc4589740 1 -- 172.21.15.16:0/4010437518 >> 172.21.15.16:0/4010437518 conn(0x559e638509c0 msgr2=0x559e63850da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:18:29.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.717+0000 7fcfc4589740 1 -- 172.21.15.16:0/4010437518 shutdown_connections 2024-03-28T22:18:29.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.717+0000 7fcfc4589740 1 -- 172.21.15.16:0/4010437518 wait complete. 2024-03-28T22:18:29.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.717+0000 7fcfc4589740 1 -- start start 2024-03-28T22:18:29.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.717+0000 7fcfc4589740 1 Processor -- start 2024-03-28T22:18:29.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.717+0000 7fcfc4589740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e63855f40 0x559e63860b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:29.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.717+0000 7fcfc4589740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e63856860 0x559e638610d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:29.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.717+0000 7fcfc4589740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e63863ba0 0x559e63861610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:29.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.717+0000 7fcfc4589740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559e638669e0 con 0x559e63855f40 2024-03-28T22:18:29.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.717+0000 7fcfc4589740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x559e637584a0 con 0x559e63856860 2024-03-28T22:18:29.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.717+0000 7fcfc4589740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x559e63866b60 con 0x559e63863ba0 2024-03-28T22:18:29.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.717+0000 7fcfc4589740 10 client.0 _ll_register_callbacks cb 0x559e6375bd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:18:29.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.717+0000 7fcfba7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e63855f40 0x559e63860b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:29.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.718+0000 7fcfba7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e63855f40 0x559e63860b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53562/0 (socket says 172.21.15.16:53562) 2024-03-28T22:18:29.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.718+0000 7fcfba7fc640 1 -- 172.21.15.16:0/1952101465 learned_addr learned my addr 172.21.15.16:0/1952101465 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:18:29.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.718+0000 7fcfbaffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e63856860 0x559e638610d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:29.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.718+0000 7fcfba7fc640 1 -- 172.21.15.16:0/1952101465 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e63863ba0 msgr2=0x559e63861610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:29.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.718+0000 7fcfba7fc640 1 --2- 172.21.15.16:0/1952101465 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e63863ba0 0x559e63861610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:29.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.718+0000 7fcfba7fc640 1 -- 172.21.15.16:0/1952101465 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e63856860 msgr2=0x559e638610d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:29.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.718+0000 7fcfba7fc640 1 --2- 172.21.15.16:0/1952101465 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e63856860 0x559e638610d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:29.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.718+0000 7fcfba7fc640 1 -- 172.21.15.16:0/1952101465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e635987d0 con 0x559e63855f40 2024-03-28T22:18:29.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.718+0000 7fcfba7fc640 1 --2- 172.21.15.16:0/1952101465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e63855f40 0x559e63860b90 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fcfb00056d0 tx=0x7fcfb0009370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbcbeecac4280791 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:18:29.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.718+0000 7fcf9affd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:18:29.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.718+0000 7fcf9affd640 1 -- 172.21.15.16:0/1952101465 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb0003d10 con 0x559e63855f40 2024-03-28T22:18:29.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.718+0000 7fcf9affd640 1 -- 172.21.15.16:0/1952101465 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7fcfb0003eb0 con 0x559e63855f40 2024-03-28T22:18:29.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.718+0000 7fcfc4589740 1 -- 172.21.15.16:0/1952101465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfb000fdf0 con 0x559e63855f40 2024-03-28T22:18:29.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.718+0000 7fcf9affd640 1 -- 172.21.15.16:0/1952101465 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcfb00040b0 con 0x559e63855f40 2024-03-28T22:18:29.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.718+0000 7fcfc4589740 10 client.17130 fetch_fsmap learned FSMap version 27 2024-03-28T22:18:29.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.718+0000 7fcfc4589740 1 -- 172.21.15.16:0/1952101465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559e634ce3f0 con 0x559e63855f40 2024-03-28T22:18:29.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.719+0000 7fcf9affd640 1 -- 172.21.15.16:0/1952101465 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfb000fdf0 con 0x559e63855f40 2024-03-28T22:18:29.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.719+0000 7fcf9affd640 1 -- 172.21.15.16:0/1952101465 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7fcfb0034cc0 con 0x559e63855f40 2024-03-28T22:18:29.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.719+0000 7fcfc4589740 10 client.17130 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:18:29.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.719+0000 7fcfc4589740 10 client.17130 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:18:29.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.719+0000 7fcfc4589740 20 client.17130 populate_metadata read hostname 'smithi016' 2024-03-28T22:18:29.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.719+0000 7fcfc4589740 1 --2- 172.21.15.16:0/1952101465 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x559e63873e80 0x559e63876270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:29.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.719+0000 7fcfc4589740 4 client.17130 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:18:29.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.719+0000 7fcfc4589740 1 -- 172.21.15.16:0/1952101465 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559e637c76c0 con 0x559e63873e80 2024-03-28T22:18:29.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.719+0000 7fcfbaffd640 1 --2- 172.21.15.16:0/1952101465 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x559e63873e80 0x559e63876270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:29.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.719+0000 7fcf9affd640 10 client.17130 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:18:29.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.719+0000 7fcfbaffd640 1 --2- 172.21.15.16:0/1952101465 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x559e63873e80 0x559e63876270 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0d63a549dc8cc18 server_cookie=34320a6ab1cc7cc5 in_seq=0 out_seq=0 2024-03-28T22:18:29.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.720+0000 7fcf9affd640 1 -- 172.21.15.16:0/1952101465 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559e637c76c0 con 0x559e63873e80 2024-03-28T22:18:29.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.720+0000 7fcf9affd640 10 client.17130 handle_command_reply: tid=0 2024-03-28T22:18:29.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.720+0000 7fcfc4589740 1 client.17130 shutdown 2024-03-28T22:18:29.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.720+0000 7fcfc4589740 1 -- 172.21.15.16:0/1952101465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e63855f40 msgr2=0x559e63860b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:29.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.720+0000 7fcfc4589740 1 --2- 172.21.15.16:0/1952101465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e63855f40 0x559e63860b90 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fcfb00056d0 tx=0x7fcfb0009370 comp rx=0 tx=0).stop 2024-03-28T22:18:29.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.720+0000 7fcfc4589740 1 -- 172.21.15.16:0/1952101465 shutdown_connections 2024-03-28T22:18:29.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.720+0000 7fcfc4589740 1 --2- 172.21.15.16:0/1952101465 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x559e63873e80 0x559e63876270 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:29.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.720+0000 7fcfc4589740 1 --2- 172.21.15.16:0/1952101465 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e63863ba0 0x559e63861610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:29.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.720+0000 7fcfc4589740 1 --2- 172.21.15.16:0/1952101465 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e63856860 0x559e638610d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:29.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.720+0000 7fcfc4589740 1 --2- 172.21.15.16:0/1952101465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e63855f40 0x559e63860b90 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:29.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.720+0000 7fcf9affd640 0 client.17130 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:18:29.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.720+0000 7fcfc4589740 1 -- 172.21.15.16:0/1952101465 >> 172.21.15.16:0/1952101465 conn(0x559e638509c0 msgr2=0x559e63852710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:18:29.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.720+0000 7fcfc4589740 1 -- 172.21.15.16:0/1952101465 shutdown_connections 2024-03-28T22:18:29.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.720+0000 7fcfc4589740 1 -- 172.21.15.16:0/1952101465 wait complete. 2024-03-28T22:18:29.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.720+0000 7fcfc4589740 20 client.17130 trim_cache size 0 max 16384 2024-03-28T22:18:29.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.738+0000 7fcfc4589740 1 Processor -- start 2024-03-28T22:18:29.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.738+0000 7fcfc4589740 1 -- start start 2024-03-28T22:18:29.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.738+0000 7fcfc4589740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e63855f40 0x559e63860470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:29.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.738+0000 7fcfc4589740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e63856860 0x559e6386f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:29.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.738+0000 7fcfc4589740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e63863ba0 0x559e6387ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:29.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.738+0000 7fcfc4589740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcfbc1adae0 con 0x559e63863ba0 2024-03-28T22:18:29.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.738+0000 7fcfc4589740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcfbc1ab3b0 con 0x559e63855f40 2024-03-28T22:18:29.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.738+0000 7fcfc4589740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcfbc1a9220 con 0x559e63856860 2024-03-28T22:18:29.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.738+0000 7fcfba7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e63863ba0 0x559e6387ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:29.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.738+0000 7fcfba7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e63863ba0 0x559e6387ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53576/0 (socket says 172.21.15.16:53576) 2024-03-28T22:18:29.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.738+0000 7fcfba7fc640 1 -- 172.21.15.16:0/4048198773 learned_addr learned my addr 172.21.15.16:0/4048198773 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:18:29.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.738+0000 7fcfba7fc640 1 --2- 172.21.15.16:0/4048198773 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e63855f40 0x559e63860470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:29.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.739+0000 7fcfbaffd640 1 --2- 172.21.15.16:0/4048198773 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e63856860 0x559e6386f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:29.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.739+0000 7fcfba7fc640 1 -- 172.21.15.16:0/4048198773 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e63856860 msgr2=0x559e6386f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:29.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.739+0000 7fcfba7fc640 1 --2- 172.21.15.16:0/4048198773 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e63856860 0x559e6386f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:29.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.739+0000 7fcfba7fc640 1 -- 172.21.15.16:0/4048198773 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e63855f40 msgr2=0x559e63860470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:29.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.739+0000 7fcfba7fc640 1 --2- 172.21.15.16:0/4048198773 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e63855f40 0x559e63860470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:29.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.739+0000 7fcfba7fc640 1 -- 172.21.15.16:0/4048198773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e6348e0f0 con 0x559e63863ba0 2024-03-28T22:18:29.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.739+0000 7fcfba7fc640 1 --2- 172.21.15.16:0/4048198773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e63863ba0 0x559e6387ba30 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fcfb002da30 tx=0x7fcfb0005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e25e0a9a0e9f98d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:18:29.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.739+0000 7fcfa17fa640 1 -- 172.21.15.16:0/4048198773 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb0009ea0 con 0x559e63863ba0 2024-03-28T22:18:29.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.739+0000 7fcfa17fa640 1 -- 172.21.15.16:0/4048198773 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfb0023040 con 0x559e63863ba0 2024-03-28T22:18:29.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.739+0000 7fcfa17fa640 1 -- 172.21.15.16:0/4048198773 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb0003f00 con 0x559e63863ba0 2024-03-28T22:18:29.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.740+0000 7fcfc4589740 1 -- 172.21.15.16:0/4048198773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e63863ba0 msgr2=0x559e6387ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:29.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.740+0000 7fcfc4589740 1 --2- 172.21.15.16:0/4048198773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e63863ba0 0x559e6387ba30 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fcfb002da30 tx=0x7fcfb0005970 comp rx=0 tx=0).stop 2024-03-28T22:18:29.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.740+0000 7fcfc4589740 1 -- 172.21.15.16:0/4048198773 shutdown_connections 2024-03-28T22:18:29.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.740+0000 7fcfc4589740 1 --2- 172.21.15.16:0/4048198773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e63863ba0 0x559e6387ba30 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:29.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.740+0000 7fcfc4589740 1 --2- 172.21.15.16:0/4048198773 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e63856860 0x559e6386f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:29.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.740+0000 7fcfc4589740 1 --2- 172.21.15.16:0/4048198773 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e63855f40 0x559e63860470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:29.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.740+0000 7fcfc4589740 1 -- 172.21.15.16:0/4048198773 >> 172.21.15.16:0/4048198773 conn(0x559e638509c0 msgr2=0x559e638526a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:18:29.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.740+0000 7fcfc4589740 1 -- 172.21.15.16:0/4048198773 shutdown_connections 2024-03-28T22:18:29.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.740+0000 7fcfc4589740 1 -- 172.21.15.16:0/4048198773 wait complete. 2024-03-28T22:18:29.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.740+0000 7fcfc4589740 1 -- start start 2024-03-28T22:18:29.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.740+0000 7fcfc4589740 1 Processor -- start 2024-03-28T22:18:29.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.741+0000 7fcfc4589740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e63855f40 0x559e6386bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:29.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.741+0000 7fcfc4589740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e63856860 0x559e6385ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:29.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.741+0000 7fcfc4589740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e63863ba0 0x559e6385d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:29.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.741+0000 7fcfc4589740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcfbc1a9870 con 0x559e63856860 2024-03-28T22:18:29.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.741+0000 7fcfc4589740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcfbc1aace0 con 0x559e63855f40 2024-03-28T22:18:29.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.741+0000 7fcfc4589740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcfbc1a9f40 con 0x559e63863ba0 2024-03-28T22:18:29.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.741+0000 7fcfc4589740 10 client.0 _ll_register_callbacks cb 0x7fcfb00116f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:18:29.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.741+0000 7fcfbaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e63856860 0x559e6385ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:29.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.741+0000 7fcfbaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e63856860 0x559e6385ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53580/0 (socket says 172.21.15.16:53580) 2024-03-28T22:18:29.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.741+0000 7fcfba7fc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e63855f40 0x559e6386bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:29.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.741+0000 7fcfbaffd640 1 -- 172.21.15.16:0/2346164592 learned_addr learned my addr 172.21.15.16:0/2346164592 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:18:29.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.741+0000 7fcfba7fc640 1 --2- 172.21.15.16:0/2346164592 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e63863ba0 0x559e6385d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:29.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.741+0000 7fcfbaffd640 1 -- 172.21.15.16:0/2346164592 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e63863ba0 msgr2=0x559e6385d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:29.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.741+0000 7fcfbaffd640 1 --2- 172.21.15.16:0/2346164592 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e63863ba0 0x559e6385d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:29.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.741+0000 7fcfbaffd640 1 -- 172.21.15.16:0/2346164592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e63855f40 msgr2=0x559e6386bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:29.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.741+0000 7fcfbaffd640 1 --2- 172.21.15.16:0/2346164592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e63855f40 0x559e6386bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:29.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.741+0000 7fcfbaffd640 1 -- 172.21.15.16:0/2346164592 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e6386c3b0 con 0x559e63856860 2024-03-28T22:18:29.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.741+0000 7fcfbaffd640 1 --2- 172.21.15.16:0/2346164592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e63856860 0x559e6385ce00 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fcfac005ec0 tx=0x7fcfac02cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44576bd7fd760123 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:18:29.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.741+0000 7fcf9affd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:18:29.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.741+0000 7fcf9affd640 1 -- 172.21.15.16:0/2346164592 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfac046c30 con 0x559e63856860 2024-03-28T22:18:29.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.741+0000 7fcf9affd640 1 -- 172.21.15.16:0/2346164592 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7fcfac046dd0 con 0x559e63856860 2024-03-28T22:18:29.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.741+0000 7fcfc4589740 1 -- 172.21.15.16:0/2346164592 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfb0023040 con 0x559e63856860 2024-03-28T22:18:29.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.742+0000 7fcf9affd640 1 -- 172.21.15.16:0/2346164592 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcfac004040 con 0x559e63856860 2024-03-28T22:18:29.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.742+0000 7fcfc4589740 10 client.17136 fetch_fsmap learned FSMap version 27 2024-03-28T22:18:29.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.742+0000 7fcfc4589740 1 -- 172.21.15.16:0/2346164592 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559e638d9a60 con 0x559e63856860 2024-03-28T22:18:29.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.742+0000 7fcf9affd640 1 -- 172.21.15.16:0/2346164592 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfb0023040 con 0x559e63856860 2024-03-28T22:18:29.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.742+0000 7fcf9affd640 1 -- 172.21.15.16:0/2346164592 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7fcfac005160 con 0x559e63856860 2024-03-28T22:18:29.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.742+0000 7fcfc4589740 10 client.17136 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:18:29.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.742+0000 7fcfc4589740 10 client.17136 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:18:29.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.742+0000 7fcfc4589740 20 client.17136 populate_metadata read hostname 'smithi016' 2024-03-28T22:18:29.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.742+0000 7fcfc4589740 1 --2- 172.21.15.16:0/2346164592 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x559e638da1c0 0x559e638dc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:29.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.742+0000 7fcfc4589740 4 client.17136 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:18:29.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.742+0000 7fcfc4589740 1 -- 172.21.15.16:0/2346164592 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559e637c1a10 con 0x559e638da1c0 2024-03-28T22:18:29.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.742+0000 7fcfba7fc640 1 --2- 172.21.15.16:0/2346164592 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x559e638da1c0 0x559e638dc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:29.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.743+0000 7fcf9affd640 10 client.17136 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:18:29.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.743+0000 7fcfba7fc640 1 --2- 172.21.15.16:0/2346164592 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x559e638da1c0 0x559e638dc5e0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46bbd17d8e7f15ee server_cookie=51575a5bb67e4afb in_seq=0 out_seq=0 2024-03-28T22:18:29.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.743+0000 7fcf9affd640 1 -- 172.21.15.16:0/2346164592 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x559e637c1a10 con 0x559e638da1c0 2024-03-28T22:18:29.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.743+0000 7fcf9affd640 10 client.17136 handle_command_reply: tid=0 2024-03-28T22:18:29.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.743+0000 7fcfc4589740 1 client.17136 shutdown 2024-03-28T22:18:29.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.743+0000 7fcfc4589740 1 -- 172.21.15.16:0/2346164592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e63856860 msgr2=0x559e6385ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:29.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.743+0000 7fcfc4589740 1 --2- 172.21.15.16:0/2346164592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e63856860 0x559e6385ce00 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fcfac005ec0 tx=0x7fcfac02cf90 comp rx=0 tx=0).stop 2024-03-28T22:18:29.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.743+0000 7fcfc4589740 1 -- 172.21.15.16:0/2346164592 shutdown_connections 2024-03-28T22:18:29.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.743+0000 7fcfc4589740 1 --2- 172.21.15.16:0/2346164592 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x559e638da1c0 0x559e638dc5e0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:29.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.743+0000 7fcfc4589740 1 --2- 172.21.15.16:0/2346164592 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e63863ba0 0x559e6385d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:29.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.743+0000 7fcfc4589740 1 --2- 172.21.15.16:0/2346164592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559e63856860 0x559e6385ce00 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:29.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.743+0000 7fcfc4589740 1 --2- 172.21.15.16:0/2346164592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559e63855f40 0x559e6386bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:29.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.743+0000 7fcf9affd640 0 client.17136 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:18:29.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.743+0000 7fcfc4589740 1 -- 172.21.15.16:0/2346164592 >> 172.21.15.16:0/2346164592 conn(0x559e638509c0 msgr2=0x559e63851e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:18:29.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.743+0000 7fcfc4589740 1 -- 172.21.15.16:0/2346164592 shutdown_connections 2024-03-28T22:18:29.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.743+0000 7fcfc4589740 1 -- 172.21.15.16:0/2346164592 wait complete. 2024-03-28T22:18:29.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.743+0000 7fcfc4589740 20 client.17136 trim_cache size 0 max 16384 2024-03-28T22:18:29.754 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:18:29.755 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-03-28T22:18:29.755 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-03-28T22:18:29.755 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:18:29.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.743+0000 7fcfc19a1640 1 -- 172.21.15.16:0/2851107300 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fcfa40a1ef0 msgr2=0x7fcfa40a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:29.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.743+0000 7fcfc19a1640 1 --2- 172.21.15.16:0/2851107300 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fcfa40a1ef0 0x7fcfa40a43b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fcfb40cc090 tx=0x7fcfb4018400 comp rx=0 tx=0).stop 2024-03-28T22:18:29.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.744+0000 7fcfc19a1640 1 -- 172.21.15.16:0/2851107300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfbc10bd40 msgr2=0x7fcfbc19c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:29.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.744+0000 7fcfc19a1640 1 --2- 172.21.15.16:0/2851107300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfbc10bd40 0x7fcfbc19c810 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fcfb400cab0 tx=0x7fcfb400cf70 comp rx=0 tx=0).stop 2024-03-28T22:18:29.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.744+0000 7fcfc19a1640 1 -- 172.21.15.16:0/2851107300 shutdown_connections 2024-03-28T22:18:29.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.744+0000 7fcfc19a1640 1 --2- 172.21.15.16:0/2851107300 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fcfa40a1ef0 0x7fcfa40a43b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:29.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.744+0000 7fcfc19a1640 1 --2- 172.21.15.16:0/2851107300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfbc10bd40 0x7fcfbc19c810 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:29.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.744+0000 7fcfc19a1640 1 --2- 172.21.15.16:0/2851107300 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcfbc103290 0x7fcfbc198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:29.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.744+0000 7fcfc19a1640 1 --2- 172.21.15.16:0/2851107300 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcfbc102970 0x7fcfbc197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:29.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.744+0000 7fcfc19a1640 1 -- 172.21.15.16:0/2851107300 >> 172.21.15.16:0/2851107300 conn(0x7fcfbc077800 msgr2=0x7fcfbc0fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:18:29.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.744+0000 7fcfc19a1640 1 -- 172.21.15.16:0/2851107300 shutdown_connections 2024-03-28T22:18:29.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.744+0000 7fcfc19a1640 1 -- 172.21.15.16:0/2851107300 wait complete. 2024-03-28T22:18:29.759 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:18:29.759 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:18:29.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.861+0000 7f1fa54e1640 1 -- 172.21.15.16:0/2160219422 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fa010bd20 msgr2=0x7f1fa010e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:29.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.861+0000 7f1fa54e1640 1 --2- 172.21.15.16:0/2160219422 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fa010bd20 0x7f1fa010e170 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1f90009ca0 tx=0x7f1f9002c180 comp rx=0 tx=0).stop 2024-03-28T22:18:29.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.861+0000 7f1fa54e1640 1 -- 172.21.15.16:0/2160219422 shutdown_connections 2024-03-28T22:18:29.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.861+0000 7f1fa54e1640 1 --2- 172.21.15.16:0/2160219422 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fa010bd20 0x7f1fa010e170 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:29.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.861+0000 7f1fa54e1640 1 --2- 172.21.15.16:0/2160219422 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1fa0103270 0x7f1fa010b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:29.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.861+0000 7f1fa54e1640 1 --2- 172.21.15.16:0/2160219422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fa0102950 0x7f1fa0102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:29.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.861+0000 7f1fa54e1640 1 -- 172.21.15.16:0/2160219422 >> 172.21.15.16:0/2160219422 conn(0x7f1fa00fc0d0 msgr2=0x7f1fa00fe4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:18:29.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.862+0000 7f1fa54e1640 1 -- 172.21.15.16:0/2160219422 shutdown_connections 2024-03-28T22:18:29.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.862+0000 7f1fa54e1640 1 -- 172.21.15.16:0/2160219422 wait complete. 2024-03-28T22:18:29.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.862+0000 7f1fa54e1640 1 Processor -- start 2024-03-28T22:18:29.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.862+0000 7f1fa54e1640 1 -- start start 2024-03-28T22:18:29.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.862+0000 7f1fa54e1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1fa0102950 0x7f1fa0197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:29.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.862+0000 7f1fa54e1640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fa0103270 0x7f1fa0198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:29.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.862+0000 7f1fa54e1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fa010bd20 0x7f1fa019c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:29.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.862+0000 7f1fa54e1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1fa0111ac0 con 0x7f1fa010bd20 2024-03-28T22:18:29.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.862+0000 7f1fa54e1640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f1fa0111940 con 0x7f1fa0103270 2024-03-28T22:18:29.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.862+0000 7f1fa54e1640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1fa0111c40 con 0x7f1fa0102950 2024-03-28T22:18:29.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.863+0000 7f1f9f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fa010bd20 0x7f1fa019c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:29.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.863+0000 7f1f9f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fa010bd20 0x7f1fa019c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53590/0 (socket says 172.21.15.16:53590) 2024-03-28T22:18:29.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.863+0000 7f1f9f7fe640 1 -- 172.21.15.16:0/3996612685 learned_addr learned my addr 172.21.15.16:0/3996612685 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:18:29.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.863+0000 7f1f9e7fc640 1 --2- 172.21.15.16:0/3996612685 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fa0103270 0x7f1fa0198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:29.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.863+0000 7f1f9effd640 1 --2- 172.21.15.16:0/3996612685 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1fa0102950 0x7f1fa0197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:29.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.863+0000 7f1f9f7fe640 1 -- 172.21.15.16:0/3996612685 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1fa0102950 msgr2=0x7f1fa0197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:29.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.863+0000 7f1f9f7fe640 1 --2- 172.21.15.16:0/3996612685 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1fa0102950 0x7f1fa0197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:29.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.863+0000 7f1f9f7fe640 1 -- 172.21.15.16:0/3996612685 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fa0103270 msgr2=0x7f1fa0198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:29.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.863+0000 7f1f9f7fe640 1 --2- 172.21.15.16:0/3996612685 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fa0103270 0x7f1fa0198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:29.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.863+0000 7f1f9f7fe640 1 -- 172.21.15.16:0/3996612685 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fa019cfd0 con 0x7f1fa010bd20 2024-03-28T22:18:29.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.863+0000 7f1f9e7fc640 1 --2- 172.21.15.16:0/3996612685 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fa0103270 0x7f1fa0198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:18:29.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.863+0000 7f1f9f7fe640 1 --2- 172.21.15.16:0/3996612685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fa010bd20 0x7f1fa019c8a0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f1f9002c790 tx=0x7f1f90035ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:18:29.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.863+0000 7f1f97fff640 1 -- 172.21.15.16:0/3996612685 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f90039040 con 0x7f1fa010bd20 2024-03-28T22:18:29.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.863+0000 7f1fa54e1640 1 -- 172.21.15.16:0/3996612685 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fa019d260 con 0x7f1fa010bd20 2024-03-28T22:18:29.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.863+0000 7f1f97fff640 1 -- 172.21.15.16:0/3996612685 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f90044070 con 0x7f1fa010bd20 2024-03-28T22:18:29.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.863+0000 7f1f97fff640 1 -- 172.21.15.16:0/3996612685 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f900329f0 con 0x7f1fa010bd20 2024-03-28T22:18:29.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.863+0000 7f1fa54e1640 1 -- 172.21.15.16:0/3996612685 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fa01a4b50 con 0x7f1fa010bd20 2024-03-28T22:18:29.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.864+0000 7f1fa54e1640 1 -- 172.21.15.16:0/3996612685 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fa006a670 con 0x7f1fa010bd20 2024-03-28T22:18:29.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.865+0000 7f1f97fff640 1 -- 172.21.15.16:0/3996612685 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f1f90032b90 con 0x7f1fa010bd20 2024-03-28T22:18:29.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.866+0000 7f1f97fff640 1 --2- 172.21.15.16:0/3996612685 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f1f8c0a1ef0 0x7f1f8c0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:29.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.866+0000 7f1f9effd640 1 --2- 172.21.15.16:0/3996612685 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f1f8c0a1ef0 0x7f1f8c0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:29.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.866+0000 7f1f97fff640 1 -- 172.21.15.16:0/3996612685 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7f1f900f0bc0 con 0x7f1fa010bd20 2024-03-28T22:18:29.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.867+0000 7f1f9effd640 1 --2- 172.21.15.16:0/3996612685 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f1f8c0a1ef0 0x7f1f8c0a43b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1f980095f0 tx=0x7f1f98005f90 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:18:29.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:29.870+0000 7f1f97fff640 1 -- 172.21.15.16:0/3996612685 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1f900bdd60 con 0x7f1fa010bd20 2024-03-28T22:18:29.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:29 smithi063 ceph-mon[36249]: pgmap v244: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-28T22:18:29.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:29 smithi016 ceph-mon[32374]: pgmap v244: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-28T22:18:30.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:29 smithi053 ceph-mon[35979]: pgmap v244: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-28T22:18:30.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.095+0000 7f1fa54e1640 1 -- 172.21.15.16:0/3996612685 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1fa0199330 con 0x7f1fa010bd20 2024-03-28T22:18:30.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.096+0000 7f1f97fff640 1 -- 172.21.15.16:0/3996612685 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11851 (secure 0 0 0) 0x7f1f900bd660 con 0x7f1fa010bd20 2024-03-28T22:18:30.097 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:18:30.097 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/1644581227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1644581227},{"type":"v1","addr":"172.21.15.16:6839","nonce":1644581227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/2593062669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2593062669},{"type":"v1","addr":"172.21.15.16:6841","nonce":2593062669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:08:29.672976+0000","modified":"2024-03-28T22:14:44.922044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24566,"mds_1":24524,"mds_2":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24466":{"gid":24466,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/247249857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":247249857},{"type":"v1","addr":"172.21.15.63:6835","nonce":247249857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6837/2324231032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":2324231032},{"type":"v1","addr":"172.21.15.63:6837","nonce":2324231032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6839/1177197077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":1177197077},{"type":"v1","addr":"172.21.15.63:6839","nonce":1177197077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"e","rank":2,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.53:6835/350727634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":350727634},{"type":"v1","addr":"172.21.15.53:6835","nonce":350727634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/349877573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":349877573},{"type":"v1","addr":"172.21.15.53:6837","nonce":349877573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6839/3714153956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":3714153956},{"type":"v1","addr":"172.21.15.53:6839","nonce":3714153956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24566,"qdb_cluster":[24500,24524,24566]},"id":1}]} 2024-03-28T22:18:30.098 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2024-03-28T22:18:30.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.098+0000 7f1fa54e1640 1 -- 172.21.15.16:0/3996612685 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f1f8c0a1ef0 msgr2=0x7f1f8c0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:30.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.098+0000 7f1fa54e1640 1 --2- 172.21.15.16:0/3996612685 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f1f8c0a1ef0 0x7f1f8c0a43b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1f980095f0 tx=0x7f1f98005f90 comp rx=0 tx=0).stop 2024-03-28T22:18:30.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.098+0000 7f1fa54e1640 1 -- 172.21.15.16:0/3996612685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fa010bd20 msgr2=0x7f1fa019c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:30.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.098+0000 7f1fa54e1640 1 --2- 172.21.15.16:0/3996612685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fa010bd20 0x7f1fa019c8a0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f1f9002c790 tx=0x7f1f90035ed0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.098+0000 7f1fa54e1640 1 -- 172.21.15.16:0/3996612685 shutdown_connections 2024-03-28T22:18:30.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.098+0000 7f1fa54e1640 1 --2- 172.21.15.16:0/3996612685 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f1f8c0a1ef0 0x7f1f8c0a43b0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.098+0000 7f1fa54e1640 1 --2- 172.21.15.16:0/3996612685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fa010bd20 0x7f1fa019c8a0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.098+0000 7f1fa54e1640 1 --2- 172.21.15.16:0/3996612685 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fa0103270 0x7f1fa0198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.098+0000 7f1fa54e1640 1 --2- 172.21.15.16:0/3996612685 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1fa0102950 0x7f1fa0197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.098+0000 7f1fa54e1640 1 -- 172.21.15.16:0/3996612685 >> 172.21.15.16:0/3996612685 conn(0x7f1fa00fc0d0 msgr2=0x7f1fa00fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:18:30.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.098+0000 7f1fa54e1640 1 -- 172.21.15.16:0/3996612685 shutdown_connections 2024-03-28T22:18:30.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.098+0000 7f1fa54e1640 1 -- 172.21.15.16:0/3996612685 wait complete. 2024-03-28T22:18:30.110 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:18:30.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.214+0000 7f65b04ae640 1 -- 172.21.15.16:0/2341176519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65a81076f0 msgr2=0x7f65a8107ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:30.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.214+0000 7f65b04ae640 1 --2- 172.21.15.16:0/2341176519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65a81076f0 0x7f65a8107ad0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f65a4009890 tx=0x7f65a402bb70 comp rx=0 tx=0).stop 2024-03-28T22:18:30.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.214+0000 7f65b04ae640 1 -- 172.21.15.16:0/2341176519 shutdown_connections 2024-03-28T22:18:30.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.214+0000 7f65b04ae640 1 --2- 172.21.15.16:0/2341176519 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65a81020a0 0x7f65a810e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.214+0000 7f65b04ae640 1 --2- 172.21.15.16:0/2341176519 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65a81016e0 0x7f65a8101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.214+0000 7f65b04ae640 1 --2- 172.21.15.16:0/2341176519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65a81076f0 0x7f65a8107ad0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.214+0000 7f65b04ae640 1 -- 172.21.15.16:0/2341176519 >> 172.21.15.16:0/2341176519 conn(0x7f65a80fd1f0 msgr2=0x7f65a80ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:18:30.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.214+0000 7f65b04ae640 1 -- 172.21.15.16:0/2341176519 shutdown_connections 2024-03-28T22:18:30.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.214+0000 7f65b04ae640 1 -- 172.21.15.16:0/2341176519 wait complete. 2024-03-28T22:18:30.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.215+0000 7f65b04ae640 1 Processor -- start 2024-03-28T22:18:30.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.215+0000 7f65b04ae640 1 -- start start 2024-03-28T22:18:30.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.215+0000 7f65b04ae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65a81016e0 0x7f65a8198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.215+0000 7f65b04ae640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65a81020a0 0x7f65a8198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.215+0000 7f65b04ae640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65a81076f0 0x7f65a819c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.215+0000 7f65b04ae640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f65a8111b00 con 0x7f65a81016e0 2024-03-28T22:18:30.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.215+0000 7f65b04ae640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f65a8111980 con 0x7f65a81076f0 2024-03-28T22:18:30.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.215+0000 7f65b04ae640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f65a8111c80 con 0x7f65a81020a0 2024-03-28T22:18:30.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.215+0000 7f65ae24a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65a81016e0 0x7f65a8198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.215+0000 7f65ae24a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65a81016e0 0x7f65a8198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53616/0 (socket says 172.21.15.16:53616) 2024-03-28T22:18:30.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.215+0000 7f65ae24a640 1 -- 172.21.15.16:0/2954866258 learned_addr learned my addr 172.21.15.16:0/2954866258 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:18:30.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.215+0000 7f65aea4b640 1 --2- 172.21.15.16:0/2954866258 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65a81076f0 0x7f65a819c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.215+0000 7f65ada49640 1 --2- 172.21.15.16:0/2954866258 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65a81020a0 0x7f65a8198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.215+0000 7f65ae24a640 1 -- 172.21.15.16:0/2954866258 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65a81020a0 msgr2=0x7f65a8198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:30.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.215+0000 7f65ae24a640 1 --2- 172.21.15.16:0/2954866258 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65a81020a0 0x7f65a8198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.215+0000 7f65ae24a640 1 -- 172.21.15.16:0/2954866258 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65a81076f0 msgr2=0x7f65a819c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:30.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.215+0000 7f65ae24a640 1 --2- 172.21.15.16:0/2954866258 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65a81076f0 0x7f65a819c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.215+0000 7f65ae24a640 1 -- 172.21.15.16:0/2954866258 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65a819d030 con 0x7f65a81016e0 2024-03-28T22:18:30.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.216+0000 7f65ae24a640 1 --2- 172.21.15.16:0/2954866258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65a81016e0 0x7f65a8198040 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f65a4000c00 tx=0x7f65a4004550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:18:30.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.216+0000 7f659f7fe640 1 -- 172.21.15.16:0/2954866258 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a4041020 con 0x7f65a81016e0 2024-03-28T22:18:30.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.216+0000 7f659f7fe640 1 -- 172.21.15.16:0/2954866258 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65a402c9f0 con 0x7f65a81016e0 2024-03-28T22:18:30.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.216+0000 7f65b04ae640 1 -- 172.21.15.16:0/2954866258 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65a819d2c0 con 0x7f65a81016e0 2024-03-28T22:18:30.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.216+0000 7f659f7fe640 1 -- 172.21.15.16:0/2954866258 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a4039040 con 0x7f65a81016e0 2024-03-28T22:18:30.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.216+0000 7f65b04ae640 1 -- 172.21.15.16:0/2954866258 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65a81a4bb0 con 0x7f65a81016e0 2024-03-28T22:18:30.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.218+0000 7f659f7fe640 1 -- 172.21.15.16:0/2954866258 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f65a4044020 con 0x7f65a81016e0 2024-03-28T22:18:30.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.218+0000 7f65b316b740 1 Processor -- start 2024-03-28T22:18:30.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.219+0000 7f65b316b740 1 -- start start 2024-03-28T22:18:30.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.219+0000 7f65b316b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0b73ee470 0x55d0b73ee850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.219+0000 7f65b316b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0b73eed90 0x55d0b73f7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.219+0000 7f659f7fe640 1 --2- 172.21.15.16:0/2954866258 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f65940a1ef0 0x7f65940a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.219+0000 7f65b316b740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d0b73f7970 0x55d0b73f9d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.219+0000 7f65b316b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d0b73521b0 con 0x55d0b73eed90 2024-03-28T22:18:30.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.219+0000 7f65b316b740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55d0b72fac00 con 0x55d0b73f7970 2024-03-28T22:18:30.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.219+0000 7f65b316b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d0b734e250 con 0x55d0b73ee470 2024-03-28T22:18:30.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.219+0000 7f65ae24a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0b73eed90 0x55d0b73f7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.219+0000 7f65ada49640 1 --2- 172.21.15.16:0/2954866258 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f65940a1ef0 0x7f65940a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.219+0000 7f65aea4b640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d0b73f7970 0x55d0b73f9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.219+0000 7f65ae24a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0b73eed90 0x55d0b73f7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53622/0 (socket says 172.21.15.16:53622) 2024-03-28T22:18:30.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.219+0000 7f65ae24a640 1 -- 172.21.15.16:0/4107612385 learned_addr learned my addr 172.21.15.16:0/4107612385 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:18:30.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.219+0000 7f65aea4b640 1 --2- 172.21.15.16:0/4107612385 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0b73ee470 0x55d0b73ee850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.219+0000 7f659f7fe640 1 -- 172.21.15.16:0/2954866258 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7f65a40ef8f0 con 0x7f65a81016e0 2024-03-28T22:18:30.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.219+0000 7f65aea4b640 1 -- 172.21.15.16:0/4107612385 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0b73ee470 msgr2=0x55d0b73ee850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:30.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.219+0000 7f65aea4b640 1 --2- 172.21.15.16:0/4107612385 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0b73ee470 0x55d0b73ee850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.219+0000 7f65aea4b640 1 -- 172.21.15.16:0/4107612385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0b73eed90 msgr2=0x55d0b73f7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:30.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.219+0000 7f65aea4b640 1 --2- 172.21.15.16:0/4107612385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0b73eed90 0x55d0b73f7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.219+0000 7f65aea4b640 1 -- 172.21.15.16:0/4107612385 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0b731a3d0 con 0x55d0b73f7970 2024-03-28T22:18:30.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.220+0000 7f65ada49640 1 --2- 172.21.15.16:0/2954866258 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f65940a1ef0 0x7f65940a43b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f65a80ff910 tx=0x7f6598005f90 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:18:30.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.220+0000 7f65aea4b640 1 --2- 172.21.15.16:0/4107612385 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d0b73f7970 0x55d0b73f9d60 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f65a000eb30 tx=0x7f65a0010040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e98c83feea2d6669 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:18:30.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.220+0000 7f659cbfe640 1 -- 172.21.15.16:0/4107612385 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a0004b20 con 0x55d0b73f7970 2024-03-28T22:18:30.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.220+0000 7f659cbfe640 1 -- 172.21.15.16:0/4107612385 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65a0004cc0 con 0x55d0b73f7970 2024-03-28T22:18:30.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.220+0000 7f659cbfe640 1 -- 172.21.15.16:0/4107612385 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a0004290 con 0x55d0b73f7970 2024-03-28T22:18:30.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.221+0000 7f65b316b740 1 -- 172.21.15.16:0/4107612385 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d0b73f7970 msgr2=0x55d0b73f9d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:30.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.221+0000 7f65b316b740 1 --2- 172.21.15.16:0/4107612385 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d0b73f7970 0x55d0b73f9d60 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f65a000eb30 tx=0x7f65a0010040 comp rx=0 tx=0).stop 2024-03-28T22:18:30.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.221+0000 7f65b316b740 1 -- 172.21.15.16:0/4107612385 shutdown_connections 2024-03-28T22:18:30.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.221+0000 7f65b316b740 1 --2- 172.21.15.16:0/4107612385 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d0b73f7970 0x55d0b73f9d60 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.221+0000 7f65b316b740 1 --2- 172.21.15.16:0/4107612385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0b73eed90 0x55d0b73f7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.221+0000 7f65b316b740 1 --2- 172.21.15.16:0/4107612385 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0b73ee470 0x55d0b73ee850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.221+0000 7f65b316b740 1 -- 172.21.15.16:0/4107612385 >> 172.21.15.16:0/4107612385 conn(0x55d0b73e8ba0 msgr2=0x55d0b73e8f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:18:30.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.221+0000 7f65b316b740 1 -- 172.21.15.16:0/4107612385 shutdown_connections 2024-03-28T22:18:30.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.221+0000 7f65b316b740 1 -- 172.21.15.16:0/4107612385 wait complete. 2024-03-28T22:18:30.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.221+0000 7f65b316b740 1 -- start start 2024-03-28T22:18:30.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.221+0000 7f65b316b740 1 Processor -- start 2024-03-28T22:18:30.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.221+0000 7f65b316b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0b73ee470 0x55d0b73f4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.221+0000 7f65b316b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0b73eed90 0x55d0b73f4f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.221+0000 7f65b316b740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d0b73f7970 0x55d0b73f54c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.221+0000 7f65b316b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d0b73fa760 con 0x55d0b73ee470 2024-03-28T22:18:30.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.221+0000 7f65b316b740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55d0b72f0470 con 0x55d0b73f7970 2024-03-28T22:18:30.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.221+0000 7f65b316b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d0b73fa8e0 con 0x55d0b73eed90 2024-03-28T22:18:30.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.221+0000 7f65b316b740 10 client.0 _ll_register_callbacks cb 0x55d0b730dcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:18:30.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.221+0000 7f65ae24a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0b73ee470 0x55d0b73f4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.221+0000 7f65ae24a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0b73ee470 0x55d0b73f4a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53630/0 (socket says 172.21.15.16:53630) 2024-03-28T22:18:30.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.221+0000 7f65ae24a640 1 -- 172.21.15.16:0/710809818 learned_addr learned my addr 172.21.15.16:0/710809818 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:18:30.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.221+0000 7f65aea4b640 1 --2- 172.21.15.16:0/710809818 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d0b73f7970 0x55d0b73f54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.222+0000 7f65ada49640 1 --2- 172.21.15.16:0/710809818 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0b73eed90 0x55d0b73f4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.222+0000 7f65ae24a640 1 -- 172.21.15.16:0/710809818 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0b73eed90 msgr2=0x55d0b73f4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:30.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.222+0000 7f65ae24a640 1 --2- 172.21.15.16:0/710809818 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0b73eed90 0x55d0b73f4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.222+0000 7f65ae24a640 1 -- 172.21.15.16:0/710809818 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d0b73f7970 msgr2=0x55d0b73f54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:30.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.222+0000 7f65ae24a640 1 --2- 172.21.15.16:0/710809818 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d0b73f7970 0x55d0b73f54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.222+0000 7f65ae24a640 1 -- 172.21.15.16:0/710809818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0b70663f0 con 0x55d0b73ee470 2024-03-28T22:18:30.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.222+0000 7f65ae24a640 1 --2- 172.21.15.16:0/710809818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0b73ee470 0x55d0b73f4a40 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f65a40b8d60 tx=0x7f65a40b8760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5a7339971bf2dfb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:18:30.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.222+0000 7f6591ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:18:30.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.222+0000 7f6591ffb640 1 -- 172.21.15.16:0/710809818 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a40bca90 con 0x55d0b73ee470 2024-03-28T22:18:30.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.222+0000 7f6591ffb640 1 -- 172.21.15.16:0/710809818 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7f65a4032d80 con 0x55d0b73ee470 2024-03-28T22:18:30.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.222+0000 7f65b316b740 1 -- 172.21.15.16:0/710809818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65a0004cc0 con 0x55d0b73ee470 2024-03-28T22:18:30.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.222+0000 7f6591ffb640 1 -- 172.21.15.16:0/710809818 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f65a40bc390 con 0x55d0b73ee470 2024-03-28T22:18:30.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.222+0000 7f65b316b740 10 client.17154 fetch_fsmap learned FSMap version 27 2024-03-28T22:18:30.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.222+0000 7f65b316b740 1 -- 172.21.15.16:0/710809818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d0b71307d0 con 0x55d0b73ee470 2024-03-28T22:18:30.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.222+0000 7f6591ffb640 1 -- 172.21.15.16:0/710809818 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65a0004cc0 con 0x55d0b73ee470 2024-03-28T22:18:30.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.223+0000 7f6591ffb640 1 -- 172.21.15.16:0/710809818 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f65a40a4020 con 0x55d0b73ee470 2024-03-28T22:18:30.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.223+0000 7f65b316b740 10 client.17154 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:18:30.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.223+0000 7f65b316b740 10 client.17154 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-28T22:18:30.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.223+0000 7f65b316b740 20 client.17154 populate_metadata read hostname 'smithi016' 2024-03-28T22:18:30.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.223+0000 7f65b316b740 1 --2- 172.21.15.16:0/710809818 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55d0b7407a10 0x55d0b7409e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.223+0000 7f65b316b740 4 client.17154 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:18:30.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.223+0000 7f65b316b740 1 -- 172.21.15.16:0/710809818 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d0b735f6f0 con 0x55d0b7407a10 2024-03-28T22:18:30.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.223+0000 7f65ada49640 1 --2- 172.21.15.16:0/710809818 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55d0b7407a10 0x55d0b7409e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.223+0000 7f6591ffb640 10 client.17154 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:18:30.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.223+0000 7f65ada49640 1 --2- 172.21.15.16:0/710809818 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55d0b7407a10 0x55d0b7409e80 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=118f0c6346b0d2ed server_cookie=93082e53b01480ff in_seq=0 out_seq=0 2024-03-28T22:18:30.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.224+0000 7f6591ffb640 1 -- 172.21.15.16:0/710809818 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d0b735f6f0 con 0x55d0b7407a10 2024-03-28T22:18:30.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.224+0000 7f6591ffb640 10 client.17154 handle_command_reply: tid=0 2024-03-28T22:18:30.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.224+0000 7f65b316b740 1 client.17154 shutdown 2024-03-28T22:18:30.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.224+0000 7f65b316b740 1 -- 172.21.15.16:0/710809818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0b73ee470 msgr2=0x55d0b73f4a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:30.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.224+0000 7f65b316b740 1 --2- 172.21.15.16:0/710809818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0b73ee470 0x55d0b73f4a40 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f65a40b8d60 tx=0x7f65a40b8760 comp rx=0 tx=0).stop 2024-03-28T22:18:30.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.224+0000 7f65b316b740 1 -- 172.21.15.16:0/710809818 shutdown_connections 2024-03-28T22:18:30.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.224+0000 7f65b316b740 1 --2- 172.21.15.16:0/710809818 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55d0b7407a10 0x55d0b7409e80 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.224+0000 7f65b316b740 1 --2- 172.21.15.16:0/710809818 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d0b73f7970 0x55d0b73f54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.224+0000 7f65b316b740 1 --2- 172.21.15.16:0/710809818 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0b73eed90 0x55d0b73f4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.224+0000 7f6591ffb640 0 client.17154 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:18:30.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.224+0000 7f65b316b740 1 --2- 172.21.15.16:0/710809818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0b73ee470 0x55d0b73f4a40 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.224+0000 7f65b316b740 1 -- 172.21.15.16:0/710809818 >> 172.21.15.16:0/710809818 conn(0x55d0b73e8ba0 msgr2=0x55d0b73ea8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:18:30.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.225+0000 7f65b316b740 1 -- 172.21.15.16:0/710809818 shutdown_connections 2024-03-28T22:18:30.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.225+0000 7f65b316b740 1 -- 172.21.15.16:0/710809818 wait complete. 2024-03-28T22:18:30.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.225+0000 7f65b316b740 20 client.17154 trim_cache size 0 max 16384 2024-03-28T22:18:30.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.238+0000 7f65b316b740 1 Processor -- start 2024-03-28T22:18:30.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.238+0000 7f65b316b740 1 -- start start 2024-03-28T22:18:30.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.238+0000 7f65b316b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0b73ee470 0x55d0b7403530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.238+0000 7f65b316b740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d0b73eed90 0x55d0b740f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.238+0000 7f65b316b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0b73f7970 0x55d0b73f1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.238+0000 7f65b316b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f65a81afd70 con 0x55d0b73ee470 2024-03-28T22:18:30.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.238+0000 7f65b316b740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f65a81ad640 con 0x55d0b73eed90 2024-03-28T22:18:30.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.238+0000 7f65b316b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f65a81ab4b0 con 0x55d0b73f7970 2024-03-28T22:18:30.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.238+0000 7f65ae24a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0b73ee470 0x55d0b7403530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.238+0000 7f65ae24a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0b73ee470 0x55d0b7403530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53644/0 (socket says 172.21.15.16:53644) 2024-03-28T22:18:30.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.238+0000 7f65ae24a640 1 -- 172.21.15.16:0/4285888428 learned_addr learned my addr 172.21.15.16:0/4285888428 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:18:30.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.238+0000 7f65ada49640 1 --2- 172.21.15.16:0/4285888428 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d0b73eed90 0x55d0b740f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.238+0000 7f65ae24a640 1 -- 172.21.15.16:0/4285888428 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0b73f7970 msgr2=0x55d0b73f1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:30.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.238+0000 7f65ae24a640 1 --2- 172.21.15.16:0/4285888428 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0b73f7970 0x55d0b73f1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.238+0000 7f65ae24a640 1 -- 172.21.15.16:0/4285888428 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d0b73eed90 msgr2=0x55d0b740f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:30.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.238+0000 7f65ae24a640 1 --2- 172.21.15.16:0/4285888428 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d0b73eed90 0x55d0b740f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.238+0000 7f65ae24a640 1 -- 172.21.15.16:0/4285888428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0b73eebc0 con 0x55d0b73ee470 2024-03-28T22:18:30.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.238+0000 7f65ada49640 1 --2- 172.21.15.16:0/4285888428 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d0b73eed90 0x55d0b740f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:18:30.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.239+0000 7f65ae24a640 1 --2- 172.21.15.16:0/4285888428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0b73ee470 0x55d0b7403530 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f65a4032860 tx=0x7f65a4005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9254b223a3a60a67 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:18:30.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.239+0000 7f6593fff640 1 -- 172.21.15.16:0/4285888428 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a40bbed0 con 0x55d0b73ee470 2024-03-28T22:18:30.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.239+0000 7f6593fff640 1 -- 172.21.15.16:0/4285888428 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65a40bca90 con 0x55d0b73ee470 2024-03-28T22:18:30.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.239+0000 7f6593fff640 1 -- 172.21.15.16:0/4285888428 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a40c46a0 con 0x55d0b73ee470 2024-03-28T22:18:30.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.239+0000 7f65b316b740 1 -- 172.21.15.16:0/4285888428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0b73ee470 msgr2=0x55d0b7403530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:30.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.239+0000 7f65b316b740 1 --2- 172.21.15.16:0/4285888428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0b73ee470 0x55d0b7403530 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f65a4032860 tx=0x7f65a4005f90 comp rx=0 tx=0).stop 2024-03-28T22:18:30.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.239+0000 7f65b316b740 1 -- 172.21.15.16:0/4285888428 shutdown_connections 2024-03-28T22:18:30.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.239+0000 7f65b316b740 1 --2- 172.21.15.16:0/4285888428 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0b73f7970 0x55d0b73f1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.239+0000 7f65b316b740 1 --2- 172.21.15.16:0/4285888428 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d0b73eed90 0x55d0b740f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.239+0000 7f65b316b740 1 --2- 172.21.15.16:0/4285888428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0b73ee470 0x55d0b7403530 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.239+0000 7f65b316b740 1 -- 172.21.15.16:0/4285888428 >> 172.21.15.16:0/4285888428 conn(0x55d0b73e8ba0 msgr2=0x55d0b73e7d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:18:30.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.239+0000 7f65b316b740 1 -- 172.21.15.16:0/4285888428 shutdown_connections 2024-03-28T22:18:30.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.239+0000 7f65b316b740 1 -- 172.21.15.16:0/4285888428 wait complete. 2024-03-28T22:18:30.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.240+0000 7f65b316b740 1 -- start start 2024-03-28T22:18:30.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.240+0000 7f65b316b740 1 Processor -- start 2024-03-28T22:18:30.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.240+0000 7f65b316b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0b73ee470 0x55d0b73ffc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.240+0000 7f65b316b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0b73eed90 0x55d0b73fc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.240+0000 7f65b316b740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d0b73f7970 0x55d0b73fc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.240+0000 7f65b316b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f65a81abb00 con 0x55d0b73eed90 2024-03-28T22:18:30.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.240+0000 7f65b316b740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f65a81acf70 con 0x55d0b73f7970 2024-03-28T22:18:30.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.240+0000 7f65b316b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f65a81ac1d0 con 0x55d0b73ee470 2024-03-28T22:18:30.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.240+0000 7f65b316b740 10 client.0 _ll_register_callbacks cb 0x7f65a4005ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:18:30.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.240+0000 7f65ada49640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0b73eed90 0x55d0b73fc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.240+0000 7f65ada49640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0b73eed90 0x55d0b73fc0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53650/0 (socket says 172.21.15.16:53650) 2024-03-28T22:18:30.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.240+0000 7f65ada49640 1 -- 172.21.15.16:0/1043809961 learned_addr learned my addr 172.21.15.16:0/1043809961 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:18:30.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.240+0000 7f65aea4b640 1 --2- 172.21.15.16:0/1043809961 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d0b73f7970 0x55d0b73fc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.240+0000 7f65ae24a640 1 --2- 172.21.15.16:0/1043809961 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0b73ee470 0x55d0b73ffc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.240+0000 7f65ada49640 1 -- 172.21.15.16:0/1043809961 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0b73ee470 msgr2=0x55d0b73ffc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:30.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.240+0000 7f65ada49640 1 --2- 172.21.15.16:0/1043809961 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0b73ee470 0x55d0b73ffc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.240+0000 7f65ada49640 1 -- 172.21.15.16:0/1043809961 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d0b73f7970 msgr2=0x55d0b73fc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:30.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.240+0000 7f65ada49640 1 --2- 172.21.15.16:0/1043809961 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d0b73f7970 0x55d0b73fc5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.240+0000 7f65ada49640 1 -- 172.21.15.16:0/1043809961 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0b73fcc50 con 0x55d0b73eed90 2024-03-28T22:18:30.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.240+0000 7f65ae24a640 1 --2- 172.21.15.16:0/1043809961 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0b73ee470 0x55d0b73ffc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:18:30.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.240+0000 7f65ada49640 1 --2- 172.21.15.16:0/1043809961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0b73eed90 0x55d0b73fc0a0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f6598009460 tx=0x7f6598004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=867f1188e6b6821b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:18:30.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.240+0000 7f6591ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:18:30.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.241+0000 7f6591ffb640 1 -- 172.21.15.16:0/1043809961 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f659801fe10 con 0x55d0b73eed90 2024-03-28T22:18:30.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.241+0000 7f6591ffb640 1 -- 172.21.15.16:0/1043809961 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7f6598028510 con 0x55d0b73eed90 2024-03-28T22:18:30.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.241+0000 7f65b316b740 1 -- 172.21.15.16:0/1043809961 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65a40bca90 con 0x55d0b73eed90 2024-03-28T22:18:30.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.241+0000 7f6591ffb640 1 -- 172.21.15.16:0/1043809961 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6598028710 con 0x55d0b73eed90 2024-03-28T22:18:30.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.241+0000 7f65b316b740 10 client.17160 fetch_fsmap learned FSMap version 27 2024-03-28T22:18:30.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.241+0000 7f65b316b740 1 -- 172.21.15.16:0/1043809961 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d0b746ec60 con 0x55d0b73eed90 2024-03-28T22:18:30.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.241+0000 7f6591ffb640 1 -- 172.21.15.16:0/1043809961 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65a40bca90 con 0x55d0b73eed90 2024-03-28T22:18:30.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.241+0000 7f6591ffb640 1 -- 172.21.15.16:0/1043809961 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f65980316c0 con 0x55d0b73eed90 2024-03-28T22:18:30.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.241+0000 7f65b316b740 10 client.17160 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:18:30.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.241+0000 7f65b316b740 10 client.17160 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-28T22:18:30.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.241+0000 7f65b316b740 20 client.17160 populate_metadata read hostname 'smithi016' 2024-03-28T22:18:30.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.241+0000 7f65b316b740 1 --2- 172.21.15.16:0/1043809961 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55d0b746f3c0 0x55d0b74717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.241+0000 7f65b316b740 4 client.17160 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:18:30.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.241+0000 7f65b316b740 1 -- 172.21.15.16:0/1043809961 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d0b7359a40 con 0x55d0b746f3c0 2024-03-28T22:18:30.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.241+0000 7f65ae24a640 1 --2- 172.21.15.16:0/1043809961 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55d0b746f3c0 0x55d0b74717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.242+0000 7f65ae24a640 1 --2- 172.21.15.16:0/1043809961 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55d0b746f3c0 0x55d0b74717e0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=99efb758aa1ff42c server_cookie=105bdabf3b7d4aa4 in_seq=0 out_seq=0 2024-03-28T22:18:30.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.242+0000 7f6591ffb640 10 client.17160 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:18:30.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.242+0000 7f6591ffb640 1 -- 172.21.15.16:0/1043809961 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d0b7359a40 con 0x55d0b746f3c0 2024-03-28T22:18:30.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.242+0000 7f6591ffb640 10 client.17160 handle_command_reply: tid=0 2024-03-28T22:18:30.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.242+0000 7f65b316b740 1 client.17160 shutdown 2024-03-28T22:18:30.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.242+0000 7f65b316b740 1 -- 172.21.15.16:0/1043809961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0b73eed90 msgr2=0x55d0b73fc0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:30.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.242+0000 7f65b316b740 1 --2- 172.21.15.16:0/1043809961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0b73eed90 0x55d0b73fc0a0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f6598009460 tx=0x7f6598004730 comp rx=0 tx=0).stop 2024-03-28T22:18:30.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.242+0000 7f65b316b740 1 -- 172.21.15.16:0/1043809961 shutdown_connections 2024-03-28T22:18:30.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.242+0000 7f65b316b740 1 --2- 172.21.15.16:0/1043809961 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55d0b746f3c0 0x55d0b74717e0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.242+0000 7f65b316b740 1 --2- 172.21.15.16:0/1043809961 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d0b73f7970 0x55d0b73fc5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.242+0000 7f65b316b740 1 --2- 172.21.15.16:0/1043809961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0b73eed90 0x55d0b73fc0a0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.242+0000 7f6591ffb640 0 client.17160 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:18:30.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.242+0000 7f65b316b740 1 --2- 172.21.15.16:0/1043809961 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0b73ee470 0x55d0b73ffc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.242+0000 7f65b316b740 1 -- 172.21.15.16:0/1043809961 >> 172.21.15.16:0/1043809961 conn(0x55d0b73e8ba0 msgr2=0x55d0b73e9fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:18:30.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.242+0000 7f65b316b740 1 -- 172.21.15.16:0/1043809961 shutdown_connections 2024-03-28T22:18:30.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.242+0000 7f65b316b740 1 -- 172.21.15.16:0/1043809961 wait complete. 2024-03-28T22:18:30.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.242+0000 7f65b316b740 20 client.17160 trim_cache size 0 max 16384 2024-03-28T22:18:30.254 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:18:30.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.243+0000 7f65b04ae640 1 -- 172.21.15.16:0/2954866258 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f65940a1ef0 msgr2=0x7f65940a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:30.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.243+0000 7f65b04ae640 1 --2- 172.21.15.16:0/2954866258 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f65940a1ef0 0x7f65940a43b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f65a80ff910 tx=0x7f6598005f90 comp rx=0 tx=0).stop 2024-03-28T22:18:30.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.243+0000 7f65b04ae640 1 -- 172.21.15.16:0/2954866258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65a81016e0 msgr2=0x7f65a8198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:30.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.243+0000 7f65b04ae640 1 --2- 172.21.15.16:0/2954866258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65a81016e0 0x7f65a8198040 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f65a4000c00 tx=0x7f65a4004550 comp rx=0 tx=0).stop 2024-03-28T22:18:30.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.243+0000 7f65b04ae640 1 -- 172.21.15.16:0/2954866258 shutdown_connections 2024-03-28T22:18:30.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.243+0000 7f65b04ae640 1 --2- 172.21.15.16:0/2954866258 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f65940a1ef0 0x7f65940a43b0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.243+0000 7f65b04ae640 1 --2- 172.21.15.16:0/2954866258 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65a81076f0 0x7f65a819c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.243+0000 7f65b04ae640 1 --2- 172.21.15.16:0/2954866258 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65a81020a0 0x7f65a8198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.243+0000 7f65b04ae640 1 --2- 172.21.15.16:0/2954866258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65a81016e0 0x7f65a8198040 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.243+0000 7f65b04ae640 1 -- 172.21.15.16:0/2954866258 >> 172.21.15.16:0/2954866258 conn(0x7f65a80fd1f0 msgr2=0x7f65a80fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:18:30.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.243+0000 7f65b04ae640 1 -- 172.21.15.16:0/2954866258 shutdown_connections 2024-03-28T22:18:30.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.243+0000 7f65b04ae640 1 -- 172.21.15.16:0/2954866258 wait complete. 2024-03-28T22:18:30.257 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:18:30.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.361+0000 7fef3c484640 1 -- 172.21.15.16:0/3729027082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef341005f0 msgr2=0x7fef34100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:30.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.361+0000 7fef3c484640 1 --2- 172.21.15.16:0/3729027082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef341005f0 0x7fef34100a70 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fef24009930 tx=0x7fef2402bc30 comp rx=0 tx=0).stop 2024-03-28T22:18:30.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.361+0000 7fef3c484640 1 -- 172.21.15.16:0/3729027082 shutdown_connections 2024-03-28T22:18:30.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.361+0000 7fef3c484640 1 --2- 172.21.15.16:0/3729027082 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef34100fb0 0x7fef3410d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.361+0000 7fef3c484640 1 --2- 172.21.15.16:0/3729027082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef341005f0 0x7fef34100a70 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.361+0000 7fef3c484640 1 --2- 172.21.15.16:0/3729027082 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef34106690 0x7fef34106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.361+0000 7fef3c484640 1 -- 172.21.15.16:0/3729027082 >> 172.21.15.16:0/3729027082 conn(0x7fef34077800 msgr2=0x7fef34077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:18:30.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.361+0000 7fef3c484640 1 -- 172.21.15.16:0/3729027082 shutdown_connections 2024-03-28T22:18:30.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.361+0000 7fef3c484640 1 -- 172.21.15.16:0/3729027082 wait complete. 2024-03-28T22:18:30.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.361+0000 7fef3c484640 1 Processor -- start 2024-03-28T22:18:30.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.361+0000 7fef3c484640 1 -- start start 2024-03-28T22:18:30.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.362+0000 7fef3c484640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef341005f0 0x7fef34198e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.362+0000 7fef3c484640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef34100fb0 0x7fef34199390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.362+0000 7fef3c484640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef34106690 0x7fef341a3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.362+0000 7fef3c484640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fef341109d0 con 0x7fef34100fb0 2024-03-28T22:18:30.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.362+0000 7fef3c484640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fef34110850 con 0x7fef341005f0 2024-03-28T22:18:30.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.362+0000 7fef3c484640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fef34110b50 con 0x7fef34106690 2024-03-28T22:18:30.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.362+0000 7fef39a1f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef34100fb0 0x7fef34199390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.362+0000 7fef39a1f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef34100fb0 0x7fef34199390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53668/0 (socket says 172.21.15.16:53668) 2024-03-28T22:18:30.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.362+0000 7fef39a1f640 1 -- 172.21.15.16:0/3653622871 learned_addr learned my addr 172.21.15.16:0/3653622871 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:18:30.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.362+0000 7fef3a220640 1 --2- 172.21.15.16:0/3653622871 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef341005f0 0x7fef34198e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.362+0000 7fef3aa21640 1 --2- 172.21.15.16:0/3653622871 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef34106690 0x7fef341a3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.362+0000 7fef39a1f640 1 -- 172.21.15.16:0/3653622871 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef34106690 msgr2=0x7fef341a3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:30.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.362+0000 7fef39a1f640 1 --2- 172.21.15.16:0/3653622871 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef34106690 0x7fef341a3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.362+0000 7fef39a1f640 1 -- 172.21.15.16:0/3653622871 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef341005f0 msgr2=0x7fef34198e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:30.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.362+0000 7fef39a1f640 1 --2- 172.21.15.16:0/3653622871 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef341005f0 0x7fef34198e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.362+0000 7fef39a1f640 1 -- 172.21.15.16:0/3653622871 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef341a4410 con 0x7fef34100fb0 2024-03-28T22:18:30.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.362+0000 7fef3aa21640 1 --2- 172.21.15.16:0/3653622871 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef34106690 0x7fef341a3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:18:30.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.362+0000 7fef39a1f640 1 --2- 172.21.15.16:0/3653622871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef34100fb0 0x7fef34199390 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fef24009230 tx=0x7fef2402cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:18:30.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.362+0000 7fef2b7fe640 1 -- 172.21.15.16:0/3653622871 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef24041070 con 0x7fef34100fb0 2024-03-28T22:18:30.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.362+0000 7fef2b7fe640 1 -- 172.21.15.16:0/3653622871 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef24004450 con 0x7fef34100fb0 2024-03-28T22:18:30.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.362+0000 7fef3c484640 1 -- 172.21.15.16:0/3653622871 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef341a45e0 con 0x7fef34100fb0 2024-03-28T22:18:30.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.362+0000 7fef2b7fe640 1 -- 172.21.15.16:0/3653622871 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef24003e50 con 0x7fef34100fb0 2024-03-28T22:18:30.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.363+0000 7fef3c484640 1 -- 172.21.15.16:0/3653622871 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef341a49e0 con 0x7fef34100fb0 2024-03-28T22:18:30.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.365+0000 7fef2b7fe640 1 -- 172.21.15.16:0/3653622871 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7fef24044070 con 0x7fef34100fb0 2024-03-28T22:18:30.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.366+0000 7fef2b7fe640 1 --2- 172.21.15.16:0/3653622871 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fef200a2090 0x7fef200a4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.366+0000 7fef3f067740 1 Processor -- start 2024-03-28T22:18:30.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.366+0000 7fef3a220640 1 --2- 172.21.15.16:0/3653622871 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fef200a2090 0x7fef200a4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.366+0000 7fef3f067740 1 -- start start 2024-03-28T22:18:30.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.366+0000 7fef2b7fe640 1 -- 172.21.15.16:0/3653622871 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7fef240efa60 con 0x7fef34100fb0 2024-03-28T22:18:30.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.366+0000 7fef3f067740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570d0ad8470 0x5570d0ad8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.366+0000 7fef3f067740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570d0ad8d90 0x5570d0ae1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.366+0000 7fef3f067740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570d0ae1970 0x5570d0ae3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.366+0000 7fef3f067740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5570d0a3c1b0 con 0x5570d0ad8d90 2024-03-28T22:18:30.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.366+0000 7fef3f067740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5570d09e4c00 con 0x5570d0ad8470 2024-03-28T22:18:30.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.366+0000 7fef3f067740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5570d0a38250 con 0x5570d0ae1970 2024-03-28T22:18:30.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.366+0000 7fef3a220640 1 --2- 172.21.15.16:0/3653622871 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fef200a2090 0x7fef200a4550 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fef34101730 tx=0x7fef30005f90 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:18:30.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.366+0000 7fef39a1f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570d0ad8d90 0x5570d0ae1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.366+0000 7fef3a220640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570d0ad8470 0x5570d0ad8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.366+0000 7fef39a1f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570d0ad8d90 0x5570d0ae1430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53670/0 (socket says 172.21.15.16:53670) 2024-03-28T22:18:30.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.366+0000 7fef3a220640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570d0ad8470 0x5570d0ad8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:47602/0 (socket says 172.21.15.16:47602) 2024-03-28T22:18:30.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.366+0000 7fef39a1f640 1 -- 172.21.15.16:0/3792924415 learned_addr learned my addr 172.21.15.16:0/3792924415 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:18:30.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.366+0000 7fef3aa21640 1 --2- 172.21.15.16:0/3792924415 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570d0ae1970 0x5570d0ae3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.366+0000 7fef39a1f640 1 -- 172.21.15.16:0/3792924415 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570d0ae1970 msgr2=0x5570d0ae3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:30.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.366+0000 7fef39a1f640 1 --2- 172.21.15.16:0/3792924415 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570d0ae1970 0x5570d0ae3d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.366+0000 7fef39a1f640 1 -- 172.21.15.16:0/3792924415 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570d0ad8470 msgr2=0x5570d0ad8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:30.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.366+0000 7fef39a1f640 1 --2- 172.21.15.16:0/3792924415 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570d0ad8470 0x5570d0ad8850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.366+0000 7fef39a1f640 1 -- 172.21.15.16:0/3792924415 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570d0a043d0 con 0x5570d0ad8d90 2024-03-28T22:18:30.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.367+0000 7fef39a1f640 1 --2- 172.21.15.16:0/3792924415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570d0ad8d90 0x5570d0ae1430 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fef240b89a0 tx=0x7fef24009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d883d7dc0d95682c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:18:30.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.367+0000 7fef28bfe640 1 -- 172.21.15.16:0/3792924415 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef2404b040 con 0x5570d0ad8d90 2024-03-28T22:18:30.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.367+0000 7fef28bfe640 1 -- 172.21.15.16:0/3792924415 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef24004c80 con 0x5570d0ad8d90 2024-03-28T22:18:30.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.367+0000 7fef28bfe640 1 -- 172.21.15.16:0/3792924415 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef240c4ce0 con 0x5570d0ad8d90 2024-03-28T22:18:30.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.367+0000 7fef3f067740 1 -- 172.21.15.16:0/3792924415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570d0ad8d90 msgr2=0x5570d0ae1430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:30.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.367+0000 7fef3f067740 1 --2- 172.21.15.16:0/3792924415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570d0ad8d90 0x5570d0ae1430 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fef240b89a0 tx=0x7fef24009260 comp rx=0 tx=0).stop 2024-03-28T22:18:30.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.367+0000 7fef3f067740 1 -- 172.21.15.16:0/3792924415 shutdown_connections 2024-03-28T22:18:30.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.367+0000 7fef3f067740 1 --2- 172.21.15.16:0/3792924415 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570d0ae1970 0x5570d0ae3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.368+0000 7fef3f067740 1 --2- 172.21.15.16:0/3792924415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570d0ad8d90 0x5570d0ae1430 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.368+0000 7fef3f067740 1 --2- 172.21.15.16:0/3792924415 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570d0ad8470 0x5570d0ad8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.368+0000 7fef3f067740 1 -- 172.21.15.16:0/3792924415 >> 172.21.15.16:0/3792924415 conn(0x5570d0ad2ba0 msgr2=0x5570d0ad2f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:18:30.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.368+0000 7fef3f067740 1 -- 172.21.15.16:0/3792924415 shutdown_connections 2024-03-28T22:18:30.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.368+0000 7fef3f067740 1 -- 172.21.15.16:0/3792924415 wait complete. 2024-03-28T22:18:30.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.368+0000 7fef3f067740 1 -- start start 2024-03-28T22:18:30.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.368+0000 7fef3f067740 1 Processor -- start 2024-03-28T22:18:30.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.368+0000 7fef3f067740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570d0ad8470 0x5570d0adea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.368+0000 7fef3f067740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570d0ad8d90 0x5570d0adefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.368+0000 7fef3f067740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570d0ae1970 0x5570d0adf4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.368+0000 7fef3f067740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5570d0ae4520 con 0x5570d0ad8d90 2024-03-28T22:18:30.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.368+0000 7fef3f067740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5570d09da470 con 0x5570d0ae1970 2024-03-28T22:18:30.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.368+0000 7fef3f067740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5570d0ae46a0 con 0x5570d0ad8470 2024-03-28T22:18:30.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.368+0000 7fef3f067740 10 client.0 _ll_register_callbacks cb 0x5570d09f7cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:18:30.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.368+0000 7fef39a1f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570d0ad8d90 0x5570d0adefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.368+0000 7fef39a1f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570d0ad8d90 0x5570d0adefa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53674/0 (socket says 172.21.15.16:53674) 2024-03-28T22:18:30.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.368+0000 7fef39a1f640 1 -- 172.21.15.16:0/3511268124 learned_addr learned my addr 172.21.15.16:0/3511268124 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:18:30.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.368+0000 7fef3aa21640 1 --2- 172.21.15.16:0/3511268124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570d0ae1970 0x5570d0adf4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.368+0000 7fef3a220640 1 --2- 172.21.15.16:0/3511268124 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570d0ad8470 0x5570d0adea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.369+0000 7fef39a1f640 1 -- 172.21.15.16:0/3511268124 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570d0ad8470 msgr2=0x5570d0adea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:30.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.369+0000 7fef39a1f640 1 --2- 172.21.15.16:0/3511268124 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570d0ad8470 0x5570d0adea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.369+0000 7fef39a1f640 1 -- 172.21.15.16:0/3511268124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570d0ae1970 msgr2=0x5570d0adf4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:30.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.369+0000 7fef39a1f640 1 --2- 172.21.15.16:0/3511268124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570d0ae1970 0x5570d0adf4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.369+0000 7fef39a1f640 1 -- 172.21.15.16:0/3511268124 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570d07503f0 con 0x5570d0ad8d90 2024-03-28T22:18:30.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.369+0000 7fef39a1f640 1 --2- 172.21.15.16:0/3511268124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570d0ad8d90 0x5570d0adefa0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fef240b8950 tx=0x7fef240ed290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aa81c0dfaf50c1b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:18:30.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.369+0000 7fef1dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:18:30.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.369+0000 7fef1dffb640 1 -- 172.21.15.16:0/3511268124 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef2404b030 con 0x5570d0ad8d90 2024-03-28T22:18:30.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.369+0000 7fef1dffb640 1 -- 172.21.15.16:0/3511268124 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7fef240ed410 con 0x5570d0ad8d90 2024-03-28T22:18:30.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.369+0000 7fef3f067740 1 -- 172.21.15.16:0/3511268124 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef24004c80 con 0x5570d0ad8d90 2024-03-28T22:18:30.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.369+0000 7fef1dffb640 1 -- 172.21.15.16:0/3511268124 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fef240c4ce0 con 0x5570d0ad8d90 2024-03-28T22:18:30.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.369+0000 7fef3f067740 10 client.17172 fetch_fsmap learned FSMap version 27 2024-03-28T22:18:30.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.369+0000 7fef3f067740 1 -- 172.21.15.16:0/3511268124 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5570d081a7d0 con 0x5570d0ad8d90 2024-03-28T22:18:30.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.369+0000 7fef1dffb640 1 -- 172.21.15.16:0/3511268124 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef24004c80 con 0x5570d0ad8d90 2024-03-28T22:18:30.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.369+0000 7fef1dffb640 1 -- 172.21.15.16:0/3511268124 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7fef240bb990 con 0x5570d0ad8d90 2024-03-28T22:18:30.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.370+0000 7fef3f067740 10 client.17172 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:18:30.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.370+0000 7fef3f067740 10 client.17172 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T22:18:30.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.370+0000 7fef3f067740 20 client.17172 populate_metadata read hostname 'smithi016' 2024-03-28T22:18:30.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.370+0000 7fef3f067740 1 --2- 172.21.15.16:0/3511268124 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5570d0af1560 0x5570d0af3950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.370+0000 7fef3f067740 4 client.17172 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:18:30.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.370+0000 7fef3f067740 1 -- 172.21.15.16:0/3511268124 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5570d0a496f0 con 0x5570d0af1560 2024-03-28T22:18:30.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.370+0000 7fef3a220640 1 --2- 172.21.15.16:0/3511268124 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5570d0af1560 0x5570d0af3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.370+0000 7fef3a220640 1 --2- 172.21.15.16:0/3511268124 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5570d0af1560 0x5570d0af3950 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b9a13e032b5a831d server_cookie=9e8275d014022de8 in_seq=0 out_seq=0 2024-03-28T22:18:30.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.370+0000 7fef1dffb640 10 client.17172 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:18:30.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.371+0000 7fef1dffb640 1 -- 172.21.15.16:0/3511268124 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5570d0a496f0 con 0x5570d0af1560 2024-03-28T22:18:30.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.371+0000 7fef1dffb640 10 client.17172 handle_command_reply: tid=0 2024-03-28T22:18:30.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.371+0000 7fef3f067740 1 client.17172 shutdown 2024-03-28T22:18:30.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.371+0000 7fef3f067740 1 -- 172.21.15.16:0/3511268124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570d0ad8d90 msgr2=0x5570d0adefa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:30.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.371+0000 7fef3f067740 1 --2- 172.21.15.16:0/3511268124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570d0ad8d90 0x5570d0adefa0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fef240b8950 tx=0x7fef240ed290 comp rx=0 tx=0).stop 2024-03-28T22:18:30.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.371+0000 7fef3f067740 1 -- 172.21.15.16:0/3511268124 shutdown_connections 2024-03-28T22:18:30.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.371+0000 7fef3f067740 1 --2- 172.21.15.16:0/3511268124 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5570d0af1560 0x5570d0af3950 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.371+0000 7fef3f067740 1 --2- 172.21.15.16:0/3511268124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570d0ae1970 0x5570d0adf4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.371+0000 7fef1dffb640 0 client.17172 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:18:30.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.371+0000 7fef3f067740 1 --2- 172.21.15.16:0/3511268124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570d0ad8d90 0x5570d0adefa0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.371+0000 7fef3f067740 1 --2- 172.21.15.16:0/3511268124 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570d0ad8470 0x5570d0adea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.371+0000 7fef3f067740 1 -- 172.21.15.16:0/3511268124 >> 172.21.15.16:0/3511268124 conn(0x5570d0ad2ba0 msgr2=0x5570d0ad48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:18:30.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.371+0000 7fef3f067740 1 -- 172.21.15.16:0/3511268124 shutdown_connections 2024-03-28T22:18:30.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.371+0000 7fef3f067740 1 -- 172.21.15.16:0/3511268124 wait complete. 2024-03-28T22:18:30.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.371+0000 7fef3f067740 20 client.17172 trim_cache size 0 max 16384 2024-03-28T22:18:30.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.385+0000 7fef3f067740 1 Processor -- start 2024-03-28T22:18:30.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.385+0000 7fef3f067740 1 -- start start 2024-03-28T22:18:30.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.385+0000 7fef3f067740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570d0ad8470 0x5570d0ae1190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.385+0000 7fef3f067740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570d0ad8d90 0x5570d0adf300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.385+0000 7fef3f067740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570d0ae1970 0x5570d0af8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.385+0000 7fef3f067740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fef341afcb0 con 0x5570d0ae1970 2024-03-28T22:18:30.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.385+0000 7fef3f067740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fef341ad580 con 0x5570d0ad8470 2024-03-28T22:18:30.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.385+0000 7fef3f067740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fef341ab3f0 con 0x5570d0ad8d90 2024-03-28T22:18:30.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.385+0000 7fef3aa21640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570d0ae1970 0x5570d0af8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.385+0000 7fef3aa21640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570d0ae1970 0x5570d0af8fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53676/0 (socket says 172.21.15.16:53676) 2024-03-28T22:18:30.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.385+0000 7fef3aa21640 1 -- 172.21.15.16:0/2955904776 learned_addr learned my addr 172.21.15.16:0/2955904776 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:18:30.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.385+0000 7fef3a220640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570d0ad8470 0x5570d0ae1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.386+0000 7fef39a1f640 1 --2- 172.21.15.16:0/2955904776 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570d0ad8d90 0x5570d0adf300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.386+0000 7fef3aa21640 1 -- 172.21.15.16:0/2955904776 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570d0ad8d90 msgr2=0x5570d0adf300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:30.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.386+0000 7fef3aa21640 1 --2- 172.21.15.16:0/2955904776 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570d0ad8d90 0x5570d0adf300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.386+0000 7fef3aa21640 1 -- 172.21.15.16:0/2955904776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570d0ad8470 msgr2=0x5570d0ae1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:30.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.386+0000 7fef3aa21640 1 --2- 172.21.15.16:0/2955904776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570d0ad8470 0x5570d0ae1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.386+0000 7fef3aa21640 1 -- 172.21.15.16:0/2955904776 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570d0ad26d0 con 0x5570d0ae1970 2024-03-28T22:18:30.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.386+0000 7fef3a220640 1 --2- 172.21.15.16:0/2955904776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570d0ad8470 0x5570d0ae1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:18:30.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.386+0000 7fef39a1f640 1 --2- 172.21.15.16:0/2955904776 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570d0ad8d90 0x5570d0adf300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:18:30.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.386+0000 7fef3aa21640 1 --2- 172.21.15.16:0/2955904776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570d0ae1970 0x5570d0af8fe0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fef2c003aa0 tx=0x7fef2c003ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4916c0e1c572c74 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:18:30.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.386+0000 7fef1ffff640 1 -- 172.21.15.16:0/2955904776 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef2c004830 con 0x5570d0ae1970 2024-03-28T22:18:30.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.386+0000 7fef1ffff640 1 -- 172.21.15.16:0/2955904776 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef2c012930 con 0x5570d0ae1970 2024-03-28T22:18:30.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.386+0000 7fef1ffff640 1 -- 172.21.15.16:0/2955904776 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef2c00eb30 con 0x5570d0ae1970 2024-03-28T22:18:30.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.387+0000 7fef3f067740 1 -- 172.21.15.16:0/2955904776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570d0ae1970 msgr2=0x5570d0af8fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:30.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.387+0000 7fef3f067740 1 --2- 172.21.15.16:0/2955904776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570d0ae1970 0x5570d0af8fe0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fef2c003aa0 tx=0x7fef2c003ea0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.387+0000 7fef3f067740 1 -- 172.21.15.16:0/2955904776 shutdown_connections 2024-03-28T22:18:30.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.387+0000 7fef3f067740 1 --2- 172.21.15.16:0/2955904776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570d0ae1970 0x5570d0af8fe0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.387+0000 7fef3f067740 1 --2- 172.21.15.16:0/2955904776 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570d0ad8d90 0x5570d0adf300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.387+0000 7fef3f067740 1 --2- 172.21.15.16:0/2955904776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570d0ad8470 0x5570d0ae1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.387+0000 7fef3f067740 1 -- 172.21.15.16:0/2955904776 >> 172.21.15.16:0/2955904776 conn(0x5570d0ad2ba0 msgr2=0x5570d0ad3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:18:30.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.387+0000 7fef3f067740 1 -- 172.21.15.16:0/2955904776 shutdown_connections 2024-03-28T22:18:30.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.387+0000 7fef3f067740 1 -- 172.21.15.16:0/2955904776 wait complete. 2024-03-28T22:18:30.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.387+0000 7fef3f067740 1 -- start start 2024-03-28T22:18:30.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.387+0000 7fef3f067740 1 Processor -- start 2024-03-28T22:18:30.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.387+0000 7fef3f067740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570d0ad8470 0x5570d0adbe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.387+0000 7fef3f067740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570d0ad8d90 0x5570d0adc360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.387+0000 7fef3f067740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570d0ae1970 0x5570d0b56610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.387+0000 7fef3f067740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fef341aba40 con 0x5570d0ad8d90 2024-03-28T22:18:30.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.387+0000 7fef3f067740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fef341aceb0 con 0x5570d0ad8470 2024-03-28T22:18:30.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.387+0000 7fef3f067740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fef341ac110 con 0x5570d0ae1970 2024-03-28T22:18:30.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.388+0000 7fef3f067740 10 client.0 _ll_register_callbacks cb 0x7fef24005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:18:30.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.388+0000 7fef39a1f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570d0ad8d90 0x5570d0adc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.388+0000 7fef3a220640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570d0ad8470 0x5570d0adbe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.388+0000 7fef39a1f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570d0ad8d90 0x5570d0adc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53678/0 (socket says 172.21.15.16:53678) 2024-03-28T22:18:30.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.388+0000 7fef3a220640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570d0ad8470 0x5570d0adbe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:47616/0 (socket says 172.21.15.16:47616) 2024-03-28T22:18:30.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.388+0000 7fef39a1f640 1 -- 172.21.15.16:0/2964604514 learned_addr learned my addr 172.21.15.16:0/2964604514 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:18:30.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.388+0000 7fef3aa21640 1 --2- 172.21.15.16:0/2964604514 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570d0ae1970 0x5570d0b56610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.388+0000 7fef39a1f640 1 -- 172.21.15.16:0/2964604514 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570d0ae1970 msgr2=0x5570d0b56610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:30.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.388+0000 7fef39a1f640 1 --2- 172.21.15.16:0/2964604514 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570d0ae1970 0x5570d0b56610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.388+0000 7fef39a1f640 1 -- 172.21.15.16:0/2964604514 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570d0ad8470 msgr2=0x5570d0adbe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:30.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.388+0000 7fef39a1f640 1 --2- 172.21.15.16:0/2964604514 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570d0ad8470 0x5570d0adbe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.388+0000 7fef39a1f640 1 -- 172.21.15.16:0/2964604514 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570d0b56c00 con 0x5570d0ad8d90 2024-03-28T22:18:30.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.388+0000 7fef39a1f640 1 --2- 172.21.15.16:0/2964604514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570d0ad8d90 0x5570d0adc360 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fef240ba7f0 tx=0x7fef240b8680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4cf5a099f064e06 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:18:30.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.388+0000 7fef1dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:18:30.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.388+0000 7fef1dffb640 1 -- 172.21.15.16:0/2964604514 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef2404b020 con 0x5570d0ad8d90 2024-03-28T22:18:30.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.388+0000 7fef1dffb640 1 -- 172.21.15.16:0/2964604514 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7fef240bb990 con 0x5570d0ad8d90 2024-03-28T22:18:30.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.388+0000 7fef3f067740 1 -- 172.21.15.16:0/2964604514 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef2c012930 con 0x5570d0ad8d90 2024-03-28T22:18:30.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.388+0000 7fef1dffb640 1 -- 172.21.15.16:0/2964604514 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fef24004c80 con 0x5570d0ad8d90 2024-03-28T22:18:30.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.388+0000 7fef3f067740 10 client.17178 fetch_fsmap learned FSMap version 27 2024-03-28T22:18:30.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.388+0000 7fef3f067740 1 -- 172.21.15.16:0/2964604514 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5570d0b56c00 con 0x5570d0ad8d90 2024-03-28T22:18:30.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.388+0000 7fef1dffb640 1 -- 172.21.15.16:0/2964604514 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef2c012930 con 0x5570d0ad8d90 2024-03-28T22:18:30.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.389+0000 7fef1dffb640 1 -- 172.21.15.16:0/2964604514 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7fef240bb400 con 0x5570d0ad8d90 2024-03-28T22:18:30.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.389+0000 7fef3f067740 10 client.17178 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:18:30.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.389+0000 7fef3f067740 10 client.17178 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T22:18:30.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.389+0000 7fef3f067740 20 client.17178 populate_metadata read hostname 'smithi016' 2024-03-28T22:18:30.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.389+0000 7fef3f067740 1 --2- 172.21.15.16:0/2964604514 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5570d0b58180 0x5570d0b5a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.389+0000 7fef3f067740 4 client.17178 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:18:30.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.389+0000 7fef3f067740 1 -- 172.21.15.16:0/2964604514 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5570d0a43a40 con 0x5570d0b58180 2024-03-28T22:18:30.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.389+0000 7fef3a220640 1 --2- 172.21.15.16:0/2964604514 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5570d0b58180 0x5570d0b5a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.389+0000 7fef3a220640 1 --2- 172.21.15.16:0/2964604514 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5570d0b58180 0x5570d0b5a5a0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=188474855dea59e4 server_cookie=b6119d9f24c2b4e8 in_seq=0 out_seq=0 2024-03-28T22:18:30.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.389+0000 7fef1dffb640 10 client.17178 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:18:30.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.389+0000 7fef1dffb640 1 -- 172.21.15.16:0/2964604514 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5570d0a43a40 con 0x5570d0b58180 2024-03-28T22:18:30.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.389+0000 7fef1dffb640 10 client.17178 handle_command_reply: tid=0 2024-03-28T22:18:30.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.389+0000 7fef3f067740 1 client.17178 shutdown 2024-03-28T22:18:30.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.390+0000 7fef3f067740 1 -- 172.21.15.16:0/2964604514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570d0ad8d90 msgr2=0x5570d0adc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:30.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.390+0000 7fef3f067740 1 --2- 172.21.15.16:0/2964604514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570d0ad8d90 0x5570d0adc360 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fef240ba7f0 tx=0x7fef240b8680 comp rx=0 tx=0).stop 2024-03-28T22:18:30.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.390+0000 7fef3f067740 1 -- 172.21.15.16:0/2964604514 shutdown_connections 2024-03-28T22:18:30.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.390+0000 7fef3f067740 1 --2- 172.21.15.16:0/2964604514 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5570d0b58180 0x5570d0b5a5a0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.390+0000 7fef3f067740 1 --2- 172.21.15.16:0/2964604514 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570d0ae1970 0x5570d0b56610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.390+0000 7fef3f067740 1 --2- 172.21.15.16:0/2964604514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570d0ad8d90 0x5570d0adc360 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.390+0000 7fef1dffb640 0 client.17178 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:18:30.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.390+0000 7fef3f067740 1 --2- 172.21.15.16:0/2964604514 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570d0ad8470 0x5570d0adbe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.390+0000 7fef3f067740 1 -- 172.21.15.16:0/2964604514 >> 172.21.15.16:0/2964604514 conn(0x5570d0ad2ba0 msgr2=0x5570d0ad45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:18:30.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.390+0000 7fef3f067740 1 -- 172.21.15.16:0/2964604514 shutdown_connections 2024-03-28T22:18:30.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.390+0000 7fef3f067740 1 -- 172.21.15.16:0/2964604514 wait complete. 2024-03-28T22:18:30.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.390+0000 7fef3f067740 20 client.17178 trim_cache size 0 max 16384 2024-03-28T22:18:30.403 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:18:30.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.390+0000 7fef3c484640 1 -- 172.21.15.16:0/3653622871 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fef200a2090 msgr2=0x7fef200a4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:30.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.390+0000 7fef3c484640 1 --2- 172.21.15.16:0/3653622871 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fef200a2090 0x7fef200a4550 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fef34101730 tx=0x7fef30005f90 comp rx=0 tx=0).stop 2024-03-28T22:18:30.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.390+0000 7fef3c484640 1 -- 172.21.15.16:0/3653622871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef34100fb0 msgr2=0x7fef34199390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:30.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.390+0000 7fef3c484640 1 --2- 172.21.15.16:0/3653622871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef34100fb0 0x7fef34199390 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fef24009230 tx=0x7fef2402cb30 comp rx=0 tx=0).stop 2024-03-28T22:18:30.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.390+0000 7fef3c484640 1 -- 172.21.15.16:0/3653622871 shutdown_connections 2024-03-28T22:18:30.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.390+0000 7fef3c484640 1 --2- 172.21.15.16:0/3653622871 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fef200a2090 0x7fef200a4550 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.390+0000 7fef3c484640 1 --2- 172.21.15.16:0/3653622871 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef34106690 0x7fef341a3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.390+0000 7fef3c484640 1 --2- 172.21.15.16:0/3653622871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef34100fb0 0x7fef34199390 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.390+0000 7fef3c484640 1 --2- 172.21.15.16:0/3653622871 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef341005f0 0x7fef34198e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.390+0000 7fef3c484640 1 -- 172.21.15.16:0/3653622871 >> 172.21.15.16:0/3653622871 conn(0x7fef34077800 msgr2=0x7fef340ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:18:30.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.390+0000 7fef3c484640 1 -- 172.21.15.16:0/3653622871 shutdown_connections 2024-03-28T22:18:30.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.390+0000 7fef3c484640 1 -- 172.21.15.16:0/3653622871 wait complete. 2024-03-28T22:18:30.406 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:18:30.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.510+0000 7ff6b4fe5640 1 -- 172.21.15.16:0/1854759486 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6b0102b80 msgr2=0x7ff6b010d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:30.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.510+0000 7ff6b4fe5640 1 --2- 172.21.15.16:0/1854759486 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6b0102b80 0x7ff6b010d5e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff6a8009cd0 tx=0x7ff6a802c180 comp rx=0 tx=0).stop 2024-03-28T22:18:30.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.510+0000 7ff6b4fe5640 1 -- 172.21.15.16:0/1854759486 shutdown_connections 2024-03-28T22:18:30.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.510+0000 7ff6b4fe5640 1 --2- 172.21.15.16:0/1854759486 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6b0102b80 0x7ff6b010d5e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.510+0000 7ff6b4fe5640 1 --2- 172.21.15.16:0/1854759486 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6b01021e0 0x7ff6b0102640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.510+0000 7ff6b4fe5640 1 --2- 172.21.15.16:0/1854759486 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6b01009d0 0x7ff6b0100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.510+0000 7ff6b4fe5640 1 -- 172.21.15.16:0/1854759486 >> 172.21.15.16:0/1854759486 conn(0x7ff6b0077830 msgr2=0x7ff6b0077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:18:30.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.510+0000 7ff6b4fe5640 1 -- 172.21.15.16:0/1854759486 shutdown_connections 2024-03-28T22:18:30.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.510+0000 7ff6b4fe5640 1 -- 172.21.15.16:0/1854759486 wait complete. 2024-03-28T22:18:30.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.511+0000 7ff6b4fe5640 1 Processor -- start 2024-03-28T22:18:30.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.511+0000 7ff6b4fe5640 1 -- start start 2024-03-28T22:18:30.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.511+0000 7ff6b4fe5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6b01009d0 0x7ff6b0198150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.511+0000 7ff6b4fe5640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6b01021e0 0x7ff6b0198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.511+0000 7ff6b4fe5640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6b0102b80 0x7ff6b019c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.511+0000 7ff6b4fe5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b0110ac0 con 0x7ff6b01009d0 2024-03-28T22:18:30.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.511+0000 7ff6b4fe5640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b0110940 con 0x7ff6b01021e0 2024-03-28T22:18:30.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.511+0000 7ff6b4fe5640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b0110c40 con 0x7ff6b0102b80 2024-03-28T22:18:30.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.511+0000 7ff6ae59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6b01009d0 0x7ff6b0198150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.511+0000 7ff6ae59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6b01009d0 0x7ff6b0198150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53704/0 (socket says 172.21.15.16:53704) 2024-03-28T22:18:30.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.511+0000 7ff6ae59c640 1 -- 172.21.15.16:0/933832655 learned_addr learned my addr 172.21.15.16:0/933832655 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:18:30.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.511+0000 7ff6add9b640 1 --2- 172.21.15.16:0/933832655 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6b01021e0 0x7ff6b0198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.511+0000 7ff6aed9d640 1 --2- 172.21.15.16:0/933832655 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6b0102b80 0x7ff6b019c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.511+0000 7ff6ae59c640 1 -- 172.21.15.16:0/933832655 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6b0102b80 msgr2=0x7ff6b019c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:30.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.511+0000 7ff6ae59c640 1 --2- 172.21.15.16:0/933832655 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6b0102b80 0x7ff6b019c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.511+0000 7ff6ae59c640 1 -- 172.21.15.16:0/933832655 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6b01021e0 msgr2=0x7ff6b0198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:30.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.511+0000 7ff6ae59c640 1 --2- 172.21.15.16:0/933832655 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6b01021e0 0x7ff6b0198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.511+0000 7ff6ae59c640 1 -- 172.21.15.16:0/933832655 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6b019d050 con 0x7ff6b01009d0 2024-03-28T22:18:30.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.512+0000 7ff6ae59c640 1 --2- 172.21.15.16:0/933832655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6b01009d0 0x7ff6b0198150 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7ff6a000d950 tx=0x7ff6a000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:18:30.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.512+0000 7ff69f7fe640 1 -- 172.21.15.16:0/933832655 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a0017070 con 0x7ff6b01009d0 2024-03-28T22:18:30.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.512+0000 7ff69f7fe640 1 -- 172.21.15.16:0/933832655 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6a0009c30 con 0x7ff6b01009d0 2024-03-28T22:18:30.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.512+0000 7ff6b4fe5640 1 -- 172.21.15.16:0/933832655 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6b019d340 con 0x7ff6b01009d0 2024-03-28T22:18:30.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.512+0000 7ff69f7fe640 1 -- 172.21.15.16:0/933832655 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a000f6b0 con 0x7ff6b01009d0 2024-03-28T22:18:30.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.512+0000 7ff6b4fe5640 1 -- 172.21.15.16:0/933832655 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6b01a4bd0 con 0x7ff6b01009d0 2024-03-28T22:18:30.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.514+0000 7ff69f7fe640 1 -- 172.21.15.16:0/933832655 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7ff6a00109d0 con 0x7ff6b01009d0 2024-03-28T22:18:30.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.514+0000 7ff6b7bc6740 1 Processor -- start 2024-03-28T22:18:30.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.515+0000 7ff6b7bc6740 1 -- start start 2024-03-28T22:18:30.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.515+0000 7ff6b7bc6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617fc113470 0x5617fc113850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.515+0000 7ff69f7fe640 1 --2- 172.21.15.16:0/933832655 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff6980a1ef0 0x7ff6980a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.515+0000 7ff6b7bc6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617fc113d90 0x5617fc11c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.515+0000 7ff6b7bc6740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5617fc11c970 0x5617fc11ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.515+0000 7ff6b7bc6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5617fc0771b0 con 0x5617fc113470 2024-03-28T22:18:30.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.515+0000 7ff6b7bc6740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5617fc01fc00 con 0x5617fc11c970 2024-03-28T22:18:30.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.515+0000 7ff6b7bc6740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5617fc073250 con 0x5617fc113d90 2024-03-28T22:18:30.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.515+0000 7ff6aed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617fc113470 0x5617fc113850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.515+0000 7ff6ae59c640 1 --2- 172.21.15.16:0/933832655 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff6980a1ef0 0x7ff6980a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.515+0000 7ff6aed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617fc113470 0x5617fc113850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53716/0 (socket says 172.21.15.16:53716) 2024-03-28T22:18:30.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.515+0000 7ff6aed9d640 1 -- 172.21.15.16:0/2714207327 learned_addr learned my addr 172.21.15.16:0/2714207327 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:18:30.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.515+0000 7ff6aed9d640 1 --2- 172.21.15.16:0/2714207327 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5617fc11c970 0x5617fc11ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.515+0000 7ff69f7fe640 1 -- 172.21.15.16:0/933832655 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7ff6a00cdd90 con 0x7ff6b01009d0 2024-03-28T22:18:30.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.515+0000 7ff6add9b640 1 --2- 172.21.15.16:0/2714207327 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617fc113d90 0x5617fc11c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.515+0000 7ff6aed9d640 1 -- 172.21.15.16:0/2714207327 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617fc113d90 msgr2=0x5617fc11c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:30.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.515+0000 7ff6aed9d640 1 --2- 172.21.15.16:0/2714207327 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617fc113d90 0x5617fc11c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.515+0000 7ff6aed9d640 1 -- 172.21.15.16:0/2714207327 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5617fc11c970 msgr2=0x5617fc11ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:30.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.515+0000 7ff6aed9d640 1 --2- 172.21.15.16:0/2714207327 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5617fc11c970 0x5617fc11ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.515+0000 7ff6aed9d640 1 -- 172.21.15.16:0/2714207327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617fc03f3d0 con 0x5617fc113470 2024-03-28T22:18:30.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.515+0000 7ff6aed9d640 1 --2- 172.21.15.16:0/2714207327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617fc113470 0x5617fc113850 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7ff6a8009cd0 tx=0x7ff6a80093d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af9bad7f6ccb26ad server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:18:30.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.515+0000 7ff6ae59c640 1 --2- 172.21.15.16:0/933832655 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff6980a1ef0 0x7ff6980a43b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff6a009a4d0 tx=0x7ff6a008fc60 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:18:30.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.516+0000 7ff69cff9640 1 -- 172.21.15.16:0/2714207327 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a8048070 con 0x5617fc113470 2024-03-28T22:18:30.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.516+0000 7ff69cff9640 1 -- 172.21.15.16:0/2714207327 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6a8003970 con 0x5617fc113470 2024-03-28T22:18:30.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.516+0000 7ff69cff9640 1 -- 172.21.15.16:0/2714207327 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a8039040 con 0x5617fc113470 2024-03-28T22:18:30.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.516+0000 7ff6b7bc6740 1 -- 172.21.15.16:0/2714207327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617fc113470 msgr2=0x5617fc113850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:30.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.516+0000 7ff6b7bc6740 1 --2- 172.21.15.16:0/2714207327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617fc113470 0x5617fc113850 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7ff6a8009cd0 tx=0x7ff6a80093d0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.516+0000 7ff6b7bc6740 1 -- 172.21.15.16:0/2714207327 shutdown_connections 2024-03-28T22:18:30.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.516+0000 7ff6b7bc6740 1 --2- 172.21.15.16:0/2714207327 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5617fc11c970 0x5617fc11ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.516+0000 7ff6b7bc6740 1 --2- 172.21.15.16:0/2714207327 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617fc113d90 0x5617fc11c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.516+0000 7ff6b7bc6740 1 --2- 172.21.15.16:0/2714207327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617fc113470 0x5617fc113850 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.516+0000 7ff6b7bc6740 1 -- 172.21.15.16:0/2714207327 >> 172.21.15.16:0/2714207327 conn(0x5617fc10dba0 msgr2=0x5617fc10df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:18:30.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.516+0000 7ff6b7bc6740 1 -- 172.21.15.16:0/2714207327 shutdown_connections 2024-03-28T22:18:30.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.517+0000 7ff6b7bc6740 1 -- 172.21.15.16:0/2714207327 wait complete. 2024-03-28T22:18:30.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.517+0000 7ff6b7bc6740 1 -- start start 2024-03-28T22:18:30.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.517+0000 7ff6b7bc6740 1 Processor -- start 2024-03-28T22:18:30.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.517+0000 7ff6b7bc6740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5617fc113470 0x5617fc119970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.517+0000 7ff6b7bc6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617fc113d90 0x5617fc119eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.517+0000 7ff6b7bc6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617fc11c970 0x5617fc11a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.517+0000 7ff6b7bc6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5617fc11f700 con 0x5617fc113d90 2024-03-28T22:18:30.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.517+0000 7ff6b7bc6740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5617fc015470 con 0x5617fc113470 2024-03-28T22:18:30.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.517+0000 7ff6b7bc6740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5617fc11f880 con 0x5617fc11c970 2024-03-28T22:18:30.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.517+0000 7ff6b7bc6740 10 client.0 _ll_register_callbacks cb 0x5617fc032cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:18:30.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.517+0000 7ff6add9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617fc113d90 0x5617fc119eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.517+0000 7ff6add9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617fc113d90 0x5617fc119eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53724/0 (socket says 172.21.15.16:53724) 2024-03-28T22:18:30.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.517+0000 7ff6add9b640 1 -- 172.21.15.16:0/1144002455 learned_addr learned my addr 172.21.15.16:0/1144002455 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:18:30.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.517+0000 7ff6aed9d640 1 --2- 172.21.15.16:0/1144002455 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5617fc113470 0x5617fc119970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.517+0000 7ff6aed9d640 1 --2- 172.21.15.16:0/1144002455 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617fc11c970 0x5617fc11a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.518+0000 7ff6add9b640 1 -- 172.21.15.16:0/1144002455 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617fc11c970 msgr2=0x5617fc11a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:30.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.518+0000 7ff6add9b640 1 --2- 172.21.15.16:0/1144002455 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617fc11c970 0x5617fc11a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.518+0000 7ff6add9b640 1 -- 172.21.15.16:0/1144002455 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5617fc113470 msgr2=0x5617fc119970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:30.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.518+0000 7ff6add9b640 1 --2- 172.21.15.16:0/1144002455 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5617fc113470 0x5617fc119970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.518+0000 7ff6add9b640 1 -- 172.21.15.16:0/1144002455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617fbd8b3f0 con 0x5617fc113d90 2024-03-28T22:18:30.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.518+0000 7ff6add9b640 1 --2- 172.21.15.16:0/1144002455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617fc113d90 0x5617fc119eb0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4012700 tx=0x7ff6a4012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22aec25073f3c1ff server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:18:30.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.518+0000 7ff695ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:18:30.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.518+0000 7ff695ffb640 1 -- 172.21.15.16:0/1144002455 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a4004050 con 0x5617fc113d90 2024-03-28T22:18:30.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.518+0000 7ff695ffb640 1 -- 172.21.15.16:0/1144002455 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7ff6a40041f0 con 0x5617fc113d90 2024-03-28T22:18:30.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.518+0000 7ff6b7bc6740 1 -- 172.21.15.16:0/1144002455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6a8003970 con 0x5617fc113d90 2024-03-28T22:18:30.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.518+0000 7ff695ffb640 1 -- 172.21.15.16:0/1144002455 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff6a40043f0 con 0x5617fc113d90 2024-03-28T22:18:30.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.518+0000 7ff6b7bc6740 10 client.17190 fetch_fsmap learned FSMap version 27 2024-03-28T22:18:30.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.518+0000 7ff6b7bc6740 1 -- 172.21.15.16:0/1144002455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5617fbe557d0 con 0x5617fc113d90 2024-03-28T22:18:30.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.518+0000 7ff695ffb640 1 -- 172.21.15.16:0/1144002455 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6a8003970 con 0x5617fc113d90 2024-03-28T22:18:30.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.518+0000 7ff695ffb640 1 -- 172.21.15.16:0/1144002455 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7ff6a4003bc0 con 0x5617fc113d90 2024-03-28T22:18:30.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.518+0000 7ff6b7bc6740 10 client.17190 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:18:30.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.518+0000 7ff6b7bc6740 10 client.17190 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:18:30.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.518+0000 7ff6b7bc6740 20 client.17190 populate_metadata read hostname 'smithi016' 2024-03-28T22:18:30.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.518+0000 7ff6b7bc6740 1 --2- 172.21.15.16:0/1144002455 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5617fc12c930 0x5617fc12ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.518+0000 7ff6b7bc6740 4 client.17190 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:18:30.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.518+0000 7ff6b7bc6740 1 -- 172.21.15.16:0/1144002455 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5617fc0846f0 con 0x5617fc12c930 2024-03-28T22:18:30.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.519+0000 7ff6aed9d640 1 --2- 172.21.15.16:0/1144002455 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5617fc12c930 0x5617fc12ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.519+0000 7ff6aed9d640 1 --2- 172.21.15.16:0/1144002455 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5617fc12c930 0x5617fc12ed80 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb9ab295841d810d server_cookie=a0f27c229ba100c in_seq=0 out_seq=0 2024-03-28T22:18:30.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.519+0000 7ff695ffb640 10 client.17190 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:18:30.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.520+0000 7ff695ffb640 1 -- 172.21.15.16:0/1144002455 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5617fc0846f0 con 0x5617fc12c930 2024-03-28T22:18:30.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.520+0000 7ff695ffb640 10 client.17190 handle_command_reply: tid=0 2024-03-28T22:18:30.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.520+0000 7ff6b7bc6740 1 client.17190 shutdown 2024-03-28T22:18:30.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.520+0000 7ff6b7bc6740 1 -- 172.21.15.16:0/1144002455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617fc113d90 msgr2=0x5617fc119eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:30.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.520+0000 7ff6b7bc6740 1 --2- 172.21.15.16:0/1144002455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617fc113d90 0x5617fc119eb0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4012700 tx=0x7ff6a4012bc0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.520+0000 7ff6b7bc6740 1 -- 172.21.15.16:0/1144002455 shutdown_connections 2024-03-28T22:18:30.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.520+0000 7ff6b7bc6740 1 --2- 172.21.15.16:0/1144002455 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5617fc12c930 0x5617fc12ed80 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.520+0000 7ff6b7bc6740 1 --2- 172.21.15.16:0/1144002455 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617fc11c970 0x5617fc11a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.520+0000 7ff6b7bc6740 1 --2- 172.21.15.16:0/1144002455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617fc113d90 0x5617fc119eb0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.520+0000 7ff695ffb640 0 client.17190 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:18:30.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.520+0000 7ff6b7bc6740 1 --2- 172.21.15.16:0/1144002455 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5617fc113470 0x5617fc119970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.520+0000 7ff6b7bc6740 1 -- 172.21.15.16:0/1144002455 >> 172.21.15.16:0/1144002455 conn(0x5617fc10dba0 msgr2=0x5617fc10f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:18:30.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.520+0000 7ff6b7bc6740 1 -- 172.21.15.16:0/1144002455 shutdown_connections 2024-03-28T22:18:30.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.520+0000 7ff6b7bc6740 1 -- 172.21.15.16:0/1144002455 wait complete. 2024-03-28T22:18:30.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.520+0000 7ff6b7bc6740 20 client.17190 trim_cache size 0 max 16384 2024-03-28T22:18:30.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.533+0000 7ff6b7bc6740 1 Processor -- start 2024-03-28T22:18:30.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.534+0000 7ff6b7bc6740 1 -- start start 2024-03-28T22:18:30.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.534+0000 7ff6b7bc6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617fc113470 0x5617fc11c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.534+0000 7ff6b7bc6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617fc113d90 0x5617fc108bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.534+0000 7ff6b7bc6740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5617fc11c970 0x5617fc1284d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.534+0000 7ff6b7bc6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b01afd90 con 0x5617fc113470 2024-03-28T22:18:30.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.534+0000 7ff6b7bc6740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b01ad660 con 0x5617fc11c970 2024-03-28T22:18:30.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.534+0000 7ff6b7bc6740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b01ab4d0 con 0x5617fc113d90 2024-03-28T22:18:30.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.534+0000 7ff6aed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617fc113470 0x5617fc11c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.534+0000 7ff6aed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617fc113470 0x5617fc11c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53738/0 (socket says 172.21.15.16:53738) 2024-03-28T22:18:30.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.534+0000 7ff6aed9d640 1 -- 172.21.15.16:0/3894931418 learned_addr learned my addr 172.21.15.16:0/3894931418 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:18:30.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.534+0000 7ff6aed9d640 1 --2- 172.21.15.16:0/3894931418 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5617fc11c970 0x5617fc1284d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.534+0000 7ff6add9b640 1 --2- 172.21.15.16:0/3894931418 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617fc113d90 0x5617fc108bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.534+0000 7ff6aed9d640 1 -- 172.21.15.16:0/3894931418 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617fc113d90 msgr2=0x5617fc108bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:30.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.534+0000 7ff6aed9d640 1 --2- 172.21.15.16:0/3894931418 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617fc113d90 0x5617fc108bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.534+0000 7ff6aed9d640 1 -- 172.21.15.16:0/3894931418 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5617fc11c970 msgr2=0x5617fc1284d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:30.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.534+0000 7ff6aed9d640 1 --2- 172.21.15.16:0/3894931418 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5617fc11c970 0x5617fc1284d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.534+0000 7ff6aed9d640 1 -- 172.21.15.16:0/3894931418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617fc118940 con 0x5617fc113470 2024-03-28T22:18:30.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.534+0000 7ff6aed9d640 1 --2- 172.21.15.16:0/3894931418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617fc113470 0x5617fc11c0a0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7ff6a8009620 tx=0x7ff6a8005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f26d803c1c699e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:18:30.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.534+0000 7ff697fff640 1 -- 172.21.15.16:0/3894931418 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a8039070 con 0x5617fc113470 2024-03-28T22:18:30.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.534+0000 7ff697fff640 1 -- 172.21.15.16:0/3894931418 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6a8032b50 con 0x5617fc113470 2024-03-28T22:18:30.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.534+0000 7ff697fff640 1 -- 172.21.15.16:0/3894931418 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a8048040 con 0x5617fc113470 2024-03-28T22:18:30.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.535+0000 7ff6b7bc6740 1 -- 172.21.15.16:0/3894931418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617fc113470 msgr2=0x5617fc11c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:30.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.535+0000 7ff6b7bc6740 1 --2- 172.21.15.16:0/3894931418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617fc113470 0x5617fc11c0a0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7ff6a8009620 tx=0x7ff6a8005ef0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.535+0000 7ff6b7bc6740 1 -- 172.21.15.16:0/3894931418 shutdown_connections 2024-03-28T22:18:30.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.535+0000 7ff6b7bc6740 1 --2- 172.21.15.16:0/3894931418 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5617fc11c970 0x5617fc1284d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.535+0000 7ff6b7bc6740 1 --2- 172.21.15.16:0/3894931418 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617fc113d90 0x5617fc108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.535+0000 7ff6b7bc6740 1 --2- 172.21.15.16:0/3894931418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617fc113470 0x5617fc11c0a0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.535+0000 7ff6b7bc6740 1 -- 172.21.15.16:0/3894931418 >> 172.21.15.16:0/3894931418 conn(0x5617fc10dba0 msgr2=0x5617fc10cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:18:30.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.535+0000 7ff6b7bc6740 1 -- 172.21.15.16:0/3894931418 shutdown_connections 2024-03-28T22:18:30.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.535+0000 7ff6b7bc6740 1 -- 172.21.15.16:0/3894931418 wait complete. 2024-03-28T22:18:30.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.535+0000 7ff6b7bc6740 1 -- start start 2024-03-28T22:18:30.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.535+0000 7ff6b7bc6740 1 Processor -- start 2024-03-28T22:18:30.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.535+0000 7ff6b7bc6740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5617fc113470 0x5617fc124b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.535+0000 7ff6b7bc6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617fc113d90 0x5617fc121ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.536+0000 7ff6b7bc6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617fc11c970 0x5617fc122010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.536+0000 7ff6b7bc6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b01abb20 con 0x5617fc113d90 2024-03-28T22:18:30.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.536+0000 7ff6b7bc6740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b01acf90 con 0x5617fc113470 2024-03-28T22:18:30.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.536+0000 7ff6b7bc6740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b01ac1f0 con 0x5617fc11c970 2024-03-28T22:18:30.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.536+0000 7ff6b7bc6740 10 client.0 _ll_register_callbacks cb 0x7ff6a4009d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:18:30.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.536+0000 7ff6add9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617fc113d90 0x5617fc121ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.536+0000 7ff6add9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617fc113d90 0x5617fc121ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53740/0 (socket says 172.21.15.16:53740) 2024-03-28T22:18:30.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.536+0000 7ff6add9b640 1 -- 172.21.15.16:0/2057683987 learned_addr learned my addr 172.21.15.16:0/2057683987 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:18:30.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.536+0000 7ff6aed9d640 1 --2- 172.21.15.16:0/2057683987 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5617fc113470 0x5617fc124b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.536+0000 7ff6aed9d640 1 --2- 172.21.15.16:0/2057683987 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617fc11c970 0x5617fc122010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.536+0000 7ff6add9b640 1 -- 172.21.15.16:0/2057683987 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617fc11c970 msgr2=0x5617fc122010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:30.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.536+0000 7ff6add9b640 1 --2- 172.21.15.16:0/2057683987 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617fc11c970 0x5617fc122010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.536+0000 7ff6add9b640 1 -- 172.21.15.16:0/2057683987 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5617fc113470 msgr2=0x5617fc124b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:30.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.536+0000 7ff6add9b640 1 --2- 172.21.15.16:0/2057683987 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5617fc113470 0x5617fc124b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.536+0000 7ff6add9b640 1 -- 172.21.15.16:0/2057683987 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617fc1251f0 con 0x5617fc113d90 2024-03-28T22:18:30.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.536+0000 7ff6add9b640 1 --2- 172.21.15.16:0/2057683987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617fc113d90 0x5617fc121ad0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4000c00 tx=0x7ff6a40042e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b7edf3fdeebacf3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:18:30.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.536+0000 7ff695ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:18:30.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.536+0000 7ff695ffb640 1 -- 172.21.15.16:0/2057683987 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a4029040 con 0x5617fc113d90 2024-03-28T22:18:30.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.536+0000 7ff695ffb640 1 -- 172.21.15.16:0/2057683987 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7ff6a400f920 con 0x5617fc113d90 2024-03-28T22:18:30.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.536+0000 7ff6b7bc6740 1 -- 172.21.15.16:0/2057683987 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6a8032b50 con 0x5617fc113d90 2024-03-28T22:18:30.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.536+0000 7ff695ffb640 1 -- 172.21.15.16:0/2057683987 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff6a403a050 con 0x5617fc113d90 2024-03-28T22:18:30.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.536+0000 7ff6b7bc6740 10 client.17196 fetch_fsmap learned FSMap version 27 2024-03-28T22:18:30.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.536+0000 7ff6b7bc6740 1 -- 172.21.15.16:0/2057683987 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5617fc193790 con 0x5617fc113d90 2024-03-28T22:18:30.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.537+0000 7ff695ffb640 1 -- 172.21.15.16:0/2057683987 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6a8032b50 con 0x5617fc113d90 2024-03-28T22:18:30.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.537+0000 7ff695ffb640 1 -- 172.21.15.16:0/2057683987 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7ff6a4035030 con 0x5617fc113d90 2024-03-28T22:18:30.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.537+0000 7ff6b7bc6740 10 client.17196 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:18:30.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.537+0000 7ff6b7bc6740 10 client.17196 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:18:30.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.537+0000 7ff6b7bc6740 20 client.17196 populate_metadata read hostname 'smithi016' 2024-03-28T22:18:30.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.537+0000 7ff6b7bc6740 1 --2- 172.21.15.16:0/2057683987 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5617fc12c930 0x5617fc195fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:30.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.537+0000 7ff6b7bc6740 4 client.17196 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:18:30.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.537+0000 7ff6b7bc6740 1 -- 172.21.15.16:0/2057683987 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5617fc07ea40 con 0x5617fc12c930 2024-03-28T22:18:30.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.537+0000 7ff6aed9d640 1 --2- 172.21.15.16:0/2057683987 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5617fc12c930 0x5617fc195fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:30.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.537+0000 7ff695ffb640 10 client.17196 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:18:30.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.537+0000 7ff6aed9d640 1 --2- 172.21.15.16:0/2057683987 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5617fc12c930 0x5617fc195fd0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9558a18206469a50 server_cookie=5932aeedb82bbce4 in_seq=0 out_seq=0 2024-03-28T22:18:30.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.537+0000 7ff695ffb640 1 -- 172.21.15.16:0/2057683987 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5617fc07ea40 con 0x5617fc12c930 2024-03-28T22:18:30.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.537+0000 7ff695ffb640 10 client.17196 handle_command_reply: tid=0 2024-03-28T22:18:30.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.537+0000 7ff6b7bc6740 1 client.17196 shutdown 2024-03-28T22:18:30.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.538+0000 7ff6b7bc6740 1 -- 172.21.15.16:0/2057683987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617fc113d90 msgr2=0x5617fc121ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:30.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.538+0000 7ff6b7bc6740 1 --2- 172.21.15.16:0/2057683987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617fc113d90 0x5617fc121ad0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4000c00 tx=0x7ff6a40042e0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.538+0000 7ff6b7bc6740 1 -- 172.21.15.16:0/2057683987 shutdown_connections 2024-03-28T22:18:30.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.538+0000 7ff6b7bc6740 1 --2- 172.21.15.16:0/2057683987 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5617fc12c930 0x5617fc195fd0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.538+0000 7ff6b7bc6740 1 --2- 172.21.15.16:0/2057683987 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617fc11c970 0x5617fc122010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.538+0000 7ff6b7bc6740 1 --2- 172.21.15.16:0/2057683987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5617fc113d90 0x5617fc121ad0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.538+0000 7ff695ffb640 0 client.17196 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:18:30.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.538+0000 7ff6b7bc6740 1 --2- 172.21.15.16:0/2057683987 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5617fc113470 0x5617fc124b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.538+0000 7ff6b7bc6740 1 -- 172.21.15.16:0/2057683987 >> 172.21.15.16:0/2057683987 conn(0x5617fc10dba0 msgr2=0x5617fc10ef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:18:30.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.538+0000 7ff6b7bc6740 1 -- 172.21.15.16:0/2057683987 shutdown_connections 2024-03-28T22:18:30.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.538+0000 7ff6b7bc6740 1 -- 172.21.15.16:0/2057683987 wait complete. 2024-03-28T22:18:30.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.538+0000 7ff6b7bc6740 20 client.17196 trim_cache size 0 max 16384 2024-03-28T22:18:30.561 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:18:30.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.538+0000 7ff6b4fe5640 1 -- 172.21.15.16:0/933832655 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff6980a1ef0 msgr2=0x7ff6980a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:30.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.538+0000 7ff6b4fe5640 1 --2- 172.21.15.16:0/933832655 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff6980a1ef0 0x7ff6980a43b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff6a009a4d0 tx=0x7ff6a008fc60 comp rx=0 tx=0).stop 2024-03-28T22:18:30.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.538+0000 7ff6b4fe5640 1 -- 172.21.15.16:0/933832655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6b01009d0 msgr2=0x7ff6b0198150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:30.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.538+0000 7ff6b4fe5640 1 --2- 172.21.15.16:0/933832655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6b01009d0 0x7ff6b0198150 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7ff6a000d950 tx=0x7ff6a000de10 comp rx=0 tx=0).stop 2024-03-28T22:18:30.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.538+0000 7ff6b4fe5640 1 -- 172.21.15.16:0/933832655 shutdown_connections 2024-03-28T22:18:30.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.538+0000 7ff6b4fe5640 1 --2- 172.21.15.16:0/933832655 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff6980a1ef0 0x7ff6980a43b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.538+0000 7ff6b4fe5640 1 --2- 172.21.15.16:0/933832655 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6b0102b80 0x7ff6b019c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.538+0000 7ff6b4fe5640 1 --2- 172.21.15.16:0/933832655 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6b01021e0 0x7ff6b0198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.538+0000 7ff6b4fe5640 1 --2- 172.21.15.16:0/933832655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6b01009d0 0x7ff6b0198150 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:30.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.538+0000 7ff6b4fe5640 1 -- 172.21.15.16:0/933832655 >> 172.21.15.16:0/933832655 conn(0x7ff6b0077830 msgr2=0x7ff6b006a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:18:30.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.538+0000 7ff6b4fe5640 1 -- 172.21.15.16:0/933832655 shutdown_connections 2024-03-28T22:18:30.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:30.538+0000 7ff6b4fe5640 1 -- 172.21.15.16:0/933832655 wait complete. 2024-03-28T22:18:30.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:30 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3996612685' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:18:30.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:30 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/3996612685' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:18:31.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:30 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3996612685' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:18:31.565 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:18:31.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:18:31.566 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:18:31.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.684+0000 7f7963c0b640 1 -- 172.21.15.16:0/4262734329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f795c0ffc60 msgr2=0x7f795c1000e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:31.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.684+0000 7f7963c0b640 1 --2- 172.21.15.16:0/4262734329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f795c0ffc60 0x7f795c1000e0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f794c009900 tx=0x7f794c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:18:31.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.684+0000 7f7963c0b640 1 -- 172.21.15.16:0/4262734329 shutdown_connections 2024-03-28T22:18:31.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.684+0000 7f7963c0b640 1 --2- 172.21.15.16:0/4262734329 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f795c100620 0x7f795c10e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.684+0000 7f7963c0b640 1 --2- 172.21.15.16:0/4262734329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f795c0ffc60 0x7f795c1000e0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.684+0000 7f7963c0b640 1 --2- 172.21.15.16:0/4262734329 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f795c1076f0 0x7f795c107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.684+0000 7f7963c0b640 1 -- 172.21.15.16:0/4262734329 >> 172.21.15.16:0/4262734329 conn(0x7f795c076b50 msgr2=0x7f795c0fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:18:31.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.684+0000 7f7963c0b640 1 -- 172.21.15.16:0/4262734329 shutdown_connections 2024-03-28T22:18:31.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.684+0000 7f7963c0b640 1 -- 172.21.15.16:0/4262734329 wait complete. 2024-03-28T22:18:31.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.685+0000 7f7963c0b640 1 Processor -- start 2024-03-28T22:18:31.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.685+0000 7f7963c0b640 1 -- start start 2024-03-28T22:18:31.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.685+0000 7f7963c0b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f795c0ffc60 0x7f795c197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:31.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.685+0000 7f7963c0b640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f795c100620 0x7f795c198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:31.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.685+0000 7f7963c0b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f795c1076f0 0x7f795c19c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:31.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.685+0000 7f7963c0b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f795c111ae0 con 0x7f795c0ffc60 2024-03-28T22:18:31.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.685+0000 7f7963c0b640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f795c111960 con 0x7f795c100620 2024-03-28T22:18:31.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.685+0000 7f7963c0b640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f795c111c60 con 0x7f795c1076f0 2024-03-28T22:18:31.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.685+0000 7f79619a7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f795c0ffc60 0x7f795c197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:31.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.685+0000 7f79619a7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f795c0ffc60 0x7f795c197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47474/0 (socket says 172.21.15.16:47474) 2024-03-28T22:18:31.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.685+0000 7f79619a7640 1 -- 172.21.15.16:0/327014322 learned_addr learned my addr 172.21.15.16:0/327014322 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:18:31.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.685+0000 7f79611a6640 1 --2- 172.21.15.16:0/327014322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f795c100620 0x7f795c198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:31.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.686+0000 7f79621a8640 1 --2- 172.21.15.16:0/327014322 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f795c1076f0 0x7f795c19c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:31.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.686+0000 7f79619a7640 1 -- 172.21.15.16:0/327014322 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f795c1076f0 msgr2=0x7f795c19c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:31.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.686+0000 7f79619a7640 1 --2- 172.21.15.16:0/327014322 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f795c1076f0 0x7f795c19c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.686+0000 7f79619a7640 1 -- 172.21.15.16:0/327014322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f795c100620 msgr2=0x7f795c198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:31.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.686+0000 7f79619a7640 1 --2- 172.21.15.16:0/327014322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f795c100620 0x7f795c198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.686+0000 7f79619a7640 1 -- 172.21.15.16:0/327014322 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f795c19cfb0 con 0x7f795c0ffc60 2024-03-28T22:18:31.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.686+0000 7f79619a7640 1 --2- 172.21.15.16:0/327014322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f795c0ffc60 0x7f795c197fc0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f795800e9f0 tx=0x7f795800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:18:31.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.686+0000 7f7952ffd640 1 -- 172.21.15.16:0/327014322 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795800cd80 con 0x7f795c0ffc60 2024-03-28T22:18:31.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.686+0000 7f7952ffd640 1 -- 172.21.15.16:0/327014322 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7958005500 con 0x7f795c0ffc60 2024-03-28T22:18:31.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.686+0000 7f7952ffd640 1 -- 172.21.15.16:0/327014322 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7958010690 con 0x7f795c0ffc60 2024-03-28T22:18:31.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.686+0000 7f7963c0b640 1 -- 172.21.15.16:0/327014322 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f795c19d2a0 con 0x7f795c0ffc60 2024-03-28T22:18:31.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.686+0000 7f7963c0b640 1 -- 172.21.15.16:0/327014322 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f795c1a4b30 con 0x7f795c0ffc60 2024-03-28T22:18:31.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.688+0000 7f7952ffd640 1 -- 172.21.15.16:0/327014322 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f79580046c0 con 0x7f795c0ffc60 2024-03-28T22:18:31.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.689+0000 7f79667df740 1 Processor -- start 2024-03-28T22:18:31.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.689+0000 7f79667df740 1 -- start start 2024-03-28T22:18:31.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.689+0000 7f79667df740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55789a1589f0 0x55789a158dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:31.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.689+0000 7f79667df740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55789a159310 0x55789a1619b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:31.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.689+0000 7f79667df740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55789a161ef0 0x55789a1642e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:31.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.689+0000 7f79667df740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55789a0590b0 con 0x55789a161ef0 2024-03-28T22:18:31.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.689+0000 7f79667df740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55789a0bbea0 con 0x55789a159310 2024-03-28T22:18:31.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.689+0000 7f79667df740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55789a064c00 con 0x55789a1589f0 2024-03-28T22:18:31.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.689+0000 7f7952ffd640 1 --2- 172.21.15.16:0/327014322 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f79480a1ef0 0x7f79480a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:31.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.690+0000 7f79611a6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55789a161ef0 0x55789a1642e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:31.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.690+0000 7f79619a7640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55789a159310 0x55789a1619b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:31.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.690+0000 7f79611a6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55789a161ef0 0x55789a1642e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47486/0 (socket says 172.21.15.16:47486) 2024-03-28T22:18:31.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.690+0000 7f79611a6640 1 -- 172.21.15.16:0/1971436413 learned_addr learned my addr 172.21.15.16:0/1971436413 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:18:31.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.690+0000 7f79621a8640 1 --2- 172.21.15.16:0/327014322 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f79480a1ef0 0x7f79480a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:31.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.690+0000 7f79621a8640 1 --2- 172.21.15.16:0/1971436413 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55789a1589f0 0x55789a158dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:31.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.690+0000 7f7952ffd640 1 -- 172.21.15.16:0/327014322 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7f79580ce670 con 0x7f795c0ffc60 2024-03-28T22:18:31.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.690+0000 7f79611a6640 1 -- 172.21.15.16:0/1971436413 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55789a1589f0 msgr2=0x55789a158dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:31.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.690+0000 7f79611a6640 1 --2- 172.21.15.16:0/1971436413 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55789a1589f0 0x55789a158dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.690+0000 7f79611a6640 1 -- 172.21.15.16:0/1971436413 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55789a159310 msgr2=0x55789a1619b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:31.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.690+0000 7f79611a6640 1 --2- 172.21.15.16:0/1971436413 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55789a159310 0x55789a1619b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.690+0000 7f79611a6640 1 -- 172.21.15.16:0/1971436413 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557899e987e0 con 0x55789a161ef0 2024-03-28T22:18:31.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.690+0000 7f79619a7640 1 --2- 172.21.15.16:0/1971436413 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55789a159310 0x55789a1619b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:18:31.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.690+0000 7f79611a6640 1 --2- 172.21.15.16:0/1971436413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55789a161ef0 0x55789a1642e0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f794c02cfd0 tx=0x7f794c005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f848bf18feec2f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:18:31.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.690+0000 7f79621a8640 1 --2- 172.21.15.16:0/327014322 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f79480a1ef0 0x7f79480a43b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7954009fd0 tx=0x7f79540091e0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:18:31.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.690+0000 7f7947fff640 1 -- 172.21.15.16:0/1971436413 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f794c042070 con 0x55789a161ef0 2024-03-28T22:18:31.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.690+0000 7f7947fff640 1 -- 172.21.15.16:0/1971436413 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f794c02ca80 con 0x55789a161ef0 2024-03-28T22:18:31.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.690+0000 7f7947fff640 1 -- 172.21.15.16:0/1971436413 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f794c004f00 con 0x55789a161ef0 2024-03-28T22:18:31.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.691+0000 7f79667df740 1 -- 172.21.15.16:0/1971436413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55789a161ef0 msgr2=0x55789a1642e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:31.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.691+0000 7f79667df740 1 --2- 172.21.15.16:0/1971436413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55789a161ef0 0x55789a1642e0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f794c02cfd0 tx=0x7f794c005c00 comp rx=0 tx=0).stop 2024-03-28T22:18:31.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.691+0000 7f79667df740 1 -- 172.21.15.16:0/1971436413 shutdown_connections 2024-03-28T22:18:31.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.691+0000 7f79667df740 1 --2- 172.21.15.16:0/1971436413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55789a161ef0 0x55789a1642e0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.691+0000 7f79667df740 1 --2- 172.21.15.16:0/1971436413 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55789a159310 0x55789a1619b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.691+0000 7f79667df740 1 --2- 172.21.15.16:0/1971436413 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55789a1589f0 0x55789a158dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.691+0000 7f79667df740 1 -- 172.21.15.16:0/1971436413 >> 172.21.15.16:0/1971436413 conn(0x55789a153120 msgr2=0x55789a153500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:18:31.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.691+0000 7f79667df740 1 -- 172.21.15.16:0/1971436413 shutdown_connections 2024-03-28T22:18:31.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.691+0000 7f79667df740 1 -- 172.21.15.16:0/1971436413 wait complete. 2024-03-28T22:18:31.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.692+0000 7f79667df740 1 -- start start 2024-03-28T22:18:31.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.692+0000 7f79667df740 1 Processor -- start 2024-03-28T22:18:31.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.692+0000 7f79667df740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55789a1589f0 0x55789a15ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:31.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.692+0000 7f79667df740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55789a159310 0x55789a15f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:31.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.692+0000 7f79667df740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55789a161ef0 0x55789a15f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:31.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.692+0000 7f79667df740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55789a05a3d0 con 0x55789a161ef0 2024-03-28T22:18:31.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.692+0000 7f79667df740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55789a0b7f40 con 0x55789a1589f0 2024-03-28T22:18:31.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.692+0000 7f79667df740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55789a164c80 con 0x55789a159310 2024-03-28T22:18:31.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.692+0000 7f79667df740 10 client.0 _ll_register_callbacks cb 0x55789a077b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:18:31.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.692+0000 7f79621a8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55789a161ef0 0x55789a15f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:31.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.692+0000 7f79619a7640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55789a1589f0 0x55789a15ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:31.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.692+0000 7f79621a8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55789a161ef0 0x55789a15f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47502/0 (socket says 172.21.15.16:47502) 2024-03-28T22:18:31.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.692+0000 7f79621a8640 1 -- 172.21.15.16:0/3265624584 learned_addr learned my addr 172.21.15.16:0/3265624584 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:18:31.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.692+0000 7f79611a6640 1 --2- 172.21.15.16:0/3265624584 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55789a159310 0x55789a15f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:31.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.693+0000 7f79621a8640 1 -- 172.21.15.16:0/3265624584 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55789a159310 msgr2=0x55789a15f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:31.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.693+0000 7f79621a8640 1 --2- 172.21.15.16:0/3265624584 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55789a159310 0x55789a15f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.693+0000 7f79621a8640 1 -- 172.21.15.16:0/3265624584 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55789a1589f0 msgr2=0x55789a15ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:31.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.693+0000 7f79621a8640 1 --2- 172.21.15.16:0/3265624584 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55789a1589f0 0x55789a15ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.693+0000 7f79621a8640 1 -- 172.21.15.16:0/3265624584 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557899ef8f70 con 0x55789a161ef0 2024-03-28T22:18:31.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.693+0000 7f79619a7640 1 --2- 172.21.15.16:0/3265624584 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55789a1589f0 0x55789a15ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:18:31.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.693+0000 7f79621a8640 1 --2- 172.21.15.16:0/3265624584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55789a161ef0 0x55789a15f8b0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f7954012a00 tx=0x7f7954004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=641e937b1650417f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:18:31.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.693+0000 7f79457fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:18:31.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.693+0000 7f79457fa640 1 -- 172.21.15.16:0/3265624584 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7954010c90 con 0x55789a161ef0 2024-03-28T22:18:31.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.693+0000 7f79457fa640 1 -- 172.21.15.16:0/3265624584 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7f7954004b20 con 0x55789a161ef0 2024-03-28T22:18:31.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.693+0000 7f79667df740 1 -- 172.21.15.16:0/3265624584 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f794c02ca80 con 0x55789a161ef0 2024-03-28T22:18:31.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.693+0000 7f79457fa640 1 -- 172.21.15.16:0/3265624584 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7954010e30 con 0x55789a161ef0 2024-03-28T22:18:31.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.693+0000 7f79667df740 10 client.17208 fetch_fsmap learned FSMap version 27 2024-03-28T22:18:31.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.693+0000 7f79667df740 1 -- 172.21.15.16:0/3265624584 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557899d90170 con 0x55789a161ef0 2024-03-28T22:18:31.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.693+0000 7f79457fa640 1 -- 172.21.15.16:0/3265624584 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f794c02ca80 con 0x55789a161ef0 2024-03-28T22:18:31.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.693+0000 7f79457fa640 1 -- 172.21.15.16:0/3265624584 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f7954032480 con 0x55789a161ef0 2024-03-28T22:18:31.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.693+0000 7f79667df740 10 client.17208 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:18:31.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.693+0000 7f79667df740 10 client.17208 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:18:31.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.693+0000 7f79667df740 20 client.17208 populate_metadata read hostname 'smithi016' 2024-03-28T22:18:31.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.693+0000 7f79667df740 1 --2- 172.21.15.16:0/3265624584 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55789a171d10 0x55789a174160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:31.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.693+0000 7f79667df740 4 client.17208 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:18:31.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.693+0000 7f79667df740 1 -- 172.21.15.16:0/3265624584 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55789a0c93e0 con 0x55789a171d10 2024-03-28T22:18:31.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.694+0000 7f79619a7640 1 --2- 172.21.15.16:0/3265624584 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55789a171d10 0x55789a174160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:31.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.694+0000 7f79457fa640 10 client.17208 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:18:31.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.694+0000 7f79619a7640 1 --2- 172.21.15.16:0/3265624584 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55789a171d10 0x55789a174160 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7202e67870fbd29 server_cookie=2eaefa37889f6dba in_seq=0 out_seq=0 2024-03-28T22:18:31.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.695+0000 7f79457fa640 1 -- 172.21.15.16:0/3265624584 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55789a0c93e0 con 0x55789a171d10 2024-03-28T22:18:31.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.695+0000 7f79457fa640 10 client.17208 handle_command_reply: tid=0 2024-03-28T22:18:31.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.695+0000 7f79667df740 1 client.17208 shutdown 2024-03-28T22:18:31.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.695+0000 7f79667df740 1 -- 172.21.15.16:0/3265624584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55789a161ef0 msgr2=0x55789a15f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:31.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.695+0000 7f79667df740 1 --2- 172.21.15.16:0/3265624584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55789a161ef0 0x55789a15f8b0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f7954012a00 tx=0x7f7954004ab0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.695+0000 7f79667df740 1 -- 172.21.15.16:0/3265624584 shutdown_connections 2024-03-28T22:18:31.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.695+0000 7f79667df740 1 --2- 172.21.15.16:0/3265624584 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55789a171d10 0x55789a174160 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.695+0000 7f79667df740 1 --2- 172.21.15.16:0/3265624584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55789a161ef0 0x55789a15f8b0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.695+0000 7f79667df740 1 --2- 172.21.15.16:0/3265624584 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55789a159310 0x55789a15f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.695+0000 7f79457fa640 0 client.17208 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:18:31.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.695+0000 7f79667df740 1 --2- 172.21.15.16:0/3265624584 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55789a1589f0 0x55789a15ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.695+0000 7f79667df740 1 -- 172.21.15.16:0/3265624584 >> 172.21.15.16:0/3265624584 conn(0x55789a153120 msgr2=0x55789a154e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:18:31.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.695+0000 7f79667df740 1 -- 172.21.15.16:0/3265624584 shutdown_connections 2024-03-28T22:18:31.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.695+0000 7f79667df740 1 -- 172.21.15.16:0/3265624584 wait complete. 2024-03-28T22:18:31.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.695+0000 7f79667df740 20 client.17208 trim_cache size 0 max 16384 2024-03-28T22:18:31.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.713+0000 7f79667df740 1 Processor -- start 2024-03-28T22:18:31.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.713+0000 7f79667df740 1 -- start start 2024-03-28T22:18:31.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.713+0000 7f79667df740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55789a1589f0 0x55789a161560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:31.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.713+0000 7f79667df740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55789a159310 0x55789a16d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:31.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.713+0000 7f79667df740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55789a161ef0 0x55789a15dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:31.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.713+0000 7f79667df740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f795c1afb40 con 0x55789a159310 2024-03-28T22:18:31.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.713+0000 7f79667df740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f795c1ad410 con 0x55789a161ef0 2024-03-28T22:18:31.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.713+0000 7f79667df740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f795c1ab280 con 0x55789a1589f0 2024-03-28T22:18:31.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.713+0000 7f79611a6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55789a159310 0x55789a16d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:31.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.713+0000 7f79611a6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55789a159310 0x55789a16d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47504/0 (socket says 172.21.15.16:47504) 2024-03-28T22:18:31.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.713+0000 7f79611a6640 1 -- 172.21.15.16:0/3114131986 learned_addr learned my addr 172.21.15.16:0/3114131986 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:18:31.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.713+0000 7f79621a8640 1 --2- 172.21.15.16:0/3114131986 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55789a161ef0 0x55789a15dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:31.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.713+0000 7f79619a7640 1 --2- 172.21.15.16:0/3114131986 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55789a1589f0 0x55789a161560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:31.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.713+0000 7f79611a6640 1 -- 172.21.15.16:0/3114131986 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55789a1589f0 msgr2=0x55789a161560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:31.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.713+0000 7f79611a6640 1 --2- 172.21.15.16:0/3114131986 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55789a1589f0 0x55789a161560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.713+0000 7f79611a6640 1 -- 172.21.15.16:0/3114131986 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55789a161ef0 msgr2=0x55789a15dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:31.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.713+0000 7f79611a6640 1 --2- 172.21.15.16:0/3114131986 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55789a161ef0 0x55789a15dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.713+0000 7f79611a6640 1 -- 172.21.15.16:0/3114131986 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55789a157ae0 con 0x55789a159310 2024-03-28T22:18:31.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.713+0000 7f79611a6640 1 --2- 172.21.15.16:0/3114131986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55789a159310 0x55789a16d930 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f794c02f020 tx=0x7f794c005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=436709f535e28b8e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:18:31.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.714+0000 7f79477fe640 1 -- 172.21.15.16:0/3114131986 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f794c042040 con 0x55789a159310 2024-03-28T22:18:31.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.714+0000 7f79477fe640 1 -- 172.21.15.16:0/3114131986 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f794c04a020 con 0x55789a159310 2024-03-28T22:18:31.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.714+0000 7f79477fe640 1 -- 172.21.15.16:0/3114131986 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f794c004c20 con 0x55789a159310 2024-03-28T22:18:31.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.714+0000 7f79667df740 1 -- 172.21.15.16:0/3114131986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55789a159310 msgr2=0x55789a16d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:31.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.714+0000 7f79667df740 1 --2- 172.21.15.16:0/3114131986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55789a159310 0x55789a16d930 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f794c02f020 tx=0x7f794c005850 comp rx=0 tx=0).stop 2024-03-28T22:18:31.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.714+0000 7f79667df740 1 -- 172.21.15.16:0/3114131986 shutdown_connections 2024-03-28T22:18:31.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.714+0000 7f79667df740 1 --2- 172.21.15.16:0/3114131986 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55789a161ef0 0x55789a15dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.714+0000 7f79667df740 1 --2- 172.21.15.16:0/3114131986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55789a159310 0x55789a16d930 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.714+0000 7f79667df740 1 --2- 172.21.15.16:0/3114131986 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55789a1589f0 0x55789a161560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.714+0000 7f79667df740 1 -- 172.21.15.16:0/3114131986 >> 172.21.15.16:0/3114131986 conn(0x55789a153120 msgr2=0x55789a15d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:18:31.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.714+0000 7f79667df740 1 -- 172.21.15.16:0/3114131986 shutdown_connections 2024-03-28T22:18:31.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.714+0000 7f79667df740 1 -- 172.21.15.16:0/3114131986 wait complete. 2024-03-28T22:18:31.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.715+0000 7f79667df740 1 -- start start 2024-03-28T22:18:31.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.715+0000 7f79667df740 1 Processor -- start 2024-03-28T22:18:31.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.715+0000 7f79667df740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55789a1589f0 0x55789a15bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:31.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.715+0000 7f79667df740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55789a159310 0x55789a15c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:31.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.715+0000 7f79667df740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55789a161ef0 0x55789a1d7060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:31.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.715+0000 7f79667df740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f795c1ab8d0 con 0x55789a161ef0 2024-03-28T22:18:31.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.715+0000 7f79667df740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f795c1acd40 con 0x55789a159310 2024-03-28T22:18:31.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.715+0000 7f79667df740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f795c1abfa0 con 0x55789a1589f0 2024-03-28T22:18:31.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.715+0000 7f79667df740 10 client.0 _ll_register_callbacks cb 0x7f79540123e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:18:31.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.715+0000 7f79621a8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55789a161ef0 0x55789a1d7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:31.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.715+0000 7f79621a8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55789a161ef0 0x55789a1d7060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47510/0 (socket says 172.21.15.16:47510) 2024-03-28T22:18:31.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.715+0000 7f79621a8640 1 -- 172.21.15.16:0/2851278811 learned_addr learned my addr 172.21.15.16:0/2851278811 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:18:31.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.715+0000 7f79611a6640 1 --2- 172.21.15.16:0/2851278811 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55789a159310 0x55789a15c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:31.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.715+0000 7f79619a7640 1 --2- 172.21.15.16:0/2851278811 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55789a1589f0 0x55789a15bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:31.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.715+0000 7f79621a8640 1 -- 172.21.15.16:0/2851278811 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55789a1589f0 msgr2=0x55789a15bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:31.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.715+0000 7f79621a8640 1 --2- 172.21.15.16:0/2851278811 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55789a1589f0 0x55789a15bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.715+0000 7f79621a8640 1 -- 172.21.15.16:0/2851278811 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55789a159310 msgr2=0x55789a15c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:31.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.715+0000 7f79621a8640 1 --2- 172.21.15.16:0/2851278811 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55789a159310 0x55789a15c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.716+0000 7f79621a8640 1 -- 172.21.15.16:0/2851278811 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55789a1d75a0 con 0x55789a161ef0 2024-03-28T22:18:31.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.716+0000 7f79611a6640 1 --2- 172.21.15.16:0/2851278811 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55789a159310 0x55789a15c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:18:31.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.716+0000 7f79621a8640 1 --2- 172.21.15.16:0/2851278811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55789a161ef0 0x55789a1d7060 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f7954032a90 tx=0x7f79540326b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=326877958764c8b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:18:31.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.716+0000 7f79457fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:18:31.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.716+0000 7f79457fa640 1 -- 172.21.15.16:0/2851278811 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7954010c90 con 0x55789a161ef0 2024-03-28T22:18:31.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.716+0000 7f79457fa640 1 -- 172.21.15.16:0/2851278811 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7f7954032d20 con 0x55789a161ef0 2024-03-28T22:18:31.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.716+0000 7f79667df740 1 -- 172.21.15.16:0/2851278811 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f794c04a020 con 0x55789a161ef0 2024-03-28T22:18:31.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.716+0000 7f79457fa640 1 -- 172.21.15.16:0/2851278811 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7954042050 con 0x55789a161ef0 2024-03-28T22:18:31.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.716+0000 7f79667df740 10 client.17214 fetch_fsmap learned FSMap version 27 2024-03-28T22:18:31.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.716+0000 7f79667df740 1 -- 172.21.15.16:0/2851278811 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55789a1d8780 con 0x55789a161ef0 2024-03-28T22:18:31.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.716+0000 7f79457fa640 1 -- 172.21.15.16:0/2851278811 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f794c04a020 con 0x55789a161ef0 2024-03-28T22:18:31.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.716+0000 7f79457fa640 1 -- 172.21.15.16:0/2851278811 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f7954004370 con 0x55789a161ef0 2024-03-28T22:18:31.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.716+0000 7f79667df740 10 client.17214 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:18:31.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.716+0000 7f79667df740 10 client.17214 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:18:31.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.716+0000 7f79667df740 20 client.17214 populate_metadata read hostname 'smithi016' 2024-03-28T22:18:31.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.716+0000 7f79667df740 1 --2- 172.21.15.16:0/2851278811 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55789a1d8ea0 0x55789a1db2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:31.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.716+0000 7f79667df740 4 client.17214 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:18:31.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.716+0000 7f79667df740 1 -- 172.21.15.16:0/2851278811 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55789a0c3730 con 0x55789a1d8ea0 2024-03-28T22:18:31.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.717+0000 7f79619a7640 1 --2- 172.21.15.16:0/2851278811 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55789a1d8ea0 0x55789a1db2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:31.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.717+0000 7f79457fa640 10 client.17214 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:18:31.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.717+0000 7f79619a7640 1 --2- 172.21.15.16:0/2851278811 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55789a1d8ea0 0x55789a1db2c0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df7d9fab63085fcf server_cookie=5419c8c0812fe602 in_seq=0 out_seq=0 2024-03-28T22:18:31.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.718+0000 7f79457fa640 1 -- 172.21.15.16:0/2851278811 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55789a0c3730 con 0x55789a1d8ea0 2024-03-28T22:18:31.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.718+0000 7f79457fa640 10 client.17214 handle_command_reply: tid=0 2024-03-28T22:18:31.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.718+0000 7f79667df740 1 client.17214 shutdown 2024-03-28T22:18:31.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.719+0000 7f79667df740 1 -- 172.21.15.16:0/2851278811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55789a161ef0 msgr2=0x55789a1d7060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:31.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.719+0000 7f79667df740 1 --2- 172.21.15.16:0/2851278811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55789a161ef0 0x55789a1d7060 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f7954032a90 tx=0x7f79540326b0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.719+0000 7f79667df740 1 -- 172.21.15.16:0/2851278811 shutdown_connections 2024-03-28T22:18:31.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.719+0000 7f79667df740 1 --2- 172.21.15.16:0/2851278811 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55789a1d8ea0 0x55789a1db2c0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.719+0000 7f79667df740 1 --2- 172.21.15.16:0/2851278811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55789a161ef0 0x55789a1d7060 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.719+0000 7f79667df740 1 --2- 172.21.15.16:0/2851278811 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55789a159310 0x55789a15c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.719+0000 7f79457fa640 0 client.17214 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:18:31.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.719+0000 7f79667df740 1 --2- 172.21.15.16:0/2851278811 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55789a1589f0 0x55789a15bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.719+0000 7f79667df740 1 -- 172.21.15.16:0/2851278811 >> 172.21.15.16:0/2851278811 conn(0x55789a153120 msgr2=0x55789a15f680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:18:31.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.719+0000 7f79667df740 1 -- 172.21.15.16:0/2851278811 shutdown_connections 2024-03-28T22:18:31.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.719+0000 7f79667df740 1 -- 172.21.15.16:0/2851278811 wait complete. 2024-03-28T22:18:31.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.719+0000 7f79667df740 20 client.17214 trim_cache size 0 max 16384 2024-03-28T22:18:31.745 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:18:31.746 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-03-28T22:18:31.746 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "0de73dc8-8346-40d1-b6a4-15b31b742d27", 2024-03-28T22:18:31.746 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-03-28T22:18:31.746 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:18:31.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.719+0000 7f7963c0b640 1 -- 172.21.15.16:0/327014322 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f79480a1ef0 msgr2=0x7f79480a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:31.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.719+0000 7f7963c0b640 1 --2- 172.21.15.16:0/327014322 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f79480a1ef0 0x7f79480a43b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7954009fd0 tx=0x7f79540091e0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.719+0000 7f7963c0b640 1 -- 172.21.15.16:0/327014322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f795c0ffc60 msgr2=0x7f795c197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:31.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.719+0000 7f7963c0b640 1 --2- 172.21.15.16:0/327014322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f795c0ffc60 0x7f795c197fc0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f795800e9f0 tx=0x7f795800edf0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.719+0000 7f7963c0b640 1 -- 172.21.15.16:0/327014322 shutdown_connections 2024-03-28T22:18:31.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.719+0000 7f7963c0b640 1 --2- 172.21.15.16:0/327014322 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f79480a1ef0 0x7f79480a43b0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.719+0000 7f7963c0b640 1 --2- 172.21.15.16:0/327014322 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f795c1076f0 0x7f795c19c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.719+0000 7f7963c0b640 1 --2- 172.21.15.16:0/327014322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f795c100620 0x7f795c198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.719+0000 7f7963c0b640 1 --2- 172.21.15.16:0/327014322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f795c0ffc60 0x7f795c197fc0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.719+0000 7f7963c0b640 1 -- 172.21.15.16:0/327014322 >> 172.21.15.16:0/327014322 conn(0x7f795c076b50 msgr2=0x7f795c0fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:18:31.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.720+0000 7f7963c0b640 1 -- 172.21.15.16:0/327014322 shutdown_connections 2024-03-28T22:18:31.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.720+0000 7f7963c0b640 1 -- 172.21.15.16:0/327014322 wait complete. 2024-03-28T22:18:31.749 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:18:31.750 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:18:31.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.855+0000 7f9aa359e640 1 -- 172.21.15.16:0/3734027448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9aa4107600 msgr2=0x7f9aa41079e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:31.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.856+0000 7f9aa359e640 1 --2- 172.21.15.16:0/3734027448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9aa4107600 0x7f9aa41079e0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f9a94009900 tx=0x7f9a9402bc30 comp rx=0 tx=0).stop 2024-03-28T22:18:31.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.856+0000 7f9aa359e640 1 -- 172.21.15.16:0/3734027448 shutdown_connections 2024-03-28T22:18:31.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.856+0000 7f9aa359e640 1 --2- 172.21.15.16:0/3734027448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9aa4101f00 0x7f9aa410e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.856+0000 7f9aa359e640 1 --2- 172.21.15.16:0/3734027448 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9aa4101540 0x7f9aa41019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.856+0000 7f9aa359e640 1 --2- 172.21.15.16:0/3734027448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9aa4107600 0x7f9aa41079e0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.856+0000 7f9aa359e640 1 -- 172.21.15.16:0/3734027448 >> 172.21.15.16:0/3734027448 conn(0x7f9aa40fd030 msgr2=0x7f9aa40ff450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:18:31.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.856+0000 7f9aa359e640 1 -- 172.21.15.16:0/3734027448 shutdown_connections 2024-03-28T22:18:31.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.856+0000 7f9aa359e640 1 -- 172.21.15.16:0/3734027448 wait complete. 2024-03-28T22:18:31.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.856+0000 7f9aa359e640 1 Processor -- start 2024-03-28T22:18:31.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.857+0000 7f9aa359e640 1 -- start start 2024-03-28T22:18:31.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.857+0000 7f9aa359e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9aa4101540 0x7f9aa4197d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:31.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.857+0000 7f9aa359e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9aa4101f00 0x7f9aa41982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:31.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.857+0000 7f9aa359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9aa4107600 0x7f9aa419c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:31.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.857+0000 7f9aa359e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa4111930 con 0x7f9aa4107600 2024-03-28T22:18:31.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.857+0000 7f9aa359e640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa41117b0 con 0x7f9aa4101f00 2024-03-28T22:18:31.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.857+0000 7f9aa359e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa4111ab0 con 0x7f9aa4101540 2024-03-28T22:18:31.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.857+0000 7f9aa2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9aa4107600 0x7f9aa419c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:31.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.857+0000 7f9aa1d9b640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9aa4101f00 0x7f9aa41982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:31.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.857+0000 7f9aa2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9aa4107600 0x7f9aa419c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47522/0 (socket says 172.21.15.16:47522) 2024-03-28T22:18:31.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.857+0000 7f9aa2d9d640 1 -- 172.21.15.16:0/3781940784 learned_addr learned my addr 172.21.15.16:0/3781940784 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:18:31.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.857+0000 7f9aa259c640 1 --2- 172.21.15.16:0/3781940784 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9aa4101540 0x7f9aa4197d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:31.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.857+0000 7f9aa2d9d640 1 -- 172.21.15.16:0/3781940784 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9aa4101540 msgr2=0x7f9aa4197d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:31.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.857+0000 7f9aa2d9d640 1 --2- 172.21.15.16:0/3781940784 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9aa4101540 0x7f9aa4197d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.857+0000 7f9aa2d9d640 1 -- 172.21.15.16:0/3781940784 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9aa4101f00 msgr2=0x7f9aa41982d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:31.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.857+0000 7f9aa2d9d640 1 --2- 172.21.15.16:0/3781940784 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9aa4101f00 0x7f9aa41982d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.857+0000 7f9aa2d9d640 1 -- 172.21.15.16:0/3781940784 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aa419cd80 con 0x7f9aa4107600 2024-03-28T22:18:31.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.857+0000 7f9aa259c640 1 --2- 172.21.15.16:0/3781940784 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9aa4101540 0x7f9aa4197d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:18:31.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.858+0000 7f9aa2d9d640 1 --2- 172.21.15.16:0/3781940784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9aa4107600 0x7f9aa419c650 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c00eaa0 tx=0x7f9a9c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:18:31.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.858+0000 7f9a937fe640 1 -- 172.21.15.16:0/3781940784 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c00cdb0 con 0x7f9aa4107600 2024-03-28T22:18:31.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.858+0000 7f9aa359e640 1 -- 172.21.15.16:0/3781940784 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9aa419d070 con 0x7f9aa4107600 2024-03-28T22:18:31.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.858+0000 7f9a937fe640 1 -- 172.21.15.16:0/3781940784 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a9c005020 con 0x7f9aa4107600 2024-03-28T22:18:31.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.858+0000 7f9a937fe640 1 -- 172.21.15.16:0/3781940784 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c0116b0 con 0x7f9aa4107600 2024-03-28T22:18:31.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.858+0000 7f9aa359e640 1 -- 172.21.15.16:0/3781940784 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9aa41a4900 con 0x7f9aa4107600 2024-03-28T22:18:31.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.861+0000 7f9a937fe640 1 -- 172.21.15.16:0/3781940784 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f9a9c005560 con 0x7f9aa4107600 2024-03-28T22:18:31.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.861+0000 7f9aab9df740 1 Processor -- start 2024-03-28T22:18:31.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.861+0000 7f9aab9df740 1 -- start start 2024-03-28T22:18:31.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.862+0000 7f9aab9df740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5587edd53470 0x5587edd53850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:31.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.862+0000 7f9aab9df740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587edd53d90 0x5587edd5c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:31.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.862+0000 7f9a937fe640 1 --2- 172.21.15.16:0/3781940784 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f9a8c0a1ef0 0x7f9a8c0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:31.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.862+0000 7f9aab9df740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587edd5c970 0x5587edd5ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:31.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.862+0000 7f9aab9df740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5587edcb7180 con 0x5587edd5c970 2024-03-28T22:18:31.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.862+0000 7f9aab9df740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5587edc5fb70 con 0x5587edd53470 2024-03-28T22:18:31.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.862+0000 7f9aab9df740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5587edcb3220 con 0x5587edd53d90 2024-03-28T22:18:31.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.862+0000 7f9aa259c640 1 --2- 172.21.15.16:0/3781940784 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f9a8c0a1ef0 0x7f9a8c0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:31.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.862+0000 7f9aa2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587edd5c970 0x5587edd5ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:31.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.862+0000 7f9aa1d9b640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5587edd53470 0x5587edd53850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:31.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.862+0000 7f9aa1d9b640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5587edd53470 0x5587edd53850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:44034/0 (socket says 172.21.15.16:44034) 2024-03-28T22:18:31.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.862+0000 7f9aa1d9b640 1 -- 172.21.15.16:0/1698931978 learned_addr learned my addr 172.21.15.16:0/1698931978 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:18:31.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.862+0000 7f9aa1d9b640 1 --2- 172.21.15.16:0/1698931978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587edd53d90 0x5587edd5c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:31.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.862+0000 7f9a937fe640 1 -- 172.21.15.16:0/3781940784 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7f9a9c0ce400 con 0x7f9aa4107600 2024-03-28T22:18:31.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.862+0000 7f9aa2d9d640 1 -- 172.21.15.16:0/1698931978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587edd53d90 msgr2=0x5587edd5c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:31.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.862+0000 7f9aa2d9d640 1 --2- 172.21.15.16:0/1698931978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587edd53d90 0x5587edd5c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.862+0000 7f9aa2d9d640 1 -- 172.21.15.16:0/1698931978 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5587edd53470 msgr2=0x5587edd53850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:31.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.862+0000 7f9aa2d9d640 1 --2- 172.21.15.16:0/1698931978 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5587edd53470 0x5587edd53850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.862+0000 7f9aa2d9d640 1 -- 172.21.15.16:0/1698931978 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587edc737c0 con 0x5587edd5c970 2024-03-28T22:18:31.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.863+0000 7f9aa259c640 1 --2- 172.21.15.16:0/3781940784 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f9a8c0a1ef0 0x7f9a8c0a43b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9a94006010 tx=0x7f9a94005a20 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:18:31.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.863+0000 7f9aa2d9d640 1 --2- 172.21.15.16:0/1698931978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587edd5c970 0x5587edd5ed60 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c0cbb50 tx=0x7f9a9c015610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=329f115c07ddcfd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:18:31.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.863+0000 7f9a90ff9640 1 -- 172.21.15.16:0/1698931978 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c0b3b20 con 0x5587edd5c970 2024-03-28T22:18:31.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.863+0000 7f9a90ff9640 1 -- 172.21.15.16:0/1698931978 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a9c09b4f0 con 0x5587edd5c970 2024-03-28T22:18:31.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.863+0000 7f9a90ff9640 1 -- 172.21.15.16:0/1698931978 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c097460 con 0x5587edd5c970 2024-03-28T22:18:31.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.864+0000 7f9aab9df740 1 -- 172.21.15.16:0/1698931978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587edd5c970 msgr2=0x5587edd5ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:31.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.864+0000 7f9aab9df740 1 --2- 172.21.15.16:0/1698931978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587edd5c970 0x5587edd5ed60 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c0cbb50 tx=0x7f9a9c015610 comp rx=0 tx=0).stop 2024-03-28T22:18:31.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.864+0000 7f9aab9df740 1 -- 172.21.15.16:0/1698931978 shutdown_connections 2024-03-28T22:18:31.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.864+0000 7f9aab9df740 1 --2- 172.21.15.16:0/1698931978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587edd5c970 0x5587edd5ed60 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.864+0000 7f9aab9df740 1 --2- 172.21.15.16:0/1698931978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587edd53d90 0x5587edd5c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.864+0000 7f9aab9df740 1 --2- 172.21.15.16:0/1698931978 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5587edd53470 0x5587edd53850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.864+0000 7f9aab9df740 1 -- 172.21.15.16:0/1698931978 >> 172.21.15.16:0/1698931978 conn(0x5587edd4dba0 msgr2=0x5587edd4df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:18:31.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.864+0000 7f9aab9df740 1 -- 172.21.15.16:0/1698931978 shutdown_connections 2024-03-28T22:18:31.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.864+0000 7f9aab9df740 1 -- 172.21.15.16:0/1698931978 wait complete. 2024-03-28T22:18:31.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.864+0000 7f9aab9df740 1 -- start start 2024-03-28T22:18:31.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.864+0000 7f9aab9df740 1 Processor -- start 2024-03-28T22:18:31.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.865+0000 7f9aab9df740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5587edd53470 0x5587edd59a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:31.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.865+0000 7f9aab9df740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587edd53d90 0x5587edd59f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:31.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.865+0000 7f9aab9df740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587edd5c970 0x5587edd5a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:31.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.865+0000 7f9aab9df740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5587edd5f760 con 0x5587edd5c970 2024-03-28T22:18:31.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.865+0000 7f9aab9df740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5587edc554a0 con 0x5587edd53470 2024-03-28T22:18:31.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.865+0000 7f9aab9df740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5587edd5f8e0 con 0x5587edd53d90 2024-03-28T22:18:31.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.865+0000 7f9aab9df740 10 client.0 _ll_register_callbacks cb 0x5587edc58d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:18:31.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.865+0000 7f9aa2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587edd5c970 0x5587edd5a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:31.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.865+0000 7f9aa2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587edd5c970 0x5587edd5a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47546/0 (socket says 172.21.15.16:47546) 2024-03-28T22:18:31.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.865+0000 7f9aa2d9d640 1 -- 172.21.15.16:0/2818521961 learned_addr learned my addr 172.21.15.16:0/2818521961 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:18:31.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.865+0000 7f9aa1d9b640 1 --2- 172.21.15.16:0/2818521961 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587edd53d90 0x5587edd59f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:31.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.865+0000 7f9aa259c640 1 --2- 172.21.15.16:0/2818521961 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5587edd53470 0x5587edd59a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:31.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.865+0000 7f9aa2d9d640 1 -- 172.21.15.16:0/2818521961 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587edd53d90 msgr2=0x5587edd59f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:31.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.865+0000 7f9aa2d9d640 1 --2- 172.21.15.16:0/2818521961 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587edd53d90 0x5587edd59f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.865+0000 7f9aa2d9d640 1 -- 172.21.15.16:0/2818521961 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5587edd53470 msgr2=0x5587edd59a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:31.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.865+0000 7f9aa2d9d640 1 --2- 172.21.15.16:0/2818521961 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5587edd53470 0x5587edd59a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.865+0000 7f9aa2d9d640 1 -- 172.21.15.16:0/2818521961 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587eda957d0 con 0x5587edd5c970 2024-03-28T22:18:31.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.865+0000 7f9aa2d9d640 1 --2- 172.21.15.16:0/2818521961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587edd5c970 0x5587edd5a4c0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c0990a0 tx=0x7f9a9c097670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2052446e70ccb63f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:18:31.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.865+0000 7f9a89ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:18:31.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.865+0000 7f9a89ffb640 1 -- 172.21.15.16:0/2818521961 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c0b3b20 con 0x5587edd5c970 2024-03-28T22:18:31.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.865+0000 7f9a89ffb640 1 -- 172.21.15.16:0/2818521961 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7f9a9c0153c0 con 0x5587edd5c970 2024-03-28T22:18:31.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.865+0000 7f9aab9df740 1 -- 172.21.15.16:0/2818521961 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a9c09b4f0 con 0x5587edd5c970 2024-03-28T22:18:31.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.866+0000 7f9a89ffb640 1 -- 172.21.15.16:0/2818521961 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9a9c09ef40 con 0x5587edd5c970 2024-03-28T22:18:31.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.866+0000 7f9aab9df740 10 client.17226 fetch_fsmap learned FSMap version 27 2024-03-28T22:18:31.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.866+0000 7f9aab9df740 1 -- 172.21.15.16:0/2818521961 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5587ed9cb3f0 con 0x5587edd5c970 2024-03-28T22:18:31.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.866+0000 7f9a89ffb640 1 -- 172.21.15.16:0/2818521961 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a9c09b4f0 con 0x5587edd5c970 2024-03-28T22:18:31.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.866+0000 7f9a89ffb640 1 -- 172.21.15.16:0/2818521961 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f9a9c00f040 con 0x5587edd5c970 2024-03-28T22:18:31.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.866+0000 7f9aab9df740 10 client.17226 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:18:31.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.866+0000 7f9aab9df740 10 client.17226 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:18:31.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.866+0000 7f9aab9df740 20 client.17226 populate_metadata read hostname 'smithi016' 2024-03-28T22:18:31.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.866+0000 7f9aab9df740 1 --2- 172.21.15.16:0/2818521961 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5587edd6cc10 0x5587edd6f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:31.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.866+0000 7f9aab9df740 4 client.17226 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:18:31.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.866+0000 7f9aab9df740 1 -- 172.21.15.16:0/2818521961 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5587edcc46c0 con 0x5587edd6cc10 2024-03-28T22:18:31.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.866+0000 7f9aa259c640 1 --2- 172.21.15.16:0/2818521961 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5587edd6cc10 0x5587edd6f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:31.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.867+0000 7f9aa259c640 1 --2- 172.21.15.16:0/2818521961 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5587edd6cc10 0x5587edd6f000 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=500b268157eebea3 server_cookie=f535fde9828e330 in_seq=0 out_seq=0 2024-03-28T22:18:31.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.867+0000 7f9a89ffb640 10 client.17226 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:18:31.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.868+0000 7f9a89ffb640 1 -- 172.21.15.16:0/2818521961 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5587edcc46c0 con 0x5587edd6cc10 2024-03-28T22:18:31.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.868+0000 7f9a89ffb640 10 client.17226 handle_command_reply: tid=0 2024-03-28T22:18:31.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.868+0000 7f9aab9df740 1 client.17226 shutdown 2024-03-28T22:18:31.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.868+0000 7f9aab9df740 1 -- 172.21.15.16:0/2818521961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587edd5c970 msgr2=0x5587edd5a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:31.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.868+0000 7f9aab9df740 1 --2- 172.21.15.16:0/2818521961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587edd5c970 0x5587edd5a4c0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c0990a0 tx=0x7f9a9c097670 comp rx=0 tx=0).stop 2024-03-28T22:18:31.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.868+0000 7f9aab9df740 1 -- 172.21.15.16:0/2818521961 shutdown_connections 2024-03-28T22:18:31.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.868+0000 7f9aab9df740 1 --2- 172.21.15.16:0/2818521961 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5587edd6cc10 0x5587edd6f000 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.868+0000 7f9aab9df740 1 --2- 172.21.15.16:0/2818521961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587edd5c970 0x5587edd5a4c0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.868+0000 7f9aab9df740 1 --2- 172.21.15.16:0/2818521961 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587edd53d90 0x5587edd59f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.868+0000 7f9aab9df740 1 --2- 172.21.15.16:0/2818521961 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5587edd53470 0x5587edd59a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.868+0000 7f9aab9df740 1 -- 172.21.15.16:0/2818521961 >> 172.21.15.16:0/2818521961 conn(0x5587edd4dba0 msgr2=0x5587edd4f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:18:31.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.868+0000 7f9a89ffb640 0 client.17226 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:18:31.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.868+0000 7f9aab9df740 1 -- 172.21.15.16:0/2818521961 shutdown_connections 2024-03-28T22:18:31.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.868+0000 7f9aab9df740 1 -- 172.21.15.16:0/2818521961 wait complete. 2024-03-28T22:18:31.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.868+0000 7f9aab9df740 20 client.17226 trim_cache size 0 max 16384 2024-03-28T22:18:31.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.888+0000 7f9aab9df740 1 Processor -- start 2024-03-28T22:18:31.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.888+0000 7f9aab9df740 1 -- start start 2024-03-28T22:18:31.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.888+0000 7f9aab9df740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587edd53470 0x5587edd5c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:31.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.888+0000 7f9aab9df740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5587edd53d90 0x5587edd58ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:31.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.888+0000 7f9aab9df740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587edd5c970 0x5587edd68240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:31.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.888+0000 7f9aab9df740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa41afa90 con 0x5587edd53470 2024-03-28T22:18:31.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.888+0000 7f9aab9df740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa41ad360 con 0x5587edd53d90 2024-03-28T22:18:31.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.888+0000 7f9aab9df740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa41ab1d0 con 0x5587edd5c970 2024-03-28T22:18:31.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.888+0000 7f9aa259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587edd53470 0x5587edd5c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:31.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.888+0000 7f9aa259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587edd53470 0x5587edd5c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47554/0 (socket says 172.21.15.16:47554) 2024-03-28T22:18:31.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.888+0000 7f9aa259c640 1 -- 172.21.15.16:0/260064672 learned_addr learned my addr 172.21.15.16:0/260064672 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:18:31.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.888+0000 7f9aa1d9b640 1 --2- 172.21.15.16:0/260064672 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5587edd53d90 0x5587edd58ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:31.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.888+0000 7f9aa2d9d640 1 --2- 172.21.15.16:0/260064672 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587edd5c970 0x5587edd68240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:31.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.888+0000 7f9aa259c640 1 -- 172.21.15.16:0/260064672 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587edd5c970 msgr2=0x5587edd68240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:31.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.888+0000 7f9aa259c640 1 --2- 172.21.15.16:0/260064672 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587edd5c970 0x5587edd68240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.888+0000 7f9aa259c640 1 -- 172.21.15.16:0/260064672 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5587edd53d90 msgr2=0x5587edd58ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:31.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.888+0000 7f9aa259c640 1 --2- 172.21.15.16:0/260064672 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5587edd53d90 0x5587edd58ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.888+0000 7f9aa259c640 1 -- 172.21.15.16:0/260064672 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587edd53bc0 con 0x5587edd53470 2024-03-28T22:18:31.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.889+0000 7f9aa259c640 1 --2- 172.21.15.16:0/260064672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587edd53470 0x5587edd5c170 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f9a9402efd0 tx=0x7f9a9403d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=621dcb7b9bbeb116 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:18:31.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.889+0000 7f9a8bfff640 1 -- 172.21.15.16:0/260064672 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a94005240 con 0x5587edd53470 2024-03-28T22:18:31.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.889+0000 7f9a8bfff640 1 -- 172.21.15.16:0/260064672 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a94054070 con 0x5587edd53470 2024-03-28T22:18:31.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.889+0000 7f9a8bfff640 1 -- 172.21.15.16:0/260064672 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a94004920 con 0x5587edd53470 2024-03-28T22:18:31.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.890+0000 7f9aab9df740 1 -- 172.21.15.16:0/260064672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587edd53470 msgr2=0x5587edd5c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:31.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.890+0000 7f9aab9df740 1 --2- 172.21.15.16:0/260064672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587edd53470 0x5587edd5c170 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f9a9402efd0 tx=0x7f9a9403d000 comp rx=0 tx=0).stop 2024-03-28T22:18:31.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.890+0000 7f9aab9df740 1 -- 172.21.15.16:0/260064672 shutdown_connections 2024-03-28T22:18:31.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.890+0000 7f9aab9df740 1 --2- 172.21.15.16:0/260064672 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587edd5c970 0x5587edd68240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.890+0000 7f9aab9df740 1 --2- 172.21.15.16:0/260064672 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5587edd53d90 0x5587edd58ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.890+0000 7f9aab9df740 1 --2- 172.21.15.16:0/260064672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587edd53470 0x5587edd5c170 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.890+0000 7f9aab9df740 1 -- 172.21.15.16:0/260064672 >> 172.21.15.16:0/260064672 conn(0x5587edd4dba0 msgr2=0x5587edd67b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:18:31.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.890+0000 7f9aab9df740 1 -- 172.21.15.16:0/260064672 shutdown_connections 2024-03-28T22:18:31.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.890+0000 7f9aab9df740 1 -- 172.21.15.16:0/260064672 wait complete. 2024-03-28T22:18:31.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.890+0000 7f9aab9df740 1 -- start start 2024-03-28T22:18:31.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.890+0000 7f9aab9df740 1 Processor -- start 2024-03-28T22:18:31.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.890+0000 7f9aab9df740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587edd53470 0x5587edd71eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:31.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.890+0000 7f9aab9df740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5587edd53d90 0x5587edd6f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:31.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.890+0000 7f9aab9df740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587edd5c970 0x5587edd6f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:31.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.890+0000 7f9aab9df740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa41ab820 con 0x5587edd53470 2024-03-28T22:18:31.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.890+0000 7f9aab9df740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa41acc90 con 0x5587edd53d90 2024-03-28T22:18:31.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.890+0000 7f9aab9df740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa41abef0 con 0x5587edd5c970 2024-03-28T22:18:31.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.890+0000 7f9aab9df740 10 client.0 _ll_register_callbacks cb 0x7f9a98009ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:18:31.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.890+0000 7f9aa259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587edd53470 0x5587edd71eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:31.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.890+0000 7f9aa259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587edd53470 0x5587edd71eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47556/0 (socket says 172.21.15.16:47556) 2024-03-28T22:18:31.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.890+0000 7f9aa259c640 1 -- 172.21.15.16:0/4174763629 learned_addr learned my addr 172.21.15.16:0/4174763629 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:18:31.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.891+0000 7f9aa2d9d640 1 --2- 172.21.15.16:0/4174763629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587edd5c970 0x5587edd6f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:31.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.891+0000 7f9aa1d9b640 1 --2- 172.21.15.16:0/4174763629 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5587edd53d90 0x5587edd6f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:31.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.891+0000 7f9aa259c640 1 -- 172.21.15.16:0/4174763629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587edd5c970 msgr2=0x5587edd6f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:31.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.891+0000 7f9aa259c640 1 --2- 172.21.15.16:0/4174763629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587edd5c970 0x5587edd6f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.891+0000 7f9aa259c640 1 -- 172.21.15.16:0/4174763629 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5587edd53d90 msgr2=0x5587edd6f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:18:31.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.891+0000 7f9aa259c640 1 --2- 172.21.15.16:0/4174763629 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5587edd53d90 0x5587edd6f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.891+0000 7f9aa259c640 1 -- 172.21.15.16:0/4174763629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587ed98b0f0 con 0x5587edd53470 2024-03-28T22:18:31.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.891+0000 7f9aa259c640 1 --2- 172.21.15.16:0/4174763629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587edd53470 0x5587edd71eb0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f9a9402cee0 tx=0x7f9a94004350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=408eab3baadf82d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:18:31.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.891+0000 7f9a89ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:18:31.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.891+0000 7f9a89ffb640 1 -- 172.21.15.16:0/4174763629 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a9404f300 con 0x5587edd53470 2024-03-28T22:18:31.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.891+0000 7f9a89ffb640 1 -- 172.21.15.16:0/4174763629 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7f9a9404f4a0 con 0x5587edd53470 2024-03-28T22:18:31.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.891+0000 7f9aab9df740 1 -- 172.21.15.16:0/4174763629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a94054070 con 0x5587edd53470 2024-03-28T22:18:31.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.891+0000 7f9a89ffb640 1 -- 172.21.15.16:0/4174763629 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9a94068050 con 0x5587edd53470 2024-03-28T22:18:31.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.891+0000 7f9aab9df740 10 client.17232 fetch_fsmap learned FSMap version 27 2024-03-28T22:18:31.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.891+0000 7f9aab9df740 1 -- 172.21.15.16:0/4174763629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5587eddd29c0 con 0x5587edd53470 2024-03-28T22:18:31.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.892+0000 7f9a89ffb640 1 -- 172.21.15.16:0/4174763629 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a94054070 con 0x5587edd53470 2024-03-28T22:18:31.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.892+0000 7f9a89ffb640 1 -- 172.21.15.16:0/4174763629 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f9a94049040 con 0x5587edd53470 2024-03-28T22:18:31.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.892+0000 7f9aab9df740 10 client.17232 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:18:31.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.892+0000 7f9aab9df740 10 client.17232 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:18:31.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.892+0000 7f9aab9df740 20 client.17232 populate_metadata read hostname 'smithi016' 2024-03-28T22:18:31.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.892+0000 7f9aab9df740 1 --2- 172.21.15.16:0/4174763629 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5587eddd3050 0x5587eddd5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:31.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.892+0000 7f9aab9df740 4 client.17232 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:18:31.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.892+0000 7f9aab9df740 1 -- 172.21.15.16:0/4174763629 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5587edcbea10 con 0x5587eddd3050 2024-03-28T22:18:31.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.892+0000 7f9aa1d9b640 1 --2- 172.21.15.16:0/4174763629 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5587eddd3050 0x5587eddd5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:18:31.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.893+0000 7f9aa1d9b640 1 --2- 172.21.15.16:0/4174763629 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5587eddd3050 0x5587eddd5470 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5a613925e091be6 server_cookie=f6139dcbab6ecf85 in_seq=0 out_seq=0 2024-03-28T22:18:31.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.893+0000 7f9a89ffb640 10 client.17232 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:18:31.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.893+0000 7f9a89ffb640 1 -- 172.21.15.16:0/4174763629 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5587edcbea10 con 0x5587eddd3050 2024-03-28T22:18:31.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.893+0000 7f9a89ffb640 10 client.17232 handle_command_reply: tid=0 2024-03-28T22:18:31.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.893+0000 7f9aab9df740 1 client.17232 shutdown 2024-03-28T22:18:31.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.893+0000 7f9aab9df740 1 -- 172.21.15.16:0/4174763629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587edd53470 msgr2=0x5587edd71eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:31.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.893+0000 7f9aab9df740 1 --2- 172.21.15.16:0/4174763629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587edd53470 0x5587edd71eb0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f9a9402cee0 tx=0x7f9a94004350 comp rx=0 tx=0).stop 2024-03-28T22:18:31.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.893+0000 7f9aab9df740 1 -- 172.21.15.16:0/4174763629 shutdown_connections 2024-03-28T22:18:31.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.893+0000 7f9aab9df740 1 --2- 172.21.15.16:0/4174763629 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5587eddd3050 0x5587eddd5470 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.893+0000 7f9aab9df740 1 --2- 172.21.15.16:0/4174763629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587edd5c970 0x5587edd6f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.893+0000 7f9aab9df740 1 --2- 172.21.15.16:0/4174763629 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5587edd53d90 0x5587edd6f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.893+0000 7f9aab9df740 1 --2- 172.21.15.16:0/4174763629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5587edd53470 0x5587edd71eb0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.893+0000 7f9aab9df740 1 -- 172.21.15.16:0/4174763629 >> 172.21.15.16:0/4174763629 conn(0x5587edd4dba0 msgr2=0x5587edd683c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:18:31.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.893+0000 7f9a89ffb640 0 client.17232 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:18:31.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.893+0000 7f9aab9df740 1 -- 172.21.15.16:0/4174763629 shutdown_connections 2024-03-28T22:18:31.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.893+0000 7f9aab9df740 1 -- 172.21.15.16:0/4174763629 wait complete. 2024-03-28T22:18:31.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.893+0000 7f9aab9df740 20 client.17232 trim_cache size 0 max 16384 2024-03-28T22:18:31.905 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:18:31.905 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T22:18:31.905 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-03-28T22:18:31.905 INFO:teuthology.orchestra.run.smithi016.stdout: "0de73dc8-8346-40d1-b6a4-15b31b742d27": { 2024-03-28T22:18:31.905 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-03-28T22:18:31.905 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "0de73dc8-8346-40d1-b6a4-15b31b742d27", 2024-03-28T22:18:31.905 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-03-28T22:18:31.906 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:18:31.906 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:18:31.906 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:18:31.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.894+0000 7f9aa359e640 1 -- 172.21.15.16:0/3781940784 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f9a8c0a1ef0 msgr2=0x7f9a8c0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:31.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.894+0000 7f9aa359e640 1 --2- 172.21.15.16:0/3781940784 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f9a8c0a1ef0 0x7f9a8c0a43b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9a94006010 tx=0x7f9a94005a20 comp rx=0 tx=0).stop 2024-03-28T22:18:31.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.894+0000 7f9aa359e640 1 -- 172.21.15.16:0/3781940784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9aa4107600 msgr2=0x7f9aa419c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:18:31.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.894+0000 7f9aa359e640 1 --2- 172.21.15.16:0/3781940784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9aa4107600 0x7f9aa419c650 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c00eaa0 tx=0x7f9a9c00ef60 comp rx=0 tx=0).stop 2024-03-28T22:18:31.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.894+0000 7f9aa359e640 1 -- 172.21.15.16:0/3781940784 shutdown_connections 2024-03-28T22:18:31.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.894+0000 7f9aa359e640 1 --2- 172.21.15.16:0/3781940784 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f9a8c0a1ef0 0x7f9a8c0a43b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.894+0000 7f9aa359e640 1 --2- 172.21.15.16:0/3781940784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9aa4107600 0x7f9aa419c650 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.894+0000 7f9aa359e640 1 --2- 172.21.15.16:0/3781940784 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9aa4101f00 0x7f9aa41982d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.894+0000 7f9aa359e640 1 --2- 172.21.15.16:0/3781940784 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9aa4101540 0x7f9aa4197d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:31.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.894+0000 7f9aa359e640 1 -- 172.21.15.16:0/3781940784 >> 172.21.15.16:0/3781940784 conn(0x7f9aa40fd030 msgr2=0x7f9aa40fe790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:18:31.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.894+0000 7f9aa359e640 1 -- 172.21.15.16:0/3781940784 shutdown_connections 2024-03-28T22:18:31.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:18:31.894+0000 7f9aa359e640 1 -- 172.21.15.16:0/3781940784 wait complete. 2024-03-28T22:18:31.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:31 smithi016 ceph-mon[32374]: pgmap v245: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-03-28T22:18:31.909 INFO:tasks.cephfs.filesystem:scrub status for tag:0de73dc8-8346-40d1-b6a4-15b31b742d27 - {'path': '/', 'tag': '0de73dc8-8346-40d1-b6a4-15b31b742d27', 'options': 'recursive,force'} 2024-03-28T22:18:31.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:31 smithi063 ceph-mon[36249]: pgmap v245: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-03-28T22:18:32.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:31 smithi053 ceph-mon[35979]: pgmap v245: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-03-28T22:18:32.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:32 smithi063 ceph-mon[36249]: scrub queued for path: / 2024-03-28T22:18:32.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:32 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:18:32.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:32 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:18:32.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:32 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:18:32.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:32 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:18:32.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:32 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:18:32.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:32 smithi016 ceph-mon[32374]: scrub queued for path: / 2024-03-28T22:18:32.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:32 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:18:32.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:32 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:18:32.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:32 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:18:32.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:32 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:18:32.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:32 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:18:33.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:32 smithi053 ceph-mon[35979]: scrub queued for path: / 2024-03-28T22:18:33.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:32 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:18:33.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:32 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:18:33.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:32 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:18:33.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:32 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:18:33.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:32 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:18:33.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:33 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:18:33.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:33 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:18:33.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:33 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:18:33.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:33 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:18:33.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:33 smithi063 ceph-mon[36249]: pgmap v246: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:18:33.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:33 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:18:33.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:33 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:18:33.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:33 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:18:33.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:33 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:18:33.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:33 smithi016 ceph-mon[32374]: pgmap v246: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:18:34.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:33 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:18:34.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:33 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:18:34.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:33 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:18:34.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:33 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:18:34.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:33 smithi053 ceph-mon[35979]: pgmap v246: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:18:35.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:35 smithi063 ceph-mon[36249]: pgmap v247: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:18:35.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:35 smithi016 ceph-mon[32374]: pgmap v247: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:18:36.046 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:35 smithi053 ceph-mon[35979]: pgmap v247: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:18:37.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:37 smithi063 ceph-mon[36249]: pgmap v248: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:18:37.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:37 smithi016 ceph-mon[32374]: pgmap v248: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:18:38.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:37 smithi053 ceph-mon[35979]: pgmap v248: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:18:39.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:39 smithi063 ceph-mon[36249]: pgmap v249: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:18:39.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:39 smithi016 ceph-mon[32374]: pgmap v249: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:18:40.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:39 smithi053 ceph-mon[35979]: pgmap v249: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:18:41.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:41 smithi063 ceph-mon[36249]: pgmap v250: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:18:41.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:41 smithi016 ceph-mon[32374]: pgmap v250: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:18:42.046 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:41 smithi053 ceph-mon[35979]: pgmap v250: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:18:43.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:43 smithi063 ceph-mon[36249]: pgmap v251: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:18:43.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:43 smithi016 ceph-mon[32374]: pgmap v251: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:18:44.046 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:43 smithi053 ceph-mon[35979]: pgmap v251: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:18:45.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:45 smithi063 ceph-mon[36249]: pgmap v252: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:18:45.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:45 smithi063 ceph-mon[36249]: scrub summary: idle 2024-03-28T22:18:45.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:45 smithi016 ceph-mon[32374]: pgmap v252: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:18:45.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:45 smithi016 ceph-mon[32374]: scrub summary: idle 2024-03-28T22:18:46.046 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:45 smithi053 ceph-mon[35979]: pgmap v252: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:18:46.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:45 smithi053 ceph-mon[35979]: scrub summary: idle 2024-03-28T22:18:46.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:46 smithi063 ceph-mon[36249]: scrub summary: idle 2024-03-28T22:18:46.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:46 smithi016 ceph-mon[32374]: scrub summary: idle 2024-03-28T22:18:47.046 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:46 smithi053 ceph-mon[35979]: scrub summary: idle 2024-03-28T22:18:47.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:47 smithi063 ceph-mon[36249]: pgmap v253: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:18:47.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:47 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:18:47.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:47 smithi016 ceph-mon[32374]: pgmap v253: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:18:47.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:47 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:18:48.046 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:47 smithi053 ceph-mon[35979]: pgmap v253: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:18:48.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:47 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:18:48.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:48 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:18:48.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:48 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:18:48.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:48 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:18:48.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:48 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:18:48.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:48 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:18:48.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:48 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:18:49.046 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:48 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:18:49.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:48 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:18:49.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:48 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:18:50.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:50 smithi063 ceph-mon[36249]: pgmap v254: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:18:50.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:50 smithi016 ceph-mon[32374]: pgmap v254: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:18:50.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:50 smithi053 ceph-mon[35979]: pgmap v254: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:18:52.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:52 smithi063 ceph-mon[36249]: pgmap v255: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:18:52.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:52 smithi016 ceph-mon[32374]: pgmap v255: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:18:52.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:52 smithi053 ceph-mon[35979]: pgmap v255: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:18:54.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:54 smithi063 ceph-mon[36249]: pgmap v256: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:18:54.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:54 smithi016 ceph-mon[32374]: pgmap v256: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:18:54.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:54 smithi053 ceph-mon[35979]: pgmap v256: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:18:56.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:56 smithi063 ceph-mon[36249]: pgmap v257: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:18:56.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:56 smithi016 ceph-mon[32374]: pgmap v257: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:18:56.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:56 smithi053 ceph-mon[35979]: pgmap v257: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:18:58.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:18:58 smithi063 ceph-mon[36249]: pgmap v258: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:18:58.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:18:58 smithi016 ceph-mon[32374]: pgmap v258: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:18:58.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:18:58 smithi053 ceph-mon[35979]: pgmap v258: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:19:00.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:00 smithi063 ceph-mon[36249]: pgmap v259: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:19:00.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:00 smithi016 ceph-mon[32374]: pgmap v259: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:19:00.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:00 smithi053 ceph-mon[35979]: pgmap v259: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:19:01.911 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:19:01.911 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:19:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.028+0000 7f681f73e640 1 -- 172.21.15.16:0/2664948351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6818100fb0 msgr2=0x7f681810d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.029+0000 7f681f73e640 1 --2- 172.21.15.16:0/2664948351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6818100fb0 0x7f681810d4c0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f6808009cd0 tx=0x7f680802c0f0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.029+0000 7f681f73e640 1 -- 172.21.15.16:0/2664948351 shutdown_connections 2024-03-28T22:19:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.029+0000 7f681f73e640 1 --2- 172.21.15.16:0/2664948351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6818100fb0 0x7f681810d4c0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.029+0000 7f681f73e640 1 --2- 172.21.15.16:0/2664948351 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68181005f0 0x7f6818100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.029+0000 7f681f73e640 1 --2- 172.21.15.16:0/2664948351 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6818106690 0x7f6818106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.029+0000 7f681f73e640 1 -- 172.21.15.16:0/2664948351 >> 172.21.15.16:0/2664948351 conn(0x7f6818077800 msgr2=0x7f6818077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.029+0000 7f681f73e640 1 -- 172.21.15.16:0/2664948351 shutdown_connections 2024-03-28T22:19:02.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.029+0000 7f681f73e640 1 -- 172.21.15.16:0/2664948351 wait complete. 2024-03-28T22:19:02.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.029+0000 7f681f73e640 1 Processor -- start 2024-03-28T22:19:02.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.029+0000 7f681f73e640 1 -- start start 2024-03-28T22:19:02.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.030+0000 7f681f73e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68181005f0 0x7f6818196e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.030+0000 7f681f73e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6818100fb0 0x7f6818197350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.030+0000 7f681f73e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6818106690 0x7f681819b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.030+0000 7f681f73e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f68181109d0 con 0x7f6818100fb0 2024-03-28T22:19:02.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.030+0000 7f681f73e640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f6818110850 con 0x7f6818106690 2024-03-28T22:19:02.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.030+0000 7f681f73e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6818110b50 con 0x7f68181005f0 2024-03-28T22:19:02.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.030+0000 7f681ccd9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6818100fb0 0x7f6818197350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.030+0000 7f681ccd9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6818100fb0 0x7f6818197350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59882/0 (socket says 172.21.15.16:59882) 2024-03-28T22:19:02.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.030+0000 7f681ccd9640 1 -- 172.21.15.16:0/2316746066 learned_addr learned my addr 172.21.15.16:0/2316746066 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:02.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.030+0000 7f681dcdb640 1 --2- 172.21.15.16:0/2316746066 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6818106690 0x7f681819b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.030+0000 7f681d4da640 1 --2- 172.21.15.16:0/2316746066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68181005f0 0x7f6818196e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.030+0000 7f681ccd9640 1 -- 172.21.15.16:0/2316746066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68181005f0 msgr2=0x7f6818196e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.030+0000 7f681ccd9640 1 --2- 172.21.15.16:0/2316746066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68181005f0 0x7f6818196e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.030+0000 7f681ccd9640 1 -- 172.21.15.16:0/2316746066 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6818106690 msgr2=0x7f681819b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.030+0000 7f681ccd9640 1 --2- 172.21.15.16:0/2316746066 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6818106690 0x7f681819b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.030+0000 7f681ccd9640 1 -- 172.21.15.16:0/2316746066 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f681819bdb0 con 0x7f6818100fb0 2024-03-28T22:19:02.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.030+0000 7f681ccd9640 1 --2- 172.21.15.16:0/2316746066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6818100fb0 0x7f6818197350 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f680c00e9f0 tx=0x7f680c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:02.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.030+0000 7f68167fc640 1 -- 172.21.15.16:0/2316746066 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680c00cd80 con 0x7f6818100fb0 2024-03-28T22:19:02.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.030+0000 7f68167fc640 1 -- 172.21.15.16:0/2316746066 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f680c010910 con 0x7f6818100fb0 2024-03-28T22:19:02.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.030+0000 7f681f73e640 1 -- 172.21.15.16:0/2316746066 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f681819c0a0 con 0x7f6818100fb0 2024-03-28T22:19:02.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.030+0000 7f68167fc640 1 -- 172.21.15.16:0/2316746066 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680c018990 con 0x7f6818100fb0 2024-03-28T22:19:02.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.030+0000 7f681f73e640 1 -- 172.21.15.16:0/2316746066 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68181a3930 con 0x7f6818100fb0 2024-03-28T22:19:02.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.033+0000 7f68167fc640 1 -- 172.21.15.16:0/2316746066 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f680c010ac0 con 0x7f6818100fb0 2024-03-28T22:19:02.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.033+0000 7f6822403740 1 Processor -- start 2024-03-28T22:19:02.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.033+0000 7f6822403740 1 -- start start 2024-03-28T22:19:02.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.033+0000 7f68167fc640 1 --2- 172.21.15.16:0/2316746066 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f68000a1ef0 0x7f68000a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.033+0000 7f6822403740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f284a6290 0x556f284a6670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.033+0000 7f6822403740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f284a6bb0 0x556f284af250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.034+0000 7f6822403740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f284af790 0x556f284b1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.034+0000 7f6822403740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556f2840a180 con 0x556f284a6bb0 2024-03-28T22:19:02.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.034+0000 7f6822403740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x556f283b2b70 con 0x556f284af790 2024-03-28T22:19:02.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.034+0000 7f6822403740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x556f28406220 con 0x556f284a6290 2024-03-28T22:19:02.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.034+0000 7f681dcdb640 1 --2- 172.21.15.16:0/2316746066 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f68000a1ef0 0x7f68000a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.034+0000 7f68167fc640 1 -- 172.21.15.16:0/2316746066 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7f680c0ce540 con 0x7f6818100fb0 2024-03-28T22:19:02.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.034+0000 7f681ccd9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f284a6bb0 0x556f284af250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.034+0000 7f681ccd9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f284a6bb0 0x556f284af250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59890/0 (socket says 172.21.15.16:59890) 2024-03-28T22:19:02.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.034+0000 7f681ccd9640 1 -- 172.21.15.16:0/802439684 learned_addr learned my addr 172.21.15.16:0/802439684 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:02.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.034+0000 7f681dcdb640 1 --2- 172.21.15.16:0/802439684 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f284af790 0x556f284b1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.034+0000 7f681d4da640 1 --2- 172.21.15.16:0/802439684 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f284a6290 0x556f284a6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.034+0000 7f681ccd9640 1 -- 172.21.15.16:0/802439684 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f284a6290 msgr2=0x556f284a6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:02.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.034+0000 7f681ccd9640 1 --2- 172.21.15.16:0/802439684 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f284a6290 0x556f284a6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.034+0000 7f681ccd9640 1 -- 172.21.15.16:0/802439684 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f284af790 msgr2=0x556f284b1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:02.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.034+0000 7f681ccd9640 1 --2- 172.21.15.16:0/802439684 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f284af790 0x556f284b1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.034+0000 7f681ccd9640 1 -- 172.21.15.16:0/802439684 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f283c67c0 con 0x556f284a6bb0 2024-03-28T22:19:02.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.034+0000 7f681dcdb640 1 --2- 172.21.15.16:0/2316746066 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f68000a1ef0 0x7f68000a43b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6808005cc0 tx=0x7f6808032450 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:02.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.034+0000 7f681ccd9640 1 --2- 172.21.15.16:0/802439684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f284a6bb0 0x556f284af250 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f680c0a31a0 tx=0x7f680c097300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=830349260d63e8ea server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:02.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.034+0000 7f67ff7fe640 1 -- 172.21.15.16:0/802439684 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680c0071c0 con 0x556f284a6bb0 2024-03-28T22:19:02.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.034+0000 7f67ff7fe640 1 -- 172.21.15.16:0/802439684 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f680c01c860 con 0x556f284a6bb0 2024-03-28T22:19:02.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.034+0000 7f67ff7fe640 1 -- 172.21.15.16:0/802439684 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680c0cbf10 con 0x556f284a6bb0 2024-03-28T22:19:02.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.035+0000 7f6822403740 1 -- 172.21.15.16:0/802439684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f284a6bb0 msgr2=0x556f284af250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.035+0000 7f6822403740 1 --2- 172.21.15.16:0/802439684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f284a6bb0 0x556f284af250 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f680c0a31a0 tx=0x7f680c097300 comp rx=0 tx=0).stop 2024-03-28T22:19:02.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.035+0000 7f6822403740 1 -- 172.21.15.16:0/802439684 shutdown_connections 2024-03-28T22:19:02.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.035+0000 7f6822403740 1 --2- 172.21.15.16:0/802439684 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f284af790 0x556f284b1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.035+0000 7f6822403740 1 --2- 172.21.15.16:0/802439684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f284a6bb0 0x556f284af250 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.035+0000 7f6822403740 1 --2- 172.21.15.16:0/802439684 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f284a6290 0x556f284a6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.035+0000 7f6822403740 1 -- 172.21.15.16:0/802439684 >> 172.21.15.16:0/802439684 conn(0x556f284a09c0 msgr2=0x556f284a0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:02.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.035+0000 7f6822403740 1 -- 172.21.15.16:0/802439684 shutdown_connections 2024-03-28T22:19:02.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.035+0000 7f6822403740 1 -- 172.21.15.16:0/802439684 wait complete. 2024-03-28T22:19:02.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.036+0000 7f6822403740 1 -- start start 2024-03-28T22:19:02.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.036+0000 7f6822403740 1 Processor -- start 2024-03-28T22:19:02.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.036+0000 7f6822403740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f284a6290 0x556f284b2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.036+0000 7f6822403740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f284a6bb0 0x556f284b2e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.036+0000 7f6822403740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f284af790 0x556f284b33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.036+0000 7f6822403740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556f284b2520 con 0x556f284a6bb0 2024-03-28T22:19:02.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.036+0000 7f6822403740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x556f283a84a0 con 0x556f284a6290 2024-03-28T22:19:02.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.036+0000 7f6822403740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x556f284b26a0 con 0x556f284af790 2024-03-28T22:19:02.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.036+0000 7f6822403740 10 client.0 _ll_register_callbacks cb 0x556f283abd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:19:02.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.036+0000 7f681ccd9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f284a6bb0 0x556f284b2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.036+0000 7f681ccd9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f284a6bb0 0x556f284b2e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59894/0 (socket says 172.21.15.16:59894) 2024-03-28T22:19:02.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.036+0000 7f681ccd9640 1 -- 172.21.15.16:0/414193853 learned_addr learned my addr 172.21.15.16:0/414193853 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:02.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.036+0000 7f681d4da640 1 --2- 172.21.15.16:0/414193853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f284a6290 0x556f284b2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.036+0000 7f681dcdb640 1 --2- 172.21.15.16:0/414193853 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f284af790 0x556f284b33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.036+0000 7f681ccd9640 1 -- 172.21.15.16:0/414193853 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f284af790 msgr2=0x556f284b33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:02.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.036+0000 7f681ccd9640 1 --2- 172.21.15.16:0/414193853 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f284af790 0x556f284b33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.036+0000 7f681ccd9640 1 -- 172.21.15.16:0/414193853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f284a6290 msgr2=0x556f284b2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:02.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.036+0000 7f681ccd9640 1 --2- 172.21.15.16:0/414193853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f284a6290 0x556f284b2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.036+0000 7f681ccd9640 1 -- 172.21.15.16:0/414193853 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f281e87d0 con 0x556f284a6bb0 2024-03-28T22:19:02.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.037+0000 7f681ccd9640 1 --2- 172.21.15.16:0/414193853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f284a6bb0 0x556f284b2e70 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f680c097390 tx=0x7f680c01d350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4873b7c828b4191 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:02.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.037+0000 7f67fcff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:19:02.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.037+0000 7f67fcff9640 1 -- 172.21.15.16:0/414193853 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680c0b2070 con 0x556f284a6bb0 2024-03-28T22:19:02.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.037+0000 7f67fcff9640 1 -- 172.21.15.16:0/414193853 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7f680c0b9310 con 0x556f284a6bb0 2024-03-28T22:19:02.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.037+0000 7f6822403740 1 -- 172.21.15.16:0/414193853 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f680c01c860 con 0x556f284a6bb0 2024-03-28T22:19:02.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.037+0000 7f67fcff9640 1 -- 172.21.15.16:0/414193853 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f680c09f080 con 0x556f284a6bb0 2024-03-28T22:19:02.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.037+0000 7f6822403740 10 client.17244 fetch_fsmap learned FSMap version 27 2024-03-28T22:19:02.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.037+0000 7f6822403740 1 -- 172.21.15.16:0/414193853 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556f2811e3f0 con 0x556f284a6bb0 2024-03-28T22:19:02.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.037+0000 7f67fcff9640 1 -- 172.21.15.16:0/414193853 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f680c01c860 con 0x556f284a6bb0 2024-03-28T22:19:02.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.037+0000 7f67fcff9640 1 -- 172.21.15.16:0/414193853 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f680c09a380 con 0x556f284a6bb0 2024-03-28T22:19:02.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.037+0000 7f6822403740 10 client.17244 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:19:02.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.037+0000 7f6822403740 10 client.17244 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:19:02.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.037+0000 7f6822403740 20 client.17244 populate_metadata read hostname 'smithi016' 2024-03-28T22:19:02.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.037+0000 7f6822403740 1 --2- 172.21.15.16:0/414193853 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x556f284c1ba0 0x556f284c3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.037+0000 7f6822403740 4 client.17244 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:19:02.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.037+0000 7f6822403740 1 -- 172.21.15.16:0/414193853 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556f284176c0 con 0x556f284c1ba0 2024-03-28T22:19:02.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.038+0000 7f681d4da640 1 --2- 172.21.15.16:0/414193853 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x556f284c1ba0 0x556f284c3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.038+0000 7f67fcff9640 10 client.17244 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:19:02.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.038+0000 7f681d4da640 1 --2- 172.21.15.16:0/414193853 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x556f284c1ba0 0x556f284c3f90 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ffbcc9a187ea3cf server_cookie=19fa6706a51ab953 in_seq=0 out_seq=0 2024-03-28T22:19:02.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.038+0000 7f67fcff9640 1 -- 172.21.15.16:0/414193853 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556f284176c0 con 0x556f284c1ba0 2024-03-28T22:19:02.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.038+0000 7f67fcff9640 10 client.17244 handle_command_reply: tid=0 2024-03-28T22:19:02.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.039+0000 7f6822403740 1 client.17244 shutdown 2024-03-28T22:19:02.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.039+0000 7f6822403740 1 -- 172.21.15.16:0/414193853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f284a6bb0 msgr2=0x556f284b2e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.039+0000 7f6822403740 1 --2- 172.21.15.16:0/414193853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f284a6bb0 0x556f284b2e70 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f680c097390 tx=0x7f680c01d350 comp rx=0 tx=0).stop 2024-03-28T22:19:02.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.039+0000 7f6822403740 1 -- 172.21.15.16:0/414193853 shutdown_connections 2024-03-28T22:19:02.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.039+0000 7f6822403740 1 --2- 172.21.15.16:0/414193853 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x556f284c1ba0 0x556f284c3f90 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.039+0000 7f6822403740 1 --2- 172.21.15.16:0/414193853 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f284af790 0x556f284b33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.039+0000 7f6822403740 1 --2- 172.21.15.16:0/414193853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f284a6bb0 0x556f284b2e70 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.039+0000 7f67fcff9640 0 client.17244 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:19:02.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.039+0000 7f6822403740 1 --2- 172.21.15.16:0/414193853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f284a6290 0x556f284b2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.039+0000 7f6822403740 1 -- 172.21.15.16:0/414193853 >> 172.21.15.16:0/414193853 conn(0x556f284a09c0 msgr2=0x556f2849fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:02.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.039+0000 7f6822403740 1 -- 172.21.15.16:0/414193853 shutdown_connections 2024-03-28T22:19:02.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.039+0000 7f6822403740 1 -- 172.21.15.16:0/414193853 wait complete. 2024-03-28T22:19:02.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.039+0000 7f6822403740 20 client.17244 trim_cache size 0 max 16384 2024-03-28T22:19:02.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.056+0000 7f6822403740 1 Processor -- start 2024-03-28T22:19:02.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.056+0000 7f6822403740 1 -- start start 2024-03-28T22:19:02.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.057+0000 7f6822403740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f284a6290 0x556f284c6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.057+0000 7f6822403740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f284a6bb0 0x556f284abca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.057+0000 7f6822403740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f284af790 0x556f284ac1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.057+0000 7f6822403740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f68181ada40 con 0x556f284af790 2024-03-28T22:19:02.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.057+0000 7f6822403740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f68181ab310 con 0x556f284a6290 2024-03-28T22:19:02.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.057+0000 7f6822403740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f68181a9180 con 0x556f284a6bb0 2024-03-28T22:19:02.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.057+0000 7f681dcdb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f284af790 0x556f284ac1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.057+0000 7f681dcdb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f284af790 0x556f284ac1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59896/0 (socket says 172.21.15.16:59896) 2024-03-28T22:19:02.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.057+0000 7f681dcdb640 1 -- 172.21.15.16:0/3195970525 learned_addr learned my addr 172.21.15.16:0/3195970525 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:02.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.057+0000 7f681d4da640 1 --2- 172.21.15.16:0/3195970525 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f284a6290 0x556f284c6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.057+0000 7f681dcdb640 1 -- 172.21.15.16:0/3195970525 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f284a6bb0 msgr2=0x556f284abca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:02.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.057+0000 7f681ccd9640 1 --2- 172.21.15.16:0/3195970525 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f284a6bb0 0x556f284abca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.057+0000 7f681dcdb640 1 --2- 172.21.15.16:0/3195970525 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f284a6bb0 0x556f284abca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.057+0000 7f681dcdb640 1 -- 172.21.15.16:0/3195970525 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f284a6290 msgr2=0x556f284c6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:02.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.057+0000 7f681dcdb640 1 --2- 172.21.15.16:0/3195970525 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f284a6290 0x556f284c6fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.057+0000 7f681dcdb640 1 -- 172.21.15.16:0/3195970525 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f280de0f0 con 0x556f284af790 2024-03-28T22:19:02.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.057+0000 7f681d4da640 1 --2- 172.21.15.16:0/3195970525 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f284a6290 0x556f284c6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:19:02.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.057+0000 7f681dcdb640 1 --2- 172.21.15.16:0/3195970525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f284af790 0x556f284ac1e0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f6808009cd0 tx=0x7f6808047000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f39cc0a621e86e01 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:02.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.057+0000 7f67feffd640 1 -- 172.21.15.16:0/3195970525 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680802ce60 con 0x556f284af790 2024-03-28T22:19:02.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.057+0000 7f67feffd640 1 -- 172.21.15.16:0/3195970525 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6808003e50 con 0x556f284af790 2024-03-28T22:19:02.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.057+0000 7f67feffd640 1 -- 172.21.15.16:0/3195970525 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6808039040 con 0x556f284af790 2024-03-28T22:19:02.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.058+0000 7f6822403740 1 -- 172.21.15.16:0/3195970525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f284af790 msgr2=0x556f284ac1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.058+0000 7f6822403740 1 --2- 172.21.15.16:0/3195970525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f284af790 0x556f284ac1e0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f6808009cd0 tx=0x7f6808047000 comp rx=0 tx=0).stop 2024-03-28T22:19:02.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.058+0000 7f6822403740 1 -- 172.21.15.16:0/3195970525 shutdown_connections 2024-03-28T22:19:02.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.058+0000 7f6822403740 1 --2- 172.21.15.16:0/3195970525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f284af790 0x556f284ac1e0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.058+0000 7f6822403740 1 --2- 172.21.15.16:0/3195970525 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f284a6bb0 0x556f284abca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.058+0000 7f6822403740 1 --2- 172.21.15.16:0/3195970525 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f284a6290 0x556f284c6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.058+0000 7f6822403740 1 -- 172.21.15.16:0/3195970525 >> 172.21.15.16:0/3195970525 conn(0x556f284a09c0 msgr2=0x556f2849fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:02.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.058+0000 7f6822403740 1 -- 172.21.15.16:0/3195970525 shutdown_connections 2024-03-28T22:19:02.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.058+0000 7f6822403740 1 -- 172.21.15.16:0/3195970525 wait complete. 2024-03-28T22:19:02.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.058+0000 7f6822403740 1 -- start start 2024-03-28T22:19:02.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.058+0000 7f6822403740 1 Processor -- start 2024-03-28T22:19:02.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.059+0000 7f6822403740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f284a6290 0x556f284c4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.059+0000 7f6822403740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f284a6bb0 0x556f284c4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.059+0000 7f6822403740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f284af790 0x556f284c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.059+0000 7f6822403740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f68181a97d0 con 0x556f284a6290 2024-03-28T22:19:02.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.059+0000 7f6822403740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f68181aac40 con 0x556f284a6bb0 2024-03-28T22:19:02.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.059+0000 7f6822403740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f68181a9ea0 con 0x556f284af790 2024-03-28T22:19:02.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.059+0000 7f6822403740 10 client.0 _ll_register_callbacks cb 0x7f6808037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:19:02.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.059+0000 7f681d4da640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f284a6290 0x556f284c4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.059+0000 7f681ccd9640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f284a6bb0 0x556f284c4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.059+0000 7f681d4da640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f284a6290 0x556f284c4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59908/0 (socket says 172.21.15.16:59908) 2024-03-28T22:19:02.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.059+0000 7f681d4da640 1 -- 172.21.15.16:0/4202187314 learned_addr learned my addr 172.21.15.16:0/4202187314 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:02.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.059+0000 7f681dcdb640 1 --2- 172.21.15.16:0/4202187314 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f284af790 0x556f284c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.059+0000 7f681ccd9640 1 -- 172.21.15.16:0/4202187314 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f284af790 msgr2=0x556f284c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:02.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.059+0000 7f681ccd9640 1 --2- 172.21.15.16:0/4202187314 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f284af790 0x556f284c54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.059+0000 7f681ccd9640 1 -- 172.21.15.16:0/4202187314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f284a6290 msgr2=0x556f284c4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:02.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.059+0000 7f681ccd9640 1 --2- 172.21.15.16:0/4202187314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f284a6290 0x556f284c4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.059+0000 7f681ccd9640 1 -- 172.21.15.16:0/4202187314 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f28526d10 con 0x556f284a6bb0 2024-03-28T22:19:02.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.059+0000 7f681d4da640 1 --2- 172.21.15.16:0/4202187314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f284a6290 0x556f284c4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:19:02.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.059+0000 7f681ccd9640 1 --2- 172.21.15.16:0/4202187314 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f284a6bb0 0x556f284c4fb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x556f284aecb0 tx=0x7f680c0977d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec63df697035559c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:02.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.059+0000 7f67fcff9640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-03-28T22:19:02.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.059+0000 7f67fcff9640 1 -- 172.21.15.16:0/4202187314 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680c0b2070 con 0x556f284a6bb0 2024-03-28T22:19:02.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.060+0000 7f67fcff9640 1 -- 172.21.15.16:0/4202187314 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7f680c010430 con 0x556f284a6bb0 2024-03-28T22:19:02.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.060+0000 7f6822403740 1 -- 172.21.15.16:0/4202187314 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6808003e50 con 0x556f284a6bb0 2024-03-28T22:19:02.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.060+0000 7f67fcff9640 1 -- 172.21.15.16:0/4202187314 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f680c0a36d0 con 0x556f284a6bb0 2024-03-28T22:19:02.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.060+0000 7f6822403740 10 client.27152 fetch_fsmap learned FSMap version 27 2024-03-28T22:19:02.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.060+0000 7f6822403740 1 -- 172.21.15.16:0/4202187314 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556f28527fe0 con 0x556f284a6bb0 2024-03-28T22:19:02.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.060+0000 7f67fcff9640 1 -- 172.21.15.16:0/4202187314 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6808003e50 con 0x556f284a6bb0 2024-03-28T22:19:02.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.060+0000 7f67fcff9640 1 -- 172.21.15.16:0/4202187314 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f680c099df0 con 0x556f284a6bb0 2024-03-28T22:19:02.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.060+0000 7f6822403740 10 client.27152 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:19:02.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.060+0000 7f6822403740 10 client.27152 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:19:02.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.060+0000 7f6822403740 20 client.27152 populate_metadata read hostname 'smithi016' 2024-03-28T22:19:02.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.060+0000 7f6822403740 1 --2- 172.21.15.16:0/4202187314 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x556f28528510 0x556f2852a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.060+0000 7f6822403740 4 client.27152 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:19:02.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.060+0000 7f6822403740 1 -- 172.21.15.16:0/4202187314 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556f28411a10 con 0x556f28528510 2024-03-28T22:19:02.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.060+0000 7f681d4da640 1 --2- 172.21.15.16:0/4202187314 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x556f28528510 0x556f2852a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.061+0000 7f67fcff9640 10 client.27152 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:19:02.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.061+0000 7f681d4da640 1 --2- 172.21.15.16:0/4202187314 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x556f28528510 0x556f2852a900 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfedf01e5caa6a17 server_cookie=c77f3b47c621e1e in_seq=0 out_seq=0 2024-03-28T22:19:02.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.061+0000 7f67fcff9640 1 -- 172.21.15.16:0/4202187314 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x556f28411a10 con 0x556f28528510 2024-03-28T22:19:02.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.061+0000 7f67fcff9640 10 client.27152 handle_command_reply: tid=0 2024-03-28T22:19:02.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.061+0000 7f6822403740 1 client.27152 shutdown 2024-03-28T22:19:02.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.061+0000 7f6822403740 1 -- 172.21.15.16:0/4202187314 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f284a6bb0 msgr2=0x556f284c4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.061+0000 7f6822403740 1 --2- 172.21.15.16:0/4202187314 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f284a6bb0 0x556f284c4fb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x556f284aecb0 tx=0x7f680c0977d0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.061+0000 7f6822403740 1 -- 172.21.15.16:0/4202187314 shutdown_connections 2024-03-28T22:19:02.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.061+0000 7f6822403740 1 --2- 172.21.15.16:0/4202187314 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x556f28528510 0x556f2852a900 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.061+0000 7f6822403740 1 --2- 172.21.15.16:0/4202187314 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f284af790 0x556f284c54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.061+0000 7f67fcff9640 0 client.27152 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:19:02.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.061+0000 7f6822403740 1 --2- 172.21.15.16:0/4202187314 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556f284a6bb0 0x556f284c4fb0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.061+0000 7f6822403740 1 --2- 172.21.15.16:0/4202187314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556f284a6290 0x556f284c4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.061+0000 7f6822403740 1 -- 172.21.15.16:0/4202187314 >> 172.21.15.16:0/4202187314 conn(0x556f284a09c0 msgr2=0x556f284ad1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:02.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.061+0000 7f6822403740 1 -- 172.21.15.16:0/4202187314 shutdown_connections 2024-03-28T22:19:02.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.061+0000 7f6822403740 1 -- 172.21.15.16:0/4202187314 wait complete. 2024-03-28T22:19:02.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.061+0000 7f6822403740 20 client.27152 trim_cache size 0 max 16384 2024-03-28T22:19:02.074 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:19:02.074 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-03-28T22:19:02.074 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-03-28T22:19:02.074 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:19:02.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.062+0000 7f681f73e640 1 -- 172.21.15.16:0/2316746066 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f68000a1ef0 msgr2=0x7f68000a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.062+0000 7f681f73e640 1 --2- 172.21.15.16:0/2316746066 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f68000a1ef0 0x7f68000a43b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6808005cc0 tx=0x7f6808032450 comp rx=0 tx=0).stop 2024-03-28T22:19:02.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.062+0000 7f681f73e640 1 -- 172.21.15.16:0/2316746066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6818100fb0 msgr2=0x7f6818197350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.062+0000 7f681f73e640 1 --2- 172.21.15.16:0/2316746066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6818100fb0 0x7f6818197350 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f680c00e9f0 tx=0x7f680c00eeb0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.062+0000 7f681f73e640 1 -- 172.21.15.16:0/2316746066 shutdown_connections 2024-03-28T22:19:02.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.062+0000 7f681f73e640 1 --2- 172.21.15.16:0/2316746066 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f68000a1ef0 0x7f68000a43b0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.062+0000 7f681f73e640 1 --2- 172.21.15.16:0/2316746066 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6818106690 0x7f681819b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.062+0000 7f681f73e640 1 --2- 172.21.15.16:0/2316746066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6818100fb0 0x7f6818197350 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.062+0000 7f681f73e640 1 --2- 172.21.15.16:0/2316746066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68181005f0 0x7f6818196e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.062+0000 7f681f73e640 1 -- 172.21.15.16:0/2316746066 >> 172.21.15.16:0/2316746066 conn(0x7f6818077800 msgr2=0x7f68180ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:02.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.062+0000 7f681f73e640 1 -- 172.21.15.16:0/2316746066 shutdown_connections 2024-03-28T22:19:02.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.062+0000 7f681f73e640 1 -- 172.21.15.16:0/2316746066 wait complete. 2024-03-28T22:19:02.077 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:19:02.078 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:19:02.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.179+0000 7fbae3014640 1 -- 172.21.15.16:0/1883768605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbadc0ffbe0 msgr2=0x7fbadc100060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.179+0000 7fbae3014640 1 --2- 172.21.15.16:0/1883768605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbadc0ffbe0 0x7fbadc100060 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fbad0009930 tx=0x7fbad002bc30 comp rx=0 tx=0).stop 2024-03-28T22:19:02.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.179+0000 7fbae3014640 1 -- 172.21.15.16:0/1883768605 shutdown_connections 2024-03-28T22:19:02.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.179+0000 7fbae3014640 1 --2- 172.21.15.16:0/1883768605 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbadc1005a0 0x7fbadc10e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.179+0000 7fbae3014640 1 --2- 172.21.15.16:0/1883768605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbadc0ffbe0 0x7fbadc100060 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.179+0000 7fbae3014640 1 --2- 172.21.15.16:0/1883768605 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbadc107700 0x7fbadc107ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.179+0000 7fbae3014640 1 -- 172.21.15.16:0/1883768605 >> 172.21.15.16:0/1883768605 conn(0x7fbadc076b60 msgr2=0x7fbadc0fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:02.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.179+0000 7fbae3014640 1 -- 172.21.15.16:0/1883768605 shutdown_connections 2024-03-28T22:19:02.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.179+0000 7fbae3014640 1 -- 172.21.15.16:0/1883768605 wait complete. 2024-03-28T22:19:02.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.180+0000 7fbae3014640 1 Processor -- start 2024-03-28T22:19:02.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.180+0000 7fbae3014640 1 -- start start 2024-03-28T22:19:02.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.180+0000 7fbae3014640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbadc0ffbe0 0x7fbadc198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.180+0000 7fbae3014640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbadc1005a0 0x7fbadc198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.180+0000 7fbae3014640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbadc107700 0x7fbadc19c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.180+0000 7fbae3014640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbadc111b10 con 0x7fbadc0ffbe0 2024-03-28T22:19:02.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.180+0000 7fbae3014640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fbadc111990 con 0x7fbadc107700 2024-03-28T22:19:02.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.180+0000 7fbae3014640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbadc111c90 con 0x7fbadc1005a0 2024-03-28T22:19:02.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.180+0000 7fbae0db0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbadc0ffbe0 0x7fbadc198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.180+0000 7fbae0db0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbadc0ffbe0 0x7fbadc198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59924/0 (socket says 172.21.15.16:59924) 2024-03-28T22:19:02.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.180+0000 7fbae0db0640 1 -- 172.21.15.16:0/452480470 learned_addr learned my addr 172.21.15.16:0/452480470 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:02.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.180+0000 7fbae15b1640 1 --2- 172.21.15.16:0/452480470 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbadc107700 0x7fbadc19c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.180+0000 7fbadbfff640 1 --2- 172.21.15.16:0/452480470 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbadc1005a0 0x7fbadc198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.181+0000 7fbae0db0640 1 -- 172.21.15.16:0/452480470 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbadc1005a0 msgr2=0x7fbadc198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.181+0000 7fbae0db0640 1 --2- 172.21.15.16:0/452480470 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbadc1005a0 0x7fbadc198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.181+0000 7fbae0db0640 1 -- 172.21.15.16:0/452480470 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbadc107700 msgr2=0x7fbadc19c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.181+0000 7fbae0db0640 1 --2- 172.21.15.16:0/452480470 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbadc107700 0x7fbadc19c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.181+0000 7fbae0db0640 1 -- 172.21.15.16:0/452480470 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbadc19d000 con 0x7fbadc0ffbe0 2024-03-28T22:19:02.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.181+0000 7fbae0db0640 1 --2- 172.21.15.16:0/452480470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbadc0ffbe0 0x7fbadc198010 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fbad400ea80 tx=0x7fbad400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:02.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.181+0000 7fbad9ffb640 1 -- 172.21.15.16:0/452480470 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad400cd80 con 0x7fbadc0ffbe0 2024-03-28T22:19:02.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.181+0000 7fbad9ffb640 1 -- 172.21.15.16:0/452480470 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbad4005020 con 0x7fbadc0ffbe0 2024-03-28T22:19:02.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.181+0000 7fbae3014640 1 -- 172.21.15.16:0/452480470 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbadc19d2f0 con 0x7fbadc0ffbe0 2024-03-28T22:19:02.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.181+0000 7fbad9ffb640 1 -- 172.21.15.16:0/452480470 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad40116d0 con 0x7fbadc0ffbe0 2024-03-28T22:19:02.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.181+0000 7fbae3014640 1 -- 172.21.15.16:0/452480470 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbadc1a4b80 con 0x7fbadc0ffbe0 2024-03-28T22:19:02.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.182+0000 7fbae3014640 1 -- 172.21.15.16:0/452480470 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbadc102f00 con 0x7fbadc0ffbe0 2024-03-28T22:19:02.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.184+0000 7fbad9ffb640 1 -- 172.21.15.16:0/452480470 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7fbad4005560 con 0x7fbadc0ffbe0 2024-03-28T22:19:02.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.184+0000 7fbad9ffb640 1 --2- 172.21.15.16:0/452480470 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fbac40a1f70 0x7fbac40a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.184+0000 7fbad9ffb640 1 -- 172.21.15.16:0/452480470 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7fbad40ce730 con 0x7fbadc0ffbe0 2024-03-28T22:19:02.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.184+0000 7fbadbfff640 1 --2- 172.21.15.16:0/452480470 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fbac40a1f70 0x7fbac40a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.185+0000 7fbadbfff640 1 --2- 172.21.15.16:0/452480470 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fbac40a1f70 0x7fbac40a4430 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fbad0005ab0 tx=0x7fbad0005fd0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:02.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.188+0000 7fbad9ffb640 1 -- 172.21.15.16:0/452480470 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fbad4016030 con 0x7fbadc0ffbe0 2024-03-28T22:19:02.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.409+0000 7fbae3014640 1 -- 172.21.15.16:0/452480470 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbadc068a20 con 0x7fbadc0ffbe0 2024-03-28T22:19:02.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.409+0000 7fbad9ffb640 1 -- 172.21.15.16:0/452480470 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11851 (secure 0 0 0) 0x7fbad409b8a0 con 0x7fbadc0ffbe0 2024-03-28T22:19:02.411 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:19:02.412 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/1644581227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1644581227},{"type":"v1","addr":"172.21.15.16:6839","nonce":1644581227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/2593062669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2593062669},{"type":"v1","addr":"172.21.15.16:6841","nonce":2593062669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:08:29.672976+0000","modified":"2024-03-28T22:14:44.922044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24566,"mds_1":24524,"mds_2":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24466":{"gid":24466,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/247249857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":247249857},{"type":"v1","addr":"172.21.15.63:6835","nonce":247249857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6837/2324231032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":2324231032},{"type":"v1","addr":"172.21.15.63:6837","nonce":2324231032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6839/1177197077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":1177197077},{"type":"v1","addr":"172.21.15.63:6839","nonce":1177197077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"e","rank":2,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.53:6835/350727634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":350727634},{"type":"v1","addr":"172.21.15.53:6835","nonce":350727634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/349877573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":349877573},{"type":"v1","addr":"172.21.15.53:6837","nonce":349877573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6839/3714153956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":3714153956},{"type":"v1","addr":"172.21.15.53:6839","nonce":3714153956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24566,"qdb_cluster":[24500,24524,24566]},"id":1}]} 2024-03-28T22:19:02.412 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2024-03-28T22:19:02.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.412+0000 7fbae3014640 1 -- 172.21.15.16:0/452480470 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fbac40a1f70 msgr2=0x7fbac40a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.412+0000 7fbae3014640 1 --2- 172.21.15.16:0/452480470 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fbac40a1f70 0x7fbac40a4430 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fbad0005ab0 tx=0x7fbad0005fd0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.412+0000 7fbae3014640 1 -- 172.21.15.16:0/452480470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbadc0ffbe0 msgr2=0x7fbadc198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.412+0000 7fbae3014640 1 --2- 172.21.15.16:0/452480470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbadc0ffbe0 0x7fbadc198010 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fbad400ea80 tx=0x7fbad400ee80 comp rx=0 tx=0).stop 2024-03-28T22:19:02.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.412+0000 7fbae3014640 1 -- 172.21.15.16:0/452480470 shutdown_connections 2024-03-28T22:19:02.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.412+0000 7fbae3014640 1 --2- 172.21.15.16:0/452480470 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fbac40a1f70 0x7fbac40a4430 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.412+0000 7fbae3014640 1 --2- 172.21.15.16:0/452480470 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbadc107700 0x7fbadc19c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.412+0000 7fbae3014640 1 --2- 172.21.15.16:0/452480470 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbadc1005a0 0x7fbadc198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.412+0000 7fbae3014640 1 --2- 172.21.15.16:0/452480470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbadc0ffbe0 0x7fbadc198010 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.412+0000 7fbae3014640 1 -- 172.21.15.16:0/452480470 >> 172.21.15.16:0/452480470 conn(0x7fbadc076b60 msgr2=0x7fbadc104710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:02.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.412+0000 7fbae3014640 1 -- 172.21.15.16:0/452480470 shutdown_connections 2024-03-28T22:19:02.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.412+0000 7fbae3014640 1 -- 172.21.15.16:0/452480470 wait complete. 2024-03-28T22:19:02.423 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:02 smithi016 ceph-mon[32374]: pgmap v260: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:19:02.424 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:19:02.430 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:02 smithi063 ceph-mon[36249]: pgmap v260: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:19:02.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.527+0000 7fec23e49640 1 -- 172.21.15.16:0/625591679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec1c103290 msgr2=0x7fec1c10b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.527+0000 7fec23e49640 1 --2- 172.21.15.16:0/625591679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec1c103290 0x7fec1c10b770 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fec0c009900 tx=0x7fec0c02bcd0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.527+0000 7fec23e49640 1 -- 172.21.15.16:0/625591679 shutdown_connections 2024-03-28T22:19:02.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.527+0000 7fec23e49640 1 --2- 172.21.15.16:0/625591679 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fec1c10bd40 0x7fec1c10e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.527+0000 7fec23e49640 1 --2- 172.21.15.16:0/625591679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec1c103290 0x7fec1c10b770 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.527+0000 7fec23e49640 1 --2- 172.21.15.16:0/625591679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec1c102970 0x7fec1c102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.527+0000 7fec23e49640 1 -- 172.21.15.16:0/625591679 >> 172.21.15.16:0/625591679 conn(0x7fec1c077770 msgr2=0x7fec1c077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:02.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.527+0000 7fec23e49640 1 -- 172.21.15.16:0/625591679 shutdown_connections 2024-03-28T22:19:02.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.527+0000 7fec23e49640 1 -- 172.21.15.16:0/625591679 wait complete. 2024-03-28T22:19:02.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.528+0000 7fec23e49640 1 Processor -- start 2024-03-28T22:19:02.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.528+0000 7fec23e49640 1 -- start start 2024-03-28T22:19:02.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.528+0000 7fec23e49640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec1c102970 0x7fec1c1980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.528+0000 7fec23e49640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec1c103290 0x7fec1c1985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.528+0000 7fec23e49640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fec1c10bd40 0x7fec1c19c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.528+0000 7fec23e49640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fec1c111b20 con 0x7fec1c103290 2024-03-28T22:19:02.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.528+0000 7fec23e49640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fec1c1119a0 con 0x7fec1c10bd40 2024-03-28T22:19:02.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.528+0000 7fec23e49640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fec1c111ca0 con 0x7fec1c102970 2024-03-28T22:19:02.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.528+0000 7fec213e4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec1c103290 0x7fec1c1985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.528+0000 7fec223e6640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fec1c10bd40 0x7fec1c19c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.528+0000 7fec213e4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec1c103290 0x7fec1c1985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59948/0 (socket says 172.21.15.16:59948) 2024-03-28T22:19:02.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.528+0000 7fec223e6640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fec1c10bd40 0x7fec1c19c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:38866/0 (socket says 172.21.15.16:38866) 2024-03-28T22:19:02.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.528+0000 7fec213e4640 1 -- 172.21.15.16:0/2225014947 learned_addr learned my addr 172.21.15.16:0/2225014947 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:02.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.529+0000 7fec21be5640 1 --2- 172.21.15.16:0/2225014947 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec1c102970 0x7fec1c1980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.529+0000 7fec213e4640 1 -- 172.21.15.16:0/2225014947 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec1c102970 msgr2=0x7fec1c1980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.529+0000 7fec213e4640 1 --2- 172.21.15.16:0/2225014947 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec1c102970 0x7fec1c1980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.529+0000 7fec213e4640 1 -- 172.21.15.16:0/2225014947 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fec1c10bd40 msgr2=0x7fec1c19c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.529+0000 7fec213e4640 1 --2- 172.21.15.16:0/2225014947 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fec1c10bd40 0x7fec1c19c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.529+0000 7fec213e4640 1 -- 172.21.15.16:0/2225014947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec1c19d090 con 0x7fec1c103290 2024-03-28T22:19:02.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.529+0000 7fec223e6640 1 --2- 172.21.15.16:0/2225014947 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fec1c10bd40 0x7fec1c19c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:19:02.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.529+0000 7fec21be5640 1 --2- 172.21.15.16:0/2225014947 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec1c102970 0x7fec1c1980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:19:02.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.529+0000 7fec213e4640 1 --2- 172.21.15.16:0/2225014947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec1c103290 0x7fec1c1985e0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fec0c000c00 tx=0x7fec0c0043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:02.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.529+0000 7fec12ffd640 1 -- 172.21.15.16:0/2225014947 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec0c042020 con 0x7fec1c103290 2024-03-28T22:19:02.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.529+0000 7fec12ffd640 1 -- 172.21.15.16:0/2225014947 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec0c0054e0 con 0x7fec1c103290 2024-03-28T22:19:02.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.529+0000 7fec23e49640 1 -- 172.21.15.16:0/2225014947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec1c19d320 con 0x7fec1c103290 2024-03-28T22:19:02.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.529+0000 7fec12ffd640 1 -- 172.21.15.16:0/2225014947 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec0c003a80 con 0x7fec1c103290 2024-03-28T22:19:02.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.529+0000 7fec23e49640 1 -- 172.21.15.16:0/2225014947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec1c1a4c10 con 0x7fec1c103290 2024-03-28T22:19:02.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.531+0000 7fec12ffd640 1 -- 172.21.15.16:0/2225014947 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7fec0c045020 con 0x7fec1c103290 2024-03-28T22:19:02.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.532+0000 7fec26a20740 1 Processor -- start 2024-03-28T22:19:02.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.532+0000 7fec26a20740 1 -- start start 2024-03-28T22:19:02.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.532+0000 7fec26a20740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f409a7c470 0x55f409a7c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.532+0000 7fec26a20740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f409a7cd90 0x55f409a85430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.532+0000 7fec26a20740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f409a85970 0x55f409a87d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.532+0000 7fec26a20740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f4099e01b0 con 0x55f409a85970 2024-03-28T22:19:02.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.532+0000 7fec26a20740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f409988c00 con 0x55f409a7c470 2024-03-28T22:19:02.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.532+0000 7fec26a20740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f4099dc250 con 0x55f409a7cd90 2024-03-28T22:19:02.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.532+0000 7fec12ffd640 1 --2- 172.21.15.16:0/2225014947 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fec080a1ef0 0x7fec080a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.532+0000 7fec223e6640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f409a7c470 0x55f409a7c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.532+0000 7fec213e4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f409a85970 0x55f409a87d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.532+0000 7fec223e6640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f409a7c470 0x55f409a7c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:38874/0 (socket says 172.21.15.16:38874) 2024-03-28T22:19:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.532+0000 7fec223e6640 1 -- 172.21.15.16:0/614316332 learned_addr learned my addr 172.21.15.16:0/614316332 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.532+0000 7fec223e6640 1 --2- 172.21.15.16:0/2225014947 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fec080a1ef0 0x7fec080a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.532+0000 7fec21be5640 1 --2- 172.21.15.16:0/614316332 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f409a7cd90 0x55f409a85430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.533+0000 7fec12ffd640 1 -- 172.21.15.16:0/2225014947 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7fec0c0efd80 con 0x7fec1c103290 2024-03-28T22:19:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.533+0000 7fec223e6640 1 -- 172.21.15.16:0/614316332 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f409a7cd90 msgr2=0x55f409a85430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.533+0000 7fec223e6640 1 --2- 172.21.15.16:0/614316332 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f409a7cd90 0x55f409a85430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.533+0000 7fec223e6640 1 -- 172.21.15.16:0/614316332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f409a85970 msgr2=0x55f409a87d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.533+0000 7fec223e6640 1 --2- 172.21.15.16:0/614316332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f409a85970 0x55f409a87d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.533+0000 7fec223e6640 1 -- 172.21.15.16:0/614316332 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4099a83d0 con 0x55f409a7c470 2024-03-28T22:19:02.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.533+0000 7fec223e6640 1 --2- 172.21.15.16:0/614316332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f409a7c470 0x55f409a7c850 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fec140099a0 tx=0x7fec14011f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fa1c0656a36f688 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:02.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.533+0000 7fec223e6640 1 --2- 172.21.15.16:0/2225014947 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fec080a1ef0 0x7fec080a43b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fec1400fee0 tx=0x7fec1401e000 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:02.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.533+0000 7fec07fff640 1 -- 172.21.15.16:0/614316332 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec14022070 con 0x55f409a7c470 2024-03-28T22:19:02.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.533+0000 7fec07fff640 1 -- 172.21.15.16:0/614316332 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec14003970 con 0x55f409a7c470 2024-03-28T22:19:02.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.533+0000 7fec07fff640 1 -- 172.21.15.16:0/614316332 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec14004350 con 0x55f409a7c470 2024-03-28T22:19:02.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.534+0000 7fec26a20740 1 -- 172.21.15.16:0/614316332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f409a7c470 msgr2=0x55f409a7c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.534+0000 7fec26a20740 1 --2- 172.21.15.16:0/614316332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f409a7c470 0x55f409a7c850 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fec140099a0 tx=0x7fec14011f90 comp rx=0 tx=0).stop 2024-03-28T22:19:02.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.534+0000 7fec26a20740 1 -- 172.21.15.16:0/614316332 shutdown_connections 2024-03-28T22:19:02.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.534+0000 7fec26a20740 1 --2- 172.21.15.16:0/614316332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f409a85970 0x55f409a87d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.534+0000 7fec26a20740 1 --2- 172.21.15.16:0/614316332 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f409a7cd90 0x55f409a85430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.534+0000 7fec26a20740 1 --2- 172.21.15.16:0/614316332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f409a7c470 0x55f409a7c850 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.534+0000 7fec26a20740 1 -- 172.21.15.16:0/614316332 >> 172.21.15.16:0/614316332 conn(0x55f409a76ba0 msgr2=0x55f409a76f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:02.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.534+0000 7fec26a20740 1 -- 172.21.15.16:0/614316332 shutdown_connections 2024-03-28T22:19:02.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.534+0000 7fec26a20740 1 -- 172.21.15.16:0/614316332 wait complete. 2024-03-28T22:19:02.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.534+0000 7fec26a20740 1 -- start start 2024-03-28T22:19:02.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.534+0000 7fec26a20740 1 Processor -- start 2024-03-28T22:19:02.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.535+0000 7fec26a20740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f409a7c470 0x55f409a82a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.535+0000 7fec26a20740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f409a7cd90 0x55f409a82f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.535+0000 7fec26a20740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f409a85970 0x55f409a834c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.535+0000 7fec26a20740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f409a886a0 con 0x55f409a7c470 2024-03-28T22:19:02.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.535+0000 7fec26a20740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f40997e470 con 0x55f409a85970 2024-03-28T22:19:02.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.535+0000 7fec26a20740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f409a88820 con 0x55f409a7cd90 2024-03-28T22:19:02.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.535+0000 7fec26a20740 10 client.0 _ll_register_callbacks cb 0x55f40999bcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:19:02.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.535+0000 7fec21be5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f409a7c470 0x55f409a82a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.535+0000 7fec21be5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f409a7c470 0x55f409a82a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59962/0 (socket says 172.21.15.16:59962) 2024-03-28T22:19:02.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.535+0000 7fec21be5640 1 -- 172.21.15.16:0/413400042 learned_addr learned my addr 172.21.15.16:0/413400042 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:02.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.535+0000 7fec223e6640 1 --2- 172.21.15.16:0/413400042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f409a85970 0x55f409a834c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.535+0000 7fec213e4640 1 --2- 172.21.15.16:0/413400042 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f409a7cd90 0x55f409a82f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.535+0000 7fec21be5640 1 -- 172.21.15.16:0/413400042 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f409a7cd90 msgr2=0x55f409a82f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:02.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.535+0000 7fec21be5640 1 --2- 172.21.15.16:0/413400042 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f409a7cd90 0x55f409a82f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.535+0000 7fec21be5640 1 -- 172.21.15.16:0/413400042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f409a85970 msgr2=0x55f409a834c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:02.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.535+0000 7fec21be5640 1 --2- 172.21.15.16:0/413400042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f409a85970 0x55f409a834c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.535+0000 7fec21be5640 1 -- 172.21.15.16:0/413400042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4096f43f0 con 0x55f409a7c470 2024-03-28T22:19:02.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.535+0000 7fec21be5640 1 --2- 172.21.15.16:0/413400042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f409a7c470 0x55f409a82a40 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fec180107c0 tx=0x7fec18010c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d8df4640f19270e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:02.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.535+0000 7fec057fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:19:02.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.536+0000 7fec057fa640 1 -- 172.21.15.16:0/413400042 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec1800ed40 con 0x55f409a7c470 2024-03-28T22:19:02.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.536+0000 7fec057fa640 1 -- 172.21.15.16:0/413400042 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7fec18005020 con 0x55f409a7c470 2024-03-28T22:19:02.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.536+0000 7fec26a20740 1 -- 172.21.15.16:0/413400042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec14003970 con 0x55f409a7c470 2024-03-28T22:19:02.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.536+0000 7fec057fa640 1 -- 172.21.15.16:0/413400042 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fec18005220 con 0x55f409a7c470 2024-03-28T22:19:02.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.536+0000 7fec26a20740 10 client.17268 fetch_fsmap learned FSMap version 27 2024-03-28T22:19:02.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.536+0000 7fec26a20740 1 -- 172.21.15.16:0/413400042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f4097be7d0 con 0x55f409a7c470 2024-03-28T22:19:02.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.536+0000 7fec057fa640 1 -- 172.21.15.16:0/413400042 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec14003970 con 0x55f409a7c470 2024-03-28T22:19:02.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.536+0000 7fec057fa640 1 -- 172.21.15.16:0/413400042 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7fec18011440 con 0x55f409a7c470 2024-03-28T22:19:02.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.536+0000 7fec26a20740 10 client.17268 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:19:02.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.536+0000 7fec26a20740 10 client.17268 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-28T22:19:02.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.536+0000 7fec26a20740 20 client.17268 populate_metadata read hostname 'smithi016' 2024-03-28T22:19:02.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.536+0000 7fec26a20740 1 --2- 172.21.15.16:0/413400042 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55f409a95a10 0x55f409a97e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.536+0000 7fec26a20740 4 client.17268 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:19:02.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.536+0000 7fec26a20740 1 -- 172.21.15.16:0/413400042 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f4099ed6f0 con 0x55f409a95a10 2024-03-28T22:19:02.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.536+0000 7fec213e4640 1 --2- 172.21.15.16:0/413400042 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55f409a95a10 0x55f409a97e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.537+0000 7fec057fa640 10 client.17268 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:19:02.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.537+0000 7fec213e4640 1 --2- 172.21.15.16:0/413400042 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55f409a95a10 0x55f409a97e60 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=450c4cc140d64036 server_cookie=1015f3cc621a70c4 in_seq=0 out_seq=0 2024-03-28T22:19:02.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.537+0000 7fec057fa640 1 -- 172.21.15.16:0/413400042 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f4099ed6f0 con 0x55f409a95a10 2024-03-28T22:19:02.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.537+0000 7fec057fa640 10 client.17268 handle_command_reply: tid=0 2024-03-28T22:19:02.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.537+0000 7fec26a20740 1 client.17268 shutdown 2024-03-28T22:19:02.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.538+0000 7fec26a20740 1 -- 172.21.15.16:0/413400042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f409a7c470 msgr2=0x55f409a82a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.538+0000 7fec26a20740 1 --2- 172.21.15.16:0/413400042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f409a7c470 0x55f409a82a40 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fec180107c0 tx=0x7fec18010c80 comp rx=0 tx=0).stop 2024-03-28T22:19:02.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.538+0000 7fec26a20740 1 -- 172.21.15.16:0/413400042 shutdown_connections 2024-03-28T22:19:02.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.538+0000 7fec26a20740 1 --2- 172.21.15.16:0/413400042 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55f409a95a10 0x55f409a97e60 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.538+0000 7fec26a20740 1 --2- 172.21.15.16:0/413400042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f409a85970 0x55f409a834c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.538+0000 7fec26a20740 1 --2- 172.21.15.16:0/413400042 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f409a7cd90 0x55f409a82f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.538+0000 7fec057fa640 0 client.17268 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:19:02.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.538+0000 7fec26a20740 1 --2- 172.21.15.16:0/413400042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f409a7c470 0x55f409a82a40 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.538+0000 7fec26a20740 1 -- 172.21.15.16:0/413400042 >> 172.21.15.16:0/413400042 conn(0x55f409a76ba0 msgr2=0x55f409a788f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:02.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.538+0000 7fec26a20740 1 -- 172.21.15.16:0/413400042 shutdown_connections 2024-03-28T22:19:02.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.538+0000 7fec26a20740 1 -- 172.21.15.16:0/413400042 wait complete. 2024-03-28T22:19:02.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.538+0000 7fec26a20740 20 client.17268 trim_cache size 0 max 16384 2024-03-28T22:19:02.554 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:02 smithi053 ceph-mon[35979]: pgmap v260: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:19:02.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.551+0000 7fec26a20740 1 Processor -- start 2024-03-28T22:19:02.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.551+0000 7fec26a20740 1 -- start start 2024-03-28T22:19:02.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.551+0000 7fec26a20740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f409a7c470 0x55f409a85170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.551+0000 7fec26a20740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f409a7cd90 0x55f409a8de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.551+0000 7fec26a20740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f409a85970 0x55f409a91470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.551+0000 7fec26a20740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fec1c1afc20 con 0x55f409a7c470 2024-03-28T22:19:02.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.551+0000 7fec26a20740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fec1c1ad4f0 con 0x55f409a7cd90 2024-03-28T22:19:02.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.551+0000 7fec26a20740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fec1c1ab360 con 0x55f409a85970 2024-03-28T22:19:02.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.551+0000 7fec21be5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f409a7c470 0x55f409a85170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.551+0000 7fec21be5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f409a7c470 0x55f409a85170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59970/0 (socket says 172.21.15.16:59970) 2024-03-28T22:19:02.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.551+0000 7fec21be5640 1 -- 172.21.15.16:0/231887608 learned_addr learned my addr 172.21.15.16:0/231887608 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:02.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.551+0000 7fec213e4640 1 --2- 172.21.15.16:0/231887608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f409a7cd90 0x55f409a8de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.552+0000 7fec223e6640 1 --2- 172.21.15.16:0/231887608 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f409a85970 0x55f409a91470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.552+0000 7fec21be5640 1 -- 172.21.15.16:0/231887608 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f409a85970 msgr2=0x55f409a91470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:02.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.552+0000 7fec21be5640 1 --2- 172.21.15.16:0/231887608 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f409a85970 0x55f409a91470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.552+0000 7fec21be5640 1 -- 172.21.15.16:0/231887608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f409a7cd90 msgr2=0x55f409a8de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:02.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.552+0000 7fec21be5640 1 --2- 172.21.15.16:0/231887608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f409a7cd90 0x55f409a8de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.552+0000 7fec21be5640 1 -- 172.21.15.16:0/231887608 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f409a945b0 con 0x55f409a7c470 2024-03-28T22:19:02.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.552+0000 7fec21be5640 1 --2- 172.21.15.16:0/231887608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f409a7c470 0x55f409a85170 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fec18000c00 tx=0x7fec1800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac65005784619dd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:02.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.552+0000 7fec077fe640 1 -- 172.21.15.16:0/231887608 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec18028070 con 0x55f409a7c470 2024-03-28T22:19:02.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.552+0000 7fec077fe640 1 -- 172.21.15.16:0/231887608 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec18029040 con 0x55f409a7c470 2024-03-28T22:19:02.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.552+0000 7fec077fe640 1 -- 172.21.15.16:0/231887608 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec18004660 con 0x55f409a7c470 2024-03-28T22:19:02.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.552+0000 7fec26a20740 1 -- 172.21.15.16:0/231887608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f409a7c470 msgr2=0x55f409a85170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.553+0000 7fec26a20740 1 --2- 172.21.15.16:0/231887608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f409a7c470 0x55f409a85170 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fec18000c00 tx=0x7fec1800cce0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.553+0000 7fec26a20740 1 -- 172.21.15.16:0/231887608 shutdown_connections 2024-03-28T22:19:02.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.553+0000 7fec26a20740 1 --2- 172.21.15.16:0/231887608 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f409a85970 0x55f409a91470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.553+0000 7fec26a20740 1 --2- 172.21.15.16:0/231887608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f409a7cd90 0x55f409a8de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.553+0000 7fec26a20740 1 --2- 172.21.15.16:0/231887608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f409a7c470 0x55f409a85170 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.553+0000 7fec26a20740 1 -- 172.21.15.16:0/231887608 >> 172.21.15.16:0/231887608 conn(0x55f409a76ba0 msgr2=0x55f409a77fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:02.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.553+0000 7fec26a20740 1 -- 172.21.15.16:0/231887608 shutdown_connections 2024-03-28T22:19:02.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.553+0000 7fec26a20740 1 -- 172.21.15.16:0/231887608 wait complete. 2024-03-28T22:19:02.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.553+0000 7fec26a20740 1 -- start start 2024-03-28T22:19:02.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.553+0000 7fec26a20740 1 Processor -- start 2024-03-28T22:19:02.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.553+0000 7fec26a20740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f409a7c470 0x55f409af9460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.553+0000 7fec26a20740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f409a7cd90 0x55f409af99a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.553+0000 7fec26a20740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f409a85970 0x55f409af3d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.553+0000 7fec26a20740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fec1c1ab9b0 con 0x55f409a7cd90 2024-03-28T22:19:02.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.553+0000 7fec26a20740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fec1c1ace20 con 0x55f409a85970 2024-03-28T22:19:02.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.553+0000 7fec26a20740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fec1c1ac080 con 0x55f409a7c470 2024-03-28T22:19:02.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.553+0000 7fec26a20740 10 client.0 _ll_register_callbacks cb 0x7fec180103d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:19:02.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.554+0000 7fec213e4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f409a7cd90 0x55f409af99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.554+0000 7fec213e4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f409a7cd90 0x55f409af99a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59980/0 (socket says 172.21.15.16:59980) 2024-03-28T22:19:02.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.554+0000 7fec213e4640 1 -- 172.21.15.16:0/2099914644 learned_addr learned my addr 172.21.15.16:0/2099914644 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:02.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.554+0000 7fec223e6640 1 --2- 172.21.15.16:0/2099914644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f409a85970 0x55f409af3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.554+0000 7fec21be5640 1 --2- 172.21.15.16:0/2099914644 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f409a7c470 0x55f409af9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.554+0000 7fec213e4640 1 -- 172.21.15.16:0/2099914644 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f409a7c470 msgr2=0x55f409af9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:02.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.554+0000 7fec213e4640 1 --2- 172.21.15.16:0/2099914644 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f409a7c470 0x55f409af9460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.554+0000 7fec213e4640 1 -- 172.21.15.16:0/2099914644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f409a85970 msgr2=0x55f409af3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:02.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.554+0000 7fec213e4640 1 --2- 172.21.15.16:0/2099914644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f409a85970 0x55f409af3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.554+0000 7fec213e4640 1 -- 172.21.15.16:0/2099914644 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f409a87d60 con 0x55f409a7cd90 2024-03-28T22:19:02.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.554+0000 7fec213e4640 1 --2- 172.21.15.16:0/2099914644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f409a7cd90 0x55f409af99a0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fec0c0bc580 tx=0x7fec0c08c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=131e419802501333 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:02.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.554+0000 7fec057fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:19:02.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.554+0000 7fec057fa640 1 -- 172.21.15.16:0/2099914644 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec0c08b070 con 0x55f409a7cd90 2024-03-28T22:19:02.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.554+0000 7fec057fa640 1 -- 172.21.15.16:0/2099914644 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7fec0c0a70b0 con 0x55f409a7cd90 2024-03-28T22:19:02.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.554+0000 7fec26a20740 1 -- 172.21.15.16:0/2099914644 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec18029040 con 0x55f409a7cd90 2024-03-28T22:19:02.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.554+0000 7fec057fa640 1 -- 172.21.15.16:0/2099914644 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fec0c0bcf80 con 0x55f409a7cd90 2024-03-28T22:19:02.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.554+0000 7fec26a20740 10 client.17274 fetch_fsmap learned FSMap version 27 2024-03-28T22:19:02.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.554+0000 7fec26a20740 1 -- 172.21.15.16:0/2099914644 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f409b00b00 con 0x55f409a7cd90 2024-03-28T22:19:02.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.554+0000 7fec057fa640 1 -- 172.21.15.16:0/2099914644 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec18029040 con 0x55f409a7cd90 2024-03-28T22:19:02.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.555+0000 7fec057fa640 1 -- 172.21.15.16:0/2099914644 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7fec0c02cbc0 con 0x55f409a7cd90 2024-03-28T22:19:02.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.555+0000 7fec26a20740 10 client.17274 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:19:02.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.555+0000 7fec26a20740 10 client.17274 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-28T22:19:02.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.555+0000 7fec26a20740 20 client.17274 populate_metadata read hostname 'smithi016' 2024-03-28T22:19:02.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.555+0000 7fec26a20740 1 --2- 172.21.15.16:0/2099914644 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55f409b01200 0x55f409b03620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.555+0000 7fec26a20740 4 client.17274 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:19:02.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.555+0000 7fec26a20740 1 -- 172.21.15.16:0/2099914644 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f4099e7a40 con 0x55f409b01200 2024-03-28T22:19:02.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.555+0000 7fec21be5640 1 --2- 172.21.15.16:0/2099914644 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55f409b01200 0x55f409b03620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.555+0000 7fec057fa640 10 client.17274 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:19:02.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.555+0000 7fec21be5640 1 --2- 172.21.15.16:0/2099914644 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55f409b01200 0x55f409b03620 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dd1b68a8707ded7f server_cookie=33c4215c53c5faf in_seq=0 out_seq=0 2024-03-28T22:19:02.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.555+0000 7fec057fa640 1 -- 172.21.15.16:0/2099914644 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f4099e7a40 con 0x55f409b01200 2024-03-28T22:19:02.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.555+0000 7fec057fa640 10 client.17274 handle_command_reply: tid=0 2024-03-28T22:19:02.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.555+0000 7fec26a20740 1 client.17274 shutdown 2024-03-28T22:19:02.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.556+0000 7fec26a20740 1 -- 172.21.15.16:0/2099914644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f409a7cd90 msgr2=0x55f409af99a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.556+0000 7fec26a20740 1 --2- 172.21.15.16:0/2099914644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f409a7cd90 0x55f409af99a0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fec0c0bc580 tx=0x7fec0c08c6e0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.556+0000 7fec26a20740 1 -- 172.21.15.16:0/2099914644 shutdown_connections 2024-03-28T22:19:02.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.556+0000 7fec26a20740 1 --2- 172.21.15.16:0/2099914644 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55f409b01200 0x55f409b03620 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.556+0000 7fec26a20740 1 --2- 172.21.15.16:0/2099914644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f409a85970 0x55f409af3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.556+0000 7fec057fa640 0 client.17274 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:19:02.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.556+0000 7fec26a20740 1 --2- 172.21.15.16:0/2099914644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f409a7cd90 0x55f409af99a0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.556+0000 7fec26a20740 1 --2- 172.21.15.16:0/2099914644 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f409a7c470 0x55f409af9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.556+0000 7fec26a20740 1 -- 172.21.15.16:0/2099914644 >> 172.21.15.16:0/2099914644 conn(0x55f409a76ba0 msgr2=0x55f409a78860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:02.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.556+0000 7fec26a20740 1 -- 172.21.15.16:0/2099914644 shutdown_connections 2024-03-28T22:19:02.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.556+0000 7fec26a20740 1 -- 172.21.15.16:0/2099914644 wait complete. 2024-03-28T22:19:02.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.556+0000 7fec26a20740 20 client.17274 trim_cache size 0 max 16384 2024-03-28T22:19:02.570 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:19:02.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.556+0000 7fec23e49640 1 -- 172.21.15.16:0/2225014947 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fec080a1ef0 msgr2=0x7fec080a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.556+0000 7fec23e49640 1 --2- 172.21.15.16:0/2225014947 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fec080a1ef0 0x7fec080a43b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fec1400fee0 tx=0x7fec1401e000 comp rx=0 tx=0).stop 2024-03-28T22:19:02.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.556+0000 7fec23e49640 1 -- 172.21.15.16:0/2225014947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec1c103290 msgr2=0x7fec1c1985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.556+0000 7fec23e49640 1 --2- 172.21.15.16:0/2225014947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec1c103290 0x7fec1c1985e0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fec0c000c00 tx=0x7fec0c0043c0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.556+0000 7fec23e49640 1 -- 172.21.15.16:0/2225014947 shutdown_connections 2024-03-28T22:19:02.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.556+0000 7fec23e49640 1 --2- 172.21.15.16:0/2225014947 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fec080a1ef0 0x7fec080a43b0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.556+0000 7fec23e49640 1 --2- 172.21.15.16:0/2225014947 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fec1c10bd40 0x7fec1c19c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.556+0000 7fec23e49640 1 --2- 172.21.15.16:0/2225014947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec1c103290 0x7fec1c1985e0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.556+0000 7fec23e49640 1 --2- 172.21.15.16:0/2225014947 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec1c102970 0x7fec1c1980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.556+0000 7fec23e49640 1 -- 172.21.15.16:0/2225014947 >> 172.21.15.16:0/2225014947 conn(0x7fec1c077770 msgr2=0x7fec1c0fe9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:02.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.556+0000 7fec23e49640 1 -- 172.21.15.16:0/2225014947 shutdown_connections 2024-03-28T22:19:02.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.557+0000 7fec23e49640 1 -- 172.21.15.16:0/2225014947 wait complete. 2024-03-28T22:19:02.573 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:19:02.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.676+0000 7f780f2ed640 1 -- 172.21.15.16:0/3391981310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7808103290 msgr2=0x7f780810b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.676+0000 7f780f2ed640 1 --2- 172.21.15.16:0/3391981310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7808103290 0x7f780810b770 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f77f8009900 tx=0x7f77f802bcd0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.677+0000 7f780f2ed640 1 -- 172.21.15.16:0/3391981310 shutdown_connections 2024-03-28T22:19:02.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.677+0000 7f780f2ed640 1 --2- 172.21.15.16:0/3391981310 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f780810bd40 0x7f780810e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.677+0000 7f780f2ed640 1 --2- 172.21.15.16:0/3391981310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7808103290 0x7f780810b770 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.677+0000 7f780f2ed640 1 --2- 172.21.15.16:0/3391981310 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7808102970 0x7f7808102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.677+0000 7f780f2ed640 1 -- 172.21.15.16:0/3391981310 >> 172.21.15.16:0/3391981310 conn(0x7f7808077770 msgr2=0x7f7808077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:02.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.677+0000 7f780f2ed640 1 -- 172.21.15.16:0/3391981310 shutdown_connections 2024-03-28T22:19:02.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.677+0000 7f780f2ed640 1 -- 172.21.15.16:0/3391981310 wait complete. 2024-03-28T22:19:02.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.677+0000 7f780f2ed640 1 Processor -- start 2024-03-28T22:19:02.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.677+0000 7f780f2ed640 1 -- start start 2024-03-28T22:19:02.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.677+0000 7f780f2ed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7808102970 0x7f78081005c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.677+0000 7f780f2ed640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7808103290 0x7f7808100b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.677+0000 7f780f2ed640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f780810bd40 0x7f7808101040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.677+0000 7f780f2ed640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7808111ad0 con 0x7f7808102970 2024-03-28T22:19:02.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.677+0000 7f780f2ed640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7808111950 con 0x7f780810bd40 2024-03-28T22:19:02.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.677+0000 7f780f2ed640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7808111c50 con 0x7f7808103290 2024-03-28T22:19:02.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.677+0000 7f780d089640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7808102970 0x7f78081005c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.678+0000 7f780d089640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7808102970 0x7f78081005c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60008/0 (socket says 172.21.15.16:60008) 2024-03-28T22:19:02.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.678+0000 7f780d089640 1 -- 172.21.15.16:0/2337426332 learned_addr learned my addr 172.21.15.16:0/2337426332 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:02.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.678+0000 7f780d88a640 1 --2- 172.21.15.16:0/2337426332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f780810bd40 0x7f7808101040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.678+0000 7f780c888640 1 --2- 172.21.15.16:0/2337426332 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7808103290 0x7f7808100b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.678+0000 7f780d089640 1 -- 172.21.15.16:0/2337426332 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7808103290 msgr2=0x7f7808100b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.678+0000 7f780d089640 1 --2- 172.21.15.16:0/2337426332 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7808103290 0x7f7808100b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.678+0000 7f780d089640 1 -- 172.21.15.16:0/2337426332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f780810bd40 msgr2=0x7f7808101040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.678+0000 7f780d089640 1 --2- 172.21.15.16:0/2337426332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f780810bd40 0x7f7808101040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.678+0000 7f780d089640 1 -- 172.21.15.16:0/2337426332 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78081a6530 con 0x7f7808102970 2024-03-28T22:19:02.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.678+0000 7f780c888640 1 --2- 172.21.15.16:0/2337426332 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7808103290 0x7f7808100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:19:02.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.678+0000 7f780d089640 1 --2- 172.21.15.16:0/2337426332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7808102970 0x7f78081005c0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f780000ea80 tx=0x7f780000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:02.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.678+0000 7f77fe7fc640 1 -- 172.21.15.16:0/2337426332 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780000cd80 con 0x7f7808102970 2024-03-28T22:19:02.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.678+0000 7f780f2ed640 1 -- 172.21.15.16:0/2337426332 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78081a6820 con 0x7f7808102970 2024-03-28T22:19:02.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.678+0000 7f77fe7fc640 1 -- 172.21.15.16:0/2337426332 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7800005020 con 0x7f7808102970 2024-03-28T22:19:02.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.678+0000 7f77fe7fc640 1 -- 172.21.15.16:0/2337426332 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78000116d0 con 0x7f7808102970 2024-03-28T22:19:02.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.678+0000 7f780f2ed640 1 -- 172.21.15.16:0/2337426332 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78081a6d70 con 0x7f7808102970 2024-03-28T22:19:02.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.680+0000 7f77fe7fc640 1 -- 172.21.15.16:0/2337426332 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f7800005560 con 0x7f7808102970 2024-03-28T22:19:02.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.681+0000 7f7811faa740 1 Processor -- start 2024-03-28T22:19:02.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.681+0000 7f7811faa740 1 -- start start 2024-03-28T22:19:02.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.681+0000 7f7811faa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559601ef2470 0x559601ef2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.681+0000 7f77fe7fc640 1 --2- 172.21.15.16:0/2337426332 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f77f40a1ef0 0x7f77f40a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.681+0000 7f7811faa740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559601ef2d90 0x559601efb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.681+0000 7f7811faa740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559601efb970 0x559601efdd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.681+0000 7f7811faa740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559601e561b0 con 0x559601ef2470 2024-03-28T22:19:02.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.681+0000 7f7811faa740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x559601dfec00 con 0x559601ef2d90 2024-03-28T22:19:02.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.681+0000 7f7811faa740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x559601e52250 con 0x559601efb970 2024-03-28T22:19:02.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.681+0000 7f780d089640 1 --2- 172.21.15.16:0/2337426332 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f77f40a1ef0 0x7f77f40a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.681+0000 7f780d88a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559601ef2470 0x559601ef2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.681+0000 7f780d88a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559601ef2470 0x559601ef2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60020/0 (socket says 172.21.15.16:60020) 2024-03-28T22:19:02.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.681+0000 7f780c888640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559601ef2d90 0x559601efb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.681+0000 7f780d88a640 1 -- 172.21.15.16:0/976444327 learned_addr learned my addr 172.21.15.16:0/976444327 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:02.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.681+0000 7f77fe7fc640 1 -- 172.21.15.16:0/2337426332 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7f78000ce810 con 0x7f7808102970 2024-03-28T22:19:02.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.681+0000 7f780d88a640 1 --2- 172.21.15.16:0/976444327 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559601efb970 0x559601efdd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.681+0000 7f780d88a640 1 -- 172.21.15.16:0/976444327 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559601efb970 msgr2=0x559601efdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:02.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.681+0000 7f780d88a640 1 --2- 172.21.15.16:0/976444327 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559601efb970 0x559601efdd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.681+0000 7f780d88a640 1 -- 172.21.15.16:0/976444327 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559601ef2d90 msgr2=0x559601efb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:02.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.681+0000 7f780d88a640 1 --2- 172.21.15.16:0/976444327 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559601ef2d90 0x559601efb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.681+0000 7f780d88a640 1 -- 172.21.15.16:0/976444327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559601e1e3d0 con 0x559601ef2470 2024-03-28T22:19:02.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.681+0000 7f780c888640 1 --2- 172.21.15.16:0/976444327 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559601ef2d90 0x559601efb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:19:02.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.682+0000 7f780d089640 1 --2- 172.21.15.16:0/2337426332 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f77f40a1ef0 0x7f77f40a43b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f78000cc010 tx=0x7f7800015b80 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:02.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.682+0000 7f780d88a640 1 --2- 172.21.15.16:0/976444327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559601ef2470 0x559601ef2850 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f78040095f0 tx=0x7f7804009760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5910c13951e2f28 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:02.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.682+0000 7f77f37fe640 1 -- 172.21.15.16:0/976444327 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7804004050 con 0x559601ef2470 2024-03-28T22:19:02.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.682+0000 7f77f37fe640 1 -- 172.21.15.16:0/976444327 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78040041f0 con 0x559601ef2470 2024-03-28T22:19:02.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.682+0000 7f77f37fe640 1 -- 172.21.15.16:0/976444327 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780400f6e0 con 0x559601ef2470 2024-03-28T22:19:02.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.682+0000 7f7811faa740 1 -- 172.21.15.16:0/976444327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559601ef2470 msgr2=0x559601ef2850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.682+0000 7f7811faa740 1 --2- 172.21.15.16:0/976444327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559601ef2470 0x559601ef2850 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f78040095f0 tx=0x7f7804009760 comp rx=0 tx=0).stop 2024-03-28T22:19:02.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.683+0000 7f7811faa740 1 -- 172.21.15.16:0/976444327 shutdown_connections 2024-03-28T22:19:02.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.683+0000 7f7811faa740 1 --2- 172.21.15.16:0/976444327 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559601efb970 0x559601efdd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.683+0000 7f7811faa740 1 --2- 172.21.15.16:0/976444327 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559601ef2d90 0x559601efb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.683+0000 7f7811faa740 1 --2- 172.21.15.16:0/976444327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559601ef2470 0x559601ef2850 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.683+0000 7f7811faa740 1 -- 172.21.15.16:0/976444327 >> 172.21.15.16:0/976444327 conn(0x559601eecba0 msgr2=0x559601eecf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:02.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.683+0000 7f7811faa740 1 -- 172.21.15.16:0/976444327 shutdown_connections 2024-03-28T22:19:02.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.683+0000 7f7811faa740 1 -- 172.21.15.16:0/976444327 wait complete. 2024-03-28T22:19:02.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.683+0000 7f7811faa740 1 -- start start 2024-03-28T22:19:02.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.683+0000 7f7811faa740 1 Processor -- start 2024-03-28T22:19:02.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.683+0000 7f7811faa740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559601ef2470 0x559601ef8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.683+0000 7f7811faa740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559601ef2d90 0x559601ef8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.683+0000 7f7811faa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559601efb970 0x559601ef93e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.683+0000 7f7811faa740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559601efe760 con 0x559601efb970 2024-03-28T22:19:02.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.683+0000 7f7811faa740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x559601df4470 con 0x559601ef2470 2024-03-28T22:19:02.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.683+0000 7f7811faa740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x559601efe8e0 con 0x559601ef2d90 2024-03-28T22:19:02.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.684+0000 7f7811faa740 10 client.0 _ll_register_callbacks cb 0x559601e11cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:19:02.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.684+0000 7f780d88a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559601efb970 0x559601ef93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.684+0000 7f780d88a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559601efb970 0x559601ef93e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60024/0 (socket says 172.21.15.16:60024) 2024-03-28T22:19:02.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.684+0000 7f780d88a640 1 -- 172.21.15.16:0/2092223641 learned_addr learned my addr 172.21.15.16:0/2092223641 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:02.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.684+0000 7f780c888640 1 --2- 172.21.15.16:0/2092223641 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559601ef2d90 0x559601ef8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.684+0000 7f780d88a640 1 --2- 172.21.15.16:0/2092223641 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559601ef2470 0x559601ef8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.684+0000 7f780d88a640 1 -- 172.21.15.16:0/2092223641 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559601ef2d90 msgr2=0x559601ef8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:02.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.684+0000 7f780d88a640 1 --2- 172.21.15.16:0/2092223641 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559601ef2d90 0x559601ef8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.684+0000 7f780d88a640 1 -- 172.21.15.16:0/2092223641 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559601ef2470 msgr2=0x559601ef8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:02.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.684+0000 7f780d88a640 1 --2- 172.21.15.16:0/2092223641 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559601ef2470 0x559601ef8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.684+0000 7f780d88a640 1 -- 172.21.15.16:0/2092223641 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559601b6a3f0 con 0x559601efb970 2024-03-28T22:19:02.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.684+0000 7f780d88a640 1 --2- 172.21.15.16:0/2092223641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559601efb970 0x559601ef93e0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f7804009ef0 tx=0x7f7804003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f4fd45966c0e5a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:02.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.684+0000 7f77f0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:19:02.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.684+0000 7f77f0ff9640 1 -- 172.21.15.16:0/2092223641 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7804028040 con 0x559601efb970 2024-03-28T22:19:02.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.684+0000 7f77f0ff9640 1 -- 172.21.15.16:0/2092223641 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7f7804005050 con 0x559601efb970 2024-03-28T22:19:02.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.684+0000 7f7811faa740 1 -- 172.21.15.16:0/2092223641 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78040041f0 con 0x559601efb970 2024-03-28T22:19:02.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.684+0000 7f77f0ff9640 1 -- 172.21.15.16:0/2092223641 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7804005250 con 0x559601efb970 2024-03-28T22:19:02.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.684+0000 7f7811faa740 10 client.17286 fetch_fsmap learned FSMap version 27 2024-03-28T22:19:02.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.684+0000 7f7811faa740 1 -- 172.21.15.16:0/2092223641 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559601c347d0 con 0x559601efb970 2024-03-28T22:19:02.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.685+0000 7f77f0ff9640 1 -- 172.21.15.16:0/2092223641 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78040041f0 con 0x559601efb970 2024-03-28T22:19:02.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.685+0000 7f77f0ff9640 1 -- 172.21.15.16:0/2092223641 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f780400f480 con 0x559601efb970 2024-03-28T22:19:02.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.685+0000 7f7811faa740 10 client.17286 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:19:02.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.685+0000 7f7811faa740 10 client.17286 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T22:19:02.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.685+0000 7f7811faa740 20 client.17286 populate_metadata read hostname 'smithi016' 2024-03-28T22:19:02.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.685+0000 7f7811faa740 1 --2- 172.21.15.16:0/2092223641 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x559601f0b7a0 0x559601f0db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.685+0000 7f7811faa740 4 client.17286 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:19:02.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.685+0000 7f7811faa740 1 -- 172.21.15.16:0/2092223641 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559601e636f0 con 0x559601f0b7a0 2024-03-28T22:19:02.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.685+0000 7f780c888640 1 --2- 172.21.15.16:0/2092223641 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x559601f0b7a0 0x559601f0db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.685+0000 7f77f0ff9640 10 client.17286 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:19:02.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.685+0000 7f780c888640 1 --2- 172.21.15.16:0/2092223641 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x559601f0b7a0 0x559601f0db90 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d9778c27094bf51 server_cookie=c3ef9c6bc051930b in_seq=0 out_seq=0 2024-03-28T22:19:02.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.686+0000 7f77f0ff9640 1 -- 172.21.15.16:0/2092223641 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559601e636f0 con 0x559601f0b7a0 2024-03-28T22:19:02.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.686+0000 7f77f0ff9640 10 client.17286 handle_command_reply: tid=0 2024-03-28T22:19:02.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.686+0000 7f7811faa740 1 client.17286 shutdown 2024-03-28T22:19:02.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.686+0000 7f7811faa740 1 -- 172.21.15.16:0/2092223641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559601efb970 msgr2=0x559601ef93e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.686+0000 7f7811faa740 1 --2- 172.21.15.16:0/2092223641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559601efb970 0x559601ef93e0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f7804009ef0 tx=0x7f7804003d50 comp rx=0 tx=0).stop 2024-03-28T22:19:02.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.687+0000 7f7811faa740 1 -- 172.21.15.16:0/2092223641 shutdown_connections 2024-03-28T22:19:02.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.687+0000 7f7811faa740 1 --2- 172.21.15.16:0/2092223641 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x559601f0b7a0 0x559601f0db90 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.687+0000 7f7811faa740 1 --2- 172.21.15.16:0/2092223641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559601efb970 0x559601ef93e0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.687+0000 7f77f0ff9640 0 client.17286 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:19:02.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.687+0000 7f7811faa740 1 --2- 172.21.15.16:0/2092223641 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559601ef2d90 0x559601ef8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.687+0000 7f7811faa740 1 --2- 172.21.15.16:0/2092223641 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559601ef2470 0x559601ef8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.687+0000 7f7811faa740 1 -- 172.21.15.16:0/2092223641 >> 172.21.15.16:0/2092223641 conn(0x559601eecba0 msgr2=0x559601eee8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:02.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.687+0000 7f7811faa740 1 -- 172.21.15.16:0/2092223641 shutdown_connections 2024-03-28T22:19:02.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.687+0000 7f7811faa740 1 -- 172.21.15.16:0/2092223641 wait complete. 2024-03-28T22:19:02.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.687+0000 7f7811faa740 20 client.17286 trim_cache size 0 max 16384 2024-03-28T22:19:02.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.700+0000 7f7811faa740 1 Processor -- start 2024-03-28T22:19:02.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.700+0000 7f7811faa740 1 -- start start 2024-03-28T22:19:02.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.700+0000 7f7811faa740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559601ef2470 0x559601efb090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.700+0000 7f7811faa740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559601ef2d90 0x559601f07500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.700+0000 7f7811faa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559601efb970 0x559601f133a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.700+0000 7f7811faa740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f78081b1f00 con 0x559601efb970 2024-03-28T22:19:02.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.700+0000 7f7811faa740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f78081af7d0 con 0x559601ef2470 2024-03-28T22:19:02.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.700+0000 7f7811faa740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f78081ad640 con 0x559601ef2d90 2024-03-28T22:19:02.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.700+0000 7f780d88a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559601efb970 0x559601f133a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.700+0000 7f780d88a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559601efb970 0x559601f133a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60038/0 (socket says 172.21.15.16:60038) 2024-03-28T22:19:02.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.700+0000 7f780d88a640 1 -- 172.21.15.16:0/107829422 learned_addr learned my addr 172.21.15.16:0/107829422 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:02.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.700+0000 7f780d88a640 1 --2- 172.21.15.16:0/107829422 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559601ef2470 0x559601efb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.700+0000 7f780d88a640 1 -- 172.21.15.16:0/107829422 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559601ef2d90 msgr2=0x559601f07500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:02.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.700+0000 7f780d88a640 1 --2- 172.21.15.16:0/107829422 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559601ef2d90 0x559601f07500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.700+0000 7f780d88a640 1 -- 172.21.15.16:0/107829422 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559601ef2470 msgr2=0x559601efb090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:02.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.700+0000 7f780d88a640 1 --2- 172.21.15.16:0/107829422 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559601ef2470 0x559601efb090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.700+0000 7f780d88a640 1 -- 172.21.15.16:0/107829422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559601efe090 con 0x559601efb970 2024-03-28T22:19:02.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.701+0000 7f780d88a640 1 --2- 172.21.15.16:0/107829422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559601efb970 0x559601f133a0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x559601f0a5f0 tx=0x7f78040053f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec24c7b314d6171a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:02.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.701+0000 7f77f2ffd640 1 -- 172.21.15.16:0/107829422 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780400f480 con 0x559601efb970 2024-03-28T22:19:02.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.701+0000 7f77f2ffd640 1 -- 172.21.15.16:0/107829422 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f780400f620 con 0x559601efb970 2024-03-28T22:19:02.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.701+0000 7f77f2ffd640 1 -- 172.21.15.16:0/107829422 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780401f430 con 0x559601efb970 2024-03-28T22:19:02.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.701+0000 7f7811faa740 1 -- 172.21.15.16:0/107829422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559601efb970 msgr2=0x559601f133a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.701+0000 7f7811faa740 1 --2- 172.21.15.16:0/107829422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559601efb970 0x559601f133a0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x559601f0a5f0 tx=0x7f78040053f0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.701+0000 7f7811faa740 1 -- 172.21.15.16:0/107829422 shutdown_connections 2024-03-28T22:19:02.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.701+0000 7f7811faa740 1 --2- 172.21.15.16:0/107829422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559601efb970 0x559601f133a0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.701+0000 7f7811faa740 1 --2- 172.21.15.16:0/107829422 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559601ef2d90 0x559601f07500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.701+0000 7f7811faa740 1 --2- 172.21.15.16:0/107829422 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559601ef2470 0x559601efb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.701+0000 7f7811faa740 1 -- 172.21.15.16:0/107829422 >> 172.21.15.16:0/107829422 conn(0x559601eecba0 msgr2=0x559601eebd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:02.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.701+0000 7f7811faa740 1 -- 172.21.15.16:0/107829422 shutdown_connections 2024-03-28T22:19:02.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.701+0000 7f7811faa740 1 -- 172.21.15.16:0/107829422 wait complete. 2024-03-28T22:19:02.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.702+0000 7f7811faa740 1 -- start start 2024-03-28T22:19:02.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.702+0000 7f7811faa740 1 Processor -- start 2024-03-28T22:19:02.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.702+0000 7f7811faa740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559601ef2470 0x559601f03b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.702+0000 7f7811faa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559601ef2d90 0x559601f00b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.702+0000 7f7811faa740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559601efb970 0x559601f01040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.702+0000 7f7811faa740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f78081adc90 con 0x559601ef2d90 2024-03-28T22:19:02.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.702+0000 7f7811faa740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f78081af100 con 0x559601ef2470 2024-03-28T22:19:02.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.702+0000 7f7811faa740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f78081ae360 con 0x559601efb970 2024-03-28T22:19:02.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.702+0000 7f7811faa740 10 client.0 _ll_register_callbacks cb 0x7f78040039d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:19:02.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.702+0000 7f780c888640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559601ef2d90 0x559601f00b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.702+0000 7f780d88a640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559601ef2470 0x559601f03b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.702+0000 7f780c888640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559601ef2d90 0x559601f00b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60052/0 (socket says 172.21.15.16:60052) 2024-03-28T22:19:02.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.702+0000 7f780c888640 1 -- 172.21.15.16:0/1376974169 learned_addr learned my addr 172.21.15.16:0/1376974169 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:02.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.702+0000 7f780d88a640 1 --2- 172.21.15.16:0/1376974169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559601efb970 0x559601f01040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.703+0000 7f780c888640 1 -- 172.21.15.16:0/1376974169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559601efb970 msgr2=0x559601f01040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:02.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.703+0000 7f780c888640 1 --2- 172.21.15.16:0/1376974169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559601efb970 0x559601f01040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.703+0000 7f780c888640 1 -- 172.21.15.16:0/1376974169 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559601ef2470 msgr2=0x559601f03b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:02.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.703+0000 7f780c888640 1 --2- 172.21.15.16:0/1376974169 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559601ef2470 0x559601f03b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.703+0000 7f780c888640 1 -- 172.21.15.16:0/1376974169 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559601f01580 con 0x559601ef2d90 2024-03-28T22:19:02.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.703+0000 7f780d88a640 1 --2- 172.21.15.16:0/1376974169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559601efb970 0x559601f01040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:19:02.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.703+0000 7f780c888640 1 --2- 172.21.15.16:0/1376974169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559601ef2d90 0x559601f00b00 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f77f8009230 tx=0x7f77f802ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86b843133b987239 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:02.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.703+0000 7f77f0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:19:02.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.703+0000 7f77f0ff9640 1 -- 172.21.15.16:0/1376974169 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f8045ca0 con 0x559601ef2d90 2024-03-28T22:19:02.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.703+0000 7f77f0ff9640 1 -- 172.21.15.16:0/1376974169 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7f77f8004030 con 0x559601ef2d90 2024-03-28T22:19:02.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.703+0000 7f7811faa740 1 -- 172.21.15.16:0/1376974169 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f780400f620 con 0x559601ef2d90 2024-03-28T22:19:02.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.703+0000 7f77f0ff9640 1 -- 172.21.15.16:0/1376974169 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f77f8053050 con 0x559601ef2d90 2024-03-28T22:19:02.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.703+0000 7f7811faa740 10 client.17292 fetch_fsmap learned FSMap version 27 2024-03-28T22:19:02.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.703+0000 7f7811faa740 1 -- 172.21.15.16:0/1376974169 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559601f72790 con 0x559601ef2d90 2024-03-28T22:19:02.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.703+0000 7f77f0ff9640 1 -- 172.21.15.16:0/1376974169 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f780400f620 con 0x559601ef2d90 2024-03-28T22:19:02.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.703+0000 7f77f0ff9640 1 -- 172.21.15.16:0/1376974169 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f77f8005460 con 0x559601ef2d90 2024-03-28T22:19:02.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.703+0000 7f7811faa740 10 client.17292 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:19:02.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.703+0000 7f7811faa740 10 client.17292 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T22:19:02.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.703+0000 7f7811faa740 20 client.17292 populate_metadata read hostname 'smithi016' 2024-03-28T22:19:02.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.703+0000 7f7811faa740 1 --2- 172.21.15.16:0/1376974169 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x559601f72ee0 0x559601f75300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.703+0000 7f7811faa740 4 client.17292 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:19:02.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.703+0000 7f7811faa740 1 -- 172.21.15.16:0/1376974169 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559601e5da40 con 0x559601f72ee0 2024-03-28T22:19:02.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.704+0000 7f780d88a640 1 --2- 172.21.15.16:0/1376974169 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x559601f72ee0 0x559601f75300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.704+0000 7f77f0ff9640 10 client.17292 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:19:02.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.704+0000 7f780d88a640 1 --2- 172.21.15.16:0/1376974169 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x559601f72ee0 0x559601f75300 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dadd67f610366019 server_cookie=abaf8fd4775c0e39 in_seq=0 out_seq=0 2024-03-28T22:19:02.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.704+0000 7f77f0ff9640 1 -- 172.21.15.16:0/1376974169 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559601e5da40 con 0x559601f72ee0 2024-03-28T22:19:02.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.704+0000 7f77f0ff9640 10 client.17292 handle_command_reply: tid=0 2024-03-28T22:19:02.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.704+0000 7f7811faa740 1 client.17292 shutdown 2024-03-28T22:19:02.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.704+0000 7f7811faa740 1 -- 172.21.15.16:0/1376974169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559601ef2d90 msgr2=0x559601f00b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.704+0000 7f7811faa740 1 --2- 172.21.15.16:0/1376974169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559601ef2d90 0x559601f00b00 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f77f8009230 tx=0x7f77f802ccb0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.704+0000 7f7811faa740 1 -- 172.21.15.16:0/1376974169 shutdown_connections 2024-03-28T22:19:02.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.704+0000 7f7811faa740 1 --2- 172.21.15.16:0/1376974169 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x559601f72ee0 0x559601f75300 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.704+0000 7f7811faa740 1 --2- 172.21.15.16:0/1376974169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559601efb970 0x559601f01040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.704+0000 7f7811faa740 1 --2- 172.21.15.16:0/1376974169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559601ef2d90 0x559601f00b00 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.704+0000 7f7811faa740 1 --2- 172.21.15.16:0/1376974169 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559601ef2470 0x559601f03b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.704+0000 7f77f0ff9640 0 client.17292 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:19:02.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.704+0000 7f7811faa740 1 -- 172.21.15.16:0/1376974169 >> 172.21.15.16:0/1376974169 conn(0x559601eecba0 msgr2=0x559601eeb4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:02.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.704+0000 7f7811faa740 1 -- 172.21.15.16:0/1376974169 shutdown_connections 2024-03-28T22:19:02.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.704+0000 7f7811faa740 1 -- 172.21.15.16:0/1376974169 wait complete. 2024-03-28T22:19:02.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.704+0000 7f7811faa740 20 client.17292 trim_cache size 0 max 16384 2024-03-28T22:19:02.719 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:19:02.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.705+0000 7f780f2ed640 1 -- 172.21.15.16:0/2337426332 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f77f40a1ef0 msgr2=0x7f77f40a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.705+0000 7f780f2ed640 1 --2- 172.21.15.16:0/2337426332 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f77f40a1ef0 0x7f77f40a43b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f78000cc010 tx=0x7f7800015b80 comp rx=0 tx=0).stop 2024-03-28T22:19:02.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.705+0000 7f780f2ed640 1 -- 172.21.15.16:0/2337426332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7808102970 msgr2=0x7f78081005c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.705+0000 7f780f2ed640 1 --2- 172.21.15.16:0/2337426332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7808102970 0x7f78081005c0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f780000ea80 tx=0x7f780000ee80 comp rx=0 tx=0).stop 2024-03-28T22:19:02.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.705+0000 7f780f2ed640 1 -- 172.21.15.16:0/2337426332 shutdown_connections 2024-03-28T22:19:02.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.705+0000 7f780f2ed640 1 --2- 172.21.15.16:0/2337426332 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f77f40a1ef0 0x7f77f40a43b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.705+0000 7f780f2ed640 1 --2- 172.21.15.16:0/2337426332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f780810bd40 0x7f7808101040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.705+0000 7f780f2ed640 1 --2- 172.21.15.16:0/2337426332 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7808103290 0x7f7808100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.705+0000 7f780f2ed640 1 --2- 172.21.15.16:0/2337426332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7808102970 0x7f78081005c0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.705+0000 7f780f2ed640 1 -- 172.21.15.16:0/2337426332 >> 172.21.15.16:0/2337426332 conn(0x7f7808077770 msgr2=0x7f78080fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:02.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.705+0000 7f780f2ed640 1 -- 172.21.15.16:0/2337426332 shutdown_connections 2024-03-28T22:19:02.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.705+0000 7f780f2ed640 1 -- 172.21.15.16:0/2337426332 wait complete. 2024-03-28T22:19:02.722 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:19:02.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.825+0000 7f263f60e640 1 -- 172.21.15.16:0/3876534158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f263810be30 msgr2=0x7f263810e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.825+0000 7f263f60e640 1 --2- 172.21.15.16:0/3876534158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f263810be30 0x7f263810e2a0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f2630009be0 tx=0x7f263002bf80 comp rx=0 tx=0).stop 2024-03-28T22:19:02.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.825+0000 7f263f60e640 1 -- 172.21.15.16:0/3876534158 shutdown_connections 2024-03-28T22:19:02.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.825+0000 7f263f60e640 1 --2- 172.21.15.16:0/3876534158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f263810be30 0x7f263810e2a0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.825+0000 7f263f60e640 1 --2- 172.21.15.16:0/3876534158 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f26381032e0 0x7f263810b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.825+0000 7f263f60e640 1 --2- 172.21.15.16:0/3876534158 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f26381029c0 0x7f2638102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.825+0000 7f263f60e640 1 -- 172.21.15.16:0/3876534158 >> 172.21.15.16:0/3876534158 conn(0x7f26380777c0 msgr2=0x7f2638077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:02.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.825+0000 7f263f60e640 1 -- 172.21.15.16:0/3876534158 shutdown_connections 2024-03-28T22:19:02.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.825+0000 7f263f60e640 1 -- 172.21.15.16:0/3876534158 wait complete. 2024-03-28T22:19:02.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.826+0000 7f263f60e640 1 Processor -- start 2024-03-28T22:19:02.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.826+0000 7f263f60e640 1 -- start start 2024-03-28T22:19:02.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.826+0000 7f263f60e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f26381029c0 0x7f26381980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.826+0000 7f263f60e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26381032e0 0x7f26381985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.826+0000 7f263f60e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f263810be30 0x7f263819c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.826+0000 7f263f60e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2638111c00 con 0x7f26381032e0 2024-03-28T22:19:02.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.826+0000 7f263f60e640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f2638111a80 con 0x7f26381029c0 2024-03-28T22:19:02.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.826+0000 7f263f60e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2638111d80 con 0x7f263810be30 2024-03-28T22:19:02.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.826+0000 7f263cba9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26381032e0 0x7f26381985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.826+0000 7f263cba9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26381032e0 0x7f26381985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60066/0 (socket says 172.21.15.16:60066) 2024-03-28T22:19:02.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.826+0000 7f263cba9640 1 -- 172.21.15.16:0/2552984136 learned_addr learned my addr 172.21.15.16:0/2552984136 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:02.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.826+0000 7f263d3aa640 1 --2- 172.21.15.16:0/2552984136 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f26381029c0 0x7f26381980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.826+0000 7f263dbab640 1 --2- 172.21.15.16:0/2552984136 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f263810be30 0x7f263819c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.826+0000 7f263cba9640 1 -- 172.21.15.16:0/2552984136 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f263810be30 msgr2=0x7f263819c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.826+0000 7f263cba9640 1 --2- 172.21.15.16:0/2552984136 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f263810be30 0x7f263819c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.826+0000 7f263cba9640 1 -- 172.21.15.16:0/2552984136 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f26381029c0 msgr2=0x7f26381980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.826+0000 7f263cba9640 1 --2- 172.21.15.16:0/2552984136 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f26381029c0 0x7f26381980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.826+0000 7f263cba9640 1 -- 172.21.15.16:0/2552984136 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f263819d090 con 0x7f26381032e0 2024-03-28T22:19:02.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.827+0000 7f263cba9640 1 --2- 172.21.15.16:0/2552984136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26381032e0 0x7f26381985e0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f262800d770 tx=0x7f262800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:02.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.827+0000 7f262e7fc640 1 -- 172.21.15.16:0/2552984136 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2628017070 con 0x7f26381032e0 2024-03-28T22:19:02.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.827+0000 7f262e7fc640 1 -- 172.21.15.16:0/2552984136 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2628005500 con 0x7f26381032e0 2024-03-28T22:19:02.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.827+0000 7f263f60e640 1 -- 172.21.15.16:0/2552984136 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f263819d380 con 0x7f26381032e0 2024-03-28T22:19:02.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.827+0000 7f262e7fc640 1 -- 172.21.15.16:0/2552984136 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2628004e00 con 0x7f26381032e0 2024-03-28T22:19:02.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.827+0000 7f263f60e640 1 -- 172.21.15.16:0/2552984136 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26381a4c10 con 0x7f26381032e0 2024-03-28T22:19:02.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.829+0000 7f262e7fc640 1 -- 172.21.15.16:0/2552984136 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f2628004fa0 con 0x7f26381032e0 2024-03-28T22:19:02.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.829+0000 7f26421f5740 1 Processor -- start 2024-03-28T22:19:02.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.830+0000 7f26421f5740 1 -- start start 2024-03-28T22:19:02.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.830+0000 7f262e7fc640 1 --2- 172.21.15.16:0/2552984136 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f26240a1ef0 0x7f26240a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.830+0000 7f26421f5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5589ff6ae1a0 0x5589ff6ae580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.830+0000 7f26421f5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ff6aeac0 0x5589ff6bb840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.830+0000 7f26421f5740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5589ff6bbd80 0x5589ff6be170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.830+0000 7f26421f5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5589ff6121b0 con 0x5589ff6aeac0 2024-03-28T22:19:02.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.830+0000 7f26421f5740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5589ff5bac00 con 0x5589ff6bbd80 2024-03-28T22:19:02.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.830+0000 7f26421f5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5589ff60e250 con 0x5589ff6ae1a0 2024-03-28T22:19:02.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.830+0000 7f263dbab640 1 --2- 172.21.15.16:0/2552984136 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f26240a1ef0 0x7f26240a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.830+0000 7f263cba9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ff6aeac0 0x5589ff6bb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.830+0000 7f263cba9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ff6aeac0 0x5589ff6bb840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60068/0 (socket says 172.21.15.16:60068) 2024-03-28T22:19:02.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.830+0000 7f263cba9640 1 -- 172.21.15.16:0/2639460975 learned_addr learned my addr 172.21.15.16:0/2639460975 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:02.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.830+0000 7f263dbab640 1 --2- 172.21.15.16:0/2639460975 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5589ff6bbd80 0x5589ff6be170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.830+0000 7f262e7fc640 1 -- 172.21.15.16:0/2552984136 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7f26280cdab0 con 0x7f26381032e0 2024-03-28T22:19:02.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.830+0000 7f263d3aa640 1 --2- 172.21.15.16:0/2639460975 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5589ff6ae1a0 0x5589ff6ae580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.830+0000 7f263cba9640 1 -- 172.21.15.16:0/2639460975 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5589ff6ae1a0 msgr2=0x5589ff6ae580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:02.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.830+0000 7f263cba9640 1 --2- 172.21.15.16:0/2639460975 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5589ff6ae1a0 0x5589ff6ae580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.830+0000 7f263cba9640 1 -- 172.21.15.16:0/2639460975 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5589ff6bbd80 msgr2=0x5589ff6be170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:02.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.830+0000 7f263cba9640 1 --2- 172.21.15.16:0/2639460975 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5589ff6bbd80 0x5589ff6be170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.830+0000 7f263cba9640 1 -- 172.21.15.16:0/2639460975 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589ff5da3d0 con 0x5589ff6aeac0 2024-03-28T22:19:02.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.830+0000 7f263dbab640 1 --2- 172.21.15.16:0/2552984136 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f26240a1ef0 0x7f26240a43b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2630005f00 tx=0x7f263002d930 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:02.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.830+0000 7f263cba9640 1 --2- 172.21.15.16:0/2639460975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ff6aeac0 0x5589ff6bb840 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f2628096f20 tx=0x7f2628009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c497a0626afac95 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:02.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.831+0000 7f26237fe640 1 -- 172.21.15.16:0/2639460975 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2628020070 con 0x5589ff6aeac0 2024-03-28T22:19:02.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.831+0000 7f26237fe640 1 -- 172.21.15.16:0/2639460975 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26280046c0 con 0x5589ff6aeac0 2024-03-28T22:19:02.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.831+0000 7f26237fe640 1 -- 172.21.15.16:0/2639460975 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26280cb480 con 0x5589ff6aeac0 2024-03-28T22:19:02.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.831+0000 7f26421f5740 1 -- 172.21.15.16:0/2639460975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ff6aeac0 msgr2=0x5589ff6bb840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.831+0000 7f26421f5740 1 --2- 172.21.15.16:0/2639460975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ff6aeac0 0x5589ff6bb840 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f2628096f20 tx=0x7f2628009c30 comp rx=0 tx=0).stop 2024-03-28T22:19:02.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.831+0000 7f26421f5740 1 -- 172.21.15.16:0/2639460975 shutdown_connections 2024-03-28T22:19:02.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.831+0000 7f26421f5740 1 --2- 172.21.15.16:0/2639460975 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5589ff6bbd80 0x5589ff6be170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.831+0000 7f26421f5740 1 --2- 172.21.15.16:0/2639460975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ff6aeac0 0x5589ff6bb840 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.831+0000 7f26421f5740 1 --2- 172.21.15.16:0/2639460975 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5589ff6ae1a0 0x5589ff6ae580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.831+0000 7f26421f5740 1 -- 172.21.15.16:0/2639460975 >> 172.21.15.16:0/2639460975 conn(0x5589ff6a8ba0 msgr2=0x5589ff6a8f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:02.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.831+0000 7f26421f5740 1 -- 172.21.15.16:0/2639460975 shutdown_connections 2024-03-28T22:19:02.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.831+0000 7f26421f5740 1 -- 172.21.15.16:0/2639460975 wait complete. 2024-03-28T22:19:02.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.832+0000 7f26421f5740 1 -- start start 2024-03-28T22:19:02.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.832+0000 7f26421f5740 1 Processor -- start 2024-03-28T22:19:02.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.832+0000 7f26421f5740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5589ff6ae1a0 0x5589ff6b8e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.832+0000 7f26421f5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ff6aeac0 0x5589ff6b9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.832+0000 7f26421f5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5589ff6bbd80 0x5589ff6b98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.832+0000 7f26421f5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5589ff6bea80 con 0x5589ff6aeac0 2024-03-28T22:19:02.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.832+0000 7f26421f5740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5589ff5b0470 con 0x5589ff6ae1a0 2024-03-28T22:19:02.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.832+0000 7f26421f5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5589ff6bec00 con 0x5589ff6bbd80 2024-03-28T22:19:02.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.832+0000 7f26421f5740 10 client.0 _ll_register_callbacks cb 0x5589ff5cdcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:19:02.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.832+0000 7f263cba9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ff6aeac0 0x5589ff6b9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.832+0000 7f263cba9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ff6aeac0 0x5589ff6b9360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60084/0 (socket says 172.21.15.16:60084) 2024-03-28T22:19:02.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.832+0000 7f263cba9640 1 -- 172.21.15.16:0/2511000179 learned_addr learned my addr 172.21.15.16:0/2511000179 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:02.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.832+0000 7f263d3aa640 1 --2- 172.21.15.16:0/2511000179 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5589ff6ae1a0 0x5589ff6b8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.832+0000 7f263dbab640 1 --2- 172.21.15.16:0/2511000179 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5589ff6bbd80 0x5589ff6b98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.833+0000 7f263cba9640 1 -- 172.21.15.16:0/2511000179 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5589ff6bbd80 msgr2=0x5589ff6b98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:02.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.833+0000 7f263cba9640 1 --2- 172.21.15.16:0/2511000179 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5589ff6bbd80 0x5589ff6b98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.833+0000 7f263cba9640 1 -- 172.21.15.16:0/2511000179 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5589ff6ae1a0 msgr2=0x5589ff6b8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:02.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.833+0000 7f263cba9640 1 --2- 172.21.15.16:0/2511000179 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5589ff6ae1a0 0x5589ff6b8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.833+0000 7f263cba9640 1 -- 172.21.15.16:0/2511000179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589ff3263f0 con 0x5589ff6aeac0 2024-03-28T22:19:02.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.833+0000 7f263cba9640 1 --2- 172.21.15.16:0/2511000179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ff6aeac0 0x5589ff6b9360 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f262809a4e0 tx=0x7f26280cb1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8364ec08a0315e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:02.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.833+0000 7f2620ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:19:02.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.833+0000 7f2620ff9640 1 -- 172.21.15.16:0/2511000179 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2628020050 con 0x5589ff6aeac0 2024-03-28T22:19:02.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.833+0000 7f2620ff9640 1 -- 172.21.15.16:0/2511000179 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7f2628096bc0 con 0x5589ff6aeac0 2024-03-28T22:19:02.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.833+0000 7f26421f5740 1 -- 172.21.15.16:0/2511000179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5589ff61f6f0 con 0x5589ff6aeac0 2024-03-28T22:19:02.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.833+0000 7f2620ff9640 1 -- 172.21.15.16:0/2511000179 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f262809e5f0 con 0x5589ff6aeac0 2024-03-28T22:19:02.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.833+0000 7f26421f5740 10 client.17304 fetch_fsmap learned FSMap version 27 2024-03-28T22:19:02.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.833+0000 7f26421f5740 1 -- 172.21.15.16:0/2511000179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5589ff3f07d0 con 0x5589ff6aeac0 2024-03-28T22:19:02.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.833+0000 7f2620ff9640 1 -- 172.21.15.16:0/2511000179 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5589ff61f6f0 con 0x5589ff6aeac0 2024-03-28T22:19:02.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.833+0000 7f2620ff9640 1 -- 172.21.15.16:0/2511000179 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f26280b3470 con 0x5589ff6aeac0 2024-03-28T22:19:02.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.833+0000 7f26421f5740 10 client.17304 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:19:02.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.833+0000 7f26421f5740 10 client.17304 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:19:02.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.833+0000 7f26421f5740 20 client.17304 populate_metadata read hostname 'smithi016' 2024-03-28T22:19:02.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.833+0000 7f26421f5740 1 --2- 172.21.15.16:0/2511000179 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5589ff6cbbe0 0x5589ff6cdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.833+0000 7f26421f5740 4 client.17304 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:19:02.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.833+0000 7f26421f5740 1 -- 172.21.15.16:0/2511000179 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5589ff619a40 con 0x5589ff6cbbe0 2024-03-28T22:19:02.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.834+0000 7f263d3aa640 1 --2- 172.21.15.16:0/2511000179 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5589ff6cbbe0 0x5589ff6cdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.834+0000 7f263d3aa640 1 --2- 172.21.15.16:0/2511000179 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5589ff6cbbe0 0x5589ff6cdfd0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27b734ea3b09b5ad server_cookie=1469b62cfec80699 in_seq=0 out_seq=0 2024-03-28T22:19:02.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.834+0000 7f2620ff9640 10 client.17304 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:19:02.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.835+0000 7f2620ff9640 1 -- 172.21.15.16:0/2511000179 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5589ff619a40 con 0x5589ff6cbbe0 2024-03-28T22:19:02.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.835+0000 7f2620ff9640 10 client.17304 handle_command_reply: tid=0 2024-03-28T22:19:02.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.835+0000 7f26421f5740 1 client.17304 shutdown 2024-03-28T22:19:02.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.835+0000 7f26421f5740 1 -- 172.21.15.16:0/2511000179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ff6aeac0 msgr2=0x5589ff6b9360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.835+0000 7f26421f5740 1 --2- 172.21.15.16:0/2511000179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ff6aeac0 0x5589ff6b9360 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f262809a4e0 tx=0x7f26280cb1a0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.835+0000 7f26421f5740 1 -- 172.21.15.16:0/2511000179 shutdown_connections 2024-03-28T22:19:02.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.835+0000 7f26421f5740 1 --2- 172.21.15.16:0/2511000179 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5589ff6cbbe0 0x5589ff6cdfd0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.835+0000 7f26421f5740 1 --2- 172.21.15.16:0/2511000179 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5589ff6bbd80 0x5589ff6b98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.835+0000 7f26421f5740 1 --2- 172.21.15.16:0/2511000179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ff6aeac0 0x5589ff6b9360 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.835+0000 7f2620ff9640 0 client.17304 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:19:02.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.835+0000 7f26421f5740 1 --2- 172.21.15.16:0/2511000179 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5589ff6ae1a0 0x5589ff6b8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.835+0000 7f26421f5740 1 -- 172.21.15.16:0/2511000179 >> 172.21.15.16:0/2511000179 conn(0x5589ff6a8ba0 msgr2=0x5589ff6aa8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:02.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.835+0000 7f26421f5740 1 -- 172.21.15.16:0/2511000179 shutdown_connections 2024-03-28T22:19:02.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.835+0000 7f26421f5740 1 -- 172.21.15.16:0/2511000179 wait complete. 2024-03-28T22:19:02.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.835+0000 7f26421f5740 20 client.17304 trim_cache size 0 max 16384 2024-03-28T22:19:02.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.849+0000 7f26421f5740 1 Processor -- start 2024-03-28T22:19:02.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.849+0000 7f26421f5740 1 -- start start 2024-03-28T22:19:02.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.849+0000 7f26421f5740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5589ff6ae1a0 0x5589ff6bb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.849+0000 7f26421f5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ff6aeac0 0x5589ff6c7890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.849+0000 7f26421f5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5589ff6bbd80 0x5589ff6d3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.849+0000 7f26421f5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f26381adac0 con 0x5589ff6aeac0 2024-03-28T22:19:02.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.849+0000 7f26421f5740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f26381ab390 con 0x5589ff6ae1a0 2024-03-28T22:19:02.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.849+0000 7f26421f5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f26381a9200 con 0x5589ff6bbd80 2024-03-28T22:19:02.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.849+0000 7f263cba9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ff6aeac0 0x5589ff6c7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.849+0000 7f263cba9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ff6aeac0 0x5589ff6c7890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60090/0 (socket says 172.21.15.16:60090) 2024-03-28T22:19:02.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.849+0000 7f263cba9640 1 -- 172.21.15.16:0/3809357820 learned_addr learned my addr 172.21.15.16:0/3809357820 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:02.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.849+0000 7f263d3aa640 1 --2- 172.21.15.16:0/3809357820 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5589ff6ae1a0 0x5589ff6bb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.849+0000 7f263cba9640 1 -- 172.21.15.16:0/3809357820 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5589ff6bbd80 msgr2=0x5589ff6d3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.849+0000 7f263dbab640 1 --2- 172.21.15.16:0/3809357820 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5589ff6bbd80 0x5589ff6d3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.849+0000 7f263cba9640 1 --2- 172.21.15.16:0/3809357820 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5589ff6bbd80 0x5589ff6d3570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.849+0000 7f263cba9640 1 -- 172.21.15.16:0/3809357820 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5589ff6ae1a0 msgr2=0x5589ff6bb4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.849+0000 7f263cba9640 1 --2- 172.21.15.16:0/3809357820 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5589ff6ae1a0 0x5589ff6bb4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.849+0000 7f263cba9640 1 -- 172.21.15.16:0/3809357820 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589ff6a4460 con 0x5589ff6aeac0 2024-03-28T22:19:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.849+0000 7f263d3aa640 1 --2- 172.21.15.16:0/3809357820 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5589ff6ae1a0 0x5589ff6bb4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:19:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.850+0000 7f263cba9640 1 --2- 172.21.15.16:0/3809357820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ff6aeac0 0x5589ff6c7890 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f262800d740 tx=0x7f2628009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=607f52b48c21f0e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.850+0000 7f2622ffd640 1 -- 172.21.15.16:0/3809357820 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2628020070 con 0x5589ff6aeac0 2024-03-28T22:19:02.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.850+0000 7f2622ffd640 1 -- 172.21.15.16:0/3809357820 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26280988e0 con 0x5589ff6aeac0 2024-03-28T22:19:02.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.850+0000 7f2622ffd640 1 -- 172.21.15.16:0/3809357820 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2628096bc0 con 0x5589ff6aeac0 2024-03-28T22:19:02.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.850+0000 7f26421f5740 1 -- 172.21.15.16:0/3809357820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ff6aeac0 msgr2=0x5589ff6c7890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.850+0000 7f26421f5740 1 --2- 172.21.15.16:0/3809357820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ff6aeac0 0x5589ff6c7890 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f262800d740 tx=0x7f2628009c30 comp rx=0 tx=0).stop 2024-03-28T22:19:02.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.850+0000 7f26421f5740 1 -- 172.21.15.16:0/3809357820 shutdown_connections 2024-03-28T22:19:02.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.850+0000 7f26421f5740 1 --2- 172.21.15.16:0/3809357820 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5589ff6bbd80 0x5589ff6d3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.850+0000 7f26421f5740 1 --2- 172.21.15.16:0/3809357820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ff6aeac0 0x5589ff6c7890 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.850+0000 7f26421f5740 1 --2- 172.21.15.16:0/3809357820 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5589ff6ae1a0 0x5589ff6bb4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.850+0000 7f26421f5740 1 -- 172.21.15.16:0/3809357820 >> 172.21.15.16:0/3809357820 conn(0x5589ff6a8ba0 msgr2=0x5589ff6a3bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:02.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.850+0000 7f26421f5740 1 -- 172.21.15.16:0/3809357820 shutdown_connections 2024-03-28T22:19:02.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.850+0000 7f26421f5740 1 -- 172.21.15.16:0/3809357820 wait complete. 2024-03-28T22:19:02.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.851+0000 7f26421f5740 1 -- start start 2024-03-28T22:19:02.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.851+0000 7f26421f5740 1 Processor -- start 2024-03-28T22:19:02.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.851+0000 7f26421f5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5589ff6ae1a0 0x5589ff6bf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.851+0000 7f26421f5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ff6aeac0 0x5589ff6bfc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.851+0000 7f26421f5740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5589ff6bbd80 0x5589ff6c01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.851+0000 7f26421f5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f26381a9850 con 0x5589ff6aeac0 2024-03-28T22:19:02.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.851+0000 7f26421f5740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f26381aacc0 con 0x5589ff6bbd80 2024-03-28T22:19:02.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.851+0000 7f26421f5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f26381a9f20 con 0x5589ff6ae1a0 2024-03-28T22:19:02.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.851+0000 7f26421f5740 10 client.0 _ll_register_callbacks cb 0x7f2628096130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:19:02.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.851+0000 7f263cba9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ff6aeac0 0x5589ff6bfc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.851+0000 7f263cba9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ff6aeac0 0x5589ff6bfc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60100/0 (socket says 172.21.15.16:60100) 2024-03-28T22:19:02.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.851+0000 7f263cba9640 1 -- 172.21.15.16:0/3490972722 learned_addr learned my addr 172.21.15.16:0/3490972722 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:02.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.851+0000 7f263dbab640 1 --2- 172.21.15.16:0/3490972722 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5589ff6bbd80 0x5589ff6c01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.851+0000 7f263d3aa640 1 --2- 172.21.15.16:0/3490972722 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5589ff6ae1a0 0x5589ff6bf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.851+0000 7f263cba9640 1 -- 172.21.15.16:0/3490972722 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5589ff6ae1a0 msgr2=0x5589ff6bf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:02.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.851+0000 7f263cba9640 1 --2- 172.21.15.16:0/3490972722 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5589ff6ae1a0 0x5589ff6bf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.851+0000 7f263cba9640 1 -- 172.21.15.16:0/3490972722 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5589ff6bbd80 msgr2=0x5589ff6c01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:02.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.851+0000 7f263cba9640 1 --2- 172.21.15.16:0/3490972722 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5589ff6bbd80 0x5589ff6c01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.851+0000 7f263cba9640 1 -- 172.21.15.16:0/3490972722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589ff730ba0 con 0x5589ff6aeac0 2024-03-28T22:19:02.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.852+0000 7f263cba9640 1 --2- 172.21.15.16:0/3490972722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ff6aeac0 0x5589ff6bfc60 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f26280a2710 tx=0x7f2628005bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d95379f1098b0213 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:02.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.852+0000 7f2620ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:19:02.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.852+0000 7f2620ff9640 1 -- 172.21.15.16:0/3490972722 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2628020050 con 0x5589ff6aeac0 2024-03-28T22:19:02.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.852+0000 7f2620ff9640 1 -- 172.21.15.16:0/3490972722 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7f2628099ba0 con 0x5589ff6aeac0 2024-03-28T22:19:02.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.852+0000 7f26421f5740 1 -- 172.21.15.16:0/3490972722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26280988e0 con 0x5589ff6aeac0 2024-03-28T22:19:02.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.852+0000 7f2620ff9640 1 -- 172.21.15.16:0/3490972722 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f26280a2c40 con 0x5589ff6aeac0 2024-03-28T22:19:02.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.852+0000 7f26421f5740 10 client.17310 fetch_fsmap learned FSMap version 27 2024-03-28T22:19:02.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.852+0000 7f26421f5740 1 -- 172.21.15.16:0/3490972722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5589ff730ba0 con 0x5589ff6aeac0 2024-03-28T22:19:02.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.852+0000 7f2620ff9640 1 -- 172.21.15.16:0/3490972722 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26280988e0 con 0x5589ff6aeac0 2024-03-28T22:19:02.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.852+0000 7f2620ff9640 1 -- 172.21.15.16:0/3490972722 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f2628083a90 con 0x5589ff6aeac0 2024-03-28T22:19:02.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.852+0000 7f26421f5740 10 client.17310 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:19:02.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.852+0000 7f26421f5740 10 client.17310 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:19:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.852+0000 7f26421f5740 20 client.17310 populate_metadata read hostname 'smithi016' 2024-03-28T22:19:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.852+0000 7f26421f5740 1 --2- 172.21.15.16:0/3490972722 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5589ff7320e0 0x5589ff7344d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.852+0000 7f26421f5740 4 client.17310 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:19:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.852+0000 7f26421f5740 1 -- 172.21.15.16:0/3490972722 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5589ff5976a0 con 0x5589ff7320e0 2024-03-28T22:19:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.853+0000 7f263d3aa640 1 --2- 172.21.15.16:0/3490972722 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5589ff7320e0 0x5589ff7344d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.853+0000 7f2620ff9640 10 client.17310 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:19:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.853+0000 7f263d3aa640 1 --2- 172.21.15.16:0/3490972722 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5589ff7320e0 0x5589ff7344d0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=465a9a3cb31f9ec server_cookie=6df511e3b1004f4b in_seq=0 out_seq=0 2024-03-28T22:19:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.853+0000 7f2620ff9640 1 -- 172.21.15.16:0/3490972722 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5589ff5976a0 con 0x5589ff7320e0 2024-03-28T22:19:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.853+0000 7f2620ff9640 10 client.17310 handle_command_reply: tid=0 2024-03-28T22:19:02.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.853+0000 7f26421f5740 1 client.17310 shutdown 2024-03-28T22:19:02.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.853+0000 7f26421f5740 1 -- 172.21.15.16:0/3490972722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ff6aeac0 msgr2=0x5589ff6bfc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.853+0000 7f26421f5740 1 --2- 172.21.15.16:0/3490972722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ff6aeac0 0x5589ff6bfc60 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f26280a2710 tx=0x7f2628005bb0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.853+0000 7f26421f5740 1 -- 172.21.15.16:0/3490972722 shutdown_connections 2024-03-28T22:19:02.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.853+0000 7f26421f5740 1 --2- 172.21.15.16:0/3490972722 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5589ff7320e0 0x5589ff7344d0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.853+0000 7f26421f5740 1 --2- 172.21.15.16:0/3490972722 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5589ff6bbd80 0x5589ff6c01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.853+0000 7f2620ff9640 0 client.17310 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:19:02.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.853+0000 7f26421f5740 1 --2- 172.21.15.16:0/3490972722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5589ff6aeac0 0x5589ff6bfc60 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.853+0000 7f26421f5740 1 --2- 172.21.15.16:0/3490972722 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5589ff6ae1a0 0x5589ff6bf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.853+0000 7f26421f5740 1 -- 172.21.15.16:0/3490972722 >> 172.21.15.16:0/3490972722 conn(0x5589ff6a8ba0 msgr2=0x5589ff6b8dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:02.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.853+0000 7f26421f5740 1 -- 172.21.15.16:0/3490972722 shutdown_connections 2024-03-28T22:19:02.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.853+0000 7f26421f5740 1 -- 172.21.15.16:0/3490972722 wait complete. 2024-03-28T22:19:02.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.853+0000 7f26421f5740 20 client.17310 trim_cache size 0 max 16384 2024-03-28T22:19:02.867 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:19:02.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.854+0000 7f263f60e640 1 -- 172.21.15.16:0/2552984136 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f26240a1ef0 msgr2=0x7f26240a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.854+0000 7f263f60e640 1 --2- 172.21.15.16:0/2552984136 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f26240a1ef0 0x7f26240a43b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2630005f00 tx=0x7f263002d930 comp rx=0 tx=0).stop 2024-03-28T22:19:02.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.854+0000 7f263f60e640 1 -- 172.21.15.16:0/2552984136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26381032e0 msgr2=0x7f26381985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:02.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.854+0000 7f263f60e640 1 --2- 172.21.15.16:0/2552984136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26381032e0 0x7f26381985e0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f262800d770 tx=0x7f262800dc30 comp rx=0 tx=0).stop 2024-03-28T22:19:02.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.854+0000 7f263f60e640 1 -- 172.21.15.16:0/2552984136 shutdown_connections 2024-03-28T22:19:02.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.854+0000 7f263f60e640 1 --2- 172.21.15.16:0/2552984136 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f26240a1ef0 0x7f26240a43b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.854+0000 7f263f60e640 1 --2- 172.21.15.16:0/2552984136 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f263810be30 0x7f263819c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.854+0000 7f263f60e640 1 --2- 172.21.15.16:0/2552984136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26381032e0 0x7f26381985e0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.854+0000 7f263f60e640 1 --2- 172.21.15.16:0/2552984136 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f26381029c0 0x7f26381980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:02.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.854+0000 7f263f60e640 1 -- 172.21.15.16:0/2552984136 >> 172.21.15.16:0/2552984136 conn(0x7f26380777c0 msgr2=0x7f26380fe7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:02.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.854+0000 7f263f60e640 1 -- 172.21.15.16:0/2552984136 shutdown_connections 2024-03-28T22:19:02.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:02.854+0000 7f263f60e640 1 -- 172.21.15.16:0/2552984136 wait complete. 2024-03-28T22:19:03.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:03 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/452480470' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:19:04.400 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:19:04.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:19:04.401 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:19:04.402 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:03 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/452480470' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:19:04.402 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:03 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/452480470' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:19:04.430 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:04 smithi016 ceph-mon[32374]: pgmap v261: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:19:04.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:04 smithi063 ceph-mon[36249]: pgmap v261: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:19:04.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.520+0000 7fcae95b5640 1 -- 172.21.15.16:0/2120552304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae4103b90 msgr2=0x7fcae410e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:04.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.520+0000 7fcae95b5640 1 --2- 172.21.15.16:0/2120552304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae4103b90 0x7fcae410e6c0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fcadc009ca0 tx=0x7fcadc02c1d0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.520+0000 7fcae95b5640 1 -- 172.21.15.16:0/2120552304 shutdown_connections 2024-03-28T22:19:04.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.520+0000 7fcae95b5640 1 --2- 172.21.15.16:0/2120552304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae4103b90 0x7fcae410e6c0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.520+0000 7fcae95b5640 1 --2- 172.21.15.16:0/2120552304 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcae41031f0 0x7fcae4103650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.520+0000 7fcae95b5640 1 --2- 172.21.15.16:0/2120552304 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcae4101ab0 0x7fcae4101e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.520+0000 7fcae95b5640 1 -- 172.21.15.16:0/2120552304 >> 172.21.15.16:0/2120552304 conn(0x7fcae40fd1e0 msgr2=0x7fcae40ff600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:04.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.520+0000 7fcae95b5640 1 -- 172.21.15.16:0/2120552304 shutdown_connections 2024-03-28T22:19:04.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.520+0000 7fcae95b5640 1 -- 172.21.15.16:0/2120552304 wait complete. 2024-03-28T22:19:04.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.521+0000 7fcae95b5640 1 Processor -- start 2024-03-28T22:19:04.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.521+0000 7fcae95b5640 1 -- start start 2024-03-28T22:19:04.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.521+0000 7fcae95b5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae4101ab0 0x7fcae4198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:04.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.521+0000 7fcae95b5640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcae41031f0 0x7fcae4198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:04.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.521+0000 7fcae95b5640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcae4103b90 0x7fcae419c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:04.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.521+0000 7fcae95b5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcae4111c00 con 0x7fcae4101ab0 2024-03-28T22:19:04.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.521+0000 7fcae95b5640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcae4111a80 con 0x7fcae4103b90 2024-03-28T22:19:04.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.521+0000 7fcae95b5640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcae4111d80 con 0x7fcae41031f0 2024-03-28T22:19:04.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.521+0000 7fcae2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae4101ab0 0x7fcae4198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:04.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.521+0000 7fcae2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae4101ab0 0x7fcae4198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60106/0 (socket says 172.21.15.16:60106) 2024-03-28T22:19:04.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.521+0000 7fcae2ffd640 1 -- 172.21.15.16:0/3400930133 learned_addr learned my addr 172.21.15.16:0/3400930133 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:04.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.521+0000 7fcae37fe640 1 --2- 172.21.15.16:0/3400930133 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcae4103b90 0x7fcae419c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:04.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.522+0000 7fcae27fc640 1 --2- 172.21.15.16:0/3400930133 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcae41031f0 0x7fcae4198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:04.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.522+0000 7fcae2ffd640 1 -- 172.21.15.16:0/3400930133 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcae41031f0 msgr2=0x7fcae4198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:04.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.522+0000 7fcae2ffd640 1 --2- 172.21.15.16:0/3400930133 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcae41031f0 0x7fcae4198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.522+0000 7fcae2ffd640 1 -- 172.21.15.16:0/3400930133 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcae4103b90 msgr2=0x7fcae419c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:04.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.522+0000 7fcae2ffd640 1 --2- 172.21.15.16:0/3400930133 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcae4103b90 0x7fcae419c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.522+0000 7fcae2ffd640 1 -- 172.21.15.16:0/3400930133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcae419d010 con 0x7fcae4101ab0 2024-03-28T22:19:04.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.522+0000 7fcae2ffd640 1 --2- 172.21.15.16:0/3400930133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae4101ab0 0x7fcae4198020 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fcad400d950 tx=0x7fcad400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:04.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.522+0000 7fcacbfff640 1 -- 172.21.15.16:0/3400930133 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad4017070 con 0x7fcae4101ab0 2024-03-28T22:19:04.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.522+0000 7fcae95b5640 1 -- 172.21.15.16:0/3400930133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcae419d300 con 0x7fcae4101ab0 2024-03-28T22:19:04.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.522+0000 7fcacbfff640 1 -- 172.21.15.16:0/3400930133 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcad4009c30 con 0x7fcae4101ab0 2024-03-28T22:19:04.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.522+0000 7fcacbfff640 1 -- 172.21.15.16:0/3400930133 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad400f6b0 con 0x7fcae4101ab0 2024-03-28T22:19:04.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.522+0000 7fcae95b5640 1 -- 172.21.15.16:0/3400930133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcae41a4b90 con 0x7fcae4101ab0 2024-03-28T22:19:04.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.524+0000 7fcacbfff640 1 -- 172.21.15.16:0/3400930133 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7fcad40109d0 con 0x7fcae4101ab0 2024-03-28T22:19:04.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.525+0000 7fcaec196740 1 Processor -- start 2024-03-28T22:19:04.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.525+0000 7fcaec196740 1 -- start start 2024-03-28T22:19:04.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.525+0000 7fcacbfff640 1 --2- 172.21.15.16:0/3400930133 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fcacc0a1ef0 0x7fcacc0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:04.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.525+0000 7fcaec196740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c45d2879f0 0x55c45d287dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:04.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.525+0000 7fcaec196740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c45d288310 0x55c45d2909b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:04.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.525+0000 7fcaec196740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c45d290ef0 0x55c45d2932e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:04.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.525+0000 7fcae37fe640 1 --2- 172.21.15.16:0/3400930133 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fcacc0a1ef0 0x7fcacc0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:04.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.525+0000 7fcaec196740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c45d1880b0 con 0x55c45d290ef0 2024-03-28T22:19:04.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.525+0000 7fcacbfff640 1 -- 172.21.15.16:0/3400930133 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7fcad40cdde0 con 0x7fcae4101ab0 2024-03-28T22:19:04.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.525+0000 7fcaec196740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55c45d1eaea0 con 0x55c45d288310 2024-03-28T22:19:04.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.525+0000 7fcaec196740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55c45d193c00 con 0x55c45d2879f0 2024-03-28T22:19:04.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.525+0000 7fcae37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c45d290ef0 0x55c45d2932e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:04.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.525+0000 7fcae27fc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c45d288310 0x55c45d2909b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:04.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.525+0000 7fcae37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c45d290ef0 0x55c45d2932e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60110/0 (socket says 172.21.15.16:60110) 2024-03-28T22:19:04.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.525+0000 7fcae37fe640 1 -- 172.21.15.16:0/2786047392 learned_addr learned my addr 172.21.15.16:0/2786047392 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:04.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.525+0000 7fcae27fc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c45d288310 0x55c45d2909b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:39036/0 (socket says 172.21.15.16:39036) 2024-03-28T22:19:04.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.525+0000 7fcae2ffd640 1 --2- 172.21.15.16:0/2786047392 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c45d2879f0 0x55c45d287dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:04.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.525+0000 7fcae27fc640 1 -- 172.21.15.16:0/2786047392 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c45d2879f0 msgr2=0x55c45d287dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:04.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.525+0000 7fcae27fc640 1 --2- 172.21.15.16:0/2786047392 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c45d2879f0 0x55c45d287dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.525+0000 7fcae27fc640 1 -- 172.21.15.16:0/2786047392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c45d290ef0 msgr2=0x55c45d2932e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:04.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.525+0000 7fcae27fc640 1 --2- 172.21.15.16:0/2786047392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c45d290ef0 0x55c45d2932e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.525+0000 7fcae27fc640 1 -- 172.21.15.16:0/2786047392 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c45cfc77e0 con 0x55c45d288310 2024-03-28T22:19:04.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.525+0000 7fcae37fe640 1 --2- 172.21.15.16:0/2786047392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c45d290ef0 0x55c45d2932e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:19:04.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.526+0000 7fcae37fe640 1 --2- 172.21.15.16:0/3400930133 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fcacc0a1ef0 0x7fcacc0a43b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fcadc006010 tx=0x7fcadc0323c0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:04.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.526+0000 7fcae27fc640 1 --2- 172.21.15.16:0/2786047392 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c45d288310 0x55c45d2909b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fcad8009b40 tx=0x7fcad80113e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71284bec5481b42e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:04.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.526+0000 7fcac97fa640 1 -- 172.21.15.16:0/2786047392 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad8011e10 con 0x55c45d288310 2024-03-28T22:19:04.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.526+0000 7fcac97fa640 1 -- 172.21.15.16:0/2786047392 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcad8012930 con 0x55c45d288310 2024-03-28T22:19:04.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.526+0000 7fcac97fa640 1 -- 172.21.15.16:0/2786047392 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad800fa00 con 0x55c45d288310 2024-03-28T22:19:04.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.527+0000 7fcaec196740 1 -- 172.21.15.16:0/2786047392 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c45d288310 msgr2=0x55c45d2909b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:04.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.527+0000 7fcaec196740 1 --2- 172.21.15.16:0/2786047392 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c45d288310 0x55c45d2909b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fcad8009b40 tx=0x7fcad80113e0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.527+0000 7fcaec196740 1 -- 172.21.15.16:0/2786047392 shutdown_connections 2024-03-28T22:19:04.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.527+0000 7fcaec196740 1 --2- 172.21.15.16:0/2786047392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c45d290ef0 0x55c45d2932e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.527+0000 7fcaec196740 1 --2- 172.21.15.16:0/2786047392 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c45d288310 0x55c45d2909b0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.527+0000 7fcaec196740 1 --2- 172.21.15.16:0/2786047392 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c45d2879f0 0x55c45d287dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.527+0000 7fcaec196740 1 -- 172.21.15.16:0/2786047392 >> 172.21.15.16:0/2786047392 conn(0x55c45d282120 msgr2=0x55c45d282500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:04.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.527+0000 7fcaec196740 1 -- 172.21.15.16:0/2786047392 shutdown_connections 2024-03-28T22:19:04.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.527+0000 7fcaec196740 1 -- 172.21.15.16:0/2786047392 wait complete. 2024-03-28T22:19:04.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.527+0000 7fcaec196740 1 -- start start 2024-03-28T22:19:04.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.527+0000 7fcaec196740 1 Processor -- start 2024-03-28T22:19:04.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.527+0000 7fcaec196740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c45d2879f0 0x55c45d28de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:04.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.527+0000 7fcaec196740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c45d288310 0x55c45d28e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:04.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.527+0000 7fcaec196740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c45d290ef0 0x55c45d28e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:04.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.527+0000 7fcaec196740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c45d1893d0 con 0x55c45d288310 2024-03-28T22:19:04.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.527+0000 7fcaec196740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55c45d1e6f40 con 0x55c45d290ef0 2024-03-28T22:19:04.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.527+0000 7fcaec196740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55c45d293ce0 con 0x55c45d2879f0 2024-03-28T22:19:04.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.527+0000 7fcaec196740 10 client.0 _ll_register_callbacks cb 0x55c45d1a6b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:19:04.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.527+0000 7fcae27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c45d288310 0x55c45d28e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:04.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.527+0000 7fcae27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c45d288310 0x55c45d28e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60120/0 (socket says 172.21.15.16:60120) 2024-03-28T22:19:04.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.527+0000 7fcae27fc640 1 -- 172.21.15.16:0/1512510378 learned_addr learned my addr 172.21.15.16:0/1512510378 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:04.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.527+0000 7fcae37fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c45d290ef0 0x55c45d28e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:04.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.528+0000 7fcae2ffd640 1 --2- 172.21.15.16:0/1512510378 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c45d2879f0 0x55c45d28de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:04.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.528+0000 7fcae27fc640 1 -- 172.21.15.16:0/1512510378 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c45d2879f0 msgr2=0x55c45d28de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:04.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.528+0000 7fcae27fc640 1 --2- 172.21.15.16:0/1512510378 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c45d2879f0 0x55c45d28de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.528+0000 7fcae27fc640 1 -- 172.21.15.16:0/1512510378 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c45d290ef0 msgr2=0x55c45d28e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:04.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.528+0000 7fcae27fc640 1 --2- 172.21.15.16:0/1512510378 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c45d290ef0 0x55c45d28e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.528+0000 7fcae27fc640 1 -- 172.21.15.16:0/1512510378 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c45d027f70 con 0x55c45d288310 2024-03-28T22:19:04.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.528+0000 7fcae37fe640 1 --2- 172.21.15.16:0/1512510378 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c45d290ef0 0x55c45d28e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:19:04.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.528+0000 7fcae27fc640 1 --2- 172.21.15.16:0/1512510378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c45d288310 0x55c45d28e3d0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fcad8009740 tx=0x7fcad8012700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c85b89fec21ffa1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:04.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.528+0000 7fcac6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:19:04.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.528+0000 7fcac6ff5640 1 -- 172.21.15.16:0/1512510378 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad8028070 con 0x55c45d288310 2024-03-28T22:19:04.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.528+0000 7fcac6ff5640 1 -- 172.21.15.16:0/1512510378 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7fcad8005040 con 0x55c45d288310 2024-03-28T22:19:04.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.528+0000 7fcaec196740 1 -- 172.21.15.16:0/1512510378 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcad8012930 con 0x55c45d288310 2024-03-28T22:19:04.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.528+0000 7fcac6ff5640 1 -- 172.21.15.16:0/1512510378 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcad8005240 con 0x55c45d288310 2024-03-28T22:19:04.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.528+0000 7fcaec196740 10 client.17322 fetch_fsmap learned FSMap version 27 2024-03-28T22:19:04.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.528+0000 7fcaec196740 1 -- 172.21.15.16:0/1512510378 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c45cebf170 con 0x55c45d288310 2024-03-28T22:19:04.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.528+0000 7fcac6ff5640 1 -- 172.21.15.16:0/1512510378 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcad8012930 con 0x55c45d288310 2024-03-28T22:19:04.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.529+0000 7fcac6ff5640 1 -- 172.21.15.16:0/1512510378 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7fcad8004920 con 0x55c45d288310 2024-03-28T22:19:04.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.529+0000 7fcaec196740 10 client.17322 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:19:04.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.529+0000 7fcaec196740 10 client.17322 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:19:04.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.529+0000 7fcaec196740 20 client.17322 populate_metadata read hostname 'smithi016' 2024-03-28T22:19:04.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.529+0000 7fcaec196740 1 --2- 172.21.15.16:0/1512510378 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55c45d2a0ca0 0x55c45d2a3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:04.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.529+0000 7fcaec196740 4 client.17322 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:19:04.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.529+0000 7fcaec196740 1 -- 172.21.15.16:0/1512510378 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c45d1f83e0 con 0x55c45d2a0ca0 2024-03-28T22:19:04.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.529+0000 7fcae2ffd640 1 --2- 172.21.15.16:0/1512510378 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55c45d2a0ca0 0x55c45d2a3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:04.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.529+0000 7fcae2ffd640 1 --2- 172.21.15.16:0/1512510378 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55c45d2a0ca0 0x55c45d2a3090 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ae37a058d336981 server_cookie=fb424a370d276cc3 in_seq=0 out_seq=0 2024-03-28T22:19:04.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.529+0000 7fcac6ff5640 10 client.17322 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:19:04.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.530+0000 7fcac6ff5640 1 -- 172.21.15.16:0/1512510378 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c45d1f83e0 con 0x55c45d2a0ca0 2024-03-28T22:19:04.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.530+0000 7fcac6ff5640 10 client.17322 handle_command_reply: tid=0 2024-03-28T22:19:04.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.530+0000 7fcaec196740 1 client.17322 shutdown 2024-03-28T22:19:04.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.531+0000 7fcaec196740 1 -- 172.21.15.16:0/1512510378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c45d288310 msgr2=0x55c45d28e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:04.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.531+0000 7fcaec196740 1 --2- 172.21.15.16:0/1512510378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c45d288310 0x55c45d28e3d0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fcad8009740 tx=0x7fcad8012700 comp rx=0 tx=0).stop 2024-03-28T22:19:04.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.531+0000 7fcaec196740 1 -- 172.21.15.16:0/1512510378 shutdown_connections 2024-03-28T22:19:04.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.531+0000 7fcaec196740 1 --2- 172.21.15.16:0/1512510378 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55c45d2a0ca0 0x55c45d2a3090 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.531+0000 7fcaec196740 1 --2- 172.21.15.16:0/1512510378 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c45d290ef0 0x55c45d28e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.531+0000 7fcaec196740 1 --2- 172.21.15.16:0/1512510378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c45d288310 0x55c45d28e3d0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.531+0000 7fcaec196740 1 --2- 172.21.15.16:0/1512510378 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c45d2879f0 0x55c45d28de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.531+0000 7fcaec196740 1 -- 172.21.15.16:0/1512510378 >> 172.21.15.16:0/1512510378 conn(0x55c45d282120 msgr2=0x55c45d283e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:04.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.531+0000 7fcac6ff5640 0 client.17322 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:19:04.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.531+0000 7fcaec196740 1 -- 172.21.15.16:0/1512510378 shutdown_connections 2024-03-28T22:19:04.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.531+0000 7fcaec196740 1 -- 172.21.15.16:0/1512510378 wait complete. 2024-03-28T22:19:04.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.531+0000 7fcaec196740 20 client.17322 trim_cache size 0 max 16384 2024-03-28T22:19:04.573 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:04 smithi053 ceph-mon[35979]: pgmap v261: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:19:04.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.548+0000 7fcaec196740 1 Processor -- start 2024-03-28T22:19:04.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.548+0000 7fcaec196740 1 -- start start 2024-03-28T22:19:04.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.549+0000 7fcaec196740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c45d2879f0 0x55c45d2905c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:04.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.549+0000 7fcaec196740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c45d288310 0x55c45d29c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:04.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.549+0000 7fcaec196740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c45d290ef0 0x55c45d2a8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:04.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.549+0000 7fcaec196740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcae41afd50 con 0x55c45d290ef0 2024-03-28T22:19:04.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.549+0000 7fcaec196740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcae41ad620 con 0x55c45d288310 2024-03-28T22:19:04.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.549+0000 7fcaec196740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcae41ab490 con 0x55c45d2879f0 2024-03-28T22:19:04.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.549+0000 7fcae37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c45d290ef0 0x55c45d2a8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:04.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.549+0000 7fcae37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c45d290ef0 0x55c45d2a8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60136/0 (socket says 172.21.15.16:60136) 2024-03-28T22:19:04.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.549+0000 7fcae27fc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c45d288310 0x55c45d29c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:04.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.549+0000 7fcae37fe640 1 -- 172.21.15.16:0/961412086 learned_addr learned my addr 172.21.15.16:0/961412086 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:04.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.549+0000 7fcae2ffd640 1 --2- 172.21.15.16:0/961412086 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c45d2879f0 0x55c45d2905c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:04.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.549+0000 7fcae37fe640 1 -- 172.21.15.16:0/961412086 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c45d2879f0 msgr2=0x55c45d2905c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:04.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.549+0000 7fcae37fe640 1 --2- 172.21.15.16:0/961412086 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c45d2879f0 0x55c45d2905c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.549+0000 7fcae37fe640 1 -- 172.21.15.16:0/961412086 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c45d288310 msgr2=0x55c45d29c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:04.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.549+0000 7fcae37fe640 1 --2- 172.21.15.16:0/961412086 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c45d288310 0x55c45d29c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.549+0000 7fcae37fe640 1 -- 172.21.15.16:0/961412086 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c45d28c420 con 0x55c45d290ef0 2024-03-28T22:19:04.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.549+0000 7fcae27fc640 1 --2- 172.21.15.16:0/961412086 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c45d288310 0x55c45d29c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:19:04.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.549+0000 7fcae37fe640 1 --2- 172.21.15.16:0/961412086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c45d290ef0 0x55c45d2a8800 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fcadc037040 tx=0x7fcadc047000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e7aa9aed44a5cf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:04.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.549+0000 7fcac8ff9640 1 -- 172.21.15.16:0/961412086 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcadc058070 con 0x55c45d290ef0 2024-03-28T22:19:04.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.549+0000 7fcac8ff9640 1 -- 172.21.15.16:0/961412086 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcadc03a040 con 0x55c45d290ef0 2024-03-28T22:19:04.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.549+0000 7fcac8ff9640 1 -- 172.21.15.16:0/961412086 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcadc034a50 con 0x55c45d290ef0 2024-03-28T22:19:04.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.550+0000 7fcaec196740 1 -- 172.21.15.16:0/961412086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c45d290ef0 msgr2=0x55c45d2a8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:04.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.550+0000 7fcaec196740 1 --2- 172.21.15.16:0/961412086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c45d290ef0 0x55c45d2a8800 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fcadc037040 tx=0x7fcadc047000 comp rx=0 tx=0).stop 2024-03-28T22:19:04.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.550+0000 7fcaec196740 1 -- 172.21.15.16:0/961412086 shutdown_connections 2024-03-28T22:19:04.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.550+0000 7fcaec196740 1 --2- 172.21.15.16:0/961412086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c45d290ef0 0x55c45d2a8800 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.550+0000 7fcaec196740 1 --2- 172.21.15.16:0/961412086 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c45d288310 0x55c45d29c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.550+0000 7fcaec196740 1 --2- 172.21.15.16:0/961412086 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c45d2879f0 0x55c45d2905c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.550+0000 7fcaec196740 1 -- 172.21.15.16:0/961412086 >> 172.21.15.16:0/961412086 conn(0x55c45d282120 msgr2=0x55c45d2808f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:04.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.550+0000 7fcaec196740 1 -- 172.21.15.16:0/961412086 shutdown_connections 2024-03-28T22:19:04.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.550+0000 7fcaec196740 1 -- 172.21.15.16:0/961412086 wait complete. 2024-03-28T22:19:04.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.550+0000 7fcaec196740 1 -- start start 2024-03-28T22:19:04.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.550+0000 7fcaec196740 1 Processor -- start 2024-03-28T22:19:04.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.551+0000 7fcaec196740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c45d2879f0 0x55c45d298fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:04.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.551+0000 7fcaec196740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c45d288310 0x55c45d295f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:04.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.551+0000 7fcaec196740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c45d290ef0 0x55c45d2964d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:04.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.551+0000 7fcaec196740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcae41abae0 con 0x55c45d2879f0 2024-03-28T22:19:04.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.551+0000 7fcaec196740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcae41acf50 con 0x55c45d288310 2024-03-28T22:19:04.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.551+0000 7fcaec196740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcae41ac1b0 con 0x55c45d290ef0 2024-03-28T22:19:04.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.551+0000 7fcaec196740 10 client.0 _ll_register_callbacks cb 0x7fcad80099c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:19:04.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.551+0000 7fcae2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c45d2879f0 0x55c45d298fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:04.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.551+0000 7fcae2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c45d2879f0 0x55c45d298fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60142/0 (socket says 172.21.15.16:60142) 2024-03-28T22:19:04.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.551+0000 7fcae2ffd640 1 -- 172.21.15.16:0/2446826489 learned_addr learned my addr 172.21.15.16:0/2446826489 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:04.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.551+0000 7fcae27fc640 1 --2- 172.21.15.16:0/2446826489 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c45d288310 0x55c45d295f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:04.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.551+0000 7fcae2ffd640 1 -- 172.21.15.16:0/2446826489 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c45d290ef0 msgr2=0x55c45d2964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:04.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.551+0000 7fcae2ffd640 1 --2- 172.21.15.16:0/2446826489 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c45d290ef0 0x55c45d2964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.551+0000 7fcae2ffd640 1 -- 172.21.15.16:0/2446826489 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c45d288310 msgr2=0x55c45d295f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:04.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.551+0000 7fcae2ffd640 1 --2- 172.21.15.16:0/2446826489 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c45d288310 0x55c45d295f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.551+0000 7fcae2ffd640 1 -- 172.21.15.16:0/2446826489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c45d299730 con 0x55c45d2879f0 2024-03-28T22:19:04.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.551+0000 7fcae2ffd640 1 --2- 172.21.15.16:0/2446826489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c45d2879f0 0x55c45d298fc0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fcad409a760 tx=0x7fcad40ba040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6347e3287c31a2f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:04.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.551+0000 7fcac6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:19:04.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.551+0000 7fcac6ff5640 1 -- 172.21.15.16:0/2446826489 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad4020040 con 0x55c45d2879f0 2024-03-28T22:19:04.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.551+0000 7fcac6ff5640 1 -- 172.21.15.16:0/2446826489 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7fcad40cb7b0 con 0x55c45d2879f0 2024-03-28T22:19:04.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.551+0000 7fcaec196740 1 -- 172.21.15.16:0/2446826489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcadc03a040 con 0x55c45d2879f0 2024-03-28T22:19:04.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.552+0000 7fcac6ff5640 1 -- 172.21.15.16:0/2446826489 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcad409a7d0 con 0x55c45d2879f0 2024-03-28T22:19:04.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.552+0000 7fcaec196740 10 client.17328 fetch_fsmap learned FSMap version 27 2024-03-28T22:19:04.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.552+0000 7fcaec196740 1 -- 172.21.15.16:0/2446826489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c45d30bdd0 con 0x55c45d2879f0 2024-03-28T22:19:04.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.552+0000 7fcac6ff5640 1 -- 172.21.15.16:0/2446826489 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcadc03a040 con 0x55c45d2879f0 2024-03-28T22:19:04.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.552+0000 7fcac6ff5640 1 -- 172.21.15.16:0/2446826489 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7fcad40b3160 con 0x55c45d2879f0 2024-03-28T22:19:04.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.552+0000 7fcaec196740 10 client.17328 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:19:04.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.552+0000 7fcaec196740 10 client.17328 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:19:04.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.552+0000 7fcaec196740 20 client.17328 populate_metadata read hostname 'smithi016' 2024-03-28T22:19:04.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.552+0000 7fcaec196740 1 --2- 172.21.15.16:0/2446826489 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55c45d30c4a0 0x55c45d30e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:04.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.552+0000 7fcaec196740 4 client.17328 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:19:04.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.552+0000 7fcaec196740 1 -- 172.21.15.16:0/2446826489 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c45d1f2730 con 0x55c45d30c4a0 2024-03-28T22:19:04.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.552+0000 7fcae27fc640 1 --2- 172.21.15.16:0/2446826489 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55c45d30c4a0 0x55c45d30e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:04.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.552+0000 7fcac6ff5640 10 client.17328 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:19:04.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.552+0000 7fcae27fc640 1 --2- 172.21.15.16:0/2446826489 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55c45d30c4a0 0x55c45d30e8c0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b11176391eb17dd server_cookie=42b57086210863d in_seq=0 out_seq=0 2024-03-28T22:19:04.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.553+0000 7fcac6ff5640 1 -- 172.21.15.16:0/2446826489 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c45d1f2730 con 0x55c45d30c4a0 2024-03-28T22:19:04.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.553+0000 7fcac6ff5640 10 client.17328 handle_command_reply: tid=0 2024-03-28T22:19:04.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.553+0000 7fcaec196740 1 client.17328 shutdown 2024-03-28T22:19:04.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.554+0000 7fcaec196740 1 -- 172.21.15.16:0/2446826489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c45d2879f0 msgr2=0x55c45d298fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:04.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.554+0000 7fcaec196740 1 --2- 172.21.15.16:0/2446826489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c45d2879f0 0x55c45d298fc0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fcad409a760 tx=0x7fcad40ba040 comp rx=0 tx=0).stop 2024-03-28T22:19:04.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.554+0000 7fcaec196740 1 -- 172.21.15.16:0/2446826489 shutdown_connections 2024-03-28T22:19:04.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.554+0000 7fcaec196740 1 --2- 172.21.15.16:0/2446826489 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55c45d30c4a0 0x55c45d30e8c0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.554+0000 7fcaec196740 1 --2- 172.21.15.16:0/2446826489 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c45d290ef0 0x55c45d2964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.554+0000 7fcaec196740 1 --2- 172.21.15.16:0/2446826489 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c45d288310 0x55c45d295f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.554+0000 7fcaec196740 1 --2- 172.21.15.16:0/2446826489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c45d2879f0 0x55c45d298fc0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.554+0000 7fcac6ff5640 0 client.17328 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:19:04.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.554+0000 7fcaec196740 1 -- 172.21.15.16:0/2446826489 >> 172.21.15.16:0/2446826489 conn(0x55c45d282120 msgr2=0x55c45d28e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:04.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.554+0000 7fcaec196740 1 -- 172.21.15.16:0/2446826489 shutdown_connections 2024-03-28T22:19:04.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.554+0000 7fcaec196740 1 -- 172.21.15.16:0/2446826489 wait complete. 2024-03-28T22:19:04.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.554+0000 7fcaec196740 20 client.17328 trim_cache size 0 max 16384 2024-03-28T22:19:04.599 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:19:04.600 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-03-28T22:19:04.600 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "e6778206-7e54-4256-a562-09e0f35e48f4", 2024-03-28T22:19:04.600 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-03-28T22:19:04.601 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:19:04.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.554+0000 7fcae95b5640 1 -- 172.21.15.16:0/3400930133 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fcacc0a1ef0 msgr2=0x7fcacc0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:04.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.554+0000 7fcae95b5640 1 --2- 172.21.15.16:0/3400930133 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fcacc0a1ef0 0x7fcacc0a43b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fcadc006010 tx=0x7fcadc0323c0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.554+0000 7fcae95b5640 1 -- 172.21.15.16:0/3400930133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae4101ab0 msgr2=0x7fcae4198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:04.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.554+0000 7fcae95b5640 1 --2- 172.21.15.16:0/3400930133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae4101ab0 0x7fcae4198020 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fcad400d950 tx=0x7fcad400de10 comp rx=0 tx=0).stop 2024-03-28T22:19:04.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.554+0000 7fcae95b5640 1 -- 172.21.15.16:0/3400930133 shutdown_connections 2024-03-28T22:19:04.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.554+0000 7fcae95b5640 1 --2- 172.21.15.16:0/3400930133 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fcacc0a1ef0 0x7fcacc0a43b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.554+0000 7fcae95b5640 1 --2- 172.21.15.16:0/3400930133 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcae4103b90 0x7fcae419c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.554+0000 7fcae95b5640 1 --2- 172.21.15.16:0/3400930133 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcae41031f0 0x7fcae4198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.554+0000 7fcae95b5640 1 --2- 172.21.15.16:0/3400930133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae4101ab0 0x7fcae4198020 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.554+0000 7fcae95b5640 1 -- 172.21.15.16:0/3400930133 >> 172.21.15.16:0/3400930133 conn(0x7fcae40fd1e0 msgr2=0x7fcae4107600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:04.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.555+0000 7fcae95b5640 1 -- 172.21.15.16:0/3400930133 shutdown_connections 2024-03-28T22:19:04.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.555+0000 7fcae95b5640 1 -- 172.21.15.16:0/3400930133 wait complete. 2024-03-28T22:19:04.605 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:19:04.606 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:19:04.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.754+0000 7faa5182a640 1 -- 172.21.15.16:0/1331793255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa4c1004f0 msgr2=0x7faa4c1008d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:04.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.755+0000 7faa5182a640 1 --2- 172.21.15.16:0/1331793255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa4c1004f0 0x7faa4c1008d0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7faa3c009900 tx=0x7faa3c02bcd0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.755+0000 7faa5182a640 1 -- 172.21.15.16:0/1331793255 shutdown_connections 2024-03-28T22:19:04.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.755+0000 7faa5182a640 1 --2- 172.21.15.16:0/1331793255 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa4c10e0a0 0x7faa4c110490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.755+0000 7faa5182a640 1 --2- 172.21.15.16:0/1331793255 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa4c100e10 0x7faa4c10db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.755+0000 7faa5182a640 1 --2- 172.21.15.16:0/1331793255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa4c1004f0 0x7faa4c1008d0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.755+0000 7faa5182a640 1 -- 172.21.15.16:0/1331793255 >> 172.21.15.16:0/1331793255 conn(0x7faa4c077800 msgr2=0x7faa4c077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:04.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.755+0000 7faa5182a640 1 -- 172.21.15.16:0/1331793255 shutdown_connections 2024-03-28T22:19:04.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.755+0000 7faa5182a640 1 -- 172.21.15.16:0/1331793255 wait complete. 2024-03-28T22:19:04.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.755+0000 7faa5182a640 1 Processor -- start 2024-03-28T22:19:04.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.755+0000 7faa5182a640 1 -- start start 2024-03-28T22:19:04.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.756+0000 7faa5182a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa4c1004f0 0x7faa4c19a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:04.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.756+0000 7faa5182a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa4c100e10 0x7faa4c19a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:04.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.756+0000 7faa5182a640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa4c10e0a0 0x7faa4c19eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:04.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.756+0000 7faa5182a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c113d20 con 0x7faa4c100e10 2024-03-28T22:19:04.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.756+0000 7faa5182a640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c113ba0 con 0x7faa4c10e0a0 2024-03-28T22:19:04.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.756+0000 7faa5182a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c113ea0 con 0x7faa4c1004f0 2024-03-28T22:19:04.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.756+0000 7faa4a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa4c100e10 0x7faa4c19a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:04.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.756+0000 7faa4a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa4c100e10 0x7faa4c19a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60160/0 (socket says 172.21.15.16:60160) 2024-03-28T22:19:04.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.756+0000 7faa4a7fc640 1 -- 172.21.15.16:0/2116117135 learned_addr learned my addr 172.21.15.16:0/2116117135 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:04.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.756+0000 7faa4affd640 1 --2- 172.21.15.16:0/2116117135 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa4c1004f0 0x7faa4c19a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:04.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.756+0000 7faa4b7fe640 1 --2- 172.21.15.16:0/2116117135 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa4c10e0a0 0x7faa4c19eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:04.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.756+0000 7faa4a7fc640 1 -- 172.21.15.16:0/2116117135 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa4c1004f0 msgr2=0x7faa4c19a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:04.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.756+0000 7faa4a7fc640 1 --2- 172.21.15.16:0/2116117135 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa4c1004f0 0x7faa4c19a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.756+0000 7faa4a7fc640 1 -- 172.21.15.16:0/2116117135 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa4c10e0a0 msgr2=0x7faa4c19eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:04.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.756+0000 7faa4a7fc640 1 --2- 172.21.15.16:0/2116117135 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa4c10e0a0 0x7faa4c19eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.756+0000 7faa4a7fc640 1 -- 172.21.15.16:0/2116117135 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa4c19f210 con 0x7faa4c100e10 2024-03-28T22:19:04.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.756+0000 7faa4a7fc640 1 --2- 172.21.15.16:0/2116117135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa4c100e10 0x7faa4c19a760 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7faa4000e9f0 tx=0x7faa4000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:04.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.756+0000 7faa50828640 1 -- 172.21.15.16:0/2116117135 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa4000cd80 con 0x7faa4c100e10 2024-03-28T22:19:04.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.757+0000 7faa5182a640 1 -- 172.21.15.16:0/2116117135 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa4c19f500 con 0x7faa4c100e10 2024-03-28T22:19:04.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.757+0000 7faa50828640 1 -- 172.21.15.16:0/2116117135 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa40010910 con 0x7faa4c100e10 2024-03-28T22:19:04.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.757+0000 7faa50828640 1 -- 172.21.15.16:0/2116117135 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa400188f0 con 0x7faa4c100e10 2024-03-28T22:19:04.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.757+0000 7faa5182a640 1 -- 172.21.15.16:0/2116117135 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa4c1a6d90 con 0x7faa4c100e10 2024-03-28T22:19:04.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.759+0000 7faa50828640 1 -- 172.21.15.16:0/2116117135 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7faa40010ac0 con 0x7faa4c100e10 2024-03-28T22:19:04.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.759+0000 7faa5440f740 1 Processor -- start 2024-03-28T22:19:04.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.759+0000 7faa5440f740 1 -- start start 2024-03-28T22:19:04.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.760+0000 7faa5440f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d24e426290 0x55d24e426670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:04.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.760+0000 7faa5440f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d24e426bb0 0x55d24e42f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:04.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.760+0000 7faa5440f740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d24e42f790 0x55d24e431b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:04.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.760+0000 7faa5440f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d24e38a180 con 0x55d24e426290 2024-03-28T22:19:04.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.760+0000 7faa5440f740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55d24e332b70 con 0x55d24e42f790 2024-03-28T22:19:04.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.760+0000 7faa5440f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d24e386220 con 0x55d24e426bb0 2024-03-28T22:19:04.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.760+0000 7faa4b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d24e426290 0x55d24e426670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:04.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.760+0000 7faa50828640 1 --2- 172.21.15.16:0/2116117135 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7faa340a1ef0 0x7faa340a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:04.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.760+0000 7faa4b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d24e426290 0x55d24e426670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60174/0 (socket says 172.21.15.16:60174) 2024-03-28T22:19:04.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.760+0000 7faa4b7fe640 1 -- 172.21.15.16:0/1677422298 learned_addr learned my addr 172.21.15.16:0/1677422298 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:04.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.760+0000 7faa4affd640 1 --2- 172.21.15.16:0/1677422298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d24e426bb0 0x55d24e42f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:04.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.760+0000 7faa4a7fc640 1 --2- 172.21.15.16:0/1677422298 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d24e42f790 0x55d24e431b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:04.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.760+0000 7faa4b7fe640 1 --2- 172.21.15.16:0/2116117135 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7faa340a1ef0 0x7faa340a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:04.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.760+0000 7faa50828640 1 -- 172.21.15.16:0/2116117135 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7faa400ce510 con 0x7faa4c100e10 2024-03-28T22:19:04.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.760+0000 7faa4b7fe640 1 -- 172.21.15.16:0/1677422298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d24e426bb0 msgr2=0x55d24e42f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:04.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.760+0000 7faa4b7fe640 1 --2- 172.21.15.16:0/1677422298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d24e426bb0 0x55d24e42f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.760+0000 7faa4b7fe640 1 -- 172.21.15.16:0/1677422298 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d24e42f790 msgr2=0x55d24e431b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:04.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.760+0000 7faa4b7fe640 1 --2- 172.21.15.16:0/1677422298 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d24e42f790 0x55d24e431b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.760+0000 7faa4b7fe640 1 -- 172.21.15.16:0/1677422298 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d24e3467c0 con 0x55d24e426290 2024-03-28T22:19:04.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.760+0000 7faa4a7fc640 1 --2- 172.21.15.16:0/1677422298 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d24e42f790 0x55d24e431b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:19:04.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.760+0000 7faa4b7fe640 1 --2- 172.21.15.16:0/1677422298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d24e426290 0x55d24e426670 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7faa44013a00 tx=0x7faa44013ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aede716e606e51b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:04.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.761+0000 7faa31ffb640 1 -- 172.21.15.16:0/1677422298 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa44022020 con 0x55d24e426290 2024-03-28T22:19:04.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.761+0000 7faa31ffb640 1 -- 172.21.15.16:0/1677422298 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa44012d00 con 0x55d24e426290 2024-03-28T22:19:04.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.761+0000 7faa31ffb640 1 -- 172.21.15.16:0/1677422298 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa4401ea60 con 0x55d24e426290 2024-03-28T22:19:04.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.761+0000 7faa5440f740 1 -- 172.21.15.16:0/1677422298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d24e426290 msgr2=0x55d24e426670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:04.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.761+0000 7faa4b7fe640 1 --2- 172.21.15.16:0/2116117135 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7faa340a1ef0 0x7faa340a43b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7faa44023000 tx=0x7faa44021000 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:04.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.761+0000 7faa5440f740 1 --2- 172.21.15.16:0/1677422298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d24e426290 0x55d24e426670 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7faa44013a00 tx=0x7faa44013ec0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.761+0000 7faa5440f740 1 -- 172.21.15.16:0/1677422298 shutdown_connections 2024-03-28T22:19:04.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.761+0000 7faa5440f740 1 --2- 172.21.15.16:0/1677422298 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d24e42f790 0x55d24e431b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.761+0000 7faa5440f740 1 --2- 172.21.15.16:0/1677422298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d24e426bb0 0x55d24e42f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.761+0000 7faa5440f740 1 --2- 172.21.15.16:0/1677422298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d24e426290 0x55d24e426670 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.761+0000 7faa5440f740 1 -- 172.21.15.16:0/1677422298 >> 172.21.15.16:0/1677422298 conn(0x55d24e4209c0 msgr2=0x55d24e420da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:04.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.762+0000 7faa5440f740 1 -- 172.21.15.16:0/1677422298 shutdown_connections 2024-03-28T22:19:04.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.762+0000 7faa5440f740 1 -- 172.21.15.16:0/1677422298 wait complete. 2024-03-28T22:19:04.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.762+0000 7faa5440f740 1 -- start start 2024-03-28T22:19:04.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.762+0000 7faa5440f740 1 Processor -- start 2024-03-28T22:19:04.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.762+0000 7faa5440f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d24e426290 0x55d24e42c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:04.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.762+0000 7faa5440f740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d24e426bb0 0x55d24e42cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:04.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.762+0000 7faa5440f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d24e42f790 0x55d24e42d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:04.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.762+0000 7faa5440f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d24e432580 con 0x55d24e426290 2024-03-28T22:19:04.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.762+0000 7faa5440f740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55d24e3284a0 con 0x55d24e426bb0 2024-03-28T22:19:04.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.762+0000 7faa5440f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d24e432700 con 0x55d24e42f790 2024-03-28T22:19:04.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.763+0000 7faa5440f740 10 client.0 _ll_register_callbacks cb 0x55d24e32bd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:19:04.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.763+0000 7faa4affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d24e426290 0x55d24e42c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:04.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.763+0000 7faa4a7fc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d24e426bb0 0x55d24e42cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:04.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.763+0000 7faa4affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d24e426290 0x55d24e42c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60178/0 (socket says 172.21.15.16:60178) 2024-03-28T22:19:04.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.763+0000 7faa4affd640 1 -- 172.21.15.16:0/3905398421 learned_addr learned my addr 172.21.15.16:0/3905398421 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:04.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.763+0000 7faa4b7fe640 1 --2- 172.21.15.16:0/3905398421 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d24e42f790 0x55d24e42d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:04.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.763+0000 7faa4affd640 1 -- 172.21.15.16:0/3905398421 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d24e42f790 msgr2=0x55d24e42d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:04.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.763+0000 7faa4affd640 1 --2- 172.21.15.16:0/3905398421 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d24e42f790 0x55d24e42d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.763+0000 7faa4affd640 1 -- 172.21.15.16:0/3905398421 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d24e426bb0 msgr2=0x55d24e42cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:04.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.763+0000 7faa4affd640 1 --2- 172.21.15.16:0/3905398421 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d24e426bb0 0x55d24e42cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.763+0000 7faa4affd640 1 -- 172.21.15.16:0/3905398421 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d24e1687d0 con 0x55d24e426290 2024-03-28T22:19:04.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.763+0000 7faa4a7fc640 1 --2- 172.21.15.16:0/3905398421 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d24e426bb0 0x55d24e42cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:19:04.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.763+0000 7faa4affd640 1 --2- 172.21.15.16:0/3905398421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d24e426290 0x55d24e42c7c0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7faa3c009230 tx=0x7faa3c005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98f3a3e2634caf68 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:04.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.763+0000 7faa2affd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:19:04.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.763+0000 7faa2affd640 1 -- 172.21.15.16:0/3905398421 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3c004450 con 0x55d24e426290 2024-03-28T22:19:04.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.763+0000 7faa2affd640 1 -- 172.21.15.16:0/3905398421 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7faa3c0046c0 con 0x55d24e426290 2024-03-28T22:19:04.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.763+0000 7faa5440f740 1 -- 172.21.15.16:0/3905398421 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa44012d00 con 0x55d24e426290 2024-03-28T22:19:04.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.764+0000 7faa2affd640 1 -- 172.21.15.16:0/3905398421 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7faa3c0048c0 con 0x55d24e426290 2024-03-28T22:19:04.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.764+0000 7faa5440f740 10 client.17340 fetch_fsmap learned FSMap version 27 2024-03-28T22:19:04.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.764+0000 7faa5440f740 1 -- 172.21.15.16:0/3905398421 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d24e09e3f0 con 0x55d24e426290 2024-03-28T22:19:04.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.764+0000 7faa2affd640 1 -- 172.21.15.16:0/3905398421 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa44012d00 con 0x55d24e426290 2024-03-28T22:19:04.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.764+0000 7faa2affd640 1 -- 172.21.15.16:0/3905398421 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7faa3c046070 con 0x55d24e426290 2024-03-28T22:19:04.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.764+0000 7faa5440f740 10 client.17340 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:19:04.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.764+0000 7faa5440f740 10 client.17340 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:19:04.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.764+0000 7faa5440f740 20 client.17340 populate_metadata read hostname 'smithi016' 2024-03-28T22:19:04.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.764+0000 7faa5440f740 1 --2- 172.21.15.16:0/3905398421 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d24e43fae0 0x55d24e441f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:04.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.764+0000 7faa5440f740 4 client.17340 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:19:04.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.764+0000 7faa5440f740 1 -- 172.21.15.16:0/3905398421 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d24e3976c0 con 0x55d24e43fae0 2024-03-28T22:19:04.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.764+0000 7faa4a7fc640 1 --2- 172.21.15.16:0/3905398421 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d24e43fae0 0x55d24e441f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:04.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.765+0000 7faa4a7fc640 1 --2- 172.21.15.16:0/3905398421 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d24e43fae0 0x55d24e441f30 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa944adcd1308e0d server_cookie=11707912187788be in_seq=0 out_seq=0 2024-03-28T22:19:04.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.765+0000 7faa2affd640 10 client.17340 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:19:04.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.766+0000 7faa2affd640 1 -- 172.21.15.16:0/3905398421 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d24e3976c0 con 0x55d24e43fae0 2024-03-28T22:19:04.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.766+0000 7faa2affd640 10 client.17340 handle_command_reply: tid=0 2024-03-28T22:19:04.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.766+0000 7faa5440f740 1 client.17340 shutdown 2024-03-28T22:19:04.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.766+0000 7faa5440f740 1 -- 172.21.15.16:0/3905398421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d24e426290 msgr2=0x55d24e42c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:04.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.766+0000 7faa5440f740 1 --2- 172.21.15.16:0/3905398421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d24e426290 0x55d24e42c7c0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7faa3c009230 tx=0x7faa3c005d50 comp rx=0 tx=0).stop 2024-03-28T22:19:04.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.766+0000 7faa5440f740 1 -- 172.21.15.16:0/3905398421 shutdown_connections 2024-03-28T22:19:04.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.766+0000 7faa5440f740 1 --2- 172.21.15.16:0/3905398421 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d24e43fae0 0x55d24e441f30 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.766+0000 7faa5440f740 1 --2- 172.21.15.16:0/3905398421 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d24e42f790 0x55d24e42d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.766+0000 7faa2affd640 0 client.17340 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:19:04.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.766+0000 7faa5440f740 1 --2- 172.21.15.16:0/3905398421 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d24e426bb0 0x55d24e42cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.766+0000 7faa5440f740 1 --2- 172.21.15.16:0/3905398421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d24e426290 0x55d24e42c7c0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.766+0000 7faa5440f740 1 -- 172.21.15.16:0/3905398421 >> 172.21.15.16:0/3905398421 conn(0x55d24e4209c0 msgr2=0x55d24e422710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:04.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.766+0000 7faa5440f740 1 -- 172.21.15.16:0/3905398421 shutdown_connections 2024-03-28T22:19:04.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.766+0000 7faa5440f740 1 -- 172.21.15.16:0/3905398421 wait complete. 2024-03-28T22:19:04.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.766+0000 7faa5440f740 20 client.17340 trim_cache size 0 max 16384 2024-03-28T22:19:04.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.786+0000 7faa5440f740 1 Processor -- start 2024-03-28T22:19:04.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.786+0000 7faa5440f740 1 -- start start 2024-03-28T22:19:04.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.786+0000 7faa5440f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d24e426290 0x55d24e41fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:04.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.786+0000 7faa5440f740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d24e426bb0 0x55d24e43b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:04.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.786+0000 7faa5440f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d24e42f790 0x55d24e447800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:04.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.786+0000 7faa5440f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c1b1ef0 con 0x55d24e42f790 2024-03-28T22:19:04.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.786+0000 7faa5440f740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c1af7c0 con 0x55d24e426bb0 2024-03-28T22:19:04.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.786+0000 7faa5440f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c1ad630 con 0x55d24e426290 2024-03-28T22:19:04.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.787+0000 7faa4b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d24e42f790 0x55d24e447800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:04.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.787+0000 7faa4affd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d24e426290 0x55d24e41fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:04.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.787+0000 7faa4b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d24e42f790 0x55d24e447800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60188/0 (socket says 172.21.15.16:60188) 2024-03-28T22:19:04.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.787+0000 7faa4b7fe640 1 -- 172.21.15.16:0/4150791813 learned_addr learned my addr 172.21.15.16:0/4150791813 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:04.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.787+0000 7faa4a7fc640 1 --2- 172.21.15.16:0/4150791813 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d24e426bb0 0x55d24e43b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:04.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.787+0000 7faa4b7fe640 1 -- 172.21.15.16:0/4150791813 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d24e426290 msgr2=0x55d24e41fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:04.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.787+0000 7faa4b7fe640 1 --2- 172.21.15.16:0/4150791813 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d24e426290 0x55d24e41fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.787+0000 7faa4b7fe640 1 -- 172.21.15.16:0/4150791813 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d24e426bb0 msgr2=0x55d24e43b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:04.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.787+0000 7faa4b7fe640 1 --2- 172.21.15.16:0/4150791813 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d24e426bb0 0x55d24e43b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.787+0000 7faa4b7fe640 1 -- 172.21.15.16:0/4150791813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d24e41c070 con 0x55d24e42f790 2024-03-28T22:19:04.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.787+0000 7faa4affd640 1 --2- 172.21.15.16:0/4150791813 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d24e426290 0x55d24e41fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:19:04.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.787+0000 7faa4b7fe640 1 --2- 172.21.15.16:0/4150791813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d24e42f790 0x55d24e447800 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7faa44013950 tx=0x7faa44012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c05a5b80d6f48a70 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:04.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.787+0000 7faa317fa640 1 -- 172.21.15.16:0/4150791813 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa44022040 con 0x55d24e42f790 2024-03-28T22:19:04.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.787+0000 7faa317fa640 1 -- 172.21.15.16:0/4150791813 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa44035070 con 0x55d24e42f790 2024-03-28T22:19:04.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.787+0000 7faa317fa640 1 -- 172.21.15.16:0/4150791813 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa4401e770 con 0x55d24e42f790 2024-03-28T22:19:04.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.788+0000 7faa5440f740 1 -- 172.21.15.16:0/4150791813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d24e42f790 msgr2=0x55d24e447800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:04.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.788+0000 7faa5440f740 1 --2- 172.21.15.16:0/4150791813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d24e42f790 0x55d24e447800 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7faa44013950 tx=0x7faa44012000 comp rx=0 tx=0).stop 2024-03-28T22:19:04.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.788+0000 7faa5440f740 1 -- 172.21.15.16:0/4150791813 shutdown_connections 2024-03-28T22:19:04.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.788+0000 7faa5440f740 1 --2- 172.21.15.16:0/4150791813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d24e42f790 0x55d24e447800 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.788+0000 7faa5440f740 1 --2- 172.21.15.16:0/4150791813 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d24e426bb0 0x55d24e43b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.788+0000 7faa5440f740 1 --2- 172.21.15.16:0/4150791813 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d24e426290 0x55d24e41fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.788+0000 7faa5440f740 1 -- 172.21.15.16:0/4150791813 >> 172.21.15.16:0/4150791813 conn(0x55d24e4209c0 msgr2=0x55d24e42c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:04.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.788+0000 7faa5440f740 1 -- 172.21.15.16:0/4150791813 shutdown_connections 2024-03-28T22:19:04.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.788+0000 7faa5440f740 1 -- 172.21.15.16:0/4150791813 wait complete. 2024-03-28T22:19:04.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.788+0000 7faa5440f740 1 -- start start 2024-03-28T22:19:04.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.789+0000 7faa5440f740 1 Processor -- start 2024-03-28T22:19:04.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.789+0000 7faa5440f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d24e426290 0x55d24e437950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:04.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.789+0000 7faa5440f740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d24e426bb0 0x55d24e433cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:04.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.789+0000 7faa5440f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d24e42f790 0x55d24e434210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:04.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.789+0000 7faa5440f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c1adc80 con 0x55d24e426290 2024-03-28T22:19:04.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.789+0000 7faa5440f740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c1af0f0 con 0x55d24e426bb0 2024-03-28T22:19:04.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.789+0000 7faa5440f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c1ae350 con 0x55d24e42f790 2024-03-28T22:19:04.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.789+0000 7faa5440f740 10 client.0 _ll_register_callbacks cb 0x7faa3c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:19:04.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.789+0000 7faa4affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d24e426290 0x55d24e437950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:04.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.789+0000 7faa4affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d24e426290 0x55d24e437950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60202/0 (socket says 172.21.15.16:60202) 2024-03-28T22:19:04.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.789+0000 7faa4affd640 1 -- 172.21.15.16:0/1916180932 learned_addr learned my addr 172.21.15.16:0/1916180932 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:04.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.789+0000 7faa4a7fc640 1 --2- 172.21.15.16:0/1916180932 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d24e426bb0 0x55d24e433cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:04.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.789+0000 7faa4b7fe640 1 --2- 172.21.15.16:0/1916180932 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d24e42f790 0x55d24e434210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:04.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.789+0000 7faa4affd640 1 -- 172.21.15.16:0/1916180932 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d24e42f790 msgr2=0x55d24e434210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:04.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.789+0000 7faa4affd640 1 --2- 172.21.15.16:0/1916180932 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d24e42f790 0x55d24e434210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.789+0000 7faa4affd640 1 -- 172.21.15.16:0/1916180932 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d24e426bb0 msgr2=0x55d24e433cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:04.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.789+0000 7faa4affd640 1 --2- 172.21.15.16:0/1916180932 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d24e426bb0 0x55d24e433cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.789+0000 7faa4affd640 1 -- 172.21.15.16:0/1916180932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d24e05e0f0 con 0x55d24e426290 2024-03-28T22:19:04.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.790+0000 7faa4affd640 1 --2- 172.21.15.16:0/1916180932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d24e426290 0x55d24e437950 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7faa3c0037c0 tx=0x7faa3c02cd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8213684f87ff2673 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:04.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.790+0000 7faa2affd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:19:04.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.790+0000 7faa2affd640 1 -- 172.21.15.16:0/1916180932 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3c009520 con 0x55d24e426290 2024-03-28T22:19:04.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.790+0000 7faa2affd640 1 -- 172.21.15.16:0/1916180932 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7faa3c004460 con 0x55d24e426290 2024-03-28T22:19:04.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.790+0000 7faa5440f740 1 -- 172.21.15.16:0/1916180932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa44035070 con 0x55d24e426290 2024-03-28T22:19:04.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.790+0000 7faa2affd640 1 -- 172.21.15.16:0/1916180932 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7faa3c004a60 con 0x55d24e426290 2024-03-28T22:19:04.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.790+0000 7faa5440f740 10 client.17346 fetch_fsmap learned FSMap version 27 2024-03-28T22:19:04.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.790+0000 7faa5440f740 1 -- 172.21.15.16:0/1916180932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d24e4a69b0 con 0x55d24e426290 2024-03-28T22:19:04.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.790+0000 7faa2affd640 1 -- 172.21.15.16:0/1916180932 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa44035070 con 0x55d24e426290 2024-03-28T22:19:04.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.790+0000 7faa2affd640 1 -- 172.21.15.16:0/1916180932 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7faa3c046070 con 0x55d24e426290 2024-03-28T22:19:04.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.790+0000 7faa5440f740 10 client.17346 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:19:04.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.790+0000 7faa5440f740 10 client.17346 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:19:04.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.790+0000 7faa5440f740 20 client.17346 populate_metadata read hostname 'smithi016' 2024-03-28T22:19:04.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.790+0000 7faa5440f740 1 --2- 172.21.15.16:0/1916180932 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d24e43fae0 0x55d24e4a91b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:04.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.790+0000 7faa5440f740 4 client.17346 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:19:04.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.790+0000 7faa5440f740 1 -- 172.21.15.16:0/1916180932 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d24e391a10 con 0x55d24e43fae0 2024-03-28T22:19:04.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.791+0000 7faa4a7fc640 1 --2- 172.21.15.16:0/1916180932 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d24e43fae0 0x55d24e4a91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:04.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.791+0000 7faa2affd640 10 client.17346 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:19:04.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.791+0000 7faa4a7fc640 1 --2- 172.21.15.16:0/1916180932 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d24e43fae0 0x55d24e4a91b0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64c66321a6147bcc server_cookie=2315efd55e11f9db in_seq=0 out_seq=0 2024-03-28T22:19:04.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.791+0000 7faa2affd640 1 -- 172.21.15.16:0/1916180932 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55d24e391a10 con 0x55d24e43fae0 2024-03-28T22:19:04.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.791+0000 7faa2affd640 10 client.17346 handle_command_reply: tid=0 2024-03-28T22:19:04.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.791+0000 7faa5440f740 1 client.17346 shutdown 2024-03-28T22:19:04.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.792+0000 7faa5440f740 1 -- 172.21.15.16:0/1916180932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d24e426290 msgr2=0x55d24e437950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:04.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.792+0000 7faa5440f740 1 --2- 172.21.15.16:0/1916180932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d24e426290 0x55d24e437950 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7faa3c0037c0 tx=0x7faa3c02cd90 comp rx=0 tx=0).stop 2024-03-28T22:19:04.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.792+0000 7faa5440f740 1 -- 172.21.15.16:0/1916180932 shutdown_connections 2024-03-28T22:19:04.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.792+0000 7faa5440f740 1 --2- 172.21.15.16:0/1916180932 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55d24e43fae0 0x55d24e4a91b0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.792+0000 7faa5440f740 1 --2- 172.21.15.16:0/1916180932 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d24e42f790 0x55d24e434210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.792+0000 7faa2affd640 0 client.17346 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:19:04.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.792+0000 7faa5440f740 1 --2- 172.21.15.16:0/1916180932 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d24e426bb0 0x55d24e433cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.792+0000 7faa5440f740 1 --2- 172.21.15.16:0/1916180932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d24e426290 0x55d24e437950 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.792+0000 7faa5440f740 1 -- 172.21.15.16:0/1916180932 >> 172.21.15.16:0/1916180932 conn(0x55d24e4209c0 msgr2=0x55d24e41faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:04.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.792+0000 7faa5440f740 1 -- 172.21.15.16:0/1916180932 shutdown_connections 2024-03-28T22:19:04.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.792+0000 7faa5440f740 1 -- 172.21.15.16:0/1916180932 wait complete. 2024-03-28T22:19:04.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.792+0000 7faa5440f740 20 client.17346 trim_cache size 0 max 16384 2024-03-28T22:19:04.805 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:19:04.805 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T22:19:04.805 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-03-28T22:19:04.805 INFO:teuthology.orchestra.run.smithi016.stdout: "e6778206-7e54-4256-a562-09e0f35e48f4": { 2024-03-28T22:19:04.805 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-03-28T22:19:04.806 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "e6778206-7e54-4256-a562-09e0f35e48f4", 2024-03-28T22:19:04.806 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-03-28T22:19:04.806 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:19:04.806 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:19:04.806 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:19:04.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.792+0000 7faa5182a640 1 -- 172.21.15.16:0/2116117135 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7faa340a1ef0 msgr2=0x7faa340a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:04.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.792+0000 7faa5182a640 1 --2- 172.21.15.16:0/2116117135 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7faa340a1ef0 0x7faa340a43b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7faa44023000 tx=0x7faa44021000 comp rx=0 tx=0).stop 2024-03-28T22:19:04.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.792+0000 7faa5182a640 1 -- 172.21.15.16:0/2116117135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa4c100e10 msgr2=0x7faa4c19a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:04.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.792+0000 7faa5182a640 1 --2- 172.21.15.16:0/2116117135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa4c100e10 0x7faa4c19a760 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7faa4000e9f0 tx=0x7faa4000eeb0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.792+0000 7faa5182a640 1 -- 172.21.15.16:0/2116117135 shutdown_connections 2024-03-28T22:19:04.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.793+0000 7faa5182a640 1 --2- 172.21.15.16:0/2116117135 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7faa340a1ef0 0x7faa340a43b0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.793+0000 7faa5182a640 1 --2- 172.21.15.16:0/2116117135 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa4c10e0a0 0x7faa4c19eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.793+0000 7faa5182a640 1 --2- 172.21.15.16:0/2116117135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa4c100e10 0x7faa4c19a760 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.793+0000 7faa5182a640 1 --2- 172.21.15.16:0/2116117135 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa4c1004f0 0x7faa4c19a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:04.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.793+0000 7faa5182a640 1 -- 172.21.15.16:0/2116117135 >> 172.21.15.16:0/2116117135 conn(0x7faa4c077800 msgr2=0x7faa4c10f690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:04.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.793+0000 7faa5182a640 1 -- 172.21.15.16:0/2116117135 shutdown_connections 2024-03-28T22:19:04.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:04.793+0000 7faa5182a640 1 -- 172.21.15.16:0/2116117135 wait complete. 2024-03-28T22:19:04.809 INFO:tasks.cephfs.filesystem:scrub status for tag:e6778206-7e54-4256-a562-09e0f35e48f4 - {'path': '/', 'tag': 'e6778206-7e54-4256-a562-09e0f35e48f4', 'options': 'recursive,force'} 2024-03-28T22:19:05.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:05 smithi063 ceph-mon[36249]: scrub queued for path: / 2024-03-28T22:19:05.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:05 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:19:05.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:05 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:19:05.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:05 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:19:05.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:05 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:19:05.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:05 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:19:05.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:05 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:19:05.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:05 smithi016 ceph-mon[32374]: scrub queued for path: / 2024-03-28T22:19:05.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:05 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:19:05.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:05 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:19:05.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:05 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:19:05.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:05 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:19:05.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:05 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:19:05.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:05 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:19:05.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:05 smithi053 ceph-mon[35979]: scrub queued for path: / 2024-03-28T22:19:05.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:05 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:19:05.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:05 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:19:05.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:05 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:19:05.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:05 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:19:05.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:05 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:19:05.548 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:05 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:19:06.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:06 smithi063 ceph-mon[36249]: pgmap v262: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:19:06.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:06 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:19:06.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:06 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:19:06.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:06 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:19:06.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:06 smithi016 ceph-mon[32374]: pgmap v262: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:19:06.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:06 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:19:06.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:06 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:19:06.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:06 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:19:06.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:06 smithi053 ceph-mon[35979]: pgmap v262: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:19:06.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:06 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:19:06.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:06 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:19:06.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:06 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:19:08.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:08 smithi063 ceph-mon[36249]: pgmap v263: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-28T22:19:08.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:08 smithi016 ceph-mon[32374]: pgmap v263: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-28T22:19:08.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:08 smithi053 ceph-mon[35979]: pgmap v263: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-28T22:19:10.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:10 smithi063 ceph-mon[36249]: pgmap v264: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-28T22:19:10.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:10 smithi016 ceph-mon[32374]: pgmap v264: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-28T22:19:10.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:10 smithi053 ceph-mon[35979]: pgmap v264: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-28T22:19:11.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:11 smithi063 ceph-mon[36249]: osdmap e94: 12 total, 12 up, 12 in 2024-03-28T22:19:11.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:11 smithi016 ceph-mon[32374]: osdmap e94: 12 total, 12 up, 12 in 2024-03-28T22:19:11.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:11 smithi053 ceph-mon[35979]: osdmap e94: 12 total, 12 up, 12 in 2024-03-28T22:19:12.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:12 smithi063 ceph-mon[36249]: pgmap v266: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-03-28T22:19:12.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:12 smithi016 ceph-mon[32374]: pgmap v266: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-03-28T22:19:12.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:12 smithi053 ceph-mon[35979]: pgmap v266: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-03-28T22:19:14.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:14 smithi063 ceph-mon[36249]: pgmap v267: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-03-28T22:19:14.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:14 smithi016 ceph-mon[32374]: pgmap v267: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-03-28T22:19:14.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:14 smithi053 ceph-mon[35979]: pgmap v267: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-03-28T22:19:16.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:16 smithi063 ceph-mon[36249]: pgmap v268: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-03-28T22:19:16.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:16 smithi016 ceph-mon[32374]: pgmap v268: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-03-28T22:19:16.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:16 smithi053 ceph-mon[35979]: pgmap v268: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-03-28T22:19:18.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:18 smithi063 ceph-mon[36249]: pgmap v269: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T22:19:18.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:18 smithi063 ceph-mon[36249]: osdmap e95: 12 total, 12 up, 12 in 2024-03-28T22:19:18.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:18 smithi016 ceph-mon[32374]: pgmap v269: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T22:19:18.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:18 smithi016 ceph-mon[32374]: osdmap e95: 12 total, 12 up, 12 in 2024-03-28T22:19:18.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:18 smithi053 ceph-mon[35979]: pgmap v269: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T22:19:18.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:18 smithi053 ceph-mon[35979]: osdmap e95: 12 total, 12 up, 12 in 2024-03-28T22:19:20.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:20 smithi063 ceph-mon[36249]: pgmap v271: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 245 B/s wr, 7 op/s 2024-03-28T22:19:20.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:20 smithi063 ceph-mon[36249]: scrub summary: idle 2024-03-28T22:19:20.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:20 smithi016 ceph-mon[32374]: pgmap v271: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 245 B/s wr, 7 op/s 2024-03-28T22:19:20.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:20 smithi016 ceph-mon[32374]: scrub summary: idle 2024-03-28T22:19:20.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:20 smithi053 ceph-mon[35979]: pgmap v271: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 245 B/s wr, 7 op/s 2024-03-28T22:19:20.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:20 smithi053 ceph-mon[35979]: scrub summary: idle 2024-03-28T22:19:21.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:21 smithi063 ceph-mon[36249]: scrub summary: idle 2024-03-28T22:19:21.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:21 smithi016 ceph-mon[32374]: scrub summary: idle 2024-03-28T22:19:21.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:21 smithi053 ceph-mon[35979]: scrub summary: idle 2024-03-28T22:19:22.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:22 smithi063 ceph-mon[36249]: pgmap v272: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T22:19:22.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:22 smithi016 ceph-mon[32374]: pgmap v272: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T22:19:22.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:22 smithi053 ceph-mon[35979]: pgmap v272: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T22:19:24.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:24 smithi063 ceph-mon[36249]: pgmap v273: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T22:19:24.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:24 smithi016 ceph-mon[32374]: pgmap v273: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T22:19:24.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:24 smithi053 ceph-mon[35979]: pgmap v273: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T22:19:26.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:26 smithi063 ceph-mon[36249]: pgmap v274: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T22:19:26.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:26 smithi016 ceph-mon[32374]: pgmap v274: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T22:19:26.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:26 smithi053 ceph-mon[35979]: pgmap v274: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T22:19:28.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:28 smithi063 ceph-mon[36249]: pgmap v275: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-28T22:19:28.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:28 smithi016 ceph-mon[32374]: pgmap v275: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-28T22:19:28.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:28 smithi053 ceph-mon[35979]: pgmap v275: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-28T22:19:30.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:30 smithi063 ceph-mon[36249]: pgmap v276: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-28T22:19:30.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:30 smithi016 ceph-mon[32374]: pgmap v276: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-28T22:19:30.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:30 smithi053 ceph-mon[35979]: pgmap v276: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-28T22:19:32.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:32 smithi063 ceph-mon[36249]: pgmap v277: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-03-28T22:19:32.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:32 smithi016 ceph-mon[32374]: pgmap v277: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-03-28T22:19:32.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:32 smithi053 ceph-mon[35979]: pgmap v277: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-03-28T22:19:34.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:34 smithi063 ceph-mon[36249]: pgmap v278: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:19:34.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:34 smithi016 ceph-mon[32374]: pgmap v278: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:19:34.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:34 smithi053 ceph-mon[35979]: pgmap v278: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:19:34.810 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:19:34.810 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:19:34.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.927+0000 7f062fd3e640 1 -- 172.21.15.16:0/277440126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f062810bd40 msgr2=0x7f062810e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:34.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.927+0000 7f062fd3e640 1 --2- 172.21.15.16:0/277440126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f062810bd40 0x7f062810e170 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f0620009900 tx=0x7f062002bcd0 comp rx=0 tx=0).stop 2024-03-28T22:19:34.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.927+0000 7f062fd3e640 1 -- 172.21.15.16:0/277440126 shutdown_connections 2024-03-28T22:19:34.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.927+0000 7f062fd3e640 1 --2- 172.21.15.16:0/277440126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f062810bd40 0x7f062810e170 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:34.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.927+0000 7f062fd3e640 1 --2- 172.21.15.16:0/277440126 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0628103290 0x7f062810b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:34.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.927+0000 7f062fd3e640 1 --2- 172.21.15.16:0/277440126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0628102970 0x7f0628102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:34.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.927+0000 7f062fd3e640 1 -- 172.21.15.16:0/277440126 >> 172.21.15.16:0/277440126 conn(0x7f0628077800 msgr2=0x7f0628077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:34.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.927+0000 7f062fd3e640 1 -- 172.21.15.16:0/277440126 shutdown_connections 2024-03-28T22:19:34.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.927+0000 7f062fd3e640 1 -- 172.21.15.16:0/277440126 wait complete. 2024-03-28T22:19:34.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.928+0000 7f062fd3e640 1 Processor -- start 2024-03-28T22:19:34.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.928+0000 7f062fd3e640 1 -- start start 2024-03-28T22:19:34.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.928+0000 7f062fd3e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0628102970 0x7f0628197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:34.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.928+0000 7f062fd3e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0628103290 0x7f0628198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:34.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.928+0000 7f062fd3e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f062810bd40 0x7f062819c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:34.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.928+0000 7f062fd3e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0628111ad0 con 0x7f062810bd40 2024-03-28T22:19:34.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.928+0000 7f062fd3e640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0628111950 con 0x7f0628102970 2024-03-28T22:19:34.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.928+0000 7f062fd3e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0628111c50 con 0x7f0628103290 2024-03-28T22:19:34.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.928+0000 7f062e2db640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f062810bd40 0x7f062819c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:34.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.928+0000 7f062dada640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0628102970 0x7f0628197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:34.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.928+0000 7f062e2db640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f062810bd40 0x7f062819c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57814/0 (socket says 172.21.15.16:57814) 2024-03-28T22:19:34.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.928+0000 7f062e2db640 1 -- 172.21.15.16:0/2662895606 learned_addr learned my addr 172.21.15.16:0/2662895606 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:34.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.928+0000 7f062d2d9640 1 --2- 172.21.15.16:0/2662895606 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0628103290 0x7f0628198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:34.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.928+0000 7f062e2db640 1 -- 172.21.15.16:0/2662895606 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0628103290 msgr2=0x7f0628198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:34.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.928+0000 7f062e2db640 1 --2- 172.21.15.16:0/2662895606 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0628103290 0x7f0628198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:34.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.928+0000 7f062e2db640 1 -- 172.21.15.16:0/2662895606 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0628102970 msgr2=0x7f0628197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:34.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.928+0000 7f062e2db640 1 --2- 172.21.15.16:0/2662895606 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0628102970 0x7f0628197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:34.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.928+0000 7f062e2db640 1 -- 172.21.15.16:0/2662895606 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f062819cf70 con 0x7f062810bd40 2024-03-28T22:19:34.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.928+0000 7f062dada640 1 --2- 172.21.15.16:0/2662895606 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0628102970 0x7f0628197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:19:34.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.929+0000 7f062e2db640 1 --2- 172.21.15.16:0/2662895606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f062810bd40 0x7f062819c840 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f0620005940 tx=0x7f062002cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:34.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.929+0000 7f061effd640 1 -- 172.21.15.16:0/2662895606 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0620041070 con 0x7f062810bd40 2024-03-28T22:19:34.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.929+0000 7f061effd640 1 -- 172.21.15.16:0/2662895606 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0620004450 con 0x7f062810bd40 2024-03-28T22:19:34.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.929+0000 7f062fd3e640 1 -- 172.21.15.16:0/2662895606 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f062819d200 con 0x7f062810bd40 2024-03-28T22:19:34.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.929+0000 7f061effd640 1 -- 172.21.15.16:0/2662895606 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0620003de0 con 0x7f062810bd40 2024-03-28T22:19:34.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.929+0000 7f062fd3e640 1 -- 172.21.15.16:0/2662895606 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06281a4af0 con 0x7f062810bd40 2024-03-28T22:19:34.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.931+0000 7f061effd640 1 -- 172.21.15.16:0/2662895606 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f0620044070 con 0x7f062810bd40 2024-03-28T22:19:34.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.931+0000 7f0632a04740 1 Processor -- start 2024-03-28T22:19:34.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.932+0000 7f0632a04740 1 -- start start 2024-03-28T22:19:34.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.932+0000 7f0632a04740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc6552d290 0x55fc6552d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:34.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.932+0000 7f0632a04740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc6552dbb0 0x55fc65536250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:34.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.932+0000 7f061effd640 1 --2- 172.21.15.16:0/2662895606 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f06140a1ef0 0x7f06140a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:34.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.932+0000 7f0632a04740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc65536790 0x55fc65538b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:34.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.932+0000 7f0632a04740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55fc65491180 con 0x55fc6552d290 2024-03-28T22:19:34.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.932+0000 7f0632a04740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55fc65439b70 con 0x55fc65536790 2024-03-28T22:19:34.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.932+0000 7f0632a04740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55fc6548d220 con 0x55fc6552dbb0 2024-03-28T22:19:34.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.932+0000 7f062d2d9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc6552d290 0x55fc6552d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:34.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.932+0000 7f062e2db640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc65536790 0x55fc65538b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:34.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.932+0000 7f062d2d9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc6552d290 0x55fc6552d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57828/0 (socket says 172.21.15.16:57828) 2024-03-28T22:19:34.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.932+0000 7f062d2d9640 1 -- 172.21.15.16:0/4023501422 learned_addr learned my addr 172.21.15.16:0/4023501422 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:34.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.932+0000 7f061effd640 1 -- 172.21.15.16:0/2662895606 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7f06200efc00 con 0x7f062810bd40 2024-03-28T22:19:34.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.932+0000 7f062d2d9640 1 --2- 172.21.15.16:0/2662895606 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f06140a1ef0 0x7f06140a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:34.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.932+0000 7f062dada640 1 --2- 172.21.15.16:0/4023501422 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc6552dbb0 0x55fc65536250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:34.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.933+0000 7f062e2db640 1 -- 172.21.15.16:0/4023501422 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc6552dbb0 msgr2=0x55fc65536250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:34.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.933+0000 7f062e2db640 1 --2- 172.21.15.16:0/4023501422 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc6552dbb0 0x55fc65536250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:34.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.933+0000 7f062e2db640 1 -- 172.21.15.16:0/4023501422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc6552d290 msgr2=0x55fc6552d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:34.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.933+0000 7f062e2db640 1 --2- 172.21.15.16:0/4023501422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc6552d290 0x55fc6552d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:34.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.933+0000 7f062e2db640 1 -- 172.21.15.16:0/4023501422 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc6544d7c0 con 0x55fc65536790 2024-03-28T22:19:34.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.933+0000 7f062e2db640 1 --2- 172.21.15.16:0/4023501422 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc65536790 0x55fc65538b80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f06200ed460 tx=0x7f06200038b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0fc26c010bb2c6a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:34.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.933+0000 7f062d2d9640 1 --2- 172.21.15.16:0/2662895606 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f06140a1ef0 0x7f06140a43b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f06180058c0 tx=0x7f0618005f90 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:34.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.933+0000 7f0613fff640 1 -- 172.21.15.16:0/4023501422 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f062004b070 con 0x55fc65536790 2024-03-28T22:19:34.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.933+0000 7f0613fff640 1 -- 172.21.15.16:0/4023501422 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06200bc7b0 con 0x55fc65536790 2024-03-28T22:19:34.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.933+0000 7f0613fff640 1 -- 172.21.15.16:0/4023501422 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06200c4ea0 con 0x55fc65536790 2024-03-28T22:19:34.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.934+0000 7f0632a04740 1 -- 172.21.15.16:0/4023501422 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc65536790 msgr2=0x55fc65538b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:34.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.934+0000 7f0632a04740 1 --2- 172.21.15.16:0/4023501422 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc65536790 0x55fc65538b80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f06200ed460 tx=0x7f06200038b0 comp rx=0 tx=0).stop 2024-03-28T22:19:34.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.934+0000 7f0632a04740 1 -- 172.21.15.16:0/4023501422 shutdown_connections 2024-03-28T22:19:34.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.934+0000 7f0632a04740 1 --2- 172.21.15.16:0/4023501422 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc65536790 0x55fc65538b80 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:34.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.934+0000 7f0632a04740 1 --2- 172.21.15.16:0/4023501422 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc6552dbb0 0x55fc65536250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:34.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.934+0000 7f0632a04740 1 --2- 172.21.15.16:0/4023501422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc6552d290 0x55fc6552d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:34.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.934+0000 7f0632a04740 1 -- 172.21.15.16:0/4023501422 >> 172.21.15.16:0/4023501422 conn(0x55fc655279c0 msgr2=0x55fc65527da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:34.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.934+0000 7f0632a04740 1 -- 172.21.15.16:0/4023501422 shutdown_connections 2024-03-28T22:19:34.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.934+0000 7f0632a04740 1 -- 172.21.15.16:0/4023501422 wait complete. 2024-03-28T22:19:34.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.934+0000 7f0632a04740 1 -- start start 2024-03-28T22:19:34.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.934+0000 7f0632a04740 1 Processor -- start 2024-03-28T22:19:34.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.935+0000 7f0632a04740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc6552d290 0x55fc65533800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:34.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.935+0000 7f0632a04740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc6552dbb0 0x55fc65533d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:34.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.935+0000 7f0632a04740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc65536790 0x55fc65534280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:34.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.935+0000 7f0632a04740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55fc65439b70 con 0x55fc6552d290 2024-03-28T22:19:34.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.935+0000 7f0632a04740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55fc6542f4a0 con 0x55fc6552dbb0 2024-03-28T22:19:34.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.935+0000 7f0632a04740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55fc65539340 con 0x55fc65536790 2024-03-28T22:19:34.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.935+0000 7f0632a04740 10 client.0 _ll_register_callbacks cb 0x55fc65432d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:19:34.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.935+0000 7f062dada640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc6552d290 0x55fc65533800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:34.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.935+0000 7f062dada640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc6552d290 0x55fc65533800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57836/0 (socket says 172.21.15.16:57836) 2024-03-28T22:19:34.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.935+0000 7f062dada640 1 -- 172.21.15.16:0/644380533 learned_addr learned my addr 172.21.15.16:0/644380533 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:34.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.935+0000 7f062d2d9640 1 --2- 172.21.15.16:0/644380533 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc6552dbb0 0x55fc65533d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:34.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.935+0000 7f062e2db640 1 --2- 172.21.15.16:0/644380533 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc65536790 0x55fc65534280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:34.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.935+0000 7f062dada640 1 -- 172.21.15.16:0/644380533 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc65536790 msgr2=0x55fc65534280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:34.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.935+0000 7f062dada640 1 --2- 172.21.15.16:0/644380533 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc65536790 0x55fc65534280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:34.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.935+0000 7f062dada640 1 -- 172.21.15.16:0/644380533 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc6552dbb0 msgr2=0x55fc65533d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:34.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.935+0000 7f062dada640 1 --2- 172.21.15.16:0/644380533 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc6552dbb0 0x55fc65533d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:34.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.935+0000 7f062dada640 1 -- 172.21.15.16:0/644380533 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc6526f7d0 con 0x55fc6552d290 2024-03-28T22:19:34.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.935+0000 7f062d2d9640 1 --2- 172.21.15.16:0/644380533 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc6552dbb0 0x55fc65533d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:19:34.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.935+0000 7f062dada640 1 --2- 172.21.15.16:0/644380533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc6552d290 0x55fc65533800 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f0624012ab0 tx=0x7f0624012f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd715c93e527e201 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:34.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.935+0000 7f06117fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:19:34.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.935+0000 7f06117fa640 1 -- 172.21.15.16:0/644380533 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f062401f060 con 0x55fc6552d290 2024-03-28T22:19:34.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.935+0000 7f06117fa640 1 -- 172.21.15.16:0/644380533 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7f062401a3f0 con 0x55fc6552d290 2024-03-28T22:19:34.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.936+0000 7f0632a04740 1 -- 172.21.15.16:0/644380533 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06200bc7b0 con 0x55fc6552d290 2024-03-28T22:19:34.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.936+0000 7f06117fa640 1 -- 172.21.15.16:0/644380533 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f062401a5f0 con 0x55fc6552d290 2024-03-28T22:19:34.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.936+0000 7f0632a04740 10 client.17358 fetch_fsmap learned FSMap version 27 2024-03-28T22:19:34.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.936+0000 7f0632a04740 1 -- 172.21.15.16:0/644380533 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fc651a53f0 con 0x55fc6552d290 2024-03-28T22:19:34.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.936+0000 7f06117fa640 1 -- 172.21.15.16:0/644380533 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06200bc7b0 con 0x55fc6552d290 2024-03-28T22:19:34.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.936+0000 7f06117fa640 1 -- 172.21.15.16:0/644380533 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f0624019030 con 0x55fc6552d290 2024-03-28T22:19:34.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.936+0000 7f0632a04740 10 client.17358 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:19:34.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.936+0000 7f0632a04740 10 client.17358 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:19:34.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.936+0000 7f0632a04740 20 client.17358 populate_metadata read hostname 'smithi016' 2024-03-28T22:19:34.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.936+0000 7f0632a04740 1 --2- 172.21.15.16:0/644380533 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55fc65546850 0x55fc65548cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:34.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.936+0000 7f0632a04740 4 client.17358 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:19:34.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.936+0000 7f0632a04740 1 -- 172.21.15.16:0/644380533 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fc6549e6c0 con 0x55fc65546850 2024-03-28T22:19:34.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.936+0000 7f062d2d9640 1 --2- 172.21.15.16:0/644380533 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55fc65546850 0x55fc65548cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:34.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.937+0000 7f06117fa640 10 client.17358 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:19:34.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.937+0000 7f062d2d9640 1 --2- 172.21.15.16:0/644380533 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55fc65546850 0x55fc65548cc0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfd2e4cffe0ff9ca server_cookie=8738e1003196603f in_seq=0 out_seq=0 2024-03-28T22:19:34.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.938+0000 7f06117fa640 1 -- 172.21.15.16:0/644380533 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fc6549e6c0 con 0x55fc65546850 2024-03-28T22:19:34.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.938+0000 7f06117fa640 10 client.17358 handle_command_reply: tid=0 2024-03-28T22:19:34.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.938+0000 7f0632a04740 1 client.17358 shutdown 2024-03-28T22:19:34.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.938+0000 7f0632a04740 1 -- 172.21.15.16:0/644380533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc6552d290 msgr2=0x55fc65533800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:34.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.938+0000 7f0632a04740 1 --2- 172.21.15.16:0/644380533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc6552d290 0x55fc65533800 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f0624012ab0 tx=0x7f0624012f70 comp rx=0 tx=0).stop 2024-03-28T22:19:34.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.938+0000 7f0632a04740 1 -- 172.21.15.16:0/644380533 shutdown_connections 2024-03-28T22:19:34.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.938+0000 7f0632a04740 1 --2- 172.21.15.16:0/644380533 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55fc65546850 0x55fc65548cc0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:34.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.938+0000 7f0632a04740 1 --2- 172.21.15.16:0/644380533 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc65536790 0x55fc65534280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:34.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.938+0000 7f0632a04740 1 --2- 172.21.15.16:0/644380533 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc6552dbb0 0x55fc65533d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:34.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.938+0000 7f0632a04740 1 --2- 172.21.15.16:0/644380533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc6552d290 0x55fc65533800 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:34.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.938+0000 7f06117fa640 0 client.17358 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:19:34.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.938+0000 7f0632a04740 1 -- 172.21.15.16:0/644380533 >> 172.21.15.16:0/644380533 conn(0x55fc655279c0 msgr2=0x55fc65529710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:34.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.938+0000 7f0632a04740 1 -- 172.21.15.16:0/644380533 shutdown_connections 2024-03-28T22:19:34.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.938+0000 7f0632a04740 1 -- 172.21.15.16:0/644380533 wait complete. 2024-03-28T22:19:34.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.938+0000 7f0632a04740 20 client.17358 trim_cache size 0 max 16384 2024-03-28T22:19:34.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.956+0000 7f0632a04740 1 Processor -- start 2024-03-28T22:19:34.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.956+0000 7f0632a04740 1 -- start start 2024-03-28T22:19:34.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.956+0000 7f0632a04740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc6552d290 0x55fc65535f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:34.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.956+0000 7f0632a04740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc6552dbb0 0x55fc65541f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:34.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.956+0000 7f0632a04740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc65536790 0x55fc6554e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:34.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.956+0000 7f0632a04740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f06281afc80 con 0x55fc6552dbb0 2024-03-28T22:19:34.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.956+0000 7f0632a04740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f06281ad550 con 0x55fc65536790 2024-03-28T22:19:34.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.956+0000 7f0632a04740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f06281ab3c0 con 0x55fc6552d290 2024-03-28T22:19:34.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.956+0000 7f062d2d9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc6552dbb0 0x55fc65541f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:34.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.956+0000 7f062d2d9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc6552dbb0 0x55fc65541f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57850/0 (socket says 172.21.15.16:57850) 2024-03-28T22:19:34.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.956+0000 7f062d2d9640 1 -- 172.21.15.16:0/1449976094 learned_addr learned my addr 172.21.15.16:0/1449976094 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:34.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.956+0000 7f062e2db640 1 --2- 172.21.15.16:0/1449976094 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc65536790 0x55fc6554e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:34.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.956+0000 7f062dada640 1 --2- 172.21.15.16:0/1449976094 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc6552d290 0x55fc65535f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:34.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.956+0000 7f062d2d9640 1 -- 172.21.15.16:0/1449976094 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc6552d290 msgr2=0x55fc65535f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:34.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.956+0000 7f062d2d9640 1 --2- 172.21.15.16:0/1449976094 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc6552d290 0x55fc65535f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:34.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.956+0000 7f062d2d9640 1 -- 172.21.15.16:0/1449976094 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc65536790 msgr2=0x55fc6554e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:34.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.956+0000 7f062d2d9640 1 --2- 172.21.15.16:0/1449976094 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc65536790 0x55fc6554e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:34.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.956+0000 7f062d2d9640 1 -- 172.21.15.16:0/1449976094 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc6552d9e0 con 0x55fc6552dbb0 2024-03-28T22:19:34.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.956+0000 7f062e2db640 1 --2- 172.21.15.16:0/1449976094 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc65536790 0x55fc6554e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:19:34.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.956+0000 7f062d2d9640 1 --2- 172.21.15.16:0/1449976094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc6552dbb0 0x55fc65541f60 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x55fc65533060 tx=0x7f0618005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a1f6b8d012096fb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:34.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.957+0000 7f06137fe640 1 -- 172.21.15.16:0/1449976094 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f061802cb90 con 0x55fc6552dbb0 2024-03-28T22:19:34.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.957+0000 7f06137fe640 1 -- 172.21.15.16:0/1449976094 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f061802cd30 con 0x55fc6552dbb0 2024-03-28T22:19:34.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.957+0000 7f06137fe640 1 -- 172.21.15.16:0/1449976094 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06180047a0 con 0x55fc6552dbb0 2024-03-28T22:19:34.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.957+0000 7f0632a04740 1 -- 172.21.15.16:0/1449976094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc6552dbb0 msgr2=0x55fc65541f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:34.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.957+0000 7f0632a04740 1 --2- 172.21.15.16:0/1449976094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc6552dbb0 0x55fc65541f60 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x55fc65533060 tx=0x7f0618005850 comp rx=0 tx=0).stop 2024-03-28T22:19:34.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.957+0000 7f0632a04740 1 -- 172.21.15.16:0/1449976094 shutdown_connections 2024-03-28T22:19:34.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.957+0000 7f0632a04740 1 --2- 172.21.15.16:0/1449976094 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc65536790 0x55fc6554e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:34.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.957+0000 7f0632a04740 1 --2- 172.21.15.16:0/1449976094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc6552dbb0 0x55fc65541f60 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:34.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.957+0000 7f0632a04740 1 --2- 172.21.15.16:0/1449976094 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc6552d290 0x55fc65535f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:34.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.957+0000 7f0632a04740 1 -- 172.21.15.16:0/1449976094 >> 172.21.15.16:0/1449976094 conn(0x55fc655279c0 msgr2=0x55fc65526c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:34.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.957+0000 7f0632a04740 1 -- 172.21.15.16:0/1449976094 shutdown_connections 2024-03-28T22:19:34.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.957+0000 7f0632a04740 1 -- 172.21.15.16:0/1449976094 wait complete. 2024-03-28T22:19:34.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.958+0000 7f0632a04740 1 -- start start 2024-03-28T22:19:34.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.958+0000 7f0632a04740 1 Processor -- start 2024-03-28T22:19:34.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.958+0000 7f0632a04740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc6552d290 0x55fc6553a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:34.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.958+0000 7f0632a04740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc6552dbb0 0x55fc6553aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:34.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.958+0000 7f0632a04740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc65536790 0x55fc655b0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:34.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.958+0000 7f0632a04740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f06281aba10 con 0x55fc65536790 2024-03-28T22:19:34.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.958+0000 7f0632a04740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f06281ace80 con 0x55fc6552dbb0 2024-03-28T22:19:34.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.958+0000 7f0632a04740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f06281ac0e0 con 0x55fc6552d290 2024-03-28T22:19:34.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.958+0000 7f0632a04740 10 client.0 _ll_register_callbacks cb 0x7f0624009f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:19:34.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.958+0000 7f062e2db640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc65536790 0x55fc655b0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:34.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.958+0000 7f062e2db640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc65536790 0x55fc655b0010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57866/0 (socket says 172.21.15.16:57866) 2024-03-28T22:19:34.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.958+0000 7f062e2db640 1 -- 172.21.15.16:0/2417878828 learned_addr learned my addr 172.21.15.16:0/2417878828 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:34.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.958+0000 7f062d2d9640 1 --2- 172.21.15.16:0/2417878828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc6552dbb0 0x55fc6553aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:34.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.958+0000 7f062dada640 1 --2- 172.21.15.16:0/2417878828 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc6552d290 0x55fc6553a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:34.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.958+0000 7f062e2db640 1 -- 172.21.15.16:0/2417878828 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc6552d290 msgr2=0x55fc6553a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:34.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.958+0000 7f062e2db640 1 --2- 172.21.15.16:0/2417878828 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc6552d290 0x55fc6553a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:34.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.958+0000 7f062e2db640 1 -- 172.21.15.16:0/2417878828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc6552dbb0 msgr2=0x55fc6553aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:34.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.958+0000 7f062e2db640 1 --2- 172.21.15.16:0/2417878828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc6552dbb0 0x55fc6553aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:34.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.958+0000 7f062e2db640 1 -- 172.21.15.16:0/2417878828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc651650f0 con 0x55fc65536790 2024-03-28T22:19:34.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.959+0000 7f062e2db640 1 --2- 172.21.15.16:0/2417878828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc65536790 0x55fc655b0010 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f0620009430 tx=0x7f06200b8b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=457d87a752b4f926 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:34.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.959+0000 7f06117fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:19:34.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.959+0000 7f06117fa640 1 -- 172.21.15.16:0/2417878828 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06200d4020 con 0x55fc65536790 2024-03-28T22:19:34.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.959+0000 7f06117fa640 1 -- 172.21.15.16:0/2417878828 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7f06200ed5d0 con 0x55fc65536790 2024-03-28T22:19:34.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.959+0000 7f0632a04740 1 -- 172.21.15.16:0/2417878828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f061802cd30 con 0x55fc65536790 2024-03-28T22:19:34.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.959+0000 7f06117fa640 1 -- 172.21.15.16:0/2417878828 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f062004b040 con 0x55fc65536790 2024-03-28T22:19:34.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.959+0000 7f0632a04740 10 client.17364 fetch_fsmap learned FSMap version 27 2024-03-28T22:19:34.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.959+0000 7f0632a04740 1 -- 172.21.15.16:0/2417878828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fc655b1600 con 0x55fc65536790 2024-03-28T22:19:34.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.959+0000 7f06117fa640 1 -- 172.21.15.16:0/2417878828 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f061802cd30 con 0x55fc65536790 2024-03-28T22:19:34.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.959+0000 7f06117fa640 1 -- 172.21.15.16:0/2417878828 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f06200b8e20 con 0x55fc65536790 2024-03-28T22:19:34.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.959+0000 7f0632a04740 10 client.17364 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:19:34.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.959+0000 7f0632a04740 10 client.17364 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:19:34.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.959+0000 7f0632a04740 20 client.17364 populate_metadata read hostname 'smithi016' 2024-03-28T22:19:34.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.959+0000 7f0632a04740 1 --2- 172.21.15.16:0/2417878828 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55fc655b1b70 0x55fc655b3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:34.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.959+0000 7f0632a04740 4 client.17364 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:19:34.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.959+0000 7f0632a04740 1 -- 172.21.15.16:0/2417878828 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fc65498a10 con 0x55fc655b1b70 2024-03-28T22:19:34.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.960+0000 7f062dada640 1 --2- 172.21.15.16:0/2417878828 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55fc655b1b70 0x55fc655b3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:34.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.960+0000 7f06117fa640 10 client.17364 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:19:34.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.960+0000 7f062dada640 1 --2- 172.21.15.16:0/2417878828 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55fc655b1b70 0x55fc655b3f90 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ddfc6e395762bd50 server_cookie=661248a35c76190f in_seq=0 out_seq=0 2024-03-28T22:19:34.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.960+0000 7f06117fa640 1 -- 172.21.15.16:0/2417878828 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55fc65498a10 con 0x55fc655b1b70 2024-03-28T22:19:34.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.960+0000 7f06117fa640 10 client.17364 handle_command_reply: tid=0 2024-03-28T22:19:34.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.960+0000 7f0632a04740 1 client.17364 shutdown 2024-03-28T22:19:34.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.960+0000 7f0632a04740 1 -- 172.21.15.16:0/2417878828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc65536790 msgr2=0x55fc655b0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:34.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.960+0000 7f0632a04740 1 --2- 172.21.15.16:0/2417878828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc65536790 0x55fc655b0010 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f0620009430 tx=0x7f06200b8b50 comp rx=0 tx=0).stop 2024-03-28T22:19:34.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.960+0000 7f0632a04740 1 -- 172.21.15.16:0/2417878828 shutdown_connections 2024-03-28T22:19:34.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.960+0000 7f0632a04740 1 --2- 172.21.15.16:0/2417878828 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55fc655b1b70 0x55fc655b3f90 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:34.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.960+0000 7f0632a04740 1 --2- 172.21.15.16:0/2417878828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fc65536790 0x55fc655b0010 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:34.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.960+0000 7f0632a04740 1 --2- 172.21.15.16:0/2417878828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fc6552dbb0 0x55fc6553aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:34.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.960+0000 7f0632a04740 1 --2- 172.21.15.16:0/2417878828 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc6552d290 0x55fc6553a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:34.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.960+0000 7f06117fa640 0 client.17364 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:19:34.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.960+0000 7f0632a04740 1 -- 172.21.15.16:0/2417878828 >> 172.21.15.16:0/2417878828 conn(0x55fc655279c0 msgr2=0x55fc65528e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:34.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.960+0000 7f0632a04740 1 -- 172.21.15.16:0/2417878828 shutdown_connections 2024-03-28T22:19:34.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.960+0000 7f0632a04740 1 -- 172.21.15.16:0/2417878828 wait complete. 2024-03-28T22:19:34.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.961+0000 7f0632a04740 20 client.17364 trim_cache size 0 max 16384 2024-03-28T22:19:34.972 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:19:34.972 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-03-28T22:19:34.972 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-03-28T22:19:34.972 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:19:34.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.961+0000 7f062fd3e640 1 -- 172.21.15.16:0/2662895606 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f06140a1ef0 msgr2=0x7f06140a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:34.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.961+0000 7f062fd3e640 1 --2- 172.21.15.16:0/2662895606 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f06140a1ef0 0x7f06140a43b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f06180058c0 tx=0x7f0618005f90 comp rx=0 tx=0).stop 2024-03-28T22:19:34.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.961+0000 7f062fd3e640 1 -- 172.21.15.16:0/2662895606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f062810bd40 msgr2=0x7f062819c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:34.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.961+0000 7f062fd3e640 1 --2- 172.21.15.16:0/2662895606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f062810bd40 0x7f062819c840 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f0620005940 tx=0x7f062002cc40 comp rx=0 tx=0).stop 2024-03-28T22:19:34.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.961+0000 7f062fd3e640 1 -- 172.21.15.16:0/2662895606 shutdown_connections 2024-03-28T22:19:34.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.961+0000 7f062fd3e640 1 --2- 172.21.15.16:0/2662895606 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f06140a1ef0 0x7f06140a43b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:34.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.961+0000 7f062fd3e640 1 --2- 172.21.15.16:0/2662895606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f062810bd40 0x7f062819c840 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:34.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.961+0000 7f062fd3e640 1 --2- 172.21.15.16:0/2662895606 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0628103290 0x7f0628198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:34.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.961+0000 7f062fd3e640 1 --2- 172.21.15.16:0/2662895606 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0628102970 0x7f0628197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:34.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.961+0000 7f062fd3e640 1 -- 172.21.15.16:0/2662895606 >> 172.21.15.16:0/2662895606 conn(0x7f0628077800 msgr2=0x7f06280fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:34.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.961+0000 7f062fd3e640 1 -- 172.21.15.16:0/2662895606 shutdown_connections 2024-03-28T22:19:34.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:34.961+0000 7f062fd3e640 1 -- 172.21.15.16:0/2662895606 wait complete. 2024-03-28T22:19:34.975 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:19:34.976 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:19:35.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.078+0000 7f24d9fa0640 1 -- 172.21.15.16:0/446282518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24d41004a0 msgr2=0x7f24d410e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:35.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.078+0000 7f24d9fa0640 1 --2- 172.21.15.16:0/446282518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24d41004a0 0x7f24d410e4c0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f24cc009cd0 tx=0x7f24cc02c0f0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.078+0000 7f24d9fa0640 1 -- 172.21.15.16:0/446282518 shutdown_connections 2024-03-28T22:19:35.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.078+0000 7f24d9fa0640 1 --2- 172.21.15.16:0/446282518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24d41004a0 0x7f24d410e4c0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.078+0000 7f24d9fa0640 1 --2- 172.21.15.16:0/446282518 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24d40ffae0 0x7f24d40fff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.078+0000 7f24d9fa0640 1 --2- 172.21.15.16:0/446282518 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f24d4107600 0x7f24d41079e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.078+0000 7f24d9fa0640 1 -- 172.21.15.16:0/446282518 >> 172.21.15.16:0/446282518 conn(0x7f24d4076a60 msgr2=0x7f24d40fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:35.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.078+0000 7f24d9fa0640 1 -- 172.21.15.16:0/446282518 shutdown_connections 2024-03-28T22:19:35.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.078+0000 7f24d9fa0640 1 -- 172.21.15.16:0/446282518 wait complete. 2024-03-28T22:19:35.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.079+0000 7f24d9fa0640 1 Processor -- start 2024-03-28T22:19:35.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.079+0000 7f24d9fa0640 1 -- start start 2024-03-28T22:19:35.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.079+0000 7f24d9fa0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24d40ffae0 0x7f24d4197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.079+0000 7f24d9fa0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f24d41004a0 0x7f24d41983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.079+0000 7f24d9fa0640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24d4107600 0x7f24d419c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.079+0000 7f24d9fa0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f24d4111a10 con 0x7f24d40ffae0 2024-03-28T22:19:35.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.079+0000 7f24d9fa0640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f24d4111890 con 0x7f24d4107600 2024-03-28T22:19:35.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.079+0000 7f24d9fa0640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f24d4111b90 con 0x7f24d41004a0 2024-03-28T22:19:35.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.079+0000 7f24d8f9e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24d40ffae0 0x7f24d4197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.079+0000 7f24d8f9e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24d40ffae0 0x7f24d4197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57882/0 (socket says 172.21.15.16:57882) 2024-03-28T22:19:35.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.079+0000 7f24d8f9e640 1 -- 172.21.15.16:0/3251641273 learned_addr learned my addr 172.21.15.16:0/3251641273 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:35.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.079+0000 7f24d979f640 1 --2- 172.21.15.16:0/3251641273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24d4107600 0x7f24d419c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.079+0000 7f24d3fff640 1 --2- 172.21.15.16:0/3251641273 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f24d41004a0 0x7f24d41983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.079+0000 7f24d8f9e640 1 -- 172.21.15.16:0/3251641273 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f24d41004a0 msgr2=0x7f24d41983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:35.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.079+0000 7f24d8f9e640 1 --2- 172.21.15.16:0/3251641273 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f24d41004a0 0x7f24d41983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.079+0000 7f24d8f9e640 1 -- 172.21.15.16:0/3251641273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24d4107600 msgr2=0x7f24d419c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:35.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.079+0000 7f24d8f9e640 1 --2- 172.21.15.16:0/3251641273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24d4107600 0x7f24d419c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.079+0000 7f24d8f9e640 1 -- 172.21.15.16:0/3251641273 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24d419cea0 con 0x7f24d40ffae0 2024-03-28T22:19:35.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.080+0000 7f24d8f9e640 1 --2- 172.21.15.16:0/3251641273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24d40ffae0 0x7f24d4197eb0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f24c800e9f0 tx=0x7f24c800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:35.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.080+0000 7f24d1ffb640 1 -- 172.21.15.16:0/3251641273 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c800cd80 con 0x7f24d40ffae0 2024-03-28T22:19:35.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.080+0000 7f24d1ffb640 1 -- 172.21.15.16:0/3251641273 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24c8005500 con 0x7f24d40ffae0 2024-03-28T22:19:35.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.080+0000 7f24d9fa0640 1 -- 172.21.15.16:0/3251641273 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24d419d190 con 0x7f24d40ffae0 2024-03-28T22:19:35.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.080+0000 7f24d1ffb640 1 -- 172.21.15.16:0/3251641273 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c8010690 con 0x7f24d40ffae0 2024-03-28T22:19:35.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.080+0000 7f24d9fa0640 1 -- 172.21.15.16:0/3251641273 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24d41a4a20 con 0x7f24d40ffae0 2024-03-28T22:19:35.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.081+0000 7f24d9fa0640 1 -- 172.21.15.16:0/3251641273 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24d404f520 con 0x7f24d40ffae0 2024-03-28T22:19:35.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.082+0000 7f24d1ffb640 1 -- 172.21.15.16:0/3251641273 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f24c80046c0 con 0x7f24d40ffae0 2024-03-28T22:19:35.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.082+0000 7f24d1ffb640 1 --2- 172.21.15.16:0/3251641273 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f24c40a1f60 0x7f24c40a4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.082+0000 7f24d1ffb640 1 -- 172.21.15.16:0/3251641273 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7f24c80ce550 con 0x7f24d40ffae0 2024-03-28T22:19:35.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.082+0000 7f24d3fff640 1 --2- 172.21.15.16:0/3251641273 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f24c40a1f60 0x7f24c40a4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.083+0000 7f24d3fff640 1 --2- 172.21.15.16:0/3251641273 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f24c40a1f60 0x7f24c40a4420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f24d40ff030 tx=0x7f24c0005f90 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:35.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.087+0000 7f24d1ffb640 1 -- 172.21.15.16:0/3251641273 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f24c809b640 con 0x7f24d40ffae0 2024-03-28T22:19:35.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.308+0000 7f24d9fa0640 1 -- 172.21.15.16:0/3251641273 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f24d4199200 con 0x7f24d40ffae0 2024-03-28T22:19:35.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.309+0000 7f24d1ffb640 1 -- 172.21.15.16:0/3251641273 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11851 (secure 0 0 0) 0x7f24c809af40 con 0x7f24d40ffae0 2024-03-28T22:19:35.309 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:19:35.310 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/1644581227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1644581227},{"type":"v1","addr":"172.21.15.16:6839","nonce":1644581227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/2593062669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2593062669},{"type":"v1","addr":"172.21.15.16:6841","nonce":2593062669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:08:29.672976+0000","modified":"2024-03-28T22:14:44.922044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24566,"mds_1":24524,"mds_2":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24466":{"gid":24466,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/247249857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":247249857},{"type":"v1","addr":"172.21.15.63:6835","nonce":247249857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6837/2324231032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":2324231032},{"type":"v1","addr":"172.21.15.63:6837","nonce":2324231032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6839/1177197077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":1177197077},{"type":"v1","addr":"172.21.15.63:6839","nonce":1177197077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"e","rank":2,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.53:6835/350727634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":350727634},{"type":"v1","addr":"172.21.15.53:6835","nonce":350727634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/349877573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":349877573},{"type":"v1","addr":"172.21.15.53:6837","nonce":349877573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6839/3714153956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":3714153956},{"type":"v1","addr":"172.21.15.53:6839","nonce":3714153956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24566,"qdb_cluster":[24500,24524,24566]},"id":1}]} 2024-03-28T22:19:35.310 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2024-03-28T22:19:35.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.311+0000 7f24d9fa0640 1 -- 172.21.15.16:0/3251641273 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f24c40a1f60 msgr2=0x7f24c40a4420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:35.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.311+0000 7f24d9fa0640 1 --2- 172.21.15.16:0/3251641273 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f24c40a1f60 0x7f24c40a4420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f24d40ff030 tx=0x7f24c0005f90 comp rx=0 tx=0).stop 2024-03-28T22:19:35.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.311+0000 7f24d9fa0640 1 -- 172.21.15.16:0/3251641273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24d40ffae0 msgr2=0x7f24d4197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:35.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.311+0000 7f24d9fa0640 1 --2- 172.21.15.16:0/3251641273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24d40ffae0 0x7f24d4197eb0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f24c800e9f0 tx=0x7f24c800edf0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.311+0000 7f24d9fa0640 1 -- 172.21.15.16:0/3251641273 shutdown_connections 2024-03-28T22:19:35.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.311+0000 7f24d9fa0640 1 --2- 172.21.15.16:0/3251641273 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f24c40a1f60 0x7f24c40a4420 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.311+0000 7f24d9fa0640 1 --2- 172.21.15.16:0/3251641273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24d4107600 0x7f24d419c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.311+0000 7f24d9fa0640 1 --2- 172.21.15.16:0/3251641273 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f24d41004a0 0x7f24d41983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.311+0000 7f24d9fa0640 1 --2- 172.21.15.16:0/3251641273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24d40ffae0 0x7f24d4197eb0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.311+0000 7f24d9fa0640 1 -- 172.21.15.16:0/3251641273 >> 172.21.15.16:0/3251641273 conn(0x7f24d4076a60 msgr2=0x7f24d41043f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:35.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.311+0000 7f24d9fa0640 1 -- 172.21.15.16:0/3251641273 shutdown_connections 2024-03-28T22:19:35.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.311+0000 7f24d9fa0640 1 -- 172.21.15.16:0/3251641273 wait complete. 2024-03-28T22:19:35.322 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:19:35.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.427+0000 7f30f95b5640 1 -- 172.21.15.16:0/1818296857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30f4106690 msgr2=0x7f30f4106a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:35.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.428+0000 7f30f95b5640 1 --2- 172.21.15.16:0/1818296857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30f4106690 0x7f30f4106a70 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f30ec009930 tx=0x7f30ec02bc30 comp rx=0 tx=0).stop 2024-03-28T22:19:35.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.428+0000 7f30f95b5640 1 -- 172.21.15.16:0/1818296857 shutdown_connections 2024-03-28T22:19:35.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.428+0000 7f30f95b5640 1 --2- 172.21.15.16:0/1818296857 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30f4100fb0 0x7f30f410d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.428+0000 7f30f95b5640 1 --2- 172.21.15.16:0/1818296857 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30f41005f0 0x7f30f4100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.428+0000 7f30f95b5640 1 --2- 172.21.15.16:0/1818296857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30f4106690 0x7f30f4106a70 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.428+0000 7f30f95b5640 1 -- 172.21.15.16:0/1818296857 >> 172.21.15.16:0/1818296857 conn(0x7f30f40fc0a0 msgr2=0x7f30f40fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:35.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.428+0000 7f30f95b5640 1 -- 172.21.15.16:0/1818296857 shutdown_connections 2024-03-28T22:19:35.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.428+0000 7f30f95b5640 1 -- 172.21.15.16:0/1818296857 wait complete. 2024-03-28T22:19:35.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.428+0000 7f30f95b5640 1 Processor -- start 2024-03-28T22:19:35.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.428+0000 7f30f95b5640 1 -- start start 2024-03-28T22:19:35.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.429+0000 7f30f95b5640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30f41005f0 0x7f30f4197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.429+0000 7f30f95b5640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30f4100fb0 0x7f30f41984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.429+0000 7f30f95b5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30f4106690 0x7f30f419c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.429+0000 7f30f95b5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f30f4111b10 con 0x7f30f4106690 2024-03-28T22:19:35.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.429+0000 7f30f95b5640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f30f4111990 con 0x7f30f4100fb0 2024-03-28T22:19:35.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.429+0000 7f30f95b5640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f30f4111c90 con 0x7f30f41005f0 2024-03-28T22:19:35.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.429+0000 7f30f37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30f4106690 0x7f30f419c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.429+0000 7f30f37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30f4106690 0x7f30f419c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57904/0 (socket says 172.21.15.16:57904) 2024-03-28T22:19:35.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.429+0000 7f30f37fe640 1 -- 172.21.15.16:0/2031668414 learned_addr learned my addr 172.21.15.16:0/2031668414 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:35.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.429+0000 7f30f27fc640 1 --2- 172.21.15.16:0/2031668414 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30f4100fb0 0x7f30f41984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.429+0000 7f30f2ffd640 1 --2- 172.21.15.16:0/2031668414 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30f41005f0 0x7f30f4197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.429+0000 7f30f37fe640 1 -- 172.21.15.16:0/2031668414 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30f41005f0 msgr2=0x7f30f4197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:35.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.429+0000 7f30f37fe640 1 --2- 172.21.15.16:0/2031668414 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30f41005f0 0x7f30f4197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.429+0000 7f30f37fe640 1 -- 172.21.15.16:0/2031668414 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30f4100fb0 msgr2=0x7f30f41984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:35.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.429+0000 7f30f37fe640 1 --2- 172.21.15.16:0/2031668414 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30f4100fb0 0x7f30f41984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.429+0000 7f30f37fe640 1 -- 172.21.15.16:0/2031668414 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30f419cfa0 con 0x7f30f4106690 2024-03-28T22:19:35.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.429+0000 7f30f37fe640 1 --2- 172.21.15.16:0/2031668414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30f4106690 0x7f30f419c870 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f30e400eb10 tx=0x7f30e400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:35.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.429+0000 7f30dbfff640 1 -- 172.21.15.16:0/2031668414 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e4004b20 con 0x7f30f4106690 2024-03-28T22:19:35.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.429+0000 7f30dbfff640 1 -- 172.21.15.16:0/2031668414 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30e4004cc0 con 0x7f30f4106690 2024-03-28T22:19:35.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.429+0000 7f30dbfff640 1 -- 172.21.15.16:0/2031668414 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e4010760 con 0x7f30f4106690 2024-03-28T22:19:35.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.429+0000 7f30f95b5640 1 -- 172.21.15.16:0/2031668414 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30f419d290 con 0x7f30f4106690 2024-03-28T22:19:35.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.429+0000 7f30f95b5640 1 -- 172.21.15.16:0/2031668414 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30f41a4b20 con 0x7f30f4106690 2024-03-28T22:19:35.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.431+0000 7f30dbfff640 1 -- 172.21.15.16:0/2031668414 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f30e4005020 con 0x7f30f4106690 2024-03-28T22:19:35.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.432+0000 7f30fc198740 1 Processor -- start 2024-03-28T22:19:35.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.432+0000 7f30fc198740 1 -- start start 2024-03-28T22:19:35.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.432+0000 7f30fc198740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6449d6120 0x55b6449d6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.432+0000 7f30fc198740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6449d6a40 0x55b6449e3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.432+0000 7f30fc198740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b6449e3d80 0x55b6449e6170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.432+0000 7f30fc198740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b64493a1b0 con 0x55b6449d6a40 2024-03-28T22:19:35.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.432+0000 7f30fc198740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55b6448e2c00 con 0x55b6449d6120 2024-03-28T22:19:35.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.432+0000 7f30fc198740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55b644936250 con 0x55b6449e3d80 2024-03-28T22:19:35.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.432+0000 7f30dbfff640 1 --2- 172.21.15.16:0/2031668414 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f30dc0a1f70 0x7f30dc0a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.432+0000 7f30f2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6449d6a40 0x55b6449e3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.432+0000 7f30f2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6449d6a40 0x55b6449e3840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57912/0 (socket says 172.21.15.16:57912) 2024-03-28T22:19:35.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.432+0000 7f30f2ffd640 1 -- 172.21.15.16:0/1006067322 learned_addr learned my addr 172.21.15.16:0/1006067322 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:35.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.432+0000 7f30f27fc640 1 --2- 172.21.15.16:0/1006067322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6449d6120 0x55b6449d6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.432+0000 7f30f37fe640 1 --2- 172.21.15.16:0/2031668414 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f30dc0a1f70 0x7f30dc0a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.432+0000 7f30f27fc640 1 --2- 172.21.15.16:0/1006067322 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b6449e3d80 0x55b6449e6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.432+0000 7f30dbfff640 1 -- 172.21.15.16:0/2031668414 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7f30e40ce8c0 con 0x7f30f4106690 2024-03-28T22:19:35.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.432+0000 7f30f27fc640 1 -- 172.21.15.16:0/1006067322 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b6449e3d80 msgr2=0x55b6449e6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:35.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.432+0000 7f30f27fc640 1 --2- 172.21.15.16:0/1006067322 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b6449e3d80 0x55b6449e6170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.432+0000 7f30f27fc640 1 -- 172.21.15.16:0/1006067322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6449d6a40 msgr2=0x55b6449e3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:35.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.432+0000 7f30f27fc640 1 --2- 172.21.15.16:0/1006067322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6449d6a40 0x55b6449e3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.432+0000 7f30f27fc640 1 -- 172.21.15.16:0/1006067322 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6449023d0 con 0x55b6449d6120 2024-03-28T22:19:35.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.432+0000 7f30f2ffd640 1 --2- 172.21.15.16:0/1006067322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6449d6a40 0x55b6449e3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:19:35.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.433+0000 7f30f27fc640 1 --2- 172.21.15.16:0/1006067322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6449d6120 0x55b6449d6500 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f30f4105d20 tx=0x7f30e80096c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c50048c017542c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:35.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.433+0000 7f30f37fe640 1 --2- 172.21.15.16:0/2031668414 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f30dc0a1f70 0x7f30dc0a4430 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f30e409b240 tx=0x7f30e406b110 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:35.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.433+0000 7f30d97fa640 1 -- 172.21.15.16:0/1006067322 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e8004250 con 0x55b6449d6120 2024-03-28T22:19:35.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.433+0000 7f30d97fa640 1 -- 172.21.15.16:0/1006067322 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30e8011e50 con 0x55b6449d6120 2024-03-28T22:19:35.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.433+0000 7f30d97fa640 1 -- 172.21.15.16:0/1006067322 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e801b830 con 0x55b6449d6120 2024-03-28T22:19:35.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.433+0000 7f30fc198740 1 -- 172.21.15.16:0/1006067322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6449d6120 msgr2=0x55b6449d6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:35.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.433+0000 7f30fc198740 1 --2- 172.21.15.16:0/1006067322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6449d6120 0x55b6449d6500 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f30f4105d20 tx=0x7f30e80096c0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.433+0000 7f30fc198740 1 -- 172.21.15.16:0/1006067322 shutdown_connections 2024-03-28T22:19:35.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.433+0000 7f30fc198740 1 --2- 172.21.15.16:0/1006067322 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b6449e3d80 0x55b6449e6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.433+0000 7f30fc198740 1 --2- 172.21.15.16:0/1006067322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6449d6a40 0x55b6449e3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.433+0000 7f30fc198740 1 --2- 172.21.15.16:0/1006067322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6449d6120 0x55b6449d6500 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.433+0000 7f30fc198740 1 -- 172.21.15.16:0/1006067322 >> 172.21.15.16:0/1006067322 conn(0x55b6449d0ba0 msgr2=0x55b6449d0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:35.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.434+0000 7f30fc198740 1 -- 172.21.15.16:0/1006067322 shutdown_connections 2024-03-28T22:19:35.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.434+0000 7f30fc198740 1 -- 172.21.15.16:0/1006067322 wait complete. 2024-03-28T22:19:35.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.434+0000 7f30fc198740 1 -- start start 2024-03-28T22:19:35.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.434+0000 7f30fc198740 1 Processor -- start 2024-03-28T22:19:35.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.434+0000 7f30fc198740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6449d6120 0x55b6449e0d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.434+0000 7f30fc198740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6449d6a40 0x55b6449e12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.434+0000 7f30fc198740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b6449e3d80 0x55b6449e1800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.434+0000 7f30fc198740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b6449e6bc0 con 0x55b6449d6120 2024-03-28T22:19:35.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.434+0000 7f30fc198740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55b6448d8470 con 0x55b6449d6a40 2024-03-28T22:19:35.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.434+0000 7f30fc198740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55b6449e6d40 con 0x55b6449e3d80 2024-03-28T22:19:35.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.434+0000 7f30fc198740 10 client.0 _ll_register_callbacks cb 0x55b6448f5cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:19:35.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.434+0000 7f30f27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6449d6120 0x55b6449e0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.434+0000 7f30f27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6449d6120 0x55b6449e0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57928/0 (socket says 172.21.15.16:57928) 2024-03-28T22:19:35.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.434+0000 7f30f27fc640 1 -- 172.21.15.16:0/931258482 learned_addr learned my addr 172.21.15.16:0/931258482 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:35.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.434+0000 7f30f2ffd640 1 --2- 172.21.15.16:0/931258482 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6449d6a40 0x55b6449e12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.434+0000 7f30f27fc640 1 --2- 172.21.15.16:0/931258482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b6449e3d80 0x55b6449e1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.434+0000 7f30f27fc640 1 -- 172.21.15.16:0/931258482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b6449e3d80 msgr2=0x55b6449e1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:35.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.434+0000 7f30f27fc640 1 --2- 172.21.15.16:0/931258482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b6449e3d80 0x55b6449e1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.434+0000 7f30f27fc640 1 -- 172.21.15.16:0/931258482 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6449d6a40 msgr2=0x55b6449e12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:35.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.434+0000 7f30f27fc640 1 --2- 172.21.15.16:0/931258482 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6449d6a40 0x55b6449e12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.434+0000 7f30f27fc640 1 -- 172.21.15.16:0/931258482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b64464e3f0 con 0x55b6449d6120 2024-03-28T22:19:35.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.435+0000 7f30f27fc640 1 --2- 172.21.15.16:0/931258482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6449d6120 0x55b6449e0d80 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f30e8009730 tx=0x7f30e801b510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe605c86d737ab16 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:35.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.435+0000 7f30d6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:19:35.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.435+0000 7f30d6ff5640 1 -- 172.21.15.16:0/931258482 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e8005030 con 0x55b6449d6120 2024-03-28T22:19:35.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.435+0000 7f30d6ff5640 1 -- 172.21.15.16:0/931258482 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7f30e80051d0 con 0x55b6449d6120 2024-03-28T22:19:35.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.435+0000 7f30fc198740 1 -- 172.21.15.16:0/931258482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30e8011e50 con 0x55b6449d6120 2024-03-28T22:19:35.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.435+0000 7f30d6ff5640 1 -- 172.21.15.16:0/931258482 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f30e80053d0 con 0x55b6449d6120 2024-03-28T22:19:35.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.435+0000 7f30fc198740 10 client.17382 fetch_fsmap learned FSMap version 27 2024-03-28T22:19:35.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.435+0000 7f30fc198740 1 -- 172.21.15.16:0/931258482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b6447187d0 con 0x55b6449d6120 2024-03-28T22:19:35.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.435+0000 7f30d6ff5640 1 -- 172.21.15.16:0/931258482 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30e8011e50 con 0x55b6449d6120 2024-03-28T22:19:35.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.435+0000 7f30d6ff5640 1 -- 172.21.15.16:0/931258482 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f30e8003890 con 0x55b6449d6120 2024-03-28T22:19:35.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.435+0000 7f30fc198740 10 client.17382 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:19:35.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.435+0000 7f30fc198740 10 client.17382 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-28T22:19:35.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.435+0000 7f30fc198740 20 client.17382 populate_metadata read hostname 'smithi016' 2024-03-28T22:19:35.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.435+0000 7f30fc198740 1 --2- 172.21.15.16:0/931258482 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55b6449f3e20 0x55b6449f6210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.435+0000 7f30fc198740 4 client.17382 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:19:35.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.435+0000 7f30fc198740 1 -- 172.21.15.16:0/931258482 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b6449476f0 con 0x55b6449f3e20 2024-03-28T22:19:35.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.436+0000 7f30f2ffd640 1 --2- 172.21.15.16:0/931258482 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55b6449f3e20 0x55b6449f6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.436+0000 7f30d6ff5640 10 client.17382 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:19:35.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.436+0000 7f30f2ffd640 1 --2- 172.21.15.16:0/931258482 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55b6449f3e20 0x55b6449f6210 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d0961cef9c36522d server_cookie=7aa6a2b7b970f73e in_seq=0 out_seq=0 2024-03-28T22:19:35.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.436+0000 7f30d6ff5640 1 -- 172.21.15.16:0/931258482 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b6449476f0 con 0x55b6449f3e20 2024-03-28T22:19:35.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.436+0000 7f30d6ff5640 10 client.17382 handle_command_reply: tid=0 2024-03-28T22:19:35.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.437+0000 7f30fc198740 1 client.17382 shutdown 2024-03-28T22:19:35.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.437+0000 7f30fc198740 1 -- 172.21.15.16:0/931258482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6449d6120 msgr2=0x55b6449e0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:35.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.437+0000 7f30fc198740 1 --2- 172.21.15.16:0/931258482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6449d6120 0x55b6449e0d80 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f30e8009730 tx=0x7f30e801b510 comp rx=0 tx=0).stop 2024-03-28T22:19:35.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.437+0000 7f30fc198740 1 -- 172.21.15.16:0/931258482 shutdown_connections 2024-03-28T22:19:35.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.437+0000 7f30fc198740 1 --2- 172.21.15.16:0/931258482 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55b6449f3e20 0x55b6449f6210 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.437+0000 7f30fc198740 1 --2- 172.21.15.16:0/931258482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b6449e3d80 0x55b6449e1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.437+0000 7f30fc198740 1 --2- 172.21.15.16:0/931258482 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6449d6a40 0x55b6449e12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.437+0000 7f30d6ff5640 0 client.17382 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:19:35.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.437+0000 7f30fc198740 1 --2- 172.21.15.16:0/931258482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6449d6120 0x55b6449e0d80 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.437+0000 7f30fc198740 1 -- 172.21.15.16:0/931258482 >> 172.21.15.16:0/931258482 conn(0x55b6449d0ba0 msgr2=0x55b6449d28f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:35.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.437+0000 7f30fc198740 1 -- 172.21.15.16:0/931258482 shutdown_connections 2024-03-28T22:19:35.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.437+0000 7f30fc198740 1 -- 172.21.15.16:0/931258482 wait complete. 2024-03-28T22:19:35.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.437+0000 7f30fc198740 20 client.17382 trim_cache size 0 max 16384 2024-03-28T22:19:35.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.450+0000 7f30fc198740 1 Processor -- start 2024-03-28T22:19:35.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.450+0000 7f30fc198740 1 -- start start 2024-03-28T22:19:35.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.450+0000 7f30fc198740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b6449d6120 0x55b6449e34b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.450+0000 7f30fc198740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6449d6a40 0x55b6449efa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.450+0000 7f30fc198740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6449e3d80 0x55b6449fb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.450+0000 7f30fc198740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f30f41adbf0 con 0x55b6449d6a40 2024-03-28T22:19:35.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.450+0000 7f30fc198740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f30f41ab4c0 con 0x55b6449e3d80 2024-03-28T22:19:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.450+0000 7f30fc198740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f30f41a9330 con 0x55b6449d6120 2024-03-28T22:19:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.451+0000 7f30f2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6449d6a40 0x55b6449efa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.451+0000 7f30f2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6449d6a40 0x55b6449efa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57940/0 (socket says 172.21.15.16:57940) 2024-03-28T22:19:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.451+0000 7f30f2ffd640 1 -- 172.21.15.16:0/2205055562 learned_addr learned my addr 172.21.15.16:0/2205055562 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.451+0000 7f30f27fc640 1 --2- 172.21.15.16:0/2205055562 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6449e3d80 0x55b6449fb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.451+0000 7f30f2ffd640 1 -- 172.21.15.16:0/2205055562 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b6449d6120 msgr2=0x55b6449e34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.451+0000 7f30f2ffd640 1 --2- 172.21.15.16:0/2205055562 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b6449d6120 0x55b6449e34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.451+0000 7f30f2ffd640 1 -- 172.21.15.16:0/2205055562 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6449e3d80 msgr2=0x55b6449fb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.451+0000 7f30f2ffd640 1 --2- 172.21.15.16:0/2205055562 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6449e3d80 0x55b6449fb980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.451+0000 7f30f2ffd640 1 -- 172.21.15.16:0/2205055562 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6449df2d0 con 0x55b6449d6a40 2024-03-28T22:19:35.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.451+0000 7f30f2ffd640 1 --2- 172.21.15.16:0/2205055562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6449d6a40 0x55b6449efa00 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f30ec036040 tx=0x7f30ec02e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cb35dcbf4d9d63e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:35.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.451+0000 7f30d8ff9640 1 -- 172.21.15.16:0/2205055562 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30ec045a30 con 0x55b6449d6a40 2024-03-28T22:19:35.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.451+0000 7f30d8ff9640 1 -- 172.21.15.16:0/2205055562 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30ec04a070 con 0x55b6449d6a40 2024-03-28T22:19:35.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.451+0000 7f30d8ff9640 1 -- 172.21.15.16:0/2205055562 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30ec0452d0 con 0x55b6449d6a40 2024-03-28T22:19:35.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.452+0000 7f30fc198740 1 -- 172.21.15.16:0/2205055562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6449d6a40 msgr2=0x55b6449efa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:35.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.452+0000 7f30fc198740 1 --2- 172.21.15.16:0/2205055562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6449d6a40 0x55b6449efa00 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f30ec036040 tx=0x7f30ec02e040 comp rx=0 tx=0).stop 2024-03-28T22:19:35.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.452+0000 7f30fc198740 1 -- 172.21.15.16:0/2205055562 shutdown_connections 2024-03-28T22:19:35.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.452+0000 7f30fc198740 1 --2- 172.21.15.16:0/2205055562 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6449e3d80 0x55b6449fb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.452+0000 7f30fc198740 1 --2- 172.21.15.16:0/2205055562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6449d6a40 0x55b6449efa00 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.452+0000 7f30fc198740 1 --2- 172.21.15.16:0/2205055562 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b6449d6120 0x55b6449e34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.452+0000 7f30fc198740 1 -- 172.21.15.16:0/2205055562 >> 172.21.15.16:0/2205055562 conn(0x55b6449d0ba0 msgr2=0x55b6449d2880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:35.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.452+0000 7f30fc198740 1 -- 172.21.15.16:0/2205055562 shutdown_connections 2024-03-28T22:19:35.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.452+0000 7f30fc198740 1 -- 172.21.15.16:0/2205055562 wait complete. 2024-03-28T22:19:35.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.452+0000 7f30fc198740 1 -- start start 2024-03-28T22:19:35.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.452+0000 7f30fc198740 1 Processor -- start 2024-03-28T22:19:35.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.452+0000 7f30fc198740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6449d6120 0x55b6449e7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.452+0000 7f30fc198740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b6449d6a40 0x55b6449e7f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.452+0000 7f30fc198740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6449e3d80 0x55b6449e8480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.452+0000 7f30fc198740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f30f41a9980 con 0x55b6449d6120 2024-03-28T22:19:35.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.452+0000 7f30fc198740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f30f41aadf0 con 0x55b6449e3d80 2024-03-28T22:19:35.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.452+0000 7f30fc198740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f30f41aa050 con 0x55b6449d6a40 2024-03-28T22:19:35.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.452+0000 7f30fc198740 10 client.0 _ll_register_callbacks cb 0x7f30e8004180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:19:35.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.452+0000 7f30f27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6449d6120 0x55b6449e7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.452+0000 7f30f27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6449d6120 0x55b6449e7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57956/0 (socket says 172.21.15.16:57956) 2024-03-28T22:19:35.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.452+0000 7f30f27fc640 1 -- 172.21.15.16:0/2300602510 learned_addr learned my addr 172.21.15.16:0/2300602510 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:35.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.453+0000 7f30f2ffd640 1 --2- 172.21.15.16:0/2300602510 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b6449d6a40 0x55b6449e7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.453+0000 7f30f27fc640 1 --2- 172.21.15.16:0/2300602510 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6449e3d80 0x55b6449e8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.453+0000 7f30f27fc640 1 -- 172.21.15.16:0/2300602510 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b6449d6a40 msgr2=0x55b6449e7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:35.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.453+0000 7f30f27fc640 1 --2- 172.21.15.16:0/2300602510 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b6449d6a40 0x55b6449e7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.453+0000 7f30f27fc640 1 -- 172.21.15.16:0/2300602510 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6449e3d80 msgr2=0x55b6449e8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:35.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.453+0000 7f30f27fc640 1 --2- 172.21.15.16:0/2300602510 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6449e3d80 0x55b6449e8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.453+0000 7f30f27fc640 1 -- 172.21.15.16:0/2300602510 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b644a59080 con 0x55b6449d6120 2024-03-28T22:19:35.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.453+0000 7f30f27fc640 1 --2- 172.21.15.16:0/2300602510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6449d6120 0x55b6449e7a00 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f30e8003c60 tx=0x7f30e801b410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd5f8b49a1e8f76d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:35.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.453+0000 7f30d6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:19:35.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.453+0000 7f30d6ff5640 1 -- 172.21.15.16:0/2300602510 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e80035f0 con 0x55b6449d6120 2024-03-28T22:19:35.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.453+0000 7f30d6ff5640 1 -- 172.21.15.16:0/2300602510 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7f30e8003790 con 0x55b6449d6120 2024-03-28T22:19:35.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.453+0000 7f30fc198740 1 -- 172.21.15.16:0/2300602510 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30ec04a070 con 0x55b6449d6120 2024-03-28T22:19:35.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.453+0000 7f30d6ff5640 1 -- 172.21.15.16:0/2300602510 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f30e80039b0 con 0x55b6449d6120 2024-03-28T22:19:35.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.453+0000 7f30fc198740 10 client.17388 fetch_fsmap learned FSMap version 27 2024-03-28T22:19:35.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.453+0000 7f30fc198740 1 -- 172.21.15.16:0/2300602510 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b644a5a380 con 0x55b6449d6120 2024-03-28T22:19:35.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.453+0000 7f30d6ff5640 1 -- 172.21.15.16:0/2300602510 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30ec04a070 con 0x55b6449d6120 2024-03-28T22:19:35.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.453+0000 7f30d6ff5640 1 -- 172.21.15.16:0/2300602510 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f30e801c6a0 con 0x55b6449d6120 2024-03-28T22:19:35.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.454+0000 7f30fc198740 10 client.17388 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:19:35.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.454+0000 7f30fc198740 10 client.17388 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-28T22:19:35.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.454+0000 7f30fc198740 20 client.17388 populate_metadata read hostname 'smithi016' 2024-03-28T22:19:35.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.454+0000 7f30fc198740 1 --2- 172.21.15.16:0/2300602510 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55b644a5aa70 0x55b644a5ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.454+0000 7f30fc198740 4 client.17388 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:19:35.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.454+0000 7f30fc198740 1 -- 172.21.15.16:0/2300602510 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b644941a40 con 0x55b644a5aa70 2024-03-28T22:19:35.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.454+0000 7f30f2ffd640 1 --2- 172.21.15.16:0/2300602510 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55b644a5aa70 0x55b644a5ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.454+0000 7f30f2ffd640 1 --2- 172.21.15.16:0/2300602510 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55b644a5aa70 0x55b644a5ce90 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cf4781f5822f9692 server_cookie=3d46635d3b3253ae in_seq=0 out_seq=0 2024-03-28T22:19:35.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.454+0000 7f30d6ff5640 10 client.17388 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:19:35.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.454+0000 7f30d6ff5640 1 -- 172.21.15.16:0/2300602510 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b644941a40 con 0x55b644a5aa70 2024-03-28T22:19:35.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.454+0000 7f30d6ff5640 10 client.17388 handle_command_reply: tid=0 2024-03-28T22:19:35.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.454+0000 7f30fc198740 1 client.17388 shutdown 2024-03-28T22:19:35.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.454+0000 7f30fc198740 1 -- 172.21.15.16:0/2300602510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6449d6120 msgr2=0x55b6449e7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:35.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.454+0000 7f30fc198740 1 --2- 172.21.15.16:0/2300602510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6449d6120 0x55b6449e7a00 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f30e8003c60 tx=0x7f30e801b410 comp rx=0 tx=0).stop 2024-03-28T22:19:35.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.454+0000 7f30fc198740 1 -- 172.21.15.16:0/2300602510 shutdown_connections 2024-03-28T22:19:35.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.454+0000 7f30fc198740 1 --2- 172.21.15.16:0/2300602510 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x55b644a5aa70 0x55b644a5ce90 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.454+0000 7f30fc198740 1 --2- 172.21.15.16:0/2300602510 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6449e3d80 0x55b6449e8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.454+0000 7f30fc198740 1 --2- 172.21.15.16:0/2300602510 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b6449d6a40 0x55b6449e7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.454+0000 7f30d6ff5640 0 client.17388 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:19:35.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.454+0000 7f30fc198740 1 --2- 172.21.15.16:0/2300602510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b6449d6120 0x55b6449e7a00 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.454+0000 7f30fc198740 1 -- 172.21.15.16:0/2300602510 >> 172.21.15.16:0/2300602510 conn(0x55b6449d0ba0 msgr2=0x55b6449e0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:35.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.455+0000 7f30fc198740 1 -- 172.21.15.16:0/2300602510 shutdown_connections 2024-03-28T22:19:35.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.455+0000 7f30fc198740 1 -- 172.21.15.16:0/2300602510 wait complete. 2024-03-28T22:19:35.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.455+0000 7f30fc198740 20 client.17388 trim_cache size 0 max 16384 2024-03-28T22:19:35.470 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:19:35.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.455+0000 7f30f95b5640 1 -- 172.21.15.16:0/2031668414 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f30dc0a1f70 msgr2=0x7f30dc0a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:35.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.455+0000 7f30f95b5640 1 --2- 172.21.15.16:0/2031668414 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f30dc0a1f70 0x7f30dc0a4430 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f30e409b240 tx=0x7f30e406b110 comp rx=0 tx=0).stop 2024-03-28T22:19:35.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.455+0000 7f30f95b5640 1 -- 172.21.15.16:0/2031668414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30f4106690 msgr2=0x7f30f419c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:35.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.455+0000 7f30f95b5640 1 --2- 172.21.15.16:0/2031668414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30f4106690 0x7f30f419c870 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f30e400eb10 tx=0x7f30e400c2d0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.455+0000 7f30f95b5640 1 -- 172.21.15.16:0/2031668414 shutdown_connections 2024-03-28T22:19:35.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.455+0000 7f30f95b5640 1 --2- 172.21.15.16:0/2031668414 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f30dc0a1f70 0x7f30dc0a4430 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.455+0000 7f30f95b5640 1 --2- 172.21.15.16:0/2031668414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30f4106690 0x7f30f419c870 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.455+0000 7f30f95b5640 1 --2- 172.21.15.16:0/2031668414 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30f4100fb0 0x7f30f41984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.455+0000 7f30f95b5640 1 --2- 172.21.15.16:0/2031668414 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30f41005f0 0x7f30f4197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.455+0000 7f30f95b5640 1 -- 172.21.15.16:0/2031668414 >> 172.21.15.16:0/2031668414 conn(0x7f30f40fc0a0 msgr2=0x7f30f40fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:35.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.455+0000 7f30f95b5640 1 -- 172.21.15.16:0/2031668414 shutdown_connections 2024-03-28T22:19:35.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.455+0000 7f30f95b5640 1 -- 172.21.15.16:0/2031668414 wait complete. 2024-03-28T22:19:35.473 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:19:35.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.577+0000 7f5437e9d640 1 -- 172.21.15.16:0/3776808405 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f54301020a0 msgr2=0x7f543010e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:35.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.577+0000 7f5437e9d640 1 --2- 172.21.15.16:0/3776808405 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f54301020a0 0x7f543010e5b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5428009ca0 tx=0x7f542802c0f0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.577+0000 7f5437e9d640 1 -- 172.21.15.16:0/3776808405 shutdown_connections 2024-03-28T22:19:35.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.577+0000 7f5437e9d640 1 --2- 172.21.15.16:0/3776808405 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f54301020a0 0x7f543010e5b0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.577+0000 7f5437e9d640 1 --2- 172.21.15.16:0/3776808405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54301016e0 0x7f5430101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.577+0000 7f5437e9d640 1 --2- 172.21.15.16:0/3776808405 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54301076f0 0x7f5430107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.577+0000 7f5437e9d640 1 -- 172.21.15.16:0/3776808405 >> 172.21.15.16:0/3776808405 conn(0x7f54300fd1f0 msgr2=0x7f54300ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:35.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.577+0000 7f5437e9d640 1 -- 172.21.15.16:0/3776808405 shutdown_connections 2024-03-28T22:19:35.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.577+0000 7f5437e9d640 1 -- 172.21.15.16:0/3776808405 wait complete. 2024-03-28T22:19:35.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.578+0000 7f5437e9d640 1 Processor -- start 2024-03-28T22:19:35.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.578+0000 7f5437e9d640 1 -- start start 2024-03-28T22:19:35.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.578+0000 7f5437e9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54301016e0 0x7f5430197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.578+0000 7f5437e9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f54301020a0 0x7f54301984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.578+0000 7f5437e9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54301076f0 0x7f543019c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.578+0000 7f5437e9d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5430111b00 con 0x7f54301076f0 2024-03-28T22:19:35.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.578+0000 7f5437e9d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5430111980 con 0x7f54301020a0 2024-03-28T22:19:35.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.578+0000 7f5437e9d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5430111c80 con 0x7f54301016e0 2024-03-28T22:19:35.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.578+0000 7f543643a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54301076f0 0x7f543019c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.578+0000 7f543643a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54301076f0 0x7f543019c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57978/0 (socket says 172.21.15.16:57978) 2024-03-28T22:19:35.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.578+0000 7f5435438640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f54301020a0 0x7f54301984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.578+0000 7f543643a640 1 -- 172.21.15.16:0/1372777618 learned_addr learned my addr 172.21.15.16:0/1372777618 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:35.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.578+0000 7f5435c39640 1 --2- 172.21.15.16:0/1372777618 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54301016e0 0x7f5430197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.578+0000 7f543643a640 1 -- 172.21.15.16:0/1372777618 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54301016e0 msgr2=0x7f5430197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:35.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.578+0000 7f543643a640 1 --2- 172.21.15.16:0/1372777618 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54301016e0 0x7f5430197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.578+0000 7f543643a640 1 -- 172.21.15.16:0/1372777618 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f54301020a0 msgr2=0x7f54301984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:35.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.578+0000 7f543643a640 1 --2- 172.21.15.16:0/1372777618 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f54301020a0 0x7f54301984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.578+0000 7f543643a640 1 -- 172.21.15.16:0/1372777618 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f543019cf70 con 0x7f54301076f0 2024-03-28T22:19:35.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.578+0000 7f5435c39640 1 --2- 172.21.15.16:0/1372777618 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54301016e0 0x7f5430197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:19:35.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.578+0000 7f5435438640 1 --2- 172.21.15.16:0/1372777618 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f54301020a0 0x7f54301984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:19:35.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.579+0000 7f543643a640 1 --2- 172.21.15.16:0/1372777618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54301076f0 0x7f543019c840 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f5428005e30 tx=0x7f5428032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:35.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.579+0000 7f5426ffd640 1 -- 172.21.15.16:0/1372777618 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5428042020 con 0x7f54301076f0 2024-03-28T22:19:35.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.579+0000 7f5437e9d640 1 -- 172.21.15.16:0/1372777618 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f543019d200 con 0x7f54301076f0 2024-03-28T22:19:35.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.579+0000 7f5426ffd640 1 -- 172.21.15.16:0/1372777618 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5428034640 con 0x7f54301076f0 2024-03-28T22:19:35.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.579+0000 7f5426ffd640 1 -- 172.21.15.16:0/1372777618 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5428039040 con 0x7f54301076f0 2024-03-28T22:19:35.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.579+0000 7f5437e9d640 1 -- 172.21.15.16:0/1372777618 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54301a4af0 con 0x7f54301076f0 2024-03-28T22:19:35.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.581+0000 7f5426ffd640 1 -- 172.21.15.16:0/1372777618 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f5428045020 con 0x7f54301076f0 2024-03-28T22:19:35.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.581+0000 7f543aa82740 1 Processor -- start 2024-03-28T22:19:35.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.581+0000 7f543aa82740 1 -- start start 2024-03-28T22:19:35.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.582+0000 7f543aa82740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5649a39e0470 0x5649a39e0850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.582+0000 7f543aa82740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5649a39e0d90 0x5649a39e9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.582+0000 7f543aa82740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5649a39e9970 0x5649a39ebd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.582+0000 7f543aa82740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5649a39441b0 con 0x5649a39e9970 2024-03-28T22:19:35.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.582+0000 7f543aa82740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5649a38ecc00 con 0x5649a39e0d90 2024-03-28T22:19:35.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.582+0000 7f543aa82740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5649a3940250 con 0x5649a39e0470 2024-03-28T22:19:35.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.582+0000 7f5435438640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5649a39e9970 0x5649a39ebd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.582+0000 7f5435c39640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5649a39e0d90 0x5649a39e9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.582+0000 7f5435438640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5649a39e9970 0x5649a39ebd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57984/0 (socket says 172.21.15.16:57984) 2024-03-28T22:19:35.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.582+0000 7f5435c39640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5649a39e0d90 0x5649a39e9430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:45260/0 (socket says 172.21.15.16:45260) 2024-03-28T22:19:35.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.582+0000 7f5435438640 1 -- 172.21.15.16:0/3144617825 learned_addr learned my addr 172.21.15.16:0/3144617825 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:35.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.582+0000 7f5426ffd640 1 --2- 172.21.15.16:0/1372777618 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f541c0a1ef0 0x7f541c0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.582+0000 7f5435438640 1 --2- 172.21.15.16:0/3144617825 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5649a39e0470 0x5649a39e0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.582+0000 7f543643a640 1 --2- 172.21.15.16:0/1372777618 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f541c0a1ef0 0x7f541c0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.582+0000 7f5435c39640 1 -- 172.21.15.16:0/3144617825 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5649a39e0470 msgr2=0x5649a39e0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:35.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.582+0000 7f5435c39640 1 --2- 172.21.15.16:0/3144617825 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5649a39e0470 0x5649a39e0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.582+0000 7f5435c39640 1 -- 172.21.15.16:0/3144617825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5649a39e9970 msgr2=0x5649a39ebd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:35.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.582+0000 7f5435c39640 1 --2- 172.21.15.16:0/3144617825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5649a39e9970 0x5649a39ebd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.582+0000 7f5435c39640 1 -- 172.21.15.16:0/3144617825 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649a390c3d0 con 0x5649a39e0d90 2024-03-28T22:19:35.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.582+0000 7f5426ffd640 1 -- 172.21.15.16:0/1372777618 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7f54280f0200 con 0x7f54301076f0 2024-03-28T22:19:35.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.582+0000 7f5435c39640 1 --2- 172.21.15.16:0/3144617825 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5649a39e0d90 0x5649a39e9430 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f542c00dba0 tx=0x7f542c00f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24194a3c604cd1ab server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:35.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.582+0000 7f543643a640 1 --2- 172.21.15.16:0/1372777618 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f541c0a1ef0 0x7f541c0a43b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f54280edb10 tx=0x7f542808ecc0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:35.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.582+0000 7f541bfff640 1 -- 172.21.15.16:0/3144617825 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f542c00de90 con 0x5649a39e0d90 2024-03-28T22:19:35.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.583+0000 7f541bfff640 1 -- 172.21.15.16:0/3144617825 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f542c005500 con 0x5649a39e0d90 2024-03-28T22:19:35.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.583+0000 7f541bfff640 1 -- 172.21.15.16:0/3144617825 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f542c0116a0 con 0x5649a39e0d90 2024-03-28T22:19:35.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.583+0000 7f543aa82740 1 -- 172.21.15.16:0/3144617825 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5649a39e0d90 msgr2=0x5649a39e9430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:35.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.583+0000 7f543aa82740 1 --2- 172.21.15.16:0/3144617825 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5649a39e0d90 0x5649a39e9430 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f542c00dba0 tx=0x7f542c00f730 comp rx=0 tx=0).stop 2024-03-28T22:19:35.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.583+0000 7f543aa82740 1 -- 172.21.15.16:0/3144617825 shutdown_connections 2024-03-28T22:19:35.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.583+0000 7f543aa82740 1 --2- 172.21.15.16:0/3144617825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5649a39e9970 0x5649a39ebd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.583+0000 7f543aa82740 1 --2- 172.21.15.16:0/3144617825 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5649a39e0d90 0x5649a39e9430 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.583+0000 7f543aa82740 1 --2- 172.21.15.16:0/3144617825 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5649a39e0470 0x5649a39e0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.583+0000 7f543aa82740 1 -- 172.21.15.16:0/3144617825 >> 172.21.15.16:0/3144617825 conn(0x5649a39daba0 msgr2=0x5649a39daf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:35.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.583+0000 7f543aa82740 1 -- 172.21.15.16:0/3144617825 shutdown_connections 2024-03-28T22:19:35.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.583+0000 7f543aa82740 1 -- 172.21.15.16:0/3144617825 wait complete. 2024-03-28T22:19:35.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.584+0000 7f543aa82740 1 -- start start 2024-03-28T22:19:35.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.584+0000 7f543aa82740 1 Processor -- start 2024-03-28T22:19:35.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.584+0000 7f543aa82740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5649a39e0470 0x5649a39e6a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.584+0000 7f543aa82740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5649a39e0d90 0x5649a39e6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.584+0000 7f543aa82740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5649a39e9970 0x5649a39e7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.584+0000 7f543aa82740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5649a39ec760 con 0x5649a39e0d90 2024-03-28T22:19:35.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.584+0000 7f543aa82740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5649a38e2470 con 0x5649a39e0470 2024-03-28T22:19:35.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.584+0000 7f543aa82740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5649a39ec8e0 con 0x5649a39e9970 2024-03-28T22:19:35.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.584+0000 7f543aa82740 10 client.0 _ll_register_callbacks cb 0x5649a38ffcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:19:35.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.584+0000 7f5435c39640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5649a39e0d90 0x5649a39e6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.584+0000 7f5435c39640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5649a39e0d90 0x5649a39e6f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57988/0 (socket says 172.21.15.16:57988) 2024-03-28T22:19:35.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.584+0000 7f5435c39640 1 -- 172.21.15.16:0/3757418768 learned_addr learned my addr 172.21.15.16:0/3757418768 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:35.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.584+0000 7f5435438640 1 --2- 172.21.15.16:0/3757418768 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5649a39e0470 0x5649a39e6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.584+0000 7f5435438640 1 --2- 172.21.15.16:0/3757418768 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5649a39e9970 0x5649a39e7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.584+0000 7f5435c39640 1 -- 172.21.15.16:0/3757418768 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5649a39e9970 msgr2=0x5649a39e7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:35.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.584+0000 7f5435c39640 1 --2- 172.21.15.16:0/3757418768 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5649a39e9970 0x5649a39e7480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.584+0000 7f5435c39640 1 -- 172.21.15.16:0/3757418768 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5649a39e0470 msgr2=0x5649a39e6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:35.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.584+0000 7f5435c39640 1 --2- 172.21.15.16:0/3757418768 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5649a39e0470 0x5649a39e6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.584+0000 7f5435c39640 1 -- 172.21.15.16:0/3757418768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649a36583f0 con 0x5649a39e0d90 2024-03-28T22:19:35.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.585+0000 7f5435c39640 1 --2- 172.21.15.16:0/3757418768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5649a39e0d90 0x5649a39e6f40 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f542c000c00 tx=0x7f542c00d910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae4f68c30592d15b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:35.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.585+0000 7f54197fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:19:35.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.585+0000 7f54197fa640 1 -- 172.21.15.16:0/3757418768 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f542c02a060 con 0x5649a39e0d90 2024-03-28T22:19:35.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.585+0000 7f54197fa640 1 -- 172.21.15.16:0/3757418768 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7f542c00dc90 con 0x5649a39e0d90 2024-03-28T22:19:35.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.585+0000 7f543aa82740 1 -- 172.21.15.16:0/3757418768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f542c005500 con 0x5649a39e0d90 2024-03-28T22:19:35.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.585+0000 7f54197fa640 1 -- 172.21.15.16:0/3757418768 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f542c00de90 con 0x5649a39e0d90 2024-03-28T22:19:35.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.585+0000 7f543aa82740 10 client.17400 fetch_fsmap learned FSMap version 27 2024-03-28T22:19:35.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.585+0000 7f543aa82740 1 -- 172.21.15.16:0/3757418768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5649a37227d0 con 0x5649a39e0d90 2024-03-28T22:19:35.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.585+0000 7f54197fa640 1 -- 172.21.15.16:0/3757418768 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f542c005500 con 0x5649a39e0d90 2024-03-28T22:19:35.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.585+0000 7f54197fa640 1 -- 172.21.15.16:0/3757418768 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f542c011bd0 con 0x5649a39e0d90 2024-03-28T22:19:35.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.585+0000 7f543aa82740 10 client.17400 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:19:35.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.585+0000 7f543aa82740 10 client.17400 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T22:19:35.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.585+0000 7f543aa82740 20 client.17400 populate_metadata read hostname 'smithi016' 2024-03-28T22:19:35.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.585+0000 7f543aa82740 1 --2- 172.21.15.16:0/3757418768 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5649a39f99e0 0x5649a39fbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.585+0000 7f543aa82740 4 client.17400 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:19:35.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.585+0000 7f543aa82740 1 -- 172.21.15.16:0/3757418768 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5649a39516f0 con 0x5649a39f99e0 2024-03-28T22:19:35.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.586+0000 7f5435438640 1 --2- 172.21.15.16:0/3757418768 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5649a39f99e0 0x5649a39fbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.586+0000 7f54197fa640 10 client.17400 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:19:35.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.586+0000 7f5435438640 1 --2- 172.21.15.16:0/3757418768 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5649a39f99e0 0x5649a39fbdd0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2dad2bf1d18c3c1b server_cookie=1f3ed3e62e99193 in_seq=0 out_seq=0 2024-03-28T22:19:35.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.587+0000 7f54197fa640 1 -- 172.21.15.16:0/3757418768 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5649a39516f0 con 0x5649a39f99e0 2024-03-28T22:19:35.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.587+0000 7f54197fa640 10 client.17400 handle_command_reply: tid=0 2024-03-28T22:19:35.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.587+0000 7f543aa82740 1 client.17400 shutdown 2024-03-28T22:19:35.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.587+0000 7f543aa82740 1 -- 172.21.15.16:0/3757418768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5649a39e0d90 msgr2=0x5649a39e6f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:35.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.587+0000 7f543aa82740 1 --2- 172.21.15.16:0/3757418768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5649a39e0d90 0x5649a39e6f40 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f542c000c00 tx=0x7f542c00d910 comp rx=0 tx=0).stop 2024-03-28T22:19:35.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.587+0000 7f543aa82740 1 -- 172.21.15.16:0/3757418768 shutdown_connections 2024-03-28T22:19:35.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.587+0000 7f543aa82740 1 --2- 172.21.15.16:0/3757418768 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5649a39f99e0 0x5649a39fbdd0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.587+0000 7f543aa82740 1 --2- 172.21.15.16:0/3757418768 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5649a39e9970 0x5649a39e7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.587+0000 7f543aa82740 1 --2- 172.21.15.16:0/3757418768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5649a39e0d90 0x5649a39e6f40 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.587+0000 7f54197fa640 0 client.17400 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:19:35.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.587+0000 7f543aa82740 1 --2- 172.21.15.16:0/3757418768 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5649a39e0470 0x5649a39e6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.587+0000 7f543aa82740 1 -- 172.21.15.16:0/3757418768 >> 172.21.15.16:0/3757418768 conn(0x5649a39daba0 msgr2=0x5649a39dc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:35.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.587+0000 7f543aa82740 1 -- 172.21.15.16:0/3757418768 shutdown_connections 2024-03-28T22:19:35.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.587+0000 7f543aa82740 1 -- 172.21.15.16:0/3757418768 wait complete. 2024-03-28T22:19:35.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.587+0000 7f543aa82740 20 client.17400 trim_cache size 0 max 16384 2024-03-28T22:19:35.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.600+0000 7f543aa82740 1 Processor -- start 2024-03-28T22:19:35.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.601+0000 7f543aa82740 1 -- start start 2024-03-28T22:19:35.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.601+0000 7f543aa82740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5649a39e0470 0x5649a39e9130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.601+0000 7f543aa82740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5649a39e0d90 0x5649a39f5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.601+0000 7f543aa82740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5649a39e9970 0x5649a3a01560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.601+0000 7f543aa82740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f54301afbf0 con 0x5649a39e0d90 2024-03-28T22:19:35.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.601+0000 7f543aa82740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f54301ad4c0 con 0x5649a39e0470 2024-03-28T22:19:35.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.601+0000 7f543aa82740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f54301ab330 con 0x5649a39e9970 2024-03-28T22:19:35.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.601+0000 7f5435c39640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5649a39e0d90 0x5649a39f5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.601+0000 7f5435c39640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5649a39e0d90 0x5649a39f5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58000/0 (socket says 172.21.15.16:58000) 2024-03-28T22:19:35.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.601+0000 7f5435c39640 1 -- 172.21.15.16:0/1656702530 learned_addr learned my addr 172.21.15.16:0/1656702530 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:35.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.601+0000 7f5435438640 1 --2- 172.21.15.16:0/1656702530 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5649a39e0470 0x5649a39e9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.601+0000 7f5435c39640 1 -- 172.21.15.16:0/1656702530 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5649a39e9970 msgr2=0x5649a3a01560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:35.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.601+0000 7f5435c39640 1 --2- 172.21.15.16:0/1656702530 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5649a39e9970 0x5649a3a01560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.601+0000 7f5435c39640 1 -- 172.21.15.16:0/1656702530 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5649a39e0470 msgr2=0x5649a39e9130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:35.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.601+0000 7f5435c39640 1 --2- 172.21.15.16:0/1656702530 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5649a39e0470 0x5649a39e9130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.601+0000 7f5435c39640 1 -- 172.21.15.16:0/1656702530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649a39d5150 con 0x5649a39e0d90 2024-03-28T22:19:35.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.601+0000 7f5435c39640 1 --2- 172.21.15.16:0/1656702530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5649a39e0d90 0x5649a39f5530 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f542c014040 tx=0x7f542c00fd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87d1368151601b25 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:35.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.601+0000 7f541b7fe640 1 -- 172.21.15.16:0/1656702530 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f542c00dc90 con 0x5649a39e0d90 2024-03-28T22:19:35.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.601+0000 7f541b7fe640 1 -- 172.21.15.16:0/1656702530 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f542c00de30 con 0x5649a39e0d90 2024-03-28T22:19:35.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.601+0000 7f541b7fe640 1 -- 172.21.15.16:0/1656702530 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f542c004520 con 0x5649a39e0d90 2024-03-28T22:19:35.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.602+0000 7f543aa82740 1 -- 172.21.15.16:0/1656702530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5649a39e0d90 msgr2=0x5649a39f5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:35.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.602+0000 7f543aa82740 1 --2- 172.21.15.16:0/1656702530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5649a39e0d90 0x5649a39f5530 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f542c014040 tx=0x7f542c00fd60 comp rx=0 tx=0).stop 2024-03-28T22:19:35.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.602+0000 7f543aa82740 1 -- 172.21.15.16:0/1656702530 shutdown_connections 2024-03-28T22:19:35.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.602+0000 7f543aa82740 1 --2- 172.21.15.16:0/1656702530 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5649a39e9970 0x5649a3a01560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.602+0000 7f543aa82740 1 --2- 172.21.15.16:0/1656702530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5649a39e0d90 0x5649a39f5530 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.602+0000 7f543aa82740 1 --2- 172.21.15.16:0/1656702530 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5649a39e0470 0x5649a39e9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.602+0000 7f543aa82740 1 -- 172.21.15.16:0/1656702530 >> 172.21.15.16:0/1656702530 conn(0x5649a39daba0 msgr2=0x5649a39d9d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:35.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.602+0000 7f543aa82740 1 -- 172.21.15.16:0/1656702530 shutdown_connections 2024-03-28T22:19:35.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.602+0000 7f543aa82740 1 -- 172.21.15.16:0/1656702530 wait complete. 2024-03-28T22:19:35.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.602+0000 7f543aa82740 1 -- start start 2024-03-28T22:19:35.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.602+0000 7f543aa82740 1 Processor -- start 2024-03-28T22:19:35.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.603+0000 7f543aa82740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5649a39e0470 0x5649a39f1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.603+0000 7f543aa82740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5649a39e0d90 0x5649a39eeb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.603+0000 7f543aa82740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5649a39e9970 0x5649a39ef070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.603+0000 7f543aa82740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f54301ab980 con 0x5649a39e9970 2024-03-28T22:19:35.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.603+0000 7f543aa82740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f54301acdf0 con 0x5649a39e0470 2024-03-28T22:19:35.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.603+0000 7f543aa82740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f54301ac050 con 0x5649a39e0d90 2024-03-28T22:19:35.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.603+0000 7f543aa82740 10 client.0 _ll_register_callbacks cb 0x7f542c00da20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:19:35.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.603+0000 7f5435438640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5649a39e9970 0x5649a39ef070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.603+0000 7f5435438640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5649a39e9970 0x5649a39ef070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58012/0 (socket says 172.21.15.16:58012) 2024-03-28T22:19:35.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.603+0000 7f5435438640 1 -- 172.21.15.16:0/823564687 learned_addr learned my addr 172.21.15.16:0/823564687 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:35.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.603+0000 7f5435438640 1 --2- 172.21.15.16:0/823564687 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5649a39e0470 0x5649a39f1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.603+0000 7f5435c39640 1 --2- 172.21.15.16:0/823564687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5649a39e0d90 0x5649a39eeb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.603+0000 7f5435438640 1 -- 172.21.15.16:0/823564687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5649a39e0d90 msgr2=0x5649a39eeb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:35.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.603+0000 7f5435438640 1 --2- 172.21.15.16:0/823564687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5649a39e0d90 0x5649a39eeb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.603+0000 7f5435438640 1 -- 172.21.15.16:0/823564687 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5649a39e0470 msgr2=0x5649a39f1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:35.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.603+0000 7f5435438640 1 --2- 172.21.15.16:0/823564687 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5649a39e0470 0x5649a39f1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.603+0000 7f5435438640 1 -- 172.21.15.16:0/823564687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649a39f2330 con 0x5649a39e9970 2024-03-28T22:19:35.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.603+0000 7f5435438640 1 --2- 172.21.15.16:0/823564687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5649a39e9970 0x5649a39ef070 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f5420009ce0 tx=0x7f5420009d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2306bdb89308e80f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:35.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.603+0000 7f54197fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:19:35.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.603+0000 7f54197fa640 1 -- 172.21.15.16:0/823564687 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f542001c040 con 0x5649a39e9970 2024-03-28T22:19:35.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.603+0000 7f54197fa640 1 -- 172.21.15.16:0/823564687 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7f5420011ad0 con 0x5649a39e9970 2024-03-28T22:19:35.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.603+0000 7f543aa82740 1 -- 172.21.15.16:0/823564687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f542c00de30 con 0x5649a39e9970 2024-03-28T22:19:35.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.604+0000 7f54197fa640 1 -- 172.21.15.16:0/823564687 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5420035050 con 0x5649a39e9970 2024-03-28T22:19:35.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.604+0000 7f543aa82740 10 client.17406 fetch_fsmap learned FSMap version 27 2024-03-28T22:19:35.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.604+0000 7f543aa82740 1 -- 172.21.15.16:0/823564687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5649a3a606f0 con 0x5649a39e9970 2024-03-28T22:19:35.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.604+0000 7f54197fa640 1 -- 172.21.15.16:0/823564687 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f542c00de30 con 0x5649a39e9970 2024-03-28T22:19:35.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.604+0000 7f54197fa640 1 -- 172.21.15.16:0/823564687 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f54200151a0 con 0x5649a39e9970 2024-03-28T22:19:35.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.604+0000 7f543aa82740 10 client.17406 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:19:35.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.604+0000 7f543aa82740 10 client.17406 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T22:19:35.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.604+0000 7f543aa82740 20 client.17406 populate_metadata read hostname 'smithi016' 2024-03-28T22:19:35.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.604+0000 7f543aa82740 1 --2- 172.21.15.16:0/823564687 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5649a3a60ea0 0x5649a3a632c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.604+0000 7f543aa82740 4 client.17406 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:19:35.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.604+0000 7f543aa82740 1 -- 172.21.15.16:0/823564687 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5649a394ba40 con 0x5649a3a60ea0 2024-03-28T22:19:35.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.604+0000 7f5435c39640 1 --2- 172.21.15.16:0/823564687 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5649a3a60ea0 0x5649a3a632c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.604+0000 7f5435c39640 1 --2- 172.21.15.16:0/823564687 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5649a3a60ea0 0x5649a3a632c0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c04f89b1ca76f78f server_cookie=54515746ff2dec9c in_seq=0 out_seq=0 2024-03-28T22:19:35.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.604+0000 7f54197fa640 10 client.17406 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:19:35.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.605+0000 7f54197fa640 1 -- 172.21.15.16:0/823564687 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5649a394ba40 con 0x5649a3a60ea0 2024-03-28T22:19:35.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.605+0000 7f54197fa640 10 client.17406 handle_command_reply: tid=0 2024-03-28T22:19:35.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.605+0000 7f543aa82740 1 client.17406 shutdown 2024-03-28T22:19:35.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.605+0000 7f543aa82740 1 -- 172.21.15.16:0/823564687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5649a39e9970 msgr2=0x5649a39ef070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:35.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.605+0000 7f543aa82740 1 --2- 172.21.15.16:0/823564687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5649a39e9970 0x5649a39ef070 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f5420009ce0 tx=0x7f5420009d10 comp rx=0 tx=0).stop 2024-03-28T22:19:35.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.605+0000 7f543aa82740 1 -- 172.21.15.16:0/823564687 shutdown_connections 2024-03-28T22:19:35.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.605+0000 7f543aa82740 1 --2- 172.21.15.16:0/823564687 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5649a3a60ea0 0x5649a3a632c0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.605+0000 7f543aa82740 1 --2- 172.21.15.16:0/823564687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5649a39e9970 0x5649a39ef070 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.605+0000 7f54197fa640 0 client.17406 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:19:35.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.605+0000 7f543aa82740 1 --2- 172.21.15.16:0/823564687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5649a39e0d90 0x5649a39eeb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.605+0000 7f543aa82740 1 --2- 172.21.15.16:0/823564687 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5649a39e0470 0x5649a39f1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.605+0000 7f543aa82740 1 -- 172.21.15.16:0/823564687 >> 172.21.15.16:0/823564687 conn(0x5649a39daba0 msgr2=0x5649a39e6930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:35.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.605+0000 7f543aa82740 1 -- 172.21.15.16:0/823564687 shutdown_connections 2024-03-28T22:19:35.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.605+0000 7f543aa82740 1 -- 172.21.15.16:0/823564687 wait complete. 2024-03-28T22:19:35.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.605+0000 7f543aa82740 20 client.17406 trim_cache size 0 max 16384 2024-03-28T22:19:35.619 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:19:35.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.605+0000 7f5437e9d640 1 -- 172.21.15.16:0/1372777618 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f541c0a1ef0 msgr2=0x7f541c0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:35.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.605+0000 7f5437e9d640 1 --2- 172.21.15.16:0/1372777618 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f541c0a1ef0 0x7f541c0a43b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f54280edb10 tx=0x7f542808ecc0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.605+0000 7f5437e9d640 1 -- 172.21.15.16:0/1372777618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54301076f0 msgr2=0x7f543019c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:35.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.605+0000 7f5437e9d640 1 --2- 172.21.15.16:0/1372777618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54301076f0 0x7f543019c840 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f5428005e30 tx=0x7f5428032f40 comp rx=0 tx=0).stop 2024-03-28T22:19:35.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.606+0000 7f5437e9d640 1 -- 172.21.15.16:0/1372777618 shutdown_connections 2024-03-28T22:19:35.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.606+0000 7f5437e9d640 1 --2- 172.21.15.16:0/1372777618 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f541c0a1ef0 0x7f541c0a43b0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.606+0000 7f5437e9d640 1 --2- 172.21.15.16:0/1372777618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54301076f0 0x7f543019c840 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.606+0000 7f5437e9d640 1 --2- 172.21.15.16:0/1372777618 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f54301020a0 0x7f54301984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.606+0000 7f5437e9d640 1 --2- 172.21.15.16:0/1372777618 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54301016e0 0x7f5430197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.606+0000 7f5437e9d640 1 -- 172.21.15.16:0/1372777618 >> 172.21.15.16:0/1372777618 conn(0x7f54300fd1f0 msgr2=0x7f54300fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:35.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.606+0000 7f5437e9d640 1 -- 172.21.15.16:0/1372777618 shutdown_connections 2024-03-28T22:19:35.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.606+0000 7f5437e9d640 1 -- 172.21.15.16:0/1372777618 wait complete. 2024-03-28T22:19:35.623 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:19:35.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.727+0000 7f067c7a1640 1 -- 172.21.15.16:0/19920077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06741009d0 msgr2=0x7f0674100db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:35.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.727+0000 7f067c7a1640 1 --2- 172.21.15.16:0/19920077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06741009d0 0x7f0674100db0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f0670009900 tx=0x7f067002bd10 comp rx=0 tx=0).stop 2024-03-28T22:19:35.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.727+0000 7f067c7a1640 1 -- 172.21.15.16:0/19920077 shutdown_connections 2024-03-28T22:19:35.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.727+0000 7f067c7a1640 1 --2- 172.21.15.16:0/19920077 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0674102b80 0x7f067410d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.727+0000 7f067c7a1640 1 --2- 172.21.15.16:0/19920077 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06741021e0 0x7f0674102640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.727+0000 7f067c7a1640 1 --2- 172.21.15.16:0/19920077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06741009d0 0x7f0674100db0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.727+0000 7f067c7a1640 1 -- 172.21.15.16:0/19920077 >> 172.21.15.16:0/19920077 conn(0x7f0674077830 msgr2=0x7f0674077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:35.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.727+0000 7f067c7a1640 1 -- 172.21.15.16:0/19920077 shutdown_connections 2024-03-28T22:19:35.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.727+0000 7f067c7a1640 1 -- 172.21.15.16:0/19920077 wait complete. 2024-03-28T22:19:35.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.728+0000 7f067c7a1640 1 Processor -- start 2024-03-28T22:19:35.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.728+0000 7f067c7a1640 1 -- start start 2024-03-28T22:19:35.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.728+0000 7f067c7a1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06741009d0 0x7f0674198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.728+0000 7f067c7a1640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06741021e0 0x7f0674198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.728+0000 7f067c7a1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0674102b80 0x7f067419c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.728+0000 7f067c7a1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0674111c10 con 0x7f06741009d0 2024-03-28T22:19:35.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.728+0000 7f067c7a1640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0674111a90 con 0x7f06741021e0 2024-03-28T22:19:35.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.728+0000 7f067c7a1640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0674111d90 con 0x7f0674102b80 2024-03-28T22:19:35.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.728+0000 7f067a53d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06741009d0 0x7f0674198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.728+0000 7f067a53d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06741009d0 0x7f0674198110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58024/0 (socket says 172.21.15.16:58024) 2024-03-28T22:19:35.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.728+0000 7f067a53d640 1 -- 172.21.15.16:0/1845892029 learned_addr learned my addr 172.21.15.16:0/1845892029 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:35.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.728+0000 7f0679d3c640 1 --2- 172.21.15.16:0/1845892029 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06741021e0 0x7f0674198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.728+0000 7f067a53d640 1 -- 172.21.15.16:0/1845892029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0674102b80 msgr2=0x7f067419c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:35.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.728+0000 7f067a53d640 1 --2- 172.21.15.16:0/1845892029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0674102b80 0x7f067419c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.728+0000 7f067a53d640 1 -- 172.21.15.16:0/1845892029 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06741021e0 msgr2=0x7f0674198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:35.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.728+0000 7f067a53d640 1 --2- 172.21.15.16:0/1845892029 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06741021e0 0x7f0674198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.728+0000 7f067a53d640 1 -- 172.21.15.16:0/1845892029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f067419d0b0 con 0x7f06741009d0 2024-03-28T22:19:35.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.728+0000 7f067a53d640 1 --2- 172.21.15.16:0/1845892029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06741009d0 0x7f0674198110 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f0670006010 tx=0x7f06700042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:35.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.729+0000 7f066b7fe640 1 -- 172.21.15.16:0/1845892029 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0670042020 con 0x7f06741009d0 2024-03-28T22:19:35.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.729+0000 7f066b7fe640 1 -- 172.21.15.16:0/1845892029 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06700054e0 con 0x7f06741009d0 2024-03-28T22:19:35.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.729+0000 7f067c7a1640 1 -- 172.21.15.16:0/1845892029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f067419d340 con 0x7f06741009d0 2024-03-28T22:19:35.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.729+0000 7f066b7fe640 1 -- 172.21.15.16:0/1845892029 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0670003ad0 con 0x7f06741009d0 2024-03-28T22:19:35.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.729+0000 7f067c7a1640 1 -- 172.21.15.16:0/1845892029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06741a4c30 con 0x7f06741009d0 2024-03-28T22:19:35.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.731+0000 7f066b7fe640 1 -- 172.21.15.16:0/1845892029 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f0670045020 con 0x7f06741009d0 2024-03-28T22:19:35.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.731+0000 7f067f384740 1 Processor -- start 2024-03-28T22:19:35.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.731+0000 7f067f384740 1 -- start start 2024-03-28T22:19:35.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.732+0000 7f066b7fe640 1 --2- 172.21.15.16:0/1845892029 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f06600a1f70 0x7f06600a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.732+0000 7f067f384740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af85ea8470 0x55af85ea8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.732+0000 7f067f384740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af85ea8d90 0x55af85eb1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.732+0000 7f067f384740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af85eb1970 0x55af85eb3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.732+0000 7f067f384740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55af85e0c1b0 con 0x55af85ea8470 2024-03-28T22:19:35.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.732+0000 7f067f384740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55af85db4c00 con 0x55af85eb1970 2024-03-28T22:19:35.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.732+0000 7f067f384740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55af85e08250 con 0x55af85ea8d90 2024-03-28T22:19:35.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.732+0000 7f067a53d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af85ea8470 0x55af85ea8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.732+0000 7f067ad3e640 1 --2- 172.21.15.16:0/1845892029 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f06600a1f70 0x7f06600a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.732+0000 7f067a53d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af85ea8470 0x55af85ea8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58030/0 (socket says 172.21.15.16:58030) 2024-03-28T22:19:35.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.732+0000 7f067a53d640 1 -- 172.21.15.16:0/3213909611 learned_addr learned my addr 172.21.15.16:0/3213909611 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:35.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.732+0000 7f066b7fe640 1 -- 172.21.15.16:0/1845892029 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7f06700efcf0 con 0x7f06741009d0 2024-03-28T22:19:35.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.732+0000 7f067ad3e640 1 --2- 172.21.15.16:0/3213909611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af85eb1970 0x55af85eb3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.732+0000 7f0679d3c640 1 --2- 172.21.15.16:0/3213909611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af85ea8d90 0x55af85eb1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.732+0000 7f067a53d640 1 -- 172.21.15.16:0/3213909611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af85ea8d90 msgr2=0x55af85eb1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:35.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.732+0000 7f067a53d640 1 --2- 172.21.15.16:0/3213909611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af85ea8d90 0x55af85eb1430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.732+0000 7f067a53d640 1 -- 172.21.15.16:0/3213909611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af85eb1970 msgr2=0x55af85eb3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:35.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.732+0000 7f067a53d640 1 --2- 172.21.15.16:0/3213909611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af85eb1970 0x55af85eb3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.732+0000 7f067a53d640 1 -- 172.21.15.16:0/3213909611 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af85dd43d0 con 0x55af85ea8470 2024-03-28T22:19:35.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.732+0000 7f067ad3e640 1 --2- 172.21.15.16:0/1845892029 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f06600a1f70 0x7f06600a4430 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f066c009fd0 tx=0x7f066c009d20 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:35.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.732+0000 7f067a53d640 1 --2- 172.21.15.16:0/3213909611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af85ea8470 0x55af85ea8850 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f06700bc780 tx=0x7f067004ff90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=529a2cc7457c637f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:35.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.732+0000 7f0668ff9640 1 -- 172.21.15.16:0/3213909611 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06700bc150 con 0x55af85ea8470 2024-03-28T22:19:35.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.732+0000 7f0668ff9640 1 -- 172.21.15.16:0/3213909611 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0670004c80 con 0x55af85ea8470 2024-03-28T22:19:35.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.732+0000 7f0668ff9640 1 -- 172.21.15.16:0/3213909611 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06700d5520 con 0x55af85ea8470 2024-03-28T22:19:35.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.733+0000 7f067f384740 1 -- 172.21.15.16:0/3213909611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af85ea8470 msgr2=0x55af85ea8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:35.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.733+0000 7f067f384740 1 --2- 172.21.15.16:0/3213909611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af85ea8470 0x55af85ea8850 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f06700bc780 tx=0x7f067004ff90 comp rx=0 tx=0).stop 2024-03-28T22:19:35.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.733+0000 7f067f384740 1 -- 172.21.15.16:0/3213909611 shutdown_connections 2024-03-28T22:19:35.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.733+0000 7f067f384740 1 --2- 172.21.15.16:0/3213909611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af85eb1970 0x55af85eb3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.733+0000 7f067f384740 1 --2- 172.21.15.16:0/3213909611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af85ea8d90 0x55af85eb1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.733+0000 7f067f384740 1 --2- 172.21.15.16:0/3213909611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af85ea8470 0x55af85ea8850 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.733+0000 7f067f384740 1 -- 172.21.15.16:0/3213909611 >> 172.21.15.16:0/3213909611 conn(0x55af85ea2ba0 msgr2=0x55af85ea2f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:35.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.733+0000 7f067f384740 1 -- 172.21.15.16:0/3213909611 shutdown_connections 2024-03-28T22:19:35.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.733+0000 7f067f384740 1 -- 172.21.15.16:0/3213909611 wait complete. 2024-03-28T22:19:35.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.734+0000 7f067f384740 1 -- start start 2024-03-28T22:19:35.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.734+0000 7f067f384740 1 Processor -- start 2024-03-28T22:19:35.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.734+0000 7f067f384740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af85ea8470 0x55af85eae9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.734+0000 7f067f384740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af85ea8d90 0x55af85eaeee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.734+0000 7f067f384740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af85eb1970 0x55af85eaf420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.734+0000 7f067f384740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55af85eb4520 con 0x55af85ea8d90 2024-03-28T22:19:35.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.734+0000 7f067f384740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55af85daa470 con 0x55af85eb1970 2024-03-28T22:19:35.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.734+0000 7f067f384740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55af85eb46a0 con 0x55af85ea8470 2024-03-28T22:19:35.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.734+0000 7f067f384740 10 client.0 _ll_register_callbacks cb 0x55af85dc7cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:19:35.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.734+0000 7f0679d3c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af85ea8d90 0x55af85eaeee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.734+0000 7f067ad3e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af85eb1970 0x55af85eaf420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.734+0000 7f0679d3c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af85ea8d90 0x55af85eaeee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58036/0 (socket says 172.21.15.16:58036) 2024-03-28T22:19:35.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.734+0000 7f0679d3c640 1 -- 172.21.15.16:0/1843728403 learned_addr learned my addr 172.21.15.16:0/1843728403 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:35.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.734+0000 7f067a53d640 1 --2- 172.21.15.16:0/1843728403 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af85ea8470 0x55af85eae9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.734+0000 7f0679d3c640 1 -- 172.21.15.16:0/1843728403 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af85ea8470 msgr2=0x55af85eae9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:35.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.734+0000 7f0679d3c640 1 --2- 172.21.15.16:0/1843728403 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af85ea8470 0x55af85eae9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.734+0000 7f0679d3c640 1 -- 172.21.15.16:0/1843728403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af85eb1970 msgr2=0x55af85eaf420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:35.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.734+0000 7f0679d3c640 1 --2- 172.21.15.16:0/1843728403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af85eb1970 0x55af85eaf420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.734+0000 7f0679d3c640 1 -- 172.21.15.16:0/1843728403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af85b203f0 con 0x55af85ea8d90 2024-03-28T22:19:35.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.734+0000 7f0679d3c640 1 --2- 172.21.15.16:0/1843728403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af85ea8d90 0x55af85eaeee0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f0664012750 tx=0x7f0664012c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd2fbd89ddcddb43 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:35.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.734+0000 7f065dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:19:35.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.735+0000 7f065dffb640 1 -- 172.21.15.16:0/1843728403 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0664012e60 con 0x55af85ea8d90 2024-03-28T22:19:35.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.735+0000 7f065dffb640 1 -- 172.21.15.16:0/1843728403 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7f066400f920 con 0x55af85ea8d90 2024-03-28T22:19:35.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.735+0000 7f067f384740 1 -- 172.21.15.16:0/1843728403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0670004c80 con 0x55af85ea8d90 2024-03-28T22:19:35.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.735+0000 7f065dffb640 1 -- 172.21.15.16:0/1843728403 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f066400fb20 con 0x55af85ea8d90 2024-03-28T22:19:35.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.735+0000 7f067f384740 10 client.17418 fetch_fsmap learned FSMap version 27 2024-03-28T22:19:35.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.735+0000 7f067f384740 1 -- 172.21.15.16:0/1843728403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55af85bea7d0 con 0x55af85ea8d90 2024-03-28T22:19:35.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.735+0000 7f065dffb640 1 -- 172.21.15.16:0/1843728403 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0670004c80 con 0x55af85ea8d90 2024-03-28T22:19:35.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.735+0000 7f065dffb640 1 -- 172.21.15.16:0/1843728403 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f066401c420 con 0x55af85ea8d90 2024-03-28T22:19:35.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.735+0000 7f067f384740 10 client.17418 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:19:35.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.735+0000 7f067f384740 10 client.17418 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:19:35.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.735+0000 7f067f384740 20 client.17418 populate_metadata read hostname 'smithi016' 2024-03-28T22:19:35.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.735+0000 7f067f384740 1 --2- 172.21.15.16:0/1843728403 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55af85ec1790 0x55af85ec3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.735+0000 7f067f384740 4 client.17418 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:19:35.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.735+0000 7f067f384740 1 -- 172.21.15.16:0/1843728403 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55af85e196f0 con 0x55af85ec1790 2024-03-28T22:19:35.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.735+0000 7f067a53d640 1 --2- 172.21.15.16:0/1843728403 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55af85ec1790 0x55af85ec3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.736+0000 7f065dffb640 10 client.17418 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:19:35.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.736+0000 7f067a53d640 1 --2- 172.21.15.16:0/1843728403 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55af85ec1790 0x55af85ec3bf0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb65ba9a39e8d42b server_cookie=99bd74bae381d84c in_seq=0 out_seq=0 2024-03-28T22:19:35.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.736+0000 7f065dffb640 1 -- 172.21.15.16:0/1843728403 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55af85e196f0 con 0x55af85ec1790 2024-03-28T22:19:35.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.736+0000 7f065dffb640 10 client.17418 handle_command_reply: tid=0 2024-03-28T22:19:35.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.736+0000 7f067f384740 1 client.17418 shutdown 2024-03-28T22:19:35.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.737+0000 7f067f384740 1 -- 172.21.15.16:0/1843728403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af85ea8d90 msgr2=0x55af85eaeee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:35.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.737+0000 7f067f384740 1 --2- 172.21.15.16:0/1843728403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af85ea8d90 0x55af85eaeee0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f0664012750 tx=0x7f0664012c10 comp rx=0 tx=0).stop 2024-03-28T22:19:35.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.737+0000 7f067f384740 1 -- 172.21.15.16:0/1843728403 shutdown_connections 2024-03-28T22:19:35.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.737+0000 7f067f384740 1 --2- 172.21.15.16:0/1843728403 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55af85ec1790 0x55af85ec3bf0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.737+0000 7f067f384740 1 --2- 172.21.15.16:0/1843728403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af85eb1970 0x55af85eaf420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.737+0000 7f065dffb640 0 client.17418 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:19:35.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.737+0000 7f067f384740 1 --2- 172.21.15.16:0/1843728403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af85ea8d90 0x55af85eaeee0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.737+0000 7f067f384740 1 --2- 172.21.15.16:0/1843728403 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af85ea8470 0x55af85eae9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.737+0000 7f067f384740 1 -- 172.21.15.16:0/1843728403 >> 172.21.15.16:0/1843728403 conn(0x55af85ea2ba0 msgr2=0x55af85ea48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:35.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.737+0000 7f067f384740 1 -- 172.21.15.16:0/1843728403 shutdown_connections 2024-03-28T22:19:35.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.737+0000 7f067f384740 1 -- 172.21.15.16:0/1843728403 wait complete. 2024-03-28T22:19:35.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.737+0000 7f067f384740 20 client.17418 trim_cache size 0 max 16384 2024-03-28T22:19:35.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.750+0000 7f067f384740 1 Processor -- start 2024-03-28T22:19:35.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.750+0000 7f067f384740 1 -- start start 2024-03-28T22:19:35.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.750+0000 7f067f384740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af85ea8470 0x55af85eb10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.750+0000 7f067f384740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af85ea8d90 0x55af85ebd100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.750+0000 7f067f384740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af85eb1970 0x55af85ec94c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.750+0000 7f067f384740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f06741afd40 con 0x55af85ea8d90 2024-03-28T22:19:35.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.750+0000 7f067f384740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f06741ad610 con 0x55af85ea8470 2024-03-28T22:19:35.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.750+0000 7f067f384740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f06741ab480 con 0x55af85eb1970 2024-03-28T22:19:35.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.750+0000 7f0679d3c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af85ea8d90 0x55af85ebd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.750+0000 7f0679d3c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af85ea8d90 0x55af85ebd100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58038/0 (socket says 172.21.15.16:58038) 2024-03-28T22:19:35.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.750+0000 7f0679d3c640 1 -- 172.21.15.16:0/1502619679 learned_addr learned my addr 172.21.15.16:0/1502619679 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:35.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.750+0000 7f067a53d640 1 --2- 172.21.15.16:0/1502619679 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af85ea8470 0x55af85eb10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.751+0000 7f0679d3c640 1 -- 172.21.15.16:0/1502619679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af85eb1970 msgr2=0x55af85ec94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:35.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.751+0000 7f0679d3c640 1 --2- 172.21.15.16:0/1502619679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af85eb1970 0x55af85ec94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.751+0000 7f0679d3c640 1 -- 172.21.15.16:0/1502619679 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af85ea8470 msgr2=0x55af85eb10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:35.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.751+0000 7f0679d3c640 1 --2- 172.21.15.16:0/1502619679 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af85ea8470 0x55af85eb10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.751+0000 7f0679d3c640 1 -- 172.21.15.16:0/1502619679 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af85ea8bc0 con 0x55af85ea8d90 2024-03-28T22:19:35.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.751+0000 7f0679d3c640 1 --2- 172.21.15.16:0/1502619679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af85ea8d90 0x55af85ebd100 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f066401ca80 tx=0x7f0664009e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa802773fd3f969e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:35.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.751+0000 7f065ffff640 1 -- 172.21.15.16:0/1502619679 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0664012e60 con 0x55af85ea8d90 2024-03-28T22:19:35.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.751+0000 7f065ffff640 1 -- 172.21.15.16:0/1502619679 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f066400f920 con 0x55af85ea8d90 2024-03-28T22:19:35.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.751+0000 7f065ffff640 1 -- 172.21.15.16:0/1502619679 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0664005540 con 0x55af85ea8d90 2024-03-28T22:19:35.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.751+0000 7f067f384740 1 -- 172.21.15.16:0/1502619679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af85ea8d90 msgr2=0x55af85ebd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:35.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.751+0000 7f067f384740 1 --2- 172.21.15.16:0/1502619679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af85ea8d90 0x55af85ebd100 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f066401ca80 tx=0x7f0664009e50 comp rx=0 tx=0).stop 2024-03-28T22:19:35.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.751+0000 7f067f384740 1 -- 172.21.15.16:0/1502619679 shutdown_connections 2024-03-28T22:19:35.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.751+0000 7f067f384740 1 --2- 172.21.15.16:0/1502619679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af85eb1970 0x55af85ec94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.751+0000 7f067f384740 1 --2- 172.21.15.16:0/1502619679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af85ea8d90 0x55af85ebd100 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.751+0000 7f067f384740 1 --2- 172.21.15.16:0/1502619679 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af85ea8470 0x55af85eb10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.751+0000 7f067f384740 1 -- 172.21.15.16:0/1502619679 >> 172.21.15.16:0/1502619679 conn(0x55af85ea2ba0 msgr2=0x55af85ea1d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:35.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.751+0000 7f067f384740 1 -- 172.21.15.16:0/1502619679 shutdown_connections 2024-03-28T22:19:35.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.751+0000 7f067f384740 1 -- 172.21.15.16:0/1502619679 wait complete. 2024-03-28T22:19:35.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.752+0000 7f067f384740 1 -- start start 2024-03-28T22:19:35.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.752+0000 7f067f384740 1 Processor -- start 2024-03-28T22:19:35.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.752+0000 7f067f384740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af85ea8470 0x55af85eb5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.752+0000 7f067f384740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af85ea8d90 0x55af85eb5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.752+0000 7f067f384740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af85eb1970 0x55af85eb5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.752+0000 7f067f384740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f06741abad0 con 0x55af85ea8470 2024-03-28T22:19:35.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.752+0000 7f067f384740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f06741acf40 con 0x55af85eb1970 2024-03-28T22:19:35.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.752+0000 7f067f384740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f06741ac1a0 con 0x55af85ea8d90 2024-03-28T22:19:35.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.752+0000 7f067f384740 10 client.0 _ll_register_callbacks cb 0x7f0664009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:19:35.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.752+0000 7f067a53d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af85ea8470 0x55af85eb5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.752+0000 7f067a53d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af85ea8470 0x55af85eb5220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58040/0 (socket says 172.21.15.16:58040) 2024-03-28T22:19:35.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.752+0000 7f067a53d640 1 -- 172.21.15.16:0/3002515318 learned_addr learned my addr 172.21.15.16:0/3002515318 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:35.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.752+0000 7f067ad3e640 1 --2- 172.21.15.16:0/3002515318 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af85eb1970 0x55af85eb5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.752+0000 7f0679d3c640 1 --2- 172.21.15.16:0/3002515318 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af85ea8d90 0x55af85eb5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.752+0000 7f067a53d640 1 -- 172.21.15.16:0/3002515318 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af85ea8d90 msgr2=0x55af85eb5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:35.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.752+0000 7f067a53d640 1 --2- 172.21.15.16:0/3002515318 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af85ea8d90 0x55af85eb5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.752+0000 7f067a53d640 1 -- 172.21.15.16:0/3002515318 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af85eb1970 msgr2=0x55af85eb5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:35.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.752+0000 7f067a53d640 1 --2- 172.21.15.16:0/3002515318 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af85eb1970 0x55af85eb5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.752+0000 7f067a53d640 1 -- 172.21.15.16:0/3002515318 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af85f26cf0 con 0x55af85ea8470 2024-03-28T22:19:35.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.753+0000 7f067a53d640 1 --2- 172.21.15.16:0/3002515318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af85ea8470 0x55af85eb5220 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f0670000c00 tx=0x7f067002bd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8989313375d7ec02 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:35.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.753+0000 7f065dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:19:35.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.753+0000 7f065dffb640 1 -- 172.21.15.16:0/3002515318 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0670004c80 con 0x55af85ea8470 2024-03-28T22:19:35.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.753+0000 7f065dffb640 1 -- 172.21.15.16:0/3002515318 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7f06700a70b0 con 0x55af85ea8470 2024-03-28T22:19:35.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.753+0000 7f067f384740 1 -- 172.21.15.16:0/3002515318 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f066400f920 con 0x55af85ea8470 2024-03-28T22:19:35.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.753+0000 7f065dffb640 1 -- 172.21.15.16:0/3002515318 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0670005690 con 0x55af85ea8470 2024-03-28T22:19:35.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.753+0000 7f067f384740 10 client.17424 fetch_fsmap learned FSMap version 27 2024-03-28T22:19:35.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.753+0000 7f067f384740 1 -- 172.21.15.16:0/3002515318 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55af85f27fd0 con 0x55af85ea8470 2024-03-28T22:19:35.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.753+0000 7f065dffb640 1 -- 172.21.15.16:0/3002515318 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f066400f920 con 0x55af85ea8470 2024-03-28T22:19:35.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.753+0000 7f065dffb640 1 -- 172.21.15.16:0/3002515318 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f06700ed6c0 con 0x55af85ea8470 2024-03-28T22:19:35.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.753+0000 7f067f384740 10 client.17424 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:19:35.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.753+0000 7f067f384740 10 client.17424 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:19:35.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.753+0000 7f067f384740 20 client.17424 populate_metadata read hostname 'smithi016' 2024-03-28T22:19:35.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.753+0000 7f067f384740 1 --2- 172.21.15.16:0/3002515318 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55af85ec1790 0x55af85f2a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:35.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.753+0000 7f067f384740 4 client.17424 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:19:35.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.753+0000 7f067f384740 1 -- 172.21.15.16:0/3002515318 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55af85e13a40 con 0x55af85ec1790 2024-03-28T22:19:35.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.753+0000 7f0679d3c640 1 --2- 172.21.15.16:0/3002515318 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55af85ec1790 0x55af85f2a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:35.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.754+0000 7f065dffb640 10 client.17424 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:19:35.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.754+0000 7f0679d3c640 1 --2- 172.21.15.16:0/3002515318 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55af85ec1790 0x55af85f2a5d0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76492b142233715b server_cookie=4508533036fae948 in_seq=0 out_seq=0 2024-03-28T22:19:35.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.754+0000 7f065dffb640 1 -- 172.21.15.16:0/3002515318 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55af85e13a40 con 0x55af85ec1790 2024-03-28T22:19:35.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.754+0000 7f065dffb640 10 client.17424 handle_command_reply: tid=0 2024-03-28T22:19:35.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.754+0000 7f067f384740 1 client.17424 shutdown 2024-03-28T22:19:35.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.754+0000 7f067f384740 1 -- 172.21.15.16:0/3002515318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af85ea8470 msgr2=0x55af85eb5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:35.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.754+0000 7f067f384740 1 --2- 172.21.15.16:0/3002515318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af85ea8470 0x55af85eb5220 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f0670000c00 tx=0x7f067002bd10 comp rx=0 tx=0).stop 2024-03-28T22:19:35.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.754+0000 7f067f384740 1 -- 172.21.15.16:0/3002515318 shutdown_connections 2024-03-28T22:19:35.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.754+0000 7f067f384740 1 --2- 172.21.15.16:0/3002515318 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55af85ec1790 0x55af85f2a5d0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.754+0000 7f067f384740 1 --2- 172.21.15.16:0/3002515318 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af85eb1970 0x55af85eb5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.754+0000 7f067f384740 1 --2- 172.21.15.16:0/3002515318 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af85ea8d90 0x55af85eb5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.754+0000 7f067f384740 1 --2- 172.21.15.16:0/3002515318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55af85ea8470 0x55af85eb5220 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.754+0000 7f065dffb640 0 client.17424 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:19:35.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.754+0000 7f067f384740 1 -- 172.21.15.16:0/3002515318 >> 172.21.15.16:0/3002515318 conn(0x55af85ea2ba0 msgr2=0x55af85ea3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:35.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.754+0000 7f067f384740 1 -- 172.21.15.16:0/3002515318 shutdown_connections 2024-03-28T22:19:35.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.754+0000 7f067f384740 1 -- 172.21.15.16:0/3002515318 wait complete. 2024-03-28T22:19:35.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.754+0000 7f067f384740 20 client.17424 trim_cache size 0 max 16384 2024-03-28T22:19:35.769 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:19:35.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.755+0000 7f067c7a1640 1 -- 172.21.15.16:0/1845892029 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f06600a1f70 msgr2=0x7f06600a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:35.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.755+0000 7f067c7a1640 1 --2- 172.21.15.16:0/1845892029 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f06600a1f70 0x7f06600a4430 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f066c009fd0 tx=0x7f066c009d20 comp rx=0 tx=0).stop 2024-03-28T22:19:35.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.755+0000 7f067c7a1640 1 -- 172.21.15.16:0/1845892029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06741009d0 msgr2=0x7f0674198110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:35.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.755+0000 7f067c7a1640 1 --2- 172.21.15.16:0/1845892029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06741009d0 0x7f0674198110 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f0670006010 tx=0x7f06700042f0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.755+0000 7f067c7a1640 1 -- 172.21.15.16:0/1845892029 shutdown_connections 2024-03-28T22:19:35.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.755+0000 7f067c7a1640 1 --2- 172.21.15.16:0/1845892029 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f06600a1f70 0x7f06600a4430 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.755+0000 7f067c7a1640 1 --2- 172.21.15.16:0/1845892029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0674102b80 0x7f067419c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.755+0000 7f067c7a1640 1 --2- 172.21.15.16:0/1845892029 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06741021e0 0x7f0674198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.755+0000 7f067c7a1640 1 --2- 172.21.15.16:0/1845892029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06741009d0 0x7f0674198110 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:35.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.755+0000 7f067c7a1640 1 -- 172.21.15.16:0/1845892029 >> 172.21.15.16:0/1845892029 conn(0x7f0674077830 msgr2=0x7f0674104400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:35.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.755+0000 7f067c7a1640 1 -- 172.21.15.16:0/1845892029 shutdown_connections 2024-03-28T22:19:35.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:35.755+0000 7f067c7a1640 1 -- 172.21.15.16:0/1845892029 wait complete. 2024-03-28T22:19:36.430 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:36 smithi063 ceph-mon[36249]: pgmap v279: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:19:36.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:36 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/3251641273' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:19:36.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:36 smithi016 ceph-mon[32374]: pgmap v279: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:19:36.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:36 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3251641273' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:19:36.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:36 smithi053 ceph-mon[35979]: pgmap v279: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:19:36.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:36 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3251641273' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:19:36.773 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:19:36.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:19:36.774 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:19:36.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.892+0000 7fabab4e1640 1 -- 172.21.15.16:0/4115727734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faba41009d0 msgr2=0x7faba4100db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:36.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.892+0000 7fabab4e1640 1 --2- 172.21.15.16:0/4115727734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faba41009d0 0x7faba4100db0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7faba0009900 tx=0x7faba002bd10 comp rx=0 tx=0).stop 2024-03-28T22:19:36.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.892+0000 7fabab4e1640 1 -- 172.21.15.16:0/4115727734 shutdown_connections 2024-03-28T22:19:36.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.892+0000 7fabab4e1640 1 --2- 172.21.15.16:0/4115727734 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faba4102b80 0x7faba410d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:36.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.892+0000 7fabab4e1640 1 --2- 172.21.15.16:0/4115727734 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faba41021e0 0x7faba4102640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:36.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.892+0000 7fabab4e1640 1 --2- 172.21.15.16:0/4115727734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faba41009d0 0x7faba4100db0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:36.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.892+0000 7fabab4e1640 1 -- 172.21.15.16:0/4115727734 >> 172.21.15.16:0/4115727734 conn(0x7faba4077830 msgr2=0x7faba4077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:36.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.892+0000 7fabab4e1640 1 -- 172.21.15.16:0/4115727734 shutdown_connections 2024-03-28T22:19:36.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.892+0000 7fabab4e1640 1 -- 172.21.15.16:0/4115727734 wait complete. 2024-03-28T22:19:36.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.893+0000 7fabab4e1640 1 Processor -- start 2024-03-28T22:19:36.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.893+0000 7fabab4e1640 1 -- start start 2024-03-28T22:19:36.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.893+0000 7fabab4e1640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faba41009d0 0x7faba40fd590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:36.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.893+0000 7fabab4e1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faba41021e0 0x7faba40fdad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:36.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.893+0000 7fabab4e1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faba4102b80 0x7faba40fe010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:36.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.893+0000 7fabab4e1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faba4110b20 con 0x7faba41021e0 2024-03-28T22:19:36.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.893+0000 7fabab4e1640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7faba41109a0 con 0x7faba41009d0 2024-03-28T22:19:36.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.893+0000 7fabab4e1640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7faba4110ca0 con 0x7faba4102b80 2024-03-28T22:19:36.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.893+0000 7faba8a7c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faba41021e0 0x7faba40fdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:36.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.893+0000 7faba8a7c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faba41021e0 0x7faba40fdad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58062/0 (socket says 172.21.15.16:58062) 2024-03-28T22:19:36.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.893+0000 7faba927d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faba41009d0 0x7faba40fd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:36.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.893+0000 7faba8a7c640 1 -- 172.21.15.16:0/390893947 learned_addr learned my addr 172.21.15.16:0/390893947 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:36.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.893+0000 7faba9a7e640 1 --2- 172.21.15.16:0/390893947 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faba4102b80 0x7faba40fe010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:36.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.893+0000 7faba8a7c640 1 -- 172.21.15.16:0/390893947 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faba4102b80 msgr2=0x7faba40fe010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:36.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.893+0000 7faba8a7c640 1 --2- 172.21.15.16:0/390893947 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faba4102b80 0x7faba40fe010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:36.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.894+0000 7faba8a7c640 1 -- 172.21.15.16:0/390893947 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faba41009d0 msgr2=0x7faba40fd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:36.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.894+0000 7faba8a7c640 1 --2- 172.21.15.16:0/390893947 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faba41009d0 0x7faba40fd590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:36.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.894+0000 7faba8a7c640 1 -- 172.21.15.16:0/390893947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faba41a65e0 con 0x7faba41021e0 2024-03-28T22:19:36.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.894+0000 7faba8a7c640 1 --2- 172.21.15.16:0/390893947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faba41021e0 0x7faba40fdad0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fab9400d770 tx=0x7fab9400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:36.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.894+0000 7fab9a7fc640 1 -- 172.21.15.16:0/390893947 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab94017070 con 0x7faba41021e0 2024-03-28T22:19:36.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.894+0000 7fab9a7fc640 1 -- 172.21.15.16:0/390893947 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab94005500 con 0x7faba41021e0 2024-03-28T22:19:36.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.894+0000 7fabab4e1640 1 -- 172.21.15.16:0/390893947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faba41a68d0 con 0x7faba41021e0 2024-03-28T22:19:36.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.894+0000 7fab9a7fc640 1 -- 172.21.15.16:0/390893947 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab94004e00 con 0x7faba41021e0 2024-03-28T22:19:36.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.894+0000 7fabab4e1640 1 -- 172.21.15.16:0/390893947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faba41a6e20 con 0x7faba41021e0 2024-03-28T22:19:36.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.896+0000 7fab9a7fc640 1 -- 172.21.15.16:0/390893947 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7fab94004fa0 con 0x7faba41021e0 2024-03-28T22:19:36.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.897+0000 7fabae19b740 1 Processor -- start 2024-03-28T22:19:36.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.897+0000 7fabae19b740 1 -- start start 2024-03-28T22:19:36.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.897+0000 7fab9a7fc640 1 --2- 172.21.15.16:0/390893947 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fab900a1f70 0x7fab900a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:36.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.897+0000 7fabae19b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24f9ca9f0 0x55e24f9cadd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:36.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.897+0000 7fabae19b740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e24f9cb310 0x55e24f9d39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:36.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.897+0000 7fabae19b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e24f9d3ef0 0x55e24f9d62e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:36.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.897+0000 7fabae19b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e24f8cb0b0 con 0x55e24f9d3ef0 2024-03-28T22:19:36.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.897+0000 7fabae19b740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55e24f92dea0 con 0x55e24f9cb310 2024-03-28T22:19:36.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.897+0000 7fabae19b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e24f8d6c00 con 0x55e24f9ca9f0 2024-03-28T22:19:36.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.897+0000 7faba9a7e640 1 --2- 172.21.15.16:0/390893947 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fab900a1f70 0x7fab900a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:36.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.897+0000 7faba9a7e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e24f9d3ef0 0x55e24f9d62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:36.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.897+0000 7faba8a7c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e24f9cb310 0x55e24f9d39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:36.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.897+0000 7faba9a7e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e24f9d3ef0 0x55e24f9d62e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58064/0 (socket says 172.21.15.16:58064) 2024-03-28T22:19:36.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.897+0000 7faba927d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24f9ca9f0 0x55e24f9cadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:36.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.897+0000 7faba9a7e640 1 -- 172.21.15.16:0/4061965641 learned_addr learned my addr 172.21.15.16:0/4061965641 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:36.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.897+0000 7fab9a7fc640 1 -- 172.21.15.16:0/390893947 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7fab940cdb20 con 0x7faba41021e0 2024-03-28T22:19:36.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.898+0000 7faba9a7e640 1 -- 172.21.15.16:0/4061965641 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24f9ca9f0 msgr2=0x55e24f9cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:36.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.898+0000 7faba9a7e640 1 --2- 172.21.15.16:0/4061965641 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24f9ca9f0 0x55e24f9cadd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:36.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.898+0000 7faba9a7e640 1 -- 172.21.15.16:0/4061965641 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e24f9cb310 msgr2=0x55e24f9d39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:36.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.898+0000 7faba9a7e640 1 --2- 172.21.15.16:0/4061965641 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e24f9cb310 0x55e24f9d39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:36.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.898+0000 7faba9a7e640 1 -- 172.21.15.16:0/4061965641 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e24f70a7e0 con 0x55e24f9d3ef0 2024-03-28T22:19:36.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.898+0000 7faba8a7c640 1 --2- 172.21.15.16:0/4061965641 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e24f9cb310 0x55e24f9d39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:19:36.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.898+0000 7faba9a7e640 1 --2- 172.21.15.16:0/4061965641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e24f9d3ef0 0x55e24f9d62e0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fab9c010dc0 tx=0x7fab9c00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bb53bb7fff32bbb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:36.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.898+0000 7faba9a7e640 1 --2- 172.21.15.16:0/390893947 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fab900a1f70 0x7fab900a4430 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fab9c00fc90 tx=0x7fab9c018000 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:36.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.898+0000 7fab8f7fe640 1 -- 172.21.15.16:0/4061965641 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab9c019040 con 0x55e24f9d3ef0 2024-03-28T22:19:36.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.898+0000 7fab8f7fe640 1 -- 172.21.15.16:0/4061965641 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab9c009d70 con 0x55e24f9d3ef0 2024-03-28T22:19:36.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.898+0000 7fab8f7fe640 1 -- 172.21.15.16:0/4061965641 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab9c005560 con 0x55e24f9d3ef0 2024-03-28T22:19:36.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.899+0000 7fabae19b740 1 -- 172.21.15.16:0/4061965641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e24f9d3ef0 msgr2=0x55e24f9d62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:36.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.899+0000 7fabae19b740 1 --2- 172.21.15.16:0/4061965641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e24f9d3ef0 0x55e24f9d62e0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fab9c010dc0 tx=0x7fab9c00f790 comp rx=0 tx=0).stop 2024-03-28T22:19:36.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.899+0000 7fabae19b740 1 -- 172.21.15.16:0/4061965641 shutdown_connections 2024-03-28T22:19:36.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.899+0000 7fabae19b740 1 --2- 172.21.15.16:0/4061965641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e24f9d3ef0 0x55e24f9d62e0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:36.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.899+0000 7fabae19b740 1 --2- 172.21.15.16:0/4061965641 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e24f9cb310 0x55e24f9d39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:36.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.899+0000 7fabae19b740 1 --2- 172.21.15.16:0/4061965641 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24f9ca9f0 0x55e24f9cadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:36.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.899+0000 7fabae19b740 1 -- 172.21.15.16:0/4061965641 >> 172.21.15.16:0/4061965641 conn(0x55e24f9c5120 msgr2=0x55e24f9c5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:36.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.899+0000 7fabae19b740 1 -- 172.21.15.16:0/4061965641 shutdown_connections 2024-03-28T22:19:36.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.899+0000 7fabae19b740 1 -- 172.21.15.16:0/4061965641 wait complete. 2024-03-28T22:19:36.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.899+0000 7fabae19b740 1 -- start start 2024-03-28T22:19:36.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.899+0000 7fabae19b740 1 Processor -- start 2024-03-28T22:19:36.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.900+0000 7fabae19b740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e24f9ca9f0 0x55e24f9d0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:36.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.900+0000 7fabae19b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24f9cb310 0x55e24f9d1410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:36.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.900+0000 7fabae19b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e24f9d3ef0 0x55e24f9d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:36.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.900+0000 7fabae19b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e24f8cc3d0 con 0x55e24f9d3ef0 2024-03-28T22:19:36.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.900+0000 7fabae19b740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55e24f929f40 con 0x55e24f9ca9f0 2024-03-28T22:19:36.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.900+0000 7fabae19b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e24f9d6c80 con 0x55e24f9cb310 2024-03-28T22:19:36.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.900+0000 7fabae19b740 10 client.0 _ll_register_callbacks cb 0x55e24f8e9b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:19:36.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.900+0000 7faba9a7e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e24f9d3ef0 0x55e24f9d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:36.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.900+0000 7faba927d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e24f9ca9f0 0x55e24f9d0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:36.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.900+0000 7faba9a7e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e24f9d3ef0 0x55e24f9d1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58066/0 (socket says 172.21.15.16:58066) 2024-03-28T22:19:36.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.900+0000 7faba9a7e640 1 -- 172.21.15.16:0/1387583046 learned_addr learned my addr 172.21.15.16:0/1387583046 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:36.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.900+0000 7faba8a7c640 1 --2- 172.21.15.16:0/1387583046 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24f9cb310 0x55e24f9d1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:36.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.900+0000 7faba927d640 1 -- 172.21.15.16:0/1387583046 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24f9cb310 msgr2=0x55e24f9d1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:36.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.900+0000 7faba927d640 1 --2- 172.21.15.16:0/1387583046 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24f9cb310 0x55e24f9d1410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:36.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.900+0000 7faba927d640 1 -- 172.21.15.16:0/1387583046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e24f9d3ef0 msgr2=0x55e24f9d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:36.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.900+0000 7faba927d640 1 --2- 172.21.15.16:0/1387583046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e24f9d3ef0 0x55e24f9d1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:36.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.900+0000 7faba927d640 1 -- 172.21.15.16:0/1387583046 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e24f76af70 con 0x55e24f9ca9f0 2024-03-28T22:19:36.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.900+0000 7faba9a7e640 1 --2- 172.21.15.16:0/1387583046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e24f9d3ef0 0x55e24f9d1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:19:36.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.900+0000 7faba927d640 1 --2- 172.21.15.16:0/1387583046 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e24f9ca9f0 0x55e24f9d0ed0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7faba0000c00 tx=0x7faba002c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da950d07f4c44d03 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:36.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.900+0000 7fab8cff9640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-03-28T22:19:36.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.901+0000 7fab8cff9640 1 -- 172.21.15.16:0/1387583046 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba0044070 con 0x55e24f9ca9f0 2024-03-28T22:19:36.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.901+0000 7fab8cff9640 1 -- 172.21.15.16:0/1387583046 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7faba00046c0 con 0x55e24f9ca9f0 2024-03-28T22:19:36.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.901+0000 7fabae19b740 1 -- 172.21.15.16:0/1387583046 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab9c009d70 con 0x55e24f9ca9f0 2024-03-28T22:19:36.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.901+0000 7fab8cff9640 1 -- 172.21.15.16:0/1387583046 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7faba00048c0 con 0x55e24f9ca9f0 2024-03-28T22:19:36.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.901+0000 7fabae19b740 10 client.27338 fetch_fsmap learned FSMap version 27 2024-03-28T22:19:36.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.901+0000 7fabae19b740 1 -- 172.21.15.16:0/1387583046 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e24f602170 con 0x55e24f9ca9f0 2024-03-28T22:19:36.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.901+0000 7fab8cff9640 1 -- 172.21.15.16:0/1387583046 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab9c009d70 con 0x55e24f9ca9f0 2024-03-28T22:19:36.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.901+0000 7fab8cff9640 1 -- 172.21.15.16:0/1387583046 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7faba0003780 con 0x55e24f9ca9f0 2024-03-28T22:19:36.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.901+0000 7fabae19b740 10 client.27338 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:19:36.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.901+0000 7fabae19b740 10 client.27338 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:19:36.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.901+0000 7fabae19b740 20 client.27338 populate_metadata read hostname 'smithi016' 2024-03-28T22:19:36.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.901+0000 7fabae19b740 1 --2- 172.21.15.16:0/1387583046 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e24f9e3d60 0x55e24f9e6170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:36.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.901+0000 7fabae19b740 4 client.27338 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:19:36.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.901+0000 7fabae19b740 1 -- 172.21.15.16:0/1387583046 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e24f93b3e0 con 0x55e24f9e3d60 2024-03-28T22:19:36.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.901+0000 7faba8a7c640 1 --2- 172.21.15.16:0/1387583046 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e24f9e3d60 0x55e24f9e6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:36.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.902+0000 7faba8a7c640 1 --2- 172.21.15.16:0/1387583046 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e24f9e3d60 0x55e24f9e6170 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3343126423cd34c9 server_cookie=48db6ae4d72124e1 in_seq=0 out_seq=0 2024-03-28T22:19:36.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.902+0000 7fab8cff9640 10 client.27338 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:19:36.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.902+0000 7fab8cff9640 1 -- 172.21.15.16:0/1387583046 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e24f93b3e0 con 0x55e24f9e3d60 2024-03-28T22:19:36.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.902+0000 7fab8cff9640 10 client.27338 handle_command_reply: tid=0 2024-03-28T22:19:36.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.902+0000 7fabae19b740 1 client.27338 shutdown 2024-03-28T22:19:36.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.902+0000 7fabae19b740 1 -- 172.21.15.16:0/1387583046 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e24f9ca9f0 msgr2=0x55e24f9d0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:36.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.902+0000 7fabae19b740 1 --2- 172.21.15.16:0/1387583046 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e24f9ca9f0 0x55e24f9d0ed0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7faba0000c00 tx=0x7faba002c880 comp rx=0 tx=0).stop 2024-03-28T22:19:36.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.902+0000 7fabae19b740 1 -- 172.21.15.16:0/1387583046 shutdown_connections 2024-03-28T22:19:36.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.902+0000 7fabae19b740 1 --2- 172.21.15.16:0/1387583046 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e24f9e3d60 0x55e24f9e6170 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:36.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.902+0000 7fabae19b740 1 --2- 172.21.15.16:0/1387583046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e24f9d3ef0 0x55e24f9d1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:36.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.902+0000 7fabae19b740 1 --2- 172.21.15.16:0/1387583046 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24f9cb310 0x55e24f9d1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:36.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.902+0000 7fabae19b740 1 --2- 172.21.15.16:0/1387583046 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e24f9ca9f0 0x55e24f9d0ed0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:36.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.902+0000 7fab8cff9640 0 client.27338 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:19:36.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.902+0000 7fabae19b740 1 -- 172.21.15.16:0/1387583046 >> 172.21.15.16:0/1387583046 conn(0x55e24f9c5120 msgr2=0x55e24f9c6e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:36.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.903+0000 7fabae19b740 1 -- 172.21.15.16:0/1387583046 shutdown_connections 2024-03-28T22:19:36.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.903+0000 7fabae19b740 1 -- 172.21.15.16:0/1387583046 wait complete. 2024-03-28T22:19:36.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.903+0000 7fabae19b740 20 client.27338 trim_cache size 0 max 16384 2024-03-28T22:19:36.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.920+0000 7fabae19b740 1 Processor -- start 2024-03-28T22:19:36.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.920+0000 7fabae19b740 1 -- start start 2024-03-28T22:19:36.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.921+0000 7fabae19b740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e24f9ca9f0 0x55e24f9d3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:36.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.921+0000 7fabae19b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e24f9cb310 0x55e24f9df900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:36.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.921+0000 7fabae19b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24f9d3ef0 0x55e24f9cfd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:36.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.921+0000 7fabae19b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faba41b1e10 con 0x55e24f9cb310 2024-03-28T22:19:36.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.921+0000 7fabae19b740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7faba41af6e0 con 0x55e24f9ca9f0 2024-03-28T22:19:36.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.921+0000 7fabae19b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7faba41ad550 con 0x55e24f9d3ef0 2024-03-28T22:19:36.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.921+0000 7faba8a7c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e24f9cb310 0x55e24f9df900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:36.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.921+0000 7faba8a7c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e24f9cb310 0x55e24f9df900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58068/0 (socket says 172.21.15.16:58068) 2024-03-28T22:19:36.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.921+0000 7faba8a7c640 1 -- 172.21.15.16:0/2124167113 learned_addr learned my addr 172.21.15.16:0/2124167113 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:36.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.921+0000 7faba927d640 1 --2- 172.21.15.16:0/2124167113 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e24f9ca9f0 0x55e24f9d3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:36.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.921+0000 7faba8a7c640 1 -- 172.21.15.16:0/2124167113 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24f9d3ef0 msgr2=0x55e24f9cfd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:36.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.921+0000 7faba8a7c640 1 --2- 172.21.15.16:0/2124167113 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24f9d3ef0 0x55e24f9cfd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:36.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.921+0000 7faba8a7c640 1 -- 172.21.15.16:0/2124167113 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e24f9ca9f0 msgr2=0x55e24f9d3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:36.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.921+0000 7faba8a7c640 1 --2- 172.21.15.16:0/2124167113 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e24f9ca9f0 0x55e24f9d3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:36.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.921+0000 7faba8a7c640 1 -- 172.21.15.16:0/2124167113 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e24f9cbc90 con 0x55e24f9cb310 2024-03-28T22:19:36.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.921+0000 7faba8a7c640 1 --2- 172.21.15.16:0/2124167113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e24f9cb310 0x55e24f9df900 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fab9409a260 tx=0x7fab940048a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efa7cbaa23bca51a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:36.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.921+0000 7fab8effd640 1 -- 172.21.15.16:0/2124167113 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab94020040 con 0x55e24f9cb310 2024-03-28T22:19:36.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.921+0000 7fab8effd640 1 -- 172.21.15.16:0/2124167113 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab940a2c50 con 0x55e24f9cb310 2024-03-28T22:19:36.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.921+0000 7fab8effd640 1 -- 172.21.15.16:0/2124167113 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab940bb410 con 0x55e24f9cb310 2024-03-28T22:19:36.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.922+0000 7fabae19b740 1 -- 172.21.15.16:0/2124167113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e24f9cb310 msgr2=0x55e24f9df900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:36.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.922+0000 7fabae19b740 1 --2- 172.21.15.16:0/2124167113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e24f9cb310 0x55e24f9df900 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fab9409a260 tx=0x7fab940048a0 comp rx=0 tx=0).stop 2024-03-28T22:19:36.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.922+0000 7fabae19b740 1 -- 172.21.15.16:0/2124167113 shutdown_connections 2024-03-28T22:19:36.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.922+0000 7fabae19b740 1 --2- 172.21.15.16:0/2124167113 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24f9d3ef0 0x55e24f9cfd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:36.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.922+0000 7fabae19b740 1 --2- 172.21.15.16:0/2124167113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e24f9cb310 0x55e24f9df900 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:36.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.922+0000 7fabae19b740 1 --2- 172.21.15.16:0/2124167113 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e24f9ca9f0 0x55e24f9d3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:36.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.922+0000 7fabae19b740 1 -- 172.21.15.16:0/2124167113 >> 172.21.15.16:0/2124167113 conn(0x55e24f9c5120 msgr2=0x55e24f9cf670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:36.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.922+0000 7fabae19b740 1 -- 172.21.15.16:0/2124167113 shutdown_connections 2024-03-28T22:19:36.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.922+0000 7fabae19b740 1 -- 172.21.15.16:0/2124167113 wait complete. 2024-03-28T22:19:36.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.922+0000 7fabae19b740 1 -- start start 2024-03-28T22:19:36.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.922+0000 7fabae19b740 1 Processor -- start 2024-03-28T22:19:36.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.923+0000 7fabae19b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e24f9ca9f0 0x55e24f9ce2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:36.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.923+0000 7fabae19b740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e24f9cb310 0x55e24f9ce7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:36.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.923+0000 7fabae19b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24f9d3ef0 0x55e24fa43040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:36.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.923+0000 7fabae19b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faba41adba0 con 0x55e24f9ca9f0 2024-03-28T22:19:36.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.923+0000 7fabae19b740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7faba41af010 con 0x55e24f9cb310 2024-03-28T22:19:36.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.923+0000 7fabae19b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7faba41ae270 con 0x55e24f9d3ef0 2024-03-28T22:19:36.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.923+0000 7fabae19b740 10 client.0 _ll_register_callbacks cb 0x7faba00044b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:19:36.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.923+0000 7faba927d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e24f9ca9f0 0x55e24f9ce2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:36.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.923+0000 7faba927d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e24f9ca9f0 0x55e24f9ce2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58078/0 (socket says 172.21.15.16:58078) 2024-03-28T22:19:36.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.923+0000 7faba927d640 1 -- 172.21.15.16:0/1127904149 learned_addr learned my addr 172.21.15.16:0/1127904149 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:36.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.923+0000 7faba8a7c640 1 --2- 172.21.15.16:0/1127904149 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e24f9cb310 0x55e24f9ce7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:36.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.923+0000 7faba9a7e640 1 --2- 172.21.15.16:0/1127904149 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24f9d3ef0 0x55e24fa43040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:36.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.923+0000 7faba927d640 1 -- 172.21.15.16:0/1127904149 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24f9d3ef0 msgr2=0x55e24fa43040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:36.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.923+0000 7faba927d640 1 --2- 172.21.15.16:0/1127904149 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24f9d3ef0 0x55e24fa43040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:36.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.923+0000 7faba927d640 1 -- 172.21.15.16:0/1127904149 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e24f9cb310 msgr2=0x55e24f9ce7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:36.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.923+0000 7faba927d640 1 --2- 172.21.15.16:0/1127904149 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e24f9cb310 0x55e24f9ce7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:36.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.923+0000 7faba927d640 1 -- 172.21.15.16:0/1127904149 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e24fa43600 con 0x55e24f9ca9f0 2024-03-28T22:19:36.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.923+0000 7faba9a7e640 1 --2- 172.21.15.16:0/1127904149 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24f9d3ef0 0x55e24fa43040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:19:36.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.923+0000 7faba927d640 1 --2- 172.21.15.16:0/1127904149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e24f9ca9f0 0x55e24f9ce2a0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7faba0003bd0 tx=0x7faba0009830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99c96bdf066eb4c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:36.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.923+0000 7fab8cff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:19:36.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.924+0000 7fab8cff9640 1 -- 172.21.15.16:0/1127904149 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba0055040 con 0x55e24f9ca9f0 2024-03-28T22:19:36.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.924+0000 7fab8cff9640 1 -- 172.21.15.16:0/1127904149 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7faba0034560 con 0x55e24f9ca9f0 2024-03-28T22:19:36.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.924+0000 7fabae19b740 1 -- 172.21.15.16:0/1127904149 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab940a2c50 con 0x55e24f9ca9f0 2024-03-28T22:19:36.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.924+0000 7fab8cff9640 1 -- 172.21.15.16:0/1127904149 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7faba0044040 con 0x55e24f9ca9f0 2024-03-28T22:19:36.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.924+0000 7fabae19b740 10 client.17442 fetch_fsmap learned FSMap version 27 2024-03-28T22:19:36.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.924+0000 7fabae19b740 1 -- 172.21.15.16:0/1127904149 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e24fa4a9d0 con 0x55e24f9ca9f0 2024-03-28T22:19:36.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.924+0000 7fab8cff9640 1 -- 172.21.15.16:0/1127904149 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab940a2c50 con 0x55e24f9ca9f0 2024-03-28T22:19:36.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.924+0000 7fab8cff9640 1 -- 172.21.15.16:0/1127904149 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7faba005a600 con 0x55e24f9ca9f0 2024-03-28T22:19:36.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.924+0000 7fabae19b740 10 client.17442 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:19:36.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.924+0000 7fabae19b740 10 client.17442 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:19:36.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.924+0000 7fabae19b740 20 client.17442 populate_metadata read hostname 'smithi016' 2024-03-28T22:19:36.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.924+0000 7fabae19b740 1 --2- 172.21.15.16:0/1127904149 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e24fa4b180 0x55e24fa4d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:36.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.924+0000 7fabae19b740 4 client.17442 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:19:36.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.924+0000 7fabae19b740 1 -- 172.21.15.16:0/1127904149 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55e24f935730 con 0x55e24fa4b180 2024-03-28T22:19:36.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.924+0000 7faba8a7c640 1 --2- 172.21.15.16:0/1127904149 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e24fa4b180 0x55e24fa4d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:36.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.925+0000 7fab8cff9640 10 client.17442 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:19:36.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.925+0000 7faba8a7c640 1 --2- 172.21.15.16:0/1127904149 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e24fa4b180 0x55e24fa4d5a0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa54555c8412d526 server_cookie=bfa3179f5a3e445e in_seq=0 out_seq=0 2024-03-28T22:19:36.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.926+0000 7fab8cff9640 1 -- 172.21.15.16:0/1127904149 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55e24f935730 con 0x55e24fa4b180 2024-03-28T22:19:36.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.926+0000 7fab8cff9640 10 client.17442 handle_command_reply: tid=0 2024-03-28T22:19:36.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.926+0000 7fabae19b740 1 client.17442 shutdown 2024-03-28T22:19:36.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.926+0000 7fabae19b740 1 -- 172.21.15.16:0/1127904149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e24f9ca9f0 msgr2=0x55e24f9ce2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:36.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.926+0000 7fabae19b740 1 --2- 172.21.15.16:0/1127904149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e24f9ca9f0 0x55e24f9ce2a0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7faba0003bd0 tx=0x7faba0009830 comp rx=0 tx=0).stop 2024-03-28T22:19:36.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.926+0000 7fabae19b740 1 -- 172.21.15.16:0/1127904149 shutdown_connections 2024-03-28T22:19:36.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.926+0000 7fabae19b740 1 --2- 172.21.15.16:0/1127904149 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55e24fa4b180 0x55e24fa4d5a0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:36.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.926+0000 7fabae19b740 1 --2- 172.21.15.16:0/1127904149 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24f9d3ef0 0x55e24fa43040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:36.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.926+0000 7fabae19b740 1 --2- 172.21.15.16:0/1127904149 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e24f9cb310 0x55e24f9ce7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:36.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.926+0000 7fabae19b740 1 --2- 172.21.15.16:0/1127904149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e24f9ca9f0 0x55e24f9ce2a0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:36.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.926+0000 7fab8cff9640 0 client.17442 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:19:36.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.926+0000 7fabae19b740 1 -- 172.21.15.16:0/1127904149 >> 172.21.15.16:0/1127904149 conn(0x55e24f9c5120 msgr2=0x55e24f9cff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:36.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.926+0000 7fabae19b740 1 -- 172.21.15.16:0/1127904149 shutdown_connections 2024-03-28T22:19:36.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.926+0000 7fabae19b740 1 -- 172.21.15.16:0/1127904149 wait complete. 2024-03-28T22:19:36.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.926+0000 7fabae19b740 20 client.17442 trim_cache size 0 max 16384 2024-03-28T22:19:36.936 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:19:36.936 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-03-28T22:19:36.936 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "d437028f-3977-44ff-b7f7-7f2c6cbb2ce3", 2024-03-28T22:19:36.936 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-03-28T22:19:36.937 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:19:36.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.927+0000 7fabab4e1640 1 -- 172.21.15.16:0/390893947 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fab900a1f70 msgr2=0x7fab900a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:36.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.927+0000 7fabab4e1640 1 --2- 172.21.15.16:0/390893947 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fab900a1f70 0x7fab900a4430 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fab9c00fc90 tx=0x7fab9c018000 comp rx=0 tx=0).stop 2024-03-28T22:19:36.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.927+0000 7fabab4e1640 1 -- 172.21.15.16:0/390893947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faba41021e0 msgr2=0x7faba40fdad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:36.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.927+0000 7fabab4e1640 1 --2- 172.21.15.16:0/390893947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faba41021e0 0x7faba40fdad0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fab9400d770 tx=0x7fab9400dc30 comp rx=0 tx=0).stop 2024-03-28T22:19:36.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.927+0000 7fabab4e1640 1 -- 172.21.15.16:0/390893947 shutdown_connections 2024-03-28T22:19:36.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.927+0000 7fabab4e1640 1 --2- 172.21.15.16:0/390893947 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fab900a1f70 0x7fab900a4430 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:36.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.927+0000 7fabab4e1640 1 --2- 172.21.15.16:0/390893947 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faba4102b80 0x7faba40fe010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:36.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.927+0000 7fabab4e1640 1 --2- 172.21.15.16:0/390893947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faba41021e0 0x7faba40fdad0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:36.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.927+0000 7fabab4e1640 1 --2- 172.21.15.16:0/390893947 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faba41009d0 0x7faba40fd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:36.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.927+0000 7fabab4e1640 1 -- 172.21.15.16:0/390893947 >> 172.21.15.16:0/390893947 conn(0x7faba4077830 msgr2=0x7faba406a150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:36.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.927+0000 7fabab4e1640 1 -- 172.21.15.16:0/390893947 shutdown_connections 2024-03-28T22:19:36.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:36.927+0000 7fabab4e1640 1 -- 172.21.15.16:0/390893947 wait complete. 2024-03-28T22:19:36.940 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:19:36.940 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:19:37.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.046+0000 7f870ec84640 1 -- 172.21.15.16:0/3555748740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8708106680 msgr2=0x7f8708106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:37.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.046+0000 7f870ec84640 1 --2- 172.21.15.16:0/3555748740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8708106680 0x7f8708106a60 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f86fc009930 tx=0x7f86fc02bc30 comp rx=0 tx=0).stop 2024-03-28T22:19:37.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.046+0000 7f870ec84640 1 -- 172.21.15.16:0/3555748740 shutdown_connections 2024-03-28T22:19:37.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.046+0000 7f870ec84640 1 --2- 172.21.15.16:0/3555748740 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8708100fa0 0x7f870810d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:37.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.046+0000 7f870ec84640 1 --2- 172.21.15.16:0/3555748740 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87081005e0 0x7f8708100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:37.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.046+0000 7f870ec84640 1 --2- 172.21.15.16:0/3555748740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8708106680 0x7f8708106a60 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:37.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.046+0000 7f870ec84640 1 -- 172.21.15.16:0/3555748740 >> 172.21.15.16:0/3555748740 conn(0x7f8708077770 msgr2=0x7f8708077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:37.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.046+0000 7f870ec84640 1 -- 172.21.15.16:0/3555748740 shutdown_connections 2024-03-28T22:19:37.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.046+0000 7f870ec84640 1 -- 172.21.15.16:0/3555748740 wait complete. 2024-03-28T22:19:37.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.046+0000 7f870ec84640 1 Processor -- start 2024-03-28T22:19:37.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.046+0000 7f870ec84640 1 -- start start 2024-03-28T22:19:37.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.047+0000 7f870ec84640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87081005e0 0x7f8708198dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:37.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.047+0000 7f870ec84640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8708100fa0 0x7f8708199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:37.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.047+0000 7f870ec84640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8708106680 0x7f87081a3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:37.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.047+0000 7f870ec84640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8708110a00 con 0x7f87081005e0 2024-03-28T22:19:37.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.047+0000 7f870ec84640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8708110880 con 0x7f8708100fa0 2024-03-28T22:19:37.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.047+0000 7f870ec84640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8708110b80 con 0x7f8708106680 2024-03-28T22:19:37.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.047+0000 7f870ca20640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87081005e0 0x7f8708198dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:37.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.047+0000 7f870ca20640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87081005e0 0x7f8708198dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58094/0 (socket says 172.21.15.16:58094) 2024-03-28T22:19:37.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.047+0000 7f870ca20640 1 -- 172.21.15.16:0/3766275985 learned_addr learned my addr 172.21.15.16:0/3766275985 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:37.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.047+0000 7f8707fff640 1 --2- 172.21.15.16:0/3766275985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8708100fa0 0x7f8708199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:37.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.047+0000 7f870d221640 1 --2- 172.21.15.16:0/3766275985 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8708106680 0x7f87081a3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:37.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.047+0000 7f870ca20640 1 -- 172.21.15.16:0/3766275985 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8708106680 msgr2=0x7f87081a3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:37.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.047+0000 7f870ca20640 1 --2- 172.21.15.16:0/3766275985 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8708106680 0x7f87081a3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:37.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.047+0000 7f870ca20640 1 -- 172.21.15.16:0/3766275985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8708100fa0 msgr2=0x7f8708199300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:37.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.047+0000 7f870ca20640 1 --2- 172.21.15.16:0/3766275985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8708100fa0 0x7f8708199300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:37.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.047+0000 7f870ca20640 1 -- 172.21.15.16:0/3766275985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87081a4380 con 0x7f87081005e0 2024-03-28T22:19:37.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.047+0000 7f870ca20640 1 --2- 172.21.15.16:0/3766275985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87081005e0 0x7f8708198dc0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f86fc009230 tx=0x7f86fc02cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:37.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.047+0000 7f8705ffb640 1 -- 172.21.15.16:0/3766275985 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86fc041070 con 0x7f87081005e0 2024-03-28T22:19:37.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.047+0000 7f8705ffb640 1 -- 172.21.15.16:0/3766275985 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86fc004450 con 0x7f87081005e0 2024-03-28T22:19:37.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.047+0000 7f870ec84640 1 -- 172.21.15.16:0/3766275985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87081a4550 con 0x7f87081005e0 2024-03-28T22:19:37.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.047+0000 7f8705ffb640 1 -- 172.21.15.16:0/3766275985 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86fc034440 con 0x7f87081005e0 2024-03-28T22:19:37.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.048+0000 7f870ec84640 1 -- 172.21.15.16:0/3766275985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87081a4a10 con 0x7f87081005e0 2024-03-28T22:19:37.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.050+0000 7f8705ffb640 1 -- 172.21.15.16:0/3766275985 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f86fc044070 con 0x7f87081005e0 2024-03-28T22:19:37.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.050+0000 7f8711869740 1 Processor -- start 2024-03-28T22:19:37.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.050+0000 7f8711869740 1 -- start start 2024-03-28T22:19:37.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.051+0000 7f8711869740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558e2c401290 0x558e2c401670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:37.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.051+0000 7f8711869740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558e2c401bb0 0x558e2c40a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:37.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.051+0000 7f8711869740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e2c40a790 0x558e2c40cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:37.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.051+0000 7f8711869740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558e2c365180 con 0x558e2c40a790 2024-03-28T22:19:37.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.051+0000 7f8711869740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x558e2c30db70 con 0x558e2c401bb0 2024-03-28T22:19:37.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.051+0000 7f8711869740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x558e2c361220 con 0x558e2c401290 2024-03-28T22:19:37.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.051+0000 7f8705ffb640 1 --2- 172.21.15.16:0/3766275985 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f86f80a1f70 0x7f86f80a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:37.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.051+0000 7f8707fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e2c40a790 0x558e2c40cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:37.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.051+0000 7f870ca20640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558e2c401bb0 0x558e2c40a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:37.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.051+0000 7f8707fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e2c40a790 0x558e2c40cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58096/0 (socket says 172.21.15.16:58096) 2024-03-28T22:19:37.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.051+0000 7f8707fff640 1 -- 172.21.15.16:0/3558615903 learned_addr learned my addr 172.21.15.16:0/3558615903 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:37.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.051+0000 7f870d221640 1 --2- 172.21.15.16:0/3766275985 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f86f80a1f70 0x7f86f80a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:37.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.051+0000 7f870d221640 1 --2- 172.21.15.16:0/3558615903 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558e2c401290 0x558e2c401670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:37.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.051+0000 7f8705ffb640 1 -- 172.21.15.16:0/3766275985 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7f86fc0efc40 con 0x7f87081005e0 2024-03-28T22:19:37.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.051+0000 7f8707fff640 1 -- 172.21.15.16:0/3558615903 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558e2c401290 msgr2=0x558e2c401670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:37.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.051+0000 7f8707fff640 1 --2- 172.21.15.16:0/3558615903 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558e2c401290 0x558e2c401670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:37.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.051+0000 7f8707fff640 1 -- 172.21.15.16:0/3558615903 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558e2c401bb0 msgr2=0x558e2c40a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:37.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.051+0000 7f8707fff640 1 --2- 172.21.15.16:0/3558615903 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558e2c401bb0 0x558e2c40a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:37.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.051+0000 7f8707fff640 1 -- 172.21.15.16:0/3558615903 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e2c3217c0 con 0x558e2c40a790 2024-03-28T22:19:37.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.051+0000 7f870d221640 1 --2- 172.21.15.16:0/3558615903 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558e2c401290 0x558e2c401670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:19:37.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.051+0000 7f870d221640 1 --2- 172.21.15.16:0/3766275985 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f86f80a1f70 0x7f86f80a4430 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8700005c40 tx=0x7f87000094f0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:37.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.051+0000 7f8707fff640 1 --2- 172.21.15.16:0/3558615903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e2c40a790 0x558e2c40cb80 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f86f4009c90 tx=0x7f86f4011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=458d9ec69330c6d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:37.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.052+0000 7f86f2ffd640 1 -- 172.21.15.16:0/3558615903 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86f401e070 con 0x558e2c40a790 2024-03-28T22:19:37.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.052+0000 7f86f2ffd640 1 -- 172.21.15.16:0/3558615903 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86f4019b10 con 0x558e2c40a790 2024-03-28T22:19:37.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.052+0000 7f86f2ffd640 1 -- 172.21.15.16:0/3558615903 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86f400ea60 con 0x558e2c40a790 2024-03-28T22:19:37.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.052+0000 7f8711869740 1 -- 172.21.15.16:0/3558615903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e2c40a790 msgr2=0x558e2c40cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:37.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.052+0000 7f8711869740 1 --2- 172.21.15.16:0/3558615903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e2c40a790 0x558e2c40cb80 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f86f4009c90 tx=0x7f86f4011810 comp rx=0 tx=0).stop 2024-03-28T22:19:37.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.052+0000 7f8711869740 1 -- 172.21.15.16:0/3558615903 shutdown_connections 2024-03-28T22:19:37.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.052+0000 7f8711869740 1 --2- 172.21.15.16:0/3558615903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e2c40a790 0x558e2c40cb80 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:37.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.052+0000 7f8711869740 1 --2- 172.21.15.16:0/3558615903 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558e2c401bb0 0x558e2c40a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:37.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.052+0000 7f8711869740 1 --2- 172.21.15.16:0/3558615903 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558e2c401290 0x558e2c401670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:37.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.052+0000 7f8711869740 1 -- 172.21.15.16:0/3558615903 >> 172.21.15.16:0/3558615903 conn(0x558e2c3fb9c0 msgr2=0x558e2c3fbda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:37.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.052+0000 7f8711869740 1 -- 172.21.15.16:0/3558615903 shutdown_connections 2024-03-28T22:19:37.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.052+0000 7f8711869740 1 -- 172.21.15.16:0/3558615903 wait complete. 2024-03-28T22:19:37.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.053+0000 7f8711869740 1 -- start start 2024-03-28T22:19:37.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.053+0000 7f8711869740 1 Processor -- start 2024-03-28T22:19:37.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.053+0000 7f8711869740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558e2c401290 0x558e2c407720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:37.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.053+0000 7f8711869740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558e2c401bb0 0x558e2c407c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:37.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.053+0000 7f8711869740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e2c40a790 0x558e2c4081a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:37.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.053+0000 7f8711869740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558e2c40d580 con 0x558e2c40a790 2024-03-28T22:19:37.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.053+0000 7f8711869740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x558e2c3034a0 con 0x558e2c401290 2024-03-28T22:19:37.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.053+0000 7f8711869740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x558e2c40d700 con 0x558e2c401bb0 2024-03-28T22:19:37.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.053+0000 7f8711869740 10 client.0 _ll_register_callbacks cb 0x558e2c306d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:19:37.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.053+0000 7f870d221640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e2c40a790 0x558e2c4081a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:37.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.053+0000 7f870ca20640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558e2c401290 0x558e2c407720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:37.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.053+0000 7f870d221640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e2c40a790 0x558e2c4081a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58112/0 (socket says 172.21.15.16:58112) 2024-03-28T22:19:37.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.053+0000 7f870d221640 1 -- 172.21.15.16:0/952352025 learned_addr learned my addr 172.21.15.16:0/952352025 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:37.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.053+0000 7f8707fff640 1 --2- 172.21.15.16:0/952352025 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558e2c401bb0 0x558e2c407c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:37.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.053+0000 7f870d221640 1 -- 172.21.15.16:0/952352025 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558e2c401bb0 msgr2=0x558e2c407c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:37.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.053+0000 7f870d221640 1 --2- 172.21.15.16:0/952352025 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558e2c401bb0 0x558e2c407c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:37.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.053+0000 7f870d221640 1 -- 172.21.15.16:0/952352025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558e2c401290 msgr2=0x558e2c407720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:37.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.053+0000 7f870d221640 1 --2- 172.21.15.16:0/952352025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558e2c401290 0x558e2c407720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:37.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.053+0000 7f870d221640 1 -- 172.21.15.16:0/952352025 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e2c1437d0 con 0x558e2c40a790 2024-03-28T22:19:37.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.054+0000 7f870d221640 1 --2- 172.21.15.16:0/952352025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e2c40a790 0x558e2c4081a0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f87000125e0 tx=0x7f8700004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7def7270eec8b116 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:37.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.054+0000 7f86e7fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:19:37.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.054+0000 7f86e7fff640 1 -- 172.21.15.16:0/952352025 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f870001c960 con 0x558e2c40a790 2024-03-28T22:19:37.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.054+0000 7f86e7fff640 1 -- 172.21.15.16:0/952352025 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7f870001cb00 con 0x558e2c40a790 2024-03-28T22:19:37.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.054+0000 7f8711869740 1 -- 172.21.15.16:0/952352025 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86f4019b10 con 0x558e2c40a790 2024-03-28T22:19:37.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.054+0000 7f86e7fff640 1 -- 172.21.15.16:0/952352025 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f870001cd00 con 0x558e2c40a790 2024-03-28T22:19:37.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.054+0000 7f8711869740 10 client.17454 fetch_fsmap learned FSMap version 27 2024-03-28T22:19:37.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.054+0000 7f8711869740 1 -- 172.21.15.16:0/952352025 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558e2c0793f0 con 0x558e2c40a790 2024-03-28T22:19:37.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.054+0000 7f86e7fff640 1 -- 172.21.15.16:0/952352025 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86f4019b10 con 0x558e2c40a790 2024-03-28T22:19:37.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.054+0000 7f86e7fff640 1 -- 172.21.15.16:0/952352025 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f8700025430 con 0x558e2c40a790 2024-03-28T22:19:37.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.054+0000 7f8711869740 10 client.17454 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:19:37.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.054+0000 7f8711869740 10 client.17454 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:19:37.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.054+0000 7f8711869740 20 client.17454 populate_metadata read hostname 'smithi016' 2024-03-28T22:19:37.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.054+0000 7f8711869740 1 --2- 172.21.15.16:0/952352025 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558e2c41ab60 0x558e2c41cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:37.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.054+0000 7f8711869740 4 client.17454 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:19:37.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.054+0000 7f8711869740 1 -- 172.21.15.16:0/952352025 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558e2c3726c0 con 0x558e2c41ab60 2024-03-28T22:19:37.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.055+0000 7f870ca20640 1 --2- 172.21.15.16:0/952352025 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558e2c41ab60 0x558e2c41cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:37.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.055+0000 7f86e7fff640 10 client.17454 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:19:37.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.055+0000 7f870ca20640 1 --2- 172.21.15.16:0/952352025 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558e2c41ab60 0x558e2c41cfe0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=790e02b0296edcf0 server_cookie=cbceb340af7d33b0 in_seq=0 out_seq=0 2024-03-28T22:19:37.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.056+0000 7f86e7fff640 1 -- 172.21.15.16:0/952352025 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558e2c3726c0 con 0x558e2c41ab60 2024-03-28T22:19:37.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.056+0000 7f86e7fff640 10 client.17454 handle_command_reply: tid=0 2024-03-28T22:19:37.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.056+0000 7f8711869740 1 client.17454 shutdown 2024-03-28T22:19:37.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.056+0000 7f8711869740 1 -- 172.21.15.16:0/952352025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e2c40a790 msgr2=0x558e2c4081a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:37.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.056+0000 7f8711869740 1 --2- 172.21.15.16:0/952352025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e2c40a790 0x558e2c4081a0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f87000125e0 tx=0x7f8700004ab0 comp rx=0 tx=0).stop 2024-03-28T22:19:37.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.056+0000 7f8711869740 1 -- 172.21.15.16:0/952352025 shutdown_connections 2024-03-28T22:19:37.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.056+0000 7f8711869740 1 --2- 172.21.15.16:0/952352025 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558e2c41ab60 0x558e2c41cfe0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:37.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.056+0000 7f8711869740 1 --2- 172.21.15.16:0/952352025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e2c40a790 0x558e2c4081a0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:37.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.056+0000 7f8711869740 1 --2- 172.21.15.16:0/952352025 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558e2c401bb0 0x558e2c407c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:37.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.056+0000 7f8711869740 1 --2- 172.21.15.16:0/952352025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558e2c401290 0x558e2c407720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:37.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.056+0000 7f86e7fff640 0 client.17454 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:19:37.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.056+0000 7f8711869740 1 -- 172.21.15.16:0/952352025 >> 172.21.15.16:0/952352025 conn(0x558e2c3fb9c0 msgr2=0x558e2c3fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:37.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.056+0000 7f8711869740 1 -- 172.21.15.16:0/952352025 shutdown_connections 2024-03-28T22:19:37.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.056+0000 7f8711869740 1 -- 172.21.15.16:0/952352025 wait complete. 2024-03-28T22:19:37.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.056+0000 7f8711869740 20 client.17454 trim_cache size 0 max 16384 2024-03-28T22:19:37.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.073+0000 7f8711869740 1 Processor -- start 2024-03-28T22:19:37.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.073+0000 7f8711869740 1 -- start start 2024-03-28T22:19:37.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.074+0000 7f8711869740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558e2c401290 0x558e2c3fac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:37.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.074+0000 7f8711869740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558e2c401bb0 0x558e2c416350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:37.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.074+0000 7f8711869740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e2c40a790 0x558e2c4228b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:37.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.074+0000 7f8711869740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f87081afcf0 con 0x558e2c40a790 2024-03-28T22:19:37.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.074+0000 7f8711869740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f87081ad5c0 con 0x558e2c401290 2024-03-28T22:19:37.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.074+0000 7f8711869740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f87081ab430 con 0x558e2c401bb0 2024-03-28T22:19:37.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.074+0000 7f870d221640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e2c40a790 0x558e2c4228b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:37.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.074+0000 7f870d221640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e2c40a790 0x558e2c4228b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58116/0 (socket says 172.21.15.16:58116) 2024-03-28T22:19:37.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.074+0000 7f870d221640 1 -- 172.21.15.16:0/2442041359 learned_addr learned my addr 172.21.15.16:0/2442041359 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:37.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.074+0000 7f870ca20640 1 --2- 172.21.15.16:0/2442041359 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558e2c401290 0x558e2c3fac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:37.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.074+0000 7f870d221640 1 -- 172.21.15.16:0/2442041359 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558e2c401bb0 msgr2=0x558e2c416350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:37.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.074+0000 7f870d221640 1 --2- 172.21.15.16:0/2442041359 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558e2c401bb0 0x558e2c416350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:37.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.074+0000 7f870d221640 1 -- 172.21.15.16:0/2442041359 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558e2c401290 msgr2=0x558e2c3fac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:37.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.074+0000 7f870d221640 1 --2- 172.21.15.16:0/2442041359 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558e2c401290 0x558e2c3fac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:37.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.074+0000 7f870d221640 1 -- 172.21.15.16:0/2442041359 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e2c0390f0 con 0x558e2c40a790 2024-03-28T22:19:37.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.074+0000 7f870d221640 1 --2- 172.21.15.16:0/2442041359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e2c40a790 0x558e2c4228b0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f8700025850 tx=0x7f8700010dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22695735da143c8f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:37.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.074+0000 7f86f27fc640 1 -- 172.21.15.16:0/2442041359 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8700025cc0 con 0x558e2c40a790 2024-03-28T22:19:37.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.074+0000 7f86f27fc640 1 -- 172.21.15.16:0/2442041359 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8700012e70 con 0x558e2c40a790 2024-03-28T22:19:37.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.074+0000 7f86f27fc640 1 -- 172.21.15.16:0/2442041359 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f870002f460 con 0x558e2c40a790 2024-03-28T22:19:37.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.075+0000 7f8711869740 1 -- 172.21.15.16:0/2442041359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e2c40a790 msgr2=0x558e2c4228b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:37.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.075+0000 7f8711869740 1 --2- 172.21.15.16:0/2442041359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e2c40a790 0x558e2c4228b0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f8700025850 tx=0x7f8700010dc0 comp rx=0 tx=0).stop 2024-03-28T22:19:37.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.075+0000 7f8711869740 1 -- 172.21.15.16:0/2442041359 shutdown_connections 2024-03-28T22:19:37.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.075+0000 7f8711869740 1 --2- 172.21.15.16:0/2442041359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e2c40a790 0x558e2c4228b0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:37.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.075+0000 7f8711869740 1 --2- 172.21.15.16:0/2442041359 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558e2c401bb0 0x558e2c416350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:37.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.075+0000 7f8711869740 1 --2- 172.21.15.16:0/2442041359 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558e2c401290 0x558e2c3fac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:37.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.075+0000 7f8711869740 1 -- 172.21.15.16:0/2442041359 >> 172.21.15.16:0/2442041359 conn(0x558e2c3fb9c0 msgr2=0x558e2c4076e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:37.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.075+0000 7f8711869740 1 -- 172.21.15.16:0/2442041359 shutdown_connections 2024-03-28T22:19:37.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.075+0000 7f8711869740 1 -- 172.21.15.16:0/2442041359 wait complete. 2024-03-28T22:19:37.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.075+0000 7f8711869740 1 -- start start 2024-03-28T22:19:37.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.075+0000 7f8711869740 1 Processor -- start 2024-03-28T22:19:37.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.076+0000 7f8711869740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558e2c401290 0x558e2c40e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:37.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.076+0000 7f8711869740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e2c401bb0 0x558e2c40eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:37.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.076+0000 7f8711869740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558e2c40a790 0x558e2c40f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:37.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.076+0000 7f8711869740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f87081aba80 con 0x558e2c401bb0 2024-03-28T22:19:37.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.076+0000 7f8711869740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f87081acef0 con 0x558e2c40a790 2024-03-28T22:19:37.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.076+0000 7f8711869740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f87081ac150 con 0x558e2c401290 2024-03-28T22:19:37.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.076+0000 7f8711869740 10 client.0 _ll_register_callbacks cb 0x7f8700009d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:19:37.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.076+0000 7f8707fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e2c401bb0 0x558e2c40eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:37.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.076+0000 7f8707fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e2c401bb0 0x558e2c40eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58126/0 (socket says 172.21.15.16:58126) 2024-03-28T22:19:37.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.076+0000 7f8707fff640 1 -- 172.21.15.16:0/649221448 learned_addr learned my addr 172.21.15.16:0/649221448 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:19:37.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.076+0000 7f870d221640 1 --2- 172.21.15.16:0/649221448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558e2c40a790 0x558e2c40f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:37.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.076+0000 7f870ca20640 1 --2- 172.21.15.16:0/649221448 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558e2c401290 0x558e2c40e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:37.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.076+0000 7f8707fff640 1 -- 172.21.15.16:0/649221448 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558e2c401290 msgr2=0x558e2c40e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:37.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.076+0000 7f8707fff640 1 --2- 172.21.15.16:0/649221448 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558e2c401290 0x558e2c40e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:37.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.076+0000 7f8707fff640 1 -- 172.21.15.16:0/649221448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558e2c40a790 msgr2=0x558e2c40f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:19:37.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.076+0000 7f8707fff640 1 --2- 172.21.15.16:0/649221448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558e2c40a790 0x558e2c40f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:37.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.076+0000 7f8707fff640 1 -- 172.21.15.16:0/649221448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e2c480200 con 0x558e2c401bb0 2024-03-28T22:19:37.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.076+0000 7f870ca20640 1 --2- 172.21.15.16:0/649221448 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558e2c401290 0x558e2c40e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:19:37.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.076+0000 7f8707fff640 1 --2- 172.21.15.16:0/649221448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e2c401bb0 0x558e2c40eb10 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f86f4009be0 tx=0x7f86f4005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76c8292ec2f0b6ac server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:19:37.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.076+0000 7f86e7fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:19:37.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.076+0000 7f86e7fff640 1 -- 172.21.15.16:0/649221448 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86f401e040 con 0x558e2c401bb0 2024-03-28T22:19:37.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.076+0000 7f86e7fff640 1 -- 172.21.15.16:0/649221448 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7f86f400e750 con 0x558e2c401bb0 2024-03-28T22:19:37.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.076+0000 7f8711869740 1 -- 172.21.15.16:0/649221448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8700012e70 con 0x558e2c401bb0 2024-03-28T22:19:37.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.077+0000 7f86e7fff640 1 -- 172.21.15.16:0/649221448 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f86f400e3b0 con 0x558e2c401bb0 2024-03-28T22:19:37.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.077+0000 7f8711869740 10 client.17460 fetch_fsmap learned FSMap version 27 2024-03-28T22:19:37.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.077+0000 7f8711869740 1 -- 172.21.15.16:0/649221448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558e2c4814b0 con 0x558e2c401bb0 2024-03-28T22:19:37.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.077+0000 7f86e7fff640 1 -- 172.21.15.16:0/649221448 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8700012e70 con 0x558e2c401bb0 2024-03-28T22:19:37.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.077+0000 7f86e7fff640 1 -- 172.21.15.16:0/649221448 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f86f4004b90 con 0x558e2c401bb0 2024-03-28T22:19:37.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.077+0000 7f8711869740 10 client.17460 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:19:37.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.077+0000 7f8711869740 10 client.17460 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:19:37.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.077+0000 7f8711869740 20 client.17460 populate_metadata read hostname 'smithi016' 2024-03-28T22:19:37.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.077+0000 7f8711869740 1 --2- 172.21.15.16:0/649221448 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558e2c481a50 0x558e2c483e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:19:37.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.077+0000 7f8711869740 4 client.17460 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:19:37.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.077+0000 7f8711869740 1 -- 172.21.15.16:0/649221448 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558e2c36ca10 con 0x558e2c481a50 2024-03-28T22:19:37.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.077+0000 7f870ca20640 1 --2- 172.21.15.16:0/649221448 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558e2c481a50 0x558e2c483e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:19:37.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.078+0000 7f86e7fff640 10 client.17460 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:19:37.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.078+0000 7f870ca20640 1 --2- 172.21.15.16:0/649221448 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558e2c481a50 0x558e2c483e70 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ff215fcdd3c9d9e server_cookie=3356a3192c81a82f in_seq=0 out_seq=0 2024-03-28T22:19:37.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.078+0000 7f86e7fff640 1 -- 172.21.15.16:0/649221448 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x558e2c36ca10 con 0x558e2c481a50 2024-03-28T22:19:37.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.078+0000 7f86e7fff640 10 client.17460 handle_command_reply: tid=0 2024-03-28T22:19:37.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.078+0000 7f8711869740 1 client.17460 shutdown 2024-03-28T22:19:37.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.078+0000 7f8711869740 1 -- 172.21.15.16:0/649221448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e2c401bb0 msgr2=0x558e2c40eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:37.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.078+0000 7f8711869740 1 --2- 172.21.15.16:0/649221448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e2c401bb0 0x558e2c40eb10 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f86f4009be0 tx=0x7f86f4005850 comp rx=0 tx=0).stop 2024-03-28T22:19:37.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.078+0000 7f8711869740 1 -- 172.21.15.16:0/649221448 shutdown_connections 2024-03-28T22:19:37.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.078+0000 7f8711869740 1 --2- 172.21.15.16:0/649221448 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x558e2c481a50 0x558e2c483e70 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:37.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.078+0000 7f8711869740 1 --2- 172.21.15.16:0/649221448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558e2c40a790 0x558e2c40f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:37.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.078+0000 7f8711869740 1 --2- 172.21.15.16:0/649221448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558e2c401bb0 0x558e2c40eb10 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:37.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.078+0000 7f86e7fff640 0 client.17460 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:19:37.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.078+0000 7f8711869740 1 --2- 172.21.15.16:0/649221448 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558e2c401290 0x558e2c40e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:37.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.078+0000 7f8711869740 1 -- 172.21.15.16:0/649221448 >> 172.21.15.16:0/649221448 conn(0x558e2c3fb9c0 msgr2=0x558e2c3faaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:37.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.078+0000 7f8711869740 1 -- 172.21.15.16:0/649221448 shutdown_connections 2024-03-28T22:19:37.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.078+0000 7f8711869740 1 -- 172.21.15.16:0/649221448 wait complete. 2024-03-28T22:19:37.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.078+0000 7f8711869740 20 client.17460 trim_cache size 0 max 16384 2024-03-28T22:19:37.090 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:19:37.090 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T22:19:37.090 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-03-28T22:19:37.090 INFO:teuthology.orchestra.run.smithi016.stdout: "d437028f-3977-44ff-b7f7-7f2c6cbb2ce3": { 2024-03-28T22:19:37.090 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-03-28T22:19:37.090 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "d437028f-3977-44ff-b7f7-7f2c6cbb2ce3", 2024-03-28T22:19:37.091 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-03-28T22:19:37.091 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:19:37.091 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:19:37.091 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:19:37.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.079+0000 7f870ec84640 1 -- 172.21.15.16:0/3766275985 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f86f80a1f70 msgr2=0x7f86f80a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:37.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.079+0000 7f870ec84640 1 --2- 172.21.15.16:0/3766275985 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f86f80a1f70 0x7f86f80a4430 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8700005c40 tx=0x7f87000094f0 comp rx=0 tx=0).stop 2024-03-28T22:19:37.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.079+0000 7f870ec84640 1 -- 172.21.15.16:0/3766275985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87081005e0 msgr2=0x7f8708198dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:19:37.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.079+0000 7f870ec84640 1 --2- 172.21.15.16:0/3766275985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87081005e0 0x7f8708198dc0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f86fc009230 tx=0x7f86fc02cbf0 comp rx=0 tx=0).stop 2024-03-28T22:19:37.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.079+0000 7f870ec84640 1 -- 172.21.15.16:0/3766275985 shutdown_connections 2024-03-28T22:19:37.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.079+0000 7f870ec84640 1 --2- 172.21.15.16:0/3766275985 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f86f80a1f70 0x7f86f80a4430 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:37.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.079+0000 7f870ec84640 1 --2- 172.21.15.16:0/3766275985 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8708106680 0x7f87081a3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:37.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.079+0000 7f870ec84640 1 --2- 172.21.15.16:0/3766275985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8708100fa0 0x7f8708199300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:37.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.079+0000 7f870ec84640 1 --2- 172.21.15.16:0/3766275985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87081005e0 0x7f8708198dc0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:19:37.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.079+0000 7f870ec84640 1 -- 172.21.15.16:0/3766275985 >> 172.21.15.16:0/3766275985 conn(0x7f8708077770 msgr2=0x7f87080ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:19:37.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.079+0000 7f870ec84640 1 -- 172.21.15.16:0/3766275985 shutdown_connections 2024-03-28T22:19:37.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:19:37.079+0000 7f870ec84640 1 -- 172.21.15.16:0/3766275985 wait complete. 2024-03-28T22:19:37.094 INFO:tasks.cephfs.filesystem:scrub status for tag:d437028f-3977-44ff-b7f7-7f2c6cbb2ce3 - {'path': '/', 'tag': 'd437028f-3977-44ff-b7f7-7f2c6cbb2ce3', 'options': 'recursive,force'} 2024-03-28T22:19:37.430 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:37 smithi063 ceph-mon[36249]: scrub queued for path: / 2024-03-28T22:19:37.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:37 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:19:37.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:37 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:19:37.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:37 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:19:37.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:37 smithi016 ceph-mon[32374]: scrub queued for path: / 2024-03-28T22:19:37.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:37 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:19:37.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:37 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:19:37.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:37 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:19:37.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:37 smithi053 ceph-mon[35979]: scrub queued for path: / 2024-03-28T22:19:37.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:37 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:19:37.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:37 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:19:37.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:37 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:19:38.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:38 smithi063 ceph-mon[36249]: pgmap v280: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:19:38.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:38 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:19:38.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:38 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:19:38.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:38 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:19:38.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:38 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:19:38.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:38 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:19:38.432 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:38 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:19:38.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:38 smithi016 ceph-mon[32374]: pgmap v280: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:19:38.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:38 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:19:38.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:38 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:19:38.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:38 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:19:38.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:38 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:19:38.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:38 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:19:38.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:38 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:19:38.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:38 smithi053 ceph-mon[35979]: pgmap v280: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:19:38.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:38 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:19:38.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:38 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:19:38.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:38 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:19:38.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:38 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:19:38.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:38 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:19:38.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:38 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:19:40.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:40 smithi063 ceph-mon[36249]: pgmap v281: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:19:40.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:40 smithi016 ceph-mon[32374]: pgmap v281: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:19:40.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:40 smithi053 ceph-mon[35979]: pgmap v281: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:19:42.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:42 smithi063 ceph-mon[36249]: pgmap v282: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:19:42.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:42 smithi016 ceph-mon[32374]: pgmap v282: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:19:42.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:42 smithi053 ceph-mon[35979]: pgmap v282: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:19:44.430 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:44 smithi063 ceph-mon[36249]: pgmap v283: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:19:44.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:44 smithi016 ceph-mon[32374]: pgmap v283: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:19:44.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:44 smithi053 ceph-mon[35979]: pgmap v283: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:19:46.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:46 smithi063 ceph-mon[36249]: pgmap v284: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:19:46.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:46 smithi016 ceph-mon[32374]: pgmap v284: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:19:46.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:46 smithi053 ceph-mon[35979]: pgmap v284: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:19:48.430 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:48 smithi063 ceph-mon[36249]: pgmap v285: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:19:48.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:48 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:19:48.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:48 smithi016 ceph-mon[32374]: pgmap v285: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:19:48.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:48 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:19:48.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:48 smithi053 ceph-mon[35979]: pgmap v285: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:19:48.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:48 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:19:49.430 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:49 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:19:49.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:49 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:19:49.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:49 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:19:49.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:49 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:19:49.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:49 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:19:49.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:49 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:19:49.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:49 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:19:49.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:49 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:19:49.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:49 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:19:50.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:50 smithi063 ceph-mon[36249]: pgmap v286: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:19:50.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:50 smithi063 ceph-mon[36249]: scrub summary: idle 2024-03-28T22:19:50.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:50 smithi016 ceph-mon[32374]: pgmap v286: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:19:50.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:50 smithi016 ceph-mon[32374]: scrub summary: idle 2024-03-28T22:19:50.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:50 smithi053 ceph-mon[35979]: pgmap v286: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-28T22:19:50.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:50 smithi053 ceph-mon[35979]: scrub summary: idle 2024-03-28T22:19:51.430 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:51 smithi063 ceph-mon[36249]: scrub summary: idle 2024-03-28T22:19:51.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:51 smithi016 ceph-mon[32374]: scrub summary: idle 2024-03-28T22:19:51.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:51 smithi053 ceph-mon[35979]: scrub summary: idle 2024-03-28T22:19:52.430 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:52 smithi063 ceph-mon[36249]: pgmap v287: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:19:52.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:52 smithi016 ceph-mon[32374]: pgmap v287: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:19:52.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:52 smithi053 ceph-mon[35979]: pgmap v287: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-28T22:19:54.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:54 smithi016 ceph-mon[32374]: pgmap v288: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:19:54.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:54 smithi053 ceph-mon[35979]: pgmap v288: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:19:54.680 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:54 smithi063 ceph-mon[36249]: pgmap v288: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:19:56.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:56 smithi016 ceph-mon[32374]: pgmap v289: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:19:56.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:56 smithi053 ceph-mon[35979]: pgmap v289: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:19:56.680 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:56 smithi063 ceph-mon[36249]: pgmap v289: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:19:58.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:19:58 smithi053 ceph-mon[35979]: pgmap v290: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:19:58.680 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:19:58 smithi063 ceph-mon[36249]: pgmap v290: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:19:58.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:19:58 smithi016 ceph-mon[32374]: pgmap v290: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:20:00.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:00 smithi053 ceph-mon[35979]: pgmap v291: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:20:00.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:00 smithi053 ceph-mon[35979]: overall HEALTH_OK 2024-03-28T22:20:00.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:00 smithi063 ceph-mon[36249]: pgmap v291: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:20:00.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:00 smithi063 ceph-mon[36249]: overall HEALTH_OK 2024-03-28T22:20:00.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:00 smithi016 ceph-mon[32374]: pgmap v291: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:20:00.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:00 smithi016 ceph-mon[32374]: overall HEALTH_OK 2024-03-28T22:20:02.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:02 smithi053 ceph-mon[35979]: pgmap v292: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:20:02.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:02 smithi063 ceph-mon[36249]: pgmap v292: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:20:02.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:02 smithi016 ceph-mon[32374]: pgmap v292: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:20:04.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:04 smithi053 ceph-mon[35979]: pgmap v293: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:20:04.680 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:04 smithi063 ceph-mon[36249]: pgmap v293: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:20:04.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:04 smithi016 ceph-mon[32374]: pgmap v293: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:20:06.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:06 smithi053 ceph-mon[35979]: pgmap v294: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:20:06.680 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:06 smithi063 ceph-mon[36249]: pgmap v294: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:20:06.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:06 smithi016 ceph-mon[32374]: pgmap v294: 97 pgs: 97 active+clean; 91 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:20:07.095 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:20:07.096 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:20:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.213+0000 7fed51180640 1 -- 172.21.15.16:0/564086945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed4c1076f0 msgr2=0x7fed4c107ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.213+0000 7fed51180640 1 --2- 172.21.15.16:0/564086945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed4c1076f0 0x7fed4c107ad0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fed3c009cd0 tx=0x7fed3c02c0f0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.213+0000 7fed51180640 1 -- 172.21.15.16:0/564086945 shutdown_connections 2024-03-28T22:20:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.213+0000 7fed51180640 1 --2- 172.21.15.16:0/564086945 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed4c1020a0 0x7fed4c10e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.213+0000 7fed51180640 1 --2- 172.21.15.16:0/564086945 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed4c1016e0 0x7fed4c101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.213+0000 7fed51180640 1 --2- 172.21.15.16:0/564086945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed4c1076f0 0x7fed4c107ad0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.213+0000 7fed51180640 1 -- 172.21.15.16:0/564086945 >> 172.21.15.16:0/564086945 conn(0x7fed4c0fd1f0 msgr2=0x7fed4c0ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:07.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.213+0000 7fed51180640 1 -- 172.21.15.16:0/564086945 shutdown_connections 2024-03-28T22:20:07.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.213+0000 7fed51180640 1 -- 172.21.15.16:0/564086945 wait complete. 2024-03-28T22:20:07.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.214+0000 7fed51180640 1 Processor -- start 2024-03-28T22:20:07.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.214+0000 7fed51180640 1 -- start start 2024-03-28T22:20:07.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.214+0000 7fed51180640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed4c1016e0 0x7fed4c198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.214+0000 7fed51180640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed4c1020a0 0x7fed4c198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.214+0000 7fed51180640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed4c1076f0 0x7fed4c19c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.214+0000 7fed51180640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fed4c111ac0 con 0x7fed4c1076f0 2024-03-28T22:20:07.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.214+0000 7fed51180640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fed4c111940 con 0x7fed4c1016e0 2024-03-28T22:20:07.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.214+0000 7fed51180640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fed4c111c40 con 0x7fed4c1020a0 2024-03-28T22:20:07.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.214+0000 7fed4b59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed4c1076f0 0x7fed4c19c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.214+0000 7fed4ad9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed4c1016e0 0x7fed4c198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.214+0000 7fed4ad9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed4c1016e0 0x7fed4c198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:37756/0 (socket says 172.21.15.16:37756) 2024-03-28T22:20:07.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.214+0000 7fed4b59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed4c1076f0 0x7fed4c19c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50022/0 (socket says 172.21.15.16:50022) 2024-03-28T22:20:07.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.214+0000 7fed4ad9d640 1 -- 172.21.15.16:0/572699030 learned_addr learned my addr 172.21.15.16:0/572699030 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:07.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.214+0000 7fed4a59c640 1 --2- 172.21.15.16:0/572699030 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed4c1020a0 0x7fed4c198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.215+0000 7fed4b59e640 1 -- 172.21.15.16:0/572699030 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed4c1020a0 msgr2=0x7fed4c198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:07.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.215+0000 7fed4b59e640 1 --2- 172.21.15.16:0/572699030 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed4c1020a0 0x7fed4c198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.215+0000 7fed4b59e640 1 -- 172.21.15.16:0/572699030 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed4c1016e0 msgr2=0x7fed4c198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:07.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.215+0000 7fed4b59e640 1 --2- 172.21.15.16:0/572699030 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed4c1016e0 0x7fed4c198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.215+0000 7fed4b59e640 1 -- 172.21.15.16:0/572699030 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed4c19d010 con 0x7fed4c1076f0 2024-03-28T22:20:07.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.215+0000 7fed4b59e640 1 --2- 172.21.15.16:0/572699030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed4c1076f0 0x7fed4c19c8e0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fed4400e970 tx=0x7fed4400ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:07.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.215+0000 7fed33fff640 1 -- 172.21.15.16:0/572699030 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed4400ccc0 con 0x7fed4c1076f0 2024-03-28T22:20:07.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.215+0000 7fed51180640 1 -- 172.21.15.16:0/572699030 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed4c19d300 con 0x7fed4c1076f0 2024-03-28T22:20:07.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.215+0000 7fed33fff640 1 -- 172.21.15.16:0/572699030 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed440035f0 con 0x7fed4c1076f0 2024-03-28T22:20:07.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.215+0000 7fed33fff640 1 -- 172.21.15.16:0/572699030 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed4400ce60 con 0x7fed4c1076f0 2024-03-28T22:20:07.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.215+0000 7fed51180640 1 -- 172.21.15.16:0/572699030 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed4c1a4b90 con 0x7fed4c1076f0 2024-03-28T22:20:07.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.217+0000 7fed33fff640 1 -- 172.21.15.16:0/572699030 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7fed440037a0 con 0x7fed4c1076f0 2024-03-28T22:20:07.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.218+0000 7fed53e34740 1 Processor -- start 2024-03-28T22:20:07.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.218+0000 7fed53e34740 1 -- start start 2024-03-28T22:20:07.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.218+0000 7fed53e34740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5567de707290 0x5567de707670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.218+0000 7fed33fff640 1 --2- 172.21.15.16:0/572699030 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fed340a1ef0 0x7fed340a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.218+0000 7fed53e34740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567de707bb0 0x5567de710250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.218+0000 7fed53e34740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5567de710790 0x5567de712b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.218+0000 7fed53e34740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5567de66b180 con 0x5567de707bb0 2024-03-28T22:20:07.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.218+0000 7fed53e34740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5567de613b70 con 0x5567de707290 2024-03-28T22:20:07.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.218+0000 7fed53e34740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5567de667220 con 0x5567de710790 2024-03-28T22:20:07.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.218+0000 7fed4a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567de707bb0 0x5567de710250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.218+0000 7fed4ad9d640 1 --2- 172.21.15.16:0/572699030 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fed340a1ef0 0x7fed340a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.218+0000 7fed4a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567de707bb0 0x5567de710250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50036/0 (socket says 172.21.15.16:50036) 2024-03-28T22:20:07.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.219+0000 7fed4a59c640 1 -- 172.21.15.16:0/1730090126 learned_addr learned my addr 172.21.15.16:0/1730090126 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:07.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.219+0000 7fed33fff640 1 -- 172.21.15.16:0/572699030 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7fed440ce360 con 0x7fed4c1076f0 2024-03-28T22:20:07.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.219+0000 7fed4a59c640 1 --2- 172.21.15.16:0/1730090126 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5567de707290 0x5567de707670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.219+0000 7fed4b59e640 1 --2- 172.21.15.16:0/1730090126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5567de710790 0x5567de712b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.219+0000 7fed4a59c640 1 -- 172.21.15.16:0/1730090126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5567de710790 msgr2=0x5567de712b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:07.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.219+0000 7fed4a59c640 1 --2- 172.21.15.16:0/1730090126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5567de710790 0x5567de712b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.219+0000 7fed4a59c640 1 -- 172.21.15.16:0/1730090126 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5567de707290 msgr2=0x5567de707670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:07.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.219+0000 7fed4a59c640 1 --2- 172.21.15.16:0/1730090126 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5567de707290 0x5567de707670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.219+0000 7fed4a59c640 1 -- 172.21.15.16:0/1730090126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567de6277c0 con 0x5567de707bb0 2024-03-28T22:20:07.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.219+0000 7fed4ad9d640 1 --2- 172.21.15.16:0/572699030 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fed340a1ef0 0x7fed340a43b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fed3c000f80 tx=0x7fed3c0098e0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:07.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.219+0000 7fed4a59c640 1 --2- 172.21.15.16:0/1730090126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567de707bb0 0x5567de710250 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fed40011730 tx=0x7fed4000a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c58ff749c74d0ae server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:07.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.219+0000 7fed317fa640 1 -- 172.21.15.16:0/1730090126 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed40004020 con 0x5567de707bb0 2024-03-28T22:20:07.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.219+0000 7fed317fa640 1 -- 172.21.15.16:0/1730090126 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed40005500 con 0x5567de707bb0 2024-03-28T22:20:07.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.219+0000 7fed317fa640 1 -- 172.21.15.16:0/1730090126 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed40004ba0 con 0x5567de707bb0 2024-03-28T22:20:07.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.220+0000 7fed53e34740 1 -- 172.21.15.16:0/1730090126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567de707bb0 msgr2=0x5567de710250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:07.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.220+0000 7fed53e34740 1 --2- 172.21.15.16:0/1730090126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567de707bb0 0x5567de710250 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fed40011730 tx=0x7fed4000a6a0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.220+0000 7fed53e34740 1 -- 172.21.15.16:0/1730090126 shutdown_connections 2024-03-28T22:20:07.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.220+0000 7fed53e34740 1 --2- 172.21.15.16:0/1730090126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5567de710790 0x5567de712b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.220+0000 7fed53e34740 1 --2- 172.21.15.16:0/1730090126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567de707bb0 0x5567de710250 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.220+0000 7fed53e34740 1 --2- 172.21.15.16:0/1730090126 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5567de707290 0x5567de707670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.220+0000 7fed53e34740 1 -- 172.21.15.16:0/1730090126 >> 172.21.15.16:0/1730090126 conn(0x5567de7019c0 msgr2=0x5567de701da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:07.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.220+0000 7fed53e34740 1 -- 172.21.15.16:0/1730090126 shutdown_connections 2024-03-28T22:20:07.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.220+0000 7fed53e34740 1 -- 172.21.15.16:0/1730090126 wait complete. 2024-03-28T22:20:07.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.220+0000 7fed53e34740 1 -- start start 2024-03-28T22:20:07.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.220+0000 7fed53e34740 1 Processor -- start 2024-03-28T22:20:07.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.221+0000 7fed53e34740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567de707290 0x5567de70d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.221+0000 7fed53e34740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5567de707bb0 0x5567de70dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.221+0000 7fed53e34740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5567de710790 0x5567de70e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.221+0000 7fed53e34740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5567de713580 con 0x5567de707290 2024-03-28T22:20:07.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.221+0000 7fed53e34740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5567de6094a0 con 0x5567de707bb0 2024-03-28T22:20:07.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.221+0000 7fed53e34740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5567de713700 con 0x5567de710790 2024-03-28T22:20:07.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.221+0000 7fed53e34740 10 client.0 _ll_register_callbacks cb 0x5567de60cd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:20:07.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.221+0000 7fed4ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567de707290 0x5567de70d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.221+0000 7fed4ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567de707290 0x5567de70d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50050/0 (socket says 172.21.15.16:50050) 2024-03-28T22:20:07.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.221+0000 7fed4ad9d640 1 -- 172.21.15.16:0/1752045305 learned_addr learned my addr 172.21.15.16:0/1752045305 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:07.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.221+0000 7fed4a59c640 1 --2- 172.21.15.16:0/1752045305 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5567de707bb0 0x5567de70dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.221+0000 7fed4b59e640 1 --2- 172.21.15.16:0/1752045305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5567de710790 0x5567de70e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.221+0000 7fed4ad9d640 1 -- 172.21.15.16:0/1752045305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5567de710790 msgr2=0x5567de70e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:07.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.221+0000 7fed4ad9d640 1 --2- 172.21.15.16:0/1752045305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5567de710790 0x5567de70e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.221+0000 7fed4ad9d640 1 -- 172.21.15.16:0/1752045305 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5567de707bb0 msgr2=0x5567de70dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:07.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.221+0000 7fed4ad9d640 1 --2- 172.21.15.16:0/1752045305 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5567de707bb0 0x5567de70dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.221+0000 7fed4ad9d640 1 -- 172.21.15.16:0/1752045305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567de4497d0 con 0x5567de707290 2024-03-28T22:20:07.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.221+0000 7fed4b59e640 1 --2- 172.21.15.16:0/1752045305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5567de710790 0x5567de70e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:20:07.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.221+0000 7fed4ad9d640 1 --2- 172.21.15.16:0/1752045305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567de707290 0x5567de70d800 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fed3c006010 tx=0x7fed3c02c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cae021e75626a402 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:07.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.221+0000 7fed2eff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:20:07.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.221+0000 7fed2eff5640 1 -- 172.21.15.16:0/1752045305 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3c044070 con 0x5567de707290 2024-03-28T22:20:07.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.221+0000 7fed2eff5640 1 -- 172.21.15.16:0/1752045305 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7fed3c04b670 con 0x5567de707290 2024-03-28T22:20:07.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.222+0000 7fed53e34740 1 -- 172.21.15.16:0/1752045305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed40005500 con 0x5567de707290 2024-03-28T22:20:07.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.222+0000 7fed2eff5640 1 -- 172.21.15.16:0/1752045305 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fed3c059050 con 0x5567de707290 2024-03-28T22:20:07.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.222+0000 7fed53e34740 10 client.17472 fetch_fsmap learned FSMap version 27 2024-03-28T22:20:07.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.222+0000 7fed53e34740 1 -- 172.21.15.16:0/1752045305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5567de37f3f0 con 0x5567de707290 2024-03-28T22:20:07.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.222+0000 7fed2eff5640 1 -- 172.21.15.16:0/1752045305 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed40005500 con 0x5567de707290 2024-03-28T22:20:07.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.222+0000 7fed2eff5640 1 -- 172.21.15.16:0/1752045305 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7fed3c034a40 con 0x5567de707290 2024-03-28T22:20:07.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.222+0000 7fed53e34740 10 client.17472 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:20:07.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.222+0000 7fed53e34740 10 client.17472 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:20:07.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.222+0000 7fed53e34740 20 client.17472 populate_metadata read hostname 'smithi016' 2024-03-28T22:20:07.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.222+0000 7fed53e34740 1 --2- 172.21.15.16:0/1752045305 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5567de720bb0 0x5567de723020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.222+0000 7fed53e34740 4 client.17472 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:20:07.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.222+0000 7fed53e34740 1 -- 172.21.15.16:0/1752045305 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5567de6786c0 con 0x5567de720bb0 2024-03-28T22:20:07.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.222+0000 7fed4a59c640 1 --2- 172.21.15.16:0/1752045305 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5567de720bb0 0x5567de723020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.223+0000 7fed2eff5640 10 client.17472 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:20:07.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.223+0000 7fed4a59c640 1 --2- 172.21.15.16:0/1752045305 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5567de720bb0 0x5567de723020 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96e79ede25422e27 server_cookie=3bea609cc1310524 in_seq=0 out_seq=0 2024-03-28T22:20:07.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.223+0000 7fed2eff5640 1 -- 172.21.15.16:0/1752045305 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5567de6786c0 con 0x5567de720bb0 2024-03-28T22:20:07.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.223+0000 7fed2eff5640 10 client.17472 handle_command_reply: tid=0 2024-03-28T22:20:07.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.223+0000 7fed53e34740 1 client.17472 shutdown 2024-03-28T22:20:07.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.224+0000 7fed53e34740 1 -- 172.21.15.16:0/1752045305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567de707290 msgr2=0x5567de70d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:07.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.224+0000 7fed53e34740 1 --2- 172.21.15.16:0/1752045305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567de707290 0x5567de70d800 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fed3c006010 tx=0x7fed3c02c0f0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.224+0000 7fed53e34740 1 -- 172.21.15.16:0/1752045305 shutdown_connections 2024-03-28T22:20:07.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.224+0000 7fed53e34740 1 --2- 172.21.15.16:0/1752045305 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5567de720bb0 0x5567de723020 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.224+0000 7fed53e34740 1 --2- 172.21.15.16:0/1752045305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5567de710790 0x5567de70e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.224+0000 7fed53e34740 1 --2- 172.21.15.16:0/1752045305 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5567de707bb0 0x5567de70dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.224+0000 7fed2eff5640 0 client.17472 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:20:07.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.224+0000 7fed53e34740 1 --2- 172.21.15.16:0/1752045305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567de707290 0x5567de70d800 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.224+0000 7fed53e34740 1 -- 172.21.15.16:0/1752045305 >> 172.21.15.16:0/1752045305 conn(0x5567de7019c0 msgr2=0x5567de703710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:07.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.224+0000 7fed53e34740 1 -- 172.21.15.16:0/1752045305 shutdown_connections 2024-03-28T22:20:07.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.224+0000 7fed53e34740 1 -- 172.21.15.16:0/1752045305 wait complete. 2024-03-28T22:20:07.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.224+0000 7fed53e34740 20 client.17472 trim_cache size 0 max 16384 2024-03-28T22:20:07.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.241+0000 7fed53e34740 1 Processor -- start 2024-03-28T22:20:07.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.241+0000 7fed53e34740 1 -- start start 2024-03-28T22:20:07.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.241+0000 7fed53e34740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5567de707290 0x5567de70ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.241+0000 7fed53e34740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5567de707bb0 0x5567de71c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.242+0000 7fed53e34740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567de710790 0x5567de703500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.242+0000 7fed53e34740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fed4c1afd10 con 0x5567de710790 2024-03-28T22:20:07.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.242+0000 7fed53e34740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fed4c1ad5e0 con 0x5567de707290 2024-03-28T22:20:07.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.242+0000 7fed53e34740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fed4c1ab450 con 0x5567de707bb0 2024-03-28T22:20:07.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.242+0000 7fed4b59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567de710790 0x5567de703500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.242+0000 7fed4ad9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5567de707290 0x5567de70ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.242+0000 7fed4b59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567de710790 0x5567de703500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50060/0 (socket says 172.21.15.16:50060) 2024-03-28T22:20:07.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.242+0000 7fed4b59e640 1 -- 172.21.15.16:0/2231796608 learned_addr learned my addr 172.21.15.16:0/2231796608 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:07.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.242+0000 7fed4a59c640 1 --2- 172.21.15.16:0/2231796608 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5567de707bb0 0x5567de71c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.242+0000 7fed4b59e640 1 -- 172.21.15.16:0/2231796608 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5567de707bb0 msgr2=0x5567de71c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:07.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.242+0000 7fed4b59e640 1 --2- 172.21.15.16:0/2231796608 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5567de707bb0 0x5567de71c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.242+0000 7fed4b59e640 1 -- 172.21.15.16:0/2231796608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5567de707290 msgr2=0x5567de70ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:07.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.242+0000 7fed4b59e640 1 --2- 172.21.15.16:0/2231796608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5567de707290 0x5567de70ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.242+0000 7fed4b59e640 1 -- 172.21.15.16:0/2231796608 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567de33f0f0 con 0x5567de710790 2024-03-28T22:20:07.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.242+0000 7fed4b59e640 1 --2- 172.21.15.16:0/2231796608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567de710790 0x5567de703500 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fed440cbb10 tx=0x7fed440b3d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2254f66c66a925bb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:07.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.242+0000 7fed30ff9640 1 -- 172.21.15.16:0/2231796608 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed4409ad50 con 0x5567de710790 2024-03-28T22:20:07.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.242+0000 7fed30ff9640 1 -- 172.21.15.16:0/2231796608 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed4409eea0 con 0x5567de710790 2024-03-28T22:20:07.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.242+0000 7fed30ff9640 1 -- 172.21.15.16:0/2231796608 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed440cbd30 con 0x5567de710790 2024-03-28T22:20:07.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.243+0000 7fed53e34740 1 -- 172.21.15.16:0/2231796608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567de710790 msgr2=0x5567de703500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:07.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.243+0000 7fed53e34740 1 --2- 172.21.15.16:0/2231796608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567de710790 0x5567de703500 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fed440cbb10 tx=0x7fed440b3d90 comp rx=0 tx=0).stop 2024-03-28T22:20:07.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.243+0000 7fed53e34740 1 -- 172.21.15.16:0/2231796608 shutdown_connections 2024-03-28T22:20:07.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.243+0000 7fed53e34740 1 --2- 172.21.15.16:0/2231796608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567de710790 0x5567de703500 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.243+0000 7fed53e34740 1 --2- 172.21.15.16:0/2231796608 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5567de707bb0 0x5567de71c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.243+0000 7fed53e34740 1 --2- 172.21.15.16:0/2231796608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5567de707290 0x5567de70ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.243+0000 7fed53e34740 1 -- 172.21.15.16:0/2231796608 >> 172.21.15.16:0/2231796608 conn(0x5567de7019c0 msgr2=0x5567de702df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:07.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.243+0000 7fed53e34740 1 -- 172.21.15.16:0/2231796608 shutdown_connections 2024-03-28T22:20:07.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.243+0000 7fed53e34740 1 -- 172.21.15.16:0/2231796608 wait complete. 2024-03-28T22:20:07.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.243+0000 7fed53e34740 1 -- start start 2024-03-28T22:20:07.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.243+0000 7fed53e34740 1 Processor -- start 2024-03-28T22:20:07.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.243+0000 7fed53e34740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5567de707290 0x5567de714260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.243+0000 7fed53e34740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567de707bb0 0x5567de7147a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.243+0000 7fed53e34740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5567de710790 0x5567de714ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.243+0000 7fed53e34740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fed4c1abaa0 con 0x5567de707bb0 2024-03-28T22:20:07.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.243+0000 7fed53e34740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fed4c1acf10 con 0x5567de710790 2024-03-28T22:20:07.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.243+0000 7fed53e34740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fed4c1ac170 con 0x5567de707290 2024-03-28T22:20:07.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.244+0000 7fed53e34740 10 client.0 _ll_register_callbacks cb 0x7fed44005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:20:07.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.244+0000 7fed4a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567de707bb0 0x5567de7147a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.244+0000 7fed4a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567de707bb0 0x5567de7147a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50062/0 (socket says 172.21.15.16:50062) 2024-03-28T22:20:07.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.244+0000 7fed4a59c640 1 -- 172.21.15.16:0/3758018493 learned_addr learned my addr 172.21.15.16:0/3758018493 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:07.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.244+0000 7fed4b59e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5567de710790 0x5567de714ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.244+0000 7fed4ad9d640 1 --2- 172.21.15.16:0/3758018493 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5567de707290 0x5567de714260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.244+0000 7fed4a59c640 1 -- 172.21.15.16:0/3758018493 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5567de707290 msgr2=0x5567de714260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:07.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.244+0000 7fed4a59c640 1 --2- 172.21.15.16:0/3758018493 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5567de707290 0x5567de714260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.244+0000 7fed4a59c640 1 -- 172.21.15.16:0/3758018493 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5567de710790 msgr2=0x5567de714ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:07.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.244+0000 7fed4a59c640 1 --2- 172.21.15.16:0/3758018493 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5567de710790 0x5567de714ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.244+0000 7fed4a59c640 1 -- 172.21.15.16:0/3758018493 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567de785f50 con 0x5567de707bb0 2024-03-28T22:20:07.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.244+0000 7fed4a59c640 1 --2- 172.21.15.16:0/3758018493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567de707bb0 0x5567de7147a0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fed40011730 tx=0x7fed4000a4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5fb057f6e639365 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:07.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.244+0000 7fed2eff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:20:07.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.244+0000 7fed2eff5640 1 -- 172.21.15.16:0/3758018493 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed400044a0 con 0x5567de707bb0 2024-03-28T22:20:07.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.244+0000 7fed2eff5640 1 -- 172.21.15.16:0/3758018493 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7fed4002d580 con 0x5567de707bb0 2024-03-28T22:20:07.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.244+0000 7fed53e34740 1 -- 172.21.15.16:0/3758018493 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed4409eea0 con 0x5567de707bb0 2024-03-28T22:20:07.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.244+0000 7fed2eff5640 1 -- 172.21.15.16:0/3758018493 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fed4003d050 con 0x5567de707bb0 2024-03-28T22:20:07.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.244+0000 7fed53e34740 10 client.17478 fetch_fsmap learned FSMap version 27 2024-03-28T22:20:07.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.244+0000 7fed53e34740 1 -- 172.21.15.16:0/3758018493 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5567de787220 con 0x5567de707bb0 2024-03-28T22:20:07.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.245+0000 7fed2eff5640 1 -- 172.21.15.16:0/3758018493 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed4409eea0 con 0x5567de707bb0 2024-03-28T22:20:07.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.245+0000 7fed2eff5640 1 -- 172.21.15.16:0/3758018493 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7fed4001e600 con 0x5567de707bb0 2024-03-28T22:20:07.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.245+0000 7fed53e34740 10 client.17478 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:20:07.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.245+0000 7fed53e34740 10 client.17478 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:20:07.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.245+0000 7fed53e34740 20 client.17478 populate_metadata read hostname 'smithi016' 2024-03-28T22:20:07.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.245+0000 7fed53e34740 1 --2- 172.21.15.16:0/3758018493 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5567de787840 0x5567de789c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.245+0000 7fed53e34740 4 client.17478 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:20:07.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.245+0000 7fed53e34740 1 -- 172.21.15.16:0/3758018493 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5567de672a10 con 0x5567de787840 2024-03-28T22:20:07.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.245+0000 7fed4ad9d640 1 --2- 172.21.15.16:0/3758018493 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5567de787840 0x5567de789c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.245+0000 7fed2eff5640 10 client.17478 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:20:07.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.245+0000 7fed4ad9d640 1 --2- 172.21.15.16:0/3758018493 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5567de787840 0x5567de789c60 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a47a9882479f2b2 server_cookie=c0cdd05d0d212824 in_seq=0 out_seq=0 2024-03-28T22:20:07.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.245+0000 7fed2eff5640 1 -- 172.21.15.16:0/3758018493 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5567de672a10 con 0x5567de787840 2024-03-28T22:20:07.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.245+0000 7fed2eff5640 10 client.17478 handle_command_reply: tid=0 2024-03-28T22:20:07.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.245+0000 7fed53e34740 1 client.17478 shutdown 2024-03-28T22:20:07.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.246+0000 7fed53e34740 1 -- 172.21.15.16:0/3758018493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567de707bb0 msgr2=0x5567de7147a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:07.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.246+0000 7fed53e34740 1 --2- 172.21.15.16:0/3758018493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567de707bb0 0x5567de7147a0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fed40011730 tx=0x7fed4000a4e0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.246+0000 7fed53e34740 1 -- 172.21.15.16:0/3758018493 shutdown_connections 2024-03-28T22:20:07.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.246+0000 7fed53e34740 1 --2- 172.21.15.16:0/3758018493 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5567de787840 0x5567de789c60 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.246+0000 7fed53e34740 1 --2- 172.21.15.16:0/3758018493 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5567de710790 0x5567de714ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.246+0000 7fed53e34740 1 --2- 172.21.15.16:0/3758018493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5567de707bb0 0x5567de7147a0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.246+0000 7fed2eff5640 0 client.17478 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:20:07.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.246+0000 7fed53e34740 1 --2- 172.21.15.16:0/3758018493 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5567de707290 0x5567de714260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.246+0000 7fed53e34740 1 -- 172.21.15.16:0/3758018493 >> 172.21.15.16:0/3758018493 conn(0x5567de7019c0 msgr2=0x5567de700c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:07.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.246+0000 7fed53e34740 1 -- 172.21.15.16:0/3758018493 shutdown_connections 2024-03-28T22:20:07.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.246+0000 7fed53e34740 1 -- 172.21.15.16:0/3758018493 wait complete. 2024-03-28T22:20:07.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.246+0000 7fed53e34740 20 client.17478 trim_cache size 0 max 16384 2024-03-28T22:20:07.259 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:20:07.259 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-03-28T22:20:07.259 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-03-28T22:20:07.259 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:20:07.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.246+0000 7fed51180640 1 -- 172.21.15.16:0/572699030 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fed340a1ef0 msgr2=0x7fed340a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:07.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.246+0000 7fed51180640 1 --2- 172.21.15.16:0/572699030 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fed340a1ef0 0x7fed340a43b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fed3c000f80 tx=0x7fed3c0098e0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.246+0000 7fed51180640 1 -- 172.21.15.16:0/572699030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed4c1076f0 msgr2=0x7fed4c19c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:07.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.246+0000 7fed51180640 1 --2- 172.21.15.16:0/572699030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed4c1076f0 0x7fed4c19c8e0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fed4400e970 tx=0x7fed4400ee30 comp rx=0 tx=0).stop 2024-03-28T22:20:07.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.246+0000 7fed51180640 1 -- 172.21.15.16:0/572699030 shutdown_connections 2024-03-28T22:20:07.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.246+0000 7fed51180640 1 --2- 172.21.15.16:0/572699030 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fed340a1ef0 0x7fed340a43b0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.246+0000 7fed51180640 1 --2- 172.21.15.16:0/572699030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed4c1076f0 0x7fed4c19c8e0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.246+0000 7fed51180640 1 --2- 172.21.15.16:0/572699030 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed4c1020a0 0x7fed4c198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.246+0000 7fed51180640 1 --2- 172.21.15.16:0/572699030 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed4c1016e0 0x7fed4c198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.246+0000 7fed51180640 1 -- 172.21.15.16:0/572699030 >> 172.21.15.16:0/572699030 conn(0x7fed4c0fd1f0 msgr2=0x7fed4c0fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:07.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.247+0000 7fed51180640 1 -- 172.21.15.16:0/572699030 shutdown_connections 2024-03-28T22:20:07.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.247+0000 7fed51180640 1 -- 172.21.15.16:0/572699030 wait complete. 2024-03-28T22:20:07.262 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:20:07.263 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:20:07.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.364+0000 7f4b6399a640 1 -- 172.21.15.16:0/4260444910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b5c106ad0 msgr2=0x7f4b5c10e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:07.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.364+0000 7f4b6399a640 1 --2- 172.21.15.16:0/4260444910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b5c106ad0 0x7f4b5c10e650 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f4b54009ca0 tx=0x7f4b5402c0f0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.364+0000 7f4b6399a640 1 -- 172.21.15.16:0/4260444910 shutdown_connections 2024-03-28T22:20:07.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.364+0000 7f4b6399a640 1 --2- 172.21.15.16:0/4260444910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b5c106ad0 0x7f4b5c10e650 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.364+0000 7f4b6399a640 1 --2- 172.21.15.16:0/4260444910 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b5c100b60 0x7f4b5c1063a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.364+0000 7f4b6399a640 1 --2- 172.21.15.16:0/4260444910 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b5c1001b0 0x7f4b5c100590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.364+0000 7f4b6399a640 1 -- 172.21.15.16:0/4260444910 >> 172.21.15.16:0/4260444910 conn(0x7f4b5c076aa0 msgr2=0x7f4b5c076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:07.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.364+0000 7f4b6399a640 1 -- 172.21.15.16:0/4260444910 shutdown_connections 2024-03-28T22:20:07.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.364+0000 7f4b6399a640 1 -- 172.21.15.16:0/4260444910 wait complete. 2024-03-28T22:20:07.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.365+0000 7f4b6399a640 1 Processor -- start 2024-03-28T22:20:07.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.365+0000 7f4b6399a640 1 -- start start 2024-03-28T22:20:07.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.365+0000 7f4b6399a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b5c1001b0 0x7f4b5c197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.365+0000 7f4b6399a640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b5c106ad0 0x7f4b5c1984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.365+0000 7f4b6399a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b5c19c870 0x7f4b5c19cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.365+0000 7f4b6399a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4b5c111b30 con 0x7f4b5c1001b0 2024-03-28T22:20:07.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.365+0000 7f4b6399a640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4b5c1119b0 con 0x7f4b5c106ad0 2024-03-28T22:20:07.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.365+0000 7f4b6399a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4b5c111cb0 con 0x7f4b5c19c870 2024-03-28T22:20:07.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.365+0000 7f4b61736640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b5c1001b0 0x7f4b5c197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.365+0000 7f4b61736640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b5c1001b0 0x7f4b5c197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50088/0 (socket says 172.21.15.16:50088) 2024-03-28T22:20:07.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.365+0000 7f4b61736640 1 -- 172.21.15.16:0/3490739160 learned_addr learned my addr 172.21.15.16:0/3490739160 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:07.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.365+0000 7f4b60f35640 1 --2- 172.21.15.16:0/3490739160 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b5c106ad0 0x7f4b5c1984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.365+0000 7f4b61f37640 1 --2- 172.21.15.16:0/3490739160 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b5c19c870 0x7f4b5c19cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.365+0000 7f4b61736640 1 -- 172.21.15.16:0/3490739160 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b5c19c870 msgr2=0x7f4b5c19cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:07.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.365+0000 7f4b61736640 1 --2- 172.21.15.16:0/3490739160 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b5c19c870 0x7f4b5c19cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.365+0000 7f4b61736640 1 -- 172.21.15.16:0/3490739160 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b5c106ad0 msgr2=0x7f4b5c1984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:07.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.365+0000 7f4b61736640 1 --2- 172.21.15.16:0/3490739160 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b5c106ad0 0x7f4b5c1984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.365+0000 7f4b61736640 1 -- 172.21.15.16:0/3490739160 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b5c19d360 con 0x7f4b5c1001b0 2024-03-28T22:20:07.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.365+0000 7f4b61736640 1 --2- 172.21.15.16:0/3490739160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b5c1001b0 0x7f4b5c197fb0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f4b5800e9f0 tx=0x7f4b5800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:07.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.366+0000 7f4b527fc640 1 -- 172.21.15.16:0/3490739160 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b5800cd80 con 0x7f4b5c1001b0 2024-03-28T22:20:07.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.366+0000 7f4b527fc640 1 -- 172.21.15.16:0/3490739160 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b58005500 con 0x7f4b5c1001b0 2024-03-28T22:20:07.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.366+0000 7f4b6399a640 1 -- 172.21.15.16:0/3490739160 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b5c19d650 con 0x7f4b5c1001b0 2024-03-28T22:20:07.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.366+0000 7f4b527fc640 1 -- 172.21.15.16:0/3490739160 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b58010690 con 0x7f4b5c1001b0 2024-03-28T22:20:07.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.366+0000 7f4b6399a640 1 -- 172.21.15.16:0/3490739160 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b5c1a4e40 con 0x7f4b5c1001b0 2024-03-28T22:20:07.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.367+0000 7f4b6399a640 1 -- 172.21.15.16:0/3490739160 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b5c068a20 con 0x7f4b5c1001b0 2024-03-28T22:20:07.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.368+0000 7f4b527fc640 1 -- 172.21.15.16:0/3490739160 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f4b580046c0 con 0x7f4b5c1001b0 2024-03-28T22:20:07.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.368+0000 7f4b527fc640 1 --2- 172.21.15.16:0/3490739160 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4b480a1f70 0x7f4b480a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.368+0000 7f4b527fc640 1 -- 172.21.15.16:0/3490739160 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7f4b580ce550 con 0x7f4b5c1001b0 2024-03-28T22:20:07.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.368+0000 7f4b60f35640 1 --2- 172.21.15.16:0/3490739160 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4b480a1f70 0x7f4b480a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.369+0000 7f4b60f35640 1 --2- 172.21.15.16:0/3490739160 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4b480a1f70 0x7f4b480a4430 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c1063e0 tx=0x7f4b4c005f90 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:07.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.373+0000 7f4b527fc640 1 -- 172.21.15.16:0/3490739160 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4b5809b640 con 0x7f4b5c1001b0 2024-03-28T22:20:07.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.594+0000 7f4b6399a640 1 -- 172.21.15.16:0/3490739160 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b5c100b60 con 0x7f4b5c1001b0 2024-03-28T22:20:07.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.595+0000 7f4b527fc640 1 -- 172.21.15.16:0/3490739160 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11851 (secure 0 0 0) 0x7f4b5809af40 con 0x7f4b5c1001b0 2024-03-28T22:20:07.597 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:20:07.597 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/1644581227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1644581227},{"type":"v1","addr":"172.21.15.16:6839","nonce":1644581227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/2593062669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2593062669},{"type":"v1","addr":"172.21.15.16:6841","nonce":2593062669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:08:29.672976+0000","modified":"2024-03-28T22:14:44.922044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24566,"mds_1":24524,"mds_2":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24466":{"gid":24466,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/247249857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":247249857},{"type":"v1","addr":"172.21.15.63:6835","nonce":247249857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6837/2324231032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":2324231032},{"type":"v1","addr":"172.21.15.63:6837","nonce":2324231032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6839/1177197077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":1177197077},{"type":"v1","addr":"172.21.15.63:6839","nonce":1177197077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"e","rank":2,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.53:6835/350727634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":350727634},{"type":"v1","addr":"172.21.15.53:6835","nonce":350727634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/349877573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":349877573},{"type":"v1","addr":"172.21.15.53:6837","nonce":349877573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6839/3714153956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":3714153956},{"type":"v1","addr":"172.21.15.53:6839","nonce":3714153956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24566,"qdb_cluster":[24500,24524,24566]},"id":1}]} 2024-03-28T22:20:07.597 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2024-03-28T22:20:07.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.597+0000 7f4b6399a640 1 -- 172.21.15.16:0/3490739160 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4b480a1f70 msgr2=0x7f4b480a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:07.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.597+0000 7f4b6399a640 1 --2- 172.21.15.16:0/3490739160 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4b480a1f70 0x7f4b480a4430 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c1063e0 tx=0x7f4b4c005f90 comp rx=0 tx=0).stop 2024-03-28T22:20:07.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.597+0000 7f4b6399a640 1 -- 172.21.15.16:0/3490739160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b5c1001b0 msgr2=0x7f4b5c197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:07.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.597+0000 7f4b6399a640 1 --2- 172.21.15.16:0/3490739160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b5c1001b0 0x7f4b5c197fb0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f4b5800e9f0 tx=0x7f4b5800edf0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.597+0000 7f4b6399a640 1 -- 172.21.15.16:0/3490739160 shutdown_connections 2024-03-28T22:20:07.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.597+0000 7f4b6399a640 1 --2- 172.21.15.16:0/3490739160 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f4b480a1f70 0x7f4b480a4430 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.597+0000 7f4b6399a640 1 --2- 172.21.15.16:0/3490739160 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b5c19c870 0x7f4b5c19cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.597+0000 7f4b6399a640 1 --2- 172.21.15.16:0/3490739160 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b5c106ad0 0x7f4b5c1984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.597+0000 7f4b6399a640 1 --2- 172.21.15.16:0/3490739160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b5c1001b0 0x7f4b5c197fb0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.597+0000 7f4b6399a640 1 -- 172.21.15.16:0/3490739160 >> 172.21.15.16:0/3490739160 conn(0x7f4b5c076aa0 msgr2=0x7f4b5c0fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:07.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.597+0000 7f4b6399a640 1 -- 172.21.15.16:0/3490739160 shutdown_connections 2024-03-28T22:20:07.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.597+0000 7f4b6399a640 1 -- 172.21.15.16:0/3490739160 wait complete. 2024-03-28T22:20:07.610 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:20:07.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.712+0000 7f0511e15640 1 -- 172.21.15.16:0/3361954424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f050c1005f0 msgr2=0x7f050c100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:07.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.712+0000 7f0511e15640 1 --2- 172.21.15.16:0/3361954424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f050c1005f0 0x7f050c100a70 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f0500009930 tx=0x7f050002bc30 comp rx=0 tx=0).stop 2024-03-28T22:20:07.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.712+0000 7f0511e15640 1 -- 172.21.15.16:0/3361954424 shutdown_connections 2024-03-28T22:20:07.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.712+0000 7f0511e15640 1 --2- 172.21.15.16:0/3361954424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f050c100fb0 0x7f050c10d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.712+0000 7f0511e15640 1 --2- 172.21.15.16:0/3361954424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f050c1005f0 0x7f050c100a70 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.712+0000 7f0511e15640 1 --2- 172.21.15.16:0/3361954424 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f050c106690 0x7f050c106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.713+0000 7f0511e15640 1 -- 172.21.15.16:0/3361954424 >> 172.21.15.16:0/3361954424 conn(0x7f050c077800 msgr2=0x7f050c077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:07.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.713+0000 7f0511e15640 1 -- 172.21.15.16:0/3361954424 shutdown_connections 2024-03-28T22:20:07.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.713+0000 7f0511e15640 1 -- 172.21.15.16:0/3361954424 wait complete. 2024-03-28T22:20:07.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.713+0000 7f0511e15640 1 Processor -- start 2024-03-28T22:20:07.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.713+0000 7f0511e15640 1 -- start start 2024-03-28T22:20:07.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.713+0000 7f0511e15640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f050c1005f0 0x7f050c198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.713+0000 7f0511e15640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f050c100fb0 0x7f050c198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.713+0000 7f0511e15640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f050c106690 0x7f050c19c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.713+0000 7f0511e15640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f050c111b10 con 0x7f050c106690 2024-03-28T22:20:07.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.713+0000 7f0511e15640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f050c111990 con 0x7f050c100fb0 2024-03-28T22:20:07.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.713+0000 7f0511e15640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f050c111c90 con 0x7f050c1005f0 2024-03-28T22:20:07.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.714+0000 7f050bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f050c106690 0x7f050c19c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.714+0000 7f050bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f050c106690 0x7f050c19c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50106/0 (socket says 172.21.15.16:50106) 2024-03-28T22:20:07.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.714+0000 7f050affd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f050c100fb0 0x7f050c198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.714+0000 7f050bfff640 1 -- 172.21.15.16:0/467948326 learned_addr learned my addr 172.21.15.16:0/467948326 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:07.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.714+0000 7f050b7fe640 1 --2- 172.21.15.16:0/467948326 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f050c1005f0 0x7f050c198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.714+0000 7f050bfff640 1 -- 172.21.15.16:0/467948326 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f050c1005f0 msgr2=0x7f050c198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:07.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.714+0000 7f050bfff640 1 --2- 172.21.15.16:0/467948326 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f050c1005f0 0x7f050c198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.714+0000 7f050bfff640 1 -- 172.21.15.16:0/467948326 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f050c100fb0 msgr2=0x7f050c198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:07.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.714+0000 7f050bfff640 1 --2- 172.21.15.16:0/467948326 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f050c100fb0 0x7f050c198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.714+0000 7f050bfff640 1 -- 172.21.15.16:0/467948326 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f050c19d040 con 0x7f050c106690 2024-03-28T22:20:07.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.714+0000 7f050bfff640 1 --2- 172.21.15.16:0/467948326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f050c106690 0x7f050c19c910 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f050400ea60 tx=0x7f050400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:07.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.714+0000 7f0508ff9640 1 -- 172.21.15.16:0/467948326 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f050400ce40 con 0x7f050c106690 2024-03-28T22:20:07.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.714+0000 7f0511e15640 1 -- 172.21.15.16:0/467948326 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f050c19d330 con 0x7f050c106690 2024-03-28T22:20:07.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.714+0000 7f0508ff9640 1 -- 172.21.15.16:0/467948326 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0504005020 con 0x7f050c106690 2024-03-28T22:20:07.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.714+0000 7f0508ff9640 1 -- 172.21.15.16:0/467948326 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0504010690 con 0x7f050c106690 2024-03-28T22:20:07.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.714+0000 7f0511e15640 1 -- 172.21.15.16:0/467948326 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f050c1a4bc0 con 0x7f050c106690 2024-03-28T22:20:07.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.716+0000 7f0508ff9640 1 -- 172.21.15.16:0/467948326 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f0504005520 con 0x7f050c106690 2024-03-28T22:20:07.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.717+0000 7f05149f6740 1 Processor -- start 2024-03-28T22:20:07.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.717+0000 7f0508ff9640 1 --2- 172.21.15.16:0/467948326 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f04f40a1ef0 0x7f04f40a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.717+0000 7f05149f6740 1 -- start start 2024-03-28T22:20:07.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.717+0000 7f050affd640 1 --2- 172.21.15.16:0/467948326 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f04f40a1ef0 0x7f04f40a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.717+0000 7f0508ff9640 1 -- 172.21.15.16:0/467948326 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7f05040ce780 con 0x7f050c106690 2024-03-28T22:20:07.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.717+0000 7f05149f6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562ed21df470 0x562ed21df850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.717+0000 7f05149f6740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ed21dfd90 0x562ed21e8430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.717+0000 7f05149f6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ed21e8970 0x562ed21ead60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.717+0000 7f05149f6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562ed21431b0 con 0x562ed21e8970 2024-03-28T22:20:07.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.717+0000 7f05149f6740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x562ed20ebc00 con 0x562ed21dfd90 2024-03-28T22:20:07.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.717+0000 7f05149f6740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x562ed213f250 con 0x562ed21df470 2024-03-28T22:20:07.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.717+0000 7f050bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ed21e8970 0x562ed21ead60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.717+0000 7f050affd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ed21dfd90 0x562ed21e8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.717+0000 7f050bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ed21e8970 0x562ed21ead60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50110/0 (socket says 172.21.15.16:50110) 2024-03-28T22:20:07.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.717+0000 7f050b7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562ed21df470 0x562ed21df850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.717+0000 7f050bfff640 1 -- 172.21.15.16:0/2308843734 learned_addr learned my addr 172.21.15.16:0/2308843734 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:07.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.717+0000 7f050bfff640 1 -- 172.21.15.16:0/2308843734 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562ed21df470 msgr2=0x562ed21df850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:07.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.717+0000 7f050bfff640 1 --2- 172.21.15.16:0/2308843734 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562ed21df470 0x562ed21df850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.717+0000 7f050bfff640 1 -- 172.21.15.16:0/2308843734 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ed21dfd90 msgr2=0x562ed21e8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:07.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.717+0000 7f050bfff640 1 --2- 172.21.15.16:0/2308843734 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ed21dfd90 0x562ed21e8430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.718+0000 7f050bfff640 1 -- 172.21.15.16:0/2308843734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ed210b3d0 con 0x562ed21e8970 2024-03-28T22:20:07.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.718+0000 7f050affd640 1 --2- 172.21.15.16:0/2308843734 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ed21dfd90 0x562ed21e8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:20:07.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.718+0000 7f050affd640 1 --2- 172.21.15.16:0/467948326 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f04f40a1ef0 0x7f04f40a43b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0500005ab0 tx=0x7f050002c7a0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:07.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.718+0000 7f050bfff640 1 --2- 172.21.15.16:0/2308843734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ed21e8970 0x562ed21ead60 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f0504097ae0 tx=0x7f050409b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b3fafc7d75d7b99 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:07.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.718+0000 7f04f1ffb640 1 -- 172.21.15.16:0/2308843734 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05040a38b0 con 0x562ed21e8970 2024-03-28T22:20:07.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.718+0000 7f04f1ffb640 1 -- 172.21.15.16:0/2308843734 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05040051d0 con 0x562ed21e8970 2024-03-28T22:20:07.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.718+0000 7f04f1ffb640 1 -- 172.21.15.16:0/2308843734 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05040a3680 con 0x562ed21e8970 2024-03-28T22:20:07.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.718+0000 7f05149f6740 1 -- 172.21.15.16:0/2308843734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ed21e8970 msgr2=0x562ed21ead60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:07.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.718+0000 7f05149f6740 1 --2- 172.21.15.16:0/2308843734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ed21e8970 0x562ed21ead60 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f0504097ae0 tx=0x7f050409b8d0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.718+0000 7f05149f6740 1 -- 172.21.15.16:0/2308843734 shutdown_connections 2024-03-28T22:20:07.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.718+0000 7f05149f6740 1 --2- 172.21.15.16:0/2308843734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ed21e8970 0x562ed21ead60 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.718+0000 7f05149f6740 1 --2- 172.21.15.16:0/2308843734 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ed21dfd90 0x562ed21e8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.718+0000 7f05149f6740 1 --2- 172.21.15.16:0/2308843734 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562ed21df470 0x562ed21df850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.718+0000 7f05149f6740 1 -- 172.21.15.16:0/2308843734 >> 172.21.15.16:0/2308843734 conn(0x562ed21d9ba0 msgr2=0x562ed21d9f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:07.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.719+0000 7f05149f6740 1 -- 172.21.15.16:0/2308843734 shutdown_connections 2024-03-28T22:20:07.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.719+0000 7f05149f6740 1 -- 172.21.15.16:0/2308843734 wait complete. 2024-03-28T22:20:07.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.719+0000 7f05149f6740 1 -- start start 2024-03-28T22:20:07.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.719+0000 7f05149f6740 1 Processor -- start 2024-03-28T22:20:07.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.719+0000 7f05149f6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ed21df470 0x562ed21e59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.719+0000 7f05149f6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562ed21dfd90 0x562ed21e5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.719+0000 7f05149f6740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ed21e8970 0x562ed21e6450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.719+0000 7f05149f6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562ed21eb760 con 0x562ed21df470 2024-03-28T22:20:07.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.719+0000 7f05149f6740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x562ed20e1470 con 0x562ed21e8970 2024-03-28T22:20:07.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.719+0000 7f05149f6740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x562ed21eb8e0 con 0x562ed21dfd90 2024-03-28T22:20:07.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.719+0000 7f05149f6740 10 client.0 _ll_register_callbacks cb 0x562ed20fecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:20:07.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.719+0000 7f050b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ed21df470 0x562ed21e59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.719+0000 7f050b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ed21df470 0x562ed21e59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50114/0 (socket says 172.21.15.16:50114) 2024-03-28T22:20:07.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.719+0000 7f050b7fe640 1 -- 172.21.15.16:0/4280840418 learned_addr learned my addr 172.21.15.16:0/4280840418 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:07.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.719+0000 7f050bfff640 1 --2- 172.21.15.16:0/4280840418 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ed21e8970 0x562ed21e6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.720+0000 7f050affd640 1 --2- 172.21.15.16:0/4280840418 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562ed21dfd90 0x562ed21e5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.720+0000 7f050b7fe640 1 -- 172.21.15.16:0/4280840418 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562ed21dfd90 msgr2=0x562ed21e5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:07.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.720+0000 7f050b7fe640 1 --2- 172.21.15.16:0/4280840418 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562ed21dfd90 0x562ed21e5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.720+0000 7f050b7fe640 1 -- 172.21.15.16:0/4280840418 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ed21e8970 msgr2=0x562ed21e6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:07.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.720+0000 7f050b7fe640 1 --2- 172.21.15.16:0/4280840418 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ed21e8970 0x562ed21e6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.720+0000 7f050b7fe640 1 -- 172.21.15.16:0/4280840418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ed1e573f0 con 0x562ed21df470 2024-03-28T22:20:07.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.720+0000 7f050bfff640 1 --2- 172.21.15.16:0/4280840418 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ed21e8970 0x562ed21e6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:20:07.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.720+0000 7f050b7fe640 1 --2- 172.21.15.16:0/4280840418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ed21df470 0x562ed21e59d0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f04fc00cd00 tx=0x7f04fc0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8125f1e8bfc436b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:07.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.720+0000 7f04eaffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:20:07.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.720+0000 7f04eaffd640 1 -- 172.21.15.16:0/4280840418 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04fc005220 con 0x562ed21df470 2024-03-28T22:20:07.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.720+0000 7f05149f6740 1 -- 172.21.15.16:0/4280840418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05040051d0 con 0x562ed21df470 2024-03-28T22:20:07.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.720+0000 7f04eaffd640 1 -- 172.21.15.16:0/4280840418 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7f04fc0053c0 con 0x562ed21df470 2024-03-28T22:20:07.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.720+0000 7f04eaffd640 1 -- 172.21.15.16:0/4280840418 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f04fc0055c0 con 0x562ed21df470 2024-03-28T22:20:07.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.720+0000 7f05149f6740 10 client.17496 fetch_fsmap learned FSMap version 27 2024-03-28T22:20:07.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.720+0000 7f05149f6740 1 -- 172.21.15.16:0/4280840418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562ed1f217d0 con 0x562ed21df470 2024-03-28T22:20:07.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.720+0000 7f04eaffd640 1 -- 172.21.15.16:0/4280840418 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05040051d0 con 0x562ed21df470 2024-03-28T22:20:07.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.720+0000 7f04eaffd640 1 -- 172.21.15.16:0/4280840418 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f04fc018030 con 0x562ed21df470 2024-03-28T22:20:07.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.721+0000 7f05149f6740 10 client.17496 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:20:07.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.721+0000 7f05149f6740 10 client.17496 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-28T22:20:07.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.721+0000 7f05149f6740 20 client.17496 populate_metadata read hostname 'smithi016' 2024-03-28T22:20:07.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.721+0000 7f05149f6740 1 --2- 172.21.15.16:0/4280840418 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x562ed21f8940 0x562ed21fad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.721+0000 7f05149f6740 4 client.17496 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:20:07.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.721+0000 7f05149f6740 1 -- 172.21.15.16:0/4280840418 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562ed21506f0 con 0x562ed21f8940 2024-03-28T22:20:07.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.721+0000 7f050affd640 1 --2- 172.21.15.16:0/4280840418 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x562ed21f8940 0x562ed21fad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.721+0000 7f050affd640 1 --2- 172.21.15.16:0/4280840418 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x562ed21f8940 0x562ed21fad90 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=436f68b500ca2f17 server_cookie=a247f0fd391fd3c8 in_seq=0 out_seq=0 2024-03-28T22:20:07.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.721+0000 7f04eaffd640 10 client.17496 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:20:07.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.722+0000 7f04eaffd640 1 -- 172.21.15.16:0/4280840418 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562ed21506f0 con 0x562ed21f8940 2024-03-28T22:20:07.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.722+0000 7f04eaffd640 10 client.17496 handle_command_reply: tid=0 2024-03-28T22:20:07.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.722+0000 7f05149f6740 1 client.17496 shutdown 2024-03-28T22:20:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.722+0000 7f05149f6740 1 -- 172.21.15.16:0/4280840418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ed21df470 msgr2=0x562ed21e59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.722+0000 7f05149f6740 1 --2- 172.21.15.16:0/4280840418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ed21df470 0x562ed21e59d0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f04fc00cd00 tx=0x7f04fc0039c0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.722+0000 7f05149f6740 1 -- 172.21.15.16:0/4280840418 shutdown_connections 2024-03-28T22:20:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.722+0000 7f05149f6740 1 --2- 172.21.15.16:0/4280840418 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x562ed21f8940 0x562ed21fad90 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.722+0000 7f05149f6740 1 --2- 172.21.15.16:0/4280840418 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ed21e8970 0x562ed21e6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.722+0000 7f04eaffd640 0 client.17496 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:20:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.722+0000 7f05149f6740 1 --2- 172.21.15.16:0/4280840418 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562ed21dfd90 0x562ed21e5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.722+0000 7f05149f6740 1 --2- 172.21.15.16:0/4280840418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ed21df470 0x562ed21e59d0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.722+0000 7f05149f6740 1 -- 172.21.15.16:0/4280840418 >> 172.21.15.16:0/4280840418 conn(0x562ed21d9ba0 msgr2=0x562ed21db8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:07.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.723+0000 7f05149f6740 1 -- 172.21.15.16:0/4280840418 shutdown_connections 2024-03-28T22:20:07.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.723+0000 7f05149f6740 1 -- 172.21.15.16:0/4280840418 wait complete. 2024-03-28T22:20:07.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.723+0000 7f05149f6740 20 client.17496 trim_cache size 0 max 16384 2024-03-28T22:20:07.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.736+0000 7f05149f6740 1 Processor -- start 2024-03-28T22:20:07.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.736+0000 7f05149f6740 1 -- start start 2024-03-28T22:20:07.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.736+0000 7f05149f6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562ed21df470 0x562ed21e8100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.736+0000 7f05149f6740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ed21dfd90 0x562ed21e4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.736+0000 7f05149f6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ed21e8970 0x562ed21f4500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.736+0000 7f05149f6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f050c1afd20 con 0x562ed21e8970 2024-03-28T22:20:07.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.736+0000 7f05149f6740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f050c1ad5f0 con 0x562ed21dfd90 2024-03-28T22:20:07.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.736+0000 7f05149f6740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f050c1ab460 con 0x562ed21df470 2024-03-28T22:20:07.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.736+0000 7f050bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ed21e8970 0x562ed21f4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.736+0000 7f050bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ed21e8970 0x562ed21f4500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50120/0 (socket says 172.21.15.16:50120) 2024-03-28T22:20:07.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.736+0000 7f050bfff640 1 -- 172.21.15.16:0/89324053 learned_addr learned my addr 172.21.15.16:0/89324053 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:07.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.736+0000 7f050affd640 1 --2- 172.21.15.16:0/89324053 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ed21dfd90 0x562ed21e4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.736+0000 7f050b7fe640 1 --2- 172.21.15.16:0/89324053 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562ed21df470 0x562ed21e8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.736+0000 7f050bfff640 1 -- 172.21.15.16:0/89324053 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562ed21df470 msgr2=0x562ed21e8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:07.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.736+0000 7f050bfff640 1 --2- 172.21.15.16:0/89324053 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562ed21df470 0x562ed21e8100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.736+0000 7f050bfff640 1 -- 172.21.15.16:0/89324053 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ed21dfd90 msgr2=0x562ed21e4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:07.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.736+0000 7f050bfff640 1 --2- 172.21.15.16:0/89324053 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ed21dfd90 0x562ed21e4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.736+0000 7f050bfff640 1 -- 172.21.15.16:0/89324053 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ed21dfbc0 con 0x562ed21e8970 2024-03-28T22:20:07.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.736+0000 7f050affd640 1 --2- 172.21.15.16:0/89324053 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ed21dfd90 0x562ed21e4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:20:07.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.736+0000 7f050b7fe640 1 --2- 172.21.15.16:0/89324053 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562ed21df470 0x562ed21e8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:20:07.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.737+0000 7f050bfff640 1 --2- 172.21.15.16:0/89324053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ed21e8970 0x562ed21f4500 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f0504009730 tx=0x7f05040096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e067f9db92ed4445 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:07.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.737+0000 7f04f17fa640 1 -- 172.21.15.16:0/89324053 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0504065070 con 0x562ed21e8970 2024-03-28T22:20:07.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.737+0000 7f04f17fa640 1 -- 172.21.15.16:0/89324053 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05040b9070 con 0x562ed21e8970 2024-03-28T22:20:07.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.737+0000 7f04f17fa640 1 -- 172.21.15.16:0/89324053 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f050409a560 con 0x562ed21e8970 2024-03-28T22:20:07.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.737+0000 7f05149f6740 1 -- 172.21.15.16:0/89324053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ed21e8970 msgr2=0x562ed21f4500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:07.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.737+0000 7f05149f6740 1 --2- 172.21.15.16:0/89324053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ed21e8970 0x562ed21f4500 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f0504009730 tx=0x7f05040096c0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.737+0000 7f05149f6740 1 -- 172.21.15.16:0/89324053 shutdown_connections 2024-03-28T22:20:07.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.737+0000 7f05149f6740 1 --2- 172.21.15.16:0/89324053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ed21e8970 0x562ed21f4500 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.737+0000 7f05149f6740 1 --2- 172.21.15.16:0/89324053 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ed21dfd90 0x562ed21e4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.737+0000 7f05149f6740 1 --2- 172.21.15.16:0/89324053 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562ed21df470 0x562ed21e8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.737+0000 7f05149f6740 1 -- 172.21.15.16:0/89324053 >> 172.21.15.16:0/89324053 conn(0x562ed21d9ba0 msgr2=0x562ed21d84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:07.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.737+0000 7f05149f6740 1 -- 172.21.15.16:0/89324053 shutdown_connections 2024-03-28T22:20:07.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.737+0000 7f05149f6740 1 -- 172.21.15.16:0/89324053 wait complete. 2024-03-28T22:20:07.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.738+0000 7f05149f6740 1 -- start start 2024-03-28T22:20:07.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.738+0000 7f05149f6740 1 Processor -- start 2024-03-28T22:20:07.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.738+0000 7f05149f6740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ed21df470 0x562ed21f0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.738+0000 7f05149f6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562ed21dfd90 0x562ed21ec290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.738+0000 7f05149f6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ed21e8970 0x562ed21ec9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.738+0000 7f05149f6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f050c1abab0 con 0x562ed21e8970 2024-03-28T22:20:07.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.738+0000 7f05149f6740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f050c1acf20 con 0x562ed21df470 2024-03-28T22:20:07.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.738+0000 7f05149f6740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f050c1ac180 con 0x562ed21dfd90 2024-03-28T22:20:07.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.738+0000 7f05149f6740 10 client.0 _ll_register_callbacks cb 0x7f04fc004520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:20:07.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.738+0000 7f050bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ed21e8970 0x562ed21ec9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.738+0000 7f050bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ed21e8970 0x562ed21ec9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50134/0 (socket says 172.21.15.16:50134) 2024-03-28T22:20:07.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.738+0000 7f050bfff640 1 -- 172.21.15.16:0/147756795 learned_addr learned my addr 172.21.15.16:0/147756795 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:07.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.738+0000 7f050b7fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ed21df470 0x562ed21f0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.738+0000 7f050affd640 1 --2- 172.21.15.16:0/147756795 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562ed21dfd90 0x562ed21ec290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.738+0000 7f050bfff640 1 -- 172.21.15.16:0/147756795 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562ed21dfd90 msgr2=0x562ed21ec290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:07.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.738+0000 7f050bfff640 1 --2- 172.21.15.16:0/147756795 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562ed21dfd90 0x562ed21ec290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.738+0000 7f050bfff640 1 -- 172.21.15.16:0/147756795 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ed21df470 msgr2=0x562ed21f0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:07.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.738+0000 7f050bfff640 1 --2- 172.21.15.16:0/147756795 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ed21df470 0x562ed21f0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.738+0000 7f050bfff640 1 -- 172.21.15.16:0/147756795 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ed225dc90 con 0x562ed21e8970 2024-03-28T22:20:07.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.739+0000 7f050bfff640 1 --2- 172.21.15.16:0/147756795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ed21e8970 0x562ed21ec9d0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f050400c760 tx=0x7f050409f3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=446d94dedadc309a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:07.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.739+0000 7f04eaffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:20:07.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.739+0000 7f04eaffd640 1 -- 172.21.15.16:0/147756795 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0504065070 con 0x562ed21e8970 2024-03-28T22:20:07.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.739+0000 7f04eaffd640 1 -- 172.21.15.16:0/147756795 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7f050409a760 con 0x562ed21e8970 2024-03-28T22:20:07.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.739+0000 7f05149f6740 1 -- 172.21.15.16:0/147756795 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05040b9070 con 0x562ed21e8970 2024-03-28T22:20:07.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.739+0000 7f04eaffd640 1 -- 172.21.15.16:0/147756795 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f05040a36a0 con 0x562ed21e8970 2024-03-28T22:20:07.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.739+0000 7f05149f6740 10 client.17502 fetch_fsmap learned FSMap version 27 2024-03-28T22:20:07.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.739+0000 7f05149f6740 1 -- 172.21.15.16:0/147756795 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562ed225ed40 con 0x562ed21e8970 2024-03-28T22:20:07.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.739+0000 7f04eaffd640 1 -- 172.21.15.16:0/147756795 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05040b9070 con 0x562ed21e8970 2024-03-28T22:20:07.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.739+0000 7f04eaffd640 1 -- 172.21.15.16:0/147756795 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f050409a060 con 0x562ed21e8970 2024-03-28T22:20:07.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.739+0000 7f05149f6740 10 client.17502 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:20:07.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.739+0000 7f05149f6740 10 client.17502 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-28T22:20:07.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.739+0000 7f05149f6740 20 client.17502 populate_metadata read hostname 'smithi016' 2024-03-28T22:20:07.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.739+0000 7f05149f6740 1 --2- 172.21.15.16:0/147756795 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x562ed225efc0 0x562ed22613b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.739+0000 7f05149f6740 4 client.17502 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:20:07.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.739+0000 7f05149f6740 1 -- 172.21.15.16:0/147756795 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562ed214aa40 con 0x562ed225efc0 2024-03-28T22:20:07.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.739+0000 7f050b7fe640 1 --2- 172.21.15.16:0/147756795 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x562ed225efc0 0x562ed22613b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.740+0000 7f050b7fe640 1 --2- 172.21.15.16:0/147756795 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x562ed225efc0 0x562ed22613b0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=181837da5b954f07 server_cookie=c73dc892b5824eb5 in_seq=0 out_seq=0 2024-03-28T22:20:07.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.740+0000 7f04eaffd640 10 client.17502 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:20:07.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.740+0000 7f04eaffd640 1 -- 172.21.15.16:0/147756795 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562ed214aa40 con 0x562ed225efc0 2024-03-28T22:20:07.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.740+0000 7f04eaffd640 10 client.17502 handle_command_reply: tid=0 2024-03-28T22:20:07.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.740+0000 7f05149f6740 1 client.17502 shutdown 2024-03-28T22:20:07.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.740+0000 7f05149f6740 1 -- 172.21.15.16:0/147756795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ed21e8970 msgr2=0x562ed21ec9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:07.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.740+0000 7f05149f6740 1 --2- 172.21.15.16:0/147756795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ed21e8970 0x562ed21ec9d0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f050400c760 tx=0x7f050409f3e0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.740+0000 7f05149f6740 1 -- 172.21.15.16:0/147756795 shutdown_connections 2024-03-28T22:20:07.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.740+0000 7f05149f6740 1 --2- 172.21.15.16:0/147756795 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x562ed225efc0 0x562ed22613b0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.740+0000 7f05149f6740 1 --2- 172.21.15.16:0/147756795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562ed21e8970 0x562ed21ec9d0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.740+0000 7f05149f6740 1 --2- 172.21.15.16:0/147756795 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562ed21dfd90 0x562ed21ec290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.740+0000 7f04eaffd640 0 client.17502 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:20:07.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.740+0000 7f05149f6740 1 --2- 172.21.15.16:0/147756795 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ed21df470 0x562ed21f0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.740+0000 7f05149f6740 1 -- 172.21.15.16:0/147756795 >> 172.21.15.16:0/147756795 conn(0x562ed21d9ba0 msgr2=0x562ed21d84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:07.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.740+0000 7f05149f6740 1 -- 172.21.15.16:0/147756795 shutdown_connections 2024-03-28T22:20:07.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.740+0000 7f05149f6740 1 -- 172.21.15.16:0/147756795 wait complete. 2024-03-28T22:20:07.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.740+0000 7f05149f6740 20 client.17502 trim_cache size 0 max 16384 2024-03-28T22:20:07.759 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:20:07.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.741+0000 7f0511e15640 1 -- 172.21.15.16:0/467948326 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f04f40a1ef0 msgr2=0x7f04f40a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:07.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.741+0000 7f0511e15640 1 --2- 172.21.15.16:0/467948326 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f04f40a1ef0 0x7f04f40a43b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0500005ab0 tx=0x7f050002c7a0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.741+0000 7f0511e15640 1 -- 172.21.15.16:0/467948326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f050c106690 msgr2=0x7f050c19c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:07.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.741+0000 7f0511e15640 1 --2- 172.21.15.16:0/467948326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f050c106690 0x7f050c19c910 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f050400ea60 tx=0x7f050400ef20 comp rx=0 tx=0).stop 2024-03-28T22:20:07.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.741+0000 7f0511e15640 1 -- 172.21.15.16:0/467948326 shutdown_connections 2024-03-28T22:20:07.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.741+0000 7f0511e15640 1 --2- 172.21.15.16:0/467948326 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f04f40a1ef0 0x7f04f40a43b0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.741+0000 7f0511e15640 1 --2- 172.21.15.16:0/467948326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f050c106690 0x7f050c19c910 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.741+0000 7f0511e15640 1 --2- 172.21.15.16:0/467948326 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f050c100fb0 0x7f050c198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.741+0000 7f0511e15640 1 --2- 172.21.15.16:0/467948326 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f050c1005f0 0x7f050c198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.741+0000 7f0511e15640 1 -- 172.21.15.16:0/467948326 >> 172.21.15.16:0/467948326 conn(0x7f050c077800 msgr2=0x7f050c0ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:07.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.741+0000 7f0511e15640 1 -- 172.21.15.16:0/467948326 shutdown_connections 2024-03-28T22:20:07.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.741+0000 7f0511e15640 1 -- 172.21.15.16:0/467948326 wait complete. 2024-03-28T22:20:07.765 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:20:07.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.869+0000 7efd4b0cc640 1 -- 172.21.15.16:0/2718692802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd441020a0 msgr2=0x7efd4410e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.869+0000 7efd4b0cc640 1 --2- 172.21.15.16:0/2718692802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd441020a0 0x7efd4410e5b0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7efd3c009cd0 tx=0x7efd3c02c0f0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.869+0000 7efd4b0cc640 1 -- 172.21.15.16:0/2718692802 shutdown_connections 2024-03-28T22:20:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.869+0000 7efd4b0cc640 1 --2- 172.21.15.16:0/2718692802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd441020a0 0x7efd4410e5b0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.869+0000 7efd4b0cc640 1 --2- 172.21.15.16:0/2718692802 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd441016e0 0x7efd44101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.869+0000 7efd4b0cc640 1 --2- 172.21.15.16:0/2718692802 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efd441076f0 0x7efd44107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.869+0000 7efd4b0cc640 1 -- 172.21.15.16:0/2718692802 >> 172.21.15.16:0/2718692802 conn(0x7efd440fd1f0 msgr2=0x7efd440ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:07.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.869+0000 7efd4b0cc640 1 -- 172.21.15.16:0/2718692802 shutdown_connections 2024-03-28T22:20:07.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.869+0000 7efd4b0cc640 1 -- 172.21.15.16:0/2718692802 wait complete. 2024-03-28T22:20:07.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.869+0000 7efd4b0cc640 1 Processor -- start 2024-03-28T22:20:07.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.869+0000 7efd4b0cc640 1 -- start start 2024-03-28T22:20:07.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.870+0000 7efd4b0cc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd441016e0 0x7efd44197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.870+0000 7efd4b0cc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efd441020a0 0x7efd441984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.870+0000 7efd4b0cc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd441076f0 0x7efd4419c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.870+0000 7efd4b0cc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efd44111b00 con 0x7efd441016e0 2024-03-28T22:20:07.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.870+0000 7efd4b0cc640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7efd44111980 con 0x7efd441076f0 2024-03-28T22:20:07.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.870+0000 7efd4b0cc640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7efd44111c80 con 0x7efd441020a0 2024-03-28T22:20:07.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.870+0000 7efd48e68640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd441016e0 0x7efd44197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.870+0000 7efd48e68640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd441016e0 0x7efd44197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50146/0 (socket says 172.21.15.16:50146) 2024-03-28T22:20:07.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.870+0000 7efd48e68640 1 -- 172.21.15.16:0/2776415322 learned_addr learned my addr 172.21.15.16:0/2776415322 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:07.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.870+0000 7efd49669640 1 --2- 172.21.15.16:0/2776415322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd441076f0 0x7efd4419c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.870+0000 7efd43fff640 1 --2- 172.21.15.16:0/2776415322 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efd441020a0 0x7efd441984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.870+0000 7efd48e68640 1 -- 172.21.15.16:0/2776415322 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efd441020a0 msgr2=0x7efd441984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:07.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.870+0000 7efd48e68640 1 --2- 172.21.15.16:0/2776415322 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efd441020a0 0x7efd441984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.870+0000 7efd48e68640 1 -- 172.21.15.16:0/2776415322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd441076f0 msgr2=0x7efd4419c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:07.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.870+0000 7efd48e68640 1 --2- 172.21.15.16:0/2776415322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd441076f0 0x7efd4419c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.870+0000 7efd48e68640 1 -- 172.21.15.16:0/2776415322 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd4419cf70 con 0x7efd441016e0 2024-03-28T22:20:07.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.870+0000 7efd43fff640 1 --2- 172.21.15.16:0/2776415322 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efd441020a0 0x7efd441984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:20:07.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.870+0000 7efd48e68640 1 --2- 172.21.15.16:0/2776415322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd441016e0 0x7efd44197f80 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7efd3800d950 tx=0x7efd3800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:07.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.870+0000 7efd41ffb640 1 -- 172.21.15.16:0/2776415322 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd38017070 con 0x7efd441016e0 2024-03-28T22:20:07.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.870+0000 7efd4b0cc640 1 -- 172.21.15.16:0/2776415322 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd4419d260 con 0x7efd441016e0 2024-03-28T22:20:07.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.870+0000 7efd41ffb640 1 -- 172.21.15.16:0/2776415322 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd38009c30 con 0x7efd441016e0 2024-03-28T22:20:07.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.870+0000 7efd41ffb640 1 -- 172.21.15.16:0/2776415322 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd3800f6b0 con 0x7efd441016e0 2024-03-28T22:20:07.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.870+0000 7efd4b0cc640 1 -- 172.21.15.16:0/2776415322 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd441a4af0 con 0x7efd441016e0 2024-03-28T22:20:07.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.873+0000 7efd41ffb640 1 -- 172.21.15.16:0/2776415322 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7efd380109d0 con 0x7efd441016e0 2024-03-28T22:20:07.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.873+0000 7efd4dd83740 1 Processor -- start 2024-03-28T22:20:07.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.873+0000 7efd4dd83740 1 -- start start 2024-03-28T22:20:07.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.873+0000 7efd4dd83740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574e2602470 0x5574e2602850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.873+0000 7efd4dd83740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e2602d90 0x5574e260b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.873+0000 7efd4dd83740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574e260b970 0x5574e260dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.873+0000 7efd4dd83740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5574e25661b0 con 0x5574e260b970 2024-03-28T22:20:07.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.873+0000 7efd4dd83740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5574e250ec00 con 0x5574e2602d90 2024-03-28T22:20:07.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.873+0000 7efd4dd83740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5574e2562250 con 0x5574e2602470 2024-03-28T22:20:07.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.873+0000 7efd41ffb640 1 --2- 172.21.15.16:0/2776415322 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7efd340a1ef0 0x7efd340a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.873+0000 7efd48e68640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e2602d90 0x5574e260b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.873+0000 7efd43fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574e260b970 0x5574e260dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.873+0000 7efd43fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574e260b970 0x5574e260dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50148/0 (socket says 172.21.15.16:50148) 2024-03-28T22:20:07.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.873+0000 7efd43fff640 1 -- 172.21.15.16:0/925287358 learned_addr learned my addr 172.21.15.16:0/925287358 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:07.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.873+0000 7efd49669640 1 --2- 172.21.15.16:0/2776415322 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7efd340a1ef0 0x7efd340a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.873+0000 7efd49669640 1 --2- 172.21.15.16:0/925287358 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574e2602470 0x5574e2602850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.873+0000 7efd43fff640 1 -- 172.21.15.16:0/925287358 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574e2602470 msgr2=0x5574e2602850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:07.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.873+0000 7efd43fff640 1 --2- 172.21.15.16:0/925287358 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574e2602470 0x5574e2602850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.873+0000 7efd43fff640 1 -- 172.21.15.16:0/925287358 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e2602d90 msgr2=0x5574e260b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:07.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.873+0000 7efd41ffb640 1 -- 172.21.15.16:0/2776415322 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7efd380cdd70 con 0x7efd441016e0 2024-03-28T22:20:07.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.873+0000 7efd43fff640 1 --2- 172.21.15.16:0/925287358 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e2602d90 0x5574e260b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.873+0000 7efd43fff640 1 -- 172.21.15.16:0/925287358 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574e252e3d0 con 0x5574e260b970 2024-03-28T22:20:07.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.874+0000 7efd48e68640 1 --2- 172.21.15.16:0/925287358 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e2602d90 0x5574e260b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:20:07.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.874+0000 7efd43fff640 1 --2- 172.21.15.16:0/925287358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574e260b970 0x5574e260dd60 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7efd30009cf0 tx=0x7efd300117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47481c9e817e6f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:07.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.874+0000 7efd49669640 1 --2- 172.21.15.16:0/2776415322 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7efd340a1ef0 0x7efd340a43b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7efd3c02c630 tx=0x7efd3c02e040 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:07.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.874+0000 7efd2effd640 1 -- 172.21.15.16:0/925287358 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd30012450 con 0x5574e260b970 2024-03-28T22:20:07.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.874+0000 7efd2effd640 1 -- 172.21.15.16:0/925287358 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd30012ad0 con 0x5574e260b970 2024-03-28T22:20:07.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.874+0000 7efd2effd640 1 -- 172.21.15.16:0/925287358 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd3000fbf0 con 0x5574e260b970 2024-03-28T22:20:07.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.874+0000 7efd4dd83740 1 -- 172.21.15.16:0/925287358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574e260b970 msgr2=0x5574e260dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:07.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.874+0000 7efd4dd83740 1 --2- 172.21.15.16:0/925287358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574e260b970 0x5574e260dd60 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7efd30009cf0 tx=0x7efd300117b0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.875+0000 7efd4dd83740 1 -- 172.21.15.16:0/925287358 shutdown_connections 2024-03-28T22:20:07.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.875+0000 7efd4dd83740 1 --2- 172.21.15.16:0/925287358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574e260b970 0x5574e260dd60 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.875+0000 7efd4dd83740 1 --2- 172.21.15.16:0/925287358 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e2602d90 0x5574e260b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.875+0000 7efd4dd83740 1 --2- 172.21.15.16:0/925287358 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574e2602470 0x5574e2602850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.875+0000 7efd4dd83740 1 -- 172.21.15.16:0/925287358 >> 172.21.15.16:0/925287358 conn(0x5574e25fcba0 msgr2=0x5574e25fcf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:07.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.875+0000 7efd4dd83740 1 -- 172.21.15.16:0/925287358 shutdown_connections 2024-03-28T22:20:07.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.875+0000 7efd4dd83740 1 -- 172.21.15.16:0/925287358 wait complete. 2024-03-28T22:20:07.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.875+0000 7efd4dd83740 1 -- start start 2024-03-28T22:20:07.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.875+0000 7efd4dd83740 1 Processor -- start 2024-03-28T22:20:07.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.875+0000 7efd4dd83740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574e2602470 0x5574e2608930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.875+0000 7efd4dd83740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e2602d90 0x5574e2608e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.875+0000 7efd4dd83740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574e260b970 0x5574e26093b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.875+0000 7efd4dd83740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5574e260e760 con 0x5574e2602470 2024-03-28T22:20:07.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.875+0000 7efd4dd83740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5574e2504470 con 0x5574e2602d90 2024-03-28T22:20:07.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.875+0000 7efd4dd83740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5574e260e8e0 con 0x5574e260b970 2024-03-28T22:20:07.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.875+0000 7efd4dd83740 10 client.0 _ll_register_callbacks cb 0x5574e2521cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:20:07.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.875+0000 7efd48e68640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574e2602470 0x5574e2608930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.875+0000 7efd48e68640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574e2602470 0x5574e2608930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50154/0 (socket says 172.21.15.16:50154) 2024-03-28T22:20:07.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.875+0000 7efd48e68640 1 -- 172.21.15.16:0/1998236169 learned_addr learned my addr 172.21.15.16:0/1998236169 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:07.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.876+0000 7efd43fff640 1 --2- 172.21.15.16:0/1998236169 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e2602d90 0x5574e2608e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.876+0000 7efd49669640 1 --2- 172.21.15.16:0/1998236169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574e260b970 0x5574e26093b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.876+0000 7efd48e68640 1 -- 172.21.15.16:0/1998236169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574e260b970 msgr2=0x5574e26093b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:07.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.876+0000 7efd48e68640 1 --2- 172.21.15.16:0/1998236169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574e260b970 0x5574e26093b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.876+0000 7efd48e68640 1 -- 172.21.15.16:0/1998236169 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e2602d90 msgr2=0x5574e2608e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:07.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.876+0000 7efd48e68640 1 --2- 172.21.15.16:0/1998236169 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e2602d90 0x5574e2608e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.876+0000 7efd48e68640 1 -- 172.21.15.16:0/1998236169 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574e227a3f0 con 0x5574e2602470 2024-03-28T22:20:07.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.876+0000 7efd49669640 1 --2- 172.21.15.16:0/1998236169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574e260b970 0x5574e26093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:20:07.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.876+0000 7efd48e68640 1 --2- 172.21.15.16:0/1998236169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574e2602470 0x5574e2608930 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7efd3809a6f0 tx=0x7efd3800bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79c5b668df803a81 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:07.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.876+0000 7efd23fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:20:07.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.876+0000 7efd23fff640 1 -- 172.21.15.16:0/1998236169 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd38020070 con 0x5574e2602470 2024-03-28T22:20:07.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.876+0000 7efd23fff640 1 -- 172.21.15.16:0/1998236169 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7efd38096dd0 con 0x5574e2602470 2024-03-28T22:20:07.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.876+0000 7efd4dd83740 1 -- 172.21.15.16:0/1998236169 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd30012ad0 con 0x5574e2602470 2024-03-28T22:20:07.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.876+0000 7efd23fff640 1 -- 172.21.15.16:0/1998236169 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7efd3809a760 con 0x5574e2602470 2024-03-28T22:20:07.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.876+0000 7efd4dd83740 10 client.17514 fetch_fsmap learned FSMap version 27 2024-03-28T22:20:07.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.876+0000 7efd4dd83740 1 -- 172.21.15.16:0/1998236169 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5574e23447d0 con 0x5574e2602470 2024-03-28T22:20:07.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.876+0000 7efd23fff640 1 -- 172.21.15.16:0/1998236169 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd30012ad0 con 0x5574e2602470 2024-03-28T22:20:07.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.876+0000 7efd23fff640 1 -- 172.21.15.16:0/1998236169 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7efd38099bb0 con 0x5574e2602470 2024-03-28T22:20:07.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.877+0000 7efd4dd83740 10 client.17514 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:20:07.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.877+0000 7efd4dd83740 10 client.17514 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T22:20:07.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.877+0000 7efd4dd83740 20 client.17514 populate_metadata read hostname 'smithi016' 2024-03-28T22:20:07.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.877+0000 7efd4dd83740 1 --2- 172.21.15.16:0/1998236169 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5574e261b920 0x5574e261dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.877+0000 7efd4dd83740 4 client.17514 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:20:07.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.877+0000 7efd4dd83740 1 -- 172.21.15.16:0/1998236169 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5574e25736f0 con 0x5574e261b920 2024-03-28T22:20:07.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.877+0000 7efd43fff640 1 --2- 172.21.15.16:0/1998236169 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5574e261b920 0x5574e261dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.877+0000 7efd43fff640 1 --2- 172.21.15.16:0/1998236169 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5574e261b920 0x5574e261dd50 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b8a437d2e80be694 server_cookie=a47d087a077d972e in_seq=0 out_seq=0 2024-03-28T22:20:07.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.877+0000 7efd23fff640 10 client.17514 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:20:07.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.878+0000 7efd23fff640 1 -- 172.21.15.16:0/1998236169 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5574e25736f0 con 0x5574e261b920 2024-03-28T22:20:07.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.878+0000 7efd23fff640 10 client.17514 handle_command_reply: tid=0 2024-03-28T22:20:07.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.878+0000 7efd4dd83740 1 client.17514 shutdown 2024-03-28T22:20:07.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.879+0000 7efd4dd83740 1 -- 172.21.15.16:0/1998236169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574e2602470 msgr2=0x5574e2608930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:07.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.879+0000 7efd4dd83740 1 --2- 172.21.15.16:0/1998236169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574e2602470 0x5574e2608930 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7efd3809a6f0 tx=0x7efd3800bf60 comp rx=0 tx=0).stop 2024-03-28T22:20:07.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.879+0000 7efd4dd83740 1 -- 172.21.15.16:0/1998236169 shutdown_connections 2024-03-28T22:20:07.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.879+0000 7efd4dd83740 1 --2- 172.21.15.16:0/1998236169 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5574e261b920 0x5574e261dd50 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.879+0000 7efd4dd83740 1 --2- 172.21.15.16:0/1998236169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574e260b970 0x5574e26093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.879+0000 7efd23fff640 0 client.17514 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:20:07.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.879+0000 7efd4dd83740 1 --2- 172.21.15.16:0/1998236169 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e2602d90 0x5574e2608e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.879+0000 7efd4dd83740 1 --2- 172.21.15.16:0/1998236169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574e2602470 0x5574e2608930 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.879+0000 7efd4dd83740 1 -- 172.21.15.16:0/1998236169 >> 172.21.15.16:0/1998236169 conn(0x5574e25fcba0 msgr2=0x5574e25fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:07.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.879+0000 7efd4dd83740 1 -- 172.21.15.16:0/1998236169 shutdown_connections 2024-03-28T22:20:07.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.879+0000 7efd4dd83740 1 -- 172.21.15.16:0/1998236169 wait complete. 2024-03-28T22:20:07.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.879+0000 7efd4dd83740 20 client.17514 trim_cache size 0 max 16384 2024-03-28T22:20:07.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.892+0000 7efd4dd83740 1 Processor -- start 2024-03-28T22:20:07.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.892+0000 7efd4dd83740 1 -- start start 2024-03-28T22:20:07.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.892+0000 7efd4dd83740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574e2602470 0x5574e2617530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.892+0000 7efd4dd83740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e2602d90 0x5574e26235d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.892+0000 7efd4dd83740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574e260b970 0x5574e2613b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.892+0000 7efd4dd83740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efd441afc00 con 0x5574e260b970 2024-03-28T22:20:07.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.892+0000 7efd4dd83740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7efd441ad4d0 con 0x5574e2602d90 2024-03-28T22:20:07.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.892+0000 7efd4dd83740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7efd441ab340 con 0x5574e2602470 2024-03-28T22:20:07.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.892+0000 7efd49669640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574e260b970 0x5574e2613b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.892+0000 7efd49669640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574e260b970 0x5574e2613b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50166/0 (socket says 172.21.15.16:50166) 2024-03-28T22:20:07.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.892+0000 7efd49669640 1 -- 172.21.15.16:0/1595495838 learned_addr learned my addr 172.21.15.16:0/1595495838 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:07.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.892+0000 7efd43fff640 1 --2- 172.21.15.16:0/1595495838 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e2602d90 0x5574e26235d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.893+0000 7efd48e68640 1 --2- 172.21.15.16:0/1595495838 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574e2602470 0x5574e2617530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.893+0000 7efd49669640 1 -- 172.21.15.16:0/1595495838 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574e2602470 msgr2=0x5574e2617530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:07.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.893+0000 7efd49669640 1 --2- 172.21.15.16:0/1595495838 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574e2602470 0x5574e2617530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.893+0000 7efd49669640 1 -- 172.21.15.16:0/1595495838 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e2602d90 msgr2=0x5574e26235d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:07.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.893+0000 7efd49669640 1 --2- 172.21.15.16:0/1595495838 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e2602d90 0x5574e26235d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.893+0000 7efd49669640 1 -- 172.21.15.16:0/1595495838 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574e25fc9d0 con 0x5574e260b970 2024-03-28T22:20:07.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.893+0000 7efd43fff640 1 --2- 172.21.15.16:0/1595495838 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e2602d90 0x5574e26235d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:20:07.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.893+0000 7efd48e68640 1 --2- 172.21.15.16:0/1595495838 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574e2602470 0x5574e2617530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:20:07.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.893+0000 7efd49669640 1 --2- 172.21.15.16:0/1595495838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574e260b970 0x5574e2613b50 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7efd3c036000 tx=0x7efd3c005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34bef53676ae7ea8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:07.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.893+0000 7efd2e7fc640 1 -- 172.21.15.16:0/1595495838 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd3c039040 con 0x5574e260b970 2024-03-28T22:20:07.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.893+0000 7efd2e7fc640 1 -- 172.21.15.16:0/1595495838 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd3c047070 con 0x5574e260b970 2024-03-28T22:20:07.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.893+0000 7efd2e7fc640 1 -- 172.21.15.16:0/1595495838 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd3c043400 con 0x5574e260b970 2024-03-28T22:20:07.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.894+0000 7efd4dd83740 1 -- 172.21.15.16:0/1595495838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574e260b970 msgr2=0x5574e2613b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:07.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.894+0000 7efd4dd83740 1 --2- 172.21.15.16:0/1595495838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574e260b970 0x5574e2613b50 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7efd3c036000 tx=0x7efd3c005d80 comp rx=0 tx=0).stop 2024-03-28T22:20:07.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.894+0000 7efd4dd83740 1 -- 172.21.15.16:0/1595495838 shutdown_connections 2024-03-28T22:20:07.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.894+0000 7efd4dd83740 1 --2- 172.21.15.16:0/1595495838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574e260b970 0x5574e2613b50 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.894+0000 7efd4dd83740 1 --2- 172.21.15.16:0/1595495838 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e2602d90 0x5574e26235d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.894+0000 7efd4dd83740 1 --2- 172.21.15.16:0/1595495838 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574e2602470 0x5574e2617530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.894+0000 7efd4dd83740 1 -- 172.21.15.16:0/1595495838 >> 172.21.15.16:0/1595495838 conn(0x5574e25fcba0 msgr2=0x5574e26088f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:07.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.894+0000 7efd4dd83740 1 -- 172.21.15.16:0/1595495838 shutdown_connections 2024-03-28T22:20:07.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.894+0000 7efd4dd83740 1 -- 172.21.15.16:0/1595495838 wait complete. 2024-03-28T22:20:07.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.894+0000 7efd4dd83740 1 -- start start 2024-03-28T22:20:07.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.894+0000 7efd4dd83740 1 Processor -- start 2024-03-28T22:20:07.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.894+0000 7efd4dd83740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574e2602470 0x5574e2679f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.894+0000 7efd4dd83740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574e2602d90 0x5574e267a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.894+0000 7efd4dd83740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e260b970 0x5574e260fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.894+0000 7efd4dd83740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efd441ab990 con 0x5574e2602470 2024-03-28T22:20:07.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.894+0000 7efd4dd83740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7efd441ace00 con 0x5574e260b970 2024-03-28T22:20:07.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.894+0000 7efd4dd83740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7efd441ac060 con 0x5574e2602d90 2024-03-28T22:20:07.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.894+0000 7efd4dd83740 10 client.0 _ll_register_callbacks cb 0x7efd38009de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:20:07.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.894+0000 7efd48e68640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574e2602470 0x5574e2679f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.895+0000 7efd48e68640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574e2602470 0x5574e2679f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50176/0 (socket says 172.21.15.16:50176) 2024-03-28T22:20:07.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.895+0000 7efd48e68640 1 -- 172.21.15.16:0/1000745789 learned_addr learned my addr 172.21.15.16:0/1000745789 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:07.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.895+0000 7efd49669640 1 --2- 172.21.15.16:0/1000745789 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e260b970 0x5574e260fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.895+0000 7efd43fff640 1 --2- 172.21.15.16:0/1000745789 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574e2602d90 0x5574e267a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.895+0000 7efd48e68640 1 -- 172.21.15.16:0/1000745789 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574e2602d90 msgr2=0x5574e267a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:07.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.895+0000 7efd48e68640 1 --2- 172.21.15.16:0/1000745789 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574e2602d90 0x5574e267a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.895+0000 7efd48e68640 1 -- 172.21.15.16:0/1000745789 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e260b970 msgr2=0x5574e260fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:07.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.895+0000 7efd48e68640 1 --2- 172.21.15.16:0/1000745789 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e260b970 0x5574e260fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.895+0000 7efd48e68640 1 -- 172.21.15.16:0/1000745789 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574e2610430 con 0x5574e2602470 2024-03-28T22:20:07.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.895+0000 7efd48e68640 1 --2- 172.21.15.16:0/1000745789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574e2602470 0x5574e2679f00 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7efd38097130 tx=0x7efd38005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=236901f2785a4f38 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:07.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.895+0000 7efd23fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:20:07.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.895+0000 7efd23fff640 1 -- 172.21.15.16:0/1000745789 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd380b8060 con 0x5574e2602470 2024-03-28T22:20:07.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.895+0000 7efd23fff640 1 -- 172.21.15.16:0/1000745789 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7efd38099bb0 con 0x5574e2602470 2024-03-28T22:20:07.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.895+0000 7efd4dd83740 1 -- 172.21.15.16:0/1000745789 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd3c047070 con 0x5574e2602470 2024-03-28T22:20:07.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.895+0000 7efd23fff640 1 -- 172.21.15.16:0/1000745789 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7efd38020050 con 0x5574e2602470 2024-03-28T22:20:07.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.895+0000 7efd4dd83740 10 client.17520 fetch_fsmap learned FSMap version 27 2024-03-28T22:20:07.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.895+0000 7efd4dd83740 1 -- 172.21.15.16:0/1000745789 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5574e26827d0 con 0x5574e2602470 2024-03-28T22:20:07.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.895+0000 7efd23fff640 1 -- 172.21.15.16:0/1000745789 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd3c047070 con 0x5574e2602470 2024-03-28T22:20:07.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.896+0000 7efd23fff640 1 -- 172.21.15.16:0/1000745789 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7efd38099620 con 0x5574e2602470 2024-03-28T22:20:07.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.896+0000 7efd4dd83740 10 client.17520 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:20:07.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.896+0000 7efd4dd83740 10 client.17520 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T22:20:07.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.896+0000 7efd4dd83740 20 client.17520 populate_metadata read hostname 'smithi016' 2024-03-28T22:20:07.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.896+0000 7efd4dd83740 1 --2- 172.21.15.16:0/1000745789 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5574e2682cb0 0x5574e26850a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:07.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.896+0000 7efd4dd83740 4 client.17520 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:20:07.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.896+0000 7efd4dd83740 1 -- 172.21.15.16:0/1000745789 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5574e256da40 con 0x5574e2682cb0 2024-03-28T22:20:07.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.896+0000 7efd43fff640 1 --2- 172.21.15.16:0/1000745789 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5574e2682cb0 0x5574e26850a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:07.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.896+0000 7efd43fff640 1 --2- 172.21.15.16:0/1000745789 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5574e2682cb0 0x5574e26850a0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ae44e7dfa295250d server_cookie=d850b227d28165f0 in_seq=0 out_seq=0 2024-03-28T22:20:07.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.896+0000 7efd23fff640 10 client.17520 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:20:07.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.896+0000 7efd23fff640 1 -- 172.21.15.16:0/1000745789 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5574e256da40 con 0x5574e2682cb0 2024-03-28T22:20:07.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.896+0000 7efd23fff640 10 client.17520 handle_command_reply: tid=0 2024-03-28T22:20:07.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.896+0000 7efd4dd83740 1 client.17520 shutdown 2024-03-28T22:20:07.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.896+0000 7efd4dd83740 1 -- 172.21.15.16:0/1000745789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574e2602470 msgr2=0x5574e2679f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:07.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.896+0000 7efd4dd83740 1 --2- 172.21.15.16:0/1000745789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574e2602470 0x5574e2679f00 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7efd38097130 tx=0x7efd38005bd0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.896+0000 7efd4dd83740 1 -- 172.21.15.16:0/1000745789 shutdown_connections 2024-03-28T22:20:07.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.896+0000 7efd4dd83740 1 --2- 172.21.15.16:0/1000745789 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x5574e2682cb0 0x5574e26850a0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.896+0000 7efd4dd83740 1 --2- 172.21.15.16:0/1000745789 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e260b970 0x5574e260fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.896+0000 7efd4dd83740 1 --2- 172.21.15.16:0/1000745789 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574e2602d90 0x5574e267a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.896+0000 7efd23fff640 0 client.17520 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:20:07.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.896+0000 7efd4dd83740 1 --2- 172.21.15.16:0/1000745789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5574e2602470 0x5574e2679f00 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.897+0000 7efd4dd83740 1 -- 172.21.15.16:0/1000745789 >> 172.21.15.16:0/1000745789 conn(0x5574e25fcba0 msgr2=0x5574e26088f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:07.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.897+0000 7efd4dd83740 1 -- 172.21.15.16:0/1000745789 shutdown_connections 2024-03-28T22:20:07.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.897+0000 7efd4dd83740 1 -- 172.21.15.16:0/1000745789 wait complete. 2024-03-28T22:20:07.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.897+0000 7efd4dd83740 20 client.17520 trim_cache size 0 max 16384 2024-03-28T22:20:07.913 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:20:07.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.897+0000 7efd4b0cc640 1 -- 172.21.15.16:0/2776415322 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7efd340a1ef0 msgr2=0x7efd340a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:07.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.897+0000 7efd4b0cc640 1 --2- 172.21.15.16:0/2776415322 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7efd340a1ef0 0x7efd340a43b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7efd3c02c630 tx=0x7efd3c02e040 comp rx=0 tx=0).stop 2024-03-28T22:20:07.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.897+0000 7efd4b0cc640 1 -- 172.21.15.16:0/2776415322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd441016e0 msgr2=0x7efd44197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:07.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.897+0000 7efd4b0cc640 1 --2- 172.21.15.16:0/2776415322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd441016e0 0x7efd44197f80 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7efd3800d950 tx=0x7efd3800de10 comp rx=0 tx=0).stop 2024-03-28T22:20:07.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.897+0000 7efd4b0cc640 1 -- 172.21.15.16:0/2776415322 shutdown_connections 2024-03-28T22:20:07.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.897+0000 7efd4b0cc640 1 --2- 172.21.15.16:0/2776415322 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7efd340a1ef0 0x7efd340a43b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.897+0000 7efd4b0cc640 1 --2- 172.21.15.16:0/2776415322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd441076f0 0x7efd4419c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.897+0000 7efd4b0cc640 1 --2- 172.21.15.16:0/2776415322 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efd441020a0 0x7efd441984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.897+0000 7efd4b0cc640 1 --2- 172.21.15.16:0/2776415322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd441016e0 0x7efd44197f80 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:07.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.897+0000 7efd4b0cc640 1 -- 172.21.15.16:0/2776415322 >> 172.21.15.16:0/2776415322 conn(0x7efd440fd1f0 msgr2=0x7efd440fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:07.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.897+0000 7efd4b0cc640 1 -- 172.21.15.16:0/2776415322 shutdown_connections 2024-03-28T22:20:07.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:07.897+0000 7efd4b0cc640 1 -- 172.21.15.16:0/2776415322 wait complete. 2024-03-28T22:20:07.916 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:20:08.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.018+0000 7ff3a9055640 1 -- 172.21.15.16:0/767268703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3a41009d0 msgr2=0x7ff3a4100db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:08.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.018+0000 7ff3a9055640 1 --2- 172.21.15.16:0/767268703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3a41009d0 0x7ff3a4100db0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7ff394009900 tx=0x7ff39402bd10 comp rx=0 tx=0).stop 2024-03-28T22:20:08.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.019+0000 7ff3a9055640 1 -- 172.21.15.16:0/767268703 shutdown_connections 2024-03-28T22:20:08.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.019+0000 7ff3a9055640 1 --2- 172.21.15.16:0/767268703 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3a4102b80 0x7ff3a410d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:08.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.019+0000 7ff3a9055640 1 --2- 172.21.15.16:0/767268703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3a41021e0 0x7ff3a4102640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:08.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.019+0000 7ff3a9055640 1 --2- 172.21.15.16:0/767268703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3a41009d0 0x7ff3a4100db0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:08.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.019+0000 7ff3a9055640 1 -- 172.21.15.16:0/767268703 >> 172.21.15.16:0/767268703 conn(0x7ff3a4077830 msgr2=0x7ff3a4077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:08.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.019+0000 7ff3a9055640 1 -- 172.21.15.16:0/767268703 shutdown_connections 2024-03-28T22:20:08.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.019+0000 7ff3a9055640 1 -- 172.21.15.16:0/767268703 wait complete. 2024-03-28T22:20:08.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.019+0000 7ff3a9055640 1 Processor -- start 2024-03-28T22:20:08.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.019+0000 7ff3a9055640 1 -- start start 2024-03-28T22:20:08.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.020+0000 7ff3a9055640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3a41009d0 0x7ff3a4198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:08.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.020+0000 7ff3a9055640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3a41021e0 0x7ff3a4198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:08.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.020+0000 7ff3a9055640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3a4102b80 0x7ff3a419c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:08.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.020+0000 7ff3a9055640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a4111c10 con 0x7ff3a41009d0 2024-03-28T22:20:08.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.020+0000 7ff3a9055640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a4111a90 con 0x7ff3a41021e0 2024-03-28T22:20:08.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.020+0000 7ff3a9055640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a4111d90 con 0x7ff3a4102b80 2024-03-28T22:20:08.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.020+0000 7ff3a2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3a41009d0 0x7ff3a4198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:08.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.020+0000 7ff3a2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3a41009d0 0x7ff3a4198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50204/0 (socket says 172.21.15.16:50204) 2024-03-28T22:20:08.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.020+0000 7ff3a2d9d640 1 -- 172.21.15.16:0/14706669 learned_addr learned my addr 172.21.15.16:0/14706669 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:08.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.020+0000 7ff3a259c640 1 --2- 172.21.15.16:0/14706669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3a41021e0 0x7ff3a4198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:08.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.020+0000 7ff3a359e640 1 --2- 172.21.15.16:0/14706669 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3a4102b80 0x7ff3a419c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:08.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.020+0000 7ff3a2d9d640 1 -- 172.21.15.16:0/14706669 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3a4102b80 msgr2=0x7ff3a419c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:08.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.020+0000 7ff3a2d9d640 1 --2- 172.21.15.16:0/14706669 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3a4102b80 0x7ff3a419c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:08.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.020+0000 7ff3a2d9d640 1 -- 172.21.15.16:0/14706669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3a41021e0 msgr2=0x7ff3a4198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:08.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.020+0000 7ff3a2d9d640 1 --2- 172.21.15.16:0/14706669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3a41021e0 0x7ff3a4198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:08.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.020+0000 7ff3a2d9d640 1 -- 172.21.15.16:0/14706669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3a419d040 con 0x7ff3a41009d0 2024-03-28T22:20:08.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.020+0000 7ff3a2d9d640 1 --2- 172.21.15.16:0/14706669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3a41009d0 0x7ff3a4198050 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7ff394006010 tx=0x7ff3940042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:08.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.020+0000 7ff38bfff640 1 -- 172.21.15.16:0/14706669 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff394042020 con 0x7ff3a41009d0 2024-03-28T22:20:08.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.020+0000 7ff3a9055640 1 -- 172.21.15.16:0/14706669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3a419d2d0 con 0x7ff3a41009d0 2024-03-28T22:20:08.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.020+0000 7ff38bfff640 1 -- 172.21.15.16:0/14706669 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3940054e0 con 0x7ff3a41009d0 2024-03-28T22:20:08.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.021+0000 7ff38bfff640 1 -- 172.21.15.16:0/14706669 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff394003ad0 con 0x7ff3a41009d0 2024-03-28T22:20:08.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.021+0000 7ff3a9055640 1 -- 172.21.15.16:0/14706669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3a41a4bc0 con 0x7ff3a41009d0 2024-03-28T22:20:08.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.023+0000 7ff38bfff640 1 -- 172.21.15.16:0/14706669 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7ff394045020 con 0x7ff3a41009d0 2024-03-28T22:20:08.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.023+0000 7ff3abc3a740 1 Processor -- start 2024-03-28T22:20:08.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.023+0000 7ff3abc3a740 1 -- start start 2024-03-28T22:20:08.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.023+0000 7ff3abc3a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c6622fc470 0x55c6622fc850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:08.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.023+0000 7ff3abc3a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6622fcd90 0x55c662305430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:08.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.023+0000 7ff3abc3a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c662305970 0x55c662307d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:08.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.023+0000 7ff3abc3a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c6622601b0 con 0x55c6622fc470 2024-03-28T22:20:08.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.023+0000 7ff3abc3a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55c662208c00 con 0x55c6622fcd90 2024-03-28T22:20:08.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.023+0000 7ff3abc3a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55c66225c250 con 0x55c662305970 2024-03-28T22:20:08.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.023+0000 7ff38bfff640 1 --2- 172.21.15.16:0/14706669 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff38c0a1ef0 0x7ff38c0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:08.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.023+0000 7ff3a359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c6622fc470 0x55c6622fc850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:08.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.023+0000 7ff3a2d9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6622fcd90 0x55c662305430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:08.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.023+0000 7ff3a359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c6622fc470 0x55c6622fc850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50216/0 (socket says 172.21.15.16:50216) 2024-03-28T22:20:08.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.023+0000 7ff3a359e640 1 -- 172.21.15.16:0/637244492 learned_addr learned my addr 172.21.15.16:0/637244492 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:08.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.023+0000 7ff3a359e640 1 --2- 172.21.15.16:0/14706669 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff38c0a1ef0 0x7ff38c0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:08.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.023+0000 7ff38bfff640 1 -- 172.21.15.16:0/14706669 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7ff3940efd60 con 0x7ff3a41009d0 2024-03-28T22:20:08.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.023+0000 7ff3a259c640 1 --2- 172.21.15.16:0/637244492 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c662305970 0x55c662307d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:08.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.024+0000 7ff3a359e640 1 -- 172.21.15.16:0/637244492 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c662305970 msgr2=0x55c662307d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:08.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.024+0000 7ff3a359e640 1 --2- 172.21.15.16:0/637244492 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c662305970 0x55c662307d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:08.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.024+0000 7ff3a359e640 1 -- 172.21.15.16:0/637244492 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6622fcd90 msgr2=0x55c662305430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:08.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.024+0000 7ff3a359e640 1 --2- 172.21.15.16:0/637244492 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6622fcd90 0x55c662305430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:08.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.024+0000 7ff3a359e640 1 -- 172.21.15.16:0/637244492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6622283d0 con 0x55c6622fc470 2024-03-28T22:20:08.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.024+0000 7ff3a2d9d640 1 --2- 172.21.15.16:0/637244492 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6622fcd90 0x55c662305430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:20:08.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.024+0000 7ff3a359e640 1 --2- 172.21.15.16:0/637244492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c6622fc470 0x55c6622fc850 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7ff39c013de0 tx=0x7ff39c0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=458f12ffd2158a4d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:08.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.024+0000 7ff3a359e640 1 --2- 172.21.15.16:0/14706669 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff38c0a1ef0 0x7ff38c0a43b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff39c005c40 tx=0x7ff39c01b000 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:08.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.024+0000 7ff3897fa640 1 -- 172.21.15.16:0/637244492 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff39c00bde0 con 0x55c6622fc470 2024-03-28T22:20:08.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.024+0000 7ff3897fa640 1 -- 172.21.15.16:0/637244492 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff39c003e50 con 0x55c6622fc470 2024-03-28T22:20:08.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.024+0000 7ff3897fa640 1 -- 172.21.15.16:0/637244492 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff39c0047d0 con 0x55c6622fc470 2024-03-28T22:20:08.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.025+0000 7ff3abc3a740 1 -- 172.21.15.16:0/637244492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c6622fc470 msgr2=0x55c6622fc850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:08.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.025+0000 7ff3abc3a740 1 --2- 172.21.15.16:0/637244492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c6622fc470 0x55c6622fc850 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7ff39c013de0 tx=0x7ff39c0107a0 comp rx=0 tx=0).stop 2024-03-28T22:20:08.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.025+0000 7ff3abc3a740 1 -- 172.21.15.16:0/637244492 shutdown_connections 2024-03-28T22:20:08.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.025+0000 7ff3abc3a740 1 --2- 172.21.15.16:0/637244492 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c662305970 0x55c662307d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:08.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.025+0000 7ff3abc3a740 1 --2- 172.21.15.16:0/637244492 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6622fcd90 0x55c662305430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:08.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.025+0000 7ff3abc3a740 1 --2- 172.21.15.16:0/637244492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c6622fc470 0x55c6622fc850 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:08.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.025+0000 7ff3abc3a740 1 -- 172.21.15.16:0/637244492 >> 172.21.15.16:0/637244492 conn(0x55c6622f6ba0 msgr2=0x55c6622f6f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:08.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.025+0000 7ff3abc3a740 1 -- 172.21.15.16:0/637244492 shutdown_connections 2024-03-28T22:20:08.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.025+0000 7ff3abc3a740 1 -- 172.21.15.16:0/637244492 wait complete. 2024-03-28T22:20:08.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.025+0000 7ff3abc3a740 1 -- start start 2024-03-28T22:20:08.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.025+0000 7ff3abc3a740 1 Processor -- start 2024-03-28T22:20:08.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.025+0000 7ff3abc3a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6622fc470 0x55c6623029d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:08.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.025+0000 7ff3abc3a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c6622fcd90 0x55c662302f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:08.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.025+0000 7ff3abc3a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c662305970 0x55c662303450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:08.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.025+0000 7ff3abc3a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c662308700 con 0x55c662305970 2024-03-28T22:20:08.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.025+0000 7ff3abc3a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55c6621fe470 con 0x55c6622fc470 2024-03-28T22:20:08.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.025+0000 7ff3abc3a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55c662308880 con 0x55c6622fcd90 2024-03-28T22:20:08.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.025+0000 7ff3abc3a740 10 client.0 _ll_register_callbacks cb 0x55c66221bcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:20:08.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.025+0000 7ff3a359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c662305970 0x55c662303450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:08.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.025+0000 7ff3a2d9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6622fc470 0x55c6623029d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:08.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.025+0000 7ff3a359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c662305970 0x55c662303450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50232/0 (socket says 172.21.15.16:50232) 2024-03-28T22:20:08.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.025+0000 7ff3a359e640 1 -- 172.21.15.16:0/2445161505 learned_addr learned my addr 172.21.15.16:0/2445161505 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:08.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.025+0000 7ff3a259c640 1 --2- 172.21.15.16:0/2445161505 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c6622fcd90 0x55c662302f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:08.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.026+0000 7ff3a2d9d640 1 -- 172.21.15.16:0/2445161505 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c6622fcd90 msgr2=0x55c662302f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:08.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.026+0000 7ff3a2d9d640 1 --2- 172.21.15.16:0/2445161505 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c6622fcd90 0x55c662302f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:08.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.026+0000 7ff3a2d9d640 1 -- 172.21.15.16:0/2445161505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c662305970 msgr2=0x55c662303450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:08.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.026+0000 7ff3a2d9d640 1 --2- 172.21.15.16:0/2445161505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c662305970 0x55c662303450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:08.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.026+0000 7ff3a2d9d640 1 -- 172.21.15.16:0/2445161505 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c661f743f0 con 0x55c6622fc470 2024-03-28T22:20:08.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.026+0000 7ff3a359e640 1 --2- 172.21.15.16:0/2445161505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c662305970 0x55c662303450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:20:08.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.026+0000 7ff3a2d9d640 1 --2- 172.21.15.16:0/2445161505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6622fc470 0x55c6623029d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff3940ed5c0 tx=0x7ff39408e820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=919642747973340e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:08.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.026+0000 7ff386ff5640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-03-28T22:20:08.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.026+0000 7ff386ff5640 1 -- 172.21.15.16:0/2445161505 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff394005690 con 0x55c6622fc470 2024-03-28T22:20:08.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.026+0000 7ff3abc3a740 1 -- 172.21.15.16:0/2445161505 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff39c003e50 con 0x55c6622fc470 2024-03-28T22:20:08.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.026+0000 7ff386ff5640 1 -- 172.21.15.16:0/2445161505 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7ff39402cbd0 con 0x55c6622fc470 2024-03-28T22:20:08.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.026+0000 7ff386ff5640 1 -- 172.21.15.16:0/2445161505 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff3940bcf60 con 0x55c6622fc470 2024-03-28T22:20:08.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.026+0000 7ff3abc3a740 10 client.27434 fetch_fsmap learned FSMap version 27 2024-03-28T22:20:08.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.026+0000 7ff3abc3a740 1 -- 172.21.15.16:0/2445161505 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c66203e7d0 con 0x55c6622fc470 2024-03-28T22:20:08.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.026+0000 7ff386ff5640 1 -- 172.21.15.16:0/2445161505 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff39c003e50 con 0x55c6622fc470 2024-03-28T22:20:08.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.027+0000 7ff386ff5640 1 -- 172.21.15.16:0/2445161505 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7ff394003c70 con 0x55c6622fc470 2024-03-28T22:20:08.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.027+0000 7ff3abc3a740 10 client.27434 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:20:08.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.027+0000 7ff3abc3a740 10 client.27434 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:20:08.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.027+0000 7ff3abc3a740 20 client.27434 populate_metadata read hostname 'smithi016' 2024-03-28T22:20:08.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.027+0000 7ff3abc3a740 1 --2- 172.21.15.16:0/2445161505 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55c6623158c0 0x55c662317d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:08.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.027+0000 7ff3abc3a740 4 client.27434 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:20:08.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.027+0000 7ff3abc3a740 1 -- 172.21.15.16:0/2445161505 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c66226d6f0 con 0x55c6623158c0 2024-03-28T22:20:08.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.027+0000 7ff3a259c640 1 --2- 172.21.15.16:0/2445161505 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55c6623158c0 0x55c662317d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:08.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.027+0000 7ff3a259c640 1 --2- 172.21.15.16:0/2445161505 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55c6623158c0 0x55c662317d30 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7cc18f231c3452fa server_cookie=7e3481f459af77e0 in_seq=0 out_seq=0 2024-03-28T22:20:08.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.027+0000 7ff386ff5640 10 client.27434 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:20:08.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.028+0000 7ff386ff5640 1 -- 172.21.15.16:0/2445161505 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c66226d6f0 con 0x55c6623158c0 2024-03-28T22:20:08.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.028+0000 7ff386ff5640 10 client.27434 handle_command_reply: tid=0 2024-03-28T22:20:08.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.028+0000 7ff3abc3a740 1 client.27434 shutdown 2024-03-28T22:20:08.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.028+0000 7ff3abc3a740 1 -- 172.21.15.16:0/2445161505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6622fc470 msgr2=0x55c6623029d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:08.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.028+0000 7ff3abc3a740 1 --2- 172.21.15.16:0/2445161505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6622fc470 0x55c6623029d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff3940ed5c0 tx=0x7ff39408e820 comp rx=0 tx=0).stop 2024-03-28T22:20:08.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.028+0000 7ff3abc3a740 1 -- 172.21.15.16:0/2445161505 shutdown_connections 2024-03-28T22:20:08.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.028+0000 7ff3abc3a740 1 --2- 172.21.15.16:0/2445161505 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55c6623158c0 0x55c662317d30 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:08.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.028+0000 7ff3abc3a740 1 --2- 172.21.15.16:0/2445161505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c662305970 0x55c662303450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:08.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.028+0000 7ff3abc3a740 1 --2- 172.21.15.16:0/2445161505 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c6622fcd90 0x55c662302f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:08.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.028+0000 7ff3abc3a740 1 --2- 172.21.15.16:0/2445161505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6622fc470 0x55c6623029d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:08.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.028+0000 7ff386ff5640 0 client.27434 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:20:08.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.028+0000 7ff3abc3a740 1 -- 172.21.15.16:0/2445161505 >> 172.21.15.16:0/2445161505 conn(0x55c6622f6ba0 msgr2=0x55c6622f88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:08.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.028+0000 7ff3abc3a740 1 -- 172.21.15.16:0/2445161505 shutdown_connections 2024-03-28T22:20:08.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.028+0000 7ff3abc3a740 1 -- 172.21.15.16:0/2445161505 wait complete. 2024-03-28T22:20:08.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.028+0000 7ff3abc3a740 20 client.27434 trim_cache size 0 max 16384 2024-03-28T22:20:08.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.041+0000 7ff3abc3a740 1 Processor -- start 2024-03-28T22:20:08.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.041+0000 7ff3abc3a740 1 -- start start 2024-03-28T22:20:08.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.041+0000 7ff3abc3a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6622fc470 0x55c6622f5f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:08.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.041+0000 7ff3abc3a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c6622fcd90 0x55c6623022d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:08.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.041+0000 7ff3abc3a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c662305970 0x55c6623114a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:08.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.041+0000 7ff3abc3a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a41afbd0 con 0x55c662305970 2024-03-28T22:20:08.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.041+0000 7ff3abc3a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a41ad4a0 con 0x55c6622fc470 2024-03-28T22:20:08.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.041+0000 7ff3abc3a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a41ab310 con 0x55c6622fcd90 2024-03-28T22:20:08.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.042+0000 7ff3a359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c662305970 0x55c6623114a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:08.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.042+0000 7ff3a359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c662305970 0x55c6623114a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50242/0 (socket says 172.21.15.16:50242) 2024-03-28T22:20:08.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.042+0000 7ff3a359e640 1 -- 172.21.15.16:0/1822845484 learned_addr learned my addr 172.21.15.16:0/1822845484 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:08.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.042+0000 7ff3a2d9d640 1 --2- 172.21.15.16:0/1822845484 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6622fc470 0x55c6622f5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:08.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.042+0000 7ff3a359e640 1 -- 172.21.15.16:0/1822845484 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c6622fcd90 msgr2=0x55c6623022d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:08.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.042+0000 7ff3a259c640 1 --2- 172.21.15.16:0/1822845484 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c6622fcd90 0x55c6623022d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:08.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.042+0000 7ff3a359e640 1 --2- 172.21.15.16:0/1822845484 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c6622fcd90 0x55c6623022d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:08.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.042+0000 7ff3a359e640 1 -- 172.21.15.16:0/1822845484 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6622fc470 msgr2=0x55c6622f5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:08.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.042+0000 7ff3a359e640 1 --2- 172.21.15.16:0/1822845484 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6622fc470 0x55c6622f5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:08.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.042+0000 7ff3a359e640 1 -- 172.21.15.16:0/1822845484 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6623119e0 con 0x55c662305970 2024-03-28T22:20:08.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.042+0000 7ff3a2d9d640 1 --2- 172.21.15.16:0/1822845484 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6622fc470 0x55c6622f5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:20:08.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.042+0000 7ff3a259c640 1 --2- 172.21.15.16:0/1822845484 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c6622fcd90 0x55c6623022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:20:08.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.042+0000 7ff3a359e640 1 --2- 172.21.15.16:0/1822845484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c662305970 0x55c6623114a0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7ff39c00ab90 tx=0x7ff39c015040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68a39ae1f78fea8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:08.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.042+0000 7ff388ff9640 1 -- 172.21.15.16:0/1822845484 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff39c004140 con 0x55c662305970 2024-03-28T22:20:08.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.042+0000 7ff388ff9640 1 -- 172.21.15.16:0/1822845484 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff39c00ac10 con 0x55c662305970 2024-03-28T22:20:08.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.042+0000 7ff388ff9640 1 -- 172.21.15.16:0/1822845484 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff39c003bd0 con 0x55c662305970 2024-03-28T22:20:08.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.043+0000 7ff3abc3a740 1 -- 172.21.15.16:0/1822845484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c662305970 msgr2=0x55c6623114a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:08.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.043+0000 7ff3abc3a740 1 --2- 172.21.15.16:0/1822845484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c662305970 0x55c6623114a0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7ff39c00ab90 tx=0x7ff39c015040 comp rx=0 tx=0).stop 2024-03-28T22:20:08.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.043+0000 7ff3abc3a740 1 -- 172.21.15.16:0/1822845484 shutdown_connections 2024-03-28T22:20:08.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.043+0000 7ff3abc3a740 1 --2- 172.21.15.16:0/1822845484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c662305970 0x55c6623114a0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:08.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.043+0000 7ff3abc3a740 1 --2- 172.21.15.16:0/1822845484 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c6622fcd90 0x55c6623022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:08.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.043+0000 7ff3abc3a740 1 --2- 172.21.15.16:0/1822845484 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6622fc470 0x55c6622f5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:08.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.043+0000 7ff3abc3a740 1 -- 172.21.15.16:0/1822845484 >> 172.21.15.16:0/1822845484 conn(0x55c6622f6ba0 msgr2=0x55c662303220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:08.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.043+0000 7ff3abc3a740 1 -- 172.21.15.16:0/1822845484 shutdown_connections 2024-03-28T22:20:08.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.043+0000 7ff3abc3a740 1 -- 172.21.15.16:0/1822845484 wait complete. 2024-03-28T22:20:08.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.043+0000 7ff3abc3a740 1 -- start start 2024-03-28T22:20:08.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.043+0000 7ff3abc3a740 1 Processor -- start 2024-03-28T22:20:08.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.043+0000 7ff3abc3a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6622fc470 0x55c6622ff6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:08.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.043+0000 7ff3abc3a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c6622fcd90 0x55c6622ffc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:08.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.043+0000 7ff3abc3a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c662305970 0x55c662300170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:08.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.043+0000 7ff3abc3a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a41ab960 con 0x55c6622fcd90 2024-03-28T22:20:08.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.043+0000 7ff3abc3a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a41acdd0 con 0x55c6622fc470 2024-03-28T22:20:08.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.043+0000 7ff3abc3a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a41ac030 con 0x55c662305970 2024-03-28T22:20:08.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.043+0000 7ff3abc3a740 10 client.0 _ll_register_callbacks cb 0x7ff394009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:20:08.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.044+0000 7ff3a259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c6622fcd90 0x55c6622ffc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:08.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.044+0000 7ff3a259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c6622fcd90 0x55c6622ffc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50248/0 (socket says 172.21.15.16:50248) 2024-03-28T22:20:08.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.044+0000 7ff3a2d9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6622fc470 0x55c6622ff6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:08.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.044+0000 7ff3a259c640 1 -- 172.21.15.16:0/1537489528 learned_addr learned my addr 172.21.15.16:0/1537489528 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:08.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.044+0000 7ff3a359e640 1 --2- 172.21.15.16:0/1537489528 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c662305970 0x55c662300170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:08.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.044+0000 7ff3a259c640 1 -- 172.21.15.16:0/1537489528 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c662305970 msgr2=0x55c662300170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:08.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.044+0000 7ff3a259c640 1 --2- 172.21.15.16:0/1537489528 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c662305970 0x55c662300170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:08.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.044+0000 7ff3a259c640 1 -- 172.21.15.16:0/1537489528 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6622fc470 msgr2=0x55c6622ff6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:08.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.044+0000 7ff3a259c640 1 --2- 172.21.15.16:0/1537489528 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6622fc470 0x55c6622ff6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:08.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.044+0000 7ff3a259c640 1 -- 172.21.15.16:0/1537489528 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c66237ad90 con 0x55c6622fcd90 2024-03-28T22:20:08.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.044+0000 7ff3a359e640 1 --2- 172.21.15.16:0/1537489528 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c662305970 0x55c662300170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:20:08.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.044+0000 7ff3a259c640 1 --2- 172.21.15.16:0/1537489528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c6622fcd90 0x55c6622ffc30 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x55c6623185a0 tx=0x7ff398010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bc3a5e5e4746dae server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:08.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.044+0000 7ff386ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:20:08.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.044+0000 7ff386ff5640 1 -- 172.21.15.16:0/1537489528 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff398023020 con 0x55c6622fcd90 2024-03-28T22:20:08.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.044+0000 7ff386ff5640 1 -- 172.21.15.16:0/1537489528 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7ff398004050 con 0x55c6622fcd90 2024-03-28T22:20:08.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.044+0000 7ff3abc3a740 1 -- 172.21.15.16:0/1537489528 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff39c00ac10 con 0x55c6622fcd90 2024-03-28T22:20:08.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.044+0000 7ff386ff5640 1 -- 172.21.15.16:0/1537489528 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff39802e050 con 0x55c6622fcd90 2024-03-28T22:20:08.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.044+0000 7ff3abc3a740 10 client.17538 fetch_fsmap learned FSMap version 27 2024-03-28T22:20:08.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.044+0000 7ff3abc3a740 1 -- 172.21.15.16:0/1537489528 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c66237c010 con 0x55c6622fcd90 2024-03-28T22:20:08.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.044+0000 7ff386ff5640 1 -- 172.21.15.16:0/1537489528 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff39c00ac10 con 0x55c6622fcd90 2024-03-28T22:20:08.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.045+0000 7ff386ff5640 1 -- 172.21.15.16:0/1537489528 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7ff39801e390 con 0x55c6622fcd90 2024-03-28T22:20:08.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.045+0000 7ff3abc3a740 10 client.17538 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:20:08.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.045+0000 7ff3abc3a740 10 client.17538 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:20:08.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.045+0000 7ff3abc3a740 20 client.17538 populate_metadata read hostname 'smithi016' 2024-03-28T22:20:08.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.045+0000 7ff3abc3a740 1 --2- 172.21.15.16:0/1537489528 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55c66237c810 0x55c66237ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:08.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.045+0000 7ff3abc3a740 4 client.17538 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:20:08.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.045+0000 7ff3abc3a740 1 -- 172.21.15.16:0/1537489528 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c662267a40 con 0x55c66237c810 2024-03-28T22:20:08.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.045+0000 7ff3a2d9d640 1 --2- 172.21.15.16:0/1537489528 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55c66237c810 0x55c66237ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:08.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.045+0000 7ff386ff5640 10 client.17538 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:20:08.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.045+0000 7ff3a2d9d640 1 --2- 172.21.15.16:0/1537489528 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55c66237c810 0x55c66237ec30 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed3f014bd1e3449 server_cookie=a8c19952596e01fe in_seq=0 out_seq=0 2024-03-28T22:20:08.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.045+0000 7ff386ff5640 1 -- 172.21.15.16:0/1537489528 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c662267a40 con 0x55c66237c810 2024-03-28T22:20:08.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.045+0000 7ff386ff5640 10 client.17538 handle_command_reply: tid=0 2024-03-28T22:20:08.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.045+0000 7ff3abc3a740 1 client.17538 shutdown 2024-03-28T22:20:08.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.045+0000 7ff3abc3a740 1 -- 172.21.15.16:0/1537489528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c6622fcd90 msgr2=0x55c6622ffc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:08.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.045+0000 7ff3abc3a740 1 --2- 172.21.15.16:0/1537489528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c6622fcd90 0x55c6622ffc30 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x55c6623185a0 tx=0x7ff398010800 comp rx=0 tx=0).stop 2024-03-28T22:20:08.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.045+0000 7ff3abc3a740 1 -- 172.21.15.16:0/1537489528 shutdown_connections 2024-03-28T22:20:08.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.045+0000 7ff3abc3a740 1 --2- 172.21.15.16:0/1537489528 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55c66237c810 0x55c66237ec30 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:08.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.045+0000 7ff3abc3a740 1 --2- 172.21.15.16:0/1537489528 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c662305970 0x55c662300170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:08.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.045+0000 7ff386ff5640 0 client.17538 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:20:08.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.045+0000 7ff3abc3a740 1 --2- 172.21.15.16:0/1537489528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c6622fcd90 0x55c6622ffc30 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:08.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.045+0000 7ff3abc3a740 1 --2- 172.21.15.16:0/1537489528 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6622fc470 0x55c6622ff6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:08.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.045+0000 7ff3abc3a740 1 -- 172.21.15.16:0/1537489528 >> 172.21.15.16:0/1537489528 conn(0x55c6622f6ba0 msgr2=0x55c6622f7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:08.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.046+0000 7ff3abc3a740 1 -- 172.21.15.16:0/1537489528 shutdown_connections 2024-03-28T22:20:08.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.046+0000 7ff3abc3a740 1 -- 172.21.15.16:0/1537489528 wait complete. 2024-03-28T22:20:08.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.046+0000 7ff3abc3a740 20 client.17538 trim_cache size 0 max 16384 2024-03-28T22:20:08.064 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:20:08.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.046+0000 7ff3a9055640 1 -- 172.21.15.16:0/14706669 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff38c0a1ef0 msgr2=0x7ff38c0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:08.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.046+0000 7ff3a9055640 1 --2- 172.21.15.16:0/14706669 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff38c0a1ef0 0x7ff38c0a43b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff39c005c40 tx=0x7ff39c01b000 comp rx=0 tx=0).stop 2024-03-28T22:20:08.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.046+0000 7ff3a9055640 1 -- 172.21.15.16:0/14706669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3a41009d0 msgr2=0x7ff3a4198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:08.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.046+0000 7ff3a9055640 1 --2- 172.21.15.16:0/14706669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3a41009d0 0x7ff3a4198050 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7ff394006010 tx=0x7ff3940042f0 comp rx=0 tx=0).stop 2024-03-28T22:20:08.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.046+0000 7ff3a9055640 1 -- 172.21.15.16:0/14706669 shutdown_connections 2024-03-28T22:20:08.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.046+0000 7ff3a9055640 1 --2- 172.21.15.16:0/14706669 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff38c0a1ef0 0x7ff38c0a43b0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:08.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.046+0000 7ff3a9055640 1 --2- 172.21.15.16:0/14706669 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3a4102b80 0x7ff3a419c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:08.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.046+0000 7ff3a9055640 1 --2- 172.21.15.16:0/14706669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3a41021e0 0x7ff3a4198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:08.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.046+0000 7ff3a9055640 1 --2- 172.21.15.16:0/14706669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3a41009d0 0x7ff3a4198050 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:08.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.046+0000 7ff3a9055640 1 -- 172.21.15.16:0/14706669 >> 172.21.15.16:0/14706669 conn(0x7ff3a4077830 msgr2=0x7ff3a41043c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:08.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.046+0000 7ff3a9055640 1 -- 172.21.15.16:0/14706669 shutdown_connections 2024-03-28T22:20:08.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:08.046+0000 7ff3a9055640 1 -- 172.21.15.16:0/14706669 wait complete. 2024-03-28T22:20:08.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:08 smithi053 ceph-mon[35979]: pgmap v295: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-28T22:20:08.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:08 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3490739160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:20:08.680 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:08 smithi063 ceph-mon[36249]: pgmap v295: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-28T22:20:08.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:08 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/3490739160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:20:08.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:08 smithi016 ceph-mon[32374]: pgmap v295: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-28T22:20:08.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:08 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3490739160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:20:09.068 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:20:09.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:20:09.069 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:20:09.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.187+0000 7f97cb4fb640 1 -- 172.21.15.16:0/2727055692 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f97c4107700 msgr2=0x7f97c4107ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:09.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.187+0000 7f97cb4fb640 1 --2- 172.21.15.16:0/2727055692 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f97c4107700 0x7f97c4107ae0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f97bc009860 tx=0x7f97bc02bb70 comp rx=0 tx=0).stop 2024-03-28T22:20:09.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.187+0000 7f97cb4fb640 1 -- 172.21.15.16:0/2727055692 shutdown_connections 2024-03-28T22:20:09.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.187+0000 7f97cb4fb640 1 --2- 172.21.15.16:0/2727055692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c41020b0 0x7f97c410e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.187+0000 7f97cb4fb640 1 --2- 172.21.15.16:0/2727055692 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f97c41016f0 0x7f97c4101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.187+0000 7f97cb4fb640 1 --2- 172.21.15.16:0/2727055692 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f97c4107700 0x7f97c4107ae0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.187+0000 7f97cb4fb640 1 -- 172.21.15.16:0/2727055692 >> 172.21.15.16:0/2727055692 conn(0x7f97c40fd1e0 msgr2=0x7f97c40ff600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:09.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.187+0000 7f97cb4fb640 1 -- 172.21.15.16:0/2727055692 shutdown_connections 2024-03-28T22:20:09.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.187+0000 7f97cb4fb640 1 -- 172.21.15.16:0/2727055692 wait complete. 2024-03-28T22:20:09.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.187+0000 7f97cb4fb640 1 Processor -- start 2024-03-28T22:20:09.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.188+0000 7f97cb4fb640 1 -- start start 2024-03-28T22:20:09.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.188+0000 7f97cb4fb640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f97c41016f0 0x7f97c4198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:09.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.188+0000 7f97cb4fb640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f97c41020b0 0x7f97c41985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:09.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.188+0000 7f97cb4fb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c4107700 0x7f97c419c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:09.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.188+0000 7f97cb4fb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f97c4111ad0 con 0x7f97c4107700 2024-03-28T22:20:09.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.188+0000 7f97cb4fb640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f97c4111950 con 0x7f97c41020b0 2024-03-28T22:20:09.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.188+0000 7f97cb4fb640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f97c4111c50 con 0x7f97c41016f0 2024-03-28T22:20:09.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.188+0000 7f97c9a98640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c4107700 0x7f97c419c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:09.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.188+0000 7f97c8a96640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f97c41020b0 0x7f97c41985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:09.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.188+0000 7f97c8a96640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f97c41020b0 0x7f97c41985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:38036/0 (socket says 172.21.15.16:38036) 2024-03-28T22:20:09.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.188+0000 7f97c9a98640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c4107700 0x7f97c419c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50258/0 (socket says 172.21.15.16:50258) 2024-03-28T22:20:09.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.188+0000 7f97c8a96640 1 -- 172.21.15.16:0/2051327326 learned_addr learned my addr 172.21.15.16:0/2051327326 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:09.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.188+0000 7f97c9297640 1 --2- 172.21.15.16:0/2051327326 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f97c41016f0 0x7f97c4198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:09.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.188+0000 7f97c8a96640 1 -- 172.21.15.16:0/2051327326 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f97c41016f0 msgr2=0x7f97c4198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:09.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.188+0000 7f97c8a96640 1 --2- 172.21.15.16:0/2051327326 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f97c41016f0 0x7f97c4198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.188+0000 7f97c8a96640 1 -- 172.21.15.16:0/2051327326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c4107700 msgr2=0x7f97c419c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:09.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.188+0000 7f97c8a96640 1 --2- 172.21.15.16:0/2051327326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c4107700 0x7f97c419c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.188+0000 7f97c8a96640 1 -- 172.21.15.16:0/2051327326 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97c419d010 con 0x7f97c41020b0 2024-03-28T22:20:09.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.188+0000 7f97c9a98640 1 --2- 172.21.15.16:0/2051327326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c4107700 0x7f97c419c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:20:09.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.188+0000 7f97c8a96640 1 --2- 172.21.15.16:0/2051327326 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f97c41020b0 0x7f97c41985b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f97b800d950 tx=0x7f97b800de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:09.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.189+0000 7f97c27fc640 1 -- 172.21.15.16:0/2051327326 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b8017070 con 0x7f97c41020b0 2024-03-28T22:20:09.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.189+0000 7f97c27fc640 1 -- 172.21.15.16:0/2051327326 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97b8009c30 con 0x7f97c41020b0 2024-03-28T22:20:09.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.189+0000 7f97cb4fb640 1 -- 172.21.15.16:0/2051327326 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97c419d300 con 0x7f97c41020b0 2024-03-28T22:20:09.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.189+0000 7f97c27fc640 1 -- 172.21.15.16:0/2051327326 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b800f6b0 con 0x7f97c41020b0 2024-03-28T22:20:09.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.189+0000 7f97cb4fb640 1 -- 172.21.15.16:0/2051327326 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97c41a4b90 con 0x7f97c41020b0 2024-03-28T22:20:09.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.191+0000 7f97c27fc640 1 -- 172.21.15.16:0/2051327326 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f97b80109d0 con 0x7f97c41020b0 2024-03-28T22:20:09.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.191+0000 7f97ce1bc740 1 Processor -- start 2024-03-28T22:20:09.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.191+0000 7f97ce1bc740 1 -- start start 2024-03-28T22:20:09.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.192+0000 7f97c27fc640 1 --2- 172.21.15.16:0/2051327326 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f97ac0a1ef0 0x7f97ac0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:09.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.192+0000 7f97ce1bc740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aa7f718720 0x55aa7f718b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:09.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.192+0000 7f97ce1bc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa7f719040 0x55aa7f725dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:09.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.192+0000 7f97ce1bc740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55aa7f726300 0x55aa7f7286f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:09.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.192+0000 7f97c27fc640 1 -- 172.21.15.16:0/2051327326 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7f97b80cde40 con 0x7f97c41020b0 2024-03-28T22:20:09.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.192+0000 7f97c9a98640 1 --2- 172.21.15.16:0/2051327326 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f97ac0a1ef0 0x7f97ac0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:09.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.192+0000 7f97ce1bc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55aa7f6190b0 con 0x55aa7f719040 2024-03-28T22:20:09.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.192+0000 7f97ce1bc740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55aa7f67bea0 con 0x55aa7f726300 2024-03-28T22:20:09.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.192+0000 7f97ce1bc740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55aa7f624c00 con 0x55aa7f718720 2024-03-28T22:20:09.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.192+0000 7f97c8a96640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa7f719040 0x55aa7f725dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:09.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.192+0000 7f97c9a98640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55aa7f726300 0x55aa7f7286f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:09.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.192+0000 7f97c8a96640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa7f719040 0x55aa7f725dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50272/0 (socket says 172.21.15.16:50272) 2024-03-28T22:20:09.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.192+0000 7f97c8a96640 1 -- 172.21.15.16:0/2690523504 learned_addr learned my addr 172.21.15.16:0/2690523504 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:09.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.192+0000 7f97c9297640 1 --2- 172.21.15.16:0/2690523504 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aa7f718720 0x55aa7f718b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:09.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.192+0000 7f97c8a96640 1 -- 172.21.15.16:0/2690523504 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aa7f718720 msgr2=0x55aa7f718b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:09.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.192+0000 7f97c8a96640 1 --2- 172.21.15.16:0/2690523504 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aa7f718720 0x55aa7f718b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.192+0000 7f97c8a96640 1 -- 172.21.15.16:0/2690523504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55aa7f726300 msgr2=0x55aa7f7286f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:09.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.192+0000 7f97c8a96640 1 --2- 172.21.15.16:0/2690523504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55aa7f726300 0x55aa7f7286f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.192+0000 7f97c8a96640 1 -- 172.21.15.16:0/2690523504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa7f4587e0 con 0x55aa7f719040 2024-03-28T22:20:09.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.192+0000 7f97c9a98640 1 --2- 172.21.15.16:0/2051327326 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f97ac0a1ef0 0x7f97ac0a43b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f97b4005fd0 tx=0x7f97b400eeb0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:09.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.192+0000 7f97c8a96640 1 --2- 172.21.15.16:0/2690523504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa7f719040 0x55aa7f725dc0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f97b80a2a40 tx=0x7f97b8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7e5e338cbbca34d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:09.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.193+0000 7f97ab7fe640 1 -- 172.21.15.16:0/2690523504 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b8020030 con 0x55aa7f719040 2024-03-28T22:20:09.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.193+0000 7f97ab7fe640 1 -- 172.21.15.16:0/2690523504 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97b809e920 con 0x55aa7f719040 2024-03-28T22:20:09.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.193+0000 7f97ab7fe640 1 -- 172.21.15.16:0/2690523504 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b80cb910 con 0x55aa7f719040 2024-03-28T22:20:09.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.193+0000 7f97ce1bc740 1 -- 172.21.15.16:0/2690523504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa7f719040 msgr2=0x55aa7f725dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:09.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.193+0000 7f97ce1bc740 1 --2- 172.21.15.16:0/2690523504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa7f719040 0x55aa7f725dc0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f97b80a2a40 tx=0x7f97b8005f90 comp rx=0 tx=0).stop 2024-03-28T22:20:09.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.193+0000 7f97ce1bc740 1 -- 172.21.15.16:0/2690523504 shutdown_connections 2024-03-28T22:20:09.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.193+0000 7f97ce1bc740 1 --2- 172.21.15.16:0/2690523504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55aa7f726300 0x55aa7f7286f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.193+0000 7f97ce1bc740 1 --2- 172.21.15.16:0/2690523504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa7f719040 0x55aa7f725dc0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.193+0000 7f97ce1bc740 1 --2- 172.21.15.16:0/2690523504 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aa7f718720 0x55aa7f718b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.193+0000 7f97ce1bc740 1 -- 172.21.15.16:0/2690523504 >> 172.21.15.16:0/2690523504 conn(0x55aa7f713120 msgr2=0x55aa7f713500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:09.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.193+0000 7f97ce1bc740 1 -- 172.21.15.16:0/2690523504 shutdown_connections 2024-03-28T22:20:09.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.193+0000 7f97ce1bc740 1 -- 172.21.15.16:0/2690523504 wait complete. 2024-03-28T22:20:09.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.194+0000 7f97ce1bc740 1 -- start start 2024-03-28T22:20:09.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.194+0000 7f97ce1bc740 1 Processor -- start 2024-03-28T22:20:09.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.194+0000 7f97ce1bc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa7f718720 0x55aa7f723380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:09.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.194+0000 7f97ce1bc740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aa7f719040 0x55aa7f7238c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:09.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.194+0000 7f97ce1bc740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55aa7f726300 0x55aa7f723e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:09.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.194+0000 7f97ce1bc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55aa7f61a3d0 con 0x55aa7f718720 2024-03-28T22:20:09.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.194+0000 7f97ce1bc740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55aa7f677f40 con 0x55aa7f726300 2024-03-28T22:20:09.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.194+0000 7f97ce1bc740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55aa7f729000 con 0x55aa7f719040 2024-03-28T22:20:09.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.194+0000 7f97ce1bc740 10 client.0 _ll_register_callbacks cb 0x55aa7f637b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:20:09.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.194+0000 7f97c9297640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa7f718720 0x55aa7f723380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:09.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.194+0000 7f97c9297640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa7f718720 0x55aa7f723380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50282/0 (socket says 172.21.15.16:50282) 2024-03-28T22:20:09.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.194+0000 7f97c9297640 1 -- 172.21.15.16:0/476711317 learned_addr learned my addr 172.21.15.16:0/476711317 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:09.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.194+0000 7f97c9a98640 1 --2- 172.21.15.16:0/476711317 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55aa7f726300 0x55aa7f723e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:09.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.195+0000 7f97c8a96640 1 --2- 172.21.15.16:0/476711317 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aa7f719040 0x55aa7f7238c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:09.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.195+0000 7f97c9297640 1 -- 172.21.15.16:0/476711317 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aa7f719040 msgr2=0x55aa7f7238c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:09.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.195+0000 7f97c9297640 1 --2- 172.21.15.16:0/476711317 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aa7f719040 0x55aa7f7238c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.195+0000 7f97c9297640 1 -- 172.21.15.16:0/476711317 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55aa7f726300 msgr2=0x55aa7f723e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:09.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.195+0000 7f97c9297640 1 --2- 172.21.15.16:0/476711317 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55aa7f726300 0x55aa7f723e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.195+0000 7f97c9297640 1 -- 172.21.15.16:0/476711317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa7f4b8f70 con 0x55aa7f718720 2024-03-28T22:20:09.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.195+0000 7f97c8a96640 1 --2- 172.21.15.16:0/476711317 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aa7f719040 0x55aa7f7238c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:20:09.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.195+0000 7f97c9297640 1 --2- 172.21.15.16:0/476711317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa7f718720 0x55aa7f723380 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f97bc0095f0 tx=0x7f97bc005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb99198492533d11 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:09.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.195+0000 7f97a8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:20:09.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.195+0000 7f97a8ff9640 1 -- 172.21.15.16:0/476711317 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97bc039040 con 0x55aa7f718720 2024-03-28T22:20:09.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.195+0000 7f97a8ff9640 1 -- 172.21.15.16:0/476711317 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7f97bc004480 con 0x55aa7f718720 2024-03-28T22:20:09.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.195+0000 7f97ce1bc740 1 -- 172.21.15.16:0/476711317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97b809e920 con 0x55aa7f718720 2024-03-28T22:20:09.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.195+0000 7f97a8ff9640 1 -- 172.21.15.16:0/476711317 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f97bc04d050 con 0x55aa7f718720 2024-03-28T22:20:09.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.195+0000 7f97ce1bc740 10 client.17550 fetch_fsmap learned FSMap version 27 2024-03-28T22:20:09.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.195+0000 7f97ce1bc740 1 -- 172.21.15.16:0/476711317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55aa7f350170 con 0x55aa7f718720 2024-03-28T22:20:09.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.195+0000 7f97a8ff9640 1 -- 172.21.15.16:0/476711317 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97b809e920 con 0x55aa7f718720 2024-03-28T22:20:09.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.195+0000 7f97a8ff9640 1 -- 172.21.15.16:0/476711317 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f97bc02c510 con 0x55aa7f718720 2024-03-28T22:20:09.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.196+0000 7f97ce1bc740 10 client.17550 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:20:09.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.196+0000 7f97ce1bc740 10 client.17550 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:20:09.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.196+0000 7f97ce1bc740 20 client.17550 populate_metadata read hostname 'smithi016' 2024-03-28T22:20:09.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.196+0000 7f97ce1bc740 1 --2- 172.21.15.16:0/476711317 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55aa7f736230 0x55aa7f738680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:09.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.196+0000 7f97ce1bc740 4 client.17550 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:20:09.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.196+0000 7f97ce1bc740 1 -- 172.21.15.16:0/476711317 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aa7f6893e0 con 0x55aa7f736230 2024-03-28T22:20:09.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.196+0000 7f97c8a96640 1 --2- 172.21.15.16:0/476711317 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55aa7f736230 0x55aa7f738680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:09.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.196+0000 7f97c8a96640 1 --2- 172.21.15.16:0/476711317 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55aa7f736230 0x55aa7f738680 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7219e81fbd473689 server_cookie=bb1e1d8d7fd6c087 in_seq=0 out_seq=0 2024-03-28T22:20:09.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.196+0000 7f97a8ff9640 10 client.17550 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:20:09.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.197+0000 7f97a8ff9640 1 -- 172.21.15.16:0/476711317 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55aa7f6893e0 con 0x55aa7f736230 2024-03-28T22:20:09.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.197+0000 7f97a8ff9640 10 client.17550 handle_command_reply: tid=0 2024-03-28T22:20:09.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.197+0000 7f97ce1bc740 1 client.17550 shutdown 2024-03-28T22:20:09.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.197+0000 7f97ce1bc740 1 -- 172.21.15.16:0/476711317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa7f718720 msgr2=0x55aa7f723380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:09.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.197+0000 7f97ce1bc740 1 --2- 172.21.15.16:0/476711317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa7f718720 0x55aa7f723380 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f97bc0095f0 tx=0x7f97bc005dc0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.197+0000 7f97ce1bc740 1 -- 172.21.15.16:0/476711317 shutdown_connections 2024-03-28T22:20:09.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.197+0000 7f97ce1bc740 1 --2- 172.21.15.16:0/476711317 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55aa7f736230 0x55aa7f738680 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.197+0000 7f97ce1bc740 1 --2- 172.21.15.16:0/476711317 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55aa7f726300 0x55aa7f723e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.197+0000 7f97a8ff9640 0 client.17550 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:20:09.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.197+0000 7f97ce1bc740 1 --2- 172.21.15.16:0/476711317 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aa7f719040 0x55aa7f7238c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.197+0000 7f97ce1bc740 1 --2- 172.21.15.16:0/476711317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa7f718720 0x55aa7f723380 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.197+0000 7f97ce1bc740 1 -- 172.21.15.16:0/476711317 >> 172.21.15.16:0/476711317 conn(0x55aa7f713120 msgr2=0x55aa7f714e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:09.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.197+0000 7f97ce1bc740 1 -- 172.21.15.16:0/476711317 shutdown_connections 2024-03-28T22:20:09.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.197+0000 7f97ce1bc740 1 -- 172.21.15.16:0/476711317 wait complete. 2024-03-28T22:20:09.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.197+0000 7f97ce1bc740 20 client.17550 trim_cache size 0 max 16384 2024-03-28T22:20:09.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.215+0000 7f97ce1bc740 1 Processor -- start 2024-03-28T22:20:09.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.216+0000 7f97ce1bc740 1 -- start start 2024-03-28T22:20:09.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.216+0000 7f97ce1bc740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aa7f718720 0x55aa7f725ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:09.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.216+0000 7f97ce1bc740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55aa7f719040 0x55aa7f7224e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:09.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.216+0000 7f97ce1bc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa7f726300 0x55aa7f731c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:09.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.216+0000 7f97ce1bc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f97c41adaf0 con 0x55aa7f726300 2024-03-28T22:20:09.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.216+0000 7f97ce1bc740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f97c41ab3c0 con 0x55aa7f719040 2024-03-28T22:20:09.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.216+0000 7f97ce1bc740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f97c41a9210 con 0x55aa7f718720 2024-03-28T22:20:09.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.216+0000 7f97c9a98640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa7f726300 0x55aa7f731c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:09.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.216+0000 7f97c9a98640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa7f726300 0x55aa7f731c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50298/0 (socket says 172.21.15.16:50298) 2024-03-28T22:20:09.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.216+0000 7f97c8a96640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55aa7f719040 0x55aa7f7224e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.216+0000 7f97c9a98640 1 -- 172.21.15.16:0/1497502870 learned_addr learned my addr 172.21.15.16:0/1497502870 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.216+0000 7f97c9297640 1 --2- 172.21.15.16:0/1497502870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aa7f718720 0x55aa7f725ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.216+0000 7f97c9a98640 1 -- 172.21.15.16:0/1497502870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aa7f718720 msgr2=0x55aa7f725ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.216+0000 7f97c9a98640 1 --2- 172.21.15.16:0/1497502870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aa7f718720 0x55aa7f725ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.216+0000 7f97c9a98640 1 -- 172.21.15.16:0/1497502870 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55aa7f719040 msgr2=0x55aa7f7224e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.216+0000 7f97c9a98640 1 --2- 172.21.15.16:0/1497502870 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55aa7f719040 0x55aa7f7224e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.216+0000 7f97c9a98640 1 -- 172.21.15.16:0/1497502870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa7f7218d0 con 0x55aa7f726300 2024-03-28T22:20:09.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.216+0000 7f97c9a98640 1 --2- 172.21.15.16:0/1497502870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa7f726300 0x55aa7f731c90 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f97b400ccc0 tx=0x7f97b401c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7541f275a423f6fc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:09.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.217+0000 7f97aaffd640 1 -- 172.21.15.16:0/1497502870 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b402d070 con 0x55aa7f726300 2024-03-28T22:20:09.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.217+0000 7f97aaffd640 1 -- 172.21.15.16:0/1497502870 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97b4023040 con 0x55aa7f726300 2024-03-28T22:20:09.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.217+0000 7f97aaffd640 1 -- 172.21.15.16:0/1497502870 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b4017040 con 0x55aa7f726300 2024-03-28T22:20:09.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.217+0000 7f97ce1bc740 1 -- 172.21.15.16:0/1497502870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa7f726300 msgr2=0x55aa7f731c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:09.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.217+0000 7f97ce1bc740 1 --2- 172.21.15.16:0/1497502870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa7f726300 0x55aa7f731c90 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f97b400ccc0 tx=0x7f97b401c000 comp rx=0 tx=0).stop 2024-03-28T22:20:09.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.217+0000 7f97ce1bc740 1 -- 172.21.15.16:0/1497502870 shutdown_connections 2024-03-28T22:20:09.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.217+0000 7f97ce1bc740 1 --2- 172.21.15.16:0/1497502870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa7f726300 0x55aa7f731c90 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.217+0000 7f97ce1bc740 1 --2- 172.21.15.16:0/1497502870 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55aa7f719040 0x55aa7f7224e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.217+0000 7f97ce1bc740 1 --2- 172.21.15.16:0/1497502870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aa7f718720 0x55aa7f725ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.217+0000 7f97ce1bc740 1 -- 172.21.15.16:0/1497502870 >> 172.21.15.16:0/1497502870 conn(0x55aa7f713120 msgr2=0x55aa7f7118f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:09.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.217+0000 7f97ce1bc740 1 -- 172.21.15.16:0/1497502870 shutdown_connections 2024-03-28T22:20:09.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.217+0000 7f97ce1bc740 1 -- 172.21.15.16:0/1497502870 wait complete. 2024-03-28T22:20:09.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.218+0000 7f97ce1bc740 1 -- start start 2024-03-28T22:20:09.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.218+0000 7f97ce1bc740 1 Processor -- start 2024-03-28T22:20:09.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.218+0000 7f97ce1bc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa7f718720 0x55aa7f72e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:09.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.218+0000 7f97ce1bc740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aa7f719040 0x55aa7f72b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:09.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.218+0000 7f97ce1bc740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55aa7f726300 0x55aa7f72b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:09.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.218+0000 7f97ce1bc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f97c41a9880 con 0x55aa7f718720 2024-03-28T22:20:09.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.218+0000 7f97ce1bc740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f97c41aacf0 con 0x55aa7f726300 2024-03-28T22:20:09.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.218+0000 7f97ce1bc740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f97c41a9f50 con 0x55aa7f719040 2024-03-28T22:20:09.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.218+0000 7f97ce1bc740 10 client.0 _ll_register_callbacks cb 0x7f97bc035f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:20:09.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.218+0000 7f97c9297640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa7f718720 0x55aa7f72e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:09.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.218+0000 7f97c9297640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa7f718720 0x55aa7f72e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50312/0 (socket says 172.21.15.16:50312) 2024-03-28T22:20:09.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.218+0000 7f97c9297640 1 -- 172.21.15.16:0/39264036 learned_addr learned my addr 172.21.15.16:0/39264036 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:09.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.218+0000 7f97c9a98640 1 --2- 172.21.15.16:0/39264036 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55aa7f726300 0x55aa7f72b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:09.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.218+0000 7f97c8a96640 1 --2- 172.21.15.16:0/39264036 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aa7f719040 0x55aa7f72b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:09.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.219+0000 7f97c9297640 1 -- 172.21.15.16:0/39264036 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aa7f719040 msgr2=0x55aa7f72b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:09.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.219+0000 7f97c9297640 1 --2- 172.21.15.16:0/39264036 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aa7f719040 0x55aa7f72b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.219+0000 7f97c9297640 1 -- 172.21.15.16:0/39264036 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55aa7f726300 msgr2=0x55aa7f72b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:09.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.219+0000 7f97c9297640 1 --2- 172.21.15.16:0/39264036 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55aa7f726300 0x55aa7f72b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.219+0000 7f97c9297640 1 -- 172.21.15.16:0/39264036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa7f72bd10 con 0x55aa7f718720 2024-03-28T22:20:09.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.219+0000 7f97c9297640 1 --2- 172.21.15.16:0/39264036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa7f718720 0x55aa7f72e2c0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f97bc02c960 tx=0x7f97bc035e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b73c1ce93a2ea124 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:09.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.219+0000 7f97a8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:20:09.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.219+0000 7f97a8ff9640 1 -- 172.21.15.16:0/39264036 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97bc039040 con 0x55aa7f718720 2024-03-28T22:20:09.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.219+0000 7f97a8ff9640 1 -- 172.21.15.16:0/39264036 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7f97bc004480 con 0x55aa7f718720 2024-03-28T22:20:09.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.219+0000 7f97ce1bc740 1 -- 172.21.15.16:0/39264036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97b4023040 con 0x55aa7f718720 2024-03-28T22:20:09.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.219+0000 7f97a8ff9640 1 -- 172.21.15.16:0/39264036 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f97bc034440 con 0x55aa7f718720 2024-03-28T22:20:09.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.219+0000 7f97ce1bc740 10 client.17556 fetch_fsmap learned FSMap version 27 2024-03-28T22:20:09.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.219+0000 7f97ce1bc740 1 -- 172.21.15.16:0/39264036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55aa7f79d1d0 con 0x55aa7f718720 2024-03-28T22:20:09.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.219+0000 7f97a8ff9640 1 -- 172.21.15.16:0/39264036 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97b4023040 con 0x55aa7f718720 2024-03-28T22:20:09.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.219+0000 7f97a8ff9640 1 -- 172.21.15.16:0/39264036 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f97bc03f480 con 0x55aa7f718720 2024-03-28T22:20:09.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.219+0000 7f97ce1bc740 10 client.17556 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:20:09.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.219+0000 7f97ce1bc740 10 client.17556 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:20:09.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.219+0000 7f97ce1bc740 20 client.17556 populate_metadata read hostname 'smithi016' 2024-03-28T22:20:09.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.220+0000 7f97ce1bc740 1 --2- 172.21.15.16:0/39264036 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55aa7f79d910 0x55aa7f79fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:09.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.220+0000 7f97ce1bc740 4 client.17556 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:20:09.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.220+0000 7f97ce1bc740 1 -- 172.21.15.16:0/39264036 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55aa7f683730 con 0x55aa7f79d910 2024-03-28T22:20:09.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.220+0000 7f97c8a96640 1 --2- 172.21.15.16:0/39264036 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55aa7f79d910 0x55aa7f79fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:09.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.220+0000 7f97c8a96640 1 --2- 172.21.15.16:0/39264036 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55aa7f79d910 0x55aa7f79fd30 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dba7c5c858b83742 server_cookie=98ca6a3abee14e53 in_seq=0 out_seq=0 2024-03-28T22:20:09.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.220+0000 7f97a8ff9640 10 client.17556 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:20:09.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.222+0000 7f97a8ff9640 1 -- 172.21.15.16:0/39264036 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55aa7f683730 con 0x55aa7f79d910 2024-03-28T22:20:09.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.222+0000 7f97a8ff9640 10 client.17556 handle_command_reply: tid=0 2024-03-28T22:20:09.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.222+0000 7f97ce1bc740 1 client.17556 shutdown 2024-03-28T22:20:09.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.222+0000 7f97ce1bc740 1 -- 172.21.15.16:0/39264036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa7f718720 msgr2=0x55aa7f72e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:09.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.222+0000 7f97ce1bc740 1 --2- 172.21.15.16:0/39264036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa7f718720 0x55aa7f72e2c0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f97bc02c960 tx=0x7f97bc035e90 comp rx=0 tx=0).stop 2024-03-28T22:20:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.222+0000 7f97ce1bc740 1 -- 172.21.15.16:0/39264036 shutdown_connections 2024-03-28T22:20:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.222+0000 7f97ce1bc740 1 --2- 172.21.15.16:0/39264036 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55aa7f79d910 0x55aa7f79fd30 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.222+0000 7f97ce1bc740 1 --2- 172.21.15.16:0/39264036 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55aa7f726300 0x55aa7f72b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.222+0000 7f97ce1bc740 1 --2- 172.21.15.16:0/39264036 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aa7f719040 0x55aa7f72b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.222+0000 7f97a8ff9640 0 client.17556 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:20:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.222+0000 7f97ce1bc740 1 --2- 172.21.15.16:0/39264036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aa7f718720 0x55aa7f72e2c0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.222+0000 7f97ce1bc740 1 -- 172.21.15.16:0/39264036 >> 172.21.15.16:0/39264036 conn(0x55aa7f713120 msgr2=0x55aa7f712180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.222+0000 7f97ce1bc740 1 -- 172.21.15.16:0/39264036 shutdown_connections 2024-03-28T22:20:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.222+0000 7f97ce1bc740 1 -- 172.21.15.16:0/39264036 wait complete. 2024-03-28T22:20:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.222+0000 7f97ce1bc740 20 client.17556 trim_cache size 0 max 16384 2024-03-28T22:20:09.234 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:20:09.234 INFO:teuthology.orchestra.run.smithi016.stdout: "return_code": 0, 2024-03-28T22:20:09.234 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_tag": "fb2941ee-edb3-4281-b3c1-b3cc0a9bbfa7", 2024-03-28T22:20:09.234 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": "asynchronous" 2024-03-28T22:20:09.235 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:20:09.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.222+0000 7f97cb4fb640 1 -- 172.21.15.16:0/2051327326 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f97ac0a1ef0 msgr2=0x7f97ac0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:09.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.222+0000 7f97cb4fb640 1 --2- 172.21.15.16:0/2051327326 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f97ac0a1ef0 0x7f97ac0a43b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f97b4005fd0 tx=0x7f97b400eeb0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.222+0000 7f97cb4fb640 1 -- 172.21.15.16:0/2051327326 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f97c41020b0 msgr2=0x7f97c41985b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:09.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.222+0000 7f97cb4fb640 1 --2- 172.21.15.16:0/2051327326 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f97c41020b0 0x7f97c41985b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f97b800d950 tx=0x7f97b800de10 comp rx=0 tx=0).stop 2024-03-28T22:20:09.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.223+0000 7f97cb4fb640 1 -- 172.21.15.16:0/2051327326 shutdown_connections 2024-03-28T22:20:09.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.223+0000 7f97cb4fb640 1 --2- 172.21.15.16:0/2051327326 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f97ac0a1ef0 0x7f97ac0a43b0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.223+0000 7f97cb4fb640 1 --2- 172.21.15.16:0/2051327326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c4107700 0x7f97c419c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.223+0000 7f97cb4fb640 1 --2- 172.21.15.16:0/2051327326 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f97c41020b0 0x7f97c41985b0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.223+0000 7f97cb4fb640 1 --2- 172.21.15.16:0/2051327326 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f97c41016f0 0x7f97c4198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.223+0000 7f97cb4fb640 1 -- 172.21.15.16:0/2051327326 >> 172.21.15.16:0/2051327326 conn(0x7f97c40fd1e0 msgr2=0x7f97c40fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:09.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.223+0000 7f97cb4fb640 1 -- 172.21.15.16:0/2051327326 shutdown_connections 2024-03-28T22:20:09.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.223+0000 7f97cb4fb640 1 -- 172.21.15.16:0/2051327326 wait complete. 2024-03-28T22:20:09.238 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:20:09.238 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:20:09.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.343+0000 7f401b5ae640 1 -- 172.21.15.16:0/2941018766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4014106680 msgr2=0x7f4014106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:09.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.343+0000 7f401b5ae640 1 --2- 172.21.15.16:0/2941018766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4014106680 0x7f4014106a60 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f4010009930 tx=0x7f401002bc30 comp rx=0 tx=0).stop 2024-03-28T22:20:09.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.343+0000 7f401b5ae640 1 -- 172.21.15.16:0/2941018766 shutdown_connections 2024-03-28T22:20:09.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.343+0000 7f401b5ae640 1 --2- 172.21.15.16:0/2941018766 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4014100fa0 0x7f401410d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.343+0000 7f401b5ae640 1 --2- 172.21.15.16:0/2941018766 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40141005e0 0x7f4014100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.343+0000 7f401b5ae640 1 --2- 172.21.15.16:0/2941018766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4014106680 0x7f4014106a60 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.343+0000 7f401b5ae640 1 -- 172.21.15.16:0/2941018766 >> 172.21.15.16:0/2941018766 conn(0x7f4014077770 msgr2=0x7f4014077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:09.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.343+0000 7f401b5ae640 1 -- 172.21.15.16:0/2941018766 shutdown_connections 2024-03-28T22:20:09.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.343+0000 7f401b5ae640 1 -- 172.21.15.16:0/2941018766 wait complete. 2024-03-28T22:20:09.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.344+0000 7f401b5ae640 1 Processor -- start 2024-03-28T22:20:09.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.344+0000 7f401b5ae640 1 -- start start 2024-03-28T22:20:09.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.344+0000 7f401b5ae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40141005e0 0x7f40141980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:09.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.344+0000 7f401b5ae640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4014100fa0 0x7f4014198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:09.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.344+0000 7f401b5ae640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4014106680 0x7f401419c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:09.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.344+0000 7f401b5ae640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4014111b00 con 0x7f40141005e0 2024-03-28T22:20:09.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.344+0000 7f401b5ae640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4014111980 con 0x7f4014100fa0 2024-03-28T22:20:09.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.344+0000 7f401b5ae640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4014111c80 con 0x7f4014106680 2024-03-28T22:20:09.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.344+0000 7f401934a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40141005e0 0x7f40141980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:09.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.344+0000 7f401934a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40141005e0 0x7f40141980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50328/0 (socket says 172.21.15.16:50328) 2024-03-28T22:20:09.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.344+0000 7f401934a640 1 -- 172.21.15.16:0/2047433448 learned_addr learned my addr 172.21.15.16:0/2047433448 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:09.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.344+0000 7f4018b49640 1 --2- 172.21.15.16:0/2047433448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4014100fa0 0x7f4014198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:09.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.344+0000 7f4019b4b640 1 --2- 172.21.15.16:0/2047433448 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4014106680 0x7f401419c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:09.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.344+0000 7f401934a640 1 -- 172.21.15.16:0/2047433448 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4014106680 msgr2=0x7f401419c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:09.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.344+0000 7f401934a640 1 --2- 172.21.15.16:0/2047433448 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4014106680 0x7f401419c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.344+0000 7f401934a640 1 -- 172.21.15.16:0/2047433448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4014100fa0 msgr2=0x7f4014198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:09.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.344+0000 7f401934a640 1 --2- 172.21.15.16:0/2047433448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4014100fa0 0x7f4014198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.344+0000 7f401934a640 1 -- 172.21.15.16:0/2047433448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f401419d0b0 con 0x7f40141005e0 2024-03-28T22:20:09.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.344+0000 7f4019b4b640 1 --2- 172.21.15.16:0/2047433448 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4014106680 0x7f401419c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:20:09.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.345+0000 7f401934a640 1 --2- 172.21.15.16:0/2047433448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40141005e0 0x7f40141980c0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f4010009230 tx=0x7f401002cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:09.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.345+0000 7f400a7fc640 1 -- 172.21.15.16:0/2047433448 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4010041070 con 0x7f40141005e0 2024-03-28T22:20:09.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.345+0000 7f400a7fc640 1 -- 172.21.15.16:0/2047433448 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4010004450 con 0x7f40141005e0 2024-03-28T22:20:09.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.345+0000 7f401b5ae640 1 -- 172.21.15.16:0/2047433448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f401419d340 con 0x7f40141005e0 2024-03-28T22:20:09.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.345+0000 7f400a7fc640 1 -- 172.21.15.16:0/2047433448 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4010034440 con 0x7f40141005e0 2024-03-28T22:20:09.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.345+0000 7f401b5ae640 1 -- 172.21.15.16:0/2047433448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40141a4c30 con 0x7f40141005e0 2024-03-28T22:20:09.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.347+0000 7f400a7fc640 1 -- 172.21.15.16:0/2047433448 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f4010044070 con 0x7f40141005e0 2024-03-28T22:20:09.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.347+0000 7f401e18a740 1 Processor -- start 2024-03-28T22:20:09.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.347+0000 7f401e18a740 1 -- start start 2024-03-28T22:20:09.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.348+0000 7f400a7fc640 1 --2- 172.21.15.16:0/2047433448 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f40000a1ef0 0x7f40000a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:09.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.348+0000 7f401e18a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5591c0d44f40 0x5591c0d45320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:09.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.348+0000 7f401e18a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5591c0d45860 0x5591c0d52660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:09.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.348+0000 7f401e18a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c0d52ba0 0x5591c0d54f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:09.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.348+0000 7f401e18a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5591c0ca9180 con 0x5591c0d52ba0 2024-03-28T22:20:09.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.348+0000 7f401e18a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5591c0c51b70 con 0x5591c0d44f40 2024-03-28T22:20:09.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.348+0000 7f401e18a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5591c0ca5220 con 0x5591c0d45860 2024-03-28T22:20:09.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.348+0000 7f4019b4b640 1 --2- 172.21.15.16:0/2047433448 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f40000a1ef0 0x7f40000a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:09.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.348+0000 7f400a7fc640 1 -- 172.21.15.16:0/2047433448 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7f40100efd20 con 0x7f40141005e0 2024-03-28T22:20:09.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.348+0000 7f401934a640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5591c0d44f40 0x5591c0d45320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:09.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.348+0000 7f401934a640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5591c0d44f40 0x5591c0d45320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:38110/0 (socket says 172.21.15.16:38110) 2024-03-28T22:20:09.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.348+0000 7f401934a640 1 -- 172.21.15.16:0/1649423281 learned_addr learned my addr 172.21.15.16:0/1649423281 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:09.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.348+0000 7f4019b4b640 1 --2- 172.21.15.16:0/1649423281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c0d52ba0 0x5591c0d54f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:09.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.348+0000 7f4018b49640 1 --2- 172.21.15.16:0/1649423281 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5591c0d45860 0x5591c0d52660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:09.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.348+0000 7f401934a640 1 -- 172.21.15.16:0/1649423281 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5591c0d45860 msgr2=0x5591c0d52660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:09.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.348+0000 7f401934a640 1 --2- 172.21.15.16:0/1649423281 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5591c0d45860 0x5591c0d52660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.348+0000 7f401934a640 1 -- 172.21.15.16:0/1649423281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c0d52ba0 msgr2=0x5591c0d54f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:09.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.348+0000 7f401934a640 1 --2- 172.21.15.16:0/1649423281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c0d52ba0 0x5591c0d54f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.348+0000 7f401934a640 1 -- 172.21.15.16:0/1649423281 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591c0c657c0 con 0x5591c0d44f40 2024-03-28T22:20:09.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.348+0000 7f4019b4b640 1 --2- 172.21.15.16:0/2047433448 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f40000a1ef0 0x7f40000a43b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f400c005fd0 tx=0x7f400c00ac20 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:09.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.348+0000 7f401934a640 1 --2- 172.21.15.16:0/1649423281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5591c0d44f40 0x5591c0d45320 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f40100b8c20 tx=0x7f401002bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a822375b9c3a8c9a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:09.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.349+0000 7f3fff7fe640 1 -- 172.21.15.16:0/1649423281 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f401004b040 con 0x5591c0d44f40 2024-03-28T22:20:09.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.349+0000 7f3fff7fe640 1 -- 172.21.15.16:0/1649423281 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4010004c80 con 0x5591c0d44f40 2024-03-28T22:20:09.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.349+0000 7f3fff7fe640 1 -- 172.21.15.16:0/1649423281 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40100c0910 con 0x5591c0d44f40 2024-03-28T22:20:09.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.349+0000 7f401e18a740 1 -- 172.21.15.16:0/1649423281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5591c0d44f40 msgr2=0x5591c0d45320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:09.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.349+0000 7f401e18a740 1 --2- 172.21.15.16:0/1649423281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5591c0d44f40 0x5591c0d45320 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f40100b8c20 tx=0x7f401002bc30 comp rx=0 tx=0).stop 2024-03-28T22:20:09.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.349+0000 7f401e18a740 1 -- 172.21.15.16:0/1649423281 shutdown_connections 2024-03-28T22:20:09.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.349+0000 7f401e18a740 1 --2- 172.21.15.16:0/1649423281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c0d52ba0 0x5591c0d54f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.349+0000 7f401e18a740 1 --2- 172.21.15.16:0/1649423281 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5591c0d45860 0x5591c0d52660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.349+0000 7f401e18a740 1 --2- 172.21.15.16:0/1649423281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5591c0d44f40 0x5591c0d45320 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.349+0000 7f401e18a740 1 -- 172.21.15.16:0/1649423281 >> 172.21.15.16:0/1649423281 conn(0x5591c0d3f9c0 msgr2=0x5591c0d3fda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:09.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.349+0000 7f401e18a740 1 -- 172.21.15.16:0/1649423281 shutdown_connections 2024-03-28T22:20:09.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.349+0000 7f401e18a740 1 -- 172.21.15.16:0/1649423281 wait complete. 2024-03-28T22:20:09.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.350+0000 7f401e18a740 1 -- start start 2024-03-28T22:20:09.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.350+0000 7f401e18a740 1 Processor -- start 2024-03-28T22:20:09.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.350+0000 7f401e18a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5591c0d44f40 0x5591c0d4fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:09.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.350+0000 7f401e18a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c0d45860 0x5591c0d50130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:09.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.350+0000 7f401e18a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5591c0d52ba0 0x5591c0d50670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:09.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.350+0000 7f401e18a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5591c0d557a0 con 0x5591c0d45860 2024-03-28T22:20:09.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.350+0000 7f401e18a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5591c0c474a0 con 0x5591c0d52ba0 2024-03-28T22:20:09.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.350+0000 7f401e18a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5591c0d55920 con 0x5591c0d44f40 2024-03-28T22:20:09.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.350+0000 7f401e18a740 10 client.0 _ll_register_callbacks cb 0x5591c0c4ad90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:20:09.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.350+0000 7f4018b49640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c0d45860 0x5591c0d50130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:09.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.350+0000 7f4018b49640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c0d45860 0x5591c0d50130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50340/0 (socket says 172.21.15.16:50340) 2024-03-28T22:20:09.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.350+0000 7f4018b49640 1 -- 172.21.15.16:0/1452024571 learned_addr learned my addr 172.21.15.16:0/1452024571 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:09.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.350+0000 7f401934a640 1 --2- 172.21.15.16:0/1452024571 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5591c0d44f40 0x5591c0d4fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:09.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.350+0000 7f4019b4b640 1 --2- 172.21.15.16:0/1452024571 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5591c0d52ba0 0x5591c0d50670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:09.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.350+0000 7f4018b49640 1 -- 172.21.15.16:0/1452024571 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5591c0d44f40 msgr2=0x5591c0d4fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:09.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.350+0000 7f4018b49640 1 --2- 172.21.15.16:0/1452024571 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5591c0d44f40 0x5591c0d4fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.350+0000 7f4018b49640 1 -- 172.21.15.16:0/1452024571 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5591c0d52ba0 msgr2=0x5591c0d50670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:09.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.350+0000 7f4018b49640 1 --2- 172.21.15.16:0/1452024571 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5591c0d52ba0 0x5591c0d50670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.350+0000 7f4018b49640 1 -- 172.21.15.16:0/1452024571 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591c0a877d0 con 0x5591c0d45860 2024-03-28T22:20:09.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.351+0000 7f4018b49640 1 --2- 172.21.15.16:0/1452024571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c0d45860 0x5591c0d50130 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f4004012700 tx=0x7f4004012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56b3d79a1751800c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:09.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.351+0000 7f3ffcff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:20:09.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.351+0000 7f3ffcff9640 1 -- 172.21.15.16:0/1452024571 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4004004050 con 0x5591c0d45860 2024-03-28T22:20:09.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.351+0000 7f3ffcff9640 1 -- 172.21.15.16:0/1452024571 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7f40040041f0 con 0x5591c0d45860 2024-03-28T22:20:09.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.351+0000 7f401e18a740 1 -- 172.21.15.16:0/1452024571 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4010004c80 con 0x5591c0d45860 2024-03-28T22:20:09.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.351+0000 7f3ffcff9640 1 -- 172.21.15.16:0/1452024571 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f40040043f0 con 0x5591c0d45860 2024-03-28T22:20:09.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.351+0000 7f401e18a740 10 client.17568 fetch_fsmap learned FSMap version 27 2024-03-28T22:20:09.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.351+0000 7f401e18a740 1 -- 172.21.15.16:0/1452024571 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5591c09bd3f0 con 0x5591c0d45860 2024-03-28T22:20:09.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.351+0000 7f3ffcff9640 1 -- 172.21.15.16:0/1452024571 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4010004c80 con 0x5591c0d45860 2024-03-28T22:20:09.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.351+0000 7f3ffcff9640 1 -- 172.21.15.16:0/1452024571 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f4004003bc0 con 0x5591c0d45860 2024-03-28T22:20:09.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.351+0000 7f401e18a740 10 client.17568 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:20:09.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.351+0000 7f401e18a740 10 client.17568 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:20:09.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.351+0000 7f401e18a740 20 client.17568 populate_metadata read hostname 'smithi016' 2024-03-28T22:20:09.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.351+0000 7f401e18a740 1 --2- 172.21.15.16:0/1452024571 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5591c0d62dc0 0x5591c0d651d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:09.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.351+0000 7f401e18a740 4 client.17568 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:20:09.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.351+0000 7f401e18a740 1 -- 172.21.15.16:0/1452024571 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5591c0cb66c0 con 0x5591c0d62dc0 2024-03-28T22:20:09.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.352+0000 7f401934a640 1 --2- 172.21.15.16:0/1452024571 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5591c0d62dc0 0x5591c0d651d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:09.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.352+0000 7f3ffcff9640 10 client.17568 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:20:09.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.352+0000 7f401934a640 1 --2- 172.21.15.16:0/1452024571 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5591c0d62dc0 0x5591c0d651d0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0f012addef7e784 server_cookie=8a1a0233baa75696 in_seq=0 out_seq=0 2024-03-28T22:20:09.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.353+0000 7f3ffcff9640 1 -- 172.21.15.16:0/1452024571 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5591c0cb66c0 con 0x5591c0d62dc0 2024-03-28T22:20:09.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.353+0000 7f3ffcff9640 10 client.17568 handle_command_reply: tid=0 2024-03-28T22:20:09.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.353+0000 7f401e18a740 1 client.17568 shutdown 2024-03-28T22:20:09.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.353+0000 7f401e18a740 1 -- 172.21.15.16:0/1452024571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c0d45860 msgr2=0x5591c0d50130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:09.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.353+0000 7f401e18a740 1 --2- 172.21.15.16:0/1452024571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c0d45860 0x5591c0d50130 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f4004012700 tx=0x7f4004012bc0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.353+0000 7f401e18a740 1 -- 172.21.15.16:0/1452024571 shutdown_connections 2024-03-28T22:20:09.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.353+0000 7f401e18a740 1 --2- 172.21.15.16:0/1452024571 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5591c0d62dc0 0x5591c0d651d0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.353+0000 7f401e18a740 1 --2- 172.21.15.16:0/1452024571 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5591c0d52ba0 0x5591c0d50670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.353+0000 7f401e18a740 1 --2- 172.21.15.16:0/1452024571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c0d45860 0x5591c0d50130 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.353+0000 7f3ffcff9640 0 client.17568 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:20:09.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.353+0000 7f401e18a740 1 --2- 172.21.15.16:0/1452024571 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5591c0d44f40 0x5591c0d4fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.353+0000 7f401e18a740 1 -- 172.21.15.16:0/1452024571 >> 172.21.15.16:0/1452024571 conn(0x5591c0d3f9c0 msgr2=0x5591c0d41710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:09.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.353+0000 7f401e18a740 1 -- 172.21.15.16:0/1452024571 shutdown_connections 2024-03-28T22:20:09.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.353+0000 7f401e18a740 1 -- 172.21.15.16:0/1452024571 wait complete. 2024-03-28T22:20:09.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.353+0000 7f401e18a740 20 client.17568 trim_cache size 0 max 16384 2024-03-28T22:20:09.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.373+0000 7f401e18a740 1 Processor -- start 2024-03-28T22:20:09.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.373+0000 7f401e18a740 1 -- start start 2024-03-28T22:20:09.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.373+0000 7f401e18a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5591c0d44f40 0x5591c0d3ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:09.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.373+0000 7f401e18a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5591c0d45860 0x5591c0d5e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:09.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.373+0000 7f401e18a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c0d52ba0 0x5591c0d6aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:09.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.373+0000 7f401e18a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f40141adc00 con 0x5591c0d52ba0 2024-03-28T22:20:09.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.373+0000 7f401e18a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f40141ab4d0 con 0x5591c0d44f40 2024-03-28T22:20:09.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.373+0000 7f401e18a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f40141a9340 con 0x5591c0d45860 2024-03-28T22:20:09.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.373+0000 7f4019b4b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c0d52ba0 0x5591c0d6aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:09.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.373+0000 7f4019b4b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c0d52ba0 0x5591c0d6aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50346/0 (socket says 172.21.15.16:50346) 2024-03-28T22:20:09.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.373+0000 7f4019b4b640 1 -- 172.21.15.16:0/4097733556 learned_addr learned my addr 172.21.15.16:0/4097733556 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:09.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.373+0000 7f401934a640 1 --2- 172.21.15.16:0/4097733556 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5591c0d44f40 0x5591c0d3ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:09.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.373+0000 7f4018b49640 1 --2- 172.21.15.16:0/4097733556 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5591c0d45860 0x5591c0d5e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:09.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.374+0000 7f4019b4b640 1 -- 172.21.15.16:0/4097733556 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5591c0d45860 msgr2=0x5591c0d5e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:09.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.374+0000 7f4019b4b640 1 --2- 172.21.15.16:0/4097733556 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5591c0d45860 0x5591c0d5e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.374+0000 7f4019b4b640 1 -- 172.21.15.16:0/4097733556 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5591c0d44f40 msgr2=0x5591c0d3ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:09.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.374+0000 7f4019b4b640 1 --2- 172.21.15.16:0/4097733556 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5591c0d44f40 0x5591c0d3ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.374+0000 7f4019b4b640 1 -- 172.21.15.16:0/4097733556 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591c0d45690 con 0x5591c0d52ba0 2024-03-28T22:20:09.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.374+0000 7f4018b49640 1 --2- 172.21.15.16:0/4097733556 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5591c0d45860 0x5591c0d5e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:20:09.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.374+0000 7f4019b4b640 1 --2- 172.21.15.16:0/4097733556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c0d52ba0 0x5591c0d6aae0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f400c0120f0 tx=0x7f400c01c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=531ba9372d976abe server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:09.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.374+0000 7f3ffeffd640 1 -- 172.21.15.16:0/4097733556 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f400c011b70 con 0x5591c0d52ba0 2024-03-28T22:20:09.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.374+0000 7f3ffeffd640 1 -- 172.21.15.16:0/4097733556 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f400c011d10 con 0x5591c0d52ba0 2024-03-28T22:20:09.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.374+0000 7f3ffeffd640 1 -- 172.21.15.16:0/4097733556 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f400c01f610 con 0x5591c0d52ba0 2024-03-28T22:20:09.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.375+0000 7f401e18a740 1 -- 172.21.15.16:0/4097733556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c0d52ba0 msgr2=0x5591c0d6aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:09.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.375+0000 7f401e18a740 1 --2- 172.21.15.16:0/4097733556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c0d52ba0 0x5591c0d6aae0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f400c0120f0 tx=0x7f400c01c000 comp rx=0 tx=0).stop 2024-03-28T22:20:09.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.375+0000 7f401e18a740 1 -- 172.21.15.16:0/4097733556 shutdown_connections 2024-03-28T22:20:09.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.375+0000 7f401e18a740 1 --2- 172.21.15.16:0/4097733556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c0d52ba0 0x5591c0d6aae0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.375+0000 7f401e18a740 1 --2- 172.21.15.16:0/4097733556 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5591c0d45860 0x5591c0d5e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.375+0000 7f401e18a740 1 --2- 172.21.15.16:0/4097733556 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5591c0d44f40 0x5591c0d3ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.375+0000 7f401e18a740 1 -- 172.21.15.16:0/4097733556 >> 172.21.15.16:0/4097733556 conn(0x5591c0d3f9c0 msgr2=0x5591c0d416a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:09.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.375+0000 7f401e18a740 1 -- 172.21.15.16:0/4097733556 shutdown_connections 2024-03-28T22:20:09.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.375+0000 7f401e18a740 1 -- 172.21.15.16:0/4097733556 wait complete. 2024-03-28T22:20:09.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.375+0000 7f401e18a740 1 -- start start 2024-03-28T22:20:09.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.375+0000 7f401e18a740 1 Processor -- start 2024-03-28T22:20:09.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.376+0000 7f401e18a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5591c0d44f40 0x5591c0d5ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:09.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.376+0000 7f401e18a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5591c0d45860 0x5591c0d57b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:09.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.376+0000 7f401e18a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c0d52ba0 0x5591c0d580c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:09.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.376+0000 7f401e18a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f40141a9990 con 0x5591c0d52ba0 2024-03-28T22:20:09.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.376+0000 7f401e18a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f40141aae00 con 0x5591c0d44f40 2024-03-28T22:20:09.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.376+0000 7f401e18a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f40141aa060 con 0x5591c0d45860 2024-03-28T22:20:09.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.376+0000 7f401e18a740 10 client.0 _ll_register_callbacks cb 0x7f40040097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:20:09.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.376+0000 7f4019b4b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c0d52ba0 0x5591c0d580c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:09.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.376+0000 7f4019b4b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c0d52ba0 0x5591c0d580c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50360/0 (socket says 172.21.15.16:50360) 2024-03-28T22:20:09.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.376+0000 7f4019b4b640 1 -- 172.21.15.16:0/1682586448 learned_addr learned my addr 172.21.15.16:0/1682586448 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:09.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.376+0000 7f401934a640 1 --2- 172.21.15.16:0/1682586448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5591c0d44f40 0x5591c0d5ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:09.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.376+0000 7f4018b49640 1 --2- 172.21.15.16:0/1682586448 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5591c0d45860 0x5591c0d57b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:09.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.376+0000 7f4019b4b640 1 -- 172.21.15.16:0/1682586448 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5591c0d45860 msgr2=0x5591c0d57b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:09.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.376+0000 7f4019b4b640 1 --2- 172.21.15.16:0/1682586448 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5591c0d45860 0x5591c0d57b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.376+0000 7f4019b4b640 1 -- 172.21.15.16:0/1682586448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5591c0d44f40 msgr2=0x5591c0d5ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:09.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.376+0000 7f4019b4b640 1 --2- 172.21.15.16:0/1682586448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5591c0d44f40 0x5591c0d5ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.376+0000 7f4019b4b640 1 -- 172.21.15.16:0/1682586448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591c097d0f0 con 0x5591c0d52ba0 2024-03-28T22:20:09.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.376+0000 7f4019b4b640 1 --2- 172.21.15.16:0/1682586448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c0d52ba0 0x5591c0d580c0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f400c029e00 tx=0x7f400c0123f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c13f3edf3977a02c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:09.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.376+0000 7f3ffcff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:20:09.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.376+0000 7f3ffcff9640 1 -- 172.21.15.16:0/1682586448 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f400c011b70 con 0x5591c0d52ba0 2024-03-28T22:20:09.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.376+0000 7f3ffcff9640 1 -- 172.21.15.16:0/1682586448 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7f400c029480 con 0x5591c0d52ba0 2024-03-28T22:20:09.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.376+0000 7f401e18a740 1 -- 172.21.15.16:0/1682586448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f400c011d10 con 0x5591c0d52ba0 2024-03-28T22:20:09.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.377+0000 7f3ffcff9640 1 -- 172.21.15.16:0/1682586448 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f400c029680 con 0x5591c0d52ba0 2024-03-28T22:20:09.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.377+0000 7f401e18a740 10 client.17574 fetch_fsmap learned FSMap version 27 2024-03-28T22:20:09.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.377+0000 7f401e18a740 1 -- 172.21.15.16:0/1682586448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5591c0dca000 con 0x5591c0d52ba0 2024-03-28T22:20:09.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.377+0000 7f3ffcff9640 1 -- 172.21.15.16:0/1682586448 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f400c011d10 con 0x5591c0d52ba0 2024-03-28T22:20:09.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.377+0000 7f3ffcff9640 1 -- 172.21.15.16:0/1682586448 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f400c03ca10 con 0x5591c0d52ba0 2024-03-28T22:20:09.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.377+0000 7f401e18a740 10 client.17574 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:20:09.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.377+0000 7f401e18a740 10 client.17574 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:20:09.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.377+0000 7f401e18a740 20 client.17574 populate_metadata read hostname 'smithi016' 2024-03-28T22:20:09.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.377+0000 7f401e18a740 1 --2- 172.21.15.16:0/1682586448 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5591c0dca6f0 0x5591c0dccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:09.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.377+0000 7f401e18a740 4 client.17574 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:20:09.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.377+0000 7f401e18a740 1 -- 172.21.15.16:0/1682586448 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5591c0cb0a10 con 0x5591c0dca6f0 2024-03-28T22:20:09.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.377+0000 7f401934a640 1 --2- 172.21.15.16:0/1682586448 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5591c0dca6f0 0x5591c0dccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:09.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.378+0000 7f3ffcff9640 10 client.17574 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:20:09.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.378+0000 7f401934a640 1 --2- 172.21.15.16:0/1682586448 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5591c0dca6f0 0x5591c0dccb10 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6cf96938a0f20bdb server_cookie=273381f43064180a in_seq=0 out_seq=0 2024-03-28T22:20:09.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.378+0000 7f3ffcff9640 1 -- 172.21.15.16:0/1682586448 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5591c0cb0a10 con 0x5591c0dca6f0 2024-03-28T22:20:09.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.378+0000 7f3ffcff9640 10 client.17574 handle_command_reply: tid=0 2024-03-28T22:20:09.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.378+0000 7f401e18a740 1 client.17574 shutdown 2024-03-28T22:20:09.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.378+0000 7f401e18a740 1 -- 172.21.15.16:0/1682586448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c0d52ba0 msgr2=0x5591c0d580c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:09.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.378+0000 7f401e18a740 1 --2- 172.21.15.16:0/1682586448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c0d52ba0 0x5591c0d580c0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f400c029e00 tx=0x7f400c0123f0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.378+0000 7f401e18a740 1 -- 172.21.15.16:0/1682586448 shutdown_connections 2024-03-28T22:20:09.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.378+0000 7f401e18a740 1 --2- 172.21.15.16:0/1682586448 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x5591c0dca6f0 0x5591c0dccb10 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.378+0000 7f401e18a740 1 --2- 172.21.15.16:0/1682586448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5591c0d52ba0 0x5591c0d580c0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.378+0000 7f3ffcff9640 0 client.17574 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:20:09.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.378+0000 7f401e18a740 1 --2- 172.21.15.16:0/1682586448 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5591c0d45860 0x5591c0d57b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.378+0000 7f401e18a740 1 --2- 172.21.15.16:0/1682586448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5591c0d44f40 0x5591c0d5ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.378+0000 7f401e18a740 1 -- 172.21.15.16:0/1682586448 >> 172.21.15.16:0/1682586448 conn(0x5591c0d3f9c0 msgr2=0x5591c0d4fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:09.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.378+0000 7f401e18a740 1 -- 172.21.15.16:0/1682586448 shutdown_connections 2024-03-28T22:20:09.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.378+0000 7f401e18a740 1 -- 172.21.15.16:0/1682586448 wait complete. 2024-03-28T22:20:09.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.378+0000 7f401e18a740 20 client.17574 trim_cache size 0 max 16384 2024-03-28T22:20:09.393 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:20:09.393 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T22:20:09.394 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": { 2024-03-28T22:20:09.394 INFO:teuthology.orchestra.run.smithi016.stdout: "fb2941ee-edb3-4281-b3c1-b3cc0a9bbfa7": { 2024-03-28T22:20:09.394 INFO:teuthology.orchestra.run.smithi016.stdout: "path": "/", 2024-03-28T22:20:09.394 INFO:teuthology.orchestra.run.smithi016.stdout: "tag": "fb2941ee-edb3-4281-b3c1-b3cc0a9bbfa7", 2024-03-28T22:20:09.395 INFO:teuthology.orchestra.run.smithi016.stdout: "options": "recursive,force" 2024-03-28T22:20:09.395 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:09.395 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:09.395 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:20:09.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.379+0000 7f401b5ae640 1 -- 172.21.15.16:0/2047433448 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f40000a1ef0 msgr2=0x7f40000a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:09.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.379+0000 7f401b5ae640 1 --2- 172.21.15.16:0/2047433448 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f40000a1ef0 0x7f40000a43b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f400c005fd0 tx=0x7f400c00ac20 comp rx=0 tx=0).stop 2024-03-28T22:20:09.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.379+0000 7f401b5ae640 1 -- 172.21.15.16:0/2047433448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40141005e0 msgr2=0x7f40141980c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:09.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.379+0000 7f401b5ae640 1 --2- 172.21.15.16:0/2047433448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40141005e0 0x7f40141980c0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f4010009230 tx=0x7f401002cbf0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.379+0000 7f401b5ae640 1 -- 172.21.15.16:0/2047433448 shutdown_connections 2024-03-28T22:20:09.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.379+0000 7f401b5ae640 1 --2- 172.21.15.16:0/2047433448 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f40000a1ef0 0x7f40000a43b0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.379+0000 7f401b5ae640 1 --2- 172.21.15.16:0/2047433448 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4014106680 0x7f401419c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.379+0000 7f401b5ae640 1 --2- 172.21.15.16:0/2047433448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4014100fa0 0x7f4014198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.379+0000 7f401b5ae640 1 --2- 172.21.15.16:0/2047433448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40141005e0 0x7f40141980c0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:09.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.379+0000 7f401b5ae640 1 -- 172.21.15.16:0/2047433448 >> 172.21.15.16:0/2047433448 conn(0x7f4014077770 msgr2=0x7f40140ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:09.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.379+0000 7f401b5ae640 1 -- 172.21.15.16:0/2047433448 shutdown_connections 2024-03-28T22:20:09.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:09.379+0000 7f401b5ae640 1 -- 172.21.15.16:0/2047433448 wait complete. 2024-03-28T22:20:09.398 INFO:tasks.cephfs.filesystem:scrub status for tag:fb2941ee-edb3-4281-b3c1-b3cc0a9bbfa7 - {'path': '/', 'tag': 'fb2941ee-edb3-4281-b3c1-b3cc0a9bbfa7', 'options': 'recursive,force'} 2024-03-28T22:20:10.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:10 smithi053 ceph-mon[35979]: pgmap v296: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-28T22:20:10.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:10 smithi053 ceph-mon[35979]: scrub queued for path: / 2024-03-28T22:20:10.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:10 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:20:10.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:10 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:20:10.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:10 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:20:10.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:10 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:20:10.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:10 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:20:10.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:10 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:20:10.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:10 smithi053 ceph-mon[35979]: scrub summary: active paths [/] 2024-03-28T22:20:10.547 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:10 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:20:10.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:10 smithi063 ceph-mon[36249]: pgmap v296: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-28T22:20:10.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:10 smithi063 ceph-mon[36249]: scrub queued for path: / 2024-03-28T22:20:10.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:10 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:20:10.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:10 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:20:10.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:10 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:20:10.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:10 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:20:10.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:10 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:20:10.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:10 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:20:10.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:10 smithi063 ceph-mon[36249]: scrub summary: active paths [/] 2024-03-28T22:20:10.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:10 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:20:10.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:10 smithi016 ceph-mon[32374]: pgmap v296: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-28T22:20:10.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:10 smithi016 ceph-mon[32374]: scrub queued for path: / 2024-03-28T22:20:10.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:10 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:20:10.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:10 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:20:10.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:10 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:20:10.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:10 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:20:10.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:10 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:20:10.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:10 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:20:10.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:10 smithi016 ceph-mon[32374]: scrub summary: active paths [/] 2024-03-28T22:20:10.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:10 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:20:11.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:11 smithi053 ceph-mon[35979]: scrub summary: idle+waiting paths [/] 2024-03-28T22:20:11.297 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:11 smithi053 ceph-mon[35979]: osdmap e96: 12 total, 12 up, 12 in 2024-03-28T22:20:11.680 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:11 smithi063 ceph-mon[36249]: scrub summary: idle+waiting paths [/] 2024-03-28T22:20:11.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:11 smithi063 ceph-mon[36249]: osdmap e96: 12 total, 12 up, 12 in 2024-03-28T22:20:11.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:11 smithi016 ceph-mon[32374]: scrub summary: idle+waiting paths [/] 2024-03-28T22:20:11.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:11 smithi016 ceph-mon[32374]: osdmap e96: 12 total, 12 up, 12 in 2024-03-28T22:20:12.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:12 smithi053 ceph-mon[35979]: pgmap v298: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-03-28T22:20:12.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:12 smithi063 ceph-mon[36249]: pgmap v298: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-03-28T22:20:12.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:12 smithi016 ceph-mon[32374]: pgmap v298: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-03-28T22:20:14.483 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:14 smithi063 ceph-mon[36249]: pgmap v299: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-03-28T22:20:14.486 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:14 smithi016 ceph-mon[32374]: pgmap v299: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-03-28T22:20:14.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:14 smithi053 ceph-mon[35979]: pgmap v299: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-03-28T22:20:16.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:16 smithi053 ceph-mon[35979]: pgmap v300: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-03-28T22:20:16.680 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:16 smithi063 ceph-mon[36249]: pgmap v300: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-03-28T22:20:16.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:16 smithi016 ceph-mon[32374]: pgmap v300: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-03-28T22:20:18.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:18 smithi053 ceph-mon[35979]: pgmap v301: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T22:20:18.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:18 smithi053 ceph-mon[35979]: osdmap e97: 12 total, 12 up, 12 in 2024-03-28T22:20:18.680 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:18 smithi063 ceph-mon[36249]: pgmap v301: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T22:20:18.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:18 smithi063 ceph-mon[36249]: osdmap e97: 12 total, 12 up, 12 in 2024-03-28T22:20:18.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:18 smithi016 ceph-mon[32374]: pgmap v301: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T22:20:18.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:18 smithi016 ceph-mon[32374]: osdmap e97: 12 total, 12 up, 12 in 2024-03-28T22:20:20.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:20 smithi053 ceph-mon[35979]: pgmap v303: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 247 B/s wr, 7 op/s 2024-03-28T22:20:20.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:20 smithi053 ceph-mon[35979]: scrub summary: idle 2024-03-28T22:20:20.680 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:20 smithi063 ceph-mon[36249]: pgmap v303: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 247 B/s wr, 7 op/s 2024-03-28T22:20:20.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:20 smithi063 ceph-mon[36249]: scrub summary: idle 2024-03-28T22:20:20.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:20 smithi016 ceph-mon[32374]: pgmap v303: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 247 B/s wr, 7 op/s 2024-03-28T22:20:20.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:20 smithi016 ceph-mon[32374]: scrub summary: idle 2024-03-28T22:20:21.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:21 smithi053 ceph-mon[35979]: scrub summary: idle 2024-03-28T22:20:21.680 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:21 smithi063 ceph-mon[36249]: scrub summary: idle 2024-03-28T22:20:21.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:21 smithi016 ceph-mon[32374]: scrub summary: idle 2024-03-28T22:20:22.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:22 smithi053 ceph-mon[35979]: pgmap v304: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T22:20:22.680 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:22 smithi063 ceph-mon[36249]: pgmap v304: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T22:20:22.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:22 smithi016 ceph-mon[32374]: pgmap v304: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T22:20:24.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:24 smithi053 ceph-mon[35979]: pgmap v305: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T22:20:24.680 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:24 smithi063 ceph-mon[36249]: pgmap v305: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T22:20:24.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:24 smithi016 ceph-mon[32374]: pgmap v305: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T22:20:26.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:26 smithi053 ceph-mon[35979]: pgmap v306: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T22:20:26.680 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:26 smithi063 ceph-mon[36249]: pgmap v306: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T22:20:26.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:26 smithi016 ceph-mon[32374]: pgmap v306: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-28T22:20:28.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:28 smithi053 ceph-mon[35979]: pgmap v307: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-28T22:20:28.680 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:28 smithi063 ceph-mon[36249]: pgmap v307: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-28T22:20:28.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:28 smithi016 ceph-mon[32374]: pgmap v307: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-28T22:20:30.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:30 smithi053 ceph-mon[35979]: pgmap v308: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-28T22:20:30.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:30 smithi063 ceph-mon[36249]: pgmap v308: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-28T22:20:30.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:30 smithi016 ceph-mon[32374]: pgmap v308: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-28T22:20:32.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:32 smithi053 ceph-mon[35979]: pgmap v309: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:20:32.680 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:32 smithi063 ceph-mon[36249]: pgmap v309: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:20:32.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:32 smithi016 ceph-mon[32374]: pgmap v309: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:20:34.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:34 smithi053 ceph-mon[35979]: pgmap v310: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T22:20:34.680 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:34 smithi063 ceph-mon[36249]: pgmap v310: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T22:20:34.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:34 smithi016 ceph-mon[32374]: pgmap v310: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T22:20:36.453 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-28T22:20:36.549 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:36 smithi053 ceph-mon[35979]: pgmap v311: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T22:20:36.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.569+0000 7f6a45b48640 1 -- 172.21.15.16:0/1035765378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a40100fb0 msgr2=0x7f6a4010d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:36.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.569+0000 7f6a45b48640 1 --2- 172.21.15.16:0/1035765378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a40100fb0 0x7f6a4010d4c0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f6a38009930 tx=0x7f6a3802bc30 comp rx=0 tx=0).stop 2024-03-28T22:20:36.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.569+0000 7f6a45b48640 1 -- 172.21.15.16:0/1035765378 shutdown_connections 2024-03-28T22:20:36.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.569+0000 7f6a45b48640 1 --2- 172.21.15.16:0/1035765378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a40100fb0 0x7f6a4010d4c0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:36.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.569+0000 7f6a45b48640 1 --2- 172.21.15.16:0/1035765378 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a401005f0 0x7f6a40100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:36.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.569+0000 7f6a45b48640 1 --2- 172.21.15.16:0/1035765378 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a40106690 0x7f6a40106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:36.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.569+0000 7f6a45b48640 1 -- 172.21.15.16:0/1035765378 >> 172.21.15.16:0/1035765378 conn(0x7f6a40077740 msgr2=0x7f6a40077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:36.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.569+0000 7f6a45b48640 1 -- 172.21.15.16:0/1035765378 shutdown_connections 2024-03-28T22:20:36.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.569+0000 7f6a45b48640 1 -- 172.21.15.16:0/1035765378 wait complete. 2024-03-28T22:20:36.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.569+0000 7f6a45b48640 1 Processor -- start 2024-03-28T22:20:36.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.570+0000 7f6a45b48640 1 -- start start 2024-03-28T22:20:36.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.570+0000 7f6a45b48640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a401005f0 0x7f6a40196e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:36.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.570+0000 7f6a45b48640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a40100fb0 0x7f6a40197370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:36.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.570+0000 7f6a45b48640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a40106690 0x7f6a4019b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:36.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.570+0000 7f6a45b48640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6a401109d0 con 0x7f6a40100fb0 2024-03-28T22:20:36.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.570+0000 7f6a45b48640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f6a40110850 con 0x7f6a40106690 2024-03-28T22:20:36.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.570+0000 7f6a45b48640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6a40110b50 con 0x7f6a401005f0 2024-03-28T22:20:36.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.570+0000 7f6a3effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a40100fb0 0x7f6a40197370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:36.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.570+0000 7f6a3effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a40100fb0 0x7f6a40197370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40882/0 (socket says 172.21.15.16:40882) 2024-03-28T22:20:36.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.570+0000 7f6a3effd640 1 -- 172.21.15.16:0/2924233813 learned_addr learned my addr 172.21.15.16:0/2924233813 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:36.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.570+0000 7f6a3ffff640 1 --2- 172.21.15.16:0/2924233813 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a40106690 0x7f6a4019b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:36.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.570+0000 7f6a3f7fe640 1 --2- 172.21.15.16:0/2924233813 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a401005f0 0x7f6a40196e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:36.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.570+0000 7f6a3effd640 1 -- 172.21.15.16:0/2924233813 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a401005f0 msgr2=0x7f6a40196e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:36.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.570+0000 7f6a3effd640 1 --2- 172.21.15.16:0/2924233813 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a401005f0 0x7f6a40196e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:36.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.570+0000 7f6a3effd640 1 -- 172.21.15.16:0/2924233813 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a40106690 msgr2=0x7f6a4019b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:36.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.570+0000 7f6a3effd640 1 --2- 172.21.15.16:0/2924233813 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a40106690 0x7f6a4019b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:36.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.570+0000 7f6a3effd640 1 -- 172.21.15.16:0/2924233813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a4019be20 con 0x7f6a40100fb0 2024-03-28T22:20:36.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.570+0000 7f6a3f7fe640 1 --2- 172.21.15.16:0/2924233813 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a401005f0 0x7f6a40196e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:20:36.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.570+0000 7f6a3effd640 1 --2- 172.21.15.16:0/2924233813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a40100fb0 0x7f6a40197370 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f6a2800c890 tx=0x7f6a2800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:36.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.571+0000 7f6a3cff9640 1 -- 172.21.15.16:0/2924233813 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a28004220 con 0x7f6a40100fb0 2024-03-28T22:20:36.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.571+0000 7f6a3cff9640 1 -- 172.21.15.16:0/2924233813 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a280043c0 con 0x7f6a40100fb0 2024-03-28T22:20:36.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.571+0000 7f6a45b48640 1 -- 172.21.15.16:0/2924233813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a4019c110 con 0x7f6a40100fb0 2024-03-28T22:20:36.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.571+0000 7f6a3cff9640 1 -- 172.21.15.16:0/2924233813 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a280049e0 con 0x7f6a40100fb0 2024-03-28T22:20:36.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.571+0000 7f6a45b48640 1 -- 172.21.15.16:0/2924233813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a401a39a0 con 0x7f6a40100fb0 2024-03-28T22:20:36.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.572+0000 7f6a45b48640 1 -- 172.21.15.16:0/2924233813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a4004fd00 con 0x7f6a40100fb0 2024-03-28T22:20:36.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.573+0000 7f6a3cff9640 1 -- 172.21.15.16:0/2924233813 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f6a280079b0 con 0x7f6a40100fb0 2024-03-28T22:20:36.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.573+0000 7f6a3cff9640 1 --2- 172.21.15.16:0/2924233813 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f6a2c0a1ee0 0x7f6a2c0a43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:36.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.573+0000 7f6a3cff9640 1 -- 172.21.15.16:0/2924233813 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8527+0+0 (secure 0 0 0) 0x7f6a280ce200 con 0x7f6a40100fb0 2024-03-28T22:20:36.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.573+0000 7f6a3f7fe640 1 --2- 172.21.15.16:0/2924233813 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f6a2c0a1ee0 0x7f6a2c0a43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:36.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.574+0000 7f6a3f7fe640 1 --2- 172.21.15.16:0/2924233813 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f6a2c0a1ee0 0x7f6a2c0a43a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6a40101730 tx=0x7f6a30009500 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:36.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.577+0000 7f6a3cff9640 1 -- 172.21.15.16:0/2924233813 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6a28014030 con 0x7f6a40100fb0 2024-03-28T22:20:36.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:36 smithi063 ceph-mon[36249]: pgmap v311: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T22:20:36.684 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:36 smithi016 ceph-mon[32374]: pgmap v311: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T22:20:36.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.781+0000 7f6a45b48640 1 -- 172.21.15.16:0/2924233813 --> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f6a40061b90 con 0x7f6a2c0a1ee0 2024-03-28T22:20:36.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.789+0000 7f6a3cff9640 1 -- 172.21.15.16:0/2924233813 <== mgr.15441 v2:172.21.15.16:6800/1819900910 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7f6a40061b90 con 0x7f6a2c0a1ee0 2024-03-28T22:20:36.791 INFO:teuthology.orchestra.run.smithi016.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-28T00:00:00", "created": "2024-03-28T22:10:35", "first": "2024-03-28T22:11:00", "last": "2024-03-28T22:20:00", "last_pruned": "2024-03-28T22:20:00", "created_count": 10, "pruned_count": 4, "active": true} 2024-03-28T22:20:36.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.791+0000 7f6a45b48640 1 -- 172.21.15.16:0/2924233813 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f6a2c0a1ee0 msgr2=0x7f6a2c0a43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:36.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.791+0000 7f6a45b48640 1 --2- 172.21.15.16:0/2924233813 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f6a2c0a1ee0 0x7f6a2c0a43a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6a40101730 tx=0x7f6a30009500 comp rx=0 tx=0).stop 2024-03-28T22:20:36.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.791+0000 7f6a45b48640 1 -- 172.21.15.16:0/2924233813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a40100fb0 msgr2=0x7f6a40197370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:36.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.791+0000 7f6a45b48640 1 --2- 172.21.15.16:0/2924233813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a40100fb0 0x7f6a40197370 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f6a2800c890 tx=0x7f6a2800cd50 comp rx=0 tx=0).stop 2024-03-28T22:20:36.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.791+0000 7f6a45b48640 1 -- 172.21.15.16:0/2924233813 shutdown_connections 2024-03-28T22:20:36.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.791+0000 7f6a45b48640 1 --2- 172.21.15.16:0/2924233813 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f6a2c0a1ee0 0x7f6a2c0a43a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:36.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.791+0000 7f6a45b48640 1 --2- 172.21.15.16:0/2924233813 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a40106690 0x7f6a4019b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:36.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.791+0000 7f6a45b48640 1 --2- 172.21.15.16:0/2924233813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a40100fb0 0x7f6a40197370 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:36.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.791+0000 7f6a45b48640 1 --2- 172.21.15.16:0/2924233813 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a401005f0 0x7f6a40196e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:36.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.791+0000 7f6a45b48640 1 -- 172.21.15.16:0/2924233813 >> 172.21.15.16:0/2924233813 conn(0x7f6a40077740 msgr2=0x7f6a400ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:36.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.791+0000 7f6a45b48640 1 -- 172.21.15.16:0/2924233813 shutdown_connections 2024-03-28T22:20:36.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.791+0000 7f6a45b48640 1 -- 172.21.15.16:0/2924233813 wait complete. 2024-03-28T22:20:36.802 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-28T22:20:36.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.907+0000 7ff83f22e640 1 -- 172.21.15.16:0/1964964576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff838100fd0 msgr2=0x7ff83810d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:36.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.907+0000 7ff83f22e640 1 --2- 172.21.15.16:0/1964964576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff838100fd0 0x7ff83810d4e0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7ff830009930 tx=0x7ff83002bc30 comp rx=0 tx=0).stop 2024-03-28T22:20:36.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.907+0000 7ff83f22e640 1 -- 172.21.15.16:0/1964964576 shutdown_connections 2024-03-28T22:20:36.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.907+0000 7ff83f22e640 1 --2- 172.21.15.16:0/1964964576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff838100fd0 0x7ff83810d4e0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:36.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.907+0000 7ff83f22e640 1 --2- 172.21.15.16:0/1964964576 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff838100610 0x7ff838100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:36.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.907+0000 7ff83f22e640 1 --2- 172.21.15.16:0/1964964576 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8381066b0 0x7ff838106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:36.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.907+0000 7ff83f22e640 1 -- 172.21.15.16:0/1964964576 >> 172.21.15.16:0/1964964576 conn(0x7ff838077770 msgr2=0x7ff838077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:36.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.907+0000 7ff83f22e640 1 -- 172.21.15.16:0/1964964576 shutdown_connections 2024-03-28T22:20:36.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.907+0000 7ff83f22e640 1 -- 172.21.15.16:0/1964964576 wait complete. 2024-03-28T22:20:36.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.908+0000 7ff83f22e640 1 Processor -- start 2024-03-28T22:20:36.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.908+0000 7ff83f22e640 1 -- start start 2024-03-28T22:20:36.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.908+0000 7ff83f22e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff838100610 0x7ff838196e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:36.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.908+0000 7ff83f22e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff838100fd0 0x7ff838197350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:36.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.908+0000 7ff83f22e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8381066b0 0x7ff83819b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:36.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.908+0000 7ff83f22e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff8381109f0 con 0x7ff838100fd0 2024-03-28T22:20:36.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.908+0000 7ff83f22e640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff838110870 con 0x7ff8381066b0 2024-03-28T22:20:36.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.908+0000 7ff83f22e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff838110b70 con 0x7ff838100610 2024-03-28T22:20:36.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.908+0000 7ff837fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff838100fd0 0x7ff838197350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:36.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.908+0000 7ff837fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff838100fd0 0x7ff838197350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40886/0 (socket says 172.21.15.16:40886) 2024-03-28T22:20:36.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.908+0000 7ff837fff640 1 -- 172.21.15.16:0/464343994 learned_addr learned my addr 172.21.15.16:0/464343994 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:36.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.908+0000 7ff83d7cb640 1 --2- 172.21.15.16:0/464343994 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8381066b0 0x7ff83819b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:36.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.908+0000 7ff83cfca640 1 --2- 172.21.15.16:0/464343994 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff838100610 0x7ff838196e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:36.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.908+0000 7ff837fff640 1 -- 172.21.15.16:0/464343994 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff838100610 msgr2=0x7ff838196e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:36.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.908+0000 7ff837fff640 1 --2- 172.21.15.16:0/464343994 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff838100610 0x7ff838196e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:36.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.908+0000 7ff837fff640 1 -- 172.21.15.16:0/464343994 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8381066b0 msgr2=0x7ff83819b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:36.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.908+0000 7ff837fff640 1 --2- 172.21.15.16:0/464343994 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8381066b0 0x7ff83819b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:36.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.908+0000 7ff837fff640 1 -- 172.21.15.16:0/464343994 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff83819be00 con 0x7ff838100fd0 2024-03-28T22:20:36.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.908+0000 7ff837fff640 1 --2- 172.21.15.16:0/464343994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff838100fd0 0x7ff838197350 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7ff82000c890 tx=0x7ff82000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:36.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.909+0000 7ff835ffb640 1 -- 172.21.15.16:0/464343994 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff820004220 con 0x7ff838100fd0 2024-03-28T22:20:36.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.909+0000 7ff835ffb640 1 -- 172.21.15.16:0/464343994 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8200043c0 con 0x7ff838100fd0 2024-03-28T22:20:36.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.909+0000 7ff83f22e640 1 -- 172.21.15.16:0/464343994 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff83819c0f0 con 0x7ff838100fd0 2024-03-28T22:20:36.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.909+0000 7ff835ffb640 1 -- 172.21.15.16:0/464343994 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8200049e0 con 0x7ff838100fd0 2024-03-28T22:20:36.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.909+0000 7ff83f22e640 1 -- 172.21.15.16:0/464343994 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8381a3980 con 0x7ff838100fd0 2024-03-28T22:20:36.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.910+0000 7ff83f22e640 1 -- 172.21.15.16:0/464343994 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff838101d50 con 0x7ff838100fd0 2024-03-28T22:20:36.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.911+0000 7ff835ffb640 1 -- 172.21.15.16:0/464343994 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7ff8200079b0 con 0x7ff838100fd0 2024-03-28T22:20:36.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.911+0000 7ff835ffb640 1 --2- 172.21.15.16:0/464343994 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff8240a1ee0 0x7ff8240a43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:36.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.911+0000 7ff835ffb640 1 -- 172.21.15.16:0/464343994 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8527+0+0 (secure 0 0 0) 0x7ff8200ce240 con 0x7ff838100fd0 2024-03-28T22:20:36.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.911+0000 7ff83cfca640 1 --2- 172.21.15.16:0/464343994 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff8240a1ee0 0x7ff8240a43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:36.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.912+0000 7ff83cfca640 1 --2- 172.21.15.16:0/464343994 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff8240a1ee0 0x7ff8240a43a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff838101750 tx=0x7ff828009500 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:36.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:36.916+0000 7ff835ffb640 1 -- 172.21.15.16:0/464343994 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff820014030 con 0x7ff838100fd0 2024-03-28T22:20:37.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:37.117+0000 7ff83f22e640 1 -- 172.21.15.16:0/464343994 --> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7ff838061b90 con 0x7ff8240a1ee0 2024-03-28T22:20:37.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:37.124+0000 7ff835ffb640 1 -- 172.21.15.16:0/464343994 <== mgr.15441 v2:172.21.15.16:6800/1819900910 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7ff838061b90 con 0x7ff8240a1ee0 2024-03-28T22:20:37.124 INFO:teuthology.orchestra.run.smithi016.stdout:/ 1m 6m3h 2024-03-28T22:20:37.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:37.126+0000 7ff83f22e640 1 -- 172.21.15.16:0/464343994 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff8240a1ee0 msgr2=0x7ff8240a43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:37.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:37.126+0000 7ff83f22e640 1 --2- 172.21.15.16:0/464343994 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff8240a1ee0 0x7ff8240a43a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff838101750 tx=0x7ff828009500 comp rx=0 tx=0).stop 2024-03-28T22:20:37.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:37.126+0000 7ff83f22e640 1 -- 172.21.15.16:0/464343994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff838100fd0 msgr2=0x7ff838197350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:37.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:37.126+0000 7ff83f22e640 1 --2- 172.21.15.16:0/464343994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff838100fd0 0x7ff838197350 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7ff82000c890 tx=0x7ff82000cd50 comp rx=0 tx=0).stop 2024-03-28T22:20:37.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:37.126+0000 7ff83f22e640 1 -- 172.21.15.16:0/464343994 shutdown_connections 2024-03-28T22:20:37.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:37.126+0000 7ff83f22e640 1 --2- 172.21.15.16:0/464343994 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7ff8240a1ee0 0x7ff8240a43a0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:37.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:37.126+0000 7ff83f22e640 1 --2- 172.21.15.16:0/464343994 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8381066b0 0x7ff83819b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:37.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:37.126+0000 7ff83f22e640 1 --2- 172.21.15.16:0/464343994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff838100fd0 0x7ff838197350 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:37.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:37.126+0000 7ff83f22e640 1 --2- 172.21.15.16:0/464343994 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff838100610 0x7ff838196e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:37.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:37.126+0000 7ff83f22e640 1 -- 172.21.15.16:0/464343994 >> 172.21.15.16:0/464343994 conn(0x7ff838077770 msgr2=0x7ff8380ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:37.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:37.126+0000 7ff83f22e640 1 -- 172.21.15.16:0/464343994 shutdown_connections 2024-03-28T22:20:37.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:37.126+0000 7ff83f22e640 1 -- 172.21.15.16:0/464343994 wait complete. 2024-03-28T22:20:37.137 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-03-28T22:20:37.296 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-03-28T22:20:38.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:38 smithi053 ceph-mon[35979]: pgmap v312: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:20:38.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:38 smithi053 ceph-mon[35979]: from='client.17580 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:20:38.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:38 smithi053 ceph-mon[35979]: from='client.17586 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:20:38.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:38 smithi063 ceph-mon[36249]: pgmap v312: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:20:38.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:38 smithi063 ceph-mon[36249]: from='client.17580 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:20:38.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:38 smithi063 ceph-mon[36249]: from='client.17586 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:20:38.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:38 smithi016 ceph-mon[32374]: pgmap v312: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:20:38.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:38 smithi016 ceph-mon[32374]: from='client.17580 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:20:38.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:38 smithi016 ceph-mon[32374]: from='client.17586 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:20:39.399 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:20:39.399 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:20:39.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.516+0000 7f382b47c640 1 -- 172.21.15.16:0/2965419495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38241009d0 msgr2=0x7f3824100db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:39.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.516+0000 7f382b47c640 1 --2- 172.21.15.16:0/2965419495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38241009d0 0x7f3824100db0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f3820009900 tx=0x7f382002bd10 comp rx=0 tx=0).stop 2024-03-28T22:20:39.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.516+0000 7f382b47c640 1 -- 172.21.15.16:0/2965419495 shutdown_connections 2024-03-28T22:20:39.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.516+0000 7f382b47c640 1 --2- 172.21.15.16:0/2965419495 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3824102b80 0x7f382410d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.516+0000 7f382b47c640 1 --2- 172.21.15.16:0/2965419495 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38241021e0 0x7f3824102640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.516+0000 7f382b47c640 1 --2- 172.21.15.16:0/2965419495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38241009d0 0x7f3824100db0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.516+0000 7f382b47c640 1 -- 172.21.15.16:0/2965419495 >> 172.21.15.16:0/2965419495 conn(0x7f3824077830 msgr2=0x7f3824077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:39.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.516+0000 7f382b47c640 1 -- 172.21.15.16:0/2965419495 shutdown_connections 2024-03-28T22:20:39.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.516+0000 7f382b47c640 1 -- 172.21.15.16:0/2965419495 wait complete. 2024-03-28T22:20:39.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.517+0000 7f382b47c640 1 Processor -- start 2024-03-28T22:20:39.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.517+0000 7f382b47c640 1 -- start start 2024-03-28T22:20:39.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.517+0000 7f382b47c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38241009d0 0x7f3824198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:39.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.517+0000 7f382b47c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f38241021e0 0x7f3824198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:39.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.517+0000 7f382b47c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3824102b80 0x7f382419c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:39.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.517+0000 7f382b47c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3824110b20 con 0x7f3824102b80 2024-03-28T22:20:39.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.517+0000 7f382b47c640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f38241109a0 con 0x7f38241009d0 2024-03-28T22:20:39.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.517+0000 7f382b47c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3824110ca0 con 0x7f38241021e0 2024-03-28T22:20:39.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.517+0000 7f3829a19640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3824102b80 0x7f382419c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:39.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.517+0000 7f3829218640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38241009d0 0x7f3824198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:39.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.517+0000 7f3829a19640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3824102b80 0x7f382419c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40904/0 (socket says 172.21.15.16:40904) 2024-03-28T22:20:39.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.517+0000 7f3829218640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38241009d0 0x7f3824198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:36862/0 (socket says 172.21.15.16:36862) 2024-03-28T22:20:39.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.517+0000 7f3829a19640 1 -- 172.21.15.16:0/274515842 learned_addr learned my addr 172.21.15.16:0/274515842 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:39.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.517+0000 7f3828a17640 1 --2- 172.21.15.16:0/274515842 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f38241021e0 0x7f3824198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:39.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.517+0000 7f3829218640 1 -- 172.21.15.16:0/274515842 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f38241021e0 msgr2=0x7f3824198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:39.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.517+0000 7f3829218640 1 --2- 172.21.15.16:0/274515842 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f38241021e0 0x7f3824198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.517+0000 7f3829218640 1 -- 172.21.15.16:0/274515842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3824102b80 msgr2=0x7f382419c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:39.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.517+0000 7f3829218640 1 --2- 172.21.15.16:0/274515842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3824102b80 0x7f382419c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.517+0000 7f3829218640 1 -- 172.21.15.16:0/274515842 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f382419d030 con 0x7f38241009d0 2024-03-28T22:20:39.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.517+0000 7f3829a19640 1 --2- 172.21.15.16:0/274515842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3824102b80 0x7f382419c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:20:39.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.518+0000 7f3829218640 1 --2- 172.21.15.16:0/274515842 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38241009d0 0x7f3824198040 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3820036040 tx=0x7f38200043b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:39.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.518+0000 7f381a7fc640 1 -- 172.21.15.16:0/274515842 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3820042020 con 0x7f38241009d0 2024-03-28T22:20:39.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.518+0000 7f382b47c640 1 -- 172.21.15.16:0/274515842 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f382419d2c0 con 0x7f38241009d0 2024-03-28T22:20:39.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.518+0000 7f381a7fc640 1 -- 172.21.15.16:0/274515842 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38200054e0 con 0x7f38241009d0 2024-03-28T22:20:39.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.518+0000 7f381a7fc640 1 -- 172.21.15.16:0/274515842 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3820003ad0 con 0x7f38241009d0 2024-03-28T22:20:39.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.518+0000 7f382b47c640 1 -- 172.21.15.16:0/274515842 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38241a4bb0 con 0x7f38241009d0 2024-03-28T22:20:39.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.520+0000 7f381a7fc640 1 -- 172.21.15.16:0/274515842 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f3820045020 con 0x7f38241009d0 2024-03-28T22:20:39.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.520+0000 7f382e05c740 1 Processor -- start 2024-03-28T22:20:39.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.521+0000 7f382e05c740 1 -- start start 2024-03-28T22:20:39.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.521+0000 7f382e05c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fae6c2efc0 0x55fae6c2f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:39.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.521+0000 7f382e05c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fae6c2f8e0 0x55fae6c3c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:39.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.521+0000 7f382e05c740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fae6c3cba0 0x55fae6c3ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:39.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.521+0000 7f382e05c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55fae6b93180 con 0x55fae6c2f8e0 2024-03-28T22:20:39.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.521+0000 7f382e05c740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55fae6b3bb70 con 0x55fae6c3cba0 2024-03-28T22:20:39.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.521+0000 7f382e05c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55fae6b8f220 con 0x55fae6c2efc0 2024-03-28T22:20:39.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.521+0000 7f381a7fc640 1 --2- 172.21.15.16:0/274515842 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f38100a1ef0 0x7f38100a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:39.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.521+0000 7f3829218640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fae6c2f8e0 0x55fae6c3c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:39.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.521+0000 7f3829218640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fae6c2f8e0 0x55fae6c3c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40906/0 (socket says 172.21.15.16:40906) 2024-03-28T22:20:39.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.521+0000 7f3829218640 1 -- 172.21.15.16:0/3230056445 learned_addr learned my addr 172.21.15.16:0/3230056445 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:39.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.521+0000 7f3828a17640 1 --2- 172.21.15.16:0/3230056445 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fae6c3cba0 0x55fae6c3ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:39.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.521+0000 7f3829a19640 1 --2- 172.21.15.16:0/274515842 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f38100a1ef0 0x7f38100a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:39.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.521+0000 7f3829a19640 1 --2- 172.21.15.16:0/3230056445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fae6c2efc0 0x55fae6c2f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:39.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.521+0000 7f381a7fc640 1 -- 172.21.15.16:0/274515842 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8527+0+0 (secure 0 0 0) 0x7f38200efd40 con 0x7f38241009d0 2024-03-28T22:20:39.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.521+0000 7f3829218640 1 -- 172.21.15.16:0/3230056445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fae6c2efc0 msgr2=0x55fae6c2f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:39.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.521+0000 7f3829218640 1 --2- 172.21.15.16:0/3230056445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fae6c2efc0 0x55fae6c2f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.521+0000 7f3829218640 1 -- 172.21.15.16:0/3230056445 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fae6c3cba0 msgr2=0x55fae6c3ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:39.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.521+0000 7f3829218640 1 --2- 172.21.15.16:0/3230056445 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fae6c3cba0 0x55fae6c3ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.521+0000 7f3829218640 1 -- 172.21.15.16:0/3230056445 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fae6b4f7c0 con 0x55fae6c2f8e0 2024-03-28T22:20:39.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.521+0000 7f3828a17640 1 --2- 172.21.15.16:0/3230056445 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fae6c3cba0 0x55fae6c3ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:20:39.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.522+0000 7f3829218640 1 --2- 172.21.15.16:0/3230056445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fae6c2f8e0 0x55fae6c3c660 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f38200ed5a0 tx=0x7f38200045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69abf7340b36c5ab server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:39.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.522+0000 7f3829a19640 1 --2- 172.21.15.16:0/274515842 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f38100a1ef0 0x7f38100a43b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f381c007350 tx=0x7f381c0072c0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:39.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.522+0000 7f380f7fe640 1 -- 172.21.15.16:0/3230056445 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38200bc7e0 con 0x55fae6c2f8e0 2024-03-28T22:20:39.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.522+0000 7f380f7fe640 1 -- 172.21.15.16:0/3230056445 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38200c0930 con 0x55fae6c2f8e0 2024-03-28T22:20:39.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.522+0000 7f380f7fe640 1 -- 172.21.15.16:0/3230056445 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382002cbd0 con 0x55fae6c2f8e0 2024-03-28T22:20:39.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.522+0000 7f382e05c740 1 -- 172.21.15.16:0/3230056445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fae6c2f8e0 msgr2=0x55fae6c3c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:39.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.522+0000 7f382e05c740 1 --2- 172.21.15.16:0/3230056445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fae6c2f8e0 0x55fae6c3c660 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f38200ed5a0 tx=0x7f38200045e0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.522+0000 7f382e05c740 1 -- 172.21.15.16:0/3230056445 shutdown_connections 2024-03-28T22:20:39.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.522+0000 7f382e05c740 1 --2- 172.21.15.16:0/3230056445 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fae6c3cba0 0x55fae6c3ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.522+0000 7f382e05c740 1 --2- 172.21.15.16:0/3230056445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fae6c2f8e0 0x55fae6c3c660 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.522+0000 7f382e05c740 1 --2- 172.21.15.16:0/3230056445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fae6c2efc0 0x55fae6c2f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.522+0000 7f382e05c740 1 -- 172.21.15.16:0/3230056445 >> 172.21.15.16:0/3230056445 conn(0x55fae6c299c0 msgr2=0x55fae6c29da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:39.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.522+0000 7f382e05c740 1 -- 172.21.15.16:0/3230056445 shutdown_connections 2024-03-28T22:20:39.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.523+0000 7f382e05c740 1 -- 172.21.15.16:0/3230056445 wait complete. 2024-03-28T22:20:39.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.523+0000 7f382e05c740 1 -- start start 2024-03-28T22:20:39.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.523+0000 7f382e05c740 1 Processor -- start 2024-03-28T22:20:39.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.523+0000 7f382e05c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fae6c2efc0 0x55fae6c39c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:39.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.523+0000 7f382e05c740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fae6c2f8e0 0x55fae6c3a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:39.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.523+0000 7f382e05c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fae6c3cba0 0x55fae6c3a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:39.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.523+0000 7f382e05c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55fae6c3f720 con 0x55fae6c3cba0 2024-03-28T22:20:39.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.523+0000 7f382e05c740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55fae6b314a0 con 0x55fae6c2f8e0 2024-03-28T22:20:39.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.523+0000 7f382e05c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55fae6c3f8a0 con 0x55fae6c2efc0 2024-03-28T22:20:39.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.523+0000 7f382e05c740 10 client.0 _ll_register_callbacks cb 0x55fae6b34d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:20:39.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.523+0000 7f3829a19640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fae6c3cba0 0x55fae6c3a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:39.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.524+0000 7f3829a19640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fae6c3cba0 0x55fae6c3a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40908/0 (socket says 172.21.15.16:40908) 2024-03-28T22:20:39.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.524+0000 7f3829a19640 1 -- 172.21.15.16:0/1070067029 learned_addr learned my addr 172.21.15.16:0/1070067029 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:39.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.524+0000 7f3828a17640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fae6c2f8e0 0x55fae6c3a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:39.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.524+0000 7f3829218640 1 --2- 172.21.15.16:0/1070067029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fae6c2efc0 0x55fae6c39c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:39.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.524+0000 7f3829a19640 1 -- 172.21.15.16:0/1070067029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fae6c2efc0 msgr2=0x55fae6c39c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:39.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.524+0000 7f3829a19640 1 --2- 172.21.15.16:0/1070067029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fae6c2efc0 0x55fae6c39c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.524+0000 7f3829a19640 1 -- 172.21.15.16:0/1070067029 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fae6c2f8e0 msgr2=0x55fae6c3a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:39.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.524+0000 7f3829a19640 1 --2- 172.21.15.16:0/1070067029 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fae6c2f8e0 0x55fae6c3a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.524+0000 7f3829a19640 1 -- 172.21.15.16:0/1070067029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fae69717d0 con 0x55fae6c3cba0 2024-03-28T22:20:39.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.524+0000 7f3828a17640 1 --2- 172.21.15.16:0/1070067029 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fae6c2f8e0 0x55fae6c3a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:20:39.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.524+0000 7f3829a19640 1 --2- 172.21.15.16:0/1070067029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fae6c3cba0 0x55fae6c3a6b0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f381c0126d0 tx=0x7f381c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bda14dfb5b8b0ce7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:39.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.524+0000 7f380cff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:20:39.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.524+0000 7f380cff9640 1 -- 172.21.15.16:0/1070067029 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f381c024030 con 0x55fae6c3cba0 2024-03-28T22:20:39.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.524+0000 7f380cff9640 1 -- 172.21.15.16:0/1070067029 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8527+0+0 (secure 0 0 0) 0x7f381c020750 con 0x55fae6c3cba0 2024-03-28T22:20:39.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.524+0000 7f382e05c740 1 -- 172.21.15.16:0/1070067029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38200c0930 con 0x55fae6c3cba0 2024-03-28T22:20:39.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.524+0000 7f380cff9640 1 -- 172.21.15.16:0/1070067029 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f381c020950 con 0x55fae6c3cba0 2024-03-28T22:20:39.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.524+0000 7f382e05c740 10 client.17598 fetch_fsmap learned FSMap version 27 2024-03-28T22:20:39.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.524+0000 7f382e05c740 1 -- 172.21.15.16:0/1070067029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fae68a73f0 con 0x55fae6c3cba0 2024-03-28T22:20:39.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.525+0000 7f380cff9640 1 -- 172.21.15.16:0/1070067029 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38200c0930 con 0x55fae6c3cba0 2024-03-28T22:20:39.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.525+0000 7f380cff9640 1 -- 172.21.15.16:0/1070067029 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f381c01d050 con 0x55fae6c3cba0 2024-03-28T22:20:39.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.525+0000 7f382e05c740 10 client.17598 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:20:39.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.525+0000 7f382e05c740 10 client.17598 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:20:39.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.525+0000 7f382e05c740 20 client.17598 populate_metadata read hostname 'smithi016' 2024-03-28T22:20:39.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.525+0000 7f382e05c740 1 --2- 172.21.15.16:0/1070067029 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55fae6c4cd30 0x55fae6c4f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:39.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.525+0000 7f382e05c740 4 client.17598 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:20:39.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.525+0000 7f382e05c740 1 -- 172.21.15.16:0/1070067029 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fae6ba06c0 con 0x55fae6c4cd30 2024-03-28T22:20:39.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.525+0000 7f3829218640 1 --2- 172.21.15.16:0/1070067029 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55fae6c4cd30 0x55fae6c4f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:39.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.525+0000 7f380cff9640 10 client.17598 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:20:39.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.525+0000 7f3829218640 1 --2- 172.21.15.16:0/1070067029 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55fae6c4cd30 0x55fae6c4f150 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9747d250fb64a522 server_cookie=39bba6b6b68e946b in_seq=0 out_seq=0 2024-03-28T22:20:39.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.526+0000 7f380cff9640 1 -- 172.21.15.16:0/1070067029 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fae6ba06c0 con 0x55fae6c4cd30 2024-03-28T22:20:39.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.526+0000 7f380cff9640 10 client.17598 handle_command_reply: tid=0 2024-03-28T22:20:39.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.526+0000 7f382e05c740 1 client.17598 shutdown 2024-03-28T22:20:39.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.526+0000 7f382e05c740 1 -- 172.21.15.16:0/1070067029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fae6c3cba0 msgr2=0x55fae6c3a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:39.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.526+0000 7f382e05c740 1 --2- 172.21.15.16:0/1070067029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fae6c3cba0 0x55fae6c3a6b0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f381c0126d0 tx=0x7f381c004a40 comp rx=0 tx=0).stop 2024-03-28T22:20:39.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.526+0000 7f382e05c740 1 -- 172.21.15.16:0/1070067029 shutdown_connections 2024-03-28T22:20:39.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.526+0000 7f382e05c740 1 --2- 172.21.15.16:0/1070067029 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55fae6c4cd30 0x55fae6c4f150 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.526+0000 7f382e05c740 1 --2- 172.21.15.16:0/1070067029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fae6c3cba0 0x55fae6c3a6b0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.526+0000 7f382e05c740 1 --2- 172.21.15.16:0/1070067029 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fae6c2f8e0 0x55fae6c3a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.526+0000 7f380cff9640 0 client.17598 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:20:39.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.526+0000 7f382e05c740 1 --2- 172.21.15.16:0/1070067029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fae6c2efc0 0x55fae6c39c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.526+0000 7f382e05c740 1 -- 172.21.15.16:0/1070067029 >> 172.21.15.16:0/1070067029 conn(0x55fae6c299c0 msgr2=0x55fae6c2b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:39.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.526+0000 7f382e05c740 1 -- 172.21.15.16:0/1070067029 shutdown_connections 2024-03-28T22:20:39.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.526+0000 7f382e05c740 1 -- 172.21.15.16:0/1070067029 wait complete. 2024-03-28T22:20:39.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.527+0000 7f382e05c740 20 client.17598 trim_cache size 0 max 16384 2024-03-28T22:20:39.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.544+0000 7f382e05c740 1 Processor -- start 2024-03-28T22:20:39.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.544+0000 7f382e05c740 1 -- start start 2024-03-28T22:20:39.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.544+0000 7f382e05c740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fae6c2efc0 0x55fae6c39510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:39.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.544+0000 7f382e05c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fae6c2f8e0 0x55fae6c484c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:39.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.544+0000 7f382e05c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fae6c3cba0 0x55fae6c3ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:39.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.544+0000 7f382e05c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f38241ada00 con 0x55fae6c3cba0 2024-03-28T22:20:39.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.544+0000 7f382e05c740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f38241ab2d0 con 0x55fae6c2efc0 2024-03-28T22:20:39.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.544+0000 7f382e05c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f38241a9140 con 0x55fae6c2f8e0 2024-03-28T22:20:39.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.544+0000 7f3829a19640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fae6c3cba0 0x55fae6c3ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:39.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.544+0000 7f3829a19640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fae6c3cba0 0x55fae6c3ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40914/0 (socket says 172.21.15.16:40914) 2024-03-28T22:20:39.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.544+0000 7f3829a19640 1 -- 172.21.15.16:0/3664930648 learned_addr learned my addr 172.21.15.16:0/3664930648 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:39.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.544+0000 7f3829218640 1 --2- 172.21.15.16:0/3664930648 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fae6c2efc0 0x55fae6c39510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:39.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.545+0000 7f3828a17640 1 --2- 172.21.15.16:0/3664930648 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fae6c2f8e0 0x55fae6c484c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:39.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.545+0000 7f3829a19640 1 -- 172.21.15.16:0/3664930648 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fae6c2f8e0 msgr2=0x55fae6c484c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:39.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.545+0000 7f3829a19640 1 --2- 172.21.15.16:0/3664930648 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fae6c2f8e0 0x55fae6c484c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.545+0000 7f3829a19640 1 -- 172.21.15.16:0/3664930648 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fae6c2efc0 msgr2=0x55fae6c39510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:39.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.545+0000 7f3829a19640 1 --2- 172.21.15.16:0/3664930648 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fae6c2efc0 0x55fae6c39510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.545+0000 7f3829a19640 1 -- 172.21.15.16:0/3664930648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fae68670f0 con 0x55fae6c3cba0 2024-03-28T22:20:39.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.545+0000 7f3828a17640 1 --2- 172.21.15.16:0/3664930648 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fae6c2f8e0 0x55fae6c484c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:20:39.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.545+0000 7f3829a19640 1 --2- 172.21.15.16:0/3664930648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fae6c3cba0 0x55fae6c3ba60 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f381c007e90 tx=0x7f381c014660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31f2f35942c1dae8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:39.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.545+0000 7f380effd640 1 -- 172.21.15.16:0/3664930648 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f381c024040 con 0x55fae6c3cba0 2024-03-28T22:20:39.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.545+0000 7f380effd640 1 -- 172.21.15.16:0/3664930648 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f381c037070 con 0x55fae6c3cba0 2024-03-28T22:20:39.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.545+0000 7f380effd640 1 -- 172.21.15.16:0/3664930648 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f381c004cf0 con 0x55fae6c3cba0 2024-03-28T22:20:39.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.546+0000 7f382e05c740 1 -- 172.21.15.16:0/3664930648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fae6c3cba0 msgr2=0x55fae6c3ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:39.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.546+0000 7f382e05c740 1 --2- 172.21.15.16:0/3664930648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fae6c3cba0 0x55fae6c3ba60 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f381c007e90 tx=0x7f381c014660 comp rx=0 tx=0).stop 2024-03-28T22:20:39.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.546+0000 7f382e05c740 1 -- 172.21.15.16:0/3664930648 shutdown_connections 2024-03-28T22:20:39.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.546+0000 7f382e05c740 1 --2- 172.21.15.16:0/3664930648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fae6c3cba0 0x55fae6c3ba60 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.546+0000 7f382e05c740 1 --2- 172.21.15.16:0/3664930648 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fae6c2f8e0 0x55fae6c484c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.546+0000 7f382e05c740 1 --2- 172.21.15.16:0/3664930648 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fae6c2efc0 0x55fae6c39510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.546+0000 7f382e05c740 1 -- 172.21.15.16:0/3664930648 >> 172.21.15.16:0/3664930648 conn(0x55fae6c299c0 msgr2=0x55fae6c3b350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:39.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.546+0000 7f382e05c740 1 -- 172.21.15.16:0/3664930648 shutdown_connections 2024-03-28T22:20:39.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.546+0000 7f382e05c740 1 -- 172.21.15.16:0/3664930648 wait complete. 2024-03-28T22:20:39.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.546+0000 7f382e05c740 1 -- start start 2024-03-28T22:20:39.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.546+0000 7f382e05c740 1 Processor -- start 2024-03-28T22:20:39.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.546+0000 7f382e05c740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fae6c2efc0 0x55fae6c40380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:39.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.546+0000 7f382e05c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fae6c2f8e0 0x55fae6c408c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:39.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.546+0000 7f382e05c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fae6c3cba0 0x55fae6c40e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:39.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.546+0000 7f382e05c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f38241a9790 con 0x55fae6c2f8e0 2024-03-28T22:20:39.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.546+0000 7f382e05c740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f38241aac00 con 0x55fae6c2efc0 2024-03-28T22:20:39.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.546+0000 7f382e05c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f38241a9e60 con 0x55fae6c3cba0 2024-03-28T22:20:39.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.546+0000 7f382e05c740 10 client.0 _ll_register_callbacks cb 0x7f381c014850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:20:39.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.547+0000 7f3828a17640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fae6c2f8e0 0x55fae6c408c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:39.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.547+0000 7f3828a17640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fae6c2f8e0 0x55fae6c408c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40920/0 (socket says 172.21.15.16:40920) 2024-03-28T22:20:39.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.547+0000 7f3828a17640 1 -- 172.21.15.16:0/3325246771 learned_addr learned my addr 172.21.15.16:0/3325246771 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:39.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.547+0000 7f3829218640 1 --2- 172.21.15.16:0/3325246771 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fae6c2efc0 0x55fae6c40380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:39.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.547+0000 7f3828a17640 1 -- 172.21.15.16:0/3325246771 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fae6c3cba0 msgr2=0x55fae6c40e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:39.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.547+0000 7f3829a19640 1 --2- 172.21.15.16:0/3325246771 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fae6c3cba0 0x55fae6c40e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:39.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.547+0000 7f3828a17640 1 --2- 172.21.15.16:0/3325246771 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fae6c3cba0 0x55fae6c40e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.547+0000 7f3828a17640 1 -- 172.21.15.16:0/3325246771 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fae6c2efc0 msgr2=0x55fae6c40380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:39.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.547+0000 7f3828a17640 1 --2- 172.21.15.16:0/3325246771 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fae6c2efc0 0x55fae6c40380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.547+0000 7f3828a17640 1 -- 172.21.15.16:0/3325246771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fae6cb2050 con 0x55fae6c2f8e0 2024-03-28T22:20:39.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.547+0000 7f3829a19640 1 --2- 172.21.15.16:0/3325246771 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fae6c3cba0 0x55fae6c40e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:20:39.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.547+0000 7f3828a17640 1 --2- 172.21.15.16:0/3325246771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fae6c2f8e0 0x55fae6c408c0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x55fae6c2ab50 tx=0x7f381400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2988ce27034097b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:39.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.547+0000 7f380cff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:20:39.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.547+0000 7f380cff9640 1 -- 172.21.15.16:0/3325246771 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f381401d410 con 0x55fae6c2f8e0 2024-03-28T22:20:39.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.547+0000 7f380cff9640 1 -- 172.21.15.16:0/3325246771 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8527+0+0 (secure 0 0 0) 0x7f381401d5b0 con 0x55fae6c2f8e0 2024-03-28T22:20:39.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.547+0000 7f382e05c740 1 -- 172.21.15.16:0/3325246771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f381c037070 con 0x55fae6c2f8e0 2024-03-28T22:20:39.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.547+0000 7f380cff9640 1 -- 172.21.15.16:0/3325246771 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f381402b050 con 0x55fae6c2f8e0 2024-03-28T22:20:39.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.547+0000 7f382e05c740 10 client.17604 fetch_fsmap learned FSMap version 27 2024-03-28T22:20:39.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.547+0000 7f382e05c740 1 -- 172.21.15.16:0/3325246771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fae6cb32e0 con 0x55fae6c2f8e0 2024-03-28T22:20:39.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.548+0000 7f380cff9640 1 -- 172.21.15.16:0/3325246771 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f381c037070 con 0x55fae6c2f8e0 2024-03-28T22:20:39.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.548+0000 7f380cff9640 1 -- 172.21.15.16:0/3325246771 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f38140268a0 con 0x55fae6c2f8e0 2024-03-28T22:20:39.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.548+0000 7f382e05c740 10 client.17604 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:20:39.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.548+0000 7f382e05c740 10 client.17604 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:20:39.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.548+0000 7f382e05c740 20 client.17604 populate_metadata read hostname 'smithi016' 2024-03-28T22:20:39.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.548+0000 7f382e05c740 1 --2- 172.21.15.16:0/3325246771 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55fae6cb3b10 0x55fae6cb5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:39.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.548+0000 7f382e05c740 4 client.17604 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:20:39.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.548+0000 7f382e05c740 1 -- 172.21.15.16:0/3325246771 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fae6b9aa10 con 0x55fae6cb3b10 2024-03-28T22:20:39.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.548+0000 7f3829218640 1 --2- 172.21.15.16:0/3325246771 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55fae6cb3b10 0x55fae6cb5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:39.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.548+0000 7f380cff9640 10 client.17604 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:20:39.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.548+0000 7f3829218640 1 --2- 172.21.15.16:0/3325246771 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55fae6cb3b10 0x55fae6cb5f30 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7762bb20884d7b99 server_cookie=1d3b873fe96b3823 in_seq=0 out_seq=0 2024-03-28T22:20:39.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.548+0000 7f380cff9640 1 -- 172.21.15.16:0/3325246771 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55fae6b9aa10 con 0x55fae6cb3b10 2024-03-28T22:20:39.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.548+0000 7f380cff9640 10 client.17604 handle_command_reply: tid=0 2024-03-28T22:20:39.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.549+0000 7f382e05c740 1 client.17604 shutdown 2024-03-28T22:20:39.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.549+0000 7f382e05c740 1 -- 172.21.15.16:0/3325246771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fae6c2f8e0 msgr2=0x55fae6c408c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:39.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.549+0000 7f382e05c740 1 --2- 172.21.15.16:0/3325246771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fae6c2f8e0 0x55fae6c408c0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x55fae6c2ab50 tx=0x7f381400f790 comp rx=0 tx=0).stop 2024-03-28T22:20:39.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.549+0000 7f382e05c740 1 -- 172.21.15.16:0/3325246771 shutdown_connections 2024-03-28T22:20:39.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.549+0000 7f382e05c740 1 --2- 172.21.15.16:0/3325246771 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x55fae6cb3b10 0x55fae6cb5f30 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.549+0000 7f382e05c740 1 --2- 172.21.15.16:0/3325246771 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fae6c3cba0 0x55fae6c40e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.549+0000 7f382e05c740 1 --2- 172.21.15.16:0/3325246771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fae6c2f8e0 0x55fae6c408c0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.549+0000 7f382e05c740 1 --2- 172.21.15.16:0/3325246771 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fae6c2efc0 0x55fae6c40380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.549+0000 7f380cff9640 0 client.17604 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:20:39.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.549+0000 7f382e05c740 1 -- 172.21.15.16:0/3325246771 >> 172.21.15.16:0/3325246771 conn(0x55fae6c299c0 msgr2=0x55fae6c3a480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:39.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.549+0000 7f382e05c740 1 -- 172.21.15.16:0/3325246771 shutdown_connections 2024-03-28T22:20:39.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.549+0000 7f382e05c740 1 -- 172.21.15.16:0/3325246771 wait complete. 2024-03-28T22:20:39.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.549+0000 7f382e05c740 20 client.17604 trim_cache size 0 max 16384 2024-03-28T22:20:39.560 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:20:39.560 INFO:teuthology.orchestra.run.smithi016.stdout: "status": "no active scrubs running", 2024-03-28T22:20:39.560 INFO:teuthology.orchestra.run.smithi016.stdout: "scrubs": {} 2024-03-28T22:20:39.561 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:20:39.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.549+0000 7f382b47c640 1 -- 172.21.15.16:0/274515842 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f38100a1ef0 msgr2=0x7f38100a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:39.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.549+0000 7f382b47c640 1 --2- 172.21.15.16:0/274515842 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f38100a1ef0 0x7f38100a43b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f381c007350 tx=0x7f381c0072c0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.549+0000 7f382b47c640 1 -- 172.21.15.16:0/274515842 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38241009d0 msgr2=0x7f3824198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:39.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.549+0000 7f382b47c640 1 --2- 172.21.15.16:0/274515842 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38241009d0 0x7f3824198040 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3820036040 tx=0x7f38200043b0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.549+0000 7f382b47c640 1 -- 172.21.15.16:0/274515842 shutdown_connections 2024-03-28T22:20:39.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.549+0000 7f382b47c640 1 --2- 172.21.15.16:0/274515842 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f38100a1ef0 0x7f38100a43b0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.549+0000 7f382b47c640 1 --2- 172.21.15.16:0/274515842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3824102b80 0x7f382419c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.549+0000 7f382b47c640 1 --2- 172.21.15.16:0/274515842 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f38241021e0 0x7f3824198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.549+0000 7f382b47c640 1 --2- 172.21.15.16:0/274515842 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38241009d0 0x7f3824198040 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.549+0000 7f382b47c640 1 -- 172.21.15.16:0/274515842 >> 172.21.15.16:0/274515842 conn(0x7f3824077830 msgr2=0x7f382406a0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:39.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.550+0000 7f382b47c640 1 -- 172.21.15.16:0/274515842 shutdown_connections 2024-03-28T22:20:39.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.550+0000 7f382b47c640 1 -- 172.21.15.16:0/274515842 wait complete. 2024-03-28T22:20:39.563 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:20:39.564 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:20:39.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.667+0000 7f2c17b8c640 1 -- 172.21.15.16:0/75932174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c10104d70 msgr2=0x7f2c10105150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:39.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.667+0000 7f2c17b8c640 1 --2- 172.21.15.16:0/75932174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c10104d70 0x7f2c10105150 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c009890 tx=0x7f2c0c02bb70 comp rx=0 tx=0).stop 2024-03-28T22:20:39.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.667+0000 7f2c17b8c640 1 -- 172.21.15.16:0/75932174 shutdown_connections 2024-03-28T22:20:39.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.667+0000 7f2c17b8c640 1 --2- 172.21.15.16:0/75932174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c101008e0 0x7f2c1010cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.667+0000 7f2c17b8c640 1 --2- 172.21.15.16:0/75932174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c100fff40 0x7f2c101003a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.667+0000 7f2c17b8c640 1 --2- 172.21.15.16:0/75932174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c10104d70 0x7f2c10105150 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.667+0000 7f2c17b8c640 1 -- 172.21.15.16:0/75932174 >> 172.21.15.16:0/75932174 conn(0x7f2c10076aa0 msgr2=0x7f2c10076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:39.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.667+0000 7f2c17b8c640 1 -- 172.21.15.16:0/75932174 shutdown_connections 2024-03-28T22:20:39.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.667+0000 7f2c17b8c640 1 -- 172.21.15.16:0/75932174 wait complete. 2024-03-28T22:20:39.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.667+0000 7f2c17b8c640 1 Processor -- start 2024-03-28T22:20:39.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.667+0000 7f2c17b8c640 1 -- start start 2024-03-28T22:20:39.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.668+0000 7f2c17b8c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c100fff40 0x7f2c10198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:39.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.668+0000 7f2c17b8c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c101008e0 0x7f2c10198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:39.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.668+0000 7f2c17b8c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c10104d70 0x7f2c1019c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:39.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.668+0000 7f2c17b8c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2c1010f990 con 0x7f2c101008e0 2024-03-28T22:20:39.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.668+0000 7f2c17b8c640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f2c1010f810 con 0x7f2c10104d70 2024-03-28T22:20:39.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.668+0000 7f2c17b8c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2c1010fb10 con 0x7f2c100fff40 2024-03-28T22:20:39.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.668+0000 7f2c15127640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c101008e0 0x7f2c10198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:39.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.668+0000 7f2c15127640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c101008e0 0x7f2c10198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40934/0 (socket says 172.21.15.16:40934) 2024-03-28T22:20:39.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.668+0000 7f2c15127640 1 -- 172.21.15.16:0/85712313 learned_addr learned my addr 172.21.15.16:0/85712313 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:39.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.668+0000 7f2c16129640 1 --2- 172.21.15.16:0/85712313 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c10104d70 0x7f2c1019c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:39.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.668+0000 7f2c15928640 1 --2- 172.21.15.16:0/85712313 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c100fff40 0x7f2c10198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:39.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.668+0000 7f2c15127640 1 -- 172.21.15.16:0/85712313 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c100fff40 msgr2=0x7f2c10198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:39.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.668+0000 7f2c15127640 1 --2- 172.21.15.16:0/85712313 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c100fff40 0x7f2c10198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.668+0000 7f2c15127640 1 -- 172.21.15.16:0/85712313 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c10104d70 msgr2=0x7f2c1019c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:39.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.668+0000 7f2c15127640 1 --2- 172.21.15.16:0/85712313 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c10104d70 0x7f2c1019c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.668+0000 7f2c15127640 1 -- 172.21.15.16:0/85712313 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c1019d0d0 con 0x7f2c101008e0 2024-03-28T22:20:39.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.668+0000 7f2c15127640 1 --2- 172.21.15.16:0/85712313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c101008e0 0x7f2c10198670 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f2c0000c970 tx=0x7f2c0000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:39.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.668+0000 7f2c06ffd640 1 -- 172.21.15.16:0/85712313 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c00007bb0 con 0x7f2c101008e0 2024-03-28T22:20:39.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.668+0000 7f2c17b8c640 1 -- 172.21.15.16:0/85712313 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c1019d3c0 con 0x7f2c101008e0 2024-03-28T22:20:39.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.668+0000 7f2c06ffd640 1 -- 172.21.15.16:0/85712313 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c00007d50 con 0x7f2c101008e0 2024-03-28T22:20:39.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.668+0000 7f2c06ffd640 1 -- 172.21.15.16:0/85712313 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c0000f450 con 0x7f2c101008e0 2024-03-28T22:20:39.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.668+0000 7f2c17b8c640 1 -- 172.21.15.16:0/85712313 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c101a4c50 con 0x7f2c101008e0 2024-03-28T22:20:39.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.669+0000 7f2c17b8c640 1 -- 172.21.15.16:0/85712313 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c10068a20 con 0x7f2c101008e0 2024-03-28T22:20:39.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.670+0000 7f2c06ffd640 1 -- 172.21.15.16:0/85712313 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f2c00004fe0 con 0x7f2c101008e0 2024-03-28T22:20:39.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.671+0000 7f2c06ffd640 1 --2- 172.21.15.16:0/85712313 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f2bfc0a1ef0 0x7f2bfc0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:39.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.671+0000 7f2c06ffd640 1 -- 172.21.15.16:0/85712313 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8527+0+0 (secure 0 0 0) 0x7f2c000ce3d0 con 0x7f2c101008e0 2024-03-28T22:20:39.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.671+0000 7f2c15928640 1 --2- 172.21.15.16:0/85712313 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f2bfc0a1ef0 0x7f2bfc0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:39.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.671+0000 7f2c15928640 1 --2- 172.21.15.16:0/85712313 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f2bfc0a1ef0 0x7f2bfc0a43b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c0095f0 tx=0x7f2c0c009580 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:39.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.675+0000 7f2c06ffd640 1 -- 172.21.15.16:0/85712313 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2c00014030 con 0x7f2c101008e0 2024-03-28T22:20:39.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.894+0000 7f2c17b8c640 1 -- 172.21.15.16:0/85712313 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c10199430 con 0x7f2c101008e0 2024-03-28T22:20:39.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.894+0000 7f2c06ffd640 1 -- 172.21.15.16:0/85712313 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11851 (secure 0 0 0) 0x7f2c10199430 con 0x7f2c101008e0 2024-03-28T22:20:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:20:39.895 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/1644581227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1644581227},{"type":"v1","addr":"172.21.15.16:6839","nonce":1644581227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/2593062669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2593062669},{"type":"v1","addr":"172.21.15.16:6841","nonce":2593062669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:08:29.672976+0000","modified":"2024-03-28T22:14:44.922044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24566,"mds_1":24524,"mds_2":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24466":{"gid":24466,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/247249857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":247249857},{"type":"v1","addr":"172.21.15.63:6835","nonce":247249857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6837/2324231032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":2324231032},{"type":"v1","addr":"172.21.15.63:6837","nonce":2324231032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6839/1177197077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":1177197077},{"type":"v1","addr":"172.21.15.63:6839","nonce":1177197077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"e","rank":2,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.53:6835/350727634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":350727634},{"type":"v1","addr":"172.21.15.53:6835","nonce":350727634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/349877573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":349877573},{"type":"v1","addr":"172.21.15.53:6837","nonce":349877573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6839/3714153956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":3714153956},{"type":"v1","addr":"172.21.15.53:6839","nonce":3714153956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24566,"qdb_cluster":[24500,24524,24566]},"id":1}]} 2024-03-28T22:20:39.895 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2024-03-28T22:20:39.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.896+0000 7f2c17b8c640 1 -- 172.21.15.16:0/85712313 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f2bfc0a1ef0 msgr2=0x7f2bfc0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:39.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.896+0000 7f2c17b8c640 1 --2- 172.21.15.16:0/85712313 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f2bfc0a1ef0 0x7f2bfc0a43b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c0095f0 tx=0x7f2c0c009580 comp rx=0 tx=0).stop 2024-03-28T22:20:39.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.896+0000 7f2c17b8c640 1 -- 172.21.15.16:0/85712313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c101008e0 msgr2=0x7f2c10198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:39.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.896+0000 7f2c17b8c640 1 --2- 172.21.15.16:0/85712313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c101008e0 0x7f2c10198670 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f2c0000c970 tx=0x7f2c0000ce30 comp rx=0 tx=0).stop 2024-03-28T22:20:39.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.896+0000 7f2c17b8c640 1 -- 172.21.15.16:0/85712313 shutdown_connections 2024-03-28T22:20:39.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.896+0000 7f2c17b8c640 1 --2- 172.21.15.16:0/85712313 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f2bfc0a1ef0 0x7f2bfc0a43b0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.896+0000 7f2c17b8c640 1 --2- 172.21.15.16:0/85712313 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c10104d70 0x7f2c1019c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.896+0000 7f2c17b8c640 1 --2- 172.21.15.16:0/85712313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c101008e0 0x7f2c10198670 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.896+0000 7f2c17b8c640 1 --2- 172.21.15.16:0/85712313 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c100fff40 0x7f2c10198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:39.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.896+0000 7f2c17b8c640 1 -- 172.21.15.16:0/85712313 >> 172.21.15.16:0/85712313 conn(0x7f2c10076aa0 msgr2=0x7f2c10101e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:39.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.896+0000 7f2c17b8c640 1 -- 172.21.15.16:0/85712313 shutdown_connections 2024-03-28T22:20:39.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:39.896+0000 7f2c17b8c640 1 -- 172.21.15.16:0/85712313 wait complete. 2024-03-28T22:20:39.909 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:20:40.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.013+0000 7fc33b72c640 1 -- 172.21.15.16:0/2486954023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc334102b80 msgr2=0x7fc33410d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:40.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.013+0000 7fc33b72c640 1 --2- 172.21.15.16:0/2486954023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc334102b80 0x7fc33410d5e0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fc330009ca0 tx=0x7fc33002c1d0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.013+0000 7fc33b72c640 1 -- 172.21.15.16:0/2486954023 shutdown_connections 2024-03-28T22:20:40.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.013+0000 7fc33b72c640 1 --2- 172.21.15.16:0/2486954023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc334102b80 0x7fc33410d5e0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.013+0000 7fc33b72c640 1 --2- 172.21.15.16:0/2486954023 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3341021e0 0x7fc334102640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.013+0000 7fc33b72c640 1 --2- 172.21.15.16:0/2486954023 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc3341009d0 0x7fc334100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.013+0000 7fc33b72c640 1 -- 172.21.15.16:0/2486954023 >> 172.21.15.16:0/2486954023 conn(0x7fc334077830 msgr2=0x7fc334077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:40.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.013+0000 7fc33b72c640 1 -- 172.21.15.16:0/2486954023 shutdown_connections 2024-03-28T22:20:40.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.013+0000 7fc33b72c640 1 -- 172.21.15.16:0/2486954023 wait complete. 2024-03-28T22:20:40.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.014+0000 7fc33b72c640 1 Processor -- start 2024-03-28T22:20:40.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.014+0000 7fc33b72c640 1 -- start start 2024-03-28T22:20:40.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.014+0000 7fc33b72c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc3341009d0 0x7fc334198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.014+0000 7fc33b72c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3341021e0 0x7fc334198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.014+0000 7fc33b72c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc334102b80 0x7fc33419c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.014+0000 7fc33b72c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc334111c50 con 0x7fc334102b80 2024-03-28T22:20:40.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.014+0000 7fc33b72c640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc334111ad0 con 0x7fc3341009d0 2024-03-28T22:20:40.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.014+0000 7fc33b72c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc334111dd0 con 0x7fc3341021e0 2024-03-28T22:20:40.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.014+0000 7fc339cc9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc334102b80 0x7fc33419c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.014+0000 7fc339cc9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc334102b80 0x7fc33419c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40956/0 (socket says 172.21.15.16:40956) 2024-03-28T22:20:40.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.014+0000 7fc3394c8640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc3341009d0 0x7fc334198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.014+0000 7fc339cc9640 1 -- 172.21.15.16:0/1216191103 learned_addr learned my addr 172.21.15.16:0/1216191103 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:40.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.015+0000 7fc338cc7640 1 --2- 172.21.15.16:0/1216191103 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3341021e0 0x7fc334198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.015+0000 7fc339cc9640 1 -- 172.21.15.16:0/1216191103 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3341021e0 msgr2=0x7fc334198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:40.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.015+0000 7fc339cc9640 1 --2- 172.21.15.16:0/1216191103 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3341021e0 0x7fc334198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.015+0000 7fc339cc9640 1 -- 172.21.15.16:0/1216191103 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc3341009d0 msgr2=0x7fc334198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:40.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.015+0000 7fc339cc9640 1 --2- 172.21.15.16:0/1216191103 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc3341009d0 0x7fc334198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.015+0000 7fc339cc9640 1 -- 172.21.15.16:0/1216191103 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc33419d040 con 0x7fc334102b80 2024-03-28T22:20:40.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.015+0000 7fc339cc9640 1 --2- 172.21.15.16:0/1216191103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc334102b80 0x7fc33419c910 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fc33002c7e0 tx=0x7fc330034740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:40.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.015+0000 7fc32a7fc640 1 -- 172.21.15.16:0/1216191103 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc330039050 con 0x7fc334102b80 2024-03-28T22:20:40.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.015+0000 7fc33b72c640 1 -- 172.21.15.16:0/1216191103 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc33419d2d0 con 0x7fc334102b80 2024-03-28T22:20:40.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.015+0000 7fc32a7fc640 1 -- 172.21.15.16:0/1216191103 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc330045070 con 0x7fc334102b80 2024-03-28T22:20:40.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.015+0000 7fc32a7fc640 1 -- 172.21.15.16:0/1216191103 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc330032ac0 con 0x7fc334102b80 2024-03-28T22:20:40.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.015+0000 7fc33b72c640 1 -- 172.21.15.16:0/1216191103 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3341a4bc0 con 0x7fc334102b80 2024-03-28T22:20:40.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.017+0000 7fc32a7fc640 1 -- 172.21.15.16:0/1216191103 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7fc3300045b0 con 0x7fc334102b80 2024-03-28T22:20:40.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.017+0000 7fc33e3ec740 1 Processor -- start 2024-03-28T22:20:40.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.018+0000 7fc33e3ec740 1 -- start start 2024-03-28T22:20:40.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.018+0000 7fc32a7fc640 1 --2- 172.21.15.16:0/1216191103 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fc3200a1ef0 0x7fc3200a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.018+0000 7fc33e3ec740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56310d1df470 0x56310d1df850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.018+0000 7fc33e3ec740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56310d1dfd90 0x56310d1e8430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.018+0000 7fc33e3ec740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56310d1e8970 0x56310d1ead60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.018+0000 7fc33e3ec740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56310d1431b0 con 0x56310d1e8970 2024-03-28T22:20:40.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.018+0000 7fc33e3ec740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x56310d0ebc00 con 0x56310d1df470 2024-03-28T22:20:40.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.018+0000 7fc33e3ec740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56310d13f250 con 0x56310d1dfd90 2024-03-28T22:20:40.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.018+0000 7fc338cc7640 1 --2- 172.21.15.16:0/1216191103 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fc3200a1ef0 0x7fc3200a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.018+0000 7fc339cc9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56310d1e8970 0x56310d1ead60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.018+0000 7fc3394c8640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56310d1df470 0x56310d1df850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.018+0000 7fc32a7fc640 1 -- 172.21.15.16:0/1216191103 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8527+0+0 (secure 0 0 0) 0x7fc3300f0a10 con 0x7fc334102b80 2024-03-28T22:20:40.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.018+0000 7fc339cc9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56310d1e8970 0x56310d1ead60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40962/0 (socket says 172.21.15.16:40962) 2024-03-28T22:20:40.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.018+0000 7fc339cc9640 1 -- 172.21.15.16:0/1404869965 learned_addr learned my addr 172.21.15.16:0/1404869965 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:40.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.018+0000 7fc3394c8640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56310d1df470 0x56310d1df850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:36916/0 (socket says 172.21.15.16:36916) 2024-03-28T22:20:40.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.018+0000 7fc338cc7640 1 --2- 172.21.15.16:0/1404869965 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56310d1dfd90 0x56310d1e8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.018+0000 7fc3394c8640 1 -- 172.21.15.16:0/1404869965 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56310d1dfd90 msgr2=0x56310d1e8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:40.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.018+0000 7fc3394c8640 1 --2- 172.21.15.16:0/1404869965 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56310d1dfd90 0x56310d1e8430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.018+0000 7fc3394c8640 1 -- 172.21.15.16:0/1404869965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56310d1e8970 msgr2=0x56310d1ead60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:40.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.018+0000 7fc3394c8640 1 --2- 172.21.15.16:0/1404869965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56310d1e8970 0x56310d1ead60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.018+0000 7fc3394c8640 1 -- 172.21.15.16:0/1404869965 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56310d10b3d0 con 0x56310d1df470 2024-03-28T22:20:40.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.018+0000 7fc339cc9640 1 --2- 172.21.15.16:0/1404869965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56310d1e8970 0x56310d1ead60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:20:40.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.018+0000 7fc338cc7640 1 --2- 172.21.15.16:0/1216191103 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fc3200a1ef0 0x7fc3200a43b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc324006fd0 tx=0x7fc324013650 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:40.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.019+0000 7fc3394c8640 1 --2- 172.21.15.16:0/1404869965 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56310d1df470 0x56310d1df850 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc32c009c20 tx=0x7fc32c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf3adff987a4cfad server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:40.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.019+0000 7fc31f7fe640 1 -- 172.21.15.16:0/1404869965 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc32c01e070 con 0x56310d1df470 2024-03-28T22:20:40.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.019+0000 7fc31f7fe640 1 -- 172.21.15.16:0/1404869965 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc32c009dd0 con 0x56310d1df470 2024-03-28T22:20:40.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.019+0000 7fc31f7fe640 1 -- 172.21.15.16:0/1404869965 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc32c019670 con 0x56310d1df470 2024-03-28T22:20:40.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.019+0000 7fc33e3ec740 1 -- 172.21.15.16:0/1404869965 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56310d1df470 msgr2=0x56310d1df850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:40.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.019+0000 7fc33e3ec740 1 --2- 172.21.15.16:0/1404869965 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56310d1df470 0x56310d1df850 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc32c009c20 tx=0x7fc32c011810 comp rx=0 tx=0).stop 2024-03-28T22:20:40.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.019+0000 7fc33e3ec740 1 -- 172.21.15.16:0/1404869965 shutdown_connections 2024-03-28T22:20:40.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.019+0000 7fc33e3ec740 1 --2- 172.21.15.16:0/1404869965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56310d1e8970 0x56310d1ead60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.019+0000 7fc33e3ec740 1 --2- 172.21.15.16:0/1404869965 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56310d1dfd90 0x56310d1e8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.019+0000 7fc33e3ec740 1 --2- 172.21.15.16:0/1404869965 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56310d1df470 0x56310d1df850 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.019+0000 7fc33e3ec740 1 -- 172.21.15.16:0/1404869965 >> 172.21.15.16:0/1404869965 conn(0x56310d1d9ba0 msgr2=0x56310d1d9f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:40.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.019+0000 7fc33e3ec740 1 -- 172.21.15.16:0/1404869965 shutdown_connections 2024-03-28T22:20:40.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.019+0000 7fc33e3ec740 1 -- 172.21.15.16:0/1404869965 wait complete. 2024-03-28T22:20:40.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.020+0000 7fc33e3ec740 1 -- start start 2024-03-28T22:20:40.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.020+0000 7fc33e3ec740 1 Processor -- start 2024-03-28T22:20:40.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.020+0000 7fc33e3ec740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56310d1df470 0x56310d1e5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.020+0000 7fc33e3ec740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56310d1dfd90 0x56310d1e5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.020+0000 7fc33e3ec740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56310d1e8970 0x56310d1e6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.020+0000 7fc33e3ec740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56310d1efb70 con 0x56310d1e8970 2024-03-28T22:20:40.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.020+0000 7fc33e3ec740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x56310d0e1470 con 0x56310d1df470 2024-03-28T22:20:40.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.020+0000 7fc33e3ec740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56310d1efcf0 con 0x56310d1dfd90 2024-03-28T22:20:40.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.020+0000 7fc33e3ec740 10 client.0 _ll_register_callbacks cb 0x56310d0fecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:20:40.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.020+0000 7fc3394c8640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56310d1df470 0x56310d1e5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.020+0000 7fc339cc9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56310d1e8970 0x56310d1e6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.020+0000 7fc3394c8640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56310d1df470 0x56310d1e5990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:36926/0 (socket says 172.21.15.16:36926) 2024-03-28T22:20:40.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.020+0000 7fc3394c8640 1 -- 172.21.15.16:0/2746919536 learned_addr learned my addr 172.21.15.16:0/2746919536 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:40.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.020+0000 7fc338cc7640 1 --2- 172.21.15.16:0/2746919536 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56310d1dfd90 0x56310d1e5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.021+0000 7fc3394c8640 1 -- 172.21.15.16:0/2746919536 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56310d1dfd90 msgr2=0x56310d1e5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:40.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.021+0000 7fc3394c8640 1 --2- 172.21.15.16:0/2746919536 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56310d1dfd90 0x56310d1e5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.021+0000 7fc3394c8640 1 -- 172.21.15.16:0/2746919536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56310d1e8970 msgr2=0x56310d1e6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:40.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.021+0000 7fc3394c8640 1 --2- 172.21.15.16:0/2746919536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56310d1e8970 0x56310d1e6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.021+0000 7fc3394c8640 1 -- 172.21.15.16:0/2746919536 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56310ce573f0 con 0x56310d1df470 2024-03-28T22:20:40.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.021+0000 7fc339cc9640 1 --2- 172.21.15.16:0/2746919536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56310d1e8970 0x56310d1e6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:20:40.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.021+0000 7fc338cc7640 1 --2- 172.21.15.16:0/2746919536 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56310d1dfd90 0x56310d1e5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:20:40.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.021+0000 7fc3394c8640 1 --2- 172.21.15.16:0/2746919536 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56310d1df470 0x56310d1e5990 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc32c009730 tx=0x7fc32c005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98a10f00893dffbf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:40.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.021+0000 7fc31cff9640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-03-28T22:20:40.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.021+0000 7fc31cff9640 1 -- 172.21.15.16:0/2746919536 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc32c01e040 con 0x56310d1df470 2024-03-28T22:20:40.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.021+0000 7fc33e3ec740 1 -- 172.21.15.16:0/2746919536 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc32c009dd0 con 0x56310d1df470 2024-03-28T22:20:40.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.021+0000 7fc31cff9640 1 -- 172.21.15.16:0/2746919536 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8527+0+0 (secure 0 0 0) 0x7fc32c0035f0 con 0x56310d1df470 2024-03-28T22:20:40.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.021+0000 7fc31cff9640 1 -- 172.21.15.16:0/2746919536 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc32c031050 con 0x56310d1df470 2024-03-28T22:20:40.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.021+0000 7fc33e3ec740 10 client.27524 fetch_fsmap learned FSMap version 27 2024-03-28T22:20:40.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.021+0000 7fc33e3ec740 1 -- 172.21.15.16:0/2746919536 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56310cf217d0 con 0x56310d1df470 2024-03-28T22:20:40.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.021+0000 7fc31cff9640 1 -- 172.21.15.16:0/2746919536 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc32c009dd0 con 0x56310d1df470 2024-03-28T22:20:40.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.022+0000 7fc31cff9640 1 -- 172.21.15.16:0/2746919536 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7fc32c02c030 con 0x56310d1df470 2024-03-28T22:20:40.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.022+0000 7fc33e3ec740 10 client.27524 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:20:40.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.022+0000 7fc33e3ec740 10 client.27524 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-28T22:20:40.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.022+0000 7fc33e3ec740 20 client.27524 populate_metadata read hostname 'smithi016' 2024-03-28T22:20:40.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.022+0000 7fc33e3ec740 1 --2- 172.21.15.16:0/2746919536 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x56310d1fcd20 0x56310d1ff110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.022+0000 7fc33e3ec740 4 client.27524 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:20:40.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.022+0000 7fc33e3ec740 1 -- 172.21.15.16:0/2746919536 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56310d1506f0 con 0x56310d1fcd20 2024-03-28T22:20:40.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.022+0000 7fc338cc7640 1 --2- 172.21.15.16:0/2746919536 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x56310d1fcd20 0x56310d1ff110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.022+0000 7fc338cc7640 1 --2- 172.21.15.16:0/2746919536 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x56310d1fcd20 0x56310d1ff110 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f07768451ce33bbd server_cookie=70fd903c6a1c6322 in_seq=0 out_seq=0 2024-03-28T22:20:40.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.022+0000 7fc31cff9640 10 client.27524 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:20:40.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.023+0000 7fc31cff9640 1 -- 172.21.15.16:0/2746919536 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56310d1506f0 con 0x56310d1fcd20 2024-03-28T22:20:40.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.023+0000 7fc31cff9640 10 client.27524 handle_command_reply: tid=0 2024-03-28T22:20:40.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.023+0000 7fc33e3ec740 1 client.27524 shutdown 2024-03-28T22:20:40.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.023+0000 7fc33e3ec740 1 -- 172.21.15.16:0/2746919536 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56310d1df470 msgr2=0x56310d1e5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:40.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.023+0000 7fc33e3ec740 1 --2- 172.21.15.16:0/2746919536 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56310d1df470 0x56310d1e5990 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc32c009730 tx=0x7fc32c005740 comp rx=0 tx=0).stop 2024-03-28T22:20:40.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.023+0000 7fc33e3ec740 1 -- 172.21.15.16:0/2746919536 shutdown_connections 2024-03-28T22:20:40.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.023+0000 7fc33e3ec740 1 --2- 172.21.15.16:0/2746919536 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x56310d1fcd20 0x56310d1ff110 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.023+0000 7fc33e3ec740 1 --2- 172.21.15.16:0/2746919536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56310d1e8970 0x56310d1e6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.023+0000 7fc31cff9640 0 client.27524 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:20:40.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.023+0000 7fc33e3ec740 1 --2- 172.21.15.16:0/2746919536 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56310d1dfd90 0x56310d1e5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.023+0000 7fc33e3ec740 1 --2- 172.21.15.16:0/2746919536 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56310d1df470 0x56310d1e5990 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.023+0000 7fc33e3ec740 1 -- 172.21.15.16:0/2746919536 >> 172.21.15.16:0/2746919536 conn(0x56310d1d9ba0 msgr2=0x56310d1db8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:40.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.023+0000 7fc33e3ec740 1 -- 172.21.15.16:0/2746919536 shutdown_connections 2024-03-28T22:20:40.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.023+0000 7fc33e3ec740 1 -- 172.21.15.16:0/2746919536 wait complete. 2024-03-28T22:20:40.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.023+0000 7fc33e3ec740 20 client.27524 trim_cache size 0 max 16384 2024-03-28T22:20:40.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.036+0000 7fc33e3ec740 1 Processor -- start 2024-03-28T22:20:40.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.037+0000 7fc33e3ec740 1 -- start start 2024-03-28T22:20:40.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.037+0000 7fc33e3ec740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56310d1df470 0x56310d1e80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.037+0000 7fc33e3ec740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56310d1dfd90 0x56310d1f8940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.037+0000 7fc33e3ec740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56310d1e8970 0x56310d1e4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.037+0000 7fc33e3ec740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc3341ad9b0 con 0x56310d1e8970 2024-03-28T22:20:40.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.037+0000 7fc33e3ec740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc3341ab280 con 0x56310d1dfd90 2024-03-28T22:20:40.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.037+0000 7fc33e3ec740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc3341a90f0 con 0x56310d1df470 2024-03-28T22:20:40.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.037+0000 7fc339cc9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56310d1e8970 0x56310d1e4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.037+0000 7fc338cc7640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56310d1dfd90 0x56310d1f8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.037+0000 7fc339cc9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56310d1e8970 0x56310d1e4840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40980/0 (socket says 172.21.15.16:40980) 2024-03-28T22:20:40.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.037+0000 7fc339cc9640 1 -- 172.21.15.16:0/1667081274 learned_addr learned my addr 172.21.15.16:0/1667081274 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:40.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.037+0000 7fc3394c8640 1 --2- 172.21.15.16:0/1667081274 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56310d1df470 0x56310d1e80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.037+0000 7fc339cc9640 1 -- 172.21.15.16:0/1667081274 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56310d1df470 msgr2=0x56310d1e80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:40.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.037+0000 7fc339cc9640 1 --2- 172.21.15.16:0/1667081274 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56310d1df470 0x56310d1e80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.037+0000 7fc339cc9640 1 -- 172.21.15.16:0/1667081274 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56310d1dfd90 msgr2=0x56310d1f8940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:40.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.037+0000 7fc339cc9640 1 --2- 172.21.15.16:0/1667081274 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56310d1dfd90 0x56310d1f8940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.037+0000 7fc339cc9640 1 -- 172.21.15.16:0/1667081274 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56310d1e3ee0 con 0x56310d1e8970 2024-03-28T22:20:40.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.037+0000 7fc3394c8640 1 --2- 172.21.15.16:0/1667081274 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56310d1df470 0x56310d1e80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:20:40.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.037+0000 7fc339cc9640 1 --2- 172.21.15.16:0/1667081274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56310d1e8970 0x56310d1e4840 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fc3300bd170 tx=0x7fc33002c1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0f1706c9254b724 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:40.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.037+0000 7fc31effd640 1 -- 172.21.15.16:0/1667081274 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3300bcfb0 con 0x56310d1e8970 2024-03-28T22:20:40.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.038+0000 7fc31effd640 1 -- 172.21.15.16:0/1667081274 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc33003a040 con 0x56310d1e8970 2024-03-28T22:20:40.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.038+0000 7fc31effd640 1 -- 172.21.15.16:0/1667081274 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc33004b720 con 0x56310d1e8970 2024-03-28T22:20:40.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.038+0000 7fc33e3ec740 1 -- 172.21.15.16:0/1667081274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56310d1e8970 msgr2=0x56310d1e4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:40.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.038+0000 7fc33e3ec740 1 --2- 172.21.15.16:0/1667081274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56310d1e8970 0x56310d1e4840 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fc3300bd170 tx=0x7fc33002c1d0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.038+0000 7fc33e3ec740 1 -- 172.21.15.16:0/1667081274 shutdown_connections 2024-03-28T22:20:40.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.038+0000 7fc33e3ec740 1 --2- 172.21.15.16:0/1667081274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56310d1e8970 0x56310d1e4840 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.038+0000 7fc33e3ec740 1 --2- 172.21.15.16:0/1667081274 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56310d1dfd90 0x56310d1f8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.038+0000 7fc33e3ec740 1 --2- 172.21.15.16:0/1667081274 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56310d1df470 0x56310d1e80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.038+0000 7fc33e3ec740 1 -- 172.21.15.16:0/1667081274 >> 172.21.15.16:0/1667081274 conn(0x56310d1d9ba0 msgr2=0x56310d1e4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:40.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.038+0000 7fc33e3ec740 1 -- 172.21.15.16:0/1667081274 shutdown_connections 2024-03-28T22:20:40.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.038+0000 7fc33e3ec740 1 -- 172.21.15.16:0/1667081274 wait complete. 2024-03-28T22:20:40.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.038+0000 7fc33e3ec740 1 -- start start 2024-03-28T22:20:40.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.038+0000 7fc33e3ec740 1 Processor -- start 2024-03-28T22:20:40.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.039+0000 7fc33e3ec740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56310d1df470 0x56310d1f5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.039+0000 7fc33e3ec740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56310d1dfd90 0x56310d1e2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.039+0000 7fc33e3ec740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56310d1e8970 0x56310d1e26a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.039+0000 7fc33e3ec740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc3341a9740 con 0x56310d1e8970 2024-03-28T22:20:40.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.039+0000 7fc33e3ec740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc3341aabb0 con 0x56310d1df470 2024-03-28T22:20:40.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.039+0000 7fc33e3ec740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc3341a9e10 con 0x56310d1dfd90 2024-03-28T22:20:40.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.039+0000 7fc33e3ec740 10 client.0 _ll_register_callbacks cb 0x7fc32c009a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:20:40.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.039+0000 7fc339cc9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56310d1e8970 0x56310d1e26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.039+0000 7fc3394c8640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56310d1df470 0x56310d1f5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.039+0000 7fc339cc9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56310d1e8970 0x56310d1e26a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40982/0 (socket says 172.21.15.16:40982) 2024-03-28T22:20:40.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.039+0000 7fc339cc9640 1 -- 172.21.15.16:0/3982332175 learned_addr learned my addr 172.21.15.16:0/3982332175 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:40.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.039+0000 7fc338cc7640 1 --2- 172.21.15.16:0/3982332175 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56310d1dfd90 0x56310d1e2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.039+0000 7fc339cc9640 1 -- 172.21.15.16:0/3982332175 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56310d1dfd90 msgr2=0x56310d1e2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:40.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.039+0000 7fc339cc9640 1 --2- 172.21.15.16:0/3982332175 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56310d1dfd90 0x56310d1e2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.039+0000 7fc339cc9640 1 -- 172.21.15.16:0/3982332175 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56310d1df470 msgr2=0x56310d1f5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:40.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.039+0000 7fc339cc9640 1 --2- 172.21.15.16:0/3982332175 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56310d1df470 0x56310d1f5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.039+0000 7fc339cc9640 1 -- 172.21.15.16:0/3982332175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56310d1e2d10 con 0x56310d1e8970 2024-03-28T22:20:40.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.039+0000 7fc338cc7640 1 --2- 172.21.15.16:0/3982332175 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56310d1dfd90 0x56310d1e2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:20:40.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.039+0000 7fc339cc9640 1 --2- 172.21.15.16:0/3982332175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56310d1e8970 0x56310d1e26a0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fc3300ee320 tx=0x7fc3300bd540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a4013e180b3d8ea server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:40.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.039+0000 7fc31cff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:20:40.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.039+0000 7fc31cff9640 1 -- 172.21.15.16:0/3982332175 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3300a4070 con 0x56310d1e8970 2024-03-28T22:20:40.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.040+0000 7fc31cff9640 1 -- 172.21.15.16:0/3982332175 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8527+0+0 (secure 0 0 0) 0x7fc33004b910 con 0x56310d1e8970 2024-03-28T22:20:40.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.040+0000 7fc33e3ec740 1 -- 172.21.15.16:0/3982332175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc33003a040 con 0x56310d1e8970 2024-03-28T22:20:40.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.040+0000 7fc31cff9640 1 -- 172.21.15.16:0/3982332175 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc3300bcdd0 con 0x56310d1e8970 2024-03-28T22:20:40.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.040+0000 7fc33e3ec740 10 client.17628 fetch_fsmap learned FSMap version 27 2024-03-28T22:20:40.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.040+0000 7fc33e3ec740 1 -- 172.21.15.16:0/3982332175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56310d262d30 con 0x56310d1e8970 2024-03-28T22:20:40.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.040+0000 7fc31cff9640 1 -- 172.21.15.16:0/3982332175 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc33003a040 con 0x56310d1e8970 2024-03-28T22:20:40.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.040+0000 7fc31cff9640 1 -- 172.21.15.16:0/3982332175 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7fc3300dbbe0 con 0x56310d1e8970 2024-03-28T22:20:40.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.040+0000 7fc33e3ec740 10 client.17628 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:20:40.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.040+0000 7fc33e3ec740 10 client.17628 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-28T22:20:40.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.040+0000 7fc33e3ec740 20 client.17628 populate_metadata read hostname 'smithi016' 2024-03-28T22:20:40.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.040+0000 7fc33e3ec740 1 --2- 172.21.15.16:0/3982332175 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x56310d263260 0x56310d265650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.040+0000 7fc33e3ec740 4 client.17628 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:20:40.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.040+0000 7fc33e3ec740 1 -- 172.21.15.16:0/3982332175 --> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56310d14aa40 con 0x56310d263260 2024-03-28T22:20:40.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.040+0000 7fc3394c8640 1 --2- 172.21.15.16:0/3982332175 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x56310d263260 0x56310d265650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.041+0000 7fc31cff9640 10 client.17628 ms_handle_connect on v2:172.21.15.53:6834/350727634 2024-03-28T22:20:40.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.041+0000 7fc3394c8640 1 --2- 172.21.15.16:0/3982332175 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x56310d263260 0x56310d265650 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=90208ad1d073f616 server_cookie=b642e8a9ec690749 in_seq=0 out_seq=0 2024-03-28T22:20:40.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.041+0000 7fc31cff9640 1 -- 172.21.15.16:0/3982332175 <== mds.2 v2:172.21.15.53:6834/350727634 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56310d14aa40 con 0x56310d263260 2024-03-28T22:20:40.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.041+0000 7fc31cff9640 10 client.17628 handle_command_reply: tid=0 2024-03-28T22:20:40.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.041+0000 7fc33e3ec740 1 client.17628 shutdown 2024-03-28T22:20:40.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.041+0000 7fc33e3ec740 1 -- 172.21.15.16:0/3982332175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56310d1e8970 msgr2=0x56310d1e26a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:40.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.041+0000 7fc33e3ec740 1 --2- 172.21.15.16:0/3982332175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56310d1e8970 0x56310d1e26a0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fc3300ee320 tx=0x7fc3300bd540 comp rx=0 tx=0).stop 2024-03-28T22:20:40.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.041+0000 7fc33e3ec740 1 -- 172.21.15.16:0/3982332175 shutdown_connections 2024-03-28T22:20:40.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.041+0000 7fc33e3ec740 1 --2- 172.21.15.16:0/3982332175 >> [v2:172.21.15.53:6834/350727634,v1:172.21.15.53:6835/350727634] conn(0x56310d263260 0x56310d265650 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.041+0000 7fc33e3ec740 1 --2- 172.21.15.16:0/3982332175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56310d1e8970 0x56310d1e26a0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.041+0000 7fc33e3ec740 1 --2- 172.21.15.16:0/3982332175 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56310d1dfd90 0x56310d1e2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.041+0000 7fc31cff9640 0 client.17628 ms_handle_reset on v2:172.21.15.53:6834/350727634 2024-03-28T22:20:40.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.041+0000 7fc33e3ec740 1 --2- 172.21.15.16:0/3982332175 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56310d1df470 0x56310d1f5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.041+0000 7fc33e3ec740 1 -- 172.21.15.16:0/3982332175 >> 172.21.15.16:0/3982332175 conn(0x56310d1d9ba0 msgr2=0x56310d1e49c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:40.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.041+0000 7fc33e3ec740 1 -- 172.21.15.16:0/3982332175 shutdown_connections 2024-03-28T22:20:40.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.041+0000 7fc33e3ec740 1 -- 172.21.15.16:0/3982332175 wait complete. 2024-03-28T22:20:40.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.041+0000 7fc33e3ec740 20 client.17628 trim_cache size 0 max 16384 2024-03-28T22:20:40.055 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:20:40.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.042+0000 7fc33b72c640 1 -- 172.21.15.16:0/1216191103 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fc3200a1ef0 msgr2=0x7fc3200a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:40.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.042+0000 7fc33b72c640 1 --2- 172.21.15.16:0/1216191103 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fc3200a1ef0 0x7fc3200a43b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc324006fd0 tx=0x7fc324013650 comp rx=0 tx=0).stop 2024-03-28T22:20:40.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.042+0000 7fc33b72c640 1 -- 172.21.15.16:0/1216191103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc334102b80 msgr2=0x7fc33419c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:40.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.042+0000 7fc33b72c640 1 --2- 172.21.15.16:0/1216191103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc334102b80 0x7fc33419c910 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fc33002c7e0 tx=0x7fc330034740 comp rx=0 tx=0).stop 2024-03-28T22:20:40.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.042+0000 7fc33b72c640 1 -- 172.21.15.16:0/1216191103 shutdown_connections 2024-03-28T22:20:40.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.042+0000 7fc33b72c640 1 --2- 172.21.15.16:0/1216191103 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fc3200a1ef0 0x7fc3200a43b0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.042+0000 7fc33b72c640 1 --2- 172.21.15.16:0/1216191103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc334102b80 0x7fc33419c910 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.042+0000 7fc33b72c640 1 --2- 172.21.15.16:0/1216191103 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3341021e0 0x7fc334198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.042+0000 7fc33b72c640 1 --2- 172.21.15.16:0/1216191103 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc3341009d0 0x7fc334198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.042+0000 7fc33b72c640 1 -- 172.21.15.16:0/1216191103 >> 172.21.15.16:0/1216191103 conn(0x7fc334077830 msgr2=0x7fc334104280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:40.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.042+0000 7fc33b72c640 1 -- 172.21.15.16:0/1216191103 shutdown_connections 2024-03-28T22:20:40.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.042+0000 7fc33b72c640 1 -- 172.21.15.16:0/1216191103 wait complete. 2024-03-28T22:20:40.058 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:20:40.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.162+0000 7fcdddcbb640 1 -- 172.21.15.16:0/283786298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdd81005f0 msgr2=0x7fcdd8100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:40.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.162+0000 7fcdddcbb640 1 --2- 172.21.15.16:0/283786298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdd81005f0 0x7fcdd8100a70 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc009930 tx=0x7fcdcc02bc30 comp rx=0 tx=0).stop 2024-03-28T22:20:40.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.162+0000 7fcdddcbb640 1 -- 172.21.15.16:0/283786298 shutdown_connections 2024-03-28T22:20:40.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.162+0000 7fcdddcbb640 1 --2- 172.21.15.16:0/283786298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcdd8100fb0 0x7fcdd810d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.162+0000 7fcdddcbb640 1 --2- 172.21.15.16:0/283786298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdd81005f0 0x7fcdd8100a70 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.162+0000 7fcdddcbb640 1 --2- 172.21.15.16:0/283786298 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcdd8106690 0x7fcdd8106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.162+0000 7fcdddcbb640 1 -- 172.21.15.16:0/283786298 >> 172.21.15.16:0/283786298 conn(0x7fcdd8077800 msgr2=0x7fcdd8077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:40.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.162+0000 7fcdddcbb640 1 -- 172.21.15.16:0/283786298 shutdown_connections 2024-03-28T22:20:40.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.162+0000 7fcdddcbb640 1 -- 172.21.15.16:0/283786298 wait complete. 2024-03-28T22:20:40.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.163+0000 7fcdddcbb640 1 Processor -- start 2024-03-28T22:20:40.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.163+0000 7fcdddcbb640 1 -- start start 2024-03-28T22:20:40.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.163+0000 7fcdddcbb640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcdd81005f0 0x7fcdd8102290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.163+0000 7fcdddcbb640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcdd8100fb0 0x7fcdd81027d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.163+0000 7fcdddcbb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdd8106690 0x7fcdd8102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.163+0000 7fcdddcbb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd81109d0 con 0x7fcdd8106690 2024-03-28T22:20:40.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.163+0000 7fcdddcbb640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd8110850 con 0x7fcdd8100fb0 2024-03-28T22:20:40.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.163+0000 7fcdddcbb640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd8110b50 con 0x7fcdd81005f0 2024-03-28T22:20:40.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.163+0000 7fcdd7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdd8106690 0x7fcdd8102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.163+0000 7fcdd7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdd8106690 0x7fcdd8102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41000/0 (socket says 172.21.15.16:41000) 2024-03-28T22:20:40.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.163+0000 7fcdd6ffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcdd8100fb0 0x7fcdd81027d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.163+0000 7fcdd7fff640 1 -- 172.21.15.16:0/2817405190 learned_addr learned my addr 172.21.15.16:0/2817405190 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:40.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.163+0000 7fcdd77fe640 1 --2- 172.21.15.16:0/2817405190 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcdd81005f0 0x7fcdd8102290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.163+0000 7fcdd7fff640 1 -- 172.21.15.16:0/2817405190 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcdd81005f0 msgr2=0x7fcdd8102290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:40.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.163+0000 7fcdd7fff640 1 --2- 172.21.15.16:0/2817405190 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcdd81005f0 0x7fcdd8102290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.163+0000 7fcdd7fff640 1 -- 172.21.15.16:0/2817405190 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcdd8100fb0 msgr2=0x7fcdd81027d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:40.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.163+0000 7fcdd7fff640 1 --2- 172.21.15.16:0/2817405190 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcdd8100fb0 0x7fcdd81027d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.163+0000 7fcdd7fff640 1 -- 172.21.15.16:0/2817405190 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdd81a64d0 con 0x7fcdd8106690 2024-03-28T22:20:40.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.163+0000 7fcdd77fe640 1 --2- 172.21.15.16:0/2817405190 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcdd81005f0 0x7fcdd8102290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:20:40.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.164+0000 7fcdd7fff640 1 --2- 172.21.15.16:0/2817405190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdd8106690 0x7fcdd8102d10 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fcdd000da40 tx=0x7fcdd000df00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:40.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.164+0000 7fcdd4ff9640 1 -- 172.21.15.16:0/2817405190 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd0017070 con 0x7fcdd8106690 2024-03-28T22:20:40.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.164+0000 7fcdd4ff9640 1 -- 172.21.15.16:0/2817405190 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdd0003e50 con 0x7fcdd8106690 2024-03-28T22:20:40.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.164+0000 7fcdddcbb640 1 -- 172.21.15.16:0/2817405190 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdd81a67c0 con 0x7fcdd8106690 2024-03-28T22:20:40.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.164+0000 7fcdd4ff9640 1 -- 172.21.15.16:0/2817405190 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd0010460 con 0x7fcdd8106690 2024-03-28T22:20:40.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.164+0000 7fcdddcbb640 1 -- 172.21.15.16:0/2817405190 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdd81a6c90 con 0x7fcdd8106690 2024-03-28T22:20:40.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.166+0000 7fcdd4ff9640 1 -- 172.21.15.16:0/2817405190 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7fcdd0003970 con 0x7fcdd8106690 2024-03-28T22:20:40.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.166+0000 7fcde0970740 1 Processor -- start 2024-03-28T22:20:40.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.166+0000 7fcde0970740 1 -- start start 2024-03-28T22:20:40.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.166+0000 7fcde0970740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d4d11470 0x55a3d4d11850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.166+0000 7fcde0970740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3d4d11d90 0x55a3d4d1a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.166+0000 7fcde0970740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3d4d1a970 0x55a3d4d1cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.166+0000 7fcde0970740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a3d4c751b0 con 0x55a3d4d11470 2024-03-28T22:20:40.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.166+0000 7fcde0970740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55a3d4c1dc00 con 0x55a3d4d11d90 2024-03-28T22:20:40.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.166+0000 7fcde0970740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55a3d4c71250 con 0x55a3d4d1a970 2024-03-28T22:20:40.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.167+0000 7fcdd6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d4d11470 0x55a3d4d11850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.167+0000 7fcdd6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d4d11470 0x55a3d4d11850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41012/0 (socket says 172.21.15.16:41012) 2024-03-28T22:20:40.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.167+0000 7fcdd6ffd640 1 -- 172.21.15.16:0/987730419 learned_addr learned my addr 172.21.15.16:0/987730419 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:40.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.167+0000 7fcdd77fe640 1 --2- 172.21.15.16:0/987730419 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3d4d11d90 0x55a3d4d1a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.167+0000 7fcdd4ff9640 1 --2- 172.21.15.16:0/2817405190 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fcdc00a1ef0 0x7fcdc00a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.167+0000 7fcdd7fff640 1 --2- 172.21.15.16:0/2817405190 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fcdc00a1ef0 0x7fcdc00a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.167+0000 7fcdd6ffd640 1 -- 172.21.15.16:0/987730419 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3d4d1a970 msgr2=0x55a3d4d1cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:40.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.167+0000 7fcdd6ffd640 1 --2- 172.21.15.16:0/987730419 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3d4d1a970 0x55a3d4d1cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.167+0000 7fcdd6ffd640 1 -- 172.21.15.16:0/987730419 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3d4d11d90 msgr2=0x55a3d4d1a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:40.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.167+0000 7fcdd6ffd640 1 --2- 172.21.15.16:0/987730419 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3d4d11d90 0x55a3d4d1a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.167+0000 7fcdd6ffd640 1 -- 172.21.15.16:0/987730419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3d4c3d3d0 con 0x55a3d4d11470 2024-03-28T22:20:40.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.167+0000 7fcdd77fe640 1 --2- 172.21.15.16:0/987730419 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3d4d11d90 0x55a3d4d1a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:20:40.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.167+0000 7fcdd4ff9640 1 -- 172.21.15.16:0/2817405190 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8527+0+0 (secure 0 0 0) 0x7fcdd00cde90 con 0x7fcdd8106690 2024-03-28T22:20:40.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.167+0000 7fcdd6ffd640 1 --2- 172.21.15.16:0/987730419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d4d11470 0x55a3d4d11850 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc02cfd0 tx=0x7fcdcc005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=933021772d0bf3eb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:40.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.167+0000 7fcdbdffb640 1 -- 172.21.15.16:0/987730419 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc02ca80 con 0x55a3d4d11470 2024-03-28T22:20:40.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.167+0000 7fcdd7fff640 1 --2- 172.21.15.16:0/2817405190 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fcdc00a1ef0 0x7fcdc00a43b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fcdd0096c80 tx=0x7fcdd0085040 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:40.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.167+0000 7fcdbdffb640 1 -- 172.21.15.16:0/987730419 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdcc02cc20 con 0x55a3d4d11470 2024-03-28T22:20:40.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.167+0000 7fcdbdffb640 1 -- 172.21.15.16:0/987730419 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc0051b0 con 0x55a3d4d11470 2024-03-28T22:20:40.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.168+0000 7fcde0970740 1 -- 172.21.15.16:0/987730419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d4d11470 msgr2=0x55a3d4d11850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:40.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.168+0000 7fcde0970740 1 --2- 172.21.15.16:0/987730419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d4d11470 0x55a3d4d11850 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc02cfd0 tx=0x7fcdcc005e00 comp rx=0 tx=0).stop 2024-03-28T22:20:40.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.168+0000 7fcde0970740 1 -- 172.21.15.16:0/987730419 shutdown_connections 2024-03-28T22:20:40.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.168+0000 7fcde0970740 1 --2- 172.21.15.16:0/987730419 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3d4d1a970 0x55a3d4d1cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.168+0000 7fcde0970740 1 --2- 172.21.15.16:0/987730419 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3d4d11d90 0x55a3d4d1a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.168+0000 7fcde0970740 1 --2- 172.21.15.16:0/987730419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d4d11470 0x55a3d4d11850 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.168+0000 7fcde0970740 1 -- 172.21.15.16:0/987730419 >> 172.21.15.16:0/987730419 conn(0x55a3d4d0bba0 msgr2=0x55a3d4d0bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:40.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.168+0000 7fcde0970740 1 -- 172.21.15.16:0/987730419 shutdown_connections 2024-03-28T22:20:40.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.168+0000 7fcde0970740 1 -- 172.21.15.16:0/987730419 wait complete. 2024-03-28T22:20:40.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.168+0000 7fcde0970740 1 -- start start 2024-03-28T22:20:40.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.169+0000 7fcde0970740 1 Processor -- start 2024-03-28T22:20:40.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.169+0000 7fcde0970740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3d4d11470 0x55a3d4d179c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.169+0000 7fcde0970740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3d4d11d90 0x55a3d4d17f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.169+0000 7fcde0970740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d4d1a970 0x55a3d4d18440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.169+0000 7fcde0970740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a3d4d1d700 con 0x55a3d4d1a970 2024-03-28T22:20:40.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.169+0000 7fcde0970740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55a3d4c13470 con 0x55a3d4d11470 2024-03-28T22:20:40.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.169+0000 7fcde0970740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55a3d4d1d880 con 0x55a3d4d11d90 2024-03-28T22:20:40.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.169+0000 7fcde0970740 10 client.0 _ll_register_callbacks cb 0x55a3d4c30cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:20:40.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.169+0000 7fcdd6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d4d1a970 0x55a3d4d18440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.169+0000 7fcdd6ffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3d4d11470 0x55a3d4d179c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.169+0000 7fcdd6ffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3d4d11470 0x55a3d4d179c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:36970/0 (socket says 172.21.15.16:36970) 2024-03-28T22:20:40.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.169+0000 7fcdd6ffd640 1 -- 172.21.15.16:0/2102643171 learned_addr learned my addr 172.21.15.16:0/2102643171 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:40.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.169+0000 7fcdd77fe640 1 --2- 172.21.15.16:0/2102643171 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3d4d11d90 0x55a3d4d17f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.169+0000 7fcdd6ffd640 1 -- 172.21.15.16:0/2102643171 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3d4d11d90 msgr2=0x55a3d4d17f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:40.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.169+0000 7fcdd6ffd640 1 --2- 172.21.15.16:0/2102643171 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3d4d11d90 0x55a3d4d17f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.169+0000 7fcdd6ffd640 1 -- 172.21.15.16:0/2102643171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d4d1a970 msgr2=0x55a3d4d18440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:40.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.169+0000 7fcdd6ffd640 1 --2- 172.21.15.16:0/2102643171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d4d1a970 0x55a3d4d18440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.169+0000 7fcdd6ffd640 1 -- 172.21.15.16:0/2102643171 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3d49893f0 con 0x55a3d4d11470 2024-03-28T22:20:40.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.169+0000 7fcdd77fe640 1 --2- 172.21.15.16:0/2102643171 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3d4d11d90 0x55a3d4d17f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:20:40.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.169+0000 7fcdd6ffd640 1 --2- 172.21.15.16:0/2102643171 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3d4d11470 0x55a3d4d179c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc005ab0 tx=0x7fcdcc005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb20cba07fb15ebe server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:40.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.169+0000 7fcdb6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-03-28T22:20:40.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.170+0000 7fcdb6ffd640 1 -- 172.21.15.16:0/2102643171 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc04e070 con 0x55a3d4d11470 2024-03-28T22:20:40.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.170+0000 7fcdb6ffd640 1 -- 172.21.15.16:0/2102643171 <== mon.2 v2:172.21.15.53:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8527+0+0 (secure 0 0 0) 0x7fcdcc0046c0 con 0x55a3d4d11470 2024-03-28T22:20:40.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.170+0000 7fcde0970740 1 -- 172.21.15.16:0/2102643171 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcdcc02cc20 con 0x55a3d4d11470 2024-03-28T22:20:40.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.170+0000 7fcdb6ffd640 1 -- 172.21.15.16:0/2102643171 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcdcc05e050 con 0x55a3d4d11470 2024-03-28T22:20:40.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.170+0000 7fcde0970740 10 client.27542 fetch_fsmap learned FSMap version 27 2024-03-28T22:20:40.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.170+0000 7fcde0970740 1 -- 172.21.15.16:0/2102643171 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a3d4a537d0 con 0x55a3d4d11470 2024-03-28T22:20:40.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.170+0000 7fcdb6ffd640 1 -- 172.21.15.16:0/2102643171 <== mon.2 v2:172.21.15.53:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdcc02cc20 con 0x55a3d4d11470 2024-03-28T22:20:40.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.170+0000 7fcdb6ffd640 1 -- 172.21.15.16:0/2102643171 <== mon.2 v2:172.21.15.53:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7fcdcc0039d0 con 0x55a3d4d11470 2024-03-28T22:20:40.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.170+0000 7fcde0970740 10 client.27542 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:20:40.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.170+0000 7fcde0970740 10 client.27542 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T22:20:40.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.170+0000 7fcde0970740 20 client.27542 populate_metadata read hostname 'smithi016' 2024-03-28T22:20:40.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.170+0000 7fcde0970740 1 --2- 172.21.15.16:0/2102643171 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55a3d4d2a7b0 0x55a3d4d2cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.170+0000 7fcde0970740 4 client.27542 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:20:40.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.170+0000 7fcde0970740 1 -- 172.21.15.16:0/2102643171 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a3d4c826f0 con 0x55a3d4d2a7b0 2024-03-28T22:20:40.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.170+0000 7fcdd77fe640 1 --2- 172.21.15.16:0/2102643171 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55a3d4d2a7b0 0x55a3d4d2cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.171+0000 7fcdd77fe640 1 --2- 172.21.15.16:0/2102643171 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55a3d4d2a7b0 0x55a3d4d2cba0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1846a09728984cd9 server_cookie=35bb31405250f88f in_seq=0 out_seq=0 2024-03-28T22:20:40.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.171+0000 7fcdb6ffd640 10 client.27542 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:20:40.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.172+0000 7fcdb6ffd640 1 -- 172.21.15.16:0/2102643171 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a3d4c826f0 con 0x55a3d4d2a7b0 2024-03-28T22:20:40.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.172+0000 7fcdb6ffd640 10 client.27542 handle_command_reply: tid=0 2024-03-28T22:20:40.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.172+0000 7fcde0970740 1 client.27542 shutdown 2024-03-28T22:20:40.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.172+0000 7fcde0970740 1 -- 172.21.15.16:0/2102643171 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3d4d11470 msgr2=0x55a3d4d179c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:40.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.172+0000 7fcde0970740 1 --2- 172.21.15.16:0/2102643171 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3d4d11470 0x55a3d4d179c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc005ab0 tx=0x7fcdcc005fd0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.172+0000 7fcde0970740 1 -- 172.21.15.16:0/2102643171 shutdown_connections 2024-03-28T22:20:40.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.172+0000 7fcde0970740 1 --2- 172.21.15.16:0/2102643171 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55a3d4d2a7b0 0x55a3d4d2cba0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.172+0000 7fcde0970740 1 --2- 172.21.15.16:0/2102643171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d4d1a970 0x55a3d4d18440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.172+0000 7fcde0970740 1 --2- 172.21.15.16:0/2102643171 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3d4d11d90 0x55a3d4d17f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.172+0000 7fcdb6ffd640 0 client.27542 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:20:40.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.172+0000 7fcde0970740 1 --2- 172.21.15.16:0/2102643171 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3d4d11470 0x55a3d4d179c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.172+0000 7fcde0970740 1 -- 172.21.15.16:0/2102643171 >> 172.21.15.16:0/2102643171 conn(0x55a3d4d0bba0 msgr2=0x55a3d4d0d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:40.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.172+0000 7fcde0970740 1 -- 172.21.15.16:0/2102643171 shutdown_connections 2024-03-28T22:20:40.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.172+0000 7fcde0970740 1 -- 172.21.15.16:0/2102643171 wait complete. 2024-03-28T22:20:40.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.172+0000 7fcde0970740 20 client.27542 trim_cache size 0 max 16384 2024-03-28T22:20:40.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.186+0000 7fcde0970740 1 Processor -- start 2024-03-28T22:20:40.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.186+0000 7fcde0970740 1 -- start start 2024-03-28T22:20:40.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.186+0000 7fcde0970740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d4d11470 0x55a3d4d1a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.186+0000 7fcde0970740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3d4d11d90 0x55a3d4d264a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.186+0000 7fcde0970740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3d4d1a970 0x55a3d4d32140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.186+0000 7fcde0970740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd81b1d40 con 0x55a3d4d11470 2024-03-28T22:20:40.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.186+0000 7fcde0970740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd81af610 con 0x55a3d4d1a970 2024-03-28T22:20:40.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.186+0000 7fcde0970740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd81ad480 con 0x55a3d4d11d90 2024-03-28T22:20:40.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.186+0000 7fcdd6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d4d11470 0x55a3d4d1a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.186+0000 7fcdd6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d4d11470 0x55a3d4d1a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41040/0 (socket says 172.21.15.16:41040) 2024-03-28T22:20:40.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.186+0000 7fcdd6ffd640 1 -- 172.21.15.16:0/897273174 learned_addr learned my addr 172.21.15.16:0/897273174 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:40.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.186+0000 7fcdd6ffd640 1 --2- 172.21.15.16:0/897273174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3d4d1a970 0x55a3d4d32140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.186+0000 7fcdd77fe640 1 --2- 172.21.15.16:0/897273174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3d4d11d90 0x55a3d4d264a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.186+0000 7fcdd6ffd640 1 -- 172.21.15.16:0/897273174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3d4d11d90 msgr2=0x55a3d4d264a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:40.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.186+0000 7fcdd6ffd640 1 --2- 172.21.15.16:0/897273174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3d4d11d90 0x55a3d4d264a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.186+0000 7fcdd6ffd640 1 -- 172.21.15.16:0/897273174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3d4d1a970 msgr2=0x55a3d4d32140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:40.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.186+0000 7fcdd6ffd640 1 --2- 172.21.15.16:0/897273174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3d4d1a970 0x55a3d4d32140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.186+0000 7fcdd6ffd640 1 -- 172.21.15.16:0/897273174 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3d4d2cf10 con 0x55a3d4d11470 2024-03-28T22:20:40.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.186+0000 7fcdd6ffd640 1 --2- 172.21.15.16:0/897273174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d4d11470 0x55a3d4d1a0f0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc005d00 tx=0x7fcdcc005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be6e9cef264abc42 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:40.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.186+0000 7fcdbd7fa640 1 -- 172.21.15.16:0/897273174 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc067070 con 0x55a3d4d11470 2024-03-28T22:20:40.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.187+0000 7fcdbd7fa640 1 -- 172.21.15.16:0/897273174 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdcc0404a0 con 0x55a3d4d11470 2024-03-28T22:20:40.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.187+0000 7fcdbd7fa640 1 -- 172.21.15.16:0/897273174 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc04e050 con 0x55a3d4d11470 2024-03-28T22:20:40.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.187+0000 7fcde0970740 1 -- 172.21.15.16:0/897273174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d4d11470 msgr2=0x55a3d4d1a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:40.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.187+0000 7fcde0970740 1 --2- 172.21.15.16:0/897273174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d4d11470 0x55a3d4d1a0f0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc005d00 tx=0x7fcdcc005c90 comp rx=0 tx=0).stop 2024-03-28T22:20:40.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.187+0000 7fcde0970740 1 -- 172.21.15.16:0/897273174 shutdown_connections 2024-03-28T22:20:40.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.187+0000 7fcde0970740 1 --2- 172.21.15.16:0/897273174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3d4d1a970 0x55a3d4d32140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.187+0000 7fcde0970740 1 --2- 172.21.15.16:0/897273174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3d4d11d90 0x55a3d4d264a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.187+0000 7fcde0970740 1 --2- 172.21.15.16:0/897273174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d4d11470 0x55a3d4d1a0f0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.187+0000 7fcde0970740 1 -- 172.21.15.16:0/897273174 >> 172.21.15.16:0/897273174 conn(0x55a3d4d0bba0 msgr2=0x55a3d4d0ad40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:40.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.187+0000 7fcde0970740 1 -- 172.21.15.16:0/897273174 shutdown_connections 2024-03-28T22:20:40.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.187+0000 7fcde0970740 1 -- 172.21.15.16:0/897273174 wait complete. 2024-03-28T22:20:40.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.187+0000 7fcde0970740 1 -- start start 2024-03-28T22:20:40.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.188+0000 7fcde0970740 1 Processor -- start 2024-03-28T22:20:40.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.188+0000 7fcde0970740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3d4d11470 0x55a3d4d146c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.188+0000 7fcde0970740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d4d11d90 0x55a3d4d14c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.188+0000 7fcde0970740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3d4d1a970 0x55a3d4d15140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.188+0000 7fcde0970740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd81adad0 con 0x55a3d4d11d90 2024-03-28T22:20:40.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.188+0000 7fcde0970740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd81aef40 con 0x55a3d4d11470 2024-03-28T22:20:40.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.188+0000 7fcde0970740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd81ae1a0 con 0x55a3d4d1a970 2024-03-28T22:20:40.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.188+0000 7fcde0970740 10 client.0 _ll_register_callbacks cb 0x7fcdcc02c9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:20:40.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.188+0000 7fcdd77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d4d11d90 0x55a3d4d14c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.188+0000 7fcdd77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d4d11d90 0x55a3d4d14c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41052/0 (socket says 172.21.15.16:41052) 2024-03-28T22:20:40.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.188+0000 7fcdd77fe640 1 -- 172.21.15.16:0/437223770 learned_addr learned my addr 172.21.15.16:0/437223770 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:40.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.188+0000 7fcdd6ffd640 1 --2- 172.21.15.16:0/437223770 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3d4d11470 0x55a3d4d146c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.188+0000 7fcdd6ffd640 1 --2- 172.21.15.16:0/437223770 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3d4d1a970 0x55a3d4d15140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.188+0000 7fcdd77fe640 1 -- 172.21.15.16:0/437223770 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3d4d1a970 msgr2=0x55a3d4d15140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:40.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.188+0000 7fcdd77fe640 1 --2- 172.21.15.16:0/437223770 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3d4d1a970 0x55a3d4d15140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.188+0000 7fcdd77fe640 1 -- 172.21.15.16:0/437223770 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3d4d11470 msgr2=0x55a3d4d146c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:40.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.188+0000 7fcdd77fe640 1 --2- 172.21.15.16:0/437223770 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3d4d11470 0x55a3d4d146c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.188+0000 7fcdd77fe640 1 -- 172.21.15.16:0/437223770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3d4d8f890 con 0x55a3d4d11d90 2024-03-28T22:20:40.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.188+0000 7fcdd77fe640 1 --2- 172.21.15.16:0/437223770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d4d11d90 0x55a3d4d14c00 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8101730 tx=0x7fcdc8004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a2b92f1dcccb9a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:40.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.188+0000 7fcdb6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:20:40.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.189+0000 7fcdb6ffd640 1 -- 172.21.15.16:0/437223770 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc8016070 con 0x55a3d4d11d90 2024-03-28T22:20:40.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.189+0000 7fcdb6ffd640 1 -- 172.21.15.16:0/437223770 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8527+0+0 (secure 0 0 0) 0x7fcdc8003cd0 con 0x55a3d4d11d90 2024-03-28T22:20:40.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.189+0000 7fcde0970740 1 -- 172.21.15.16:0/437223770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcdcc0404a0 con 0x55a3d4d11d90 2024-03-28T22:20:40.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.189+0000 7fcdb6ffd640 1 -- 172.21.15.16:0/437223770 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcdc801e3b0 con 0x55a3d4d11d90 2024-03-28T22:20:40.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.189+0000 7fcde0970740 10 client.17646 fetch_fsmap learned FSMap version 27 2024-03-28T22:20:40.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.189+0000 7fcde0970740 1 -- 172.21.15.16:0/437223770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a3d4d90b40 con 0x55a3d4d11d90 2024-03-28T22:20:40.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.189+0000 7fcdb6ffd640 1 -- 172.21.15.16:0/437223770 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdcc0404a0 con 0x55a3d4d11d90 2024-03-28T22:20:40.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.189+0000 7fcdb6ffd640 1 -- 172.21.15.16:0/437223770 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7fcdc80047a0 con 0x55a3d4d11d90 2024-03-28T22:20:40.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.189+0000 7fcde0970740 10 client.17646 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:20:40.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.189+0000 7fcde0970740 10 client.17646 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-28T22:20:40.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.189+0000 7fcde0970740 20 client.17646 populate_metadata read hostname 'smithi016' 2024-03-28T22:20:40.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.189+0000 7fcde0970740 1 --2- 172.21.15.16:0/437223770 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55a3d4d912f0 0x55a3d4d93710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.189+0000 7fcde0970740 4 client.17646 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:20:40.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.189+0000 7fcde0970740 1 -- 172.21.15.16:0/437223770 --> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a3d4c7ca40 con 0x55a3d4d912f0 2024-03-28T22:20:40.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.189+0000 7fcdd6ffd640 1 --2- 172.21.15.16:0/437223770 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55a3d4d912f0 0x55a3d4d93710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.190+0000 7fcdb6ffd640 10 client.17646 ms_handle_connect on v2:172.21.15.53:6836/349877573 2024-03-28T22:20:40.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.190+0000 7fcdd6ffd640 1 --2- 172.21.15.16:0/437223770 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55a3d4d912f0 0x55a3d4d93710 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b1795e39684f6c5b server_cookie=3f30c1a4372feb2b in_seq=0 out_seq=0 2024-03-28T22:20:40.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.190+0000 7fcdb6ffd640 1 -- 172.21.15.16:0/437223770 <== mds.1 v2:172.21.15.53:6836/349877573 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a3d4c7ca40 con 0x55a3d4d912f0 2024-03-28T22:20:40.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.190+0000 7fcdb6ffd640 10 client.17646 handle_command_reply: tid=0 2024-03-28T22:20:40.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.190+0000 7fcde0970740 1 client.17646 shutdown 2024-03-28T22:20:40.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.190+0000 7fcde0970740 1 -- 172.21.15.16:0/437223770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d4d11d90 msgr2=0x55a3d4d14c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:40.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.190+0000 7fcde0970740 1 --2- 172.21.15.16:0/437223770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d4d11d90 0x55a3d4d14c00 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8101730 tx=0x7fcdc8004410 comp rx=0 tx=0).stop 2024-03-28T22:20:40.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.190+0000 7fcde0970740 1 -- 172.21.15.16:0/437223770 shutdown_connections 2024-03-28T22:20:40.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.190+0000 7fcde0970740 1 --2- 172.21.15.16:0/437223770 >> [v2:172.21.15.53:6836/349877573,v1:172.21.15.53:6837/349877573] conn(0x55a3d4d912f0 0x55a3d4d93710 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.190+0000 7fcde0970740 1 --2- 172.21.15.16:0/437223770 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a3d4d1a970 0x55a3d4d15140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.190+0000 7fcde0970740 1 --2- 172.21.15.16:0/437223770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a3d4d11d90 0x55a3d4d14c00 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.190+0000 7fcdb6ffd640 0 client.17646 ms_handle_reset on v2:172.21.15.53:6836/349877573 2024-03-28T22:20:40.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.190+0000 7fcde0970740 1 --2- 172.21.15.16:0/437223770 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3d4d11470 0x55a3d4d146c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.190+0000 7fcde0970740 1 -- 172.21.15.16:0/437223770 >> 172.21.15.16:0/437223770 conn(0x55a3d4d0bba0 msgr2=0x55a3d4d0a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:40.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.190+0000 7fcde0970740 1 -- 172.21.15.16:0/437223770 shutdown_connections 2024-03-28T22:20:40.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.190+0000 7fcde0970740 1 -- 172.21.15.16:0/437223770 wait complete. 2024-03-28T22:20:40.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.190+0000 7fcde0970740 20 client.17646 trim_cache size 0 max 16384 2024-03-28T22:20:40.201 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:20:40.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.191+0000 7fcdddcbb640 1 -- 172.21.15.16:0/2817405190 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fcdc00a1ef0 msgr2=0x7fcdc00a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:40.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.191+0000 7fcdddcbb640 1 --2- 172.21.15.16:0/2817405190 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fcdc00a1ef0 0x7fcdc00a43b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fcdd0096c80 tx=0x7fcdd0085040 comp rx=0 tx=0).stop 2024-03-28T22:20:40.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.191+0000 7fcdddcbb640 1 -- 172.21.15.16:0/2817405190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdd8106690 msgr2=0x7fcdd8102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:40.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.191+0000 7fcdddcbb640 1 --2- 172.21.15.16:0/2817405190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdd8106690 0x7fcdd8102d10 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fcdd000da40 tx=0x7fcdd000df00 comp rx=0 tx=0).stop 2024-03-28T22:20:40.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.191+0000 7fcdddcbb640 1 -- 172.21.15.16:0/2817405190 shutdown_connections 2024-03-28T22:20:40.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.191+0000 7fcdddcbb640 1 --2- 172.21.15.16:0/2817405190 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fcdc00a1ef0 0x7fcdc00a43b0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.191+0000 7fcdddcbb640 1 --2- 172.21.15.16:0/2817405190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdd8106690 0x7fcdd8102d10 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.191+0000 7fcdddcbb640 1 --2- 172.21.15.16:0/2817405190 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcdd8100fb0 0x7fcdd81027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.191+0000 7fcdddcbb640 1 --2- 172.21.15.16:0/2817405190 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcdd81005f0 0x7fcdd8102290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.191+0000 7fcdddcbb640 1 -- 172.21.15.16:0/2817405190 >> 172.21.15.16:0/2817405190 conn(0x7fcdd8077800 msgr2=0x7fcdd80ff8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:40.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.191+0000 7fcdddcbb640 1 -- 172.21.15.16:0/2817405190 shutdown_connections 2024-03-28T22:20:40.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.191+0000 7fcdddcbb640 1 -- 172.21.15.16:0/2817405190 wait complete. 2024-03-28T22:20:40.204 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:20:40.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.309+0000 7f7cb1fb6640 1 -- 172.21.15.16:0/244525539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cac1032e0 msgr2=0x7f7cac10b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:40.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.309+0000 7f7cb1fb6640 1 --2- 172.21.15.16:0/244525539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cac1032e0 0x7f7cac10b7c0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f7ca0009900 tx=0x7f7ca002bcd0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.309+0000 7f7cb1fb6640 1 -- 172.21.15.16:0/244525539 shutdown_connections 2024-03-28T22:20:40.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.309+0000 7f7cb1fb6640 1 --2- 172.21.15.16:0/244525539 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7cac10be30 0x7f7cac10e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.309+0000 7f7cb1fb6640 1 --2- 172.21.15.16:0/244525539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cac1032e0 0x7f7cac10b7c0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.309+0000 7f7cb1fb6640 1 --2- 172.21.15.16:0/244525539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7cac1029c0 0x7f7cac102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.309+0000 7f7cb1fb6640 1 -- 172.21.15.16:0/244525539 >> 172.21.15.16:0/244525539 conn(0x7f7cac0777c0 msgr2=0x7f7cac077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:40.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.309+0000 7f7cb1fb6640 1 -- 172.21.15.16:0/244525539 shutdown_connections 2024-03-28T22:20:40.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.309+0000 7f7cb1fb6640 1 -- 172.21.15.16:0/244525539 wait complete. 2024-03-28T22:20:40.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.310+0000 7f7cb1fb6640 1 Processor -- start 2024-03-28T22:20:40.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.310+0000 7f7cb1fb6640 1 -- start start 2024-03-28T22:20:40.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.310+0000 7f7cb1fb6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cac1029c0 0x7f7cac198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.310+0000 7f7cb1fb6640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7cac1032e0 0x7f7cac198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.310+0000 7f7cb1fb6640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7cac10be30 0x7f7cac19c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.310+0000 7f7cb1fb6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7cac111c00 con 0x7f7cac1029c0 2024-03-28T22:20:40.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.310+0000 7f7cb1fb6640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7cac111a80 con 0x7f7cac1032e0 2024-03-28T22:20:40.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.310+0000 7f7cb1fb6640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7cac111d80 con 0x7f7cac10be30 2024-03-28T22:20:40.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.310+0000 7f7cab7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cac1029c0 0x7f7cac198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.310+0000 7f7cab7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cac1029c0 0x7f7cac198120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41072/0 (socket says 172.21.15.16:41072) 2024-03-28T22:20:40.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.310+0000 7f7cab7fe640 1 -- 172.21.15.16:0/62947120 learned_addr learned my addr 172.21.15.16:0/62947120 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:40.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.310+0000 7f7caaffd640 1 --2- 172.21.15.16:0/62947120 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7cac1032e0 0x7f7cac198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.310+0000 7f7cabfff640 1 --2- 172.21.15.16:0/62947120 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7cac10be30 0x7f7cac19c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.311+0000 7f7cab7fe640 1 -- 172.21.15.16:0/62947120 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7cac10be30 msgr2=0x7f7cac19c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:40.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.311+0000 7f7cab7fe640 1 --2- 172.21.15.16:0/62947120 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7cac10be30 0x7f7cac19c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.311+0000 7f7cab7fe640 1 -- 172.21.15.16:0/62947120 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7cac1032e0 msgr2=0x7f7cac198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:40.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.311+0000 7f7cab7fe640 1 --2- 172.21.15.16:0/62947120 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7cac1032e0 0x7f7cac198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.311+0000 7f7cab7fe640 1 -- 172.21.15.16:0/62947120 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cac19d110 con 0x7f7cac1029c0 2024-03-28T22:20:40.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.311+0000 7f7cab7fe640 1 --2- 172.21.15.16:0/62947120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cac1029c0 0x7f7cac198120 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c00d950 tx=0x7f7c9c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:40.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.311+0000 7f7ca8ff9640 1 -- 172.21.15.16:0/62947120 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c9c017070 con 0x7f7cac1029c0 2024-03-28T22:20:40.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.311+0000 7f7ca8ff9640 1 -- 172.21.15.16:0/62947120 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c9c009c30 con 0x7f7cac1029c0 2024-03-28T22:20:40.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.311+0000 7f7cb1fb6640 1 -- 172.21.15.16:0/62947120 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cac19d400 con 0x7f7cac1029c0 2024-03-28T22:20:40.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.311+0000 7f7ca8ff9640 1 -- 172.21.15.16:0/62947120 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c9c00f6b0 con 0x7f7cac1029c0 2024-03-28T22:20:40.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.311+0000 7f7cb1fb6640 1 -- 172.21.15.16:0/62947120 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cac1a4c90 con 0x7f7cac1029c0 2024-03-28T22:20:40.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.313+0000 7f7ca8ff9640 1 -- 172.21.15.16:0/62947120 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f7c9c0109d0 con 0x7f7cac1029c0 2024-03-28T22:20:40.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.314+0000 7f7cb4b93740 1 Processor -- start 2024-03-28T22:20:40.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.314+0000 7f7cb4b93740 1 -- start start 2024-03-28T22:20:40.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.314+0000 7f7ca8ff9640 1 --2- 172.21.15.16:0/62947120 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f7c940a1ef0 0x7f7c940a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.314+0000 7f7cb4b93740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56328d477470 0x56328d477850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.314+0000 7f7cb4b93740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56328d477d90 0x56328d480430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.314+0000 7f7cb4b93740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56328d480970 0x56328d482d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.314+0000 7f7cb4b93740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56328d3db1b0 con 0x56328d477d90 2024-03-28T22:20:40.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.314+0000 7f7cb4b93740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x56328d383c00 con 0x56328d477470 2024-03-28T22:20:40.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.314+0000 7f7cb4b93740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56328d3d7250 con 0x56328d480970 2024-03-28T22:20:40.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.314+0000 7f7caaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56328d477d90 0x56328d480430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.314+0000 7f7cabfff640 1 --2- 172.21.15.16:0/62947120 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f7c940a1ef0 0x7f7c940a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.314+0000 7f7cab7fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56328d477470 0x56328d477850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.314+0000 7f7caaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56328d477d90 0x56328d480430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41082/0 (socket says 172.21.15.16:41082) 2024-03-28T22:20:40.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.314+0000 7f7caaffd640 1 -- 172.21.15.16:0/676053677 learned_addr learned my addr 172.21.15.16:0/676053677 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:40.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.314+0000 7f7cabfff640 1 --2- 172.21.15.16:0/676053677 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56328d480970 0x56328d482d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.314+0000 7f7ca8ff9640 1 -- 172.21.15.16:0/62947120 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8527+0+0 (secure 0 0 0) 0x7f7c9c0cdd90 con 0x7f7cac1029c0 2024-03-28T22:20:40.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.314+0000 7f7caaffd640 1 -- 172.21.15.16:0/676053677 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56328d480970 msgr2=0x56328d482d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:40.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.314+0000 7f7caaffd640 1 --2- 172.21.15.16:0/676053677 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56328d480970 0x56328d482d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.314+0000 7f7caaffd640 1 -- 172.21.15.16:0/676053677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56328d477470 msgr2=0x56328d477850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:40.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.314+0000 7f7caaffd640 1 --2- 172.21.15.16:0/676053677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56328d477470 0x56328d477850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.314+0000 7f7caaffd640 1 -- 172.21.15.16:0/676053677 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56328d3a33d0 con 0x56328d477d90 2024-03-28T22:20:40.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.314+0000 7f7cab7fe640 1 --2- 172.21.15.16:0/676053677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56328d477470 0x56328d477850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:20:40.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.315+0000 7f7caaffd640 1 --2- 172.21.15.16:0/676053677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56328d477d90 0x56328d480430 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f7ca0009230 tx=0x7f7ca0005bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87058de8df3b54bf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:40.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.315+0000 7f7cabfff640 1 --2- 172.21.15.16:0/62947120 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f7c940a1ef0 0x7f7c940a43b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x56328d472d30 tx=0x7f7ca4009b60 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:40.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.315+0000 7f7c91ffb640 1 -- 172.21.15.16:0/676053677 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca0042070 con 0x56328d477d90 2024-03-28T22:20:40.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.315+0000 7f7c91ffb640 1 -- 172.21.15.16:0/676053677 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ca002c730 con 0x56328d477d90 2024-03-28T22:20:40.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.315+0000 7f7c91ffb640 1 -- 172.21.15.16:0/676053677 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca0004f10 con 0x56328d477d90 2024-03-28T22:20:40.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.315+0000 7f7cb4b93740 1 -- 172.21.15.16:0/676053677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56328d477d90 msgr2=0x56328d480430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:40.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.315+0000 7f7cb4b93740 1 --2- 172.21.15.16:0/676053677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56328d477d90 0x56328d480430 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f7ca0009230 tx=0x7f7ca0005bf0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.315+0000 7f7cb4b93740 1 -- 172.21.15.16:0/676053677 shutdown_connections 2024-03-28T22:20:40.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.315+0000 7f7cb4b93740 1 --2- 172.21.15.16:0/676053677 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56328d480970 0x56328d482d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.315+0000 7f7cb4b93740 1 --2- 172.21.15.16:0/676053677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56328d477d90 0x56328d480430 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.315+0000 7f7cb4b93740 1 --2- 172.21.15.16:0/676053677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56328d477470 0x56328d477850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.315+0000 7f7cb4b93740 1 -- 172.21.15.16:0/676053677 >> 172.21.15.16:0/676053677 conn(0x56328d471ba0 msgr2=0x56328d471f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:40.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.315+0000 7f7cb4b93740 1 -- 172.21.15.16:0/676053677 shutdown_connections 2024-03-28T22:20:40.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.315+0000 7f7cb4b93740 1 -- 172.21.15.16:0/676053677 wait complete. 2024-03-28T22:20:40.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.316+0000 7f7cb4b93740 1 -- start start 2024-03-28T22:20:40.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.316+0000 7f7cb4b93740 1 Processor -- start 2024-03-28T22:20:40.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.316+0000 7f7cb4b93740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56328d477470 0x56328d47d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.316+0000 7f7cb4b93740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56328d477d90 0x56328d47dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.316+0000 7f7cb4b93740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56328d480970 0x56328d47e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.316+0000 7f7cb4b93740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56328d483700 con 0x56328d477470 2024-03-28T22:20:40.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.316+0000 7f7cb4b93740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x56328d379470 con 0x56328d480970 2024-03-28T22:20:40.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.316+0000 7f7cb4b93740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56328d483880 con 0x56328d477d90 2024-03-28T22:20:40.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.316+0000 7f7cb4b93740 10 client.0 _ll_register_callbacks cb 0x56328d396cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:20:40.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.316+0000 7f7cab7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56328d477470 0x56328d47d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.316+0000 7f7cab7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56328d477470 0x56328d47d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41094/0 (socket says 172.21.15.16:41094) 2024-03-28T22:20:40.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.316+0000 7f7cab7fe640 1 -- 172.21.15.16:0/1878336746 learned_addr learned my addr 172.21.15.16:0/1878336746 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:40.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.316+0000 7f7cabfff640 1 --2- 172.21.15.16:0/1878336746 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56328d480970 0x56328d47e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.317+0000 7f7cab7fe640 1 -- 172.21.15.16:0/1878336746 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56328d477d90 msgr2=0x56328d47dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:40.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.317+0000 7f7cab7fe640 1 --2- 172.21.15.16:0/1878336746 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56328d477d90 0x56328d47dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.317+0000 7f7cab7fe640 1 -- 172.21.15.16:0/1878336746 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56328d480970 msgr2=0x56328d47e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:40.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.317+0000 7f7cab7fe640 1 --2- 172.21.15.16:0/1878336746 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56328d480970 0x56328d47e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.317+0000 7f7cab7fe640 1 -- 172.21.15.16:0/1878336746 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56328d0ef3f0 con 0x56328d477470 2024-03-28T22:20:40.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.317+0000 7f7cab7fe640 1 --2- 172.21.15.16:0/1878336746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56328d477470 0x56328d47d9a0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c0cb4e0 tx=0x7f7c9c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f10ca66126cf846e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:40.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.317+0000 7f7c8affd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:20:40.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.317+0000 7f7c8affd640 1 -- 172.21.15.16:0/1878336746 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c9c020070 con 0x56328d477470 2024-03-28T22:20:40.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.317+0000 7f7c8affd640 1 -- 172.21.15.16:0/1878336746 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8527+0+0 (secure 0 0 0) 0x7f7c9c0850b0 con 0x56328d477470 2024-03-28T22:20:40.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.317+0000 7f7cb4b93740 1 -- 172.21.15.16:0/1878336746 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56328d3e86f0 con 0x56328d477470 2024-03-28T22:20:40.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.317+0000 7f7c8affd640 1 -- 172.21.15.16:0/1878336746 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7c9c09e870 con 0x56328d477470 2024-03-28T22:20:40.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.317+0000 7f7cb4b93740 10 client.17658 fetch_fsmap learned FSMap version 27 2024-03-28T22:20:40.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.317+0000 7f7cb4b93740 1 -- 172.21.15.16:0/1878336746 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56328d1b97d0 con 0x56328d477470 2024-03-28T22:20:40.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.317+0000 7f7c8affd640 1 -- 172.21.15.16:0/1878336746 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56328d3e86f0 con 0x56328d477470 2024-03-28T22:20:40.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.317+0000 7f7c8affd640 1 -- 172.21.15.16:0/1878336746 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f7c9c07dce0 con 0x56328d477470 2024-03-28T22:20:40.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.317+0000 7f7cb4b93740 10 client.17658 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:20:40.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.317+0000 7f7cb4b93740 10 client.17658 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:20:40.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.317+0000 7f7cb4b93740 20 client.17658 populate_metadata read hostname 'smithi016' 2024-03-28T22:20:40.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.317+0000 7f7cb4b93740 1 --2- 172.21.15.16:0/1878336746 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x56328d4908d0 0x56328d492d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.317+0000 7f7cb4b93740 4 client.17658 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:20:40.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.317+0000 7f7cb4b93740 1 -- 172.21.15.16:0/1878336746 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56328d3e2a40 con 0x56328d4908d0 2024-03-28T22:20:40.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.318+0000 7f7caaffd640 1 --2- 172.21.15.16:0/1878336746 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x56328d4908d0 0x56328d492d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.318+0000 7f7caaffd640 1 --2- 172.21.15.16:0/1878336746 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x56328d4908d0 0x56328d492d20 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48c9c7c0cb8c9067 server_cookie=bb8713ac541fc2e9 in_seq=0 out_seq=0 2024-03-28T22:20:40.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.318+0000 7f7c8affd640 10 client.17658 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:20:40.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.319+0000 7f7c8affd640 1 -- 172.21.15.16:0/1878336746 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56328d3e2a40 con 0x56328d4908d0 2024-03-28T22:20:40.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.319+0000 7f7c8affd640 10 client.17658 handle_command_reply: tid=0 2024-03-28T22:20:40.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.319+0000 7f7cb4b93740 1 client.17658 shutdown 2024-03-28T22:20:40.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.319+0000 7f7cb4b93740 1 -- 172.21.15.16:0/1878336746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56328d477470 msgr2=0x56328d47d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:40.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.319+0000 7f7cb4b93740 1 --2- 172.21.15.16:0/1878336746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56328d477470 0x56328d47d9a0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c0cb4e0 tx=0x7f7c9c005f90 comp rx=0 tx=0).stop 2024-03-28T22:20:40.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.319+0000 7f7cb4b93740 1 -- 172.21.15.16:0/1878336746 shutdown_connections 2024-03-28T22:20:40.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.319+0000 7f7cb4b93740 1 --2- 172.21.15.16:0/1878336746 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x56328d4908d0 0x56328d492d20 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.319+0000 7f7cb4b93740 1 --2- 172.21.15.16:0/1878336746 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56328d480970 0x56328d47e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.319+0000 7f7cb4b93740 1 --2- 172.21.15.16:0/1878336746 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56328d477d90 0x56328d47dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.319+0000 7f7c8affd640 0 client.17658 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:20:40.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.319+0000 7f7cb4b93740 1 --2- 172.21.15.16:0/1878336746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56328d477470 0x56328d47d9a0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.319+0000 7f7cb4b93740 1 -- 172.21.15.16:0/1878336746 >> 172.21.15.16:0/1878336746 conn(0x56328d471ba0 msgr2=0x56328d4738f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:40.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.319+0000 7f7cb4b93740 1 -- 172.21.15.16:0/1878336746 shutdown_connections 2024-03-28T22:20:40.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.319+0000 7f7cb4b93740 1 -- 172.21.15.16:0/1878336746 wait complete. 2024-03-28T22:20:40.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.319+0000 7f7cb4b93740 20 client.17658 trim_cache size 0 max 16384 2024-03-28T22:20:40.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.332+0000 7f7cb4b93740 1 Processor -- start 2024-03-28T22:20:40.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.332+0000 7f7cb4b93740 1 -- start start 2024-03-28T22:20:40.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.332+0000 7f7cb4b93740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56328d477470 0x56328d488ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.332+0000 7f7cb4b93740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56328d477d90 0x56328d48c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.332+0000 7f7cb4b93740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56328d480970 0x56328d4985f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.332+0000 7f7cb4b93740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7cac1afcd0 con 0x56328d480970 2024-03-28T22:20:40.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.332+0000 7f7cb4b93740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7cac1ad5a0 con 0x56328d477d90 2024-03-28T22:20:40.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.332+0000 7f7cb4b93740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7cac1ab410 con 0x56328d477470 2024-03-28T22:20:40.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.332+0000 7f7cabfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56328d480970 0x56328d4985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.332+0000 7f7cabfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56328d480970 0x56328d4985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41096/0 (socket says 172.21.15.16:41096) 2024-03-28T22:20:40.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.333+0000 7f7caaffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56328d477d90 0x56328d48c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.333+0000 7f7cabfff640 1 -- 172.21.15.16:0/3952318559 learned_addr learned my addr 172.21.15.16:0/3952318559 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:40.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.333+0000 7f7cab7fe640 1 --2- 172.21.15.16:0/3952318559 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56328d477470 0x56328d488ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.333+0000 7f7cabfff640 1 -- 172.21.15.16:0/3952318559 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56328d477470 msgr2=0x56328d488ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:40.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.333+0000 7f7cabfff640 1 --2- 172.21.15.16:0/3952318559 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56328d477470 0x56328d488ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.333+0000 7f7cabfff640 1 -- 172.21.15.16:0/3952318559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56328d477d90 msgr2=0x56328d48c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:40.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.333+0000 7f7cabfff640 1 --2- 172.21.15.16:0/3952318559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56328d477d90 0x56328d48c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.333+0000 7f7cabfff640 1 -- 172.21.15.16:0/3952318559 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56328d47bf10 con 0x56328d480970 2024-03-28T22:20:40.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.333+0000 7f7cab7fe640 1 --2- 172.21.15.16:0/3952318559 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56328d477470 0x56328d488ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:20:40.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.333+0000 7f7caaffd640 1 --2- 172.21.15.16:0/3952318559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56328d477d90 0x56328d48c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:20:40.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.333+0000 7f7cabfff640 1 --2- 172.21.15.16:0/3952318559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56328d480970 0x56328d4985f0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4003ca0 tx=0x7f7ca4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f4994d1126e55ef server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:40.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.333+0000 7f7c917fa640 1 -- 172.21.15.16:0/3952318559 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca4013e60 con 0x56328d480970 2024-03-28T22:20:40.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.333+0000 7f7c917fa640 1 -- 172.21.15.16:0/3952318559 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ca4003cd0 con 0x56328d480970 2024-03-28T22:20:40.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.333+0000 7f7c917fa640 1 -- 172.21.15.16:0/3952318559 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca40293e0 con 0x56328d480970 2024-03-28T22:20:40.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.334+0000 7f7cb4b93740 1 -- 172.21.15.16:0/3952318559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56328d480970 msgr2=0x56328d4985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:40.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.334+0000 7f7cb4b93740 1 --2- 172.21.15.16:0/3952318559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56328d480970 0x56328d4985f0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4003ca0 tx=0x7f7ca4004ab0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.334+0000 7f7cb4b93740 1 -- 172.21.15.16:0/3952318559 shutdown_connections 2024-03-28T22:20:40.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.334+0000 7f7cb4b93740 1 --2- 172.21.15.16:0/3952318559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56328d480970 0x56328d4985f0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.334+0000 7f7cb4b93740 1 --2- 172.21.15.16:0/3952318559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56328d477d90 0x56328d48c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.334+0000 7f7cb4b93740 1 --2- 172.21.15.16:0/3952318559 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56328d477470 0x56328d488ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.334+0000 7f7cb4b93740 1 -- 172.21.15.16:0/3952318559 >> 172.21.15.16:0/3952318559 conn(0x56328d471ba0 msgr2=0x56328d470d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:40.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.334+0000 7f7cb4b93740 1 -- 172.21.15.16:0/3952318559 shutdown_connections 2024-03-28T22:20:40.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.334+0000 7f7cb4b93740 1 -- 172.21.15.16:0/3952318559 wait complete. 2024-03-28T22:20:40.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.334+0000 7f7cb4b93740 1 -- start start 2024-03-28T22:20:40.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.334+0000 7f7cb4b93740 1 Processor -- start 2024-03-28T22:20:40.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.335+0000 7f7cb4b93740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56328d477470 0x56328d47b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.335+0000 7f7cb4b93740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56328d477d90 0x56328d479c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.335+0000 7f7cb4b93740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56328d480970 0x56328d47a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.335+0000 7f7cb4b93740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7cac1aba60 con 0x56328d477470 2024-03-28T22:20:40.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.335+0000 7f7cb4b93740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7cac1aced0 con 0x56328d480970 2024-03-28T22:20:40.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.335+0000 7f7cb4b93740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7cac1ac130 con 0x56328d477d90 2024-03-28T22:20:40.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.335+0000 7f7cb4b93740 10 client.0 _ll_register_callbacks cb 0x7f7c9c09a130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:20:40.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.335+0000 7f7cab7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56328d477470 0x56328d47b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.335+0000 7f7cab7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56328d477470 0x56328d47b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41112/0 (socket says 172.21.15.16:41112) 2024-03-28T22:20:40.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.335+0000 7f7cab7fe640 1 -- 172.21.15.16:0/313021233 learned_addr learned my addr 172.21.15.16:0/313021233 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:40.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.335+0000 7f7cabfff640 1 --2- 172.21.15.16:0/313021233 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56328d480970 0x56328d47a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.335+0000 7f7caaffd640 1 --2- 172.21.15.16:0/313021233 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56328d477d90 0x56328d479c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.335+0000 7f7cab7fe640 1 -- 172.21.15.16:0/313021233 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56328d477d90 msgr2=0x56328d479c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:40.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.335+0000 7f7cab7fe640 1 --2- 172.21.15.16:0/313021233 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56328d477d90 0x56328d479c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.335+0000 7f7cab7fe640 1 -- 172.21.15.16:0/313021233 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56328d480970 msgr2=0x56328d47a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:20:40.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.335+0000 7f7cab7fe640 1 --2- 172.21.15.16:0/313021233 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56328d480970 0x56328d47a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.335+0000 7f7cab7fe640 1 -- 172.21.15.16:0/313021233 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56328d47a970 con 0x56328d477470 2024-03-28T22:20:40.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.335+0000 7f7caaffd640 1 --2- 172.21.15.16:0/313021233 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56328d477d90 0x56328d479c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:20:40.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.335+0000 7f7cab7fe640 1 --2- 172.21.15.16:0/313021233 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56328d477470 0x56328d47b4b0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c09a760 tx=0x7f7c9c09a890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28c3f6138f91c940 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:40.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.335+0000 7f7c8affd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-03-28T22:20:40.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.335+0000 7f7c8affd640 1 -- 172.21.15.16:0/313021233 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c9c020040 con 0x56328d477470 2024-03-28T22:20:40.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.335+0000 7f7c8affd640 1 -- 172.21.15.16:0/313021233 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8527+0+0 (secure 0 0 0) 0x7f7c9c0cbdd0 con 0x56328d477470 2024-03-28T22:20:40.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.335+0000 7f7cb4b93740 1 -- 172.21.15.16:0/313021233 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ca4003cd0 con 0x56328d477470 2024-03-28T22:20:40.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.335+0000 7f7c8affd640 1 -- 172.21.15.16:0/313021233 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7c9c0a2ec0 con 0x56328d477470 2024-03-28T22:20:40.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.335+0000 7f7cb4b93740 10 client.17664 fetch_fsmap learned FSMap version 27 2024-03-28T22:20:40.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.335+0000 7f7cb4b93740 1 -- 172.21.15.16:0/313021233 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56328d4fbc50 con 0x56328d477470 2024-03-28T22:20:40.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.336+0000 7f7c8affd640 1 -- 172.21.15.16:0/313021233 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ca4003cd0 con 0x56328d477470 2024-03-28T22:20:40.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.336+0000 7f7c8affd640 1 -- 172.21.15.16:0/313021233 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f7c9c099cb0 con 0x56328d477470 2024-03-28T22:20:40.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.336+0000 7f7cb4b93740 10 client.17664 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T22:20:40.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.336+0000 7f7cb4b93740 10 client.17664 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:20:40.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.336+0000 7f7cb4b93740 20 client.17664 populate_metadata read hostname 'smithi016' 2024-03-28T22:20:40.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.336+0000 7f7cb4b93740 1 --2- 172.21.15.16:0/313021233 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x56328d4908d0 0x56328d4fe3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:40.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.336+0000 7f7cb4b93740 4 client.17664 mds_command: new command op to 24566 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:20:40.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.336+0000 7f7cb4b93740 1 -- 172.21.15.16:0/313021233 --> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56328d3606a0 con 0x56328d4908d0 2024-03-28T22:20:40.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.336+0000 7f7caaffd640 1 --2- 172.21.15.16:0/313021233 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x56328d4908d0 0x56328d4fe3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:40.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.336+0000 7f7caaffd640 1 --2- 172.21.15.16:0/313021233 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x56328d4908d0 0x56328d4fe3b0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=363eba83c7943119 server_cookie=545c4424198b7b89 in_seq=0 out_seq=0 2024-03-28T22:20:40.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.336+0000 7f7c8affd640 10 client.17664 ms_handle_connect on v2:172.21.15.53:6838/3714153956 2024-03-28T22:20:40.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.337+0000 7f7c8affd640 1 -- 172.21.15.16:0/313021233 <== mds.0 v2:172.21.15.53:6838/3714153956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56328d3606a0 con 0x56328d4908d0 2024-03-28T22:20:40.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.337+0000 7f7c8affd640 10 client.17664 handle_command_reply: tid=0 2024-03-28T22:20:40.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.337+0000 7f7cb4b93740 1 client.17664 shutdown 2024-03-28T22:20:40.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.337+0000 7f7cb4b93740 1 -- 172.21.15.16:0/313021233 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56328d477470 msgr2=0x56328d47b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:40.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.337+0000 7f7cb4b93740 1 --2- 172.21.15.16:0/313021233 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56328d477470 0x56328d47b4b0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c09a760 tx=0x7f7c9c09a890 comp rx=0 tx=0).stop 2024-03-28T22:20:40.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.337+0000 7f7cb4b93740 1 -- 172.21.15.16:0/313021233 shutdown_connections 2024-03-28T22:20:40.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.337+0000 7f7cb4b93740 1 --2- 172.21.15.16:0/313021233 >> [v2:172.21.15.53:6838/3714153956,v1:172.21.15.53:6839/3714153956] conn(0x56328d4908d0 0x56328d4fe3b0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.337+0000 7f7cb4b93740 1 --2- 172.21.15.16:0/313021233 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56328d480970 0x56328d47a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.337+0000 7f7c8affd640 0 client.17664 ms_handle_reset on v2:172.21.15.53:6838/3714153956 2024-03-28T22:20:40.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.337+0000 7f7cb4b93740 1 --2- 172.21.15.16:0/313021233 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56328d477d90 0x56328d479c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.337+0000 7f7cb4b93740 1 --2- 172.21.15.16:0/313021233 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56328d477470 0x56328d47b4b0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.337+0000 7f7cb4b93740 1 -- 172.21.15.16:0/313021233 >> 172.21.15.16:0/313021233 conn(0x56328d471ba0 msgr2=0x56328d47d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:40.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.337+0000 7f7cb4b93740 1 -- 172.21.15.16:0/313021233 shutdown_connections 2024-03-28T22:20:40.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.337+0000 7f7cb4b93740 1 -- 172.21.15.16:0/313021233 wait complete. 2024-03-28T22:20:40.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.337+0000 7f7cb4b93740 20 client.17664 trim_cache size 0 max 16384 2024-03-28T22:20:40.352 INFO:teuthology.orchestra.run.smithi016.stdout:[] 2024-03-28T22:20:40.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.337+0000 7f7cb1fb6640 1 -- 172.21.15.16:0/62947120 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f7c940a1ef0 msgr2=0x7f7c940a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:40.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.337+0000 7f7cb1fb6640 1 --2- 172.21.15.16:0/62947120 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f7c940a1ef0 0x7f7c940a43b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x56328d472d30 tx=0x7f7ca4009b60 comp rx=0 tx=0).stop 2024-03-28T22:20:40.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.337+0000 7f7cb1fb6640 1 -- 172.21.15.16:0/62947120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cac1029c0 msgr2=0x7f7cac198120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:40.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.337+0000 7f7cb1fb6640 1 --2- 172.21.15.16:0/62947120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cac1029c0 0x7f7cac198120 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c00d950 tx=0x7f7c9c00de10 comp rx=0 tx=0).stop 2024-03-28T22:20:40.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.337+0000 7f7cb1fb6640 1 -- 172.21.15.16:0/62947120 shutdown_connections 2024-03-28T22:20:40.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.337+0000 7f7cb1fb6640 1 --2- 172.21.15.16:0/62947120 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f7c940a1ef0 0x7f7c940a43b0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.337+0000 7f7cb1fb6640 1 --2- 172.21.15.16:0/62947120 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7cac10be30 0x7f7cac19c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.337+0000 7f7cb1fb6640 1 --2- 172.21.15.16:0/62947120 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7cac1032e0 0x7f7cac198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.337+0000 7f7cb1fb6640 1 --2- 172.21.15.16:0/62947120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cac1029c0 0x7f7cac198120 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:40.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.337+0000 7f7cb1fb6640 1 -- 172.21.15.16:0/62947120 >> 172.21.15.16:0/62947120 conn(0x7f7cac0777c0 msgr2=0x7f7cac0fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:40.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.338+0000 7f7cb1fb6640 1 -- 172.21.15.16:0/62947120 shutdown_connections 2024-03-28T22:20:40.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:40.338+0000 7f7cb1fb6640 1 -- 172.21.15.16:0/62947120 wait complete. 2024-03-28T22:20:40.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:40 smithi053 ceph-mon[35979]: pgmap v313: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T22:20:40.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:40 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/85712313' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:20:40.680 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:40 smithi063 ceph-mon[36249]: pgmap v313: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T22:20:40.680 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:40 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/85712313' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:20:40.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:40 smithi016 ceph-mon[32374]: pgmap v313: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-28T22:20:40.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:40 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/85712313' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:20:41.356 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-03-28T22:20:41.356 INFO:tasks.fwd_scrub:done joining 2024-03-28T22:20:41.356 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-28T22:20:41.367 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:20:41.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.483+0000 7fb11f11e640 1 -- 172.21.15.16:0/2718131411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1181016f0 msgr2=0x7fb118101b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:41.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.483+0000 7fb11f11e640 1 --2- 172.21.15.16:0/2718131411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1181016f0 0x7fb118101b70 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fb100009930 tx=0x7fb10002bc30 comp rx=0 tx=0).stop 2024-03-28T22:20:41.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.483+0000 7fb11f11e640 1 -- 172.21.15.16:0/2718131411 shutdown_connections 2024-03-28T22:20:41.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.483+0000 7fb11f11e640 1 --2- 172.21.15.16:0/2718131411 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1181020b0 0x7fb11810e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:41.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.483+0000 7fb11f11e640 1 --2- 172.21.15.16:0/2718131411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1181016f0 0x7fb118101b70 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:41.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.483+0000 7fb11f11e640 1 --2- 172.21.15.16:0/2718131411 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb118107700 0x7fb118107ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:41.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.483+0000 7fb11f11e640 1 -- 172.21.15.16:0/2718131411 >> 172.21.15.16:0/2718131411 conn(0x7fb1180fd200 msgr2=0x7fb1180ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:41.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.483+0000 7fb11f11e640 1 -- 172.21.15.16:0/2718131411 shutdown_connections 2024-03-28T22:20:41.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.483+0000 7fb11f11e640 1 -- 172.21.15.16:0/2718131411 wait complete. 2024-03-28T22:20:41.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.483+0000 7fb11f11e640 1 Processor -- start 2024-03-28T22:20:41.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.483+0000 7fb11f11e640 1 -- start start 2024-03-28T22:20:41.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.484+0000 7fb11f11e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1181016f0 0x7fb118073a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:41.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.484+0000 7fb11f11e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb1181020b0 0x7fb118073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:41.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.484+0000 7fb11f11e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb118107700 0x7fb1180744d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:41.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.484+0000 7fb11f11e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb118111b10 con 0x7fb1181016f0 2024-03-28T22:20:41.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.484+0000 7fb11f11e640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb118111990 con 0x7fb118107700 2024-03-28T22:20:41.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.484+0000 7fb11f11e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb118111c90 con 0x7fb1181020b0 2024-03-28T22:20:41.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.484+0000 7fb11ceba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1181016f0 0x7fb118073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:41.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.484+0000 7fb11ceba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1181016f0 0x7fb118073a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41706/0 (socket says 172.21.15.16:41706) 2024-03-28T22:20:41.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.484+0000 7fb11ceba640 1 -- 172.21.15.16:0/3242338280 learned_addr learned my addr 172.21.15.16:0/3242338280 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:41.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.484+0000 7fb11d6bb640 1 --2- 172.21.15.16:0/3242338280 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb118107700 0x7fb1180744d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:41.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.484+0000 7fb117fff640 1 --2- 172.21.15.16:0/3242338280 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb1181020b0 0x7fb118073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:41.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.484+0000 7fb11ceba640 1 -- 172.21.15.16:0/3242338280 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb1181020b0 msgr2=0x7fb118073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:41.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.484+0000 7fb11ceba640 1 --2- 172.21.15.16:0/3242338280 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb1181020b0 0x7fb118073f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:41.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.484+0000 7fb11ceba640 1 -- 172.21.15.16:0/3242338280 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb118107700 msgr2=0x7fb1180744d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:41.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.484+0000 7fb11ceba640 1 --2- 172.21.15.16:0/3242338280 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb118107700 0x7fb1180744d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:41.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.484+0000 7fb11ceba640 1 -- 172.21.15.16:0/3242338280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1181a6530 con 0x7fb1181016f0 2024-03-28T22:20:41.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.484+0000 7fb11ceba640 1 --2- 172.21.15.16:0/3242338280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1181016f0 0x7fb118073a50 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fb10800ea80 tx=0x7fb10800ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:41.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.484+0000 7fb115ffb640 1 -- 172.21.15.16:0/3242338280 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb10800ce40 con 0x7fb1181016f0 2024-03-28T22:20:41.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.484+0000 7fb115ffb640 1 -- 172.21.15.16:0/3242338280 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb108005020 con 0x7fb1181016f0 2024-03-28T22:20:41.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.484+0000 7fb115ffb640 1 -- 172.21.15.16:0/3242338280 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb108011760 con 0x7fb1181016f0 2024-03-28T22:20:41.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.485+0000 7fb11f11e640 1 -- 172.21.15.16:0/3242338280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1181a6820 con 0x7fb1181016f0 2024-03-28T22:20:41.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.485+0000 7fb11f11e640 1 -- 172.21.15.16:0/3242338280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1181a6cf0 con 0x7fb1181016f0 2024-03-28T22:20:41.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.486+0000 7fb11f11e640 1 -- 172.21.15.16:0/3242338280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb118068a20 con 0x7fb1181016f0 2024-03-28T22:20:41.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.487+0000 7fb115ffb640 1 -- 172.21.15.16:0/3242338280 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7fb1080051d0 con 0x7fb1181016f0 2024-03-28T22:20:41.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.487+0000 7fb115ffb640 1 --2- 172.21.15.16:0/3242338280 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fb1040a1ef0 0x7fb1040a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:41.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.488+0000 7fb115ffb640 1 -- 172.21.15.16:0/3242338280 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8527+0+0 (secure 0 0 0) 0x7fb1080ce7f0 con 0x7fb1181016f0 2024-03-28T22:20:41.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.488+0000 7fb117fff640 1 --2- 172.21.15.16:0/3242338280 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fb1040a1ef0 0x7fb1040a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:41.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.488+0000 7fb117fff640 1 --2- 172.21.15.16:0/3242338280 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fb1040a1ef0 0x7fb1040a43b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb100006010 tx=0x7fb100000bc0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:41.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.492+0000 7fb115ffb640 1 -- 172.21.15.16:0/3242338280 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb108016030 con 0x7fb1181016f0 2024-03-28T22:20:41.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.675+0000 7fb11f11e640 1 -- 172.21.15.16:0/3242338280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb118109c60 con 0x7fb1181016f0 2024-03-28T22:20:41.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.677+0000 7fb115ffb640 1 -- 172.21.15.16:0/3242338280 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) ==== 74+0+22967 (secure 0 0 0) 0x7fb10809b880 con 0x7fb1181016f0 2024-03-28T22:20:41.678 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:20:41.678 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":97,"fsid":"798d9884-ed4e-11ee-95d0-87774f69a715","created":"2024-03-28T22:00:27.216099+0000","modified":"2024-03-28T22:20:17.454175+0000","last_up_change":"2024-03-28T22:07:40.251141+0000","last_in_change":"2024-03-28T22:07:19.456451+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:03:59.327832+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:08:28.123529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:08:28.919037+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":96,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"46e7c8f8-ecc8-43d6-ae53-0655728f3240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6803","nonce":1589692483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6805","nonce":1589692483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6809","nonce":1589692483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6807","nonce":1589692483}]},"public_addr":"172.21.15.16:6803/1589692483","cluster_addr":"172.21.15.16:6805/1589692483","heartbeat_back_addr":"172.21.15.16:6809/1589692483","heartbeat_front_addr":"172.21.15.16:6807/1589692483","state":["exists","up"]},{"osd":1,"uuid":"af6d9c22-2830-4ce2-a547-011df0c6cbad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6801","nonce":795531234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6803","nonce":795531234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6807","nonce":795531234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6805","nonce":795531234}]},"public_addr":"172.21.15.53:6801/795531234","cluster_addr":"172.21.15.53:6803/795531234","heartbeat_back_addr":"172.21.15.53:6807/795531234","heartbeat_front_addr":"172.21.15.53:6805/795531234","state":["exists","up"]},{"osd":2,"uuid":"b10adca9-110c-4f87-85e0-b7542709d31a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6801","nonce":2214131086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6803","nonce":2214131086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6807","nonce":2214131086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6805","nonce":2214131086}]},"public_addr":"172.21.15.63:6801/2214131086","cluster_addr":"172.21.15.63:6803/2214131086","heartbeat_back_addr":"172.21.15.63:6807/2214131086","heartbeat_front_addr":"172.21.15.63:6805/2214131086","state":["exists","up"]},{"osd":3,"uuid":"c83cf4bb-395a-4786-abf7-74ca93a33eb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6811","nonce":13176789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6813","nonce":13176789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6817","nonce":13176789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6815","nonce":13176789}]},"public_addr":"172.21.15.16:6811/13176789","cluster_addr":"172.21.15.16:6813/13176789","heartbeat_back_addr":"172.21.15.16:6817/13176789","heartbeat_front_addr":"172.21.15.16:6815/13176789","state":["exists","up"]},{"osd":4,"uuid":"941b3e54-47d6-4e85-8069-e730c3fa8eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6809","nonce":772161849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6811","nonce":772161849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6815","nonce":772161849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6813","nonce":772161849}]},"public_addr":"172.21.15.53:6809/772161849","cluster_addr":"172.21.15.53:6811/772161849","heartbeat_back_addr":"172.21.15.53:6815/772161849","heartbeat_front_addr":"172.21.15.53:6813/772161849","state":["exists","up"]},{"osd":5,"uuid":"51d89826-1d60-40fc-8fd8-09bfe1ab4125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6809","nonce":1602554852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6811","nonce":1602554852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6815","nonce":1602554852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6813","nonce":1602554852}]},"public_addr":"172.21.15.63:6809/1602554852","cluster_addr":"172.21.15.63:6811/1602554852","heartbeat_back_addr":"172.21.15.63:6815/1602554852","heartbeat_front_addr":"172.21.15.63:6813/1602554852","state":["exists","up"]},{"osd":6,"uuid":"978efe5d-2c55-4ff8-8a1e-9f891388945e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6819","nonce":4290459224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6821","nonce":4290459224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6825","nonce":4290459224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6823","nonce":4290459224}]},"public_addr":"172.21.15.16:6819/4290459224","cluster_addr":"172.21.15.16:6821/4290459224","heartbeat_back_addr":"172.21.15.16:6825/4290459224","heartbeat_front_addr":"172.21.15.16:6823/4290459224","state":["exists","up"]},{"osd":7,"uuid":"50d489cb-dbb8-48df-b868-b29edf12e04d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6817","nonce":1056467525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6819","nonce":1056467525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6823","nonce":1056467525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6821","nonce":1056467525}]},"public_addr":"172.21.15.53:6817/1056467525","cluster_addr":"172.21.15.53:6819/1056467525","heartbeat_back_addr":"172.21.15.53:6823/1056467525","heartbeat_front_addr":"172.21.15.53:6821/1056467525","state":["exists","up"]},{"osd":8,"uuid":"49b39414-cc9c-4057-ae3f-cc109cd979da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6817","nonce":2918516794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6819","nonce":2918516794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6823","nonce":2918516794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6821","nonce":2918516794}]},"public_addr":"172.21.15.63:6817/2918516794","cluster_addr":"172.21.15.63:6819/2918516794","heartbeat_back_addr":"172.21.15.63:6823/2918516794","heartbeat_front_addr":"172.21.15.63:6821/2918516794","state":["exists","up"]},{"osd":9,"uuid":"4c873e51-b783-4eb6-83a3-fee31722d60b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6827","nonce":1585624367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6829","nonce":1585624367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6833","nonce":1585624367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6831","nonce":1585624367}]},"public_addr":"172.21.15.16:6827/1585624367","cluster_addr":"172.21.15.16:6829/1585624367","heartbeat_back_addr":"172.21.15.16:6833/1585624367","heartbeat_front_addr":"172.21.15.16:6831/1585624367","state":["exists","up"]},{"osd":10,"uuid":"085a78bf-e2ad-481a-85a3-71d1fc549a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6825","nonce":3357446484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6827","nonce":3357446484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6831","nonce":3357446484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6829","nonce":3357446484}]},"public_addr":"172.21.15.53:6825/3357446484","cluster_addr":"172.21.15.53:6827/3357446484","heartbeat_back_addr":"172.21.15.53:6831/3357446484","heartbeat_front_addr":"172.21.15.53:6829/3357446484","state":["exists","up"]},{"osd":11,"uuid":"13302aa3-08b9-4639-9042-27d850b4d5f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6825","nonce":2167866570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6827","nonce":2167866570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6831","nonce":2167866570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6829","nonce":2167866570}]},"public_addr":"172.21.15.63:6825/2167866570","cluster_addr":"172.21.15.63:6827/2167866570","heartbeat_back_addr":"172.21.15.63:6831/2167866570","heartbeat_front_addr":"172.21.15.63:6829/2167866570","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:07.450323+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:31.363172+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:55.295021+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:19.794760+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:44.063492+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:08.195064+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:33.655328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:57.639414+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:22.402394+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:47.462050+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:12.226307+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:37.495625+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3631746518":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6801/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/2311413235":"2024-03-29T22:10:34.301217+0000","172.21.15.16:0/3116652786":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/317557779":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/1062264551":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/996598130":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6801/21986004":"2024-03-29T22:10:34.301217+0000","172.21.15.16:6801/1832422779":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/3821408150":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/3070600736":"2024-03-29T22:10:34.301217+0000","172.21.15.16:0/2117914406":"2024-03-29T22:10:34.301217+0000","172.21.15.16:0/4029580607":"2024-03-29T22:10:34.301217+0000","172.21.15.16:6800/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/2787995271":"2024-03-29T22:10:34.301217+0000","172.21.15.16:6800/1832422779":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6800/21986004":"2024-03-29T22:10:34.301217+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:20:41.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.679+0000 7fb11f11e640 1 -- 172.21.15.16:0/3242338280 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fb1040a1ef0 msgr2=0x7fb1040a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:41.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.679+0000 7fb11f11e640 1 --2- 172.21.15.16:0/3242338280 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fb1040a1ef0 0x7fb1040a43b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb100006010 tx=0x7fb100000bc0 comp rx=0 tx=0).stop 2024-03-28T22:20:41.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.679+0000 7fb11f11e640 1 -- 172.21.15.16:0/3242338280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1181016f0 msgr2=0x7fb118073a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:41.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.679+0000 7fb11f11e640 1 --2- 172.21.15.16:0/3242338280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1181016f0 0x7fb118073a50 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fb10800ea80 tx=0x7fb10800ef40 comp rx=0 tx=0).stop 2024-03-28T22:20:41.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.679+0000 7fb11f11e640 1 -- 172.21.15.16:0/3242338280 shutdown_connections 2024-03-28T22:20:41.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.679+0000 7fb11f11e640 1 --2- 172.21.15.16:0/3242338280 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fb1040a1ef0 0x7fb1040a43b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:41.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.679+0000 7fb11f11e640 1 --2- 172.21.15.16:0/3242338280 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb118107700 0x7fb1180744d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:41.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.679+0000 7fb11f11e640 1 --2- 172.21.15.16:0/3242338280 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb1181020b0 0x7fb118073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:41.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.679+0000 7fb11f11e640 1 --2- 172.21.15.16:0/3242338280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1181016f0 0x7fb118073a50 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:41.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.679+0000 7fb11f11e640 1 -- 172.21.15.16:0/3242338280 >> 172.21.15.16:0/3242338280 conn(0x7fb1180fd200 msgr2=0x7fb1180fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:41.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.679+0000 7fb11f11e640 1 -- 172.21.15.16:0/3242338280 shutdown_connections 2024-03-28T22:20:41.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.679+0000 7fb11f11e640 1 -- 172.21.15.16:0/3242338280 wait complete. 2024-03-28T22:20:41.690 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T22:03:59.327832+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T22:08:28.123529+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T22:08:28.919037+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '96', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 96, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T22:20:41.691 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T22:20:41.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.796+0000 7f29284cd640 1 -- 172.21.15.16:0/2125749820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29201016d0 msgr2=0x7f2920101b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:41.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.796+0000 7f29284cd640 1 --2- 172.21.15.16:0/2125749820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29201016d0 0x7f2920101b50 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f2910009930 tx=0x7f291002bc30 comp rx=0 tx=0).stop 2024-03-28T22:20:41.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.797+0000 7f29284cd640 1 -- 172.21.15.16:0/2125749820 shutdown_connections 2024-03-28T22:20:41.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.797+0000 7f29284cd640 1 --2- 172.21.15.16:0/2125749820 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2920102090 0x7f292010e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:41.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.797+0000 7f29284cd640 1 --2- 172.21.15.16:0/2125749820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29201016d0 0x7f2920101b50 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:41.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.797+0000 7f29284cd640 1 --2- 172.21.15.16:0/2125749820 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29201076e0 0x7f2920107ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:41.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.797+0000 7f29284cd640 1 -- 172.21.15.16:0/2125749820 >> 172.21.15.16:0/2125749820 conn(0x7f29200fd1e0 msgr2=0x7f29200ff600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:41.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.797+0000 7f29284cd640 1 -- 172.21.15.16:0/2125749820 shutdown_connections 2024-03-28T22:20:41.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.797+0000 7f29284cd640 1 -- 172.21.15.16:0/2125749820 wait complete. 2024-03-28T22:20:41.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.797+0000 7f29284cd640 1 Processor -- start 2024-03-28T22:20:41.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.797+0000 7f29284cd640 1 -- start start 2024-03-28T22:20:41.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.797+0000 7f29284cd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29201016d0 0x7f2920197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:41.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.797+0000 7f29284cd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2920102090 0x7f2920198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:41.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.797+0000 7f29284cd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29201076e0 0x7f292019c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:41.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.797+0000 7f29284cd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2920111af0 con 0x7f2920102090 2024-03-28T22:20:41.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.797+0000 7f29284cd640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f2920111970 con 0x7f29201076e0 2024-03-28T22:20:41.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.797+0000 7f29284cd640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2920111c70 con 0x7f29201016d0 2024-03-28T22:20:41.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.797+0000 7f2925a68640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2920102090 0x7f2920198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:41.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.797+0000 7f2925a68640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2920102090 0x7f2920198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41728/0 (socket says 172.21.15.16:41728) 2024-03-28T22:20:41.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.797+0000 7f2925a68640 1 -- 172.21.15.16:0/2391227249 learned_addr learned my addr 172.21.15.16:0/2391227249 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:41.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.797+0000 7f2926a6a640 1 --2- 172.21.15.16:0/2391227249 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29201076e0 0x7f292019c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:41.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.798+0000 7f2926269640 1 --2- 172.21.15.16:0/2391227249 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29201016d0 0x7f2920197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:41.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.798+0000 7f2925a68640 1 -- 172.21.15.16:0/2391227249 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29201016d0 msgr2=0x7f2920197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:41.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.798+0000 7f2925a68640 1 --2- 172.21.15.16:0/2391227249 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29201016d0 0x7f2920197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:41.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.798+0000 7f2925a68640 1 -- 172.21.15.16:0/2391227249 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29201076e0 msgr2=0x7f292019c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:41.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.798+0000 7f2925a68640 1 --2- 172.21.15.16:0/2391227249 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29201076e0 0x7f292019c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:41.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.798+0000 7f2925a68640 1 -- 172.21.15.16:0/2391227249 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f292019cfe0 con 0x7f2920102090 2024-03-28T22:20:41.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.798+0000 7f2926269640 1 --2- 172.21.15.16:0/2391227249 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29201016d0 0x7f2920197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:20:41.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.798+0000 7f2926a6a640 1 --2- 172.21.15.16:0/2391227249 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29201076e0 0x7f292019c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:20:41.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.798+0000 7f2925a68640 1 --2- 172.21.15.16:0/2391227249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2920102090 0x7f2920198530 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f2910009230 tx=0x7f291002cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:41.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.798+0000 7f29177fe640 1 -- 172.21.15.16:0/2391227249 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2910041070 con 0x7f2920102090 2024-03-28T22:20:41.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.798+0000 7f29177fe640 1 -- 172.21.15.16:0/2391227249 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2910004450 con 0x7f2920102090 2024-03-28T22:20:41.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.798+0000 7f29284cd640 1 -- 172.21.15.16:0/2391227249 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f292019d270 con 0x7f2920102090 2024-03-28T22:20:41.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.798+0000 7f29177fe640 1 -- 172.21.15.16:0/2391227249 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2910003e50 con 0x7f2920102090 2024-03-28T22:20:41.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.798+0000 7f29284cd640 1 -- 172.21.15.16:0/2391227249 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29201a4b60 con 0x7f2920102090 2024-03-28T22:20:41.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.799+0000 7f29284cd640 1 -- 172.21.15.16:0/2391227249 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29200685d0 con 0x7f2920102090 2024-03-28T22:20:41.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.800+0000 7f29177fe640 1 -- 172.21.15.16:0/2391227249 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f2910044070 con 0x7f2920102090 2024-03-28T22:20:41.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.801+0000 7f29177fe640 1 --2- 172.21.15.16:0/2391227249 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f290c0a2090 0x7f290c0a4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:41.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.801+0000 7f29177fe640 1 -- 172.21.15.16:0/2391227249 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8527+0+0 (secure 0 0 0) 0x7f29100efad0 con 0x7f2920102090 2024-03-28T22:20:41.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.801+0000 7f2926269640 1 --2- 172.21.15.16:0/2391227249 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f290c0a2090 0x7f290c0a4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:41.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.801+0000 7f2926269640 1 --2- 172.21.15.16:0/2391227249 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f290c0a2090 0x7f290c0a4550 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2918005fd0 tx=0x7f29180093c0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:41.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.805+0000 7f29177fe640 1 -- 172.21.15.16:0/2391227249 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f29100bccd0 con 0x7f2920102090 2024-03-28T22:20:41.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.986+0000 7f29284cd640 1 -- 172.21.15.16:0/2391227249 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f29201096b0 con 0x7f2920102090 2024-03-28T22:20:41.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.987+0000 7f29177fe640 1 -- 172.21.15.16:0/2391227249 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v97) ==== 93+0+10 (secure 0 0 0) 0x7f29100bc5d0 con 0x7f2920102090 2024-03-28T22:20:41.987 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-28T22:20:41.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.989+0000 7f29284cd640 1 -- 172.21.15.16:0/2391227249 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f290c0a2090 msgr2=0x7f290c0a4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:41.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.989+0000 7f29284cd640 1 --2- 172.21.15.16:0/2391227249 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f290c0a2090 0x7f290c0a4550 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2918005fd0 tx=0x7f29180093c0 comp rx=0 tx=0).stop 2024-03-28T22:20:41.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.989+0000 7f29284cd640 1 -- 172.21.15.16:0/2391227249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2920102090 msgr2=0x7f2920198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:41.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.989+0000 7f29284cd640 1 --2- 172.21.15.16:0/2391227249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2920102090 0x7f2920198530 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f2910009230 tx=0x7f291002cba0 comp rx=0 tx=0).stop 2024-03-28T22:20:41.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.989+0000 7f29284cd640 1 -- 172.21.15.16:0/2391227249 shutdown_connections 2024-03-28T22:20:41.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.989+0000 7f29284cd640 1 --2- 172.21.15.16:0/2391227249 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f290c0a2090 0x7f290c0a4550 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:41.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.989+0000 7f29284cd640 1 --2- 172.21.15.16:0/2391227249 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29201076e0 0x7f292019c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:41.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.989+0000 7f29284cd640 1 --2- 172.21.15.16:0/2391227249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2920102090 0x7f2920198530 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:41.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.989+0000 7f29284cd640 1 --2- 172.21.15.16:0/2391227249 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29201016d0 0x7f2920197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:41.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.989+0000 7f29284cd640 1 -- 172.21.15.16:0/2391227249 >> 172.21.15.16:0/2391227249 conn(0x7f29200fd1e0 msgr2=0x7f29200fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:41.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.989+0000 7f29284cd640 1 -- 172.21.15.16:0/2391227249 shutdown_connections 2024-03-28T22:20:41.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:41.989+0000 7f29284cd640 1 -- 172.21.15.16:0/2391227249 wait complete. 2024-03-28T22:20:42.000 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T22:20:42.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.105+0000 7fb9637fd640 1 -- 172.21.15.16:0/1984638041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb95c100fa0 msgr2=0x7fb95c10d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:42.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.105+0000 7fb9637fd640 1 --2- 172.21.15.16:0/1984638041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb95c100fa0 0x7fb95c10d4b0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fb954009cd0 tx=0x7fb95402c0f0 comp rx=0 tx=0).stop 2024-03-28T22:20:42.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.105+0000 7fb9637fd640 1 -- 172.21.15.16:0/1984638041 shutdown_connections 2024-03-28T22:20:42.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.105+0000 7fb9637fd640 1 --2- 172.21.15.16:0/1984638041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb95c100fa0 0x7fb95c10d4b0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:42.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.105+0000 7fb9637fd640 1 --2- 172.21.15.16:0/1984638041 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb95c1005e0 0x7fb95c100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:42.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.105+0000 7fb9637fd640 1 --2- 172.21.15.16:0/1984638041 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb95c106680 0x7fb95c106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:42.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.105+0000 7fb9637fd640 1 -- 172.21.15.16:0/1984638041 >> 172.21.15.16:0/1984638041 conn(0x7fb95c077770 msgr2=0x7fb95c077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:42.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.105+0000 7fb9637fd640 1 -- 172.21.15.16:0/1984638041 shutdown_connections 2024-03-28T22:20:42.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.105+0000 7fb9637fd640 1 -- 172.21.15.16:0/1984638041 wait complete. 2024-03-28T22:20:42.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.106+0000 7fb9637fd640 1 Processor -- start 2024-03-28T22:20:42.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.106+0000 7fb9637fd640 1 -- start start 2024-03-28T22:20:42.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.106+0000 7fb9637fd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb95c1005e0 0x7fb95c197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:42.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.106+0000 7fb9637fd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb95c100fa0 0x7fb95c1984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:42.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.106+0000 7fb9637fd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb95c106680 0x7fb95c19c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:42.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.106+0000 7fb9637fd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb95c110a00 con 0x7fb95c100fa0 2024-03-28T22:20:42.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.106+0000 7fb9637fd640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb95c110880 con 0x7fb95c1005e0 2024-03-28T22:20:42.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.106+0000 7fb9637fd640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb95c110b80 con 0x7fb95c106680 2024-03-28T22:20:42.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.106+0000 7fb960d98640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb95c100fa0 0x7fb95c1984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:42.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.106+0000 7fb960d98640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb95c100fa0 0x7fb95c1984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41756/0 (socket says 172.21.15.16:41756) 2024-03-28T22:20:42.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.106+0000 7fb961599640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb95c1005e0 0x7fb95c197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:42.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.106+0000 7fb960d98640 1 -- 172.21.15.16:0/2407516669 learned_addr learned my addr 172.21.15.16:0/2407516669 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:42.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.106+0000 7fb961d9a640 1 --2- 172.21.15.16:0/2407516669 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb95c106680 0x7fb95c19c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:42.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.106+0000 7fb960d98640 1 -- 172.21.15.16:0/2407516669 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb95c106680 msgr2=0x7fb95c19c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:42.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.106+0000 7fb960d98640 1 --2- 172.21.15.16:0/2407516669 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb95c106680 0x7fb95c19c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:42.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.106+0000 7fb960d98640 1 -- 172.21.15.16:0/2407516669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb95c1005e0 msgr2=0x7fb95c197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:42.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.106+0000 7fb960d98640 1 --2- 172.21.15.16:0/2407516669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb95c1005e0 0x7fb95c197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:42.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.106+0000 7fb960d98640 1 -- 172.21.15.16:0/2407516669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb95c19cf50 con 0x7fb95c100fa0 2024-03-28T22:20:42.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.106+0000 7fb960d98640 1 --2- 172.21.15.16:0/2407516669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb95c100fa0 0x7fb95c1984a0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fb95800b4a0 tx=0x7fb95800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:42.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.107+0000 7fb9527fc640 1 -- 172.21.15.16:0/2407516669 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb958014070 con 0x7fb95c100fa0 2024-03-28T22:20:42.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.107+0000 7fb9527fc640 1 -- 172.21.15.16:0/2407516669 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb958010af0 con 0x7fb95c100fa0 2024-03-28T22:20:42.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.107+0000 7fb9637fd640 1 -- 172.21.15.16:0/2407516669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb95c19d240 con 0x7fb95c100fa0 2024-03-28T22:20:42.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.107+0000 7fb9527fc640 1 -- 172.21.15.16:0/2407516669 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb958017d10 con 0x7fb95c100fa0 2024-03-28T22:20:42.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.107+0000 7fb9637fd640 1 -- 172.21.15.16:0/2407516669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb95c1a4ad0 con 0x7fb95c100fa0 2024-03-28T22:20:42.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.108+0000 7fb9637fd640 1 -- 172.21.15.16:0/2407516669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb95c101d20 con 0x7fb95c100fa0 2024-03-28T22:20:42.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.109+0000 7fb9527fc640 1 -- 172.21.15.16:0/2407516669 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7fb958017070 con 0x7fb95c100fa0 2024-03-28T22:20:42.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.110+0000 7fb9527fc640 1 --2- 172.21.15.16:0/2407516669 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fb9480a1ef0 0x7fb9480a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:42.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.110+0000 7fb961599640 1 --2- 172.21.15.16:0/2407516669 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fb9480a1ef0 0x7fb9480a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:42.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.110+0000 7fb9527fc640 1 -- 172.21.15.16:0/2407516669 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8527+0+0 (secure 0 0 0) 0x7fb9580cd820 con 0x7fb95c100fa0 2024-03-28T22:20:42.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.110+0000 7fb961599640 1 --2- 172.21.15.16:0/2407516669 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fb9480a1ef0 0x7fb9480a43b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb95c101720 tx=0x7fb94c005f90 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:42.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.113+0000 7fb9527fc640 1 -- 172.21.15.16:0/2407516669 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb95809aa20 con 0x7fb95c100fa0 2024-03-28T22:20:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.293+0000 7fb9637fd640 1 -- 172.21.15.16:0/2407516669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb95c1a4d70 con 0x7fb95c100fa0 2024-03-28T22:20:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.294+0000 7fb9527fc640 1 -- 172.21.15.16:0/2407516669 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v97) ==== 104+0+11 (secure 0 0 0) 0x7fb95809a320 con 0x7fb95c100fa0 2024-03-28T22:20:42.294 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-28T22:20:42.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.296+0000 7fb9637fd640 1 -- 172.21.15.16:0/2407516669 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fb9480a1ef0 msgr2=0x7fb9480a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:42.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.296+0000 7fb9637fd640 1 --2- 172.21.15.16:0/2407516669 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fb9480a1ef0 0x7fb9480a43b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb95c101720 tx=0x7fb94c005f90 comp rx=0 tx=0).stop 2024-03-28T22:20:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.296+0000 7fb9637fd640 1 -- 172.21.15.16:0/2407516669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb95c100fa0 msgr2=0x7fb95c1984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.296+0000 7fb9637fd640 1 --2- 172.21.15.16:0/2407516669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb95c100fa0 0x7fb95c1984a0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fb95800b4a0 tx=0x7fb95800b960 comp rx=0 tx=0).stop 2024-03-28T22:20:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.296+0000 7fb9637fd640 1 -- 172.21.15.16:0/2407516669 shutdown_connections 2024-03-28T22:20:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.296+0000 7fb9637fd640 1 --2- 172.21.15.16:0/2407516669 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fb9480a1ef0 0x7fb9480a43b0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.296+0000 7fb9637fd640 1 --2- 172.21.15.16:0/2407516669 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb95c106680 0x7fb95c19c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.296+0000 7fb9637fd640 1 --2- 172.21.15.16:0/2407516669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb95c100fa0 0x7fb95c1984a0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.296+0000 7fb9637fd640 1 --2- 172.21.15.16:0/2407516669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb95c1005e0 0x7fb95c197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.296+0000 7fb9637fd640 1 -- 172.21.15.16:0/2407516669 >> 172.21.15.16:0/2407516669 conn(0x7fb95c077770 msgr2=0x7fb95c0ff7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.296+0000 7fb9637fd640 1 -- 172.21.15.16:0/2407516669 shutdown_connections 2024-03-28T22:20:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.296+0000 7fb9637fd640 1 -- 172.21.15.16:0/2407516669 wait complete. 2024-03-28T22:20:42.307 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T22:20:42.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.413+0000 7f13f8264640 1 -- 172.21.15.16:0/690004897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13f01005c0 msgr2=0x7f13f0100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:42.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.413+0000 7f13f8264640 1 --2- 172.21.15.16:0/690004897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13f01005c0 0x7f13f0100a40 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f13e0009930 tx=0x7f13e002bc30 comp rx=0 tx=0).stop 2024-03-28T22:20:42.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.413+0000 7f13f8264640 1 -- 172.21.15.16:0/690004897 shutdown_connections 2024-03-28T22:20:42.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.413+0000 7f13f8264640 1 --2- 172.21.15.16:0/690004897 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13f0100f80 0x7f13f010d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:42.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.413+0000 7f13f8264640 1 --2- 172.21.15.16:0/690004897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13f01005c0 0x7f13f0100a40 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:42.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.413+0000 7f13f8264640 1 --2- 172.21.15.16:0/690004897 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f13f0106660 0x7f13f0106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:42.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.413+0000 7f13f8264640 1 -- 172.21.15.16:0/690004897 >> 172.21.15.16:0/690004897 conn(0x7f13f0077750 msgr2=0x7f13f0077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:42.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.413+0000 7f13f8264640 1 -- 172.21.15.16:0/690004897 shutdown_connections 2024-03-28T22:20:42.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.413+0000 7f13f8264640 1 -- 172.21.15.16:0/690004897 wait complete. 2024-03-28T22:20:42.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.414+0000 7f13f8264640 1 Processor -- start 2024-03-28T22:20:42.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.414+0000 7f13f8264640 1 -- start start 2024-03-28T22:20:42.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.414+0000 7f13f8264640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13f01005c0 0x7f13f0197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:42.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.414+0000 7f13f8264640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f13f0100f80 0x7f13f01984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:42.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.414+0000 7f13f8264640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13f0106660 0x7f13f019c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:42.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.414+0000 7f13f8264640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f13f01109a0 con 0x7f13f01005c0 2024-03-28T22:20:42.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.414+0000 7f13f8264640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f13f0110820 con 0x7f13f0106660 2024-03-28T22:20:42.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.414+0000 7f13f8264640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f13f0110b20 con 0x7f13f0100f80 2024-03-28T22:20:42.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.414+0000 7f13f6000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13f01005c0 0x7f13f0197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:42.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.414+0000 7f13f6000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13f01005c0 0x7f13f0197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41780/0 (socket says 172.21.15.16:41780) 2024-03-28T22:20:42.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.414+0000 7f13f6000640 1 -- 172.21.15.16:0/1531988680 learned_addr learned my addr 172.21.15.16:0/1531988680 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:42.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.414+0000 7f13f6801640 1 --2- 172.21.15.16:0/1531988680 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13f0106660 0x7f13f019c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:42.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.414+0000 7f13f57ff640 1 --2- 172.21.15.16:0/1531988680 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f13f0100f80 0x7f13f01984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:42.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.414+0000 7f13f6000640 1 -- 172.21.15.16:0/1531988680 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f13f0100f80 msgr2=0x7f13f01984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:42.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.414+0000 7f13f6000640 1 --2- 172.21.15.16:0/1531988680 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f13f0100f80 0x7f13f01984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:42.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.414+0000 7f13f6000640 1 -- 172.21.15.16:0/1531988680 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13f0106660 msgr2=0x7f13f019c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:42.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.414+0000 7f13f6000640 1 --2- 172.21.15.16:0/1531988680 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13f0106660 0x7f13f019c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:42.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.414+0000 7f13f6000640 1 -- 172.21.15.16:0/1531988680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13f019cf30 con 0x7f13f01005c0 2024-03-28T22:20:42.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.414+0000 7f13f57ff640 1 --2- 172.21.15.16:0/1531988680 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f13f0100f80 0x7f13f01984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:20:42.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.414+0000 7f13f6000640 1 --2- 172.21.15.16:0/1531988680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13f01005c0 0x7f13f0197f90 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f13ec00c970 tx=0x7f13ec00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:42.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.415+0000 7f13e6ffd640 1 -- 172.21.15.16:0/1531988680 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13ec004220 con 0x7f13f01005c0 2024-03-28T22:20:42.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.415+0000 7f13f8264640 1 -- 172.21.15.16:0/1531988680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13f019d220 con 0x7f13f01005c0 2024-03-28T22:20:42.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.415+0000 7f13e6ffd640 1 -- 172.21.15.16:0/1531988680 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13ec0043c0 con 0x7f13f01005c0 2024-03-28T22:20:42.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.415+0000 7f13e6ffd640 1 -- 172.21.15.16:0/1531988680 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13ec004a00 con 0x7f13f01005c0 2024-03-28T22:20:42.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.415+0000 7f13f8264640 1 -- 172.21.15.16:0/1531988680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13f01a4ab0 con 0x7f13f01005c0 2024-03-28T22:20:42.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.416+0000 7f13f8264640 1 -- 172.21.15.16:0/1531988680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13f0101d00 con 0x7f13f01005c0 2024-03-28T22:20:42.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.417+0000 7f13e6ffd640 1 -- 172.21.15.16:0/1531988680 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f13ec0079b0 con 0x7f13f01005c0 2024-03-28T22:20:42.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.417+0000 7f13e6ffd640 1 --2- 172.21.15.16:0/1531988680 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f13dc0a1ef0 0x7f13dc0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:42.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.417+0000 7f13f57ff640 1 --2- 172.21.15.16:0/1531988680 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f13dc0a1ef0 0x7f13dc0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:42.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.417+0000 7f13e6ffd640 1 -- 172.21.15.16:0/1531988680 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8527+0+0 (secure 0 0 0) 0x7f13ec0ce300 con 0x7f13f01005c0 2024-03-28T22:20:42.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.418+0000 7f13f57ff640 1 --2- 172.21.15.16:0/1531988680 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f13dc0a1ef0 0x7f13dc0a43b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f13e0006010 tx=0x7f13e002c7d0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:42.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.421+0000 7f13e6ffd640 1 -- 172.21.15.16:0/1531988680 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f13ec014030 con 0x7f13f01005c0 2024-03-28T22:20:42.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:42 smithi053 ceph-mon[35979]: pgmap v314: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-28T22:20:42.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:42 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3242338280' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:20:42.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:42 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2391227249' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:20:42.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.604+0000 7f13f8264640 1 -- 172.21.15.16:0/1531988680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f13f01a4ce0 con 0x7f13f01005c0 2024-03-28T22:20:42.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.604+0000 7f13e6ffd640 1 -- 172.21.15.16:0/1531988680 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v97) ==== 100+0+11 (secure 0 0 0) 0x7f13ec09b3f0 con 0x7f13f01005c0 2024-03-28T22:20:42.605 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-28T22:20:42.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.606+0000 7f13f8264640 1 -- 172.21.15.16:0/1531988680 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f13dc0a1ef0 msgr2=0x7f13dc0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:42.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.606+0000 7f13f8264640 1 --2- 172.21.15.16:0/1531988680 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f13dc0a1ef0 0x7f13dc0a43b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f13e0006010 tx=0x7f13e002c7d0 comp rx=0 tx=0).stop 2024-03-28T22:20:42.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.606+0000 7f13f8264640 1 -- 172.21.15.16:0/1531988680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13f01005c0 msgr2=0x7f13f0197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:42.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.606+0000 7f13f8264640 1 --2- 172.21.15.16:0/1531988680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13f01005c0 0x7f13f0197f90 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f13ec00c970 tx=0x7f13ec00ce30 comp rx=0 tx=0).stop 2024-03-28T22:20:42.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.606+0000 7f13f8264640 1 -- 172.21.15.16:0/1531988680 shutdown_connections 2024-03-28T22:20:42.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.606+0000 7f13f8264640 1 --2- 172.21.15.16:0/1531988680 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f13dc0a1ef0 0x7f13dc0a43b0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:42.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.606+0000 7f13f8264640 1 --2- 172.21.15.16:0/1531988680 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13f0106660 0x7f13f019c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:42.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.606+0000 7f13f8264640 1 --2- 172.21.15.16:0/1531988680 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f13f0100f80 0x7f13f01984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:42.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.606+0000 7f13f8264640 1 --2- 172.21.15.16:0/1531988680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13f01005c0 0x7f13f0197f90 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:42.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.606+0000 7f13f8264640 1 -- 172.21.15.16:0/1531988680 >> 172.21.15.16:0/1531988680 conn(0x7f13f0077750 msgr2=0x7f13f0104770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:42.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.606+0000 7f13f8264640 1 -- 172.21.15.16:0/1531988680 shutdown_connections 2024-03-28T22:20:42.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.606+0000 7f13f8264640 1 -- 172.21.15.16:0/1531988680 wait complete. 2024-03-28T22:20:42.616 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:42 smithi016 ceph-mon[32374]: pgmap v314: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-28T22:20:42.616 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:42 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3242338280' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:20:42.616 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:42 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2391227249' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:20:42.617 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-28T22:20:42.680 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:42 smithi063 ceph-mon[36249]: pgmap v314: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-28T22:20:42.680 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:42 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/3242338280' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:20:42.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:42 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2391227249' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:20:42.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.719+0000 7f071d33e640 1 -- 172.21.15.16:0/1737397597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07181001b0 msgr2=0x7f0718100590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:42.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.719+0000 7f071d33e640 1 --2- 172.21.15.16:0/1737397597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07181001b0 0x7f0718100590 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f0710009900 tx=0x7f071002bc30 comp rx=0 tx=0).stop 2024-03-28T22:20:42.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.719+0000 7f071d33e640 1 -- 172.21.15.16:0/1737397597 shutdown_connections 2024-03-28T22:20:42.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.719+0000 7f071d33e640 1 --2- 172.21.15.16:0/1737397597 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0718106a80 0x7f071810e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:42.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.719+0000 7f071d33e640 1 --2- 172.21.15.16:0/1737397597 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0718100b60 0x7f07181063a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:42.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.719+0000 7f071d33e640 1 --2- 172.21.15.16:0/1737397597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07181001b0 0x7f0718100590 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:42.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.719+0000 7f071d33e640 1 -- 172.21.15.16:0/1737397597 >> 172.21.15.16:0/1737397597 conn(0x7f0718076aa0 msgr2=0x7f0718076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:42.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.719+0000 7f071d33e640 1 -- 172.21.15.16:0/1737397597 shutdown_connections 2024-03-28T22:20:42.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.719+0000 7f071d33e640 1 -- 172.21.15.16:0/1737397597 wait complete. 2024-03-28T22:20:42.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.720+0000 7f071d33e640 1 Processor -- start 2024-03-28T22:20:42.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.720+0000 7f071d33e640 1 -- start start 2024-03-28T22:20:42.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.720+0000 7f071d33e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07181001b0 0x7f0718198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:42.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.720+0000 7f071d33e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0718106a80 0x7f0718198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:42.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.720+0000 7f071d33e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f071819c8c0 0x7f071819cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:42.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.720+0000 7f071d33e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0718111b40 con 0x7f071819c8c0 2024-03-28T22:20:42.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.720+0000 7f071d33e640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f07181119c0 con 0x7f07181001b0 2024-03-28T22:20:42.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.720+0000 7f071d33e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0718111cc0 con 0x7f0718106a80 2024-03-28T22:20:42.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.720+0000 7f07177fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f071819c8c0 0x7f071819cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:42.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.720+0000 7f0716ffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07181001b0 0x7f0718198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:42.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.720+0000 7f0716ffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07181001b0 0x7f0718198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:47058/0 (socket says 172.21.15.16:47058) 2024-03-28T22:20:42.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.720+0000 7f0716ffd640 1 -- 172.21.15.16:0/3959160727 learned_addr learned my addr 172.21.15.16:0/3959160727 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:20:42.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.720+0000 7f07167fc640 1 --2- 172.21.15.16:0/3959160727 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0718106a80 0x7f0718198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:42.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.721+0000 7f0716ffd640 1 -- 172.21.15.16:0/3959160727 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0718106a80 msgr2=0x7f0718198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:42.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.721+0000 7f0716ffd640 1 --2- 172.21.15.16:0/3959160727 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0718106a80 0x7f0718198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:42.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.721+0000 7f0716ffd640 1 -- 172.21.15.16:0/3959160727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f071819c8c0 msgr2=0x7f071819cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:42.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.721+0000 7f0716ffd640 1 --2- 172.21.15.16:0/3959160727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f071819c8c0 0x7f071819cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:42.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.721+0000 7f0716ffd640 1 -- 172.21.15.16:0/3959160727 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f071819d3b0 con 0x7f07181001b0 2024-03-28T22:20:42.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.721+0000 7f07177fe640 1 --2- 172.21.15.16:0/3959160727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f071819c8c0 0x7f071819cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:20:42.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.721+0000 7f0716ffd640 1 --2- 172.21.15.16:0/3959160727 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07181001b0 0x7f0718198000 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0710006010 tx=0x7f071002cb30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:42.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.721+0000 7f070ffff640 1 -- 172.21.15.16:0/3959160727 <== mon.2 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0710041070 con 0x7f07181001b0 2024-03-28T22:20:42.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.721+0000 7f071d33e640 1 -- 172.21.15.16:0/3959160727 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f071819d640 con 0x7f07181001b0 2024-03-28T22:20:42.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.721+0000 7f071d33e640 1 -- 172.21.15.16:0/3959160727 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07181a4e60 con 0x7f07181001b0 2024-03-28T22:20:42.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.721+0000 7f070ffff640 1 -- 172.21.15.16:0/3959160727 <== mon.2 v2:172.21.15.53:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0710004450 con 0x7f07181001b0 2024-03-28T22:20:42.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.721+0000 7f070ffff640 1 -- 172.21.15.16:0/3959160727 <== mon.2 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f071004b060 con 0x7f07181001b0 2024-03-28T22:20:42.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.722+0000 7f071d33e640 1 -- 172.21.15.16:0/3959160727 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0718068a20 con 0x7f07181001b0 2024-03-28T22:20:42.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.724+0000 7f070ffff640 1 -- 172.21.15.16:0/3959160727 <== mon.2 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f0710044070 con 0x7f07181001b0 2024-03-28T22:20:42.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.724+0000 7f070ffff640 1 --2- 172.21.15.16:0/3959160727 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f07040a1ef0 0x7f07040a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:20:42.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.724+0000 7f07167fc640 1 --2- 172.21.15.16:0/3959160727 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f07040a1ef0 0x7f07040a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:20:42.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.724+0000 7f070ffff640 1 -- 172.21.15.16:0/3959160727 <== mon.2 v2:172.21.15.53:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8527+0+0 (secure 0 0 0) 0x7f07100efa40 con 0x7f07181001b0 2024-03-28T22:20:42.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.725+0000 7f07167fc640 1 --2- 172.21.15.16:0/3959160727 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f07040a1ef0 0x7f07040a43b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f07180ffa50 tx=0x7f0700009110 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:20:42.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.728+0000 7f070ffff640 1 -- 172.21.15.16:0/3959160727 <== mon.2 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f07100bcc40 con 0x7f07181001b0 2024-03-28T22:20:42.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.957+0000 7f071d33e640 1 -- 172.21.15.16:0/3959160727 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f0718199350 con 0x7f07181001b0 2024-03-28T22:20:42.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.969+0000 7f070ffff640 1 -- 172.21.15.16:0/3959160727 <== mon.2 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) ==== 81+0+736930 (secure 0 0 0) 0x7f0718199350 con 0x7f07181001b0 2024-03-28T22:20:42.981 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:20:42.981 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:20:42.981 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 26, 2024-03-28T22:20:42.981 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:42.981 INFO:teuthology.orchestra.run.smithi016.stdout: "active_gid": 15441, 2024-03-28T22:20:42.981 INFO:teuthology.orchestra.run.smithi016.stdout: "active_name": "x", 2024-03-28T22:20:42.982 INFO:teuthology.orchestra.run.smithi016.stdout: "active_addrs": { 2024-03-28T22:20:42.982 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-03-28T22:20:42.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:42.982 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-03-28T22:20:42.982 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6800", 2024-03-28T22:20:42.982 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1819900910 2024-03-28T22:20:42.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:42.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:42.983 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-03-28T22:20:42.983 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6801", 2024-03-28T22:20:42.983 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1819900910 2024-03-28T22:20:42.983 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:42.983 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-03-28T22:20:42.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:42.983 INFO:teuthology.orchestra.run.smithi016.stdout: "active_addr": "172.21.15.16:6801/1819900910", 2024-03-28T22:20:42.983 INFO:teuthology.orchestra.run.smithi016.stdout: "active_change": "2024-03-28T22:10:34.301417+0000", 2024-03-28T22:20:42.983 INFO:teuthology.orchestra.run.smithi016.stdout: "active_mgr_features": 4540701547738038271, 2024-03-28T22:20:42.984 INFO:teuthology.orchestra.run.smithi016.stdout: "available": true, 2024-03-28T22:20:42.984 INFO:teuthology.orchestra.run.smithi016.stdout: "standbys": [ 2024-03-28T22:20:42.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:42.984 INFO:teuthology.orchestra.run.smithi016.stdout: "gid": 15438, 2024-03-28T22:20:42.984 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "y", 2024-03-28T22:20:42.984 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_features": 4540701547738038271, 2024-03-28T22:20:42.984 INFO:teuthology.orchestra.run.smithi016.stdout: "available_modules": [ 2024-03-28T22:20:42.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:42.985 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "alerts", 2024-03-28T22:20:42.985 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:42.985 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:42.985 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:42.985 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-03-28T22:20:42.985 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-03-28T22:20:42.985 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:42.985 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:42.985 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:42.986 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-03-28T22:20:42.986 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:42.986 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:42.986 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:42.986 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How frequently to reexamine health status", 2024-03-28T22:20:42.986 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:42.986 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:42.986 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:42.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:42.987 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:42.987 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:42.987 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:42.987 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:42.987 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:42.987 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:42.987 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:42.987 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:42.987 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:42.988 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:42.988 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:42.988 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:42.988 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:42.988 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:42.988 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:42.988 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:42.988 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:42.988 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:42.988 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:42.989 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:42.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:42.989 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:42.989 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:42.989 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:42.989 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:42.989 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:42.989 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:42.989 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:42.990 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:42.990 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:42.990 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:42.990 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:42.990 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:42.990 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:42.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:42.990 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:42.990 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:42.991 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:42.991 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:42.991 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:42.991 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:42.991 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:42.991 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:42.991 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:42.991 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:42.992 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:42.992 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:42.992 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:42.992 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:42.992 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:42.992 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:42.992 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:42.992 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:42.993 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:42.993 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:42.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:42.993 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:42.993 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:42.993 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:42.993 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:42.993 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:42.993 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:42.994 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:42.994 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:42.994 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:42.994 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:42.994 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:42.994 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:42.994 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:42.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:42.995 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_destination": { 2024-03-28T22:20:42.995 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_destination", 2024-03-28T22:20:42.995 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:42.995 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:42.995 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:42.995 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:42.995 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:42.996 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:42.996 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:42.996 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email address to send alerts to", 2024-03-28T22:20:42.996 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:42.996 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:42.996 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:42.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:42.997 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_from_name": { 2024-03-28T22:20:42.997 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_from_name", 2024-03-28T22:20:42.997 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:42.997 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:42.997 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:42.997 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "Ceph", 2024-03-28T22:20:42.997 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:42.997 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:42.998 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:42.998 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email From: name", 2024-03-28T22:20:42.998 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:42.998 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:42.998 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:42.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:42.998 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_host": { 2024-03-28T22:20:42.998 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_host", 2024-03-28T22:20:42.999 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:42.999 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:42.999 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:42.999 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:42.999 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:42.999 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:42.999 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:42.999 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP server", 2024-03-28T22:20:43.000 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.000 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.000 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.000 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_password": { 2024-03-28T22:20:43.000 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_password", 2024-03-28T22:20:43.000 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.000 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.001 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.001 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.001 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.001 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.001 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.001 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Password to authenticate with", 2024-03-28T22:20:43.001 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.001 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.001 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.002 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_port": { 2024-03-28T22:20:43.002 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_port", 2024-03-28T22:20:43.002 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.002 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.002 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.002 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "465", 2024-03-28T22:20:43.002 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.003 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.003 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.003 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP port", 2024-03-28T22:20:43.003 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.003 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.003 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.003 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_sender": { 2024-03-28T22:20:43.004 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_sender", 2024-03-28T22:20:43.004 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.004 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.004 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.004 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.004 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.004 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.004 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.005 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP envelope sender", 2024-03-28T22:20:43.005 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.005 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.005 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.005 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_ssl": { 2024-03-28T22:20:43.005 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_ssl", 2024-03-28T22:20:43.005 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.006 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.006 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.006 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.006 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.006 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.006 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.006 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-28T22:20:43.006 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.007 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.007 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.007 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_user": { 2024-03-28T22:20:43.007 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_user", 2024-03-28T22:20:43.007 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.007 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.007 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.007 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.008 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.008 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.008 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.008 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "User to authenticate as", 2024-03-28T22:20:43.008 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.008 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.008 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.008 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.009 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.009 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "balancer", 2024-03-28T22:20:43.009 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:43.009 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:43.009 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:43.009 INFO:teuthology.orchestra.run.smithi016.stdout: "active": { 2024-03-28T22:20:43.009 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "active", 2024-03-28T22:20:43.010 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.010 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.010 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.010 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.010 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.010 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.010 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.010 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically balance PGs across cluster", 2024-03-28T22:20:43.011 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.011 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.011 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.011 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_time": { 2024-03-28T22:20:43.011 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_time", 2024-03-28T22:20:43.011 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.011 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.011 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.012 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0000", 2024-03-28T22:20:43.012 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.012 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.012 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.012 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "beginning time of day to automatically balance", 2024-03-28T22:20:43.012 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T22:20:43.012 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.012 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.013 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_weekday": { 2024-03-28T22:20:43.013 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_weekday", 2024-03-28T22:20:43.013 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-03-28T22:20:43.013 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.013 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.013 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-03-28T22:20:43.013 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-03-28T22:20:43.013 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-03-28T22:20:43.014 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.014 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-28T22:20:43.014 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T22:20:43.014 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.014 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.014 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_max_iterations": { 2024-03-28T22:20:43.015 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_max_iterations", 2024-03-28T22:20:43.015 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-03-28T22:20:43.015 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.015 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.015 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "25", 2024-03-28T22:20:43.015 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-03-28T22:20:43.015 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "250", 2024-03-28T22:20:43.015 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.015 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-28T22:20:43.016 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.016 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.016 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.016 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_metrics": { 2024-03-28T22:20:43.016 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_metrics", 2024-03-28T22:20:43.016 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.016 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.017 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.017 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "pgs,objects,bytes", 2024-03-28T22:20:43.017 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.017 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.017 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.017 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-28T22:20:43.017 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-28T22:20:43.017 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.018 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.018 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_step": { 2024-03-28T22:20:43.018 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_step", 2024-03-28T22:20:43.018 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-03-28T22:20:43.018 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.018 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.018 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.5", 2024-03-28T22:20:43.018 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0.001", 2024-03-28T22:20:43.019 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "0.999", 2024-03-28T22:20:43.019 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.019 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "aggressiveness of optimization", 2024-03-28T22:20:43.019 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-28T22:20:43.019 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.019 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.020 INFO:teuthology.orchestra.run.smithi016.stdout: "end_time": { 2024-03-28T22:20:43.020 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_time", 2024-03-28T22:20:43.020 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.020 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.020 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.020 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2359", 2024-03-28T22:20:43.020 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.020 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.020 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.021 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ending time of day to automatically balance", 2024-03-28T22:20:43.021 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T22:20:43.021 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.021 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.021 INFO:teuthology.orchestra.run.smithi016.stdout: "end_weekday": { 2024-03-28T22:20:43.021 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_weekday", 2024-03-28T22:20:43.021 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-03-28T22:20:43.022 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.022 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.022 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-03-28T22:20:43.022 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-03-28T22:20:43.022 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-03-28T22:20:43.022 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.022 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-28T22:20:43.023 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T22:20:43.023 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.023 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.023 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:43.023 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:43.023 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.023 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.023 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.024 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.024 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.024 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.024 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.024 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.024 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.024 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.024 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.025 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.025 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.025 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.025 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.025 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.025 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.025 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.025 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:43.026 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:43.026 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.026 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.026 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.026 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.026 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.026 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.027 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.027 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.027 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.027 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.027 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.027 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:43.027 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:43.028 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.028 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.028 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.028 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:43.028 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.028 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.028 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.028 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.028 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.028 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.029 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.029 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.029 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.029 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.029 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.029 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.029 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.029 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.030 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:43.030 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:43.030 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.030 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.030 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.030 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.030 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.031 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.031 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.031 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.031 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.031 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.031 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.031 INFO:teuthology.orchestra.run.smithi016.stdout: "min_score": { 2024-03-28T22:20:43.032 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_score", 2024-03-28T22:20:43.032 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-03-28T22:20:43.032 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.032 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.032 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-03-28T22:20:43.032 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.032 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.032 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.033 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-28T22:20:43.033 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.033 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.033 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.033 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-03-28T22:20:43.033 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-03-28T22:20:43.033 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.034 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.034 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.034 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "upmap", 2024-03-28T22:20:43.034 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.034 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.034 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.034 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-compat", 2024-03-28T22:20:43.034 INFO:teuthology.orchestra.run.smithi016.stdout: "none", 2024-03-28T22:20:43.035 INFO:teuthology.orchestra.run.smithi016.stdout: "read", 2024-03-28T22:20:43.035 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap", 2024-03-28T22:20:43.035 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap-read" 2024-03-28T22:20:43.035 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.035 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Balancer mode", 2024-03-28T22:20:43.035 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.035 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.036 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.036 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_ids": { 2024-03-28T22:20:43.036 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_ids", 2024-03-28T22:20:43.036 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.036 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.036 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.036 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.037 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.037 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.037 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.037 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-28T22:20:43.037 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.037 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.037 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.038 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-03-28T22:20:43.038 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-03-28T22:20:43.038 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:43.038 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.038 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.038 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-03-28T22:20:43.038 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.038 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.038 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.039 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-28T22:20:43.039 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.039 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.039 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.039 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_deviation": { 2024-03-28T22:20:43.039 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_deviation", 2024-03-28T22:20:43.039 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.039 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.040 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.040 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-03-28T22:20:43.040 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-03-28T22:20:43.040 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.040 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.040 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-28T22:20:43.040 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-28T22:20:43.040 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.040 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.041 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_optimizations": { 2024-03-28T22:20:43.041 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_optimizations", 2024-03-28T22:20:43.041 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-03-28T22:20:43.041 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.041 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.041 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-03-28T22:20:43.041 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.041 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.042 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.042 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-28T22:20:43.042 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.042 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.042 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.042 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.042 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.042 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.043 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm", 2024-03-28T22:20:43.043 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:43.043 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:43.043 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:43.043 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_down_multiplier": { 2024-03-28T22:20:43.043 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_down_multiplier", 2024-03-28T22:20:43.043 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-03-28T22:20:43.043 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.043 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.044 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-03-28T22:20:43.044 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.044 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.044 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.044 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-28T22:20:43.044 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.044 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.044 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.045 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_refresh_rate": { 2024-03-28T22:20:43.045 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_refresh_rate", 2024-03-28T22:20:43.045 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:43.045 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.045 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.045 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "20", 2024-03-28T22:20:43.045 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.046 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.046 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.046 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-28T22:20:43.046 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.046 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.046 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.046 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_starting_port": { 2024-03-28T22:20:43.047 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_starting_port", 2024-03-28T22:20:43.047 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.047 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.047 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.047 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4721", 2024-03-28T22:20:43.047 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.047 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.047 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.048 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-28T22:20:43.048 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.048 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.048 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.048 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_ptrace": { 2024-03-28T22:20:43.049 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_ptrace", 2024-03-28T22:20:43.049 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.049 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.049 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.049 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.049 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.049 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.049 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.050 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-28T22:20:43.050 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-28T22:20:43.050 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.050 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.050 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_interval": { 2024-03-28T22:20:43.050 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_interval", 2024-03-28T22:20:43.051 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:43.051 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.051 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.051 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-03-28T22:20:43.051 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.051 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.051 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.052 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-28T22:20:43.052 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.052 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.052 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.052 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_memory_target_ratio": { 2024-03-28T22:20:43.052 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_memory_target_ratio", 2024-03-28T22:20:43.052 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-03-28T22:20:43.052 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.053 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.053 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.7", 2024-03-28T22:20:43.053 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.053 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.053 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.053 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-28T22:20:43.053 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.054 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.054 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.054 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm_log_destination": { 2024-03-28T22:20:43.054 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm_log_destination", 2024-03-28T22:20:43.054 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.054 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.055 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.055 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.055 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.055 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.055 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.055 INFO:teuthology.orchestra.run.smithi016.stdout: "file", 2024-03-28T22:20:43.055 INFO:teuthology.orchestra.run.smithi016.stdout: "file,syslog", 2024-03-28T22:20:43.055 INFO:teuthology.orchestra.run.smithi016.stdout: "syslog" 2024-03-28T22:20:43.056 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.056 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-28T22:20:43.056 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.056 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.056 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.056 INFO:teuthology.orchestra.run.smithi016.stdout: "cgroups_split": { 2024-03-28T22:20:43.057 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cgroups_split", 2024-03-28T22:20:43.057 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.057 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.057 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.057 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.057 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.057 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.057 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.058 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-28T22:20:43.058 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.058 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.058 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.058 INFO:teuthology.orchestra.run.smithi016.stdout: "config_checks_enabled": { 2024-03-28T22:20:43.058 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_checks_enabled", 2024-03-28T22:20:43.058 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.059 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.059 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.059 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.059 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.059 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.059 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.059 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-28T22:20:43.059 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.059 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.060 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.060 INFO:teuthology.orchestra.run.smithi016.stdout: "config_dashboard": { 2024-03-28T22:20:43.060 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_dashboard", 2024-03-28T22:20:43.060 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.060 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.060 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.060 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.061 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.061 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.061 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.061 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-28T22:20:43.061 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.061 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.061 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.062 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_alertmanager": { 2024-03-28T22:20:43.062 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_alertmanager", 2024-03-28T22:20:43.062 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.062 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.062 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.062 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-28T22:20:43.062 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.063 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.063 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.063 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-03-28T22:20:43.063 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.063 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.063 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.063 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_base": { 2024-03-28T22:20:43.064 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_base", 2024-03-28T22:20:43.064 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.064 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.064 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.064 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-28T22:20:43.064 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.064 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.064 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.064 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Container image name, without the tag", 2024-03-28T22:20:43.065 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.065 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.065 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.065 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_elasticsearch": { 2024-03-28T22:20:43.065 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_elasticsearch", 2024-03-28T22:20:43.065 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.065 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.066 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.066 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-28T22:20:43.066 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.066 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.066 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.066 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "elasticsearch container image", 2024-03-28T22:20:43.066 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.066 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.067 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.067 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_grafana": { 2024-03-28T22:20:43.067 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_grafana", 2024-03-28T22:20:43.067 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.067 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.067 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.067 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-28T22:20:43.067 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.068 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.068 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.068 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-03-28T22:20:43.068 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.068 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.068 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.069 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_haproxy": { 2024-03-28T22:20:43.069 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_haproxy", 2024-03-28T22:20:43.069 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.069 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.069 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.069 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-28T22:20:43.069 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.069 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.069 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.070 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "HAproxy container image", 2024-03-28T22:20:43.070 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.070 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.070 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.070 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_agent": { 2024-03-28T22:20:43.070 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_agent", 2024-03-28T22:20:43.070 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.071 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.071 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.071 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-28T22:20:43.071 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.071 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.071 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.071 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger agent container image", 2024-03-28T22:20:43.071 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.071 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.071 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.072 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_collector": { 2024-03-28T22:20:43.072 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_collector", 2024-03-28T22:20:43.072 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.072 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.072 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.072 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-28T22:20:43.072 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.072 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.073 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.073 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger collector container image", 2024-03-28T22:20:43.073 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.073 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.073 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.073 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_query": { 2024-03-28T22:20:43.073 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_query", 2024-03-28T22:20:43.073 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.074 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.074 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.074 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-28T22:20:43.074 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.074 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.074 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.074 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger query container image", 2024-03-28T22:20:43.074 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.075 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.075 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.075 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_keepalived": { 2024-03-28T22:20:43.075 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_keepalived", 2024-03-28T22:20:43.075 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.075 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.075 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.076 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-28T22:20:43.076 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.076 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.076 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.076 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Keepalived container image", 2024-03-28T22:20:43.076 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.076 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.076 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.077 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_loki": { 2024-03-28T22:20:43.077 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_loki", 2024-03-28T22:20:43.077 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.077 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.077 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.077 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-28T22:20:43.077 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.077 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.078 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.078 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Loki container image", 2024-03-28T22:20:43.078 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.078 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.078 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.078 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_node_exporter": { 2024-03-28T22:20:43.078 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_node_exporter", 2024-03-28T22:20:43.079 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.079 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.079 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.079 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-28T22:20:43.079 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.079 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.079 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.079 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-03-28T22:20:43.080 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.080 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.080 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.080 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_nvmeof": { 2024-03-28T22:20:43.080 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_nvmeof", 2024-03-28T22:20:43.080 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.080 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.080 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.081 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-28T22:20:43.081 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.081 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.081 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.081 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Nvme-of container image", 2024-03-28T22:20:43.081 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.081 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.081 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.082 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_prometheus": { 2024-03-28T22:20:43.082 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_prometheus", 2024-03-28T22:20:43.082 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.082 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.082 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.082 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-28T22:20:43.082 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.083 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.083 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.083 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-03-28T22:20:43.083 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.083 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.083 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.083 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_promtail": { 2024-03-28T22:20:43.084 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_promtail", 2024-03-28T22:20:43.084 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.084 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.084 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.084 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-28T22:20:43.084 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.084 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.084 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.085 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Promtail container image", 2024-03-28T22:20:43.085 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.085 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.085 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.085 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_snmp_gateway": { 2024-03-28T22:20:43.085 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_snmp_gateway", 2024-03-28T22:20:43.085 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.086 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.086 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.086 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-28T22:20:43.086 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.086 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.086 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.086 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SNMP Gateway container image", 2024-03-28T22:20:43.086 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.087 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.087 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.087 INFO:teuthology.orchestra.run.smithi016.stdout: "container_init": { 2024-03-28T22:20:43.087 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_init", 2024-03-28T22:20:43.087 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.087 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.087 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.087 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.088 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.088 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.088 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.088 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Run podman/docker with `--init`", 2024-03-28T22:20:43.088 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.088 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.088 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.088 INFO:teuthology.orchestra.run.smithi016.stdout: "daemon_cache_timeout": { 2024-03-28T22:20:43.089 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "daemon_cache_timeout", 2024-03-28T22:20:43.089 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:43.089 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.089 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.089 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-03-28T22:20:43.089 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.089 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.089 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.090 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-28T22:20:43.090 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.090 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.090 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.090 INFO:teuthology.orchestra.run.smithi016.stdout: "default_cephadm_command_timeout": { 2024-03-28T22:20:43.090 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_cephadm_command_timeout", 2024-03-28T22:20:43.090 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.091 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.091 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.091 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "900", 2024-03-28T22:20:43.091 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.091 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.091 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.091 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-28T22:20:43.091 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.091 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.092 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.092 INFO:teuthology.orchestra.run.smithi016.stdout: "default_registry": { 2024-03-28T22:20:43.092 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_registry", 2024-03-28T22:20:43.092 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.092 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.093 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.093 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io", 2024-03-28T22:20:43.093 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.093 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.093 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.093 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-28T22:20:43.093 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.093 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.094 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.094 INFO:teuthology.orchestra.run.smithi016.stdout: "device_cache_timeout": { 2024-03-28T22:20:43.094 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_cache_timeout", 2024-03-28T22:20:43.094 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:43.094 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.094 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.094 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1800", 2024-03-28T22:20:43.094 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.095 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.095 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.095 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache device inventory", 2024-03-28T22:20:43.095 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.095 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.095 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.095 INFO:teuthology.orchestra.run.smithi016.stdout: "device_enhanced_scan": { 2024-03-28T22:20:43.095 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_enhanced_scan", 2024-03-28T22:20:43.096 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.096 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.096 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.096 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.096 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.096 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.096 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.096 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-28T22:20:43.096 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.096 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.097 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.097 INFO:teuthology.orchestra.run.smithi016.stdout: "facts_cache_timeout": { 2024-03-28T22:20:43.097 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "facts_cache_timeout", 2024-03-28T22:20:43.097 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:43.097 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.097 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.097 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-03-28T22:20:43.097 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.098 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.098 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.098 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache host facts data", 2024-03-28T22:20:43.098 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.098 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.098 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.098 INFO:teuthology.orchestra.run.smithi016.stdout: "host_check_interval": { 2024-03-28T22:20:43.099 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "host_check_interval", 2024-03-28T22:20:43.099 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:43.099 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.099 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.099 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-03-28T22:20:43.099 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.099 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.099 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.100 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to perform a host check", 2024-03-28T22:20:43.100 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.100 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.100 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.100 INFO:teuthology.orchestra.run.smithi016.stdout: "hw_monitoring": { 2024-03-28T22:20:43.100 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hw_monitoring", 2024-03-28T22:20:43.100 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.101 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.101 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.101 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.101 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.101 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.101 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.101 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-28T22:20:43.101 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.101 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.102 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.102 INFO:teuthology.orchestra.run.smithi016.stdout: "inventory_list_all": { 2024-03-28T22:20:43.102 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "inventory_list_all", 2024-03-28T22:20:43.102 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.102 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.102 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.102 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.103 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.103 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.103 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.103 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-28T22:20:43.103 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.103 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.103 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.103 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:43.104 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:43.104 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.104 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.104 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.104 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.104 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.104 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.104 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.104 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.105 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.105 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.105 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.105 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.105 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.105 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.105 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.105 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.106 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.106 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.106 INFO:teuthology.orchestra.run.smithi016.stdout: "log_refresh_metadata": { 2024-03-28T22:20:43.106 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_refresh_metadata", 2024-03-28T22:20:43.106 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.106 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.106 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.107 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.107 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.107 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.107 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.107 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-28T22:20:43.107 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.107 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.107 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.107 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:43.108 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:43.108 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.108 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.108 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.108 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.108 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.108 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.108 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.108 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-28T22:20:43.109 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.109 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.109 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.109 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:43.109 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:43.109 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.109 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.109 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.110 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:43.110 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.110 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.110 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.110 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.110 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.110 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.110 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.110 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.111 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.111 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.111 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.111 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.111 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.111 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.111 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:43.111 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:43.112 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.112 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.112 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.112 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.112 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.112 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.112 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.112 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.112 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.112 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.113 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.113 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-28T22:20:43.113 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-28T22:20:43.113 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.113 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.113 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.113 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.113 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.114 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.114 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.114 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-28T22:20:43.114 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.114 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.114 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.114 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-28T22:20:43.114 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-28T22:20:43.115 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.115 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.115 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.115 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "*", 2024-03-28T22:20:43.115 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.115 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.115 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.115 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-28T22:20:43.115 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.116 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.116 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.116 INFO:teuthology.orchestra.run.smithi016.stdout: "max_count_per_host": { 2024-03-28T22:20:43.116 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_count_per_host", 2024-03-28T22:20:43.116 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.116 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.116 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.116 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-03-28T22:20:43.117 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.117 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.117 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.117 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of daemons per service per host", 2024-03-28T22:20:43.117 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.117 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.117 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.117 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd_draining_count": { 2024-03-28T22:20:43.118 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_osd_draining_count", 2024-03-28T22:20:43.118 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.118 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.118 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.118 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-03-28T22:20:43.118 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.118 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.119 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.119 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-28T22:20:43.119 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.119 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.119 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.119 INFO:teuthology.orchestra.run.smithi016.stdout: "migration_current": { 2024-03-28T22:20:43.119 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "migration_current", 2024-03-28T22:20:43.120 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.120 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.120 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.120 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.120 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.120 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.120 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.120 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "internal - do not modify", 2024-03-28T22:20:43.121 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.121 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.121 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.121 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-03-28T22:20:43.121 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-03-28T22:20:43.121 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.121 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.121 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.122 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "root", 2024-03-28T22:20:43.122 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.122 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.122 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.122 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm-package", 2024-03-28T22:20:43.122 INFO:teuthology.orchestra.run.smithi016.stdout: "root" 2024-03-28T22:20:43.122 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.122 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "mode for remote execution of cephadm", 2024-03-28T22:20:43.123 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.123 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.123 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.123 INFO:teuthology.orchestra.run.smithi016.stdout: "oob_default_addr": { 2024-03-28T22:20:43.123 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "oob_default_addr", 2024-03-28T22:20:43.123 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.123 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.123 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.124 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "169.254.1.1", 2024-03-28T22:20:43.124 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.124 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.124 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.124 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-28T22:20:43.124 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.124 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.125 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.125 INFO:teuthology.orchestra.run.smithi016.stdout: "prometheus_alerts_path": { 2024-03-28T22:20:43.125 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus_alerts_path", 2024-03-28T22:20:43.125 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.125 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.125 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.125 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-28T22:20:43.126 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.126 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.126 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.126 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-28T22:20:43.126 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.126 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.126 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.127 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_insecure": { 2024-03-28T22:20:43.127 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_insecure", 2024-03-28T22:20:43.127 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.127 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.127 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.127 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.127 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.127 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.127 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.127 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-28T22:20:43.128 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.128 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.128 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.128 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_password": { 2024-03-28T22:20:43.128 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_password", 2024-03-28T22:20:43.128 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.128 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.128 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.129 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.129 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.129 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.129 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.129 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-28T22:20:43.129 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.129 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.129 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.130 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_url": { 2024-03-28T22:20:43.130 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_url", 2024-03-28T22:20:43.130 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.130 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.130 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.130 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.130 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.131 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.131 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.131 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-28T22:20:43.131 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.131 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.131 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.131 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_username": { 2024-03-28T22:20:43.132 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_username", 2024-03-28T22:20:43.132 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.132 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.132 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.132 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.132 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.132 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.133 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.133 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-28T22:20:43.133 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.133 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.133 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.133 INFO:teuthology.orchestra.run.smithi016.stdout: "secure_monitoring_stack": { 2024-03-28T22:20:43.133 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "secure_monitoring_stack", 2024-03-28T22:20:43.134 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.134 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.134 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.134 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.134 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.134 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.134 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.134 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-28T22:20:43.135 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.135 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.135 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.135 INFO:teuthology.orchestra.run.smithi016.stdout: "service_discovery_port": { 2024-03-28T22:20:43.135 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "service_discovery_port", 2024-03-28T22:20:43.135 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.135 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.136 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.136 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8765", 2024-03-28T22:20:43.136 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.136 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.136 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.136 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "cephadm service discovery port", 2024-03-28T22:20:43.136 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.136 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.137 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.137 INFO:teuthology.orchestra.run.smithi016.stdout: "ssh_config_file": { 2024-03-28T22:20:43.137 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssh_config_file", 2024-03-28T22:20:43.137 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.137 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.137 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.137 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.137 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.138 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.138 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.138 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-28T22:20:43.138 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.138 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.138 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.139 INFO:teuthology.orchestra.run.smithi016.stdout: "use_agent": { 2024-03-28T22:20:43.139 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_agent", 2024-03-28T22:20:43.139 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.139 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.139 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.139 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.139 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.139 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.139 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.140 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-28T22:20:43.140 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.140 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.140 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.140 INFO:teuthology.orchestra.run.smithi016.stdout: "use_repo_digest": { 2024-03-28T22:20:43.140 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_repo_digest", 2024-03-28T22:20:43.140 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.140 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.141 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.141 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.141 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.141 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.141 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.141 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-28T22:20:43.141 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.141 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.141 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.142 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_failed_host_check": { 2024-03-28T22:20:43.142 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_failed_host_check", 2024-03-28T22:20:43.142 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.142 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.142 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.142 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.142 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.142 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.143 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.143 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if the host check fails", 2024-03-28T22:20:43.143 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.143 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.143 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.143 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.143 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_daemons": { 2024-03-28T22:20:43.143 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_daemons", 2024-03-28T22:20:43.143 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.144 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.144 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.144 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.144 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.144 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.144 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.144 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-28T22:20:43.144 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.145 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.145 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.145 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_hosts": { 2024-03-28T22:20:43.145 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_hosts", 2024-03-28T22:20:43.145 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.145 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.145 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.146 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.146 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.146 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.146 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.146 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-28T22:20:43.146 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.146 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.146 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.146 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.147 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.147 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crash", 2024-03-28T22:20:43.147 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:43.147 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:43.147 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:43.147 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:43.148 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:43.148 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.148 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.148 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.148 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.148 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.148 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.148 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.148 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.149 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.149 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.149 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.149 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.149 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.149 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.149 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.149 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.150 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.150 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.150 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:43.150 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:43.150 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.150 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.150 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.151 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.151 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.151 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.151 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.151 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.151 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.151 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.151 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.152 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:43.152 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:43.152 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.152 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.152 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.152 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:43.152 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.152 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.152 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.153 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.153 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.153 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.153 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.153 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.153 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.153 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.153 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.153 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.153 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.154 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.154 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.154 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:43.154 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:43.154 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.154 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.154 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.154 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.154 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.155 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.155 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.155 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.155 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.155 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.155 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.155 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.155 INFO:teuthology.orchestra.run.smithi016.stdout: "retain_interval": { 2024-03-28T22:20:43.156 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retain_interval", 2024-03-28T22:20:43.156 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:43.156 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.156 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.156 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "31536000", 2024-03-28T22:20:43.156 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.156 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.156 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.156 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-28T22:20:43.157 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.157 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.157 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.157 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_recent_interval": { 2024-03-28T22:20:43.157 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_recent_interval", 2024-03-28T22:20:43.157 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:43.157 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.157 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.158 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1209600", 2024-03-28T22:20:43.158 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.158 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.158 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.158 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-28T22:20:43.158 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.158 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.158 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.158 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.159 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.159 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dashboard", 2024-03-28T22:20:43.159 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:43.159 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:43.159 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:43.159 INFO:teuthology.orchestra.run.smithi016.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-28T22:20:43.160 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-28T22:20:43.160 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.160 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.160 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.160 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-03-28T22:20:43.160 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.160 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.161 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.161 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.161 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.161 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.161 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.161 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_HOST": { 2024-03-28T22:20:43.161 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-28T22:20:43.161 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.162 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.162 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.162 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.162 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.162 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.162 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.162 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.162 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.163 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.163 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.163 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-28T22:20:43.163 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-28T22:20:43.163 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.163 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.163 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.163 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.164 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.164 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.164 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.164 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.164 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.164 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.164 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.164 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_ENABLED": { 2024-03-28T22:20:43.164 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_ENABLED", 2024-03-28T22:20:43.165 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.165 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.165 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.165 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.165 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.165 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.165 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.165 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.165 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.165 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.166 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.166 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-28T22:20:43.166 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-28T22:20:43.166 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.166 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.166 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.166 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.167 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.167 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.167 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.167 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.167 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.167 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.167 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.167 INFO:teuthology.orchestra.run.smithi016.stdout: "ENABLE_BROWSABLE_API": { 2024-03-28T22:20:43.167 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-28T22:20:43.168 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.168 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.168 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.168 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.168 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.168 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.168 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.168 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.169 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.169 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.169 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.169 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-28T22:20:43.169 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-28T22:20:43.169 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.169 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.169 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.170 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.170 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.170 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.170 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.170 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.170 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.170 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.170 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.171 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-28T22:20:43.171 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-28T22:20:43.171 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.171 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.171 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.171 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.171 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.171 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.172 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.172 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.172 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.172 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.172 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.172 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-28T22:20:43.172 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-28T22:20:43.172 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.173 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.173 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.173 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.173 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.173 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.173 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.173 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.173 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.173 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.173 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.174 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-28T22:20:43.174 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-28T22:20:43.174 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.174 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.174 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.174 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.174 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.175 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.175 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.175 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.177 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.177 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.177 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.177 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-28T22:20:43.178 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-28T22:20:43.178 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.178 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.178 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.178 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.178 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.178 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.179 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.179 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.179 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.179 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.179 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.179 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-28T22:20:43.179 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-28T22:20:43.179 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.180 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.180 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.180 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.180 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.180 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.180 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.180 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.180 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.180 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.181 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.181 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-28T22:20:43.181 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-28T22:20:43.181 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.181 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.181 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.181 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.182 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.182 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.182 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.182 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.182 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.182 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.182 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.182 INFO:teuthology.orchestra.run.smithi016.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-28T22:20:43.183 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-28T22:20:43.183 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.183 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.183 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.183 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.183 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.183 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.183 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.184 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.184 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.184 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.184 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.184 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_PASSWORD": { 2024-03-28T22:20:43.184 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-28T22:20:43.184 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.185 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.185 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.185 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-03-28T22:20:43.185 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.185 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.185 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.185 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.186 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.186 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.186 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.186 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-28T22:20:43.186 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-28T22:20:43.186 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.186 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.187 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.187 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.187 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.187 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.187 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.187 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.187 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.187 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.187 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.188 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_URL": { 2024-03-28T22:20:43.188 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_URL", 2024-03-28T22:20:43.188 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.188 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.188 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.188 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.188 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.189 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.189 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.189 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.189 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.189 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.189 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.189 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_USERNAME": { 2024-03-28T22:20:43.189 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-28T22:20:43.190 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.190 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.190 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.190 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-03-28T22:20:43.190 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.190 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.190 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.190 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.191 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.191 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.191 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.191 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-28T22:20:43.191 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-28T22:20:43.191 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.191 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.192 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.192 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.192 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.192 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.192 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.192 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.192 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.192 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.192 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.193 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-28T22:20:43.193 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-28T22:20:43.193 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.193 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.193 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.193 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.193 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.193 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.194 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.194 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.194 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.194 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.194 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.194 INFO:teuthology.orchestra.run.smithi016.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-28T22:20:43.194 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-28T22:20:43.194 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.195 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.195 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.195 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.195 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.195 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.195 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.195 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.195 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.196 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.196 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.196 INFO:teuthology.orchestra.run.smithi016.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-28T22:20:43.196 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-28T22:20:43.196 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.196 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.196 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.197 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.197 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.197 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.197 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.197 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.197 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.197 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.197 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.197 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_HOST": { 2024-03-28T22:20:43.198 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-28T22:20:43.198 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.198 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.198 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.198 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.198 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.198 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.198 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.199 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.199 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.199 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.199 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.199 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-28T22:20:43.199 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-28T22:20:43.199 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.199 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.199 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.200 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.200 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.200 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.200 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.200 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.200 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.200 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.200 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.201 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-28T22:20:43.201 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-28T22:20:43.201 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.201 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.201 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.201 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.201 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.201 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.201 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.202 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.202 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.202 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.202 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.202 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-28T22:20:43.202 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-28T22:20:43.202 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.203 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.203 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.203 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.203 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.203 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.203 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.203 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.203 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.204 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.204 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.204 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-28T22:20:43.204 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-28T22:20:43.204 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.204 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.204 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.204 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.205 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.205 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.205 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.205 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.205 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.205 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.205 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.205 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-28T22:20:43.206 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-28T22:20:43.206 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.206 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.206 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.206 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.206 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.206 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.206 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.207 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.207 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.207 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.207 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.207 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-28T22:20:43.207 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-28T22:20:43.207 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.208 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.208 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.208 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.208 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.208 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.208 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.208 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.208 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.208 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.209 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.209 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-28T22:20:43.209 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-28T22:20:43.209 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.209 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.209 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.209 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.209 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.210 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.210 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.210 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.210 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.210 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.210 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.210 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-28T22:20:43.210 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-28T22:20:43.211 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.211 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.211 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.211 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.211 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.211 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.211 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.211 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.212 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.212 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.212 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.212 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_ENABLED": { 2024-03-28T22:20:43.212 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-28T22:20:43.212 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.212 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.212 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.213 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.213 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.213 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.213 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.213 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.213 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.213 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.213 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.214 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-28T22:20:43.214 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-28T22:20:43.214 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.214 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.214 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.214 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-28T22:20:43.214 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.214 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.215 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.215 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.215 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.215 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.215 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.215 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-28T22:20:43.215 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-28T22:20:43.215 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.216 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.216 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.216 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-03-28T22:20:43.216 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.216 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.216 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.216 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.216 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.216 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.216 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.217 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-28T22:20:43.217 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-28T22:20:43.217 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.217 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.217 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.217 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8", 2024-03-28T22:20:43.217 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.218 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.218 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.218 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.218 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.218 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.218 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.218 INFO:teuthology.orchestra.run.smithi016.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-28T22:20:43.219 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-28T22:20:43.219 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.219 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.219 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.219 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "45", 2024-03-28T22:20:43.219 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.219 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.219 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.219 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.220 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.220 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.220 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.220 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ACCESS_KEY": { 2024-03-28T22:20:43.220 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-28T22:20:43.220 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.220 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.220 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.221 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.221 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.221 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.221 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.221 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.221 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.221 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.221 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.222 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-28T22:20:43.222 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-28T22:20:43.222 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.222 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.222 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.222 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-03-28T22:20:43.222 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.222 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.223 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.223 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.223 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.223 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.223 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.223 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SECRET_KEY": { 2024-03-28T22:20:43.223 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-28T22:20:43.223 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.224 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.224 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.224 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.224 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.224 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.224 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.224 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.224 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.224 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.225 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.225 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SSL_VERIFY": { 2024-03-28T22:20:43.225 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-28T22:20:43.225 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.225 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.225 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.225 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.225 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.226 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.226 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.226 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.226 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.226 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.226 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.226 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-28T22:20:43.227 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-28T22:20:43.227 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.227 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.227 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.227 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-03-28T22:20:43.227 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.227 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.228 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.228 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.228 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.228 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.228 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.228 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-28T22:20:43.228 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-28T22:20:43.228 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.229 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.229 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.229 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-03-28T22:20:43.229 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.229 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.229 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.229 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.229 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.229 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.230 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.230 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-28T22:20:43.230 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-28T22:20:43.230 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.230 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.230 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.230 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-03-28T22:20:43.231 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.231 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.231 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.231 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.231 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.231 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.231 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.231 INFO:teuthology.orchestra.run.smithi016.stdout: "cross_origin_url": { 2024-03-28T22:20:43.232 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cross_origin_url", 2024-03-28T22:20:43.232 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.232 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.232 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.232 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.232 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.232 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.232 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.232 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.233 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.233 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.233 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.233 INFO:teuthology.orchestra.run.smithi016.stdout: "crt_file": { 2024-03-28T22:20:43.233 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crt_file", 2024-03-28T22:20:43.233 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.233 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.233 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.234 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.234 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.234 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.234 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.234 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.234 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.234 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.234 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.235 INFO:teuthology.orchestra.run.smithi016.stdout: "debug": { 2024-03-28T22:20:43.235 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "debug", 2024-03-28T22:20:43.235 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.235 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.235 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.235 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.235 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.236 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.236 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.236 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable/disable debug options", 2024-03-28T22:20:43.236 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.236 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.236 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.236 INFO:teuthology.orchestra.run.smithi016.stdout: "jwt_token_ttl": { 2024-03-28T22:20:43.237 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "jwt_token_ttl", 2024-03-28T22:20:43.237 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.237 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.237 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.237 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "28800", 2024-03-28T22:20:43.237 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.237 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.237 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.237 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.238 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.238 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.238 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.238 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-03-28T22:20:43.238 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-03-28T22:20:43.238 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.239 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.239 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.239 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.239 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.239 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.239 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.239 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.239 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.239 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.240 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.240 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:43.240 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:43.240 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.240 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.240 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.240 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.241 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.241 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.241 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.241 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.241 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.241 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.241 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.241 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.241 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.242 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.242 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.242 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.242 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.242 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.242 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:43.242 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:43.242 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.243 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.243 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.243 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.243 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.243 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.243 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.243 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.243 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.243 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.244 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.244 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:43.244 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:43.244 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.244 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.244 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.244 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:43.244 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.245 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.245 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.245 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.245 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.245 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.245 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.245 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.245 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.245 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.246 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.246 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.246 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.246 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.246 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:43.246 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:43.247 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.247 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.247 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.247 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.247 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.247 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.247 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.247 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.247 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.248 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.248 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.248 INFO:teuthology.orchestra.run.smithi016.stdout: "motd": { 2024-03-28T22:20:43.248 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "motd", 2024-03-28T22:20:43.248 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.248 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.248 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.248 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.249 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.249 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.249 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.249 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "The message of the day", 2024-03-28T22:20:43.249 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.249 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.249 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.250 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect_resolve_ip_addr": { 2024-03-28T22:20:43.250 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "redirect_resolve_ip_addr", 2024-03-28T22:20:43.250 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.250 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.250 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.250 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.250 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.250 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.251 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.251 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.251 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.251 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.251 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.251 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-03-28T22:20:43.252 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-03-28T22:20:43.252 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.252 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.252 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.252 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-03-28T22:20:43.252 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.252 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.253 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.253 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.253 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.253 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.253 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.253 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-03-28T22:20:43.253 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-03-28T22:20:43.254 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.254 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.254 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.254 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8080", 2024-03-28T22:20:43.254 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.254 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.254 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.254 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.254 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.255 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.255 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.255 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-03-28T22:20:43.255 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-03-28T22:20:43.255 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.255 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.255 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.256 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.256 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.256 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.256 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.256 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.256 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.256 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.256 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.257 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl_server_port": { 2024-03-28T22:20:43.257 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl_server_port", 2024-03-28T22:20:43.257 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.257 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.257 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.257 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8443", 2024-03-28T22:20:43.257 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.257 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.258 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.258 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.258 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.258 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.258 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.258 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-03-28T22:20:43.259 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-03-28T22:20:43.259 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.259 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.259 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.259 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "redirect", 2024-03-28T22:20:43.259 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.259 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.259 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.260 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.260 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect" 2024-03-28T22:20:43.260 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.260 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.260 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.260 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.260 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.261 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-03-28T22:20:43.261 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-03-28T22:20:43.261 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.261 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.261 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.261 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-03-28T22:20:43.261 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-03-28T22:20:43.261 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-03-28T22:20:43.261 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.262 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.262 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.262 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.262 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.262 INFO:teuthology.orchestra.run.smithi016.stdout: "url_prefix": { 2024-03-28T22:20:43.262 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url_prefix", 2024-03-28T22:20:43.263 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.263 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.263 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.263 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.263 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.263 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.263 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.263 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.264 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.264 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.264 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.264 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.264 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.265 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "devicehealth", 2024-03-28T22:20:43.265 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:43.265 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:43.265 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:43.265 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_monitoring": { 2024-03-28T22:20:43.265 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_monitoring", 2024-03-28T22:20:43.265 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.265 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.265 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.266 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.266 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.266 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.266 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.266 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "monitor device health metrics", 2024-03-28T22:20:43.266 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.266 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.266 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.267 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:43.267 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:43.267 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.267 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.267 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.267 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.267 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.267 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.268 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.268 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.268 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.268 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.268 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.268 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.268 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.268 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.268 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.268 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.269 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.269 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.269 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:43.269 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:43.269 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.269 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.269 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.269 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.270 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.270 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.270 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.270 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.270 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.270 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.270 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.271 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:43.271 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:43.271 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.271 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.271 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.271 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:43.271 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.271 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.271 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.271 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.272 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.272 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.272 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.272 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.272 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.272 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.272 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.272 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.272 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.273 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.273 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:43.273 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:43.273 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.273 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.273 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.273 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.273 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.274 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.274 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.274 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.274 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.274 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.274 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.274 INFO:teuthology.orchestra.run.smithi016.stdout: "mark_out_threshold": { 2024-03-28T22:20:43.275 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mark_out_threshold", 2024-03-28T22:20:43.275 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:43.275 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.275 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.275 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2419200", 2024-03-28T22:20:43.275 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.275 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.275 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.276 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-28T22:20:43.276 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.276 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.276 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.276 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": { 2024-03-28T22:20:43.276 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_name", 2024-03-28T22:20:43.276 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.276 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.277 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.277 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "device_health_metrics", 2024-03-28T22:20:43.277 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.277 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.277 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.277 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-28T22:20:43.277 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.277 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.278 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.278 INFO:teuthology.orchestra.run.smithi016.stdout: "retention_period": { 2024-03-28T22:20:43.278 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retention_period", 2024-03-28T22:20:43.278 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:43.278 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.278 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.278 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15552000", 2024-03-28T22:20:43.278 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.279 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.279 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.279 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain device health metrics", 2024-03-28T22:20:43.279 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.279 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.279 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.279 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_frequency": { 2024-03-28T22:20:43.280 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_frequency", 2024-03-28T22:20:43.280 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:43.280 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.280 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.280 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-03-28T22:20:43.280 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.280 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.280 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.280 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-28T22:20:43.281 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.281 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.281 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.281 INFO:teuthology.orchestra.run.smithi016.stdout: "self_heal": { 2024-03-28T22:20:43.281 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "self_heal", 2024-03-28T22:20:43.281 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.281 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.281 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.282 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.282 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.282 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.282 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.282 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-28T22:20:43.282 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.282 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.282 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.283 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-03-28T22:20:43.283 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-03-28T22:20:43.283 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:43.283 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.283 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.283 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-03-28T22:20:43.283 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.283 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.283 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.284 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and check device health", 2024-03-28T22:20:43.284 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.284 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.284 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.284 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_threshold": { 2024-03-28T22:20:43.284 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_threshold", 2024-03-28T22:20:43.284 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:43.285 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.285 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.285 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7257600", 2024-03-28T22:20:43.285 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.285 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.285 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.285 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-28T22:20:43.285 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.285 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.285 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.286 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.286 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.286 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "diskprediction_local", 2024-03-28T22:20:43.286 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:43.286 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:43.286 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:43.287 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:43.287 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:43.287 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.287 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.287 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.287 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.287 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.287 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.288 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.288 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.288 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.288 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.288 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.288 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.288 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.288 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.289 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.289 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.289 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.289 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.289 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:43.289 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:43.289 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.290 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.290 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.290 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.290 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.290 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.290 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.290 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.291 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.291 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.291 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.291 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:43.291 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:43.291 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.291 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.292 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.292 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:43.292 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.292 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.292 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.292 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.292 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.292 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.293 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.293 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.293 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.293 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.293 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.293 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.293 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.293 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.294 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:43.294 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:43.294 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.294 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.294 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.294 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.295 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.295 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.295 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.295 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.295 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.295 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.295 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.295 INFO:teuthology.orchestra.run.smithi016.stdout: "predict_interval": { 2024-03-28T22:20:43.296 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predict_interval", 2024-03-28T22:20:43.296 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.296 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.296 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.296 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-03-28T22:20:43.296 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.296 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.296 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.296 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.297 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.297 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.297 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.297 INFO:teuthology.orchestra.run.smithi016.stdout: "predictor_model": { 2024-03-28T22:20:43.297 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predictor_model", 2024-03-28T22:20:43.297 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.297 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.297 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.298 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "prophetstor", 2024-03-28T22:20:43.298 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.298 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.298 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.298 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.298 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.298 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.298 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.299 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-03-28T22:20:43.299 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-03-28T22:20:43.299 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.299 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.299 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.299 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-03-28T22:20:43.299 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.299 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.299 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.300 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.300 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.300 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.300 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.300 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.300 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.300 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "influx", 2024-03-28T22:20:43.301 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": false, 2024-03-28T22:20:43.301 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "influxdb python module not found", 2024-03-28T22:20:43.301 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:43.301 INFO:teuthology.orchestra.run.smithi016.stdout: "batch_size": { 2024-03-28T22:20:43.301 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "batch_size", 2024-03-28T22:20:43.301 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.301 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.301 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.302 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5000", 2024-03-28T22:20:43.302 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.302 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.302 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.302 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-28T22:20:43.302 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.302 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.302 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.303 INFO:teuthology.orchestra.run.smithi016.stdout: "database": { 2024-03-28T22:20:43.303 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "database", 2024-03-28T22:20:43.303 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.303 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.303 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.303 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "ceph", 2024-03-28T22:20:43.303 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.304 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.304 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.304 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-28T22:20:43.304 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.304 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.304 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.304 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": { 2024-03-28T22:20:43.305 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hostname", 2024-03-28T22:20:43.305 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.305 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.305 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.305 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.305 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.305 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.305 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.305 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server hostname", 2024-03-28T22:20:43.306 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.306 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.306 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.306 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-03-28T22:20:43.306 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-03-28T22:20:43.306 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:43.307 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.307 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.307 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "30", 2024-03-28T22:20:43.307 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "5", 2024-03-28T22:20:43.307 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.307 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.307 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-28T22:20:43.307 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.308 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.308 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.308 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:43.308 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:43.308 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.308 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.308 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.309 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.309 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.309 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.309 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.309 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.309 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.309 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.309 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.310 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.310 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.310 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.310 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.310 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.310 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.310 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.311 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:43.311 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:43.311 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.311 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.311 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.311 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.311 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.311 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.312 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.312 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.312 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.312 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.312 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.312 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:43.312 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:43.313 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.313 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.313 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.313 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:43.313 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.313 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.313 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.313 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.313 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.314 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.314 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.314 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.314 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.314 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.314 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.314 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.314 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.315 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.315 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:43.315 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:43.315 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.315 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.315 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.315 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.315 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.315 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.316 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.316 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.316 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.316 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.316 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.316 INFO:teuthology.orchestra.run.smithi016.stdout: "password": { 2024-03-28T22:20:43.316 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "password", 2024-03-28T22:20:43.316 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.317 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.317 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.317 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.317 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.317 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.317 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.317 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "password of InfluxDB server user", 2024-03-28T22:20:43.317 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.317 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.318 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.318 INFO:teuthology.orchestra.run.smithi016.stdout: "port": { 2024-03-28T22:20:43.318 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "port", 2024-03-28T22:20:43.318 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.318 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.318 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.318 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8086", 2024-03-28T22:20:43.318 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.319 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.319 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.319 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server port", 2024-03-28T22:20:43.319 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.319 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.319 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.320 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-03-28T22:20:43.320 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-03-28T22:20:43.320 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.320 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.320 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.320 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "false", 2024-03-28T22:20:43.320 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.320 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.321 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.321 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T22:20:43.321 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.321 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.321 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.321 INFO:teuthology.orchestra.run.smithi016.stdout: "threads": { 2024-03-28T22:20:43.321 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threads", 2024-03-28T22:20:43.321 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.322 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.322 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.322 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-03-28T22:20:43.322 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-03-28T22:20:43.322 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "32", 2024-03-28T22:20:43.322 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.322 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-28T22:20:43.323 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.323 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.323 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.323 INFO:teuthology.orchestra.run.smithi016.stdout: "username": { 2024-03-28T22:20:43.323 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "username", 2024-03-28T22:20:43.323 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.323 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.324 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.324 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.324 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.324 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.324 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.324 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "username of InfluxDB server user", 2024-03-28T22:20:43.324 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.324 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.325 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.325 INFO:teuthology.orchestra.run.smithi016.stdout: "verify_ssl": { 2024-03-28T22:20:43.325 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "verify_ssl", 2024-03-28T22:20:43.325 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.325 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.325 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.325 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "true", 2024-03-28T22:20:43.326 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.326 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.326 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.326 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T22:20:43.326 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.326 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.326 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.326 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.327 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.327 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "insights", 2024-03-28T22:20:43.327 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:43.327 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:43.327 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:43.327 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:43.328 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:43.328 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.328 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.328 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.328 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.328 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.328 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.329 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.329 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.329 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.329 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.329 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.329 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.330 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.330 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.330 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.330 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.330 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.330 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.330 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:43.330 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:43.331 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.331 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.331 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.331 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.331 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.331 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.331 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.332 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.332 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.332 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.332 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.332 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:43.332 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:43.332 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.332 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.333 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.333 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:43.333 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.333 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.333 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.333 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.334 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.334 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.334 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.334 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.334 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.334 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.334 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.335 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.335 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.335 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.335 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:43.335 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:43.335 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.335 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.336 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.336 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.336 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.336 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.336 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.336 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.336 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.336 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.336 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.337 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.337 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.337 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "iostat", 2024-03-28T22:20:43.337 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:43.337 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:43.337 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:43.338 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:43.338 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:43.338 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.338 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.338 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.338 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.338 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.338 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.339 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.339 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.339 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.339 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.339 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.339 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.339 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.339 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.340 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.340 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.340 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.340 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.340 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:43.340 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:43.340 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.341 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.341 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.341 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.341 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.341 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.341 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.341 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.341 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.342 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.342 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.342 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:43.342 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:43.342 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.342 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.342 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.343 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:43.343 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.343 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.343 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.343 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.343 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.343 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.344 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.344 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.344 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.344 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.344 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.344 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.344 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.344 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.345 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:43.345 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:43.345 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.345 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.345 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.345 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.345 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.346 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.346 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.346 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.346 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.346 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.346 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.346 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.347 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.347 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "k8sevents", 2024-03-28T22:20:43.347 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:43.347 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:43.348 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:43.348 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_event_retention_days": { 2024-03-28T22:20:43.348 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ceph_event_retention_days", 2024-03-28T22:20:43.348 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.348 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.348 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.348 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7", 2024-03-28T22:20:43.349 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.349 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.349 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.349 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-28T22:20:43.349 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.349 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.349 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.349 INFO:teuthology.orchestra.run.smithi016.stdout: "config_check_secs": { 2024-03-28T22:20:43.350 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_check_secs", 2024-03-28T22:20:43.350 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.350 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.350 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.350 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-03-28T22:20:43.350 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "10", 2024-03-28T22:20:43.350 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.350 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.351 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-28T22:20:43.351 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.351 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.351 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.351 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:43.351 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:43.351 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.351 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.352 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.352 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.352 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.352 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.352 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.352 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.352 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.352 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.353 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.353 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.353 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.353 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.353 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.353 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.353 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.353 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.354 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:43.354 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:43.354 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.354 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.354 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.354 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.354 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.355 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.355 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.355 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.355 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.355 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.355 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.355 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:43.356 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:43.356 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.356 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.356 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.356 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:43.356 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.356 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.356 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.356 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.357 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.357 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.357 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.357 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.357 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.357 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.357 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.357 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.357 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.358 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.358 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:43.358 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:43.358 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.358 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.358 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.358 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.358 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.358 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.359 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.359 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.359 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.359 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.359 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.359 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.359 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.360 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "localpool", 2024-03-28T22:20:43.360 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:43.360 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:43.360 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:43.360 INFO:teuthology.orchestra.run.smithi016.stdout: "failure_domain": { 2024-03-28T22:20:43.360 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "failure_domain", 2024-03-28T22:20:43.361 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.361 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.361 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.361 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "host", 2024-03-28T22:20:43.361 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.361 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.361 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.361 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "failure domain for any created local pool", 2024-03-28T22:20:43.361 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-28T22:20:43.362 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.362 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.362 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:43.362 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:43.362 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.362 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.363 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.363 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.363 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.363 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.363 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.363 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.363 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.363 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.363 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.364 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.364 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.364 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.364 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.364 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.364 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.364 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.364 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:43.364 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:43.365 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.365 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.365 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.365 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.365 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.365 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.365 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.365 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.365 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.365 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.366 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.366 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:43.366 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:43.366 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.366 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.366 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.366 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:43.367 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.367 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.367 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.367 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.367 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.367 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.367 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.367 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.367 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.367 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.368 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.368 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.368 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.368 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.368 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:43.368 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:43.369 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.369 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.369 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.369 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.369 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.369 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.369 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.369 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.370 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.370 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.370 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.370 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": { 2024-03-28T22:20:43.370 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_size", 2024-03-28T22:20:43.370 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.371 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.371 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.371 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.371 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.371 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.371 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.371 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default min_size for any created local pool", 2024-03-28T22:20:43.371 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-28T22:20:43.371 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.372 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.372 INFO:teuthology.orchestra.run.smithi016.stdout: "num_rep": { 2024-03-28T22:20:43.372 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "num_rep", 2024-03-28T22:20:43.372 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.372 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.372 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.372 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3", 2024-03-28T22:20:43.373 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.373 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.373 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.373 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default replica count for any created local pool", 2024-03-28T22:20:43.373 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.373 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.373 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.374 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": { 2024-03-28T22:20:43.374 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_num", 2024-03-28T22:20:43.374 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.374 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.374 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.374 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "128", 2024-03-28T22:20:43.374 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.375 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.375 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.375 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default pg_num for any created local pool", 2024-03-28T22:20:43.375 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.375 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.375 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.376 INFO:teuthology.orchestra.run.smithi016.stdout: "prefix": { 2024-03-28T22:20:43.377 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prefix", 2024-03-28T22:20:43.377 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.377 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.377 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.378 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.378 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.378 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.378 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.378 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name prefix for any created local pool", 2024-03-28T22:20:43.378 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.378 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.379 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.379 INFO:teuthology.orchestra.run.smithi016.stdout: "subtree": { 2024-03-28T22:20:43.379 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "subtree", 2024-03-28T22:20:43.379 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.379 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.379 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.379 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "rack", 2024-03-28T22:20:43.380 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.380 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.380 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.380 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-28T22:20:43.380 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-28T22:20:43.380 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.380 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.380 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.381 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.381 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mds_autoscaler", 2024-03-28T22:20:43.381 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:43.381 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:43.381 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:43.381 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:43.382 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:43.382 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.382 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.382 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.382 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.382 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.382 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.382 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.383 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.383 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.383 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.383 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.383 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.383 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.383 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.384 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.384 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.384 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.384 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.384 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:43.384 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:43.384 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.384 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.385 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.385 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.385 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.385 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.385 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.385 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.385 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.386 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.386 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.386 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:43.386 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:43.386 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.386 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.386 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.387 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:43.387 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.387 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.387 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.387 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.387 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.387 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.387 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.388 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.388 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.388 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.388 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.388 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.388 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.388 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.389 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:43.389 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:43.389 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.389 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.389 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.389 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.389 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.389 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.390 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.390 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.390 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.390 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.390 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.390 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.390 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.391 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirroring", 2024-03-28T22:20:43.391 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:43.391 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:43.391 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:43.391 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:43.391 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:43.391 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.391 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.392 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.392 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.392 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.392 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.392 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.392 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.392 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.392 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.393 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.393 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.393 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.393 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.393 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.393 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.393 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.393 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.394 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:43.394 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:43.394 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.394 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.394 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.394 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.394 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.395 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.395 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.395 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.395 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.395 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.395 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.395 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:43.395 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:43.396 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.396 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.396 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.396 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:43.396 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.396 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.396 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.396 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.397 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.397 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.397 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.397 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.397 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.397 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.397 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.398 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.398 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.398 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.398 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:43.398 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:43.398 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.398 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.399 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.399 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.399 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.399 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.399 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.399 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.399 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.399 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.399 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.400 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.400 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.400 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "nfs", 2024-03-28T22:20:43.400 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:43.400 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:43.401 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:43.401 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:43.401 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:43.401 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.401 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.401 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.401 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.401 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.401 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.402 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.402 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.402 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.402 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.402 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.402 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.402 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.402 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.403 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.403 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.403 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.403 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.403 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:43.403 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:43.403 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.404 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.404 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.404 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.404 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.404 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.404 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.404 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.404 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.405 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.405 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.405 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:43.405 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:43.405 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.405 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.405 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.406 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:43.406 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.406 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.406 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.406 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.406 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.406 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.406 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.407 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.407 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.407 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.407 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.407 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.407 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.407 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.408 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:43.408 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:43.408 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.408 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.408 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.408 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.408 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.409 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.409 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.409 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.409 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.409 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.409 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.409 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.409 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.410 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-03-28T22:20:43.410 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:43.410 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:43.410 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:43.410 INFO:teuthology.orchestra.run.smithi016.stdout: "fail_fs": { 2024-03-28T22:20:43.411 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "fail_fs", 2024-03-28T22:20:43.411 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.411 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.411 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.411 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.411 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.411 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.411 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.411 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-28T22:20:43.412 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.412 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.412 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.412 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:43.412 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:43.412 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.412 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.413 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.413 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.413 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.413 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.413 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.413 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.413 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.413 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.413 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.414 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.414 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.414 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.414 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.414 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.414 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.414 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.418 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.418 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.418 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.418 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.418 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.418 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.418 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.418 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.420 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.420 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.420 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.420 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.420 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.420 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.420 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator": { 2024-03-28T22:20:43.420 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-03-28T22:20:43.421 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.421 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.421 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.421 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.421 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.421 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.421 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.421 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm", 2024-03-28T22:20:43.422 INFO:teuthology.orchestra.run.smithi016.stdout: "rook", 2024-03-28T22:20:43.422 INFO:teuthology.orchestra.run.smithi016.stdout: "test_orchestrator" 2024-03-28T22:20:43.422 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.422 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Orchestrator backend", 2024-03-28T22:20:43.422 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.422 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.422 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.423 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.423 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.423 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_perf_query", 2024-03-28T22:20:43.423 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:43.423 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:43.423 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:43.424 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:43.424 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:43.424 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.424 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.424 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.424 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.424 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.424 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.425 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.425 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.425 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.425 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.425 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.425 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.425 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.425 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.426 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.426 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.426 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.426 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.426 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:43.426 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:43.427 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.427 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.427 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.427 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.427 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.427 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.427 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.427 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.428 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.428 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.428 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.428 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:43.428 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:43.428 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.429 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.429 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.429 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:43.429 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.429 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.429 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.429 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.430 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.430 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.430 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.430 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.430 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.430 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.430 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.430 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.431 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.431 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.431 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:43.431 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:43.431 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.431 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.431 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.431 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.432 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.432 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.432 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.432 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.432 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.432 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.432 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.432 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.432 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.433 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.433 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_support", 2024-03-28T22:20:43.433 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:43.433 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:43.433 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:43.433 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:43.433 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:43.433 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.434 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.434 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.434 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.434 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.434 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.434 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.434 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.434 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.435 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.435 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.435 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.435 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.435 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.435 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.435 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.435 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.435 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.436 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:43.436 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:43.436 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.436 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.436 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.436 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.436 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.436 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.437 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.437 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.437 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.437 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.437 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.437 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:43.437 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:43.437 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.438 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.438 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.438 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:43.438 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.438 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.438 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.438 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.438 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.439 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.439 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.439 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.439 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.439 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.439 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.439 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.440 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.440 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.440 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:43.440 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:43.440 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.440 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.440 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.441 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.441 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.441 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.441 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.441 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.441 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.441 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.441 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.441 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.442 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.442 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_autoscaler", 2024-03-28T22:20:43.442 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:43.442 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:43.442 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:43.442 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:43.443 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:43.443 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.443 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.443 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.443 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.443 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.443 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.443 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.443 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.443 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.444 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.444 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.444 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.444 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.444 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.444 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.444 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.444 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.445 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.445 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:43.445 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:43.445 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.445 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.445 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.445 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.445 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.446 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.446 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.446 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.446 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.446 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.446 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.446 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:43.447 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:43.447 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.447 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.447 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.447 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:43.447 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.447 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.448 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.448 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.448 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.448 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.448 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.448 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.448 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.448 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.450 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.450 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.450 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.450 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.450 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.450 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.450 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.451 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.451 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.451 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-03-28T22:20:43.451 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-03-28T22:20:43.451 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:43.451 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.451 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.452 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-03-28T22:20:43.452 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.452 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.452 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.452 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.452 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.452 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.453 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.453 INFO:teuthology.orchestra.run.smithi016.stdout: "threshold": { 2024-03-28T22:20:43.453 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threshold", 2024-03-28T22:20:43.453 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-03-28T22:20:43.453 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.453 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.453 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-03-28T22:20:43.454 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1.0", 2024-03-28T22:20:43.454 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.454 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.454 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "scaling threshold", 2024-03-28T22:20:43.454 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-28T22:20:43.454 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.454 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.454 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.455 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.455 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "progress", 2024-03-28T22:20:43.455 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:43.455 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:43.455 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:43.455 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_pg_recovery_event": { 2024-03-28T22:20:43.455 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_pg_recovery_event", 2024-03-28T22:20:43.456 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.456 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.456 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.456 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.456 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.456 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.456 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.456 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-28T22:20:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-03-28T22:20:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-03-28T22:20:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.458 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.458 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.458 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.458 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.458 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.458 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.458 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.460 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.460 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.460 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.460 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.460 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.460 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.460 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.460 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.460 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.461 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.461 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.461 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:43.461 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:43.461 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.461 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.461 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.462 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.462 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.462 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.462 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.462 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.462 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.462 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.462 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.463 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:43.463 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:43.463 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.463 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.463 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.463 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:43.463 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.464 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.464 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.464 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.464 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.464 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.464 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.464 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.465 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.465 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.465 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.465 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.465 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.465 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.465 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:43.465 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:43.466 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.466 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.466 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.466 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.466 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.466 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.466 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.466 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.467 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.467 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.467 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.467 INFO:teuthology.orchestra.run.smithi016.stdout: "max_completed_events": { 2024-03-28T22:20:43.467 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_completed_events", 2024-03-28T22:20:43.467 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.467 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.467 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.468 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "50", 2024-03-28T22:20:43.468 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.468 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.468 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.468 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "number of past completed events to remember", 2024-03-28T22:20:43.468 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.468 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.468 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-03-28T22:20:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-03-28T22:20:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-03-28T22:20:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.470 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long the module is going to sleep", 2024-03-28T22:20:43.470 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.470 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.470 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.470 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.470 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.470 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.470 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus", 2024-03-28T22:20:43.471 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:43.471 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:43.471 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:43.471 INFO:teuthology.orchestra.run.smithi016.stdout: "cache": { 2024-03-28T22:20:43.471 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cache", 2024-03-28T22:20:43.471 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.471 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.471 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.472 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.472 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.472 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.472 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.472 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.472 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.472 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.472 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.473 INFO:teuthology.orchestra.run.smithi016.stdout: "exclude_perf_counters": { 2024-03-28T22:20:43.473 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "exclude_perf_counters", 2024-03-28T22:20:43.473 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.473 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.473 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.473 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.473 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.474 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.474 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.474 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-28T22:20:43.474 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-28T22:20:43.474 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.474 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.474 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:43.475 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:43.475 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.475 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.475 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.475 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.475 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.475 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.475 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.475 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.476 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.476 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.476 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.476 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.476 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.476 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.476 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.476 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.476 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.476 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.478 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.478 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.478 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.478 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.478 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.478 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:43.478 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:43.478 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.479 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.479 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.479 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:43.479 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.479 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.479 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.479 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.479 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.480 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.480 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.480 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.480 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.480 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.480 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.480 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.480 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.481 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.481 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:43.481 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:43.481 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.481 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.481 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.481 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.481 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.482 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.482 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.482 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.482 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.482 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.482 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.482 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools": { 2024-03-28T22:20:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools", 2024-03-28T22:20:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.484 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.484 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.484 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.484 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.484 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-28T22:20:43.484 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-28T22:20:43.484 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "300", 2024-03-28T22:20:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_interval": { 2024-03-28T22:20:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_interval", 2024-03-28T22:20:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-03-28T22:20:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15.0", 2024-03-28T22:20:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.488 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-03-28T22:20:43.488 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-03-28T22:20:43.488 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.488 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.488 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.488 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-03-28T22:20:43.488 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.488 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.488 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-28T22:20:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-03-28T22:20:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-03-28T22:20:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.490 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.490 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.490 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "9283", 2024-03-28T22:20:43.490 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.490 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.490 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.490 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-28T22:20:43.490 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.490 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.491 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.491 INFO:teuthology.orchestra.run.smithi016.stdout: "stale_cache_strategy": { 2024-03-28T22:20:43.491 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stale_cache_strategy", 2024-03-28T22:20:43.491 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.491 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.491 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.491 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "log", 2024-03-28T22:20:43.491 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-03-28T22:20:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-03-28T22:20:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.493 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.493 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.493 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "default", 2024-03-28T22:20:43.493 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.493 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.493 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.493 INFO:teuthology.orchestra.run.smithi016.stdout: "default", 2024-03-28T22:20:43.494 INFO:teuthology.orchestra.run.smithi016.stdout: "error" 2024-03-28T22:20:43.494 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.494 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.494 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.494 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.494 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.495 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-03-28T22:20:43.495 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-03-28T22:20:43.495 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.495 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.495 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.495 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-03-28T22:20:43.495 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-03-28T22:20:43.495 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-03-28T22:20:43.495 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.496 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.496 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.496 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.496 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.496 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.496 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.496 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_support", 2024-03-28T22:20:43.497 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:43.497 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:43.497 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:43.497 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:43.497 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:43.497 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.497 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.497 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.497 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.498 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.498 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.498 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.498 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.498 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.498 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.498 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.498 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.498 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.499 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.499 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.499 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.499 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.499 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.499 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:43.499 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:43.499 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.500 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.500 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.500 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.500 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.500 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.500 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.500 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.500 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.500 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.501 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.501 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:43.501 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:43.501 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.501 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.501 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.501 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:43.502 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.502 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.502 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.502 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.502 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.502 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.502 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.502 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.503 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.503 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.503 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.503 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.503 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.503 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.503 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:43.504 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:43.504 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.504 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.504 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.504 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.504 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.504 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.504 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.505 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.505 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.505 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.505 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.505 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_snap_create": { 2024-03-28T22:20:43.505 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_snap_create", 2024-03-28T22:20:43.505 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.506 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.506 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.506 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-03-28T22:20:43.506 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.506 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.506 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.506 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.506 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.506 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.507 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.507 INFO:teuthology.orchestra.run.smithi016.stdout: "mirror_snapshot_schedule": { 2024-03-28T22:20:43.507 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirror_snapshot_schedule", 2024-03-28T22:20:43.507 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.507 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.507 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.507 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.507 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.508 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.508 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.508 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.508 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.508 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.508 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.508 INFO:teuthology.orchestra.run.smithi016.stdout: "trash_purge_schedule": { 2024-03-28T22:20:43.509 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "trash_purge_schedule", 2024-03-28T22:20:43.509 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.509 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.509 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.509 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.509 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.509 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.509 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.510 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.510 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.510 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.510 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.510 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.510 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.511 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "restful", 2024-03-28T22:20:43.511 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:43.511 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:43.511 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:43.511 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_auth": { 2024-03-28T22:20:43.511 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_auth", 2024-03-28T22:20:43.511 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.511 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.512 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.512 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.512 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.512 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.512 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.512 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.512 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.512 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.512 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.513 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-03-28T22:20:43.513 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-03-28T22:20:43.513 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.513 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.513 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.513 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.513 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.513 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.513 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.514 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.514 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.514 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.514 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.514 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:43.514 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:43.514 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.515 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.515 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.515 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.515 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.515 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.515 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.515 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.515 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.516 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.516 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.516 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.516 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.516 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.516 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.516 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.516 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.517 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.517 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:43.517 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:43.517 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.517 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.517 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.517 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.517 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.518 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.518 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.518 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.518 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.518 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.519 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.519 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:43.519 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:43.519 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.519 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.519 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.519 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:43.520 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.520 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.520 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.520 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.520 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.520 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.520 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.520 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.521 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.521 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.521 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.521 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.521 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.521 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.521 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:43.521 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:43.522 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.522 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.522 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.522 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.522 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.522 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.522 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.522 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.523 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.523 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.523 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.523 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-03-28T22:20:43.523 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-03-28T22:20:43.523 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.523 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.523 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.524 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.524 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.524 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.524 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.524 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.524 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.524 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.524 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.525 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-03-28T22:20:43.525 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-03-28T22:20:43.525 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.525 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.525 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.525 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.525 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.526 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.526 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.526 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.526 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.526 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.526 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.526 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.526 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.527 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rgw", 2024-03-28T22:20:43.527 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:43.527 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:43.527 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:43.527 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:43.527 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:43.527 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.528 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.528 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.528 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.528 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.528 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.528 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.528 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.528 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.528 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.528 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.529 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.529 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.529 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.529 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.529 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.529 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.529 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.530 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:43.530 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:43.530 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.530 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.530 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.530 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.530 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.531 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.531 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.531 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.531 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.531 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.531 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.531 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:43.531 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:43.532 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.532 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.532 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.532 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:43.532 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.532 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.532 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.533 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.533 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.533 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.533 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.533 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.533 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.533 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.533 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.533 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.534 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.534 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.534 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:43.534 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:43.534 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.534 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.534 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.535 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.535 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.535 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.535 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.535 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.535 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.535 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.535 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.535 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.536 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.536 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rook", 2024-03-28T22:20:43.536 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:43.536 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:43.536 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:43.537 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:43.537 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:43.537 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.537 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.537 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.537 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.537 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.537 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.537 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.538 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.538 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.538 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.538 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.538 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.538 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.538 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.538 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.538 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.539 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.539 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.539 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:43.539 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:43.539 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.539 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.539 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.539 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.540 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.540 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.540 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.540 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.540 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.540 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.540 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.541 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:43.541 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:43.541 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.541 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.541 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.541 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:43.541 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.541 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.542 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.542 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.542 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.542 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.542 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.542 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.542 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.542 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.543 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.543 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.543 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.543 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.543 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:43.543 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:43.543 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.543 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.544 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.544 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.544 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.544 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.544 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.544 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.544 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.544 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.544 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.545 INFO:teuthology.orchestra.run.smithi016.stdout: "storage_class": { 2024-03-28T22:20:43.545 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "storage_class", 2024-03-28T22:20:43.545 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.545 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.545 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.545 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "local", 2024-03-28T22:20:43.545 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.545 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.546 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.546 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-28T22:20:43.546 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.546 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.546 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.546 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.546 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.547 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "selftest", 2024-03-28T22:20:43.547 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:43.547 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:43.547 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:43.547 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:43.547 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:43.547 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.548 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.548 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.548 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.548 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.548 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.548 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.548 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.549 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.549 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.549 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.549 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.549 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.549 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.549 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.549 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.550 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.550 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.550 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:43.550 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:43.550 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.550 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.550 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.550 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.551 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.551 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.551 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.551 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.551 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.551 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.551 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.551 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:43.552 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:43.552 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.552 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.552 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.552 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:43.552 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.552 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.552 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.552 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.553 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.553 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.553 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.553 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.553 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.553 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.553 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.553 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.553 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.553 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.554 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:43.554 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:43.554 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.554 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.554 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.554 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.554 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.555 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.555 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.555 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.555 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.555 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.555 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.555 INFO:teuthology.orchestra.run.smithi016.stdout: "roption1": { 2024-03-28T22:20:43.555 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption1", 2024-03-28T22:20:43.556 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.556 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.556 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.556 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.556 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.556 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.556 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.556 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.557 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.557 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.557 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.557 INFO:teuthology.orchestra.run.smithi016.stdout: "roption2": { 2024-03-28T22:20:43.557 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption2", 2024-03-28T22:20:43.557 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.557 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.557 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.558 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "xyz", 2024-03-28T22:20:43.558 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.558 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.558 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.558 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.558 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.558 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.558 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.559 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption1": { 2024-03-28T22:20:43.559 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption1", 2024-03-28T22:20:43.559 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.559 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.559 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.559 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.559 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.559 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.560 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.560 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.560 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.560 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.560 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.560 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption2": { 2024-03-28T22:20:43.560 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption2", 2024-03-28T22:20:43.561 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.561 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.561 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.561 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.561 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.561 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.561 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.561 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.562 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.562 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.562 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.562 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption3": { 2024-03-28T22:20:43.562 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption3", 2024-03-28T22:20:43.562 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-03-28T22:20:43.562 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.563 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.563 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.563 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.563 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.563 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.563 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.563 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.563 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.564 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.564 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption4": { 2024-03-28T22:20:43.564 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption4", 2024-03-28T22:20:43.564 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.564 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.564 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.564 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.565 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.565 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.565 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.565 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.565 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.565 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.565 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.566 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption5": { 2024-03-28T22:20:43.566 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption5", 2024-03-28T22:20:43.566 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.566 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.566 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.566 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.566 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.567 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.567 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.567 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.567 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.567 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.567 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.567 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption6": { 2024-03-28T22:20:43.567 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption6", 2024-03-28T22:20:43.568 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.568 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.568 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.568 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.568 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.568 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.568 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.568 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.569 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.569 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.569 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.569 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption7": { 2024-03-28T22:20:43.569 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption7", 2024-03-28T22:20:43.569 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.569 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.569 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.569 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.570 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-03-28T22:20:43.570 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "42", 2024-03-28T22:20:43.570 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.570 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.570 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.570 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.570 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.571 INFO:teuthology.orchestra.run.smithi016.stdout: "testkey": { 2024-03-28T22:20:43.571 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testkey", 2024-03-28T22:20:43.571 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.571 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.571 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.571 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.571 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.571 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.571 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.571 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.572 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.572 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.572 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.572 INFO:teuthology.orchestra.run.smithi016.stdout: "testlkey": { 2024-03-28T22:20:43.572 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testlkey", 2024-03-28T22:20:43.572 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.572 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.572 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.573 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.573 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.573 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.573 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.573 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.573 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.573 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.573 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.573 INFO:teuthology.orchestra.run.smithi016.stdout: "testnewline": { 2024-03-28T22:20:43.574 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testnewline", 2024-03-28T22:20:43.574 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.574 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.574 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.574 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.574 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.574 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.574 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.575 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.575 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.575 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.575 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.575 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.575 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.575 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snap_schedule", 2024-03-28T22:20:43.576 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:43.576 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:43.576 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:43.576 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_m_granularity": { 2024-03-28T22:20:43.576 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_m_granularity", 2024-03-28T22:20:43.576 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.576 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.576 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.577 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.577 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.577 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.577 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.577 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow minute scheduled snapshots", 2024-03-28T22:20:43.577 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.577 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.577 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.578 INFO:teuthology.orchestra.run.smithi016.stdout: "dump_on_update": { 2024-03-28T22:20:43.578 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dump_on_update", 2024-03-28T22:20:43.578 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.578 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.578 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.578 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.578 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.579 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.579 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.579 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "dump database to debug log on update", 2024-03-28T22:20:43.579 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.579 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.579 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.579 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:43.581 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:43.581 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.581 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.582 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.582 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.582 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.582 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.582 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.582 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.582 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.583 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.583 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.583 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.583 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.583 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.583 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.583 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.583 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.583 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.584 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:43.584 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:43.584 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.584 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.584 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.584 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.584 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.585 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.585 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.585 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.585 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.585 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.585 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.585 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:43.585 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:43.586 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.586 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.586 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.586 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:43.586 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.586 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.586 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.587 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.587 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.587 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.587 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.587 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.587 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.587 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.587 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.588 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.588 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.588 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.588 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:43.588 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:43.588 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.588 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.589 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.589 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.589 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.589 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.589 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.589 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.589 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.589 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.590 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.590 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.590 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.590 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stats", 2024-03-28T22:20:43.590 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:43.591 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:43.591 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:43.591 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:43.591 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:43.591 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.591 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.591 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.592 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.592 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.592 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.592 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.592 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.592 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.592 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.592 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.593 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.593 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.593 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.593 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.593 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.593 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.593 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.594 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:43.594 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:43.594 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.594 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.594 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.594 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.594 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.595 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.595 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.595 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.595 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.595 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.595 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.595 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:43.595 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:43.596 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.596 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.596 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.596 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:43.596 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.596 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.596 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.596 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.597 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.597 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.597 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.597 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.597 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.597 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.597 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.597 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.597 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.598 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.598 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:43.598 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:43.598 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.598 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.598 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.598 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.599 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.599 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.599 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.599 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.599 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.599 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.599 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.599 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.600 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.600 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "status", 2024-03-28T22:20:43.600 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:43.600 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:43.600 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:43.600 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:43.600 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:43.601 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.601 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.601 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.601 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.601 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.601 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.601 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.601 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.602 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.602 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.602 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.602 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.602 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.603 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.603 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.603 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.603 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.603 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.603 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:43.603 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:43.604 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.604 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.604 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.604 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.604 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.604 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.604 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.605 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.605 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.605 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.605 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.605 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:43.605 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:43.605 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.605 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.606 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.606 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:43.606 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.606 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.606 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.606 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.606 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.607 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.607 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.607 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.607 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.607 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.607 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.607 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.607 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.608 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.608 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:43.608 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:43.608 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.608 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.608 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.608 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.609 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.609 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.609 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.609 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.609 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.609 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.609 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.609 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.610 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.610 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telegraf", 2024-03-28T22:20:43.610 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:43.610 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:43.610 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:43.610 INFO:teuthology.orchestra.run.smithi016.stdout: "address": { 2024-03-28T22:20:43.611 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "address", 2024-03-28T22:20:43.611 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.611 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.611 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.611 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-28T22:20:43.611 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.611 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.611 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.612 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.612 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.612 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.612 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.612 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-03-28T22:20:43.612 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-03-28T22:20:43.612 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:43.613 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.613 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.613 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15", 2024-03-28T22:20:43.613 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.613 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.613 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.613 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.613 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.614 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.614 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.614 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:43.614 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:43.614 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.614 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.614 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.615 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.615 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.615 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.615 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.615 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.615 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.615 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.616 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.616 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.616 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.616 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.616 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.616 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.616 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.616 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.617 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:43.617 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:43.617 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.617 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.617 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.617 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.617 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.617 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.618 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.618 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.618 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.618 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.618 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.618 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:43.618 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:43.619 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.619 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.619 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.619 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:43.619 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.619 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.619 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.619 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.620 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.620 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.620 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.620 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.620 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.620 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.620 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.620 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.620 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.621 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.621 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:43.621 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:43.621 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.621 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.621 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.621 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.622 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.622 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.622 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.622 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.622 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.622 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.622 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.623 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.623 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.623 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telemetry", 2024-03-28T22:20:43.623 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:43.623 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:43.623 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:43.623 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_basic": { 2024-03-28T22:20:43.624 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_basic", 2024-03-28T22:20:43.624 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.624 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.624 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.624 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.624 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.624 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.624 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.625 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-28T22:20:43.625 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.625 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.625 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.625 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_crash": { 2024-03-28T22:20:43.625 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_crash", 2024-03-28T22:20:43.625 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.626 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.626 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.626 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.626 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.626 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.626 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.626 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-28T22:20:43.626 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.627 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.627 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.627 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_device": { 2024-03-28T22:20:43.627 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_device", 2024-03-28T22:20:43.627 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.627 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.627 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.627 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.628 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.628 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.628 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.628 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-28T22:20:43.628 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.628 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.628 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.629 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_ident": { 2024-03-28T22:20:43.629 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_ident", 2024-03-28T22:20:43.629 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.629 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.629 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.629 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.629 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.629 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.629 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.630 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-28T22:20:43.630 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.630 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.630 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.630 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_perf": { 2024-03-28T22:20:43.630 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_perf", 2024-03-28T22:20:43.630 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.631 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.631 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.631 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.631 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.631 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.631 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.631 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-28T22:20:43.631 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.632 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.632 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.632 INFO:teuthology.orchestra.run.smithi016.stdout: "contact": { 2024-03-28T22:20:43.632 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "contact", 2024-03-28T22:20:43.632 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.632 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.632 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.633 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.633 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.633 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.633 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.633 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.633 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.633 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.633 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.634 INFO:teuthology.orchestra.run.smithi016.stdout: "description": { 2024-03-28T22:20:43.634 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "description", 2024-03-28T22:20:43.634 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.634 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.634 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.634 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.635 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.635 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.635 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.635 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.635 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.635 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.635 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.636 INFO:teuthology.orchestra.run.smithi016.stdout: "device_url": { 2024-03-28T22:20:43.636 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_url", 2024-03-28T22:20:43.636 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.636 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.636 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.636 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-28T22:20:43.637 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.637 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.637 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.637 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.637 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.637 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.637 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.637 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-03-28T22:20:43.638 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-03-28T22:20:43.638 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.638 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.638 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.638 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.638 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.638 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.639 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.639 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.639 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.639 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.639 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.639 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-03-28T22:20:43.639 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-03-28T22:20:43.640 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.640 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.640 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.640 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "24", 2024-03-28T22:20:43.640 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "8", 2024-03-28T22:20:43.640 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.640 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.640 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.641 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.641 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.641 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.641 INFO:teuthology.orchestra.run.smithi016.stdout: "last_opt_revision": { 2024-03-28T22:20:43.641 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "last_opt_revision", 2024-03-28T22:20:43.641 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.641 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.642 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.642 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1", 2024-03-28T22:20:43.642 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.642 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.642 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.642 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.642 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.642 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.643 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.643 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard": { 2024-03-28T22:20:43.643 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard", 2024-03-28T22:20:43.643 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.643 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.643 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.643 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.644 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.644 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.644 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.644 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.644 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.644 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.644 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.645 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard_description": { 2024-03-28T22:20:43.645 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard_description", 2024-03-28T22:20:43.645 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.645 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.645 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.645 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.645 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.645 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.646 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.646 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.646 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.646 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.646 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.646 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:43.646 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:43.647 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.647 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.647 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.647 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.647 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.647 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.647 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.647 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.648 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.648 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.648 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.648 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.648 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.648 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.648 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.649 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.649 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.649 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.649 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:43.649 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:43.649 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.649 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.650 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.650 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.650 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.650 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.650 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.650 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.650 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.651 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.651 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.651 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:43.651 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:43.651 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.651 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.651 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.652 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:43.652 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.652 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.652 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.652 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.652 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.652 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.652 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.653 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.653 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.653 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.653 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.653 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.653 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.653 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.654 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:43.654 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:43.654 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.654 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.654 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.654 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.654 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.655 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.655 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.655 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.655 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.655 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.655 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.655 INFO:teuthology.orchestra.run.smithi016.stdout: "organization": { 2024-03-28T22:20:43.656 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "organization", 2024-03-28T22:20:43.656 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.656 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.656 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.656 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.656 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.656 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.656 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.656 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.657 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.657 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.657 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.657 INFO:teuthology.orchestra.run.smithi016.stdout: "proxy": { 2024-03-28T22:20:43.657 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "proxy", 2024-03-28T22:20:43.657 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.657 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.658 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.658 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.658 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.658 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.658 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.658 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.658 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.659 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.659 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.659 INFO:teuthology.orchestra.run.smithi016.stdout: "url": { 2024-03-28T22:20:43.659 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url", 2024-03-28T22:20:43.659 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.659 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.659 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.660 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-28T22:20:43.660 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.660 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.660 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.660 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.660 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.660 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.661 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.661 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.661 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.661 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "test_orchestrator", 2024-03-28T22:20:43.661 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:43.662 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:43.662 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:43.662 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:43.662 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:43.662 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.662 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.662 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.662 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.663 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.663 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.663 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.663 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.663 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.663 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.663 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.664 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.664 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.664 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.664 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.664 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.664 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.664 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.665 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:43.665 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:43.665 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.665 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.665 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.665 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.665 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.665 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.665 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.666 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.666 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.666 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.666 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.666 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:43.666 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:43.667 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.667 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.667 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.667 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:43.667 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.667 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.667 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.668 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.668 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.668 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.668 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.668 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.668 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.668 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.668 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.668 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.669 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.669 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.669 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:43.669 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:43.669 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.669 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.670 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.670 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.670 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.670 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.670 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.670 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.670 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.670 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.671 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.671 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.671 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.671 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "volumes", 2024-03-28T22:20:43.671 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:43.671 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:43.672 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:43.672 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:43.672 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:43.672 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.672 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.672 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.672 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.672 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.673 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.673 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.673 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.673 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.673 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.673 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.673 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.673 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.674 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.674 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.674 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.674 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.674 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.674 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:43.675 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:43.675 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.675 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.675 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.675 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.675 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.675 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.675 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.676 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.676 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.676 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.676 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.676 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:43.676 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:43.676 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.676 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.677 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.677 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:43.677 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.677 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.677 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.677 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.677 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.677 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.678 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.678 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.678 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.678 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.678 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.678 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.678 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.678 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.679 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:43.679 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:43.679 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.679 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.679 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.679 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.679 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.680 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.680 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.680 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.680 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.680 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.680 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.680 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_clones": { 2024-03-28T22:20:43.680 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_clones", 2024-03-28T22:20:43.681 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.681 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.681 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.681 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4", 2024-03-28T22:20:43.681 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.681 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.681 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.681 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-28T22:20:43.682 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.682 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.682 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.682 INFO:teuthology.orchestra.run.smithi016.stdout: "periodic_async_work": { 2024-03-28T22:20:43.682 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "periodic_async_work", 2024-03-28T22:20:43.682 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.683 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.683 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.683 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.683 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.683 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.683 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.683 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Periodically check for async work", 2024-03-28T22:20:43.684 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.684 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.684 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.684 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_delay": { 2024-03-28T22:20:43.684 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_delay", 2024-03-28T22:20:43.684 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.684 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.685 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.685 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-03-28T22:20:43.685 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.685 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.685 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.685 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-28T22:20:43.685 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.685 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.685 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.686 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_no_wait": { 2024-03-28T22:20:43.686 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_no_wait", 2024-03-28T22:20:43.686 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.686 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.686 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.686 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.687 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.687 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.687 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.687 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-28T22:20:43.687 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.687 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.687 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.688 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.688 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.688 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix", 2024-03-28T22:20:43.688 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:43.688 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:43.688 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:43.688 INFO:teuthology.orchestra.run.smithi016.stdout: "discovery_interval": { 2024-03-28T22:20:43.689 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "discovery_interval", 2024-03-28T22:20:43.689 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-03-28T22:20:43.689 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.689 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.689 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "100", 2024-03-28T22:20:43.689 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.689 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.690 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.690 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.690 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.690 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.690 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.690 INFO:teuthology.orchestra.run.smithi016.stdout: "identifier": { 2024-03-28T22:20:43.691 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "identifier", 2024-03-28T22:20:43.691 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.691 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.691 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.691 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.691 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.691 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.691 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.692 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.692 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.692 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.692 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.692 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-03-28T22:20:43.692 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-03-28T22:20:43.693 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:43.693 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.693 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.693 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-03-28T22:20:43.693 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.693 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.693 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.693 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.694 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.694 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.694 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.694 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:43.694 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:43.694 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.695 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.695 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.695 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.695 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.695 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.695 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.695 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.695 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.696 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.696 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.696 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.696 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.696 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.696 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.696 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.696 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.697 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.697 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:43.697 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:43.697 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.697 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.697 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.697 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.698 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.698 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.698 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.698 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.698 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.698 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.698 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.699 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:43.699 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:43.699 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.699 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.699 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.699 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:43.699 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.699 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.699 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.700 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.700 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.700 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.700 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.700 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.700 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.700 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.700 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.701 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.701 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.701 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.701 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:43.701 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:43.701 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.701 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.702 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.702 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.702 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.702 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.702 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.702 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.702 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.703 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.703 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.703 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_host": { 2024-03-28T22:20:43.703 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_host", 2024-03-28T22:20:43.703 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.703 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.703 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.704 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.704 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.704 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.704 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.704 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.704 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.704 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.704 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.705 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_port": { 2024-03-28T22:20:43.705 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_port", 2024-03-28T22:20:43.705 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.705 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.705 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.705 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10051", 2024-03-28T22:20:43.705 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.706 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.706 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.706 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.706 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.706 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.706 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.706 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_sender": { 2024-03-28T22:20:43.707 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_sender", 2024-03-28T22:20:43.707 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.707 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.707 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.707 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-28T22:20:43.707 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.707 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.707 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.708 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.708 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.708 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.708 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.708 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.708 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.708 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.709 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-03-28T22:20:43.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.709 INFO:teuthology.orchestra.run.smithi016.stdout: "gid": 25349, 2024-03-28T22:20:43.709 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "z", 2024-03-28T22:20:43.709 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_features": 4540701547738038271, 2024-03-28T22:20:43.709 INFO:teuthology.orchestra.run.smithi016.stdout: "available_modules": [ 2024-03-28T22:20:43.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.709 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "alerts", 2024-03-28T22:20:43.710 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:43.710 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:43.710 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:43.710 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-03-28T22:20:43.710 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-03-28T22:20:43.710 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:43.710 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.711 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.711 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-03-28T22:20:43.711 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.711 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.711 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.711 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How frequently to reexamine health status", 2024-03-28T22:20:43.711 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.711 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.712 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.712 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:43.712 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:43.712 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.712 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.712 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.712 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.713 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.713 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.713 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.713 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.713 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.713 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.713 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.714 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.714 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.714 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.714 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.714 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.714 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.714 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.715 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:43.715 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:43.715 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.715 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.715 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.715 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.715 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.715 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.716 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.716 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.716 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.716 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.716 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.716 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:43.716 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:43.716 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.717 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.717 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.717 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:43.717 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.717 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.717 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.717 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.717 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.718 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.718 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.718 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.718 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.718 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.718 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.718 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.719 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.719 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.719 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:43.719 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:43.719 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.719 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.719 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.720 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.720 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.720 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.720 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.720 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.720 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.720 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.720 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.721 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_destination": { 2024-03-28T22:20:43.721 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_destination", 2024-03-28T22:20:43.721 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.721 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.721 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.721 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.721 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.722 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.722 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.722 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email address to send alerts to", 2024-03-28T22:20:43.722 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.722 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.722 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.722 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_from_name": { 2024-03-28T22:20:43.723 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_from_name", 2024-03-28T22:20:43.723 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.723 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.723 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.723 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "Ceph", 2024-03-28T22:20:43.723 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.723 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.724 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.724 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email From: name", 2024-03-28T22:20:43.724 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.724 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.724 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.724 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_host": { 2024-03-28T22:20:43.724 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_host", 2024-03-28T22:20:43.725 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.725 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.725 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.725 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.725 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.725 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.725 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.725 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP server", 2024-03-28T22:20:43.726 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.726 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.726 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.726 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_password": { 2024-03-28T22:20:43.726 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_password", 2024-03-28T22:20:43.726 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.726 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.727 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.727 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.727 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.727 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.727 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.727 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Password to authenticate with", 2024-03-28T22:20:43.727 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.728 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.728 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.728 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_port": { 2024-03-28T22:20:43.728 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_port", 2024-03-28T22:20:43.728 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.728 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.728 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.728 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "465", 2024-03-28T22:20:43.729 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.729 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.729 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.729 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP port", 2024-03-28T22:20:43.729 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.729 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.729 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.730 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_sender": { 2024-03-28T22:20:43.730 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_sender", 2024-03-28T22:20:43.730 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.730 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.730 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.730 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.730 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.731 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.731 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.731 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP envelope sender", 2024-03-28T22:20:43.731 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.731 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.731 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.731 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_ssl": { 2024-03-28T22:20:43.731 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_ssl", 2024-03-28T22:20:43.732 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.732 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.732 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.732 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.732 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.732 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.732 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.732 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-28T22:20:43.733 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.733 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.733 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.733 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_user": { 2024-03-28T22:20:43.733 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_user", 2024-03-28T22:20:43.733 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.733 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.734 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.734 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.734 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.734 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.734 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.734 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "User to authenticate as", 2024-03-28T22:20:43.734 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.734 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.735 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.735 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.735 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.735 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "balancer", 2024-03-28T22:20:43.735 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:43.735 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:43.736 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:43.736 INFO:teuthology.orchestra.run.smithi016.stdout: "active": { 2024-03-28T22:20:43.736 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "active", 2024-03-28T22:20:43.736 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.736 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.736 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.736 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.736 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.737 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.737 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.737 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically balance PGs across cluster", 2024-03-28T22:20:43.737 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.737 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.737 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.737 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_time": { 2024-03-28T22:20:43.738 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_time", 2024-03-28T22:20:43.738 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.738 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.738 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.738 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0000", 2024-03-28T22:20:43.738 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.738 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.739 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.739 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "beginning time of day to automatically balance", 2024-03-28T22:20:43.739 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T22:20:43.739 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.739 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.739 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_weekday": { 2024-03-28T22:20:43.739 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_weekday", 2024-03-28T22:20:43.740 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-03-28T22:20:43.740 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.740 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.740 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-03-28T22:20:43.740 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-03-28T22:20:43.740 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-03-28T22:20:43.740 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.740 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-28T22:20:43.741 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T22:20:43.741 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.741 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.741 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_max_iterations": { 2024-03-28T22:20:43.741 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_max_iterations", 2024-03-28T22:20:43.741 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-03-28T22:20:43.741 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.742 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.742 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "25", 2024-03-28T22:20:43.742 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-03-28T22:20:43.742 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "250", 2024-03-28T22:20:43.742 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.742 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-28T22:20:43.742 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.742 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.743 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.743 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_metrics": { 2024-03-28T22:20:43.743 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_metrics", 2024-03-28T22:20:43.743 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.743 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.743 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.743 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "pgs,objects,bytes", 2024-03-28T22:20:43.744 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.744 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.744 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.744 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-28T22:20:43.744 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-28T22:20:43.744 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.744 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.745 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_step": { 2024-03-28T22:20:43.745 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_step", 2024-03-28T22:20:43.745 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-03-28T22:20:43.745 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.745 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.745 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.5", 2024-03-28T22:20:43.745 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0.001", 2024-03-28T22:20:43.745 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "0.999", 2024-03-28T22:20:43.746 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.746 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "aggressiveness of optimization", 2024-03-28T22:20:43.746 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-28T22:20:43.746 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.746 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.746 INFO:teuthology.orchestra.run.smithi016.stdout: "end_time": { 2024-03-28T22:20:43.747 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_time", 2024-03-28T22:20:43.747 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.747 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.747 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.747 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2359", 2024-03-28T22:20:43.747 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.747 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.747 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.747 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ending time of day to automatically balance", 2024-03-28T22:20:43.748 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T22:20:43.748 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.748 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.748 INFO:teuthology.orchestra.run.smithi016.stdout: "end_weekday": { 2024-03-28T22:20:43.748 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_weekday", 2024-03-28T22:20:43.748 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-03-28T22:20:43.748 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.749 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.749 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-03-28T22:20:43.749 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-03-28T22:20:43.749 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-03-28T22:20:43.749 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.749 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-28T22:20:43.749 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T22:20:43.749 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.750 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.750 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:43.750 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:43.750 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.750 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.750 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.750 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.751 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.751 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.751 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.751 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.751 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.751 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.751 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.751 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.752 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.752 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.752 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.752 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.752 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.752 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.753 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:43.753 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:43.753 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.753 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.753 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.753 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.753 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.754 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.754 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.754 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.754 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.754 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.754 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.754 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:43.755 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:43.755 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.755 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.755 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.755 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:43.755 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.755 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.755 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.755 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.756 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.756 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.756 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.756 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.756 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.756 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.756 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.756 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.756 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.757 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.757 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:43.757 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:43.757 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.757 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.757 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.757 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.758 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.758 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.758 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.758 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.758 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.758 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.758 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.759 INFO:teuthology.orchestra.run.smithi016.stdout: "min_score": { 2024-03-28T22:20:43.759 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_score", 2024-03-28T22:20:43.759 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-03-28T22:20:43.759 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.759 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.759 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-03-28T22:20:43.759 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.759 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.760 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.760 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-28T22:20:43.760 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.760 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.760 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.760 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-03-28T22:20:43.760 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-03-28T22:20:43.761 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.761 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.761 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.761 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "upmap", 2024-03-28T22:20:43.761 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.761 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.761 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.761 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-compat", 2024-03-28T22:20:43.761 INFO:teuthology.orchestra.run.smithi016.stdout: "none", 2024-03-28T22:20:43.762 INFO:teuthology.orchestra.run.smithi016.stdout: "read", 2024-03-28T22:20:43.762 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap", 2024-03-28T22:20:43.762 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap-read" 2024-03-28T22:20:43.762 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.762 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Balancer mode", 2024-03-28T22:20:43.762 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.762 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.763 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.763 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_ids": { 2024-03-28T22:20:43.763 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_ids", 2024-03-28T22:20:43.763 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.763 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.763 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.763 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.764 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.764 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.764 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.764 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-28T22:20:43.764 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.764 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.764 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.764 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-03-28T22:20:43.765 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-03-28T22:20:43.765 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:43.765 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.765 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.765 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-03-28T22:20:43.765 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.765 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.765 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.765 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-28T22:20:43.766 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.766 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.766 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.766 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_deviation": { 2024-03-28T22:20:43.766 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_deviation", 2024-03-28T22:20:43.766 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.766 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.767 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.767 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-03-28T22:20:43.767 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-03-28T22:20:43.767 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.767 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.767 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-28T22:20:43.767 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-28T22:20:43.767 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.768 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.768 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_optimizations": { 2024-03-28T22:20:43.768 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_optimizations", 2024-03-28T22:20:43.768 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-03-28T22:20:43.768 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.768 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.768 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-03-28T22:20:43.768 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.769 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.769 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.769 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-28T22:20:43.769 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.769 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.769 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.769 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.769 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.770 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm", 2024-03-28T22:20:43.770 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:43.770 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:43.770 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:43.770 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_down_multiplier": { 2024-03-28T22:20:43.771 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_down_multiplier", 2024-03-28T22:20:43.771 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-03-28T22:20:43.771 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.771 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.771 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-03-28T22:20:43.771 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.771 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.771 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.771 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-28T22:20:43.772 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.772 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.772 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.772 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_refresh_rate": { 2024-03-28T22:20:43.772 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_refresh_rate", 2024-03-28T22:20:43.772 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:43.772 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.773 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.773 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "20", 2024-03-28T22:20:43.773 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.773 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.773 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.773 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-28T22:20:43.773 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.773 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.774 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.774 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_starting_port": { 2024-03-28T22:20:43.774 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_starting_port", 2024-03-28T22:20:43.774 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.774 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.774 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.774 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4721", 2024-03-28T22:20:43.775 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.775 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.775 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.775 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-28T22:20:43.775 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.775 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.775 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.776 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_ptrace": { 2024-03-28T22:20:43.776 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_ptrace", 2024-03-28T22:20:43.776 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.776 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.776 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.776 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.776 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.776 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.777 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.777 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-28T22:20:43.777 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-28T22:20:43.777 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.777 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.777 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_interval": { 2024-03-28T22:20:43.777 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_interval", 2024-03-28T22:20:43.777 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:43.778 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.778 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.778 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-03-28T22:20:43.778 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.778 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.778 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.778 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-28T22:20:43.778 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.779 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.779 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.779 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_memory_target_ratio": { 2024-03-28T22:20:43.779 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_memory_target_ratio", 2024-03-28T22:20:43.779 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-03-28T22:20:43.779 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.779 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.779 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.7", 2024-03-28T22:20:43.780 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.780 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.780 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.780 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-28T22:20:43.780 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.780 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.780 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.781 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm_log_destination": { 2024-03-28T22:20:43.781 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm_log_destination", 2024-03-28T22:20:43.781 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.781 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.781 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.781 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.781 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.781 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.782 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.782 INFO:teuthology.orchestra.run.smithi016.stdout: "file", 2024-03-28T22:20:43.782 INFO:teuthology.orchestra.run.smithi016.stdout: "file,syslog", 2024-03-28T22:20:43.782 INFO:teuthology.orchestra.run.smithi016.stdout: "syslog" 2024-03-28T22:20:43.782 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.782 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-28T22:20:43.782 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.782 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.783 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.783 INFO:teuthology.orchestra.run.smithi016.stdout: "cgroups_split": { 2024-03-28T22:20:43.783 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cgroups_split", 2024-03-28T22:20:43.783 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.783 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.783 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.783 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.784 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.785 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.785 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.785 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-28T22:20:43.785 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.786 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.786 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.786 INFO:teuthology.orchestra.run.smithi016.stdout: "config_checks_enabled": { 2024-03-28T22:20:43.786 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_checks_enabled", 2024-03-28T22:20:43.786 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.786 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.786 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.787 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.787 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.787 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.787 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.787 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-28T22:20:43.787 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.787 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.787 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.788 INFO:teuthology.orchestra.run.smithi016.stdout: "config_dashboard": { 2024-03-28T22:20:43.788 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_dashboard", 2024-03-28T22:20:43.788 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.788 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.788 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.788 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.788 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.789 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.789 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.789 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-28T22:20:43.789 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.789 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.789 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.789 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_alertmanager": { 2024-03-28T22:20:43.790 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_alertmanager", 2024-03-28T22:20:43.790 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.790 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.790 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.790 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-28T22:20:43.790 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.790 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.790 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.791 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-03-28T22:20:43.791 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.791 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.791 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.791 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_base": { 2024-03-28T22:20:43.791 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_base", 2024-03-28T22:20:43.791 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.792 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.792 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.792 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-28T22:20:43.792 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.792 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.792 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.792 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Container image name, without the tag", 2024-03-28T22:20:43.792 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.793 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.793 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.793 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_elasticsearch": { 2024-03-28T22:20:43.793 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_elasticsearch", 2024-03-28T22:20:43.793 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.793 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.793 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.794 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-28T22:20:43.794 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.794 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.794 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.794 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "elasticsearch container image", 2024-03-28T22:20:43.794 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.794 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.795 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.795 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_grafana": { 2024-03-28T22:20:43.795 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_grafana", 2024-03-28T22:20:43.795 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.795 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.795 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.795 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-28T22:20:43.796 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.796 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.796 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.796 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-03-28T22:20:43.796 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.796 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.796 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.797 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_haproxy": { 2024-03-28T22:20:43.797 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_haproxy", 2024-03-28T22:20:43.797 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.797 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.797 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.797 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-28T22:20:43.797 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.798 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.798 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.798 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "HAproxy container image", 2024-03-28T22:20:43.798 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.798 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.798 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.799 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_agent": { 2024-03-28T22:20:43.799 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_agent", 2024-03-28T22:20:43.799 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.799 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.799 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.799 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-28T22:20:43.799 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.799 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.800 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.800 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger agent container image", 2024-03-28T22:20:43.800 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.800 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.800 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.800 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_collector": { 2024-03-28T22:20:43.800 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_collector", 2024-03-28T22:20:43.800 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.801 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.801 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.801 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-28T22:20:43.801 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.801 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.801 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.801 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger collector container image", 2024-03-28T22:20:43.801 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.802 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.802 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.802 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_query": { 2024-03-28T22:20:43.802 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_query", 2024-03-28T22:20:43.802 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.802 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.803 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.803 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-28T22:20:43.803 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.803 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.803 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.803 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger query container image", 2024-03-28T22:20:43.803 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.803 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.804 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.804 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_keepalived": { 2024-03-28T22:20:43.804 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_keepalived", 2024-03-28T22:20:43.804 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.804 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.804 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.804 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-28T22:20:43.804 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.805 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.805 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.805 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Keepalived container image", 2024-03-28T22:20:43.805 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.805 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.805 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.805 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_loki": { 2024-03-28T22:20:43.806 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_loki", 2024-03-28T22:20:43.806 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.806 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.806 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.806 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-28T22:20:43.806 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.806 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.807 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.807 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Loki container image", 2024-03-28T22:20:43.807 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.807 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.807 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.807 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_node_exporter": { 2024-03-28T22:20:43.807 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_node_exporter", 2024-03-28T22:20:43.808 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.808 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.808 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.808 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-28T22:20:43.808 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.808 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.808 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.808 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-03-28T22:20:43.809 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.809 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.809 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.809 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_nvmeof": { 2024-03-28T22:20:43.809 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_nvmeof", 2024-03-28T22:20:43.809 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.809 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.809 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.810 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-28T22:20:43.810 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.810 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.810 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.810 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Nvme-of container image", 2024-03-28T22:20:43.810 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.810 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.810 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.811 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_prometheus": { 2024-03-28T22:20:43.811 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_prometheus", 2024-03-28T22:20:43.811 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.811 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.811 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.811 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-28T22:20:43.811 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.812 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.812 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.812 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-03-28T22:20:43.812 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.812 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.812 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.812 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_promtail": { 2024-03-28T22:20:43.813 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_promtail", 2024-03-28T22:20:43.813 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.813 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.813 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.813 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-28T22:20:43.813 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.813 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.813 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.814 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Promtail container image", 2024-03-28T22:20:43.814 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.814 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.814 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.814 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_snmp_gateway": { 2024-03-28T22:20:43.814 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_snmp_gateway", 2024-03-28T22:20:43.814 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.815 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.815 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.815 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-28T22:20:43.815 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.815 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.815 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.815 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SNMP Gateway container image", 2024-03-28T22:20:43.815 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.816 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.816 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.816 INFO:teuthology.orchestra.run.smithi016.stdout: "container_init": { 2024-03-28T22:20:43.816 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_init", 2024-03-28T22:20:43.816 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.816 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.816 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.817 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.817 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.817 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.817 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.817 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Run podman/docker with `--init`", 2024-03-28T22:20:43.817 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.817 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.817 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.818 INFO:teuthology.orchestra.run.smithi016.stdout: "daemon_cache_timeout": { 2024-03-28T22:20:43.818 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "daemon_cache_timeout", 2024-03-28T22:20:43.818 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:43.818 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.818 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.818 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-03-28T22:20:43.818 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.819 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.819 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.819 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-28T22:20:43.819 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.819 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.819 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.820 INFO:teuthology.orchestra.run.smithi016.stdout: "default_cephadm_command_timeout": { 2024-03-28T22:20:43.820 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_cephadm_command_timeout", 2024-03-28T22:20:43.820 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.820 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.820 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.820 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "900", 2024-03-28T22:20:43.820 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.820 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.821 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.821 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-28T22:20:43.821 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.821 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.821 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.821 INFO:teuthology.orchestra.run.smithi016.stdout: "default_registry": { 2024-03-28T22:20:43.821 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_registry", 2024-03-28T22:20:43.822 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.822 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.822 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.822 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io", 2024-03-28T22:20:43.822 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.822 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.822 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.823 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-28T22:20:43.823 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.823 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.823 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.823 INFO:teuthology.orchestra.run.smithi016.stdout: "device_cache_timeout": { 2024-03-28T22:20:43.823 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_cache_timeout", 2024-03-28T22:20:43.824 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:43.824 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.824 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.824 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1800", 2024-03-28T22:20:43.824 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.824 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.824 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.825 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache device inventory", 2024-03-28T22:20:43.825 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.825 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.825 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.825 INFO:teuthology.orchestra.run.smithi016.stdout: "device_enhanced_scan": { 2024-03-28T22:20:43.825 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_enhanced_scan", 2024-03-28T22:20:43.826 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.826 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.826 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.826 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.826 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.827 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.827 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.827 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-28T22:20:43.827 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.827 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.828 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.828 INFO:teuthology.orchestra.run.smithi016.stdout: "facts_cache_timeout": { 2024-03-28T22:20:43.828 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "facts_cache_timeout", 2024-03-28T22:20:43.829 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:43.829 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.829 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.829 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-03-28T22:20:43.829 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.830 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.830 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.830 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache host facts data", 2024-03-28T22:20:43.830 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.830 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.830 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.831 INFO:teuthology.orchestra.run.smithi016.stdout: "host_check_interval": { 2024-03-28T22:20:43.831 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "host_check_interval", 2024-03-28T22:20:43.831 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:43.831 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.831 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.831 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-03-28T22:20:43.831 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.831 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.832 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.832 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to perform a host check", 2024-03-28T22:20:43.832 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.832 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.832 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.832 INFO:teuthology.orchestra.run.smithi016.stdout: "hw_monitoring": { 2024-03-28T22:20:43.832 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hw_monitoring", 2024-03-28T22:20:43.833 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.833 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.833 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.833 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.833 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.833 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.833 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.833 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-28T22:20:43.834 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.834 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.834 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.834 INFO:teuthology.orchestra.run.smithi016.stdout: "inventory_list_all": { 2024-03-28T22:20:43.834 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "inventory_list_all", 2024-03-28T22:20:43.834 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.835 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.835 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.835 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.835 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.835 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.835 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.835 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-28T22:20:43.835 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.836 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.836 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.836 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:43.836 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:43.836 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.836 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.837 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.837 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.837 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.837 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.837 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.837 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.837 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.837 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.838 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.838 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.838 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.838 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.838 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.838 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.838 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.838 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.839 INFO:teuthology.orchestra.run.smithi016.stdout: "log_refresh_metadata": { 2024-03-28T22:20:43.839 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_refresh_metadata", 2024-03-28T22:20:43.839 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.839 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.839 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.839 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.839 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.840 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.840 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.840 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-28T22:20:43.840 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.840 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.840 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.840 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:43.841 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:43.841 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.841 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.841 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.841 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.841 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.841 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.841 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.842 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-28T22:20:43.842 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.842 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.842 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.842 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:43.842 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:43.843 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.843 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.843 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.843 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:43.843 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.843 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.843 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.843 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.844 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.844 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.844 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.844 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.844 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.844 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.844 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.844 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.845 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.845 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.845 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:43.845 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:43.845 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.845 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.845 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.846 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.846 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.846 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.846 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.846 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.846 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.846 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.846 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.847 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-28T22:20:43.847 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-28T22:20:43.847 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.847 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.847 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.847 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.847 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.848 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.848 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.848 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-28T22:20:43.848 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.848 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.848 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.848 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-28T22:20:43.849 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-28T22:20:43.849 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.849 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.849 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.849 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "*", 2024-03-28T22:20:43.849 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.849 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.849 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.850 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-28T22:20:43.850 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.850 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.850 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.850 INFO:teuthology.orchestra.run.smithi016.stdout: "max_count_per_host": { 2024-03-28T22:20:43.850 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_count_per_host", 2024-03-28T22:20:43.850 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.851 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.851 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.851 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-03-28T22:20:43.851 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.851 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.851 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.851 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of daemons per service per host", 2024-03-28T22:20:43.851 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.852 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.852 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.852 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd_draining_count": { 2024-03-28T22:20:43.852 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_osd_draining_count", 2024-03-28T22:20:43.852 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.852 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.852 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.853 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-03-28T22:20:43.853 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.853 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.853 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.853 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-28T22:20:43.853 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.853 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.853 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.854 INFO:teuthology.orchestra.run.smithi016.stdout: "migration_current": { 2024-03-28T22:20:43.854 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "migration_current", 2024-03-28T22:20:43.854 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.854 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.854 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.854 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.854 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.855 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.855 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.855 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "internal - do not modify", 2024-03-28T22:20:43.855 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.855 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.855 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.855 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-03-28T22:20:43.856 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-03-28T22:20:43.856 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.856 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.856 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.856 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "root", 2024-03-28T22:20:43.856 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.856 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.856 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.857 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm-package", 2024-03-28T22:20:43.857 INFO:teuthology.orchestra.run.smithi016.stdout: "root" 2024-03-28T22:20:43.857 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.857 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "mode for remote execution of cephadm", 2024-03-28T22:20:43.857 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.857 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.857 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.858 INFO:teuthology.orchestra.run.smithi016.stdout: "oob_default_addr": { 2024-03-28T22:20:43.858 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "oob_default_addr", 2024-03-28T22:20:43.858 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.858 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.858 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.858 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "169.254.1.1", 2024-03-28T22:20:43.858 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.859 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.859 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.859 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-28T22:20:43.859 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.859 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.859 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.859 INFO:teuthology.orchestra.run.smithi016.stdout: "prometheus_alerts_path": { 2024-03-28T22:20:43.860 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus_alerts_path", 2024-03-28T22:20:43.860 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.860 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.860 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.860 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-28T22:20:43.860 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.860 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.860 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.861 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-28T22:20:43.861 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.861 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.861 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.861 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_insecure": { 2024-03-28T22:20:43.861 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_insecure", 2024-03-28T22:20:43.862 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.862 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.862 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.862 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.862 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.862 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.862 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.862 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-28T22:20:43.863 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.863 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.863 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.863 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.863 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_password": { 2024-03-28T22:20:43.863 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_password", 2024-03-28T22:20:43.863 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.863 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.863 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.864 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.864 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.864 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.864 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.864 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-28T22:20:43.864 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.864 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.864 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.865 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_url": { 2024-03-28T22:20:43.865 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_url", 2024-03-28T22:20:43.865 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.865 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.865 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.865 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.865 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.866 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.866 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.866 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-28T22:20:43.866 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.866 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.866 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.866 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.867 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_username": { 2024-03-28T22:20:43.867 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_username", 2024-03-28T22:20:43.867 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.867 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.867 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.867 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.867 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.867 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.867 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.868 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-28T22:20:43.868 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.868 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.868 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.868 INFO:teuthology.orchestra.run.smithi016.stdout: "secure_monitoring_stack": { 2024-03-28T22:20:43.868 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "secure_monitoring_stack", 2024-03-28T22:20:43.868 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.869 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.869 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.869 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.869 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.869 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.869 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.869 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-28T22:20:43.869 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.870 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.870 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.870 INFO:teuthology.orchestra.run.smithi016.stdout: "service_discovery_port": { 2024-03-28T22:20:43.870 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "service_discovery_port", 2024-03-28T22:20:43.870 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.870 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.870 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.871 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8765", 2024-03-28T22:20:43.871 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.871 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.871 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.871 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "cephadm service discovery port", 2024-03-28T22:20:43.871 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.871 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.871 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.872 INFO:teuthology.orchestra.run.smithi016.stdout: "ssh_config_file": { 2024-03-28T22:20:43.872 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssh_config_file", 2024-03-28T22:20:43.872 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.872 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.872 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.872 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.873 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.873 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.873 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.873 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-28T22:20:43.873 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.873 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.873 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.874 INFO:teuthology.orchestra.run.smithi016.stdout: "use_agent": { 2024-03-28T22:20:43.874 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_agent", 2024-03-28T22:20:43.874 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.874 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.874 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.874 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.874 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.874 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.875 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.875 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-28T22:20:43.875 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.875 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.875 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.875 INFO:teuthology.orchestra.run.smithi016.stdout: "use_repo_digest": { 2024-03-28T22:20:43.875 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_repo_digest", 2024-03-28T22:20:43.876 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.876 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.876 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.876 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.876 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.876 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.876 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.876 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-28T22:20:43.877 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.877 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.877 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.877 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.877 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_failed_host_check": { 2024-03-28T22:20:43.877 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_failed_host_check", 2024-03-28T22:20:43.877 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.878 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.878 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.878 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.878 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.878 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.878 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.878 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if the host check fails", 2024-03-28T22:20:43.878 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.879 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.879 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.879 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.879 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_daemons": { 2024-03-28T22:20:43.879 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_daemons", 2024-03-28T22:20:43.879 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.879 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.879 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.880 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.880 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.880 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.880 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.880 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-28T22:20:43.880 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.880 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.880 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.881 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_hosts": { 2024-03-28T22:20:43.881 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_hosts", 2024-03-28T22:20:43.881 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.881 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.881 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.881 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.882 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.882 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.882 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.882 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-28T22:20:43.882 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.882 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.882 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.882 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.883 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.883 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.883 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crash", 2024-03-28T22:20:43.883 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:43.883 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:43.883 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:43.883 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:43.884 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:43.884 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.884 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.884 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.884 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.884 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.884 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.885 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.885 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.885 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.885 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.885 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.885 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.885 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.885 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.885 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.886 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.886 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.886 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.886 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:43.886 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:43.886 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.886 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.887 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.887 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.887 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.887 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.887 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.887 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.887 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.887 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.888 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.888 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:43.888 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:43.888 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.888 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.888 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.888 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:43.889 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.889 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.889 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.889 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.889 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.889 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.889 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.889 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.890 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.890 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.890 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.890 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.890 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.890 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.890 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:43.891 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:43.891 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.891 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.891 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.891 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.891 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.891 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.891 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.892 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.892 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.892 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.892 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.892 INFO:teuthology.orchestra.run.smithi016.stdout: "retain_interval": { 2024-03-28T22:20:43.892 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retain_interval", 2024-03-28T22:20:43.893 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:43.893 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.893 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.893 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "31536000", 2024-03-28T22:20:43.893 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.893 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.893 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.894 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-28T22:20:43.894 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.894 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.894 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.894 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.894 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_recent_interval": { 2024-03-28T22:20:43.894 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_recent_interval", 2024-03-28T22:20:43.894 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:43.895 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.895 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.895 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1209600", 2024-03-28T22:20:43.895 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.895 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.895 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.895 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-28T22:20:43.895 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.895 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.896 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.896 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.896 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:43.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.896 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:43.896 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dashboard", 2024-03-28T22:20:43.896 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:43.896 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:43.897 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:43.897 INFO:teuthology.orchestra.run.smithi016.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-28T22:20:43.897 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-28T22:20:43.897 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.897 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.897 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.897 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-03-28T22:20:43.897 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.898 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.898 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.898 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.898 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.898 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.898 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.898 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_HOST": { 2024-03-28T22:20:43.899 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-28T22:20:43.899 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.899 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.899 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.899 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.899 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.899 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.899 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.900 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.900 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.900 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.900 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.900 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-28T22:20:43.900 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-28T22:20:43.901 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.901 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.901 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.901 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.901 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.901 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.901 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.901 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.902 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.902 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.902 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.902 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_ENABLED": { 2024-03-28T22:20:43.902 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_ENABLED", 2024-03-28T22:20:43.902 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.902 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.903 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.903 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.903 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.903 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.903 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.903 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.903 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.904 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.904 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.904 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-28T22:20:43.904 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-28T22:20:43.904 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.904 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.904 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.905 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.905 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.905 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.905 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.905 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.905 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.905 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.905 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.906 INFO:teuthology.orchestra.run.smithi016.stdout: "ENABLE_BROWSABLE_API": { 2024-03-28T22:20:43.906 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-28T22:20:43.906 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.906 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.906 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.906 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.907 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.907 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.907 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.907 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.907 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.907 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.907 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.908 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-28T22:20:43.908 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-28T22:20:43.908 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.908 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.908 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.908 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.908 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.909 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.909 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.909 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.909 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.909 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.909 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.909 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-28T22:20:43.909 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-28T22:20:43.910 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.910 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.910 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.910 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.910 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.910 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.910 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.910 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.911 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.911 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.911 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.911 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-28T22:20:43.911 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-28T22:20:43.911 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.912 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.912 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.912 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.912 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.912 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.912 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.912 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.912 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.912 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.913 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.913 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-28T22:20:43.913 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-28T22:20:43.913 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.913 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.913 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.913 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.914 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.914 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.914 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.914 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.914 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.914 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.914 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.915 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-28T22:20:43.915 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-28T22:20:43.915 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.915 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.915 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.915 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.915 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.915 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.916 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.916 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.916 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.916 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.916 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.916 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-28T22:20:43.916 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-28T22:20:43.917 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.917 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.917 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.917 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.917 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.917 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.917 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.918 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.918 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.918 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.918 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.918 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-28T22:20:43.918 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-28T22:20:43.918 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.919 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.919 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.919 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.919 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.919 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.919 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.919 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.920 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.920 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.920 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.920 INFO:teuthology.orchestra.run.smithi016.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-28T22:20:43.920 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-28T22:20:43.920 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.920 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.921 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.921 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.921 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.921 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.921 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.921 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.921 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.921 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.922 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.922 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_PASSWORD": { 2024-03-28T22:20:43.922 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-28T22:20:43.922 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.922 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.922 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.922 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-03-28T22:20:43.923 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.923 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.923 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.923 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.923 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.923 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.923 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.924 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-28T22:20:43.924 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-28T22:20:43.924 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.924 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.924 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.924 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.924 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.925 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.925 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.925 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.925 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.925 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.925 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.926 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_URL": { 2024-03-28T22:20:43.926 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_URL", 2024-03-28T22:20:43.926 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.926 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.926 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.926 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.926 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.926 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.927 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.927 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.927 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.927 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.927 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.927 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_USERNAME": { 2024-03-28T22:20:43.928 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-28T22:20:43.928 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.928 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.928 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.928 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-03-28T22:20:43.928 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.929 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.929 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.929 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.929 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.929 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.930 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.930 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-28T22:20:43.930 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-28T22:20:43.931 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.931 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.931 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.931 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.931 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.932 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.932 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.932 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.932 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.932 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.932 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.933 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-28T22:20:43.933 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-28T22:20:43.933 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.933 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.933 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.933 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.933 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.933 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.934 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.934 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.934 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.934 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.934 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.934 INFO:teuthology.orchestra.run.smithi016.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-28T22:20:43.934 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-28T22:20:43.935 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.935 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.935 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.935 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.935 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.935 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.935 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.935 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.936 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.936 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.936 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.936 INFO:teuthology.orchestra.run.smithi016.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-28T22:20:43.936 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-28T22:20:43.936 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.936 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.937 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.937 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.937 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.937 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.937 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.937 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.937 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.938 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.938 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.938 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_HOST": { 2024-03-28T22:20:43.938 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-28T22:20:43.938 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.938 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.938 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.939 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.939 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.939 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.939 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.939 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.939 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.939 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.940 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.940 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-28T22:20:43.940 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-28T22:20:43.940 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.940 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.940 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.940 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.941 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.941 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.941 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.941 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.941 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.941 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.941 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.942 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-28T22:20:43.942 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-28T22:20:43.942 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.942 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.942 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.942 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.942 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.943 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.943 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.943 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.943 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.943 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.943 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.944 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-28T22:20:43.944 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-28T22:20:43.944 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.944 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.944 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.944 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.944 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.944 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.945 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.945 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.945 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.945 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.945 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.945 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-28T22:20:43.946 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-28T22:20:43.946 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.946 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.946 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.946 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.946 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.946 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.947 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.947 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.947 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.947 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.947 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.947 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-28T22:20:43.947 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-28T22:20:43.948 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.948 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.948 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.948 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.948 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.948 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.948 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.948 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.949 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.949 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.949 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.949 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-28T22:20:43.949 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-28T22:20:43.949 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.949 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.950 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.950 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.950 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.950 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.950 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.950 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.950 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.950 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.951 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.951 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-28T22:20:43.951 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-28T22:20:43.951 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.951 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.951 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.951 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.952 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.952 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.952 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.952 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.952 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.952 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.952 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.953 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-28T22:20:43.953 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-28T22:20:43.953 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.953 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.953 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.953 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.953 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.954 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.954 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.954 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.954 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.954 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.954 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.954 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_ENABLED": { 2024-03-28T22:20:43.955 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-28T22:20:43.955 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.955 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.955 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.955 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.955 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.955 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.956 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.956 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.956 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.956 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.956 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.956 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-28T22:20:43.956 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-28T22:20:43.957 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.957 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.957 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.957 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-28T22:20:43.957 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.957 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.957 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.958 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.958 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.958 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.958 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.958 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-28T22:20:43.958 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-28T22:20:43.958 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.959 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.959 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.959 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-03-28T22:20:43.959 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.959 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.959 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.959 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.960 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.960 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.960 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.960 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-28T22:20:43.960 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-28T22:20:43.960 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.960 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.960 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.961 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8", 2024-03-28T22:20:43.961 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.961 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.961 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.961 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.961 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.961 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.961 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.962 INFO:teuthology.orchestra.run.smithi016.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-28T22:20:43.962 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-28T22:20:43.962 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.962 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.962 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.962 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "45", 2024-03-28T22:20:43.963 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.963 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.963 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.963 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.963 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.963 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.963 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.964 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ACCESS_KEY": { 2024-03-28T22:20:43.964 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-28T22:20:43.964 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.964 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.964 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.964 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.964 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.964 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.965 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.965 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.965 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.965 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.965 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.965 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-28T22:20:43.965 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-28T22:20:43.966 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.966 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.966 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.966 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-03-28T22:20:43.966 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.966 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.966 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.967 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.967 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.967 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.967 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.967 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SECRET_KEY": { 2024-03-28T22:20:43.967 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-28T22:20:43.967 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.968 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.968 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.968 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.968 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.968 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.968 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.969 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.969 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.969 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.969 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.969 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SSL_VERIFY": { 2024-03-28T22:20:43.969 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-28T22:20:43.969 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.970 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.970 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.970 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:43.970 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.970 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.970 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.970 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.971 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.971 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.971 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.971 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-28T22:20:43.971 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-28T22:20:43.971 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.971 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.972 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.972 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-03-28T22:20:43.972 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.972 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.972 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.972 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.972 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.972 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.973 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.973 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-28T22:20:43.973 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-28T22:20:43.973 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.973 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.973 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.973 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-03-28T22:20:43.974 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.974 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.974 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.974 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.974 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.974 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.974 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.975 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-28T22:20:43.975 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-28T22:20:43.975 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.975 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.975 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.975 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-03-28T22:20:43.975 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.976 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.976 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.976 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.976 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.976 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.976 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.977 INFO:teuthology.orchestra.run.smithi016.stdout: "cross_origin_url": { 2024-03-28T22:20:43.977 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cross_origin_url", 2024-03-28T22:20:43.977 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.977 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.977 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.977 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.977 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.978 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.978 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.978 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.978 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.978 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.978 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.979 INFO:teuthology.orchestra.run.smithi016.stdout: "crt_file": { 2024-03-28T22:20:43.979 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crt_file", 2024-03-28T22:20:43.979 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.979 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.979 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.979 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.979 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.980 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.980 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.980 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.980 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.980 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.980 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.980 INFO:teuthology.orchestra.run.smithi016.stdout: "debug": { 2024-03-28T22:20:43.981 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "debug", 2024-03-28T22:20:43.981 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.981 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.981 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.981 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.981 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.981 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.981 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.982 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable/disable debug options", 2024-03-28T22:20:43.982 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.982 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.982 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.982 INFO:teuthology.orchestra.run.smithi016.stdout: "jwt_token_ttl": { 2024-03-28T22:20:43.982 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "jwt_token_ttl", 2024-03-28T22:20:43.982 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:43.982 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.983 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.983 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "28800", 2024-03-28T22:20:43.983 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.983 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.983 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.983 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.983 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.983 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.984 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.984 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-03-28T22:20:43.984 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-03-28T22:20:43.984 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.984 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.984 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.984 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.985 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.985 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.985 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.985 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.985 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.985 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.985 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.986 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:43.986 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:43.986 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.986 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.986 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.986 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.986 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.986 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.987 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.987 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.987 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.987 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.987 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.987 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.987 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.989 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.989 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.989 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.989 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.990 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.990 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:43.990 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:43.990 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.990 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.990 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.991 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.991 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.991 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.991 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.991 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.991 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.991 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.992 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.992 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:43.992 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:43.992 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.992 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.992 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.993 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:43.993 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.993 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.993 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:43.993 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:43.993 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:43.993 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:43.993 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:43.994 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:43.994 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:43.994 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:43.994 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.994 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.994 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.994 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.995 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:43.995 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:43.995 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.995 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.995 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:43.995 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.995 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.996 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.996 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.996 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:43.996 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.996 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.996 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.996 INFO:teuthology.orchestra.run.smithi016.stdout: "motd": { 2024-03-28T22:20:43.997 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "motd", 2024-03-28T22:20:43.997 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:43.997 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.997 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.997 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:43.997 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.997 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.997 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.998 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "The message of the day", 2024-03-28T22:20:43.998 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:43.998 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:43.998 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:43.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:43.998 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect_resolve_ip_addr": { 2024-03-28T22:20:43.998 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "redirect_resolve_ip_addr", 2024-03-28T22:20:43.999 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:43.999 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:43.999 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:43.999 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:43.999 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:43.999 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:43.999 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:43.999 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.000 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.000 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.000 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.000 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-03-28T22:20:44.000 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-03-28T22:20:44.000 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.000 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.000 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.001 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-03-28T22:20:44.001 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.001 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.001 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.001 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.001 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.001 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.001 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.002 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-03-28T22:20:44.002 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-03-28T22:20:44.002 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.002 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.002 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.002 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8080", 2024-03-28T22:20:44.003 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.003 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.003 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.003 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.003 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.003 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.003 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.004 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-03-28T22:20:44.004 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-03-28T22:20:44.004 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.004 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.004 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.004 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.004 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.004 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.005 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.005 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.005 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.005 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.005 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.005 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl_server_port": { 2024-03-28T22:20:44.005 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl_server_port", 2024-03-28T22:20:44.006 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.006 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.006 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.006 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8443", 2024-03-28T22:20:44.006 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.006 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.006 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.006 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.007 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.007 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.007 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.007 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-03-28T22:20:44.007 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-03-28T22:20:44.007 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.007 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.008 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.008 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "redirect", 2024-03-28T22:20:44.008 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.008 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.008 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.008 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.008 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect" 2024-03-28T22:20:44.008 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.009 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.009 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.009 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.009 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.009 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-03-28T22:20:44.009 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-03-28T22:20:44.009 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.010 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.010 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.010 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-03-28T22:20:44.010 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-03-28T22:20:44.010 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-03-28T22:20:44.010 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.011 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.011 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.011 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.011 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.011 INFO:teuthology.orchestra.run.smithi016.stdout: "url_prefix": { 2024-03-28T22:20:44.011 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url_prefix", 2024-03-28T22:20:44.012 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.012 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.012 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.012 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.012 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.012 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.012 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.012 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.013 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.013 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.013 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.013 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.013 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.014 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "devicehealth", 2024-03-28T22:20:44.014 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.014 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.014 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.014 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_monitoring": { 2024-03-28T22:20:44.014 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_monitoring", 2024-03-28T22:20:44.014 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.014 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.015 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.015 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.015 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.015 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.015 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.015 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "monitor device health metrics", 2024-03-28T22:20:44.015 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.015 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.016 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.016 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.016 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.016 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.016 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.016 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.016 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.016 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.017 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.017 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.017 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.017 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.017 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.017 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.017 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.018 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.018 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.018 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.018 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.018 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.018 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.019 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.019 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.019 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.019 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.019 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.019 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.020 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.020 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.020 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.020 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.020 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.020 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.020 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.020 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.021 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.021 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.021 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.021 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.021 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.021 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.021 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.022 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.022 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.022 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.022 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.022 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.022 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.022 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.023 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.023 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.023 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.023 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.023 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.023 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.023 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.024 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.024 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.024 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.024 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.024 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.024 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.024 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.025 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.025 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.025 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.025 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.025 INFO:teuthology.orchestra.run.smithi016.stdout: "mark_out_threshold": { 2024-03-28T22:20:44.025 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mark_out_threshold", 2024-03-28T22:20:44.025 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:44.026 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.026 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.026 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2419200", 2024-03-28T22:20:44.026 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.026 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.026 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.027 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-28T22:20:44.027 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.027 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.027 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.027 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": { 2024-03-28T22:20:44.028 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_name", 2024-03-28T22:20:44.028 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.028 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.028 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.028 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "device_health_metrics", 2024-03-28T22:20:44.029 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.029 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.029 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.029 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-28T22:20:44.029 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.030 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.030 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.030 INFO:teuthology.orchestra.run.smithi016.stdout: "retention_period": { 2024-03-28T22:20:44.030 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retention_period", 2024-03-28T22:20:44.031 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:44.031 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.031 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.031 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15552000", 2024-03-28T22:20:44.031 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.032 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.032 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.032 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain device health metrics", 2024-03-28T22:20:44.032 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.033 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.033 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.033 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_frequency": { 2024-03-28T22:20:44.033 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_frequency", 2024-03-28T22:20:44.033 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:44.033 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.034 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.034 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-03-28T22:20:44.034 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.034 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.034 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.034 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-28T22:20:44.034 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.034 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.035 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.035 INFO:teuthology.orchestra.run.smithi016.stdout: "self_heal": { 2024-03-28T22:20:44.035 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "self_heal", 2024-03-28T22:20:44.035 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.035 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.035 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.035 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.035 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.036 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.036 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.036 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-28T22:20:44.036 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.036 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.036 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.036 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-03-28T22:20:44.036 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-03-28T22:20:44.037 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:44.037 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.037 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.037 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-03-28T22:20:44.037 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.037 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.037 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.037 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and check device health", 2024-03-28T22:20:44.038 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.038 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.038 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.038 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_threshold": { 2024-03-28T22:20:44.038 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_threshold", 2024-03-28T22:20:44.038 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:44.039 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.039 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.039 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7257600", 2024-03-28T22:20:44.039 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.039 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.039 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.039 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-28T22:20:44.039 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.040 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.040 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.040 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.040 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.040 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "diskprediction_local", 2024-03-28T22:20:44.040 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.041 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.041 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.041 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.041 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.041 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.041 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.041 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.042 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.042 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.042 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.042 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.042 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.042 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.042 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.043 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.043 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.043 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.043 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.043 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.043 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.043 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.044 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.044 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.044 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.044 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.044 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.044 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.045 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.045 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.045 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.045 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.045 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.045 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.045 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.046 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.046 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.046 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.046 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.046 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.046 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.046 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.047 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.047 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.047 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.047 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.047 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.047 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.047 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.048 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.048 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.048 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.048 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.048 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.048 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.048 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.048 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.049 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.049 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.049 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.049 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.049 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.049 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.049 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.050 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.050 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.050 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.050 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.050 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.050 INFO:teuthology.orchestra.run.smithi016.stdout: "predict_interval": { 2024-03-28T22:20:44.051 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predict_interval", 2024-03-28T22:20:44.051 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.051 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.051 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.051 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-03-28T22:20:44.051 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.051 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.052 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.052 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.052 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.052 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.052 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.052 INFO:teuthology.orchestra.run.smithi016.stdout: "predictor_model": { 2024-03-28T22:20:44.053 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predictor_model", 2024-03-28T22:20:44.053 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.053 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.053 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.053 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "prophetstor", 2024-03-28T22:20:44.053 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.053 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.053 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.054 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.054 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.054 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.054 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.054 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-03-28T22:20:44.054 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-03-28T22:20:44.055 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.055 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.055 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.055 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-03-28T22:20:44.055 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.055 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.055 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.055 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.055 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.056 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.056 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.056 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.056 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.056 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "influx", 2024-03-28T22:20:44.056 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": false, 2024-03-28T22:20:44.057 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "influxdb python module not found", 2024-03-28T22:20:44.057 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.057 INFO:teuthology.orchestra.run.smithi016.stdout: "batch_size": { 2024-03-28T22:20:44.057 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "batch_size", 2024-03-28T22:20:44.057 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.057 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.057 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.057 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5000", 2024-03-28T22:20:44.058 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.058 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.058 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.058 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-28T22:20:44.058 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.058 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.058 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.059 INFO:teuthology.orchestra.run.smithi016.stdout: "database": { 2024-03-28T22:20:44.059 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "database", 2024-03-28T22:20:44.059 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.059 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.059 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.059 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "ceph", 2024-03-28T22:20:44.059 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.059 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.060 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.060 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-28T22:20:44.060 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.060 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.060 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.060 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": { 2024-03-28T22:20:44.060 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hostname", 2024-03-28T22:20:44.061 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.061 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.061 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.061 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.061 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.061 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.061 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.061 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server hostname", 2024-03-28T22:20:44.062 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.062 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.062 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.062 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-03-28T22:20:44.062 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-03-28T22:20:44.062 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:44.063 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.063 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.063 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "30", 2024-03-28T22:20:44.063 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "5", 2024-03-28T22:20:44.063 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.063 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.063 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-28T22:20:44.063 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.064 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.064 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.064 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.064 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.064 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.064 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.064 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.065 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.065 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.065 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.065 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.065 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.065 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.065 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.066 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.066 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.066 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.066 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.066 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.066 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.066 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.067 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.067 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.067 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.067 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.067 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.067 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.067 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.068 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.068 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.068 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.068 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.068 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.068 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.068 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.069 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.069 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.069 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.069 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.069 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.069 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.069 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.070 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.070 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.070 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.070 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.070 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.070 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.070 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.071 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.071 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.071 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.071 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.071 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.071 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.071 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.072 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.072 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.072 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.072 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.072 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.072 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.072 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.072 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.072 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.073 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.073 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.073 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.073 INFO:teuthology.orchestra.run.smithi016.stdout: "password": { 2024-03-28T22:20:44.073 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "password", 2024-03-28T22:20:44.073 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.074 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.074 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.074 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.074 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.074 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.074 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.074 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "password of InfluxDB server user", 2024-03-28T22:20:44.074 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.075 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.075 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.075 INFO:teuthology.orchestra.run.smithi016.stdout: "port": { 2024-03-28T22:20:44.075 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "port", 2024-03-28T22:20:44.075 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.075 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.075 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.075 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8086", 2024-03-28T22:20:44.076 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.076 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.076 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.076 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server port", 2024-03-28T22:20:44.076 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.076 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.076 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.076 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-03-28T22:20:44.077 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-03-28T22:20:44.077 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.077 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.077 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.077 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "false", 2024-03-28T22:20:44.077 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.077 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.077 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.077 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T22:20:44.078 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.078 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.078 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.078 INFO:teuthology.orchestra.run.smithi016.stdout: "threads": { 2024-03-28T22:20:44.078 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threads", 2024-03-28T22:20:44.078 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.078 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.079 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.079 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-03-28T22:20:44.079 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-03-28T22:20:44.079 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "32", 2024-03-28T22:20:44.079 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.079 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-28T22:20:44.079 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.079 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.080 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.080 INFO:teuthology.orchestra.run.smithi016.stdout: "username": { 2024-03-28T22:20:44.080 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "username", 2024-03-28T22:20:44.080 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.080 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.080 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.080 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.080 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.081 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.081 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.081 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "username of InfluxDB server user", 2024-03-28T22:20:44.081 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.081 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.081 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.081 INFO:teuthology.orchestra.run.smithi016.stdout: "verify_ssl": { 2024-03-28T22:20:44.082 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "verify_ssl", 2024-03-28T22:20:44.082 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.082 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.082 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.082 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "true", 2024-03-28T22:20:44.082 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.082 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.083 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.083 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T22:20:44.083 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.083 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.083 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.083 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.083 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.084 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "insights", 2024-03-28T22:20:44.084 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.084 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.084 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.084 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.084 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.084 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.085 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.085 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.085 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.085 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.085 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.085 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.085 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.085 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.086 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.086 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.086 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.086 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.086 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.086 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.086 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.086 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.087 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.087 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.087 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.087 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.087 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.087 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.087 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.088 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.088 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.088 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.088 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.088 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.088 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.088 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.088 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.089 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.089 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.089 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.089 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.089 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.089 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.089 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.089 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.090 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.090 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.090 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.090 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.090 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.090 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.090 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.090 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.091 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.091 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.091 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.091 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.091 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.091 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.091 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.092 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.092 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.092 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.092 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.092 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.092 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.092 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.092 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.093 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.093 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.093 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.093 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "iostat", 2024-03-28T22:20:44.093 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.093 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.094 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.094 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.094 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.094 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.094 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.094 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.094 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.094 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.095 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.095 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.095 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.095 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.095 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.095 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.095 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.095 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.096 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.096 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.096 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.096 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.096 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.096 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.097 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.097 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.097 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.097 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.097 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.097 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.097 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.097 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.098 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.098 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.098 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.098 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.098 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.098 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.098 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.099 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.099 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.099 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.099 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.099 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.099 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.099 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.099 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.100 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.100 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.100 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.100 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.100 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.100 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.100 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.100 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.101 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.101 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.101 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.101 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.101 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.101 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.102 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.102 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.102 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.102 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.102 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.102 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.103 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.103 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.103 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.103 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.103 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "k8sevents", 2024-03-28T22:20:44.103 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.103 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.104 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.104 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_event_retention_days": { 2024-03-28T22:20:44.104 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ceph_event_retention_days", 2024-03-28T22:20:44.104 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.104 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.104 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.104 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7", 2024-03-28T22:20:44.104 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.104 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.105 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.105 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-28T22:20:44.105 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.105 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.105 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.105 INFO:teuthology.orchestra.run.smithi016.stdout: "config_check_secs": { 2024-03-28T22:20:44.105 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_check_secs", 2024-03-28T22:20:44.105 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.106 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.106 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.106 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-03-28T22:20:44.106 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "10", 2024-03-28T22:20:44.106 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.106 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.106 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-28T22:20:44.106 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.107 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.107 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.107 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.107 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.107 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.107 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.107 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.107 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.108 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.108 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.108 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.108 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.108 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.108 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.108 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.108 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.108 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.109 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.109 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.109 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.109 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.109 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.109 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.110 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.110 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.110 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.110 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.110 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.110 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.110 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.110 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.111 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.111 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.111 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.111 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.111 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.111 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.112 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.112 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.112 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.112 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.112 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.112 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.112 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.112 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.112 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.113 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.113 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.113 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.113 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.113 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.113 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.113 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.113 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.114 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.114 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.114 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.114 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.114 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.114 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.114 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.115 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.115 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.115 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.115 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.115 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.115 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.115 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.115 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.115 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.116 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.116 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "localpool", 2024-03-28T22:20:44.116 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.116 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.116 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.116 INFO:teuthology.orchestra.run.smithi016.stdout: "failure_domain": { 2024-03-28T22:20:44.116 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "failure_domain", 2024-03-28T22:20:44.117 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.117 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.117 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.117 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "host", 2024-03-28T22:20:44.117 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.117 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.117 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.117 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "failure domain for any created local pool", 2024-03-28T22:20:44.118 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-28T22:20:44.118 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.118 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.118 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.118 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.118 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.119 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.119 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.119 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.119 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.119 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.119 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.119 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.119 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.119 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.120 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.120 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.120 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.120 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.120 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.120 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.120 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.120 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.121 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.121 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.121 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.121 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.121 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.121 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.121 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.121 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.121 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.122 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.122 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.122 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.122 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.122 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.122 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.122 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.123 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.123 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.123 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.123 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.123 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.123 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.123 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.123 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.123 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.124 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.124 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.124 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.124 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.124 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.124 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.124 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.125 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.125 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.125 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.125 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.125 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.125 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.126 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.126 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.126 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.126 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.126 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.126 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.127 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.127 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.127 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": { 2024-03-28T22:20:44.127 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_size", 2024-03-28T22:20:44.127 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.127 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.127 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.127 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.128 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.128 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.128 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.128 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default min_size for any created local pool", 2024-03-28T22:20:44.128 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-28T22:20:44.128 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.128 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.129 INFO:teuthology.orchestra.run.smithi016.stdout: "num_rep": { 2024-03-28T22:20:44.129 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "num_rep", 2024-03-28T22:20:44.129 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.129 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.129 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.129 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3", 2024-03-28T22:20:44.129 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.129 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.129 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.130 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default replica count for any created local pool", 2024-03-28T22:20:44.130 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.130 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.130 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.130 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": { 2024-03-28T22:20:44.130 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_num", 2024-03-28T22:20:44.130 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.131 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.131 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.131 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "128", 2024-03-28T22:20:44.131 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.131 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.131 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.131 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default pg_num for any created local pool", 2024-03-28T22:20:44.132 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.132 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.132 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.132 INFO:teuthology.orchestra.run.smithi016.stdout: "prefix": { 2024-03-28T22:20:44.132 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prefix", 2024-03-28T22:20:44.132 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.132 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.133 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.133 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.133 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.133 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.133 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.133 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name prefix for any created local pool", 2024-03-28T22:20:44.133 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.133 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.133 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.134 INFO:teuthology.orchestra.run.smithi016.stdout: "subtree": { 2024-03-28T22:20:44.134 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "subtree", 2024-03-28T22:20:44.134 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.134 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.134 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.134 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "rack", 2024-03-28T22:20:44.135 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.135 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.135 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.135 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-28T22:20:44.135 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-28T22:20:44.135 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.135 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.135 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.135 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.136 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mds_autoscaler", 2024-03-28T22:20:44.136 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.136 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.136 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.136 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.136 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.137 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.137 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.137 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.137 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.137 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.137 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.137 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.137 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.138 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.138 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.138 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.138 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.138 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.138 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.138 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.138 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.139 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.139 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.139 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.139 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.139 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.140 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.140 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.140 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.140 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.140 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.140 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.140 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.140 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.141 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.141 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.141 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.141 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.141 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.141 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.141 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.142 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.142 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.142 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.142 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.142 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.142 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.142 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.143 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.143 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.143 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.143 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.143 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.143 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.143 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.143 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.144 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.144 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.144 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.144 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.144 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.144 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.144 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.145 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.145 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.145 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.145 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.145 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.145 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.145 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.145 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.146 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.146 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.146 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirroring", 2024-03-28T22:20:44.146 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.146 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.146 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.146 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.147 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.147 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.147 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.147 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.147 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.147 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.147 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.148 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.148 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.148 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.148 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.148 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.148 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.148 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.148 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.149 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.149 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.149 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.149 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.149 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.149 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.149 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.150 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.150 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.150 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.150 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.150 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.150 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.151 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.151 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.151 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.151 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.151 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.151 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.152 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.152 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.152 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.152 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.152 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.152 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.152 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.152 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.153 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.153 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.153 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.153 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.153 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.153 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.153 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.154 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.154 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.154 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.154 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.154 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.154 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.155 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.155 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.155 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.155 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.155 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.156 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.156 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.156 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.156 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.157 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.157 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.157 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.157 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.158 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "nfs", 2024-03-28T22:20:44.158 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.158 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.158 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.158 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.158 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.159 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.159 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.159 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.159 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.159 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.159 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.159 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.159 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.160 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.160 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.160 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.160 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.160 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.160 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.160 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.160 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.160 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.161 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.161 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.161 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.161 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.161 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.161 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.162 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.162 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.162 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.162 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.162 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.162 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.162 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.163 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.163 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.163 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.163 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.163 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.163 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.163 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.164 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.164 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.164 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.164 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.164 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.164 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.164 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.164 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.165 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.165 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.165 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.165 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.165 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.165 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.165 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.165 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.166 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.166 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.166 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.166 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.166 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.166 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.166 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.167 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.167 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.167 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.167 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.167 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.167 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.168 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-03-28T22:20:44.168 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.168 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.168 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.168 INFO:teuthology.orchestra.run.smithi016.stdout: "fail_fs": { 2024-03-28T22:20:44.168 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "fail_fs", 2024-03-28T22:20:44.168 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.169 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.169 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.169 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.169 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.169 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.169 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.169 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-28T22:20:44.169 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.170 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.170 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.170 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.170 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.170 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.170 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.171 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.171 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.171 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.171 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.171 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.171 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.171 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.171 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.172 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.172 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.172 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.172 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.172 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.172 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.172 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.172 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.173 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.173 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.173 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.173 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.173 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.173 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.173 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.174 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.174 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.174 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.174 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.174 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.174 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.174 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.175 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.175 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.175 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.175 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.175 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.175 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.175 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.175 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.176 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.176 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.176 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.176 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.176 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.176 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.176 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.176 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.177 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.177 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.177 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.177 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.177 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.177 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.177 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.178 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.178 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.178 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.178 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.178 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.178 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.178 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.179 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.179 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.179 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator": { 2024-03-28T22:20:44.179 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-03-28T22:20:44.179 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.179 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.179 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.180 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.180 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.180 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.180 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.180 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm", 2024-03-28T22:20:44.180 INFO:teuthology.orchestra.run.smithi016.stdout: "rook", 2024-03-28T22:20:44.180 INFO:teuthology.orchestra.run.smithi016.stdout: "test_orchestrator" 2024-03-28T22:20:44.180 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.180 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Orchestrator backend", 2024-03-28T22:20:44.181 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.181 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.181 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.181 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.181 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.181 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_perf_query", 2024-03-28T22:20:44.182 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.182 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.182 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.182 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.182 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.182 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.182 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.182 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.183 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.183 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.183 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.183 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.183 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.183 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.183 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.183 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.184 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.184 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.184 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.184 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.184 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.184 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.184 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.185 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.185 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.185 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.185 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.185 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.185 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.185 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.185 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.186 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.186 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.186 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.186 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.186 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.186 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.187 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.187 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.187 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.187 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.187 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.187 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.187 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.187 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.188 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.188 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.188 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.188 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.188 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.188 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.188 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.188 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.189 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.189 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.189 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.189 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.189 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.189 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.189 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.190 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.190 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.190 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.190 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.190 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.190 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.190 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.191 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.191 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.191 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.191 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.191 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_support", 2024-03-28T22:20:44.191 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.193 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.193 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.193 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.194 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.194 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.194 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.194 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.194 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.194 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.194 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.194 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.195 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.195 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.195 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.195 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.195 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.195 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.195 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.195 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.196 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.196 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.196 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.196 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.196 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.196 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.196 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.196 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.197 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.197 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.197 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.197 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.197 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.197 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.197 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.197 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.198 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.198 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.198 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.198 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.198 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.198 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.198 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.199 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.199 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.199 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.199 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.199 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.199 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.199 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.199 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.199 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.200 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.200 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.200 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.200 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.200 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.200 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.200 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.201 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.201 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.201 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.201 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.201 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.201 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.201 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.201 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.202 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.202 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.202 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.202 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.202 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.202 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_autoscaler", 2024-03-28T22:20:44.202 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.203 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.203 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.203 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.203 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.203 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.203 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.203 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.203 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.204 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.204 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.204 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.204 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.204 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.204 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.204 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.204 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.205 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.205 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.205 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.205 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.205 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.205 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.205 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.206 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.206 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.206 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.206 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.206 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.206 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.206 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.206 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.207 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.207 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.207 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.207 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.207 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.207 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.207 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.208 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.208 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.208 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.208 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.208 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.208 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.208 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.208 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.209 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.209 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.209 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.209 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.209 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.209 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.209 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.209 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.210 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.210 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.210 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.210 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.210 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.210 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.210 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.211 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.211 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.211 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.211 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.211 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.211 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.211 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.212 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-03-28T22:20:44.212 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-03-28T22:20:44.212 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:44.212 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.212 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.212 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-03-28T22:20:44.212 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.213 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.213 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.213 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.213 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.213 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.213 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.213 INFO:teuthology.orchestra.run.smithi016.stdout: "threshold": { 2024-03-28T22:20:44.214 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threshold", 2024-03-28T22:20:44.214 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-03-28T22:20:44.214 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.214 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.214 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-03-28T22:20:44.214 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1.0", 2024-03-28T22:20:44.214 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.215 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.215 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "scaling threshold", 2024-03-28T22:20:44.215 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-28T22:20:44.215 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.215 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.215 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.215 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.216 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "progress", 2024-03-28T22:20:44.216 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.216 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.216 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.216 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_pg_recovery_event": { 2024-03-28T22:20:44.216 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_pg_recovery_event", 2024-03-28T22:20:44.216 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.217 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.217 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.217 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.217 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.217 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.217 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.217 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-28T22:20:44.218 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.218 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.218 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.218 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-03-28T22:20:44.218 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-03-28T22:20:44.218 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.218 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.219 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.219 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.219 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.219 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.219 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.219 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.219 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.219 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.220 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.220 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.220 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.220 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.220 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.220 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.220 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.221 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.221 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.221 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.221 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.221 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.221 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.221 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.221 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.222 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.222 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.222 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.222 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.222 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.222 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.222 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.223 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.223 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.223 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.223 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.223 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.223 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.223 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.223 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.224 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.224 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.224 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.224 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.224 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.224 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.225 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.225 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.225 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.225 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.225 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.225 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.225 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.225 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.226 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.226 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.226 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.226 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.226 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.226 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.226 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.227 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.227 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.227 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.227 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.227 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.227 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.228 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.228 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.228 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.228 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.228 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.228 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.228 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.228 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.229 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.229 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.229 INFO:teuthology.orchestra.run.smithi016.stdout: "max_completed_events": { 2024-03-28T22:20:44.229 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_completed_events", 2024-03-28T22:20:44.229 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.229 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.230 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.230 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "50", 2024-03-28T22:20:44.230 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.230 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.230 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.230 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "number of past completed events to remember", 2024-03-28T22:20:44.230 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.231 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.231 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.231 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-03-28T22:20:44.231 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-03-28T22:20:44.231 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:44.231 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.232 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.232 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-03-28T22:20:44.232 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.232 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.232 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.232 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long the module is going to sleep", 2024-03-28T22:20:44.232 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.233 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.233 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.233 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.233 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.233 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus", 2024-03-28T22:20:44.233 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.234 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.234 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.234 INFO:teuthology.orchestra.run.smithi016.stdout: "cache": { 2024-03-28T22:20:44.234 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cache", 2024-03-28T22:20:44.234 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.234 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.234 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.235 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.235 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.235 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.235 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.235 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.235 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.235 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.235 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.236 INFO:teuthology.orchestra.run.smithi016.stdout: "exclude_perf_counters": { 2024-03-28T22:20:44.236 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "exclude_perf_counters", 2024-03-28T22:20:44.236 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.236 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.236 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.236 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.236 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.236 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.237 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.237 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-28T22:20:44.237 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-28T22:20:44.237 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.237 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.237 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.237 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.238 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.238 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.238 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.238 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.238 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.238 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.238 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.239 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.239 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.239 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.239 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.239 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.239 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.239 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.239 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.240 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.240 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.240 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.240 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.240 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.240 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.241 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.241 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.241 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.241 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.241 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.241 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.241 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.241 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.242 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.242 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.242 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.242 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.242 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.243 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.243 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.243 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.243 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.243 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.243 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.243 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.243 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.244 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.244 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.244 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.244 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.244 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.244 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.244 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.244 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.245 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.245 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.245 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.245 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.245 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.245 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.245 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.246 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.246 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.246 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.246 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.246 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.246 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.246 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.247 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools": { 2024-03-28T22:20:44.247 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools", 2024-03-28T22:20:44.247 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.247 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.247 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.247 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.248 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.248 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.248 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.248 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.248 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.248 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.248 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.249 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-28T22:20:44.249 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-28T22:20:44.249 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.249 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.249 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.249 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "300", 2024-03-28T22:20:44.249 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.250 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.250 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.250 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.250 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.250 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.250 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.250 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_interval": { 2024-03-28T22:20:44.251 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_interval", 2024-03-28T22:20:44.251 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-03-28T22:20:44.251 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.251 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.251 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15.0", 2024-03-28T22:20:44.251 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.251 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.252 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.252 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.252 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.252 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.252 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.252 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-03-28T22:20:44.252 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-03-28T22:20:44.253 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.253 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.253 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.253 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-03-28T22:20:44.253 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.253 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.253 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.254 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-28T22:20:44.254 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.254 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.254 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.254 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-03-28T22:20:44.254 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-03-28T22:20:44.255 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.255 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.255 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.255 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "9283", 2024-03-28T22:20:44.255 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.255 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.255 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.255 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-28T22:20:44.256 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.256 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.256 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.256 INFO:teuthology.orchestra.run.smithi016.stdout: "stale_cache_strategy": { 2024-03-28T22:20:44.256 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stale_cache_strategy", 2024-03-28T22:20:44.256 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.257 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.257 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.257 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "log", 2024-03-28T22:20:44.257 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.257 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.257 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.257 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.257 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.258 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.258 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.258 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-03-28T22:20:44.258 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-03-28T22:20:44.258 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.258 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.259 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.259 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "default", 2024-03-28T22:20:44.259 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.259 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.259 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.259 INFO:teuthology.orchestra.run.smithi016.stdout: "default", 2024-03-28T22:20:44.259 INFO:teuthology.orchestra.run.smithi016.stdout: "error" 2024-03-28T22:20:44.260 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.260 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.260 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.260 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.260 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.260 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-03-28T22:20:44.260 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-03-28T22:20:44.261 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.261 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.261 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.261 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-03-28T22:20:44.261 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-03-28T22:20:44.261 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-03-28T22:20:44.261 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.261 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.262 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.262 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.262 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.262 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.262 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.263 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_support", 2024-03-28T22:20:44.263 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.263 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.263 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.263 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.263 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.263 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.264 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.264 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.264 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.264 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.264 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.264 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.264 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.265 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.265 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.265 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.265 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.265 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.265 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.265 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.265 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.266 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.266 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.266 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.266 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.266 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.266 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.267 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.267 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.267 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.267 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.267 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.267 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.267 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.268 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.268 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.268 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.268 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.268 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.268 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.269 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.269 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.269 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.269 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.269 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.269 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.269 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.269 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.270 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.270 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.270 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.270 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.270 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.270 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.270 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.271 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.271 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.271 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.271 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.271 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.271 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.272 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.272 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.272 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.272 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.272 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.272 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.272 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.272 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.273 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_snap_create": { 2024-03-28T22:20:44.273 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_snap_create", 2024-03-28T22:20:44.273 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.273 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.273 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.273 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-03-28T22:20:44.273 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.274 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.274 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.274 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.274 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.274 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.274 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.275 INFO:teuthology.orchestra.run.smithi016.stdout: "mirror_snapshot_schedule": { 2024-03-28T22:20:44.275 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirror_snapshot_schedule", 2024-03-28T22:20:44.275 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.275 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.275 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.275 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.275 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.275 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.276 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.276 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.276 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.276 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.276 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.276 INFO:teuthology.orchestra.run.smithi016.stdout: "trash_purge_schedule": { 2024-03-28T22:20:44.277 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "trash_purge_schedule", 2024-03-28T22:20:44.277 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.277 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.277 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.278 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.278 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.278 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.278 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.279 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.279 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.279 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.279 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.279 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.280 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.280 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "restful", 2024-03-28T22:20:44.280 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.280 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.281 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.281 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_auth": { 2024-03-28T22:20:44.281 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_auth", 2024-03-28T22:20:44.281 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.281 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.281 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.281 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.281 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.282 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.282 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.282 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.282 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.282 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.282 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.282 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-03-28T22:20:44.283 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-03-28T22:20:44.283 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.283 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.283 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.283 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.283 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.283 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.283 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.284 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.284 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.284 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.284 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.284 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.284 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.284 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.285 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.285 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.285 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.285 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.285 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.285 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.285 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.286 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.286 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.286 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.286 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.286 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.286 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.286 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.286 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.287 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.287 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.287 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.287 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.287 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.287 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.288 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.288 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.288 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.288 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.288 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.288 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.288 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.288 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.289 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.289 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.289 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.289 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.289 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.289 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.289 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.290 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.290 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.290 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.290 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.290 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.290 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.290 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.290 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.291 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.291 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.291 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.291 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.291 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.291 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.291 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.292 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.292 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.292 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.292 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.292 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.292 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.293 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.293 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.293 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.293 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.294 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.294 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.294 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-03-28T22:20:44.295 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-03-28T22:20:44.295 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.295 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.295 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.296 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.296 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.296 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.296 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.296 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.296 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.296 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.297 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.297 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-03-28T22:20:44.297 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-03-28T22:20:44.297 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.297 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.297 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.298 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.298 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.298 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.298 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.298 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.298 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.298 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.299 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.299 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.299 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.299 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rgw", 2024-03-28T22:20:44.299 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.299 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.299 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.300 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.300 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.300 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.300 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.300 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.300 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.300 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.300 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.301 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.301 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.301 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.301 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.301 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.301 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.301 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.302 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.302 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.302 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.302 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.302 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.302 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.303 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.303 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.303 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.303 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.303 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.303 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.303 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.304 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.304 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.304 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.304 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.304 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.304 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.304 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.304 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.304 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.305 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.305 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.305 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.305 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.305 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.305 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.305 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.305 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.306 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.306 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.306 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.306 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.306 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.306 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.306 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.307 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.307 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.307 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.307 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.307 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.307 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.307 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.308 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.308 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.308 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.308 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.308 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.309 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.309 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.309 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.309 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.310 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rook", 2024-03-28T22:20:44.310 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.310 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.311 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.311 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.311 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.311 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.311 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.312 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.312 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.312 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.312 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.312 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.313 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.313 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.313 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.313 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.313 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.314 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.314 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.314 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.314 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.314 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.315 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.315 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.315 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.315 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.315 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.316 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.316 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.316 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.316 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.316 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.316 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.316 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.316 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.317 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.317 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.317 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.317 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.317 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.318 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.318 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.318 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.318 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.318 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.318 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.319 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.319 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.319 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.319 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.319 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.319 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.320 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.320 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.320 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.320 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.320 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.320 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.320 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.321 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.321 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.321 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.321 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.321 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.321 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.321 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.321 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.322 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.322 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.322 INFO:teuthology.orchestra.run.smithi016.stdout: "storage_class": { 2024-03-28T22:20:44.322 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "storage_class", 2024-03-28T22:20:44.322 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.322 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.323 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.323 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "local", 2024-03-28T22:20:44.323 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.323 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.323 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.323 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-28T22:20:44.323 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.324 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.324 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.324 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.324 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.324 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "selftest", 2024-03-28T22:20:44.324 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.325 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.325 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.325 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.325 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.325 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.325 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.325 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.325 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.326 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.326 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.326 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.326 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.326 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.326 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.326 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.327 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.327 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.327 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.327 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.327 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.327 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.327 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.327 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.328 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.328 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.328 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.328 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.328 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.328 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.328 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.329 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.329 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.329 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.329 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.329 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.329 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.329 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.329 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.330 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.330 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.330 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.330 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.330 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.330 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.330 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.331 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.331 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.331 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.331 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.331 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.331 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.331 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.331 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.332 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.332 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.332 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.332 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.332 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.333 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.333 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.333 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.333 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.333 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.333 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.333 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.333 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.334 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.334 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.334 INFO:teuthology.orchestra.run.smithi016.stdout: "roption1": { 2024-03-28T22:20:44.334 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption1", 2024-03-28T22:20:44.334 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.334 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.334 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.335 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.335 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.335 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.335 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.335 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.335 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.335 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.336 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.336 INFO:teuthology.orchestra.run.smithi016.stdout: "roption2": { 2024-03-28T22:20:44.336 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption2", 2024-03-28T22:20:44.336 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.336 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.336 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.336 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "xyz", 2024-03-28T22:20:44.337 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.337 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.337 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.337 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.337 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.337 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.337 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.338 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption1": { 2024-03-28T22:20:44.338 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption1", 2024-03-28T22:20:44.338 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.338 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.338 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.338 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.338 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.338 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.339 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.339 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.339 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.339 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.339 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.339 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption2": { 2024-03-28T22:20:44.339 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption2", 2024-03-28T22:20:44.339 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.340 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.340 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.340 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.340 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.340 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.340 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.340 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.340 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.341 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.341 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.341 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption3": { 2024-03-28T22:20:44.341 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption3", 2024-03-28T22:20:44.341 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-03-28T22:20:44.341 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.341 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.342 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.342 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.342 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.342 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.342 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.342 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.342 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.343 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.343 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption4": { 2024-03-28T22:20:44.343 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption4", 2024-03-28T22:20:44.343 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.343 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.343 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.343 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.344 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.344 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.344 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.344 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.344 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.344 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.344 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.345 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption5": { 2024-03-28T22:20:44.345 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption5", 2024-03-28T22:20:44.345 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.345 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.345 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.345 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.345 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.345 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.346 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.346 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.346 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.346 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.346 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.346 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption6": { 2024-03-28T22:20:44.346 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption6", 2024-03-28T22:20:44.347 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.347 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.347 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.347 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.347 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.347 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.347 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.347 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.348 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.348 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.348 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.348 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption7": { 2024-03-28T22:20:44.348 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption7", 2024-03-28T22:20:44.348 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.348 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.349 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.349 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.349 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-03-28T22:20:44.349 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "42", 2024-03-28T22:20:44.349 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.349 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.349 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.350 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.350 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.350 INFO:teuthology.orchestra.run.smithi016.stdout: "testkey": { 2024-03-28T22:20:44.350 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testkey", 2024-03-28T22:20:44.350 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.350 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.351 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.351 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.351 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.351 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.351 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.351 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.351 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.351 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.352 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.352 INFO:teuthology.orchestra.run.smithi016.stdout: "testlkey": { 2024-03-28T22:20:44.352 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testlkey", 2024-03-28T22:20:44.352 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.352 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.352 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.352 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.353 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.353 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.353 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.353 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.353 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.353 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.353 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.354 INFO:teuthology.orchestra.run.smithi016.stdout: "testnewline": { 2024-03-28T22:20:44.354 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testnewline", 2024-03-28T22:20:44.354 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.354 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.354 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.354 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.354 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.355 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.355 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.355 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.355 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.355 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.355 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.355 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.356 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.356 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snap_schedule", 2024-03-28T22:20:44.356 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.356 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.356 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.356 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_m_granularity": { 2024-03-28T22:20:44.357 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_m_granularity", 2024-03-28T22:20:44.357 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.357 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.357 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.357 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.357 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.357 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.357 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.358 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow minute scheduled snapshots", 2024-03-28T22:20:44.358 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.358 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.358 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.358 INFO:teuthology.orchestra.run.smithi016.stdout: "dump_on_update": { 2024-03-28T22:20:44.358 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dump_on_update", 2024-03-28T22:20:44.358 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.359 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.359 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.359 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.359 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.359 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.359 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.359 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "dump database to debug log on update", 2024-03-28T22:20:44.359 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.360 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.360 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.360 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.360 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.360 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.360 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.360 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.361 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.361 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.361 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.361 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.361 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.361 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.361 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.361 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.362 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.362 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.362 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.362 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.362 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.362 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.362 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.363 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.363 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.363 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.363 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.363 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.363 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.363 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.364 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.364 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.364 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.364 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.364 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.364 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.364 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.365 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.365 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.365 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.365 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.365 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.365 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.365 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.365 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.366 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.366 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.366 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.366 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.366 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.366 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.366 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.367 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.367 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.367 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.367 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.367 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.367 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.367 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.368 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.368 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.368 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.368 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.368 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.368 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.368 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.368 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.369 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.369 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.369 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.369 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.369 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stats", 2024-03-28T22:20:44.369 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.369 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.370 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.370 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.370 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.370 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.370 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.370 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.370 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.370 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.371 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.371 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.371 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.371 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.371 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.371 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.371 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.371 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.372 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.372 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.372 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.372 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.372 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.372 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.372 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.373 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.373 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.373 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.373 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.373 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.373 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.374 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.374 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.374 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.374 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.375 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.375 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.375 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.375 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.376 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.376 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.376 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.376 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.377 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.377 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.377 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.377 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.377 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.378 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.378 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.378 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.378 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.378 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.378 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.378 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.379 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.379 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.379 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.379 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.379 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.379 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.379 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.379 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.380 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.380 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.380 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.380 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.380 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.380 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.380 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.380 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.381 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "status", 2024-03-28T22:20:44.381 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.381 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.381 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.381 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.381 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.381 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.382 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.382 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.382 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.382 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.382 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.382 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.382 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.382 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.383 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.383 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.383 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.383 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.383 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.383 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.383 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.383 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.384 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.384 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.384 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.384 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.384 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.384 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.384 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.385 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.385 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.385 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.385 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.385 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.385 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.385 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.385 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.386 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.386 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.386 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.386 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.386 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.386 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.386 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.386 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.387 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.387 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.387 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.387 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.387 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.387 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.387 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.387 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.388 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.388 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.388 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.388 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.388 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.388 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.388 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.388 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.389 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.389 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.389 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.389 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.389 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.389 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.389 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.389 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.390 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.390 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.390 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telegraf", 2024-03-28T22:20:44.390 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.390 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.390 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.391 INFO:teuthology.orchestra.run.smithi016.stdout: "address": { 2024-03-28T22:20:44.391 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "address", 2024-03-28T22:20:44.391 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.391 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.391 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.391 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-28T22:20:44.391 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.392 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.392 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.392 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.392 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.392 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.392 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.392 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-03-28T22:20:44.393 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-03-28T22:20:44.393 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:44.393 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.393 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.393 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15", 2024-03-28T22:20:44.393 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.393 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.393 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.393 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.394 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.394 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.394 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.394 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.394 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.394 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.394 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.395 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.395 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.395 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.395 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.395 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.395 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.395 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.395 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.395 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.396 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.397 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.397 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.397 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.397 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.398 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.398 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.398 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.398 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.398 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.398 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.398 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.399 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.399 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.399 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.399 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.399 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.399 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.399 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.399 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.399 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.400 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.400 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.400 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.400 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.400 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.400 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.400 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.400 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.400 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.401 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.401 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.401 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.401 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.401 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.401 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.401 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.402 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.402 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.402 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.402 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.402 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.402 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.402 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.403 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.403 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.403 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.403 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.403 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.403 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.403 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.403 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.404 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.404 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.404 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.404 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telemetry", 2024-03-28T22:20:44.404 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.404 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.405 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.405 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_basic": { 2024-03-28T22:20:44.405 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_basic", 2024-03-28T22:20:44.405 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.405 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.405 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.405 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.405 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.406 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.406 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.406 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-28T22:20:44.406 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.407 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.407 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.407 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_crash": { 2024-03-28T22:20:44.407 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_crash", 2024-03-28T22:20:44.407 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.407 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.407 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.408 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.408 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.408 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.408 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.408 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-28T22:20:44.408 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.408 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.409 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.409 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_device": { 2024-03-28T22:20:44.409 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_device", 2024-03-28T22:20:44.409 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.409 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.409 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.409 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.409 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.410 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.410 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.410 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-28T22:20:44.410 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.410 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.410 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.410 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_ident": { 2024-03-28T22:20:44.411 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_ident", 2024-03-28T22:20:44.411 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.411 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.411 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.411 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.411 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.411 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.411 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.412 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-28T22:20:44.412 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.412 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.412 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.412 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_perf": { 2024-03-28T22:20:44.412 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_perf", 2024-03-28T22:20:44.412 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.413 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.413 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.413 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.413 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.413 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.413 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.413 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-28T22:20:44.413 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.414 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.414 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.414 INFO:teuthology.orchestra.run.smithi016.stdout: "contact": { 2024-03-28T22:20:44.414 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "contact", 2024-03-28T22:20:44.414 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.414 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.414 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.415 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.415 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.415 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.415 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.415 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.415 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.415 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.415 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.416 INFO:teuthology.orchestra.run.smithi016.stdout: "description": { 2024-03-28T22:20:44.416 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "description", 2024-03-28T22:20:44.416 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.416 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.416 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.416 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.416 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.416 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.417 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.417 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.417 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.417 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.417 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.417 INFO:teuthology.orchestra.run.smithi016.stdout: "device_url": { 2024-03-28T22:20:44.417 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_url", 2024-03-28T22:20:44.418 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.418 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.418 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.418 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-28T22:20:44.418 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.418 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.418 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.418 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.419 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.419 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.419 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.419 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-03-28T22:20:44.419 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-03-28T22:20:44.419 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.419 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.420 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.420 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.420 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.420 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.420 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.420 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.420 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.420 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.420 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.421 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-03-28T22:20:44.421 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-03-28T22:20:44.421 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.421 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.421 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.421 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "24", 2024-03-28T22:20:44.421 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "8", 2024-03-28T22:20:44.422 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.422 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.422 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.422 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.422 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.422 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.423 INFO:teuthology.orchestra.run.smithi016.stdout: "last_opt_revision": { 2024-03-28T22:20:44.423 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "last_opt_revision", 2024-03-28T22:20:44.423 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.423 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.423 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.423 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1", 2024-03-28T22:20:44.423 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.423 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.423 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.424 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.424 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.424 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.424 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.424 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard": { 2024-03-28T22:20:44.424 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard", 2024-03-28T22:20:44.424 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.425 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.425 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.425 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.425 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.425 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.425 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.425 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.425 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.426 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.426 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.426 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard_description": { 2024-03-28T22:20:44.426 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard_description", 2024-03-28T22:20:44.426 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.426 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.427 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.427 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.427 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.427 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.427 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.427 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.427 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.428 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.428 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.428 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.428 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.428 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.428 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.428 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.428 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.429 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.429 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.429 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.429 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.429 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.429 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.429 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.430 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.430 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.430 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.430 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.430 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.430 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.430 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.431 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.431 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.431 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.431 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.431 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.431 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.431 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.432 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.432 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.432 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.432 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.432 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.432 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.432 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.432 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.433 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.433 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.433 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.433 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.433 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.433 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.433 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.433 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.433 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.434 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.434 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.434 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.434 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.434 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.434 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.434 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.435 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.435 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.435 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.435 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.435 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.435 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.435 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.436 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.436 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.436 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.436 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.436 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.436 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.436 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.436 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.437 INFO:teuthology.orchestra.run.smithi016.stdout: "organization": { 2024-03-28T22:20:44.437 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "organization", 2024-03-28T22:20:44.437 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.437 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.437 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.437 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.437 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.437 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.438 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.438 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.438 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.438 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.438 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.438 INFO:teuthology.orchestra.run.smithi016.stdout: "proxy": { 2024-03-28T22:20:44.438 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "proxy", 2024-03-28T22:20:44.439 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.439 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.439 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.439 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.439 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.439 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.439 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.439 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.439 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.440 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.440 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.440 INFO:teuthology.orchestra.run.smithi016.stdout: "url": { 2024-03-28T22:20:44.440 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url", 2024-03-28T22:20:44.440 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.440 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.440 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.440 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-28T22:20:44.441 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.441 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.441 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.441 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.441 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.441 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.441 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.441 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.441 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.442 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "test_orchestrator", 2024-03-28T22:20:44.442 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.442 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.442 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.442 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.442 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.442 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.443 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.443 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.443 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.443 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.443 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.443 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.443 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.443 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.443 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.444 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.444 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.444 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.444 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.444 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.444 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.444 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.444 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.445 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.445 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.445 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.445 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.445 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.445 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.445 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.445 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.446 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.446 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.446 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.446 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.446 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.446 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.446 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.446 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.447 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.447 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.447 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.447 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.447 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.447 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.447 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.447 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.448 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.448 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.448 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.448 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.448 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.448 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.448 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.449 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.449 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.449 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.449 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.449 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.449 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.450 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.450 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.450 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.450 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.450 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.450 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.450 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.450 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.451 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.451 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.451 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.451 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "volumes", 2024-03-28T22:20:44.451 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.451 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.452 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.452 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.452 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.452 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.452 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.452 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.452 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.453 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.453 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.453 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.453 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.453 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.453 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.453 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.454 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.454 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.454 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.454 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.454 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.454 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.454 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.455 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.455 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.455 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.455 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.455 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.455 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.455 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.456 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.456 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.456 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.456 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.456 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.456 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.456 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.457 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.457 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.457 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.457 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.457 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.457 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.457 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.457 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.458 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.458 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.458 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.458 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.458 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.458 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.458 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.458 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.458 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.459 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.459 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.459 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.459 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.459 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.459 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.459 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.460 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.460 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.460 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.460 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.460 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.460 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.460 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.460 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.461 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_clones": { 2024-03-28T22:20:44.461 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_clones", 2024-03-28T22:20:44.461 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.461 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.461 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.461 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4", 2024-03-28T22:20:44.462 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.462 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.462 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.462 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-28T22:20:44.462 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.462 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.462 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.463 INFO:teuthology.orchestra.run.smithi016.stdout: "periodic_async_work": { 2024-03-28T22:20:44.463 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "periodic_async_work", 2024-03-28T22:20:44.463 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.463 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.463 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.463 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.463 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.464 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.464 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.464 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Periodically check for async work", 2024-03-28T22:20:44.464 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.464 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.464 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.464 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_delay": { 2024-03-28T22:20:44.465 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_delay", 2024-03-28T22:20:44.465 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.465 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.465 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.465 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-03-28T22:20:44.465 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.465 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.465 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.466 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-28T22:20:44.466 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.466 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.466 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.466 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_no_wait": { 2024-03-28T22:20:44.466 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_no_wait", 2024-03-28T22:20:44.467 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.467 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.467 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.467 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.467 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.467 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.467 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.467 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-28T22:20:44.467 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.468 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.468 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.468 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.468 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.468 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix", 2024-03-28T22:20:44.468 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.469 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.469 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.469 INFO:teuthology.orchestra.run.smithi016.stdout: "discovery_interval": { 2024-03-28T22:20:44.469 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "discovery_interval", 2024-03-28T22:20:44.469 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-03-28T22:20:44.469 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.469 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.469 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "100", 2024-03-28T22:20:44.469 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.470 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.470 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.470 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.470 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.470 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.470 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.470 INFO:teuthology.orchestra.run.smithi016.stdout: "identifier": { 2024-03-28T22:20:44.471 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "identifier", 2024-03-28T22:20:44.471 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.471 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.471 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.471 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.472 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.472 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.472 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.472 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.472 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.472 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.473 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.473 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-03-28T22:20:44.473 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-03-28T22:20:44.473 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:44.474 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.474 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.474 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-03-28T22:20:44.474 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.474 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.474 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.475 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.475 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.475 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.475 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.475 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.475 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.476 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.476 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.476 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.476 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.476 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.476 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.476 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.477 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.477 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.477 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.477 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.477 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.477 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.477 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.477 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.478 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.478 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.478 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.478 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.478 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.478 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.479 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.479 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.479 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.479 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.479 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.479 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.479 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.479 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.480 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.480 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.480 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.480 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.480 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.480 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.480 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.480 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.481 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.481 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.481 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.481 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.481 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.481 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.481 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.481 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.481 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.482 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.482 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.482 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.482 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.482 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.482 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.482 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.483 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.483 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.483 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.483 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.483 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.483 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.483 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.483 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.484 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.484 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.484 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.484 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_host": { 2024-03-28T22:20:44.484 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_host", 2024-03-28T22:20:44.484 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.484 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.485 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.485 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.485 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.485 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.485 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.485 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.485 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.485 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.485 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.486 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_port": { 2024-03-28T22:20:44.486 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_port", 2024-03-28T22:20:44.486 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.486 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.486 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.486 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10051", 2024-03-28T22:20:44.487 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.487 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.487 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.487 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.487 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.487 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.487 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.488 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_sender": { 2024-03-28T22:20:44.488 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_sender", 2024-03-28T22:20:44.488 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.488 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.488 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.488 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-28T22:20:44.488 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.488 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.489 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.489 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.489 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.489 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.489 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.489 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.489 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.489 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.490 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-03-28T22:20:44.490 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.490 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.490 INFO:teuthology.orchestra.run.smithi016.stdout: "modules": [ 2024-03-28T22:20:44.490 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm", 2024-03-28T22:20:44.490 INFO:teuthology.orchestra.run.smithi016.stdout: "dashboard", 2024-03-28T22:20:44.490 INFO:teuthology.orchestra.run.smithi016.stdout: "iostat", 2024-03-28T22:20:44.491 INFO:teuthology.orchestra.run.smithi016.stdout: "nfs", 2024-03-28T22:20:44.491 INFO:teuthology.orchestra.run.smithi016.stdout: "restful", 2024-03-28T22:20:44.491 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_schedule" 2024-03-28T22:20:44.491 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.491 INFO:teuthology.orchestra.run.smithi016.stdout: "available_modules": [ 2024-03-28T22:20:44.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.491 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "alerts", 2024-03-28T22:20:44.491 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.491 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.492 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.492 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-03-28T22:20:44.492 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-03-28T22:20:44.492 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:44.492 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.492 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.492 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-03-28T22:20:44.493 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.493 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.493 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.493 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How frequently to reexamine health status", 2024-03-28T22:20:44.493 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.493 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.493 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.494 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.494 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.494 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.494 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.494 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.494 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.494 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.495 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.495 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.495 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.495 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.495 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.495 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.495 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.495 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.495 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.496 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.496 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.496 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.496 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.496 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.496 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.496 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.497 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.497 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.497 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.497 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.497 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.497 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.497 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.497 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.497 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.498 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.498 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.498 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.498 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.498 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.498 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.498 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.499 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.499 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.499 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.499 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.499 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.499 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.499 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.499 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.500 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.500 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.500 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.500 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.500 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.500 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.500 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.501 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.501 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.501 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.501 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.501 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.501 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.501 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.501 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.502 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.502 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.502 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.502 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.502 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_destination": { 2024-03-28T22:20:44.502 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_destination", 2024-03-28T22:20:44.503 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.503 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.503 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.503 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.503 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.503 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.503 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.503 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email address to send alerts to", 2024-03-28T22:20:44.503 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.504 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.504 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.504 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_from_name": { 2024-03-28T22:20:44.504 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_from_name", 2024-03-28T22:20:44.504 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.504 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.504 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.505 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "Ceph", 2024-03-28T22:20:44.505 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.505 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.505 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.505 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email From: name", 2024-03-28T22:20:44.505 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.505 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.505 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.506 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_host": { 2024-03-28T22:20:44.506 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_host", 2024-03-28T22:20:44.506 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.506 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.506 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.506 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.506 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.507 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.507 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.507 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP server", 2024-03-28T22:20:44.507 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.507 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.507 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.507 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_password": { 2024-03-28T22:20:44.508 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_password", 2024-03-28T22:20:44.508 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.508 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.508 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.508 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.508 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.508 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.508 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.509 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Password to authenticate with", 2024-03-28T22:20:44.509 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.509 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.509 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.509 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_port": { 2024-03-28T22:20:44.509 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_port", 2024-03-28T22:20:44.509 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.510 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.510 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.510 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "465", 2024-03-28T22:20:44.510 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.510 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.510 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.510 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP port", 2024-03-28T22:20:44.510 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.511 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.511 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.511 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_sender": { 2024-03-28T22:20:44.511 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_sender", 2024-03-28T22:20:44.511 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.511 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.511 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.511 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.512 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.512 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.512 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.512 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP envelope sender", 2024-03-28T22:20:44.512 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.512 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.512 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.513 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_ssl": { 2024-03-28T22:20:44.513 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_ssl", 2024-03-28T22:20:44.513 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.513 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.513 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.513 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.513 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.513 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.514 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.514 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-28T22:20:44.514 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.514 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.514 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.514 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_user": { 2024-03-28T22:20:44.514 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_user", 2024-03-28T22:20:44.515 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.515 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.515 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.515 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.515 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.515 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.515 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.515 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "User to authenticate as", 2024-03-28T22:20:44.515 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.516 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.516 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.516 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.516 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.516 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.516 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "balancer", 2024-03-28T22:20:44.516 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.517 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.517 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.517 INFO:teuthology.orchestra.run.smithi016.stdout: "active": { 2024-03-28T22:20:44.517 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "active", 2024-03-28T22:20:44.517 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.517 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.517 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.517 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.517 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.518 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.518 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.518 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically balance PGs across cluster", 2024-03-28T22:20:44.518 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.518 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.518 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.519 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_time": { 2024-03-28T22:20:44.519 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_time", 2024-03-28T22:20:44.519 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.519 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.519 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.519 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0000", 2024-03-28T22:20:44.519 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.519 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.520 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.520 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "beginning time of day to automatically balance", 2024-03-28T22:20:44.520 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T22:20:44.520 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.520 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.520 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_weekday": { 2024-03-28T22:20:44.521 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_weekday", 2024-03-28T22:20:44.521 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-03-28T22:20:44.521 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.521 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.521 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-03-28T22:20:44.521 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-03-28T22:20:44.521 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-03-28T22:20:44.522 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.522 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-28T22:20:44.522 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T22:20:44.522 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.522 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.522 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_max_iterations": { 2024-03-28T22:20:44.523 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_max_iterations", 2024-03-28T22:20:44.523 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-03-28T22:20:44.523 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.523 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.523 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "25", 2024-03-28T22:20:44.523 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-03-28T22:20:44.523 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "250", 2024-03-28T22:20:44.523 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.524 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-28T22:20:44.524 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.524 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.524 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.524 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_metrics": { 2024-03-28T22:20:44.524 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_metrics", 2024-03-28T22:20:44.524 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.525 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.525 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.525 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "pgs,objects,bytes", 2024-03-28T22:20:44.525 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.525 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.525 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.525 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-28T22:20:44.525 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-28T22:20:44.526 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.526 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.526 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_step": { 2024-03-28T22:20:44.526 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_step", 2024-03-28T22:20:44.526 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-03-28T22:20:44.526 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.527 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.527 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.5", 2024-03-28T22:20:44.527 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0.001", 2024-03-28T22:20:44.527 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "0.999", 2024-03-28T22:20:44.527 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.527 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "aggressiveness of optimization", 2024-03-28T22:20:44.527 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-28T22:20:44.527 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.528 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.528 INFO:teuthology.orchestra.run.smithi016.stdout: "end_time": { 2024-03-28T22:20:44.528 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_time", 2024-03-28T22:20:44.528 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.528 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.528 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.528 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2359", 2024-03-28T22:20:44.529 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.529 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.529 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.529 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ending time of day to automatically balance", 2024-03-28T22:20:44.529 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T22:20:44.529 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.529 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.530 INFO:teuthology.orchestra.run.smithi016.stdout: "end_weekday": { 2024-03-28T22:20:44.530 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_weekday", 2024-03-28T22:20:44.530 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-03-28T22:20:44.530 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.530 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.530 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-03-28T22:20:44.530 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-03-28T22:20:44.531 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-03-28T22:20:44.531 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.531 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-28T22:20:44.531 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T22:20:44.531 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.531 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.531 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.532 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.532 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.532 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.532 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.532 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.532 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.532 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.533 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.533 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.533 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.533 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.533 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.534 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.534 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.534 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.534 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.534 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.535 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.535 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.535 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.536 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.536 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.536 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.536 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.536 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.537 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.537 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.537 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.537 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.537 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.537 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.537 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.538 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.538 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.538 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.538 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.538 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.538 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.538 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.538 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.539 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.539 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.539 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.539 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.539 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.539 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.539 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.540 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.540 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.540 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.540 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.540 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.540 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.540 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.541 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.541 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.541 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.541 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.541 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.541 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.541 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.541 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.542 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.542 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.542 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.542 INFO:teuthology.orchestra.run.smithi016.stdout: "min_score": { 2024-03-28T22:20:44.542 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_score", 2024-03-28T22:20:44.542 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-03-28T22:20:44.542 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.542 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.543 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-03-28T22:20:44.543 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.543 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.543 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.543 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-28T22:20:44.543 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.543 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.543 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.544 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-03-28T22:20:44.544 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-03-28T22:20:44.544 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.544 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.544 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.544 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "upmap", 2024-03-28T22:20:44.544 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.544 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.545 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.545 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-compat", 2024-03-28T22:20:44.545 INFO:teuthology.orchestra.run.smithi016.stdout: "none", 2024-03-28T22:20:44.545 INFO:teuthology.orchestra.run.smithi016.stdout: "read", 2024-03-28T22:20:44.545 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap", 2024-03-28T22:20:44.545 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap-read" 2024-03-28T22:20:44.545 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.545 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Balancer mode", 2024-03-28T22:20:44.546 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.546 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.546 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.546 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_ids": { 2024-03-28T22:20:44.546 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_ids", 2024-03-28T22:20:44.546 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.546 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.546 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.547 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.547 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.547 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.547 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.547 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-28T22:20:44.547 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.547 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.547 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.548 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-03-28T22:20:44.548 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-03-28T22:20:44.548 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:44.548 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.548 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.548 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-03-28T22:20:44.548 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.549 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.549 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.549 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-28T22:20:44.549 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.549 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.549 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.549 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_deviation": { 2024-03-28T22:20:44.550 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_deviation", 2024-03-28T22:20:44.550 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.550 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.550 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.550 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-03-28T22:20:44.550 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-03-28T22:20:44.550 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.550 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.551 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-28T22:20:44.551 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-28T22:20:44.551 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.551 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.551 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_optimizations": { 2024-03-28T22:20:44.551 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_optimizations", 2024-03-28T22:20:44.551 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-03-28T22:20:44.551 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.552 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.552 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-03-28T22:20:44.552 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.552 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.552 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.552 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-28T22:20:44.552 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.552 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.553 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.553 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.553 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.553 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm", 2024-03-28T22:20:44.553 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.553 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.553 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.554 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_down_multiplier": { 2024-03-28T22:20:44.554 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_down_multiplier", 2024-03-28T22:20:44.554 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-03-28T22:20:44.554 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.554 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.554 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-03-28T22:20:44.554 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.554 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.555 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.555 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-28T22:20:44.555 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.555 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.555 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.555 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_refresh_rate": { 2024-03-28T22:20:44.555 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_refresh_rate", 2024-03-28T22:20:44.556 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:44.556 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.556 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.556 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "20", 2024-03-28T22:20:44.556 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.556 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.556 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.556 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-28T22:20:44.557 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.557 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.557 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.557 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_starting_port": { 2024-03-28T22:20:44.557 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_starting_port", 2024-03-28T22:20:44.557 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.557 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.558 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.558 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4721", 2024-03-28T22:20:44.558 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.558 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.558 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.558 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-28T22:20:44.558 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.559 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.559 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.559 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_ptrace": { 2024-03-28T22:20:44.559 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_ptrace", 2024-03-28T22:20:44.559 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.559 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.559 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.559 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.560 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.560 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.560 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.560 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-28T22:20:44.560 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-28T22:20:44.560 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.560 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.561 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_interval": { 2024-03-28T22:20:44.561 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_interval", 2024-03-28T22:20:44.561 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:44.561 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.561 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.561 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-03-28T22:20:44.561 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.561 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.562 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.562 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-28T22:20:44.562 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.562 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.562 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.562 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_memory_target_ratio": { 2024-03-28T22:20:44.562 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_memory_target_ratio", 2024-03-28T22:20:44.562 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-03-28T22:20:44.563 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.563 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.563 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.7", 2024-03-28T22:20:44.563 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.563 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.563 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.563 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-28T22:20:44.563 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.564 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.564 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.564 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm_log_destination": { 2024-03-28T22:20:44.564 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm_log_destination", 2024-03-28T22:20:44.564 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.564 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.565 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.565 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.565 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.565 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.565 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.565 INFO:teuthology.orchestra.run.smithi016.stdout: "file", 2024-03-28T22:20:44.565 INFO:teuthology.orchestra.run.smithi016.stdout: "file,syslog", 2024-03-28T22:20:44.565 INFO:teuthology.orchestra.run.smithi016.stdout: "syslog" 2024-03-28T22:20:44.565 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.566 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-28T22:20:44.566 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.566 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.566 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.566 INFO:teuthology.orchestra.run.smithi016.stdout: "cgroups_split": { 2024-03-28T22:20:44.566 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cgroups_split", 2024-03-28T22:20:44.566 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.567 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.567 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.567 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.567 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.567 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.567 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.567 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-28T22:20:44.567 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.567 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.568 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.568 INFO:teuthology.orchestra.run.smithi016.stdout: "config_checks_enabled": { 2024-03-28T22:20:44.568 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_checks_enabled", 2024-03-28T22:20:44.568 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.568 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.568 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.568 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.569 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.569 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.569 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.569 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-28T22:20:44.569 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.569 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.569 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.570 INFO:teuthology.orchestra.run.smithi016.stdout: "config_dashboard": { 2024-03-28T22:20:44.570 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_dashboard", 2024-03-28T22:20:44.570 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.570 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.570 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.570 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.570 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.570 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.570 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.571 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-28T22:20:44.571 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.571 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.571 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.571 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_alertmanager": { 2024-03-28T22:20:44.571 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_alertmanager", 2024-03-28T22:20:44.571 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.572 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.572 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.572 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-28T22:20:44.572 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.572 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.572 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.572 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-03-28T22:20:44.572 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.573 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.573 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.573 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_base": { 2024-03-28T22:20:44.573 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_base", 2024-03-28T22:20:44.573 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.573 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.573 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.574 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-28T22:20:44.574 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.574 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.574 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.574 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Container image name, without the tag", 2024-03-28T22:20:44.574 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.574 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.574 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.575 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_elasticsearch": { 2024-03-28T22:20:44.575 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_elasticsearch", 2024-03-28T22:20:44.575 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.575 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.575 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.575 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-28T22:20:44.575 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.576 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.576 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.576 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "elasticsearch container image", 2024-03-28T22:20:44.576 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.576 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.576 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.576 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_grafana": { 2024-03-28T22:20:44.577 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_grafana", 2024-03-28T22:20:44.577 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.577 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.577 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.577 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-28T22:20:44.577 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.577 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.577 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.578 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-03-28T22:20:44.578 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.578 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.578 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.578 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_haproxy": { 2024-03-28T22:20:44.578 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_haproxy", 2024-03-28T22:20:44.579 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.579 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.579 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.579 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-28T22:20:44.579 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.579 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.579 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.579 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "HAproxy container image", 2024-03-28T22:20:44.580 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.580 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.580 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.580 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_agent": { 2024-03-28T22:20:44.580 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_agent", 2024-03-28T22:20:44.580 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.580 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.581 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.581 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-28T22:20:44.581 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.581 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.581 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.581 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger agent container image", 2024-03-28T22:20:44.581 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.582 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.582 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.582 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_collector": { 2024-03-28T22:20:44.582 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_collector", 2024-03-28T22:20:44.582 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.582 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.582 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.583 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-28T22:20:44.583 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.583 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.583 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.583 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger collector container image", 2024-03-28T22:20:44.583 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.583 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.583 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.584 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_query": { 2024-03-28T22:20:44.584 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_query", 2024-03-28T22:20:44.584 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.584 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.584 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.584 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-28T22:20:44.584 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.585 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.585 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.585 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger query container image", 2024-03-28T22:20:44.585 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.585 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.585 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.585 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_keepalived": { 2024-03-28T22:20:44.586 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_keepalived", 2024-03-28T22:20:44.586 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.586 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.586 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.586 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-28T22:20:44.586 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.586 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.587 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.587 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Keepalived container image", 2024-03-28T22:20:44.587 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.587 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.587 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.587 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_loki": { 2024-03-28T22:20:44.587 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_loki", 2024-03-28T22:20:44.588 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.588 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.588 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.588 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-28T22:20:44.588 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.588 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.588 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.589 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Loki container image", 2024-03-28T22:20:44.589 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.589 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.589 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.589 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_node_exporter": { 2024-03-28T22:20:44.589 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_node_exporter", 2024-03-28T22:20:44.589 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.590 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.590 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.590 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-28T22:20:44.590 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.590 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.590 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.590 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-03-28T22:20:44.590 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.591 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.591 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.591 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_nvmeof": { 2024-03-28T22:20:44.591 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_nvmeof", 2024-03-28T22:20:44.591 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.591 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.591 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.592 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-28T22:20:44.592 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.592 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.592 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.592 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Nvme-of container image", 2024-03-28T22:20:44.592 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.592 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.592 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.593 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_prometheus": { 2024-03-28T22:20:44.593 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_prometheus", 2024-03-28T22:20:44.593 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.593 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.593 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.593 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-28T22:20:44.593 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.594 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.594 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.594 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-03-28T22:20:44.594 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.594 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.594 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.595 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_promtail": { 2024-03-28T22:20:44.595 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_promtail", 2024-03-28T22:20:44.595 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.595 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.595 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.595 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-28T22:20:44.595 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.595 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.596 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.596 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Promtail container image", 2024-03-28T22:20:44.596 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.596 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.596 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.596 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_snmp_gateway": { 2024-03-28T22:20:44.597 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_snmp_gateway", 2024-03-28T22:20:44.597 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.597 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.597 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.597 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-28T22:20:44.597 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.597 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.597 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.597 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SNMP Gateway container image", 2024-03-28T22:20:44.598 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.598 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.598 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.598 INFO:teuthology.orchestra.run.smithi016.stdout: "container_init": { 2024-03-28T22:20:44.598 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_init", 2024-03-28T22:20:44.598 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.599 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.599 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.599 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.599 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.599 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.599 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.599 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Run podman/docker with `--init`", 2024-03-28T22:20:44.599 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.599 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.601 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.601 INFO:teuthology.orchestra.run.smithi016.stdout: "daemon_cache_timeout": { 2024-03-28T22:20:44.601 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "daemon_cache_timeout", 2024-03-28T22:20:44.602 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:44.602 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.602 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.602 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-03-28T22:20:44.602 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.602 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.602 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.602 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-28T22:20:44.603 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.603 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.603 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.603 INFO:teuthology.orchestra.run.smithi016.stdout: "default_cephadm_command_timeout": { 2024-03-28T22:20:44.603 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_cephadm_command_timeout", 2024-03-28T22:20:44.603 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.603 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.603 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.604 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "900", 2024-03-28T22:20:44.604 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.604 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.604 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.604 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-28T22:20:44.604 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.604 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.604 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: "default_registry": { 2024-03-28T22:20:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_registry", 2024-03-28T22:20:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io", 2024-03-28T22:20:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.606 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.606 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-28T22:20:44.606 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.606 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.606 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.606 INFO:teuthology.orchestra.run.smithi016.stdout: "device_cache_timeout": { 2024-03-28T22:20:44.606 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_cache_timeout", 2024-03-28T22:20:44.607 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:44.607 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.607 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.607 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1800", 2024-03-28T22:20:44.607 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.607 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.607 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.607 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache device inventory", 2024-03-28T22:20:44.607 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.608 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.608 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.608 INFO:teuthology.orchestra.run.smithi016.stdout: "device_enhanced_scan": { 2024-03-28T22:20:44.608 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_enhanced_scan", 2024-03-28T22:20:44.608 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.608 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.608 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.609 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.609 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.609 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.609 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.609 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-28T22:20:44.609 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.609 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.609 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.610 INFO:teuthology.orchestra.run.smithi016.stdout: "facts_cache_timeout": { 2024-03-28T22:20:44.610 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "facts_cache_timeout", 2024-03-28T22:20:44.610 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:44.610 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.610 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.610 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-03-28T22:20:44.610 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache host facts data", 2024-03-28T22:20:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: "host_check_interval": { 2024-03-28T22:20:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "host_check_interval", 2024-03-28T22:20:44.612 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:44.612 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.612 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.612 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-03-28T22:20:44.612 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.612 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.612 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.613 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to perform a host check", 2024-03-28T22:20:44.613 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.613 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.613 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.613 INFO:teuthology.orchestra.run.smithi016.stdout: "hw_monitoring": { 2024-03-28T22:20:44.613 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hw_monitoring", 2024-03-28T22:20:44.613 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.613 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.614 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.614 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.614 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.614 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.614 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.614 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-28T22:20:44.614 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.614 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.615 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.615 INFO:teuthology.orchestra.run.smithi016.stdout: "inventory_list_all": { 2024-03-28T22:20:44.615 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "inventory_list_all", 2024-03-28T22:20:44.615 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.615 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.615 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.615 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.616 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.616 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.616 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.616 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-28T22:20:44.616 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.616 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.616 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.618 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.618 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.618 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.618 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.618 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.618 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.618 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.618 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.619 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.619 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.619 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.619 INFO:teuthology.orchestra.run.smithi016.stdout: "log_refresh_metadata": { 2024-03-28T22:20:44.619 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_refresh_metadata", 2024-03-28T22:20:44.619 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.619 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.620 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.620 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.620 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.620 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.620 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.620 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-28T22:20:44.620 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.620 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.621 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.621 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.621 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.621 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.621 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.621 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.621 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.622 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.622 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.622 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.622 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-28T22:20:44.622 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.622 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.622 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.623 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.623 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.623 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.623 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.623 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.623 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.623 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.623 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.624 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.624 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.624 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.624 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.624 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.624 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.624 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.626 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.626 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.626 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.626 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.626 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.626 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.626 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.626 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.627 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.627 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.627 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-28T22:20:44.627 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-28T22:20:44.627 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.627 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.627 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.628 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.628 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.628 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.628 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.628 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-28T22:20:44.628 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.628 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.628 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-28T22:20:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-28T22:20:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "*", 2024-03-28T22:20:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.630 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.630 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.630 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-28T22:20:44.630 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.630 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.630 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.630 INFO:teuthology.orchestra.run.smithi016.stdout: "max_count_per_host": { 2024-03-28T22:20:44.631 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_count_per_host", 2024-03-28T22:20:44.631 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.631 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.631 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.631 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-03-28T22:20:44.631 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.631 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.631 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.632 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of daemons per service per host", 2024-03-28T22:20:44.632 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.632 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.632 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.632 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd_draining_count": { 2024-03-28T22:20:44.632 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_osd_draining_count", 2024-03-28T22:20:44.632 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.633 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.633 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.633 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-03-28T22:20:44.633 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.633 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.633 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.633 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-28T22:20:44.633 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.633 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.634 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.634 INFO:teuthology.orchestra.run.smithi016.stdout: "migration_current": { 2024-03-28T22:20:44.634 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "migration_current", 2024-03-28T22:20:44.634 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.634 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.634 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.634 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.635 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.635 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.635 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.635 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "internal - do not modify", 2024-03-28T22:20:44.635 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.635 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.635 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.636 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-03-28T22:20:44.636 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-03-28T22:20:44.636 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.636 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.636 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.636 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "root", 2024-03-28T22:20:44.636 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.636 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.637 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.637 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm-package", 2024-03-28T22:20:44.637 INFO:teuthology.orchestra.run.smithi016.stdout: "root" 2024-03-28T22:20:44.637 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.637 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "mode for remote execution of cephadm", 2024-03-28T22:20:44.637 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.637 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.637 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.638 INFO:teuthology.orchestra.run.smithi016.stdout: "oob_default_addr": { 2024-03-28T22:20:44.638 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "oob_default_addr", 2024-03-28T22:20:44.638 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.638 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.638 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.638 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "169.254.1.1", 2024-03-28T22:20:44.638 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.639 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.639 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.639 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-28T22:20:44.639 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.639 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.639 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.639 INFO:teuthology.orchestra.run.smithi016.stdout: "prometheus_alerts_path": { 2024-03-28T22:20:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus_alerts_path", 2024-03-28T22:20:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-28T22:20:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-28T22:20:44.641 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.641 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.641 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.641 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_insecure": { 2024-03-28T22:20:44.641 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_insecure", 2024-03-28T22:20:44.641 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.641 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.642 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.642 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.642 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.642 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.642 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.642 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-28T22:20:44.642 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.642 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.643 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.643 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_password": { 2024-03-28T22:20:44.643 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_password", 2024-03-28T22:20:44.643 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.643 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.643 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.643 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.643 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.644 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.644 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.644 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-28T22:20:44.644 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.644 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.644 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.644 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_url": { 2024-03-28T22:20:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_url", 2024-03-28T22:20:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-28T22:20:44.646 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.646 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.646 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.646 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_username": { 2024-03-28T22:20:44.646 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_username", 2024-03-28T22:20:44.646 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.646 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.647 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.647 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.647 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.647 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.647 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.647 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-28T22:20:44.647 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.647 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.648 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.648 INFO:teuthology.orchestra.run.smithi016.stdout: "secure_monitoring_stack": { 2024-03-28T22:20:44.648 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "secure_monitoring_stack", 2024-03-28T22:20:44.648 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.648 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.648 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-28T22:20:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.650 INFO:teuthology.orchestra.run.smithi016.stdout: "service_discovery_port": { 2024-03-28T22:20:44.650 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "service_discovery_port", 2024-03-28T22:20:44.650 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.650 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.650 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.650 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8765", 2024-03-28T22:20:44.650 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.651 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.651 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.651 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "cephadm service discovery port", 2024-03-28T22:20:44.651 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.651 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.651 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.651 INFO:teuthology.orchestra.run.smithi016.stdout: "ssh_config_file": { 2024-03-28T22:20:44.652 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssh_config_file", 2024-03-28T22:20:44.652 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.652 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.652 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.652 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.652 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.652 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.652 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-28T22:20:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: "use_agent": { 2024-03-28T22:20:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_agent", 2024-03-28T22:20:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.654 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.654 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.654 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.654 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.654 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.654 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-28T22:20:44.654 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: "use_repo_digest": { 2024-03-28T22:20:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_repo_digest", 2024-03-28T22:20:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.656 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.656 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.656 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.656 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.656 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.656 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-28T22:20:44.656 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.656 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.657 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.657 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_failed_host_check": { 2024-03-28T22:20:44.657 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_failed_host_check", 2024-03-28T22:20:44.657 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.657 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.657 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.657 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.658 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.658 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.658 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.658 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if the host check fails", 2024-03-28T22:20:44.658 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.658 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.658 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.659 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_daemons": { 2024-03-28T22:20:44.659 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_daemons", 2024-03-28T22:20:44.659 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.659 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.659 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.659 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.659 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.659 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.660 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.660 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-28T22:20:44.660 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.660 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.660 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.660 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_hosts": { 2024-03-28T22:20:44.660 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_hosts", 2024-03-28T22:20:44.660 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.661 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.661 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.661 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.661 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.661 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.661 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.661 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-28T22:20:44.661 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.662 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.662 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.662 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.662 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.662 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crash", 2024-03-28T22:20:44.662 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.664 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.664 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.664 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.664 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.664 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.664 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.664 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.664 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.665 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.665 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.665 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.665 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.665 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.665 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.665 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.666 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.666 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.666 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.666 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.666 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.666 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.666 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.666 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.667 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.667 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.667 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.667 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.667 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.667 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.667 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.667 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.668 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.668 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.668 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.668 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.668 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.668 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.668 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.668 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.669 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.669 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.669 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.669 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.669 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.669 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.669 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.669 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.670 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.670 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.670 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.670 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.670 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.670 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.671 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.671 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.671 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.671 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.671 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.671 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.671 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.672 INFO:teuthology.orchestra.run.smithi016.stdout: "retain_interval": { 2024-03-28T22:20:44.672 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retain_interval", 2024-03-28T22:20:44.672 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:44.672 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.672 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.672 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "31536000", 2024-03-28T22:20:44.672 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.672 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.673 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.673 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-28T22:20:44.673 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.673 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.673 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.673 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_recent_interval": { 2024-03-28T22:20:44.673 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_recent_interval", 2024-03-28T22:20:44.674 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:44.674 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.674 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.674 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1209600", 2024-03-28T22:20:44.674 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.674 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.674 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-28T22:20:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dashboard", 2024-03-28T22:20:44.676 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.676 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.676 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.676 INFO:teuthology.orchestra.run.smithi016.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-28T22:20:44.676 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-28T22:20:44.676 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.676 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.677 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.677 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-03-28T22:20:44.677 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.677 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.677 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.677 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.677 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.677 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.678 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.678 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_HOST": { 2024-03-28T22:20:44.678 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-28T22:20:44.678 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.678 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.678 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.678 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-28T22:20:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-28T22:20:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.681 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.681 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.681 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.681 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.681 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_ENABLED": { 2024-03-28T22:20:44.681 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_ENABLED", 2024-03-28T22:20:44.681 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.682 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.682 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.682 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.682 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.682 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.682 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.682 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.682 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.683 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.683 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.683 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-28T22:20:44.683 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-28T22:20:44.683 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.683 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.683 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.683 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.684 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.684 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.684 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.684 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.684 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.684 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.684 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.685 INFO:teuthology.orchestra.run.smithi016.stdout: "ENABLE_BROWSABLE_API": { 2024-03-28T22:20:44.685 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-28T22:20:44.685 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.685 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.685 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.685 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.685 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.685 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.685 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.686 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.686 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.686 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.686 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.686 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-28T22:20:44.686 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-28T22:20:44.686 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.687 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.687 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.687 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.687 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.687 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.687 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.687 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.687 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-28T22:20:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-28T22:20:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.689 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.689 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.689 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.689 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.689 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.689 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.689 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.690 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-28T22:20:44.690 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-28T22:20:44.690 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.690 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.690 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.690 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.690 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.691 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.691 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.691 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.691 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.691 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.691 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.692 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-28T22:20:44.692 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-28T22:20:44.692 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.692 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.692 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.692 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.692 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-28T22:20:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-28T22:20:44.694 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.694 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.694 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.694 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.694 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.694 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.694 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-28T22:20:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-28T22:20:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.696 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.696 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.696 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.696 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.696 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.696 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.696 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.696 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-28T22:20:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-28T22:20:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.698 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.698 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.698 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.698 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.698 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.698 INFO:teuthology.orchestra.run.smithi016.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-28T22:20:44.698 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-28T22:20:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.700 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.700 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.700 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_PASSWORD": { 2024-03-28T22:20:44.700 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-28T22:20:44.700 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.700 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.700 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.701 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-03-28T22:20:44.701 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.701 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.701 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.701 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.701 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.701 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.701 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.702 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-28T22:20:44.702 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-28T22:20:44.702 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.702 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.702 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.702 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.702 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.703 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.703 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.703 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.703 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.703 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.703 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.703 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_URL": { 2024-03-28T22:20:44.704 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_URL", 2024-03-28T22:20:44.704 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.704 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.704 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.704 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.704 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.704 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.704 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.705 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.705 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.705 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.705 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.705 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_USERNAME": { 2024-03-28T22:20:44.705 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-28T22:20:44.705 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.706 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.706 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.706 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-03-28T22:20:44.706 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.706 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.706 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.706 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.707 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.707 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.707 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.707 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-28T22:20:44.707 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-28T22:20:44.707 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.707 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.708 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.708 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.708 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.708 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.708 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.708 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.708 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.708 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.709 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.709 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-28T22:20:44.709 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-28T22:20:44.709 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.709 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.709 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.709 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.710 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.710 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.710 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.710 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.710 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.710 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.710 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.711 INFO:teuthology.orchestra.run.smithi016.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-28T22:20:44.711 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-28T22:20:44.711 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.711 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.711 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.711 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.711 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.711 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.712 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.712 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.712 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.712 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.712 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.712 INFO:teuthology.orchestra.run.smithi016.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-28T22:20:44.712 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-28T22:20:44.713 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.713 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.713 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.713 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.713 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.713 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.713 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.713 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.714 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.714 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.714 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.714 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_HOST": { 2024-03-28T22:20:44.714 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-28T22:20:44.714 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.714 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.715 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.715 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.715 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.715 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.715 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.715 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.715 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.716 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.716 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.716 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-28T22:20:44.716 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-28T22:20:44.716 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.717 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.717 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.717 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.717 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.717 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.717 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.718 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.718 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.718 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.718 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.719 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-28T22:20:44.719 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-28T22:20:44.719 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.719 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.719 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.720 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.720 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.720 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.720 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.720 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.721 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.721 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.721 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.721 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-28T22:20:44.721 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-28T22:20:44.722 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.722 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.722 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.722 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.722 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.723 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.723 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.723 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.723 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.723 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.723 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.723 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-28T22:20:44.724 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-28T22:20:44.724 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.724 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.724 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.724 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.724 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.724 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.724 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.725 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.725 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.725 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.725 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.725 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-28T22:20:44.725 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-28T22:20:44.725 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.725 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.726 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.726 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.726 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.726 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.726 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.726 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.726 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.726 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.727 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.727 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-28T22:20:44.727 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-28T22:20:44.727 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.727 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.727 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.728 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.728 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.728 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.728 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.728 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.728 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.728 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.728 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.729 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-28T22:20:44.729 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-28T22:20:44.729 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.729 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.729 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.729 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.729 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.729 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.730 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.730 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.730 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.730 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.730 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.730 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-28T22:20:44.730 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-28T22:20:44.731 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.731 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.731 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.731 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.731 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.731 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.731 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.731 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.732 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.732 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.732 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.732 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_ENABLED": { 2024-03-28T22:20:44.732 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-28T22:20:44.732 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.732 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.733 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.733 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.733 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.733 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.733 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.733 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.733 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.733 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.734 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.734 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-28T22:20:44.734 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-28T22:20:44.734 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.734 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.734 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.735 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-28T22:20:44.735 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.735 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.735 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.735 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.735 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.735 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.735 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.736 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-28T22:20:44.736 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-28T22:20:44.736 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.736 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.736 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.736 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-03-28T22:20:44.737 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.737 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.737 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.737 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.737 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.737 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.737 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.738 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-28T22:20:44.738 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-28T22:20:44.738 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.738 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.738 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.738 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8", 2024-03-28T22:20:44.738 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.739 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.739 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.739 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.739 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.739 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.739 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.739 INFO:teuthology.orchestra.run.smithi016.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-28T22:20:44.740 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-28T22:20:44.740 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.740 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.740 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.740 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "45", 2024-03-28T22:20:44.740 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.740 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.740 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.740 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.741 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.741 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.741 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.741 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ACCESS_KEY": { 2024-03-28T22:20:44.741 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-28T22:20:44.741 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.741 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.742 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.742 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.742 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.742 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.742 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.742 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.742 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.742 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.743 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.743 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-28T22:20:44.743 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-28T22:20:44.743 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.743 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.743 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.743 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-03-28T22:20:44.743 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.744 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.744 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.744 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.744 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.744 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.744 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.744 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SECRET_KEY": { 2024-03-28T22:20:44.745 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-28T22:20:44.745 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.745 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.745 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.745 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.745 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.745 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.745 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.745 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.746 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.746 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.746 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.746 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SSL_VERIFY": { 2024-03-28T22:20:44.746 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-28T22:20:44.746 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.746 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.747 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.747 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.747 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.747 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.747 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.747 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.747 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.747 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.748 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.748 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-28T22:20:44.748 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-28T22:20:44.748 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.748 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.748 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.748 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-03-28T22:20:44.749 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.749 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.749 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.749 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.749 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.749 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.749 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.750 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-28T22:20:44.750 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-28T22:20:44.750 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.750 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.750 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.750 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-03-28T22:20:44.750 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.750 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.751 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.751 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.751 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.751 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.751 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.751 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-28T22:20:44.751 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-28T22:20:44.752 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.752 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.752 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.752 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-03-28T22:20:44.752 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.752 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.752 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.752 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.753 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.753 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.753 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.753 INFO:teuthology.orchestra.run.smithi016.stdout: "cross_origin_url": { 2024-03-28T22:20:44.753 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cross_origin_url", 2024-03-28T22:20:44.753 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.753 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.754 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.754 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.754 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.754 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.754 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.754 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.754 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.754 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.755 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.755 INFO:teuthology.orchestra.run.smithi016.stdout: "crt_file": { 2024-03-28T22:20:44.755 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crt_file", 2024-03-28T22:20:44.755 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.755 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.755 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.755 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.755 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.756 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.756 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.756 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.756 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.756 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.756 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.757 INFO:teuthology.orchestra.run.smithi016.stdout: "debug": { 2024-03-28T22:20:44.757 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "debug", 2024-03-28T22:20:44.757 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.757 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.757 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.757 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.757 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.757 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.757 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.758 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable/disable debug options", 2024-03-28T22:20:44.758 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.758 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.758 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.758 INFO:teuthology.orchestra.run.smithi016.stdout: "jwt_token_ttl": { 2024-03-28T22:20:44.758 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "jwt_token_ttl", 2024-03-28T22:20:44.759 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.759 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.759 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.759 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "28800", 2024-03-28T22:20:44.759 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.759 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.759 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.759 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.760 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.760 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.760 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.760 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-03-28T22:20:44.760 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-03-28T22:20:44.760 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.760 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.761 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.761 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.761 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.761 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.761 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.761 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.761 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.761 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.761 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.762 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.762 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.762 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.762 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.762 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.762 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.762 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.762 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.763 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.763 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.763 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.763 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.763 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.763 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.763 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.764 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.764 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.764 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.764 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.764 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.764 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.764 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.764 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.765 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.765 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.765 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.765 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.765 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.765 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.765 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.765 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.766 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.766 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.766 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.766 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.766 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.766 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.766 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.767 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.767 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.767 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.767 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.767 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.767 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.767 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.768 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.768 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.768 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.768 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.768 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.768 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.768 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.768 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.769 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.769 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.769 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.769 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.769 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.769 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.769 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.769 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.770 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.770 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.770 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.770 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.770 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.770 INFO:teuthology.orchestra.run.smithi016.stdout: "motd": { 2024-03-28T22:20:44.770 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "motd", 2024-03-28T22:20:44.770 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.771 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.771 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.771 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.771 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.771 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.771 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.771 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "The message of the day", 2024-03-28T22:20:44.771 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.771 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.772 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.772 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect_resolve_ip_addr": { 2024-03-28T22:20:44.772 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "redirect_resolve_ip_addr", 2024-03-28T22:20:44.772 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.772 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.772 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.772 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.773 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.773 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.773 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.773 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.773 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.773 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.773 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.773 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-03-28T22:20:44.774 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-03-28T22:20:44.774 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.774 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.774 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.774 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-03-28T22:20:44.774 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.774 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.774 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.775 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.775 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.775 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.775 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.775 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-03-28T22:20:44.775 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-03-28T22:20:44.775 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.775 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.776 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.776 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8080", 2024-03-28T22:20:44.776 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.776 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.776 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.776 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.776 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.776 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.777 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.777 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-03-28T22:20:44.777 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-03-28T22:20:44.777 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.777 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.777 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.777 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.778 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.778 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.778 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.778 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.778 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.778 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.778 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.779 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl_server_port": { 2024-03-28T22:20:44.779 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl_server_port", 2024-03-28T22:20:44.779 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.779 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.779 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.779 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8443", 2024-03-28T22:20:44.779 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.779 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.779 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.780 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.780 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.780 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.780 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.780 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-03-28T22:20:44.780 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-03-28T22:20:44.780 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.781 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.781 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.781 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "redirect", 2024-03-28T22:20:44.781 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.781 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.781 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.781 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.781 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect" 2024-03-28T22:20:44.782 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.782 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.782 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.782 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.782 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.782 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-03-28T22:20:44.782 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-03-28T22:20:44.783 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.783 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.783 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.783 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-03-28T22:20:44.783 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-03-28T22:20:44.783 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-03-28T22:20:44.783 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.783 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.784 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.784 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.784 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.784 INFO:teuthology.orchestra.run.smithi016.stdout: "url_prefix": { 2024-03-28T22:20:44.784 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url_prefix", 2024-03-28T22:20:44.784 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.784 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.785 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.785 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.785 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.785 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.785 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.785 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.785 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.785 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.786 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.786 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.786 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.786 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "devicehealth", 2024-03-28T22:20:44.786 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.786 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.786 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.787 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_monitoring": { 2024-03-28T22:20:44.787 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_monitoring", 2024-03-28T22:20:44.787 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.787 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.787 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.787 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.787 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.787 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.788 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.788 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "monitor device health metrics", 2024-03-28T22:20:44.788 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.788 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.788 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.788 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.788 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.788 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.789 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.789 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.789 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.789 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.789 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.789 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.789 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.789 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.790 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.790 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.790 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.790 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.790 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.790 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.790 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.790 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.790 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.791 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.791 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.791 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.791 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.791 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.791 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.791 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.792 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.792 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.792 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.792 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.792 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.792 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.792 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.792 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.793 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.793 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.793 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.793 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.793 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.793 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.793 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.793 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.794 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.794 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.794 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.794 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.794 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.794 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.794 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.794 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.794 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.795 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.795 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.795 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.795 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.795 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.795 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.795 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.796 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.796 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.796 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.796 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.796 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.796 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.796 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.796 INFO:teuthology.orchestra.run.smithi016.stdout: "mark_out_threshold": { 2024-03-28T22:20:44.797 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mark_out_threshold", 2024-03-28T22:20:44.797 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:44.797 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.797 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.797 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2419200", 2024-03-28T22:20:44.797 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.797 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.797 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.797 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-28T22:20:44.798 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.798 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.798 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.798 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": { 2024-03-28T22:20:44.798 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_name", 2024-03-28T22:20:44.798 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.798 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.799 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.799 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "device_health_metrics", 2024-03-28T22:20:44.799 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.799 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.799 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.799 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-28T22:20:44.799 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.799 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.800 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.800 INFO:teuthology.orchestra.run.smithi016.stdout: "retention_period": { 2024-03-28T22:20:44.800 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retention_period", 2024-03-28T22:20:44.800 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:44.800 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.800 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.800 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15552000", 2024-03-28T22:20:44.800 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.801 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.801 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.801 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain device health metrics", 2024-03-28T22:20:44.801 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.801 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.801 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.801 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_frequency": { 2024-03-28T22:20:44.802 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_frequency", 2024-03-28T22:20:44.802 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:44.802 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.802 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.802 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-03-28T22:20:44.802 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.802 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.802 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.802 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-28T22:20:44.803 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.803 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.803 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.803 INFO:teuthology.orchestra.run.smithi016.stdout: "self_heal": { 2024-03-28T22:20:44.803 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "self_heal", 2024-03-28T22:20:44.803 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.803 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.805 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.805 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.805 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.805 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.806 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.806 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-28T22:20:44.806 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.806 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.806 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.806 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-03-28T22:20:44.806 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-03-28T22:20:44.807 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:44.807 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.807 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.807 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-03-28T22:20:44.807 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.807 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.807 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.807 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and check device health", 2024-03-28T22:20:44.808 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.808 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.808 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.808 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_threshold": { 2024-03-28T22:20:44.808 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_threshold", 2024-03-28T22:20:44.808 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:44.808 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.809 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.809 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7257600", 2024-03-28T22:20:44.809 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.809 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.809 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.809 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-28T22:20:44.809 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.809 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.809 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.810 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.810 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.810 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "diskprediction_local", 2024-03-28T22:20:44.810 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.810 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.810 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.811 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.811 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.811 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.811 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.811 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.811 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.811 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.811 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.812 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.812 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.812 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.812 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.812 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.812 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.812 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.812 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.812 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.813 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.813 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.813 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.813 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.813 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.813 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.813 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.813 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.814 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.814 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.814 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.814 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.814 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.814 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.814 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.814 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.815 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.815 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.815 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.815 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.815 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.815 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.815 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.815 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.816 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.816 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.816 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.816 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.816 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.816 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.816 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.816 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.817 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.817 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.817 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.817 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.817 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.817 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.817 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.818 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.818 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.818 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.818 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.818 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.818 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.818 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.818 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.818 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.819 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.819 INFO:teuthology.orchestra.run.smithi016.stdout: "predict_interval": { 2024-03-28T22:20:44.819 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predict_interval", 2024-03-28T22:20:44.819 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.819 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.819 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.819 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-03-28T22:20:44.819 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.820 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.820 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.820 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.820 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.820 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.820 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.820 INFO:teuthology.orchestra.run.smithi016.stdout: "predictor_model": { 2024-03-28T22:20:44.821 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predictor_model", 2024-03-28T22:20:44.821 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.821 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.821 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.821 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "prophetstor", 2024-03-28T22:20:44.821 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.821 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.821 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.821 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.822 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.822 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.822 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.822 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-03-28T22:20:44.822 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-03-28T22:20:44.822 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.822 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.823 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.823 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-03-28T22:20:44.823 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.823 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.823 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.823 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.823 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.823 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.823 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.824 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.824 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.824 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "influx", 2024-03-28T22:20:44.824 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": false, 2024-03-28T22:20:44.824 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "influxdb python module not found", 2024-03-28T22:20:44.824 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.825 INFO:teuthology.orchestra.run.smithi016.stdout: "batch_size": { 2024-03-28T22:20:44.825 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "batch_size", 2024-03-28T22:20:44.825 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.825 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.825 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.825 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5000", 2024-03-28T22:20:44.825 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.825 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.825 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.826 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-28T22:20:44.826 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.826 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.826 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.826 INFO:teuthology.orchestra.run.smithi016.stdout: "database": { 2024-03-28T22:20:44.826 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "database", 2024-03-28T22:20:44.826 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.827 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.827 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.827 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "ceph", 2024-03-28T22:20:44.827 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.827 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.827 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.827 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-28T22:20:44.827 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.827 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.828 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.828 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": { 2024-03-28T22:20:44.828 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hostname", 2024-03-28T22:20:44.828 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.828 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.828 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.828 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.829 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.829 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.829 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.829 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server hostname", 2024-03-28T22:20:44.829 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.829 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.829 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.829 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-03-28T22:20:44.830 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-03-28T22:20:44.830 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:44.830 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.830 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.830 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "30", 2024-03-28T22:20:44.830 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "5", 2024-03-28T22:20:44.830 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.830 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.831 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-28T22:20:44.831 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.831 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.831 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.831 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.831 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.831 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.831 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.832 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.832 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.832 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.832 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.832 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.832 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.833 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.833 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.833 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.833 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.833 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.833 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.833 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.833 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.834 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.834 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.834 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.834 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.834 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.834 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.834 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.834 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.835 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.835 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.835 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.835 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.835 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.835 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.835 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.836 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.836 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.836 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.836 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.836 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.836 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.836 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.836 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.837 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.837 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.837 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.837 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.837 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.837 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.837 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.837 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.837 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.838 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.838 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.838 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.838 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.838 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.838 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.839 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.839 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.839 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.839 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.839 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.839 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.839 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.839 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.840 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.840 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.840 INFO:teuthology.orchestra.run.smithi016.stdout: "password": { 2024-03-28T22:20:44.840 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "password", 2024-03-28T22:20:44.840 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.840 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.840 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.840 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.841 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.841 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.841 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.841 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "password of InfluxDB server user", 2024-03-28T22:20:44.841 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.841 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.841 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.842 INFO:teuthology.orchestra.run.smithi016.stdout: "port": { 2024-03-28T22:20:44.842 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "port", 2024-03-28T22:20:44.842 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.842 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.842 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.842 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8086", 2024-03-28T22:20:44.842 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.842 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.842 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.843 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server port", 2024-03-28T22:20:44.843 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.843 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.843 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.843 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-03-28T22:20:44.843 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-03-28T22:20:44.843 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.843 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.844 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.844 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "false", 2024-03-28T22:20:44.844 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.844 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.844 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.844 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T22:20:44.844 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.844 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.845 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.845 INFO:teuthology.orchestra.run.smithi016.stdout: "threads": { 2024-03-28T22:20:44.845 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threads", 2024-03-28T22:20:44.845 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.845 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.845 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.845 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-03-28T22:20:44.845 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-03-28T22:20:44.846 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "32", 2024-03-28T22:20:44.846 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.846 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-28T22:20:44.846 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.846 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.846 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.846 INFO:teuthology.orchestra.run.smithi016.stdout: "username": { 2024-03-28T22:20:44.847 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "username", 2024-03-28T22:20:44.847 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.847 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.847 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.847 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.847 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.847 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.847 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.847 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "username of InfluxDB server user", 2024-03-28T22:20:44.848 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.848 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.848 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.848 INFO:teuthology.orchestra.run.smithi016.stdout: "verify_ssl": { 2024-03-28T22:20:44.848 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "verify_ssl", 2024-03-28T22:20:44.848 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.848 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.849 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.849 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "true", 2024-03-28T22:20:44.849 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.849 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.849 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.849 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T22:20:44.849 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.849 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.850 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.850 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.850 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.850 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "insights", 2024-03-28T22:20:44.850 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.850 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.851 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.851 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.851 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.851 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.851 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.851 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.851 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.851 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.851 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.852 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.852 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.852 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.852 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.852 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.852 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.852 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.852 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.853 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.853 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.853 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.853 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.853 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.853 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.853 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.853 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.854 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.854 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.854 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.854 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.854 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.854 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.854 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.854 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.855 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.855 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.855 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.855 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.855 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.855 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.855 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.855 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.856 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.856 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.856 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.856 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.856 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.856 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.856 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.856 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.857 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.857 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.857 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.857 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.857 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.857 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.857 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.858 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.858 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.858 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.858 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.858 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.858 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.858 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.858 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.858 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.859 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.859 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.859 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.859 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.859 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.859 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "iostat", 2024-03-28T22:20:44.859 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.859 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.860 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.860 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.860 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.860 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.860 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.860 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.860 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.860 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.861 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.861 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.861 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.861 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.861 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.861 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.861 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.861 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.862 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.862 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.862 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.862 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.862 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.862 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.862 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.863 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.863 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.863 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.863 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.863 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.863 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.863 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.863 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.863 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.864 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.864 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.864 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.864 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.864 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.864 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.864 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.864 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.865 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.865 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.865 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.865 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.865 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.865 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.865 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.865 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.865 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.866 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.866 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.866 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.866 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.866 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.866 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.866 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.867 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.867 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.867 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.867 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.867 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.867 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.867 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.867 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.867 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.868 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.868 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.868 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.868 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.868 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.868 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.869 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.869 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "k8sevents", 2024-03-28T22:20:44.869 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.869 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.869 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.869 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_event_retention_days": { 2024-03-28T22:20:44.869 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ceph_event_retention_days", 2024-03-28T22:20:44.869 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.869 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.870 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.870 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7", 2024-03-28T22:20:44.870 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.870 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.870 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.870 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-28T22:20:44.870 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.870 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.871 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.871 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.871 INFO:teuthology.orchestra.run.smithi016.stdout: "config_check_secs": { 2024-03-28T22:20:44.871 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_check_secs", 2024-03-28T22:20:44.871 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.871 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.871 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.871 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-03-28T22:20:44.871 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "10", 2024-03-28T22:20:44.872 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.872 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.872 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-28T22:20:44.872 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.872 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.872 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.872 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.873 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.873 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.873 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.873 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.873 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.873 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.873 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.873 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.874 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.874 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.874 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.874 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.874 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.874 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.874 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.874 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.875 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.875 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.875 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.875 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.875 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.875 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.875 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.875 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.876 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.876 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.876 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.876 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.876 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.876 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.876 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.876 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.876 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.877 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.877 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.877 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.877 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.877 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.877 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.877 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.877 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.878 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.878 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.878 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.878 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.878 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.878 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.878 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.878 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.878 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.879 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.879 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.879 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.879 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.879 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.879 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.879 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.879 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.880 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.880 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.880 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.880 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.880 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.880 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.880 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.880 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.881 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.881 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.881 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.881 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.881 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "localpool", 2024-03-28T22:20:44.881 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.882 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.882 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.882 INFO:teuthology.orchestra.run.smithi016.stdout: "failure_domain": { 2024-03-28T22:20:44.882 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "failure_domain", 2024-03-28T22:20:44.882 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.882 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.882 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.882 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "host", 2024-03-28T22:20:44.883 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.883 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.883 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.883 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "failure domain for any created local pool", 2024-03-28T22:20:44.883 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-28T22:20:44.883 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.883 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.884 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.884 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.884 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.884 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.884 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.884 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.884 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.884 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.884 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.885 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.885 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.885 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.885 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.885 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.885 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.885 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.885 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.886 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.886 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.886 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.886 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.886 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.886 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.886 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.887 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.887 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.887 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.887 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.887 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.887 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.887 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.887 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.888 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.888 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.888 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.888 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.888 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.888 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.888 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.889 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.889 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.889 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.889 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.889 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.889 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.889 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.889 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.890 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.890 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.890 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.890 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.890 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.890 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.890 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.890 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.891 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.891 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.891 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.891 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.891 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.891 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.891 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.891 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.892 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.892 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.892 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.892 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": { 2024-03-28T22:20:44.892 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_size", 2024-03-28T22:20:44.892 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.892 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.893 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.893 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.893 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.893 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.893 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.893 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default min_size for any created local pool", 2024-03-28T22:20:44.893 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-28T22:20:44.893 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.894 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.894 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.894 INFO:teuthology.orchestra.run.smithi016.stdout: "num_rep": { 2024-03-28T22:20:44.894 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "num_rep", 2024-03-28T22:20:44.894 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.894 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.894 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.894 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3", 2024-03-28T22:20:44.894 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.895 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.895 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.895 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default replica count for any created local pool", 2024-03-28T22:20:44.895 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.895 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.895 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.895 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": { 2024-03-28T22:20:44.896 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_num", 2024-03-28T22:20:44.896 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.896 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.896 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.896 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "128", 2024-03-28T22:20:44.896 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.896 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.896 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.897 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default pg_num for any created local pool", 2024-03-28T22:20:44.897 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.897 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.897 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.897 INFO:teuthology.orchestra.run.smithi016.stdout: "prefix": { 2024-03-28T22:20:44.897 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prefix", 2024-03-28T22:20:44.897 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.897 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.898 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.898 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.898 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.898 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.898 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.898 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name prefix for any created local pool", 2024-03-28T22:20:44.898 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.898 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.899 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.899 INFO:teuthology.orchestra.run.smithi016.stdout: "subtree": { 2024-03-28T22:20:44.899 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "subtree", 2024-03-28T22:20:44.899 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.899 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.899 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.899 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "rack", 2024-03-28T22:20:44.900 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.900 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.900 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.900 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-28T22:20:44.900 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-28T22:20:44.900 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.900 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.900 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.900 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.901 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.901 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mds_autoscaler", 2024-03-28T22:20:44.901 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.901 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.901 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.901 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.901 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.902 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.902 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.902 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.902 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.902 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.902 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.902 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.902 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.902 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.903 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.903 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.903 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.903 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.903 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.903 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.903 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.903 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.904 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.904 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.904 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.904 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.904 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.904 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.904 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.904 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.905 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.905 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.905 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.905 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.905 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.905 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.905 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.906 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.906 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.906 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.906 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.906 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.906 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.906 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.906 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.907 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.907 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.907 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.907 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.907 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.907 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.907 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.907 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.907 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.908 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.908 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.908 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.908 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.908 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.908 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.908 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.908 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.909 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.909 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.909 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.909 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.909 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.909 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.909 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.909 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.910 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.910 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirroring", 2024-03-28T22:20:44.910 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.910 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.910 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.910 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.910 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.911 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.911 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.911 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.911 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.911 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.911 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.911 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.911 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.912 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.912 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.912 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.912 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.912 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.912 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.912 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.912 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.913 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.913 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.913 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.913 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.913 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.913 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.913 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.913 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.914 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.914 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.914 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.914 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.914 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.914 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.914 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.915 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.915 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.915 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.915 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.915 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.915 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.915 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.915 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.916 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.916 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.916 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.916 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.916 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.916 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.916 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.916 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.917 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.917 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.917 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.917 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.917 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.917 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.918 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.918 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.918 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.918 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.918 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.918 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.918 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.918 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.918 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.919 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.919 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.919 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.919 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.919 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "nfs", 2024-03-28T22:20:44.919 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.920 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.920 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.920 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.920 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.920 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.920 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.920 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.920 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.921 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.921 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.921 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.921 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.921 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.921 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.921 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.921 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.922 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.922 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.922 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.922 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.922 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.922 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.922 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.923 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.923 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.923 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.923 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.923 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.923 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.923 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.923 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.923 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.924 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.924 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.924 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.924 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.924 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.924 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.924 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.925 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.925 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.925 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.925 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.925 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.925 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.925 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.925 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.925 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.926 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.926 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.926 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.926 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.926 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.926 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.926 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.927 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.927 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.927 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.927 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.927 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.927 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.927 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.927 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.927 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.928 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.928 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.928 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.928 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.928 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.928 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.928 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.929 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-03-28T22:20:44.929 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.929 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.929 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.929 INFO:teuthology.orchestra.run.smithi016.stdout: "fail_fs": { 2024-03-28T22:20:44.929 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "fail_fs", 2024-03-28T22:20:44.929 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.929 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.930 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.930 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.930 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.930 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.930 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.930 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-28T22:20:44.930 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.930 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.931 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.931 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.931 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.931 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.931 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.931 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.931 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.931 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.932 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.932 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.932 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.932 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.932 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.932 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.932 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.932 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.932 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.933 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.933 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.933 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.933 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.933 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.933 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.933 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.934 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.934 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.934 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.934 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.934 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.934 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.935 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.935 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.935 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.935 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.935 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.935 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.935 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.936 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.936 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.936 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.936 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.936 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.936 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.936 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.936 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.936 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.937 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.937 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.937 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.937 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.937 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.937 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.937 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.937 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.938 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.938 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.938 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.938 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.938 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.938 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.938 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.938 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.939 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.939 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.939 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.939 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.939 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.939 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator": { 2024-03-28T22:20:44.939 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-03-28T22:20:44.939 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.940 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.940 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.940 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.940 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.940 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.940 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.940 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm", 2024-03-28T22:20:44.940 INFO:teuthology.orchestra.run.smithi016.stdout: "rook", 2024-03-28T22:20:44.941 INFO:teuthology.orchestra.run.smithi016.stdout: "test_orchestrator" 2024-03-28T22:20:44.941 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.941 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Orchestrator backend", 2024-03-28T22:20:44.941 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.941 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.941 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.941 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.941 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.942 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_perf_query", 2024-03-28T22:20:44.942 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.942 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.942 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.942 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.942 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.943 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.943 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.943 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.943 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.943 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.943 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.943 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.943 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.944 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.944 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.944 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.944 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.944 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.944 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.944 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.945 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.945 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.945 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.945 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.945 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.945 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.945 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.945 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.946 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.946 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.946 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.946 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.946 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.946 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.946 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.946 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.947 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.947 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.947 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.947 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.947 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.947 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.948 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.948 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.948 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.948 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.948 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.948 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.948 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.948 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.949 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.949 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.949 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.949 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.949 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.949 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.949 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.950 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.950 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.950 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.950 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.950 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.950 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.950 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.950 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.951 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.951 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.951 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.951 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.951 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.951 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.952 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_support", 2024-03-28T22:20:44.952 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.952 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.952 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.952 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.952 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.952 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.952 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.953 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.953 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.953 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.953 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.953 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.953 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.953 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.953 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.953 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.954 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.954 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.954 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.954 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.954 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.954 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.954 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.955 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.955 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.955 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.955 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.955 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.955 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.955 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.955 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.956 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.956 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.956 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.956 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.956 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.956 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.956 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.956 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.957 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.957 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.957 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.957 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.957 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.957 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.957 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.957 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.958 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.958 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.958 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.958 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.958 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.958 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.958 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.958 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.959 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.959 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.959 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.959 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.959 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.959 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.959 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.960 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.960 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.960 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.960 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.960 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.960 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.960 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.960 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.961 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.961 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_autoscaler", 2024-03-28T22:20:44.961 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.961 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.961 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.961 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.962 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.962 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.962 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.962 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.962 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.962 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.962 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.962 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.963 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.963 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.963 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.963 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.963 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.963 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.963 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.963 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.964 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.964 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.964 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.964 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.964 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.964 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.964 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.964 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.965 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.965 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.965 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.965 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.965 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.965 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.965 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.965 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.966 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.966 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.966 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.966 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.966 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.966 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.966 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.966 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.967 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.967 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.967 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.967 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.967 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.967 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.967 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.967 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.967 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.968 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.968 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.968 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.968 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.968 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.968 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.968 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.968 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.969 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.969 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.969 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.969 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.969 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.969 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.969 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.969 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.970 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-03-28T22:20:44.970 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-03-28T22:20:44.970 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:44.970 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.970 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.970 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-03-28T22:20:44.970 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.971 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.971 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.971 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.971 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.971 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.971 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.971 INFO:teuthology.orchestra.run.smithi016.stdout: "threshold": { 2024-03-28T22:20:44.972 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threshold", 2024-03-28T22:20:44.972 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-03-28T22:20:44.972 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.972 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.972 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-03-28T22:20:44.972 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1.0", 2024-03-28T22:20:44.972 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.972 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.972 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "scaling threshold", 2024-03-28T22:20:44.973 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-28T22:20:44.973 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.973 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.973 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.973 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.973 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "progress", 2024-03-28T22:20:44.973 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.974 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.974 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.974 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_pg_recovery_event": { 2024-03-28T22:20:44.974 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_pg_recovery_event", 2024-03-28T22:20:44.974 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.974 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.974 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.975 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.975 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.975 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.975 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.975 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-28T22:20:44.975 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.975 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.975 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.976 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-03-28T22:20:44.976 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-03-28T22:20:44.976 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.976 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.976 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.976 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.976 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.977 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.977 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.977 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.977 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.977 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.977 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.978 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.978 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.978 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.978 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.978 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.978 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.978 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.978 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.979 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.979 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.979 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.979 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.979 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.979 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.979 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.979 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.979 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.980 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.980 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.980 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.980 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.980 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.980 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.980 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.980 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.981 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.981 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.981 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.981 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.981 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.981 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.981 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.981 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.982 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.982 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.982 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.982 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.982 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.982 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.982 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.982 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.982 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.983 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.983 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.983 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.983 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.983 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.983 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.983 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.984 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.984 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.984 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.984 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.984 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.984 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.984 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.984 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.985 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.985 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.985 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.985 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.985 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.985 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.985 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.985 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.985 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.986 INFO:teuthology.orchestra.run.smithi016.stdout: "max_completed_events": { 2024-03-28T22:20:44.986 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_completed_events", 2024-03-28T22:20:44.986 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:44.986 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.986 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.986 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "50", 2024-03-28T22:20:44.986 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.987 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.987 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.987 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "number of past completed events to remember", 2024-03-28T22:20:44.987 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.987 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.987 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.987 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-03-28T22:20:44.987 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-03-28T22:20:44.987 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:44.988 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.988 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.988 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-03-28T22:20:44.988 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.988 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.988 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.988 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long the module is going to sleep", 2024-03-28T22:20:44.988 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.989 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.989 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.989 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.989 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:44.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:44.989 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus", 2024-03-28T22:20:44.989 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:44.989 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:44.989 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:44.990 INFO:teuthology.orchestra.run.smithi016.stdout: "cache": { 2024-03-28T22:20:44.990 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cache", 2024-03-28T22:20:44.990 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.990 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.990 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:44.990 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.990 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.991 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.991 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.991 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.991 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.991 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.991 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.991 INFO:teuthology.orchestra.run.smithi016.stdout: "exclude_perf_counters": { 2024-03-28T22:20:44.991 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "exclude_perf_counters", 2024-03-28T22:20:44.991 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.992 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.992 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.992 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:44.992 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.992 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.992 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.992 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-28T22:20:44.992 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-28T22:20:44.992 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.993 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.993 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:44.993 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:44.993 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.993 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.993 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.993 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:44.993 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.993 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.994 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.994 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.994 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.994 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.994 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.994 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.994 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.994 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.994 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.995 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.995 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.995 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.995 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:44.995 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:44.995 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.995 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.995 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.996 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:44.996 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.996 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.996 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:44.996 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.996 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.996 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.996 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.997 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:44.997 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:44.997 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:44.997 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.997 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.997 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:44.997 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:44.997 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:44.997 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:44.998 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:44.998 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:44.998 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:44.998 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:44.998 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:44.998 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:44.998 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:44.998 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:44.998 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:44.999 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:44.999 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:44.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:44.999 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:44.999 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:44.999 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:44.999 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:44.999 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:44.999 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:45.000 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.000 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.000 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.000 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.000 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.000 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.000 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.001 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools": { 2024-03-28T22:20:45.001 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools", 2024-03-28T22:20:45.001 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.001 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.001 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.001 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:45.001 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.002 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.002 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.002 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.002 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.002 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.002 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.002 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-28T22:20:45.003 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-28T22:20:45.003 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:45.003 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.003 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.003 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "300", 2024-03-28T22:20:45.003 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.003 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.003 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.004 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.004 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.004 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.004 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.004 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_interval": { 2024-03-28T22:20:45.004 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_interval", 2024-03-28T22:20:45.004 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-03-28T22:20:45.004 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.005 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.005 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15.0", 2024-03-28T22:20:45.005 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.005 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.005 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.005 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.005 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.005 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.005 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.006 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-03-28T22:20:45.006 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-03-28T22:20:45.006 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.006 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.006 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.006 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-03-28T22:20:45.006 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.007 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.007 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.007 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-28T22:20:45.007 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.007 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.007 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.007 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-03-28T22:20:45.009 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-03-28T22:20:45.009 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:45.009 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.009 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.009 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "9283", 2024-03-28T22:20:45.009 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.009 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.010 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.010 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-28T22:20:45.010 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.010 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.010 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.010 INFO:teuthology.orchestra.run.smithi016.stdout: "stale_cache_strategy": { 2024-03-28T22:20:45.010 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stale_cache_strategy", 2024-03-28T22:20:45.010 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.011 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.011 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.011 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "log", 2024-03-28T22:20:45.011 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.011 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.011 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.011 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.011 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.011 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.012 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.012 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-03-28T22:20:45.012 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-03-28T22:20:45.012 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.012 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.012 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.012 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "default", 2024-03-28T22:20:45.012 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.013 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.013 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:45.013 INFO:teuthology.orchestra.run.smithi016.stdout: "default", 2024-03-28T22:20:45.013 INFO:teuthology.orchestra.run.smithi016.stdout: "error" 2024-03-28T22:20:45.013 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:45.013 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.013 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.013 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.013 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.014 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-03-28T22:20:45.014 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-03-28T22:20:45.014 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:45.014 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.014 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.014 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-03-28T22:20:45.014 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-03-28T22:20:45.014 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-03-28T22:20:45.014 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.015 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.015 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.015 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.015 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.015 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:45.015 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:45.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:45.015 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_support", 2024-03-28T22:20:45.016 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:45.016 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:45.016 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:45.016 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:45.016 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:45.016 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.016 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.016 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.016 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:45.017 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.017 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.017 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:45.017 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:45.017 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:45.017 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:45.017 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:45.017 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:45.017 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:45.017 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:45.018 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.018 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.018 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.018 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.018 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:45.018 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:45.018 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.019 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.019 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.019 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:45.019 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.019 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.019 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.019 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.019 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.020 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.020 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.020 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:45.020 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:45.020 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.020 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.020 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.021 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:45.021 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.021 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.021 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:45.021 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:45.021 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:45.021 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:45.021 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:45.022 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:45.022 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:45.022 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:45.022 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.022 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.022 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.022 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.022 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:45.023 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:45.023 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.023 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.023 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.023 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:45.023 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.023 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.023 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.023 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.024 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.024 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.024 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.024 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_snap_create": { 2024-03-28T22:20:45.024 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_snap_create", 2024-03-28T22:20:45.024 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:45.024 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.025 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.025 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-03-28T22:20:45.025 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.025 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.025 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.025 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.025 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.025 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.025 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.026 INFO:teuthology.orchestra.run.smithi016.stdout: "mirror_snapshot_schedule": { 2024-03-28T22:20:45.026 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirror_snapshot_schedule", 2024-03-28T22:20:45.026 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.026 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.026 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.026 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:45.027 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.027 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.027 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.027 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.027 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.027 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.027 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.027 INFO:teuthology.orchestra.run.smithi016.stdout: "trash_purge_schedule": { 2024-03-28T22:20:45.028 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "trash_purge_schedule", 2024-03-28T22:20:45.028 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.028 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.028 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.028 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:45.028 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.028 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.028 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.028 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.029 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.029 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.029 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.029 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:45.029 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:45.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:45.029 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "restful", 2024-03-28T22:20:45.029 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:45.029 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:45.030 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:45.030 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_auth": { 2024-03-28T22:20:45.030 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_auth", 2024-03-28T22:20:45.030 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.030 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.030 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.030 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:45.030 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.031 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.031 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.031 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.031 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.031 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.031 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.031 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-03-28T22:20:45.031 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-03-28T22:20:45.031 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.032 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.032 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.032 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:45.032 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.032 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.032 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.032 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.032 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.033 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.033 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.033 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:45.033 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:45.033 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.033 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.033 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.033 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:45.034 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.034 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.034 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:45.034 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:45.034 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:45.034 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:45.034 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:45.034 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:45.035 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:45.035 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:45.035 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.035 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.035 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.035 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.035 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:45.035 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:45.036 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.036 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.036 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.036 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:45.036 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.036 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.036 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.036 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.036 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.037 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.037 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.037 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:45.037 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:45.037 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.037 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.037 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.037 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:45.038 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.038 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.038 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:45.038 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:45.038 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:45.038 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:45.038 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:45.038 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:45.039 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:45.039 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:45.039 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.039 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.039 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.039 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.039 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:45.039 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:45.040 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.040 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.040 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.040 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:45.040 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.040 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.040 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.040 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.040 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.041 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.041 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.041 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-03-28T22:20:45.041 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-03-28T22:20:45.041 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.041 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.041 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.042 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:45.042 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.042 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.042 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.042 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.042 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.042 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.042 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.043 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-03-28T22:20:45.043 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-03-28T22:20:45.043 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.043 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.043 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.043 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:45.043 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.043 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.044 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.044 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.044 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.044 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.044 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.044 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:45.044 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:45.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:45.045 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rgw", 2024-03-28T22:20:45.045 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:45.045 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:45.045 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:45.045 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:45.045 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:45.045 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.045 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.046 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.046 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:45.046 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.046 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.046 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:45.046 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:45.046 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:45.046 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:45.047 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:45.047 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:45.047 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:45.047 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:45.047 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.047 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.047 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.047 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.048 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:45.048 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:45.048 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.048 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.048 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.048 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:45.048 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.049 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.049 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.049 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.049 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.049 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.049 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.049 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:45.050 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:45.050 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.050 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.050 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.050 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:45.050 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.050 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.050 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:45.051 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:45.051 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:45.051 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:45.051 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:45.051 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:45.051 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:45.051 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:45.051 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.052 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.052 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.052 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.052 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:45.052 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:45.052 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.052 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.053 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.053 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:45.053 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.053 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.053 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.053 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.053 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.053 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.053 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.054 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:45.054 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:45.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:45.054 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rook", 2024-03-28T22:20:45.054 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:45.054 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:45.055 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:45.055 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:45.055 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:45.055 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.055 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.055 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.055 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:45.055 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.056 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.056 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:45.056 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:45.056 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:45.056 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:45.056 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:45.056 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:45.056 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:45.056 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:45.057 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.057 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.057 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.057 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.057 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:45.057 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:45.057 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.058 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.058 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.058 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:45.058 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.058 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.058 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.058 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.058 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.059 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.059 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.059 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:45.059 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:45.059 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.059 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.059 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.060 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:45.060 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.060 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.060 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:45.060 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:45.060 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:45.060 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:45.060 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:45.061 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:45.061 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:45.061 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:45.061 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.061 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.061 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.061 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.061 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:45.062 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:45.062 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.062 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.062 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.062 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:45.062 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.062 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.062 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.063 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.063 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.063 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.063 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.063 INFO:teuthology.orchestra.run.smithi016.stdout: "storage_class": { 2024-03-28T22:20:45.063 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "storage_class", 2024-03-28T22:20:45.063 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.063 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.064 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.064 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "local", 2024-03-28T22:20:45.064 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.064 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.064 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.064 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-28T22:20:45.064 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.064 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.064 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.064 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:45.065 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:45.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:45.065 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "selftest", 2024-03-28T22:20:45.065 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:45.065 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:45.065 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:45.065 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:45.066 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:45.066 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.066 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.066 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.066 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:45.066 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.066 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.066 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:45.066 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:45.067 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:45.067 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:45.067 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:45.067 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:45.067 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:45.067 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:45.067 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.067 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.068 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.068 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.068 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:45.068 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:45.068 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.068 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.068 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.068 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:45.069 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.069 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.069 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.069 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.069 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.069 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.069 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.069 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:45.069 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:45.070 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.070 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.070 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.070 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:45.070 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.070 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.070 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:45.070 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:45.071 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:45.071 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:45.071 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:45.071 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:45.071 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:45.071 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:45.071 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.071 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.071 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.072 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.072 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:45.072 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:45.072 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.072 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.072 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.072 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:45.073 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.073 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.073 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.073 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.073 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.073 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.073 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.074 INFO:teuthology.orchestra.run.smithi016.stdout: "roption1": { 2024-03-28T22:20:45.074 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption1", 2024-03-28T22:20:45.074 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.074 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.074 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.074 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:45.074 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.075 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.075 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.075 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.075 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.075 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.075 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.075 INFO:teuthology.orchestra.run.smithi016.stdout: "roption2": { 2024-03-28T22:20:45.075 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption2", 2024-03-28T22:20:45.076 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.076 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.076 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.076 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "xyz", 2024-03-28T22:20:45.076 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.076 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.076 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.076 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.076 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.077 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.077 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.077 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption1": { 2024-03-28T22:20:45.077 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption1", 2024-03-28T22:20:45.077 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.077 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.077 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.078 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:45.078 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.078 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.078 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.078 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.078 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.078 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.079 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.079 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption2": { 2024-03-28T22:20:45.079 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption2", 2024-03-28T22:20:45.079 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:45.079 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.079 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.079 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:45.080 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.080 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.080 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.080 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.080 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.080 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.080 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.081 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption3": { 2024-03-28T22:20:45.081 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption3", 2024-03-28T22:20:45.081 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-03-28T22:20:45.081 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.081 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.081 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:45.081 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.081 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.082 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.082 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.082 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.082 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.082 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.082 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption4": { 2024-03-28T22:20:45.082 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption4", 2024-03-28T22:20:45.083 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.083 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.083 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.083 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:45.083 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.083 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.083 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.083 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.084 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.084 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.084 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.084 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption5": { 2024-03-28T22:20:45.084 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption5", 2024-03-28T22:20:45.084 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.084 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.085 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.085 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:45.085 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.085 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.085 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.085 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.085 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.085 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.086 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.086 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption6": { 2024-03-28T22:20:45.086 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption6", 2024-03-28T22:20:45.086 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.086 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.086 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.086 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:45.087 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.087 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.087 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.087 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.087 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.087 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.087 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.088 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption7": { 2024-03-28T22:20:45.088 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption7", 2024-03-28T22:20:45.088 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:45.088 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.088 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.088 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:45.088 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-03-28T22:20:45.088 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "42", 2024-03-28T22:20:45.088 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.089 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.089 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.089 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.089 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.089 INFO:teuthology.orchestra.run.smithi016.stdout: "testkey": { 2024-03-28T22:20:45.089 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testkey", 2024-03-28T22:20:45.089 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.089 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.090 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.090 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:45.090 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.090 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.090 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.090 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.090 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.090 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.090 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.091 INFO:teuthology.orchestra.run.smithi016.stdout: "testlkey": { 2024-03-28T22:20:45.091 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testlkey", 2024-03-28T22:20:45.091 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.091 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.091 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.091 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:45.091 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.091 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.092 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.092 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.092 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.092 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.092 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.092 INFO:teuthology.orchestra.run.smithi016.stdout: "testnewline": { 2024-03-28T22:20:45.092 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testnewline", 2024-03-28T22:20:45.092 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.092 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.093 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.093 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:45.093 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.093 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.093 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.093 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.093 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.093 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.093 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.094 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:45.094 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:45.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:45.094 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snap_schedule", 2024-03-28T22:20:45.094 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:45.094 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:45.094 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:45.094 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_m_granularity": { 2024-03-28T22:20:45.095 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_m_granularity", 2024-03-28T22:20:45.095 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.095 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.095 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.095 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:45.095 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.095 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.095 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.095 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow minute scheduled snapshots", 2024-03-28T22:20:45.096 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.096 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.096 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.096 INFO:teuthology.orchestra.run.smithi016.stdout: "dump_on_update": { 2024-03-28T22:20:45.096 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dump_on_update", 2024-03-28T22:20:45.096 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.096 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.096 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.096 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:45.097 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.097 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.097 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.097 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "dump database to debug log on update", 2024-03-28T22:20:45.097 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.097 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.097 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.097 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:45.098 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:45.098 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.098 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.098 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.098 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:45.098 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.098 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.098 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:45.098 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:45.099 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:45.099 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:45.099 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:45.099 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:45.099 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:45.099 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:45.099 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.099 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.099 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.100 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.100 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:45.100 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:45.100 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.100 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.100 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.100 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:45.100 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.101 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.101 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.101 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.101 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.101 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.101 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.101 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:45.101 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:45.102 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.102 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.102 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.102 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:45.102 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.102 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.102 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:45.102 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:45.102 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:45.103 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:45.103 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:45.103 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:45.103 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:45.103 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:45.103 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.103 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.103 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.103 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.104 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:45.104 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:45.104 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.104 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.104 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.104 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:45.104 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.105 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.105 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.105 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.105 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.105 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.105 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.105 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:45.105 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:45.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:45.106 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stats", 2024-03-28T22:20:45.106 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:45.106 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:45.106 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:45.106 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:45.106 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:45.106 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.106 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.107 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.107 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:45.107 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.107 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.107 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:45.107 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:45.107 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:45.107 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:45.107 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:45.108 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:45.108 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:45.108 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:45.108 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.108 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.108 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.108 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.109 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:45.109 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:45.109 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.109 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.109 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.109 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:45.109 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.109 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.110 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.110 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.110 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.110 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.110 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.110 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:45.110 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:45.110 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.111 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.111 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.111 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:45.111 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.111 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.111 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:45.111 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:45.111 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:45.111 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:45.112 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:45.112 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:45.112 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:45.112 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:45.112 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.112 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.112 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.112 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.113 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:45.113 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:45.113 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.113 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.113 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.113 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:45.113 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.113 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.113 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.114 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.114 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.114 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.114 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.114 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:45.114 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:45.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.114 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:45.114 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "status", 2024-03-28T22:20:45.115 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:45.115 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:45.115 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:45.115 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:45.115 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:45.115 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.115 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.115 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.115 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:45.116 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.116 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.116 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:45.116 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:45.116 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:45.116 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:45.116 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:45.116 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:45.117 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:45.117 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:45.117 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.117 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.117 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.117 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.117 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:45.117 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:45.118 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.118 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.118 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.118 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:45.118 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.118 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.118 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.118 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.118 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.119 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.119 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.119 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:45.119 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:45.119 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.119 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.119 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.119 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:45.120 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.120 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.120 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:45.120 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:45.120 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:45.120 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:45.120 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:45.120 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:45.120 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:45.120 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:45.121 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.121 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.121 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.121 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.121 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:45.121 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:45.121 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.122 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.122 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.122 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:45.122 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.122 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.122 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.122 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.122 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.123 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.123 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.123 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:45.123 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:45.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:45.123 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telegraf", 2024-03-28T22:20:45.123 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:45.123 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:45.124 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:45.124 INFO:teuthology.orchestra.run.smithi016.stdout: "address": { 2024-03-28T22:20:45.124 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "address", 2024-03-28T22:20:45.124 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.124 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.124 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.124 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-28T22:20:45.124 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.124 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.125 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.125 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.125 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.125 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.125 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.125 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-03-28T22:20:45.125 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-03-28T22:20:45.126 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:45.126 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.126 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.126 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15", 2024-03-28T22:20:45.126 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.126 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.126 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.126 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.126 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.127 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.127 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.127 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:45.127 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:45.127 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.127 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.127 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.127 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:45.128 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.128 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.128 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:45.128 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:45.128 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:45.128 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:45.128 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:45.128 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:45.128 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:45.129 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:45.129 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.129 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.129 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.129 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.129 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:45.129 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:45.130 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.130 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.130 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.130 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:45.130 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.130 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.130 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.130 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.130 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.131 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.131 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.131 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:45.131 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:45.131 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.131 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.131 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.132 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:45.132 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.132 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.132 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:45.132 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:45.132 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:45.132 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:45.132 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:45.132 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:45.133 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:45.133 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:45.133 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.133 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.133 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.133 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.133 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:45.133 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:45.134 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.134 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.134 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.134 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:45.134 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.134 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.134 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.134 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.134 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.135 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.135 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.135 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:45.135 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:45.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.135 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:45.135 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telemetry", 2024-03-28T22:20:45.135 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:45.136 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:45.136 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:45.136 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_basic": { 2024-03-28T22:20:45.136 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_basic", 2024-03-28T22:20:45.136 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.136 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.136 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.136 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:45.137 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.137 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.137 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.137 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-28T22:20:45.137 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.137 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.137 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.138 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_crash": { 2024-03-28T22:20:45.138 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_crash", 2024-03-28T22:20:45.138 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.138 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.138 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.138 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:45.138 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.138 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.138 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.139 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-28T22:20:45.139 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.139 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.139 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.139 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_device": { 2024-03-28T22:20:45.139 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_device", 2024-03-28T22:20:45.139 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.140 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.140 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.140 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:45.140 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.140 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.140 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.140 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-28T22:20:45.140 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.141 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.141 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.141 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_ident": { 2024-03-28T22:20:45.141 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_ident", 2024-03-28T22:20:45.141 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.141 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.141 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.141 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:45.142 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.142 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.142 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.142 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-28T22:20:45.142 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.142 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.142 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.143 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_perf": { 2024-03-28T22:20:45.143 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_perf", 2024-03-28T22:20:45.143 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.143 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.143 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.143 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:45.143 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.143 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.143 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.143 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-28T22:20:45.144 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.144 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.144 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.144 INFO:teuthology.orchestra.run.smithi016.stdout: "contact": { 2024-03-28T22:20:45.144 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "contact", 2024-03-28T22:20:45.144 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.144 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.144 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.145 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:45.145 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.145 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.145 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.145 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.145 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.145 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.145 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.146 INFO:teuthology.orchestra.run.smithi016.stdout: "description": { 2024-03-28T22:20:45.146 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "description", 2024-03-28T22:20:45.146 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.146 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.146 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.146 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:45.146 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.146 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.146 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.147 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.147 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.147 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.147 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.147 INFO:teuthology.orchestra.run.smithi016.stdout: "device_url": { 2024-03-28T22:20:45.147 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_url", 2024-03-28T22:20:45.147 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.147 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.148 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.148 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-28T22:20:45.148 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.148 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.148 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.148 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.148 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.149 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.149 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.149 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-03-28T22:20:45.149 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-03-28T22:20:45.149 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.149 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.149 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.149 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:45.150 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.150 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.150 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.150 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.150 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.150 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.150 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.150 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-03-28T22:20:45.151 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-03-28T22:20:45.151 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:45.151 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.151 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.151 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "24", 2024-03-28T22:20:45.151 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "8", 2024-03-28T22:20:45.151 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.151 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.151 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.152 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.152 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.152 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.152 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.152 INFO:teuthology.orchestra.run.smithi016.stdout: "last_opt_revision": { 2024-03-28T22:20:45.152 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "last_opt_revision", 2024-03-28T22:20:45.152 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:45.152 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.152 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.152 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1", 2024-03-28T22:20:45.153 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.153 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.153 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.153 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.153 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.153 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.153 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.153 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard": { 2024-03-28T22:20:45.154 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard", 2024-03-28T22:20:45.154 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.154 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.154 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.154 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:45.154 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.154 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.154 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.154 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.155 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.155 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.155 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.155 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.155 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard_description": { 2024-03-28T22:20:45.155 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard_description", 2024-03-28T22:20:45.155 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.155 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.155 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.156 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:45.156 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.156 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.156 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.156 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.156 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.156 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.156 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.157 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:45.157 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:45.157 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.157 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.157 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.157 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:45.157 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.157 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.158 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:45.158 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:45.158 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:45.158 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:45.158 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:45.158 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:45.158 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:45.158 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:45.158 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.159 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.159 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.159 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.159 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:45.159 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:45.159 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.159 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.159 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.159 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:45.160 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.160 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.160 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.160 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.160 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.160 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.160 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.160 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:45.161 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:45.161 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.161 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.161 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.161 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:45.161 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.161 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.161 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:45.161 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:45.162 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:45.162 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:45.162 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:45.162 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:45.162 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:45.162 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:45.162 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.162 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.162 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.163 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.163 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:45.163 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:45.163 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.163 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.163 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.163 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:45.163 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.163 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.164 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.164 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.164 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.164 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.164 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.164 INFO:teuthology.orchestra.run.smithi016.stdout: "organization": { 2024-03-28T22:20:45.164 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "organization", 2024-03-28T22:20:45.164 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.165 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.165 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.165 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:45.165 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.165 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.165 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.165 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.165 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.165 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.165 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.166 INFO:teuthology.orchestra.run.smithi016.stdout: "proxy": { 2024-03-28T22:20:45.166 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "proxy", 2024-03-28T22:20:45.166 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.166 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.166 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.166 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:45.166 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.166 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.167 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.167 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.167 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.167 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.167 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.167 INFO:teuthology.orchestra.run.smithi016.stdout: "url": { 2024-03-28T22:20:45.167 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url", 2024-03-28T22:20:45.167 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.168 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.168 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.168 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-28T22:20:45.168 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.168 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.168 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.168 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.168 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.168 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.169 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.169 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:45.169 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:45.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:45.169 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "test_orchestrator", 2024-03-28T22:20:45.169 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:45.169 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:45.169 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:45.169 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:45.170 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:45.170 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.170 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.170 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.170 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:45.170 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.170 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.170 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:45.171 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:45.171 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:45.171 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:45.171 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:45.171 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:45.171 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:45.171 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:45.171 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.171 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.172 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.172 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.172 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:45.172 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:45.172 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.172 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.172 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.172 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:45.172 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.173 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.173 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.173 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.173 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.173 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.173 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.173 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:45.173 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:45.174 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.174 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.174 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.174 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:45.174 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.174 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.174 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:45.174 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:45.175 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:45.175 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:45.175 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:45.175 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:45.175 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:45.175 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:45.175 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.175 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.175 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.175 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.176 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:45.176 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:45.176 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.176 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.176 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.176 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:45.176 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.176 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.177 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.177 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.177 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.177 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.177 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.177 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:45.177 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:45.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:45.178 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "volumes", 2024-03-28T22:20:45.178 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:45.178 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:45.178 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:45.178 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:45.178 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:45.178 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.178 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.178 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.179 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:45.179 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.179 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.179 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:45.179 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:45.179 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:45.179 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:45.179 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:45.180 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:45.180 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:45.180 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:45.180 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.180 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.180 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.180 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.181 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:45.181 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:45.181 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.181 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.181 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.181 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:45.181 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.181 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.181 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.181 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.182 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.182 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.182 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.182 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:45.182 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:45.182 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.182 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.182 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.183 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:45.183 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.183 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.183 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:45.183 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:45.183 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:45.183 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:45.183 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:45.183 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:45.184 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:45.184 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:45.184 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.184 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.184 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.184 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.184 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:45.184 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:45.185 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.185 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.185 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.185 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:45.185 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.185 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.185 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.185 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.185 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.185 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.186 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.186 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_clones": { 2024-03-28T22:20:45.186 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_clones", 2024-03-28T22:20:45.186 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:45.186 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.186 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.186 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4", 2024-03-28T22:20:45.187 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.187 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.187 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.187 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-28T22:20:45.187 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.187 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.187 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.187 INFO:teuthology.orchestra.run.smithi016.stdout: "periodic_async_work": { 2024-03-28T22:20:45.188 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "periodic_async_work", 2024-03-28T22:20:45.188 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.188 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.188 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.188 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:45.188 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.188 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.188 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.189 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Periodically check for async work", 2024-03-28T22:20:45.189 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.189 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.189 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.189 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_delay": { 2024-03-28T22:20:45.189 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_delay", 2024-03-28T22:20:45.189 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:45.189 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.190 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.190 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-03-28T22:20:45.190 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.190 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.190 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.190 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-28T22:20:45.190 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.190 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.190 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.191 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_no_wait": { 2024-03-28T22:20:45.191 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_no_wait", 2024-03-28T22:20:45.191 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.191 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.191 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.191 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-03-28T22:20:45.191 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.192 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.192 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.192 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-28T22:20:45.192 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.192 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.192 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.192 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:45.192 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:45.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:45.193 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix", 2024-03-28T22:20:45.193 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-03-28T22:20:45.193 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-03-28T22:20:45.193 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-03-28T22:20:45.193 INFO:teuthology.orchestra.run.smithi016.stdout: "discovery_interval": { 2024-03-28T22:20:45.193 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "discovery_interval", 2024-03-28T22:20:45.193 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-03-28T22:20:45.194 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.194 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.194 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "100", 2024-03-28T22:20:45.194 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.194 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.194 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.194 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.194 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.194 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.195 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.195 INFO:teuthology.orchestra.run.smithi016.stdout: "identifier": { 2024-03-28T22:20:45.195 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "identifier", 2024-03-28T22:20:45.195 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.195 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.195 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.195 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:45.196 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.196 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.196 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.196 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.196 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.196 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.196 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.196 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-03-28T22:20:45.197 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-03-28T22:20:45.197 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-03-28T22:20:45.197 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.197 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.197 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-03-28T22:20:45.197 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.197 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.197 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.198 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.198 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.198 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.198 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.198 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-03-28T22:20:45.198 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-03-28T22:20:45.199 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.199 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.199 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.199 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:45.199 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.199 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.199 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:45.199 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:45.200 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:45.200 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:45.200 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:45.200 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:45.200 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:45.200 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:45.200 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.200 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.200 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.201 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.201 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-03-28T22:20:45.201 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-03-28T22:20:45.201 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.201 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.201 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.201 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:45.201 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.201 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.202 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.202 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.202 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.202 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.202 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.202 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-03-28T22:20:45.202 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-03-28T22:20:45.202 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.203 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.203 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.203 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-03-28T22:20:45.203 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.203 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.203 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-03-28T22:20:45.203 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-03-28T22:20:45.203 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-03-28T22:20:45.203 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-03-28T22:20:45.204 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-03-28T22:20:45.204 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-03-28T22:20:45.204 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-03-28T22:20:45.204 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:45.204 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.204 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.204 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.204 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.205 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-03-28T22:20:45.205 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-03-28T22:20:45.205 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-03-28T22:20:45.205 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.205 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-03-28T22:20:45.205 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-03-28T22:20:45.205 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.205 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.205 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.206 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.206 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.206 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.206 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.206 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_host": { 2024-03-28T22:20:45.206 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_host", 2024-03-28T22:20:45.206 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.207 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.207 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.207 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-03-28T22:20:45.207 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.207 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.207 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.207 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.207 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.207 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.208 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.208 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_port": { 2024-03-28T22:20:45.208 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_port", 2024-03-28T22:20:45.208 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-03-28T22:20:45.208 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.208 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.208 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10051", 2024-03-28T22:20:45.209 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.209 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.209 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.209 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.209 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.209 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.209 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.209 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_sender": { 2024-03-28T22:20:45.209 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_sender", 2024-03-28T22:20:45.210 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-03-28T22:20:45.210 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-03-28T22:20:45.210 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-03-28T22:20:45.210 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-28T22:20:45.210 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-03-28T22:20:45.210 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-03-28T22:20:45.210 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-03-28T22:20:45.210 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-03-28T22:20:45.210 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-03-28T22:20:45.211 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-03-28T22:20:45.213 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-03-28T22:20:45.213 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:45.213 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:45.213 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:45.213 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:45.214 INFO:teuthology.orchestra.run.smithi016.stdout: "services": { 2024-03-28T22:20:45.214 INFO:teuthology.orchestra.run.smithi016.stdout: "dashboard": "https://172.21.15.16:8443/" 2024-03-28T22:20:45.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.214 INFO:teuthology.orchestra.run.smithi016.stdout: "always_on_modules": { 2024-03-28T22:20:45.214 INFO:teuthology.orchestra.run.smithi016.stdout: "octopus": [ 2024-03-28T22:20:45.214 INFO:teuthology.orchestra.run.smithi016.stdout: "balancer", 2024-03-28T22:20:45.214 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-03-28T22:20:45.214 INFO:teuthology.orchestra.run.smithi016.stdout: "devicehealth", 2024-03-28T22:20:45.215 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator", 2024-03-28T22:20:45.215 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscaler", 2024-03-28T22:20:45.215 INFO:teuthology.orchestra.run.smithi016.stdout: "progress", 2024-03-28T22:20:45.215 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_support", 2024-03-28T22:20:45.215 INFO:teuthology.orchestra.run.smithi016.stdout: "status", 2024-03-28T22:20:45.215 INFO:teuthology.orchestra.run.smithi016.stdout: "telemetry", 2024-03-28T22:20:45.215 INFO:teuthology.orchestra.run.smithi016.stdout: "volumes" 2024-03-28T22:20:45.215 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:45.215 INFO:teuthology.orchestra.run.smithi016.stdout: "pacific": [ 2024-03-28T22:20:45.216 INFO:teuthology.orchestra.run.smithi016.stdout: "balancer", 2024-03-28T22:20:45.216 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-03-28T22:20:45.216 INFO:teuthology.orchestra.run.smithi016.stdout: "devicehealth", 2024-03-28T22:20:45.216 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator", 2024-03-28T22:20:45.216 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscaler", 2024-03-28T22:20:45.216 INFO:teuthology.orchestra.run.smithi016.stdout: "progress", 2024-03-28T22:20:45.216 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_support", 2024-03-28T22:20:45.216 INFO:teuthology.orchestra.run.smithi016.stdout: "status", 2024-03-28T22:20:45.217 INFO:teuthology.orchestra.run.smithi016.stdout: "telemetry", 2024-03-28T22:20:45.217 INFO:teuthology.orchestra.run.smithi016.stdout: "volumes" 2024-03-28T22:20:45.217 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:45.217 INFO:teuthology.orchestra.run.smithi016.stdout: "quincy": [ 2024-03-28T22:20:45.217 INFO:teuthology.orchestra.run.smithi016.stdout: "balancer", 2024-03-28T22:20:45.217 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-03-28T22:20:45.217 INFO:teuthology.orchestra.run.smithi016.stdout: "devicehealth", 2024-03-28T22:20:45.217 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator", 2024-03-28T22:20:45.218 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscaler", 2024-03-28T22:20:45.218 INFO:teuthology.orchestra.run.smithi016.stdout: "progress", 2024-03-28T22:20:45.218 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_support", 2024-03-28T22:20:45.218 INFO:teuthology.orchestra.run.smithi016.stdout: "status", 2024-03-28T22:20:45.218 INFO:teuthology.orchestra.run.smithi016.stdout: "telemetry", 2024-03-28T22:20:45.218 INFO:teuthology.orchestra.run.smithi016.stdout: "volumes" 2024-03-28T22:20:45.218 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:45.219 INFO:teuthology.orchestra.run.smithi016.stdout: "reef": [ 2024-03-28T22:20:45.219 INFO:teuthology.orchestra.run.smithi016.stdout: "balancer", 2024-03-28T22:20:45.219 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-03-28T22:20:45.219 INFO:teuthology.orchestra.run.smithi016.stdout: "devicehealth", 2024-03-28T22:20:45.219 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator", 2024-03-28T22:20:45.219 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscaler", 2024-03-28T22:20:45.219 INFO:teuthology.orchestra.run.smithi016.stdout: "progress", 2024-03-28T22:20:45.219 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_support", 2024-03-28T22:20:45.220 INFO:teuthology.orchestra.run.smithi016.stdout: "status", 2024-03-28T22:20:45.220 INFO:teuthology.orchestra.run.smithi016.stdout: "telemetry", 2024-03-28T22:20:45.220 INFO:teuthology.orchestra.run.smithi016.stdout: "volumes" 2024-03-28T22:20:45.220 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-03-28T22:20:45.220 INFO:teuthology.orchestra.run.smithi016.stdout: "squid": [ 2024-03-28T22:20:45.220 INFO:teuthology.orchestra.run.smithi016.stdout: "balancer", 2024-03-28T22:20:45.220 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-03-28T22:20:45.220 INFO:teuthology.orchestra.run.smithi016.stdout: "devicehealth", 2024-03-28T22:20:45.221 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator", 2024-03-28T22:20:45.221 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscaler", 2024-03-28T22:20:45.221 INFO:teuthology.orchestra.run.smithi016.stdout: "progress", 2024-03-28T22:20:45.221 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_support", 2024-03-28T22:20:45.221 INFO:teuthology.orchestra.run.smithi016.stdout: "status", 2024-03-28T22:20:45.221 INFO:teuthology.orchestra.run.smithi016.stdout: "telemetry", 2024-03-28T22:20:45.221 INFO:teuthology.orchestra.run.smithi016.stdout: "volumes" 2024-03-28T22:20:45.221 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-03-28T22:20:45.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.222 INFO:teuthology.orchestra.run.smithi016.stdout: "last_failure_osd_epoch": 89, 2024-03-28T22:20:45.222 INFO:teuthology.orchestra.run.smithi016.stdout: "active_clients": [ 2024-03-28T22:20:45.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:45.222 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "devicehealth", 2024-03-28T22:20:45.222 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-03-28T22:20:45.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:45.223 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-03-28T22:20:45.223 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-03-28T22:20:45.223 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3936054478 2024-03-28T22:20:45.223 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:45.223 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-03-28T22:20:45.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:45.223 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "libcephsqlite", 2024-03-28T22:20:45.224 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-03-28T22:20:45.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:45.224 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-03-28T22:20:45.224 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-03-28T22:20:45.224 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2709607521 2024-03-28T22:20:45.224 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:45.224 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-03-28T22:20:45.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:45.225 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_support", 2024-03-28T22:20:45.225 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-03-28T22:20:45.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:45.225 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-03-28T22:20:45.225 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-03-28T22:20:45.225 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1247336580 2024-03-28T22:20:45.225 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:45.226 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-03-28T22:20:45.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:45.226 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snap_schedule", 2024-03-28T22:20:45.226 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-03-28T22:20:45.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:45.226 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-03-28T22:20:45.226 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-03-28T22:20:45.227 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2465330500 2024-03-28T22:20:45.227 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:45.227 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-03-28T22:20:45.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:45.227 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snap_schedule", 2024-03-28T22:20:45.227 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-03-28T22:20:45.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:45.228 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-03-28T22:20:45.228 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-03-28T22:20:45.228 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1220981374 2024-03-28T22:20:45.228 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:45.228 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-03-28T22:20:45.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:45.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:45.228 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "volumes", 2024-03-28T22:20:45.229 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-03-28T22:20:45.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-03-28T22:20:45.229 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-03-28T22:20:45.229 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-03-28T22:20:45.229 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3010384052 2024-03-28T22:20:45.229 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:45.229 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-03-28T22:20:45.229 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:45.229 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-03-28T22:20:45.229 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:20:45.231 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:43 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2407516669' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:20:45.231 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:43 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/1531988680' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:20:45.231 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:43 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/3959160727' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-28T22:20:45.232 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:44 smithi063 ceph-mon[36249]: pgmap v315: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T22:20:45.232 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:43 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2407516669' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:20:45.232 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:43 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/1531988680' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:20:45.232 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:43 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3959160727' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-28T22:20:45.232 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:44 smithi053 ceph-mon[35979]: pgmap v315: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T22:20:45.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.975+0000 7f071d33e640 1 -- 172.21.15.16:0/3959160727 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f07040a1ef0 msgr2=0x7f07040a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:45.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.975+0000 7f071d33e640 1 --2- 172.21.15.16:0/3959160727 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f07040a1ef0 0x7f07040a43b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f07180ffa50 tx=0x7f0700009110 comp rx=0 tx=0).stop 2024-03-28T22:20:45.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.975+0000 7f071d33e640 1 -- 172.21.15.16:0/3959160727 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07181001b0 msgr2=0x7f0718198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:20:45.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.975+0000 7f071d33e640 1 --2- 172.21.15.16:0/3959160727 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07181001b0 0x7f0718198000 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0710006010 tx=0x7f071002cb30 comp rx=0 tx=0).stop 2024-03-28T22:20:45.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.975+0000 7f071d33e640 1 -- 172.21.15.16:0/3959160727 shutdown_connections 2024-03-28T22:20:45.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.975+0000 7f071d33e640 1 --2- 172.21.15.16:0/3959160727 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f07040a1ef0 0x7f07040a43b0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:45.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.975+0000 7f071d33e640 1 --2- 172.21.15.16:0/3959160727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f071819c8c0 0x7f071819cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:45.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.975+0000 7f071d33e640 1 --2- 172.21.15.16:0/3959160727 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0718106a80 0x7f0718198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:45.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.975+0000 7f071d33e640 1 --2- 172.21.15.16:0/3959160727 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07181001b0 0x7f0718198000 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:20:45.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.975+0000 7f071d33e640 1 -- 172.21.15.16:0/3959160727 >> 172.21.15.16:0/3959160727 conn(0x7f0718076aa0 msgr2=0x7f07180fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:20:45.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.975+0000 7f071d33e640 1 -- 172.21.15.16:0/3959160727 shutdown_connections 2024-03-28T22:20:45.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:20:42.975+0000 7f071d33e640 1 -- 172.21.15.16:0/3959160727 wait complete. 2024-03-28T22:20:45.241 DEBUG:tasks.check_counter:Getting stats from a 2024-03-28T22:20:45.241 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-28T22:20:45.242 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:43 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2407516669' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:20:45.242 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:43 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/1531988680' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:20:45.242 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:43 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3959160727' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-28T22:20:45.242 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:44 smithi016 ceph-mon[32374]: pgmap v315: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T22:20:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:20:46.430 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:46 smithi063 ceph-mon[36249]: pgmap v316: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T22:20:46.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:46 smithi053 ceph-mon[35979]: pgmap v316: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T22:20:46.629 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:46 smithi016 ceph-mon[32374]: pgmap v316: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T22:20:47.170 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:20:47.170 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T22:20:47.170 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 0, 2024-03-28T22:20:47.170 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 2, 2024-03-28T22:20:47.171 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 0, 2024-03-28T22:20:47.171 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 0, 2024-03-28T22:20:47.171 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 4, 2024-03-28T22:20:47.171 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 0, 2024-03-28T22:20:47.171 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.001281088, 2024-03-28T22:20:47.171 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.000000000, 2024-03-28T22:20:47.171 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.000336369, 2024-03-28T22:20:47.171 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T22:20:47.171 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:20:47.172 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-03-28T22:20:47.172 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-03-28T22:20:47.172 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-03-28T22:20:47.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:47.172 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:20:47.172 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-03-28T22:20:47.172 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-03-28T22:20:47.172 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-03-28T22:20:47.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:47.173 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T22:20:47.173 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T22:20:47.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:47.173 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T22:20:47.173 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 216, 2024-03-28T22:20:47.173 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 203, 2024-03-28T22:20:47.173 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 1569440, 2024-03-28T22:20:47.173 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 123872, 2024-03-28T22:20:47.174 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 2, 2024-03-28T22:20:47.174 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 1, 2024-03-28T22:20:47.174 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.041364940, 2024-03-28T22:20:47.174 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.013284043, 2024-03-28T22:20:47.174 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.033000052, 2024-03-28T22:20:47.174 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.003963630, 2024-03-28T22:20:47.174 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:20:47.174 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 203, 2024-03-28T22:20:47.175 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.010791138, 2024-03-28T22:20:47.175 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000053158 2024-03-28T22:20:47.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:47.175 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:20:47.175 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-03-28T22:20:47.175 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-03-28T22:20:47.175 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-03-28T22:20:47.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:47.175 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 1569440, 2024-03-28T22:20:47.175 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 123872 2024-03-28T22:20:47.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:47.176 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T22:20:47.176 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 2, 2024-03-28T22:20:47.176 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 154, 2024-03-28T22:20:47.176 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 320, 2024-03-28T22:20:47.176 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 182672, 2024-03-28T22:20:47.176 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 5, 2024-03-28T22:20:47.176 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 3, 2024-03-28T22:20:47.176 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.013113822, 2024-03-28T22:20:47.177 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.008992666, 2024-03-28T22:20:47.177 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.001833398, 2024-03-28T22:20:47.177 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T22:20:47.177 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:20:47.177 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 151, 2024-03-28T22:20:47.177 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.009500596, 2024-03-28T22:20:47.177 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000062917 2024-03-28T22:20:47.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:47.178 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:20:47.178 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-03-28T22:20:47.178 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-03-28T22:20:47.178 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-03-28T22:20:47.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:47.178 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-28T22:20:47.178 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 182672 2024-03-28T22:20:47.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:47.178 INFO:teuthology.orchestra.run.smithi016.stdout: "cct": { 2024-03-28T22:20:47.179 INFO:teuthology.orchestra.run.smithi016.stdout: "total_workers": 0, 2024-03-28T22:20:47.179 INFO:teuthology.orchestra.run.smithi016.stdout: "unhealthy_workers": 0 2024-03-28T22:20:47.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:47.179 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool": { 2024-03-28T22:20:47.179 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_bytes": 0, 2024-03-28T22:20:47.179 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_items": 0, 2024-03-28T22:20:47.179 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T22:20:47.179 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_items": 0, 2024-03-28T22:20:47.179 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T22:20:47.180 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_items": 0, 2024-03-28T22:20:47.180 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T22:20:47.180 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T22:20:47.180 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T22:20:47.180 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T22:20:47.180 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T22:20:47.180 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_items": 0, 2024-03-28T22:20:47.180 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T22:20:47.180 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T22:20:47.181 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_bytes": 0, 2024-03-28T22:20:47.181 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_items": 0, 2024-03-28T22:20:47.181 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_bytes": 0, 2024-03-28T22:20:47.181 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_items": 0, 2024-03-28T22:20:47.181 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T22:20:47.181 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T22:20:47.181 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T22:20:47.181 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T22:20:47.181 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T22:20:47.182 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_items": 0, 2024-03-28T22:20:47.182 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_bytes": 0, 2024-03-28T22:20:47.182 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_items": 0, 2024-03-28T22:20:47.182 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T22:20:47.182 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T22:20:47.182 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_bytes": 0, 2024-03-28T22:20:47.182 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_items": 0, 2024-03-28T22:20:47.182 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_bytes": 0, 2024-03-28T22:20:47.182 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_items": 0, 2024-03-28T22:20:47.183 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T22:20:47.183 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_items": 0, 2024-03-28T22:20:47.183 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T22:20:47.183 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_items": 0, 2024-03-28T22:20:47.183 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_bytes": 149034, 2024-03-28T22:20:47.183 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_items": 32, 2024-03-28T22:20:47.183 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_bytes": 0, 2024-03-28T22:20:47.183 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_items": 0, 2024-03-28T22:20:47.183 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bytes": 0, 2024-03-28T22:20:47.183 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_items": 0, 2024-03-28T22:20:47.184 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_bytes": 0, 2024-03-28T22:20:47.184 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_items": 0, 2024-03-28T22:20:47.184 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_bytes": 0, 2024-03-28T22:20:47.184 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_items": 0, 2024-03-28T22:20:47.184 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_bytes": 0, 2024-03-28T22:20:47.184 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_items": 0, 2024-03-28T22:20:47.184 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T22:20:47.184 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_items": 0, 2024-03-28T22:20:47.185 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_bytes": 0, 2024-03-28T22:20:47.185 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_items": 0, 2024-03-28T22:20:47.185 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_bytes": 928, 2024-03-28T22:20:47.185 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_items": 2, 2024-03-28T22:20:47.185 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_bytes": 0, 2024-03-28T22:20:47.185 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_items": 0, 2024-03-28T22:20:47.185 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_bytes": 0, 2024-03-28T22:20:47.185 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_items": 0 2024-03-28T22:20:47.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:47.186 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T22:20:47.186 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-03-28T22:20:47.186 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-03-28T22:20:47.186 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-03-28T22:20:47.186 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 214, 2024-03-28T22:20:47.186 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 1536484, 2024-03-28T22:20:47.186 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-03-28T22:20:47.187 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 214, 2024-03-28T22:20:47.187 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-03-28T22:20:47.187 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-03-28T22:20:47.187 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 214, 2024-03-28T22:20:47.187 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 1536484, 2024-03-28T22:20:47.187 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-03-28T22:20:47.187 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-03-28T22:20:47.187 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-03-28T22:20:47.187 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-03-28T22:20:47.188 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:47.188 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:47.188 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:20:47.762 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-03-28T22:20:47.762 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-03-28T22:20:47.762 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.a 2024-03-28T22:20:47.763 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.a 2024-03-28T22:20:47.763 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-03-28T22:20:47.763 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-03-28T22:20:47.763 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-03-28T22:20:47.763 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.a 2024-03-28T22:20:47.763 DEBUG:tasks.check_counter:Getting stats from d 2024-03-28T22:20:47.763 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-28T22:20:48.013 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:20:48.386 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:48 smithi016 ceph-mon[32374]: pgmap v317: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-28T22:20:48.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:48 smithi053 ceph-mon[35979]: pgmap v317: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-28T22:20:48.680 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:48 smithi063 ceph-mon[36249]: pgmap v317: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-28T22:20:49.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:49 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:20:49.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:49 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:20:49.680 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:49 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:20:49.690 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:20:49.690 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T22:20:49.691 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 0, 2024-03-28T22:20:49.691 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 2, 2024-03-28T22:20:49.691 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 0, 2024-03-28T22:20:49.691 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 0, 2024-03-28T22:20:49.691 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 4, 2024-03-28T22:20:49.691 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 0, 2024-03-28T22:20:49.691 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.001260884, 2024-03-28T22:20:49.692 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.000000000, 2024-03-28T22:20:49.692 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.000211951, 2024-03-28T22:20:49.692 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T22:20:49.692 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:20:49.692 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-03-28T22:20:49.692 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-03-28T22:20:49.692 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-03-28T22:20:49.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:49.693 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:20:49.693 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-03-28T22:20:49.693 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-03-28T22:20:49.693 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-03-28T22:20:49.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:49.694 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T22:20:49.694 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T22:20:49.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:49.694 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T22:20:49.694 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 217, 2024-03-28T22:20:49.694 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 206, 2024-03-28T22:20:49.694 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 1567728, 2024-03-28T22:20:49.695 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 126624, 2024-03-28T22:20:49.695 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 2, 2024-03-28T22:20:49.695 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 1, 2024-03-28T22:20:49.695 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.042093984, 2024-03-28T22:20:49.695 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.013699148, 2024-03-28T22:20:49.695 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.033795661, 2024-03-28T22:20:49.695 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.004173867, 2024-03-28T22:20:49.696 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:20:49.696 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 205, 2024-03-28T22:20:49.696 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.010211582, 2024-03-28T22:20:49.696 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000049812 2024-03-28T22:20:49.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:49.696 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:20:49.696 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-03-28T22:20:49.697 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-03-28T22:20:49.697 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-03-28T22:20:49.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:49.697 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 1567728, 2024-03-28T22:20:49.697 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 126624 2024-03-28T22:20:49.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:49.697 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T22:20:49.697 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 6, 2024-03-28T22:20:49.698 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 159, 2024-03-28T22:20:49.698 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 3392, 2024-03-28T22:20:49.698 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 183648, 2024-03-28T22:20:49.698 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 5, 2024-03-28T22:20:49.698 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 3, 2024-03-28T22:20:49.698 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.014364407, 2024-03-28T22:20:49.698 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.009760735, 2024-03-28T22:20:49.699 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.002204481, 2024-03-28T22:20:49.699 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T22:20:49.699 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:20:49.699 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 157, 2024-03-28T22:20:49.699 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.011466262, 2024-03-28T22:20:49.699 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000073033 2024-03-28T22:20:49.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:49.699 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:20:49.700 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-03-28T22:20:49.700 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-03-28T22:20:49.700 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-03-28T22:20:49.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:49.700 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 3392, 2024-03-28T22:20:49.700 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 183648 2024-03-28T22:20:49.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:49.701 INFO:teuthology.orchestra.run.smithi016.stdout: "cct": { 2024-03-28T22:20:49.701 INFO:teuthology.orchestra.run.smithi016.stdout: "total_workers": 0, 2024-03-28T22:20:49.701 INFO:teuthology.orchestra.run.smithi016.stdout: "unhealthy_workers": 0 2024-03-28T22:20:49.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:49.701 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool": { 2024-03-28T22:20:49.701 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_bytes": 0, 2024-03-28T22:20:49.701 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_items": 0, 2024-03-28T22:20:49.702 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T22:20:49.702 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_items": 0, 2024-03-28T22:20:49.702 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T22:20:49.702 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_items": 0, 2024-03-28T22:20:49.703 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T22:20:49.703 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T22:20:49.703 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T22:20:49.703 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T22:20:49.703 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T22:20:49.703 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_items": 0, 2024-03-28T22:20:49.703 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T22:20:49.703 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T22:20:49.704 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_bytes": 0, 2024-03-28T22:20:49.704 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_items": 0, 2024-03-28T22:20:49.704 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_bytes": 0, 2024-03-28T22:20:49.704 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_items": 0, 2024-03-28T22:20:49.704 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T22:20:49.704 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T22:20:49.704 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T22:20:49.704 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T22:20:49.705 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T22:20:49.705 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_items": 0, 2024-03-28T22:20:49.705 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_bytes": 0, 2024-03-28T22:20:49.705 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_items": 0, 2024-03-28T22:20:49.705 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T22:20:49.705 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T22:20:49.705 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_bytes": 0, 2024-03-28T22:20:49.705 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_items": 0, 2024-03-28T22:20:49.705 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_bytes": 0, 2024-03-28T22:20:49.706 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_items": 0, 2024-03-28T22:20:49.706 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T22:20:49.706 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_items": 0, 2024-03-28T22:20:49.706 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T22:20:49.706 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_items": 0, 2024-03-28T22:20:49.706 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_bytes": 148977, 2024-03-28T22:20:49.706 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_items": 30, 2024-03-28T22:20:49.707 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_bytes": 0, 2024-03-28T22:20:49.707 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_items": 0, 2024-03-28T22:20:49.707 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bytes": 0, 2024-03-28T22:20:49.707 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_items": 0, 2024-03-28T22:20:49.707 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_bytes": 0, 2024-03-28T22:20:49.707 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_items": 0, 2024-03-28T22:20:49.707 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_bytes": 0, 2024-03-28T22:20:49.707 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_items": 0, 2024-03-28T22:20:49.707 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_bytes": 0, 2024-03-28T22:20:49.708 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_items": 0, 2024-03-28T22:20:49.708 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T22:20:49.708 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_items": 0, 2024-03-28T22:20:49.708 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_bytes": 0, 2024-03-28T22:20:49.708 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_items": 0, 2024-03-28T22:20:49.708 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_bytes": 928, 2024-03-28T22:20:49.708 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_items": 2, 2024-03-28T22:20:49.708 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_bytes": 0, 2024-03-28T22:20:49.708 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_items": 0, 2024-03-28T22:20:49.709 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_bytes": 0, 2024-03-28T22:20:49.709 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_items": 0 2024-03-28T22:20:49.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:49.709 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T22:20:49.709 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-03-28T22:20:49.709 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-03-28T22:20:49.709 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-03-28T22:20:49.709 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 219, 2024-03-28T22:20:49.710 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 1537134, 2024-03-28T22:20:49.710 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-03-28T22:20:49.710 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 219, 2024-03-28T22:20:49.710 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-03-28T22:20:49.710 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-03-28T22:20:49.711 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 219, 2024-03-28T22:20:49.711 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 1537134, 2024-03-28T22:20:49.711 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-03-28T22:20:49.711 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-03-28T22:20:49.711 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-03-28T22:20:49.711 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-03-28T22:20:49.711 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:49.711 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:49.712 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:20:50.268 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-28T22:20:50.269 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-28T22:20:50.269 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-03-28T22:20:50.269 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-03-28T22:20:50.269 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-03-28T22:20:50.269 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-03-28T22:20:50.269 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-03-28T22:20:50.269 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2024-03-28T22:20:50.269 DEBUG:tasks.check_counter:Getting stats from g 2024-03-28T22:20:50.270 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-28T22:20:50.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:50 smithi053 ceph-mon[35979]: pgmap v318: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T22:20:50.585 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:50 smithi016 ceph-mon[32374]: pgmap v318: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T22:20:50.680 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:50 smithi063 ceph-mon[36249]: pgmap v318: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-28T22:20:50.961 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:20:51.930 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:51 smithi063 ceph-mon[36249]: pgmap v319: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-28T22:20:51.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:51 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:20:51.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:51 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:20:51.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:51 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:20:51.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:51 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:20:51.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:51 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:20:51.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:51 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:20:51.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:51 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:20:51.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:51 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:20:51.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:51 smithi063 ceph-mon[36249]: Adjusting osd_memory_target on smithi063 to 174.1M 2024-03-28T22:20:51.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:51 smithi063 ceph-mon[36249]: Unable to set osd_memory_target on smithi063 to 182642073: error parsing value: Value '182642073' is below minimum 939524096 2024-03-28T22:20:51.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:51 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:20:51.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:51 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:20:51.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:51 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:20:51.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:51 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:20:51.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:51 smithi016 ceph-mon[32374]: pgmap v319: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-28T22:20:51.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:51 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:20:51.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:51 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:20:51.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:51 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:20:51.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:51 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:20:51.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:51 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:20:51.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:51 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:20:51.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:51 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:20:51.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:51 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:20:51.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:51 smithi016 ceph-mon[32374]: Adjusting osd_memory_target on smithi063 to 174.1M 2024-03-28T22:20:51.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:51 smithi016 ceph-mon[32374]: Unable to set osd_memory_target on smithi063 to 182642073: error parsing value: Value '182642073' is below minimum 939524096 2024-03-28T22:20:51.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:51 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:20:51.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:51 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:20:51.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:51 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:20:51.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:51 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:20:52.046 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:51 smithi053 ceph-mon[35979]: pgmap v319: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-28T22:20:52.046 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:51 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:20:52.046 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:51 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:20:52.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:51 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:20:52.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:51 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:20:52.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:51 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:20:52.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:51 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:20:52.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:51 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:20:52.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:51 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:20:52.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:51 smithi053 ceph-mon[35979]: Adjusting osd_memory_target on smithi063 to 174.1M 2024-03-28T22:20:52.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:51 smithi053 ceph-mon[35979]: Unable to set osd_memory_target on smithi063 to 182642073: error parsing value: Value '182642073' is below minimum 939524096 2024-03-28T22:20:52.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:51 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:20:52.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:51 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:20:52.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:51 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:20:52.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:51 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:20:52.750 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:20:52.751 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T22:20:52.751 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 0, 2024-03-28T22:20:52.751 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 2, 2024-03-28T22:20:52.751 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 0, 2024-03-28T22:20:52.751 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 0, 2024-03-28T22:20:52.751 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 4, 2024-03-28T22:20:52.751 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 0, 2024-03-28T22:20:52.752 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.001107936, 2024-03-28T22:20:52.752 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.000000000, 2024-03-28T22:20:52.752 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.000158760, 2024-03-28T22:20:52.752 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T22:20:52.752 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:20:52.752 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-03-28T22:20:52.752 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-03-28T22:20:52.752 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-03-28T22:20:52.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:52.753 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:20:52.753 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-03-28T22:20:52.753 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-03-28T22:20:52.753 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-03-28T22:20:52.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:52.753 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T22:20:52.753 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T22:20:52.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:52.754 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T22:20:52.754 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 6, 2024-03-28T22:20:52.754 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 157, 2024-03-28T22:20:52.754 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 3392, 2024-03-28T22:20:52.754 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 183296, 2024-03-28T22:20:52.754 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 5, 2024-03-28T22:20:52.754 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 3, 2024-03-28T22:20:52.754 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.013382579, 2024-03-28T22:20:52.755 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.008985556, 2024-03-28T22:20:52.757 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.002026097, 2024-03-28T22:20:52.757 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T22:20:52.757 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:20:52.757 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 155, 2024-03-28T22:20:52.757 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.012103185, 2024-03-28T22:20:52.758 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000078085 2024-03-28T22:20:52.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:52.758 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:20:52.758 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-03-28T22:20:52.758 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-03-28T22:20:52.758 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-03-28T22:20:52.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:52.759 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 3392, 2024-03-28T22:20:52.759 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 183296 2024-03-28T22:20:52.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:52.759 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T22:20:52.759 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 214, 2024-03-28T22:20:52.759 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 203, 2024-03-28T22:20:52.760 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 1566912, 2024-03-28T22:20:52.760 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 124704, 2024-03-28T22:20:52.760 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 2, 2024-03-28T22:20:52.760 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 1, 2024-03-28T22:20:52.760 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.041861501, 2024-03-28T22:20:52.760 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.014033754, 2024-03-28T22:20:52.760 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.033257758, 2024-03-28T22:20:52.760 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.004040610, 2024-03-28T22:20:52.761 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:20:52.761 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 202, 2024-03-28T22:20:52.761 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.009057839, 2024-03-28T22:20:52.761 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000044840 2024-03-28T22:20:52.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:52.761 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:20:52.761 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-03-28T22:20:52.762 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-03-28T22:20:52.762 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-03-28T22:20:52.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:52.762 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 1566912, 2024-03-28T22:20:52.762 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 124704 2024-03-28T22:20:52.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:52.762 INFO:teuthology.orchestra.run.smithi016.stdout: "cct": { 2024-03-28T22:20:52.763 INFO:teuthology.orchestra.run.smithi016.stdout: "total_workers": 0, 2024-03-28T22:20:52.763 INFO:teuthology.orchestra.run.smithi016.stdout: "unhealthy_workers": 0 2024-03-28T22:20:52.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:52.763 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool": { 2024-03-28T22:20:52.763 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_bytes": 0, 2024-03-28T22:20:52.763 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_items": 0, 2024-03-28T22:20:52.763 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T22:20:52.764 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_items": 0, 2024-03-28T22:20:52.764 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T22:20:52.764 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_items": 0, 2024-03-28T22:20:52.764 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T22:20:52.764 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T22:20:52.764 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T22:20:52.764 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T22:20:52.765 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T22:20:52.765 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_items": 0, 2024-03-28T22:20:52.765 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T22:20:52.765 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T22:20:52.765 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_bytes": 0, 2024-03-28T22:20:52.765 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_items": 0, 2024-03-28T22:20:52.765 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_bytes": 0, 2024-03-28T22:20:52.766 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_items": 0, 2024-03-28T22:20:52.766 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T22:20:52.766 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T22:20:52.766 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T22:20:52.766 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T22:20:52.766 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T22:20:52.767 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_items": 0, 2024-03-28T22:20:52.767 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_bytes": 0, 2024-03-28T22:20:52.767 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_items": 0, 2024-03-28T22:20:52.767 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T22:20:52.767 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T22:20:52.767 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_bytes": 0, 2024-03-28T22:20:52.767 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_items": 0, 2024-03-28T22:20:52.768 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_bytes": 0, 2024-03-28T22:20:52.768 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_items": 0, 2024-03-28T22:20:52.768 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T22:20:52.768 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_items": 0, 2024-03-28T22:20:52.768 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T22:20:52.768 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_items": 0, 2024-03-28T22:20:52.768 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_bytes": 148977, 2024-03-28T22:20:52.768 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_items": 30, 2024-03-28T22:20:52.769 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_bytes": 0, 2024-03-28T22:20:52.769 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_items": 0, 2024-03-28T22:20:52.769 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bytes": 0, 2024-03-28T22:20:52.769 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_items": 0, 2024-03-28T22:20:52.769 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_bytes": 0, 2024-03-28T22:20:52.769 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_items": 0, 2024-03-28T22:20:52.769 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_bytes": 0, 2024-03-28T22:20:52.769 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_items": 0, 2024-03-28T22:20:52.770 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_bytes": 0, 2024-03-28T22:20:52.770 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_items": 0, 2024-03-28T22:20:52.770 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T22:20:52.770 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_items": 0, 2024-03-28T22:20:52.770 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_bytes": 0, 2024-03-28T22:20:52.770 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_items": 0, 2024-03-28T22:20:52.770 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_bytes": 928, 2024-03-28T22:20:52.770 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_items": 2, 2024-03-28T22:20:52.771 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_bytes": 0, 2024-03-28T22:20:52.771 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_items": 0, 2024-03-28T22:20:52.771 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_bytes": 0, 2024-03-28T22:20:52.771 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_items": 0 2024-03-28T22:20:52.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:52.771 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T22:20:52.771 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-03-28T22:20:52.771 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-03-28T22:20:52.772 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-03-28T22:20:52.772 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 216, 2024-03-28T22:20:52.772 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 1536744, 2024-03-28T22:20:52.772 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-03-28T22:20:52.772 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 216, 2024-03-28T22:20:52.772 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-03-28T22:20:52.772 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-03-28T22:20:52.772 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 216, 2024-03-28T22:20:52.773 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 1536744, 2024-03-28T22:20:52.773 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-03-28T22:20:52.773 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-03-28T22:20:52.773 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-03-28T22:20:52.773 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-03-28T22:20:52.773 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:52.773 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:52.773 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:20:52.930 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:52 smithi063 ceph-mon[36249]: Adjusting osd_memory_target on smithi053 to 175.2M 2024-03-28T22:20:52.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:52 smithi063 ceph-mon[36249]: Unable to set osd_memory_target on smithi053 to 183743795: error parsing value: Value '183743795' is below minimum 939524096 2024-03-28T22:20:52.934 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:52 smithi016 ceph-mon[32374]: Adjusting osd_memory_target on smithi053 to 175.2M 2024-03-28T22:20:52.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:52 smithi016 ceph-mon[32374]: Unable to set osd_memory_target on smithi053 to 183743795: error parsing value: Value '183743795' is below minimum 939524096 2024-03-28T22:20:53.046 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:52 smithi053 ceph-mon[35979]: Adjusting osd_memory_target on smithi053 to 175.2M 2024-03-28T22:20:53.046 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:52 smithi053 ceph-mon[35979]: Unable to set osd_memory_target on smithi053 to 183743795: error parsing value: Value '183743795' is below minimum 939524096 2024-03-28T22:20:53.315 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-03-28T22:20:53.315 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-03-28T22:20:53.316 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.g 2024-03-28T22:20:53.316 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.g 2024-03-28T22:20:53.316 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-03-28T22:20:53.316 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-03-28T22:20:53.316 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-03-28T22:20:53.316 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.g 2024-03-28T22:20:53.316 DEBUG:tasks.check_counter:Getting stats from j 2024-03-28T22:20:53.317 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-28T22:20:53.595 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:20:53.886 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:53 smithi016 ceph-mon[32374]: pgmap v320: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:20:53.930 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:53 smithi063 ceph-mon[36249]: pgmap v320: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:20:54.046 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:53 smithi053 ceph-mon[35979]: pgmap v320: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:20:54.722 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:54 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:20:54.722 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:54 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:20:54.722 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:54 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:20:54.722 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:54 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:20:54.722 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:54 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:20:54.722 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:54 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:20:54.723 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:54 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:20:54.723 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:54 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:20:54.723 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:54 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:20:54.930 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:54 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:20:54.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:54 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:20:54.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:54 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:20:54.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:54 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:20:54.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:54 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:20:54.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:54 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:20:54.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:54 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:20:54.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:54 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:20:54.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:54 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:20:55.046 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:54 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:20:55.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:54 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:20:55.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:54 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:20:55.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:54 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:20:55.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:54 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:20:55.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:54 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:20:55.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:54 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:20:55.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:54 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:20:55.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:54 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' 2024-03-28T22:20:55.220 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-28T22:20:55.220 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T22:20:55.220 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 4, 2024-03-28T22:20:55.221 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 4, 2024-03-28T22:20:55.221 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 3072, 2024-03-28T22:20:55.221 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 448, 2024-03-28T22:20:55.221 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 4, 2024-03-28T22:20:55.221 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 0, 2024-03-28T22:20:55.221 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.001461495, 2024-03-28T22:20:55.222 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.000042664, 2024-03-28T22:20:55.222 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.000624123, 2024-03-28T22:20:55.222 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T22:20:55.222 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:20:55.222 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 3, 2024-03-28T22:20:55.222 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000833965, 2024-03-28T22:20:55.222 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000277988 2024-03-28T22:20:55.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:55.223 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:20:55.223 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-03-28T22:20:55.223 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-03-28T22:20:55.223 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-03-28T22:20:55.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:55.223 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-28T22:20:55.223 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 448 2024-03-28T22:20:55.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:55.224 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T22:20:55.224 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 219, 2024-03-28T22:20:55.224 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 208, 2024-03-28T22:20:55.224 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 1568272, 2024-03-28T22:20:55.224 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 127904, 2024-03-28T22:20:55.224 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 2, 2024-03-28T22:20:55.224 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 1, 2024-03-28T22:20:55.224 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.044040121, 2024-03-28T22:20:55.225 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.013695971, 2024-03-28T22:20:55.225 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.035433079, 2024-03-28T22:20:55.225 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.004242058, 2024-03-28T22:20:55.225 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:20:55.225 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 207, 2024-03-28T22:20:55.225 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.009578276, 2024-03-28T22:20:55.225 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000046271 2024-03-28T22:20:55.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:55.226 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:20:55.226 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-03-28T22:20:55.226 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-03-28T22:20:55.226 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-03-28T22:20:55.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:55.226 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 1568272, 2024-03-28T22:20:55.227 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 127904 2024-03-28T22:20:55.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:55.227 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T22:20:55.227 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 2, 2024-03-28T22:20:55.227 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 159, 2024-03-28T22:20:55.227 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 320, 2024-03-28T22:20:55.227 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 183552, 2024-03-28T22:20:55.228 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 5, 2024-03-28T22:20:55.228 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 3, 2024-03-28T22:20:55.228 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.013511747, 2024-03-28T22:20:55.228 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.009282099, 2024-03-28T22:20:55.228 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.002056552, 2024-03-28T22:20:55.228 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T22:20:55.228 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:20:55.229 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 156, 2024-03-28T22:20:55.229 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.009801171, 2024-03-28T22:20:55.229 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000062828 2024-03-28T22:20:55.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:55.229 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:20:55.229 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-03-28T22:20:55.229 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-03-28T22:20:55.230 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-03-28T22:20:55.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:55.230 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-28T22:20:55.230 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 183552 2024-03-28T22:20:55.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:55.230 INFO:teuthology.orchestra.run.smithi016.stdout: "cct": { 2024-03-28T22:20:55.231 INFO:teuthology.orchestra.run.smithi016.stdout: "total_workers": 0, 2024-03-28T22:20:55.231 INFO:teuthology.orchestra.run.smithi016.stdout: "unhealthy_workers": 0 2024-03-28T22:20:55.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:55.231 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool": { 2024-03-28T22:20:55.231 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_bytes": 0, 2024-03-28T22:20:55.231 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_items": 0, 2024-03-28T22:20:55.232 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T22:20:55.232 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_items": 0, 2024-03-28T22:20:55.232 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T22:20:55.232 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_items": 0, 2024-03-28T22:20:55.232 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T22:20:55.232 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T22:20:55.232 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T22:20:55.233 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T22:20:55.233 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T22:20:55.233 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_items": 0, 2024-03-28T22:20:55.233 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T22:20:55.233 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T22:20:55.233 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_bytes": 0, 2024-03-28T22:20:55.233 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_items": 0, 2024-03-28T22:20:55.234 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_bytes": 0, 2024-03-28T22:20:55.234 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_items": 0, 2024-03-28T22:20:55.234 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T22:20:55.234 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T22:20:55.234 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T22:20:55.234 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T22:20:55.234 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T22:20:55.235 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_items": 0, 2024-03-28T22:20:55.235 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_bytes": 0, 2024-03-28T22:20:55.235 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_items": 0, 2024-03-28T22:20:55.235 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T22:20:55.235 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T22:20:55.235 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_bytes": 0, 2024-03-28T22:20:55.235 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_items": 0, 2024-03-28T22:20:55.236 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_bytes": 0, 2024-03-28T22:20:55.236 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_items": 0, 2024-03-28T22:20:55.236 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T22:20:55.236 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_items": 0, 2024-03-28T22:20:55.236 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T22:20:55.236 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_items": 0, 2024-03-28T22:20:55.237 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_bytes": 148977, 2024-03-28T22:20:55.237 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_items": 30, 2024-03-28T22:20:55.237 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_bytes": 0, 2024-03-28T22:20:55.237 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_items": 0, 2024-03-28T22:20:55.237 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bytes": 0, 2024-03-28T22:20:55.237 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_items": 0, 2024-03-28T22:20:55.237 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_bytes": 0, 2024-03-28T22:20:55.237 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_items": 0, 2024-03-28T22:20:55.238 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_bytes": 0, 2024-03-28T22:20:55.238 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_items": 0, 2024-03-28T22:20:55.238 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_bytes": 0, 2024-03-28T22:20:55.238 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_items": 0, 2024-03-28T22:20:55.238 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T22:20:55.238 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_items": 0, 2024-03-28T22:20:55.238 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_bytes": 0, 2024-03-28T22:20:55.238 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_items": 0, 2024-03-28T22:20:55.238 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_bytes": 928, 2024-03-28T22:20:55.239 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_items": 2, 2024-03-28T22:20:55.239 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_bytes": 0, 2024-03-28T22:20:55.239 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_items": 0, 2024-03-28T22:20:55.239 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_bytes": 0, 2024-03-28T22:20:55.239 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_items": 0 2024-03-28T22:20:55.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-03-28T22:20:55.239 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T22:20:55.239 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-03-28T22:20:55.240 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-03-28T22:20:55.240 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-03-28T22:20:55.240 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 221, 2024-03-28T22:20:55.240 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 1537394, 2024-03-28T22:20:55.240 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-03-28T22:20:55.240 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 221, 2024-03-28T22:20:55.240 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-03-28T22:20:55.240 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-03-28T22:20:55.240 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 221, 2024-03-28T22:20:55.241 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 1537394, 2024-03-28T22:20:55.241 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-03-28T22:20:55.241 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-03-28T22:20:55.241 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-03-28T22:20:55.241 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-03-28T22:20:55.241 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:55.241 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-03-28T22:20:55.241 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-28T22:20:55.839 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:55 smithi016 ceph-mon[32374]: Adjusting osd_memory_target on smithi016 to 177.2M 2024-03-28T22:20:55.839 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:55 smithi016 ceph-mon[32374]: Unable to set osd_memory_target on smithi016 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-03-28T22:20:55.839 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:55 smithi016 ceph-mon[32374]: pgmap v321: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:20:55.863 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-28T22:20:55.864 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-28T22:20:55.864 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-03-28T22:20:55.864 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-03-28T22:20:55.864 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-03-28T22:20:55.864 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-03-28T22:20:55.864 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-03-28T22:20:55.864 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2024-03-28T22:20:55.865 DEBUG:tasks.check_counter:Getting stats from b 2024-03-28T22:20:55.865 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-28T22:20:55.894 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:55 smithi053 ceph-mon[35979]: Adjusting osd_memory_target on smithi016 to 177.2M 2024-03-28T22:20:55.894 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:55 smithi053 ceph-mon[35979]: Unable to set osd_memory_target on smithi016 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-03-28T22:20:55.894 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:55 smithi053 ceph-mon[35979]: pgmap v321: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:20:55.930 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:55 smithi063 ceph-mon[36249]: Adjusting osd_memory_target on smithi016 to 177.2M 2024-03-28T22:20:55.930 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:55 smithi063 ceph-mon[36249]: Unable to set osd_memory_target on smithi016 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-03-28T22:20:55.930 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:55 smithi063 ceph-mon[36249]: pgmap v321: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:20:56.144 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T22:20:57.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:57 smithi016 ceph-mon[32374]: pgmap v322: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:20:57.930 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:57 smithi063 ceph-mon[36249]: pgmap v322: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:20:58.001 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:57 smithi053 ceph-mon[35979]: pgmap v322: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:20:58.160 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-03-28T22:20:58.160 INFO:teuthology.orchestra.run.smithi053.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T22:20:58.160 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_messages": 1537, 2024-03-28T22:20:58.160 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages": 1359, 2024-03-28T22:20:58.160 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_bytes": 599438, 2024-03-28T22:20:58.160 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_bytes": 4298961, 2024-03-28T22:20:58.161 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_created_connections": 151, 2024-03-28T22:20:58.161 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_active_connections": 3, 2024-03-28T22:20:58.161 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_total_time": 0.173541077, 2024-03-28T22:20:58.161 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_send_time": 0.049076558, 2024-03-28T22:20:58.161 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_recv_time": 0.078650238, 2024-03-28T22:20:58.161 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_fast_dispatch_time": 0.034327336, 2024-03-28T22:20:58.161 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:20:58.161 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 1360, 2024-03-28T22:20:58.162 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.480051889, 2024-03-28T22:20:58.162 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000352979 2024-03-28T22:20:58.162 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.162 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:20:58.162 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 1404, 2024-03-28T22:20:58.162 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000257249, 2024-03-28T22:20:58.162 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000183 2024-03-28T22:20:58.162 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.163 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_encrypted_bytes": 46112, 2024-03-28T22:20:58.163 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_encrypted_bytes": 2512 2024-03-28T22:20:58.163 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.163 INFO:teuthology.orchestra.run.smithi053.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T22:20:58.163 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_messages": 2488, 2024-03-28T22:20:58.163 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages": 2001, 2024-03-28T22:20:58.163 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_bytes": 2547687, 2024-03-28T22:20:58.163 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_bytes": 1671093, 2024-03-28T22:20:58.163 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_created_connections": 12, 2024-03-28T22:20:58.164 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_active_connections": 6, 2024-03-28T22:20:58.164 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_total_time": 0.195871447, 2024-03-28T22:20:58.164 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_send_time": 0.076648015, 2024-03-28T22:20:58.164 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_recv_time": 0.121728861, 2024-03-28T22:20:58.164 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_fast_dispatch_time": 0.024168860, 2024-03-28T22:20:58.164 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:20:58.164 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 2004, 2024-03-28T22:20:58.164 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.349614497, 2024-03-28T22:20:58.165 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000174458 2024-03-28T22:20:58.165 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.165 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:20:58.165 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 2901, 2024-03-28T22:20:58.165 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000611857, 2024-03-28T22:20:58.165 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000210 2024-03-28T22:20:58.165 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.165 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_encrypted_bytes": 1646752, 2024-03-28T22:20:58.165 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_encrypted_bytes": 484352 2024-03-28T22:20:58.166 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.166 INFO:teuthology.orchestra.run.smithi053.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T22:20:58.166 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_messages": 3165, 2024-03-28T22:20:58.166 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages": 3651, 2024-03-28T22:20:58.166 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_bytes": 1109699, 2024-03-28T22:20:58.166 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_bytes": 2029048, 2024-03-28T22:20:58.166 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_created_connections": 26, 2024-03-28T22:20:58.166 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_active_connections": 8, 2024-03-28T22:20:58.167 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_total_time": 0.245125163, 2024-03-28T22:20:58.167 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_send_time": 0.080419598, 2024-03-28T22:20:58.167 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_recv_time": 0.101486393, 2024-03-28T22:20:58.167 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_fast_dispatch_time": 0.062746797, 2024-03-28T22:20:58.167 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:20:58.167 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 3642, 2024-03-28T22:20:58.167 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.334219148, 2024-03-28T22:20:58.167 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000091768 2024-03-28T22:20:58.167 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.168 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:20:58.168 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 4073, 2024-03-28T22:20:58.168 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000782286, 2024-03-28T22:20:58.168 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000192 2024-03-28T22:20:58.168 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.168 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_encrypted_bytes": 410768, 2024-03-28T22:20:58.168 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_encrypted_bytes": 246768 2024-03-28T22:20:58.168 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.168 INFO:teuthology.orchestra.run.smithi053.stdout: "cct": { 2024-03-28T22:20:58.169 INFO:teuthology.orchestra.run.smithi053.stdout: "total_workers": 1, 2024-03-28T22:20:58.169 INFO:teuthology.orchestra.run.smithi053.stdout: "unhealthy_workers": 0 2024-03-28T22:20:58.169 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.169 INFO:teuthology.orchestra.run.smithi053.stdout: "finisher-MDSRank": { 2024-03-28T22:20:58.169 INFO:teuthology.orchestra.run.smithi053.stdout: "queue_len": 0, 2024-03-28T22:20:58.169 INFO:teuthology.orchestra.run.smithi053.stdout: "complete_latency": { 2024-03-28T22:20:58.169 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 722, 2024-03-28T22:20:58.169 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.729894107, 2024-03-28T22:20:58.170 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.001010933 2024-03-28T22:20:58.170 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-03-28T22:20:58.170 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.170 INFO:teuthology.orchestra.run.smithi053.stdout: "finisher-PurgeQueue": { 2024-03-28T22:20:58.170 INFO:teuthology.orchestra.run.smithi053.stdout: "queue_len": 0, 2024-03-28T22:20:58.170 INFO:teuthology.orchestra.run.smithi053.stdout: "complete_latency": { 2024-03-28T22:20:58.170 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 52, 2024-03-28T22:20:58.170 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.017904552, 2024-03-28T22:20:58.170 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000344318 2024-03-28T22:20:58.171 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-03-28T22:20:58.171 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.171 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": { 2024-03-28T22:20:58.171 INFO:teuthology.orchestra.run.smithi053.stdout: "request": 1202, 2024-03-28T22:20:58.171 INFO:teuthology.orchestra.run.smithi053.stdout: "reply": 1121, 2024-03-28T22:20:58.171 INFO:teuthology.orchestra.run.smithi053.stdout: "reply_latency": { 2024-03-28T22:20:58.171 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 1121, 2024-03-28T22:20:58.171 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 6.658356408, 2024-03-28T22:20:58.172 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.005939657 2024-03-28T22:20:58.172 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.172 INFO:teuthology.orchestra.run.smithi053.stdout: "slow_reply": 0, 2024-03-28T22:20:58.172 INFO:teuthology.orchestra.run.smithi053.stdout: "forward": 81, 2024-03-28T22:20:58.172 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_fetch_complete": 0, 2024-03-28T22:20:58.172 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_fetch_keys": 0, 2024-03-28T22:20:58.172 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_commit": 30, 2024-03-28T22:20:58.172 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_split": 2, 2024-03-28T22:20:58.172 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_merge": 0, 2024-03-28T22:20:58.173 INFO:teuthology.orchestra.run.smithi053.stdout: "inodes": 104, 2024-03-28T22:20:58.173 INFO:teuthology.orchestra.run.smithi053.stdout: "inodes_top": 18, 2024-03-28T22:20:58.173 INFO:teuthology.orchestra.run.smithi053.stdout: "inodes_bottom": 61, 2024-03-28T22:20:58.173 INFO:teuthology.orchestra.run.smithi053.stdout: "inodes_pin_tail": 25, 2024-03-28T22:20:58.173 INFO:teuthology.orchestra.run.smithi053.stdout: "inodes_pinned": 79, 2024-03-28T22:20:58.173 INFO:teuthology.orchestra.run.smithi053.stdout: "inodes_expired": 870, 2024-03-28T22:20:58.173 INFO:teuthology.orchestra.run.smithi053.stdout: "inodes_with_caps": 25, 2024-03-28T22:20:58.173 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": 25, 2024-03-28T22:20:58.174 INFO:teuthology.orchestra.run.smithi053.stdout: "subtrees": 6, 2024-03-28T22:20:58.174 INFO:teuthology.orchestra.run.smithi053.stdout: "traverse": 1382, 2024-03-28T22:20:58.174 INFO:teuthology.orchestra.run.smithi053.stdout: "traverse_hit": 953, 2024-03-28T22:20:58.174 INFO:teuthology.orchestra.run.smithi053.stdout: "traverse_forward": 78, 2024-03-28T22:20:58.174 INFO:teuthology.orchestra.run.smithi053.stdout: "traverse_discover": 8, 2024-03-28T22:20:58.174 INFO:teuthology.orchestra.run.smithi053.stdout: "traverse_dir_fetch": 0, 2024-03-28T22:20:58.174 INFO:teuthology.orchestra.run.smithi053.stdout: "traverse_remote_ino": 0, 2024-03-28T22:20:58.174 INFO:teuthology.orchestra.run.smithi053.stdout: "traverse_lock": 4, 2024-03-28T22:20:58.175 INFO:teuthology.orchestra.run.smithi053.stdout: "load_cent": 0, 2024-03-28T22:20:58.175 INFO:teuthology.orchestra.run.smithi053.stdout: "q": 0, 2024-03-28T22:20:58.175 INFO:teuthology.orchestra.run.smithi053.stdout: "exported": 26, 2024-03-28T22:20:58.175 INFO:teuthology.orchestra.run.smithi053.stdout: "exported_inodes": 866, 2024-03-28T22:20:58.175 INFO:teuthology.orchestra.run.smithi053.stdout: "imported": 3, 2024-03-28T22:20:58.175 INFO:teuthology.orchestra.run.smithi053.stdout: "imported_inodes": 22, 2024-03-28T22:20:58.175 INFO:teuthology.orchestra.run.smithi053.stdout: "openino_dir_fetch": 0, 2024-03-28T22:20:58.175 INFO:teuthology.orchestra.run.smithi053.stdout: "openino_backtrace_fetch": 560, 2024-03-28T22:20:58.175 INFO:teuthology.orchestra.run.smithi053.stdout: "openino_peer_discover": 0, 2024-03-28T22:20:58.176 INFO:teuthology.orchestra.run.smithi053.stdout: "root_rfiles": 3452, 2024-03-28T22:20:58.176 INFO:teuthology.orchestra.run.smithi053.stdout: "root_rbytes": 26525988, 2024-03-28T22:20:58.176 INFO:teuthology.orchestra.run.smithi053.stdout: "root_rsnaps": 6, 2024-03-28T22:20:58.176 INFO:teuthology.orchestra.run.smithi053.stdout: "scrub_backtrace_fetch": 560, 2024-03-28T22:20:58.176 INFO:teuthology.orchestra.run.smithi053.stdout: "scrub_set_tag": 0, 2024-03-28T22:20:58.176 INFO:teuthology.orchestra.run.smithi053.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T22:20:58.176 INFO:teuthology.orchestra.run.smithi053.stdout: "scrub_inotable_repaired": 0, 2024-03-28T22:20:58.176 INFO:teuthology.orchestra.run.smithi053.stdout: "scrub_dir_inodes": 151, 2024-03-28T22:20:58.176 INFO:teuthology.orchestra.run.smithi053.stdout: "scrub_dir_base_inodes": 19, 2024-03-28T22:20:58.177 INFO:teuthology.orchestra.run.smithi053.stdout: "scrub_dirfrag_rstats": 132, 2024-03-28T22:20:58.177 INFO:teuthology.orchestra.run.smithi053.stdout: "scrub_file_inodes": 409, 2024-03-28T22:20:58.177 INFO:teuthology.orchestra.run.smithi053.stdout: "handle_inode_file_caps": 0, 2024-03-28T22:20:58.177 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_cap_op_revoke": 31, 2024-03-28T22:20:58.177 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_cap_op_grant": 609, 2024-03-28T22:20:58.177 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T22:20:58.177 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T22:20:58.177 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T22:20:58.177 INFO:teuthology.orchestra.run.smithi053.stdout: "handle_client_caps": 116, 2024-03-28T22:20:58.178 INFO:teuthology.orchestra.run.smithi053.stdout: "handle_client_caps_dirty": 65, 2024-03-28T22:20:58.178 INFO:teuthology.orchestra.run.smithi053.stdout: "handle_client_cap_release": 18, 2024-03-28T22:20:58.178 INFO:teuthology.orchestra.run.smithi053.stdout: "process_request_cap_release": 103 2024-03-28T22:20:58.178 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.178 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_cache": { 2024-03-28T22:20:58.178 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_update": 11, 2024-03-28T22:20:58.178 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_update_receipt": 18, 2024-03-28T22:20:58.178 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_try_discover": 12, 2024-03-28T22:20:58.178 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_send_discover": 14, 2024-03-28T22:20:58.179 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_handle_discover": 44, 2024-03-28T22:20:58.179 INFO:teuthology.orchestra.run.smithi053.stdout: "num_strays": 0, 2024-03-28T22:20:58.179 INFO:teuthology.orchestra.run.smithi053.stdout: "num_strays_delayed": 0, 2024-03-28T22:20:58.179 INFO:teuthology.orchestra.run.smithi053.stdout: "num_strays_enqueuing": 0, 2024-03-28T22:20:58.179 INFO:teuthology.orchestra.run.smithi053.stdout: "strays_created": 23, 2024-03-28T22:20:58.179 INFO:teuthology.orchestra.run.smithi053.stdout: "strays_enqueued": 21, 2024-03-28T22:20:58.179 INFO:teuthology.orchestra.run.smithi053.stdout: "strays_reintegrated": 2, 2024-03-28T22:20:58.179 INFO:teuthology.orchestra.run.smithi053.stdout: "strays_migrated": 0, 2024-03-28T22:20:58.179 INFO:teuthology.orchestra.run.smithi053.stdout: "num_recovering_processing": 0, 2024-03-28T22:20:58.180 INFO:teuthology.orchestra.run.smithi053.stdout: "num_recovering_enqueued": 0, 2024-03-28T22:20:58.180 INFO:teuthology.orchestra.run.smithi053.stdout: "num_recovering_prioritized": 0, 2024-03-28T22:20:58.180 INFO:teuthology.orchestra.run.smithi053.stdout: "recovery_started": 0, 2024-03-28T22:20:58.180 INFO:teuthology.orchestra.run.smithi053.stdout: "recovery_completed": 0, 2024-03-28T22:20:58.180 INFO:teuthology.orchestra.run.smithi053.stdout: "ireq_quiesce_path": 0, 2024-03-28T22:20:58.180 INFO:teuthology.orchestra.run.smithi053.stdout: "ireq_quiesce_inode": 0, 2024-03-28T22:20:58.180 INFO:teuthology.orchestra.run.smithi053.stdout: "ireq_enqueue_scrub": 19, 2024-03-28T22:20:58.180 INFO:teuthology.orchestra.run.smithi053.stdout: "ireq_exportdir": 26, 2024-03-28T22:20:58.180 INFO:teuthology.orchestra.run.smithi053.stdout: "ireq_flush": 0, 2024-03-28T22:20:58.181 INFO:teuthology.orchestra.run.smithi053.stdout: "ireq_fragmentdir": 2, 2024-03-28T22:20:58.181 INFO:teuthology.orchestra.run.smithi053.stdout: "ireq_fragstats": 0, 2024-03-28T22:20:58.181 INFO:teuthology.orchestra.run.smithi053.stdout: "ireq_inodestats": 0 2024-03-28T22:20:58.181 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.181 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_log": { 2024-03-28T22:20:58.181 INFO:teuthology.orchestra.run.smithi053.stdout: "evlrg": 0, 2024-03-28T22:20:58.181 INFO:teuthology.orchestra.run.smithi053.stdout: "evadd": 891, 2024-03-28T22:20:58.181 INFO:teuthology.orchestra.run.smithi053.stdout: "evex": 0, 2024-03-28T22:20:58.181 INFO:teuthology.orchestra.run.smithi053.stdout: "evtrm": 0, 2024-03-28T22:20:58.181 INFO:teuthology.orchestra.run.smithi053.stdout: "ev": 891, 2024-03-28T22:20:58.182 INFO:teuthology.orchestra.run.smithi053.stdout: "evexg": 0, 2024-03-28T22:20:58.182 INFO:teuthology.orchestra.run.smithi053.stdout: "evexd": 0, 2024-03-28T22:20:58.182 INFO:teuthology.orchestra.run.smithi053.stdout: "segadd": 2, 2024-03-28T22:20:58.182 INFO:teuthology.orchestra.run.smithi053.stdout: "segex": 0, 2024-03-28T22:20:58.182 INFO:teuthology.orchestra.run.smithi053.stdout: "segtrm": 0, 2024-03-28T22:20:58.182 INFO:teuthology.orchestra.run.smithi053.stdout: "seg": 2, 2024-03-28T22:20:58.182 INFO:teuthology.orchestra.run.smithi053.stdout: "segmjr": 2, 2024-03-28T22:20:58.182 INFO:teuthology.orchestra.run.smithi053.stdout: "segexg": 0, 2024-03-28T22:20:58.183 INFO:teuthology.orchestra.run.smithi053.stdout: "segexd": 0, 2024-03-28T22:20:58.183 INFO:teuthology.orchestra.run.smithi053.stdout: "expos": 4194304, 2024-03-28T22:20:58.183 INFO:teuthology.orchestra.run.smithi053.stdout: "wrpos": 6399717, 2024-03-28T22:20:58.183 INFO:teuthology.orchestra.run.smithi053.stdout: "rdpos": 0, 2024-03-28T22:20:58.183 INFO:teuthology.orchestra.run.smithi053.stdout: "jlat": { 2024-03-28T22:20:58.183 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 158, 2024-03-28T22:20:58.183 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.657632977, 2024-03-28T22:20:58.183 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.004162234 2024-03-28T22:20:58.183 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.184 INFO:teuthology.orchestra.run.smithi053.stdout: "replayed": 0 2024-03-28T22:20:58.184 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.184 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_mem": { 2024-03-28T22:20:58.184 INFO:teuthology.orchestra.run.smithi053.stdout: "ino": 101, 2024-03-28T22:20:58.184 INFO:teuthology.orchestra.run.smithi053.stdout: "ino+": 708, 2024-03-28T22:20:58.184 INFO:teuthology.orchestra.run.smithi053.stdout: "ino-": 607, 2024-03-28T22:20:58.184 INFO:teuthology.orchestra.run.smithi053.stdout: "dir": 128, 2024-03-28T22:20:58.184 INFO:teuthology.orchestra.run.smithi053.stdout: "dir+": 156, 2024-03-28T22:20:58.185 INFO:teuthology.orchestra.run.smithi053.stdout: "dir-": 28, 2024-03-28T22:20:58.185 INFO:teuthology.orchestra.run.smithi053.stdout: "dn": 104, 2024-03-28T22:20:58.185 INFO:teuthology.orchestra.run.smithi053.stdout: "dn+": 1000, 2024-03-28T22:20:58.185 INFO:teuthology.orchestra.run.smithi053.stdout: "dn-": 896, 2024-03-28T22:20:58.185 INFO:teuthology.orchestra.run.smithi053.stdout: "cap": 25, 2024-03-28T22:20:58.185 INFO:teuthology.orchestra.run.smithi053.stdout: "cap+": 633, 2024-03-28T22:20:58.185 INFO:teuthology.orchestra.run.smithi053.stdout: "cap-": 608, 2024-03-28T22:20:58.185 INFO:teuthology.orchestra.run.smithi053.stdout: "rss": 63356, 2024-03-28T22:20:58.185 INFO:teuthology.orchestra.run.smithi053.stdout: "heap": 231804 2024-03-28T22:20:58.186 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.186 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_server": { 2024-03-28T22:20:58.186 INFO:teuthology.orchestra.run.smithi053.stdout: "dispatch_client_request": 1284, 2024-03-28T22:20:58.186 INFO:teuthology.orchestra.run.smithi053.stdout: "dispatch_server_request": 2, 2024-03-28T22:20:58.186 INFO:teuthology.orchestra.run.smithi053.stdout: "handle_client_request": 1202, 2024-03-28T22:20:58.186 INFO:teuthology.orchestra.run.smithi053.stdout: "handle_client_session": 59, 2024-03-28T22:20:58.186 INFO:teuthology.orchestra.run.smithi053.stdout: "handle_peer_request": 15, 2024-03-28T22:20:58.186 INFO:teuthology.orchestra.run.smithi053.stdout: "req_create_latency": { 2024-03-28T22:20:58.187 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 561, 2024-03-28T22:20:58.187 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.893407685, 2024-03-28T22:20:58.187 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.001592527 2024-03-28T22:20:58.187 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.187 INFO:teuthology.orchestra.run.smithi053.stdout: "req_getattr_latency": { 2024-03-28T22:20:58.187 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 101, 2024-03-28T22:20:58.187 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.021820144, 2024-03-28T22:20:58.187 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000216041 2024-03-28T22:20:58.188 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.188 INFO:teuthology.orchestra.run.smithi053.stdout: "req_getfilelock_latency": { 2024-03-28T22:20:58.188 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:20:58.188 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:20:58.188 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:20:58.188 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.188 INFO:teuthology.orchestra.run.smithi053.stdout: "req_link_latency": { 2024-03-28T22:20:58.188 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 2, 2024-03-28T22:20:58.188 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.003245183, 2024-03-28T22:20:58.189 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.001622591 2024-03-28T22:20:58.189 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.189 INFO:teuthology.orchestra.run.smithi053.stdout: "req_lookup_latency": { 2024-03-28T22:20:58.189 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 49, 2024-03-28T22:20:58.189 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 4.957901164, 2024-03-28T22:20:58.189 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.101181656 2024-03-28T22:20:58.189 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.189 INFO:teuthology.orchestra.run.smithi053.stdout: "req_lookuphash_latency": { 2024-03-28T22:20:58.190 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:20:58.190 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:20:58.190 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:20:58.190 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.190 INFO:teuthology.orchestra.run.smithi053.stdout: "req_lookupino_latency": { 2024-03-28T22:20:58.190 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:20:58.190 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:20:58.190 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:20:58.190 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.191 INFO:teuthology.orchestra.run.smithi053.stdout: "req_lookupname_latency": { 2024-03-28T22:20:58.191 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:20:58.191 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:20:58.191 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:20:58.191 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.191 INFO:teuthology.orchestra.run.smithi053.stdout: "req_lookupparent_latency": { 2024-03-28T22:20:58.191 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:20:58.191 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:20:58.192 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:20:58.192 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.192 INFO:teuthology.orchestra.run.smithi053.stdout: "req_lookupsnap_latency": { 2024-03-28T22:20:58.192 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:20:58.192 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:20:58.192 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:20:58.192 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.192 INFO:teuthology.orchestra.run.smithi053.stdout: "req_lssnap_latency": { 2024-03-28T22:20:58.192 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 10, 2024-03-28T22:20:58.193 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.001888520, 2024-03-28T22:20:58.193 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000188852 2024-03-28T22:20:58.193 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.193 INFO:teuthology.orchestra.run.smithi053.stdout: "req_mkdir_latency": { 2024-03-28T22:20:58.194 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 45, 2024-03-28T22:20:58.194 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.443033189, 2024-03-28T22:20:58.194 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.009845181 2024-03-28T22:20:58.194 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.194 INFO:teuthology.orchestra.run.smithi053.stdout: "req_mknod_latency": { 2024-03-28T22:20:58.194 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:20:58.194 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:20:58.194 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:20:58.195 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.195 INFO:teuthology.orchestra.run.smithi053.stdout: "req_mksnap_latency": { 2024-03-28T22:20:58.195 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 10, 2024-03-28T22:20:58.195 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.091437704, 2024-03-28T22:20:58.195 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.009143770 2024-03-28T22:20:58.195 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.195 INFO:teuthology.orchestra.run.smithi053.stdout: "req_open_latency": { 2024-03-28T22:20:58.195 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 290, 2024-03-28T22:20:58.196 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.082567317, 2024-03-28T22:20:58.196 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000284714 2024-03-28T22:20:58.196 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.196 INFO:teuthology.orchestra.run.smithi053.stdout: "req_readdir_latency": { 2024-03-28T22:20:58.196 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 6, 2024-03-28T22:20:58.196 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.019356633, 2024-03-28T22:20:58.196 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.003226105 2024-03-28T22:20:58.196 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.197 INFO:teuthology.orchestra.run.smithi053.stdout: "req_rename_latency": { 2024-03-28T22:20:58.197 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 19, 2024-03-28T22:20:58.197 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.066313701, 2024-03-28T22:20:58.197 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.003490194 2024-03-28T22:20:58.197 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.197 INFO:teuthology.orchestra.run.smithi053.stdout: "req_renamesnap_latency": { 2024-03-28T22:20:58.197 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:20:58.197 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:20:58.197 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:20:58.198 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.198 INFO:teuthology.orchestra.run.smithi053.stdout: "req_snapdiff_latency": { 2024-03-28T22:20:58.198 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:20:58.198 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:20:58.198 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:20:58.198 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.198 INFO:teuthology.orchestra.run.smithi053.stdout: "req_rmdir_latency": { 2024-03-28T22:20:58.198 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:20:58.199 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:20:58.199 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:20:58.199 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.199 INFO:teuthology.orchestra.run.smithi053.stdout: "req_rmsnap_latency": { 2024-03-28T22:20:58.199 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 4, 2024-03-28T22:20:58.199 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.017897457, 2024-03-28T22:20:58.199 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.004474364 2024-03-28T22:20:58.199 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.199 INFO:teuthology.orchestra.run.smithi053.stdout: "req_rmxattr_latency": { 2024-03-28T22:20:58.200 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:20:58.200 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:20:58.200 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:20:58.200 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.200 INFO:teuthology.orchestra.run.smithi053.stdout: "req_setattr_latency": { 2024-03-28T22:20:58.200 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 1, 2024-03-28T22:20:58.200 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000894484, 2024-03-28T22:20:58.200 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000894484 2024-03-28T22:20:58.201 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.201 INFO:teuthology.orchestra.run.smithi053.stdout: "req_setdirlayout_latency": { 2024-03-28T22:20:58.201 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:20:58.201 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:20:58.201 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:20:58.201 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.201 INFO:teuthology.orchestra.run.smithi053.stdout: "req_setfilelock_latency": { 2024-03-28T22:20:58.201 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:20:58.202 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:20:58.202 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:20:58.202 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.202 INFO:teuthology.orchestra.run.smithi053.stdout: "req_setlayout_latency": { 2024-03-28T22:20:58.202 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:20:58.202 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:20:58.202 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:20:58.202 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.203 INFO:teuthology.orchestra.run.smithi053.stdout: "req_setxattr_latency": { 2024-03-28T22:20:58.203 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 8, 2024-03-28T22:20:58.203 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.023516949, 2024-03-28T22:20:58.203 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.002939618 2024-03-28T22:20:58.203 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.203 INFO:teuthology.orchestra.run.smithi053.stdout: "req_symlink_latency": { 2024-03-28T22:20:58.203 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 1, 2024-03-28T22:20:58.203 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.001209657, 2024-03-28T22:20:58.203 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.001209657 2024-03-28T22:20:58.204 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.204 INFO:teuthology.orchestra.run.smithi053.stdout: "req_unlink_latency": { 2024-03-28T22:20:58.204 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 14, 2024-03-28T22:20:58.204 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.033866621, 2024-03-28T22:20:58.204 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.002419044 2024-03-28T22:20:58.204 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.204 INFO:teuthology.orchestra.run.smithi053.stdout: "cap_revoke_eviction": 0, 2024-03-28T22:20:58.204 INFO:teuthology.orchestra.run.smithi053.stdout: "cap_acquisition_throttle": 0, 2024-03-28T22:20:58.205 INFO:teuthology.orchestra.run.smithi053.stdout: "req_getvxattr_latency": { 2024-03-28T22:20:58.205 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:20:58.205 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:20:58.205 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:20:58.205 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-03-28T22:20:58.205 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.205 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_sessions": { 2024-03-28T22:20:58.205 INFO:teuthology.orchestra.run.smithi053.stdout: "session_count": 2, 2024-03-28T22:20:58.205 INFO:teuthology.orchestra.run.smithi053.stdout: "session_add": 6, 2024-03-28T22:20:58.206 INFO:teuthology.orchestra.run.smithi053.stdout: "session_remove": 4, 2024-03-28T22:20:58.206 INFO:teuthology.orchestra.run.smithi053.stdout: "sessions_open": 2, 2024-03-28T22:20:58.206 INFO:teuthology.orchestra.run.smithi053.stdout: "sessions_stale": 0, 2024-03-28T22:20:58.206 INFO:teuthology.orchestra.run.smithi053.stdout: "total_load": 30, 2024-03-28T22:20:58.206 INFO:teuthology.orchestra.run.smithi053.stdout: "average_load": 15, 2024-03-28T22:20:58.206 INFO:teuthology.orchestra.run.smithi053.stdout: "avg_session_uptime": 614, 2024-03-28T22:20:58.206 INFO:teuthology.orchestra.run.smithi053.stdout: "mdthresh_evicted": 0 2024-03-28T22:20:58.206 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.207 INFO:teuthology.orchestra.run.smithi053.stdout: "mempool": { 2024-03-28T22:20:58.207 INFO:teuthology.orchestra.run.smithi053.stdout: "bloom_filter_bytes": 0, 2024-03-28T22:20:58.207 INFO:teuthology.orchestra.run.smithi053.stdout: "bloom_filter_items": 0, 2024-03-28T22:20:58.207 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T22:20:58.207 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_alloc_items": 0, 2024-03-28T22:20:58.207 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T22:20:58.207 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_data_items": 0, 2024-03-28T22:20:58.207 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T22:20:58.207 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T22:20:58.208 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T22:20:58.208 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T22:20:58.208 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T22:20:58.208 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_other_items": 0, 2024-03-28T22:20:58.208 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T22:20:58.208 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T22:20:58.208 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_extent_bytes": 0, 2024-03-28T22:20:58.208 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_extent_items": 0, 2024-03-28T22:20:58.209 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_blob_bytes": 0, 2024-03-28T22:20:58.209 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_blob_items": 0, 2024-03-28T22:20:58.209 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T22:20:58.209 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T22:20:58.209 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T22:20:58.209 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T22:20:58.209 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T22:20:58.209 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_fsck_items": 0, 2024-03-28T22:20:58.209 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_txc_bytes": 0, 2024-03-28T22:20:58.210 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_txc_items": 0, 2024-03-28T22:20:58.210 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T22:20:58.210 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T22:20:58.210 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_writing_bytes": 0, 2024-03-28T22:20:58.210 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_writing_items": 0, 2024-03-28T22:20:58.210 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_bytes": 0, 2024-03-28T22:20:58.210 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_items": 0, 2024-03-28T22:20:58.210 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T22:20:58.211 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_file_reader_items": 0, 2024-03-28T22:20:58.211 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T22:20:58.211 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_file_writer_items": 0, 2024-03-28T22:20:58.211 INFO:teuthology.orchestra.run.smithi053.stdout: "buffer_anon_bytes": 301761, 2024-03-28T22:20:58.211 INFO:teuthology.orchestra.run.smithi053.stdout: "buffer_anon_items": 189, 2024-03-28T22:20:58.211 INFO:teuthology.orchestra.run.smithi053.stdout: "buffer_meta_bytes": 1056, 2024-03-28T22:20:58.211 INFO:teuthology.orchestra.run.smithi053.stdout: "buffer_meta_items": 12, 2024-03-28T22:20:58.211 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_bytes": 0, 2024-03-28T22:20:58.211 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_items": 0, 2024-03-28T22:20:58.212 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_mapbl_bytes": 0, 2024-03-28T22:20:58.212 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_mapbl_items": 0, 2024-03-28T22:20:58.212 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_pglog_bytes": 0, 2024-03-28T22:20:58.212 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_pglog_items": 0, 2024-03-28T22:20:58.212 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_bytes": 8160, 2024-03-28T22:20:58.212 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_items": 186, 2024-03-28T22:20:58.212 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T22:20:58.212 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_mapping_items": 0, 2024-03-28T22:20:58.213 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap_bytes": 0, 2024-03-28T22:20:58.213 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap_items": 0, 2024-03-28T22:20:58.213 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_co_bytes": 524215, 2024-03-28T22:20:58.213 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_co_items": 5226, 2024-03-28T22:20:58.213 INFO:teuthology.orchestra.run.smithi053.stdout: "unittest_1_bytes": 0, 2024-03-28T22:20:58.213 INFO:teuthology.orchestra.run.smithi053.stdout: "unittest_1_items": 0, 2024-03-28T22:20:58.213 INFO:teuthology.orchestra.run.smithi053.stdout: "unittest_2_bytes": 0, 2024-03-28T22:20:58.213 INFO:teuthology.orchestra.run.smithi053.stdout: "unittest_2_items": 0 2024-03-28T22:20:58.213 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.214 INFO:teuthology.orchestra.run.smithi053.stdout: "objecter": { 2024-03-28T22:20:58.214 INFO:teuthology.orchestra.run.smithi053.stdout: "op_active": 0, 2024-03-28T22:20:58.214 INFO:teuthology.orchestra.run.smithi053.stdout: "op_laggy": 0, 2024-03-28T22:20:58.214 INFO:teuthology.orchestra.run.smithi053.stdout: "op_send": 1403, 2024-03-28T22:20:58.214 INFO:teuthology.orchestra.run.smithi053.stdout: "op_send_bytes": 2481449, 2024-03-28T22:20:58.214 INFO:teuthology.orchestra.run.smithi053.stdout: "op_resend": 32, 2024-03-28T22:20:58.214 INFO:teuthology.orchestra.run.smithi053.stdout: "op_reply": 1403, 2024-03-28T22:20:58.214 INFO:teuthology.orchestra.run.smithi053.stdout: "op_latency": { 2024-03-28T22:20:58.214 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 1403, 2024-03-28T22:20:58.215 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 20.639577528, 2024-03-28T22:20:58.215 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.014711031 2024-03-28T22:20:58.215 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.215 INFO:teuthology.orchestra.run.smithi053.stdout: "op_inflight": 0, 2024-03-28T22:20:58.215 INFO:teuthology.orchestra.run.smithi053.stdout: "oplen_avg": { 2024-03-28T22:20:58.215 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 1403, 2024-03-28T22:20:58.215 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 1476 2024-03-28T22:20:58.215 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.216 INFO:teuthology.orchestra.run.smithi053.stdout: "op": 1403, 2024-03-28T22:20:58.216 INFO:teuthology.orchestra.run.smithi053.stdout: "op_r": 603, 2024-03-28T22:20:58.216 INFO:teuthology.orchestra.run.smithi053.stdout: "op_w": 800, 2024-03-28T22:20:58.216 INFO:teuthology.orchestra.run.smithi053.stdout: "op_rmw": 0, 2024-03-28T22:20:58.216 INFO:teuthology.orchestra.run.smithi053.stdout: "op_pg": 0, 2024-03-28T22:20:58.216 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_stat": 0, 2024-03-28T22:20:58.216 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_create": 11, 2024-03-28T22:20:58.216 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_read": 24, 2024-03-28T22:20:58.216 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_write": 163, 2024-03-28T22:20:58.217 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_writefull": 31, 2024-03-28T22:20:58.217 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_writesame": 0, 2024-03-28T22:20:58.217 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_append": 0, 2024-03-28T22:20:58.217 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_zero": 0, 2024-03-28T22:20:58.217 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_truncate": 2, 2024-03-28T22:20:58.217 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_delete": 533, 2024-03-28T22:20:58.217 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_mapext": 0, 2024-03-28T22:20:58.217 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_sparse_read": 0, 2024-03-28T22:20:58.217 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_clonerange": 0, 2024-03-28T22:20:58.218 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_getxattr": 579, 2024-03-28T22:20:58.218 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_setxattr": 22, 2024-03-28T22:20:58.218 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_cmpxattr": 0, 2024-03-28T22:20:58.218 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_rmxattr": 0, 2024-03-28T22:20:58.218 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_resetxattrs": 0, 2024-03-28T22:20:58.218 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_call": 0, 2024-03-28T22:20:58.218 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_watch": 0, 2024-03-28T22:20:58.218 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_notify": 0, 2024-03-28T22:20:58.218 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T22:20:58.219 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_pgls": 0, 2024-03-28T22:20:58.219 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_pgls_filter": 0, 2024-03-28T22:20:58.219 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_other": 0, 2024-03-28T22:20:58.219 INFO:teuthology.orchestra.run.smithi053.stdout: "linger_active": 0, 2024-03-28T22:20:58.219 INFO:teuthology.orchestra.run.smithi053.stdout: "linger_send": 0, 2024-03-28T22:20:58.219 INFO:teuthology.orchestra.run.smithi053.stdout: "linger_resend": 0, 2024-03-28T22:20:58.219 INFO:teuthology.orchestra.run.smithi053.stdout: "linger_ping": 0, 2024-03-28T22:20:58.219 INFO:teuthology.orchestra.run.smithi053.stdout: "poolop_active": 0, 2024-03-28T22:20:58.220 INFO:teuthology.orchestra.run.smithi053.stdout: "poolop_send": 0, 2024-03-28T22:20:58.220 INFO:teuthology.orchestra.run.smithi053.stdout: "poolop_resend": 0, 2024-03-28T22:20:58.220 INFO:teuthology.orchestra.run.smithi053.stdout: "poolstat_active": 0, 2024-03-28T22:20:58.220 INFO:teuthology.orchestra.run.smithi053.stdout: "poolstat_send": 0, 2024-03-28T22:20:58.220 INFO:teuthology.orchestra.run.smithi053.stdout: "poolstat_resend": 0, 2024-03-28T22:20:58.220 INFO:teuthology.orchestra.run.smithi053.stdout: "statfs_active": 0, 2024-03-28T22:20:58.220 INFO:teuthology.orchestra.run.smithi053.stdout: "statfs_send": 0, 2024-03-28T22:20:58.220 INFO:teuthology.orchestra.run.smithi053.stdout: "statfs_resend": 0, 2024-03-28T22:20:58.220 INFO:teuthology.orchestra.run.smithi053.stdout: "command_active": 0, 2024-03-28T22:20:58.221 INFO:teuthology.orchestra.run.smithi053.stdout: "command_send": 0, 2024-03-28T22:20:58.221 INFO:teuthology.orchestra.run.smithi053.stdout: "command_resend": 0, 2024-03-28T22:20:58.221 INFO:teuthology.orchestra.run.smithi053.stdout: "map_epoch": 97, 2024-03-28T22:20:58.221 INFO:teuthology.orchestra.run.smithi053.stdout: "map_full": 0, 2024-03-28T22:20:58.221 INFO:teuthology.orchestra.run.smithi053.stdout: "map_inc": 12, 2024-03-28T22:20:58.221 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_sessions": 12, 2024-03-28T22:20:58.221 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_session_open": 12, 2024-03-28T22:20:58.221 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_session_close": 0, 2024-03-28T22:20:58.221 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_laggy": 0, 2024-03-28T22:20:58.222 INFO:teuthology.orchestra.run.smithi053.stdout: "omap_wr": 97, 2024-03-28T22:20:58.222 INFO:teuthology.orchestra.run.smithi053.stdout: "omap_rd": 0, 2024-03-28T22:20:58.222 INFO:teuthology.orchestra.run.smithi053.stdout: "omap_del": 14 2024-03-28T22:20:58.222 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.222 INFO:teuthology.orchestra.run.smithi053.stdout: "oft": { 2024-03-28T22:20:58.222 INFO:teuthology.orchestra.run.smithi053.stdout: "omap_total_objs": 1, 2024-03-28T22:20:58.222 INFO:teuthology.orchestra.run.smithi053.stdout: "omap_total_kv_pairs": 0, 2024-03-28T22:20:58.222 INFO:teuthology.orchestra.run.smithi053.stdout: "omap_total_updates": 0, 2024-03-28T22:20:58.223 INFO:teuthology.orchestra.run.smithi053.stdout: "omap_total_removes": 0 2024-03-28T22:20:58.223 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.223 INFO:teuthology.orchestra.run.smithi053.stdout: "osd-slow-ops": { 2024-03-28T22:20:58.223 INFO:teuthology.orchestra.run.smithi053.stdout: "slow_ops_count": 0 2024-03-28T22:20:58.223 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.223 INFO:teuthology.orchestra.run.smithi053.stdout: "purge_queue": { 2024-03-28T22:20:58.223 INFO:teuthology.orchestra.run.smithi053.stdout: "pq_executing_ops": 0, 2024-03-28T22:20:58.223 INFO:teuthology.orchestra.run.smithi053.stdout: "pq_executing_ops_high_water": 10, 2024-03-28T22:20:58.224 INFO:teuthology.orchestra.run.smithi053.stdout: "pq_executing": 0, 2024-03-28T22:20:58.224 INFO:teuthology.orchestra.run.smithi053.stdout: "pq_executing_high_water": 10, 2024-03-28T22:20:58.224 INFO:teuthology.orchestra.run.smithi053.stdout: "pq_executed_ops": 21, 2024-03-28T22:20:58.224 INFO:teuthology.orchestra.run.smithi053.stdout: "pq_executed": 21, 2024-03-28T22:20:58.224 INFO:teuthology.orchestra.run.smithi053.stdout: "pq_item_in_journal": 0 2024-03-28T22:20:58.224 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.224 INFO:teuthology.orchestra.run.smithi053.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T22:20:58.224 INFO:teuthology.orchestra.run.smithi053.stdout: "val": 0, 2024-03-28T22:20:58.224 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 104857600, 2024-03-28T22:20:58.225 INFO:teuthology.orchestra.run.smithi053.stdout: "get_started": 0, 2024-03-28T22:20:58.225 INFO:teuthology.orchestra.run.smithi053.stdout: "get": 7186, 2024-03-28T22:20:58.225 INFO:teuthology.orchestra.run.smithi053.stdout: "get_sum": 3572914, 2024-03-28T22:20:58.225 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_fail": 0, 2024-03-28T22:20:58.225 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_success": 7186, 2024-03-28T22:20:58.225 INFO:teuthology.orchestra.run.smithi053.stdout: "take": 0, 2024-03-28T22:20:58.225 INFO:teuthology.orchestra.run.smithi053.stdout: "take_sum": 0, 2024-03-28T22:20:58.226 INFO:teuthology.orchestra.run.smithi053.stdout: "put": 7186, 2024-03-28T22:20:58.226 INFO:teuthology.orchestra.run.smithi053.stdout: "put_sum": 3572914, 2024-03-28T22:20:58.226 INFO:teuthology.orchestra.run.smithi053.stdout: "wait": { 2024-03-28T22:20:58.226 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:20:58.226 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:20:58.226 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:20:58.226 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-03-28T22:20:58.226 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.226 INFO:teuthology.orchestra.run.smithi053.stdout: "throttle-objecter_bytes": { 2024-03-28T22:20:58.227 INFO:teuthology.orchestra.run.smithi053.stdout: "val": 0, 2024-03-28T22:20:58.227 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 104857600, 2024-03-28T22:20:58.227 INFO:teuthology.orchestra.run.smithi053.stdout: "get_started": 0, 2024-03-28T22:20:58.227 INFO:teuthology.orchestra.run.smithi053.stdout: "get": 0, 2024-03-28T22:20:58.227 INFO:teuthology.orchestra.run.smithi053.stdout: "get_sum": 0, 2024-03-28T22:20:58.227 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_fail": 0, 2024-03-28T22:20:58.227 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_success": 0, 2024-03-28T22:20:58.227 INFO:teuthology.orchestra.run.smithi053.stdout: "take": 1403, 2024-03-28T22:20:58.228 INFO:teuthology.orchestra.run.smithi053.stdout: "take_sum": 2483602, 2024-03-28T22:20:58.228 INFO:teuthology.orchestra.run.smithi053.stdout: "put": 1403, 2024-03-28T22:20:58.228 INFO:teuthology.orchestra.run.smithi053.stdout: "put_sum": 2483602, 2024-03-28T22:20:58.228 INFO:teuthology.orchestra.run.smithi053.stdout: "wait": { 2024-03-28T22:20:58.228 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:20:58.228 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:20:58.228 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:20:58.228 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-03-28T22:20:58.228 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.229 INFO:teuthology.orchestra.run.smithi053.stdout: "throttle-objecter_ops": { 2024-03-28T22:20:58.229 INFO:teuthology.orchestra.run.smithi053.stdout: "val": 0, 2024-03-28T22:20:58.229 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 1024, 2024-03-28T22:20:58.229 INFO:teuthology.orchestra.run.smithi053.stdout: "get_started": 0, 2024-03-28T22:20:58.229 INFO:teuthology.orchestra.run.smithi053.stdout: "get": 0, 2024-03-28T22:20:58.229 INFO:teuthology.orchestra.run.smithi053.stdout: "get_sum": 0, 2024-03-28T22:20:58.229 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_fail": 0, 2024-03-28T22:20:58.229 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_success": 0, 2024-03-28T22:20:58.230 INFO:teuthology.orchestra.run.smithi053.stdout: "take": 1403, 2024-03-28T22:20:58.230 INFO:teuthology.orchestra.run.smithi053.stdout: "take_sum": 1403, 2024-03-28T22:20:58.230 INFO:teuthology.orchestra.run.smithi053.stdout: "put": 1403, 2024-03-28T22:20:58.230 INFO:teuthology.orchestra.run.smithi053.stdout: "put_sum": 1403, 2024-03-28T22:20:58.230 INFO:teuthology.orchestra.run.smithi053.stdout: "wait": { 2024-03-28T22:20:58.230 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:20:58.230 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:20:58.230 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:20:58.231 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-03-28T22:20:58.231 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.231 INFO:teuthology.orchestra.run.smithi053.stdout: "throttle-write_buf_throttle": { 2024-03-28T22:20:58.231 INFO:teuthology.orchestra.run.smithi053.stdout: "val": 0, 2024-03-28T22:20:58.231 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 3758096384, 2024-03-28T22:20:58.231 INFO:teuthology.orchestra.run.smithi053.stdout: "get_started": 0, 2024-03-28T22:20:58.231 INFO:teuthology.orchestra.run.smithi053.stdout: "get": 21, 2024-03-28T22:20:58.231 INFO:teuthology.orchestra.run.smithi053.stdout: "get_sum": 2153, 2024-03-28T22:20:58.231 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_fail": 0, 2024-03-28T22:20:58.232 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_success": 21, 2024-03-28T22:20:58.232 INFO:teuthology.orchestra.run.smithi053.stdout: "take": 0, 2024-03-28T22:20:58.232 INFO:teuthology.orchestra.run.smithi053.stdout: "take_sum": 0, 2024-03-28T22:20:58.232 INFO:teuthology.orchestra.run.smithi053.stdout: "put": 5, 2024-03-28T22:20:58.232 INFO:teuthology.orchestra.run.smithi053.stdout: "put_sum": 2153, 2024-03-28T22:20:58.232 INFO:teuthology.orchestra.run.smithi053.stdout: "wait": { 2024-03-28T22:20:58.232 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:20:58.232 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:20:58.233 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:20:58.233 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-03-28T22:20:58.233 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:20:58.233 INFO:teuthology.orchestra.run.smithi053.stdout: "throttle-write_buf_throttle-0x55ea94332460": { 2024-03-28T22:20:58.233 INFO:teuthology.orchestra.run.smithi053.stdout: "val": 0, 2024-03-28T22:20:58.233 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 3758096384, 2024-03-28T22:20:58.233 INFO:teuthology.orchestra.run.smithi053.stdout: "get_started": 0, 2024-03-28T22:20:58.233 INFO:teuthology.orchestra.run.smithi053.stdout: "get": 891, 2024-03-28T22:20:58.233 INFO:teuthology.orchestra.run.smithi053.stdout: "get_sum": 2205413, 2024-03-28T22:20:58.234 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_fail": 0, 2024-03-28T22:20:58.234 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_success": 891, 2024-03-28T22:20:58.234 INFO:teuthology.orchestra.run.smithi053.stdout: "take": 0, 2024-03-28T22:20:58.234 INFO:teuthology.orchestra.run.smithi053.stdout: "take_sum": 0, 2024-03-28T22:20:58.234 INFO:teuthology.orchestra.run.smithi053.stdout: "put": 158, 2024-03-28T22:20:58.234 INFO:teuthology.orchestra.run.smithi053.stdout: "put_sum": 2205413, 2024-03-28T22:20:58.234 INFO:teuthology.orchestra.run.smithi053.stdout: "wait": { 2024-03-28T22:20:58.234 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:20:58.234 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:20:58.235 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:20:58.235 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-03-28T22:20:58.235 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-03-28T22:20:58.235 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-03-28T22:20:58.859 INFO:tasks.check_counter:Daemon mds.b mds.exported=26 2024-03-28T22:20:58.860 INFO:tasks.check_counter:Daemon mds.b mds.imported=3 2024-03-28T22:20:58.860 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=11 2024-03-28T22:20:58.860 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=18 2024-03-28T22:20:58.860 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-03-28T22:20:58.860 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=10 2024-03-28T22:20:58.860 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=4 2024-03-28T22:20:58.860 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=2 2024-03-28T22:20:58.861 DEBUG:tasks.check_counter:Getting stats from e 2024-03-28T22:20:58.861 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-28T22:20:59.178 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T22:20:59.809 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:20:59 smithi053 ceph-mon[35979]: pgmap v323: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:20:59.930 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:20:59 smithi063 ceph-mon[36249]: pgmap v323: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:20:59.934 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:20:59 smithi016 ceph-mon[32374]: pgmap v323: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:21:01.097 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-03-28T22:21:01.097 INFO:teuthology.orchestra.run.smithi053.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T22:21:01.097 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_messages": 17596, 2024-03-28T22:21:01.097 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages": 18126, 2024-03-28T22:21:01.097 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_bytes": 4464128, 2024-03-28T22:21:01.097 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_bytes": 23954392, 2024-03-28T22:21:01.097 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_created_connections": 17, 2024-03-28T22:21:01.098 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_active_connections": 4, 2024-03-28T22:21:01.098 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_total_time": 1.326519280, 2024-03-28T22:21:01.098 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_send_time": 0.575270660, 2024-03-28T22:21:01.098 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_recv_time": 0.278600411, 2024-03-28T22:21:01.098 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_fast_dispatch_time": 0.381923079, 2024-03-28T22:21:01.098 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:21:01.098 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 18124, 2024-03-28T22:21:01.099 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.658121138, 2024-03-28T22:21:01.099 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000036312 2024-03-28T22:21:01.099 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.099 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:21:01.099 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 1135, 2024-03-28T22:21:01.099 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000277254, 2024-03-28T22:21:01.099 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000244 2024-03-28T22:21:01.099 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.099 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_encrypted_bytes": 1856, 2024-03-28T22:21:01.100 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_encrypted_bytes": 316112 2024-03-28T22:21:01.100 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.100 INFO:teuthology.orchestra.run.smithi053.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T22:21:01.100 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_messages": 49838, 2024-03-28T22:21:01.100 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages": 65792, 2024-03-28T22:21:01.100 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_bytes": 13726768, 2024-03-28T22:21:01.100 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_bytes": 52041884, 2024-03-28T22:21:01.100 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_created_connections": 47, 2024-03-28T22:21:01.100 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_active_connections": 10, 2024-03-28T22:21:01.101 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_total_time": 2.578957701, 2024-03-28T22:21:01.101 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_send_time": 1.051565143, 2024-03-28T22:21:01.101 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_recv_time": 5.528615347, 2024-03-28T22:21:01.101 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_fast_dispatch_time": 0.519276298, 2024-03-28T22:21:01.101 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:21:01.101 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 65790, 2024-03-28T22:21:01.101 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 3.224342547, 2024-03-28T22:21:01.101 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000049009 2024-03-28T22:21:01.102 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.102 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:21:01.102 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 37960, 2024-03-28T22:21:01.102 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.006779582, 2024-03-28T22:21:01.102 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000178 2024-03-28T22:21:01.102 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.102 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_encrypted_bytes": 4160, 2024-03-28T22:21:01.102 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_encrypted_bytes": 131440 2024-03-28T22:21:01.102 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.103 INFO:teuthology.orchestra.run.smithi053.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T22:21:01.103 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_messages": 19230, 2024-03-28T22:21:01.103 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages": 19817, 2024-03-28T22:21:01.103 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_bytes": 7440684, 2024-03-28T22:21:01.103 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_bytes": 15037212, 2024-03-28T22:21:01.103 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_created_connections": 9, 2024-03-28T22:21:01.103 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_active_connections": 6, 2024-03-28T22:21:01.103 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_total_time": 1.474328463, 2024-03-28T22:21:01.104 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_send_time": 0.643937658, 2024-03-28T22:21:01.104 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_recv_time": 0.377040203, 2024-03-28T22:21:01.104 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_fast_dispatch_time": 0.390754381, 2024-03-28T22:21:01.104 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:21:01.104 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 19816, 2024-03-28T22:21:01.104 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.708534156, 2024-03-28T22:21:01.104 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000035755 2024-03-28T22:21:01.104 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.104 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:21:01.105 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 2936, 2024-03-28T22:21:01.105 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000710077, 2024-03-28T22:21:01.105 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000241 2024-03-28T22:21:01.105 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.105 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_encrypted_bytes": 1637264, 2024-03-28T22:21:01.105 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_encrypted_bytes": 156432 2024-03-28T22:21:01.105 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.105 INFO:teuthology.orchestra.run.smithi053.stdout: "cct": { 2024-03-28T22:21:01.106 INFO:teuthology.orchestra.run.smithi053.stdout: "total_workers": 1, 2024-03-28T22:21:01.106 INFO:teuthology.orchestra.run.smithi053.stdout: "unhealthy_workers": 0 2024-03-28T22:21:01.106 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.106 INFO:teuthology.orchestra.run.smithi053.stdout: "finisher-MDSRank": { 2024-03-28T22:21:01.106 INFO:teuthology.orchestra.run.smithi053.stdout: "queue_len": 0, 2024-03-28T22:21:01.106 INFO:teuthology.orchestra.run.smithi053.stdout: "complete_latency": { 2024-03-28T22:21:01.106 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 30444, 2024-03-28T22:21:01.106 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 31.702375050, 2024-03-28T22:21:01.106 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.001041334 2024-03-28T22:21:01.107 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-03-28T22:21:01.107 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.107 INFO:teuthology.orchestra.run.smithi053.stdout: "finisher-PurgeQueue": { 2024-03-28T22:21:01.107 INFO:teuthology.orchestra.run.smithi053.stdout: "queue_len": 0, 2024-03-28T22:21:01.107 INFO:teuthology.orchestra.run.smithi053.stdout: "complete_latency": { 2024-03-28T22:21:01.107 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 4777, 2024-03-28T22:21:01.107 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 1.702096217, 2024-03-28T22:21:01.107 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000356310 2024-03-28T22:21:01.108 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-03-28T22:21:01.108 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.108 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": { 2024-03-28T22:21:01.108 INFO:teuthology.orchestra.run.smithi053.stdout: "request": 10126, 2024-03-28T22:21:01.108 INFO:teuthology.orchestra.run.smithi053.stdout: "reply": 10065, 2024-03-28T22:21:01.108 INFO:teuthology.orchestra.run.smithi053.stdout: "reply_latency": { 2024-03-28T22:21:01.108 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 10065, 2024-03-28T22:21:01.108 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 38.308585722, 2024-03-28T22:21:01.109 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.003806118 2024-03-28T22:21:01.109 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.109 INFO:teuthology.orchestra.run.smithi053.stdout: "slow_reply": 0, 2024-03-28T22:21:01.109 INFO:teuthology.orchestra.run.smithi053.stdout: "forward": 61, 2024-03-28T22:21:01.109 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_fetch_complete": 0, 2024-03-28T22:21:01.109 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_fetch_keys": 0, 2024-03-28T22:21:01.109 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_commit": 247, 2024-03-28T22:21:01.109 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_split": 29, 2024-03-28T22:21:01.109 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_merge": 0, 2024-03-28T22:21:01.110 INFO:teuthology.orchestra.run.smithi053.stdout: "inodes": 8362, 2024-03-28T22:21:01.110 INFO:teuthology.orchestra.run.smithi053.stdout: "inodes_top": 794, 2024-03-28T22:21:01.110 INFO:teuthology.orchestra.run.smithi053.stdout: "inodes_bottom": 7374, 2024-03-28T22:21:01.110 INFO:teuthology.orchestra.run.smithi053.stdout: "inodes_pin_tail": 194, 2024-03-28T22:21:01.110 INFO:teuthology.orchestra.run.smithi053.stdout: "inodes_pinned": 7227, 2024-03-28T22:21:01.110 INFO:teuthology.orchestra.run.smithi053.stdout: "inodes_expired": 24, 2024-03-28T22:21:01.110 INFO:teuthology.orchestra.run.smithi053.stdout: "inodes_with_caps": 3485, 2024-03-28T22:21:01.111 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": 3485, 2024-03-28T22:21:01.111 INFO:teuthology.orchestra.run.smithi053.stdout: "subtrees": 9, 2024-03-28T22:21:01.111 INFO:teuthology.orchestra.run.smithi053.stdout: "traverse": 10705, 2024-03-28T22:21:01.111 INFO:teuthology.orchestra.run.smithi053.stdout: "traverse_hit": 10206, 2024-03-28T22:21:01.111 INFO:teuthology.orchestra.run.smithi053.stdout: "traverse_forward": 61, 2024-03-28T22:21:01.111 INFO:teuthology.orchestra.run.smithi053.stdout: "traverse_discover": 26, 2024-03-28T22:21:01.111 INFO:teuthology.orchestra.run.smithi053.stdout: "traverse_dir_fetch": 0, 2024-03-28T22:21:01.111 INFO:teuthology.orchestra.run.smithi053.stdout: "traverse_remote_ino": 0, 2024-03-28T22:21:01.112 INFO:teuthology.orchestra.run.smithi053.stdout: "traverse_lock": 0, 2024-03-28T22:21:01.112 INFO:teuthology.orchestra.run.smithi053.stdout: "load_cent": 0, 2024-03-28T22:21:01.112 INFO:teuthology.orchestra.run.smithi053.stdout: "q": 0, 2024-03-28T22:21:01.112 INFO:teuthology.orchestra.run.smithi053.stdout: "exported": 3, 2024-03-28T22:21:01.112 INFO:teuthology.orchestra.run.smithi053.stdout: "exported_inodes": 22, 2024-03-28T22:21:01.112 INFO:teuthology.orchestra.run.smithi053.stdout: "imported": 26, 2024-03-28T22:21:01.112 INFO:teuthology.orchestra.run.smithi053.stdout: "imported_inodes": 866, 2024-03-28T22:21:01.112 INFO:teuthology.orchestra.run.smithi053.stdout: "openino_dir_fetch": 0, 2024-03-28T22:21:01.113 INFO:teuthology.orchestra.run.smithi053.stdout: "openino_backtrace_fetch": 55230, 2024-03-28T22:21:01.113 INFO:teuthology.orchestra.run.smithi053.stdout: "openino_peer_discover": 0, 2024-03-28T22:21:01.113 INFO:teuthology.orchestra.run.smithi053.stdout: "root_rfiles": 3452, 2024-03-28T22:21:01.113 INFO:teuthology.orchestra.run.smithi053.stdout: "root_rbytes": 26525988, 2024-03-28T22:21:01.113 INFO:teuthology.orchestra.run.smithi053.stdout: "root_rsnaps": 6, 2024-03-28T22:21:01.113 INFO:teuthology.orchestra.run.smithi053.stdout: "scrub_backtrace_fetch": 55230, 2024-03-28T22:21:01.113 INFO:teuthology.orchestra.run.smithi053.stdout: "scrub_set_tag": 0, 2024-03-28T22:21:01.113 INFO:teuthology.orchestra.run.smithi053.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T22:21:01.113 INFO:teuthology.orchestra.run.smithi053.stdout: "scrub_inotable_repaired": 0, 2024-03-28T22:21:01.114 INFO:teuthology.orchestra.run.smithi053.stdout: "scrub_dir_inodes": 1063, 2024-03-28T22:21:01.114 INFO:teuthology.orchestra.run.smithi053.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T22:21:01.114 INFO:teuthology.orchestra.run.smithi053.stdout: "scrub_dirfrag_rstats": 1063, 2024-03-28T22:21:01.114 INFO:teuthology.orchestra.run.smithi053.stdout: "scrub_file_inodes": 54167, 2024-03-28T22:21:01.114 INFO:teuthology.orchestra.run.smithi053.stdout: "handle_inode_file_caps": 3, 2024-03-28T22:21:01.114 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_cap_op_revoke": 555, 2024-03-28T22:21:01.114 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_cap_op_grant": 11193, 2024-03-28T22:21:01.114 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_cap_op_trunc": 8, 2024-03-28T22:21:01.115 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T22:21:01.115 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T22:21:01.115 INFO:teuthology.orchestra.run.smithi053.stdout: "handle_client_caps": 14922, 2024-03-28T22:21:01.115 INFO:teuthology.orchestra.run.smithi053.stdout: "handle_client_caps_dirty": 10043, 2024-03-28T22:21:01.115 INFO:teuthology.orchestra.run.smithi053.stdout: "handle_client_cap_release": 1747, 2024-03-28T22:21:01.115 INFO:teuthology.orchestra.run.smithi053.stdout: "process_request_cap_release": 3442 2024-03-28T22:21:01.115 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.115 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_cache": { 2024-03-28T22:21:01.116 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_update": 34, 2024-03-28T22:21:01.116 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_update_receipt": 12, 2024-03-28T22:21:01.116 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_try_discover": 8, 2024-03-28T22:21:01.116 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_send_discover": 31, 2024-03-28T22:21:01.116 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_handle_discover": 66, 2024-03-28T22:21:01.116 INFO:teuthology.orchestra.run.smithi053.stdout: "num_strays": 3, 2024-03-28T22:21:01.116 INFO:teuthology.orchestra.run.smithi053.stdout: "num_strays_delayed": 0, 2024-03-28T22:21:01.116 INFO:teuthology.orchestra.run.smithi053.stdout: "num_strays_enqueuing": 0, 2024-03-28T22:21:01.116 INFO:teuthology.orchestra.run.smithi053.stdout: "strays_created": 3210, 2024-03-28T22:21:01.117 INFO:teuthology.orchestra.run.smithi053.stdout: "strays_enqueued": 3207, 2024-03-28T22:21:01.117 INFO:teuthology.orchestra.run.smithi053.stdout: "strays_reintegrated": 0, 2024-03-28T22:21:01.117 INFO:teuthology.orchestra.run.smithi053.stdout: "strays_migrated": 0, 2024-03-28T22:21:01.117 INFO:teuthology.orchestra.run.smithi053.stdout: "num_recovering_processing": 0, 2024-03-28T22:21:01.117 INFO:teuthology.orchestra.run.smithi053.stdout: "num_recovering_enqueued": 0, 2024-03-28T22:21:01.117 INFO:teuthology.orchestra.run.smithi053.stdout: "num_recovering_prioritized": 0, 2024-03-28T22:21:01.117 INFO:teuthology.orchestra.run.smithi053.stdout: "recovery_started": 0, 2024-03-28T22:21:01.117 INFO:teuthology.orchestra.run.smithi053.stdout: "recovery_completed": 0, 2024-03-28T22:21:01.117 INFO:teuthology.orchestra.run.smithi053.stdout: "ireq_quiesce_path": 0, 2024-03-28T22:21:01.118 INFO:teuthology.orchestra.run.smithi053.stdout: "ireq_quiesce_inode": 0, 2024-03-28T22:21:01.118 INFO:teuthology.orchestra.run.smithi053.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T22:21:01.118 INFO:teuthology.orchestra.run.smithi053.stdout: "ireq_exportdir": 3, 2024-03-28T22:21:01.118 INFO:teuthology.orchestra.run.smithi053.stdout: "ireq_flush": 0, 2024-03-28T22:21:01.118 INFO:teuthology.orchestra.run.smithi053.stdout: "ireq_fragmentdir": 29, 2024-03-28T22:21:01.118 INFO:teuthology.orchestra.run.smithi053.stdout: "ireq_fragstats": 0, 2024-03-28T22:21:01.118 INFO:teuthology.orchestra.run.smithi053.stdout: "ireq_inodestats": 0 2024-03-28T22:21:01.118 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.119 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_log": { 2024-03-28T22:21:01.119 INFO:teuthology.orchestra.run.smithi053.stdout: "evlrg": 0, 2024-03-28T22:21:01.119 INFO:teuthology.orchestra.run.smithi053.stdout: "evadd": 25740, 2024-03-28T22:21:01.119 INFO:teuthology.orchestra.run.smithi053.stdout: "evex": 0, 2024-03-28T22:21:01.119 INFO:teuthology.orchestra.run.smithi053.stdout: "evtrm": 0, 2024-03-28T22:21:01.119 INFO:teuthology.orchestra.run.smithi053.stdout: "ev": 25740, 2024-03-28T22:21:01.119 INFO:teuthology.orchestra.run.smithi053.stdout: "evexg": 0, 2024-03-28T22:21:01.119 INFO:teuthology.orchestra.run.smithi053.stdout: "evexd": 0, 2024-03-28T22:21:01.120 INFO:teuthology.orchestra.run.smithi053.stdout: "segadd": 35, 2024-03-28T22:21:01.120 INFO:teuthology.orchestra.run.smithi053.stdout: "segex": 0, 2024-03-28T22:21:01.120 INFO:teuthology.orchestra.run.smithi053.stdout: "segtrm": 0, 2024-03-28T22:21:01.120 INFO:teuthology.orchestra.run.smithi053.stdout: "seg": 35, 2024-03-28T22:21:01.120 INFO:teuthology.orchestra.run.smithi053.stdout: "segmjr": 4, 2024-03-28T22:21:01.120 INFO:teuthology.orchestra.run.smithi053.stdout: "segexg": 0, 2024-03-28T22:21:01.120 INFO:teuthology.orchestra.run.smithi053.stdout: "segexd": 0, 2024-03-28T22:21:01.120 INFO:teuthology.orchestra.run.smithi053.stdout: "expos": 4194304, 2024-03-28T22:21:01.120 INFO:teuthology.orchestra.run.smithi053.stdout: "wrpos": 69476710, 2024-03-28T22:21:01.121 INFO:teuthology.orchestra.run.smithi053.stdout: "rdpos": 0, 2024-03-28T22:21:01.121 INFO:teuthology.orchestra.run.smithi053.stdout: "jlat": { 2024-03-28T22:21:01.121 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 368, 2024-03-28T22:21:01.121 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 29.626159644, 2024-03-28T22:21:01.121 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.080505868 2024-03-28T22:21:01.121 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.121 INFO:teuthology.orchestra.run.smithi053.stdout: "replayed": 0 2024-03-28T22:21:01.121 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.121 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_mem": { 2024-03-28T22:21:01.122 INFO:teuthology.orchestra.run.smithi053.stdout: "ino": 6423, 2024-03-28T22:21:01.122 INFO:teuthology.orchestra.run.smithi053.stdout: "ino+": 9651, 2024-03-28T22:21:01.122 INFO:teuthology.orchestra.run.smithi053.stdout: "ino-": 3228, 2024-03-28T22:21:01.122 INFO:teuthology.orchestra.run.smithi053.stdout: "dir": 305, 2024-03-28T22:21:01.122 INFO:teuthology.orchestra.run.smithi053.stdout: "dir+": 343, 2024-03-28T22:21:01.122 INFO:teuthology.orchestra.run.smithi053.stdout: "dir-": 38, 2024-03-28T22:21:01.122 INFO:teuthology.orchestra.run.smithi053.stdout: "dn": 8362, 2024-03-28T22:21:01.122 INFO:teuthology.orchestra.run.smithi053.stdout: "dn+": 11302, 2024-03-28T22:21:01.123 INFO:teuthology.orchestra.run.smithi053.stdout: "dn-": 2940, 2024-03-28T22:21:01.123 INFO:teuthology.orchestra.run.smithi053.stdout: "cap": 3485, 2024-03-28T22:21:01.123 INFO:teuthology.orchestra.run.smithi053.stdout: "cap+": 6728, 2024-03-28T22:21:01.123 INFO:teuthology.orchestra.run.smithi053.stdout: "cap-": 3243, 2024-03-28T22:21:01.123 INFO:teuthology.orchestra.run.smithi053.stdout: "rss": 114836, 2024-03-28T22:21:01.123 INFO:teuthology.orchestra.run.smithi053.stdout: "heap": 223612 2024-03-28T22:21:01.123 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.123 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_server": { 2024-03-28T22:21:01.124 INFO:teuthology.orchestra.run.smithi053.stdout: "dispatch_client_request": 10406, 2024-03-28T22:21:01.124 INFO:teuthology.orchestra.run.smithi053.stdout: "dispatch_server_request": 11, 2024-03-28T22:21:01.124 INFO:teuthology.orchestra.run.smithi053.stdout: "handle_client_request": 10126, 2024-03-28T22:21:01.124 INFO:teuthology.orchestra.run.smithi053.stdout: "handle_client_session": 30, 2024-03-28T22:21:01.124 INFO:teuthology.orchestra.run.smithi053.stdout: "handle_peer_request": 23, 2024-03-28T22:21:01.124 INFO:teuthology.orchestra.run.smithi053.stdout: "req_create_latency": { 2024-03-28T22:21:01.124 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 6105, 2024-03-28T22:21:01.124 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 22.758938623, 2024-03-28T22:21:01.124 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.003727917 2024-03-28T22:21:01.125 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.125 INFO:teuthology.orchestra.run.smithi053.stdout: "req_getattr_latency": { 2024-03-28T22:21:01.125 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 2, 2024-03-28T22:21:01.125 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.007256426, 2024-03-28T22:21:01.125 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.003628213 2024-03-28T22:21:01.125 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.125 INFO:teuthology.orchestra.run.smithi053.stdout: "req_getfilelock_latency": { 2024-03-28T22:21:01.125 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:01.126 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:01.126 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:01.126 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.126 INFO:teuthology.orchestra.run.smithi053.stdout: "req_link_latency": { 2024-03-28T22:21:01.126 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:01.126 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:01.126 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:01.126 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.126 INFO:teuthology.orchestra.run.smithi053.stdout: "req_lookup_latency": { 2024-03-28T22:21:01.127 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 397, 2024-03-28T22:21:01.127 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.114440661, 2024-03-28T22:21:01.127 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000288263 2024-03-28T22:21:01.127 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.127 INFO:teuthology.orchestra.run.smithi053.stdout: "req_lookuphash_latency": { 2024-03-28T22:21:01.127 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:01.127 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:01.127 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:01.128 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.128 INFO:teuthology.orchestra.run.smithi053.stdout: "req_lookupino_latency": { 2024-03-28T22:21:01.128 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 2, 2024-03-28T22:21:01.128 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000280939, 2024-03-28T22:21:01.128 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000140469 2024-03-28T22:21:01.128 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.128 INFO:teuthology.orchestra.run.smithi053.stdout: "req_lookupname_latency": { 2024-03-28T22:21:01.128 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:01.128 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:01.129 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:01.129 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.129 INFO:teuthology.orchestra.run.smithi053.stdout: "req_lookupparent_latency": { 2024-03-28T22:21:01.129 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:01.129 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:01.129 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:01.129 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.129 INFO:teuthology.orchestra.run.smithi053.stdout: "req_lookupsnap_latency": { 2024-03-28T22:21:01.130 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:01.130 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:01.130 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:01.130 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.131 INFO:teuthology.orchestra.run.smithi053.stdout: "req_lssnap_latency": { 2024-03-28T22:21:01.131 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:01.131 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:01.131 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:01.131 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.131 INFO:teuthology.orchestra.run.smithi053.stdout: "req_mkdir_latency": { 2024-03-28T22:21:01.131 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 40, 2024-03-28T22:21:01.131 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.045423268, 2024-03-28T22:21:01.131 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.001135581 2024-03-28T22:21:01.132 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.132 INFO:teuthology.orchestra.run.smithi053.stdout: "req_mknod_latency": { 2024-03-28T22:21:01.132 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:01.132 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:01.132 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:01.132 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.132 INFO:teuthology.orchestra.run.smithi053.stdout: "req_mksnap_latency": { 2024-03-28T22:21:01.132 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:01.133 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:01.133 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:01.133 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.133 INFO:teuthology.orchestra.run.smithi053.stdout: "req_open_latency": { 2024-03-28T22:21:01.133 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 152, 2024-03-28T22:21:01.133 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.052386090, 2024-03-28T22:21:01.133 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000344645 2024-03-28T22:21:01.133 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.133 INFO:teuthology.orchestra.run.smithi053.stdout: "req_readdir_latency": { 2024-03-28T22:21:01.134 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 101, 2024-03-28T22:21:01.134 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.794930257, 2024-03-28T22:21:01.134 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.007870596 2024-03-28T22:21:01.134 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.134 INFO:teuthology.orchestra.run.smithi053.stdout: "req_rename_latency": { 2024-03-28T22:21:01.134 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 62, 2024-03-28T22:21:01.134 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.166627328, 2024-03-28T22:21:01.134 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.002687537 2024-03-28T22:21:01.135 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.135 INFO:teuthology.orchestra.run.smithi053.stdout: "req_renamesnap_latency": { 2024-03-28T22:21:01.135 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:01.135 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:01.135 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:01.135 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.135 INFO:teuthology.orchestra.run.smithi053.stdout: "req_snapdiff_latency": { 2024-03-28T22:21:01.135 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:01.135 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:01.136 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:01.136 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.136 INFO:teuthology.orchestra.run.smithi053.stdout: "req_rmdir_latency": { 2024-03-28T22:21:01.136 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 22, 2024-03-28T22:21:01.136 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.980878661, 2024-03-28T22:21:01.136 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.044585393 2024-03-28T22:21:01.136 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.136 INFO:teuthology.orchestra.run.smithi053.stdout: "req_rmsnap_latency": { 2024-03-28T22:21:01.137 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:01.137 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:01.137 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:01.137 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.137 INFO:teuthology.orchestra.run.smithi053.stdout: "req_rmxattr_latency": { 2024-03-28T22:21:01.137 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:01.137 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:01.137 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:01.137 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.138 INFO:teuthology.orchestra.run.smithi053.stdout: "req_setattr_latency": { 2024-03-28T22:21:01.138 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 8, 2024-03-28T22:21:01.138 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.069640524, 2024-03-28T22:21:01.138 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.008705065 2024-03-28T22:21:01.138 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.138 INFO:teuthology.orchestra.run.smithi053.stdout: "req_setdirlayout_latency": { 2024-03-28T22:21:01.138 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:01.138 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:01.139 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:01.139 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.139 INFO:teuthology.orchestra.run.smithi053.stdout: "req_setfilelock_latency": { 2024-03-28T22:21:01.139 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 8, 2024-03-28T22:21:01.139 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.003130847, 2024-03-28T22:21:01.139 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000391355 2024-03-28T22:21:01.139 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.139 INFO:teuthology.orchestra.run.smithi053.stdout: "req_setlayout_latency": { 2024-03-28T22:21:01.139 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:01.140 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:01.140 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:01.140 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.140 INFO:teuthology.orchestra.run.smithi053.stdout: "req_setxattr_latency": { 2024-03-28T22:21:01.140 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:01.140 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:01.140 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:01.140 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.141 INFO:teuthology.orchestra.run.smithi053.stdout: "req_symlink_latency": { 2024-03-28T22:21:01.141 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 6, 2024-03-28T22:21:01.141 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.009469389, 2024-03-28T22:21:01.141 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.001578231 2024-03-28T22:21:01.141 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.141 INFO:teuthology.orchestra.run.smithi053.stdout: "req_unlink_latency": { 2024-03-28T22:21:01.141 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 3160, 2024-03-28T22:21:01.141 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 13.305182709, 2024-03-28T22:21:01.141 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.004210500 2024-03-28T22:21:01.142 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.142 INFO:teuthology.orchestra.run.smithi053.stdout: "cap_revoke_eviction": 0, 2024-03-28T22:21:01.142 INFO:teuthology.orchestra.run.smithi053.stdout: "cap_acquisition_throttle": 0, 2024-03-28T22:21:01.142 INFO:teuthology.orchestra.run.smithi053.stdout: "req_getvxattr_latency": { 2024-03-28T22:21:01.142 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:01.142 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:01.142 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:01.142 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-03-28T22:21:01.143 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.143 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_sessions": { 2024-03-28T22:21:01.143 INFO:teuthology.orchestra.run.smithi053.stdout: "session_count": 2, 2024-03-28T22:21:01.143 INFO:teuthology.orchestra.run.smithi053.stdout: "session_add": 5, 2024-03-28T22:21:01.143 INFO:teuthology.orchestra.run.smithi053.stdout: "session_remove": 3, 2024-03-28T22:21:01.143 INFO:teuthology.orchestra.run.smithi053.stdout: "sessions_open": 2, 2024-03-28T22:21:01.143 INFO:teuthology.orchestra.run.smithi053.stdout: "sessions_stale": 0, 2024-03-28T22:21:01.143 INFO:teuthology.orchestra.run.smithi053.stdout: "total_load": 6626, 2024-03-28T22:21:01.144 INFO:teuthology.orchestra.run.smithi053.stdout: "average_load": 3313, 2024-03-28T22:21:01.144 INFO:teuthology.orchestra.run.smithi053.stdout: "avg_session_uptime": 611, 2024-03-28T22:21:01.144 INFO:teuthology.orchestra.run.smithi053.stdout: "mdthresh_evicted": 0 2024-03-28T22:21:01.144 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.144 INFO:teuthology.orchestra.run.smithi053.stdout: "mempool": { 2024-03-28T22:21:01.144 INFO:teuthology.orchestra.run.smithi053.stdout: "bloom_filter_bytes": 0, 2024-03-28T22:21:01.144 INFO:teuthology.orchestra.run.smithi053.stdout: "bloom_filter_items": 0, 2024-03-28T22:21:01.144 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T22:21:01.144 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_alloc_items": 0, 2024-03-28T22:21:01.145 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T22:21:01.145 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_data_items": 0, 2024-03-28T22:21:01.145 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T22:21:01.145 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T22:21:01.145 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T22:21:01.145 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T22:21:01.145 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T22:21:01.145 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_other_items": 0, 2024-03-28T22:21:01.145 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T22:21:01.146 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T22:21:01.146 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_extent_bytes": 0, 2024-03-28T22:21:01.146 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_extent_items": 0, 2024-03-28T22:21:01.146 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_blob_bytes": 0, 2024-03-28T22:21:01.146 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_blob_items": 0, 2024-03-28T22:21:01.146 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T22:21:01.146 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T22:21:01.146 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T22:21:01.147 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T22:21:01.147 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T22:21:01.147 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_fsck_items": 0, 2024-03-28T22:21:01.147 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_txc_bytes": 0, 2024-03-28T22:21:01.147 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_txc_items": 0, 2024-03-28T22:21:01.147 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T22:21:01.147 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T22:21:01.147 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_writing_bytes": 0, 2024-03-28T22:21:01.148 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_writing_items": 0, 2024-03-28T22:21:01.148 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_bytes": 0, 2024-03-28T22:21:01.148 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_items": 0, 2024-03-28T22:21:01.148 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T22:21:01.148 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_file_reader_items": 0, 2024-03-28T22:21:01.148 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T22:21:01.148 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_file_writer_items": 0, 2024-03-28T22:21:01.148 INFO:teuthology.orchestra.run.smithi053.stdout: "buffer_anon_bytes": 518461, 2024-03-28T22:21:01.148 INFO:teuthology.orchestra.run.smithi053.stdout: "buffer_anon_items": 5407, 2024-03-28T22:21:01.149 INFO:teuthology.orchestra.run.smithi053.stdout: "buffer_meta_bytes": 968, 2024-03-28T22:21:01.149 INFO:teuthology.orchestra.run.smithi053.stdout: "buffer_meta_items": 11, 2024-03-28T22:21:01.149 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_bytes": 0, 2024-03-28T22:21:01.149 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_items": 0, 2024-03-28T22:21:01.149 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_mapbl_bytes": 0, 2024-03-28T22:21:01.149 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_mapbl_items": 0, 2024-03-28T22:21:01.149 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_pglog_bytes": 0, 2024-03-28T22:21:01.149 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_pglog_items": 0, 2024-03-28T22:21:01.150 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_bytes": 8160, 2024-03-28T22:21:01.150 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_items": 186, 2024-03-28T22:21:01.150 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T22:21:01.150 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_mapping_items": 0, 2024-03-28T22:21:01.150 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap_bytes": 0, 2024-03-28T22:21:01.150 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap_items": 0, 2024-03-28T22:21:01.150 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_co_bytes": 22462420, 2024-03-28T22:21:01.150 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_co_items": 231059, 2024-03-28T22:21:01.150 INFO:teuthology.orchestra.run.smithi053.stdout: "unittest_1_bytes": 0, 2024-03-28T22:21:01.151 INFO:teuthology.orchestra.run.smithi053.stdout: "unittest_1_items": 0, 2024-03-28T22:21:01.151 INFO:teuthology.orchestra.run.smithi053.stdout: "unittest_2_bytes": 0, 2024-03-28T22:21:01.151 INFO:teuthology.orchestra.run.smithi053.stdout: "unittest_2_items": 0 2024-03-28T22:21:01.151 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.151 INFO:teuthology.orchestra.run.smithi053.stdout: "objecter": { 2024-03-28T22:21:01.151 INFO:teuthology.orchestra.run.smithi053.stdout: "op_active": 0, 2024-03-28T22:21:01.151 INFO:teuthology.orchestra.run.smithi053.stdout: "op_laggy": 0, 2024-03-28T22:21:01.151 INFO:teuthology.orchestra.run.smithi053.stdout: "op_send": 59291, 2024-03-28T22:21:01.152 INFO:teuthology.orchestra.run.smithi053.stdout: "op_send_bytes": 69769557, 2024-03-28T22:21:01.152 INFO:teuthology.orchestra.run.smithi053.stdout: "op_resend": 33, 2024-03-28T22:21:01.152 INFO:teuthology.orchestra.run.smithi053.stdout: "op_reply": 59291, 2024-03-28T22:21:01.152 INFO:teuthology.orchestra.run.smithi053.stdout: "op_latency": { 2024-03-28T22:21:01.152 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 59291, 2024-03-28T22:21:01.152 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 36.623270178, 2024-03-28T22:21:01.152 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000617686 2024-03-28T22:21:01.152 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.152 INFO:teuthology.orchestra.run.smithi053.stdout: "op_inflight": 0, 2024-03-28T22:21:01.153 INFO:teuthology.orchestra.run.smithi053.stdout: "oplen_avg": { 2024-03-28T22:21:01.153 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 59291, 2024-03-28T22:21:01.153 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 59623 2024-03-28T22:21:01.153 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.153 INFO:teuthology.orchestra.run.smithi053.stdout: "op": 59291, 2024-03-28T22:21:01.153 INFO:teuthology.orchestra.run.smithi053.stdout: "op_r": 55268, 2024-03-28T22:21:01.153 INFO:teuthology.orchestra.run.smithi053.stdout: "op_w": 4023, 2024-03-28T22:21:01.153 INFO:teuthology.orchestra.run.smithi053.stdout: "op_rmw": 0, 2024-03-28T22:21:01.153 INFO:teuthology.orchestra.run.smithi053.stdout: "op_pg": 0, 2024-03-28T22:21:01.154 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_stat": 0, 2024-03-28T22:21:01.154 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_create": 10, 2024-03-28T22:21:01.154 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_read": 38, 2024-03-28T22:21:01.154 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_write": 406, 2024-03-28T22:21:01.154 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_writefull": 34, 2024-03-28T22:21:01.154 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_writesame": 0, 2024-03-28T22:21:01.154 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_append": 0, 2024-03-28T22:21:01.154 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_zero": 0, 2024-03-28T22:21:01.155 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_truncate": 5, 2024-03-28T22:21:01.155 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_delete": 3258, 2024-03-28T22:21:01.155 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_mapext": 0, 2024-03-28T22:21:01.155 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_sparse_read": 0, 2024-03-28T22:21:01.155 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_clonerange": 0, 2024-03-28T22:21:01.155 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_getxattr": 55230, 2024-03-28T22:21:01.155 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_setxattr": 20, 2024-03-28T22:21:01.155 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_cmpxattr": 0, 2024-03-28T22:21:01.156 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_rmxattr": 0, 2024-03-28T22:21:01.156 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_resetxattrs": 0, 2024-03-28T22:21:01.156 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_call": 0, 2024-03-28T22:21:01.156 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_watch": 0, 2024-03-28T22:21:01.156 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_notify": 0, 2024-03-28T22:21:01.156 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T22:21:01.156 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_pgls": 0, 2024-03-28T22:21:01.156 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_pgls_filter": 0, 2024-03-28T22:21:01.156 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_other": 8, 2024-03-28T22:21:01.157 INFO:teuthology.orchestra.run.smithi053.stdout: "linger_active": 0, 2024-03-28T22:21:01.157 INFO:teuthology.orchestra.run.smithi053.stdout: "linger_send": 0, 2024-03-28T22:21:01.157 INFO:teuthology.orchestra.run.smithi053.stdout: "linger_resend": 0, 2024-03-28T22:21:01.157 INFO:teuthology.orchestra.run.smithi053.stdout: "linger_ping": 0, 2024-03-28T22:21:01.157 INFO:teuthology.orchestra.run.smithi053.stdout: "poolop_active": 0, 2024-03-28T22:21:01.157 INFO:teuthology.orchestra.run.smithi053.stdout: "poolop_send": 0, 2024-03-28T22:21:01.157 INFO:teuthology.orchestra.run.smithi053.stdout: "poolop_resend": 0, 2024-03-28T22:21:01.157 INFO:teuthology.orchestra.run.smithi053.stdout: "poolstat_active": 0, 2024-03-28T22:21:01.158 INFO:teuthology.orchestra.run.smithi053.stdout: "poolstat_send": 0, 2024-03-28T22:21:01.158 INFO:teuthology.orchestra.run.smithi053.stdout: "poolstat_resend": 0, 2024-03-28T22:21:01.158 INFO:teuthology.orchestra.run.smithi053.stdout: "statfs_active": 0, 2024-03-28T22:21:01.158 INFO:teuthology.orchestra.run.smithi053.stdout: "statfs_send": 0, 2024-03-28T22:21:01.158 INFO:teuthology.orchestra.run.smithi053.stdout: "statfs_resend": 0, 2024-03-28T22:21:01.158 INFO:teuthology.orchestra.run.smithi053.stdout: "command_active": 0, 2024-03-28T22:21:01.158 INFO:teuthology.orchestra.run.smithi053.stdout: "command_send": 0, 2024-03-28T22:21:01.158 INFO:teuthology.orchestra.run.smithi053.stdout: "command_resend": 0, 2024-03-28T22:21:01.159 INFO:teuthology.orchestra.run.smithi053.stdout: "map_epoch": 97, 2024-03-28T22:21:01.159 INFO:teuthology.orchestra.run.smithi053.stdout: "map_full": 0, 2024-03-28T22:21:01.159 INFO:teuthology.orchestra.run.smithi053.stdout: "map_inc": 11, 2024-03-28T22:21:01.159 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_sessions": 12, 2024-03-28T22:21:01.159 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_session_open": 12, 2024-03-28T22:21:01.159 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_session_close": 0, 2024-03-28T22:21:01.159 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_laggy": 0, 2024-03-28T22:21:01.159 INFO:teuthology.orchestra.run.smithi053.stdout: "omap_wr": 585, 2024-03-28T22:21:01.159 INFO:teuthology.orchestra.run.smithi053.stdout: "omap_rd": 0, 2024-03-28T22:21:01.160 INFO:teuthology.orchestra.run.smithi053.stdout: "omap_del": 29 2024-03-28T22:21:01.160 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.160 INFO:teuthology.orchestra.run.smithi053.stdout: "oft": { 2024-03-28T22:21:01.160 INFO:teuthology.orchestra.run.smithi053.stdout: "omap_total_objs": 1, 2024-03-28T22:21:01.160 INFO:teuthology.orchestra.run.smithi053.stdout: "omap_total_kv_pairs": 0, 2024-03-28T22:21:01.160 INFO:teuthology.orchestra.run.smithi053.stdout: "omap_total_updates": 0, 2024-03-28T22:21:01.160 INFO:teuthology.orchestra.run.smithi053.stdout: "omap_total_removes": 0 2024-03-28T22:21:01.160 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.160 INFO:teuthology.orchestra.run.smithi053.stdout: "osd-slow-ops": { 2024-03-28T22:21:01.161 INFO:teuthology.orchestra.run.smithi053.stdout: "slow_ops_count": 4 2024-03-28T22:21:01.161 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.161 INFO:teuthology.orchestra.run.smithi053.stdout: "purge_queue": { 2024-03-28T22:21:01.161 INFO:teuthology.orchestra.run.smithi053.stdout: "pq_executing_ops": 0, 2024-03-28T22:21:01.161 INFO:teuthology.orchestra.run.smithi053.stdout: "pq_executing_ops_high_water": 10, 2024-03-28T22:21:01.161 INFO:teuthology.orchestra.run.smithi053.stdout: "pq_executing": 0, 2024-03-28T22:21:01.161 INFO:teuthology.orchestra.run.smithi053.stdout: "pq_executing_high_water": 10, 2024-03-28T22:21:01.161 INFO:teuthology.orchestra.run.smithi053.stdout: "pq_executed_ops": 3207, 2024-03-28T22:21:01.162 INFO:teuthology.orchestra.run.smithi053.stdout: "pq_executed": 3207, 2024-03-28T22:21:01.162 INFO:teuthology.orchestra.run.smithi053.stdout: "pq_item_in_journal": 0 2024-03-28T22:21:01.162 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.162 INFO:teuthology.orchestra.run.smithi053.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T22:21:01.162 INFO:teuthology.orchestra.run.smithi053.stdout: "val": 0, 2024-03-28T22:21:01.162 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 104857600, 2024-03-28T22:21:01.162 INFO:teuthology.orchestra.run.smithi053.stdout: "get_started": 0, 2024-03-28T22:21:01.162 INFO:teuthology.orchestra.run.smithi053.stdout: "get": 86660, 2024-03-28T22:21:01.163 INFO:teuthology.orchestra.run.smithi053.stdout: "get_sum": 18219513, 2024-03-28T22:21:01.163 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_fail": 0, 2024-03-28T22:21:01.163 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_success": 86660, 2024-03-28T22:21:01.163 INFO:teuthology.orchestra.run.smithi053.stdout: "take": 0, 2024-03-28T22:21:01.163 INFO:teuthology.orchestra.run.smithi053.stdout: "take_sum": 0, 2024-03-28T22:21:01.163 INFO:teuthology.orchestra.run.smithi053.stdout: "put": 86660, 2024-03-28T22:21:01.163 INFO:teuthology.orchestra.run.smithi053.stdout: "put_sum": 18219513, 2024-03-28T22:21:01.163 INFO:teuthology.orchestra.run.smithi053.stdout: "wait": { 2024-03-28T22:21:01.163 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:01.164 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:01.164 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:01.164 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-03-28T22:21:01.164 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.164 INFO:teuthology.orchestra.run.smithi053.stdout: "throttle-objecter_bytes": { 2024-03-28T22:21:01.164 INFO:teuthology.orchestra.run.smithi053.stdout: "val": 0, 2024-03-28T22:21:01.164 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 104857600, 2024-03-28T22:21:01.164 INFO:teuthology.orchestra.run.smithi053.stdout: "get_started": 0, 2024-03-28T22:21:01.164 INFO:teuthology.orchestra.run.smithi053.stdout: "get": 0, 2024-03-28T22:21:01.165 INFO:teuthology.orchestra.run.smithi053.stdout: "get_sum": 0, 2024-03-28T22:21:01.165 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_fail": 0, 2024-03-28T22:21:01.165 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_success": 0, 2024-03-28T22:21:01.165 INFO:teuthology.orchestra.run.smithi053.stdout: "take": 59291, 2024-03-28T22:21:01.165 INFO:teuthology.orchestra.run.smithi053.stdout: "take_sum": 70107584, 2024-03-28T22:21:01.165 INFO:teuthology.orchestra.run.smithi053.stdout: "put": 59291, 2024-03-28T22:21:01.165 INFO:teuthology.orchestra.run.smithi053.stdout: "put_sum": 70107584, 2024-03-28T22:21:01.165 INFO:teuthology.orchestra.run.smithi053.stdout: "wait": { 2024-03-28T22:21:01.166 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:01.166 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:01.166 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:01.166 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-03-28T22:21:01.166 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.166 INFO:teuthology.orchestra.run.smithi053.stdout: "throttle-objecter_ops": { 2024-03-28T22:21:01.166 INFO:teuthology.orchestra.run.smithi053.stdout: "val": 0, 2024-03-28T22:21:01.166 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 1024, 2024-03-28T22:21:01.166 INFO:teuthology.orchestra.run.smithi053.stdout: "get_started": 0, 2024-03-28T22:21:01.167 INFO:teuthology.orchestra.run.smithi053.stdout: "get": 0, 2024-03-28T22:21:01.167 INFO:teuthology.orchestra.run.smithi053.stdout: "get_sum": 0, 2024-03-28T22:21:01.167 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_fail": 0, 2024-03-28T22:21:01.167 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_success": 0, 2024-03-28T22:21:01.167 INFO:teuthology.orchestra.run.smithi053.stdout: "take": 59291, 2024-03-28T22:21:01.167 INFO:teuthology.orchestra.run.smithi053.stdout: "take_sum": 59291, 2024-03-28T22:21:01.167 INFO:teuthology.orchestra.run.smithi053.stdout: "put": 59291, 2024-03-28T22:21:01.167 INFO:teuthology.orchestra.run.smithi053.stdout: "put_sum": 59291, 2024-03-28T22:21:01.168 INFO:teuthology.orchestra.run.smithi053.stdout: "wait": { 2024-03-28T22:21:01.168 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:01.168 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:01.168 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:01.168 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-03-28T22:21:01.168 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.168 INFO:teuthology.orchestra.run.smithi053.stdout: "throttle-write_buf_throttle": { 2024-03-28T22:21:01.168 INFO:teuthology.orchestra.run.smithi053.stdout: "val": 0, 2024-03-28T22:21:01.168 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 3758096384, 2024-03-28T22:21:01.169 INFO:teuthology.orchestra.run.smithi053.stdout: "get_started": 0, 2024-03-28T22:21:01.169 INFO:teuthology.orchestra.run.smithi053.stdout: "get": 3207, 2024-03-28T22:21:01.169 INFO:teuthology.orchestra.run.smithi053.stdout: "get_sum": 338027, 2024-03-28T22:21:01.169 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_fail": 0, 2024-03-28T22:21:01.169 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_success": 3207, 2024-03-28T22:21:01.169 INFO:teuthology.orchestra.run.smithi053.stdout: "take": 0, 2024-03-28T22:21:01.169 INFO:teuthology.orchestra.run.smithi053.stdout: "take_sum": 0, 2024-03-28T22:21:01.169 INFO:teuthology.orchestra.run.smithi053.stdout: "put": 38, 2024-03-28T22:21:01.169 INFO:teuthology.orchestra.run.smithi053.stdout: "put_sum": 338027, 2024-03-28T22:21:01.170 INFO:teuthology.orchestra.run.smithi053.stdout: "wait": { 2024-03-28T22:21:01.170 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:01.170 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:01.170 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:01.170 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-03-28T22:21:01.170 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:01.170 INFO:teuthology.orchestra.run.smithi053.stdout: "throttle-write_buf_throttle-0x55b476bf2460": { 2024-03-28T22:21:01.170 INFO:teuthology.orchestra.run.smithi053.stdout: "val": 0, 2024-03-28T22:21:01.171 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 3758096384, 2024-03-28T22:21:01.171 INFO:teuthology.orchestra.run.smithi053.stdout: "get_started": 0, 2024-03-28T22:21:01.171 INFO:teuthology.orchestra.run.smithi053.stdout: "get": 25740, 2024-03-28T22:21:01.171 INFO:teuthology.orchestra.run.smithi053.stdout: "get_sum": 65282406, 2024-03-28T22:21:01.171 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_fail": 0, 2024-03-28T22:21:01.171 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_success": 25740, 2024-03-28T22:21:01.171 INFO:teuthology.orchestra.run.smithi053.stdout: "take": 0, 2024-03-28T22:21:01.171 INFO:teuthology.orchestra.run.smithi053.stdout: "take_sum": 0, 2024-03-28T22:21:01.171 INFO:teuthology.orchestra.run.smithi053.stdout: "put": 368, 2024-03-28T22:21:01.172 INFO:teuthology.orchestra.run.smithi053.stdout: "put_sum": 65282406, 2024-03-28T22:21:01.172 INFO:teuthology.orchestra.run.smithi053.stdout: "wait": { 2024-03-28T22:21:01.172 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:01.172 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:01.172 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:01.172 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-03-28T22:21:01.172 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-03-28T22:21:01.172 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-03-28T22:21:01.930 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:01 smithi063 ceph-mon[36249]: pgmap v324: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:21:01.934 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:01 smithi016 ceph-mon[32374]: pgmap v324: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:21:01.997 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:01 smithi053 ceph-mon[35979]: pgmap v324: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T22:21:02.028 INFO:tasks.check_counter:Daemon mds.e mds.exported=3 2024-03-28T22:21:02.028 INFO:tasks.check_counter:Daemon mds.e mds.imported=26 2024-03-28T22:21:02.028 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=34 2024-03-28T22:21:02.028 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=12 2024-03-28T22:21:02.028 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=6 2024-03-28T22:21:02.029 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-03-28T22:21:02.029 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T22:21:02.029 INFO:tasks.check_counter:Daemon mds.e mds.dir_split=29 2024-03-28T22:21:02.029 DEBUG:tasks.check_counter:Getting stats from h 2024-03-28T22:21:02.029 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-28T22:21:02.346 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T22:21:03.897 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:03 smithi053 ceph-mon[35979]: pgmap v325: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:21:03.930 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:03 smithi063 ceph-mon[36249]: pgmap v325: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:21:03.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:03 smithi016 ceph-mon[32374]: pgmap v325: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:21:03.970 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-03-28T22:21:03.971 INFO:teuthology.orchestra.run.smithi053.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T22:21:03.971 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_messages": 1347, 2024-03-28T22:21:03.971 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages": 1156, 2024-03-28T22:21:03.971 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_bytes": 453109, 2024-03-28T22:21:03.971 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_bytes": 771119, 2024-03-28T22:21:03.971 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_created_connections": 10, 2024-03-28T22:21:03.971 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_active_connections": 5, 2024-03-28T22:21:03.971 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_total_time": 0.107325749, 2024-03-28T22:21:03.972 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_send_time": 0.048877735, 2024-03-28T22:21:03.972 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_recv_time": 0.062133519, 2024-03-28T22:21:03.972 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_fast_dispatch_time": 0.003220015, 2024-03-28T22:21:03.972 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:21:03.972 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 1154, 2024-03-28T22:21:03.972 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.053141305, 2024-03-28T22:21:03.972 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000046049 2024-03-28T22:21:03.973 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:03.973 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:21:03.973 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 2260, 2024-03-28T22:21:03.973 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000415121, 2024-03-28T22:21:03.973 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000183 2024-03-28T22:21:03.973 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:03.973 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-28T22:21:03.973 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_encrypted_bytes": 313296 2024-03-28T22:21:03.974 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:03.974 INFO:teuthology.orchestra.run.smithi053.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T22:21:03.974 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_messages": 90, 2024-03-28T22:21:03.974 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages": 160, 2024-03-28T22:21:03.974 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_bytes": 19052, 2024-03-28T22:21:03.974 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_bytes": 597984, 2024-03-28T22:21:03.974 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_created_connections": 51, 2024-03-28T22:21:03.974 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_active_connections": 3, 2024-03-28T22:21:03.975 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_total_time": 0.022924756, 2024-03-28T22:21:03.975 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_send_time": 0.004070518, 2024-03-28T22:21:03.975 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_recv_time": 0.017419012, 2024-03-28T22:21:03.975 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_fast_dispatch_time": 0.000223467, 2024-03-28T22:21:03.975 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:21:03.975 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 157, 2024-03-28T22:21:03.975 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.020981732, 2024-03-28T22:21:03.975 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000133641 2024-03-28T22:21:03.976 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:03.976 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:21:03.976 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 151, 2024-03-28T22:21:03.976 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000023200, 2024-03-28T22:21:03.976 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000153 2024-03-28T22:21:03.976 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:03.976 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_encrypted_bytes": 10768, 2024-03-28T22:21:03.976 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_encrypted_bytes": 152512 2024-03-28T22:21:03.976 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:03.977 INFO:teuthology.orchestra.run.smithi053.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T22:21:03.977 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_messages": 423, 2024-03-28T22:21:03.977 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages": 427, 2024-03-28T22:21:03.977 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_bytes": 1670427, 2024-03-28T22:21:03.977 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_bytes": 211151, 2024-03-28T22:21:03.977 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_created_connections": 11, 2024-03-28T22:21:03.977 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_active_connections": 6, 2024-03-28T22:21:03.977 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_total_time": 0.049176988, 2024-03-28T22:21:03.978 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_send_time": 0.013504103, 2024-03-28T22:21:03.978 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_recv_time": 0.044883810, 2024-03-28T22:21:03.978 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_fast_dispatch_time": 0.004298001, 2024-03-28T22:21:03.978 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:21:03.978 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 427, 2024-03-28T22:21:03.978 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.017475395, 2024-03-28T22:21:03.978 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000040925 2024-03-28T22:21:03.979 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:03.979 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:21:03.979 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 149, 2024-03-28T22:21:03.979 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000050588, 2024-03-28T22:21:03.979 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000339 2024-03-28T22:21:03.979 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:03.979 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_encrypted_bytes": 1635904, 2024-03-28T22:21:03.979 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_encrypted_bytes": 135440 2024-03-28T22:21:03.980 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:03.980 INFO:teuthology.orchestra.run.smithi053.stdout: "cct": { 2024-03-28T22:21:03.980 INFO:teuthology.orchestra.run.smithi053.stdout: "total_workers": 1, 2024-03-28T22:21:03.980 INFO:teuthology.orchestra.run.smithi053.stdout: "unhealthy_workers": 0 2024-03-28T22:21:03.980 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:03.980 INFO:teuthology.orchestra.run.smithi053.stdout: "finisher-MDSRank": { 2024-03-28T22:21:03.980 INFO:teuthology.orchestra.run.smithi053.stdout: "queue_len": 0, 2024-03-28T22:21:03.980 INFO:teuthology.orchestra.run.smithi053.stdout: "complete_latency": { 2024-03-28T22:21:03.980 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 75, 2024-03-28T22:21:03.981 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.002317299, 2024-03-28T22:21:03.981 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000030897 2024-03-28T22:21:03.981 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-03-28T22:21:03.981 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:03.981 INFO:teuthology.orchestra.run.smithi053.stdout: "finisher-PurgeQueue": { 2024-03-28T22:21:03.981 INFO:teuthology.orchestra.run.smithi053.stdout: "queue_len": 0, 2024-03-28T22:21:03.981 INFO:teuthology.orchestra.run.smithi053.stdout: "complete_latency": { 2024-03-28T22:21:03.981 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 2, 2024-03-28T22:21:03.982 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000009695, 2024-03-28T22:21:03.982 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000004847 2024-03-28T22:21:03.982 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-03-28T22:21:03.982 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:03.982 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": { 2024-03-28T22:21:03.982 INFO:teuthology.orchestra.run.smithi053.stdout: "request": 132, 2024-03-28T22:21:03.982 INFO:teuthology.orchestra.run.smithi053.stdout: "reply": 12, 2024-03-28T22:21:03.982 INFO:teuthology.orchestra.run.smithi053.stdout: "reply_latency": { 2024-03-28T22:21:03.983 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 12, 2024-03-28T22:21:03.983 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.796584896, 2024-03-28T22:21:03.983 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.066382074 2024-03-28T22:21:03.983 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:03.983 INFO:teuthology.orchestra.run.smithi053.stdout: "slow_reply": 0, 2024-03-28T22:21:03.983 INFO:teuthology.orchestra.run.smithi053.stdout: "forward": 120, 2024-03-28T22:21:03.983 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_fetch_complete": 0, 2024-03-28T22:21:03.983 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_fetch_keys": 0, 2024-03-28T22:21:03.984 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_commit": 11, 2024-03-28T22:21:03.984 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_split": 0, 2024-03-28T22:21:03.984 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_merge": 0, 2024-03-28T22:21:03.984 INFO:teuthology.orchestra.run.smithi053.stdout: "inodes": 42, 2024-03-28T22:21:03.984 INFO:teuthology.orchestra.run.smithi053.stdout: "inodes_top": 13, 2024-03-28T22:21:03.984 INFO:teuthology.orchestra.run.smithi053.stdout: "inodes_bottom": 29, 2024-03-28T22:21:03.984 INFO:teuthology.orchestra.run.smithi053.stdout: "inodes_pin_tail": 0, 2024-03-28T22:21:03.984 INFO:teuthology.orchestra.run.smithi053.stdout: "inodes_pinned": 21, 2024-03-28T22:21:03.984 INFO:teuthology.orchestra.run.smithi053.stdout: "inodes_expired": 9, 2024-03-28T22:21:03.985 INFO:teuthology.orchestra.run.smithi053.stdout: "inodes_with_caps": 2, 2024-03-28T22:21:03.985 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": 2, 2024-03-28T22:21:03.985 INFO:teuthology.orchestra.run.smithi053.stdout: "subtrees": 4, 2024-03-28T22:21:03.985 INFO:teuthology.orchestra.run.smithi053.stdout: "traverse": 203, 2024-03-28T22:21:03.985 INFO:teuthology.orchestra.run.smithi053.stdout: "traverse_hit": 55, 2024-03-28T22:21:03.985 INFO:teuthology.orchestra.run.smithi053.stdout: "traverse_forward": 116, 2024-03-28T22:21:03.985 INFO:teuthology.orchestra.run.smithi053.stdout: "traverse_discover": 26, 2024-03-28T22:21:03.985 INFO:teuthology.orchestra.run.smithi053.stdout: "traverse_dir_fetch": 0, 2024-03-28T22:21:03.986 INFO:teuthology.orchestra.run.smithi053.stdout: "traverse_remote_ino": 0, 2024-03-28T22:21:03.986 INFO:teuthology.orchestra.run.smithi053.stdout: "traverse_lock": 1, 2024-03-28T22:21:03.986 INFO:teuthology.orchestra.run.smithi053.stdout: "load_cent": 0, 2024-03-28T22:21:03.986 INFO:teuthology.orchestra.run.smithi053.stdout: "q": 0, 2024-03-28T22:21:03.986 INFO:teuthology.orchestra.run.smithi053.stdout: "exported": 0, 2024-03-28T22:21:03.986 INFO:teuthology.orchestra.run.smithi053.stdout: "exported_inodes": 0, 2024-03-28T22:21:03.986 INFO:teuthology.orchestra.run.smithi053.stdout: "imported": 0, 2024-03-28T22:21:03.986 INFO:teuthology.orchestra.run.smithi053.stdout: "imported_inodes": 0, 2024-03-28T22:21:03.987 INFO:teuthology.orchestra.run.smithi053.stdout: "openino_dir_fetch": 0, 2024-03-28T22:21:03.987 INFO:teuthology.orchestra.run.smithi053.stdout: "openino_backtrace_fetch": 0, 2024-03-28T22:21:03.987 INFO:teuthology.orchestra.run.smithi053.stdout: "openino_peer_discover": 0, 2024-03-28T22:21:03.987 INFO:teuthology.orchestra.run.smithi053.stdout: "root_rfiles": 3452, 2024-03-28T22:21:03.987 INFO:teuthology.orchestra.run.smithi053.stdout: "root_rbytes": 26525988, 2024-03-28T22:21:03.987 INFO:teuthology.orchestra.run.smithi053.stdout: "root_rsnaps": 6, 2024-03-28T22:21:03.987 INFO:teuthology.orchestra.run.smithi053.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T22:21:03.987 INFO:teuthology.orchestra.run.smithi053.stdout: "scrub_set_tag": 0, 2024-03-28T22:21:03.988 INFO:teuthology.orchestra.run.smithi053.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T22:21:03.988 INFO:teuthology.orchestra.run.smithi053.stdout: "scrub_inotable_repaired": 0, 2024-03-28T22:21:03.988 INFO:teuthology.orchestra.run.smithi053.stdout: "scrub_dir_inodes": 0, 2024-03-28T22:21:03.988 INFO:teuthology.orchestra.run.smithi053.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T22:21:03.988 INFO:teuthology.orchestra.run.smithi053.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T22:21:03.988 INFO:teuthology.orchestra.run.smithi053.stdout: "scrub_file_inodes": 0, 2024-03-28T22:21:03.988 INFO:teuthology.orchestra.run.smithi053.stdout: "handle_inode_file_caps": 0, 2024-03-28T22:21:03.988 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T22:21:03.988 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_cap_op_grant": 0, 2024-03-28T22:21:03.989 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T22:21:03.989 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T22:21:03.989 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T22:21:03.989 INFO:teuthology.orchestra.run.smithi053.stdout: "handle_client_caps": 1, 2024-03-28T22:21:03.989 INFO:teuthology.orchestra.run.smithi053.stdout: "handle_client_caps_dirty": 0, 2024-03-28T22:21:03.989 INFO:teuthology.orchestra.run.smithi053.stdout: "handle_client_cap_release": 0, 2024-03-28T22:21:03.989 INFO:teuthology.orchestra.run.smithi053.stdout: "process_request_cap_release": 0 2024-03-28T22:21:03.989 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:03.990 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_cache": { 2024-03-28T22:21:03.990 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_update": 0, 2024-03-28T22:21:03.990 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_update_receipt": 95, 2024-03-28T22:21:03.990 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_try_discover": 60, 2024-03-28T22:21:03.990 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_send_discover": 65, 2024-03-28T22:21:03.990 INFO:teuthology.orchestra.run.smithi053.stdout: "dir_handle_discover": 0, 2024-03-28T22:21:03.990 INFO:teuthology.orchestra.run.smithi053.stdout: "num_strays": 0, 2024-03-28T22:21:03.990 INFO:teuthology.orchestra.run.smithi053.stdout: "num_strays_delayed": 0, 2024-03-28T22:21:03.991 INFO:teuthology.orchestra.run.smithi053.stdout: "num_strays_enqueuing": 0, 2024-03-28T22:21:03.991 INFO:teuthology.orchestra.run.smithi053.stdout: "strays_created": 0, 2024-03-28T22:21:03.991 INFO:teuthology.orchestra.run.smithi053.stdout: "strays_enqueued": 0, 2024-03-28T22:21:03.991 INFO:teuthology.orchestra.run.smithi053.stdout: "strays_reintegrated": 0, 2024-03-28T22:21:03.991 INFO:teuthology.orchestra.run.smithi053.stdout: "strays_migrated": 0, 2024-03-28T22:21:03.991 INFO:teuthology.orchestra.run.smithi053.stdout: "num_recovering_processing": 0, 2024-03-28T22:21:03.991 INFO:teuthology.orchestra.run.smithi053.stdout: "num_recovering_enqueued": 0, 2024-03-28T22:21:03.991 INFO:teuthology.orchestra.run.smithi053.stdout: "num_recovering_prioritized": 0, 2024-03-28T22:21:03.992 INFO:teuthology.orchestra.run.smithi053.stdout: "recovery_started": 0, 2024-03-28T22:21:03.992 INFO:teuthology.orchestra.run.smithi053.stdout: "recovery_completed": 0, 2024-03-28T22:21:03.992 INFO:teuthology.orchestra.run.smithi053.stdout: "ireq_quiesce_path": 0, 2024-03-28T22:21:03.992 INFO:teuthology.orchestra.run.smithi053.stdout: "ireq_quiesce_inode": 0, 2024-03-28T22:21:03.992 INFO:teuthology.orchestra.run.smithi053.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T22:21:03.992 INFO:teuthology.orchestra.run.smithi053.stdout: "ireq_exportdir": 0, 2024-03-28T22:21:03.992 INFO:teuthology.orchestra.run.smithi053.stdout: "ireq_flush": 0, 2024-03-28T22:21:03.992 INFO:teuthology.orchestra.run.smithi053.stdout: "ireq_fragmentdir": 0, 2024-03-28T22:21:03.993 INFO:teuthology.orchestra.run.smithi053.stdout: "ireq_fragstats": 0, 2024-03-28T22:21:03.993 INFO:teuthology.orchestra.run.smithi053.stdout: "ireq_inodestats": 0 2024-03-28T22:21:03.993 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:03.993 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_log": { 2024-03-28T22:21:03.993 INFO:teuthology.orchestra.run.smithi053.stdout: "evlrg": 0, 2024-03-28T22:21:03.993 INFO:teuthology.orchestra.run.smithi053.stdout: "evadd": 12, 2024-03-28T22:21:03.993 INFO:teuthology.orchestra.run.smithi053.stdout: "evex": 0, 2024-03-28T22:21:03.993 INFO:teuthology.orchestra.run.smithi053.stdout: "evtrm": 0, 2024-03-28T22:21:03.993 INFO:teuthology.orchestra.run.smithi053.stdout: "ev": 12, 2024-03-28T22:21:03.994 INFO:teuthology.orchestra.run.smithi053.stdout: "evexg": 0, 2024-03-28T22:21:03.994 INFO:teuthology.orchestra.run.smithi053.stdout: "evexd": 0, 2024-03-28T22:21:03.994 INFO:teuthology.orchestra.run.smithi053.stdout: "segadd": 2, 2024-03-28T22:21:03.994 INFO:teuthology.orchestra.run.smithi053.stdout: "segex": 0, 2024-03-28T22:21:03.994 INFO:teuthology.orchestra.run.smithi053.stdout: "segtrm": 0, 2024-03-28T22:21:03.994 INFO:teuthology.orchestra.run.smithi053.stdout: "seg": 2, 2024-03-28T22:21:03.994 INFO:teuthology.orchestra.run.smithi053.stdout: "segmjr": 2, 2024-03-28T22:21:03.994 INFO:teuthology.orchestra.run.smithi053.stdout: "segexg": 0, 2024-03-28T22:21:03.995 INFO:teuthology.orchestra.run.smithi053.stdout: "segexd": 0, 2024-03-28T22:21:03.995 INFO:teuthology.orchestra.run.smithi053.stdout: "expos": 4194304, 2024-03-28T22:21:03.995 INFO:teuthology.orchestra.run.smithi053.stdout: "wrpos": 4197719, 2024-03-28T22:21:03.995 INFO:teuthology.orchestra.run.smithi053.stdout: "rdpos": 0, 2024-03-28T22:21:03.995 INFO:teuthology.orchestra.run.smithi053.stdout: "jlat": { 2024-03-28T22:21:03.995 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 11, 2024-03-28T22:21:03.995 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.027834890, 2024-03-28T22:21:03.995 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.002530444 2024-03-28T22:21:03.996 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:03.996 INFO:teuthology.orchestra.run.smithi053.stdout: "replayed": 0 2024-03-28T22:21:03.996 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:03.996 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_mem": { 2024-03-28T22:21:03.996 INFO:teuthology.orchestra.run.smithi053.stdout: "ino": 46, 2024-03-28T22:21:03.996 INFO:teuthology.orchestra.run.smithi053.stdout: "ino+": 51, 2024-03-28T22:21:03.996 INFO:teuthology.orchestra.run.smithi053.stdout: "ino-": 5, 2024-03-28T22:21:03.996 INFO:teuthology.orchestra.run.smithi053.stdout: "dir": 51, 2024-03-28T22:21:03.996 INFO:teuthology.orchestra.run.smithi053.stdout: "dir+": 59, 2024-03-28T22:21:03.997 INFO:teuthology.orchestra.run.smithi053.stdout: "dir-": 8, 2024-03-28T22:21:03.997 INFO:teuthology.orchestra.run.smithi053.stdout: "dn": 42, 2024-03-28T22:21:03.997 INFO:teuthology.orchestra.run.smithi053.stdout: "dn+": 51, 2024-03-28T22:21:03.997 INFO:teuthology.orchestra.run.smithi053.stdout: "dn-": 9, 2024-03-28T22:21:03.997 INFO:teuthology.orchestra.run.smithi053.stdout: "cap": 2, 2024-03-28T22:21:03.997 INFO:teuthology.orchestra.run.smithi053.stdout: "cap+": 9, 2024-03-28T22:21:03.997 INFO:teuthology.orchestra.run.smithi053.stdout: "cap-": 7, 2024-03-28T22:21:03.997 INFO:teuthology.orchestra.run.smithi053.stdout: "rss": 51612, 2024-03-28T22:21:03.998 INFO:teuthology.orchestra.run.smithi053.stdout: "heap": 223612 2024-03-28T22:21:03.998 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:03.998 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_server": { 2024-03-28T22:21:03.998 INFO:teuthology.orchestra.run.smithi053.stdout: "dispatch_client_request": 137, 2024-03-28T22:21:03.998 INFO:teuthology.orchestra.run.smithi053.stdout: "dispatch_server_request": 0, 2024-03-28T22:21:03.998 INFO:teuthology.orchestra.run.smithi053.stdout: "handle_client_request": 132, 2024-03-28T22:21:03.998 INFO:teuthology.orchestra.run.smithi053.stdout: "handle_client_session": 32, 2024-03-28T22:21:03.999 INFO:teuthology.orchestra.run.smithi053.stdout: "handle_peer_request": 1, 2024-03-28T22:21:03.999 INFO:teuthology.orchestra.run.smithi053.stdout: "req_create_latency": { 2024-03-28T22:21:03.999 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:03.999 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:03.999 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:03.999 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:03.999 INFO:teuthology.orchestra.run.smithi053.stdout: "req_getattr_latency": { 2024-03-28T22:21:03.999 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 6, 2024-03-28T22:21:04.000 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.002465708, 2024-03-28T22:21:04.000 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000410951 2024-03-28T22:21:04.000 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.000 INFO:teuthology.orchestra.run.smithi053.stdout: "req_getfilelock_latency": { 2024-03-28T22:21:04.000 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:04.000 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:04.000 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:04.000 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.000 INFO:teuthology.orchestra.run.smithi053.stdout: "req_link_latency": { 2024-03-28T22:21:04.001 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:04.001 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:04.001 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:04.001 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.001 INFO:teuthology.orchestra.run.smithi053.stdout: "req_lookup_latency": { 2024-03-28T22:21:04.001 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 6, 2024-03-28T22:21:04.001 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.794119188, 2024-03-28T22:21:04.001 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.132353198 2024-03-28T22:21:04.002 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.002 INFO:teuthology.orchestra.run.smithi053.stdout: "req_lookuphash_latency": { 2024-03-28T22:21:04.002 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:04.002 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:04.002 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:04.002 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.002 INFO:teuthology.orchestra.run.smithi053.stdout: "req_lookupino_latency": { 2024-03-28T22:21:04.003 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:04.003 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:04.003 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:04.003 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.003 INFO:teuthology.orchestra.run.smithi053.stdout: "req_lookupname_latency": { 2024-03-28T22:21:04.003 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:04.003 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:04.003 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:04.003 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.004 INFO:teuthology.orchestra.run.smithi053.stdout: "req_lookupparent_latency": { 2024-03-28T22:21:04.004 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:04.004 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:04.004 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:04.004 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.004 INFO:teuthology.orchestra.run.smithi053.stdout: "req_lookupsnap_latency": { 2024-03-28T22:21:04.004 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:04.004 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:04.005 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:04.005 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.005 INFO:teuthology.orchestra.run.smithi053.stdout: "req_lssnap_latency": { 2024-03-28T22:21:04.005 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:04.005 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:04.005 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:04.005 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.006 INFO:teuthology.orchestra.run.smithi053.stdout: "req_mkdir_latency": { 2024-03-28T22:21:04.006 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:04.006 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:04.006 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:04.006 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.007 INFO:teuthology.orchestra.run.smithi053.stdout: "req_mknod_latency": { 2024-03-28T22:21:04.007 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:04.007 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:04.007 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:04.007 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.007 INFO:teuthology.orchestra.run.smithi053.stdout: "req_mksnap_latency": { 2024-03-28T22:21:04.007 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:04.007 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:04.008 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:04.008 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.008 INFO:teuthology.orchestra.run.smithi053.stdout: "req_open_latency": { 2024-03-28T22:21:04.008 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:04.008 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:04.008 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:04.008 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.008 INFO:teuthology.orchestra.run.smithi053.stdout: "req_readdir_latency": { 2024-03-28T22:21:04.009 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:04.009 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:04.009 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:04.009 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.009 INFO:teuthology.orchestra.run.smithi053.stdout: "req_rename_latency": { 2024-03-28T22:21:04.009 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:04.009 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:04.009 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:04.009 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.010 INFO:teuthology.orchestra.run.smithi053.stdout: "req_renamesnap_latency": { 2024-03-28T22:21:04.010 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:04.010 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:04.010 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:04.010 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.010 INFO:teuthology.orchestra.run.smithi053.stdout: "req_snapdiff_latency": { 2024-03-28T22:21:04.010 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:04.011 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:04.011 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:04.011 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.013 INFO:teuthology.orchestra.run.smithi053.stdout: "req_rmdir_latency": { 2024-03-28T22:21:04.013 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:04.013 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:04.013 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:04.014 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.014 INFO:teuthology.orchestra.run.smithi053.stdout: "req_rmsnap_latency": { 2024-03-28T22:21:04.014 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:04.014 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:04.014 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:04.014 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.014 INFO:teuthology.orchestra.run.smithi053.stdout: "req_rmxattr_latency": { 2024-03-28T22:21:04.014 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:04.015 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:04.015 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:04.015 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.015 INFO:teuthology.orchestra.run.smithi053.stdout: "req_setattr_latency": { 2024-03-28T22:21:04.015 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:04.015 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:04.015 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:04.015 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.016 INFO:teuthology.orchestra.run.smithi053.stdout: "req_setdirlayout_latency": { 2024-03-28T22:21:04.016 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:04.016 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:04.016 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:04.016 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.016 INFO:teuthology.orchestra.run.smithi053.stdout: "req_setfilelock_latency": { 2024-03-28T22:21:04.016 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:04.016 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:04.017 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:04.017 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.017 INFO:teuthology.orchestra.run.smithi053.stdout: "req_setlayout_latency": { 2024-03-28T22:21:04.017 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:04.017 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:04.017 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:04.017 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.017 INFO:teuthology.orchestra.run.smithi053.stdout: "req_setxattr_latency": { 2024-03-28T22:21:04.017 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:04.018 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:04.018 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:04.018 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.018 INFO:teuthology.orchestra.run.smithi053.stdout: "req_symlink_latency": { 2024-03-28T22:21:04.018 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:04.018 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:04.018 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:04.018 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.019 INFO:teuthology.orchestra.run.smithi053.stdout: "req_unlink_latency": { 2024-03-28T22:21:04.019 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:04.019 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:04.019 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:04.019 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.019 INFO:teuthology.orchestra.run.smithi053.stdout: "cap_revoke_eviction": 0, 2024-03-28T22:21:04.019 INFO:teuthology.orchestra.run.smithi053.stdout: "cap_acquisition_throttle": 0, 2024-03-28T22:21:04.019 INFO:teuthology.orchestra.run.smithi053.stdout: "req_getvxattr_latency": { 2024-03-28T22:21:04.020 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:04.020 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:04.020 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:04.020 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-03-28T22:21:04.020 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.020 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_sessions": { 2024-03-28T22:21:04.020 INFO:teuthology.orchestra.run.smithi053.stdout: "session_count": 2, 2024-03-28T22:21:04.020 INFO:teuthology.orchestra.run.smithi053.stdout: "session_add": 6, 2024-03-28T22:21:04.021 INFO:teuthology.orchestra.run.smithi053.stdout: "session_remove": 4, 2024-03-28T22:21:04.021 INFO:teuthology.orchestra.run.smithi053.stdout: "sessions_open": 2, 2024-03-28T22:21:04.021 INFO:teuthology.orchestra.run.smithi053.stdout: "sessions_stale": 0, 2024-03-28T22:21:04.021 INFO:teuthology.orchestra.run.smithi053.stdout: "total_load": 3, 2024-03-28T22:21:04.021 INFO:teuthology.orchestra.run.smithi053.stdout: "average_load": 1, 2024-03-28T22:21:04.021 INFO:teuthology.orchestra.run.smithi053.stdout: "avg_session_uptime": 579, 2024-03-28T22:21:04.021 INFO:teuthology.orchestra.run.smithi053.stdout: "mdthresh_evicted": 0 2024-03-28T22:21:04.021 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.021 INFO:teuthology.orchestra.run.smithi053.stdout: "mempool": { 2024-03-28T22:21:04.022 INFO:teuthology.orchestra.run.smithi053.stdout: "bloom_filter_bytes": 0, 2024-03-28T22:21:04.022 INFO:teuthology.orchestra.run.smithi053.stdout: "bloom_filter_items": 0, 2024-03-28T22:21:04.022 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T22:21:04.022 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_alloc_items": 0, 2024-03-28T22:21:04.022 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T22:21:04.022 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_data_items": 0, 2024-03-28T22:21:04.022 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T22:21:04.022 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T22:21:04.023 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T22:21:04.023 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T22:21:04.023 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T22:21:04.023 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_other_items": 0, 2024-03-28T22:21:04.023 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T22:21:04.023 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T22:21:04.023 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_extent_bytes": 0, 2024-03-28T22:21:04.023 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_extent_items": 0, 2024-03-28T22:21:04.024 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_blob_bytes": 0, 2024-03-28T22:21:04.024 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_blob_items": 0, 2024-03-28T22:21:04.024 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T22:21:04.024 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T22:21:04.024 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T22:21:04.024 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T22:21:04.024 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T22:21:04.024 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_fsck_items": 0, 2024-03-28T22:21:04.025 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_txc_bytes": 0, 2024-03-28T22:21:04.025 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_txc_items": 0, 2024-03-28T22:21:04.025 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T22:21:04.025 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T22:21:04.025 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_writing_bytes": 0, 2024-03-28T22:21:04.025 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_writing_items": 0, 2024-03-28T22:21:04.025 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_bytes": 0, 2024-03-28T22:21:04.025 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_items": 0, 2024-03-28T22:21:04.026 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T22:21:04.026 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_file_reader_items": 0, 2024-03-28T22:21:04.026 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T22:21:04.026 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_file_writer_items": 0, 2024-03-28T22:21:04.026 INFO:teuthology.orchestra.run.smithi053.stdout: "buffer_anon_bytes": 292301, 2024-03-28T22:21:04.026 INFO:teuthology.orchestra.run.smithi053.stdout: "buffer_anon_items": 149, 2024-03-28T22:21:04.026 INFO:teuthology.orchestra.run.smithi053.stdout: "buffer_meta_bytes": 0, 2024-03-28T22:21:04.027 INFO:teuthology.orchestra.run.smithi053.stdout: "buffer_meta_items": 0, 2024-03-28T22:21:04.027 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_bytes": 0, 2024-03-28T22:21:04.027 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_items": 0, 2024-03-28T22:21:04.027 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_mapbl_bytes": 0, 2024-03-28T22:21:04.027 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_mapbl_items": 0, 2024-03-28T22:21:04.027 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_pglog_bytes": 0, 2024-03-28T22:21:04.027 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_pglog_items": 0, 2024-03-28T22:21:04.027 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_bytes": 8160, 2024-03-28T22:21:04.027 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_items": 186, 2024-03-28T22:21:04.028 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T22:21:04.028 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_mapping_items": 0, 2024-03-28T22:21:04.028 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap_bytes": 0, 2024-03-28T22:21:04.028 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap_items": 0, 2024-03-28T22:21:04.028 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_co_bytes": 211499, 2024-03-28T22:21:04.028 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_co_items": 1244, 2024-03-28T22:21:04.028 INFO:teuthology.orchestra.run.smithi053.stdout: "unittest_1_bytes": 0, 2024-03-28T22:21:04.028 INFO:teuthology.orchestra.run.smithi053.stdout: "unittest_1_items": 0, 2024-03-28T22:21:04.029 INFO:teuthology.orchestra.run.smithi053.stdout: "unittest_2_bytes": 0, 2024-03-28T22:21:04.029 INFO:teuthology.orchestra.run.smithi053.stdout: "unittest_2_items": 0 2024-03-28T22:21:04.029 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.029 INFO:teuthology.orchestra.run.smithi053.stdout: "objecter": { 2024-03-28T22:21:04.029 INFO:teuthology.orchestra.run.smithi053.stdout: "op_active": 0, 2024-03-28T22:21:04.029 INFO:teuthology.orchestra.run.smithi053.stdout: "op_laggy": 0, 2024-03-28T22:21:04.029 INFO:teuthology.orchestra.run.smithi053.stdout: "op_send": 56, 2024-03-28T22:21:04.029 INFO:teuthology.orchestra.run.smithi053.stdout: "op_send_bytes": 14947, 2024-03-28T22:21:04.030 INFO:teuthology.orchestra.run.smithi053.stdout: "op_resend": 33, 2024-03-28T22:21:04.030 INFO:teuthology.orchestra.run.smithi053.stdout: "op_reply": 56, 2024-03-28T22:21:04.030 INFO:teuthology.orchestra.run.smithi053.stdout: "op_latency": { 2024-03-28T22:21:04.030 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 56, 2024-03-28T22:21:04.030 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.365992502, 2024-03-28T22:21:04.030 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.006535580 2024-03-28T22:21:04.030 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.030 INFO:teuthology.orchestra.run.smithi053.stdout: "op_inflight": 0, 2024-03-28T22:21:04.031 INFO:teuthology.orchestra.run.smithi053.stdout: "oplen_avg": { 2024-03-28T22:21:04.031 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 56, 2024-03-28T22:21:04.031 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 84 2024-03-28T22:21:04.031 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.031 INFO:teuthology.orchestra.run.smithi053.stdout: "op": 56, 2024-03-28T22:21:04.031 INFO:teuthology.orchestra.run.smithi053.stdout: "op_r": 0, 2024-03-28T22:21:04.031 INFO:teuthology.orchestra.run.smithi053.stdout: "op_w": 56, 2024-03-28T22:21:04.031 INFO:teuthology.orchestra.run.smithi053.stdout: "op_rmw": 0, 2024-03-28T22:21:04.031 INFO:teuthology.orchestra.run.smithi053.stdout: "op_pg": 0, 2024-03-28T22:21:04.032 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_stat": 0, 2024-03-28T22:21:04.032 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_create": 10, 2024-03-28T22:21:04.032 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_read": 0, 2024-03-28T22:21:04.032 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_write": 11, 2024-03-28T22:21:04.032 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_writefull": 10, 2024-03-28T22:21:04.032 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_writesame": 0, 2024-03-28T22:21:04.032 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_append": 0, 2024-03-28T22:21:04.032 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_zero": 0, 2024-03-28T22:21:04.033 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_truncate": 0, 2024-03-28T22:21:04.033 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_delete": 6, 2024-03-28T22:21:04.033 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_mapext": 0, 2024-03-28T22:21:04.033 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_sparse_read": 0, 2024-03-28T22:21:04.033 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_clonerange": 0, 2024-03-28T22:21:04.033 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_getxattr": 0, 2024-03-28T22:21:04.033 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_setxattr": 20, 2024-03-28T22:21:04.033 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_cmpxattr": 0, 2024-03-28T22:21:04.034 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_rmxattr": 0, 2024-03-28T22:21:04.034 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_resetxattrs": 0, 2024-03-28T22:21:04.034 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_call": 0, 2024-03-28T22:21:04.034 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_watch": 0, 2024-03-28T22:21:04.034 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_notify": 0, 2024-03-28T22:21:04.034 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T22:21:04.034 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_pgls": 0, 2024-03-28T22:21:04.034 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_pgls_filter": 0, 2024-03-28T22:21:04.035 INFO:teuthology.orchestra.run.smithi053.stdout: "osdop_other": 0, 2024-03-28T22:21:04.035 INFO:teuthology.orchestra.run.smithi053.stdout: "linger_active": 0, 2024-03-28T22:21:04.035 INFO:teuthology.orchestra.run.smithi053.stdout: "linger_send": 0, 2024-03-28T22:21:04.035 INFO:teuthology.orchestra.run.smithi053.stdout: "linger_resend": 0, 2024-03-28T22:21:04.035 INFO:teuthology.orchestra.run.smithi053.stdout: "linger_ping": 0, 2024-03-28T22:21:04.035 INFO:teuthology.orchestra.run.smithi053.stdout: "poolop_active": 0, 2024-03-28T22:21:04.035 INFO:teuthology.orchestra.run.smithi053.stdout: "poolop_send": 0, 2024-03-28T22:21:04.035 INFO:teuthology.orchestra.run.smithi053.stdout: "poolop_resend": 0, 2024-03-28T22:21:04.036 INFO:teuthology.orchestra.run.smithi053.stdout: "poolstat_active": 0, 2024-03-28T22:21:04.036 INFO:teuthology.orchestra.run.smithi053.stdout: "poolstat_send": 0, 2024-03-28T22:21:04.036 INFO:teuthology.orchestra.run.smithi053.stdout: "poolstat_resend": 0, 2024-03-28T22:21:04.036 INFO:teuthology.orchestra.run.smithi053.stdout: "statfs_active": 0, 2024-03-28T22:21:04.036 INFO:teuthology.orchestra.run.smithi053.stdout: "statfs_send": 0, 2024-03-28T22:21:04.036 INFO:teuthology.orchestra.run.smithi053.stdout: "statfs_resend": 0, 2024-03-28T22:21:04.036 INFO:teuthology.orchestra.run.smithi053.stdout: "command_active": 0, 2024-03-28T22:21:04.036 INFO:teuthology.orchestra.run.smithi053.stdout: "command_send": 0, 2024-03-28T22:21:04.037 INFO:teuthology.orchestra.run.smithi053.stdout: "command_resend": 0, 2024-03-28T22:21:04.037 INFO:teuthology.orchestra.run.smithi053.stdout: "map_epoch": 97, 2024-03-28T22:21:04.037 INFO:teuthology.orchestra.run.smithi053.stdout: "map_full": 0, 2024-03-28T22:21:04.037 INFO:teuthology.orchestra.run.smithi053.stdout: "map_inc": 11, 2024-03-28T22:21:04.037 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_sessions": 10, 2024-03-28T22:21:04.037 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_session_open": 10, 2024-03-28T22:21:04.037 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_session_close": 0, 2024-03-28T22:21:04.037 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_laggy": 0, 2024-03-28T22:21:04.038 INFO:teuthology.orchestra.run.smithi053.stdout: "omap_wr": 24, 2024-03-28T22:21:04.038 INFO:teuthology.orchestra.run.smithi053.stdout: "omap_rd": 0, 2024-03-28T22:21:04.038 INFO:teuthology.orchestra.run.smithi053.stdout: "omap_del": 3 2024-03-28T22:21:04.038 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.038 INFO:teuthology.orchestra.run.smithi053.stdout: "oft": { 2024-03-28T22:21:04.038 INFO:teuthology.orchestra.run.smithi053.stdout: "omap_total_objs": 1, 2024-03-28T22:21:04.038 INFO:teuthology.orchestra.run.smithi053.stdout: "omap_total_kv_pairs": 0, 2024-03-28T22:21:04.039 INFO:teuthology.orchestra.run.smithi053.stdout: "omap_total_updates": 0, 2024-03-28T22:21:04.039 INFO:teuthology.orchestra.run.smithi053.stdout: "omap_total_removes": 0 2024-03-28T22:21:04.039 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.039 INFO:teuthology.orchestra.run.smithi053.stdout: "osd-slow-ops": { 2024-03-28T22:21:04.039 INFO:teuthology.orchestra.run.smithi053.stdout: "slow_ops_count": 0 2024-03-28T22:21:04.039 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.039 INFO:teuthology.orchestra.run.smithi053.stdout: "purge_queue": { 2024-03-28T22:21:04.039 INFO:teuthology.orchestra.run.smithi053.stdout: "pq_executing_ops": 0, 2024-03-28T22:21:04.040 INFO:teuthology.orchestra.run.smithi053.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T22:21:04.040 INFO:teuthology.orchestra.run.smithi053.stdout: "pq_executing": 0, 2024-03-28T22:21:04.040 INFO:teuthology.orchestra.run.smithi053.stdout: "pq_executing_high_water": 0, 2024-03-28T22:21:04.040 INFO:teuthology.orchestra.run.smithi053.stdout: "pq_executed_ops": 0, 2024-03-28T22:21:04.040 INFO:teuthology.orchestra.run.smithi053.stdout: "pq_executed": 0, 2024-03-28T22:21:04.040 INFO:teuthology.orchestra.run.smithi053.stdout: "pq_item_in_journal": 0 2024-03-28T22:21:04.040 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.040 INFO:teuthology.orchestra.run.smithi053.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T22:21:04.040 INFO:teuthology.orchestra.run.smithi053.stdout: "val": 0, 2024-03-28T22:21:04.041 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 104857600, 2024-03-28T22:21:04.041 INFO:teuthology.orchestra.run.smithi053.stdout: "get_started": 0, 2024-03-28T22:21:04.041 INFO:teuthology.orchestra.run.smithi053.stdout: "get": 1856, 2024-03-28T22:21:04.041 INFO:teuthology.orchestra.run.smithi053.stdout: "get_sum": 1958718, 2024-03-28T22:21:04.041 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_fail": 0, 2024-03-28T22:21:04.041 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_success": 1856, 2024-03-28T22:21:04.041 INFO:teuthology.orchestra.run.smithi053.stdout: "take": 0, 2024-03-28T22:21:04.041 INFO:teuthology.orchestra.run.smithi053.stdout: "take_sum": 0, 2024-03-28T22:21:04.042 INFO:teuthology.orchestra.run.smithi053.stdout: "put": 1856, 2024-03-28T22:21:04.042 INFO:teuthology.orchestra.run.smithi053.stdout: "put_sum": 1958718, 2024-03-28T22:21:04.042 INFO:teuthology.orchestra.run.smithi053.stdout: "wait": { 2024-03-28T22:21:04.042 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:04.042 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:04.042 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:04.042 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-03-28T22:21:04.042 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.043 INFO:teuthology.orchestra.run.smithi053.stdout: "throttle-objecter_bytes": { 2024-03-28T22:21:04.043 INFO:teuthology.orchestra.run.smithi053.stdout: "val": 0, 2024-03-28T22:21:04.043 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 104857600, 2024-03-28T22:21:04.043 INFO:teuthology.orchestra.run.smithi053.stdout: "get_started": 0, 2024-03-28T22:21:04.043 INFO:teuthology.orchestra.run.smithi053.stdout: "get": 0, 2024-03-28T22:21:04.043 INFO:teuthology.orchestra.run.smithi053.stdout: "get_sum": 0, 2024-03-28T22:21:04.043 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_fail": 0, 2024-03-28T22:21:04.043 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_success": 0, 2024-03-28T22:21:04.043 INFO:teuthology.orchestra.run.smithi053.stdout: "take": 56, 2024-03-28T22:21:04.044 INFO:teuthology.orchestra.run.smithi053.stdout: "take_sum": 14947, 2024-03-28T22:21:04.044 INFO:teuthology.orchestra.run.smithi053.stdout: "put": 56, 2024-03-28T22:21:04.044 INFO:teuthology.orchestra.run.smithi053.stdout: "put_sum": 14947, 2024-03-28T22:21:04.044 INFO:teuthology.orchestra.run.smithi053.stdout: "wait": { 2024-03-28T22:21:04.044 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:04.044 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:04.044 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:04.044 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-03-28T22:21:04.045 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.045 INFO:teuthology.orchestra.run.smithi053.stdout: "throttle-objecter_ops": { 2024-03-28T22:21:04.045 INFO:teuthology.orchestra.run.smithi053.stdout: "val": 0, 2024-03-28T22:21:04.045 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 1024, 2024-03-28T22:21:04.045 INFO:teuthology.orchestra.run.smithi053.stdout: "get_started": 0, 2024-03-28T22:21:04.045 INFO:teuthology.orchestra.run.smithi053.stdout: "get": 0, 2024-03-28T22:21:04.045 INFO:teuthology.orchestra.run.smithi053.stdout: "get_sum": 0, 2024-03-28T22:21:04.045 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_fail": 0, 2024-03-28T22:21:04.046 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_success": 0, 2024-03-28T22:21:04.046 INFO:teuthology.orchestra.run.smithi053.stdout: "take": 56, 2024-03-28T22:21:04.046 INFO:teuthology.orchestra.run.smithi053.stdout: "take_sum": 56, 2024-03-28T22:21:04.046 INFO:teuthology.orchestra.run.smithi053.stdout: "put": 56, 2024-03-28T22:21:04.046 INFO:teuthology.orchestra.run.smithi053.stdout: "put_sum": 56, 2024-03-28T22:21:04.046 INFO:teuthology.orchestra.run.smithi053.stdout: "wait": { 2024-03-28T22:21:04.046 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:04.046 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:04.047 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:04.047 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-03-28T22:21:04.047 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.047 INFO:teuthology.orchestra.run.smithi053.stdout: "throttle-write_buf_throttle": { 2024-03-28T22:21:04.047 INFO:teuthology.orchestra.run.smithi053.stdout: "val": 0, 2024-03-28T22:21:04.047 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 3758096384, 2024-03-28T22:21:04.047 INFO:teuthology.orchestra.run.smithi053.stdout: "get_started": 0, 2024-03-28T22:21:04.047 INFO:teuthology.orchestra.run.smithi053.stdout: "get": 0, 2024-03-28T22:21:04.047 INFO:teuthology.orchestra.run.smithi053.stdout: "get_sum": 0, 2024-03-28T22:21:04.048 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_fail": 0, 2024-03-28T22:21:04.048 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_success": 0, 2024-03-28T22:21:04.048 INFO:teuthology.orchestra.run.smithi053.stdout: "take": 0, 2024-03-28T22:21:04.048 INFO:teuthology.orchestra.run.smithi053.stdout: "take_sum": 0, 2024-03-28T22:21:04.048 INFO:teuthology.orchestra.run.smithi053.stdout: "put": 0, 2024-03-28T22:21:04.048 INFO:teuthology.orchestra.run.smithi053.stdout: "put_sum": 0, 2024-03-28T22:21:04.048 INFO:teuthology.orchestra.run.smithi053.stdout: "wait": { 2024-03-28T22:21:04.048 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:04.049 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:04.049 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:04.049 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-03-28T22:21:04.049 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:04.049 INFO:teuthology.orchestra.run.smithi053.stdout: "throttle-write_buf_throttle-0x55e97b2a0460": { 2024-03-28T22:21:04.049 INFO:teuthology.orchestra.run.smithi053.stdout: "val": 0, 2024-03-28T22:21:04.049 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 3758096384, 2024-03-28T22:21:04.049 INFO:teuthology.orchestra.run.smithi053.stdout: "get_started": 0, 2024-03-28T22:21:04.050 INFO:teuthology.orchestra.run.smithi053.stdout: "get": 12, 2024-03-28T22:21:04.050 INFO:teuthology.orchestra.run.smithi053.stdout: "get_sum": 3415, 2024-03-28T22:21:04.050 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_fail": 0, 2024-03-28T22:21:04.050 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_success": 12, 2024-03-28T22:21:04.050 INFO:teuthology.orchestra.run.smithi053.stdout: "take": 0, 2024-03-28T22:21:04.050 INFO:teuthology.orchestra.run.smithi053.stdout: "take_sum": 0, 2024-03-28T22:21:04.050 INFO:teuthology.orchestra.run.smithi053.stdout: "put": 11, 2024-03-28T22:21:04.050 INFO:teuthology.orchestra.run.smithi053.stdout: "put_sum": 3415, 2024-03-28T22:21:04.051 INFO:teuthology.orchestra.run.smithi053.stdout: "wait": { 2024-03-28T22:21:04.051 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:04.051 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:04.051 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:04.051 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-03-28T22:21:04.051 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-03-28T22:21:04.051 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-03-28T22:21:04.748 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-03-28T22:21:04.749 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-03-28T22:21:04.749 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=0 2024-03-28T22:21:04.749 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=95 2024-03-28T22:21:04.749 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-03-28T22:21:04.749 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-03-28T22:21:04.749 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T22:21:04.749 INFO:tasks.check_counter:Daemon mds.h mds.dir_split=0 2024-03-28T22:21:04.749 DEBUG:tasks.check_counter:Getting stats from k 2024-03-28T22:21:04.750 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-28T22:21:05.018 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T22:21:05.796 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:05 smithi053 ceph-mon[35979]: pgmap v326: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:21:05.930 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:05 smithi063 ceph-mon[36249]: pgmap v326: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:21:05.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:05 smithi016 ceph-mon[32374]: pgmap v326: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-28T22:21:06.725 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-03-28T22:21:06.725 INFO:teuthology.orchestra.run.smithi053.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T22:21:06.725 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_messages": 0, 2024-03-28T22:21:06.726 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages": 2, 2024-03-28T22:21:06.726 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_bytes": 0, 2024-03-28T22:21:06.726 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_bytes": 0, 2024-03-28T22:21:06.726 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_created_connections": 4, 2024-03-28T22:21:06.726 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_active_connections": 0, 2024-03-28T22:21:06.726 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_total_time": 0.001396797, 2024-03-28T22:21:06.726 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_send_time": 0.000000000, 2024-03-28T22:21:06.726 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_recv_time": 0.000312654, 2024-03-28T22:21:06.727 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T22:21:06.727 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:21:06.727 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:06.727 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:06.727 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:06.727 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:06.727 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:21:06.727 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:06.728 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:06.728 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:06.728 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:06.728 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T22:21:06.728 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T22:21:06.728 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:06.728 INFO:teuthology.orchestra.run.smithi053.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T22:21:06.728 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_messages": 228, 2024-03-28T22:21:06.729 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages": 215, 2024-03-28T22:21:06.729 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_bytes": 1572768, 2024-03-28T22:21:06.729 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_bytes": 131072, 2024-03-28T22:21:06.729 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_created_connections": 2, 2024-03-28T22:21:06.729 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_active_connections": 1, 2024-03-28T22:21:06.729 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_total_time": 0.039152002, 2024-03-28T22:21:06.729 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_send_time": 0.008823262, 2024-03-28T22:21:06.729 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_recv_time": 0.037722515, 2024-03-28T22:21:06.729 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_fast_dispatch_time": 0.003931446, 2024-03-28T22:21:06.730 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:21:06.730 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 215, 2024-03-28T22:21:06.730 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.006986424, 2024-03-28T22:21:06.730 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000032494 2024-03-28T22:21:06.730 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:06.730 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:21:06.730 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:06.730 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:06.730 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:06.731 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:06.731 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_encrypted_bytes": 1572768, 2024-03-28T22:21:06.731 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_encrypted_bytes": 131072 2024-03-28T22:21:06.731 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:06.731 INFO:teuthology.orchestra.run.smithi053.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T22:21:06.731 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_messages": 2, 2024-03-28T22:21:06.731 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages": 162, 2024-03-28T22:21:06.731 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_bytes": 320, 2024-03-28T22:21:06.732 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_bytes": 184080, 2024-03-28T22:21:06.732 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_created_connections": 5, 2024-03-28T22:21:06.732 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_active_connections": 3, 2024-03-28T22:21:06.732 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_total_time": 0.007755851, 2024-03-28T22:21:06.732 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_send_time": 0.004825357, 2024-03-28T22:21:06.732 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_recv_time": 0.001401196, 2024-03-28T22:21:06.732 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T22:21:06.732 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:21:06.733 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 159, 2024-03-28T22:21:06.733 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.009300178, 2024-03-28T22:21:06.733 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000058491 2024-03-28T22:21:06.733 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:06.733 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:21:06.733 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:06.734 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:06.734 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:06.734 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:06.734 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-28T22:21:06.734 INFO:teuthology.orchestra.run.smithi053.stdout: "msgr_send_encrypted_bytes": 184080 2024-03-28T22:21:06.734 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:06.735 INFO:teuthology.orchestra.run.smithi053.stdout: "cct": { 2024-03-28T22:21:06.735 INFO:teuthology.orchestra.run.smithi053.stdout: "total_workers": 0, 2024-03-28T22:21:06.735 INFO:teuthology.orchestra.run.smithi053.stdout: "unhealthy_workers": 0 2024-03-28T22:21:06.735 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:06.735 INFO:teuthology.orchestra.run.smithi053.stdout: "mempool": { 2024-03-28T22:21:06.735 INFO:teuthology.orchestra.run.smithi053.stdout: "bloom_filter_bytes": 0, 2024-03-28T22:21:06.735 INFO:teuthology.orchestra.run.smithi053.stdout: "bloom_filter_items": 0, 2024-03-28T22:21:06.736 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T22:21:06.736 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_alloc_items": 0, 2024-03-28T22:21:06.736 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T22:21:06.736 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_data_items": 0, 2024-03-28T22:21:06.736 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T22:21:06.736 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T22:21:06.736 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T22:21:06.736 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T22:21:06.737 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T22:21:06.737 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_other_items": 0, 2024-03-28T22:21:06.737 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T22:21:06.737 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T22:21:06.737 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_extent_bytes": 0, 2024-03-28T22:21:06.737 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_extent_items": 0, 2024-03-28T22:21:06.737 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_blob_bytes": 0, 2024-03-28T22:21:06.737 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_blob_items": 0, 2024-03-28T22:21:06.738 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T22:21:06.738 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T22:21:06.738 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T22:21:06.738 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T22:21:06.738 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T22:21:06.738 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_fsck_items": 0, 2024-03-28T22:21:06.738 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_txc_bytes": 0, 2024-03-28T22:21:06.739 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_txc_items": 0, 2024-03-28T22:21:06.739 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T22:21:06.739 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T22:21:06.739 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_writing_bytes": 0, 2024-03-28T22:21:06.739 INFO:teuthology.orchestra.run.smithi053.stdout: "bluestore_writing_items": 0, 2024-03-28T22:21:06.739 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_bytes": 0, 2024-03-28T22:21:06.739 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_items": 0, 2024-03-28T22:21:06.739 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T22:21:06.740 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_file_reader_items": 0, 2024-03-28T22:21:06.740 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T22:21:06.740 INFO:teuthology.orchestra.run.smithi053.stdout: "bluefs_file_writer_items": 0, 2024-03-28T22:21:06.740 INFO:teuthology.orchestra.run.smithi053.stdout: "buffer_anon_bytes": 148995, 2024-03-28T22:21:06.740 INFO:teuthology.orchestra.run.smithi053.stdout: "buffer_anon_items": 31, 2024-03-28T22:21:06.740 INFO:teuthology.orchestra.run.smithi053.stdout: "buffer_meta_bytes": 0, 2024-03-28T22:21:06.740 INFO:teuthology.orchestra.run.smithi053.stdout: "buffer_meta_items": 0, 2024-03-28T22:21:06.741 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_bytes": 0, 2024-03-28T22:21:06.741 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_items": 0, 2024-03-28T22:21:06.741 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_mapbl_bytes": 0, 2024-03-28T22:21:06.741 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_mapbl_items": 0, 2024-03-28T22:21:06.741 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_pglog_bytes": 0, 2024-03-28T22:21:06.741 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_pglog_items": 0, 2024-03-28T22:21:06.741 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_bytes": 0, 2024-03-28T22:21:06.741 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_items": 0, 2024-03-28T22:21:06.742 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T22:21:06.742 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_mapping_items": 0, 2024-03-28T22:21:06.742 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap_bytes": 0, 2024-03-28T22:21:06.742 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap_items": 0, 2024-03-28T22:21:06.742 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_co_bytes": 928, 2024-03-28T22:21:06.742 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_co_items": 2, 2024-03-28T22:21:06.743 INFO:teuthology.orchestra.run.smithi053.stdout: "unittest_1_bytes": 0, 2024-03-28T22:21:06.743 INFO:teuthology.orchestra.run.smithi053.stdout: "unittest_1_items": 0, 2024-03-28T22:21:06.743 INFO:teuthology.orchestra.run.smithi053.stdout: "unittest_2_bytes": 0, 2024-03-28T22:21:06.743 INFO:teuthology.orchestra.run.smithi053.stdout: "unittest_2_items": 0 2024-03-28T22:21:06.743 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-28T22:21:06.743 INFO:teuthology.orchestra.run.smithi053.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T22:21:06.743 INFO:teuthology.orchestra.run.smithi053.stdout: "val": 0, 2024-03-28T22:21:06.744 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 104857600, 2024-03-28T22:21:06.744 INFO:teuthology.orchestra.run.smithi053.stdout: "get_started": 0, 2024-03-28T22:21:06.744 INFO:teuthology.orchestra.run.smithi053.stdout: "get": 226, 2024-03-28T22:21:06.744 INFO:teuthology.orchestra.run.smithi053.stdout: "get_sum": 1538108, 2024-03-28T22:21:06.744 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_fail": 0, 2024-03-28T22:21:06.744 INFO:teuthology.orchestra.run.smithi053.stdout: "get_or_fail_success": 226, 2024-03-28T22:21:06.744 INFO:teuthology.orchestra.run.smithi053.stdout: "take": 0, 2024-03-28T22:21:06.744 INFO:teuthology.orchestra.run.smithi053.stdout: "take_sum": 0, 2024-03-28T22:21:06.745 INFO:teuthology.orchestra.run.smithi053.stdout: "put": 226, 2024-03-28T22:21:06.745 INFO:teuthology.orchestra.run.smithi053.stdout: "put_sum": 1538108, 2024-03-28T22:21:06.745 INFO:teuthology.orchestra.run.smithi053.stdout: "wait": { 2024-03-28T22:21:06.745 INFO:teuthology.orchestra.run.smithi053.stdout: "avgcount": 0, 2024-03-28T22:21:06.745 INFO:teuthology.orchestra.run.smithi053.stdout: "sum": 0.000000000, 2024-03-28T22:21:06.745 INFO:teuthology.orchestra.run.smithi053.stdout: "avgtime": 0.000000000 2024-03-28T22:21:06.745 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-03-28T22:21:06.746 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-03-28T22:21:06.746 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-03-28T22:21:07.405 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-03-28T22:21:07.405 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-03-28T22:21:07.405 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.k 2024-03-28T22:21:07.405 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.k 2024-03-28T22:21:07.405 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-03-28T22:21:07.405 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-03-28T22:21:07.405 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-03-28T22:21:07.405 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.k 2024-03-28T22:21:07.406 DEBUG:tasks.check_counter:Getting stats from c 2024-03-28T22:21:07.406 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-28T22:21:07.670 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T22:21:07.792 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:07 smithi063 ceph-mon[36249]: pgmap v327: 97 pgs: 97 active+clean; 91 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-28T22:21:07.934 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:07 smithi016 ceph-mon[32374]: pgmap v327: 97 pgs: 97 active+clean; 91 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-28T22:21:08.046 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:07 smithi053 ceph-mon[35979]: pgmap v327: 97 pgs: 97 active+clean; 91 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-28T22:21:09.510 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-28T22:21:09.510 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T22:21:09.510 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 2195, 2024-03-28T22:21:09.510 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 2195, 2024-03-28T22:21:09.511 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 626126, 2024-03-28T22:21:09.511 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 697193, 2024-03-28T22:21:09.511 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 6, 2024-03-28T22:21:09.511 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 2, 2024-03-28T22:21:09.511 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.192319161, 2024-03-28T22:21:09.511 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.063335890, 2024-03-28T22:21:09.511 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.048883155, 2024-03-28T22:21:09.511 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.064173503, 2024-03-28T22:21:09.512 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:21:09.512 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 2194, 2024-03-28T22:21:09.512 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.083066019, 2024-03-28T22:21:09.512 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000037860 2024-03-28T22:21:09.512 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.512 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:21:09.512 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.512 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.513 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.513 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.513 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-28T22:21:09.513 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 448 2024-03-28T22:21:09.513 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.513 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T22:21:09.513 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 246, 2024-03-28T22:21:09.514 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 228, 2024-03-28T22:21:09.514 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 1619964, 2024-03-28T22:21:09.514 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 132170, 2024-03-28T22:21:09.514 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 3, 2024-03-28T22:21:09.514 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 2, 2024-03-28T22:21:09.514 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.048094931, 2024-03-28T22:21:09.514 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.015158711, 2024-03-28T22:21:09.514 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.036384116, 2024-03-28T22:21:09.515 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.005463815, 2024-03-28T22:21:09.515 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:21:09.515 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 227, 2024-03-28T22:21:09.515 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.014114147, 2024-03-28T22:21:09.515 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000062176 2024-03-28T22:21:09.515 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.515 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:21:09.516 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.516 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.516 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.516 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.516 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 1619408, 2024-03-28T22:21:09.516 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 131472 2024-03-28T22:21:09.516 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.516 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T22:21:09.517 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 3, 2024-03-28T22:21:09.517 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 161, 2024-03-28T22:21:09.517 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 588, 2024-03-28T22:21:09.517 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 445006, 2024-03-28T22:21:09.517 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 6, 2024-03-28T22:21:09.517 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 4, 2024-03-28T22:21:09.517 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.011290377, 2024-03-28T22:21:09.517 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.006643036, 2024-03-28T22:21:09.518 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.002528163, 2024-03-28T22:21:09.518 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.000029516, 2024-03-28T22:21:09.518 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:21:09.518 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 158, 2024-03-28T22:21:09.518 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.013443359, 2024-03-28T22:21:09.518 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000085084 2024-03-28T22:21:09.518 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.518 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:21:09.519 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.519 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.519 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.519 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.519 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-28T22:21:09.519 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 444688 2024-03-28T22:21:09.519 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.519 INFO:teuthology.orchestra.run.smithi063.stdout: "cct": { 2024-03-28T22:21:09.520 INFO:teuthology.orchestra.run.smithi063.stdout: "total_workers": 1, 2024-03-28T22:21:09.520 INFO:teuthology.orchestra.run.smithi063.stdout: "unhealthy_workers": 0 2024-03-28T22:21:09.520 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.520 INFO:teuthology.orchestra.run.smithi063.stdout: "finisher-MDSRank": { 2024-03-28T22:21:09.520 INFO:teuthology.orchestra.run.smithi063.stdout: "queue_len": 0, 2024-03-28T22:21:09.520 INFO:teuthology.orchestra.run.smithi063.stdout: "complete_latency": { 2024-03-28T22:21:09.520 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 3711, 2024-03-28T22:21:09.520 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.098689204, 2024-03-28T22:21:09.521 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000026593 2024-03-28T22:21:09.521 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-28T22:21:09.521 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.521 INFO:teuthology.orchestra.run.smithi063.stdout: "finisher-PurgeQueue": { 2024-03-28T22:21:09.521 INFO:teuthology.orchestra.run.smithi063.stdout: "queue_len": 0, 2024-03-28T22:21:09.521 INFO:teuthology.orchestra.run.smithi063.stdout: "complete_latency": { 2024-03-28T22:21:09.521 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.521 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.521 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.522 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-28T22:21:09.522 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.522 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-28T22:21:09.522 INFO:teuthology.orchestra.run.smithi063.stdout: "request": 0, 2024-03-28T22:21:09.522 INFO:teuthology.orchestra.run.smithi063.stdout: "reply": 0, 2024-03-28T22:21:09.522 INFO:teuthology.orchestra.run.smithi063.stdout: "reply_latency": { 2024-03-28T22:21:09.522 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.522 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.523 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.523 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.523 INFO:teuthology.orchestra.run.smithi063.stdout: "slow_reply": 0, 2024-03-28T22:21:09.523 INFO:teuthology.orchestra.run.smithi063.stdout: "forward": 0, 2024-03-28T22:21:09.523 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_fetch_complete": 0, 2024-03-28T22:21:09.523 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_fetch_keys": 0, 2024-03-28T22:21:09.523 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_commit": 0, 2024-03-28T22:21:09.523 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_split": 0, 2024-03-28T22:21:09.524 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_merge": 0, 2024-03-28T22:21:09.524 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes": 0, 2024-03-28T22:21:09.524 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_top": 0, 2024-03-28T22:21:09.524 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_bottom": 0, 2024-03-28T22:21:09.524 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_pin_tail": 0, 2024-03-28T22:21:09.524 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_pinned": 0, 2024-03-28T22:21:09.524 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_expired": 0, 2024-03-28T22:21:09.524 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_with_caps": 0, 2024-03-28T22:21:09.525 INFO:teuthology.orchestra.run.smithi063.stdout: "caps": 0, 2024-03-28T22:21:09.525 INFO:teuthology.orchestra.run.smithi063.stdout: "subtrees": 1, 2024-03-28T22:21:09.525 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse": 0, 2024-03-28T22:21:09.525 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_hit": 0, 2024-03-28T22:21:09.525 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_forward": 0, 2024-03-28T22:21:09.525 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_discover": 0, 2024-03-28T22:21:09.525 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_dir_fetch": 0, 2024-03-28T22:21:09.525 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_remote_ino": 0, 2024-03-28T22:21:09.526 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_lock": 0, 2024-03-28T22:21:09.526 INFO:teuthology.orchestra.run.smithi063.stdout: "load_cent": 0, 2024-03-28T22:21:09.526 INFO:teuthology.orchestra.run.smithi063.stdout: "q": 0, 2024-03-28T22:21:09.526 INFO:teuthology.orchestra.run.smithi063.stdout: "exported": 0, 2024-03-28T22:21:09.526 INFO:teuthology.orchestra.run.smithi063.stdout: "exported_inodes": 0, 2024-03-28T22:21:09.526 INFO:teuthology.orchestra.run.smithi063.stdout: "imported": 0, 2024-03-28T22:21:09.526 INFO:teuthology.orchestra.run.smithi063.stdout: "imported_inodes": 0, 2024-03-28T22:21:09.526 INFO:teuthology.orchestra.run.smithi063.stdout: "openino_dir_fetch": 0, 2024-03-28T22:21:09.527 INFO:teuthology.orchestra.run.smithi063.stdout: "openino_backtrace_fetch": 0, 2024-03-28T22:21:09.527 INFO:teuthology.orchestra.run.smithi063.stdout: "openino_peer_discover": 0, 2024-03-28T22:21:09.527 INFO:teuthology.orchestra.run.smithi063.stdout: "root_rfiles": 0, 2024-03-28T22:21:09.527 INFO:teuthology.orchestra.run.smithi063.stdout: "root_rbytes": 0, 2024-03-28T22:21:09.527 INFO:teuthology.orchestra.run.smithi063.stdout: "root_rsnaps": 0, 2024-03-28T22:21:09.527 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T22:21:09.527 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_set_tag": 0, 2024-03-28T22:21:09.527 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T22:21:09.528 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_inotable_repaired": 0, 2024-03-28T22:21:09.528 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_dir_inodes": 0, 2024-03-28T22:21:09.528 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T22:21:09.528 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T22:21:09.528 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_file_inodes": 0, 2024-03-28T22:21:09.528 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_inode_file_caps": 0, 2024-03-28T22:21:09.528 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T22:21:09.528 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_grant": 0, 2024-03-28T22:21:09.528 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T22:21:09.529 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T22:21:09.529 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T22:21:09.529 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_caps": 0, 2024-03-28T22:21:09.529 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_caps_dirty": 0, 2024-03-28T22:21:09.529 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_cap_release": 0, 2024-03-28T22:21:09.529 INFO:teuthology.orchestra.run.smithi063.stdout: "process_request_cap_release": 0 2024-03-28T22:21:09.529 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.529 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_cache": { 2024-03-28T22:21:09.530 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_update": 0, 2024-03-28T22:21:09.530 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_update_receipt": 0, 2024-03-28T22:21:09.530 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_try_discover": 0, 2024-03-28T22:21:09.530 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_send_discover": 0, 2024-03-28T22:21:09.530 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_handle_discover": 0, 2024-03-28T22:21:09.530 INFO:teuthology.orchestra.run.smithi063.stdout: "num_strays": 0, 2024-03-28T22:21:09.530 INFO:teuthology.orchestra.run.smithi063.stdout: "num_strays_delayed": 0, 2024-03-28T22:21:09.530 INFO:teuthology.orchestra.run.smithi063.stdout: "num_strays_enqueuing": 0, 2024-03-28T22:21:09.531 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_created": 0, 2024-03-28T22:21:09.531 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_enqueued": 0, 2024-03-28T22:21:09.531 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_reintegrated": 0, 2024-03-28T22:21:09.531 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_migrated": 0, 2024-03-28T22:21:09.531 INFO:teuthology.orchestra.run.smithi063.stdout: "num_recovering_processing": 0, 2024-03-28T22:21:09.531 INFO:teuthology.orchestra.run.smithi063.stdout: "num_recovering_enqueued": 0, 2024-03-28T22:21:09.531 INFO:teuthology.orchestra.run.smithi063.stdout: "num_recovering_prioritized": 0, 2024-03-28T22:21:09.531 INFO:teuthology.orchestra.run.smithi063.stdout: "recovery_started": 0, 2024-03-28T22:21:09.531 INFO:teuthology.orchestra.run.smithi063.stdout: "recovery_completed": 0, 2024-03-28T22:21:09.532 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_quiesce_path": 0, 2024-03-28T22:21:09.532 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_quiesce_inode": 0, 2024-03-28T22:21:09.532 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T22:21:09.532 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_exportdir": 0, 2024-03-28T22:21:09.532 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_flush": 0, 2024-03-28T22:21:09.532 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_fragmentdir": 0, 2024-03-28T22:21:09.532 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_fragstats": 0, 2024-03-28T22:21:09.532 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_inodestats": 0 2024-03-28T22:21:09.533 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.533 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_log": { 2024-03-28T22:21:09.533 INFO:teuthology.orchestra.run.smithi063.stdout: "evlrg": 0, 2024-03-28T22:21:09.533 INFO:teuthology.orchestra.run.smithi063.stdout: "evadd": 0, 2024-03-28T22:21:09.533 INFO:teuthology.orchestra.run.smithi063.stdout: "evex": 0, 2024-03-28T22:21:09.533 INFO:teuthology.orchestra.run.smithi063.stdout: "evtrm": 0, 2024-03-28T22:21:09.533 INFO:teuthology.orchestra.run.smithi063.stdout: "ev": 12, 2024-03-28T22:21:09.533 INFO:teuthology.orchestra.run.smithi063.stdout: "evexg": 0, 2024-03-28T22:21:09.533 INFO:teuthology.orchestra.run.smithi063.stdout: "evexd": 0, 2024-03-28T22:21:09.534 INFO:teuthology.orchestra.run.smithi063.stdout: "segadd": 0, 2024-03-28T22:21:09.534 INFO:teuthology.orchestra.run.smithi063.stdout: "segex": 0, 2024-03-28T22:21:09.534 INFO:teuthology.orchestra.run.smithi063.stdout: "segtrm": 0, 2024-03-28T22:21:09.534 INFO:teuthology.orchestra.run.smithi063.stdout: "seg": 2, 2024-03-28T22:21:09.534 INFO:teuthology.orchestra.run.smithi063.stdout: "segmjr": 2, 2024-03-28T22:21:09.534 INFO:teuthology.orchestra.run.smithi063.stdout: "segexg": 0, 2024-03-28T22:21:09.534 INFO:teuthology.orchestra.run.smithi063.stdout: "segexd": 0, 2024-03-28T22:21:09.534 INFO:teuthology.orchestra.run.smithi063.stdout: "expos": 4194304, 2024-03-28T22:21:09.535 INFO:teuthology.orchestra.run.smithi063.stdout: "wrpos": 4197719, 2024-03-28T22:21:09.535 INFO:teuthology.orchestra.run.smithi063.stdout: "rdpos": 4197578, 2024-03-28T22:21:09.535 INFO:teuthology.orchestra.run.smithi063.stdout: "jlat": { 2024-03-28T22:21:09.535 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.535 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.535 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.535 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.535 INFO:teuthology.orchestra.run.smithi063.stdout: "replayed": 12 2024-03-28T22:21:09.536 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.536 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_mem": { 2024-03-28T22:21:09.536 INFO:teuthology.orchestra.run.smithi063.stdout: "ino": 3, 2024-03-28T22:21:09.536 INFO:teuthology.orchestra.run.smithi063.stdout: "ino+": 3, 2024-03-28T22:21:09.536 INFO:teuthology.orchestra.run.smithi063.stdout: "ino-": 0, 2024-03-28T22:21:09.536 INFO:teuthology.orchestra.run.smithi063.stdout: "dir": 1, 2024-03-28T22:21:09.536 INFO:teuthology.orchestra.run.smithi063.stdout: "dir+": 1, 2024-03-28T22:21:09.537 INFO:teuthology.orchestra.run.smithi063.stdout: "dir-": 0, 2024-03-28T22:21:09.537 INFO:teuthology.orchestra.run.smithi063.stdout: "dn": 0, 2024-03-28T22:21:09.537 INFO:teuthology.orchestra.run.smithi063.stdout: "dn+": 0, 2024-03-28T22:21:09.537 INFO:teuthology.orchestra.run.smithi063.stdout: "dn-": 0, 2024-03-28T22:21:09.538 INFO:teuthology.orchestra.run.smithi063.stdout: "cap": 0, 2024-03-28T22:21:09.538 INFO:teuthology.orchestra.run.smithi063.stdout: "cap+": 0, 2024-03-28T22:21:09.538 INFO:teuthology.orchestra.run.smithi063.stdout: "cap-": 0, 2024-03-28T22:21:09.538 INFO:teuthology.orchestra.run.smithi063.stdout: "rss": 48528, 2024-03-28T22:21:09.538 INFO:teuthology.orchestra.run.smithi063.stdout: "heap": 215420 2024-03-28T22:21:09.538 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.538 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_server": { 2024-03-28T22:21:09.538 INFO:teuthology.orchestra.run.smithi063.stdout: "dispatch_client_request": 0, 2024-03-28T22:21:09.539 INFO:teuthology.orchestra.run.smithi063.stdout: "dispatch_server_request": 0, 2024-03-28T22:21:09.539 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_request": 0, 2024-03-28T22:21:09.539 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_session": 0, 2024-03-28T22:21:09.539 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_peer_request": 0, 2024-03-28T22:21:09.539 INFO:teuthology.orchestra.run.smithi063.stdout: "req_create_latency": { 2024-03-28T22:21:09.539 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.539 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.539 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.539 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.540 INFO:teuthology.orchestra.run.smithi063.stdout: "req_getattr_latency": { 2024-03-28T22:21:09.540 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.540 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.540 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.540 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.540 INFO:teuthology.orchestra.run.smithi063.stdout: "req_getfilelock_latency": { 2024-03-28T22:21:09.540 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.540 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.541 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.541 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.541 INFO:teuthology.orchestra.run.smithi063.stdout: "req_link_latency": { 2024-03-28T22:21:09.541 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.541 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.541 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.541 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.541 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookup_latency": { 2024-03-28T22:21:09.542 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.542 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.542 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.542 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.542 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookuphash_latency": { 2024-03-28T22:21:09.542 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.542 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.542 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.543 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.543 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupino_latency": { 2024-03-28T22:21:09.543 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.543 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.543 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.543 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.543 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupname_latency": { 2024-03-28T22:21:09.544 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.544 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.544 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.544 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.544 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupparent_latency": { 2024-03-28T22:21:09.544 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.544 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.544 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.545 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.545 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupsnap_latency": { 2024-03-28T22:21:09.545 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.545 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.545 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.545 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.545 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lssnap_latency": { 2024-03-28T22:21:09.545 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.546 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.546 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.546 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.546 INFO:teuthology.orchestra.run.smithi063.stdout: "req_mkdir_latency": { 2024-03-28T22:21:09.546 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.547 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.547 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.547 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.547 INFO:teuthology.orchestra.run.smithi063.stdout: "req_mknod_latency": { 2024-03-28T22:21:09.547 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.547 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.548 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.548 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.548 INFO:teuthology.orchestra.run.smithi063.stdout: "req_mksnap_latency": { 2024-03-28T22:21:09.548 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.548 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.548 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.548 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.548 INFO:teuthology.orchestra.run.smithi063.stdout: "req_open_latency": { 2024-03-28T22:21:09.549 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.549 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.549 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.549 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.549 INFO:teuthology.orchestra.run.smithi063.stdout: "req_readdir_latency": { 2024-03-28T22:21:09.549 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.549 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.549 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.550 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.550 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rename_latency": { 2024-03-28T22:21:09.550 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.550 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.550 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.550 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.550 INFO:teuthology.orchestra.run.smithi063.stdout: "req_renamesnap_latency": { 2024-03-28T22:21:09.550 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.551 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.551 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.551 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.551 INFO:teuthology.orchestra.run.smithi063.stdout: "req_snapdiff_latency": { 2024-03-28T22:21:09.551 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.551 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.551 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.551 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.552 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rmdir_latency": { 2024-03-28T22:21:09.552 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.552 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.552 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.552 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.552 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rmsnap_latency": { 2024-03-28T22:21:09.552 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.552 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.553 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.553 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.553 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rmxattr_latency": { 2024-03-28T22:21:09.553 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.553 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.553 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.553 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.553 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setattr_latency": { 2024-03-28T22:21:09.554 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.554 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.554 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.554 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.554 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setdirlayout_latency": { 2024-03-28T22:21:09.554 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.554 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.555 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.555 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.555 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setfilelock_latency": { 2024-03-28T22:21:09.555 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.555 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.555 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.555 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.555 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setlayout_latency": { 2024-03-28T22:21:09.555 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.556 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.556 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.556 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.556 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setxattr_latency": { 2024-03-28T22:21:09.556 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.556 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.556 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.556 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.557 INFO:teuthology.orchestra.run.smithi063.stdout: "req_symlink_latency": { 2024-03-28T22:21:09.557 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.557 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.557 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.557 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.557 INFO:teuthology.orchestra.run.smithi063.stdout: "req_unlink_latency": { 2024-03-28T22:21:09.557 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.557 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.558 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.558 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.558 INFO:teuthology.orchestra.run.smithi063.stdout: "cap_revoke_eviction": 0, 2024-03-28T22:21:09.558 INFO:teuthology.orchestra.run.smithi063.stdout: "cap_acquisition_throttle": 0, 2024-03-28T22:21:09.558 INFO:teuthology.orchestra.run.smithi063.stdout: "req_getvxattr_latency": { 2024-03-28T22:21:09.558 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.558 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.558 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.559 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-28T22:21:09.559 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.559 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_sessions": { 2024-03-28T22:21:09.559 INFO:teuthology.orchestra.run.smithi063.stdout: "session_count": 2, 2024-03-28T22:21:09.559 INFO:teuthology.orchestra.run.smithi063.stdout: "session_add": 6, 2024-03-28T22:21:09.559 INFO:teuthology.orchestra.run.smithi063.stdout: "session_remove": 4, 2024-03-28T22:21:09.559 INFO:teuthology.orchestra.run.smithi063.stdout: "sessions_open": 4, 2024-03-28T22:21:09.559 INFO:teuthology.orchestra.run.smithi063.stdout: "sessions_stale": 0, 2024-03-28T22:21:09.560 INFO:teuthology.orchestra.run.smithi063.stdout: "total_load": 0, 2024-03-28T22:21:09.560 INFO:teuthology.orchestra.run.smithi063.stdout: "average_load": 0, 2024-03-28T22:21:09.560 INFO:teuthology.orchestra.run.smithi063.stdout: "avg_session_uptime": 3313, 2024-03-28T22:21:09.560 INFO:teuthology.orchestra.run.smithi063.stdout: "mdthresh_evicted": 0 2024-03-28T22:21:09.560 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.560 INFO:teuthology.orchestra.run.smithi063.stdout: "mempool": { 2024-03-28T22:21:09.560 INFO:teuthology.orchestra.run.smithi063.stdout: "bloom_filter_bytes": 0, 2024-03-28T22:21:09.560 INFO:teuthology.orchestra.run.smithi063.stdout: "bloom_filter_items": 0, 2024-03-28T22:21:09.561 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T22:21:09.561 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_alloc_items": 0, 2024-03-28T22:21:09.561 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T22:21:09.561 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_data_items": 0, 2024-03-28T22:21:09.561 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T22:21:09.561 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T22:21:09.561 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T22:21:09.561 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T22:21:09.562 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T22:21:09.562 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_other_items": 0, 2024-03-28T22:21:09.562 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T22:21:09.562 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T22:21:09.562 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_extent_bytes": 0, 2024-03-28T22:21:09.562 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_extent_items": 0, 2024-03-28T22:21:09.562 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_blob_bytes": 0, 2024-03-28T22:21:09.562 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_blob_items": 0, 2024-03-28T22:21:09.563 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T22:21:09.563 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T22:21:09.563 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T22:21:09.563 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T22:21:09.563 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T22:21:09.563 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_items": 0, 2024-03-28T22:21:09.563 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_txc_bytes": 0, 2024-03-28T22:21:09.563 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_txc_items": 0, 2024-03-28T22:21:09.564 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T22:21:09.564 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T22:21:09.564 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_bytes": 0, 2024-03-28T22:21:09.564 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_items": 0, 2024-03-28T22:21:09.564 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_bytes": 0, 2024-03-28T22:21:09.564 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_items": 0, 2024-03-28T22:21:09.564 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T22:21:09.564 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_reader_items": 0, 2024-03-28T22:21:09.564 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T22:21:09.565 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_writer_items": 0, 2024-03-28T22:21:09.565 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_anon_bytes": 181313, 2024-03-28T22:21:09.565 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_anon_items": 47, 2024-03-28T22:21:09.565 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_meta_bytes": 0, 2024-03-28T22:21:09.565 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_meta_items": 0, 2024-03-28T22:21:09.565 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_bytes": 0, 2024-03-28T22:21:09.565 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_items": 0, 2024-03-28T22:21:09.565 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mapbl_bytes": 0, 2024-03-28T22:21:09.565 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mapbl_items": 0, 2024-03-28T22:21:09.566 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pglog_bytes": 0, 2024-03-28T22:21:09.566 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pglog_items": 0, 2024-03-28T22:21:09.566 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_bytes": 8160, 2024-03-28T22:21:09.566 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_items": 186, 2024-03-28T22:21:09.566 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T22:21:09.566 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_mapping_items": 0, 2024-03-28T22:21:09.566 INFO:teuthology.orchestra.run.smithi063.stdout: "pgmap_bytes": 0, 2024-03-28T22:21:09.566 INFO:teuthology.orchestra.run.smithi063.stdout: "pgmap_items": 0, 2024-03-28T22:21:09.567 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_co_bytes": 8464, 2024-03-28T22:21:09.567 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_co_items": 13, 2024-03-28T22:21:09.567 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_1_bytes": 0, 2024-03-28T22:21:09.567 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_1_items": 0, 2024-03-28T22:21:09.567 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_2_bytes": 0, 2024-03-28T22:21:09.567 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_2_items": 0 2024-03-28T22:21:09.567 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.567 INFO:teuthology.orchestra.run.smithi063.stdout: "objecter": { 2024-03-28T22:21:09.568 INFO:teuthology.orchestra.run.smithi063.stdout: "op_active": 0, 2024-03-28T22:21:09.568 INFO:teuthology.orchestra.run.smithi063.stdout: "op_laggy": 0, 2024-03-28T22:21:09.568 INFO:teuthology.orchestra.run.smithi063.stdout: "op_send": 2194, 2024-03-28T22:21:09.568 INFO:teuthology.orchestra.run.smithi063.stdout: "op_send_bytes": 21, 2024-03-28T22:21:09.568 INFO:teuthology.orchestra.run.smithi063.stdout: "op_resend": 3, 2024-03-28T22:21:09.568 INFO:teuthology.orchestra.run.smithi063.stdout: "op_reply": 2194, 2024-03-28T22:21:09.568 INFO:teuthology.orchestra.run.smithi063.stdout: "op_latency": { 2024-03-28T22:21:09.568 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 2194, 2024-03-28T22:21:09.568 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 1.045990421, 2024-03-28T22:21:09.569 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000476750 2024-03-28T22:21:09.569 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.569 INFO:teuthology.orchestra.run.smithi063.stdout: "op_inflight": 0, 2024-03-28T22:21:09.569 INFO:teuthology.orchestra.run.smithi063.stdout: "oplen_avg": { 2024-03-28T22:21:09.569 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 2194, 2024-03-28T22:21:09.569 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 2195 2024-03-28T22:21:09.569 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.569 INFO:teuthology.orchestra.run.smithi063.stdout: "op": 2194, 2024-03-28T22:21:09.570 INFO:teuthology.orchestra.run.smithi063.stdout: "op_r": 2194, 2024-03-28T22:21:09.570 INFO:teuthology.orchestra.run.smithi063.stdout: "op_w": 0, 2024-03-28T22:21:09.570 INFO:teuthology.orchestra.run.smithi063.stdout: "op_rmw": 0, 2024-03-28T22:21:09.570 INFO:teuthology.orchestra.run.smithi063.stdout: "op_pg": 0, 2024-03-28T22:21:09.570 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_stat": 1425, 2024-03-28T22:21:09.570 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_create": 0, 2024-03-28T22:21:09.570 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_read": 767, 2024-03-28T22:21:09.570 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_write": 0, 2024-03-28T22:21:09.571 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_writefull": 0, 2024-03-28T22:21:09.571 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_writesame": 0, 2024-03-28T22:21:09.571 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_append": 0, 2024-03-28T22:21:09.571 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_zero": 0, 2024-03-28T22:21:09.571 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_truncate": 0, 2024-03-28T22:21:09.571 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_delete": 0, 2024-03-28T22:21:09.571 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_mapext": 0, 2024-03-28T22:21:09.571 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_sparse_read": 0, 2024-03-28T22:21:09.572 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_clonerange": 0, 2024-03-28T22:21:09.572 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_getxattr": 1, 2024-03-28T22:21:09.572 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_setxattr": 0, 2024-03-28T22:21:09.572 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_cmpxattr": 0, 2024-03-28T22:21:09.572 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_rmxattr": 0, 2024-03-28T22:21:09.572 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_resetxattrs": 0, 2024-03-28T22:21:09.572 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_call": 0, 2024-03-28T22:21:09.572 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_watch": 0, 2024-03-28T22:21:09.572 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_notify": 0, 2024-03-28T22:21:09.573 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T22:21:09.573 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_pgls": 0, 2024-03-28T22:21:09.573 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_pgls_filter": 0, 2024-03-28T22:21:09.573 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_other": 0, 2024-03-28T22:21:09.573 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_active": 0, 2024-03-28T22:21:09.573 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_send": 0, 2024-03-28T22:21:09.573 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_resend": 0, 2024-03-28T22:21:09.573 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_ping": 0, 2024-03-28T22:21:09.574 INFO:teuthology.orchestra.run.smithi063.stdout: "poolop_active": 0, 2024-03-28T22:21:09.574 INFO:teuthology.orchestra.run.smithi063.stdout: "poolop_send": 0, 2024-03-28T22:21:09.574 INFO:teuthology.orchestra.run.smithi063.stdout: "poolop_resend": 0, 2024-03-28T22:21:09.574 INFO:teuthology.orchestra.run.smithi063.stdout: "poolstat_active": 0, 2024-03-28T22:21:09.574 INFO:teuthology.orchestra.run.smithi063.stdout: "poolstat_send": 0, 2024-03-28T22:21:09.574 INFO:teuthology.orchestra.run.smithi063.stdout: "poolstat_resend": 0, 2024-03-28T22:21:09.574 INFO:teuthology.orchestra.run.smithi063.stdout: "statfs_active": 0, 2024-03-28T22:21:09.574 INFO:teuthology.orchestra.run.smithi063.stdout: "statfs_send": 0, 2024-03-28T22:21:09.575 INFO:teuthology.orchestra.run.smithi063.stdout: "statfs_resend": 0, 2024-03-28T22:21:09.575 INFO:teuthology.orchestra.run.smithi063.stdout: "command_active": 0, 2024-03-28T22:21:09.575 INFO:teuthology.orchestra.run.smithi063.stdout: "command_send": 0, 2024-03-28T22:21:09.575 INFO:teuthology.orchestra.run.smithi063.stdout: "command_resend": 0, 2024-03-28T22:21:09.575 INFO:teuthology.orchestra.run.smithi063.stdout: "map_epoch": 97, 2024-03-28T22:21:09.575 INFO:teuthology.orchestra.run.smithi063.stdout: "map_full": 0, 2024-03-28T22:21:09.575 INFO:teuthology.orchestra.run.smithi063.stdout: "map_inc": 11, 2024-03-28T22:21:09.575 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_sessions": 4, 2024-03-28T22:21:09.576 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_session_open": 4, 2024-03-28T22:21:09.576 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_session_close": 0, 2024-03-28T22:21:09.576 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_laggy": 0, 2024-03-28T22:21:09.576 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_wr": 0, 2024-03-28T22:21:09.576 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_rd": 2, 2024-03-28T22:21:09.576 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_del": 0 2024-03-28T22:21:09.576 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.576 INFO:teuthology.orchestra.run.smithi063.stdout: "oft": { 2024-03-28T22:21:09.577 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_objs": 0, 2024-03-28T22:21:09.577 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_kv_pairs": 0, 2024-03-28T22:21:09.577 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_updates": 0, 2024-03-28T22:21:09.577 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_removes": 0 2024-03-28T22:21:09.577 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.577 INFO:teuthology.orchestra.run.smithi063.stdout: "osd-slow-ops": { 2024-03-28T22:21:09.577 INFO:teuthology.orchestra.run.smithi063.stdout: "slow_ops_count": 0 2024-03-28T22:21:09.577 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.578 INFO:teuthology.orchestra.run.smithi063.stdout: "purge_queue": { 2024-03-28T22:21:09.578 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing_ops": 0, 2024-03-28T22:21:09.578 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T22:21:09.578 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing": 0, 2024-03-28T22:21:09.578 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing_high_water": 0, 2024-03-28T22:21:09.578 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executed_ops": 0, 2024-03-28T22:21:09.578 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executed": 0, 2024-03-28T22:21:09.578 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_item_in_journal": 0 2024-03-28T22:21:09.579 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.579 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T22:21:09.579 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-03-28T22:21:09.579 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 104857600, 2024-03-28T22:21:09.579 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-03-28T22:21:09.579 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 2440, 2024-03-28T22:21:09.579 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 2011523, 2024-03-28T22:21:09.579 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-03-28T22:21:09.580 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 2440, 2024-03-28T22:21:09.580 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 0, 2024-03-28T22:21:09.580 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 0, 2024-03-28T22:21:09.580 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 2440, 2024-03-28T22:21:09.580 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 2011523, 2024-03-28T22:21:09.580 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-03-28T22:21:09.580 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.580 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.580 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.581 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-28T22:21:09.581 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.581 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-objecter_bytes": { 2024-03-28T22:21:09.581 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-03-28T22:21:09.581 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 104857600, 2024-03-28T22:21:09.581 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-03-28T22:21:09.581 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 0, 2024-03-28T22:21:09.581 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 0, 2024-03-28T22:21:09.582 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-03-28T22:21:09.582 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 0, 2024-03-28T22:21:09.582 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 2194, 2024-03-28T22:21:09.582 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 3420, 2024-03-28T22:21:09.582 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 2194, 2024-03-28T22:21:09.582 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 3420, 2024-03-28T22:21:09.582 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-03-28T22:21:09.582 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.583 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.583 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.583 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-28T22:21:09.583 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.583 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-objecter_ops": { 2024-03-28T22:21:09.583 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-03-28T22:21:09.583 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 1024, 2024-03-28T22:21:09.583 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-03-28T22:21:09.584 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 0, 2024-03-28T22:21:09.584 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 0, 2024-03-28T22:21:09.584 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-03-28T22:21:09.584 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 0, 2024-03-28T22:21:09.584 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 2194, 2024-03-28T22:21:09.584 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 2194, 2024-03-28T22:21:09.584 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 2194, 2024-03-28T22:21:09.584 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 2194, 2024-03-28T22:21:09.585 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-03-28T22:21:09.585 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.585 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.585 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.585 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-28T22:21:09.585 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.585 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-write_buf_throttle": { 2024-03-28T22:21:09.585 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-03-28T22:21:09.586 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 3758096384, 2024-03-28T22:21:09.586 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-03-28T22:21:09.586 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 0, 2024-03-28T22:21:09.586 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 0, 2024-03-28T22:21:09.586 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-03-28T22:21:09.586 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 0, 2024-03-28T22:21:09.586 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 0, 2024-03-28T22:21:09.586 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 0, 2024-03-28T22:21:09.587 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 0, 2024-03-28T22:21:09.587 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 0, 2024-03-28T22:21:09.587 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-03-28T22:21:09.587 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.587 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.587 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.587 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-28T22:21:09.587 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:09.588 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-write_buf_throttle-0x55d034eb26e0": { 2024-03-28T22:21:09.588 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-03-28T22:21:09.588 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 3758096384, 2024-03-28T22:21:09.588 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-03-28T22:21:09.588 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 0, 2024-03-28T22:21:09.588 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 0, 2024-03-28T22:21:09.588 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-03-28T22:21:09.588 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 0, 2024-03-28T22:21:09.588 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 0, 2024-03-28T22:21:09.589 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 0, 2024-03-28T22:21:09.589 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 0, 2024-03-28T22:21:09.589 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 0, 2024-03-28T22:21:09.589 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-03-28T22:21:09.589 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:09.589 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:09.589 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:09.589 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-28T22:21:09.590 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-28T22:21:09.590 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-28T22:21:09.930 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:09 smithi063 ceph-mon[36249]: pgmap v328: 97 pgs: 97 active+clean; 91 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-28T22:21:09.934 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:09 smithi016 ceph-mon[32374]: pgmap v328: 97 pgs: 97 active+clean; 91 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-28T22:21:10.046 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:09 smithi053 ceph-mon[35979]: pgmap v328: 97 pgs: 97 active+clean; 91 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-28T22:21:10.243 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-03-28T22:21:10.243 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-03-28T22:21:10.243 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-03-28T22:21:10.243 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=0 2024-03-28T22:21:10.243 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=0 2024-03-28T22:21:10.243 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-03-28T22:21:10.244 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T22:21:10.244 INFO:tasks.check_counter:Daemon mds.c mds.dir_split=0 2024-03-28T22:21:10.244 DEBUG:tasks.check_counter:Getting stats from f 2024-03-28T22:21:10.244 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-28T22:21:10.507 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T22:21:11.894 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:11 smithi063 ceph-mon[36249]: pgmap v329: 97 pgs: 97 active+clean; 91 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-28T22:21:11.894 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:11 smithi063 ceph-mon[36249]: osdmap e98: 12 total, 12 up, 12 in 2024-03-28T22:21:11.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:11 smithi016 ceph-mon[32374]: pgmap v329: 97 pgs: 97 active+clean; 91 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-28T22:21:11.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:11 smithi016 ceph-mon[32374]: osdmap e98: 12 total, 12 up, 12 in 2024-03-28T22:21:12.046 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:11 smithi053 ceph-mon[35979]: pgmap v329: 97 pgs: 97 active+clean; 91 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-28T22:21:12.046 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:11 smithi053 ceph-mon[35979]: osdmap e98: 12 total, 12 up, 12 in 2024-03-28T22:21:12.107 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-28T22:21:12.108 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T22:21:12.108 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 285, 2024-03-28T22:21:12.108 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 285, 2024-03-28T22:21:12.108 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 16857663, 2024-03-28T22:21:12.108 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 89870, 2024-03-28T22:21:12.108 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 7, 2024-03-28T22:21:12.108 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 3, 2024-03-28T22:21:12.108 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.038162254, 2024-03-28T22:21:12.109 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.007619498, 2024-03-28T22:21:12.109 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.019650540, 2024-03-28T22:21:12.109 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.007444394, 2024-03-28T22:21:12.109 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:21:12.109 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 284, 2024-03-28T22:21:12.109 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.012668661, 2024-03-28T22:21:12.109 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000044607 2024-03-28T22:21:12.109 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.109 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:21:12.110 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.110 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.110 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.110 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.110 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-28T22:21:12.110 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 448 2024-03-28T22:21:12.110 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.110 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T22:21:12.111 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 910, 2024-03-28T22:21:12.111 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 892, 2024-03-28T22:21:12.111 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 31150637, 2024-03-28T22:21:12.111 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 343445, 2024-03-28T22:21:12.111 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 5, 2024-03-28T22:21:12.111 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 4, 2024-03-28T22:21:12.111 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.130904081, 2024-03-28T22:21:12.111 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.037255753, 2024-03-28T22:21:12.112 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.069714174, 2024-03-28T22:21:12.112 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.027611600, 2024-03-28T22:21:12.112 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:21:12.112 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 891, 2024-03-28T22:21:12.112 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.038573807, 2024-03-28T22:21:12.112 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000043292 2024-03-28T22:21:12.112 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.112 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:21:12.113 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.113 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.113 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.113 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.113 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 1620544, 2024-03-28T22:21:12.113 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 132288 2024-03-28T22:21:12.113 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.113 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T22:21:12.113 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 1415, 2024-03-28T22:21:12.114 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 1574, 2024-03-28T22:21:12.114 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 19580512, 2024-03-28T22:21:12.114 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 894198, 2024-03-28T22:21:12.114 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 8, 2024-03-28T22:21:12.114 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 6, 2024-03-28T22:21:12.114 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.170160972, 2024-03-28T22:21:12.114 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.054267990, 2024-03-28T22:21:12.114 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.051927907, 2024-03-28T22:21:12.115 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.046989721, 2024-03-28T22:21:12.115 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:21:12.115 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 1571, 2024-03-28T22:21:12.115 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.071797634, 2024-03-28T22:21:12.115 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000045701 2024-03-28T22:21:12.115 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.115 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:21:12.115 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.115 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.116 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.116 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.116 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-28T22:21:12.116 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 444864 2024-03-28T22:21:12.116 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.116 INFO:teuthology.orchestra.run.smithi063.stdout: "cct": { 2024-03-28T22:21:12.116 INFO:teuthology.orchestra.run.smithi063.stdout: "total_workers": 1, 2024-03-28T22:21:12.116 INFO:teuthology.orchestra.run.smithi063.stdout: "unhealthy_workers": 0 2024-03-28T22:21:12.117 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.117 INFO:teuthology.orchestra.run.smithi063.stdout: "finisher-MDSRank": { 2024-03-28T22:21:12.117 INFO:teuthology.orchestra.run.smithi063.stdout: "queue_len": 0, 2024-03-28T22:21:12.117 INFO:teuthology.orchestra.run.smithi063.stdout: "complete_latency": { 2024-03-28T22:21:12.117 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 3845, 2024-03-28T22:21:12.117 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.116732995, 2024-03-28T22:21:12.117 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000030359 2024-03-28T22:21:12.117 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-28T22:21:12.117 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.118 INFO:teuthology.orchestra.run.smithi063.stdout: "finisher-PurgeQueue": { 2024-03-28T22:21:12.118 INFO:teuthology.orchestra.run.smithi063.stdout: "queue_len": 0, 2024-03-28T22:21:12.118 INFO:teuthology.orchestra.run.smithi063.stdout: "complete_latency": { 2024-03-28T22:21:12.118 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.118 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.118 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.118 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-28T22:21:12.118 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.119 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-28T22:21:12.119 INFO:teuthology.orchestra.run.smithi063.stdout: "request": 0, 2024-03-28T22:21:12.119 INFO:teuthology.orchestra.run.smithi063.stdout: "reply": 0, 2024-03-28T22:21:12.119 INFO:teuthology.orchestra.run.smithi063.stdout: "reply_latency": { 2024-03-28T22:21:12.119 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.119 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.119 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.119 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.120 INFO:teuthology.orchestra.run.smithi063.stdout: "slow_reply": 0, 2024-03-28T22:21:12.120 INFO:teuthology.orchestra.run.smithi063.stdout: "forward": 0, 2024-03-28T22:21:12.120 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_fetch_complete": 0, 2024-03-28T22:21:12.120 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_fetch_keys": 0, 2024-03-28T22:21:12.120 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_commit": 0, 2024-03-28T22:21:12.120 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_split": 0, 2024-03-28T22:21:12.120 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_merge": 0, 2024-03-28T22:21:12.120 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes": 11070, 2024-03-28T22:21:12.120 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_top": 195, 2024-03-28T22:21:12.121 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_bottom": 10875, 2024-03-28T22:21:12.121 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_pin_tail": 0, 2024-03-28T22:21:12.121 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_pinned": 10792, 2024-03-28T22:21:12.121 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_expired": 0, 2024-03-28T22:21:12.121 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_with_caps": 0, 2024-03-28T22:21:12.121 INFO:teuthology.orchestra.run.smithi063.stdout: "caps": 0, 2024-03-28T22:21:12.121 INFO:teuthology.orchestra.run.smithi063.stdout: "subtrees": 5, 2024-03-28T22:21:12.121 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse": 0, 2024-03-28T22:21:12.122 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_hit": 0, 2024-03-28T22:21:12.122 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_forward": 0, 2024-03-28T22:21:12.122 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_discover": 0, 2024-03-28T22:21:12.122 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_dir_fetch": 0, 2024-03-28T22:21:12.122 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_remote_ino": 0, 2024-03-28T22:21:12.122 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_lock": 0, 2024-03-28T22:21:12.122 INFO:teuthology.orchestra.run.smithi063.stdout: "load_cent": 0, 2024-03-28T22:21:12.122 INFO:teuthology.orchestra.run.smithi063.stdout: "q": 0, 2024-03-28T22:21:12.122 INFO:teuthology.orchestra.run.smithi063.stdout: "exported": 0, 2024-03-28T22:21:12.123 INFO:teuthology.orchestra.run.smithi063.stdout: "exported_inodes": 0, 2024-03-28T22:21:12.123 INFO:teuthology.orchestra.run.smithi063.stdout: "imported": 0, 2024-03-28T22:21:12.123 INFO:teuthology.orchestra.run.smithi063.stdout: "imported_inodes": 0, 2024-03-28T22:21:12.123 INFO:teuthology.orchestra.run.smithi063.stdout: "openino_dir_fetch": 0, 2024-03-28T22:21:12.123 INFO:teuthology.orchestra.run.smithi063.stdout: "openino_backtrace_fetch": 0, 2024-03-28T22:21:12.123 INFO:teuthology.orchestra.run.smithi063.stdout: "openino_peer_discover": 0, 2024-03-28T22:21:12.123 INFO:teuthology.orchestra.run.smithi063.stdout: "root_rfiles": 0, 2024-03-28T22:21:12.123 INFO:teuthology.orchestra.run.smithi063.stdout: "root_rbytes": 0, 2024-03-28T22:21:12.124 INFO:teuthology.orchestra.run.smithi063.stdout: "root_rsnaps": 0, 2024-03-28T22:21:12.124 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T22:21:12.124 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_set_tag": 0, 2024-03-28T22:21:12.124 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T22:21:12.124 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_inotable_repaired": 0, 2024-03-28T22:21:12.124 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_dir_inodes": 0, 2024-03-28T22:21:12.124 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T22:21:12.124 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T22:21:12.125 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_file_inodes": 0, 2024-03-28T22:21:12.125 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_inode_file_caps": 0, 2024-03-28T22:21:12.125 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T22:21:12.125 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_grant": 0, 2024-03-28T22:21:12.125 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T22:21:12.125 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T22:21:12.125 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T22:21:12.125 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_caps": 0, 2024-03-28T22:21:12.125 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_caps_dirty": 0, 2024-03-28T22:21:12.126 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_cap_release": 0, 2024-03-28T22:21:12.126 INFO:teuthology.orchestra.run.smithi063.stdout: "process_request_cap_release": 0 2024-03-28T22:21:12.126 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.126 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_cache": { 2024-03-28T22:21:12.126 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_update": 0, 2024-03-28T22:21:12.126 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_update_receipt": 0, 2024-03-28T22:21:12.126 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_try_discover": 0, 2024-03-28T22:21:12.126 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_send_discover": 0, 2024-03-28T22:21:12.126 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_handle_discover": 0, 2024-03-28T22:21:12.127 INFO:teuthology.orchestra.run.smithi063.stdout: "num_strays": 0, 2024-03-28T22:21:12.127 INFO:teuthology.orchestra.run.smithi063.stdout: "num_strays_delayed": 0, 2024-03-28T22:21:12.127 INFO:teuthology.orchestra.run.smithi063.stdout: "num_strays_enqueuing": 0, 2024-03-28T22:21:12.127 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_created": 0, 2024-03-28T22:21:12.127 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_enqueued": 0, 2024-03-28T22:21:12.127 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_reintegrated": 0, 2024-03-28T22:21:12.127 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_migrated": 0, 2024-03-28T22:21:12.127 INFO:teuthology.orchestra.run.smithi063.stdout: "num_recovering_processing": 0, 2024-03-28T22:21:12.127 INFO:teuthology.orchestra.run.smithi063.stdout: "num_recovering_enqueued": 0, 2024-03-28T22:21:12.128 INFO:teuthology.orchestra.run.smithi063.stdout: "num_recovering_prioritized": 0, 2024-03-28T22:21:12.128 INFO:teuthology.orchestra.run.smithi063.stdout: "recovery_started": 0, 2024-03-28T22:21:12.128 INFO:teuthology.orchestra.run.smithi063.stdout: "recovery_completed": 0, 2024-03-28T22:21:12.128 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_quiesce_path": 0, 2024-03-28T22:21:12.128 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_quiesce_inode": 0, 2024-03-28T22:21:12.128 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T22:21:12.128 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_exportdir": 0, 2024-03-28T22:21:12.128 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_flush": 0, 2024-03-28T22:21:12.128 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_fragmentdir": 0, 2024-03-28T22:21:12.129 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_fragstats": 0, 2024-03-28T22:21:12.129 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_inodestats": 0 2024-03-28T22:21:12.129 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.129 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_log": { 2024-03-28T22:21:12.129 INFO:teuthology.orchestra.run.smithi063.stdout: "evlrg": 0, 2024-03-28T22:21:12.129 INFO:teuthology.orchestra.run.smithi063.stdout: "evadd": 0, 2024-03-28T22:21:12.129 INFO:teuthology.orchestra.run.smithi063.stdout: "evex": 0, 2024-03-28T22:21:12.129 INFO:teuthology.orchestra.run.smithi063.stdout: "evtrm": 0, 2024-03-28T22:21:12.130 INFO:teuthology.orchestra.run.smithi063.stdout: "ev": 25740, 2024-03-28T22:21:12.130 INFO:teuthology.orchestra.run.smithi063.stdout: "evexg": 0, 2024-03-28T22:21:12.130 INFO:teuthology.orchestra.run.smithi063.stdout: "evexd": 0, 2024-03-28T22:21:12.130 INFO:teuthology.orchestra.run.smithi063.stdout: "segadd": 0, 2024-03-28T22:21:12.130 INFO:teuthology.orchestra.run.smithi063.stdout: "segex": 0, 2024-03-28T22:21:12.130 INFO:teuthology.orchestra.run.smithi063.stdout: "segtrm": 0, 2024-03-28T22:21:12.130 INFO:teuthology.orchestra.run.smithi063.stdout: "seg": 35, 2024-03-28T22:21:12.130 INFO:teuthology.orchestra.run.smithi063.stdout: "segmjr": 4, 2024-03-28T22:21:12.130 INFO:teuthology.orchestra.run.smithi063.stdout: "segexg": 0, 2024-03-28T22:21:12.131 INFO:teuthology.orchestra.run.smithi063.stdout: "segexd": 0, 2024-03-28T22:21:12.131 INFO:teuthology.orchestra.run.smithi063.stdout: "expos": 4194304, 2024-03-28T22:21:12.131 INFO:teuthology.orchestra.run.smithi063.stdout: "wrpos": 69476710, 2024-03-28T22:21:12.131 INFO:teuthology.orchestra.run.smithi063.stdout: "rdpos": 69476136, 2024-03-28T22:21:12.131 INFO:teuthology.orchestra.run.smithi063.stdout: "jlat": { 2024-03-28T22:21:12.131 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.131 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.131 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.131 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.132 INFO:teuthology.orchestra.run.smithi063.stdout: "replayed": 25740 2024-03-28T22:21:12.132 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.132 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_mem": { 2024-03-28T22:21:12.132 INFO:teuthology.orchestra.run.smithi063.stdout: "ino": 6423, 2024-03-28T22:21:12.132 INFO:teuthology.orchestra.run.smithi063.stdout: "ino+": 9650, 2024-03-28T22:21:12.132 INFO:teuthology.orchestra.run.smithi063.stdout: "ino-": 3227, 2024-03-28T22:21:12.132 INFO:teuthology.orchestra.run.smithi063.stdout: "dir": 305, 2024-03-28T22:21:12.132 INFO:teuthology.orchestra.run.smithi063.stdout: "dir+": 341, 2024-03-28T22:21:12.133 INFO:teuthology.orchestra.run.smithi063.stdout: "dir-": 36, 2024-03-28T22:21:12.133 INFO:teuthology.orchestra.run.smithi063.stdout: "dn": 11070, 2024-03-28T22:21:12.133 INFO:teuthology.orchestra.run.smithi063.stdout: "dn+": 11118, 2024-03-28T22:21:12.133 INFO:teuthology.orchestra.run.smithi063.stdout: "dn-": 48, 2024-03-28T22:21:12.133 INFO:teuthology.orchestra.run.smithi063.stdout: "cap": 0, 2024-03-28T22:21:12.133 INFO:teuthology.orchestra.run.smithi063.stdout: "cap+": 0, 2024-03-28T22:21:12.133 INFO:teuthology.orchestra.run.smithi063.stdout: "cap-": 0, 2024-03-28T22:21:12.133 INFO:teuthology.orchestra.run.smithi063.stdout: "rss": 82556, 2024-03-28T22:21:12.134 INFO:teuthology.orchestra.run.smithi063.stdout: "heap": 215420 2024-03-28T22:21:12.134 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.134 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_server": { 2024-03-28T22:21:12.134 INFO:teuthology.orchestra.run.smithi063.stdout: "dispatch_client_request": 0, 2024-03-28T22:21:12.134 INFO:teuthology.orchestra.run.smithi063.stdout: "dispatch_server_request": 0, 2024-03-28T22:21:12.134 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_request": 0, 2024-03-28T22:21:12.134 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_session": 0, 2024-03-28T22:21:12.134 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_peer_request": 0, 2024-03-28T22:21:12.134 INFO:teuthology.orchestra.run.smithi063.stdout: "req_create_latency": { 2024-03-28T22:21:12.135 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.135 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.135 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.135 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.135 INFO:teuthology.orchestra.run.smithi063.stdout: "req_getattr_latency": { 2024-03-28T22:21:12.135 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.135 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.135 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.136 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.136 INFO:teuthology.orchestra.run.smithi063.stdout: "req_getfilelock_latency": { 2024-03-28T22:21:12.136 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.136 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.136 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.136 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.136 INFO:teuthology.orchestra.run.smithi063.stdout: "req_link_latency": { 2024-03-28T22:21:12.136 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.136 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.137 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.137 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.137 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookup_latency": { 2024-03-28T22:21:12.137 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.137 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.137 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.137 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.137 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookuphash_latency": { 2024-03-28T22:21:12.138 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.138 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.138 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.138 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.138 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupino_latency": { 2024-03-28T22:21:12.138 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.138 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.138 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.139 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.139 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupname_latency": { 2024-03-28T22:21:12.139 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.139 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.139 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.139 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.139 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupparent_latency": { 2024-03-28T22:21:12.139 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.140 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.140 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.140 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.140 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupsnap_latency": { 2024-03-28T22:21:12.140 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.140 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.140 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.140 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.140 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lssnap_latency": { 2024-03-28T22:21:12.141 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.141 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.141 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.141 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.142 INFO:teuthology.orchestra.run.smithi063.stdout: "req_mkdir_latency": { 2024-03-28T22:21:12.142 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.142 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.142 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.142 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.142 INFO:teuthology.orchestra.run.smithi063.stdout: "req_mknod_latency": { 2024-03-28T22:21:12.142 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.143 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.143 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.143 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.143 INFO:teuthology.orchestra.run.smithi063.stdout: "req_mksnap_latency": { 2024-03-28T22:21:12.143 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.143 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.143 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.143 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.143 INFO:teuthology.orchestra.run.smithi063.stdout: "req_open_latency": { 2024-03-28T22:21:12.144 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.144 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.144 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.144 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.144 INFO:teuthology.orchestra.run.smithi063.stdout: "req_readdir_latency": { 2024-03-28T22:21:12.144 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.144 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.144 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.144 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.145 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rename_latency": { 2024-03-28T22:21:12.145 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.145 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.145 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.145 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.145 INFO:teuthology.orchestra.run.smithi063.stdout: "req_renamesnap_latency": { 2024-03-28T22:21:12.145 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.145 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.145 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.146 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.146 INFO:teuthology.orchestra.run.smithi063.stdout: "req_snapdiff_latency": { 2024-03-28T22:21:12.146 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.146 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.146 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.146 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.146 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rmdir_latency": { 2024-03-28T22:21:12.146 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.147 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.147 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.147 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.147 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rmsnap_latency": { 2024-03-28T22:21:12.147 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.147 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.147 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.147 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.147 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rmxattr_latency": { 2024-03-28T22:21:12.148 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.148 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.148 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.148 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.148 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setattr_latency": { 2024-03-28T22:21:12.148 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.148 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.148 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.148 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.149 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setdirlayout_latency": { 2024-03-28T22:21:12.149 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.149 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.149 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.149 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.149 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setfilelock_latency": { 2024-03-28T22:21:12.149 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.149 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.149 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.150 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.150 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setlayout_latency": { 2024-03-28T22:21:12.150 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.150 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.150 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.150 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.150 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setxattr_latency": { 2024-03-28T22:21:12.150 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.150 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.151 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.151 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.151 INFO:teuthology.orchestra.run.smithi063.stdout: "req_symlink_latency": { 2024-03-28T22:21:12.151 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.151 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.151 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.151 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.151 INFO:teuthology.orchestra.run.smithi063.stdout: "req_unlink_latency": { 2024-03-28T22:21:12.152 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.152 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.152 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.152 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.152 INFO:teuthology.orchestra.run.smithi063.stdout: "cap_revoke_eviction": 0, 2024-03-28T22:21:12.152 INFO:teuthology.orchestra.run.smithi063.stdout: "cap_acquisition_throttle": 0, 2024-03-28T22:21:12.152 INFO:teuthology.orchestra.run.smithi063.stdout: "req_getvxattr_latency": { 2024-03-28T22:21:12.152 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.153 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.153 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.153 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-28T22:21:12.153 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.153 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_sessions": { 2024-03-28T22:21:12.153 INFO:teuthology.orchestra.run.smithi063.stdout: "session_count": 2, 2024-03-28T22:21:12.153 INFO:teuthology.orchestra.run.smithi063.stdout: "session_add": 5, 2024-03-28T22:21:12.153 INFO:teuthology.orchestra.run.smithi063.stdout: "session_remove": 3, 2024-03-28T22:21:12.153 INFO:teuthology.orchestra.run.smithi063.stdout: "sessions_open": 3, 2024-03-28T22:21:12.154 INFO:teuthology.orchestra.run.smithi063.stdout: "sessions_stale": 0, 2024-03-28T22:21:12.154 INFO:teuthology.orchestra.run.smithi063.stdout: "total_load": 0, 2024-03-28T22:21:12.154 INFO:teuthology.orchestra.run.smithi063.stdout: "average_load": 0, 2024-03-28T22:21:12.154 INFO:teuthology.orchestra.run.smithi063.stdout: "avg_session_uptime": 2645, 2024-03-28T22:21:12.154 INFO:teuthology.orchestra.run.smithi063.stdout: "mdthresh_evicted": 0 2024-03-28T22:21:12.154 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.154 INFO:teuthology.orchestra.run.smithi063.stdout: "mempool": { 2024-03-28T22:21:12.154 INFO:teuthology.orchestra.run.smithi063.stdout: "bloom_filter_bytes": 0, 2024-03-28T22:21:12.155 INFO:teuthology.orchestra.run.smithi063.stdout: "bloom_filter_items": 0, 2024-03-28T22:21:12.155 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T22:21:12.155 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_alloc_items": 0, 2024-03-28T22:21:12.155 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T22:21:12.155 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_data_items": 0, 2024-03-28T22:21:12.155 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T22:21:12.155 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T22:21:12.155 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T22:21:12.156 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T22:21:12.156 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T22:21:12.156 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_other_items": 0, 2024-03-28T22:21:12.156 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T22:21:12.156 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T22:21:12.156 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_extent_bytes": 0, 2024-03-28T22:21:12.156 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_extent_items": 0, 2024-03-28T22:21:12.156 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_blob_bytes": 0, 2024-03-28T22:21:12.156 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_blob_items": 0, 2024-03-28T22:21:12.157 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T22:21:12.157 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T22:21:12.157 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T22:21:12.157 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T22:21:12.157 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T22:21:12.157 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_items": 0, 2024-03-28T22:21:12.157 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_txc_bytes": 0, 2024-03-28T22:21:12.157 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_txc_items": 0, 2024-03-28T22:21:12.157 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T22:21:12.158 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T22:21:12.158 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_bytes": 0, 2024-03-28T22:21:12.158 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_items": 0, 2024-03-28T22:21:12.158 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_bytes": 0, 2024-03-28T22:21:12.158 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_items": 0, 2024-03-28T22:21:12.158 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T22:21:12.158 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_reader_items": 0, 2024-03-28T22:21:12.158 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T22:21:12.158 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_writer_items": 0, 2024-03-28T22:21:12.159 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_anon_bytes": 462072, 2024-03-28T22:21:12.159 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_anon_items": 6544, 2024-03-28T22:21:12.159 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_meta_bytes": 0, 2024-03-28T22:21:12.159 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_meta_items": 0, 2024-03-28T22:21:12.159 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_bytes": 0, 2024-03-28T22:21:12.159 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_items": 0, 2024-03-28T22:21:12.159 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mapbl_bytes": 0, 2024-03-28T22:21:12.159 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mapbl_items": 0, 2024-03-28T22:21:12.159 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pglog_bytes": 0, 2024-03-28T22:21:12.160 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pglog_items": 0, 2024-03-28T22:21:12.160 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_bytes": 8160, 2024-03-28T22:21:12.160 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_items": 186, 2024-03-28T22:21:12.160 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T22:21:12.160 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_mapping_items": 0, 2024-03-28T22:21:12.160 INFO:teuthology.orchestra.run.smithi063.stdout: "pgmap_bytes": 0, 2024-03-28T22:21:12.160 INFO:teuthology.orchestra.run.smithi063.stdout: "pgmap_items": 0, 2024-03-28T22:21:12.160 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_co_bytes": 22839920, 2024-03-28T22:21:12.160 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_co_items": 198871, 2024-03-28T22:21:12.161 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_1_bytes": 0, 2024-03-28T22:21:12.161 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_1_items": 0, 2024-03-28T22:21:12.161 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_2_bytes": 0, 2024-03-28T22:21:12.161 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_2_items": 0 2024-03-28T22:21:12.161 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.161 INFO:teuthology.orchestra.run.smithi063.stdout: "objecter": { 2024-03-28T22:21:12.161 INFO:teuthology.orchestra.run.smithi063.stdout: "op_active": 0, 2024-03-28T22:21:12.161 INFO:teuthology.orchestra.run.smithi063.stdout: "op_laggy": 0, 2024-03-28T22:21:12.162 INFO:teuthology.orchestra.run.smithi063.stdout: "op_send": 2358, 2024-03-28T22:21:12.162 INFO:teuthology.orchestra.run.smithi063.stdout: "op_send_bytes": 21, 2024-03-28T22:21:12.162 INFO:teuthology.orchestra.run.smithi063.stdout: "op_resend": 3, 2024-03-28T22:21:12.162 INFO:teuthology.orchestra.run.smithi063.stdout: "op_reply": 2358, 2024-03-28T22:21:12.162 INFO:teuthology.orchestra.run.smithi063.stdout: "op_latency": { 2024-03-28T22:21:12.162 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 2358, 2024-03-28T22:21:12.162 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 1.276988321, 2024-03-28T22:21:12.162 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000541555 2024-03-28T22:21:12.162 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.163 INFO:teuthology.orchestra.run.smithi063.stdout: "op_inflight": 0, 2024-03-28T22:21:12.163 INFO:teuthology.orchestra.run.smithi063.stdout: "oplen_avg": { 2024-03-28T22:21:12.163 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 2358, 2024-03-28T22:21:12.163 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 2359 2024-03-28T22:21:12.163 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.163 INFO:teuthology.orchestra.run.smithi063.stdout: "op": 2358, 2024-03-28T22:21:12.163 INFO:teuthology.orchestra.run.smithi063.stdout: "op_r": 2358, 2024-03-28T22:21:12.163 INFO:teuthology.orchestra.run.smithi063.stdout: "op_w": 0, 2024-03-28T22:21:12.163 INFO:teuthology.orchestra.run.smithi063.stdout: "op_rmw": 0, 2024-03-28T22:21:12.164 INFO:teuthology.orchestra.run.smithi063.stdout: "op_pg": 0, 2024-03-28T22:21:12.164 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_stat": 1509, 2024-03-28T22:21:12.164 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_create": 0, 2024-03-28T22:21:12.164 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_read": 847, 2024-03-28T22:21:12.164 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_write": 0, 2024-03-28T22:21:12.164 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_writefull": 0, 2024-03-28T22:21:12.164 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_writesame": 0, 2024-03-28T22:21:12.164 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_append": 0, 2024-03-28T22:21:12.164 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_zero": 0, 2024-03-28T22:21:12.165 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_truncate": 0, 2024-03-28T22:21:12.165 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_delete": 0, 2024-03-28T22:21:12.165 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_mapext": 0, 2024-03-28T22:21:12.165 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_sparse_read": 0, 2024-03-28T22:21:12.165 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_clonerange": 0, 2024-03-28T22:21:12.165 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_getxattr": 1, 2024-03-28T22:21:12.165 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_setxattr": 0, 2024-03-28T22:21:12.165 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_cmpxattr": 0, 2024-03-28T22:21:12.165 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_rmxattr": 0, 2024-03-28T22:21:12.166 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_resetxattrs": 0, 2024-03-28T22:21:12.166 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_call": 0, 2024-03-28T22:21:12.166 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_watch": 0, 2024-03-28T22:21:12.166 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_notify": 0, 2024-03-28T22:21:12.166 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T22:21:12.166 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_pgls": 0, 2024-03-28T22:21:12.166 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_pgls_filter": 0, 2024-03-28T22:21:12.166 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_other": 0, 2024-03-28T22:21:12.167 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_active": 0, 2024-03-28T22:21:12.167 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_send": 0, 2024-03-28T22:21:12.167 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_resend": 0, 2024-03-28T22:21:12.167 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_ping": 0, 2024-03-28T22:21:12.167 INFO:teuthology.orchestra.run.smithi063.stdout: "poolop_active": 0, 2024-03-28T22:21:12.167 INFO:teuthology.orchestra.run.smithi063.stdout: "poolop_send": 0, 2024-03-28T22:21:12.167 INFO:teuthology.orchestra.run.smithi063.stdout: "poolop_resend": 0, 2024-03-28T22:21:12.167 INFO:teuthology.orchestra.run.smithi063.stdout: "poolstat_active": 0, 2024-03-28T22:21:12.167 INFO:teuthology.orchestra.run.smithi063.stdout: "poolstat_send": 0, 2024-03-28T22:21:12.167 INFO:teuthology.orchestra.run.smithi063.stdout: "poolstat_resend": 0, 2024-03-28T22:21:12.168 INFO:teuthology.orchestra.run.smithi063.stdout: "statfs_active": 0, 2024-03-28T22:21:12.168 INFO:teuthology.orchestra.run.smithi063.stdout: "statfs_send": 0, 2024-03-28T22:21:12.168 INFO:teuthology.orchestra.run.smithi063.stdout: "statfs_resend": 0, 2024-03-28T22:21:12.168 INFO:teuthology.orchestra.run.smithi063.stdout: "command_active": 0, 2024-03-28T22:21:12.168 INFO:teuthology.orchestra.run.smithi063.stdout: "command_send": 0, 2024-03-28T22:21:12.168 INFO:teuthology.orchestra.run.smithi063.stdout: "command_resend": 0, 2024-03-28T22:21:12.168 INFO:teuthology.orchestra.run.smithi063.stdout: "map_epoch": 98, 2024-03-28T22:21:12.168 INFO:teuthology.orchestra.run.smithi063.stdout: "map_full": 0, 2024-03-28T22:21:12.168 INFO:teuthology.orchestra.run.smithi063.stdout: "map_inc": 12, 2024-03-28T22:21:12.169 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_sessions": 9, 2024-03-28T22:21:12.169 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_session_open": 9, 2024-03-28T22:21:12.169 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_session_close": 0, 2024-03-28T22:21:12.169 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_laggy": 0, 2024-03-28T22:21:12.169 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_wr": 0, 2024-03-28T22:21:12.169 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_rd": 2, 2024-03-28T22:21:12.169 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_del": 0 2024-03-28T22:21:12.169 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.170 INFO:teuthology.orchestra.run.smithi063.stdout: "oft": { 2024-03-28T22:21:12.170 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_objs": 0, 2024-03-28T22:21:12.170 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_kv_pairs": 0, 2024-03-28T22:21:12.170 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_updates": 0, 2024-03-28T22:21:12.170 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_removes": 0 2024-03-28T22:21:12.170 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.170 INFO:teuthology.orchestra.run.smithi063.stdout: "osd-slow-ops": { 2024-03-28T22:21:12.170 INFO:teuthology.orchestra.run.smithi063.stdout: "slow_ops_count": 0 2024-03-28T22:21:12.171 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.171 INFO:teuthology.orchestra.run.smithi063.stdout: "purge_queue": { 2024-03-28T22:21:12.171 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing_ops": 0, 2024-03-28T22:21:12.171 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T22:21:12.171 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing": 0, 2024-03-28T22:21:12.171 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing_high_water": 0, 2024-03-28T22:21:12.171 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executed_ops": 0, 2024-03-28T22:21:12.171 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executed": 0, 2024-03-28T22:21:12.171 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_item_in_journal": 0 2024-03-28T22:21:12.172 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.172 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T22:21:12.172 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-03-28T22:21:12.172 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 104857600, 2024-03-28T22:21:12.172 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-03-28T22:21:12.172 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 2606, 2024-03-28T22:21:12.172 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 67338614, 2024-03-28T22:21:12.172 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-03-28T22:21:12.172 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 2606, 2024-03-28T22:21:12.173 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 0, 2024-03-28T22:21:12.173 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 0, 2024-03-28T22:21:12.173 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 2606, 2024-03-28T22:21:12.173 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 67338614, 2024-03-28T22:21:12.173 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-03-28T22:21:12.173 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.173 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.173 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.173 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-28T22:21:12.173 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.174 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-objecter_bytes": { 2024-03-28T22:21:12.174 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-03-28T22:21:12.174 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 104857600, 2024-03-28T22:21:12.174 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-03-28T22:21:12.174 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 0, 2024-03-28T22:21:12.174 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 0, 2024-03-28T22:21:12.174 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-03-28T22:21:12.174 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 0, 2024-03-28T22:21:12.175 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 2358, 2024-03-28T22:21:12.175 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 65301972, 2024-03-28T22:21:12.175 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 2358, 2024-03-28T22:21:12.175 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 65301972, 2024-03-28T22:21:12.175 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-03-28T22:21:12.175 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.175 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.175 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.175 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-28T22:21:12.176 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.176 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-objecter_ops": { 2024-03-28T22:21:12.176 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-03-28T22:21:12.176 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 1024, 2024-03-28T22:21:12.176 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-03-28T22:21:12.176 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 0, 2024-03-28T22:21:12.176 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 0, 2024-03-28T22:21:12.176 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-03-28T22:21:12.176 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 0, 2024-03-28T22:21:12.176 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 2358, 2024-03-28T22:21:12.177 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 2358, 2024-03-28T22:21:12.177 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 2358, 2024-03-28T22:21:12.177 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 2358, 2024-03-28T22:21:12.177 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-03-28T22:21:12.177 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.177 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.177 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.177 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-28T22:21:12.178 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.178 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-write_buf_throttle": { 2024-03-28T22:21:12.178 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-03-28T22:21:12.178 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 3758096384, 2024-03-28T22:21:12.178 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-03-28T22:21:12.178 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 0, 2024-03-28T22:21:12.178 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 0, 2024-03-28T22:21:12.178 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-03-28T22:21:12.178 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 0, 2024-03-28T22:21:12.179 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 0, 2024-03-28T22:21:12.179 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 0, 2024-03-28T22:21:12.179 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 0, 2024-03-28T22:21:12.179 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 0, 2024-03-28T22:21:12.179 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-03-28T22:21:12.179 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.179 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.179 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.180 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-28T22:21:12.180 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:12.180 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-write_buf_throttle-0x55f64eb6a6e0": { 2024-03-28T22:21:12.180 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-03-28T22:21:12.180 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 3758096384, 2024-03-28T22:21:12.180 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-03-28T22:21:12.180 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 0, 2024-03-28T22:21:12.180 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 0, 2024-03-28T22:21:12.180 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-03-28T22:21:12.181 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 0, 2024-03-28T22:21:12.181 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 0, 2024-03-28T22:21:12.181 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 0, 2024-03-28T22:21:12.181 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 0, 2024-03-28T22:21:12.181 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 0, 2024-03-28T22:21:12.181 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-03-28T22:21:12.181 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:12.181 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:12.181 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:12.182 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-28T22:21:12.182 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-28T22:21:12.182 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-28T22:21:12.869 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-28T22:21:12.869 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-03-28T22:21:12.869 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-03-28T22:21:12.870 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2024-03-28T22:21:12.870 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2024-03-28T22:21:12.870 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-03-28T22:21:12.870 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T22:21:12.870 INFO:tasks.check_counter:Daemon mds.f mds.dir_split=0 2024-03-28T22:21:12.870 DEBUG:tasks.check_counter:Getting stats from i 2024-03-28T22:21:12.870 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-28T22:21:13.123 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T22:21:13.930 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:13 smithi063 ceph-mon[36249]: pgmap v331: 97 pgs: 97 active+clean; 91 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-28T22:21:13.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:13 smithi016 ceph-mon[32374]: pgmap v331: 97 pgs: 97 active+clean; 91 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-28T22:21:14.046 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:13 smithi053 ceph-mon[35979]: pgmap v331: 97 pgs: 97 active+clean; 91 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-28T22:21:14.784 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-28T22:21:14.785 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T22:21:14.785 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 1597, 2024-03-28T22:21:14.785 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 1599, 2024-03-28T22:21:14.785 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 2682714, 2024-03-28T22:21:14.785 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 507856, 2024-03-28T22:21:14.785 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 6, 2024-03-28T22:21:14.785 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 2, 2024-03-28T22:21:14.786 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.152799872, 2024-03-28T22:21:14.786 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.049496017, 2024-03-28T22:21:14.786 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.040545410, 2024-03-28T22:21:14.786 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.046559722, 2024-03-28T22:21:14.786 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:21:14.786 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 1597, 2024-03-28T22:21:14.786 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.063313830, 2024-03-28T22:21:14.786 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000039645 2024-03-28T22:21:14.786 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.787 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:21:14.787 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.787 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.787 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.787 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.787 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T22:21:14.787 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T22:21:14.787 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.788 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T22:21:14.788 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 250, 2024-03-28T22:21:14.788 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 232, 2024-03-28T22:21:14.788 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 1622088, 2024-03-28T22:21:14.788 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 133146, 2024-03-28T22:21:14.788 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 4, 2024-03-28T22:21:14.788 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 3, 2024-03-28T22:21:14.788 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.046659627, 2024-03-28T22:21:14.788 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.014566369, 2024-03-28T22:21:14.789 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.034087796, 2024-03-28T22:21:14.789 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.005701435, 2024-03-28T22:21:14.789 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:21:14.789 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 231, 2024-03-28T22:21:14.789 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.016807745, 2024-03-28T22:21:14.789 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000072760 2024-03-28T22:21:14.789 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.789 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:21:14.789 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.789 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.790 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.790 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.790 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 1620432, 2024-03-28T22:21:14.790 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 131808 2024-03-28T22:21:14.790 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.790 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T22:21:14.790 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 746, 2024-03-28T22:21:14.790 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 902, 2024-03-28T22:21:14.791 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 185503, 2024-03-28T22:21:14.791 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 682073, 2024-03-28T22:21:14.791 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 6, 2024-03-28T22:21:14.791 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 4, 2024-03-28T22:21:14.791 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.083516491, 2024-03-28T22:21:14.791 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.031743204, 2024-03-28T22:21:14.791 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.017466069, 2024-03-28T22:21:14.791 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.025187999, 2024-03-28T22:21:14.791 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:21:14.792 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 900, 2024-03-28T22:21:14.792 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.044637163, 2024-03-28T22:21:14.792 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000049596 2024-03-28T22:21:14.792 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.792 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:21:14.792 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.792 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.792 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.792 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.793 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 3392, 2024-03-28T22:21:14.793 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 446752 2024-03-28T22:21:14.793 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.793 INFO:teuthology.orchestra.run.smithi063.stdout: "cct": { 2024-03-28T22:21:14.793 INFO:teuthology.orchestra.run.smithi063.stdout: "total_workers": 1, 2024-03-28T22:21:14.793 INFO:teuthology.orchestra.run.smithi063.stdout: "unhealthy_workers": 0 2024-03-28T22:21:14.793 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.793 INFO:teuthology.orchestra.run.smithi063.stdout: "finisher-MDSRank": { 2024-03-28T22:21:14.793 INFO:teuthology.orchestra.run.smithi063.stdout: "queue_len": 0, 2024-03-28T22:21:14.794 INFO:teuthology.orchestra.run.smithi063.stdout: "complete_latency": { 2024-03-28T22:21:14.794 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 3851, 2024-03-28T22:21:14.794 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.107132539, 2024-03-28T22:21:14.794 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000027819 2024-03-28T22:21:14.794 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-28T22:21:14.794 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.794 INFO:teuthology.orchestra.run.smithi063.stdout: "finisher-PurgeQueue": { 2024-03-28T22:21:14.794 INFO:teuthology.orchestra.run.smithi063.stdout: "queue_len": 0, 2024-03-28T22:21:14.795 INFO:teuthology.orchestra.run.smithi063.stdout: "complete_latency": { 2024-03-28T22:21:14.795 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.795 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.795 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.795 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-28T22:21:14.795 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.795 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-28T22:21:14.795 INFO:teuthology.orchestra.run.smithi063.stdout: "request": 0, 2024-03-28T22:21:14.796 INFO:teuthology.orchestra.run.smithi063.stdout: "reply": 0, 2024-03-28T22:21:14.796 INFO:teuthology.orchestra.run.smithi063.stdout: "reply_latency": { 2024-03-28T22:21:14.796 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.796 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.796 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.796 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.796 INFO:teuthology.orchestra.run.smithi063.stdout: "slow_reply": 0, 2024-03-28T22:21:14.796 INFO:teuthology.orchestra.run.smithi063.stdout: "forward": 0, 2024-03-28T22:21:14.796 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_fetch_complete": 0, 2024-03-28T22:21:14.796 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_fetch_keys": 0, 2024-03-28T22:21:14.797 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_commit": 0, 2024-03-28T22:21:14.797 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_split": 0, 2024-03-28T22:21:14.797 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_merge": 0, 2024-03-28T22:21:14.797 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes": 88, 2024-03-28T22:21:14.797 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_top": 2, 2024-03-28T22:21:14.797 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_bottom": 86, 2024-03-28T22:21:14.797 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_pin_tail": 0, 2024-03-28T22:21:14.797 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_pinned": 86, 2024-03-28T22:21:14.798 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_expired": 0, 2024-03-28T22:21:14.798 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_with_caps": 0, 2024-03-28T22:21:14.798 INFO:teuthology.orchestra.run.smithi063.stdout: "caps": 0, 2024-03-28T22:21:14.798 INFO:teuthology.orchestra.run.smithi063.stdout: "subtrees": 5, 2024-03-28T22:21:14.798 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse": 0, 2024-03-28T22:21:14.798 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_hit": 0, 2024-03-28T22:21:14.798 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_forward": 0, 2024-03-28T22:21:14.798 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_discover": 0, 2024-03-28T22:21:14.799 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_dir_fetch": 0, 2024-03-28T22:21:14.799 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_remote_ino": 0, 2024-03-28T22:21:14.799 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_lock": 0, 2024-03-28T22:21:14.799 INFO:teuthology.orchestra.run.smithi063.stdout: "load_cent": 0, 2024-03-28T22:21:14.799 INFO:teuthology.orchestra.run.smithi063.stdout: "q": 0, 2024-03-28T22:21:14.799 INFO:teuthology.orchestra.run.smithi063.stdout: "exported": 0, 2024-03-28T22:21:14.799 INFO:teuthology.orchestra.run.smithi063.stdout: "exported_inodes": 0, 2024-03-28T22:21:14.799 INFO:teuthology.orchestra.run.smithi063.stdout: "imported": 0, 2024-03-28T22:21:14.799 INFO:teuthology.orchestra.run.smithi063.stdout: "imported_inodes": 0, 2024-03-28T22:21:14.800 INFO:teuthology.orchestra.run.smithi063.stdout: "openino_dir_fetch": 0, 2024-03-28T22:21:14.800 INFO:teuthology.orchestra.run.smithi063.stdout: "openino_backtrace_fetch": 0, 2024-03-28T22:21:14.800 INFO:teuthology.orchestra.run.smithi063.stdout: "openino_peer_discover": 0, 2024-03-28T22:21:14.800 INFO:teuthology.orchestra.run.smithi063.stdout: "root_rfiles": 3452, 2024-03-28T22:21:14.800 INFO:teuthology.orchestra.run.smithi063.stdout: "root_rbytes": 26525988, 2024-03-28T22:21:14.800 INFO:teuthology.orchestra.run.smithi063.stdout: "root_rsnaps": 6, 2024-03-28T22:21:14.800 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T22:21:14.800 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_set_tag": 0, 2024-03-28T22:21:14.800 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T22:21:14.800 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_inotable_repaired": 0, 2024-03-28T22:21:14.801 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_dir_inodes": 0, 2024-03-28T22:21:14.801 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T22:21:14.801 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T22:21:14.801 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_file_inodes": 0, 2024-03-28T22:21:14.801 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_inode_file_caps": 0, 2024-03-28T22:21:14.801 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T22:21:14.801 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_grant": 0, 2024-03-28T22:21:14.801 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T22:21:14.801 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T22:21:14.802 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T22:21:14.802 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_caps": 0, 2024-03-28T22:21:14.802 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_caps_dirty": 0, 2024-03-28T22:21:14.802 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_cap_release": 0, 2024-03-28T22:21:14.802 INFO:teuthology.orchestra.run.smithi063.stdout: "process_request_cap_release": 0 2024-03-28T22:21:14.802 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.802 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_cache": { 2024-03-28T22:21:14.802 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_update": 0, 2024-03-28T22:21:14.802 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_update_receipt": 0, 2024-03-28T22:21:14.803 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_try_discover": 0, 2024-03-28T22:21:14.803 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_send_discover": 0, 2024-03-28T22:21:14.803 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_handle_discover": 0, 2024-03-28T22:21:14.803 INFO:teuthology.orchestra.run.smithi063.stdout: "num_strays": 0, 2024-03-28T22:21:14.803 INFO:teuthology.orchestra.run.smithi063.stdout: "num_strays_delayed": 0, 2024-03-28T22:21:14.803 INFO:teuthology.orchestra.run.smithi063.stdout: "num_strays_enqueuing": 0, 2024-03-28T22:21:14.803 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_created": 0, 2024-03-28T22:21:14.803 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_enqueued": 0, 2024-03-28T22:21:14.803 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_reintegrated": 0, 2024-03-28T22:21:14.804 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_migrated": 0, 2024-03-28T22:21:14.804 INFO:teuthology.orchestra.run.smithi063.stdout: "num_recovering_processing": 0, 2024-03-28T22:21:14.804 INFO:teuthology.orchestra.run.smithi063.stdout: "num_recovering_enqueued": 0, 2024-03-28T22:21:14.804 INFO:teuthology.orchestra.run.smithi063.stdout: "num_recovering_prioritized": 0, 2024-03-28T22:21:14.804 INFO:teuthology.orchestra.run.smithi063.stdout: "recovery_started": 0, 2024-03-28T22:21:14.804 INFO:teuthology.orchestra.run.smithi063.stdout: "recovery_completed": 0, 2024-03-28T22:21:14.804 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_quiesce_path": 0, 2024-03-28T22:21:14.804 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_quiesce_inode": 0, 2024-03-28T22:21:14.804 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T22:21:14.805 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_exportdir": 0, 2024-03-28T22:21:14.805 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_flush": 0, 2024-03-28T22:21:14.805 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_fragmentdir": 0, 2024-03-28T22:21:14.805 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_fragstats": 0, 2024-03-28T22:21:14.805 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_inodestats": 0 2024-03-28T22:21:14.805 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.805 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_log": { 2024-03-28T22:21:14.805 INFO:teuthology.orchestra.run.smithi063.stdout: "evlrg": 0, 2024-03-28T22:21:14.805 INFO:teuthology.orchestra.run.smithi063.stdout: "evadd": 0, 2024-03-28T22:21:14.806 INFO:teuthology.orchestra.run.smithi063.stdout: "evex": 0, 2024-03-28T22:21:14.806 INFO:teuthology.orchestra.run.smithi063.stdout: "evtrm": 0, 2024-03-28T22:21:14.806 INFO:teuthology.orchestra.run.smithi063.stdout: "ev": 899, 2024-03-28T22:21:14.806 INFO:teuthology.orchestra.run.smithi063.stdout: "evexg": 0, 2024-03-28T22:21:14.806 INFO:teuthology.orchestra.run.smithi063.stdout: "evexd": 0, 2024-03-28T22:21:14.806 INFO:teuthology.orchestra.run.smithi063.stdout: "segadd": 0, 2024-03-28T22:21:14.806 INFO:teuthology.orchestra.run.smithi063.stdout: "segex": 0, 2024-03-28T22:21:14.806 INFO:teuthology.orchestra.run.smithi063.stdout: "segtrm": 0, 2024-03-28T22:21:14.807 INFO:teuthology.orchestra.run.smithi063.stdout: "seg": 2, 2024-03-28T22:21:14.807 INFO:teuthology.orchestra.run.smithi063.stdout: "segmjr": 2, 2024-03-28T22:21:14.807 INFO:teuthology.orchestra.run.smithi063.stdout: "segexg": 0, 2024-03-28T22:21:14.807 INFO:teuthology.orchestra.run.smithi063.stdout: "segexd": 0, 2024-03-28T22:21:14.807 INFO:teuthology.orchestra.run.smithi063.stdout: "expos": 4194304, 2024-03-28T22:21:14.807 INFO:teuthology.orchestra.run.smithi063.stdout: "wrpos": 6402975, 2024-03-28T22:21:14.807 INFO:teuthology.orchestra.run.smithi063.stdout: "rdpos": 6402915, 2024-03-28T22:21:14.807 INFO:teuthology.orchestra.run.smithi063.stdout: "jlat": { 2024-03-28T22:21:14.807 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.808 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.808 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.808 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.808 INFO:teuthology.orchestra.run.smithi063.stdout: "replayed": 899 2024-03-28T22:21:14.808 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.808 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_mem": { 2024-03-28T22:21:14.808 INFO:teuthology.orchestra.run.smithi063.stdout: "ino": 64, 2024-03-28T22:21:14.808 INFO:teuthology.orchestra.run.smithi063.stdout: "ino+": 658, 2024-03-28T22:21:14.808 INFO:teuthology.orchestra.run.smithi063.stdout: "ino-": 594, 2024-03-28T22:21:14.808 INFO:teuthology.orchestra.run.smithi063.stdout: "dir": 18, 2024-03-28T22:21:14.809 INFO:teuthology.orchestra.run.smithi063.stdout: "dir+": 71, 2024-03-28T22:21:14.809 INFO:teuthology.orchestra.run.smithi063.stdout: "dir-": 53, 2024-03-28T22:21:14.809 INFO:teuthology.orchestra.run.smithi063.stdout: "dn": 88, 2024-03-28T22:21:14.809 INFO:teuthology.orchestra.run.smithi063.stdout: "dn+": 673, 2024-03-28T22:21:14.809 INFO:teuthology.orchestra.run.smithi063.stdout: "dn-": 585, 2024-03-28T22:21:14.809 INFO:teuthology.orchestra.run.smithi063.stdout: "cap": 0, 2024-03-28T22:21:14.809 INFO:teuthology.orchestra.run.smithi063.stdout: "cap+": 0, 2024-03-28T22:21:14.809 INFO:teuthology.orchestra.run.smithi063.stdout: "cap-": 0, 2024-03-28T22:21:14.810 INFO:teuthology.orchestra.run.smithi063.stdout: "rss": 48584, 2024-03-28T22:21:14.810 INFO:teuthology.orchestra.run.smithi063.stdout: "heap": 215420 2024-03-28T22:21:14.810 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.810 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_server": { 2024-03-28T22:21:14.810 INFO:teuthology.orchestra.run.smithi063.stdout: "dispatch_client_request": 0, 2024-03-28T22:21:14.810 INFO:teuthology.orchestra.run.smithi063.stdout: "dispatch_server_request": 0, 2024-03-28T22:21:14.810 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_request": 0, 2024-03-28T22:21:14.810 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_session": 0, 2024-03-28T22:21:14.811 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_peer_request": 0, 2024-03-28T22:21:14.811 INFO:teuthology.orchestra.run.smithi063.stdout: "req_create_latency": { 2024-03-28T22:21:14.811 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.811 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.811 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.811 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.811 INFO:teuthology.orchestra.run.smithi063.stdout: "req_getattr_latency": { 2024-03-28T22:21:14.811 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.811 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.812 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.812 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.812 INFO:teuthology.orchestra.run.smithi063.stdout: "req_getfilelock_latency": { 2024-03-28T22:21:14.812 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.812 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.812 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.812 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.812 INFO:teuthology.orchestra.run.smithi063.stdout: "req_link_latency": { 2024-03-28T22:21:14.813 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.813 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.813 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.813 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.813 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookup_latency": { 2024-03-28T22:21:14.813 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.813 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.813 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.814 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.814 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookuphash_latency": { 2024-03-28T22:21:14.814 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.814 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.814 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.814 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.814 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupino_latency": { 2024-03-28T22:21:14.814 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.814 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.815 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.815 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.815 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupname_latency": { 2024-03-28T22:21:14.815 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.815 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.815 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.815 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.815 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupparent_latency": { 2024-03-28T22:21:14.815 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.816 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.816 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.816 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.816 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupsnap_latency": { 2024-03-28T22:21:14.816 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.816 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.816 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.816 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.817 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lssnap_latency": { 2024-03-28T22:21:14.817 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.817 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.817 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.817 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.817 INFO:teuthology.orchestra.run.smithi063.stdout: "req_mkdir_latency": { 2024-03-28T22:21:14.818 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.818 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.818 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.818 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.818 INFO:teuthology.orchestra.run.smithi063.stdout: "req_mknod_latency": { 2024-03-28T22:21:14.818 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.819 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.819 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.819 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.819 INFO:teuthology.orchestra.run.smithi063.stdout: "req_mksnap_latency": { 2024-03-28T22:21:14.819 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.819 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.819 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.819 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.819 INFO:teuthology.orchestra.run.smithi063.stdout: "req_open_latency": { 2024-03-28T22:21:14.820 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.820 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.820 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.820 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.820 INFO:teuthology.orchestra.run.smithi063.stdout: "req_readdir_latency": { 2024-03-28T22:21:14.820 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.820 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.820 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.820 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.821 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rename_latency": { 2024-03-28T22:21:14.821 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.821 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.821 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.821 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.821 INFO:teuthology.orchestra.run.smithi063.stdout: "req_renamesnap_latency": { 2024-03-28T22:21:14.821 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.821 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.822 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.822 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.822 INFO:teuthology.orchestra.run.smithi063.stdout: "req_snapdiff_latency": { 2024-03-28T22:21:14.822 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.822 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.822 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.822 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.822 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rmdir_latency": { 2024-03-28T22:21:14.823 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.823 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.823 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.823 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.823 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rmsnap_latency": { 2024-03-28T22:21:14.823 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.823 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.823 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.824 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.824 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rmxattr_latency": { 2024-03-28T22:21:14.824 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.824 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.824 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.824 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.824 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setattr_latency": { 2024-03-28T22:21:14.824 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.824 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.825 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.825 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.825 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setdirlayout_latency": { 2024-03-28T22:21:14.825 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.825 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.825 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.825 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.825 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setfilelock_latency": { 2024-03-28T22:21:14.826 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.826 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.826 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.826 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.826 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setlayout_latency": { 2024-03-28T22:21:14.826 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.826 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.826 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.827 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.827 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setxattr_latency": { 2024-03-28T22:21:14.827 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.827 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.827 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.827 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.827 INFO:teuthology.orchestra.run.smithi063.stdout: "req_symlink_latency": { 2024-03-28T22:21:14.827 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.827 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.828 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.828 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.828 INFO:teuthology.orchestra.run.smithi063.stdout: "req_unlink_latency": { 2024-03-28T22:21:14.828 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.828 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.828 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.828 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.828 INFO:teuthology.orchestra.run.smithi063.stdout: "cap_revoke_eviction": 0, 2024-03-28T22:21:14.829 INFO:teuthology.orchestra.run.smithi063.stdout: "cap_acquisition_throttle": 0, 2024-03-28T22:21:14.829 INFO:teuthology.orchestra.run.smithi063.stdout: "req_getvxattr_latency": { 2024-03-28T22:21:14.829 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.829 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.829 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.829 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-28T22:21:14.829 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.829 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_sessions": { 2024-03-28T22:21:14.829 INFO:teuthology.orchestra.run.smithi063.stdout: "session_count": 2, 2024-03-28T22:21:14.830 INFO:teuthology.orchestra.run.smithi063.stdout: "session_add": 6, 2024-03-28T22:21:14.830 INFO:teuthology.orchestra.run.smithi063.stdout: "session_remove": 4, 2024-03-28T22:21:14.830 INFO:teuthology.orchestra.run.smithi063.stdout: "sessions_open": 4, 2024-03-28T22:21:14.830 INFO:teuthology.orchestra.run.smithi063.stdout: "sessions_stale": 0, 2024-03-28T22:21:14.830 INFO:teuthology.orchestra.run.smithi063.stdout: "total_load": 0, 2024-03-28T22:21:14.830 INFO:teuthology.orchestra.run.smithi063.stdout: "average_load": 0, 2024-03-28T22:21:14.830 INFO:teuthology.orchestra.run.smithi063.stdout: "avg_session_uptime": 5975, 2024-03-28T22:21:14.830 INFO:teuthology.orchestra.run.smithi063.stdout: "mdthresh_evicted": 0 2024-03-28T22:21:14.830 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.830 INFO:teuthology.orchestra.run.smithi063.stdout: "mempool": { 2024-03-28T22:21:14.831 INFO:teuthology.orchestra.run.smithi063.stdout: "bloom_filter_bytes": 0, 2024-03-28T22:21:14.831 INFO:teuthology.orchestra.run.smithi063.stdout: "bloom_filter_items": 0, 2024-03-28T22:21:14.831 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T22:21:14.831 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_alloc_items": 0, 2024-03-28T22:21:14.831 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T22:21:14.831 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_data_items": 0, 2024-03-28T22:21:14.831 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T22:21:14.831 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T22:21:14.832 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T22:21:14.832 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T22:21:14.832 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T22:21:14.832 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_other_items": 0, 2024-03-28T22:21:14.832 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T22:21:14.832 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T22:21:14.832 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_extent_bytes": 0, 2024-03-28T22:21:14.832 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_extent_items": 0, 2024-03-28T22:21:14.832 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_blob_bytes": 0, 2024-03-28T22:21:14.833 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_blob_items": 0, 2024-03-28T22:21:14.833 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T22:21:14.833 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T22:21:14.833 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T22:21:14.833 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T22:21:14.833 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T22:21:14.833 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_items": 0, 2024-03-28T22:21:14.833 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_txc_bytes": 0, 2024-03-28T22:21:14.833 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_txc_items": 0, 2024-03-28T22:21:14.834 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T22:21:14.834 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T22:21:14.834 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_bytes": 0, 2024-03-28T22:21:14.834 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_items": 0, 2024-03-28T22:21:14.834 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_bytes": 0, 2024-03-28T22:21:14.834 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_items": 0, 2024-03-28T22:21:14.834 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T22:21:14.834 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_reader_items": 0, 2024-03-28T22:21:14.834 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T22:21:14.835 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_writer_items": 0, 2024-03-28T22:21:14.835 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_anon_bytes": 191431, 2024-03-28T22:21:14.835 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_anon_items": 106, 2024-03-28T22:21:14.835 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_meta_bytes": 0, 2024-03-28T22:21:14.835 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_meta_items": 0, 2024-03-28T22:21:14.835 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_bytes": 0, 2024-03-28T22:21:14.835 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_items": 0, 2024-03-28T22:21:14.835 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mapbl_bytes": 0, 2024-03-28T22:21:14.835 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mapbl_items": 0, 2024-03-28T22:21:14.836 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pglog_bytes": 0, 2024-03-28T22:21:14.836 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pglog_items": 0, 2024-03-28T22:21:14.836 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_bytes": 8280, 2024-03-28T22:21:14.836 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_items": 189, 2024-03-28T22:21:14.836 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T22:21:14.836 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_mapping_items": 0, 2024-03-28T22:21:14.836 INFO:teuthology.orchestra.run.smithi063.stdout: "pgmap_bytes": 0, 2024-03-28T22:21:14.836 INFO:teuthology.orchestra.run.smithi063.stdout: "pgmap_items": 0, 2024-03-28T22:21:14.836 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_co_bytes": 250272, 2024-03-28T22:21:14.836 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_co_items": 2699, 2024-03-28T22:21:14.837 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_1_bytes": 0, 2024-03-28T22:21:14.837 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_1_items": 0, 2024-03-28T22:21:14.837 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_2_bytes": 0, 2024-03-28T22:21:14.837 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_2_items": 0 2024-03-28T22:21:14.837 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.837 INFO:teuthology.orchestra.run.smithi063.stdout: "objecter": { 2024-03-28T22:21:14.837 INFO:teuthology.orchestra.run.smithi063.stdout: "op_active": 0, 2024-03-28T22:21:14.837 INFO:teuthology.orchestra.run.smithi063.stdout: "op_laggy": 0, 2024-03-28T22:21:14.837 INFO:teuthology.orchestra.run.smithi063.stdout: "op_send": 2341, 2024-03-28T22:21:14.838 INFO:teuthology.orchestra.run.smithi063.stdout: "op_send_bytes": 26, 2024-03-28T22:21:14.838 INFO:teuthology.orchestra.run.smithi063.stdout: "op_resend": 4, 2024-03-28T22:21:14.838 INFO:teuthology.orchestra.run.smithi063.stdout: "op_reply": 2341, 2024-03-28T22:21:14.838 INFO:teuthology.orchestra.run.smithi063.stdout: "op_latency": { 2024-03-28T22:21:14.838 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 2341, 2024-03-28T22:21:14.838 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 1.075990166, 2024-03-28T22:21:14.838 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000459628 2024-03-28T22:21:14.838 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.838 INFO:teuthology.orchestra.run.smithi063.stdout: "op_inflight": 0, 2024-03-28T22:21:14.839 INFO:teuthology.orchestra.run.smithi063.stdout: "oplen_avg": { 2024-03-28T22:21:14.839 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 2341, 2024-03-28T22:21:14.839 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 2342 2024-03-28T22:21:14.839 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.839 INFO:teuthology.orchestra.run.smithi063.stdout: "op": 2341, 2024-03-28T22:21:14.839 INFO:teuthology.orchestra.run.smithi063.stdout: "op_r": 2341, 2024-03-28T22:21:14.839 INFO:teuthology.orchestra.run.smithi063.stdout: "op_w": 0, 2024-03-28T22:21:14.839 INFO:teuthology.orchestra.run.smithi063.stdout: "op_rmw": 0, 2024-03-28T22:21:14.839 INFO:teuthology.orchestra.run.smithi063.stdout: "op_pg": 0, 2024-03-28T22:21:14.840 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_stat": 1499, 2024-03-28T22:21:14.840 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_create": 0, 2024-03-28T22:21:14.840 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_read": 839, 2024-03-28T22:21:14.840 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_write": 0, 2024-03-28T22:21:14.840 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_writefull": 0, 2024-03-28T22:21:14.840 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_writesame": 0, 2024-03-28T22:21:14.840 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_append": 0, 2024-03-28T22:21:14.840 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_zero": 0, 2024-03-28T22:21:14.840 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_truncate": 0, 2024-03-28T22:21:14.840 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_delete": 0, 2024-03-28T22:21:14.841 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_mapext": 0, 2024-03-28T22:21:14.841 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_sparse_read": 0, 2024-03-28T22:21:14.841 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_clonerange": 0, 2024-03-28T22:21:14.841 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_getxattr": 2, 2024-03-28T22:21:14.841 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_setxattr": 0, 2024-03-28T22:21:14.841 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_cmpxattr": 0, 2024-03-28T22:21:14.841 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_rmxattr": 0, 2024-03-28T22:21:14.841 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_resetxattrs": 0, 2024-03-28T22:21:14.841 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_call": 0, 2024-03-28T22:21:14.842 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_watch": 0, 2024-03-28T22:21:14.842 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_notify": 0, 2024-03-28T22:21:14.842 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T22:21:14.842 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_pgls": 0, 2024-03-28T22:21:14.842 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_pgls_filter": 0, 2024-03-28T22:21:14.842 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_other": 0, 2024-03-28T22:21:14.842 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_active": 0, 2024-03-28T22:21:14.842 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_send": 0, 2024-03-28T22:21:14.842 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_resend": 0, 2024-03-28T22:21:14.843 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_ping": 0, 2024-03-28T22:21:14.843 INFO:teuthology.orchestra.run.smithi063.stdout: "poolop_active": 0, 2024-03-28T22:21:14.843 INFO:teuthology.orchestra.run.smithi063.stdout: "poolop_send": 0, 2024-03-28T22:21:14.843 INFO:teuthology.orchestra.run.smithi063.stdout: "poolop_resend": 0, 2024-03-28T22:21:14.843 INFO:teuthology.orchestra.run.smithi063.stdout: "poolstat_active": 0, 2024-03-28T22:21:14.843 INFO:teuthology.orchestra.run.smithi063.stdout: "poolstat_send": 0, 2024-03-28T22:21:14.843 INFO:teuthology.orchestra.run.smithi063.stdout: "poolstat_resend": 0, 2024-03-28T22:21:14.843 INFO:teuthology.orchestra.run.smithi063.stdout: "statfs_active": 0, 2024-03-28T22:21:14.843 INFO:teuthology.orchestra.run.smithi063.stdout: "statfs_send": 0, 2024-03-28T22:21:14.844 INFO:teuthology.orchestra.run.smithi063.stdout: "statfs_resend": 0, 2024-03-28T22:21:14.844 INFO:teuthology.orchestra.run.smithi063.stdout: "command_active": 0, 2024-03-28T22:21:14.844 INFO:teuthology.orchestra.run.smithi063.stdout: "command_send": 0, 2024-03-28T22:21:14.844 INFO:teuthology.orchestra.run.smithi063.stdout: "command_resend": 0, 2024-03-28T22:21:14.844 INFO:teuthology.orchestra.run.smithi063.stdout: "map_epoch": 98, 2024-03-28T22:21:14.844 INFO:teuthology.orchestra.run.smithi063.stdout: "map_full": 0, 2024-03-28T22:21:14.844 INFO:teuthology.orchestra.run.smithi063.stdout: "map_inc": 12, 2024-03-28T22:21:14.844 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_sessions": 5, 2024-03-28T22:21:14.844 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_session_open": 5, 2024-03-28T22:21:14.845 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_session_close": 0, 2024-03-28T22:21:14.845 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_laggy": 0, 2024-03-28T22:21:14.845 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_wr": 0, 2024-03-28T22:21:14.845 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_rd": 2, 2024-03-28T22:21:14.845 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_del": 0 2024-03-28T22:21:14.845 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.845 INFO:teuthology.orchestra.run.smithi063.stdout: "oft": { 2024-03-28T22:21:14.845 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_objs": 0, 2024-03-28T22:21:14.845 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_kv_pairs": 0, 2024-03-28T22:21:14.845 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_updates": 0, 2024-03-28T22:21:14.846 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_removes": 0 2024-03-28T22:21:14.846 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.846 INFO:teuthology.orchestra.run.smithi063.stdout: "osd-slow-ops": { 2024-03-28T22:21:14.846 INFO:teuthology.orchestra.run.smithi063.stdout: "slow_ops_count": 0 2024-03-28T22:21:14.846 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.846 INFO:teuthology.orchestra.run.smithi063.stdout: "purge_queue": { 2024-03-28T22:21:14.846 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing_ops": 0, 2024-03-28T22:21:14.846 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T22:21:14.846 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing": 0, 2024-03-28T22:21:14.847 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing_high_water": 0, 2024-03-28T22:21:14.847 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executed_ops": 0, 2024-03-28T22:21:14.847 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executed": 0, 2024-03-28T22:21:14.847 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_item_in_journal": 0 2024-03-28T22:21:14.847 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.847 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T22:21:14.847 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-03-28T22:21:14.847 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 104857600, 2024-03-28T22:21:14.847 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-03-28T22:21:14.848 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 2589, 2024-03-28T22:21:14.848 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 4241643, 2024-03-28T22:21:14.848 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-03-28T22:21:14.848 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 2589, 2024-03-28T22:21:14.848 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 0, 2024-03-28T22:21:14.848 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 0, 2024-03-28T22:21:14.848 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 2589, 2024-03-28T22:21:14.848 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 4241643, 2024-03-28T22:21:14.849 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-03-28T22:21:14.849 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.849 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.849 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.849 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-28T22:21:14.849 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.849 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-objecter_bytes": { 2024-03-28T22:21:14.849 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-03-28T22:21:14.849 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 104857600, 2024-03-28T22:21:14.850 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-03-28T22:21:14.850 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 0, 2024-03-28T22:21:14.850 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 0, 2024-03-28T22:21:14.850 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-03-28T22:21:14.850 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 0, 2024-03-28T22:21:14.850 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 2341, 2024-03-28T22:21:14.850 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 2208681, 2024-03-28T22:21:14.850 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 2341, 2024-03-28T22:21:14.851 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 2208681, 2024-03-28T22:21:14.851 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-03-28T22:21:14.851 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.851 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.851 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.851 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-28T22:21:14.851 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.851 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-objecter_ops": { 2024-03-28T22:21:14.851 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-03-28T22:21:14.852 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 1024, 2024-03-28T22:21:14.852 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-03-28T22:21:14.852 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 0, 2024-03-28T22:21:14.852 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 0, 2024-03-28T22:21:14.852 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-03-28T22:21:14.852 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 0, 2024-03-28T22:21:14.852 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 2341, 2024-03-28T22:21:14.852 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 2341, 2024-03-28T22:21:14.853 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 2341, 2024-03-28T22:21:14.853 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 2341, 2024-03-28T22:21:14.853 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-03-28T22:21:14.853 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.853 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.853 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.853 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-28T22:21:14.853 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.854 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-write_buf_throttle": { 2024-03-28T22:21:14.854 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-03-28T22:21:14.854 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 3758096384, 2024-03-28T22:21:14.854 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-03-28T22:21:14.854 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 0, 2024-03-28T22:21:14.854 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 0, 2024-03-28T22:21:14.854 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-03-28T22:21:14.854 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 0, 2024-03-28T22:21:14.854 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 0, 2024-03-28T22:21:14.855 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 0, 2024-03-28T22:21:14.855 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 0, 2024-03-28T22:21:14.855 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 0, 2024-03-28T22:21:14.855 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-03-28T22:21:14.855 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.855 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.855 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.855 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-28T22:21:14.856 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:14.856 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-write_buf_throttle-0x55d9e07c66e0": { 2024-03-28T22:21:14.856 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-03-28T22:21:14.856 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 3758096384, 2024-03-28T22:21:14.856 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-03-28T22:21:14.856 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 0, 2024-03-28T22:21:14.856 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 0, 2024-03-28T22:21:14.856 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-03-28T22:21:14.856 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 0, 2024-03-28T22:21:14.857 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 0, 2024-03-28T22:21:14.857 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 0, 2024-03-28T22:21:14.857 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 0, 2024-03-28T22:21:14.857 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 0, 2024-03-28T22:21:14.857 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-03-28T22:21:14.857 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:14.857 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:14.857 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:14.858 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-28T22:21:14.858 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-28T22:21:14.858 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-28T22:21:15.643 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-03-28T22:21:15.643 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-03-28T22:21:15.643 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=0 2024-03-28T22:21:15.643 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=0 2024-03-28T22:21:15.643 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-03-28T22:21:15.643 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-03-28T22:21:15.643 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T22:21:15.644 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=0 2024-03-28T22:21:15.644 DEBUG:tasks.check_counter:Getting stats from l 2024-03-28T22:21:15.644 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-28T22:21:15.896 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T22:21:15.930 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:15 smithi063 ceph-mon[36249]: pgmap v332: 97 pgs: 97 active+clean; 91 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-28T22:21:15.934 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:15 smithi016 ceph-mon[32374]: pgmap v332: 97 pgs: 97 active+clean; 91 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-28T22:21:16.046 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:15 smithi053 ceph-mon[35979]: pgmap v332: 97 pgs: 97 active+clean; 91 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-28T22:21:17.456 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-28T22:21:17.456 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T22:21:17.456 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 0, 2024-03-28T22:21:17.456 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 2, 2024-03-28T22:21:17.456 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 0, 2024-03-28T22:21:17.457 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 0, 2024-03-28T22:21:17.457 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 4, 2024-03-28T22:21:17.457 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 0, 2024-03-28T22:21:17.457 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.001262594, 2024-03-28T22:21:17.457 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.000000000, 2024-03-28T22:21:17.457 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.000336934, 2024-03-28T22:21:17.457 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T22:21:17.457 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:21:17.458 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:17.458 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:17.458 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:17.458 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:17.458 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:21:17.458 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:17.458 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:17.458 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:17.459 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:17.459 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T22:21:17.459 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T22:21:17.459 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:17.459 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T22:21:17.459 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 2, 2024-03-28T22:21:17.459 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 165, 2024-03-28T22:21:17.459 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 320, 2024-03-28T22:21:17.460 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 184608, 2024-03-28T22:21:17.460 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 5, 2024-03-28T22:21:17.460 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 3, 2024-03-28T22:21:17.460 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.012413700, 2024-03-28T22:21:17.460 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.008453174, 2024-03-28T22:21:17.460 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.001559801, 2024-03-28T22:21:17.460 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T22:21:17.460 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:21:17.461 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 162, 2024-03-28T22:21:17.461 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.012820160, 2024-03-28T22:21:17.461 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000079136 2024-03-28T22:21:17.461 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:17.461 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:21:17.461 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:17.461 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:17.461 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:17.461 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:17.462 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-28T22:21:17.462 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 184608 2024-03-28T22:21:17.462 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:17.462 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T22:21:17.462 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 231, 2024-03-28T22:21:17.462 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 218, 2024-03-28T22:21:17.462 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 1573584, 2024-03-28T22:21:17.462 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 132992, 2024-03-28T22:21:17.463 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 2, 2024-03-28T22:21:17.463 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 1, 2024-03-28T22:21:17.463 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.047985910, 2024-03-28T22:21:17.463 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.014033972, 2024-03-28T22:21:17.463 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.037011778, 2024-03-28T22:21:17.463 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.005868669, 2024-03-28T22:21:17.463 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:21:17.463 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 218, 2024-03-28T22:21:17.464 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.013269077, 2024-03-28T22:21:17.464 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000060867 2024-03-28T22:21:17.464 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:17.464 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:21:17.464 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:17.464 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:17.464 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:17.464 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:17.464 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 1573584, 2024-03-28T22:21:17.465 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 132992 2024-03-28T22:21:17.465 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:17.465 INFO:teuthology.orchestra.run.smithi063.stdout: "cct": { 2024-03-28T22:21:17.465 INFO:teuthology.orchestra.run.smithi063.stdout: "total_workers": 0, 2024-03-28T22:21:17.465 INFO:teuthology.orchestra.run.smithi063.stdout: "unhealthy_workers": 0 2024-03-28T22:21:17.465 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:17.465 INFO:teuthology.orchestra.run.smithi063.stdout: "mempool": { 2024-03-28T22:21:17.465 INFO:teuthology.orchestra.run.smithi063.stdout: "bloom_filter_bytes": 0, 2024-03-28T22:21:17.465 INFO:teuthology.orchestra.run.smithi063.stdout: "bloom_filter_items": 0, 2024-03-28T22:21:17.466 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T22:21:17.466 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_alloc_items": 0, 2024-03-28T22:21:17.466 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T22:21:17.466 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_data_items": 0, 2024-03-28T22:21:17.466 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T22:21:17.466 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T22:21:17.466 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T22:21:17.466 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T22:21:17.466 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T22:21:17.467 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_other_items": 0, 2024-03-28T22:21:17.467 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T22:21:17.467 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T22:21:17.467 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_extent_bytes": 0, 2024-03-28T22:21:17.467 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_extent_items": 0, 2024-03-28T22:21:17.467 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_blob_bytes": 0, 2024-03-28T22:21:17.467 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_blob_items": 0, 2024-03-28T22:21:17.467 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T22:21:17.468 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T22:21:17.468 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T22:21:17.468 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T22:21:17.468 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T22:21:17.468 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_items": 0, 2024-03-28T22:21:17.468 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_txc_bytes": 0, 2024-03-28T22:21:17.468 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_txc_items": 0, 2024-03-28T22:21:17.468 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T22:21:17.469 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T22:21:17.469 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_bytes": 0, 2024-03-28T22:21:17.469 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_items": 0, 2024-03-28T22:21:17.469 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_bytes": 0, 2024-03-28T22:21:17.469 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_items": 0, 2024-03-28T22:21:17.469 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T22:21:17.469 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_reader_items": 0, 2024-03-28T22:21:17.469 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T22:21:17.469 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_writer_items": 0, 2024-03-28T22:21:17.470 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_anon_bytes": 150048, 2024-03-28T22:21:17.470 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_anon_items": 32, 2024-03-28T22:21:17.470 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_meta_bytes": 0, 2024-03-28T22:21:17.470 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_meta_items": 0, 2024-03-28T22:21:17.470 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_bytes": 0, 2024-03-28T22:21:17.470 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_items": 0, 2024-03-28T22:21:17.470 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mapbl_bytes": 0, 2024-03-28T22:21:17.470 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mapbl_items": 0, 2024-03-28T22:21:17.470 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pglog_bytes": 0, 2024-03-28T22:21:17.471 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pglog_items": 0, 2024-03-28T22:21:17.471 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_bytes": 0, 2024-03-28T22:21:17.471 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_items": 0, 2024-03-28T22:21:17.471 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T22:21:17.471 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_mapping_items": 0, 2024-03-28T22:21:17.471 INFO:teuthology.orchestra.run.smithi063.stdout: "pgmap_bytes": 0, 2024-03-28T22:21:17.471 INFO:teuthology.orchestra.run.smithi063.stdout: "pgmap_items": 0, 2024-03-28T22:21:17.471 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_co_bytes": 928, 2024-03-28T22:21:17.472 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_co_items": 2, 2024-03-28T22:21:17.472 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_1_bytes": 0, 2024-03-28T22:21:17.472 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_1_items": 0, 2024-03-28T22:21:17.472 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_2_bytes": 0, 2024-03-28T22:21:17.472 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_2_items": 0 2024-03-28T22:21:17.472 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-28T22:21:17.472 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T22:21:17.472 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-03-28T22:21:17.473 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 104857600, 2024-03-28T22:21:17.473 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-03-28T22:21:17.473 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 229, 2024-03-28T22:21:17.473 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 1538498, 2024-03-28T22:21:17.473 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-03-28T22:21:17.473 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 229, 2024-03-28T22:21:17.473 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 0, 2024-03-28T22:21:17.473 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 0, 2024-03-28T22:21:17.473 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 229, 2024-03-28T22:21:17.474 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 1538498, 2024-03-28T22:21:17.474 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-03-28T22:21:17.474 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-28T22:21:17.474 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-28T22:21:17.474 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-28T22:21:17.474 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-28T22:21:17.474 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-28T22:21:17.474 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-28T22:21:17.822 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:17 smithi063 ceph-mon[36249]: pgmap v333: 97 pgs: 97 active+clean; 91 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T22:21:17.822 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:17 smithi063 ceph-mon[36249]: osdmap e99: 12 total, 12 up, 12 in 2024-03-28T22:21:17.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:17 smithi016 ceph-mon[32374]: pgmap v333: 97 pgs: 97 active+clean; 91 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T22:21:17.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:17 smithi016 ceph-mon[32374]: osdmap e99: 12 total, 12 up, 12 in 2024-03-28T22:21:18.046 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:17 smithi053 ceph-mon[35979]: pgmap v333: 97 pgs: 97 active+clean; 91 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-28T22:21:18.046 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:17 smithi053 ceph-mon[35979]: osdmap e99: 12 total, 12 up, 12 in 2024-03-28T22:21:18.143 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-03-28T22:21:18.144 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-03-28T22:21:18.144 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.l 2024-03-28T22:21:18.144 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.l 2024-03-28T22:21:18.144 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-03-28T22:21:18.144 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-03-28T22:21:18.144 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-03-28T22:21:18.144 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.l 2024-03-28T22:21:18.145 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-28T22:21:18.155 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-28T22:21:18.155 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-28T22:21:18.155 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T22:21:18.176 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-28T22:21:18.176 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T22:21:18.234 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:21:18.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.389+0000 7fca09602640 1 -- 172.21.15.16:0/390548978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca041065f0 msgr2=0x7fca041069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:18.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.389+0000 7fca09602640 1 --2- 172.21.15.16:0/390548978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca041065f0 0x7fca041069d0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4009900 tx=0x7fc9f402bc30 comp rx=0 tx=0).stop 2024-03-28T22:21:18.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.389+0000 7fca09602640 1 -- 172.21.15.16:0/390548978 shutdown_connections 2024-03-28T22:21:18.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.389+0000 7fca09602640 1 --2- 172.21.15.16:0/390548978 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca04100ef0 0x7fca0410d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:18.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.389+0000 7fca09602640 1 --2- 172.21.15.16:0/390548978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca04100530 0x7fca041009b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:18.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.389+0000 7fca09602640 1 --2- 172.21.15.16:0/390548978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca041065f0 0x7fca041069d0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:18.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.389+0000 7fca09602640 1 -- 172.21.15.16:0/390548978 >> 172.21.15.16:0/390548978 conn(0x7fca040fbfe0 msgr2=0x7fca040fe400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:21:18.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.389+0000 7fca09602640 1 -- 172.21.15.16:0/390548978 shutdown_connections 2024-03-28T22:21:18.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.389+0000 7fca09602640 1 -- 172.21.15.16:0/390548978 wait complete. 2024-03-28T22:21:18.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.390+0000 7fca09602640 1 Processor -- start 2024-03-28T22:21:18.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.390+0000 7fca09602640 1 -- start start 2024-03-28T22:21:18.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.390+0000 7fca09602640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca04100530 0x7fca04198cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:18.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.390+0000 7fca09602640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca04100ef0 0x7fca04199230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:18.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.390+0000 7fca09602640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca041065f0 0x7fca041a3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:18.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.390+0000 7fca09602640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fca041108d0 con 0x7fca04100530 2024-03-28T22:21:18.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.390+0000 7fca09602640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fca04110750 con 0x7fca04100ef0 2024-03-28T22:21:18.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.390+0000 7fca09602640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fca04110a50 con 0x7fca041065f0 2024-03-28T22:21:18.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.390+0000 7fca02ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca04100530 0x7fca04198cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:18.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.390+0000 7fca02ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca04100530 0x7fca04198cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41290/0 (socket says 172.21.15.16:41290) 2024-03-28T22:21:18.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.390+0000 7fca02ffd640 1 -- 172.21.15.16:0/1714932605 learned_addr learned my addr 172.21.15.16:0/1714932605 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:21:18.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.390+0000 7fca027fc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca04100ef0 0x7fca04199230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:18.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.390+0000 7fca037fe640 1 --2- 172.21.15.16:0/1714932605 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca041065f0 0x7fca041a3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:18.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.390+0000 7fca02ffd640 1 -- 172.21.15.16:0/1714932605 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca041065f0 msgr2=0x7fca041a3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:18.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.390+0000 7fca02ffd640 1 --2- 172.21.15.16:0/1714932605 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca041065f0 0x7fca041a3d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:18.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.390+0000 7fca02ffd640 1 -- 172.21.15.16:0/1714932605 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca04100ef0 msgr2=0x7fca04199230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:18.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.390+0000 7fca02ffd640 1 --2- 172.21.15.16:0/1714932605 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca04100ef0 0x7fca04199230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:18.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.390+0000 7fca02ffd640 1 -- 172.21.15.16:0/1714932605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca041a42b0 con 0x7fca04100530 2024-03-28T22:21:18.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.390+0000 7fca027fc640 1 --2- 172.21.15.16:0/1714932605 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca04100ef0 0x7fca04199230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:21:18.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.391+0000 7fca02ffd640 1 --2- 172.21.15.16:0/1714932605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca04100530 0x7fca04198cf0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4005940 tx=0x7fc9f4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:21:18.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.391+0000 7fc9ebfff640 1 -- 172.21.15.16:0/1714932605 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f4041070 con 0x7fca04100530 2024-03-28T22:21:18.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.391+0000 7fc9ebfff640 1 -- 172.21.15.16:0/1714932605 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9f40054e0 con 0x7fca04100530 2024-03-28T22:21:18.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.391+0000 7fca09602640 1 -- 172.21.15.16:0/1714932605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca041a4480 con 0x7fca04100530 2024-03-28T22:21:18.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.391+0000 7fc9ebfff640 1 -- 172.21.15.16:0/1714932605 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f4003d00 con 0x7fca04100530 2024-03-28T22:21:18.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.391+0000 7fca09602640 1 -- 172.21.15.16:0/1714932605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca041a48e0 con 0x7fca04100530 2024-03-28T22:21:18.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.392+0000 7fca09602640 1 -- 172.21.15.16:0/1714932605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca0404f520 con 0x7fca04100530 2024-03-28T22:21:18.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.393+0000 7fc9ebfff640 1 -- 172.21.15.16:0/1714932605 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7fc9f4044070 con 0x7fca04100530 2024-03-28T22:21:18.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.393+0000 7fc9ebfff640 1 --2- 172.21.15.16:0/1714932605 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fc9ec0a1ee0 0x7fc9ec0a43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:18.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.394+0000 7fca027fc640 1 --2- 172.21.15.16:0/1714932605 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fc9ec0a1ee0 0x7fc9ec0a43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:18.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.394+0000 7fc9ebfff640 1 -- 172.21.15.16:0/1714932605 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8527+0+0 (secure 0 0 0) 0x7fc9f40effa0 con 0x7fca04100530 2024-03-28T22:21:18.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.394+0000 7fca027fc640 1 --2- 172.21.15.16:0/1714932605 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fc9ec0a1ee0 0x7fc9ec0a43a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fca04069540 tx=0x7fc9f8005f90 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:21:18.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.397+0000 7fc9ebfff640 1 -- 172.21.15.16:0/1714932605 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc9f40bd1a0 con 0x7fca04100530 2024-03-28T22:21:18.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.580+0000 7fca09602640 1 -- 172.21.15.16:0/1714932605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fca0402d170 con 0x7fca04100530 2024-03-28T22:21:18.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.582+0000 7fc9ebfff640 1 -- 172.21.15.16:0/1714932605 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) ==== 74+0+22967 (secure 0 0 0) 0x7fc9f40bcaa0 con 0x7fca04100530 2024-03-28T22:21:18.583 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:21:18.584 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":99,"fsid":"798d9884-ed4e-11ee-95d0-87774f69a715","created":"2024-03-28T22:00:27.216099+0000","modified":"2024-03-28T22:21:17.467135+0000","last_up_change":"2024-03-28T22:07:40.251141+0000","last_in_change":"2024-03-28T22:07:19.456451+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:03:59.327832+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:08:28.123529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:08:28.919037+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":98,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"46e7c8f8-ecc8-43d6-ae53-0655728f3240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6803","nonce":1589692483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6805","nonce":1589692483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6809","nonce":1589692483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6807","nonce":1589692483}]},"public_addr":"172.21.15.16:6803/1589692483","cluster_addr":"172.21.15.16:6805/1589692483","heartbeat_back_addr":"172.21.15.16:6809/1589692483","heartbeat_front_addr":"172.21.15.16:6807/1589692483","state":["exists","up"]},{"osd":1,"uuid":"af6d9c22-2830-4ce2-a547-011df0c6cbad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6801","nonce":795531234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6803","nonce":795531234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6807","nonce":795531234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6805","nonce":795531234}]},"public_addr":"172.21.15.53:6801/795531234","cluster_addr":"172.21.15.53:6803/795531234","heartbeat_back_addr":"172.21.15.53:6807/795531234","heartbeat_front_addr":"172.21.15.53:6805/795531234","state":["exists","up"]},{"osd":2,"uuid":"b10adca9-110c-4f87-85e0-b7542709d31a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6801","nonce":2214131086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6803","nonce":2214131086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6807","nonce":2214131086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6805","nonce":2214131086}]},"public_addr":"172.21.15.63:6801/2214131086","cluster_addr":"172.21.15.63:6803/2214131086","heartbeat_back_addr":"172.21.15.63:6807/2214131086","heartbeat_front_addr":"172.21.15.63:6805/2214131086","state":["exists","up"]},{"osd":3,"uuid":"c83cf4bb-395a-4786-abf7-74ca93a33eb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6811","nonce":13176789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6813","nonce":13176789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6817","nonce":13176789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6815","nonce":13176789}]},"public_addr":"172.21.15.16:6811/13176789","cluster_addr":"172.21.15.16:6813/13176789","heartbeat_back_addr":"172.21.15.16:6817/13176789","heartbeat_front_addr":"172.21.15.16:6815/13176789","state":["exists","up"]},{"osd":4,"uuid":"941b3e54-47d6-4e85-8069-e730c3fa8eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6809","nonce":772161849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6811","nonce":772161849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6815","nonce":772161849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6813","nonce":772161849}]},"public_addr":"172.21.15.53:6809/772161849","cluster_addr":"172.21.15.53:6811/772161849","heartbeat_back_addr":"172.21.15.53:6815/772161849","heartbeat_front_addr":"172.21.15.53:6813/772161849","state":["exists","up"]},{"osd":5,"uuid":"51d89826-1d60-40fc-8fd8-09bfe1ab4125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6809","nonce":1602554852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6811","nonce":1602554852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6815","nonce":1602554852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6813","nonce":1602554852}]},"public_addr":"172.21.15.63:6809/1602554852","cluster_addr":"172.21.15.63:6811/1602554852","heartbeat_back_addr":"172.21.15.63:6815/1602554852","heartbeat_front_addr":"172.21.15.63:6813/1602554852","state":["exists","up"]},{"osd":6,"uuid":"978efe5d-2c55-4ff8-8a1e-9f891388945e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6819","nonce":4290459224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6821","nonce":4290459224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6825","nonce":4290459224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6823","nonce":4290459224}]},"public_addr":"172.21.15.16:6819/4290459224","cluster_addr":"172.21.15.16:6821/4290459224","heartbeat_back_addr":"172.21.15.16:6825/4290459224","heartbeat_front_addr":"172.21.15.16:6823/4290459224","state":["exists","up"]},{"osd":7,"uuid":"50d489cb-dbb8-48df-b868-b29edf12e04d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6817","nonce":1056467525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6819","nonce":1056467525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6823","nonce":1056467525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6821","nonce":1056467525}]},"public_addr":"172.21.15.53:6817/1056467525","cluster_addr":"172.21.15.53:6819/1056467525","heartbeat_back_addr":"172.21.15.53:6823/1056467525","heartbeat_front_addr":"172.21.15.53:6821/1056467525","state":["exists","up"]},{"osd":8,"uuid":"49b39414-cc9c-4057-ae3f-cc109cd979da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6817","nonce":2918516794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6819","nonce":2918516794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6823","nonce":2918516794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6821","nonce":2918516794}]},"public_addr":"172.21.15.63:6817/2918516794","cluster_addr":"172.21.15.63:6819/2918516794","heartbeat_back_addr":"172.21.15.63:6823/2918516794","heartbeat_front_addr":"172.21.15.63:6821/2918516794","state":["exists","up"]},{"osd":9,"uuid":"4c873e51-b783-4eb6-83a3-fee31722d60b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6827","nonce":1585624367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6829","nonce":1585624367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6833","nonce":1585624367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6831","nonce":1585624367}]},"public_addr":"172.21.15.16:6827/1585624367","cluster_addr":"172.21.15.16:6829/1585624367","heartbeat_back_addr":"172.21.15.16:6833/1585624367","heartbeat_front_addr":"172.21.15.16:6831/1585624367","state":["exists","up"]},{"osd":10,"uuid":"085a78bf-e2ad-481a-85a3-71d1fc549a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6825","nonce":3357446484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6827","nonce":3357446484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6831","nonce":3357446484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6829","nonce":3357446484}]},"public_addr":"172.21.15.53:6825/3357446484","cluster_addr":"172.21.15.53:6827/3357446484","heartbeat_back_addr":"172.21.15.53:6831/3357446484","heartbeat_front_addr":"172.21.15.53:6829/3357446484","state":["exists","up"]},{"osd":11,"uuid":"13302aa3-08b9-4639-9042-27d850b4d5f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6825","nonce":2167866570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6827","nonce":2167866570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6831","nonce":2167866570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6829","nonce":2167866570}]},"public_addr":"172.21.15.63:6825/2167866570","cluster_addr":"172.21.15.63:6827/2167866570","heartbeat_back_addr":"172.21.15.63:6831/2167866570","heartbeat_front_addr":"172.21.15.63:6829/2167866570","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:07.450323+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:31.363172+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:55.295021+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:19.794760+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:44.063492+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:08.195064+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:33.655328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:57.639414+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:22.402394+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:47.462050+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:12.226307+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:37.495625+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3631746518":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6801/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/2311413235":"2024-03-29T22:10:34.301217+0000","172.21.15.16:0/3116652786":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/317557779":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/1062264551":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/996598130":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6801/21986004":"2024-03-29T22:10:34.301217+0000","172.21.15.16:6801/1832422779":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/3821408150":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/3070600736":"2024-03-29T22:10:34.301217+0000","172.21.15.16:0/2117914406":"2024-03-29T22:10:34.301217+0000","172.21.15.16:0/4029580607":"2024-03-29T22:10:34.301217+0000","172.21.15.16:6800/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/2787995271":"2024-03-29T22:10:34.301217+0000","172.21.15.16:6800/1832422779":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6800/21986004":"2024-03-29T22:10:34.301217+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:21:18.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.584+0000 7fca09602640 1 -- 172.21.15.16:0/1714932605 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fc9ec0a1ee0 msgr2=0x7fc9ec0a43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:18.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.584+0000 7fca09602640 1 --2- 172.21.15.16:0/1714932605 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fc9ec0a1ee0 0x7fc9ec0a43a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fca04069540 tx=0x7fc9f8005f90 comp rx=0 tx=0).stop 2024-03-28T22:21:18.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.584+0000 7fca09602640 1 -- 172.21.15.16:0/1714932605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca04100530 msgr2=0x7fca04198cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:18.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.584+0000 7fca09602640 1 --2- 172.21.15.16:0/1714932605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca04100530 0x7fca04198cf0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4005940 tx=0x7fc9f4004270 comp rx=0 tx=0).stop 2024-03-28T22:21:18.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.584+0000 7fca09602640 1 -- 172.21.15.16:0/1714932605 shutdown_connections 2024-03-28T22:21:18.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.584+0000 7fca09602640 1 --2- 172.21.15.16:0/1714932605 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fc9ec0a1ee0 0x7fc9ec0a43a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:18.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.584+0000 7fca09602640 1 --2- 172.21.15.16:0/1714932605 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca041065f0 0x7fca041a3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:18.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.584+0000 7fca09602640 1 --2- 172.21.15.16:0/1714932605 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca04100ef0 0x7fca04199230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:18.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.584+0000 7fca09602640 1 --2- 172.21.15.16:0/1714932605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca04100530 0x7fca04198cf0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:18.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.584+0000 7fca09602640 1 -- 172.21.15.16:0/1714932605 >> 172.21.15.16:0/1714932605 conn(0x7fca040fbfe0 msgr2=0x7fca040fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:21:18.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.584+0000 7fca09602640 1 -- 172.21.15.16:0/1714932605 shutdown_connections 2024-03-28T22:21:18.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.584+0000 7fca09602640 1 -- 172.21.15.16:0/1714932605 wait complete. 2024-03-28T22:21:18.596 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T22:03:59.327832+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T22:08:28.123529+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T22:08:28.919037+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '98', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 98, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T22:21:18.597 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T22:21:18.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.702+0000 7fb016474640 1 -- 172.21.15.16:0/2996149109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0101076f0 msgr2=0x7fb010107ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:18.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.702+0000 7fb016474640 1 --2- 172.21.15.16:0/2996149109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0101076f0 0x7fb010107ad0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fb000009930 tx=0x7fb00002bc30 comp rx=0 tx=0).stop 2024-03-28T22:21:18.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.702+0000 7fb016474640 1 -- 172.21.15.16:0/2996149109 shutdown_connections 2024-03-28T22:21:18.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.702+0000 7fb016474640 1 --2- 172.21.15.16:0/2996149109 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0101020a0 0x7fb01010e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:18.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.702+0000 7fb016474640 1 --2- 172.21.15.16:0/2996149109 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0101016e0 0x7fb010101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:18.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.702+0000 7fb016474640 1 --2- 172.21.15.16:0/2996149109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0101076f0 0x7fb010107ad0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:18.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.702+0000 7fb016474640 1 -- 172.21.15.16:0/2996149109 >> 172.21.15.16:0/2996149109 conn(0x7fb0100fd1b0 msgr2=0x7fb0100ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:21:18.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.702+0000 7fb016474640 1 -- 172.21.15.16:0/2996149109 shutdown_connections 2024-03-28T22:21:18.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.702+0000 7fb016474640 1 -- 172.21.15.16:0/2996149109 wait complete. 2024-03-28T22:21:18.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.702+0000 7fb016474640 1 Processor -- start 2024-03-28T22:21:18.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.702+0000 7fb016474640 1 -- start start 2024-03-28T22:21:18.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.703+0000 7fb016474640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0101016e0 0x7fb010198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:18.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.703+0000 7fb016474640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0101020a0 0x7fb0101985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:18.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.703+0000 7fb016474640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0101076f0 0x7fb01019c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:18.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.703+0000 7fb016474640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb010111b00 con 0x7fb0101016e0 2024-03-28T22:21:18.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.703+0000 7fb016474640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb010111980 con 0x7fb0101020a0 2024-03-28T22:21:18.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.703+0000 7fb016474640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb010111c80 con 0x7fb0101076f0 2024-03-28T22:21:18.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.703+0000 7fb00ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0101016e0 0x7fb010198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:18.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.703+0000 7fb00ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0101016e0 0x7fb010198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41296/0 (socket says 172.21.15.16:41296) 2024-03-28T22:21:18.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.703+0000 7fb00ffff640 1 -- 172.21.15.16:0/2875686944 learned_addr learned my addr 172.21.15.16:0/2875686944 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:21:18.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.703+0000 7fb00f7fe640 1 --2- 172.21.15.16:0/2875686944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0101020a0 0x7fb0101985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:18.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.703+0000 7fb014a11640 1 --2- 172.21.15.16:0/2875686944 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0101076f0 0x7fb01019c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:18.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.703+0000 7fb00ffff640 1 -- 172.21.15.16:0/2875686944 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0101076f0 msgr2=0x7fb01019c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:18.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.703+0000 7fb00ffff640 1 --2- 172.21.15.16:0/2875686944 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0101076f0 0x7fb01019c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:18.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.703+0000 7fb00ffff640 1 -- 172.21.15.16:0/2875686944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0101020a0 msgr2=0x7fb0101985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:18.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.703+0000 7fb00ffff640 1 --2- 172.21.15.16:0/2875686944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0101020a0 0x7fb0101985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:18.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.703+0000 7fb00ffff640 1 -- 172.21.15.16:0/2875686944 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb01019d050 con 0x7fb0101016e0 2024-03-28T22:21:18.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.703+0000 7fb00ffff640 1 --2- 172.21.15.16:0/2875686944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0101016e0 0x7fb010198060 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fb000009230 tx=0x7fb00002cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:21:18.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.703+0000 7fb00d7fa640 1 -- 172.21.15.16:0/2875686944 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb000041070 con 0x7fb0101016e0 2024-03-28T22:21:18.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.703+0000 7fb016474640 1 -- 172.21.15.16:0/2875686944 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb01019d2e0 con 0x7fb0101016e0 2024-03-28T22:21:18.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.703+0000 7fb00d7fa640 1 -- 172.21.15.16:0/2875686944 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb000004450 con 0x7fb0101016e0 2024-03-28T22:21:18.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.703+0000 7fb00d7fa640 1 -- 172.21.15.16:0/2875686944 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb000003e50 con 0x7fb0101016e0 2024-03-28T22:21:18.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.703+0000 7fb016474640 1 -- 172.21.15.16:0/2875686944 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0101a4bd0 con 0x7fb0101016e0 2024-03-28T22:21:18.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.704+0000 7fb016474640 1 -- 172.21.15.16:0/2875686944 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb010068a20 con 0x7fb0101016e0 2024-03-28T22:21:18.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.705+0000 7fb00d7fa640 1 -- 172.21.15.16:0/2875686944 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7fb000044070 con 0x7fb0101016e0 2024-03-28T22:21:18.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.706+0000 7fb00d7fa640 1 --2- 172.21.15.16:0/2875686944 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7faff80a2090 0x7faff80a4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:18.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.706+0000 7fb00d7fa640 1 -- 172.21.15.16:0/2875686944 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8527+0+0 (secure 0 0 0) 0x7fb0000efe60 con 0x7fb0101016e0 2024-03-28T22:21:18.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.706+0000 7fb00f7fe640 1 --2- 172.21.15.16:0/2875686944 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7faff80a2090 0x7faff80a4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:18.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.707+0000 7fb00f7fe640 1 --2- 172.21.15.16:0/2875686944 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7faff80a2090 0x7faff80a4550 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb0100ff850 tx=0x7fb004009110 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:21:18.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.710+0000 7fb00d7fa640 1 -- 172.21.15.16:0/2875686944 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb0000bd060 con 0x7fb0101016e0 2024-03-28T22:21:18.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.892+0000 7fb016474640 1 -- 172.21.15.16:0/2875686944 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb010109c50 con 0x7fb0101016e0 2024-03-28T22:21:18.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.892+0000 7fb00d7fa640 1 -- 172.21.15.16:0/2875686944 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v99) ==== 93+0+10 (secure 0 0 0) 0x7fb0000bc960 con 0x7fb0101016e0 2024-03-28T22:21:18.894 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-28T22:21:18.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.895+0000 7fb016474640 1 -- 172.21.15.16:0/2875686944 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7faff80a2090 msgr2=0x7faff80a4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:18.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.895+0000 7fb016474640 1 --2- 172.21.15.16:0/2875686944 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7faff80a2090 0x7faff80a4550 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb0100ff850 tx=0x7fb004009110 comp rx=0 tx=0).stop 2024-03-28T22:21:18.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.895+0000 7fb016474640 1 -- 172.21.15.16:0/2875686944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0101016e0 msgr2=0x7fb010198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:18.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.895+0000 7fb016474640 1 --2- 172.21.15.16:0/2875686944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0101016e0 0x7fb010198060 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fb000009230 tx=0x7fb00002cba0 comp rx=0 tx=0).stop 2024-03-28T22:21:18.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.895+0000 7fb016474640 1 -- 172.21.15.16:0/2875686944 shutdown_connections 2024-03-28T22:21:18.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.895+0000 7fb016474640 1 --2- 172.21.15.16:0/2875686944 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7faff80a2090 0x7faff80a4550 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:18.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.895+0000 7fb016474640 1 --2- 172.21.15.16:0/2875686944 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0101076f0 0x7fb01019c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:18.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.895+0000 7fb016474640 1 --2- 172.21.15.16:0/2875686944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0101020a0 0x7fb0101985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:18.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.895+0000 7fb016474640 1 --2- 172.21.15.16:0/2875686944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0101016e0 0x7fb010198060 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:18.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.895+0000 7fb016474640 1 -- 172.21.15.16:0/2875686944 >> 172.21.15.16:0/2875686944 conn(0x7fb0100fd1b0 msgr2=0x7fb0100fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:21:18.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.895+0000 7fb016474640 1 -- 172.21.15.16:0/2875686944 shutdown_connections 2024-03-28T22:21:18.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:18.895+0000 7fb016474640 1 -- 172.21.15.16:0/2875686944 wait complete. 2024-03-28T22:21:18.905 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:18 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/1714932605' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:21:18.906 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T22:21:18.930 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:18 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/1714932605' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:21:19.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.011+0000 7f235c055640 1 -- 172.21.15.16:0/3091746307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2354102970 msgr2=0x7f2354102d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:19.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.011+0000 7f235c055640 1 --2- 172.21.15.16:0/3091746307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2354102970 0x7f2354102d50 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f2344009900 tx=0x7f234402bcd0 comp rx=0 tx=0).stop 2024-03-28T22:21:19.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.011+0000 7f235c055640 1 -- 172.21.15.16:0/3091746307 shutdown_connections 2024-03-28T22:21:19.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.011+0000 7f235c055640 1 --2- 172.21.15.16:0/3091746307 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f235410bd40 0x7f235410e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:19.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.011+0000 7f235c055640 1 --2- 172.21.15.16:0/3091746307 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2354103290 0x7f235410b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:19.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.011+0000 7f235c055640 1 --2- 172.21.15.16:0/3091746307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2354102970 0x7f2354102d50 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:19.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.011+0000 7f235c055640 1 -- 172.21.15.16:0/3091746307 >> 172.21.15.16:0/3091746307 conn(0x7f2354077800 msgr2=0x7f2354077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:21:19.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.011+0000 7f235c055640 1 -- 172.21.15.16:0/3091746307 shutdown_connections 2024-03-28T22:21:19.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.011+0000 7f235c055640 1 -- 172.21.15.16:0/3091746307 wait complete. 2024-03-28T22:21:19.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.011+0000 7f235c055640 1 Processor -- start 2024-03-28T22:21:19.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.011+0000 7f235c055640 1 -- start start 2024-03-28T22:21:19.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.012+0000 7f235c055640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2354102970 0x7f23541980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:19.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.012+0000 7f235c055640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2354103290 0x7f2354198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:19.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.012+0000 7f235c055640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f235410bd40 0x7f235419c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:19.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.012+0000 7f235c055640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2354111b60 con 0x7f235410bd40 2024-03-28T22:21:19.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.012+0000 7f235c055640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f23541119e0 con 0x7f2354102970 2024-03-28T22:21:19.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.012+0000 7f235c055640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2354111ce0 con 0x7f2354103290 2024-03-28T22:21:19.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.012+0000 7f235a5f2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f235410bd40 0x7f235419c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:19.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.012+0000 7f2359df1640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2354102970 0x7f23541980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:19.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.012+0000 7f2359df1640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2354102970 0x7f23541980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.16:52154/0 (socket says 172.21.15.16:52154) 2024-03-28T22:21:19.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.012+0000 7f235a5f2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f235410bd40 0x7f235419c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41306/0 (socket says 172.21.15.16:41306) 2024-03-28T22:21:19.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.012+0000 7f2359df1640 1 -- 172.21.15.16:0/1891577488 learned_addr learned my addr 172.21.15.16:0/1891577488 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:21:19.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.012+0000 7f23595f0640 1 --2- 172.21.15.16:0/1891577488 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2354103290 0x7f2354198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:19.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.012+0000 7f235a5f2640 1 -- 172.21.15.16:0/1891577488 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2354103290 msgr2=0x7f2354198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:19.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.012+0000 7f235a5f2640 1 --2- 172.21.15.16:0/1891577488 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2354103290 0x7f2354198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:19.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.012+0000 7f235a5f2640 1 -- 172.21.15.16:0/1891577488 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2354102970 msgr2=0x7f23541980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:19.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.012+0000 7f235a5f2640 1 --2- 172.21.15.16:0/1891577488 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2354102970 0x7f23541980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:19.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.012+0000 7f235a5f2640 1 -- 172.21.15.16:0/1891577488 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f235419d0d0 con 0x7f235410bd40 2024-03-28T22:21:19.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.012+0000 7f2359df1640 1 --2- 172.21.15.16:0/1891577488 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2354102970 0x7f23541980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:21:19.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.012+0000 7f235a5f2640 1 --2- 172.21.15.16:0/1891577488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f235410bd40 0x7f235419c9a0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f234c00be20 tx=0x7f234c00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:21:19.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.012+0000 7f234affd640 1 -- 172.21.15.16:0/1891577488 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f234c019070 con 0x7f235410bd40 2024-03-28T22:21:19.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.012+0000 7f234affd640 1 -- 172.21.15.16:0/1891577488 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f234c00cdf0 con 0x7f235410bd40 2024-03-28T22:21:19.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.012+0000 7f235c055640 1 -- 172.21.15.16:0/1891577488 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f235419d3c0 con 0x7f235410bd40 2024-03-28T22:21:19.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.012+0000 7f234affd640 1 -- 172.21.15.16:0/1891577488 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f234c012920 con 0x7f235410bd40 2024-03-28T22:21:19.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.012+0000 7f235c055640 1 -- 172.21.15.16:0/1891577488 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23541a4c50 con 0x7f235410bd40 2024-03-28T22:21:19.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.014+0000 7f235c055640 1 -- 172.21.15.16:0/1891577488 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23540685d0 con 0x7f235410bd40 2024-03-28T22:21:19.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.015+0000 7f234affd640 1 -- 172.21.15.16:0/1891577488 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f234c01c070 con 0x7f235410bd40 2024-03-28T22:21:19.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.015+0000 7f234affd640 1 --2- 172.21.15.16:0/1891577488 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f23400a1ef0 0x7f23400a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:19.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.015+0000 7f2359df1640 1 --2- 172.21.15.16:0/1891577488 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f23400a1ef0 0x7f23400a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:19.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.015+0000 7f234affd640 1 -- 172.21.15.16:0/1891577488 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8527+0+0 (secure 0 0 0) 0x7f234c0cde00 con 0x7f235410bd40 2024-03-28T22:21:19.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.016+0000 7f2359df1640 1 --2- 172.21.15.16:0/1891577488 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f23400a1ef0 0x7f23400a43b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f23440096c0 tx=0x7f2344005a20 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:21:19.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.019+0000 7f234affd640 1 -- 172.21.15.16:0/1891577488 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f234c09a910 con 0x7f235410bd40 2024-03-28T22:21:19.046 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:18 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/1714932605' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:21:19.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.200+0000 7f235c055640 1 -- 172.21.15.16:0/1891577488 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2354106e90 con 0x7f235410bd40 2024-03-28T22:21:19.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.200+0000 7f234affd640 1 -- 172.21.15.16:0/1891577488 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v99) ==== 104+0+11 (secure 0 0 0) 0x7f234c09e9b0 con 0x7f235410bd40 2024-03-28T22:21:19.202 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-03-28T22:21:19.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.202+0000 7f235c055640 1 -- 172.21.15.16:0/1891577488 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f23400a1ef0 msgr2=0x7f23400a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:19.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.202+0000 7f235c055640 1 --2- 172.21.15.16:0/1891577488 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f23400a1ef0 0x7f23400a43b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f23440096c0 tx=0x7f2344005a20 comp rx=0 tx=0).stop 2024-03-28T22:21:19.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.202+0000 7f235c055640 1 -- 172.21.15.16:0/1891577488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f235410bd40 msgr2=0x7f235419c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:19.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.202+0000 7f235c055640 1 --2- 172.21.15.16:0/1891577488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f235410bd40 0x7f235419c9a0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f234c00be20 tx=0x7f234c00bf10 comp rx=0 tx=0).stop 2024-03-28T22:21:19.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.202+0000 7f235c055640 1 -- 172.21.15.16:0/1891577488 shutdown_connections 2024-03-28T22:21:19.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.202+0000 7f235c055640 1 --2- 172.21.15.16:0/1891577488 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f23400a1ef0 0x7f23400a43b0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:19.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.202+0000 7f235c055640 1 --2- 172.21.15.16:0/1891577488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f235410bd40 0x7f235419c9a0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:19.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.202+0000 7f235c055640 1 --2- 172.21.15.16:0/1891577488 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2354103290 0x7f2354198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:19.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.202+0000 7f235c055640 1 --2- 172.21.15.16:0/1891577488 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2354102970 0x7f23541980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:19.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.202+0000 7f235c055640 1 -- 172.21.15.16:0/1891577488 >> 172.21.15.16:0/1891577488 conn(0x7f2354077800 msgr2=0x7f23540fe9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:21:19.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.202+0000 7f235c055640 1 -- 172.21.15.16:0/1891577488 shutdown_connections 2024-03-28T22:21:19.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.202+0000 7f235c055640 1 -- 172.21.15.16:0/1891577488 wait complete. 2024-03-28T22:21:19.214 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T22:21:19.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.318+0000 7f2f6264e640 1 -- 172.21.15.16:0/2504887652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f5c069480 msgr2=0x7f2f5c10e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:19.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.318+0000 7f2f6264e640 1 --2- 172.21.15.16:0/2504887652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f5c069480 0x7f2f5c10e5b0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f2f54009cd0 tx=0x7f2f5402c0f0 comp rx=0 tx=0).stop 2024-03-28T22:21:19.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.319+0000 7f2f6264e640 1 -- 172.21.15.16:0/2504887652 shutdown_connections 2024-03-28T22:21:19.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.319+0000 7f2f6264e640 1 --2- 172.21.15.16:0/2504887652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f5c069480 0x7f2f5c10e5b0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:19.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.319+0000 7f2f6264e640 1 --2- 172.21.15.16:0/2504887652 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f5c068ac0 0x7f2f5c068f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:19.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.319+0000 7f2f6264e640 1 --2- 172.21.15.16:0/2504887652 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f5c1077a0 0x7f2f5c107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:19.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.319+0000 7f2f6264e640 1 -- 172.21.15.16:0/2504887652 >> 172.21.15.16:0/2504887652 conn(0x7f2f5c077750 msgr2=0x7f2f5c077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:21:19.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.319+0000 7f2f6264e640 1 -- 172.21.15.16:0/2504887652 shutdown_connections 2024-03-28T22:21:19.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.319+0000 7f2f6264e640 1 -- 172.21.15.16:0/2504887652 wait complete. 2024-03-28T22:21:19.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.319+0000 7f2f6264e640 1 Processor -- start 2024-03-28T22:21:19.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.319+0000 7f2f6264e640 1 -- start start 2024-03-28T22:21:19.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.319+0000 7f2f6264e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f5c068ac0 0x7f2f5c100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:19.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.319+0000 7f2f6264e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f5c069480 0x7f2f5c100b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:19.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.319+0000 7f2f6264e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f5c1077a0 0x7f2f5c101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:19.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.319+0000 7f2f6264e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2f5c111af0 con 0x7f2f5c068ac0 2024-03-28T22:21:19.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.319+0000 7f2f6264e640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f2f5c111970 con 0x7f2f5c1077a0 2024-03-28T22:21:19.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.319+0000 7f2f6264e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2f5c111c70 con 0x7f2f5c069480 2024-03-28T22:21:19.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.320+0000 7f2f5bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f5c068ac0 0x7f2f5c100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:19.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.320+0000 7f2f5bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f5c068ac0 0x7f2f5c100610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41320/0 (socket says 172.21.15.16:41320) 2024-03-28T22:21:19.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.320+0000 7f2f5bfff640 1 -- 172.21.15.16:0/86246643 learned_addr learned my addr 172.21.15.16:0/86246643 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:21:19.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.320+0000 7f2f60beb640 1 --2- 172.21.15.16:0/86246643 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f5c1077a0 0x7f2f5c101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:19.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.320+0000 7f2f5b7fe640 1 --2- 172.21.15.16:0/86246643 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f5c069480 0x7f2f5c100b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:19.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.320+0000 7f2f5bfff640 1 -- 172.21.15.16:0/86246643 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f5c069480 msgr2=0x7f2f5c100b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:19.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.320+0000 7f2f5bfff640 1 --2- 172.21.15.16:0/86246643 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f5c069480 0x7f2f5c100b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:19.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.320+0000 7f2f5bfff640 1 -- 172.21.15.16:0/86246643 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f5c1077a0 msgr2=0x7f2f5c101090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:19.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.320+0000 7f2f5bfff640 1 --2- 172.21.15.16:0/86246643 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f5c1077a0 0x7f2f5c101090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:19.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.320+0000 7f2f5bfff640 1 -- 172.21.15.16:0/86246643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f5c1a63f0 con 0x7f2f5c068ac0 2024-03-28T22:21:19.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.320+0000 7f2f5bfff640 1 --2- 172.21.15.16:0/86246643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f5c068ac0 0x7f2f5c100610 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c00e9f0 tx=0x7f2f4c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:21:19.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.320+0000 7f2f597fa640 1 -- 172.21.15.16:0/86246643 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f4c00cd80 con 0x7f2f5c068ac0 2024-03-28T22:21:19.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.320+0000 7f2f597fa640 1 -- 172.21.15.16:0/86246643 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f4c010910 con 0x7f2f5c068ac0 2024-03-28T22:21:19.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.320+0000 7f2f6264e640 1 -- 172.21.15.16:0/86246643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f5c1a66e0 con 0x7f2f5c068ac0 2024-03-28T22:21:19.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.320+0000 7f2f597fa640 1 -- 172.21.15.16:0/86246643 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f4c018990 con 0x7f2f5c068ac0 2024-03-28T22:21:19.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.320+0000 7f2f6264e640 1 -- 172.21.15.16:0/86246643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f5c1a6c30 con 0x7f2f5c068ac0 2024-03-28T22:21:19.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.321+0000 7f2f6264e640 1 -- 172.21.15.16:0/86246643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f5c102e20 con 0x7f2f5c068ac0 2024-03-28T22:21:19.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.322+0000 7f2f597fa640 1 -- 172.21.15.16:0/86246643 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f2f4c010ac0 con 0x7f2f5c068ac0 2024-03-28T22:21:19.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.323+0000 7f2f597fa640 1 --2- 172.21.15.16:0/86246643 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f2f440a1ef0 0x7f2f440a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:19.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.323+0000 7f2f5b7fe640 1 --2- 172.21.15.16:0/86246643 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f2f440a1ef0 0x7f2f440a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:19.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.323+0000 7f2f597fa640 1 -- 172.21.15.16:0/86246643 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8527+0+0 (secure 0 0 0) 0x7f2f4c0ce4f0 con 0x7f2f5c068ac0 2024-03-28T22:21:19.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.323+0000 7f2f5b7fe640 1 --2- 172.21.15.16:0/86246643 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f2f440a1ef0 0x7f2f440a43b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c1020a0 tx=0x7f2f50005f90 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:21:19.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.327+0000 7f2f597fa640 1 -- 172.21.15.16:0/86246643 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2f4c09b5e0 con 0x7f2f5c068ac0 2024-03-28T22:21:19.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.508+0000 7f2f6264e640 1 -- 172.21.15.16:0/86246643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2f5c1a6e90 con 0x7f2f5c068ac0 2024-03-28T22:21:19.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.508+0000 7f2f597fa640 1 -- 172.21.15.16:0/86246643 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v99) ==== 100+0+11 (secure 0 0 0) 0x7f2f4c09aee0 con 0x7f2f5c068ac0 2024-03-28T22:21:19.510 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-03-28T22:21:19.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.510+0000 7f2f6264e640 1 -- 172.21.15.16:0/86246643 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f2f440a1ef0 msgr2=0x7f2f440a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:19.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.510+0000 7f2f6264e640 1 --2- 172.21.15.16:0/86246643 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f2f440a1ef0 0x7f2f440a43b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c1020a0 tx=0x7f2f50005f90 comp rx=0 tx=0).stop 2024-03-28T22:21:19.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.510+0000 7f2f6264e640 1 -- 172.21.15.16:0/86246643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f5c068ac0 msgr2=0x7f2f5c100610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:19.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.510+0000 7f2f6264e640 1 --2- 172.21.15.16:0/86246643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f5c068ac0 0x7f2f5c100610 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c00e9f0 tx=0x7f2f4c00eeb0 comp rx=0 tx=0).stop 2024-03-28T22:21:19.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.510+0000 7f2f6264e640 1 -- 172.21.15.16:0/86246643 shutdown_connections 2024-03-28T22:21:19.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.510+0000 7f2f6264e640 1 --2- 172.21.15.16:0/86246643 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f2f440a1ef0 0x7f2f440a43b0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:19.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.510+0000 7f2f6264e640 1 --2- 172.21.15.16:0/86246643 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f5c1077a0 0x7f2f5c101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:19.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.510+0000 7f2f6264e640 1 --2- 172.21.15.16:0/86246643 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f5c069480 0x7f2f5c100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:19.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.510+0000 7f2f6264e640 1 --2- 172.21.15.16:0/86246643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f5c068ac0 0x7f2f5c100610 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:19.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.510+0000 7f2f6264e640 1 -- 172.21.15.16:0/86246643 >> 172.21.15.16:0/86246643 conn(0x7f2f5c077750 msgr2=0x7f2f5c0fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:21:19.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.511+0000 7f2f6264e640 1 -- 172.21.15.16:0/86246643 shutdown_connections 2024-03-28T22:21:19.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.511+0000 7f2f6264e640 1 -- 172.21.15.16:0/86246643 wait complete. 2024-03-28T22:21:19.522 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-28T22:21:19.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.625+0000 7f7b4c8c0640 1 -- 172.21.15.16:0/2008878260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b481005f0 msgr2=0x7f7b48100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:19.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.625+0000 7f7b4c8c0640 1 --2- 172.21.15.16:0/2008878260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b481005f0 0x7f7b48100a70 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c009930 tx=0x7f7b3c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:21:19.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.626+0000 7f7b4c8c0640 1 -- 172.21.15.16:0/2008878260 shutdown_connections 2024-03-28T22:21:19.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.626+0000 7f7b4c8c0640 1 --2- 172.21.15.16:0/2008878260 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b48100fb0 0x7f7b4810d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:19.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.626+0000 7f7b4c8c0640 1 --2- 172.21.15.16:0/2008878260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b481005f0 0x7f7b48100a70 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:19.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.626+0000 7f7b4c8c0640 1 --2- 172.21.15.16:0/2008878260 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b48106690 0x7f7b48106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:19.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.626+0000 7f7b4c8c0640 1 -- 172.21.15.16:0/2008878260 >> 172.21.15.16:0/2008878260 conn(0x7f7b48077800 msgr2=0x7f7b48077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:21:19.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.626+0000 7f7b4c8c0640 1 -- 172.21.15.16:0/2008878260 shutdown_connections 2024-03-28T22:21:19.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.626+0000 7f7b4c8c0640 1 -- 172.21.15.16:0/2008878260 wait complete. 2024-03-28T22:21:19.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.626+0000 7f7b4c8c0640 1 Processor -- start 2024-03-28T22:21:19.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.626+0000 7f7b4c8c0640 1 -- start start 2024-03-28T22:21:19.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.627+0000 7f7b4c8c0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b481005f0 0x7f7b48102290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:19.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.627+0000 7f7b4c8c0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b48100fb0 0x7f7b481027d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:19.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.627+0000 7f7b4c8c0640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b48106690 0x7f7b48102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:19.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.627+0000 7f7b4c8c0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7b481109d0 con 0x7f7b481005f0 2024-03-28T22:21:19.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.627+0000 7f7b4c8c0640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7b48110850 con 0x7f7b48106690 2024-03-28T22:21:19.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.627+0000 7f7b4c8c0640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7b48110b50 con 0x7f7b48100fb0 2024-03-28T22:21:19.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.627+0000 7f7b4659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b481005f0 0x7f7b48102290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:19.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.627+0000 7f7b4659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b481005f0 0x7f7b48102290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41326/0 (socket says 172.21.15.16:41326) 2024-03-28T22:21:19.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.627+0000 7f7b4659c640 1 -- 172.21.15.16:0/2675407932 learned_addr learned my addr 172.21.15.16:0/2675407932 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:21:19.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.627+0000 7f7b46d9d640 1 --2- 172.21.15.16:0/2675407932 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b48106690 0x7f7b48102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:19.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.627+0000 7f7b45d9b640 1 --2- 172.21.15.16:0/2675407932 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b48100fb0 0x7f7b481027d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:19.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.627+0000 7f7b4659c640 1 -- 172.21.15.16:0/2675407932 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b48100fb0 msgr2=0x7f7b481027d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:19.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.627+0000 7f7b4659c640 1 --2- 172.21.15.16:0/2675407932 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b48100fb0 0x7f7b481027d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:19.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.627+0000 7f7b4659c640 1 -- 172.21.15.16:0/2675407932 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b48106690 msgr2=0x7f7b48102d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:19.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.627+0000 7f7b4659c640 1 --2- 172.21.15.16:0/2675407932 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b48106690 0x7f7b48102d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:19.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.627+0000 7f7b4659c640 1 -- 172.21.15.16:0/2675407932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b481a64a0 con 0x7f7b481005f0 2024-03-28T22:21:19.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.627+0000 7f7b4659c640 1 --2- 172.21.15.16:0/2675407932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b481005f0 0x7f7b48102290 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f7b3800d950 tx=0x7f7b3800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:21:19.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.627+0000 7f7b377fe640 1 -- 172.21.15.16:0/2675407932 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b38017070 con 0x7f7b481005f0 2024-03-28T22:21:19.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.627+0000 7f7b377fe640 1 -- 172.21.15.16:0/2675407932 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b38009c30 con 0x7f7b481005f0 2024-03-28T22:21:19.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.627+0000 7f7b4c8c0640 1 -- 172.21.15.16:0/2675407932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b481a6790 con 0x7f7b481005f0 2024-03-28T22:21:19.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.627+0000 7f7b377fe640 1 -- 172.21.15.16:0/2675407932 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3800f600 con 0x7f7b481005f0 2024-03-28T22:21:19.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.627+0000 7f7b4c8c0640 1 -- 172.21.15.16:0/2675407932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b481a6c60 con 0x7f7b481005f0 2024-03-28T22:21:19.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.628+0000 7f7b4c8c0640 1 -- 172.21.15.16:0/2675407932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b480685d0 con 0x7f7b481005f0 2024-03-28T22:21:19.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.629+0000 7f7b377fe640 1 -- 172.21.15.16:0/2675407932 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f7b3801a070 con 0x7f7b481005f0 2024-03-28T22:21:19.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.630+0000 7f7b377fe640 1 --2- 172.21.15.16:0/2675407932 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f7b300a1ef0 0x7f7b300a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:19.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.630+0000 7f7b45d9b640 1 --2- 172.21.15.16:0/2675407932 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f7b300a1ef0 0x7f7b300a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:19.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.630+0000 7f7b377fe640 1 -- 172.21.15.16:0/2675407932 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8527+0+0 (secure 0 0 0) 0x7f7b380cdb20 con 0x7f7b481005f0 2024-03-28T22:21:19.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.630+0000 7f7b45d9b640 1 --2- 172.21.15.16:0/2675407932 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f7b300a1ef0 0x7f7b300a43b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c005ab0 tx=0x7f7b3c005fd0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:21:19.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.633+0000 7f7b377fe640 1 -- 172.21.15.16:0/2675407932 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7b38009de0 con 0x7f7b481005f0 2024-03-28T22:21:19.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.829+0000 7f7b4c8c0640 1 -- 172.21.15.16:0/2675407932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f7b481a6ef0 con 0x7f7b481005f0 2024-03-28T22:21:19.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.829+0000 7f7b377fe640 1 -- 172.21.15.16:0/2675407932 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 17 entries v99) ==== 81+0+998 (secure 0 0 0) 0x7f7b3809a560 con 0x7f7b481005f0 2024-03-28T22:21:19.831 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3631746518 2024-03-29T22:01:28.993434+0000 2024-03-28T22:21:19.831 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/2859950725 2024-03-29T22:01:02.448898+0000 2024-03-28T22:21:19.831 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2311413235 2024-03-29T22:10:34.301217+0000 2024-03-28T22:21:19.831 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3116652786 2024-03-29T22:01:02.448898+0000 2024-03-28T22:21:19.832 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/317557779 2024-03-29T22:01:02.448898+0000 2024-03-28T22:21:19.832 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1062264551 2024-03-29T22:01:28.993434+0000 2024-03-28T22:21:19.832 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/996598130 2024-03-29T22:01:02.448898+0000 2024-03-28T22:21:19.832 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/21986004 2024-03-29T22:10:34.301217+0000 2024-03-28T22:21:19.832 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/1832422779 2024-03-29T22:01:28.993434+0000 2024-03-28T22:21:19.832 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3821408150 2024-03-29T22:01:28.993434+0000 2024-03-28T22:21:19.832 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3070600736 2024-03-29T22:10:34.301217+0000 2024-03-28T22:21:19.832 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2117914406 2024-03-29T22:10:34.301217+0000 2024-03-28T22:21:19.832 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4029580607 2024-03-29T22:10:34.301217+0000 2024-03-28T22:21:19.833 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/2859950725 2024-03-29T22:01:02.448898+0000 2024-03-28T22:21:19.833 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2787995271 2024-03-29T22:10:34.301217+0000 2024-03-28T22:21:19.833 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/1832422779 2024-03-29T22:01:28.993434+0000 2024-03-28T22:21:19.833 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/21986004 2024-03-29T22:10:34.301217+0000 2024-03-28T22:21:19.834 INFO:teuthology.orchestra.run.smithi016.stderr:listed 17 entries 2024-03-28T22:21:19.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.831+0000 7f7b4c8c0640 1 -- 172.21.15.16:0/2675407932 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f7b300a1ef0 msgr2=0x7f7b300a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:19.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.831+0000 7f7b4c8c0640 1 --2- 172.21.15.16:0/2675407932 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f7b300a1ef0 0x7f7b300a43b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c005ab0 tx=0x7f7b3c005fd0 comp rx=0 tx=0).stop 2024-03-28T22:21:19.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.831+0000 7f7b4c8c0640 1 -- 172.21.15.16:0/2675407932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b481005f0 msgr2=0x7f7b48102290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:19.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.831+0000 7f7b4c8c0640 1 --2- 172.21.15.16:0/2675407932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b481005f0 0x7f7b48102290 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f7b3800d950 tx=0x7f7b3800de10 comp rx=0 tx=0).stop 2024-03-28T22:21:19.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.831+0000 7f7b4c8c0640 1 -- 172.21.15.16:0/2675407932 shutdown_connections 2024-03-28T22:21:19.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.831+0000 7f7b4c8c0640 1 --2- 172.21.15.16:0/2675407932 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f7b300a1ef0 0x7f7b300a43b0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:19.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.831+0000 7f7b4c8c0640 1 --2- 172.21.15.16:0/2675407932 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b48106690 0x7f7b48102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:19.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.831+0000 7f7b4c8c0640 1 --2- 172.21.15.16:0/2675407932 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b48100fb0 0x7f7b481027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:19.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.831+0000 7f7b4c8c0640 1 --2- 172.21.15.16:0/2675407932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b481005f0 0x7f7b48102290 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:19.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.831+0000 7f7b4c8c0640 1 -- 172.21.15.16:0/2675407932 >> 172.21.15.16:0/2675407932 conn(0x7f7b48077800 msgr2=0x7f7b480ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:21:19.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.831+0000 7f7b4c8c0640 1 -- 172.21.15.16:0/2675407932 shutdown_connections 2024-03-28T22:21:19.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:19.831+0000 7f7b4c8c0640 1 -- 172.21.15.16:0/2675407932 wait complete. 2024-03-28T22:21:19.842 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:19 smithi016 ceph-mon[32374]: pgmap v335: 97 pgs: 97 active+clean; 91 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-28T22:21:19.842 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:19 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2875686944' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:21:19.843 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:19 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/1891577488' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:21:19.843 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:19 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/86246643' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:21:19.844 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-28T22:21:19.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:21:19.844 DEBUG:teuthology.orchestra.run.smithi016:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-28T22:21:19.930 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:19 smithi063 ceph-mon[36249]: pgmap v335: 97 pgs: 97 active+clean; 91 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-28T22:21:19.930 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:19 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2875686944' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:21:19.930 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:19 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/1891577488' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:21:19.930 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:19 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/86246643' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:21:19.967 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-03-28T22:21:19.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:21:19.967 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-28T22:21:19.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:21:19.984 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-03-28T22:21:20.039 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-28T22:21:20.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:21:20.039 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T22:21:20.046 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:19 smithi053 ceph-mon[35979]: pgmap v335: 97 pgs: 97 active+clean; 91 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-28T22:21:20.046 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:19 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2875686944' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:21:20.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:19 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/1891577488' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:21:20.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:19 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/86246643' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:21:20.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:21:20.106 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link delete ceph-brx 2024-03-28T22:21:20.201 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-28T22:21:20.213 INFO:tasks.cephadm:Teardown begin 2024-03-28T22:21:20.213 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-28T22:21:20.214 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:21:20.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.318+0000 7fc8d8381640 1 -- 172.21.15.16:0/2346925315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8d01005f0 msgr2=0x7fc8d0100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:20.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.318+0000 7fc8d8381640 1 --2- 172.21.15.16:0/2346925315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8d01005f0 0x7fc8d0100a70 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4009930 tx=0x7fc8c402bc30 comp rx=0 tx=0).stop 2024-03-28T22:21:20.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.318+0000 7fc8d8381640 1 -- 172.21.15.16:0/2346925315 shutdown_connections 2024-03-28T22:21:20.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.318+0000 7fc8d8381640 1 --2- 172.21.15.16:0/2346925315 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8d0100fb0 0x7fc8d010d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:20.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.318+0000 7fc8d8381640 1 --2- 172.21.15.16:0/2346925315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8d01005f0 0x7fc8d0100a70 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:20.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.318+0000 7fc8d8381640 1 --2- 172.21.15.16:0/2346925315 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8d0106690 0x7fc8d0106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:20.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.318+0000 7fc8d8381640 1 -- 172.21.15.16:0/2346925315 >> 172.21.15.16:0/2346925315 conn(0x7fc8d0077800 msgr2=0x7fc8d0077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:21:20.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.318+0000 7fc8d8381640 1 -- 172.21.15.16:0/2346925315 shutdown_connections 2024-03-28T22:21:20.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.318+0000 7fc8d8381640 1 -- 172.21.15.16:0/2346925315 wait complete. 2024-03-28T22:21:20.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.318+0000 7fc8d8381640 1 Processor -- start 2024-03-28T22:21:20.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.318+0000 7fc8d8381640 1 -- start start 2024-03-28T22:21:20.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.319+0000 7fc8d8381640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8d01005f0 0x7fc8d0197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:20.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.319+0000 7fc8d8381640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8d0100fb0 0x7fc8d01984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:20.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.319+0000 7fc8d8381640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8d0106690 0x7fc8d019c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:20.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.319+0000 7fc8d8381640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d0111b10 con 0x7fc8d0100fb0 2024-03-28T22:21:20.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.319+0000 7fc8d8381640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d0111990 con 0x7fc8d0106690 2024-03-28T22:21:20.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.319+0000 7fc8d8381640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d0111c90 con 0x7fc8d01005f0 2024-03-28T22:21:20.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.319+0000 7fc8d591c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8d0100fb0 0x7fc8d01984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:20.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.319+0000 7fc8d591c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8d0100fb0 0x7fc8d01984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41340/0 (socket says 172.21.15.16:41340) 2024-03-28T22:21:20.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.319+0000 7fc8d591c640 1 -- 172.21.15.16:0/4052158464 learned_addr learned my addr 172.21.15.16:0/4052158464 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:21:20.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.319+0000 7fc8d611d640 1 --2- 172.21.15.16:0/4052158464 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8d01005f0 0x7fc8d0197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:20.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.319+0000 7fc8d691e640 1 --2- 172.21.15.16:0/4052158464 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8d0106690 0x7fc8d019c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:20.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.319+0000 7fc8d591c640 1 -- 172.21.15.16:0/4052158464 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8d01005f0 msgr2=0x7fc8d0197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:20.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.319+0000 7fc8d591c640 1 --2- 172.21.15.16:0/4052158464 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8d01005f0 0x7fc8d0197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:20.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.319+0000 7fc8d591c640 1 -- 172.21.15.16:0/4052158464 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8d0106690 msgr2=0x7fc8d019c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:20.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.319+0000 7fc8d591c640 1 --2- 172.21.15.16:0/4052158464 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8d0106690 0x7fc8d019c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:20.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.319+0000 7fc8d591c640 1 -- 172.21.15.16:0/4052158464 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8d019cf60 con 0x7fc8d0100fb0 2024-03-28T22:21:20.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.319+0000 7fc8d591c640 1 --2- 172.21.15.16:0/4052158464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8d0100fb0 0x7fc8d01984b0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4009230 tx=0x7fc8c402cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:21:20.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.319+0000 7fc8cf7fe640 1 -- 172.21.15.16:0/4052158464 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c4041070 con 0x7fc8d0100fb0 2024-03-28T22:21:20.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.319+0000 7fc8cf7fe640 1 -- 172.21.15.16:0/4052158464 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8c4004450 con 0x7fc8d0100fb0 2024-03-28T22:21:20.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.319+0000 7fc8d8381640 1 -- 172.21.15.16:0/4052158464 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8d019d1f0 con 0x7fc8d0100fb0 2024-03-28T22:21:20.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.319+0000 7fc8cf7fe640 1 -- 172.21.15.16:0/4052158464 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c4034440 con 0x7fc8d0100fb0 2024-03-28T22:21:20.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.319+0000 7fc8d8381640 1 -- 172.21.15.16:0/4052158464 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8d01a4ae0 con 0x7fc8d0100fb0 2024-03-28T22:21:20.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.320+0000 7fc8d8381640 1 -- 172.21.15.16:0/4052158464 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8d0101d30 con 0x7fc8d0100fb0 2024-03-28T22:21:20.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.321+0000 7fc8cf7fe640 1 -- 172.21.15.16:0/4052158464 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7fc8c4044070 con 0x7fc8d0100fb0 2024-03-28T22:21:20.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.322+0000 7fc8cf7fe640 1 --2- 172.21.15.16:0/4052158464 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fc8b80a1ef0 0x7fc8b80a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:20.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.322+0000 7fc8d611d640 1 --2- 172.21.15.16:0/4052158464 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fc8b80a1ef0 0x7fc8b80a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:20.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.322+0000 7fc8cf7fe640 1 -- 172.21.15.16:0/4052158464 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8527+0+0 (secure 0 0 0) 0x7fc8c40efa50 con 0x7fc8d0100fb0 2024-03-28T22:21:20.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.322+0000 7fc8d611d640 1 --2- 172.21.15.16:0/4052158464 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fc8b80a1ef0 0x7fc8b80a43b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc8d0101730 tx=0x7fc8c8009500 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:21:20.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.326+0000 7fc8cf7fe640 1 -- 172.21.15.16:0/4052158464 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc8c40bc600 con 0x7fc8d0100fb0 2024-03-28T22:21:20.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.549+0000 7fc8d8381640 1 -- 172.21.15.16:0/4052158464 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc8d0068a20 con 0x7fc8d0100fb0 2024-03-28T22:21:20.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.550+0000 7fc8cf7fe640 1 -- 172.21.15.16:0/4052158464 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11851 (secure 0 0 0) 0x7fc8c40c06a0 con 0x7fc8d0100fb0 2024-03-28T22:21:20.552 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:21:20.552 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/1644581227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1644581227},{"type":"v1","addr":"172.21.15.16:6839","nonce":1644581227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/2593062669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2593062669},{"type":"v1","addr":"172.21.15.16:6841","nonce":2593062669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:08:29.672976+0000","modified":"2024-03-28T22:14:44.922044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24566,"mds_1":24524,"mds_2":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24466":{"gid":24466,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/247249857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":247249857},{"type":"v1","addr":"172.21.15.63:6835","nonce":247249857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6837/2324231032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":2324231032},{"type":"v1","addr":"172.21.15.63:6837","nonce":2324231032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6839/1177197077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":1177197077},{"type":"v1","addr":"172.21.15.63:6839","nonce":1177197077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"e","rank":2,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.53:6835/350727634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":350727634},{"type":"v1","addr":"172.21.15.53:6835","nonce":350727634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/349877573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":349877573},{"type":"v1","addr":"172.21.15.53:6837","nonce":349877573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6839/3714153956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":3714153956},{"type":"v1","addr":"172.21.15.53:6839","nonce":3714153956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24566,"qdb_cluster":[24500,24524,24566]},"id":1}]} 2024-03-28T22:21:20.552 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2024-03-28T22:21:20.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.552+0000 7fc8d8381640 1 -- 172.21.15.16:0/4052158464 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fc8b80a1ef0 msgr2=0x7fc8b80a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:20.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.552+0000 7fc8d8381640 1 --2- 172.21.15.16:0/4052158464 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fc8b80a1ef0 0x7fc8b80a43b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc8d0101730 tx=0x7fc8c8009500 comp rx=0 tx=0).stop 2024-03-28T22:21:20.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.552+0000 7fc8d8381640 1 -- 172.21.15.16:0/4052158464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8d0100fb0 msgr2=0x7fc8d01984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:20.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.552+0000 7fc8d8381640 1 --2- 172.21.15.16:0/4052158464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8d0100fb0 0x7fc8d01984b0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4009230 tx=0x7fc8c402cba0 comp rx=0 tx=0).stop 2024-03-28T22:21:20.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.552+0000 7fc8d8381640 1 -- 172.21.15.16:0/4052158464 shutdown_connections 2024-03-28T22:21:20.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.552+0000 7fc8d8381640 1 --2- 172.21.15.16:0/4052158464 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fc8b80a1ef0 0x7fc8b80a43b0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:20.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.552+0000 7fc8d8381640 1 --2- 172.21.15.16:0/4052158464 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8d0106690 0x7fc8d019c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:20.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.552+0000 7fc8d8381640 1 --2- 172.21.15.16:0/4052158464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8d0100fb0 0x7fc8d01984b0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:20.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.552+0000 7fc8d8381640 1 --2- 172.21.15.16:0/4052158464 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8d01005f0 0x7fc8d0197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:20.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.552+0000 7fc8d8381640 1 -- 172.21.15.16:0/4052158464 >> 172.21.15.16:0/4052158464 conn(0x7fc8d0077800 msgr2=0x7fc8d00ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:21:20.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.552+0000 7fc8d8381640 1 -- 172.21.15.16:0/4052158464 shutdown_connections 2024-03-28T22:21:20.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.552+0000 7fc8d8381640 1 -- 172.21.15.16:0/4052158464 wait complete. 2024-03-28T22:21:20.564 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:21:20.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.666+0000 7f09fb61c640 1 -- 172.21.15.16:0/2190694662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09f4100fa0 msgr2=0x7f09f410d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:20.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.666+0000 7f09fb61c640 1 --2- 172.21.15.16:0/2190694662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09f4100fa0 0x7f09f410d4b0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f09ec009cd0 tx=0x7f09ec02c0f0 comp rx=0 tx=0).stop 2024-03-28T22:21:20.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.666+0000 7f09fb61c640 1 -- 172.21.15.16:0/2190694662 shutdown_connections 2024-03-28T22:21:20.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.666+0000 7f09fb61c640 1 --2- 172.21.15.16:0/2190694662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09f4100fa0 0x7f09f410d4b0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:20.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.666+0000 7f09fb61c640 1 --2- 172.21.15.16:0/2190694662 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09f41005e0 0x7f09f4100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:20.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.666+0000 7f09fb61c640 1 --2- 172.21.15.16:0/2190694662 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09f4106680 0x7f09f4106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:20.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.666+0000 7f09fb61c640 1 -- 172.21.15.16:0/2190694662 >> 172.21.15.16:0/2190694662 conn(0x7f09f40fc0d0 msgr2=0x7f09f40fe4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:21:20.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.666+0000 7f09fb61c640 1 -- 172.21.15.16:0/2190694662 shutdown_connections 2024-03-28T22:21:20.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.666+0000 7f09fb61c640 1 -- 172.21.15.16:0/2190694662 wait complete. 2024-03-28T22:21:20.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.667+0000 7f09fb61c640 1 Processor -- start 2024-03-28T22:21:20.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.667+0000 7f09fb61c640 1 -- start start 2024-03-28T22:21:20.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.667+0000 7f09fb61c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09f41005e0 0x7f09f4197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:20.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.667+0000 7f09fb61c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09f4100fa0 0x7f09f41984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:20.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.667+0000 7f09fb61c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09f4106680 0x7f09f419c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:20.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.667+0000 7f09fb61c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f09f4110a00 con 0x7f09f41005e0 2024-03-28T22:21:20.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.667+0000 7f09fb61c640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f09f4110880 con 0x7f09f4106680 2024-03-28T22:21:20.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.667+0000 7f09fb61c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f09f4110b80 con 0x7f09f4100fa0 2024-03-28T22:21:20.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.667+0000 7f09f93b8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09f41005e0 0x7f09f4197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:20.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.667+0000 7f09f93b8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09f41005e0 0x7f09f4197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47528/0 (socket says 172.21.15.16:47528) 2024-03-28T22:21:20.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.667+0000 7f09f93b8640 1 -- 172.21.15.16:0/1288041097 learned_addr learned my addr 172.21.15.16:0/1288041097 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:21:20.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.667+0000 7f09f9bb9640 1 --2- 172.21.15.16:0/1288041097 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09f4106680 0x7f09f419c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:20.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.667+0000 7f09f8bb7640 1 --2- 172.21.15.16:0/1288041097 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09f4100fa0 0x7f09f41984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:20.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.667+0000 7f09f93b8640 1 -- 172.21.15.16:0/1288041097 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09f4100fa0 msgr2=0x7f09f41984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:20.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.667+0000 7f09f93b8640 1 --2- 172.21.15.16:0/1288041097 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09f4100fa0 0x7f09f41984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:20.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.667+0000 7f09f93b8640 1 -- 172.21.15.16:0/1288041097 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09f4106680 msgr2=0x7f09f419c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:20.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.667+0000 7f09f93b8640 1 --2- 172.21.15.16:0/1288041097 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09f4106680 0x7f09f419c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:20.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.667+0000 7f09f93b8640 1 -- 172.21.15.16:0/1288041097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09f419cf80 con 0x7f09f41005e0 2024-03-28T22:21:20.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.667+0000 7f09f93b8640 1 --2- 172.21.15.16:0/1288041097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09f41005e0 0x7f09f4197f90 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f09f000e9c0 tx=0x7f09f000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:21:20.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.667+0000 7f09ea7fc640 1 -- 172.21.15.16:0/1288041097 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09f000cd80 con 0x7f09f41005e0 2024-03-28T22:21:20.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.667+0000 7f09ea7fc640 1 -- 172.21.15.16:0/1288041097 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09f00035f0 con 0x7f09f41005e0 2024-03-28T22:21:20.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.668+0000 7f09fb61c640 1 -- 172.21.15.16:0/1288041097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09f419d270 con 0x7f09f41005e0 2024-03-28T22:21:20.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.668+0000 7f09ea7fc640 1 -- 172.21.15.16:0/1288041097 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09f0010690 con 0x7f09f41005e0 2024-03-28T22:21:20.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.668+0000 7f09fb61c640 1 -- 172.21.15.16:0/1288041097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09f41a4b00 con 0x7f09f41005e0 2024-03-28T22:21:20.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.669+0000 7f09fb61c640 1 -- 172.21.15.16:0/1288041097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09f4101d20 con 0x7f09f41005e0 2024-03-28T22:21:20.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.670+0000 7f09ea7fc640 1 -- 172.21.15.16:0/1288041097 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f09f00037a0 con 0x7f09f41005e0 2024-03-28T22:21:20.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.670+0000 7f09ea7fc640 1 --2- 172.21.15.16:0/1288041097 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f09e00a1ef0 0x7f09e00a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:20.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.670+0000 7f09ea7fc640 1 -- 172.21.15.16:0/1288041097 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8527+0+0 (secure 0 0 0) 0x7f09f00ce690 con 0x7f09f41005e0 2024-03-28T22:21:20.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.670+0000 7f09f8bb7640 1 --2- 172.21.15.16:0/1288041097 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f09e00a1ef0 0x7f09e00a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:20.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.671+0000 7f09f8bb7640 1 --2- 172.21.15.16:0/1288041097 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f09e00a1ef0 0x7f09e00a43b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f09f4105430 tx=0x7f09e4005f90 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:21:20.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.674+0000 7f09ea7fc640 1 -- 172.21.15.16:0/1288041097 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f09f009b810 con 0x7f09f41005e0 2024-03-28T22:21:20.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.895+0000 7f09fb61c640 1 -- 172.21.15.16:0/1288041097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09f4068a20 con 0x7f09f41005e0 2024-03-28T22:21:20.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.896+0000 7f09ea7fc640 1 -- 172.21.15.16:0/1288041097 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11851 (secure 0 0 0) 0x7f09f009b110 con 0x7f09f41005e0 2024-03-28T22:21:20.899 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:21:20.899 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/2478458611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2478458611},{"type":"v1","addr":"172.21.15.16:6835","nonce":2478458611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4019085314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":4019085314},{"type":"v1","addr":"172.21.15.16:6837","nonce":4019085314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/1644581227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1644581227},{"type":"v1","addr":"172.21.15.16:6839","nonce":1644581227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/2593062669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":2593062669},{"type":"v1","addr":"172.21.15.16:6841","nonce":2593062669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/3228503882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":3228503882},{"type":"v1","addr":"172.21.15.63:6833","nonce":3228503882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3748831330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3748831330},{"type":"v1","addr":"172.21.15.53:6833","nonce":3748831330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:08:29.672976+0000","modified":"2024-03-28T22:14:44.922044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24566,"mds_1":24524,"mds_2":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24466":{"gid":24466,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/247249857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":247249857},{"type":"v1","addr":"172.21.15.63:6835","nonce":247249857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6837/2324231032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":2324231032},{"type":"v1","addr":"172.21.15.63:6837","nonce":2324231032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6839/1177197077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":1177197077},{"type":"v1","addr":"172.21.15.63:6839","nonce":1177197077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"e","rank":2,"incarnation":18,"state":"up:active","state_seq":8,"addr":"172.21.15.53:6835/350727634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":350727634},{"type":"v1","addr":"172.21.15.53:6835","nonce":350727634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/349877573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":349877573},{"type":"v1","addr":"172.21.15.53:6837","nonce":349877573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6839/3714153956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":3714153956},{"type":"v1","addr":"172.21.15.53:6839","nonce":3714153956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24566,"qdb_cluster":[24500,24524,24566]},"id":1}]} 2024-03-28T22:21:20.900 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2024-03-28T22:21:20.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.898+0000 7f09fb61c640 1 -- 172.21.15.16:0/1288041097 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f09e00a1ef0 msgr2=0x7f09e00a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:20.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.898+0000 7f09fb61c640 1 --2- 172.21.15.16:0/1288041097 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f09e00a1ef0 0x7f09e00a43b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f09f4105430 tx=0x7f09e4005f90 comp rx=0 tx=0).stop 2024-03-28T22:21:20.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.898+0000 7f09fb61c640 1 -- 172.21.15.16:0/1288041097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09f41005e0 msgr2=0x7f09f4197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:20.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.898+0000 7f09fb61c640 1 --2- 172.21.15.16:0/1288041097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09f41005e0 0x7f09f4197f90 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f09f000e9c0 tx=0x7f09f000ee80 comp rx=0 tx=0).stop 2024-03-28T22:21:20.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.898+0000 7f09fb61c640 1 -- 172.21.15.16:0/1288041097 shutdown_connections 2024-03-28T22:21:20.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.898+0000 7f09fb61c640 1 --2- 172.21.15.16:0/1288041097 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f09e00a1ef0 0x7f09e00a43b0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:20.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.898+0000 7f09fb61c640 1 --2- 172.21.15.16:0/1288041097 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09f4106680 0x7f09f419c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:20.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.898+0000 7f09fb61c640 1 --2- 172.21.15.16:0/1288041097 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09f4100fa0 0x7f09f41984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:20.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.898+0000 7f09fb61c640 1 --2- 172.21.15.16:0/1288041097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09f41005e0 0x7f09f4197f90 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:20.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.898+0000 7f09fb61c640 1 -- 172.21.15.16:0/1288041097 >> 172.21.15.16:0/1288041097 conn(0x7f09f40fc0d0 msgr2=0x7f09f40fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:21:20.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.898+0000 7f09fb61c640 1 -- 172.21.15.16:0/1288041097 shutdown_connections 2024-03-28T22:21:20.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:20.898+0000 7f09fb61c640 1 -- 172.21.15.16:0/1288041097 wait complete. 2024-03-28T22:21:20.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:20 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2675407932' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T22:21:20.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:20 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/4052158464' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:21:20.910 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:21:21.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.012+0000 7fbe79d8a640 1 -- 172.21.15.16:0/403174109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe7410bce0 msgr2=0x7fbe7410e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:21.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.012+0000 7fbe79d8a640 1 --2- 172.21.15.16:0/403174109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe7410bce0 0x7fbe7410e160 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c009ca0 tx=0x7fbe6c02c180 comp rx=0 tx=0).stop 2024-03-28T22:21:21.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.012+0000 7fbe79d8a640 1 -- 172.21.15.16:0/403174109 shutdown_connections 2024-03-28T22:21:21.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.012+0000 7fbe79d8a640 1 --2- 172.21.15.16:0/403174109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe7410bce0 0x7fbe7410e160 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:21.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.012+0000 7fbe79d8a640 1 --2- 172.21.15.16:0/403174109 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe74103230 0x7fbe7410b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:21.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.012+0000 7fbe79d8a640 1 --2- 172.21.15.16:0/403174109 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe74102910 0x7fbe74102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:21.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.012+0000 7fbe79d8a640 1 -- 172.21.15.16:0/403174109 >> 172.21.15.16:0/403174109 conn(0x7fbe740fc0d0 msgr2=0x7fbe740fe4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:21:21.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.012+0000 7fbe79d8a640 1 -- 172.21.15.16:0/403174109 shutdown_connections 2024-03-28T22:21:21.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.012+0000 7fbe79d8a640 1 -- 172.21.15.16:0/403174109 wait complete. 2024-03-28T22:21:21.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.013+0000 7fbe79d8a640 1 Processor -- start 2024-03-28T22:21:21.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.013+0000 7fbe79d8a640 1 -- start start 2024-03-28T22:21:21.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.013+0000 7fbe79d8a640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe74102910 0x7fbe74197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:21.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.013+0000 7fbe79d8a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe74103230 0x7fbe74198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:21.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.013+0000 7fbe79d8a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe7410bce0 0x7fbe7419c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:21.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.013+0000 7fbe79d8a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbe74111ac0 con 0x7fbe74103230 2024-03-28T22:21:21.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.013+0000 7fbe79d8a640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fbe74111940 con 0x7fbe74102910 2024-03-28T22:21:21.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.013+0000 7fbe79d8a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbe74111c40 con 0x7fbe7410bce0 2024-03-28T22:21:21.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.013+0000 7fbe72ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe74103230 0x7fbe74198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:21.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.013+0000 7fbe72ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe74103230 0x7fbe74198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47546/0 (socket says 172.21.15.16:47546) 2024-03-28T22:21:21.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.013+0000 7fbe72ffd640 1 -- 172.21.15.16:0/3669558334 learned_addr learned my addr 172.21.15.16:0/3669558334 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:21:21.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.013+0000 7fbe737fe640 1 --2- 172.21.15.16:0/3669558334 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe74102910 0x7fbe74197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:21.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.013+0000 7fbe73fff640 1 --2- 172.21.15.16:0/3669558334 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe7410bce0 0x7fbe7419c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:21.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.014+0000 7fbe72ffd640 1 -- 172.21.15.16:0/3669558334 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe7410bce0 msgr2=0x7fbe7419c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:21.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.014+0000 7fbe72ffd640 1 --2- 172.21.15.16:0/3669558334 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe7410bce0 0x7fbe7419c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:21.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.014+0000 7fbe72ffd640 1 -- 172.21.15.16:0/3669558334 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe74102910 msgr2=0x7fbe74197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:21.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.014+0000 7fbe72ffd640 1 --2- 172.21.15.16:0/3669558334 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe74102910 0x7fbe74197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:21.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.014+0000 7fbe72ffd640 1 -- 172.21.15.16:0/3669558334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe7419cf60 con 0x7fbe74103230 2024-03-28T22:21:21.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.014+0000 7fbe73fff640 1 --2- 172.21.15.16:0/3669558334 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe7410bce0 0x7fbe7419c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:21:21.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.014+0000 7fbe72ffd640 1 --2- 172.21.15.16:0/3669558334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe74103230 0x7fbe74198500 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fbe6800e9f0 tx=0x7fbe6800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:21:21.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.014+0000 7fbe70ff9640 1 -- 172.21.15.16:0/3669558334 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe6800cd80 con 0x7fbe74103230 2024-03-28T22:21:21.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.014+0000 7fbe70ff9640 1 -- 172.21.15.16:0/3669558334 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe68010910 con 0x7fbe74103230 2024-03-28T22:21:21.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.014+0000 7fbe79d8a640 1 -- 172.21.15.16:0/3669558334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe7419d250 con 0x7fbe74103230 2024-03-28T22:21:21.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.014+0000 7fbe70ff9640 1 -- 172.21.15.16:0/3669558334 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe68018990 con 0x7fbe74103230 2024-03-28T22:21:21.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.014+0000 7fbe79d8a640 1 -- 172.21.15.16:0/3669558334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe741a4ae0 con 0x7fbe74103230 2024-03-28T22:21:21.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.015+0000 7fbe79d8a640 1 -- 172.21.15.16:0/3669558334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe7406a6b0 con 0x7fbe74103230 2024-03-28T22:21:21.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.016+0000 7fbe70ff9640 1 -- 172.21.15.16:0/3669558334 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7fbe68010ac0 con 0x7fbe74103230 2024-03-28T22:21:21.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.017+0000 7fbe70ff9640 1 --2- 172.21.15.16:0/3669558334 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fbe5c0a1ef0 0x7fbe5c0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:21.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.017+0000 7fbe737fe640 1 --2- 172.21.15.16:0/3669558334 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fbe5c0a1ef0 0x7fbe5c0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:21.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.017+0000 7fbe70ff9640 1 -- 172.21.15.16:0/3669558334 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8527+0+0 (secure 0 0 0) 0x7fbe680ce550 con 0x7fbe74103230 2024-03-28T22:21:21.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.017+0000 7fbe737fe640 1 --2- 172.21.15.16:0/3669558334 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fbe5c0a1ef0 0x7fbe5c0a43b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbe64009bd0 tx=0x7fbe64005f90 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:21:21.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.021+0000 7fbe70ff9640 1 -- 172.21.15.16:0/3669558334 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fbe6809b5e0 con 0x7fbe74103230 2024-03-28T22:21:21.046 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:20 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2675407932' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T22:21:21.046 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:20 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/4052158464' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:21:21.180 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:20 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2675407932' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T22:21:21.180 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:20 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/4052158464' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:21:21.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.204+0000 7fbe79d8a640 1 -- 172.21.15.16:0/3669558334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbe74106840 con 0x7fbe74103230 2024-03-28T22:21:21.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.206+0000 7fbe70ff9640 1 -- 172.21.15.16:0/3669558334 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) ==== 74+0+22967 (secure 0 0 0) 0x7fbe6809aee0 con 0x7fbe74103230 2024-03-28T22:21:21.208 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-28T22:21:21.208 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":99,"fsid":"798d9884-ed4e-11ee-95d0-87774f69a715","created":"2024-03-28T22:00:27.216099+0000","modified":"2024-03-28T22:21:17.467135+0000","last_up_change":"2024-03-28T22:07:40.251141+0000","last_in_change":"2024-03-28T22:07:19.456451+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:03:59.327832+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:08:28.123529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:08:28.919037+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":98,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"46e7c8f8-ecc8-43d6-ae53-0655728f3240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6803","nonce":1589692483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6805","nonce":1589692483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6809","nonce":1589692483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1589692483},{"type":"v1","addr":"172.21.15.16:6807","nonce":1589692483}]},"public_addr":"172.21.15.16:6803/1589692483","cluster_addr":"172.21.15.16:6805/1589692483","heartbeat_back_addr":"172.21.15.16:6809/1589692483","heartbeat_front_addr":"172.21.15.16:6807/1589692483","state":["exists","up"]},{"osd":1,"uuid":"af6d9c22-2830-4ce2-a547-011df0c6cbad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6801","nonce":795531234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6803","nonce":795531234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6807","nonce":795531234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":795531234},{"type":"v1","addr":"172.21.15.53:6805","nonce":795531234}]},"public_addr":"172.21.15.53:6801/795531234","cluster_addr":"172.21.15.53:6803/795531234","heartbeat_back_addr":"172.21.15.53:6807/795531234","heartbeat_front_addr":"172.21.15.53:6805/795531234","state":["exists","up"]},{"osd":2,"uuid":"b10adca9-110c-4f87-85e0-b7542709d31a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6801","nonce":2214131086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6803","nonce":2214131086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6807","nonce":2214131086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":2214131086},{"type":"v1","addr":"172.21.15.63:6805","nonce":2214131086}]},"public_addr":"172.21.15.63:6801/2214131086","cluster_addr":"172.21.15.63:6803/2214131086","heartbeat_back_addr":"172.21.15.63:6807/2214131086","heartbeat_front_addr":"172.21.15.63:6805/2214131086","state":["exists","up"]},{"osd":3,"uuid":"c83cf4bb-395a-4786-abf7-74ca93a33eb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6811","nonce":13176789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6813","nonce":13176789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6817","nonce":13176789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":13176789},{"type":"v1","addr":"172.21.15.16:6815","nonce":13176789}]},"public_addr":"172.21.15.16:6811/13176789","cluster_addr":"172.21.15.16:6813/13176789","heartbeat_back_addr":"172.21.15.16:6817/13176789","heartbeat_front_addr":"172.21.15.16:6815/13176789","state":["exists","up"]},{"osd":4,"uuid":"941b3e54-47d6-4e85-8069-e730c3fa8eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6809","nonce":772161849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6811","nonce":772161849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6815","nonce":772161849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":772161849},{"type":"v1","addr":"172.21.15.53:6813","nonce":772161849}]},"public_addr":"172.21.15.53:6809/772161849","cluster_addr":"172.21.15.53:6811/772161849","heartbeat_back_addr":"172.21.15.53:6815/772161849","heartbeat_front_addr":"172.21.15.53:6813/772161849","state":["exists","up"]},{"osd":5,"uuid":"51d89826-1d60-40fc-8fd8-09bfe1ab4125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6809","nonce":1602554852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6811","nonce":1602554852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6815","nonce":1602554852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":1602554852},{"type":"v1","addr":"172.21.15.63:6813","nonce":1602554852}]},"public_addr":"172.21.15.63:6809/1602554852","cluster_addr":"172.21.15.63:6811/1602554852","heartbeat_back_addr":"172.21.15.63:6815/1602554852","heartbeat_front_addr":"172.21.15.63:6813/1602554852","state":["exists","up"]},{"osd":6,"uuid":"978efe5d-2c55-4ff8-8a1e-9f891388945e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6819","nonce":4290459224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6821","nonce":4290459224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6825","nonce":4290459224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":4290459224},{"type":"v1","addr":"172.21.15.16:6823","nonce":4290459224}]},"public_addr":"172.21.15.16:6819/4290459224","cluster_addr":"172.21.15.16:6821/4290459224","heartbeat_back_addr":"172.21.15.16:6825/4290459224","heartbeat_front_addr":"172.21.15.16:6823/4290459224","state":["exists","up"]},{"osd":7,"uuid":"50d489cb-dbb8-48df-b868-b29edf12e04d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6817","nonce":1056467525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6819","nonce":1056467525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6823","nonce":1056467525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":1056467525},{"type":"v1","addr":"172.21.15.53:6821","nonce":1056467525}]},"public_addr":"172.21.15.53:6817/1056467525","cluster_addr":"172.21.15.53:6819/1056467525","heartbeat_back_addr":"172.21.15.53:6823/1056467525","heartbeat_front_addr":"172.21.15.53:6821/1056467525","state":["exists","up"]},{"osd":8,"uuid":"49b39414-cc9c-4057-ae3f-cc109cd979da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6817","nonce":2918516794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6819","nonce":2918516794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6823","nonce":2918516794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":2918516794},{"type":"v1","addr":"172.21.15.63:6821","nonce":2918516794}]},"public_addr":"172.21.15.63:6817/2918516794","cluster_addr":"172.21.15.63:6819/2918516794","heartbeat_back_addr":"172.21.15.63:6823/2918516794","heartbeat_front_addr":"172.21.15.63:6821/2918516794","state":["exists","up"]},{"osd":9,"uuid":"4c873e51-b783-4eb6-83a3-fee31722d60b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6827","nonce":1585624367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6829","nonce":1585624367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6833","nonce":1585624367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":1585624367},{"type":"v1","addr":"172.21.15.16:6831","nonce":1585624367}]},"public_addr":"172.21.15.16:6827/1585624367","cluster_addr":"172.21.15.16:6829/1585624367","heartbeat_back_addr":"172.21.15.16:6833/1585624367","heartbeat_front_addr":"172.21.15.16:6831/1585624367","state":["exists","up"]},{"osd":10,"uuid":"085a78bf-e2ad-481a-85a3-71d1fc549a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6825","nonce":3357446484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6827","nonce":3357446484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6831","nonce":3357446484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3357446484},{"type":"v1","addr":"172.21.15.53:6829","nonce":3357446484}]},"public_addr":"172.21.15.53:6825/3357446484","cluster_addr":"172.21.15.53:6827/3357446484","heartbeat_back_addr":"172.21.15.53:6831/3357446484","heartbeat_front_addr":"172.21.15.53:6829/3357446484","state":["exists","up"]},{"osd":11,"uuid":"13302aa3-08b9-4639-9042-27d850b4d5f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6825","nonce":2167866570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6827","nonce":2167866570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6831","nonce":2167866570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":2167866570},{"type":"v1","addr":"172.21.15.63:6829","nonce":2167866570}]},"public_addr":"172.21.15.63:6825/2167866570","cluster_addr":"172.21.15.63:6827/2167866570","heartbeat_back_addr":"172.21.15.63:6831/2167866570","heartbeat_front_addr":"172.21.15.63:6829/2167866570","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:07.450323+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:31.363172+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:03:55.295021+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:19.794760+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:04:44.063492+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:08.195064+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:33.655328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:05:57.639414+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:22.402394+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:06:47.462050+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:12.226307+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:07:37.495625+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3631746518":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6801/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/2311413235":"2024-03-29T22:10:34.301217+0000","172.21.15.16:0/3116652786":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/317557779":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/1062264551":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/996598130":"2024-03-29T22:01:02.448898+0000","172.21.15.16:6801/21986004":"2024-03-29T22:10:34.301217+0000","172.21.15.16:6801/1832422779":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/3821408150":"2024-03-29T22:01:28.993434+0000","172.21.15.16:0/3070600736":"2024-03-29T22:10:34.301217+0000","172.21.15.16:0/2117914406":"2024-03-29T22:10:34.301217+0000","172.21.15.16:0/4029580607":"2024-03-29T22:10:34.301217+0000","172.21.15.16:6800/2859950725":"2024-03-29T22:01:02.448898+0000","172.21.15.16:0/2787995271":"2024-03-29T22:10:34.301217+0000","172.21.15.16:6800/1832422779":"2024-03-29T22:01:28.993434+0000","172.21.15.16:6800/21986004":"2024-03-29T22:10:34.301217+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:21:21.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.208+0000 7fbe79d8a640 1 -- 172.21.15.16:0/3669558334 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fbe5c0a1ef0 msgr2=0x7fbe5c0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:21.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.208+0000 7fbe79d8a640 1 --2- 172.21.15.16:0/3669558334 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fbe5c0a1ef0 0x7fbe5c0a43b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbe64009bd0 tx=0x7fbe64005f90 comp rx=0 tx=0).stop 2024-03-28T22:21:21.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.208+0000 7fbe79d8a640 1 -- 172.21.15.16:0/3669558334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe74103230 msgr2=0x7fbe74198500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:21.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.208+0000 7fbe79d8a640 1 --2- 172.21.15.16:0/3669558334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe74103230 0x7fbe74198500 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fbe6800e9f0 tx=0x7fbe6800eeb0 comp rx=0 tx=0).stop 2024-03-28T22:21:21.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.208+0000 7fbe79d8a640 1 -- 172.21.15.16:0/3669558334 shutdown_connections 2024-03-28T22:21:21.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.208+0000 7fbe79d8a640 1 --2- 172.21.15.16:0/3669558334 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fbe5c0a1ef0 0x7fbe5c0a43b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:21.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.208+0000 7fbe79d8a640 1 --2- 172.21.15.16:0/3669558334 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe7410bce0 0x7fbe7419c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:21.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.208+0000 7fbe79d8a640 1 --2- 172.21.15.16:0/3669558334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe74103230 0x7fbe74198500 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:21.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.208+0000 7fbe79d8a640 1 --2- 172.21.15.16:0/3669558334 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe74102910 0x7fbe74197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:21.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.208+0000 7fbe79d8a640 1 -- 172.21.15.16:0/3669558334 >> 172.21.15.16:0/3669558334 conn(0x7fbe740fc0d0 msgr2=0x7fbe740fd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:21:21.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.208+0000 7fbe79d8a640 1 -- 172.21.15.16:0/3669558334 shutdown_connections 2024-03-28T22:21:21.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.208+0000 7fbe79d8a640 1 -- 172.21.15.16:0/3669558334 wait complete. 2024-03-28T22:21:21.220 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-28T22:21:21.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.323+0000 7f871975c640 1 -- 172.21.15.16:0/1630383857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8714103290 msgr2=0x7f871410b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:21.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.323+0000 7f871975c640 1 --2- 172.21.15.16:0/1630383857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8714103290 0x7f871410b770 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f8708009900 tx=0x7f870802bcd0 comp rx=0 tx=0).stop 2024-03-28T22:21:21.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.323+0000 7f871975c640 1 -- 172.21.15.16:0/1630383857 shutdown_connections 2024-03-28T22:21:21.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.323+0000 7f871975c640 1 --2- 172.21.15.16:0/1630383857 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f871410bd40 0x7f871410e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:21.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.323+0000 7f871975c640 1 --2- 172.21.15.16:0/1630383857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8714103290 0x7f871410b770 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:21.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.323+0000 7f871975c640 1 --2- 172.21.15.16:0/1630383857 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8714102970 0x7f8714102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:21.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.323+0000 7f871975c640 1 -- 172.21.15.16:0/1630383857 >> 172.21.15.16:0/1630383857 conn(0x7f8714077800 msgr2=0x7f8714077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:21:21.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.323+0000 7f871975c640 1 -- 172.21.15.16:0/1630383857 shutdown_connections 2024-03-28T22:21:21.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.323+0000 7f871975c640 1 -- 172.21.15.16:0/1630383857 wait complete. 2024-03-28T22:21:21.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.324+0000 7f871975c640 1 Processor -- start 2024-03-28T22:21:21.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.324+0000 7f871975c640 1 -- start start 2024-03-28T22:21:21.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.324+0000 7f871975c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8714102970 0x7f8714198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:21.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.324+0000 7f871975c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8714103290 0x7f8714198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:21.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.324+0000 7f871975c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f871410bd40 0x7f871419c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:21.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.324+0000 7f871975c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8714111ad0 con 0x7f871410bd40 2024-03-28T22:21:21.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.324+0000 7f871975c640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8714111950 con 0x7f8714102970 2024-03-28T22:21:21.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.324+0000 7f871975c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8714111c50 con 0x7f8714103290 2024-03-28T22:21:21.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.324+0000 7f87137fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f871410bd40 0x7f871419c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:21.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.324+0000 7f87137fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f871410bd40 0x7f871419c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47566/0 (socket says 172.21.15.16:47566) 2024-03-28T22:21:21.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.324+0000 7f87137fe640 1 -- 172.21.15.16:0/2177575816 learned_addr learned my addr 172.21.15.16:0/2177575816 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:21:21.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.324+0000 7f8712ffd640 1 --2- 172.21.15.16:0/2177575816 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8714102970 0x7f8714198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:21.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.324+0000 7f87127fc640 1 --2- 172.21.15.16:0/2177575816 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8714103290 0x7f8714198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:21.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.325+0000 7f87137fe640 1 -- 172.21.15.16:0/2177575816 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8714103290 msgr2=0x7f8714198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:21.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.325+0000 7f87137fe640 1 --2- 172.21.15.16:0/2177575816 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8714103290 0x7f8714198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:21.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.325+0000 7f87137fe640 1 -- 172.21.15.16:0/2177575816 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8714102970 msgr2=0x7f8714198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:21.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.325+0000 7f87137fe640 1 --2- 172.21.15.16:0/2177575816 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8714102970 0x7f8714198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:21.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.325+0000 7f87137fe640 1 -- 172.21.15.16:0/2177575816 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f871419d040 con 0x7f871410bd40 2024-03-28T22:21:21.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.325+0000 7f87127fc640 1 --2- 172.21.15.16:0/2177575816 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8714103290 0x7f8714198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:21:21.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.325+0000 7f87137fe640 1 --2- 172.21.15.16:0/2177575816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f871410bd40 0x7f871419c910 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f870400ea60 tx=0x7f870400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:21:21.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.325+0000 7f86fbfff640 1 -- 172.21.15.16:0/2177575816 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f870400ce40 con 0x7f871410bd40 2024-03-28T22:21:21.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.325+0000 7f871975c640 1 -- 172.21.15.16:0/2177575816 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f871419d330 con 0x7f871410bd40 2024-03-28T22:21:21.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.325+0000 7f86fbfff640 1 -- 172.21.15.16:0/2177575816 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8704005020 con 0x7f871410bd40 2024-03-28T22:21:21.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.325+0000 7f86fbfff640 1 -- 172.21.15.16:0/2177575816 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8704010690 con 0x7f871410bd40 2024-03-28T22:21:21.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.325+0000 7f871975c640 1 -- 172.21.15.16:0/2177575816 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87141a4bc0 con 0x7f871410bd40 2024-03-28T22:21:21.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.326+0000 7f871975c640 1 -- 172.21.15.16:0/2177575816 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87140685d0 con 0x7f871410bd40 2024-03-28T22:21:21.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.327+0000 7f86fbfff640 1 -- 172.21.15.16:0/2177575816 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f8704005520 con 0x7f871410bd40 2024-03-28T22:21:21.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.328+0000 7f86fbfff640 1 --2- 172.21.15.16:0/2177575816 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f86fc0a1ef0 0x7f86fc0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:21.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.328+0000 7f8712ffd640 1 --2- 172.21.15.16:0/2177575816 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f86fc0a1ef0 0x7f86fc0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:21.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.328+0000 7f86fbfff640 1 -- 172.21.15.16:0/2177575816 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8527+0+0 (secure 0 0 0) 0x7f87040ce740 con 0x7f871410bd40 2024-03-28T22:21:21.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.328+0000 7f8712ffd640 1 --2- 172.21.15.16:0/2177575816 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f86fc0a1ef0 0x7f86fc0a43b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f870c009bd0 tx=0x7f870c005f90 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:21:21.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.332+0000 7f86fbfff640 1 -- 172.21.15.16:0/2177575816 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f870409b7d0 con 0x7f871410bd40 2024-03-28T22:21:21.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.552+0000 7f871975c640 1 -- 172.21.15.16:0/2177575816 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f87141068a0 con 0x7f871410bd40 2024-03-28T22:21:21.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.554+0000 7f86fbfff640 1 -- 172.21.15.16:0/2177575816 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v28) ==== 161+0+0 (secure 0 0 0) 0x7f870409b0d0 con 0x7f871410bd40 2024-03-28T22:21:21.555 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-28T22:21:21.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.557+0000 7f871975c640 1 -- 172.21.15.16:0/2177575816 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f86fc0a1ef0 msgr2=0x7f86fc0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:21.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.557+0000 7f871975c640 1 --2- 172.21.15.16:0/2177575816 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f86fc0a1ef0 0x7f86fc0a43b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f870c009bd0 tx=0x7f870c005f90 comp rx=0 tx=0).stop 2024-03-28T22:21:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.557+0000 7f871975c640 1 -- 172.21.15.16:0/2177575816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f871410bd40 msgr2=0x7f871419c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.557+0000 7f871975c640 1 --2- 172.21.15.16:0/2177575816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f871410bd40 0x7f871419c910 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f870400ea60 tx=0x7f870400ef20 comp rx=0 tx=0).stop 2024-03-28T22:21:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.557+0000 7f871975c640 1 -- 172.21.15.16:0/2177575816 shutdown_connections 2024-03-28T22:21:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.557+0000 7f871975c640 1 --2- 172.21.15.16:0/2177575816 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f86fc0a1ef0 0x7f86fc0a43b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.557+0000 7f871975c640 1 --2- 172.21.15.16:0/2177575816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f871410bd40 0x7f871419c910 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.557+0000 7f871975c640 1 --2- 172.21.15.16:0/2177575816 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8714103290 0x7f8714198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.557+0000 7f871975c640 1 --2- 172.21.15.16:0/2177575816 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8714102970 0x7f8714198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.557+0000 7f871975c640 1 -- 172.21.15.16:0/2177575816 >> 172.21.15.16:0/2177575816 conn(0x7f8714077800 msgr2=0x7f87140fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:21:21.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.557+0000 7f871975c640 1 -- 172.21.15.16:0/2177575816 shutdown_connections 2024-03-28T22:21:21.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.557+0000 7f871975c640 1 -- 172.21.15.16:0/2177575816 wait complete. 2024-03-28T22:21:21.569 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-28T22:21:21.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.675+0000 7fcadf4f4640 1 -- 172.21.15.16:0/1232449133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcad81009d0 msgr2=0x7fcad8100db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:21.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.675+0000 7fcadf4f4640 1 --2- 172.21.15.16:0/1232449133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcad81009d0 0x7fcad8100db0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fcad0009ca0 tx=0x7fcad002c1d0 comp rx=0 tx=0).stop 2024-03-28T22:21:21.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.675+0000 7fcadf4f4640 1 -- 172.21.15.16:0/1232449133 shutdown_connections 2024-03-28T22:21:21.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.675+0000 7fcadf4f4640 1 --2- 172.21.15.16:0/1232449133 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcad8102b80 0x7fcad810d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:21.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.675+0000 7fcadf4f4640 1 --2- 172.21.15.16:0/1232449133 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcad81021e0 0x7fcad8102640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:21.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.675+0000 7fcadf4f4640 1 --2- 172.21.15.16:0/1232449133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcad81009d0 0x7fcad8100db0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:21.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.675+0000 7fcadf4f4640 1 -- 172.21.15.16:0/1232449133 >> 172.21.15.16:0/1232449133 conn(0x7fcad8077830 msgr2=0x7fcad8077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:21:21.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.675+0000 7fcadf4f4640 1 -- 172.21.15.16:0/1232449133 shutdown_connections 2024-03-28T22:21:21.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.675+0000 7fcadf4f4640 1 -- 172.21.15.16:0/1232449133 wait complete. 2024-03-28T22:21:21.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.675+0000 7fcadf4f4640 1 Processor -- start 2024-03-28T22:21:21.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.675+0000 7fcadf4f4640 1 -- start start 2024-03-28T22:21:21.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.676+0000 7fcadf4f4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcad81009d0 0x7fcad8196f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:21.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.676+0000 7fcadf4f4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcad81021e0 0x7fcad81974a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:21.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.676+0000 7fcadf4f4640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcad8102b80 0x7fcad819b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:21.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.676+0000 7fcadf4f4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcad8110b20 con 0x7fcad81009d0 2024-03-28T22:21:21.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.676+0000 7fcadf4f4640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcad81109a0 con 0x7fcad8102b80 2024-03-28T22:21:21.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.676+0000 7fcadf4f4640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcad8110ca0 con 0x7fcad81021e0 2024-03-28T22:21:21.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.676+0000 7fcadd290640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcad81009d0 0x7fcad8196f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:21.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.676+0000 7fcadd290640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcad81009d0 0x7fcad8196f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47582/0 (socket says 172.21.15.16:47582) 2024-03-28T22:21:21.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.676+0000 7fcadd290640 1 -- 172.21.15.16:0/969433713 learned_addr learned my addr 172.21.15.16:0/969433713 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:21:21.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.676+0000 7fcadda91640 1 --2- 172.21.15.16:0/969433713 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcad8102b80 0x7fcad819b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:21.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.676+0000 7fcadca8f640 1 --2- 172.21.15.16:0/969433713 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcad81021e0 0x7fcad81974a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:21.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.676+0000 7fcadd290640 1 -- 172.21.15.16:0/969433713 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcad81021e0 msgr2=0x7fcad81974a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:21.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.676+0000 7fcadd290640 1 --2- 172.21.15.16:0/969433713 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcad81021e0 0x7fcad81974a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:21.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.676+0000 7fcadd290640 1 -- 172.21.15.16:0/969433713 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcad8102b80 msgr2=0x7fcad819b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:21.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.676+0000 7fcadd290640 1 --2- 172.21.15.16:0/969433713 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcad8102b80 0x7fcad819b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:21.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.676+0000 7fcadd290640 1 -- 172.21.15.16:0/969433713 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcad819bf50 con 0x7fcad81009d0 2024-03-28T22:21:21.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.676+0000 7fcadd290640 1 --2- 172.21.15.16:0/969433713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcad81009d0 0x7fcad8196f60 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fcad0000f80 tx=0x7fcad0034830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:21:21.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.676+0000 7fcad67fc640 1 -- 172.21.15.16:0/969433713 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad0044020 con 0x7fcad81009d0 2024-03-28T22:21:21.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.676+0000 7fcadf4f4640 1 -- 172.21.15.16:0/969433713 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcad819c1e0 con 0x7fcad81009d0 2024-03-28T22:21:21.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.676+0000 7fcad67fc640 1 -- 172.21.15.16:0/969433713 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcad003a040 con 0x7fcad81009d0 2024-03-28T22:21:21.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.676+0000 7fcad67fc640 1 -- 172.21.15.16:0/969433713 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad0039070 con 0x7fcad81009d0 2024-03-28T22:21:21.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.676+0000 7fcadf4f4640 1 -- 172.21.15.16:0/969433713 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcad8069df0 con 0x7fcad81009d0 2024-03-28T22:21:21.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.677+0000 7fcadf4f4640 1 -- 172.21.15.16:0/969433713 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcad8068a20 con 0x7fcad81009d0 2024-03-28T22:21:21.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.678+0000 7fcad67fc640 1 -- 172.21.15.16:0/969433713 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7fcad0047020 con 0x7fcad81009d0 2024-03-28T22:21:21.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.679+0000 7fcad67fc640 1 --2- 172.21.15.16:0/969433713 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fcac00a1ef0 0x7fcac00a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:21.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.679+0000 7fcadca8f640 1 --2- 172.21.15.16:0/969433713 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fcac00a1ef0 0x7fcac00a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:21.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.679+0000 7fcad67fc640 1 -- 172.21.15.16:0/969433713 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8741+0+0 (secure 0 0 0) 0x7fcad00f2e60 con 0x7fcad81009d0 2024-03-28T22:21:21.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.679+0000 7fcadca8f640 1 --2- 172.21.15.16:0/969433713 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fcac00a1ef0 0x7fcac00a43b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fcacc005fd0 tx=0x7fcacc005850 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:21:21.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.682+0000 7fcad67fc640 1 -- 172.21.15.16:0/969433713 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fcad004e020 con 0x7fcad81009d0 2024-03-28T22:21:21.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:21.906+0000 7fcadf4f4640 1 -- 172.21.15.16:0/969433713 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fcad806a810 con 0x7fcad81009d0 2024-03-28T22:21:21.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:21 smithi063 ceph-mon[36249]: pgmap v336: 97 pgs: 97 active+clean; 91 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 937 B/s wr, 7 op/s 2024-03-28T22:21:21.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:21 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/1288041097' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:21:21.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:21 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/3669558334' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:21:21.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:21 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2177575816' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T22:21:21.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:21 smithi063 ceph-mon[36249]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T22:21:21.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:21 smithi063 ceph-mon[36249]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T22:21:21.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:21 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/2177575816' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-28T22:21:21.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:21 smithi063 ceph-mon[36249]: osdmap e100: 12 total, 12 up, 12 in 2024-03-28T22:21:21.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:21 smithi063 ceph-mon[36249]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-28T22:21:21.933 INFO:journalctl@ceph.mds.c.smithi063.stdout:Mar 28 22:21:21 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-c[66857]: ignoring --setuser ceph since I am not root 2024-03-28T22:21:21.933 INFO:journalctl@ceph.mds.c.smithi063.stdout:Mar 28 22:21:21 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-c[66857]: ignoring --setgroup ceph since I am not root 2024-03-28T22:21:21.933 INFO:journalctl@ceph.mds.c.smithi063.stdout:Mar 28 22:21:21 smithi063 ceph-mds[66861]: -- 172.21.15.63:0/2845342998 <== mon.2 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5619a6ce1a40 con 0x5619a6cb4800 2024-03-28T22:21:21.933 INFO:journalctl@ceph.mds.c.smithi063.stdout:Mar 28 22:21:21 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-c[66857]: starting mds.c at 2024-03-28T22:21:21.934 INFO:journalctl@ceph.mds.f.smithi063.stdout:Mar 28 22:21:21 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-f[66546]: ignoring --setuser ceph since I am not root 2024-03-28T22:21:21.934 INFO:journalctl@ceph.mds.f.smithi063.stdout:Mar 28 22:21:21 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-f[66546]: ignoring --setgroup ceph since I am not root 2024-03-28T22:21:21.934 INFO:journalctl@ceph.mds.f.smithi063.stdout:Mar 28 22:21:21 smithi063 ceph-mds[66550]: -- 172.21.15.63:0/350186234 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5615ca527a40 con 0x5615ca4fa800 2024-03-28T22:21:21.934 INFO:journalctl@ceph.mds.f.smithi063.stdout:Mar 28 22:21:21 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-f[66546]: starting mds.f at 2024-03-28T22:21:21.935 INFO:journalctl@ceph.mds.i.smithi063.stdout:Mar 28 22:21:21 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-i[67179]: ignoring --setuser ceph since I am not root 2024-03-28T22:21:21.935 INFO:journalctl@ceph.mds.i.smithi063.stdout:Mar 28 22:21:21 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-i[67179]: ignoring --setgroup ceph since I am not root 2024-03-28T22:21:21.935 INFO:journalctl@ceph.mds.i.smithi063.stdout:Mar 28 22:21:21 smithi063 ceph-mds[67183]: -- 172.21.15.63:0/1129719956 <== mon.1 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55909e5de4e0 con 0x55909e5b5000 2024-03-28T22:21:21.935 INFO:journalctl@ceph.mds.i.smithi063.stdout:Mar 28 22:21:21 smithi063 ceph-mds[67183]: -- 172.21.15.63:0/1129719956 <== mon.1 v2:172.21.15.63:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55909e5e1a40 con 0x55909e5b5000 2024-03-28T22:21:21.936 INFO:journalctl@ceph.mds.i.smithi063.stdout:Mar 28 22:21:21 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-i[67179]: starting mds.i at 2024-03-28T22:21:21.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:21 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mon-a[32370]: 2024-03-28T22:21:21.553+0000 7efcc7d54700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T22:21:21.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:21 smithi016 ceph-mon[32374]: pgmap v336: 97 pgs: 97 active+clean; 91 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 937 B/s wr, 7 op/s 2024-03-28T22:21:21.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:21 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/1288041097' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:21:21.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:21 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3669558334' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:21:21.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:21 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2177575816' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T22:21:21.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:21 smithi016 ceph-mon[32374]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T22:21:21.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:21 smithi016 ceph-mon[32374]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T22:21:21.938 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:21 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/2177575816' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-28T22:21:21.939 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:21 smithi016 ceph-mon[32374]: osdmap e100: 12 total, 12 up, 12 in 2024-03-28T22:21:21.939 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:21 smithi016 ceph-mon[32374]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-28T22:21:22.046 INFO:journalctl@ceph.mds.b.smithi053.stdout:Mar 28 22:21:21 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-b[66455]: ignoring --setuser ceph since I am not root 2024-03-28T22:21:22.047 INFO:journalctl@ceph.mds.b.smithi053.stdout:Mar 28 22:21:21 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-b[66455]: ignoring --setgroup ceph since I am not root 2024-03-28T22:21:22.047 INFO:journalctl@ceph.mds.b.smithi053.stdout:Mar 28 22:21:21 smithi053 ceph-mds[66459]: -- 172.21.15.53:0/2691666867 <== mon.2 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5602ab247a40 con 0x5602ab21b000 2024-03-28T22:21:22.047 INFO:journalctl@ceph.mds.b.smithi053.stdout:Mar 28 22:21:21 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-b[66455]: starting mds.b at 2024-03-28T22:21:22.047 INFO:journalctl@ceph.mds.h.smithi053.stdout:Mar 28 22:21:21 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-h[66151]: ignoring --setuser ceph since I am not root 2024-03-28T22:21:22.047 INFO:journalctl@ceph.mds.h.smithi053.stdout:Mar 28 22:21:21 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-h[66151]: ignoring --setgroup ceph since I am not root 2024-03-28T22:21:22.048 INFO:journalctl@ceph.mds.h.smithi053.stdout:Mar 28 22:21:21 smithi053 ceph-mds[66155]: -- 172.21.15.53:0/780877644 <== mon.2 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e50a0bda40 con 0x55e50a090800 2024-03-28T22:21:22.048 INFO:journalctl@ceph.mds.h.smithi053.stdout:Mar 28 22:21:21 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-h[66151]: starting mds.h at 2024-03-28T22:21:22.048 INFO:journalctl@ceph.mds.e.smithi053.stdout:Mar 28 22:21:21 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-e[65829]: ignoring --setuser ceph since I am not root 2024-03-28T22:21:22.048 INFO:journalctl@ceph.mds.e.smithi053.stdout:Mar 28 22:21:21 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-e[65829]: ignoring --setgroup ceph since I am not root 2024-03-28T22:21:22.048 INFO:journalctl@ceph.mds.e.smithi053.stdout:Mar 28 22:21:21 smithi053 ceph-mds[65833]: -- 172.21.15.53:0/3667877424 <== mon.1 v2:172.21.15.63:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55909c1b7a40 con 0x55909c18a800 2024-03-28T22:21:22.048 INFO:journalctl@ceph.mds.e.smithi053.stdout:Mar 28 22:21:21 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-e[65829]: starting mds.e at 2024-03-28T22:21:22.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:21 smithi053 ceph-mon[35979]: pgmap v336: 97 pgs: 97 active+clean; 91 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 937 B/s wr, 7 op/s 2024-03-28T22:21:22.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:21 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/1288041097' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:21:22.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:21 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3669558334' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:21:22.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:21 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2177575816' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T22:21:22.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:21 smithi053 ceph-mon[35979]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T22:21:22.049 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:21 smithi053 ceph-mon[35979]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T22:21:22.050 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:21 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/2177575816' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-28T22:21:22.050 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:21 smithi053 ceph-mon[35979]: osdmap e100: 12 total, 12 up, 12 in 2024-03-28T22:21:22.050 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:21 smithi053 ceph-mon[35979]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-28T22:21:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.556+0000 7fcad67fc640 1 -- 172.21.15.16:0/969433713 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v29) ==== 104+0+0 (secure 0 0 0) 0x7fcad00bff80 con 0x7fcad81009d0 2024-03-28T22:21:22.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.559+0000 7fcadf4f4640 1 -- 172.21.15.16:0/969433713 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fcac00a1ef0 msgr2=0x7fcac00a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:22.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.559+0000 7fcadf4f4640 1 --2- 172.21.15.16:0/969433713 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fcac00a1ef0 0x7fcac00a43b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fcacc005fd0 tx=0x7fcacc005850 comp rx=0 tx=0).stop 2024-03-28T22:21:22.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.559+0000 7fcadf4f4640 1 -- 172.21.15.16:0/969433713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcad81009d0 msgr2=0x7fcad8196f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:22.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.559+0000 7fcadf4f4640 1 --2- 172.21.15.16:0/969433713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcad81009d0 0x7fcad8196f60 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fcad0000f80 tx=0x7fcad0034830 comp rx=0 tx=0).stop 2024-03-28T22:21:22.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.559+0000 7fcadf4f4640 1 -- 172.21.15.16:0/969433713 shutdown_connections 2024-03-28T22:21:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.559+0000 7fcadf4f4640 1 --2- 172.21.15.16:0/969433713 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7fcac00a1ef0 0x7fcac00a43b0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.559+0000 7fcadf4f4640 1 --2- 172.21.15.16:0/969433713 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcad8102b80 0x7fcad819b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.559+0000 7fcadf4f4640 1 --2- 172.21.15.16:0/969433713 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcad81021e0 0x7fcad81974a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.559+0000 7fcadf4f4640 1 --2- 172.21.15.16:0/969433713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcad81009d0 0x7fcad8196f60 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.559+0000 7fcadf4f4640 1 -- 172.21.15.16:0/969433713 >> 172.21.15.16:0/969433713 conn(0x7fcad8077830 msgr2=0x7fcad8104320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:21:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.559+0000 7fcadf4f4640 1 -- 172.21.15.16:0/969433713 shutdown_connections 2024-03-28T22:21:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.559+0000 7fcadf4f4640 1 -- 172.21.15.16:0/969433713 wait complete. 2024-03-28T22:21:22.576 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-28T22:21:22.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.689+0000 7f552ce4e640 1 -- 172.21.15.16:0/3483681648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5528106660 msgr2=0x7f5528106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:22.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.689+0000 7f552ce4e640 1 --2- 172.21.15.16:0/3483681648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5528106660 0x7f5528106a40 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f5518009cd0 tx=0x7f551802c0f0 comp rx=0 tx=0).stop 2024-03-28T22:21:22.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.689+0000 7f552ce4e640 1 -- 172.21.15.16:0/3483681648 shutdown_connections 2024-03-28T22:21:22.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.689+0000 7f552ce4e640 1 --2- 172.21.15.16:0/3483681648 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5528100f80 0x7f552810d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:22.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.689+0000 7f552ce4e640 1 --2- 172.21.15.16:0/3483681648 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f55281005c0 0x7f5528100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:22.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.689+0000 7f552ce4e640 1 --2- 172.21.15.16:0/3483681648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5528106660 0x7f5528106a40 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:22.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.689+0000 7f552ce4e640 1 -- 172.21.15.16:0/3483681648 >> 172.21.15.16:0/3483681648 conn(0x7f5528077750 msgr2=0x7f5528077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:21:22.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.689+0000 7f552ce4e640 1 -- 172.21.15.16:0/3483681648 shutdown_connections 2024-03-28T22:21:22.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.689+0000 7f552ce4e640 1 -- 172.21.15.16:0/3483681648 wait complete. 2024-03-28T22:21:22.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.690+0000 7f552ce4e640 1 Processor -- start 2024-03-28T22:21:22.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.690+0000 7f552ce4e640 1 -- start start 2024-03-28T22:21:22.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.690+0000 7f552ce4e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55281005c0 0x7f5528102260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:22.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.690+0000 7f552ce4e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5528100f80 0x7f55281027a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:22.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.690+0000 7f552ce4e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5528106660 0x7f5528102ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:22.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.690+0000 7f552ce4e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f55281109e0 con 0x7f55281005c0 2024-03-28T22:21:22.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.690+0000 7f552ce4e640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5528110860 con 0x7f5528106660 2024-03-28T22:21:22.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.690+0000 7f552ce4e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5528110b60 con 0x7f5528100f80 2024-03-28T22:21:22.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.690+0000 7f552659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55281005c0 0x7f5528102260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:22.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.690+0000 7f552659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55281005c0 0x7f5528102260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47602/0 (socket says 172.21.15.16:47602) 2024-03-28T22:21:22.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.690+0000 7f552659c640 1 -- 172.21.15.16:0/3758753064 learned_addr learned my addr 172.21.15.16:0/3758753064 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:21:22.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.690+0000 7f5526d9d640 1 --2- 172.21.15.16:0/3758753064 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5528106660 0x7f5528102ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:22.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.691+0000 7f5525d9b640 1 --2- 172.21.15.16:0/3758753064 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5528100f80 0x7f55281027a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:22.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.691+0000 7f552659c640 1 -- 172.21.15.16:0/3758753064 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5528100f80 msgr2=0x7f55281027a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:22.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.691+0000 7f552659c640 1 --2- 172.21.15.16:0/3758753064 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5528100f80 0x7f55281027a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:22.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.691+0000 7f552659c640 1 -- 172.21.15.16:0/3758753064 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5528106660 msgr2=0x7f5528102ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:22.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.691+0000 7f552659c640 1 --2- 172.21.15.16:0/3758753064 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5528106660 0x7f5528102ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:22.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.691+0000 7f552659c640 1 -- 172.21.15.16:0/3758753064 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55281a6470 con 0x7f55281005c0 2024-03-28T22:21:22.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.691+0000 7f5526d9d640 1 --2- 172.21.15.16:0/3758753064 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5528106660 0x7f5528102ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:21:22.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.691+0000 7f552659c640 1 --2- 172.21.15.16:0/3758753064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55281005c0 0x7f5528102260 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f5518000f80 tx=0x7f5518032ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:21:22.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.693+0000 7f55177fe640 1 -- 172.21.15.16:0/3758753064 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5518041020 con 0x7f55281005c0 2024-03-28T22:21:22.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.693+0000 7f55177fe640 1 -- 172.21.15.16:0/3758753064 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5518034cd0 con 0x7f55281005c0 2024-03-28T22:21:22.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.693+0000 7f552ce4e640 1 -- 172.21.15.16:0/3758753064 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55281a6700 con 0x7f55281005c0 2024-03-28T22:21:22.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.693+0000 7f55177fe640 1 -- 172.21.15.16:0/3758753064 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5518039040 con 0x7f55281005c0 2024-03-28T22:21:22.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.693+0000 7f552ce4e640 1 -- 172.21.15.16:0/3758753064 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55281a6c10 con 0x7f55281005c0 2024-03-28T22:21:22.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.695+0000 7f552ce4e640 1 -- 172.21.15.16:0/3758753064 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5528101d00 con 0x7f55281005c0 2024-03-28T22:21:22.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.696+0000 7f55177fe640 1 -- 172.21.15.16:0/3758753064 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7f5518044020 con 0x7f55281005c0 2024-03-28T22:21:22.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.696+0000 7f55177fe640 1 --2- 172.21.15.16:0/3758753064 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f55100a1f70 0x7f55100a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:22.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.696+0000 7f55177fe640 1 -- 172.21.15.16:0/3758753064 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8741+0+0 (secure 0 0 0) 0x7f55180efe40 con 0x7f55281005c0 2024-03-28T22:21:22.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.696+0000 7f5525d9b640 1 --2- 172.21.15.16:0/3758753064 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f55100a1f70 0x7f55100a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:22.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.697+0000 7f5525d9b640 1 --2- 172.21.15.16:0/3758753064 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f55100a1f70 0x7f55100a4430 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5528103cf0 tx=0x7f551c00b390 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:21:22.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.701+0000 7f55177fe640 1 -- 172.21.15.16:0/3758753064 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f551804b020 con 0x7f55281005c0 2024-03-28T22:21:22.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:22.888+0000 7f552ce4e640 1 -- 172.21.15.16:0/3758753064 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f55281a6f30 con 0x7f55281005c0 2024-03-28T22:21:22.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:22 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/969433713' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T22:21:22.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:22 smithi016 ceph-mon[32374]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T22:21:22.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:22 smithi016 ceph-mon[32374]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-28T22:21:22.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:22 smithi016 ceph-mon[32374]: Cluster is now healthy 2024-03-28T22:21:22.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:22 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/969433713' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-28T22:21:22.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:22 smithi016 ceph-mon[32374]: mds.? [v2:172.21.15.63:6835/2940737058,v1:172.21.15.63:6838/2940737058] up:boot 2024-03-28T22:21:22.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:22 smithi016 ceph-mon[32374]: mds.? [v2:172.21.15.63:6834/1691112328,v1:172.21.15.63:6837/1691112328] up:boot 2024-03-28T22:21:22.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:22 smithi016 ceph-mon[32374]: mds.? [v2:172.21.15.63:6836/3060361665,v1:172.21.15.63:6839/3060361665] up:boot 2024-03-28T22:21:22.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:22 smithi016 ceph-mon[32374]: mds.? [v2:172.21.15.53:6834/30955306,v1:172.21.15.53:6835/30955306] up:boot 2024-03-28T22:21:22.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:22 smithi016 ceph-mon[32374]: mds.? [v2:172.21.15.53:6836/1074345704,v1:172.21.15.53:6837/1074345704] up:boot 2024-03-28T22:21:22.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:22 smithi016 ceph-mon[32374]: mds.? [v2:172.21.15.53:6838/1001999842,v1:172.21.15.53:6839/1001999842] up:boot 2024-03-28T22:21:22.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:22 smithi016 ceph-mon[32374]: fsmap 12 up:standby 2024-03-28T22:21:22.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:22 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T22:21:22.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:22 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T22:21:22.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:22 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T22:21:22.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:22 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T22:21:22.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:22 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T22:21:22.937 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:22 smithi016 ceph-mon[32374]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T22:21:23.046 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:22 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/969433713' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T22:21:23.046 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:22 smithi053 ceph-mon[35979]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T22:21:23.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:22 smithi053 ceph-mon[35979]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-28T22:21:23.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:22 smithi053 ceph-mon[35979]: Cluster is now healthy 2024-03-28T22:21:23.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:22 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/969433713' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-28T22:21:23.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:22 smithi053 ceph-mon[35979]: mds.? [v2:172.21.15.63:6835/2940737058,v1:172.21.15.63:6838/2940737058] up:boot 2024-03-28T22:21:23.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:22 smithi053 ceph-mon[35979]: mds.? [v2:172.21.15.63:6834/1691112328,v1:172.21.15.63:6837/1691112328] up:boot 2024-03-28T22:21:23.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:22 smithi053 ceph-mon[35979]: mds.? [v2:172.21.15.63:6836/3060361665,v1:172.21.15.63:6839/3060361665] up:boot 2024-03-28T22:21:23.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:22 smithi053 ceph-mon[35979]: mds.? [v2:172.21.15.53:6834/30955306,v1:172.21.15.53:6835/30955306] up:boot 2024-03-28T22:21:23.047 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:22 smithi053 ceph-mon[35979]: mds.? [v2:172.21.15.53:6836/1074345704,v1:172.21.15.53:6837/1074345704] up:boot 2024-03-28T22:21:23.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:22 smithi053 ceph-mon[35979]: mds.? [v2:172.21.15.53:6838/1001999842,v1:172.21.15.53:6839/1001999842] up:boot 2024-03-28T22:21:23.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:22 smithi053 ceph-mon[35979]: fsmap 12 up:standby 2024-03-28T22:21:23.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:22 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T22:21:23.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:22 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T22:21:23.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:22 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T22:21:23.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:22 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T22:21:23.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:22 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T22:21:23.048 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:22 smithi053 ceph-mon[35979]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T22:21:23.180 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:22 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/969433713' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T22:21:23.181 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:22 smithi063 ceph-mon[36249]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T22:21:23.181 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:22 smithi063 ceph-mon[36249]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-28T22:21:23.181 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:22 smithi063 ceph-mon[36249]: Cluster is now healthy 2024-03-28T22:21:23.181 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:22 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/969433713' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-28T22:21:23.181 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:22 smithi063 ceph-mon[36249]: mds.? [v2:172.21.15.63:6835/2940737058,v1:172.21.15.63:6838/2940737058] up:boot 2024-03-28T22:21:23.181 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:22 smithi063 ceph-mon[36249]: mds.? [v2:172.21.15.63:6834/1691112328,v1:172.21.15.63:6837/1691112328] up:boot 2024-03-28T22:21:23.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:22 smithi063 ceph-mon[36249]: mds.? [v2:172.21.15.63:6836/3060361665,v1:172.21.15.63:6839/3060361665] up:boot 2024-03-28T22:21:23.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:22 smithi063 ceph-mon[36249]: mds.? [v2:172.21.15.53:6834/30955306,v1:172.21.15.53:6835/30955306] up:boot 2024-03-28T22:21:23.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:22 smithi063 ceph-mon[36249]: mds.? [v2:172.21.15.53:6836/1074345704,v1:172.21.15.53:6837/1074345704] up:boot 2024-03-28T22:21:23.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:22 smithi063 ceph-mon[36249]: mds.? [v2:172.21.15.53:6838/1001999842,v1:172.21.15.53:6839/1001999842] up:boot 2024-03-28T22:21:23.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:22 smithi063 ceph-mon[36249]: fsmap 12 up:standby 2024-03-28T22:21:23.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:22 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T22:21:23.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:22 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T22:21:23.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:22 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T22:21:23.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:22 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T22:21:23.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:22 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T22:21:23.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:22 smithi063 ceph-mon[36249]: from='mgr.15441 172.21.15.16:0/1146192585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T22:21:23.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.697+0000 7f55177fe640 1 -- 172.21.15.16:0/3758753064 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v101) ==== 181+0+0 (secure 0 0 0) 0x7f55180bcf00 con 0x7f55281005c0 2024-03-28T22:21:23.699 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2024-03-28T22:21:23.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.702+0000 7f552ce4e640 1 -- 172.21.15.16:0/3758753064 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f55100a1f70 msgr2=0x7f55100a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:23.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.702+0000 7f552ce4e640 1 --2- 172.21.15.16:0/3758753064 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f55100a1f70 0x7f55100a4430 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5528103cf0 tx=0x7f551c00b390 comp rx=0 tx=0).stop 2024-03-28T22:21:23.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.702+0000 7f552ce4e640 1 -- 172.21.15.16:0/3758753064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55281005c0 msgr2=0x7f5528102260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:23.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.702+0000 7f552ce4e640 1 --2- 172.21.15.16:0/3758753064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55281005c0 0x7f5528102260 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f5518000f80 tx=0x7f5518032ea0 comp rx=0 tx=0).stop 2024-03-28T22:21:23.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.702+0000 7f552ce4e640 1 -- 172.21.15.16:0/3758753064 shutdown_connections 2024-03-28T22:21:23.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.702+0000 7f552ce4e640 1 --2- 172.21.15.16:0/3758753064 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7f55100a1f70 0x7f55100a4430 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:23.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.702+0000 7f552ce4e640 1 --2- 172.21.15.16:0/3758753064 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5528106660 0x7f5528102ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:23.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.702+0000 7f552ce4e640 1 --2- 172.21.15.16:0/3758753064 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5528100f80 0x7f55281027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:23.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.702+0000 7f552ce4e640 1 --2- 172.21.15.16:0/3758753064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55281005c0 0x7f5528102260 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:23.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.702+0000 7f552ce4e640 1 -- 172.21.15.16:0/3758753064 >> 172.21.15.16:0/3758753064 conn(0x7f5528077750 msgr2=0x7f55280ff800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:21:23.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.702+0000 7f552ce4e640 1 -- 172.21.15.16:0/3758753064 shutdown_connections 2024-03-28T22:21:23.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.702+0000 7f552ce4e640 1 -- 172.21.15.16:0/3758753064 wait complete. 2024-03-28T22:21:23.719 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-28T22:21:23.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.830+0000 7efc9dc72640 1 -- 172.21.15.16:0/2555258342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc98100e10 msgr2=0x7efc9810db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:23.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.830+0000 7efc9dc72640 1 --2- 172.21.15.16:0/2555258342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc98100e10 0x7efc9810db60 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7efc8c009900 tx=0x7efc8c02bce0 comp rx=0 tx=0).stop 2024-03-28T22:21:23.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.830+0000 7efc9dc72640 1 -- 172.21.15.16:0/2555258342 shutdown_connections 2024-03-28T22:21:23.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.830+0000 7efc9dc72640 1 --2- 172.21.15.16:0/2555258342 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efc9810e0a0 0x7efc98110490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:23.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.830+0000 7efc9dc72640 1 --2- 172.21.15.16:0/2555258342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc98100e10 0x7efc9810db60 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:23.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.830+0000 7efc9dc72640 1 --2- 172.21.15.16:0/2555258342 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc981004f0 0x7efc981008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:23.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.830+0000 7efc9dc72640 1 -- 172.21.15.16:0/2555258342 >> 172.21.15.16:0/2555258342 conn(0x7efc98077800 msgr2=0x7efc98077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:21:23.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.830+0000 7efc9dc72640 1 -- 172.21.15.16:0/2555258342 shutdown_connections 2024-03-28T22:21:23.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.830+0000 7efc9dc72640 1 -- 172.21.15.16:0/2555258342 wait complete. 2024-03-28T22:21:23.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.831+0000 7efc9dc72640 1 Processor -- start 2024-03-28T22:21:23.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.831+0000 7efc9dc72640 1 -- start start 2024-03-28T22:21:23.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.831+0000 7efc9dc72640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc981004f0 0x7efc9819a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:23.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.831+0000 7efc9dc72640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efc98100e10 0x7efc9819a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:23.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.831+0000 7efc9dc72640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc9810e0a0 0x7efc9819ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:23.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.831+0000 7efc9dc72640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efc98113d60 con 0x7efc9810e0a0 2024-03-28T22:21:23.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.831+0000 7efc9dc72640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7efc98113be0 con 0x7efc981004f0 2024-03-28T22:21:23.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.831+0000 7efc9dc72640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7efc98113ee0 con 0x7efc98100e10 2024-03-28T22:21:23.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.832+0000 7efc97fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc9810e0a0 0x7efc9819ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:23.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.832+0000 7efc97fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc9810e0a0 0x7efc9819ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47626/0 (socket says 172.21.15.16:47626) 2024-03-28T22:21:23.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.832+0000 7efc97fff640 1 -- 172.21.15.16:0/568706228 learned_addr learned my addr 172.21.15.16:0/568706228 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-03-28T22:21:23.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.832+0000 7efc96ffd640 1 --2- 172.21.15.16:0/568706228 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efc98100e10 0x7efc9819a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:23.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.832+0000 7efc977fe640 1 --2- 172.21.15.16:0/568706228 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc981004f0 0x7efc9819a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:23.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.832+0000 7efc97fff640 1 -- 172.21.15.16:0/568706228 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efc98100e10 msgr2=0x7efc9819a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:23.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.832+0000 7efc97fff640 1 --2- 172.21.15.16:0/568706228 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efc98100e10 0x7efc9819a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:23.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.832+0000 7efc97fff640 1 -- 172.21.15.16:0/568706228 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc981004f0 msgr2=0x7efc9819a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:23.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.832+0000 7efc97fff640 1 --2- 172.21.15.16:0/568706228 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc981004f0 0x7efc9819a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:23.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.832+0000 7efc97fff640 1 -- 172.21.15.16:0/568706228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc9819f320 con 0x7efc9810e0a0 2024-03-28T22:21:23.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.832+0000 7efc97fff640 1 --2- 172.21.15.16:0/568706228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc9810e0a0 0x7efc9819ebf0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7efc9000b9e0 tx=0x7efc9000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:21:23.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.832+0000 7efc94ff9640 1 -- 172.21.15.16:0/568706228 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc90019070 con 0x7efc9810e0a0 2024-03-28T22:21:23.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.832+0000 7efc94ff9640 1 -- 172.21.15.16:0/568706228 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc90014440 con 0x7efc9810e0a0 2024-03-28T22:21:23.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.832+0000 7efc9dc72640 1 -- 172.21.15.16:0/568706228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc9819f610 con 0x7efc9810e0a0 2024-03-28T22:21:23.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.832+0000 7efc94ff9640 1 -- 172.21.15.16:0/568706228 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc900125a0 con 0x7efc9810e0a0 2024-03-28T22:21:23.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.832+0000 7efc9dc72640 1 -- 172.21.15.16:0/568706228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc981a6ea0 con 0x7efc9810e0a0 2024-03-28T22:21:23.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.833+0000 7efc9dc72640 1 -- 172.21.15.16:0/568706228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc980685d0 con 0x7efc9810e0a0 2024-03-28T22:21:23.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.834+0000 7efc94ff9640 1 -- 172.21.15.16:0/568706228 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 136887+0+0 (secure 0 0 0) 0x7efc9001c070 con 0x7efc9810e0a0 2024-03-28T22:21:23.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.835+0000 7efc94ff9640 1 --2- 172.21.15.16:0/568706228 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7efc800a1ef0 0x7efc800a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:21:23.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.835+0000 7efc977fe640 1 --2- 172.21.15.16:0/568706228 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7efc800a1ef0 0x7efc800a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:21:23.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.835+0000 7efc94ff9640 1 -- 172.21.15.16:0/568706228 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8245+0+0 (secure 0 0 0) 0x7efc900cd9d0 con 0x7efc9810e0a0 2024-03-28T22:21:23.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.836+0000 7efc977fe640 1 --2- 172.21.15.16:0/568706228 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7efc800a1ef0 0x7efc800a43b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7efc88005e10 tx=0x7efc88005da0 comp rx=0 tx=0).ready entity=mgr.15441 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:21:23.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:23.839+0000 7efc94ff9640 1 -- 172.21.15.16:0/568706228 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7efc9009ac40 con 0x7efc9810e0a0 2024-03-28T22:21:24.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:24.020+0000 7efc9dc72640 1 -- 172.21.15.16:0/568706228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7efc98109260 con 0x7efc9810e0a0 2024-03-28T22:21:24.046 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:23 smithi053 ceph-mon[35979]: pgmap v338: 97 pgs: 97 active+clean; 91 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-03-28T22:21:24.046 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:23 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3758753064' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T22:21:24.180 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:23 smithi063 ceph-mon[36249]: pgmap v338: 97 pgs: 97 active+clean; 91 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-03-28T22:21:24.180 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:23 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/3758753064' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T22:21:24.184 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:23 smithi016 ceph-mon[32374]: pgmap v338: 97 pgs: 97 active+clean; 91 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-03-28T22:21:24.185 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:23 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3758753064' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T22:21:24.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:24.703+0000 7efc94ff9640 1 -- 172.21.15.16:0/568706228 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v102) ==== 169+0+0 (secure 0 0 0) 0x7efc9009a540 con 0x7efc9810e0a0 2024-03-28T22:21:24.705 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2024-03-28T22:21:24.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:24.707+0000 7efc9dc72640 1 -- 172.21.15.16:0/568706228 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7efc800a1ef0 msgr2=0x7efc800a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:24.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:24.707+0000 7efc9dc72640 1 --2- 172.21.15.16:0/568706228 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7efc800a1ef0 0x7efc800a43b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7efc88005e10 tx=0x7efc88005da0 comp rx=0 tx=0).stop 2024-03-28T22:21:24.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:24.707+0000 7efc9dc72640 1 -- 172.21.15.16:0/568706228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc9810e0a0 msgr2=0x7efc9819ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:21:24.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:24.707+0000 7efc9dc72640 1 --2- 172.21.15.16:0/568706228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc9810e0a0 0x7efc9819ebf0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7efc9000b9e0 tx=0x7efc9000bea0 comp rx=0 tx=0).stop 2024-03-28T22:21:24.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:24.707+0000 7efc9dc72640 1 -- 172.21.15.16:0/568706228 shutdown_connections 2024-03-28T22:21:24.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:24.707+0000 7efc9dc72640 1 --2- 172.21.15.16:0/568706228 >> [v2:172.21.15.16:6800/1819900910,v1:172.21.15.16:6801/1819900910] conn(0x7efc800a1ef0 0x7efc800a43b0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:24.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:24.707+0000 7efc9dc72640 1 --2- 172.21.15.16:0/568706228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc9810e0a0 0x7efc9819ebf0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:24.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:24.707+0000 7efc9dc72640 1 --2- 172.21.15.16:0/568706228 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efc98100e10 0x7efc9819a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:24.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:24.707+0000 7efc9dc72640 1 --2- 172.21.15.16:0/568706228 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc981004f0 0x7efc9819a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:21:24.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:24.707+0000 7efc9dc72640 1 -- 172.21.15.16:0/568706228 >> 172.21.15.16:0/568706228 conn(0x7efc98077800 msgr2=0x7efc9810f590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:21:24.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:24.707+0000 7efc9dc72640 1 -- 172.21.15.16:0/568706228 shutdown_connections 2024-03-28T22:21:24.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-28T22:21:24.707+0000 7efc9dc72640 1 -- 172.21.15.16:0/568706228 wait complete. 2024-03-28T22:21:24.726 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T22:21:24.755 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T22:21:24.795 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T22:21:24.832 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-28T22:21:24.832 DEBUG:teuthology.orchestra.run.smithi016:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-28T22:21:24.848 DEBUG:teuthology.orchestra.run.smithi053:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-28T22:21:24.865 DEBUG:teuthology.orchestra.run.smithi063:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-28T22:21:24.889 INFO:tasks.cephadm:Stopping all daemons... 2024-03-28T22:21:24.889 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-28T22:21:24.889 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mon.a 2024-03-28T22:21:25.046 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:24 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/3758753064' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T22:21:25.046 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:24 smithi053 ceph-mon[35979]: osdmap e101: 12 total, 12 up, 12 in 2024-03-28T22:21:25.046 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:24 smithi053 ceph-mon[35979]: from='client.? 172.21.15.16:0/568706228' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T22:21:25.180 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:24 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/3758753064' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T22:21:25.180 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:24 smithi063 ceph-mon[36249]: osdmap e101: 12 total, 12 up, 12 in 2024-03-28T22:21:25.180 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:24 smithi063 ceph-mon[36249]: from='client.? 172.21.15.16:0/568706228' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T22:21:25.185 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:24 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/3758753064' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T22:21:25.185 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:24 smithi016 ceph-mon[32374]: osdmap e101: 12 total, 12 up, 12 in 2024-03-28T22:21:25.185 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:24 smithi016 ceph-mon[32374]: from='client.? 172.21.15.16:0/568706228' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T22:21:25.185 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:24 smithi016 systemd[1]: Stopping Ceph mon.a for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:21:25.515 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:25 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mon-a[32370]: 2024-03-28T22:21:25.197+0000 7efccd55f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:21:25.515 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:25 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mon-a[32370]: 2024-03-28T22:21:25.197+0000 7efccd55f700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-28T22:21:25.515 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:25 smithi016 podman[103136]: 2024-03-28 22:21:25.240372727 +0000 UTC m=+0.163789904 container died 89528bfea73262e13c40e4724f4059497f435302350965eed236cd717fbd566c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mon-a, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-323d23d, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-28T22:21:25.515 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:25 smithi016 podman[103136]: 2024-03-28 22:21:25.515093998 +0000 UTC m=+0.438511176 container cleanup 89528bfea73262e13c40e4724f4059497f435302350965eed236cd717fbd566c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mon-a, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-323d23d, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-28T22:21:25.872 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:25 smithi016 bash[103136]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mon-a 2024-03-28T22:21:25.872 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 28 22:21:25 smithi016 podman[103150]: 2024-03-28 22:21:25.758007006 +0000 UTC m=+0.513371238 container remove 89528bfea73262e13c40e4724f4059497f435302350965eed236cd717fbd566c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mon-a, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-323d23d, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2) 2024-03-28T22:21:25.887 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mon.a.service' 2024-03-28T22:21:25.932 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:21:25.933 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-28T22:21:25.933 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-28T22:21:25.933 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mon.b 2024-03-28T22:21:26.229 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:25 smithi053 systemd[1]: Stopping Ceph mon.b for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:21:26.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:26 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mon-b[35974]: 2024-03-28T22:21:26.227+0000 7f7186414700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:21:26.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:26 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mon-b[35974]: 2024-03-28T22:21:26.227+0000 7f7186414700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-28T22:21:26.546 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:26 smithi053 podman[71718]: 2024-03-28 22:21:26.350888133 +0000 UTC m=+0.206758434 container died 1b0c77331568b2e2a6d2abee909eac4ac1c299df8fc8986de0cedaef3934fed2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mon-b, io.buildah.version=1.33.5, RELEASE=squid-323d23d, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-28T22:21:26.946 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:26 smithi053 podman[71718]: 2024-03-28 22:21:26.703113089 +0000 UTC m=+0.558983384 container cleanup 1b0c77331568b2e2a6d2abee909eac4ac1c299df8fc8986de0cedaef3934fed2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mon-b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-323d23d, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-28T22:21:26.946 INFO:journalctl@ceph.mon.b.smithi053.stdout:Mar 28 22:21:26 smithi053 bash[71718]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mon-b 2024-03-28T22:21:27.050 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mon.b.service' 2024-03-28T22:21:27.103 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:21:27.104 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-28T22:21:27.104 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-28T22:21:27.104 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mon.c 2024-03-28T22:21:27.430 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:27 smithi063 systemd[1]: Stopping Ceph mon.c for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:21:27.708 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:27 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mon-c[36245]: 2024-03-28T22:21:27.442+0000 7fbb39d54700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:21:27.708 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:27 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mon-c[36245]: 2024-03-28T22:21:27.442+0000 7fbb39d54700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-28T22:21:28.010 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:27 smithi063 podman[72671]: 2024-03-28 22:21:27.708522062 +0000 UTC m=+0.385888557 container died 1283d1c3415addeafafc8d38af49df66cfbc8c716c75f9eb2019ec3fe9823e8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-323d23d, ceph=True) 2024-03-28T22:21:28.336 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:28 smithi063 podman[72671]: 2024-03-28 22:21:28.118387021 +0000 UTC m=+0.795753561 container cleanup 1283d1c3415addeafafc8d38af49df66cfbc8c716c75f9eb2019ec3fe9823e8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mon-c, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, RELEASE=squid-323d23d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-28T22:21:28.336 INFO:journalctl@ceph.mon.c.smithi063.stdout:Mar 28 22:21:28 smithi063 bash[72671]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mon-c 2024-03-28T22:21:28.552 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mon.c.service' 2024-03-28T22:21:28.657 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:21:28.658 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-28T22:21:28.658 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-28T22:21:28.658 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mgr.x 2024-03-28T22:21:28.962 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:21:28 smithi016 systemd[1]: Stopping Ceph mgr.x for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:21:29.236 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:21:28 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x[32604]: 2024-03-28T22:21:28.959+0000 7f8ad5a60700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:21:29.236 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:21:28 smithi016 podman[103260]: 2024-03-28 22:21:28.991490553 +0000 UTC m=+0.141302118 container died ebef8c7b1dbd41105072d4cb6ef197b32ec610643cbee18ebaa6bf57aa591597 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x, org.label-schema.build-date=20240326, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-323d23d, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T22:21:29.674 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:21:29 smithi016 podman[103260]: 2024-03-28 22:21:29.336043439 +0000 UTC m=+0.485854991 container cleanup ebef8c7b1dbd41105072d4cb6ef197b32ec610643cbee18ebaa6bf57aa591597 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=squid-323d23d, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-28T22:21:29.675 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:21:29 smithi016 bash[103260]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x 2024-03-28T22:21:29.675 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Mar 28 22:21:29 smithi016 podman[103274]: 2024-03-28 22:21:29.478846465 +0000 UTC m=+0.483508756 container remove ebef8c7b1dbd41105072d4cb6ef197b32ec610643cbee18ebaa6bf57aa591597 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-323d23d, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-28T22:21:29.696 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mgr.x.service' 2024-03-28T22:21:29.733 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:21:29.733 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-28T22:21:29.733 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-28T22:21:29.733 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mgr.y 2024-03-28T22:21:30.046 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:21:29 smithi053 systemd[1]: Stopping Ceph mgr.y for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:21:30.394 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:21:30 smithi053 podman[71840]: 2024-03-28 22:21:30.061441606 +0000 UTC m=+0.122250340 container died 6153d3a6250b7e53fd847d31e0337869267c967dec5cd64a264228f7aa490f72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-323d23d, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-28T22:21:30.394 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:21:30 smithi053 podman[71840]: 2024-03-28 22:21:30.298491002 +0000 UTC m=+0.359299745 container cleanup 6153d3a6250b7e53fd847d31e0337869267c967dec5cd64a264228f7aa490f72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-323d23d, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=) 2024-03-28T22:21:30.394 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:21:30 smithi053 bash[71840]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y 2024-03-28T22:21:30.698 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:21:30 smithi053 podman[71854]: 2024-03-28 22:21:30.500280213 +0000 UTC m=+0.435000733 container remove 6153d3a6250b7e53fd847d31e0337869267c967dec5cd64a264228f7aa490f72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-323d23d, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5) 2024-03-28T22:21:30.699 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 28 22:21:30 smithi053 systemd[1]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-03-28T22:21:30.710 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mgr.y.service' 2024-03-28T22:21:30.750 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:21:30.750 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-28T22:21:30.751 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-28T22:21:30.751 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mgr.z 2024-03-28T22:21:31.077 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:21:30 smithi063 systemd[1]: Stopping Ceph mgr.z for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:21:31.349 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:21:31 smithi063 podman[72793]: 2024-03-28 22:21:31.077610536 +0000 UTC m=+0.120661292 container died 7a7c1ac7b208adb1c6070b943bc965f73704dc587b7f5ff0da79839cdbe4c30e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-323d23d, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-28T22:21:31.623 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:21:31 smithi063 podman[72793]: 2024-03-28 22:21:31.423208275 +0000 UTC m=+0.466259015 container cleanup 7a7c1ac7b208adb1c6070b943bc965f73704dc587b7f5ff0da79839cdbe4c30e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240326, GIT_CLEAN=True, RELEASE=squid-323d23d, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD) 2024-03-28T22:21:31.624 INFO:journalctl@ceph.mgr.z.smithi063.stdout:Mar 28 22:21:31 smithi063 bash[72793]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mgr-z 2024-03-28T22:21:31.840 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mgr.z.service' 2024-03-28T22:21:31.881 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:21:31.881 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-28T22:21:31.881 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-28T22:21:31.881 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.0 2024-03-28T22:21:32.181 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 28 22:21:31 smithi016 systemd[1]: Stopping Ceph osd.0 for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:21:32.435 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 28 22:21:32 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-0[44837]: 2024-03-28T22:21:32.181+0000 7fb9d28eb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:21:32.435 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 28 22:21:32 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-0[44837]: 2024-03-28T22:21:32.181+0000 7fb9d28eb700 -1 osd.0 102 *** Got signal Terminated *** 2024-03-28T22:21:32.435 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 28 22:21:32 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-0[44837]: 2024-03-28T22:21:32.181+0000 7fb9d28eb700 -1 osd.0 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T22:21:37.935 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 28 22:21:37 smithi016 podman[103382]: 2024-03-28 22:21:37.617836599 +0000 UTC m=+5.545233448 container died a445589113201e6d525633b7b03222a4a5d922957c71f655cbf43dad3071f82a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-0, RELEASE=squid-323d23d, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-28T22:21:38.435 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 28 22:21:37 smithi016 podman[103382]: 2024-03-28 22:21:37.956299108 +0000 UTC m=+5.883695946 container cleanup a445589113201e6d525633b7b03222a4a5d922957c71f655cbf43dad3071f82a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-0, RELEASE=squid-323d23d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-28T22:21:38.435 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 28 22:21:37 smithi016 bash[103382]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-0 2024-03-28T22:21:38.435 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 28 22:21:38 smithi016 podman[103401]: 2024-03-28 22:21:38.140861155 +0000 UTC m=+0.521479241 container remove a445589113201e6d525633b7b03222a4a5d922957c71f655cbf43dad3071f82a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-323d23d) 2024-03-28T22:21:38.860 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 28 22:21:38 smithi016 podman[103488]: 2024-03-28 22:21:38.473827502 +0000 UTC m=+0.017809397 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:21:38.861 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 28 22:21:38 smithi016 podman[103488]: 2024-03-28 22:21:38.575049743 +0000 UTC m=+0.119031619 container create 95d03ac5ce3e2fc015916e20b11e646e9a2b9d020f5f368f1afcea9c1d79c1d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-0-deactivate, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-323d23d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326) 2024-03-28T22:21:38.861 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 28 22:21:38 smithi016 podman[103488]: 2024-03-28 22:21:38.80246697 +0000 UTC m=+0.346448852 container init 95d03ac5ce3e2fc015916e20b11e646e9a2b9d020f5f368f1afcea9c1d79c1d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-323d23d, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-28T22:21:38.861 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 28 22:21:38 smithi016 podman[103488]: 2024-03-28 22:21:38.805660995 +0000 UTC m=+0.349642878 container start 95d03ac5ce3e2fc015916e20b11e646e9a2b9d020f5f368f1afcea9c1d79c1d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-0-deactivate, RELEASE=squid-323d23d, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-28T22:21:38.861 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 28 22:21:38 smithi016 podman[103488]: 2024-03-28 22:21:38.860722694 +0000 UTC m=+0.404704581 container attach 95d03ac5ce3e2fc015916e20b11e646e9a2b9d020f5f368f1afcea9c1d79c1d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-323d23d, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-28T22:21:39.133 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 28 22:21:39 smithi016 podman[103488]: 2024-03-28 22:21:39.100382613 +0000 UTC m=+0.644364489 container died 95d03ac5ce3e2fc015916e20b11e646e9a2b9d020f5f368f1afcea9c1d79c1d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, RELEASE=squid-323d23d, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T22:21:39.644 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.0.service' 2024-03-28T22:21:39.686 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:21:39.687 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-28T22:21:39.687 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-28T22:21:39.687 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.1 2024-03-28T22:21:39.998 INFO:journalctl@ceph.osd.1.smithi053.stdout:Mar 28 22:21:39 smithi053 systemd[1]: Stopping Ceph osd.1 for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:21:40.296 INFO:journalctl@ceph.osd.1.smithi053.stdout:Mar 28 22:21:39 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-1[40596]: 2024-03-28T22:21:39.994+0000 7f793f440700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:21:40.296 INFO:journalctl@ceph.osd.1.smithi053.stdout:Mar 28 22:21:39 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-1[40596]: 2024-03-28T22:21:39.994+0000 7f793f440700 -1 osd.1 102 *** Got signal Terminated *** 2024-03-28T22:21:40.296 INFO:journalctl@ceph.osd.1.smithi053.stdout:Mar 28 22:21:39 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-1[40596]: 2024-03-28T22:21:39.994+0000 7f793f440700 -1 osd.1 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T22:21:45.894 INFO:journalctl@ceph.osd.1.smithi053.stdout:Mar 28 22:21:45 smithi053 podman[71964]: 2024-03-28 22:21:45.557883345 +0000 UTC m=+5.668526068 container died 5250da249d5e852b4720f00364b197fff2a1d4531aae157b35c588e6e73c089d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-1, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-323d23d, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T22:21:46.296 INFO:journalctl@ceph.osd.1.smithi053.stdout:Mar 28 22:21:45 smithi053 podman[71964]: 2024-03-28 22:21:45.894383942 +0000 UTC m=+6.005026663 container cleanup 5250da249d5e852b4720f00364b197fff2a1d4531aae157b35c588e6e73c089d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-323d23d, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-28T22:21:46.296 INFO:journalctl@ceph.osd.1.smithi053.stdout:Mar 28 22:21:45 smithi053 bash[71964]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-1 2024-03-28T22:21:46.296 INFO:journalctl@ceph.osd.1.smithi053.stdout:Mar 28 22:21:46 smithi053 podman[71981]: 2024-03-28 22:21:46.095674113 +0000 UTC m=+0.535381411 container remove 5250da249d5e852b4720f00364b197fff2a1d4531aae157b35c588e6e73c089d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-323d23d, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-28T22:21:46.765 INFO:journalctl@ceph.osd.1.smithi053.stdout:Mar 28 22:21:46 smithi053 podman[72068]: 2024-03-28 22:21:46.417867691 +0000 UTC m=+0.018220035 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:21:46.766 INFO:journalctl@ceph.osd.1.smithi053.stdout:Mar 28 22:21:46 smithi053 podman[72068]: 2024-03-28 22:21:46.538239249 +0000 UTC m=+0.138591577 container create 17941b0ea5167e23f87d2b2a5fd1e204817069536a1565ce4270a1ea85a7fc6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, RELEASE=squid-323d23d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-28T22:21:47.046 INFO:journalctl@ceph.osd.1.smithi053.stdout:Mar 28 22:21:46 smithi053 podman[72068]: 2024-03-28 22:21:46.765747778 +0000 UTC m=+0.366100121 container init 17941b0ea5167e23f87d2b2a5fd1e204817069536a1565ce4270a1ea85a7fc6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-323d23d, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-28T22:21:47.046 INFO:journalctl@ceph.osd.1.smithi053.stdout:Mar 28 22:21:46 smithi053 podman[72068]: 2024-03-28 22:21:46.769414497 +0000 UTC m=+0.369766831 container start 17941b0ea5167e23f87d2b2a5fd1e204817069536a1565ce4270a1ea85a7fc6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-323d23d, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=) 2024-03-28T22:21:47.046 INFO:journalctl@ceph.osd.1.smithi053.stdout:Mar 28 22:21:46 smithi053 podman[72068]: 2024-03-28 22:21:46.874120729 +0000 UTC m=+0.474473096 container attach 17941b0ea5167e23f87d2b2a5fd1e204817069536a1565ce4270a1ea85a7fc6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-323d23d, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-28T22:21:47.327 INFO:journalctl@ceph.osd.1.smithi053.stdout:Mar 28 22:21:47 smithi053 podman[72068]: 2024-03-28 22:21:47.066697378 +0000 UTC m=+0.667049723 container died 17941b0ea5167e23f87d2b2a5fd1e204817069536a1565ce4270a1ea85a7fc6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5, RELEASE=squid-323d23d, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0) 2024-03-28T22:21:47.620 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.1.service' 2024-03-28T22:21:47.665 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:21:47.665 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-28T22:21:47.666 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-28T22:21:47.666 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.2 2024-03-28T22:21:47.969 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 28 22:21:47 smithi063 systemd[1]: Stopping Ceph osd.2 for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:21:48.239 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 28 22:21:47 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-2[41397]: 2024-03-28T22:21:47.969+0000 7f0142a6b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:21:48.240 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 28 22:21:47 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-2[41397]: 2024-03-28T22:21:47.969+0000 7f0142a6b700 -1 osd.2 102 *** Got signal Terminated *** 2024-03-28T22:21:48.240 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 28 22:21:47 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-2[41397]: 2024-03-28T22:21:47.969+0000 7f0142a6b700 -1 osd.2 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T22:21:53.680 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 28 22:21:53 smithi063 podman[72919]: 2024-03-28 22:21:53.413364735 +0000 UTC m=+5.544845967 container died 7ab4716535cb2da9c0be59bae2cf81df25c1b3d2a37ba250a4f7e59051f254e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-2, org.label-schema.license=GPLv2, RELEASE=squid-323d23d, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5) 2024-03-28T22:21:53.952 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 28 22:21:53 smithi063 podman[72919]: 2024-03-28 22:21:53.685340801 +0000 UTC m=+5.816821999 container cleanup 7ab4716535cb2da9c0be59bae2cf81df25c1b3d2a37ba250a4f7e59051f254e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-323d23d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-28T22:21:53.953 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 28 22:21:53 smithi063 bash[72919]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-2 2024-03-28T22:21:54.315 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 28 22:21:53 smithi063 podman[72948]: 2024-03-28 22:21:53.953214348 +0000 UTC m=+0.536222343 container remove 7ab4716535cb2da9c0be59bae2cf81df25c1b3d2a37ba250a4f7e59051f254e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-2, CEPH_POINT_RELEASE=, RELEASE=squid-323d23d, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-28T22:21:54.569 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 28 22:21:54 smithi063 podman[73040]: 2024-03-28 22:21:54.21878554 +0000 UTC m=+0.018525156 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:21:54.569 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 28 22:21:54 smithi063 podman[73040]: 2024-03-28 22:21:54.345925324 +0000 UTC m=+0.145664931 container create 987e0b0b705d267b1ae5a1737e9a6023d8463763354a9e4d792fea89328bb5ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-323d23d, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-28T22:21:54.569 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 28 22:21:54 smithi063 podman[73040]: 2024-03-28 22:21:54.565658811 +0000 UTC m=+0.365398433 container init 987e0b0b705d267b1ae5a1737e9a6023d8463763354a9e4d792fea89328bb5ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-2-deactivate, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-323d23d) 2024-03-28T22:21:54.569 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 28 22:21:54 smithi063 podman[73040]: 2024-03-28 22:21:54.569166669 +0000 UTC m=+0.368906292 container start 987e0b0b705d267b1ae5a1737e9a6023d8463763354a9e4d792fea89328bb5ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-323d23d, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-28T22:21:54.871 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 28 22:21:54 smithi063 podman[73040]: 2024-03-28 22:21:54.665318093 +0000 UTC m=+0.465057729 container attach 987e0b0b705d267b1ae5a1737e9a6023d8463763354a9e4d792fea89328bb5ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-323d23d, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-28T22:21:55.159 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 28 22:21:54 smithi063 podman[73040]: 2024-03-28 22:21:54.871691633 +0000 UTC m=+0.671431262 container died 987e0b0b705d267b1ae5a1737e9a6023d8463763354a9e4d792fea89328bb5ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-323d23d) 2024-03-28T22:21:55.435 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.2.service' 2024-03-28T22:21:55.482 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:21:55.482 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-28T22:21:55.482 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-28T22:21:55.482 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.3 2024-03-28T22:21:55.766 INFO:journalctl@ceph.osd.3.smithi016.stdout:Mar 28 22:21:55 smithi016 systemd[1]: Stopping Ceph osd.3 for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:21:56.045 INFO:journalctl@ceph.osd.3.smithi016.stdout:Mar 28 22:21:55 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-3[51407]: 2024-03-28T22:21:55.766+0000 7f1aca4e9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:21:56.045 INFO:journalctl@ceph.osd.3.smithi016.stdout:Mar 28 22:21:55 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-3[51407]: 2024-03-28T22:21:55.766+0000 7f1aca4e9700 -1 osd.3 102 *** Got signal Terminated *** 2024-03-28T22:21:56.045 INFO:journalctl@ceph.osd.3.smithi016.stdout:Mar 28 22:21:55 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-3[51407]: 2024-03-28T22:21:55.766+0000 7f1aca4e9700 -1 osd.3 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T22:21:58.385 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:21:57 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:21:57.966+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:21:58.684 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:21:58 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:21:58.385+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.652839+0000 front 2024-03-28T22:21:33.652795+0000 (oldest deadline 2024-03-28T22:21:58.352870+0000) 2024-03-28T22:21:59.184 INFO:journalctl@ceph.osd.3.smithi016.stdout:Mar 28 22:21:58 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-3[51407]: 2024-03-28T22:21:58.764+0000 7f1ac72de700 -1 osd.3 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.846357+0000 front 2024-03-28T22:21:34.846396+0000 (oldest deadline 2024-03-28T22:21:58.346504+0000) 2024-03-28T22:21:59.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:21:58 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:21:58.952+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:21:59.680 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:21:59 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5[48377]: 2024-03-28T22:21:59.211+0000 7fc50fbc2700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:32.925505+0000 front 2024-03-28T22:21:32.925258+0000 (oldest deadline 2024-03-28T22:21:58.225370+0000) 2024-03-28T22:21:59.684 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:21:59 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:21:59.416+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.652839+0000 front 2024-03-28T22:21:33.652795+0000 (oldest deadline 2024-03-28T22:21:58.352870+0000) 2024-03-28T22:22:00.180 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:21:59 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:21:59.888+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:00.184 INFO:journalctl@ceph.osd.3.smithi016.stdout:Mar 28 22:21:59 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-3[51407]: 2024-03-28T22:21:59.730+0000 7f1ac72de700 -1 osd.3 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.846357+0000 front 2024-03-28T22:21:34.846396+0000 (oldest deadline 2024-03-28T22:21:58.346504+0000) 2024-03-28T22:22:00.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:21:59 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:21:59.931+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:00.680 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:00 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5[48377]: 2024-03-28T22:22:00.253+0000 7fc50fbc2700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:32.925505+0000 front 2024-03-28T22:21:32.925258+0000 (oldest deadline 2024-03-28T22:21:58.225370+0000) 2024-03-28T22:22:00.684 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:00 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:00.409+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.652839+0000 front 2024-03-28T22:21:33.652795+0000 (oldest deadline 2024-03-28T22:21:58.352870+0000) 2024-03-28T22:22:00.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:00 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:00.382+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:01.180 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:00 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:00.909+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:01.336 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:00 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:00.954+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:01.337 INFO:journalctl@ceph.osd.3.smithi016.stdout:Mar 28 22:22:01 smithi016 podman[103595]: 2024-03-28 22:22:01.079084637 +0000 UTC m=+5.398204665 container died c4470d6e66af0f748995ba71ed98220baad937f39df3baa92065479543d8d858 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-3, RELEASE=squid-323d23d, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326) 2024-03-28T22:22:01.680 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:01 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5[48377]: 2024-03-28T22:22:01.303+0000 7fc50fbc2700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:32.925505+0000 front 2024-03-28T22:21:32.925258+0000 (oldest deadline 2024-03-28T22:21:58.225370+0000) 2024-03-28T22:22:01.684 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:01 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:01.408+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.652839+0000 front 2024-03-28T22:21:33.652795+0000 (oldest deadline 2024-03-28T22:21:58.352870+0000) 2024-03-28T22:22:01.685 INFO:journalctl@ceph.osd.3.smithi016.stdout:Mar 28 22:22:01 smithi016 podman[103595]: 2024-03-28 22:22:01.543179224 +0000 UTC m=+5.862299251 container cleanup c4470d6e66af0f748995ba71ed98220baad937f39df3baa92065479543d8d858 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-3, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-323d23d, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-28T22:22:01.685 INFO:journalctl@ceph.osd.3.smithi016.stdout:Mar 28 22:22:01 smithi016 bash[103595]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-3 2024-03-28T22:22:01.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:01 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:01.358+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:01.979 INFO:journalctl@ceph.osd.3.smithi016.stdout:Mar 28 22:22:01 smithi016 podman[103626]: 2024-03-28 22:22:01.713376605 +0000 UTC m=+0.625245540 container remove c4470d6e66af0f748995ba71ed98220baad937f39df3baa92065479543d8d858 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-3, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-323d23d, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True) 2024-03-28T22:22:02.180 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:01 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:01.927+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:02.301 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:01 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:01.978+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:02.301 INFO:journalctl@ceph.osd.3.smithi016.stdout:Mar 28 22:22:02 smithi016 podman[103718]: 2024-03-28 22:22:02.064037621 +0000 UTC m=+0.017970960 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:22:02.302 INFO:journalctl@ceph.osd.3.smithi016.stdout:Mar 28 22:22:02 smithi016 podman[103718]: 2024-03-28 22:22:02.203550008 +0000 UTC m=+0.157483319 container create b5f5b36e172d205de53fc47eec0b355ce80cc568f0e4268a19fe490e722b452f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-323d23d, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-28T22:22:02.680 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:02 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5[48377]: 2024-03-28T22:22:02.283+0000 7fc50fbc2700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:32.925505+0000 front 2024-03-28T22:21:32.925258+0000 (oldest deadline 2024-03-28T22:21:58.225370+0000) 2024-03-28T22:22:02.684 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:02 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:02.365+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.652839+0000 front 2024-03-28T22:21:33.652795+0000 (oldest deadline 2024-03-28T22:21:58.352870+0000) 2024-03-28T22:22:02.684 INFO:journalctl@ceph.osd.3.smithi016.stdout:Mar 28 22:22:02 smithi016 podman[103718]: 2024-03-28 22:22:02.447791808 +0000 UTC m=+0.401725123 container init b5f5b36e172d205de53fc47eec0b355ce80cc568f0e4268a19fe490e722b452f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, RELEASE=squid-323d23d, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T22:22:02.685 INFO:journalctl@ceph.osd.3.smithi016.stdout:Mar 28 22:22:02 smithi016 podman[103718]: 2024-03-28 22:22:02.450796831 +0000 UTC m=+0.404730143 container start b5f5b36e172d205de53fc47eec0b355ce80cc568f0e4268a19fe490e722b452f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, RELEASE=squid-323d23d, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T22:22:02.685 INFO:journalctl@ceph.osd.3.smithi016.stdout:Mar 28 22:22:02 smithi016 podman[103718]: 2024-03-28 22:22:02.53931204 +0000 UTC m=+0.493245362 container attach b5f5b36e172d205de53fc47eec0b355ce80cc568f0e4268a19fe490e722b452f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, RELEASE=squid-323d23d, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-28T22:22:02.795 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:02 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:02.362+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:02.990 INFO:journalctl@ceph.osd.3.smithi016.stdout:Mar 28 22:22:02 smithi016 podman[103718]: 2024-03-28 22:22:02.74428904 +0000 UTC m=+0.698222358 container died b5f5b36e172d205de53fc47eec0b355ce80cc568f0e4268a19fe490e722b452f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-323d23d, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-28T22:22:03.180 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:02 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:02.902+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:03.341 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:02 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:02.990+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:03.481 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.3.service' 2024-03-28T22:22:03.528 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:22:03.528 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-28T22:22:03.529 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-28T22:22:03.529 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.4 2024-03-28T22:22:03.681 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:03 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5[48377]: 2024-03-28T22:22:03.298+0000 7fc50fbc2700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:32.925505+0000 front 2024-03-28T22:21:32.925258+0000 (oldest deadline 2024-03-28T22:21:58.225370+0000) 2024-03-28T22:22:03.684 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:03 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:03.341+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.652839+0000 front 2024-03-28T22:21:33.652795+0000 (oldest deadline 2024-03-28T22:21:58.352870+0000) 2024-03-28T22:22:03.796 INFO:journalctl@ceph.osd.4.smithi053.stdout:Mar 28 22:22:03 smithi053 systemd[1]: Stopping Ceph osd.4 for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:22:03.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:03 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:03.377+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:04.296 INFO:journalctl@ceph.osd.4.smithi053.stdout:Mar 28 22:22:03 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-4[47197]: 2024-03-28T22:22:03.821+0000 7ff1efee0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:22:04.296 INFO:journalctl@ceph.osd.4.smithi053.stdout:Mar 28 22:22:03 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-4[47197]: 2024-03-28T22:22:03.821+0000 7ff1efee0700 -1 osd.4 102 *** Got signal Terminated *** 2024-03-28T22:22:04.296 INFO:journalctl@ceph.osd.4.smithi053.stdout:Mar 28 22:22:03 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-4[47197]: 2024-03-28T22:22:03.821+0000 7ff1efee0700 -1 osd.4 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T22:22:04.297 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:03 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:03.948+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:04.341 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:03 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:03.956+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:04.680 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:04 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5[48377]: 2024-03-28T22:22:04.297+0000 7fc50fbc2700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:32.925505+0000 front 2024-03-28T22:21:32.925258+0000 (oldest deadline 2024-03-28T22:21:58.225370+0000) 2024-03-28T22:22:04.684 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:04 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:04.342+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.652839+0000 front 2024-03-28T22:21:33.652795+0000 (oldest deadline 2024-03-28T22:21:58.352870+0000) 2024-03-28T22:22:04.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:04 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:04.404+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:05.301 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:04 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:04.954+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:05.317 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:04 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:04.988+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:05.680 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:05 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5[48377]: 2024-03-28T22:22:05.316+0000 7fc50fbc2700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:32.925505+0000 front 2024-03-28T22:21:32.925258+0000 (oldest deadline 2024-03-28T22:21:58.225370+0000) 2024-03-28T22:22:05.684 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:05 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:05.300+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.652839+0000 front 2024-03-28T22:21:33.652795+0000 (oldest deadline 2024-03-28T22:21:58.352870+0000) 2024-03-28T22:22:05.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:05 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:05.433+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:06.301 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:05 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:05.979+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:06.367 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:05 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:05.987+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:06.680 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:06 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5[48377]: 2024-03-28T22:22:06.367+0000 7fc50fbc2700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:32.925505+0000 front 2024-03-28T22:21:32.925258+0000 (oldest deadline 2024-03-28T22:21:58.225370+0000) 2024-03-28T22:22:06.684 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:06 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:06.301+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.652839+0000 front 2024-03-28T22:21:33.652795+0000 (oldest deadline 2024-03-28T22:21:58.352870+0000) 2024-03-28T22:22:06.796 INFO:journalctl@ceph.osd.4.smithi053.stdout:Mar 28 22:22:06 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-4[47197]: 2024-03-28T22:22:06.380+0000 7ff1e8ccd700 -1 osd.4 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:42.166400+0000 front 2024-03-28T22:21:42.166277+0000 (oldest deadline 2024-03-28T22:22:05.666401+0000) 2024-03-28T22:22:06.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:06 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:06.406+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:07.344 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:06 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:06.951+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:07.366 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:06 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:06.973+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:07.680 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:07 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5[48377]: 2024-03-28T22:22:07.366+0000 7fc50fbc2700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:32.925505+0000 front 2024-03-28T22:21:32.925258+0000 (oldest deadline 2024-03-28T22:21:58.225370+0000) 2024-03-28T22:22:07.680 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:07 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:07.520+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:41.373318+0000 front 2024-03-28T22:21:41.373414+0000 (oldest deadline 2024-03-28T22:22:06.673341+0000) 2024-03-28T22:22:07.684 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:07 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:07.343+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.652839+0000 front 2024-03-28T22:21:33.652795+0000 (oldest deadline 2024-03-28T22:21:58.352870+0000) 2024-03-28T22:22:07.796 INFO:journalctl@ceph.osd.4.smithi053.stdout:Mar 28 22:22:07 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-4[47197]: 2024-03-28T22:22:07.395+0000 7ff1e8ccd700 -1 osd.4 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:42.166400+0000 front 2024-03-28T22:21:42.166277+0000 (oldest deadline 2024-03-28T22:22:05.666401+0000) 2024-03-28T22:22:07.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:07 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:07.364+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:08.184 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:07 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:07.905+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:08.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:07 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:07.905+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:08.296 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:07 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:07.866+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.211688+0000 front 2024-03-28T22:21:43.211660+0000 (oldest deadline 2024-03-28T22:22:07.311798+0000) 2024-03-28T22:22:08.329 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:07 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:07.983+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:08.680 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:08 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5[48377]: 2024-03-28T22:22:08.329+0000 7fc50fbc2700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:32.925505+0000 front 2024-03-28T22:21:32.925258+0000 (oldest deadline 2024-03-28T22:21:58.225370+0000) 2024-03-28T22:22:08.680 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:08 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5[48377]: 2024-03-28T22:22:08.329+0000 7fc50fbc2700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:42.926578+0000 front 2024-03-28T22:21:42.926540+0000 (oldest deadline 2024-03-28T22:22:08.226459+0000) 2024-03-28T22:22:08.680 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:08 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:08.556+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:41.373318+0000 front 2024-03-28T22:21:41.373414+0000 (oldest deadline 2024-03-28T22:22:06.673341+0000) 2024-03-28T22:22:08.684 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:08 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:08.356+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.652839+0000 front 2024-03-28T22:21:33.652795+0000 (oldest deadline 2024-03-28T22:21:58.352870+0000) 2024-03-28T22:22:08.684 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:08 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:08.356+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.053957+0000 front 2024-03-28T22:21:43.053845+0000 (oldest deadline 2024-03-28T22:22:08.353799+0000) 2024-03-28T22:22:08.796 INFO:journalctl@ceph.osd.4.smithi053.stdout:Mar 28 22:22:08 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-4[47197]: 2024-03-28T22:22:08.382+0000 7ff1e8ccd700 -1 osd.4 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:42.166400+0000 front 2024-03-28T22:21:42.166277+0000 (oldest deadline 2024-03-28T22:22:05.666401+0000) 2024-03-28T22:22:08.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:08 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:08.340+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:09.184 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:08 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:08.913+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:09.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:08 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:08.913+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:09.202 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:08 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:08.858+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.211688+0000 front 2024-03-28T22:21:43.211660+0000 (oldest deadline 2024-03-28T22:22:07.311798+0000) 2024-03-28T22:22:09.360 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:09 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:08.999+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:09.360 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:09 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:08.999+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:09.521 INFO:journalctl@ceph.osd.4.smithi053.stdout:Mar 28 22:22:09 smithi053 podman[72184]: 2024-03-28 22:22:09.20287423 +0000 UTC m=+5.473021287 container died be327efe30f31592273c387f251f133bcd63709602776982645608b232e7111c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-4, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-323d23d, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-28T22:22:09.521 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:09 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:09.329+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:09.521 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:09 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:09.329+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:09.680 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:09 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5[48377]: 2024-03-28T22:22:09.360+0000 7fc50fbc2700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:32.925505+0000 front 2024-03-28T22:21:32.925258+0000 (oldest deadline 2024-03-28T22:21:58.225370+0000) 2024-03-28T22:22:09.680 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:09 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5[48377]: 2024-03-28T22:22:09.360+0000 7fc50fbc2700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:42.926578+0000 front 2024-03-28T22:21:42.926540+0000 (oldest deadline 2024-03-28T22:22:08.226459+0000) 2024-03-28T22:22:09.681 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:09 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:09.530+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:41.373318+0000 front 2024-03-28T22:21:41.373414+0000 (oldest deadline 2024-03-28T22:22:06.673341+0000) 2024-03-28T22:22:09.684 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:09 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:09.350+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.652839+0000 front 2024-03-28T22:21:33.652795+0000 (oldest deadline 2024-03-28T22:21:58.352870+0000) 2024-03-28T22:22:09.684 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:09 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:09.350+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.053957+0000 front 2024-03-28T22:21:43.053845+0000 (oldest deadline 2024-03-28T22:22:08.353799+0000) 2024-03-28T22:22:09.796 INFO:journalctl@ceph.osd.4.smithi053.stdout:Mar 28 22:22:09 smithi053 podman[72184]: 2024-03-28 22:22:09.52153682 +0000 UTC m=+5.791683868 container cleanup be327efe30f31592273c387f251f133bcd63709602776982645608b232e7111c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-4, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-323d23d, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-28T22:22:09.796 INFO:journalctl@ceph.osd.4.smithi053.stdout:Mar 28 22:22:09 smithi053 bash[72184]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-4 2024-03-28T22:22:09.796 INFO:journalctl@ceph.osd.4.smithi053.stdout:Mar 28 22:22:09 smithi053 podman[72201]: 2024-03-28 22:22:09.706216193 +0000 UTC m=+0.503312179 container remove be327efe30f31592273c387f251f133bcd63709602776982645608b232e7111c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-4, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=squid-323d23d) 2024-03-28T22:22:10.115 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:09 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:09.845+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.211688+0000 front 2024-03-28T22:21:43.211660+0000 (oldest deadline 2024-03-28T22:22:07.311798+0000) 2024-03-28T22:22:10.184 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:09 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:09.897+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:10.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:09 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:09.897+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:10.363 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:10 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:10.007+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:10.363 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:10 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:10.007+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:10.417 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:10 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:10.337+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:10.417 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:10 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:10.337+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:10.418 INFO:journalctl@ceph.osd.4.smithi053.stdout:Mar 28 22:22:10 smithi053 podman[72294]: 2024-03-28 22:22:10.115322002 +0000 UTC m=+0.108255743 container create ca1055015b4965c1c37ab00c4646bfd91940482ec0e547c8a06789d0bbf0ab1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-323d23d, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326) 2024-03-28T22:22:10.418 INFO:journalctl@ceph.osd.4.smithi053.stdout:Mar 28 22:22:10 smithi053 podman[72294]: 2024-03-28 22:22:10.024926835 +0000 UTC m=+0.017860574 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:22:10.418 INFO:journalctl@ceph.osd.4.smithi053.stdout:Mar 28 22:22:10 smithi053 podman[72294]: 2024-03-28 22:22:10.325819363 +0000 UTC m=+0.318753081 container init ca1055015b4965c1c37ab00c4646bfd91940482ec0e547c8a06789d0bbf0ab1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, RELEASE=squid-323d23d, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T22:22:10.418 INFO:journalctl@ceph.osd.4.smithi053.stdout:Mar 28 22:22:10 smithi053 podman[72294]: 2024-03-28 22:22:10.328970719 +0000 UTC m=+0.321904439 container start ca1055015b4965c1c37ab00c4646bfd91940482ec0e547c8a06789d0bbf0ab1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-323d23d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0) 2024-03-28T22:22:10.421 INFO:journalctl@ceph.osd.4.smithi053.stdout:Mar 28 22:22:10 smithi053 podman[72294]: 2024-03-28 22:22:10.417441356 +0000 UTC m=+0.410375096 container attach ca1055015b4965c1c37ab00c4646bfd91940482ec0e547c8a06789d0bbf0ab1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-323d23d, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-28T22:22:10.680 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:10 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:10.503+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:41.373318+0000 front 2024-03-28T22:21:41.373414+0000 (oldest deadline 2024-03-28T22:22:06.673341+0000) 2024-03-28T22:22:10.681 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:10 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5[48377]: 2024-03-28T22:22:10.362+0000 7fc50fbc2700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:32.925505+0000 front 2024-03-28T22:21:32.925258+0000 (oldest deadline 2024-03-28T22:21:58.225370+0000) 2024-03-28T22:22:10.681 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:10 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5[48377]: 2024-03-28T22:22:10.362+0000 7fc50fbc2700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:42.926578+0000 front 2024-03-28T22:21:42.926540+0000 (oldest deadline 2024-03-28T22:22:08.226459+0000) 2024-03-28T22:22:10.684 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:10 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:10.392+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.652839+0000 front 2024-03-28T22:21:33.652795+0000 (oldest deadline 2024-03-28T22:21:58.352870+0000) 2024-03-28T22:22:10.684 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:10 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:10.392+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.053957+0000 front 2024-03-28T22:21:43.053845+0000 (oldest deadline 2024-03-28T22:22:08.353799+0000) 2024-03-28T22:22:10.796 INFO:journalctl@ceph.osd.4.smithi053.stdout:Mar 28 22:22:10 smithi053 podman[72294]: 2024-03-28 22:22:10.633874485 +0000 UTC m=+0.626808211 container died ca1055015b4965c1c37ab00c4646bfd91940482ec0e547c8a06789d0bbf0ab1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-323d23d) 2024-03-28T22:22:11.182 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:10 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:10.798+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.211688+0000 front 2024-03-28T22:21:43.211660+0000 (oldest deadline 2024-03-28T22:22:07.311798+0000) 2024-03-28T22:22:11.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:10 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:10.860+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:11.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:10 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:10.860+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:11.200 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.4.service' 2024-03-28T22:22:11.251 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:22:11.251 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-28T22:22:11.252 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-28T22:22:11.252 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.5 2024-03-28T22:22:11.334 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:11 smithi063 systemd[1]: Stopping Ceph osd.5 for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:22:11.335 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:11 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:11.055+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:11.335 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:11 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:11.055+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:11.545 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:11 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:11.349+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:11.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:11 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:11.349+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:11.680 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:11 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:11.463+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:41.373318+0000 front 2024-03-28T22:21:41.373414+0000 (oldest deadline 2024-03-28T22:22:06.673341+0000) 2024-03-28T22:22:11.680 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:11 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5[48377]: 2024-03-28T22:22:11.332+0000 7fc50fbc2700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:32.925505+0000 front 2024-03-28T22:21:32.925258+0000 (oldest deadline 2024-03-28T22:21:58.225370+0000) 2024-03-28T22:22:11.681 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:11 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5[48377]: 2024-03-28T22:22:11.333+0000 7fc50fbc2700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:42.926578+0000 front 2024-03-28T22:21:42.926540+0000 (oldest deadline 2024-03-28T22:22:08.226459+0000) 2024-03-28T22:22:11.681 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:11 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5[48377]: 2024-03-28T22:22:11.554+0000 7fc512dcd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:22:11.681 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:11 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5[48377]: 2024-03-28T22:22:11.554+0000 7fc512dcd700 -1 osd.5 102 *** Got signal Terminated *** 2024-03-28T22:22:11.681 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:11 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5[48377]: 2024-03-28T22:22:11.554+0000 7fc512dcd700 -1 osd.5 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T22:22:11.684 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:11 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:11.351+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.652839+0000 front 2024-03-28T22:21:33.652795+0000 (oldest deadline 2024-03-28T22:21:58.352870+0000) 2024-03-28T22:22:11.684 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:11 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:11.351+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.053957+0000 front 2024-03-28T22:21:43.053845+0000 (oldest deadline 2024-03-28T22:22:08.353799+0000) 2024-03-28T22:22:12.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:11 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:11.780+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.211688+0000 front 2024-03-28T22:21:43.211660+0000 (oldest deadline 2024-03-28T22:22:07.311798+0000) 2024-03-28T22:22:12.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:11 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:11.867+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:12.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:11 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:11.867+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:12.290 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:12 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:12.008+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:12.290 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:12 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:12.008+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:12.680 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:12 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:12.485+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:41.373318+0000 front 2024-03-28T22:21:41.373414+0000 (oldest deadline 2024-03-28T22:22:06.673341+0000) 2024-03-28T22:22:12.680 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:12 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5[48377]: 2024-03-28T22:22:12.289+0000 7fc50fbc2700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:32.925505+0000 front 2024-03-28T22:21:32.925258+0000 (oldest deadline 2024-03-28T22:21:58.225370+0000) 2024-03-28T22:22:12.681 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:12 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5[48377]: 2024-03-28T22:22:12.289+0000 7fc50fbc2700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:42.926578+0000 front 2024-03-28T22:21:42.926540+0000 (oldest deadline 2024-03-28T22:22:08.226459+0000) 2024-03-28T22:22:12.684 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:12 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:12.326+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.652839+0000 front 2024-03-28T22:21:33.652795+0000 (oldest deadline 2024-03-28T22:21:58.352870+0000) 2024-03-28T22:22:12.685 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:12 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:12.326+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.053957+0000 front 2024-03-28T22:21:43.053845+0000 (oldest deadline 2024-03-28T22:22:08.353799+0000) 2024-03-28T22:22:12.738 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:12 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:12.315+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:12.738 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:12 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:12.315+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:13.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:12 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:12.738+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.211688+0000 front 2024-03-28T22:21:43.211660+0000 (oldest deadline 2024-03-28T22:22:07.311798+0000) 2024-03-28T22:22:13.184 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:12 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:12.831+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:13.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:12 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:12.831+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:13.287 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:13 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:13.009+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:13.287 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:13 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:13.009+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:13.531 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:13 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:13.281+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.652839+0000 front 2024-03-28T22:21:33.652795+0000 (oldest deadline 2024-03-28T22:21:58.352870+0000) 2024-03-28T22:22:13.532 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:13 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:13.281+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.053957+0000 front 2024-03-28T22:21:43.053845+0000 (oldest deadline 2024-03-28T22:22:08.353799+0000) 2024-03-28T22:22:13.680 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:13 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:13.517+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:41.373318+0000 front 2024-03-28T22:21:41.373414+0000 (oldest deadline 2024-03-28T22:22:06.673341+0000) 2024-03-28T22:22:13.680 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:13 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5[48377]: 2024-03-28T22:22:13.287+0000 7fc50fbc2700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:32.925505+0000 front 2024-03-28T22:21:32.925258+0000 (oldest deadline 2024-03-28T22:21:58.225370+0000) 2024-03-28T22:22:13.680 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:13 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5[48377]: 2024-03-28T22:22:13.287+0000 7fc50fbc2700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:42.926578+0000 front 2024-03-28T22:21:42.926540+0000 (oldest deadline 2024-03-28T22:22:08.226459+0000) 2024-03-28T22:22:13.693 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:13 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:13.313+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:13.693 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:13 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:13.313+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:14.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:13 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:13.693+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.211688+0000 front 2024-03-28T22:21:43.211660+0000 (oldest deadline 2024-03-28T22:22:07.311798+0000) 2024-03-28T22:22:14.184 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:13 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:13.811+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:14.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:13 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:13.811+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:14.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:13 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:13.811+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.710716+0000 front 2024-03-28T22:21:50.710773+0000 (oldest deadline 2024-03-28T22:22:13.610874+0000) 2024-03-28T22:22:14.277 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:14 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:13.999+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:14.277 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:14 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:13.999+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:14.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:14 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:14.280+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:14.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:14 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:14.280+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:14.547 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:14 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:14.280+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:14.680 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:14 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5[48377]: 2024-03-28T22:22:14.276+0000 7fc50fbc2700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:32.925505+0000 front 2024-03-28T22:21:32.925258+0000 (oldest deadline 2024-03-28T22:21:58.225370+0000) 2024-03-28T22:22:14.680 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:14 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5[48377]: 2024-03-28T22:22:14.276+0000 7fc50fbc2700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:42.926578+0000 front 2024-03-28T22:21:42.926540+0000 (oldest deadline 2024-03-28T22:22:08.226459+0000) 2024-03-28T22:22:14.680 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:14 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5[48377]: 2024-03-28T22:22:14.276+0000 7fc50fbc2700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.226947+0000 front 2024-03-28T22:21:48.227106+0000 (oldest deadline 2024-03-28T22:22:13.527148+0000) 2024-03-28T22:22:14.681 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:14 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:14.517+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:41.373318+0000 front 2024-03-28T22:21:41.373414+0000 (oldest deadline 2024-03-28T22:22:06.673341+0000) 2024-03-28T22:22:14.681 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:14 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:14.517+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:47.774560+0000 front 2024-03-28T22:21:47.774226+0000 (oldest deadline 2024-03-28T22:22:13.674264+0000) 2024-03-28T22:22:14.684 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:14 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:14.238+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.652839+0000 front 2024-03-28T22:21:33.652795+0000 (oldest deadline 2024-03-28T22:21:58.352870+0000) 2024-03-28T22:22:14.684 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:14 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:14.238+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.053957+0000 front 2024-03-28T22:21:43.053845+0000 (oldest deadline 2024-03-28T22:22:08.353799+0000) 2024-03-28T22:22:14.685 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:14 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:14.238+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.654922+0000 front 2024-03-28T22:21:50.655059+0000 (oldest deadline 2024-03-28T22:22:13.554819+0000) 2024-03-28T22:22:15.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:14 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:14.728+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.211688+0000 front 2024-03-28T22:21:43.211660+0000 (oldest deadline 2024-03-28T22:22:07.311798+0000) 2024-03-28T22:22:15.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:14 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:14.780+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:15.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:14 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:14.780+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:15.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:14 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:14.780+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.710716+0000 front 2024-03-28T22:21:50.710773+0000 (oldest deadline 2024-03-28T22:22:13.610874+0000) 2024-03-28T22:22:15.269 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:15 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:15.015+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:15.269 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:15 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:15.015+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:15.541 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:15 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5[48377]: 2024-03-28T22:22:15.269+0000 7fc50fbc2700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:32.925505+0000 front 2024-03-28T22:21:32.925258+0000 (oldest deadline 2024-03-28T22:21:58.225370+0000) 2024-03-28T22:22:15.541 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:15 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5[48377]: 2024-03-28T22:22:15.269+0000 7fc50fbc2700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:42.926578+0000 front 2024-03-28T22:21:42.926540+0000 (oldest deadline 2024-03-28T22:22:08.226459+0000) 2024-03-28T22:22:15.541 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:15 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5[48377]: 2024-03-28T22:22:15.269+0000 7fc50fbc2700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.226947+0000 front 2024-03-28T22:21:48.227106+0000 (oldest deadline 2024-03-28T22:22:13.527148+0000) 2024-03-28T22:22:15.545 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:15 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:15.280+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:15.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:15 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:15.280+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:15.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:15 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:15.280+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:15.684 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:15 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:15.282+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.652839+0000 front 2024-03-28T22:21:33.652795+0000 (oldest deadline 2024-03-28T22:21:58.352870+0000) 2024-03-28T22:22:15.684 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:15 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:15.282+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.053957+0000 front 2024-03-28T22:21:43.053845+0000 (oldest deadline 2024-03-28T22:22:08.353799+0000) 2024-03-28T22:22:15.685 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:15 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:15.282+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.654922+0000 front 2024-03-28T22:21:50.655059+0000 (oldest deadline 2024-03-28T22:22:13.554819+0000) 2024-03-28T22:22:15.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:15 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:15.540+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:41.373318+0000 front 2024-03-28T22:21:41.373414+0000 (oldest deadline 2024-03-28T22:22:06.673341+0000) 2024-03-28T22:22:15.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:15 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:15.540+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:47.774560+0000 front 2024-03-28T22:21:47.774226+0000 (oldest deadline 2024-03-28T22:22:13.674264+0000) 2024-03-28T22:22:16.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:15 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:15.769+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.211688+0000 front 2024-03-28T22:21:43.211660+0000 (oldest deadline 2024-03-28T22:22:07.311798+0000) 2024-03-28T22:22:16.184 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:15 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:15.812+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:16.184 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:15 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:15.812+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:16.184 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:15 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:15.812+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.710716+0000 front 2024-03-28T22:21:50.710773+0000 (oldest deadline 2024-03-28T22:22:13.610874+0000) 2024-03-28T22:22:16.293 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:16 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:16.014+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:16.294 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:16 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:16.014+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:16.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:16 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:16.266+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:16.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:16 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:16.266+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:16.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:16 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:16.266+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:16.587 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:16 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5[48377]: 2024-03-28T22:22:16.293+0000 7fc50fbc2700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:32.925505+0000 front 2024-03-28T22:21:32.925258+0000 (oldest deadline 2024-03-28T22:21:58.225370+0000) 2024-03-28T22:22:16.587 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:16 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5[48377]: 2024-03-28T22:22:16.293+0000 7fc50fbc2700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:42.926578+0000 front 2024-03-28T22:21:42.926540+0000 (oldest deadline 2024-03-28T22:22:08.226459+0000) 2024-03-28T22:22:16.587 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:16 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5[48377]: 2024-03-28T22:22:16.293+0000 7fc50fbc2700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.226947+0000 front 2024-03-28T22:21:48.227106+0000 (oldest deadline 2024-03-28T22:22:13.527148+0000) 2024-03-28T22:22:16.684 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:16 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:16.286+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.652839+0000 front 2024-03-28T22:21:33.652795+0000 (oldest deadline 2024-03-28T22:21:58.352870+0000) 2024-03-28T22:22:16.685 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:16 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:16.286+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.053957+0000 front 2024-03-28T22:21:43.053845+0000 (oldest deadline 2024-03-28T22:22:08.353799+0000) 2024-03-28T22:22:16.685 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:16 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:16.286+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.654922+0000 front 2024-03-28T22:21:50.655059+0000 (oldest deadline 2024-03-28T22:22:13.554819+0000) 2024-03-28T22:22:16.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:16 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:16.587+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:41.373318+0000 front 2024-03-28T22:21:41.373414+0000 (oldest deadline 2024-03-28T22:22:06.673341+0000) 2024-03-28T22:22:16.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:16 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:16.587+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:47.774560+0000 front 2024-03-28T22:21:47.774226+0000 (oldest deadline 2024-03-28T22:22:13.674264+0000) 2024-03-28T22:22:17.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:16 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:16.736+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.211688+0000 front 2024-03-28T22:21:43.211660+0000 (oldest deadline 2024-03-28T22:22:07.311798+0000) 2024-03-28T22:22:17.184 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:16 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:16.803+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:17.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:16 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:16.803+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:17.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:16 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:16.803+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.710716+0000 front 2024-03-28T22:21:50.710773+0000 (oldest deadline 2024-03-28T22:22:13.610874+0000) 2024-03-28T22:22:17.188 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:16 smithi063 podman[73144]: 2024-03-28 22:22:16.933659414 +0000 UTC m=+5.479057279 container died 9835696fdd4019d4c86133818f621e36888e06a1afef72224ab42629e246c705 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5, RELEASE=squid-323d23d, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T22:22:17.189 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:16 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:16.969+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:17.189 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:16 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:16.969+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:17.189 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:16 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:16.969+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:17.334 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:17 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:17.289+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:17.334 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:17 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:17.289+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:17.334 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:17 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:17.289+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:17.439 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:17 smithi063 podman[73144]: 2024-03-28 22:22:17.438198007 +0000 UTC m=+5.983595842 container cleanup 9835696fdd4019d4c86133818f621e36888e06a1afef72224ab42629e246c705 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-323d23d, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-28T22:22:17.440 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:17 smithi063 bash[73144]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5 2024-03-28T22:22:17.685 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:17 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:17.303+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.652839+0000 front 2024-03-28T22:21:33.652795+0000 (oldest deadline 2024-03-28T22:21:58.352870+0000) 2024-03-28T22:22:17.685 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:17 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:17.303+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.053957+0000 front 2024-03-28T22:21:43.053845+0000 (oldest deadline 2024-03-28T22:22:08.353799+0000) 2024-03-28T22:22:17.686 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:17 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:17.303+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.654922+0000 front 2024-03-28T22:21:50.655059+0000 (oldest deadline 2024-03-28T22:22:13.554819+0000) 2024-03-28T22:22:17.931 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:17 smithi063 podman[73160]: 2024-03-28 22:22:17.66484018 +0000 UTC m=+0.728762022 container remove 9835696fdd4019d4c86133818f621e36888e06a1afef72224ab42629e246c705 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-323d23d, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux ) 2024-03-28T22:22:17.931 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:17 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:17.595+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:41.373318+0000 front 2024-03-28T22:21:41.373414+0000 (oldest deadline 2024-03-28T22:22:06.673341+0000) 2024-03-28T22:22:17.931 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:17 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:17.595+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:47.774560+0000 front 2024-03-28T22:21:47.774226+0000 (oldest deadline 2024-03-28T22:22:13.674264+0000) 2024-03-28T22:22:18.045 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:17 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:17.711+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.211688+0000 front 2024-03-28T22:21:43.211660+0000 (oldest deadline 2024-03-28T22:22:07.311798+0000) 2024-03-28T22:22:18.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:17 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:17.711+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:51.313162+0000 front 2024-03-28T22:21:51.312999+0000 (oldest deadline 2024-03-28T22:22:17.213156+0000) 2024-03-28T22:22:18.184 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:17 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:17.839+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:18.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:17 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:17.839+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:18.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:17 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:17.839+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.710716+0000 front 2024-03-28T22:21:50.710773+0000 (oldest deadline 2024-03-28T22:22:13.610874+0000) 2024-03-28T22:22:18.292 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:18 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:18.012+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:18.293 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:18 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:18.012+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:18.293 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:18 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:18.012+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:18.293 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:17 smithi063 podman[73252]: 2024-03-28 22:22:17.882338298 +0000 UTC m=+0.018405800 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:22:18.294 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:17 smithi063 podman[73252]: 2024-03-28 22:22:17.998947658 +0000 UTC m=+0.135015142 container create a37e1bc20c86af1fbf083d2be51c5155e95dc4cb63d623638e004a5120a5b7ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-323d23d, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T22:22:18.294 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:18 smithi063 podman[73252]: 2024-03-28 22:22:18.209835392 +0000 UTC m=+0.345902868 container init a37e1bc20c86af1fbf083d2be51c5155e95dc4cb63d623638e004a5120a5b7ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-323d23d, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-28T22:22:18.294 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:18 smithi063 podman[73252]: 2024-03-28 22:22:18.212972932 +0000 UTC m=+0.349040408 container start a37e1bc20c86af1fbf083d2be51c5155e95dc4cb63d623638e004a5120a5b7ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5-deactivate, RELEASE=squid-323d23d, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-28T22:22:18.294 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:18 smithi063 podman[73252]: 2024-03-28 22:22:18.292973593 +0000 UTC m=+0.429041094 container attach a37e1bc20c86af1fbf083d2be51c5155e95dc4cb63d623638e004a5120a5b7ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-323d23d, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_CLEAN=True) 2024-03-28T22:22:18.637 INFO:journalctl@ceph.osd.5.smithi063.stdout:Mar 28 22:22:18 smithi063 podman[73252]: 2024-03-28 22:22:18.513638114 +0000 UTC m=+0.649705614 container died a37e1bc20c86af1fbf083d2be51c5155e95dc4cb63d623638e004a5120a5b7ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-323d23d, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-28T22:22:18.684 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:18 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:18.308+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.652839+0000 front 2024-03-28T22:21:33.652795+0000 (oldest deadline 2024-03-28T22:21:58.352870+0000) 2024-03-28T22:22:18.684 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:18 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:18.308+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.053957+0000 front 2024-03-28T22:21:43.053845+0000 (oldest deadline 2024-03-28T22:22:08.353799+0000) 2024-03-28T22:22:18.685 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:18 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:18.308+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.654922+0000 front 2024-03-28T22:21:50.655059+0000 (oldest deadline 2024-03-28T22:22:13.554819+0000) 2024-03-28T22:22:18.736 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:18 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:18.304+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:18.737 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:18 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:18.304+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:18.737 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:18 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:18.304+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:18.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:18 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:18.637+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:41.373318+0000 front 2024-03-28T22:21:41.373414+0000 (oldest deadline 2024-03-28T22:22:06.673341+0000) 2024-03-28T22:22:18.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:18 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:18.637+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:47.774560+0000 front 2024-03-28T22:21:47.774226+0000 (oldest deadline 2024-03-28T22:22:13.674264+0000) 2024-03-28T22:22:19.045 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:18 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:18.736+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.211688+0000 front 2024-03-28T22:21:43.211660+0000 (oldest deadline 2024-03-28T22:22:07.311798+0000) 2024-03-28T22:22:19.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:18 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:18.736+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:51.313162+0000 front 2024-03-28T22:21:51.312999+0000 (oldest deadline 2024-03-28T22:22:17.213156+0000) 2024-03-28T22:22:19.184 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:18 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:18.842+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:19.184 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:18 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:18.842+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:19.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:18 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:18.842+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.710716+0000 front 2024-03-28T22:21:50.710773+0000 (oldest deadline 2024-03-28T22:22:13.610874+0000) 2024-03-28T22:22:19.217 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.5.service' 2024-03-28T22:22:19.267 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:19 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:19.007+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:19.267 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:19 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:19.007+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:19.267 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:19 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:19.007+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:19.269 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:22:19.269 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-28T22:22:19.269 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-28T22:22:19.269 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.6 2024-03-28T22:22:19.579 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:19 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:19.288+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.652839+0000 front 2024-03-28T22:21:33.652795+0000 (oldest deadline 2024-03-28T22:21:58.352870+0000) 2024-03-28T22:22:19.579 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:19 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:19.288+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.053957+0000 front 2024-03-28T22:21:43.053845+0000 (oldest deadline 2024-03-28T22:22:08.353799+0000) 2024-03-28T22:22:19.579 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:19 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:19.288+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.654922+0000 front 2024-03-28T22:21:50.655059+0000 (oldest deadline 2024-03-28T22:22:13.554819+0000) 2024-03-28T22:22:19.579 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:19 smithi016 systemd[1]: Stopping Ceph osd.6 for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:22:19.754 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:19 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:19.328+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:19.755 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:19 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:19.328+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:19.755 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:19 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:19.328+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:19.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:19 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:19.618+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:41.373318+0000 front 2024-03-28T22:21:41.373414+0000 (oldest deadline 2024-03-28T22:22:06.673341+0000) 2024-03-28T22:22:19.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:19 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:19.618+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:47.774560+0000 front 2024-03-28T22:21:47.774226+0000 (oldest deadline 2024-03-28T22:22:13.674264+0000) 2024-03-28T22:22:19.934 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:19 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:19.814+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:19.934 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:19 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:19.814+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:19.935 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:19 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:19.814+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.710716+0000 front 2024-03-28T22:21:50.710773+0000 (oldest deadline 2024-03-28T22:22:13.610874+0000) 2024-03-28T22:22:19.935 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:19 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:19.577+0000 7f91d83e5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:22:19.935 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:19 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:19.577+0000 7f91d83e5700 -1 osd.6 102 *** Got signal Terminated *** 2024-03-28T22:22:19.935 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:19 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:19.577+0000 7f91d83e5700 -1 osd.6 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T22:22:20.045 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:19 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:19.754+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.211688+0000 front 2024-03-28T22:21:43.211660+0000 (oldest deadline 2024-03-28T22:22:07.311798+0000) 2024-03-28T22:22:20.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:19 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:19.754+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:51.313162+0000 front 2024-03-28T22:21:51.312999+0000 (oldest deadline 2024-03-28T22:22:17.213156+0000) 2024-03-28T22:22:20.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:20 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:20.034+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:20.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:20 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:20.034+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:20.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:20 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:20.034+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:20.685 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:20 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:20.319+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.652839+0000 front 2024-03-28T22:21:33.652795+0000 (oldest deadline 2024-03-28T22:21:58.352870+0000) 2024-03-28T22:22:20.685 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:20 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:20.319+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.053957+0000 front 2024-03-28T22:21:43.053845+0000 (oldest deadline 2024-03-28T22:22:08.353799+0000) 2024-03-28T22:22:20.685 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:20 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:20.319+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.654922+0000 front 2024-03-28T22:21:50.655059+0000 (oldest deadline 2024-03-28T22:22:13.554819+0000) 2024-03-28T22:22:20.790 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:20 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:20.312+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:20.790 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:20 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:20.312+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:20.790 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:20 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:20.312+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:20.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:20 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:20.595+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:41.373318+0000 front 2024-03-28T22:21:41.373414+0000 (oldest deadline 2024-03-28T22:22:06.673341+0000) 2024-03-28T22:22:20.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:20 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:20.595+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:47.774560+0000 front 2024-03-28T22:21:47.774226+0000 (oldest deadline 2024-03-28T22:22:13.674264+0000) 2024-03-28T22:22:21.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:20 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:20.790+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.211688+0000 front 2024-03-28T22:21:43.211660+0000 (oldest deadline 2024-03-28T22:22:07.311798+0000) 2024-03-28T22:22:21.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:20 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:20.790+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:51.313162+0000 front 2024-03-28T22:21:51.312999+0000 (oldest deadline 2024-03-28T22:22:17.213156+0000) 2024-03-28T22:22:21.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:20 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:20.799+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:21.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:20 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:20.799+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:21.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:20 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:20.799+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.710716+0000 front 2024-03-28T22:21:50.710773+0000 (oldest deadline 2024-03-28T22:22:13.610874+0000) 2024-03-28T22:22:21.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:21 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:21.036+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:21.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:21 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:21.036+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:21.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:21 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:21.036+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:21.685 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:21 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:21.344+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.652839+0000 front 2024-03-28T22:21:33.652795+0000 (oldest deadline 2024-03-28T22:21:58.352870+0000) 2024-03-28T22:22:21.685 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:21 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:21.344+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.053957+0000 front 2024-03-28T22:21:43.053845+0000 (oldest deadline 2024-03-28T22:22:08.353799+0000) 2024-03-28T22:22:21.685 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:21 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:21.344+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.654922+0000 front 2024-03-28T22:21:50.655059+0000 (oldest deadline 2024-03-28T22:22:13.554819+0000) 2024-03-28T22:22:21.764 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:21 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:21.308+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:21.765 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:21 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:21.308+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:21.765 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:21 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:21.308+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:21.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:21 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:21.613+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:41.373318+0000 front 2024-03-28T22:21:41.373414+0000 (oldest deadline 2024-03-28T22:22:06.673341+0000) 2024-03-28T22:22:21.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:21 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:21.613+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:47.774560+0000 front 2024-03-28T22:21:47.774226+0000 (oldest deadline 2024-03-28T22:22:13.674264+0000) 2024-03-28T22:22:22.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:21 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:21.764+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.211688+0000 front 2024-03-28T22:21:43.211660+0000 (oldest deadline 2024-03-28T22:22:07.311798+0000) 2024-03-28T22:22:22.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:21 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:21.764+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:51.313162+0000 front 2024-03-28T22:21:51.312999+0000 (oldest deadline 2024-03-28T22:22:17.213156+0000) 2024-03-28T22:22:22.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:21 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:21.764+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:59.513990+0000 front 2024-03-28T22:21:59.514141+0000 (oldest deadline 2024-03-28T22:22:21.214162+0000) 2024-03-28T22:22:22.184 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:21 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:21.796+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:22.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:21 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:21.796+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:22.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:21 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:21.797+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.710716+0000 front 2024-03-28T22:21:50.710773+0000 (oldest deadline 2024-03-28T22:22:13.610874+0000) 2024-03-28T22:22:22.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:22 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:22.028+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:22.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:22 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:22.028+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:22.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:22 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:22.028+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:22.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:22 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:22.286+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:22.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:22 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:22.286+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:22.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:22 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:22.286+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:22.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:22 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:22.286+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:57.638872+0000 front 2024-03-28T22:21:57.638939+0000 (oldest deadline 2024-03-28T22:22:21.739096+0000) 2024-03-28T22:22:22.685 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:22 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:22.344+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.652839+0000 front 2024-03-28T22:21:33.652795+0000 (oldest deadline 2024-03-28T22:21:58.352870+0000) 2024-03-28T22:22:22.685 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:22 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:22.344+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.053957+0000 front 2024-03-28T22:21:43.053845+0000 (oldest deadline 2024-03-28T22:22:08.353799+0000) 2024-03-28T22:22:22.685 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:22 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:22.344+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.654922+0000 front 2024-03-28T22:21:50.655059+0000 (oldest deadline 2024-03-28T22:22:13.554819+0000) 2024-03-28T22:22:22.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:22 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:22.573+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:41.373318+0000 front 2024-03-28T22:21:41.373414+0000 (oldest deadline 2024-03-28T22:22:06.673341+0000) 2024-03-28T22:22:22.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:22 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:22.573+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:47.774560+0000 front 2024-03-28T22:21:47.774226+0000 (oldest deadline 2024-03-28T22:22:13.674264+0000) 2024-03-28T22:22:22.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:22 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:22.573+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:58.375448+0000 front 2024-03-28T22:21:58.375558+0000 (oldest deadline 2024-03-28T22:22:21.875338+0000) 2024-03-28T22:22:23.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:22 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:22.747+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.211688+0000 front 2024-03-28T22:21:43.211660+0000 (oldest deadline 2024-03-28T22:22:07.311798+0000) 2024-03-28T22:22:23.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:22 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:22.747+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:51.313162+0000 front 2024-03-28T22:21:51.312999+0000 (oldest deadline 2024-03-28T22:22:17.213156+0000) 2024-03-28T22:22:23.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:22 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:22.747+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:59.513990+0000 front 2024-03-28T22:21:59.514141+0000 (oldest deadline 2024-03-28T22:22:21.214162+0000) 2024-03-28T22:22:23.184 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:22 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:22.812+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:23.184 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:22 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:22.812+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:23.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:22 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:22.812+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.710716+0000 front 2024-03-28T22:21:50.710773+0000 (oldest deadline 2024-03-28T22:22:13.610874+0000) 2024-03-28T22:22:23.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:23 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:23.020+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:23.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:23 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:23.021+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:23.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:23 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:23.021+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:23.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:23 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:23.021+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:22:23.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:23 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:23.243+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:23.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:23 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:23.243+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:23.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:23 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:23.243+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:23.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:23 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:23.243+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:57.638872+0000 front 2024-03-28T22:21:57.638939+0000 (oldest deadline 2024-03-28T22:22:21.739096+0000) 2024-03-28T22:22:23.684 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:23 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:23.308+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.652839+0000 front 2024-03-28T22:21:33.652795+0000 (oldest deadline 2024-03-28T22:21:58.352870+0000) 2024-03-28T22:22:23.685 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:23 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:23.308+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.053957+0000 front 2024-03-28T22:21:43.053845+0000 (oldest deadline 2024-03-28T22:22:08.353799+0000) 2024-03-28T22:22:23.685 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:23 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:23.308+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.654922+0000 front 2024-03-28T22:21:50.655059+0000 (oldest deadline 2024-03-28T22:22:13.554819+0000) 2024-03-28T22:22:23.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:23 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:23.598+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:41.373318+0000 front 2024-03-28T22:21:41.373414+0000 (oldest deadline 2024-03-28T22:22:06.673341+0000) 2024-03-28T22:22:23.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:23 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:23.598+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:47.774560+0000 front 2024-03-28T22:21:47.774226+0000 (oldest deadline 2024-03-28T22:22:13.674264+0000) 2024-03-28T22:22:23.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:23 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:23.598+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:58.375448+0000 front 2024-03-28T22:21:58.375558+0000 (oldest deadline 2024-03-28T22:22:21.875338+0000) 2024-03-28T22:22:24.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:23 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:23.773+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.211688+0000 front 2024-03-28T22:21:43.211660+0000 (oldest deadline 2024-03-28T22:22:07.311798+0000) 2024-03-28T22:22:24.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:23 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:23.773+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:51.313162+0000 front 2024-03-28T22:21:51.312999+0000 (oldest deadline 2024-03-28T22:22:17.213156+0000) 2024-03-28T22:22:24.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:23 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:23.773+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:59.513990+0000 front 2024-03-28T22:21:59.514141+0000 (oldest deadline 2024-03-28T22:22:21.214162+0000) 2024-03-28T22:22:24.184 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:23 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:23.775+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:24.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:23 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:23.775+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:24.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:23 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:23.775+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.710716+0000 front 2024-03-28T22:21:50.710773+0000 (oldest deadline 2024-03-28T22:22:13.610874+0000) 2024-03-28T22:22:24.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:24 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:24.035+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:24.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:24 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:24.035+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:24.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:24 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:24.035+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:24.431 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:24 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:24.035+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:22:24.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:24 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:24.270+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:24.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:24 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:24.270+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:24.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:24 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:24.270+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:24.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:24 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:24.270+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:57.638872+0000 front 2024-03-28T22:21:57.638939+0000 (oldest deadline 2024-03-28T22:22:21.739096+0000) 2024-03-28T22:22:24.685 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:24 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:24.273+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.652839+0000 front 2024-03-28T22:21:33.652795+0000 (oldest deadline 2024-03-28T22:21:58.352870+0000) 2024-03-28T22:22:24.685 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:24 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:24.273+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.053957+0000 front 2024-03-28T22:21:43.053845+0000 (oldest deadline 2024-03-28T22:22:08.353799+0000) 2024-03-28T22:22:24.685 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:24 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6[58126]: 2024-03-28T22:22:24.273+0000 7f91d11d2700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.654922+0000 front 2024-03-28T22:21:50.655059+0000 (oldest deadline 2024-03-28T22:22:13.554819+0000) 2024-03-28T22:22:24.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:24 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:24.551+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:41.373318+0000 front 2024-03-28T22:21:41.373414+0000 (oldest deadline 2024-03-28T22:22:06.673341+0000) 2024-03-28T22:22:24.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:24 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:24.551+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:47.774560+0000 front 2024-03-28T22:21:47.774226+0000 (oldest deadline 2024-03-28T22:22:13.674264+0000) 2024-03-28T22:22:24.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:24 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:24.551+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:58.375448+0000 front 2024-03-28T22:21:58.375558+0000 (oldest deadline 2024-03-28T22:22:21.875338+0000) 2024-03-28T22:22:25.045 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:24 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:24.766+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.211688+0000 front 2024-03-28T22:21:43.211660+0000 (oldest deadline 2024-03-28T22:22:07.311798+0000) 2024-03-28T22:22:25.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:24 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:24.766+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:51.313162+0000 front 2024-03-28T22:21:51.312999+0000 (oldest deadline 2024-03-28T22:22:17.213156+0000) 2024-03-28T22:22:25.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:24 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:24.766+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:59.513990+0000 front 2024-03-28T22:21:59.514141+0000 (oldest deadline 2024-03-28T22:22:21.214162+0000) 2024-03-28T22:22:25.184 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:25 smithi016 podman[103822]: 2024-03-28 22:22:25.036361298 +0000 UTC m=+5.575229599 container died 905373d56540f5763ae58c0b9881c63879e39ab140ee58a2ac166fa71c15bf0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-323d23d, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T22:22:25.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:24 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:24.820+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:25.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:24 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:24.820+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:25.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:24 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:24.820+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.710716+0000 front 2024-03-28T22:21:50.710773+0000 (oldest deadline 2024-03-28T22:22:13.610874+0000) 2024-03-28T22:22:25.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:24 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:24.820+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:22:00.612619+0000 front 2024-03-28T22:22:00.612649+0000 (oldest deadline 2024-03-28T22:22:24.112843+0000) 2024-03-28T22:22:25.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:25 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:25.041+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:25.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:25 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:25.041+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:25.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:25 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:25.041+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:25.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:25 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:25.041+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:22:25.471 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:25 smithi016 podman[103822]: 2024-03-28 22:22:25.344856122 +0000 UTC m=+5.883724419 container cleanup 905373d56540f5763ae58c0b9881c63879e39ab140ee58a2ac166fa71c15bf0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-323d23d, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-28T22:22:25.471 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:25 smithi016 bash[103822]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6 2024-03-28T22:22:25.770 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:25 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:25.300+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:25.770 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:25 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:25.300+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:25.770 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:25 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:25.300+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:25.770 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:25 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:25.300+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:57.638872+0000 front 2024-03-28T22:21:57.638939+0000 (oldest deadline 2024-03-28T22:22:21.739096+0000) 2024-03-28T22:22:25.845 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:25 smithi016 podman[103839]: 2024-03-28 22:22:25.637716685 +0000 UTC m=+0.594912781 container remove 905373d56540f5763ae58c0b9881c63879e39ab140ee58a2ac166fa71c15bf0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-323d23d, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326) 2024-03-28T22:22:25.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:25 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:25.586+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:41.373318+0000 front 2024-03-28T22:21:41.373414+0000 (oldest deadline 2024-03-28T22:22:06.673341+0000) 2024-03-28T22:22:25.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:25 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:25.586+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:47.774560+0000 front 2024-03-28T22:21:47.774226+0000 (oldest deadline 2024-03-28T22:22:13.674264+0000) 2024-03-28T22:22:25.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:25 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:25.586+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:58.375448+0000 front 2024-03-28T22:21:58.375558+0000 (oldest deadline 2024-03-28T22:22:21.875338+0000) 2024-03-28T22:22:26.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:25 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:25.770+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.211688+0000 front 2024-03-28T22:21:43.211660+0000 (oldest deadline 2024-03-28T22:22:07.311798+0000) 2024-03-28T22:22:26.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:25 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:25.770+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:51.313162+0000 front 2024-03-28T22:21:51.312999+0000 (oldest deadline 2024-03-28T22:22:17.213156+0000) 2024-03-28T22:22:26.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:25 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:25.770+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:59.513990+0000 front 2024-03-28T22:21:59.514141+0000 (oldest deadline 2024-03-28T22:22:21.214162+0000) 2024-03-28T22:22:26.121 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:25 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:25.843+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:26.122 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:25 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:25.843+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:26.122 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:25 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:25.843+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.710716+0000 front 2024-03-28T22:21:50.710773+0000 (oldest deadline 2024-03-28T22:22:13.610874+0000) 2024-03-28T22:22:26.122 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:25 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:25.843+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:22:00.612619+0000 front 2024-03-28T22:22:00.612649+0000 (oldest deadline 2024-03-28T22:22:24.112843+0000) 2024-03-28T22:22:26.416 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:26 smithi016 podman[103941]: 2024-03-28 22:22:26.025643769 +0000 UTC m=+0.018286129 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:22:26.416 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:26 smithi016 podman[103941]: 2024-03-28 22:22:26.138720675 +0000 UTC m=+0.131363014 container create 872856c0fb3958ff99b004f753271e80fd9d2069a9e12ea588404750c68e3c30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=squid-323d23d, ceph=True) 2024-03-28T22:22:26.416 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:26 smithi016 podman[103941]: 2024-03-28 22:22:26.349628295 +0000 UTC m=+0.342270635 container init 872856c0fb3958ff99b004f753271e80fd9d2069a9e12ea588404750c68e3c30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-323d23d, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326) 2024-03-28T22:22:26.416 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:26 smithi016 podman[103941]: 2024-03-28 22:22:26.352684127 +0000 UTC m=+0.345326467 container start 872856c0fb3958ff99b004f753271e80fd9d2069a9e12ea588404750c68e3c30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-323d23d, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-28T22:22:26.417 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:26 smithi016 podman[103941]: 2024-03-28 22:22:26.416271259 +0000 UTC m=+0.408913608 container attach 872856c0fb3958ff99b004f753271e80fd9d2069a9e12ea588404750c68e3c30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=squid-323d23d, ceph=True) 2024-03-28T22:22:26.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:26 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:26.051+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:26.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:26 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:26.051+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:26.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:26 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:26.051+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:26.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:26 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:26.051+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:22:26.684 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:26 smithi016 podman[103941]: 2024-03-28 22:22:26.662025989 +0000 UTC m=+0.654668346 container died 872856c0fb3958ff99b004f753271e80fd9d2069a9e12ea588404750c68e3c30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=squid-323d23d, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-28T22:22:26.753 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:26 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:26.341+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:26.754 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:26 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:26.341+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:26.754 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:26 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:26.341+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:26.754 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:26 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:26.341+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:57.638872+0000 front 2024-03-28T22:21:57.638939+0000 (oldest deadline 2024-03-28T22:22:21.739096+0000) 2024-03-28T22:22:26.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:26 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:26.589+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:41.373318+0000 front 2024-03-28T22:21:41.373414+0000 (oldest deadline 2024-03-28T22:22:06.673341+0000) 2024-03-28T22:22:26.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:26 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:26.589+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:47.774560+0000 front 2024-03-28T22:21:47.774226+0000 (oldest deadline 2024-03-28T22:22:13.674264+0000) 2024-03-28T22:22:26.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:26 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:26.589+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:58.375448+0000 front 2024-03-28T22:21:58.375558+0000 (oldest deadline 2024-03-28T22:22:21.875338+0000) 2024-03-28T22:22:27.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:26 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:26.751+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.211688+0000 front 2024-03-28T22:21:43.211660+0000 (oldest deadline 2024-03-28T22:22:07.311798+0000) 2024-03-28T22:22:27.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:26 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:26.751+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:51.313162+0000 front 2024-03-28T22:21:51.312999+0000 (oldest deadline 2024-03-28T22:22:17.213156+0000) 2024-03-28T22:22:27.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:26 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:26.751+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:59.513990+0000 front 2024-03-28T22:21:59.514141+0000 (oldest deadline 2024-03-28T22:22:21.214162+0000) 2024-03-28T22:22:27.102 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:26 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:26.830+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:27.102 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:26 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:26.830+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:27.102 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:26 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:26.830+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.710716+0000 front 2024-03-28T22:21:50.710773+0000 (oldest deadline 2024-03-28T22:22:13.610874+0000) 2024-03-28T22:22:27.103 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:26 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:26.830+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:22:00.612619+0000 front 2024-03-28T22:22:00.612649+0000 (oldest deadline 2024-03-28T22:22:24.112843+0000) 2024-03-28T22:22:27.351 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.6.service' 2024-03-28T22:22:27.378 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:27 smithi016 podman[103969]: 2024-03-28 22:22:27.336184237 +0000 UTC m=+0.665328804 container remove 872856c0fb3958ff99b004f753271e80fd9d2069a9e12ea588404750c68e3c30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-323d23d, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-28T22:22:27.378 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:27 smithi016 systemd[1]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.6.service: Deactivated successfully. 2024-03-28T22:22:27.378 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:27 smithi016 systemd[1]: Stopped Ceph osd.6 for 798d9884-ed4e-11ee-95d0-87774f69a715. 2024-03-28T22:22:27.378 INFO:journalctl@ceph.osd.6.smithi016.stdout:Mar 28 22:22:27 smithi016 systemd[1]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.6.service: Consumed 15.093s CPU time. 2024-03-28T22:22:27.393 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:22:27.393 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-28T22:22:27.394 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-28T22:22:27.394 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.7 2024-03-28T22:22:27.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:27 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:27.034+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:27.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:27 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:27.034+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:27.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:27 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:27.034+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:27.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:27 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:27.034+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:22:27.692 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:27 smithi053 systemd[1]: Stopping Ceph osd.7 for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:22:27.693 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:27 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:27.299+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:27.693 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:27 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:27.299+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:27.693 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:27 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:27.299+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:27.693 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:27 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:27.299+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:57.638872+0000 front 2024-03-28T22:21:57.638939+0000 (oldest deadline 2024-03-28T22:22:21.739096+0000) 2024-03-28T22:22:27.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:27 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:27.550+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:41.373318+0000 front 2024-03-28T22:21:41.373414+0000 (oldest deadline 2024-03-28T22:22:06.673341+0000) 2024-03-28T22:22:27.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:27 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:27.550+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:47.774560+0000 front 2024-03-28T22:21:47.774226+0000 (oldest deadline 2024-03-28T22:22:13.674264+0000) 2024-03-28T22:22:27.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:27 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:27.550+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:58.375448+0000 front 2024-03-28T22:21:58.375558+0000 (oldest deadline 2024-03-28T22:22:21.875338+0000) 2024-03-28T22:22:28.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:27 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:27.692+0000 7f10a8a47700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:22:28.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:27 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:27.692+0000 7f10a8a47700 -1 osd.7 102 *** Got signal Terminated *** 2024-03-28T22:22:28.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:27 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:27.692+0000 7f10a8a47700 -1 osd.7 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T22:22:28.047 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:27 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:27.710+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.211688+0000 front 2024-03-28T22:21:43.211660+0000 (oldest deadline 2024-03-28T22:22:07.311798+0000) 2024-03-28T22:22:28.047 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:27 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:27.710+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:51.313162+0000 front 2024-03-28T22:21:51.312999+0000 (oldest deadline 2024-03-28T22:22:17.213156+0000) 2024-03-28T22:22:28.047 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:27 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:27.710+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:59.513990+0000 front 2024-03-28T22:21:59.514141+0000 (oldest deadline 2024-03-28T22:22:21.214162+0000) 2024-03-28T22:22:28.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:27 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:27.879+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:28.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:27 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:27.879+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:28.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:27 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:27.879+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.710716+0000 front 2024-03-28T22:21:50.710773+0000 (oldest deadline 2024-03-28T22:22:13.610874+0000) 2024-03-28T22:22:28.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:27 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:27.879+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:22:00.612619+0000 front 2024-03-28T22:22:00.612649+0000 (oldest deadline 2024-03-28T22:22:24.112843+0000) 2024-03-28T22:22:28.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:28 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:28.015+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:28.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:28 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:28.015+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:28.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:28 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:28.015+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:28.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:28 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:28.015+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:22:28.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:28 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:28.268+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:28.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:28 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:28.268+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:28.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:28 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:28.268+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:28.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:28 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:28.268+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:57.638872+0000 front 2024-03-28T22:21:57.638939+0000 (oldest deadline 2024-03-28T22:22:21.739096+0000) 2024-03-28T22:22:28.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:28 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:28.574+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:41.373318+0000 front 2024-03-28T22:21:41.373414+0000 (oldest deadline 2024-03-28T22:22:06.673341+0000) 2024-03-28T22:22:28.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:28 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:28.574+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:47.774560+0000 front 2024-03-28T22:21:47.774226+0000 (oldest deadline 2024-03-28T22:22:13.674264+0000) 2024-03-28T22:22:28.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:28 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:28.574+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:58.375448+0000 front 2024-03-28T22:21:58.375558+0000 (oldest deadline 2024-03-28T22:22:21.875338+0000) 2024-03-28T22:22:29.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:28 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:28.761+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.211688+0000 front 2024-03-28T22:21:43.211660+0000 (oldest deadline 2024-03-28T22:22:07.311798+0000) 2024-03-28T22:22:29.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:28 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:28.761+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:51.313162+0000 front 2024-03-28T22:21:51.312999+0000 (oldest deadline 2024-03-28T22:22:17.213156+0000) 2024-03-28T22:22:29.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:28 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:28.761+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:59.513990+0000 front 2024-03-28T22:21:59.514141+0000 (oldest deadline 2024-03-28T22:22:21.214162+0000) 2024-03-28T22:22:29.184 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:28 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:28.903+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:29.184 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:28 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:28.903+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:29.184 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:28 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:28.903+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.710716+0000 front 2024-03-28T22:21:50.710773+0000 (oldest deadline 2024-03-28T22:22:13.610874+0000) 2024-03-28T22:22:29.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:28 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:28.903+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:22:00.612619+0000 front 2024-03-28T22:22:00.612649+0000 (oldest deadline 2024-03-28T22:22:24.112843+0000) 2024-03-28T22:22:29.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:28 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:28.976+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:29.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:28 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:28.976+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:29.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:28 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:28.976+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:29.431 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:28 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:28.976+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:22:29.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:29 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:29.275+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:29.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:29 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:29.275+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:29.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:29 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:29.275+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:29.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:29 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:29.275+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:57.638872+0000 front 2024-03-28T22:21:57.638939+0000 (oldest deadline 2024-03-28T22:22:21.739096+0000) 2024-03-28T22:22:29.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:29 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:29.557+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:41.373318+0000 front 2024-03-28T22:21:41.373414+0000 (oldest deadline 2024-03-28T22:22:06.673341+0000) 2024-03-28T22:22:29.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:29 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:29.557+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:47.774560+0000 front 2024-03-28T22:21:47.774226+0000 (oldest deadline 2024-03-28T22:22:13.674264+0000) 2024-03-28T22:22:29.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:29 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:29.557+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:58.375448+0000 front 2024-03-28T22:21:58.375558+0000 (oldest deadline 2024-03-28T22:22:21.875338+0000) 2024-03-28T22:22:30.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:29 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:29.729+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.211688+0000 front 2024-03-28T22:21:43.211660+0000 (oldest deadline 2024-03-28T22:22:07.311798+0000) 2024-03-28T22:22:30.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:29 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:29.729+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:51.313162+0000 front 2024-03-28T22:21:51.312999+0000 (oldest deadline 2024-03-28T22:22:17.213156+0000) 2024-03-28T22:22:30.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:29 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:29.729+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:59.513990+0000 front 2024-03-28T22:21:59.514141+0000 (oldest deadline 2024-03-28T22:22:21.214162+0000) 2024-03-28T22:22:30.184 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:29 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:29.906+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:30.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:29 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:29.906+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:30.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:29 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:29.906+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.710716+0000 front 2024-03-28T22:21:50.710773+0000 (oldest deadline 2024-03-28T22:22:13.610874+0000) 2024-03-28T22:22:30.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:29 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:29.906+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:22:00.612619+0000 front 2024-03-28T22:22:00.612649+0000 (oldest deadline 2024-03-28T22:22:24.112843+0000) 2024-03-28T22:22:30.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:30 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:30.024+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:30.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:30 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:30.024+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:30.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:30 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:30.024+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:30.431 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:30 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:30.024+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:22:30.545 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:30 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:30.277+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:30.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:30 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:30.277+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:30.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:30 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:30.277+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:30.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:30 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:30.277+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:57.638872+0000 front 2024-03-28T22:21:57.638939+0000 (oldest deadline 2024-03-28T22:22:21.739096+0000) 2024-03-28T22:22:30.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:30 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:30.540+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:41.373318+0000 front 2024-03-28T22:21:41.373414+0000 (oldest deadline 2024-03-28T22:22:06.673341+0000) 2024-03-28T22:22:30.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:30 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:30.540+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:47.774560+0000 front 2024-03-28T22:21:47.774226+0000 (oldest deadline 2024-03-28T22:22:13.674264+0000) 2024-03-28T22:22:30.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:30 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:30.540+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:58.375448+0000 front 2024-03-28T22:21:58.375558+0000 (oldest deadline 2024-03-28T22:22:21.875338+0000) 2024-03-28T22:22:31.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:30 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:30.746+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.211688+0000 front 2024-03-28T22:21:43.211660+0000 (oldest deadline 2024-03-28T22:22:07.311798+0000) 2024-03-28T22:22:31.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:30 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:30.746+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:51.313162+0000 front 2024-03-28T22:21:51.312999+0000 (oldest deadline 2024-03-28T22:22:17.213156+0000) 2024-03-28T22:22:31.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:30 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:30.746+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:59.513990+0000 front 2024-03-28T22:21:59.514141+0000 (oldest deadline 2024-03-28T22:22:21.214162+0000) 2024-03-28T22:22:31.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:30 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:30.746+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:06.916205+0000 front 2024-03-28T22:22:06.916252+0000 (oldest deadline 2024-03-28T22:22:29.816439+0000) 2024-03-28T22:22:31.184 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:30 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:30.863+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:31.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:30 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:30.863+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:31.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:30 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:30.863+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.710716+0000 front 2024-03-28T22:21:50.710773+0000 (oldest deadline 2024-03-28T22:22:13.610874+0000) 2024-03-28T22:22:31.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:30 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:30.863+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:22:00.612619+0000 front 2024-03-28T22:22:00.612649+0000 (oldest deadline 2024-03-28T22:22:24.112843+0000) 2024-03-28T22:22:31.186 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:30 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:30.863+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:04.113537+0000 front 2024-03-28T22:22:04.113671+0000 (oldest deadline 2024-03-28T22:22:30.013514+0000) 2024-03-28T22:22:31.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:31 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:31.034+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:31.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:31 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:31.034+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:31.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:31 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:31.034+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:31.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:31 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:31.034+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:22:31.737 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:31 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:31.313+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:31.737 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:31 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:31.313+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:31.738 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:31 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:31.313+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:31.738 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:31 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:31.313+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:57.638872+0000 front 2024-03-28T22:21:57.638939+0000 (oldest deadline 2024-03-28T22:22:21.739096+0000) 2024-03-28T22:22:31.738 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:31 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:31.313+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:05.140891+0000 front 2024-03-28T22:22:05.140920+0000 (oldest deadline 2024-03-28T22:22:30.441027+0000) 2024-03-28T22:22:31.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:31 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:31.545+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:41.373318+0000 front 2024-03-28T22:21:41.373414+0000 (oldest deadline 2024-03-28T22:22:06.673341+0000) 2024-03-28T22:22:31.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:31 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:31.545+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:47.774560+0000 front 2024-03-28T22:21:47.774226+0000 (oldest deadline 2024-03-28T22:22:13.674264+0000) 2024-03-28T22:22:31.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:31 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:31.545+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:58.375448+0000 front 2024-03-28T22:21:58.375558+0000 (oldest deadline 2024-03-28T22:22:21.875338+0000) 2024-03-28T22:22:32.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:31 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:31.737+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.211688+0000 front 2024-03-28T22:21:43.211660+0000 (oldest deadline 2024-03-28T22:22:07.311798+0000) 2024-03-28T22:22:32.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:31 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:31.737+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:51.313162+0000 front 2024-03-28T22:21:51.312999+0000 (oldest deadline 2024-03-28T22:22:17.213156+0000) 2024-03-28T22:22:32.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:31 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:31.737+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:59.513990+0000 front 2024-03-28T22:21:59.514141+0000 (oldest deadline 2024-03-28T22:22:21.214162+0000) 2024-03-28T22:22:32.046 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:31 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7[54227]: 2024-03-28T22:22:31.737+0000 7f10a1834700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:06.916205+0000 front 2024-03-28T22:22:06.916252+0000 (oldest deadline 2024-03-28T22:22:29.816439+0000) 2024-03-28T22:22:32.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:31 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:31.868+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:32.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:31 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:31.868+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:32.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:31 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:31.868+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.710716+0000 front 2024-03-28T22:21:50.710773+0000 (oldest deadline 2024-03-28T22:22:13.610874+0000) 2024-03-28T22:22:32.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:31 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:31.868+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:22:00.612619+0000 front 2024-03-28T22:22:00.612649+0000 (oldest deadline 2024-03-28T22:22:24.112843+0000) 2024-03-28T22:22:32.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:31 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:31.868+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:04.113537+0000 front 2024-03-28T22:22:04.113671+0000 (oldest deadline 2024-03-28T22:22:30.013514+0000) 2024-03-28T22:22:32.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:31 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:31.994+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:32.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:31 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:31.994+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:32.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:31 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:31.994+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:32.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:31 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:31.994+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:22:32.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:32 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:32.283+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:32.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:32 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:32.283+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:32.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:32 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:32.283+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:32.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:32 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:32.283+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:57.638872+0000 front 2024-03-28T22:21:57.638939+0000 (oldest deadline 2024-03-28T22:22:21.739096+0000) 2024-03-28T22:22:32.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:32 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:32.283+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:05.140891+0000 front 2024-03-28T22:22:05.140920+0000 (oldest deadline 2024-03-28T22:22:30.441027+0000) 2024-03-28T22:22:32.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:32 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:32.561+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:41.373318+0000 front 2024-03-28T22:21:41.373414+0000 (oldest deadline 2024-03-28T22:22:06.673341+0000) 2024-03-28T22:22:32.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:32 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:32.561+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:47.774560+0000 front 2024-03-28T22:21:47.774226+0000 (oldest deadline 2024-03-28T22:22:13.674264+0000) 2024-03-28T22:22:32.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:32 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:32.561+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:58.375448+0000 front 2024-03-28T22:21:58.375558+0000 (oldest deadline 2024-03-28T22:22:21.875338+0000) 2024-03-28T22:22:33.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:32 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:32.863+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:33.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:32 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:32.863+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:33.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:32 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:32.863+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.710716+0000 front 2024-03-28T22:21:50.710773+0000 (oldest deadline 2024-03-28T22:22:13.610874+0000) 2024-03-28T22:22:33.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:32 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:32.863+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:22:00.612619+0000 front 2024-03-28T22:22:00.612649+0000 (oldest deadline 2024-03-28T22:22:24.112843+0000) 2024-03-28T22:22:33.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:32 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:32.863+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:04.113537+0000 front 2024-03-28T22:22:04.113671+0000 (oldest deadline 2024-03-28T22:22:30.013514+0000) 2024-03-28T22:22:33.394 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:33 smithi053 podman[72411]: 2024-03-28 22:22:33.127386686 +0000 UTC m=+5.532442707 container died 0d27146c6f1206d8be3a88d6525073df4a44885cc5831a5daf1327df34d17d12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-323d23d, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T22:22:33.395 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:33 smithi053 podman[72411]: 2024-03-28 22:22:33.375624433 +0000 UTC m=+5.780680466 container cleanup 0d27146c6f1206d8be3a88d6525073df4a44885cc5831a5daf1327df34d17d12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-323d23d, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS) 2024-03-28T22:22:33.395 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:33 smithi053 bash[72411]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7 2024-03-28T22:22:33.395 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:33 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:33.278+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:33.395 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:33 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:33.278+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:33.395 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:33 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:33.278+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:33.395 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:33 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:33.278+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:57.638872+0000 front 2024-03-28T22:21:57.638939+0000 (oldest deadline 2024-03-28T22:22:21.739096+0000) 2024-03-28T22:22:33.395 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:33 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:33.278+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:05.140891+0000 front 2024-03-28T22:22:05.140920+0000 (oldest deadline 2024-03-28T22:22:30.441027+0000) 2024-03-28T22:22:33.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:33 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:33.004+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:33.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:33 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:33.004+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:33.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:33 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:33.004+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:33.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:33 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:33.004+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:22:33.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:33 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:33.515+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:41.373318+0000 front 2024-03-28T22:21:41.373414+0000 (oldest deadline 2024-03-28T22:22:06.673341+0000) 2024-03-28T22:22:33.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:33 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:33.515+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:47.774560+0000 front 2024-03-28T22:21:47.774226+0000 (oldest deadline 2024-03-28T22:22:13.674264+0000) 2024-03-28T22:22:33.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:33 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:33.515+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:58.375448+0000 front 2024-03-28T22:21:58.375558+0000 (oldest deadline 2024-03-28T22:22:21.875338+0000) 2024-03-28T22:22:33.977 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:33 smithi053 podman[72428]: 2024-03-28 22:22:33.6604316 +0000 UTC m=+0.530788045 container remove 0d27146c6f1206d8be3a88d6525073df4a44885cc5831a5daf1327df34d17d12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-323d23d, org.label-schema.vendor=CentOS) 2024-03-28T22:22:34.184 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:33 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:33.866+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:34.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:33 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:33.866+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:34.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:33 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:33.866+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.710716+0000 front 2024-03-28T22:21:50.710773+0000 (oldest deadline 2024-03-28T22:22:13.610874+0000) 2024-03-28T22:22:34.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:33 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:33.866+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:22:00.612619+0000 front 2024-03-28T22:22:00.612649+0000 (oldest deadline 2024-03-28T22:22:24.112843+0000) 2024-03-28T22:22:34.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:33 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:33.866+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:04.113537+0000 front 2024-03-28T22:22:04.113671+0000 (oldest deadline 2024-03-28T22:22:30.013514+0000) 2024-03-28T22:22:34.271 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:33 smithi053 podman[72517]: 2024-03-28 22:22:33.881480988 +0000 UTC m=+0.018306813 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:22:34.272 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:33 smithi053 podman[72517]: 2024-03-28 22:22:33.994483046 +0000 UTC m=+0.131308863 container create cda91547495de10c705501ee46dcc34976367604e2bb3c4fdc22d89d23ef219d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-323d23d, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T22:22:34.272 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:34 smithi053 podman[72517]: 2024-03-28 22:22:34.205311823 +0000 UTC m=+0.342137647 container init cda91547495de10c705501ee46dcc34976367604e2bb3c4fdc22d89d23ef219d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, GIT_CLEAN=True, RELEASE=squid-323d23d) 2024-03-28T22:22:34.272 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:34 smithi053 podman[72517]: 2024-03-28 22:22:34.208428504 +0000 UTC m=+0.345254323 container start cda91547495de10c705501ee46dcc34976367604e2bb3c4fdc22d89d23ef219d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=squid-323d23d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-28T22:22:34.272 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:34 smithi053 podman[72517]: 2024-03-28 22:22:34.271918065 +0000 UTC m=+0.408743901 container attach cda91547495de10c705501ee46dcc34976367604e2bb3c4fdc22d89d23ef219d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=squid-323d23d, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True) 2024-03-28T22:22:34.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:34 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:34.006+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:34.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:34 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:34.006+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:34.431 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:34 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:34.006+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:34.431 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:34 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:34.006+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:22:34.431 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:34 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:34.006+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:07.780894+0000 front 2024-03-28T22:22:07.780921+0000 (oldest deadline 2024-03-28T22:22:33.080880+0000) 2024-03-28T22:22:34.546 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:34 smithi053 podman[72517]: 2024-03-28 22:22:34.50958166 +0000 UTC m=+0.646407470 container died cda91547495de10c705501ee46dcc34976367604e2bb3c4fdc22d89d23ef219d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, RELEASE=squid-323d23d, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-28T22:22:34.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:34 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:34.326+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:34.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:34 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:34.326+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:34.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:34 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:34.326+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:34.547 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:34 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:34.326+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:57.638872+0000 front 2024-03-28T22:21:57.638939+0000 (oldest deadline 2024-03-28T22:22:21.739096+0000) 2024-03-28T22:22:34.547 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:34 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:34.326+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:05.140891+0000 front 2024-03-28T22:22:05.140920+0000 (oldest deadline 2024-03-28T22:22:30.441027+0000) 2024-03-28T22:22:34.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:34 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:34.539+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:41.373318+0000 front 2024-03-28T22:21:41.373414+0000 (oldest deadline 2024-03-28T22:22:06.673341+0000) 2024-03-28T22:22:34.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:34 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:34.539+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:47.774560+0000 front 2024-03-28T22:21:47.774226+0000 (oldest deadline 2024-03-28T22:22:13.674264+0000) 2024-03-28T22:22:34.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:34 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:34.539+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:58.375448+0000 front 2024-03-28T22:21:58.375558+0000 (oldest deadline 2024-03-28T22:22:21.875338+0000) 2024-03-28T22:22:34.931 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:34 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:34.539+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:07.677063+0000 front 2024-03-28T22:22:07.677117+0000 (oldest deadline 2024-03-28T22:22:33.577063+0000) 2024-03-28T22:22:35.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:34 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:34.836+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:35.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:34 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:34.836+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:35.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:34 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:34.836+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.710716+0000 front 2024-03-28T22:21:50.710773+0000 (oldest deadline 2024-03-28T22:22:13.610874+0000) 2024-03-28T22:22:35.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:34 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:34.836+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:22:00.612619+0000 front 2024-03-28T22:22:00.612649+0000 (oldest deadline 2024-03-28T22:22:24.112843+0000) 2024-03-28T22:22:35.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:34 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:34.836+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:04.113537+0000 front 2024-03-28T22:22:04.113671+0000 (oldest deadline 2024-03-28T22:22:30.013514+0000) 2024-03-28T22:22:35.212 INFO:journalctl@ceph.osd.7.smithi053.stdout:Mar 28 22:22:35 smithi053 podman[72543]: 2024-03-28 22:22:35.191965484 +0000 UTC m=+0.676694201 container remove cda91547495de10c705501ee46dcc34976367604e2bb3c4fdc22d89d23ef219d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, RELEASE=squid-323d23d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0) 2024-03-28T22:22:35.218 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.7.service' 2024-03-28T22:22:35.268 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:22:35.269 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-28T22:22:35.269 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-28T22:22:35.269 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.8 2024-03-28T22:22:35.304 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:34 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:34.970+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:35.305 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:34 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:34.970+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:35.305 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:34 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:34.970+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:35.305 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:34 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:34.970+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:22:35.305 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:34 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:34.970+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:07.780894+0000 front 2024-03-28T22:22:07.780921+0000 (oldest deadline 2024-03-28T22:22:33.080880+0000) 2024-03-28T22:22:35.490 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:35 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:35.306+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:35.490 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:35 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:35.306+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:35.490 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:35 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:35.306+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:35.490 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:35 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:35.307+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:57.638872+0000 front 2024-03-28T22:21:57.638939+0000 (oldest deadline 2024-03-28T22:22:21.739096+0000) 2024-03-28T22:22:35.491 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:35 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:35.307+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:05.140891+0000 front 2024-03-28T22:22:05.140920+0000 (oldest deadline 2024-03-28T22:22:30.441027+0000) 2024-03-28T22:22:35.566 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:35 smithi063 systemd[1]: Stopping Ceph osd.8 for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:22:35.566 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:35 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:35.550+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:41.373318+0000 front 2024-03-28T22:21:41.373414+0000 (oldest deadline 2024-03-28T22:22:06.673341+0000) 2024-03-28T22:22:35.566 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:35 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:35.550+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:47.774560+0000 front 2024-03-28T22:21:47.774226+0000 (oldest deadline 2024-03-28T22:22:13.674264+0000) 2024-03-28T22:22:35.566 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:35 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:35.550+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:58.375448+0000 front 2024-03-28T22:21:58.375558+0000 (oldest deadline 2024-03-28T22:22:21.875338+0000) 2024-03-28T22:22:35.567 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:35 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:35.550+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:07.677063+0000 front 2024-03-28T22:22:07.677117+0000 (oldest deadline 2024-03-28T22:22:33.577063+0000) 2024-03-28T22:22:35.567 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:35 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:35.565+0000 7fc64c92c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:22:35.567 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:35 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:35.565+0000 7fc64c92c700 -1 osd.8 102 *** Got signal Terminated *** 2024-03-28T22:22:35.567 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:35 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:35.565+0000 7fc64c92c700 -1 osd.8 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T22:22:36.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:35 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:35.820+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:36.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:35 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:35.820+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:36.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:35 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:35.820+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.710716+0000 front 2024-03-28T22:21:50.710773+0000 (oldest deadline 2024-03-28T22:22:13.610874+0000) 2024-03-28T22:22:36.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:35 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:35.820+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:22:00.612619+0000 front 2024-03-28T22:22:00.612649+0000 (oldest deadline 2024-03-28T22:22:24.112843+0000) 2024-03-28T22:22:36.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:35 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:35.820+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:04.113537+0000 front 2024-03-28T22:22:04.113671+0000 (oldest deadline 2024-03-28T22:22:30.013514+0000) 2024-03-28T22:22:36.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:35 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:35.997+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:36.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:35 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:35.997+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:36.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:35 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:35.997+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:36.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:35 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:35.997+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:22:36.431 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:35 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:35.997+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:07.780894+0000 front 2024-03-28T22:22:07.780921+0000 (oldest deadline 2024-03-28T22:22:33.080880+0000) 2024-03-28T22:22:36.547 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:36 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:36.289+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:36.547 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:36 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:36.289+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:36.547 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:36 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:36.289+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:36.547 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:36 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:36.289+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:57.638872+0000 front 2024-03-28T22:21:57.638939+0000 (oldest deadline 2024-03-28T22:22:21.739096+0000) 2024-03-28T22:22:36.547 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:36 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:36.289+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:05.140891+0000 front 2024-03-28T22:22:05.140920+0000 (oldest deadline 2024-03-28T22:22:30.441027+0000) 2024-03-28T22:22:36.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:36 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:36.514+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:41.373318+0000 front 2024-03-28T22:21:41.373414+0000 (oldest deadline 2024-03-28T22:22:06.673341+0000) 2024-03-28T22:22:36.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:36 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:36.514+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:47.774560+0000 front 2024-03-28T22:21:47.774226+0000 (oldest deadline 2024-03-28T22:22:13.674264+0000) 2024-03-28T22:22:36.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:36 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:36.514+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:58.375448+0000 front 2024-03-28T22:21:58.375558+0000 (oldest deadline 2024-03-28T22:22:21.875338+0000) 2024-03-28T22:22:36.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:36 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:36.514+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:07.677063+0000 front 2024-03-28T22:22:07.677117+0000 (oldest deadline 2024-03-28T22:22:33.577063+0000) 2024-03-28T22:22:36.934 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:36 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:36.775+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:36.934 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:36 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:36.775+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:36.935 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:36 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:36.775+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.710716+0000 front 2024-03-28T22:21:50.710773+0000 (oldest deadline 2024-03-28T22:22:13.610874+0000) 2024-03-28T22:22:36.935 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:36 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:36.775+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:22:00.612619+0000 front 2024-03-28T22:22:00.612649+0000 (oldest deadline 2024-03-28T22:22:24.112843+0000) 2024-03-28T22:22:36.935 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:36 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:36.775+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:04.113537+0000 front 2024-03-28T22:22:04.113671+0000 (oldest deadline 2024-03-28T22:22:30.013514+0000) 2024-03-28T22:22:37.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:37 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:37.040+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:37.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:37 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:37.040+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:37.431 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:37 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:37.040+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:37.431 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:37 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:37.040+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:22:37.431 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:37 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:37.040+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:07.780894+0000 front 2024-03-28T22:22:07.780921+0000 (oldest deadline 2024-03-28T22:22:33.080880+0000) 2024-03-28T22:22:37.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:37 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:37.287+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:37.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:37 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:37.287+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:37.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:37 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:37.287+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:37.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:37 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:37.287+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:57.638872+0000 front 2024-03-28T22:21:57.638939+0000 (oldest deadline 2024-03-28T22:22:21.739096+0000) 2024-03-28T22:22:37.546 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:37 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:37.287+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:05.140891+0000 front 2024-03-28T22:22:05.140920+0000 (oldest deadline 2024-03-28T22:22:30.441027+0000) 2024-03-28T22:22:37.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:37 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:37.469+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:41.373318+0000 front 2024-03-28T22:21:41.373414+0000 (oldest deadline 2024-03-28T22:22:06.673341+0000) 2024-03-28T22:22:37.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:37 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:37.469+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:47.774560+0000 front 2024-03-28T22:21:47.774226+0000 (oldest deadline 2024-03-28T22:22:13.674264+0000) 2024-03-28T22:22:37.931 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:37 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:37.469+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:58.375448+0000 front 2024-03-28T22:21:58.375558+0000 (oldest deadline 2024-03-28T22:22:21.875338+0000) 2024-03-28T22:22:37.931 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:37 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:37.469+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:07.677063+0000 front 2024-03-28T22:22:07.677117+0000 (oldest deadline 2024-03-28T22:22:33.577063+0000) 2024-03-28T22:22:38.184 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:37 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:37.812+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:38.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:37 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:37.812+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:38.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:37 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:37.812+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.710716+0000 front 2024-03-28T22:21:50.710773+0000 (oldest deadline 2024-03-28T22:22:13.610874+0000) 2024-03-28T22:22:38.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:37 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:37.812+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:22:00.612619+0000 front 2024-03-28T22:22:00.612649+0000 (oldest deadline 2024-03-28T22:22:24.112843+0000) 2024-03-28T22:22:38.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:37 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:37.812+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:04.113537+0000 front 2024-03-28T22:22:04.113671+0000 (oldest deadline 2024-03-28T22:22:30.013514+0000) 2024-03-28T22:22:38.428 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:38 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:38.083+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:38.428 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:38 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:38.083+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:38.428 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:38 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:38.083+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:38.428 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:38 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:38.083+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:22:38.429 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:38 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:38.083+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:07.780894+0000 front 2024-03-28T22:22:07.780921+0000 (oldest deadline 2024-03-28T22:22:33.080880+0000) 2024-03-28T22:22:38.429 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:38 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:38.083+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:13.081407+0000 front 2024-03-28T22:22:13.081352+0000 (oldest deadline 2024-03-28T22:22:37.181460+0000) 2024-03-28T22:22:38.680 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:38 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:38.428+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:41.373318+0000 front 2024-03-28T22:21:41.373414+0000 (oldest deadline 2024-03-28T22:22:06.673341+0000) 2024-03-28T22:22:38.680 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:38 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:38.428+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:47.774560+0000 front 2024-03-28T22:21:47.774226+0000 (oldest deadline 2024-03-28T22:22:13.674264+0000) 2024-03-28T22:22:38.680 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:38 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:38.428+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:58.375448+0000 front 2024-03-28T22:21:58.375558+0000 (oldest deadline 2024-03-28T22:22:21.875338+0000) 2024-03-28T22:22:38.680 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:38 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:38.428+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:07.677063+0000 front 2024-03-28T22:22:07.677117+0000 (oldest deadline 2024-03-28T22:22:33.577063+0000) 2024-03-28T22:22:38.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:38 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:38.308+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:38.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:38 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:38.308+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:38.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:38 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:38.308+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:38.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:38 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:38.308+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:57.638872+0000 front 2024-03-28T22:21:57.638939+0000 (oldest deadline 2024-03-28T22:22:21.739096+0000) 2024-03-28T22:22:38.797 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:38 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:38.308+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:05.140891+0000 front 2024-03-28T22:22:05.140920+0000 (oldest deadline 2024-03-28T22:22:30.441027+0000) 2024-03-28T22:22:38.797 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:38 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:38.308+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:11.542058+0000 front 2024-03-28T22:22:11.542081+0000 (oldest deadline 2024-03-28T22:22:37.442130+0000) 2024-03-28T22:22:39.184 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:38 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:38.790+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:39.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:38 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:38.790+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:39.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:38 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:38.790+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.710716+0000 front 2024-03-28T22:21:50.710773+0000 (oldest deadline 2024-03-28T22:22:13.610874+0000) 2024-03-28T22:22:39.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:38 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:38.790+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:22:00.612619+0000 front 2024-03-28T22:22:00.612649+0000 (oldest deadline 2024-03-28T22:22:24.112843+0000) 2024-03-28T22:22:39.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:38 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:38.790+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:04.113537+0000 front 2024-03-28T22:22:04.113671+0000 (oldest deadline 2024-03-28T22:22:30.013514+0000) 2024-03-28T22:22:39.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:39 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:39.084+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:39.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:39 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:39.084+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:39.431 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:39 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:39.084+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:39.431 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:39 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:39.084+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:22:39.431 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:39 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:39.084+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:07.780894+0000 front 2024-03-28T22:22:07.780921+0000 (oldest deadline 2024-03-28T22:22:33.080880+0000) 2024-03-28T22:22:39.431 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:39 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:39.084+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:13.081407+0000 front 2024-03-28T22:22:13.081352+0000 (oldest deadline 2024-03-28T22:22:37.181460+0000) 2024-03-28T22:22:39.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:39 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:39.343+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:39.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:39 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:39.343+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:39.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:39 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:39.343+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:39.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:39 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:39.343+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:57.638872+0000 front 2024-03-28T22:21:57.638939+0000 (oldest deadline 2024-03-28T22:22:21.739096+0000) 2024-03-28T22:22:39.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:39 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:39.343+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:05.140891+0000 front 2024-03-28T22:22:05.140920+0000 (oldest deadline 2024-03-28T22:22:30.441027+0000) 2024-03-28T22:22:39.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:39 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:39.343+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:11.542058+0000 front 2024-03-28T22:22:11.542081+0000 (oldest deadline 2024-03-28T22:22:37.442130+0000) 2024-03-28T22:22:39.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:39 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:39.447+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:41.373318+0000 front 2024-03-28T22:21:41.373414+0000 (oldest deadline 2024-03-28T22:22:06.673341+0000) 2024-03-28T22:22:39.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:39 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:39.447+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:47.774560+0000 front 2024-03-28T22:21:47.774226+0000 (oldest deadline 2024-03-28T22:22:13.674264+0000) 2024-03-28T22:22:39.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:39 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:39.447+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:58.375448+0000 front 2024-03-28T22:21:58.375558+0000 (oldest deadline 2024-03-28T22:22:21.875338+0000) 2024-03-28T22:22:39.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:39 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:39.447+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:07.677063+0000 front 2024-03-28T22:22:07.677117+0000 (oldest deadline 2024-03-28T22:22:33.577063+0000) 2024-03-28T22:22:40.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:39 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:39.805+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:40.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:39 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:39.805+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:40.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:39 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:39.805+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.710716+0000 front 2024-03-28T22:21:50.710773+0000 (oldest deadline 2024-03-28T22:22:13.610874+0000) 2024-03-28T22:22:40.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:39 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:39.805+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:22:00.612619+0000 front 2024-03-28T22:22:00.612649+0000 (oldest deadline 2024-03-28T22:22:24.112843+0000) 2024-03-28T22:22:40.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:39 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:39.805+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:04.113537+0000 front 2024-03-28T22:22:04.113671+0000 (oldest deadline 2024-03-28T22:22:30.013514+0000) 2024-03-28T22:22:40.186 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:39 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:39.805+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:15.315002+0000 front 2024-03-28T22:22:15.314870+0000 (oldest deadline 2024-03-28T22:22:39.414712+0000) 2024-03-28T22:22:40.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:40 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:40.122+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:40.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:40 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:40.122+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:40.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:40 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:40.122+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:40.431 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:40 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:40.122+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:22:40.431 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:40 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:40.122+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:07.780894+0000 front 2024-03-28T22:22:07.780921+0000 (oldest deadline 2024-03-28T22:22:33.080880+0000) 2024-03-28T22:22:40.431 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:40 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:40.122+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:13.081407+0000 front 2024-03-28T22:22:13.081352+0000 (oldest deadline 2024-03-28T22:22:37.181460+0000) 2024-03-28T22:22:40.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:40 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:40.350+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:40.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:40 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:40.351+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:40.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:40 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:40.351+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:40.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:40 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:40.351+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:57.638872+0000 front 2024-03-28T22:21:57.638939+0000 (oldest deadline 2024-03-28T22:22:21.739096+0000) 2024-03-28T22:22:40.797 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:40 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:40.351+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:05.140891+0000 front 2024-03-28T22:22:05.140920+0000 (oldest deadline 2024-03-28T22:22:30.441027+0000) 2024-03-28T22:22:40.797 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:40 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:40.351+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:11.542058+0000 front 2024-03-28T22:22:11.542081+0000 (oldest deadline 2024-03-28T22:22:37.442130+0000) 2024-03-28T22:22:40.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:40 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:40.444+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:41.373318+0000 front 2024-03-28T22:21:41.373414+0000 (oldest deadline 2024-03-28T22:22:06.673341+0000) 2024-03-28T22:22:40.930 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:40 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:40.445+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:47.774560+0000 front 2024-03-28T22:21:47.774226+0000 (oldest deadline 2024-03-28T22:22:13.674264+0000) 2024-03-28T22:22:40.931 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:40 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:40.445+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:58.375448+0000 front 2024-03-28T22:21:58.375558+0000 (oldest deadline 2024-03-28T22:22:21.875338+0000) 2024-03-28T22:22:40.931 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:40 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8[55006]: 2024-03-28T22:22:40.445+0000 7fc649721700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:07.677063+0000 front 2024-03-28T22:22:07.677117+0000 (oldest deadline 2024-03-28T22:22:33.577063+0000) 2024-03-28T22:22:41.184 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:40 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:40.844+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:41.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:40 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:40.844+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:41.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:40 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:40.844+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.710716+0000 front 2024-03-28T22:21:50.710773+0000 (oldest deadline 2024-03-28T22:22:13.610874+0000) 2024-03-28T22:22:41.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:40 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:40.844+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:22:00.612619+0000 front 2024-03-28T22:22:00.612649+0000 (oldest deadline 2024-03-28T22:22:24.112843+0000) 2024-03-28T22:22:41.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:40 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:40.844+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:04.113537+0000 front 2024-03-28T22:22:04.113671+0000 (oldest deadline 2024-03-28T22:22:30.013514+0000) 2024-03-28T22:22:41.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:40 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:40.844+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:15.315002+0000 front 2024-03-28T22:22:15.314870+0000 (oldest deadline 2024-03-28T22:22:39.414712+0000) 2024-03-28T22:22:41.391 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:41 smithi063 podman[73370]: 2024-03-28 22:22:41.101764854 +0000 UTC m=+5.622699274 container died ef945d9ba655b1c844b6732629681193b66e70b9bb38458c0c0dc5e26a37d2f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, RELEASE=squid-323d23d, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T22:22:41.392 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:41 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:41.105+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:41.392 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:41 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:41.105+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:41.392 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:41 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:41.105+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:41.393 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:41 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:41.105+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:22:41.393 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:41 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:41.105+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:07.780894+0000 front 2024-03-28T22:22:07.780921+0000 (oldest deadline 2024-03-28T22:22:33.080880+0000) 2024-03-28T22:22:41.393 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:41 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:41.105+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:13.081407+0000 front 2024-03-28T22:22:13.081352+0000 (oldest deadline 2024-03-28T22:22:37.181460+0000) 2024-03-28T22:22:41.680 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:41 smithi063 podman[73370]: 2024-03-28 22:22:41.491201883 +0000 UTC m=+6.012136301 container cleanup ef945d9ba655b1c844b6732629681193b66e70b9bb38458c0c0dc5e26a37d2f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, RELEASE=squid-323d23d, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T22:22:41.680 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:41 smithi063 bash[73370]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8 2024-03-28T22:22:41.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:41 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:41.378+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:41.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:41 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:41.378+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:41.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:41 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:41.378+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:41.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:41 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:41.378+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:57.638872+0000 front 2024-03-28T22:21:57.638939+0000 (oldest deadline 2024-03-28T22:22:21.739096+0000) 2024-03-28T22:22:41.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:41 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:41.378+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:05.140891+0000 front 2024-03-28T22:22:05.140920+0000 (oldest deadline 2024-03-28T22:22:30.441027+0000) 2024-03-28T22:22:41.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:41 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:41.378+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:11.542058+0000 front 2024-03-28T22:22:11.542081+0000 (oldest deadline 2024-03-28T22:22:37.442130+0000) 2024-03-28T22:22:42.041 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:41 smithi063 podman[73388]: 2024-03-28 22:22:41.71735905 +0000 UTC m=+0.607448583 container remove ef945d9ba655b1c844b6732629681193b66e70b9bb38458c0c0dc5e26a37d2f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, RELEASE=squid-323d23d) 2024-03-28T22:22:42.184 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:41 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:41.885+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:42.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:41 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:41.885+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:42.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:41 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:41.885+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.710716+0000 front 2024-03-28T22:21:50.710773+0000 (oldest deadline 2024-03-28T22:22:13.610874+0000) 2024-03-28T22:22:42.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:41 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:41.885+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:22:00.612619+0000 front 2024-03-28T22:22:00.612649+0000 (oldest deadline 2024-03-28T22:22:24.112843+0000) 2024-03-28T22:22:42.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:41 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:41.885+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:04.113537+0000 front 2024-03-28T22:22:04.113671+0000 (oldest deadline 2024-03-28T22:22:30.013514+0000) 2024-03-28T22:22:42.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:41 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:41.885+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:15.315002+0000 front 2024-03-28T22:22:15.314870+0000 (oldest deadline 2024-03-28T22:22:39.414712+0000) 2024-03-28T22:22:42.345 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:42 smithi063 podman[73481]: 2024-03-28 22:22:41.94507619 +0000 UTC m=+0.018365935 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:22:42.345 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:42 smithi063 podman[73481]: 2024-03-28 22:22:42.051436564 +0000 UTC m=+0.124726295 container create 43a9da3c3669eda4a1fe4245ac4aeaab706e554a494810a568ab831803c694c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-323d23d, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-28T22:22:42.345 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:42 smithi063 podman[73481]: 2024-03-28 22:22:42.279529561 +0000 UTC m=+0.352819307 container init 43a9da3c3669eda4a1fe4245ac4aeaab706e554a494810a568ab831803c694c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8-deactivate, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-323d23d, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-28T22:22:42.345 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:42 smithi063 podman[73481]: 2024-03-28 22:22:42.283118899 +0000 UTC m=+0.356408633 container start 43a9da3c3669eda4a1fe4245ac4aeaab706e554a494810a568ab831803c694c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-323d23d, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-28T22:22:42.346 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:42 smithi063 podman[73481]: 2024-03-28 22:22:42.345688482 +0000 UTC m=+0.418978234 container attach 43a9da3c3669eda4a1fe4245ac4aeaab706e554a494810a568ab831803c694c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8-deactivate, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-323d23d, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-28T22:22:42.346 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:42 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:42.153+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:42.346 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:42 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:42.153+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:42.346 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:42 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:42.153+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:42.346 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:42 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:42.153+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:22:42.347 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:42 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:42.153+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:07.780894+0000 front 2024-03-28T22:22:07.780921+0000 (oldest deadline 2024-03-28T22:22:33.080880+0000) 2024-03-28T22:22:42.347 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:42 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:42.153+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:13.081407+0000 front 2024-03-28T22:22:13.081352+0000 (oldest deadline 2024-03-28T22:22:37.181460+0000) 2024-03-28T22:22:42.680 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:42 smithi063 podman[73481]: 2024-03-28 22:22:42.583235765 +0000 UTC m=+0.656525525 container died 43a9da3c3669eda4a1fe4245ac4aeaab706e554a494810a568ab831803c694c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-323d23d, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T22:22:42.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:42 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:42.391+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:42.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:42 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:42.391+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:42.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:42 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:42.391+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:42.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:42 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:42.391+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:57.638872+0000 front 2024-03-28T22:21:57.638939+0000 (oldest deadline 2024-03-28T22:22:21.739096+0000) 2024-03-28T22:22:42.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:42 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:42.391+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:05.140891+0000 front 2024-03-28T22:22:05.140920+0000 (oldest deadline 2024-03-28T22:22:30.441027+0000) 2024-03-28T22:22:42.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:42 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:42.391+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:11.542058+0000 front 2024-03-28T22:22:11.542081+0000 (oldest deadline 2024-03-28T22:22:37.442130+0000) 2024-03-28T22:22:43.095 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.8.service' 2024-03-28T22:22:43.129 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:43 smithi063 podman[73506]: 2024-03-28 22:22:43.074367216 +0000 UTC m=+0.486076352 container remove 43a9da3c3669eda4a1fe4245ac4aeaab706e554a494810a568ab831803c694c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-8-deactivate, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-323d23d, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-28T22:22:43.130 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:43 smithi063 systemd[1]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.8.service: Deactivated successfully. 2024-03-28T22:22:43.130 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:43 smithi063 systemd[1]: Stopped Ceph osd.8 for 798d9884-ed4e-11ee-95d0-87774f69a715. 2024-03-28T22:22:43.130 INFO:journalctl@ceph.osd.8.smithi063.stdout:Mar 28 22:22:43 smithi063 systemd[1]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.8.service: Consumed 13.585s CPU time. 2024-03-28T22:22:43.175 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:22:43.175 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-28T22:22:43.176 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-28T22:22:43.176 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.9 2024-03-28T22:22:43.184 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:42 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:42.854+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:43.184 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:42 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:42.854+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:43.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:42 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:42.854+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.710716+0000 front 2024-03-28T22:21:50.710773+0000 (oldest deadline 2024-03-28T22:22:13.610874+0000) 2024-03-28T22:22:43.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:42 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:42.854+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:22:00.612619+0000 front 2024-03-28T22:22:00.612649+0000 (oldest deadline 2024-03-28T22:22:24.112843+0000) 2024-03-28T22:22:43.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:42 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:42.854+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:04.113537+0000 front 2024-03-28T22:22:04.113671+0000 (oldest deadline 2024-03-28T22:22:30.013514+0000) 2024-03-28T22:22:43.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:42 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:42.854+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:15.315002+0000 front 2024-03-28T22:22:15.314870+0000 (oldest deadline 2024-03-28T22:22:39.414712+0000) 2024-03-28T22:22:43.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:43 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:43.128+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:43.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:43 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:43.128+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:43.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:43 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:43.128+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:43.431 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:43 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:43.128+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:22:43.431 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:43 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:43.128+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:07.780894+0000 front 2024-03-28T22:22:07.780921+0000 (oldest deadline 2024-03-28T22:22:33.080880+0000) 2024-03-28T22:22:43.431 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:43 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:43.128+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:13.081407+0000 front 2024-03-28T22:22:13.081352+0000 (oldest deadline 2024-03-28T22:22:37.181460+0000) 2024-03-28T22:22:43.504 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:43 smithi016 systemd[1]: Stopping Ceph osd.9 for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:22:43.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:43 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:43.438+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:43.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:43 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:43.438+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:43.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:43 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:43.438+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:43.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:43 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:43.438+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:57.638872+0000 front 2024-03-28T22:21:57.638939+0000 (oldest deadline 2024-03-28T22:22:21.739096+0000) 2024-03-28T22:22:43.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:43 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:43.438+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:05.140891+0000 front 2024-03-28T22:22:05.140920+0000 (oldest deadline 2024-03-28T22:22:30.441027+0000) 2024-03-28T22:22:43.797 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:43 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:43.438+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:11.542058+0000 front 2024-03-28T22:22:11.542081+0000 (oldest deadline 2024-03-28T22:22:37.442130+0000) 2024-03-28T22:22:43.853 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:43 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:43.504+0000 7ff80178e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:22:43.854 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:43 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:43.504+0000 7ff80178e700 -1 osd.9 102 *** Got signal Terminated *** 2024-03-28T22:22:43.854 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:43 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:43.504+0000 7ff80178e700 -1 osd.9 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T22:22:43.854 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:43 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:43.853+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:43.854 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:43 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:43.853+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:44.184 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:43 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:43.853+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.710716+0000 front 2024-03-28T22:21:50.710773+0000 (oldest deadline 2024-03-28T22:22:13.610874+0000) 2024-03-28T22:22:44.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:43 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:43.853+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:22:00.612619+0000 front 2024-03-28T22:22:00.612649+0000 (oldest deadline 2024-03-28T22:22:24.112843+0000) 2024-03-28T22:22:44.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:43 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:43.853+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:04.113537+0000 front 2024-03-28T22:22:04.113671+0000 (oldest deadline 2024-03-28T22:22:30.013514+0000) 2024-03-28T22:22:44.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:43 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:43.853+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:15.315002+0000 front 2024-03-28T22:22:15.314870+0000 (oldest deadline 2024-03-28T22:22:39.414712+0000) 2024-03-28T22:22:44.340 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:44 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:44.089+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:44.340 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:44 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:44.089+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:44.340 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:44 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:44.089+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:44.341 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:44 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:44.089+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:22:44.341 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:44 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:44.089+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:07.780894+0000 front 2024-03-28T22:22:07.780921+0000 (oldest deadline 2024-03-28T22:22:33.080880+0000) 2024-03-28T22:22:44.341 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:44 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:44.089+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:13.081407+0000 front 2024-03-28T22:22:13.081352+0000 (oldest deadline 2024-03-28T22:22:37.181460+0000) 2024-03-28T22:22:44.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:44 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:44.476+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:44.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:44 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:44.476+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:44.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:44 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:44.476+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:44.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:44 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:44.476+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:57.638872+0000 front 2024-03-28T22:21:57.638939+0000 (oldest deadline 2024-03-28T22:22:21.739096+0000) 2024-03-28T22:22:44.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:44 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:44.476+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:05.140891+0000 front 2024-03-28T22:22:05.140920+0000 (oldest deadline 2024-03-28T22:22:30.441027+0000) 2024-03-28T22:22:44.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:44 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:44.476+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:11.542058+0000 front 2024-03-28T22:22:11.542081+0000 (oldest deadline 2024-03-28T22:22:37.442130+0000) 2024-03-28T22:22:45.184 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:44 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:44.812+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:45.184 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:44 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:44.812+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:45.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:44 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:44.812+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.710716+0000 front 2024-03-28T22:21:50.710773+0000 (oldest deadline 2024-03-28T22:22:13.610874+0000) 2024-03-28T22:22:45.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:44 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:44.812+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:22:00.612619+0000 front 2024-03-28T22:22:00.612649+0000 (oldest deadline 2024-03-28T22:22:24.112843+0000) 2024-03-28T22:22:45.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:44 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:44.812+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:04.113537+0000 front 2024-03-28T22:22:04.113671+0000 (oldest deadline 2024-03-28T22:22:30.013514+0000) 2024-03-28T22:22:45.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:44 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:44.812+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:15.315002+0000 front 2024-03-28T22:22:15.314870+0000 (oldest deadline 2024-03-28T22:22:39.414712+0000) 2024-03-28T22:22:45.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:45 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:45.135+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:45.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:45 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:45.135+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:45.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:45 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:45.135+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:45.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:45 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:45.135+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:22:45.431 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:45 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:45.135+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:07.780894+0000 front 2024-03-28T22:22:07.780921+0000 (oldest deadline 2024-03-28T22:22:33.080880+0000) 2024-03-28T22:22:45.431 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:45 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:45.135+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:13.081407+0000 front 2024-03-28T22:22:13.081352+0000 (oldest deadline 2024-03-28T22:22:37.181460+0000) 2024-03-28T22:22:45.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:45 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:45.466+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:45.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:45 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:45.466+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:45.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:45 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:45.466+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:45.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:45 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:45.466+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:57.638872+0000 front 2024-03-28T22:21:57.638939+0000 (oldest deadline 2024-03-28T22:22:21.739096+0000) 2024-03-28T22:22:45.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:45 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:45.466+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:05.140891+0000 front 2024-03-28T22:22:05.140920+0000 (oldest deadline 2024-03-28T22:22:30.441027+0000) 2024-03-28T22:22:45.797 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:45 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:45.466+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:11.542058+0000 front 2024-03-28T22:22:11.542081+0000 (oldest deadline 2024-03-28T22:22:37.442130+0000) 2024-03-28T22:22:46.184 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:45 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:45.779+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:46.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:45 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:45.779+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:46.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:45 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:45.779+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.710716+0000 front 2024-03-28T22:21:50.710773+0000 (oldest deadline 2024-03-28T22:22:13.610874+0000) 2024-03-28T22:22:46.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:45 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:45.779+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:22:00.612619+0000 front 2024-03-28T22:22:00.612649+0000 (oldest deadline 2024-03-28T22:22:24.112843+0000) 2024-03-28T22:22:46.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:45 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:45.779+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:04.113537+0000 front 2024-03-28T22:22:04.113671+0000 (oldest deadline 2024-03-28T22:22:30.013514+0000) 2024-03-28T22:22:46.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:45 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:45.779+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:15.315002+0000 front 2024-03-28T22:22:15.314870+0000 (oldest deadline 2024-03-28T22:22:39.414712+0000) 2024-03-28T22:22:46.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:46 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:46.167+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:46.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:46 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:46.167+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:46.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:46 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:46.167+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:46.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:46 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:46.167+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:22:46.431 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:46 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:46.167+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:07.780894+0000 front 2024-03-28T22:22:07.780921+0000 (oldest deadline 2024-03-28T22:22:33.080880+0000) 2024-03-28T22:22:46.431 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:46 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:46.167+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:13.081407+0000 front 2024-03-28T22:22:13.081352+0000 (oldest deadline 2024-03-28T22:22:37.181460+0000) 2024-03-28T22:22:46.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:46 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:46.473+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:46.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:46 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:46.473+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:46.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:46 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:46.473+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:46.797 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:46 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:46.473+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:57.638872+0000 front 2024-03-28T22:21:57.638939+0000 (oldest deadline 2024-03-28T22:22:21.739096+0000) 2024-03-28T22:22:46.797 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:46 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:46.473+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:05.140891+0000 front 2024-03-28T22:22:05.140920+0000 (oldest deadline 2024-03-28T22:22:30.441027+0000) 2024-03-28T22:22:46.797 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:46 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:46.473+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:11.542058+0000 front 2024-03-28T22:22:11.542081+0000 (oldest deadline 2024-03-28T22:22:37.442130+0000) 2024-03-28T22:22:47.184 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:46 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:46.805+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:47.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:46 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:46.805+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:47.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:46 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:46.805+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.710716+0000 front 2024-03-28T22:21:50.710773+0000 (oldest deadline 2024-03-28T22:22:13.610874+0000) 2024-03-28T22:22:47.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:46 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:46.805+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:22:00.612619+0000 front 2024-03-28T22:22:00.612649+0000 (oldest deadline 2024-03-28T22:22:24.112843+0000) 2024-03-28T22:22:47.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:46 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:46.805+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:04.113537+0000 front 2024-03-28T22:22:04.113671+0000 (oldest deadline 2024-03-28T22:22:30.013514+0000) 2024-03-28T22:22:47.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:46 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:46.805+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:15.315002+0000 front 2024-03-28T22:22:15.314870+0000 (oldest deadline 2024-03-28T22:22:39.414712+0000) 2024-03-28T22:22:47.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:47 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:47.119+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:47.431 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:47 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:47.119+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:47.431 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:47 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:47.119+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:47.431 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:47 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:47.119+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:22:47.431 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:47 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:47.119+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:07.780894+0000 front 2024-03-28T22:22:07.780921+0000 (oldest deadline 2024-03-28T22:22:33.080880+0000) 2024-03-28T22:22:47.431 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:47 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:47.119+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:13.081407+0000 front 2024-03-28T22:22:13.081352+0000 (oldest deadline 2024-03-28T22:22:37.181460+0000) 2024-03-28T22:22:47.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:47 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:47.498+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:47.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:47 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:47.498+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:47.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:47 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:47.498+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:47.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:47 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:47.498+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:57.638872+0000 front 2024-03-28T22:21:57.638939+0000 (oldest deadline 2024-03-28T22:22:21.739096+0000) 2024-03-28T22:22:47.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:47 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:47.498+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:05.140891+0000 front 2024-03-28T22:22:05.140920+0000 (oldest deadline 2024-03-28T22:22:30.441027+0000) 2024-03-28T22:22:47.797 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:47 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:47.498+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:11.542058+0000 front 2024-03-28T22:22:11.542081+0000 (oldest deadline 2024-03-28T22:22:37.442130+0000) 2024-03-28T22:22:48.184 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:47 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:47.850+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.207527+0000 front 2024-03-28T22:21:33.207593+0000 (oldest deadline 2024-03-28T22:21:57.907714+0000) 2024-03-28T22:22:48.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:47 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:47.850+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:43.709384+0000 front 2024-03-28T22:21:43.709420+0000 (oldest deadline 2024-03-28T22:22:07.809497+0000) 2024-03-28T22:22:48.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:47 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:47.850+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:50.710716+0000 front 2024-03-28T22:21:50.710773+0000 (oldest deadline 2024-03-28T22:22:13.610874+0000) 2024-03-28T22:22:48.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:47 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:47.850+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:22:00.612619+0000 front 2024-03-28T22:22:00.612649+0000 (oldest deadline 2024-03-28T22:22:24.112843+0000) 2024-03-28T22:22:48.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:47 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:47.850+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:04.113537+0000 front 2024-03-28T22:22:04.113671+0000 (oldest deadline 2024-03-28T22:22:30.013514+0000) 2024-03-28T22:22:48.185 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:47 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9[65244]: 2024-03-28T22:22:47.850+0000 7ff7fe583700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:15.315002+0000 front 2024-03-28T22:22:15.314870+0000 (oldest deadline 2024-03-28T22:22:39.414712+0000) 2024-03-28T22:22:48.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:48 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:48.168+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:48.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:48 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:48.168+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:48.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:48 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:48.168+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:48.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:48 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:48.168+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:22:48.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:48 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:48.168+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:07.780894+0000 front 2024-03-28T22:22:07.780921+0000 (oldest deadline 2024-03-28T22:22:33.080880+0000) 2024-03-28T22:22:48.431 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:48 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:48.168+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:13.081407+0000 front 2024-03-28T22:22:13.081352+0000 (oldest deadline 2024-03-28T22:22:37.181460+0000) 2024-03-28T22:22:48.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:48 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:48.532+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:48.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:48 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:48.532+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:48.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:48 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:48.532+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:48.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:48 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:48.532+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:57.638872+0000 front 2024-03-28T22:21:57.638939+0000 (oldest deadline 2024-03-28T22:22:21.739096+0000) 2024-03-28T22:22:48.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:48 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:48.532+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:05.140891+0000 front 2024-03-28T22:22:05.140920+0000 (oldest deadline 2024-03-28T22:22:30.441027+0000) 2024-03-28T22:22:48.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:48 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:48.532+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:11.542058+0000 front 2024-03-28T22:22:11.542081+0000 (oldest deadline 2024-03-28T22:22:37.442130+0000) 2024-03-28T22:22:49.105 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:48 smithi016 podman[104049]: 2024-03-28 22:22:48.843734234 +0000 UTC m=+5.466415331 container died 10ce830a882198055a63438efc9a574581483a1cf217890705e6067443077a4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, RELEASE=squid-323d23d, ceph=True) 2024-03-28T22:22:49.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:49 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:49.178+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:49.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:49 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:49.178+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:49.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:49 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:49.178+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:49.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:49 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:49.178+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:22:49.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:49 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:49.178+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:07.780894+0000 front 2024-03-28T22:22:07.780921+0000 (oldest deadline 2024-03-28T22:22:33.080880+0000) 2024-03-28T22:22:49.431 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:49 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:49.178+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:13.081407+0000 front 2024-03-28T22:22:13.081352+0000 (oldest deadline 2024-03-28T22:22:37.181460+0000) 2024-03-28T22:22:49.434 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:49 smithi016 podman[104049]: 2024-03-28 22:22:49.163032931 +0000 UTC m=+5.785714019 container cleanup 10ce830a882198055a63438efc9a574581483a1cf217890705e6067443077a4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-323d23d, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-28T22:22:49.434 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:49 smithi016 bash[104049]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9 2024-03-28T22:22:49.434 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:49 smithi016 podman[104066]: 2024-03-28 22:22:49.347458509 +0000 UTC m=+0.505062575 container remove 10ce830a882198055a63438efc9a574581483a1cf217890705e6067443077a4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-323d23d, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T22:22:49.947 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:49 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:49.552+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:49.947 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:49 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:49.552+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:49.947 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:49 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:49.552+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:49.948 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:49 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:49.552+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:57.638872+0000 front 2024-03-28T22:21:57.638939+0000 (oldest deadline 2024-03-28T22:22:21.739096+0000) 2024-03-28T22:22:49.948 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:49 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:49.552+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:05.140891+0000 front 2024-03-28T22:22:05.140920+0000 (oldest deadline 2024-03-28T22:22:30.441027+0000) 2024-03-28T22:22:49.948 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:49 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:49.552+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:11.542058+0000 front 2024-03-28T22:22:11.542081+0000 (oldest deadline 2024-03-28T22:22:37.442130+0000) 2024-03-28T22:22:50.058 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:49 smithi016 podman[104155]: 2024-03-28 22:22:49.66665268 +0000 UTC m=+0.017858493 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:22:50.058 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:49 smithi016 podman[104155]: 2024-03-28 22:22:49.773193501 +0000 UTC m=+0.124399331 container create 3c1eb52a2a8c81c42ed77a0fb6809e31bfced8d811da7a52cbb9b4fd6696cb9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-323d23d, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-28T22:22:50.058 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:49 smithi016 podman[104155]: 2024-03-28 22:22:49.98382404 +0000 UTC m=+0.335029837 container init 3c1eb52a2a8c81c42ed77a0fb6809e31bfced8d811da7a52cbb9b4fd6696cb9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-323d23d, GIT_CLEAN=True, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-28T22:22:50.059 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:49 smithi016 podman[104155]: 2024-03-28 22:22:49.987046153 +0000 UTC m=+0.338251939 container start 3c1eb52a2a8c81c42ed77a0fb6809e31bfced8d811da7a52cbb9b4fd6696cb9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-323d23d, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T22:22:50.059 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:50 smithi016 podman[104155]: 2024-03-28 22:22:50.058586142 +0000 UTC m=+0.409791932 container attach 3c1eb52a2a8c81c42ed77a0fb6809e31bfced8d811da7a52cbb9b4fd6696cb9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, RELEASE=squid-323d23d, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T22:22:50.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:50 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:50.154+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:50.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:50 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:50.154+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:50.431 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:50 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:50.154+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:50.431 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:50 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:50.154+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:22:50.431 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:50 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:50.154+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:07.780894+0000 front 2024-03-28T22:22:07.780921+0000 (oldest deadline 2024-03-28T22:22:33.080880+0000) 2024-03-28T22:22:50.431 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:50 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:50.154+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:13.081407+0000 front 2024-03-28T22:22:13.081352+0000 (oldest deadline 2024-03-28T22:22:37.181460+0000) 2024-03-28T22:22:50.434 INFO:journalctl@ceph.osd.9.smithi016.stdout:Mar 28 22:22:50 smithi016 podman[104155]: 2024-03-28 22:22:50.285347153 +0000 UTC m=+0.636552954 container died 3c1eb52a2a8c81c42ed77a0fb6809e31bfced8d811da7a52cbb9b4fd6696cb9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-323d23d, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-28T22:22:50.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:50 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:50.526+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:50.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:50 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:50.526+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:50.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:50 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:50.526+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:50.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:50 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:50.526+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:57.638872+0000 front 2024-03-28T22:21:57.638939+0000 (oldest deadline 2024-03-28T22:22:21.739096+0000) 2024-03-28T22:22:50.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:50 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:50.526+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:05.140891+0000 front 2024-03-28T22:22:05.140920+0000 (oldest deadline 2024-03-28T22:22:30.441027+0000) 2024-03-28T22:22:50.797 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:50 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:50.526+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:11.542058+0000 front 2024-03-28T22:22:11.542081+0000 (oldest deadline 2024-03-28T22:22:37.442130+0000) 2024-03-28T22:22:50.797 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:50 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:50.526+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-03-28T22:22:24.444000+0000 front 2024-03-28T22:22:24.443895+0000 (oldest deadline 2024-03-28T22:22:50.343719+0000) 2024-03-28T22:22:51.009 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.9.service' 2024-03-28T22:22:51.092 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:22:51.093 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-28T22:22:51.093 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-28T22:22:51.093 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.10 2024-03-28T22:22:51.395 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:51 smithi053 systemd[1]: Stopping Ceph osd.10 for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:22:51.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:51 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:51.165+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:51.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:51 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:51.165+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:51.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:51 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:51.165+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:51.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:51 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:51.165+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:22:51.431 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:51 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:51.165+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:07.780894+0000 front 2024-03-28T22:22:07.780921+0000 (oldest deadline 2024-03-28T22:22:33.080880+0000) 2024-03-28T22:22:51.431 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:51 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:51.165+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:13.081407+0000 front 2024-03-28T22:22:13.081352+0000 (oldest deadline 2024-03-28T22:22:37.181460+0000) 2024-03-28T22:22:51.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:51 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:51.394+0000 7fad488ed700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:22:51.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:51 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:51.394+0000 7fad488ed700 -1 osd.10 102 *** Got signal Terminated *** 2024-03-28T22:22:51.797 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:51 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:51.394+0000 7fad488ed700 -1 osd.10 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T22:22:51.797 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:51 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:51.535+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:51.797 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:51 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:51.535+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:51.797 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:51 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:51.535+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:51.797 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:51 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:51.535+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:57.638872+0000 front 2024-03-28T22:21:57.638939+0000 (oldest deadline 2024-03-28T22:22:21.739096+0000) 2024-03-28T22:22:51.797 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:51 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:51.535+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:05.140891+0000 front 2024-03-28T22:22:05.140920+0000 (oldest deadline 2024-03-28T22:22:30.441027+0000) 2024-03-28T22:22:51.798 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:51 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:51.535+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:11.542058+0000 front 2024-03-28T22:22:11.542081+0000 (oldest deadline 2024-03-28T22:22:37.442130+0000) 2024-03-28T22:22:51.798 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:51 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:51.535+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-03-28T22:22:24.444000+0000 front 2024-03-28T22:22:24.443895+0000 (oldest deadline 2024-03-28T22:22:50.343719+0000) 2024-03-28T22:22:52.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:52 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:52.197+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:52.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:52 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:52.197+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:52.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:52 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:52.197+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:52.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:52 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:52.197+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:22:52.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:52 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:52.197+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:07.780894+0000 front 2024-03-28T22:22:07.780921+0000 (oldest deadline 2024-03-28T22:22:33.080880+0000) 2024-03-28T22:22:52.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:52 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:52.197+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:13.081407+0000 front 2024-03-28T22:22:13.081352+0000 (oldest deadline 2024-03-28T22:22:37.181460+0000) 2024-03-28T22:22:53.046 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:52 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:52.564+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:53.046 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:52 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:52.564+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:53.046 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:52 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:52.564+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:53.046 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:52 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:52.564+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:57.638872+0000 front 2024-03-28T22:21:57.638939+0000 (oldest deadline 2024-03-28T22:22:21.739096+0000) 2024-03-28T22:22:53.047 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:52 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:52.564+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:05.140891+0000 front 2024-03-28T22:22:05.140920+0000 (oldest deadline 2024-03-28T22:22:30.441027+0000) 2024-03-28T22:22:53.047 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:52 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:52.564+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:11.542058+0000 front 2024-03-28T22:22:11.542081+0000 (oldest deadline 2024-03-28T22:22:37.442130+0000) 2024-03-28T22:22:53.047 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:52 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:52.564+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-03-28T22:22:24.444000+0000 front 2024-03-28T22:22:24.443895+0000 (oldest deadline 2024-03-28T22:22:50.343719+0000) 2024-03-28T22:22:53.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:53 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:53.212+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:53.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:53 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:53.213+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:53.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:53 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:53.213+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:53.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:53 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:53.213+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:22:53.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:53 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:53.213+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:07.780894+0000 front 2024-03-28T22:22:07.780921+0000 (oldest deadline 2024-03-28T22:22:33.080880+0000) 2024-03-28T22:22:53.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:53 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:53.213+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:13.081407+0000 front 2024-03-28T22:22:13.081352+0000 (oldest deadline 2024-03-28T22:22:37.181460+0000) 2024-03-28T22:22:53.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:53 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:53.515+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:53.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:53 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:53.515+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:53.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:53 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:53.515+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:53.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:53 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:53.515+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:57.638872+0000 front 2024-03-28T22:21:57.638939+0000 (oldest deadline 2024-03-28T22:22:21.739096+0000) 2024-03-28T22:22:53.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:53 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:53.515+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:05.140891+0000 front 2024-03-28T22:22:05.140920+0000 (oldest deadline 2024-03-28T22:22:30.441027+0000) 2024-03-28T22:22:53.797 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:53 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:53.515+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:11.542058+0000 front 2024-03-28T22:22:11.542081+0000 (oldest deadline 2024-03-28T22:22:37.442130+0000) 2024-03-28T22:22:53.797 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:53 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:53.515+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-03-28T22:22:24.444000+0000 front 2024-03-28T22:22:24.443895+0000 (oldest deadline 2024-03-28T22:22:50.343719+0000) 2024-03-28T22:22:54.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:54 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:54.258+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:54.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:54 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:54.258+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:54.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:54 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:54.258+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:54.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:54 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:54.258+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:22:54.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:54 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:54.258+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:07.780894+0000 front 2024-03-28T22:22:07.780921+0000 (oldest deadline 2024-03-28T22:22:33.080880+0000) 2024-03-28T22:22:54.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:54 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:54.258+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:13.081407+0000 front 2024-03-28T22:22:13.081352+0000 (oldest deadline 2024-03-28T22:22:37.181460+0000) 2024-03-28T22:22:54.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:54 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:54.466+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:54.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:54 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:54.466+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:54.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:54 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:54.466+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:54.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:54 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:54.466+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:57.638872+0000 front 2024-03-28T22:21:57.638939+0000 (oldest deadline 2024-03-28T22:22:21.739096+0000) 2024-03-28T22:22:54.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:54 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:54.466+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:05.140891+0000 front 2024-03-28T22:22:05.140920+0000 (oldest deadline 2024-03-28T22:22:30.441027+0000) 2024-03-28T22:22:54.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:54 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:54.466+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:11.542058+0000 front 2024-03-28T22:22:11.542081+0000 (oldest deadline 2024-03-28T22:22:37.442130+0000) 2024-03-28T22:22:54.797 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:54 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:54.466+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-03-28T22:22:24.444000+0000 front 2024-03-28T22:22:24.443895+0000 (oldest deadline 2024-03-28T22:22:50.343719+0000) 2024-03-28T22:22:55.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:55 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:55.279+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:55.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:55 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:55.279+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:55.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:55 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:55.279+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:55.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:55 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:55.279+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:22:55.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:55 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:55.279+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:07.780894+0000 front 2024-03-28T22:22:07.780921+0000 (oldest deadline 2024-03-28T22:22:33.080880+0000) 2024-03-28T22:22:55.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:55 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:55.279+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:13.081407+0000 front 2024-03-28T22:22:13.081352+0000 (oldest deadline 2024-03-28T22:22:37.181460+0000) 2024-03-28T22:22:55.795 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:55 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:55.470+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:34.836190+0000 front 2024-03-28T22:21:34.836069+0000 (oldest deadline 2024-03-28T22:22:00.136187+0000) 2024-03-28T22:22:55.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:55 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:55.470+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:44.837066+0000 front 2024-03-28T22:21:44.837110+0000 (oldest deadline 2024-03-28T22:22:08.937290+0000) 2024-03-28T22:22:55.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:55 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:55.470+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.937584+0000 front 2024-03-28T22:21:48.937621+0000 (oldest deadline 2024-03-28T22:22:14.237734+0000) 2024-03-28T22:22:55.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:55 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:55.470+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:57.638872+0000 front 2024-03-28T22:21:57.638939+0000 (oldest deadline 2024-03-28T22:22:21.739096+0000) 2024-03-28T22:22:55.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:55 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:55.470+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:05.140891+0000 front 2024-03-28T22:22:05.140920+0000 (oldest deadline 2024-03-28T22:22:30.441027+0000) 2024-03-28T22:22:55.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:55 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:55.470+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:11.542058+0000 front 2024-03-28T22:22:11.542081+0000 (oldest deadline 2024-03-28T22:22:37.442130+0000) 2024-03-28T22:22:55.796 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:55 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10[61174]: 2024-03-28T22:22:55.470+0000 7fad456e2700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-03-28T22:22:24.444000+0000 front 2024-03-28T22:22:24.443895+0000 (oldest deadline 2024-03-28T22:22:50.343719+0000) 2024-03-28T22:22:56.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:56 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:56.260+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:56.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:56 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:56.260+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:56.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:56 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:56.260+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:56.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:56 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:56.260+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:22:56.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:56 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:56.260+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:07.780894+0000 front 2024-03-28T22:22:07.780921+0000 (oldest deadline 2024-03-28T22:22:33.080880+0000) 2024-03-28T22:22:56.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:56 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:56.260+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:13.081407+0000 front 2024-03-28T22:22:13.081352+0000 (oldest deadline 2024-03-28T22:22:37.181460+0000) 2024-03-28T22:22:57.046 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:56 smithi053 podman[72646]: 2024-03-28 22:22:56.760810261 +0000 UTC m=+5.463511547 container died 95a527461e986bd902654afd6a3121c0f4698eea623d94925a34cd5ee9fa4db4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-323d23d, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T22:22:57.436 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:57 smithi053 podman[72646]: 2024-03-28 22:22:57.077436888 +0000 UTC m=+5.780138193 container cleanup 95a527461e986bd902654afd6a3121c0f4698eea623d94925a34cd5ee9fa4db4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-323d23d, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-28T22:22:57.437 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:57 smithi053 bash[72646]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10 2024-03-28T22:22:57.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:57 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:57.304+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:57.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:57 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:57.304+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:57.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:57 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:57.304+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:57.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:57 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:57.304+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:22:57.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:57 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:57.304+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:07.780894+0000 front 2024-03-28T22:22:07.780921+0000 (oldest deadline 2024-03-28T22:22:33.080880+0000) 2024-03-28T22:22:57.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:57 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:57.304+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:13.081407+0000 front 2024-03-28T22:22:13.081352+0000 (oldest deadline 2024-03-28T22:22:37.181460+0000) 2024-03-28T22:22:57.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:57 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:57.304+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6820 osd.7 since back 2024-03-28T22:22:32.085409+0000 front 2024-03-28T22:22:32.085369+0000 (oldest deadline 2024-03-28T22:22:56.785212+0000) 2024-03-28T22:22:57.795 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:57 smithi053 podman[72663]: 2024-03-28 22:22:57.436840721 +0000 UTC m=+0.673285494 container remove 95a527461e986bd902654afd6a3121c0f4698eea623d94925a34cd5ee9fa4db4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-323d23d, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-28T22:22:58.131 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:57 smithi053 podman[72758]: 2024-03-28 22:22:57.837790076 +0000 UTC m=+0.102946337 container create e3c757368eee2efd4c528159e2abc293ea08398957141a663401eede0b1cdc22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, RELEASE=squid-323d23d, maintainer=Guillaume Abrioux , GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T22:22:58.131 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:57 smithi053 podman[72758]: 2024-03-28 22:22:57.752709274 +0000 UTC m=+0.017865552 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:22:58.131 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:58 smithi053 podman[72758]: 2024-03-28 22:22:58.048884483 +0000 UTC m=+0.314040838 container init e3c757368eee2efd4c528159e2abc293ea08398957141a663401eede0b1cdc22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10-deactivate, org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-323d23d, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5) 2024-03-28T22:22:58.132 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:58 smithi053 podman[72758]: 2024-03-28 22:22:58.052639802 +0000 UTC m=+0.317796076 container start e3c757368eee2efd4c528159e2abc293ea08398957141a663401eede0b1cdc22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10-deactivate, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-323d23d, org.label-schema.license=GPLv2, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-28T22:22:58.132 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:58 smithi053 podman[72758]: 2024-03-28 22:22:58.131643896 +0000 UTC m=+0.396800192 container attach e3c757368eee2efd4c528159e2abc293ea08398957141a663401eede0b1cdc22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, RELEASE=squid-323d23d, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True) 2024-03-28T22:22:58.545 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:58 smithi053 podman[72758]: 2024-03-28 22:22:58.351412838 +0000 UTC m=+0.616569101 container died e3c757368eee2efd4c528159e2abc293ea08398957141a663401eede0b1cdc22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-323d23d, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-28T22:22:58.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:58 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:58.334+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:58.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:58 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:58.334+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:58.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:58 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:58.334+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:58.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:58 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:58.334+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:22:58.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:58 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:58.334+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:07.780894+0000 front 2024-03-28T22:22:07.780921+0000 (oldest deadline 2024-03-28T22:22:33.080880+0000) 2024-03-28T22:22:58.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:58 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:58.334+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:13.081407+0000 front 2024-03-28T22:22:13.081352+0000 (oldest deadline 2024-03-28T22:22:37.181460+0000) 2024-03-28T22:22:58.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:58 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:58.334+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6820 osd.7 since back 2024-03-28T22:22:32.085409+0000 front 2024-03-28T22:22:32.085369+0000 (oldest deadline 2024-03-28T22:22:56.785212+0000) 2024-03-28T22:22:58.900 INFO:journalctl@ceph.osd.10.smithi053.stdout:Mar 28 22:22:58 smithi053 podman[72783]: 2024-03-28 22:22:58.885167135 +0000 UTC m=+0.527895750 container remove e3c757368eee2efd4c528159e2abc293ea08398957141a663401eede0b1cdc22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-10-deactivate, org.label-schema.build-date=20240326, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-323d23d, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-28T22:22:58.906 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.10.service' 2024-03-28T22:22:58.951 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:22:58.952 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-28T22:22:58.952 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-28T22:22:58.952 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.11 2024-03-28T22:22:59.260 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:59 smithi063 systemd[1]: Stopping Ceph osd.11 for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:22:59.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:59 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:59.259+0000 7f7fc0e0d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:22:59.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:59 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:59.259+0000 7f7fc0e0d700 -1 osd.11 102 *** Got signal Terminated *** 2024-03-28T22:22:59.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:59 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:59.259+0000 7f7fc0e0d700 -1 osd.11 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T22:22:59.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:59 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:59.298+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:22:59.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:59 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:59.298+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:22:59.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:59 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:59.298+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:22:59.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:59 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:59.298+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:22:59.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:59 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:59.298+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:07.780894+0000 front 2024-03-28T22:22:07.780921+0000 (oldest deadline 2024-03-28T22:22:33.080880+0000) 2024-03-28T22:22:59.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:59 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:59.298+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:13.081407+0000 front 2024-03-28T22:22:13.081352+0000 (oldest deadline 2024-03-28T22:22:37.181460+0000) 2024-03-28T22:22:59.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:22:59 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:22:59.298+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6820 osd.7 since back 2024-03-28T22:22:32.085409+0000 front 2024-03-28T22:22:32.085369+0000 (oldest deadline 2024-03-28T22:22:56.785212+0000) 2024-03-28T22:23:00.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:00 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:23:00.340+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:23:00.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:00 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:23:00.340+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:23:00.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:00 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:23:00.340+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:23:00.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:00 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:23:00.340+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:23:00.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:00 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:23:00.340+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:07.780894+0000 front 2024-03-28T22:22:07.780921+0000 (oldest deadline 2024-03-28T22:22:33.080880+0000) 2024-03-28T22:23:00.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:00 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:23:00.340+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:13.081407+0000 front 2024-03-28T22:22:13.081352+0000 (oldest deadline 2024-03-28T22:22:37.181460+0000) 2024-03-28T22:23:00.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:00 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:23:00.340+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6820 osd.7 since back 2024-03-28T22:22:32.085409+0000 front 2024-03-28T22:22:32.085369+0000 (oldest deadline 2024-03-28T22:22:56.785212+0000) 2024-03-28T22:23:01.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:01 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:23:01.364+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:23:01.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:01 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:23:01.364+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:23:01.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:01 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:23:01.364+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:23:01.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:01 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:23:01.364+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:23:01.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:01 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:23:01.364+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:07.780894+0000 front 2024-03-28T22:22:07.780921+0000 (oldest deadline 2024-03-28T22:22:33.080880+0000) 2024-03-28T22:23:01.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:01 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:23:01.364+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:13.081407+0000 front 2024-03-28T22:22:13.081352+0000 (oldest deadline 2024-03-28T22:22:37.181460+0000) 2024-03-28T22:23:01.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:01 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:23:01.364+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6820 osd.7 since back 2024-03-28T22:22:32.085409+0000 front 2024-03-28T22:22:32.085369+0000 (oldest deadline 2024-03-28T22:22:56.785212+0000) 2024-03-28T22:23:02.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:02 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:23:02.382+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:23:02.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:02 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:23:02.382+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:23:02.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:02 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:23:02.382+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:23:02.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:02 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:23:02.382+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:23:02.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:02 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:23:02.382+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:07.780894+0000 front 2024-03-28T22:22:07.780921+0000 (oldest deadline 2024-03-28T22:22:33.080880+0000) 2024-03-28T22:23:02.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:02 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:23:02.382+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:13.081407+0000 front 2024-03-28T22:22:13.081352+0000 (oldest deadline 2024-03-28T22:22:37.181460+0000) 2024-03-28T22:23:02.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:02 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:23:02.382+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6820 osd.7 since back 2024-03-28T22:22:32.085409+0000 front 2024-03-28T22:22:32.085369+0000 (oldest deadline 2024-03-28T22:22:56.785212+0000) 2024-03-28T22:23:02.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:02 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:23:02.382+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6820 osd.8 since back 2024-03-28T22:22:36.785571+0000 front 2024-03-28T22:22:36.785593+0000 (oldest deadline 2024-03-28T22:23:02.085569+0000) 2024-03-28T22:23:03.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:03 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:23:03.365+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-03-28T22:21:33.275682+0000 front 2024-03-28T22:21:33.275508+0000 (oldest deadline 2024-03-28T22:21:59.175619+0000) 2024-03-28T22:23:03.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:03 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:23:03.365+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6804 osd.1 since back 2024-03-28T22:21:39.176564+0000 front 2024-03-28T22:21:39.176457+0000 (oldest deadline 2024-03-28T22:22:08.576958+0000) 2024-03-28T22:23:03.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:03 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:23:03.365+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6804 osd.2 since back 2024-03-28T22:21:48.577262+0000 front 2024-03-28T22:21:48.577280+0000 (oldest deadline 2024-03-28T22:22:16.178094+0000) 2024-03-28T22:23:03.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:03 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:23:03.365+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-03-28T22:21:56.178729+0000 front 2024-03-28T22:21:56.178579+0000 (oldest deadline 2024-03-28T22:22:22.078702+0000) 2024-03-28T22:23:03.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:03 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:23:03.365+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6812 osd.4 since back 2024-03-28T22:22:07.780894+0000 front 2024-03-28T22:22:07.780921+0000 (oldest deadline 2024-03-28T22:22:33.080880+0000) 2024-03-28T22:23:03.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:03 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:23:03.365+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6812 osd.5 since back 2024-03-28T22:22:13.081407+0000 front 2024-03-28T22:22:13.081352+0000 (oldest deadline 2024-03-28T22:22:37.181460+0000) 2024-03-28T22:23:03.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:03 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:23:03.365+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.53:6820 osd.7 since back 2024-03-28T22:22:32.085409+0000 front 2024-03-28T22:22:32.085369+0000 (oldest deadline 2024-03-28T22:22:56.785212+0000) 2024-03-28T22:23:03.681 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:03 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11[61638]: 2024-03-28T22:23:03.365+0000 7f7fb9bfa700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.63:6820 osd.8 since back 2024-03-28T22:22:36.785571+0000 front 2024-03-28T22:22:36.785593+0000 (oldest deadline 2024-03-28T22:23:02.085569+0000) 2024-03-28T22:23:05.052 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:04 smithi063 podman[73597]: 2024-03-28 22:23:04.782771453 +0000 UTC m=+5.622230500 container died 166707260b9517961b176299e72879f5afdee829d624e44d145e09fc3916d762 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, RELEASE=squid-323d23d, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-28T22:23:05.410 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:05 smithi063 podman[73597]: 2024-03-28 22:23:05.143377456 +0000 UTC m=+5.982836495 container cleanup 166707260b9517961b176299e72879f5afdee829d624e44d145e09fc3916d762 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-323d23d, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS) 2024-03-28T22:23:05.411 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:05 smithi063 bash[73597]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11 2024-03-28T22:23:05.680 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:05 smithi063 podman[73614]: 2024-03-28 22:23:05.411308146 +0000 UTC m=+0.625116420 container remove 166707260b9517961b176299e72879f5afdee829d624e44d145e09fc3916d762 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=squid-323d23d, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-28T22:23:06.038 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:05 smithi063 podman[73701]: 2024-03-28 22:23:05.76205592 +0000 UTC m=+0.108634185 container create 132c53575fd14ccd7f7ba1a241ecaf6bf26de8cf87f5e6ac9d4c85c43fc10e4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=squid-323d23d, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-28T22:23:06.039 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:05 smithi063 podman[73701]: 2024-03-28 22:23:05.671470738 +0000 UTC m=+0.018049011 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:23:06.039 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:05 smithi063 podman[73701]: 2024-03-28 22:23:05.956305889 +0000 UTC m=+0.302884166 container init 132c53575fd14ccd7f7ba1a241ecaf6bf26de8cf87f5e6ac9d4c85c43fc10e4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11-deactivate, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-323d23d, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T22:23:06.039 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:05 smithi063 podman[73701]: 2024-03-28 22:23:05.959932887 +0000 UTC m=+0.306511163 container start 132c53575fd14ccd7f7ba1a241ecaf6bf26de8cf87f5e6ac9d4c85c43fc10e4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=squid-323d23d, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-28T22:23:06.039 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:06 smithi063 podman[73701]: 2024-03-28 22:23:06.03933539 +0000 UTC m=+0.385913663 container attach 132c53575fd14ccd7f7ba1a241ecaf6bf26de8cf87f5e6ac9d4c85c43fc10e4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11-deactivate, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-323d23d, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T22:23:06.430 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:06 smithi063 podman[73701]: 2024-03-28 22:23:06.260392225 +0000 UTC m=+0.606970500 container died 132c53575fd14ccd7f7ba1a241ecaf6bf26de8cf87f5e6ac9d4c85c43fc10e4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=squid-323d23d, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True) 2024-03-28T22:23:06.955 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.11.service' 2024-03-28T22:23:06.986 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:06 smithi063 podman[73728]: 2024-03-28 22:23:06.934307996 +0000 UTC m=+0.667912799 container remove 132c53575fd14ccd7f7ba1a241ecaf6bf26de8cf87f5e6ac9d4c85c43fc10e4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-323d23d, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-28T22:23:06.987 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:06 smithi063 systemd[1]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.11.service: Deactivated successfully. 2024-03-28T22:23:06.987 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:06 smithi063 systemd[1]: Stopped Ceph osd.11 for 798d9884-ed4e-11ee-95d0-87774f69a715. 2024-03-28T22:23:06.987 INFO:journalctl@ceph.osd.11.smithi063.stdout:Mar 28 22:23:06 smithi063 systemd[1]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715@osd.11.service: Consumed 14.677s CPU time. 2024-03-28T22:23:07.003 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:23:07.003 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-28T22:23:07.004 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-28T22:23:07.004 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mds.a 2024-03-28T22:23:07.314 INFO:journalctl@ceph.mds.a.smithi016.stdout:Mar 28 22:23:07 smithi016 systemd[1]: Stopping Ceph mds.a for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:23:07.684 INFO:journalctl@ceph.mds.a.smithi016.stdout:Mar 28 22:23:07 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-a[72078]: 2024-03-28T22:23:07.314+0000 7efdaf3f8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:23:07.684 INFO:journalctl@ceph.mds.a.smithi016.stdout:Mar 28 22:23:07 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-a[72078]: 2024-03-28T22:23:07.314+0000 7efdaf3f8700 -1 mds.a *** got signal Terminated *** 2024-03-28T22:23:10.340 INFO:journalctl@ceph.mds.a.smithi016.stdout:Mar 28 22:23:10 smithi016 podman[104277]: 2024-03-28 22:23:10.065705191 +0000 UTC m=+2.864991085 container died 6cf1ec98898ed670c6cf70c2fd5120bd85450d6888d22a3d301ef74b2dd211b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-323d23d, org.label-schema.license=GPLv2) 2024-03-28T22:23:10.684 INFO:journalctl@ceph.mds.a.smithi016.stdout:Mar 28 22:23:10 smithi016 podman[104277]: 2024-03-28 22:23:10.422320222 +0000 UTC m=+3.221606113 container cleanup 6cf1ec98898ed670c6cf70c2fd5120bd85450d6888d22a3d301ef74b2dd211b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-a, RELEASE=squid-323d23d, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-28T22:23:10.684 INFO:journalctl@ceph.mds.a.smithi016.stdout:Mar 28 22:23:10 smithi016 bash[104277]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-a 2024-03-28T22:23:10.913 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mds.a.service' 2024-03-28T22:23:10.951 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:23:10.951 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-28T22:23:10.951 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-28T22:23:10.952 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mds.d 2024-03-28T22:23:11.434 INFO:journalctl@ceph.mds.d.smithi016.stdout:Mar 28 22:23:11 smithi016 systemd[1]: Stopping Ceph mds.d for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:23:11.435 INFO:journalctl@ceph.mds.d.smithi016.stdout:Mar 28 22:23:11 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-d[71117]: 2024-03-28T22:23:11.258+0000 7f1a447d1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:23:11.435 INFO:journalctl@ceph.mds.d.smithi016.stdout:Mar 28 22:23:11 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-d[71117]: 2024-03-28T22:23:11.258+0000 7f1a447d1700 -1 mds.d *** got signal Terminated *** 2024-03-28T22:23:15.251 INFO:journalctl@ceph.mds.d.smithi016.stdout:Mar 28 22:23:14 smithi016 podman[104399]: 2024-03-28 22:23:14.990045502 +0000 UTC m=+3.814695378 container died ef92b8d3de0cf5b93271e267ea76be0895d9de863f6a0870e6d33d438520ca1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=squid-323d23d, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-28T22:23:15.684 INFO:journalctl@ceph.mds.d.smithi016.stdout:Mar 28 22:23:15 smithi016 podman[104399]: 2024-03-28 22:23:15.442454326 +0000 UTC m=+4.267104191 container cleanup ef92b8d3de0cf5b93271e267ea76be0895d9de863f6a0870e6d33d438520ca1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-d, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-323d23d, org.label-schema.build-date=20240326) 2024-03-28T22:23:15.685 INFO:journalctl@ceph.mds.d.smithi016.stdout:Mar 28 22:23:15 smithi016 bash[104399]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-d 2024-03-28T22:23:15.812 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mds.d.service' 2024-03-28T22:23:15.847 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:23:15.847 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-28T22:23:15.848 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-28T22:23:15.848 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mds.g 2024-03-28T22:23:16.184 INFO:journalctl@ceph.mds.g.smithi016.stdout:Mar 28 22:23:15 smithi016 systemd[1]: Stopping Ceph mds.g for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:23:16.684 INFO:journalctl@ceph.mds.g.smithi016.stdout:Mar 28 22:23:16 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-g[71726]: 2024-03-28T22:23:16.270+0000 7f66da48d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:23:16.685 INFO:journalctl@ceph.mds.g.smithi016.stdout:Mar 28 22:23:16 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-g[71726]: 2024-03-28T22:23:16.270+0000 7f66da48d700 -1 mds.g *** got signal Terminated *** 2024-03-28T22:23:19.812 INFO:journalctl@ceph.mds.g.smithi016.stdout:Mar 28 22:23:19 smithi016 podman[104519]: 2024-03-28 22:23:19.501443756 +0000 UTC m=+3.436154797 container died ed462ec3a52b19bbb79d4ffc4e74b4c8f95cdd3bddd2d5984a20208310841091 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-g, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_CLEAN=True, RELEASE=squid-323d23d, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-28T22:23:19.812 INFO:journalctl@ceph.mds.g.smithi016.stdout:Mar 28 22:23:19 smithi016 podman[104519]: 2024-03-28 22:23:19.812615502 +0000 UTC m=+3.747326528 container cleanup ed462ec3a52b19bbb79d4ffc4e74b4c8f95cdd3bddd2d5984a20208310841091 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-g, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-323d23d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-28T22:23:20.098 INFO:journalctl@ceph.mds.g.smithi016.stdout:Mar 28 22:23:19 smithi016 bash[104519]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-g 2024-03-28T22:23:20.099 INFO:journalctl@ceph.mds.g.smithi016.stdout:Mar 28 22:23:19 smithi016 podman[104531]: 2024-03-28 22:23:19.988659271 +0000 UTC m=+0.481567346 container remove ed462ec3a52b19bbb79d4ffc4e74b4c8f95cdd3bddd2d5984a20208310841091 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-g, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-323d23d, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-28T22:23:20.115 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mds.g.service' 2024-03-28T22:23:20.149 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:23:20.149 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-28T22:23:20.149 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-28T22:23:20.149 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mds.j 2024-03-28T22:23:20.434 INFO:journalctl@ceph.mds.j.smithi016.stdout:Mar 28 22:23:20 smithi016 systemd[1]: Stopping Ceph mds.j for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:23:20.934 INFO:journalctl@ceph.mds.j.smithi016.stdout:Mar 28 22:23:20 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-j[70770]: 2024-03-28T22:23:20.465+0000 7ff9c0528700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:23:20.934 INFO:journalctl@ceph.mds.j.smithi016.stdout:Mar 28 22:23:20 smithi016 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-j[70770]: 2024-03-28T22:23:20.465+0000 7ff9c0528700 -1 mds.j *** got signal Terminated *** 2024-03-28T22:23:24.599 INFO:journalctl@ceph.mds.j.smithi016.stdout:Mar 28 22:23:24 smithi016 podman[104638]: 2024-03-28 22:23:24.218918516 +0000 UTC m=+3.848823951 container died 83061c52f5f61b02eb1c9a788df9845a9e56991953c010642c83c67fab95e06e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-j, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-323d23d, GIT_BRANCH=HEAD, ceph=True) 2024-03-28T22:23:24.933 INFO:journalctl@ceph.mds.j.smithi016.stdout:Mar 28 22:23:24 smithi016 podman[104638]: 2024-03-28 22:23:24.67407856 +0000 UTC m=+4.303983989 container cleanup 83061c52f5f61b02eb1c9a788df9845a9e56991953c010642c83c67fab95e06e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-j, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-323d23d, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-28T22:23:24.933 INFO:journalctl@ceph.mds.j.smithi016.stdout:Mar 28 22:23:24 smithi016 bash[104638]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-j 2024-03-28T22:23:25.099 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mds.j.service' 2024-03-28T22:23:25.133 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:23:25.133 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-28T22:23:25.133 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-28T22:23:25.133 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mds.b 2024-03-28T22:23:25.436 INFO:journalctl@ceph.mds.b.smithi053.stdout:Mar 28 22:23:25 smithi053 systemd[1]: Stopping Ceph mds.b for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:23:25.718 INFO:journalctl@ceph.mds.b.smithi053.stdout:Mar 28 22:23:25 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-b[66455]: 2024-03-28T22:23:25.435+0000 7f76e8909700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:23:25.718 INFO:journalctl@ceph.mds.b.smithi053.stdout:Mar 28 22:23:25 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-b[66455]: 2024-03-28T22:23:25.435+0000 7f76e8909700 -1 mds.b *** got signal Terminated *** 2024-03-28T22:23:29.797 INFO:journalctl@ceph.mds.b.smithi053.stdout:Mar 28 22:23:29 smithi053 podman[72862]: 2024-03-28 22:23:29.449476564 +0000 UTC m=+4.113228006 container died 19904610fabe9d4a77a857d8b1f474031e557a481a5f34d9395ec9433588a45a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-b, org.label-schema.vendor=CentOS, RELEASE=squid-323d23d, org.label-schema.build-date=20240326, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-28T22:23:30.188 INFO:journalctl@ceph.mds.b.smithi053.stdout:Mar 28 22:23:29 smithi053 podman[72862]: 2024-03-28 22:23:29.927023169 +0000 UTC m=+4.590774608 container cleanup 19904610fabe9d4a77a857d8b1f474031e557a481a5f34d9395ec9433588a45a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-323d23d, ceph=True) 2024-03-28T22:23:30.188 INFO:journalctl@ceph.mds.b.smithi053.stdout:Mar 28 22:23:29 smithi053 bash[72862]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-b 2024-03-28T22:23:30.317 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mds.b.service' 2024-03-28T22:23:30.359 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:23:30.359 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-28T22:23:30.360 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-28T22:23:30.360 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mds.e 2024-03-28T22:23:30.715 INFO:journalctl@ceph.mds.e.smithi053.stdout:Mar 28 22:23:30 smithi053 systemd[1]: Stopping Ceph mds.e for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:23:31.046 INFO:journalctl@ceph.mds.e.smithi053.stdout:Mar 28 22:23:30 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-e[65829]: 2024-03-28T22:23:30.715+0000 7fa93bd78700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:23:31.046 INFO:journalctl@ceph.mds.e.smithi053.stdout:Mar 28 22:23:30 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-e[65829]: 2024-03-28T22:23:30.715+0000 7fa93bd78700 -1 mds.e *** got signal Terminated *** 2024-03-28T22:23:34.031 INFO:journalctl@ceph.mds.e.smithi053.stdout:Mar 28 22:23:33 smithi053 podman[72996]: 2024-03-28 22:23:33.641216017 +0000 UTC m=+3.058568151 container died 71e30f2bb06caa0523c2a575d9a073e33918894a1d13aea86e7b33c9f5b21f72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-e, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, RELEASE=squid-323d23d, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T22:23:34.295 INFO:journalctl@ceph.mds.e.smithi053.stdout:Mar 28 22:23:34 smithi053 podman[72996]: 2024-03-28 22:23:34.10591315 +0000 UTC m=+3.523265284 container cleanup 71e30f2bb06caa0523c2a575d9a073e33918894a1d13aea86e7b33c9f5b21f72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-323d23d, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-28T22:23:34.296 INFO:journalctl@ceph.mds.e.smithi053.stdout:Mar 28 22:23:34 smithi053 bash[72996]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-e 2024-03-28T22:23:34.536 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mds.e.service' 2024-03-28T22:23:34.576 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:23:34.576 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-28T22:23:34.576 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-28T22:23:34.577 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mds.h 2024-03-28T22:23:34.937 INFO:journalctl@ceph.mds.h.smithi053.stdout:Mar 28 22:23:34 smithi053 systemd[1]: Stopping Ceph mds.h for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:23:35.295 INFO:journalctl@ceph.mds.h.smithi053.stdout:Mar 28 22:23:34 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-h[66151]: 2024-03-28T22:23:34.934+0000 7f31db4bd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:23:35.296 INFO:journalctl@ceph.mds.h.smithi053.stdout:Mar 28 22:23:34 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-h[66151]: 2024-03-28T22:23:34.934+0000 7f31db4bd700 -1 mds.h *** got signal Terminated *** 2024-03-28T22:23:37.959 INFO:journalctl@ceph.mds.h.smithi053.stdout:Mar 28 22:23:37 smithi053 podman[73116]: 2024-03-28 22:23:37.639526551 +0000 UTC m=+2.831602985 container died e38cdb5da5853660cc07004d82e98094409a198744ba5c12a1306f73730960d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-h, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-323d23d) 2024-03-28T22:23:38.295 INFO:journalctl@ceph.mds.h.smithi053.stdout:Mar 28 22:23:37 smithi053 podman[73116]: 2024-03-28 22:23:37.959366366 +0000 UTC m=+3.151442802 container cleanup e38cdb5da5853660cc07004d82e98094409a198744ba5c12a1306f73730960d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-h, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-323d23d, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-28T22:23:38.296 INFO:journalctl@ceph.mds.h.smithi053.stdout:Mar 28 22:23:37 smithi053 bash[73116]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-h 2024-03-28T22:23:38.296 INFO:journalctl@ceph.mds.h.smithi053.stdout:Mar 28 22:23:38 smithi053 podman[73130]: 2024-03-28 22:23:38.152384889 +0000 UTC m=+0.512820868 container remove e38cdb5da5853660cc07004d82e98094409a198744ba5c12a1306f73730960d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-h, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-323d23d, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240326, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T22:23:38.401 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mds.h.service' 2024-03-28T22:23:38.441 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:23:38.441 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-28T22:23:38.441 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-28T22:23:38.442 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mds.k 2024-03-28T22:23:38.771 INFO:journalctl@ceph.mds.k.smithi053.stdout:Mar 28 22:23:38 smithi053 systemd[1]: Stopping Ceph mds.k for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:23:39.045 INFO:journalctl@ceph.mds.k.smithi053.stdout:Mar 28 22:23:38 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-k[65504]: 2024-03-28T22:23:38.770+0000 7fd6e4a50700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:23:39.046 INFO:journalctl@ceph.mds.k.smithi053.stdout:Mar 28 22:23:38 smithi053 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-k[65504]: 2024-03-28T22:23:38.770+0000 7fd6e4a50700 -1 mds.k *** got signal Terminated *** 2024-03-28T22:23:41.779 INFO:journalctl@ceph.mds.k.smithi053.stdout:Mar 28 22:23:41 smithi053 podman[73238]: 2024-03-28 22:23:41.521933186 +0000 UTC m=+2.856090247 container died 321408088b92313a15bc2b8aaf5ff1250c5defaeab95ecc4ab0a8105d08f4373 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-323d23d, org.label-schema.license=GPLv2) 2024-03-28T22:23:41.779 INFO:journalctl@ceph.mds.k.smithi053.stdout:Mar 28 22:23:41 smithi053 podman[73238]: 2024-03-28 22:23:41.779898824 +0000 UTC m=+3.114055897 container cleanup 321408088b92313a15bc2b8aaf5ff1250c5defaeab95ecc4ab0a8105d08f4373 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-k, org.label-schema.vendor=CentOS, RELEASE=squid-323d23d, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-28T22:23:42.045 INFO:journalctl@ceph.mds.k.smithi053.stdout:Mar 28 22:23:41 smithi053 bash[73238]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-k 2024-03-28T22:23:42.363 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mds.k.service' 2024-03-28T22:23:42.401 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:23:42.401 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-28T22:23:42.402 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-28T22:23:42.402 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mds.c 2024-03-28T22:23:42.701 INFO:journalctl@ceph.mds.c.smithi063.stdout:Mar 28 22:23:42 smithi063 systemd[1]: Stopping Ceph mds.c for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:23:43.030 INFO:journalctl@ceph.mds.c.smithi063.stdout:Mar 28 22:23:42 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-c[66857]: 2024-03-28T22:23:42.700+0000 7fc252225700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:23:43.030 INFO:journalctl@ceph.mds.c.smithi063.stdout:Mar 28 22:23:42 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-c[66857]: 2024-03-28T22:23:42.700+0000 7fc252225700 -1 mds.c *** got signal Terminated *** 2024-03-28T22:23:45.930 INFO:journalctl@ceph.mds.c.smithi063.stdout:Mar 28 22:23:45 smithi063 podman[73807]: 2024-03-28 22:23:45.630252518 +0000 UTC m=+3.028637680 container died 60896cd5fa5f2236598d0550ecee9e0778470a91d942c5d45f82678f40d2b6ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-c, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, RELEASE=squid-323d23d, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-28T22:23:46.218 INFO:journalctl@ceph.mds.c.smithi063.stdout:Mar 28 22:23:45 smithi063 podman[73807]: 2024-03-28 22:23:45.958997293 +0000 UTC m=+3.357382465 container cleanup 60896cd5fa5f2236598d0550ecee9e0778470a91d942c5d45f82678f40d2b6ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-c, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5, RELEASE=squid-323d23d, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-28T22:23:46.218 INFO:journalctl@ceph.mds.c.smithi063.stdout:Mar 28 22:23:45 smithi063 bash[73807]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-c 2024-03-28T22:23:46.429 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mds.c.service' 2024-03-28T22:23:46.471 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:23:46.472 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-28T22:23:46.472 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-28T22:23:46.472 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mds.f 2024-03-28T22:23:46.779 INFO:journalctl@ceph.mds.f.smithi063.stdout:Mar 28 22:23:46 smithi063 systemd[1]: Stopping Ceph mds.f for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:23:47.179 INFO:journalctl@ceph.mds.f.smithi063.stdout:Mar 28 22:23:46 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-f[66546]: 2024-03-28T22:23:46.779+0000 7fb188a17700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:23:47.180 INFO:journalctl@ceph.mds.f.smithi063.stdout:Mar 28 22:23:46 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-f[66546]: 2024-03-28T22:23:46.779+0000 7fb188a17700 -1 mds.f *** got signal Terminated *** 2024-03-28T22:23:49.880 INFO:journalctl@ceph.mds.f.smithi063.stdout:Mar 28 22:23:49 smithi063 podman[73939]: 2024-03-28 22:23:49.621114402 +0000 UTC m=+2.926446799 container died ce5a887a2420436a5c5431b71ea4e726a1a579bf40bce69d678834ae0cef7b8d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-323d23d, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-28T22:23:50.180 INFO:journalctl@ceph.mds.f.smithi063.stdout:Mar 28 22:23:49 smithi063 podman[73939]: 2024-03-28 22:23:49.979709935 +0000 UTC m=+3.285042290 container cleanup ce5a887a2420436a5c5431b71ea4e726a1a579bf40bce69d678834ae0cef7b8d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-f, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=squid-323d23d, GIT_CLEAN=True) 2024-03-28T22:23:50.180 INFO:journalctl@ceph.mds.f.smithi063.stdout:Mar 28 22:23:49 smithi063 bash[73939]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-f 2024-03-28T22:23:50.404 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mds.f.service' 2024-03-28T22:23:50.444 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:23:50.444 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-28T22:23:50.445 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-28T22:23:50.445 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mds.i 2024-03-28T22:23:50.930 INFO:journalctl@ceph.mds.i.smithi063.stdout:Mar 28 22:23:50 smithi063 systemd[1]: Stopping Ceph mds.i for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:23:50.930 INFO:journalctl@ceph.mds.i.smithi063.stdout:Mar 28 22:23:50 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-i[67179]: 2024-03-28T22:23:50.757+0000 7f94f69ba700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:23:50.930 INFO:journalctl@ceph.mds.i.smithi063.stdout:Mar 28 22:23:50 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-i[67179]: 2024-03-28T22:23:50.757+0000 7f94f69ba700 -1 mds.i *** got signal Terminated *** 2024-03-28T22:23:53.883 INFO:journalctl@ceph.mds.i.smithi063.stdout:Mar 28 22:23:53 smithi063 podman[74059]: 2024-03-28 22:23:53.629376654 +0000 UTC m=+2.962020812 container died 4092192cc725d3ddd0ae54b343e545724479d9337e701c096ba10bf06b99086e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-i, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_CLEAN=True, RELEASE=squid-323d23d, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=) 2024-03-28T22:23:54.141 INFO:journalctl@ceph.mds.i.smithi063.stdout:Mar 28 22:23:54 smithi063 podman[74059]: 2024-03-28 22:23:54.124995958 +0000 UTC m=+3.457640087 container cleanup 4092192cc725d3ddd0ae54b343e545724479d9337e701c096ba10bf06b99086e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-i, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, RELEASE=squid-323d23d, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T22:23:54.141 INFO:journalctl@ceph.mds.i.smithi063.stdout:Mar 28 22:23:54 smithi063 bash[74059]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-i 2024-03-28T22:23:54.486 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mds.i.service' 2024-03-28T22:23:54.526 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:23:54.527 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-28T22:23:54.527 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-28T22:23:54.527 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mds.l 2024-03-28T22:23:54.861 INFO:journalctl@ceph.mds.l.smithi063.stdout:Mar 28 22:23:54 smithi063 systemd[1]: Stopping Ceph mds.l for 798d9884-ed4e-11ee-95d0-87774f69a715... 2024-03-28T22:23:55.179 INFO:journalctl@ceph.mds.l.smithi063.stdout:Mar 28 22:23:54 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-l[66239]: 2024-03-28T22:23:54.860+0000 7f90a9683700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:23:55.180 INFO:journalctl@ceph.mds.l.smithi063.stdout:Mar 28 22:23:54 smithi063 ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-l[66239]: 2024-03-28T22:23:54.860+0000 7f90a9683700 -1 mds.l *** got signal Terminated *** 2024-03-28T22:23:58.870 INFO:journalctl@ceph.mds.l.smithi063.stdout:Mar 28 22:23:58 smithi063 podman[74182]: 2024-03-28 22:23:58.577135912 +0000 UTC m=+3.817333088 container died b1246fcb93bfc98de573ce20067cfc91f54cb5dea9a836bf6ac126d8d28a1159 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-l, org.label-schema.build-date=20240326, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-323d23d, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T22:23:59.179 INFO:journalctl@ceph.mds.l.smithi063.stdout:Mar 28 22:23:58 smithi063 podman[74182]: 2024-03-28 22:23:58.953377667 +0000 UTC m=+4.193574827 container cleanup b1246fcb93bfc98de573ce20067cfc91f54cb5dea9a836bf6ac126d8d28a1159 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-l, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-323d23d) 2024-03-28T22:23:59.180 INFO:journalctl@ceph.mds.l.smithi063.stdout:Mar 28 22:23:58 smithi063 bash[74182]: ceph-798d9884-ed4e-11ee-95d0-87774f69a715-mds-l 2024-03-28T22:23:59.440 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-798d9884-ed4e-11ee-95d0-87774f69a715@mds.l.service' 2024-03-28T22:23:59.479 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:23:59.480 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-28T22:23:59.480 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-28T22:23:59.677 INFO:teuthology.orchestra.run.smithi016.stdout:Deleting cluster with fsid: 798d9884-ed4e-11ee-95d0-87774f69a715 2024-03-28T22:24:02.166 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-28T22:24:02.370 INFO:teuthology.orchestra.run.smithi053.stdout:Deleting cluster with fsid: 798d9884-ed4e-11ee-95d0-87774f69a715 2024-03-28T22:24:04.958 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-28T22:24:05.159 INFO:teuthology.orchestra.run.smithi063.stdout:Deleting cluster with fsid: 798d9884-ed4e-11ee-95d0-87774f69a715 2024-03-28T22:24:07.683 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T22:24:07.718 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T22:24:07.757 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T22:24:07.791 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-28T22:24:07.793 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627081/remote/smithi016/crash 2024-03-28T22:24:07.794 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/crash -- . 2024-03-28T22:24:07.846 INFO:teuthology.orchestra.run.smithi016.stderr:tar: /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-28T22:24:07.847 INFO:teuthology.orchestra.run.smithi016.stderr:tar: Error is not recoverable: exiting now 2024-03-28T22:24:07.848 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627081/remote/smithi053/crash 2024-03-28T22:24:07.849 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/crash -- . 2024-03-28T22:24:07.896 INFO:teuthology.orchestra.run.smithi053.stderr:tar: /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-28T22:24:07.896 INFO:teuthology.orchestra.run.smithi053.stderr:tar: Error is not recoverable: exiting now 2024-03-28T22:24:07.898 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627081/remote/smithi063/crash 2024-03-28T22:24:07.898 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/crash -- . 2024-03-28T22:24:07.936 INFO:teuthology.orchestra.run.smithi063.stderr:tar: /var/lib/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-28T22:24:07.936 INFO:teuthology.orchestra.run.smithi063.stderr:tar: Error is not recoverable: exiting now 2024-03-28T22:24:07.937 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-28T22:24:07.937 DEBUG:teuthology.orchestra.run.smithi016:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-28T22:24:07.974 INFO:tasks.cephadm:Compressing logs... 2024-03-28T22:24:07.974 DEBUG:teuthology.orchestra.run.smithi016:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-28T22:24:08.018 DEBUG:teuthology.orchestra.run.smithi053:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-28T22:24:08.021 DEBUG:teuthology.orchestra.run.smithi063:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-28T22:24:08.045 INFO:teuthology.orchestra.run.smithi016.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-28T22:24:08.045 INFO:teuthology.orchestra.run.smithi016.stderr:: No such file or directory 2024-03-28T22:24:08.045 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-osd.6.log 2024-03-28T22:24:08.045 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mon.a.log 2024-03-28T22:24:08.047 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-28T22:24:08.047 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-osd.6.log: /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mds.g.log 2024-03-28T22:24:08.047 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mds.j.log 2024-03-28T22:24:08.047 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-osd.3.log 2024-03-28T22:24:08.049 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mgr.x.log 2024-03-28T22:24:08.049 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-osd.0.log 2024-03-28T22:24:08.050 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mgr.x.log: 90.1% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-28T22:24:08.051 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-28T22:24:08.051 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-osd.0.log: 94.6% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mds.g.log.gz 2024-03-28T22:24:08.054 INFO:teuthology.orchestra.run.smithi063.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-28T22:24:08.054 INFO:teuthology.orchestra.run.smithi063.stderr:: No such file or directory 2024-03-28T22:24:08.054 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mds.l.log 2024-03-28T22:24:08.054 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-28T22:24:08.055 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mds.i.log 2024-03-28T22:24:08.055 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mds.l.log: /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-osd.11.log 2024-03-28T22:24:08.055 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mgr.z.log 2024-03-28T22:24:08.055 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-osd.11.log: gzip -5 --verbose 92.4% -- -- replaced with /var/log/ceph/cephadm.log.gz /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-osd.8.log 2024-03-28T22:24:08.055 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-03-28T22:24:08.055 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-osd.5.log 2024-03-28T22:24:08.055 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mds.f.log 2024-03-28T22:24:08.055 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-28T22:24:08.064 INFO:teuthology.orchestra.run.smithi016.stderr: 94.5% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mds.j.log.gz 2024-03-28T22:24:08.064 INFO:teuthology.orchestra.run.smithi016.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-28T22:24:08.064 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph.log 2024-03-28T22:24:08.064 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-28T22:24:08.064 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph.log: 83.9% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-28T22:24:08.065 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-osd.9.log 2024-03-28T22:24:08.065 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mds.d.log 2024-03-28T22:24:08.065 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-osd.9.log: /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-volume.log: 87.2% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-28T22:24:08.068 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mds.f.log: 90.3% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-28T22:24:08.068 INFO:teuthology.orchestra.run.smithi063.stderr: 94.6% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mds.l.log.gz 2024-03-28T22:24:08.069 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph.log 2024-03-28T22:24:08.069 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-osd.2.log 2024-03-28T22:24:08.069 INFO:teuthology.orchestra.run.smithi063.stderr: 83.6% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-28T22:24:08.069 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mds.c.log 2024-03-28T22:24:08.069 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-osd.2.log: 87.2% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-28T22:24:08.069 INFO:teuthology.orchestra.run.smithi063.stderr: 91.8% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mgr.z.log.gz 2024-03-28T22:24:08.069 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-28T22:24:08.069 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mon.c.log 2024-03-28T22:24:08.070 INFO:teuthology.orchestra.run.smithi053.stderr:find: ‘/var/log/rbd-target-api’gzip -5: No such file or directory --verbose 2024-03-28T22:24:08.070 INFO:teuthology.orchestra.run.smithi053.stderr: -- /var/log/ceph/cephadm.log 2024-03-28T22:24:08.070 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-osd.10.log 2024-03-28T22:24:08.070 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-28T22:24:08.070 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mds.b.log 2024-03-28T22:24:08.071 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-osd.4.log 2024-03-28T22:24:08.071 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-osd.1.log 2024-03-28T22:24:08.071 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mds.b.log: /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mds.k.log 2024-03-28T22:24:08.071 INFO:teuthology.orchestra.run.smithi053.stderr: 92.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-28T22:24:08.071 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mds.h.log 2024-03-28T22:24:08.071 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose --/var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mds.k.log: /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-osd.7.log 2024-03-28T22:24:08.071 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mds.e.log 2024-03-28T22:24:08.071 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-osd.7.log: 90.2% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-28T22:24:08.071 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mgr.y.log 2024-03-28T22:24:08.072 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mds.e.log: 94.5% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mds.k.log.gz 2024-03-28T22:24:08.072 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-28T22:24:08.074 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mds.a.log 2024-03-28T22:24:08.074 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mds.d.log: 91.0% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-28T22:24:08.074 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mds.a.log: 94.5% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mds.d.log.gz 2024-03-28T22:24:08.075 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mgr.y.log: 91.8% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mgr.y.log.gz 2024-03-28T22:24:08.076 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph.log 2024-03-28T22:24:08.076 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph.cephadm.log: 83.6% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-28T22:24:08.076 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-28T22:24:08.077 INFO:teuthology.orchestra.run.smithi016.stderr: 94.5% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mds.a.log.gz 2024-03-28T22:24:08.078 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph.log: 87.2% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-28T22:24:08.078 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mon.b.log 2024-03-28T22:24:08.089 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-volume.log: /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mon.c.log: 90.9% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-28T22:24:08.092 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-volume.log: /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mon.b.log: 91.0% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-28T22:24:08.128 INFO:teuthology.orchestra.run.smithi063.stderr: 94.5% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mds.c.log.gz 2024-03-28T22:24:08.178 INFO:teuthology.orchestra.run.smithi053.stderr: 93.2% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mds.h.log.gz 2024-03-28T22:24:08.210 INFO:teuthology.orchestra.run.smithi063.stderr: 94.4% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mds.i.log.gz 2024-03-28T22:24:08.358 INFO:teuthology.orchestra.run.smithi063.stderr: 92.1% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mon.c.log.gz 2024-03-28T22:24:08.420 INFO:teuthology.orchestra.run.smithi016.stderr: 90.3% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mgr.x.log.gz 2024-03-28T22:24:08.489 INFO:teuthology.orchestra.run.smithi053.stderr: 92.1% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mon.b.log.gz 2024-03-28T22:24:08.987 INFO:teuthology.orchestra.run.smithi016.stderr: 90.6% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mon.a.log.gz 2024-03-28T22:24:09.060 INFO:teuthology.orchestra.run.smithi053.stderr: 93.6% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mds.b.log.gz 2024-03-28T22:24:09.255 INFO:teuthology.orchestra.run.smithi063.stderr: 94.0% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-osd.8.log.gz 2024-03-28T22:24:09.265 INFO:teuthology.orchestra.run.smithi053.stderr: 93.7% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-osd.4.log.gz 2024-03-28T22:24:09.340 INFO:teuthology.orchestra.run.smithi016.stderr: 94.1% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-osd.6.log.gz 2024-03-28T22:24:09.422 INFO:teuthology.orchestra.run.smithi063.stderr: 93.7% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-osd.5.log.gz 2024-03-28T22:24:09.425 INFO:teuthology.orchestra.run.smithi063.stderr: 95.3% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mds.f.log.gz 2024-03-28T22:24:09.448 INFO:teuthology.orchestra.run.smithi016.stderr: 94.0% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-osd.3.log.gz 2024-03-28T22:24:09.462 INFO:teuthology.orchestra.run.smithi016.stderr: 93.9% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-osd.9.log.gz 2024-03-28T22:24:09.492 INFO:teuthology.orchestra.run.smithi063.stderr: 93.8% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-osd.2.log.gz 2024-03-28T22:24:09.693 INFO:teuthology.orchestra.run.smithi063.stderr: 94.2% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-osd.11.log.gz 2024-03-28T22:24:09.695 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-03-28T22:24:09.695 INFO:teuthology.orchestra.run.smithi063.stderr:real 0m1.660s 2024-03-28T22:24:09.695 INFO:teuthology.orchestra.run.smithi063.stderr:user 0m7.326s 2024-03-28T22:24:09.695 INFO:teuthology.orchestra.run.smithi063.stderr:sys 0m0.255s 2024-03-28T22:24:09.714 INFO:teuthology.orchestra.run.smithi016.stderr: 94.1% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-osd.0.log.gz 2024-03-28T22:24:09.716 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-28T22:24:09.716 INFO:teuthology.orchestra.run.smithi016.stderr:real 0m1.685s 2024-03-28T22:24:09.716 INFO:teuthology.orchestra.run.smithi016.stderr:user 0m6.906s 2024-03-28T22:24:09.716 INFO:teuthology.orchestra.run.smithi016.stderr:sys 0m0.249s 2024-03-28T22:24:09.745 INFO:teuthology.orchestra.run.smithi053.stderr: 93.8% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-osd.10.log.gz 2024-03-28T22:24:09.748 INFO:teuthology.orchestra.run.smithi053.stderr: 94.0% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-osd.1.log.gz 2024-03-28T22:24:09.812 INFO:teuthology.orchestra.run.smithi053.stderr: 94.2% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-osd.7.log.gz 2024-03-28T22:24:28.074 INFO:teuthology.orchestra.run.smithi053.stderr: 93.4% -- replaced with /var/log/ceph/798d9884-ed4e-11ee-95d0-87774f69a715/ceph-mds.e.log.gz 2024-03-28T22:24:28.076 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-28T22:24:28.076 INFO:teuthology.orchestra.run.smithi053.stderr:real 0m20.041s 2024-03-28T22:24:28.076 INFO:teuthology.orchestra.run.smithi053.stderr:user 0m27.113s 2024-03-28T22:24:28.076 INFO:teuthology.orchestra.run.smithi053.stderr:sys 0m0.806s 2024-03-28T22:24:28.076 INFO:tasks.cephadm:Archiving logs... 2024-03-28T22:24:28.077 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627081/remote/smithi016/log 2024-03-28T22:24:28.078 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-28T22:24:28.991 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627081/remote/smithi053/log 2024-03-28T22:24:28.992 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-28T22:24:32.607 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627081/remote/smithi063/log 2024-03-28T22:24:32.608 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-28T22:24:33.488 INFO:tasks.cephadm:Removing cluster... 2024-03-28T22:24:33.489 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 --force 2024-03-28T22:24:33.687 INFO:teuthology.orchestra.run.smithi016.stdout:Deleting cluster with fsid: 798d9884-ed4e-11ee-95d0-87774f69a715 2024-03-28T22:24:33.946 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 --force 2024-03-28T22:24:34.141 INFO:teuthology.orchestra.run.smithi053.stdout:Deleting cluster with fsid: 798d9884-ed4e-11ee-95d0-87774f69a715 2024-03-28T22:24:34.433 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 798d9884-ed4e-11ee-95d0-87774f69a715 --force 2024-03-28T22:24:34.629 INFO:teuthology.orchestra.run.smithi063.stdout:Deleting cluster with fsid: 798d9884-ed4e-11ee-95d0-87774f69a715 2024-03-28T22:24:34.900 INFO:tasks.cephadm:Removing cephadm ... 2024-03-28T22:24:34.900 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-28T22:24:34.918 DEBUG:teuthology.orchestra.run.smithi053:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-28T22:24:34.936 DEBUG:teuthology.orchestra.run.smithi063:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-28T22:24:34.951 INFO:tasks.cephadm:Teardown complete 2024-03-28T22:24:34.952 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-28T22:24:34.965 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-28T22:24:34.965 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-28T22:24:34.968 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-28T22:24:34.978 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-28T22:24:35.096 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-28T22:24:35.107 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-28T22:24:35.108 DEBUG:teuthology.orchestra.run.smithi016:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T22:24:35.111 DEBUG:teuthology.orchestra.run.smithi053:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T22:24:35.113 DEBUG:teuthology.orchestra.run.smithi063:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T22:24:35.123 INFO:teuthology.orchestra.run.smithi016.stderr:bash: line 1: ntpq: command not found 2024-03-28T22:24:35.126 INFO:teuthology.orchestra.run.smithi053.stderr:bash: line 1: ntpq: command not found 2024-03-28T22:24:35.127 INFO:teuthology.orchestra.run.smithi063.stderr:bash: line 1: ntpq: command not found 2024-03-28T22:24:35.128 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T22:24:35.128 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2024-03-28T22:24:35.128 INFO:teuthology.orchestra.run.smithi016.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 8 -797us[ -799us] +/- 98ms 2024-03-28T22:24:35.129 INFO:teuthology.orchestra.run.smithi016.stdout:^* hv02.front.sepia.ceph.com 3 8 377 5 -837us[ -839us] +/- 74ms 2024-03-28T22:24:35.129 INFO:teuthology.orchestra.run.smithi016.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 7 +1719us[+1718us] +/- 87ms 2024-03-28T22:24:35.129 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T22:24:35.132 INFO:teuthology.orchestra.run.smithi053.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T22:24:35.133 INFO:teuthology.orchestra.run.smithi053.stdout:=============================================================================== 2024-03-28T22:24:35.133 INFO:teuthology.orchestra.run.smithi053.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 11 -771us[ -770us] +/- 98ms 2024-03-28T22:24:35.133 INFO:teuthology.orchestra.run.smithi053.stdout:^* hv02.front.sepia.ceph.com 3 6 377 9 -835us[ -833us] +/- 74ms 2024-03-28T22:24:35.133 INFO:teuthology.orchestra.run.smithi053.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 10 +1727us[+1728us] +/- 87ms 2024-03-28T22:24:35.133 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T22:24:35.133 INFO:teuthology.orchestra.run.smithi063.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T22:24:35.133 INFO:teuthology.orchestra.run.smithi063.stdout:=============================================================================== 2024-03-28T22:24:35.134 INFO:teuthology.orchestra.run.smithi063.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 6 -782us[ -771us] +/- 98ms 2024-03-28T22:24:35.134 INFO:teuthology.orchestra.run.smithi063.stdout:^* hv02.front.sepia.ceph.com 3 6 377 5 -848us[ -837us] +/- 74ms 2024-03-28T22:24:35.134 INFO:teuthology.orchestra.run.smithi063.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 5 +1707us[+1719us] +/- 87ms 2024-03-28T22:24:35.134 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T22:24:35.134 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-28T22:24:35.145 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-28T22:24:35.146 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-28T22:24:35.157 DEBUG:teuthology.orchestra.run.smithi016:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-28T22:24:35.257 DEBUG:teuthology.orchestra.run.smithi053:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-28T22:24:35.341 DEBUG:teuthology.orchestra.run.smithi063:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-28T22:24:35.421 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T22:24:35.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T22:24:35.453 DEBUG:teuthology.orchestra.run.smithi053:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T22:24:35.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T22:24:35.488 DEBUG:teuthology.orchestra.run.smithi063:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T22:24:35.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T22:24:35.522 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-28T22:24:35.534 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-28T22:24:35.544 INFO:teuthology.task.internal:Duration was 1941.326653 seconds 2024-03-28T22:24:35.545 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-28T22:24:35.556 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-28T22:24:35.556 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-28T22:24:35.559 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-28T22:24:35.561 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-28T22:24:35.600 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T22:24:35.612 INFO:teuthology.orchestra.run.smithi053.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T22:24:35.613 INFO:teuthology.orchestra.run.smithi063.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T22:24:36.075 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-28T22:24:36.075 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi016.front.sepia.ceph.com 2024-03-28T22:24:36.075 DEBUG:teuthology.orchestra.run.smithi016:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-28T22:24:36.096 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi053.front.sepia.ceph.com 2024-03-28T22:24:36.096 DEBUG:teuthology.orchestra.run.smithi053:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-28T22:24:36.120 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi063.front.sepia.ceph.com 2024-03-28T22:24:36.120 DEBUG:teuthology.orchestra.run.smithi063:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-28T22:24:36.144 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-28T22:24:36.144 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-28T22:24:36.147 DEBUG:teuthology.orchestra.run.smithi053:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-28T22:24:36.162 DEBUG:teuthology.orchestra.run.smithi063:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-28T22:24:36.219 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-28T22:24:36.219 DEBUG:teuthology.orchestra.run.smithi016:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-28T22:24:36.222 DEBUG:teuthology.orchestra.run.smithi053:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-28T22:24:36.237 DEBUG:teuthology.orchestra.run.smithi063:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-28T22:24:36.606 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-28T22:24:36.619 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-28T22:24:36.619 DEBUG:teuthology.orchestra.run.smithi016:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-28T22:24:36.646 DEBUG:teuthology.orchestra.run.smithi053:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-28T22:24:36.678 DEBUG:teuthology.orchestra.run.smithi063:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-28T22:24:36.709 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-28T22:24:36.722 DEBUG:teuthology.orchestra.run.smithi016:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-28T22:24:36.725 DEBUG:teuthology.orchestra.run.smithi053:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-28T22:24:36.727 DEBUG:teuthology.orchestra.run.smithi063:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-28T22:24:36.748 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = core 2024-03-28T22:24:36.754 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern = core 2024-03-28T22:24:36.779 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern = core 2024-03-28T22:24:36.831 DEBUG:teuthology.orchestra.run.smithi016:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-28T22:24:36.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T22:24:36.845 DEBUG:teuthology.orchestra.run.smithi053:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-28T22:24:36.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T22:24:36.859 DEBUG:teuthology.orchestra.run.smithi063:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-28T22:24:36.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T22:24:36.886 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-28T22:24:36.899 INFO:teuthology.task.internal:Transferring archived files... 2024-03-28T22:24:36.899 DEBUG:teuthology.misc:Transferring archived files from smithi016:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627081/remote/smithi016 2024-03-28T22:24:36.900 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-28T22:24:36.944 DEBUG:teuthology.misc:Transferring archived files from smithi053:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627081/remote/smithi053 2024-03-28T22:24:36.944 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-28T22:24:36.986 DEBUG:teuthology.misc:Transferring archived files from smithi063:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627081/remote/smithi063 2024-03-28T22:24:36.987 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-28T22:24:37.031 INFO:teuthology.task.internal:Removing archive directory... 2024-03-28T22:24:37.031 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-28T22:24:37.034 DEBUG:teuthology.orchestra.run.smithi053:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-28T22:24:37.036 DEBUG:teuthology.orchestra.run.smithi063:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-28T22:24:37.080 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-28T22:24:37.094 INFO:teuthology.task.internal:Not uploading archives. 2024-03-28T22:24:37.094 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-28T22:24:37.108 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-28T22:24:37.108 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-28T22:24:37.110 DEBUG:teuthology.orchestra.run.smithi053:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-28T22:24:37.113 DEBUG:teuthology.orchestra.run.smithi063:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-28T22:24:37.122 INFO:teuthology.orchestra.run.smithi016.stdout: 265819 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 28 22:24 /home/ubuntu/cephtest 2024-03-28T22:24:37.125 INFO:teuthology.orchestra.run.smithi053.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 28 22:24 /home/ubuntu/cephtest 2024-03-28T22:24:37.134 INFO:teuthology.orchestra.run.smithi063.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 28 22:24 /home/ubuntu/cephtest 2024-03-28T22:24:37.136 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-28T22:24:37.157 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-28T22:24:37.188 INFO:teuthology.nuke:Checking targets against current locks 2024-03-28T22:24:37.213 DEBUG:teuthology.nuke:shortname: smithi016 2024-03-28T22:24:37.213 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T22:24:37.237 DEBUG:teuthology.nuke:shortname: smithi053 2024-03-28T22:24:37.237 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T22:24:37.242 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627081', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 21:43:11.861994', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBALRE86IiYBSvcxgSkbfx534/kRLCWStoyl6wAPDRl4nc3gdufJHPiscJ+bHK493R7gtQKDv2W7wkh+bP78Mi14='} 2024-03-28T22:24:37.273 DEBUG:teuthology.nuke:shortname: smithi063 2024-03-28T22:24:37.273 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T22:24:37.280 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi053.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627081', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 21:43:11.859516', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM44VrSbAkAdZ43rcJvDCkX44DcpMU2oZk2CR5TPqYqHHdYimKvUvbqpfps/UbdzGC+6zQbJKOyiFYYufqvMNu8='} 2024-03-28T22:24:37.293 INFO:teuthology.orchestra.console.smithi016:Power off 2024-03-28T22:24:37.293 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T22:24:37.305 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi063.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627081', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 21:43:11.860884', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEkuF2zVDPcz+eNFgDlEeVqrx7lY4M0fPlPf9F4EV5kLZ4JP9oA3g8gk9DFuASMkR4P2mAXA8O8T1rStkYerqTM='} 2024-03-28T22:24:37.317 DEBUG:teuthology.orchestra.console.smithi016:power off output: Chassis Power Control: Down/Off 2024-03-28T22:24:37.318 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T22:24:37.346 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is on 2024-03-28T22:24:37.376 INFO:teuthology.orchestra.console.smithi063:Power off 2024-03-28T22:24:37.377 DEBUG:teuthology.orchestra.console.smithi063:pexpect command: ipmitool -H smithi063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T22:24:37.399 DEBUG:teuthology.orchestra.console.smithi063:power off output: Chassis Power Control: Down/Off 2024-03-28T22:24:37.400 DEBUG:teuthology.orchestra.console.smithi063:pexpect command: ipmitool -H smithi063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T22:24:37.426 DEBUG:teuthology.orchestra.console.smithi063:check power output: Chassis Power is on 2024-03-28T22:24:38.052 INFO:teuthology.orchestra.console.smithi053:Power off 2024-03-28T22:24:38.052 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T22:24:38.076 DEBUG:teuthology.orchestra.console.smithi053:power off output: Chassis Power Control: Down/Off 2024-03-28T22:24:38.076 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T22:24:38.100 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is on 2024-03-28T22:24:41.347 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T22:24:41.427 DEBUG:teuthology.orchestra.console.smithi063:pexpect command: ipmitool -H smithi063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T22:24:41.458 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is on 2024-03-28T22:24:41.537 DEBUG:teuthology.orchestra.console.smithi063:check power output: Chassis Power is on 2024-03-28T22:24:42.102 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T22:24:42.212 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is on 2024-03-28T22:24:45.463 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T22:24:45.537 DEBUG:teuthology.orchestra.console.smithi063:pexpect command: ipmitool -H smithi063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T22:24:45.574 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is off 2024-03-28T22:24:45.648 DEBUG:teuthology.orchestra.console.smithi063:check power output: Chassis Power is off 2024-03-28T22:24:45.676 INFO:teuthology.orchestra.console.smithi016:Power off completed 2024-03-28T22:24:45.749 INFO:teuthology.orchestra.console.smithi063:Power off completed 2024-03-28T22:24:46.214 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T22:24:46.323 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is on 2024-03-28T22:24:50.328 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T22:24:50.438 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2024-03-28T22:24:50.539 INFO:teuthology.orchestra.console.smithi053:Power off completed 2024-03-28T22:24:50.645 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} duration: 1941.326652765274 failure_reason: 'Command failed (workunit test suites/fsx.sh) on smithi016 with status 2: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=55f5138942aa4866e49329c9103468889cec3f73 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh''' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=7f9a46a26efd4604b4fc653ab336cdc4 status: fail success: false 2024-03-28T22:24:50.646 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-28T22:24:50.741 INFO:teuthology.run:FAIL